Search Results: "cas"

10 May 2010

Stefano Zacchiroli: UDD - consolidating bazaar metadata for QA and data mining

Eclectic paper on the Ultimate Debian Database A few months ago, I've co-authored with Lucas a paper on UDD, which has just been presented at this year IEEE's Mining Software Repository conference, continuing my recent tradition of eclectic papers. The paper is titled The Ultimate Debian Database: Consolidating Bazaar Metadata for Quality Assurance and Data Mining and is available for download from my publications page. For Debian people already familiar with UDD there is probably not much to learn from it, as the main target of the paper is the community of scientists doing data mining on software repositories. For them, UDD offers a valuable entry point to Debian "facts", as data sources reflected in the database are easily joinable together and to some extent already validated by other UDD users (e.g. QA people). Nevertheless the first two sections of the paper are probably of more broad interest. There we have given our point of view on the so called Debian Data Hell: why it exists, how it's related to the nature of Debian and similar distros, etc. I've already noted in the past how that is also related to the culture of freedom that in Debian we value not only in our software, but also in our infrastructure and procedures. We should just get rid of a bit of inertia, and total world domination will then be just around the corner :-) I'm happy to conclude quoting the acknowledgments section of the paper: Acknowledgments
The authors would like to thank all UDD contributors, and in particular: Christian von Essen and Marc Brockschmidt (student and co-mentor in the Google Summer of Code which witnessed the first UDD implementation), Olivier Berger for his support and FLOSSmole contacts, Andreas Tille who contributed several gatherers, the Debian community at large, the "German cabal" and Debian System Administrators for their UDD hosting and support.

9 May 2010

Robert Collins: Maintainable pyunit test suites

There s a test code maintenance issue I ve been grappling with, and watching others grapple with for a while now. I ve blogged about some infrastructural things related to it before, but now I think its time to talk about the problem itself. The problem shows up as soon as you start writing setUp functions, or custom assertThing functions. And the problem is where do you put this code?
If you have a single TestCase, its easy. But as soon as you have two test classes it becomes more difficult. If you choose either class, the other class cannot use your setUp or assertion code. If you create a base class for your tests and put the code there you end up with a huge base class, and every test paying the total overhead of your test needs, rather than just the overhead needed to test the particular system you want to test. Or with a large and growing list of assertions most of which are irrelevant for most tests.
The reason the choices have to be made is because test code is just code; and all the normal issues there separation of concerns, composition often being better than inheritance, do-one-thing-well all apply to our test code. These issues are exacerbated by pyunit (that is the Python unittest module included with the standard library and extended by various projects)
Lets look some (some) of the concerns involved in a test environment: Test execution, fixture management, outcome decision making. I m using slightly abstract terms here because I don t want to bind the discussion down to an existing implementation. However the down side is that I need to define these terms a little.
Test execution by this I mean the basic machinery of running a single test: the test framework calling into user code and receiving back an outcome with details. E.g. in pyunit your test_method() code is called, success is determined by it returning successfully, and other outcomes by raising specific exceptions. Other languages without exceptions might do this returning an outcome object, or passing some object into the user code to be called by the test.
Fixture management the non trivial code that prepares a situation where you can make assertions. On the small side, creating a few object instances and glueing them together, on the large end, loading data into a database (and creating the database instance at the same time). Isolation issues such as masking out environment variables and creating temp directories are included in this category in my opinion.
Outcome decision making possibly the most obtuse label I ve ever given this, I m referring the process of deciding *what* outcome you wish to have happen. This takes different forms depending on your testing framework. For instance, in Python s doctest:
>>> x
45
provides a specification the test framework calls str(x) and then compares that to the string 45 . In pyunit assertions are typically used:
self.assertEqual(45, x)
Will call 45 == x and if the result is not True, raise an exception indicating a Failure has occured. Unexpected exceptions cause Errors, and in the most recent pyunit, and some extensions, other exceptions can signal that a test should not be run, or should have failed.
So, those are the three concerns that we have when testing; where should each be expressed (in pyunit)? Pragmatically the test execution code is the hardest to separate out: Its partly outside of user control , in that the contract is with the test framework. So lets start by saying that this core facility, which we should very rarely need to change, should be in TestCase.
That leaves fixture management and outcome decision making. Lets tackle decision making if you consider the earlier doctest and assertion examples, I think its fairly clear that there are multiple discrete components at play. Two in particular I d like to highlight are: matching and signalling. In the doctest example the matching is done by string matching the reference object(s) are stringified and compared to an example the test writer provides. In the pyunit example the matching is done by the __eq__ protocol. The signalling in the doctest example is done inside the test framework (so we don t see any evidence of it at all). In the pyunit example the signalling is done by the assertion method calling self.fail(), that being the defined contract for causing a failure. Now for a more complex example: testing a float. In doctest:
>>> %0.3f % x
0.123
In pyunit:
self.assertAlmostEqual(0.123, x, places=3)
This very simple check that a floating point number is effectively 0.123 exposes two problems immediately. The first, in doctest, is that literal string comparisons are extremely limited. A regex or other language would be much more powerful (and there are some extensions to doctest; the point remains though the operator is not enough). The second problem is in pyunit. It is that the contract of assertEqual and assertAlmostEqual are different: you cannot substitute one in where the other was expected without partial function application something that while powerful is not the most obvious thing to reach for, or to read in code. The JUnit folk came up with a nice way to address this: they decoupled /matching/ and /deciding/ with a new assertion called assertThat and a language for matching expressed as classes. The initial matcher library, hamcrest, is pretty ugly in Python; I don t use it because it tries too hard to be english like rather than being honest about being code. (Aside, what would is_() in a python library mean to you? Unless you ve read the hamcrest code, or are not a Python programmer, you ll probably get it wrong. However the concept is totally sound. So, outcome decision making should be done by using a matching language totally seperate from testing, and a small bit of glue for your test framework. In testtools that glue is assertThat , and the matching language is a narrow Matcher contract (in testtools.matchers) which I m going to describe here, in case you cannot or don t want to use the testtools one.
class Matcher:
    def __str__(self):
        "Describe this matcher."""
    def match(self, something):
        """Determine if something is matched.
        :param something: Something to match.
        :return: None if something matched, or a Mismatch object otherwise.
        """
class Mismatch:
    def describe(self):
        """Describe a mismatch that has occured."""
This permits composition and inheritance within your matching code in a pretty clean way. Using == only permits this if you can simultaneously define an __eq__ for your objects that matches with arbitrarily sensitivity (e.g. you might not want to be examining the process_id value for a process a test ran, but do want to check other fields).
Now for fixture management. This one is pretty simple really: stop using setUp (and other similar on-TestCase methods). If you use them, you will end up with a hierarchy like this:
BaseTestCase1
 +TestCase1
 +TestCase2
 +BaseTestCase2
   +TestCase3
   +TestCase4
   +BaseTestCase3
     +TestCase5
     ...
That is, you ll have a tree of base classes, and hanging off them actual test cases. Instead, write on your base TestCase a single glue method e.g.
def useFixture(self, fixture):
      fixture.setUp()
      self.addCleanup(fixture.tearDown)
      return fixture
And then rather than having a setUp function which performs complex operations, define a fixture an object with a setUp and a tearDown method. Use this in tests that need that code::
def test_foo(self):
      server = self.useFixture(NewServerWithUsers())
      self.assertThat(server, HasUser('fred'))
Note that there are some things around that offer this sort of convention already: thats all it is convention. Pick one, and run with it. But please don t use setUp; it was a conflated idea in the first place and is a concrete problem. Something like testresources or testscenarios may fit your needs if it does, great! However they are not the last word they aren t convenient enough to replace just calling a simple helper like I ve presented here.
To conclude, the short story is:
  • use assertThat and have a seperate hierarchy of composable matchers
  • use or create a fixture/resouce framework rather than setUp/tearDown
  • any old TestCase that has the outcomes you want should do at this point (but I love testtools).

6 May 2010

Marco d'Itri: Ubuntu still bad at contributing back their changes

Not just to Debian (we are mostly used to it now), but also to upstream maintainers. Is it so hard? Case study: whois. Discuss. In related news, I just uploaded a whois release which supports the new IDN ccTLDs. I only had to add the respective servers to the list since I had already added IDN support in 2003.

Ian Wienand: How much slower is cross compiling to your own host?

The usual case for cross-compiling is that your target is so woefully slow and under-powered that you would be insane to do anything else. However, sometimes for one of the best reasons of all, "historical reasons", you might ship a 64-bit product but support building on 32-bit hosts, and thus cross-compile even on a very fast architecture like x86. How much does this cost, even though almost everyone is running the 32-bit cross-compiler on a modern 64-bit machine? To test, I got a a 32-bit cross and a 64-bit native x86_64 compiler and toolchain; in this case based on gcc-4.1.2 and binutils 2.17. I then did a allyesconfig build of Linux 2.6.33 x86_64 kernel 3 times using the cross compilier toolchain and then native one. The results (in seconds):
32-bit64-bit
60905684
60505616
60635652
average
60675650
So, all up, ~7% less by building your 64-bit code on a 64-bit machine with a 32-bit cross-compiler.

4 May 2010

Wouter Verhelst: On clue and opinions

How does one define one's "national identity"? One way is to look at where a person is born. By that definion, I could be called Flemish, Belgian, or European, whatever you think about it. Another way to look at it is how one feels about one's own identity. This, of course, is a much more accurate way to describe a person; and by that definition, I am either Belgian or European. I do not feel any affinity with much of the so-called "flemish" identity, nor do I wish to be associated with it any more than is strictly necessary. Obviously, some people feel different about that for themselves. This I can understand. In fact, the statement to feel more Belgian than Flemish is probably not one that is very common in this general area. I am not stupid; I know that Belgium is not a Nation (it is a state, but that is a different matter). I just do not feel a lot of warm fuzzy feelings about Flanders. Naturally, other people have other feelings about that. This, too, I can understand. The logical result of the above is that my personal opinions about the somewhat convoluted politics in this country are not shared by many; that when I state my opinion, I may expect opposition, or at the very least someone stating why they disagree with me. This, of course, I can also understand. But what I cannot understand is that every time I state my opinion, someone must come out and insult my intelligence by telling me I "do not have a clue". Please do not mistake my disagreement for misunderstanding. Martijn asks me to "check the facts". Let's do that, shall we?
  1. There's been a judgement by the constitutional court that the current situation regarding which party may run for election in the BHV region is unconstitutional.
  2. One group of politicians wishes to resolve this issue by splitting the BHV region in two. That would certainly solve the issue.
  3. Another group of politicians dislikes this solution. They may be willing to go along, but they want other concessions before they will do so. This is what I referred to when I said that there were "a huge heap of all kinds of side issues" in my previous post.
  4. The first group does not wish to make such concessions. Some of their numbers have even stated in the press that there is no need to make them, because, given the first point, there's only one solution and that is to split.
That last point is, of course, complete and utter bullshit. It would hold merit if there was only one way to solve a constitutional crisis; but by definition, there are always at least two ways: one is to change the law to comply with the constitution, the other is to change the constitution to make the law no longer be unconstitutional. So a group that feels strongly about one of these two possible solutions should be prepared to negotiate with the other group so as to hammer out an agreement that benefits both parties. In some cases, this will mean doing things that they do not completely like; but if they're not willing to do that, at all, they have no business being politicians. For the longest time during the early negotiations, however, both parties have stubbornly refused to even acknowledge the other position, let alone talk about it. Now, for clarity, I'm not advocating that the latter of the two options in that previous paragraph is what should happen. Frankly, I don't even care what happens, so long as they manage to resolve the issue at hand in some way. What I do care about, however, is that this country has been in a near-constant state of stalemate over these past few years for an issue that only matters to politicians; I don't care who or what a group of people in a (to me) obscure part of the country can vote for. This country needs a stable government which can act on things that actually matter. Like, say, the international economic crisis of the past two years; I think it matters much more whether the man in the street has a job than whether John Q. Politician will past these next elections.

MJ Ray: Day Against DRM, 4 May 2010

Today is the Day Against DRM, working against Digital Restrictions Management or Technological Protection Measures (TPM). I ve two tips to help with our public disservice British Broadcasting Corporation which has consistently failed to act in the public interest and reject DRM:
  1. If you want to link to a BBC radio show like Global Business, you can often get mp3 and avoid Flash-forcing DRM-wannabe iPlayer (and cool toys like get_iplayer) by using BBC Mobile s Podcasts Index.
  2. Satellite TV broadcasts can be captured as MP2 by MythTV or even 50-quid black box home entertainment devices. The DRM-free file can be played back on a computer with mplayer -demuxer +mpegts filename.dvr or re-encoded for other devices with mencoder.
What DRM-avoiding tips would you give?

3 May 2010

Matthew Garrett: PCI power management problems

I've previously written about some of the implementation details of runtime PCI power management. One of the key aspects for PCI devices is that the PME line on the device be able to generate a wakeup event. Unfortunately, it turns out that since Windows makes no use of this functionality at present, some vendors have started failing to wire this up. This is problematic because the device itself still announces PME support and will even raise a PME signal when it gets woken up - but it's pretty much screaming as hard as it can and nobody's listening because somebody's replaced the air with cheese.

Bother, etc.

The obvious question is what to do next. We could poll all the PCI devices every second or so to see if there's a PME status register that's been set. This would be reasonably cheap (we're going to wake up at least once a second on x86 anyway, so it makes no real difference to power consumption) but would introduce obvious latency in the wakeup path. This would be fine for certain types of situation (you're probably not going to be too sad if your SD reader takes a second longer to notice a card insertion) but not others (if an sdio wireless card generates a wakeup interrupt then we really ought to do something about it in a more sensible timeframe). A second is to force users to pass a boottime argument to enable this at all. That kind of sucks.

The third possibility is limited to a subset of hardware, but does allow the introduction of some kind of elegance into what would otherwise be one of those nightmarish scenarios that makes me consider taking up farming instead. If we can trigger a PME ourselves then we can test whether the line is connected. This doesn't appear to be possible on SDHCI but the spec for firewire makes it look like we can do it there - one of the interrupt sources is the port enable/disable register, and we can hit toggle that ourselves. I haven't actually tested this yet, but if it works that would let us make this determination.

It's depressing that doing anything interesting with power management is still heavily determined by what Microsoft have bothered to implement - to the extent that on some machines (hello, Thinkpads) there are no GPE methods at all for PME signals and you don't get runtime power management at all. The only thing that saves us is that (a) it's pretty hard to be able to screw up stuff that's already all glued into one chip package, so integrated stuff like USB should be fine anyway, and (b) PCIe does this as part of the normal traffic stream so there's no way to get that wrong. Unless you're still missing the GPE methods for the signals (hello, Thinkpads) in which case you get nothing unless the native PCIe signalling works. I suspect that we can always force that, so there's some hope yet.

In summary, then: dispassionate.

Lucas Nussbaum: But why isn t Debian using Launchpad?

Due to my work on Ultimate Debian Database, I m sometimes asked why Debian isn t using Launchpad. First, using Launchpad can mean two different things. The first possibility is using Canonical s instance of Launchpad. That would mean creating a Debian project on launchpad.net, and using Canonical s infrastructure. Well, that s clearly not a good idea. We (Debian) do not want to depend on Canonical to fix bugs or make enhancements that we require to improve Debian. For example, Canonical imposes the use of the Bazaar Version Control System in Launchpad: you simply can t use Git instead (git-bzr hacks don t count). We want to stay in control of our infrastructure, for obvious reasons. So the other possibility would be to setup our own instance of Launchpad, given that Launchpad is now Free Software. However, it is not clear if it is actually possible: I was told by a Launchpad developer that they didn t know of any external (outside Canonical) installation of Launchpad. Even if this was possible, it is not clear at all that the Ubuntu infrastructure is superior to the Debian infrastructure. The Debian infrastructure has many nice features that are missing in Launchpad, for example version-tracking in the Bug Tracking System, which allows to track (by parsing the changelog) the versions of a package where a bug has been fixed or not (example with iceweasel). So, when switching to Launchpad, we would have to reimplement quite a lot of needed features in in, with no clear benefit: the Debian infrastructure works fine, and is actively maintained. It s also interesting to note that while the Ubuntu infrastructure is centered on Launchpad, there are quite a lot of external services that are not integrated in Launchpad: the Ubuntu popularity contest, merges.u.c, patches.u.c, the various services on qa.ubuntuwire.org and qa.u.c, etc. With the Debian model, it is very easy to add a new service and get it integrated with the various dashboard (Debian Developer s Packages Overview, Packages Tracking System). Within Ubuntu, those external services are really second-class citizens. All in all, the infrastructures of the two projects reflect their organizations: bazaar model for Debian, with an emphasis on collaboration between the services, controlled by Canonical for Ubuntu.

Lucas Nussbaum: Castles in the air

When people ask me why I m so fond of computers , I already answer that it s like being able to build very large buildings or bridges, super-fast cars, etc without having to care about buying the raw material needed, or about the consequences: computers are like a giant Lego box which is only limited by the programmer s abilities. What I discovered today is that there s a quote from Frederick P. Brooks that says about the same thing:
The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds his castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures.

David Welton: Mochiweb proto-deb and hacking

I had some time over the weekend to play a bit with Mochiweb. It's a very nice little bit of work, but kind of strange for an open source project. I think both things come from the fact that it was built to do some very specific things in a business, and once those goals were accomplished, that suited the owner just fine. There is no nice/fancy project home page, no links to documentation, and in general, all of the stuff you're supposed to do for a successful open source project. The code, however, is good, fast, and pretty easy to deal with. Anyway, I wanted to use it for a thing or two I'm working on, so I created some .deb packaging for it. It's been ages since I've done that (indeed, I officially retired from Debian), so it's probably missing something, but I thought I'd throw it out there for people to have a look. The git repository is here: http://github.com/davidw/mochiweb - my changes live in the 'deb' branch. The big thing I've tried to do is make it so that if you have several mochiweb projects on the same machine, they'll all be started/stopped by the init scripts, if you so choose. What you do is add a file with the path to the application to the /etc/mochiweb/apps directory, and when you do /etc/init.d/mochiweb start, it will start all of those. At stop time, it will run the stop.sh script, which shuts things down. It runs things as the user that owns the start.sh script, unless it's root, in which case it simply doesn't run. The thing that's missing right now is a way to get the thing to write logs somewhere, and I'm not quite sure how to do that, so here's hoping someone will write in with a suggestion. The .deb file I generated ought to be attached to this post, although there are no guarantees that I'll keep it up to date.

Christine Spang: Christine Spang: in case it wasn't already obvious

Mode of transportation has not yet been determined. Possibly bike, possibly train, possibly bus. It feels strange not needing to buy plane tickets. I should be there for all of the conference proper but not for DebCamp.

2 May 2010

Dominique Dumont: Improve config file upgrades Ep 01

Hello The GSoC project Krzysztof and I are working aims to facilitate configuration upgrade during package upgrades. Most of the cases, this upgrade should not require any input from the user, even if the user had previously customized his configuration file. Currently configuration upgrade may ask questions to the user which may be hard to answer for most non-technical users. For more details see PackageConfigUpgrade on Debian wiki. (If you are not familiar with upgrade issues, you might want to read this page.) At the 2 extremes of upgrade tools, we have: Each tool has its own advantages and drawbacks. On the plus side of diff3 tools : On the other hand, here are the issues with diff3 based tools: For more details on diff3 issues, see Benjamin Pierce s paper (thanks to David Lutterkort from Augeas for the info) On the other end of the spectrum, Config::Model pros are: But, Config::Model main problem is: diff3 tools is lackluster for users. While Config::Model puts too much strain on packagers because it requires configuration models where all parameters are defined and declared in the model. For fast evolving projects, this can be a huge recurrent task (been there, done that for Xorg) . (Well, this recurrent task is a problem until doc, implementation and config model can be extracted from a single meta-description But I digress, this may be a project for GSoC 2011 :-p ) So, this GSoC project (for 2010) aims to find a compromise between these 2 extreme cases. On Config::Model, I ve always had a drawer statement whenever I was asked why Config::Model does not handle unknown parameter:
a configuration file cannot be validated if an unknown parameter can t be distinguished from a user typo.
This stance is fine for casual user interaction on a stable system: an unknown parameter is most likely an error. And the user must be warned of his error as soon as possible. For a stable distro release, packagers should have the time to provide complete models, so basing upgrade on complete models should be possible (famous last words ?). On the other hand, a lot of configuration upgrades will be done on unstable systems where software evolve quite fast, where models will likely fall behind software evolutions. Another shorcoming of my drawer statement is that it assumes user interaction where erroneous input is more likely. In case of configuration upgrades, there should be no user interaction. And we can assume that the config file to be upgraded was used (and thus validated) by the old version of the software. So unknown parameters should be tolerated and managed as best as possible. Now, the big question is: how ? In all attempts I ve seen in managing upgrades that goes beyond diff3 tools, the configuration is stored as structured data. Sometimes this structure can be infered from the configuration file syntax (shell vars, INI files, YAML, XML), sometimes not (sshd config file). In the latter case, some structural knowledge must be built in the upgrade tool. Currently, Config::Model is designed to bring a complete description of the structure and semantics of a configuration file. This is perceived as bringing too much structural and semantic knowledge ( too semantic ). So the maintenance cost is perceived as too important. That may be right or wrong, but that does not matter. Perception is more important here. So, let s figure out a way to reduce the amount of work required to perform upgrades (you may want to read a little bit more on configuration model creation before going on ) One possibility would be to support configuration models that allow unknown leaves in nodes. Such a model would provide a kind of fallback leaf description. Something like: this parameter is unknown, let s assume it s a simple string value . Let s consider sshd_config example to see how such a behavior would pan out for some upgrade scenarios. sshd model is made of 2 main classes:
- Sshd
- SshdMatchElement The second one contains all parameters allowed in a Match class (in fact, Match parameter is the only one in sshd that defines a new hierarchical level. All other parameters are simple leaves). Now let s consider typical upgrade scenarios from old version (with old models) to a more modern version. Let s assume that the model is lagging behind and the model does not follow sshd evolutions. So Config::Model will face 3 upgrade cases (extracted from sshd config history):
  1. Unexpected parameter that are actually simple leaf parameters. E.g. introduction of ClientAliveInterval.
  2. Obsolete parameters that should be migrated. E.g KeepAlive is renamed into TCPKeepAlive
  3. New structural parameters (introduction of Match element)
So what would happen with a model allowing unknown parameters but lacking specific instructions for these upgrades:
  1. ClientAliveInterval value would be kept. :-)
  2. KeepAlive would be kept as is.TCPKeepAlive would also be kept. No migration is performed, but the configuration would be good. As long as sshd supports the old parameter, we re good. :-/
  3. Without knowledge of the structure brought by Match parameters, Config::Model would consider that parameter in Match block would duplicate parameters in the main block and remove the duplicate. User data may be lost. :-(
So, there s not silver bullet. Let s resume what we ve learnt: Next step is to figure out what would happen if the same strategy is applied to other types of configuration files. But that will be for another blog. All the best

Dominique Dumont: Config-Model related project for GSoC 2010

Hello It s official: I m going to mentor a project related to config-model (without forgetting Augeas which might also be involved) this year:

Content-aware Config Files Upgrading
by Krzysztof Tyszecki, mentored by Dominique Dumont When a package deliver configuration files, the problem of merging user data with new configuration instructions will arise during package upgrades on users systems. Sometimes merging can be done with 3 way merge, but this process does not insure that the resulting file is correct or even legal. This project intends to create standards,
tools and heuristics to make the scary config file conflict resolution debconf prompt a thing of the past.
This project is hosted by Debian organization. Since this project might interest a lot of people, I will (try to) blog regularly on progress, issues, ideas.

From you people, I d like feedback and more importantly, I d like to get real life use cases.

What are the configuration upgrade scenario that bothered you ? What was your configuration file ? What was the setup done by package maintainer ? What result did you get ? What result did you expected to get ?

All the best


Julien Blache: forked-daapd: now with AirTunes v2 streaming

I ve been working on this for a few months now; it s taken quite some time because I wanted to do it right and deliver something better than what already existed in the OpenSource world as far as that particular technology was (not) concerned. So here it is:

AirTunes v2 UDP streaming forked-daapd can now do streaming to multiple AirTunes devices (as well as the local soundcard) and can be controlled from Remote (and, ideally, from anything that speaks DACP). I m releasing this early while the paint is still wet and not everything works on the DACP-side of things. Case in point, while playing a song or an album is supported, playing a playlist isn t. There s some work needed on this particular feature, and it ll get there eventually. There have been a couple of other changes in the codebase in recent weeks too; details after the jump. Implementing AirTunes v2 has been quite fun and, in addition to releasing the forked-daapd code today, I ve also taken the time to write down my findings about the protocol so others can experiment with it. This documentation is by no means exhaustive nor complete as far as the protocol goes; if you find out anything I haven t, let me know so I can update this document. Let s review the recent changes in the forked-daapd codebase over the past few weeks. AirTunes v2 You can now use Remote from your iSomething together with your AirTunes devices and forked-daapd to listen to your library. As mentioned above, playing a playlist is not supported; I have a few things to sort out before this can work and I suspect this will bring changes not only to the DACP interface but also to other parts of the code. So, stay tuned. forked-daapd supports synchronized playback to multiple AirTunes devices together with the local soundcard if so desired. You can expect the AirTunes devices to be in sync with each other, but the soundcard will usually run in circles around them. It looks like AirTunes plays sound a bit slower than 44.1 kHz; not a big deal, though, as AirTunes will catch up periodically and the desync, while it can be noticeable, isn t big (ie we re not talking seconds, not even one full second). Make sure your network can handle the load, as streaming to a single AirTunes device requires in excess of 1.6 Mbps (1.8 Mbps is actually closer to the real thing). Local audio output is supported via ALSA and OSS4, although note that the OSS4 code hasn t been very well tested. I intend to test and debug it fully at a later date, but feel free to beat me to it and send patches. Also note that, while there is code for FreeBSD in the player/streaming part of forked-daapd, it doesn t work well at all. The streaming code needs a timer with a higher resolution than kqueue/kevent has to offer, and I haven t found anything as capable as timerfd. So it s there, but it s broken. Read on about FreeBSD support. Metadata scanning Several fixes and enhancements have been made to the filescanner so it picks more and better metadata. Format-specific tagnames have been added to work around ffmpeg s limitations. Using a recent enough ffmpeg version (anything after mid-december 2009) will work best, especially for MP3 files. SQLite concurrency fixes The database code has been reworked to handle concurrency on the database. Issues with locked tables (or the whole database) were noticed pretty easily with big libraries when accessing forked-daapd during the library update after startup. On uniprocessor machines, this was even easier to hit. This, however, comes with new requirements for SQLite3: you ll now need SQLite3 v3.5.0 at least and it must be built with the unlock notify API enabled. Not all distributions ship SQLite3 with the unlock notify API enabled, so you may need to do a rebuild. Debian, for instance, doesn t; this is #579266. A word about FreeBSD support I m giving up on FreeBSD support. The streaming code doesn t work on FreeBSD due to timer issues, and I don t intend to fix it. I also don t intend to invest much time in porting future features to FreeBSD; if I can make it work, then fine, otherwise, too bad. Someone needs to step up if there s interest in having forked-daapd work on FreeBSD.

1 May 2010

Sean Finney: Nouveau, Gallium3D, and Compiz on Debian

update: 20100503: It's been brought to my attention that there might have been a regression in one of the libraries, mesa, maybe, so that master might not be as reliable a reference point for this guide. so, just in case, here are the commit hashes that were used from the different projects: for the non-git-savvy: git checkout -b mybranch <hash> we now return to your regular broadcasting... A week or two ago I figured it was about time that I took the plung and tried out the latest nouveau drivers from experimental on my laptop. Unfortunately, it meant bidding adieu to compiz (apart from the bling it does actually provide some pretty damned useful features). I thought I'd give it a shot at least for a while on principal, since I've always had to think of the nvidia drivers as a necessary evil, and their lack of proper xrandr support is really, really annoying. Since then I haven't noticed any problems, apart from the following message after every suspend/resume: 
[  931.213239] Uhhuh. NMI received for unknown reason 80 on CPU 0.
[  931.213240] You have some hardware problem, likely on the PCI bus.
[  931.213242] Dazed and confused, but trying to continue
No idea what's causing it, but otherwise the system is stable and functional. If anyone knows what can be done about this, I'd appreciate hearing from you. My graphics card, by the way: 
01:00.0 VGA compatible controller: nVidia Corporation G72M [Quadro NVS 110M/GeForce Go 7300] (rev a1)
In the time since then, the nouveau driver has made its way into unstable, and a lot of distros (ubuntu, fedora, ...) are even using it as the default driver. So before going any further with the 3d/compiz stuff, I highly recommend that you get the packaged version working first. This will not only make sure that the basic driver works, but it will also make recovery/rollback a lot easier. Since it's all packaged in the standard repositories, it's not much harder than installing a few packages and editing a few files... so come on give it a try :) Getting the 2d driver working in Debian sid Really, this is so easy it's hardly worth writing up, but i'm running with the -v flag this morning... Make sure you don't have any reference to nvidia in /etc/modules or other places that might cause it to be automatically loaded. Then remove the proprietary glx bindings, which would otherwise interfere with the nouveau driver. 
sudo apt-get remove nvidia-glx
Next, you'll need to install a kernel that includes updated drm support. Normally this would require a kernel >= 2.6.33.1, but the debian kernel maintainers have graciously backported this code to the 2.6.32-4 kernel package. note this isn't the default kernel version in testing/unstable at the time of writing so you have to explicitly install it along with the nouveau driver. 
sudo apt-get install linux-image-2.6.32-4-amd64 xserver-xorg-video-nouveau
If you are already running this kernel and using the proprietary nvidia drivers, via module-assistant or otherwise, you /might/ need to uninstall them first. On my system I left them installed in the old kernel version but did not install them on the new kernel version. Required xorg configuration I don't believe that nouveau is picked by default if available at the time of writing, so you have to have the following in your xorg.conf: 
sudo sh -c "cat > /etc/X11/xorg.conf" << EOF
Section "Device"
    Identifier     "nVidia Corporation G72M [Quadro NVS 110M/GeForce Go 7300]"
    Driver         "nouveau"
EndSection
EOF
This was all I needed to do, apart from rebooting into the new kernel. For those who don't want to venture outside the packaging system, this is about as far as you can go for the time being. Though really, if you're bothering to read this far along, you're probably more interested in what comes next :) Getting 3d working (without voiding your distro warranty) So I thought I'd give the 3d-accelerated gallium a go. Their wiki page certainly makes one think twice, but at this point I was seriously fiending to get my bling back (well, the compiz scale plugin, anyway) and figured it wouldn't hurt too much to try. Via google and trawling around in various forums I found a number of how-to documents for getting Gallium3d on debian/ubuntu systems, but most of them recommended doing things in a way that wasn't easily reversible and that could totally hose your system in a not-so-hypothetical worst-case scenario. It's pretty hard to roll back a make install done into /usr, for example, and subsequent package upgrades can leave your system even worse off as they will probably clobber half of what you just did, leaving the other half totally broken. And if you have a laptop which requires the gui for networking (i.e. network-manager), then fixing/rolling-back the changes is not entirely trivial, since you'd likely have to re-download the various drm/mesa/xorg packages. So enter the tried and true unix utility stow (apt-get install stow, of course). stow is a utility that lets you install software into arbitrary folders, and then manage the system-wide installation of the software via a farm of symlinks pointing at the installed location. It's probably more well known among the sysadmin crowd than the desktop user crowd, but it's incredibly useful for the task at hand as it will let us easily install multiple copies of different software in a way that's very easy to update and/or roll-back. RTFM stow(1) for more information. So, on to the juicy stuff... Install needed software, remove problematic packages, setup, etc At least on my system, I had to remove libdrm-dev to keep things from accidentally linking against it instead of the updated version that we install later on below. There's probably a way to override this with CFLAGS but I figure this is simpler. 
sudo apt-get remove libdrm-dev
sudo apt-get install stow build-essential xorg-dev git-core libtool mesa-common-dev automake autoconf 
mkdir ~/nouveau
Compile a linux-2.6.34 RC kernel 
cd ~/nouveau
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
cd linux-2.6
git remote add -f nouveau git://anongit.freedesktop.org/nouveau/linux-2.6
git checkout -b nouveau-master nouveau/master
make menuconfig
Enable device drivers -> staging drivers -> nouveau. Make any other changes you might want to make (I just took defaults for everything else). 
make deb-pkg
The linux-firmware-free debian package ships some of the same stuff from the generated linux-firmware-image package. I decided to opt for the upstream firmware, though I'm not sure that this is necessary: 
sudo apt-get remove linux-firmware-free
sudo dpkg -i ~/nouveau/linux-image*.deb ~/nouveau/linux-firmware-image*.deb
sudo update-grub
Install a copy of dri2proto 
cd ~/nouveau
git clone git://anongit.freedesktop.org/xorg/proto/dri2proto
cd dri2proto
./autogen.sh --prefix=/usr/local/stow/dri2proto-20100428
sudo mkdir -p /usr/local/stow/dri2proto-20100428
sudo make install
sudo stow -d /usr/local/stow dri2proto-20100428
Install a cutting-edge version of libdrm 
cd ~/nouveau
git clone git://anongit.freedesktop.org/git/mesa/drm/
cd drm
./autogen.sh --prefix=/usr/local/stow/drm-20100428 --disable-intel --disable-radeon --enable-nouveau-experimental-api
make
sudo mkdir -p /usr/local/stow/drm-20100428
sudo make install
sudo stow  -d /usr/local/stow  drm-20100428
Likewise for mesa 
cd ~/nouveau
git clone git://anongit.freedesktop.org/git/mesa/mesa
cd mesa
./autogen.sh --prefix=/usr/local/stow/mesa-20100428  --enable-debug --enable-glx-tls --disable-asm --with-dri-drivers= --enable-gallium-nouveau --disable-gallium-intel --disable-gallium-radeon --disable-gallium-svga --with-state-trackers=glx,dri --with-demos=xdemos,demos,trivial,tests
make
sudo mkdir -p /usr/local/stow/mesa-20100428 
sudo make install
sudo stow -d /usr/local/stow mesa-20100428
And now the xorg driver... 
cd ~/nouveau
git clone git://anongit.freedesktop.org/git/nouveau/xf86-video-nouveau/
cd xf86-video-nouveau
./autogen.sh --prefix=/usr/local/stow/xf86-video-nouveau-20100428
make
sudo mkdir -p /usr/local/stow/xf86-video-nouveau-20100428
sudo make install
sudo stow -d /usr/local/stow xf86-video-nouveau-20100428
Configuration Configure xorg to look in alternate location for xorg modules and set up DRI 
sudo sh -c "cat > /etc/X11/xorg.conf" << EOF
Section "Files"
    ModulePath "/usr/local/lib/xorg/modules,/usr/lib/xorg/modules"
EndSection
Section "ServerFlags"
    Option "GlxVisuals" "all"
Endsection
Section "Device"
    Identifier     "nVidia Corporation G72M [Quadro NVS 110M/GeForce Go 7300]"
    Driver         "nouveau"
EndSection
EOF
I've /heard/ that disabling TV out is a necessity on some cards, though I haven't seen this myself: 
sudo sh -c "cat > /etc/modprobe.d/nouveau.conf" << EOF
# not entirely sure this is necessary, but heard it helped with vsync
options nouveau tv_disable=1
EOF
Finally, the one thing that can not be done in /usr/local is making the nouveau DRI driver available. From what I could tell the path that mesa uses to find the DRI libraries is hard-coded (corrections welcome). Since you won't already have this file (otherwise, why would you be following along here?), just drop in a symlink to the file in /usr/local: 
sudo ln -s /usr/local/lib/dri/nouveau_dri.so /usr/lib/dri/
Finally, reboot, and hope for the best :) Riding the bleeding edge: how to upgrade later on Again, stow really shows its usefullness here. Using drm as an example case, upgrading an individual component would look something like: 
cd ~/nouveau/drm
git clean -dfx
git pull
today= date +%Y%m%d 
./autogen.sh --prefix=/usr/local/stow/drm-$today --disable-intel --disable-radeon --enable-nouveau-experimental-api
make
sudo mkdir -p /usr/local/stow/drm-$today
sudo make install
sudo stow -d /usr/local/stow -D drm-20100428
sudo stow -d /usr/local/stow  drm-$today
Note how the old version is still installed, so you can switch components around pretty freely. Abort! Abort!: How to roll back changes In case your system becomes totally unusable, or you've decided that you just want to go back to the safe and cozy world of what's provided by the debian packages, simply do the following: Assuming that you're using the nouveau driver from unstable, this should be all that is necessary. Conclusion Overall impression I am very, very happy with the resulting setup. I was led to believe that this whole thing was much more unstable and unusable than it actually is! While there are certainly some glitches and missing features, I feel comfortable enough using it that I don't have plans to switch back. But if I find myself in a situation where I can't use the new setup, rolling back shouldn't be a problem either. I figure I'm happy with what I have now and it will only get better. What works, at least to a "kinda works" level. What still doesn't work Further reading, references Thanks A couple shoutouts to those who helped me with this on the way: Feedback / Corrections Unforunately, I've had to disable comments for the time being due to overwhelming quantities of spam. So please drop me a line via email or on IRC (seanius on oftc/freenode).

30 April 2010

Biella Coleman: Baby Panda wants YOU to submit to DEBCONF



bad mood destroyer, originally uploaded by the biella.
People s the deadline is fast approaching. Don t worry though, the best ideas come under deadlines and your proposal need not be lengthy. So get those typing hands out and submit a proposal (a few sentences suffices) for a presentation! The annual Debian conference is being held in NYC this year. First time
setting foot in the imperial empire. Are you doing something related to
free software that might be interesting to share, show off or talk about
at Debconf? I bet you are! We are looking for art installations,
performances, movie screenings, maybe you are wanting to host a
roundtable/panel discussion, coordinate a workshop (you dont have to
know the subject deeply to do this!). I know the deadline is soon, but don t let that scare you. The
submission requirement bar is very low, and we are flexible! here is the submission information: [1] ********************* Calling all potential contributors to DebConf10! One more week until
the final submission deadline! The DebConf team is excited to announce that we are now accepting
proposals for contributions to this year s Debian conference! We
invite you to submit proposals for how you would contribute until the
deadline: soon There are many ways you can contribute, you could present a technical
paper, host a panel discussion, put on a tutorial, do a performance,
an art installation, a debate, host a meeting (BoFS, or Birds of a
Feather Session), or other possibilities that you devise. This year we
are also accepting proposals for tracks=E2=80=94a thematic grouping around a
particular subject, and people to coordinate those tracks. If you are
looking for ideas of things that you could contribute, or have ideas
for things that you would like to see happen at DebConf, have a look
at the Contribution Brainstorm[0] page. DebConf talks will be broadcast live on the Internet where possible,
unless otherwise requested. Videos of the talks will be published on
the web along with the presentation slides and papers. =3D=3D Step 1. Submit Proposal =3D=3D Proposals should provide an overview of your proposed contribution,
similar to an abstract, and be no more than 600 words. Since
contributions can cover a range of issues, please describe what your
proposal covers, be it political (Free Software, law, advocacy,
access, etc.), technical (d-i, OpenPGP, etc.), social (Debian
structures and groups). It s okay to hit all three of these. Your
proposal should also include some information on why you re qualified
to present, or coordinate this topic. For example, you can include
links to posts you have made to Debian mailing lists or other material
that indicates your qualifications. Proposals and their abstracts will
be accepted until May 1st, 2010, 23h59 UTC. To submit your proposal, go to Penta[1] and register as an
attendee. Once you are registered, there is a Submit a proposal
link. You will see your proposal on the site. You can choose between
different presentation types. If you are unsure what type you should
use, please read our glossary of definitions[2]. If it s not there for
some reason, or you have questions, please contact us immediately at
mailto:talks@debconf.org. =3D=3D Step 2. Committee Review =3D=3D The review committee for this year will make its decision by May 1st,
2010. All correspondence will be done by email. =3D=3D Step 3. Paper Submission =3D=3D Like last year, we are asking for papers to be submitted along with
the presentations. A full paper is optional, however we strongly
encourage you to submit one if possible as we intend to provide
written information along with transcripts of the sessions for later
viewing. Having written papers in advance will allow us to get
translations done to help non-native-English speakers feel more
comfortable with the topics presented. Papers are due by July 1st,
2010, and should cover the topic in reasonable depth (3 pages A4 text,
plus pictures and diagrams). We will use LaTeX to typeset the
proceedings. Please submit your paper formatted in LaTeX. Should you
be unfamiliar with LaTeX earlier submission in plain text is also
fine. =3D=3D Step 4. Live Presentation =3D=3D Longer presentations may have a break in the middle and should include
workshop items that directly involve the participants. If using slides
or any other presentation, please consider that your audience will
consist of people who use free software, and your choice of
application to prepare and display the slides should reflect this if
at all possible. =3D=3D Fine Print Publication Rights =3D=3D DebConf requires non-exclusive publication rights to papers,
presentations, and any additional handouts or audio/visual materials
used in conjunction with the presentation. The authors have the
freedom to pick a DFSG-free license for the papers themselves and
retain all copyrights. The presentations will be recorded, and may be
broadcast over the Internet. Any copies of the presentation will be
made available =3D=3D Failure to Submit =3D=3D In the event that a deadline is missed we reserve the right to revoke
any offer to present. =3D=3D About DebConf =3D=3D The annual DebConf conference is a technical and social forum for
Debian developers, sponsors, affiliates, and friends. It allows
various groups within Debian a chance to come together, network, and
share their work. Until we see you in the big city,
DebConf Talks Team [1] introduction written by Micah Anderson

Erich Schubert: Oil spill: fortunately it was "just oil" - and not nuclear.

Face it: technology fails. Sometimes. Sooner or later.Statistics are a nice way to fool people, because they can't interpret the numbers properly. And usually this is done to make things sound less risky than they really are. But it's just too easy to lie with numbers. Say, they risk of a major incident at a plant is less than 1% (per year). Sounds good, doesn't it? But if you have 100 plants, that means it will happen about once a year. Make that 1000. Or even more.There are currently around 440 nuclear reactors in use for electricity. Probably some more the IAEA doesn't know about. There are definitely extra reactors in various military vehicles including unmaintained russian submarines. So make that 500. So looking forward to the next 200 years, we want to make sure things are not likely to happen at a rate of 0.001%. Unless our atomic plants are way more than 99.999% secure, we're likely to see a major nuclear accident sooner or later.Which brings us to a totally differnt scale when we look at photovoltaics or wind power. There are just so many more of these plants, that you can bet on some of them failing completely every day. The good news about these plants is that their damage potential is just so much lower. They might be as much as thousandfold more likely to fail than a nuclear plant. And there are probably a thousand more, too. But even if we completely underestimate the risk they just don't do as much damage even in the worst case. Much unlike a nuclear meltdown or such an oil spill. I bet noone would have thought of oil drilling being this risky. After all, it has been done for years.So how is your risk assessment of nuclear power? Is it less risky than oil drilling?

Russell Coker: CPL Still Sucks

I previously described my experience with Computers and Parts Land (CPL) [1] who gave me a product that didn t do what I wanted (because they thought that they knew better than me) and then gave me attitude when I returned it. As it s almost a year since that incident I had to buy something else from them to use the credit note as I decided that it s not worth making a Trade Practices Act (TPA) [2] issue out of it. Thanks Tim for the suggestion though. Firstly I went to their new store that is advertised on their web site. The people at the new store refused to honor the credit note (they probably hoped that I would just let it go and give them an extra $60 in profit). Claiming that I need to take a credit note back to the store it came from is bogus. Anyway I went back to the original store and bought a new 1.5TB SATA hard drive which seems to work well enough. The service was really slow, I was the only customer in the store and there were several employees not doing much but it still took them ages to print a receipt and give me the hard drive. It will be the last CPL purchase I ever make. In the unlikely situation that there is ever any new gear that is only sold by CPL then I will wait a year and buy it on ebay. It would be good if someone started working as a TPA complainant. They would take donations from dis-satisfied customers who want to pay for an investigation of a shonky company (I d pay $10 for an investigation of CPL). When enough donations were collected they would buy some stuff and make a video of every interaction with the employees and then launch a private legal action under the TPA if they do anything wrong. The TPA complainant would get to keep everything that they buy, any excess donation money after proving the case, and any money offered by the company to settle law suits. While it s not a good use of my time and money to go after CPL directly it would be good to give some money to someone who would then see them dealt with properly.

29 April 2010

David Welton: Erlang vs node.js

I've written about Erlang in the past, and my suspicion that, soon or later, other languages/systems will come along and "eat its lunch". Scala is one such potential contender. Another that has been gaining some visiblity lately is node.js, a simple framework for creating networked applications on top of Google's V8 Javascript engine. I should define what Erlang's lunch is a little bit better before we go on. Erlang does several things really, really well; better than many existing, mainstream systems:
  1. Concurrency - the Actor model that it uses is much easier and more pleasant than dealing with threads.
  2. Distributed systems - Erlang makes this fairly easy and pleasant as well.
  3. Fault tolerant systems - using its distributed powers, Erlang is good for writing things like telephone switches that can't spend any significant time down.
Of these, I think the big one is concurrency. Distributed systems are nice, but not a critical point for most people. Same goes with fault-tollerant: those who need it really need it, but the rest of us are willing to make some compromises. Our web applications are important, but generally not something where people's lives hang in the balance. How does Erlang do "concurrency"? The developer creates Erlang "processes" that interact with one another by passing messages. These are not real, OS level processes, though, and this is critical to how Erlang operates. Since these processes must all coexist within one real, system level process, it is absolutely essential that no operation that they perform hangs the entire system! The Erlang runtime system is built around this concept. Any Erlang process can do things like read and write to the disk or network, or have a "while (true) .... " loop (it doesn't actually look quite like that in Erlang, but that's the idea), and it won't wedge the system. This knowledge is also critical when you want to interface Erlang to the outside world: if your C library contains a routine that might block for a long time, you can't just call it from Erlang as it won't be a well-behaved part of Erlang's world (there are ways around this of course, but make life a bit more complicated for the developer). All this is done with Erlang's scheduler: each Erlang process gets a number of operations it can run before some other process gets to run, so even our while loop will only run for a bit before the system moves on to something else. IO is rigorously done with non-blocking calls internally in order to keep that from becoming an issue. No other system that I know of has such a focus on being non-blocking, and node.js is no exception: a while(true) loop is perfectly capable of wedging the system. Node.js works by passing functions (closures, in many cases) around so that work can be performed as needs be. However, the actual functions that run actually do block the system, and thus must be written in order to not run for too long. Also, and this is important, Node.js also does its best to make IO non-blocking by default, so that you don't have to worry about IO calls. Node.js isn't up to the level Erlang is at, because it requires more manual intervention and thinking about how to do things, but it's probably "good enough" for many tasks. How often do you really write code with so many calculations that it slows things down? Not often in my case - the real world problem I most often encounter is IO, and node.js does its best to make that non-blocking, so that it can be handled in the "background" or a bit at a time, without wedging the system. And if you really needed to write a long-running calculation (say you want to stream the digits of PI or something), you can break up your calculation manually, which may not be quite as elegant as Erlang, but is "good enough" for many people. "Good enough" concurrency, combined with a language that is at least an order of magnitude more popular than Erlang, and a fast runtime, combined with ease of use in general (it's way easier to get started with node.js than with most Erlang web stuff) make for a system that's likely to do fairly well in terms of diffusion and popularity, and is going to "eat some of Erlang's lunch". Or perhaps, rather than actually taking users away from Erlang, it's likely to attract people that might have otherwise gone to Erlang.

28 April 2010

Petter Reinholdtsen: Thoughts on roaming laptop setup for Debian Edu

For some years now, I have wondered how we should handle laptops in Debian Edu. The Debian Edu infrastructure is mostly designed to handle stationary computers, and less suited for computers that come and go. Now I finally believe I have an sensible idea on how to adjust Debian Edu for laptops, by introducing a new profile for them, for example called Roaming Workstations. Here are my thought on this. The setup would consist of the following: I believe all the pieces to implement this are in Debian/testing at the moment. If we work quickly, we should be able to get this ready in time for the Squeeze release to freeze. Some of the pieces need tweaking, like libpam-ccreds should get support for pam-auth-update (#566718) and nslcd (or perhaps debian-edu-config) should get some integration code to stop its daemon when the LDAP server is unavailable to avoid long timeouts when disconnected from the net. If we get Kerberos enabled, we need to make sure we avoid long timeouts there too. If you want to help out with implementing this for Debian Edu, please contact us on debian-edu@lists.debian.org.

Next.

Previous.