In the last years, I have seen many people try to use dpkg --build to create Debian packages. Indeed, if you look up dpkg s and dpkg-deb s manual pages, this option seems to be what you have to use:

-b, --build directory [archive directory] Creates a debian archive from the filesystem tree stored in directory. directory must have a DEBIAN subdirectory, which contains the control information files such as the control file itself. This directory will not appear in the binary package s filesystem archive, but instead the files in it will be put in the binary package s control information area.

And indeed, dpkg-deb is what ultimately creates the .deb files (aka binary packages). But it s a low-level tool that you should not call yourself. If you want to properly package a new software, you should rather create a Debian source package that will transform upstream source code into policy-compliant binary packages. Creating a source package also involves preparing a directory tree (but with a debian sub-directory), it s probably more complicated than calling dpkg -b on a manually crafted directory. But the result is much more versatile: the tools used bring value by dynamically analyzing/modifying the files within your package (for example, the dependencies on C libraries that your package needs are automatically inserted). If this is news to you, you might want to check out the New Maintainers Guide and the Debian Policy.

Found it useful? Be sure to not miss other packaging tips (or lessons), click here to subscribe to my free newsletter and get new articles by email.

10 comments Liked this article? Click here. My blog is Flattr-enabled.

I've recently spent five days debugging a problem in obnam, my backup program. It turned out to be a bug in my B-tree implementation, which I wrote for obnam. The process was pretty long, and it might be instructive to de-brief myself. The bug manifested itself when I ran a backup test that was larger than what I had ran before: one terabyte of live data. The symptom was a key lookup problem in a B-tree. Good: the problem was entirely deterministic, and easy to reproduce. Bad: it took hours to reproduce it. My first step was to fiddle with the obnam parameters to see if I could make it fail faster, but in the same way. I got the time down to half an hour. After this, I started fiddling with the code to see if I can figure out what happens, and perhaps find a quick fix to the bug. After some hours, I realized I was being stupid. I was making essentially random changes to the code, instead of hunting down the actual cause. I had no idea what was going wrong, so trying to fix it was an act of utter stupidty. So I went back to where I had started from. The next step was to run "obnam fsck" on the failing backup store. This did not find any problems. The fsck I've written is a bit simplistic, and merely checks that for every client in the backup store, every generation is found, and within every generation each file's metadata, and data chunks are found. I will improve fsck in the future, but right now it was not helpful. I did not have a tool to verify that the actual B-trees were OK. So I wrote one. It, too, is quite simplistic, but at least it proved that the key was still present in the on-disk version of the B-tree. At this point, I did a code review of the B-tree code. It had been a some months since I'd touched the code, and hence the review brought up a bunch of minor issues: outdated comments, badly formatted code, stupid variable names, etc. One or two buglets, too, but nothing relevant to the case of the missing key. At this point I made a mistake: I concluded that the error was not in the B-tree code, but the obnam code using the B-trees. I spent the next couple of days running the test case, adding more logging, and analyzing the results. Without my mistake, I would not have ignored indicators that would have led me faster into the depths of the B-tree code. I might have saved a day or two of the process. Eventually I narrowed down the problem to a single B-tree insertion call in the obnam code. The call inserted a new value for a given key into one tree, and the value in another tree changed as well. Quite mysterious. My B-trees are an implementation of the Rodeh B-trees. All updates are done using copy-on-write, and this allows efficient cloning of trees. However, as an optimization, a node that is used by only one tree at a time can be modified in-place. This speeds things up a lot when the same tree is updated many times. In obnam, each B-tree represents a backup generation. The specific bug I was hunting was that the key I use for storing the unique identifier for a generation changed in two trees at the same time. Now that I have actually found and fixed the problem, it is clear that I should have realized at once what was happening, but I didn't, so I spent another couple of days adding logging statements into the B-tree insertion code, and tracing where things went wrong. Since B-trees are updated quite heavily by obnam, the log files grew quite big, and the test runs grew quite long. The last run I did resulted in a 94 gigabyte log file, and lasted over 12 hours. less had trouble browsing the log file, probaby due to the rather long lines generated by dumps of the B-trees. I ended up splitting the large file into quite a number of small ones, using csplit. After some more analysis, I figured out the problem: the code that shadowed a B-tree node, i.e., implemented the copy-on-write modification, failed to update the reference counts of the children of a shadowed index node. The Rodeh B-trees use lazy reference counts: when an index node is shadowed, only the immediate children get reference count updates. This turns out to work quite well, and saves quite massive amounts of updates. Unfortunately, my code was too lazy, and did not update any reference counts. After I had come this far, it only took a couple of minutes to write a unit test to catch the problem, fix the code, run the unit tests successfully, and commit. Then I ran the big test case again, without the extra logging I had added, and the code worked fine. At least it worked fine for about 24 hours until it ate up all memory in the VM I was running it on, but that's a separate problem. What can I do to prevent this kind of bug happening in the future? Unit testing didn't catch it, and I have 100% statement coverage with unit tests (not counting 19 excluded statements). Code review by other programmers would surely help. I should perhaps attract others to the project. Please excuse the length of this blog post. It is a distillation of the about ten A4-pages' worth of notes I made during this debugging process. (Always make notes when debugging. Your brain can't remember all details.)

It's been a while since I started working on this project and even longer
since I had the idea. It's therefore a pleasure to finally announce the DACA project.


What is DACA?

Automated Code Analysis helps detect and fix bugs and other issues in source
code. The project aims to give users easy access to a wide variety of tools to
improve quality of software distributed by Debian, while giving the tool's
developers a test bed, more visibility, and more feedback.
This is achieved by running those tools on the complete Debian archive.

http://qa.debian.org/daca/

What is there for everyone?

At the moment there are only partial reports from two tools, but the list of
tools to be evaluated and possibly included goes over twenty.

Current tools: cppcheck, and checkbashisms (at the source package level.)

Limitations

Most of the tools are CPU-bound, limiting considerably the number of tools and
time it takes to check the whole Debian archive. For example, with the typical
sid repository update (i.e. not during the freeze and with a working ftp-
master) it is impossible for the server running cppcheck to keep up with all
the changes.

How can you help?

* First of all you can go and squash bugs!
Please keep in mind what's in the notices at the bottom of the pages. They are
rather static now, but they may change later.

* Second, report false positives, fix bugs, improve the tools
Every report page (at the footer) should mention the version of the tool used
to generate it. They are usually the latest.

* Third, join the DACA project
More hands are needed to evaluate other tools, setup an infrastructure for
running them, and finally generating the web reports.
Discussing tools already available at DACA is also welcome.

There's a project request at Alioth pending its approval, but once accepted
access to the repository and mailing lists will be found at:

http://alioth.debian.org/projects/daca

(the contact email address at the DACA website will be updated accordingly)

* Fourth, donate hardware

If you have equipment with very powerful CPUs or other hardware you (or your
company) can donate, please take a look at the following page:

http://www.debian.org/donations#equipment_donations

Note that this may be handled by people unrelated to the DACA project, and as
such you should mention what you would prefer your hardware to be used for.

* And finally, figure out what I missed, provide feedback, and go back to step
one.


Thanks!

By the way, thanks to the DSA team for tolerating a few hiccups on the host running
cppcheck, and to those who previewed the website and provided feedback.

Debian 6.0 Squeeze to be released with completely free Kernel(s) Today we have announced that, starting with the upcoming release of Squeeze, Debian will be even Free-er. Exceptions to the DFSG for non-free Linux firmware blobs, which have been granted in the past, will no longer be granted. Starting from Squeeze, Debian will be Free the bottom up, no matter where your own definition of software ends. The news has been known for a while within the Debian development community, but we had the pleasure of sharing that with our users only today. According to the reaction on identi.ca, where the news quickly made into the most popular notices, people are enthusiastic about the news and that in turn is very rewarding. If you are as happy and proud about this achievement as I am, please direct your kudos to all the teams who made this possible:

• the Debian kernel team who worked for the past 2 release cycles, together with Linux upstream, to split free-vs-non-free firmware bits and monitor new releases to avoid freedom regressions;
• the Debian CD team who basically "forked" different media production lines, for different user needs wrt firmware;
• the d-i team who added, long ago, support for loading user-provided firmware at installation time;
• whoever else I might have forgotten who contributed to this result (feel free to leave a comment to refresh my memory!).

Having been trol^W asked several times in the past about the "real freeness" of Debian at events, talks, and the like, I consider this episode to be a very nice story to tell. While we have granted exceptions for Linux firmware at the time of Etch and Lenny releases, during this time Debian people have worked, together with upstream, to make this final result possible. All this work has improved the situation not only for Debian users, but also for users of those derivatives which rely upon Debian kernels. Nonetheless, don't expect us to live under a rock. We know that there are users out there who need nasty non-free firmware bits to boot their machines or to access the Internet. That is why separate images with firmware included are being provided, although they are not part of Debian and are properly "discriminated": they should be looked for explicitly by users and can't be supported to the same extent of Free firmware, simply because we (as anybody else other than the hardware manufacturer) do not have access to the corresponding source code. This separation is a perfect example of the pragmatism embodied by the Debian Social contract: Debian consists of 100% Free software; at the same time, Debian enables its users to make their own choices in terms of Freedom, being explicit about what users risk if they cross the boundaries of DFSG. Awareness is the key here and I find it to be a very honest yet effective way of explaining to the world why Free Software is better. This is not the end of the story though, as we need to document this change properly to both increase awareness and avoid leaving users in the dark. A couple of bugs filed today might benefit from the help of some kind doc-writers: Debian bug #607191 (documenting the firmware change in the Squeeze release notes) and Debian bug #607193 (documenting the firmware change on the website).

Being able to rebuild an existing Debian package is a very useful skill. It s a prerequisite for many tasks that an admin might want to perform at some point: enable a feature that is disabled in the official Debian package, rebuild a source package for another suite (for example build a Debian Testing package for use on Debian Stable, we call that backporting), include a bug fix that upstream developers prepared, etc. Discover the 4 steps to rebuild a Debian package. 1. Download the source package The preferred way to download source packages is to use APT. It can download them from the source repositories that you have configured in /etc/apt/sources.list, for example:

deb-src http://ftp.debian.org/debian unstable main contrib non-free
deb-src http://ftp.debian.org/debian testing main contrib non-free
deb-src http://ftp.debian.org/debian stable main contrib non-free

Note that the lines start with deb-src instead of the usual deb . This tells APT that we are interested in the source packages and not in the binary packages. After an apt-get update you can use apt-get source publican to retrieve the latest version of the source package publican . You can also indicate the distribution where the source package must be fetched with the syntax package/distribution . apt-get source publican/testing will grab the source package publican in the testing distribution and extract it in the current directory (with dpkg-source -x, thus you need to have installed the dpkg-dev package).

$ apt-get source publican/testing
Reading package lists... Done
Building dependency tree
Reading state information... Done
NOTICE: 'publican' packaging is maintained in the 'Git' version control system at:
git://git.debian.org/collab-maint/publican.git
Need to get 727 kB of source archives.
Get:1 http://nas/debian/ squeeze/main publican 2.1-2 (dsc) [2253 B]
Get:2 http://nas/debian/ squeeze/main publican 2.1-2 (tar) [720 kB]
Get:3 http://nas/debian/ squeeze/main publican 2.1-2 (diff) [4728 B]
Fetched 727 kB in 0s (2970 kB/s)
dpkg-source: info: extracting publican in publican-2.1
dpkg-source: info: unpacking publican_2.1.orig.tar.gz
dpkg-source: info: unpacking publican_2.1-2.debian.tar.gz
$ ls -dF publican*
publican-2.1/                 publican_2.1-2.dsc
publican_2.1-2.debian.tar.gz  publican_2.1.orig.tar.gz

If you don t want to use APT, or if the source package is not hosted in an APT source repository, you can download a complete source package with dget -u dsc-url where dsc-url is the URL of the .dsc file representing the source package. dget is provided by the devscripts package. Note that the -u option means that the origin of the source package is not verified before extraction. 2. Install the build-dependencies Again APT can do the grunt work for you, you just have to use apt-get build-dep foo to install the build-dependencies for the last version of the source package foo. It supports the same syntactic sugar than apt-get source so that you can run apt-get build-dep publican/testing to install the build-dependencies required to build the testing version of the publican source package. If you can t use APT for this, enter the directory where the source package has been unpacked and run dpkg-checkbuilddeps. It will spit out a list of unmet build dependencies (if there are any, otherwise it will print nothing and you can go ahead safely). With a bit of copy and paste and a apt-get install invocation, you ll install the required packages in a few seconds. 3. Do whatever changes you need I won t detail this step since it depends on your specific goal with the rebuild. You might have to edit debian/rules, or to apply a patch. But one thing is sure, if you have made any change or have recompiled the package in a different environment, you should really change its version number. You can do this with dch --local foo (again from the devscripts package), replace foo by a short name identifying you as the supplier of the updated version. It will update debian/changelog and invite you to write a small entry documenting your change. 4. Build the package The last step is also the simplest one now that everything is in place. You must be in the directory of the unpacked source package.
Now run either debuild -us -uc (recommended, requires the devscripts package) or directly dpkg-buildpackage -us -uc . The -us -uc options avoid the signature step in the build process that would generate a (harmless) failure at the end if you have no GPG key matching the name entered in the top entry of the Debian changelog.

$ cd publican-2.1
$ debuild -us -uc
 dpkg-buildpackage -rfakeroot -D -us -uc
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2
dpkg-buildpackage: export CPPFLAGS from dpkg-buildflags (origin: vendor):
dpkg-buildpackage: export CXXFLAGS from dpkg-buildflags (origin: vendor): -g -O2
dpkg-buildpackage: export FFLAGS from dpkg-buildflags (origin: vendor): -g -O2
dpkg-buildpackage: export LDFLAGS from dpkg-buildflags (origin: vendor):
dpkg-buildpackage: source package publican
dpkg-buildpackage: source version 2.1-2rh1
dpkg-buildpackage: source changed by Rapha l Hertzog <hertzog>
 dpkg-source --before-build publican-2.1
dpkg-buildpackage: host architecture i386
[...]
dpkg-deb: building package  publican' in  ../publican_2.1-2rh1_all.deb'.
 dpkg-genchanges  >../publican_2.1-2rh1_i386.changes
dpkg-genchanges: not including original source code in upload
 dpkg-source --after-build publican-2.1
dpkg-buildpackage: binary and diff upload (original source NOT included)
Now running lintian...
Finished running lintian.

The build is over, the updated source and binary packages have been generated in the parent directory.

$ cd ..
$ ls -dF publican*
publican-2.1/                    publican_2.1-2rh1.dsc
publican_2.1-2.debian.tar.gz     publican_2.1-2rh1_i386.changes
publican_2.1-2.dsc               publican_2.1-2rh1_source.changes
publican_2.1-2rh1_all.deb        publican_2.1.orig.tar.gz
publican_2.1-2rh1.debian.tar.gz

Do you want to read more tutorials like this one? Click here to subscribe to my free newsletter, you can opt to receive future articles by email.

11 comments Liked this article? Click here. My blog is Flattr-enabled.

We've just posted a news item about the removal of non-free firmware from Debian's Linux kernel packages. Kudos to the kernel team and others for the work involved to make this happen! While we would be much happier if these binary blobs were not necessary at all, it's clear that some of our users may depend on them to support the hardware they are using. So, as mentioned in that news item, we've started making some unofficial CD images available that include the non-free firmware packages that we can redistribute. If you need one of these CDs, daily builds are to be found at http://cdimage.debian.org/cdimage/unofficial/non-free/cd-including-firmware/ . Starting with the next debian-installer release (hopefully RC1, coming soon!), we'll be generating fixed versions of these images to match each release too. Be aware that these are only netinstall images, equivalent to the normal free netinst images that we normally produce but with the firmware files added. If you want to do an installation entirely from a set of CDs/DVDs/BDs then you will need to burn one of these in addition to that set, then feed the rest of the set when prompted to by the installer.

As I'm not currently working on any suitable paper, I'll just post this to my blog so it does not completely slip off my radar ;-) Also, it might be interesting to my reader. Readers? Oh, there are two of you now? Good! Yesterday, I learnt thanks to Beatriz Busaniche that a group of South American Free Culture activists launched number zero of a magazine that promises to be very interesting: Cultura RWX, cultura en modo lectura, escritura y acci n (culture in reading, writing and action mode). Guys, best luck with this new project! Anyway, reading it, I found this asseveration I want to keep at hand:

( )cuando surge la industria musical aparecen los derechos de autor como forma de defensa de los productores musicales, espec ficamente los m sicos. No tanto frente a los usuarios, porque hasta el cassette no existi posibilidad de copiar una obra musical. Era una defensa frente a las discogr ficas, que buscaban cerrar contratos muchas veces abusivos con los artistas. M sica en Libertad: La industria musical frente al cambio de paradigma; Mat as Lennie, adaptaci n: Sebasti n Vazquez

Yes, yes, translating to English:

( )when the musical industry was born, copyright appeared as a means of defense of the musical producers, specifically of the musicians. Not so much against the users, because up until the invention of the cassette there was no possibility to copy a musical creation. It was a defense against the discographic companies, which tried to close often abusive contracts with the artists. Music in Freedom: Musical industry and the paradigm shift; Mat as Lennie, adaptation: Sebasti n V zquez

I have argued (i.e. in here) in this same line regarding the birth of copyright itself It was an arrangement that had to be made between writers and printers, back in the XVI/XVII centuries. Simple individuals were just unable to get anything of value out of the copying technology they had at hand. Copyright was born in a time where reproduction required specialized equipment. Today, massive reproduction technology is a given for a good portion of the planet's population. Copyright now only defends big corporations And will inevitably fade away as anachronic. Of course, it refuses to go without a fight... But it cannot win long-term. We cannot afford to allow it!

There s been a lot of discussion regarding whether or not Nokia is Doomed or not. The people who say Nokia are doomed basically point out that Nokia doesn t have any attractive products at the high end, and at the low end the margins are extremely thin. The high end products suffer from the Symbian being essentially dead (even Nokia is recommending that developers not develop native applications for Symbian, but to use Qt instead), and Nokia doesn t have much of a development community following it, and it certainly does have much in the way of 3rd party applications, either targetting Symbian or Qt at the moment. So what do I think of the whole debate between Tomi and Scoble? First of all, I think there is a huge difference in American and European assumptions and perspectives, and a big question is whether the rest of the world will end up looking more like Europe or America vis-a-vis two key areas: cost of data plans, and whether phones become much more application centric. Tomi took Apple to task in the comments section of his 2nd article for not having an SD card slot (how else would people share photos with their friends?) and for not supporting MMS in its earlier phones. My first reaction to that was: Um, isn t that what photo-sharing sites are for? Is it really that hard to attach a photo to an e-mail? And then it hit me. In Europe, data is still like MMS a few years ago a place for rapacious carriers to make way too much money. Many European telco s don t have unlimited data plans, and charge by the megabyte and even if you re lucky enough to live in a country which does have an American-like data plan, the cost of data roaming is still incredibly expensive. In contrast, in the US, I can pay $30/month for an unlimited data plan, and I can travel 2000 miles south or west and it will still be valid. Try doing that in Europe! The US had consumer-friendly data plans much earlier than Europe did, and so perhaps it s not surprising that Nokia has engineered phones that were far more optimized for the limitations caused by the Europe s Wireless carriers. The second area of debate where I think Scoble and Tomi are far apart is whether phones of the future are fundamentally about applications or well, making phone calls. Here I don t have proof that this is a fundamentally European vs. US difference, but I have my suspicions that it might be. Tomi spent a lot of time dwelling on how Nokia was much better at making phone calls (i.e., better microphones, better radios, etc). And my reaction to that was, Who cares? I rarely use my phone for making phone calls these days! And that was certainly one of the reasons why I gave up on Nokia after the E70 its contacts database was garbage! It was OK as a phone directory, but as a place for storing multiple addresses and e-mail addresses, it didn t hold a candle to the Palm PDA. And that s perhaps the key question how much is a smart phone and about being a phone , versus being a PDA (and these days I want a cloud-synchronized PDA, for my calendar, contacts, and todo lists), and how much is it about applications? This is getting long, so I think I ll save my comments about whether I think Meego will be an adequate savior for Nokia for another post. But it s worthwhile to talk here about Tomi s comments about most smartphones being much cheaper than the luxury iPhone, and so it doesn t matter that Nokia s attempt in the higher end smart phones has been a continuous history of fail. First of all, it s worth noting that there are much cheaper Android phones available on the market today, which are price-competitive with Nokia s low-end smartphones (i.e., available for free from T-Mobile in the States with a two year commitment). Secondly, the history in the computer market over the last twenty years is that features inevitably waterfall into the cheaper models, and prices will tend to drop over time as well. Apple started only with the iPod, but over time they added the iPod Nano and the iPod Shuffle. And it would not surprise me if they introduce a lower-end iPhone as well in time as well. It would shock me if they aren t experimenting with such models even as we speak, and have simply chosen not to push one out to the market yet. So even if you buy Tomi s argument that the high-end smartphones don t matter, and you only care about volume, and not about profit margins (talk to the people at Nokia that will need to be laid off to make their expenses match with their lowered revenue run rates; I bet they will care), the question is really about whether Nokia has time to execute on the Meego vision before it s too late and the current application-centric smartphone ecosystems (Android and iPhone) start eating into the lower-end smartphone segment. More on that in my next post. No related posts.

There s been a lot of discussion regarding whether or not Nokia is Doomed or not. The people who say Nokia are doomed basically point out that Nokia doesn t have any attractive products at the high end, and at the low end the margins are extremely thin. The high end products suffer from the Symbian being essentially dead (even Nokia is recommending that developers not develop native applications for Symbian, but to use Qt instead), and Nokia doesn t have much of a development community following it, and it certainly does have much in the way of 3rd party applications, either targetting Symbian or Qt at the moment. So what do I think of the whole debate between Tomi and Scoble? First of all, I think there is a huge difference in American and European assumptions and perspectives, and a big question is whether the rest of the world will end up looking more like Europe or America vis-a-vis two key areas: cost of data plans, and whether phones become much more application centric. Tomi took Apple to task in the comments section of his 2nd article for not having an SD card slot (how else would people share photos with their friends?) and for not supporting MMS in its earlier phones. My first reaction to that was: Um, isn t that what photo-sharing sites are for? Is it really that hard to attach a photo to an e-mail? And then it hit me. In Europe, data is still like MMS a few years ago a place for rapacious carriers to make way too much money. Many European telco s don t have unlimited data plans, and charge by the megabyte and even if you re lucky enough to live in a country which does have an American-like data plan, the cost of data roaming is still incredibly expensive. In contrast, in the US, I can pay $30/month for an unlimited data plan, and I can travel 2000 miles south or west and it will still be valid. Try doing that in Europe! The US had consumer-friendly data plans much earlier than Europe did, and so perhaps it s not surprising that Nokia has engineered phones that were far more optimized for the limitations caused by the Europe s Wireless carriers. The second area of debate where I think Scoble and Tomi are far apart is whether phones of the future are fundamentally about applications or well, making phone calls. Here I don t have proof that this is a fundamentally European vs. US difference, but I have my suspicions that it might be. Tomi spent a lot of time dwelling on how Nokia was much better at making phone calls (i.e., better microphones, better radios, etc). And my reaction to that was, Who cares? I rarely use my phone for making phone calls these days! And that was certainly one of the reasons why I gave up on Nokia after the E70 its contacts database was garbage! It was OK as a phone directory, but as a place for storing multiple addresses and e-mail addresses, it didn t hold a candle to the Palm PDA. And that s perhaps the key question how much is a smart phone and about being a phone , versus being a PDA (and these days I want a cloud-synchronized PDA, for my calendar, contacts, and todo lists), and how much is it about applications? This is getting long, so I think I ll save my comments about whether I think Meego will be an adequate savior for Nokia for another post. But it s worthwhile to talk here about Tomi s comments about most smartphones being much cheaper than the luxury iPhone, and so it doesn t matter that Nokia s attempt in the higher end smart phones has been a continuous history of fail. First of all, it s worth noting that there are much cheaper Android phones available on the market today, which are price-competitive with Nokia s low-end smartphones (i.e., available for free from T-Mobile in the States with a two year commitment). Secondly, the history in the computer market over the last twenty years is that features inevitably waterfall into the cheaper models, and prices will tend to drop over time as well. Apple started only with the iPod, but over time they added the iPod Nano and the iPod Shuffle. And it would not surprise me if they introduce a lower-end iPhone as well in time as well. It would shock me if they aren t experimenting with such models even as we speak, and have simply chosen not to push one out to the market yet. So even if you buy Tomi s argument that the high-end smartphones don t matter, and you only care about volume, and not about profit margins (talk to the people at Nokia that will need to be laid off to make their expenses match with their lowered revenue run rates; I bet they will care), the question is really about whether Nokia has time to execute on the Meego vision before it s too late and the current application-centric smartphone ecosystems (Android and iPhone) start eating into the lower-end smartphone segment. More on that in my next post. No related posts.

It's been an intense couple of days out at the Hollow. Did it only start to snow yesterday? It feels like a long time in winter. Yesterday was the first day that the solar panels produced no power, it was too cloudy and snowing most of the time. I cleaned an inch of slushy snow off the panels to no avail, and wondered if the power would last. But the wood stove has kept it nice and warm, so yesterday was spent cosily in the living room enjoying the snowfall, nethack, and podcasts. Also, I'm very much enjoying cooking on the wood stove. I've never been in a position to cook on one before; when I lived at Wortroot the stove was in the cabin, and the cold kitchen in a different building. So cooking on the stove, or even keeping a kettle on, was mostly only done in emergencies. Here, I am constantly using it, and the propane stove is still on its first 20 pound tank. Long slow cooked beany or roastish stuff does especially well. Yesterday I did a great pork loin roast with homemade quick sauerkraut. Today, after cleaning another 3 inches of snow off the solar panels, and scraping off a layer of ice, I am getting a decent amount of power for a cloudy day with some snow still falling. Turns out it was good I cleaned them yesterday after all, since a part I missed has a inch+ layer of hard ice, which I don't dare chip away at for fear of damaging the panels. I'd not want to live someplace where it snowed a lot with solar panels that didn't have some kind of a heating system. I wish I had a camera to show the long walk down the driveway, through the winter woods, and past the field where the wind blows gusts of snow into the air. But my cell phone is on a Caribbean cruise. I have been recording data ever since the fall equinox, and it's almost to the winter solstice now. Time to geek out with some graphs! Solar production has surely been falling off with the waning days, but it seems like it will get through the shortest day with enough, and from then on, no worries.

Highest watt production seen each day I've been here.
Zero all day yesterday.

Power system voltages.
Charge controller limits it to a maximum of 15 volts during the day.
Originally, it was limiting it to 13 volts.

we release when it's ready ... Members of the Debian release team clearly take very seriously Debian release mantras such as we release when it's ready and good things come to those who wait. Julien Cristau is no exception. After a freeze period used to improve the quality of his research work bringing down to 0 the number of release critical bugs affecting it, Julien has just released his PhD thesis titled Jeux et automates sur les ordres: Congrats, Doctor! PS earlier on today, Julien has promised that, as a gift to the Debian project for his new title, he will fix 50 Squeeze RC bugs by the end of the week. \o/
(Not. But that would indeed be a nice gift to him from all of us, don't you think?)

So, last week I bricked a device and I think I can fix it over a RS-232 cable from a borrowed Windows computer. However, I don t have the right cable in my office and the Windows computers I can borrow no longer have old-style serial ports. So I had two choices:

1. Order a USB-serial converter and the right cable from an online supplier and pay a premium for next-day delivery (so it doesn t arrive when I m on the trip);
2. Make a half-hour detour on a trip two days later to visit a high street electronics store and pay fairly high prices (like three times the online price).

Of course, I ordered online. And that s where the trouble started. My order was sent by Initial Citylink, who make Parcelfarce look good. Next day came and went with no delivery. About 2pm, the parcel s web page changed to say There was no one to receive the goods at the delivery point so a card was left and that it would be delivered today, Monday. Of course, there had been someone here all day and no card was left. Phoning Citylink gets a recorded message saying they re too busy to take calls. No offer to call me back. Just suggests trying the useless website. Over the weekend, the page changed again. Now it says Thank you for your rescheduling request. Your parcel(s) will be delivered on Tuesday 14th December between 07:30 and 17:30 . I didn t request that, I don t know who did and there s no way to tell from the web page. There s no way to undo that request and ask for it to be delivered on Monday instead. I wanted the device fixed by today if possible. Now I m going to have to bodge something together from spares (for once I m glad I have a cupboard of old kit here ). Still no answer from Citylink s phones and the supplier doesn t seem to know what s going on either. I m not paying for next-day delivery from anyone who uses Citylink again.

Sometimes, free software isn't better than proprietary, it's true - and one of the principal reasons is that the majority of free software does not use the full power of the freedom of the software. Far too many free software projects are single-developer projects. Many single-developer projects at SourceForge or Savannah and elsewhere have a single maintainer for the distribution providing the packages too and some (including nearly all of my own) have one person doing both jobs. In other words, there is no collaboration upstream or in packaging, so there is little or no benefit to the code of being free software...

It begs the question of what happens when the omnipresent developer-under-a-bus risk actually bites.

When comparing free software and proprietary, considering end-of-life effects can be very illuminating, especially when considering the interests of third-parties who come to rely upon the output of the project. When a proprietary software producer declares a software product to be end-of-life (whether due to collapse/purchase of the company or some other restructuring) third parties are left high and dry. In effect, all proprietary software is directly equivalent to a single developer writing free software. The more important the software, the worse this gets because at least if the single free software developer does disappear, the third party can try and pick up the pieces.

The bigger problem for free software is the common failure to actually collaborate. This has been my main upstream bug bear for a long time now. The model must change. New contributors must be dissuaded from writing new code just because they prefer one language over another. Distributions need to change their guidelines for new developers to actively discourage new packagers from starting with new upstream code. We are in danger of undermining the main appeal of free software by not stamping harder on the NotInventedHere (NIH) syndrome. NIH is pernicious, NIH is dangerous and there is even an argument for considering NIH as an anti-feature.
New is considered harmful
New is not necessarily a good idea, new can be positively harmful to the interests of the wider software community. New promotes reinventing the wheel. It's another incidence of "Just because we can, does not mean we should."

If free software advocates neglect to counteract the poison of NIH, then free software will lose the argument by losing the fundamental merit of free software. A freedom which is not utilised is akin to not having that freedom at all. Yes, someone can pick up a free software project after the single developer abandons it but, in practise, what actually happens is that someone else comes along whilst the single developer is struggling along alone and decides to fall prey to NIH simply because the original is written in C and their preferred language is Python or Ruby or Haskell or whatever.

The "choice" argument is simply invalid - providing another implementation of program foo in a different language is just a way of reinventing the bugs already fixed in foo in new ways in a new language - and then adding some more which are unique to that language. Yes, the new developer might argue that the new code is more active but for how long? Long enough to become just as stable as the abandoned code? What benefit is that? In a few years time we will have two abandoned codebases in two different languages written by two different developers who never collaborated on the actual problem both of them were trying to solve independently. Lunacy.

The dilemma for distributions like Debian is that the list of Debian Packages that Need Lovin' is a burden to Debian during a release freeze yet provides an ideal launchpad for new developers to refresh stale code. I've long thought that Debian needs to be more strict on removing such packages, yet at the same time I cannot help but feel that this tends to encourage more NIH behaviour.

It's a common mantra amongst free software developers that many eyes make all bugs shallow. The problem for free software is that outside certain key teams, there are simply too few eyes and the community is not doing enough to shine a light on existing code before dancing to the sound of the NIH drum.

Far too much free software is at risk from that omnipresent bus called "RealLife" and this makes far too much free software little better than proprietary software. Pandering to the fashions of NIH only makes things worse.

This isn't a technical problem that can be solved with a new cool tool (which itself is subject to the NIH poison), it is a social problem and, sadly, the free software community has a history of failing to tackle purely social problems successfully.

There is a small role for existing technical solutions to make it easier to spot NIH tendencies, including sites like SourceForge and Savannah having greater scrutiny about new projects and distributors like Debian being more averse to adding yet more ITP bugs (or closing them automatically unless something happens within a month). Fundamentally, it comes down to how new and existing members of the free software community decide to handle their own projects and bugs. The model of "do it this way because that's more fun" has got us so far, the challenge is to remove NIH tendencies without removing that fun.

I received a trackback from Tim Bray s Saving Data Safely post on the Android Developer s blog to my Don t fear the fsync! blog entry, so I guess the cat s out of the bag. Starting with Gingerbread, newer Android phones (starting with the Nexus S) will be using the ext4 file system. Very cool! So just as IBM used to promote Linux by saying that it was scalable enough to run on everything between watches and mainframes, I can now talk about ext4 as running in production on cell phones to Google data centers. How much am I worried about Tim Bray s caution to Android programmers that they use fsync() correctly? Not a lot. Sure, they should make sure they use fsync(), or if they want to be clever, sync_file_range(), to make sure files are appropriately written to disk (or, in Android s case, to flash). But unlike Ubuntu s running on random PC s, with users downloading the latest (possibly buggy) Nvidia drivers, handset manufacturers test their systems very carefully before they let them ship. So if there are any problems, they tend to be found before the phone ships to end-users. So at least in my experience, my Nexus One has been very reliable; it s never crashed on me. So the chances of random crashes when you exit the 3D game (as one Ubuntu user reported and considered acceptable?!? I d be roasting Ubuntu and/or Nvidia under a slow fire if that was a reliably reproducible bug, not considering it par for the course) are very remote. And fsync() is important if systems crash or aren t shut down cleanly. Still, if users are randomly ripping out their batteries to turn off their cell phone in a hurry, because they re too impatient to wait for a controlled shutdown, then sure, we might have problems, and it s a good reason for application writers to use fsync() correctly. By the way, I had nothing to do with the choice to use ext4 on Android. So if you re curious about why ext4 was chosen, I can t say anything authoritatively, since I wasn t consulted before the decision was made (although obviously I m delighted). As far as I can tell after the fact, one of the reasons for choosing ext4 was better performance, especially in the light of dual-core ARM CPU s which are becoming available in large quantities in the near future; YAFFS is single threaded, so it would have been a bottleneck on dual-core systems. Why not btrfs? Well, for all of btrfs s features, it s not out of beta yet, and Gingerbread is shipping today. (Well, in less than a week to Best Buy, if we want to be precise.) This is another reason why I m glad to see ext4 being used on Android is that it validates my decision to keep working on ext4 2-3 years ago, even though newer filesystems like btrfs were promised to be coming down the pike. As I ve said many times before, file systems are like fine wine, and they take many years to mature. So having ext4 ready today is a great way of giving more choices to developers and system administrators about what file system they want to use. No related posts.

I received a trackback from Tim Bray s Saving Data Safely post on the Android Developer s blog to my Don t fear the fsync! blog entry, so I guess the cat s out of the bag. Starting with Gingerbread, newer Android phones (starting with the Nexus S) will be using the ext4 file system. Very cool! So just as IBM used to promote Linux by saying that it was scalable enough to run on everything between watches and mainframes, I can now talk about ext4 as running in production on cell phones to Google data centers. How much am I worried about Tim Bray s caution to Android programmers that they use fsync() correctly? Not a lot. Sure, they should make sure they use fsync(), or if they want to be clever, sync_file_range(), to make sure files are appropriately written to disk (or, in Android s case, to flash). But unlike Ubuntu s running on random PC s, with users downloading the latest (possibly buggy) Nvidia drivers, handset manufacturers test their systems very carefully before they let them ship. So if there are any problems, they tend to be found before the phone ships to end-users. So at least in my experience, my Nexus One has been very reliable; it s never crashed on me. So the chances of random crashes when you exit the 3D game (as one Ubuntu user reported and considered acceptable?!? I d be roasting Ubuntu and/or Nvidia under a slow fire if that was a reliably reproducible bug, not considering it par for the course) are very remote. And fsync() is important if systems crash or aren t shut down cleanly. Still, if users are randomly ripping out their batteries to turn off their cell phone in a hurry, because they re too impatient to wait for a controlled shutdown, then sure, we might have problems, and it s a good reason for application writers to use fsync() correctly. By the way, I had nothing to do with the choice to use ext4 on Android. So if you re curious about why ext4 was chosen, I can t say anything authoritatively, since I wasn t consulted before the decision was made (although obviously I m delighted). As far as I can tell after the fact, one of the reasons for choosing ext4 was better performance, especially in the light of dual-core ARM CPU s which are becoming available in large quantities in the near future; YAFFS is single threaded, so it would have been a bottleneck on dual-core systems. Why not btrfs? Well, for all of btrfs s features, it s not out of beta yet, and Gingerbread is shipping today. (Well, in less than a week to Best Buy, if we want to be precise.) This is another reason why I m glad to see ext4 being used on Android is that it validates my decision to keep working on ext4 2-3 years ago, even though newer filesystems like btrfs were promised to be coming down the pike. As I ve said many times before, file systems are like fine wine, and they take many years to mature. So having ext4 ready today is a great way of giving more choices to developers and system administrators about what file system they want to use. No related posts.

Ok, here we go: It seems that I'll receive the umbrella (big picture) in the first week of June, so I'm taking orders now. Please read this posting carefully if you want a Debian umbrella. Update IV 2010-12-12: Orders are now processed via debian.ch, so just go over there for your Umbrella. I still have a very few umbrellas here in Basel, so if you want to pick one up locally you're still welcome. About CHF 5 to 6 per umbrella will go to debian.ch (where it is held as official Debian money under the authority of the DPL.) (Old version of this article removed. You're still welcome to send money to my bank account, but you won't get an umbrella in return.)

Many times we can recognise different periods in our lives, but the line that divides those periods is not always clear. As I write this on the plane that is going to take me back to Argentina, I know I am closing something. 20 months ago, I took a plane in the opposite direction. The idea was to live and work in France for something like 7-9 months; I stayed more, got a Master's degree, met a lot of people, made some good friends, found a place that we would call and feel as home; and generally had a great time, so much that I almost didn't have time for any hacking. Good bye, France, the fond memories of this time are going to stay with me. Maybe someday, life brings me back. Tags: Planet Debian, Planet Lugfi, Vida

Went to the Chicago Symphony yesterday to see Regina Carter as well as Jazz wunderkind Esperanza Spalding perform one seat each with their respective bands. Regina Carter was presenting material from her current record 'Reverse Thread'. This was a real nice set of African-themed world music featuring Carter herself on violin, Yacouba Sissoko on kora, Will Holshouser on accordion, Chris Lightcap on bass and Alvester Garnett on drums. Some of pieces were really, really nicely done and I particularly enjoyed Holshouser on the accordion. After the break, Esperanza Spalding come on for her Chamber Music Society'. Lovely setup with Spalding on acoustic bass and vocals, Leo Genovese on piano/keyboards, Sara Caswell on violin, Lois Martin on viola, Jody Redhage on cello, the always impressice Terry Lyne Carrington on drums and Leala Cyr on backing vocals (and one co-lead in a really nice duet with Spalding). This was clearly more experimental and a chunk of the audience left during the act. But there is room for improvided chamber music, and it was a good modern music act. And Spalding is really quite impressive and I will gladly go and see her again.

As I continue to explore BitCoin, I've starting to wonder what properties the system have, and how it will be affected by laws and regulations here in Norway. Here are some random notes. One interesting thing to note is that since the transactions are verified using a peer to peer network, all details about a transaction is known to everyone. This means that if a BitCoin address has been published like I did with mine in my initial post about BitCoin, it is possible for everyone to see how many BitCoins have been transfered to that address. There is even a web service to look at the details for all transactions. There I can see that my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b have received 16.06 Bitcoin, the 1LfdGnGuWkpSJgbQySxxCWhv8MHqvwst3 address of Simon Phipps have received 181.97 BitCoin and the address 1MCwBbhNGp5hRm5rC1Aims2YFRe2SXPYKt of EFF have received 2447.38 BitCoins so far. Thank you to each and every one of you that donated bitcoins to support my activity. The fact that anyone can see how much money was transfered to a given address make it more obvious why the BitCoin community recommend to generate and hand out a new address for each transaction. I'm told there is no way to track which addresses belong to a given person or organisation without the person or organisation revealing it themselves, as Simon, EFF and I have done. In Norway, and in most other countries, there are laws and regulations limiting how much money one can transfer across the border without declaring it. There are money laundering, tax and accounting laws and regulations I would expect to apply to the use of BitCoin. If the Skolelinux foundation (SLX Debian Labs) were to accept donations in BitCoin in addition to normal bank transfers like EFF is doing, how should this be accounted? Given that it is impossible to know if money can across the border or not, should everything or nothing be declared? What exchange rate should be used when calculating taxes? Would receivers have to pay income tax if the foundation were to pay Skolelinux contributors in BitCoin? I have no idea, but it would be interesting to know. For a currency to be useful and successful, it must be trusted and accepted by a lot of users. It must be possible to get easy access to the currency (as a wage or using currency exchanges), and it must be easy to spend it. At the moment BitCoin seem fairly easy to get access to, but there are very few places to spend it. I am not really a regular user of any of the vendor types currently accepting BitCoin, so I wonder when my kind of shop would start accepting BitCoins. I would like to buy electronics, travels and subway tickets, not herbs and books. :) The currency is young, and this will improve over time if it become popular, but I suspect regular banks will start to lobby to get BitCoin declared illegal if it become popular. I'm sure they will claim it is helping fund terrorism and money laundering (which probably would be true, as is any currency in existence), but I believe the problems should be solved elsewhere and not by blaming currencies. The process of creating new BitCoins is called mining, and it is CPU intensive process that depend on a bit of luck as well (as one is competing against all the other miners currently spending CPU cycles to see which one get the next lump of cash). The "winner" get 50 BitCoin when this happen. Yesterday I came across the obvious way to join forces to increase ones changes of getting at least some coins, by coordinating the work on mining BitCoins across several machines and people, and sharing the result if one is lucky and get the 50 BitCoins. Check out BitCoin Pool if this sounds interesting. I have not had time to try to set up a machine to participate there yet, but have seen that running on ones own for a few days have not yield any BitCoins througth mining yet. Update 2010-12-15: Found an interesting criticism of bitcoin. Not quite sure how valid it is, but thought it was interesting to read. The arguments presented seem to be equally valid for gold, which was used as a currency for many years.