... or why PHP sucks so much?

It is quite common (at least for me) finding abused web applications on shared hosts around. The most typical case is finding some kind of IRC bot running as www-data and a few unauthorized files around used for phishing, cross-site scripting or what else. That motivated me to try playing with mod_chroot in order to minimize the possibility of webapps abuse from our friendly kiddies^Wcrackers. Mod_chroot is a nice tiny Apache module whose purpose is the confinment of webapps within a limited tree, where nothing but a few files and dirs are available to try exploits and abuse badly written code.

Unfortunately, I found that mod_chroot poses one major problem (among others, see its CAVEATS doc): the sucking mail() function is not working out of the box, because PHP folks - God knows why - decided to implement that stuff by calling a local /usr/sbin/sendmail program within a shell call. The most sane option is simply ignoring the issue and living happy with a disabled local mail() function. A nice solution in that case is using a PEAR Mail module, which is able to send mails via SMTP in much more elegant way. Unfortunately, there are quite a good number of morons out there, and that could be not a viable option if you are the system administrator of a bounce of shared hosts which are exposed to those morons, who absolutely need a working mail() function.

After a few googling around, I did find that many people had the same problem, but none found (or explained) a working and neat solution for a working mail() implementation, so I'm writing some notes about that. My implementation is based on a nice tiny program (mini_sendmail) which has the great advantage of not requiring a configuration file or a spool directory to work. It also works without suid bit on, which is a good thing as well. It is statically compiled by default, so it simplifies things a lot. In order to have mail() working you need also to install a (statically compiled) shell, e.g. bash-static as sh under the /bin directory of your chroot tree (a more tiny shell would be appropriate). I commented out the silly username autodetection code in the mini_sendmail.c source, because it creates some problem within an empty chroot tree (it failed with a "can't determine username" message with or without the /etc/passwd file available).

Now, the next step is using an entry like

sendmail_path = /usr/sbin/sendmail -t -fwww-data@your.domain -s127.0.0.1

in your php.ini file. You can also use another IP address if your SMTP server is not your localhost or you used some more exotic routing setup. Having a suitable /etc/hosts is also useful. A final trick is adding a var/lib/php4 or what ever directory is required to store PHP session files. No other files should be required in order to have all working. Of course, you have also to use inet sockets for any required connection (e.g. mysql) but this is typically the default in etch. I would also suggest to add only ad hoc statically-compiled binaries within the chroot tree when required, else you will need to store all required shared libs: in that case you will need to find binaries requirement by using ldd and chroot to find loading/running errors by means of an usual trial-and-error cycle.

Using XMPP4R, I did some stats about Jabber clients usage on the Apinc Jabber server, which hosts im.apinc.org, jabber.fr, and many more using virtual hosting. I already did similar stats in March 2006 and September 2005. The poll was done by sending jabber:iq:version to online users, around 1:00 PM (french local time, most of the users are french). 1343 clients were pinged, and 1315 answered, which is better than last year (1145 answers). Of the 1145 jids that answered last year, 368 were also part of the poll this year (I don’t know if this is good, or not enough). Systems:

• GNU/Linux: 43% (2006: 38% ; 2005: 34%)
• Windows: 35% (2006: 37% ; 2005: 34%)
• Mac OS: 16% (2006: 18% ; 2005: 23%)
• Unknown: 4 (2006: 5% ; 2005: 6%)
• Others: 0% (12 clients ; 2006: 0% ; 2005: 1%)

Clients:

• Psi: 24% (2006: 28% ; 2005: 28%)
• gaim: 22% (2006: 25% ; 2005: 25%)
• Gajim: 12% (2006: 5% ; 2005: 3%)
• Kopete: 11% (2006: 7% ; 2005: 7%)
• iChatAgent: 9% (2006: 13% ; 2005: 18%)
• libgaim (Adium): 5% (2006: 4% ; 2005: 3%)
• Pandion: 4% (2006: 4% ; 2005: 2%)
• Miranda: 2% (2006: 2% ; 2005: 1%)
• BitlBee: 2%
• neos: 1%
• Unknown client: 0%
• Exodus: 0%
• Imendio Gossip: 0%
• Jabbin: 0%
• Spark IM Client: 0%
• Trillian: 0%
• JBother: 0%
• jabber.el: 0%
• JETI: 0%
• JAJC: 0%
• Class.Jabber.PHP: 0%
• Jabberwocky: 0%
• Tkabber: 0%
• Gush: 0%

(all clients with at least one reply are listed here) I also did some stats on the Linux distros. With 566 Linux users, one can consider that statistically signifiant.

• Client answers with the kernel version, not including distribution information: 62%
• Debian: 14%
• Ubuntu: 12%
• Gentoo: 2%
• Unknown distros (not provided by the client): 2%
• Fedora Core: 1%
• Arch Linux: 1%
• Mandriva: 1% (despite the fact that most users are french!)
• Slackware: 0%

(Other distros are below 5 replies) The Debian/Ubuntu situation is interesting. Debian is not dying, even on the desktop ! The server is hosting jabber.ubuntu-fr.org, so the results are biased towards Ubuntu. Update: it seems that Psi reports “Debian GNU/Linux (testing/unstable)” even when running on Ubuntu. This is the case for ~5% of linux users. To server admins: if you are running a large jabber server with jabberd 1.6, it’s easy to give me the right to get the list of online users, so contact me if your are interested in me doing your users stats. It would be interesting to see if users from different servers/countries have different behaviours. I’m not sure if it’s possible with ejabberd and other servers.

FSFE launched a raffle to encourage more people to join. I don't find it particularly attractive, which is probably partly because of my statistics education. I'll join FSFE when they stop promoting non-free-software like the notorious FDL. As it turns out, my reaction was pretty mild compared to some existing fellows. There was a discussion on the main FSFE international list flaming them for including the "likeable mask"-wearing Nokia N800 and an ex-Qt-locked Greenphone among the prizes. GNU Project Chief Webmaster Matt Lee wrote:

"I don't think this is appropriate, or wise. Further, it makes me want to not support FSF Europe."

Pretty early on, FSFE's Jonas Oberg wrote:

"Actually, this is news to all of us. [...] if there's no way to liberate the hardware before sending it out, we might well have to withdraw those prizes from the raffle to avoid sending non-free software to anyone."

but the flamefest continues unabated. As is sadly common on free software-related lists, the moderators have not stepped in with the fire blankets yet. (Some also found it worth reposting old lies about debian, which drew me onto the list to rebut them. sigh Way to go FSF fans.) Aside: Savannah has suffered a hardware failure in case you were wondering where cvs.savannah.* went.

Erich Schubert comments on the issues relating to getting big changes into Debian. This is something that I had also noticed. I started work on SE Linux in Debian in 2001 and continued it actively until 2003 when I joined Red Hat. Less than a year after I joined Red Hat there was a Fedora release with SE Linux fully integrated and shortly after that there was a release with SE Linux on by default. The reason for this was that Red Hat management supported the idea of SE Linux and everyone had to accept it. There was no option for a package maintainer to refuse to support SE Linux.

Recently in a discussion on debian-devel one DD (who I won't name in this blog post) advocated removing SE Linux support from dpkg. I then asked him whether he had the same attitude towards non-executable stack
(Exec-Shield/PaX/OpenWall), Poly-Instantiated directories, and PIE executables. When he expressed interest in having those features I pointed out that one of the enemies of security in Debian is the fact that every person controls their little area and has no requirement to work towards common goals (apart from the most obvious ones of making the system work).

This means that instead of having a little cooperation from other developers anyone who wants to get a significant change included will have to fight hundreds of battles.

SE Linux is a classic example of this. Debian could have had SE Linux support long before Fedora, but instead it gets it long afterwards.

The same battles occur with regard to all the other security measures I mentioned (and some others I didn't). We could made Debian the most secure Linux distribution, there are many people who have the skills and the interest in doing so.

If you want features such as exec-shield, then you are missing out - largely because the people with the skill and time to work on them are too busy fighting trench-warfare rather than actively coding.

Now while I strongly object to most incarnations of the "you can't force a volunteer to do anything" meme that infects Debian I do agree that we can't force developers to write new code. We can however strongly discourate an antagonistic attitude towards new features. If someone proposes a feature
that you don't plan to use but which doesn't hurt you then there's no reason to attack - you can just ignore it. If someone sends in a patch that adds a feature which is requested by many people but you personally don't use, then if it has little or no down-side (linking against a couple of shared objects as is the case for many SE Linux enabled programs provides no measurable overhead) and the code is good it should be merged!

The real problem is that some DDs are more concerned about what is best for them personally (in the most short-term manner) than about what is best for the users.

The aim of Debian is to integrate software produced by other projects to create a complete operating system based on free software. In recent years, the project has faced increasingly delayed and unpredictable releases. Most notably, the release process of Debian 3.1 was characterized by major delays. Initially announced for December 1, 2003, the software was finally released in June 2005: a delay of one and a half years. Since then, the project has made a number of improvements to its release process.

Version	Date	Months
1.1	1996-06-17
1.2	1996-12-12	6
1.3	1997-06-02	6
2.0	1998-07-24	14
2.1	1999-03-09	7
2.2	2000-08-14	17
3.0	2002-07-19	23
3.1	2005-06-06	35

Past Problems

• Release management was not very organized and release updates were posted only infrequently. Because of this and the lack of a roadmap, freezes were often announced out of the blue.
• Due to the unorganized nature of the release, several new and unexpected blockers were found during the release process, leading to delays.
• The unexpected delays meant that software was frozen for a long time, in the case of Debian 3.1 for over a year. When this release was finally published, many components were already out of date and would often not meet user demands.
• The fact that Debian has experienced significant delays with several of its recent releases has led to problems with the image of the project. There is a perception that Debian is slow and cannot meet deadlines. This is also associated with frustration in the developer and user community.

Solutions

• The project has implemented better release management structures. In particular, Debian moved from a single release manager to a team during the 3.1 release cycle. The release is now handled by two release managers with the help of several release assistants.
• A release date for the next release was set well in advance and there is more planning.
• Release announcements are sent more frequently and various information sources have been implemented through which developers can stay informed about the release status.
• Release targets have been defined better and there is a distinction between blockers and goals. Only blockers hold up the release whereas goals can be postponed for a future release. For example, the XFree86 to X.org transition and the integration of AMD64 support were blockers, while LSB 3.1 compatibility and SE Linux support were only goals for Debian 4.0.
• The release team has shifted the responsibility of achieving targets to specific developers and teams. There are also better criteria now defining these responsibilities. For example, criteria have been published that architectures need to fulfil for inclusion in Debian 4.0 and there is a page indicating the status of each architecture.
• The release team is increasingly giving encouragement to developers to help out with software packages and bugs which normally do not fall into their domain.
• A staged freeze has been implemented according to which software is frozen in stages according to its importance. The toolchain and base system are frozen earlier than the majority of package. The majority of software is now frozen when most blockers have been resolved.
• The use of the experimental repository has been promoted to make sure that the main development repository is in a good shape most of the time.

Outstanding problems

• Developers still need to be convinced that targets can be met, that deadlines are real and that Debian can release on time.

[ Warning: Long post follows ]
Just today, I read that H01ger is happy he can wear just a T-shirt somewhere in Northern Germany. Well, from my point of view, much the opposite has happened. And I'm not just happy - I'm fucking proud!
About four years ago, Nadezhda went almost every weekend to hike to the mountains with the Grupo de los Cien people. I'll sidetrack a bit to speak about them: Grupo de los Cien is a group of mountain lovers (no relation with the enviromentalist group which later took the same name) that, since the 1950s, have built and maintained the high mountain shelters in Central Mexico. It is a group, by the way, with which Nadezhda fell completely in love - and now it's up to me to find out why ;-) But lets go back to the main topic.
Almost every week, she came back delighted and loaded of energy from a mountain hike. Almost every week, I told her I wished to go with her, of course, if they went to a simpler route. Of course, lugging around over 130Kg of humanity is no easy deal. And, after over a year and because of many problems that came together at once, she stopped going to the mountain as well. She always kept an eye on their friends, but so far had not been able to return to the mountain.

Ok, so last week she heard the group was organizing volunteers to go fix the Ayoloco 2 shelter. She signed up, and invited me. I was afraid, but accepted the challenge - Of course, full of fear.
Being this a historical event (at least in my life), I cannot but offer you some of the photos I took. And, yes, I've just started playing with Flickr. Lets see if it works for me ;-)
Saturday, 5 AM, we woke up. 5:50AM, we met part of the group near Daniel's house, in Condesa. 7 AM, we were having a nice breakfast (tamales and coffee, yum!) in Amecameca, still 28km from La Joya. From Amecameca, we went to Paso de Cort s, where we met the rest of the group, and from there we went by a little, bumpy and unpaved road until La Joya. Somewhat around 20 people started the hike at around 9 AM.


According to Google Earth, La Joya (just South of the Iztacc huatl - white woman or sleeping woman in N huatl) is at 3995m - And, of course, to start at that altitude is not easy. The air is very thin, and walking a little bit too fast quickly makes you feel the blood pumping, trying to get some more oxygen. I often remembered how my Bolivian friends behave in Potos , walking slowly even when in a hurry :)

We first went down a little valley, then up. The first portion of the hike was on soft, wet sand, with some grass (what we call zacate - Strong, hard grass, not what you commonly see in a garden). I soon learnt that was a blessing, as it was by far the easiest part. Then we went up a steep and quite rocky area - It started getting tricky for somebody not used to trusting his own feet. And it was steep, yes. This was the first time (of many, of course) I started thinking whether I should head back.

We reached what I foolishly thought was the summit, only to find a steeper, longer way to go. It started getting common to find little deposits of snow/ice (I don't know the exact term in English - it's called aguanieve in Spanish, literally snow-water. [[Update]: Thanks to the now-DD H01ger, I now know that aguanieve in English is called sleet]. I understand it's a bit harder than snow, but still much lighter than hail). We crossed a completely rocky section - No sand to hold the rocks to their place. That meant being extra careful (and thus, extra slow).

But just afterwards, in case it was not enough, Mother Nature answered my pleads for some sand - we crossed a section made exclusively of loose sand. And, please, if you have ever walked upwards on sand, imagine doing so at over 4000 meters. I was taking so much care of how and where I stepped that I strayed not more than 5 meters down of the path the group was following - Being able to climb back to the right path was really not easy. This was the point that not only I was thinking about heading back, but I'm sure Nadezhda thought I would abort the mission. She cheered me up, took some pictures of me as soon as I got out of the sand trap, and we were able to move on.

After the sand, some more rocks, and we started feeling the chilly wind. We entered a cloudy region - Humidity does get the coldness into your bones! We could not really see where was the rest of the group, so we proceeded the best way we could. Fortunately for Nadezhda and me, just behind us came the very experienced Mario Corsalini and led us. And, yes, we finally saw the Ayoloco 2 shelter.

I was by then exhausted, after some four hours of ascent. I ate one of my apples - The sweetest apple I have tasted so far. After resting a bit, and realizing I was almost frozen (we were at 2 Celsius under, with constant chilly, humid wind)... Then I started working with the guys, reinforcing the shelter, while Nadezhda and some other people painted the inside with. People started eating - of course, I joined them. And, after some time up there, we started going back down. Miguel ngel, a volunteer in the Izta-Popo park, walked a good portion of the descent with me, giving me tons of help and tips. Thank you, thank you, really. The descent is, of course, much easier - We were chatting most of the time, and we made only around two hours back. The weather was quite nice on us - We had some aguanieve falling every now and then (yes! Yes! many of you guys have heard/read me bitch on how I had never seen snow falling before - Ok, this is not exactly snow, but I think it qualifies as my first experience ;-) ), but aguanieve does not make you wet, as it bounces off the clothes. When we approached La Joya we started having some light rain. We hopped in the cars, and then the rain really began.
But even if this was not enough: Grupo de los Cien was celebrating the 83th birthday of one of its senior members, Poncho Rico. No, being 83 years old is not an excuse for skipping this memorable hike.

We went to Anatolio's house, very close to Amecameca, and had a delicious dinner, cake and drinks.
Nadezhda: Thank you. So, so, so very much. Thank you for being so patient with me, for dragging me up the Sleeping Woman. Thank you for getting me into this so very important and beautiful experience. I hope this will be the first of many. There is still a very long way for me to go before I'm really up to the challenge, but believe me: This will not be my last time on the mighty mountains.

(Update: Yes, Gunnar, the policy says that the debian/rules file must be a makefile, so if you do something else, you have a serious bug because of a policy break. Of course, probably there is nothing that should stop one from using, let's say bash, or python or perl, from a technical PoV.)

Erich has a wish list for Lenny.

What draw my attention was the very first "define and deploy a standard way of packaging software, that can be used for like 90% of packages", later he mentions rationale "other distributions (e.g. *BSD ports, Gentoo?) are ahead of us in this respect, having a standard way of packaging and building things and keeping track of changes."; these two quotes contain the real essence of Gentoo's success in the area of fast development.

A half a year ago I did an experiment in packaging a rpm, a deb and an ebuild for the same piece of software. What I observed back then made me think a lot about the entry level knowledge needed for a person to create a decent package for those types of package systems.

With the risk of being flamed by many people that will miss my point, I will tell you from the start that the deb would have been the most painful if had to start from scratch. Second would have been the rpm due to lack of good/updated documentation and the clear winner in lack of pain points was the ebuild. In reality places were switched between the deb and the rpm.

And before I start, please understand that what I want to point out is that the building system in debian is showing its age and we should do something about it, and is not about pointing people to documentation; it is about making things behave more sanely by default and not about having to read a whole day to understand why the package fails to install a file you just added then to find that you had to add the destination path to a debian/dirs file etc...


The deb had to be the most difficult. Not because I didn't do it before, or because it was hard, but because there are a lot of things that need to be done because the package management is not smart enough to do it by default (i.e. class inheritance does not exist, like is the case for the ebuilds or the macros for rpms).

The Debian approach is backwards, is reversed with regard to the others and the way it should be by default, to allow easier understanding of the packaging. You can, by default, make a package that is tweaked a lot. There is no concept (if you don't count cdbs which is boo-ed a lot by some people) or possibility to just say "make this package a package that is a standard linux app with the regular ./configure && make && make install recipe". And I don't count such a recipe the templates based on dh_make, that is way too verbose. A new maintainer wannabe has difficulties understanding why are those lines in there, and what they do, and most importantly, why is that order correct and no other. Another question which is legitimate is "is that the minimum or can that be trimmed"? (Follow debian-mentors and see how many people leave the .ex files just because they are afraid they will not know how to generate them again.)

Even the rpm has a concept of standard stuff to do for a package, there are some macros that implement that. And that is more logical for an untainted mind (as in not yet debianized) than what we have in the dh_make template, or worse, a clean and new debian/rules file. Why should one care to learn the ways and philosophy of make just to make a small package?


Let me tell you something that will feel like knife stab for many people in our community: the clear winner was the ebuild, not because I was smart or exciting, but because:

• it was dead simple to make a working one
  
• the documentation is really verbose and complete
• there is a default behaviour for every ebuild, even for a minimalist ebuild
• if your ebuild has a special need, the ebuilt can override the default behaviour by redefining the function (target, in debian/rules-speech), but you don't need to, in most cases
• there are some already defined classes which can be inherited to obtain a tweaked default behaviour (imagine typical cmake build system, typical autotools build system, typical kde build system - and iirc, this can include build dependencies)
• all the information about the package is centralized in one file which is human readable from the start, plus that is obvious, or one can understand fast what a field/function/variable means from the start, without digging through the documentation continuously
  

I did the ebuild in one day's work (it was less, but some will not believe me, so I'd say 1 day is reasonable enough) starting from absolutely 0 knowledge about the packaging system (if you don't count knowing to emerge package as knowledge).

What is done properly in the portage system is the fact that there are some default classes which your ebuild can inherit to behave more like a certain type of packages (base - a generic recipe, gnome, kde, games, sourceforge - they download tarballs directly, if needed - ...)

Why this is not possible for deb right now? Simple, we have it as a rule in the policy that the debian/rules file is a makefile. So even if one would implement a class-like model for deb packages, you'd still have the debian/rules file as a make file...

Gentoo implemented initially portage's default behaviour (and as a consequence, the "classes") in shell, later they migrated some parts to python and the whole portage is implemented in python now (while classes are implemented in bash). I am not saying that we should put python in essential, God forbid, probably the emdebian guys would stab me :-), I am saying that possibilities exist, alternatives should be possible.

I know, some will just say "show me the code", "making such a change over the entire archive is insane" or "the makefile is a really good way to do things like installing and building, after all, that's what it was designed for". Yes, all of these will miss the point, which is having nothing-to-do as a default policy for a package is a bad policy, it should be an empty/default package will use a standard recipe to build itself, and that recipe should be implemented in the build system.


And if you still don't believe me, just look at these few ebuilds and tell me you'd not be able to write them after a quarter after seeing one the first time and skimming over the docs or/and knowing what a regular gentoo user knows (what a keyword is, what is a slot). Also compare with the debian counter parts and don't forget the different building aids that we have.

http://sources.gentoo.org/viewcvs.py/gentoo-x86/games-arcade/amphetamine/amphetamine-0.8.10.ebuild?view=markup
http://svn.debian.org/wsvn/pkg-games/packages/trunk/amphetamine/debian/rules?op=file&rev=0&sc=0
http://svn.debian.org/wsvn/pkg-games/packages/trunk/amphetamine/debian/control?op=file&rev=0&sc=0
http://svn.debian.org/wsvn/pkg-games/packages/trunk/amphetamine/debian/postinst?op=file&rev=0&sc=0

http://sources.gentoo.org/viewcvs.py/gentoo-x86/games-arcade/holotz-castle/holotz-castle-1.3.8.ebuild?rev=1.3&view=markup
http://svn.debian.org/wsvn/pkg-games/packages/trunk/holotz-castle/debian/rules?op=file&rev=0&sc=0
http://svn.debian.org/wsvn/pkg-games/packages/trunk/holotz-castle/debian/control?op=file&rev=0&sc=0


http://sources.gentoo.org/viewcvs.py/gentoo-x86/games-arcade/late/late-0.1.0.ebuild?rev=1.10&view=markup
http://svn.debian.org/wsvn/pkg-games/packages/trunk/late/debian/rules?op=file&rev=0&sc=0
http://svn.debian.org/wsvn/pkg-games/packages/trunk/late/debian/postinst?op=file&rev=0&sc=0
http://svn.debian.org/wsvn/pkg-games/packages/trunk/late/debian/postrm?op=file&rev=0&sc=0


http://sources.gentoo.org/viewcvs.py/gentoo-x86/games-arcade/ppracer/ppracer-0.3.1.ebuild?rev=1.15&view=markup
http://svn.debian.org/wsvn/pkg-games/packages/trunk/ppracer/debian/rules?op=file&rev=0&sc=0


One more thing, the ebuilds mostly need not be updated when an upstream release occurs, they just need to be copied and given a different filename.

Big night last night at South by Southwest. I went to the Yahoo! party, which was packed and had an open bar with call drinks. I drank Bombay Sapphire with tonic and picked up a pipes.yahoo.com and a upcoming.org T-shirt. Clean T-shirts are the best thing to have at the end of a long conference. Then the "South by Northwest" party, for Canadians and those who love them, and the Creative Commons party across the street. I got to hang out with my friend Nicolas Ritoux and Armand Geddyn, which was a pretty potent recipe for staying out late and acting weird. It was a little hard to get to the conference this morning at 10AM, but I was a little relieved that my dozens of tipsy twitter messages didn't actually get broadcast for some reason. There was some funny material in there, but I also lost my ability to spell after the third party. tags: sxsw07 austin niko party

Eddy, while I agree that ebuilds look a lot cleaner, shorter or easier than typical debian/rules + debian/control, you should consider a few things:

• There is cdbs in Debian, which provides a lot of help in packaging “easy” packages. Note though that I don’t really like the hidden magic it does.
• ebuilds don’t have to handle architecture independent data, since Gentoo doesn’t have such packages. If Debian wouldn’t have architecture independent data, debian/rules would look easier immediately.
• Updating an easy package (i.e. one without Debian specific patches) to a new upstream usually is a no-brainer in Debian as much as in Gentoo.
• With ebuild classes, you still have to know/learn which class matches your package best, so you still need to know what each class actually does. That isn’t too different to understanding dh_* stuff, if you ask me.

So all in all, I’m quite certain that the amount of learning needed to create a correct ebuild is lower than that needed to create a Debian source package, but I don’t think that packaging for Debian is hard either. And finally, Erich is quite right that a central repository for all Debian packaging scripts would be a nice thing. However it would be quite hard to determine the right VCS to use for that. Most de-central VCSes I glanced over need manual intervention on the “server” side to include patches from local developer branches. And of course there is the problem of several packages having a non-clean upstream tarball they need to repack for each release as well as several packages having a tarball which includes more than one single upstream tarball. You would need to find a way to handle those as well.

Been trying out upstart today. It's a nice piece of code, but unfortunately it's totally useless today. Not because its concepts suck; rather, because Debian packages today ship with SysV-style initscripts, and those do not integrate well with upstart: most of the features that make upstart so interesting require you to create an event.d script, which has a totally different syntax. The Debian Developer inside me couldn't help but start to think about a way to fix this, but that seems harder than what it initially would sound like to the uninitiated. My first idea was to add some /sbin executable to upstart or one of its supporting packages, so that a SysV-style initscript could exit if that binary told it that upstart was running, before doing anything. That way, a package could ship both upstart event.d files and SysV-style initscripts during a migration period, and we could get rid of the SysV-style initscripts when and if Debian eventually decides to migrate to upstar. But when I sent that past #debian-devel, someone mentioned initng, which is yet a different init implementation. If this would work for upstart, it could also work for initng, and I don't think many packages would want to ship initscripts for three different init implementations. So scratch that. My next idea was to generate initscripts, rather than have packages ship them. The amount of common code in our initscripts is huge, anyway, and it's not as if it's all that hard to write an initscript. The things an initscript needs to do are limited, anyway: start a daemon when requested, stop it, restart it, perhaps send it a SIGHUP if or when configuration updates have been performed, and provide some useful output on all that. In most cases, that's about it. So, I thought, why not create an RFC822-style file (analogously to other files in the debian/ directory of a debian source package) containing all the metadata on an initscript, and generate the correct style of initscript out of that? That should work, I thought. So I started writing a specification, and had a look at the initscript of one of my own packages, nbd-server. Which does this at some point (nonrelevant content cut out):

i=0
while [ ! -z $ NBD_FILE[$i]  ]
do
  start-stop-daemon --start --quiet --exec /bin/nbd-server (...)
  echo -n " $ NBD_FILE[$i]  ]
  i=$(( $i + 1 ))
done

There's no way you'll be able to express that type of flexibility with an RFC822-style file containing all the metadata. So, scratch that too. I still do think generating initscripts is the right thing to do, however; the question that remains is "how". One idea I had was to use m4 in an autoconf kind of way, since autoconf allows one to mix autoconf macros with "regular" shell snippets, which would be exactly the kind of flexibility that one would need. However, that totally ignores the fact that autoconf is only able to pull this off because its output is shellscript in every case. An m4 macro that has to produce output which are not, in fact, shellscripts (such as would be the case for this project) could not do that. I think I'm kinda stuck now...

Eddy, the problem with packaging software is that it is not a once-off thing, but the package needs constant maintenance, and for that, I need to be able to understand how the build process works (on a large scale). While at first the rules file generated by dh_make seems huge, it is actually quite easy to understand if you have the output of "apropos dh_" printed out and are not afraid to look at the occasional manpage.

Here's an "interesting" phenomenon I noticed on my new (not-Linux-hating) 4GB portable music player:

daniel@jeeves:~$ df -h /media/IAUDIO
Filesystem Size Used Avail Use% Mounted on
/dev/sdb 3.8G 3.8G 0 100% /media/IAUDIO
daniel@jeeves:~$ du -hcs /media/IAUDIO
2.3G /media/IAUDIO
2.3G total

Just in case "df" and "du" had different notions of how large a gigabyte was, I asked them to tell me the number of bytes instead, with about the same result.

I won't comment on the fact that I paid for 4GB but got 3.8GB; that's typical (whoops, I just did). But where'd that extra 1.5GB go? FAT can't seriously be imposing a >50% overhead on file size, can it?

The end result here is that on my 4GB music player, I can store only a little more than 2GB of music. Boo. :-(

[UPDATE]: running fsck revealed that I had 1.5 GB in lost clusters on this filesystem and gave me my space back. Since this device is almost brand-new, I'm puzzled about how this happened...but oh well, it seems to be fixed easily enough.

München (Munich!) has a great swing scene - alive and kicking. Lindy Hop [wikipedia], Balboa [wikipedia], Boogie Woogie [wikipedia] (~ East Coast Swing) and Rock'n'Roll [wikipedia] (the acrobatic version) all over the place. Since starting dancing (RnR some years ago, Lindy a year ago, all just socially until about half a year where I took on the others as well and started going to some real trainings), they've become an essential part of my life that I wouldn't want to miss. If you happen to come to Munich, here are some links for you:

• Swing and the City, the homepage of Christine von Scheidt, a world class dancer and trainer as well as judge in some contests gives Lindy Hop classes that are excellent.
• The Cord (Sonnenstraße, very central location in Munich) has a great cosmopolitan flair and a swing night every monday, often with live music (which may or may not be great for dancing). Cat and Chris of 'Swing and the City' give teaser classes there, too.
• Salon Erna at Optimolwerke (near Ostbahnhof, also quite central, at the big party areal in Munich) is the home of Swing on Sunday. There is a beginners class as 19:30 with changing trainers and topic (usually Lindy Hop). While a bar like Cord, there are mostly dancers here.
• Swing it! is organizing social dancing events with livebands about once a month. Unfortunately, the previous location closed down (not due to Swing It, which got the place really crowded), but they'll hopefully find a new home soon.
• Boogie-Bären is the dancing club I'm in. They have four Lindy Hop classes on Tuesday, two Balboa classes on thursday and Boogie Woogie classes on most days.
• Near Fürstenfeldbruck (north-west of Munich) there is the Tanzfabrik Überacker where Fabi and Gino will take good care of you; I learned a lot from them in their classes at Salon Erna. They got me addicted to Lindy Hop. :-)
• Keep on Swinging, is the home of Fabi and Chrissi. They are giving occasional workshops at the Tanzfabrik
• If you are at the university for some time, got to the University Sports Programm. It's exceptionally cheap (like 7.50 EUR for a term for a regular student), and that includes lots of sport offers, such as a Rock'n'Roll and Boogie Woogie class. Thats where I started dancing before going to Berkeley, discovering Lindy Hop there and continuing it in Munich later on.
• Claudia and Horst, who also teach at Boogie-Bären, also give classes in Freising, north of Munich.
• And of course World of Swing: Markus and Bärbel are very well known dancers and trainers. If you are into swing, you probably have seen or met them already. They are giving Balboa and Boogie Woogie-Classes and organizing large dance events such as Rock that Swing or the "Munich Balboa Weekend".
• In summer, there will hopefully be a weekly dance event in Hofgarten park again, at the Diana temple.

P.S. My personal favourites / recommendations: Go to Cord or Salon Erna to see how much fun the community is, then take classes with Swing and the City or Keep on Swinging.

I've started using Yahoo Pipes today, and it's pretty cool. I wouldn't go so far as to call it a "milestone in the history of the Internet", but there are a few interesting use cases. Say for example if you're an avid reader of Planet Debian and you don't like the French Revolutionary Calendar, then you can use Pipes to filter out entries whose title matches a French Revolutionary month and enjoy a 100% revolution-free Planet. Or you could split matching entries into a separate feed, or just use regexp replacement to translate them into something else, etc. The possibilities are endless.

There's also a handy Babelfish module, so you can create a pipe that takes Planet Debian, translates it to French, then back to English. Hours of fun!

We the undersigned petition the Prime Minister to promote the use of Open Document Format within the UK government.

Government documents must be available for tens if not hundreds of years. Currently much electronic documentation is stored in proprietary formats, such as Microsoft's .doc format. In order to allow future generations access to these documents it is imperative that they be in a fully documented standard. Open Document Format (ISO/IEC 26300:2006) is now the international document standard and as such should be supported by the Government.

The Weston Mercury - News: Council chief probe costs 34k

This happened in North Somerset and I remember something similar happening in West Norfolk. When will councillors stop using expensive lawyers to defend themselves at the taxpayers' expense?

opensource - epetition response

Ambivalent reply from the PM's office.

Town and parish councils

Kewstoke is used as a case study, but the link goes nowhere now...

The Irascible Professor-commentary of the day 02-08-07. Movie star teachers.

I've given up watching hero-teacher films. It's really not like that. I guess the oppressive bureaucracy of state education, with pressure from executive teachers to fiddle the student numbers, wouldn't make a good film.

The Irascible Professor-commentary of the day 03-06-07. Why I hope mhy children won't become educators.

Don't let your children grow up to become salesm^Wteachers.

NO2ID:newsletter 66

"Saturday, 10th March - King's Lynn street stall to advertise forthcoming public meeting, from 10am. Volunteers needed! Contact kings.lynn@no2id.net or text/call James on 07778 966395."

Tories want to free (software) Whitehall

The shadow chancellor gets all over the BBC as a result of his party's latest conversion.

Neighbourhood Fix-It

Another better-than-local-councils system from MySociety.

Concerto for blunt instrument: Ode to the No Longer Active

"We held a place for you // But you were busy // You've been so busy lately."

Yesterday ICANN published an interesting article about last attack of root servers (6 February 2007): see http://www.icann.org/announcements/announcement-08mar07.htm . The paper is not so detailed and technical as one should expect, anyway it is worth to read it.

It seems that also the remaining root servers should move to anycast. I think this is good, but it will works reliable with TCP? Do someone use TCP to query DNS servers? Why is it not disabled? It seems that all root servers accept TCP ( dig @l.root-servers.net. . NS +tcp).

A side note: as you can see, root servers don't give you yet the IPv6 addresses of root servers. Now they do some test, prior to broke the root server 512-bytes packet rule. We are still away to the full IPv4/IPv6 inter-operative nets.

The last recommendation is: "ISPs should only accept DNS queries from trusted sources (i.e., their own customers) rather than allow anyone to use their servers." . This rule (on recursive queries) is already a well know rule, and I I think it is less problematic of mail-rely (ev. with SPF), but on the other side, we are moving to the point that we should trust our ISP and our ISP will firewall "non-proxied" traffic.

As last point, the fact sheet cite two wikipedia articles. Wikipedia is so good, or there are not better (updated) documentation on the net?

It's been a good few days on our trip to South by Southwest. We flew out of Montreal on Tuesday afternoon with temperatures around -22C and got to Houston that evening with a temperature of 77F. Nice. We stayed in a Best Western near the airport in Houston, but with the Houston Rodeo currently going on, we didn't see much point in staying in the overcrowded city. So yesterday morning we did a nice drive from H-town to San Antonio, with a stop in Schulenberg for lunch at the Oakridge Smokehouse. We checked into the Quality Inn and Suites - Northwoods just inside the 1604 loop, and took a well-deserved nap before going for a stroll on River Walk and getting dinner at Casa Rio, right on the water. This morning we drove from San Antonio to Austin, with lunch in New Braunfels. Great Mexican food at El Tapatio, which was nice. After negotiating Austin's perpetually crappy I-35 traffic, we checked into the venerable Driskill Hotel right by the Austin Convention Centre and 6th Street, and took a break for a while. To top off our driving week, we headed up to Pflugerville where our friends Armand and Barbie Geddyn live -- operators of Austin's local SubGenius clench, the Ministry of Truth. Fun folks, and we got to do some grilling (fresh shrimp from the coast) and hang out with old pals. Tomorrow is the first day of registration for SxSW, and we're going to have a Quebecker meetup at the Registrants' Lounge at Brush Creek Park at 4:30PM. Should be a good time. I still have to put the finishing touches on my presentation for Saturday afternoon, so I'll probably spend tomorrow in the hotel room or in a nearby Internet caf . My presentation is at 3:35PM on Saturday; I'll post the slides here and on SlideShare after it's over. tags: travel sxsw07 minitru austin houston san antonio schulenberg new braunfels

CACert on certifi.ca I've had some good feedback from CACert users on certifi.ca this week. The OpenID server now supports CACert certs, and it's a pretty active community, so folks have been very supportive. Thanks to all for your feedback and help. tags: cacert certifi.ca

Here are some scenarios for ways that I want to use Debian on semi-embedded and embedded devices. So far there's not a good way to accomplish most of these, and each scenario needs at best a different approach, from flashybrid to Embdebian to ADS root builder to hand hacked images. That will change, I hope, if the grand unified scheme I'm plotting comes to fruition. If some of these scenarios interest you, please check out the design at http://wiki.debian.org/RootSync. openmoko First, install Debian to a 2 gb microSD card in the phone, using d-i. Bog-standard Debian install (except for it being g-i running on a cell phone!), standard partitioning. Boot the install and use it for a while. Install more stuff and tweak the system. Now you want to remove that microSD card, to save power. So, unmount it, and remove it. Debian continues running; all the files you normally use, including everything needed to boot the system, have already been copied over to the flash. If the flash gets too full, just delete any files you don't need right now. If you later find you're missing a file, just plug the microSD back in, mount it, and you can access the file again. This is also an easy way to do a backups, a simple command can sync your changes up to the microSD card. Also the microSD card can be plugged into a different openmoko phone, chrooted into, a command run, the phone rebooted, and it's been taken over, now running the same Debian install. Great party trick. ;-) slug with a 32 mb USB stick This is my sister's nslu2 box, that only does one thing, really: Dial up to the internet and route packets to the access point. It's also nice to get Debian security updates from time to time. And I don't want to have a noisy hard drive on it, so it runs from a small USB stick. The initial install is done to USB drive, then I add the USB stick to the picture, boot the nslu2 up, configure ppp, run it, and I can remove the drive, it's ready to run in embedded mode from the USB stick. Later I need to apply a security update, so I plug in the USB drive, mount it, run aptitude, and I'm done, just unmount the USB drive again, and the USB stick now has updated versions of the necessary files. true embedded system development A company is creating a real embedded arm system, it needs to run from 64 mb of flash, with a fair chunk for their own application, and a fair chunk free for user data. An engineer is using his development system to work on creating the jffs2 filesystem that will go on the production hardware. He starts by untarring the Debian armel port to a drive, and copies in the custom application. Then he sets up an empty jffs2 filesystem on mtdblockN. After rebooting the system, the jffs2 contains the files needed to boot Debian on its own. The engineer runs the custom application, and the jffs2 is now ready to run that too. But it's getting a bit tight for space on the jffs2. Time for busybox. He runs "busybox install [long list of busybox commands]", replacing lots of core commands with their smaller busybox versions. After a few reboots, all the init scripts are hacked up to work with the busybox stuff. (Argh.. Debian should work with it by default. Anyone want patches?) Maybe he goes on to run mklibs or do something with uclibc. This might take a few iterations to get right, but he doesn't worry about screwing up the system -- he knows that at any point he can delete all the files on the jffs2, reboot, and be back to his unmodified system on the disk drive. Once everything is ready, he can load the jffs2 onto production hardware and the new embedded system is ready to be tested and qualified. slug thick client My nslu2 box, used for handling some USB devices like my UPS, security camera, etc. When the network is up, and the storage server is up, it can nfs mount root, but that doesn't work when the power goes out and the server needs to come down. I want it to keep working in that case, as a kind of detached thick client. After hacking nfs root support into d-i at some future point :-) I have it booting with nfs root already. There's around 3 mb of unused flash on the slug, so I set up a jffs2 partition using that, and through some trial and error get the right files on there to let the system keep running in a degraded mode when the nfs mount goes down. 3 mb does fill up pretty quickly if the camera notices motion, but it's good enough, so I decide not to put a USB stick on the box.

Yesterday I mentioned a small project of mine called reportbug-ng. The feedback so far was very positive. Looks like quite a few people out there would like to see a desktop version of reportbug. Since I’ve already received a few patches via email, I decided to move the project to alioth so contributers can easily keep track of the changes. So here it is: the official homepage and teh almighty SVN repository. Comments and contributions are very welcome! (Hint, hint: The source-tree also contains a TODO list. Once all the points under “urgend” are processed, I’ll consider uploading it to Debian.)

I listened to my YP-U2 quite a bit on my recent travels. It worked fairly well. On my return, I remembered that I had a few comments to post from my old YP-U2 upgrade problems article... David Weinehall wrote:

"FWIW, my YP-U2 also has problems playing some oggs (nothing low bit-rate about them though; they're encoded with -q6); this is with version 1.418 of the firmware =/ On Samsung's site there seems to be a new version (1.43) available; too bad they don't mention what has been fixed."

dann frazier commented:

"Links/discussion about ogg support in these device can be found here: http://wiki.xiph.org/index.php/Talk:PortablePlayersSamsungYepp#YP-U2 I had to use a friend's Windows computer to do the upgrade :("

Erik Rossen writes:

"I have a Samsung YP-U2RZW (code for YP-U2, with Radio, 1GB of space, White case) that I bought today. The firmware version is "1.41 EU" according to system/system.var. And playback of medium quality OGG (44100 Hz, ~160032 bps) has serious stuttering problems. The latest firmware available on the UK Samsung site is 1.43. I too would love to know what files to copy to system/ (or wherever) so that I could do the upgrade from Linux, but I am probably going to end up booting WinXP for this task."

Scott Testerman comments:

"I don't have one of these players, but I wonder if this post is of any help to you? http://err.no/personal/blog/tech/2006-12-13-22-25_Samsung_Z5F.html"

Interesting, but sadly, the YP-U2 upgrades I have don't include a dat file. I wonder if this is what Samsung's support engineer was trying when he told me to copy all the files to the device's root. Does anyone have more info? Last time I chased Samsung about their promise of Linux upgrade instructions, I was asked to phone back in an evening when the member of staff who promised would be there. I still haven't had time to do that (there are other things to do most evenings - why won't they call me?). As far as I can tell, Samsung's Linux upgrade support sucks. I've bought a few Samsung devices and been fairly happy (far better than Canon, for example), but if I buy one again, I'll expect upgrades not to work.