I came across the mingetty changelog by chance while researching something totally unrelated. Paul Martin, you are a genius.

mingetty (1.07-2) unstable; urgency=high
   * Critical security patch: Fix unsafe chroot call. (Closes: #597382)
   * Checked dependencies for locusts. (Closes: http://xkcd.com/797/)
  -- Paul Martin <masked>  Sat, 25 Sep 2010 01:51:12 +0100

Original cartoon: http://xkcd.com/797/

Lucas has written two long, insightful posts on the frustration about the social aspects of integrating Ruby stuff Debian The first one, on September 12 and the second one, on September 29. I cannot really blame (thoroughly) the Ruby guys for their position. After all, they have a vibrant community, and they are advancing great pieces of work. And they know who that code is meant for Fellow programmers. And yes, although it is a pain to follow their API changes (and several of the Gems I regularly use do often get refactorings and functionality enhancements which break compatibility but introduce very nice new features), they say that's solved with one of Gems' main features being the simultaneous installability of different versions. The key difference in Debian's worldview with Ruby's is they cater to Fellow programmers. Even leaving aside heaps of different positions and worldview/mindset, we have a fundamental difference: Debian cares about its users, whatever that means. So, our users should not even care what language a given application is implemented in They should only care that it works. We, as packagers, should take care of all the infrastructural stuff. And yes, that's where we find the conflicting spot: We don't want to ship many versions of a system library (that in this case would be a Gem). Specially if later versions fix known bugs in earlier versions and backports are not available or supported. Specially if upstream authors' only response to a bug in an older release will be "upgrade and rewrite whatever breaks in your application". As an example of this, I am not currently updating the gems I maintain, as Debian is on a freeze to get out the next stable release. Or if at all, I am targetting those uploads to our Experimental branch, in order not to create a huge backlog for me when the freeze is over (just a series of rebuilds targetted at unstable). And yes, I will have to be responsible for any bugs that will most likely not be supported by most of my upstreams during the next ~2 years. That's the role of a Linux distribution. And yes, as Lucas writes in the comments he got as responses to the first post This dissonance comes in no small part because the Ruby developer community is mostly made from non-linuxers. People coming from a background where (mostly propietary) applications bundle up everything they need, where static linking is more popular than dynamic libraries, where there is no coordination between parts of the system are much less likely to understand our work. And yes, the Perl community is a joy to work with in this regard. And that's the same I understand from the Python one. Because of their origins and where their main strength was grown and remains. PS - And yes, I will join the flock of people saying that... The specific person that attacked your work is a great programmer, but well known as intolerant and obnoxious. Fortunately, even if our respective cultures fail to mix so much, most of our interactions just end with a "sigh" of lack of understanding, and not with the flames you got targetted with :-/

In my Ruby packaging in Debian and Ubuntu: Mythbusting and FAQ, I wrote The Ruby developers (or expert users) community is generally very hostile towards Debian. Many harsh words and insults are used in discussions mentioning Debian in the ruby-core@ mailing list, and there are frequent recommendations to avoid the Debian packages and install from sources, which is quite demotivating (Actually, what prompted that blog post is someone calling Debian-specific changes unforgivable). Zed A. Shaw just provided a very good example of that. Quoting the most juicy parts:

It s simply a tactic to make sure that you are stuck on Debian.

If we look at this problem from an open source perspective, Debian is at a minimum being a horrible citizen.

What is happening is control is not aligned with responsibility. Debian has taken control of the software package and recrafted it for their very narrow set of system administrator users. However, they refuse to take responsibility for the defect and quality problems they create. Instead I end up being responsible for their mistakes, which is wrong.

If Debian is in control, then Debian should be made responsible. That s the only way to get them to change it. If they don t want to be responsible, then they should let me be in control.

I m imagining a mass petition, maybe some funny ad campaigns, in-person confrontations meant to embarrass package maintainers, SEO tricks to get people off Debian, promotion of any alternative to Debian, anything to make them pay, apologize, and listen.

If you write software and you re sick of Debian screwing up your gear, then shoot me ideas for how we can make this happen. I think that if enough of us band together to make Debian responsible for its actions we can actually get them to start asking us before they package our software to find out how we think it should be packaged.

On the technical part of the post: The problem that Zed is experiencing is unknown to me, and also to the Debian and Ubuntu bug trackers. To my knowledge, rubygems doesn t not require openssl or net/scp to work, and a simple grep through the sources kind-of confirms that. But of course, that blog post was written as a rant, not as a proper bug report. My best guess is that the gem that Zed is installing requires openssl and net/scp. But of course, rubygems can t just depend on all the libraries that applications or libraries installed via rubygems could possibly require. Regarding openssl, we had a problem in Ruby because of the licensing situation of Ruby (dual licensed GPLv2+ and Ruby license, linked with openssl, which might raise some eyebrows), so it was split off the main standard library package until recently (and, apparently, in the version Zed is using). For net/scp, it is not part of the standard library, and not a dependency of rubygems, so I don t see why Zed expects it to be installed. On the rest of the post: Following my last post, several people volunteered to help with Ruby on Debian (and Ubuntu). In particular, Laurent Arnoud has been doing some fantastic work fixing the pkg-ruby-extras team s RC bugs. Also, all the messages of support both in the comments of my last post were very much appreciated. But I m really wondering why the Ruby community generates so many poisonous people. All my work on Debian is done on a volunteer basis, and it s really hard to get so much shit from it.

I'm one of those people who tries to keep a very consistent coding style across all of my projects, including project documentation and build system layout, and also like reading other coding style guides to look for good ideas and clean ways of organizing packages. I first wrote a coding style guide for INN once I started working on it regularly, then moved and expanded on that guide and added Perl for my work at Stanford. Over time, I wrote a guide for C build systems and package documentation. These guides were all stranded in our old group web pages, which I'm working on retiring, and since they're my personal styles (although largely used by my group at work), it seemed to make the most sense to pull them back into my personal web pages. They're now available in a new coding style section of my technical notes. The build system document is heavily revised, since I originally wrote it before I started using Automake, and the documentation guide now has much more detailed documentation of how I lay out README files. The other documents just have minor revisions.

Can you believe it? I still talk to people who don t know how to play blackjack. There must be something in the mechanics or tableau of blackjack that people find intimidating; for some reason, people can be a little cautious about taking the first move.
Photo courtesy of Winquerque on Net via Flickr. Clearly, there s no cause to hesitate. Players find quite quickly that it is actually not hard to figure out, and the strategies aren t even terribly complicated. For sure, if you re used to playing online slots, you are probably going to need to work a little harder to master the nuts and bolts. But it s still nothing like getting to grips with Texas Hold em from the beginning, for example!. Quite frankly, you very likely already know a fair portion of what most players have to know. I am able to give you the essentials in one simple little paragraph: If the card has aparticular number, that s the amount it will be worth. If it doesn t, it s worth 10. The only exception to this rule are Aces, which are worth either 1 or 11 (more about that later). Say hit when you have to get closer to 21, and/or stand when you don t. If you are closest to 21, you win. If you bust (get more than 21), you lose. Getting dealt blackjack (a natural 21 from the first two cards), makes you an instant winner (well, you and the dealer can also tie, or push ). Got it? OK, so it is not completely all you need to know. To begin with, most professional blackjack fans understand all too well that your goal has less to do with trying to get to 21, and just about everything to do with defeating an opponent. The dealer will not have the luxury of actually playing , precisely. Whether you re playing at a land-based casino or online (speaking of which, for additional info on USA casino players, all you have to do is click the link!), the house rules specify each individual play they can make. Mainly, she or he needs to take another card until their cards add up to the house rule limit (generally 17 and over). Of course, one of you will get an ill-fated draw and find themselves busted , but this decides a fewer number of games than you might expect. The tricky aspect of blackjack is the decision that you have to make if the cards add up somewhere around 12 and 16 (approximately, subject to other details). If you only had your own cards to consider, it would just be nothing but chance. Don t get me wrong, dumb luck has quite a lot to do with it. It is a card game, of course, and all card games are based on luck to some extent. But the serious gamer recognizes that luck is essentially a code word for statistics ; if you know what you have, and you possess a clue about the dealer s cards, you may make a knowledgeable judgment about which cards you re more inclined to end up with if you decide to hit another time. The most useful clue that you require to evaluate whether to play risky or conservative (sorry, I meant loose or tight ) is the dealer s upcard. Well, that s all for now, but for more information, you can always take a look at our friends at USA online casinos accepting Webmoney , who will get you up to speed. A bit later, we re going to get down to the serious nitty-gritty: doubling down, splitting, and much much more. Until then !

Many Unix daemons use a technique known as pre-forking . This means that to save the amount of time taken to fork a child process they will keep a pool of processes waiting for work to come in. When a job arrives then one of the existing processes is used and the overhead of the fork() system call is saved. I decided to write a little benchmark to see how much overhead a fork() really has. I wrote the below program (which is released under the GPL 3.0 license) to test this. It gives the performance of a fork() operation followed by a waitpid() operation in fork()s per second and also the performance of running a trivial program via system which uses /bin/sh to execute the given command. On my Thinkpad T61 with a Intel(R) Core(TM)2 Duo CPU T7500 @ 2.20GHz I could get 2429.85 forks per second when running Linux 2.6.32 in 64bit mode. On a Thinkpad T20 with a 500MHz P3 CPU I could get 341.74 forks per second. In both cases is seems that the number of forks per second is significantly greater than the number of real-world requests. If each request on average took one disk seek then neither system would have the fork performance as any sort of bottleneck. Also if each request took more than a couple of milliseconds of CPU time on the T7500 or 10ms of CPU time on the 500MHz P3 then the benefits of pre-forking would be very small. Finally it s worth noting that the overhead of fork() + waitpid() in a loop will not be the same as the overhead of just fork()ing off processes and calling waitpid() when there s nothing else to do. I had a brief look at some of my servers to see how many operations they perform. One busy front-end mail server has about 3,000,000 log entries in mail.log per day, that is about 35 per second. These log entries include calling SpamAssassin and Clamav, which are fairly heavy operations. The system in question averages one Intel(R) Xeon(R) CPU L5420 @ 2.50GHz core being used 24*7, I can t do a good benchmark run on that system as it s always busy but I think it s reasonable to assume for the sake of discussion that it s about the same speed as the T7500 (it may be 5* faster, but that won t change things much). At 2429 forks per second (or 0.4ms per fork/wait) if that time is entirely reduced to zero that won t make any noticeable difference to a system that has an average operation taking 1000/35= 28ms! Now if a daemon was to use fork() + system() to launch a child process (which is a really slow way of doing it) then the T7500 gets 248.51 fork()+system() operations per second with bash and 305.63 per second with dash. The P3-500 gets 24.48 with bash and 33.06 with dash. So it seems that if every log entry on my busy mail server involved using a fork()+system() operation and it was replaced to use pre-forked daemons then it might be possible to save almost 10% of the CPU time on that system in question. Now it is theoretically possible that the setup of a daemon process can take more CPU time than fork()+system(). EG a daemon could have some really complex data structures to initialise. If the structures in question were initialised in the same way for each request then a viable design would be to have the master process initialise all the data which would then be inherited by the children. The only way I can imagine for a daemon child process to take any significant amount of time on modern hardware is for it to generate a session encryption key, and there s really nothing stopping a single master process from generating several such keys in advance and then passing them to child processes as needed. In conclusion I think that the meme about pre-forking is based on hardware that was used at a time when a 500MHz 32bit system (like my ancient Thinkpad T20) was unimaginably fast and when operating systems were less efficient than a modern Linux kernel. The only corner case might be daemons which do relatively simple CPU bound operations such as serving static files from a web server where the data all fits into the system cache, but even then I expect that the benefit is a lot smaller than most people think and the number of pre-forked processes is probably best kept very low. One final thing to note is that if you compare fork()+exec() with an operation to instruct a running daemon (via Unix domain sockets perhaps) to provide access to a new child (which may be pre-forked or may be forked on demand) then you have the potential to save a moderate amount of CPU time. The initialisation of a new process has some overhead that is greater than calling fork(), and when you fork() a new process there are usually lots of data structures which are not written after that time which means that on Linux they remain as shared memory and thus reduce the system memory use (and improve cache efficiency when they are read). #include <unistd.h>
#include <stdio.h>
#include <sys/time.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <stdlib.h> #define NUM_FORKS 10000
#define NUM_SHELLS 1000 int main()

struct timeval start, end;
if(gettimeofday(&start, NULL) == -1)

fprintf(stderr, "Can't get time of day\n");
return 1;
int i = 0;
while(i < NUM_FORKS)

pid_t pid = fork();
if(pid == 0)
return 0;
if(pid > 0)

int status;
pid_t rc = waitpid(-1, &status, 0);
if(rc != pid)

fprintf(stderr, "waidpid() failed\n");
return 1;


else

fprintf(stderr, "fork() failed\n");
return 1;

i++;
if(gettimeofday(&end, NULL) == -1)

fprintf(stderr, "Can't get time of day\n");
return 1;
printf("%.2f fork()s per second\n", double(NUM_FORKS)/(double(end.tv_sec start.tv_sec) + double(end.tv_usec start.tv_usec) / 1000000.0) ); if(gettimeofday(&start, NULL) == -1)

fprintf(stderr, "Can't get time of day\n");
return 1;
i = 0;
while(i < NUM_SHELLS)

pid_t pid = fork();
if(pid == 0)

if(system("id > /dev/null") == -1)
fprintf(stderr, "system() failed\n");
return 0;

if(pid > 0)

int status;
pid_t rc = waitpid(-1, &status, 0);
if(rc != pid)

fprintf(stderr, "waidpid() failed\n");
return 1;


else

fprintf(stderr, "fork() failed\n");
return 1;

i++;
if(gettimeofday(&end, NULL) == -1)

fprintf(stderr, "Can't get time of day\n");
return 1;
printf("%.2f fork() and system() calls per second\n", double(NUM_SHELLS)/(double(end.tv_sec start.tv_sec) + double(end.tv_usec start.tv_usec) / 1000000.0) );
return 0;

It has been 12 hours since the LibreOffice announcement, and the projects starts to do its first steps.

1. Neil Brown had submitted the first patch to LibreOffice, after waiting for 3 years (!) while the patch was available at the openoffice.org issues system (#80637).
2. Michael Meeks had published Minutes of first LibreOffice technical group call . The general plan is to accept patches ( no large / risky / de-stabilising (new) feature work ) from the community and also merge the changes already done to openoffice.org (existing CWS).
3. The public is welcome to help cleaning the existing code, Gil Forcada already started. see the EasyHacks list for the tasks to be done.
4. LibreOffice mirrors now also contain the source code for the beta release done in the morning.
5. Patches are welcome at libreoffice@lists.freedesktop.org or using the Bugzilla. See Get Involved Developing LibreOffice for more info.

You re welcome to join us
Filed under: LibreOffice, Openoffice.org Tagged: LibreOffice

If I'll make it through this week it's because of these special people in my life. They help me through thick and thin, they believe in me, they are there for me. There is one universal term for them, friends. The term in its real meaning, in its original meaning, not in the perverted sense that sites like Facebook want to make you believe that it's alright to apply to random bystanders. You won't go as far for those as you'd go for your true friends, so don't let them steal away the meaning of the word from you. As music is one of the most driving force for me and this song makes me wanna cry, this is what I want to send out to my friends who are able to motivate me to keep me going: That's What Friends Are For. And as this song is so special I won't drown it in two more like I usual do, this blog entry goes to my personal section anyway. There is though a second video of a live version of it that in my opinion adds quite something to it, it contains a short interview with Dionne Warwick at the end. Enjoy!

/personal permanent link Comments: 2

Daily builds As of a month ago there are Ubuntu archives with fresh packages of Samba 4 and Openchange, built on a daily basis day from the latest upstream revision. This means that it is now possible to run a version of Samba 4 that is less than 24 hours old, without having to know how to extract source code from the version control system that upstream is using, without having to know how to build and install an application from source, but perhaps most importantly: without having to go through the tedious process of manually updating the source code and rebuilding. OpenChange is tightly coupled to Samba 4, so installing a new version of OpenChange usually involves installing a new version of Samba 4 as well. To make matters more confusing, the two projects use different version control systems (Samba 4 is in Git, while OpenChange is in Subversion) and different build systems (Samba 4 uses waf, OpenChange uses autoconf and make). I have been involved in Samba 4 and OpenChange as an upstream developer and more recently also as a packager for both Debian and Ubuntu. As an upstream developer for both these projects it is important for me that users can easily run the development versions. It makes it possible for interested users to confirm the fixes for issues they have reported and to test new features. The more users run the development version, the more confident I can be as a developer that doing a release will not cause any unexpected surprises. As a packager it is useful to know when there are upstream changes that are going to break my package with the next release.

Recipes The daily builds work using so-called recipes which describe how to build a Debian source package from a set of Bazaar branches. For example, the Samba 4 recipe looks like this:

# bzr-builder format 0.2 deb-version 4.0.0~alpha14~bzr revno ~ppa revno:packaging + revno:debian 
lp:samba
merge debian lp:~samba-team/samba/unstable
merge packaging lp:~samba-team/samba/4.0-ppa-maverick

This dictates that a source package should be built by taking the upstream Samba branch and merging the Debian packaging and some recipe-specific tweaking. The last bit on the first line indicates the version string to be used when generating a changelog entry for the daily build. Every night Launchpad (through bzr-builder) merges these branches and attempts to build the resulting source package, e-mailing me in case of build problems. Generally I fix issues that come up by committing directly to upstream VCS or to the Debian packaging branch. There is no overhead in maintaining the daily build after I've set it up. For more information on creating source package recipes, see getting started.

Toolchain The entire toolchain that does the daily package builds for Ubuntu is Free Software, and I have contributed to various bits of that toolchain over the years. It's exciting to see everything come together.

Soyuz Launchpad consists of multiple pillars - one of those pillars is Soyuz, which I hack on as part of my day job at Canonical. Soyuz is responsible for the archive management and package building. Debian source packages (a combination of upstream source code and packaging metadata) get uploaded by users and then built for various architectures on our buildfarm and published to the Ubuntu archive or to users personal package archives.

Launchpad-code Another pillar of Launchpad is Launchpad-code, which is responsible for the hosting and management of version control branches. Launchpad users can either host their branches on Launchpad directly or mirror branches (either native Bazaar branches or branches in a foreign format such as Subversion, Git or Mercurial). The mirrorring of native and foreign branches happens using standard Bazaar API's. In the case of Samba and OpenChange we import the branches of the upstream projects (Samba is in Git, OpenChange is in Subversion) and the packaging for both projects is in Bazaar. Launchad-code calls out to Bazaar to do the actual mirrorring. Over the last few years I have done a lot of work to improve Bazaars support for foreign branches, in particular on supporting Subversion, Git and Mercurial. As the code mirrorring in Launchpad is one of the biggest users of bzr-svn and bzr-git it has helped find some of the more obscure bugs in those plugins over the last few years, to the point where there are only a handful of issues with Git and Subversion imports left.

bzr-git and dulwich bzr-git provides transparent access to Git repositories from within Bazaar and is built on top of Dulwich. Dulwich is a Python library that provides access to the Git file formats and protocols that is completely independent of Bazaar. James Westby originally started it and I adopted it for bzr-git and further extended it. There are now several other projects that use it as well, including hg-git, and rabbitvcs. Apart from James and me almost two dozen other people have contributed it so far.

bzr-svn and subvertpy bzr-svn provides transparant access to Subversion repositories in Bazaar. When I grew frustrated with the existing Subversion Python bindings for various reasons, I decided to create independent Python bindings for Subversion from scratch. These bindings have since been split out into a separate project - subvertpy - and other projects have since also started using them, e.g. hgsubversion and basie.

Using the daily builds To use the Samba 4 and OpenChange daily builds (Ubuntu Maverick only for now), run:

$ apt-add-repository ppa:samba-team/ppa
$ apt-add-repository ppa:openchange/daily-builds

cp: Karnivool - Themata

For a security update of a shared library to take effect processes need to be restarted in order to use the libraries from the newly installed package. The openssl and freetype updates got me thinking again if I caught all the affected services on servers and desktops/laptop systems I'm running. So I put together a small script that takes a package name and scans /proc/pid/maps for processes that map libraries from this package, it then looks for init scripts in these packages. It can also hook into apt and restart the services after the upgrade. Since the mapping to the init scripts is quiet dumb at the moment this should be used with a bit of caution though. The script whatmaps can be fetched form here. Debs are also available. Besides Debian packages it also handles RPMs and already proved useful since it pointed me to clamav being possibly affected by the recent bzip2 advisory.

There are situations in which a given package needs to have different dependencies in Debian and in Ubuntu. Despite this difference it s possible to keep a single source package that will build both variants of the package. Continue reading to discover a step-by-step explanation. 1. When is that needed? While it is possible to have different dependencies depending on the distribution on which the package is built, it should be usually avoided when possible. This infrastructure should only be used as a last resort when there are no better alternatives. Here are some examples of when it might be needed:

• Ubuntu has some packages that Debian does not have (or vice-versa), and the resulting package would benefit from having them installed.
• The package names differ between Ubuntu and Debian and it s on purpose (i.e. the difference is a justified choice and not a mistake because both distributions failed to coordinate).
• The packages are built differently in both distributions, and the run time dependencies are not the same due to this. Maybe some associated patches are only applied in Ubuntu.

2. Using substitution variables in debian/control The dependency that varies between both distributions can t be hardcoded in debian/control, instead you should put a substitution variable (substvar) that will be replaced at build time by dpkg-gencontrol. You can name it $ dist:Depends for example:

[...]
Depends: bzip2, $ shlibs:Depends , $ misc:Depends , $ dist:Depends 
[...]

Note that you typically already have other substitution variables ($ shlibs:Depends comes from dpkg-shlibdeps and $ misc:Depends from debhelper and its dh_* scripts). 3. dpkg-gencontrol needs a -V option The value used to replace this new variable needs to be communicated to dpkg-gencontrol. You can use the -V option for this, the syntax would be something like this:

dpkg-gencontrol [...] -Vdist:Depends="foo (>= 2), bar"

If you use debhelper, you have to pass the option to dh_gencontrol after two dashes (--):

dh_gencontrol -- -Vdist:Depends="foo (>= 2), bar"

If you use CDBS, you can set the DEB_DH_GENCONTROL_ARGS_ALL make variable:

include /usr/share/cdbs/1/rules/debhelper.mk
DEB_DH_GENCONTROL_ARGS_ALL = -Vdist:Depends="foo (>= 2), bar"

The value given to dpkg-gencontrol is static in all those examples, now let s see how we can use give a different value depending on the distribution that we re targetting. 4. Using dpkg-vendor in debian/rules dpkg-vendor is a small tool (provided by the dpkg-dev package) that parses the /etc/dpkg/origins/default file (provided by the base-files package) to know the current distribution and its ancestry. It can be used in debian/rules to adjust the behavior depending on the current distribution. You can check its man-page to learn about the various options supported but we re only going to use --derives-from <vendor> in this sample. With this option the script exits with zero if the current distribution is or derives from the indicated distribution, or with 1 otherwise. Now combining all together, we can use dpkg-vendor to dynamically define the content of the substitution variable in debian/rules. Let s suppose that you want $ dist:Depends to be foo (>= 2) on Ubuntu (and its derivatives) and bar everywhere else. Using debhelper s 7 tiny rules file, this example could be:

ifeq ($(shell dpkg-vendor --derives-from Ubuntu && echo yes),yes)
	SUBSTVARS = -Vdist:Depends="foo (>= 2)"
else
	SUBSTVARS = -Vdist:Depends="bar"
endif
%:
	dh $@
override_dh_gencontrol:
	dh_gencontrol -- $(SUBSTVARS)

If you use CDBS, it could be this:

include /usr/share/cdbs/1/rules/debhelper.mk
ifeq ($(shell dpkg-vendor --derives-from Ubuntu && echo yes),yes)
	DEB_DH_GENCONTROL_ARGS_ALL = -Vdist:Depends="foo (>= 2)"
else
	DEB_DH_GENCONTROL_ARGS_ALL = -Vdist:Depends="bar"
endif

Do you want to read more tutorials like this one? Click here to subscribe to my free newsletter, you can opt to receive future articles by email.

9 comments Support my work

One of the nice things that I've been involved with since starting to work at ARM in Cambridge is setting up newer, faster machines to help with the armel port. We have six machines hosted in the machine room here now:

• abel is a porter box
• arnold, alain, alwyn and antheil are running buildd software
• arne is a hot spare (currently on my desk, but ready to go back in the rack any minute now)

All of these machines are Marvell DB-78x00-BP development boards, each configured with a 1GHz Feroceon processor (ARM v5t), 1.5GB of RAM and a 250GB drive attached via SATA. They're nice machines, reasonably powerful yet (as with many ARM-based machines) they draw very very little electrical power even when working hard. These very boards were used for a while by the folks at Canonical to help build the Ubuntu armel port, but now we've got them. In terms of configuration, these machines are not quite fully supported in Debian yet, though. The kernels we're using are locally-built, based on the Debian linux-source-2.6.32 package but with a .config (marvell.config) that's tweaked slightly to add the support for these boards. There aren't any source changes needed, so I'm hoping to get support added directly in Debian, either as a new kernel flavour or (preferred) as a patch to an existing flavour. I've had conflicting advice about whether the latter is possible, so I'm going to have to experiment and find out for myself. UPDATE 2010-09-28: I've tested, and it seems that the boards will need a new flavour after all, as the config is incompatible with the closest other config (kirkwood). Ah well... I had no end of trouble trying to get make-kpkg do the right thing, so on advice from Ben I built the kernel using "make deb-pkg", a standard target in the Linux kernel's build system: fakeroot make -j2 deb-pkg DEBEMAIL=93sam@debian.org DEBFULLNAME="Steve McIntyre" KDEB_PKGVERSION=buildd23 Annoyingly, that wouldn't work when cross-compiling either so I had to build the kernel natively. To make the resulting kernel image package install properly (and, just as importantly, allow for future easy upgrades for the DSA folks), I also needed the following tweaks to the Debian system:

• depmod:
  • Need to make sure that depmod is run so the new kernel can find and load modules at boot. Added trivial script in /etc/kernel/postinst.d/depmod to do this.
• initramfs-tools:
  • Needed to copy the file /etc/kernel/postinst.d/initramfs-tools into place from my amd64 machine; I'm guessing this would be there automatically on a new-enough version of initramfs-tools on the armel machines, but we're still using Lenny as a base system for now even if I'm using a Squeeze-based kernel.
• flash-kernel:
  • #594878: support for these boards. Hector did most of the work, and I've worked on it a little more.
  • #550584: kernel postinst hook script (/etc/kernel/postinst.d/zz-flash-kernel) to create uImage and uInitrd files from the kernel zImage and the initramfs.

Finally, I've tweaked the uboot config on the machines to use the uImage and uInitrd files that are generated:

Marvell>> setenv IDE ide reset
Marvell>> setenv loadkernel ext2load ide 0:1 0x2000000 /uImage
Marvell>> setenv loadinitramfs ext2load ide 0:1 0x3000000 /uInitrd
Marvell>> setenv bootboth bootm 0x2000000 0x3000000
Marvell>> setenv bootcmd setenv bootargs \$\(bootargs\)\;$(IDE)\;$(loadkernel)\;$(loadinitramfs)\;$(bootboth)
Marvell>> saveenv

And that's it, as far as I can see. I'll now wait for people to tell me what I've got wrong above... :-)

I use to take my (company) notebook with me on business travels.
Two times I now had the unlucky situation that something bad happened to it on such an occassion. Whenever you get in the situation that you need to reinstall your system in a hotel room you'll might have the same wish that I got: A way to quickly bring the system in a state where I could work with it.

Well, I used FAI a while back for a customer. Its a real great tool for automated installations and I really prefer it over debian-installer preseeding. Apart from the fact that the partitioning is way easier it also gives me the power to complete the whole installation up to a point where I've got almost nothing to do anymore. It also features an installation completely from CD or USB-Stick which makes it suitable for me.

However, my notebook installation has a little "caveat" which made that a little bit more harder as previously thought. As it is a notebook and I carry company data on it it has to be encrypted. Disk encryption at a whole.
The stable FAI version does not support this.
The problem is: The current support for crypto in setup-storage (FAIs disk setup tool) is not very far. Supported is the creating of a LUKS container with a keyfile, saving this keyfile to the FAI $LOGDIR and creating a crypttab.
Unfortunately for a root filesystem this would leave us with an unbootable system, because this requires manual interaction. And on the other hand using a keyfile for a cryptoroot is a no-go anyway. We want a passphrase.
On a side-note: cryptoroot support with a keyfile is more complex than with a passphrase, as you have to provide a script that knows how to get to the key.

So I started experiments with scripts in the FAI-configuration that added a passphrase, changed the crypttab and recreated the crypttab. That worked, although it was very ugly.
But due to a good coorperation with Michael Tautschnig, a FAI- and Debian-Developer, on this, the FAI experimental version 4.0~beta2+experimental18
now supports LUKS-volumes with a passphrase that can be specified in the disk_config.

Now its actually possible to setup a system like mine with FAI out-of-the-box. One thing (apart from the FAI configuration and setup as you want and need it) has to be done, anyway:
The initrd support of cryptsetup requires busybox (otherwise you will see a lot of "command not found" errors and you system won't boot) and it requires initramfs-tools, which is standard nowadays.
So you have to make sure that these packages are in your package config!

So now I can define a FAI-profile for my notebook, create a partial fai mirror with the packages it needs and put all this together on an USB stick with fai-cd (don't worry about the name, it can be used to create ISO images as well). I can carry this with me and if I need it I stick it into my notebook and let FAI automatically reinstall my system. Nice :)

Update: Somebody asked me, weither he understood me right, that I'd put my LUKS passphrase on a FAI usbstick clear-text. Obviously, the answer is and should be NO. What I do and what I'd suggest to others: Use a default passphrase in the FAI configuration, install with it - after all on a fresh installation there is not much to protect - and once it is finished *change* the passphrase to something secure by adding a new keyslot and removing the old.

Just so I don t forget it for a second time: esmtp can be pretty obscure when it comes to error messages.
When it encounters an .authenticate/ca.pem file with public permissions it connects to the server and only after seeing the extensions list does it tell you StartTLS extension not supported by MTA . Not exactly straightforward.

After the famous Facebook outage a few days ago, Facebook Engineering made a to be fair, reasonably detailed statement on what went wrong and how it affected the service. One particular comment attached to it drew my attention:

It s great to hear and see that big companies like Facebook are so open with what they do. That s rare, very rare. Thanks!

Huh? Did I miss some announcement of openness and freedom?
<script type="text/javascript"> var flattr_wp_ver = '0.9.16'; var flattr_uid = '32970'; var flattr_url = 'http://www.jwiltshire.org.uk/content'; var flattr_lng = 'en_GB'; var flattr_cat = 'text'; var flattr_tag = 'blog,wordpress,rss,feed'; var flattr_btn = 'large'; var flattr_tle = 'jwiltshire.org.uk'; var flattr_dsc = 'The public face of jwiltshire'; </script> <script src="https://api.flattr.com/js/0.5.0/load.js?mode=auto" type="text/javascript"></script>

Working on a new feature for eSpeak GUI I started looking into language identification. Forcing users to manually choose the text s language is a botheration, so trying to guess it by checking which system dictionary contains the most words from the text or some other method would surely be beneficial. After a quick search I learned that it s much easier than this: it s possible to reliably determine the language based on statistic n-gram information. Ignoring the fact that now I officially hate Firefox, Chromium, OpenOffice.org and everyone else there for not implementing this and having me spend the day changing the spell-checker s language, I was left with the choice on how to use this in eSpeak GUI. The first option I found was TextCat, which is also the only library I ve found to be packaged for Debian. However, ignoring the fact that upstream isn t maintaining it any more (such a library shouldn t need too much maintainance, after all), the package declares incorrect dependencies (bug filled a month ago, no response yet) and the API is also pretty crappy (it requires a physical file indicating the location of the statistic models). Unrelated to that, I ve also found that the Catalan text samples it includes are incorrect, so the same may be true for other languages. I guess it d make sense to work on a new (and completely Unicode) language samples collection. I ve thought of using something like the Universal Declaration of Human Rights since this way all languages can have the same text, but being more of a legal thing it may be biased by some words being too repetitive. Looking for other alternatives to the TextCat library I ve only found the following:

• TextCat (same name, different code): PHP licensed, so incompatible with GPL projects.
• Mguesser (part of mnogosearch-mysql): it s a standalone executable and not a library.
• SpamAssassin s TextCat.pm: also a standalone executable, this time written in Perl. Apparently they were using a fork of TextCat (the original library, not the PHP licensed one) before that.

So it looks like I ll have to start by getting a good collection of text samples I can use to generate the statistic data. Then I have several options on how to actually use it. As I see it, those are my possibilities:

1. Fixing libtextcat s packaging and just using that.
2. Taking it over as new upstream maintainer. Not my preferred option as I don t really feel like maintaining a C library at this point.
3. Trying to convince the maintainer of the new TextCat (with last commit January this year and a more sane API) to re-license it in a GPL-compatible way, packaging that and seeing how that one works (haven t tried it out yet).
4. Writing my own implementation in Python, maybe based upon this example or TextCat.pm.

Any other ideas, pointers to some library I may have missed or offers to collaborate are very welcome. Please also note that my intention in writing this post is not only to rant about there being no well-maintained ready-to-use library being available, but especially raising awareness on the topic of language identification. I d love to see this feature all around the desktop, just like (and in combination with) spell-checking, which is already omnipresent. Related posts:

1. espeak-gui 0.2
2. Introducing espeak-gui

---

11 comments
Siegfried-Angel Gevatter Pujals, 2010. Permalink License Post tags: language identification, Programari lliure, text categorization

I recently started using btrfs, in both raid0 and raid1 setups, and migrated some data there. Once all the data was copied, I wanted to check how much overhead moving to btrfs meant, space-wise.

# df -h /mnt/share
Filesystem Size Used Avail Use% Mounted on
/dev/sdb3 932G 682G 250G 74% /mnt/share # btrfs-show /dev/sdb3
Label: none uuid: e06484de-fb18-4564-b6e6-adbaed8bebec
Total devices 2 FS bytes used 681.61GB
devid 2 size 465.66GB used 427.13GB path /dev/sdb3
devid 1 size 465.66GB used 427.15GB path /dev/sda3


The filesystem was freshly created, in a raid0 fashion, so both 465GB partitions are used to form (roughly) a 930GB volume.
So, I had filled it with 682GB of data, and df was telling me 250GB were left, which is not unexpected. What I didn t expect, though, is btrfs-show telling me 427GB were used on each partition. Again, the filesystem was freshly created and the data had just been copied over, so I wasn t expecting these numbers to be off. Which meant 854GB were being allocated on disk for 681GB of data, 25% overhead ! Curious to know how much left that really meant, I just tried a stupid thing: filling as much as btrfs permits:

# dd if=/dev/zero of=/mnt/share/zero bs=1M
dd: writing /mnt/share/zero': No space left on device
253359+0 records in
253358+0 records out
265665118208 bytes (266 GB) copied, 1144.41 s, 232 MB/s
# du -sh /mnt/share/zero
248G /mnt/share/zero


It turns out I actually could write for the amount of free space df was claiming was free (modulo GiB-GB conversions).

# df -h /mnt/share
Filesystem Size Used Avail Use% Mounted on
/dev/sdb3 932G 930G 2.0G 100% /mnt/share # btrfs-show /dev/sdb3
Label: none uuid: e06484de-fb18-4564-b6e6-adbaed8bebec
Total devices 2 FS bytes used 929.36GB
devid 2 size 465.66GB used 465.13GB path /dev/sdb3
devid 1 size 465.66GB used 465.15GB path /dev/sda3


So, this time df is slightly off, and btrfs-show somehow agrees the filesystem is full. But surprises don t stop there:

# rm /mnt/share/zero # df -h /mnt/share
Filesystem Size Used Avail Use% Mounted on
/dev/sdb3 932G 682G 250G 74% /mnt/share # btrfs-show /dev/sda3
Label: none uuid: e06484de-fb18-4564-b6e6-adbaed8bebec
Total devices 2 FS bytes used 929.36GB
devid 2 size 465.66GB used 465.13GB path /dev/sdb3
devid 1 size 465.66GB used 465.15GB path /dev/sda3


That s right, even after the file removal, for btrfs-show, everything is still in use. Even after several sync or several hours later. All in all, just don t trust what btrfs-show says. NB: one also needs to know that df reports wrong sizes for raid1 setups, but that s another story.

This last couple of days I've been dedicated making Gnus fresher. I've decided to give a whirl on Gravatar support. I already tried the gravatar.el lying on the Interweb, but well, it was crap: it used wget to fetch pictures, therefore was totally synchronous. Reading each mail was slower. The cache did not even have TTL, as far as I recall. So, I've now wrote gravatar.el implementing the Gravatar API. Asynchronously of course. With cache, TTL, etc. Perfect. :-) Then I've composed gnus-gravatar.el, implementing a washing function adding Gravatar for From field and/or Cc/To fields, like done for picons. As I was expecting, the patch was badly received by the GNU guys, which start talking about thinks like external resources, privacy, non-free software, etc. Boring. Fortunately, Lars allowed me to push the patch in git so everybody can give it a try. I'm now waiting for feedbacks in order to know if I will have to maintain this patch outside Gnus, or not. Here's the mandatory screen-shot.

I guess it's time again to push for another band. This time I present you Garbage. I love them, but one has to be careful in what mood one is when listening to them; it's possible that they move you in a direction you don't feel comfortable with at that time.

• Bleed Like Me: If you listen closely to the lyrics of the second verse you might get an idea why had to love that song.
• The World Is Not Enough: One might argue that a band has made it when they do a James Bond title song. This is it.
• Only Happy When It Rains: Probably the best known song from Garbage. Damn fine, great tune and groove for the message it contains.

Enjoy!

/music permanent link Comments: 0

My comment form on my blog contains the following, right above the <textarea> element where the text can be entered:

<!-- To any spammers out there: Any submission to this form is
moderated: this means that only I will see your URLs, and I will not
visit them nor link to them. Thus, you can stop wasting our collective
time. -->

That, combined with the fact that my blog comment system is a home-grown thing, has meant that I haven't seen a single spammy comment anymore in months. The same comment form has shown the same thing since ages in a <p> element right above the <form> element, but that apparently was missed by some spammers. Since adding the above comment, however, I've not received a single spam submission to that form in over a year. Who says spammers are stupid?