I ve been doing some code lately, fixing things here and there, and over the last few days gave mt-daapd some attention. First, I was reported a segfault of mt-daapd when reloading web pages too fast. Surprisingly, it turned out to be a stupid omission in a simple routine and it is hard to hit without disabling the cache in the browser. Now, I just fixed something that s been bugging me for a while: mt-daapd did not handle Avahi daemon restarts, leading to mt-daapd becoming invisible to clients relying on mDNS until you restarted it. And, wow, that went a bit further than expected. As I expected, the simple piece of code needed to handle that wasn t there. But still it did not work, as it turns out the event loop for the Avahi polling wasn t being run. Replace stupid code reinventing the wheel by the Avahi-provided wheel and there, it works. But not when run as a daemon (aka foreground only). Well yes, starting the thread before daemonizing isn t going to work well, so fix that. EPIC WIN \o/ As mt-daapd s upstream is not active anymore I m accumulating fixes for mt-daapd in the Debian package, to the point I m starting to consider setting up a git repository somewhere with those fixes and possibly some more code rework if I feel like it.

I may be wrong on this but it looks like Microsoft SMTP clients (at least Windows Mail and Outlook) don t like being sent a large volume of SSL certificate information when opening a TLS connection. They appear to assume that the data they are being sent is malformed and assume that STARTTLS failed, continuing with an unencrypted SMTP dialogue. This can be triggered relatively easily on a Debian system by telling Exim to use all the certificates provided by the ca-certificates package (which is the default configuration). The Windows clients will give an unhelpful the remote end dropped the connection style error, caused by the server getting upset by the unexpected fallback to unencrypted SMTP. The server logs will show something like this:

2009-02-17 21:32:55 TLS error on connection from client.example.com (Client) [192.168.192.168] (gnutls_handshake): A TLS packet with unexpected length was received.
2009-02-17 21:33:00 SMTP protocol synchronization error (input sent without waiting for greeting): rejected connection from H=client.example.com [192.168.192.168] input="EHLO Client\r\n"

Configuring the MAIN_VERIFY_TLS_CERTIFICATES option in the Debian Exim configuration (which sets the tls_verifiy_certificates option in the actual Exim configuration) to point to something with less certificates in should avoid the issue. On the bright side, at least they re making an effort to avoid overflows.

Yesterday, I posted part 1 of how to think about compression. If you haven t read it already, take a look now, so this post makes sense. Introduction In the part 1 test, I compressed a 6GB tar file with various tools. This is a good test if you are writing an entire tar file to disk, or if you are writing to tape. For part 2, I will be compressing each individual file contained in that tarball individually. This is a good test if you back up to hard disk and want quick access to your files. Quite a few tools take this approach rdiff-backup, rdup, and backuppc are among them. We can expect performance to be worse both in terms of size and speed for this test. The compressor tool will be executed once per file, instead of once for the entire group of files. This will magnify any startup costs in the tool. It will also reduce compression ratios, because the tools won t have as large a data set to draw on to look for redundancy. To add to that, we have the block size of the filesystem 4K on most Linux systems. Any file s actual disk consumption is always rounded up to the next multiple of 4K. So a 5-byte file takes up the same amount of space as a 3000-byte file. (This behavior is not unique to Linux.) If a compressor can t shrink enough space out of a file to cross at least one 4K barrier, it effectively doesn t save any disk space. On the other hand, in certain situations, saving one byte of data could free 4K of disk space. So, for the results below, I use du to calculate disk usage, which reflects the actual amount of space consumed by files on disk. The Tools Based on comments in part 1, I added tests for lzop and xz to this iteration. I attempted to test pbzip2, but it would have taken 3 days to complete, so it is not included here more on that issue below. The Numbers Let s start with the table, using the same metrics as with part 1:

Tool	MB saved	Space vs. gzip	Time vs. gzip	Cost
gzip	3081	100.00%	100.00%	0.41
gzip -1	2908	104.84%	82.34%	0.36
gzip -9	3091	99.72%	141.60%	0.58
bzip2	3173	97.44%	201.87%	0.81
bzip2 -1	3126	98.75%	182.22%	0.74
lzma -1	3280	94.44%	163.31%	0.63
lzma -2	3320	93.33%	217.94%	0.83
xz -1	3270	94.73%	176.52%	0.68
xz -2	3309	93.63%	200.05%	0.76
lzop -1	2508	116.01%	77.49%	0.39
lzop -2	2498	116.30%	76.59%	0.39

As before, in the MB saved column, higher numbers are better; in all other columns, lower numbers are better. I m using clock seconds here on a dual-core machine. The cost column is clock seconds per MB saved. Let s draw some initial conclusions:

• lzma -1 continues to be both faster and smaller than bzip2. lzma -2 is still smaller than bzip2, but unlike the test in part 1, is now a bit slower.
• As you ll see below, lzop ran as fast as cat. Strangely, lzop -3 produced larger output than lzop -1.
• gzip -9 is probably not worth it it saved less than 1% more space and took 42% longer.
• xz -1 is not as good as lzma -1 in either way, though xz -2 is faster than lzma -2, at the cost of some storage space.
• Among the tools also considered for part 1, the difference in space and time were both smaller. Across all tools, the difference in time is still far more significant than the difference in space.

The Pretty Charts Now, let s look at an illustration of this. As before, the sweet spot is the lower left, and the worst spot is the upper right. First, let s look at the compression tools themselves: At the extremely fast, but not as good compression, end is lzop. gzip is still the balanced performer, bzip2 still looks really bad, and lzma -1 is still the best high-compression performer. Now, let s throw cat into the mix: Here s something notable, that this graph makes crystal clear: lzop was just as fast as cat. In other words, it is likely that lzop was faster than the disk, and using lzop compression would be essentially free in terms of time consumed. And finally, look at the cost: What happened to pbzip2? I tried the parallel bzip2 implementation just like last time, but it ran extremely slow. Interestingly, pbzip2 < notes.txt > notes.txt.bz2 took 1.002 wall seconds, but pbzip2 notes.txt finished almost instantaneously. This 1-second startup time for pbzip2 was a killer, and the test would have taken more than 3 days to complete. I killed it early and omitted it from my results. Hopefully this bug can be fixed. I didn t expect pbzip2 to help much in this test, and perhaps even to see a slight degradation, but not like THAT. Conclusions As before, the difference in time was far more significant than the difference in space. By compressing files individually, we lost about 400MB (about 7%) space compared to making a tar file and then combining that. My test set contained 270,101 files. gzip continues to be a strong all-purpose contender, posting fast compression time and respectable compression ratios. lzop is a very interesting tool, running as fast as cat and yet turning in reasonable compression though 25% worse than gzip on its default settings. gzip -1 was almost as fast, though, and compressed better. If gzip weren t fast enough with -6, I d be likely to try gzip -1 before using lzop, since the gzip format is far more widely supported, and that s important to me for backups. These results still look troubling for bzip2. lzma -1 continued to turn in far better times and compression ratios that bzip2. Even bzip2 -1 couldn t match the speed of lzma -1, and compressed barely better than gzip. I think bzip2 would be hard-pressed to find a comfortable niche anywhere by now. As before, you can download my spreadsheet with all the numbers behind these charts and the table.

If you re reading this post via the website rather than a feed/planet then you will notice that the site has gone completely black in support of the Creative Freedom Foundation s campaign against S92A of the NZ Copyright Amendment Act which is due to come into effect on 28th February 2009. I ve also joined the wave of people blacking out their avatar on Facebook/Jabber/MSN, etc. S92A introduces Guilt Upon Accusation whereby if you are accused of copyright infringement (downloading music and movies, etc) repeatedly (likely 3 or more times) you are at risk of being disconnected from the Internet by your ISP. The law does not require any proof or substantiation of the accusations and the entire process would occur outside of the courts and the established legal system. Not only does it place every user at risk, the wording is very unclear on exactly what type of organisation is considered an ISP and there is significant concern that schools, businesses, libraries and hospitals will be placed in the difficult position of determining whether their users have broken the law and require disconnection. Opposition to the law is not an attack on copyright, or a statement that we should be free to download all the movies and music that we desire. Those sorts of activities are clearly wrong, and I don t have any issue with copyright holders wanting to enforce their rights when their content is illegally copied. However, disconnecting people upon accusation, with no proof or formal legal process to prove guilt is not the right way to go about it. The fact that the law does not require proof of guilt is only the tip of the iceberg in terms of problems with it. For further background on the problems it causes for ISPs by placing them as the middle-man in copyright disputes you should refer to the following posts:

• Russel Brown of Public Address, points out that among other faults the law stipulates a penalty that no court would impose in adjudicating a copyright complaint
• David Farrar of Kiwiblog, has a guest post at the NZ Center for Political Research explaining how the law affects businesses
• Creative Freedom Foundation Background on S92

Finally, I think it is worth pointing out that S92A was removed from the proposed Amendment at the select committee stage, but was later reintroduced by Judith Tizard during the final reading of the bill. Mark Harris has an excellent post on the history of the amendment which includes facts such as the official report on the amendment also recommended removing S92A as it was unecessary given existing ISP terms and conditions which forbid illegal activity. The fact that the select committee (based on public submissions) recognised the problems with S92A and removed it, only to have it added back in again at the last stage when we no longer had any say on it really hacks me off and I cant help but feel the influence of the big money American media companies pressuring our politicians to pass a law that they don t really understand the full consequences of. So what is to be done? The Blacked Out campaign, being run by the Creative Freedom Foundation is gathering steam and international attention. Peter Dunne of United Future (who originally voted for the amendment) has declared that the amendment is wrong, and doesn t do what they thought they were voting to do, we need to convince National and the rest of the house of the same. Time is running out for this to happen before the amendment comes into effect on Feb 28th, but there is still time to write to your local MP and sign the petition against S92A Guilt Upon Accusation . The Creative Freedom Foundation site has a nice easy list of what you can do to register your protest.

It's time to drive some buzz this way ;-) Although this post will only be a pointer towards the Spanish post I made on Planeta EDUSOL, for reasons soon to become obvious. In any case, the information I'm posting here is not exactly the same. Can you read Spanish? Please go on to the invitation for the first videoconference for the EDUSOL Seminar. This year, we the organizers of the On-line Encounter of Education and Free Software (EDUSOL) are aiming higher - we are not "just" having a two-week encounter at the end of the year - We are having an all-year-long Seminar, focusing on the collaborative construction of knowledge. People from quite distinct backgrounds will be part of this project, and we aim to drive it towards the publication of a book. We (mainly, Alejandro Miranda and me) have been quite busy bootstrapping this seminar, getting the proposed authors, thinking over the intended communication channels and ways, and setting up the needed infrastructure) and are ready to start the public-facing activities. We will be having monthly videoconferencing sessions, the last Thursday each month, 16:00-18:00 Mexico Central time (currently GMT-6; GMT-5 after the beginning of April). The VC sessions will be also relayed through Ogg streams, and we will have an IRC channel available to offer full interactivity for those who do not have access to a H.323 VC setup. This first session will be moderated by Victor Manuel Mart nez; the speakers will be Alejandro Miranda and myself - The topics we will present are:

• Short project presentation, delineating the list of invited authors and tematic lines we will pursue
• Description of the collaboration scheme we expect to hold, including how everybody (not just invited authors) can participate
• Presentation of one of the topics we will work into in the Seminar: Free Software and the Democratic Construction of the Society

If you have access to videoconferencing facilities, please get in touch with Carlos Cruz, the Videoconferences Coordinator at the Economic Research Institute, as soon as possible for all the needed coordination.

After dropping all (pretense at) regular exercise after leaving high/grammar/secondary school (the german word is Gymnasium) and doing more or less nothing for more than 10 years, I've taken up weight training (after a not very successful attempt at volleyball; I like the sport but the team fell apart) two years ago. Since I actually managed to work out regularly (twice a week) in the first year, and since membership in a gym is not cheap, and since I moved and thus had to find a different solution anyway, I decided to buy a Bowflex Sport home gym. I can't compare it to other home equipment, and it apparently it has been discontinued by Nautilus, but nonetheless, a few thoughts: The conclusion first: Would I buy it again? No. Was it a waste of money? No, it's not that bad. In detail:

• The free-running seat that should allow cardio rowing is not very useful because there's no good foot-rest, and because the range of motion is too limited. This is my main grievance because being able to do cardio was one of the main reasons for chosing this engine over other offers.
• Go for the 310lbs extension at least (I did, luckily.) With leg exercises, 210lbs are just not enough (I'm 70kg/180cm and have been doing weight training two years. Certainly no muscle freak...)
• It's slightly too small. For many exercises, I'd wish for a bigger range of motion. Especially the lat tower could be 10cm higher.
• While the famous power rods haven't shown signs of wearing out after this first year, the resistance is not consistent. The 310lbs is split in rods of 5, 10, 10, 30, 50, 50lbs per side, and I've noticed that 30 + 10 + 10lbs is noticeably lighter than 50lbs. (I haven't measured at which point of the range of motion the nominal resistance applies. Resistance goes from very light at the start of the motion to hard towards the end, so the numbers don't really compare to training with real weights anyway. I haven't had any problem with this, though.)
• The documentation is lacking. Besides quality issues (wrong images, spelling), some of the stuff delivered in the box as well as some of the possible exercises are simply not used/described. (Example: it is possible to do squats, but the manual doesn't describe them.) So don't get this machine if you don't have an idea about what kinds of exercises you want to do.
• Assembling it is quite a project. But the documentation is good enough on the assembly if you have ever managed to assemble a Kinder Surprise toy, and you only do it once anyway, so it's not really an issue.

Closing remarks: I'll probably keep it for quite a while. I've thought about spending much less money and just getting a bench and a few weights, but I really like to have a lat tower. Overall I can do what I want to do with the exception of cardio workouts. The other angle is that there is quite a bit of bad press around about the founder of Nautilus, Arhur Jones. I can't comment on how trustworthy these articles are, and I can't comment on Nautilus' business practices today (Mr. Jones seems to be dead), so take this with a large pinch of salt before making a buying decision based on moral/ethical standards.

So a customer of mine needs to use RedHat. It's not their choice, the choice was forced upon them by the people who wrote the tools they need to use. The new version of this particular tool which they wanted to use requires RHEL5, and they're currently running RHEL4. So I had to upgrade their systems from RHEL4 to RHEL5. No sweat, right? Yeah, right. RedHat has this nice 'Red Hat Network', where you can manage your servers, and remotely mark packages for upgrade. You can also, by way of a command line tool, say that you want it to upgrade from one release to another. This can't be done remotely, which sortof makes sense; upgrading from one release to another might cause problems that you really really don't want if you don't have an immediate shell. So you should use 'up2date --upgrade-to-release=5' to upgrade from RHEL4 to RHEL5. Except that that doesn't work. For some reason, RedHat chose not to support in-place upgrades from RedHat 4 to RedHat 5. Instead, you have to download installation media, boot from them (thereby inducing downtime) and select 'upgrade' in the installer. Which part of this distribution is "Enterprise", again? Oh well. So I download the first ISO, and start the upgrade process. Surely that'll work, right? Hah. No, I really need all six images. So I download them, and start again. Some minutes into the installation, the HP iLO session that I'm using times out, and takes my Virtual CD device with it. As such, the installer-in-upgrade-mode obviously can't find the CD-ROM anymore, and produces a read error. This is totally normal. The system provides me with a 'retry' button. That button doesn't work. That is to say, it works, but after two or so files, the installation goes to a grinding halt. I had to reboot, and start over. So after rebooting, changing my iLO session settings so the timeout is at 120 minutes (the maximum) rather than 30 minutes (what appears to be the default), and I restart the upgrade process, making sure to click in the iLO interface every once in a while to keep it active. When it's somewhere at 80% and 3 hours into the upgrade process (god, why does that have to take so long?), it suddenly finds itself out of disk space. Well, shit happens, of course; guess there's nothing to blame RedHat with that one. But the way this situation is handled, is beyond my comprehension: The only 'recover' option is 'reboot'. There's no 'retry'. I don't mind diving to a second console, mucking about a bit on the mounted partition, and doing 'retry', but the installer doesn't even offer me that option. Instead, I have to choose 'reboot'. Never mind that the system is halfway through an installation process and might be totally and utterly broken, and not even bootable. Well, that's what rescue modes on installation media are for, I guess, but still. After cleaning out some old files that aren't needed anymore and going back to the installer, you'd suppose that the system could figure out what it'd already done, and continue from there, right? Well, no, it can't. It starts over from scratch, thereby forcing me to sit and wait for another three hours. Times like these make me remember why I'm a Debian developer, and not a RedHat one. Why I chose not to finish that Linux From Scratch project that I'd started after using RedHat for three years and about six months before becoming a Debian Developer. Why I stopped hitting my computer screen around that time. Stupid piece of shit. disclaimer: this is all IMAO, and you don't have to agree with me. But what's a blog for, if not to rant? ;-)

So, I've had this plan for ages to follow up with what I did with my nslu2, and hide a Debian box inside a working stereo. Finally I've started work on this little project. I bought a cheap stereo with CD/tape/radio, mostly on the criteria:

• It needed to be big enough to stand a chance of fitting my qnap board and a full size SATA drive in it. But not too big..
• It needed to have line in. Luckily, most stereos do, thanks to ipods.

I got lucky.. after removing the CD drive, ripping out the battery compartment, and making other modifications to the interior of the case, I had just enough space to fit the qnap board, with centimeters to spare in some places, and millimeters in others. It fits perfectly. The board's LEDs and reset button will be accessible via the CD slot. I was able to keep the radio and amp working. Wiring up a USB sound card to that was easy. I could have kept the cassette deck working, but since I don't have any more of that media, I stripped it, and am planning to put a USB hub in there, accessible by pressing Stop/Eject. ;-) It seems I will even have a fair bit of empty space. Pondering adding a small UPS. Or, I might toss in an access point.

As an obligatory note, Debian Lenny was released earlier today. Which means that sysadmins all over the world are starting to upgrade their servers. There is an oddly little known tool that each and every sysadmin should install on at least one server they maintain, called apt-listchanges. It lists changes made to packages since the currently installed version. Sure that information will be overwhelming on major upgrades, but what is useful even on major upgrades is the capability to parse News files in the same way. News files contain important information about the package in question. For example a maintainer could list known upgrade problems there, like is done in the lighttpd package. Or list changes in package specific default behaviour, like is done in Vim package. Sure, you will notice these in time, but it's nice to get a heads up before a problem bites you.

and the case against bzip2 Compression is with us all the time. I want to talk about general-purpose lossless compression here. There is a lot of agonizing over compression ratios: the size of output for various sizes of input. For some situations, this is of course the single most important factor. For instance, if you re Linus Torvalds putting your code out there for millions of people to download, the benefit of saving even a few percent of file size is well worth the cost of perhaps 50% worse compression performance. He compresses a source tarball once a month maybe, and we are all downloading it thousands of times a day. On the other hand, when you re doing backups, the calculation is different. Your storage media costs money, but so does your CPU. If you have a large photo collection or edit digital video, you may create 50GB of new data in a day. If you use a compression algorithm that s too slow, your backup for one day may not complete before your backup for the next day starts. This is even more significant a problem when you consider enterprises backing up terabytes of data each day. So I want to think of compression both in terms of resulting size and performance. Onward Starting Point I started by looking at the practical compression test, which has some very useful charts. He has charted savings vs. runtime for a number of different compressors, and with the range of different settings for each. If you look at his first chart, you ll notice several interesting things:

• gzip performance flattens at about -5 or -6, right where the manpage tells us it will, and in line with its defaults.
• 7za -2 (the LZMA algorithm used in 7-Zip and p7zip) is both faster and smaller than any possible bzip2 combination. 7za -3 gets much slower.
• bzip2 s performance is more tightly clustered than the others, both in terms of speed and space. bzip2 -3 is about the same speed as -1, but gains some space.

All this was very interesting, but had one limitation: it applied only to the gimp source tree, which is something of a best-case scenario for compression tools. A 6GB Test
I wanted to try something a bit more interesting. I made an uncompressed tar file of /usr on my workstation, which comes to 6GB of data. My /usr contains highly compressible data such as header files and source code, ELF binaries and libraries, already-compressed documentation files, small icons, and the like. It is a large, real-world mix of data. In fact, every compression comparison I saw was using data sets less than 1GB in size hardly representative of backup workloads. Let s start with the numbers:

Tool	MB saved	Space vs. gzip	Time vs. gzip	Cost
gzip	3398	100.00%	100.00%	0.15
bzip2	3590	92.91%	333.05%	0.48
pbzip2	3587	92.99%	183.77%	0.26
lzma -1	3641	91.01%	195.58%	0.28
lzma -2	3783	85.76%	273.83%	0.37

In the MB saved column, higher numbers are better; in all other columns, lower numbers are better. I m using clock seconds here on a dual-core machine. The cost column is clock seconds per MB saved. What does this tell us?

• bzip2 can do roughly 7% better than gzip, at a cost of a compression time more than 3 times as long.
• lzma -1 compresses better than bzip2 -9 in less than twice the time of gzip. That is, it is significantly faster and marginally smaller than bzip2.
• lzma -2 is significantly smaller and still somewhat faster than bzip2.
• pbzip2 achieves better wall clock performance, though not better CPU time performance, than bzip2 though even then, it is only marginally better than lzma -1 on a dual-core machine.

Some Pretty Charts First, let s see how the time vs. size numbers look: Like the other charts, the best area is the lower left, and worst is upper right. It s clear we have two outliers: gzip and bzip2. And a cluster of pretty similar performers. This view somewhat magnifies the differences, though. Let s add cat to the mix: And finally, look at the cost: Conclusions First off, the difference in time is far larger than the difference in space. We re talking a difference of 15% at the most in terms of space, but orders of magnitude for time. I think this pretty definitively is a death knell for bzip2. lzma -1 can achieve better compression in significantly less time, and lzma -2 can achieve significantly better compression in a little less time. pbzip2 can help even that out in terms of clock time on multicore machines, but 7za already has a parallel LZMA implementation, and it seems only a matter of time before /usr/bin/lzma gets it too. Also, if I were to chart CPU time, the numbers would be even less kind to pbzip2 than to bzip2. bzip2 does have some interesting properties, such as resetting everything every 900K, which could provide marginally better safety than any other compressor here though I don t know if lzma provides similar properties, or could. I think a strong argument remains that gzip is most suitable for backups in the general case. lzma -1 makes a good contender when space is at more of a premium. bzip2 doesn t seem to make a good contender at all now that we have lzma. I have also made my spreadsheet (OpenOffice format) containing the raw numbers and charts available for those interested. Update Part 2 of this story is now available, which considers more compression tools, and looks at performance compressing files individually rather than the large tar file.

Started updating our Debian Machines at Hosting Nation to Lenny. Smooth so far.

This is a followup to the earlier design of dh. After converting many packages to minimal rules files using dh, I find that it works very well for the 3 line rules file case, which works for 80% of packages. But getting from there to a slightly more complex rules file that 20% of packages need can be non-obvious. So, let's make it obvious. Suppose I want to run dh_compress -Xsomefile. But my rules file as it currently stands doesn't offer an easy place to run that command:

#!/usr/bin/make -f
%:
    dh $@

To use dh in this situation you had to go look up rather a lot of boilerplate, and to insert an explicit version of the right target, with the right dependencies, that called dh before and after your command:

#!/usr/bin/make -f
%:
    dh $@
binary: binary-indep
binary-indep:
    dh $@ --before dh_compress
    dh_compress -Xsomefile
    dh $@ --after dh_compress

That took me half an hour to come up with, and I wrote this stuff! Also, if I want to override a second command, I have to do it in the right order. So, the full flexability of debhelper is there, but it's too hard to get at. Here's the new new thing.

#!/usr/bin/make -f
%:
    dh $@
override_dh_compress:
        dh_compress -Xsomefile

What this does should be obvious. These override targets can be written for any debhelper command that you need to change the default behavior of, or even avoid running at all.

override_dh_auto_configure:
    ./configure --with-kitchen-sink
override_dh_auto_test:
    # The test suite for this package is busted on s390.
    if [ $DEB_BUILD_ARCH != s390 ]; then dh_auto_test; fi

The overrides can also be used to run additional stuff before, or after any debhelper command:

override_dh_clean:
    dh_clean
    rm -rf build_dir

So, I think that's beautiful. Simple, clean, no boilerplate, and intuitively obvious if you are familiar with debhelper.

---

FWIW, the implementation, which will be in the next debhelper release unless someone screams loudly, is not beautiful. Mostly because I sorta shot myself in the foot with the cute implicit "%:" rules. But, I got around it. :-)

"Hard SF" is a sub-genre of Science Fiction where "Science" is key. It's very hard to draw solid lines around stories that are examples of Hard SF, but it's easier to provide examples of things which are not: "Star Wars", for example, where space ships, travel and light sabres are window-dressing around a fantasy story. If you look at the Wikipedia article on Hard SF, in particular it's edit history, you will see people continually argue back and forth. Editor Jonathan Strahan recently asked for some recommendations of Hard SF authors to be included in a new Hard SF anthology. I stumbled across this via sfsignal. There are some good comments at both sites, and a fair amount of disagreement over whether certain authors should be considered Hard SF authors or not. Alastair Reynolds was cited by people several times. I really enjoy much of Reynold's writing, but I wouldn't consider it Hard SF. I argued this point and Reynolds himself joined the discussion.

I took the package of blackberries from the freezer, where it had sat, neglected since I picked them from my fecund vines in August. They had some frost on top; it didn't seem that the wait could have improved them. But, I figured, let's do this right. No store-bought pie crust, made my own with lots of love, and no little trepidation, since pie crust is not my strong suite. Put it back in the fridge to rest as I melted the frozen berries. Added some honey, then worried I'd sweetened it too much. Seemed too runny, so added some flour. Spent what seemed a long time criss-crossing strips of the pastry on top. Which, now that I look at it, doesn't seem to be quite the way it's typically done, but whatever. 45 minutes later, it came out, and bliss, it's the most perfect pie I've ever made. Crust acceptably flaky, berries just the right mix of tart and sweet. But mostly, it tastes of summer and care, and the wait seems to have been worth it. So that's my Debian release pie.

---

On the software side of things, I didn't do anything much toward making this release happen. Indeed, I have forgotten most of what I put into the mix earlier. (Something to do with armel, and firmware, and maybe boot menus?) It was a nice change to sit back and watch and not worry (much) about doing anything myself. I think that someone fell asleep in his chair while running the CD builds and had to get a wakeup call early this morning to finish. Dedication. Seems like there should be a better way, but you can't argue with the result. Previously: Easter Etch, Releases and Such, Where was I When ..

Alright, Debian GNU/Linux 5.0 AKA as Lenny has been released. Time for a Debian unstable unfreeze party! What does the new stable release bring for system administrators? I ll give an overview what news you might expect when upgrading from Debian GNU/Linux 4.0, codename Etch (released on 8th April 2007) to the current version Debian GNU/Linux 5.0, codename Lenny (released on 14th February 2009). I try to avoid duplicated information so make sure to read the release announcement and the official release notes for Lenny beforehand. Noteworthy Changes

• initrd-tools got replaced by initramfs-tools
• netkit-inetd got replaced by openbsd-inetd
• the default syslog daemon sysklogd got replaced by rsyslog
• new defaults when creating ext2/ext3 file systems: dir_index and resize_inode feature enabled by default and use blocksize = 4096, inode_size = 256 and inode_ratio = 16384 (see /etc/mke2fs.conf)
• improved IPv6 support
• init.d-scripts for dependency-based init systems
• Debian-Volatile (hosting packages providing data that needs to be regularly updated over time, such as timezones definitions, anti-virus signature files, ) is an official service
• EVMS (Enterprise Volume Management System) was removed
• compatibility with the FHS v2.3
• software developed for version 3.2 of the LSB
• official Debian Lenny live systems for the amd64 and i386 architectures
• several new d-i features

Virtualisation

• kvm: works out-of-the-box with Lenny
• OpenVZ: deploying OpenVZ systems works out-of-the-box now thanks to new OpenVZ kernel image flavors (no need for any further, external repositories)
• vserver: the according kernel images are available
• Xen: works (dom0 as well as domU support), see Xen @ Debian-Wiki

Virtualisation related new tools:

• ganeti: Cluster-based virtualization management software
• libvirt-bin: Libvirt is a C toolkit to interact with the virtualization capabilities of recent versions of Linux (and other OSes). The library aims at providing a long term stable C API for different virtualization mechanisms.
• virtinst: Programs to create and clone virtual machines
• virt-manager: desktop application for managing virtual machines
• xen-shell: Console based Xen administration utility
• xenstore-utils: Xenstore utilities for Xen
• xenwatch: Virtualization utilities, mostly for Xen

Desktop oriented packages like virtualbox and qemu are available as well of course. Noteworthy Updates This is a (selective) list of some noteworthy updates:

• apache2: upgrade from 2.2.3-4+etch6 to 2.2.9-10+lenny2
• apt: upgrade from version 0.6.46.4-0.1 to 0.7.20.1, bringing apt-get autoremove
• aptitude: upgrade from version 0.4.4-4 to 0.4.11.11-1~lenny1, bringing aptitude safe-upgrade which replaces aptitude upgrade and aptitude full-upgrade replacing aptitude dist-upgrade
• bash: upgrade from 3.1dfsg-8 to 3.2-4
• bind: upgrade from 1:8.4.7-1 / 1:9.3.4-2etch4 to 1:9.5.0.dfsg.P2-5.1
• cfengine: upgrade from 2.1.20-1 to 2.2.8-1
• courier: upgrade from 0.53.3-5 to 0.60.0-2
• dpkg: upgrade from 1.13.26 to 1.14.24, bringing dpkg triggers
• drbd: upgrade from 0.7.21-4 to 2:8.0.14-2
• dstat: upgrade from 0.6.3-2 to 0.6.7-1, the new version for example provides the topcpu and topmem modules so you can run dstat -c -M topcpu -dng -M topmem to identify cpu and memory intensive jobs
• fai: upgrade from 3.1.8 to version 3.2.16, see NEWS
• gcc/g++: upgrade from 4:4.1.1-15 to 4.3.2-1.1
• git: upgrade from 1:1.4.4.4-4+etch1 to 1:1.5.6.5-2
• initramfs-tools: upgrade from 0.85i to 0.92o, see new Lenny features
• iproute: upgrade from 20061002-3 to 20080725-2
• iptables: upgrade from 1.3.6.0debian1-5 to 1.4.2-5
• gnu libc: upgrade from 2.3.6.ds1-13etch9+b1 to 2.7-18
• lighttpd: upgrade from 1.4.13-4etch11 to 1.4.19-5
• linux-kernel: upgrade from kernel 2.6.18 to 2.6.26, see wiki.debian.org/KernelFAQ#new-features-in-lenny
• lvm2: upgrade from 2.02.06-4etch1 to 2.02.39-6
• mdadm: upgrade from 2.5.6-9 to 2.6.7.1-1, see README.upgrading-2.5.3
• mercurial: upgrade from 0.9.1-1+etch1 to 1.0.1-5.1
• mount: upgrade from 2.12r-19etch1 to 2.13.1.1-1, bringing read-only binds, make-shared, make-slave, make-private, make-unbindable
• mysql: upgrade from 5.0.32-7etch8 to 5.0.51a-23
• nagios: upgrade from 2.6-2+etch1 to 3.0.6-3, see Upgrading From Nagios 2.x and What s New in Nagios 3 for details
• openldap: upgrade from 2.3.30-5+etch2 to 2.4.11-1
• openssh: upgrade from 1:4.3p2-9etch3 to 1:5.1p1-5
• postgresql: upgrade from 7.5.22 / 8.1.15-0etch1 to 8.3.5-1
• perl: upgrade from 5.8.8-7etch6 to 5.10.0-19
• php: upgrade from 5.2.0-8+etch13 to 5.2.6.dfsg.1-1+lenny2
• postfix: upgrade from 2.3.8-2+etch1 to 2.5.5-1.1
• puppet: upgrade from 0.20.1-1 to 0.24.5-3
• python: upgrade from 2.4.4-2 to 2.5.2-3
• ruby: upgrade from 1.8.5-4etch4 to 1.8.7.72-3
• samba: upgrade from 3.0.24-6etch10 to 2:3.2.5-4
• subversion: upgrade from 1.4.2dfsg1-2 to 1.5.1dfsg1-2
• tcl/tk: upgrade from 8.4.12-1etch2 to 8.5.3-4
• utilx-linux: upgrade from 2.12r-19etch1 to 2.13.1.1-1, brings new utilities like:
  • chrt: manipulate real-time attributes of a process
  • ionice: get/set program io scheduling class and priority
  • rename.ul: rename the specified files by replacing the first occurrence of from in their name by to
  • taskset: retrieve or set a process s CPU affinity
• zsh: upgrade from 4.3.2-25 to 4.3.6-6

New packages Lenny ships over 7000 new packages. Lists of new/removed/replaced packages are available online. I ll name 238 sysadmin related packages that might be worth a look. (Note: I don t list addon stuff like optional server-modules, docs-only and kernel-source related packages. I plan to present some of the following packages in more detail in separate blog entries.)

• ack-grep: A grep-like program specifically for large source trees
• acpitail: Show ACPI information in a tail-like style
• adns-tools: Asynchronous-capable DNS client library and utilities
• aggregate: ipv4 cidr prefix aggregator
• aosd-cat: an on screen display tool which uses libaosd
• apt-cacher-ng: Caching proxy for distribution of software packages
• apt-cross: retrieve, build and install libraries for cross-compiling
• aptfs: FUSE filesystem for APT source repositories
• apt-p2p: apt helper for peer-to-peer downloads of Debian packages
• apt-transport-https: APT https transport, use deb https://foo distro main lines in the sources.list
• arp-scan: arp scanning and fingerprinting tool
• array-info: command line tool reporting RAID status for several RAID types
• balance: Load balancing solution and generic tcp proxy
• bash-completion: programmable completion for the bash shell
• blktrace: utilities for block layer IO tracing
• daemonlogger: simple network packet logger and soft tap daemon
• daemontools: a collection of tools for managing UNIX services
• dbndns: Debian fork of djbdns, a collection of Domain Name System tools
• dcfldd: enhanced version of dd for forensics and security
• dctrl2xml: Debian control data to XML converter
• debomatic: automatic build machine for Debian source packages
• desproxy: tunnel TCP traffic through a HTTP proxy
• detox: utility to replace problematic characters in filenames
• di-netboot-assistant: Debian-Installer netboot assistant
• dish: the diligence/distributed shell for parallel sysadmin
• djbdns: a collection of Domain Name System tools
• dns2tcp: TCP over DNS tunnel client and server
• dnscache-run: djbdns dnscache service
• dnshistory: Translating and storing of IP addresses from log files
• dnsproxy: proxy for DNS queries
• dsyslog: advanced modular syslog daemon
• etckeeper: store /etc in git, mercurial, or bzr
• ext3grep: Tool to help recover deleted files on ext3 filesystems
• fair: high availability load balancer for TCP connections
• fatresize: FAT16/FAT32 filesystem resizer
• flog: dump STDIN to file and reopen on SIGHUP
• freeradius-utils: FreeRadius client utilities
• ganeti: Cluster-based virtualization management software
• gfs2-tools: Red Hat cluster suite - global file system 2 tools
• gitosis: git repository hosting application
• gptsync: GPT and MBR partition tables synchronisation tool
• grokevt: scripts for reading Microsoft Windows event log files
• grub2: GRand Unified Bootloader, version 2
• gt5: shell program to display visual disk usage with navigation
• haproxy: fast and reliable load balancing reverse proxy
• havp: HTTP Anti Virus Proxy
• heirloom-mailx: feature-rich BSD mail(1)
• hfsprogs: mkfs and fsck for HFS and HFS+ file systems
• hinfo: Check address ownership and DNSBL listings for spam reporting
• hlbr: IPS that runs over layer 2 (no TCP/IP stack required)
• hobbit: monitoring system for systems, networks and applications - server
• hotwire: Extensible graphical command execution shell
• hunchentoot: the Common Lisp web server formerly known as TBNL
• ifupdown-extra: Network scripts for ifupdown
• ike: Shrew Soft VPN client - Daemon and libraries
• incron: cron-like daemon which handles filesystem events
• inoticoming: trigger actions when files hit an incoming directory
• iodine: tool for tunneling IPv4 data through a DNS server
• iotop: simple top-like I/O monitor
• ipplan: web-based IP address manager and tracker
• ips: Intelligent process status
• iscsitarget: iSCSI Enterprise Target userland tools
• isns: Internet Storage Naming Service
• itop: simple top-like interrupt load monitor
• iwatch: realtime filesystem monitoring program using inotify
• jetring: gpg keyring mantainance using changesets
• john: active password cracking tool
• kanif: cluster management and administration swiss army knife
• keepassx: Cross Platform Password Manager
• keysafe: A safe to put your passwords in
• killer: Background job killer
• kpartx: create device mappings for partitions
• kvm: Full virtualization on x86 hardware
• latencytop: A tool for developers to visualize system latencies
• lbcd: Return system load via UDP for remote load balancers
• ldb-tools: LDAP-like embedded database - tools
• ldnsutils: ldns library for DNS programming
• lfhex: large file hex editor
• live-helper: Debian Live build scripts
• live-magic: GUI frontend to create Debian LiveCDs, netboot images, etc.
• logapp: supervise execution of applications producing heavy output
• lsat: Security auditor tool
• lustre-utils: Userspace utilities for the Lustre filesystem
• lwat: LDAP Web-based Administration Tool
• maatkit: Command-line utilities for MySQL
• mantis: web-based bug tracking system
• memdump: memory dumper
• memlockd: daemon to lock files into RAM
• metainit: Generates init scripts
• mirmon: monitor the state of mirrors
• mkelfimage: utility to create ELF boot images from Linux kernel images
• mongrel: A small fast HTTP library and server for Ruby
• monkey: fast, efficient, small and easy to configure web server
• monkeytail: tail variant designed for web developers monitoring logfiles
• mpy-svn-stats: Simple and easy to use svn statistics generator
• mr: a Multiple Repository management tool
• msr-tools: Utilities for modifying MSRs from userspace
• mtd-utils: Memory Technology Device Utilities
• munge: authentication service to create and validate credentials
• mxallowd: Anti-Spam-Daemon using nolisting/iptables
• mylvmbackup: quickly creating backups of MySQL server s data files
• myrescue: rescue data from damaged harddisks
• mysql-proxy: high availability, load balancing and query modification for mysql
• mysqltuner: high-performance MySQL tuning script
• nagvis: Visualization addon for Nagios
• ncdu: ncurses disk usage viewer
• netrw: netcat like tool with nice features to transport files over network
• netsend: a speedy filetransfer and network diagnostic program
• network-config: Simple network configuration tool
• nfdump: netflow capture daemon
• ngetty: getty replacement - one single daemon for all consoles
• nilfs2-tools: Continuous Snapshotting Log-structured Filesystem
• ninja: Privilege escalation detection system for GNU\Linux
• noip2: client for dynamic DNS service
• nsd3: authoritative domain name server (3.x series)
• ntfs-3g: read-write NTFS driver for FUSE
• nulog: Graphical firewall log analysis interface
• nuttcp: network performance measurement tool
• ocsinventory-server: Hardware and software inventory tool (Communication Server)
• odt2txt: simple converter from OpenDocument Text to plain text
• olsrd: optimized link-state routing daemon (unik-olsrd)
• onesixtyone: fast and simple SNMP scanner
• openais: Standards-based cluster framework (daemon and modules)
• opencryptoki: PKCS#11 implementation for Linux (daemon)
• openvas-client: Remote network security auditor, the client
• ophcrack: Microsoft Windows password cracker using rainbow tables
• op: sudo like controlled privilege escalation
• otpw-bin: OTPW programs for generating OTPW lists
• packeth: Ethernet packet generator
• paperkey: extract just the secret information out ouf OpenPGP secret key
• paris-traceroute: New version of well known tool traceroute
• password-gorilla: a cross-platform password manager
• pathfinderd: Daemon for X.509 Path Discovery and Validation
• pathfinder-utils: Utilities to use with the Pathfinder Daemon
• pcaputils: specialized libpcap utilities
• pcp: System level performance monitoring and performance management
• perlconsole: small program that lets you evaluate Perl code interactively
• pgloader: loads flat data files into PostgreSQL
• pgpool2: connection pool server and replication proxy for PostgreSQL
• pgsnap: PostgreSQL report tool
• pmailq: postfix mail queue manager
• pnputils: Plug and Play BIOS utilities
• policykit: framework for managing administrative policies and privileges
• postfwd: Postfix policyd to combine complex restrictions in a ruleset
• postpone: schedules commands to be executed later
• powertop: Linux tool to find out what is using power on a laptop
• prayer: standalone IMAP-based webmail server
• prelude-correlator: Hybrid Intrusion Detection System [ Correlator ]
• privbind: Allow unprivileged apps to bind to a privileged port
• pssh: Parallel versions of SSH-based tools
• ptop: PostgreSQL performance monitoring tool akin to top
• pyftpd: ftp daemon with advanced features
• rancid-core: rancid Really Awesome New Cisco confIg Differ
• rancid-util: Utilities for rancid
• rdnssd: IPv6 recursive DNS server discovery daemon
• rdup: utility to create a file list suitable for making backups
• reglookup: utility to read and query Windows NT/2000/XP registry
• rgmanager: Red Hat cluster suite - clustered resource group manager
• rinse: RPM installation environment
• rofs: Read-Only Filesystem for FUSE
• rsyslog: enhanced multi-threaded syslogd
• safe-rm: wrapper around the rm command to prevent accidental deletions
• samba-tools: tools provided by the Samba suite
• samdump2: Dump Windows 2k/NT/XP password hashes
• scalpel: A Frugal, High Performance File Carver
• scamper: advanced traceroute and network measurement utility
• scanmem: Locate and modify a variable in a running process
• schedtool: Queries/alters process scheduling policy and CPU affinity
• screenie: a small and lightweight GNU screen(1) wrapper
• scrounge-ntfs: Data recovery program for NTFS filesystems
• ser: Sip Express Router, very fast and configurable SIP proxy
• serverstats: a simple tool for creating graphs using rrdtool
• shutdown-at-night: System to shut down clients at night, and wake them in the morning
• sipcrack: SIP login dumper/cracker
• sks: Synchronizing OpenPGP Key Server
• slack: configuration management program for lazy admin
• sma: Sendmail log analyser
• smbind: PHP-based tool for managing DNS zones for BIND
• smbnetfs: User-space filesystem for SMB/NMB (Windows) network servers and shares
• softflowd: Flow-based network traffic analyser
• speedometer: measure and display the rate of data across a network connection
• spf-milter-python: RFC 4408 compliant Python SPF Milter for Sendmail and Postfix
• spf-tools-perl: SPF tools (spfquery, spfd) based on the Mail::SPF Perl module
• spf-tools-python: sender policy framework (SPF) tools for Python
• sqlgrey: Postfix Greylisting Policy Server
• ssdeep: Recursive piecewise hashing tool
• sshfp: DNS SSHFP records generator
• sshm: A command-line tool to manage your ssh servers
• sshproxy: ssh gateway to apply ACLs on ssh connections
• sslscan: Fast SSL scanner
• strace64: A system call tracer for 64bit binaries
• sucrack: multithreaded su bruteforcer
• supercat: program that colorizes text for terminals and HTML
• superiotool: Super I/O detection tool
• system-config-lvm: A utility for graphically configuring Logical Volumes
• system-config-printer: graphical interface to configure the printing system
• tack: terminfo action checker
• taktuk: efficient, large scale, parallel remote execution of commands
• tcpwatch-httpproxy: TCP monitoring and logging tool with support for HTTP 1.1
• terminator: Multiple GNOME terminals in one window
• timelimit: Simple utility to limit a process s absolute execution time
• tipcutils: TIPC utilities
• tor: anonymizing overlay network for TCP
• tpm-tools: Management tools for the TPM hardware (tools)
• tracker-utils: metadata database, indexer and search tool - commandline tools
• tumgreyspf: external policy checker for the postfix mail server
• ucspi-tcp: command-line tools for building TCP client-server applications
• unbound: validating, recursive, caching DNS resolver
• unhide: Forensic tool to find hidden processes and ports
• uniutils: Tools for finding out what is in a Unicode file
• unsort: reorders lines in a file in semirandom ways
• uphpmvault: upload recovery images to HP MediaVault2 via Ethernet
• usermode: Graphical tools for certain user account management tasks
• utf8-migration-tool: Debian UTF-8 migration wizard
• uuid-runtime: universally unique id library
• vblade-persist: create/manage supervised AoE exports
• vde2: Virtual Distributed Ethernet
• vdmfec: recover lost blocks using Forward Error Correction
• virtinst: Programs to create and clone virtual machines
• virt-manager: desktop application for managing virtual machines
• virtualbox-ose: x86 virtualization solution - binaries
• virt-viewer: Displaying the graphical console of a virtual machine
• watchupstream: Look for newer upstream releases
• whirlpool: Implementation of the whirlpool hash algorithm
• win32-loader: Debian-Installer loader for win32
• xavante: Lua HTTP 1.1 Web server
• xdelta3: A diff utility which works with binary files
• xen-shell: Console based Xen administration utility
• xenstore-utils: Xenstore utilities for Xen
• xenwatch: Virtualization utilities, mostly for Xen
• xfingerd: BSD-like finger daemon with qmail support
• xl2tpd: a layer 2 tunneling protocol implementation
• xrdp: Remote Desktop Protocol (RDP) server
• yersinia: Network vulnerabilities check software
• zerofree: zero free blocks from ext2/3 file-systems
• zipcmp: compare contents of zip archives
• zipmerge: merge zip archives
• ziproxy: compressing HTTP proxy server

Further Ressources

• Debian GNU/Linux 5.0 released - Release Announcement
• Debian GNU/Linux 5.0 - Release Notes
• Etch2Lenny Upgrade @ Debian-Wiki
• New in Lenny @ Debian-Wiki

If you are following this blog, you probably remember per-process namespaces. Today, I m going to tell you how I did use them in the process of preparing this server to be upgraded to Lenny. I must say I was not using the most recent stuff on this server, and this is why I needed such preparation. First, this server is still running with php4. And well, the following line in the apt-get dist-upgrade output got my full attention (emphasis mine):

The following packages will be REMOVED:
cacti libapache2-mod-php4 libarchive-tar-perl libcurl3-openssl-dev
libgssapi2 libpci2 librpm4 libssp0 linux-kernel-headers modutils php4-mysql


While this server is not important enough not to be broken for a couple hours, I do like to test procedures that could help on servers that are. The first thing I needed to do on this server was obviously to upgrade php. But we all know how php applications are not fully compatible with all versions of php, so I also needed to test the upgrade was not breaking anything. On a server you don t care much about, you can just upgrade, test if all is okay, and be done with it. Obviously, if all is not okay, your visitors will see it, and you may also have a bad time downgrading. Another way to perform the upgrade is to have a similarly installed server on the side, test and validate the upgrade there, and replicate the upgrade on the production server if everything is fine. However, that does require additional resources, and possibly to setup them if they don t already exist. A cheaper way to do the above is to do it on the production server, both in-place and on the side (you ll see what I mean), using unionfs and per-process namespaces. Full containers could be used instead of per-process namespaces (such as openvz, vserver or lxc), but they still require much more setup, especially when you don t use them in the first place. Chroots could work just as well as per-process namespaces, but one of the ideas here is to expose the per-process namespaces feature, and allow for improvement of this procedure with pid and network namespaces, which are not available in Etch (where I m starting from), but are in Lenny. Unionfs allows to merge several directories into a single one, accessing some read-only, and others read-write. Installing unionfs is as easy as running the following command:

apt-get install unionfs-modules- uname -r

I won t describe all the kinds of setups that are possible with unionfs, but only one typical use case, which is what we will be using here:

# mkdir /tmp/root-cow
# mount -t unionfs -o dirs=/tmp/root-cow:/=ro none /mnt


The first thing we do here is to create an empty directory. Next, we merge it with the root filesystem (/), that we will keep read-only (meaning unionfs won t allow itself to write there), and mount the merged filesystem under /mnt. none could just be anything, as there is no device to be mounted. The result, in /mnt, is just something that looks like the root filesystem:

# ls /
bin boot dev etc home initrd lib media mnt opt proc root sbin srv sys tmp usr var
# ls /mnt
bin boot dev etc home initrd lib media mnt opt proc root sbin srv sys tmp usr var


But creating or modifying a file will do so in /tmp/root-cow:

# echo a > /mnt/a
# cat /a
cat: /a: No such file or directory
# cat /tmp/root-cow/a
a
# echo foo.com > /mnt/etc/mailname
# cat /etc/mailname
glandium.org
# cat /tmp/root-cow/etc/mailname
foo.com
# find /tmp/root-cow
/tmp/root-cow
/tmp/root-cow/etc
/tmp/root-cow/etc/mailname
/tmp/root-cow/a


Keep in mind that it doesn t include submounts, though:

# ls /var
backups cache lib local lock log mail opt run spool tmp www
# ls /mnt/var


This means we ll have to also mount /var, /proc, /dev, and /sys. That s enough testing for now, and we ll first do some cleanup before going after the real job:

# umount /mnt
# rm -rf /tmp/root-cow


Using the newns utility from my first post on per-process namespaces, let s create a new namespace to keep our testing private, and populate it with the necessary mount points:

# newns
# umount /tmp
# mount -t tmpfs tmpfs /tmp
# mkdir /tmp/root-cow /tmp/var-cow
# mount -t unionfs -o dirs=/tmp/root-cow:/=ro none /mnt
# mount -t unionfs -o dirs=/tmp/var-cow:/var=ro none /mnt/var
# cd /mnt
# pivot_root . mnt
# mount --move /mnt/proc /proc
# mount --move /mnt/sys /sys
# mount --move /mnt/lib/init/rw /lib/init/rw
# mount --move /mnt/tmp /tmp
# mount --move /mnt/dev /dev


The second and third statements are useful to avoid sharing /tmp with the main namespace, which means the directories we create in the fourth statement won t be visible in /tmp outside this namespace. The fifth and sixth statements put the union filesystems in place. As my system only has a separate /var filesystem (no /usr, and I don t care about /home here), I only need to setup these two. Feel free to add more union filesystems as necessary. The pivot_root call allows to switch to the unionfs ed root: everything under /mnt (/mnt and /mnt/var in our case) will be remounted under /, while what was under / is remounter under /mnt under the new root.
This means, in our case, that we have / and /var as union filesystems, while the old / and /var are respectively in /mnt and /mnt/var. It also means /dev, /proc, /sys and other filesystems are remounted as /mnt/dev, /mnt/proc, /mnt/sys, etc., which is why we next mount --move all of them to a better place in our namespace. Once all this setup is done, we are ready to do our php upgrade test. As Etch doesn t have support for neither PID namespaces nor network namespaces, we ll still have some conflicts with the main namespace for TCP port binding and process handling, so we need to be a little careful:

# rm /var/run/apache2.pid
# echo Listen 8080 > /etc/apache2/ports.conf


With these changes, we can now safely start apache2 in this namespace, at the same time the main apache2 runs in the main namespace:

# /etc/init.d/apache2 start


Now, there are actually 2 problems showing up in apache in this setup. The first one is that displaying static files doesn t work. At all. It appears unionfs doesn t support sendfile(). Which is fine. But apache doesn t check for sendfile() s error cases and doesn t fallback to a working solution when sendfile() doesn t work. So we have to manually disable it:

# echo EnableSendfile off > /etc/apache2/conf.d/sendfile.conf


The second one is that access to the mysql socket doesn t work properly under the unionfs. I didn t want to investigate further, so I only worked around the issue by forcing to use the socket outside the unionfs:

# mount --bind /mnt/var/run/mysqld /var/run/mysqld


Finally, we can test and validate our php upgrade:

# apt-get install php5-mysql php5-cli php5-snmp libapache2-mod-php5
# /etc/init.d/apache2 restart
# apt-get remove --purge php4-common


Note that installing libapache2-mod-php5 will remove libapache2-mod-php4, and apache2 gets restarted, taking into account this change. But the php modules (php5-mysql and php5-snmp) install is only going to happen after that, and no apache2 restart is triggered then, which leaves a half working php setup
Also note that the cacti setup in etch supposes php4 is installed, using a IfModule statement against mod_php4.c and none for mod_php5.c, which means a part of its setup doesn t work out of the box (most notably, the DirectoryIndex). Once all is validated, all we need to do is to stop apache2 and exit from the namespace. The union filesystems and temporary filesystems are then automagically cleaned-up and the namespace disappears as well as all modifications we just did, since all processes using the namespace ended. We are now free to upgrade on the production server, as we know all the side effects. I ll now be able to upgrade to Lenny without php getting removed.

This weekend I have mostly been investigating Common Lisp. If you're new to the environment then you may get started via a simple:

apt-get install clisp

Once you've done that the following resources will help you decide whether you're wasting your time or not:

• Practical Common Lisp
• The Common Lisp Cookbook

It has been a long time since I've had so much fun - even though the first thing I did was experiment with implementing a spam filter - although in my defense I was only classifying based upon the links contained in the mails. So for example:

• http://gmail.com/ - Good.
• http://www.facebook.com/ - Good. (ha!)
• http://tallbear.com - Spam
• http://pamperflow.com - Spam

(NSFW links) ObFilm: Batman

What a happy day on the wedding anniversary of our civil marriage we got a great gift. Our favorite distribution released a new major version.
Like every time it s the best distribution our project released so far. Many thanks to all those people who did make that happen. If you not married today, maybe you have the possibility to dedicate Lenny to your valentine. :)

The long-awaited, big and stable Debian GNU/Linux Lenny 5.0 has been released! Great job everyone and congratulations! In this release the Debian/Ruby Extra team has an ever greater presence. We were able to increase the number of mainted libraries and applications to over 95 packages. Also quite a lot of issues were resolved and I am proud that we obtained this quality level. For Squeeze we plan to deal with the transition to Ruby 1.9 and with possible multiple interpreter-support (still Ruby 1.8, JRuby?). We might also deal with the gems, who knows? :) P.S. Indeed, did I just started posting without explaining why I was away for over 7 months?