Joys of schroot Schroot is good! Try schroot. You have LVM set up and some free extents in a vg? Try this: Then apt-get install schroot and put this in your /etc/schroot/schroot.conf: Now, schroot -c sid will make a LVM snapshot of /dev/YOUR-VG-NAME/schroot-sid, mount it, set it up as a nice working system, and chroot into it running a shell. Have a look around: resolv.conf is set up, your home directory is bind mounted, /proc and /sys are mounted. When you exit the chroot, the LVM snapshot disappears, and /dev/YOUR-VG-NAME/schroot-sid is unchanged. If you want to modify /dev/YOUR-VG-NAME/schroot-sid, you just schroot -c sid-source. In my laptop, I did this: At this point, to build a debian package like if I were using pbuilder, I do this: You can repeat this whole process with lenny and etch, for example, and if you are on an amd64 system, you can also do sid32, lenny32 and etch32. If you don't want to redownload packages all the time, install approx and configure the -source chroots to use it. This also allows you to easily switch all systems from one mirror to another if, for example, you are traveling abroad. If you want to do the same as pbuilder update, you can do this: You can use schroot -c sid-source for all sorts of permanent maintenance works. You can also create persistent named snapshots: This will only disappear when you do: And you can enter the schroot as many time as you want with: You can also enter a temporary snapshot as many times as you want, by using the random name that schroot generated for it (you can see it, for example, when running df or mount). You fear that lvm snapshots are a little fragile and a little slow at the moment? It's not a problem: your home is bind-mounted inside the chroot, so you can do all the work in your normal home filesystem, without stressing the snapshot. This is good not only to build packages. For example, now that svk is broken in sid and lenny, I just did schroot -b -c etch -n svk, installed svk on it, and then every time I need svk, I can do schroot -r -c svk and have it ready.

Monkeytail is a neat little tool which allows web developers and sysadmins to easily watch new additions to their log files by tailing them (i.e. tail -f) and providing helpful vertical spacing after detecting pauses in the output.

What makes monkeytail so useful is its ability to easily tail multiple files together, even when it involves connecting to these machines over ssh and running sudo on them. For example, you could tail all of your load-balanced servers by doing:

mtail @webservers

where webservers is an alias defined in ~/.mtailrc:

<group webservers>
    sudo yes
    <file>
        filename /var/log/apache2/error.log
        prefix "server1: "
        host server1.example.com
    </file>
    <file>
        filename /var/log/apache2/error.log
        prefix "server2: "
        host server2.example.com
    </file>
</group>

It is available in Debian unstable and from its official homepage.

Kudos to Martyn Smith for turning this idea into a release!

Ten years ago, most of us wouldn't have dreamt we'd be managing terabits of storage, tens of megabits of bandwidth, arrays of network-distributed services. The height of a programmer's worry would likely be choice of UI toolkit or finding the right way to indent code, and the height of consumer concern deciding which room to put the new computer in. Now the problems associated with managing large networks are becoming real for everyone, right down to the consumer level. Stupendously large amounts of computing resource are available at an instant. Your household probably has more than a terabyte of storage already. Issues such as single sign-on are going to hit home over the next year, as networked computing and entertainment devices profilerate. Features such as Apple's Time Machine will be increasingly vital — software that makes traditionally gnarly sysadmin tasks consumer-friendly. The rebranding of .Mac into "Mobile Me" is also a step in this direction. The impact on developers As software developers, we also have to cope with the effects of this resource-richness. For small sums of money we can get access to large computing clusters, geographically redundant hosting services. Our programs have left the desktop and found their new home on the web. System administration issues loom large upon us, security concerns lurk auspiciously in the corners of our minds. Although the cost of infrastructure has dropped radically, other costs remain high and are going to stay that way. System administrators are not only grumpy, they demand high wages. Commercial software license fees spiral out of control: traditional per-CPU licensing models make little sense when you can quickly bring up tens of machines. The cost in power is already troubling large companies, and there's no reason to suspect the problems won't ripple down. Help is at hand from a variety of technologies. If they don't yet make massive resource management trivial, they at least make it possible. Some of these also inhabit the weird territory of being both the source of a problem and a solution at the same time: virtualization, for example. Distributed revision control systems Distributed revision control is a technology whose time has finally come in popular circles, thanks in part to Linus Torvald's Git system. DRCS has several important impacts on today's developer:

• Branching and, importantly, merging become much cheaper, allowing agile and flexible iterations of development.
• Loosely connected and geographically diverse development becomes much easier. Even within a single organization it is not uncommon to find teams spanning countries, time zones. Complex multi-site VPN setups aren't necessary when a few SSH keys can do the job.
• Revision control becomes packaging on the cheap. Like it or not, the mere tagging of a source tree has now become a valid option for releasing software.

All these trends lower the barrier to entry, increase collaboration and agility of development. You can the value of this as more software tools become free. Selling such tools is rapidly becoming a thing of the past, the advantages of sharing enable the developers at the sharp end to get their jobs done quicker. However, such increased agility and, well, messiness leave other problems to solve, which the next two technologies address. Virtualization Hardware-as-a-service, infrastructure-as-a-service, call it what you will. The ability to create what we used to call entire machines, pick them up and move them around the network is revolutionary, and it's something that will have a real impact on regular developers. The benefits are at several levels.

• Agile infrastructure — a ready supply of new machines makes it a lot easier and cheaper to test different scenarios, architectures, and to separate concerns. If things go wrong, throw away the image and start over. It's all about cutting the administration load.
• A packaging solution — the new macroeconomics of software distribution mean that distributing entire machine images which communicate exclusively via the network is now a feasible way to distribute your software. We must adjust to the notion of distributing appliances, not code. We may mourn the lost crafts of creating RPMs or installers, but let's face it, it's now a waste of time.
• New business models — your application can now be delivered as a black-box appliance, circumventing compatibility issues, or as a service, with virtualization part of the solution to scaling.

Configuration management Computing is a zero-sum game, and despite our increased ability to create and distribute software, problems still exist. We just pushed them to the next level. In good part, this next level is the problem of configuration management. We now have networks and clusters of (virtual) machines, software so agile we need six decimal places to describe its revision levels, and network and authentication paths that are starting to tangle. How do we manage that? One thing developers crave is repeatability. That's why we love our makefiles, autoconf, Ant, rake and so on. It's the one time even the most imperative-minded programmer writes declarative code. We like to say "let the world be like this." Our new sprawling world lacks this feature, and the best of our old toolkits — .debs, RPMs — address things only at the level of packages in a single environment. So developers must look to the world of operations, a territory we probably thought we needn't enter. In this world the new "make" is called Puppet. You write recipes to describe how things ought to be, and Puppet will make it so. I've been spending some time digging into Puppet, and feel excited by the confidence it's giving me. Now my applications exceed single source trees, and single machines, it gives me the means to tie the whole together. This article was going to be solely about Puppet, but that will have to wait now for another time. It's likely you'll have played with virtual machines and distributed revision control, but have you tried Puppet yet? Give it a spin, and let your mind wander over the benefits for your organization and development approaches. Conclusions
For developers and users alike, our world is changing. Hardware, connectivity and increasingly software is becoming cheap or free. The solidity of the old things we put value on — real things you can touch like disks — is eroding. What really matters is our data, our creations, and their communication. If they don't quite yet exist in a universal "cloud" yet, they're certainly getting frisky. As vendors provide solutions for consumers to manage their new domestic infrastructure, developers must look to network-aware toolkits and operations techniques to manage and get the best from their emergent infrastructures. Also on this topic:

• In search of agile infrastructure for web applications (June 2006)
  

Join the conversation about this post

Even though the Debian installer can set up encrypted partitions, it is optimised for systems with a single data partition, unless you want to enter multiple passphrases when the system boots. The installer configures a LUKS volume using cryptsetup, but it provides no mechanism for the use of key files, only interactive passphrases. I like partitioning my disks and use different filesystems for /tmp, /home, /var, and /usr/local for a number of reasons. I don t like entering more passphrases than necessary. If you can identify with that, the following is for you.

Installing the system The first step to setting up an encrypted Debian system is to perform a normal Debian system installation. When you are asked to partition your harddrive and create filesystems, set up all partitions as encrypted volumes (I suggest to go with the installer defaults and use dm-crypt and AES with the default settings, simply because I have no reason to doubt the installer dveeloper team s choices). Make sure to erase the disks in the process the installer has an option for that. Set up the swap partition as an encrypted volume too, but don t worry too much about the settings at this point; we will recreate the swap partition later. Unless you want to boot off an external medium, such as a USB stick, you will need to create an unencrypted partition for /boot. I will return to this topic, which has security implications, further down. The installer will ask you for passphrases for each of the volumes you create. I suggest you pick a secure passphrase for the root volume (/), but simple passphrases for all the other ones (such as a ), since we will reconfigure them to use key files instead.

Using key files A key file is like a passphrase stored in a file on disk; as opposed to what you know , it s a what you have security asset. Thus, you need to store the file somewhere. When I boot up my system, I unlock the root partition with a passphrase entered interactively, which makes the root filesystem available. I store key files for all other volumes in /etc/keys. Obviously, I need to tell cryptsetup to use those. The first step is to create a key file for each partition and to add it as a decryption key to the LUKS volume. You can do all of the following without unmounting the filesystems. See the following example for hda6, which will prompt for the simple passphrase we entered above to unlock the key ring when adding the key file:

umask 077
mkdir /etc/keys
dd if=/dev/urandom of=/etc/keys/hda6.luks bs=4k count=1
cryptsetup luksAddKey /dev/hda6 /etc/keys/hda6.luks
cryptsetup luksKillSlot /dev/hda6 0 --key-file /etc/keys/hda6.luks

The last command wipes the simple passphrase from the key ring and thus makes it unusable. Now we need to tell cryptsetup to use the key file by editing /etc/crypttab and ensuring a line such as the following exists:

hda6_crypt /dev/hda6 /etc/keys/hda6.luks luks

This tells cryptsetup to create the cryptographic volume hda6_crypt from the base device /dev/hda6, using the key file we created above, and letting it know that it s dealing with a LUKS volume. Repeat this for every partition except your root and swap partitions.

Encrypting the swap partition If you are using an encrypted Debian system, you likely have some security requirements to meet. If that s the case, you must also use an encrypted swap partition. The swap partition can be encrypted in two ways:

• it can be recreated on every boot, using a random passphrase, or
• it can be created like the other encrypted volumes with a persistent passphrase

If you want to use suspend-to-disk, you cannot use the first approach as it would overwrite your memory footprint stored in the swap partition. Furthermore, you cannot use a key file like the other partitions, since the root filesystem is not (and must not) be mounted by the time the resume process starts and needs to read the decrypted swap partition. The way I solved this is by telling cryptsetup to compute the passphrase of the swap partition from the decryption key of the volume holding the root filesystem; the cryptsetup package implements this with /lib/cryptsetup/scripts/decrypt_derived. Thus, to set up the swap partition, I do the following, assuming hda2 is the partition holding the encrypted swap and the root filesystem is in hda5_crypt:

swapoff /dev/mapper/hda2_crypt
cryptsetup luksClose hda2_crypt
dd if=/dev/urandom of=/dev/hda2
/lib/cryptsetup/scripts/decrypt_derived hda5_crypt \
    cryptsetup luksFormat /dev/hda2 --key-file -
/lib/cryptsetup/scripts/decrypt_derived hda5_crypt \
    cryptsetup luksOpen /dev/hda2 hda2_crypt --key-file -
mkswap /dev/mapper/hda2_crypt

To tell the system about this swap partition, we need to add it to /etc/crypttab and /etc/fstab; make sure, those files contain lines like the following:

/etc/crypttab:
  hda2_crypt /dev/hda2 hda5_crypt luks,keyscript=/lib/cryptsetup/scripts/decrypt_derived
/etc/fstab:
  /dev/mapper/hda2_crypt swap swap sw 0 0

With this in place, as soon as you configure the system for suspend-to-disk, the swap partition will be automatically set up alongside the root filesystem very early during the boot sequence. To figure out which swap partition to make available at that point, cryptsetup checks the following:

• a line like RESUME=/dev/mapper/hda2_crypt in /etc/initramfs-tools/conf.d/resume
• a resume device setting in /etc/uswsusp.conf (see uswsusp.conf(5))
• an entry in /etc/suspend.conf
• a resume=/dev/mapper/hda2_crypt in the kernel command line

You can inspect /usr/share/initramfs-tools/hooks/cryptroot if you want to know more about this.

Using UUIDs Even though the above is all you have to do, you might want to consider replacing the device paths in /etc/crypttab with persistent ones. One motivation might be the ability to boot off your drive via a USB adapter, which might cause it to appear as /dev/sda instead of /dev/hda. udev is installed by default on Debian systems and it makes persistent links to partitions available under /dev/disk/by-uuid, using the UUID of the content structure (e.g. the LUKS header). It uses /lib/udev/vol_id to determine those link names. If you replace the entries in /etc/crypttab, make sure to update the initramfs (update-initramfs -u -k all) and consider to use the same approach for the /boot filesystem in /etc/fstab; you ll note that all other filesystems use persistent device paths thanks to the dm-crypt layer. If you ever end up booting off the disk through a USB adapter, you might face the problem where the usb_storage subsystem takes too long to activate, so that the cryptsetup script does not find the devices in time. You can either solve this by adding the rootdelay=x parameter or break=mount to the kernel command line. The first will cause the scripts to wait x seconds before trying to configure and mount the root filesystem; the second would give you a shell that you can exit as soon as the kernel spouted its device initialisation messages at you. If you got this far, you might even want to take it further and replace hda5_crypt with cr_root or the like to abstract those silly partition numbers away even further. This is easier than it sounds, but does require several steps. Do not do this if you re not comfortable reviving your system in case it fails to come back up!

1. replace the old names with the new names in /etc/crypttab for all volumes except the root volume. If you are using a derived passphrase for the swap partition, make sure to put the new name for the root volume into the third column of the swap partition s configuration line.
2. modify /etc/fstab accordingly, leave the root filesystem s device path alone.
3. call update-initramfs -u
4. modify your bootloader to ask the kernel to boot off the new root volume.
5. replace the root volume s old name with the new one in /etc/fstab.
6. reboot, and add break=mount to the kernel command line.
7. at the busybox prompt, edit (vi) /conf/conf.d/cryptroot and change the first field of the root volume s line to the new name.
8. exit the shell and watch the boot process complete.
9. finally, replace the root volume s old name with the new one in /etc/crypttab.

If the system fails to boot up again, you can use the backup initial ramdisk, which update-initramfs left in /boot.

Security implications Apart from the usual security implications related to cryptosystems, passphrases, mathematics, user stupidity, and so on, the approach I outlined will leave you with a pretty well-secured system. Obviously, you should make sure to lock your screen whenever you leave the system unattended or the entire encryption is basically useless. There are two attack vectors on your system, both involving physical access to the machine:

• theft of the machine or its RAM chip, freezing the latter, and scanning the working memory for the passphrase
• manipulation of the kernel or initramfs stored on the (unencrypted) /boot filesystem, in such a way as to obtain the passphrase. To do this, you would have to have access to my machine and return it to me without me noticing; once I restart and enter the passphrase, you d have to steal it again

Other than that, you should be careful when travelling to totalitarian countries, like the Excited States of America, China, and probably the UK. First off, encryption arouses suspicion, and second, border agents might ask you to decrypt the partitions for them to copy or scan, and refusal to do so might get your turned away at the border. When travelling to those countries, make sure to hide your data properly.

Speed implications Obviously, having your entire system encrypted (including swap) will slow it down. I don t have any quantitative information on that, but after several years of using full-disk encryption on my laptop (an X40, which isn t very powerful), I can say that it remains usable, if you don t rely on disk-intensive operations, such as compiling kernels and the like.

Alternative approaches Several alternative approaches exist, all involving an additional device:

• if you don t like to type in the passphrase for the root filesystem, you could store it on a USB key (or the like); cryptsetup provides /lib/cryptsetup/scripts/passdev, which can be used to deal with such a situation. You can find more information in cryptsetup s README.Debian file.
• if you don t like the unencrypted /boot partition, you could boot the system off a USB key, which you can keep separate from the system except for when booting and upgrading the kernel. All you need to do for that is install the bootloader and kernel onto the device and configure it to use the proper encryption volume on the harddisk as root filesystem.
• cryptsetup also comes with support for PKCS#15 smartcards (opensc and openct).

I have chosen neither of these approaches, because the extra security does not make up for the inconvenience, and the danger of an unbootable system in case of loss of forgetting of the additional device.

Users of Debian derivatives sometimes report bugs that are not reported in the
Debian BTS, but that also affect Debian. It already happened a few times that
looking at the Ubuntu bugs for my packages allowed me to fix an unreported bug
in my Debian packages. But it’s difficult to keep track of the status of our packages in Ubuntu, since
Launchpad doesn’t provide a per-Debian-maintainer summary <insert here a rant
about closed-source software and how it restricts users>. Since it’s always fun to abuse proprietary software, I fetched all the bug data from Launchpad and inserted it in an SQLite DB (takes about 30 mins at 1200 HTTP requests/minute — it would be so much easier if the Launchpad devs added a text export of all bugs). The result is that there’s now an “Ubuntu” box on the Packages Tracking System, giving the current version in Ubuntu, a link to the Ubuntu patch (if any), and the number of open bugs. An Ubuntu column has also been added to the Debian Developer Packages Overview by Christoph Berg, with the current version in Ubuntu and the number of open bugs. It’s hidden by default: click on Display Configuration to enable it (then it’s stored in a cookie). I hope that this will help Debian maintainers to track what has been reported/fixed in Ubuntu. Also, if other Debian derivatives want to export the same kind of information, don’t hesitate to contact us. See for example:

• dpkg page on the PTS
• The Debian QA packages on DDPO

PS: the data might be slightly outdated, as it is processed on merkel.d.o, which was offline until recently. Expect it to be up-to-date in the next 24 hours.

postfwd 1.10pre7c-1 has been accepted and did hit experimental yesterday.
If you using policyd-weight you might want to give it a try, since its active development stoped. If you are annoyed by too much spam, it maybe a good idea as well.
Please have a look on postfwd.cf, there is no default config provided, since the configuration depends heavily on your environment. But maybe you have a look at /usr/share/doc/postfwd/examples, there is what you expect.
If you have any comments, send a mail to the maintainer or if there are bugs I didn’t spotted yet, please fill one!
In case I don’t respond quit fast, maybe I’m in the delivery room or anywhere else in the hospital or something like that, so please give me some extra time.
In case there are no big issues, I plan to upload the package to unstable in a reasonable time.

Colin Charles writes about a woman who is selling advertising space on herself [1]. Like Colin I haven’t bought a t-shirt in about 9 years (apart from some Cafepress ones I designed myself). So it seems that the price for getting some significant advertising at a computer conference is to buy a few hundred t-shirts (they cost $7 each when buying one at a time from Cafepress, I assume that the price gets lower than $3 each when buying truck-loads). I have been given boxer-shorts and socks with company logos on them (which I never wore), I think that very few people will show their underwear to enough people to make boxer-shorts a useful advertising mechanism, socks would probably work well in Japan though. It seems to me that many people regard accepting free t-shirts as being an exception to all the usual conventions regarding advertising. Accepting gifts from companies that you do business with is generally regarded as a bad idea, except of course when t-shirts and other apparel are given out then it’s OK. Being paid to wear a placard advertising a product is regarded as degrading by many people, but accepting a free t-shirt (effectively being paid $7 for wearing advertising) is regarded as OK by almost everyone. I don’t mind being a walking advert for a company such as Google. I use many Google products a lot and I can be described as a satisfied customer. There are some companies that have given me shirts which I only wear in winter under a jumper. The Oracle Unbreakable Linux [2] shirt is one that I wear in winter. Now I would not consider accepting an offer to have advertising on my butt (although I’m pretty sure that it doesn’t get enough attention that anyone would make such an offer). I would however be happy to talk with someone who wants to pay me to wear a t-shirt with advertising when giving a lecture at a conference. I am not aware of any conference which has any real dress requirement for speakers (apart from the basic idea of not offending the audience). The standard practice is that if your employer pays you to give a lecture as part of their marketing operation then they give you a shirt to wear (polo more often than t-shirt). I am currently working on some things which could end up as papers for presentation at Linux conferences. If someone wanted to sponsor my work on one of those free software related projects and then get the recognition of having me wear their shirt while giving a lecture and have me listed as being sponsored by that company in the conference proceedings then that seems like a reasonable deal for everyone. One thing that you need to keep in mind when accepting or soliciting for advertising is the effect it has on your reputation. Being known as someone who wants advertising on their butt probably wouldn’t be fun for very long. On the Internet advertising seems to be almost everywhere. It seems that more than half the content on the net (by the number of pages or by the number of hits) either has an option to donate (as Wikipedia does and some blogs are starting to do), has Google advertising (or a similar type of adverts from another company), is a sales site (IE you can buy online), or is a marketing site (IE provides background information and PR to make you want to buy at some other time). Note that my definition of advertising is quite broad, for example the NSA web site [3] has a lot of content that I regard as advertising/marketing - with the apparent aim of encouraging skilled people to apply for jobs. Not that I’m complaining, I’ve visited the National Cryptologic Museum [4] several times and learned many interesting things! I think that Internet advertising that doesn’t intrude on the content (IE no pop-ups, page diversions, or overly large adverts) is fine. If the advertising money either entirely pays people to produce useful content or simply encourages them to do so (as in the case of all the blogs which earn $10 a month) then I’m happy with that. I have previously written about some of my experience advertising on my blog [5] and how I encourage others to do the same. I don’t think that space on a t-shirt is any more or less appropriate for advertising than space on a web site hosting someone’s blog. Finally there is one thing I disagree with in Colin’s post, that is the use of the word “whore“. It’s not uncommon to hear the term “whoring” used as a slang term for doing unreasonable or unworthy things to make money (where “unreasonable” and “unworthy” often merely means doing something that the speaker wouldn’t be prepared to do). But using the term when talking about a woman is quite likely to cause offense and is quite unlikely to do any good. The Wikipedia page about prostitution [6] has some interesting background information.

• [1] http://www.bytebot.net/blog/archives/2008/06/13/on-whoring-oneself


• [2] http://etbe.coker.com.au/2008/02/18/oracle-unbreakable-linux/


• [3] http://www.nsa.gov/


• [4] http://www.nsa.gov/museum/index.cfm


• [5] http://etbe.coker.com.au/2008/04/14/friends-and-adverts/


• [6] http://en.wikipedia.org/wiki/Prostitution

Share This

I have been tracking the Debian release goals progress for a while now, through bug-squashing parties and using various iterations of wiki pages. It's time to write down some thoughts.

First of all, I think the concept of release goals is a fantastic idea. It may be one of the most important conceptual moves in Debian of late. It allows developers to implement distribution-wide changes without having to negotiate with every package maintainer individually and without having to go through the vicious no-change-without-policy/no-policy-without-established-practice maze. It moves the focus back on operating system development instead of package hoarding.

In the future, I think we should come up with some "sexier" release goal descriptions. If you look over the current list, many of these tasks are implementation-level, without any obvious benefit to users: double-build support? no unmet recommends? Or they have incomprehensible descriptions. The pretty much only release goal that has a real user benefit, namely the piuparts cleanness, ensuring clean distribution upgrades, is both the most cryptic and actually the most neglected one in terms of development attention. Don't get me wrong, all these goals are very valid. For better or worse, the press has started to pick up these release goals to put in their various Debian release candidate timeline pre-announcement articles, and next time around we should have something cool in there. If armel support or KDE 4 integration had been release goals, the marketing effect would be better.

The next thing is tracking bugs. If you have some time on your hands, how do you find a release goal bug that you can help with? I had worked up this wiki page to help myself, but it is still too weird. The fact that bts.turmzimmer.net exists shows that the BTS is lacking features or is too hard to use or both. I'm thinking, we should regularly copy the whole BTS into an SQL database and make queries from there. Think aggregation, grouping, full-text search! I'm not familiar with the BTS internals or the LDAP gateway, but I know my SQL. So if anyone knows the other end of this deal, I'd love to chat about setting something like this up.

So then, what happens to the release goals after the release? Has this been thought through?

Obviously, some release goals become technically self-enforcing. Python 2.5 and GCC 4.3 are now the default, and anything that doesn't work with these will have permanent trouble in the future.

Release goals that implement a distribution-wide feature, such as the LSB-based init system enhancements, will really need to become part of policy after the release. Otherwise they would revert to being normal bugs without any NMU powers and the other special attention given to release goals. So over time, the support for the former release goals across packages would again decline. To keep it up, you'd either have to repeat the release goal indefinitely, which would be silly, especially when there are no problems right now, or you'd have to make it a part of policy to enforce it in the future. Essentially, you'd have to consider most release goals trial policy changes, which become permanent if successful.

Near-examples of the above are the eternal release goals of IPv6 support and LFS support. We really have these under control quite well now -- although I still wouldn't dare to call Debian or any general-purpose Linux distribution fully IPv6-ready -- but nowadays these problems are almost all upstream code problems that Debian can't really fix very well on the packaging side. So under the current approach these release goals will exist forever, or worse, will be dropped from release goal status.

And then of course, some release goals fail to be reached. These should reapply for the next release, if interest persists. The dash goal comes to mind.

By the way, we had 13 release goals. 3 are completely done, 5 are pretty much done (less than 10 bugs), 1 more looks like it could get pretty much done before the release, and 4 will most likely fail to be accomplished.

The favorites mosaic shown in my previous post was a good excuse for me to try out Flickr's API, and it turned out to be a lot easier than I thought. So if, like me, you feel like playing with 2.5 billion photos (and 5,000 new uploads every minute), here's a quick guide.

All you need is an API key (get one here) and a set of bindings for your favorite language. I'm a Python person so I had two choices: Beej's Flickr API and flickr.py; the latter seemed to have everything I needed in a single file, so I went with it. It provides classes directly mapped to most of Flickr's concepts: photos, tags, sets, groups, users, etc.

To get started, let's see what Flickr knows about me (if you don't know your Flickr user id, get it from the RSS link in your photo page):

>>> me = flickr.User('99746300@N00')
>>> me.username, me.ispro,  me.location, me.photos_count
(u'orebokech', u'1', u'Lyon, France', u'254')
>>>

Sounds right. Now, to create my favorites mosaic I need to get the list of my favorites, actually I only need 36 of them. Easy:

>>> faves = me.getPublicFavorites(per_page=36)
>>> len(faves)
36
>>>

You can iterate over the list and get flickr.Photo objects which have all the information you need, like the available sizes of the image:

>>> photo = faves[0] # let's just get one for this example
>>> for size in photo.getSizes():
...     print "%s: %dx%d" % (size['label'], size['width'], size['height'])
Square: 75x75
Thumbnail: 100x67
Small: 240x160
Medium: 500x333
Original: 1024x683
>>>

And for each available size you can get the corresponding photo page, or the URL to the image itself:

>>> photo.getURL() # photo page, default Medium
u'http://www.flickr.com/photos/o0off/2363681897/sizes/m/'
>>> photo.getURL(size='Square', urlType='source') # img
u'http://farm3.static.flickr.com/2107/2363681897_f6528a14a4_s.jpg'
>>>

Add a little bit of HTML formatting around this, and you get a neat mosaic. The concept can trivially be extended to sets, tag searches, etc. It's all very straightforward.

Phew, this post is getting pretty long, so I think I'll leave you to explore the rest for yourself. Have fun!

One of the more useful features of the UPnP specification is that devices have a standard way of specifying a "presentation URL", a human-readable web page representing the device. For example, my SoundBridge has a web page which shows the currently playing music and lets me switch radio station, whilst my router's presentation URL is the administration page. Useful, but not exposed anywhere. Until now... This is a small Epiphany extension which adds all presentation URLs it finds to the Nearby Sites menu, just like the URLs discovered using Avahi. It needs a bit more work as it doesn't yet handle being unloaded or devices disappearing, but it is certainly usable now. If anyone else wants to have a go with it, the source can be fetched using Bazaar from here. Watch out for the currently hard-coded paths...

The GNU C library (packaged as libc6 in Debian) is the bedrock of a lot of code. What are the packages that it depends on? (Yo, DebGraph!) Original image [997 kB] libc6 is one of the well-connected vertices in the lower-left corner (third quadrant, if you like). For graphs like this one, we would like DebGraph to identify dependency cycles (loops). For example, libc6 depends on libgcc1, which in turn depends on libc6. This can be seen in the graph above by applying some careful scrutiny, but why not let the computer work its magic for us? (To be continued.) As a bonus, who depends on arping 2.05-2 for i386?

A common myth in the computer industry seems to be that ECC (Error Correcting Code - a Hamming Code [0]) RAM is only a server feature. The difference between a server and a desktop machine (in terms of utility) is that a server performs tasks for many people while a desktop machine only performs tasks for one person. Therefore when purchasing a desktop machine you can decide how much you are willing to spend for the safety and continuity of your work. For a server it’s more difficult as everyone has a different idea of how reliable a server should be in terms of uptime and in terms of data security. When running a server for a business there is the additional issue of customer confidence. If a server goes down occasionally customers start wondering what else might be wrong and considering whether they should trust their credit card details to the online ordering system. So it is obviously apparent that servers need a different degree of reliability - and it’s easy to justify spending the money. Desktop machines also need reliability, more so than most people expect. In a business when a desktop machine crashes it wastes employee time. If a crash wastes an hour (which is not unlikely given that previously saved work may need to be re-checked) then it can easily cost the business $100 (the value of the other work that the employee might have done). Two such crashes per week could cost the business as much as $8000. The price difference between a typical desktop machine and a low-end workstation (or deskside server) is considerably less than that (when I investigated the prices almost a year ago desktop machines with server features ranged in price from $800 to $2400 [1]). Some machines in a home environment need significant reliability. For example when students are completing high-school their assignments have a lot of time invested in them. Losing an assignment due to a computer problem shortly before it’s due in could impact their ability to get a place in the university course that they most desire! Then there is also data which is irreplaceable, one example I heard of was of a woman who’s computer had a factory pre-load of Windows, during a storm the machine rebooted and reinstalled itself to the factory defaults - wiping several years of baby photos… In both cases better backups would mostly solve the problem. For business use the common scenario is to have file servers storing all data and have very little data stored on the PC (ideally have no data on the PC). In this case a disk error would not lose any data (unless the swap space was corrupted and something important was paged out when the disk failed). For home use the backup requirements are quite small. If a student is working on an important assignment then they can back it up to removable media whenever they reach a milestone. Probably the best protection against disk errors destroying assignments would be a bulk purchase of USB flash storage sticks. Disk errors are usually easy to detect. Most errors are in the form of data which can not be read back, when that happens the OS will give an error message to the user explaining what happened. Then if you have good backups you revert to them and hope that you didn’t lose too much work in the mean-time (you also hope that your backups are actually readable - but that’s another issue). The less common errors are lost-writes - where the OS writes data to disk but the disk doesn’t store it. This is a little more difficult to discover as the drive will return bad data (maybe an old version of the file data or maybe data from a different file) and claim it to be good. The general idea nowadays is that a filesystem should check the consistency of the data it returns. Two new filesystems, ZFS from Sun [2] and BTRFS from Oracle [3] implement checksums of data stored on disk. ZFS is apparently production ready while BTRFS is apparently not nearly ready. I expect that from now on whenever anyone designs a filesystem for anything but the smallest machines (EG PDAs and phones) they will include data integrity mechanisms in the design. I believe that one such features become commonly used the need for RAID on low-end systems will dramatically decrease. A combination of good backups and knowing when your live data is corrupted will often be a good substitute for preserving the integrity of the live data. Not that RAID will necessarily protect your data - with most RAID configurations if a hard disk returns bad data and claims it to be good (the case of lost writes) then the system will not read data from other disks for checksum validation and the bad data will be accepted. It’s easy to compute checksums of important files and verify them later. One simple way of doing so is to compress the files, every file compression program that I’ve seen has some degree of error detection. Now the real problem with RAM which lacks ECC is that it can lose data without the user knowing. There is no possibility of software checks because any software which checks for data integrity could itself be mislead by memory errors. I once had a machine which experienced filesystem corruption on occasion, eventually I discovered that it had a memory error (memtest86+ reported a problem). I will never know whether some data was corrupted on disk because of this. Sifting through a large amount of stored data for some files which may have been corrupted due to memory errors is almost impossible. Especially when there was a period of weeks of unreliable operation of the machine in question. Checking the integrity of file data by using the verify option of a file compression utility, fsck on a filesystem that stores checksums on data, or any of the other methods is not difficult. I have a lot of important data on machines that don’t have ECC. One reason is that machines which have ECC cost more and have other trade-offs (more expensive parts, more noise, more electricity use, and the small supply makes it difficult to get good deals). Another is that there appear to be no laptops which support ECC (I use a laptop for most of my work). On the other hand RAID is very cheap and simple to implement, just buy a second hard disk and install software RAID - I think that all modern OSs support RAID as a standard installation option. So in spite of the fact that RAID does less good than a combination of ECC RAM and good backups (which are necessary even if you have RAID), it’s going to remain more popular in high-end desktop systems for a long time. The next development that seems interesting is the large portion of the PC market which is designed not to have the space for more than one hard disk. Such compact machines (known as Small Form Factor or SFF) could easily be designed to support ECC RAM. Hopefully the PC companies will add reliability features in one area while removing them in another.

• [0] http://en.wikipedia.org/wiki/Hamming_code


• [1] http://etbe.coker.com.au/2007/08/01/ecc-ram-in-a-cheap-machine/


• [2] http://en.wikipedia.org/wiki/ZFS


• [3] http://en.wikipedia.org/wiki/Btrfs

Share This

It seems that many blog posts related to fuel use (such as my post from yesterday about record oil prices [1]) are getting adverts about perpetual motion [2]. Note that the common usage of the term “Perpetual Motion” does not actually require something to move. A battery that gives out electricity forever would be regarded as fitting the description, as does any power source which doesn’t have an external source of energy. The most common examples of this are claims about Oxyhydrogen [3], this is a mixture of hydrogen and oxygen in a 2:1 ratio. The wikipedia page is interesting, apparently oxyhydrogen is used for welding metals, glass, and plastics, and it was also used to heat lime to provide theatrical lighting (”lime light”). So a mixture of hydrogen and oxygen does have real-world uses. The fraud comes in the issue of the claims about magnecules [4]. Magnecules are supposedly the reason for the “atomic” power of HHO gas (AKA Oxyhydrogen) which are repeated on many web sites. In brief, one mad so-called scientist (of course if he was a real scientist he would have experimental evidence to support his claims and such experiments would be repeatable) has invented entirely new areas of science, one of which involves magnetic bonds between atoms. He claims that such chemical species can be used to obtain free energy. The idea is that you start with water, end with water plus energy - then reuse the water in a closed system. Strangely the web sites promoting water fueled cars don’t seem to mention magnecules and just leave the “atomic energy” claim with no support - maybe magnecules are simply too crazy for them. The water fuelled car wikpedia page is interesting - it lists five different ways that water can actually be used in a car engine (which are based on sound scientific principles and which have been tested) and compares them to the various water fueled car frauds [5]. I’m not accepting any more comments on my blog about perpetual motion solutions to the petrol crisis (they just take up valuable space and distract people who want to discuss science). I’ll allow some comments about such things on this post though.

• [1] http://etbe.coker.com.au/2008/06/09/record-oil-prices/


• [2] http://en.wikipedia.org/wiki/Perpetual_motion


• [3] http://en.wikipedia.org/wiki/Oxyhydrogen


• [4] http://en.wikipedia.org/wiki/Magnecule


• [5] http://en.wikipedia.org/wiki/Water-fuelled_car

Share This

I maintain a number of web shops for our webmaster cooperative and one of our main challenges is to encourage people who put things into their basket/cart to actually buy them. How to avoid shopping cart abandonment by Graham Jones makes some points that I've identified as possible reasons for people not buying in the past: comparisons, robots and not trusting the site enough to give payment details. There's not much we can do about robots or people comparison-shopping at a technical level, but we try to build some trust by publishing the shop owner's geographic address and telephone number (which I think is required by law in England for most web shops now), making sure the SSL certificate and domain registration details are correct, using reputable payment providers and being clear about delivery charges and terms. The point about the slickness of the checkout process is a good one and one that we've only recently started to work on. We've had pretty good results from making the checkout slicker on one site. It looks like two-thirds of people who click the checkout button now continue to buy, putting it comfortably ahead of current UK averages but I need to tweak our stats calculator to make the report directly comparable. Nevertheless, I think those improvements will be added to our other shops as soon as possible. I share Graham's low opinion of the oft-quoted Amazon. We've also been looking at other web shop software besides OSCommerce for a new project, so now would be a good time to change to something new if it improves the checkout a lot. We've made OSCommerce's checkout a lot smoother, but it's still essentially OSC. Is there a good checkout which you'd want to use as an example? The other challenge is getting visitors onto the site in the first place. How To Build Links By Patrick Altoft explains the basics as well as I've seen recently.

We eventually released liquidsoap 0.3.7 !!
For those who don't know about this software, liquidsoap is a scripting language designed for building audio streams. It was initially aimed at web radios, as an alternative shoucast/icecast source client, but quickly turned to a much more general tool.
For those who read debaday, I already blogged about it few months before. There, you'll find more explanations on how the program operates.
A simple example could be: #!/usr/bin/liquidsoap # (...)

Aaron Swartz writes about what government might do to encourage entrepreneurship: http://www.aaronsw.com/weblog/prostartup As an American who has lived in Europe for a while, I've seen both sides of the coin (or pond, as it were), and feel qualified to add a few things to the conversation, even if many of these points merit a discussion of their own.

• He's right about health care. It's one of those things with a huge downside risk that people in Europe simply don't have to think about. You can change jobs, take a break from a job, start a company...whatever, and you never have to think about "what if". Even if you have the sort of preexisting condition that can hang like a stone around your neck in the US. How to implement such a system (which of course has potential disadvantages as well) is well beyond the scope of this journal.
• Universal education. I don't think it's as great as Aaron seems to think, but I don't think it's very influential one way or the other on startups. People get good educations in both places, although, by and large, the best Universities are in the US because they have more money, and have in the past attracted highly qualified people from all over the world. In both places, students manage to meet other like minded people with whom they can start companies. There are definitely improvements to be made to the US system, but that's really just a political discussion: like I said, differences in universities are not the limiting factor.
• Minimum income without having to work or even having to look for work sounds like an enormous can of worms. Might be good for startups, but has big risks associated with it, and wouldn't be politically feasible in the US. Even in most European countries with that sort of thing, you don't just get money: you have to look for work, do job training, or prove that you're just not capable of doing anything productive.

And here are some thoughts of my own: I think it's best to focus on the startup "curve", and costs associated with various points on it: when people are just getting started, keep state-imposed costs low. If someone makes a lot of money, worry about taxing it then. This is quite different than what has previously been the norm in many places in Europe, which have policies akin to slaughtering the goose before it lays any eggs, let alone golden ones. Italy is a particularly bad example, but it's the place I know best, and not too far off what happens elsewhere in terms of policy: fixed, sunk, startup costs are in the order of 10,000 Euro, which for most people is a lot of money. This creates a bias towards people who already have a lot of money, and away from random people who happen to have good ideas and a strong desire to execute on them. This is something that the US more or less gets right: it cost me $55 in state fees to start DedaSys LLC as a limited liability company (even throwing in money for lawyers and the like, you're still far under the thousands it costs in many European countries). Incidentally, it looks like Germany may be on the right track, if you look at the "Mini-GmbH" mentioned in the article. It's progress, if nothing else. Beyond that, there is a lot more red tape in Europe, and that, I think, rather than a bit higher taxes for companies that are profitable, is the thing that, outside of culture, probably weighs most heavily on European startups. Culture, however, is also a very key factor, and not something that will change soon or easily. In Italy, there's a lot of pressure to land a permanent job, from which you can basically never be fired. The idea of striking out on your own is probably simply not on most people's radar screens. I think culture is what means that Europe produces tons of great open source software, for example, but relatively fewer startups (if you're just counting open source startups though, Europe does ok). Perhaps this is slowly changing, but it will take a while, and without some policy changes, the incentives won't even materialize. Returning to things that government can do, something critical that many people don't get is that government funded startups are usually big failures (see the French "Google competitor") and a big waste of money that would probably simply be better to spend in lowering taxes, improving education, or other "level playing field" initiatives, rather than having government employees attempting to "pick winners". This is a common problem in Europe, but also in many areas of the US that want to replicate Silicon Valley's success. Going back to the US, which was Aaron's original point of focus, beyond health care, what could the US improve? At the moment, immigration seems to be the big problem: people from all over the world want to go to the US to work and start companies, but look at what kinds of hoops someone (who is coming to spend money and create jobs!) has to jump through to do so: http://www.techcrunch.com/2008/05/02/coming-to-america-getting-visas-to-do-business-in-silicon-valley/ Embarrassing and shameful. In conclusion, I think the common theme is that the government can't really do much to actively promote startups, but can certainly hinder them by setting too high a bar in terms of bureaucracy and fixed costs. Lowering those fixed costs for new companies, be it through at least basic health care for everyone, or lower taxes and less paperwork should be what governments aim for. Given the right conditions and incentives, bright, hard workers will do the rest.

Dear Bastian let's fix your blog post.

Philipp thinks, the fact that rng is not using the information in /usr/share/bug renders rng unfit for release and upgraded the corresponding bugreport from wishlist to serious. Moreover: since I dared to downgrade the report back to wishlist he decided to remove rng from testing and block it until the bug is fixed.

s/Philipp/the Release Team/g. And yes, we believe that reportbug-ng goes against the Maintainers wishes, and doesn't help users that don't know how to report a bug (and which are rng audience unless I'm mistaken).

I don t want to heat the debate about this bug again, but Philipp s decision seems arbitrary for me and I wonder if the same standard is applied to every other Debian package.

It's not, it was discussed in the Release Team and was not Philipp on his own alone. And yes, the same standards of quality is wanted for every package in Debian.

I mean, rng has no release critical defects.

That's you claiming that, there have been really constructive remarks in both bug that we upgraded, and that I think explain wisely why the current reportbug-ng behaviour doesn't makes it a suitable reportbug tool right now. I think for example that Sam's and Michael Biebl contributions in both bugs are good explanations of what is wrong and why.

It just does not use the aforementioned scripts as additional information in bugreports does this really render the software unfit for release ?

Yes it does. Again, people that use rng are I think, less experienced users, who need guidance through the bug reporting process. The kind of users that often send annoying bug reports. Instead of lowering the annoyance factor, you aggravate it. In the end, rng makes the frustration higher, and totally fails to meet its primary goals: making bug reporting nicer, simpler, and more efficient for the Reporter and the Maintainer. While it's probably nicer for the reporter, you totally fail on the other part of the contract. And if you don't care about the Maintainer, then well, reportbug-ng is just a spam machine to them, and your careless attitude toward them is unfit for release for sure.

When I write text documents (using LaTeX or Docbook), I like to wrap lines, as it makes them easier to edit (less things moving on the screen), and allow to have easy-to-read diffs. However, I always hesitate before rewrapping paragraphs (using vim’s gqap): this mean that I will add noise to my git history. So I only do that from time to time, making “rewrapping-only commits”. But that sucks, since in the meantime, I sometimes make a lot of changes, and my lines grow long again. Of course, I could rewrap my paragraphs before each commit, but if I simply add a word to a paragraph, it might cause all the lines to be rewrapped. So I what I would need is some kind of “text normalizer” that will:

• split lines at The Right Place. After ‘.’, ‘,’, ‘:’, ‘;’, etc. So rewrapping won’t propagate changes too far away.
• understand the basics of LaTeX, so it won’t rewrap

  \begin tabular l l \hline
  x & y \\
  1 & 2 \\\hline
  \end tabular 

  or

  \begin figure 
  \centerline \includegraphics fig 
  \caption Cool stuff 
  \label coolstuff 
  \end figure 

  (vim does rewrap those examples.)
• be editor-agnostic. So other committers could use it as well.
• support for other document formats (docbook XML) would be nice too.

I’ve looked at plasTeX: I could use it to parse a LaTeX document, and export it as LaTeX. But then it would be a LaTeX-only solution. Does anyone have a better solution?

Philipp thinks, the fact that rng is not using the information in /usr/share/bug renders rng “unfit for release” and upgraded the corresponding bugreport from wishlist to serious. Moreover: since I dared to downgrade the report back to wishlist he decided to remove rng from testing and block it until the bug is fixed. I don’t want to heat the debate about this bug again, but Philipp’s decision seems arbitrary for me and I wonder if the same standard is applied to every other Debian package. I mean, rng has no release critical defects. It just does not use the aforementioned scripts as additional information in bugreports — does this really render the software “unfit for release”? For example: In Etch we shipped a famous email client with a known bug which lead reproducibly to loss of emails on IMAP accounts, just because removing the program, or one of it’s components was not possible in the remaining time.

The most dynamic segment of the software industry that I'm aware of is web programming. Flamewars rage about frameworks and browsers and standards and whatnot, and the openness of the whole thing feels like some of the things that made me fall in love with computing as a whole and Free Software in particular. The problem with all that openness is that it's rather difficult to navigate. Aside from the obvious issues with browser incompatibilities, there's an enormous amount of software solutions on the server side, spanning languages, libraries, and frameworks. Each one is an ecosystem to itself, often being large and complicated.

This makes it rather difficult to do certain aspects of web programming well. You are constantly switching between tools and languages, as well as coding paradigms, in order to build a web app in full. You'll be writing HTML, CSS, and Javascript on the client side, each with its own peculiarities. On the server side you'll probably have settled on one language with its own very different varieties of coding style and patterns. Some, like Ruby on Rails, are their own unique brand of language that makes you work both in the DSL (Rails in this case) and the parent language (Ruby here). Template engines often will do the same thing, such as in the case of Django.

It's pretty obvious that having to juggle all these tools places a fairly large burden on the coder. Much to their credit, it's not difficult to work in any of these languages, but to work in them at a high level becomes more difficult the deeper and wider the software stack reaches. You get this a bit in systems programming, juggling C/C++ with a build system and a scripting language, but that's usually it. You can get that number of moving parts just on the client side of the web with HTML, CSS, and Javascript.

Something interesting that seems to be happening is that people are moving to decrease this cognitive burden. One good example is the Google Web Toolkit which lets you write all your AJAXy client-side stuff in Java, and let it get compiled in to Javascript. That way you cut the languages you have to work in. A similar concept is behind Microsoft allowing Ruby (and Python, if I've heard correctly) in Silverlight. Alternately, you can bring Javascript (back) to the server side, which is the motivation behind Helma and Rhino on Rails (the latter of which I hope sees the light of day).

Personally, I'm rooting for Javascript to not become an assembly language, but to take over the server side again. It's a capable and powerful language (much better than Java in my opinion), and we're collectively leveraging it very poorly. Rhino is interesting and has enormous potential, but it needs library binds in the worst way. Currently it's basically using Javascript to write Java, which is rather atrocious (COBOL in any language, etc) but with some Javascript-style wrappers around the Java classes it could be phenomenal. Alternately, there's Spidermonkey, which could be wrapped in a more capable shell with a good FFI, and we could easily have wrappers to our favorite Free Software libraries. This project seems far along here, but it's currently very windows-centric.

There's a lot of potential here though. Unifying the development language for both web and system apps would benefit the Free Software desktop, and give us the ability to better integrate our stuff with the web. One of the best things about Free Software is that it dropped the barrier to entry in software development. I think we can repeat that success again here.