Traditionally, Unix has operated by piping between commands ("find grep sort uniq"). I wonder if we are experiencing a shift to command composition being a more common model? I started thinking about this when I considered the following command line:

chronic mr run git annex get

Unpacking, that uses chronic, which arranges for quiet running of mr (unless it fails, then it shows the full output), which in turn runs git in every repository it knows about, and git is asked to run the git-annex subcommand, which in turn has a "get" subcommand of its own. So, in other words, this is a cron job that syncs large files in a set of git repos, quietly. You might want to make it run niced also, which adds 3 more commands:

chronic nice ionice -c3 trickle -d100 mr run git annex get

This is the most extreme example I know of (unless you tack in a "sudo"), but there does seem to have been a gradual shift to using more composition, and more subcommands. In git in particular, two-levels of subcommands is not unknown, for example "git remote add". Subcommands, especially as implemented in git (or earlier, in arch), are kind of a special case of command composition that adds a namespace. I'm sure that a count of commands with subcommands would show it exploded in the past 10-12 years. I'll bet a count of composable commands would also show a significant increase. Anyone have other examples of long compositions of commands?

Joey has proposed a new concept of Continuously Usable Testing for Debian [1], basically testing should be usable at all times and packages that aren t usable should be dropped. But to properly achieve this goal we need continual testing of usability.The Plan For SE LinuxTo do this for SE Linux I m setting up a Xen server which will have a number of different DomUs for testing a variety of server applications. The system has 1.5G of RAM and 160G of mirrored storage. An image of a typical server will take about 4G of disk space, so we could have something like 40 images online and ready for testing. I have already setup Squid on another system on the same LAN to cache Debian packages, so running apt-get dist-upgrade on a number of DomUs won t take that long. With 256M for a typical server image I could have 5 images running at the same time. If the hardware isn t enough then I can expand it, I hope to get some donations of DDR-266 or DDR-333 RAM (or maybe DDR-400) to upgrade the system to 4G of RAM, I can add more hard drives, and I could even install more servers.I want to have testing be very usable for SE Linux throughout the development cycle so that I don t have to rush things before release.At this stage I m not sure whether to track Unstable or Testing for this. I guess it might be best to track Testing most of the time and only track Unstable for daemons that are changing rapidly. It might get boring testing every version that comes through Unstable, but if people want to do this then I won t stop them.Setting up the TestsWhat I need are interested people who want to install server configurations for testing. If you have some favorite combination of daemons that you want tested for SE Linux support (even if it s daemons that have no current policy) then I can give you root access to a DomU to develop test cases. Ideally there would be automated tests used for most things for example testing a mail server by using swaks to deliver mail and a POP or IMAP client script to retrieve it. But some things can t be tested properly without human intervention.For the automated tests I want to script the creation of DomUs, upgrading the packages in the DomU, testing it, and then shutting down the DomU if it all works. If at any time the tests fail (or the upgrade fails) then it would wait for human intervention. That would be me fixing SE Linux problems and other people fixing the application problems. I think that discovering SE Linux issues will only be a part of this project.For the manual tests I will grant access to create and destroy the DomUs in question to people who can run the tests.I m thinking of having a couple of DomUs running permanently for things which are test candidates but also useful for the project, such as a MediaWiki instance. It really depends on the interest of people who might use such things.Also I m thinking of setting up some Ubuntu DomUs too, I probably should join Ubuntu and get involved with SE Linux there.Sharing the ImagesI have a web server in Germany with almost unlimited bandwidth and storage. For every image that is created I want to upload a version to the server in Germany to allow anyone in the world to test it. There are lots of possible ways of using this for software development. For example if you had a patched version of Apache that you wanted to test then you could download every image that had Apache installed and test that they all work. That would be easier than configuring Apache in different ways and also possibly provide better coverage.Also if someone can t figure out how to configure a daemon correctly then downloading a Xen image of a working configuration could be helpful (if a little bandwidth intensive). Note that deploying such an image in production would be a really bad idea, among other things there are lots of places where passwords are stored and you wouldn t want to risk missing one.I also plan to share the scripts used in running the Dom0 and anything else that seems useful along the way.What We NeedThe main thing we need is volunteers to configure virtual machines with their favorite daemons. Note that I don t plan to have only one daemon per DomU, if we can get multiple daemons running that don t conflict (EG file server and mail server) or multiple daemons that can interact (EG database server and a mail server or anything else that can be a database client) running on the same system then that s a good thing. So there will be some degree of interaction with other people.I m happy to accept contributions from people who aren t interested in SE Linux. But SE Linux will run on all DomUs.Finally I also need more RAM for a HP D530S, DU875PA (that s a Celeron 2.4GHz). I ll accept donations of complete systems too once my HP system gets full, preferably relatively low power systems as they will be housed in a location that s not as well ventilated as I would like (cost and availability of IP addresses were the main criteria). A laptop with a broken screen would be great!The system won t go live until Monday, but I think that probably people won t be ready to do much work with less than two days notice anyway.

• [1] http://kitenet.net/~joey/code/debian/cut/

Match package names across distributions What would happen if we had a quick and reliable way to match package names across distributions? These ideas came up at the appinstaller2011 meeting:

• it would be easy to lookup screenshots in the local distro, and if there are none then fall back on other distributions;
• it would be easy to port Debtags to other distributions, and possibly get changes back;
• it would be trivial to add a [patches in $DISTRO] link to the PTS
• it would be easy to point to other BTSes

We thought they were good ideas, so we started hacking. To try it, you need to get the code and build the index first:

git clone git://git.debian.org/users/enrico/distromatch.git
cd distromatch
# Careful: 90Mb
wget http://people.debian.org/~enrico/dist-info.tar.gz
tar zxf dist-info.tar.gz
# Takes a long time to do the indexing
./distromatch --reindex --verbose

Then you can query it this way:

./distromatch $DISTRO $PKGNAME [$PKGNAME1 ...]

This would give you, for the package $PKGNAME in $DISTRO, the corresponding package names in all other distros for which we have data. If you do not provide package names, it automatically shows output for all packages in $DISTRO. For example:

$ time ./distromatch debian libdigest-sha1-perl
debian:libdigest-sha1-perl fedora:perl-Digest-SHA1
debian:libdigest-sha1-perl mandriva:perl-Digest-SHA1
debian:libdigest-sha1-perl suse:perl-Digest-SHA1
real    0m0.073s
user    0m0.056s
sys 0m0.016s

Yes it's quick. It builds a Xapian index with the information it needs, and then it reuses it. As soon as I find a moment, I intend to deploy an instance of it on DDE. It is using a range of different heuristics:

• match packages by name;
• match packages by desktop files contained within;
• match packages by pkg-config metadata files contained within;
• match packages by [/usr]/bin/* files contained within;
• match packages by shared library files contained within;
• match packages by devel library files contained within;
• match packages by man pages contained within;
• match stemmed form of development library package names;
• match stemmed form of shared library package names;
• match stemmed form of perl library package names;
• match stemmed form of python library package names.

This list may get obsolete soon as more heuristics get implemented. Euristics will never cover all corner cases we surely have, but the idea is that if we can match a sizable amout of packages, the rest can be somehow fixed by hand as needed. The data it requires for a distribution should be rather straightforward to generate:

1. a file which maps binary package names to source package names
2. a file with the list of files in all the packages

For example:

$ ls -l dist-debian/
total 39688
-rw-r--r--  1 enrico enrico  1688249 Jan 20 17:37 binsrc
drwxr-xr-x  2 enrico enrico     4096 Jan 21 19:12 db
-rw-r--r--  1 enrico enrico 29960406 Jan 21 10:02 files.gz
-rw-r--r--  1 enrico enrico  8914771 Jan 21 18:39 interesting-files
$ head dist-debian/binsrc 
openoffice.org-dev openoffice.org
ext4-modules-2.6.32-5-4kc-malta-di linux-kernel-di-mipsel-2.6
linux-headers-2.6.30-2-common linux-2.6
libnspr4 nspr
ipfm ipfm
libforks-perl libforks-perl
med-physics debian-med
libntfs-3g-dev ntfs-3g
libguppi16 guppi
selinux selinux
$ zcat dist-debian/files.gz   head
memstat etc/memstat.conf
memstat usr/bin/memstat
memstat usr/share/doc/memstat/changelog.gz
memstat usr/share/doc/memstat/copyright
memstat usr/share/doc/memstat/memstat-tutorial.txt.gz
memstat usr/share/man/man1/memstat.1.gz
libdirectfb-dev usr/bin/directfb-config
libdirectfb-dev usr/bin/directfb-csource
libdirectfb-dev usr/include/directfb-internal/core/clipboard.h
libdirectfb-dev usr/include/directfb-internal/core/colorhash.h

interesting-files and db are generated when indexing. To prove the usefulness of the idea (but does it need proving?), you can find in the same git repo a little example app (it took me 10 minutes to write it), that uses the distromatch engine to export Debtags tags to other distributions:

$ ./exportdebtags fedora   head
memstat: admin::benchmarking, interface::commandline, role::program, use::monitor
libdirectfb-dev: devel::lang:c, devel::library, implemented-in::c, interface::framebuffer, role::devel-lib
libkonqsidebarplugin4a: implemented-in::c++, role::shared-lib, suite::kde, uitoolkit::qt
libemail-simple-perl: devel::lang:perl, devel::library, implemented-in::perl, role::devel-lib, role::shared-lib, works-with::mail
libpoe-component-pluggable-perl: devel::lang:perl, devel::library, implemented-in::perl, role::shared-lib
manpages-ja: culture::japanese, made-of::man, role::documentation
libhippocanvas-dev: devel::library, qa::low-popcon, role::devel-lib
libexpat-ocaml-dev: devel::lang:ocaml, devel::library, implemented-in::c, implemented-in::ocaml, role::devel-lib, works-with-format::xml
libgnutls-dev: devel::library, role::devel-lib, suite::gnu

Just in case this made you itch to play with Debtags in a non-Debian distribution, I've generated the full datasets for Fedora, Mandriva and OpenSUSE. Others have been working on the same matching problem. After we started writing code we started to become aware of existing work:

• whohas
• PackageMap
• Equivalent-Packages, statistically generated from package contents, more info in this post

I'd like to make use of those efforts, maybe to cross-validate results, maybe even better as yet another heuristics.

I had some hope when I read Girish Ramakrishnan's blog post that starts with To my knowledge, there are 3 Qt based JSON parsers out there . But I was really disappointed: instead of trying to get a consolidation going or at least highlighting the need for three different parsers, he announces yet another implementation. And, to make matters worse, it is intended to be statically linked whenever it is used. A friendly wave to all security conscious engineers who will now have to hunt down and kill security issues in various places wherever this json parser was used, in various different versions, possibly with local modifications. Girish, please do not take this as a personal attack, but what you're doing is just bad engineering practice. I don't claim qjsonparser is buggy. I haven't even looked at the code. But let's face it: bugs happen, and json is often passed over the net, so any parser is attack surface. So it should be as easy as possible to get fixed versions of the code out to the users. The means to do this is by allowing distribution builders to be aware of where the code in question was used, and to get fixed versions of it distributed easily. In other words: such code should always be in a shared library. Take, for example, the history of xpdf/poppler: many people spent countless hours chasing copies of xpdf code in many applications before they finally had enough, forked xpdf (if I have the history correctly) and created the poppler library which is now widely used. Now security issues with the PDF parser require one security fix, not 10.

Yesterday, I held a small Haskell demonstration talk at the FOSSEE group, an advocacy group for Free Software in engineering education located at the IIT Bombay in Mumbai, India, where am I currently staying as a visiting student. We had to move out of the proper lecture hall into a small room with no projector, due to an exam taken in the other room, but as the audience was small enough (seven people), using an LCD monitor was sufficient. My talk explained some the features of Haskell, namely functional, pure, lazy, strictly typed, interpreted or compiled and its library ecosystem (often forgotten in academical treatments of Haskell), and then demonstrated these features by some live coding action where I used the nice gloss vector library to render drawing commands given in a comma-separated value files, which is also parsed by a library from hackage. I have put the slides, the example csv file and the code online.

Yesterday, I wrote about backing up to the cloud. I specifically was looking at cloud backup services. I ve been looking into various options there, but also various options for disk-based backups. I d like to have both onsite and offsite backups, so both types of backup are needed. Also, it is useful to think about how the two types of backups can be combined with minimal overhead. For the onsite backups, I d want to see:

1. Preservation of ownership, permissions, etc.
2. Preservation of symlinks and hardlinks
3. Space-efficient representation of changes ideally binary deltas or block-level deduplication
4. Ease of restoring
5. Support for backing up Linux and Windows machines

Deduplicating Filesystems for Local Storage Although I initially thought of block-level deduplicating file systems as something to use for offsite backups, they could also make an excellent choice for onsite disk-based backups. rsync-based dedup backups One way to use them would be to simply rsync data to them each night. Since copies are essentially free, we could do (or use some optimized version of) cp -r current snapshot/2011-01-20 or some such to save off historic backups. Moreover, we d get dedup both across and within machines. And, many of these can use filesystem-level compression. The real upshot of this is that the entire history of the backups can be browsed as a mounted filesystem. It would be fast and easy to find files, especially when users call about that file that they deleted at some point in the past but they don t remember when, exactly what it was called, or exactly where it was stored. We can do a lot more with find and grep to locate these things than we could do with tools in Bacula (or any other backup program) restore console. Since it is a real mounted filesystem, we could also do fun things like make tarballs of it at will, zip parts up, scp them back to the file server, whatever. We could potentially even give users direct access to their files to restore things they need for themselves. The downside of this approach is that rsync can t store all the permissions unless it s running as root on the system. Wrappers such as rdup around rsync could help with that. Another downside is that there isn t a central scheduling/statistics service. We wouldn t want the backup system to be hammered by 20 servers trying to send it data at once. So there d be an element of rolling our own scripts, though not too bad. I d have preferred not to authorize a backup server with root-level access to dozens of machines, but may be inescapable in this instance. Bacula and dedup The other alternative I thought of system such as Bacula with disk-based volumes . A Bacula volume is normally a tape, but Bacula can just write them to disk files. This lets us use the powerful Bacula scheduling engine, logging service, pre-backup and post-backup jobs, etc. Normally this would be an egregious waste of disk space. Bacula, like most tape-heritage programs, will write out an entire new copy of a file if even one byte changes. I had thought that I could let block-level dedupe reduce the storage size of Bacula volumes, but after looking at the Bacula block format spec, this won t be possible as each block will have timestamps and such in it. The good things about this setup revolve around using the central Bacula director. We need only install bacula-fd on each server to be backed up, and it has a fairly limited set of things it can do. Bacula already has built-in support for defining simple or complicated retention policies. Its director will email us if there is a problem with anything. And its logs and catalog are already extensive and enable us to easily find out things such as how long backups take, how much space they consume, etc. And it backs up Windows machines intelligently and comprehensively in addition to POSIX ones. The downsides are, of course, that we don t have all the features we d get from having the entire history on the filesystem all at once, and far less efficient use of space. Not only that, but recovering from a disaster would require a more extensive bootstrapping process. A hybrid option may be possible: automatically unpacking bacula backups after they ve run onto the local filesystem. Dedupe should ensure this doesn t take additional space if the Bacula blocksize aligns with the filesystem blocksize. This is certainly not a given however. It may also make sense to use Bacula for Windows and rsync/rdup for Linux systems. This seems, however, rather wasteful and useless. Evaluation of deduplicating filesystems I set up and tested three deduplicating filesystems available for Linux: S3QL, SDFS, and zfs-fuse. I did not examine lessfs. I ran a similar set of tests for each:

1. Copy /usr/bin into the fs with tar -cpf - /usr/bin tar -xvpf - -C /mnt/testfs
2. Run commands to sync/flush the disk cache. Evaluate time and disk used at this point.
3. Rerun the tar command, putting the contents into a slightly different path in the test filesystem. This should consume very little additional space since the files will have already been there. This will validate that dedupe works as expected, and provide a hint about its efficiency.
4. Make a tarball of both directories from the dedup filesystem, writing it to /dev/zero (to test read performance)

I did not attempt to flush read caches during this, but I did flush write caches. The test system has 8GB RAM, 5GB of which was free or in use by a cache. The CPU is a Core2 6420 at 2.13GHz. The filesystems which created files atop an existing filesystem had ext4 mounted noatime beneath them. ZFS was mounted on an LVM LV. I also benchmarked native performance on ext4 as a baseline. The data set consists of 3232 files and 516MB. It contains hardlinks and symlinks. Here are my results. Please note the comments below as SDFS could not accurately complete the test.

Test	ext4	S3QL	SDFS	zfs-fuse
First copy	1.59s	6m20s	2m2s	0m25s
Sync/Flush	8.0s	1m1s	0s	0s
Second copy+sync	N/A	0m48s	1m48s	0m24s
Disk usage after 1st copy	516MB	156MB	791MB	201MB
Disk usage after 2nd copy	N/A	157MB	823MB	208MB
Make tarball	0.2s	1m1s	2m22s	0m54s
Max RAM usage	N/A	150MB	350MB	153MB
Compression	none	lzma	none	gzip-2

It should be mentioned that these tests pretty much ruled out SDFS. SDFS doesn t appear to support local compression, and it severely bloated the data store, which was much larger than the original data. Moreover, it permitted any user to create and modify files, even if the permissions bits said that the user couldn t. tar gave many errors unpacking symlinks onto the SDFS filesystem, and du -s on the result threw up errors as well. Besides that, I noted that find found 10 fewer files than in my source data. Between the huge memory consumption, the data integrity concerns, and inefficient disk storage, SDFS is out of the running for this project. S3QL is optimized for storage to S3, though it can also store its files locally or on an sftp server a nice touch. I suspect part of its performance problem stems from being designed for network backends, and using slow compression algorithms. S3QL worked fine, however, and produced no problems. Creating a checkpoint using s3qlcp (faster than cp since it doesn t have to read the data from the store) took 16s. zfs-fuse appears to be the most-used ZFS implementation on Linux at the moment. I set up a 2GB ZFS pool for this test, and set dedupe=on and compress=gzip-2. When I evaluated compression in the past, I hadn t looked at lzjb. I found a blog post comparing lzjb to the gzip options supported by zfs and wound up using gzip-2 for this test. ZFS really shone here. Compared to S3QL, it took 25s instead of over 6 minutes to copy the data over and took only 28% more space. I suspect that if I selected gzip -9 compression it would have been closer both in time and space to S3QL. But creating a ZFS snapshot was nearly instantaneous. Although ZFS-fuse probably doesn t have as many users as ZFS on Solaris, still it is available in Debian, and has a good backing behind it. I feel safer using it than I do using S3QL. So I think ZFS wins this comparison. I spent quite some time testing ZFS snapshots, which are instantaneous. (Incidentally, ZFS-fuse can t mount them directly as documented, so you create a clone of the snapshot and mount that.) They worked out as well as could be hoped. Due to dedupe, even deleting and recreating the entire content of the original filesystem resulted in less than 1MB additional storage used. I also tested creating multiple filesystems in the zpool, and confirmed that dedupe even works between filesystems. Incidentally wow, ZFS has a ton of awesome features. I see why you OpenSolaris people kept looking at us Linux folks with a sneer now. Only our project hasn t been killed by a new corporate overlord, so guess that maybe didn t work out so well for you <grin>. The Cloud Tie-In This discussion leaves another discussion: what to do about offsite backups? Assuming for the moment that I want to back them up over the Internet to some sort of cloud storage facility, there are about 3 options:

1. Get an Amazon EC2 instance with EBS storage and rsync files to it. Perhaps run ZFS on that thing.
2. Use a filesystem that can efficiently store data in S3 or Cloud Files (S3QL is the only contender here)
3. Use a third-party backup product (JungleDisk appears to be the leading option)

There is something to be said for using a different tool for offsite backups if there is some tool-level issue, that could be helpful. One of the nice things about JungleDisk is that bandwidth is free, and disk is the same $0.15/GB-mo that RackSpace normally charges. JungleDisk also does block-level dedup, and has a central management interface. This all spells nice for us. The only remaining question would be whether to just use JungleDisk to back up the backup server, or to put it on each individual machine as well. If it just backs up the backup server, then administrative burdens are lower; we can back everything there up by default and just not worry about it. On the other hand, if there is a problem with our main backups, we could be really stuck. So I d say I m leaning towards ZFS plus some sort of rsync solution and JungleDisk for offsite. I had two people suggest CrashPlan Pro on my blog. It looks interesting, but is a very closed product which makes me nervous. I like using standard tools and formats gives me more peace of mind, control, and recovery options. CrashPlan Pro supports multiple destinations and says that they do cloud hosting, but don t list pricing anywhere. So I ll probably not mess with it. I m still very interested in what comments people may have on all this. Let me know!

Debian's announcement to release Squeeze with a completely free Linux kernel caused quite some attention, which is actually a good thing. However, it also seems to have caused quite some uncertainty and was often partially misunderstood and miss quoted. I'll try to summarize and answer some of them in this blog post:

• Myth: Debian removed all firmware files from its kernels!
  Fact: No, it's just about the kernels, which will be shipped with the upcoming release Debian 6.0 Squeeze. The kernels in the current stable release Debian 5.0 Lenny remain as they are... Well, of course we will release security updates for them, but they will still contain the same firmware files as the present kernels.
• Myth: Debian is ripping stuff out of it's kernels
  Fact: Debian moved some firmware files from its main archive to the non-free part of the archive. They are still there, just in the part of archive for stuff not satisfying our Free Software Guidelines.
• Myth: Debian will be uninstallable for many users
  Fact: The non-free firmware files are still available over our infrastructure. Those, who are needed during installation (e.g. for network or storage controllers) can also be loaded during the installation (be it from CD or USB-Stick). We offer tarballs with these files (just unpack them on a usb-stick and plug it in when asked) as well as netinstall iso images already containing these non-free files. Of course these will stay there, after Squeeze has been released, too.
• Myth: Firmware files are needed, ripping them out doesn't accomplish anything and isn't good for our users
  Fact: Firmware files are needed by some drivers for some specific hardware, yes. But not all users want them. And, as we are now able to load all these firmware files when needed (in stead of compiling them into the driver itself), we are now able so ship them separately, why not do so? That allows those, who need non-free firmware to use them, while those, who don't want them, to not install them at all.
• Myth: Ah, those Debian freedom zealots again...
  Fact: It's not only us, actually, without the cooperation of many Linux Kernel developers, we couldn't have achieved that goal again. And it's not only us, who's interested in creating a free Linux Kernel, other major distributions see the problems, too. See for example the recent comment by a Fedora developer about changes in such a non-free firmware file. It just seems, that Debian was one of the first to realise the problem of non-free firmware files.
• Myth: Debian is going down on it's knees before Stallman
  Fact: I haven't talked with RMS about that, but I think Debian is still not free enough for him; as far as I know, he would like to see the non-free archive vanish completely, or at least not mentioned anywhere at all.

So, one question remains: What is to bad about non-free firmware files? Aren't they just some tiny programs executed in the CPU of the device? Why care about them Good question! Let's take possible legal issues aside, and just look the practical side. The core problem is: Without source (and tool chain to use the source) firmware files are just some random numbers for us. We don't know what they are doing, we can't analyse and improve them. We can't change them, we can't support them. Maybe you already followed the link above to comment of the Fedora developer. I cite him here again, because I think he summarized the problem so well:

Updated qlogic 2400 and 2500 firmware to 5.03.13. What does 5.03.13 do? No one knows, except for QLogic, and they're not telling. I asked, and they told me that information was only available under NDA. So, I encourage you to imagine what this firmware does, and the bugs it fixes. While you're at it, imagine a world where vendors release source code for their firmware.

So, now that we established the fact, that we can't support firmware files, one could wonder, if we actually need to do so. What harm can a simple, tiny program in a peripheral device do to your computer? Well, scientist already managed to created trojaned firmwares for some network cards. So it is a problem, and can even under some circumstanced lead to security problems! So, to summarise: Yes, Debian changed something in the Kernels. No, it will continue to work as well as usual. Some users might need to enable the non-free repository in their sources list, but those who won't, don't have to. Firmware files needed during installation are also available, and can be loaded by the installation system. So, what's all the fuss about? BTW: Those of you, who can't remember links very well, but fear, they still might need the links to the non-free images and tarballs. just remember two things: wiki and Firmware, as you'll find all you need on the Firmware page of Debian's wiki.

The DebConf talk selection process is in charge of selecting the talks at DebConf. Some of the reasons for having a talks team are to help ensure a high quality of talks, maintain a somewhat academic feeling for the conference, and to let attendees know what talks can be expected to be good.Even though there is a big talk selection process in advance, DebConf has always allowed attendees to schedule new talks during the conference - at DebConf10, these were called "ad-hoc" talks. At DC10, attendees could sign up for a room+time slot on a wiki page, and each night they would be added to the main schedule. In past years, this distinction had been called "official" (pre-selected) vs "non-official" (new ideas during DebConf) talks, which was extremely confusing to people. "Ad-hoc" talks are a very important part of DebConf: not all development meetings are pre-planned.Since we pretty much always surplus of talk rooms and times and we let people schedule talks during DebConf, people have asked: what is the purpose of talk selection? Are we going to let all talks be given? Are we going to reject talks just to say we selected them? When talks were "rejected" in the past, the speakers were told to just ask for a room during DebConf if they still wanted to give it. This question was raised on our mailing list. Even if there is a surplus of talk rooms, there are reasons for talk selection: it allows attendees to know what is most prepared, helps the video team to capture the best talks, lets talks be scheduled in the best room. You could also say it helps to organize talks into tracks and to collect abstracts for proceedings.For DebConf10, we introduced a "conference track" concept, a set of related talks which would be scheduled in the same room in sequence. This allowed a more structured conference for attendees, and perhaps recruiting speakers to fill in requested talks for the tracks. Example tracks from DebConf10 are Science, Enterprise, and Community Outreach. I would have liked to see some tracks more Debian related, such as: project infrastructure, project communication, project social issues, ..., but what we had was a good start that can be improved on, if the new team desires to.As for scheduling, there are two main parts: one phase before DebConf, assigning the selected talks to rooms and making the advance schedule. The other phase happens during DebConf and handles assigning rooms to people who want to do "ad-hoc" talks. The two phases aren't always done by the same person. The schedulers need to balance the expected audience and room capacity, need for video coverage, BOF vs auditorium style seating, not making talks overlap that are likely to appeal to the same audience, ... . The scheduling team works with the talk team, but doesn't necessarily have to be the same people.I'm very happy that there was good discussion about the selection and scheduling process for DebConf10. In the end, a talk selection and scheduling wiki page was created. This page had two main purposes. First, it discussed and documented the selection and scheduling process, so that it was transparent to people that year and provides a reference for future teams. Secondly, other people have added their proposed talks workflows, that they can be considered in future years.
Talk selection has many competing demands - because of the completeness of the wiki page, I'll just refer you there. The rest of this post mainly frames the types of questions the talks team has to face. If you have thoughts on the DebConf talk selection process, feel free to add your ideas to the talk selection process wiki page, or better yet, help out with the team this year.

I lived in Israel between 1994 and 1996 This was just after the signature of the Oslo agreements, the Rabin-Arafat signature. 1994-95 was a beautiful time to be in Israel; there was the excitement of a prospect of real peace with the Palestinians, something that had been just seen as unattainable for the past 50 years. But we also saw the buildup of a huge tension, of real hatred. And the dream of the end of the conflict, of two people living in peace side by side was heavily shattered on November 4 1995, when prime minister Rabin was assassinated. But for the sake of where I'm trying to get to (this post is meant to be about current Mexico, not of Israel 15 years ago), lets go a couple of months back. The Israeli society is a deeply democratic one (although I understand many people feel it is losing that trait). Lots of political propaganda of all sorts, many of it very creative, are posted on the streets And not by the government or by political parties, but by citizen movements. I tried to follow it... I remember a particular sticker that I started seeing very often: Rabin has no mandate over the Golan . At first, I understod that the Golan residents rejected Rabin's authority (but, why was it posted all over the country? And what specifically was it about?), until a friend explained to me that it meant that it rather meant that the right wing all over the country felt that Rabin had no right to push for negotiations involving in any way the Golan heights return to Syria because that was not his electoral platform. So, fast-forward in time, and 12500Km to the west. If you have followed my ramblings for the past ~five years, you are aware of what I think about Mexico's government It is an illegitimate government, which got the power thanks to a dirty and illegal (as per Mexico's legal system) campaign and to voting fraud. And, according to what I have read, around 30% of the population still believe firmly the elections were stolen. But even if they were legal, valid and legitimate What has happened since 2006 should not be seen as valid. In Mexico, we don't have such a democratic culture. The society is not used to demanding anything from the politicians. We often see, as an example, that when talking about political organs, there is a divide everybody understands although makes no sense: We talk about citizens as people unaffiliated to political parties (v.gr. the 1996 Federal Electoral Institute was very well regarded as it was "el IFE ciudadano", and has since then been overtaken by the parties). Why are the parties' members not seen as citizens? Well, maybe because of the same distortion field that makes us believe our de-facto president when he stated that, from the (then) over 28000 people killed in the war campaign against crime (he oficially declared he never used the term war... Of course, he was proved wrong), only ~10% were civilians. In my book, that would mean that ~25000 soldiers have been killed But no, he means mostly drug trafficants. So, as they are not innocent bystanders, they somehow stopped being civilians? But again, I'm straying off the topic. Let us assume for a minute that the 2006 elections were clean legal, and that the campaigns were not diffamatory, and all that. And tha we had a modern government, with the stated separation between the three powers and all that. By today, the Legislative power would have surely called for the government's disolution. Why? Again, we Mexicans are not used to what a democratic system means. We can see over and over examples of people who have been voted to power and forgot/betrayed everything they campaigned for. And we are used to that being... Normal. But, who did we vote for? Given the suposition of this last paragraphs (clean elections led to our current government), did the majority of the population vote for Felipe Calder n, the person? Or did it vote for Partido Acci n Nacional, the abstract entity organized as a party? No. The vote was (should have been) for the political campaign, for the things he promised he would work for, the priorities in the direction he would give the nation. Not too long ago, we took pride of living in a peaceful country. Yes, everybody feared the crime rate in the streets, everybody knew big cities such as the one I live in (and love living in) have deep problems. But the country had good relations and good respect from the international players, and it was mostly safe. And, I don't think any of the few readers that have read this far (regardless of where they live or how much they know me) has that opinion of Mexico anymore. What is now heard inside and outside is that we have a huge organized crime (mostly due to drug traffic), and that it's taking over the country. Why did this happen? The fact is, Calder n lacked legitimacy when he reached the presidency in December 2006. And although his campaign was centered on pushing work generation (his slogan was El presidente del empleo, The president of the jobs), lifting restrictions to entreprises, opening borders, getting more money in our economy... He knew he had to please the people that would protect him from any (then, probable) insurrection: Military and police. Our army has always been more a joke than an army... But they still have their guns. So, contrary to everything he promised, the first things he signed was a salary increase to all order-keeping forces in the country, and an all-out attack against drug traffic cartels. He had to make the order-keeping forces protagonist, useful and loyal. Now, fast-forward again to 2011. The country is living the worst violence since the end of the 1910-1920 revolution. Over 30,000 people have been killed. Economy has fallen as it had not for many years (this was the worst-performing country in America in 2009). Work generation was said to be high, but then corrected to indicate it was mediocre (just about the baseline of population growth). So, the balance so far of four years of... Government? Economic disaster. Social disaster. Promises unachieved. Starting actions he had no mandate to do. Felipe Calder n has no mandate to get us in this war. Even if he had been democratically elected, he would have to step down.

The family of Rcpp packages just grew by one: the first 0.1.0 release of RcppBDT is now on CRAN. RcppBDT stands for Rcpp Boost Date_Time. It employs what we call Rcpp modules: a mechanism which provides easier ways to expose C++ functions and classes to R (and which bears some resemblance to Boost.Python---see this vignette about Rcpp modules for more details). And thus RcppBDT provides R users with access to (some) Boost Date.Time functionality. I used Boost Date.Time because it is widely known, useful and well tested. It can also be used in pure template mode not requiring linking (if one foregoes string parsing and formatting which is fine here as we get this from R already; it can be added via a #define in the header file). With that, the package is rather portable as it only needs to find the Boost headers. (And no, I have not yet added a configure option for that.) Basic usage follows Rcpp modules and provides the C++ class via a Reference Class in R. That means using a new constructor and assessing functions via the $ operator. Here is an example:

R> library(RcppBDT)
Creating a new generic function for "print" in "RcppBDT"
Creating a new generic function for "format" in "RcppBDT"
R> d <- new(bdtMod$date, 2011, 1, 18)
R> format(d)
[1] "2011-01-18"
R>

Not very exiting yet: we create a date, using explicit year, months and date arguments and then format it. Something more useful follows:

R> d$getDayOfWeek()
[1] 2
R> d$getModJulian()
[1] 55579
R> d$getFirstOfNextMonth()
[1] "2011-02-01"
R>

We can use accessors to extracts parts of the date, or use functions to convert the date to different representations such as a modified Julian date. Moreover, given a date, we can apply helper functions such as get me the date of the beginning of the next month. More interesting still are helper functions such as the ones below. Note that this also shows the alternate access method using wrapper functions I added for the package, this may be more familiar to most R users:

R> getNthDayOfWeek(third, Wed, Dec, 2010)
[1] "2010-12-15"
R> getLastDayOfWeekInMonth(Sat, Dec, 2010)
[1] "2010-12-25"
R> getFirstDayOfWeekInMonth(Sat, Dec, 2010)
[1] "2010-12-04"
R> getFirstDayOfWeekAfter(Wed, as.Date("2010-12-31"))
[1] "2011-01-05"
R>

This uses some of the constants Jan, Feb, ... , Dec; Sun, Mon, ..., Sat and first ... fifth defined in the package; normal integers can also be used. The third Wednesday in a month is also known as the IMM Date to Finance-heads; the example was borrowed from Whit's earlier rboostdatetime code on github. More examples are in the demo available with the package and accessible via demo(RcppBDT) once you install the package. As always, feedback would be welcome, both on the usefulness (or lack thereof) of the Boost Date.Time functionality as well as on the Rcpp modules wrapping. Additional Boost Date.Time functionality (durations, times, ...) may be added; contributions would be welcome. The rcpp-devel mailing list off the R-Forge page for Rcpp is the best place to start a discussion.

Dear lazy web, I am writing a library that implements the Units Policy of Ubuntu. This library can be used to format sizes (of files, disks, memory, etc.) for displaying them and the other way around. For example, the size 12345 bytes can be formatted to 12.3 kB, 12.1 KiB, or 12.1 KB depending on the configuration. An other function will give you the possibility to convert 12.3 kB, 12.1 KiB, or 12.1 KB back to 12345 bytes. The preferred binary unit can be configured by an text file (per system and per user) and by an environment variable. The library is written in C and doesn t have any dependency. Therefore it can be used by any program on any desktop environment. Binding to other languages like C++, Python, Java are planned. Before creating a project for this library, I need a name for the library. The current prototype is called libdisplayunits, but this name reflect only the half of the project. It does not indicate that the library can be used for inputting sizes. What do you think? What is the best name for the library? Please poll and/or comment below.

<script charset="utf-8" language="javascript" src="http://static.polldaddy.com/p/3972978.js" type="text/javascript"></script> <noscript> View This Poll
survey software </noscript>

My friend Jeff deserves a sincere congratulation for finally unveiling his rebranded R consultancy Lemnica. One notable feature of the new website is a section called esoteric R which discusses less frequently-visited corners of the R world. It even boasts its own CRAN package esotericR with the example sources. esoteric R currently holds two articles. Jeff had sent me the one about introducing closures a while back, and I like it and may comment at another time. What caught me by surprise when Lemnica finally opened was the other article: R calling C. It is a fine article motivated by all the usual reasons that are e.g. mentioned in the Google Tech Talk which Romain and I gave last October about our work around Rcpp. But it is just not simple. Allow me to explain. When Jeff showed this C language file

#include <R.h>
#include <Rinternals.h>
SEXP esoteric_rev (SEXP x)  
  SEXP res;
  int i, r, P=0;
  PROTECT(res = allocVector(REALSXP, length(x))); P++;
  for(i=length(x), r=0; i>0; i--, r++)  
     REAL(res)[r] = REAL(x)[i-1];
   
  copyMostAttrib(x, res);
  UNPROTECT(P);
  return res;
 

and then needs several paragraphs to explain what is going on, what is needed to compile and then how to load it --- I simply could not resist. Almost immediately, I emailed back to him something as simple as this using both our Rcpp package as well as the wonderful inline package by Oleg which Romain and I more or less adopted:

library(inline)  ## for cxxfunction()
src <- 'Rcpp::NumericVector x = Rcpp::NumericVector(xs);
        std::reverse(x.begin(), x.end());
        return(x);'
fun <- cxxfunction(signature(xs="numeric"), body=src, plugin="Rcpp")
fun( seq(0, 1, 0.1) )

Here we load inline, and then define a three-line C++ program using facilities from our Rcpp package. All we need to revert a vector is to first access its R object in C++ by instantiating the R vector as a NumericVector. These C++ classes then provide iterators which are compatible with the Standard Template Library (STL). So we simply call the STL function reverse pointing the beginning and end of the vector, and are done! Rcpp then allows us the return the C++ vector which it turns into an R vector. Efficient in-place reversal, just like Jeff had motivated, in three lines. Best of all, we can execute this from within R itself:

R> library(inline)  ## for cxxfunction()
R> src <- 'Rcpp::NumericVector x = Rcpp::NumericVector(xs);
+         std::reverse(x.begin(), x.end());
+         return(x);'
R> fun <- cxxfunction(signature(xs="numeric"), body=src, plugin="Rcpp")
R> fun( seq(0, 1, 0.1) )
 [1] 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0
R>

Lastly, Jeff shows a more complete example wherein a new vector is created, and any potential attributes are copied as well. Naturally, we can do that too. First, we used clone() to make a deep copy (ie forcing creation of a new object rather than a mere proxy) and use the same R API function he accessed---but it our case both prefixed with ::Rf_ for R remapping (to protect clashed with other functions with identical names) and a global namespace identifier (as it is a global C function from R).

R> library(inline)
R> src <- 'Rcpp::NumericVector x = Rcpp::clone<Rcpp::NumericVector>(xs);
+         std::reverse(x.begin(), x.end());
+         ::Rf_copyMostAttrib(xs, x);
+         return(x);'
R> fun <- cxxfunction(signature(xs="numeric"), body=src, plugin="Rcpp")
R> obj <- structure(seq(0, 1, 0.1), obligatory="hello, world!")
R> fun(obj)
 [1] 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0
attr(,"obligatory")
[1] "hello, world!"
R> obj
 [1] 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
attr(,"obligatory")
[1] "hello, world!"
R>

Both the obj variable and the new copy contain the desired data attribute, the new copy is reversed, the original is untouched---and all in four lines of C++ called via one inline call. I have now been going on for over one hundred lines yet I never had to mention memory management, pointers, PROTECT or other components of the R API for C. Hopefully, this short writeup provided an idea of why Romain and I think Rcpp is the way to go for creating C/C++ functions for extending and enhancing R.

Every time I review some package for sponsoring, I usually end up suggesting to fix some of more picky things which lintian finds. It looks like everybody is using default set of checks and does not look at others. Indeed the defaults tell about things which are really errors or need override, but --pedantic and --display-info show also useful hints. However you need to take them more as a suggestions than as a errors. What I usually find interesting is --pedantic, because it tells about several things, which might be good idea to have in your package. Missing Homepage in debian/control is one of them and I believe this should exist for every package which has a homepage. The --display-info (or just -I) is also useful, for example it does spell checking of texts in binaries and man pages. Registration into doc-base is also useful integration into the system and you should definitely do it if the package provides documentation in HTML. To run these checks every time, you can create alias in your ~/.bashrc:

alias lintian='lintian -I --pedantic'

Filed under: Debian English 7 comments Flattr this!

The good folks at Bespoke Investment Group frequently show charts of so-called overbought or oversold levels; see e.g. here for the most recent global markets snapshot. Classifying markets as overbought or oversold is a popular heuristic. It starts from computing a rolling smoothed estimate of the prices, usually via a (exponential or standard) moving average over a suitable number of days (where Bespoke uses 50 days, see here). This is typically coupled with a (simple) rolling standard deviation. Overbought and oversold regions are then constructed by taking the smoothed mean plus/minus one and two standard deviations. Doing this is in R is pretty easy thanks to the combination of R's rich base functions and its add-on packages from CRAN. Below is a simply function I wrote a couple of months ago---and I figured I might as well release. It relies on the powerful packages quantmod and TTR by my pals Jeff Ryan and Josh Ulrich, respectively.

## plotOBOS -- displaying overbough/oversold as eg in Bespoke's plots
##
## Copyright (C) 2010 - 2011  Dirk Eddelbuettel
##
## This is free software: you can redistribute it and/or modify it
## under the terms of the GNU General Public License as published by
## the Free Software Foundation, either version 2 of the License, or
## (at your option) any later version.
suppressMessages(library(quantmod))     # for getSymbols(), brings in xts too
suppressMessages(library(TTR))          # for various moving averages
plotOBOS <- function(symbol, n=50, type=c("sma", "ema", "zlema"), years=1, blue=TRUE)  
    today <- Sys.Date()
    X <- getSymbols(symbol, src="yahoo", from=format(today-365*years-2*n), auto.assign=FALSE)
    x <- X[,6]                          # use Adjusted
    type <- match.arg(type)
    xd <- switch(type,                  # compute xd as the central location via selected MA smoother
                 sma = SMA(x,n),
                 ema = EMA(x,n),
                 zlema = ZLEMA(x,n))
    xv <- runSD(x, n)                   # compute xv as the rolling volatility
    strt <- paste(format(today-365*years), "::", sep="")
    x  <- x[strt]                       # subset plotting range using xts' nice functionality
    xd <- xd[strt]
    xv <- xv[strt]
    xyd <- xy.coords(.index(xd),xd[,1]) # xy coordinates for direct plot commands
    xyv <- xy.coords(.index(xv),xv[,1])
    n <- length(xyd$x)
    xx <- xyd$x[c(1,1:n,n:1)]           # for polygon(): from first point to last and back
    if (blue)  
        blues5 <- c("#EFF3FF", "#BDD7E7", "#6BAED6", "#3182BD", "#08519C") # cf brewer.pal(5, "Blues")
        fairlylight <- rgb(189/255, 215/255, 231/255, alpha=0.625) # aka blues5[2]
        verylight <- rgb(239/255, 243/255, 255/255, alpha=0.625) # aka blues5[1]
        dark <- rgb(8/255, 81/255, 156/255, alpha=0.625) # aka blues5[5]
      else  
        fairlylight <- rgb(204/255, 204/255, 204/255, alpha=0.5)         # grays with alpha-blending at 50%
        verylight <- rgb(242/255, 242/255, 242/255, alpha=0.5)
        dark <- 'black'
     
    plot(x, ylim=range(range(xd+2*xv, xd-2*xv, na.rm=TRUE)), main=symbol, col=fairlylight) 		# basic xts plot
    polygon(x=xx, y=c(xyd$y[1]+xyv$y[1], xyd$y+2*xyv$y, rev(xyd$y+xyv$y)), border=NA, col=fairlylight) 	# upper
    polygon(x=xx, y=c(xyd$y[1]-1*xyv$y[1], xyd$y+1*xyv$y, rev(xyd$y-1*xyv$y)), border=NA, col=verylight)# center
    polygon(x=xx, y=c(xyd$y[1]-xyv$y[1], xyd$y-2*xyv$y, rev(xyd$y-xyv$y)), border=NA, col=fairlylight) 	# lower
    lines(xd, lwd=2, col=fairlylight)   # central smooted location
    lines(x, lwd=3, col=dark)           # actual price, thicker
    invisible(NULL)
 

After downloading data and computing the rolling smoothed mean and standard deviation, it really is just a matter of plotting (appropriate) filled polygons. Here I used colors from the neat RColorBrewer package with some alpha blending. Colors can be turned off via an option to the function; ranges, data length and type of smoother can also be picked. To call this in R, simply source the file and the call, say, plotOBOS("^GSPC", years=2) which creates a two-year plot of the SP500 as shown here: This shows the market did indeed bounce off the oversold lows nicely on a few occassions in 2009 and 2010 --- but also continued to slide after hitting the condition. Nothing is foolproof, and certainly nothing as simple as this is, so buyer beware. But it may prove useful in conjunction with other tools. The code for the script is here and of course available under GPL 2 or later. I'd be happy to help incorporate it into some other finance package. Lastly, if you read this post this far, also consider our R / Finance conference coming at the end of April.

Edit: Corrected several typos with thanks to Josh.

Let's say you set up a machine using an encrypted disk with LUKS (debian-installer's partman makes this wonderfully easy!). You choose an initial passphrase, get the machine working, and it's working great. Then, you need to restart it, and realize that (for whatever reason) you've forgotten or lost the passphrase for the volume. oops! (i'm sure this has never happened to you -- let's just pretend it's your less-fortunate friend). If your system is still running, and you have superuser access to it, you can actually set a new passphrase for the LUKS volume using information that the dm-crypt kernel module has about the in-use mapping. In my examples, i'll imagine that the source volume is /dev/XXX2 and the exported cleartext volume is known by the device-mapper as XXX2_crypt In the bigger picture, this should serve as a reminder that even though your disk is encrypted, if someone gets live access to the superuser account on a system with the encryption keys loaded, your data is no longer secret. Before you do any tweaking, you might want to back up your LUKS header, just in case:

0 root@example:~# umask 077
0 root@example:~# cryptsetup luksHeaderBackup /dev/XXX2 --header-backup-file XXX2.luksheader.backup
0 root@example:~# 

Maybe also copy that off the machine, since a copy of the LUKS header stored within its own volume isn't terribly useful for a backup-recovery situation. You might also be interested in looking at the contents of the LUKS header:

0 root@example:~# cryptsetup luksDump /dev/XXX2
LUKS header information for /dev/XXX2
Version:       	1
Cipher name:   	aes
Cipher mode:   	cbc-essiv:sha256
Hash spec:     	sha1
Payload offset:	2056
MK bits:       	256
MK digest:      93 51 6c 66 ec ce 32 54 6f 6b 52 d1 27 9b 5a 62 6f 6b 52 d1
MK salt:       	b2 ca 20 49 9f 78 49 c2 fe 15 b4 0f 74 11 23 49
               	64 9e 61 bb f2 82 60 47 a5 76 fa a4 24 0e 5a 7e
MK iterations: 	10
UUID:          	052f1da0-21a1-11e0-ac64-0800200c9a66
Key Slot 0: ENABLED
	Iterations:         	218733
	Salt:               	f2 ae 8c 53 48 a5 f0 bf e1 2c 06 5f 5a bd ff d9
	                      	9a 2e d1 49 3a 63 f8 49 82 ed ae 86 7b 7b 7e 76 
	Key material offset:	8
	AF stripes:            	4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED
0 root@example:~# 

Now, the fix: We pull the live "master key" from the running device map, and fill a new luksKeySlot from it (this example uses bash's <() syntax for process substitution -- if you use a different shell, i'm sure you can find a different way to do it):

0 root@example:~# cryptsetup --master-key-file <( 
> dmsetup --showkeys table   awk '/^XXX2_crypt: /  print $6  '   tr -d '\n'   perl -e 'print pack("H*", <STDIN>);' 
> ) luksAddKey /dev/XXX2
Enter new passphrase for key slot: abc123
Verify passphrase: abc123
2 root@example:~# 

(note that the luksAddKey invocation above returned an error code of 2 even though it succeeded. I think this is a bug in cryptsetup's return code, not a bug in the password resetting -- it should have returned 0 instead of 2). You can check to see that a new key slot was enabled by re-running cryptsetup luksDump And if you really want to double-check before you reboot, you can try enabling a third keyslot using the passphrase you just added, since this would not succeed if your new passphrase failed to unlock any of the existing keyslots:

0 root@example:~# cryptsetup luksAddKey /dev/XXX2
Enter any passphrase: abc123
Enter new passphrase for key slot: anotherpassphrase
Verify passphrase: anotherpassphrase
0 root@example:~# 

You can also get rid of the original keyslot (which you don't know the passphrase to) like this:

0 root@example:~# cryptsetup luksKillSlot /dev/XXX2 0
Enter any remaining LUKS passphrase: abc123
0 root@example:~# 

(the above commands were all demonstrated using debian testing, with cryptsetup 2:1.1.3-4 and dmsetup 2:1.02.48-4) Tags: cryptsetup, dmsetup, luks, tip

The video format struggle on the web continues, and the three contenders seem to be Ogg Theora, H.264 and WebM. Most video sites seem to use H.264, while others use Ogg Theora. Interestingly enough, the comments I see give me the feeling that a lot of people believe H.264 is the most supported video format in browsers, but according to the Wikipedia article on HTML5 video, this is not true. Check out the nice table of supprted formats in different browsers there. The format supported by most browsers is Ogg Theora, supported by released versions of Mozilla Firefox, Google Chrome, Chromium, Opera, Konqueror, Epiphany, Origyn Web Browser and BOLT browser, while not supported by Internet Explorer nor Safari. The runner up is WebM supported by released versions of Google Chrome Chromium Opera and Origyn Web Browser, and test versions of Mozilla Firefox. H.264 is supported by released versions of Safari, Origyn Web Browser and BOLT browser, and the test version of Internet Explorer. Those wanting Ogg Theora support in Internet Explorer and Safari can install plugins to get it. To me, the simple conclusion from this is that to reach most users without any extra software installed, one uses Ogg Theora with the HTML5 video tag. Of course to reach all those without a browser handling HTML5, one need fallback mechanisms. In NUUG, we provide first fallback to a plugin capable of playing MPEG1 video, and those without such support we have a second fallback to the Cortado java applet playing Ogg Theora. This seem to work quite well, as can be seen in an example from last week. The reason Ogg Theora is the most supported format, and H.264 is the least supported is simple. Implementing and using H.264 require royalty payment to MPEG-LA, and the terms of use from MPEG-LA are incompatible with free software licensing. If you believed H.264 was without royalties and license terms, check out "H.264 Not The Kind Of Free That Matters" by Simon Phipps. A incomplete list of sites providing video in Ogg Theora is available from the Xiph.org wiki, if you want to have a look. I'm not aware of a similar list for WebM nor H.264. Update 2011-01-16 09:40: A question from Tollef on IRC made me realise that I failed to make it clear enough this text is about the <video> tag support in browsers and not the video support provided by external plugins like the Flash plugins.

I'm thrilled to discover that GNU-Darwin, which carried the bold claim that "GNU-Darwin aims to be the most free software distribution", distributes copies of the XNU source code with Apple's additional license rider of "The rights granted to you under the License may not be used to create, or enable the creation or redistribution of, unlawful or unlicensed copies of an Apple operating system, or to circumvent, violate, or enable the circumvention or violation of, any terms of an Apple operating system software license agreement", which certainly sounds like a restriction of use to me.

Free software's awfully like sausages - wonderfully tasty, but sometimes you suddenly discover that you've been eating sheep nostrils for the past 15 years of your life. The typical direction is realising that you've build your entire business model on something that you're actually legally obliged to give to your customers and now they're actually trying to assert their rights, but sometimes people make assumptions about freedom that don't hold. Apple last released a free version of their kernel source in 2006. Everything since then has been encumbered by additional use restrictions.

Still, much better than Microsoft - huge portions of the OS source code are available for anyone to download, examine, modify and rebuild. Apple's to be commended for dropping so much code to the community despite it having bought them almost nothing in return. But if anyone from Apple's reading, I'd love to fix up the thermal handling on your hardware under Linux. I suspect we're doing something horrible to your machines, but I don't really know what yet.

Reading Scott's recent announcement on his move to Google was both surprising and a pleasure. Surprising, because it'll take time to stop associating his name to Ubuntu, Canonical, and the nice experiences I had while I worked with them. A pleasure, because his blog post was full of reminiscences of the very early days of a project that ended up being way more successful in just a few years than probably anyone in the Oxford conference could imagine. Scott, best of luck for this new adventure! Scott's write-up includes a sentence that made me remember I had been wanting to write a blog post related to all of this, but was pending Mark Shuttleworth's permission for posting:

Ok, Mark wasn t really a Nigerian 419 scammer, but some people did discard his e-mail as spam! Scott James Remnant

Many know the story of how I ended not being part of the Super-Secret-Debian-Startup Scott mentions. I even wrote about it in a blog post, 3 years ago:

[...] nothing beats the next email which sat for some dramatic 6 months in my messy inbox until I found out in the worst of the possible scenarios. Let's go back to late February, 2004, when I had no job, and I didn't have a clue on what to do with my life.

From: Mark Shuttleworth <mark@hbd.com>
Subject: New project to discuss
To: Jordi Mallach <jordi@debian.org>
Date: Sun, 29 Feb 2004 18:33:51 +0000
[...]
I'm hiring a team of debian developers to work full time on a new
distribution based on Debian. We're making internationalisation a prime
focus, together with Python and regular release management. I've discussed
it with a number of Debian leaders and they're all very positive about it.
[...]

I'm not sure if I totally missed it as it came in, or I skimmed through it and thought WTF?! Dude on crack or I just forgot I need to reply to this email , but I'd swear it was the former. Not long after, no-name-yet.com popped up, the rumours started spreading around Debian channels. Luckily, I got a job at LliureX two months later, where I worked during the following 2 years, but that's another story. I guess it was July or so when Ubuntu was made public, and Mark and his secret team organised a conference (blog entries [1] [2] [3] [4] [5]), just before the Warty release, and I was invited to it, for the same reasons I got that email. During that conference, probably because Mark sent me some email and I applied a filter to get to it, I found the lost email, and felt like digging a hole to hide for a LONG while. I couldn't believe the incredible opportunity I had missed. I went to Mark and said "hey, you're not going to believe this", and he did look quite surprised about someone being such an idiot. I wonder if I should reply to his email today...

When the usual suspects in the secret Spanish Debian Cabal channel read this blog post, they decided Mark deserved a reply, even if it would hit his inbox more than three and a half years late. :) With great care, we crafted an email that would look genuinely stupid in late 2007, but just arrogant and idiotic in 2004, when Ubuntu was just an African word, and the GNU/Linux distribution landscape was quickly evolving at the time, Gentoo Linux had the posh distribution crown, that Debian had held for quite a few years. I even took enough care to forge the X-Operating-System and User-Agent headers so they matched whatever was current in Debian in February 2004, and of course, top-posting seemed most appropriate. So Mark woke up that Monday, fired up his email client, and got... this:

Date: Mon, 1 Mar 2004 09:47:55 +0100
From: Jordi Mallach <jordi@sindominio.net>
To: Mark Shuttleworth <mark@hbd.com>
Subject: Re: New project to discuss
Organization: SinDominio
X-Operating-System: Debian GNU/Linux sid (Linux 2.6.3 i686)
User-Agent: Mutt/1.5.5.1+cvs20040105i
Hi Mark,
Thanks for your email. I nearly deleted this e-mail because for some
reason I thought it was targetted spam.
Your project looks very interesting, almost like a dream come true.
However, I feel a bit uneasy about your proposal. Something just doesn't
fit.
Why would someone start a company to work on /yet another/ Debian
derivative? Have you heard about Progeny's sad story? I think it's a
great example to show that Debian users don't want Debian-based distros,
they want people to work on the "real thing". Besides, I don't think
there's much more place for successful commercial distros, with Red Hat
and SuSE having well-established niches in the US and Europe.
Also, why focus on Debian specifically, Why not, for example, Gentoo,
which has a lot of buzz these days, and looks poised to be the next big
distribution?
To be honest, I think only a few people have the stamina or financial
stability to undertake a project like this, so I'd like to know
a bit more about you, and details on how you plan to sustain the
expenses.
Those are the main issues that worry me about your project. Other than
that, I would be interested in taking part in it, as I'm currently
unemployed and working on something Debian-based would be just too good
to miss.
You can reach me at +34 123 45 67 89, or if you feel like flying people
around Europe, I probably can be in the UK whenever it fits you.
Thanks, and hoping to hear from you again,
Jordi
On Sun, Feb 29, 2004 at 06:33:51PM +0000, Mark Shuttleworth wrote:
> Hi Jordi
>
> We haven't met, but both Jeff Waugh and Martin Michelmayr recommended that
> I get in touch with you in connection with a new project that I'm starting.
>
> I'm hiring a team of debian developers to work full time on a new
> distribution based on Debian. We're making internationalisation a prime
> focus, together with Python and regular release management. I've discussed
> it with a number of Debian leaders and they're all very positive about it.
>
> Would you be available to discuss it by telephone? I'm in the UK, so we
> could probably find a good timezoine easily enough ;-) Let me knof if
> you're keen to discuss it, when and what number to call.
>
> Cheers,
> Mark
>
> --
> Try Debian GNU/Linux. Software freedom for the bold, at www.debian.org
> http://www.markshuttleworth.com/

As you can imagine, his reaction was immediate:

Date: Mon, 22 Oct 2007 11:13:54 +0100
From: Mark Shuttleworth <mark@hbd.com>
To: Jordi Mallach <jordi@sindominio.net>
Subject: Re: New project to discuss
Jordi! I just got this now! Did you recently flush an old mail queue?

With thanks to all the Spanish Cabal members who were involved!

Not for the first time, I ve seen the suggestion in the echo chamber that Mono packages should be moved from Debian into the non-free repository, which is not formally part of Debian. The reason, as it so often is, is patents specifically this time, the searing risk posed to Debian and its users that Mono s packaging does not (and technically could not without forking from upstream) provide base class libraries which implement only the content of ECMA-335 4th Edition. As I understand it, this implies three things about the suggestion/demand: firstly, that the individual in question believes that Mono end users are at risk from patent litigation from Microsoft Corp because Mono s implementation of Microsoft.NET beyond the content of ECMA 334/335 infringes on Microsoft patents; secondly, that the Microsoft Community Promise which promises not to assert legal claims over third party implementations of ECMA-335 4th Edition (and ECMA-334 4th Edition which defines the C# language) would render a pure ECMA-only runtime safe if it existed (which it does not); thirdly that without the protection offered by the Microsoft Community Promise, the source code licenses of Mono are irrelevant the patent risk renders the software non-Free. It appears, unfortunately, that the community of Free Software Advocates don t actually understand what Free Software actually IS. That explains an awful lot, but should surprise nobody. So here s a lesson on what, exactly, is being advocated for. The Free Software Foundation defines four Software Freedoms these are the minimum criteria required for something to be considered Free Software by the FSF:

• The freedom to run the program, for any purpose (freedom 0).
• The freedom to study how the program works, and change it to make it do what you wish (freedom 1). Access to the source code is a precondition for this.
• The freedom to redistribute copies so you can help your neighbor (freedom 2).
• The freedom to distribute copies of your modified versions to others (freedom 3). By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.

Other groups have their own variants on these, but those are really just clarifications on the FSF definition - for example, the Debian Free Software Guidelines mostly line up, but have some additional clauses such as clause 4 which allows software to be considered Free if source code may be redistributed without modifications, as long as patches may be shipped alongside. These four freedoms are offered to you by the software s copyright holders only, and apply regardless of their choice of license any Free license, from a lengthy legal tome like the MPL to the completely-Free WTFPL, will offer you these four basic freedoms as a minimum, and any additional clauses in their licenses cannot seek to restrict these freedoms. These four freedoms represent the beginning, and end, of whether a piece of software is Free or not. Software does not need to be developed in the open, in a community-responsive way, in order to qualify as Free projects such as Google s Android, which are developed under a throw a final release over the wall, bugs and all, and expect people to thank you for it model, are still free, even if contribution is difficult. Actually, on a related note, software does not need to solicit upstream contribution of any kind in order to qualify as Free as long as you personally can redistribute the code with changes, then that s enough. Software does not need to serve a fully or even partially legal purpose in order to qualify as Free the favoured tool for causing distributed denial of service attacks, Low Orbit Ion Cannon, is Free Software, even though realistically it serves no legal purpose. DeCSS, the code initially used to allow DVD media to play on Linux (by breaking the CSS encryption mechanism) is Free Software. Software does not need to be useful or tasteful in order to qualify as Free the Last Measure Operating System, a minimalist OS primarily designed to loudly display the famous shock site images from goatse and related, is Free Software. Even in somewhat less clear-cut cases of taste, your personal opinion of software has no bearing on whether it is Free Software, as long as the four freedoms are guaranteed by the author(s). Software does not need to have only Free dependencies to qualify as Free Software it is entirely permissible to write software which relies upon a non-Free framework or library, and release your code under a Free license. It is the downstream recipient s problem to provide the dependencies, including their choice to craft a Free replacement for any non-Free code you make use of. Debian has a special repository called contrib, where Free software which only works with non-Free data, lives for example, Free game engines which require the insertion of proprietary game data in order to operate. You could write Free addons for expensive proprietary software such as Matlab, and as long as your code is Free, your responsibilities are met. Software does not need to avoid patents software, algorithmic, or otherwise in order to qualify as Free. The Freetype font library was still entirely Free Software when Apple were slinging threats around regarding font hinting data. FFmpeg, the Swiss Army knife of media codec libraries, is Free Software regardless of the number of codec patents it breaks. Software does not need any third party approval to be Free Software the rights of Free Software can only be offered by the copyright holders, and the opinion of third parties is an irrelevance as to whether software is Free. The GPL d clone of Blizzard Entertainment s Battle.net servers, bnetd, is Free Software, regardless of legal takedown notices. Third parties cannot influence whether or not a piece of software is Free. They can influence tangentially related topics, such as whether the software can be legally used, but that s the limit. And even within a given piece of software, where copyright is shared by contributors, the author of one component has no say on other components. And you can t make code which is already released as Free, suddenly un-Free you can, if you hold all the copyrights, close up future versions, but your existing code remains Free forever. Reasonably, you can opt to avoid using a piece of software because you have requirements beyond it merely being Free Software Cdrtools has been avoided in Debian for a long time due to the upstream author s legal threats and rambling but that is a side issue as to the question of whether or not the software is Free. Patents are simply not involved in the question of whether or not something is Free Software, except for one narrow case: where Free Software is released by somebody who also holds related patents, and uses a license such as Ms-PL or Apache 2.0 or GPLv3 which requires them to also release those patents to those using/distributing the software. Outside that narrow situation, patents do not relate to the question of whether something is Free Software even if a company releases some source code under a license like BSD then demands patent fees from end users. So, on to the original topic of Mono. Every piece of Mono s source code is released by its authors under a license which guarantees the FSF s Four Freedoms. Whether or not you find Mono useful or tasteful does not affect that Free status. Whether or not Mono infringes upon any laws or patents does not affect that Free status. That Mono contains some libraries whose upstream author is Microsoft does not give Microsoft even the remotest claim to a single line of code outside the code they wrote and even then, it wouldn t be an issue, since the licenses they use are Free. In fact, both the licenses used in the Microsoft portions of the source base make patent grants to all users, in addition to guaranteeing the FSF s Four Freedoms and any license contamination would decrease, not increase, any risk of legal attack from Microsoft. There s even plenty of Microsoft code available for re-use at a lower level than the currently re-used libraries: The Microsoft.NET Micro Framework (for use on embedded platforms direct to the metal) is under the Free Software Apache 2.0 license, and would provide access to some of Microsoft s runtime and class library code, complete with a patent grant, if it were desired. Please try to keep your thought processes straight. If you want to argue that you re all for Free Software, please remember that there s plenty of Free Software you might not approve of and don t claim to be a Free Software advocate then use bogus arguments to claim that Free Software is not Free. Free Software includes LMOS and LOIC and Mono, whether you like it or not.

I'm working on improving various bits of Apple hardware support, which includes trying to make EFI booting on Macs more sensible. One of my test machines is an 11" Macbook Air, which is a beautiful piece of hardware. If you want something small and light that's approximately a proper computer rather than a netbook, there's nothing else in the x86 market to touch it. Nothing.

So it's a shame that while Apple employs hardware designers who are absolutely the best in their field, their firmware developers are utterly incompetent. The EFI spec describes a protocol called GOP (Graphics Output Protocol) which allows the operating system to find out things like the screen resolution, framebuffer location and stride. This machine kindly passes everything back correctly, except for stride being wrong. Like, utterly wrong. Implausibly wrong. Obviously, terrifyingly wrong. So wrong that if you trust it, your screen image looks like it's been involved in some sort of awful pasta making accident. But never mind, that's a single line workaround.

But there's far, far worse. EFI has three modes of operation - boot services, runtime services, and runtime services with a virtual memory map. Boot services are, as the name implies, intended to be used by the bootloader. They're contained within memory marked with a type of either BOOT_SERVICES_CODE or BOOT_SERVICES_DATA. Once the bootloader has loaded the OS, just before handing over to it, it calls ExitBootServices() and the firmware is then at liberty to clear those areas of memory. The OS never executes boot services code and can use the memory that was allocated to them.

So then we have runtime services. EFI is, depending on how you want to look at it, either saner or much less sane than traditional BIOS access. The firmware gives you a bunch of function pointers and you then simply call them with native calling convention (this, incidentally, is why it's pretty much impossible to run a 32-bit OS on 64-bit EFI, or a 64-bit OS on a 64-bit system that happens to have 32-bit EFI). To begin with the firmware assumes flat physical mapping, which isn't entirely ideal when you're an OS with virtual address space - so there's a SetVirtualAddressMap() call which lets the OS pass a memory map to the firmware, and from then on everything just works.

Except on this machine. On this machine, you call SetVirtualAddressMap() and the kernel faults because it's just tried to execute code from a region of address space that's marked as non-executable. Closer examination revealed that, yes, this area of address space is marked as RAM and so the kernel's doing the right thing. Except, obviously, the right thing generally doesn't involve crashing the machine at boot time, so there was something else up.

That something else turned out to be SetVirtualAddressMap() calling into boot services code. The boot services code that's not supposed to be touched after ExitBootServices() has been called. The boot services code that's flagged as BOOT_SERVICES_CODE and was therefore mapped as RAM by us because, well, nothing's there any more. The boot services code that we're ignoring exactly as the spec says we should.

So, a quick change to the bootloader to pass us an e820 map[1] that marks the boot services regions as reserved, a quick change to the kernel to mark BOOT_SERVICES_CODE regions as executable and we're in business. Except that when we call SetVariable() to configure the bootloader, the firmware tries to read from an address that's approximately 54TB above anything else.

BlApple.

[1] Oh, yes, despite E820 being an x86 BIOS-specific thing, we use it for the in-kernel representation of the address map under EFI as well. So despite the EFI memory map being available to the kernel, the bootloader has to construct something that looks like an E820 map, losing information in the process. So even though it's been given the E820 map, the kernel has to look through the EFI memory map in order to find the EFI runtime services section in order to mark it executable. This is, arguably, insane, but so is the entirety of EFI so that's ok then.