I m writing a bit about ZFS these days, and I thought I d write a bit about why I am using it, why it might or might not be interesting for you, and what you might do about it. ZFS Features and Background ZFS is not just a filesystem in the traditional sense, though you can use it that way. It is an integrated storage stack, which can completely replace the need for LVM, md-raid, and even hardware RAID controllers. This permits quite a bit of flexibility and optimization not present when building a stack involving those components. For instance, if a drive in a RAID fails, it needs only rebuild the parts that have actual data stored on them. Let s look at some of the features of ZFS:

• Full checksumming of all data and metadata, providing protection against silent data corruption. The only other Linux filesystem to offer this is btrfs.
• ZFS is a transactional filesystem that ensures consistent data and metadata.
• ZFS is copy-on-write, with snapshots that are cheap to create and impose virtually undetectable performance hits. Compare to LVM snapshots, which make writes notoriously slow and require an fsck and mount to get to a readable point.
• ZFS supports easy rollback to previous snapshots.
• ZFS send/receive can perform incremental backups much faster than rsync, particularly on systems with many unmodified files. Since it works from snapshots, it guarantees a consistent point-in-time image as well.
• Snapshots can be turned into writeable clones , which simply use copy-on-write semantics. It s like a cp -r that completes almost instantly and takes no space until you change it.
• The datasets ( filesystems or logical volumes in LVM terms) in a zpool ( volume group , to use LVM terms) can shrink or grow dynamically. They can have individual maximum and minimum sizes set, but unlike LVM, where if, say, /usr gets bigger than you thought, you have to manually allocate more space to it, ZFS datasets can use any space available in the pool.
• ZFS is designed to run well in big iron, and scales to massive amounts of storage. It supports SSDs as L2 cache and ZIL (intent log) devices.
• ZFS has some built-in compression methods that are quite CPU-efficient and can yield not just space but performance benefits in almost all cases involving compressible data.
• ZFS pools can host zvols, a block device under /dev that stores its data in the zpool. zvols support TRIM/DISCARD, so are ideal for storing VM images, as they can instantly release space released by the guest OS. They can also be snapshotted and backed up like the rest of ZFS.

Although it is often considered a server filesystem, ZFS has been used in plenty of other situations for some time now, with ports to FreeBSD, Linux, and MacOS. I find it particularly useful:

• To have faith that my photos, backups, and paperwork archives are intact. zpool scrub at any time will read the entire dataset and verify the integrity of every bit.
• I can create snapshots of my system before running apt-get dist-upgrade, making it easy to track down issues or roll back to a known-good configuration. Ideal for people tracking sid or testing. One can also easily simply boot from a previous snapshot.
• Many scripts exist that make frequent snapshots, and retain the for a period of time as a way of protecting work in progress against an accidental rm. There is no reason not to snapshot /home every 5 minutes, for instance. It s almost as good as storing / in git.

The added level of security in having cheap snapshots available is almost worth it by itself. ZFS drawbacks Compared to other Linux filesystems, there are a few drawbacks of ZFS:

• CDDL will prevent it from ever being part of the Linus kernel tree
• It is more RAM-hungry than most, although with tuning it can even run on the Raspberry Pi.
• A 64-bit kernel is strongly preferred, even in low-memory situations.
• Performance on many small files may be less than ext4
• The ZFS cache does not shrink and expand in response to changing RAM usage conditions on the system as well as the normal Linux cache does.
• Compared to btrfs, ZFS lacks some features of btrfs, such as being able to shrink an existing pool or easily change storage allocation on the fly. On the other hand, the features in ZFS have never caused me a kernel panic, and half the things I liked about btrfs seem to have.
• ZFS is already quite stable on Linux. However, the GRUB, init, and initramfs code supporting booting from a ZFS root and /boot is less stable. If you want to go 100% ZFS, be prepared to tweak your system to get it to boot properly. Once done, however, it is quite stable.

Converting to ZFS I have written up an extensive HOWTO on converting an existing system to use ZFS. It covers workarounds for all the boot-time bugs I have encountered as well as documenting all steps needed to make it happen. It works quite well. Additional Hints If setting up zvols to be used by VirtualBox or some such system, you might be interested in managing zvol ownership and permissions with udev.

I m a geek. I enjoy playing with different filesystems, version control systems, and, well, for that matter, radios. I have lately started to worry about the risks of silent data corruption, and as such, looked to switch my personal systems to either ZFS or btrfs, both of which offer built-in checksumming of all data and metadata. I initially opted for btrfs, because of its tighter integration into the Linux kernel and ability to shrink an existing btrfs filesystem. However, as I wrote last month, that experiment was not a success. I had too many serious performance regressions and one too many kernel panics and decided it wasn t worth it. And that the SuSE people got it wrong, deeply wrong, when they declared btrfs ready for production. I never lost any data, to its credit. But it simply reduces uptime too much. That left ZFS. Before I build a system, I always want to make sure I can repair it. So I started with the Debian Live rescue image, and added the zfsonlinux.org repository to it, along with some key packages to enable the ZFS kernel modules, GRUB support, and initramfs support. The resulting image is described, and can be downloaded from, my ZFS Rescue Disc wiki page, which also has a link to my source tree on github. In future blog posts in the series, I will describe the process of converting existing Debian installations to use ZFS, of getting them to boot from ZFS, some bugs I encountered along the way, and some surprising performance regressions in ZFS compared to ext4 and btrfs.

Well if you can read this then you know it s working. After way too many weeks, Debian will have WordPress version 3.8. Thanks to Rapha l for his kind assistance and answering my questions about how it was built. The upload is still gurgling along and will make it there in its own time. He said Handing over packages is hard, I d agree but say taking over them is too. So, what does WordPress 3.8 look like? From the frontend I didn t really notice much. The big changes, at least cosmetically, seem to be for the admin backend. It just look slicker and cleaner. Hopefully Debian users find the update useful and I ve not broken anything. There s always the BTS if there is. I ve deliberately tried to minimise the changes for this version to limit the breakage.

A bug fix release of Armadillo came out and was turned into a new RcppArmadillo release which is now on CRAN and will go shortly to Debian. The complete list of changes is below.

Changes in RcppArmadillo version 0.4.000.2 (2014-01-21)

• Upgraded to Armadillo release Version 4.000.2
  • fix for randi() generating out-of-interval values
  • workaround for a bug in the Intel compiler

Courtesy of CRANberries, there is also a diffstat report for the most recent release As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Formally, today is my first day as a student on a formal, scholarized institution Basically for the first time in almost twenty years! Yes, those that know me know that I aspire to live the life of academia. I have worked at public universities for almost all of my adult life (between 1997 and 1999 I worked at a local ISP and at a private school), and have had a minor academic position ( T cnico Acad mico ) for almost ten years. And not having a proper degree limited me from pursuing anything further. Then, in early 2010 I presented the written exam. By late 2010, the corresponding oral exam. That allowed me to get my formal diploma in December 2010. By the end of 2011, I requested to be a teacher in the Engineering Faculty of UNAM, and started teaching Operating Systems a year ago, in January 2012. So, a good advance in the last few years... But I know that if I just sit here, I won't be able to advance my position towards really entering the Sacred Halls of Academia. And there are some rituals I have to comply with. One of those rituals is... Devoting some long time to studying under the formal structures. Ok, so I'm finally a postgraduate student I have enrolled in Especialidad en Seguridad Inform tica y Tecnolog as de la Informaci n, a short (one year) postgraduate program in ESIME Culhuac n, of Instituto Polit cnico Nacional (a small campus of Mexico's second-largest university). Some friends have asked me, why am I starting with a Specialization and not a Masters degree. Some simple reasons: Just as when I went to Tijuana in 2010 to do my written exam, once I got and started with the paperwork, I didn't want to let it go If I postpone it, I will probably lose the push to do it by May-July, when the Masters admission process starts. Also, this specialization can be linked with the masters degree on the same topic given at the same campus. This program is one year long, and the masters two But having them both takes 2.5 years. So, not such a bad deal after all. And finally, because, after such a long time without being scholarized, I fear not having an easy time getting to grips with the discipline. I can commit to overworking myself for a year If it's too much for me, I'll just stay with that degree and give up. I expect to like it and continue... But it's also a safe bet :-) Now, there has to be a downside to picking up this path: Of course, my free time will be harshly reduced. I have reduced my Debian involvement in the last year, as I devoted a huge chunk of my time to teaching and book-writing... This year... We shall see what happens. I can for now only confirm what I have said publicly but inside our team only: I have requested to my peers and to our DPL to step down as a DebConf chair. I love organizing DebConf, but I don't want to be formally committed to a position I just cannot fulfill as I did when I started with it. As for package maintenance, by far most of my packges are team maintained, and those that are not are relatively easy to keep track of. And of course, I'll keep an eye on my keyring-maint duties as well Will even try to link that work with what I do at school! Anyway, lets see what comes now!

The decision on which init system to use has plagued Debian for a fairly long time now. We've had people talking about it as early as debconf11 in Banja Luka, and we still haven't got a decision. Instead, the decision has been going round in circles, in a typical bikeshed fashion. Originally, my opinion on the subject was "sysv-rc is good enough for everyone, and it's portable, which other alternatives aren't". By now, however, I've become convinced that the first part of that statement isn't true, and that as a result a switch of a default init system is indeed appropriate; that we should indeed switch to "something else", at least for our Linux ports. Once I came to that conclusion, my opinion on what, exactly, we should use turned out to be nonexistent. That is, I don't care. Anything will be fine. What is bothering me, though, is that things keep dragging on and on and on and on and on and on and on, ad infinitum. We should just pick one and be done with it, dammit. The fact that sysv-rc is replaced by "something else" does not mean people must stop using sysv init scripts. Even if the maintainer of some random package "foo" refuses to accept patches to support sysv-rc for his package, there's nothing stopping anyone from providing a package "sysv-support" containing init scripts (and nothing else) for sysv-rc. The same goes for a hypothetical upstart-support or systemd-support package, of course: if you want to continue living in the stone age and keep using sysv-rc forever, then there is nothing stopping you. Even if we decide to use something else for our Linux ports. Ideally the non-Linux ports would move to something more modern too; but there really really really isn't any reason why it would be a problem if they didn't. Can we stop painting the bikeshed now? If we don't, soon we'll have to call it "hunk of paint with some bikes inside" rather than "bikeshed". Thanks.

Contributor License Agreements ("CLAs") are a mechanism for an upstream software developer to insist that contributors grant the upstream developer some additional set of rights. These range in extent - some CLAs require that the contributor reassign their copyright over the contribution to the upstream developer, some merely provide the upstream developer with a grant of rights that aren't explicit in the software license (such as an explicit patent grant for a contribution licensed under a BSD-style license).

CLAs aren't new. FSF-copyrighted projects have been using copyright assignment since at least 1985 - in return, the FSF promise that the software will always be distributed under a copyleft-style license. For over a decade, Apache Software Foundation projects have required that contributors sign a CLA that allows them to retain copyright, but grants the ASF the right to relicense the work as it wishes. For the most part, this hasn't been terribly controversial.

So why do people object so much when Canonical do it? I've written about this in the context of Mir before, but it's worth expanding on the general case. The FSF's copyright assignment ensures that contributions to GPLed software will only be distributed under GPL-style licenses. The Apache CLA permits the ASF to relicense a contribution under a proprietary license, but the Apache license allows anyone to do that anyway. Going through Wikipedia's list of CLA users, the majority cover projects that are under BSD- or Apache-style licenses, with a couple of cases covering GPLed projects with a promise that any contributions will only be distributed under GPL-like licenses[1]. Either everyone can produce proprietary derivative works, or nobody can.

In contrast, Canonical ship software under the GPLv3 family of licenses (GPL, AGPL and LGPL) but require that contributors sign an agreement that permits Canonical to relicense their contributions under a proprietary license. This is a fundamentally different situation to almost all widely accepted CLAs, and it's disingenuous for Canonical to defend their CLA by pointing out the broad community uptake of, for instance, the Apache CLA.

Canonical could easily replace their CLA with one that removed this asymmetry - Project Harmony, the basis of Canonical's CLA, permits you to specify an "inbound equals outbound" agreement that prevents upstream from relicensing under a proprietary license[2]. Canonical's deliberate choice not to do so just strengthens the argument that the CLA is primarily about wanting to produce proprietary versions of software rather than wanting to strengthen their case in any copyright or patent disputes. It's unsurprising that people feel disinclined to contribute to projects under those circumstances, and it's difficult to understand why Canonical simultaneously insist on this hostile behaviour and bemoan the lack of community contribution to Canonical projects.

[1] The one major exception is the Digia/Qt project CLA, which covers an LGPLed work but makes it entirely clear that Digia will ship your contributions under proprietary licenses as well. At least they're honest.

[2] See the various options in section 2.1(d) here. Canonical chose option five. If they'd chosen option one instead, this wouldn't be a problem.

comments

While writing this post, I recognize that we might need a news-blog for Tanglu Since yesterday, we have an (installable!) Beta2 release of Tanglu! Compared to the previous snapshot, it comes with a huge amount of improvements and bugfixes:

• The systemd journal is now used for logging exclusively. You can install rsyslog to get the old behaviour back, and switch the journal to non-persistent mode. If you want to test the journal, type journalctl -b to get all messages from last boot. If we experience issues, we might revert to rsyslog for the final release.
• KDE received a new theming for KDM/KSplash, so we can now show a face-browser in KDM, which is considered more user-friendly. User faces are now loaded from the user s settings, instead of system directories by default. If you want to load a large user list (> 100) over network, you might not want this, but in that case you can easily revert the change.
• Tanglu does no longer include an MTA by default. If you want one, install one and configure it. On desktop systems, having an MTA makes no sense, especially when it is not properly configured (we follow Ubuntu/Fedora here).
• We ship with a functional live-installer now! You can install Tanglu easily to try it out. Unfortunately, the Debian installer is still not working, so you will have to boot into the live session to install Tanglu.
• GNOME is now working. GDM and g-s-d will no longer crash, and Mutter is able to utilize llvmpipe for rendering in a virtual machine. The Shell and various other GNOME parts were updated to GNOME 3.10
• We now ship the 1.14 version of Xorg now
• Linux 3.12 is now default

Known issues:

• People with ATi graphics cards might have trouble with installing the proprietary fglrx driver
• In some very rare cases, the live-cd won t auto-login. In that case, just reboot and try again, or use the password live to access the account.
• Some packages are not installable, there is a list compiled here, with detailed info why they are broken (they will be removed from the release if they are not fixed) Some transitions are still pending (causes broken/not-coinstallable packages)
• Users with a non-English locale will have an additional Public folder in their home-directory on KDE
• We don t support UEFI at time, please use the legacy mode to boot
• We just noticed that the GNOME images won t start the GNOME-Session for some reason. [UPDATE:21.01]: We published new live-cd images which have this issue fixed.

You can download the release from one of these mirrors: Mirror1 Mirror2 Mirror3 Additionally, there is now a release schedule for Tanglu Aequorea, and we will do a vote on the name for the next release soon (proposed names). As a note to Debian: systemd is working pretty well for Tanglu so far I am pretty happy with this Beta2 release, because Tanglu is shaping out to be the distribution we imagined in the beginning. Have fun! And as bonus, here are some images from Beta2:

I have just uploaded a newer version of L ve to Debian, 0.9.0. As usual, this version breaks compatibility with the API of previous versions. Literally: L VE 0.9.0 breaks compatibility with nearly every 0.8.0 game . It s a hard to fix situation from a package maintainer s point of view, at least until they agree on a stable API, hopefully in a 1.0 version sometime. L ve has been in Debian official repositories since 2008. As major changes, we can see that it s using SDL2 and LuaJIT now. Depending on where the bottlenecks were in some of the demos and games, the performance might have improved a lot. The improvements have been a lot, and the structure of the API is more consistent and clean. Congratulations to everyone that has made it possible. On the bitter part, well, most of the previous games and demos will most likely not work any more without some changes in the code. As we don t have any reverse dependencies in the archive (yet), this won t cause any severe problems. But, of course, Debian is not an isolated island, and people might need to execute some old code without being able to migrate it. I have prepared some packages for older versions of L ve that might make the situation more bearable for some, until code is migrated to the new API. These versions can be co-installed with the latest version in the archive (0.9.0). I m not sure if it will be needed, but if it was, I might consider putting previous 0.8 version in the official repositories. I would prefer not to do it, though, as that would make me the de facto maintainer of the upstream code, as L ve community is moving forwards with newer versions.

Now it is time for me to go silent for a while, and not talk about jobs, unemployment, or puppies. This past week has also been full of software releases. Some of the public ones include:

Lumail - My console mail client, with integrated lua scripting

After three months of slow work I've issued a new release today. This release features several bugfixes for dealing with malformed MIME messages, and similar fun. The core set of lua primitives hasn't changed very much for a good six months now, which means I guess rightly what kind of things would be useful.

Templer - My perl-based static-site generator.

This was recently updated to add two new plugins to the core:

• A redis plugin to allow you to set variables to values retrieved from redis.
• An RSS plugin to allow you to inline (remote) RSS feeds into your static HTML. Useful for building news-pages, etc.

Although there are a million static-site generators I still think mine has value, and I am consistently using it.

Months ago when I said "I'm writing a mail-client", all I need to do is handle three cases:

• Display a list of folders.
• Display index of messages.
• Display a single message.

Then some new things like "Compose", "Reply", "Forward", I remember somebody commented along the lines of "Yeah, but MIME will make you hate your life" I laughed. Now I know better. Still it works, it works well, and I'm glad I did it.

This is mostly a repost of my ubuntu-devel mail for a wider audience, but see below for some additions. I'd like to upgrade to GRUB 2.02 for Ubuntu 14.04; it's currently in beta. This represents a year and a half of upstream development, and contains many new features, which you can see in the NEWS file. Obviously I want to be very careful with substantial upgrades to the default boot loader. So, I've put this in trusty-proposed, and filed a blocking bug to ensure that it doesn't reach trusty proper until it's had a reasonable amount of manual testing. If you are already using trusty and have some time to try this out, it would be very helpful to me. I suggest that you only attempt this if you're comfortable driving apt-get directly and recovering from errors at that level, and if you're willing to spend time working with me on narrowing down any problems that arise. Please ensure that you have rescue media to hand before starting testing. The simplest way to upgrade is to enable trusty-proposed, upgrade ONLY packages whose names start with "grub" (e.g. use apt-get dist-upgrade to show the full list, say no to the upgrade, and then pass all the relevant package names to apt-get install), and then (very important!) disable trusty-proposed again. Provided that there were no errors in this process, you should be safe to reboot. If there were errors, you should be able to downgrade back to 2.00-22 (or 1.27+2.00-22 in the case of grub-efi-amd64-signed). Please report your experiences (positive and negative) with this upgrade in the tracking bug. I'm particularly interested in systems that are complex in any way: UEFI Secure Boot, non-trivial disk setups, manual configuration, that kind of thing. If any of the problems you see are also ones you saw with earlier versions of GRUB, please identify those clearly, as I want to prioritise handling regressions over anything else. I've assigned myself to that bug to ensure that messages to it are filtered directly into my inbox. I'll add a couple of things that weren't in my ubuntu-devel mail. Firstly, this is all in Debian experimental as well (I do all the work in Debian and sync it across, so the grub2 source package in Ubuntu is a verbatim copy of the one in Debian these days). There are some configuration differences applied at build time, but a large fraction of test cases will apply equally well to both. I don't have a definite schedule for pushing this into jessie yet - I only just finished getting 2.00 in place there, and the release schedule gives me a bit more time - but I certainly want to ship jessie with 2.02 or newer, and any test feedback would be welcome. It's probably best to just e-mail feedback to me directly for now, or to the pkg-grub-devel list. Secondly, a couple of news sites have picked this up and run it as "Canonical intends to ship Ubuntu 14.04 LTS with a beta version of GRUB". This isn't in fact my intent at all. I'm doing this now because I think GRUB 2.02 will be ready in non-beta form in time for Ubuntu 14.04, and indeed that putting it in our development release will help to stabilise it; I'm an upstream GRUB developer too and I find the exposure of widely-used packages very helpful in that context. It will certainly be much easier to upgrade to a beta now and a final release later than it would be to try to jump from 2.00 to 2.02 in a month or two's time. Even if there's some unforeseen delay and 2.02 isn't released in time, though, I think nearly three months of stabilisation is still plenty to yield a boot loader that I'm comfortable with shipping in an LTS. I've been backporting a lot of changes to 2.00 and even 1.99, and, as ever for an actively-developed codebase, it gets harder and harder over time (in particular, I've spent longer than I'd like hunting down and backporting fixes for non-512-byte sector disks). While I can still manage it, I don't want to be supporting 2.00 for five more years after upstream has moved on; I don't think that would be in anyone's best interests. And I definitely want some of the new features which aren't sensibly backportable, such as several of the new platforms (ARM, ARM64, Xen) and various networking improvements; I can imagine a number of our users being interested in things like optional signature verification of files GRUB reads from disk, improved Mac support, and the TrueCrypt ISO loader, just to name a few. This should be a much stronger base for five-year support.

I have just enrolled in a MOOC to learn about programming with Android. There are many tutorials and reports of people using Linux for the development of the apps out there, but, unfortunately, many of them are outdated or contain misleading information. You will find bazillion instructions out there telling you to install JDK6. (Even the instructions in the MOOCs say so, and Google says that it is the only supported version of Java). Even more than that, some of them will tell you that you should not use OpenJDK, but use Oracle's JDK instead. So, if you have a pure amd64 system and you try to install the x86-64 version of Google's Android Developer Tools (ADT), you are going to have a surprise: you will have all sorts of programs telling you that you have file not found, even though the required binaries are there. (This is, of course, besides install openjdk-7-jdk, if you wish to use the integrated Eclipse environment). Turns out that the so-called 64-bit ADT/SDK actually has many 32-bit executables(!) and they aren't statically linked. Inspecting them, I needed the following packages:

    gcc-4.8-base:i386
    libc6-i686:i386
    libc6:i386
    libgcc1:i386
    libstdc++6:i386
    zlib1g:i386

It is worth observing that:

• libstdc++6:i386 depends (transitively) on libc6:i386, libgcc1:i386, and gcc-4.8-base:i386.
• zlib1g:i386 depends (transitively) on libc6:i386, libgcc1:i386, and gcc-4.8-base:i386.

Given the above, and given that the version of the GCC libraries can vary a lot, it is perhaps better to simply do something like:

apt-get install libstdc++6:i386 zlib1g:i386

Although not necessary, since you are using an amd64 machine, your computer already supports many more recent instructions that will be used if you install the package libc6-i686:i386. To avoid the emulator from spitting out error messages related to libGL.so, install libgl1-mesa-dev (note: not i386 here). This way, you will be able to run the embedded QEMU with full KVM acceleration if you pass the options -qemu -m 1024 -enable-kvm on Eclipse's "Run > Run Configurations... > Target > Additional Emulator Command Line Options". After that, you are probably good to go.

As explained in more detail in my my last blog post, Rackspace is providing hosting for Debian Code Search. For those of you who don t know, Rackspace is a cloud company that provides (among other services) a public cloud based on OpenStack. That means you can easily (and programmatically, if you want) bring up virtual servers, block storage volumes, configure the network between them, etc. As part of my initial performance experiments, I was running debmirror to clone a full Debian source mirror, and I noticed this took about one hour. Given that the peak network and storage write rates I have observed in my benchmarks are much higher, I wondered why it took so long. About 43 GB (that s how big the Debian sid sources are) in 60 minutes means 12 MB/s download rate, which is a bit more than a sustained 100 MBit/s connection. Luckily, at least the Rackspace servers I looked at are connected with 1 GBit/s to the internet, so more should be possible. Note that these are the old Rackspace servers. There is a new generation of servers, which I have not yet tried, that apparently offer even higher performance. After a brief look at the debmirror source code, I concluded that it can only use a single mirror and downloads files sequentially. There is some obvious potential for improvement here, and the fact that I could come up with a proof of concept written in Go to determine the files to download in a couple of minutes encouraged me to spend my saturday on this problem :-). About 7 hours later, my prototype had gone through various iterations and the code could sustain about 115 MB/s incoming bandwidth for most of the time. Here is a screenshot of dstat measuring the performance: Another 3 hours later, the most obvious bugs were weeded out and the code successfully cloned an entire mirror for the first time. I verified the correctness by running debmirror afterwards, and no files were downloaded that had not actually arrived on the mirror in the meantime. Features

1. Parallel downloading from multiple mirrors.
   Of course, it is not guaranteed that the mirrors are consistent, but that is a solvable problem: whenever a non-200 HTTP response is encountered, the file is put back into the queue and gets rescheduled onto any mirror. Eventually, it will be rescheduled onto the mirror from which the sources.gz was downloaded, and that mirror has to have it.
2. HTTP keep-alive and pipelining.
   debmirror also does HTTP keep-alive (I read that in the source, did not verify it), but it downloads files sequentially, i.e. it downloads one file, then sends the request for the next file. This means that after one file was received, no data is being received until another round-trip finished. For a lot of small files (think of .dsc and .debian.tar.gz files or small orig tarballs) that really adds up.
   Therefore, I use HTTP pipelining: I send 99 requests and then just receive all the responses at full speed. The magic number 99 comes from the fact that this is the smallest common denominator that all mirrors I have tested allow. nginx by default allows 100 requests.
3. Request ordering.
   Instead of sending requests at random, I sort the download queue by size and make sure that the first request on each connection is for a big file. The intention is that TCP will adapt to that quickly and use a bigger window size as soon as possible instead of ramping up slower later on. Note that I have not actually measured the window sizes, so this is just a hunch.
   Another nice side-effect of the ordering is that the amount of data that is being sent through each connection is roughly equal. This means we don t waste TCP connections by sending 99 requests for small files only.
4. (Goroutines).
   I m a bit hesitant to even write about it, so let s make it quick: Perl (and thus debmirror) is typically single-threaded, whereas Goroutines scale nicely on multi-core systems. I don t think it makes a big difference on the machines I am running this on as they have only two cores and are not maxed out on CPU usage at all. But maybe this will be more important for saturating 10 GBit/s? :-)
5. (Mirror selection).
   It s not a feature of the code, but certainly relevant: of course I selected mirrors that are connected to the internet with at least a Gigabit link, serve files reasonably fast and have low latency to Rackspace s Chicago presence.
   Interestingly, while I used to think that universities have access to big uplinks and beefy machines, none of the mirrors that I use are located at a university. In fact, every mirror ending in .edu that I tried was really slow most of them providing something like 2-4 MB/s.

Results With the latest iteration of the code, I can clone an entire Debian source mirror with 43 GB of data in about 11.5 minutes, which corresponds to a 63 MB/s download rate:

GOMAXPROCS=20 ./dcs-debmirror -num_workers=20
68,24s user 384,90s system 66% cpu 11:25,97 total

It should be noted that the download rate is very low in the first couple of seconds since the sources.gz file is downloaded from a single mirror, then unpacked and analyzed. The peak download rate is about 115 MB/s (= 920 MBit/s) which is reasonably close at what you can achieve with a Gigabit link, I think. If the entire uplink was available to me at all time, the Rackspace hardware would be able to saturate that easily, both in terms of reading from the network and in terms of writing to block storage. I tested this on an SSD volume, but I see about 113 MB/s throughput with the internal hard disk, so I think that should work, too. There is another dstat screenshot of the final version (writing to disk). Perhaps even more interesting to some readers is the time for an incremental update:

GOMAXPROCS=20 ./dcs-debmirror -tcp_conns=20
2013/10/13 11:09:55 Downloading 307 files (447 MB total)
 
2013/10/13 11:10:04 All 307 files downloaded in 9.105605931s. Download rate is 49.186567 MB/s
4,91s user 3,99s system 67% cpu 13,205 total

The wall clock time is higher than the time reported by the code because the code does not count the time for downloading and parsing the sources.gz file. The entire program is about 400 lines (not SLOC) of Go code. It s part of the Debian Code Search source. If you re interested, you can read dcs-debmirror.go in your browser. Conclusions The outcome of this experiment is that I now know (and have shown!) that there are significantly more efficient ways of cloning a Debian mirror than what debmirror does. Furthermore, I have a good grasp on what kind of performance the Rackspace cloud offers and I am fairly happy with the numbers :-). My code is useful to me in context of Debian Code Search, but unless you need a sid-only source-only mirror, it will not be useful to you directly. Of course you can take the ideas that I implemented and implement them elsewhere personally, I don t plan to do that. If you have hardware, bandwidth and a use-case for 10 GBit/s+ mirroring, I d like to hear about it! :-)

Being a bit hyperactive, I have been involved in the organization of two events. I am the main organizer with Alexandre Delano of the Mini Debconf 2014 in Paris, January 18 & 19th. The (great) planning is available here:
https://france.debian.net/events/minidebconf2014/
Saturday morning presentations will be general public, the beginning of Saturday afternoon will be used by the Debian France association to vote the new status (1901 law and Debian Trusted Organization).
Sunday will be more focused on Debian itself.
During the week end, I will be talking about the Debile project, the finance of Debian France and be part of the round table on compiler selection for Debian.
The (mandatory) registration should be done on the Wiki or meetup.com In parallel, with Tobias Grosser, we organized the LLVM devroom track at FOSDEM (Bruxelles), February 2nd (Sunday).
The schedule is a mix between core developers, third party software using LLVM / Clang and academic users.
https://fosdem.org/2014/schedule/track/llvm/
I will be talking on how to become a LLVM contributor. Both events should be recorded.

Somalia's chief exports appear to be morally-ambiguous Salon articles about piracy and sophomoric evidence against libertarianism. However, it is the former topic that Captain Phillips concerns itself with, inspired by the hijacking of the Maersk Alabama container ship in 2009. What is truth? In the end, Captain Phillips does not rise above Pontius Pilate in providing an answer, but it certainly tries using more refined instruments than irony or leaden sarcasm. This motif pervades the film. Obviously, it is based on a "true story" and brings aboard that baggage, but it also permeates the plot in a much deeper sense. For example, Phillips and the US Navy lie almost compulsively to the pirates, whilst the pirates only really lie once (where they put Phillips in greater danger). Notice further that Phillips only starts to tell the truth when he thinks all hope is lost. These telling observations become even more fascinating when you realise that they must be based on the testimony of the, well, liars. Clearly, deception is a weapon to be monopolised and there are few limits on what good guys can or should lie about if they believe they can save lives. Futhermore, there is an utterly disarming epilogue where Phillips is being treated for shock by clinical efficient medical staff. Not only will this scuttle any "blanket around the shoulders" clich but is probably a highly accurate portrayal of what actually happens post-trauma. This echoes the kind of truth Werner Herzog aims for in his filmmaking as well his guilt-inducing duality between uncomfortable lingering and compulsive viewing. Lastly, a starter for a meta-discussion: can a film based on real-world events even be "spoilered"? Hearing headlines on the radio before you read your newspaper hardly robs you of a literary journey... Captain Phillips does have some quotidian problems. Firstly, the only tool for ratcheting up tension is for the Somalians to launch verbal broadsides at the Americans, with each compromise somehow escalating the situation. This technique is effective but well before the climatic rescue scene where it is really needed it has been subject to the most extreme diminishing returns. (I cannot be the first to notice the "Africans festooned with guns shouting incomphensively" trope I hope it is based on a Babel-esque mechanism of disorientation from miscommunication rather than anything more unsavoury.) Secondly, the US Navy acts like a teacher with an Ofsted inspector observing quietly from the corner of the classroom; far too well-behaved it suspends belief, with no post-kill gloating or even the tiniest of post-arrest congratulations. Whilst nobody wants to see the Navy overreact badly to other military branches getting all the glory, nobody wants to see a suspiciously bland recruitment vehicle either. Paradoxically, this hermetic treatment made me unduly fascinated by them as if they were part of some military "uncanny valley". Two quick observations:

• All US Somali interactions are recorded by a naval officer. No doubt a value-for-money defense against a USS Abu Ghraib, but knowing the plot is based on factual events, it was perhaps a little too Baudrillardian to ponder how the presence of the Navy's cameras in a scene actually lent weight to the film's version of events, crucially without me even knowing whether the parallel "real-life" footage is verifiable or not.
• The navigational computers not only seem to require lines to drawn repeatedly between points of interest, but the Maersk Alabama's arbitrary relabelling as MOTHERSHIP seems to imply that an officer could humourously rename a radar contact to something unbecoming of a 12A classification.

Finally, despite the title, the film is actually about two captains; the skillful liar Phillips and ... well, that's the real problem. Whilst Captain Muse is certainly no caricatured Hook, we are offered little depth beyond a "You're not just a fisherman" faux-revelation that leads nowhere. I was left inventing reasons for his akrasia so that he made any sense whatsoever. One could charitably argue that the film attempts to stay objective on Muse, but the inability for the film to take any obvious ethical stance actually seems to confuse and then compromise the narrative. What deeper truth is actually being revealed? Is this film or documentary? Worse still, the moral vacuum is invariably filled by the viewer's existing political outlook: are Somali pirates victims of circumstance who are forced into (alas, regrettable) swashbuckling adventures to pacify plank-threatening warlords? Or are they violent and dangerous criminals who habour an irrational resentment against the West, flimsily represented by material goods in shipping containers? Your improvised answer to this Rorschach test will always sit more haphazardly in the film than any pre-constructed treatment ever could. 6/10