Vijay Kumar gave an interesting TED talk about autonomous UAVs [1]. His research is based on helicopters with 4 sets of blades and his group has developed software to allow them to develop maps, fly in formation, and more. Hadiyah wrote an interesting post about networking at TED 2012 [2]. It seems that giving every delegate the opportunity to have their bio posted is a good conference feature that others could copy. Bruce Schneier wrote a good summary of the harm that post-911 airport security has caused [3]. Chris Neugebauer wrote an insightful post about the drinking culture in conferences, how it excludes people and distracts everyone from the educational purpose of the conference [4]. Matthew Wright wrote an informative article for Beyond Zero Emissions comparing current options for renewable power with the unproven plans for new nuclear and fossil fuel power plants [5]. The Free Universal Construction Kit is a set of design files to allow 3D printing of connectors between different types of construction kits (Lego, Fischer Technic, etc) [6]. Jay Bradner gave an interesting TED talk about the use of Open Source principles in cancer research [7]. He described his research into drugs which block cancer by converting certain types of cancer cell into normal cells and how he shared that research to allow the drugs to be developed for clinical use as fast as possible. Christopher Priest wrote an epic blog post roasting everyone currently associated with the Arthur C. Clarke awards, he took particular care to flame Charles Stross who celebrated The Prestige of such a great flaming by releasing a t-shirt [8]. For a while I ve been hoping that an author like Charles Stross would manage to make more money from t-shirt sales than from book sales, Charles is already publishing some of his work for free on the Internet and it would be good if he could publish it all for free. Erich Schubert wrote an interesting post about the utility and evolution of Favebook likes [9]. Richard Hartmann wrote an interesting summary of the problems with Google products that annoy him the most [10]. Sam Varghese wrote an insightful article about the political situation in China [11]. The part about the downside of allowing poorly educated people to vote seems to apply to the US as well. Sociological Images has an article about the increased rate of Autism diagnosis as social contagion [12]. People who get their children diagnosed encourage others with similar children to do the same. Vivek wrote a great little post about setting up WPA on Debian [13]. It was much easier than expected once I followed that post. Of course I probably could have read the documentation for ifupdown, but who reads docs when Google is available?

• [1] http://www.ted.com/talks/vijay_kumar_robots_that_fly_and_cooperate.html
• [2] http://hadiyahdotme.wordpress.com/2012/03/08/ted2012-and-why-conferences-will-never-be-the-same/
• [3] http://www.schneier.com/blog/archives/2012/03/harms_of_post-9.html
• [4] http://chris.neugebauer.id.au/2012/04/06/do-australian-tech-conferences-suffer-from-a-culture-of-exclusion-how-do-we-avoid-it/
• [5] http://beyondzeroemissions.org/media/newswire/clean-fossils-future-how-about-ready-renewables-120404
• [6] http://fffff.at/free-universal-construction-kit/
• [7] http://www.ted.com/talks/jay_bradner_open_source_cancer_research.html
• [8] http://www.antipope.org/charlie/blog-static/2012/03/buy-the-tee-shirt.html
• [9] http://www.vitavonni.de/blog/201204/2012041101-are-likes-still-worth-anything.html
• [10] http://richardhartmann.de/blog/posts/2012/04/13-woegle-is-me/
• [11] http://wildcard.gnubies.com/?p=1940
• [12] http://thesocietypages.org/socimages/2012/03/21/the-rise-of-autism-social-contagion-or-environmental-causation/
• [13] http://www.cyberciti.biz/faq/debian-linux-wpa-wpa2-wireless-wifi-networking/

Related posts:

1. Links March 2012 Washington s Blog has an informative summary of recent articles about...
2. Links April 2012 Karen Tse gave an interesting TED talk about how to...
3. Links February 2012 Sociological Images has an interesting article about the attempts to...

Hello I ve released Config::Model 2.018 which is now able to track the changes done with cme to your configuration data. Thw GUI now features a new menu entry File -> show unsaved change that will give you a list of the changes done since running the command or since the last save (whichever occured last). For instance: This feature is also available in non-graphic mode. As an example, let s refresh the lcdproc package:

$ cme fix dpkg
Fixing...
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Warning in 'control source Standards-Version' value '3.9.2': Current standards version is 3.9.3
Warning in 'copyright Format' value 'http://dep.debian.net/deps/dep5/': Format does not match the recommended URL for DEP-5
Changes:
- control source Standards-Version: '3.9.2' -> '3.9.3' # applied fix
- copyright Format: 'http://dep.debian.net/deps/dep5/' -> 'http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/' # applied fix

The Changes list can be reused to update the changelog, but the author does not dare injecting this directly in debian/changelog Let s check the resulting changes:

$ git diff
diff --git a/debian/control b/debian/control
index 11a5b8a..2b81192 100644
--- a/debian/control
+++ b/debian/control
@@ -20,7 +20,7 @@ Build-Depends: autoconf,
                libxosd-dev,
                pkg-config,
                texinfo
-Standards-Version: 3.9.2
+Standards-Version: 3.9.3
 Vcs-Browser: http://git.debian.org/?p=collab-maint/lcdproc.git;a=summary
 Vcs-Git: git://git.debian.org/collab-maint/lcdproc.git
 Homepage: http://www.lcdproc.org/
diff --git a/debian/copyright b/debian/copyright
index 6cea444..4b48619 100644
--- a/debian/copyright
+++ b/debian/copyright
@@ -1,4 +1,4 @@
-Format: http://dep.debian.net/deps/dep5/
+Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
 Upstream-Name: lcdproc
 Upstream-Contact: William W. Ferrell 
 Source: http://www.lcdproc.org/
domi@ylum:~/debian-dev/lcdproc-stuff/lcdproc$ 

Last but not least, this track and display changes features is also available for other models, like OpenSsh, lcdproc, multistrap All the best
Tagged: Config::Model, dpkg, Perl

In the past days, I looked at various memory allocators in a quest to improve performance in my multi-threaded test cases of a reference counting language runtime / object allocator (a fun project). It turns out the glibc s memory allocator is relatively slow, especially if you do loops that create one element and destroy one element at the same time (for example, map() on a list you no longer use after you passed it to map). To fix this problem, I considered various options. The first option was to add a thread-local cache around malloc(). So whenever we want to free() an object, we place it on a thread-local list instead, and if a malloc() request for an object of that size comes in, we reuse the old object. This fixes the problem with the lists, but experiments shown another problem with the glibc allocator: Allocating many objects without releasing others (let s say appending an element to a functional list). I started testing with tcmalloc instead, and noticed that it was several times faster (reducing run time from 6.75 seconds to 1.33 seconds (5 times faster)). As I do not want to depend on a C++ code base, I decided to write a simple allocator that allocates blocks of memory via mmap(), splits those into objects, and puts the objects on a local free list. That allocator performed faster than tcmalloc(), but was also just a simple test case, and not really useable, due to potential fragmentation problems (and because the code was statically linked in, causing thread-local storage to be considerably faster, as it does not need to call a function on every TLS access, but can rather go through a segment register). I then discovered jemalloc, and tried testing with this. It turned out that jemalloc was even faster than tcmalloc in all test cases, and also required about the same amount of memory as tcmalloc (and slightly more than the custom test code, as that combined reference counting with memory allocator metadata and thus had less meta data overhead). Using jemalloc, the list building test performs in 0.9X seconds now (instead of tcmalloc s 1.33 or glibc s 6.75 seconds), requires 0 major page faults (tcmalloc has 2), and uses 5361472 kb memory (tcmalloc uses 5290240, glibc requires 7866608). Given that jemalloc is written in C, I can only recommend it to everyone in need of a scalable memory allocator. Depending on your workload, it might require less memory than tcmalloc (at least that s the case at Facebook) and is most likely faster than tcmalloc. It also provides tcmalloc-compatible profiling facilities (as Facebook needed them). Furthermore, jemalloc is also the memory allocator used by FreeBSD, and is used by Mozilla projects, and on several Facebook projects, and should thus be relatively stable and useable. All tests were run with 4 threads, on a dual-core laptop with hyper-threading, using (e)glibc 2.13, tcmalloc 2.0, and jemalloc 3.0. The tests may not be representative of real world performance, and do not account for fragmentation. Should I try replacing Python s custom caching of malloc() calls with simply calling jemalloc, and run a few Python benchmarks? Anyone interested in that?
Filed under: General

Traffic jams suck; but they do give you a chance to think and blog. Truckers hams are telling me that four trucks are totalled and need to be removed by the THW. As the trucker next to me just broke out his gas grill and several people are joining him in a spontaneous BBQ, I don't expect to be moving any time soon. Kicking things along This is coming along nicely RAID 4 Unless you are forced to support legacy systems or playing RAIDemon and gotta try them all, there is no valid use case for RAID 4. So doing so is a patently bad idea. RAID 5 is always the better option as you are not relying on a single disk which contains the checksums, but use striping to distribute them. RAID 4 has a chance of 1/n that a failed disk will force all remaining disks to read all respective sectors and then the controller/CPU to XOR everything. There is a reason why RAID 3 was replaced with RAID 4; just as there's a reason why RAID 4 was replaced by RAID 5. Don't use RAID 4. Seriously. Tom Tom vs OSM In case you have been following the news in the GPS circles, you will have noticed that TomTom pretty gave up on innovating and resigned themselves to dying a slow death. Well that and to start a nice amount of FUD. While they have been rebuked pretty thoroughly already, I wanted to add my own POV. There is a kernel of truth in what TomTom said inasmuch they state that pedestrian paths are better than car paths. Of course, while they got the relative quality right, they are wrong about the absolute quality of the data. They claim pedestrian is OK and motorized is bad. In my experience, maps for motorized traffic are good while pedestrian tracks are mind-boggingly great. These days, most people (and probably all who read this) will be at least somewhat familiar with the quality of OSM in the random cities they visit over the year. The common verdict is usually "very nice". But that's not where OSM shines. It's in the lesser-known places where you don't expect too many tech-minded people to come along where it starts to really impress me. While I found a few stubs on a dead-end street in Akureyri which were not mapped, every foot trail I ever walked along in Iceland has been mapped. ski-doo tracks on Svalbard? Check. Backyard in Novosibirsk? Check, as well. And we are talking "packed earth track in between garages opening up into a 'playground' where a man and a pregnant women are passing a bottle back and forth while smoking and watching over their two kids who play next to a middle-aged man who is lying on the ground wasted, using a rock as a pillow all the while an extremely blingy Mercedes with two muscle-packed thugs is cruising by, all in the time it took us to walk through". Mapped. Horse tracks and hiking trails in Mongolia? Yep. Hutongs in Beijing including alleys so narrow you have to walk single file? Of course. Richard Fairhurst is right that the plural of anecdote is not data, but OSM is good. TomTom is afraid for a reason. As an aside, I find the picture used in TomTom's newsletter extremely funny; I am sure they didn't intend this, but hey. Windows 100-300 MiB for any random driver installation package is par for the course. Be it GFX, audio, or network card. If you plug in a different USB mouse, Windows 7 will take several seconds while "setting up" a standard USB HID device. And hitting ctrl-alt-del in the login screen or while logged in does not give you a quick way to shut down or reboot the machine. As I am breaking out the Windows installation every month to install updates and every six months to actually use this does not affect me a lot. Still, it's amazing that so many people are putting up with this. UEFI It's nice to see that Matthew Garrett is still going strong. I am not sure if this is the best way, but it seems to be a workable solution while in between a rock and a hard place. Not sure about the requirements of GPLv3 with regards to the keys Fedora will use to sign GRUB 2 with. From what I remember without rereading the license, as long as Fedora is not selling a system on which you can not disable secure boot they should be in the clear. IANAL, IIRC, etc. Wrapping up And I have not moved a single meter, other than making way to a police motorbike, in over an hour. Pity I don't have anything to BBQ with me; smells delicious over there.

Eight days ago, I uploaded atop 1.26-1 to DELAYED/8, listing me as new maintainer. This means that the package has in the mean time appeared in unstable, and I hope that it ll swiftly migrate to testing.

Edit 16:17EDT 31/5/12: Clarification of who gets the $99
Edit 02:10EDT 01/6/12: Clarification that it's a one-off payment

(Brief disclaimer - while I work for Red Hat, I'm only going to be talking about Fedora here. Anything written below represents only my opinions and my work on Fedora, not Red Hat's opinions or future plans)

Fedora 17 was released this week. It's both useful and free, and serves as a welcome addition to any family gathering. Do give it a go. But it's also noteworthy for another reason - it's the last Fedora release in the pre-UEFI secure boot era. Fedora 18 will be released at around the same time as Windows 8, and as previously discussed all Windows 8 hardware will be shipping with secure boot enabled by default. While Microsoft have modified their original position and all x86 Windows machines will be required to have a firmware option to disable this or to permit users to enrol their own keys, it's not really an option to force all our users to play with hard to find firmware settings before they can run Fedora. We've been working on a plan for dealing with this. It's not ideal, but of all the approaches we've examined we feel that this one offers the best balance between letting users install Fedora while still permitting user freedom.

Getting the machine booted
Most hardware you'll be able to buy towards the end of the year will be Windows 8 certified. That means that it'll be carrying a set of secure boot keys, and if it comes with Windows 8 pre-installed then secure boot will be enabled by default. This set of keys isn't absolutely fixed and will probably vary between manufacturers, but anything with a Windows logo will carry the Microsoft key[1].

We explored the possibility of producing a Fedora key and encouraging hardware vendors to incorporate it, but turned it down for a couple of reasons. First, while we had a surprisingly positive response from the vendors, there was no realistic chance that we could get all of them to carry it. That would mean going back to the bad old days of scouring compatibility lists before buying hardware, and that's fundamentally user-hostile. Secondly, it would put Fedora in a privileged position. As one of the larger distributions, we have more opportunity to talk to hardware manufacturers than most distributions do. Systems with a Fedora key would boot Fedora fine, but would they boot Mandriva? Arch? Mint? Mepis? Adopting a distribution-specific key and encouraging hardware companies to adopt it would have been hostile to other distributions. We want to compete on merit, not because we have better links to OEMs.

An alternative was producing some sort of overall Linux key. It turns out that this is also difficult, since it would mean finding an entity who was willing to take responsibility for managing signing or key distribution. That means having the ability to keep the root key absolutely secure and perform adequate validation of people asking for signing. That's expensive. Like millions of dollars expensive. It would also take a lot of time to set up, and that's not really time we had. And, finally, nobody was jumping at the opportunity to volunteer. So no generic Linux key.

The last option wasn't hugely attractive, but is probably the least worst. Microsoft will be offering signing services through their sysdev portal. It's not entirely free (there's a one-off $99 fee to gain access edit: The $99 goes to Verisign, not Microsoft - further edit: once paid you can sign as many binaries as you want), but it's cheaper than any realistic alternative would have been. It ensures compatibility with as wide a range of hardware as possible and it avoids Fedora having any special privileges over other Linux distributions. If there are better options then we haven't found them. So, in all probability, this is the approach we'll take. Our first stage bootloader will be signed with a Microsoft key.

Bootloaders
We've decided to take a multi-layer approach to our signing for a fairly simple reason. Signing through the Microsoft signing service is a manual process, and that's a pain. We don't want to have bootloader updates delayed because someone needs to find a copy of Internet Explorer and a smartcard and build packages by hand. Instead we're writing a very simple bootloader[2]. This will do nothing other than load a real bootloader (grub 2), validate that it's signed with a Fedora signing key and then execute it. Using the Fedora signing key there means that we can build grub updates in our existing build infrastructure and sign them ourselves. The first stage bootloader should change very rarely, and we don't envisage updating it more than once per release cycle. It shouldn't be much of a burden on release management.

What about grub? We've already switched Fedora 18 over to using grub 2 by default on EFI systems, but it still needs some work before it's ready for secure boot. The first thing is that we'll be disabling the module loading. Right now you can load arbitrary code into grub 2 at runtime, and that defeats the point of secure boot. So that'll be disabled. Next we'll be adding support for verifying that the kernel it's about to boot is signed with a trusted key. And finally we'll be sanitising the kernel command line to avoid certain bits of functionality that would permit an attacker to cause even a signed kernel to launch arbitrary code. These restrictions will all vanish if secure boot is disabled.

Kernel
Secure boot is built on the idea that all code that can touch the hardware directly is trusted, and any untrusted code must go through the trusted code. This can be circumvented if users can execute arbitrary code in the kernel. So, we'll be moving to requiring signed kernel modules and locking down certain aspects of kernel functionality. The most obvious example is that it won't be possible to access PCI regions directly from userspace, which means all graphics cards will need kernel drivers. Userspace modesetting will be a thing of the past. Again, disabling secure boot will disable these restrictions.

Signed modules are obviously troubling from a user perspective. We'll be signing all the drivers that we ship, but what about out of tree drivers? We don't have a good answer for that yet. As before, we don't want any kind of solution that works for us but doesn't work for other distributions. Fedora-only or Ubuntu-only drivers are the last thing anyone wants, and this really needs to be handled in a cross-distribution way.

Wait signed what
Secure boot is designed to protect against malware code running before the operating system. This isn't a hypothetical threat. Pre-boot malware exists in the wild, and some of it is nastier than you expect. So obviously bootloaders need to be signed, since otherwise you'd just replace the signed bootloader with an unsigned one that installed malware and booted your OS.

But what about the kernel? The kernel is just code. If I take a signed Linux bootloader and then use it to boot something that looks like an unsigned Linux kernel, I've instead potentially just booted a piece of malware. And if that malware can attack Windows then the signed Linux bootloader is no longer just a signed Linux bootloader, it's a signed Windows malware launcher and that's the kind of thing that results in that bootloader being added to the list of blacklisted binaries and suddenly your signed Linux bootloader isn't even a signed Linux bootloader. So kernels need to be signed.

And modules? Again, modules are just code. It's a little trickier, but if your signed kernel loads an unsigned module then that unsigned module can set up a fake UEFI environment and chain into a compromised OS bootloader. Now the attacker just has to include a signed kernel and a minimal initramfs that loads their malware module. It'd slow down boot by a couple of seconds, but other than that it'd be undetectable. X? If you can access the registers on a GPU then you can get the GPU to DMA over the kernel and execute arbitrary code. Trickier again, but still achievable - and if you've locked down every other avenue of attack, it's even attractive.

If we produce signed code that can be used to attack other operating systems then those other operating systems are justified in blacklisting us. That doesn't seem like a good outcome.

Customisation
A lot of our users want to build their own kernels. Some even want to build their own distributions. Signing our bootloader and kernel is an impediment to that. We'll be providing all the tools we use for signing our binaries, but for obvious reasons we can't hand out our keys. There's three approaches here. The first is for a user to generate their own key and enrol it in their system firmware. We'll trust anything that's signed with a key that's present in the firmware. The second is to rebuild the shim loader with their own key installed and then pay $99 and sign that with Microsoft. That means that they'll be able to give copies to anyone else and let them install it without any fiddling. The third is to just disable secure boot entirely, at which point the machine should return to granting the same set of freedoms as it currently does.

But I don't trust Microsoft
A system in custom mode should allow you to delete all existing keys and replace them with your own. After that it's just a matter of re-signing the Fedora bootloader (like I said, we'll be providing tools and documentation for that) and you'll have a computer that will boot Fedora but which will refuse to boot any Microsoft code. It may be a little more awkward for desktops because you may have to handle the Microsoft-signed UEFI drivers on your graphics and network cards, but this is also solvable. I'm looking at ways to implement a tool to allow you to automatically whitelist the installed drivers. Barring firmware backdoors, it's possible to configure secure boot such that your computer will only run software you trust. Freedom means being allowed to run the software you want to run, but it also means being able to choose the software you don't want to run.

You've sold out
We've been working on this for months. This isn't an attractive solution, but it is a workable one. We came to the conclusion that every other approach was unworkable. The cause of free software isn't furthered by making it difficult or impossible for unskilled users to run Linux, and while this approach does have its downsides it does also avoid us ending up where we were in the 90s. Users will retain the freedom to run modified software and we wouldn't have accepted any solution that made that impossible.

But is this a compromise? Of course. There's already inequalities between Fedora and users - trademarks prevent the distribution of the Fedora artwork with modified distributions, and much of the Fedora infrastructure is licensed such that some people have more power than others. This adds to that inequality. It's not the ideal outcome for anyone, and I'm genuinely sorry that we weren't able to come up with a solution that was better. This isn't as bad as I feared it would be, but nor is it as good as I hoped it would be.

What about ARM
Microsoft's certification requirements for ARM machines forbid vendors from offering the ability to disable secure boot or enrol user keys. While we could support secure boot in the same way as we plan to on x86, it would prevent users from running modified software unless they paid money for a signing key. We don't find that acceptable and so have no plans to support it.

Thankfully this shouldn't be anywhere near as much of a problem as it would be in the x86 world. Microsoft have far less influence over the ARM market, and the only machines affected by this will be the ones explicitly designed to support Windows. If you want to run Linux on ARM then there'll be no shortage of hardware available to you.

Is this all set in stone?
No. We've spent some time thinking about all of this and are happy that we can implement it in the Fedora 18 timescale, but there's always the possibility that we've missed something or that a new idea will come up. If we can increase user freedom without making awful compromises somewhere else then we'll do it.

[1] In fact, chances are that everything will carry the Microsoft key. Secure boot requires that UEFI drivers also be signed. The signing format only permits a single signature per binary. For compatibility, approximately all add-on hardware shipped will be signed with Microsoft's key, and that means that all system vendors have to recognise Microsoft's key in order to permit that hardware to run on their systems.
[2] Current source is here. It relies on a port of the UEFI crypto library and OpenSSL that I have building with some handholding, and which I'll upload as soon as possible.

comments

I ve expanded a bit on my idea (that is, the sbuilder wrapper to help aid with managing more then one sbuild chroot), and it got out of hand very quickly. It s gone from one small shell script to a set of 8, and growing. Quickly. Until I spin these off into their own repo, they ll be in my dot-bin Before getting too far, please relize this is only how I m using sbuild. There are many other (more) sane configurations that you can (and should) use. This was not made to be useful for anyone other then me :) I ve added the ability to use unionfs-fuse, set up a way to login interactively and a script to create new chroots without having them register as a global schroot. Firstly, forgive me for my sins. This is all a huge hack, but it s use isn t the worst thing in the world. As of the time of this writing, I now have a gcc and clang chroot for each of stable, testing and unstable. Now the big question how do you use this. Rather, this is more of a how do I use sbuilder :) Firstly, I ve created a file for each schroot set that I m using. Here s my /etc/schroot/chroot.d/sid.

[sid]
type=directory
description=Debian sid/amd64 autobuilder
directory=/var/sbuild/sid
groups=root,sbuild
root-groups=root,sbuild
script-config=sbuild/config

It s very similar to the one that sbuilder-createchroot creates. If you re just looking for aufs / unionfs support, keep in mind schroot has some great support. check man 5 schroot.conf. Next, create a dist.d directory in /var/sbuild such as /var/sbuild/sid.d. In the future, this might get automagically created by the sb-create script. For now, it just expects that exists. The arguments for all the sb* commands is always in the order of variant, dist. variant defaults to generic , which I ve set as an alias to the generic chroot, and the dist defaults to sid . At some point, I ll change the default to be the last UNRELEASED entry in the changelog. So, to create a new chroot to become your clang chroot, you may choose to invoke it as sb-create clang, or more explicitly sb-create clang sid. To create a new chroot to become your stable / gcc chroot, you may create it with sb-create gcc stable. After you have a chroot, you can start to use the sb or sbo scripts. sbo is just a wrapper around the sb script that sets an envvar to tell the sb-prehook and sb-posthook to set up the unionfs overlay. The unionfs overlay routine replaces the dist symbolic link (such as /var/sbuild/testing) with a folder, and mounts the real chroot (such as /var/sbuild/testing.d/testing-gcc) under the unionfs divert point (which is something like /var/sbuild/testing.overlay.d/testing-gcc). This lets us make sure the chroot is never affected by a routine build, even if it s designed to trash a chroot (unless, of course, it uses any number of ways to get outside the chroot and trash the pristine directory this is to prevent mistakes, not malicious scripts). By the way, for those interested, there s a small issue with what I m describing it s very subtle. Basically, there s a few assumptions made by how schroot operates, and handles sessions . Since it ll keep stuff mounted if the session is still active, it makes purging all this a real pain. The magic behind the scripts is in the sb-endsessions routine which will properly end all the schroot sessions for that dist, and since we have our own high-level locking system, we shouldn t have a situation where this kills off another build. The code (which might have issues) looks like:

SESSIONS= schroot --all-sessions -l   grep "^session:$DIST-" 
for session in $SESSIONS; do
    echo "W: Ending session $session"
    schroot -e -c $session
done

By backing of schroot s mounts, we can do our mounting / removing without worrying about klobbering schroot. Now, there s also the case of wanting to interactively log into a chroot. For that, there s sb-login and sb-maint, both take the usual variant / dist arguments. sb-login logs in as the current user, and sb-maint logs in as the root user. Eventually, I ll be adding some scripts to upgrade all the build chroots for a dist at once.

This is the latest upstream version and will, barring any unforseen circumstances, be the version in wheezy, the upcoming stable release of Debian GNU/Linux. In order to ensure it is the high quality our users have come to expect from Debian stable, I need your help. First and foremost, please test the hell out of it. Upgrades during a stable release are in general not allowed unless the problem is security related so we need to shake all the bugs out now. I am particularly interested in knowing if there are any issues with upgrades from squeeze. I waved my hands ineffectually at the bug list but it needs more attention. Could some kind person go through them and let me know which ones are no longer valid, need further attention etc.? I attempted to add hardening flags but lintian still complains a lot. I think the warnings are probably false positives but I'm not a 100% sure so I would appreciate some advice from those who know more about such things. Those are the issues that come to mind at the moment but feel free to let me know if there's something I've missed.

I recently got a question by email about the lecture I presented at FOSDEM, and by that mail I discovered that the video of the lecture was uploaded to YouTube (FOSDEM 2012 Debian packaging for beginners). While I thank the person who did the upload, he/she did not keep the right license for the video which is CC-BY-SA 3.0 (as noted at FOSDEM archive). You can also grab the webm file directly. Well, the next task will be to listen to myself in order to improve for FOSDEM 2013 See you there.
Filed under: Debian GNU/Linux, FOSDEM

MLMMJ stands for Mailing List Manager Made Joyful. To me, it s the best mailing list manager available. Not only its written in C (no slow interpreted language here ), but also it is easy and very convenient to setup. No hack is needed to make it run on a multi-domain environment, and no need to run a special subdomain for your lists (yes, I m thinking about you, silly Mailman ). If you didn t know, MLMMJ is used to handle relatively high traffic lists: all mailing lists for Gentoo are using MLMMJ for example. MLMMJ 1.2.18.0 has just been release upstream, few days ago (on the 29 of may). It s with a great pleasure that I packaged (and uploaded to SID) the latest version, and wrote the changelog, where I wrote that 6 Debian specific patches (out of 8) could be removed! One of the funny changes are the renaming of the mlmmj-recieve into mlmmj-receive as it always should have been (with a symlink to keep backward compatibility), and the removal of the .sh extension to the mlmmj-make-ml (which was patched in Debian to be kept policy compliant: that s one of the removed patches ). Even if I did a few quick functional tests before uploading, I d be happy to get some feedback before Wheezy is out, so please test and eventually bug-report!

Hello!

I'm Sorina and this summer I will be working on Debian Installer network setup during Google Summer of Code. I am a second year bachelor's student at Politehnica University of Bucharest, I am studying Computer Science. I very much enjoy what I am doing even if sometimes I feel that the amount of stuff I want/have to learn/do/work on is infinite and I don't really know where to start.

I am at the beginning of the project, I believe now I am accustomed to the way things work and I am waiting forward to finishing the exam period so that I could really start working.

The hardest part for me in what I've done so far is testing what I am doing. It took me a while to figure out at least a part of how the installer functions and to solve the problems that occurred while building. Luckily, there is documentation and internet and even more important, I had help and support from my mentors.

So far I have only written two patches for netcfg, for bugs #610752 and #636211. For the first one I had a version for a while now, but I was unable to test it (unfortunately, it's likely that you will hear this a lot in the future). The idea was to make the link detection timeout preseedable and for this I have added two new templates, asking the user to provide a value for that matter.

The second one involved a missing "Go Back" button. This seemed to be the result of not letting debconf know that the program is capable of handling the user pressing a back button. That could be solved using deb_capb.

As I said before, testing was my main problem. Eventually, I built the image, wrote it to an USB and rebooted my laptop. This served its purpose, since I had access to anything I needed such as the wireless interface and physical access to the network cable. Running it through a virtual machine did not help much in this case, because I was unable to provide those.

Another thing I considered to be hard was blog posting, but now that I have just finished my first post it seems that I overreacted.

That's all for now, have a nice day!

I'm participating on organizing this year's openSUSE conference and as we've finally finalized place and time and CFP is open, it's time to share some information. This will be special - there is not a single conference, but four of them are sharing same space and the motto - Bootstrapping awesome!!!. The 4th openSUSE conference will be held together with LinuxDays, Czech conference following tradition of canceled LinuxExpo, what should be the biggest free software event in Czech republic. In addition there will be also 12th SUSE Labs conference (so you can meet quite a lot of kernel hackers and other strange guys) and first Gentoo mini summit (the website is empty so far). All that will happen on weekend from 20th October to 21st October, the SUSE conferences will then continue for 22nd and 23rd October. I believe it will be great mixture of conferences and I hope to meet lot of people there.

Filed under: English Gentoo Suse 0 comments Flattr this!

I like to think that archive rebuilds play an important role in Debian Quality Assurance and Release Management efforts. By trying to rebuild every Debian package from source, one can identify packages that do not build anymore due to changes in other packages (compilers, interpreters, libraries, ). It is also a good way to stress-test all packages that are involved in building other packages. Since 2007, I had been running Debian archive rebuilds on the Grid 5000 testbed, a research infrastructure for performing experiments on distributed systems HPC/Grid/Cloud/P2P. I filed more than 6000 release-critical bugs in the process. Late last year, Amazon kindly offered us a grant to allow us to run such QA tests on Amazon Web Services. With S bastien Badia, we ported the rebuild infrastructure to AWS (scripts), and several rebuilds have already been carried out on AWS. On the technical level, 50 to 100 EC2 spot instances are started, and then controlled from a master instance using SSH. On build instances, a classic sbuild setup is used. Logs are retrieved to the master node after rebuilds, and build instances are simply shut down when there are no more tasks to process. Several tasks are processed simultaneously on each instance, and when they fail, they are retried again with no other concurrent build on the same instance, to eliminate random failures caused by load or timing issues. All the scripts are designed to support other kind of QA tests, not just rebuilds. Moving to Amazon Web Services will facilitate sharing the human workload of doing those tests. It is now possible for developers interested in custom tests to do them themselves (hint hint).

When fixing or modifying a package I at times change the installed files directly. However, once the changes are okay and it is time to prepare a diff, having to keep track and manually diff'ing the installed files is time consuming.

That's when ldebdiff does its thing: it runs diff against the original files of a given package and the ones installed on a system.
For example:

$ ldebdiff acpi-fakekey
--- unpacked/etc/init.d/acpi-fakekey    2012-04-05 05:14:21.000000000 -0500
+++ /etc/init.d/acpi-fakekey    2012-04-12 12:10:55.000000000 -0500
@@ -4,8 +4,8 @@
 
 ### BEGIN INIT INFO
 # Provides:          acpi-fakekey
-# Required-Start:    $local_fs $remote_fs
-# Required-Stop:     $local_fs $remote_fs
+# Required-Start:    $remote_fs
+# Required-Stop:     $remote_fs
 # Default-Start:     2 3 4 5
 # Default-Stop:      
 # Short-Description: Start acpi_fakekey daemon

($remote_fs implies $local_fs)

• The "Save the baltic languages" week..:-)
• Latvian translations completed for level 1. Latvian won't be deactivated!
• Also progress in levels 2 and 3 for Latvian
• Lithuanian progress for level 1. Language won't be deactivated!
• Estonian was already safe..:-)

Status for D-I level 1 (core D-I files):

• 51 languages 100%: ar ast be bg bn bs ca cs da de el eo es et eu fa fi fr gu he hi id it ja kk km kn ko lo lv mr nb nl pa pl pt pt_BR ru si sk sl sr sv te th tr ug uk vi zh_CN zh_TW
• 1 languages 99%: ta
• 2 languages 97%: ga hu
• 5 languages 95%: dz is lt mk ro
• 1 language 92%: ml
• others are 90% or below

Status for D-I level 2 (packages that have localized material that may appear during default installs, such as iso-codes, tasksel, etc.):

• 31 languages 100%: ast be bg ca cs da de eo es fa fr gu he id is it ja kk km mr nl pl pt ru si sk sl sv tr uk zh_CN
• 8 languages 99%: eu fi hr lv nb th vi zh_TW
• 3 languages 98%: dz pt_BR ro
• 1 language 97%: ta
• 1 language 96%: ug
• 3 languages 94%: ar el gl
• 1 language 92%: sr
• 5 languages 92%: bn bs hu ko ne
• others are 90% or below

Status for D-I level 3 (packages that have localized material that may appear during non-default installs, such as win32-loader)

• 38 languages 100%: ast be bg bs ca cs da de el eo es fa fi fr ga gl he hr id is it ja kk km nb nl pl pt ru sk sl sr sv th tr vi zh_CN zh_TW
• 3 languages 98%: hu ug uk
• 1 language 95%: lv
• others are 90% or below

Full 100% completeness (hall of fame) for 26 languages: Asturian, Belarusian, Bulgarian, Catalan, Czech, Danish, German, Esperanto, Spanish, Persian, French, Hebrew, Indonesian, Italian, Japanese, Kazakh, Khmer, Dutch, Polish, Portuguese, Russian, Slovak, Slovenian, Swedish, Turkish, Simplified Chinese

• Latvian is now completed.
• Lithuanian is now completed.
• I got an offer for Welsh

At this very moment, it means that I would deactivate 14 languages:

• Amharic (5f17u in sublevel 2)
• Welsh (39f82u, 32f47u)
• Galician (7f10u, 6f9u)
• Croatian (16f19u, 6f9u)
• Georgian (17f47u, 10f21u)
• Kurdish (12f25u, 6f22u)
• Malayalam (5u in sublevel 2 only!)
• Nepali (16f54u, 10f21u)
• Norwegian Nynorsk (13f34u, 13f23u)
• Albanian (19f50u, 13f23u)
• Tamil (2f1u, 1f)
• Tagalog (18f54u, 24f39u)

I ve recently found myself in need of more then one sbuild chroot, each one set up slightly differently then the next. For a while, providing different behavior with unstable and sid was all well and good, but I figure, like any good computer scientist, I should solve the general problem. As a result, I ve written an sbuild wrapper to help maintain sanity. It uses a series of symbolic links to hide everything out of the way, and some bash hackery. Basically, I ve set up a /var/sbuild directory, full of a few different files. For each distribution, there is one folder, one symbolic link and up to two other files. There s the dist.d file (such as /var/sbuild/sid.d) that contains all the variants , dist symbolic link (such as /var/sbuild/sid) which points to the currently active chroot, and up to two lock files. The script will default to sid-default (that is to say, exactly, /var/sbuild/sid.d/sid-default - which is a symbolc link to the default chroot), but may take two arguments, in the order of variant, then dist. Currently, I have two chroots, one for gcc, and one for clang (set up like Sylvestre used for cland.d.n) but I may add some more with different setups. Currently the behavior is to have the second wait kindly if you kick off two sbuilds at once, and fail out a third. There are a bunch of bugs, and I m not sure that this will work very well, but it s working. The next step is to combine this with unionfs overlays to help protect accidentally trashing the chroots. You can find the script on my GitHub, pasted below.

#!/bin/bash
DIST="sid"
VARIANT="default"
SBUILD=/var/sbuild
if [ "x$1" != "x" ]; then
    VARIANT=$1
fi
if [ "x$2" != "x" ]; then
    DIST=$2
fi
if [ -e $SBUILD/$DIST.lock ]; then
    PID= cat $SBUILD/$DIST.lock 
    echo "W: We *must* wait for the finish of PID $PID"
    echo "W: Blocking execution until this file is removed."
    if [ -e $SBUILD/$DIST.wait ]; then
        echo "E: Someone else is waiting. Bombing out."
        exit 2
    fi
    echo $$ > $SBUILD/$DIST.wait
    while [ -e $SBUILD/$DIST.lock ]; do
        sleep 1
    done
    echo "I: Dropping out of lock. Purging wait."
    rm $SBUILD/$DIST.wait
fi
echo "I: Aquiring lock"
echo $$ > $SBUILD/$DIST.lock
CURRENT=$(basename  ls -xl $SBUILD/$DIST   awk ' print $11 ' )
# We set up the name like:
#   dist-variant
# such as "sid-clang" is cool.
#echo "I: We're currently $CURRENT."
if [ "x$CURRENT" == "x$DIST-$VARIANT" ]; then
    echo "I: This matches the desired setup. No action."
else
    if [ -e $SBUILD/$DIST.d/$DIST-$VARIANT ]; then
        echo "I: Re-setting $DIST symlink."
        rm $SBUILD/$DIST
        ln -s $SBUILD/$DIST.d/$DIST-$VARIANT $SBUILD/$DIST
    else
        echo "E: No such dist! Looking in $SBUILD/$DIST.d/$DIST-$VARIANT."
        exit 1
    fi
fi
echo "I: "
echo "I: Dist:    $DIST"
echo "I: Variant: $VARIANT"
echo "I: "
# echo "I: Before we go farther, we're updating."
# sbuild-update -d $DIST --upgrade
sbuild -d $DIST
echo "I: Releasing lock"
rm $SBUILD/$DIST.lock

I ve decided to start putting together packages to rebuild the bare bones install I use for most of my systems, since rebuilding everything by hand is starting to get annoying. I call it hairy candy. Because my setup relies on some chem interesting chem hacks, none of this is fit in any way for the archive. If the vim team was to catch wind of what I was doing, they d be none too happy :) While these packages work great for me, I can t stress enough how badly they can break something. Using Zack s guide, I was able to get an archive up and running, no problem. Taking his advice, I even got signing working. Simple. mini-dinstall rocks a nice alternative to a PPA, for sure. At least, for one arch :) Now, the one spin I took on Zack s work was to add a landing page a listing of what s in the archive. I did this using a bit of Python & Jinja2. The script follows.

#!/usr/bin/env python
# Copyright (c) Paul Tagliamonte, under the terms of the Expat license.
from jinja2 import Template
import sys
def parse_packages(fd):
    ret = []
    cur =  
    key = None
    for line in fd.readlines():
    if line.strip() == "":
        ret.append(cur)
        cur =  
        continue
    if line[0] == " ":
        cur[key] += "\n" + line.strip()
    else:
        key, val = line.split(":", 1)
        cur[key] = val.strip()
    return ret
pkgs = parse_packages(open(sys.argv[1], 'r'))
context =  
    "pkgs": pkgs
 
t = Template(open(sys.argv[2], 'r').read())
print t.render(**context)

Invocation looks something like:

render.py $ARCHIVE/debian/wicked/Packages $DATA/index.html

I kick this script off in the singing script, since mini-dinstall doesn t have a exit hook. Works great, thanks everyone!

i dislike having to run Windows as much as the next free software developer, but like many sysadmins, i am occasionally asked to maintain some legacy systems. A nice way to keep these systems available (while not having to physically maintain them) is to put them in a virtual sandbox using a tool like kvm. While kvm makes it relatively straightforward to install WinXP from a CD (as long as you have the proper licensing key), it is more challenging to transition a pre-existing hardware windows XP installation into a virtual instance, due to Windows only wanting to boot to ide chipsets that it remembers being installed to. In particular, booting a disk image pulled from a soon-to-be-discarded physical disk can produce a Blue Screen of Death (BSOD) with the message:

Stop error code 0x0000007B

or

(INACCESSABLE_BOOT_DEVICE)

This seems like it's roughly the equivalent (in a standard debian GNU/Linux environment) of specifying MODULES=dep in /etc/initramfs-tools/initramfs.conf, and then trying to swap out all the hardware. At first blush, Microsoft's knowledge base suggests doing an in-place upgrade or full repartition and reinstall, which are both fairly drastic measures -- you might as well just start from scratch, which is exactly what you don't want to have to do for a nursed-along legacy system which no one who originally set it up is even with the organization any more. Fortunately, a bit more digging in the Knowledge Base turned up an unsupported set of steps that appears to be the equivalent of setting MODULES=most (at least for the IDE chipsets). Running this on the old hardware before imaging the disk worked for me, though i did need to re-validate Windows XP after the reboot by typing in the long magic code again. i guess they're keying it to the hardware, which clearly changed in this instance. Such silliness to spend time working around, really, when i'd rather be spending my time working on free software. :/

After a relatively long period of flux, I've managed to settle into a routine where I've got all the time I can dedicate to open source stuff bunched together: every Tuesday night. I've started calling it "Debian Day", although Debian won't get all that time exclusively. One evening a week doesn't sound like much, but that's unfortunately a realistic amount given my other commitments. Still, having a firm idea of how much resource I have makes planning what I can and can't do much easier. It also means I need to be quite selective about what I spend my time on. This also impacts the type of interactions I can use. I can respond to email, bugs, etc., but I cannot effectively interact with folks on IRC, for example: the period between windows is too large for it to work effectively. Therefore, I can't work reliably with any team or area of Debian that relies on IRC to get things done. Tonight, then. I have recently been ripping some CDs, a task for which I use grip, despite it long since having been dropped by Debian. It still works very well, whereas Sound Juicer, which still pops up whenever I insert an audio CD, does not. I've been interested in helping out with Debian GNOME packaging for a while, and it seems the issue has been fixed upstream, but after asking around in #debian-gnome, the root cause is that the newer musicbrainz API packages have not been uploaded to Debian. (the existing package uses the old API). Two separate people appear to have been working on a new Music Brainz package, but neither filed an ITP bug. I've spent the largest chunk of my Debian time just trying to figure out what has been going on, sadly, but at least there's the likelyhood this will be fixed for wheezy. There's also the tail end of a musicbrainz-2.1 transition finishing up. Another thing I considered this evening was the future of debgtd in Debian stable. On reflection, and after sitting on the fence for several months, I've decided I don't have the time to fix it up enough to have it ready for wheezy, so with regret I've filed a removal bug. With any luck, at some point in the wheezy cycle I'll manage to put more time towards this, because I think it addresses a genuine need and could be useful to people. Finally, whenever I sit down and look at Debian stuff to finish up, I have a strong temptation to fix some issues belonging to the games team. In particular, Gustavo Panizzo has put a lot of work into vavoom packages, but sadly his requests for sponsorship are falling on deaf ears. There's also Johey Shmit's work on packaging zdoom which needs some attention to finish removing some embedded libraries. I have to keep reminding myself I left the team for a reason, to focus on something else, and finding that something else should be the priority. So all in all not the most uplifting evening: I'm starting to think I simply couldn't work effectively with the Debian GNOME team without being able to dedicate more time to use IRC effectively; I spent more time that could be necessary figuring out a bunch of other folks have potentially wasted each other's time by working on packages and not filing ITPs; and I've removed one of my own packages from the distribution altogether. It's not all doom and gloom, though: last week I packaged Simon Howard's lhasa, a library and tool for decompressing LHA files, as used by various old DOS games. (See? I just can't escape that stuff ) I also managed to 'resolve' two bup bugs, one RC. I also finished transitioning from my old PGP key to my new one.