Ever since my company moved offices back in June, I've been wanting to do this: organize a BSP. It just hadn't happened yet. Today, I decided that unless I start actually making this a concrete plan rather than some vague idea, it isn't going to happen. So, here we go: during the weekend of the 15th and 16th of December, I expect some people to show up and help stamping out those pesky RC bugs. If you've been wanting to help Debian somehow but don't know where to start, this is your chance! Don't worry about not knowing how all this works; there will be Debian Developers attending who can help you out (two DDs have told me that they'd like to attend, and of course I will be there too). If you've got some experience in Debian packaging, your help would of course be all the more appreciated. Even if you can't make it the whole weekend, that's fine too! Just be there for as long as you can, and help stamp out those pesky RC bugs. The more people help out, the better!

As the openSUSE Conference and all related events are over, it's time to share my thoughts how it was. This was actually first conference ever where I helped with organizing, so it was quite new experience for me. Week before it has happened I expected quite a lot of problems, but in the end everything went more or less smoothly. Having something like 700-800 visitors is IMHO great for first year of LinuxDays, even though it has been collocated with other conferences. Most people came on Saturday and there were less of them on Sunday. Maybe it was result of the party, which lasted for many of them till early morning (I've left around 1 AM and there were still lot of people drinking). Besides organizing stuff, I also had my own talk about Weblate. Unfortunately not much people has attended it, but those who did gave me some valuable feedback which will be reflected in next Weblate release. Anyway if you want to see my slides from this talk, they are now available on my website.

Filed under: English Suse Weblate 0 comments Flattr this!

I've had a few weeks with very little time spent on Debian. I did set aside some time to try to find and fix an RC bug, to help the release. Steven C commented to let me know about the UDD Developer's Dashboard which is a great way to find RC bugs relevant to a team. I sat down to try and fix #648272 in 'Abuse' but sadly I didn't manage, instead finding another possible RC bug #689457 on the way. I was curious as to how long the version 1 web services API for musicbrainz will be kept around, in case there was a strong likelyhood it would be turned off during the lifetime of wheezy. If that were the case, then it might have been worth requesting freeze exceptions to patch the last few WS1 clients to use WS2 instead. However, the musicbrainz developers have no plans to turn off WS1 as long as there are people still using it. This is a bit of a chicken and egg situation for wheezy, but probably not strong enough grounds to persue it any further. So last night was my first "Debian Day" in a while, and by chance I've had a number of enquiries for software and packages which I haven't touched in a while. For the prboom package, we rebuild the upstream tarball to remove some material we don't have the right to distribute. The code that does the repackaging lives in a script called debian/fix_upstream.sh. Prboom upstream release very infrequently, so the script hasn't been run in a while. Fabian Greffrath noticed some bashisms which meant it would not work reliably with a non-bash /bin/sh. He also noticed that we couldn't build the binary lump data/prboom.wad using only free software since some changes that were made to freedoom. One possible fix was to expand the deutex package to recognise freedoom.wad, which I've now applied. Fabian has fixed the bashism issue, but I suspect a more robust solution long-term would be to always rebuild data/prboom.wad during package build, so that code gets exercised more often. The prboom changes are in the Git repository but I haven't release them yet. I need to decide whether it's worth requesting a freeze exception for the orig tarball build issue. It's not an FTBFS but it's similar: you could not rebuild a binary distributed in the source tarball using software only in Debian, at least not without some workarounds. We've had another request to package Prboom+. I've never been particularly interested in doing this myself, but it would probably be very little work, you could more-or-less take the prboom packaging as it stands. One thing I thought would be interesting to try would be to set up a git repository for the packaging and import the upstream Subversion repository in its entirety via git svn, rather than just import the released tarballs. Sadly their repository layout is not quite ideal for this. game-data-packager continues to grow. Following hot on the heels of Hexen 2 support, Markus Koschany has written preliminary support for LGeneral, a game which has been in Debian, at least in part, on and off. Hopefully we'll have that merged soon, and finish of Hexen 2 support whilst we're at it. On that note, here's a lazyweb question: what's the most convenient way to determine the length (at microsecond resolution) of a PCM WAV file from the command-line? Finally, we've been building Debian-based virtual machines at work, which meant dusting off some old preseed files and tweaking them to have unattended installs. This took a lot longer than I thought. It's certainly a lot more painful than kickstart and Redhat-based systems, and it's very unlikely that a preseed file that worked for one version of Debian would fully work for another. I'm not certain but I think even different releases of d-i targetting the same release "break" assumptions you might make, particularly about which questions you need to answer via the kernel command line, rather than in the preseed file. That's one thing we really need to sort. It's crazy having to specify keyboard layouts in the command line for VMs which won't ever have a keyboard. The disparity between what the user is asked, what value is written to the debconf database at that point in the install process, and what ends up in the debconf database at the end of the process is also a real pain: answering a question about keyboard layout sets half a dozen different debconf values in a cascade that is probably sensitive to other things like priority or whether d-i is in expert mode, too. So working backwards from the debconf database of an installed machine is an exercise in frustration, and figuring out which questions are asked prior to the preseed file being read for any given d-i or Debian release is a slow trial and error process. I'm going to get in touch with the d-i folks and see what kind of continuous integration systems they have in place. It would be great to have commit-level notification that you've broken preseed builds. If not for the d-i folks, then certainly at work.

There have been a few interesting discussions going on in parallel about my slaughter sysadmin tool. I've now decided there will be a 2.0 release, and that will change things for the better. At the moment there are two main parts to the system:

Downloading polices

These are instructions/perl code that are applied to the local host.

Downloading files

Polices are allowed to download files. e.g. /etc/ssh/sshd_config templates, etc.

Both these occur over HTTP fetches (SSL may be used), and there is a different root for the two trees. For example you can see the two public examples I have here:

• http://www.steve.org.uk/slaughter/policies/
• http://www.steve.org.uk/slaughter/files

A fetch of the policy "foo.policy" uses the first prefix, and a fetch of the file "bar" uses the latter prefix. (In actual live usage I use a restricted location because I figured I might end up storing sensitive things, though I suspect I don't.) The plan is to update the configuration file to read something like this:

transport = http
#
# Valid options will be
#    rsync   http   git   mercurial   ftp
#
#
# each transport will have a different prefix
#
prefix = http://static.steve.org.uk/private
# for rsync:
#  prefix=rsync.example.com::module/
#
# for ftp:
#  prefix=ftp://ftp.example.com/pub/
#
#  for git:
#  prefix=git://github.com/user/repo.git
#
#  for mercurial
#  prefix=http://repo.example.com/path/to/repo
#

I anticipate that the HTTP transport will continue to work the way it currently does. The other transports will clone/fetch the appropriate resource recursively to a local directory - say /var/cache/slaughter. So the complete archive of files/policies will be available locally. The HTTP transport will continue to work the same way with regard to file fetching, i.e. fetching them remotely on-demand. For all other transports the "remote" file being copied will be pulled from the local cache. So assuming this:

transport = rsync
prefix    = rsync.company.com::module/

Then the following policy will result in the expected action:

if ( UserExists( User => "skx" ) )
 
    # copy
    FetchFile(
            Source => "/global-keys",
              Dest => "/home/skx/.ssh/authorized_keys2",
             Owner => "skx",
             Group => "skx",
              Mode => "600" );
 

The file "/global-keys" will refer to /var/cache/slaughter/global-keys which will have been already downloaded. I see zero downside to this approach; it allows HTTP stuff to continue to work as it did before, and it allows more flexibility. We can benefit from knowing that the remote policies are untampered with, for example, via the checking built into git/mercurial, and the speed gains of rsync. There will also be an optional verification stage. So the code will roughly go like this:

• 1. Fetch the policy using the specified transport.
• 2. (Optionally) run some local command to verify the local policies.
• 3. Execute policies.

I'm not anticipating additional changes, but I'm open to persuasion.

I get a lot of spam. Most of it, thankfully is blocked by dspam but occasionally i get some through the filter. One that particularly caught my eye was interesting not so much what it was advertising (I don t read that part of the email) but where it came from and goes to. Normally there are two service providers involved in spam. The email comes from (or via) one and then the spamvertised website is another. The interesting thing is for this spam both of these were the same service provider. The email came from 174.34.168.85 and the spamvertised website was 70.32.40.194. Both of these addresses are owned by nobistech.net. I punted the email to spamcop and it said that they re not interested in spam reports. A few google queries shows that these guys seem quite happy to have spam sources and destinations and have been doing it for years. They either appear as nobistech.net or unbiquity servers but they are one and the same organisation, or at least related. I won t bother to send anything to them, it seems this has been done many many times by others with no results. Instead some CIDR blocks will be put into my blacklist.

A blog post from Martin Bekkelund today tell the story of how Amazon erased the books from a customer's kindle, locked the account and refuse to tell the customer why. If a real book store did this to a customer, it would be called breaking into private property and theft. The story has spread around the net today. A bit more background information is available in Norwegian from digi.no. It is no surprise that digital restriction mechanisms (DRM) are used this way, as it has been warned about such abuse since DRM was introduced many years back. And Amazon proved in 2009 that it was willing to break into customers equipment and remove the books people had bought, when it removed the book 1984 by George Orwell from all the customers who had bought it. From the official comments, it even sounded like Amazon would never do that again. And here we are, three years later. And thought this action is against Norwegian regulations and law, it is according to the terms of use as written by Amazon, and it is hard to hold Amazon accountable to Norwegian laws. It is just yet another example of unacceptable terms of use on the web, and how they are used to remove customer rights. Luckily for electronic books, there are alternatives without unacceptable terms. For example Project Gutenberg (about 40,000 books), Project Runenberg (1,652 books) and The Internet Archive (3,641,797 books) have heaps of books without DRM, which can read by anyone and shared with anyone. Update 2012-10-23: This story broke in the morning on Monday. In the evening after the story had spread all across the Internet, Amazon restored the account of the user, as reported by digi.no and NRK. Apparently public pressure work. The story from Martin have seen several twitter messages per minute the last 24 hours, which is quite a lot, and is still drawing a lot of attention. But even when the account is restored, the fundamental problem still exist. I recommend reading two opinions from Simon Phipps and Glen Moody if you want to learn more about the fundamentals and more details about the original story.

$ gsettings set \
org.gnome.desktop.interface gtk-color-scheme '
os_chrome_fg_color:black;
os_chrome_bg_color:#FED;
os_chrome_selected_fg_color:black;
os_chrome_selected_bg_color:#f5a089;
selected_fg_color:white;
selected_bg_color:#c50039;
theme_selected_fg_color:black;
theme_selected_bg_color:#c50039;
tooltip_fg_color:black;
tooltip_bg_color:#FFC;
'

A few days ago, Samsung released the source code of Synaps, an implementation of the Amazon Web Service CloudWatch API for OpenStack. Being a developer on the Ceilometer project, I've been curious to look on this project and how it could overlap with Ceilometer or other projects like Heat. What is CloudWatch? CloudWatch is a monitoring system provided by Amazon on its Web Services platform to monitor services. This allows you get notifications and trigger an action on certain threshold. For example, this can be used to scale your architecture by monitoring the number of requests you get on it and its general load by starting new servers. Synaps Synaps is written in around 7k lines of Python (with 28 % of which are comments), reuses at least one common module of OpenStack (openstack.common.cfg) and copy some modules from Nova. One thing that strikes me, is that there seems to be only a few unit tests compared to most OpenStack projects. Also, many parts of the code and documentation contains text written in korean, which won't be very helpful for most people! :-) It uses some external technologies, like Storm, Cassandra to store its persistent data and Pandas to do data analysis. The API server provides an EC2 compatible API only: no OpenStack specific API. This is probably not a bad thing for now, since I am not aware of any work in this direction. The API access directly the Cassandra back-end for read operation, but relies on RPC to do writes. This way, a set of daemon handles the write using the Storm part of Synaps and do data aggregation. The authentication only supports LDAP, but it should still be possible to add a driver for Keystone. A Java and a Python SDK are provided to record metrics into Synaps, but there's not enough documentation for it to be useful. Overlap with Heat For now, there's not a lot of overlap with Heat, because Heat does not implement completely the CloudWatch API. Heat actually still misses a lot of the CloudWatch functions. But as soon as it will implement the CloudWatch API completely, the overlap will be complete with Synaps in this regard. One divergence point however, is that Heat uses RPC to access data from the storage back-end via its engine (the central daemon), whereas Synaps directly connects to Cassandra. Also, Heat relies on SQLAlchemy, like most OpenStack projects needing a database. Overlap with Ceilometer One of the goal of Ceilometer is to provide data probes and pollsters for all OpenStack components (Nova, Swift, Quantum ) whereas Synaps let the OpenStack users to put any kind of metric inside it, and therefore doesn't provide anything for now. But the storage of metrics is the main common point between Synaps and Ceilometer. Synaps chose only one technology, Cassandra, to store its metrics, whereas Ceilometer took care of building an abstraction layer for the storage engine. Ceilometer currently allows an operator to use SQL or MongoDB, but Cassandra could likely be added. Data metric consolidation is done by Synaps. This makes sense, since Synaps don't need to have the full data history to trigger alarms. On the opposite, Ceilometer needs to have a full history to allow things like billing, and don't do any aggregation on data. Also, in Synaps, the data analysis is done using Pandas. This means the data used are retrieved from the Cassandra back-end, and then transformed by Pandas inside Synaps in something else. It's likely that in such a case, Synaps should use CQL to achieve that. Ceilometer manipulates the data near their storage: it means that the computation are done by back-end to be efficient (SQL, mapreduce ). Conclusion Considering Samsung open-sourced Synaps late in the development process, I don't feel like they aimed to have it becoming a core component. This is always sad, because the effort put into this implementation are big and it would have probably little to add some abstraction layers to follow what other OpenStack projects do. But this takes time and energy, and it's understandable that Samsung didn't want to achieve this in a short time frame. There's a part of the code and architecture that overlaps with Ceilometer and Heat. Ceilometer is becoming a specialized point to store data metrics from any source: so it's sad, but understandable, that Synaps did not tried to reuse it. Fortunately, Heat is working with Ceilometer to achieve exactly that. This means OpenStack would have only one metrics storage point, used for billing, for monitoring and alarming. Therefore, I think Synaps is an implementation of CloudWatch that should be looked at as an inspiration for Heat and Ceilometer to build a better and more integrated solution!

Christopher complained that in android, you need a user account to install software. That is not true, there is F-droid which is a catalog of Free/Open Source Software, no user account needed. Lack of multitasking is yes annoying sometimes, yes, but that is really a poweruser problem. Most users don't load web pages in background while playing. Most users will simply not bother with web pages that take too long to load. Given the choice, most users will prefer a snappy UI over a UI that allows proper multitasking. N900 was never snappy, stutters were common, and it would often hang for long times if you had too many apps or browser windows open at the same time. Even for Android there are way more complaints about non-fluid UI than lack of proper multitasking. It is thus only logical for the Android developers to concentrate in developing a snappier UI over improving multitasking. The fact that Android has given 400+ million Linux computers out in the hands of consumers is one of the MOST AMAZING THINGS ever. But it seems others on FOSS community seem to consider Android a rather unfortunate event, because it's not "Real Linux" or "100% free software" or "because it's not bug free". Err, like Maemo or MeeGo ever is any of those... It is fair to complain about usability quirks Android as enduser. But if you believe Free Software, you should also see the opportunity the open source parts of Android provides to you fix it to your needs. After all, the point of Free Software is that you don't need to depend on the upstream to fix you everything? Christoph complains that the stock email app is lacking. Android Email app is Open Source (Maemo and N9 email apps are not). Android email has been forked as K-9 Mail. The app lifecycle is fixable allowing you to return from browser to the mail you were at - at least a plenty of other apps manage to do that. The question is, are we consumers or creators. If we are just sitting waiting to Google provide us a perfect shrinkwrapped Android, we could just as well use iPhone or Windows phone. We should instead see Android as an opportunity - when it's 90% open source, fix the remaining closed source bits like open source 3D drivers and open source replacements for proprietary interfaces.

It's been six months since Xfce 4.10 has been released. And it's been four months since Wheezy is frozen. Due to this timing (and the fact Squeeze has 4.6 and doing a 4.6 4.10 upgrade needed some tuning in various packages), it was decided to not try to push 4.10 into Wheezy that late in the release cycle. So Xfce 4.10 was uploaded to experimental instead, and as it needs a full rebuild of all panel plugins against 4.10 panel (another reason for not trying to push it to Wheezy), those have not been uploaded. You can try Xfce 4.10 using experimental, but you'll need to remove the xfce4-goodies metapackage and the various depending plugin (since they'll just crash if you try to load them on 4.10 panel). Multiple people asked me (either on IRC, by private or public mail) when 4.10 will be uploaded to unstable and transition to testing. Like last time, the answer is : not before Wheezy is released. Right now, we're more interested in stabilizing Wheezy and squashing the bugs there than adding new ones in unstable. So, if you want to have Xfce 4.10 in Debian sid/testing sooner, then the easiest and fastest way is to fix some release critical bugs so we can release sooner, and then start breaking sid by uploading a whole lot of new stuff there. Note that this is true also for other software like GNOME, KDE stack (I have no idea how to call it these days), the Linux kernel, strongswan or whatever. About development releases of Xfce 4.11 (like the recently released exo 0.9 and Thunar 1.5), well, since we already use experimental for 4.10, there's not much chance they get uploaded anytime soon. We could try to package them and let people build it themselves, or I could host it somewhere on my server for people to try. But as I already said, we're more interested in fixing bug in Wheezy right now, and people interested in finding bugs in Xfce development releases (so they are fixed for the final one) should build it themselves and report everything they find on the Xfce bugzilla. TL;DR: if you care about new, shiny stuff, please help fixing RC bugs in Wheezy.

That was not my target in today's marathon (my second one in the year). The target was breaking my 3h38'45" record, set on June 10th in Caen. And, well, I did it. By more than 5 minutes \o/. Of course, the Seine-Eure marathon is the perfect place for this:

• really flat race with no up climbs above 5 meters and an overall up climbing of about 30 meters
• high level marathon. Even though it only has 600 runners, it is the fastest in France, when considering the "median" time (the time of the runner arrivng exactly half of all runners) of about 3h35.
• it is located in Normandy so the likeliness of heat in October is just science-fiction..:-)

My target was running 5'06"/km, so 3h35'11" (but the target was running this as long as possible and see what happens after km 35). I did this nearly perfectly in the first half, reaching the half-marathon in 1h47'30". Double this and you get 3h35'..:-) "However", I slightly accelerated in the second half, first without noticing, then because....I just could do it..:-). So I managed to run the second half at an average 5'/km, with even the last 2 kilometers around 4'50" and an amazing sprint at the end. Usain bubulle. The outcome is 3h33'35". Doh. Never thought I could do this. It seems that my unusual preparation (remember the "let's run a marathon when coming back from work, at night", in September? or the 5 half-marathons in a row....culminating in a 3/4 marathon in the 6th week-end?)...was not so bad. I'm now very close to qualifying times for Boston, my dream marathon (much more than NYC) as they are 3h30' for my age category. Well, another option is to wait turning 55, where the qualifying time is then 3h40..:-)...but it woudl be quite good to complete one under 3h30. Now, it doesn't look like a dream. We'll see at the end of next year as I will only run one marathon, in autumn, more focusing Spring on ultra running (and, technically, a marathon, but that will be the Mont-Blanc marathon, just before Debconf13....and these 42.195km are quite different from those in Normandy!

To experiment with network stuff, I was using UML-based network labs. Many alternatives exist, like GNS3, Netkit, Marionnet or Cloonix. All of them are great viable solutions but I still prefer to stick to my minimal home-made solution with UML virtual machines. Here is why:

• I didn t want to use disk images. They take a lot of space and they have to be maintained. They also become cluttered, especially if you try to reuse them across several labs. They are also difficult to share.
• I want to be able to access my home directory. It contains the important configuration files related to the lab and I can put them in the right place thanks to symbolic links when the lab starts. It also makes exchanging files with the lab quite easy.
• I don t want to boot a complete system. This allows me to be cheap on memory and each virtual system should boot in a few seconds.

The use of UML had some drawbacks:

• It may be buggy. For example, it is currently not possible to use gdbserver inside UML without a patch. Sometimes, the kernel won t even compile.
• It is slow.

However, UML features HostFS, a filesystem providing access to any part of the host filesystem. This is the killer feature which allows me to not use any virtual disk image and to get access to my home directory right from the guest. I discovered recently that KVM provided 9P, a similar filesystem on top of VirtIO, the paravirtualized IO framework.

Setting up the lab The setup of the lab is done with a single self-contained shell file. The layout is similar to what I have done with UML. I will only highlight here the most interesting steps.

Booting KVM with a minimal kernel My initial goal was to experiment with Nicolas Dichtel s IPv6 ECMP patch. Therefore, I needed to configure a custom kernel. I have started from make defconfig, removed everything that was not necessary, added what I needed for my lab (mostly network stuff) and added the appropriate options for VirtIO drivers:

CONFIG_NET_9P_VIRTIO=y
CONFIG_VIRTIO_BLK=y
CONFIG_VIRTIO_NET=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_VIRTIO=y
CONFIG_VIRTIO_RING=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_BALLOON=y
CONFIG_VIRTIO_MMIO=y

No modules. Grab the complete configuration if you want to have a look. From here, you can start your kernel with the following command ($LINUX is the appropriate bzImage):

kvm \
  -m 256m \
  -display none \
  -nodefconfig -no-user-config -nodefaults \
  \
  -chardev stdio,id=charserial0,signal=off \
  -device isa-serial,chardev=charserial0,id=serial0 \
  \
  -chardev socket,id=con0,path=$TMP/vm-$name-console.pipe,server,nowait \
  -mon chardev=con0,mode=readline,default \
  \
  -kernel $LINUX \
  -append "init=/bin/sh console=ttyS0"

Of course, since there is no disk to boot from, the kernel will panic when trying to mount the root filesystem. KVM is configured to not display video output (-display none). A serial port is defined and uses stdio as a backend¹. The kernel is configured to use this serial port as a console (console=ttyS0). A VirtIO console could have been used instead but it seems this is not possible to make it work early in the boot process. The KVM monitor is setup to listen on an Unix socket. It is possible to connect to it with socat UNIX:$TMP/vm-$name-console.pipe -.

Initial ramdisk UPDATED: I was initially unable to mount the host filesystem as the root filesystem for the guest directly by the kernel. In a comment, Josh Triplett told me to use /dev/root as the mount tag to solve this problem. I keep using an initrd in this post but the lab on Github has been updated to not use one. Here is how to build a small initial ramdisk:

# Setup initrd
setup_initrd()  
    info "Build initrd"
    DESTDIR=$TMP/initrd
    mkdir -p $DESTDIR
    # Setup busybox
    copy_exec $($WHICH busybox) /bin/busybox
    for applet in $($ DESTDIR /bin/busybox --list); do
        ln -s busybox $ DESTDIR /bin/$ applet 
    done
    # Setup init
    cp $PROGNAME $ DESTDIR /init
    cd "$ DESTDIR " && find .   \
       cpio --quiet -R 0:0 -o -H newc   \
       gzip > $TMP/initrd.gz
 

The copy_exec function is stolen from the initramfs-tools package in Debian. It will ensure that the appropriate libraries are also copied. Another solution would have been to use a static busybox. The setup script is copied as /init in the initial ramdisk. It will detect it has been invoked as such. If it was omitted, a shell would be spawned instead. Remove the cp call if you want to experiment manually. The flag -initrd allows KVM to use this initial ramdisk.

Root filesystem Let s mount our root filesystem using 9P. This is quite easy. First KVM needs to be configured to export the host filesystem to the guest:

kvm \
  $ PREVIOUS_ARGS  \
  -fsdev local,security_model=passthrough,id=fsdev-root,path=$ ROOT ,readonly \
  -device virtio-9p-pci,id=fs-root,fsdev=fsdev-root,mount_tag=rootshare

$ ROOT can either be / or any directory containing a complete filesystem. Mounting it from the guest is quite easy:

mkdir -p /target/ro
mount -t 9p rootshare /target/ro -o trans=virtio,version=9p2000.u

You should find a complete root filesystem inside /target/ro. I have used version=9p2000.u instead of version=9p2000.L because the later does not allow a program to mount() a host mount point². Now, you have a read-only root filesystem (because you don t want to mess with your existing root filesystem and moreover, you did not run this lab as root, did you?). Let s use an union filesystem. Debian comes with AUFS while Ubuntu and OpenWRT have migrated to overlayfs. I was previously using AUFS but got errors on some specific cases. It is still not clear which one will end up in the kernel. So, let s try overlayfs. I didn t find any patchset ready to be applied on top of my kernel tree. I was working with David Miller s net-next tree. Here is how I have applied the overlayfs patch on top of it:

$ git remote add torvalds git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
$ git fetch torvalds
$ git remote add overlayfs git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/vfs.git
$ git fetch overlayfs
$ git merge-base overlayfs.v15 v3.6
4cbe5a555fa58a79b6ecbb6c531b8bab0650778d
$ git checkout -b net-next+overlayfs
$ git cherry-pick 4cbe5a555fa58a79b6ecbb6c531b8bab0650778d..overlayfs.v15

Don t forget to enable CONFIG_OVERLAYFS_FS in .config. Here is how I configured the whole root filesystem:

info "Setup overlayfs"
mkdir /target
mkdir /target/ro
mkdir /target/rw
mkdir /target/overlay
# Version 9p2000.u allows to access /dev, /sys and mount new
# partitions over them. This is not the case for 9p2000.L.
mount -t 9p        rootshare /target/ro      -o trans=virtio,version=9p2000.u
mount -t tmpfs     tmpfs     /target/rw      -o rw
mount -t overlayfs overlayfs /target/overlay -o lowerdir=/target/ro,upperdir=/target/rw
mount -n -t proc  proc /target/overlay/proc
mount -n -t sysfs sys  /target/overlay/sys
info "Mount home directory on /root"
mount -t 9p homeshare /target/overlay/root -o trans=virtio,version=9p2000.L,access=0,rw
info "Mount lab directory on /lab"
mkdir /target/overlay/lab
mount -t 9p labshare /target/overlay/lab -o trans=virtio,version=9p2000.L,access=0,rw
info "Chroot"
export STATE=1
cp "$PROGNAME" /target/overlay
exec chroot /target/overlay "$PROGNAME"

You have to export your $ HOME and the lab directory from host:

kvm \
  $ PREVIOUS_ARGS  \
  -fsdev local,security_model=passthrough,id=fsdev-root,path=$ ROOT ,readonly \
  -device virtio-9p-pci,id=fs-root,fsdev=fsdev-root,mount_tag=rootshare \
  -fsdev local,security_model=none,id=fsdev-home,path=$ HOME  \
  -device virtio-9p-pci,id=fs-home,fsdev=fsdev-home,mount_tag=homeshare \
  -fsdev local,security_model=none,id=fsdev-lab,path=$(dirname "$PROGNAME") \
  -device virtio-9p-pci,id=fs-lab,fsdev=fsdev-lab,mount_tag=labshare

Network You know what is missing from our network lab? Network setup. For each LAN that I will need, I spawn a VDE switch:

# Setup a VDE switch
setup_switch()  
    info "Setup switch $1"
    screen -t "sw-$1" \
        start-stop-daemon --make-pidfile --pidfile "$TMP/switch-$1.pid" \
        --start --startas $($WHICH vde_switch) -- \
        --sock "$TMP/switch-$1.sock"
    screen -X select 0
 

To attach an interface to the newly created LAN, I use:

mac=$(echo $name-$net   sha1sum   \
            awk ' print "52:54:" substr($1,0,2) ":" substr($1, 2, 2) ":" substr($1, 4, 2) ":" substr($1, 6, 2) ')
kvm \
  $ PREVIOUS_ARGS  \
  -net nic,model=virtio,macaddr=$mac,vlan=$net \
  -net vde,sock=$TMP/switch-$net.sock,vlan=$net

The use of a VDE switch allows me to run the lab as a non-root user. It is possible to give Internet access to each VM, either by using -net user flag or using slirpvde on a special switch. I prefer the latest solution since it will allow the VM to speak to each others.

Debugging This lab was mostly done to debug both the kernel and Quagga. Each of them can be debugged remotely.

Kernel debugging While the kernel features KGDB, its own debugger, compatible with GDB, it is easier to use the remote GDB server built inside KVM.

kvm \
  $ PREVIOUS_ARGS  \
  -gdb unix:$TMP/vm-$name-gdb.pipe,server,nowait

To connect to the remote GDB server from the host, first locate the vmlinux file at the root of the source tree and run GDB on it. The kernel has to be compiled with CONFIG_DEBUG_INFO=y to get the appropriate debugging symbols. Then, use socat with the Unix socket to attach to the remote debugger:

$ gdb vmlinux
GNU gdb (GDB) 7.4.1-debian
Reading symbols from /home/bernat/src/linux/vmlinux...done.
(gdb) target remote   socat UNIX:$TMP/vm-$name-gdb.pipe -
Remote debugging using   socat UNIX:/tmp/tmp.W36qWnrCEj/vm-r1-gdb.pipe -
native_safe_halt () at /home/bernat/src/linux/arch/x86/include/asm/irqflags.h:50
50   
(gdb)

You can now set breakpoints and resume the execution of the kernel. It is easier to debug the kernel if optimizations are not enabled. However, it is not possible to disable them globally. You can however disable them for some files. For example, to debug net/ipv6/route.c, just add CFLAGS_route.o = -O0 to net/ipv6/Makefile, remove net/ipv6/route.o and type make.

Userland debugging To debug a program inside KVM, you can just use gdb as usual. Your $HOME directory is available and it should be therefore straightforward. However, if you want to perform some remote debugging, that s quite easy. Add a new serial port to KVM:

kvm \
  $ PREVIOUS_ARGS  \
  -chardev socket,id=charserial1,path=$TMP/vm-$name-serial.pipe,server,nowait \
  -device isa-serial,chardev=charserial1,id=serial1

Starts gdbserver in the guest:

$ libtool execute gdbserver /dev/ttyS1 zebra/zebra
Process /root/code/orange/quagga/build/zebra/.libs/lt-zebra created; pid = 800
Remote debugging using /dev/ttyS1

And from the host, you can attach to the remote process:

$ libtool execute gdb zebra/zebra
GNU gdb (GDB) 7.4.1-debian
Reading symbols from /home/bernat/code/orange/quagga/build/zebra/.libs/lt-zebra...done.
(gdb) target remote   socat UNIX:/tmp/tmp.W36qWnrCEj/vm-r1-serial.pipe
Remote debugging using   socat UNIX:/tmp/tmp.W36qWnrCEj/vm-r1-serial.pipe
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x00007ffff7dddaf0 in ?? () from /lib64/ld-linux-x86-64.so.2
(gdb)

Demo For a demo, have a look at the following video (it is also available as an Ogg Theora video).

<iframe frameborder="0" height="270" src="http://www.dailymotion.com/embed/video/xuglsg" width="480"></iframe>

---

1. stdio is configured such that signals are not enabled. KVM won t stop when receiving SIGINT. This is important for the usage we want to have.
2. Therefore, it is not possible to mound a fresh /proc on top of the existing one. I have searched a bit but didn t find why. Any comments on this is welcome.

7 years, 8 months, 17 days, 22 hours and 25 minutes ago I wrote the first post for this blog, inspired by a talk by Andrew Tridgell at linux.conf.au 2004 on the value of a personal "junkcode" repository. I know this because I've recently gone through considerable effort to change the format of the blog and keeping the timestamps in sync was one challenge. I've now updated to use Pelican, massaged all the old posts into reStructuredText, done a fair bit of HTML and CSS hacking and hopefully redirected all the old URLs to the correct place. Old comments didn't make the cut, but there wasn't much value there and I'm happy to outsouce to Disqus. How things have changed! As I was writing the CSS to round the corners of some elements, I remembered back to cribbing from Slashdot HTML to do the same thing -- except in those days you used a table with blank elements in each corner with 4 different rounded-corner gif files! Yet I'm still writing in emacs, so some things will forever remain the same I guess.

I ve started the morning with the easy hacks for non developers talks, which for my opinion lacked some code related tasks that even non developers can do like fixing translation problems in the source (in the English strings), fixing some simple logic bugs which doesn t require reading complex code (e.g. traversing records and first/last special cases). The other options mentioned in the talk aren t less important, but I think people shouldn t be afraid reading/touching the code to some level even with very limited coding skills. Later I ve used Rene s 3.7/master builds for Debian to verify two RTL bugs (fdo#43210, fdo#44925) fixed by the Motah guys from Saudi Arabia. After the verification, I pushed the changes to the 3.6 branch (should appear in 3.6.4).
Katya, which I ve met in the last conference, told me see watched my talked remotely, and took interest in RTL ui forcing right alignment to non aligned text. While reproducing the bug on her computer we were able to narrow the problem a bit, to be related to the locale and not only to the RTL interface. Till the end of the day she d identified the problem in the odp file and I hope a fix will be available by tomorrow. Jan (kendy) had a good progress with fdo#44657 (RTL problem #1). Checking the fix is quite confusing, and we had to use two laptops to understand and compare the right and desired behavior. Hopefully this problem will end soon, which will make coming to the conference worthwhile (as for the RTL status and efforts). Later I started working on moving the RTL status page from a wiki in Hebrew to the TDF wiki RTL Bugs page in English . By this I hope to get more visibility to the current status, and as a base to collaboration, let others help with RTL more easily. During one of the breaks I heard Lionel encourages someone to do builds on his own, and I decided that after avoiding this tasks for two years, it s about time. By the evening I got a build running according to the native build guide in the wiki (which took half the night on my laptop). There are still a few things to learn how to do with this build, but I how to now be more active in testing things without waiting for official Debian builds (which are quite hard when Debian is frozen). Thanks to Lionel for the encouragement. http://cgit.freedesktop.org/libreoffice/core/commit/?id=a662d6d8c00f5a1cff724db51b0914aab36e201b
Filed under: LibreOffice

Puppet I've really had it with Puppet. I used to be able to put up with all its downsides

• Non-Unix approach to everything (own transport, self-made PKI, non-intuitive configuration language, a faint attempt at versioning (bitbucket), and much much more )
• Ruby
• Abysmal slowness
• Lack of basic functionality (e.g. replace a line of text)
• Host management and configuration programming intertwined, lack of a high-level approach to defining functionality
• Horrific error messages
• Catastrophic upgrade paths
• Did I mention Ruby and its speed?
• Lack of IPv6 support
• [I could keep going ]

but now that my fourth attempt to upgrade my complex configuration from version 0.25.5 to version 2.7 failed due to a myriad of completely incomprehensible errors ("err: Could not run Puppet configuration client: interning empty string") and many hours were lost in trying to hunt these down using binary searches, I am giving up. Bye bye Puppet.

An alternative But I need an alternative. I want a system that is capable of handling a large number of hosts, but not so complex that one wouldn't put it to use for half a dozen machines. The configuration management system I want looks about as follows: It

• makes use of existing infrastructure (e.g. SSH transport and public keys, Unix toolchain, Debian package management and debconf)
• interacts with the package management system (Debian only in my case)
• can provision files whose contents might depend on context, particular machine data and conditionals. There should be a unified templating approach for static and dynamic files, with the ability to override the source of data (e.g. a default template used unless a template exists for a class of machine, or a specific hostname)
• can edit files on the target machine in a flexible and robust manner
• can remove files
• can run commands when files change
• can reference data from other machines (e.g. obtain the certificate fingerprint of each hosts that define me as their SMTP smarthost)
• can control running services (i.e. enable init.d scripts, check that a process is running
• is written in a sensible language
• is modular and easily extensible, ideally using a well-known language (e.g. Python!)
• allows to specify infrastructure with tags ("all webservers", "all machines in Zurich", "machines that are in Munich and receive mail"), but with the ability to override every parameter for a specific host
• should just do configuration management, and not try to take away jobs from monitoring software
• logs changes per-machine and collects data about applied configurations in a central location
• is configured using flat files that are human-readable so that the configuration may be stored in Git (e.g. YAML, not XML)
• can be configured using scripts in a flexible way

Since for me, Ruby is a downside of Puppet, I won't look at Chef, but from this page, I gleaned a couple of links: Ansible, Quattor, Salt, and bcfg2 (which uses XML though). And of course, there remains the ephemeral cfengine.

cfengine I haven't used cfengine since 2002, but I am not convinced it's worth a new look because it seems to be an academic project with gigantic complexity and a whole vernacular to its own. There is no doubt that it is a powerful solution, and the most mature of all of them, but it's far away from the Unix-like simplicity that I've come to love in almost 20 years of Debian. Do correct me if I am wrong.

Ansible Ansible looks interesting. It seems rather bottom-up, first introducing a way to remotely execute commands on hosts, which you can then later extend/automate to manage the host configurations. It uses SSH for transport, and its reason-to-be made me want to look at it. My ventures into the Ansible domain are not over yet, but I've put them on hold. First of all, it's not yet packaged for Debian (Ubuntu-PPA packages work on Debian squeeze and wheezy). Second, I was put off a bit by its gratuitous use of the shell to run commands, as well as other design decisions. Check this out: there are modules for the remote execution of commands, namely "shell", "command", and "raw". The shell modules should be self-explanatory; the command module provides some idempotency, such as not running the command if a file exists (or not). To do this, it creates a Python script in /tmp on the target and then executes that like so:

$SHELL -c /tmp/ansible/ansible-1350291485.22-74945524909437/command

Correct me if I am wrong, but there is zero need for this shell indirection. My attempts at finding an answer on IRC were met by user "daniel_hozac" with a reason along the lines of "it's needed, believe me", and on the mailing list, I am told that only the shell can execute a script by parsing the interpreter line at the top of the module. Finally, the raw execution module also executes using the shell And there a few other design decisions that I can't quite explain, around the command-line switch --sudo see the aforementioned message In short: running a command like

ansible -v arnold.madduck.net -a "/usr/bin/apt-get update" --sudo

does not invoke apt-get with sudo, as one might like; it invokes the shell that runs the Python script that runs the command. Effectively therefore, you need to allow sudo shell execution, and for proper automation, this has to be possible without a password. And then you might just as well allow root logins again. The author seems to think that "core behaviour" is that sudo allows all execution and that limiting the commands to run is not a use-case that Ansible will support. Apparently, I was the first to ever suggest this. There are always ways around (e.g. skip --sudo and just use sudo as the command, simply ignore the useless shell invocation and trust that your machine can handle it, but when such design decisions remain incomprehensible and get defended by the project people, then I am hesitant to invest more time on principle.

Salt Finally, I've looked at Salt, which is what I've spent most time on so far. From the discussions I started on host targeting and data collection, it soon became apparent that Salt is very thin and flexible, and that the user community is accomodating. Unfortunately, Salt does not use SSH, but at least it reuses existing functionality (ZeroMQ). As opposed to the push/pull model, Salt "minions" interestingly maintain a persistent connection to the server (which is not yet very stable), and while non-root usage is still not unproblematic, at least there has already been work done in this direction. I think I will investigate Salt more as it does look like it can do what I want. The YAML-based syntax does seem a bit brittle, but it's the best I've found so far. NP: The Pineapple Thief: Someone Here is Missing

I ll be attending Google Summer of Code s Mentor s Summit this weekend. I leave tomorrow for Mountain View if anyone else will be there, be sure to let me know! Looking forward to as great of an event as last year!

As you may know, when Phil Zimmermann published PGP, his goal was to counter the wide spying possibilities the development of electronic communications offered to the authorities, by promoting the general use of encryption so that 1. governments would not be able to restrict it afterwards and 2. it becomes possible to encrypt messages for privacy without drawing suspicion. Now, cryptography has made its way and is widely use, but mostly for commercial website and only sparsely for mail. Indeed, contrary to signing, encrypting a message requires that the recipient has a crypto key and is able and willing to use it to decrypt your message. To alleviate this problem, I thought that it would be useful to have a way, when sending a message, to tell your recipients that you would like that they encrypt their answers, because you are able and willing to get encrypted mail. I think the most relevant way to do that would be by adding a dedicated header field:

Encryption-Desired: PGP

In that header, one would be able to indicate what protocols he accepts. I suggest a comma-separated list, with the following possible values: PGP (or PGP/MIME and PGP/traditional for people that only accept one of these two PGP formats), S/MIME. For instance:

Encryption-Desired: PGP/MIME, S/MIME

It may be useful to be able to indicate that you either always wish your messages to be encrypted, or that it is only for this message's replies. If this is really relevant, it would take two distinct headers: Encryption-Desired and Reply-Encryption-Desired. Any thought about that proposal? Next thing to do, determine how to have that standardized at IANA and how to promote it.

Credit Unions are financial co-operatives where a community s savings are used to fund other community borrowing and both savers and borrowers can become members. Thursday 18 October 2012 is International Credit Union Day, as well as being in the United Nations International Year of Co-operatives. This year s theme is Members Matter Most which was suggested from Ireland. In the UK, there will be events including one at Westminster with speeches from Damian Hind MP, Chair of All Party Parliamentary Group on Credit Unions and Simon Hughes MP, its Vice Chair. I m a member of my local credit union. I can pay in at my bank or any PayPoint and withdraw by bank transfer. If you re fed up with the fat cat plc bankers, there s no better time to Move Your Money in general and Find Your Credit Union (international visitors, start at WOCCU) in particular, is there?

Yesterday I went to FES Iztacala, the faculty where I worked between 1999 and 2003. It's nice to go visit some good friends (even if to talk for work issues). It is somewhat far from my usual roaming area (~25Km straight to the North), so I cannot do it as often as I'd like. But anyway - I had to be at work early in the morning, but leaving from here a bit early for lunchtime, and leaving home at ~14:30, I managed to arrive to Iztacala in ~75 minutes. Sustained cycling for 20 Km/h, even counting stops at traffic lights on the way, yay! Anyway, had a productive and fun evening there, but around 18:00 I decided to head back before night got me Specially for the first part of the way, as I'm not familiar with Atzcapozalco. Alejandro suggested me to go by the recently (some months ago) opened cycle path that covers 4Km and almost exactly crosses the delegation (each of the 16 constitutive parts of Distrito Federal, where an important part of Mexico City is located). The cycle path is a good initiative... But I must say, I'm very very glad I took it still with good daylight. As well as the Recreative cyclepath that goes to the South, until the border of Morelos state, this one was built over abandoned rail tracks. Good use for a vacant and useless public space Rail tracks which lay unclaimed in the city are uncomfortable to walk, and useless for anything else, so they basically mean a useless 2m-wide strip of common grounds. So, I welcome any initiatives that make it into a useful space again! And two meters are just enough for a comfortable cycling path - Yes, which will surely be shared with pedestrians, and sometimes becomes uncomfortable. But lets try it! However... When rail tracks are decomissioned and cycle paths built over them... the metal should be dismounted! Not only because of economic concerns (good metal used for rail tracks is much more expensive and useful than asphalt), but because if it stays there, it just becomes a danger. Specially, as is the case, if the asphalt is just deep enough to sometimes cover the tracks And sometimes not. Had I known, I would have taken several photographs of important mistakes in the rail layout. I know I was very close to having an accident at least once (this means, I lost balance and miraclously managed to slow down from ~15Km/h by running with the bike between my legs!), and got in uncomfortable situations several more times. For a good portion of the track, there is a train track running at about of its width, so I had to constantly ring the bell or shout whenever I saw pedestrians As changing from side to side to route around them would put both them and me in danger. Towards the Southern part of the cycle path, as it is a much more active industrial area, there are many places where multiple tracks cross each other under the thin asphalt, sometimes completely unpaved. In one of those points I even decided to step down of the bike and make ~20m walking. This cycle path seems like it was done in a great hurry to present a successful project to the Politicians in Charge, without much thought on what it requires to be really a good project. It provides, yes, a very useful and good mobility solution for cyclists in the North-West. But it is too dangerous... And I am not sure whether I'd take it again. Probably not. So, all in all...

• Thanks to the Atzcapozalco authorities who thought of the cyclists and claimed back unused public space for us all to enjoy!
• The job done was too hasty. Valuable money (selling the tracks) was lost by burying it under asphalt. But it was not enough asphalt It needs attention before somebody gets hurt (or, before more people I'd be surprised if nobody had yet felt adrenaline over there)
• I hope the entrant city government takes the self-powered vehicle promotion more seriously than the outgoing one Which has done great advances (such as the Ecobici program), but is still far from reaching its promises... As always, I know.
• I managed to get back home alive and complete, yay!
• Of course, the cycle path has been mapped into OpenStreetMap. My SportsTracker tracks are also there, but they require Flash (ugh): the way to Iztacala and the way back.

Oh, and lastly: Some might be surprised I'm using bits of Twitterspeak here. But well, I now have presence a bot repeating my posts over there, so I'd better get Alejandro to read this using the proper channels ;-)