The Debian package of the Linux kernel has some additional features not present in mainline Linux 3.2. This continues from part 1 and part 2. Real-time scheduling The 'PREEMPT_RT' aka 'linux-rt' patchset provides true real-time scheduling under Linux. This can allow applications to respond to I/O with very low latency and jitter, although the results depend on application design and on the hardware and firmware used. Real-time scheduling is commonly used in industrial control, live audio/video applications, and high-frequency trading systems; it can also be useful for tightly-coupled cluster computing. However, most applications will run more slowly under these kernels, and may not be any more responsive. The Linux 3.2-rt stable series maintained by Steven Rostedt is used to build alternate kernels for amd64 and i386. You can install these using the metapackage linux-image-rt-amd64 or linux-image-rt-686-pae. Hyper-V guest drivers Like most VM hypervisors, Microsoft Hyper-V supports paravirtualised drivers which can provide much better performance than emulation of old hardware devices. Linux 3.2 had quite early versions of these drivers that were relegated to the 'staging' area, but we've updated them to the much improved versions found in Linux 3.4. I hoped we would also be able to package the userland daemon that supports management of Hyper-V guests, but this also needed many fixes and wasn't ready in time.

The Debian package has some additional features not present in mainline Linux 3.2. Some of these were described in part 1, and this continues the list. CPU devices CPUs are exposed in the device model as normal devices, not 'sysdevs'. For x86 CPUs, model and feature information is included in the uevent for the CPU device. This results in automatic loading of appropriate device drivers with the help of udev, rather than specialised init scripts:

• CPU frequency scaling drivers
• The intel_idle driver
• CPU temperature monitoring drivers
• Crypto drivers that use extensions such as SSE or VIA Padlock

Control groups Most control groups (cgroups) features are included, but some are disabled by default at run-time:

• The memory controller is disabled and can be enabled using the kernel parameter cgroup_enable=memory.
• Scheduler auto-grouping is disabled and can be enabled through the sysctl kernel.sched_autogroup_enabled.

EFI stub On x86 architectures the kernel is built as an EFI executable which can be directly loaded from an EFI shell, e.g.:

Shell> vmlinuz-3.2.0-4-amd64 root=/dev/sda2 initrd=initrd.img-3.2.0-4-amd64

Debian will usually boot on an EFI system via grub-efi, but this can be useful for some unusual configurations and some of the simple EFI boot loaders that are currently being developed. Linux Security Modules The kernel side of AppArmor has some extensions for backward compatibility with AppArmor userland versions 2.4-2.7, matching wheezy userland. Debian appears to be the only major distribution to provide a choice of LSMs: AppArmor, SELinux or TOMOYO. Use the security= boot parameter to enable any of them. CPU microcode updates for Xen Like all hardware, CPUs have bugs, which can sometimes be fixed in microcode. x86 microcode updates may be applied by either the BIOS or operating system, but it's easier to upgrade packages than to upgrade the BIOS. Xen 4.1 can apply microcode updates but relies on the dom0 kernel to load them, and the mainline Linux kernel does not do this. We include an extension of the microcode loader that does so.

As you may know, Debian 7.0 (codename 'wheezy') includes Linux 3.2. I'm maintaining the 3.2.y stable series at kernel.org which will collect bug fixes and some new hardware support as nominated by kernel developers and distributors, following the usual stable kernel rules. The Debian package has some additional features not present in mainline Linux 3.2. Union filesystem Debian Live needs a union filesystem to allow writing files on top of the read-only live system image. To support this, we include Aufs developed by Junjiro R. Okajima. I hope that Linux will gain union mount or union filesystem support in time for the next release, but currently it doesn't take much work on our part to integrate. Security hardening

• Magic SysRq functions are restricted by default. This can be changed through sysctl kernel.sysrq.
• Network protocols AF_IEEE802154, AF_RDS and AF_DECnet must be explicitly loaded; they cannot be loaded just by creating a socket.
• The default permissions on the debugfs root are 700, i.e. accessible only to root. (This change has also been made in Linux 3.7.)
• The symlink and hard link restrictions from Linux 3.6 (and previously grsecurity, Openwall and Ubuntu) are included and enabled by default. This can be changed through sysctls fs.protected_symlinks and fs.protected_hardlinks.
• The procfs hidepid/hidegid feature from Linux 3.4 is included, though not enabled by default. When enabled, this prevents users from reading the state of each others' processes through procfs, which can in some cases be security-sensitive.

Bufferbloat I backported two big pieces of the solution to bufferbloat:

• Byte Queue Limits (BQL) allows network drivers to reduce unnecessary buffering in their hardware transmit queues. This has been implemented in the bnx2, bnx2x, e1000e, forcedeth, igb, ixgbe, sfc, skge and tg3 drivers in Linux 3.3 onwards. Documentation is in the commit that added it, but you're unlikely to need it.
• The CoDel queue management algorithm can reduce unnecessary buffering in software transmit queues. FQ CoDel adds an element of 'fairness' between multiple network flows. These were implemented as the queue disciplines codel and fq_codel in Linux 3.5. They can be selected using the tc command, e.g.:
  tc qdisc replace dev eth0 root fq_codel

Alternate title: 'for those who care about channels 12-13'. For background, see the earlier entry on wireless regulation. If your wifi interface no longer supports channel 12 or 13, this may be because the kernel does not know the regulations for the domain code detected by the driver. Prior to Linux 2.6.34, regulations for 'US', 'JP' (Japan) and 'EU' (ETSI members) were built-in, but now there is only '00' (world, conservative regulations) which allows only channels 1-11 in the 2.4 GHz band. For any driver that detects specific European country codes, the 'EU' regulatory information was useless even in 2.6.32 (squeeze) and the unknown code would be mapped to the '00' (world) domain. Since wireless regulations are subject to change from time to time, they are now provided by the wireless-regdb package and supposed to be loaded into the kernel by crda. These packages should have been included in squeeze, but for various reasons they missed it. But they are now available in squeeze-backports.

Already chilled by Wheezy freeze? It s been a long ride since the release of Squeeze, and your beloved FTP Team tried to assist our tireless developers and contributors at its best. Here are some hot stats to give you a figure about what happened behind the scenes. Since the release of Squeeze, 7462 .changes files with NEW components were processed by dak, with an average of 14.660 NEW packages per day. On the FTP Team side, we had 6877 accepts (13.511 per day), 641 rejects (1.259 per day) and 280 comments to maintainers (0.550 per day). This table represents the activity by single team member:

Login	Accepts	Rejects	Comments
ansgar	407 accepts (0.800 per day)	71 rejects (0.139 per day)	53 comments (0.104 per day)
dak	12 accepts (0.024 per day)	1 rejects (0.002 per day)	0 comments (0.000 per day)
dktrkranz	4319 accepts (8.485 per day)	381 rejects (0.749 per day)	104 comments (0.204 per day)
joerg	100 accepts (0.196 per day)	12 rejects (0.024 per day)	1 comments (0.002 per day)
mhy	214 accepts (0.420 per day)	14 rejects (0.028 per day)	5 comments (0.010 per day)
stew	67 accepts (0.132 per day)	16 rejects (0.031 per day)	7 comments (0.014 per day)
tolimar	1480 accepts (2.908 per day)	93 rejects (0.183 per day)	84 comments (0.165 per day)
twerner	278 accepts (0.546 per day)	53 rejects (0.104 per day)	26 comments (0.051 per day)

---

Who were the most prolific maintainers who got a NEW processing? Here is our special top ten:

1. Debian Perl Group (559 accepts)
2. Debian Haskell Group (491 accepts)
3. Debian Ruby Extras Maintainers (285 accepts)
4. Debian Java Maintainers (257 accepts)
5. Debian Med Packaging Team (164 accepts)
6. Debian Multimedia Maintainers (160 accepts)
7. Debian Fonts Task Force (156 accepts)
8. Debian Javascript Maintainers (137 accepts)
9. Debian Python Modules Team (129 accepts)
10. Debian Qt/KDE Maintainers (98 accepts)

That doesn t reflect the real developers, though. Here s our Changed-By top ten:

1. Clint Adams (216 accepts)
2. Jonas Smedegaard (208 accepts)
3. Ben Hutchings (203 accepts)
4. Joachim Breitner (153 accepts)
5. TANIGUCHI Takaki (112 accepts)
6. Alessio Treglia (101 accepts)
7. David Paleino (95 accepts)
8. Nicholas Bamber (76 accepts)
9. Mathieu Parent (68 accepts)
10. Jeff Breidenbach (63 accepts)

Clint rocks with tons of Haskell packages, followed by Jonas (mostly Perl packages), and Ben (kernel uploads). Italian cabal stands still, with Alessio and David respectively at 6th and 7th place

---

How long does a package stay in NEW? Some more, some less, but the average is 3 days, 15 minutes and 21 seconds. Now go and check your dak mails to see whether you had a fast processing or not liblog4ada 1.2-1 surely had, as it was accepted after 30 seconds! gsoap 2.7.17-1 was not so lucky, it took 103 days, 8 hours, 20 minutes and 43 seconds to clear NEW, but then made its way to the archive. Better late than never

---

FTP Team is not just accepting NEW packages, but also removing obsolete ones. Here are some details about this task:

• Source packages removed: 2159
• Binaries for architecture all: 8773
• Binaries for architecture alpha: 3646
• Binaries for architecture amd64: 14653
• Binaries for architecture arm: 3
• Binaries for architecture armel: 13283
• Binaries for architecture armhf: 3423
• Binaries for architecture hppa: 1418
• Binaries for architecture hurd-i386: 6923
• Binaries for architecture i386: 15655
• Binaries for architecture ia64: 12365
• Binaries for architecture kfreebsd-amd64: 10972
• Binaries for architecture kfreebsd-i386: 10998
• Binaries for architecture mips: 12478
• Binaries for architecture mipsel: 12479
• Binaries for architecture powerpc: 13945
• Binaries for architecture s390: 12374
• Binaries for architecture s390x: 3884
• Binaries for architecture sparc: 12295

---

FTP Team also took care of override changes:

• Number of priority changes: 853
• Number of section changes: 975

Thursday saw all the Collabora employees at Cambridge office go out and socialise at the beer festival. They seemed to have selected a wonderful day for it, the sun was shining and it was warm and blue sky day.

Alas, I had to attend some customer conference calls and work on some time sensitive research so I could not go to the ball as it were. At about eight my brain had run out of steam so I decided to call it a day and go and meetup with people at the festival for an hour or two.

The queue when I arrived dissuaded me from that notion. I asked one of the stewards and they indicated it would take at least an hour from where the queue finished.

So I decided to wend my way home along the bank of the Cam. I proceeded slowly along and to my utter surprise bumped into Ben Hutchings and his Solarflare work colleagues having their own soiree. I was immediately invited to sit and converse. Pretty quickly I was inveigled into accepting a glass of wine by John Aspden from his floating bar (AKA houseboat).

From here on my evening was a pleasant one of amusing new people, easy conversation and a definite pondering if the host would be discovering the delights of Cam swimming as he became progressively inebriated!

So although I missed the festival I did manage to have an enjoyable time. A big thanks to the solarflare guys and especially John who was the consummate host and provided me with far too much alcohol.

The statistics of the "who wrote Linux x.y.z" series date back to at least 2.6.20. According to my experience talking with users and Free Software enthusiasts, those statistics really make a dent in the public perception of who is giving back upstream. Obviously, one should not take a single upstream, even if it is as important as the Linux kernel, as a measure of how much a given Free Software entity is giving back upstream overall. But users still seem to be fascinated by them. As a result, I have often had to answer the question: why Debian doesn't show up on those statistics?. My answer has always been something along the lines that Debian Developers who maintain Linux kernel packages, the almighty Debian Kernel Team, do that mostly as part of their volunteer engagement in Debian. As a consequence, they do not earmark their contributions as if they worked for a company and they add up to the hobbyist count instead (although you can you can routinely spot individual Debian Kernel Team members among the most active contributors for specific Linux releases). The above is the true and honest answer. But every time I've given it, I couldn't help feeling that the user who asked went home with a "yeah, well" afterthought. If you don't want to take my word of it, fine. Here is what Greg K-H had to say about Debian contributions in a recent blog post about the stable Linux kernel:

I would personally like to thank the Debian kernel developers, specifically Ben Hutchings, Maximilian Attems, Dann Frazier, Bastian Blank, and Moritz Muehlenhoff. They went above and beyond what any "normal" developer would have done, ferreting patches out of the kernel.org releases and the different vendor kernels and bug tracking systems, backporting them to the 2.6.32 kernel, testing, and then forwarding them on to me. Their dedication to their user community is amazing for such a "volunteer" group of developers. I firmly believe that without their help, the 2.6.32 kernel would not have been the success that it was. The users of Red Hat and SuSE products owe them a great debt. Buy them a beer the next time you see them, they more than deserve it.

I'll take good care of following his wise advice. Please do the same.

---

(Thanks to Sylvestre for pointing me to Greg's blog post.)

Ben Hutchings writes:

There is an alternative to using C-c, and that is M-#. This adds a '#' to the start of the line, commenting it out, and then behaves as if you pressed Return. The result is that the command is cancelled but still remains on-screen and in your history. You can then copy the command later using the mouse or keyboard.

Naturally, there is a zsh equivalent for this (there always is...): pound-insert. Quoth the manpage:

pound-insert
If there is no # character at the beginning of the buffer, add one to the beginning of each line. If there is one, remove a # from each line that has one. In either case, accept the current line. The INTERACTIVE_COMMENTS option must be set for this to have any usefulness.

So if you want the same for zsh, a simple

bindkey '^[#' pound-insert

is all you need. Update: Clint Adams noted another way to do (almost) the same.

I switched from Google Reader to Liferea a while back. I was mostly happy with the functionality of Reader but I don't really want my reading history to be recorded or my feed list to be vulnerable to the whim of Google's inscrutable abuse department. I'm still using Liferea but unfortunately it has a lot of problems. Yes I do have bug numbers:

• Calls sync() many times while refreshing feeds (#623619). This is probably related to use of SQLite and not properly batching writes. I work around this by using 'eatmydata' to nullify those calls, but that's not ideal.
• The UI thread also blocks on feed updates. Together with the previous bug, this can make it unresponsive for minutes while updating feeds (#542276).
• It often leaks memory and other resources when I play embedded video (#660593).
• There is no progress or 'busy' indicator for the browser (#660595).
• Error/prompt callbacks are not handled (#660597) - for example, if I follow a link that happens to lead to a PDF, nothing happens.
• There is no progress indicator for downloading 'enclosed' files (typically podcast episodes), and errors are simply ignored (#660599).
• While updating feeds, it may reset the item pane back to the summary (#660602).

So, I'm looking for an alternative again. My requirements are:

1. MAY be a desktop or web application.
2. If it's a web application, it MUST be reasonably secure, e.g. it must not be written in PHP.
3. If it's a web application, it MUST allow for multiple independent users on the same server.
4. If it's a desktop application, it MUST embed a browser engine (presumably Gecko or WebKit) so I can follow links without having to switch windows.
5. MUST support organisation of feeds by folders or tags, including combined item lists.
6. MUST keep track of which items have been read.
7. MUST support a global 'unread items' list. SHOULD only remove items from this list when I refresh it, not as soon as I move away from an item.
8. SHOULD support a three-pane (folder/list/item) view or something similar. Google Reader's list view with expanding items is perhaps even better, though it means links must be opened in a separate tab.
9. SHOULD support folder and item navigation by keyboard.
10. SHOULD have some way to flag/bookmark items for later attention.
11. If it's a desktop application, it SHOULD have some sort of download manager to support podcasts.

~$ dpkg --print-architecture
i386
~$ dpkg -s linux-image-$(uname -r)
Package  linux-image-3.2.0-1-amd64' is not installed and no info is available.
Use dpkg --info (= dpkg-deb --info) to examine archive files,
and dpkg --contents (= dpkg-deb --contents) to list their contents.
~$ dpkg -s linux-image-$(uname -r):amd64
Package: linux-image-3.2.0-1-amd64
Status: install ok installed
Priority: optional
Section: kernel
Installed-Size: 104822
Maintainer: Debian Kernel Team <debian-kernel@lists.debian.org>
Architecture: amd64
Source: linux-2.6
Version: 3.2.6-1

...

Phil Kern made a general call for testing of Debian 6.0.4. I would like to more specifically point out that I backported the isci driver for Intel C600 SAS/SATA controllers. Unfortunately I have not yet had any testing results for this. If you have any machines with this hardware not yet in production, please do consider testing the new Linux kernel package, version 2.6.32-41. Updated: I now have a positive test result from the user who requested this driver in the installer.

Our Dear Project Leader, Stefano Zacchiroli, regularly mentions the fact that there's an amount of Debian money available for hardware sponsoring of Debian developers, but it seems that not much people benefit from it. Each time I saw one of this reminder, I wonder if I should apply, and the anser is usually no. The fact is that I don't think any new laptop or desktop to do my Debian stuff, and the last time I bought a box (my x201s last summer) it was not really specifically for Debian tasks so I didn't dare to ask (not to mention the fact I bought it because I did have the money to do so). And I think this is mostly the problem. I might be wrong, but I think that most people which could benefit from this just don't dare asking or don't estimate themselves eligible for it. When I saw Ben Hutchings post, where the first thing he says is about how hardware is expensive, I thought hey, he should get some Debian money for buying new hardware: building kernel is really time consuming and having multiple powerful cores, more ram and fast disks/SSDs really helps . Turns out that Ben just didn't really want to spend too much money there, but the case still stands. We also see from time to time people saying they'll be offline for a while because of broken laptop or something like that. Once again, maybe those people wouldn't mind some help from the Debian project, and maybe they just don't think about asking, or they don't dare. So thinking about it a bit more, I think I wouldn't dare asking money for myself, but maybe I could dare asking money for other people (this is a bit like the flattr posts by Rapha l Hertzog, where he incited people to give money to projects he liked). If I'm not alone in this case, maybe those Debian developers who think some of their peers would benefit some hardware could drop them a mail with leader@ on copy, to propose just that. No need for huge publicity on that (in order to not embarass people), though the transparency rules still apply when it comes to Debian money.

I thought it was about time to get a faster machine for... well, mostly for compiling kernels faster, but I'm sure I will find other applications for it. I was originally intending to upgrade my laptop (Thinkpad T61, 1.8 GHz Core 2 Duo) but it seems that in the 3 years since I bought it second-hand laptop prices have not dropped in line with Moore's law. So instead I decided to upgrade (well, replace most of) a desktop PC. The last time I got a new desktop at home was in 2002. It has an Athlon XP, PATA drives and a huge CRT monitor. It has had some upgrades since then (DVD writer, more RAM, slightly faster CPU, replacement fan and PSU). Obviously it was time to replace motherboard, CPU and RAM. Not so obviously, the PSU needed changing again as modern CPUs apparently need more 12V electrons (?!). The DVD writer was salvageable via a USB adapter and its local HD is currently a laptop drive (the only spare SATA HD I had). The monitor is of course reusable, though the firmware setup program doesn't entirely agree with it. Windows Yes, I do sometimes run Windows. It's for games, and also seems to be needed for streaming video with stupid DRM. On the assumption that Windows is less likely to play nice with an existing installation, I tried installing it first. The firmware appear to allow me to boot the DVD in either BIOS or UEFI mode, so I selected the UEFI option. This worked so far as loading the initramfs (or whatever the Windows equivalent is) but then Windows was lacking a driver to access the DVD. This stumped me for a day or so until I realised I had plugged the drive into a USB 3 (XHCI) port. Apparently Windows 7 SP1 still doesn't include an XHCI driver in the installer (but neither does Debian 6.0.3 - we should fix that). The installation was mostly uneventful after this. The Windows installer includes a reasonably capable partition editor. Debian Installation of files was entirely uneventful. The installer correctly detected the GPT (GUID partition table) created by Windows. However, it was not able to install a boot loader. Currently it only includes support for the LILO and GRUB-PC boot loaders that run under BIOS. After much experimentation and cursing, I found instructions for installing Debian Squeeze on XServe 3.1 by Jan Peter Alexander Rajagukguk, which put me on the right track. It seems that the grub-install script for GRUB-EFI in squeeze (and maybe other parts of the package) doesn't actually work. However. JP's direction to build GRUB from source is no longer necessary. Here's what I did (minus the mis-steps):

1. Boot the installer in rescue mode and open a shell on the installed system:

   # # We start in dash, so switch to bash.
   # exec bash
   # mkdir -p /boot/efi
   # # Mount the EFI boot partition by UUID.
   # # It's probably called sda1 now, but is subject to change.
   # ls -l /dev/disk/by-uuid
   ...
   # echo >>/etc/fstab 'UUID=efi-boot-uuid /boot/efi vfat defaults 0 0'
   # mount /boot/efi
   # # Enable unstable sources, but use stable by default
   # echo >>/etc/apt/apt.conf 'APT::Default-Release "squeeze";'
   # echo >>/etc/apt/sources.list 'deb http://cdn.debian.net/debian/ sid main'
   # apt-get update
   ...
   # # Install unstable grub-efi-amd64
   # apt-get install -t unstable grub-efi-amd64
   ...
   # # Install GRUB to the EFI partition.  The '--removable' options
   # # inhibits setting EFI variables, which won't work because we booted
   # # through BIOS emulation.  Instead, it installs as the default boot
   # # loader.
   # grub-install --bootloader-id=GRUB --removable
   # # On the AMI/ASUS firmware, you cannot select a new bootloader
   # # directly but can invoke a 'shell'.  So I made GRUB the shell:
   # cp /boot/efi/boot/grub/grub.efi /boot/efi/shellx64.efi
   # exit
   

2. Reboot and select GRUB as the boot loader if necessary. In my case I had to enter the setup program and select 'EFI shell'. Debian should now boot from the hard drive.
3. Add a non-volatile boot option for GRUB (and make it the default):

   # apt-get install efibootmgr
   ...
   # modprobe efivars
   # efibootmgr -c -l '\efi\grub\grubx64.efi' -L GRUB
   

The bug report Simon McVittie reported in Debian bug #654876 that his laptop running Linux 3.0 or 3.1 would sometimes crash (panic) while idle. He initially suspected a driver bug, but the screen did not show any information about where the original fault occurred. However, using netconsole, he was able to capture a full log of the crash. This showed that a packet received through the wireless interface was being processed by IGMP, which then divided by zero. IGMP IGMP is part of the IPv4 protocol suite, supporting multicast routing. Every multicast address corresponds to a dynamic set of hosts, called a multicast group. Multicast routers can send query messages asking which hosts belong to which groups, and hosts using multicast report back at intervals. Routers can then limit forwarding of multicast packets to the interfaces where the group has members. More sophisticated switches can also snoop IGMP and use it to limit their multicast forwarding. There are unfortunately three different versions with semi-compatible message formats. In version 1, the maximum reporting interval (Max Response Time) is fixed as 10 seconds, but from version 2 it is specified in query messages. The Linux IGMP implementation supports all three versions. It distinguishes query messages as specified in RFC 3376 section 7.1: v3 messages are longer than v1 or v2; v2 messages have a non-zero Max Response Time whereas v1 messages always have zero. It is possible to force use of a specific protocol version, but normally if there are multiple multicast routers using different protocol versions it will respond according to the earliest protocol version in use so that all routers can understand its responses. Source and fix for the bug Linux 2.6.36 included two fixes to the version selection logic. Unfortunately, the second of these introduced the bug in question. While v2 query messages cannot possibly have zero Max Response Time (as that would make them v1), v3 query messages can. What this means is unspecified, but the Linux IGMP code previously treated it as the minimum valid value of 1/10 second. But in the case where a v3 query is received and a v2 query has also recently been received, this is no longer done. This results in a reporting interval of 0 seconds and a division by zero when deciding the initial random delay. The fix for this is pretty obvious. It is included in Linux 3.0.17, 3.1.9, 3.2.1, and the Debian package version 3.1.8-2. Security impact This is easily exploitable for denial of service within a local network. Linux does not check the destination address of IGMP queries, so it may also be possible to attack a target through its unicast address from several hops away. The attacker only needs to send a single IGMPv2 query followed by a single IGMPv3 query with zero Max Response Time. All systems running Linux 2.6.36 or later (up to the above fixed versions) with any active IPv4 multicast listeners (other than for the 'all hosts' address) are vulnerable.

Ben Hutchings is a rather unassuming guy but hiding behind his hat, there s a real kernel hacker who backports new drivers for the kernel in Debian stable so that our flagship release supports very recent hardware. Read on to learn more about Ben and the kernel team s projects for Debian Wheezy! Raphael: Who are you? Ben: I m a professional programmer, living in Cambridge, England with my long-suffering wife Nattie. In Debian, I mostly work on the Linux kernel and related packages. Raphael: How did you start contributing to Debian? Ben: I started using Debian in 1998 and at some point I subscribed to Debian Weekly News. So in 2003 I heard about the planned Debian 10th birthday party in Cambridge, and thought I would like to go to that. Somehow I persuaded Nattie that we should go, even though it was on the day of our wedding anniversary! We both enjoyed it; we made new friends and met some old ones (small world). From then on we have
both been socially involved in Debian UK. In 2004 there was a bug-squashing party in Cambridge, and we attended that as well. That s where I really started contributing fixing bugs and learning about Debian packaging. Then in 2005 I made my first package (sgt-puzzles), attended DebConf, and was persuaded to enter the New Maintainer process. NM involved a lot of waiting, but by the time I was given questions and tasks to do I had learned enough to get through quite quickly. In April 2006 I was approved as a Debian Developer. Meanwhile, I looked at the videos from DebConf 5 and thought that it would be useful to distribute them on a DVD. That led me to start writing video software and to get involved in the video team for the next year s DebConf. Raphael: You have been one the main driver behind the removal of non-free firmwares from the kernel. Explain us what you did and what s the status nowadays? Ben: That s giving me a bit more credit than I deserve. For a long time the easy way for drivers to load firmware programs was to include them as a blob in their static data, but more recently the kernel has included a simple method for drivers to request a named blob at run-time. These requests are normally handled by udev by reading from files on disk, although there is a build-time option to include blobs in the kernel. Several upstream and distribution developers worked to convert the older drivers to use this method. I converted the last few of these drivers that Debian included in its binary packages. In the upstream Linux source, those blobs have not actually been removed; they have been moved to a firmware subdirectory. The long-term plan is to remove this while still allowing the inclusion of blobs at build-time from the separate linux-firmware repository. For now, the Debian source package excludes this subdirectory from the upstream tarball, so it is all free software. There are still a few drivers that have not been converted, and in Debian we just exclude the firmware from them (so they cannot be built). And from time to time a driver will be added to the staging section of Linux that includes firmware in the old way. But it s understood in the kernel community that it s one of the bugs that will have to be fixed before the driver can move out of staging . Raphael: Do you believe that Debian has done enough to make it easy for users to install the non-free firmwares that they need? Ben: The installer, the Linux binary packages and initramfs-tools will warn about specific files that may be needed but are missing. Users who have enabled the non-free section should then be able to find the necessary package with apt-cache search, because each of the
binaries built from the firmware-nonfree source package includes driver and file names within its description. For the installer, there is a single tarball that provides everything. We could make this easier, but I think we have gone about as far as we can while following the Debian Social Contract and Debian policy. Raphael: At some point in the past, the Debian kernel team was not working very well. Did the situation improve? Ben: Back in 2008 when I started working on the Linux kernel package to sort out the firmware issues, I think there were some problems of communication and coordination, and quite possibly some members were burned-out. Since then, many of the most active kernel team members have been able to meet face-to-face to discuss future plans at LPC 2009 in Portland and the 2010 mini-DebConf in Paris. We generally seem to have productive discussions on the debian-kernel mailing list and elsewhere, and I think the team is working quite well. Several new contributors have joined after me. I would say our biggest problem today is that we just don t have enough time to do all we want to. Certainly, almost all my Debian time is now taken up with integrating upstream kernel releases and handling some fraction of the incoming bug reports. Occasionally I can take the time to work on actual features or the other packages I m neglecting!

Our biggest problem today is that we just don t have enough time to do all we want to.

Raphael: It is widely known that Linux is maintained in a git repository. But the Debian kernel team is using Subversion. I believe a switch is planned. Why was not git used from the start? Ben: The linux-2.6 source package dates from the time when Linus made his first release using git. I wasn t part of the team back then so I don t know for sure why it was imported to Subversion. However, at that time hardly anyone knew how to use git, no-one had experience hosting public git repositories, and Alioth certainly didn t offer that option. Today there are no real blockers: everyone on the kernel team is familiar with using git; Alioth is ready to host it; we don t have per-architecture patches that would require large numbers of branches. But it still takes time to plan such a conversion for what is a relatively complex source package (actually a small set of related source packages). Raphael: What are your plans for Debian Wheezy? Ben: Something I ve already done, in conjunction with the installer team, is to start generating udebs from the linux-2.6 source package. The kernel and modules have to be repacked into lots of little udebs to avoid using too much memory during installation. The configuration for this used to be in a bunch of separate source packages; these could get out of step with the kernel build configuration and this would only be noticed some time later. Now we can update them both at the same time, they are effectively cross-checked on every upload, and the installer can always be built from the latest kernel version in testing or unstable. I think that we should be encouraging PC users to install the 64-bit build (amd64), but many users will still use 32-bit (i386) for backward compatibility or out of habit. On i386, we ve slightly reduced the variety of kernel flavours by getting rid of 686 and making 686-pae the default (previously this was called 686-bigmem ). This means that the NX security feature will be used on all systems that support it. It should also mean that the first i386 CD can have suitable kernel packages for all systems. I have been trying to work on providing a full choice of Linux Security Modules (LSMs). Despite their name, they cannot be built as kernel modules, so every enabled LSM is a waste of memory on the systems that don t use it. This is a significant concern for smaller Debian systems. My intent is to allow all unused LSMs to be freed at boot time so that we can happily enable all of them. I recently proposed to drop support for older x86 systems, starting with 486-class processors in wheezy. In general, this would allow the use of more compiler optimisations throughout userland and the kernel. However it seems that there isn t that much to be gained unless we also drop 586-class processors, and there are still quite a few of those in use. So I think this will have to wait. Uwe Kleine-K nig has been working to include real-time support (also known as PREEMPT_RT). This can provide low and very predictable I/O latency, which is useful for live audio synthesis, for example. It still requires a number of patches and a build configuration change, resulting in a separate binary package. We re currently only building that for 64-bit PCs. (You may notice this is missing for Linux 3.1, because the real-time developers skipped this release.) Raphael: What s the biggest problem of Debian? Ben: I think we try too hard to accommodate every possible option, without regard for the cost to developers and users in general. As an example, we now have sysvinit, file-rc, upstart and systemd all in testing. Daemon maintainers can t rely on any advanced features of upstart or systemd because we refuse to choose between them. And the decision to support the FreeBSD kernel means that we cannot choose upstart or systemd as the only option. So all daemon maintainers will have to maintain those baroque init scripts for the indefinite future. We really should be able to decide as a distribution that when one option is technically good and popular then it can be made the only option. But no-one really has the authority to do that, so we muddle along with the pretence that all the options are equally valid and functional, while none of them is supported as well as they should be.

We try too hard to accommodate every possible option, without regard for the cost to developers and users in general.

We also try to build every package on every architecture, in general. I m quite sure there are many (package, architecture) combinations that have no users, ever. But if at some point that combination FTBFS, developers will waste time investigating and fixing that time that could have been spent working on bugs and features that users actually care about. Yes, sure, portability is good but you can t prove portability just by making a package compile on every architecture. This also applies to the selection of drivers for the kernel, by the way. Raphael: Is there someone in Debian that you admire for their contributions? Well, there are many people, but I will pick out just a few: Steve McIntyre, for his work as DPL to improve communication with the various Debian derivatives and to bring fresh blood into various core teams. Also for being a generous host for countless Debian social and bug-squashing events. Stefano Zacchiroli, for improving further on communications with both downstream and upstream projects, and for regularly exercising his power to lead discussions to the benefit of the project. Julien Cristau, for maintaining good humour while not only fighting against the tide of graphics driver regressions in X and the Linux kernel but also working on release management. Jonathan Nieder, for taking on the unglamorous and frustrating task of kernel bug triage as a non-maintainer and developing it to a fine art.

---

Thank you to Ben for the time spent answering my questions. I hope you enjoyed reading his answers as I did.

Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Google+, Twitter and Facebook .

4 comments Liked this article? Click here. My blog is Flattr-enabled.

Screensaver control on the Linux desktop is a total mess, but the xdg-screensaver command provided an interim cross-desktop solution by detecting and using the appropriate underlying interfaces. For GNOME, it depends on the gnome-screensaver-command --poke option, but that has been broken and then removed in favour of the cross-desktop D-Bus API org.freedesktop.ScreenSaver another GNOME-specific API which can't practically be used by xdg-screensaver. What's more, switching to the new API seems to have introduced various bugs in GNOME's own video player, Totem, such that it often failed to inhibit the screensaver in GNOME 2.32 and 3.0. Apparently everything is wonderful again in 3.2; here's hoping. (In case anyone asks where my bug report on the totem package is, I already looked at the bug list and it's clear that there's no point.) Updated: Well I said the GNOME Session D-Bus API couldn't practically be used in xdg-screensaver, but then set out to prove myself wrong. And with the aid of Perl, I may have succeeded.

We all love stats, don t we? So, here we go! Let s start with a graph: It shows the number of packages in the NEW queue since last year. You can see a big drop during April 2011, and a reasonably low rate during the last six months. You could think fellow Debian Developers stopped to upload NEW packages. Sorry, you re wrong! Since Squeeze release, 3.832 .changes files with NEW components were processed by dak, with an average of 14,85 NEW packages per day. On the FTP Team side, we had 3.732 accepts (14,47 per day), 339 rejects (1,31 per day) and 178 comments to maintainers (0,69 per day).

---

Who were the most prolific maintainers who got a NEW processing? Here is our special top ten:

1. Debian Haskell Group (362 packages)
2. Debian Perl Group (343 packages)
3. Debian Java Maintainers (161 packages)
4. Debian Ruby Extras Maintainers (124 packages)
5. Debian Multimedia Maintainers (100 packages)
6. Debian Fonts Task Force (96 packages)
7. Debian Med Packaging Team (79 packages)
8. Debian Install System Team (61 packages)
9. Debian Javascript Maintainers (54 packages)
10. Debian Python Modules Team (50 packages)

That s bad packaging teams cannot bake cookies!
Let s do the same with Changed By, this time:

1. Ben Hutchings (159 packages)
2. Joachim Breitner (138 packages)
3. Clint Adams (134 packages)
4. Jonas Smedegaard (124 packages)
5. TANIGUCHI Takaki (97 packages)
6. Nicholas Bamber (61 packages)
7. Alessio Treglia (60 packages)
8. maximilian attems (54 packages)
9. David Paleino (51 packages)
10. Torsten Werner (45 packages)

Much better now go and heat up your ovens, we know who you are

---

Another nice aspect to look at is the speed of NEW processing. Some maintainers were very happy for a fast NEW processing, someone even complained for having been too quick! So, let s find out which upload was the quickest ever. Try to gamble a bit before reading the answer, to see whether you are near to the real value Alessio Treglia, you probably already know, because your gwc_0.21.16~dfsg-1 upload has been processed in 41 seconds (yes, forty-one seconds!). Here s an excerpt from ftp-master log to certify it:

20110516120252 process-upload dak Processing changes file gwc_0.21.16~dfsg-1_amd64.changes
20110516120258 process-upload dak Moving to new gwc_0.21.16~dfsg-1_amd64.changes
20110516120339 process-new tolimar NEW ACCEPT: gwc_0.21.16~dfsg-1_amd64.changes

Alex was the super-fast FTP Team member behind the quickest accept, do you want to beat him? Join FTP Team

I mentioned this briefly yesterday, but now I'll try to summarize the story of a great surprise and a big moment for me. All this started when my wife Elizabeth and my son Jean-Baptiste wanted to do something special for my 50th birthday. So, it indeed all started months ago, probably early March or something (I don't yet have all the details). Jean-Baptiste described this well on the web site, so I won't go again into details, but basically, this was about getting birthday wishes from my "free software family" in, as you might guess, as many languages as possible. Elizabeth brought the original idea and JB helped her by setting up the website and collecting e-mail addresses of people I usually work with: he grabbed addresses from PO files on Debian website, plus some in his own set of GPG signatures and here we go. And then he started poking dozens of you folks in order to get your wishes for this birthday. Gradually, contributions accumulated on the website, with many challenges for them: be sure to get as many people as possible, poking and re-poking all those FLOSS people who keep forgetting things... It seems that poking people is something that's probably in the Perrier's genes! And they were doing all this without me noticing. As usually in Debian, releasing on time is a no-no. So, it quickly turned out that having everything ready by April 2nd wouldn't be possible. So, their new goal was offering this to me on Pentecost Sunday, which was yesterday. And...here comes the gift. Aha, this looks like a photo album. Could it be a "50 years of Christian" album? But, EH, why is that pic of me, with the red Debconf5 tee-shirt (that features a world map) and a "bubulle" sign, in front of the book? But, EH EH EH, what the .... are doing these word by H0lger, then Fil, then Joey doing on the following pages? And only then, OMG, I discover the real gift they prepared. 106, often bilingual, wishes from 110 people (some were couples!). 18 postcards (one made of wood). 45 languages. One postcard with wishes from nearly every distro representatives at LinuxTag 2011. Dozens of photos from my friends all around the world. All this in a wonderful album. I can't tell what I said. Anyway, JB was shooting a video, so...we'll see. OK, I didn't cry...but it wasn't that far and emotion was really really intense. Guys, ladies, gentlemen, friends....it took me a while to realize what you contributed to. It took me the entire afternoon to realize the investment put by Elizabeth and JB (and JB's sisters support) into this. Yes, as many of you wrote, I have an awesome family and they really know how to share their love. I also have an awesome virtual family all around the world. Your words are wholeheartedly appreciated and some were indeed much much much appreciated. Of course, I'll have the book in Banja Luka so that you can see the result. I know (because JB and Elizabeth told me) that many of you were really awaiting to see how it would be received (yes, that includes you, in Germany, who I visited in early May!!!). Again, thank you so much for this incredible gift. Thank you Holger Levsen, Phil Hands, Joey Hess, Lior Kaplan, Martin Michlmayr, Alberto Gonzalez Iniesta, Kenshi "best friend" Muto, Praveen Arimbrathodiyil, Felipe Augusto van de Wiel, Ana Carolina Comandulli (5 postcards!), Stefano Zacchiroli (1st contribution received by JB, of course), Gunnar Wolf, Enriiiiiico Zini, Clytie Siddall, Frans Pop (by way of Clytie), Tenzin Dendup, Otavio Salvador, Neil McGovern, Konstantinos Margaritis, Luk Claes, Jonas Smedegaard, Pema Geyleg, Meike "sp tzle queen" Reichle, Alexander Reichle-Schmehl, Torsten Werner, "nette BSD" folks, CentOS Ralph and Brian, Fedora people, SUSE's Jan, Ubuntu's Lucia Tamara, Skolelinux' Paul, Rapha l Hertzog, Lars Wirzenius, Andrew McMillan (revenge in September!), Yasa Giridhar Appaji Nag (now I know my name in Telugu), Amaya Rodrigo, St phane Glondu, Martin Krafft, Jon "maddog" Hall (and God save the queen), Eddy Petri or, Daniel Nylander, Aiet Kolkhi, Andreas "die Katze geht in die K che, wunderbar" Tille, Paul "lets bend the elbow" Wise, Jordi "half-marathon in Banja Luka" Mallach, Steve "as ever-young as I am" Langasek, Obey Arthur Liu, YAMANE Hideki, Jaldhar H. Vyas, Vikram Vincent, Margarita "Bronx cross-country queen" Manterola, Patty Langasek, Aigars Mahinovs (finding a pic *with* you on it is tricky!), Thepittak Karoonboonyanan, Javier "nobody expects the Spanish inquisition" Fern ndez-Sanguino, Varun Hiremath, Moray Allan, David Moreno Garza, Ralf "marathon-man" Treinen, Arief S Fitrianto, Penny Leach, Adam D. Barrat, Wolfgang Martin Borgert, Christine "the mentee overtakes the mentor" Spang, Arjuna Rao Chevala, Gerfried "my best contradictor" Fuchs, Stefano Canepa, Samuel Thibault, Eloy "first samba maintainer" Par s, Josip Rodin, Daniel Kahn Gillmor, Steve McIntyre, Guntupalli Karunakar, Jano Gulja , Karolina Kali , Ben Hutchings, Matej Kova i , Khoem Sokhem, Lisandro "I have the longest name in this list" Dami n Nicanor P rez-Meyer, Amanpreet Singh Alam, H ctor Or n, Hans Nordhaugn, Ivan Mas r, Dr. Tirumurti Vasudevan, John "yes, Kansas is as flat as you can imagine" Goerzen, Jean-Baptiste "Piwet" Perrier, Elizabeth "I love you" Perrier, Peter Eisentraut, Jesus "enemy by nature" Climent, Peter Palfrader, Vasudev Kamath, Miroslav "Chicky" Ku e, Mart n Ferrari, Ollivier Robert, Jure uhalev, Yunqiang Su, Jonathan McDowell, Sampada Nakhare, Nayan Nakhare, Dirk "rendez-vous for Chicago marathon" Eddelbuettel, Elian Myftiu, Tim Retout, Giuseppe Sacco, Changwoo Ryu, Pedro Ribeoro, Miguel "oh no, not him again" Figueiredo, Ana Guerrero, Aur lien Jarno, Kumar Appaiah, Arangel Angov, Faidon Liambotis, Mehdi Dogguy, Andrew Lee, Russ Allbery, Bj rn Steensrud, Mathieu Parent, Davide Viti, Steinar H. Gunderson, Kurt Gramlich, Vanja Cvelbar, Adam Conrad, Armi Be irovi , Nattie Mayer-Hutchings, Joerg "dis shuld be REJECTed" Jaspert and Luca Capello. Let's say it gain:

• Danke
• Thank you
• (RTL or not?) / toda
• Gracias
• / arigat
• / n ndi
• Obrigado
• Grazie
• C m n
• Kadin chey la
• / Efharisto
• Dank je wel
• Tak
• D ngge
• Merci
• Kiitos
• Tika hoki
• / dhanyavaadhamulu
• Thank you and God save the Queen
• Mul umesc
• Tack
• / didi madloba
• Gr cies
• Dakujem
• / Aabhar
• / Dhanyavadagalu
• Paldies
• / kop khun krap
• Terima kasih
• Hvala (hr)
• / Dhanyavad
• / Hvala (sr)
• Hvala (sl)
• / Akun
• / Sukriya
• Takk
• / Nandri
• Maki biyen jianbakik
• D kuji
• / Xiexie
• / Aabhari aahe
• Faleminderit
• / Kam sah ham ni da
• / Blagodaram
• / Xiexie
• Hvala (bs)

I mentioned this briefly yesterday, but now I'll try to summarize the story of a great surprise and a big moment for me. All this started when my wife Elizabeth and my son Jean-Baptiste wanted to do something special for my 50th birthday. So, it indeed all started months ago, probably early March or something (I don't yet have all the details). Jean-Baptiste described this well on the web site, so I won't go again into details, but basically, this was about getting birthday wishes from my "free software family" in, as you might guess, as many languages as possible. Elizabeth brought the original idea and JB helped her by setting up the website and collecting e-mail addresses of people I usually work with: he grabbed addresses from PO files on Debian website, plus some in his own set of GPG signatures and here we go. And then he started poking dozens of you folks in order to get your wishes for this birthday. Gradually, contributions accumulated on the website, with many challenges for them: be sure to get as many people as possible, poking and re-poking all those FLOSS people who keep forgetting things... It seems that poking people is something that's probably in the Perrier's genes! And they were doing all this without me noticing. As usually in Debian, releasing on time is a no-no. So, it quickly turned out that having everything ready by April 2nd wouldn't be possible. So, their new goal was offering this to me on Pentecost Sunday, which was yesterday. And...here comes the gift. Aha, this looks like a photo album. Could it be a "50 years of Christian" album? But, EH, why is that pic of me, with the red Debconf5 tee-shirt (that features a world map) and a "bubulle" sign, in front of the book? But, EH EH EH, what the .... are doing these word by H0lger, then Fil, then Joey doing on the following pages? And only then, OMG, I discover the real gift they prepared. 106, often bilingual, wishes from 110 people (some were couples!). 18 postcards (one made of wood). 45 languages. One postcard with wishes from nearly every distro representatives at LinuxTag 2011. Dozens of photos from my friends all around the world. All this in a wonderful album. I can't tell what I said. Anyway, JB was shooting a video, so...we'll see. OK, I didn't cry...but it wasn't that far and emotion was really really intense. Guys, ladies, gentlemen, friends....it took me a while to realize what you contributed to. It took me the entire afternoon to realize the investment put by Elizabeth and JB (and JB's sisters support) into this. Yes, as many of you wrote, I have an awesome family and they really know how to share their love. I also have an awesome virtual family all around the world. Your words are wholeheartedly appreciated and some were indeed much much much appreciated. Of course, I'll have the book in Banja Luka so that you can see the result. I know (because JB and Elizabeth told me) that many of you were really awaiting to see how it would be received (yes, that includes you, in Germany, who I visited in early May!!!). Again, thank you so much for this incredible gift. Thank you Holger Levsen, Phil Hands, Joey Hess, Lior Kaplan, Martin Michlmayr, Alberto Gonzalez Iniesta, Kenshi "best friend" Muto, Praveen Arimbrathodiyil, Felipe Augusto van de Wiel, Ana Carolina Comandulli (5 postcards!), Stefano Zacchiroli (1st contribution received by JB, of course), Gunnar Wolf, Enriiiiiico Zini, Clytie Siddall, Frans Pop (by way of Clytie), Tenzin Dendup, Otavio Salvador, Neil McGovern, Konstantinos Margaritis, Luk Claes, Jonas Smedegaard, Pema Geyleg, Meike "sp tzle queen" Reichle, Alexander Reichle-Schmehl, Torsten Werner, "nette BSD" folks, CentOS Ralph and Brian, Fedora people, SUSE's Jan, Ubuntu's Lucia Tamara, Skolelinux' Paul, Rapha l Hertzog, Lars Wirzenius, Andrew McMillan (revenge in September!), Yasa Giridhar Appaji Nag (now I know my name in Telugu), Amaya Rodrigo, St phane Glondu, Martin Krafft, Jon "maddog" Hall (and God save the queen), Eddy Petri or, Daniel Nylander, Aiet Kolkhi, Andreas "die Katze geht in die K che, wunderbar" Tille, Paul "lets bend the elbow" Wise, Jordi "half-marathon in Banja Luka" Mallach, Steve "as ever-young as I am" Langasek, Obey Arthur Liu, YAMANE Hideki, Jaldhar H. Vyas, Vikram Vincent, Margarita "Bronx cross-country queen" Manterola, Patty Langasek, Aigars Mahinovs (finding a pic *with* you on it is tricky!), Thepittak Karoonboonyanan, Javier "nobody expects the Spanish inquisition" Fern ndez-Sanguino, Varun Hiremath, Moray Allan, David Moreno Garza, Ralf "marathon-man" Treinen, Arief S Fitrianto, Penny Leach, Adam D. Barrat, Wolfgang Martin Borgert, Christine "the mentee overtakes the mentor" Spang, Arjuna Rao Chevala, Gerfried "my best contradictor" Fuchs, Stefano Canepa, Samuel Thibault, Eloy "first samba maintainer" Par s, Josip Rodin, Daniel Kahn Gillmor, Steve McIntyre, Guntupalli Karunakar, Jano Gulja , Karolina Kali , Ben Hutchings, Matej Kova i , Khoem Sokhem, Lisandro "I have the longest name in this list" Dami n Nicanor P rez-Meyer, Amanpreet Singh Alam, H ctor Or n, Hans Nordhaugn, Ivan Mas r, Dr. Tirumurti Vasudevan, John "yes, Kansas is as flat as you can imagine" Goerzen, Jean-Baptiste "Piwet" Perrier, Elizabeth "I love you" Perrier, Peter Eisentraut, Jesus "enemy by nature" Climent, Peter Palfrader, Vasudev Kamath, Miroslav "Chicky" Ku e, Mart n Ferrari, Ollivier Robert, Jure uhalev, Yunqiang Su, Jonathan McDowell, Sampada Nakhare, Nayan Nakhare, Dirk "rendez-vous for Chicago marathon" Eddelbuettel, Elian Myftiu, Tim Retout, Giuseppe Sacco, Changwoo Ryu, Pedro Ribeoro, Miguel "oh no, not him again" Figueiredo, Ana Guerrero, Aur lien Jarno, Kumar Appaiah, Arangel Angov, Faidon Liambotis, Mehdi Dogguy, Andrew Lee, Russ Allbery, Bj rn Steensrud, Mathieu Parent, Davide Viti, Steinar H. Gunderson, Kurt Gramlich, Vanja Cvelbar, Adam Conrad, Armi Be irovi , Nattie Mayer-Hutchings, Joerg "dis shuld be REJECTed" Jaspert and Luca Capello. Let's say it gain:

• Danke
• Thank you
• (RTL or not?) / toda
• Gracias
• / arigat
• / n ndi
• Obrigado
• Grazie
• C m n
• Kadin chey la
• / Efharisto
• Dank je wel
• Tak
• D ngge
• Merci
• Kiitos
• Tika hoki
• / dhanyavaadhamulu
• Thank you and God save the Queen
• Mul umesc
• Tack
• / didi madloba
• Gr cies
• Dakujem
• / Aabhar li> / Dhanyavadagalu
• Paldies
• / kop khun krap
• Terima kasih
• Hvala (hr)
• / Dhanyavad
• / Hvala (sr)
• Hvala (sl)
• / Akun
• / Sukriya
• Takk
• / Nandri
• Maki biyen jianbakik
• D kuji
• / Xiexie
• / Aabhari aahe
• Faleminderit
• / Kam sah ham ni da
• / Blagodaram
• / Xiexie
• Hvala (bs)

I have a talk to give, hacking to do, and I should be helping with video as well. Hope to see you in Banja Luka!