This is a long story. I wanted to document which tools I used on my desktop, but then realized it was difficult to do this without an historical context. This therefore, became a history of my computer usage and programming habits.

Before the desktop: vic 20 and apple My first contacts with computers were when my uncle showed me his Apple computers at his home. He was working at UQAM and had access to all sorts of crazy computers. My memory is vague, but I believe he showed me a turtle moving around the screen, which I suspect was Logo, and some rabbit game which I was quite fond of. Later, I hooked up with a Vic-20 (similar to a Commodore 64) and typed in my first programs. Back then, you would painstakingly transcribe the program from a book into the screen, type RUN at the end and really hope that you didn't make a typo. You could also read and write data off audio tape with an external tape drive, which was really amazing. To put things into perspective, the Vic-20 would plug into your television for a display, and the computer was "in the keyboard". Old school shit. The language was BASIC, which was awful. Later, we got luxurious and had the privilege of a Macintosh Plus computer. 3000$ plus the printer, that was quite something. Still, that modern system could run Word, had games, and a 9" black and white screen. But the best part, and where I really started to learn to program, is with Hypercard, the crazy software written by Bill Atkinson that became the inspiration for wikis, the Myst computer game and, to a certain extent, popularized the concept of hyperlinking, which is the basic principle of the World Wide Web.

The early years: afterstep and FreeBSD But my real computer life all started with FreeBSD 2.2.x (probably 2.2.5), around 1997, when I installed my first FLOSS operating system. This was just amazing: source code for everything, down to the compilers themselves, I was amazed, since I was coming from a Mac (still had that Mac Plus lying around) and Windows (95!) world. Back then, X11 (XFree86, it was called) was a fantasy. Something I would compile with the FreeBSD ports system and sometimes would be able to run. Window managers were some weird things I wasn't sure how to use, but I figured out twm (yuck!) but was still amazed. At the university, we had SGI workstations (remember the Indigo machines?) on which I started setting up the Afterstep window manager, which I used for a long time. The web browser was Netscape Communicator 4, and lo and behold, it also did email, although I mostly used Pine over a modem terminal to the university. Internet access at home was spotty and was dependent on students compiling discreetly copies of slirp which were allowing us, in a really clunky way, to run a PPP session over those terminal lines. That actually got my account suspended because I messed up my sendmail configuration and was sending emails that appeared to be coming as , which freaked out the sysadmins that be over there. The hardware was a Pentium 166Mhz with 32MB of ram and 20GB of disk space, which was a huge improvement over the 1MB of ram and 20MB of (external!) hard disk that the Mac Plus was proud of. By then, I started programming in more languages: I learned the basics of shell scripting, C programming and I had learned Pascal in a college class.

The real freedom: Debian and... enlightenment? I then switched to Debian, thanks to my friend scyrma, which showed me the power of dselect which, for a FreeBSD user, was really impressive. Imagine that: you could just install software and all dependencies without compiling anything! ;) I don't exactly remember when that was, but it was probably Woody. I probably still used afterstep, but at that time, changing window managers was like changing t-shirts and we would waste countless hours fiddling around with borders and background images and other silly things. I remember running Enlightenment and writing software for it, for example, but not much else. Back then, email was still over pine, but a new web browser was rearing its ugly head, Mozilla, and was quickly trimmed down to Phoenix, shortly lived as Mozilla Firebird, only to be finally renamed Mozilla Firefox, and was unleashed as a stable release to an unsuspecting Internet Explorer and Microsoft-dominated world. Throughout this chaos, I also used various Mozilla derivatives like Galeon since all of this was pretty unstable and heavy for my poor computers. This was probably around 2002 to 2005. By then, I had formal classes on programming languages, was forced to learn Java, discovered C++, and learned more C as the university assumed that you knew that language in the data structures and algorithm class, which some students really didn't find funny at all. I also learned some Prolog and Haskell back then. Through my experiences with free software, I discovered various eclectic programming languages like TCL, and kept an eye out for other languages like lisp (thanks to Emacs), Forth (thanks to the FreeBSD boot loader), HTML/CSS/Javascript (no thanks to the web). There are probably others I am forgetting here, voluntarily or not.

The years of the desktops: rox, sawfish and gnome Another stable increment on my desktop was the use of the sawfish window manager and the rox desktop, mostly because of the awesome Rox file manager that had no equal in the rest of the free software world. Gnome was an atrocity and KDE even more so: bloated and unstable. Eventually, however, KDE and Gnome not only matured, but simply took over the whole of the graphical environment psyche of the free world. People nowadays don't even know what a window manager is and assume that you now have a "desktop environment", which, for me, is a real desk with a bunch of papers, a radio transceiver and maybe a computer with a screen lying around. So I did use KDE, and then Gnome. I switched from KDE to Gnome because I felt that Gnome left me more space to run my own stuff without starting crazy daemons to preload everything. At this point, Firefox was the new thing and basically stable enough for daily use. This lasted until around 2009, I guess. On the programming side, I had picked up Perl because I had to learn it to work at Ericsson (don't ask) and learned PHP to pay my rent, which I consider one of the biggest professional mistakes I have made so far in that story. It is around that time that Koumbit was founded by me and other friends.

The tiles fall on the gnomes: awesome desktop and sysadmin programming At some point, friends (mostly comar and millette) introduced me to wmii as the best thing since sliced bread. I banged my head against it with great pains, but I really enjoyed the hands off approach to window management. Tiling window managers really got a hold on me there, and in fact we can say on the whole desktop, with recent Gnome 3 taking up some tiling concepts itself. I have since then switched to the Awesome Window Manager which is probably the worst name you can use for software (try googling for "Awesome crash" to get an idea). I have switched from Firefox to Chromium, after having repeated crashes from Firefox on a site that Chrome would load fine. I was expecting Chrome to crash only one tab, because of its design, and it turned out to not crash at all, and I have since given up on Firefox totally. I have also given up on desktop environments. While I have given them a chance in quick setups and still use them when forced to, I find them too bloated and single-minded for my needs. I want control over my desktop, and those tools work very hard to remove control and hide it behind user interfaces that treat me like a brain-dead dog. Finally, I have kept on programming in PHP for work, because that's what we do over there (unfortunately). Mostly because of Drupal and Aegir, in which I was and still am deeply involved. But I am quite fond of Python for my personnal project and still use shell scripting to get myself out of trouble or automate data processing from time to time. I have a rule though: once a program hits 100 lines of shell scripting, it needs to be rewritten, usually in Perl because the syntax is closer. During the years 2009 to 2012, I have started to work more and more on system administration however. In 2010, I became a Debian Developer and a lot of my work at Koumbit revolves around system administration, optimisation and maintenance. I work oncall and sometimes get called during the night to nurse servers (which is why I am up writing this right now).

The future of programming I have become seriously disgruntled with programming. I have seen the emergence of the web, where programmers felt it was relevant to rewrite 30 years of programs into the web framework (webmail? wtf?), and now we are seeing the same thing happen with mobile devices, the tracking devices they call phones and "pads", we should rather say. And again they are trying to make people pay for trivial software. I am tired of seeing our community reinvent the wheel every 10 years because of infatuated egos and marketing sprints. We are wasting great minds on this, a lot of energy and precious time. I would rather maintain 5 year old imperfect software than coordinate a 5 year project to rewrite that software into another imperfect software. That is what is happening right now, and it's going nowhere. So here's the current state of affairs, some perls, some tiles, but mostly system administration and less programming. Through all this time, the common pattern is that I have learned to solve problems and document my work, which are key concepts of the job of a system administrator, which is what I really am now. Somehow, that popped into my life like a natural fit. Maybe, one day, I will retire and become a shepherd in far away mountains, but I will always have in my mind the strong desire to fix shit up and make things right with the world. Maybe get involved in a project like Open Source Ecology and especially their Global Village Construction Set to make sure humankind survives all that stupidity.

The F Word has an informative post about men commenting on Feminist blogs [1]. Most of it applies to any situation where a member of a powerful group comments on an issue related to a minority group. Near the end they say: I ll also paraphrase and flesh out the most useful piece of advice I ever read, when I made the effort to research white privilege: don t expect the minority to trust you. Trust is earned, and you re just another commenter who they can t tell apart from any other commenter. You re entering someone else s space, where different rules apply. You get to have the rest of the world for people to assume you re a wonderful person. Here, you re just another one of them , and given the track record of them , it s up to you to listen, learn and prove that you re being thoughtful and honestly trying to examine your privilege. Sociological Images has an interesting article on people s perception of the sky colour it seems that the blue sky meme is modern [2]. Charles Stross wrote an interesting article about the possible uses for future low power computers [3]. He gets a bit over-excited about the possibilities for sensing making a tiny computer that can sense so many things isn t going to be easy (DSLRs are big for a reason). But having lots of powerful computers everywhere does provide lots of interesting and potentially bad opportunities. Martin Bekkelund has an interesting article about Amazon wiping DRM infected books that it had sold to a customer without giving a refund or an explanation [4]. If you want to buy ebooks it seems that the sensible thing to do would be to immediately crack them or download them from The Pirate Bay so Amazon can t steal them back. Rebecca Saxe gave an interesting talk about trying to develop methods for conflict resolution through neuroscience [5]. Barry Eisler has written an interesting article about the corruption of journalists [6]. It s really worth reading, some of the methods of corruption apply to even the more casual bloggers. Krebs on Security has an informative article about the Microsoft Tech Support phone scams [7]

• [1] http://www.thefword.org.uk/features/2008/06/men_feminism_ne
• [2] http://thesocietypages.org/socimages/2012/07/30/is-the-sky-blue
• [3] http://www.antipope.org/charlie/blog-static/2012/08/how-low-power-can-you-go.html
• [4] http://www.bekkelund.net/2012/10/22/outlawed-by-amazon-drm/
• [5] http://edge.org/conversation/imaging-conflict-resolution
• [6] http://barryeisler.blogspot.com.au/2012/08/you-will-be-assimilated.html
• [7] http://krebsonsecurity.com/2012/08/tech-support-phone-scams-surge/

I'm the maintainer of MySQL Connector/Python Debian package, and it's shipped with several unittests, but they require a MySQL instance to be run against, so up to know I wasn't able to run them at build-time.

But I want to change that, so I wanted to find a way to spawn a MySQL instance just for testing purposes. I've searched a lot, and then found a package solving the same problem: php5.

The package has a setup-mysql.sh script that takes care of starting/stopping the instances, and debian/rules file sets up instance parameters and then it starts the instance before the tests and then stops it after them.

It's a very interesting solution, so I thought of sharing + taking notes to self, just in case I need it in other situations.

This entry is not really about computers, technology, or other work-related topics, but more about a hard-hack that I wanted to try for a while. How to make a kilt !!! After a bit of duck-ducking, I decided to follow this excellent tutorial. At first sight the entire process seems a bit long, but you will realize that after the first read, that everything boils down to 3 steps: measure, fold and pin, sew. For the measure part, I've the impression that the formula that is given in the instructable (waist/3*8+1) is a bit short for my comfort and taste. This is the size for the internal apron, the folded part that goes all around your left hip, back, and right hip, and the front apron. My suggestion would be to make the inner apron a bit longer then the front apron. This way the kilt will feel in my opinion more comfortable and it will envelop you body completely. For the fold and pin part, you just need a bit of patience and a ruler. Put the pins parallel to the folding as in the instructable and not perpendicular. This will help you later when sewing everything. The sewing ... If you know how to use a sewing machine, this is going to be a piece of cake. Otherwise, well, I spent more time troubleshooting the machine then sewing the kilt. I've broken few needles in the process and learned how to thread the machine with my eyes blindfolded. Not to mention that you have to learn how to disassemble this machines in a thousand parts to understand how the thread got stuck. It was fun. A lesson that I've learned is that a sewing machine works much better in the morning than late at night when you are tired and sleepy. Really ! Other then that, it was a fun experience. Maybe I'll make another one to commit this skill to mind. Maybe I'll run a kilt making workshop at the next debconf :)

Attachment	Size
PA250002.JPG	134.19 KB
PA250003.JPG	131.11 KB

So you ve got lots of shiny spam-detection software (not eyetests or similar rubbish) installed but are still getting some spam on your email and your website? Why aren t your spam detectors and preventative measures effective at dealing with it? Basically, the spam detectors are pretty effective, but it s a problem of scale. The underlying problem is that there s so much spam now something like 73% of email is spam just now (I suspect the web is worse). I expect much of the rest is legitimate robots too, like newsletters, automated billing, or notifications about social network activity. So, we want to trap the spam, while letting humans and good robots through. We can t use physical ability tests because there are both human spammers who are paid to spam trickier sites manually, and people like me who fail things like Google s human test because we use technology to overcome our physical limitations: there are now robots that are better than me at voice recognition or passing eyetests! We try to design websites so that the return on investment for spammers is too low (don t give untrusted users outgoing links automatically, basically). Even so, when we re using some popular software like WordPress, our site settings don t give them a return, but most stupid automatic spammers don t bother to check and still have a go. After that, the main things we re trying are rules of thumb to trap spammers (which is usually enough to filter out 90% or so) and to group sites together in informal co-operative spam-fighting networks like blogspam.net, so that once a spammer is spotted, they should get blocked on lots of sites (which blocks a bit over half of the remaining 10%). Sadly, the rest gets shown to humans for decision. Real comments are so few and far between now that we really don t want to risk turning real people away and killing discussions. We used to go after spammers who got shown to humans, but there are now too many spammers and too many service providers who won t kick spammers off their services: the spammers pay them and we don t: all we could do was waste their money in support, so they stopped offering any support to non-customers. Is that a flaw in the co-operative nature of the Internet? Can we overcome it? Wish I knew

It has been brought to my attention that a company selling some so-called colon cleansing product wanted to threat with a law suit a Romanian skeptical blogger because he wrote some articles showing that any such products (the one produced by the said company is the most known/popular in Romania) are pure quackery and there is no scientific basis for the claim they make in order to promote their products.

In his articles he also explained how, in fact, the mucoid plaque, the thing that supposedly proves the efficiency of the product, it is a result of taking the product due to its ingredients, and how no such mucoid plaque was ever observed in any colonoscopy, colon surgery or any other situation where you'd expect it to be seen. He also quoted specialists and lots of other scientific references, showing an honest approach to the issue.

As a response to the initial take-down message from the company doing business with people's crap, the blogger said would like to see scientific proof for the claims made for the product, and when that was to happen, he would take down the articles and publish a correction.

The company decided that the best way to continue this was to try to make a legal threat and ask 100.000 euros (one hundred thousands Euros) as damage in a country where, according to the latest data from the National Statistics Institute, the total average monthly personal income is about 180 Euros.

The blogger, as a reply, decided the threat should be made public and wrote another article which probably made the company very unhappy, because they decided to sue Wordpress so they would take down the blog.

And that's exactly what they did, they sued Wordpress, and sent some documents to Wordpress who sent them to the blogger. Among the documents there were 4 pdf files containing each an original article (in Romanian) from the blog and only 3 pdf containg English translation for only 3 of them. The one missing was the one where the blogger himself showed there wasn't any legal basis for the threats they made initially against him.

Here are the translations (ironically, made on the company's own expense):

Initial article entitled "ColonHelp doesn't help the colon. But it empties your wallet!" (original here)
Initial Article.en<iframe class="scribd_iframe_embed" data-aspect-ratio="0.701030927835051" data-auto-height="true" frameborder="0" height="600" id="doc_83859" scrolling="no" src="http://www.scribd.com/embeds/109756494/content?start_page=1&amp;view_mode=scroll&amp;access_key=key-19l4sbejfk4l4na4feoe" width="100%"></iframe>



The second article entitled "Again about ColonHelp and intestinal cleansers" (original here)
Follow Up.en<iframe class="scribd_iframe_embed" data-aspect-ratio="0.70264064293915" data-auto-height="true" frameborder="0" height="600" id="doc_46722" scrolling="no" src="http://www.scribd.com/embeds/109757646/content?start_page=1&amp;view_mode=scroll&amp;access_key=key-1d0z2779liaialmv90r5" width="100%"></iframe>



** Missing translation of the first reply to threats (original Romanian text here)



The second article about the threats entitled "People who clean the colon have filled the fan with shit" (original here)
Threats 2<iframe class="scribd_iframe_embed" data-aspect-ratio="0.705069124423963" data-auto-height="true" frameborder="0" height="600" id="doc_72956" scrolling="no" src="http://www.scribd.com/embeds/109758037/content?start_page=1&amp;view_mode=scroll&amp;access_key=key-2d3c3eq1d6lo313fogho" width="100%"></iframe>


The Romanian blogger explains himself more of the details on this issue in his latest article on his blog.

The company is called Zenyth Pharmaceuticals and Wordpress will probably lose the lawsuit by not presenting themselves in any way in the Romanian courts, but I think some Streisand effect would really help the asses of this company to get them kicked in their rightful place, at the top of the hall of shame.

The product name is called ColonHelp.

Please spread this information as wide as possible.
Do NOT link to the company's site (it would raise its search engine rank), but link to the blogger' article or the translations.


If any Romanian speaker cares to translate the untranslated article and publish it somewhere on the web, I would be more than glad to update this article and add a link to that translation.

It's quite some time since I didn't report about Debian Installer localization. Indeed, thanks to the tireless activity of Cyril Brulebois, we focused on releasing D-I and I indeed stopped harassing translators for updates around July 2012. Why July? Well, you probably know we have "something" to release as soon as we can and most, if not all, Debian energies should be focused on this! Or, more precisely speaking, I only focused on getting the newly introduced material translated: we had several changes recently in D-I, mostly focused on important features, such as IPv6 support (thanks to Phil Kern who worked on this), better wireless networking support (thanks to Sorina Sandu) and EFI boot support (thanks to Steve McIntyre). And, of course, when people change code in D-I, they want to add questions to users, display error and informative messages, etc, etc. And these need translations..:-) Most translators coped with all this (sometimes with /me hitting them hard on the head to get updates) and D-I beta2 was released with 37 complete translations out of 73 supported languages. Yesterday, I just resumed the activity of trying to get more updates not only for those recent changes, but also for other older changes...or for languages that never got completed in the past. As a result, we bumped from 37 compelte languages up to 45 this morning: look for level 1 here (level 2 has been hit by a change in iso-codes, but that change doesn't really affect D-I). If you language is not 100% in the leftmost column on the stats page, you can probably help. Just get in touch with me and we'll check if somebody is already working on this or not.

(This is something I did a few months ago, and I really should have noted down all the details then so I covered everything. However hopefully these notes will remind me enough next time.) When I first wanted to reverse engineer a USB device that only had Windows drivers the "easy" option was to take a Windows machine, install usbsnoop on it and capture the traffic as a bunch of verbose text files. It was a cumbersome procedure. Recently I wanted to do a firmware upgrade of a ZTE 3G modem dongle, partly to provider unlock it and partly to try and enable some voice functionality. I was also hoping to sniff the traffic to see how to drive the voice side of things. These days I don't have a dedicated Windows box, but I do have a Windows 7 KVM virtual machine. I hadn't yet used the USB support in this, but I thought I'd see what it was capable of. Firstly I had to explicitly enable USB2 - the device wasn't happy with the default USB1 only stack that KVM enabled. That involved passing -usb -device usb-ehci,id=ehci to KVM. I also told KVM to grab all the ZTE devices with -device usb-host,bus=ehci.0,vendorid=0x19d2. I dropped a udev rules file into /etc/udev/rules.d to ensure any device nodes created belonged to my normal user. This gave me a Windows setup that could see the USB dongle and install the appropriate drivers. It was also happy to do the firmware update (along with various device resets on the way as it changed USB ID - this is why I needed the udev rule, to ensure every time the device re-appeared it would be seen by the KVM instance without manual intervention). After that was complete I investigated usbmon. modprobe usbmon created the appropriate /dev/usbmon<n> files and I chowned the appropriate bus to my normal user. Once this was done I was able to start Wireshark which rather nicely has full USB decoding support. Firing up the custom app in the Windows KVM guest I could see the traffic going back and forth to each of the device endpoints and work out what was going on. All of this was much more flexible than using a standalone Windows box. Once I figured out that I needed to explicitly enable USB2 I was quite pleased with how simple USB access under KVM was. I also used lvm to create a snapshot of the Windows guest so at the end I could roll back all of the drivers I'd installed for the dongle. And being able to use Wireshark instead of trawling through dense text files helped a lot in seeing the command stream.

Problem #2: How to add a wireless module to a d-i image? Slightly longer version: As reported in #686605, a whole wireless module family is missing from the linux kernel udebs, meaning they re missing from the d-i images. Please note the hypothesis here is that the regular linux kernel images contain the relevant modules, only the udebs don t. Recipe #2: Craft a local udeb!

1. The following assumes one is interested in adding a wireless module, but that can be adapted to other module families. The relevant udeb base name is nic-wireless-modules.
2. When it comes to linux kernel modules, picking the right ones is important, so one has to consider the ABI. Right now, the current ABI in testing is 3.2.0-3-amd64. The complete udeb name is therefore nic-wireless-modules-3.2.0-3-amd64-di.
3. If you re running a kernel with the same ABI, all good, relevant modules should be available under /lib/modules/$(uname -r), which is what will be assumed below. Otherwise, one would have to download a proper linux-image-$ABI package, and extract it into a temporary directory to grab the wanted module(s).
4. Determine the relevant module and its dependencies. As an example, I ll pick a Realtek module: rtl8192cu.ko. If it s loaded, one can use the following to detect its dependencies:

   $ lsmod grep rtl8192cu
   rtl8192cu              78863  0
   rtlwifi                81350  1 rtl8192cu
   rtl8192c_common        52602  1 rtl8192cu
   mac80211              192768  4 iwlwifi,rtl8192c_common,rtlwifi,rtl8192cu
   cfg80211              137140  3 mac80211,iwlwifi,rtlwifi
   usbcore               128498  7 ehci_hcd,usbhid,uvcvideo,rtlwifi,rtl8192cu
   

   But it isn t even needed to load the module to get those dependencies:

   $ /sbin/modinfo /lib/modules/3.2.0-3-amd64/kernel/drivers/net/wireless/rtlwifi/rtl8192cu/rtl8192cu.ko grep ^depends
   depends:        rtlwifi,mac80211,rtl8192c-common,usbcore
   

   Compared to what the d-i image contains, only rtlwifi and rtl8192c-common are missing.
5. Fetch the udeb using http://packages.debian.org/ or browsing your favourite mirror if you know the paths by heart.
6. Extract the udeb into a temporary data/ directory, along with its meta data:

   dpkg -x nic-wireless-modules-3.2.0-3-amd64-di_3.2.23-1_amd64.udeb data
   dpkg --control nic-wireless-modules-3.2.0-3-amd64-di_3.2.23-1_amd64.udeb data/DEBIAN
   

7. Copy the 3 modules under the relevant subdirectory under data/.
8. Update the version in data/DEBIAN/control, for example by adding the +local1 suffix.
9. Build a new binary package, using fakeroot to avoid owner/group noise in the diff:

   fakeroot dpkg -b data nic-wireless-modules-3.2.0-3-amd64-di_3.2.23-1+local1_amd64.udeb
   

10. Use debdiff to check the results:

    $ debdiff nic-wireless-modules-3.2.0-3-amd64-di_3.2.23-1*_amd64.udeb
    Files in second .deb but not in first
    -------------------------------------
    -rw-r--r--  root/root   /lib/modules/3.2.0-3-amd64/kernel/drivers/net/wireless/rtlwifi/rtl8192cu/rtl8192c-common.ko
    -rw-r--r--  root/root   /lib/modules/3.2.0-3-amd64/kernel/drivers/net/wireless/rtlwifi/rtl8192cu/rtl8192cu.ko
    -rw-r--r--  root/root   /lib/modules/3.2.0-3-amd64/kernel/drivers/net/wireless/rtlwifi/rtlwifi.ko
    Control files: lines which differ (wdiff format)
    ------------------------------------------------
    Version: [-3.2.23-1-]  +3.2.23-1+local1+ 
    

11. Put the resulting udeb under localudebs/ in your d-i build tree, and rebuild your image as usual. Almost-tada!

Why almost ? Because a firmware is needed, so one can go back to the 7th step, extract the proper file (lib/firmware/rtlwifi/rtl8192cufw.bin) from the non-free firmware-realtek package, and add it under data/lib/firmware/realtek/. Rebuild the udeb, rebuild the d-i image. Then: tada!

Foreword: This is the first blog post of a mini-series, trying to document some recipes about Debian Installer hacking. VirtualBox will probably be mentioned on a regular fashion, but most of those recipes should be easily adaptable to qemu/kvm. Problem #1: How to share some d-i logs without using the network? Slightly longer version: Investigating some issues on a wired-less virtual machine (Network Adapter 1 Disable Network Adapter), it s hard to get log files out of the virtual machine. Recipe #1: Use serial ports! Serial Ports Port 1 Enable Serial Port, then pick Raw File as Port Mode, and specify a file on the host system to gather logs. There s nothing to do on d-i side, the serial console appears automatically as /dev/ttyS0. From a console, one can extract the needed bits:

cp /var/log/syslog /dev/ttyS0
dmesg > /dev/ttyS0

Just about three years ago, I created the automatic rule-based time tracker (arbtt), a tool that quietly runs in the background, records your current windows and their titles, and allows you to later process this data using, well, rules. By now, I have almost half a million records, spanning 317 days of me working at the computer. Processing this amount of data has become very memory-hungry: It takes 1G of RAM to process my data. Other users, with a presumably higher sampling rate, report even worse performance, which finally made me approach the problem again. Naturally, one wonders why it is such a memory hog, after all it takes just one run over the data to produce a report. The data itself is a lazy list so the garbage collector should the be able to drop the seen elements immediately. But arbtt is able to print several reports in one run; there is even an option (--each-category) that creates a number of reports based on the data itself. The code was structured that first a number of values that are possibly used by several reports are first created one and then re-used by the reports. And then there is the issue of the filters: The user specifies a what predicates are interesting (e.g., those where he was active) and some reports need to know not only which samples were matched, but also distinguish the consecutive runs of selected samples. Lots of reasons that made it hard to not use the list of of samples several times, which causes it to be completely retained in memory. The idea of completely rewriting everything in a much more imperative way was not appealing. But luckily Haskell s good abstraction possibilities can help here. What is the correct way to traverse a lazy list and calculate some information about it? It is a strict left fold, i.e. an algorithm that can be represented as arguments to

foldl' :: (s -> x -> s) -> s -> [x] -> s

For those who are new to folds: The first argument is a function that processes one element from the list, returning the new state of the processing. The second argument is the initial state. And foldl' can apply such an algorithm to a list to return a final state. So obviously I had to express my reports as left folds. To think clearer about it, I gave them a name. I also added a finishing step to the mix, which is handy:

data LeftFold x a = forall s. LeftFold  
  start :: s,
  process :: s -> x -> s,
  finish :: s -> a
   

A value of type LeftFold x a is then an algorithm that traverses a list of xes and returns something of type a. The forall s. means that the user of such a processor must not make any assumptions about the type of the state. I also have a function that runs such a list processor, simply by invoking foldl' with the arguments stored in the LeftFold constructor:

runLeftFold :: LeftFold x a -> [x] -> a
runLeftFold (LeftFold st1 p1 f1) xs = f1 (foldl' p1 st1 xs)

Now, as a Haskell programmer, I am searching for monads everywhere, and it really would have been nice if this were a monad, as they are very easy to compose. But unfortunately, they do not form a monad. If they did then I could decide after running one processor and looking at the result how I would want to process the list now, but this obviously contradicts with the desire to traverse the list only once. But there is the monad s smaller ugly sibling, the applicative functor. One can still somewhat nicely compose them to form more complex algorithms, just with more restrictions. Here is my instance:

instance Functor (LeftFold x) where
  fmap f (LeftFold st1 p1 f2) = LeftFold st1 p1 (f . f2)
instance Applicative (LeftFold x) where
  pure x = LeftFold () const (const x)
  LeftFold st1 p1 f1 <*> LeftFold st2 p2 f2 = LeftFold  
  start  =  st1 :!: st2,
  process = \(s1 :!: s2) x -> p1 s1 x :!: p2 s2 x,
  finish  = \(s1 :!: s2)  -> f1 s1 (f2 s2)
   

The :!: operator is a strict tuple constructor. This is important, as otherwise the fold would be come a lazy one, I would be building up huge numbers of thunks and performance would be very bad. For the same reason every list processor should make sure that its state does not build thunks once it is in weak head normal form. Now I am at a abstraction level that I feel comfortable working in, by writing functions that combine the list processors in various ways. Probably the most important one for me is already present in the libraries (here with a specialized type):

sequenceA :: [LeftFold x a] -> LeftFold x [a]

This is the whole trick: Now I can, based on the user input, run any number and selection of reports in one single list traversal. I find it very elegant that under the hood , the strict tuples are nested at precisely the right depth depending on the length of the list. As hinted at before, I do some interesting stuff that was previously represented as lists of lists of values. Expressing these as left folds was a bit tricky, but again, the complexity is contained in the combinators that I wrote (although it still gets confusing, check out processIntervalReport in Stats.hs):

monoidFold :: Monoid m => LeftFold m m
mapElems :: LeftFold y a -> (x -> y) -> LeftFold x a 
filterWith :: (x -> Bool) -> LeftFold (Bool :!: x) a -> LeftFold x a
onSelected :: LeftFold x a -> LeftFold (Bool :!: x) a
onJusts :: LeftFold x a -> LeftFold (Maybe x) a
onAll :: LeftFold x a -> LeftFold (Bool :!: x) a
runOnGroups :: (x -> x -> Bool) -> LeftFold x y -> LeftFold y z -> LeftFold x z
runOnIntervals :: LeftFold x y -> LeftFold y z -> LeftFold (Bool :!: x) z
lfLength :: LeftFold x Int
lfFirst :: LeftFold x (Maybe x)
lfLast :: LeftFold x (Maybe x)
toList :: LeftFold x [x]
concatFold :: LeftFold [x] [x]

As a result, the program now only requires 10MB and runs 18% faster. If you wonder why, in the definition of runLeftFold, I do not apply the finish function strictly: That is intentional; I am actually passing the result of a processor that calculates some often required data (e.g. total time recorded) as an argument to all the list processors. (Near the end of the main function in stats-main.hs) This loop works fine as long as I am using these only in the finish functions.
There is one downside to this approach: I cannot easily calculate stuff on demand and share it between two reports. Previously, I could just bind this to a variable, pass it to all reports, only if at least one is interested it would be evaluated and if several are interested, the result would be shared. I do not have a good solution for this yet. The code above is in the self-contained module LeftFold.hs in the arbtt sources.
Several People have had this idea before, here is one blog post by Max Rabkin.
Flattr this post

DPL August report, posted on d-d-a a while ago (yep, I forgot to blog it up to now!, sorry for the oldies).

---

Dear project members, August has been a month with a good deal of vacations for many of us, including yours truly. Therefore the monthly report of DPL activities will be briefer than usual. Which is good, as it'll leave all my readers more time to do NMUs and fix RC bugs! Ongoing discussions

• Discussion about the new trademark policy has continued on -project. I've collected quite some feedback and summarized it in a request for change that I've passed on to SFLC. I'm waiting for a second draft from them, before reviving the thread.
• on the long-running issue of de-duplicating DebConf infrastructure wrt Debian's, I've been happy to support the proposal of moving DebConf's list to Debian's software and infrastructure. There seems to be interest and consensus, but the DebConf team could probably use a couple more helping hands to make it real
• starting from the practical need of dealing with minified JavaScript files, an interesting discussion sparkled on -devel. It quickly turned in both musing on the notion of archive-wide DFSG-freeness and some new devscript tooling by Andreas Tille. AFAIU some extra testing would be welcome

Assets

• The DFGS-free logo relicensing I've mentioned last month is now on its way. Following advice from SFLC, I've slightly patched the current trademark policy (mainly to clarify scope), just enough to decouple logo relicensing under LGPLv3/CC-BY-SA 3.0 from the finalization of a new trademark policy (see above). I've also verified that the license choice is fine with teams that regularly deal with the logos. Ball is now in SPI camp (as copyright holder). They should finalize the relicensing during the forthcoming board meeting, on September 13th.
• We heard back from the trademark office in Japan, who accepted our request to transfer it from local Debian contributors to SPI. That being mainly a bureaucratic acknowledgement, we'll now need to way for a while until they confirm the transfer has been finalized. My eternal thanks for help with this matter goes to Kenshi Muto.

Core teams

• Manoj Srivastava has stepped down as tech-ctte member. I hereby thank him for his long and valuable service in that capacity. The tech-ctte has now one vacant seat, and I've encouraged present tech-ctte members to propose to me a suitable candidate. In an attempt to promote diversity of views within the tech-ctte, they are now considering various ways to solicit candidatures. You'll hear from them about this soon, I suspect.
• In the meantime, monthly IRC meetings of the tech-ctte are continuing, as you might have noticed from tech-ctte decision reports posted on this list. As it seemed to be lingering, at the last meeting I've solicited progress on the Python maintenance issue.
• The auditors have noticed the unclear situation of the Policy Editor delegation, which seems to rely on a "base" delegation dating back to Antony Towns' office and various subsequent delegation "patches". After discussion with Russ Allbery I've then proceeded to a public "ping" to assess the interest of current editors in continuing their work in such capacity, and encourage new volunteers to apply as editors. Based on the feedback I've got, I'm now preparing a new Policy Editor delegation.

Legal and RC fun

• Solicited by Jonathan Nieder and Helge Kreutzmann, I've looked into #555168 and sought legal advice to answer various questions related to the non-copyrightability of locale data. Aaron Williamson of SFLC has been blazing fast in answering them (thanks!), and Jonathan and Helge has now all the needed information to help addressing that (RC) issue. Keep an eye on the bug log for more details.

Hardware

• I've approved buying 6 console servers to improve DSA ability to act on remote machines, without having to bother^Wrely on local admins when not strictly needed. ~550 USD well spent.

See? It's been quick(er)! Talk to you here next month, with a much lower count of Wheezy RC Bugs on the horizon, hopefully.
Cheers.

---

PS the boring day-to-day activity log for August 2012 is available at master:/srv/leader/news/bits-from-the-DPL.txt.201208

I just uploaded Debian Policy 3.9.4.0, which includes the implementation of the Technical Committee decision to make build-arch and build-indep mandatory targets and which significantly reworks the section on shared libraries. Nothing here is release-critical for wheezy, and uploads targeted for wheezy should not attempt to update packaging for a new version of Policy. I went into some more detail on debian-devel-announce. The version number probably should have been 3.10.0.0 given the build-arch and build-indep changes and the work on symbols files. Ah well. Here's the upgrading checklist:

2.4

New tasks archive section.

4.9

build-arch and build-indep are now mandatory targets in debian/rules.

5.6.26

New section documenting the Vcs-* fields, which are already in widespread use. Note the mechanism for specifying the Git branch used for packaging in the Vcs-Git field.

7.1

The deprecated relations < and > now must not be used.

7.8

New Built-Using field, which must be used to document the source packages for any binaries that are incorporated into this package at build time. This is used to ensure that the archive meets license requirements for providing source for all binaries.

8.6

Policy for dependencies between shared libraries and other packages has been largely rewritten to document the symbols system and more clearly document handling of shared library ABI changes. symbols files are now recommended over shlibs files in most situations. All maintainers of shared library packages should review the entirety of this section.

9.1.1

Packages must not assume the /run directory exists or is usable without a dependency on initscripts (>= 2.88dsf-13.3) until the stable release of Debian supports /run.

9.7

Packages including MIME configuration can now rely on triggers and do not need to call update-mime.

9.11

New section documenting general requirements for alternate init systems and specific requirements for integrating with upstart.

12.5

All copyright files must be encoded in UTF-8.

If you want to add some rules to your Apache based on the clients country, mod_geoip is perfect for it. Installation On Squeeze following is enough: # apt-get install libapache2-mod-geoip geoip-database/squeeze-backports Note that you should use the geoip-database version from squeeze-backports to have got the most up to date database version, I am updating it every month. Configuration You can add the rules to your VirtualHost, Directory, Location directives and also to your apache2.conf ( serverwide ). So you are flexible with where to use it. Blocking countries On some servers I have got more than 90 percent of spam requests only from three countries, so I blocked them with: <DirectoryMatch ^/var/www/.*/html >
SetEnvIf GEOIP_COUNTRY_CODE RU BlockCountry
SetEnvIf GEOIP_COUNTRY_CODE CN BlockCountry
SetEnvIf GEOIP_COUNTRY_CODE UA BlockCountry
Deny from env=BlockCountry
</DirectoryMatch> Allow only specific countries In the other way you also can allow specific countries to have got access to your website, this also may be a good idea for extranets, where you know from where your customers are: <Directory /var/www/my.site.com/html/login >
SetEnvIf GEOIP_COUNTRY_CODE DE AllowCountry
SetEnvIf GEOIP_COUNTRY_CODE CH AllowCountry
Deny from all
Allow from env=AllowCountry
</Directory> Very easy! Rewrite Rules You can also use it for mod_rewrite. Within a project, customers from CN and TW should be redirected to the chinese page: RewriteCond % ENV:GEOIP_COUNTRY_CODE ^(CN TW)$
RewriteRule ^(.*)$ http://some.example.cn/site.php [L] mod_geoip with proxy frontends Normaly mod_geoip works behinds load balancers and proxy servers, since it also take care of the HTTP_X_FORWARDED_FOR header. But with haproxy it looks problematic, since it does not add the HTTP_X_FORWARDED_FOR header to KeepAlive d requests :( Disabling KeepAlive is a bad idea on this cluster, so we decided to also use php5-geoip in our application, so everything is working nice now.. What mod_geoip is NOT is mod_geoip helps you to block/allow specific countries, but it does not protect you from them.
Also keep in mind that the database is only ~ 99,8% accurate, so you may have got false positives/negatives. If you only allow german users, a german IP could be listed as russian.
This is much more problematic with mobile/satellite connections and surely you can also not access your page, if you are on vacation in another country. ;)

Today was a black day for democracy in Germany. The German constitutional court ruled in favour of the European Stability Mechanism. In combination with last week s announcement by the European Central Bank to purchase government bonds without limits (breaking the No-Bail-Out clause at the core of their mandate more obviously and irreversably than ever before), the German people have lost a good deal of democracy today. Why? you may ask because from now on, fiscal and financial policy will be made in Brussels, by people enjoying full immunity, but who are not elected democratically by the European people, let alone the Germans, and they will freely decide over who has to pay and be liable for whom. I am talking about people like Klaus Regling, who was already involved the very first time the Maastricht Criteria were violated. He is now at the front of the largest and most powerful financial weapon ever conceived. With immunity. And people like Mario Draghi, whom I would possibly call the most corrupt person I know. His announcement to save the Euro at whatever cost accidentally came only a day before his motherland Italy had to go to the market for more money and was able to place a bond at such ridiculously low interest rates that anyone who s kept up to speed with Italy s development had to rightfully ask how that was possible. While in the past, for whatever reason, the European people have let the ECB get by saying that they are not bailing out countries when they buy bonds on the secondary market (wtf!), they have finally dropped that restriction (the law). And as of today, the ESM is ready to go, along with the fiscal pact. Germany is now liable for more than quarter of all of the Eurozone s past and future debts. And no citizen will be able to have any more influence in this, or reverse it. Budget, fiscal policy and currency control are forever gone. Not that parliamentarian democracies were ever direct. Yet, in the past, one could at least vote for those people whose promises one was inclined to believe the most. You can still do that in the future, but those people won t be able to influence fiscal or financial policy anymore. There is no way back. The ESM and its employees enjoy full immunity, and the ESM is forever-binding. There is no exit clause. Thanks to the ECB s law breaking and the ESM, which I consider highly unconstitutional, at least in Germany, Eurozone-countries may refinance their debts at interest rates that are in no way related to their ability to pay back loans. All other countries foremost Germany are henceforth liable for others debts. The fundamental rule of the EU that no country would have to stand up for another country, is gone with the wind. Within an hour, the markets reacted. Germany, which previously had to pay negative interest (a sign of stability) saw interests on its bond shoot up. And Spain, Portugal, Greece and others who couldn t previously refinance their old debts, are now getting fresh money cheaper than ever. Spain s president Rajoy today didn t even bother beating around the bush anymore, he s now going to apply for fresh money but won t bother with any saving schemes or other restructurings. Monti in Italy has suggested the same. Wouldn t you take money if you were offered it for free, without the need to pay it back? This is more than inflation, in my opinion. What is currently happening in Europe is active depreciation of individual wealth. Our heads of state are actively working against the people. The Euro has lost all credibility and everyone knows it. It is only a question of time until it will tremble and fall. Meanwhile, the market celebrates and continues their gambles while they still can, on the backs of our currency and our wealth. Most affected are the people who have savings in Euros, whose life insurances are decreasing in worth and who cannot afford to diversify into other asset classes or currencies. On the other hand, those who let their money do the work are being saved. Whoever previously invested into bonds of struggling states, hoping to reap massive interest gains, is now proven right. Brussels has eliminated the risk factor. What kind of message does this send??? Hands up if you thought that our politicians are even interested in closing the rapidly widening gap between rich and poor. Really? That s naive. The Eurozone is corrupt, and our currency has never been as virtual as today. Nobody can say whether saving the Euro at all cost is the right thing and noone knows whether what s currently happening is just bad. I would have wished that our politicians had taken the crisis as an incentive to fix the system in the interest of the people and with a long-term focus:

• Europe should have returned to the law it gave itself, without exceptions: The No-Bailout-Clause, as well as the Masstricht Criteria. In general, you cannot spend more than you earn, and this principle seems to have been forgotten entirely, and those who had most actively forgotten it, are being rewarded.
• A shared currency requires a shared fiscal policy. This policy has to be in place before the currency, the currency can only ever be the result of a common strategy. The EU has done this the wrong way around and it failed. Trying to put a fiscal policy in place now is patchwork that is guaranteed to fail. And putting fiscal policy into the hands of undemocratically chosen people in Brussels is not representing the people.

But on the contrary! Europe s policiticans are making it crystal clear that the foundation upon which it was built, the laws and rules, the promises and guarantees, no longer apply. The people were not asked. The promises once made were broken. Our politicians have ruled over our heads. More debts are being made, and more debts to pay off debts, and so on. It s long gotten out of control, now the process is institutionalised. I feel sorry for our kids. I find it irresponsible what is being done to them (in addition to the way we rape the environment). I also feel deeply with the people in the struggling countries who are being screwed by the crisis and are not at fault. What our politicians are doing is unfortunately not going to help long term. The problems are just postponed, and with every day, the inevitable crash will be more painful. I am sorry. Today is a black day for democracy. We have lost souvereignity. We have lost control over our currency. We have lost our budget rights. And I have lost my faith in the last instance of the German government that I trusted. As of today, I know that the German constitutional court is nothing more than a puppet in the hands of the politicians (who are themselves puppets of Brussels and the banks). The limit they imposed (Germany s liability must not increase beyond 190 billion Euros without the federal parliament s consent) is worthless. Soon the politicians will explain to us why it s inevitable that we must raise this limit. Not that the people could prevent it, but still I had hoped for a fundamental ruling. They should not have touched numbers. The EU had a no-bailout-clause from day one. It was conditional from the start. If one of the fundamental principles of a contract is broken, the contract becomes invalid. Not only did I expect the court to rule against socialised debt, I would have wished them to go a step further. The German national bank gave up control over the currency to the ECB only because the ECB incorporated the principles of the German national bank. Once the ECB overturned those principles, Germany should have reclaimed their souvereignity. But noone else in Europe would have wanted that. Merkel became a puppet herself. I am grateful that our daughter has dual citizenship. NP: Porcupine Tree: Live at Atlanta 2010

Hi, It's been three weeks already since the end of the Google Summer of Code 2012, and I've been told I still haven't blogged about the state of multiarch cross-toolchains. So, here it is! Building multiarch cross-toolchains This section is mostly a rehash of my previous post. Cross-toolchains build properly for most architectures in Debian, with the exception of arches having clashing ld.so paths. The following instructions assume you are using Wheezy or unstable, and have applied my patches. Adding a foreign architecture You are going to use some packages meant for a different architecture than your native one. This means you'll have to tell dpkg about a foreign architecture, for instance, armel.

# dpkg --add-architecture armel

Then, you need to update apt's database.

# apt-get update

If one of the repository you use only supports a subset of your configured architecture (for instance, when working with an unofficial port), you will need to use arch restrictions in your sources.list (see the manpage). Cross-binutils Absolutely nothing has changed there, it works exactly the same way it did before the GSoC. That is, grab binutils' source package, set $TARGET to a GNU triplet or a Debian arch, and run dpkg-buildpackage.

$ apt-get source binutils
$ cd binutils-*/
$ TARGET=arm-linux-gnueabihf dpkg-buildpackage -uc -us -b

Cross-gcc Building cross-gcc is done the same way it was before the GSoC, except it uses cross-arch dependencies instead of mangled packages for foreign packages. This means you shouldn't use dpkg-cross at all, but use multiarch instead. Then, building the cross-compiler is as easy as running (with $target being a Debian arch name):

$ export DEB_CROSS_NO_BIARCH=yes
$ echo $target > debian/arch
$ debian/rules control
$ dpkg-buildpackage -uc -us -b

Biarch/multilib and multiarch do not really play well. Using some trickery, you might be able to build multilib cross-compilers, but I would recommend building two cross-compilers instead. You will of course need to install the required build-dependencies, some of which are packages for the target architecture, but that is no different from building any other Debian package, provided that multiarch is correctly configured. This will produce cross-compilers packages using multiarch paths and cross-arch deps, along with some binary packages for the target architecture. Those binary packages are cross-built versions of the runtime libs. They are not needed, as they should be identical to the natively-built ones present in the archive, but using them shouldn't be a problem. Installing the cross-toolchains Installing cross-binutils is straightforward, installing cross-gcc itself is, but installing cross-g++ is not. Indeed there is bug #678623: g++-4.7 depends on libstdc++6-4.7-dev, which is multiarch-unaware and depends on g++-4.7. Assuming you've compiled your cross-gcc using my patches, the cross-compiled libstdc++6-4.7-dev will not have these problems, but still won't be co-installable with your native libstdc++6-4.7-dev, since it's not M-A: same. The easiest way to work around this issue is probably to download the binary package, modify its control file to include Multi-Arch: same , and install the modified package. Using the cross-toolchains Once installed, using the cross-toolchains should be as easy as running $triplet-gcc-4.7 file.c . Note the -4.7 part. Indeed, the cross-compilers packages do not provide the $triplet-gcc symlink. This matches what is done with the native compiler pase, where the gcc symlink is not provided by the compiler's package, but by the gcc package. However, there is no cross-gcc package or anything like that yet, which means you have to use the full name, or provide the symlink yourself. Another issue is pkg-config. To know where to search for libraries, pkg-config has to be told which architecture you are compiling for. Autotools does that by calling $triplet-pkg-config instead of pkg-config if it exists. The pkg-config binary packages comes with a wrapper you can symlink to: /usr/share/pkg-config-crosswrapper . Conclusion Despite a few shortcomings (bug #678623, missing symlinks, multilib issues and clashing ld.so paths), multiarch cross-toolchains are already usable on Wheezy. There is still quite a lot of work left (like fixing #678623, splitting gcc-4.7 in two, or changing wanna-build, britney etc. to handle cross-arch deps) but hopefully this will be done for Jessie.

I just posted the following question to ServerFault .and then realised there might be people out there in magical internetland who know the answer but never visit any of the SO sites, so i ve posted it here too. Feel free to respond here on on serverfault.

---

In a recent upgrade (from Openstack Diablo on Ubuntu Lucid to Openstack Essex on Ubuntu Precise), we found that DNS packets were frequently (almost always) dropped on the bridge interface (br100). For our compute-node hosts, that s a Mellanox MT26428 using the mlx4_en driver module. 1. Use an old lucid kernel (e.g. 2.6.32-41-generic). This causes other problems, in particular the lack of cgroups and the old version of the kvm and kvm_amd modules (we suspect the kvm module version is the source of a bug we re seeing where occasionally a VM will use 100% CPU). We ve been running with this for the last few months, but can t stay here forever.

net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-ip6tables = 0

---

Update 2012-09-12: Something I should have mentioned earlier this happens even on machines that don t have any openstack or even libvirt packages installed. Same hardware, same everything, but with not much more than the Ubuntu 12.04 base system installed. On kernel 2.6.32-41-generic, the bridge works as expected. On kernel 3.2.0-29-generic, using the ethernet interface, it works perfectly.
Using a bridge on that same NIC fails unless net.bridge.bridge-nf-call-iptables=0 So, it seems pretty clear that the problem is either in the mellanox driver, the updated kernel s bridging code, netfilter code. or some interaction between them. Interestingly, I have other machines (without a mellanox card) with a bridge interface that don t exhibit this problem. with NICs ranging from cheap r8169 cards to better quality broadcom tg3 Gbit cards in some Sun Fire X2200 M2 servers and intel gb cards in supermicro motherboards. Like our openstack compute nodes, they all use the bridge interfaces as their primary (or sometimes only) interface with an IP address they re configured that way so we can run VMs using libvirt & kvm with real IP addresses rather than NAT. So, that indicates that the problem is specific to the mellanox driver, although the blog post I mentioned above had a similar problem with some broadcom NICs that used the bnx2 driver. openstack, bridging, netfilter and dnat is a post from: Errata

This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (88.41 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. This month has again been a short one since I have mostly been in vacation during the last 2 weeks. Dpkg Things are relatively quiet during the freeze. I only took care of fixing 3 bugs: a regression of 3.0 (quilt) (#683547), a segfault of dpkg-query -W -f (commit) and a bad auto-completion for French users (#685863). Testing the upgrade to wheezy We got several reports of wheezy upgrade that failed because dpkg ran the trigger while the dependencies of the package with pending triggers are not satisfied. Unfortunately fixing this in dpkg is not without problems (see #671711 for details) so Guillem decided to defer this fix for Jessie. My suggestion of an intermediary solution has fallen in limbo. Instead we now have to find solutions for each case where this can fail (example of failure: 680626). Another way to avoid those errors is to ensure that triggers are run as late as possible. We can improve this in multiple ways. The first way is to modify most triggers so that they use the interest-noawait directive. In that case, the packages activating the trigger will be immediately marked as configured (instead of triggers-awaited ) and the trigger will thus not need to be run as part of further dependency solving logic. But as of today, there s no package using this new feature yet despite a nudge on debian-devel-announce. The second way is to modify APT to use dpkg no-triggers, and to let the trigger processing for the end (with a last dpkg configure -a call). I requested this early in the wheezy timeframe but for various reasons, the APT maintainers did not act on it. I pinged them again in #626599 but it s now too late for wheezy. I find this a bit sad because I have been using those options for the entire wheezy cycle and it worked fine for me (and I used them for a dist-upgrade on my wife s laptop too). It would have been good to have all this in place for wheezy so that we don t have to suffer from the same problems during the jessie upgrade, but unless someone steps up to steer those changes, it seems unlikely to happen. Instead, we re back to finding klumsy work-arounds in individual packages. Packaging I prepared security updates for python-django (1.4.1 for unstable,
1.2.3-3+squeeze3 for stable). I packaged a new upstream version for cpputest (3.2-1). I reviewed ledgersmb 1.3.21-1 prepared by Robert James Clay and asked him to prepare another version with further fixes. I released nautilus-dropbox 1.4.0-2 with supplementary changes of my own to support https_proxy and to display better diagnostic information when the download fails. With the help of Paul van der Vlis and Michael Ziegler, we did what was required to be able to migrate python-django-registration 0.8 to Wheezy even though it s a new upstream version with backwards incompatible changes. Thanks to Adam D. Barratt who unblocked the package, we now have the right version in Wheezy despite the fact that I missed the freeze deadline. Debian France Julien Cristau reminded the board of Debian France that we have to elect officers (President, Secretary, Treasurer) as the current officers have withdrawn. I was somewhat afraid that nobody would take over so I pinged each member to try to get new volunteers. We now have volunteers (me, Julien Danjou and Sylvestre Ledru) and we re waiting until Julien finds some time to run the election. Misc With the help of DSA, I setup antispam rules for the owner@packages.qa.debian.org alias because I was getting tired by the amount of spam. In the process, they asked me to write a wiki page for dsa.debian.org to document everything so that they can refer to it for future queries. I did it but it looks like that they did not apply my patch yet. I also tested an upstream patch for gnome-keyring (see bugzilla #681081) that reintroduces the support of forgetting GPG passphrases after a specified amount of time. Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

I've been slowly noticing, over a number of years now, a place where I don't react to other people the way that I want to. I've been stumbling over defensive reactions that I didn't realize I had. These are a few rules of thumb that I'm formulating for myself to try to improve that, and I thought it might help to write them down. And maybe they'll be meaningful for someone else.

1. Things people are excited about are probably exciting. If I don't find them exciting, that's probably because I don't know enough about them to understand why they're exciting. Or perhaps I have a different mindset, and therefore the exciting parts don't connect with me. Both of those are okay! There are way more things to learn about than I will ever possibly have time to learn, and I don't have to get excited about everything other people are excited about. But, equally, just because I'm not excited doesn't mean that the thing is not exciting.
2. My misplaced sense of obligation should not be someone else's buzzkill. This requires a bit of unpacking, since I only intermittantly realize that I'm doing this. But (partly for cultural reasons, partly due to natural inclinations) I've always felt a strong sense of obligation to "keep up" with things. To know a lot about a lot of topics. And one of the things I've realized is that my brain occasionally twists an interest in knowing lots about the world into a sense of obligation to "stay current" and know something about everything important, at least within my vague field of expertise. And usually that part of my brain has a very expansive notion of field of expertise; all of software development, for example, which is, when I pull it out into the light, ridiculously broad. This then sparks one of the standard failure modes of human cognition. I have an unrealistic expectation of myself. I fail to meet that expectation. This makes me feel guilty and stressed. At that point, my brain starts trying to find targets for me to displace negative feelings onto, and that generally ends up being the next person who is excited about something new that I don't know anything about but feel like I should. This is all very interesting psychologically, in the abstract. However, the concrete result is that I sometimes get all snarky and negative about something that someone else is excited about because it happens to be the moment I hit overload, or because I didn't want to hear about another thing that I feel like I should know more about. And that, in turn, can be either confrontational or discouraging to someone who is legitimately excited (and may rarely get the chance to be excited). And that's not the person I want to be.
3. It's okay not to have an opinion. Seriously, it's okay. The world is full of stuff, far more stuff than any one person can know anything about. It's okay not to go read about everything people say is exciting. It's okay not to know about something new. It's okay to focus on the things I'm already doing. Really. What's not okay is to make up an opinion about something I know nothing about just so that I can feel better about myself by having an opinion. Particularly since that opinion formed under those circumstances tends to be negative, despite the fact that I have no actual basis for having any opinion at all. I need to practice just quietly not having an opinion. No one else needs to know about my lack of opinion.
4. Before bristling at imposition, be sure someone is actually imposing. I frequently leap to the conclusion that, when someone raises a topic, they want me to do something about it. It's a gut emotional reaction, I think in part because the emotional part of my brain likes to make everything about me. Intellectually, if I can engage that part of my brain, it's often obvious that it's not about me, or that they're not asking me to do anything. Often, when someone brings something up, they just want to tell someone else about it, or make me aware of it. If I've listened to, or read, all the sentences and understood then, then congratulations! I've done everything that I needed to do and was expected to do and need not feel any further obligation. Even if something further needs to be done, it doesn't necessarily need to be done by me. I'm very protective of my time and available resources because I almost always feel overwhelmed by all the things that I want to work on. This does not entitle me to snap at random people who are being excited around me on the grounds that they're trying to put more obligations on me. They're probably not. If it becomes clear that they really want me to do something I don't have time to do, I can politely decline, but if I just shut up and listen for longer, usually that's not the case at all.

(Please note that this is not prompted by any current discussion happening anywhere, and paticularly not by anything in Debian. Insofar as it was prompted by anything, it was by a throwaway comment in an unrelated private discussion. It's just something that gelled enough to write down while I was walking home from work.)

ELKI is a data mining software project that I have been working on for the last years as part of my PhD research. It is open source (AGPL-3 licensed) and avilable as both a Debian package and Ubuntu package in the official repositories. So a simple aptitude install elki should get you going and give you a menu entry for ELKI. These packages come with the scripts elki to launch the MiniGUI and elki-cli to run from command line. The key feature that sets ELKI apart from existing open source tools used in data mining (e.g. Weka and R) is that it has support for index structures to speed up algorithms, and a very modular architecture that allows various combinations of data types, distance functions, index structures and algorithms. When looking for performance regressions and optimization potential in ELKI, I recently ran some benchmarks on a data set with 110250 images described by 8 dimensional color histograms. This is a decently sized dataset: it takes long enough (usually in the range of 1-10 minutes) to measure true hotspots. When including Weka and R in the comarison I was quite surprised: our k-means implementation runs at the same speed as Rs implementation in C (and around twice that of the more flexible "flexclus" version). For some of the key agorithms (DBSCAN, OPTICS, LOF) we are an order of magnitude faster than Weka and R, and adding index support speeds up the computation by another factor of 5-10x. In the most extreme case - DBSCAN in Weka vs. DBSCAN with R-tree in ELKI - the speedup was a factor of 330x, or 2 minutes (ELKI) as opposed to 11 hours (Weka).
The reason why I was suprised is that I expected ELKI to perform much worse. It is written in Java (as opposed to R's kmeans, which is in C), uses a very flexible architecture which for example does not assume distances to be of type double and just has a lot of glue code inbetween. However, obviously, the Java Hotspot compiler actually lives up to its expectations and manages to inline the whole distance computations into k-means, and then compiles it at a level comparable to C. R executes vectorized operations quite fast, but on non-native code as in the LOF example it can become quite slow, too. (I would not take Weka as reference, in particular with DBSCAN and OPTICS there seems to be something seriously broken. Judging from a quick look at it, the OPTICS implementation actually is not even complete, and both implementations actually copy all data out of Weka into a custom linear database, process it there, then feed back the result into Weka. They should just drop that "extension" altogether. The much newer and Weka-like LOF module is much more comparable.) Note that we also have a different focus than Weka. Weka is really popular for machine learning, in particular for classification. In ELKI, we do not have a single classification algorithm because there is Weka for that. Instead, ELKI focuses on cluster analysis and outlier detection. And ELKI has a lot of algorithms in this domain, I dare to say the largest collection. In particular, they are all in the same framework, so they can be easily compared. R does of course have an impressive collection in CRAN, but in the end they do not really fit together. Anyway, ELKI is a cool research project. It keeps on growing, we have a number of students writing extensions as part of their thesis. It has been extremely helpful for me in my own research, as I could quickly prototype some algorithms, then try different combinations and use my existing evaluation and benchmarking. You need some time to get started (largely because of the modular architecture, Java generics and such hurdles), but then it is a very powerful research tool. But there are just many more algorithms, published sometime, somewhere, but barely with source code available. We'd love to get all these published algorithms into ELKI, so researchers can try them out. And enhance them. And use them for their actual data. So far, ELKI was mostly used for algorithmic research, but it's starting to move out into the "real" world. More and more people that are not computer scientists start using ELKI to analyze their data. Because it has algorithms that no other tools have. I tried to get ELKI into the "Google Summer of Code", but it was not accepted. But I'd really like to see it gain more traction outside the university world. There are a number of cool projects associated with ELKI that I will not be able to do myself the next years, unfortunately.

• A web browser frontend would be cool. Maybe even connected to Google Refine, using Refine for preprocessing the data, then migrating it into ELKI for analysis. The current visualization engine of ELKI is using SVG - this should be fairly easy to port into the web browser. Likely, the web browers will even be faster than the current Apache Batik renderer.
• Visual programming frontend. Weka, RapidMiner, Orange: they all have visual programming style UIs. This seems to work quite well to model the data flow within the analysis. I'd love to see this for ELKI, too.
• Cluster/Cloud backend. ELKI can already handle fairly large data sets on a big enough system. If someone spends extra effort on the index structures, the data won't even need to fit into main memory anymore. Yet, everybody now wants "big data", and parallel computation probably is the future. I'm currently working on some first Hadoop YARN based experiments with ELKI. But this is a huge field, turning this into true "elk yarn". I will likely only lay some foundations (unless I get funding to continue as a PostDoc on this project. I sure hope to get to do at least a few years of postdoc somewhere, as I really enjoy working with students on this kind of project)
• New visualization engine. The current visualization engine, based on Apache Batik and SVG is quite okay. It does what I need, which is to get a quick glance at the results and the ability to export them for publications in print quality. (in particular, I can easily edit the SVG files with Inkscape) But it is not really something fancy (although we have a lot of cool visualizations). And it is slow. I havn't found a portable and fast graphics toolkit for Java yet that can produce SVG files. There is a lot of hype around processing, for example, but it seems to be too much about art for me. In fact, I'd love to use either something like Clutter or Cairo. But getting them to work for Windows and Mac OSX will likely be a pain.
• Human Computer Interaction (HCI). This is in my opinion the biggest challenge we are facing with all the "big data" stuff. If you really go into big data (and not just run Hadoop and Mahout on a single system; yes - a lot of people seem to do this), you will at some point need to go beyond just crunching the numbers. So far, the challenges that we are tackling are largely data summarization and selection. TeraSort is a cool project, and a challenge. Yet, what do you actually get from sorting this large amount of data? What do you get from running k-means on a terabyte? When doing data mining on a small data set, you quickly learn that the main challenge actually is preprocessing the data and choosing parameters the right way so that your result is not bogus. Unless you are doing simple prediction tasks, you often don't have a clearly defined objective. Sure, when predicting churn rates, you can hope to just throw all the data into a big cloud and hope you get some enlightement out. But when you are doing cluster analysis or outlier detection - unsupervised methods - the actual objective by definition cannot be hardcoded into a computer. The key objective then is learn something new on the data set. But if you want to have your user learn something on the data set, you will have to have the user guide the whole process, and you will have to present results to the user. Which gets immensely more difficult with larger data. Big data just does no longer look like this. And neither are the algorithms as simple as k-means or hierarchical clustering. Hierarchical clustering is good for teaching the basic ideas of clustering. But you will not be using a dendrogram for a huge data set. Plus, it has a naive complexity of O(n^3) and for some special cases O(n^2) - too slow for truly big data.
  For the "big data future" once we get over all the excitement of being able to just somehow crunch these numbers we will need to seriously look into what to do with the results (in particular, how to present them to the user), and how to make the algorithms accessible and usable for non-techies. Right now, you cannot expect a social sciences researcher to be able to use a Hadoop cluster. Yet to make sense of the results. But if you are a smart guy to actually solve this, and open up "big data processing" to the average non-IT user, this will be big.
• Oh, and of course there are just hundreds of algorithms not yet available (accessible) as open source. Not in ELKI, and usually not anywhere else either. Just to name a few from my wishlist (I could probably implement many of them in a few hours in ELKI, but I don't have the time to do so myself, plus they are good student or starter project to get used to ELKI): BIRCH, CLARA, CLARANS, CLINK, COBWEB, CURE, DOC, DUSC, EDSC, INSCY, MAFIA, P3C, SCHISM, STATPC, SURFING, ... just to name a few.

If you are a researcher in cluster analysis or outlier detection, consider contributing your algorithms to ELKI. Spend some time optimizing them, adding some documentation. Because, if ELKI keeps on growing and gaining popularity, it will be the future benchmark platform. And this can give you citations, which are somewhat the currency of science these days. Algorithms available in the major toolkits just do get cited more, because people compare to them. See this list for an overview of work cited by ELKI - scientific work that we reimplemented at least to some extend for ELKI. It is one of the services that we provide with ELKI for researchers: not only the algorithm, but also the appropriate citation.