If you re having issues with video playback in unstable on Intel hardware, this post is for you. Some buffer freeing code was added in libdrm 2.4.28 to try and fix exhaustion of per-process limits. Some safeguards (assert()) were put in place to see whether some other components needed fixing (like unbalanced map()/unmap()). It s hard to miss those, since the server crashes when something s wrong. Good news: for affected users, downgrading to libdrm 2.4.27 (currently available in testing) should restore a functional setup. So far, patches have floated around for libva[1], xserver-xorg-video-intel[1,2] and libdrm[1,2,3] itself. Since libdrm has already been exposed to a wide audience for a while, a new upstream release is planned in a few days, disabling those assert()s. Other packages will then have some extra time to get new releases with the appropriate patches.

Perhaps the immortal bard was making a more noble and deep reference to the stages of ones life but I feel no compunction lifting the line for my purpose.

Cocolo Overview I recently wrote about buying a fridge for storing chocolate [1]. Jason Lewis (the co-founder of Organic Trader [2]) read that post and sent me some free samples of Cocolo chocolate [3] (Cocolo is an Organic Trader product that is made in Switzerland). It s interesting to note that Cocolo seem very focussed on a net presence [3], their URL is printed on the back of the packet in an equal size font to the main label on the front (although the front label is in upper case). The main web page has a prominent link to their Twitter page which appears to be updated a couple of times a month. Cocolo makes only organic fair-trade chocolate. Every pack lists the percentage of ingredients that are Fairtrade (presumably milk and some other ingredients are sourced locally in Switzerland and Fairtrade doesn t apply to them). Their chocolate packages have the URL www.fairtrade.com.au printed on them and their web site links to an international Fairtrade organisation. The packages also list the organic and Fairtrade certification details and state that they are GMO free. The final geek data on the package is advise to store the chocolate at a temperature between 16C and 18C (I have now set my fridge thermostat to 17C). The above picture shows the front of a pack of Dark Orange chocolate and the back of a pack of Milk chocolate. Reviews One thing that is different about Cocolo is that they use only unrefined evaporated organic cane sugar juice to sweeten their chocolate. This gives it a hint of molasses in the flavor. Children who like white sugar with brown coloring might not appreciate this, but I think that the use of natural cane sugar juice will be appreciated by most people who appreciate products with complex and subtle flavors. The Milk chocolate contains a minimum of 32% cocoa solids, this compares to the EU standard of a minimum of 25% for milk chocolate and the UK standard of a minimum of 20% for Family Milk Chocolate . The EU standard for dark chocolate specifies a minimum of 35% cocoa solids, so it seems that Cocolo milk chocolate is almost as strong as dark chocolate. If you are used to eating dark and bittersweet chocolate then the Cocolo milk chocolate is obviously not that strong, but it is also significantly more concentrated than most milk chocolate that is on the market. The high chocolate content combined with the evaporated cane sugar extract gives a much stronger flavor than any of the milk chocolates that I have eaten in recent times. The Dark Mint Crisp chocolate has a minimum of 61% cocoa mass. The mint crisp is in very small pieces that give a good texture to the chocolate with a faint crunch when you bite it. It has a good balance of mint and chocolate flavors. The Dark Orange chocolate contains 58% cocoa solids and has a subtle orange flavor. The white chocolate tastes quite different from most white chocolate. While most white chocolate is marketed to children the Cocolo white chocolate will probably appeal more to adults than children. This is one of the few white chocolates that I ve wanted to eat since the age of about 14. They also have many other flavors, most common types of chocolate (such as with almonds or hazelnuts) are available. I highly recommend Cocolo products!

• [1] http://etbe.coker.com.au/2010/02/28/chocolate-fridge/
• [2] http://organictrader.com.au/
• [3] http://cocolo.com.au/

RInside 0.2.5 arrived on CRAN earlier today. RInside is a set of convenience classes which facilitate embedding of R inside of C++ applications and programs, using the classes and functions provided by the Rcpp R and C++ integration package. The big news for this release is that Windows applications once again run! James Bates gets full credit for noticing why Windows binaries died on startup, and provided a short patch which we (after some further simplicifcation) added. So for the first time since the 0.2.0 release, Windows users can just change into the examples/standard directory and issue a simple make -f Makefile.win, and the twelve example binaries will actually work. Yay, and kudos to James. The other main addition is a new examples directory examples/wt containing the web-application of the density estimation demo I had blogged about last week. By combining the Wt toolkit with RInside, we get C++ "web" applications containing R which is pretty nifty. The example otherwise follows the earlier Qt-based example of a standard desktop application with RInside. One minor snafu I just noticed is that I was overzealous with the .Rbuildignore file. By excluding the wt and qtbinaries, I also excluded their source files with the same name (up to the .cpp or .h ending). If you desire to run the examples, just grab these examples files from the SVN repo at R-Forge. All changes since the last release are summarized below:

0.2.5   2011-12-07
   o    Applied (somewhat simplified) patch by James Bates which restores
        RInside to working on Windows -- with a big Thank You! to James for
        fixing a long-standing bug we inadvertendly introduced right after
        0.2.0 almost two years ago
   o    New example embedding R inside a Wt (aka Webtoolkit, pronounced 
        'witty') application, mirroring the previous Qt application
   o    Qt example qtdensity now uses the new svg() device in base R; removed
        test for cairoDevice package as well as fallback png code
   o    Very minor fix to qmake.pro file for Qt app correcting link order

CRANberries also provides a short report with changes from the previous release. More information is on the RInside page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page.

According to Eric S. Raymond, one rule for Unix programming is the Rule of Silence: When a program has nothing surprising to say, it should say nothing. Since return codes are a form of information, I would like to add: when a program has carried out its task with no surprise, it should exit(0).Take xscreensaver-command -lock, for instance. That command is meant to make a running XScreenSaver to lock the screen. I have never seen it fail in doing so, however I sometimes get a non-zero code:

% xscreensaver -lock
xscreensaver-command: could not frobnicate window 4212
zsh: exit 1   xscreensaver-command -lock

Well, xscreensaver-command, I have the deep regret to announce you that despite of your surprising return code, you successfully succeeded to lock my screen. But that return code is wrong and annoying. Wrong, because as a user, I do not care if xscreensaver-command -lock failed to frobnicate some window he would have loved to as long as it does its job and locks my screen. To the user's point of view, locking the screen is binary: either it is done and it is a success or it is not done and that is a failure. And it is annoying, because it makes the return code meaningless and prevents you from using it to do things like:

% # Lock screen and suspend to RAM (only if locking was successful)
% xscreensaver -lock && sudo pm-suspend

With xscreensaver-command sometimes returning a non-zero code with no apparent reason, you end with a locked screen and your computer still running, and have to unlock it and try again. This looks like this meaningless dialogue:

• Scott to bridge.
• Kirk here. Scotty, could you restore impulse power?
• I'm sorry captain, I was not able to align the coaxial stabilizers, we just sustained too much damage here.
• How long would it take to get that power back?
• But you have it captain! Impulse power is working all right, it's just that
• Good job Scotty, do you think will hold long enough?
• Well, I'd say it should hold until our next scheduled revision if we take care, but captain, I could not align

So please, when you write a utility, design your return code for the user, not for you. If your utility has done the job, it should exit(0). What happened inside does may interest you as the developer but it does not matter to the user as long as the job is done, so it should not be reported to him, especially not with a misleading failure code.

Today, I've released GePeS 0.2.0. The only major difference is that it should reduce power consumption of your phone while it is running. Reducing CPU usage when application is not active was definitely on my roadmap, though email from "Publisher Support Specialist for Nokia" about this actually pushed me to investigate the issue. It turned out to be quite easy to disable sensors while application is not active and it did the trick. While doing this I also moved to using AlignedTimer to synchronize wakeups with other possible wakeups in the device. As usual, the application sources can be downloaded on GePeS page, binary will be soon available in Nokia store.

Filed under: English Meego 0 comments Flattr this!

It s been a couple days since I returned from this year s WebKitGTK+ hackfest in A Coru a, Spain. The weather was very nice, not too cold and not too rainy, we had great food, great drinks and I got to meet new people, and hang out with old friends, which is always great! I think this was a very productive hackfest, and as usual a very well organized one! Thanks to the GNOME Foundation for the travel sponsorship, to our friends at Igalia for doing an awesome job at making it happen, and to Collabora for sponsoring it and granting me the time to go there! We got a lot done, and although, as usual, our goals list had many items not crossed, we did cross a few very important ones. I took part in discussions about the new WebKit2 APIs, got to know the new design for GNOME s Web application, which looks great, discussed about Accelerated Compositing along with Joone, Alex, Nayan and Martin Robinson, hacked libsoup a bit to port the multipart/x-mixed-replace patch I wrote to the awesome gio-based infrastructure Dan Winship is building, and some random misc. The biggest chunk of time, though, ended up being devoted to a very uninteresting (to outsiders, at least), but very important task: making it possible to more easily reproduce our test results. TL;DR? We made our bots and development builds use jhbuild to automatically install dependencies; if you re using tarballs, don t worry, your usual autogen/configure/make/make install have not been touched. Now to the more verbose version! The need For a couple years now we have supported an increasingly complex and very demanding automated testing infrastructure. We have three buildbot slaves, one provided by Collabora (which I maintain), and two provided by Igalia (maintained by their WebKitGTK+ folks). Those bots build as many check ins as possible with 3 different configurations: 32 bits release, 64 bits release, and 64 bits debug. In addition to those, we have another bot called the EWS, or Early Warning System. There are two of those at this moment: one VM provided by Collabora and my desktop, provided by myself. These bots build every patch uploaded to the bugzilla, and report build failures or passes (you can see the green bubbles). They are very important to our development process because if the patch causes a build failure for our port people can often know that before landing, and try fixes by uploading them to bugzilla instead of doing additional commits. And people are usually very receptive to waiting for EWS output and acting on it, except when they take way too long. You can have an idea of what the life of an EWS bot looks like by looking at the recent status for the WebKitGTK+ bots. Maintaining all of those bots is at times a rather daunting task. The tests require a very specific set of packages, fonts, themes and icons to always report the same size for objects in a render. Upgrades, for instance, had to be synchronized, and usually involve generating new baselines for a large number of tests. You can see in these instructions, for instance, how strict the environment requirements are yes, we need specific versions of fonts, because they often cause layouts to change in size! At one point we had tests fail after a compiler upgrade, which made rounding act a bit different! So stability was a very important aspect of maintaining these bots. All of them have the same version of Debian, and most of the packages are pinned to the same version. On the other hand, and in direct contradition to the stability requirement, we often require bleeding edge versions of some libraries we rely on, such as libsoup. Since we started pushing WebKitGTK+ to be libsoup-only, its own progress has been pretty much driven by WebKitGTK+ s requirements, and Dan Winship has made it possible to make our soup backend much, much simpler and way more featureful. That meant, though, requiring very recent versions of soup. To top it off, for anyone not running Debian testing and tracking the exact same versions of packages as the bots it was virtually impossible to get the tests to pass, which made it very difficult for even ourselves to make sure all patches were still passing before committing something. Wow, what a mess. The explosion^Wsolution So a few weeks back Martin Robinson came up with a proposed solution, which, as he says, is the nuclear bomb solution. We would have a jhbuild environment which would build and install all of the dependencies necessary for reproducing the test expectations the bots have. So over the first three days of the hackfest Martin and myself hacked away in building scripts, buildmaster integration, a jhbuild configuration, a jhbuild modules file, setting up tarballs, and wiring it all in a way that makes it convenient for the contributors to get along with. You ll notice that our buildslaves now have a step just before compiling called updated gtk dependencies (gtk is the name we use for our port in the context of WebKit), which runs jhbuild to install any new dependencies or version bumps we added. You can also see that those instructions I mentioned above became a tad simpler. It took us way more time than we thought for the dust to settle, but it eventually began to. The great thing of doing it during the hackfest was that we could find and fix issues with weird configurations on the spot! Oh, you build with AR_FLAGS=cruT and something doesn t like it? OK, we fix it so that the jhbuild modules are not affected by that variable. Oh, turns out we missed a dependency, no problem, we add it to the modules file or install them on the bots, and then document the dependency. I set up a very clean chroot which we could use for trying out changes so as to not disrupt the tree too much for the other hackfest participants, and I think overall we did good. The aftermath By the time we were done our colleagues who ran other distributions such as Fedora were already being able to get a substantial improvements to the number of tests passing, and so did we! Also, the ability to seamlessly upgrade all the bots with a simple commit made it possible for us to very easily land a change that required a very recent (as in unreleased) version of soup which made our networking backend way simpler. All that red looks great, doesn t it? And we aren t done yet, we ll certainly be making more tweaks to this infrastructure to make it more transparent and more helpful to the users (contributors and other people interested in running the tests). If you ve been hit by the instability we caused, sorry about that, poke mrobinson or myself in the #webkitgtk+ IRC channel on FreeNode, and we ll help you out or fix any issues. If you haven t, we hope you enjoy all the goodness that a reproducible testing suite has to offer! That s it for now, folks, I ll have more to report on follow-up work started at the hackfest soon enough, hopefully =).

Prepare a location to run juju and install it:

mkdir ~/party
cd ~/party
sudo apt-get install juju

Initialize your juju environment. Be sure to add juju-origin: ppa to your environment, along with filling in your access-key and secret-key from your Amazon AWS account. Note that control-bucket and admin-secret should not be used by any other environment or juju won t be able to distinguish them. Other variables are good to set now too. I wanted my instances close to me, use I set region: us-west-1 . I also wanted a 64bit system, so using the AMI list, I chose default-series: oneiric , default-instance-type: m1.large and default-image-id: ami-7b772b3e

juju
$EDITOR ~/.juju/environments.yaml

Get my sbuild charm, and configure some types of builders. The salt should be something used only for this party; it is used to generate the random passwords for the builder accounts. The distro and releases can be set to whatever the mk-sbuild tool understands.

bzr co lp:~kees/charm/oneiric/sbuild/trunk sbuild-charm
cat >local.yaml <<EOM
builder-debian:
    salt: some-secret-phrase-for-this-party
    distro: debian
    releases: unstable
builder-ubuntu:
    salt: some-secret-phrase-for-this-party
    distro: ubuntu
    releases: precise,oneiric
EOM

Bootstrap juju and wait for ec2 instance to come up.

juju bootstrap

Before running the status, you can either accept the SSH key blindly, or use ec2-describe-instances to find the instance and public host name, and use my wait-for-ssh tool to inject the SSH host key into your ~/.ssh/known_hosts file. This requires having set up the environment variables needed by ec2-describe-instances , though.

ec2-describe-instances --region REGION
./sbuild-charm/wait-for-ssh INSTANCE HOST REGION

Get status:

juju status

Deploy a builder:

juju deploy --config local.yaml --repository $PWD local:sbuild-charm builder-debian

Deploy more of the same type:

juju add-unit builder-debian
juju add-unit builder-debian
juju add-unit builder-debian

Now you have to wait for them to finish installing, which will take a while. Once they re at least partially up (the builder user has been created), you can print out the slips of paper to hand out to your party attendees:

./sbuild-charm/slips   mpage -1 > /tmp/slips.ps
ps2pdf /tmp/slips.ps /tmp/slips.pdf

They look like this:

Unit: builder-debian/3
Host: ec2-256-1-1-1.us-west-1.compute.amazonaws.com
SSH key fingerprints:
  1024 3e:f7:66:53:a9:e8:96:c7:27:36:71:ce:2a:cf:65:31 (DSA)
  256 53:a9:e8:96:c7:20:6f:8f:4a:de:b2:a3:b7:6b:34:f7 (ECDSA)
  2048 3b:29:99:20:6f:8f:4a:de:b2:a3:b7:6b:34:bc:7a:e3 (RSA)
Username: builder
Password: 68b329da9893

To admin the machines, you can use juju itself, where N is the machine number from the juju status output:

juju ssh N

To add additional chroots to the entire builder service, add them to the config:

juju set builder-debian release=unstable,testing,stable
juju set builder-ubuntu release=precise,oneiric,lucid,natty

Notes about some of the terrible security hacks this charm does:

• enables password-based SSH access (and locks the default ubuntu account), so party attendees don t need anything but the ssh client itself to get to the builders.
• starts rngd -r /dev/urandom to create terrible but plentiful entropy for the sbuild GPG key generation.

Enjoy!

2011, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.

My earlier post on juju described a number of weird glitches I ran into. I got invited by hazmat on IRC (freenode #juju) to try to reproduce the problems so we could isolate the trouble. Fix #1: use the version from the PPA. The juju setup documentation doesn t mention this, but it seems that adding juju-origin: ppa to your ~/.juju/environment.yaml is a good idea. I suggest it be made the default, and to link to the full list of legal syntax for the environment.yaml file. I was not able to reproduce the missing-machines-at-startup problem after doing this, but perhaps it s a hard race to lose. Fix #2: don t use terminate-machine . :P There seems to be a problem around doing the following series of commands: juju remove-unit FOO/N; juju terminate-machine X; juju add-unit FOO . This makes the provisioner go crazy, and leaves all further attempts to add units stick in pending forever. Big thank you to hazmat and SpamapS for helping debug this.

2011, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.

Updating the bios on my X200 Tablet was surprisingly simple in Debian. Often, updating bios requires some sort of Windows or Dos boot disk and all kinds of machinations to make that happen. Lenovo provides an iso image to update the bios which alleviates some of these problems, but as the X series don't have built-in CDROM drives (and I don't have an ultrabay), All you have to do is download the iso, install syslinux aptitude install syslinux, and configure grub to start memdisk and boot off of the iso. You can do this by copying the iso to your boot partition (cp 7wuj43uc.iso /boot/;), copying memdisk to the boot partition (cp /usr/lib/syslinux/memdisk /boot/;) and adding an entry to your grub.cfg like the following (putting it in /etc/grub.d/40_custom is probably the most reasonable):

menuentry "Thinkpad x200 BIOS Upgrade to 3.19"  
       insmod part_msdos
       insmod ext2
       set root='(hd0,msdos1)'
       linux16 /memdisk iso
       initrd16 /7wuj43uc.iso
 

Note that your configuration will be slightly different depending on whether you have a separate /boot partition (I do) or not. Then you just need to boot the laptop, select the bios upgrade option in grub, and tell the program that you really do want to upgrade the bios and that you really do have a full battery and the laptop plugged into an AC adapter. The major idea for this method came from ThinkWiki's entry.

Updating the bios on my X200 Tablet was surprisingly simple in Debian. Often, updating bios requires some sort of Windows or Dos boot disk and all kinds of machinations to make that happen. Lenovo provides an iso image to update the bios which alleviates some of these problems, but as the X series don't have built-in CDROM drives (and I don't have an ultrabay), All you have to do is download the iso, install syslinux aptitude install syslinux, and configure grub to start memdisk and boot off of the iso. You can do this by copying the iso to your boot partition (cp 7wuj43uc.iso /boot/;), copying memdisk to the boot partition (cp /usr/lib/syslinux/memdisk /boot/;) and adding an entry to your grub.cfg like the following (putting it in /etc/grub.d/40_custom is probably the most reasonable):

menuentry "Thinkpad x200 BIOS Upgrade to 3.19"  
       insmod part_msdos
       insmod ext2
       set root='(hd0,msdos1)'
       linux16 /memdisk iso
       initrd16 /7wuj43uc.iso
 

Note that your configuration will be slightly different depending on whether you have a separate /boot partition (I do) or not. Then you just need to boot the laptop, select the bios upgrade option in grub, and tell the program that you really do want to upgrade the bios and that you really do have a full battery and the laptop plugged into an AC adapter. The major idea for this method came from ThinkWiki's entry.

This weekend, we held a combined Debian Bug Squashing Party and Ubuntu Local Jam in Portland, OR. A big thank you to PuppetLabs for hosting! Thanks to a brilliant insight from Kees Cook, we were able to give everyone access to their own pre-configured build environment as soon as they walked in the door by deploying schroot/sbuild instances in "the cloud" (in this case, Amazon EC2). Small blips with the mirrors notwithstanding, this worked out pretty well, and let people start to get their hands dirty as soon as they walked in the door instead of spending a lot of time up front doing the boring work of setting up a build environment. This was a big win for people who had never done a package build before, and I highly recommend it for future BSPs. You can read about the build environment setup in the Debian wiki, and details on setting up your own BSP cloud in Kees's blog. (And the cloud instances were running Ubuntu 11.10 guests, with Debian unstable chroots - a perfect pairing for our joint Debian/Ubuntu event!) So how did this curious foray into a combined Ubuntu/Debian event go? Not too shabby:

• Roughly 25 people participated in the event - a pretty good turnout considering the short notice we gave. Thanks to everyone who turned up!
• Multiarch patches were submitted for 14 library packages by 9 distinct contributors
• Four of these people submitted their first patch to Debian!
• Three more contributors worked on patches that were not submitted to Debian by the end of the event, but we will stalk them and see to it that their patches make it in ;)
• 8 Ubuntu Stable Release Updates were looked at for verification of fixes
• 7 of these fixes were successfully verified (one bug was not reproducible)
• 6 of those packages have already been moved to the -updates pocket, where all of Ubuntu's users can now benefit from them

When all was said and done, we didn't get a chance to tackle any wheezy release critical bugs like we'd hoped. That's ok, that leaves us something to do for our next event, which will be bigger and even better than this one. Maybe even big enough to rival one of those crazy, all-weekend BSPs that they have in Germany...

On Sunday, I brought up EC2 instances to support the combined Debian Bug Squashing Party/Ubuntu Local Jam that took place at PuppetLabs in Portland, OR, USA. The intent was to provide each participant with their own sbuild environment on a 64bit machine, since we were going to be working on Multi-Arch support, and having both 64bit and 32bit chroots would be helpful. The host was an Ubuntu 11.10 (Oneiric) instance so it would be possible to do SRU verifications in the cloud too. I was curious about the juju provisioning system, since it has an interesting plugin system, called charms , that can be used to build out services. I decided to write an sbuild charm, which was pretty straight forward and quite powerful (using this charm it would be possible to trigger the creation of new schroots across all instances at any time, etc). The juju service itself works really well when it works correctly. When something goes wrong, unfortunately, it becomes nearly impossible to debug or fix. Repeatedly while working on charm development, the provisioning system would lose its mind, and I d have to destroy the entire environment and re-bootstrap to get things running again. I had hoped this wouldn t be the case while I was using it during production on Sunday, but the provisioner broke spectacularly on Sunday too. Due to the fragility of the juju agents, it wasn t possible to restart the provisioner it lost its mind, the other agent s couldn t talk to it any more, etc. I would expect the master services on a cloud instance manager to be extremely robust since having it die would mean totally losing control of all your instances. On Sunday morning, I started 8 instances. 6 came up perfectly and were excellent work-horses all day at the BSP. 2 never came up. The EC2 instances started, but the service provisioner never noticed them. Adding new units didn t work (instances would start, but no services would notice them), and when I tried to remove the seemingly broken machines, the instance provisioner completely went crazy and started dumping Python traces into the logs (which seems to be related to this bug, though some kind of race condition seems to have confused it much earlier than this total failure), and that was it. We used the instances we had, and I spent 3 hours trying to fix the provisioner, eventually giving up on it. I was very pleased with EC2 and Ubuntu Server itself on the instances. The schroots worked, sbuild worked (though I identified some additional things that the charm should likely do for setup). I think juju has a lot of potential, but I m surprised at how fragile it is. It didn t help that Amazon had rebooted the entire West Coast the day before and there were dead Ubuntu Archive Mirrors in the DNS rotation. For anyone else wanting to spin up builders in the cloud using juju, I have a run-down of what this looks like from the admin s perspective, and even include a little script to produce little slips of paper to hand out to attendees with an instance s hostname, ssh keys, and builder SSH password. Seemed to work pretty well overall; I just wish I could have spun up a few more. :) So, even with the fighting with juju and a few extra instances that came up and I had to shut down again without actually using them, the total cost to run the instances for the whole BSP was about US$40, and including the charm development time, about US$45. UPDATE: some more details on how to avoid the glitches I hit.

2011, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.

At work we use .procmailrc files generated by CGIpaf to let non-technical users create forwards, out-of-office mails, etc. and any combination thereof. This also has the advantage that we can filter out double bounces and spam (which also prevents us from being listed in spammer blacklists). Unfortunately autofs (seems independent if autofs4 or autofs5 is used) seems to be unreliable if there are bursts of mount or umount requests, resulting either in File or directory not found error message while trying to access the home directory of a user, or Directory not empty error messages if the automounter tries to remove the mount point after unmounting. In that case a not mounted directory owned by root is left over. In the end both cases lead to procmail behaving as if that user does not have a .procmailrc which looks like sporadically lost mails to those who forward all mails. (The mails then can be found in the local default INBOX for that user.) Additionally there are similar issues when the NFS servers are not available. The most effective countermeasure we found so far was adding tests to the global /etc/procmailrc to check if the user s home directory exists and belongs to the correct user:

# -----------------
# Global procmailrc
# -----------------
# For debugging, turn off if everything works well
VERBOSE=1
LOGFILE=/var/log/procmail.log
# This only works with bourne shells, $SHELL defaults to the user's
# login shell. And by experience dash seems not work, so we use bash.
OLDSHELL=$SHELL
SHELL=/bin/bash
# temporary failure (see EX_TEMPFAIL in /usr/include/sysexits.h) if
# $LOGNAME is not set for some reason. (Just to be sure our paths
# later on are not senseless.
:0
* ? test -z "$LOGNAME"
 
    LOG="Expected variable LOGNAME not set. "
    EXITCODE=75
    :0
    /dev/null
 
# temporary failure (see EX_TEMPFAIL in /usr/include/sysexits.h) if
# $HOME is not readable. ~$LOGNAME does not seem to work, so this uses
# a hard wired /home/.
:0
* ? test ! -r /home/$LOGNAME
 
    LOG="Home of user $LOGNAME not readable: /home/$LOGNAME "
    EXITCODE=75
    :0
    /dev/null
 
# temporary failure (see EX_TEMPFAIL in /usr/include/sysexits.h) if
# $HOME has wrong owner. ~$LOGNAME does not seem to work, so this uses
# a hard wired /home/.
:0
* ? test ! -O /home/$LOGNAME
 
    LOG="Home of user $LOGNAME has wrong owner: /home/$LOGNAME "
    EXITCODE=75
    :0
    /dev/null
 
[ ]

If you want to store a copy of these mails for debugging purposes on every delivery attempt, replace /dev/null with some Maildir or mbox only accessible for root. One small but important part was to explicitly declare bash as shell for executing the tests, otherwise mails for users with tcsh or zsh as login shell filled up the mail queue and never get delivered (if the SHELL variable never gets fixed). Only drawback so far: This leads to more lagging e-mail on e-mail bursts also for those users who have no .procmailrc because procmail can t check if there s really no .procmailrc. Extensive procmail documentation can be found online at the Procmail Documentation Project as well as in the man pages procmail(1), procmailrc(5) and procmailex(5).

Alexander has already announced the final results of this weekend's BSP, but I'd like to add my personal thanks. So, kudos to all attendees, for helping to make the BSP such a success and also to my employer for the generous budget and my colleagues, especially Wolfram to whose initiative we owe this BSP, for helping out before, during and after the party. All in all it's been an event well worth repeating!

Uff. Two and a half days of hunting release critical bugs ended yesterday evening. It think it was quite a success. According to the statistics we closed 23 bugs with direct uploads, we created patches for 11, 9 bugs got solved by package removals, 6 will be closed by uploads via the delayed queue, 2 bugs were closed as unreproducible, and 9 other bugs where closed as well (e.g. already fixed in older version). On the other hand we opened 3 and upgraded 6 bugs to be release critical. So in the end, we solved about... 60 bugs! Wow, I'm really impressed! I'm also really pleased how well we integrated the newbs. As I already mentioned, when planing for this BSP started, we were a bit worried about the DD / non-dd ratio, but in the end everything worked out very well. All of those, who registered as interested users where able to fix some rc bugs, or help sort out some BTS hickups, e.g. bugs opened with package a but closed in package b. And as a special pleasure it seems, that the longest outstanding rc bug might be fixed soonish, too :) I'd also like to thank Pengutronix for hosting and sponsoring the BSP! The infrastructure worked very well: You arrived, plugged in your notebook, had a local mirror, IPv6 and fast enough connection to the outside. They also provided lots of snacks, cooked food and drinks (later they complained Debian people won't drink enough; apparently they calculated with way more drinks) and a general good atmosphere. PS: To the best of my knowledge, it was also the greenest bug squashing party ever, with most foods and all the electricity coming from certified ecological production. Cool, eh?

When someone wants to package a piece of software for Debian, he has to file an Intent to Package, by filing a special bug report against the pseudo-package wnpp (work-needing and prospective packages) with a title of ITP: software name . This way, we can synchronize our efforts, for instance if someone else wants to package the same piece of software. I have seen two recent examples of a misunderstanding of the correct use of such reports, so perhaps this reminder may be of some help for newcomers.While an Request for Package means that its author would like someone to package a piece of software, and ITP means that its author wants to package it himself and to have it integrated to Debian (in fact, an RFP also becomes an ITP when someone is willing to do the work). This means that the owner of an ITP bug is responsible for the corresponding work. Given that the ITP report is meant to inform others, that implies that he should document his progress by appending messages to the bug report. In particular, when something is blocking the work (typically a failure to find a sponsor), that should be indicated in the ITP report so that it does not appear as being abandoned. So, in a nutshell, when you have opened an ITP, do not expect someone to take it, to reply to it or whatever deus ex machina; you are now the owner of that bug, which means you are responsible of making it live and eventually closing it by sending your package to Debian. The progress of a packaging work should be documented, especially if it requires some specific help such as sponsoring, and the ITP report is the best place to do that since this is where interested people will look for status information.

Four years ago, during DebConf 7 in Edinburgh, I had the idea of creating init scripts not by hand, but rather describe them in a declarative way that handles most common cases and generate the real init scripts from them. Back than, this was partly motivated by the varying quality of init scripts and the advent of an alternative init system, upstart, for which proper service files could be generated as well. I implemented the idea and uploaded it as metainit to Debian, but it did not take off. Only one package uses it, and this package is not used much either. The project basically fell dormant. Yesterday, I got this mail:

Hi Joachim, I'm not very familiar with Debian ecosystem, however our company has recently switched to exclusively using Debian based servers, please forgive me if there exists a better channel for sending patches. We need (in house) to create a lot of trivial services and the sysv init system seems an ideal way to manage them, and metainit an great way to keep the config simple. However all our scripts require a heartbeat, the way we normally do this is with a bunch of checkers launched from cron. Metainit was ideal, but lacked the simple 'status' task found in most init scripts. This small patch adds that task, I hope you and others find it usefull. Regards,

together with the mentioned patch. So it turned out that the idea was a useful one after all, although it found its use on our user s machines, and not within the Debian packages, as I originally anticipated. Nice! During the recent discussions about the latest init system, systemd, there were more thoughts about generating init scripts/upstart service descriptions/systemd units from a declarative description, and if metainit had taken off back then, things would be easier now. So if anyone feels like reviving the project: Please do.

Flattr this post

We have customers, which require a report about what we've done during maintenance windows. Usually this includes a report about upgrades, newly installed packages etc. and obviously everything we've done apart from that.
Till now we've prepared them manually. For a greater bunch of systems this is a big PITA, because to be somehow useful you have to collect the data for all systems and after that, prepare a report where you have:

• The upgrades done on all systems (e.g. a libc or kernel update)
  
  seperated from
  
• the updates specific to a certain system or system class

Its also error-prone because humans make mistakes.

Perl to the rescue!
At least the part about generating a report about installed/upgraded/removed packages could be automated, because dpkg writes a well-formed logfile in /var/log/dpkg.log. But I noticed that there appearently is no library specialised at parsing that file. Its not a big deal, because the format of that file is really simple, but a proper library would be nice anyway.
And so I wrote such a library.
It basically takes a logfile, reads it line by line and stores each line parameterized into a generic object.

Its features include:

• Parsing the logfile (obviously)
• Storing each line in a logfile as a DPKG::Log::Entry object that holds informations about the entry type (e.g. action or status update), the package associated (if any), timestamp, verbatim line, line number etc.

• Limiting the number of lines parsed to a time range specified by DateTime objects

Based on that, I wrote another library DPKG::Log::Analyse, which takes a log, parses it with DPKG::Log and then extracts the more relevant information such as installed packages, upgraded packages, removed packages etc.

This, in turn, features:
- Info about newly installed packages
- Info about removed packages
- Info about upgraded packages
- Info about packages which got installed and removed again
- Info about packages which stayed halfinstalled or halfconfigured at the end of the logfile (or defined reporting period)


These libraries are already uploaded to CPAN and packaged for Debian.
They passed the NEW queue very quickly and are therefore available for Sid:

http://packages.debian.org/sid/libdpkg-log-perl

As an example use (and for my own use case, as stated above), I wrote dpkg-report, which uses the module and a Template::Toolkit based template to generate a report about what happened in the given logfile.
It currently misses some documentation, but it works somehow like this:

Report for a single host over the full log:

dpkg-report

Report for a single host for the last two days:

dpkg-report --last-two-days

Report for multiple hosts (and logfiles):
The script expects that each log file has the name <systemname>.dpkg.log so that it can guess the hostname from the system and can grab all such log files from a directory if a directory is specified as log-file arguments:</systemname>

dpkg-report --log-file /path/to/logs

This will generate a report about all systems without any merging.

Report for multiple hosts with merging:

dpkg-report --log-file /path/to/logs --merge

This will do the following:

• Report packages which where installed/removed/etc. on all systems seperate from the upgrades only done on the specific systems
• If systems start with a common name and end on a number (e.g. mail1, mail2, mail3 etc.), report packages which where installed on all systems with that common name seperate from the upgrades only done on the specific systems
• For a specific system list only the changes on that particular system and nothing else.

A (fictive) report could look some what like this:

dpkg-Report for all:
------------------------
Newly Installed:
abc
Removed:
foo
Upgraded:
bar (0.99-12 - 1.00-1)
dpkg-Report for test*:
------------------------
Newly Installed:
blub
dpkg-Report for test1:
------------------------
Upgraded:
baz (1.0.0-1 -> 1.0.0-2)
dpkg-Report for test2:
------------------------
Upgraded:
zab (0.9.7 -> 0.9.8)

Currently this report generator is only included in the source package or in the git(hub) repository of the library. I wonder if it makes sense to let the source package build another binary package for it.
But its only a 238 lines perl script with a dependency on the perl library so I'm unsure if it warrants a new binary package. What do others think?