release is the tool that I use to manage software releases. This version adds support for signing a tarball with multiple PGP keys at once. For WebAuth, we have an official signing key, but I also wanted to add my personal signature. The minimum supported version of Python was also bumped to 2.5 (see below for why). You can get the latest version from my scripts page. Normally, I wouldn't announce this to everyone, since it's a fairly minor change, except I had a question and a quick Python note. The question: how does one get GnuPG to verify multiple signatures? When I run GnuPG on the resulting detached *.asc file created by this new code, I get the following:

gpg: WARNING: multiple signatures detected.  Only the first will be checked.
gpg: Signature made Thu 10 Jan 2013 03:58:43 PM PST using RSA key ID 5736DE75
gpg: Good signature from "Russ Allbery <rra@stanford.edu>"
gpg:                 aka "Russ Allbery <rra@debian.org>"
gpg:                 aka "Russ Allbery <eagle@windlord.stanford.edu>"
gpg:                 aka "Russ Allbery <eagle@eyrie.org>"

That's nice, I suppose, but it's not as helpful as it could be. Why not verify both signatures and report all of the results? I searched both the gpg man page and the Internet for some option I'm missing, without any luck. (Searching Google for that exact message amusingly turns up lots of verbatim output from Debian repository verification and almost nothing else.) If anyone knows, send me email (rra@stanford.edu or any other convenient address), and I'll update this post with the answer. UPDATE: gpg can currently only verify multiple signatures if the signatures are from keys with the same class and digest. I was attempting this with a 2048-bit RSA key and a 1024-bit DSA key, which gpg does not support. (Yet another reason to replace the WebAuth signing key with a newly-generated key.) Thanks to Philip Martin and Bernhard R. Link for the information! The note is that, way back when I was first learning Python, one of the things that bothered me the most was the lack of safe program execution. The best available was os.spawnv(), which left something to be desired. If you too were bothered by this but don't program in Python enough to keep track of the current state of the art, there's now a subprocess module that has much saner interfaces that work properly, do reasonable things, and avoid the shell. It was added in Python 2.4, with some nicer interfaces added in Python 2.5, so it's fairly safe to use at this point.

The QNAP TS-XXX NAS boxes are nice little systems but one thing I really miss is a console. Martin Michlmayr has some instructions (TS-119, TS-219, TS-41x) on his excellent Debian On QNAP pages on how to build a suitable adaptor but sadly even though I've managed to get all the parts I've been too lazy to actually solder the thing together (it doesn't help that I nearly always burn myself when I use a soldering iron!). This became more pressing when someone reported that Debian bug #693263 in qcontrol was not fixed in Wheezy since the issue appeared to be in the initramfs hook. Worse it seemed like something in my proposed fix was causing the system to not boot at all! Having finally found the hardware needed to reproduce the issue my first thought was to try netconsole. To do this I edited /etc/initramfs-tools/modules to add:

      mv643xx_eth
      netconsole netconsole=@<IP>/eth0,@<DST-IP>/<DST-MAC>

The option syntax is described in netconsole.txt but briefly: <IP> is the address of the TS-419P II I'm debugging on, and <DST-IP> and <DST-MAC> are the IP and MAC address of another machine on the network. Having done that, running update-initramfs -u and rebooting I can use netcat -u -l -p 6666 on the other machine to see all the kernel messages. So far so good but this doesn't get me any debugging from the userspace portions of the initramfs. To get those we have to get a bit hacky by editing /usr/share/initramfs-tools/init, first to change:

     # Parse command line options
    -for x in $(cat /proc/cmdline); do
    +for x in $(cat /proc/cmdline) debug; do
            case $x in

and secondly:

             debug)
                     debug=y
                     quiet=n
    -                exec >/run/initramfs/initramfs.debug 2>&1
    +                exec >/dev/kmsg 2>&1
                     set -x
                     ;;

The first of these simulates adding debug to the kernel command line (which can't otherwise easily be edited on these systems) and the second redirects the initramfs process's output to the kernel log. The overall effect is that the output of the initramfs processes appears over netcat.

When writing system integration tests it often happens that I want to mount some tmpfses over directories like /etc/postgresql/ or /home, and run the whole script with an unshared mount namespace so that (1) it does not interfere with the real system, and (2) is guaranteed to clean up after itself (unmounting etc.) after it ends in any possible way (including SIGKILL, which breaks usual cleanup methods like trap , finally , def tearDown() , atexit() and so on). In gvfs and postgresql-common s tests, which both have been around for a while, I prepare a set of shell commands in a variable and pipe that into unshare -m sh, but that has some major problems: It doesn t scale well to large programs, looks rather ugly, breaks syntax highlighting in editors, and it destroys the real stdin, so you cannot e. g. call a bash -i in your test for interactively debugging a failed test. I just changed postgresql-common s test runner to use unshare/tmpfses as well, and needed a better approach. What I eventually figured out preserves stdin, $0, and $@, and still looks like a normal script (i. e. not just a single big string). It still looks a bit hackish, but I can live with that:

#!/bin/sh
set -e
# call ourselves through unshare in a way that keeps normal stdin, $0, and CLI args
unshare -uim sh -- -c " tail -n +7 $0 " "$0" "$@"
exit $?
# unshared program starts here
set -e
echo "args: $@"
echo "mounting tmpfs"
mount -n -t tmpfs tmpfs /etc
grep /etc /proc/mounts
echo "done"

As Unix/Linux shebang parsing is rather limited, I didn t find a way to do something like

#!/usr/bin/env unshare -m sh

If anyone knows a trick which avoids the tail -n +7 hack and having to pay attention to passing around $@ , I d appreciate a comment how to simplify this.

A few days ago, I pushed out version 2.11.14 of the Geode X.org driver. This is the driver used by the OLPC XO-1 and by a plethora of low-power desktops, micro notebooks and thin clients.
This release mostly features long-overdue fixes to rendering issues under GTK3+ and xulrunner, plus yet more ongoing changes to make this driver compile under recent X servers.
Sadly, the release took place much too late to be included into the upcoming stable Debian version, Wheezy, which is already deeply into freeze, pending publication.

With python-dbusmock you can provide mocks for arbitrary D-BUS services for your test suites or if you want to reproduce a bug. However, when writing actual tests for gnome-settings-daemon etc. I noticed that it is rather cumbersome to always have to set up the skeleton of common services such as UPower. python-dbusmock 0.2 now introduces the concept of templates which provide those skeletons for common standard services so that your code only needs to set up the particular properties and specific D-BUS objects that you need. These templates can be parameterized for common customizations, and they can provide additional convenience methods on the org.freedesktop.DBus.Mock interface to provide more abstract functionality like add a battery . So if you want to pretend you have one AC and a half-charged battery, you can now simply do

  def setUp(self):
     (self.p_mock, self.obj_upower) = self.spawn_server_template('upower',  )
  def test_ac_bat(self):
     self.obj_upower.AddAC('mock_AC', 'Mock AC')
     self.obj_upower.AddChargingBattery('mock_BAT', 'Mock Battery', 50.0, 1200)

Or, if your code is not in Python, use the CLI/D-BUS interface, like in shell:

  # start a fake system bus
  eval  dbus-launch 
  export DBUS_SYSTEM_BUS_ADDRESS=$DBUS_SESSION_BUS_ADDRESS
  # start mock upower on the fake bus
  python3 -m dbusmock --template upower &
  # add devices
  gdbus call --system -d org.freedesktop.UPower -o /org/freedesktop/UPower \
      -m org.freedesktop.DBus.Mock.AddAC mock_ac 'Mock AC'
  gdbus call --system -d org.freedesktop.UPower -o /org/freedesktop/UPower \
      -m org.freedesktop.DBus.Mock.AddChargingBattery mock_bat 'Mock Bat' 50.0 1200

In both cases upower --dump or gnome-power-statistics will show you the expected devices (of course you need to run that within the environment of the fake $DBUS_SYSTEM_BUS_ADDRESS, or run the mock on the real system bus as root). Iftikhar Ahmad contributed a template for NetworkManager, which allows you to easily set up ethernet and wifi devices and wifi access points. See pydoc3 dbusmock.templates.networkmanager for details and the test cases for how this looks like in practice. I just released python-dbusmock 0.2.1 and uploaded the new version to Debian experimental. I will sync it into Ubuntu Raring in a few hours.

Dear lazyweb, we need a new printer-scanner combination for our new office, ideally a laser printer and a high-resolution scanner (1200dpi). It must integrate nicely with CUPS and SANE, without requiring any proprietary, closed-licence drivers or plugins. It also has to come with an automatic document feeder in addition to the plain flatbed scanner. Additional, would-be-nice features in decreasing order of preference are

• Colour printing (separate cartridges a must)
• Duplex printing (real duplex preferred over soft solutions)
• Direct scanning to PDF (distribution by E-mail, Samba or FTP)

I have investigated the Samsung CLX-6220FX, but the driver situation was such a nightmare (and the printer so loud), that we returned it right away. I should have read this article by Peter beforehand I have recently seen the HP Laserjet Pro 200 M276n, the successor of the HP Laserjet CM1415nf which looks promising, but according to LinuxPrinting.org, a proprietary plugin is required (WTF HP?). Does anyone have experience with these HP models and can recommend them for use with Debian stable? Does anyone have another recommendation? Please send me a message. Thanks!

The F Word has an informative post about men commenting on Feminist blogs [1]. Most of it applies to any situation where a member of a powerful group comments on an issue related to a minority group. Near the end they say: I ll also paraphrase and flesh out the most useful piece of advice I ever read, when I made the effort to research white privilege: don t expect the minority to trust you. Trust is earned, and you re just another commenter who they can t tell apart from any other commenter. You re entering someone else s space, where different rules apply. You get to have the rest of the world for people to assume you re a wonderful person. Here, you re just another one of them , and given the track record of them , it s up to you to listen, learn and prove that you re being thoughtful and honestly trying to examine your privilege. Sociological Images has an interesting article on people s perception of the sky colour it seems that the blue sky meme is modern [2]. Charles Stross wrote an interesting article about the possible uses for future low power computers [3]. He gets a bit over-excited about the possibilities for sensing making a tiny computer that can sense so many things isn t going to be easy (DSLRs are big for a reason). But having lots of powerful computers everywhere does provide lots of interesting and potentially bad opportunities. Martin Bekkelund has an interesting article about Amazon wiping DRM infected books that it had sold to a customer without giving a refund or an explanation [4]. If you want to buy ebooks it seems that the sensible thing to do would be to immediately crack them or download them from The Pirate Bay so Amazon can t steal them back. Rebecca Saxe gave an interesting talk about trying to develop methods for conflict resolution through neuroscience [5]. Barry Eisler has written an interesting article about the corruption of journalists [6]. It s really worth reading, some of the methods of corruption apply to even the more casual bloggers. Krebs on Security has an informative article about the Microsoft Tech Support phone scams [7]

• [1] http://www.thefword.org.uk/features/2008/06/men_feminism_ne
• [2] http://thesocietypages.org/socimages/2012/07/30/is-the-sky-blue
• [3] http://www.antipope.org/charlie/blog-static/2012/08/how-low-power-can-you-go.html
• [4] http://www.bekkelund.net/2012/10/22/outlawed-by-amazon-drm/
• [5] http://edge.org/conversation/imaging-conflict-resolution
• [6] http://barryeisler.blogspot.com.au/2012/08/you-will-be-assimilated.html
• [7] http://krebsonsecurity.com/2012/08/tech-support-phone-scams-surge/

A blog post from Martin Bekkelund today tell the story of how Amazon erased the books from a customer's kindle, locked the account and refuse to tell the customer why. If a real book store did this to a customer, it would be called breaking into private property and theft. The story has spread around the net today. A bit more background information is available in Norwegian from digi.no. It is no surprise that digital restriction mechanisms (DRM) are used this way, as it has been warned about such abuse since DRM was introduced many years back. And Amazon proved in 2009 that it was willing to break into customers equipment and remove the books people had bought, when it removed the book 1984 by George Orwell from all the customers who had bought it. From the official comments, it even sounded like Amazon would never do that again. And here we are, three years later. And thought this action is against Norwegian regulations and law, it is according to the terms of use as written by Amazon, and it is hard to hold Amazon accountable to Norwegian laws. It is just yet another example of unacceptable terms of use on the web, and how they are used to remove customer rights. Luckily for electronic books, there are alternatives without unacceptable terms. For example Project Gutenberg (about 40,000 books), Project Runenberg (1,652 books) and The Internet Archive (3,641,797 books) have heaps of books without DRM, which can read by anyone and shared with anyone. Update 2012-10-23: This story broke in the morning on Monday. In the evening after the story had spread all across the Internet, Amazon restored the account of the user, as reported by digi.no and NRK. Apparently public pressure work. The story from Martin have seen several twitter messages per minute the last 24 hours, which is quite a lot, and is still drawing a lot of attention. But even when the account is restored, the fundamental problem still exist. I recommend reading two opinions from Simon Phipps and Glen Moody if you want to learn more about the fundamentals and more details about the original story.

I am moving into an office. And as any proud husband and father, I want to have photos of my wife and daughter on my desk. The year being 2012, I did not hike to the photo shop to get a print framed, but instead thought to myself that I should put a digital photo frame onto my table. This idea exposed me to the ridiculous world of consumer electronics. It led me to conclude that digital picture frame designers need to be whacked with cluebats. Step by step Once accepted, the thought of a digital photo frame developed into a product definition along the following lines: my idea photo frame would connect to my Wifi-network, and obtain the photos on-the-go from a folder exposed via HTTP or CIFS, and then go on to display them in random order, incorporating new photos as it encounters them. With this in mind, I went to the shops, and since I believe in specialised retailers and want to support them, my first stop was Foto Sauter at Sendlinger Tor. Unfortunately, none of the frames they had came with Wifi, so I decided to look further. I vehemently oppose to the business practices of the Metro group, thus skipped Saturn and MediaMarkt, and eventually ended up at Conrad. They had a frame with Wifi! I jumped for joy, until I read the manual: pictures can be obtained from Flickr and Picasa. Period. All other models on the Internet seem to be similarly limited, including the new Sony S-frame. The night before, Penny had researched the field a bit and came to the conclusion that the S-frame would be the best product available. This led me to scratch Wifi off my requirements list and get a model that would read photos off a USB stick. I went back to the photo store and bought a "Sony S-frame", only to discover that it cannot show photos in random order. It has three viewing modes (single photo, collage, single photo with clock), and a random mode, but guess what: the random mode randomly switches the viewing modes, which then display the photos in lexicographical order. How stupid is that??? I returned the product and left the store after discovering that none of their products could do random playback. I went back to Conrad and found an "Intenso MediaCreator" (what media does it create???), which displayed the photos seemingly randomly. But at home I found out that the "random" order is always the same, probably because the bright engineer that programmed this thought it was better to sort filenames by last letter and call it random, than to figure out a way to roll a dice on the device. I wrote to the support team and asked them. The response was that the desired functionality (random selection) is not possible and won't be made available. So I am returning the product. Gah! Would someone please tell me about a digital picture frame (8 inch or so) that can display images in random order, ideally loading them off a CIFS share via Wifi? Or is it really the case that consumer electronics are completely useless these days, by which I mean that "consumers" have dumbed down so far to buy this crap? Update: a lot of people wrote in suggesting to invest in a cheap Android tablet. Some suggested Raspberry Pis in USB host mode (emulating the USB stick and hence the source of the images, provided that the frame doesn't cache). Other suggestions included the Samsung SPF-85V which can display images according to an RSS feed but needs Microsoft for that (or maybe not), and the community-developed, Linux-based Joggler. Regarding the non-random order on the Intenso frame, Paul Hedderly postulated that the order comes from the filesystems (FAT order) and can be changed by writing the files differently.

Puppet I've really had it with Puppet. I used to be able to put up with all its downsides

• Non-Unix approach to everything (own transport, self-made PKI, non-intuitive configuration language, a faint attempt at versioning (bitbucket), and much much more )
• Ruby
• Abysmal slowness
• Lack of basic functionality (e.g. replace a line of text)
• Host management and configuration programming intertwined, lack of a high-level approach to defining functionality
• Horrific error messages
• Catastrophic upgrade paths
• Did I mention Ruby and its speed?
• Lack of IPv6 support
• [I could keep going ]

but now that my fourth attempt to upgrade my complex configuration from version 0.25.5 to version 2.7 failed due to a myriad of completely incomprehensible errors ("err: Could not run Puppet configuration client: interning empty string") and many hours were lost in trying to hunt these down using binary searches, I am giving up. Bye bye Puppet.

An alternative But I need an alternative. I want a system that is capable of handling a large number of hosts, but not so complex that one wouldn't put it to use for half a dozen machines. The configuration management system I want looks about as follows: It

• makes use of existing infrastructure (e.g. SSH transport and public keys, Unix toolchain, Debian package management and debconf)
• interacts with the package management system (Debian only in my case)
• can provision files whose contents might depend on context, particular machine data and conditionals. There should be a unified templating approach for static and dynamic files, with the ability to override the source of data (e.g. a default template used unless a template exists for a class of machine, or a specific hostname)
• can edit files on the target machine in a flexible and robust manner
• can remove files
• can run commands when files change
• can reference data from other machines (e.g. obtain the certificate fingerprint of each hosts that define me as their SMTP smarthost)
• can control running services (i.e. enable init.d scripts, check that a process is running
• is written in a sensible language
• is modular and easily extensible, ideally using a well-known language (e.g. Python!)
• allows to specify infrastructure with tags ("all webservers", "all machines in Zurich", "machines that are in Munich and receive mail"), but with the ability to override every parameter for a specific host
• should just do configuration management, and not try to take away jobs from monitoring software
• logs changes per-machine and collects data about applied configurations in a central location
• is configured using flat files that are human-readable so that the configuration may be stored in Git (e.g. YAML, not XML)
• can be configured using scripts in a flexible way

Since for me, Ruby is a downside of Puppet, I won't look at Chef, but from this page, I gleaned a couple of links: Ansible, Quattor, Salt, and bcfg2 (which uses XML though). And of course, there remains the ephemeral cfengine.

cfengine I haven't used cfengine since 2002, but I am not convinced it's worth a new look because it seems to be an academic project with gigantic complexity and a whole vernacular to its own. There is no doubt that it is a powerful solution, and the most mature of all of them, but it's far away from the Unix-like simplicity that I've come to love in almost 20 years of Debian. Do correct me if I am wrong.

Ansible Ansible looks interesting. It seems rather bottom-up, first introducing a way to remotely execute commands on hosts, which you can then later extend/automate to manage the host configurations. It uses SSH for transport, and its reason-to-be made me want to look at it. My ventures into the Ansible domain are not over yet, but I've put them on hold. First of all, it's not yet packaged for Debian (Ubuntu-PPA packages work on Debian squeeze and wheezy). Second, I was put off a bit by its gratuitous use of the shell to run commands, as well as other design decisions. Check this out: there are modules for the remote execution of commands, namely "shell", "command", and "raw". The shell modules should be self-explanatory; the command module provides some idempotency, such as not running the command if a file exists (or not). To do this, it creates a Python script in /tmp on the target and then executes that like so:

$SHELL -c /tmp/ansible/ansible-1350291485.22-74945524909437/command

Correct me if I am wrong, but there is zero need for this shell indirection. My attempts at finding an answer on IRC were met by user "daniel_hozac" with a reason along the lines of "it's needed, believe me", and on the mailing list, I am told that only the shell can execute a script by parsing the interpreter line at the top of the module. Finally, the raw execution module also executes using the shell And there a few other design decisions that I can't quite explain, around the command-line switch --sudo see the aforementioned message In short: running a command like

ansible -v arnold.madduck.net -a "/usr/bin/apt-get update" --sudo

does not invoke apt-get with sudo, as one might like; it invokes the shell that runs the Python script that runs the command. Effectively therefore, you need to allow sudo shell execution, and for proper automation, this has to be possible without a password. And then you might just as well allow root logins again. The author seems to think that "core behaviour" is that sudo allows all execution and that limiting the commands to run is not a use-case that Ansible will support. Apparently, I was the first to ever suggest this. There are always ways around (e.g. skip --sudo and just use sudo as the command, simply ignore the useless shell invocation and trust that your machine can handle it, but when such design decisions remain incomprehensible and get defended by the project people, then I am hesitant to invest more time on principle.

Salt Finally, I've looked at Salt, which is what I've spent most time on so far. From the discussions I started on host targeting and data collection, it soon became apparent that Salt is very thin and flexible, and that the user community is accomodating. Unfortunately, Salt does not use SSH, but at least it reuses existing functionality (ZeroMQ). As opposed to the push/pull model, Salt "minions" interestingly maintain a persistent connection to the server (which is not yet very stable), and while non-root usage is still not unproblematic, at least there has already been work done in this direction. I think I will investigate Salt more as it does look like it can do what I want. The YAML-based syntax does seem a bit brittle, but it's the best I've found so far. NP: The Pineapple Thief: Someone Here is Missing

For writing tests for GVFS (current tests, proposed improvements) I want to run Samba as normal user, so that we can test gvfs smb backend without root privileges and thus can run them safely and conveniently in a make check environment for developers and in JHBuild for continuous integration testing. Before these tests could only run under gvfs-testbed, which needs root. Unlike other servers such as ssh or ftp, this turned out surprisingly non-obvious and hard, so I want to document it in this blog post for posterity s benefit. Running the server Running smbd itself is mainly an exercise of figuring out all the options that you need to set; Alex Larsson and I had some fun figuring out all the quirks and hiccups that happen between Ubuntu s and Fedora s packaging and 3.6 vs. 4.0, but finally arrived at something working. First, you need to create an empty directory where smbd can put all its databases and state files in. For tests you would use mkdtemp(), but for easier reading I just assume mkdir /tmp/samba here. The main knowledge is in the Samba configuration file, let s call it /tmp/smb.conf:

[global]
workgroup = TESTGROUP
interfaces = lo 127.0.0.0/8
smb ports = 1445
log level = 2
map to guest = Bad User
passdb backend = smbpasswd
smb passwd file = /tmp/smbpasswd
lock directory = /tmp/samba
state directory = /tmp/samba
cache directory = /tmp/samba
pid directory = /tmp/samba
private dir = /tmp/samba
ncalrpc dir = /tmp/samba
[public]
path = /tmp/public
guest ok = yes
[private]
path = /tmp/private
read only = no

For running this as a normal user you need to set a port > 1024, so this uses 1445 to resemble the original (privileged) port 445. The map to guest line makes anonymous logins work on Fedora/Samba 4.0 (I m not sure whether it s a distribution or a version issue). Don t ask about dir vs. directory , that s an inconsistency in Samba; with above names it works on both 3.6 and 4.0. We use the old smbpasswd backend as shipping large tdb files is usually too inconvenient and brittle for test suites. I created an smbpasswd file by running smbpasswd on a real Samba installation, and then using pdbedit to convert it to a smbpasswd file:

sudo smbpasswd -a martin
sudo pdbedit -i tdbsam:/var/lib/samba/passdb.tdb -e smbpasswd:/tmp/smbpasswd

The result for password foo is

myuser:0:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:AC8E657F83DF82BEEA5D43BDAF7800CC:[U          ]:LCT-507C14C7:

which you are welcome to copy&paste (you can replace myuser with any valid user name, of course). This also defines two shares, one public, one authenticated. You need to create the directories and populate them a bit:

mkdir /tmp/public /tmp/private
echo hello > /tmp/public/hello.txt
echo secret > /tmp/private/myfile.txt

Now you can run the server with

smbd -iFS -s /tmp/smb.conf

The main problem with this approach is that smbd exits ( Server exit (failed to receive smb request) ) after a client terminates, so you need to write your tests in a way to only run one connection/request per test, or to start smbd in a loop. Running the client If you merely use the smbclient command line tool, this is rather simple: It has a -p option for specifying the port:

$ smbclient -p 1445 //localhost/private
Enter martin's password: [enter "foo" here]
Domain=[TESTGROUP] OS=[Unix] Server=[Samba 3.6.6]
smb: \> dir
  .                                   D        0  Wed Oct 17 08:28:23 2012
  ..                                  D        0  Wed Oct 17 08:31:24 2012
  myfile.txt                                   7  Wed Oct 17 08:28:23 2012

In the case of gvfs it wasn t so simple, however. Surprisingly, libsmbclient does not have an API to set the port, it always assumes 445. smbclient itself uses some internal libcli API which does have a way to change the port, but it s not exposed through libsmbclient. However, Alex and I found some mailing list posts ([1], [2]) that mention $LIBSMB_PROG, and it s also mentioned in smbclient s manpage. It doesn t quite work as advertised in the second ML post (you can t set it to smbd, smbd apparently doesn t speak the socket protocol over stdin/stdout), and it s not being used anywhere in the current Samba sources, but what does work is to use good old netcat:

export LIBSMB_PROG="nc localhost 1445"

with that, you can use smbclient or any program using libsmbclient to talk to our test smb server running as user.

Rcpp release 0.9.15 is now on CRAN and being uploaded to Debian. Martin Morgan provided a clever fix for a header search needed between clang++ (especially on OS X) and g++ (which still provided libstdc++ and headers for clang++). This should hopefully put the clang issues to bed. Ben North noticed an unprotected string conversion when exception messages are turned into R errors which got fixed, and I expanded the coverage of Date (and Datetime) types to deal properly with non-finite values NA, NaN and Inf. The complete NEWS entry for 0.9.15 is below; more details are in the ChangeLog file in the package and on the Rcpp Changelog page.

Changes in Rcpp version 0.9.15 (2012-10-13)

• Untangling the clang++ build issue about the location of the exceptions header by directly checking for the include file an approach provided by Martin Morgan in a kindly contributed patch as unit tests for them.
• The Date and Datetime types now correctly handles NA, NaN and Inf representation; the Date type switched to an internal representation via double
• Added Date and Datetime unit tests for the new features
• An additional PROTECT was added for parsing exception messages before returning them to R, following a report by Ben North

Thanks to CRANberries, you can also look at a diff to the previous release 0.9.14. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

I found it surprisingly hard to determine in tearDown() whether or not the test that currently ran succeeded or not. I am writing some tests for gnome-settings-daemon and want to show the log output of the daemon if a test failed. I now cobbled together the following hack, but I wonder if there s a more elegant way? The interwebs don t seem to have a good solution for this either.

    def tearDown(self):
        [...]
        # collect log, run() shows it on failures
        with open(self.daemon_log.name) as f:
            self.log_output = f.read()
    def run(self, result=None):
        '''Show log output on failed tests'''
        if result:
            orig_err_fail = result.errors + result.failures
        super().run(result)
        if result and result.errors + result.failures > orig_err_fail:
            print('\n----- daemon log -----\n%s\n------\n' % self.log_output)

I've just made release 1.2 of Obnam, my backup program. The NEWS file entry:

• Added a note to --node-size that it only affects new B-trees. Thanks, Michael Brown.
• New obnam diff subcommand to show differences (added/removed/modified files) between two generations, by Peter Valdemar M rch.
• obnam backup now logs the names of files that are getting backed up at the INFO level rather than DEBUG.
• The command synopsises for backup, restore, and verify commands now make it clearer that Obnam only accepts directories, not individual files, as arguments. (For now.)
• The output from the show plugin can now be redirected with the --output=FILE option. Affected subcommands: clients, generations, genids, ls, diff, nagios-last-backup-age.

Bug fixes:

• Notify user of errors during backups.
• The SFTP plugin now manages to deal with repository paths starting with /~/ which already exist without crashing.
• Character and block device nodes are now restored correctly. Thanks to Martin Dummer for the bug report.
• The symmteric key for a toplevel repository directory is re-encrypted when a public key is added or removed to the toplevel using the add-key or remove-key subcommands.
• Manual page typo fix. Thanks, Steve Kemp.

This release has been uploaded to my own apt repository on code.liw.fi, for squeeze and unstable. Due to the Debian freeze, I am not uploading this to Debian at this time, though some changes may eventually be uploaded there.

This past month has been the most difficult in my life and that of my family as well. I m not going to go into it for the whole Internet, but any well wishes, happy thoughts, prayers, whatever you want to send our way, whether we know about it or not, would certainly be welcome. And, of course, the reason I m not very active online right now is that I m focusing on family, work, and other pressing matters. I have had some measure of comfort from hearing from others that have had pain in their lives. It is good to know I m not alone, good to have people to talk and share with. And it is good to find some way to have hope in the midst of difficulty and uncertainty. I sing with the Kansas Mennonite Men s Chorus, and when listening to one of our recordings recently, was struck by these words in a whole new way. Let music never die in me is a powerful message.

I dreamed a dream, a silent dream,
of a land not far away.
Where no bird sang,
no steeples rang
and teardrops fell like rain. I dreamed a dream.
No alleluia, not one hosanna,
No song of love, no lullaby. And no choir sang to change the world.
No pipers played, no dancers twirled.
I dreamed a dream, a silent dream.
Silent.
Silent. Awake! Awake!
Awake, my soul and sing!
The time for praise has come.
The silence of the night has passed;
a new day has begun.
Let music never die in me!
Forever let my spirit sing!
Wherever emptiness is found,
Let there be joy and glorious sound. Let music never die in me!
Forever let my spirit sing! Let all our voices join as one
to praise the giver of the song! Awake! Awake!
Let music live! The Awakening, Joseph M. Martin (excerpts)

See a performance here. Even when hope is dim, or the music is playing only softly, it s not dead. I hear it when an 85-year-old person in church, comes up to me with tears in her eyes and gives me a big, silent hug. I feel the music when when I can share about things with people, when I sing, when the beautiful Kansas sunset peeks out, when I share a smile or laugh with someone, and when I see the delight and happiness of children. Let music live!

I was working on writing tests for gnome-settings-daemon a week or so ago, and finally got blocked on being unable to set up upower/ConsoleKit/etc. the way I need them. Also, doing so needs root privileges, I don t want my test suite to actually suspend my machine, and using the real service is generally not suitable for test suites that are supposed to run during make check , in jhbuild, and the like these do not have the polkit privileges to do all that, and may not even have a system D-Bus running in the first place. So I wrote a little test_upower.py helper, then realized that I need another one for systemd/ConsoleKit (for the system idle property), also looked at the mock polkit in udisks and finally sat down for two days to generalize this and do this properly. The result is python-dbusmock, I just released the first tarball. With this you can easily create mock objects on D-Bus from any programming language with a D-Bus binding, or even from the shell. The mock objects look like the real API (or at least the parts that you actually need), but they do not actually do anything (or only some action that you specify yourself). You can configure their state, behaviour and responses as you like in your test, without making any assumptions about the real system status. When using a local system/session bus, you can do unit or integration testing without needing root privileges or disturbing a running system. The Python API offers some convenience functions like start_session_bus() and start_system_bus() for this, in a DBusTestCase class (subclass of the standard unittest.TestCase ). Surprisingly I found very little precedence here. There is a Perl module, but that s not particuarly helpful for test suites in C/Vala/Python. And there is Phil s excellent Bendy Bus, but this has a different goal: If you want to thoroughly test a particular D-BUS service, such as ensuring that it does the right thing, doesn t crash on bad input, etc., then Bendy Bus is for you (and python-dbusmock isn t). However, it is too much overhead and rather inconvenient if you want to test a client-side program and just need a few system services around it which you want to set up in different states for each test. You can use python-dbusmock with any programming language, as you can run the mocker as a normal program. The actual setup of the mock (adding objects, methods, properties, etc.) all happen via D-Bus methods on the org.freedesktop.DBus.Mock interface. You just don t have the convenience D-Bus launch API. The simplest possible example is to create a mock upower with a single Suspend() method, which you can set up like this from Python:

import dbus
import dbusmock
class TestMyProgram(dbusmock.DBusTestCase):
[...]
    def setUp(self):
        self.p_mock = self.spawn_server('org.freedesktop.UPower',
                                        '/org/freedesktop/UPower',
                                        'org.freedesktop.UPower',
                                        system_bus=True,
                                        stdout=subprocess.PIPE)
        # Get a proxy for the UPower object's Mock interface
        self.dbus_upower_mock = dbus.Interface(self.dbus_con.get_object(
            'org.freedesktop.UPower', '/org/freedesktop/UPower'),
            'org.freedesktop.DBus.Mock')
        self.dbus_upower_mock.AddMethod('', 'Suspend', '', '', '')
[...]
    def test_suspend_on_idle(self):
        # run your program in a way that should trigger one suspend call
        # now check the log that we got one Suspend() call
        self.assertRegex(self.p_mock.stdout.readline(), b'^[0-9.]+ Suspend$')

This doesn t depend on Python, you can just as well run the mocker like this:

python3 -m dbusmock org.freedesktop.UPower /org/freedesktop/UPower org.freedesktop.UPower

and then set up the mocks through D-Bus like

gdbus call --system -d org.freedesktop.UPower -o /org/freedesktop/UPower \
      -m org.freedesktop.DBus.Mock.AddMethod '' Suspend '' '' ''

If you use it with Python, you get access to the dbusmock.DBusTestCase class which provides some convenience functions to set up and tear down local private session and system buses. If you use it from another language, you have to call dbus-launch yourself. Please see the README for some more details, pointers to documentation and examples. Update: You can now install this via pip from PyPI or from the daily builds PPA. Update 2: Adjusted blog entry for version 0.0.3 API, to avoid spreading now false information too far.

Thoughts about DRI.Next On the way to the X Developer s Conference in Nuremberg, Eric and I chatted about how the DRI2 extension wasn t really doing what we wanted. We came up with some fairly rough ideas and even held an informal presentation about it. We didn t have slides that day, having come up with the content for the presentation in the hours just before the conference started. This article is my attempt to capture both that discussion and further conversations held over roast pork dinners that week. A brief overview of DRI2 Here s a list of the three things that DRI2 currently offers. Application authentication. The current kernel DRM authentication mechanism restricts access to the GPU to applications connected to the DRM master. DRI2 implements this by having the application request the DRM cookie from the X server which can then be passed to the kernel to gain access to the device. This is fairly important because once given access to the GPU, an application can access any flink d global buffers in the system. Given that the application sends screen data to the X server using flink d buffers, that means all screen data is visible to any GPU-accessing application. This bypasses any GPU hardware access controls. Allocating buffers. DRI2 defines a set of attachment points for buffers which can be associated with an X drawable. An application needing a specific set of buffers for a particular rendering operation makes a request of the X server which allocates the buffers and passes back their flink names. The server automatically allocates new buffers when window sizes change, sending an event to the application so that it knows to request the new buffers at some point in the future. Presenting data to the user. The original DRI2 protocol defined only the DRI2CopyRegion request which copied data between the allocated buffers. SwapBuffers was implemented by simply copy data from the back buffer to the front buffer. This didn t provide any explicit control over frame synchronization, so a new request, DRI2SwapBuffers, was added to expose controls for that. This new request only deals with the front and back buffers, and either copies from back to front or exchanges those two buffers. Along with DRI2SwapBuffers, there are new requests that wait for various frame counters and expose those to GL applications through the OMLsynccontrol extension What s wrong with DRI2? DRI2 fixed a lot of the problems present with the original DRI extension, and made reliable 3D graphics on the Linux desktop possible. However, in the four years since it was designed, we ve learned a lot, and the graphics environment has become more complex. Here s a short list of some DRI2 issues that we d like to see fixed.

• InvalidateBuffers events. When the X window size changes, the buffers created by the X server for rendering must change size to match. The problem is that the client is presumably drawing to the old buffers when the new ones are allocated. Delivering an event to the client is supposed to make it possible for the client to keep up, but the reality is that the event is delivered at some random time to some random thread within the application. This leads to general confusion within the application, and often results in a damaged frame on the screen. Fortunately, applications tend to draw their contents often, so the damaged frame only appears briefly.
• No information about new back buffer contents. When a buffer swap happens and the client learns about the new back buffer, the back buffer contents are always undefined. For most applications, this isn t a big deal as they re going to draw the whole window. However, compositing managers really want to reduce rendering by only repainting small damaged areas of the window. Knowing what previous frame contents are present in the back buffer allows the compositing manager to repaint just the affected area.
• Un-purgable stale buffers. Between the X server finishing with a buffer and the client picking it up for a future frame, we don t need to save the buffer contents and should mark the buffer as purgable. With the current DRI2 protocols, this can t be done, which leaves all of those buffers hanging around in memory.
• Driver-specific buffers. The DRI2 buffer handles are device specific, and so we can t use buffers from other devices on the screen. External video encoders/cameras/encoders can t be used with the DRI2 extension.
• GEM flink has lots of issues. The flink names are global, allowing anyone with access to the device to access the flink data contents. There is also no reference to the underlying object, so the X server and client must carefully hold references to GEM objects during various operations.

Proposed changes for DRI.Next Given the three basic DRI2 operations (authentication, allocation, presentation), how can those be improved? Eliminate DRI/DRM magic-cookie based authentication Kristian H gsberg, Martin Peres, Timoth e Ravier & Daniel Vetter gave a talk on DRM2 authentication at XDC this year that outlined the problems with the current DRM access control model and proposed some fairly simple solutions, including using separate device nodes one for access to the GPU execution environment and a separate, more tightly controlled one, for access to the display engine. Combining that with the elimination of flink for communicating data between applications and there isn t a need for the current magic-cookie based authentication mechanism; simple file permissions should suffice to control access to the GPU. Of course, this ignores the whole memory protection issue when running on a GPU that doesn t provide access control, but we already have that problem today, and this doesn t change that, other than to eliminate the global uncontrolled flink namespace. Allocate all buffers in the application DRI2 does buffer allocation in the X server. This ensures that that multiple (presumably cooperating) applications drawing to the same window will see the same buffers, as is required by the GLX extension. We suspected that this wasn t all that necessary, and it turns out to have been broken several years ago. This is the traditional way in X to phase out undesirable code, and provides an excellent opportunity to revisit the original design. Doing buffer allocations within the client has several benefits:

• No longer need DRI2 additions to manage new GL buffers. Adding HiZ to the intel driver required new DRI2 code in the X server, even though X wasn t doing anything with those buffers at all.
• Eliminate some X round trips currently required for GL buffer allocation.
• Knowing what s in each buffer. Because the client allocates each buffer, it can track the contents of them.
• Size tracking is trivial. The application sends the GL the of the viewport, and the union of all viewports should be the same as the size of the window (or there will be undefined contents on the screen). The driver can use the viewport information to size the buffers and ensure that every frame on the screen is complete.

Present buffers through DMA-buf The new DMA-buf infrastructure provides a cross-driver/cross-process mechanism for sharing blobs of data. DMA-buf provides a way to take a chunk of memory used by one driver and pass it to another. It also allows applications to create file descriptors that reference these objects. For our purposes, it s the file descriptor which is immediately useful. This provides a reliable and secure way to pass a reference from an underlying graphics buffer from the client to the X server by sending the file descriptor over the local X socket. An additional benefit is that we get automatic integration of data from other devices in the system, like video decoders or non-primary GPUs. The Prime support added in DRI version 2.8 hacks around this by sticking a driver identifier in the driverType value. Once the buffer is available to the X server, we can create a request much like the current DRI2SwapBuffers request, except instead of implicitly naming the back and front buffers, we can pass an arbitrary buffer and have those contents copied or swapped to the drawable. We also need a way to copy a region into the drawable. I don t know if that needs the same level of swap control, but it seems like it would be nice. Perhaps the new SwapBuffers request could take a region and offset as well, copying data when swapping isn t possible. Managing buffer allocations One trivial way to use this new buffer allocation mechanism would be to have applications allocate a buffer, pass it to the X server and then simply drop their reference to it. The X server would keep a reference until the buffer was no longer in use, at which point the buffer memory would be reclaimed. However, this would eliminate a key optimization in current drivers the ability to re-use buffers instead of freeing and allocating new ones. Re-using buffers takes advantage of the work necessary to setup the buffer, including constructing page tables, allocating GPU memory space and flushing caches. Notifying the application of idle buffers Once the X server is finished using a buffer, it needs to notify the application so that the buffer can be re-used. We could send these notifications in X events, but that ends up in the twisty mess of X client event handling which has already caused so much pain with Invalidate events. The obvious alternative is to send them back in a reply. That nicely controls where the data are delivered, but causes the application to block waiting for the X server to send the reply. Fortunately, applications already want to block when swapping buffers so that they get throttled to the swap buffers rate. That is currently done by having them wait for the DRI2SwapBuffers reply. This provides a nice place to stick the idle buffer data. We can simply list buffers which have become idle since the last SwapBuffers reply was delivered. Releasing buffer memory Applications which update only infrequently end up with a back buffer allocated after their last frame which can t be freed by the system. The fix for this is to mark the buffer purgable, but that can only be done after all users of the buffer are finished with it. With this new buffer management model, the application effectively passes ownership of its buffers to the X server, and the X server knows when all use of the buffer are finished. It could mark buffers as purgable at that point. When the buffer was sent back in the SwapBuffers reply, the application would be able to ask the kernel to mark it un-purgable again. A new extension? Or just a new DRI2 version? If we eliminate the authentication model and replace the buffer allocation and presentation interfaces, what of the existing DRI2 protocol remains useful? The only remaining bits are the other synchronization requests: DRI2GetMSC, DRI2WaitMSC, DRI2WaitSBC and DRI2SwapInterval. Given this, does it make more sense to leave DRI2 as it is and plan on deprecating, and eventually eliminating, it? Doing so would place a support burden on existing applications, as they d need to have code to use the right extension for the common requests. They ll already need to support two separate buffer management versions though, so perhaps this burden isn t that onerous?

I just released PyGObject 3.3.92, for GNOME 3.5.92. There is nothing too exciting in this release; a couple of small bug fixes and a lot of new test cases. See the detailled list of changes below. Thanks to all contributors! Changes:

• release-news: Generate HTML changelog (Martin Pitt)
• [API add] Add ObjectInfo.get_abstract method (Simon Feltman) (#675581)
• Add deprecation warning when setting gpointers to anything other than int. (Simon Feltman) (#683599)
• test_properties: Test accessing a property from a superclass (Martin Pitt) (#684058)
• test_properties.py: Consistent test names (Martin Pitt)
• test_everything: Ensure TestSignals callback does get called (Martin Pitt)
• argument: Fix 64bit integer convertion from GValue (Nicolas Dufresne) (#683596)
• Add Simon Feltman as a project maintainer (Martin Pitt)
• test_signals.py: Drop global type variables (Martin Pitt)
• test_signals.py: Consistent test names (Martin Pitt)
• Add test cases for GValue signal arguments (Martin Pitt) (#683775)
• Add test for GValue signal return values (Martin Pitt) (#683596)
• Improve setting pointer fields/arguments to NULL using None (Simon Feltman) (#683150)
• Test gint64 C signal arguments and return values (Martin Pitt)
• Test in/out int64 GValue method arguments. (Martin Pitt) (#683596)
• Bump g-i dependency to 1.33.10 (Martin Pitt)
• Fix -uninstalled.pc.in file (Thibault Saunier) (#683379)

Over the past few days, I decided to purge Gnash and LightSpark from my laptop to see whether my Internet experience would be affected in any dramatic way. Amazingly, a number of sites seem to offer video content encoded in Theora (VP3) or WebM (VP8). Sadly, a handful of popular sites such as Vimeo insist upon using the H.264 (MPEG-4) CODEC, which cannot be safely supported on Free Software because of unclear licensing issues that might impose an ulterior usage fee onto the end-users.

The good news is that the Mozilla Foundation has decided to avoid the issue together: starting with the most recent release of the mobile version of Firefox, content decoding is offloaded to the platform's native media CODEC library. On most Free Software platforms, this means that Gstreamer will handle the content and, in turn, use gstreamer-plugins-bad to perform the decoding. However, Gstreamer support is still rather sketchy, as attested by this Debian bug report and thus disabled by default. This leaves me wondering how little is missing for this to properly work. Would either Canonical or Red Hat perhaps be interested in funding this?