Here is a slightly overdue status on Jessie.

• We are not ready to set a date on a Jessie release yet. Even if we were, it would be unlikely that said date would be in January. Accordingly, it is safe to assume that the rapid automatic removals clause of the freeze policy will be applied before our release. If your package (or a package you use) depend on any package in this list, then they are at risk.
• It is unclear to me whether our CD1s are able to contain all the necessary packages. This was a major issue for Wheezy. So far I have only done some minor prodding here. However, I really want this item done soon as it can easily take a month or more to resolve this.
• While the release notes have been improved quite a bit, I am certain that there are more cases we can cover. As an example, I am looking for input on #773386.
• Despite the declining number of RC bugs, we still have some particular unhealthy ones left. E.g. #759530. There is also a view of some of the known unfixed Jessie blockers.
• We have had a number of issues with APT and dpkg (e.g. trigger cycles, APT breaking under puppet) that caused upgrade failures or were a severe regression. Most of these have been resolved now. There are some trigger issues left and I have pushed for a fixed dpkg at the cost of possibly removing some packages. See #772866 (among others).

Stricter freeze policy per January 5th The next timed change of the freeze policy will apply per January 5th. After that date, we will only accept RC bugs fixes. Which means it is final chance for translation updates. More on RC bugs In absolute numbers, the RC bugs have declined quite well. We are below 150 now. We lost quite a bit of traction in December compared to November. However, November was an extremely efficient month. However, we still need the final push here. Debian installer release pending Yesterday, we received a list of packages that needed to be unblocked for d-i with a remark that a release of d-i might follow. Based on what we have unblocked previously, it will likely contain some (improved?) UEFI support. Pending Debian 7.8 release While not directly relevant to Jessie, we also got a pending Wheezy release planned for the 10th of January. The window for getting changes into the 7.8 release closes this weekend. Want to help?

• File bugs against release-notes (source at [RN source]) and installation-guide for missing or outdated documentation. Patches and drafts especially welcome.
• Fix RC bugs especially the known Jessie blockers.
• Test upgrade paths and installation media. Although in both cases, you may want to wait a bit (for the new dpkg to migrate and for the new debian-installer release respectively).
• Consider offering your help to teams such as the CD team or debian-installer team.

Thank you, [RN source]: https://anonscm.debian.org/viewvc/ddp/manuals/trunk/release-notes/ svn co https://anonscm.debian.org/viewvc/ddp/manuals/trunk/release-notes/ Git Repo: http://anonscm.debian.org/cgit/users/jcristau/release-notes.git/
Filed under: Debian, Release-Team

Here is a slightly overdue status on Jessie.

• We are not ready to set a date on a Jessie release yet. Even if we were, it would be unlikely that said date would be in January. Accordingly, it is safe to assume that the rapid automatic removals clause of the freeze policy will be applied before our release. If your package (or a package you use) depend on any package in this list, then they are at risk.
• It is unclear to me whether our CD1s are able to contain all the necessary packages. This was a major issue for Wheezy. So far I have only done some minor prodding here. However, I really want this item done soon as it can easily take a month or more to resolve this.
• While the release notes have been improved quite a bit, I am certain that there are more cases we can cover. As an example, I am looking for input on #773386.
• Despite the declining number of RC bugs, we still have some particular unhealthy ones left. E.g. #759530. There is also a view of some of the known unfixed Jessie blockers.
• We have had a number of issues with APT and dpkg (e.g. trigger cycles, APT breaking under puppet) that caused upgrade failures or were a severe regression. Most of these have been resolved now. There are some trigger issues left and I have pushed for a fixed dpkg at the cost of possibly removing some packages. See #772866 (among others).

Stricter freeze policy per January 5th The next timed change of the freeze policy will apply per January 5th. After that date, we will only accept RC bugs fixes. Which means it is final chance for translation updates. More on RC bugs In absolute numbers, the RC bugs have declined quite well. We are below 150 now. We lost quite a bit of traction in December compared to November. However, November was an extremely efficient month. However, we still need the final push here. Debian installer release pending Yesterday, we received a list of packages that needed to be unblocked for d-i with a remark that a release of d-i might follow. Based on what we have unblocked previously, it will likely contain some (improved?) UEFI support. Pending Debian 7.8 release While not directly relevant to Jessie, we also got a pending Wheezy release planned for the 10th of January. The window for getting changes into the 7.8 release closes this weekend. Want to help?

• File bugs against release-notes (source at [RN source]) and installation-guide for missing or outdated documentation. Patches and drafts especially welcome.
• Fix RC bugs especially the known Jessie blockers.
• Test upgrade paths and installation media. Although in both cases, you may want to wait a bit (for the new dpkg to migrate and for the new debian-installer release respectively).
• Consider offering your help to teams such as the CD team or debian-installer team.

Thank you, [RN source]: https://anonscm.debian.org/viewvc/ddp/manuals/trunk/release-notes/ svn co https://anonscm.debian.org/viewvc/ddp/manuals/trunk/release-notes/ Git Repo: http://anonscm.debian.org/cgit/users/jcristau/release-notes.git/
Filed under: Debian, Release-Team

For a recent customer setup of Debian/wheezy on a IBM x3630 M4 server we used my blog entry State of the art Debian/wheezy deployments with GRUB and LVM/SW-RAID/Crypto as a base. But this time we wanted to use (U)EFI instead of BIOS legacy boot. As usual we went for installing via Grml and grml-debootstrap. We started by dd-ing the Grml ISO to a USB stick ( dd grml64-full_2014.11.iso of=/dev/sdX bs=1M ). The IBM server couldn t boot from it though, as far as we could identify it seems to be related to a problem with the IBM server not being able to properly recognize USB sticks that are registering themselves as mass storage device instead of removable storage devices (you can check your device via the /sys/devices/ /removable setting). So we enabled Legacy Boot and USB Storage in the boot manager of the server to be able to boot Grml in BIOS/legacy mode from this specific USB stick. To install the GRUB boot loader in (U)EFI mode you need to be able to execute modprobe efivars . But our system was booted via BIOS/legacy and in that mode the modprobe efivars doesnt work. We could have used a different USB device for booting Grml in UEFI mode but because we are lazy sysadmins and wanted to save time we went for a different route instead: First of all we write the Grml 64bit ISO (which is (U)EFI capable out-of-the-box, also when dd-ing it) to the local RAID disk (being /dev/sdb in this example):

root@grml ~ # dd if=grml64-full_2014.11.iso of=/dev/sdb bs=1M

Now we should be able to boot in (U)EFI mode from the local RAID disk. To verify this before actually physically rebooting the system (and possibly getting into trouble) we can use qemu with OVMF:

root@grml ~ # apt-get update
root@grml ~ # apt-get install ovmf
root@grml ~ # qemu-system-x86_64 -bios /usr/share/qemu/OVMF.fd -hda /dev/sdb

The Grml boot splash comes up as expected, perfect. Now we actually reboot the live system and boot the ISO from the local disks in (U)EFI mode. Then we put the running Grml live system into RAM to not use and block the local disks any longer since we want to install Debian there. This can be achieved not just by the toram boot option, but also by executing grml2ram on-demand as needed from user space:

root@grml ~ # grml2ram

Now having the local disks available we verify that we re running in (U)EFI mode by executing:

root@grml ~ # modprobe efivars
root@grml ~ # echo $?
0

Great, so we can install the system in (U)EFI mode now. Starting with the according partitioning (/dev/sda being the local RAID disk here):

root@grml ~ # parted /dev/sda
GNU Parted 3.2
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel gpt
(parted) mkpart primary fat16 2048s 4095s
(parted) name 1 "EFI System"
(parted) mkpart primary 4096s 100%
(parted) name 2 "Linux LVM"
(parted) print
Model: IBM ServeRAID M5110 (scsi)
Disk /dev/sda: 9000GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:
  Number  Start   End     Size    File system  Name        Flags
   1      1049kB  2097kB  1049kB  fat16        EFI System
   2      2097kB  9000GB  9000GB               Linux LVM
(parted) quit
Information: You may need to update /etc/fstab.

Then setting up LVM with a logical volume for the root fs and installing Debian via grml-debootstrap on it:

root@grml ~ # pvcreate /dev/sda2
  Physical volume "/dev/sda2" successfully created
root@grml ~ # vgcreate vg0 /dev/sda2
  Volume group "vg0" successfully created
root@grml ~ # lvcreate -n rootfs -L16G vg0
  Logical volume "rootfs" created
root@grml ~ # grml-debootstrap --target /dev/mapper/vg0-rootfs --password secret --hostname foobar --release wheezy
[...]

Now finally set up the (U)EFI partition and properly install GRUB in (U)EFI mode:

root@grml ~ # mkfs.fat -F 16 /dev/sda1
mkfs.fat 3.0.26 (2014-03-07)
WARNING: Not enough clusters for a 16 bit FAT! The filesystem will be
misinterpreted as having a 12 bit FAT without mount option "fat=16".
root@grml ~ # mount /dev/mapper/vg0-rootfs /mnt
root@grml ~ # grml-chroot /mnt /bin/bash
Writing /etc/debian_chroot ...
(foobar)root@grml:/# mkdir -p /boot/efi
(foobar)root@grml:/# mount /dev/sda1 /boot/efi
(foobar)root@grml:/# apt-get install grub-efi-amd64
[...]
(foobar)root@grml:/# grub-install /dev/sda
Timeout: 10 seconds
BootOrder: 0003,0000,0001,0002,0004,0005
Boot0000* CD/DVD Rom
Boot0001* Hard Disk 0
Boot0002* PXE Network
Boot0004* USB Storage
Boot0005* Legacy Only
Boot0003* debian
Installation finished. No error reported.
(foobar)root@grml:/# ls /boot/efi/EFI/debian/
grubx64.efi
(foobar)root@grml:/# update-grub
[...]
(foobar)root@grml:/# exit
root@grml ~ # umount /mnt/boot/efi
root@grml ~ # umount /mnt/
root@grml ~ # vgchange -an
  0 logical volume(s) in volume group "vg0" now active

That s it. Now rebooting the system should bring you to your Debian installation running in (U)EFI mode. You can verify this before actually rebooting into the system by using the qemu/OVMF trick from above once again.

A bit of news on the development of OpenTAC the Open Hardware Test Automation Controller. I ve talked about this at the MiniDebConf 2014 in Cambridge. (Video available). The development is being tracked on the Vero-Apparatus wiki and as this is Open Hardware, the files are attached to the wiki (All files are CC BY-SA 4.0). Andy completed the schematics at the start of November, allowing work to start on the routing. With routing completed, orders were placed for manufacture of the first PCBs on the 19th December 2014. Whilst waiting for the PCBs to arrive we re working on the device tree database (my first real experience with creating a device tree) which will underpin the PDU and serial console services available to the user as well as the admin interface for control of the USB subsystems, fan control, power control and thermal monitoring. The first thing we need to do is create a data dictionary a table to correlate software identifiers with the real hardware pins. We ll then follow that through with a default device tree overlay that will leave all the associated I/O lines in a safe initial state. Once we have some code, I ll be pushing to a branch on GitHub. We ve also got an internal git repo on vero-apparatus for OpenTAC files which we will add to in due course. Image of the board as rendered prior to prototype production: More to come once we have the hardware on the desk

Review: Some Remarks, by Neal Stephenson

Publisher:	William Morrow
Copyright:	June 2013
ISBN:	0-06-202444-2
Format:	Trade paperback
Pages:	336

This is going to be another weird review, since I read this essay collection about three months ago, and I borrowed it from a friend. So this is both from fading memory and without a handy reference other than Internet searches. Apologies in advance for any important details that I miss. The advantage is that you'll see what parts of this collection stuck in my memory. Some Remarks is, as you might guess from the title, a rather random collection of material. There's one long essay that for me was the heart of the book (more on that in a moment), two other longer essays, two short stories, and thirteen other bits of miscellaneous writing of varying lengths. I found most of the short essays unremarkable. Stephenson uses a walking desk because sitting is bad for you that sentence contains basically all of the interesting content of one of the essays. I think it takes a large topic and some running room before Stephenson can get up to speed and produce something that's more satisfying than technological boosterism. That means the most interesting parts of this book are the three longer works. "In the Kingdom of Mao Bell" was previously published in Wired and is still available. Some Remarks contains only excerpts; Stephenson says that some of the original essay is no longer that interesting. I had mixed feelings about this one. Some of the sense of place he creates was fun to read, but Stephenson can't seem to quite believe that the Chinese don't care about "freedom" according to his definitions in the same way and therefore don't have the same political reaction to hacker culture that he does. This could have been an opportunity for him to question assumptions, but instead it's mostly an exercise in dubious, sweeping cultural evaluation, such as "the country has a long history of coming up with technologies before anyone else and then not doing a lot with them." A reminder that the detail with which Stephenson crams his writing is not always... true. Stronger is "Atoms of Cognition: Metaphysics in the Royal Society 1715 2010," which covers material familiar to readers of Stephenson's Baroque Cycle. The story of Newton, Leibniz, their rivalry, and the competing approaches to thinking about mathematics and science was my favorite part of that series, and in some ways the non-fiction treatment is better than the fictional one. If you liked the Baroque Cycle, this is worth reading. But the highlight of the book for me was "Mother Earth Mother Board." This is a long essay (50,000 words, practically a small book and the largest part of this collection) about the laying of undersea fiber-optic cables. Those who have read Cryptonomicon will recognize some of the themes here, but there's way more to this essay than there was to the bits about undersea cables in Cryptonomicon. It's mostly about technology, rather than people, which puts Stephenson on firmer ground. The bit about people reads more like a travelogue, full of Stephenson's whole-hearted admiration of people who build things and make things work. There's a bit of politics, a bit of history, a bit of tourism, and a lot of neat trivia about a part of the technological world that I'd not known much about before. I would say this is worth the price of the collection, but it too was previously published in Wired, so you can just read it online. Those reading this review on my web site will notice that I filed it in non-fiction. There are a couple of stories, but they're entirely forgettable (in fact, I had entirely forgotten them, and had to skim them again). But, for the record, here are short reviews of those: "Spew": This originally appeared in Wired and can still be read on-line. The protagonist takes a job as a sort of Internet marketing inspector who looks for deviations from expected profiles. While tracing down an anomaly, though, he finds another use of the Internet that's outside of the marketing framework he's using. It's unlikely that anyone who's been online for long will find much new in this story. Some of that is because it was originally published in 1994, but most of it is just that this isn't a very good story. Stephenson seems to have turned up his normal manic infodump to 11 to satisfy the early Wired aesthetic, and the result is a train wreck of jargon, bad slang, and superficial social observation. (3) "The Great Simoleon Caper": Originally published in TIME, this story too is still available online. It's primarily interesting because it's a story about Bitcoin (basically), written in 1995. And it's irritating for exactly the same reason that Bitcoin enthusiasm often tends to be irritating: the assumption that cryptocurrency is somehow a revolutionary attack on government-run currency systems. I'm not going to get into the ways in which this doesn't make sense given how money is used socially (David Graeber's Debt is the book to read if you want more information); just know that the story follows that path and doesn't engage with any of the social reasons why that outcome is highly unlikely. Indeed, the lengths to which the government tries to go to discredit cryptocurrency in this story are rather silly. Apart from that, this is typical early Stephenson writing. It's very in love with ideas, not as much with characterization, and consists mostly of people explaining things to each other. Sometimes this is fun, but when focused on topics about which considerably more information has become available, it doesn't age very well. (5) Overall, there was one great essay and a few interesting bits, but I wouldn't have felt I was missing much if I'd never read this collection. I borrowed Some Remarks from a friend, and I think that's about the right level of effort. If it falls into your hands, or you see it in a library, some of the essays, particularly "Mother Earth Mother Board," are worth reading, but given that the best parts are available on-line for free, I don't think it's worth a purchase. Rating: 6 out of 10

the last year hasn't been an easy one for debian. we've seen lots of fights, unproductive discussions, & in general behaviour which contributed to what enrico in his brilliant blog post called the "stink in the kitchen". people's feelings were hurt, some became less active, others resigned from a specific position or retired completely. it happened to me as well that reading through threads full of trolling, aggressiveness, finger-pointing, unrespectful or abusive behaviour, etc. made me frustrated, or sad, or angry, or all kinds of other negative feelings. but then I usually told myself: this is not the debian project as I know it; this is only a small part a part which urgently needs improvement! ; but if we only look at it alone, our picture of debian at large is distorted. the much bigger part of the debian life I know doesn't happen on this handful of high-profile mailing lists; it happens on dozens of specialized mailing lists, in many small IRC channels, in the BTS, & in in-person meetings. & what I see there is most of the time constructive, collaborative, respectful communication, & committed, helpful, funny, awesome people. the idea of this advent calendar was to give examples of some of my recent experiences which demonstrate what the bright side of debian is in my opinion & what contributes to my fun in debian. with the aim of sharing my impression that the proverbial "stink" is, if we take a step back, only one piece of the bigger picture, & that we shouldn't let ourselves get demotivated by only staring at this puzzle piece. thanks to all the awesome contributors for polishing existing & adding new shiny puzzle pieces to our common picture every day! finally: thanks for reading this series of posts, & especially thank you to all who provided positive feedback this in turn motivated me to follow through! & now: back to work, & don't forget to help cleaning the kitchen :)

---

this posting is part of GDAC (gregoa's debian advent calendar), a project to show the bright side of debian & why it's fun for me to contribute.

If you want to contribute to existing/upstream AppArmor profiles, you need to create an account on Canonical s launchpad. You ll also need to upload an SSH key to be able to push your changes. You will also need to install Canonical s version control system, called Bazaar:

apt-get install bzr

Go or create a repository where you want to checkout the changes:

mkdir apparmor-dev
cd apparmor-dev
bzr branch lp:apparmor-profiles master
ls master

Define your identity like this:

bzr whoami "My Name <my@email.com>"

Bazaar does not handle branches like Git does, unfortunately. It is a little bit weird in first place that you will need to create a directory where you would create a copy of the master branch. You would then work on this branch and later ask for a merge and pull into the master branch.

mkdir myname
bzr branch master myname/pidgin
ls myname/pidgin

Then, start modifying the profiles using a test editor and test them. Testing is done through dis/enabling the profile.
Once done, you can push the changes to your distant repository:

bzr push

Then, you need to connect to Launchpad s web interface. Go to your page, and look for the branch you just pushed. Click on change branch details and link the branch to apparmor-profiles. Then you will be able to do a merge request through the webinterface.

I have spent the last days trying to understand the relationship between AppArmor, the AppArmor profiles and Debian s AppArmor profiles. Let me try to briefly introduce you to this. First of all, let me emphasize that one needs to distinguish between AppArmor, the kernel module, and the AppArmor profiles, which define rules for application confinement. As stated on the project s wiki:

The AppArmor project source is split between the kernel module, available in the Linux kernel and git development tree and the user space tools available in launchpad.

So, we can find the upstream AppArmor profile development taking place at Canonical s launchpad. The profiles which are developed there serve as a basis for those included in Debian. It is thus useful to try to contribute to this upstream when updating a profile for Debian. Since AppArmor 2.9 has been introduced, the rules of a profiles which are not applicable for a system are ignored. Thus, collaboration between different Linux distributions shipping AppArmor will become much simpler. In the aforementioned Ubuntu repository you can find the profiles that are currently in development. Once tested and ready, they are removed from this repository and finally included into the corresponding package. For example, the profile for Evince, the GNOME PDF viewer, is available in the evince package. In Debian on the contrary, there are only some packages which ship their own profiles. This concernes for example bind, clamav, cups, tor.
But many profiles are not yet included with their package, but instead delivered through the apparmor-profiles and apparmor-profiles-extra packages. These are maintained by the the Debian AppArmor packaging team. The team takes care of merging changes from the upstream profiles into Debian. Today, the relationship between Upstream and Debian looks like this:

Upstream/bzr	Debian source package	Debian binary package	Ubuntu source package	Ubuntu binary package
apparmor	apparmor	apparmor-profiles	-	-
apparmor-profiles	apparmor-profiles-extra	apparmor-profiles-extra	-	-
ubuntu-evince	apparmor-profiles-extra	apparmor-profiles-extra	evince	evince

In the future, this should change. It would indeed be desirable that package maintainers include profiles with the packages they maintain and take care of updating them accordingly when the package itself is updated. It is much more logical to to it that way, as changes in a package like Pidgin or Evince can occur at a different point in time then when the apparmor-profiles-extra package is updated. I started working on a documentation for Debian Package Maintainers which describes how to do this.

It s been a couple of weeks since I ve launched Debian Code Search Instant, so people have had the chance to use it for a while and that gives me plenty of data points to look at :-). For every query, I log the search term itself as well as the duration the query took to execute. That way, I can easily identify queries that take a long time and see why that is. There is a class of queries for which Debian Code Search (DCS) doesn t perform so well, and that s queries that consist of trigrams which are extremely common. Whenever DCS receives such a query, it needs to search through a lot of files. Note that it doesn t really matter if there are plenty of results or not it s the number of files that potentially contain a result which matters. One such query is arse (we get a lot of curse words). It consists of only two trigrams ( ars and rse ), which are extremely common in program source code. As a couple of examples, the terms parse , sparse , charset and coarse are all matched by that. As an aside, if you really want to search for just arse , use word boundaries, i.e. \barse\b , which also makes the query significantly faster. Fixing the overloaded frontend When DCS first received the query, arse would lead to our frontend server crashing. That was due to (intentionally) unoptimized code we were aggregating all search results from all 6 source backends in memory, sorted them, and then wrote them out to disk. I addressed this in commit d2922fe92 with the following measures:

1. Instead of keeping the entire result in memory, just write the result to a temporary file on disk ( unsorted.json ) and store pointers into that file in memory, i.e. (offset, length) tuples. In order to sort the results, we also need to store the ranking and the path (to resolve ties and thereby guarantee a stable result order over multiple search queries). For grouping the results by source package, we need to keep the package name.
2. If you think about it, you don t need the entire path in order to break a tie the hash is enough, as it defines an ordering. That ordering may be different, but any ordering is good enough for the purpose of merely breaking a tie in a deterministic way. I m using Go s hash/fnv, the only non-cryptographic (fast!) hash function that is included in Go s standard library.
3. Since this was still leading to Out Of Memory errors, I decided to not store a copy of the package name in each search result, but rather use a pointer into a string pool containing all package names. The number of source package names is relatively small, in the order of 20,000, whereas the number of search results can be in the millions. Using the stringpool is a clear win the overhead in the case where #results < #srcpackages is negligible, but as soon as #results > #srcpackages, you save memory.

With all of that fixed, the query became at all possible, albeit with a runtime of around 20 minutes. Double Writing When running such a long-running query, I noticed that the query ran smooth for a while, but then it took multiple seconds without any visible progress at the end of the query before the results appeared. This was due to the frontend ranking the results and then converting unsorted.json into actual result pages. Since we provide results ordered by ranking, but also results grouped by source packages, it was writing every result twice to disk. What s even worse is that due to re-ordering, every read was essentially random (as opposed to sequential reads). What s even worse is that nobody will ever click through all the hundreds of thousands of result pages, so they are prepared entirely in vain. Therefore, with commit c744b236e I made the frontend generate these result pages on demand. This cut down the time for the ranking phase at the end of each query from 20-30 seconds (for big queries) to typically less than one second. Profiling/Monitoring After adding monitoring to each of the source-backends, I realized that during these long-running queries, the disk I/O and network I/O was nowhere near my expectations: each source-backend was sending only a low single-digit number of megabytes per second back to the frontend (typically somewhere between 1 MB/s and 3 MB/s). This didn t match up at all with the bandwidth I observed in earlier performance tests, so I used wget -O /dev/null to send a query and discard the result in order to get some theoretical performance numbers. Suddenly, I was getting more than 10 MB/s worth of results, maxing out the disks with a read rate of about 200 MB/s. So where is the bottleneck? I double-checked that neither the CPU on any of our VMs, nor the network between them was saturated. Note that as of this point, the CPU of the frontend was at roughly 70% (of one core), which didn t seem a lot to me. Then, I followed this excellent tutorial on profiling Go programs to see where the frontend is spending its time. Turns out, the biggest consumer of CPU time was the encoding/json Go package, which is used for deserializing results received from the backend and serializing them again before sending them to the client. Since I was curious about it for a while already, I decided to give cap n proto a try to replace JSON as serialization mechanism for communication between the source backends and the frontend. Switching to it (commit 8efd3b41) brought down the CPU load immensely, and made the query a bit faster. In addition, I killed the next biggest consumer: the lseek(2) syscall, which we used to call with SEEK_CUR and an offset of 0 so that it would tell us the current position. This was necessary in the first place because we don t know in advance how many bytes we re going to write when serializing a result to disk. The replacement is a neat little trick:

type countingWriter int64
func (c *countingWriter) Write(p []byte) (n int, err error)  
    *c += countingWriter(len(p))
    return len(p), nil
 
// [ ]
// Then, use an io.MultiWriter like this:
var written countingWriter
w := io.MultiWriter(s.tempFiles[backendidx], &written)
result.WriteJSON(w)

With some more profiling, the new bottleneck was suddenly the read(2) syscall, issued by the cap n proto deserialization, operating directly on the network connection buffer. strace revealed that crunching through the results of one source backend for a long query, read(2) was called about 250,000 times. By simply using a buffered reader (commit 684467ae), I could reduce that to about 2,000 times. Another bottleneck was the fact that for every processed result, the frontend needed to update the query state, which is shared amongst all goroutines (there is one goroutine for each source backend). All that parallelism isn t very effective if you need to synchronize the state updates in the end. So with commit 5d46a572, I refactored the state to be per-backend, so that locking is only necessary for the first couple of results, and the vast vast majority of results can be processed entirely without locking. This brought down the query time from 20 minutes to about 5 minutes, but I still wasn t happy with the bandwidth: the frontend was doing a bit over 10 MB/s of reads from all source backends combined, whereas with wget I could get around 40 MB/s with the same query. At this point, the CPU utilization was around 7% of one core on the frontend, and profiling didn t immediately reveal an obvious culprit. After a bit of experimenting (by commenting out code here and there ;-)), I figured out that the problem was that the frontend was still converting these results from capnproto buffers to JSON. While that doesn t take a lot of CPU time, it delays the network stream from the source-backend: once the local and remote TCP buffers are full, the source-backend will (intentionally!) not continue with its search, so that it doesn t run out of memory. I m still convinced that s a good idea, and in fact I was able to solve the problem in an elegant way: instead of writing JSON to disk and generating result pages on demand, we now write capnproto directly to disk (commit 466b7f3e) and convert it to JSON only before sending out the result pages. That decreases the overall CPU time since we only need to convert a small fraction of the results to JSON, but most importantly, the frontend is now not in the critical path anymore. It can directly pass the data through, and in fact it uses an io.TeeReader to do exactly that. Conclusion With all of these optimizations, we re now down to about 2.5 minutes for the search query arse , and the architecture of the system actually got simpler to reason about. Most importantly, though, the optimizations don t only play out for a single query, but for many different queries. I ve deployed the optimized version at the 15th of December 2014, and you can see that the 99th, 95th and 90th percentile latency dropped significantly, i.e. there are a lot fewer spikes than before, and more queries are processed faster, which is particularly obvious in the third graph (which is capped at 2 minutes):

A few weeks ago I went to see Blade Runner: The Final Cut in a one-off showing at the Tyneside Cinema. I've only watched the Final Cut once and that viewing was a bit compromised, so it was nice to see it properly, and on a massive screen with decent surround sound too. Whilst watching it I saw something that I thought might potentially have been a visual reference to Scott's earlier movie, Alien. When Deckard is climbing onto the roof of the Bradbury building, there's a decorative motif that to me, looks very Giger-esque, biomechanical, a bit like a chest-burster. I went back and took a screenshot from the Blu Ray at the same point. What do you think? As it happens, whilst the inner regions of the building in the movie are the Bradbury building (or at least the entrance hall), I don't think the upper exterior shots are.

---

There are a few other, well-known references to Alien in Blade Runner, although they are likely simply re-used effects rather than explicitly easter eggs. The display screen in Gaff's Spinner is re-used from the Narcissus (here's a comparison) and the ambience in Deckard's apartment also featured in the Nostromo's medical bay. Some enterprising person has put together a 12 hour session of that particular sound effect looping. There's a number of other comparisons and spots at this propsummit.com thread.

today I got two private mails from fellow DDs; both were personal messages & a pleasure to read. but what I also realized is that both emails were encrypted. which reminded me how hard it is to communicate securely with most people, & how easy it is with debian people. & this is something we can be proud of!

---

this posting is part of GDAC (gregoa's debian advent calendar), a project to show the bright side of debian & why it's fun for me to contribute.

Review: 2014 Hugos: Novelettes, edited by Loncon 3

Publisher:	Loncon 3
Copyright:	2014
Format:	Kindle

This is another weird "book review" covering the Hugo-nominated novelettes for the 2014 Hugos (given for works published in 2014) at Loncon 3, the 2014 Worldcon. The "editor" is the pool of attendees and supporting members who chose to nominate works, all of which had been previously edited by other editors in their original publication. I received all of these as part of the Hugo voter's packet for being a supporting member, but they all appear to be available for free on the Internet (at least at the time of this writing). "The Exchange Officers" by Brad Torgersen: An okay, if not particularly ground-breaking, military SF story, ruined for me by the ham-handed introduction of superficial jingoism. The protagonists are doing a tour as remote operators of humanoid battle suits in orbit: not a new premise, but a servicable one. Since this is military SF, they predictably have to defend a space installation against attackers. So we get a bit of drama, a bit of zero-g combat, and the fun of people learning how to remotely operate suits. You've probably read this before, but it passes the time reasonably well. Unfortunately, Torgersen decided to make the villains the Chinese military for no adequately-explained reason. (Well, I'm being kind; I suspect the reason is the standard yellow peril nonsense, but that's less generous.) So there is snide commentary about how only the military understand the Chinese threat and a fair bit of old-fashioned jingoism mixed in to the story, to its detriment. If you like this sort of thing, it's a typical example, although it escapes me why people thought it was exceptional enough to warrant nomination. (5) "The Lady Astronaut of Mars" by Mary Robinette Kowal: Once again, my clear favorite among the stories also won, which is a lovely pattern. Elma was the female astronaut in an alternate history in which manned space exploration continued to grow, leading to permanent settlement on Mars. She spent lots of time being photographed, being the smiling face of the space program, while her husband worked on the math and engineering of the launches. Now, she's an old woman, taking care of her failing and frail husband, her career behind her. Or so she thinks, before an offer that forces an impossible choice between space and staying with her husband for his final days. This is indeed the tear-jerker that it sounds like, but it's not as maudlin as it might sound. Kowal does an excellent job with Elma's characterization: she's no-nonsense, old enough to be confident in her opinions, and knows how to navigate through the world. The story is mixed with nostalgia and memories, including a reminder of just what Elma meant to others. It touches on heroism, symbolism, and the horrible choices around dying loved ones, but I thought it did so deftly and with grace. I was expecting the story to be too obvious, but I found I enjoyed the quotidian feel. It's not a story to read if you want to be surprised, but I loved the small touches. (9) "Opera Vita Aeterna" by Vox Day: Before the review, a note that I consider obligatory. The author of this story is an aggressively misogynistic white supremacist, well-known online for referring to black people as savages and arguing women should not be allowed to vote. To what extent you choose to take that into account when judging his fiction is up to you, but I don't think it should go unsaid. "Opera Vita Aeterna" is the story of a monastery in a typical fantasy world (at least as far as one can tell from this story; readers of Vox Day's fantasy series will probably know more background). At the start of the story, it gets an unexpected visit from an elf. Not just any elf, either, but one of the most powerful magicians of his society. He comes to the monastery out of curiousity about the god that the monks worship and stays for a project of illuminating their scriptures, while having theological debates with the abbot. This story is certainly not the offensive tirade that you might expect from its author. Its biggest problem is that nothing of substance happens in the story, either theologically or via more conventional action. It's a lot of description, a lot of talking, a lot of warmed-over Christian apologetics that dodges most of the hard problems, and a lot of assertions that the elf finds something of interest in this monastery. I can believe this could be the case, but Vox Day doesn't really show why. There is, at the end of the story, some actual drama, but I found it disappointing and pointless. It leads nowhere. The theology has the same problem: elves supposedly have no souls, which is should be the heart of a theological question or conflict Vox Day is constructing, but that conflict dies without any resolution. We know nothing more about the theology of this world at the end of the story than we do at the beginning. Some of the descriptions here aren't bad, and the atmosphere seems to want to develop into a story. But that development never happens, leaving the whole work feeling fundamentally pointless. (4) "The Truth of Fact, the Truth of Feelng" by Ted Chiang: This is another oddly-constructed story, although I think a bit more successful. It's a story in two interwoven parts. One is a fictional essay, told in a non-fiction style, about a man living in a future world with ubiquitous life recording and very efficient search software. Any part of one's life can be easily found and reviewed. The other is the story of a boy from a tribal culture during European colonialism. He learns to read and write, and from that a respect for written records, which come into conflict with the stories that the tribe elders tell about the past. The purpose of both of these stories is to question both the value and the implications of recording everything in a way that preserves and guarantees the facts instead of individual interpretations. The boy's story calls this into question; the narrator's story offers ambiguous support for its value and a deeper plea for giving people space to change. I found the style a bit difficult to get used to, since much of it did not feel like a story. But it grew on me as I read it, and the questions Chiang raises have stuck with me since. The problem of how and when to allow for change in others when we have perfect (or at least vastly improved) memory is both important and complicated, and this is one of the better presentations of the problem that I've seen. It's more of a think-y piece, and closer to non-fiction than a story, but I thought it was worth reading. (8) "The Waiting Stars" by Aliette de Bodard: I keep wanting to like de Bodard's space opera world of AIs and living ships, but it never quite works for me. I've read several stories set in this universe now, and it has some neat ideas, but I always struggle with the characters. This story at least doesn't have quite as much gruesome pregnancy as the previous ones (although there's still some). "The Waiting Stars" opens with a raid on a ship graveyard, an attempt to rescue and "reboot" an AI starship under the guidance of another Mind. This is intermixed with another story about a woman who was apparently rescued in childhood from birthing ship Minds and raised in a sort of foster institution. This feels like a flashback at first, but its interaction with the rest of the story is something more complicated. The conceptual trick de Bodard pulls here is thought-provoking, but once again I struggled to care about all of the characters. I also found the ending discouraging and unsatisfying, which didn't help. Someone who isn't me might really like this story, but it wasn't my thing. (6) Rating: 6 out of 10

Something happened a little while ago which we did not have time to commensurate properly. Our Rocker image for R is now the official R image for Docker itself. So getting R (via Docker) is now as simple as saying docker pull r-base. This particular container is essentially just the standard r-base Debian package for R (which is one of a few I maintain there) plus a mininal set of extras. This r-base forms the basis of our other containers as e.g. the rather popular r-studio container wrapping the excellent RStudio Server. A lot of work went into this. Carl and I also got a tremendous amount of help from the good folks at Docker. Details are as always at the Rocker repo at GitHub. Docker itself continues to make great strides, and it has been great fun help to help along. With this post I achieved another goal: blog about Docker with an image not containing shipping containers. Just kidding.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

I've owned a Logitech Wingman Gamepad Extreme since pretty much forever, and although it's been battered over the years, it's still mostly functional. As a gamepad, it has 10 buttons. What's special about it, though, is that the device also has a mode in which a gravity sensor kicks in and produces two extra axes, allowing me to pretend I'm really talking to a joystick. It looks a bit weird though, since you end up playing your games by wobbling the gamepad around a bit. About 10 years ago, I first learned how to write GObjects by writing a GObject-based joystick API. Unfortunately, I lost the code at some point due to an overzealous rm -rf call. I had planned to rewrite it, but that never really happened. About a year back, I needed to write a user interface for a customer where a joystick would be a major part of the interaction. The code there was written in Qt, so I write an event-based joystick API in Qt. As it happened, I also noticed that jstest would output names for the actual buttons and axes; I had never noticed this, because due to my 10 buttons and 4 axes, which by default produce a lot of output, the jstest program would just scroll the names off my screen whenever I plugged it in. But the names are there, and it's not too difficult. Refreshing my memory on the joystick API made me remember how much fun it is, and I wrote the beginnings of what I (at the time) called "libgjs", for "Gobject JoyStick". I didn't really finish it though, until today. I did notice in the mean time that someone else released GObject bindings for javascript and also called that gjs, so in the interest of avoiding confusion I decided to rename my library to libjoy. Not only will this allow me all kinds of interesting puns like "today I am releasing more joy", it also makes for a more compact API (compare joy_stick_open() against gjs_joystick_open()). The library also comes with a libjoy-gtk that creates a GtkListStore* which is automatically updated as joysticks are added and removed to the system; and a joytest program, a graphical joystick test program which also serves as an example of how to use the API. still TODO:

• Clean up the API a bit. There's a bit too much use of GError in there.
• Improve the UI. I suck at interface design. Patches are welcome.
• Differentiate between JS_EVENT_INIT kernel-level events, and normal events.
• Improve the documentation to the extent that gtk-doc (and, thus, GObject-Introspection) will work.

What's there is functional, though. Update: if you're going to talk about code, it's usually a good idea to link to said code. Thanks, Emanuele, for pointing that out

How cool would it be to fix that one bug you just found without having to download a source package? and without leaving your browser?

Inspired by github's online code editing, during Debconf 14 I worked on integrating an online editor on debsources (the software behind sources.debian.net). Long story short: it is available today, for users of chromium (or anything supporting chrome extensions).

After installing the editor for sources.debian.net extension, go straight to sources.debian.net and enjoy!

Go from simple debsources:


To debsources on steroids:


All in all, it brings:

• Online editing of all of Debian
• In-browser patch generation, available for download
• Downloading the modified file
• Sending the patch to the BTS
• Syntax highlighting for over 120 file formats!
• More hidden gems from Ace editor that can be integrated thanks to patches from you

Clone it or fork it:

git clone https://github.com/rgeissert/ace-sourced.n.git

For example, head to apt's source code, find a typo and correct it online: open apt.cc, click on edit, make the changes, click on email patch. Yes! it can generate a mail template for sending the patch to the BTS: just add a nice message and your patch is ready to be sent.

Didn't find any typo to fix? how sad, head to codesearch and search Debian for a spelling mistake, click on any result, edit, correct, email! you will have contributed to Debian in less than 5 minutes without leaving your browser.

The editor was meant to be integrated into debsources itself, without the need of a browser extension. This is expected to be done when the requirements imposed by debsources maintainers are sorted out.

Kudos to Harlan Lieberman who helped debug some performance issues in the early implementations of the integration and for working on the packaging of the Ace editor.

For the 6th year in a row, Igalia has organized a hackfest focused on web engines. The 5 years before this one were actually focused on the GTK+ port of WebKit, but the number of web engines that matter to us as Free Software developers and consultancies has grown, and so has the scope of the hackfest. It was a very productive and exciting event. It has already been covered by Manuel Rego, Philippe Normand, Sebastian Dr ge and Andy Wingo! I am sure more blog posts will pop up. We had Martin Robinson telling us about the new Servo engine that Mozilla has been developing as a proof of concept for both Rust as a language for building big, complex products and for doing layout in parallel. Andy gave us a very good summary of where JS engines are in terms of performance and features. We had talks about CSS grid layouts, TyGL a GL-powered implementation of the 2D painting backend in WebKit, the new Wayland port, announced by Zan Dobersek, and a lot more. With help from my colleague ChangSeok OH, I presented a description of how a team at Collabora led by Marco Barisione made the combination of WebKitGTK+ and GNOME s web browser a pretty good experience for the Raspberry Pi. It took a not so small amount of both pragmatic limitations and hacks to get to a multi-tab browser that can play youtube videos and be quite responsive, but we were very happy with how well WebKitGTK+ worked as a base for that. One of my main goals for the hackfest was to help drive features that were lingering in the bug tracker for WebKitGTK+. I picked up a patch that had gone through a number of iterations and rewrites: the HTML5 notifications support, and with help from Carlos Garcia, managed to finish it and land it at the last day of the hackfest! It provides new signals that can be used to authorize notifications, show and close them. To make notifications work in the best case scenario, the only thing that the API user needs to do is handle the permission request, since we provide a default implementation for the show and close signals that uses libnotify if it is available when building WebKitGTK+. Originally our intention was to use GNotification for the default implementation of those signals in WebKitGTK+, but it turned out to be a pain to use for our purposes. GNotification is tied to GApplication. This allows for some interesting features, like notifications being persistent and able to reactivate the application, but those make no sense in our current use case, although that may change once service workers become a thing. It can also be a bit problematic given we are a library and thus have no GApplication of our own. That was easily overcome by using the default GApplication of the process for notifications, though. The show stopper for us using GNotification was the way GNOME Shell currently deals with notifications sent using this mechanism. It will look for a .desktop file named after the application ID used to initialize the GApplication instance and reject the notification if it cannot find that. Besides making this a pain to test our test browser would need a .desktop file to be installed, that would not work for our main API user! The application ID used for all Web instances is org.gnome.Epiphany at the moment, and that is not the same as any of the desktop files used either by the main browser or by the web apps created with it. For the future we will probably move Epiphany towards this new era, and all users of the WebKitGTK+ API as well, but the strictness of GNOME Shell would hurt the usefulness of our default implementation right now, so we decided to stick to libnotify for the time being. Other than that, I managed to review a bunch of patches during the hackfest, and took part in many interesting discussions regarding the next steps for GNOME Web and the GTK+ and Wayland ports of WebKit, such as the potential introduction of a threaded compositor, which is pretty exciting. We also tried to have Bastien Nocera as a guest participant for one of our sessions, but it turns out that requires more than a notebook on top of a bench hooked up to a TV to work well. We could think of something next time ;D. I d like to thank Igalia for organizing and sponsoring the event, Collabora for sponsoring and sending ChangSeok and myself over to Spain from far away Brazil and South Korea, and Adobe for also sponsoring the event! Hope to see you all next year!

tl;dr: Providing support for all 3 init systems (sysv-rc, Upstart and systemd) isn t hard, and generating the init scripts / Upstart job / systemd using a template system is a lot easier than I previously thought. As always, when writing this kind of blog post, I do expect that others will not like what I did. But that s the point: give me your opinion in a constructive way (please be polite even if you don t like what you see I had too many times had to read harsh comments), and I ll implement your ideas if I find it nice. History of the implementation: how we came to the idea I had no plan to do this. I don t believe what I wrote can be generalized to all of the Debian archive. It s just that I started doing things, and it made sense when I did it. Let me explain how it happened. Since it s clear that many, and especially the most advanced one, may have an opinion about which init system they prefer, and because I also support Ubuntu (at least Trusty), I though it was a good idea to support all the main init system: sysv-rc, Upstart and systemd. Though I have counted (for the sake of being exact in this blog) : OpenStack in Debian contains currently 64 init scripts to run daemons in total. That s quite a lot. A way too much to just write them, all by hand. Though that s what I was doing for the last years until this the end of this last summer! So, doing all by hand, I first started implementing Upstart. Its support was there only when building in Ubuntu (which isn t the correct thing to do, this is now fixed, read further ). Then we thought about adding support for systemd. Gustavo Panizzo, one of the contributors in the OpenStack packages, started implementing it in Keystone (the auth server for OpenStack) for the Juno release which was released this October. He did that last summer, early enough so we didn t expect anyone to use the Juno branch Keystone. After some experiments, we had kind of working. What he did was invoking /etc/init.d/keystone start-systemd , which was still using start-stop-daemon. Yes, that s not perfect, and it s better to use systemd foreground process handling, but at least, we had a unique place where to write the startup scripts, where we check the /etc/default for the logging configuration, configure the log file, and so on. Then around in october, I took a step backward to see the whole picture with sysv-rc scripts, and saw the mess, with all the tiny, small difference between them. It became clear that I had to do something to make sure they were all the same, with the support for the same things (like which log system to use, where to store the PID, create /var/lib/project, /var/run/project and so on ). Last, on this month of December, I was able to fix the remaining issues for systemd support, thanks to the awesome contribution of Mikael Cluseau on the Alioth OpenStack packaging list. Now, the systemd unit file is still invoking the init script, but it s not using start-stop-daemon anymore, no PID file involved, and daemons are used as systemd foreground processes. Finally, daemons service files are also activated on installation (they were not previously). Implementation So I took the simplistic approach to use always the same template for the sysv-rc switch/case, and the start and stop functions, happening it at the end of all debian/*.init.in scripts. I started to try to reduce the number of variables, and I was surprised of the result: only a very small part of the init scripts need to change from daemon to daemon. For example, for nova-api, here s the init script (LSB header stripped-out):

DESC="OpenStack Compute API"
PROJECT_NAME=nova
NAME=$ PROJECT_NAME -api

That is it: only 3 lines, defining only the name of the daemon, the name of the project it attaches (eg: nova, cinder, etc.), and a long description. There s of course much more complicated init scripts (see the one for neutron-server in the Debian archive for example), but the vast majority only needs the above. Here s the sysv-rc init script template that I currently use:

#!/bin/sh
# The content after this line comes from openstack-pkg-tools
# and has been automatically added to a .init.in script, which
# contains only the descriptive part for the daemon. Everything
# else is standardized as a single unique script.
# Author: Thomas Goirand <zigo@debian.org>
# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin
if [ -z "$ DAEMON " ] ; then
	DAEMON=/usr/bin/$ NAME 
fi
PIDFILE=/var/run/$ PROJECT_NAME /$ NAME .pid
if [ -z "$ SCRIPTNAME " ] ; then
	SCRIPTNAME=/etc/init.d/$ NAME 
fi
if [ -z "$ SYSTEM_USER " ] ; then
	SYSTEM_USER=$ PROJECT_NAME 
fi
if [ -z "$ SYSTEM_USER " ] ; then
	SYSTEM_GROUP=$ PROJECT_NAME 
fi
if [ "$ SYSTEM_USER " != "root" ] ; then
	STARTDAEMON_CHUID="--chuid $ SYSTEM_USER :$ SYSTEM_GROUP "
fi
if [ -z "$ CONFIG_FILE " ] ; then
	CONFIG_FILE=/etc/$ PROJECT_NAME /$ PROJECT_NAME .conf
fi
LOGFILE=/var/log/$ PROJECT_NAME /$ NAME .log
if [ -z "$ NO_OPENSTACK_CONFIG_FILE_DAEMON_ARG " ] ; then
	DAEMON_ARGS="$ DAEMON_ARGS  --config-file=$ CONFIG_FILE "
fi
# Exit if the package is not installed
[ -x $DAEMON ]   exit 0
# If ran as root, create /var/lock/X, /var/run/X, /var/lib/X and /var/log/X as needed
if [ "x$USER" = "xroot" ] ; then
	for i in lock run log lib ; do
		mkdir -p /var/$i/$ PROJECT_NAME 
		chown $ SYSTEM_USER  /var/$i/$ PROJECT_NAME 
	done
fi
# This defines init_is_upstart which we use later on (+ more...)
. /lib/lsb/init-functions
# Manage log options: logfile and/or syslog, depending on user's choosing
[ -r /etc/default/openstack ] && . /etc/default/openstack
[ -r /etc/default/$NAME ] && . /etc/default/$NAME
[ "x$USE_SYSLOG" = "xyes" ] && DAEMON_ARGS="$DAEMON_ARGS --use-syslog"
[ "x$USE_LOGFILE" != "xno" ] && DAEMON_ARGS="$DAEMON_ARGS --log-file=$LOGFILE"
do_start()  
	start-stop-daemon --start --quiet --background $ STARTDAEMON_CHUID  --make-pidfile --pidfile $ PIDFILE  --chdir /var/lib/$ PROJECT_NAME  --startas $DAEMON \
			--test > /dev/null   return 1
	start-stop-daemon --start --quiet --background $ STARTDAEMON_CHUID  --make-pidfile --pidfile $ PIDFILE  --chdir /var/lib/$ PROJECT_NAME  --startas $DAEMON \
			-- $DAEMON_ARGS   return 2
 
do_stop()  
	start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE
	RETVAL=$?
	rm -f $PIDFILE
	return "$RETVAL"
 
do_systemd_start()  
	exec $DAEMON $DAEMON_ARGS
 
case "$1" in
start)
	init_is_upstart > /dev/null 2>&1 && exit 1
	log_daemon_msg "Starting $DESC" "$NAME"
	do_start
	case $? in
		0 1) log_end_msg 0 ;;
		2) log_end_msg 1 ;;
	esac
;;
stop)
	init_is_upstart > /dev/null 2>&1 && exit 0
	log_daemon_msg "Stopping $DESC" "$NAME"
	do_stop
	case $? in
		0 1) log_end_msg 0 ;;
		2) log_end_msg 1 ;;
	esac
;;
status)
	status_of_proc "$DAEMON" "$NAME" && exit 0   exit $?
;;
systemd-start)
	do_systemd_start
;;  
restart force-reload)
	init_is_upstart > /dev/null 2>&1 && exit 1
	log_daemon_msg "Restarting $DESC" "$NAME"
	do_stop
	case $? in
	0 1)
		do_start
		case $? in
			0) log_end_msg 0 ;;
			1) log_end_msg 1 ;; # Old process is still running
			*) log_end_msg 1 ;; # Failed to start
		esac
	;;
	*) log_end_msg 1 ;; # Failed to stop
	esac
;;
*)
	echo "Usage: $SCRIPTNAME  start stop status restart force-reload systemd-start " >&2
	exit 3
;;
esac
exit 0

Nothing particularly fancy here You ll noticed that it s really OpenStack centric (see the LOGFILE and CONFIGFILE things ). You may have also noticed the call to init_is_upstart which is needed for upstart support. I m not sure if it s at the correct place in the init script. Should I put that on top of the script? Was I right with the exit values for it? Please send me your comments Then I thought about generalizing all of this. Because not only the sysv-rc scripts needed to be squared-up, but also Upstart. The approach here was to source the sysv-rc script in debian/*.init.in, and then generate the Upstart job accordingly, using the above 3 variables (or more as needed). Here, the fun is that, instead of taking the approach of calculating everything at runtime with the sysv-rc, for Upstart jobs, many things are calculated at build time. For each debian/*.init.in script that the debian/rules finds, pkgos-gen-upstart-job is called. Here s pkgos-gen-upstart-job:

#!/bin/sh
INIT_TEMPLATE=$ 1 
UPSTART_FILE= echo $ INIT_TEMPLATE    sed 's/.init.in/.upstart/' 
# Get the variables defined in the init template
. $ INIT_TEMPLATE 
## Find out what should go in After=
#SHOULD_START= cat $ INIT_TEMPLATE    grep "# Should-Start:"   sed 's/# Should-Start://' 
#
#if [ -n "$ SHOULD_START " ] ; then
#	AFTER="After="
#	for i in $ SHOULD_START  ; do
#		AFTER="$ AFTER $ i .service "
#	done
#fi
if [ -z "$ DAEMON " ] ; then
        DAEMON=/usr/bin/$ NAME 
fi
PIDFILE=/var/run/$ PROJECT_NAME /$ NAME .pid
if [ -z "$ SCRIPTNAME " ] ; then
	SCRIPTNAME=/etc/init.d/$ NAME 
fi
if [ -z "$ SYSTEM_USER " ] ; then
	SYSTEM_USER=$ PROJECT_NAME 
fi
if [ -z "$ SYSTEM_GROUP " ] ; then
	SYSTEM_GROUP=$ PROJECT_NAME 
fi
if [ "$ SYSTEM_USER " != "root" ] ; then
	STARTDAEMON_CHUID="--chuid $ SYSTEM_USER :$ SYSTEM_GROUP "
fi
if [ -z "$ CONFIG_FILE " ] ; then
	CONFIG_FILE=/etc/$ PROJECT_NAME /$ PROJECT_NAME .conf
fi
LOGFILE=/var/log/$ PROJECT_NAME /$ NAME .log
DAEMON_ARGS="$ DAEMON_ARGS  --config-file=$ CONFIG_FILE "
echo "description \"$ DESC \"
author \"Thomas Goirand <zigo@debian.org>\"
start on runlevel [2345]
stop on runlevel [!2345]
chdir /var/run
pre-start script
	for i in lock run log lib ; do
		mkdir -p /var/\$i/$ PROJECT_NAME 
		chown $ SYSTEM_USER  /var/\$i/$ PROJECT_NAME 
	done
end script
script
	[ -x \"$ DAEMON \" ]   exit 0
	DAEMON_ARGS=\"$ DAEMON_ARGS \"
	[ -r /etc/default/openstack ] && . /etc/default/openstack
	[ -r /etc/default/\$UPSTART_JOB ] && . /etc/default/\$UPSTART_JOB
	[ \"x\$USE_SYSLOG\" = \"xyes\" ] && DAEMON_ARGS=\"\$DAEMON_ARGS --use-syslog\"
	[ \"x\$USE_LOGFILE\" != \"xno\" ] && DAEMON_ARGS=\"\$DAEMON_ARGS --log-file=$ LOGFILE \"
	exec start-stop-daemon --start --chdir /var/lib/$ PROJECT_NAME  \\
		$ STARTDAEMON_CHUID  --make-pidfile --pidfile $ PIDFILE  \\
		--exec $ DAEMON  -- --config-file=$ CONFIG_FILE  \$ DAEMON_ARGS 
end script
" >$ UPSTART_FILE 

The only thing which I don t know how to do, is how to implement the Should-Start / Should-Stop in an Upstart job. Can anyone shoot me a mail and tell me the solution? Then, I wanted to add support for systemd. Here, we cheated, since we only just called the sysv-rc script from the systemd unit, however, the systemd-start target uses exec, so the process stays in the foreground. It s also much smaller than the Upstart thing. However, here, I could implement the After thing, corresponding to the Should-Start:

#!/bin/sh
INIT_TEMPLATE=$ 1 
SERVICE_FILE= echo $ INIT_TEMPLATE    sed 's/.init.in/.service/' 
# Get the variables defined in the init template
. $ INIT_TEMPLATE 
if [ -z "$ SCRIPTNAME " ] ; then
	SCRIPTNAME=/etc/init.d/$ NAME 
fi
if [ -z "$ SYSTEM_USER " ] ; then
	SYSTEM_USER=$ PROJECT_NAME 
fi
if [ -z "$ SYSTEM_GROUP " ] ; then
	SYSTEM_GROUP=$ PROJECT_NAME 
fi
# Find out what should go in After=
SHOULD_START= cat $ INIT_TEMPLATE    grep "# Should-Start:"   sed 's/# Should-Start://' 
if [ -n "$ SHOULD_START " ] ; then
	AFTER="After="
	for i in $ SHOULD_START  ; do
		AFTER="$ AFTER $ i .service "
	done
fi
echo "[Unit]
Description=$ DESC 
$AFTER
[Service]
User=$ SYSTEM_USER 
Group=$ SYSTEM_GROUP 
WorkingDirectory=/var/lib/$ PROJECT_NAME 
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p /var/lock/$ PROJECT_NAME  /var/log/$ PROJECT_NAME  /var/lib/$ PROJECT_NAME 
ExecStartPre=/bin/chown $ SYSTEM_USER :$ SYSTEM_GROUP  /var/lock/$ PROJECT_NAME  /var/log/$ PROJECT_NAME  /var/lib/$ PROJECT_NAME 
ExecStart=$ SCRIPTNAME  systemd-start
Restart=on-failure
[Install]
WantedBy=multi-user.target
" >$ SERVICE_FILE 

As you can see, it s calling /etc/init.d/$ SCRIPTNAME sytemd-start, which isn t great. I d be happy to have comments from systemd user / maintainers on how to fix it to make it better. Integrating in debian/rules To integrate with the Debian package build system, we only need had to write this:

override_dh_installinit:
	# Create the init scripts from the template
	for i in  ls -1 debian/*.init.in  ; do \
		MYINIT= echo $$i   sed s/.init.in//  ; \
		cp $$i $$MYINIT.init ; \
		cat /usr/share/openstack-pkg-tools/init-script-template >>$$MYINIT.init ; \
		pkgos-gen-systemd-unit $$i ; \
	done
	# If there's an upstart.in file, use that one instead of the generated one
	for i in  ls -1 debian/*.upstart.in  ; do \
		MYPKG= echo $$i   sed s/.upstart.in//  ; \
		cp $$MYPKG.upstart.in $$MYPKG.upstart ; \
	done
	# Generate the upstart job if there's no already existing .upstart.in
	for i in  ls debian/*.init.in  ; do \
		MYINIT= echo $$i   sed s/.init.in/.upstart.in/  ; \
		if ! [ -e $$MYINIT ] ; then \
			pkgos-gen-upstart-job $$i ; \
		fi \
	done
	dh_installinit --error-handler=true
	# Generate the systemd unit file
	# Note: because dh_systemd_enable is called by the
	# dh sequencer *before* dh_installinit, we have
	# to process it manually.
	for i in  ls debian/*.init.in  ; do \
		pkgos-gen-systemd-unit $$i ; \
		MYSERVICE= echo $$i   sed 's/debian\///'  ; \
		MYSERVICE= echo $$MYSERVICE   sed 's/.init.in/.service/'  ; \
		dh_systemd_enable $$MYSERVICE ; \
	done

As you can see, it s possible to use a debian/*.upstart.in and not use the templating system, in the more complicated case (I needed it mostly for neutron-server and neutron-plugin-openvswitch-agent). Conclusion I do not pretend that what I wrote in the openstack-pkg-tools is the ultimate solution. But I m convince that it answers our own need as the OpenStack maintainers in Debian. There s a lot of room for improvements (like implementing the Should-Start in Upstart jobs, or stop calling the sysv-rc script in the systemd units), but that this is a very good move that we did to use templates and generated scripts, as the init scripts are a way more easy to maintain now, in a much more unified way. As I m not completely satisfied for the systemd and Upstart implementation, I m sure that there s already a huge improvements on the sysv-rc script maintainability. Last and again: please send your comments and help improving the above! :)

My long-term free software project (Braille Music Compiler) just produced some offspring! xsdcxx-musicxml is now available on GitHub. I used CodeSynthesis XSD to generate a rather complete object model for MusicXML 3.0 documents. Some of the classes needed a bit of manual adjustment, to make the client API really nice and tidy. During the process, I have learnt (as is almost always the case when programming) quite a lot. I have to say, once you got the hang of it, CodeSynthesis XSD is really a very powerful tool. I definitely prefer having these 100k lines of code auto-generated from a XML Schema, instead of having to implement small parts of it by hand. If you are into MusicXML for any reason, and you like C++, give this library a whirl. At least to me, it is what I was always looking for: Rather type-safe, with a quite self-explanatory API. For added ease of integration, xsdcxx-musicxml is sub-project friendly. In other words, if your project uses CMake and Git, adding xsdcxx-musicxml as a subproject is as easy as using git submodule add and putting add_subdirectory(xsdcxx-musicxml) into your CMakeLists.txt. Finally, if you want to see how this library can be put to use: The MusicXML export functionality of BMC is all in one C++ source file: musicxml.cpp.

Like each month, here comes a report about the work of paid contributors to Debian LTS. Individual reports In November 42.5 work hours have been equally split among 3 paid contributors. Their reports are available:

• Thorsten Alteholz did his share as usual.
• Rapha l Hertzog worked 18 hours (catching up the remaining 4 hours of October).
• Holger Levsen did his share but did not manage to catch up with the backlog of the previous months. As such, those unused work hours have been redispatched among other contributors for the month of December.

New paid contributors Last month we mentioned the possibility to recruit more paid contributors to better share the work load and this has already happened: Ben Hutchings and Mike Gabriel join the list of paid contributors. Ben, as a kernel maintainer, will obviously take care of releasing Linux security updates. We are glad to have him on board because backporting kernel fixes really need some skills that nobody else had within the team of paid contributors. Evolution of the situation Compared to last month, the number of paid work hours has almost not increased (we are at 45.7 hours per month) but we are in the process of adding a few more sponsors: Roche Diagnostics International AG, Misal-System, Bitfolk LTD. And we are still in contact with a couple of other companies which have announced their willingness to contribute but which are waiting the new fiscal year. But even with those new sponsors, we still have some way to go to reach our minimal goal of funding the equivalent of a half-time position. So consider asking your company representative to join this project! In terms of security updates waiting to be handled, the situation looks better than last month: the dla-needed.txt file lists 27 packages awaiting an update (6 less than last month), the list of open vulnerabilities in Squeeze shows about 58 affected packages in total. Like last month, we re a bit behind in terms of CVE triaging and there are still many packages using SSLv3 where we have no clear plan (in response to the POODLE issues). The good side is that even though the kernel update spent a large chunk of time to Holger and Rapha l, we still managed to further reduce the backlog of security issues. Thanks to our sponsors

• Gold sponsors:
• The Positive Internet
• Silver sponsors:
• AD&D David Ayers IntarS Austria
• Blablacar
• Domeneshop AS
• Evolix
• Trollweb Solutions
• Universit Lille 3
• Bronze sponsors:
• Daevel SARL
• FOSSter
• Freeside Internet Service
• Intevation GmbH
• Linuxhotel GmbH
• MyTux
• Nantes M tropole
• Offensive Security
• Seznam.cz, a.s.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Sophie stood before me. I'm leaving with that guy, she gestured. Yes, I thought that would happen, I chuckled. She hugged me. The guy, whose name we managed to never utter, did not hug me, though he usually does. They went home together. That was the last time I saw Sophie. The rest of us sat down, finished our drinks, and split up. I went with Sophie's ex-girlfriend and the guy who sometimes serves as her ironic beard. They smoked their disgusting light cigarettes, the kind with very little tobacco but lots of horrible chemicals that make me cough and hopefully fail to give me lung cancer, because watching someone else die of that was excruciating enough. So we get to our next destination and there is a Peruvian girl sitting on a stool and shopping for shoes on her phone. I am fascinated. Phone app developers had told me that people actually did this but I thought it was just wishful thinking on their part. The Peruvian girl, who is named something that sounds like it was uttered accidentally by Tommy Gnosis, complains to Sophie's ex-girlfriend that some guy keeps harassing her. We instinctively form a human barrier to shield her from this alleged transgressor, who, it turns out, is the pompous drug dealer with whom Sophie's ex-girlfriend is just about to conduct business. I'll be right back, she says. Hit on her. What Why I shout after her. There is no response. Sophie's ex-girlfriend and the drug dealer return from the darkness, having swapped possessions. The drug dealer is a blowhard and proceeds to regale us with stories so little interest to me that I can't even remember what they were about, but as drug dealers are wont to do, he abuses the power of his possession to maintain the delusion that people would tolerate his presence even if he didn't have illegal commodities to sell them. When the beard and Sophie's ex-girlfriend go out for a smoke break, I went home.