Today was the first weekday of me taking care of Zoe. We got things started more or less on the same routine that we used to do when she had day care, so we were dressed and ready to roll by about 8am. We baked a batch of mini cornbread muffins, and had them ready by the time my 9:30am chiropractic appointment rolled around, so we took a few with us for morning tea for my chiropractor and his receptionist. I've taken Zoe with me on a few occasions, and speak of her often to my chiropractor. He's a dad himself, and the first few times I brought her in, he juggled for her, so she thinks he's pretty cool. Last night, Zoe had expressed a desire to go to Southbank today. It was initially looking like it'd be too rainy, but by the time I was done with the chiropractor, it looked safe enough to venture out, so we went back home, and I prepared a picnic lunch for us, and we caught a bus over to Southbank. We had a bit of a splash around in the water and then ate lunch. Zoe had a great time alternating between feeding and chasing seagulls, crows, and an ibis. My girlfriend, who works in South Brisbane, walked over and joined us briefly and then we all walked back towards the bus and parted ways. Southbank mid-week is pretty nice. Not too busy. It was a slow walk home from the bus, but the reward of a My Little Pony biscuit was enough to get her racing ahead of me from the corner. After a biscuit and some quiet time, she fell asleep and I've had a couple of hours to read and write blog posts, as well as plan future activities. I'm hoping that we'll have enough time to go birthday present shopping for a 4th birthday party (for twins from Zoe's old day care) that we're going to on Saturday morning after swim class, before Sarah arrives to pick her up, but I'm dubious we'll get time. Zoe's still asleep as I write. There's always tomorrow. We might be best off just baking the second batch of cornbread muffins.

A new minor / maintenance release RQuantLib 0.3.11 is now on CRAN and in Debian. Like the RcppClassic upload two days ago and the RcppZiggurat and RcppEigen uploads yesterday, this release was motivated at least in part by an upcoming Rcpp release about which I should have more to day soon. A few things got polished and updated, and we added a plugin which should make use via Rcpp Attribute easier. RQuantLib combines (some of) the quantitative analytics of QuantLib with the R statistical computing environment and language. Thanks to CRANberries, there is also a diff to the previous release 0.3.10. Full changelog details, examples and more details about this package are at my RQuantLib page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

It is unfortunate that I've been exposed to several companies that all need the same kind of problems solving, again and again:

• Imaging systems. Quickly.
• Configuring MySQL & Postgres replication and fail-over.
• Configuring local users.
• Solving the problem of distributing usernames/passwords to 500+ client-systems within a team.

A lot of these are solved problems, yet they seem to keep cropping up. I guess the fact I've done some of these things more than once means I'm the local-expert, so that's why I get asked. But still.. There was a time when I thought my Debian Administration site would help solve these kind of problems; that writing documentation would encourage people to do things properly. Certainly it has helped me, and some other people are greatful, but it didn't do enough to help. I'm not sure if the problem is that my documentation is a little ideosyncratic, isn't good enough, or isn't reaching the right kind of people. I suspect a combination of that and scale - You can't walk somebody though the idea of setting up a ten-node database-cluster, they need to suffer, they need to break things, they need to sweat on Christmas Day, at 5AM, as everything goes to hell. Then the next time they'll do it properly. I'd love to take the time to write out recipes in Salt, Ansible, Puppet, Chef, CFengine, Slaughter, whatever, and support them.

Remote. Automated. System management.

Throw in monitoring of metrics, security fixes, and reporting and there's probably a valuable service there. It probably can't happen though, for three main reasons:

• The people that need it don't know they need it. They're fighting fires, they know it is important and they will fix things "soon", but other work takes priority.
• The people that are tempted will baulk at the idea of unknown code from an external source running on their system(s).
• Companies managed by one sysadmin will wonder why they need more help, because "everything is working, right?".

I did recently write some policies for setting up a two-node Master-Slave MySQL setup, with reporting, monitoring, and custom SMS-based alerts. I guess I'm wondering if I can be cheeky and sell the same work twice. ;)

There will be a new option in gcc 4.9 named -fstack-protector-strong , which offers an improved version of -fstack-protector without going all the way to -fstack-protector-all . The stack protector feature itself adds a known canary to the stack during function preamble, and checks it when the function returns. If it changed, there was a stack overflow, and the program aborts. This is fine, but figuring out when to include it is the reason behind the various options. Since traditionally stack overflows happen with string-based manipulations, the default (-fstack-protector), only includes the canary code when a function defines an 8 (--param=ssp-buffer-size=N, N=8 by default) or more byte local character array. This means just a few functions get the checking, but they re probably the most likely to need it, so it s an okay balance. Various distributions ended up lowering their default --param=ssp-buffer-size option down to 4, since there were still cases of functions that should have been protected but the conservative gcc upstream default of 8 wasn t covering them. However, even with the increased function coverage, there are rare cases when a stack overflow happens on other kinds of stack variables. To handle this more paranoid concern, -fstack-protector-all was defined to add the canary to all functions. This results in substantial use of stack space for saving the canary on deep stack users, and measurable (though surprisingly still relatively low) performance hit due to all the saving/checking. For a long time, Chrome OS used this, since we re paranoid. :) In the interest of gaining back some of the lost performance and not hitting our Chrome OS build images with such a giant stack-protector hammer, Han Shen from the Chrome OS compiler team created the new option -fstack-protector-strong, which enables the canary in many more conditions:

• local variable s address used as part of the right hand side of an assignment or function argument
• local variable is an array (or union containing an array), regardless of array type or length
• uses register local variables

This meant we were covering all the more paranoid conditions that might lead to a stack overflow. Chrome OS has been using this option instead of -fstack-protector-all for about 10 months now. As a quick demonstration of the options, you can see this example program under various conditions. It tries to show off an example of shoving serialized data into a non-character variable, like might happen in some network address manipulations or streaming data parsing. Since I m using memcpy here for clarity, the builds will need to turn off FORTIFY_SOURCE, which would also notice the overflow.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
struct no_chars  
    unsigned int len;
    unsigned int data;
 ;
int main(int argc, char * argv[])
 
    struct no_chars info =    ;
    if (argc < 3)  
        fprintf(stderr, "Usage: %s LENGTH DATA...\n", argv[0]);
        return 1;
     
    info.len = atoi(argv[1]);
    memcpy(&info.data, argv[2], info.len);
    return 0;
 

Built with everything disabled, this faults trying to return to an invalid VMA:

$ gcc -Wall -O2 -U_FORTIFY_SOURCE -fno-stack-protector /tmp/boom.c -o /tmp/boom
$ /tmp/boom 64 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Segmentation fault (core dumped)

Built with FORTIFY_SOURCE enabled, we see the expected catch of the overflow in memcpy:

$ gcc -Wall -O2 -D_FORTIFY_SOURCE=2 -fno-stack-protector /tmp/boom.c -o /tmp/boom
$ /tmp/boom 64 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
*** buffer overflow detected ***: /tmp/boom terminated
...

So, we ll leave FORTIFY_SOURCE disabled for our comparisons. With pre-4.9 gcc, we can see that -fstack-protector does not get triggered to protect this function:

$ gcc -Wall -O2 -U_FORTIFY_SOURCE -fstack-protector /tmp/boom.c -o /tmp/boom
$ /tmp/boom 64 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Segmentation fault (core dumped)

However, using -fstack-protector-all does trigger the protection, as expected:

$ gcc -Wall -O2 -U_FORTIFY_SOURCE -fstack-protector-all /tmp/boom.c -o /tmp/boom
$ /tmp/boom 64 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
*** stack smashing detected ***: /tmp/boom terminated
Aborted (core dumped)

And finally, using the gcc snapshot of 4.9, here is -fstack-protector-strong doing its job:

$ /usr/lib/gcc-snapshot/bin/gcc -Wall -O2 -U_FORTIFY_SOURCE -fstack-protector-strong /tmp/boom.c -o /tmp/boom
$ /tmp/boom 64 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
*** stack smashing detected ***: /tmp/boom terminated
Aborted (core dumped)

For Linux 3.14, I ve added support for -fstack-protector-strong via the new CONFIG_CC_STACKPROTECTOR_STRONG option. The old CONFIG_CC_STACKPROTECTOR will be available as CONFIG_CC_STACKPROTECTOR_REGULAR. When comparing the results on builds via size and objdump -d analysis, here s what I found with gcc 4.9: A normal x86_64 defconfig build, without stack protector had a kernel text size of 11430641 bytes with 36110 function bodies. Adding CONFIG_CC_STACKPROTECTOR_REGULAR increased the kernel text size to 11468490 (a +0.33% change), with 1015 of 36110 functions stack-protected (2.81%). Using CONFIG_CC_STACKPROTECTOR_STRONG increased the kernel text size to 11692790 (+2.24%), with 7401 of 36110 functions stack-protected (20.5%). And 20% is a far-cry from 100% if support for -fstack-protector-all was added back to the kernel. The next bit of work will be figuring out the best way to detect the version of gcc in use when doing Debian package builds, and using -fstack-protector-strong instead of -fstack-protector. For Ubuntu, it s much simpler because it ll just be the compiler default.

2014, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.

You probably didn't know there's a Public Consultation on the review of the EU copyright rules under way as of right now. It states explicitly that "All stakeholders are welcome to contribute to this consultation." This means you. No matter where you live or what you do, this is your chance to potentially influence copyright reform. You have time until 2014-02-05. Don't wait that long, fill the survey out now. Else you will forget. And then lament the fact that legacy industries are upheld with artifical government-sanctioned monopolies. If you find the ~40 A4 pages of text scary, there's a handy guide as to what questions will interest you the most, but I suggest you fill it out on your own. Stop what you're doing, fill this out, and return. Thanks.

Seems I had exactly the same idea two years ago, down to choosing exactly the same name and directory structure... In case anyone finds it useful, there's Conference proceedings: a public git-annex repository which contains videos from various conferences. As of right now, it contains only FOSDEM up to 2008 and last year's Chaos Communication Congress, 30c3. I expect to add more and more conferences over time and patches are always welcome. Just try and make sure you don't include location data about your own repository in said commits. But there's already something in the works to fix that problem.

Behind the scenes, all the pieces are falling together to make this FOSDEM the best ever. As every year ;) Part of making things even more awesome is a great effort: To tape ALL the things. As in, literally, every single talk, discussion, and lightning talk; devrooms included. We have the extra equipment, we have the extra storage space, but we don't have all the extra manpower, yet. Why not sign up as a volunteer? There's a self-checkout system where you simply sign up for tasks. Heralding and video-taping are especially simple to do if you plan to attend the respective track/devroom, anyway. If you prefer talking over listening, cloak room and infodesk are available asa well; they are a lot more social and people always tell us it's great fun. Finally, if your schedule is full to the brim, you can help us with build-up and teardown. As an aside, there's a little experiment with IPv6 in the works.

For some strange reason, I ve been receiving a lot of GPG-signed mail from Debian Developers and Maintainers lately. In response to each of these mails, I need to send a GPG-encrypted reply. The rate at which I m able to send replies has been significantly hampered by the poor state in which many DD/DM s maintain their GPG keys. Here are a few common mistakes, so you can consider correcting them. Ensure you have a UID for the email address(es) you use When I send an encrypted mail, I need to be sure that the recipient is legit. This means any decent mail client should refuse to send an encrypted message to foo@bar.com unless that email address is known somehow to GPG. In many cases, someone with a valid key for foo@bar.com would send their signed mail from foobar@gmail.com without that being a valid UID. In some cases, foo@bar.com isn t even a valid email address anymore (i.e. the bar.com mail server says no such mailbox). You should have a UID for each address you use. Signatures are per-UID You may well have a valid UID for foo@debian.org, foo@bar.com, and foobar@gmail.com but the PGP trust model doesn t automatically trust every UID as much as its peers. Each individual UID needs to be trusted (i.e. signed/uploaded) by others. What if you added billg@microsoft.com as a UID should that automatically be trusted? Clearly not. Just because you have foo@debian.org doesn t mean it s trusted for encryption without some signatures. Make sure you actually have an encrypting subkey GPG sucks, and as a result, it reports Skipping unusable pubkey when the issue is a lack of valid encrypting subkeys. If you have revoked all encrypting subkeys, or allowed them to expire, then I cannot send you encrypted mail. Exact naming matters Bob Bobbertson <foo@bar.com> and B b B bb rts n <foo@bar.com> are different people. Check your mail client s From: setting, to ensure it matches your UID. If not, fix one of them. Check your webmail plugin isn t shit Some people use third party plugins to integrate GPG into their webmail client (e.g. Hotmail or GMail). Make sure this actually works. Don t use Enigmail Enigmail is a popular plugin to integrate GPG into Mozilla Thunderbird. It doesn t work, in most cases. Almost every single BADSIG in my inbox is due to Enigmail. Thunderbird will insert spurious line wraps and escape characters into your mail after signing, resulting in invalid signatures. It s mostly okay if you never quote mail, and restrict messages to about 70 characters. I know plenty of Debian Developers don t care about GPG other than for package signing but please, for the sanity of the rest of us, take an occasional moment to care a little. I should note that the worst offenders for keys which don t just work were Developers with 1024D keys the best behaved were Maintainers of all stripes.

In 2011, Mika and I bought in big at the Boston Red Bones party s charity raffle supporting MassBike and NEMBA and came out huge. I won $500 off a custom frame at Geekouse Bikes. For years, Mika and I have been planning to do the Tour d Afrique route (Capetown to Cairo), unsupported, on bike. People that do this type of ride sometimes use an expedition touring frame. I worked with Marty Walsh at Geekhouse to design a bike based on this idea. The concept was a rugged steel touring frame, built for my body and comfortable over long distances, with two quirks:

1. It s designed for 26 inch mountain bike wheels and mountain bike components to ensure that the bike is repairable with parts from the kinds of cheap mountain bikes that can be found almost everywhere in the world.
2. It includes S&S torque couplers that let me split the frame in half to travel with the bike as standard luggage.

As our pan-Africa trip kept getting pushed back, so did the need for the bike. Last week, I finally picked up the finished bike from Marty s shop in Boston. It is gorgeous. I absolutely love it. I m looking forward to building up the bicycle over the next couple months and I ll post more pictures when it s finished. I am blown away by Marty s craftsmanship and attention to detail. I am psyched that his donation made this bike possible and that I was able to get the frame while helping cycling in Massachusetts!

Today, we held an one-hour IRC-meeting to debate the status of the current architectures in sid. During that one hour, we brought up all 13 architectures. We made a decision on 9 of the architectures. The remaining 4 will be revisited within 2 months. As for the actual results, we will announce these on debian-devel-announce with our next bits mail. I suspect the announcement to happen within a couple of days. I am quite pleased that we managed to keep the meeting to one hour. Especially considering that it was planned less than a week ago. One of the things, that I believe helped us, was that we had summarised the status for each architecture prior to the meeting. In my view, the summary allowed us to finish at least half of the architectures within their allocated 5 minutes. In fact, we had 20 minutes for the last 2 architectures partly because amd64 + i386 combined took about a minute. A second thing that helped us stay on time was cutting debates short. Obviously, with less than 5 minutes per architecture, there was little time for debate and, I admit, that was by design. This meant that we had to defer 4 decisions for later. We are currently waiting for information, which we hope to have soon. More on that in the announcement later.
I would like to thank the attendees for contributing to a productive and short meeting. It was a pleasure and I hope we can repeat the success next time.

The uses of this particular rack I ll cover in future entries this is about how I made the rack itself, with help from friends (Steve McIntyre & Andy Simpkins). A common theme is making allowances for using dev kit boards ready-to-rack ARM servers are not here yet. My aim was to have 4, possibly 6 ARM dev kit boards running services from home, so there was no need for a standard 42U rack, a 9U rack is enough. Hence:
Wall Mounted 9U 450mm deep 19 inch rack enclosure glass door To run various other services around the house (firewall, file server etc.), a microserver was also necessary:
HP 704941-421 ProLiant Micro Server (AMD Turion II Neo N54L 2.2GHz, 2GB RAM, 250GB HDD, 2 Core, 7th Generation) I chose to mount that on a bookcase beneath the wall mounted rack as it kept all the cables at the bottom of the rack itself. The microserver needed a second gigabit network card fitted to cope with being a firewall as well, if you do the same, ensure you find a suitable card with a low profile bracket. Some are described as being low profile but do not package the low profile bracket, only a low profile card and a full height bracket.
Intel EXPI9301CTBLK PRO1000 Network Card note the low profile bracket in the pack. The first of the dev kit requirements is the lack of boards which can be racked, so shelves are going to be needed, leading on to something to stop the boards wandering across the shelf when the cables are adjusted velcro pads in my case. Second requirement is that dev kit boards are notorious for not rebooting cleanly. Nothing to do with the image being run, the board just doesn t cut power or doesn t come back after cutting power. Sometimes this is down to early revisions of the board, sometimes the board pulls enough power through the USB serial converter to remain alive, whatever the cause, it won t reboot without user involvement. So a PDU becomes necessary remotely controllable. New units tend to be expensive and/or only suitable for larger racks, I managed to find an older 8 port APC unit, something like: (Don t be surprised if that becomes a dead link search for APC Smart Slot Master Switch Power Distribution Unit). Talking of power, I m looking to use SATA drives with these boards and the boards themselves come with a variety of wall wart plugs or USB cables, so a number of IEC sockets are needed not the usual plugs:
Power cable IEC C14 plug 13A socket 25 cm or, for devices which genuinely need 2A to boot (use the 1A for attached SATA or leave empty):

Black Universal 3.1A 15W 5V Dual USB Rapid Mains Charger Plug Check the power output rating of the USB plugs used to connect to the mains as well many are 1A or less. Keep the right plug for the right board.
Power is going to also be a problem if, like me, you want to run SATA drives off boards which support SATA. The lack of a standard case means that ATX power is awkward, so check out some cheap SATA enclosures to get a SATA connector with USB power. I am using these enclosures (prices seem to have risen since I obtained them):

Startech 2.5 inch eSATA USB External Hard Drive Enclosure for SATA HDD Along with these:
eSATA to SATA Serial External Shielded Cable 1m because the iMx53 boards have SATA connectors but the enclosure exports eSATA. Whilst this might seem awkward, the merit of having both eSATA and simple USB power on one enclosure is not to be under-estimated. (Avoid the stiffer black cables space will get tight inside the rack.) Naturally, a 2.5 inch SATA drive is going to be needed for each enclosure, I m using HDD but SSD is also an option. Also, consider simple 2 or 3 way fused adaptors so that the board and the SATA drive can be powered from a single PDU port, this makes rebooting much simpler if the board needs a power supply with integrated plug instead of over USB. Now to the networking (2 x 8 port was cheaper than 1 x 16):
Netgear GS108 8-port Gigabit Ethernet Unmanaged Switch Don t forget the cat5 cables too you ll want lots of short cables, 1m or shorter inside the rack and a few longer ones going to the microserver and wall socket. I used 8x1m. Naturally, on the floor below your rack you are going to put a UPS, so the PDU power then needs to be converted to draw from the UPS via IEC plugs instead of standard mains. I decided to use a 6 gang extension 1m cable with an IEC plug it was the only bit of wiring I had to do and even those are available ready-made if you want to do it that way. Depending on the board, you may need your own serial to USB converters, you ll certainly need some powered USB hubs. I m using a wall mounted 9U rack, so I also needed a masonry drill and 4 heavy duty masonry M8 bolts. The rack comes with a mounting plate which needs to be bolted to the wall but nothing else is included. This step is much easier with someone else to take the weight of the rack as it is guided into the brackets on the mounting plate the bracket may need a little persuasion to allow for the bolt heads to not get in the way during mounting. Once mounted, the holes in the back of the rack allow for plenty of room, it s just getting to that point. The rack has side panels which latch out easily, providing easy maintenance. The glass door can be easily reversed to open from the opposite side. However, the key in the glass door is largely useless. The expectation is that the units in the rack are attached at the front but the dev boards on shelves are not going to be protected by a key in the front door. The key therefore ends up being little more than a handle for the glass door. OK. If you ve got this far, it s a case of putting things together:
Economy Cage Nut Tool 19 inch racking for cagenut extraction Yes, you really do want one. Fine, do without the premium one, but the economy one will save you a lot of (physical) pain. At this stage, it becomes clear that the normal 19 inch server rack shelves don t leave a lot of room at the back of the rack for the cables and there are a lot of cables. Each board has power, USB serial connection and network. The SATA has power too. The PDU has a power lead and you ll need network switches too. The network switches need power and an uplink cable. I positioned the supports in the rack as far forward as possible and attached the shelves to give enough room for the PDU on the base of the rack, the network switches resting on top and the extension bar (with the heavier, stiffer cables) at the back. (I need to bring my shelves another 1 or 2 positions further forward as there is barely enough room for one cable between the shelf and the back of the rack and that makes moving boards around harder than it needs to be.) The PDU defaults to enabling all ports at startup, so connect to it over telnet and turn off the ports before connecting things and sorting out the network interface to what the rest of the lab needs. (I m using a 10. range and the PDU was originally set to use 192.168.1.) That s about it as far as the hardware setup is concerned. Just time to label up each board, note down which PDU port does which device, which serial to USB converter is on which device on the microserver and check the power my initial problem with one board was traced to the inability of the board to power SATA off the on-board USB port even when using the provided 2A power supply. That meant adding a standard mains adaptor to feed both the SATA power and the board power off the one PDU port there is little point powering off the board but not the SATA, or vice versa. I haven t totalled up the expenditure but the biggest expenses were the microserver and the wall mounted rack but don t underestimate how much it will cost to buy 6 IEC plugs, various USB serial converters and how much you may spend on peripheral items. There is quite a lot of room on the 2 shelves for more boards, what will limit the deployment in this rack is space for the cables, especially power. The shorter the power cables, the easier it will be to maintain the devices in the rack. It might be worth looking at a 12U rack, if only to give plenty of space for cables. Once I ve got the software working, I ll describe what this rack will be doing it s got something to do with Debian, ARM, Linux and tests but you ve probably already guessed that much

A new maintenance release of RcppEigen is now on CRAN. And just like yesterday's RcppClassic upload and today's RcppZiggurat upload, the timing was motivated chiefly by some minor caretaking for an upcoming Rcpp release -- hopefully soon, and with more details to follow. But this release also contains some minor cleanups. The NEWS file entry shown below lists all changes.

Changes in RcppEigen version 0.3.2.0.2 (2014-01-26)

• Converted three unused unit test files to RUnit and removed Suggests: of testthat
• Add declaration to import a symbol from Rcpp to NAMESPACE to ensure proper instantiation with the upcoming Rcpp version
• Retire SHLIB.maker function

Courtesy of CRANberries, there are diffstat reports for the most recent release relative to the preceding version. Questions, comments etc about RcppEigen should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

A new maintenance release of RcppZiggurat is now on CRAN. And just like yesterday's RcppClassic upload, this is motivated chiefly by some minor caretaking for an upcoming Rcpp release -- hopefully soon, and with more details to follow. This release also contains some small extensions and work-in-progress. By using R's uniform generator, we can fall back to R's seeding of the RNG -- but going through the R data structures has a performance hit. Similarly, and somewhat surprisingly, the default generators in both Boost and the C++11 library are not all that speedy. The NEWS file entry shown below lists all changes.

Changes in version 0.1.2 (2013-12-28)

• Add a new generator deploying R's unif_rand which faciliates use of Ziggurat as the user-supplied generator for N(0,1)
• Update a 'local' demo comparing normal distribution RNGs from Boost, C++11 and Armadillo none of which are particularly speedy
• Add declaration to import a symbol from Rcpp to NAMESPACE to ensure proper instantiation with the upcoming Rcpp version

Courtesy of CRANberries, there are diffstat reports for the most recent release relative to the preceding version. More detailed information is on the RcppZiggurat page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

There is one thing in the current init system debate that irritates me. It is about the portability of init systems. The new init systems are evaluated by how portable they are. This is one of the arguments that is most often brought against systemd, which is understandable given the polarizing attitude of one of systemd's authors. In this context, the currently used sysvinit is considered portable. But in my book, it is not portable at all.

• There is no standard for the proc filesystem.
• Any program using the proc filesystem is therefore not portable.
• sysvinit uses the proc filesystem (in src/ bootlog,hddown,killall5 .c).
• Therefore, sysvinit is not portable.

But why can we use sysvinit to boot Debian/ kFreeBSD,Hurd ? Debian/kFreeBSD uses linprocfs for /proc to provide a familiar Linux-like environment for the userspace tools available in Debian (say, pgrep). linprocfs was originally written by the FreeBSD folks to support running Linux binaries using the Linux Binary Compatibility layer. Debian/Hurd uses procfs to provide /poc. This procfs translator is written mainly to provide a Linux-compatible /proc filesystem for the same reason linprocfs is used by Debian/kFreeBSD. So sysvinit works on those system not because it is portable, but because the environment has been made Linux-like enough for sysvinit. We (most likely anyone not using Linux) often do this, because it is the easiest way to run popular software developed (mainly) for Linux. This is often the path of the least resistance, as opposed to getting the upstream project to support the native way of doing things on platform X. During gsoc last year I had to patch our procfs to finally be able to safely shut down Debian/Hurd systems using sysvinit. The problem was, that sysvinit at certain runlevel transitions (like shutting down, or I guess, switching to single user mode), sysvinit assumes that it is okay to stop and kill (almost) all processes on the system (that's what killall5 does). This might be okay on monolithic systems, but on (multiserver) microkernel systems like the Hurd, where your root filesystem and your network driver and stack are running as userspace processes, it is clearly not. I wonder how Linux systems using a FUSE-based root filesystem get away with this. This highlights that not only sysvinit depends on a Linux-specific kernel interface (/proc), but it also hard-codes assumptions about the system architecture. Amusingly, systemd get's this right (ok, I'm not sure if it does, but it could get this right...). systemd organizes processes in cgroups, one for each service it starts and one for each login session or something like that. It can (could?) kill only those processes in it was responsible for, leaving all essential translators (system servers) alone. In fact, even my tiny cgroupfs prototype can keep track of translators that are started by the root filesystem translator.

After years of missing this conference, I will finally make it to Brussels this time! I will even have some talks, one about Listaller (in the lightning-talks section) and one about AppStream and Listaller in the Distributions track. The lightning-talk will explain why distributing (some) applications independent from the distribution makes sense, and how Listaller does this. I will also very briefly talk about the concepts behind Listaller and which tools it offers for application developers to create & package cross-distro apps. The AppStream & Listaller talk will be much more detailed. It will cover the rationale for AppStream, what AppData is good for and how AppData files relate to AppStream. I will also reflect the AppData adoption in GNOME/Fedora and why GNOME-Software is the right way to go forward in developing software-centers. It will of course also include our future plans for AppStream. On the Listaller side, I will talk about how Listaller is connected to AppStream and PackageKit and why distributions should ship a way to install cross-distro packaged apps at all. I will explain module-definitions and why they are useful. An overview of the internals of Listaller and it s system integration is also included, as well as how it can be compared to competing installation solutions. If you are at FOSDEM and have questions about AppStream/PackageKit/Listaller/Tanglu/Debian/etc., please ask them! See you there!

A maintenance release of RcppClassic, now at version 0.9.5, went out to CRAN today. Nothing changed in user-facing code. The build was triggered by a change in an upcoming Rcpp release (more about that another time), and a few minor updates were needed for extended checks done by R. Courtesy of CRANberries, there is the set of changes relative to the previous release 0.9.4. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Friday was my last day at Solarflare Communications, which I joined in 2006 (originally as Level 5 Networks). When I started, Level 5 Networks (L5N) had a 2-port gigabit Ethernet controller called EF1 and a user-level network stack called EtherFabric which ran on Linux. It had nearly completed the Falcon project to develop a 10-gigabit controller (also supporing a 2-port gigabit mode) and was porting EtherFabric to Windows. Later that year, however, L5N merged with Solarflare Communications (SF), which was developing a 10GBASE-T PHY (for 10-gigabit Ethernet over twisted-pair cables). The investors and management saw the opportunity to create an integrated 10GBASE-T LAN-on-motherboard chip that would be attractive to OEMs and so had the potential for mass sales. EtherFabric was put on the back-burner, but regular net drivers for Linux, Windows and other operating systems became more important. In a change from my previous jobs, my initial role at L5N was to maintain and extend the in-house test automation software, Runbench, written in Python. I worked with Dickon Reed who had written most of Runbench up to then. One of my first major tasks was to extend it to cover DUTs running Windows. Aside from changing Runbench itself, I wrote a simple SSH server to be installed on DUTs, using the Twisted.Python framework. (Cygwin's port of OpenSSH wasn't suitable for running a lot of native Windows commands.) I rewrote the test-installation script for the Windows driver itself, converting it to Python (with a C extension to get around the WOW64 nonsense). As Windows does not allow loading a driver just once (like insmod in Linux), except by using the kernel debugger, I added a time check in the driver so the test-installation script can make it fail early when reloaded after a reboot, so that a crasher bug in the probe path would usually be recoverable. In 2007 I persuaded management in Cambridge to loan several older servers to DebConf. I took them there and back by car, and they were used to convert video for live streams and recordings. (This was repeated in 2009.) I was also asked to work on other small development tasks, including some cleanup on the Linux net driver (sfc). In mid-2007, when SF decided it was time to get sfc upstream (in-tree), I joined Steve Hodgson and Robert Stonehouse in working on that. Out-of-tree (separately distributed) drivers for Linux normally need a large number of preprocessor conditions for compatibility with Linux's kernel module API across a range of supported versions. They may also implement some features with a driver-specific interface that should be replaced with a standard interface (possibly including the work to define that standard). They may not comply with the kernel coding style or unwritten conventions for kernel code. All of these problems were present and had to be dealt with. As part of this work, I extended unifdef so that it could be used to remove all of the backward-compatibility and not-for-upstream code without introducing extra blank lines. I could then export the out-of-tree driver from CVS(!) into a kernel source tree automatically and turn it into a patch or patch series that would stand a chance of being acceptable upstream. Our first few attempts got little response, but eventually we newbies got the message that the driver was simply too big for a single submission and that putting patches on a web site because they're too big for the list was not a solution. In the next week, I removed about half of the code (on a git branch), resulting in a relatively lean driver that could be sent to the netdev list. And finally the 9th (I think) submission was accepted. Following this, I continued to work on sfc but was also brought into the Siena project. Siena (SFL9021) was the LAN-on-motherboard chip that had been planned following the merger. It combined a dual-port controller based on Falcon with a 10GBASE-T PHY and a management controller (MC) to support Lights-Out Management (LOM). (Since 10GBASE-T has not taken off, it is now mostly sold as the SFC9020 variant in which the PHY is disabled.) I worked on some of the test framework and test cases for validating the controller design in software simulation and FPGA. In the course of this I learned to read and (somewhat) understand the chip design written in Verilog, but only made a single trivial fix to it. In August 2009, the first Siena ASICs came back from the fab. There were a few ASIC-specific bugs but all of them were quickly worked around in firmware, so it would soon be ready for production. But there was much work still to be done on the driver and firmware. The firmware had until then been running in a smaller block of RAM with no peripherals to manage, while sfc had earlier been modified just enough to configure a sim or FPGA for running a userland test application. sfc now had many special cases scattered around it, and would tell the MC firmware to peek and poke registers that were no longer directly accessible from the host. The software and firmware developers had agreed that, in order to support LOM, the MC would be responsible for managing the ports and all peripherals and drivers would send higher-level requests to the MC. As a bonus, this removed the need for the driver to know the details of each new board, so that it would not be necessary to backport sfc into distribution kernels very often. Over the next few months, the firmware team did an excellent job of defining and implementing those operations, including writing the driver-side functions to invoke them. Meanwhile, Steve and I concentrated on refactoring sfc so most of the Falcon/Siena differences could be abstracted through a few structures with function pointers. The refactoring and new code for Siena were completed and submitted upstream in several large patch series in October and November. In fact, there were so many patches that Solarflare appeared in LWN's table of who wrote Linux 2.6.33 (as did I, thanks also to another patch series adding firmware metadata to many drivers). Sadly we had missed Linux 2.6.32 which was a longterm stable branch and used in many distributions, but I was able to get this version backported into all the major distributions over the next year. (To be continued.)