Recently, I ve been watching the Debian Schadenfreude related to init systems. For those who do not know, Debian is trying to decide whether Systemd or Upstart will be used to start software on Debian. There are two other options, although I think Systemd and Upstart are the main contenders. Currently the Technical Committee is considering the issue, although there s some very real chance that the entire project will get dragged through this swamp with the general resolution process. This is one of those discussions that proves that Debian truly is a community: if we didn t have something really strong holding us together we d never put up with something this painful. Between the accusations that our users now know the persecution European pagans faced at the hands of Christians as the Systemd priests drive all before them, a heated discussion of how the Debian voting system interacts with this issue, two failed calls for votes, a discussion of conflicts of interests, and a last-minute discussion of whether the matter had been appropriately brought before the Technical committee (and if so, under what authority), there s certainly schadenfreude to go around if you re into that sort of thing. However, through all this, the Technical Committee has been doing great work understanding the issues; it has been a pleasure to watch their hard work from the side lines. I d like to focus on one key question they ve found: how tightly can other software depend on the init system. Each init system offers some nice features. Upstart has an event triggering model. Systemd manages login sessions and at least in my opinion has a really nice service file format. Can I take advantage of these in my software. If I do, then users of my software might need to use a particular init system. Ian Jackson argues that we should give our users the choice of what init system to run. He reminds us that Debian is a community that supports diversity and diverse goals. We support multiple desktop systems, web browsers, etc. Wherever we can, we support people working on their goals and give developers the opportunity to make it work. When developers succeed, we give our users the opportunity to choose from among the options that have been developed. I think of Ian s argument as an appeal to part of our Social Contract. Clause 4 of the social contract begins:

Our priorities are our users and free software.
We will be guided by the needs of our users and the free software community. We will place their interests first in our priorities. We will support the needs of our users for operation in many different kinds of computing environments.

Ian is right that our users will be served by choice in init systems. However, this trade off needs to be balanced against the needs of the free software community. Diversity is an important goal but it should not come at the price of stagnation and innovation. If I want to avoid using init scripts because they don t provide restart on failure, because they are hard to write correctly, and because they don t provide access to modern security isolation techniques, I should be able to do that. If Systemd service files provide a superior solution, I should be able to work toward using them. If the desire for init system diversity shuts down my ability to find like-minded people who want to take advantage of improvements in init systems and work towards that goal, then we ve significantly damaged Debian as a forum for technical cooperation. Early in my work as a Debian Developer, Anthony Towns taught me an interesting principle. Those who do the work get significant influence over how and what gets done. Ian s right that we need to enable people to work towards choice in init systems. Those like-minded people need to be given a chance to find each other and pursue their goal. However the cost of success should be on the shoulders of those who value choice in init systems. It should not come at the cost of preventing people from depending on improvements in init systems. The best proposal I ve seen for balancing this is to enumerate stable interfaces that people want to use. Things like the logind interface or my favorite the service file interface. It needs to be possible to make another implementation of the interface. The interface needs to be stable enough that a dedicated team could have a chance of catching up with an implementation. At some point during the release cycle such interfaces would need to be frozen. However, I don t think it is reasonable to mandate that there are multiple implementations of the interface, only that there could be. The point is to give people a chance to work towards diversity in init systems, not to force it down peoples throats kicking and screaming until they leave the project or ignore our processes. Steve Langasek and Colin Watson seem to be working towards this. It s possible there s another approach besides interfaces. My main concern is the same as Ian s: maintain Debian as a forum for people to pursue their goals and work together. I suspect we see the conflict in these goals differently. I hope that we as a project can explore this conflict and figure out where we have common ground. I commit to exploring how we can work towards init choice in my frameworks; I ask those who prioritize init choice to commit to explore how we can take advantage of new features in their framework.

Since August 2011, my sites such as complete.org have been running on a Xen-backed virtual private server (VPS) at Hetzner Online, based in Germany. I had what they called their VQ19 package, which included 2GB RAM, 80GB HDD, 100Mb NIC and 4TB transfer. Unlike many other VPS hosts, I never had performance problems. However, I did sometimes have hardware problems with the host, and it could take hours to resolve. Their tech support only works business hours German time, which was also a problem. Meanwhile, OVH, a large European hosting company, recently opened a datacenter in Canada. Although they no longer offer their value-line Kimsufi dedicated servers there starting at $11.50/mo they do offer their midrange SoYouStart servers there. $50/mo gets a person a 4-core 3.2GHz Xeon server with 32GB RAM, 2x2TB SATA HDD, 200Mbps bandwidth. Not bad at all! The Kimsufi options are still good for lower-end needs as well. I signed up for one of the SoYouStart servers. I ve been pleased with my choice to migrate, and at the possibilities that having hardware like that at my disposal open up, but it is not without its downside. The primary downside is lack of any kind of KVM console. If the server doesn t boot, I can t see the Grub error message (or whatever) behind it. They do provide hardware support and automatic technician dispatching when the server isn t pingable, but they state they have no KVM access at all. They support many OS flavors, and have a premade image for them, but there is no using a custom ISO to install; if you want ZFS on Linux, for instance, you can t very easily build it into root. My server was promised within 72 hours, but delivered much quicker: within about 1. I had two times they said they had to replace a motherboard within the first day; once they did it in 30 minutes, and the other took them 2.5 hours for some reason. They do have phone support, which answers almost immediately, but the people there are not the people actually in the datacenter. It was frustrating with a server down for hours and nobody really commenting on what was going on. The server performs quite well, and after the initial issues, I ve been happy. I was initially planning an all-ZFS installation. SoYouStart does offer a rescue environment, but it doesn t support ZFS, so I figured I better stick with an ext4 root at least. The default Debian install uses RAID1 on md-raid, with a 20GB root partition and the rest of the 2TB drive in /home, and then a swap partition on each drive (mysteriously NOT in the RAID!) So I broke the mirror on /home and converted those into the two legs of a mirrored vdev for a zpool. I run all of the real work inside KVM VMs, so that should minimize the number of times I have to do anything to the root filesystem that could cause trouble. SoYouStart includes 100GB of space on a separate FTP server for backup purposes. I have scripts that upload nightly tarballs of the root filesystem, plus full zfs send streams of everything else. Every hour, it uploads an incremental zfs send stream as well. This all works quite nicely; even if the machine is a complete loss, I d never lose more than an hour s work, and could restore it completely from a rescue environment. Very nice! I ll write more in a few days about the ZFS setup I m using, and some KVM discoveries as well.

Today was a short day. Sarah is going down to Canberra for the weekend with Zoe, and wanted to pick her up after her Brazilian Jiu Jitsu class. We got away a bit late on the bike and got to the class maybe 5 minutes late, but it was just us anyway, so it was all good. I know now it's best to make sure we're out the door no later than 9am. The class was just as good as last time, and Sarah arrived halfway through to watch, so she got to see it for herself. I biked home the scenic way again, but made a slight wrong turn somewhere, so I popped out further up Wynnum Road than I wanted to be, and so had to backtrack a bit to cross it. After I got home, I decided to pop out to Jaycar again and get some more bits and pieces and do some grocery shopping. I'd made some conductive play dough on Wednesday afternoon while Megan was here, and the girls just used it as standard play dough. Today I made some non-conductive play dough to go with it, and tested it all out. At Jaycar I bought a battery pack and some banana clips, so I soldered everything together and it works nicely. We can just jam the banana clips into the appropriate bits of play dough and make LEDs light up. I bought a bunch of LEDs from Jaycar as well. So we should be all set for next Friday. After that I spent the rest of the afternoon tinkering around with my electronics project. There's nothing like learning by trial and error.

My last update about ongoing development of botch, the bootstrap/build ordering tool chain, was three months ago with the announcement of bootstrap.debian.net. Since then a number of things happened, so I thought an update was due.

New graphs for port metrics By default, a dependency graph is created by arbitrarily choosing an installation set for binary package installation or source package compilation. Installation set vertices and source vertices are connected according to this arbitrary selection. Niels Thykier approached me at Debconf13 about the possibility of using this graph to create a metric which would be able to tell for each source package, how many other source packages would become uncompilable or how many binary packages would become uninstallable, if that source package was removed from the archive. This could help deciding about the importance of packages. More about this can be found at the thread on debian-devel. For botch, this meant that two new graph graphs can now be generated. Instead of picking an arbitrary installation set for compiling a source package or installing a binary package, botch can now create a minimum graph which is created by letting dose3 calculate strong dependencies and a maximum graph by using the dependency closure.

Build profile syntax in dpkg With dpkg 1.17.2 we now have experimental build profile support in unstable. The syntax which ended up being added was:

Build-Depends: large (>= 1.0), small <!profile.stage1>

But until packages with that syntax can hit the archive, a few more tools need to understand the syntax. The patch we have for sbuild is very simple because sbuild relies on libdpkg for dependency parsing. We have a patch for apt too, but we have to rebase it for the current apt version and have to adapt it so that it works exactly like the functionality dpkg implements. But before we can do that we have to decide how to handle mixed positive and negative qualifiers or whether to remove this feature altogether because it causes too much confusion. The relevant thread on debian-dpkg starts here.

Update to latest dose3 git Botch heavily depends on libdose3 and unfortunately requires features which are only available in the current git HEAD. The latest version packaged in Debian is 3.1.3 from October 2012. Unfortunately the current dose3 git HEAD also relies on unreleased features from libcudf. On top of that, the GraphML output of the latest ocamlgraph version (1.8.3) is also broken and only fixed in the git. For now everything is set up as git submodules but this is the major blocker preventing any packaging of botch as a Debian package. Hopefully new releases will be done soon for all involved components.

Writing and reading GraphML Botch is a collection of several utilities which are connected together in a shell script. The advantage of this is, that one does not need to understand or hack the OCaml code to use botch for different purposes. In theory it also allows to insert 3rd party tools into a pipe to further modify the data. Until recently this ability was seriously hampered by the fact that many botch tools communicated with each other through marshaled OCaml binary files which prevent everything which is not written in OCaml from modifying them. The data that was passed around like this were the dependency graphs and I initially implemented it like that because I didnt want to write a GraphML parser. I now ended up writing an xmlm based GraphML parser so as of now, botch only reads and writes ASCII text files in XML (for the graphs) and in rfc822 packages format (for Packages and Sources files) which can both easily be modified by 3rd party tools. The ./tools directory contains many Python scripts using the networkx module to modify GraphML and the apt_pkg module to modify rfc822 files.

Splitting of tools To further increase the ability to modify program execution without having to know OCaml, I split up some big tools into multiple smaller ones. Some of the smaller tools are now even written in Python which is probably much more hackable for the general crowd. I converted those tools to Python which did not need any dose3 functionality and which were simple enough so that writing them didnt take much time. I could convert more tools but that might introduce bugs and takes time which I currently dont have much of (who does?).

Gzip instead of bz2 Since around January 14, snapshot.debian.org doesnt offer bzip2 compressed Packages and Sources files anymore but uses xz instead. This is awesome for must purposes but unfortunately I had to discover that there exist no OCaml bindings for libxz. Thus, botch is now using gzip instead of bz2 until either myself or anybody else finds some time to write a libxz OCaml binding.

Self hosting Fedora Paul Wise made me aware of Harald Hoyer's attempts to bootstrap Fedora. I reproduced his steps for the Debian dependency graph and it turns out that they are a little bit bigger. I'm exchanging emails with Harald Hoyer because it might not be too hard to use botch for rpm based distributions as well because dose3 supports rpm. The article also made me aware of the tred tool which is part of graphviz and allows to calculate the transitive reduction of a graph. This can help making horrible situations much better.

Dose3 bugs I planned to generate such simplified graphs for the neighborhood of each source package on bootstrap.debian.net but then binutils stopped building binutils-gold and instead provided binutils-gold while libc6-dev breaks binutils-gold (<< 2.20.1-11). This unfortunately triggered a dose3 bug and thus bootstrap.debian.net will not generate any new results until this is fixed in dose3. Another dose3 bug affects packages which Conflicts/Replaces/Provides:bar while bar is fully virtual and are Multi-Arch:same. Binaries of different architecture with this property can currently not be co-installed with dose3. Unfortunately linux-libc-dev has this property and thus botch cannot be used to analyze cross builds until that bug is fixed in dose3. I hope I get some free time soon to be able to look at these dose3 issues myself.

More documentation Since I started to like the current set of tools and how they work together I ended up writing over 2600 words of documentation in the past few days. You can start setting up and running botch by reading the first steps and get more detailed information by reading about the 28 tools that botch makes use of as of now. All existing articles, thesis and talks are linked from the wiki home.

I attended FOSDEM this year. As always, it was very busy, and the Brussels transport system was as confusing as ever. This time it was nice to accidentally bump into so many people I know from years past. Lunar's talk on reproducible builds of Debian packages was interesting - being able to independently verify that a particular binary package was built from a particular source package is quite attractive. Also Mailpile declared an alpha release. The bit in the talk that I didn't know already was the description of exactly how the search function works. Seeing how they integrate GPG into the contacts/compose features brought home how lacking most (all?) other mail clients are when it comes to usable encryption. On Sunday afternoon I managed to grab a seat in the Go devroom, to hear about crazy things like how YouTube are putting a daemon called Vitess in front of MySQL to abstract away sharding (at the cost of some transaction guarantees). You would have thought Google would already have a scalable data store of some sort? Other bits I remember: Michael Meeks talking about GPU-enabling spreadsheet formulae calculations. And hearing Wietse Venema talk about Postfix was pretty awesome.

The Debian Technical Committee has concluded that all of the available init systems were missing a key technical feature, and, in the spirit of free software, has gone ahead and fixed that lack. All init systems currently packaged for Debian can now double as test case generators for Condorcet vote analysis. I'm not sure if this counts as an NMU. In the event that the Debian project votes on a General Resolution on the default init system, this blog post should be considered striken in its entirety and replaced by the most complex email message from Anthony Towns that the reader can find.

Debops What I like the most about being a Developer is building tools to (hopefully) make someone's life better. I like it when my software gets used, and people thank me for it, because there was a need they had that wasn't met before, and thanks to my software now it is being met. I am maintaining software for meteorological research that is soon going to be 10 years old, and is still evolving and getting Real Work done. I like to develop software as if it is going to become a part of human cultural heritage, developing beyond my capacity, eventually surviving me, allowing society to declare that the need, small as it was, is now met, and move on to worry about some other problem. I feel that if I'm not thinking of my software in that way, then I am not being serious. Then I am not developing something fit for other people to use and rely on. This involves Development as much as it involves Operations: tracking security updates for all the components that make up a system. Testing. Quality assurance. Scalability. Stability. Hardening. Monitoring. Maintenance requirements. Deployment and upgrade workflows. Security. I came to learn that the requirements put forward by sysadmins are to be taken seriously, because they are the ones whose phone will ring in the middle of the night when your software breaks. I am also involved in more than one software project. I am responsible for about a dozen web applications deployed out there in the wild, and possibly another dozen of non-web projects, from terabyte-sized specialised archival tools to little utilities that are essential links in someone's complex toolchain. I build my software targeting Debian Stable + Backports. At FOSDEM I noticed that some people consider it uncool. I was perplexed. It provides me with a vast and reasonably recent set of parts to use to build my systems. It provides me with a single bug tracking system for all of them, and tools to track known issues in the systems I deployed. It provides me with a stable platform, with a well documented upgrade path to the next version. It gives me a release rhythm that allows me to enjoy the sweet hum of spinning fans thinking about my next mischief, instead of spending my waking time chasing configuration file changes and API changes deep down in my dependency chain. It allows me to rely on Debian for security updates, so I don't have to track upstream activity for each one of the building blocks of the systems I deploy. It allows me not to worry about a lot of obscure domain specific integration issues. Coinstallability of libraries with different ABI versions. Flawless support for different versions of Python, or Lua, or for different versions of C++ compilers. It has often happened to me to hear someone rant about a frustrating situation, wonder how come it had never happened to me, and realise that someone in Debian, who happens to be more expert than I can possibly be, had thought hard about how to deal with that issue, years before. I know I cannot be an expert of the entire stack from bare iron all the way up, and I have learnt to stand on the shoulders of giants. 'Devops' makes sense for me in that it hints at this cooperation between developers and operators, having constructive communication, knowing that each side has their own needs, trying their best to meet them all. It hints at a perfect world where developers and operators finally come to understand and trust each other's judgement. I don't know that perfect world, but I, a developer, do like to try to understand and trust the judgement of sysadmins. I sympathise with my sysadmin friends who feel that devops is turning into a trend of developers thinking they can do without sysadmins. Reinventing package managers. Bundling dependencies. Building "apps" instead of components. I wish that people who deploy a system built on such premises, have it become so successful that they end up being paid to maintain them for their whole career. That is certainly what I wish and strive for, for me and my own projects. In my experience, a sustainable and maintainable system won't come out of the startup mindset of building something quick&dirty, then sell it and move on to something else. In my experience, the basis for having sustainable and maintainable systems have been well known and tested in Debian, and several other distributions, for over two decades. At FOSDEM, we thought that we need a name for such a mindset. Between beers, that name came to be "debops". (It's not just Debian, though: many other distributions get it right, too)

Today is the monday after FOSDEM 2014, and I'm slowly waking up. Being in charge of the video team, and after having spent a whole week preparing followed by a whole weekend stressing about, today is a day of slowly waking up from hibernation and returning gear to various rental companies. At FOSDEM 2013, we had recorded five rooms. Even though we did lose a few talks, I felt pretty good about the whole thing, and I think we did pretty well then. It had been my plan to increase the number of rooms to be recorded from 5 to a reasonably higher number say, 10 or so but someone convinced me to do all of them. I'll readily admit I was afraid. Scratch that, I was terrified. Petrified, that we wouldn't be able to pull it off. For previous editions, we'd kept the number conservative, since video work is a high-risk business: if something goes wrong with volunteers work (say, the heralding volunteer doesn't show up), you find a replacement, or you have a staff member introduce a speaker, and the show will go on. Not so with video work: if something minor goes wrong, you will most likely lose content. If something minor goes wrong in one devroom, I'll deal with it. If something minor goes wrong in 20% of devrooms, that's 4 to 5 devrooms, and I can't deal with it. Moreover, finding enough volunteers to manage the video for five rooms was a challenge last year; I couldn't even begin to imagine how to do so for 22 of them. We did work out a way that didn't seem too far-fetched, and that might work: why not look for volunteers from the devrooms themselves? But then, they couldn't be expected to have the required experience, so we'd have to do a lot of handholding. The week before FOSDEM, four of us sat down in an office in Mechelen, and started preparing. We had a lot to do; and when you do, more jobs keep being added to the pile; TODO lists tend to grow longer, not shorter. Installing servers, testing cameras, setting twinpact dipswitches, configuring laptops. Changing a few tests in the config management system, and having to go over the laptops again. Buying extra gear. Buying yet more extra gear, several times. Buying cardboard boxes when it appeared that the plastic foldable ones we'd ordered wouldn't make it in time, and unloading the pallet of them when it did, after all, so we could load them into the van with all of the other gear. Laminating instruction sheets. Receiving shipments of rented gear. Calling another supplier last minute when one of our other suppliers had contacted us to let us know one of their cameras that they'd promised us had broken down, and couldn't be repaired in time. Calling that last-minute supplier again when we found out that some of the cameras we'd rented could do HDV only (and not DV), and we needed to find more than the ones we'd already asked for. Discovering that his cameras were of the same problematic model and so also couldn't be used. The feeling of relief when he called me a few hours later and told me he'd managed to scrounge up enough cameras to help us out after all. And then it was time to go to the ULB and start setting up everything. On friday, we had a little bit of help, but did have to do most of it ourselves. After setting up everything in every room and testing as much as possible within the time constraints that we had, we picked up the most expensive bits of the gear, brought it down to a safe place, and went home. On saturday and sunday, for me it was mostly a matter of handing out gear to various devrooms, running around to fix various issues, and then in the evening receiving boxes back, using a checklist to ensure everything expected was in the box (and that things not needed would remain in the room, or be sent back). It was exhausting, to the extent that on saturday afternoon, I had to take a bit of time off so I could go and rest. I think it's safe to say that something like this hasn't been done before. FOSDEM is a huge conference; most conferences with multiple rooms don't have more somewhere between 5 and 10 of them. The fact that we have so many more, and the fact that we record all of them, puts FOSDEM video work in a class of its own, to the extent that the professional cameramen I talked to in order to get the right cameras would incredulously ask "what do you need that for?" when I gave them the numbers we needed. As such, I didn't expect, nor was aiming for, perfection. Unfortunately, with video work it is close to impossible to attain perfection; I have told multiple people during the event that I would be happy if we reached 85% of talks recorded at acceptable quality; and though it is much too soon to say for certain, my gut feeling tells m that we've probably achieved that. One thing we hadn't planned for was streaming. During the past three years, FOSDEM enlisted the help and sponsorship of flumotion to get streaming going. Unfortunately, that did not work out this year; and since we already had far too much on our plates with everything else, we decided to forego streaming and focus on recording, only. A few days before the event, however, Steinar Gunderson took it upon himself to fix that. While we couldn't support him a great deal, we could give him access to the secondary laptops (which weren't otherwise doing much anyway) on which he could then do his thing. This was mostly transparent to us; we did communicate to some extent (e.g., when a machine that had been down had fixed and was working again), but he mostly did his thing while we did ours. Full details, for those who want it, in the linked blog post. Today, then, I spent most of my time handling hardware: waiting for other members of FOSDEM to drive the van up to my office; waiting for the laptop rental company to retrieve most of the laptops; waiting for two of the camera rental companies to retrieve their cameras and tripods; driving over to the final camera rental company to return their cameras and tripods; and finally, driving up to the ULB to retrieve the last laptops who had been copying files all night. As I write this, some video files have already been uploaded to the FOSDEM video archive. However, these are extremely low-quality renditions of video snippets that need to be reviewed; they are not ready yet for public consumption. When those files exist, expect an announcement through the main FOSDEM website.

Back in 2006, the compiler in Ubuntu was patched to enable most build-time security-hardening features (relro, stack protector, fortify source). I wasn t able to convince Debian to do the same, so Debian went the route of other distributions, adding security hardening flags during package builds only. I remain disappointed in this approach, because it means that someone who builds software without using the packaging tools on a non-Ubuntu system won t get those hardening features. Think of a sysadmin trying the latest nginx, or a vendor like Valve building games for distribution. On Ubuntu, when you do that ./configure && make you ll get the features automatically. Debian, at the time, didn t have a good way forward even for package builds since it lacked a concept of global package build flags . Happily, a solution (via dh) was developed about 2 years ago, and Debian package maintainers have been working to adopt it ever since. So, while I don t think any distro can match Ubuntu s method of security hardening compiler defaults, it is valuable to see the results of global package build flags in Debian on the package archive. I ve had an on-going graph of the state of build hardening on both Ubuntu and Debian for a while, but only recently did I put together a comparison of a default install. Very few people have all the packages in the archive installed, so it s a bit silly to only look at the archive statistics. But let s start there, just to describe what s being measured. Here s today s snapshot of Ubuntu s development archive for the past year (you can see development opening after a release every 6 months with an influx of new packages):
Here s today s snapshot of Debian s unstable archive for the past year (at the start of May you can see the archive unfreezing after the Wheezy release; the gaps were my analysis tool failing):
Ubuntu s lines are relatively flat because everything that can be built with hardening already is. Debian s graph is on a slow upward trend as more packages get migrated to dh to gain knowledge of the global flags. Each line in the graphs represents the count of source packages that contain binary packages that have at least 1 hit for a given category. ELF is just that: a source package that ultimately produces at least 1 binary package with at least 1 ELF binary in it (i.e. produces a compiled output). The Read-only Relocations ( relro ) hardening feature is almost always done for an ELF, excepting uncommon situations. As a result, the count of ELF and relro are close on Ubuntu. In fact, examining relro is a good indication of whether or not a source package got built with hardening of any kind. So, in Ubuntu, 91.5% of the archive is built with hardening, with Debian at 55.2%. The stack protector and fortify source features depend on characteristics of the source itself, and may not always be present in package s binaries even when hardening is enabled for the build (e.g. no functions got selected for stack protection, or no fortified glibc functions were used). Really these lines mostly indicate the count of packages that have a sufficiently high level of complexity that would trigger such protections. The PIE and immediate binding ( bind_now ) features are specifically enabled by a package maintainer. PIE can have a noticeable performance impact on CPU-register-starved architectures like i386 (ia32), so it is neither patched on in Ubuntu, nor part of the default flags in Debian. (And bind_now doesn t make much sense without PIE, so they usually go together.) It s worth noting, however, that it probably should be the default on amd64 (x86_64), which has plenty of available registers. Here is a comparison of default installed packages between the most recent stable releases of Ubuntu (13.10) and Debian (Wheezy). It s clear that what the average user gets with a default fresh install is better than what the archive-to-archive comparison shows. Debian s showing is better (74% built with hardening), though it is still clearly lagging behind Ubuntu (99%):

2014, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.

A few days ago I decided to try to help the Hurd people to get their changes into sysvinit, to allow them to use the normal sysvinit boot system instead of their old one. This follow up on the great Google Summer of Code work done last summer by Justus Winter to get Debian on Hurd working more like Debian on Linux. To get started, I downloaded a prebuilt hard disk image from http://ftp.debian-ports.org/debian-cd/hurd-i386/current/debian-hurd.img.tar.gz, and started it using virt-manager. The first think I had to do after logging in (root without any password) was to get the network operational. I followed the instructions on the Debian GNU/Hurd ports page and ran these commands as root to get the machine to accept a IP address from the kvm internal DHCP server:

settrans -fgap /dev/netdde /hurd/netdde 
kill $(ps -ef awk '/[p]finet/   print $2 ')
kill $(ps -ef awk '/[d]evnode/   print $2 ')
dhclient /dev/eth0

After this, the machine had internet connectivity, and I could upgrade it and install the sysvinit packages from experimental and enable it as the default boot system in Hurd. But before I did that, I set a password on the root user, as ssh is running on the machine it for ssh login to work a password need to be set. Also, note that a bug somewhere in openssh on Hurd block compression from working. Remember to turn that off on the client side. Run these commands as root to upgrade and test the new sysvinit stuff:

cat > /etc/apt/sources.list.d/experimental.list <<EOF
deb http://http.debian.net/debian/ experimental main
EOF
apt-get update
apt-get dist-upgrade
apt-get install -t experimental initscripts sysv-rc sysvinit \
    sysvinit-core sysvinit-utils 
update-alternatives --config runsystem

To reboot after switching boot system, you have to use reboot-hurd instead of just reboot, as there is not yet a sysvinit process able to receive the signals from the normal 'reboot' command. After switching to sysvinit as the boot system, upgrading every package and rebooting, the network come up with DHCP after boot as it should, and the settrans/pkill hack mentioned at the start is no longer needed. But for some strange reason, there are no longer any login prompt in the virtual console, so I logged in using ssh instead. Note that there are some race conditions in Hurd making the boot fail some times. No idea what the cause is, but hope the Hurd porters figure it out. At least Justus said on IRC (#debian-hurd on irc.debian.org) that they are aware of the problem. A way to reduce the impact is to upgrade to the Hurd packages built by Justus by adding this repository to the machine:

cat > /etc/apt/sources.list.d/hurd-ci.list <<EOF
deb http://darnassus.sceen.net/~teythoon/hurd-ci/ sid main
EOF

At the moment the prebuilt virtual machine get some packages from http://ftp.debian-ports.org/debian, because some of the packages in unstable do not yet include the required patches that are lingering in BTS. This is the completely list of "unofficial" packages installed:

# aptitude search '?narrow(?version(CURRENT),?origin(Debian Ports))'
i   emacs                   - GNU Emacs editor (metapackage)
i   gdb                     - GNU Debugger
i   hurd-recommended        - Miscellaneous translators
i   isc-dhcp-client         - ISC DHCP client
i   isc-dhcp-common         - common files used by all the isc-dhcp* packages
i   libc-bin                - Embedded GNU C Library: Binaries
i   libc-dev-bin            - Embedded GNU C Library: Development binaries
i   libc0.3                 - Embedded GNU C Library: Shared libraries
i A libc0.3-dbg             - Embedded GNU C Library: detached debugging symbols
i   libc0.3-dev             - Embedded GNU C Library: Development Libraries and Hea
i   multiarch-support       - Transitional package to ensure multiarch compatibilit
i A x11-common              - X Window System (X.Org) infrastructure
i   xorg                    - X.Org X Window System
i A xserver-xorg            - X.Org X server
i A xserver-xorg-input-all  - X.Org X server -- input driver metapackage
#

All in all, testing hurd has been an interesting experience. :) X.org did not work out of the box and I never took the time to follow the porters instructions to fix it. This time I was interested in the command line stuff.

I want Zoe to do one "extra curricular" activity per term this year. Something dance-related, something gymnastics-related and maybe some other form of sport (I'm thinking soccer). My girlfriend and I were wandering through Westfield Carindale on Saturday, and we happened upon a guy from Infinity Martial Arts touting his wares. I thought I'd suss it out, and they had an introductory offer where the sign-up fee and uniform fee were significantly reduced, and they had a pretty flexible timetable. They were just starting up their East Brisbane location, and the close proximity to home along with the reduced price sealed the deal. Long-term, I'd like for Zoe to learn Tae Kwon Do, for self-defense, but BJJ seemed as good as anything to get her introduced to the idea of martial arts. The class for 2-4 year olds was billed as "Fun and Fitness 4 Kids" so it's really a combination of listening to the instructor, some basic gymnastics-style stuff and a little bit of martial arts. We biked over this morning (going up Hawthorne Road is a slog) and got there in about 15 minutes via the direct route. It's in the upstairs of a gym in the middle of an industrial area, but it was pretty easily accessible by bike. They're still waiting on some of the equipment, so the space was a little spartan. It was just Zoe and I and a mother of two not quite 2 year old twin girls taking a trial class. First up, we got her uniform, and she looked so cute. There were pants and a jacket and a belt. I'm going to have to video the instructor tying up the belt next week so I can learn how to do it the right way. The class started with the kids standing on these flat coloured circles ("mushrooms") and effectively playing "Simon Says" ("instructor says") without being caught out. It was a pretty sneaky way of doing a bunch of warm up exercises like rotating the knees and ankles. Zoe did very well, but there were a few that she just point blank refused to do. Next the instructor set up a bunch of "stations" around the room. The first station involved me crouching in a fetal position with a football and Zoe had to try and tip me over to get the football. That was a load of fun. The next station was a few steps and foam-filled vinyl ramp, and Zoe just had to do a somersault down that. The next station was pretty much the same but taller, and Zoe had to do a "sausage roll" on her side down that. The next station was just a small exercise ball and Zoe had to do some "donkey kicks" on it. The final station involved me waving a couple of cut-off pool noodles at arm's length, and Zoe had to run in covering her head and give me a bear hug. We did a few rotations of these stations. It was heaps of fun. Next, the instructor got a whole bunch of ball pit balls of different colours, and scattered them over the floor, and put a basket at each end of the room. The kids were then instructed to retrieve specific colours as fast as possible. Zoe started out trying to get as many as possible in her arms before returning to the basket, but the idea was to do it one ball at a time. A lot of running back and forth. Finally, we did some actual BJJ (I think). It was called the "sleeping crocodile hold" or something like that. I had to lie on my back, and Zoe had to sneak up to me from my side and grapple me with one arm behind my head and the other around my waist and a knee in my side. I have no interest in Zoe learning mixed martial arts, but this class was so much fun. I was feeling a bit tired this morning before we headed out, but by the end of it I was so pumped. It was just the right combination of daddy/daughter rough and tumble, with a bit of gymnastics and following instruction. I'm pretty certain Zoe enjoyed it. I liked that the instructor stopped for a water break between each activity, so the kids were kept well hydrated throughout. We took the "scenic route" home, because we had no particular time constraints. It was more like 25 minutes and involved the Norman Park Greenway. I was so glad we went that way. It was a beautiful ride that I didn't know existed. Very indirect, it involved going through Woolloongabba, Coorparoo and Norman Park, around the back of Coorparoo State High School along the side of Norman Creek. It was semi-wetland conditions.The only part that was a bit annoying was where Norman Avenue met Wynnum Road. It was quite steep and the green light didn't last very long. I had wanted to do story time at Bulimba Library at 10:30am on Fridays, but I'd rather bike home via the Norman Park Greenway instead, because it's a nice ride. I've since decided that I'll just use the story time at the library during wet weather, when we'd be driving to BJJ anyway, and be able to make it to the library in time after class. I've also got to figure out where I'm going to fit doing some Science into the schedule. Fridays are going to be busy I think. I managed to get Zoe down for a nap by a bit after 12:30pm today. She was pretty knackered after the class (as was I from biking home) so I let her watch a bit of TV while I prepared lunch. She was funny, she saw how sweaty I was when we got home, and suggested I take a shower while she watch some TV. I was a bit unprepared for my first Science Friday, though. I'd been considering going to the Sir Thomas Brisbane Planetarium after BJJ, but I discovered it's closed until February 7 for maintenance and an upgrade, so that cunning plan was thwarted. I used her nap time to order 365 Science Experiments and 50 Dangerous Things (you Should Let Your Children Do). I should get plenty of inspiration out of those two. After she woke up, we did the old "vinegar and sodium bicarbonate" trick as our science experiment. I've finally got a use for my Google Labs Founders' Award lab coat. I need to try and find some child-sized safety glasses. The adult ones barely stay on her little nose. After that, we went for a walk to our local toy store to see if they sold child-sized safety glasses (they didn't) and then Zoe watched a little bit of TV and then we walked to the CityCat to go to Teneriffe to catch a bus to Chinatown. I couldn't have timed it better if I tried. Just as we got to Chinatown, they started doing a thing with the Chinese dragons, and I hoisted Zoe up on my shoulders so she could watch. I wasn't sure how she was going to take it, with all the noise from the drums and the dragons themselves, but she was enthralled. We watched other acts and then my girlfriend joined us, and we went to a Chinese restaurant for dinner. I bought Zoe these training chopsticks a while ago, and she's taken to them like a duck to water. I brought them with us so she could use them at the restaurant, and she ate the biggest dinner I've ever seen her eat. After dinner, we caught the tail end of a procession and had some ice cream. We were starting to leave, and there was another dragon, and Zoe was brave enough to go and touch it on her own, for good luck. We then made our way back to the bus. By the time the bus arrived and got us back to Teneriffe, and a CityCat finally arrived, it was very late, but Zoe was pretty good the whole time. It was probably the latest night she's ever had with me while we've been out on the go, and she only got a bit ratty once we were home. It didn't help that she'd forgotten that she'd put Cowie in a cupboard this morning and we burned some more time tracking her down. Today was a fantastic (and very full) day. I enjoyed it a lot, and I think Zoe did too. Fortunately, Fridays won't always be this full on.

Today was our first expedition to the local Playgroup at the Bulimba Senior Citizen's Centre, an easy 5 minute bike ride from our place. I told Zoe we were going to Playgroup when Sarah dropped her around this morning, and she got very excited and jumped up and down. When Sarah asked her if she knew what Playgroup was, she said "no!" while continuing to jump up and down excitedly. We all laughed. I must have put this on my calendar when I was in Sydney last week, because I had it down for 8:30am, and we rocked up and there wasn't anyone there. A lady arrived to set up for a music class, and she said that the Playgroup didn't usually start until much later, so I figured I'd had a timezone fail and we went for a little bike ride around Bulimba while we waited. The lady who was running the Playgroup arrived at about 9:15am, and so we helped with the set up. Basically lots of toys, a couple of slides and some basic craft stuff. Heaps of the little IKEA round plastic stools. Parents (almost exclusively mothers) started trickling in from 9:30am. I've got a total name overload. We need name tags. Most of the girls seemed to be about 2 and a half, and a lot of the mothers had very young second children. Apparently a mother's group had joined en masse because they all had second children who were crawling, and so meeting in a park wasn't practical any more. Structurally, the first hour or so was just a free for all, and the last half an hour was some songs, a story and some playing under and on a parachute. All in all it was a good morning's activity, and at $20 for the term, nice and cheap. Apparently you're supposed to join Playgroup Queensland as well, for $40, but that takes care of liability insurance and whatnot. If only their website used SSL. Zoe mostly just wanted to play with me. We did a lot of role playing with some dolls, and she checked out the various toys. She loved that there was a stage, and took it upon herself to give an impromptu dance performance for a bit. We also did a little craft activity. She seemed more interested in helping flap the parachute than going under it, but enjoyed having a ride on it when the time came. I've just finished reading How Not to F*** Them Up by Oliver James. It was quite an interesting read, although not particularly applicable for my situation, as it more dealt with the first three years. It was interesting observing a whole bunch of different mothering techniques in action, both in dealing with their toddler children, and how they responded to their second infants. After Playgroup, we biked over to the local toy shop around the corner from home, and grabbed some birthday presents for the birthday party on Saturday and came home for lunch. Chicken quesadillas and some mango (I hadn't realised how much I've missed Queensland mangoes until we moved back, and I'm glad Zoe loves them as well). I got her down for a nap earlier today, and after she woke up, I asked Zoe if she wanted to write on the birthday cards we'd bought for the twins' birthday presents. She said she didn't know how to write her name (which isn't true) and wanted to paint them instead, so we compromised and used some sponge letters to stamp "Z O E" on the cards in paint. Zoe said that Eva's favourite colour was purple and Layla's was pink (or it may have been the other way around), so we stamped one with purple paint and mixed red and white to make something that looked more orange than pink and stamped the other one. Then Zoe wanted to glue the two paper plates together that we'd been using to hold the paint, so we did that, and then she wanted to "hand paint" a cardboard tube, which seemed to involve me directly squirting various colours into her hands and then her "painting" the tube with what was in her hands. The tube quickly ended up a brown/grey colour. All in all, it was a fabulous mess, and fortunately no walls or doors copped any paint on the way to the bathroom to clean up. After that, we cleaned up Zoe's room a bit, which resulted in her going off and playing with her Magna Doodle with Smudge for a bit, and then we played Doctor with her doctor set and then we did some baking. It was a good afternoon, and she only asked to watch TV once, when she first woke up from her nap, and we managed to successfully divert to other much more fun stuff.

Bitcoin is a incredible use of peer to peer communication and encryption, allowing direct and immediate money transfer without any central control. It is sometimes claimed to be ideal for illegal activity, which I believe is quite a long way from the truth. At least I would not conduct illegal money transfers using a system where the details of every transaction are kept forever. This point is investigated in USENIX ;login: from December 2013, in the article "A Fistful of Bitcoins - Characterizing Payments Among Men with No Names" by Sarah Meiklejohn, Marjori Pomarole,Grant Jordan, Kirill Levchenko, Damon McCoy, Geoffrey M. Voelker, and Stefan Savage. They analyse the transaction log in the Bitcoin system, using it to find addresses belong to individuals and organisations and follow the flow of money from both Bitcoin theft and trades on Silk Road to where the money end up. This is how they wrap up their article:

"To demonstrate the usefulness of this type of analysis, we turned our attention to criminal activity. In the Bitcoin economy, criminal activity can appear in a number of forms, such as dealing drugs on Silk Road or simply stealing someone else s bitcoins. We followed the flow of bitcoins out of Silk Road (in particular, from one notorious address) and from a number of highly publicized thefts to see whether we could track the bitcoins to known services. Although some of the thieves attempted to use sophisticated mixing techniques (or possibly mix services) to obscure the flow of bitcoins, for the most part tracking the bitcoins was quite straightforward, and we ultimately saw large quantities of bitcoins flow to a variety of exchanges directly from the point of theft (or the withdrawal from Silk Road). As acknowledged above, following stolen bitcoins to the point at which they are deposited into an exchange does not in itself identify the thief; however, it does enable further de-anonymization in the case in which certain agencies can determine (through, for example, subpoena power) the real-world owner of the account into which the stolen bitcoins were deposited. Because such exchanges seem to serve as chokepoints into and out of the Bitcoin economy (i.e., there are few alternative ways to cash out), we conclude that using Bitcoin for money laundering or other illicit purposes does not (at least at present) seem to be particularly attractive."

These researches are not the first to analyse the Bitcoin transaction log. The 2011 paper "An Analysis of Anonymity in the Bitcoin System" by Fergal Reid and Martin Harrigan is summarized like this:

"Anonymity in Bitcoin, a peer-to-peer electronic currency system, is a complicated issue. Within the system, users are identified by public-keys only. An attacker wishing to de-anonymize its users will attempt to construct the one-to-many mapping between users and public-keys and associate information external to the system with the users. Bitcoin tries to prevent this attack by storing the mapping of a user to his or her public-keys on that user's node only and by allowing each user to generate as many public-keys as required. In this chapter we consider the topological structure of two networks derived from Bitcoin's public transaction history. We show that the two networks have a non-trivial topological structure, provide complementary views of the Bitcoin system and have implications for anonymity. We combine these structures with external information and techniques such as context discovery and flow analysis to investigate an alleged theft of Bitcoins, which, at the time of the theft, had a market value of approximately half a million U.S. dollars."

I hope these references can help kill the urban myth that Bitcoin is anonymous. It isn't really a good fit for illegal activites. Use cash if you need to stay anonymous, at least until regular DNA sampling of notes and coins become the norm. :) As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Another release of my general collection of utility libraries, Autoconf probes, general test cases, and similar things. The major change in this release is the addition of Autoconf probes and portability wrappers for libevent. Now included are Autoconf macros to probe for libevent (mandatory or optional) either via pkg-config or not, following the pattern of other probes in rra-c-util. Also included is a portability wrapper, probes, and some additional glue code that allows programs to be written against the libevent 2.x API and work (mostly) with the libevent 1.4 API, albeit with some missing functionality. This release fixes the systemd unit directory probe to not produce shell errors if pkg-config isn't available on the local system. It also fixes a syntax error when building portable/krb5.h with a C++ compiler. There are multiple improvements to the Kerberos and process TAP add-ons: memory is properly freed in all processes, making exhaustive memory allocation analysis easier; process output is flushed after stopping the process instead of before; the method used to stop processes is cleaner; a memory leak when running via fakeroot has been fixed; and full use is made of the new test_cleanup_register API in C TAP Harness. This release also adds a new message_handlers_reset function that restores all message handlers to their defaults and frees any memory allocated to set different handlers. This again is mostly useful when doing exhaustive memory allocation testing. You can get the latest release from the rra-c-util distribution page.

This is my test harness and C and shell library for writing test suites using the Test Anything Protocol (also used by Perl). The changes in this release aren't that major, but I changed one API in a non-backward-compatible way, so I bumped the major revision. That change is to test_cleanup_register, whose callback functions now take two arguments instead of one. The first, as before, indicates whether the test case succeeded or failed. The second is now true if the callback is running in the primary process (the one in which plan or plan_lazy was called), and false otherwise. Callback functions are now called for all processes, not just the primary one. This allows cleanup of per-process resources, like memory, in the child processes, which allows for easier valgrind analysis. Also in this release, the test harness was modified to reopen standard input to /dev/null when running a set of tests from a list, rather than inheriting standard input from the test driver. The test harness also now cleans up all extraneous copies of open file descriptors used during the setup of standard input, output, and error to avoid leaking harmless but noticeable file descriptors to tests. You can get the latest release from the C TAP Harness distribution page.

The 3D-printing technology has gained quite a big momentum in recent times. Its use is spreading widely even among hobbyists and many sources report daily several (sometimes weird) examples of its usage. The thing I like most, as a Free Software activist, is that several 3D-printing-related free and open source software are already available and quite good, and some printers are even released as free hardware. Unfortunately, Debian is quite lagged behind with regard to the availability of such software in our archive, especially compared to other distributions who already ship much of it. Hopefully this situation will change drastically thanks to the brand new Debian 3D-Printing team, whose purpose is to make Debian a rocking platform to do some serious 3D-printing. The team is bootstrapping right now, and we are looking for volunteers who are willing to help us. Packagers are greatly welcomed, but also triagers, users and passionate people with some experience in the field are extremely valuable. People willing to join the team can take a look at our (in-progress) wiki page and apply on the team page on Alioth. As with any good packaging team, a mailing list is available to share ideas, feel free to subscribe it, and feel free to join #debian-3dprinting on OFTC if you want to hang out with other team members.

Yesterday was the first day of my new routine, except I haven't really established the new routine yet. I think that's going to take a few weeks. It was, however, the first day of Zoe's new routine, she started Kindergarten. "Kindergarten" is a bit of an ambiguous term, even within Australia, it seems. Back when I went to school, there was "pre-school", which was, as best I can determine, what "Kindergarten" is now, except a year later. It was optional, half a week, and the year preceding the start of primary ("elementary") school. I think every other state was already doing an extra year of school (which I think they called "Kindergarten", "Kinder", or something starting with "K") so Queensland blew up "pre-school", created "Prep" and now that's the first year of primary school. Zoe will start "Prep" next year. Then the Queensland education boffins realised that kids were missing out on the stuff that "pre-school" used to offer, so they reintroduced it in the form of "Kindergarten" (by Queensland standards). It's supposed to be play-based, and isn't really "long day care". Zoe does Monday, Tuesday and every other Wednesday, from 8:30am until 2:20pm. I'd only heard about C&K from a friend after moving back to Brisbane, so we were very fortunate to secure a place without having camped out on a waiting list from birth. We got very lucky, and ended up getting multiple offers, but we're pretty thrilled with the one that Zoe got into. It's a relatively new facility, right next to a state school and some bushland. They take the kids on excursions into the bushland, and up to the state school where there's some chickens. The Kindergarten facility itself is really lovely. Anyway, yesterday was Zoe's first day. I biked there yesterday morning (without Zoe) and it took me about 11 minutes. It took me more like 15 minutes to pick her up with the bike trailer. It's uphill on the way there, but not too much of a slog. It's been really great that Zoe's BFF from day care, Megan, also scored a place in the same Kindergarten, as I think this will help the transition for both girls greatly. I was able to stay for an extended period of time to help settle Zoe in, and I was able to leave without her being particularly distressed. I then biked home, running a few errands along the way. I then got stuck into a woodworking project (I'm trying to make a little wooden clothesline for Zoe). It turns out trying to cut a halved joint with a hand saw and chisel is a lot of work. I've managed to aggravate an old rotator cuff injury and the two pieces are a tad too tight, so I'm thinking I'll redo it and cut the joint out with an electric router instead. I probably spent at least a couple of hours faffing around with that. So I banged away with a hammer and chisel for a bit, got a haircut, had a late lunch and next thing I knew it was 2pm and time to head back on the bike to pick up Zoe. She was still asleep on her stretcher when I arrived to pick her up, which wasn't that surprising, since she tends to sleep later and longer at day care. I got a good opportunity to chat with her teacher about her first day and then gently woke her up with a kiss, and we headed home. I introduced Zoe to one of our new neighbours, and after a snack, we drove into Fortitude Valley where I had to pick up some mail from my post office box. When we got back home, I started on dinner while Zoe watched a little bit of TV. I spontaneously invited my cousin over to join us for dinner, and we walked down to the Hawthorne Garage for a babyccino after dinner. It was a good day. I realised that I'm going to have to be super regimented with my time on the days Zoe is in Kindergarten, because 2pm is going to roll around very quickly. "Scarcity brings clarity", as Sergey says...

A new minor / maintenance release RQuantLib 0.3.11 is now on CRAN and in Debian. Like the RcppClassic upload two days ago and the RcppZiggurat and RcppEigen uploads yesterday, this release was motivated at least in part by an upcoming Rcpp release about which I should have more to day soon. A few things got polished and updated, and we added a plugin which should make use via Rcpp Attribute easier. RQuantLib combines (some of) the quantitative analytics of QuantLib with the R statistical computing environment and language. Thanks to CRANberries, there is also a diff to the previous release 0.3.10. Full changelog details, examples and more details about this package are at my RQuantLib page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

It is unfortunate that I've been exposed to several companies that all need the same kind of problems solving, again and again:

• Imaging systems. Quickly.
• Configuring MySQL & Postgres replication and fail-over.
• Configuring local users.
• Solving the problem of distributing usernames/passwords to 500+ client-systems within a team.

A lot of these are solved problems, yet they seem to keep cropping up. I guess the fact I've done some of these things more than once means I'm the local-expert, so that's why I get asked. But still.. There was a time when I thought my Debian Administration site would help solve these kind of problems; that writing documentation would encourage people to do things properly. Certainly it has helped me, and some other people are greatful, but it didn't do enough to help. I'm not sure if the problem is that my documentation is a little ideosyncratic, isn't good enough, or isn't reaching the right kind of people. I suspect a combination of that and scale - You can't walk somebody though the idea of setting up a ten-node database-cluster, they need to suffer, they need to break things, they need to sweat on Christmas Day, at 5AM, as everything goes to hell. Then the next time they'll do it properly. I'd love to take the time to write out recipes in Salt, Ansible, Puppet, Chef, CFengine, Slaughter, whatever, and support them.

Remote. Automated. System management.

Throw in monitoring of metrics, security fixes, and reporting and there's probably a valuable service there. It probably can't happen though, for three main reasons:

• The people that need it don't know they need it. They're fighting fires, they know it is important and they will fix things "soon", but other work takes priority.
• The people that are tempted will baulk at the idea of unknown code from an external source running on their system(s).
• Companies managed by one sysadmin will wonder why they need more help, because "everything is working, right?".

I did recently write some policies for setting up a two-node Master-Slave MySQL setup, with reporting, monitoring, and custom SMS-based alerts. I guess I'm wondering if I can be cheeky and sell the same work twice. ;)