In my previous post, I explained about the age excuse in Britney. In this post, I will cover the slightly more tricky out of date excuse. A simple example of excuse is:

out of date on mipsel: blinken (from 4:4.8.4-1)

In this simple case, Britney simply states that the binary blinken on mipsel has not yet been rebuilt. Generally, the presence of an excuse like this is sufficient to stop testing migration. The exception to this rule is if the architecture has been marked as not keeping up . At the moment we do not have any such architectures. The tricky part of an out of date -excuse is that Britney simply identifies a symptom and not a cause. In the excuse above, it is not possible to determine if the build failed on that architecture or simply has not finished yet. But the part that probably confuses the most, is when Britney lists binaries no longer built from that source:

out of date on i386: libfsoframework0 (from 0.11.0-1.1)

This case actually happens quite often and is (usually) a sign of a transition has been started. The part that confuses many is that the build-state of their package will be Installed (as in It was built successfully and has been uploaded to the archive ). From there they tend to conclude Britney is broken , where The excuse information is a bit unhelpful might be more accurate. Caveats:

1. If the source package has not yet been built on that architecture, these old binaries are interleaved into the regular list of out of date -binaries.
2. The list of excuses are updated twice a day (and there is also a delay between its update and the PTS picking it up). Thus, if the build-state of your package for that architecture has been Installed for less than 24 hours, you may just want double check before jumping to conclusions.

The problem is that the binary still has reverse dependencies in unstable, so it cannot migrate to testing. When the last reverse dependency has been updated, the FTP masters will usually semi-automatically decruft your package and the excuse will disappear. On the other hand, if the binary still has reverse dependencies, you probably started a transition. In that case, your package will be stuck in unstable until all the reverse dependencies are ready (or, at least, rebuilt to use the replacement package). Generally, it is good if the Release Team was involved before you did this (see this page ). In summary, if your package has Out of date on [architecture] in its excuses it can be one (or more) of:

• Your package is not built yet on the architecture. This is a transient problem, wait for the build servers to build your package.
• Your package FTBFS on that architecture. This is usually a permanent problem where you need to investigate why it happened and how to solve it. Sadly, often it can take weeks before someone files a bug against your package for this build failure, so being proactive might be worth your time.
• The binary is no longer built binary. This is usually a transition. If the binary packages listed has no reverse dependencies in sid and testing, then just wait for the FTP masters to decruft your package.

My second #debcraft project is a crocheted cozy for a phone, with a nice Debian swirl on it.
As the previous one this is a very quick project - I made mine in half an hour - and easy also for crochet newbies.
As I've not mastered yet the tapestry technic in crochet, I opted for a less elegant solution and used a surface slip stitch to add the swirl.
The only downside of this approach is that you'll need a bit of practice before crocheting a decent and regular swirl (it took me a couple of tries to get the hang of it).
My phone is a Samsung Galaxy, so this pattern fits a phone of 6cm x 11cm x 1cm, but you can easily adapt it to yours. This pattern is written in US crochet terms: check this page for a US/UK conversion table. What you will need

• black cotton
• red cotton
• a 3 mm crochet hook

Obviously, the crochet hook's size depends on the weight of the yarn you're using. I used a DK/8 ply cotton yarn. Finished size : 11cm x 8cm x 1cm Stitches used :

• chain stitch (ch)
• single crochet (sc)
• surface stitches

Instructions Ch 15 + 1 for turning. Continue in flat rows, working 15 sc each row, plus one ch at the end to turn.
Continue till the desired dimensions, in my case I ended with a rectangle 22cm long.
Now the fun part: with the red yarn, start creating a swirl with the surface stitches.
Here's a good tutorial on how to do these, with step-by-step photos. Now the last part: use the single crochet stitch to join the lateral seams of the cozy as showed in this tutorial. And you're finished! Pretty easy, wasn't it? For any questions feel free to contact me.
And if you are a knitter or a crocheter, did you know that on Ravelry exists the Debian Knitters group?

The Present Extension I ve finally gotten an initial implementation of the Present extension written and running and thought I should write up the current design and status. Present Design The current Present extension consists of two requests:

1. PresentRegion. Puts new bits from a pixmap in a window.
2. PresentSelectInput. Asks for Present events to be delivered.

PresentRegion This request takes a pile of arguments:

 
    PresentRegion
    window: WINDOW
    pixmap: PIXMAP
    valid-area: REGION or None
    update-area: REGION or None
    x-off, y-off: INT16
    target-msc: CARD64
    divisor: CARD64
    remainder: CARD64
    idle-fence: FENCE
 
Errors: Drawable, Pixmap, Match

Provides new content for the specified window, to be made visible at the specified time (defined by target-msc , divisor and remainder ). update-area defines the subset of the window to be updated, or None if the whole window is to be updated. valid-area defines the portion of pixmap which contains valid window contents, or None if the pixmap contains valid contents for the whole window. PresentRegion may use any region of pixmap which contains update-area and which is contained by valid-area . In other words, areas inside update-area will be presented from pixmap , areas outside valid-area will not be presented from pixmap and areas inside valid-area but outside update-area may or may not be presented at the discretion of the X server. x-off and y-off define the location in the window where the 0,0 location of the pixmap will be presented. valid-area and update-area are relative to the pixmap. If target-msc is greater than the current msc for window , the presentation will occur at (or after) the target-msc field. Otherwise, the presentation will occur after the next field where msc % divisor == remainder . If divisor is zero, then the presentation will occur after the current field. idle-fence is triggered when pixmap is no longer in use. This may be at any time following the PresentRegion request, the contents may be immediately copied to another buffer, copied just in time for the vblank interrupt or the pixmap may be used directly for display, in which case it will be busy until some future PresentRegion operation. If window is destroyed before the presentation occurs, then the presentation action will not be completed. PresentRegion holds a reference to pixmap until the presentation occurs, so pixmap may be immediately freed after the request executes, even if that is before the presentation occurs. If idle-fence is destroyed before the presentation occurs, then idle-fence will not be signaled but the presentation will occur normally. PresentSelectInput

 
    PresentSelectInput
    event-id: PRESENTEVENTID
    window: WINDOW
    eventMask: SETofPRESENTEVENT
 
Errors: Window, Value, Match, IDchoice, Access

Selects the set of Present events to be delivered for the specified window and event context. PresentSelectInput can create, modify or delete event contexts. An event context is associated with a specific window; using an existing event context with a different window generates a Match error. If eventContext specifies an existing event context, then if eventMask is empty, PresentSelectInput deletes the specified context, otherwise the specified event context is changed to select a different set of events. If eventContext is an unused XID, then if eventMask is empty no operation is performed. Otherwise, a new event context is created selecting the specified events. Present Extension Events There are three different events for the Present extension:

1. PresentConfigureNotify
2. PresentCompleteNotify
3. PresentRedirectNotify

PresentConfigureNotify This event is moving from the DRI3 extension where it doesn t belong.

 
    PresentConfigureNotify
    type: CARD8         XGE event type (35)
    extension: CARD8        Present extension request number
    length: CARD16          2
    evtype: CARD16          Present_ConfigureNotify
    eventID: PRESENTEVENTID
    window: WINDOW
    x: INT16
    y: INT16
    width: CARD16
    height: CARD16
    off_x: INT16
    off_y: INT16
    pixmap_width: CARD16
    pixmap_height: CARD16
    pixmap_flags: CARD32
 

PresentConfigureNotify events are sent when the window configuration changes if PresentSelectInput has requested it. PresentConfigureNotify events are XGE events and so do not have a unique event type. x and y are the parent-relative location of window . PresentCompleteNotify

 
    PresentCompleteNotify
    type: CARD8         XGE event type (35)
    extension: CARD8        Present extension request number
    length: CARD16          2
    evtype: CARD16          Present_CompleteNotify
    eventID: PRESENTEVENTID
    window: WINDOW
    ust: CARD64
    msc: CARD64
    sbc: CARD64
 

Notify events are delivered when a PresentRegion operation has completed and the specified contents are being displayed. sbc, msc and ust indicate the swap count, frame count and system time of the related PresentRegion request. PresentRedirectNotify This one is not specified yet, but the intent is for it to contain sufficient information for the compositing manager to be able to construct a suitable screen update that includes an application window update. Finishing GLXOMLsync_control At this point, Mesa only exposes the old swapinterval configuration value, it doesn t provide any of the GLXOMLsynccontrol APIs. However, the Present protocol does have the bits necessary to support glXSwapBuffersMscOML in the PresentRegion request. Let s see how the remaining APIs in this GL extension will be supported. glXGetSyncValuesOML This one is easy; it only needs to return the UST/MSC/SBC values from the most recent SwapBuffers request. Those are returned in the PresentCompleteNotify event, so Mesa just needs to capture that event and save the values away for return to the application. We don t need any new Present protocol for this. glXGetMscRateOML This returns the refresh rate of the monitor associated with the specified drawable. RandR exposes all of the necessary data for each monitor, but the monitor each window is going to be synchronized against isn t exposed anywhere, and is effectively implementation-dependent. So, I think the easy thing to do here is to have a Present request which reports which RandR output a window will be synchronized with. glXSwapBuffersMscOML This is the one API which is already directly supported by the Present extension. glXWaitForMscOML This is effectively the same as glXSwapBuffersMscOML, except that it doesn t actually perform a swap. For this, I think we want a new request that generates an X event when the target values are reached, and then have the client block until that event is received. This will avoid blocking other threads using the same X connection. glXWaitForSbcOML Like glXWaitForMscOML, this just needs to trigger an event to be delivered at the right time. Presentation Redirection I m focusing on finishing the above stuff before I start writing the redirection spec and code, but I ve been thinking a bit about it. What we want is for applications that are presenting a new frame within a redirected window to have that presentation delivered directly to the compositing manager instead of having the bits copied to the redirected buffer and have the compositing manager only learn about this when the damage event is received from copying the bits. Once the compositing manager receives notification that there are new bits available for a portion of a window, it can then get those bits onto the screen in one of two ways:

1. Copy them to the back buffer for the whole screen and then present that back buffer to the screen.
2. Present them directly to the screen, bypassing the compositing manager s back buffer entirely.

The first option is what you d do if there were more updates than just a single window; it would update the whole screen at the same time. The second option is what you d do if there was only the one window update to act on, and this would automatically take advantage of page flipping to avoid copying data at all. No need to un-redirect windows for the page flip to work. In both cases, we need to inform the original application both when its pixmap is idle and when the swap actually hits the screen. And we need to make sure the final swap happens when the application requested. For the pixmap idle notification, the current PresentRegion idlefence argument should suffice; we just need to pass the idlefence XID along in the redirection event. Simple. For completion notification, we need to make sure the SBC value for the original window gets incremented and that a PresentCompleteNotify event is delivered. I think that means we want to append a chunk of data to the PresentRegion request so that suitable events can be delivered when the compositing manager PresentRegion occur. I think that just needs to be the window of that original window. To make sure the swap happens at the right time, we just need to have the targetmsc value provided to the compositing manager. I m hoping that having this all event driven is that when a later PresentRegion is redirected that has an *earlier* targetmsc, we can simply queue that as well and things should just work . Current Status I ve got the protocol definitions (both X server and XCB) done, and libxcb supporting the extension. I ve got the X server bits working, but the actual updates are not synchronized to the monitor; they re using OS timers and CopyArea for now, mostly so I can test things without also hacking drivers. Mesa is using the extension, but it only provides the swapinterval value and is not yet supporting the full GLXOMLsynccontrol extension. I ve also got a simple 2D core X application using the extension which is in the shmfd repository in my home directory on freedesktop.org. Availability As always, these bits are already published in my home directory on freedesktop.org in various git repositories.

kadmin-remctl is a wrapper in front of the Kerberos kadmin protocol that exports many operations via remctl and adds some additional synchronization functionality. The primary purpose of this release is to add disallow-svr to the default flags for all newly-created principals. This prohibits obtaining service tickets for the principal, which provides some hardening against brute force attacks. Since the create command is designed for creation of user principals, not service principals, and use of service tickets for user principals is quite obscure and rare in Kerberos, this seems like a better default. This release also relaxes the default allowed principal regex to allow two-character user principals. We mostly prohibit them, but we had some old, disabled ones and at least one of those users returned. This is just a default that can be overridden by other sites. This package, as I say most times when I release it, is badly in need of a complete overhaul, a switch to JSON for the attribute format, conversion to a saner Perl kadmin API, and various other improvements. One of these days.... You can get the latest version from the kadmin-remctl distribution page.

So, writing an email client, how did that turn out? Pretty damn well as it happens. My views continue to range from "email is easy" to "email is hard". But I'm now using my home-grown mail-client for 100% of my personal mail-handling. Writing a mail client did seem a little crazy when I started, but the problem breaks down into a few distinct steps and individually they're not so hard:

• Display a list of folders.
• Display a list of messages.
• Display a single message.

Once you've got that you're almost 80% complete. You're just missing things like "delete", "reply", "compose", and those things are pretty simple to implement. I definitely made the right call in making it scriptable with Lua, because I've been able to write so many functions for working with mail. For example marking messages in all folders matching a regular expression. The code is pretty well structured, and now I've got TAB-completion support on all primitives and user-additions I'm finding it a lot nicer to use. The future? I'm not sure. But right now I'm very happy.

Want to fix that segfault you keep hitting or was reported to you? The first step is to understand the error message you get.

So you have a message like the following:
segfault at bfea3fec ip 080ee07e sp bfea3fa0 error 6

You might already know that ip means instruction pointer and sp means stack pointer and as such the addresses that follow them are the values in those registers. But what does the error number mean?

The error number, or code, actually gives you a better explanation of what the cause of the segfault is. The number's bits are flags describing the error and are architecture-dependent. For x86/x86_64 I just wrote an online converter/decoder that you can use to explain the segfault error code.

As an example, the above error code is explained as:

The cause was a user-mode write resulting in no page being found.

And the common error 4:

The cause was a user-mode read resulting in no page being found.

(also known as a null pointer dereference).

Enjoy.

not that we have a lack of RC bugs; still, I was a bit lazy in the last two weeks. here's what I've worked on:

• #675231 psad: "psad: prompting due to modified conffiles which were not modified by the user"
  propose the next fix for the preinst script, later uploaded with maintainer's permission
• #707535 iptables-dev: "libipq.h includes non-existing linux/netfilter_ipv4/ip_queue.h"
  propose a patch
• #711423 src:python-csb: "python-csb: FTBFS: tries to write to $HOME"
  set HOME for build, upload to DELAYED/2
• #713243 src:libcatalyst-perl: "libcatalyst-perl: FTBFS: tests failed"
  upload new upstream release (pkg-perl)
• #713255 src:libtest-rdf-perl: "libtest-rdf-perl: FTBFS: tests failed"
  manually close already fixed bug (pkg-perl)
• #713374 src:jifty: "jifty: FTBFS: dh_auto_test: make -j1 test returned exit code 2"
  apply patch from upstream git (pkg-perl)
• #713525 src:dvbstreamer: "dvbstreamer: FTBFS: libtool: Version mismatch error."
  add patch from Ubuntu / Logan Rosen to use autoreconf, upload to DELAYED/2
• #713564 src:libopengl-perl: "libopengl-perl: FTBFS: gl_const.h:2913:4: error: 'GL_TYPE_RGBA_FLOAT_ATI' undeclared (first use in this function)"
  test proposed upstream fix, then upload their new release (pkg-perl)
• #713588 src:pylibtiff: "pylibtiff: FTBFS: ndarraytypes.h:681:82: error: operator '<=' has no right operand"
  add patch from upstream svn, upload to DELAYED/1
• #713649 src:gdis: "gdis: FTBFS: gdis.h:269:18: fatal error: glib.h: No such file or directory"
  update bug report with new failure mode
• #713769 src:vdkbuilder2: "vdkbuilder2: FTBFS: ld: vdkb_plugin.o: undefined reference to symbol 'g_module_symbol'"
  apply patch from Ubuntu / Daniel T Chen, upload to DELAYED/2
• #714032 bti: "bti no longer works with Twitter and loses posted tweets"
  upload new upstream release
• #714251 bambam: "bambam crashes on launch due to not finding image in "data" dir."
  tag unreproducible, add some findings + questions
• #714267 iodine: "iodine: FTBFS: undefined reference to symbol '__pthread_register_cancel@@GLIBC_2.3.3'"
  make build-dep on check versioned (which now FTBFS on serveral architectures, leaving iodine unbuildable)
• #714335 bti: "bti: FTBFS on kfreebsd: error: 'EREMOTEIO' undeclared (first use in this function)"
  add patch picked from upstream git

A huge release, the combined 0.9.9 and 0.9.10 releases (I made a mistake releasing) are now tagged and pushed to pypi. It features a number of enhancements and fixes, and is just an absolute thrill to play with. Thanks to the contributors this cycle:

Bob Tolbert Christopher Allan Webber Duncan McGreggor Guillermo Vaya Joe H. Rahme Julien Danjou Konrad Hinsen Morten Linderud Nicolas Dandrimont Ralph Moritz rogererens Thomas Ballinger Tuukka Turto

Outstanding! New features are now being considered for 0.9.11. Thanks!

Someone recently asked me about the Debian Xfce 4.10 status, as apparently I forgot to update this post series. So, as you might have noticed, the full Xfce 4.10 desktop environment is currently in Debian Jessie (current name for testing). All in all, the transition went well and smooth. One of the most regular question I get about Xfce 4.10 is the panel behavior when it comes to the task bar expansion. In xfce4-panel 4.8, when people wanted to have a full side panel with a task bar plugin inside, they added a Windows buttons plugin and configured the panel to 100% length. Then the Windows buttons would expand to occupy all the free space on the panel. It was a special case plugins, as usually other plugins only used a fixed space. Now, in 4.10, this is not the case anymore. Windows button uses a fixed size. And the plugins are left-aligned, which means usually people end up with some space at the far right of the panel. To restore the previous behavior back (which is actually the pre-4.8 behavior, 4.8 was an exception by itself), one needs to add a Separator plugin, then configure it to expand (and optionnally select a transparent handle). Then move it next right to the Windows buttons plugin. Another thing which might be a bit surprising for upgrading users is the change in the Action buttons plugin, which people usually use to logout. In 4.8, by default, it's set to run the logout dialog. In 4.10, by default, it's set to logout directly (but with a confirmation dialog). If you prefer the previous behavior, you can just configure the Action buttons plugin and select the Log out item instead of the Log out one (I know, it's a bit confusing). If you have any question, don't hesitate to reach us by mail or on irc (#debian-xfce on Freenode). Other than that remember that Xfce really needs you help, both in Debian and upstream (and at that point, I'd say *especially* upstream). There's a lot of unmaintained project under the Xfce umbrella, some of them part of the core (like xfce4-power-manager), so if you have some spare time and some C/GTK+ knowledge, feel free to contact the Xfce team on the mailing list.

The next August 16th, Debian will turn 20.
What are you planning to do to to celebrate?
Being a DIY geek, I decided to create some craft projects dedicated to the Universal Operating System. I'll publish them, one per week, on this blog with a detailed tutorial, released under a CC:BY-SA 3.0 license. If you have an original project dedicated to Debian (from cookies and cakes to knitted socks, from beaded earrings to handpainted mugs, from plushies to pottery, from glass arts to lettering and so on) join me!
Post a tutorial about your project on your blog (or if you don't have one, send me a mail and you'll publish it here as guest post) with the hashtag #debcraft and a free (as in freedom) license. Bonus points for project using recycled material :). Ready to start? Below my first one: swirl charms from plastic bottles.

---

This is a little tutorial on how to create swirl charms for keyrings from plastic bottles.
The project is really easy and quick to realize, the materials very common and the result quite pretty.
I found out that there are tons of tutorial in the Internet about how to make jewelry from PET bottles, and I'll surely give them a try (earrings, in particular, seem a good idea).
If you want some inspiration, I suggest you to check out the amazing work of Turkish artist Gulnur Ozdaglar who creates ethereal jewelry and everyday objects from plastic bottles.

"What is PET bottle?
A domesticated bottle kept for companionship or amusement."
Gulnur Ozdaglar

What you will need

• one or more plastic bottles
• metal keyring
• jump rings (I made my own ones with metal wire from a spiral block note)
• red nail polish, or acrylic paint (only for the painted swirl)
• red thread and needle (only for the transparent with red embroidery)
• scissors
• a lighter
• pen and paper (if you want to use a paper swirl as guide while cutting)
• a pin (to punch holes)
• pliers and cutters

In the pic there's also a soda can, because I thought it could be a good material too.
After some tests, I decided that soda cans would probably need some resin to make the edges less sharp and avoid cuts, so I think I'll experiment a bit more with them. Step 1 First of all, you'll need to cut out the swirl from the bottle.
I cut it in the upper part of the bottle, using a paper swirl as guide to cut the swirl shape as regular as I could. Step 2 As the edges of the cut are a bit raw, I decided to lightly melt them with the lighter: be careful as if the plastic melt too much it will ruin our nice swirl. Also, remember to open a window while doing it: it's not a lot of melting but better safe than sorry. Step 3 With a pin or a needle punch a hole in the upper part of the swirl, where the jumpring will be set to connect the swirl with the keychain ring. Now you need to decide which type of swirl do: for the painted one, follow step 4a; for the transparent one with the red embroidery, follow step 4b. Step 4a Paint your swirl red, or of your favourite colour (I experimented a bit here, having lots of nail polish, and made also a purple one). Step 4b With a pin or a needle, punch a series of holes, in the middle of the swirl, following the swirl shape.
Then embroider it using backstitch. Step 5 Add the jumpring, if you have a ready made one.
If not, use a pair of cutters to cut a little bit of metal wire and the pliers to create a ring with it, insert in the hole and close it. Step 6 Connect the jumpring and the metal keyring. Congratulations: you now have a swirl shaped keycharm!

I have attended the SciPy 2013 conference in Austin, Texas. Here are my impressions.

Number one is the fact that the IPython notebook was used by pretty much everyone. I use it a lot myself, but I didn't realize how ubiquitous it has become. It is quickly becoming the standard now. The IPython notebook is using Markdown and in fact it is better than Rest. The way to remember the "[]()" syntax for links is that in regular text you put links into () parentheses, so you do the same in Markdown, and append [] for the text of the link. The other way to remember is that [] feel more serious and thus are used for the text of the link. I stressed several times to +Fernando Perez and +Brian Granger how awesome it would be to have interactive widgets in the notebook. Fortunately that was pretty much preaching to the choir, as that's one of the first things they plan to implement good foundations for and I just can't wait to use that.

It is now clear, that the IPython notebook is the way to store computations that I want to share with other people, or to use it as a "lab notebook" for myself, so that I can remember what exactly I did to obtain the results (for example how exactly I obtained some figures from raw data). In other words --- instead of having sets of scripts and manual bash commands that have to be executed in particular order to do what I want, just use IPython notebook and put everything in there.
Number two is that how big the conference has become since the last time I attended (couple years ago), yet it still has the friendly feeling. Unfortunately, I had to miss a lot of talks, due to scheduling conflicts (there were three parallel sessions), so I look forward to seeing them on video.

+Aaron Meurer and I have done the SymPy tutorial (see the link for videos and other tutorial materials). It's been nice to finally meet +Matthew Rocklin (very active SymPy contributor) in person. He also had an interesting presentation
about symbolic matrices + Lapack code generation. +Jason Moore presented PyDy.
It's been a great pleasure for us to invite +David Li (still a high school student) to attend the conference and give a presentation about his work on sympygamma.com and live.sympy.org.

It was nice to meet the Julia guys, +Jeff Bezanson and +Stefan Karpinski. I contributed the Fortran benchmarks on the Julia's website some time ago, but I had the feeling that a lot of them are quite artificial and not very meaningful. I think Jeff and Stefan confirmed my feeling. Julia seems to have quite interesting type system and multiple dispatch, that SymPy should learn from.

I met the VTK guys +Matthew McCormick and +Pat Marion. One of the keynotes was given by +Will Schroeder from Kitware about publishing. I remember him stressing to manage dependencies well as well as to use BSD like license (as opposed to viral licenses like GPL or LGPL). That opensource has pretty much won (i.e. it is now clear that that is the way to go).

I had great discussions with +Francesc Alted, +Andy Terrel, +Brett Murphy, +Jonathan Rocher, +Eric Jones, +Travis Oliphant, +Mark Wiebe, +Ilan Schnell, +St fan van der Walt, +David Cournapeau, +Anthony Scopatz, +Paul Ivanov, +Michael Droettboom, +Wes McKinney, +Jake Vanderplas, +Kurt Smith, +Aron Ahmadia, +Kyle Mandli, +Benjamin Root and others.


It's also been nice to have a chat with +Jason Vertrees and other guys from Schr dinger.

One other thing that I realized last week at the conference is that pretty much everyone agreed on the fact that NumPy should act as the default way to represent memory (no matter if the array was created in Fortran or other code) and allow manipulations on it. Faster libraries like Blaze or ODIN should then hook themselves up into NumPy using multiple dispatch. Also SymPy would then hook itself up so that it can be used with array operations natively. Currently SymPy does work with NumPy (see our tests for some examples what works), but the solution is a bit fragile (it is not possible to override NumPy behavior, but because NumPy supports general objects, we simply give it SymPy objects and things mostly work).

Similar to this, I would like to create multiple dispatch in SymPy core itself, so that other (faster) libraries for symbolic manipulation can hook themselves up, so that their own (faster) multiplication, expansion or series expansion would get called instead of the SymPy default one implemented in pure Python.

Other blog posts from the conference:

• Aaron's post
• Fernando's post

It's 9 in the evening. Do you know if your backups work? Do you know when you last made a successful backup of all of your data? Do you know whether you can restore from that backup? If not, how well can you sleep? You should verify your backups, and do it regularly, not just when you first set up the backup system. Verification means doing whatever you need to do to ensure all of your precious data has been backed up and can be correctly restored from the backups. The simplest way to do that is to restore all your data, and compare it with your live data, and note any differences. That requires you have enough free disk space to restore everything, but it's the almost the only way to be really sure. It's also a great way to ensure the restoring actually works. If you don't test that, don't be sure you can do it when needed. Some backup software has a verification mode, which doesn't actually write the restored data to disk, but compares it with live data directly. This can save space, and can be a reasonable compromise, but you should test actual restoring as well. You could do that by restoring only some of the live data, perhaps choosing files randomly for each test. If you have the disk space to do a complete restore, doing so is a great way to excercise your disaster recovery process in general. Here's one way of doing it:

• On your main computer, do a backup.
• On a second computer, perhaps borrowed for this, restore all your data, without using your main computer at all.
• Start using the restored data as your live data. Do real work, and do all the things you normally do. Pretend your main computer was eaten by your pet shark.
• If you notice something missing, or being corrupt, or being too old, get the files from your main computer, and fix your backup process so that the next time you won't have that problem.

How often should you do that? That, again, depends on how you feel about your data, and how much you trust your backup tools and processes. If it's really important that you can recover from a disaster, you need to verify more frequently. If data loss is merely inconvenient and not life-changingly disastrous, you can verify less often.

tl; dr version: shutdown works, Debian GNU/Hurd booting using sysvinit:

Loading GNU Mach ...
Loading the Hurd ...
GNU Mach 1.3.99-486
AT386 boot: physical memory map from 0x0 to 0x9f400
AT386 boot: physical memory map from 0x100000 to 0x1fffe000
AT386 boot: physical memory from 0x0 to 0x1fffe000
Enabling FXSR
pcibios_init : BIOS32 Service Directory structure at 0xfcff0
pcibios_init : BIOS32 Service Directory entry at 0xfc7ba
pcibios_init : PCI BIOS revision 2.10 entry at 0xfc78c
Probing PCI hardware.
ide: Intel 82371 PIIX3 (dual FIFO) DMA Bus Mastering IDE
    Controller on PCI bus 0 function 9
ide: BM-DMA feature is not enabled (BIOS), enabling
    ide0: BM-DMA at 0xc100-0xc107
    ide1: BM-DMA at 0xc108-0xc10f
hd0: got CHS=762/128/63 CTL=c8 from BIOS
hd0: QEMU HARDDISK, 3001MB w/256kB Cache, CHS=762/128/63, DMA
hd2: QEMU DVD-ROM, ATAPI CDROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
Floppy drive(s): fd0 is 1.44M
FDC 0 is a S82078B
probing scsi 5/22: Adapteco1542FInvalidiaddresseforishpnt.with 1542.
Invalid address for shpnt with 1542.
probing scsi 18/22:AWesternxDigitalAWD-7000nFailed9initialization of WD-7000 SCSI card!
probing eata on/1f0EATA0:Daddress 0x1f04in3use, skipping probe.
probing eata on 170EATA0: address 0x170 in use, skipping probe.
probing scsi 21/22: Iomega7parport ZIP drive ppa: Version 1.42
ppa: Probing port 03bc
ppa: Probing port 0378
ppa:         SPP port present
ppa:         PS/2 bidirectional port present
ppa: Probing port 0278
done
scsi : 0 hosts.
scsi : detected total.
Partition check (DOS partitions):
 hd0: hd0s1 hd0s2 < hd0s5 >
lpr0: at atbus0, port = 378, spl = 6, pic = 7.
2omultibootxmodulesd/execl$(exec-task=task-create)ne=$ kernel-command-line  --host-priv-port=$ host-port  --device-master-port=$ device-port  --exec-server-task=$ exec-task  -T typed $ root  $(task-create) $(task-resume)               task loaded: ext2fs --readonly --multiboot-command-line=root=device:hd0s1 console=com0 --host-priv-port=1 --device-master-port=2 --exec-server-task=3 -T typed device:hd0s1
task loaded: exec /hurd/exec
start ext2fs: Hurd server bootstrap: ext2fs[device:hd0s1] exec init proc auth
INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
Activating swap...done.
Checking root file system...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
ext2fs_check_if_mount: Can't check if filesystem is mounted due to missing mtab file while determining whether /dev/hd0s1 is mounted.
/dev/hd0s1: clean, 43930/181056 files, 281248/723200 blocks
done.
Cleaning up temporary files... /tmp.
/etc/rcS.d/S06mtab.sh: 48: /etc/rcS.d/S06mtab.sh: cannot open /proc/mounts: No such file
Activating lvm and md swap...(default pager): Already paging to partition hd0s5!
done.
Checking file systems...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
done.
Mounting local filesystems...mount: invalid option -- 'O'
Try  mount --help' or  mount --usage' for more information.
failed.
Activating swapfile swap...(default pager): Already paging to partition hd0s5!
done.
df: Warning: cannot read table of mounted file systems: No such file or directory
Cleaning up temporary files....
Configuring network interfaces...inetutils-ifconfig: invalid arguments
ifup: failed to open pid file /run/network/ifup-/dev/eth0.pid: No such file or directory
Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
Listening on Socket//dev/eth0
Sending on   Socket//dev/eth0
DHCPDISCOVER on /dev/eth0 to 255.255.255.255 port 67 interval 6
DHCPREQUEST on /dev/eth0 to 255.255.255.255 port 67
DHCPOFFER from 10.0.2.2
DHCPACK from 10.0.2.2
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
bound to 10.0.2.15 -- renewal in 38544 seconds.
done.
Cleaning up temporary files....
Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix.
INIT: Entering runlevel: 2
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Starting deferred execution scheduler: atd.
Starting periodic command scheduler: cron.
Starting system message bus: dbusFailed to set socket option"/var/run/dbus/system_bus_socket": Protocol not available.
Starting OpenBSD Secure Shell server: sshd.
GNU 0.3 (debian) (console)
login:

This is the result of:

• A patch by Pino Toscano that adds the -e and --ifexists flags to Hurds swapon: http://lists.gnu.org/archive/html/bug-hurd/2012-01/msg00060.html
• Pino Toscanos runsystem.sysv that starts sysvinits init (see below).
• A patch series to allow setting options of procfs translators at runtime (the hurdish equivalent of "remounting"): http://lists.gnu.org/archive/html/bug-hurd/2013-06/msg00119.html
• A trivial patch to tmpfs, also for updating the options at runtime: http://lists.gnu.org/archive/html/bug-hurd/2013-06/msg00126.html
• A patch series adding --fake and --no-mtab to our mount utility, fixing -oremount with one parameter (e. g. mount -oremount,ro /) and finally fixing a subtle bug in an internal look-up function: http://lists.gnu.org/archive/html/bug-hurd/2013-06/msg00135.html
• A minor tweak to checkroot.sh.
• A minor tweak to mountall.sh. Turns out that there are two init processes at this point, pidof -s takes care of that. Not sure why this does not happen on Linux, maybe it even does? Or it is related to us using inits undocumented -i flag to make it ignore the fact that it isn't pid 1. Needs further investigation.
• Disabling killprocs and sendsigs. See below.
• A minor tweak to mount-functions.sh. Same as last week though.
• Using the same inittab as Debian/Linux uses (we seem to ship our own, though I must admit I'm not sure it comes from, dpkg doesn't know about this file on my installations).
• Using the reboot and halt utilities from the sysvinit package instead of our own.

This yields a surprisingly functional Debian/Hurd system with sysvinit. There are still lot's of loose ends, I populated my status page with them. I spent quite some time in #hurd on freenode and Pino Toscano offered to send me some old notes of his about the sysvinit issue. This turned out to be most helpful as his notes were about plugging sysvinits init into the boot process. This was the next logical thing to do for me, so first thing Monday morning I just put his runsystem.sysv file with some very minor tweaks (my version) into my overlay and had a working system in front of my eyes. That certainly was a nice way to start a week of work :), so many thanks again Pino! For your convenience I've created an overlay with all necessary changes and recompiled binaries. Be careful though, you might want to backup /sbin/reboot first or you won't be able to reboot your system! Of course my tool does that for you, and I've managed to create a reasonably small (253 mb) image thanks to the zerofree tool, the process of compacting images is implemented in hurdtest as the compact subcommand. I also added sysvinit specific tests. So if you want to try hurdtest, download the image and unxz it, my tool expects the image to be in the current working directory and be named image.qcow2. Download and untar an overlay and execute:

% hurdtest -logfile log -sysvinit overlay path/to/second-week-overlay

If you specify -logfile you can execute tail -f path/to/log to see what the VM is printing to the console in real time. The sysvinit specific tests demonstrate that indeed sysvinit is managing the system and that e. g. init 0 or shutdown -r now works. There are at least three major problems that I will need to tackle for this project:

• /bin/init is not running as pid 1.
• kill(-1, SIGTERM) kills essential processes.
• There is no /proc/mounts.

I think the last issue is what I'm going to look at next week. There's information about this in the Hurd wiki and I've spoken with Richard Braun about it and he drilled me to think about the hurdish way to solve this. I think I've got a pretty good picture of how to implement this in my head and am eager to give it a go. This will involve adding a function to Hurds filesystem interface that allows one to query an active translator about his mount options and "source" and about all active and passive translators that are attached to one of its nodes. This is where the fun actually begins. This is changing an integral part of an operating system, but this is okay since it is just another program written in c, operating in userspace. But hey I know that, that's just c and if something goes wrong (and it always does) I fire up gdb and have a look. And mind you, I'm running all this as an unprivileged user, no need to jeopardize the entire system for that. Besides implementing /proc/mounts I will also add -O, --test-opts to mount, and -v, --verbose to swapo n,ff . PS: My blog is still not properly registered to planet.d.o. I asked planet@ about that, but got no response. Any help? I never imagined blogging being so stressful, I care about losing potential readers :/

Review: This Alien Shore, by C.S. Friedman

Publisher:	DAW
Copyright:	1998
Printing:	July 1999
ISBN:	0-88677-799-2
Format:	Mass market
Pages:	564

Jamisia is an orphan taken in by the Shido corporation, but she has a fairly comfortable and happy life in Shido Station. She lives in the care of her tutor, has friends, has something that feels like a relatively normal life. Then the station is attacked and destroyed by a rival corporation and she barely escapes in a one-person lifeboat, sent inexplicably by her tutor with a fake identity to rendezvous with a spaceship heading for the up-and-out. That begins a long and quite confusing flight away from everything she's known, and a discovery that her life is far more complicated than she thought. Meanwhile, far from the backwater of corporate tyranny that Earth has become, a new computer virus is wreaking havoc on Guild outpilots. The Guild navigate the ainniq: a sort of subspace that allows ships to slip between nodes without regard to distance. This is the only acceptable method of interstellar travel now that humans know that the more conventional Hausman Drive causes massive genetic mutations, and only Guerans have the secret of navigating it safely. But a virus of a sophistication no one has seen before appears to be targeting outpilot brains through their implants. This Alien Shore is told in two threads but multiple characters and viewpoints. Jamisia (yes, I stumbled over that name the entire book) spends nearly the whole book running while discovering that her head is a very complicated place for very bad reasons. Meanwhile, various other people (most notably the careful and meticulous Kio Masada) are analyzing and tracing the virus, which is of deep concern to the powers that be in the Guaran-controlled network of human colonies. The threads do come together eventually, helped by a young and much more impulsive hacker named Phoenix who is also on the trail of the virus, but it takes much longer than I expected. The book is a bit of cyberpunk, a bit of space opera, and is stuffed to the brim with world-building ideas. There's always something that feels slightly emotionally off about C.S. Friedman novels. Her characters are a bit too intense, a bit too full of primary colors and inherent natures. They don't do a lot of changing; they're just a little too much purely themselves to feel quite accurate. Sometimes it matters and sometimes it doesn't. This is one of the books where it mostly doesn't, partly because it's full of such an amazing variety of characters, and partly because Friedman does a great job adding situational complexity by playing them off of each other and showing them through each other's eyes. Pyschological realism, though, this isn't, nor would I look to Friedman for that. That's somewhat unfortunate given how central psychology is to this book. The focus that slowly emerges from this novel is the whole-hearted embrace of psychological difference as a force for creativity, power, and perspective. Acceptance, respect for, and integration of difference borders on a religious belief for Guerans, which makes this book one of the most wholeheartedly inclusive stories (at least for psychology) I've read. I loved that message enough to overlook the fact that those psychological differences are not treated with much subtlety. The most central of the differences (a very mild spoiler, but one that's essentially given away on the back cover and that is easy to guess early in the book) is multiple personalities. Sadly, I thought that was also the least successful. Friedman treats the topic respectfully and as a difficult asset, which I liked, but it's bog-standard fictional multiple personalities. The presentation owes much more to popular fiction than to the reality of disassociative identity, and the notes that Friedman hits (and the characteristics that the multiples embody) are predictable. However, her second major psychological idea is simply brilliant. The Guerans have developed a coding system for people to communicate their psychological profile to anyone who sees them, and all of the Gueran characters in the novel actively engage with that system and make extensive use of it. Friedman hints at a great deal of subtlety: a wealth of primary patterns (many of which are described in quite enjoyable animal metaphors as chapter epigraphs), secondary patterns that can be layered underneath, and a complex interaction of base patterns and patterns for daily mood. Here, I think Friedman's mild tendency towards archetypal characters helps, since it makes the psychological coding system clearer for the reader. I wish it had been more central to the story and explained further, although I can see the risk of turning the novel into an RPG sourcebook. (Although, now that I mention it, I would love to read an RPG sourcebook for this universe.) The other point for which I have to give Friedman huge credit is that at least half of the book is about a computer virus, the analysis of that virus, and the hacking culture of her future universe, and at no point during the story did I want to start throwing things. There are no completely unbelievable magic computer programs, the sense of movement of data and virtual presence through computer networks seemed vaguely plausible and less steeped in unrealistic VR metaphors, the sense of aesthetics of code is captured remarkably well for a story of this sort, and Friedman even brings some variety to her computer-savvy characters. There are a few mildly stereotyped young hackers (although even those have a tendency to have more depth than the stereotype), but Friedman also populates her electronic world with the other types you rarely see in these novels: thoughtful, brilliant, but meticulous researchers, competent technicians who don't engage as much on the aesthetic aspects, and even moderately-intelligent users who pay attention to the computers their lives are built around. I could actually believe in both these people and the software they use with only the suspension of disbelief required for any future SF. That's not to say there are no flaws, quibbles, or places where computers probably won't ever work that way. But that's part of what makes me so impressed. Friedman is still using computers and hacking in the direct service of a fairly typical SF plot to achieve specific dramatic ends and to create a mystery. She's not building the story around actual technology; she's staying fairly close to the sort of technology and plots that a lot of cyberpunk and hacking novels have. And yet she manages not to make the computing entirely stupid or unbelievable. It's existence proof that an SF author can pay attention to experts and research without compromising the story they want to tell. I want to take this book and smack Robert Sawyer upside the head with it. This is a great idea book with a heavily computer-driven plot strand that I quite enjoyed. It is, however, not without its flaws. The virus-hunting part of the plot is significantly more interesting than Jamisia's flight, particularly given that she keeps running, and running, and running, encountering various people who seem important and end up not being important at all. It takes more than half the book before anything particularly interesting happens to her; up until then, I was dragging myself through her segments so that I could get back to Masada and the other Guerans. She's also annoyingly dim about some things. I figured out what the deep secret in her brain was within the first four or five chapters (it's entirely obvious from the political structure of Friedman's universe), and I have no idea why it took everyone else so long to figure it out. There is also, as mentioned, the slightly off feeling of the characters that I find typical of Friedman, and a cut-throat power struggle at the heart of the book that just didn't have enough psychological believability to fully engage me. But the ideas are wonderful: technology-driven political structures, corporate-controlled states, ideological clashes, lots of believable cultural interactions, and a neat two-option interstellar travel system (even if the nature of the Hausman Drive damage bears no resemblence to any genetic science known to our reality). And that's on top of a wonderful visual language of psychology and discussions of computer security that I actually enjoyed reading. Friedman's the sort of author who won't be to everyone's tastes. You have to have some tolerance for her tendency to write grand clashes of power between highly flawed, aggressively emotional people. Sometimes you have to put up with rather a lot of setup. But I really liked this novel. I suspect I'm editing out a lot of the boring bits in my memory, but I fondly remember that edited version as a ton of fun. Rating: 8 out of 10

Review: The Odd Clauses, by Jay Wexler

Publisher:	Beacon
Copyright:	2011
ISBN:	0-8070-0089-2
Format:	Trade paperback
Pages:	193

The subtitle of this book summarizes it nicely: Understanding the Constitution through Ten of Its Most Curious Provisions. (The constitution in question is the United States Constitution.) It's a sort of popularization of US constitutional law, but rather than look at the most well-known and controversial parts, it takes a look at some of the odd side points, obscure clauses, or clauses that one rarely thinks about because they rarely appear in the news. This is partly a tour of curiosity aimed at those who like legal trivia, but it also provides, indirectly, a look at how constitutional law is made and debated, and at some of the burning concerns at the time of the writing that have since faded away. Sometimes why the concern has faded is as interesting as the original problem. This is the sort of book whose table of contents provides a lot of insight into what you're getting. It has ten chapters, each devoted to a particular clause or amendment: incompatibility, weights and measures, recess appointments (one of the rare ones that has made the news), original jurisdiction, natural-born citizen (odd, but sadly now not at all obscure), letters of marque and reprisal, titles of nobility, bills of attainder, and the 3rd and 21st amendments. (For those who haven't memorized all the amendments to the US Constitution, the 3rd is the prohibition on quartering soldiers in private homes and the 21st is Prohibition, which here is examined not for its core intent but for the implications for federalism.) As you can see, this is quite a selection, and while the clauses are often obscure, the issues raised by them are still generally quite active. There are some that were intensely controversial at the time of the original debate that are little more than amusing trivia at this point (the prohibition on officers of the United States taking titles of nobility), or have become so much a part of the structure of the government that people rarely given them a second thought (such as the incompatibility clause, which prohibits members of Congress from holding positions in the executive branch). But most of these clauses establish ground rules for the US government that are still quite current today, even if the legal terminology used at the time is no longer familiar. A good example is the clause prohibiting bills of attainder, something that the writers of the US Constitution saw no need to define. A bill of attainder is an act of a legislature declaring a person guilty of a crime and assigning some punishment without trial. Framed that way, it seems like an obvious extension of the separation of powers system that the US government is constructed around, but unlikely to provoke much modern controversy. However, as Wexler explains quite lucidly, there's a reason why this clause is right next to the prohibition on ex post facto laws. While no one would stand for the US Congress declaring guilt on a matter of criminal law, it has written some startlingly specific laws that seem aimed solely at specific individuals or organizations, which is exactly the sort of behavior that this prohibition, interpreted broadly, was intended to prevent. Similarly, one might think that Congress's power to issue letters of marque and reprisal is little more than ground for an interesting history lesson. But that clause is deeply entangled with the checkered history of US declarations of war and controversies over Congressional oversight, and Wexler uses it as a jumping-off point to take a look at the implications for the modern habit of undeclared wars fought by military contractors. This is a short book, so these treatments are necessarily brief. Some of them are also fairly minor; for example, the discussion of the weights and measures clause is mostly an extended rant on the US refusal to adopt the metric system. That's a sentiment with which I entirely agree, but I'm not sure it has much to do with constitutional law. But Wexler is an entertaining and concise writer; even when the discussion is relatively shallow, I enjoyed reading his analysis. (I will warn, though, that he takes a few potshots from a liberal perspective, so one's enjoyment of his humor may to some extent depend on your personal political allegiances.) I wouldn't rush out to find this book, but I found it mildly diverting and occasionally quite informative. The chapter on original jurisdiction, for instance, explained how the Supreme Court handles certain classes of cases that must be filed directly with it, despite not being a trial court. This was a question that I didn't even know I had, and a side system of judicial procedure that turned out to be surprisingly interesting. I suspect The Odd Clauses will be the most interesting to people who have only a news media understanding of the US Constitution. I went through a personal libertarian period, during which I read the constitution in detail and sought out some discussions of its specifics, so quite a bit of this book was familiar ground to me. Others without that study background may find more novelty. But even with that background, I was entertained, and don't regret buying it. One note, though: I bought this book on the recommendation of Kevin Underhill, author Lowering the Bar, which is a masterpiece of dry humor. If you're looking for legal humor, parts of The Odd Clauses are mildly funny, but Lowering the Bar is the sort of site that will have you staying up late reading just one more page of archives and reading bits aloud to anyone nearby. Rating: 7 out of 10

One of the goals of our work at Nebula is making it as easy as possible for someone to set up a private cloud. In an ideal world that would involve being able to just plug in the controller hardware, wire up a rack of servers and turn everything on, but right now there are cases where the default firmware configuration on the servers doesn't match our desired configuration. Plugging a console into individual servers just to set some BIOS options is (based on personal experience) about as much fun as writing a doctoral thesis on the experience of watching paint dry, so it seemed worth trying to find a way to avoid people having to deal with that. Thankfully, it turns out that the industry has come to a similar set of conclusions. Recent Dell hardware lets you use WS-MAN, which makes it easy to do things like enable security features as long as you have an authenticated connection to the iDRAC management system. This actually works out wonderfully - the first time a Dell node sends a PXE request to the Cloud Controller, it can push back some configuration changes, reboot the system and then boot it in a known good configuration. Thanks, Dell![1].

HP's slightly trickier. As far as I've been able to work out, the remotely-available programmatic interfaces only provide configuration interfaces to the iLO device itself, along with a range of chassis-level monitoring. Instead, HP provide a couple of utilities that allow the OS to change values. The first of these is called conrep. Conrep is able to modify BIOS settings, but needs an XML file which tells it which configuration values are at which addresses. That's a bit of a pain. Thankfully it's being replaced with a new tool called hprcu which is able to directly query the firmware in order to figure out which configuration options are available and which values have to be stored where in order to set them. You run hprcu once and it spits out a file containing your existing settings. You modify that, feed it back into hprcu, reboot and you're set.

Sounds pretty ideal. What's the problem? The first is that hprcu is currently only shipped as a 32-bit binary, and right now we don't deploy any 32-bit support code on our nodes. It'd be irritating to have to do so just to configure the firmware. The second is that it works by doing raw port IO, and that's not going to be possible from userspace once we've moved to a UEFI Secure Boot setup.

So, obviously, I've been working on reimplementing it. The first step was to figure out what it was actually doing. The first step was to strace it to see if it was using the kernel IPMI interface. strace showed no accesses to /dev/ipmi*. My next thought was that it could be using some sort of shared memory segment with the iLO hardware, so I used MMIOTrace to dump the memory accesses it made. Turned out there weren't many, and certainly not enough to do anything interesting. Then I went back to the strace logs and saw that it was calling iopl() a bunch, and then I got sad.

iopl() allows a userspace application to change its io privilege levels. The Linux manpage for the iopl() call is amazingly helpful, informing us with a straight face that "This call is necessary to allow 8514-compatible X servers to run under Linux." What it actually does is grant userspace access to the full range of io ports.

So, what's an io port? The x86 architecture (and some others) provide two ways to communicate with hardware. The one that's mostly used these days is to map the hardware into the same address space as memory (memory-mapped io), but x86 has an entirely separate address range that can be used. This is significantly more limited, as only 65,536 addresses are available for the entire system. Further, you can only read or write a maximum of 4 bytes in a single transaction. However, while slower and less generally useful than memory-mapped io, port io has the advantage that it's much simpler to implement. As a result, a lot of the original PC hardware was intended to be accessed via port io, and a bunch of that's still present. Want to reprogram your real-time clock? Port io. Want to read from the legacy keyboard controller? Port io. For low-bandwidth transactions, it's a completely reasonable way to implement hardware.

Applications[2] perform port io by executing in and out instructions. Since these instructions allow you to do things like, say, hit the keyboard controller directly[3], userspace is generally forbidden from calling them. Sufficiently privileged applications can call iopl() to raise their privileges and gain access to the full set of io ports. But, since in and out are CPU-level instructions, the actual port io accesses won't show up in strace. So I had to take another approach.

The correct way of doing this would be to LD_PRELOAD something that intercepted iopl(), didn't call it, let the application perform an in or out instruction, caught the fault, looked at the stack frame, called iopl(), performed the access, dumped debug details, dropped iopl() again and restored state. Because that seemed like a bunch of work, I took advantage of the fact that hprcu had separate in and out functions and used gdb. I told gdb to break on every call to in or out, dump the register state and then continue. Then I hacked up a script to parse the register dumps and tell me where the reads and writes were going. Astonishingly, it worked. And then I read the results and got really sad.

PCI setup is hard, if you're a BIOS. You've got PCI devices with large memory windows and you've got to arrange them all somehow and look you've only got 640K of RAM you can access while you're doing this and come on, seriously? So PC-type systems define a port IO interface to performing PCI configuration. Each PCI device has an address made up of a bus, a device and a function. You take a 32-bit integer, set the top bit to indicate that you want PCI access, put the bus in bits 16-23, the device in bits 11-15, the function in bits 8-10 and the configuration register on that device you want to access in bits 0-7. Then you write that to io port 0xcf8. Reads or writes to io port 0xcfc then read or write from that configuration register on that device. This lets the BIOS tell each PCI device where it's going to live in mmio address space without having to actually perform any mmio. Good work, BIOS developers.

This works fine in the BIOS, because nothing else is running while the BIOS is. It even works fine in the kernel, which uses this approach on older hardware that doesn't provide a memory-mapped mechanism to access PCI configuration registers. But it works really badly if you're doing it in userspace, because it shares a problem with all other indexed accesses. You write an address to 0xcf8. You write a value to 0xcfc. What happens if the kernel writes to 0xcf8 before you write to 0xcfc? Your access goes to some other register on some other device and suddenly you've just mapped a PCI device over the top of RAM and well I sure hope you saved all your work.

So, I wasn't thrilled to discover that hprcu was communicating with the iLO by using 0xcf8/0xcfc port io accesses. Not only was it going to stop working once we started deploying UEFI Secure Boot, it had the potential to cause really annoyingly hard to debug problems. Working out how to reimplement it became something of a priority.

By looking at the XML files it generated, and by following the port IO accesses it was performing, I could figure out pretty much how it worked. Some configuration values are stored in the real-time clock CMOS. These were just done in the standard way - write the address to port 0x70, read or write the value to port 0x71[4]. Other configuration values were stored in NVRAM, with accesses going via PCI. This was a little trickier to figure out, because sometimes different NVRAM addresses went to the same PCI address. I finally figured it out - there's a 48 byte window into NVRAM via the PCI configuration registers, and another register which chooses which set of NVRAM is visible. So, take the NVRAM address, divide by 48, write that value to register 0xa6, take the modulus, add it to 0x80 and write the desired value to that address.

So now I knew enough to be able to take an existing XML file and deploy it. Definitely progress, and I could even add support to the kernel. But I still wanted to know how hprcu generated that XML file in the first place. Running strings over the binary showed a bunch of debug output that it never actually printed, but immediately after the help text it also printed SsLlFf:HhAaTtOoDd. I've lost enough years of my life to this kind of thing to be able to identity that as a getopt format string, and there was that tantalising D option at the end that went entirely unmentioned in the help text. Re-running hprcu with the -D argument gave me huge piles of debug output, including references to a DMI entry and an $RBS table.

DMI is a standard for exposing system information, and one aspect of it is providing data from the firmware to the operating system. The OS can look for a defined signature in a fixed area of memory and then pull out a bunch of data about the hardware, including things like the vendor name, model, serial number, BIOS version and more. Some of these information tables are standardised and some are vendor-specific. hprcu was looking for a vendor-specific table and then scanning for it looking for a known signature. That table turned out to to contain a set of signatures and addresses. The address corresponding to the signature hprcu was contained a bunch of data starting with "$RBS". It also contained a huge quantity of ASCII that looked awfully like the strings in the XML file that hprcu had written. Success!

So, the rest of today was spent on working out the format of this table. This was only marginally less tedious than setting up BIOS settings on 20 servers by hand, but I've made enough progress to be able to figure out how to write a kernel-level driver for this. That's about half-done now - the parsing code is all implemented, I just need to add the sysfs glue and I am nowhere near drunk enough for that right now[5], so for now you're just getting a format description.

To start with, look for a DMI table with a type of 0xe5. This contains several entries of 4 bytes of signature, 8 bytes of address and 4 bytes of length. Find the entry with a signature of "$CRQ" and map the corresponding address. The first four bytes of the mapped region should be "$RBS". The first record is 21 bytes into the file. Each record starts with a single byte representing the type and two bytes representing the record length. Records of type 1 contain an ascii string that starts 8 bytes into the record, and have a further type number 6 bytes into the record that tells you what type of string it is. 0x05 indicates that it's a "Feature string", which defines the overall classification of the following features. 0x06 is an "Option string", which provides a human readable explanation of what this configuration value corresponds to. 0x60 is an "Optional warning string", which tells the user which further configuration may be required before the configuration change takes effect.

All other types appear to be configuration options. Bytes 3 to 6 provide the little-endian name of the configuration option. Bytes 7 and 8 provide a unique numberical identifier for the configuration option. Byte 13 indicates the type. 0x03 is stored in CMOS. 0x04 is stored in NVRAM. 0x05 appears to be some different kind of CMOS store. I haven't figured out the others. There then follows a set of choices. These vary in length depending on the type - 0x03 are 14 bytes long, 0x04 are 6 bytes long, 0x05 are 5 bytes long. Type 0x03 choices have the choice id at byte 0, the CMOS offset at byte 2, the mask at byte 3 and the value at byte 4. Type 0x04 choices have the choice id at byte 0, the nvram offset at bytes 1 and 2, the mask at byte 3 and the value at byte 4. Type 0x05 choices have the choice id at byte 0, the cmos offset at byte 1, the mask at byte 2 and the value at byte 3. There are optional flags following each choice - if the final byte of the choice isn't 0, there then follow 6 bytes of flags. The first four bytes provide the name of a configuration option (little-endian, again), the fifth byte refers to a choice on that option and the sixth byte indicates what kind of flag. 0x1 appears to indicate that the option is mutually exclusive with that other option. Examples include the embedded serial port and virtual serial port options, where it's impossible to map both to the same address at once. A choice can have multiple flags.

After a configuration option, there'll be a set of option string records. In turn, these correspond to the previous choices - if a configuration option had three choices, there'll be three option string records. I haven't figured out whether there's a stronger way of binding the option string records to the configuration choice yet. The other thing I haven't entirely figured out are the details of configuring the platform to perform a cold reboot on the next power cycle, but the io traces for that don't look too bad.

The aim is to provide a kernel driver that exposes all these configuration options via sysfs, including indicating the current value, the available values and letting the user set a new value. In an ideal world we'd have a wonderfully generic interface to this kind of functionality, but I'm (sadly) not sure that that's possible.

And that's how I spent the past two days.

[1] (Thell)
[2] Or, indeed, kernels
[3] And, hence, read your keystrokes
[4] Which has exactly the same problems as the PCI access, in that if you happen to ask the clock what the time is while hprcu is accessing CMOS, at least one of you is going to get very confused
[5] Obviously, I would never write kernel code while drunk. You're certainly not running any of it in your enterprise kernel.

comments

So, regarding my cry for help... I did get several replies and did more research on my own. The TL;DR up to now is "I have a fully functioning device with no input method and my data may well die on it":

• The device is passphrase-protected and encrypted so I can't simply connect an USB cable and use MTP.
• I can't connect a mouse or keyboard as LG, in their endless wisdom, didn't design the USB port with enough power in mind so it can't support USB OTG on its own.
• Google then removed USB OTG support from the Nexus 4's kernel. It's not as if powered USB hubs existed so this is obviously the correct path of action.
• While I can install new programs via Google Play, Android 4.0 and above prevents newly installed programs to start without user interaction.
• LG points towards a third-party service for out-of-warranty repairs and as part of their Terms of Service, you have to forfeit all data as they "always update the software", i.e. they will prolly ship random other devices to you on a regular basis instead of what you sent in.
• The Nexus 4 is running stock Android, locked bootloader and all

The last two options I see are

• Try to find a way to get a custom ROM onto the device with the help of USB cable and physical buttons only without destroying the encrypted data (yeah, right...)
• Try and source a display so I can repair the device myself. But as not even ifixit.com offers a howto or parts... I suspect this may fail.

And I can not even be reached under my normal number as I don't dare turning the device off and/or removing the SIM as that may prevent me from recovering with the running device, somehow.

So my partner and I spent last week in Helsinki, visiting for Midsummer. The place was lovely, albeit "too damn hot"^tm. (Despite my regularly shaved head I'm a pale-skinned redhead covered in freckles. In sustained sun I can be burned in the space of 10-20 minutes. Ouch.) I ate cake, went to a sauna, and all the usual things. I also had personal interviews with all the mosquitos. The only downside to the holiday was the timing:

• Friday Worked.
• Saturday woke up early and flew.
• Time passes, Thorin sits down and starts singing about gold.
• Sunday flew back.
• Monday .. worked.
• Tuesday worked ..

I'm a sysadmin and sometimes that involves out of hours work, so you can shuffle virtual machines around, upgrade processors, and do all the kind of "disruptive" work outside core business hours. Yesterday I was awake for 22 hours and I was working for 14 of them. Today I'm doing nothing that involves a computer other than a token check-in or three to make sure everything is fine. 99% of the time I don't mind working late, starting early, or scheduling an event outside working hours (e.g. last night I worked from 10PM-3AM ). But with the holiday I've definitely been feeling burned. My sleeping is screwed up, and I'm just getting grumpy and stupid. Still there were some highlights and I took some nice photos, met some good people, and learned some more Finnish. I actually tried to find some Finnish instructor(s) here in Edinburgh, upon my return, and was amused to discover that there is an Edinburgh-based Finnish Society. Amused? They cancelled their midsummer event due to poor weather. For the rest of the week I'm going to be very careful to count working hours and do nothing excessive. I've also got to get back into my gym-routine. Still there is good news on the horizon. I get a new computer next Monday. The new machine will run Wheezy, and Awesome. With 8GB RAM I'll stop hitting OOM conditions once I process many photographs, and I'm going to be very dedicated in using revision control for everything. (I've noticed I've gotten lazy and have started storing bookmarks locally again, instead of under revision control which is a bad sign.) Rambling. I woke up early (10:30AM) because "Microsoft Technical Support" called me and told me my computer had a virus ..

Review: River of Stars, by Guy Gavriel Kay

Publisher:	Roc
Copyright:	April 2013
ISBN:	0-451-46497-4
Format:	Hardcover
Pages:	639

In Under Heaven, Guy Gavriel Kay applied his style of slightly fantastic alternate history to Tang Dynasty China and wrote possibly the best book of his career. In River of Stars, he takes the unusual step of revisiting one of his historical worlds, something I don't think he's done explicitly apart from the Sarantime Mosaic duology. River of Stars is set in the same land of Kitai, Kay's alternate China, four centuries later. It's a stand-alone novel, not a sequel, but the effects of the events of the earlier book are deeply felt on the structure of society. Under Heaven was one of my favorite books of all time, so you can imagine how eagerly I've been awaiting this book ever since it was announced. It starts, once again, in the distant regions of Kitai, this time with a boy. Ren Daiyan is the second son of a minor clerk in a backwater town with a very atypical dream for someone of his social rank: to win back the lost provinces of Kitai from the Xiaolu and restore the glory of the empire as the center of the world. One sees little hope for this dream in a world where no one with any choice in the matter would let their sons join the discredited and suspected army. And, as River of Stars opens, it's become much less likely since the town's only teacher has closed his school and left. As with Under Heaven, the unfolding of Daiyan's life is full of surprises, all of which should be enjoyed in the flow of the story. Here, I think the cover flap even gives away too much information, despite only mentioning events at the start of the book, since the incident that it recounts has much more impact without being spoiled. I recommend avoiding any plot summary and just reading, and therefore will avoid giving any more details. River of Stars, in another echo of its predecessor novel, has a second, female protagonist. Lin Shan is the daughter of a scholar on the edges of imperial court life, a man who has quietly but persistently refused to educate her differently than he would have educated a son. This matters in the time of this novel. Kitai has turned more conservative, turned away from the roles of women in public life seen (in their limited ways) in Under Heaven, and has pressed women into new roles of formal modesty. Shan has been set up for a difficult life. But she's also brilliant, careful, and canny, making use of opportunties, and manages to make several extraordinary friends within the power structures of the court. It's impossible not to compare this novel to Under Heaven, but that's an unfair standard to which to hold any novel. So I will say up front that, at least for me, River of Stars does not capture quite the same magic. Shen Tai was a brilliant protagonist, and his unwanted journey into the circles of power was the perfect palette on which to mix personal integrity with sweeping politics. It used history as a backdrop for a novel that was always grounded in Shen Tai's modest ambitions and quiet integrity. Neither Daiyan nor Shen are that character, nor is River of Stars that book. This novel is far more political from the start, concerned directly with issues of governance, court intrigue, and military maneuvers. Even with Shen to ground the story, the personal is the political and the events of history have been moved to center stage. I've seen Kay approach a history both ways, and I prefer when he uses history as backdrop rather than focus. I find it easier to identify and celebrate with the protagonists when their journies are less caught up in grand events. When, as here, the personal becomes the political, it creates a darker tone. The stakes are higher, and there is little separable individual triumph to blunt the tragic moments. That said, Kay remains at the height of his form. Once again, this novel contains some of the best conversations I have ever read. I'm not sure what Kay is doing that other novelists aren't. It may be that he is discarding some realism in favor of presenting the purified essence of sharp and dangerous exchanges (more on that in a bit). But there are several conversations here that read like the distilled essence of a high-stakes, insightful exchange between fearsomely intelligent adversaries: steeped in strategy, carefully managed for effect, and full of thrilling rhetorical tightropes. There are also moments that literally sent shivers up my spine: moments when the story changes, and lives change, in ways that I was not expecting at all. The story is beautifully stage-managed, imbuing every moment with a feeling of a broader resonance. I do have to warn, as with all Kay but particularly recent writing, that this is done openly and explicitly via an omniscient and very obvious narrative voice. Kay writes in the style of high fantasy in which the narrator is nearly a character in the story, one who foreshadows events and shares observations directly with the reader. Some people dislike that style inherently, and if so, Kay may not be for you. I love it when it's done well, and I don't think anyone does it better than Kay. I feel like I'm in the hands of a master storyteller who is carefully crafting the story to preserve surprise and suspense and who knows exactly where he wants the story to go. That's fitting, since one of the major themes of this book is stories and legends, and how they are shaped by events but then come to shape events in turn. River of Stars is partly about the construction of stories in the way that Under Heaven was partly about poetry. Some of the characters try to control the setting and subject matter, some use stories as a way to focus their personal motivation, and some use them to define their lives. But stories also shape politics because they can define popular reaction and the limits of the possible. This theme is subtle through much of the book, but the more I thought about it the more I noticed it woven through the story. Unfortunately, and very unusually for one of Kay's novels, the journey was intense but the destination was mildly unsatisfying. I don't want to give away any detail of the plot, so this is necessarily vague, but while I intellectually appreciate the themes that I think Kay was exploring, I think River of Stars lacks the powerful catharsis that I normally get from his novels. This for me returns to the focus on the politics over the personal (which is in some ways another of the themes of the novel). The catharsis in a novel for me begins and ends with the characters, and these characters didn't have quite enough independence from the broader sweep of events. Another theme of the novel is the degree to which every character is operating within tight constraints. This also showed up in Under Heaven, but Under Heaven was full of characters who subverted and manipulated those constraints. In River of Stars, they are more often suffocating. Shen is my favorite character, and I think the strongest character in the novel, because she manages to work around the most constraints and has the most personal story. But the roles into which she's forced remain smothering and push her out of the center of the action. Her life is full of lovely details (I particularly liked her domestic situation, which shows a family model that I think was far more common in reality than it has been in fiction and one that I would like to see portrayed far more often), but she simply doesn't have realistic room, within the society portrayed here, to maneuver to the catharsis I was looking for. One other, minor note: as in Tigana, but more blatantly, unusual sexual desires make an appearance linked with the closest thing this story has to a villain. (In this case, femdom BDSM, although shown so glancingly that I suspect some readers will not pick up on what's being described.) It was neither unrealistic nor out of character, but there is a moment of nastiness that caught me by surprise and bothered me. I hope eventually to see alternative sexuality show up in a Kay novel without the negative linkages. It's not quite characterization by stereotype (I think Kay was going for a deeper point about the constrained roles available to women), but it's close enough to not work for me. I'm spending a lot of words on what didn't quite work for me, but that's because it's complex to explain, not because this is a bad story. Rather the opposite: it's an excellent novel that I didn't enjoy quite as much as its predecessor for reasons that are very individual to me. It's quite likely that those with different narrative preferences or different tolerances for certain emotional tones in novels will have a far different reaction. The quality of the writing is quite high, provided that you don't mind Kay's intrusive narrator, and several moments in this novel made me catch my breath in wonder. If you like Kay, read this; it's very much worth your time. If you're new to Kay, I would start with Under Heaven, which is near the top of my list of novels to recommend to anyone. Then read this, since seeing how the world develops and how the events of the previous novel change and shape subsequent politics is fascinating. It's not quite at the same level, but it's very much worth reading on its own terms. Rating: 9 out of 10