As previously mentioned I was looking to package pwsafe for Wheezy, as this is one of the few tools that I rely upon which isn't present. There are now packages available, with the source on github. I've also been doing some minor scripting because I've run into a few common problems recently:

run-parts

run-parts is a simple utility which will run every executable in a directory, more or less. In Debian-land run-parts is the mechanism for /etc/cron.daily and /etc/cron.hourly - and that is where I've had problems recently. Imagine you run a backup via cron.daily. Further imagine that you run a post-backup rsync and that this might take many many hours. If your backup takes >=24 hours you're screwed. To that end I've patched my run-parts tool to alert and exit if a prior invocation is still running.

silent-run

I think everybody has this script - hide all output when running a command, unless the command fails. Looking today I see chronic from Joey's excellent moreutils does this. D'oh.

I think I've done more, but I cannot remember. In conclusion software is both easy and hard - easy because these two trivial changes were within my reach, but hard because years after encountering GNU/Linux we still have to add in the missing pieces. Still could be worse, I spent four/five hours yesterday evening fighting with MS-SQL server, and that is time I'm never going to get back.

DRI3K First Steps Here s an update on DRI3000. I ll start by describing what I ve managed to get working and then summarize discussions that happened on the xorg-devel mailing list. Private Back Buffers One of the big goals for DRI3000 is to finish the job of moving buffer management out of the X server and into applications. The only thing still allocated by DRI2 in the X server are back buffers; everything else moved to the client side. Yes, I know, this breaks the GLX requirement for sharing buffers between applications, but we just don t care anymore. As a quick hack, I figured out how to do this with DRI2 today allocate our back buffers separately by creating X pixmaps for them, and then using the existing DRI2GetBuffersWithFormat request to get a GEM handle for them. Of course, now that all I ve got is a pixmap, I can t use the existing DRI2 swap buffer support, so for now I m just using CopyArea to get stuff on the screen. But, that works fine, as long as you don t care about synchronization. Handling Window Resize The biggest pain in DRI2 has been dealing with window resize. When the window resizes in the X server, a new back buffer is allocated and the old one discarded. An event is delivered to invalidate the old back buffer, but anything done between the time the back buffer is discarded and when the application responds to the event is lost. You can easily see this with any GL application today resize the window and you ll see occasional black frames. By allocating the back buffer in the application, the application handles the resize within GL; at some point in the rendering process the resize is discovered, and GL creates a new buffer, copies the existing data over, and continues rendering. So, the rendered data are never lost, and every frame gets displayed on the screen (although, perhaps at the wrong size). The puzzle here was how to tell that the window was resized. Ideally, we d have the application tell us when it received the X configure notify event and was drawing the frame at the new size. We thought of a cute hack that might do this; track GL calls to change the viewport and make sure the back buffer could hold the viewport contents. In theory, the application would receive the X configure notify event, change the viewport and render at the new size. Tracking the viewport settings for an entire frame and constructing their bounding box should describe the size of the window; at least it should describe the intended size of the window. There s at least one serious problem with this plan applications may well call glClear before calling glViewport, and as glClear does not use the current viewport, instead clearing the whole window, we couldn t use the viewport as an indication of the current window size. However, what this exercise did lead us to realize was that we don t care what size the window actually is, we only care what size the application thinks it is. More accurately, the GL library just needs to be aware of any window configuration changes before the application, so that it will construct a buffer that is not older than the application knowledge of the window size. I came up with two possible mechanisms here; the first was to construct a shared memory block between application and X server where the X server would store window configuration changes and signal the application by incrementing a sequence number in the shared page; the GL library would simply look at the sequence number and reallocate buffers when it changed. The problem with the shared memory plan was that it wouldn t work across the network, and we have a future project in mind to replace GLX indirect rendering with local direct rendering and PutImage which still needs accurate window size tracking. More about that project in a future post though X Events to the Rescue So, I decided to just have the X server send me events when the window size changed. I could simply use the existing X configure notify events, but that would require a huge infrastructure change in the application so that my GL library could get those events and have the application also see them. Not knowing what the application is up to, we d have to track every ChangeWindowAttributes call and make sure the event_mask included the right bits. Ick. Fortunately, there s another reason to use a new event we need more information than is provided in the ConfigureNotify event; as you know, the Swap extension wants to have applications draw their content within a larger buffer that can have the window decorations placed around it to avoid a copy from back buffer to window buffer. So, our new ConfigureNotify event would also contain that information. Making sure that ConfigureNotify event is delivered before the core ConfigureNotify event ensures that the GL library should always be able to know about window size changes before the application. Splitting the XCB Event Stream Ok, so I ve got these new events coming from the X server. I don t want the application to have to receive them and hand them down to the GL library; that would mean changing every application on the planet, something which doesn t seem very likely at all. Xlib does this kind of thing by allowing applications to stick themselves into the middle of the event processing code with a callback to filter out the events they re interested in before they hit the main event queue. That s how DRI2 captures Invalidate events, and it works , but using callbacks from the middle of the X event processing code creates all kinds of locking nightmares. As discussed above, I don t care when GL sees the configure events, as long as it gets them before the application finds about about the window size change. So, we don t need to synchronously handle these events, we just need to be able to know they ve arrived and then handle them on the next call to a GL drawing function. What I ve created as a prototype is the ability to identify specific events and place them in a separate event queue, and when events are placed in that event queue, to bump a sequence number so that the application can quickly identify that there s something to process. Making the Event Mask Per-API Instead of Per-Client The problem described above about using the core ConfigureNotify events made me think about how to manage multiple APIs all wanting to track window configuration. For core events, the selection of which events to receive is all based on the client; each client has a single event mask, and each client receives one copy of each event. Monolithic applications work fine with this model; there s one place in the application selecting for events and one place processing them. However, modern applications end up using different APIs for 3D, 2D and media. Getting those libraries to cooperate and use a common API for event management seems pretty intractable. Making the X server treat each API as a separate entity seemed a whole lot easier; if two APIs want events, just have them register separately and deliver two events flagged for the separate APIs. So, the new DRI3 configure notify events are created with their own XID to identify the client-side owner of the event. Within the X server, this required a tiny change; we already needed to allocate an XID for each event selection so that it could be automatically cleaned up when the client exited, so the only change was to use the one provided by the client instead of allocating one in the server. On the wire, the event includes this new XID so that the library can use it to sort out which event queue to stick the event in using the new XCB event stream splitting code. Current Status The above section describes the work that I ve got running; with it, I can run GL applications and have them correctly track window size changes without losing a frame. It s all available on the dri3 branches of my various repositories for xcb proto, libxcb, dri3proto and the X server. Future Directions The first obvious change needed is to move the configuration events from the DRI3 extension to the as-yet-unspecified new Swap extension (which I may rename as Present , as in please present this pixmap in this window ). That s because they aren t related to direct rendering, but rather to tracking window sizes for off-screen rendering, either direct, indirect or even with the CPU to memory. DRI3 and Fences Right now, I m not synchronizing the direct rendering with the CopyArea call; that means the X server will end up with essentially random contents as the application may be mid-way through the next frame before it processes the CopyArea. A simple XSync call would suffice to fix that, but I want a more efficient way of doing this. With the current Linux DRI kernel APIs, it is sufficient to serialize calls that post rendering requests to the kernel to ensure that the rendering requests are themselves serialized. So, all I need to do is have the application wait until the X server has sent the CopyArea request down to the kernel. I could do that by having the X server send me an X event, but I think there s a better way that will extend to systems that don t offer the kernel serialization guarantee. James Jones and Aaron Plattner put together a proposal to add Fences to the X Sync extension. In the X world, those offer a method to serialize rendering between two X applications, but of course the real goal is to expose those fences to GL applications through the various GL sync extensions (including GLARBsync and GLNVfence). With the current Linux DRI implementation, I think it would be pretty easy to implement these fences using pthread semaphores in a block of memory shared between the server and application. That would be DRI-specific; other direct rendering interfaces would use alternate means to share the fences between X server and application. Swap/Present The Second Extension By simply using CopyArea for my application presentation step, I think I ve neatly split this problem into manageable pieces. Once I ve got the DRI3 piece working, I ll move on to fixing the presentation issue. By making that depend solely on existing core Pixmap objects as the source of data to present, I can develop that without any reference to DRI. This will make the extension useful to existing X applications that currently have only CopyArea for this operation. Presentation of application contents occurs in two phases; the first is to identify which objects are involved in the presentation. The second is to perform the presentation operation, either using CopyArea, or by swapping pages or the entire frame buffer. For offscreen objects, these can occur at the same time. For onscreen, the presentation will likely be synchronized with the scanout engine. The second form will mean that the Fences that mark when the presentation has occurred will need to signaled only once the operation completes. A CopyArea operation means that the source pixmap is ready immediately after the Copy has completed. Doing the presentation by using the source pixmap as the new front buffer means that the source pixmap doesn t become ready until after the next swap completes. What I don t know now is whether we ll need to report up-front whether the presentation will involve a copy or a swap. At this point, I don t think so the application will need two back buffers in all cases to avoid blocking between the presentation request and the presentation execution. Yes, it could use a fence for this, but that still sticks a bubble in the 3D hardware where it s blocked waiting for vblank instead of starting on the next frame immediately. Plan of Attack Right now, I m working on finishing up the DRI3 piece:

• Replace the DRI2 buffer allocation kludge with actual local buffer allocation, mapping them into pixmaps using FD passing.
• Replace the DRI2 authentication scheme with having the X server open the DRI object, preparing it for rendering and passing it back to the application.
• Working on the XCB pieces to get the split event-queue stuff landed upstream.
• Implementing the Fencing stuff to correctly serialize access to the pixmap.

The first three seem fairly straight forward. The fencing stuff will involve working with James and Aaron to integrate their XSync changes into the server. After that, I ll start working on the presentation piece. Foremost there is figuring out the right name for this new extension; I started with the name Swap as that s the GL call it implements. However, Swap is quite misleading as to the actual functionality; a name more like Present might provide a better indication of what it actually does. Of course, Present is both a verb and a noun, with very different connotations. Suggestions on this most complicated part of the project are welcome!

Paul Wise poked me this morning about uploading fatrace ( file access trace , see the original announcement for details) to Debian, thanks for the reminder! So I filed an Intent To Package, and will upload it in a few days, unless some discussion evolves. I also took the opportunity to do some modernization: The power-usage-report script now uses the current PowerTop 2.x instead of the old 1.13, uses Python 3 now, and includes the process device activity in the report. I released this as 0.5. The actual fatrace binary didn t change its behaviour, it just got some code optimizations; thanks to Yann Droneaud for those.

I need to write up new Kindle books, of which there are now quite a few due to various sales plus the Hugo nominee slate, but I got another set of paper books and they're sitting in front of me. So here's a list. Saladin Ahmed Throne of the Crescent Moon (sff)
Elizabeth Bear Shattered Pillars (sff)
Ta-Nehisi Coates The Beautiful Struggle (non-fiction)
Guy Gavriel Kay River of Stars (sff)
Jenny Larson Let's Pretend This Never Happened (non-fiction)
Domenica Ruta With or Without You (non-fiction)
Jay Wexler The Odd Clauses (non-fiction) Ahmed's book is the remaining Hugo nominee that I didn't already pick up. I'm delighted to see the diversity on the Hugo and particularly the Nebula slate this year, and I'm curious to see the spin that Ahmed brings to the epic fantasy genre. Bear's book is the sequel to Range of Ghosts, another book I'm very much looking forward to reading to but haven't yet. I'm rather behind on reading Bear's work right now. Let's Pretend This Never Happened, With or Without You, and The Beautiful Struggle are all memoirs, of varying degrees of seriousness. I've gotten hooked on Coates's writing at The Atlantic, and I highly recommend it if you've not yet seen it. Domenica Ruta's memoir is the latest Indiespensible selection. Jay Wexler's book was recommended by Lowering the Bar, a legal humor blog whose entire archives I'm slowly reading. But the highlight of this order is Kay's River of Stars, which is a sequel of sorts to one of my favorite books ever. This is probably the book I'll read during my next vacation.

The differences between how one fuels one's car are quite pronounced, between California and Australia. Firstly, if you're paying with plastic, it's a given that you can pay at the pump. I could count on my hands the number of times I've had to walk into a gas station to pay for gas in the US. Having a small child, I was not looking forward to having to either leave her in the car so I could pay for my petrol, or having to deal with all of the rigmarole of getting her out of her car seat, just so she can accompany me inside the petrol station to make a brief transaction and then have to get her back into her car seat again. Not to mention how it drags out the whole process. Yesterday I had to wait for a pump while everyone leaves their car, queues inside to pay a single cashier, and then returns to their car and drives away. It'd be an interesting Productivity Commission report to see how much time is lost, just so people can be tempted by the high-margin items inside. Then there's pumping the petrol. California, being all hippy, requires all the fuel nozzles to have these fandangled "vapour recovery" things, which basically fit over the part of the pump that goes inside the fuel tank and does some sort of, well, vapour recovery. The upside, you're not sniffing fumes while you're pumping your petrol. The other fabulous thing about Californian fuel pumps is you can lock the handle down, so you don't have to stand there like a shag on a rock squeezing the handle while a $100 trickles into your car. You can get back in your car and listen to the radio. Or clean your windscreen. Or entertain your kid. I'd love to know why Australian pumps don't lock on any more. I have memories from my early childhood of them locking on. So yeah, I think Australians lose out quite badly when it comes to the petrol station experience. I was pleased to discover that the Woolworths branded Caltex petrol stations seem to have some sort of pay at the pump infrastructure, it just requires you to have their specific credit card or something. I need to do more research, because if I can pay at the pump, I will.

Keith and I have discovered a change in the behavior of the protection circuits integrated in the LiPo batteries we sell for use with Altus Metrum products that poses a risk for our customers. This post is meant to document what we now know, communicate changes we're planning as a result, and explain what we think flyers of our existing electronics and batteries may want to do to maximize their chances of successful flights. Background Choosing batteries and designing pyro circuits for high power rocketry avionics involves a variety of trade-offs. Reliability is the highest concern, both because nobody wants to lose an airframe due to a failed pyro event, and also because airframes recovering anomalously have safety implications. But we also care about other factors including cost and weight, and usually want to minimize the complexity of both the electronics and the overall installation. The objective of a pyro circuit is to dump enough energy rapidly into an electric match to cause it to catch fire. We need batteries both to power the electronics that decide when to fire the charge, and to provide the energy that actually ignites the match. The two most common battery types seen in the rocketry hobby are alkaline cells, often the nominal 9V rectangular variety, and rechargeable batteries based on Lithium Polymer (LiPo) chemistry. LiPo cells are 3.7 volts per cell nominal voltage, are very light, and have a high energy density. LiPo capacity is measured in units of current times time, so an 850mAh cell should be able to deliver 850 milliamps for an hour. The battery industry also uses something called a "C rate" to describe how fast the battery can be usefully discharged, wich is a multiplier relative to the battery capacity. So a battery with 850mAh capacity and a "2C" rating can deliver current at a sustained rate of 1700mA until discharged, while the same capacity at a "5C" rating can deliver 4250mA. By comparison, most 9V alkaline batteries are actually composed of 6 individual 1.5V cells enclosed in a wrapper. It's hard to get hard numbers for capacity and discharge rate, since in an alkaline cell the two are not independent, and the discharge rate is related to the volume of each individual cell. The data sheet for an Energizer 522 shows just over 600mAh at a 25mA discharge rate, dropping to about 300mAh at a 500mA discharge rate. Importantly for use in pyro circuits, LiPo cells have a very low source impedance, which means they can source immense amounts of current. It's not unusual for cells in the 1000mAh range to have ratings in excess of 30C! Because this rapid discharge ability can pose a risk of fire, it's common for LiPo cells to come with a "protection board" integrated into the battery assembly that is designed to limit the current to some rate such as 2C continuous duty. In large airframes, or projects that involve staging, air-starts, or other complex pyro event sequences, the most reliable approach will always be to use separate batteries for the control electronics and the source of pyro firing power. In the limit, having separate pyro batteries for each pyro charge with the control electronics only providing the switching to connect the batteries to the charges could even make sense. But for most airframes, this is overkill, and the increases in mass, volume, and wiring complexity just don't make sense. The challenge, then, is how to design electronics that will robustly initiate pyro events without negatively affecting the rest of the electronics when operating from a single shared battery. Altus Metrum Pyro Circuits The very first prototypes of TeleMetrum were designed to use a single-cell LiPo battery, and had an on-board 100mA charging circuit. Because we needed 5 volts to power the accelerometer, we had a small switching regulator that up-converted the LiPo voltage, and we used some of that regulator's output to charge a 1000uF capacitor. The pyro circuit used Fairchild FDN335N N-channel MOSFET switches in a low-side switching configuration to dump the energy stored in the capacitor through an attached ematch. Those FETs had an on resistance of under a tenth of an ohm in our operating conditions. The circuit worked very reliably, but the 1000uF electrolytic cap was huge and we struggled with the mechanics of such a large part hanging off the board... It turns out that 3.7 volts is way more than enough to fire a typical low-current e-match or equivalents like the Quest Q2G2 igniters. In fact, bench testing with a good digital oscilloscope showed that a typical e-match with resistance of 1.3-1.8 ohms will fire in approximately 13 microseconds when hit with the nominal 3.7 volts from a LiPo. So, starting with our v0.2 boards, we switched to using the LiPo battery voltage directly to fire the pyro charges, eliminating the clunky electrolytic capacitor entirely. We also switched to the Fairchild FDS9926A dual N-channel MOSFET whose specs are better in all regards for our application. The on resistance is down around 40 milli-ohms in our circuit, such that the current ratings are much higher (FET current limits are primarily driven by how much heat is generated due to current flowing through the channel's on resistance). Because using the LiPo voltage directly means we're effectively temporarily putting a very low resistance across the battery during the pyro events, the input voltage to the voltage regulator gets pulled down. To ensure the processor could "ride through" these events, we added a 100uF surface mount bulk capacitor on the 3.3 volt regulated voltage rail, which bench testing demonstrated was more than sufficient to maintain processor operation through pyro events. And that is essentially the same pyro circuit on all the boards, both TeleMetrum and TeleMini, that we have shipped to date. What's Changed The LiPo batteries we source and sell with our electronics come with a protection board and cable terminated in a 2-pin, 2mm "JST" connector. The specs on the protection board have always been "2C continuous", but we observed the ability to source much higher currents for short periods such as the 13 micro-seconds or so required to fire an e-match. Thus these protection circuits seemed just fine .. we could fire e-matches with a burst of current but were protected against short circuits in the wiring or our boards by the 2C continuous limit. Unfortunately, the most recent batch of batteries we sourced seem to have a much "twitchier" protection circuit. We can draw more than 2C for short bursts, but not as much as with prior boards, and not for as long an interval. With a 1 ohm power resistor on the pyro terminals of one of our boards, we get about 9 milliseconds of power before the protection circuit cuts in and shuts the battery down. The power stays down until all load is removed, which at least means the board is turned off and back on again, and in some cases could even mean the battery is unplugged and re-plugged since we draw trace current to keep the GPS memory alive even when the power is turned off, and at least some of the new batteries see that as enough to keep the power turned off after an over-current event. For many e-matches, this isn't an issue, since 9 milliseconds is way longer than the 13 microseconds needed to fire the charge. Unfortunately, we've discovered that many of the e-matches bought and used in the rocketry hobby are actually made for use in the fireworks industry, where it is desireable to retain continuity after firing so that series connections of e-matches all can fire even if some fire faster than others! This means that while the resistance goes up some after firing, sometimes the drain on the battery remains sufficient to cause the protection circuit to kick in even after the pyro charge has fired. What We're Doing About It If we remove the protection circuit from the LiPo (or jumper around it), all existing Altus Metrum products will operate successfully with pyro charges thave have an effective resistance of as low as 1 ohm. That's lower than any e-match or Q2G2 we've ever seen, so in effect what this means is that if you have an existing Altus Metrum flight computer, and you remove the LiPo protection circuit, you're good to go. This does not really make things any more "dangerous", since our battery chargers are all current limited and our discharge patterns will never cause heating of the battery. Frankly, in a rocket, the most likely way to cause a problem with a LiPo is by smashing or puncturing the actual battery during bench work or during a crash... and those cause the same problems with or without a protection board present. In the future, we will ship batteries that have either a much higher C rating on the protection circuit, or have no protection circuit at all. The number of problems reported by actual customers that we think should be attributed to LiPo protection circuit boards is very low, and we suspect most of our customers who are happily flying their boards can continue to do so. Ground testing where you fire pyro charges (or at least e-matches) using RF to issue the commands (not USB, since the LiPo charger is running any time USB is connected!) will confirm whether there's a problem. If the board resets (does startup beeps) after a pyro event, or shuts down completely (no LED activity), then you have a problem. If the matches light and the board keeps running, you're good to go. However, any Altus Metrum customer with LiPo batteries sourced from us or our distributors who is worried about this problem (even if you haven't seen problems in ground testing or previous flights), and who doesn't want to try soldering on the battery circuit board yourself, is welcome to contact us about removing the protection circuit for you. We won't charge anything other than shipping. To take advantage of this offer, just send email to fixmybattery@altusmetrum.org telling us how many of which capacity batteries you have that you'd like updated, and we'll respond with an RMA number and shipping details. Going Even Further As previously indicated, with the LiPo protection circuits removed, all of our current products will work reliably with at least 1 ohm across the pyro terminals. That should cover all real-world flying conditions just fine, but we're not satisfied yet. We've designed a new pyro control circuit that transfers the maximum possible energy to the load regardless of battery voltage without ever allowing the voltage to the processor to droop at all. We're testing this new circuit in various prototypes now, and if it pans out it will probably show up first in MegaMetrum and then trickle down to TeleMetrum and TeleMini as those products are updated later this year. The new pyro circuit tolerates 0 ohms (dead shorts) on the pyro terminals for as long as the battery can provide current, which is as good as it gets. We think the circuit is clever enough that we'll probably write more about it once we're finished validating it.

There is no common way to express license information in each source file at the moment. Some people embed license information in each file, others keep it in README or another file at the top of the source tree. Worse, there is no common syntax to express the license information in a machine-parseable way. If we had this, we could have tools that, for example, tell you if you're trying to merge code that has an incompatible license. Obviously, this kind of thing can never work perfectly. People keep inventing new licenses, and it is not possible for a computer program to fully understand any license. It is not clear humans can do that, either. However, it would be possible to do it to a number of well-known licenses, which would help most of the time. A classic 20/80 situation. I would like to suggest a syntax, similar to Emacs's "Hey Emacs" modelines, for embedding a summary of the license, or licenses, for a source file, in such a way that it can be programmatically extracted and parsed and analysed. With this syntax, one could then write a tool to ensure that all files in a project have the same license, or that all licenses are compatible with the project's overall license. Such a tool will, of course, rely on heuristics and assumptions. For example, it needs to assume the machine-parseable license summary is correct, and rely on a ruleset on what licenses are compatible with what licenses. Things can go wrong. That's life. Remember, this is aiming at doing the 20% of the work that will work 80% of the time, not perfection. I don't have a tool written, but I have a suggestion for the syntax.

/*
 * Copyright 2013 Lars Wirzenius
 *
 * tl;dr =*= Licenses: GPL-3+ or Expat, and Artistic =*=
 *
 * Blah. Blah. Blah. Imagine long, boring license texts 
 * here.
 */

The important part is this:

=*= Licenses: GPL-3+ or Expat, and Artistic =*=

The =*= prefix and suffix and the word Licenses are there to make grepping reasonably reliable without too many false positives, and to allow comment characters and other text on the same line. The actual license summary follows the syntax and semantics of the Debian copyright-format 1.0 specification, which I chose because it exists and has had a fair bit of review so far, and is reasonably expressive. The license summaries can be extracted with the following GNU sed invocation:

sed -n '/.*=\*= [Ll]icen[cs]es\?: \(.*\)=\*=.*/s//\1/p'

I allowed various forms of the word license, since it's a word that a lot of people will get wrong, and it's easy to catch all four common forms. So, does anyone else think this might be useful? Would you use it in your own projects?

Finally, finally, I've got a car. I had this grand plan of using a friend of a friend who was a car buyer to source me a car and have it ready for me at the airport when I arrived. That didn't work out so well, so I resorted to carsales.com.au and found something very quickly. I could have bought it last Friday, if I'd wanted to forego the RACQ inspection, but I hate used cars at the best of times, and so I want to do what I can do avoid buying a lemon, so I had to wait until Monday for the inspection. I got the report in the early afternoon on Monday, and the only thing it highlighted was a bit of oil on the front differential housing. I contacted the dealer and he said he'd get it looked at. I got an SMS from him on Tuesday morning saying the car would be ready after 3pm, but by the time I could arrange with Kristy for a ride, we just missed the bank, so I couldn't get the bank cheque to pay for it, so we rescheduled for this morning. The shipping container was delivered to Sarah's place on Monday, and there's been a steady stream of boxes arriving at my place. It was good to be able to transport some of those myself today, and there'll be more to move tomorrow. I need to sort out storage options, because one thing I don't want is for there to be too much clutter in my home. I think there'll be another trip to IKEA in my near future. At least I can do that all on my own now. It's so great to have independent mobility again.

Even though I haven't had much time to dedicate to http.debian.net lately, it has been up and running, or should I say serving?

Part of its job is to detect mirrors that have temporary issues or are entirely gone, down, unavailable. It does so, and many other things, by monitoring the so-called "trace files". A very important one being the "master" (or "origin") trace file.

With the recent integration of backports into the main archive, the master trace file of the backports mirrors also changed. Long story short, this change caused backports mirrors to no longer be considered by the mirror redirector as candidates. As long as they were up to date.

After the usual mirror synchronisation delay, more and more mirrors were disabled and subsets of "up to date" candidates re-calculated. This reached a critical point when only one mirror was left in the database. The mirror had not been synchronised for a couple of weeks.

This mirror is located in Costa Rica, and as the only candidate left in the database it was the only one used to serve requests for the backports archive. No matter where the client was located in the world.

The issue was later noticed and the necessary updates to the mirrors master list made. Mirrors started to be re-considered as they were re-checked (with some delay due to the rate limiter) and the subsets re-calculated. In a few hours everything was back to normality.

Correctness and fault-tolerance don't always get together very well...

I'd decided ahead of time that I wanted to use Internode as my ISP, and had ordered a Naked DSL service from them and also decided to bundle my mobile phone with them as well. For reasons that only made sense to me 7 years ago, I've been paying Telstra to keep my mobile number going, but I've long since lost the SIM. My current phone has a micro-SIM anyway, so I needed a replacement SIM. My grand plan had been to order the SIM, order the number port from Telstra to Internode, and then, well, profit from the moment I stuck the SIM in my phone. Unfortunately the port didn't go through as planned, and I was left incommunicado for the better part of two and a half days. I felt like I had my hands tied behind my back not having a mobile data service. It was also mildly annoying not being able to call people or be contactable, given the amount of running around I was doing. But it got resolved and is fast becoming a distant memory. The DSL service required a Telstra technician to come out (I'm not actually sure why) and that was scheduled for Thursday. I happened to catch him while he was at my building's MDF, and had a bit of a chat with him. He was a Scotsman, and I didn't get all the details, but he was going on about how he was only there to operate on the exchange side of the MDF, and I'd have to get someone else to jumper it up to my apartment. This wasn't what I expected from an installation service, but sure enough when I finally got around to plugging the ADSL modem in on Saturday morning, there was no line sync to be had. A call to Internode confirmed that he'd only jumpered it up to exchange side of the MDF. What was even more annoying was I'm pretty sure I saw him yanking out jumper wires from the MDF when he was working on it. Jumper wires that connected the exchange side of the MDF to my apartment. I was not thrilled with the idea of waiting (and paying) for a cabling contractor to come out and hook up a couple of bits of jumper wire, so I put out a call on Facebook for a Krone tool and a tone generator, and Brent was able to come through for me. He dropped the gear around while I was out shopping with Kristy, and when we got back, I located the pair for my unit, and rejumpered the existing jumper wire that I'm pretty sure the Telstra technician had disconnected. Lo and behold, my ADSL started working. I felt pretty proud of myself. It's fun operating at Layer 1 every now and then. The FRITZ!Box Fon WLAN 7270 is quite the beast of a box. Not only is it an ADSL modem, it's a wireless router, DECT base station, VoIP thingy and an answering machine! I've managed to connect my Engin account up to it, so once I get a DECT handset, I'll be able to make VoIP calls through it. I don't need to run Asterisk any more.

According to the update here, the signing keys are supposed to be replaced by the hardware vendor. If vendors do that, this ends up being uninteresting from a security perspective - you could generate a signed image, but nothing would trust it. It should be easy enough to verify, though. Just download a firmware image from someone using AMI firmware, pull apart the capsule file, decompress everything and check whether the leaked public key is present in the binaries.

The real risk here is that even if most vendors have replaced that key, some may not have done. There's certainly an argument that shipping test keys at all increases the probability that a vendor will accidentally end up using those rather than generating their own, and it's difficult to rule out the possibility that that's happened.

comments

(See here for an update to this)

A hardware vendor apparently had a copy of an AMI private key on a public FTP site. This is concerning, but it's not immediately obvious how dangerous this is for a few reasons. The first is that this is apparently the firmware signing key, not any of the Secure Boot keys. That means it can't be used to sign a UEFI executable or bootloader, so can't be used to sidestep Secure Boot directly. The second is that it's AMI's key, not a board vendor - we don't (yet) know if this key is used to sign any actual shipping firmware images, or whether it's effectively a reference key. And, thirdly, the code apparently dates from early 2012 - even if it was an actual signing key, it may have been replaced before any firmware based on this code shipped.

But there's still the worst case scenario that this key is used to sign most (or all) AMI-based vendor firmware. Can this be used to subvert Secure Boot? Plausibly. The attack would involve producing a new, signed firmware image with Secure Boot either disabled or with an additional key installed, and then to reflash that firmware. Firmware images are very board-specific, so unless you're engaging in a very targeted attack you either need a large repository of firmware for every board you want to attack, or you need to perform in-place modification.

Taking a look at the firmware update tool used for AMI systems, the latter might be possible. It seems that the AMI firmware driver allows you to dump the existing ROM to a file. It'd then be a matter of pulling apart the firmware image, modifying the key database, putting it back together, signing it and flashing it. It looks like doing this does require that the user enter the firmware password if one's set, so the simplest mitigation strategy would be to do that.

So. If this key is used by most vendors shipping AMI-based firmware, and if it's a current (rather than test) key, then it may well be possible for it to be deployed in an automated malware attack that subverts the Secure Boot trust model on systems running AMI-based firmware. The obvious lesson here is that handing out your private keys to third parties that you don't trust is a pretty bad idea, as is including them in source repositories.

(Wow, was this really as long ago as 2004? How little things change)

comments

Antonia came into the room. Where's Alieta? she asked. We don't know, replied Luda. It was true; we hadn't seen Alieta in quite some time, and we had no idea where she was. Antonia left. Varvara eyed me mischievously and said, You could call her. How? I asked. Touch the grandmother clock and you will figure it out. I touched the clock trepidly. Nothing happened. I opened my mouth to speak, but the word I had been about to say resonated in my mind as if spoken through a comb filter. I concentrated. Alieta, dozens of human-like voices sang over and over in different harmonies. I let go. What was that? I demanded. Clock, Varvara said dismissively. Come on, let's do laundry, Luda grunted. We carried our laundry out to the garage, which is where the washer and dryer lived. Alieta slipped in through one of the open garage doors, and surreptitiously pulled Varvara over for a whispered conversation. Antonia wandered in and squinted in Alieta's general direction. Luda quickly pulled Varvara in front of her. It's just Varvara, she announced. Antonia's eyes glazed over and she went back into the house. Alieta pursed her lips and studied the wall as if there might be some reflective object that had alerted her mother to her presence. Then she grabbed Varvara by the hand, pulled her over to push the buttons that closed the garage doors, and dragged her into the house. I watched the garage doors close. Then I watched the one on the left reverse direction. I opened my mouth to notify someone of this, but the doors reversed direction again, and proceeded to partially open and close as if possessed. I tried to hide behind Luda, without success. Now she was watching as well, as reality seemed to to shimmer and bend. The roof became arched, and there were loud mechanical noises as our environs transformed. When it finished, the place where the garage doors had been was now a wall of screens displaying video as you might encounter in a television store. More disconcertingly, we were in front of a railing which overlooked what appeared to be a car dealership showroom. Luda sucked in her breath. Benjamin Snow, you were into some serious shit, she muttered.

Not content with just one day of driving all over town, Kristy came back for another day of it. I decided that rather than specifically shopping, I needed to do some of the more bureaucratic stuff, so that I could ensure things like the electricity could get put on in my name. I determined that in order to get the electricity in my name, I'd need to provide either a driver's licence number or Medicare card number. Unfortunately I remember separating my expired ACT licence and Medicare cards from my other random cards while I was packing up my temporary apartment in the US, but I can't for the life of me recall what I did with them, so our first port of call was the Department of Transport in Zillmere to get a new licence. Talk about a painless experience. The most annoying thing was that on Wednesdays they open at 9:30am instead of 8:30am. We got there at 9:10am. I had Zoe with me, and Kristy had her daughter, and they happily played while we waited outside. I just had to fill out a fairly simple form, and I was called up promptly and there were no problems at all. In under 30 minutes of walking in the door, we were driving away with a temporary licence. Vastly different from my experience with the DMV. I'll get the genuine article in the mail in a few weeks. Then we headed over to Westfield Chermside to go to Medicare, Medibank Private, and as I was growing frustrated with how long my mobile phone number was taking to port from Telstra to Internode, a Telstra Shop to try and get a replacement Telstra SIM. This is where I ran into more of a bureaucratic brick wall. For Medicare, I wanted to get my own Medicare card (and number) again, instead of a shared one, and so I essentially had to re-apply. They wanted more than just a passport entry stamp and something with an address on it. They wanted specific documents with an address on it, and an offer letter to show I was employed, so I had to leave there empty handed. Medibank Private was even worse. In a "shut up and take my money!" kind of moment, they told me to prematurely unsuspend a suspended policy, I needed to request a document from the Department of Immigration that showed my international movements to confirm that I was indeed back in the country. I've always found this somewhat ironic, given I'm sitting in front of a Medibank Private employee when they're telling me this, and I'm trying to give them money. So I left there empty handed as well. I grabbed some cutlery and crockery from Big W. The Telstra Shop had a 45 minute wait, and as I didn't want to over-stretch Zoe, we headed back to my parents. I took the opportunity to open an electricity account with AGL, now that I had a driver's licence number. Zoe declined to take a nap, and was having a good time playing with Kristy's daughter, and they both wanted to stay at my parent's place, so we left them there and dashed over to Ikea to rectify the bed slat issue. While we were at Ikea, my number finally ported across and my phone started working, which was a huge relief. Being uncontactable during a period of many interactions with random people was highly frustrating for me. Not having mobile Internet access for a few days of extreme mobility showed how much my phone is an extension of my brain. I also picked up a bunch of other random stuff from Ikea, stools that hadn't been in stock the day before, drinking glasses, that sort of thing. We then dropped all that off at the apartment before heading back to my parents' place. So it was another busy day of running all over town, and again, I'm very grateful to Kristy for volunteering her time to make it happen. Most notable accomplishments: complete bed for Zoe, electricity, and a working mobile phone.

This was the first "normal" (i.e. not part of the Easter "long weekend") day. As it happened, Easter Monday was surprisingly retail-friendly anyway. My friend Kristy picked me up in the morning, and we dropped past the real estate agent for my apartment to see if the tenants had happened to drop the keys back yet (they hadn't) and then we went to Ikea and bought a bed for Zoe a bunch of other random stuff. I also bought a bed frame and mattress while we were at the neighbouring Logan MegaCentre (it's not a bad shopping centre) The bed frame is wooden and is taking 6-8 weeks to be made, so in the meantime, because I bought a mattress from them, they're lending me a mattress base. That's getting delivered on Saturday. We then headed over to Harvey Norman in Fortitude Valley. We'd just got started there when the real estate agent called to say the tenants had dropped the keys back, so we stopped and headed back over to meet the property manager at the apartment and get the keys. After that, we headed back to Harvey Norman and bought a fridge, TV, and a bunch of small appliances, and then headed back to the apartment to do some Ikea assembly. We'd just about finished putting Zoe's bed together when we discovered we'd gotten the wrong width slats for her bed. It turns out there are two widths of the "Sultan Lade" slats, and they're right next to each other in the warehouse. We'd picked up from the correct location, but I think the piles had become jumbled. Lesson learned: cross check the SKU as well as the pickup location. It was a long day, and I was enormously grateful to Kristy for driving me all over the place, and generally helping me shop. I think it ended up being a 15 hour day for her.

Thanks to a generous donation by Bytemark Hosting, Debian started deploying machines for its core infrastructure services in a new data center in York, UK. This hardware and hosting donation will allow the Debian Systems Administration (DSA) team to distribute Debian's core services across a greater number of geographically diverse locations, and improve, in particular, the fault-tolerance and availability of end-user facing services. Additionally, the storage component of this donation will dramatically reduce the storage challenges that Debian currently faces. The hardware provided by Bytemark Hosting consists of a fully-populated HP C7000 BladeSystem chassis containing 16 server blades:

• 12 BL495cG5 blades with 2x Opteron 2347 and 64GB RAM each
• 4 BL465cG7 blades with 2x Opteron 6100 series and 128GB RAM each

and several HP Modular Storage Arrays:

• 3 MSA2012sa
• 6 MSA2000 expansion shelves

with 108 drive bays in total, mostly 500GB SATA drives, some 2TB, some 600GB 15kRPM SAS, providing a total of 57 TB. 57 TB today could host roughly 80 times the current Debian archive or 3 times the Debian Snapshot archive. But remember both archives are constantly growing!

I've meant to look at Hy since Paul Tagliamonte started to talk to me about it, but never took a chance until now. Yesterday, Paul indicated it was a good time for me to start looking at it, so I spent a few hours playing. But what's Hy? Python is very nice: it has a great community and a wide range of useful libraries. But let's face it, it misses a great language. Hy is an implementation of a Lisp on top of Python. Technically, Hy is built directly with a custom made parser (for now) which then translates expressions using the Python AST module to generate code, which is then run by Python. Therefore, it shares the same properties as Python, and is a Lisp-1 (i.e. with a single namespace for symbols and functions). If you're interested to listen Paul talking about Hy during last PyCon US, I recommend watching his lightning talk. As the name implies, it's only a few minutes long. <iframe allowfullscreen="allowfullscreen" class="shadow" frameborder="0" height="374" src="http://www.youtube.com/embed/1vui-LupKJI?wmode=transparent&amp;autohide=1&amp;egm=0&amp;hd=1&amp;iv_load_policy=3&amp;modestbranding=1&amp;rel=0&amp;showinfo=0&amp;showsearch=0#t=16m14s" width="500"></iframe> Does it work? I've been cloning the code and played around a bit with Hy. And to my greatest surprise and pleasure, it works quite well. You can imagine writing Python from there easily. Part of the syntax smells like Clojure's, which looks like a good thing since they're playing in the same area. You can try a Hy REPL in your Web browser if you want. Here's what some code look like:
This code would ouput:

('date', 'Wed, 03 Apr 2013 12:09:23 GMT')
('connection', 'keep-alive')
('content-encoding', 'gzip')
('transfer-encoding', 'chunked')
('content-type', 'text/html; charset=utf-8')
('server', 'nginx/1.2.6')

As you can see, it's really simple to write Lispy code that really uses Python idioms. There's obviously still a lots of missing features in Hy. The language if far from complete and many parts are moving, but it's really promising, and Paul's doing a great job implementing every idea. I actually started to hack a bit on Hy, and will try to continue to do so, since I'm really eager to learn a bit more about both Lisp and Python internals in the process. I've already send a few patches on small bugs I've encountered, and proposed a few ideas. It's really exciting to be able to influence early a language design that I'll love to use! Being a recent fan of Common Lisp, I tend to grab the good stuff from it to add them into Hy.