The task: reliably synchronize my local MailDir with several remote IMAP mailboxes with IDLE support so that there is no need to poll with small time intervals to get new email immediately. Most graphical mail clients like icedove/thunderbird or evolution have IDLE support which means that their users get notified about new email as soon as it arrives. I prefer to handle email fetching/reading/sending using separate programs so after having used icedove for a long time, I switched to a offlineimap/mutt based setup a few years ago. Some while after that I discovered sup and later notmuch/alot which made me switch again. Now my only remaining problem is the synchronization between my local email and several remote IMAP servers. Using offlineimap worked fine in the beginning but soon I discovered some of its shortcomings. It would for example sometimes lock up or simply crash when I switched between different wireless networks or switched to ethernet or UMTS. Crashing was not a big problem as I just put it into a script which re-executed it every time it crashed. The problem was it locking up for one of its synchronizing email accounts while the others were kept in sync as usual. This once let to me missing five days of email because offlineimap was not crashing and I believed everything was fine (new messages from the other accounts were scrolling by as usual) while people were sending me worried emails whether I was okay or if something bad had happened. I nearly missed a paper submission deadline and another administrative deadline due to this. This was insanely annoying. It turned out that other mail synchronization programs suffered from the same lockup problem so I stuck with offlineimap and instead executed it as: Which would synchronize my email every five minutes and kill offlineimap if a synchronization took more than one minute. While I would've liked an email synchronizer which would not need this measure, this worked fine over the months. After some while it bugged me that icedove (which my girlfriend is using) was receiving email nearly instantaneously after they arrived and I couldnt have this feature with my setup. This instantaneous arrival of email works by using the IMAP IDLE command which allows the server to notify the client once new email arrives instead of the client having to poll for new email in small intervals. Unfortunately offlineimap (and any other email synchronizer I found) would not support the IDLE command. There is a fork of it which supports IDLE by using a newer python imap library but this is of little use to me as there is no possibility of a hook which executes when new email arrives so that I can execute notmuch new on the new email. At this point I could've used inotify to execute notmuch new upon arrival of new email but I went another way. Here is a short python script idle.py which connects to my IMAP servers and sends the IDLE command: It would create a connection to every directory I want to watch on every email account I want to synchronize. The select call will expire after 33 minutes as most email servers would drop the connection if nothing happens at around 30 minutes. If something happened within that time though, the script would output the arguments to offlineimap to do a quick check on just that mailbox on that account. If nothing happened, the script would output the arguments to offlineimap to do a full check on all my mailboxes. Every call which interacts with the network is wrapped in a timeout command to avoid any funny effects. Should the python script timeout, a full synchronization with offlineimap is triggered. Should offlineimap timeout, an error message is written to stderr and the script continues. The above naturally has the disadvantage of not immediately responding to new email which arrives during the time that idle.py is not running. But as this email will be fetched once the next message arrives on the same account, there is no much waiting time and so far, this problem didnt bite me. Is there a better way to synchronize my email and at a same time make use of IDLE? I'm surprised I didnt find software which would offer this feature.

Over the last few weeks I've been making several changes to http.debian.net to detect mirrors that don't follow Debian's mirroring guidelines and end up causing problems to the end users. The changes will mean less hash mismatches and similar errors.

As I wrote back in December, the redirector is becoming nicer but also stricter. Some of the changes I recently made caused over 30 mirrors to be completely disabled from the redirector. This is not ideal and I don't like having to disable mirrors. They are contributions afterall.

The only thing I can do, and can't stress enough, is encourage people to use an up to date ftpsync script (available at project/ftpsync/ on every mirror) to mirror Debian.
It takes care for you of all the little but important things needed. Really. A mirror that uses ftpsync is easier for the administrator to properly configure, and provides a consistent mirror for the benefit of the users.

Speaking of ftpsync, there is a new version! If you use ftpsync please upgrade it as soon as possible.

Other improvements are on their way. Contributions are welcome (if you like refactoring, there's quite a bit of explicitly-redundant code in check.pl that should now give a better idea of the way it needs to be refactored.)

Completing the DRI3 Extension This week marks a pretty significant milestone for the glorious DRI3000 future. The first of the two new extensions is complete and running both full Gnome and KDE desktops. DRI3 Extension Overview The DRI3 extension provides facilities for building direct rendering libraries to work with the X window system. DRI3 provides three basic mechanisms:

1. Open a DRM device.
2. Share kernel objects associated with X pixmaps. The direct rendering client may allocate kernel objects itself and ask the X server to construct a pixmap referencing them, or the client may take an existing X pixmap and discover the underlying kernel object for it.
3. Synchronize access to the kernel objects. Within the X server, Sync Fences are used to serialize access to objects. These Sync Fences are exposed via file descriptors which the underlying driver can use to implement synchronization. The current Intel DRM driver passes a shared page containing a Linux Futex.

Opening the DRM Device Ideally, the DRM application would be able to just open the graphics device and start drawing, sending the resulting buffers to the X server for display. There s work going on to make this possible, but the current situation has the X server in charge of blessing the file descriptors used by DRM clients. DRI2 does this by having the DRM client fetch a magic cookie from the kernel and pass that to the X server. The cookie is then passed to the kernel which matches it up with the DRM client and turns on rendering access for that application. For DRI3, things are much simpler the DRM client asks the X server to pass back a file descriptor for the device. The X server opens the device, does the magic cookie dance all by itself (at least for now), and then passes the file descriptor back to the application.

 
    DRI3Open
    drawable: DRAWABLE
    driverType: DRI3DRIVER
    provider: PROVIDER
       
    nfd: CARD8
    driver: STRING
    device: FD
 
    Errors: Drawable, Value, Match
    This requests that the X server open the direct rendering
    device associated with drawable, driverType and RandR
    provider. The provider must support SourceOutput or SourceOffload.
    The direct rendering library used to implement the specified
    'driverType' is returned in 'driver'. The file
    descriptor for the device is returned in 'device'. 'nfd' will
    be set to one (this is strictly a convenience for XCB which
    otherwise would need request-specific information about how
    many file descriptors were associated with this reply).

Sharing Kernel Pixel Buffers An explicit non-goal of DRI3 is support for sharing buffers that don t map directly to regular X pixmaps. So, GL ancillary buffers like depth and stencil just don t apply here. The shared buffers in DRI3 are regular X pixmaps in the X server. This provides a few obvious benefits over the DRI2 scheme: In the kernel, the buffers are referenced by DMA-BUF handles, which provides a nice driver-independent mechanism.

1. Lifetimes are easily managed. Without being associated with a separate drawable, it s easy to know when to free the Pixmap.
2. Regular X requests apply directly. For instance, copying between buffers can use the core CopyArea request.

To create back- and fake-front- buffers for Windows, the application creates a kernel buffer, associates a DMA-BUF file descriptor with that and then sends the fd to the X server with a pixmap ID to create the associated pixmap. Doing it in this direction avoids a round trip.

 
    DRI3PixmapFromBuffer
    pixmap: PIXMAP
    drawable: DRAWABLE
    size: CARD32
    width, height, stride: CARD16
    depth, bpp: CARD8
    buffer: FD
 
    Errors: Alloc, Drawable, IDChoice, Value, Match
    Creates a pixmap for the direct rendering object associated
    with 'buffer'. Changes to pixmap will be visible in that
    direct rendered object and changes to the direct rendered
    object will be visible in the pixmap.
    'size' specifies the total size of the buffer bytes. 'width',
    'height' describe the geometry (in pixels) of the underlying
    buffer. 'stride' specifies the number of bytes per scanline in
    the buffer. The pixels within the buffer may not be arranged
    in a simple linear fashion, but 'size' will be at least
    'height' * 'stride'.
    Precisely how any additional information about the buffer is
    shared is outside the scope of this extension.
    If buffer cannot be used with the screen associated with
    drawable, a Match error is returned.
    If depth or bpp are not supported by the screen, a Value error
    is returned.

To provide for texture-from-pixmap, the application takes the pixmap ID and passes that to the X server which returns the a file descriptor for a DMA-BUF which is associated with the underlying kernel buffer.

 
    DRI3BufferFromPixmap
    pixmap: PIXMAP
       
    depth: CARD8
    size: CARD32
    width, height, stride: CARD16
    depth, bpp: CARD8
    buffer: FD
 
    Errors: Pixmap, Match
    Pass back a direct rendering object associated with
    pixmap. Changes to pixmap will be visible in that
    direct rendered object and changes to the direct rendered
    object will be visible in the pixmap.
    'size' specifies the total size of the buffer bytes. 'width',
    'height' describe the geometry (in pixels) of the underlying
    buffer. 'stride' specifies the number of bytes per scanline in
    the buffer. The pixels within the buffer may not be arranged
    in a simple linear fashion, but 'size' will be at least
    'height' * 'stride'.
    Precisely how any additional information about the buffer is
    shared is outside the scope of this extension.
    If buffer cannot be used with the screen associated with
    drawable, a Match error is returned.

Tracking Window Size Changes When Eric Anholt and I first started discussing DRI3, we hoped to avoid needing to learn about the window size from the X server. The thought was that the union of all of the viewports specified by the application would form the bounds of the drawing area. When the window size changed, we expected the application would change the viewport. Alas, this simple plan isn t sufficient here a few GL functions are not limited to the viewport. So, we need to track the actual window size and monitor changes to it. DRI2 does this by delivering invalidate events to the application whenever the current buffer isn t valid; the application discovers that this event has been delivered and goes to as the X server for the new buffers. There are a couple of problems with this approach:

1. Any outstanding DRM rendering requests will still draw to the old buffers.
2. The Invalidate events must be captured before the application sees the related ConfigureNotify event so that the GL library can react appropriately.

The first problem is pretty intractable within DRI2 the application has no way of knowing whether a frame that it has drawn was delivered to the correct buffer as the underlying buffer object can change at any time. DRI3 fixes this by having the application in control of buffer management; it can easily copy data from the previous back buffer to the new back buffer synchronized to its own direct rendering. The second problem was solved in DRI2 by using the existing Xlib event hooks; the GL library directly implements the Xlib side of the DRI2 extension and captures the InvalidateBuffers events within that code, delivering those to the driver code. The problem with this solution is that Xlib holds the Display structure mutex across this whole mess, and Mesa must be very careful not to make any Xlib calls during the invalidate call. For DRI3, I considered placing the geometry data in a shared memory buffer, but my future plans for the Present extension led me to want an X event instead (more about the Present extension in a future posting). An X ConfigureNotify event is sufficient for the current requirements to track window sizes accurately. However, there s no easy way for the GL library to ensure that ConfigureNotify events will be delivered to the application other application code may (and probably will) adjust the window event mask for its own uses. I considered adding the necessary event mask tracking code within XCB, but again, knowing that the Present extension would probably need additional information anyhow, decided to create a new event instead. Using an event requires that XCB provide some mechanism to capture those events, keep them from the regular X event stream, and deliver them to the GL library. A further requirement is that the GL library be absolutely assured of receiving notification about these events before the regular event processing within the application will see a core ConfigureNotify event. The method I came up with for XCB is fairly specific to my requirements. The events are always XGE events, and are tagged with a special event context ID , an XID allocated for this purpose. The combination of the extension op-code, the event type and this event context ID are used to split off these events to custom event queues using the following APIs:

/**
 * @brief Listen for a special event
 */
xcb_special_event_t *xcb_register_for_special_event(xcb_connection_t *c,
                                                    uint8_t extension,
                                                    uint16_t evtype,
                                                    uint32_t eid,
                                                    uint32_t *stamp);

This creates a special event queue which will contain only events matching the specified extension/type/event-id triplet.

/**
 * @brief Returns the next event from a special queue
 */
xcb_generic_event_t *xcb_check_for_special_event(xcb_connection_t *c,
                                                 xcb_special_event_t *se);

This pulls an event from a special event queue. These events will not appear in the regular X event queue and so applications will never see them. There s one more piece of magic here the stamp value passed to xcbregisterforspecialevent. This pointer refers to a location in memory which will be incremented every time an event is placed in the special event queue. The application can cheaply monitor this memory location for changes and known when to check the queue for events. Within GL, the value used is the existing dri2 stamp value. That is checked at the top of the rendering operation; if it has changed, the drawing buffers will be re-acquired. Part of the buffer acquisition process is a check for special events related to the window. For now, I ve placed these events in the DRI3 extension. However, they will move to the Present extension once that is working.

 
    DRI3SelectInput
    eventContext: DRI3EVENTID
    window: WINDOW
    eventMask: SETofDRI3EVENT
 
    Errors: Window, Value, Match, IDchoice
    Selects the set of DRI3 events to be delivered for the
    specified window and event context. DRI3SelectInput can
    create, modify or delete event contexts. An event context is
    associated with a specific window; using an existing event
    context with a different window generates a Match error.
    If eventContext specifies an existing event context, then if
    eventMask is empty, DRI3SelectInput deletes the specified
    context, otherwise the specified event context is changed to
    select a different set of events.
    If eventContext is an unused XID, then if eventMask is empty
    no operation is performed. Otherwise, a new event context is
    created selecting the specified events.

The events themselves look a lot like a configure notify event:

 
    DRI3ConfigureNotify
    type: CARD8         XGE event type (35)
    extension: CARD8        DRI3 extension request number
    length: CARD16          2
    evtype: CARD16          DRI3_ConfigureNotify
    eventID: DRI3EVENTID
    window: WINDOW
    x: INT16
    y: INT16
    width: CARD16
    height: CARD16
    off_x: INT16
    off_y: INT16
    pixmap_width: CARD16
    pixmap_height: CARD16
    pixmap_flags: CARD32
 
    'x' and 'y' are the parent-relative location of 'window'.

Note that there are a couple of odd additional fields offx, offy, pixmapwidth, pixmapheight and pixmap_flags are all place-holders for what I expect to end up in the Present extension. For now, in DRI3, they should be ignored. Synchronization The DRM application needs to know when various X requests related to its buffers have finished. In particular, when performing a buffer swap, the client wants to know when that completes, and be able to block until it has. DRI2 does this by having the application make a synchronous request from the X server to get the names of the new back buffer for drawing the next frame. This has two problems:

1. The synchronous round trip to the X server isn t free. Other running applications may cause fairly arbitrary delays in getting the reply back from the X server.
2. Synchronizing with the X server doesn t ensure that GPU operations are necessarily serialized between the application and the X server.

What we want is a serialization guarantee between the X server and the DRM application that operates at the GPU level. I ve written a couple of times (dri3k first steps and Shared Memory Fences) about using X Sync extension Fences (created by James Jones and Aaron Plattner) for this synchronization and wanted to get a bit more specific here. With the X server, a Sync extension Fence is essentially driver-specific, allowing the hardware design to control how the actual synchronization is performed. DRI3 creates a way to share the underlying operating system object by passing a file descriptor from application to the X server which somehow references that device object. Both sides of the protocol need to tacitly agree on what it means.

 
    DRI3FenceFromFD
    drawable: DRAWABLE
    fence: FENCE
    initially-triggered: BOOL
    fd: FD
 
    Errors: IDchoice, Drawable
    Creates a Sync extension Fence that provides the regular Sync
    extension semantics along with a file descriptor that provides
    a device-specific mechanism to manipulate the fence directly.
    Details about the mechanism used with this file descriptor are
    outside the scope of the DRI3 extension.

For the current GEM kernel interface, because all GPU access is serialized at the kernel API, it s sufficient to serialize access to the kernel itself to ensure operations are serialized on the GPU. So, for GEM, I m using a shared memory futex for the DRI3 synchronization primitive. That does not mean that all GPUs will share this same mechanism. Eliminate the kernel serialization guarantee and some more GPU-centric design will be required. What about Swap Buffers? None of the above stuff actually gets bits onto the screen. For now, the GL implementation is simply taking the X pixmap and copying it to the window at SwapBuffers time. This is sufficient to run applications, but doesn t provide for all of the fancy swap options, like limiting to frame rate or optimizing full-screen swaps. I ve decided to relegate all of that functionality to the as-yet-unspecified Present extension. Because the whole goal of DRI3 was to get direct rendered application contents into X pixmaps, the Present extension will operate on those X objects directly. This means it will also be usable with non-DRM applications that use simple X pixmap based double buffering, a class which includes most existing non-GL based Gtk+ and Qt applications. So, I get to reduce the size of the DRI3 extension while providing additional functionality for non direct-rendered applications. Current Status As I said above, all of the above functionality is running on my systems and has booted both complete KDE and Gnome sessions. There have been some recent DMA-BUF related fixes in the kernel, so you ll need to run the latest 3.9.x stable release or a 3.10 release candidate. Here s references to all of the appropriate git repositories: DRI3 protocol and spec:

git://people.freedesktop.org/~keithp/dri3proto      master

XCB protocol

git://people.freedesktop.org/~keithp/xcb/proto.git  dri3

XCB library

git://people.freedesktop.org/~keithp/xcb/libxcb.git dri3

xshmfence library:

git://people.freedesktop.org/~keithp/libxshmfence.git   master

X server:

git://people.freedesktop.org/~keithp/xserver.git    dri3

Mesa:

git://people.freedesktop.org/~keithp/mesa.git       dri3

Next Steps Now it s time to go write the Present extension and get that working. I ll start coding and should have another posting here next week.

It has been a while since my last English Debian Edu and Skolelinux interview last November. But the developers and translators are still pulling along to get the Wheezy based release out the door, and this time I managed to get an interview from one of the French translators in the project, C dric Boutillier. Who are you, and how do you spend your days? I am 34 year old. I live near Paris, France. I am an assistant professor in probability theory. I spend my daytime teaching mathematics at the university and doing fundamental research in probability in connexion with combinatorics and statistical physics. I have been involved in the Debian project for a couple of years and became Debian Developer a few months ago. I am working on Ruby packaging, publicity and translation. How did you get in contact with the Skolelinux / Debian Edu project? I came to the Debian Edu project after a call for translation of the Debian Edu manual for the release of Debian Edu Squeeze. Since then, I have been working on updating the French translation of the manual. I had the opportunity to make an installation of Debian Edu in a virtual machine when I was preparing localised version of some screen shots for the manual. I was amazed to see it worked out of the box and how comprehensive the list of software installed by default was. What amazed me was the complete network infrastructure directly ready to use, which can and the nice administration interface provided by GOsa . What pleased me also was the fact that among the software installed by default, there were many "traditional" educative software to learn languages, to count, to program... but also software to develop creativity and artistic skills with music (Ardour, Audacity) and movies/animation (I was especially thinking of Stopmotion). I am following the development of Debian Edu and am hanging out on #debian-edu. Unfortunately, I don't much time to get more involved in this beautiful project. What do you see as the advantages of Skolelinux / Debian Edu? For me, the main advantages of Skolelinux/Debian Edu are its community of experts and its precise documentation, as well as the fact that it provides a solution ready to use. I would add also the fact that it is based on the rock solid Debian distribution, which ensures stability and provides a huge collection of educational free software. What do you see as the disadvantages of Skolelinux / Debian Edu? Maybe the lack of manpower to do lobbying on the project. Sometimes, people who need to take decisions concerning IT do not have all the elements to evaluate properly free software solutions. The fact that support by a company may be difficult to find is probably a problem if the school does not have IT personnel. One can find support from a company by looking at the wiki dokumentation, where some countries already have a number of companies providing support for Debian Edu, like Germany or Norway. This list is easy to find readily from the manual. However, for other countries, like France, the list is empty. I guess that consultants proposing support for Debian would be able to provide some support for Debian Edu as well. Which free software do you use daily? I am using the KDE Plasma Desktop. But the pieces of software I use most runs in a terminal: Mutt and OfflineIMAP for emails, latex for scientific documents, mpd for music. VIM is my editor of choice. I am also using the mathematical software Scilab and Sage (built from source as not completely packaged for Debian, yet). Do you have any suggestions for teachers interested in using the free software in Debian to teach mathematics and statistics? I do not have any "nice" recommendations for statistics. At our university, we use both R and Scilab to teach statistics and probabilistic simulations. For geometry, there are nice programs:

• drgeo and kig to do constructions in planar geometry
• kali to discover symmetry groups (the so-called wallpapers and frieze groups), although the interface looks a bit old.

I like also cantor, which provides a uniform interface to SciLab, Sage, Octave, etc... Which strategy do you believe is the right one to use to get schools to use free software? My suggestions would be to

• advertise the reduction of costs when free software is used.
• communicate about the quality of free software projects, using well known examples like Firefox, ThunderBird and OpenOffice.org/LibreOffice.
• advertise the living and strong community around the project.
• show that it is not more difficult to use than any other system.

This is my monthly summary of my free software related activities. If you re among the people who made a donation to support my work (70 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. The Debian Administrator s Handbook Spanish translation completed. The Spanish team finished the translation of the book. The PDF build process was not yet ready to build translations so I had to fix this. At the same time, I also improved the mobipocket build script to make use of Amazon s kindlegen when available (since Amazon now requires the use of this tool to generate Mobipocket files that can be distributed on their platform). Once those issues were sorted I made some promotion of this first completed translation because they really deserve some big kudos ! Plans for the French translation. You know that the Debian Administrator s Handbook came to life as a translation of the French book Cahier de l Admin Debian (published by Eyrolles). This means that we currently have a free translation of a proprietary book. It s a bid of an odd situation that I always wanted to fix. I discussed with Eyrolles to find out how we could publish the original book under the same licenses that we picked for the English book and the result is that we setup a new crowdfunding campaign to liberate the French book and then make it an official French translation of the Debian Administrator s Handbook. Read the rest and support us on the ulule project page (a kickstarter like for people who are not based in the US). Debian France I updated our membership management application (galette) to version 0.7.4.1 with numerous bug fixes but the true highlight this month was Solutions Libres et Opensource , a tradeshow in Paris where Tanguy Ortolo, me, and other volunteers (C dric Boutillier, Arnaud G., and some that I have forgotten, thanks to them!), held a Debian booth for two consecutive days (May 28-29). For once we had lots of goodies to sell (buffs, mouse pad, polos, stickers, etc.) and the booth was very well attended. Google s Summer of Code Last month I was rater overwhelmed with queries from students who were interested in applying for the Package Tracking System Rewrite project that I offered to mentor as part of Google s Summer of Code. In the end, I got 6 good student applications that Stefano and me evaluated. We selected Marko Lalic. The Community Bonding Period is just starting and we re fleshing out details on how we will organize the work. We ll try to use the IRC channel #debian-qa on OFTC for questions and answers and weekly meetings. Misc Debian packaging I packaged zim 0.60 and with the release of Wheezy, I uploaded to unstable all the packages that I staged in experimental (cpputest, publican). I sponsored the upload of libmicrohttpd 0.9.27-1. I filed a couple of bug reports that I experienced with the upcoming dpkg 1.17.0 (#709172, #709009). In both cases, the package was using a wrongly hardcoded path to dpkg-divert (the binary moved from /usr/sbin/ to /usr/bin/ a while ago and the compatibility symlink is dropped now). I also dealt with #709064 where the user reported upgrade issues related to multiarch. I also filed an upstream bug report on publican to request some way to avoid so much duplication of files (actually I filed it as a response to the Debian bug #708705 that I received). Kali work I had to update OpenVAS for Kali but some parts failed to build in a Debian 7 environment. I diagnosed the problem and submitted a patch upstream. I also got in touch with the Debian OpenVAS maintainer as I wanted to contribute the package back to Debian, but timing issues have pushed this back for a little longer. Thanks See you next month for a new summary of my activities.

One comment Liked this article? Click here. My blog is Flattr-enabled.

Since I wrote this, we've made some worthwhile progress on avoiding damaging Samsung hardware. The first is that the samsung-laptop driver appeared to be causing the firmware to attempt to write to an area of memory that was marked in the chipset, triggering a Machine Check Exception. That was what generated the pstore output that caused the problem originally. The driver now refuses to load if EFI is enabled, which avoids the problem. It's not ideal, since it's currently the only mechanism we have for certain functionality on Samsung laptops, but there you go.

The second problem was that avoiding crashing on boot didn't actually fix the problem in any fundamental way. Even with pstore disabled, it was possible for userspace to fill the nvram and trigger the same problem. Our first approach to this was to prevent any writes to nvram if the UEFI QueryVariableInfo() call reported that more than 50% of the nvram storage space would be used. That was safe, but led to another issue. The nvram storage area is typically implemented as part of the same flash chip as the firmware. Flash isn't arbitrarily accessible - changing the contents of a block typically involves rewriting the entire block. It's impractical to rewrite the entire nvram area on every write, so what actually happens is that deleting variables just results in them being marked as inactive but doesn't actually free up the space. The firmware can later perform some sort of garbage collection to free it up.

This caused us problems, since inactive space that hasn't been garbage collected yet isn't actually available, and as a result firmware implementations tend to count it as used. Say you had 64KB of nvram and wrote 32KB of variables. We'd then refuse to write any more because you'd drop below 50%. So you delete 16KB of the variables you've created and try again. Unfortunately, the firmware still thinks that there's 32KB in use and Linux would still refuse.

If you were lucky, rebooting would trigger a garbage collection run. If you weren't, it wouldn't. Problematic. Our next approach was to try to account for the space actually actively used by the variables, rather than relying on what the firmware told us via QueryVariableInfo(). This seems simple enough - just add up the size of all the variables and subtract that from the overall size to determine how much of the "used" space is actually just old inactive variables that can be ignored. However, there's still some problems there. The first is that each variable has some additional overhead associated with it, and the size of that overhead varies depending on the system vendor. We had to make a conservative guess, which could cause problems if systems had large numbers of small variables. The second is that the only variables the kernel can see are those that are flagged as runtime-visible. There may also be a significant quantity of nvram used to store variables that are only visible in boot services code. We could work around this by adding up sizes while we're still in boot services code, but on some systems calling QueryVariableInfo() before ExitBootServices() results in later calls to GetNextVariable() jumping to invalid addresses and crashing the kernel. Not a great approach.

Meanwhile, Samsung got back to us and let us know that their systems didn't require more than 5KB of nvram space to be available, which meant we could get rid of the 50% value and replace it with 5KB. The hope was that any system that booted with only 5KB of space available in nvram would trigger a garbage collection run. Unfortunately, it turned out that that wasn't true - some systems will only trigger garbage collection if the OS actually makes an attempt to write a variable that won't otherwise fit.

Hence this patch. The new approach is to ask the firmware how much space is available. If the size of the new variable would reduce this to less than 5K, we attempt to create a variable bigger than the remaining space. This should cause the firmware to realise that it's out of room and either (depending on implementation) perform a garbage collection run at runtime or set a flag that will cause the system to perform garbage collection on the next reboot. We then call QueryVariableInfo() again to see whether a garbage collection run actually happened, and if so check whether we now have enough space. If so, we go ahead and write the variable. If not, we tell userspace that there's not enough space.

This seems to work in all the situations I've tested, and it should avoid ending up in a situation where a Samsung can end up bricked. However, it's firmware, so who knows whether it's going to break things for someone else.

comments

I recently made a routine analysis of my kitchen. (Of course I make a routine analysis of my kitchen; don t you?) In it, I discovered these items, still usable, but approaching that magic throw it out date:

• Baby carrots
• Potatoes
• Lemons
• Green beans
• Corn

So, I thought, what can I make that would use all of these? And I realized I had some shrimp in the freezer, so: a shrimp boil! I tossed it all, plus some various seasonings and a few other veggies, into the Dutch oven, and boiled. Jacob and Oliver watched the activity with interest. Well, except for the potato-peeling part. For that, they went and played with their toy school buses. But the rest was good. They carefully observed me adding some spices, some vegetables, the shrimp, and watched it all simmer. Then it was time to eat. Excitement! Of course, it did take a few minutes to boil, so Jacob got down his whiteboard while Oliver looked on. They enjoyed learning how to peel the shell from the shrimp and devoured their food. And another night recently, Jacob unexpectedly showed up in the kitchen at 10PM. He said he was thirsty, so I got him some water. He asked, Dad, did you make ice cream? Earlier that day, I had prepared ice cream with Oliver, but it was a kind that had to be cooked (lemon with pureed strawberries and peaches) and it wasn t cool enough to finish before their bedtime. I did let them help add the ice and salt to the ice cream freezer just before they went to bed. So I told Jacob that yes, the ice cream was done. He stood there, tiredly, considering, with this oh he ll never say yes to ice cream at 10PM look on his face. So I said, would you like one bite right now? The look of delight on his face was amazing; a broad smile, a twinkle in his eyes, and a clap. So I got out the big bowl of ice cream and scooped up one big spoonful. He loved it. Then I said, should we go look at the stars? I carried Jacob outside to the porch. We stood there, looking up. I used to do this with him periodically, but it had been about a year. So he was thrilled. It was a partially overcast night, but there were still some stars visible. He had no idea there were some stars missing. To him, it was amazing and wonderful and infinite. Oh dad, there are way too many stars to count! He stayed there, arms around my neck, for a minute or two, then was ready to go back inside. I set him down, gave him a hug, said Goodnight, Jacob. And off he trotted, back upstairs, wearing a contented smile, and he fell asleep almost immediately. All it takes to delight children is a bit a shrimp or some stars. And those things delight me, too.

here's the overview about my RC bug fixing activities for the last week:

• #666799 libapache2-reload-perl: "libapache2-reload-perl: sourceful transition towards Apache 2.4"
  prepare adaption to apache 2.4 in git (pkg-perl)
• #666800 libapache-singleton-perl: "libapache-singleton-perl: sourceful transition towards Apache 2.4"
  prepare adaption to apache 2.4 in git (pkg-perl)
• #666807 libapache-authenhook-perl: "libapache-authenhook-perl: sourceful transition towards Apache 2.4"
  prepare adaption to apache 2.4 in git (pkg-perl)
• #666822 libapache2-mod-perl2: "libapache2-mod-perl2: sourceful transition towards Apache 2.4"
  upload the package to unstable, most of the work done by Niko Tyni (pkg-perl)
• #666837 libapache2-authcookie-perl: "libapache2-authcookie-perl: sourceful transition towards Apache 2.4"
  try to prepare adaption to apache 2.4 in git (pkg-perl)
• #678380 remotetea: "remotetea: FTBFS with Java7 as default java"
  apply patch from Ubuntu / James Page, upload to DELAYED/2
• #684296 tomcat-maven-plugin: "tomcat-maven-plugin: Drop build dependency on openjdk-6-jdk"
  apply patch from Ubuntu / James Page, upload to DELAYED/2
• #684305 umlet: "umlet: Transition package to use default java implementation"
  apply patch from Ubuntu / James Page, upload to DELAYED/2
• #709055 netmrg: "netmrg: FTBFS with multiarch net-snmp"
  apply patch from Ubuntu / Colin Watson, upload to DELAYED/2
• #709183 librep: "librep: FTBFS on kfreebsd-amd64"
  apply patch from Steven Chamberlain, upload to DELAYED/2
• #710416 courier-filter-perl: "courier-filter-perl: postinst uses /usr/share/doc content (Policy 12.3)"
  fix installation of config file (pkg-perl)

What has always been annoying me with Xen, is that xentop is well a piece of shit! It just displays the current numbers of sectors or network bytes read / write. But as an administrator, what you care about, is to know which of your VM is taking all the resources, making your whole server starve. The number of sectors read/write since the VM has started is of very low importance. What you care about is to have an idea of the current transfer rate. And the same apply for networking. So, tonight, within few hours, I hacked a small python shell script using ncurses to do what I needed, which tells how much resources have been used over the last 5 seconds (and not since the VM started). This way, it is easy to know which VM is killing your server. The script is adapted to my own needs only. Which means that it works only for DTC-Xen VMs of GPLHost. In my case, VM uses exactly 2 partitions, one for the filesystem, and one for the swap, which means that I display exactly that. I m sure it wouldn t be hard to adapt it so that it would work for all cases (which would mean finding out what device uses a VM and getting the statistics from /sys using that information, instead of determining it using the name of the VM). But I don t need it, so the script will stay this way. Before writing this tonight, I didn t know ncurses. Well, it s really not hard, especially in Python! It took me about 2 hours to write^Whack the script (cheating on the dtc-xen soap server which I already had available).

In wheezy (and all other current linux distros) there is a well-hidden bug that might freeze LVM if deleting a snapshot fails. Doesn't sound too bad one would think, but actually, creating and deleting snapshots is what buildds do all the time, and so they run into that issue quite often. It's worse on some architectures than on others, and so some packages are currently not built as fast as we're used to. Details about the bug are in #659762. Anyone fixing that would (not only) do our buildd network a big help (and you don't need to know anything about autobuilders for fixing that).

I finally used my new alarm clock for it's intended purpose, to wake up far earlier than I would normally be inclined. I set the alarm with the "at" command, playing an ogg file previously downloaded off of libre.fm... and groggily awoke to music of my choosing. Admittedly, my search began looking for a replacement for an aging music player, which had no built-in speakers, so didn't work as an alarm clock. Some time ago, the headphone jack broke, so it had kind of functioned as a glorified clock with 6GB of storage and a very little screen... I wanted something more capable; one of my requirements was that it would be feasible to run Debian a well as being relatively portable. I ended with something a little overpowered for an alarm clock... Hello, openpandora! Sporting a 1GHz ARMv7, 512MB of ram, 512MB built-in storage, two SD card slots, wifi, nearly full keyboard, 800x480 touchscreen, useable mouse nubs, 10 hours of battery life under active use... so it's nearly a netbook replacement. Yay, feature creep! It's far lighter than my netbook and fits in the palm of my hand. It fits in my pocket, but barely. Didn't spend much time using the OS it came with, ngstr m. There was already a wiki page about running Debian squeeze using the armel port, but the hardware looked like it could run armhf. Used qemu-debootstrap to build a Debian Wheezy armhf image on an SD card, and figured out the relatively simple process of booting the kernel shipped with it with an alternate root device. And it worked! But the kernel was a litle bit older than what's was in Wheezy, at the time, so I spent a couple days getting the kernel to build from the latest linux sorce package in Debian. There was a pandora git reposiory for linux 3.2.39, and most of the patches applied cleanly(those that didn't seemed already applied with the newer version). After a few build cycles, managed to get a working kernel, though haven't yet trimmed down the kernel config to my liking. Ideally, it'll get sufficiently upstreamed to be able to use the omap kernel variant. Also put together a package that configures the keyboard map, touchscreen and some cpu frequency scaling settings. Still need to document all that. Spent a week without a laptop or other computer while traveling... and the openpandora running Debian Wheezy held up pretty well.

A while ago I migrated most of my packages from Bazaar to Git. The rest of the world has decided to use Git for version control, and I don't have enough reason to stubbornly stick with Bazaar and make it harder for myself to collaborate with others. So I'm moving away from a workflow I know and have polished over the last few years - including the various bzr plugins and other tools involved. Trying to do the same thing using git is frustrating and time-consuming, but I'm sure that will improve with time. In particular, I haven't found a good way to merge in a new upstream release (from a tarball) while referencing the relevant upstream commits, like bzr merge-upstream can. Is there a good way to do this? What helper tools can you recommend for maintaining a Debian package in git? Having been upstream for bzr-git earlier, I used its git-remote-bzr implementation to do the conversions of the commits and tags:

% git clone bzr::/path/to/bzr/foo.bzr /path/to/git/foo.git

One of my last contributions to bzr-git was a bzr git-push-pristine-tar-deltas subcommand, which will export all bzr-builddeb-style pristine-tar metadata to a pristine-tar branch in a Git repository that can be used by pristine-tar directly or through something like git-buildpackage. Once you have created a git clone of your bzr branch, it should be a matter of running bzr git-push-pristine-tar-deltas with the target git repository and the Debian package name:

% cd /path/to/bzr/foo.bzr
% bzr git-push-pristine-tar-deltas /path/to/git/foo.git foo
% cd /path/to/git/foo.git foo
% git branch
*  master
   pristine-tar

Included in Debian Edu / Skolelinux, there are quite a lot of educational software. Created to help teachers teach, and pupils learn. We have tried to tag them all using debtags use::learning and role::program, and using the debtags I was happy to be able to create a collage of the educational software packages installed by default, sorted by the debtag field. Here it is. Click on a image to learn more about the program. field::arts field::astronomy field::biology:structural field::chemistry [viewmol] field::electronics [gpsim] field::geography field::linguistics field::mathematics [geomview] field::physics field::TODO In total, 61 applications. 3 of them lacked screen shots on screenshot.debian.net. If you know of some packages we should install by default, please let us know on IRC, #debian-edu on irc.debian.org, or our mailing list debian-edu@.

control-archive is the software that maintains the archive of control messages and the active and newsgroups files on ftp.isc.org. It had been a couple of years since I'd done a new release for the record, although there aren't many changes. There are a few minor fixes to the code and documentation, plus hierarchy updates for a few hierarchies. You can get the latest release from the control-archive distribution page.

PHP 5.5 rc2 was recently uploaded to Debian unstable by Ond ej Sur . Along side the new PHP version and its features the PHP team has dealt with the JSON extension being non-free because of the The Software shall be used for Good, not Evil license (see debian #692613).

php5 (5.5.0~rc1+dfsg-1) experimental; urgency=low
    [...]
  * Add dfsg-repack.sh script to remove non-free JSON module
    (Closes: #692613)
  * Remove php5-json from Provides, since that's no longer true
 -- Ond ej Sur  <ondrej@debian.org>  Fri, 17 May 2013 14:41:41 +0200

A package with a DFSG replacement for the JSON extension is waiting in the NEW queue and should be available soon. The new JSON extension was done by Remi Collet (PHP developer and Fedora s PHP maintainer). Review and patches are welcomed, code is available at https://github.com/remicollet/pecl-json-c . I hope the PHP project will adopt the solution, after claiming they won t fix it themselves (see php.net #63520).
Filed under: Debian GNU/Linux, PHP