Software in the Public Interest is one of the oldest and best non-profit associations (and not a foundation!) that supports free and open source software development and it will hold a public board of directors meeting on Wednesday at 20:30 UTC (21:30 UK). SPI meetings are held on irc.spi-inc.org (the OFTC network) in channel #spi. See the OFTC site for more information about connecting to OFTC. The agenda for the meeting is available on the web. A single resolution regarding ankur.org.in joining as an SPI associated project has been received. Please, come along and support SPI and thank its volunteers for their efforts.

Every time I think I have them done this happens

Since my laptop features WWAN connectivity, I purchased a 3G mobile broadband plan from SFR, a French mobile phone company. I took the cheapest one available: 250 MB per month for 10 .

Optimizing transparent proxy by SFR While browsing, I was curious about the fact that there were multiple accesses to non-public IP, like 10.141.0.4. I was browsing the mobile version of Le Monde, a French daily newspaper. I enabled Chromium developer tools and watched what happened in the Network tab. All images were served from IP like 10.141.0.4. Looking at the HTML source code, I discovered that <img> tags were rewritten to point to those images. The smaller one were identical to the original one but the larger ones (more than 50 KB) were of reduced quality (and therefore reduced size). <img> tags were not the only one to be rewritten. <style> and <script> tags that were pointing to external resources (with a src attribute) were also rewritten to include the CSS or the javascript inline. Before:

<script src="http://s1.lemde.fr/js/lib/jquery/1.5.1/jquery.js"></script>

After:

<script oldSrc="http://s1.lemde.fr/js/lib/jquery/1.5.1/jquery.js" NG="replaced">
/*!
 * jQuery JavaScript Library v1.5.1
 *
 * Copyright 2011, John Resig
 * Dual licensed under the MIT or GPL Version 2 licenses.
 * http://jquery.org/license
 * [...] */
</script>

This means that each page that includes jQuery, a popular javascript library, as an external resource, will be turned into a page that includes the whole jQuery code which is about 85 KB. Each page will weight more than 85 KB instead of relying on an external and cacheable resource. Nowadays, most browsers support HTTP keep-alive which allows to serve multiple resources into a single request. There are two cases:

1. If the resource is served from the same server as the HTML page, keep-alive would enable to get almost the same result without hindering the cache: the browser would first request the HTML page, then the javascript resource into the same TCP connection. There is a small penalty of about 100 ms to let the browser request the second resource. The browser may decide to open another connection because the initial document is not loaded yet. Some browsers may even negotiate this additional connection before needing it. In this case, it can use this idle connection at no cost. Otherwise, we may fall in the second case.
2. If the resource is served from another server (like in the example above) or if the browser does not have an idle connection ready for reuse, it would download the resource in parallel with the help of another connection. It will only costs about 300 ms of additional time due to the establishment of a new TCP connection with the server. Those 300 ms are a very small price to pay compared to the 2 seconds needed to download 85 KB for each page. Moreover, other static resources will likely be downloaded from the same server and therefore, this is 300 ms the first time and about 100 ms for additional resources (round-trip time for a new HTTP request into the same TCP connection).

UPDATED: In a previous version of this article, I confused pipelining and keep-alive mechanisms. While keep-alive allows to reuse an existing connection to download a new resource, only one request at a time by connection could be issued. Pipelining allows to push additional requests in the same TCP connection without waiting for answers. As noted by ric Daspet in a comment, pipelining is disabled for most browsers. However, keep-alive is well supported. Searching on the web for oldSrc and NG="replaced", I find very little information about what is used. It seems that such a feature may also break some scripts, like TinyMCE, trying to get additional resources relative to the path where the script is installed. It appears that SFR is using a product from Flash Networks called Harmony Gateway. I have asked SFR more info about this and if it was possible to disable the use of this feature but got no answer so far. I have retried a bit later and there was no rewriting for Javascript and CSS. Images were still degraded and served by a special server. The next day, my requests did not travel through a transparent proxy anymore (no more Via header in answers). Each try was done at a different location. The proxies may be installed at some locations only and configured differently. Another explanation is that SFR silently disabled this functionality for my account. If it was done correctly, such an accelerator service could be be proposed as an opt-out option since most users would appreciate faster page downloads. If you care about network neutrality, you can disable this option. Of course, it needs to be properly advertised, for example as a free option in your mobile plan.

Setting up our own optimizing proxy Google provides mod_pagespeed as a solution to optimize a website on the fly. This is an Apache module. It is targeted at optimizing your own website but since Apache can act as a regular proxy, it seems possible to enable mod_pagespeed for any website. That s pretty easy to setup if you don t mind using a precompiled version of mod_pagespeed. Download mod_pagespeed. I have downloaded the 64bit version for Debian.

# touch /etc/default/mod-pagespeed
# dpkg -i ~/download/mod-pagespeed-beta_current_amd64.deb
Selecting previously deselected package mod-pagespeed-beta.
(Reading database ... 277699 files and directories currently installed.)
Unpacking mod-pagespeed-beta (from .../mod-pagespeed-beta_current_amd64.deb) ...
dpkg: dependency problems prevent configuration of mod-pagespeed-beta:
 mod-pagespeed-beta depends on apache2.2-common; however:
  Package apache2.2-common is not installed.
dpkg: error processing mod-pagespeed-beta (--install):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 mod-pagespeed-beta
# apt-get install -f
[...]
Unpacking apache2.2-common (from .../apache2.2-common_2.2.19-1_amd64.deb) ...
Processing triggers for man-db ...
Setting up apache2.2-common (2.2.19-1) ...
[...]
# apt-get install apache2-mpm-worker
[...]
Unpacking apache2-mpm-worker (from .../apache2-mpm-worker_2.2.19-1_amd64.deb) ...
Setting up apache2-mpm-worker (2.2.19-1) ...
Starting web server: apache2.
# a2enmod proxy_http
Considering dependency proxy for proxy_http:
Enabling module proxy.
Enabling module proxy_http.
To to activate the new configuration, you need to run:
  /etc/init.d/apache2 restart

Then, we modify /etc/apache2/mods-enabled/proxy.conf:

ProxyRequests On
<Proxy *>
    AddDefaultCharset off
    Order deny,allow
    Deny from all
    Allow from 127.0.0.1
</Proxy>

Of course, this makes little sense to run the proxy on your laptop since you don t want to download the non-optimized content over your WWAN link. You need to run it on another server and therefore allow your laptop to access it. Also, we need to modify /etc/apache2/mods-enabled/pagespeed.conf to add the directive ModPagespeedDomain * to let mod_pagespeed rewrite any page. You should restart Apache after that. Now, check if the proxy is working:

$ curl -six http://localhost:80 http://www.lemonde.fr/   grep ^X-Mod-Pagespeed
X-Mod-Pagespeed: 0.9.17.7-716

All is working! Let s try a very simple site, like Debian Backports. It features one HTML file and two CSS files: We downloaded a total of 8 KB (compression included). Let s use our proxy and try again: mod_pagespeed succeeds in reducing the size of the HTML file and combined the CSS files. Moreover, it minified the resulting CSS file. There seems to be a bug in Chromium that display incorrect transfert sizes (including in the timeline representation) when using a proxy. I have checked with Wireshark that the overall quantity transferred was about 8 KB. The gain is very thin here but you can imagine that you will get better results on larger sites. mod_pagespeed seems to handle well most sites. However, it only optimizes pages that already are in its cache. The first time you grab a page, you will get the unoptimized version. I don t know if it is the expected behaviour.

I've been wondering for a long time about how to record sound which is currently being played on the computer. I know the ALSA framework is powerful enough to do that, and today just proved me so (at least to some extent). It turns out that it's pretty trivial using the file PCM plugin; all you need is a stanza like this in your $HOME/.asoundrc file:

pcm.save  
    type file
    format wav
    slave.pcm front 
    # my main PCM is called front
    file output.wav
 

Afterwards, you can use this save PCM with the tools that understand it, such as aplay or mplayer:

~ aplay -Dsave stuff.wav
~ mplayer -ao alsa:device=save stuff.wav

(for xine users, you may want to have a look at the audio.device.alsa_front_device configuration element). Funnier, though, is the possibility to use it with programs that don't provide such an easy way to change that, (say, the flash player in a browser ?) via the use of the default PCM specification in $HOME/.asoundrc:

pcm.!default save

After that, all programs using the default PCM will save the sound they play as a save.wav file in their current directory - but nothing stops you from specifiying an absolute directory. Better yet, you may apparently use pipes in stead of the file name, la popen.

Anyone who's been following anything I've written lately may be under the impression that I dislike EFI. They'd be entirely correct. It's an awful thing and I've lost far too much of my life to it. It complicates the process of booting for no real benefit to the OS. The only real advantage we've seen so far is that we can configure boot devices in a vaguely vendor-neutral manner without having to care about BIOS drive numbers. Woo.

But there is something else EFI gives us. We finally have more than 256 bytes of nvram available to us as standard. Enough nvram, in fact, for us to reasonably store crash output. Progress!

This isn't a novel concept. The UEFI spec provides for a specially segregated are of nvram for hardware error reports. This is lovely and not overly helpful for us, because they're supposed to be in a well-defined format that doesn't leave much scope for "I found a null pointer where I would really have preferred there not be one" followed by a pile of text, especially if the firmware's supposed to do something with it. Also, the record format has lots of metadata that I really don't care about. Apple have also been using EFI for this, creating a special variable that stores the crash data and letting them get away with just telling the user to turn their computer off and then turn it back on again.

EFI's not the only way this could be done, either. ACPI specifies something called the ERST, or Error Record Serialization Table. The OS can stick errors in here and then they can be retrieved later. Excellent! Except ERST is currently usually only present on high-end servers. But when ERST support was added to Linux, a generic interface called pstore went in as well.

Pstore's very simple. It's a virtual filesystem that has platform-specific plugins. The platform driver (such as ERST) registers with pstore and the ERST errors then get exposed as files in pstore. Deleting the files removes the records. pstore also registers with kmsg_dump, so when an oops happens the kernel output gets dumped back into a series of records. I'd been playing with pstore but really wanted something a little more convenient than an 8-socket server to test it with, so ended up writing a pstore backend that uses EFI variables. And now whenever I crash the kernel, pstore gives me a backtrace without me having to take photographs of the screen. Progress.

Patches are here. I should probably apologise to Seiji Aguchi, who was working on the same problem and posted a preliminary patch for some feedback last month. I replied to the thread without ever reading the patch and then promptly forgot about it, leading to me writing it all from scratch last week. Oops.

(There's an easter egg in the patchset. First person to find it doesn't win a prize. Sorry.)

I finally booked my flights to Banja LukaZagreb:

Sun 17 Jul 08:10 Innsbruck 09:20 Vienna    OS916
Sun 17 Jul 12:55 Vienna    13:45 Zagreb    OS683
Sun 31 Jul 15:10 Zagreb    16:05 Vienna    OS684
Sun 31 Jul 17:15 Vienna    18:15 Innsbruck OS901

See you at DebConf11!

Continuing my earlier musings about test driven distro development, and the tools that would require. I imagine something like this:

• create a virtual machine with a particular configuration of stable
• run a test suite verifying the VM works
• upgrade the VM to testing and reboot
• re-run test suite, verifying things work in testing

We might also want this scenario:

• install testing directly in a VM (instead of upgrading from stable)
• run test suite

There might be other scenarios that would be useful to test as well. Even from these two, it's clear there's a need for at least three separate tools:

• create a machine image with a particular configuration
  • might be based on stable or testing or unstable
  • might also be using some other package sources
  • might want particular packages installed
  • might want particular configuration settings
• run a suite of tests against the running image
  • which tests to run will depend on the scenario
• run various scenarios using the above two tools
  • needs reasonably easy ways to specify scenarios
  • needs to support tests specific to scenarios

I've written a rudimentary version of the first tool: vmdebootstrap. I've since learned there's a bunch of others that might also work. There's room for more: someone should write (or find) a tool to make a snapshot of a real system and create a VM image that mimicks it, for example. Anyway, for now, I'll assume that one of the existing tools is good enough to get started. For the second tool, I wrote a quick-and-dirty proof-of-concept thing, see systest.py. Here's a sample of how it might be used:

liw@havelock$ ./systest -v --target 192.168.122.139 --user tomjon
test 1/6: cat
test 2/6: only-ssh-port
ERROR: Assertion failed: ['22/tcp', '139/tcp', '445/tcp'] != ['22/tcp']
[status 1]
liw@havelock$ ./systest -v --target 192.168.122.139 --user tomjon \
    cat ping6-localhost ping-localhost simple-dns-lookup ssh-login
test 1/5: cat
test 2/5: ping6-localhost
test 3/5: ping-localhost
test 4/5: simple-dns-lookup
test 5/5: ssh-login
liw@havelock$ 

The first run failed, because the VM I'm testing against has some extra ports open. Some of the tests will require logging into the machine, via ssh, and for that one needs to specify the user to use. systool may overlap heavily on system monitoring tools, and possibly the production implementation should be based on those. I think it's best to design such a tool for the more general purpose of testing whether a system currently works rather than as an integrated part of a more specific larger tool. This lets the tool be useful for other things than just testing specific things about Debian. (The production implementation would then need to not have all the tests hardcoded, of course. SMOP.) The third tool I have not spent a lot of time on yet. One thing at a time. Given these tools, one would then need to decide how to use them. The easiest way would be to use them like lintian and piuparts: run them frequently on whatever packages happen to be in testing or unstable or experimental, and put links to the test reports to the PTS, and hope that people fix things. That is the easiest way to start things. Once there's a nice set of test cases and scenarios, it may be interesting to think about more aggressive ways: for example, preventing packages from migrating to testing unless the test suite passes with them. If the tests do not pass, one of four things is broken:

• the package or packages in question
• other packages already in testing
• the tests themselves
• the test environment

If things are set up properly, the last one should be rare. The other three always require manual inspection: it is not possible to automatically know whether the test itself, or the code it tests, is at fault. it is, however, enough to know that something is wrong. If the tests are written well, they should be robust enough to not be the culprits very often. (Someone wanting to make a rolling distrbution, or even better, making a monthly mini-release, might benefit from this sort of automated testing.)

... that's what I think of the work of the FTP Team. Three uploads to NEW in one day, all of which processed in less than half a day. Truly impressive ! Many thanks to them !

Whenever someone asked me about a calendar application, especially for the textmode, I always encouraged them to give pal a try. I always loved the looks of it, the interactive mode is helpful, it has HTML output format to inject the calendar into a webpage, mail output format for a daily reminder cronjob, and other useful features. I even created a file with the Austrian holidays for it which got included in the original project for the benefit of all its users.
If you haven't tried it yet and are looking for a calendar tool with support for very flexible recurring events and categories, this might be a good look. I am still happy with pal, though someone recently suggested a different tool on IRC, and that was wyrd. From a quick glance it looked promising, so I started to dig into it. My first task was to convert the former mentioned pal file for the Austrian holidays into remind format. remind is the backend for wyrd, and its definition language seems to be extremely powerful. It though took me a while to figure out how to put in Easter date related events into it, the examples weren't really hinting me in the right direction. This is part of what I am using now:
REM [trigger(easterdate(current())-47)] +6 TAG noweight MSG Faschingsdienstag %b The look and view of wyrd is different to pal in several ways. Where the granularity of pal is a pure day view, wyrd scales in hours (or half, quarter thereof). Also, wyrd offers the possibility to color the days differently by busy level. Of course it's possible to exempt tasks from adding weight to a day. pal on the other hand is able to color events differently by category. Decide yourself what you actually need, test it, and ... enjoy!

/cli permanent link Comments: 0 Flattr this

As usual many people requested travel sponsorship to attend DebConf 11 and are now waiting eagerly on the outcome of their request. We are sorry, that we can t give you much details yet. To give you an idea of the time line, here is a small explanation of the process. About 100 people requested travel sponsorship. In total that would about 50 000 Euro. Problem is currently we don t exactly know, how much money we may spend on travel sponsorship ourselves (By the way: If a potential sponsor is reading this, please contact us; DebConf also accepts donations). So, we formed a team of 13 persons (the mail only lists twelve, one was added later). The team members will start to rate each and every single request in two criteria: Contributor Level and sponsorship amount. With limited amounts we obviously want to sponsor those first, who contribute a lot to Debian, preferably if they don t travel first class. After that s done, we ll have a long, but ordered list, but the team still needs to meet and discuss different cases and similar. Our team isn t really small and consists of people living all over the world, it s quite hard to actually schedule a meeting everyone can attend. (Which is not a bug, but a feature: More people from more countries should in the end result in a fair rating of the sponsorship requests.) So, in the end, we ll meet on 19th of June. Sounds late, please keep in mind, that every team member has to rate all sponsorship requests (including some research) and all of them are volunteers with actual daytime jobs. But even if you don t hear anything from us short after that date, don t panic! As said we really don t know, yet, how much money we can spend on travel sponsorship. Our sponsor-hunting-team is constantly working on the acquisition of more money, so every now and then the budget for travel sponsorship gets increased, meaning that we can mail a couple of more people on our ordered list. So it might take a while, till we get to all, but we try very hard to do so.

I m just back from spending a few days volunteering with the tornado recovery effort in Joplin, MO. The biggest image that remains in my mind is of the first time I saw a person picking through a large pile of rubble. The person was standing on top of what used to be a house. Now it was a pile of wood, glass, carpet, siding, and roofing material. I m sure there was hope for finding some treasure or other maybe a photo album or videos of children. In any case, it made me feel so lucky, even unfairly lucky, to have not had to go through that. This scene was repeated several times, but mostly the houses that devastated appeared abandoned by the time we were there, now two weeks after the event. But I heard stories, and lots of them. The victims of the storm, who were perhaps trying to rebuild that part of their house that got smashed by a tree or a pickup, or trying to get their intact belongings out before abandoning the house, or whatever, were mostly surprisingly upbeat. They were working out in 95-degree heat, many without electricity, running water, or sewer service. Almost every person I met that suffered a loss from the tornado wanted to tell their story. Many also told of their plans for the future, which were full of hope and even upbeat. These were people doing a hard job in terrible conditions and still showing hope. Another testament to the disaster was the most unusual set of vehicles you ve ever seen parked at any hotel you can care to think of, for at least an hour-and-a-half radius in the direction I came from. Besides the usual cars and minivans, there might be FEMA vehicles, electric company trucks, Red Cross vans, construction trucks of every kind, police and law enforcement from all over, etc. There was quite obviously an influx of people helping out in Joplin. My primary task there was to provide communications support for the effort as an amateur radio operator. Amateurs (or hams ) are something of a volunteer first responder of sorts during times of crisis; most of us own and are very familiar with operating equipment that can communicate over very long distances without the need for any on-site infrastructure. Amateur radio was the only method of communication for some Joplin hospitals in the immediate aftermath. The communications emergency is over, but the response isn t. I was assigned to work with the Salvation Army. They were doing a lot of things in Joplin, and had hundreds of volunteers working with them. I don t think I even know what many of them were doing I do know they had set up several warehouses across the city working with donating clothing, food, etc. The part I was involved with was primarily the canteen operation. The SA sent in food service trucks from several parts of the country. These trucks would roam up and down the streets in the damaged area, trying to get past every single street several times a day. Anybody that we could see would be offered food and water. No strings attached, no questions asked. This included homeowners, electric line workers, construction crews, sanitation workers, and quite a few nonprofit groups that sent well-meaning and useful volunteers into the area but didn t think to provide them with a large supply of water due to sending them into an area without any. Oops. In any case, with extreme heat and no running water, conditions were dangerous. The canteens also knew of certain at-risk families that were living in homes that were mostly intact in the disaster areas, and made a special point to check in on them. They also generally looked to make sure that people looked like they were healthy. Each canteen also had a counselor on board that would visit with people while we quickly prepared their meal they all seemed to welcome that. Amateur Radio s Role The operation of this size had quite a logistics challenge. I d hear of things like an unexpected need of 70 lunches, or a semi showing up with donations before there was space, or an unexpected but very welcome donation of a large quantity of ice cream without a place to store it (so the canteen trucks, which have freezers, needed to pick it up quickly). That s where us hams came in. Each canteen had an amateur radio operator on board. Each major location also had a ham stationed there, and the head of operations also often had a shadow a ham that would follow him around wherever he went to relay messages back and forth. We also had hams with pickups (with radios in them, of course) that could transport things around the city to places that needed them, hams at headquarters managing all the communication and generally investigating questions that didn t have immediately obvious answers, etc. Radios were used instead of cellphones for a few reasons. One big one is that everybody on the operation can hear what everyone else s needs are, since it s a group communications situation rather than one-to-one. It s easy to give a general alert to everyone ( come get your ice cream now please! ) and people that have suggestions can chime in. This came in extremely handy more than once. Also, it frees the people doing other jobs from having to spend time chasing someone s voicemail, finding phone numbers, etc; that gets delegated to us in some cases. I heard from the head of canteen operations, for whom this was the first disaster he d worked that had amateur radio support, how wonderful it was to have this going on. I also heard a secondhand report that some police officers that were also amateurs had listened to our operation and reported that we sound more professional than 911 dispatch and do a better job. On Sunday I was assigned to a canteen. This meant I didn t have a lot of radio traffic to pass, so although I had it in my ear all day, I wasn t actively talking on the radio very much. So I rode in back, helping hand out water, carry meals to people, and so forth. On Saturday, I was the shadow for the head of operations. That was a difficult task, because he barely ever moved at a pace slower than a run, sometimes would abruptly zip out somewhere, etc. But it was also enlightening and vital. He was a real get it done sort of guy, and was the key to quite a few things. Having someone available to relay questions to and from him was a good thing. And today I worked as a transport person and at headquarters. Due to not having a pickup there, I didn t actually get called on much to transport things, but in general between jobs the whole time I d act as a runner if needed, or simply try to figure out the details of how things were run for next time. I wound up taking net control (being the control operator at headquarters, and generally managing communications so that people don t talk over each other and such) for about an hour. So I got to do a little of just about every amateur radio task. Thoughts I am thankful for the opportunity to go, and the good feeling of helping people in need the first I ve ever had the chance to do that in a disaster. It s a good feeling to have a skill that is useful and appreciated. Sometimes it felt like handing out food and water is something pretty small in the scheme of things. But on the other hand, it gives people a chance to have contact with someone that cares, an opportunity to have people that can notice problems drive by a few times a day, and an opportunity to help meet people s basic needs. And sometimes in a fluid situation, there might be more volunteers than are needed, so I did spend some time sitting waiting for the next task. But overall, I m convinced that the work I helped facilitate was a good thing and provided a good and needed service in Joplin. This has been quite the experience and I m sure it s changed me too, though I don t yet know how.

DECLARATIVE DIVERSIONS ====================== Introduction A diversion is when it is possible to have dpkg not overwrite a file when it reinstalls the package it belongs to, and to have it put the file from the package somewhere else instead. Declarative diversions involves a new control file with a declarative syntax which dpkg will parse and process directly as part of the package unpack and removal phases, eliminating the problems resulting from non-atomic handling of diversions. Topics There are a number of topics involved in implementing this kind of project * What syntax do we use for the new control file? * What dpkg does with the control file * How do we handle diversions to a non-existant directory? * How do we order unpacking a new package which adds a diversion? * How do we order unpacking a new package that removes a diversion? * How do we order removing a package which had a diversion? * How do we handle errors? * What happens to dpkg-divert? Details - Control File Syntax - It will conform to RFC2822 style with the following format: * Divert-From: * Divert-To: * Blank lines and lines beginning with # will be comments Divert-To will be optional and if it is ommitted then files being diverted will have their filename changed to file.distrib The above style has it s advantages, one of the main ones being that there is no need to escape whitespace within filenames. It also allows for more commands to be added at a later date. Details - Control File Handling - Within control.tar.gz the file should be named diversions This file is then copied to /var/lib/info/$package.diversions We need to store the control file for the currently-installed package so that we can detect declarative diversions that have been removed from the control file between versions (On-disk version and the to-be-unpacked version). If there is no previous version of a control file then all current diversions within the package are left in place and any new ones declared within the control file are added. If there is a previous version of the control file then the diversions listed within the file are compared with the diversions currently in place. If a diversion exists that is no longer listed in the control file then it is removed. If there is a diversion listed in the control file that doesn t currently exist then it is added. Details - Handling Diversions to non-existant directories - Diverting files to directories that don t exist can cause a number of problems. If the package does not own the directory it may be left orphaned on removal of the package The package is responsible for ensuring the availability of the target directory in the unpack phase. If a declarative diversion attempts to divert a file to a folder that neither exists already on the filesystem at unpack time, nor is shipped within the package, dpkg will return an error at unpack time. Details - Ordering Requirements - =>Unpacking a new package that adds a diversion *1. Check if package has a previous diversion control file *2. Copy control file to tmp directory *3. Calculate diversions to be added *4. Unpack folders *5. Add diversions *6. Unpack files *7. Copy control file to /var/lib/dpkg/info/$package.diversion *Performing diversions this way avoids trouble with creating folders and accidentally overwriting the wrong version of files. *Another major problem would be extracting a new version of the file and overwriting the old one. *This might cause the wrong version of the file to be diverted and the package to break. =>Unpacking a new package that removes a diversion *1. Check if package has a previous diversion control file *2. Copy control file to tmp directory *3. Calculate diversions to be removed *4. Check if dropped diversion results in a file conflict and abort the unpack if it does. *5. Unpack folders *6. Unpack files *7. Remove old files no longer needed in the package *8. Remove diversions *9. Copy control file to /var/lib/dpkg/info/$package.diversion *If a package has no previous diversion control file then no diversions will be removed and existing diversions can only be removed using dpkg-divert. *If removing a diversion results in a file conflict then dpkg will return an error at unpack time before anything is unpacked and the unpack is aborted. =>Removing a package which had a diversion *1. Remove files *2. Remove diversions *3. Remove folders *4. Remove control file at /var/lib/dpkg/info/$package.diversion *This ensures that all files, diversions and folders are removed correctly Details - Error handling Errors in diversions will have to handled with a great deal of care due to the fact that if they are not the package could be broken. This means that a great deal of checks must be done to ensure that all the files can be diverted properly before any actual diverting takes place. If they can t the package installation/update must be stopped and rolled back to avoid the package being installed incorrectly or broken. Details - dpkg-divert - When we impliment the new diversion method we should keep the current dpkg-divert. This allows maintainers to catch up with the new method without breaking their packages. It also allows maintainers to perform some operations that aren t support by the new method. All current functionality of dpkg-divert will be left intact for use by sysadmins and for maintainers to catch up with the new system. The two systems will work together without directly conflicting with each other. Example Usage #1 The file to be diverted is /usr/share/foo It needs to be moved to /usr/share/bar The syntax of the control file would be: #Start File Divert-From: /usr/share/foo Divert-To: /usr/share/bar #End File Example Usage #2 In this example the maintainer doesn t want to move the file to any specific folder The syntax of the control file would be: #Start File Divert-From: /usr/share/foo #End File This would divert the file to /usr/share/foo.distrib Footnotes - RFC2822 Guide: * http://www.faqs.org/rfcs/rfc2822.html First Email Thread on Declarative Diversions (First Message in Thread): * http://lists.debian.org/debian-dpkg/2011/05/msg00102.html Email thread on first design draft (First message in Thread): * http://lists.debian.org/debian-dpkg/2011/06/msg00019.html Declarative Diversions Wiki: * http://wiki.debian.org/SummerOfCode2011/DeclarativeDiversions My Blog for this Project: * http://blog.sam-dunne.com

Jo bought me a voucher last Christmas to spend some time on a race track, driving an Aston Martin and a Lamborghini. Finally we got around to using it last week. Jo is just as much a petrol-head as me, so I surprised her by buying the same experience voucher for her - call it a wedding present. Good call - she didn't stop grinning all day, even on the long drive home! The Aston Martin was a DB9 Volante (i.e. the convertible). Lovely car, but a little softer than I'd expected (maybe due to it being the convertible). I preferred the Lamborghini Gallardo, pictured here in a nice subtle colour. *grin* The nice folks doing the official photos at the track also took some extra photos of us sitting in another supercar - an Audi R8 V10 convertible. We've got those photos coming on CD in the post, and I'll post one when they land.

So after getting feedback on the design spec from various people I am working on a revised edition. This should be posted within a couple of hours and is hopefully a lot better than the first. Looking forward to the feedback ~Sam

(First posting to Pl net Commandline! Tag: pcli) Vutral asked in IRC how to synchronise two shells environment while they re running. As you may know, POSIX systems cannot change a process environment vector after it has been started, only the process itself can. Well, the shell can, and we ll use a variety of things for this. This trick assumes you have $HISTFILE set to the same pathname in both shells (obviously, they run under the same user). It uses export -p to render the current list of exported variables, then transforms the list from newline-separated to a single big one-line export statement.
Then it transforms all remaining newlines (which will be part of a single-quoted string, since that s mksh(1) s export format) into the sequence '$'\n'' which means: terminate current single-quoted string, append $'\n' and open up a new single-quoted string immediately; concatenate these three.
Now, $'\n' is just a fancy way of saying newline, and part of mksh because David Korn (yes, the Korn in Korn Shell) strongly suggested to me that this functionality be included but, as we can see here, it pays off.
Finally, the so transformed string is prepended by unset \$(export); which, when executed, will cause the shell to unset (and unexport) all currently exported variables. The shell parameters that are not exported, i.e. not in the environment, are not affected by this code (except for $x and $nl, but whatever).
This string is then passed to read -s (plus -r and clearing IFS to enable raw mode), which means, read into the parameter $REPLY (which we conveniently don t use but it s trashed too, thus) but store into history at the same time. Ah hah! Now, the persistent history feature comes into effect! After running the below statement in the source shell, switch into the terminal running the destination shell, press Enter once on the empty line (Ctrl-U to empty it if it wasn t), then Cursor-Up ( ) to recall voil , an insanely large line with the previously created string sorta expanded and press Enter again to run it. Now your set of exported parameters is the exact same (minus if you exported IFS, nl, x or REPLY) as in the source shell.

I ve added extra spaces and a linewrap below, this is really just one big line:

Of course, this makes a nice function, for your ~/.mkshrc or somesuch.

Grabbed a shot of my brother's Pug trying to keep cool in the garden at the weekend.

Never change a running system. Old but true saying, but sometimes there's no other chance. Until a few days ago I was happy with SSL vhosts running with a single SSL certificate. Then I needed to add another SSL certificate for another site with several subdomains like svn.site-A.de, trac.site-A.de and www.site-A.de. With Apache2 running on Squeeze it's possible to make use of Server Name Indication (SNI) mechanism in order to be able to use multiple SSL certs on a single IP based vhost setup. Well, it works for some client software, but apparently it does not work well with korganizer or Firefox Sync plugin nor with Cyberduck on OS X. Here's an example config:

SSLEngine on
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
SSLCertificateFile /etc/apache2/ssl/site-A-cert.pem
SSLCertificateKeyFile /etc/apache2/ssl/site-A-key.pem
SSLOptions StrictRequire
SSLProtocol -all +TLSv1 +SSLv3
SSLCipherSuite HIGH:MEDIUM:!aNULL:+SHA1:+MD5:+HIGH:+MEDIUM
SSLVerifyClient none
SSLProxyEngine off

This is identical to all SSL vhosts on my system. The funny thing is now that it works for two sites (site A and site B) while it doesn't work for site C. In Firefox Sync plugin I get an error that the connection couldn't be established while on Cyberduck (a webdav client for OS X) I get a requester stating that I get cert for site A on site C. Pointing the browse to the appropriate URL I get the correct cert for site C on site C. Is there anything I miss with SNI setup in Apache?

When soliciting for opinions, recently, I discovered that the python-based fabric tool was not dead, and was in fact perfect for my needs. During the process of getting acquainted with it I looked over the source code, it was mostly neat but there was a trivial (low-risk) symlink attack present. I reported that as #629003 & it is now identified more globally as CVE-2011-2185. I guess this goes to show that getting into the habit of looking over source code when you install a new package is a worthwhile thing to do; and probably easier than organising a distribution-wide security audit </irony>. In other news I'm struggling to diagnose a perl segfault, when running a search using the swish-a perl modules. Could it be security worthy? Possibly. Right now I just don't want my scripts to die when I attempt to search 20Gb of syslog data. Meh. ObQuote: "You're scared of mice and spiders, but oh-so-much greater is your fear that one day the two species will cross-breed to form an all-powerful race of mice-spiders who will immobilize human beings in giant webs in order to steal cheese. " - Spaced.

* We re doing heavy uploading and fixing in Debian-IN repository nowadays. Thanks to Vasudev for taking initiative on Git migration + various nice stuffs pending from long time. * Gwibber is till mess. Help wanted as usual! * And, MiniDebConf India at Pune is planned at Pune again. Unless, it is colliding heavily with my wife s plan, I ll be there as last time

I ve gotten a couple of complaints on the amount of daemons that is launched by default by any app built upon the KDE Frameworks, so I investigated one of them a bit. The knotify daemon currently seems to do two things:

1. Keeping some dependencies out of KDE Framework libraries
2. Recieve messages from dbus, parse them, encode differently and send them on to other places over dbus

In the past, the knotify daemon was also responsible of showing popup-messages. So, I did a quick experiment here in Switzerland to try kill off that daemon and merged the code from the daemon into the KNotification part of kdeui. As a proof of concept it works, and if I can cut out some of the newly introduced dependencies it can end up being really nice. There is much less need for a separate daemon these days where knotify doesn t actually paint the popups itself, but rather passes them on to a galago-spec implementing part of the workspace (on linux) or to growl for various non-free platforms.