Last week was a rather slow week. I did some work on allowing the dashboard to store and process information that does not come from the BTS. This exposed a few issues with the way we were storing and presenting information. After talking to Matt, we decided that the dashboard should deal with task objects. These tasks would contain basic information such as a name, person assigned to it, status, and possibly a link to a bug. This means that we would be able to start up a new DEX project by simply choosing a project name and creating a basic text file containing a list of tasks (package names, patches, bugs, etc). I also have been trying to get a hold of some people who participated in the last DEX project. I am interested in getting some more feedback about what went well, problems they encountered, and tools that might have made things easier. I am also talking to some people who are currently working on starting up new DEX projects to try and make dextools align with as many workflows as possible. If you fall into either of these categories, I would really appreciate it if you contact me. There is also a new wiki page that I created so that it is easier for people who are not interested in me or the Google Summer of Code to quickly see what dextools is about and monitor its progress. I will be adding more information over the next few days. Currently, I am continuing to work on the dashboard and talking to people about their DEX experiences. I have also begun working on some scripts to generate graphs that will go on the dashboard. These graphs will allow us to quickly check our progress on different projects. They will also allow us to see and acknowledge individuals working on the tasks. I will add more information about these graphs to this blog and to the dextools wiki page.

Last week was a rather slow week. I did some work on allowing the dashboard to store and process information that does not come from the BTS. This exposed a few issues with the way we were storing and presenting information. After talking to Matt, we decided that the dashboard should deal with task objects. These tasks would contain basic information such as a name, person assigned to it, status, and possibly a link to a bug. This means that we would be able to start up a new DEX project by simply choosing a project name and creating a basic text file containing a list of tasks (package names, patches, bugs, etc). I also have been trying to get a hold of some people who participated in the last DEX project. I am interested in getting some more feedback about what went well, problems they encountered, and tools that might have made things easier. I am also talking to some people who are currently working on starting up new DEX projects to try and make dextools align with as many workflows as possible. If you fall into either of these categories, I would really appreciate it if you contact me. There is also a new wiki page that I created so that it is easier for people who are not interested in me or the Google Summer of Code to quickly see what dextools is about and monitor its progress. I will be adding more information over the next few days. Currently, I am continuing to work on the dashboard and talking to people about their DEX experiences. I have also begun working on some scripts to generate graphs that will go on the dashboard. These graphs will allow us to quickly check our progress on different projects. They will also allow us to see and acknowledge individuals working on the tasks. I will add more information about these graphs to this blog and to the dextools wiki page.

As a freenode staffer and Ubuntu Operator, a frequent question that I am asked is, Why can t I join #channel? or Why can t I talk in #chanel? . To answer this question, I normally check the ban and quiet lists for #channel, and then I check what channel modes are applied. For small channels, this is not too bad, but for channels with very large ban/quiet lists, it can take some time. This is why I have created bansearch.pl. This irssi script will attempt to determine why a user is unable to join/talk in a channel. Currently, the script is maintained in a Launchpad bzr branch. To get and install the script, run the following commands from your home directory. bzr branch lp:~nhandler/+junk/bansearch
mkdir -p ~/.irssi/scripts/autorun
ln -s ~/bansearch/bansearch.pl ~/.irssi/scripts/bansearch.pl
ln -s ~/.irssi/scripts/bansearch.pl ~/.irssi/scripts/autorun/bansearch.pl You can leave out the last command if you do not want the script to run automatically when you start irssi. Using the script is quite simple. You can use it like this: /script load bansearch.pl
/bansearch nick [#channel] If you do not specify a channel, it will use the current channel. The script will currently look for bans and quiets matching the user. It has some limited extban support, but it does not support CIDR bans. It also supports looking at channel modes that might prevent the user from joining/talking. The script knows whether the user is identified, so $a extbans, $~a, and cmode +r are all supported, and the script will only warn you about them if they match the user. The script was written for freenode, so while basic ban/quiet searching might work on other IRC networks, the channel mode checks will probably not work properly. The script is still fairly new, so there are still bugs in it. If you find any, please let me know, and I will try and patch them as quickly as possible. I am also open to any feedback about how to improve the script.

I ve been spending the past few days working on a dashboard for DEX. I ve been using Python, javascript, and jquery to create this dashboard (partially as a learning exercise). Earlier today, I started thinking that a visual mockup of the dashboard might be helpful, as it will allow me to create and revise a design based on feedback without having to touch the code. At this point, I had already written a fair bit of code and had a rough but working early version of a dashboard coded. So for the first mockup, I simply created it to match my code. Thanks to the generosity of the folks at Balsamiq, I was able to get myself a free key to create the mockup above. As I mentioned earlier, this is what the code currently looks like, not what it will look like. Next, I m going to be spending some time revising that mockup so I have a visual goal to work towards. I also will be cleaning up the code that I currently have and making it available in a public VCS.

Here is my first official Google Summer of Code report. As I still had classes, the first week of coding was rather slow. However, after my first phone call with my mentor, Matt Zimmerman, things started to pick up. I spent most of my time working on a dashboard/portal for the Debian Derivatives Exchange (DEX). We decided that the best approach to this would be to store as little information ourselves as possible. Instead, we will try to pull most of our information from other sources, such as Debian s BTS. The main advantage to this is that package maintainers do not need to change any of their workflows, but they still get to benefit from DEX s work. Currently, I have a python script that gets a list of usertags and bugs for debian-derivatives@lists.debian.org from the BTS. This script will end up getting run by cron. It stores the information it gets in static HTML files. Finally, JQuery and Javascript are used to present the final dashboard to the user. I created a mockup showing how this looks. I also have a functioning website that looks and acts like the mockup, but the code is not publicly available yet (see below). DEX uses Alioth to host its git branches and website. I have spent some time trying to familiarize myself with the recent changes that have taken place on Alioth, as well as trying to figure out the best way to store and execute my script to generate the website. So far, it looks like people are still deciding on the best way to handle having websites that are updated from VCS hooks (due to things being split between vasks and wagner). I will continue to look into this, and might try using an alternate host until things settle down. Next, I plan to work on updating the dashboard to provide a way to edit, store, and display certain extra fields of information. This is necessary for certain pieces of information that are not appropriate for adding to bugs and in a few other cases. I will also be taking some time to talk to a few people who were active contributors to the last DEX project. This will allow me to learn more about their experience, find out what changes they feel should be made to make things run smoother, and get feedback on my current Summer of Code work. Finally, although I have been posting updates on my Summer of Code work to my blog [2][3], Matt thinks that having all of this information in one place will be useful. As a result, I will be creating another wiki page about the project that will contain all of the information people need to know. This page will be updated as the summer progresses. I am also hoping to have my code available in a public VCS by the time of my next progress update so that people can check it out and provide feedback. In addition to the dashboard, I will be working on creating a few other tools to assist with the creation and tracking of new DEX projects. I will also be coming up with a way to generate graphs to visually allow us to track our progress on projects versus time as well as show the work done by people at the individual level. This last task is part of a goal to get more people involved in the DEX project. While we may only be a few weeks into the actual Google Summer of Code, I have already learned a lot of new things. I am looking forward to working more with Matt and other members of the community over the course of the summer.

It's been a long time since I last took some time to read First Monday A great online publication, if you are not familiar with it, that I would categorize (and no, I'm not probably well-informed in it to be authoritative) as dealing with social, psychological aspects of the cultural shifts the online world has brought upon us (often dealing with topics related to Free Software communities, the reason I first met the publication). Firstmonday is an Open Access champion from early on. It follows an approachable but academic format (this means, it is peer-reviewed, its articles give extensive lists of references, and the articles are not the short reads we have got used to finding on the net, but, quoting from their audience profile, English is not the first language of many First Monday readers; A large percentage of First Monday readers are not a part of academia; Cultures, educational backgrounds, and fields of study vary greatly among First Monday readers.) This means, it's at least a great publication for me to follow :) Anyway After a long time not following it, I have just read Reading revolutions: Online digital text and implications for reading in academe, by Barry W. Cull; First Monday, Volume 16, Number 6 - 6 June 2011 Nice, interesting read. As I was planning on telling about the article to a couple of friends more into the subjects than myself, I'll comment+quote some bits on it. Before going any further: The article makes several references to Maryanne Wolf. No relation to her I'm not lulling my (two? are you both still reading?) readers towards her work ;-) The article talks about the differences social, even dips into some physiological aspects that the activity of reading is sustaining due to the shift from an activity done mainly from books (or other similar printed material) to the computer screen. Of course, we all know from our own experience many of the basic traits Shorter attention spans, a different reading pattern (skimming instead of reading; browsing through several related items instead of in-depth reading a single text as a knowledge unit). The article begins with an overview of reading and humankind. Cull quotes Maryanne Wolf's phrase, despite the fact that it took our ancestors about 2,000 years to develop an alphabetic code, children are regularly expected to crack this code in about 2,000 days . An interesting point I never thaught of is the start of reading as a purely mental activity, detaching reason from verbalization, ~1200 years ago:

Interestingly, these early scribes first did their work by reading out loud to themselves. Not until the ninth century did monastic regulations begin requiring silent reading. By the thirteenth century the practice of men reading silently and alone became commonplace. This shift to silent reading was a profound change, one that Darnton suggested involved a greater mental adjustment than the shift to printed text

. One last important point in this older history, that I'm quoting because I know I'll need the reference later on for one of my texts is about reading as a social activity Yes, also related to the quietness I just mentioned:

Interestingly, with the democratization of the printed text, there was a return to reading aloud. Reading was a solitary silent process only for the educated elite who could afford to buy books. For the rest of the population, as Darnton pointed out, reading was a social activity which took place in workshops, barns, and taverns [and] while children played, women sewed, and men repaired tools.

After this introduction (obviously one of the most interesting parts to me), Cull gives numbers showing how reading is evolving (in the USA and Canada), and quotes some prediction on how the future will end up adapting. Of course, I live in a place with a very different society, so I cannot comment much. Then he confronts some studies regarding specifically leisure reading, as it is a much more trustable factor than just literacy (in a world as highly literate as ours is, many people only read when they have to and have never or very seldom experienced the pleasure of reading just for the sake of it), bringing into the discussion the Internet (and computers in general) usage patterns. I found also very interesting the next section, regarding the pattern changes many libraries are facing now, specially academic/research-oriented libraries:

For several millennia, right up until just two decades ago, the central role of a library was to collect and house physical texts: from clay tablets, to scrolls, to printed books (Battles, 2003; Manguel, 2006). While printed text remains essential to most academic libraries, today s libraries have also become a core conduit via which researchers access scholarly texts online. Just within the last few years, Canadian academic libraries, in a situation similar to libraries throughout the Western world, have reached an interesting tipping point librarians now spend the majority of their collections budgets on electronic instead of printed texts.
( )
Libraries are convinced that digital text, now in its infancy, is likely to have a long future. Not only do they purchase electronic texts, but most academic libraries have also become publishers of electronic texts, whether they are digitizing large portions of their book holdings, or focusing on scanning a relatively small number of archival documents from their unique special collections.

And yes, doing some work with our Institute's library, I can confirm this trend. About e-books: I have got quite into that topic since the Kindle won my heart (and my money!) half a year ago. The little device completly changed my reading habits, I have read lots more since I carry it. And yes, I have never considered a full tablet-like device The article talks long about the difference, about the disadvantage that multitasking means to the human brain (oh, do I suffer from it!) I liked this snippet, quoting Steve Jobs a couple of years ago, regarding an Apple e-book reader question:

When Apple was rumoured to be working on an e book reader a few years ago, CEO Steve Jobs expressed his lack of interest: It doesn t matter how good or bad the product is, the fact is that people don t read anymore, he reportedly said, continuing by stating that 40 percent of the people in the U.S. read one book or less last year. The whole conception is flawed at the top because people don t read anymore (Markoff, 2008). Nicholas Carr (2010) has summed up Apple s involvement in the tablet phenomenon this way: Jobs is no dummy. As a text delivery system, the iPad is perfectly suited to readers who don t read anymore .

The issue for me is, I do enjoy reading, but I am an information addict. I know that if I have parallel information flows, my attention will surely dilute between them. Of course, as I read this article on-screen, it was hard for me to take the needed discipline not to be distracted by IMs or IRC highlights during the whole reading (which was also as an excercise for myself ;-) ). I am surprised to see this on student preferences (and even more surprised to see this data comes from my university):

In a study of students at the Universidad Nacional Aut noma de M xico (UNAM), the majority of students preferred print, and 63 percent reported that they could bear reading a document on a computer screen for no more than one hour (Ram rez Leyva, 2003). When it comes to course textbooks, a marked student preference for paper over e books has recently been found (Woody, 2010).

As we approach the end, it talks about another important topics I have often tried (and often failed) to communicate to my users: That of paratext, the meaning of the different texts, covers, items, layouts, etc. that are not part of the text itself but do shape the way we face it. To some of us, this seems obvious. To others, it is so hard to understand It closes with two more topics I will refer to. One is the permanent connectivity. All the time, more people are connected virtually all of their waking time. This affects not only learning habits but priorities. Will this near constant access to information interfere with students desire to comprehend and remember information, necessary to the educational process of turning it into knowledge? Author and university business school lecturer Don Tapscott recently suggested that students might not have to stress about the details those you can check Finally, regarding the continuity and ellaboration found in texts that are each time more common He quotes Maryanne Wolf:

I am worried about kids who are immersed in digital culture. They will get to college and they will have been Twittering so much that they won t have the patience to read those really long cognitively convoluted and complex sentences. They may not have developed those rich networks which are required in order to read at a high level of sophistication. The effort is what we are going to lose. They are becoming not so much a lazy reader, but an atrophied reader

As you can see, not only this post is meant to tell my couple-of-interested-friends to read the article, but it's mainly meant as a mental placeholder for myself. I will be surely refering to some of these items.

If you try to move a git submodule with git mv, you ll get the following error message:

$ git mv old/submodule new/submodule
fatal: source directory is empty, source=old/submodule, destination=new/submodule

There s a patch against git to supoort submodule moving, but it doesn t seem to be applied yet, at least not in the version which is currently in Debian Sid. What worked for me to solve this issue was the following (as posted on StackOverflow):

1. Edit .gitmodules and change the path of the submodule appropriately, and put it in the index with git add .gitmodules.
2. If needed, create the parent directory of the new location of the submodule: mkdir new.
3. Move all content from the old to the new directory: mv -vi old/submodule new/submodule.
4. Remove the old directory with git rm --cached old/submodule.

Looked like this for me afterwards:

 # On branch master
 # Changes to be committed:
 #   (use "git reset HEAD <file>..." to unstage)
 #
 #       modified:   .gitmodules
 #       renamed:    var/lib/dokuwiki/tpl -> var/lib/dokuwiki/lib/tpl
 #

Finally commit the changes. HTH.

Looking through the man page of find there is no obvious way to find broken symbolic links. But there is a simple way involving only find:

$ find -L . -type l
$ find -L . -type l -ls

The option -L (before the path!) causes find to follow symbolic links and the expression -type l causes find to report only symbolic links. But as it follows symlinks, it only reports those it can t follow, i.e. broken ones. The second line also shows where the broken links point to. To easily show that this really works, just use the color indicator of GNU ls instead of find s builtin -ls:

$ find -L . -type l -exec ls -lF --color=yes ' ' +

Et voil , all displayed links show up in red which means they re broken. Kudos to CodeSnippets for showing me the right idea. And thanks to ft of zsh and grml fame for the hint about find -exec command + instead of find -exec command ;. Hint from mika of grml fame: With zsh it is even less code to type:

% ls **/*(-@)
% ls -lF **/*(-@)

Thanks, mika!

This is a very special person. He is a very well known songwriter, at least in German language countries because he sings in German. He was that special kind of person with his lyrics when I was still a kid, and is still around continuing to write his songs in his very own special way. This person is Reinhard Mey, and if you understand German and have missed him so far, you have missed a lot. The songs that I present to you are special in the way that they are all contained in the special compilation titled Mein Apfelb umchen. The dedication he wrote for the album is also very special:

Ich glaube, Kinder zu haben ist das aufregendste Abenteuer, das wir erleben k nnen. Es ist der schwerste Beruf und die gr te Herausforderung, die ich mir denken kann, und die gl cklichste Erfahrung zugleich. Ich bin dankbar daf r! Dies sind die Lieder, die ich bis heute daf r geschrieben habe. Mein Anteil aus dem Erl s dieser Schallplatte gebe ich der Hilfe f r krebskranke Kinder.

Rough translation: I believe that having kids is the most exciting adventure that we can undergo. It is the hardest job and the biggest challenge that I can think of, and at the same time the happiest experience. I'm thankful for it! These are the songs that I wrote up to today for it. My part of the revenues of this record go to Help for children with cancer. So here are the songs:

• Mein Apfelb umchen: The song that gave this compilation its title. Extremely touching, and it manages regularly to wet my eyes... Absolutely lovely.
• Keine ruhige Minute: This is actually a live version of the song with a longer introduction that is worth listening to on its own.
• Menschenjunges: A thoughtful song about the thoughts when seeing your kid for the first time.

Enjoy! And if you feel like it, support these kind of special people.

/music permanent link Comments: 2 Flattr this

I have a WRT-54G. I've had it for some years. It's run a bunch of different firmware variants over that time, but they've all had something in common. There's no way to configure IPv6 without editing text files, installing packages and punching yourself in the face repeatedly. Adam blogged about doing so today, and I suspect he may be in need of some reconstructive surgery now.

I spent yesterday looking at disassembled ACPI tables and working out the sequence of commands the firmware was sending to the hard drive. I'm planning on spending tomorrow writing x86 assembler to parse EFI memory maps. I spend a lot of time caring about stupidly awkward implementation details worked out from staring at binary dumps. The last thing I want to do is have to spend more than three minutes working out how to get IPv6 working on my home network because that cuts into the time I can spend drinking to forget.

Thankfully this is the future and punching yourself in the face is now an optional extra rather than bundled. Recent versions of Tomato USB (ie, newer than actually released) have a nice web UI for this. I registered with Tunnelbroker.net, got a tunnel, copied the prefix and endpoint addresses into the UI, hit save and ever since then NetworkManager has given me a routable IPv6 address. It's like the future.

Because I'm lazy I ended up getting an unofficial build from here. The std built doesn't seem to include IPv6, so I grabbed the miniipv6 one. The cheat-sheet for identifying builds is here. And I didn't edit a single text file. Excellent.

Gearman is a distributed queue with several language bindings.

While Gearman has a nice Python implementation (python-gearman) of the client and worker, I chose to use the libgearman bindings (python-libgearman) directly since they are already packaged for Debian (as python-gearman.libgearman).

Unfortunately, these bindings are not very well documented, so here's the sample application I wished I had seen before I started.

Using the command-line toolsBefore diving into the Python bindings, you should make sure that you can get a quick application working on the command line (using the gearman-tools package).

Here's a very simple worker which returns verbatim the input it receives:

gearman -w -f myfunction cat

and here is the matching client:

gearman -f myfunction 'test'

You can have have a look at the status of the queues in the server by connecting to gearmand via telnet (port 4730) and issuing the status command.

Using the Python libgearman bindingsOnce your gearman setup is working (debugging is easier with the command-line tools), you can roll the gearman connection code into your application.

Here's a simple Python worker which returns what it receives:

#!/usr/bin/python

from gearman import libgearman

def work(job):
   workload = job.get_workload()
   return workload

gm_worker = libgearman.Worker()
gm_worker.add_server('localhost')
gm_worker.add_function('myfunction', work)

while True:
   gm_worker.work()

and a matching client:

#!/usr/bin/python

from gearman import libgearman

gm_client = libgearman.Client()
gm_client.add_server('localhost')

result = gm_client.do('myfunction', 'test')
print result

This should behave in exactly the same way as the command-line examples above.

Returning job errorsIf you want to expose to the client errors in the processing done by the worker, modify the worker like this:

#!/usr/bin/python

from gearman import libgearman

def work(job):
   workload = job.get_workload()
   if workload == 'fail':
       job.send_fail()
   return workload

gm_worker = libgearman.Worker()
gm_worker.add_server('localhost')
gm_worker.add_function('myfunction', work)

while True:
   gm_worker.work()

and the client this way:

#!/usr/bin/python

from gearman import libgearman

gm_client = libgearman.Client()
gm_client.add_server('localhost')

result = gm_client.do('myfunction', 'fail')
print result

LicenseThe above source code is released under the following terms:

To the extent possible under law, Francois Marier has waived all copyright and related or neighboring rights to this sample libgearman Python application. This work is published from: New Zealand.

I was laid off a couple of months ago, and have been thinking about what I want to do next. Actually, I've mostly been writing code for my pet projects, but earning a living again is occasionally on my mind. I would, very much, like to earn a living while deveoping interesting free software that I would like to use myself. There's a bunch of options available:

• I could go work for one of the large companies that use free software, and hope to find a position where I can help develop the tools they use. This would perhaps be the easiest option.
• I could perhaps find a job at a smaller company that specilizes in the development of some parts of the free software stack, or develops a Linux distribution.
• I could join or found a startup that develops free software and thinks it has a business model around it.
• I could learn web programming, and develop interesting online services that people would be willing to pay for.
• I could do consulting gigs, either via my own company (back in Finland), or by getting employed by one of the free software oriented consulting houses that exist. Again, I might need to learn some new skills, since most customers are looking for work in parts of the development stack I'm not experienced in, such as the kernel, or want web or mobile app development, neither of which I know.
• I could try to crowd-fund development of software I think the world needs, and hope to get paid enough to make a living. This would be a special case of doing consulting: I would be selling very small amounts of work to a fairly large number of customers.

In this blog entry, I want to think about that last option. Here's I imagine it would work: I would post a list of things I would like to do, estimates of how much work they'd be, and a hourly price. People would tell me how many hours of work they'd like to pay for. I would then do the work, send everyone their bill, and they'd pay it. In the ideal world, this would be an awesome way to fund development. I could, for example, write some automated tools for system testing of Debian, to help with release management, or I could finish Obnam. And that's just two things I'm already working on: there's no end of things I would like to develop, if I didn't need to worry about my next meal. However, this is not without problems. To start with, it might look like I'm trying to blackmail the world: "pay me, or this software won't be written". That's not a problem in the general case, but it'd easily look like that if the work would be to, say, maintain Debian packages. As a Debian developer, I should keep my packages in good shape whether someone is paying me or not. I would need to be careful to keep things ethical here. Another problem would be the issue of being realistic: would there be enough people willing to pay me, and how long would that last? That partly depends on the pricing. Although my needs are fairly modest, I do live in a country that is fairly expensive to live in. If I started doing this, it'd be easy for someone in a country it is cheaper to live in to compete on price (and they might well be better than I am, too). Then again, since there's no end of things to do, perhaps we could both make a living. Would you pay someone to develop free software by the hour? How much? How often? What should the software do?

Just added a TDeb test build branch for multistrap to test out the changes in the next version of the emdebian-tdeb binary package - specifically:

Move em_installtdeb into the emdebian-grip-server package so that the emdebian-tdeb package can be slimmed down to only contain what will become useful for other maintainers to use in their debian/rules files and for translators when the time comes to implement DEP-4. CDBS rules / patches to follow... bug reports against dpkg and debhelper to follow ...

DEP-4 itself is being restarted and refreshed but with some problems on alioth currently, I'm going to (try and) keep the document up to date via SGML and people.debian.org.

The changes are now in Emdebian SVN and I'm starting testing with my own packages first. Multistrap for PO and po4a-build testing, dpkg-cross for debconf testing, some others for testing with build systems other than CDBS, then the discussions will restart on debian-devel and I'll upload the TDeb support scripts to unstable.

There are a variety of things which need to happen before these scripts become useful, including:

• dpkg support for XC-Package-Type: tdeb
• debhelper support for dh_gentdeb,
• inevitable bug fixing in emdebian-tdeb

Why is the package called emdebian-tdeb? Simple - that's where the scripts have lived until now and the hope is that dpkg-gentdeb can migrate into dpkg-dev and dh_gentdeb into debhelper - eventually.

By all means play with the scripts and follow the example multistrap branch above to see what kind of changes might be needed in your own packages. I'll post again when I've worked out the changes needed in packages like dpkg-cross which only need TDebs for debconf support. (This is the easiest mode of TDeb usage but it's also the most useful during a release cycle - maintainers make a single change and then a nominated i18n boss-person - take a guess who - can update the all your debconf translations in one upload without a binary build - eventually possibly not even needing any bug reports, just a simple email advance warning.

What I've got to work on still is how the generation of an updated TDeb creates a new debian diff / debian.tar.gz to accompany the upload. For that I need XC-Package-Type: tdeb support in dpkg....

Sample output from the viewpoint of a translator updating a package already converted to support a TDeb:

neil@sylvester:tdeb-test$ dpkg-gentdeb -t

Set the changelog message to version 2.1.15+t0 first, closing any relevant bugs!

 dch -v 2.1.15+t0 

neil@sylvester:tdeb-test$ dch -v 2.1.15+t0 TDeb test

neil@sylvester:tdeb-test$ dpkg-gentdeb -t

Building TDeb update: 2.1.15+t0

 (192 entries)

Discard doc/pod/1/da/multistrap (65 of 177 strings; only 36.72% translated; need 50%).

Discard doc/pod/1/da/device-table.pl (5 of 19 strings; only 26.31% translated; need 50%).

Processing untranslated files for pod/multistrap (1) . . .

Processing untranslated files for device-table.pl (1) . . .

Processing de translations for pod/multistrap (1). . .

Processing de translations for device-table.pl (1). . .

Processing fr translations for pod/multistrap (1). . .

Processing fr translations for device-table.pl (1). . .

Processing pt translations for pod/multistrap (1). . .

Processing pt translations for device-table.pl (1). . .

dpkg-gencontrol -pmultistrap-tdeb -Pdebian/multistrap-tdeb -cdebian/control

dpkg --build debian/multistrap-tdeb ../multistrap-tdeb_2.1.15+t0_all.tdeb 

dpkg-deb: building package  multistrap-tdeb' in  ../multistrap-tdeb_2.1.15+t0_all.tdeb'.

neil@sylvester:tdeb-test$ dpkg -c ../multistrap-tdeb_2.1.15+t0_all.tdeb

drwxr-xr-x root/root         0 2011-06-08 21:06 ./

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/pt/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/pt/LC_MESSAGES/

-rw-r--r-- root/root      6427 2011-06-08 21:06 ./usr/share/locale/pt/LC_MESSAGES/multistrap.mo

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/vi/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/vi/LC_MESSAGES/

-rw-r--r-- root/root      6963 2011-06-08 21:06 ./usr/share/locale/vi/LC_MESSAGES/multistrap.mo

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/fr/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/fr/LC_MESSAGES/

-rw-r--r-- root/root      6789 2011-06-08 21:06 ./usr/share/locale/fr/LC_MESSAGES/multistrap.mo

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/da/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/da/LC_MESSAGES/

-rw-r--r-- root/root     16311 2011-06-08 21:06 ./usr/share/locale/da/LC_MESSAGES/multistrap.mo

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/pt/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/pt/man1/

-rw-r--r-- root/root     33963 2011-06-08 21:06 ./usr/share/man/pt/man1/multistrap.1

-rw-r--r-- root/root      4471 2011-06-08 21:06 ./usr/share/man/pt/man1/device-table.pl.1

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/vi/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/fr/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/fr/man1/

-rw-r--r-- root/root     33969 2011-06-08 21:06 ./usr/share/man/fr/man1/multistrap.1

-rw-r--r-- root/root      4708 2011-06-08 21:06 ./usr/share/man/fr/man1/device-table.pl.1

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/da/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/da/man1/

I'm also going to DebConf11



On the RoadTrip! all the way hrough France, Belgium, Netherlands, Germany, Austria, Slovenia, Croatia, Bosnia-Herzegovina (Banja Luka) in an MX5.

I've got a lot of things to talk to people about @ DebConf this year... you won't be able to tell if I'm just buying you a beer or trying to badger you about your packages!! Just note that TDebs have been already agreed (in terms of how TDebs work in the archive) with ftp-master at a previous meeting in Extremadura and I'm being badgered by a Release Manager and an ex-DPL to get TDebs implemented real soon now, so it is going to happen and hopefully in time for Wheezy - it's just a matter of fixing the bugs. The quicker everyone has a look at the new scripts (which have been completed re-written and immensely simplified since I first tried a dumb conversion of em_installtdeb) and the likely changes needed in packages, the quicker I can get the bugs out of the scripts.

More documentation to follow, I'll be updating the SGML whenever practical and the actual DEP just as soon as I get commit access back.

Disclaimer: There is a huge discussion going on right now in the debian-devel mailing list, about different ways to rearrange how Debian development happens, and maybe to provide a Debian users with an additional release. This blog entry is not a comment on that discussion, which I am ignoring. The discussion did, however, prompt a dent about "test driven development". That prompted a short exchange of thoughts with Tom Marble, and this blog post is a cleaned up version of my part in that discussion.

Currently Debian development happens roughly like this: all packages are uploaded to a part of the Debian archive called unstable. Once they've been in use for a while without serious problems, they get automatically copied into another part called testing. Every year or two, testing is frozen and any remaining problems are fixed, after which all of testing gets copied into yet another part of the archive called stable, and that's the actual release. There is a little bit of automatic testing, but only a little. Almost all testing is done by people, by them using unstable or testing on their usual computers. If they have problems, they report them. Contrast this with development using Test Driven Development, or TDD, and other modern development methodologies. Here's a rough summary of what I do when I write (much) of my software.

• first write one or more automatic tests (unit or functional ones)
• write the actual code, enough to make all tests pass
• add more tests
• write more code, or change existing code
• repeat this until tests describe all the behavior desired of the code

In addition, I measure coverage to make sure all parts of the code gets tested. I usually aim for 100% coverage, except for those parts that very hard or quite pointless to test. (That's easier to achieve than you'd think, but that's the topic of another blog post.) This all sounds like a lot of bureaucratic nonsense, but what I get out of this is this: once all tests pass, I have a strong confidence that the software works. As soon as I've added all the features I want to have for a release, I can push a button to push it out. (Well, actually there's a little bit more to it.) This does mean I never have bugs in my software. Of course I do. However, there's a lot fewer of them. In fact, there's so few of them, after tests pass, that I would almost be happy to make an automatic release after every successful run of "make check". Another aspect of the way I do development is distributed version control. The relevant feature is powerful branching and merging. Most things I develop happen in short-lived single-purpose branches: whenever I start work on a feature or bugfix, I create a branch for it. (Unless I'm feeling particularly lazy, which happens more often than you'd think.) When I've finished the feature, or fixed the bug, I merge the branch back into the trunk branch. This way, the trunk is always in releaseable state. It might not have all the features I want for the next release, but the features that are there, do work. The branch probably has bugs, but if I've written tests that are good enough, I know the software works well enough. Or that's the idea. Sometimes things go wrong. Then I write more tests and the next time goes better. (It doesn't have to be perfect, as long as it gets better every time.) See the contrast between automatic tests with good coverage, and the Debian style of relying on user feedback? There's no need to wait for user feedback with automatic tests. This speeds up development, makes releasing easier, but most importantly takes away any reason to fear making changes, even big changes. Automatic tests and good test coverage are easy achieve in small projects. For a system as huge as Debian, good test coverage is quite hard to achieve. The thing about automatic tests, though, is that even a little bit of it is helpful, and after you have a few tests, it gets easier to add more. The first test is the hardest to get done, since you need to set up all the infrastructure for it. Debian does do a bit of automatic testing already. (See lintian, piuparts, autopkgtest, edos, etc.) I don't want to belittle that, but I think we could do better. Here's what I would love to see:

• we have a part of the archive that corresponds to a trunk branch, i.e., it is always in a releasable state (the "testing" area was originally meant to be that; we could make it so now)
• releaseable state is determined by two things:
  • an automatic test suite
  • user feedback, particularly in the form of bug reports (as now)
• whenever changes are made, they happen in "branches"
  • the branch is not affected by changes elsewhere in the archive, except by manual synching ("merge from trunk")
  • individual package uploads, as well as groups of packages such as for transitions, are each in their own branch
  • this is sort of similar to a PPA, but more powerful
• when a branch is to be merged into trunk, the automatic tests must first pass (or at least no new failures in them can be introduced)
  • tests can also be run for any other branch, of course, so that those developing the branch know if they're ready to push their changes into trunk
• there's a culture of writing tests for bugs (whenever that is feasible), and for new features
  • particularly release goals should be expressed as automatic test suites
• there's a culture of sharing tests with upstreams, and with other distributions

Since full test coverage is going to be impossible for Debian, some subset should be targeted. Perhaps something like this would suffice to start with:

• a version of debian-installer is generated from a branch
• test installation of a new system from scratch
  • with a large set of tasks selected
• test upgrade from previous stable release to the branch
  • with a large set of packages installed (as many as possible, actually)
• test upgrade from trunk branch to branch to be merged
  • ditto large set of packages
• test with lintian, reject on specific tests failing
• test with piuparts
• test the whole system for specific functionality
  • ssh access from outside
  • sudo access for a logged in user
  • sending mail with SMTP
  • web server
  • possibly test specific web applications
  • possibly test Samba and NFS services
  • possibly test printing services (CUPS)
  • possibly test essential desktop functionality (automatic login, at least)
• if package comes with package specific tests, run those too

These tests would not guarantee that a set of changes would not break Debian, but they would give a high confidence that at least basic stuff would still work after the changes. Now, obviously implementing all of this I'm dreaming of (it is just past midnight, after all) is going to be impossible. There's way too much work to do, there's not enough tools for writing tests, and it would require too much computing power to run, and so on and so forth. But it's late, and I've had a bad day, and I might as well dream of something nice. Anyway, anyone interested in these things should perhaps help drive DEP8.

Since my laptop features WWAN connectivity, I purchased a 3G mobile broadband plan from SFR, a French mobile phone company. I took the cheapest one available: 250 MB per month for 10 .

Optimizing transparent proxy by SFR While browsing, I was curious about the fact that there were multiple accesses to non-public IP, like 10.141.0.4. I was browsing the mobile version of Le Monde, a French daily newspaper. I enabled Chromium developer tools and watched what happened in the Network tab. All images were served from IP like 10.141.0.4. Looking at the HTML source code, I discovered that <img> tags were rewritten to point to those images. The smaller one were identical to the original one but the larger ones (more than 50 KB) were of reduced quality (and therefore reduced size). <img> tags were not the only one to be rewritten. <style> and <script> tags that were pointing to external resources (with a src attribute) were also rewritten to include the CSS or the javascript inline. Before:

<script src="http://s1.lemde.fr/js/lib/jquery/1.5.1/jquery.js"></script>

After:

<script oldSrc="http://s1.lemde.fr/js/lib/jquery/1.5.1/jquery.js" NG="replaced">
/*!
 * jQuery JavaScript Library v1.5.1
 *
 * Copyright 2011, John Resig
 * Dual licensed under the MIT or GPL Version 2 licenses.
 * http://jquery.org/license
 * [...] */
</script>

This means that each page that includes jQuery, a popular javascript library, as an external resource, will be turned into a page that includes the whole jQuery code which is about 85 KB. Each page will weight more than 85 KB instead of relying on an external and cacheable resource. Nowadays, most browsers support HTTP keep-alive which allows to serve multiple resources into a single request. There are two cases:

1. If the resource is served from the same server as the HTML page, keep-alive would enable to get almost the same result without hindering the cache: the browser would first request the HTML page, then the javascript resource into the same TCP connection. There is a small penalty of about 100 ms to let the browser request the second resource. The browser may decide to open another connection because the initial document is not loaded yet. Some browsers may even negotiate this additional connection before needing it. In this case, it can use this idle connection at no cost. Otherwise, we may fall in the second case.
2. If the resource is served from another server (like in the example above) or if the browser does not have an idle connection ready for reuse, it would download the resource in parallel with the help of another connection. It will only costs about 300 ms of additional time due to the establishment of a new TCP connection with the server. Those 300 ms are a very small price to pay compared to the 2 seconds needed to download 85 KB for each page. Moreover, other static resources will likely be downloaded from the same server and therefore, this is 300 ms the first time and about 100 ms for additional resources (round-trip time for a new HTTP request into the same TCP connection).

UPDATED: In a previous version of this article, I confused pipelining and keep-alive mechanisms. While keep-alive allows to reuse an existing connection to download a new resource, only one request at a time by connection could be issued. Pipelining allows to push additional requests in the same TCP connection without waiting for answers. As noted by ric Daspet in a comment, pipelining is disabled for most browsers. However, keep-alive is well supported. Searching on the web for oldSrc and NG="replaced", I find very little information about what is used. It seems that such a feature may also break some scripts, like TinyMCE, trying to get additional resources relative to the path where the script is installed. It appears that SFR is using a product from Flash Networks called Harmony Gateway. I have asked SFR more info about this and if it was possible to disable the use of this feature but got no answer so far. I have retried a bit later and there was no rewriting for Javascript and CSS. Images were still degraded and served by a special server. The next day, my requests did not travel through a transparent proxy anymore (no more Via header in answers). Each try was done at a different location. The proxies may be installed at some locations only and configured differently. Another explanation is that SFR silently disabled this functionality for my account. If it was done correctly, such an accelerator service could be be proposed as an opt-out option since most users would appreciate faster page downloads. If you care about network neutrality, you can disable this option. Of course, it needs to be properly advertised, for example as a free option in your mobile plan.

Setting up our own optimizing proxy Google provides mod_pagespeed as a solution to optimize a website on the fly. This is an Apache module. It is targeted at optimizing your own website but since Apache can act as a regular proxy, it seems possible to enable mod_pagespeed for any website. That s pretty easy to setup if you don t mind using a precompiled version of mod_pagespeed. Download mod_pagespeed. I have downloaded the 64bit version for Debian.

# touch /etc/default/mod-pagespeed
# dpkg -i ~/download/mod-pagespeed-beta_current_amd64.deb
Selecting previously deselected package mod-pagespeed-beta.
(Reading database ... 277699 files and directories currently installed.)
Unpacking mod-pagespeed-beta (from .../mod-pagespeed-beta_current_amd64.deb) ...
dpkg: dependency problems prevent configuration of mod-pagespeed-beta:
 mod-pagespeed-beta depends on apache2.2-common; however:
  Package apache2.2-common is not installed.
dpkg: error processing mod-pagespeed-beta (--install):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 mod-pagespeed-beta
# apt-get install -f
[...]
Unpacking apache2.2-common (from .../apache2.2-common_2.2.19-1_amd64.deb) ...
Processing triggers for man-db ...
Setting up apache2.2-common (2.2.19-1) ...
[...]
# apt-get install apache2-mpm-worker
[...]
Unpacking apache2-mpm-worker (from .../apache2-mpm-worker_2.2.19-1_amd64.deb) ...
Setting up apache2-mpm-worker (2.2.19-1) ...
Starting web server: apache2.
# a2enmod proxy_http
Considering dependency proxy for proxy_http:
Enabling module proxy.
Enabling module proxy_http.
To to activate the new configuration, you need to run:
  /etc/init.d/apache2 restart

Then, we modify /etc/apache2/mods-enabled/proxy.conf:

ProxyRequests On
<Proxy *>
    AddDefaultCharset off
    Order deny,allow
    Deny from all
    Allow from 127.0.0.1
</Proxy>

Of course, this makes little sense to run the proxy on your laptop since you don t want to download the non-optimized content over your WWAN link. You need to run it on another server and therefore allow your laptop to access it. Also, we need to modify /etc/apache2/mods-enabled/pagespeed.conf to add the directive ModPagespeedDomain * to let mod_pagespeed rewrite any page. You should restart Apache after that. Now, check if the proxy is working:

$ curl -six http://localhost:80 http://www.lemonde.fr/   grep ^X-Mod-Pagespeed
X-Mod-Pagespeed: 0.9.17.7-716

All is working! Let s try a very simple site, like Debian Backports. It features one HTML file and two CSS files: We downloaded a total of 8 KB (compression included). Let s use our proxy and try again: mod_pagespeed succeeds in reducing the size of the HTML file and combined the CSS files. Moreover, it minified the resulting CSS file. There seems to be a bug in Chromium that display incorrect transfert sizes (including in the timeline representation) when using a proxy. I have checked with Wireshark that the overall quantity transferred was about 8 KB. The gain is very thin here but you can imagine that you will get better results on larger sites. mod_pagespeed seems to handle well most sites. However, it only optimizes pages that already are in its cache. The first time you grab a page, you will get the unoptimized version. I don t know if it is the expected behaviour.

Continuing my earlier musings about test driven distro development, and the tools that would require. I imagine something like this:

• create a virtual machine with a particular configuration of stable
• run a test suite verifying the VM works
• upgrade the VM to testing and reboot
• re-run test suite, verifying things work in testing

We might also want this scenario:

• install testing directly in a VM (instead of upgrading from stable)
• run test suite

There might be other scenarios that would be useful to test as well. Even from these two, it's clear there's a need for at least three separate tools:

• create a machine image with a particular configuration
  • might be based on stable or testing or unstable
  • might also be using some other package sources
  • might want particular packages installed
  • might want particular configuration settings
• run a suite of tests against the running image
  • which tests to run will depend on the scenario
• run various scenarios using the above two tools
  • needs reasonably easy ways to specify scenarios
  • needs to support tests specific to scenarios

I've written a rudimentary version of the first tool: vmdebootstrap. I've since learned there's a bunch of others that might also work. There's room for more: someone should write (or find) a tool to make a snapshot of a real system and create a VM image that mimicks it, for example. Anyway, for now, I'll assume that one of the existing tools is good enough to get started. For the second tool, I wrote a quick-and-dirty proof-of-concept thing, see systest.py. Here's a sample of how it might be used:

liw@havelock$ ./systest -v --target 192.168.122.139 --user tomjon
test 1/6: cat
test 2/6: only-ssh-port
ERROR: Assertion failed: ['22/tcp', '139/tcp', '445/tcp'] != ['22/tcp']
[status 1]
liw@havelock$ ./systest -v --target 192.168.122.139 --user tomjon \
    cat ping6-localhost ping-localhost simple-dns-lookup ssh-login
test 1/5: cat
test 2/5: ping6-localhost
test 3/5: ping-localhost
test 4/5: simple-dns-lookup
test 5/5: ssh-login
liw@havelock$ 

The first run failed, because the VM I'm testing against has some extra ports open. Some of the tests will require logging into the machine, via ssh, and for that one needs to specify the user to use. systool may overlap heavily on system monitoring tools, and possibly the production implementation should be based on those. I think it's best to design such a tool for the more general purpose of testing whether a system currently works rather than as an integrated part of a more specific larger tool. This lets the tool be useful for other things than just testing specific things about Debian. (The production implementation would then need to not have all the tests hardcoded, of course. SMOP.) The third tool I have not spent a lot of time on yet. One thing at a time. Given these tools, one would then need to decide how to use them. The easiest way would be to use them like lintian and piuparts: run them frequently on whatever packages happen to be in testing or unstable or experimental, and put links to the test reports to the PTS, and hope that people fix things. That is the easiest way to start things. Once there's a nice set of test cases and scenarios, it may be interesting to think about more aggressive ways: for example, preventing packages from migrating to testing unless the test suite passes with them. If the tests do not pass, one of four things is broken:

• the package or packages in question
• other packages already in testing
• the tests themselves
• the test environment

If things are set up properly, the last one should be rare. The other three always require manual inspection: it is not possible to automatically know whether the test itself, or the code it tests, is at fault. it is, however, enough to know that something is wrong. If the tests are written well, they should be robust enough to not be the culprits very often. (Someone wanting to make a rolling distrbution, or even better, making a monthly mini-release, might benefit from this sort of automated testing.)

Whenever someone asked me about a calendar application, especially for the textmode, I always encouraged them to give pal a try. I always loved the looks of it, the interactive mode is helpful, it has HTML output format to inject the calendar into a webpage, mail output format for a daily reminder cronjob, and other useful features. I even created a file with the Austrian holidays for it which got included in the original project for the benefit of all its users.
If you haven't tried it yet and are looking for a calendar tool with support for very flexible recurring events and categories, this might be a good look. I am still happy with pal, though someone recently suggested a different tool on IRC, and that was wyrd. From a quick glance it looked promising, so I started to dig into it. My first task was to convert the former mentioned pal file for the Austrian holidays into remind format. remind is the backend for wyrd, and its definition language seems to be extremely powerful. It though took me a while to figure out how to put in Easter date related events into it, the examples weren't really hinting me in the right direction. This is part of what I am using now:
REM [trigger(easterdate(current())-47)] +6 TAG noweight MSG Faschingsdienstag %b The look and view of wyrd is different to pal in several ways. Where the granularity of pal is a pure day view, wyrd scales in hours (or half, quarter thereof). Also, wyrd offers the possibility to color the days differently by busy level. Of course it's possible to exempt tasks from adding weight to a day. pal on the other hand is able to color events differently by category. Decide yourself what you actually need, test it, and ... enjoy!

/cli permanent link Comments: 0 Flattr this

As usual many people requested travel sponsorship to attend DebConf 11 and are now waiting eagerly on the outcome of their request. We are sorry, that we can t give you much details yet. To give you an idea of the time line, here is a small explanation of the process. About 100 people requested travel sponsorship. In total that would about 50 000 Euro. Problem is currently we don t exactly know, how much money we may spend on travel sponsorship ourselves (By the way: If a potential sponsor is reading this, please contact us; DebConf also accepts donations). So, we formed a team of 13 persons (the mail only lists twelve, one was added later). The team members will start to rate each and every single request in two criteria: Contributor Level and sponsorship amount. With limited amounts we obviously want to sponsor those first, who contribute a lot to Debian, preferably if they don t travel first class. After that s done, we ll have a long, but ordered list, but the team still needs to meet and discuss different cases and similar. Our team isn t really small and consists of people living all over the world, it s quite hard to actually schedule a meeting everyone can attend. (Which is not a bug, but a feature: More people from more countries should in the end result in a fair rating of the sponsorship requests.) So, in the end, we ll meet on 19th of June. Sounds late, please keep in mind, that every team member has to rate all sponsorship requests (including some research) and all of them are volunteers with actual daytime jobs. But even if you don t hear anything from us short after that date, don t panic! As said we really don t know, yet, how much money we can spend on travel sponsorship. Our sponsor-hunting-team is constantly working on the acquisition of more money, so every now and then the budget for travel sponsorship gets increased, meaning that we can mail a couple of more people on our ordered list. So it might take a while, till we get to all, but we try very hard to do so.

DECLARATIVE DIVERSIONS ====================== Introduction A diversion is when it is possible to have dpkg not overwrite a file when it reinstalls the package it belongs to, and to have it put the file from the package somewhere else instead. Declarative diversions involves a new control file with a declarative syntax which dpkg will parse and process directly as part of the package unpack and removal phases, eliminating the problems resulting from non-atomic handling of diversions. Topics There are a number of topics involved in implementing this kind of project * What syntax do we use for the new control file? * What dpkg does with the control file * How do we handle diversions to a non-existant directory? * How do we order unpacking a new package which adds a diversion? * How do we order unpacking a new package that removes a diversion? * How do we order removing a package which had a diversion? * How do we handle errors? * What happens to dpkg-divert? Details - Control File Syntax - It will conform to RFC2822 style with the following format: * Divert-From: * Divert-To: * Blank lines and lines beginning with # will be comments Divert-To will be optional and if it is ommitted then files being diverted will have their filename changed to file.distrib The above style has it s advantages, one of the main ones being that there is no need to escape whitespace within filenames. It also allows for more commands to be added at a later date. Details - Control File Handling - Within control.tar.gz the file should be named diversions This file is then copied to /var/lib/info/$package.diversions We need to store the control file for the currently-installed package so that we can detect declarative diversions that have been removed from the control file between versions (On-disk version and the to-be-unpacked version). If there is no previous version of a control file then all current diversions within the package are left in place and any new ones declared within the control file are added. If there is a previous version of the control file then the diversions listed within the file are compared with the diversions currently in place. If a diversion exists that is no longer listed in the control file then it is removed. If there is a diversion listed in the control file that doesn t currently exist then it is added. Details - Handling Diversions to non-existant directories - Diverting files to directories that don t exist can cause a number of problems. If the package does not own the directory it may be left orphaned on removal of the package The package is responsible for ensuring the availability of the target directory in the unpack phase. If a declarative diversion attempts to divert a file to a folder that neither exists already on the filesystem at unpack time, nor is shipped within the package, dpkg will return an error at unpack time. Details - Ordering Requirements - =>Unpacking a new package that adds a diversion *1. Check if package has a previous diversion control file *2. Copy control file to tmp directory *3. Calculate diversions to be added *4. Unpack folders *5. Add diversions *6. Unpack files *7. Copy control file to /var/lib/dpkg/info/$package.diversion *Performing diversions this way avoids trouble with creating folders and accidentally overwriting the wrong version of files. *Another major problem would be extracting a new version of the file and overwriting the old one. *This might cause the wrong version of the file to be diverted and the package to break. =>Unpacking a new package that removes a diversion *1. Check if package has a previous diversion control file *2. Copy control file to tmp directory *3. Calculate diversions to be removed *4. Check if dropped diversion results in a file conflict and abort the unpack if it does. *5. Unpack folders *6. Unpack files *7. Remove old files no longer needed in the package *8. Remove diversions *9. Copy control file to /var/lib/dpkg/info/$package.diversion *If a package has no previous diversion control file then no diversions will be removed and existing diversions can only be removed using dpkg-divert. *If removing a diversion results in a file conflict then dpkg will return an error at unpack time before anything is unpacked and the unpack is aborted. =>Removing a package which had a diversion *1. Remove files *2. Remove diversions *3. Remove folders *4. Remove control file at /var/lib/dpkg/info/$package.diversion *This ensures that all files, diversions and folders are removed correctly Details - Error handling Errors in diversions will have to handled with a great deal of care due to the fact that if they are not the package could be broken. This means that a great deal of checks must be done to ensure that all the files can be diverted properly before any actual diverting takes place. If they can t the package installation/update must be stopped and rolled back to avoid the package being installed incorrectly or broken. Details - dpkg-divert - When we impliment the new diversion method we should keep the current dpkg-divert. This allows maintainers to catch up with the new method without breaking their packages. It also allows maintainers to perform some operations that aren t support by the new method. All current functionality of dpkg-divert will be left intact for use by sysadmins and for maintainers to catch up with the new system. The two systems will work together without directly conflicting with each other. Example Usage #1 The file to be diverted is /usr/share/foo It needs to be moved to /usr/share/bar The syntax of the control file would be: #Start File Divert-From: /usr/share/foo Divert-To: /usr/share/bar #End File Example Usage #2 In this example the maintainer doesn t want to move the file to any specific folder The syntax of the control file would be: #Start File Divert-From: /usr/share/foo #End File This would divert the file to /usr/share/foo.distrib Footnotes - RFC2822 Guide: * http://www.faqs.org/rfcs/rfc2822.html First Email Thread on Declarative Diversions (First Message in Thread): * http://lists.debian.org/debian-dpkg/2011/05/msg00102.html Email thread on first design draft (First message in Thread): * http://lists.debian.org/debian-dpkg/2011/06/msg00019.html Declarative Diversions Wiki: * http://wiki.debian.org/SummerOfCode2011/DeclarativeDiversions My Blog for this Project: * http://blog.sam-dunne.com

(First posting to Pl net Commandline! Tag: pcli) Vutral asked in IRC how to synchronise two shells environment while they re running. As you may know, POSIX systems cannot change a process environment vector after it has been started, only the process itself can. Well, the shell can, and we ll use a variety of things for this. This trick assumes you have $HISTFILE set to the same pathname in both shells (obviously, they run under the same user). It uses export -p to render the current list of exported variables, then transforms the list from newline-separated to a single big one-line export statement.
Then it transforms all remaining newlines (which will be part of a single-quoted string, since that s mksh(1) s export format) into the sequence '$'\n'' which means: terminate current single-quoted string, append $'\n' and open up a new single-quoted string immediately; concatenate these three.
Now, $'\n' is just a fancy way of saying newline, and part of mksh because David Korn (yes, the Korn in Korn Shell) strongly suggested to me that this functionality be included but, as we can see here, it pays off.
Finally, the so transformed string is prepended by unset \$(export); which, when executed, will cause the shell to unset (and unexport) all currently exported variables. The shell parameters that are not exported, i.e. not in the environment, are not affected by this code (except for $x and $nl, but whatever).
This string is then passed to read -s (plus -r and clearing IFS to enable raw mode), which means, read into the parameter $REPLY (which we conveniently don t use but it s trashed too, thus) but store into history at the same time. Ah hah! Now, the persistent history feature comes into effect! After running the below statement in the source shell, switch into the terminal running the destination shell, press Enter once on the empty line (Ctrl-U to empty it if it wasn t), then Cursor-Up ( ) to recall voil , an insanely large line with the previously created string sorta expanded and press Enter again to run it. Now your set of exported parameters is the exact same (minus if you exported IFS, nl, x or REPLY) as in the source shell.

I ve added extra spaces and a linewrap below, this is really just one big line:

Of course, this makes a nice function, for your ~/.mkshrc or somesuch.