There are quite some tools to find duplicate files in Debian and depending on the task I use either hardlink (see this blog posting), fdupes (if I need output with all identical files on one line; see example below), or duff (if it has to be performant). But for code deduplication in historically grown code you sometimes need a tool which does not only find identical files, but also those which just differ in a few blanks or blank lines. I found two tools in Debian which can give you some kind of percentage of similarity: simhash (which is btw. orphaned; upstream homepage) and similarity-tester (upstream homepage). simhash has the shorter name and hecne sounds more usable on the command-line. But it seems only be able to compare two files at once and also only after first computing and writing down its similarity hash to a file. Not really usable for those one-liner cases on the command-line. similarity-tester has the longer name (and one which made me suspect that it may be a GUI tool), but provides what I was looking for:

$ find . -type f   sim_text -ipTt 75

This lists all files in the current directory which have at 75% ( -t 75 ) in common with another file in the list of files. The option -i causes sim_text to read the files to compare from standard input; -p causes sim_text to just output the similarity percentage; and -T suppresses the per-file list of found tokens. I used similarity-tester s sim_text tool to compare natural langauge as most of the files, I had to test, are shell scripts. But similarity-tester also provides tools to test the similarity of code in specific programming languages, namely C, Java, Pascal, Modula-2, Lisp and Miranda. Example output from the xen-tools project (after I already did a lot of code deduplication):

./intrepid/30-disable-gettys consists for 100 % of ./edgy/30-disable-gettys material
./edgy/30-disable-gettys consists for 100 % of ./intrepid/30-disable-gettys material
./common/90-make-fstab-rpm consists for 98 % of ./centos-5/90-make-fstab material
./centos-5/90-make-fstab consists for 98 % of ./common/90-make-fstab-rpm material
./gentoo/55-create-dev consists for 91 % of ./dapper/55-create-dev material
./dapper/55-create-dev consists for 90 % of ./gentoo/55-create-dev material
./gentoo/55-create-dev consists for 88 % of ./common/55-create-dev material
./common/90-make-fstab-deb consists for 87 % of ./common/90-make-fstab-rpm material
./common/90-make-fstab-rpm consists for 85 % of ./common/90-make-fstab-deb material
./common/30-disable-gettys consists for 81 % of ./karmic/30-disable-gettys material
./intrepid/80-install-kernel consists for 78 % of ./edgy/80-install-kernel material
./edgy/30-disable-gettys consists for 76 % of ./karmic/30-disable-gettys material
./karmic/30-disable-gettys consists for 76 % of ./edgy/30-disable-gettys material
./common/50-setup-hostname-rpm consists for 76 % of ./gentoo/50-setup-hostname material

Depending on the length of possible filenames and amount of files this can be made more readable using the column utility from the bsdmainutils package and reversed by using tac from the coreutils package:

$ find . -type f   sim_text -ipTt 75   tac   column -t
./common/50-setup-hostname-rpm  consists  for  76   %  of  ./gentoo/50-setup-hostname    material
./karmic/30-disable-gettys      consists  for  76   %  of  ./edgy/30-disable-gettys      material
./edgy/30-disable-gettys        consists  for  76   %  of  ./karmic/30-disable-gettys    material
./intrepid/80-install-kernel    consists  for  78   %  of  ./edgy/80-install-kernel      material
./common/30-disable-gettys      consists  for  81   %  of  ./karmic/30-disable-gettys    material
./common/90-make-fstab-rpm      consists  for  85   %  of  ./common/90-make-fstab-deb    material
./common/90-make-fstab-deb      consists  for  87   %  of  ./common/90-make-fstab-rpm    material
./gentoo/55-create-dev          consists  for  88   %  of  ./common/55-create-dev        material
./dapper/55-create-dev          consists  for  90   %  of  ./gentoo/55-create-dev        material
./gentoo/55-create-dev          consists  for  91   %  of  ./dapper/55-create-dev        material
./centos-5/90-make-fstab        consists  for  98   %  of  ./common/90-make-fstab-rpm    material
./common/90-make-fstab-rpm      consists  for  98   %  of  ./centos-5/90-make-fstab      material
./edgy/30-disable-gettys        consists  for  100  %  of  ./intrepid/30-disable-gettys  material
./intrepid/30-disable-gettys    consists  for  100  %  of  ./edgy/30-disable-gettys      material

Compared to that, fdupes only finds the two 100% identical files:

$ fdupes -r1 . 
./intrepid/30-disable-gettys ./edgy/30-disable-gettys 

But fdupes helped me already a lot to find the first bunch of identical files in xen-tools. :-)

From time to time I need to convert scanned documents to a pdf format.


Usage scenario 1: I scan part of a book (i.e. some article) on a school's scanner that sends me 10 big separate color pdf files (one pdf per page). I want to get one nice, small (black and white) pdf file with all the pages.


Usage scenario 2: I download a web form, print it, fill it in, sign it, scan it on my own scanner using Gimp and now I want to convert the image into a nice pdf file (either color or black & white) to send back over email.

Solution: I save the original files (be it pdf or png) into a folder and use git to track it. Then create a simple reduce script to convert it to the final format (view it as a pipeline). Often I need to tweak one or two parameters in the pipeline.

Here is a script for scenario 1:
<script src="https://gist.github.com/2871625.js?file=reduce2.sh"></script>
And here for scenario 2:
<script src="https://gist.github.com/2871625.js?file=reduce1.sh"></script>
There can be several unexpected surprises along the way. From my experience:

• If I convert png directly to tiff, sometimes the resolution can be wrong. The solution is to always convert to ppm (color) or pbm (black and white) first, which is just a simple file format containing the raw pixels. This is the "starting" format (so first I need to convert the initial pdf or png into ppm/pbm) and then do anything else. That proved to be very robust.
• The tiff2pdf utility proved to be the most robust way to convert an image to a pdf. All other ways that I have tried failed in one way or another (resolution, positioning, paper format and other things were wrong....). It can create multiple pages pdf files, set paper format (US Letter, A4, ...) and so on.
• The linux convert utility is a robust tool for cropping images, converting color to black and white (using a threshold for example) and other things. As long as the image is first converted to ppm/pbm. In principle it can also produce pdf files, but that didn't work well for me.
• I sometimes use the unpaper program in the pipeline for some automatic polishing of the images.

I never met my father's father, as he died many years before I was born. He was a soldier in the British Army and was killed in an accident in Burma in the 1950s. I'm told that back then the Army did not repatriate casualties as a matter of course like they do today, so none of the family got to see his funeral or even visit the grave. Like many of the military casualties in the Far East, he was buried in the Colonial Cemetery in Hong Kong. A few photographs were sent to my grandmother of the ceremony and the headstone, but that's all we ever saw.

Fast forward to 2012. I travelled to Hong Kong for a conference (Linaro Connect, more about that later...). I looked up the details of where grandad was laid to rest, and found a major coincidence. He was killed on 27th May 1952, almost exactly 60 years ago! I resolved to go to find him on 27th May 2012 to make the most of this lucky anniversary. Wikipedia told me that in the intervening years the Colonial Cemetery had been renamed to Happy Valley Cemetery, then simply Hong Kong Cemetery, and repurposed from military to civilian use. Jo and I bought some flowers and headed over to Happy Valley on Hong Kong Island on Sunday afternoon. We found a very large cemetery with a helpful map posted to show the different sections, but nothing to tell us where we should be looking. One of the attendants at the office on site looked at the details we had and shrugged to say "sorry, can't help". Ah well, nobody said this was going to be very easy... What we found after a few minutes was that the graves weren't laid out in any obviously logical fashion. In any given small area, you'd find people buried from roughly the same period, but people for any given period could (and would) be scattered across multiple sections at all corners of the site. Great, time for an exhaustive search then. I started combing the site, checking all the markers I could find that looked anything like the tiny grainy black and white photo we had of grandad's headstone.

Almost two hours later, I eventually found him. Two hours of heavy work: the cemetery is built on the side of quite a steep hill, and the prevailing weather was very hot with 100% humidity. But, I forgot all that as we eventually stumbled across the correct grave in (no exaggeration!) the very last part of the last section of the site, 20a.

I was relieved at this point: as things had taken so long, I had started to worry that maybe the grave had been moved or the headstone damaged and lost. But no, I found the man who had gone off to war leaving my dad and aunt as small children. We took some photos and took note of where we had found the final resting place of Sergeant William Alexander McIntyre of the Royal Signals, a man I never met in life but clearly a very important member of our family.

My curiosity was definitely nookd by this story. While this doesn't seem to have been an act of censorship or anything nefarious by Barnes and Noble itself, this does point to just how terrifying ebooks can be if we aren't careful. In the future, when we are dealing with the entirety of human knowledge, we better be careful with the find and replace function. And we better make sure we control our computer systems so that other people can't nook us without our permission. See also:

• War and Nookd
• Nookd

As someone who read War and Peace in paper form, much of it in bed, I definitely empathized with this:

Some weeks ago I decided that I wanted to read Tolstoy's War and Peace. Lou Ann loaned me her copy. At more than 1100 pages, reading it in bed required as much strength as balancing a box of bricks in my hands. In my senior years I have developed arthritis in my thumbs, which made the effort not only difficult, but painful.

A lot of people interested in free software, and user autonomy and network services are very worried about Facebook. Folks are worried for the same reason that so many investors are interested: the networks effects brought by hundreds of millions of folks signed up to use the service. Network effects -- the concept that a good or service increases in value as more people use it -- are not a new problem for free software. Software developers target Microsoft Windows because that is where the large majority of users are. Users with no love for Microsoft and who are otherwise sympathetic to free software use Windows because programs they need will only run there. Folks worried about Facebook are afraid for similar reasons. Sure, you can close down your Facebook account and move to Diaspora. But who will you talk to there? You can already hear people complaining about Facebook the same way they've been complaining about Windows or Office for years. People feel that their hands are tied and that their software, and their social network, will be determined by what everybody is doing. I'm worried about Facebook. But I'm not too intimidated by Facebook's network effects for two reasons. First, using Facebook doesn't preclude using anything else. Twitter has enormous overlapping functionality with Facebook. Sure, people use the systems very differently. But they both ask you to create lists of friends and followers and are designed around sending and receiving short status messages. Millions of people do both and both systems are thriving. For the millions of people who use both Facebook and Twitter, the two services have had to negotiate their marginal utility in a world they share with the other one. People decide that Twitter is for certain types of short messages and Facebook is for others. But these arrangements shift over time. And the relationships between services aren't always peaceful coexistence. Remember Friendster? Remember Orkut? Remember Tribe? Remember MySpace? MySpace, and all the others, are great examples of how social networks die. They very slowly fade away. MySpace users signed up for Facebook accounts and used both. They almost never just switched. Over time, as one platform became more attractive than the other, for many complicated reasons, attention and activity shifted. People logged in on MySpace less and Facebook more and, eventually, realized they were effectively no longer MySpace users. Anyone that has been on the Internet long enough to watch a few of these shifts from one platform to another knows that they're not abrupt -- even if they can be set in motion by a particular event or action. Users of social networking sites simply don't have to choose in the way that a person choosing to boot Windows and GNU/Linux does. I'm sure the vast majority of people with Diaspora accounts use Facebook actively. This is not a problem for Diaspora. It is how Diaspora -- or whatever else eventually achieves what many of us hoped Diaspora would -- could win. Second, Facebook is for the ephemeral. Facebook is primarily used for information that was produced very recently. This week if not today. If not this hour. Facebook has an enormous amount of data that users have fed it that may be hard to get out and move somewhere else. But most people don't care very much about having any regular access to the large majority of this information. What people care deeply about is having access to the data that they and their friends created today. And that data can just as easily be created somewhere else tomorrow. Or, with the right tools, created just as easily in both places. Compare this to something like Windows where moving away would require learning, converting, and perhaps even writing, new software. Perhaps even in new programming languages that most developers don't know yet. Compared to Windows, a migration away from Facebook will be easy. Facebook's photo galleries are an example of an important place where this holds less well. Social network information -- i.e., the list of who is friends with who -- is another example of something that is persistently valuable. That said, people really enjoy the act of finding and friending. Indeed, this process was part of the initial draw of Facebook and other social networks. None of this means that Facebook is over. It doesn't even mean that its ascendancy will be slowed. What it does mean is that Facebook is vulnerable to the next thing more than many technology firms that have benefited from network effects in the past. If users are given compelling reasons to switch to something else, they can with less trouble and they will. That compelling reason might be a new social network with better features or an awesome distributed architecture that allows freedom for users and the ability of those users to benefit from new and fantastic things that Facebook's overseers would never let them have and without the things Facebook's users suffer through today. Or it might be a sexier proprietary box to store users' private information. It doesn't mean that I'm not worried about Facebook. I remain deeply worried. It's just not very hard for me to imagine the end.

I just uploaded the online banking application AB-Transfers to Debian. This is not a finance management application like Gnucash or KMyMoney (i.e. it is not able to keep track of your incomes and expenses). Instead it is mainly intended as a companion that you can use for online transactions that these applications don't support yet. If you were using QBankmanager in the past for this purpose, please take a look at AB-Transfers and consider switching to it. As QBankmanager seems to be abandoned by the upstream author, I consider AB-Transfers the better choice for a pure online transaction application.

My olduse.net exhibit has now finished replaying the first year of historical Usenet posts from 30 years ago, in real time. That was only twelve thousand messages, probably less than many people get on Facebook in a year.. but if you read along this year, you probably have a much better feel for what early Usenet was like. If you didn't, it's not too early to start, 9 years of Usenet's flowering lie ahead. I don't know how many people followed along. I read .. not every message, but a large fraction of them. I see 40 or so unique NNTP connections per day, and some seem to stick around and read for quite a while, so I'm guessing there might be a few hundred people following on a weekly basis. If you're not one of them, and don't read our olduse.net blog, here are a few of the year's highlights:

• After announcing the exhibit it hit all the big tech sites like Hacker News and Slashdot, and Boing Boing. A quarter million people loaded the front page. For a while every one of those involved a login to my server to run tin. I have some really amusing who listings. Every post available initially was read an average of 100 times, which was probably more than they were read the first go around.
• Beautiful ascii art usenet maps were posted as the network mushroomed; I collected them all. In a time-delayed collaboration with Mark, I produced a more modern version.
• We watched sites struggle with the growth of usenet, and followed along as B-news was developed and deployed.
• In 2011, Dennis Richie passed. We still enjoy his wit and insight on olduse.net.
• We enjoyed the first posts of things like The Hacker's Dictionary and DEC Wars.
• Some source code posted to usenet still compiles. I haven't found anything to add to Debian, yet. :)

I will probably be expiring the first year's messages before too long, to have a more uncluttered view. So if you wanted to read them, hurry up!

I've been using a Google account since years, but having a "more serious" one along. The time has come to make the switch and elect the other as main one.

Google has a lot of services, and I'm using several of them, so changing the main account requires to migrate data of those services to the new account, given it's not possible to merge two accounts. To know the full list of services your account is signed in, go to your Account Product page. Some migrations are easy, others hell no: so I'm writing this post to keep track of the migration as it's going on.

First of all, there's a really good post about this migration at LifeHacker: it contains a lot of info I'm using here, but not all the services I need. Additionally, Google has a list of services and methods to migrate from one account to another.

Just to be generic, I'll call AccountSRC the original account and AccountDST the one I want to migrate to.

Migrate GMail

GMail is probably the most important service I have, and also the most difficult one to migrate. I have a lot of filters, so the first step is to migrate them:

1. Login into GMail for AccountSRC
2. Gear icon > Settings > Labs
3. Enable the "Filter import/export" plugin
4. Reload GMail to enable the plugin (if not done automatically)
5. Gear icon > Settings > Filters
6. At the bottom of the page, Select all
7. Export: this will download an XML file with the filters in it
8. Login into GMail for AccountDST
9. Enable "Filter import/export" plugin
10. Load the file containing the saved filters
11. Import all (or select which one) filters; this will also create automatically the labels defined in the filters.

I currently have a forward rule from AccountDST to AccountSRC: so let's revert the forward direction: from AccountSRC to AccountDST. This will allow the new account to receive the mails sent to the old one.

1. Login into GMail for AccountDST
2. Gear icon > Forwarding and POP/IMAP > Disable forwarding: this will stop redirecting mails from AccountDST to AccountSRC
3. Login into GMail for AccountSRC
4. Gear icon > Forwarding and POP/IMAP > add a forwarding address
5. Enter AccountDST as the address to forward mails to
6. A verification code is sent to AccountDST and you'll need to enter that code in AccountSRC to verify you have access to both mailboxes
7. Once verified, select "Forward a copy of incoming mail to" to AccountDST and to delete the GMail copy on AccountSRC

Migrate Contacts

Contacts are available inside GMail:

1. Login into GMail for AccountSRC
2. On the combobox on top-left, select Contacts
3. More > Export
4. Export all the contacts in Google CSV, since it's specific to be re-imported
5. Login into GMail for AccountDST
6. Contacts > More > Import
7. Select the previously saved file and import it

Migrate Reader

Migrating Reader subscriptions is easy:

1. Login into Reader for AccountSRC
2. Reader Settings -> Import/export
3. Export your subscriptions in OPML format
4. Now login into Reader for AccountDST
5. Reader Settings -> Import/export
6. And import the saved OPML file

Migrate Blogger

Login to Blogger for AccountSRC; then, for each blog you have there do this dance:

1. Select the blog
2. Go into Settings > Permissions
3. Add AccountDST as a new author: this will send an email to AccountDST GMail account
4. Accept the invitation (this will ask you to create a new Blogger account for AccountDST if you don't already have one)
5. Go back to to Blogger with AccountSRC
6. Blog > Settings > Permissions
7. Grand admin rights to AccountDST

A copy of this post is sent to soc-coordination@lists.alioth.debian.org as well as to debian-bootstrap@lists.mister-muffin.de. Diary May 21

• Cloned Dose3 and made it build
• Retrieved bootstrap.ml and bootstrap2.ml from old revisions as they were deleted
• Compiled, tested and investigated the functionality of bootstrap.ml and bootstrap2.ml on a theoretical level as no test data was available

May 22

• Pietro sends me a tarball with his current version of bootstrap.ml and dummy as well as real test data
• Created a gitorious account, project and repository
• Compiled, tested and investigated his code
• Ran into several runtime problems with the supplied dummy examples
• Created Makefile to automatically fill ./examples/real/
• Found that .dot files are too big to be rendered
• Trying to figure out how hints work, how base-system was generated and why execution takes hours

May 23

• Pietro made examples work which let me understand the code much more
• Improvement of .dot output and output formatting
• Refactored code into bootstrapCommon.ml for shared functionality and bootstrap.ml for option parsing and main()

May 24

• Play with xdeb.py
• Generate dot graphs with bootstrap.ml and analyze them with sccmap
• Try to find a way to have a reduced package selection other than main archives of ubuntu/debian
• Initial work on trying to find the list of minimal source packages that have to be cross compiled
• Create debian-bootstrap@lists.mister-muffin.de mailinglist

May 25

• Implement a replacement for apt-rdepends and grep-dctrl functionality in ocaml, both working on Package files
• Retrieve list of packages with priority:required
• Retrieve their runtime dependencies
• Retrieve the packages that are added with build-essential and dependencies
• Retrieve the list of source packages that are needed to build the above
• Retrieve list of binary packages that are build from the source packages in addition
• some more functionality in the Makefile

May 29

• Depsolver.dependency_closure replaces homebrew functionality in a better and faster way
• Only consider those binary packages that can actually be installed, given the limited amount of available packages using Depsolver.edos_install
• Create proper list diff by correctly comparing Cudf.package members

May 30

• Big code restructuring
• consider arch:all packages to be available by default
• Got helpful sourcecode comments by Pietro

May 31

• Use Depsolver.trim to reduce a universe to the installable packages
• Compile with dose 2.9.17

June 1

• Basebuildsystem now also writes output to min-cross-sources.list and base-system.list
• Begin work on basenocycles.ml to see how much the minimal system can build without cycle breaking

June 2

• Use Depsolver.trim to find source packages that can be built given the restricted universe
• Find the final list of packages that are available without solving staged build dependencies for Natty
• Many code simplifications

Results I learned a good chunk of ocaml and how to use dose3 and libcudf. I created a gitorious project and a git repository for all the sourcecode.

git clone git://gitorious.org/debian-bootstrap/bootstrap.git

The git as of now contains 30 commits and 1197 lines of ocaml code. So far, 62 emails have been exchanged between me and Pietro and Wookey. I created a mailinglist for this project where all email exchange so far is publicly accessible in the archives. You can also download all of the email exchange in mbox format. Everybody is welcome to join and/or read the list. What seems to be finished: the program that finds the minimal amount of source packages that have to be cross compiled to end up with a minimal build system. What it does is:

1. get all essential packages
2. get their runtime dependencies
3. get build-essential plus runtime dependencies
4. get all source packages that are necessary to build 1.-3. those are the packages that have to be cross compiled
5. get a list of all packages that are built by source packages from 4.
6. add all packages from 1.,2.,3. and 5. plus all arch:all packages to a universe
7. use Depsolver.trim on that universe to figure out which of those packages are actually installable

The result of 7. will then contain a list of packages that are available automatically on the foreign system due to cross compiled source packages and arch:all packages. For Debian Sid, the output of my program is:

# (1) number of packages with priority:required: 62
# (2) plus, number of dependencies of priority:required packages: 20
# (3) plus, build-essential and dependencies: 31
# number of source packages to build the above: 71
# number of additional packages built from the above source packages: 292
# (4) number of packages of those plus arch:all packages that are installable: 6421
# total number of installable packages (1)+(2)+(3)+(4): 6534

For Ubuntu Natty it is:

# (1) number of packages with priority:required: 96
# (2) plus, number of dependencies of priority:required packages: 7
# (3) plus, build-essential and dependencies: 31
# number of source packages to build the above: 87
# number of additional packages built from the above source packages: 217
# (4) number of packages of those plus arch:all packages that are installable: 2102
# total number of installable packages (1)+(2)+(3)+(4): 2236

So for Debian, 71 source packages definitely have to be made cross compilable while for Natty, the number is 87. The last two days I was toying around with these minimal systems to see how big the number of source packages is, that can be built on top of them without running into dependency cycles. After installing the binary packages that were built, I checked again until no new packages could be built. For Natty, I was only able to find 28 additional packages that can be built on top of the 2236 existing ones. This means that a number of dependency cycles prevent building anything else. In the coming two weeks I will focus on coming up with a tool that cleverly helps the user to identify packages that would be useful to have for building more packages (probably determined by how many packages depend on it - debhelper is an obvious candidate). The tool would then show why that crucial package is not available (in case of debhelper because some of its runtime dependencies are not available and require debhelper to be built) and how the situation can best be resolved. The possible methods to do so are to identify a package that is part of a cycle and either cross compile it or let it have staged build dependencies.

When packaging software for Debian, there exist two important assumptions:

1. Compilation is done natively
2. Potentially all of Debian is available at compile time

Both assumptions make the life of a package maintainer much easier and they do not create any problem unless you are one of the unlucky few who want to run Debian on an architecture that it does not yet exist for. You will then have to use other distributions like OpenEmbedded or Gentoo which you compiled (or retrieved otherwise) for that new architecture to hack a core of Debian source packages until they build a minimal Debian system that you can chroot into and continue natively building the rest of it. But even if you manage to get that far you will continue to be plagued by cyclic build and runtime dependencies. So you start to hack source packages so that they drop some dependencies and you can break enough cycles to advance step by step. The Debian ports page lists 24 ports of Debian, so despite its unpleasant nature, porting it is something that is not done seldom. The process as laid out above has a number of drawbacks:

• The process is mostly manual and reinvented every time it is done.
• You need another distribution for the bootstrapping process. Debian itself should be sufficient.
• Its complexity and manual nature prevents architectures with little workforce behind them from catching up to the main archive.
• It also avoids that Debian exists in CPU optimized sub-arch builds.

If Debian would provide a set of core packages that are cross-compilable and which suffice for a minimal foreign build system, and if it would also have enough source packages that provide a reduced build dependency set so that all dependency cycles can be broken, building Debian for a yet unknown architecture could be mostly automated. The benefits would be:

• Putting Debian on a foreign architecture would (in the best case) boil down to making the code cross-compile for and native-compile on that architecture.
• Debian would not need any other distribution to be ported to a different architecture. This would make Debian even more "universal".
• Lagging architectures can be more easily updated or rebooted than when they were initially created.
• Debian optimized for specific CPUs (Raspberry Pie, OpenMoko...) would be more attractive.

With three of this year's GSoC projects, this dream seems to come into reach. There is the "Multiarch Cross-Toolchains" project by Thibaut Girka and mentored by Hector Oron and Marcin Juszkiewicz. Cross-compiling toolchains need packages from the foreign architecture to be installed alongside the native libraries. Cross-compiler packages have been available through the emdebian repositories but always were more of a hack. With multiarch, it is now possible to install packages from multiple architectures at once, so that cross-compilation toolchains can be realized in a proper manner and therefor can also enter the main archives. Besides creating multiarch enabled toolchains, he will also be responsible for making them build on the Debian builld system as cross-architecture dependencies are not yet supported. There is also the "Bootstrappable Debian" project by Patrick "P. J." McDermott and mentored by Wookey and Jonathan Austin. He will make a small set of source packages multiarch cross-compilable (using cross-compilers provided by Thibaut Girka) and add a Build-Depends-StageN header to critical packages so that they can be built with reduced build dependencies for breaking dependency cycles. He will also patch tools as necessary to recognize the new control header. And then there is my project: "Port bootstrap build-ordering tool" (Application). It is mentored by Wookey and Pietro Abate. In contrast to the other two, my output will be more on the meta-level as I will not modify any actual Debian package or patch Debian tools with more functionality. Instead the goal of this project is threefold:

1. find the minimal set of source packages that have to be cross compiled
2. help the user to find packages that are good candidates for breaking build dependency cycles through added staged build dependencies or by making them cross-compilable
3. develop a tool that takes the information about packages that can be cross compiled or have staged build dependencies to output an ordering with which packages must be built to go from nothing to a full archive

More on that project in my follow-up post.

Users of Debian sid will have noticed: the final (and interesting) bits of GNOME 3.4 have landed and if all looks as good as it does now, they should migrate to wheezy in about a week. 3.2 3.4 hasn't been as complicated as the previous horrible transition, but still had some complications due to Cogl/Clutter incompatibilities. Other than that, our major problem has been manpower, but this isn't new for many other Debian teams. We've also seen new incarnations of Linux-only technology is now mandatory which makes our lives a bit more miserable due to kfreebsd-* and hurd-i386, but for now we've still been able to dodge it. It seems wheezy+1 will be fun in that regard though, and we might need to take drastic approaches. If all goes well and the current lot (GNOME Shell, Control Center, Settings daemon, Mutter...) transitions without additional problems, we should be wrapping up our transitions for wheezy with Evolution and friends (currently sitting in experimental), and hopefully GDM 3.4. As we get many questions regarding the status of GDM in Debian, let's add a short note on this. Packaging GDM, at least in its current upstream form, is not a matter of unpacking a new tarball and editing debian/changelog. When Joss works on a new major version, the amount of tweaking to break away from stuff that works on other distros but is not so simple in Debian is outstanding (see, for example, the current unfinished work for GDM 3.2 in our SVN repo). In our case, to handle our GDM defaults, we even need changes to the underlying configuration system, dconf. This evidently takes some effort to do, and unfortunately our GDM expert has had little time for Debian lately, but we're confident we'll end up with a GDM in wheezy that is on par with Debian standards. We are, as always, reachable at #debian-gnome in the OFTC IRC network. Have fun!

Back in 2010, Mike Gabriel showed up on the Debian Edu and Skolelinux mailing list. He quickly proved to be a valuable developer, and thanks to his tireless effort we now have Kerberos integrated into the Debian Edu Squeeze version. Who are you, and how do you spend your days? My name is Mike Gabriel, I am 38 years old and live near Kiel, Schleswig-Holstein, Germany. I live together with a wonderful partner (Angela Fu ) and two own children and two bonus children (contributed by Angela). During the day I am part-time employed as a system administrator and part-time working as an IT consultant. The consultancy work touches free software topics wherever and whenever possible. During the nights I am a free software developer. In the gaps I also train in becoming an osteopath. Starting in 2010 we (Andreas Buchholz, Angela Fu , Mike Gabriel) have set up a free software project in the area of Kiel that aims at introducing free software into schools. The project's name is "IT-Zukunft Schule" (IT future for schools). The project links IT skills with communication skills. How did you get in contact with the Skolelinux/Debian Edu project? While preparing our own customised Linux distribution for "IT-Zukunft Schule" we were repeatedly asked if we really wanted to reinvent the wheel. What schools really need is already available, people said. From this impulse we started evaluating other Linux distributions that target being used for school networks. At the end we short-listed two approaches and compared them: a commercial Linux distribution developed by a company in Bremen, Germany, and Skolelinux / Debian Edu. Between 12/2010 and 03/2011 we went to several events and met people being responsible for marketing and development of either of the distributions. Skolelinux / Debian Edu was by far much more convincing compared to the other product that got short-listed beforehand--across the full spectrum. What was most attractive for me personally: the perspective of collaboration within the developmental branch of the Debian Edu project itself. In parallel with this, we talked to many local and not-so-local people. People teaching at schools, headmasters, politicians, data protection experts, other IT professionals. We came to two conclusions: First, a technical conclusion: What schools need is available in bits and pieces here and there, and none of the solutions really fit by 100%. Any school we have seen has a very individual IT setup whereas most of each school's requirements could mapped by a standard IT solution. The requirement to this IT solution is flexibility and customisability, so that individual adaptations here and there are possible. In terms of re-distributing and rolling out such a standardised IT system for schools (a system that is still to some degree customisable) there is still a lot of work to do here locally. Debian Edu / Skolelinux has been our choice as the starting point. Second, a holistic conclusion: What schools need does not exist at all (or we missed it so far). There are several technical solutions for handling IT at schools that tend to make a good impression. What has been missing completely here in Germany, though, is the enrolment of people into using IT and teaching with IT. "IT-Zukunft Schule" tries to provide an approach for this. Only some schools have some sort of a media concept which explains, defines and gives guidance on how to use IT in class. Most schools in Northern Germany do not have an IT service provider, the school's IT equipment is managed by one or (if the school is lucky) two (admin) teachers, most of the workload these admin teachers get done in there spare time. We were surprised that only a very few admin teachers were networked with colleagues from other schools. Basically, every school here around has its individual approach of providing IT equipment to teachers and students and the exchange of ideas has been quasi non-existent until 2010/2011. Quite some (non-admin) teachers try to avoid using IT technology in class as a learning medium completely. Several reasons for this avoidance do exist. We discovered that no-one has ever taken a closer look at this social part of IT management in schools, so far. On our quest journey for a technical IT solution for schools, we discussed this issue with several teachers, headmasters, politicians, other IT professionals and they all confirmed: a holistic approach of considering IT management at schools, an approach that includes the people in place, will be new and probably a gain for all. What do you see as the advantages of Skolelinux/Debian Edu? There is a list of advantages: international context, openness to any kind of contributions, do-ocracy policy, the closeness to Debian, the different installation scenarios possible (from stand-alone workstation to complex multi-server sites), the transparency within project communication, honest communication within the group of developers, etc. What do you see as the disadvantages of Skolelinux/Debian Edu? Every coin has two sides: Technically: BTS issue #311188, tricky upgradability of a Debian Edu main server, network client installations on top of a plain vanilla Debian installation should become possible sometime in the near future, one could think about splitting the very complex package debian-edu-config into several portions (to make it easier for new developers to contribute). Another issue I see is that we (as Debian Edu developers) should find out more about the network of people who do the marketing for Debian Edu / Skolelinux. There is a very active group in Germany promoting Skolelinux on the bigger Linux Days within Germany. Are there other groups like that in other countries? How can we bring these marketing people together (marketing group A with group B and all of them with the group of Debian Edu developers)? During the last meeting of the German Skolelinux group, I got the impression of people there being rather disconnected from the development department of Debian Edu / Skolelinux. Which free software do you use daily? For my daily business, I do not use commercial software at all. For normal stuff I use Iceweasel/Firefox, Libreoffice.org. For serious text writing I prefer LaTeX. I use gimp, inkscape, scribus for more artistic tasks. I run virtual machines in KVM and Virtualbox. I am one of the upstream developers of X2Go. In 2010 I started the development of a Python based X2Go Client, called PyHoca-GUI. PyHoca-GUI has brought forth a Python X2Go Client API that currently is being integrated in Ubuntu's software center. For communications I have my own Kolab server running using Horde as web-based groupware client. For IRC I love to use irssi, for Jabber I have several clients that I use, mostly pidgin, though. I am also the Debian maintainer of Coccinella, a Jabber-based interactive whiteboard. My favourite terminal emulator is KDE's Yakuake. Which strategy do you believe is the right one to use to get schools to use free software? Communicate, communicate, communicate. Enrol people, enrol people, enrol people.

So, I m taking vacation this week and next. Here s a (partial) list of tasks that are left to do:

• Visit BC for NANOG 55
• Work with Threshold Communications to verify interop is working as well as it seems to be
• Make sure the UTC has acknowledged receipt of our 2012 filing
• Complete move of storage from old chassis to new-ish hot-ish-ness
• Bring one of these chassis online to handle VoIP for the wireless L2 segment
• Bring a 1600 online at our Bartel POP
• Take the disk image created for the above and duplicate it 3x for California-based 44Net nodes
• Bring the VMs whose storage was being provisioned via iSCSI back online
• get the ATA and DSLAM online and hook up the customers who are demanding service.
• Call CenturyLink and explain to them that I WILL report them to the consumer protection department of the Utilities and Transportation Commission if they do not take action to correct their failure in regard to contract negotiation and accounting.

That s probably enough, if not too much. Here s some of the bits I ve done this week:

• Re-establish IPv6 connectivity for many of my hosts
• Twiddled quite a few bits on the BIG-IPs
• Set up a home brew provisioning server for some 1120E hardware SIP phones I ve got laying around (DHCP + TFTP + a bunch of twiddly firmware bits and config files ping me if you want them)
• Did a lot of updates to the DNS infrastructure. We have slave DNS servers in all of our DCs now
• hung out on IRC a lot
• Started accepting IPv4 HTTP requests for web.colliertech.org in Bothell instead of Tukwila
• Put together some perl & php for the WA NGB to help with their online recruiter finder
• Reduced size of SAN base OS (squeeze) to something that fits on a CF
• Moved disks from old SAN chassis to the new one.
• Booting from CF now instead of first partition of first disk in array
• Discovered small (~3k bytes) disk error on one of the RAID members
• Made complete backup of entire array using dd
• Replaced failing disk and began recovery, with great thanks to the linux-raid list

  md0 : active raid5 sdg2[6] sdc2[4] sdb2[3] sdd2[2] sdf2[1] sde2[0]
        4864240640 blocks level 5, 64k chunk, algorithm 2 [6/5] [UUUUU_]
        [=========>...........]  recovery = 46.9% (456668928/972848128) finish=1461.9min speed=5884K/sec
  

• Brought up 802.11g AP for wireless VoIP service
• Configured asterisk to accept 24 SIP lines from GrandStream ATA
• Brought GrandStream ATA online. Currently sending SIP/RTP calls over OpenVPN tunnel to avoid NAT issues. This will be corrected when the new asterisk server is brought online. I ll have it connect to the core with IAX, and it will be on the same physical L2 as the GrandStream and 1120Es
• recovered RAID-1 array on laptop
• made sure KE7KMO is still functioning

And probably some other stuff ;-)

A few years ago I wrote how to extract support status for your Dell and HP servers. Recently I have learned from colleges here at the University of Oslo that Dell have made this even easier, by providing a SOAP based web service. Given the service tag, one can now query the Dell servers and get machine readable information about the support status. This perl code demonstrate how to do it:

use strict;
use warnings;
use SOAP::Lite;
use Data::Dumper;
my $GUID = '11111111-1111-1111-1111-111111111111';
my $App = 'test';
my $servicetag = $ARGV[0] or die "Please supply a servicetag. $!\n";
my ($deal, $latest, @dates);
my $s = SOAP::Lite
    -> uri('http://support.dell.com/WebServices/')
    -> on_action( sub   join '', @_   )
    -> proxy('http://xserv.dell.com/services/assetservice.asmx')
    ;
my $a = $s->GetAssetInformation(
    SOAP::Data->name('guid')->value($GUID)->type(''),
    SOAP::Data->name('applicationName')->value($App)->type(''),
    SOAP::Data->name('serviceTags')->value($servicetag)->type(''),
);
print Dumper($a -> result) ;

The output can look like this:

$VAR1 =  
          'Asset' =>  
                     'Entitlements' =>  
                                       'EntitlementData' => [
                                                             
                                                              'EntitlementType' => 'Expired',
                                                              'EndDate' => '2009-07-29T00:00:00',
                                                              'Provider' => '',
                                                              'StartDate' => '2006-07-29T00:00:00',
                                                              'DaysLeft' => '0'
                                                             ,
                                                             
                                                              'EntitlementType' => 'Expired',
                                                              'EndDate' => '2009-07-29T00:00:00',
                                                              'Provider' => '',
                                                              'StartDate' => '2006-07-29T00:00:00',
                                                              'DaysLeft' => '0'
                                                             ,
                                                             
                                                              'EntitlementType' => 'Expired',
                                                              'EndDate' => '2007-07-29T00:00:00',
                                                              'Provider' => '',
                                                              'StartDate' => '2006-07-29T00:00:00',
                                                              'DaysLeft' => '0'
                                                             
                                                          ]
                                      ,
                     'AssetHeaderData' =>  
                                          'SystemModel' => 'GX620',
                                          'ServiceTag' => '8DSGD2J',
                                          'SystemShipDate' => '2006-07-29T19:00:00-05:00',
                                          'Buid' => '2323',
                                          'Region' => 'Europe',
                                          'SystemID' => 'PLX_GX620',
                                          'SystemType' => 'OptiPlex'
                                         
                    
         ;

I have not been able to find any documentation from Dell about this service outside the inline documentation, and according to one comment it can have stability issues, but it is a lot better than scraping HTML pages. :) Wonder if HP and other server vendors have a similar service. If you know of one, drop me an email. :)

This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (338.26 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Dpkg Like last month, I did almost nothing concerning dpkg. This will probably change in June now that the book is out The only thing worth noting is that I have helped Carey Underwood who was trying to diagnose why btrfs was performing so badly when unpacking Debian packages (compared to ext4). Apparently this already resulted in some btrfs improvements. But not as much as what could be hoped. The sync_file_range() calls that dpkg are doing only force the writeback of the underlying data and not of the meta-data. So the numerous fsync() that follow still create many journal transactions that would be better handled as one big transaction. As a proof of this, replacing the fsync() with a sync() brings the performance on par with ext4. (Beware this is my own recollection of the discussion, while it should be close to the truth, it s probably not 100% accurate when speaking of the brtfs behaviour) Packaging I uploaded new versions of smarty-gettext and smarty-validate because they were uninstallable after the removal of smarty. The whole history of smarty in Debian/Ubuntu has been a big FAIL since the start. Once upon a time, there was a smarty package and some plugins. Everything was great except that the files were installed in a way that differs from the upstream recommendations. So Ubuntu changed the path in their version of the package and did not check whether it broke anything else (and it did break all the plugins). Despite the brokenness of the plugins, this divergence survived for years. So several packages that were using Smarty were modified to use dpkg-vendor to use the correct path depending on whether it was built on Debian or Ubuntu. In 2010, Smarty 3.0 has been released and instead of upgrading the smarty package to this version, one of the smarty co-maintainers introduced a smarty3 package that used yet another path (despite the fact that smarty 3 had a mode to be compatible with smarty 2).
At some point, I informed him that he had to handle the migration of users of smarty to smarty3 he acknowledged and then lost interest in smarty ( I m no longer using it ) and did nothing. After some more bitrot, smarty has been forcefully orphaned in August 2011 by a member of the security team. And in March this year, it has been removed from unstable despite the fact that it still had reverse dependencies (usually removals only happen when they impact no other packages, I don t know why this wasn t the case here). At least the brokenness attracted some attention to the situation and Mike Gabriel contacted me about it. I offered him to take over the various packages since they all needed a real maintainer and he accepted. I sponsored his uploads of all smarty related packages (bringing in the latest upstream versions at the same time). In the end, the situation is looking better now, except that there s no migration path from users who rely on smarty in Squeeze. They will discover that they need smarty3 in Wheezy and that the various paths have to be adjusted. It s probably acceptable since the new upstream versions are no longer backwards compatible with smarty 2 The Debian Administrator s Handbook At the start of the month, I was busy preparing the release of the book. I introduced the publican-debian package to unstable, it s a Publican brand (aka a set of CSS and XSL stylesheets to tailor the output of Publican) using the Debian colors and using the Debian logo. This brand is used by the book. I also created the debian-handbook package and setup the public Git repository on alioth.debian.org. I was ready or so I thought. A few hours after the announce, the website became unusable because the numerous visitors were exhausting the maximum number of client connections. And I could not increase the limit due to Apache s memory usage (with PHP and WordPress). We quickly off-loaded most of the static files traffic to another machine and we setup bittorrent. The problem was solved for the short term. Thousands of persons downloaded the ebook and to this date, 135 copies of the paperback have been sold. Then I took a one-week vacation. Even though I had no Internet at the place I was, I wandered in the street to find a Freewifi wifi network (customers of the Free ISP can use those freely) to stay on top of incoming email. We quickly received some bug reports and I dealt with the easy ones (typos and the like) on the fly. When I came back at home, I manually placed 54 lulu orders for the people who opted for the paperback as reward during the fundraising campaign. A bit tedious but it had to be done (if only Lulu supported a way to batch many orders at once ). I also wanted a long term solution to avoid the use of an external host to serve static files (should a new traffic spike arrive ). So I installed nginx as a front-end. It serves static files directly, as well as WordPress pages which have been cached by wp-super-cache. Apache is still here listening on a local port and responding to the remaining queries forwarded by nginx. Once I ll migrate to wheezy, I might completely ditch apache in favor of php5-fpm to handle the PHP pages. Last but not least, I wanted to bootstrap the various translations that people offered to contribute. I wrote some documentation for interested translators and blogged about it. It s shaping up nicely check it out if you re interested to help! Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

If you use emacs for Javascript or Node.js development, you should have a look at js2-mode (apt-get install js2-mode in Debian / Ubuntu). In addition to providing syntax highlight, it will parse your Javascript and issue errors and warnings. (It's not as pedantic as jslint or jshint so you'll probably want to fix all of them before committing your files.) Unfortunately the default indentation style looks a bit crazy and adds excessive indentation to your code. Here's an example:

    navigator.id.secret.generate(identity, function (plainKey, wrappedKey)  
                                     var data =  wrappedKey: wrappedKey ;
                                     $.post('/loggedin', data, function (res)  
                                                $("#message").text('All done');
                                             , 'json');
                                  , function (error)  
                                     $("#message").text('ERROR: ' + error);
                                  );

It turns out that indenting Javascript properly is really hard, but you can turn on a special flag which will cause the indentation to "bounce" between different strategies, starting with the most likely one. This is what I now have in my ~/.emacs:

(custom-set-variables
 '(js2-basic-offset 2)
 '(js2-bounce-indent-p t)
)

You can find and configure all other options by issuing the following command:

M-x customize-group [RET] js2-mode [RET]

So yes, you can have both the reasonable indentation of the standard js-mode and the helpful warnings and errors of js2-mode!

In the past days, I looked at various memory allocators in a quest to improve performance in my multi-threaded test cases of a reference counting language runtime / object allocator (a fun project). It turns out the glibc s memory allocator is relatively slow, especially if you do loops that create one element and destroy one element at the same time (for example, map() on a list you no longer use after you passed it to map). To fix this problem, I considered various options. The first option was to add a thread-local cache around malloc(). So whenever we want to free() an object, we place it on a thread-local list instead, and if a malloc() request for an object of that size comes in, we reuse the old object. This fixes the problem with the lists, but experiments shown another problem with the glibc allocator: Allocating many objects without releasing others (let s say appending an element to a functional list). I started testing with tcmalloc instead, and noticed that it was several times faster (reducing run time from 6.75 seconds to 1.33 seconds (5 times faster)). As I do not want to depend on a C++ code base, I decided to write a simple allocator that allocates blocks of memory via mmap(), splits those into objects, and puts the objects on a local free list. That allocator performed faster than tcmalloc(), but was also just a simple test case, and not really useable, due to potential fragmentation problems (and because the code was statically linked in, causing thread-local storage to be considerably faster, as it does not need to call a function on every TLS access, but can rather go through a segment register). I then discovered jemalloc, and tried testing with this. It turned out that jemalloc was even faster than tcmalloc in all test cases, and also required about the same amount of memory as tcmalloc (and slightly more than the custom test code, as that combined reference counting with memory allocator metadata and thus had less meta data overhead). Using jemalloc, the list building test performs in 0.9X seconds now (instead of tcmalloc s 1.33 or glibc s 6.75 seconds), requires 0 major page faults (tcmalloc has 2), and uses 5361472 kb memory (tcmalloc uses 5290240, glibc requires 7866608). Given that jemalloc is written in C, I can only recommend it to everyone in need of a scalable memory allocator. Depending on your workload, it might require less memory than tcmalloc (at least that s the case at Facebook) and is most likely faster than tcmalloc. It also provides tcmalloc-compatible profiling facilities (as Facebook needed them). Furthermore, jemalloc is also the memory allocator used by FreeBSD, and is used by Mozilla projects, and on several Facebook projects, and should thus be relatively stable and useable. All tests were run with 4 threads, on a dual-core laptop with hyper-threading, using (e)glibc 2.13, tcmalloc 2.0, and jemalloc 3.0. The tests may not be representative of real world performance, and do not account for fragmentation. Should I try replacing Python s custom caching of malloc() calls with simply calling jemalloc, and run a few Python benchmarks? Anyone interested in that?
Filed under: General

Traffic jams suck; but they do give you a chance to think and blog. Truckers hams are telling me that four trucks are totalled and need to be removed by the THW. As the trucker next to me just broke out his gas grill and several people are joining him in a spontaneous BBQ, I don't expect to be moving any time soon. Kicking things along This is coming along nicely RAID 4 Unless you are forced to support legacy systems or playing RAIDemon and gotta try them all, there is no valid use case for RAID 4. So doing so is a patently bad idea. RAID 5 is always the better option as you are not relying on a single disk which contains the checksums, but use striping to distribute them. RAID 4 has a chance of 1/n that a failed disk will force all remaining disks to read all respective sectors and then the controller/CPU to XOR everything. There is a reason why RAID 3 was replaced with RAID 4; just as there's a reason why RAID 4 was replaced by RAID 5. Don't use RAID 4. Seriously. Tom Tom vs OSM In case you have been following the news in the GPS circles, you will have noticed that TomTom pretty gave up on innovating and resigned themselves to dying a slow death. Well that and to start a nice amount of FUD. While they have been rebuked pretty thoroughly already, I wanted to add my own POV. There is a kernel of truth in what TomTom said inasmuch they state that pedestrian paths are better than car paths. Of course, while they got the relative quality right, they are wrong about the absolute quality of the data. They claim pedestrian is OK and motorized is bad. In my experience, maps for motorized traffic are good while pedestrian tracks are mind-boggingly great. These days, most people (and probably all who read this) will be at least somewhat familiar with the quality of OSM in the random cities they visit over the year. The common verdict is usually "very nice". But that's not where OSM shines. It's in the lesser-known places where you don't expect too many tech-minded people to come along where it starts to really impress me. While I found a few stubs on a dead-end street in Akureyri which were not mapped, every foot trail I ever walked along in Iceland has been mapped. ski-doo tracks on Svalbard? Check. Backyard in Novosibirsk? Check, as well. And we are talking "packed earth track in between garages opening up into a 'playground' where a man and a pregnant women are passing a bottle back and forth while smoking and watching over their two kids who play next to a middle-aged man who is lying on the ground wasted, using a rock as a pillow all the while an extremely blingy Mercedes with two muscle-packed thugs is cruising by, all in the time it took us to walk through". Mapped. Horse tracks and hiking trails in Mongolia? Yep. Hutongs in Beijing including alleys so narrow you have to walk single file? Of course. Richard Fairhurst is right that the plural of anecdote is not data, but OSM is good. TomTom is afraid for a reason. As an aside, I find the picture used in TomTom's newsletter extremely funny; I am sure they didn't intend this, but hey. Windows 100-300 MiB for any random driver installation package is par for the course. Be it GFX, audio, or network card. If you plug in a different USB mouse, Windows 7 will take several seconds while "setting up" a standard USB HID device. And hitting ctrl-alt-del in the login screen or while logged in does not give you a quick way to shut down or reboot the machine. As I am breaking out the Windows installation every month to install updates and every six months to actually use this does not affect me a lot. Still, it's amazing that so many people are putting up with this. UEFI It's nice to see that Matthew Garrett is still going strong. I am not sure if this is the best way, but it seems to be a workable solution while in between a rock and a hard place. Not sure about the requirements of GPLv3 with regards to the keys Fedora will use to sign GRUB 2 with. From what I remember without rereading the license, as long as Fedora is not selling a system on which you can not disable secure boot they should be in the clear. IANAL, IIRC, etc. Wrapping up And I have not moved a single meter, other than making way to a police motorbike, in over an hour. Pity I don't have anything to BBQ with me; smells delicious over there.

I ve expanded a bit on my idea (that is, the sbuilder wrapper to help aid with managing more then one sbuild chroot), and it got out of hand very quickly. It s gone from one small shell script to a set of 8, and growing. Quickly. Until I spin these off into their own repo, they ll be in my dot-bin Before getting too far, please relize this is only how I m using sbuild. There are many other (more) sane configurations that you can (and should) use. This was not made to be useful for anyone other then me :) I ve added the ability to use unionfs-fuse, set up a way to login interactively and a script to create new chroots without having them register as a global schroot. Firstly, forgive me for my sins. This is all a huge hack, but it s use isn t the worst thing in the world. As of the time of this writing, I now have a gcc and clang chroot for each of stable, testing and unstable. Now the big question how do you use this. Rather, this is more of a how do I use sbuilder :) Firstly, I ve created a file for each schroot set that I m using. Here s my /etc/schroot/chroot.d/sid.

[sid]
type=directory
description=Debian sid/amd64 autobuilder
directory=/var/sbuild/sid
groups=root,sbuild
root-groups=root,sbuild
script-config=sbuild/config

It s very similar to the one that sbuilder-createchroot creates. If you re just looking for aufs / unionfs support, keep in mind schroot has some great support. check man 5 schroot.conf. Next, create a dist.d directory in /var/sbuild such as /var/sbuild/sid.d. In the future, this might get automagically created by the sb-create script. For now, it just expects that exists. The arguments for all the sb* commands is always in the order of variant, dist. variant defaults to generic , which I ve set as an alias to the generic chroot, and the dist defaults to sid . At some point, I ll change the default to be the last UNRELEASED entry in the changelog. So, to create a new chroot to become your clang chroot, you may choose to invoke it as sb-create clang, or more explicitly sb-create clang sid. To create a new chroot to become your stable / gcc chroot, you may create it with sb-create gcc stable. After you have a chroot, you can start to use the sb or sbo scripts. sbo is just a wrapper around the sb script that sets an envvar to tell the sb-prehook and sb-posthook to set up the unionfs overlay. The unionfs overlay routine replaces the dist symbolic link (such as /var/sbuild/testing) with a folder, and mounts the real chroot (such as /var/sbuild/testing.d/testing-gcc) under the unionfs divert point (which is something like /var/sbuild/testing.overlay.d/testing-gcc). This lets us make sure the chroot is never affected by a routine build, even if it s designed to trash a chroot (unless, of course, it uses any number of ways to get outside the chroot and trash the pristine directory this is to prevent mistakes, not malicious scripts). By the way, for those interested, there s a small issue with what I m describing it s very subtle. Basically, there s a few assumptions made by how schroot operates, and handles sessions . Since it ll keep stuff mounted if the session is still active, it makes purging all this a real pain. The magic behind the scripts is in the sb-endsessions routine which will properly end all the schroot sessions for that dist, and since we have our own high-level locking system, we shouldn t have a situation where this kills off another build. The code (which might have issues) looks like:

SESSIONS= schroot --all-sessions -l   grep "^session:$DIST-" 
for session in $SESSIONS; do
    echo "W: Ending session $session"
    schroot -e -c $session
done

By backing of schroot s mounts, we can do our mounting / removing without worrying about klobbering schroot. Now, there s also the case of wanting to interactively log into a chroot. For that, there s sb-login and sb-maint, both take the usual variant / dist arguments. sb-login logs in as the current user, and sb-maint logs in as the root user. Eventually, I ll be adding some scripts to upgrade all the build chroots for a dist at once.

Hello!

I'm Sorina and this summer I will be working on Debian Installer network setup during Google Summer of Code. I am a second year bachelor's student at Politehnica University of Bucharest, I am studying Computer Science. I very much enjoy what I am doing even if sometimes I feel that the amount of stuff I want/have to learn/do/work on is infinite and I don't really know where to start.

I am at the beginning of the project, I believe now I am accustomed to the way things work and I am waiting forward to finishing the exam period so that I could really start working.

The hardest part for me in what I've done so far is testing what I am doing. It took me a while to figure out at least a part of how the installer functions and to solve the problems that occurred while building. Luckily, there is documentation and internet and even more important, I had help and support from my mentors.

So far I have only written two patches for netcfg, for bugs #610752 and #636211. For the first one I had a version for a while now, but I was unable to test it (unfortunately, it's likely that you will hear this a lot in the future). The idea was to make the link detection timeout preseedable and for this I have added two new templates, asking the user to provide a value for that matter.

The second one involved a missing "Go Back" button. This seemed to be the result of not letting debconf know that the program is capable of handling the user pressing a back button. That could be solved using deb_capb.

As I said before, testing was my main problem. Eventually, I built the image, wrote it to an USB and rebooted my laptop. This served its purpose, since I had access to anything I needed such as the wireless interface and physical access to the network cable. Running it through a virtual machine did not help much in this case, because I was unable to provide those.

Another thing I considered to be hard was blog posting, but now that I have just finished my first post it seems that I overreacted.

That's all for now, have a nice day!