Here's a script I use to speed up compilation at work. Since I work at Collabora, I call it collaboratively. It's a prefix to a command (used in the same way as sudo or nice) which sets up icecc (Icecream) and ccache to work nicely together.

#!/bin/sh
# ~/bin/collaboratively
# Typical usage: collaboratively make check
PATH=/usr/lib/ccache:/usr/lib/icecc/bin$(echo :$PATH   sed -e s@:/usr/lib/ccache@@g)
export PATH
MAKEFLAGS='-j -l3'
export MAKEFLAGS
exec "$@"

Some more explanation of what it does:

• First, I want the gcc wrappers (actually symlinks to ccache) in /usr/lib/ccache to be invoked. This uses ccache to make sure I don't actually recompile source code that hasn't changed.
• Next, I want to use icecc, which distributes builds around the office. We run an icecc scheduler on the office server, which also contributes two CPU cores and lots of RAM to the compilation effort.
• To prevent infinite recursion between icecc and ccache, I have to make sure ccache isn't in my $PATH for a second time! My bash and zsh dotfiles automatically set up ccache, so in practice it does need editing out.
• When distributing builds, make should be parallelizing as many compiles as feasible - I use the flags -j -l3 to run an unlimited number of parallel compilation processes, but stop when the load average reaches 3 (the number of CPU cores I have, plus 1, seems a reasonable maximum load-average).

apt-get install icecc-monitor and run icemon to see whether this is all working - if it is, you should see compile jobs going off to other hosts.

I spent much of this week in Alicante. It was wonderful to have a few days away from the beginnings of Winter in Cambridge - Alicante's in South West Spain, right on the Mediterranean. It's a lovely city, although it was very quiet due to the time of year. I was invited down to Alicante by organisers of the Congr s de Programari Lliure, Comunitat Valenciana, to present a talk about Debian. That I did, and as always my slides and photos are available in case people are interested. I wasn't too sure about what level to aim the talk, so I deliberately went for a general introduction to Debian. That seems to have been the correct level - nobody threw anything at me, and nobody was visibly falling asleep. *grin* It's also the first time I've ever given a talk with official translators, which is a weird experience! The main topic for the conference was the new release of Lliurex, Valencia's own GNU/Linux distribution targeted primarily at use in schools. The conference was massive; I was told there were about 1500 attendees in all, most of whom were teachers who wanted to come and learn more about Free Software. There was a good spread of talks and workshops, covering topics from development through to deployment in the classroom. The level of enthusiasm for Free Software here was immense - lots of people were keen to learn more about it and get involved. I was told that out of the 15 regions/states of Spain, 11 of them have now started their own distributions to use in schools! Extremadura were the first region to experiment with Free Software like this, and their success with LinEx was a clear inspiration to their neighbours elsewhere in Spain. As well as the obvious cost advantages of moving from proprietary to Free Software, the local governments also love the freedom to use and modify their software however they like - including the ability to provide their own translations. I met up with a lot of cool people at the conference too. The local education minister (Alejandro Font de Mora) was keen to talk with all the speakers and very enthusiastic about the future of Free Software in his schools. As is increasingly common at FS events around the world, there were of course quite a few Debian people around too. Luciano and Miriam were also giving talks, and I bumped into Rene and Miguel and a whole bunch of others whose names I'm too crap to remember - sorry! I also spoke a lot with Kurt Gramlich, the mega-enthusiastic Skolelinux developer who was also happy to push Debian to everybody he spoke to *grin*. Jon "Maddog" Hall had some excellent ideas that I'm sure we'll all hear more about soon, and his talk was excellent. He did a very good job of explaining how Free Software makes such a difference, including how it helps people to make money. He also explained that due to travel he had missed Halloween at home and he wanted to share the experience of scary masks with the people at the conference: I had a great time in Alicante, and I'm very grateful for the invitation to go there. I wish that the UK was as far as advanced as Spain in the use of Free Software in education - they've done a huge amount of good work in this area and I'm glad we can help them.

With a small mail to our mirrors-announce list, I just opened the work of more than one month to the public. Even better, I strongly encourage every mirror admin to use my scripts, or at least scripts that follow a minimal set of rules I wrote down… Some long time (somewhere in April or May this year) I got added to the Debian mirroradm group. First it was only to get some problems with packages.debian.org solved, but later on I started doing more. This group is basically the one that handles the archvsync user on .debian.org machines and also manages the official Debian mirrors listing. As the archvsync user exists on multiple .debian.org machines, and many of them do mirror a copy of our archive for various reasons, all of them had some version of a mirroring script, and sometimes also a version of a pushmirror script. Unfortunately, as they all had been maintained mostly on their own per machine - they sometimes had evolved a bit different. Additionally all of them had a few problems which could leave us with a broken mirror behind, despite it looking all well. So on Sept. 26 I decided to trash it all and rewrite the script and put it all in a VCS (and selected git). For that I set myself a few goals, like (in no particular order)

• It is all proper set -u,
• should have as less external dependencies as possible and should work by only using bash (only the runmirrors script does need egrep),
• should be as flexible as possible and support as many different configs as one can imagine without the need to change the script itself,
• should replace the existing mirror scripts of Debian and not only be for the archvsync user alone,

Now, 109 commits and more than a month later, I was finally able to open the scripts to the wide public. (Had multiple people test them before this, but thats always different than a real “release”). The scripts are in use on all .debian.org machines and some other mirrors already, so I’m fairly sure they work.

---

As I already wrote, Debian mirror admins are strongly encouraged to use this scriptset and not to invent their own. In case you miss a feature either send me a mail describing it, or better, a patch implementing it. I’m happy to implement as much as possible (and sensible :) ) And while it is discouraged to use own scripts, sometimes people have reasons to have their own, so there is now a minimal set of functionality that such a script MUST support.

• It must perform a 2-stage sync. The archive mirroring must be done in 2 stages. The first rsync run must ignore the index files. The correct exclude options for the first rsync run are:

  --exclude Packages* --exclude Sources* --exclude Release* --exclude ls-lR*
  

  The first stage must not delete any files. The second stage should then transfer the above excluded files and delete files that no longer belong on the mirror. Rationale: If archive mirroring is done in a single stage, there will be periods of time during which the index files will reference files not yet mirrored. And that will lead to a lot of confusion for our users.
• Must not ignore pushes whilst running. If a push is received during a run of the mirror sync, it must not be ignored. The whole synchronization process must be rerun. Rationale: Most implementations of Debian mirror scripts will leave the mirror in an inconsistent state in the event of a second push being received while the first sync is still running. It is likely that in the near future, the frequency of pushes will increase.
• Should understand multi-stage pushes. The script should parse the arguments it gets via ssh, and if they contain a hint to only sync stage1 or stage2, then only those steps should be performed. Rationale: This enables us to coordinate the timing of the first and second stage pushes and minimize the time during which the archive is desynchronized. This is especially important for mirrors that are involved in a round robin or GeoDNS setup. The minimum arguments the script has to understand are:
  • sync:stage1 Only sync stage1
  • sync:stage2 Only sync stage2
  • sync:all Do everything. Default if none of stage1/2 are present. There are more possible arguments, for a complete list see the ftpsync script in our git repository.

All of those requirements are easy to fulfill. Well, at least the MUST ones. The first one is basically a given, all scripts I know are doing this already. The second one sounds more complicated, but actually might just be about 4 lines of actual code changes. Just touch an extra file before you gain a lockfile (to avoid two scripts running at the same time) and then rerun rsync as long as that extra file is around. (It helps to rm this trigger file right before you call rsync :) ). The should directive is actually a little more work to implement correctly. I won’t show source here, read the ftpsync script if you want to see it, but I try some small explanation. Background: Debian mirror synchronisation is usually done by scripts that get started via ssh. As mirror operators (understandably) do not like their upstream mirror to be able to login, this is done using “ssh triggers”, a feature I wrote about earlier. Those are sshkeys using a forced command, so nothing else than this command will be run, ever. SSH, if used with such forced commands, has the helpful feature that it sets the environment variable SSH_ORIGINAL_COMMAND containing the original arguments. Which means that, if the mirror pushes you with “ssh yourhost -l mirror sleep 1” you will see *sleep 1 in that variable. It is what many mirror operators currently use when pushing their downstreams. Now, we can use that to tell the receiving end all things we want it to know. If you look in my script you will see that I defined a little more than the minimal arguments. But none of the additional ones are a requirement, they only make life much easier. My script currently understands, additional to the 3 minimal ones:

• sync:archive:$FOO This allows your upstream to tell you what archive needs an update. This way you can receive many different archives from your upstream, but only need to setup one ssh key. Adding another archive to the list you get notified about from them is now just copying the config file in place. (But the main reason is to not need dozens of keys and easy management of the authorized_keys files)
• sync:callback A feature I don’t yet use, but have implemented, are callbacks. Initially I planned to do my “update DNS RR listed mirrors at the same time” using callbacks, but then I found a better way for doing that, see further below. The idea is that you can configure a user, a target host and an ssh key. And “call back”, ie. tell your upstream you finished mirroring the archive. Could be used for statistics. Or, thinking a little more complicated, your upstream could only allow rsync access for the time between his push to you and you calling back. Or whatever, I’m sure people will come up with ideas.

---

As if that wasn’t enough, I also had to rewrite the runmirrors script. The old 10-line one just didn’t cope with the more general approach. This script is used to “push” your downstream mirrors, telling them that they need to update. (Which is the only sane way to run a mirror. cron is just not made for mirrors that can update at random times!). One nice new feature of it is the ability to update multiple mirrors nearly at the same time. This is extremely helpful in case of DNS Round Robin entries like security.debian.org (multiple hosts offering the same archive under one hostname). You do not want to have such hosts out of sync. Not even for a minute, if you can avoid that. So we now push the mirrors with sync:stage1 as argument. Then we wait until all of them have finished the stage1 sync before we push them all again, this time with sync:stage2 as argument. This will erase any update delays thanks to network latency / low bandwidth one of the security mirrors might have, as the large transfers (all the binary and source packages) are done in stage1. They can take time, a normal user won’t notice it - the Package and Source indices are not transferred yet. They arevery small and as such transfered pretty fast whenever stage1 is done everywhere and we tell mirrors to do stage2.

---

All in all it was a pretty nice exercise writing those scripts. Comments: 0

This one didn’t arrive on time for the previous post, so here it is now. I’ve been meaning to find alternatives to Konsole for a bit now. I’ve always wanted the ability to click on URLs in my terminal emulator and have them open on my browser, which the Konsole in KDE3 can’t do. When I learnt that KDE4’s Konsole can, I ooh-ooh-ooh’ed and I gave it a try. But whilst KDE3’s Konsole is a very good application, amazing, even, I found KDE4’s to suck in a variety of ways, including speed. And then my quest began. I gave rxvt-unicode a try yesterday, and so far I’m very satisfied. I quickly managed to configure it to my taste; it has tabs, which I can’t do without, and you can click on URLs as well. These two features are implemented via Perl extensions because, yes, rxvt is extendable with Perl. These two extensions are not very configurable, though, so in order to have the Next tab/Previous tab movements use my preferred keys, I had to edit the Perl source and place it under $HOME.

A couple of years ago, I started what has probably been the best job I've held so far, as a Business Development Manager for Artec Group where I focused on marketing and selling the ThinCan LTSP client. During that time, I used a combination of Guerrilla Marketing techniques to spread the word about this cool Estonian embedded platform in its LTSP client configuration and defined an extremely tight market-oriented focus that saw this relatively obscure electronic design start-up rise to the forefront of the LTSP hardware market and also brought the company commendable notoriety in the Coreboot community, thanks to its Programmable LPC Dongle, a low-cost ROM emulator that can be used to bootstrap various embedded hardware and test firmware images. Looking back at close to 3 years with Artec (if we include the stint I did at Linutop), I most fondly remember how I managed to bring the Estonian and Turkish nations closer together in collaborating on Innovation. On my first visit to Turkey, I had noticed that very little is know about Estonia in Turkey, even though both countries have a collaboration that dates back to the 1920's, so my tactic was to introduce Estonia's achievements as a whole, to establish confidence in Artec's offering. It worked: Among other things, my idea to organize an Estonian ICT Conference during the visit of Estonian PM Andrus Ansip in Turkey created a lot of extremely positive buzz that resulted in Estonia being mentioned as a trendy country in a prime time Turkish TV series. During this time, I've also made a lot of contacts among the Estonian and Turkish business communities and, more surprisingly, among the Estonian diplomacy who, much to my amazement, welcomed my ideas for better promoting Estonian interests abroad with great enthusiasm. Come December 2008, my time at Artec will become a page in my employment history. While the Thincan remains one of Artec's key products, current marketing and sales resources are focused on new markets and new technologies that completely fall outside my core expertise, which is why the company decided to let me go. I thus welcome offers from interested parties in Coreboot or LTSP markets. Other refreshing offers from outside the box - such as in business diplomacy - are equally welcome, as I've been pondering a career change for a long time.

Note: this is quite long and might not be interesting for people not involved with /in either of: debian i18n, debian l10n, git, shell scripting.


For some time now I have been the de-facto coordinator of the Romanian localization team. During this time I was faced multiple times with problems related to motivation of the team members, setting goals for a release, coordination of the changes inside the team, integration of new translators in the team, loosing team members, my lack of time on some occasions and other problems.

I thought a lot about how to improve on those points and I was never satisfied with the answers I got. What I think was the worst problem in the team was the inability to set clear goals, especially during lenny's development cycle.

During sarge's and etch's development cycles we were in an infancy state and setting as the goal to have a fully translated installation process was really enough to keep people motivated. For sarge we missed by a small margin, but the translations were of poor quality, while for etch we worked more on improving the translations.


But during lenny it was bad, really bad. My free time had been shrinking for a while, starting with etch's release, and we were unable to set clear goal, since for etch we managed to have the installation process fully translated and a few other translations. There was no way for us to reach 100% translations during lenny's development cycle, so setting that as a goal was really unrealistic. Percents by themselves don't mean anything for people, and as long as there's no substance to those numbers, there's no motivation to reach for one arbitrary percent.

I tried to set as a goal translating the packages installed by default in a new installation, but that hit the eternal question How do we know easily which are those packages?. This remained sometimes unanswered or got an unsatisfying answer. Also, there was a goal to have correct diacritics for Romanian in lenny, to have aspell-ro that uses the correct diacritics.

I even got to a point that I, myself, lost my motivation and set myself a personal goal of overrunning the level the translations that the language just above Romanian had in the po-debconf l10n statistics (ranking between languages). This was nice way for me to keep myself motivated, but I had my reserves in making this motivation public out of fear of being misinterpreted, because, by a strange coincidence, that next language was Hungarian, and in Romania's history there was some friction between Hungaria and Romania, while there are still some tensions with the Hungarian minority in Romania, in areas where they represent the local majority.

I managed to reach my personal goal, but this wasn't addressing the big picture.



So, sometime around the start of this year I started thinking about ways to coordinate the Romanian localization team in order to have:

• a clear goal at any given time
• a way to always be able to change that goal as we go
• a way to sync with eventual calls for translations, or the current sid translations
  
• stats immediately available
• automated checks for spelling, correct diacritics usage and other checks that might be useful (e.g. translation completeness)
• an easy way to assign somebody else as a language coordinator (I would appreciate some help or I might even consider stepping down)
• easy integration of new translators (by providing immediate answer to the question "What can I do to help with the translations?")
  

For short, a tool that would allow the team to work more efficiently while having the possibility to set clear goals in order to keep people motivated.



So after some pondering, I thought that creating a repository with the translations and the helper tools that would do the funky sync, checks, stats would be the best way to do that. So I started hacking on that somewhere around July-August and I published the result, but without much publicity, since it is still incomplete.

Some of the technical details are still in a haze, but I have a general idea and I got some basic functionality.


Today I decided that I should announce this semi-officially though my blog, maybe I get some input, ideas, or even contributions (I really should write a TODO).




I give you the Debian L10n Romanian coordination repository.

This is a git repository that has some tools to facilitate translation coordination and the translations that are current in the distro for the team.


Can be cloned with:

git clone git://git.debian.org/git/users/eddyp-guest/debian-ro-repo.git

or, if you're behind a restrictive firewall:

git clone http://git.debian.org/git/users/eddyp-guest/debian-ro-repo.git



Currently the work flow for updating a translation is as follows:

1. source _bin/polibs (. _bin/polibs)
2. cd foo
3. po_refresh
4. complete the translation
5. po_rearrange "ro.po"
6. git add "ro.po" && git commit -m "updated translation for foo"
7. send the translation ("git format-patch origin" and send the patches by mail, or, alternatively, just "git push")
   

Features:

• provides a po_refresh function that can import material directly from http://i18n.debian.net/material, but can also allow manual imports (template.pot from a call from translation)
• for a new translation: source _bin/polibs (. _bin/polibs), make a directory with the name of the source package, cd into it, and run po_refresh
  
• po_rearrange - beautify and unify the layout of PO files (facilitating compact and sane diff-ing for PO files)
• po_merge uses compendium, if present
  

Planned features:

• sync translations/templates from package VCS-es (Vcs-* headers and debcheckout should be the means to the end)
• po_rearrange should be called as a pre-commit hook; should either reject the commit if the po file was not rearranged, or automatically rearranged before the commit
• generate stats
• add commands for "what's outdated", "what needs review", "submit translation", and maybe "reserve translation for offline use"
• conflict merges should be done via po_merge (.gitattributes is key here)
• support other file types (?) - does this make any sense?
• periodic and automated sync with sid for all translations
  

Problems:

• security - running tests automatically from files within the repo doesn't seem too wise, but looks like the only way to get automated testing on any translator machine; maybe keeping the code in a submodule might address this issue?
• entry level translators still have a hard time - UI sucks now; there should be a wrapper command that should use the library functions and should provide a useful help
  
• is git too difficult ? - git backend usage maybe should be cloaked?
• still in development/alpha stage - I still haven't figured some of the issues
• central repo or really distributed - should there be a central git repo where the coordinator(s) do the pushes? it seems the central repo with a small pushers team for new translators (which can't commit directly) might actually facilitate interactions between experienced and new translators to instruct/bring up to speed the rookies

I was hoping that the release notes for lenny would facilitate from this infrastructure, but unfortunately I was lately in a really inactive period wrt Debian.



Questions, suggestions, ideas are welcome.

So, as everyone knows by now, the party which is a lot like our Conservatives won the US elections (instead of the other party which is a lot like our Conservatives). I’m particularly touched by Aq’s comment:-

The mood in America has been likened to the one here in 97 when we managed to kick the conservatives out for the first time in about 20 years, which amuses me. However, the fact that ours failed to live up to that optimism doesn t mean yours necessarily won t

Oh well, at least it was better than drifting further to the right (in both cases). I’ll wait and see whether it lives up to the optimism. As you may know, SPI is associated with the Open Voting Foundation. That doesn’t sit well with me - while paper voting is imperfect, it seems more verifiable and scalable than machine voting. I’ve been assured that OVF seeks to replace current election machines, rather than spread machine voting to new places and it looks like my area is not under immediate threat (ORG latest news), so I’m not working on it just now. Broadly, I agree with Simon Rumble on Voting machines: a solution searching for a problem?

So without even going into the serious problems with voting machines, it seems they don’t actually solve any actual problems, and I suspect cost a lot more to operate

I’m currently finding out more about standing for election to The Cooperative Group committee and ThePhoneCoop board - I’m not sure yet, but I think both still use paper voting. Cooperatives-SW and TTLLP both use in-person voting. What’s the current thinking here? What’s best?

So, as everyone knows by now, the party which is a lot like our Conservatives won the US elections (instead of the other party which is a lot like our Conservatives). I’m particularly touched by Aq’s comment:-

The mood in America has been likened to the one here in 97 when we managed to kick the conservatives out for the first time in about 20 years, which amuses me. However, the fact that ours failed to live up to that optimism doesn t mean yours necessarily won t

Oh well, at least it was better than drifting further to the right (in both cases). I’ll wait and see whether it lives up to the optimism. As you may know, SPI is associated with the Open Voting Foundation. That doesn’t sit well with me - while paper voting is imperfect, it seems more verifiable and scalable than machine voting. I’ve been assured that OVF seeks to replace current election machines, rather than spread machine voting to new places and it looks like my area is not under immediate threat (ORG latest news), so I’m not working on it just now. Broadly, I agree with Simon Rumble on Voting machines: a solution searching for a problem?

So without even going into the serious problems with voting machines, it seems they don’t actually solve any actual problems, and I suspect cost a lot more to operate

I’m currently finding out more about standing for election to The Cooperative Group committee and ThePhoneCoop board - I’m not sure yet, but I think both still use paper voting. Cooperatives-SW and TTLLP both use in-person voting. What’s the current thinking here? What’s best?

Hey Romain, you d be impressed by the number of upstreams whose knowledge of security measures is similarly close to the absolute zero. Since bad guys are always digging through commits for suspicious things, hiding the relevant commits is the best thing you can do to help them. Given things like this exist, It s likely that they have even automated tools to do this search for them. This way, distributors and security engineers have to do the same job, hoping to never miss a change that was actually a security fix. Unfortunately, two small scale projects that we happen to use have exactly the policy Romain described regarding security updates. I m amazed by the fantastic job the distributions security teams manage to achieve in these conditions.

OK, I know I've been very quiet on the blog for weeks now but I have my reasons and all will become clear, eventually. I'm principally waiting for the Lenny release before being able to start full discussions about certain ideas that have been consuming a lot of my time recently. I continue to do what I can to get Lenny out of the door because the wait is driving me nuts. (Fixed another RC bug yesterday, yippee!)

Anyway, something I can talk about is Emdebian Grip. Emdebian has always wanted to support "in-between" devices that don't need the full-on cross-build everything approach of the current Emdebian packages for ARM but don't quite have room for a full Debian install either. Until I had at least a basic package set available via the cross-building system, I didn't want to start working on an intermediate system, for reasons of simplicity mainly, but also because I knew that getting a cross-building package set would involve changes to the tools needed for the intermediary set.

Bah, I'm waffling again. For the background, see Emdebian, Squeeze, Grip and Crush.

The good news is that Emdebian Grip, once running, will take far, far less time (for everyone) than the full-on cross-building malarkey of Emdebian Crush. Grip can be 99.9% automated by breaking Debian into discrete package sets or 'components'.
Componentising Debian for Emdebian
Emdebian Grip is about a smaller Debian, nothing more. No changes in functionality, no changes in dependencies; only changes in final installation size and Packages.gz size. It means using the code from dpkg-cross to unpack packages, remove unwanted files and repack the .deb. Yes, yes, this can be done one package at a time at a trivial level. Emdebian Grip seeks to bring a standard method for this operation that will work for all packages on any architecture for any architecture from any suite, creating and maintaining repositories of "gripped" packages for use in devices like my Acer Aspire1 where you just don't need documentation, manpages and the rest. Squeeze is the obvious target release but any release supported by the current version of dpkg could be "gripped" too, like oldstable.

The componentisation comes in at the level of the Packages.gz file. If the device cannot realistically be expected to run CAD software or has no fancy graphics or other power-hungry hardware, why should packages that need such resources clutter up the apt-cache list? This data isn't just slowing down searching for packages you do want, it slows down every installation process (that bit about 'Reading database . . . . . '), it slows down the process of getting the updated cache data in the first place because there is more of it, it wastes space on your device in several different places when the whole point of Emdebian is that disc space is not cheap and it wastes your precious battery power if you need to do simple package management stuff away from the mains. What is needed is a mini-repository, a mini-mirror that just contains the parts of Debian that your machine needs - a "well-trimmed" mirror with sources and all dependencies satisfied, a mirror that can be automatically updated but which doesn't keep getting NEW packages that you won't use. New dependencies will be added, yes. New packages unrelated to what you currently use? Absolutely not. You could even keep a mini-mirror on your home network, updating and "gripping" while you sleep, because a mini-mirror should be a lot, lot smaller than a standard Debian mirror.

I now have a script that will do the work, what I've lacked is the idea on how to "feed" packages to the Gripper.

So, I'm going to be exploring this method:

1. An arbitrary package set - say the output of 'dpkg -l' from the target machine setup by an expert, or 'dpkg --get-selections' or just the packages from a normal debootstrap - the basic idea is to start with only the packages you absolutely must have for your target device.


2. A very small wrapper for some debootstrap code that downloads those .debs from the Debian mirrors - debootstrap can cope with choosing the mirror, the architecture, the suite etc. (already written, landing in Emdebian SVN soon).


3. Parse the list of packages (.debs) and use 'apt-cache showsrc' to convert into a list of source packages. (Script done).


4. 'apt-get source' to match the source package to the binary. At this point, we have what amounts to a partial mirror but with very well defined limits. There are other ways of doing the same thing so anything that gets you to this point is as equal as the steps above, as long as the .debs exist alongside the .dsc and all associated source package files. What you need is full control over the list of packages at this stage, we'll worry about dependencies later.


5. Simple scripting parses the .dsc to create suitable debian/files content (done).


6. dpkg-genchanges regenerates the .changes file - critical for keeping the Gripped packages with the correct source packages in the final Grip repository


7. Now run 'emgrip' against the .changes file to convert the .debs, update the .changes file with the new, smaller, .deb details and copy everything into a suitable incoming/ directory.


8. point your repository software (tested with reprepro) at the incoming/ directory and generate your Gripped repository - which only contains the packages you listed at the start, just smaller. reprepro has a neat way of allowing foobar-1.2.3-2 to be included even if foobar-1.2.3-1 does not exist in the repository, as long as foobar_1.2.3.orig.tar.gz exists somewhere that reprepro can find and even though foobar-1.2.3-2.dsc does not mention the .orig.tar.gz. Neat! (Tested and working.)


9. Run 'edos-debcheck' against the repository and "catch-up" with whatever packages are needed to make the repository as a whole "trimmed" (no unresolved dependencies).


10. Keep the update script running as a cron task to pull updated packages from incoming.debian.org, run the .changes through emgrip and update your repository.


11. Clean out the .debs, leave the source packages and restart the process for another architecture. (Crafty!)


12. Possibly, to support those who can afford to do a small amount of compilation on their Emdebian Grip device, add a second repository that provides the Build-Depends that may be missing from the initial package set - stuff like automake and pkgconfig. Depending on how that works out, it might not be many packages and it may be easier to fold them into the mini-mirror anyway.


13. Instead of 20,000 packages, I'm expecting something closer to 2,000, per mini-mirror.


14. The process of "gripping" a package also drops numerous control fields (like Homepage and VCS*), as well as Recommends, Suggests and Enhances as well as truncating the long description at a maximum of four lines - all in the name of reduced package data sizes.


15. The end result is that each mini-mirror contains a component package set for Emdebian Grip, using Debian packages. Mini-mirrors are still "well-trimmed" (no unresolved dependencies) because you get to choose which packages you include and if a particular package tries to bring in both Gtk and KDE dependencies, you can choose to omit that package and use a different one.


16. Whether Emdebian Grip will or should support dropping certain binary packages from the repository is a difficult issue - Emdebian Crush is probably the preferred route if you need to start optimising an Emdebian Grip mini-mirror that harshly.


17. Another wrapper will be needed to clean out cruft (old dependencies, outdated packages etc.) - that one may be harder to achieve by automation.

The first Emdebian Grip repositories will be for ARM, armel and i386, based on an XFCE environment package set.

"Gripping" all the packages in my /var/cache/apt/archives (about 1Gb) took about 30 minutes on a fast laptop that was doing other things at the time - the real packages will be "gripped" on the Emdebian server and for multiple architectures in sequence. It's certainly conceivable that any number of mini-mirrors can exist with any number of package sets (components), all with "gripped" packages. You can choose to "grip" .debs upon download from a normal Debian mirror but that wastes time if there is more than one device, it wastes space in /var/lib/apt/lists and /var/lib/dpkg/ and wastes bandwidth. For portable devices that you want to be using, not updating, I think that a mini-mirror is a worthwhile bonus.

Expect a 20% reduction in installation size (for more, you'll need the cross-building, functionality-modifying Emdebian Crush shebang) but a large decrease in the size of all the apt-cache data and dpkg status data. I'm hoping to push towards 30% reductions but most of these figures are guesswork right now.

And beyond the mini-mirrors? Debian-Installer, of course. I really need to set some time aside to prepare an Emdebian Grip installer using current D-I. Simple things like:

• Preferring ext2 instead of ext3 for solid state file systems,


• preferring XFCE to GNOME for the Desktop System,


• turning off Recommends before even the first package gets chosen - let alone installed,
  
• supporting machine:variant customisation scripting so that particular devices can be configured from first boot instead of having to rely on defaults or manual fixes via Wiki pages.

It really, really annoys me that I have to spend ages getting devices like the Acer Aspire1 working properly under Debian and that I have to take particular care during the installer to change the Debian default to something more sane for that kind of device. For Debian to be The Universal Operating System, the installer needs to be as smooth on an NSLU2* as on a standard desktop, a fast server and a cluster. This isn't just an architecture thing, there are big and small devices in all architectures, it needs to be a machine:variant thing. Mini-mirrors of single components are a start and part of the bigger answer is to support machine-specific configuration packages in these mini-mirrors. Packages that only exist in a specific mini-mirror and are specifically designed to setup a host of machine-specific settings, blacklist certain modules and enable others, set sensible settings in /etc/fstab and other places based on results from things like powertop, turning off features for some devices and turning them on for others. The whole point of such packages is that what they do would be insane on what would currently be considered a "typical Debian machine".

*(I'm not talking about an NSLU2 with an external hard drive here, I'm talking about fitting Emdebian Crush into the 6.4Mb of space on the NSLU2 itself so that external hard drives can be hot-plugged, as well as devices like the OpenMoko and iPAQs etc.)

Don't get me wrong, D-I does a fantastic job for the kinds of devices Debian currently supports but there are a host of other devices out there that need customised solutions where a mini-mirror and a machine-specific configuration package can remove huge amounts of hassle from the actual installation process. Emdebian Crush already uses machine:variant specific packages and the current Emdebian packages repository is, itself, a mini-mirror, albeit not well trimmed.

Let's make the most of "Debian Squeeze" - let's squeeze it onto more devices and make it even easier to get Debian onto anything.

This is the point where I should be writing something witty and intelligent about the election going on across the pond, but I can't think of anything worth saying. If you can, go and vote. Even if you think your side is going to win, go and bloody do it anyway.

Anyway. Today I achieved FDMI success; my code successfully registered my test cluster against the fabric switch it's connected to and reported various bits of information about itself. Hardly earth shattering but does at least indicate that I'm beginning to understand some details of Fibre Channel at the packet level and how to make our code send/receive them. Go 'bama me.

(FDMI is the Fabric Device Management Interface and a way for Fibre Channel devices to report things like their manufacturer, serial number, OS, HBA model and driver/firmware versions to a management controller on the switch fabric. It's not a complex protocol in any manner, but there's a distinct lack of clear Fibre Channel documentation around that's at a lower level than "So you want to hook up a SAN?")

Over the weekend I finally got fed up with Evolution struggling to connect to work's "IMAP" server (Exchange 2007), and switched to using OfflineIMAP to sync the mail to a local Maildir. This as expected worked pretty well, and I'm now hidden from the nasty lag on the server. However, I've had to write my top secret Intel password into .offlineimaprc, which sucks. Then I had a cunning plan... GNOME Keyring will store passwords in a pretty secure manner, so somehow I need to fetch the password from there. A quick look at the OfflineIMAP manual revealed that I can write Python functions which return the password, so I should be abe to hook into the keyring from OfflineIMAP. This should be fairly simple:

import gobject, gnomekeyring
# The keyring needs to know the application name
if gobject.get_application_name() is None:
  gobject.set_application_name("offlineimap")
def keyring(user, host):
  keys = gnomekeyring.find_network_password_sync(user=user, server=host, protocol="imap")
  # First one will do nicely thanks
  return keys[0]["password"]
...
remotepasseval = keyring("rburton", "imapmail.intel.com")

After writing a small tool to add the key to the keyring, to my surprise this worked first time. I bounced with glee, but ten minutes later I had error messages from OfflineIMAP running from cron in my inbox... GNOME Keyring uses an environment variable to find the daemon, which isn't set in a cron environment. GNOME Keyring will fall back to using DBus to find the daemon, but the DBus session bus environment variable isn't set. DBus will fall back to reading the session bus address from the X root window, but DISPLAY isn't set so that doesn't work either... EPIC FAIL. But, I thought, I upgraded to Network Manager 0.7 last week which bought in ConsoleKit. If I ask ConsoleKit for my sessions I should be able to find a session with has an X connection, then I can set DISPLAY appropriately and then the chain described above will work, and I'll have my password. Shockingly, this worked first time too:

import dbus, os
if not os.getenv("DISPLAY"):
  # Get the ConsoleKit manager
  bus = dbus.SystemBus()
  manager_obj = bus.get_object('org.freedesktop.ConsoleKit', '/org/freedesktop/ConsoleKit/Manager')
  manager = dbus.Interface(manager_obj, 'org.freedesktop.ConsoleKit.Manager')
  
  # For each of my sessions..
  for ssid in manager.GetSessionsForUnixUser(os.getuid()):
    obj = bus.get_object('org.freedesktop.ConsoleKit', ssid)
    session = dbus.Interface(obj, 'org.freedesktop.ConsoleKit.Session')
    # Get the X11 display name
    dpy = session.GetX11Display()
    if dpy:
      # If we have a display, set the environment variable
      os.putenv("DISPLAY", dpy);
      break

(man, I really with python-dbus had a better syntax for getting objects with a specific interface) So there you go, integrating OfflineIMAP with the GNOME Keyring via ConsoleKit and DBus. Surprisingly this was pretty easy to do, thanks to DBus and the magic provided by ConsoleKit it is 100% hack free.

I have previously written about my work packaging the tools to manage Amazon EC2 [1]. First you need to login and create a certificate (you can upload your own certificate - but this is probably only beneficial if you have two EC2 accounts and want to use the same certificate for both). Download the X509 private key file (named pk-X.pem) and the public key (named cert-X.pem). My Debian package of the EC2 API tools will look for the key files in the ~/.ec2 and /etc/ec2 directories and will take the first one it finds by default. To override the certificate (when using my Debian package) or to just have it work when using the code without my package you set the variables EC2_PRIVATE_KEY and EC2_CERT. This Amazon page describes some of the basics of setting up the client software and RSA keys [2]. I will describe some of the most important things now: The command “ec2-add-keypair gsg-keypair > id_rsa-gsg-keypair” creates a new keypair for logging in to an EC2 instance. The public key goes to amazon and the private key can be used by any ssh client to login as root when you creat an instance. To create an instance with that key you use the “-k gsg-keypair” option, so it seems a requirement to use the same working directory for creating all instances. Note that gsg-keypair could be replaced by any other string, if you are doing something really serious with EC2 you might use one account to create instances that are run by different people with different keys. But for most people I think that a single key is all that is required. Strangely they don’t provide a way of getting access to the public key, you have to create an instance and then copy the /root/.ssh/authorized_keys file for that. This Amazon page describes how to set up sample images [3]. The first thing it describes is the command ec2-describe-images -o self -o amazon which gives a list of all images owned by yourself and all public images owned by Amazon. It’s fairly clear that Amazon doesn’t expect you to use their images. The i386 OS images that they have available are Fedora Core 4 (four configurations with two versions of each) and Fedora 8 (a single configuration with two versions) as well as three other demo images that don’t indicate the version. The AMD64 OS images that they have available are Fedora Core 6 and Fedora Core 8. Obviously if they wanted customers to use their own images (which seems like a really good idea to me) they would provide images of CentOS (or one of the other recompiles of RHEL) and Debian. I have written about why I think that this is a bad idea for security [4], please make sure that you don’t use the ancient Amazon images for anything other than testing! To test choose an i386 image from Amazon’s list, i386 is best for testing because it allows the cheapest instances (currently $0.10 per hour). Before launching an instance allow ssh access to it with the command “ec2-authorize default -p 22“. Note that this command permits access for the entire world. There are options to limit access to certain IP address ranges, but at this stage it’s best to focus on getting something working. Of course you don’t want to actually use your first attempt at creating an instance, I think that setting up an instance to run in a secure and reliable manner would require many attempts and tests. As all the storage of the instance is wiped when it terminates (as we aren’t using S3 yet) and you won’t have any secret data online security doesn’t need to be the highest priority. A sample command to run an instance is “ec2-run-instances ami-2b5fba42 -k gsg-keypair” where ami-2b5fba42 is a public Fedora 8 image available at this moment. This will give output similar to the following: RESERVATION r-281fc441 999999999999 default
INSTANCE i-0c999999 ami-2b5fba42 pending gsg-keypair 0 m1.small 2008-11-04T06:03:09+0000 us-east-1c aki-a71cf9ce ari-a51cf9cc The parameter after the word INSTANCE is the serial number of the instance. The command “ec2-describe-instances i-0c999999” will provide information on the instance, once it is running (which may be a few minutes after you request it) you will see output such as the following: RESERVATION r-281fc441 999999999999 default
INSTANCE i-0c999999 ami-2b5fba42 ec2-10-11-12-13.compute-1.amazonaws.com domU-12-34-56-78-9a-bc.compute-1.internal running gsg-keypair 0 m1.small 2008-11-04T06:03:09+0000 us-east-1c aki-a71cf9ce ari-a51cf9cc The command “ssh -i id_rsa-gsg-keypair root@ec2-10-11-12-13.compute-1.amazonaws.com” will then grant you root access. The part of the name such as 10-11-12-13 is the public IP address. Naturally you won’t see 10.11.12.13, it will instead be public addresses in the Amazon range - I replaced the addresses to avoid driving bots to their site. The name domU-12-34-56-78-9a-bc.compute-1.internal is listed in Amazon’s internal DNS and returns the private IP address (in the 10.0.0.0/8 range) which is used for the instance. The instance has no public IP address, all connections (both inbound and outbound) run through some sort of NAT. This shouldn’t be a problem for HTTP, SMTP, and most protocols that are suitable for running on such a service. But for FTP or UDP based services it might be a problem. The part of the name such as12-34-56-78-9a-bc is the MAC address of the eth0 device. To halt a service you can run shutdown or halt as root in the instance, or run the ec2-terminate-instances command and give it the instance ID that you want to terminate. It seems to me that the best way of terminating an instance would be to run a script that produces a summary of whatver the instance did (you might not want to preserve all the log data, but some summary information would be useful), and give all operations that are in progress time to stop before running halt. A script could run on the management system to launch such an orderly shutdown script on the instance and then uses ec2-terminate-instances if the instance does not terminate quickly enough. In the near future I will document many aspects of using EC2. This will include dynamic configuration of the host, dynamic DNS, and S3 storage among other things.

• [1] http://etbe.coker.com.au/2008/10/04/getting-started-with-amazon-ec2/


• [2] http://docs.amazonwebservices.com/AmazonEC2/gsg/2006-06-26/running-an-instance.html


• [3] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1345&categoryID=100


• [4] http://etbe.coker.com.au/2008/10/13/ec2-security/

Share This

Every Democrat was pretty sure Al Gore was going to win in 2000. Most polls said John Kerry was going to win in 2004. And yet both times they managed to snatch defeat from the jaws of victory. Just remember how you felt then and try your best so it doesn’t happen again. There is so much at stake … the US and the world can’t afford 4 more years of a Republican US President. Even if the GOP seems to be defeated, shattered and panicking, Republican voters always end up voting for their guy on election day, while Democrats have great problems closing the deal. It has to be different this time. We need Obama to win tomorrow, and we need him to win big. And a Democratic wave to get a filibuster-proof Senate majority and a comfortable victory in the House of Representatives, so they can enact a progressive agenda in the upcoming years and restore some of the damage from the last two Bush terms. We also need a solid ‘no’ to Proposition 8 in California and similar ones in other states, so no excuse to stay at your place just because you are in one of the bluest states. I would like to ask you to do your best tomorrow, but probably Larry Lessig does a better job at it. Once we are past this, the hard work starts, trying to hold every elected representative accountable, draft better candidates for future elections, spot DINOs, …, but for now, let’s focus on tomorrow. I am hoping for the best, and I wish I could do more about it, but it’s up to you Americans. Do the right thing and leave everything on the road, you don’t want to regret it tomorrow.

I’ve always been up a bit skeptical of AI and here is one piece of AI software, Gender Analyzer that failed to predict my gender (than again I do write about the geek world). Thanks to Joe for pointing this out to me!

I’ve always been up a bit skeptical of AI and here is one piece of AI software, Gender Analyzer that failed to predict my gender (than again I do write about the geek world). Thanks to Joe for pointing this out to me!

From the “Software that recently rocked my pants” department:

• Bitlbee: talk to your Jabber/MSN/Yahoo contacs with your IRC client. I was aware of its existance, never found the time to start using it. Shame on me! I run it on my laptop rather than on a server.
• Matplotlib: plotting library for Python, and I find the results very pretty. Looks quite powerful, though I had slight problems figuring out how to do certain stuff.
• Schroot with LVM: brilliant, particularly if you make use of sessions.
• KVM with libvirt and virt-manager: seems to work well, almost as easy as VirtualBox, and (tada!) thanks to VNC you get detaching for your guests (no more Windows reboots if you restart your Xserver).

I also thank Martin F. Krafft for helpfully updating the encryption howto I followed for my previous laptop to mention that using a single encrypted device and LVM on top of it is (possibly) preferred nowadays. And to Simon McVittie for his idea of a small unencrypted Debian installation for recovery purposes in laptops without an optical drive. P.S.: I can’t believe this shit about blogging every day in November.

I know phpMyAdmin translators will love us for this, but I just commited translatable messages for new setup script. This leads to situation when even the best translations are at 75% (see statistics). I hope our brave translators will manage to handle these 350 new messages for upcoming 3.1 release, at least for major languages, which were always close to 100%. PS: Looking for translator to Czech!

The term Cloud Computing seems to be poorly defined at the moment, as an example the Wikipedia page about it is rather incoherent [1]. The one area in which all definitions of the term agree is that a service is provided by a varying number of servers of which the details are unknown to the people who use the services - including the people who program them. There seem to be four main definitions of cloud computing. The first one is Distributed Computing [2] (multiple machines in different administrative domains being used to perform a single task or several tasks), this definition does not seem to be well accepted and will probably disappear. The next one is Software As A Service [3] which is usually based on the concept of outsourcing the ownership and management of some software and accessing it over the Internet. An example is the companies that offer outsourced management of mail servers that are compatible with MS Exchange (a notoriously difficult system to manage). For an annual fee per email address you can have someone else run an Exchange compatible mail server which can talk to Blackberries etc. The main benefit of SAAS is that it saves the risk and expense of managing the software, having the software deployed at a central location is merely a way of providing further cost reductions (some companies that offer SAAS services will also install the software on your servers at greater expense). The last two definitions are the ones that I consider the most interesting, that is virtual machines (also known as Cloud Infrastructure [4]) and virtual hosting of applications (also known as Cloud Platforms [5]). The Amazon EC2 (Elastic Cloud Computing) service [6] seems to be the leading virtual machine cloud service at the moment. Google App Engine [7] seems to be the leading cloud application hosting service at the moment. It is also claimed that “Cloud Storage” is part of cloud computing. It seems to me that storing data on servers all over the world is something that was done long before the term “Cloud Computing” was invented, so I don’t think it’s deserving of the term.

• [1] http://en.wikipedia.org/wiki/Cloud_computing


• [2] http://en.wikipedia.org/wiki/Distributed_computing


• [3] http://en.wikipedia.org/wiki/Software_as_a_service


• [4] http://en.wikipedia.org/wiki/Category:Cloud_infrastructure


• [5] http://en.wikipedia.org/wiki/Category:Cloud_platforms


• [6] http://etbe.coker.com.au/2008/10/04/getting-started-with-amazon-ec2/


• [7] http://en.wikipedia.org/wiki/Google_App_Engine

Share This