I was asked for advice on how to start learning Linux. Rather than replying via email I m writing a blog post for future people who ask such questions and also to get comments from other people which may provide information I missed. Join a LUG The best thing to do is to start by joining your local Linux Users Group (LUG). Linux International maintains a list of LUGs that is reasonably comprehensive [1]. Even if there isn t a LUG near enough for you to attend meetings you can learn a lot from a mailing list of a LUG that s close to your region. There is usually no great reason not to join the mailing list of a LUG in a different region or country, but a local LUG is that the advice will often be tailored to issues such as the local prices of hardware and the practices of your government. Also note that Linux International doesn t list all LUGs, the MLUG group in Melbourne [2] and the BLUG group in Ballarat [3] aren t listed. Anyone who joins LUG (the group based in Melbourne, Victoria that I m a member of) will be advised of the smaller groups in the region if they ask on the list. As an aside it would probably make sense for the main LUV web page [4] to have links to local LUGs and to the LI page of users groups and for other LUGs to do the same. It s pretty common for a Google search to turn up the web site of a LUG that s near the ideal location but not quite right. Also it would be good if LUV could have a link to the Victorian Linux Users Group in Canada this should reduce the confusion a bit and they have a link to us [5]. Play with Linux Get a spare PC (with no important data) and try installing different distributions of Linux on it. Make sure that it never has anything particularly important so you can freely try things out without worrying about the risk of losing data. Part of the learning process usually involves breaking a system so badly that it needs to be reinstalled. Linux can run on really old hardware, an old system with 64M of RAM will do for learning (but 128M will really be preferred and 256M will be even better). Learn with other Beginners LUV has a very active beginners group, with a beginners mailing list and special beginners meetings. A group that has such things will be more helpful as you can easily learn from other people who are at a similar level to you. Also you can spend time learning Linux with friends, just spend a weekend with some friends who want to learn Linux and play with things you can often learn more by trying things than by reading books etc. Do some Programming One of the advantages of Linux (and other Free Software OSs) is that it comes with a full range of programming languages for free. You can get a much greater understanding of an OS by writing programs for it and a typical Linux distribution gives you all the tools you need. Any other Ideas? Does anyone have any other suggestions? Please leave a comment.

• [1] http://www.linux.org/groups/
• [2] http://www.mlug.org.au/
• [3] http://www.blug.asn.au/
• [4] http://luv.asn.au/
• [5] http://www.vlug.org/linux/lugs.html

dpkg-source is the program that generates the Debian source package when a new package version is built. It offers many interesting command-line options but they are often not used because people don t know how to ensure that they are used every time the package is built. Let s fill that gap! It is possible to forward some options to dpkg-source by typing them on the dpkg-buildpackage command line but you d have to remember to type them every time. You could create a shell alias to avoid typing them but then you can t have different options for different packages. Not very practical. The proper solution has been implemented last year (in dpkg 1.15.5). It is now possible to put options in debian/source/options. Any long option (those starting with -- ) can be put in that file, one option per line with the leading -- stripped. Here s an example:

# Bzip2 compression for debian.tar
compression = "bzip2"
compression-level = 7
# Do not generate diff for changes in config.(sub guess)
extend-diff-ignore = "(^ /)config.(sub guess)$"

Notice that spaces around the equal sign are possible contrary on the command line. You can use quotes around the value but it s not required. The debian/source/options file is part of the source package so if someone else grabs the resulting source package and rebuilds everything, they will use the options that you defined in that file. You can also use debian/source/local-options but this time the file will not be included in the resulting source package. This is interesting for options that you want to use when you build from the VCS (Version Control Repository, aka git/svn/bzr/etc.) but that people downloading the resulting source package should not have. Some options (like --unapply-patches) are only allowed in that file to ensure a consistent experience for users of source packages. You can learn more about the existing options in the dpkg-source manual page. Read it, I m sure you ll learn something. Did you know that you can tell dpkg-source to abort if you have upstream changes not managed by an existing patch in debian/patches? It s --abort-on-uptream-changes and it s only allowed in debian/source/local-options. Be sure to subscribe to the RSS feed or to the email newsletter to not miss useful documentation for Debian contributors! 3 comments Support my work

Hello Following a discussion on #debian-perl IRC channel, I ve proposed to provide a script to parse DEP-5 files. The goal is to be able to parse *and* validate DEP-5 files. DEP-5 is a
proposal to make debian/copyright machine-interpretable. This proposal is driven by Lars Wirzenius. With Config::Model, any evolution of DEP-5 specification will
be easy to include in the DEP-5 model read by Config::Model. What DEP-5 model ? To keep a long story short, let s say that DEP-5 model is a description of DEP-5 syntax and semantic that can be used by Config::Model to perform validation. For more detail on how to create a model, please read this doc. In other word, DEP-5 model is DEP-5 document translated into a special format. First step was to directly edit the doc and munge it into a YAML document describing the structure of DEP-5. Here s a small extract of this YAML file (slightly edited to remove most long descriptions) :

 
---
class:
  Debian::Dep5:
    class_description: >
       Machine-readable debian/copyright
    accept:
      name_match: X-.*
      type: leaf
      value_type: string
    element:
      Format-Specification:
        mandatory: 1
        type: leaf
        value_type: uniline
        description: >-
          URI of the format specification, such as ...
      Name:
        type: leaf
        value_type: uniline
      Files:
        type: hash
        index_type: string
        ordered: 1
        cargo:
          type: node
          config_class_name: Debian::Dep5::Content

During this YAML file creation step, the problem raised by the License keyword became obvious because of License properties:

• Limited number of valid License (no problem, let s use an enum)
• License names are not case sensitive (optimism goes somewhat down)
• License names have version number and an optional + suffix (ok, let s use a regular expression with Config::Model s brand new match specification)
• License can be combined with and or or . (uh oh, the match regexp will not be enough. A grammar would be better.)
• License can specify an abbreviation or the full text of the license.

Long story short, I had to add to Config::Model the possibility to specify a Parse::RecDescent grammar to validate a value. More on this later. Of course, the first draft of a model in YAML was far from being perfect. So the second step was to load it with config-edit-model. I had to fix a number of YAML errors and then some errors in the model description. Then, I had to write a parser to load the DEP-5 data into Config::Model tree. I first used Rapha l Hertzog s Dpkg::Control::Hash module. But this one is not able to cope with repeated fields without clobbering them. So I had to provide my own parser. The parser is divided in 2 parts:
- the parse function to load DEP-5 data in a simple data structure
- the read function to load the simple data structure into Config::Model s configuration tree You can view the code on Config::Model repository. Then the model is divided in 3 configuration classes:

• Debian::Dep5: the root class
• Debian::Dep5::Content: To represent Files specification
• Debian::Dep5::License

And the full model of Dep-5 can be also read on Config::Model repository:

• Dep5.pl.
• Dep5::Content
• Dep5::License

(The models are biggish because they include help text taken from DEP-5 documentation) Now some explanation is required on how is performed the License validation. The trick is that each License used must be listed in Dep5 s license parameter which is specified this way (in YAML syntax):

 
License:
  type: hash
  index_type: string
  allow_keys_matching: '^(?i:Apache Artistic BSD etc... other)[\d\.\-]*\+?$'
  cargo:
     type: leaf
     value_type: string

So this License element contains the list of licenses with their full text and only Licenses whose name matches the regular expression are accepted. Now, let s explain how the files are tied to the declaredlicenses. The Debian::Dep5::Content class had a License element that represent the relation between the files and the License(s). This bond is represented by Debian::Dep5::License configuration class. The abbrev and full_license elements are fairly obvious. The abbrev element is another matter. Here s its declaration (minus the description and help):

 abbrev:
 type: leaf
 value_type: uniline
 default: other

So far, so good. Now the meaty part: the validation requirement based on Parse::RecDescent

grammar: "license (oper license)(s?)
          oper: 'and'   'or'
          license: /[\\w\\-\\.\\+]+/i\n
               # PRD action to check if the license text is provided
               $return = $arg[0]->grab('! License')->defined($item[0]);
               "

This grammar specifies:

• The syntax of the License line itself (hence something like Perl or GPL )
• An action performed when the grammar is matched. Using Config::Model API, this Perl snippet checks that the License abbreviation has a corresponding License declared in Debian::Dep5 License hash

This way, an abbreviation cannot be used without a proper License statement. This code is the first stab. Some more work is to be done soon:

• Implement exception parsing
• Provided DEP-5 writer
• Provide config-edit-dep5 cli

Anyway, I ll soon release Config::Model with this first version of DEP-5 parser. As always feedback are more than welcome. All the best update: added missing Parse::RecDescent link

[Nothing related to Debian here, move on if you don't care about my recurring obsession with the end of the world, Cassandra pains, or Big Phat Living Room Revolution yearnings].

Although I am still looking for a community of anarcho-primitivists that want to accept me joining their frugal living, and simply slip off the social construct, the system, the fucking face of earth, call it what you will, I could find some comfort in my favourite feminist blogger's take on the subject of my midlife crisis regarding ecology, capitalism, patriarchy, child bearing, overpopulation, meat consumption and all the rumination of consuming thoughts that (still) make my life miserable and have kept me running away from human contact for a year.

[Emphasis mine]

At what point does human culture depart from the Natural? With the invention of computers? TV? Cars? The cotton gin? Electricity? Taco stands? Gunpowder? The printing press? Written language? Shoes? Crop cultivation? Yurts? The wheel? Did humans become unnatural when the good old days of picturesque, endless agrarian toil, feudal oppression, unchecked disease, ignorance, and death from dysentery at 35 turned into the bad new days of urban post-industrial capitalism where a pound of fair-trade organic coffee costs $12.99 and your email inbox is full of spam?

ah. Everything humans do, or have ever done, is natural. We can t do anything else. The idea that modern culture is un-natural is nostalgic and inaccurate. Living off the grid in a yurt is good in some absolute sense, whereas driving an SUV from a suburban bungalow to the stripmall is bad ? Come on. This a romantic, but misguided view. The cosmic reaction to a 20 Ford Expedition is the same as to a sanctimonious Prius: bupkis. The universe doesn t give a fuck about you or your lifestyle choices. It doesn t give a fuck about the economy, oil spills, or civil unrest in Blargistan. It doesn t give a fuck about katydids. Eventually our whole planet will be erased from space, and the galaxy won t bat an eye. The inevitable extinction of our species (imminent, according to research here at Spinster Laboratories) via the exhaustion of available resources is as natural as a fresh-picked peach. As Andre 3000 and other dude philosophers have observed, nothing is forever.

Yes, yes, when people use the word natural what they really mean is free of chemical additives and maybe some of the assorted hippie concepts that go with that narrative. Barter economies, home furnishings made from bamboo, vegan cookbooks, living in the country. While I would argue that it is just as natural for people to put chemical additives in things as it is to not put chemical additives in things, I admit that it is appealing to fantasize that the source of human misery is an unnatural isolation from Nature, and that doing yoga on an organic rubber mat and drinking organic spinach smoothies will put me back in sync with the cosmos.

But alas, I m already in sync with the cosmos, and so are you. In other words, this is it. This is what we ve become, and this is what we get. Which is not to say that a person can t fantasize about a verdant paradise full of songbirds and polar bears and Bengal tigers, untouched by human influence. Only, that world isn t a world we could actually live in. The minute you add contented children, lazy from a carefree day at the swimming hole, eating ripe plums on the porch at sunset to that scenario, natural history changes, and it s right back to our scorched-earth dystopia. Our giant brains use up resources, it s as simple as that.

As long as we re still here, though, we might as well try to make the best of it.

http://blog.iblamethepatriarchy.com/2010/06/19/heads-up-art-week-starts-on-monday/

Though the comfort is somewhat ephemeral, because deep inside, I know she's wrong, I know another way is feasible, but I have given up already, I have very well come to terms with extiction, I get an enormous pleasure from the thought of the very well deserved human mass suicide-by-cop^Wnature, and in fact, I trust and hope it will save me personally from facing the hard times ahead, as I am a damn coward, paralized by fear, hopeless about the flock ever getting it (is it so hard, really?), while doing as much as I can but completely overwhelmed by the need of changing a world everyone seems to like just as it is. I can certainly empathize with much more disturbed ways of airing similar anguish, among many other examples, such as The Unabomber, or The Weather Underground. I don't even want to go there as I know violence all too well and it is not fun to be on the receiving end. But I see the screaming desperation behind the motives of acts I am not at the liberty to discuss here. I wonder if consumption of resources in the form of books will numb the disorder or make it worse and what sort of medical arrangement the stablishment has for it. As you click away the previous links will mean nothing to you because a)it is Wikipedia after all, and b)you probably don't even understand what the problem of eating animal corpses is, humans are omnivores after all, and what is all this nonsense about dropping language altogether.

WTF, there was no comfort after all, and I am still a nutjob.

At some point in the past my bookmarks broke in such a way that I wasn't able anymore to alter in any way the bookmarks, deletions, additions, reorganisation, nothing worked.

Way before this event I set up a git repo in my .mozilla directory and committed everything in the directory (except obvious cache files and such) in the repo. I didn't make a cronjob with this since I didn't thought of a proper way to make sure the browser wasn't started (the data might have been inconsistent at commit time), and, since most of the time I have the browser started, it seemed rather useless to try to automate for a very, very small window of time when the browser data would be in a consistent state. As a consequence, I did occasional manual commits in the git repo.

When the bookmarks issue appeared I realised that my last git commit was so old that it didn't made any sense to try to restore since I would have had lost bookmarks, passwords and plugins. So I made a checkpoint commit os the broken profile and resorted to all sorts of manual attempts at fixing the issue, but all my efforts of playing with the bookmarks.* files proved to be useless.

I concluded that icewesel/firefox 3.5.x was using some other mechanism and decided to dig into the issue at some latter time when time allowed me to analyse and fix it. Today was that day.

I began by creating a backup for the bookmarks via the Bookmarks Manager export function. After a hickup, it managed to save a proper bookmarks.html file. I tried to import that file, but the breakage remained.

Then I decided I should prepare for the worst, installed an addon which allowed me to save all stored passwords and created a new account (started firefox from the command line with 'firefox -ProfileManager') with the intention to transplant files from the old files into the new one. Short after that I realised that it would be smarter to commit the new profile, too, import the bookmarks from the backup file and replace the touched files back into the old profile.

After the import and closing the browser, git informed me that the modified files were:

XPC.mfasl
XUL.mfasl
cookies.sqlite
localstore.rdf
places.sqlite
pluginreg.dat
urlclassifier3.sqlite

At this point the modification of the binary .mfasl files looked very ugly to me, but I had a hunch that the only relevant file to my problem was the places.sqlite file and I went on to check the contents of the file with sqlitebrowser. After inspecting also the urlclassifier3.sqlite file (whose contents were unreadable), I decided to give it a try at simply placing the places.sqlite file in the old profile and see if the problem would be fixed that way. After all, I had the whole profile's state in git so any breakage could have been undone easily.

I started firefox using the old profile and the problem was fixed. Yay!

Now I have my bookmarks back and a suggestion for the mozilla people: please don't keep useless files like bookmarks.html in the profile once they become outdated, they are misleading.

P.S.: I still have to think of a way to make automated sane commits. Maybe I'll try to make an iceweasel/firefox addon. This could be a nice idea since I could learn something new, too.

Second Sunday in September -- time for the annual Chicago Half Marathon now in its fourteenth edition (and I have been running it in 2003, 2004, 2005, 2006, 2007, 2008 and 2009 making this and the JPM Chase Corporate Challenge the races I've run most often). And the course was altered this year, alongside with an earlier start as the Bears have their season home opener today. So we started north towards 57th, then running down to 67th, turning east towards the lake at around three miles --- and having the remaining ten miles along the lakefront running up to 31st (the usual turn) and back down to 63rd. I like this course better; let's hope it sticks. Race conditions were fantastic. We had a rainy and gray day yesterday but today is pure bliss. Temperatures around 60 degrees at the 7:00am start, no wind, sunshine and not a cloud in the sky. The race itself went well. I had a pretty brutal running year suffering most of the time from some archilles tendon inflammation. It has gotten better in the last few weeks possibly thanks to some heel cups I now put in the shoes. But I had exactly one run longer than ten miles since the Boston Marathon. So I lost a lot of speed, as well as endurance and was a little nervous as to how I'd do. And considering all this, it went pretty well. I fininished in 1;41:50 or a 7:47 pace. While is easily the slowest half in a number of years, at least I got to run it evenly, pain free and with a negative split (== faster second half) and some gas left for a fast last half mile or so. So maybe I don't have to retire from running just yet. We'll see if I get some speed back in 2011.

A lot is being said on how Ruby is packaged in Debian and Ubuntu. In this post, I m trying to go through the most common myths and give my position as a Debian Ruby maintainer. Note that those are my views, not necessarily those of the other Debian Ruby maintainers. Myth: Ruby is completely outdated in Debian and Ubuntu There s a culture in the Ruby community of always using the latest bleeding-edge version. There s also a culture (though this one has recessed a bit, I think) that it s perfectly fine to change APIs in incompatible ways if it makes the software slightly better. In Debian (and Ubuntu) stable releases, we try to provide a rock-solid version of Ruby that won t be allowed to change API during the support lifetime of a release. Bugfixes for important bugs can be applied, but that s basically all. So it s not surprising that Ruby in released versions of Debian and Ubuntu is behind the latest upstream version. In development branches, we are following the latest upstream versions quite closely. Debian 6.0 squeeze , to be released by the end of the year, includes Ruby 1.8.7-p302, and will include 1.9.2-p0 (the currently of Ruby 1.9.2 in squeeze is currenty a SVN snapshot from beginning of august). There are people interested in installing a packaged Ruby that just works, and other people interested in installing their own version of Ruby (through RVM and such). People in the Ruby community tend to forget about the first category, and consider that, because they install Ruby from source or with RVM, that should be the solution for everybody. We need the Ruby community to respect that different people have different ways of using Ruby, and to stop recommending that everyone using Debian and Ubuntu should install Ruby from source. We also need to leverage the backports services for both Debian and Ubuntu to provide a way to install the latest upstream version on stable releases. The only reason why we are not doing it yet is lack of time. Myth: Ruby is split into a myriad of packages in Debian In the past (pre-2005), a decision was made to split some libraries from the standard library into separate packages. This was motivated by:

• the size of some libraries, like REXML
• the fact that some libraries, like OpenSSL, readline, gdbm and tk require other library packages to be installed
• licensing concerns about shipping some code (a .so) linked with OpenSSL, and some other code (another .so) linked with readline, in the same package.

Most of the split was undone a very long time ago, and the readline and openssl splits were undone this year. The current package in Debian squeeze has:

• ruby1.8: binaries (ruby itself, irb, rdoc)
• libruby1.8: the standard library, except ruby-tk
• libtcltk-ruby1.8: ruby-tk (merging it with libruby1.8 would add several megabytes)
• ri1.8: contains both the ri binary and documentation
• ruby1.8-dev: development headers (to build applications or gems linked against the Ruby C API), and statically built version of the library
• ruby1.8-elisp: Ruby emacs mode
• ruby1.8-examples: Examples
• libruby1.8-dbg: debug symbols

It is always possible to merge more packages, but merging any of the other packages would significantly increase the size of what you get by default, and might discourage people to use it in some contexts (disk space is not always cheap. Think embedded systems and smartphones). Myth: Rubygems is crippled in Debian There are two differences in the rubygems Debian packages. First, gems are installed in /var/lib/gems, not /usr/local/lib/gems. That is because rubygems are seen as state for the Rubygems package, not independant software installed separately. A change for this is currently being discussed, and should be done after the squeeze release. (I don t think that it s reasonable to push that change now). A consequence of installing to /var/lib/gems is that gems binary are outside of $PATH. After the change, gems binary would also be installed in /usr/local/bin. Second, gem update --sytem is disabled in the Debian package, because that would replace code from the Rubygems Debian package with code downloaded from the Internet, and can break your system in subtle ways. We have changed this (in SVN, not in any package in the archive yet) to allow gem update --system if a specific environment variable is set, which gives us the opportunity to display a warning message. Myth: Ruby is slow on Debian Ruby is built with the pthread threading code, because not using it breaks many native Ruby libraries (Ruby-gnome2, for example). And other Linux distributions are doing the same. There are some very old benchmark results floating around in some blogs that show that the pthread code has a performance impact of about 20%. There are been patches for that for a long time (applied in Ruby Enterprise Edition), but they were never properly sent back. Actually, all the discussions about this issue were rumors and blog posts, and there wasn t even a bug report about it! After the bug was reported properly, it was fixed in a timely manner. The fix is not yet included in the Ruby 1.8.7, but is being applied as a patch to the Debian package since 1.8.7.299-2. (So the Debian package is actually newer and faster than upstream Ruby in that regard). What makes Ruby so difficult to maintain in Debian? The main problems we have with Ruby are portability issues. The only officially supported platform for Ruby is IA32 (i386 in Debian), and everything else is best effort. It means that it is not even supported officially on x86_64/amd64! Debian supports a dozen of architectures, and the Ruby package has to work on each of them. Ruby 1.8 is mostly OK nowadays thanks to the help of Debian porters, but we still have serious problems with 1.9.2, since the test suite fails on sparc, hppa, ia64 and Debian GNU/kFreeBSD. Why doesn t Ruby use the alternatives system on Debian? The alternatives system provides a way to switch between different software providing the same functionality. Currently, if you want to use Ruby 1.9.2, you need to use the ruby1.9.1 executable (1.9.1 = ruby compatibility version here). The alternatives system would allow to switch ruby to ruby1.9.1 on a system-wide basis. However, the situation of applications and libraries working only with Ruby 1.8 should be clarified: those should use /usr/bin/ruby1.8 instead of /usr/bin/ruby. Another problem is that the way to specify packages dependencies still need to be determined. There s nothing impossible to do about that, it just needs someone to do the work. Where do we go from here? The Ruby developers (or expert users) community is generally very hostile towards Debian. Many harsh words and insults are used in discussions mentioning Debian in the ruby-core@ mailing list, and there are frequent recommendations to avoid the Debian packages and install from sources, which is quite demotivating (Actually, what prompted that blog post is someone calling Debian-specific changes unforgivable). This atmosphere makes it hard to recruit people, and the Debian Ruby teams are completely understaffed, which is clearly the major blocker to improving the situation further. We are 3 co-maintainers for the interpreter packages, and I m the only Debian Developer that is really active on a regular basis in the pkg-ruby-extras team (that does libraries and applications packaging). We desperately need help, but at the same time I have no time to improve our documentation and make it easier to join the team. While I haven t reached a final decision yet, I am also likely to stop doing Ruby work in Debian after squeeze is released, as I am getting increasingly tired and demotivated.

In the spur of the moment, I cycled over to Dominican, one of the two small colleges in town, to see if I could snag a remaining ticket to see Billy Bragg perform. Turned out I could, and it became a nice evening out. Darren Hanlon started up the evening as the opener for a good half hour, and was quite decent; somewhat charming in a good natured way, not taking himself too too seriously. I'd gladlt see him again. After a longer-than-needed break Billy Bragg came on stage and played for two straight hours, alternating between an electric and acoustic guitar. And also alternating between some newer material and (especially towards the end and the encore) some old crowd-pleasure. I don't know his material all that well but have of course know of his career over these last 25 years and am quite glad I went to see him. Nice way to end the week.

The unofficial rc bug tracker currently knows about the following bugs:

In Total:	420
Affecting Squeeze:	301
Squeeze only:	66
Remaining to be fixed in Squeeze:	235

Of these 235 bugs, the following tags are set:

Pending in Squeeze:	16
Patched in Squeeze:	53
Duplicates in Squeeze:	26
Contrib or non-free in Squeeze:	10
Claimed in Squeeze:	2
Delayed in Squeeze:	8
Can fixed in a security Update:	15
Otherwise fixed in Squeeze:	19

Ignoring all the above (multiple tags possible), 133 bugs need to be fixed by Debian Contributers to get Debian 6.0 Squeeze released. However, with the view of the Release Managers, 254 need to be dealt with for the release to happen. Please see my older post for an explanation of the different numbers.

Watching the Star Wars Trilogies (both old and new) has finally closed one of the major (in at least some of my collegues' view) holes in my cinematographic education. (I've seen some of the new trilogy when it came out, but not all.) Oh well ... while it's quite enjoyable to watch, it's probably at least as important to have seen them just to spot the references all over the place. Repo Man was where I found myself literally laughing out loudly all by myself. And not only because of the vintage special effects (1984). The Straight Story (Lynch) is just beautiful. Alvin Straight is your proverbial old uncle who may be a bit funny in the head but you just have to like him... If you like mind control stories, The Manchurian Candidate is for you. Now that would be a way for Debian to achieve world domination. Worth thinking about? Speaking of american politics, this reminds me of All the President's Men, which I've seen (and liked) a long time ago. I don't know enough about history to tell how close it is to reality, but it's at least a well made political thriller. Based On A True Story is the term usually used, isn't it?

omlet (or here) is a vim extension for writing OCaml code. In my opinion, it has a better indentation than the standard OCaml vim support. Unfortunately, it has a cost: the indentation vim code is more complex. And it has a few bugs :-( The main bug is that it doesn't like unbalanced comment opening "(*" and closing "*)" tags. From time to time, it enters an infinite (or very long) loop when there such a tag left in your file. It can be very far away from the point you are editing. It isn't too problematic, because unbalanced tags are a syntax error. But the problem is that it matches these tags inside strings also. So whenever you start using regular expression like "(.*)" the whole indentation fails. But there is a very ugly solution to this problem! Problematic code:

 let parse_rgxp =
   Pcre.regexp ~flags:[ CASELESS] 
     "^(?<license>[A-Z0-9\\-]*[A-Z0-9]+)\
      (?<version>-[0-9\\.]+)?(?<later>\\+)?\
      ( *with *(?<exception>.*) *exception)?$"

Solution, add ignore"(*":

 let parse_rgxp =
   ignore "(*";
   Pcre.regexp ~flags:[ CASELESS] 
     "^(?<license>[A-Z0-9\\-]*[A-Z0-9]+)\
      (?<version>-[0-9\\.]+)?(?<later>\\+)?\
      ( *with *(?<exception>.*) *exception)?$"

Very very ugly coder: you balance comment tags in dead code -- very very bad ;-) ps: another solution when the plugin enters the infinite loop, hit Ctrl-C. This will stop it and let you define your own indentation.

It has been a pretty exhausting time since my last blog post in my area of Debian land . If you followed the news you should have seen some of it. Last weekend there was the point release. For those I have the (good?) position to sit in the middle of the working queue, where the hardest part (sorting out which stuff to include, getting it all build, etc) is already done. Still, due to the amount of steps neccessary to take its usually a multi-hour thing. This one release was even slower, as we had to wait for hours until the package lists had all been built - unfortunately the thing writing that keeps berkeley db files of data around as cache - and uses the FULL path to the files as key. How stupidly annoying when the path changes during a server move. Ohwell. The other big part that got a tiny bit of press coverage was the move from www.backports.org over to backports.debian.org. This was actually waiting for me for a good longish while already (err, well, since well before DC9 this was decided to happen) but now I finally got myself moved. This turned out to be much more work than I ever feared (which is part of the reason why it took so long), but some numbers:

• At the start the code changes between the revision running on oldbpo compared to HEAD: 303 files changed, 21227 insertions(+), 468357 deletions(-) (There had been 1723 commits in the master branch since the last time bpo got it, thats about a quarter of all we have in master!)
• Database schema version oldbpo: 5, HEAD: 35
• sub-sub-sub-optimal damn old shell scripts for the cronjobs.

The db update alone took me a week to figure out which steps to do in which order. Good to have an automated system, if you the manage to have the updates run specific to the main archive. Ups. When I was done with the database (during which I managed to just erase all work done by formorer and rhonda on the webpage, ups^2) I had some code and config cleanups to do, and especially manage to get it all running with a new path layout. Much less complicated, but enough work to do, before I finally was far enough to get it all move so far that formorer could start testing it and we could turn off the old system. At this time I had to fix up our shell scripts for our cron jobs, but that (with one exception) was a nearly straight forward easy task. And on Wednesday I managed another goal - backports is fully running the master branch of dak, the exact same code as on ftp-master. The cronjobs are slightly different, as well as the config (of course, duh), but otherwise its the same. My current goal and the last things left on this move todo is to change the cronjobs so that debian as well as backports do run the SAME scripts, but their configs list which tools should be run when. Considering that my dinstall cron already contains something pretty much a state machine I think I m insane enough to get this in shell, but when I told that to Mark (you know, the second ftpmaster :) ), I heard something very much like vomiting (you know, 700km between us, this might mean something) and then the plea to use a real language for the task. Oh my. I like my shell. :)

---

There is one big black cloud on the horizon to see. I have to do something similar for security. They are at a dak (and database) version pretty similar to what bpo had. They do need to support this obnoxious spare new format too before they can support security for squeeze, so there is no way I can ignore that archive for much longer. I already asked DSA for a vm in which I can play (not good to interrupt security archive for long), but probably wont start on this before the upcoming FTPMaster meeting. Pretty busy otherwise. Wellwell, I hope it will be less painful, as I now know the steps already. :)

Backlight control is one of those things that you'd think would be simple, but ha ha this is computing so of course it's an utter disaster and everything is a huge mess. There's three main classes of backlight control in the x86 world, all of which have drawbacks:

• ACPI specifies a mechanism for backlight control, and the majority of modern machines implement it. It has the advantage that the brightness query interface is generally aware of anything else in the system which may have changed the brightness, so it's unlikely to get out of sync with reality if the platform tries to do something odd like change the backlight itself in response to an ambient light sensor or some other event. The main drawback is that there's typically a fairly small number of available backlight values, usually somewhere between 8 and 20.
• A platform-specific mechanism. This used to be more popular before the ACPI backlight interface took off, but some machines still require it. The idea here is that there's some sort of platform-specific way of requesting a backlight change, ranging from a vendor-specific ACPI method through to triggering system management calls by magic register writes. These methods usually (but not always) keep in sync with other firmware changes, but rarely provide any more brightness steps than the standard ACPI interface.
• Many mobile GPUs have backlight control registers built in. These usually give you a range of several thousand possible values, but using them will almost certainly leave you out of sync with reality if the firmware touches them at all. To make things worse, the firmware control of the backlight may occur after the gpu - so you could end up with two different controls that both need to be full in order to get maximum brightness. The worst case scenario is that the firmware gets confused by the values not being what it programmed and you end up with a hung machine.

Right now, if there's an ACPI backlight interface then that's usually the only thing we'll show you. We can do that because we can identify if there's an ACPI backlight interface when we parse the ACPI tables at the start of booting, and that information can be registered before we start setting up any other backlights. The problem comes when we have no ACPI backlight interface. We don't have any idea whether there's a platform mechanism until a platform driver loads, which could be at any time. As a result, we've been reluctant to expose GPU-level backlight control because doing so would often give you two separate backlight controls and no indication as to which should be used. Userspace doesn't really have a way to make that decision either, so everyone ends up unhappy.

This is especially problematic with some machines which provide no ACPI or known platform control (or, in the case of some Samsungs, only provide platform control if you have a special Linux BIOS that Samsung won't give you) but can control the backlight via the gpu. Right now you get nothing, because giving you something would potentially break other systems and the needs of the many etc. Sorry! But this is obviously problematic in the long term, especially because multi-GPU machines tend to have multiple ACPI backlight interfaces, so I've been working on a better approach.

When a backlight device is registered, it appears under /sys/class/backlight. If it's an ACPI device it has a symlink pointing to some random ACPI device. If it's a platform device it's pointing at something like "dell-laptop" which is approximately unhelpful when it comes to figuring out what it controls. If it's a GPU-level device then it probably points at the PCI device, which is helpful except in the case where you have multiple backlight controls on a single GPU. So, by and large, you have no good way to identify which backlight control is preferable unless you keep a huge list of all possible backlights along with some scoring.

The first thing I've added to improve this is a "type" attribute. This tells you whether a given backlight is firmware-level (like ACPI), platform-level (like the various laptop drivers) or performs raw register writes (like a GPU driver). That lets userspace decide which interface is preferable. It'll typically be the ACPI interface, because that's the most likely to keep synchronisation and so avoid bizarre brightness bugs. The next thing has been to start fixing up the parent links. There's nothing we can do for the platform level devices, but the ACPI drivers could at least point at PCI devices rather than into ACPI space. That means that multi-GPU systems can now identify which interface to use based on the currently active GPU. Finally, I've started pointing the GPU-level backlight controls at the specific output rather than merely at the PCI device. This probably makes little difference for laptops as such, but once we start exposing backlight control for monitors that support ddcci it'll make things much easier as we'll know which backlight control corresponds to which monitor.

I've then written a small library that accepts information about the output and picks the "best" backlight for the device. It's obviously based on a pile of heuristics and there's a couple of bits of API that I suspect need to be nailed down yet, but it means that this code only needs to be written once. It's then simple to glue this into X drivers, which means that they can expose a "Backlight" xrandr property on each relevant display. That means that backlight control is then handled at the session level with the X server acting as the privileged agent, which simplifies a bunch of things and means we can finally let hal die entirely. Long-term this means we'll have unified backlight control for all of your displays, which is a wonderful thing.

Summary: We kind of suck right now, but there's a reasonably clear path to getting better.

* I was thinking to move out of as many as non-free (FaiF) web services after reading mako s this post and now zack wrote about it with good explanations and reasons. What things I m using? And its replacement: - GMail [and related stuffs] (I don t own server/email setup, may be next year s plan?)
- Facebook (Big question here how will I keep touch with my wife?)
- LinkedIn (My profile there is crap, half baked. Can leave it without worry)
- Twitter (Identi.ca)
- Orkut (Already not using much)
- Github (Gitorious) Thank God. WordPress is amazing and free. I can keep in touch with this amazing blue pill virtual world with blog posts at least. Starting acting on the list from bottom-up. Lets see how far I can go.

So, conventional wisdom is that once you experience something first hand, it sits close to you, so that you can learn from it, think about it, and possibly invoke it. This rings true for me, except for one particular type of experience finishing a large and complicated project like writing a dissertation, taking PhD qualifying exams, like writing a very complicated review article. Once it is done and over with and a few months have walked away, I no longer can fathom, at all, how I even did it. It is as if I went through the experience and then amnesia set in, erasing and wiping out the fibers of events, emotions, and thoughts that went into making and finishing the project. I find this unnerving, for various reasons. First, this di-juncture is not one I experience with other experiences, even unpleasant ones, like pain (broken collarbone, burnt hand, terrible earache), for I am able, nearly always, to shore up some shadow of those experience. More practically, when I am faced again with needing to conquer what feels unconquerable (which, unsurprisingly, is my current predicament), I have no experiential reserve to guide me. Instead, it is like I am standing at a new shore, for the first time. I tell myself, I have done and I can do it but there is no concomitant emotional register or memory that assures me that this is in fact true. Maybe a few more projects like these and there will be some imprint to steer me in the future but somehow I suspect my experiential amnesia will remain the same.

The last few months have been busier than I expected, resulting in various failures to get stuff done. The good news is that things are a little more relaxed now and I'm gradually catching up, but if you've emailed me and I haven't replied then you should probably do so again.

A couple of updates - the source code for the Augen Android tablets I wrote about still hasn't been released, but the vendor does seem to be doing their best. Their supplier seems to be refusing to hand over the source code (they were given a tarball that was supposed to be it, but in fact just turned out to be the various GPLed bits of Android) and they're obviously stuck between a rock and a hard place. The obvious observation is that they should have done due diligence before starting to ship these things, but given that they're out in the wild and they're trying to improve things it seems reasonable to carry on working to try to obtain the source rather than insisting that kmart stop selling them.

Fusion Garage, on the other hand, are still failing to provide source and seem entirely unconcerned about it - they've failed to respond to any of my emails since the first. Augen aren't providing source because they can't, while Fusion Garage aren't providing source because they won't. Irked by this, I've decided to try Don Marti's suggestion and file a case with US Customs. I'll admit that I have absolutely no idea how seriously these cases get taken, and so I've no great expectation of any sort of interesting outcome. But even so, if you're in the US and try to buy a Joojoo then there's a chance that it'll be seized by US customs on the way in.

Please, those of you who send me emails, remember the following:-

• An email has a subject line use it
• If you put a proper description in your subject line, it s easier for me to find the email again
• The subject line is that what the email is about. It s NOT meant for the entire content of your message, no matter how brief
• If you send me an email entitled FYI Don t bitch at me when I don t reply.
• If you ask a question, and it s a vague one, don t get annoyed at me when I ask for clarification

Another week, another Emacs extension! I had (once again) a wonderful idea: what if I could have the weather forecasts in my Org agenda? Wouldn't that be wonderful? My quest started by looking for a service offering a good weather forecast API. I found nothing simple as the hidden Google Weather API, which is nice, but not documented. Not at all. Not a single line. Nah. Then, I wrote a google-weather extension, implementing a basic Emacs Lisp API to retrieve data from the Google service:

ELISP> (google-weather-data->forecast (google-weather-get-data "Paris"))
(((9 8 2010)
  (low "53")
  (high "63")
  (icon "http://www.google.com/ig/images/weather/rain.gif")
  (condition "Rain"))
 ((9 9 2010)
  (low "53")
  (high "69")
  (icon "http://www.google.com/ig/images/weather/chance_of_rain.gif")
  (condition "Scattered Showers"))
 ((9 10 2010)
  (low "54")
  (high "72")
  (icon "http://www.google.com/ig/images/weather/partly_cloudy.gif")
  (condition "Partly Cloudy"))
 ((9 11 2010)
  (low "55")
  (high "75")
  (icon "http://www.google.com/ig/images/weather/partly_cloudy.gif")
  (condition "Partly Cloudy")))

My API even implements data caching, which is nice to speed up the agenda display. By the way, I think my next job will be to hack on the url-cache feature of Emacs, which is utterly buggy and has probably never be used. But that's another topic. Finally, I just had to write another module on top of that to export the forecasts to Org. A screen shot is probably better than a long and boring explanation, so here's the result. My only regret is that the icons provided by Google are ugly squares, so I did not want to use them. On the other hand, I did not found any icon set that would have all the icons Google provides (around 20). So I felt back on the icon naming specification to map the Google images to standard images. Any better idea would be welcome, of course. All the information can be found on the Google Weather for Emacs extension homepage.

There was a feature missing in darcs-monitor which I wanted: to send the e-mail to multiple recipients when a new patch arrives. I tried using:

apply posthook darcs-monitor --charset=UTF-8 email addr1@example.com,addr2@example.com

But it didn't worked. I also tried "addr1@example.com, addr2@example.com", but it also didn't worked. So I included this feature in darcs-monitor and released a new version. I also updated my e-mail address and darcs repository address. While I was working at the debian package, I noticed a bug in 0.3.8-1, and made a upload of 0.3.8-2 to fix it. I also uploaded the new 0.4.0 to experimental. I'm another one using flattr now. I've created my blog as a thing: I also created a thing for darcs-monitor:

A long time ago, when I was first introduced to email, I was using the Mail program from Unix. I quickly converted to Elm, then Mutt, which were both better in terms of interface. Then I found out about Gnus, and I wouldn't dream of letting it go now. However, Gnus has started showing its age several times, and several times have I needed to upgrade the way I was using it: first because I needed to sort and split email, then because I took the sorting out of Gnus and into Procmail for more advanced filtering (including spam filtering), then because I switched to storing the emails on an IMAP server so I could read them remotely from several computers. My setup as of a few days ago was functional, but since I have grown over time to splitting emails into several hundred folders, checking for new messages was becoming more and more boring. So it's time to jump in with all the cool kids and switch to a modern solution: still Gnus of course, but with Dovecot, OfflineIMAP for synchronisation, and let's add email searches into the mix while we're at it. My web searches didn't turn up a simple step-by-step HOWTO, but I assembled bits from different places, and here's my attempt at documenting my new setup. Goals

• Gnus
• Offline operation
• Searching through emails
• and make it fast!

Assumptions

• Email gets delivered (by SMTP or fetchmail or whatever) to a remote server, which can be accessed through IMAP.
• The client is running some sort of standard Unix-like system; Debian GNU/Linux for me, but it should also work with a BSD or Solaris or something else.
• There is enough space on the client to have a copy of the whole email store (or at least of the folders you want to use with this setup Gnus can of course access the others through the slower IMAP server).

Dovecot setup We'll use Dovecot as a local IMAP server. And since we're lazy, we'll access it over a pipe, and dispense with the network part.

• Install Dovecot (aptitude install dovecot-imapd on Debian systems and related).
• Disable its automated running unless you need it for other purposes; on Debian, set ENABLED=0 in /etc/default/dovecot.
• Decide on where you'll store your emails locally; typical place would be $HOME/Maildir.
• That's it!

OfflineIMAP setup OfflineIMAP is basically an optimised two-way synchronisation mechanism between two email repositories . We'll use it in IMAP-to-IMAP mode.

• aptitude install offlineimap, or similar.
• You'll need a ~/.offlineimaprc file, with contents based on the following example:

 [general]
 accounts = MyAccount
 pythonfile = .offlineimap.py
 [Account MyAccount]
 localrepository = LocalIMAP
 remoterepository = RemoteIMAP
 # autorefresh = 5
 # postsynchook = notmuch new
 [Repository LocalIMAP]
 type = IMAP
 preauthtunnel = MAIL=maildir:$HOME/Maildir /usr/lib/dovecot/imap
 holdconnectionopen = yes
 [Repository RemoteIMAP]
 type = IMAP
 remotehost = mail.example.com
 remoteuser = jsmith
 remotepass = swordfish
 ssl = yes
 nametrans = lambda name: re.sub('^INBOX.', '', name)
 # folderfilter = lambda name: name in [ 'INBOX.important', 'INBOX.work' ]
 # folderfilter = lambda name: not (name in [ 'INBOX.spam', 'INBOX.commits' ])
 # holdconnectionopen = yes
 maxconnections = 3
 # foldersort = lld_cmp

• You will of course need to replace the remotehost, remoteuser and remotepass values with your own. If you want to synchronize only a subset of the IMAP folders, uncomment and adapt one of the folderfilter lines; the lambda can be any Python code you like, I guess.
• Run offlineimap. See the emails coming in and being replicated to your local store.
• Run MAIL=maildir:$HOME/Maildir /usr/lib/dovecot/imap. You should get * PREAUTH [CAPABILITY ... STATUS] Logged in as your-login. Type * LIST "" * in there. You should see a list of all the folders.
• The .offlineimaprc mentions a .offlineimap.py, which is where we're going to store some additional code used by OfflineIMAP. Here's a sample:

 # Propagate gnus-expire flag
 from offlineimap import imaputil
 def lld_flagsimap2maildir(flagstring):
   flagmap =  '\\seen': 'S',
              '\\answered': 'R',
              '\\flagged': 'F',
              '\\deleted': 'T',
              '\\draft': 'D',
              'gnus-expire': 'E' 
   retval = []
   imapflaglist = [x.lower() for x in flagstring[1:-1].split()]
   for imapflag in imapflaglist:
       if flagmap.has_key(imapflag):
           retval.append(flagmap[imapflag])
   retval.sort()
   return retval
 def lld_flagsmaildir2imap(list):
   flagmap =  'S': '\\Seen',
              'R': '\\Answered',
              'F': '\\Flagged',
              'T': '\\Deleted',
              'D': '\\Draft',
              'E': 'gnus-expire' 
   retval = []
   for mdflag in list:
       if flagmap.has_key(mdflag):
           retval.append(flagmap[mdflag])
   retval.sort()
   return '(' + ' '.join(retval) + ')'
 imaputil.flagsmaildir2imap = lld_flagsmaildir2imap
 imaputil.flagsimap2maildir = lld_flagsimap2maildir
 # Grab some folders first, and archives later
 high = ['^important$', '^work$']
 low = ['^archives', '^spam$']
 import re
 def lld_cmp(x, y):
   for r in high:
       xm = re.search (r, x)
       ym = re.search (r, y)
       if xm and ym:
           return cmp(x, y)
       elif xm:
           return -1
       elif ym:
           return +1
   for r in low:
       xm = re.search (r, x)
       ym = re.search (r, y)
       if xm and ym:
           return cmp(x, y)
       elif xm:
           return +1
       elif ym:
           return -1
   return cmp(x, y)

The first part of this file adds a new flag that OfflineIMAP will propagate back and forth. By default, only the standard IMAP flags are propagated; we also want to synchronize the gnus-expire flag that Gnus uses to mark expirable articles. It's a hack, but it works for now (maybe someday OfflineIMAP will propagate all the flags it finds?). The second part of that file can be dispensed with (and won't be used unless the foldersort option is uncommented in .offlineimaprc): it's only there to ensure that some important folders are propagated first, and some others go last. I don't know exactly how they are sorted by default, but I'd like the most important ones to come first, so I can start reading them while the archives and the spam are still being fetched. Gnus configuration

• You need to set up a select method for the local IMAP server. The method will be nnimap, the address can be whatever you like (it's just a name anyway), and you only need two options: nnimap-stream, set to shell, and imap-shell-program, which you set to "MAIL=maildir:$HOME/Maildir /usr/lib/dovecot/imap".
• Then browse the server, subscribe to the folders, and so on.
• From time to time, run offlineimap from a terminal, so you get the new messages, delete the old ones, and so on.
• What's that? You hate running stuff by hand? So do I. To run OfflineIMAP in a loop, uncomment the autorefresh option in .offlineimaprc. The value is the duration (in minutes) between two runs.
• Even starting OfflineIMAP is too much work? offlineimap.el to the rescue! Grab it, put it where Emacs will find it, and add the following to your .emacs:

(require 'offlineimap)
(add-hook 'gnus-before-startup-hook 'offlineimap)

Bonus: email searches The simple way:

(require 'nnir)

This goes in your .gnus. Then your group buffer will get a new command. Mark some folders with #, then M-x gnus-group-make-nnir-group (or use the G G shortcut), and type in a set of keywords. This search is performed by the IMAP server (Dovecot), which may or may not be very efficient, especially if you select many folders. Bonus+: email searches, faster The real cool kids use Notmuch nowadays, at least for email indexing and searching. It's fast, it allows complex queries, and it's generally cool. The downside is that it uses up quite some disk space for its indices, in addition to the actual emails. For that reason I'll keep it to my main computer, and I'll stick to nnir on my laptop (which has the same setup apart from that).

• aptitude install notmuch, yada yada.
• notmuch setup, tell it where you stored your Maildir. If you're only going to use Notmuch for searches, I suggest setting tags= to an empty value in .notmuch-config afterwards. We don't want Notmuch to intrude.
• notmuch new will get it indexing the messages you already have.
• Of course, being lazy is being cool, so let's uncomment the postsynchook line from .offlineimaprc.
• Now add (require 'notmuch) to your .gnus. Also (define-key gnus-group-mode-map "GG" 'notmuch-search), for the shortcut.
• But seeing just the message that matches is not enough, sometimes we want the whole thread. Here's a snippet of Lisp for your .gnus, based on Tassilo Horn's configuration. Do a Notmuch search, enter one of the results, type C-c C-c, you'll get transported to the folder where that message was, with the context. Note this requires code from org-mode, so you might need to install that.

(require 'notmuch)
(add-hook 'gnus-group-mode-hook 'lld-notmuch-shortcut)
(require 'org-gnus)
(defun lld-notmuch-shortcut ()
(define-key gnus-group-mode-map "GG" 'notmuch-search)
)
(defun lld-notmuch-file-to-group (file)
"Calculate the Gnus group name from the given file name.
"
(let ((group (file-name-directory (directory-file-name (file-name-directory file)))))
  (setq group (replace-regexp-in-string ".*/Maildir/" "nnimap+local:" group))
  (setq group (replace-regexp-in-string "/$" "" group))
  (if (string-match ":$" group)
      (concat group "INBOX")
    (replace-regexp-in-string ":\\." ":" group))))
(defun lld-notmuch-goto-message-in-gnus ()
"Open a summary buffer containing the current notmuch
article."
(interactive)
(let ((group (lld-notmuch-file-to-group (notmuch-show-get-filename)))
      (message-id (replace-regexp-in-string
                   "^id:" "" (notmuch-show-get-message-id))))
  (if (and group message-id)
      (progn 
  (switch-to-buffer "*Group*")
  (org-gnus-follow-link group message-id))
    (message "Couldn't get relevant infos for switching to Gnus."))))
(define-key notmuch-show-mode-map (kbd "C-c C-c") 'lld-notmuch-goto-message-in-gnus)

Wrap-up This setup can be replicated on several computers, of course. I have it on two, and there's no reason I couldn't have more. The flags do get propagated back and forth, including the Gnus-specific expirable flag. Accessing the local Dovecot is much faster than going through the DSL to the master IMAP server, and I'm pretty convinced that OfflineIMAP and its multi-threading is also faster than Gnus is, even talking to the same remote server. The email searching with Notmuch is a nice bonus, especially since they're fast too (and this despite my 8-year-old computer). There are a few minor glitches. I can live with them, but I should let you know anyway.

• offlineimap.el keeps an OfflineIMAP process running (if using autorefresh), which causes Emacs to complain about when you want to exit.
• OfflineIMAP propagates the new folders from the remote server to the local store, but not the other way round. Folders created from Gnus won't be propagated, so they'll only be visible on one computer.

Apart from that, I'm pretty happy with this new setup. So I hope this documentation will be useful to others, so I can spread the happiness around. Send your thanks to the authors of the software involved (Gnus, Dovecot, OfflineIMAP, offlineimap.el, Procmail, and so on). Let's see how many years I'll keep that system!