Thanks to some advice from Philipp Kern I have now got my Galaxy S running CyanogenMod 7.1.0 which is based on Android 2.3.7 [1]. CyanogenMod has lots of configuration options that seem to be lacking in the stock releases and also supports some advanced features such as OpenVPN and a command-line. I can t properly compare CyanogenMod to the stock Android as I ve only used versions 2.1 and 2.2 of the stock Android. Presumably some of the things that I like about CyanogenMod are in the stock Android 2.3.7 release. The process of updating a phone is difficult and has some risk. Fortunately Samsung provided Download mode in the BIOS to allow recovery. If you mess up the process of updating a Galaxy S and you can get Download mode by holding down volume-down, home, and then power buttons then you can almost certainly recover (so don t panic). The CyanogenMod people don t provide any documentation on upgrading from Android 2.2 (which is what Optus is still shipping AFAIK). So you will probably have some difficulty when upgrading a Galaxy S that you get in Australia (it seems that Optus is the only company shipping them in volume). As an aside if you want to buy a Gel Case for a Galaxy S in Australia then visit an Optus store. It seems that Optus is the only phone store that hasn t run out their Galaxy S accessories in favor of the Galaxy S2. I have previously written about the Galaxy S and Three Networking [2]. Now that I have the Galaxy S as my primary phone on the Virgin network all my data corruption problems are solved, the problem is entirely related to Three. With CyanogenMod there is an option to be able to toggle the LED Flash as a torch from the drop-down menu, this makes the lack of such a LED on the Galaxy S even more of an annoyance. I have also discovered that the Galaxy S apparently doesn t have a status LED! This makes it the only phone that I ve ever owned that has no clear way of informing me when the battery is charged! It s also really useful to have a flashing LED to indicate low battery when running a full screen app, and to have a flashing LED to indicate that email has been received. Someone should design a phone with multiple LEDs to indicate different things. I d like to have one LED to indicate charging status and another to indicate whether there is unread email or SMS. Whatever the cost of including a LED during manufacture it would have to be almost nothing compared to the ~$500 sale price of a phone. Wikipedia says The Samsung Galaxy S features a PowerVR graphics processor, yielding 20 million triangles per second, making it the fastest graphics processing unit in any smartphone at the time of release. Also, upon release, the Galaxy S was both the first Android phone to be certified for DivX HD, and at 9.9 mm was the thinnest smartphone available . I don t care about any of that, I want a phone with decent battery life, a LED Flash , and a status LED. The main benefit I get from the Galaxy S over the Xperia X10 is the greater storage. The Xperia X10 has a total of 1G of storage and only 465M of that is available for application install. My Galaxy S has 16G of internal storage of which 1.8G is available for phone apps and 13G is available for pictures and other mass storage. Having 1.8G for phone apps and internal phone storage used by such apps (which includes the offline IMAP cache) is a massive benefit, enough to outweigh the lack of a staus LED and a Flash LED. What I Really Want I d rather have a Samsung Galaxy Note. The Note has a LED flash, a 5.3 screen with 1280*800 resolution which is much better for running as a SSH client and also good for web browsing. I m not inclined to spend money on a phone now, so I ll probably use the Galaxy S until Virgin offers me a new phone or someone just gives me a new phone (I can always hope). One of the many nice features in the Galaxy Note is a built in stylus. When using my current phones for web browsing I sometimes find it difficult to have a touch registered to the desired part of the screen, this is a real problem with the Opera web browser which requires a long press to open a URL in a new tab.

• [1] http://www.cyanogenmod.com/
• [2] http://etbe.coker.com.au/2011/11/21/galaxy-xperia-android-network/

When someone wants to package a piece of software for Debian, he has to file an Intent to Package, by filing a special bug report against the pseudo-package wnpp (work-needing and prospective packages) with a title of ITP: software name . This way, we can synchronize our efforts, for instance if someone else wants to package the same piece of software. I have seen two recent examples of a misunderstanding of the correct use of such reports, so perhaps this reminder may be of some help for newcomers.While an Request for Package means that its author would like someone to package a piece of software, and ITP means that its author wants to package it himself and to have it integrated to Debian (in fact, an RFP also becomes an ITP when someone is willing to do the work). This means that the owner of an ITP bug is responsible for the corresponding work. Given that the ITP report is meant to inform others, that implies that he should document his progress by appending messages to the bug report. In particular, when something is blocking the work (typically a failure to find a sponsor), that should be indicated in the ITP report so that it does not appear as being abandoned. So, in a nutshell, when you have opened an ITP, do not expect someone to take it, to reply to it or whatever deus ex machina; you are now the owner of that bug, which means you are responsible of making it live and eventually closing it by sending your package to Debian. The progress of a packaging work should be documented, especially if it requires some specific help such as sponsoring, and the ITP report is the best place to do that since this is where interested people will look for status information.

We have customers, which require a report about what we've done during maintenance windows. Usually this includes a report about upgrades, newly installed packages etc. and obviously everything we've done apart from that.
Till now we've prepared them manually. For a greater bunch of systems this is a big PITA, because to be somehow useful you have to collect the data for all systems and after that, prepare a report where you have:

• The upgrades done on all systems (e.g. a libc or kernel update)
  
  seperated from
  
• the updates specific to a certain system or system class

Its also error-prone because humans make mistakes.

Perl to the rescue!
At least the part about generating a report about installed/upgraded/removed packages could be automated, because dpkg writes a well-formed logfile in /var/log/dpkg.log. But I noticed that there appearently is no library specialised at parsing that file. Its not a big deal, because the format of that file is really simple, but a proper library would be nice anyway.
And so I wrote such a library.
It basically takes a logfile, reads it line by line and stores each line parameterized into a generic object.

Its features include:

• Parsing the logfile (obviously)
• Storing each line in a logfile as a DPKG::Log::Entry object that holds informations about the entry type (e.g. action or status update), the package associated (if any), timestamp, verbatim line, line number etc.

• Limiting the number of lines parsed to a time range specified by DateTime objects

Based on that, I wrote another library DPKG::Log::Analyse, which takes a log, parses it with DPKG::Log and then extracts the more relevant information such as installed packages, upgraded packages, removed packages etc.

This, in turn, features:
- Info about newly installed packages
- Info about removed packages
- Info about upgraded packages
- Info about packages which got installed and removed again
- Info about packages which stayed halfinstalled or halfconfigured at the end of the logfile (or defined reporting period)


These libraries are already uploaded to CPAN and packaged for Debian.
They passed the NEW queue very quickly and are therefore available for Sid:

http://packages.debian.org/sid/libdpkg-log-perl

As an example use (and for my own use case, as stated above), I wrote dpkg-report, which uses the module and a Template::Toolkit based template to generate a report about what happened in the given logfile.
It currently misses some documentation, but it works somehow like this:

Report for a single host over the full log:

dpkg-report

Report for a single host for the last two days:

dpkg-report --last-two-days

Report for multiple hosts (and logfiles):
The script expects that each log file has the name <systemname>.dpkg.log so that it can guess the hostname from the system and can grab all such log files from a directory if a directory is specified as log-file arguments:</systemname>

dpkg-report --log-file /path/to/logs

This will generate a report about all systems without any merging.

Report for multiple hosts with merging:

dpkg-report --log-file /path/to/logs --merge

This will do the following:

• Report packages which where installed/removed/etc. on all systems seperate from the upgrades only done on the specific systems
• If systems start with a common name and end on a number (e.g. mail1, mail2, mail3 etc.), report packages which where installed on all systems with that common name seperate from the upgrades only done on the specific systems
• For a specific system list only the changes on that particular system and nothing else.

A (fictive) report could look some what like this:

dpkg-Report for all:
------------------------
Newly Installed:
abc
Removed:
foo
Upgraded:
bar (0.99-12 - 1.00-1)
dpkg-Report for test*:
------------------------
Newly Installed:
blub
dpkg-Report for test1:
------------------------
Upgraded:
baz (1.0.0-1 -> 1.0.0-2)
dpkg-Report for test2:
------------------------
Upgraded:
zab (0.9.7 -> 0.9.8)

Currently this report generator is only included in the source package or in the git(hub) repository of the library. I wonder if it makes sense to let the source package build another binary package for it.
But its only a 238 lines perl script with a dependency on the perl library so I'm unsure if it warrants a new binary package. What do others think?

Today I wanted to clone my dotfiles repository (no, not available online, too much private stuff in there) to a remote machine and noticed that it has grown way too big (20MiB working directory and about 200MiB in .git), so I decided to clean it up. git gc did clean up a couple of megabytes, but .git was still about 190MiB, so I wasn t satisfied. Short thinking revealed the lost megabytes are somewhere in the history when I accidentally added some files and removed them afterwards (iceweasel, icedove, it s you I m blaming ;)). But how the heck do I find and remove them? git filter-branch -f --index-filter 'git rm --cached --ignore-unmatch FILE' -- --all will remove FILE from all commits, says git-filter-branch(1), but how to find those files? They are not in my working directory anymore and I do not want to checkout every revision and look for big files in there. Let s ask git itself :)
for commit in git log --all --pretty=format:%H ; do git ls-tree -r -l $commit; done awk ' print $4 " " $5 ' sort -nu will show all files (actually all versions of all files) ever known to git, with the biggest ones at the end. Just identify the really big (unused) ones and remove them as above, thats what you think, right? Right, but .git won t be any smaller. Huh? Read git-filter-branch(1) again, just create a clone and it will be smaller, so mission accomplished! Now I had just 6MiB to push (compressed). For reference, the old tree would have used something about 150MiB to push.

0 comment(s) this blog is flattr enabled

This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (310.73 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Dpkg: Multi-Arch Saga I know lots of people are waiting the landing of multiarch in Debian unstable, and so am I. Things are progressing, though not as quickly as I hoped. Guillem merged about half of the branch between the 24th October and the 6th of November. After that most of the work happened on his personal repository in his pu/multiarch/master branch. I verify this repository from time to time because Guillem does not inform me when he has made progress. I noticed changes on his repository on the 10th, 19th, 23th, 28th of November and on the 1th of December. He announced a long time ago that he had some interface changes and up to now only wrote about the switch from the command-line option --foreign-architecture (to put in /etc/dpkg/dpkg.cfg) to the explicit command dpkg --add-architecture that only needs to be called once (see mail here). As of today (December 2th), the promised email for the other interface changes is still not here. On November 23th, I reviewed Guillem s work and tried to run the code in his branch. I spent the whole day chasing up regressions and submitted lots of fixes to Guillem. Thanks to the extensive test-suite I wrote when I developed my branch, it has been fairly easy to track them all down. All the issues I reported have been fixed in the latest version of Guillem s branch although the fixes are often slightly different from those that I submitted. Dpkg: Squeeze Backport At the start of the month, I uploaded what I expected to be a fairly uncontroversial backport of dpkg 1.16.1.1. It turns out I was wrong. After some discussion, I think we came to an agreement that it was acceptable to backport dpkg-dev and libdpkg-perl only. My goal was not to bring the latest dpkg to users but to make it easier for package maintainers to backport packages using new features provided by dpkg-dev >= 1.16 (such as hardening build flags, the makefile snippets provided in /usr/share/dpkg/, or the improved dpkg-buildflags interface). Thus I modified the source package uploaded to squeeze-backports to build only dpkg-dev and libdpkg-perl. It has been uploaded on November 23th and it s waiting in the NEW queue for a backports admin to process it. Misc Dpkg Work I merged a patch of Colin Watson to be able to verify build-dependencies for a foreign architecture (taking into account the Multi-Arch status of each package listed). I released dpkg 1.16.1.2 with two minor fixes that were sitting in the sid branch. I wanted to get rid of this so that the path is clear for a 1.16.2 upload with multiarch. The package just migrated to testing so we re fine. I spent another day doing dpkg bug triaging on Launchpad, we re now down to 77 bugs with many of them tagged as incomplete and likely to expire in 2 months. The Debian Administrator s Handbook We released a sample chapter so that it s easier to have an idea of the quality of the book. The chapter covers the APT tools quite extensively. I bet that even you could learn something about apt-get/aptitude The crowfunding campaign on Ulule ended on November 28th.
With 675 supporters, we raised 24385 EUR. Of those, 14935 EUR have been put in the liberation fund and the rest corresponds to the various pre-orders and rewards offered. This means that the translation will happen (we just started) but that the book is currently not going to be released under a free license. Don t despair As planned, the liberation campaign is carried on until the 25 K target is reached! Instead of being hosted on Ulule, this permanent campaign is on the project website at debian-handbook.info/liberation/. Note that any contribution of 10 EUR or more means that you get a copy of the ebook as soon as it s available (even if the liberation target is not reached). Package Tracking System At the start of the month, I filed two ideas of improvements for the PTS in the bug tracking system: #647258 is about showing outstanding bugs that relate to a release goal and #647901 is about warning maintainers that the package is affected by a current transition. If you re a coder and want to start contributing to Debian and its QA team, those bugs could be interesting targets for a start. In both cases, I have been in contact with members of the release team because those ideas require some structured data from the release team as input. Thanks to Meddi Dohguy and Niels Thykier for their help. Later in the month, the topic of relocating the PTS once again came up. For historical reasons, the PTS was hosted on master.debian.org together with the BTS. Nowadays the BTS has its own host and it made no sense anymore to have the PTS separate from the rest of the QA services hosted on qa.debian.org (currently quantz.debian.org). So together with Martin Zobel Helas we took care to plan the migration and on November 19th we executed the plan. It worked like a charm and almost nobody noticed (only one undocumented dependency was missed, which broke the SOAP interface). Misc Packaging Work WordPress was broken in Ubuntu and it was also not properly synchronized with Debian due to an almost useless change on their side. Thus I requested a sync so that the working version from Debian gets imported in Ubuntu. I sponsored the docbook-xsl 1.76.1 upload that I needed for Publican. Then I updated Publican just to discover that the test-suite triggers a new bug in fop (filed as #649476). I disabled the test-suite temporarily and uploaded Publican 2.8 to unstable. BTW, I also filed 2 upstream bugs with patches for issues I discovered while trying to generate the sample chapter of my book (see here and here). I uploaded a version 0.7.1 of nautilus-dropbox and fixed #648215 at the same time. I made an NMU of bison to fix a long-standing release critical bug that hit me once more during an upgrade (see #645038). I uploaded to experimental a new version of gnome-shell-timer compatible with GNOME 3.2. I took the opportunity to install from experimental the few GNOME 3.2 packages which are not yet in unstable Thanks See you next month for a new summary of my activities.

7 comments Liked this article? Click here. My blog is Flattr-enabled.

(including English!) The string freeze of Debian Installer officially ended at 23:59 yesterday (Sept. 20th). Indeed, this was extended a bit to today, with agreement by Otavio Salvador who I thank for this. That allowed Zak to "save" Tagalog and also the Welsh and Latvian translators to polish their work. We now have to decide about some of these languages: those that failed to meet the release criteria but were formerly activated in D-I. There are four such languages: Amharic, Welsh, Estonian and Northern Sami. Please find below the mail I just sent to debian-i18n and debian-boot. I promised that this discussion would happen in public. It will (but it will be short as we can't delay the release of the installer for ages....and I think that my proposals are reasonable!)

First of all, the numbers as of Sunday Sept. 21st 09:32 UTC (date of
the last commit with an l10n update):
Languages meeting the release criteria: 59
------------------------------------------
Already activated and complete for level 1: 51
 Arabic, Belarusian, Bulgarian, Bosnian, Catalan, Czech, Danish,
 German, Dzongkha, Greek, Esperanto, Spanish, Basque, Finnish, French,
 Galician, Gujarati, Hebrew, Hindi, Croatian, Hungarian, Indonesian,
 Italian, Japanese, Georgian, Khmer, Korean, Lithuanian, Latvian,
 Macedonian, Malayalam, Marathi, Norwegian Bokm l, Nepali, Dutch,
 Norwegian Nynorsk, Punjabi, Polish, Portuguese, Brazilian Portuguese,
 Romanian, Russian, Slovak, Albabian, Swedish, Tamil, Thai, Turkish,
 Vietnamese, Simplified Chinese, Traditional Chinese
Already activated and complete for sublevels 1 and 2: 6
Bengali, Kurdish, Slovenian, Tagalog, Ukrainian, Wolof
Not yet activated languages complete for sublevels 1 and 2: 2
 (the mail in -i18n and -boot says 3 but this is an error by me)
Irish, Serbian
Languages failing to meet the release criteria: 15
--------------------------------------------------
Activated languages: 4
Amharic, Welsh, Estonian, Northern Sami
Not yet activated languages: 11
Afrikaans, Persian, Armenian, Icelandic, Kazakh, Kannada,
Malagasy, Malay, Telugu, Urdu, Xhosa
Discussion
----------
(careful people will notice that I moved Welsh down to "failed to meet
the release criteria" as this is what is technically correct)
Nothing to discuss for the 57 already activated languages that meet
the defined criteria. They'll be kept or first activated in the RC1
release of Debian Installer.
Similarly, nothing to discuss for the 11 languages that were not
activated and haven't made it. They will remain unactivated.
Two languages should be activated as they have met the release
criteria for the first time during the string freeze: Irish and Serbian.
This adds more load (and size changes) to D-I but I really don't see
any reason to not follow our own rules there.
The discussion comes for the 4 languages that fail to meet the release
criteria. Here are my proposals with some rationale:
Amharic: 
  I would really dislike deactivating Amharic because it's highly
  symbolic to have the language of Ethiopia activated. We have so few
  African languages. Also, the translation is nearly complete and the
  translator was well coping with updates until July. The missing
  stuff for Amharic in sublevels 1 and 2 are messages about loading
  drivers or firmware from removable media, the rescue mode stuff for
  the graphical installer and some messages that briefly appear during
  finish-install. A little bit more important is the message warning
  that the boot partition is not ext2 or ext3, added in August by
  tbm. I think this is not enough to drop out one year of efforts for
  the translator
  As a consequence, I propose to KEEP Amharic.
Welsh:
  Only five strings are missing in sublevels 1 and 2 because of the
  small experience of PO files by the person who completed the
  translation during last week. One will make the regular user login
  name screen to be in English and others will make the GRUB password
  screen to be in English as well, that's all.
  Additionnally, we can safely assume that all potential users of
  Welsh have good skills in English...and will therefore very easily
  cope with these screens.
  As a consequence, I propose to KEEP Welsh.
Estonian:
  The translation had NO update since Etch. The last update is dated
  back to Feb. 17th 2007. I haven't got any sign of life from the
  translator and no Estonian users have volunteered to maintain the
  translation.
  Missing strings are in many places, including several screens that
  appear in default installs. Even though one can assume that the
  skills of the average Linux user in Estonia is fairly good, I think
  this is not enough to throw users in a big mix of English and
  Estonian.
  As a consequence, I propose to DROP Estonian.
Northern Sami:
  The translation is very incomplete. With about any other language,
  that would be a reason to drop the translation.
  However, a few reasons make me suggest keeping it:
   - Northern Sami is mostly used in Norway and D-I will fall back
     to Norwegian Bokm l which is understood by all potentials users
     as it is teached in all Norwegian schools. 
   - Users will be warned, *in Sami*, about this situation
   - The choice of Sami will be kept in localechooser even if the
     translations are dropped. This is on request of Debian Edu
     developers to avoid them to develop a special boot floppy
     to offer the choice of Sami (a requirement for Norwegian
     schools). I personnally think this is a reward to Debian Edu and
     its ancestor Skolelinux for their initial involvement in the
     development of D-I
  As a consequence, I propose to KEEP Northern Sami.
I understand that these choices may be debatable and some may sound
slightly subjective. I however think this is the best way to be fair
with translators' efforts without compromising the quality of D-I.
Please note that the final word on this will be by D-I release
managers...but advices are very much welcomed.

Next February, 17 to 19th, a Bug Squashing Party (BSP) is organized at IRILL offices (disclosure: I work for IRILL part time), close to Place d'Italie in Paris. The principle of a BSP is to gather Debian contributors to tackle a maximum of bugs in Debian. This event is also the opportunity for new potential contributors to meet Debian Developers or Maintainers. Numerous regular contributors will attend to this BSP and will help newcomers to fix their first bugs. For organization reasons, an inscription on the Debian wiki is mandatory. Non-Parisian can find a list of hostel close to the IRILL's offices.

uniq is a great tool, since it returns the unique (adjacent) lines of the given input. But it has a limitation: it can't check for uniqueness only the first N fields (while it allows to ignore them, weird).

So, what to do if you have a long file, and lines with several fields, but you're only interested in getting the line with the different first 2 field (but all the rest of the line content? awk to the rescue!

$ awk '!x[$1]++' file

will print the (complete) lines of file that has the first field unique. You can set $1$2 to have lines unique on the first 2 fields, and so on. Thanks to this forum post, but there's some other interesting articles.

One would expect, that one of the most used script languages of the world would be ableto do proper comparisons of numbers, even big numbers, right?Well, PHP is not such a language, at least not on 32bit systems.Given a script like this:

<?

$t1 = "1244431010010381771";

$t2 = "1244431010010381772";

if ($t1 == $t2)

print "equal\n";

?>

A current PHP version will output:

schoenfeld@homer ~ % php5 test.php
equal

It will do the right thing on 64bit systems (not claiming that the numbers are equal).Interesting enough: An equal-type-equality check (see my article from a few years ago) will not tell that the two numbers are equal.

Forbes has an interesting article about crowd-sourcing by criminals and law enforcement [1]. Ulissescastr0 made a Youtube video showing how to install SE Linux on Debian/Etch [2]. Probably no-one is using Etch nowadays so this video is outdated, but it s a good way of teaching people. It would be good if someone made a similar video showing how to do SE Linux things on Squeeze. I discovered the above SE Linux video through Explow which provides a neat interface to multiple searches and information sources [3]. I don t think I will be using Explow much in future as I could get the same result through Google video search. They also have a news portal but there are other sites for that. But it does seem that Explow would be useful for newbies. Eric Michael Johnson wrote an interesting article about the inherent bias in Psychological research based in the US [4]. People who live in urban environments think differently in some ways to people who live in different environments or who have different lifestyles. Therefore generalising from university students in the US to the entire human race is likely to get incorrect results. This is something to consider the next time you are tempted to generalise to the wider population from your own friends, colleagues, etc. The Daily Kos has a scary article about the TSA having a woman detained for reciting part of the US constitution [5]. The US will remain on my list of countries to avoid for the forseeable future. Vorlon has written an informative article about the use of hardening options when building Debian packages [6]. It s now even easier to do this, so every package that simultaneously deals with data of differing levels of integrity or sensitivity should be built this way. Bunker Roy gave an interesting TED talk about his Barefoot College that teaches useful skills to people in rural parts of India who don t have a traditional school education [7]. His talk really shows up some of the arrogance in the people who run traditional education. Justin Hall-Tipping gave an interesting TED talk about ways of solving the world energy problems [8]. He started with explaining the problems and why they need to be urgently solved and then described in detail some of the research that his group has done to solve the problems. This includes flexible photo-voltaic cells, infra-red vision to save on lighting, and a way of using carbon nano-tubes to control the thermal properties of windows.

• [1] http://www.forbes.com/sites/oreillymedia/2011/10/03/the-rise-of-crime-sourcing/
• [2] http://www.youtube.com/watch?v=FLPx7HLLteI
• [3] http://explow.com
• [4] http://blogs.plos.org/blog/2010/09/23/reflections-on-the-weird-evolution-of-human-psychology/
• [5] http://www.dailykos.com/story/2011/10/18/1027775/-TSA-Arrests-Me-for-Using-the-Fourth-Amendment-as-a-Weapon-(Tales-from-the-Edge-of-a-Revolution-2)?via=siderec
• [6] http://web.dodds.net/~vorlon/wiki/blog/Debian:_not_stale_just_hardened/
• [7] http://www.ted.com/talks/bunker_roy.html
• [8] http://www.ted.com/talks/justin_hall_tipping_freeing_energy_from_the_grid.html

Conrad Sanderson just released the second pre-release 2.3.92 of what will be Armadillo 2.4.*. This is now in RcppArmadillo release 0.2.31 which is already on CRAN as of this morning. The NEWS entries summarising the changes for both are below:

0.2.31  2011-11-28
    o   Upgraded to Armadillo test release 2.3.92 "Loco Lounge Lizard (Beta 2)"
          * fixes for linking on Ubuntu and Debian systems
          * fixes for inconsistencies in interactions between matrices and
            cubes

Courtesy of CRANberries, there is also a diffstat reports for 0.2.31 relative to 0.2.30 As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

Once the private key of some HTTPS web site is compromised, an attacker is able to build a man-in-the-middle attack to intercept and decrypt any communication with the web site. The first step against such an attack is the revocation of the associated certificate through a CRL or a protocol like OCSP. Unfortunately, the attacker could also have recorded past communications protected by this private key and therefore decrypt them. Forward secrecy allows today information to be kept secret even if the private key is compromised in the future. Achieving this property is usually costly and therefore, most web servers do not enable it on purpose. Google recently announced support of forward secrecy on their HTTPS sites. Adam Langley wrote a post with more details on what was achieved to increase efficiency of such a mechanism: with a few fellow people, he wrote an efficient implementation of some elliptic curve cryptography for OpenSSL.

Without forward secrecy To understand the problem when forward secrecy is absent, let s look at the classic TLS handshake when using a cipher suite like AES128-SHA. During this handshake, the server will present its certificate and both the client and the server will agree on a master secret. This secret is built from a 48byte premaster secret generated and encrypted by the client with the public key of the server. It is then sent in a Client Key Exchange message to the server during the third step of the TLS handshake. The master secret is derived from this premaster secret and random values sent in clear-text with Client Hello and Server Hello messages. This scheme is secure as long as only the server is able to decrypt the premaster secret (with its private key) sent by the client. Let s suppose that an attacker records all exchanges between the server and clients during a year. Two years later, the server is decommissioned and sent for recycling. The attacker is able to recover the hard drive with the private key. She can now decrypt any session she recorded: the encrypted premaster secret sent by a client is decrypted with the private key and the master secret is derived from it. The attacker can now recover passwords and other sensitive information that can still be valuable today. The main problem lies in the fact that the private key is used for two purposes: authentication of the server and encryption of a shared secret. Authentication only matters while the communication is established, but encryption is expected to last for years.

Diffie-Hellman with discrete logarithm One way to solve the problem is to keep using the private key for authentication but uses an independent mechanism to agree on a shared secret. Hopefully, there exists a well-known protocol for this: the Diffie-Hellman key exchange. It is a method of exchanging keys without any prior knowledge. Here is how it works in TLS:

1. The server needs to generate once (for example, with openssl dhparam command):
   • p , a large prime number,
   • g , a primitive root modulo p (for every integer a coprime to p , there exists an integer k such that g^k=a\mod p ).
2. The server picks a random integer a and compute g^a \mod p . After sending its regular Certificate message, it will also send a Server Key Exchange message (not included in the handshake depicted above) containing, unencrypted but signed with its private key for authentication purpose:
   • random value from the Client Hello message,
   • random value from the Server Hello message,
   • p , g ,
   • g^a\mod p=A .
3. The client checks that the signature is correct. It also picks a random integer b and sends g^b \mod p=B in a Client Key Exchange message. It will also compute A^b\mod p=g^ ab \mod p which is the premaster secret from which the master secret is derived.
4. The server will receive B and compute B^a\mod p=g^ ab \mod p which is the same premaster secret known by the client.

Again, the private key is only used for authentication purpose. An eavesdropper will only know p , g , g^a\mod p and g^b\mod p . Computing g^ ab \mod p from those values is the discrete logarithm problem for which there is no known efficient solution. Because the Diffie-Hellman exchange described above always uses new random values a and b , it is called Ephemeral Diffie-Hellman (EDH or DHE). Cipher suites like DHE-RSA-AES128-SHA use this protocol to achieve perfect forward secrecy¹. To achieve a good level of security, parameters of the same size as the key are usually used (the security provided by the discrete logarithm problem is about the same as the security provided by factorisation of two large prime numbers) and therefore, the exponentiation operations are pretty slow as we can see in the benchmark below:

Diffie-Hellman with elliptic curves Fortunately, there exists another way to achieve a Diffie-Hellman key exchange with the help of elliptic curve cryptography which is based on the algebraic structure of elliptic curves over finite fields. To get some background on this, be sure to check first Wikipedia article on elliptic curves. Elliptic curve cryptography allows one to achieve the same level of security than RSA with smaller keys. For example, a 224bit elliptic curve is believed to be as secure as a 2048bit RSA key.

Some theory Diffie-Hellman key exchange described above can easily be translated to elliptic curves. Instead of defining p and g , you get some elliptic curve, y^2=x^3+\alpha x+\beta , a prime p and a base point G . All those parameters are public. In fact, while they can be generated by the server, this is a difficult operation and they are usually chosen among a set of published ones. The use of elliptic curves is an extension of TLS described in RFC 4492. Unlike with the classic Diffie-Hellman key exchange, the client and the server need to agree on the various paremeters. Most of this agreement is done inside Client Hello and Server Hello messages. While it is possible to define some arbitrary parameters, web browsers will only support a handful of predefined curves, usually NIST P-256, P-384 and P-521. From here, the key exchange with elliptic curves is pretty similar to the classic Diffie-Hellman one:

1. The server picks a random integer a and compute aG which will be sent, unencrypted but signed with its private key for authentication purpose, in a Server Key Exchange message.
2. The client checks that the signature is correct. It also picks a random integer b and sends bG in a Client Key Exchange message. It will also compute b\cdot aG=abG which is the premaster secret from which the master secret is derived.
3. The server will receive bG and compute a\cdot bG=abG which is the same premaster secret known by the client.

An eavesdropper will only see aG and bG and won t be able to compute efficiently abG . Using ECDHE-RSA-AES128-SHA cipher suite (with P-256 for example) is already a huge speed improvement over DHE-RSA-AES128-SHA thanks to the reduced size of the various parameters involved. Web browsers only support a handful of well-defined elliptic curves, chosen to ease an efficient implementation. Bodo M ller, Emilia K sper and Adam Langley have provided 64bit optimized versions of NIST P-224, P-256 and P-521 for OpenSSL. To get even more details on the matter, you can read the end of the introduction on elliptic curves from Adam Langley, then a short paper from Emilia K sper which presents a 64bit optimized implementation of the NIST elliptic curve NIST P-224.

In practice First, keep in mind that elliptic curve cryptography is not supported by all browsers. Recent versions of Firefox and Chrome should handle NIST P-256, P-384 and P-521 but for most versions Internet Explorer, you are currently out of luck. Therefore, you need to keep accepting other cipher suites. You need a recent version of OpenSSL. Support for ECDHE cipher suites has been added in OpenSSL 1.0.0. Check with openssl ciphers ECDH that your version supports them. If you want to use the 64bit optimized version, you need to run a snapshot of OpenSSL 1.0.1, configured with enable-ec_nistp_64_gcc_128 option. A recent GCC is also required in this case. Next, you need to choose the appropriate cipher suites. If forward secrecy is an option for you, you can opt for ECDHE-RSA-AES128-SHA:AES128-SHA:RC4-SHA cipher suites which should be compatible with most browsers. If you really need forward secrecy, you may opt for ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:EDH-DSS-DES-CBC3-SHA instead. Then, you need to ensure the order of cipher suites is respected. On nginx, this is done with ssl_prefer_server_ciphers on. On Apache, this is SSLHonorCipherOrder on. UPDATED: You need to check ECDHE support for your web server. For nginx, the support has been added in 1.0.6 and 1.1.0. The curve selected defaults to NIST P-256. You can specify another one with ssl_ecdh_curve directive. For Apache, it has been added in 2.3.3 and does not exist in the current stable branch. Adding support for ECDHE is quite easy. You can check how I added it in stud. This issue also exists for DHE cipher suites, in which case you also might have to specify DH parameters to use (generated with openssl dhparam) using some special directive or by appending the parameters to the certificate. Check Immerda Techblog article for more background about this point. The implementation of TLS session tickets may be incompatible with forward secrecy, depending on how they are implemented. When they are protected by a random key generated at the start of the server, the same key could be used for months. Some implementations² may derive the key from the private key. In this case, forward secrecy is broken. If forward secrecy is a requirement for you, you need to either disable tickets or ensure that key rotation happens often. Check with openssl s_client -tls1 -cipher ECDH -connect 127.0.0.1:443 that everything works as expected.

Some benchmarks With the help of the micro-benchmark tool that I developed for my previous article, we can compare the efficiency of cipher suites providing forward secrecy: I have used a snapshot of OpenSSL 1.0.1 (2011/11/25). The optimized version of ECDHE is the one you get by using enable-ec_nistp_64_gcc_128 option when configuring OpenSSL. Let s focus on the server part. Enabling DHE-RSA-AES128-SHA cipher suite hinders the performance of TLS handshakes by a factor of 3. Using ECDHE-RSA-AES128-SHA instead only adds an overhead of 27%. However, if we use the 64bit optimized version, the cost is only 15%. The overhead is only per full TLS handshake. If 3 out of 4 of your handshakes are resumed, you need to adjust the numbers. Your mileage may vary but the computational cost for enabling perfect forward secrecy with an ECDHE cipher suite seems a small sacrifice for better security.

---

1. Perfect forward secrecy is an enhanced version of forward secrecy. It assumes each exchanged key are independent and therefore a compromised key cannot be used to compromise another one.
2. For example, this is the case of the implementation I have proposed for stud to enable sharing of tickets between multiple hosts.

I got home from work tonight and zfs had reported that one of the disks in my main pool was dying. no great problem. insert a disk i had spare, and run zpool replace like so:

ganesh:~# zpool replace -f export  scsi-SATA_WDC_WD10EARS-00_WD-WMAV50933036 scsi-SATA_ST31000528AS_9VP16X03

and now it s just a matter of waiting until the data has resilvered and i can remove the old dying drive.

ganesh:~# zpool status export -v
  pool: export
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scan: resilver in progress since Mon Nov 28 19:25:49 2011
    75.1G scanned out of 2.29T at 88.3M/s, 7h17m to go
    18.0G resilvered, 3.21% done
config:
        NAME                                                   STATE     READ WRITE CKSUM
        export                                                 ONLINE       0     0     0
          raidz1-0                                             ONLINE       0     0     0
            scsi-SATA_WDC_WD10EACS-00_WD-WCASJ2114122          ONLINE       0     0     0
            scsi-SATA_WDC_WD10EACS-00_WD-WCASJ2195141          ONLINE       0     0     0
            scsi-SATA_WDC_WD10EARS-00_WD-WMAV50817803          ONLINE       0     0     0
            replacing-3                                        ONLINE       0     0     0
              scsi-SATA_WDC_WD10EARS-00_WD-WMAV50933036        ONLINE       0     0     0
              scsi-SATA_ST31000528AS_9VP16X03                  ONLINE       0     0     0  (resilvering)
        logs
          scsi-SATA_Patriot_Torqx_278BF0715010800025492-part6  ONLINE       0     0     0
        cache
          scsi-SATA_Patriot_Torqx_278BF0715010800025492-part7  ONLINE       0     0     0
errors: No known data errors

(I use /dev/disk/by-id device names so I don t have to care about the kernel detecting the disks and disk controllers in a different order on each reboot also makes it easier to identify which disk is having a problem as I can squint at the front of the drives in my Lian Li 4-in-3 hotswap bays and read the serial number sticker) OK, yeah, sure it s not that much different from replacing a dying drive in an mdadm or hardware raid array but one nice advantage of zfs over raid is that it only resyncs the data in use, not the empty or unused blocks. given that these are 1TB drives in a raidz vdev, and the pool is 63% used that means syncing only about 630GB rather than 1TB. ps: in the time it s taken to write this, it s up to 10.11% done. 88MB/s reading data from the existing raidz drives and writing to a single drive isn t too bad at all. one of the reasons i like zfs is a post from: Errata

Commitments require care, and recently I have been suffering from the delusion that making more commitments will make me more able to achieve them. When overcommit reaches a certain point, the OOT (out of time) killer comes and reaps time from whatever it finds, often with disappointing consequences. (See also: OOM Killer.)

This week-end, after a few week-ends more targeted on real life activities (country house fall works, 70km ultra run preparation and ultra run week-end), I spent most of Sunday working on two important uploads of packages I'm involved in. So, as of Sunday evening, samba 3.6.1 is now in unstable. This is the first 3.6 version to land there, after earlier versions were in experimental while Steve Langasek was working on multi-arch support and dh7-like debian/rules transition in unstable, with 3.5 versions. Merging together changes in 3.5 and 3.6 branches has been a bit painful. I guess I had it right (thankfully, I merged 3.5 packaging changes from time to time to the 3.6 branch). If there are problems with 3.6 packages, anyway, you know what to do. I also worked to package the new version of pytrainer, a sport activity logging tool. pytrainer is maintained by the "pkg-running" team, aka No l K the, Ralf Treinen and /me. This is the software I use for keeping track of all my running activities. We have a very good link with our upstream and I even recently brought several translation updates by involving Debian translators in the loop. While working to integrate version 1.9.1 (we had 1.8.0 up to now in unstable), I also cleanedout things here orthereand I'mquite happy with the result. More testing will of course be appreciated.

The Comprehensive R Archive Network, or CRAN for short, has been a major driver in the success and rapid proliferation of the R statistical language and environment. CRAN currently hosts around 3400 packages, and is growing at a rapid rate. Not too long ago, John Fox gave a keynote lecture at the annual R conference and provided a lot of quantitative insight into R and CRAN---including an estimate of an incredible growth rate of 40% as a near-perfect straight line on a log-log chart! So CRAN does in fact grow exponentially. (His talk morphed into this paper in the R Journal, see figure 3 for this chart.) The success of CRAN is due to a lot of hard work by the CRAN maintainers, lead for many years and still today by Kurt Hornik whose dedication is unparalleled. Even at the current growth rate of several packages a day, all submissions are still rigorously quality-controlled using strong testing features available in the R system. And for all its successes, and without trying to sound ungrateful, there have always been some things missing at CRAN. It has always been difficult to keep a handle on the rapidly growing archive. Task Views for particular fields, edited by volunteers with specific domain knowledge (including yours truly) help somewhat, but still cannot keep up with the flow. What is missing are regular updates on packages. What is also missing is a better review and voting system (and while Hadley Wickham mentored a Google Summer of Code student to write CRANtastic, it seems fair to say that this subproject didn't exactly take off either). Following useR! 2007 in Ames, I decided to do something and noodled over a first design on the drive back to Chicago. A weekend of hacking lead to CRANberries. CRANberries uses existing R functions to learn which packages are available right now, and compares that to data stored in a local SQLite database. This is enough to learn two things: First, which new packages were added since the last run. That is very useful information, and it feeds a website with blog subscriptions (for the technically minded: an RSS feed, at this URL). Second, it can also compare current versions numbers with the most recent stored version number, and thereby learns about updated packages. This too is useful, and also feeds a website and RSS stream (at this URL; there is also a combined one for new and updated packages.) CRANberries writes out little summaries for both new packages (essentially copying what the DESCRIPTION file contains), and a quick diffstat summary for updated packages. A static blog compiler munges this into static html pages which I serve from here, and creates the RSS feed data at the same time. All this has been operating since 2007. Google Reader tells me the the RSS feed averages around 137 posts per week, and has about 160 subscribers. It does feed to Planet R which itself redistributes so it is hard to estimate the absolute number of readers. My weblogs also indicate a steady number of visits to the html versions. The most recent innovation was to add tweeting earlier in 2011 under the @CRANberriesFeed Twitter handle. After all, the best way to address information overload and too many posts in our RSS readers surely is to ... just generate more information and add some Twitter noise. So CRANberries now tweets a message for each new package, and a summary message for each set of new packages (or several if the total length exceeds the 140 character limit). As of today, we have sent 1723 tweets to what are currently 171 subscribers. Tweets for updated packages were added a few months later. Which leads us to today's innovation. One feature which has truly been missing from CRAN was updates about withdrawn packages. Packages can be withdrawn for a number of reasons. Back in the day, CRAN carried so-called bundles carrying packages inside. Examples were VR and gregmisc. Both had long been split into their component packages, making VR and gregmisc part of the set of packages no longer on the top page of CRAN, but only its archive section. Other examples are packages such as Design, which its author Frank Harrell renamed to rms to match to title of the book covering its methodology. And then there are of course package for which the maintainer disappeared, or lost interest, or was unable to keep up with quality requirements imposed by CRAN. All these packages are of course still in the Archive section of CRAN. But how many packages did disappear? Well, compared to the information accumulated by CRANberries over the years, as of today a staggering 282 packages have been withdrawn for various reasons. And at least I would like to know more regularly when this happens, if only so I have a chance to see if the retired package is one the 120+ packages I still look after for Debian (as happened recently with two Rmetrics packages). So starting with the next scheduled run, CRANberries will also report removed packages, in its own subtree of the website and its own RSS feed (which should appear at this URL). I made the required code changes (all of about two dozen lines), and did some light testing. To not overwhelm us all with line noise while we catch up to the current steady state of packages, I have (temporarily) lowered the frequency with which CRANberries is called by cron. I also put a cap on the number of removed packages that are reported in each run. As always with new code, there may be a bug or two but I will try to catch up in due course. I hope this is of interest and use to others. If so, please use the RSS feeds in your RSS readers, and subscribe to the @CRANberriesFeed, And keep using CRAN, and let's all say thanks to Kurt, Stefan, Uwe, and everybody who is working on CRAN (or has been in the past).

The Community Wellness team at MIT has a program on stress reduction, mindfulness, and relaxation. Among their services is a guided three-minute relaxation exercise recording (available at extension 3-2256 or 617-253-CALM). It's a very relaxing message. At the end of the recording, there's a revealing error where a standard voicemail robo-voice say "no messages are waiting" before you system hangs up on you. Turns out, the MIT wellness folks implemented this using the normal MIT voicemail system. This gave me a thought: What if my voicemail greeting included a guided relaxation message as part of its greeting so that anyone who left a message had the chance to relax a little bit first? Would messages left for me be more positive after a window of serenity? Would people ask less of me? Would my callers feel more relaxed and happier during the rest of their day? I just recorded a short version of the MIT message as my voicemail greeting. I suppose I will find out.

You make backups? That s great! But are they usable? The very day you will need your backup to restore your infrastructure, are you sure this backup you make three months ago contains the only file you need to restart your production environment? Are you sure this backup is not corrupted? Are you sure you do not store irrelevant or empty files instead of the ones you will need one day? Brebis is a fully automated backup checker. Brebis is written in Python 3 (>=3.2) and comes under the GPLv3 license. The current version is 0.4 (direct download). Some features (extensive list of features available here):

• Support of tar files, compressed with gzip or bzip2 or not, zip files, gzip or bzip2 files and uncompressed trees of files.
• Identify corrupted archives, unexpected size or type for an archive, associated wrong hash sums, Unix rights and owner users/groups of the archive thanks to a list you provide.
• Inside the archive, identify missing files, unexpected ones, unexpected size of a file, wrong associated hash sums, unexpected mode, type or owner users/groups for a file thanks to a list you provide or generate from a given archive.

I've today released version 0.21 of Obnam, my backup application. This is the first version that also got uploaded to Debian. It is currently waiting for manual processing in the NEW queue. From the NEWS file: USER VISIBLE CHANGES:

• Obnam will now unlock the repository if there's an error during a backup. For the most part, the force-lock operation should now be unnecessary, but it's still there in case it's useful some day.

BUG FIXES:

• Negative timestamps for files now work. Thanks to Jamil Djadala for reporting the bug.
• The documentation for --checkpoint units fixed. Thanks, user weinzwang from IRC.
• The connections to the repository and live data filesystem are now properly closed. This makes benchmark read/write statistics be correct.

I have just released version 0.22 of Obnam, my backup application. Snippet from the NEWS file below. This version fixes a couple of bugs that could be said to be of the brown paper bag variety. USER VISIBLE CHANGES:

• Obnam now reports its current configuration in the log file at startup. This will hopefully remove one round of "did you use the --foo option?" questions between developers and bug reporters.

BUG FIXES:

• The repository is now unlocked on exit only if it is still locked.
• A wrongly caught GeneratorExit is now dealt with properly.
• Keyboard interrupts are logged, so they don't show up as anonymous errors.

CHANGES RELEVANT TO DEVELOPERS ONLY:

• setup.py has been enhanced to work more like the old Makefile did: clean removes more artifacts. Instructions in README have been updated to point at setup.py.
• Compiler warning about _XOPEN_SOURCE re-definition fixed.
• Tests are now again run during a Debian package build.