A couple of days ago, I gave a talk to the Chicago R Users Group which is run ever-so-smoothly by Paul Teetor and Chase Carpenter. The talk provided a brief introduction to Rcpp for R and C++ integration. Slides are now up on my talks / presentations page.

As I slowly read my good friends wishing each other a good trip, telling they got home safely, and the IRC channels form thick drops of a bitter-sweet etheral substance, I cannot help feeling DebConf13 is over For me as well, from the distance. Many friends gave me warm greetings, and without being there, gave me that beautiful feeling of real community that Debian has given me for ten years already, since I met in real-life many of its developers at DebConf3 in Oslo. And yes, I have stated this far too many times I have attended every DebConf since (and worked organizing most of them). This year, over 300 people were gathered in Switzerland to enjoy the always most intense weeks of the year. This year, I was unable to attend due to calendar clashes. Even so, without the stress that organizers have, and thanks to the great work of the always-loved Video Team, I think I was able to be present at more sessions than at in any of the last few years. Oh, and for the readers of this blog who were not there Do you want to follow what was presented? You can download already the videos for all of the recorded presentations (that were, due to the planned coverage and the manageable size of the Video Team, about of the total scheduled sessions). And, as always, I was able to follow many very interesting talks and take part of a couple interesting meetings/BoF sessions. I still have a bit of catchup, partly due to the timezone difference (I was only at one of the sessions during the Swiss morning, at 02:30 local time, the pkg-ruby-extras team BoF). Anyway... Not being there, I surely was an avid consumer of the photos posted in the DebConf13 gallery, and will surely follow it for some more time as some of you upload your pending material. It was clear from the beginning that, no matter what your definition of consensus is, the chosen venue was beautiful. A beautiful place between the lake and the mountains where our sportiest guys had a very good share of morning runs, cycling sessions, competition sports of different types, outright plain fun for attendees of all sizes and all species... But, hey, wait! During a chat in the course of DebConf, a friend told me a bit worried that all this beauty and fun might make our dear and very important sponsors they are paying for a geek vacation, is it so? No, not at all. Not by a long stretch. And just looking at those same galleries makes it clear and obvious. After all, it's widely known that Debian is the operating system for the gurus. Simple: It's impossible to have all those geeks without getting amazing work done, in ways that even seem clich s (this last photo had Joey Hess explaining dpkg format version 3.0 (git) ideas, sketched after waking up at 3AM on the first sketching surface available to him). After all, Debian people are famous for their inclination to use any excuse to open their computers and hack away. We can find Debianers hacking in small spaces and also hacking out in the fields. But this time, people were able to hack indoors while enjoying the nature and hack outdoors under a tree. And, yes, one of the things that makes organizing DebConf worth it is, after eleven months having low-bandwidth meetings over IRC, having the opportunity to plan for the next days face to face, in a relaxed but work-full environment. Anyway, here at home I didn't sit idly just longing over them. How could I? We are just celebrating the Debian Project's 20th anniversary! http://gwolf.org/content/jonathan-host-and-organizer-rancho-electr-nico">Jonathan, a Debian enthusiast, student at my university, and collaborator for several free software-related collectives in Mexico City, invited me to the celebration at Rancho Electr nico (which I recently mentioned in this same blog). While I was unable to stay for the whole celebration, we had a very good time; I talked about some ways on how to contribute to Debian. Although I didn't have much of a presentation prepared for it, I feel it was successful and interesting for the attendees I just hope to start seeing some of them get into any of the ways for helping Debian soon. I also stayed as a listener and ocassional commenter for a talk on the Debian Project's history and goals, and to a presentation on a nifty electronic music programming tool called Supercollider (of course, available in Debian). Now, "regular" life should continue. For some value of "regular".

A couple of days ago, I gave a talk to the Chicago R Users Group which is run ever-so-smoothly by Paul Teetor and Chase Carpenter. The talk provided a brief introduction to Rcpp for R and C++ integration. Slides are now up on my talks / presentations page.

The Taskwarrior Team has just started a blog, kicking it off with a series of articles about development of Taskwarrior itself. Go, team! Almost from the moment I started using Taskwarrior (thanks to Jakub WIlk for an excellent job maintaining this) I knew I had finally found the todo system that I could love. Right away, I started hanging out with the Taskwarrior community at #taskwarrior @ irc.freenode.net and found out what an awesome bunch of people they are, both developers and users alike. I have plunged in with bug reports and feature requests, and am helping get more Taskwarrior-related things into Debian. In NEW right now I have uploaded several of the dependencies needed by my ITP of taskwarrior-web. It s looking like I ll have that finished later this month or early next month. Also, I have uploaded a wheezy backport of Taskwarrior itself (aka task ) which, if all goes well, enters the archive next week.

Earlier, I reported about my problems using an Intel SSD 520 Series 180 GB disk. Friday I was told by IBM that the original disk should be thrown away. And as there no longer was a problem if I bricked the firmware, I decided today to try to install Intel firmware to replace the Lenovo firmware currently on the disk. I searched the Intel site for firmware, and found issdfut_2.0.4.iso (aka Intel SATA Solid-State Drive Firmware Update Tool) which according to the site should contain the latest firmware for SSD disks. I inserted the broken disk in one of my spare laptops and booted the ISO from a USB stick. The disk was recognized, but the program claimed the newest firmware already were installed and refused to insert any Intel firmware. So no change, and the disk is still unable to handle write load. :( I guess the only way to get them working would be if Lenovo releases new firmware. No idea how likely that is. Anyway, just blogging about this test for completeness. I got a working Samsung disk, and see no point in spending more time on the broken disks.

Happy Birthday! Second oldest Linux distribution still going - Slackware is a little older. It's also been the bane of my life / my obsession / fixation and joy for about 15 of those years. I'm pleased and proud to be associated with the Debian Project as a Debian developer: many of my colleagues are at Debconf 13 in Vaumarcus in Switzerland right now but there will be celebrations around the world.

Great distribution: great hardware support on lots of architectures: huge numbers of derivatives - but what makes it is the people, including some of those who have contributed hugely but are sadly no longer with us. Debian is a phenomenal, supportive, family but, as ever, sometimes dysfunctional

Joel Klecker, Chris Rutter, Frans Pop, Adrian von Bidder and all the others - I so wish you'd all been here to see this.

As of today, thanks to the tremendous work of the French localization team in Debian, there are more web pages in French than there are in English, on Debian web site. How can this happen? Indeed, the Debian web site is not always made of pages primarily written in English. There are some French, Spanish, Chinese pages. These are not translated into English because.....there is no translation team for English (the debian-l10n-english mailing list more acts as a review board). So, given that French translators nearly translated ALL English web pages and because there are quite a few French pages, we currently have 5802 web pages in French, while there are only 5801 in English. The French Cabal definitely reached its goal: we have the DPL, we have the web site. Toutes vos possessions nous appartiennent d sormais. I'm now preparing a GR in order to turn French into the official communication language in Debian. PS: merci Thomas Vincent qui a largement contribu ce succ s ces derniers jours !

Somehow last year I missed the fact that I've been around Freenode for 10 years. It is more than 11 years now at this point. Oh my! I understand this is nothing compared to the very old wolves in free software, but bear with me on this one :)

19:25  info
19:25 -NickServ(NickServ@services.)- Information on Damog (account Damog):
19:25 -NickServ(NickServ@services.)- Registered : Jun 26 23:45:11 2002 (11 years, 7 weeks, 5 days, 17:39:50 ago)
19:25 -NickServ(NickServ@services.)- Last addr  : ~dm@debian/developer/damog
19:25 -NickServ(NickServ@services.)- vHost      : debian/developer/damog
19:25 -NickServ(NickServ@services.)- Last seen  : now
19:25 -NickServ(NickServ@services.)- Logins from: damog
19:25 -NickServ(NickServ@services.)- Nicks      : Damog
19:25 -NickServ(NickServ@services.)- Email      : damog@damog.net
19:25 -NickServ(NickServ@services.)- *** End of Info ***

Laurent Bigonville reported Debian bug #720000 on Saturday August 17th 2013, against the drizzle package. This bug is already marked pending by Tobias Frost, the package maintainer. Bug #710000 was reported as of May 27th: 2 months and 21 days for 10,000 bugs. For once, this is a rate acceleration which we can probably explain by the release of wheezy and the work strongly resumed by many maintainers for the release of jessie. It is indeed interesting to see that this 720000th bug report happened nearly on Debian's 20th birthday. To make it short, we could then say that Debian had 36,000 bug reports every year in average (which is not exactly true as the BTS records start in 1996). Funnily also, this is the first time since I'm doing this recurrent post every 10,000 bugs that one happens *during* a DebConf, a few hours before DebConf 13 officially ends up.

This weekend I'm mostly alternating between reading, writing, and trying to avoid death by the plague.[*] I've switched Lumail to using a UTF-8 aware string library, which means we can now handle the obvious case:

--
-- Prove keybindings work.
--
keymap['global'][' '] = 'msg("UTF-8 rocks!")'

Similarly we can stuff input into the buffer:

--
-- Pretend the user typed ":msg ...\n"
--
stuff( ":msg('  is pie')\n" );

This transition was annoying to handle, but wasn't too difficult. There is only one more major update required, according to the development roadmap, which is to double check that UTF-8 output is correct. Otherwise I think I'm almost done. In the sense that I don't see anything obvious missing, barring things that won't ever happen such as mutt-style "tag" support. I've updated the online examples to include some nice code:

• A nice menu for viewing/opening attachments: view_attachments().
• A nice system for marking messages as read based upon their subjects: mark_read_by_pattern().

I can't claim to have many users, so far the development has been carried out by myself and approximately four other people. But that matters not. I genuinely believe this is a good client and it really suits the way that I handle (large volumes of) email:

• Show folders with unread mail.
• Quickly read it.

Allowing you to open multiple folders at once means you get a great view into your currently-unread mail, regardless of where procmail has placed it. The overriding feeling having "completed" the client is that Lua rocks. I'm torn between wanting to sleep some more, and wondering what other system/package/tool can be extended by Lua. As epiphanies go my on_idle() update takes some beating.

* - I do not have the black death, but I'm not well.

I write a fair number of Unix command line programs, and I want to have test suites for them. A year and a half ago I wrote a tool to make that easier: cmdtest. However, it turned out to not be a particularly good one, primarily because it doesn't make it easy to write good test code. Tests and test code should be obviously correct, so you don't need a test suite for your test suite. Earlier this year, Daniel and I started a slow, but long discussion about this. We'd talk it over during many a lunch, figuring out what we actually want. We eventually converged on an approach based on for Unix command line programs, and based on Markdown documents. We then wrote that up as a README, and then took another month refining the README before writing any code. The naming discussion only took a week.

• We borrowed the given/when/then structure from BDD, and the idea of using that structure so that tests are specified using lightly structured text in a way that allowsnon-developers to participate.
• This borrowed structure gives a way to express sequences of things that happen, including setup and checks. We call these sequences test scenarios, and thus this approach to testing is called scenario testing.
• We embed the test scenarios, and the code that implements them, as code blocks in Markdown documents. This makes it easy to explain what is going on in the tests, when that's needed, right next to the tests. The Markdown files can easily be formatted as web pages or PDFs (among other formats), making it easy to share and discuss them with people who might not be comfortable accessing a git repository.

As a concrete example, here is a real test scenario, from Obnam, for testing its new FUSE plugin:

SCENARIO Browsing backups with FUSE plugin
ASSUMING user is in group fuse
GIVEN a live data directory
AND a 0 byte file called empty
AND a 1 byte file called one
AND a 4096 byte file called 4k
AND a 10485760 byte file called 10meg
AND a hardlink to 4k called 4k-hardlink
AND a symlink to 4k called 4k-symlink
AND a directory called some-dir
AND a symlink to ../4k called some-dir/other-symlink
WHEN live data is backed up
AND repository is fuse mounted
THEN latest generation can be copied correctly from fuse mount
FINALLY unmount repository

Some explanations:

• SCENARIO is the name of the scenario.
• ASSUMING is a condition for the entire scenario: the FUSE plugin can only be tested when the user can use FUSE.
• GIVEN is a setup step, WHEN is the step that does the thing that is being tested, and THEN checks that the right thing happened during WHEN.
• AND is an alias for the keyword on the previous line; it makes the test scenario seem more like normal English text
• FINALLY is for cleaning up afterwards, whether the scenario passed or not

Test scenarios like this are obviously not directly executable. We need to provide the code that actually executs each step. For this, yarn provides an IMPLEMENTS section, of which I will show one example: the creation of the files.

IMPLEMENTS GIVEN a (\d+) byte file called (\S+)
dd if=/dev/zero of="$DATADIR/live/$MATCH_2" bs=1 count="$MATCH_1"

(The other IMPLEMENTS sections for that Obnam test scenario are also only about a line of shell. The longest one is four lines, because it has an if statement that I like to format in a fairly verbose way. I'm skipping the others to avoid needless repetition.) The text after the keyword (GIVEN, WHEN, THEN, etc) in the test scenario is a single line of free-form text, which is matched by a regular expression in the IMPLEMENTS section. There must be exactly one matching IMPLEMENTS for each line in a test scenario. The body of IMPLEMENTS is a shell script snippet. Yarn provides a temporary directory for each scenario, and the path to that is given in the DATADIR environment variable. Yarn also cleans up the directory afterwards, unless the user running yarn requests otherwise, so tests that only change things in the temporary directory don't need any FINALLY steps. The above example uses FINALLY to unmount the FUSE filesystem at the end. The regular expressions can capture parts of the text in the scenario step, and these are provided to the shell script using environment variables: MATCH_1 etc. This is a simple, but powerful mechanism for implementing very flexible steps, which also means the number of steps can be kept down. The IMPLEMENTS sections are shared between all test scenarios. This makes it very easy to write new scenarios using already-existing steps, by choosing which ones to use, in which order, and making use of any variability in the steps that their implementations provide. Yarn is a new tool, and I expect it to mature in the coming months. It's included in the cmdtest Debian package, and is already in Debian jessie and unstable, for those who want to try it. (My personal apt repository has a .deb for wheezy as well; backports are welcome). The yarn scenario testing language itself is not set in stone, but Daniel and I both want to be careful about making changes to it. It's all meant to be very simple: an obviously correct testing tool is better than an intricate one. I've used yarn in several of my own little programs now, as has Daniel, and we've used it at work as well. Undoubtedly we'll find bugs in the implementation, and we're still feeling our way to figuring out how to write really excellent scenarios, and how to structure steps with the best possible taste. We'll also start designing a shell library that yarn provides to the IMPLEMENTS sections, for doing things that we find ourselves doing in many projects. It is already clear to me, however, that yarn is oh so much better than cmdtest could ever be. I'm even starting to express ideas for new tools as yarn test documents, so I'm clear what they should do. If and when I ever get to actually implementing the tools, I'll have a test suite for them already. See, for example, first attempts at specifying how a fast file transfer tool, fling, which is to use Linux-only system calls to reach wire speeds over wired LAN, and a tool to recompress compressed files with another compression tool. We hope yarn will prove to be useful for others as well. If you try it, and have suggestions or comments, please e-mail us.

Due to the extremely righteous contributions of Leo and Sylvestre, we ve opted to make the name more generic (and hilarious), to make sure people understand that this will be a more general purpose tool. Cutting edge code is currently living at debile.debian.net - my debuild.me instance will soon follow when I ve got a chance to sit down with it :) Wish I was at DebConf, have a great DC, everyone!

WebAuth is the site-wide centralized web authentication system that we use at Stanford. This release is a bug-fix release for the central login server, mostly focused on multifactor authentication but also featuring a wide variety of random bug fixes. Going into this release, I filtered the Apache error logs on all of our production servers for a period of a month and then set out to fix every log anomaly or odd behavior pattern that I detected. The most significant bug fix is to support for requiring non-password session factors. The code handled this entirely incorrectly, resulting in the user being stuck at the password authentication prompt and never able to proceed. It also cleans up several corner cases involving REMOTE_USER authentication to sites that require a Kerberos authenticator (such as for delegated credentials). This release downgrades a mismatch of login credentials and a webkdc-proxy token (single sign-on credentials) to a log warning rather than a fatal error and just discards the non-matching single sign-on credentials. This case ideally shouldn't happen, but sometimes users share devices, and if they hit a site with forced login or session factors, it's easy for this case to arise. WebLogin now retries the WebKDC POST once and ignores SIGPIPE errors to work around problems with signals from the FastCGI process manager. It also rewrites non-ASCII characters in XML from the WebKDC to work around XML validation errors created by non-ASCII characters in Kerberos error messages, pending a better fix on the WebKDC side. Password changes no longer go through the replay detection logic, which should not apply to them. Several error messages in WebLogin are now clearer. The WebKDC now requires that the return URL in a WebAuth request be an absolute URL and not contain any non-ASCII characters. A bug in the way that errors during the authentication process were reported has been fixed and should produce better error output for the user. There are also a few other, more minor bug fixes, particularly to logging and error reporting. You can get the latest release from the official WebAuth distribution site or from my WebAuth distribution pages.

The OASIS website has not been updated since a while. So I decide to take a shot at making more up to date. This blog post is about the pipeline I have but in place to automatically update the website. It is the first end to end 'continuous deployment' project I have achieved. Among the user visible changes:

• an invitation to circle the OASIS G+ page, which is now the official channel for small updates in OASIS.
• an invitation to fork the project on Github since it is now the official repository for OASIS.
• some link to documentation for the bleeding edge version of OASIS manual and API.

The OASIS website repository is also on Github. Feel free to fork it and send me pull request if you see any mistake. The website is still using a lot of markdown processed by pandoc. But there are some new technical features:

• no more index.php, we use a templating system to point to latest version
• we use a Jenkins job to generate the website daily or after any update in OASIS itself.

Since I start using quite quite a lot Python, I have decided to use it for this project. It has a lot of nice libraries and it helps me to quickly do something about the website (and provides plenty of idea to create equivalent tools in OCaml). The daily generation: Jenkins I have a Jenkins instance running, so I decided to use it to compile once a day the new website with updated documentation and links. This Jenkins instance also monitor changes of the OASIS source code. So I can do something even more precise: regenerate the website after every OASIS changes. I use the Jenkins instance to also generate a documentation tarball for OASIS manual and API. This helps a lot to be able to display the latest manual and API documentation. This way I can quickly browse the documentation and spot errors early. Another good point about Jenkins, is that it allows to store SSH credential. So I created a build user, with its own SSH key, in the OCaml Forge and I use it to publish the website at the end of the build. Right now Jenkins do the following:

• trigger a build of the OASIS website:
  • every day (cron)
  • when a push in OASIS website repository is detected
  • when a successful build of OASIS is achieved.
• get documentation artifact from the latest successful build of OASIS
• build the website
• publish it

Data gathering To build the website I need some data:

• documentation tarballs containing the API (HTML from OCamldoc) and manual (Markdow)
• list of OASIS version published
• links to each tarball (documentation and source tarball)

The OCaml Forge has a nice SOAP API. But one need to be logged in to access it. This is unfortunate, because I just want to access public data. The only way I found to gather my data was to scrape the OCaml Forge. Python has a very nice scraping library for that: beautifulsoup. I use beautifulsoup to parse the HTML downloaded from the Files tab of the OASIS project and extract all the relevant information. I use curl to download the documentation tarball (for released versions) and for the latest development version. Code Template Python has also a very nice library to process template: mako. Using the data I have gathered, I feed them to mako and I process all the .tmpl files in the repository to create matching files. Among the thing that I have transformed into template:

• the index.php has been transformed into a index.mkd.tmpl, it was a hackish PHP script scraping the RSS of updates before, it is now a clean template.
• robots.txt.tmpl, see the following section for explanation
• documentation.mkd.tmpl in order to list all version of documentation.

Fix documentation and indexing One of the problem of providing access to all versions of the documentation, is that people can end up reading an old version of the documentation. In order to prevent that, I use two different techniques:

• prevent search engine to index old version.
• warn the user that he is reading an old version.

To prevent search engine to index the file, I have created a robots.txt that list all URL of old documentation. This should be enough to prevent search engine to index the wrong page. To warn the user that he is reading the wrong version, I have added a box "you are not viewing the latest version". This part was tricky but beautifulsoup v4 provide a nice API to edit HTML in place. I just have to find the right CSS selector to define the position where I want to insert my warning box. Code Publish The ultimate goal of the project is the 'continuous deployment'. Rather than picking what version to deploy and do the process by hand, I let Jenkins deploy every version of it. Deploying the website used to be a simple rsync. But for this project I decided to use a fancier method. I spend a few hours deciding what was the best framework to do the automatic deployment. There are two main frameworks around: capistrano (Ruby) and fabric (Python). Fabric is written in Python, so I pick this one because it was a good fit for the project. Fabric biggest feature is to be a SSH wrapper. The fabric script is quite simple and to understand it, you just have to know that local run a local command and run run a command on the target host. The fabfile.py script do the following:

• create a local tarball using the OASIS website html/ directory
• upload the tarball to ssh.ocamlcore.org
• uncompress it and replace the htdocs/ directory of the oasis project
• remove the oldest tarballs, we keep a few versions to be able to perform a rollback.

Given this new process, the website is updated in 3 minutes automatically after a successful build of OASIS.

I have a recurring problem when trying to use augeas on a complex node: trying to edit a specific entry in a list which is uniquely defined by many attributes. You probably don't know that you have this problem, but it is easy to spot it into your augeas/puppet resource. Here are some symptoms of this problem:

• you need to use onlyif with multiple constraint on the selection
• you use last() and last() +1

 
 augeas  
   "setup-shorewall":
     changes =>
       [
         "set entry[last() + 1]/source 'all'",
         "set entry[last()]/dest 'all'",
         "set entry[last()]/policy 'REJECT'",
         "set entry[last()]/log_level 'info'",
       ],
     onlyif => "match entry[source = 'all'][dest = 'all'][policy = 'REJECT'] size == 0";

For a long time, I thought it was the only solution. But last week, I read again the documentation and found another solution. My main concerns are the onlyif and last() parts, it doesn't look clean to me. The problem is that I cannot define the entry all at once and if I use a value that will be set late, the node cannot be targeted in between. The clean way to do this was to define first the target attribute. Typically, in augeas changes:

 set spec[user = '$name']/user '$name'

This way if the node doesn't exist it is created and you can then use it directly:

 set spec[user = '$name']/host_group/host 'ALL'
 set spec[user = '$name']/host_group/command1 'ALL'
 set spec[user = '$name']/host_group/command1/tag 'PASSWD'

But sometimes it is not possible to set the attribute directly -- typically when you need to use multiple attribute. The solution in this case is to use defnode:

 defnode target entry[#comment = 'puppet: <%= name %>']/ "<%= name %>"
 set $target/action '<%= action %>'
 set $target/source '<%= source %>'
 set $target/#comment 'puppet: <%= name %>'
 clear $target

The big trick here is that defnode needs a value, but most of the time you cannot set a value for the node -- because it has none. To solve this, you set a value with defnode, process with your change and you clear the node at the end. This recent discovery has simplify a lot some augeas changes I use. Feel free to leave comment on your personal technique to deal with augeas and puppet.

Happy twentieth birthday, Debian! For the celebration at this year's debconf I wrote this litte haiku. I hope you like it!

Debian OS
Twenty years, still relevant
This is what we are The base of many
But without our great work
They could not exist It is an honor
To be reused in that way
Be proud of your work Enjoy the evening
Celebrate with us all night
There be poetry

Hope you had a nice celebration, I truly enjoyed this year's Poetry Night.
Enjoy!

/debian permanent link Comments: 0 Flattr this

RichiH used flint and steel to light the bonfire. We carefully fed it up from those sparks to a blaze. Put on the biggest logs we could find to make it last. Now I'm sitting on the hill above it watching folks gathered around. A poem is being read, in Hindi, then translated for us to English. Others have shared songs and poems in a dozen languages, both classics and their own. Out below the darkness of the lake. This is Debian at 20. At the start of this DebConf, I gave a talk on "Debian Cosmology. In that (and a later "dh_busfactor" talk) I shared my hopes and my fears. I was conflicted about giving the this talk, worked on it for weeks, felt it might not work, or be depressing. I've had nothing but good comments about it. 20 years is ages in internet time and technical projects ossify over time. The last session I was in this afternoon was a presentation of a new tool, which I hope & feel has the potential to fundamentally change an important and suboptimal part of Debian. Then I walked outside to a rainbow over crystal clear Swiss alps on the horizon. How encouraging, and what a nice story that will be around some future campfire.