(this is a post in two parts) At the BoF of GPG Key Management: Best Practices, dkg had two immediate recommendations for improving WoT resilience:

• Regularly refresh your local keyring from the key servers;
• If you have spare space and transit, provide a key server to take part in the pool.

Regular Refreshes This is important because unless keys are refreshed, changed expirations and (more importantly) revocations on keys are never made known to you. Making regular refreshes is easy, though I was surprised to hear people had experienced difficulty with it it s a simple entry to cron, like: 0 0 * * 1 /usr/bin/gpg --refresh-keys to run at midnight on a Sunday, or any other convenient time. Append > /dev/null to make it silent unless there are errors. Provide a key server This is nearly as trivial a refreshing the keyring, but it does take some disk space (on my machine, 4G for the decompressed dump from another server, 7.4G for the database itself and a few hundred MB for other stuff. Supposedly the dump can go after a full database build, but as I m not constrained I haven t bothered). The sks package in Lenny and Squeeze manages the database and answers to clients. However, take care to follow README.Debian carefully, since if the permissions on /var/lib/sks or /var/log/sks aren t set right it will silently fail and much head-scratching will ensue. Building the database for the first time takes some hours even on a reasonably quick machine, and takes a lot of CPU time, so it s probably best done overnight. Once sks is running, it needs to catch up from the date of the key dump to now (which is why a recent dump is always a good start). To do that, it should be listed in the pool.sks-keyservers.net and therefore needs gossip peers. http://code.google.com/p/sks-keyserver/wiki/Peering is an excellent guide, though the version of sks in Lenny does not support multiple listening addresses for either the recon or hkp processes. When a small enough delta between the server and its peers is reached, it s included in the DNS pool for use by other clients.

A DebConf doesn't just happen in a few months. It's not even a year, it's closer to two or two and a half. I've only been through two cycles, and I think this process has happened for the last three or four DebConfs. This post will talk about a "brief" summary of the general DebConf timeline, with elaboration on all the bits later. This post describes the preparation of DebConfN in year 20XX.
At DebConfN-2 (mid 20XX-2), there is a talk where interested groups can describe their location and how it would be suitable for a DebConf. This consists of a short presentation and discussion, but not many details by this point. No decision is made here.Throughout late 20XX-2, teams develop their bid as best they can. The last two years there has been a nominal deadline of December 31 20XX-2 for submitting a bid, but like all DebConf deadlines it's more for guidance than a real deadline. There are various checklists and templates to provide guidance here, which I'll describe later.Early 20XX-1 brings a lot of work to finalize bids, and then the actual selection process. There is a burst of mailing list traffic where people ask questions and help each other refine their bids. There are several meetings where bid teams give their status and people can ask questions. In February or March, there is a selection meeting.After the decision meeting, there is a big round of cheers but people soon refocus on DebConf N-1 since it is more urgent. Ideally, most of the people involved in DebConfN help with DebConfN-1 both because DebConfN-1 needs help, but also to learn how to make DebConfN go well. The DebConfN local team keeps working in the background doing whatever is needed to prepare for their conference.At DebConfN-1, there is a talk where the DebConfN team presents their progress to the DebConf attendees and tries to build support.Late 20XX-1 brings the end of the previous conference and time for N's localteam to really start getting stuff done. Throughout the last part of the year, we try to work out enough details for registration to begin: final conference dates, locations, accommodation, etc.Registration opens in early 20XX. The exact dates for opening/sponsorship deadline/reconfirmation/etc all depend on how early in the year the conference will be. There are nominal deadlines published, but they usually slip. With our limited volunteer base, it's hard to make things go according to schedule, but things always work out.Around this time, the fundraising team really starts working hard. Also, the travel sponsorship team needs to meet to decide who gets funded. The talks team needs to issue a call for events, read, and select them. The video team needs to make sure that they will be ready for the DebCamp setup period. There are many more things I'm forgetting, too.In the months before arrival, it becomes kind of crazy from a local perspective. There are many things to get organized before people arrive, mainly focusing on food, accommodation, and venue space logistics.Once DebCamp starts, things get even more complicated, but we have a flood of experienced DebConfers from around the world come in, and they all know a bit and can help out. DebCamp is effectively the DebConf preparation week to the organizers. DebCamp becomes DebConf, and much bigger, once all attendees arrive.Eventually, people leave and everyone goes home. This doesn't mean local stuff is over - I'm still engaged in DebConf10 wrap-up, making sure everyone gets paid and everything is returned.Mid to late 20XX brings a brief period of rest for most people.There is one last thing to do, though: the DebConf final report. This describes the events of DebConf, and is very useful for getting sponsors. It is also a valuable reference for future years looking back on the lessons learned. You can browse past final reports in our archive . Because everyone is tired, this too frequently gets pushed off until early 20XX+1, when DebConfN+1's fundraising team needs it. A frantic rush finishes up the missing pieces and it's done.
So, there you have it: a rough outline of the timeline of a DebConf. It extends from mid 20XX-2 to early 20XX+1, more than two and a half years. This post nowhere near describes all the details that go into DebConf, almost any of the paragraphs above could be made it's own post. Stay tuned for them.Tags: debconf, debian

At DebConf10, there was a BOF entitled "DebConf & Debian", discussing about the similarities and differences of Debian and DebConf. At times this became a little bit heated as people wanted more integration and transparency.I don't think the differences are that great. I don't even see them as separate. I see DebConf as a Debian team, one that has to release every year on a short time frame. Things are rushed, things are not perfect, but we welcome anyone who wants to help.It's my goal to start a series of posts on DebConf, in the spirit of other teams documenting themselves and calling for contributors. I hope to let people know what's the current state of things, how you can get involved.If you want to get involved, join #debconf-team on OFTC, or subscribe to the debconf-team mailing list at http://lists.debconf.org/ . There are many people who would like to help you get involved. Right now (post-debconf) is a bit of an idle time, but one of the few times that big changes can be made. DebConf is a do-ocracy, if you'd like there to be a change, the best way is to help implement it. There may be some resistance, but we (at least I) will help to find a solution.These posts are my take on things, I can't claim to speak for the DebConf team. I don't know all of the history of DebConf, and I may get things wrong. I'll run these posts by the rest of the DebConf team for suggestions/corrections, but there may still be things wrong. You should consider anything useful to be thanks to the DebConf team, and anything wrong to be my mistake.I've only been involved in DebConf for two years. I first got involved as part of DebConf10, but spent most of 2009 working on global management things for DebConf9. This year, I spent more time with local on-the-ground tasks. I'd like to keep being involved next year - and hopefully keep improving, with your help.Tags: debconf, debian

I have some mp3 files in my collection. Some years ago, back in the times, when I was a Windows-User, I used "The Godfather", to keep the chaos under control. This software, although not Open Source, is really good at what it does. And it does almost everything to organize mp3s. From (mass-)tagging to auto-renaming to auto-sorting your mp3s. But this wouldn't be a blog entry by me if this became... a praise to a windows software so..

When I became a daily Linux User I searched for alternatives. With GUI or without a GUI, but I did not really find an application that suited my needs. As far as tagging and renaming was concerned it wasn't that hard. There are really excellent command line tools (lltag, id3, etc.) and rumours have been heard that there are also GUI tools.
But what I still did not find is a simple, yet flexible, tool to sort a huge collection of mp3s into a flexible structure on the filesystem like "The Godfather" was able to do since.. ehh.. a lot of years.

So the day before yesterday I finally decided to write one on my own and came up with an ~ 280 lines perl script (POD-documentation included) which does exactly what I want and is simple.
It does no tagging.
It does no renaming.
All it does is sorting mp3s into a given template-based directory hierarchy based on their ID3 tag.

At this point a warning is due. I'm sharing that script with you under the terms of the GPL. BUT it still needs testing and therefore you probably do not want to use it without its safety measures (e.g. dry-run or copy instead of move) to avoid data loss. And notice that although it has been written with portability in mind, it has only been tested on a Debian GNU/Linux system.

The script is hosted at github. Here (or raw to download it directly).

After I've talked to a co-worker about the script, he told me that arename would do what I want. So you probably don't want to use my script, because arename is probably more tested and much more sophisticated. But on the other hand I had a quick look at the arename manpage and its so utterly feature-loaded, that it cannot avoid a certain complexicity. My tool is simple.
And there is another advantage. If you want arename to handle a certain amount of mp3s you have to use your shell magic to find the files and pipe it to arename. My script finds all mp3s recursively from where you let it start (default is $PWD so be careful) and will happily move it into a hierarchy under a directory you ask it to. For the simple job of sorting mp3s its probably easier to use.

(Oh and if its worth nothing, it might still be of use as a simple programming example, how one could solve this problem in Perl)

I ve decided to try using local DNS caching again, using the simple dnsmasq. The problem is that, even including

prepend domain-name-servers 127.0.0.1;

in /etc/dhcp3/dhclient.conf, NetworkManager will overwrite /etc/resolv.conf, and remove

nameserver 127.0.0.1

from there. I found out in a comment here, that I could create a script in /etc/NetworkManager/dispatcher.d/ to make this line be included in the file. The script is simple

#!/bin/sh
set -e
cp /etc/resolv.conf /etc/resolv.conf.tmp
echo nameserver 127.0.0.1 > /etc/resolv.conf
cat /etc/resolv.conf.tmp >> /etc/resolv.conf
rm /etc/resolv.conf.tmp

and it s working .

At some point I had to debug an issue with some code I worked on in the past. It was using OpenLDAP to connect to an Active Directory server to get some information. At some point I got a report that the authentication failed with an ugly error.

Username is stored. Authenticating as domain\user.
Enter password:
ldap_search_ext: Operations error (1)
additional info: 00000000: LdapErr: DSID-0C090627, comment: In order to perform this operation a successful bind must be completed on the connection., data 0, vece

I looked for the meaning of that error and after some staring at different possible explanations, I ended up on a page dealing with some python-ldap code. My code was in C, but it was clear it was the underlying library issuing the error message, so I looked at what it said:

Normally, this error indicates that you're attempting to bind anonymously, which Active Directory (sensibly) doesn't allow by default. We were supplying credentials to bind, though, and changing the base DN on the search to a sub-OU was all that was necessary to get the search to work. It turns out that python-ldap was binding anonymously, so the error was only sort of a red herring.

This was really strange because the authentication was actually done, as it was obvious from the messages and the traffic (analyzed with wireshark). Later in that post there were some hints that indicated that parts of the data might be stored on another server and the suggested fix was to instruct the library not to try chase referrals.

ldap.set_option(ldap.OPT_REFERRALS, 0)

I tried to see what was going on with our server using a ldapsearch command and at the end of the output there were some referrals specified.

[..]
# search reference
# refldap://ForestDnsZones.domeniu.ro/DC=ForestDnsZones,DC=domeniu,DC=ro

# search reference
# refldap://DomainDnsZones.domeniu.ro/DC=DomainDnsZones,DC=domeniu,DC=ro

# search reference
# refldap://domeniu.ro/CN=Configuration,DC=domeniu,DC=ro

# search result

# numResponses: 5
# numEntries: 1
# numReferences: 3

Bingo! So, after looking at the man page, I added this bit of code:

+ /* do not chase referrals */
+ if (ldap_set_option (ld,LDAP_OPT_REFERRALS,LDAP_OPT_OFF)!=LDAP_SUCCESS)  
+  ldap_perror(ld,"ldap_set_option");
+  return NULL;
+  
+

And then it worked. I hope this helps others that might be in the same situation as I was.

A while back I mentioned that I was going to be updating and overhauling the blogspam.net service. That process is now almost complete. A couple of nights ago I overhauled the website, and today I've finally commited my last (planned) change to the repository for the purposes of migration. I started reworking the code a week or so ago, but as of this evening the code in the repository is the code the server is actually running. The previous codebase was functional but a little hasty - and was implemented before I switched to per-UID server-hosting - so there was a need to clean things up and make sure permissions and similar niggles were checked. The new, modular, codebase requires no root access, and will store all state (logs & transient caches) in a clean extensible fasion. The code is also much more flexible making use of Module::Pluggable rather than Class::Pluggable. This allowed me to overhaul the API of the plugins (primarily to add an expire method such that each plugin has a well-defined means to expire any state they may maintain). Module::Pluggable is a great module - allows me to treat plugins as first class objects, which wasn't the case with C::P. Since all the code behind the service is Perl it is also now available on CPAN in addition to the mercurial repository where it is developed.. I see that the server is getting pretty popular these days, used by the likes of embedders.org, publiclive.com, & etc. It doesn't hurt that ikiwiki, identi.ca, and other people include support in their distributions these days. Me? I mostly use it on debian-administration.org where it does a great job. ObQuote: What's the name of that thing that if I eat it real fast, it's free? - Whip It.

Finally I m back from DebConf10 in New York. It was the first DebConf I attended and it was just great. Thanks a lot to all involved people for making DebConf such a great event. I had a big todo list for DebCamp and DebConf and managed to get a bunch of work done. Besides several discussions, bug hunting, testing stuff and attending BoFs and talks the work that I could get done for Debian included:

• released FAI 3.4.0
• released initramfs-tools 0.98
• sponsored upload of:
  • pyneighborhood
  • md5deep
  • grokevt
  • kpicosim
  • shunit2
  • mpris-remote
• uploaded new version of stressapptest (required some build tuning)
• prepared new upload of tct
• fixed RC bugs:
  • #587557 problems installing plymouth
  • #591257 stressapptest: FTBFS on sparc: configure: error: sparc is not supported!
  • #589836 mdadm: breaks initramfs on fresh (chroot) installation

My talk about "State of Debian (based) Linux live systems in 2010" went pretty well according to the feedback I got (thanks for that), even though OpenOffice failed horrible once again (one completely broken line on one slide, several broken gradients on several slides in presentation mode, the presenter screen just didn t work at all, ). I just uploaded the slides of my talk (8.4MB, PDF), though they won t be useful without the talk but thanks to the awesome videoteam my talk is available as recording. On the next weekend (August 21st/22nd) you can meet me at FrOSCon/Germany where I ll be at the Grml booth and giving a talk titled "Gute Open-Source-Projekte bestehen aus mehr als nur Code" (english: Good open source projects are more than just code). BTW: What s a wiki? Say whaaaaaat?</insider>

Today's fun idea was to put all my contacts stored into BBDB on a Google Maps' map, using my Google Maps extension for Emacs. With the help of a few lines of Lisp glue:

(google-maps-static-show
 :markers
 (mapcar
  (lambda (address-entry)
     ((,(concat
         (mapconcat
          'identity
          (elt address-entry 1) ", ") ", "
          (elt address-entry 2) ", "
          (elt address-entry 3) ", "
          (elt address-entry 4) ", "
          (elt address-entry 5)))))
  (mapcan
   (lambda (record)
     ;; We need to copy the returned list, because mapcan will modify it later
     (copy-list (bbdb-record-addresses record)))
   (bbdb-records))))

we can make that:

It's really simplistic, but I did not need more to have fun. :-) This could be extended to set a specific marker and/or color for each contact, with a legend. I'll let that as an exercise for my readers.

I've been talking about replacing my laptop since November. Now I've somewhat forced the issue by giving away my R200, leaving myself with just the EEE 901. I'm not planning to bring a desktop with me to California (well, I might ship bits for a media box, but probably not even a complete machine), so I've been looking more seriously at what my options are.

I like smaller laptops; the R200 is 12", I had a Compaq N200 previous which was 10". There seem to be a number of 13" options that are light, so I'm prepared to look at those.

This is going to be my main machine, so I need something with some grunt. A decent amount of RAM with a reasonable processor. I'm probably prepared to take the price hit in order to get SSD. Also a decent resolution screen along with built in wifi (does anything not have this these days?) + 3G would be nice too.

Of course, it turns out I can't have all of this. In particular the US market seems pretty dire for 3G support (no, something locked or that can't take a GSM SIM doesn't count. I expect to travel.) The UK market is a bit better, but there are still no perfect answers.

Lenovo X201s

(It's really hard for me to think of this as a Lenovo rather than an IBM Thinkpad.)

This comes in a 1440x900 resolution variant, which is nice for 12". I can have a touchpad too. What I can't have is 3G; with the 1440x900 screen there's apparently no space for the 3G antenna. Doh! Also Thinkpads are *ugly*.

Sony Vaio VPCZ12Z9E/X

I want this. Core i7, 1920x1080 screen, SSD, 3G (Gobi), sub 1.5Kg.

But. Sony. Ridiculously expensive. If it was half the price and made by Lenovo or Toshiba I'd have bought one by now. As it is there's no way I can reasonably expect it to last 3 years (and really I'd want 5 given the cost).

Toshiba R700-155

I've been extremely happy with my R200 and spent a long time eying the R500/R600. The -155 is a Core i7, 1366x768 screen, 13", 3G, sub 1.5Kg.

2 problems. One, I'd like a better resolution if I'm going to 13" - both the Lenovo and the Sony manage that. Secondly, and more of an issue, nowhere seems to have them in stock.

HP Elitebook 2540p

This has potential. Seems to be a little heavier than the other options, and it's only a 1280x800 screen (but in 12"). Available with Core i7, SSD & 3G. Oh, except once again stock is a problem.

Have I missed anything else out there? I can probably buy from the UK or US with equal ease.

I have just read The Gift of Fear and Other Survival Signals that Protect Us From Violence by Gavin de Becker. Like many self-help books it has a concept that can be described in a paragraph and explained in a few pages. The rest of the book shares anecdotes that help the reader understand the concept, but which are also interesting for people who get it from the first chapter. When I read the book I considered the majority of the content to be interesting stuff added to pad it out to book size because the concept seemed easy enough to get from the start, but from reading some of the reviews I get the impression that 375 pages of supporting material aren t enough to convince some people maybe this is something that you will either understand from the first few chapters or never understand at all. Gavin s writing is captivating, he has written a book about real violent crime in a style that is more readable than many detective novels, from the moment I finished the first chapter I spent all my spare time reading it. I was a little disappointed at the lack of detailed statistics, but when someone has done all the statistical analysis chooses to provide the results in the form of anecdotes rather than statistics I m prepared to tolerate that especially when the anecdotes are so interesting. I spent quite a bit of time reading the Wikipedia pages relating to some of the people and incidents that are mentioned in this book. The basic concepts of his book are to cease worrying about silly things like airline terrorists (passengers won t surrender now so that s not going to work again) and to instead take note of any real fear. For example if you are doing the same things you usually do but suddenly feel afraid then you should carefully consider what you might have subconsciously noticed that makes you feel afraid and what you can do about it. This isn t going to change my behavior much as I have mostly been doing what the book recommends for a long time. I think that everyone should read this book.

Francois Freeing the Cloud talk at Debconf10, showed members of the Debian community keen to help see opensource Web applications. The proposed open&free implementation of Gravatar is a great example, whereby a user uploads an image of him/herself which corresponds to a hash of their email address. In order to make this Web "cloud" service rock, super quick downloads of that image are needed. Debian knows about mirroring data to distribute data amongst free software agents. And we also now know about CDN, aka content delivery networks and GEO DNS, which point to your local mirror. So lets pool our resources and offer CDN Web space (100MB?) to free software Web application projects. This will make the opensource Web services competitive in all important page load times and perhaps make it easier for user contributed data to be exported and "open". I have Debian server space in Germany I'm happy to contribute to Francois's avatar project and other similar opensource Web apps. If you think this is a good idea, please get in touch.

Following up on my Flattr FOSS post, the most interesting thing about having my projects on Flattr has been watching their relative popularities in the control panel. It looks something like this, which is done using javascript, so you might need to follow the link to my blog to see it properly.

• dpkg-repack
• debhelper
• etckeeper
• mpdtoys
• mr
• alien
• pristine-tar
• moreutils
• Joey Hess's blog
• ikiwiki

I've been watching debhelper and moreutils keep pace with each other. This is interesting, because moreutils has been highlighted by Flattr FOSS, and debhelper has not. Still, the general level of interest in moreutils continues to suprise me -- and some more utils are lined up to go into it soon! Popcon tells me that alien still has more users than anything except for debhelper; easily 5 to 10 times the users of moreutils or pristine-tar. So alien's low numbers in Flattr are also surprising. I think it might be explained by more developer types than user types following Flattr FOSS right now. Perhaps it will change now that Flattr does not need invites to sign up?

Here are some notes I took about the lecture Debbugs New developments in the ongoing struggle against bugs by Don Armstrong at Debconf 10 (that I watched from the stream). Note that the slides are at : http://rzlab.ucr.edu/debian/debbugs_presentation_debconf10.pdf, and here, I ll only mention most noteworthy elements I ve noticed. There s a linear growth of number of bugs reported for several years, from the statistics made by Don. In average, there s same number of bugs filed / archived per day : Debian s not losing ground against bugs A new affect command has been added, for bugs present in a package affecting another package. The summary command which can be used to quickly spot the summary of lengthy discussions deserves more love. Included summary -1 to send a new summary by email. The new local-debbugs package : can mirror your bugs locally and allow to search, show, etc. even offline. Lots of ideas for new features still in development (but lacking manpower). Interesting idea about action required sorting which exhibits different interests for triager, maintainer, user, to maximize productivity. I guess this should be implemented in clients of the BTS also. Interesting ideas about distributed bug tracking : sharing status with other bugtrackers (see other report), sideways syncing with other distributions, and making it available to upstream also. Note this was discussed in a subsequent BoF, which I ve already blogged about. Some elements taken from the Q&A :

• To my question about addition of RDfa, not currently on the TODO list I guess I ll have to do it myself ? Need to provide more convincing hints on the usefulness.
• only 3 status in the BTS : Fixed / Found / Absent
• some other projects are using debbugs

Overall, a very interesting presentation. Well worth watching the video recording. See also my report on the presentation by Christine Spang on Distributing bugtracking for Debian with SD which also discussed some debbugs things.

Here are my notes about the very insteresting BOF on distributed bugtrackers held at DebConf This is just some elements of report (some taken from the gobby edits, and written down by me after watching the video (unfortunately missing 2 mins of sound in the beginning of the video stream/recording)). This is not a full report of all the discussions, as some discussed elements weren t of interest IMHO. You ll find here many ideas of mine, links and pointers (resulting of our work in the context of the Helios project on similar topics), so this is in no way a report of what was said exactly during the BOF. Aspects of interest:

• for Debian bugs
  • Sharing state between upstream, downstream and sidestream
    • Forwarding upstream
  • Sharing comments between the same bugs
  • merging / conflict resolution (technically challenging, as well as in order to match people s habits/process)
• Same problem in general
  • distributed bug tracking
    • working offline (modify locally, push, etc.)
  • but also distributed version/commit tracking : tracking bug presence or fix along the branches of commits in distributed VCSes
• Interop with other bugtrackers:
  • method to pass data back and forth between different bugtrackers
  • standard method / format / API : lack of standard widely used
  • visibility of issues known upstream (or elsewhere) to Debian users while using reportbug
  • scraping other trackers

Note that these topics may in principle be discussed on the Distributed bug trackers mailing list: http://kitenet.net/cgi-bin/mailman/listinfo/dist-bugs Tools / implementation ideas :

• First, there s a wiki resource about lots of software tools : http://dist-bugs.kitenet.net/software/
• SD (presented later by Christine Spang see my other blog post about her great presentation)
• LaunchPad s tracker already knows about lots of other bugs in other distros including Debian bugs (through bug tasks )
• bts-link watches links between bugs and status changes (mainly through scraping)
• Generic scraping tools could be used :
  • Scraperwiki (http://scraperwiki.com/)
  • or WebReformatter (http://code.google.com/p/webreformatter/
• Virtuoso spongers to perform on the fly conversion of bugs XML feeds to RDF to inject them in a RDF store (ex: search bugzilla inside http://www.openlinksw.com/dataspace/dav/wiki/Main/RDFMappersUseCases)

Storage of repositories / databases of remote bugs :

• Mail lots of suggestions, but to be discarded IMHO
• in users Virtuoso RDF stores, eventually in an integrated way with the Semantic Desktop (KDE 4.4 / Tracker / Zeitgeist) which may be an option (integrating bugs in other desktop tools aware of the semantic desktop).
• through desktopcouch (as used by the bughugger client for LP tracker see the bughugger presentation at RMLL/LSM 2010)

About such storage, and the RDF store option, here s a prototype of ours that exposes UDD s bug facts through a SPARQL interface, allowing to publish bug facts as RDF (setup in the frame of the Helios project). It includes both Debian and Ubuntu bugs, and allows interlinking them in a standard way (with Semantically described links in RDF). See http://testforge.int-evry.fr/d2r-server/resource/spackage/evolution for an example. The problem is not how to distribute (distributed database systems exist), but the semantics of bug properties, and the different implementations of interfaces of different tools : which standard to implement, or how many converters are necessary if no standard ? We believe that navigation/referencing/tracking of external bugs would be much easier if bugs were LinkedData citizens, i.e. they would be exposed with (standard) formats like Semantic Web ontologies served with Semantic Web standards (like RDF). The worst case is currently to have to perform web scraping on bugtracker interfaces. How to avoiding scraping ? A solution could be to publish RDFa / microformats using common standard semantics, directly from the bugtracker pages at bugs URLs (see my wishlist for debbugs on that subject : 590931: Would be great to integrate RDFa metadata into debbugs pages). Now some pointers to historical details on the same subject and Debian (old posts by Lars Wirzenius) :

• Distributed bug tracking
• Bug exchange format

Overall, the discussion was quite interesting, but didn t reach any sensitive progress regarding the state of the art. Much needs to be worked on, mainly on standardizing APIs, semantic descriptions of bugs, and implementation in tools like SD for instance. Stay tuned for more progress.

This is a quick report and notes about the great presentation made at DebConf on Distributed bug tracking for Debian with SD by Christine Spang (http://penta.debconf.org/dc10_schedule/events/591.en.html) Hopefully, the slides are gonna be published soon, but in the meantime, you have them in the video recording (see link above). For those who don t know SD, they may refer to a talk given by its author, Jesse Vincent, at the recent RMLL/LSM 2010 : Peer to peer issue tracking with SD and Prophet (video recording coming soon : EDIT: a premiere recording is available here). She draws 2 main use cases in her speech, which is exactly the kind of features we ve been thinking about for fetchbugs4.me :

• as a Debian maintainer : SD may be used to clone into your local SD repository all tickets related to your activity on your packages in debbugs.
• or as both a Debian maintainer and upstream developper, you d combine bugs from Debian s debbugs and upstream s bugtracker in one same bug database.

What would be even better is to be able to link all of these bugs to each-other but I m not sure this is yet possible. Christine started implementing debbugs (Debian s bugtracker) support in SD. For reference, there s a wishlist about it in 590156 : sd: Provide sync-ing with debbugs. The implementation is still in alpha state, in read-only mode only. Christine has implemented a new method get_{machinereadablebuglog}() for debbugs SOAP interface. It seems to send some Perl Data::Dumper output, which is probably fine for SD as SD is written in Perl, like debbugs but it may be better if it was some more generally accepted format, like JSON maybe. Among the topics discussed, was the absence of any standard API to connect to bugtrackers, meaning that SD needs to include many different client code modules for each of the trackers different APIs over different kinds of protocols. I ve raised (through my IRC comments, watching the presentation through the stream), that there s a proposal for such a standard that exists : OSLC-CM. Still, it s not widely implemented, in particular in Open Source tools (see another post of mine on the subject). I think it would be great if SD could be a client of OSLC-CM compatible trackers, hence I ve filed a wishlist. Also, I suggested that maybe it would be interesting to be able to browse the mail threads of comments on Debian bugs, much like with bts mbox show 12345 command (which opens mutt on the bugs thread, for me), when in offline mode, with SD. Maybe the bts command could just use SD as a backend for all its operations. Another very interesting feature may be to allow transfering a Debian bug to an upstream bugtracker, through SD. Dunno if that will be possible. Finally the code may be found : in github. I m looking forward to testing it.

My project. This post assumes that the reader has read my last post. I ve been trying to find a reasonable method (which computer to use, single user or multi user) to benchmark for some time, and now an e-mail from Simon Marlow calmed me down. He suggests that in this stage it s not necessary to run a long benchmark, but only a short one, that will be enought to have an idea about what each change is causing to performance. For some time I was struggling with the benchmarks to get a reasonable result. I ve started more than once to write a blog post analysing the results I ve got, until in the middle of the post I notice the results are not consistent. This usually happens when there s a big distortion in benchmarks that shouldn t present no change. For instance, in a patch that only changes code that is executed with +RTS -w -RTS, there was a difference of 50% on execution time in default mode, that is, without +RTS -w -RTS. I think that only disabling CPU scaling and running a small number of tests, which make it easier to reproduce a possibly inconsistent result, will be enough to get a good data. I ve made a stack of patches, splitting each part of my work. The stack does not reflect my preferences between the patches, that is, the complete stack applied is not what I think that is the best change to GHC. It s only a set of patches applied linearly, in a way that it s easy to see the impact in performance of each change. The patches are: Don t check for swept blocks in -DS.

The checkHeap function assumed the allocated part of the block contained only alive objects and slops. This was not true for blocks that are collected using mark sweep. The code in this patch skip the test for this kind of blocks.

This patch is only needed to make it possible to run the binary with the parameters +RTS -w -DS -RTS. I ve already explained this patch. I didn t measure the difference in performance before this patch, since it only changes code that is not going to be executed in the benchmarks, since none of them are runned with -DS. Immix: allocate and free memory on lines. This is the main patch with the initial implementation of Immix. As it is the initial implementation, it has it s problems, that will be treated in the following patches. The comparison was done using 3 programs from nofib/gc: fibheaps, fulsom and constraints, as suggested by Simon Marlow. Two tests were made for each version of the compiler code: one using the default GC strategie (copy-collection with mark-compact if there s few heap space left), another one using mark-sweep (or Immix after my changes). When comparing the default GC strategie between before and after this change, the program got 0.4% slower, with the collection of the generation 0 being 0.1% slower and the collection of the generation 1 0.4% slower. Comparing the mark-sweep with Immix, it got 13.7% slower, 15% in GC0 and 7.2% in GC1. The change in memory used is irrelevant. Chose between allocating in line or blocks in todo_block_full The first improvement I ve made is to change the place where the allocation was searching for free lines. Most of the changes were done in the function alloc_for_copy(), in rts/sm/Evac.c. This is bad because this function is called a lot of times, and should be kept fast. So I moved them to ctodo_block_full() in rts/sm/GCUtils.c. It got about 9.9% and 9.6% faster in default and sweep, which makes it 1% faster and 2.8% slower than in the original code. Improvements in sweep() I was bothered that the code I ve written to free memory in lines was messy, and I thought it could be simplified and maybe even turn faster. The general execution time increased 0.4%, but the GC1 time, which is the only one that should be affected, was reduced by 0.6%. In the default GC strategie, there was also a 0.2% increase in time, so I guess this is inside the margin of change. I have to investigate better on this one, but it seems to have improved slightly the performance. Line before inscresing block size. In a conversatio with Simon Marlow, we thought about these two options: one is to search for a line after trying to increase the block size, and that s what I did in Chose between allocating in line or blocks in todo_block_full; the other is trying to search a line first, and just increase the block size if a block (and not a line) was being used for allocation and there is no line available. With this patch I ve changed it to use the second one. Using sweep, the GC time got 0.5% slower, and with default it got 0.1%. rts/sm/Sweep.c: Mark all BF_MEDIUM blocks as BF_FRAGMENTED In my first implementation, the blocks that contain objects bigger than a line (of medium size) are marked with BF_MEDIUM, and are treated as the usual block in mark-sweep: if the block is empty, it s freed; if it s very fragmented, it s marked to be collected using copy-collection. In this patch, instead of making it use the rules of mark-sweep, I just mark it as fragmented and make it be collected by copy-collection. This made the GC0 time be reduced by 3%, but caused an increase of 509.7% in the GC1 time. This was also the first time the memory used was reduced, by 25.4%. The results in the default mode are insignificant. What it seems to me is that the copy-collection code is more efficient with memory usage, but also more slow. Marking objects as BF_FRAGMENTED make this algorithm be used even for full objects. rts/sm/Sweep.c: Don t use 3/4 heuristics to mark as BF_FRAGMENTED The mark-sweep algorithm considered that an object was fragmented if more than 1/4 of the word groups are completely unmarked. I kept using this heuristic, and this change removes it, making more use of the Immix, and less use of copy-collection. This didn t change the memory used, and made the code 10.4% slower in sweep. In default GC mode, there was no relevant difference, as expected. Allocate in lines in minor GCs. In the initial implementation I was only allocating on lines only in major GCs, because I needed the mark stack, which was only available in these kind of allocations. In this change, I created the mark stack on all GCs, and used the allocation on lines. The results for the default mode are insignificant, and the code got 9.0% faster in sweep, using the same memory. Selection I ve made a selection with the presumably better patches, which are: Don t check for swept blocks in -DS, Immix: allocate and free memory on lines, Chose between allocating in line or blocks in todo_block_full, Improvements in sweep() and Allocate in lines in minor GCs. This is an attempt to achieve the best set of patches possible, to see how it improved the original code. Comparing the default strategie for this selection and the original code, it got 3.7% slower. Comparing both sweep, it got 3.9% slower. Comparing the original default with the final sweep, it got 4.2% slower and uses 4.7% more memory. There s a lot of room for improvement, and I m willing to hear suggestions of what I could change in the code to achieve this improvement.

---

The results are available. Every time I had doubt about a comparison, I d run both versions again to check. This is way there are some backup files. The data presented here is not from all the most recent versions of each measurement, but from the ones that I thought were more similar in conditions.

3 interesting small things happened in the weekend. Due to the hard work of mostly Florian Reinhard and George Kiagiadakis, Bluedevil is now available. Bluedevil is a new and improved bluetooth handling thing targetted the KDE Workspaces. The Debconf people have uploaded the Qt Debconf frontend that I blogged about a while ago, so now it should be available. Last, but not least, applications now has more accurate data for if they are online or not, by using the ntrack library. This was especially problematic for people having some interfaces defined in /etc/network/interfaces, and other interfaces managed by NetworkManager.
This feature will be committed to upstream KDE whenever Will gets around to do it. As a added bonus, KDEBindings in Debian has seen a release critical bugfix (python plugins, e.g. plasma widgets, related), and the brokenness of Konsole (libkpty) on the kFreeBSD-arches have been tracked down to a libc issue and a patch has been made. All of this is expected to be part of next stable Debian release, codename Squeeze.

So, DebConf time is over once again. The two weeks worth of fifty weeks waiting are left behind once again, and it's back to get back to normal. DebConf was great Yes, it always is, and that's what we are all saying, but hey - Seriously! Being in the same building than 300 crazed developers is always fun, and it's always better than last year's fun. A good highlight this year is that, given the number of Free Software and Free Culture groups that exist in USA's north-eastern coast, we had the opportunity to join a large crowd which has never been part of DebConf. Also, I must agree that the USA bid for DebConf was aiming to attract as many Debian people (developers, maintainers, or just happy users) which had not yet been to a DebConf before as possible. And it was a great success! I finally met several people I have long read in the mailing lists, in blogs or in IRC. A much higher proportion than usual, I'd venture to say. Another interesting phenomenon /methinks is that this year's DebCamp started much more staffed than usual: I arrived on the first day, Sunday 25, and there were ~40 people there already; I don't have the actual numbers, but we quickly grew, and the number started to stabilize past mid-week, only to (sharply) rise in the weekend, in time for DebianDay and DebConf start. Great time! But, they say, nobody can go to the USA without buying some sweet toys, right? Well, being the proud owner of six very hairy cats, I have thought into entering the looming and weaving industry... But cat hair, while abundant, I have heard is untreadable... Maybe due to the indisciplined, natural and independent personality of the cats (catonality should I say?)... So I had two choices: Clean up my home quite often, or live in a literally hairy mess. Enter choice #3: The Roomba! I had been waiting to buy this thing for several years, as they refuse to send to Mexico or charge Mexican cards. So, I walked across Manhattan and got my very own robot cleaner! For my further surprise, although I have not yet tried it (I don't even have a suitable cable yet), I found this: Yay, the Roomba is actually hackable (via a 7 pin miniDIN serial port)! Wikipedia says that:

Roomba comes with a Mini-DIN TTL serial interface, which is incompatible with standard PC/Mac serial ports and cables, both electrically and physically. However, third-party adapters are available to access the Roomba's computer via Bluetooth, USB, or RS-232 (PC/Mac serial). New, 500-series, and 410/420 series Roombas upgraded with the OSMO hacker device allow the user to monitor Roomba's many sensors and modify its behavior. The Roomba Open Interface (formerly "Roomba Serial Command Interface") API allows programmers and roboticists to create their own enhancements to Roomba. ( )

My first impressions? Well, the Roomba lazily charged its battery throughout the day today, and was hungry and ready when I arrived home. It is a but louder than what I expected, and of course my cats were not thrilled by the presence of a eighth animated and apparently sentient being at home. Their initial reaction was of course to be verrry alert of the thing. Twelve eyes were constantly pointing at the Roomba while mine alternated between them. As they measured the thing's speed and (I guess) inferred its movement patterns, they started escaping upstairs A flat, round thing with no legs to be seen will not likely be able to climb the stairs. And they were completely right. At first, only Chupchic remained downstairs. After a bit, I went up to show them the Roomba didn't jump on us to eat our brains, and after a bit, Santa and Macusa joined. The Roomba roombed for maybe 90 minutes (this space is large, and decided it was enough... And slowly, the rest of them started coming down. I would not say Roomba's cleaning is perfect, of course. Its room discovery algorithm is funny, and it even seems it's based on the mere chance of covering most (never all) of the space it has to clean. I had, of course, not fully studied it (after a single run, how could I?). It does make a honestly good attempt at cleaning under coaches, chairs and tables. It collected a fair amount of dust (on a house that seemed quite clean to me, I cannot imagine what would happen on a messy one). I have not yet played with the virtual walls (infrared transmitters which limit rooms as if a door was closed), but given the size of this house (and that I don't want it to clean around the cats' designated bathroom area), I guess I will end up using them regularly. During DebConf, I heard one bad (stupid useless noisy thing) and two very good (it has radically changed my life) comments on the Roomba. I hope to shift the balance towards 3/4 and not towards 2/2! Anyways... Thanks to each and every one of you. DebConf is great. Always great. Always a success. I cannot even thank specific teams. Debian Rules, and DebConf Rocks!

A fresh release of RQuantLib is now on CRAN and in Debian. RQuantLib combines (some of) the quantitative analytics of QuantLib with the R statistical computing environment and language. This follows the 0.3.3 release from last week and has again a number of internal changes. All uses of objects from external namespaces are now explicit as I removed the remaining using namespace QuantLib;. This makes things a little more verbose, but should be much clearer to read, especially for those not yet up to speed on whether a given object comes from any one of the Boost, QuantLib or Rcpp namespaces. We also generalized an older three-dimensional plotting function used for option surfaces -- which had already been used in the demo() code -- and improved the code underlying this: arrays of option prices and analytics given two input vectors are now computed at the C++ level for a nice little gain in efficiency. This also illustrates the possible improvements from working with the new Rcpp API that is now used throughout the package, Full changelog details, examples and more details about this package are at my RQuantLib page.