The Bad: When I got to work today, I've found our server (white box with Debian etch responsible for networking, files, printers, etc) powered off - most likely the UPS' battery didn't survive a power outage during the night. When I turned the server on I was greeted by all those nice lines telling me I had a hard disk problem. The Ugly: Instead off marking the disk showing the read errors as bad, the RAID stack (device mapper?) somehow concluded the "good" disk of the RAID 1 array wasn't synched and kicked it out... The good The bad sectors did take only some unimportant collectd status files with them. After some poking with dd trying to force the HD to redirect the bad sectors, the read errors vanished and the Reallocated Sector Count didn't increase according to smartctl, which seems like a good signal.

This new Debian release of libgda moves gtk-doc and related packages back to build-depends-indep, and add some makefile logic on debian/rules to detect if gtk-doc is installed (looking for gtk-doc.pc) and enable or not documentation (re)generation on-the-fly. Of course, I've also been bitten by the sha1 checksums on the changes files, as I had a new version of dpkg-dev and an old devscripts/debsign installed. It took me some time to figure how to fix this one.

To my surprise, I've recently discovered that Blogger doesn't defaults to sending the comments to the blog owner (ie, myself) by e-mail. This means I've missed a lot of good feedback during my SoC project last year. Thanks for everyone who has posted comments, and sorry for not noticing/using/answering them before. /me wonders if it's time to think about moving to some other blog platform... maybe a free one...

It has been some time since my last blog post... The thing is, my university vacations ended this week, and overall left a bad taste. Unfortunately, complications ranging from buying a very big new toy from some reasonably strong illness have made an already tight schedule simple impossible to follow. On the end, I haven't dedicated enough time for anything I wanted to during these vacations: Didn't sleep as much as I would like, didn't dedicate enough time to my gf, didn't advance my Summer of Code project anything near as much as I wanted, didn't help a friend to get a Linux install in his shining new laptop, didn't fix my work's server RAID array, didn't package the newest mergeant... The list goes on...

I guess what I wanted to say is: sorry. I know I'm owing a lot of stuff for a lot of people, and I'm really sorry I haven't been able to deliver what I should. I hope to fix all of these ASAP (maybe excluding the sleeping one, seems like a lost cause :P)

Now if just the Sao Paulo's metro stopped this stupid strike...

Earlier today I've just made my first package upload to the Debian archives (and no, this isn't an April 1st. joke :P). It was a debian revision of libgda3, with the following changes:

• Updated the section of the doc-base file. For future reference, recent versions of the doc-base package install a list of valid sections on /usr/share/doc-base/data/section.list. BTW, there is an impressive number of packages which need to update their doc-base files, which is a very simple change.


• Updated the copyright file with a cooler machine-readable one; added some copyright info which was missing (about the GFDL'ed docs) and made lintian happier by using proper copyright lines


• Added a patch to fix an ia64 failure on the documentation building process (Thanks again, Steve and Chris).

Now the bad news: The building still fails in ia64... and I have no idea way. Below is an excerpt from the full build log:

        
Writing ch18.html for chapter
I/O error : No such file or directory
I/O error : No such file or directory
runtime error: file file:///usr/share/xml/docbook/stylesheet/nwalsh/html/chunker.xsl line 202 element document
xsltDocumentElem: unable to save to ch18.html

I'm not sure when I'll have time to try debugging this, so any help would be greatly appreciated.

• I'm now a Debian Maintainer. Yay :)


• New releases of libgda3 and libgnomedb3 were uploaded. On both of them, I've added the Dm-Upload-Allowed, VCS-* and Homepage fields, updated the standards version and fixed FTBFSes on BDFH's dirty chroot, along with some other minor bugfixes.

A lot has happened after the Summer of Code has ended, and I haven't been able to work on bug-triage after that, in part due to "Real Life", in part due to laziness. Anyway, I've finally got some time to work a bit on it today.

First, I've taken care of Debian's 452918, where the reporter said that using "pending" on bug status meaning open might cause confusion with the pending tag. Actually, pending is the word used internally by debbugs to mean open bugs, and its also what is returned by the debbugs' SOAP interface (bugs tagged pending get the status pending-fixed instead). Anyway, I've just mapped the internal/SOAP debbugs status to the ones used by pkgreport.cgi, that should avoid misunderstandings.

While at it, I also took some time to add a feature I've missed while using bug-triage myself: When browsing upstream bugs in search of a match for a Debian bug, it's necessary to read the upstream bug log; now double-clicking on a upstream bug on bug-triage will open it's log on a browser (the same way that selecting a Debian bug and clicking "Go to" in the toolbar works).

It's only small changes by now, and I'm not sure if I'll be able to dedicate myself much more to this in the (near) future, but I'll try my best.

The newest version of mergeant, a GNOME GUI for database manipulation, has hit the Debian archives earlier this week (hmm... actually last week, as today is sunday). Thanks Lo c for the sponsorship.

This version was a minor one, the biggest change being the use of a library to allow the binary to find its resources in different directories on runtime, which was disabled on the Debian package as the user isn't supposed to move its files around anyway.

Another change was the migration of the mime data to the newer shared mime/freedesktop style. It took me some time to figure out that I had to install the mime file through dh_installmime to have it add the mime snippets on the maintainer scripts, as dh_installmime is called before dh_install on CDBS (I wonder if there is some reason for the calling order...)

Update: I've missed that .sharedmime isn't a list of files to install like, say, .install; it is the file to install itself.

The latest version of mergeant, a GNOME GUI for database manipulation, has finally reached the Debian archives, closing all bugs which were open against the previous one. Thanks, Lo c, for the sponsorship.

As a side note, this version of mergeant has got an implicit pointer conversion bug due to missing includes in two different files. This was my first chance to test the bug forwarding feature of the Bug Triaging and Forwarding Tool (my project in Google's Summer of Code 2007). It isn't as comprehensible as I would like due to limitations on bugzilla, but was pretty handy nevertheless.

The Google Summer of Code coding period has ended earlier this week, so it seems to be a good time to stop and look what was done (and what wasn't). Let's start with an objective analysis, following the list of planned features:

Advanced query of bugs on the Debian BTS

In a 1-10 scale, I would give myself something like 6 or 7 here. bug-triage can query bugs by number, source package, package, maintainer and submitter. These parameters can be combined to make an query (reports must match all parameters) and more than one query may be added, subtracted and and'ed. A clearly missing feature is querying by usertag, support for it has been added on btsutils already and is just a question of adding a new input box on the query builder dialog (it should work if you enter the query manually in the bug-triage query line AFAICT).

Show all the information related to a bug report

bug-triage has a "goto" button which opens the bug in the user's preferred browser. Missing is the support to open the bug in a MUA, which I have promissed to Lo c.

Send additional information to a bug report

bug-triage has a "followup" button which opens the user's MUA with the right address. Maybe setting the subject would be nice...

Manipulate the tags of a bug report

I haven't had time to implement this at all. I've started it, the bugs have a context-sensitive menu which have each of the available tags as checkboxes on the right current status, but clicking on them does nothing currently.

Search for similar bugs in upstream developer BTS

This one would value something like 3 or 4 in a 1-10 scale. It searches upstream bugzillas by product and component, but doesn't do any cool trick like searching words in the bug subjects. The plus is that it does support KDE bugzilla, even though I haven't been brave enough to promisse it before SoC :P

Forward a bug report to the upstream developer BTS

There isn't really a good interface to report bugs programatically on GNOME bugzilla, which was disappointing. <rant>There is the bug-buddy xmlrpc(?) interface, but the developers say they will actively break anything using it, which I find very free-software-unlike.</rant>. The current bug-triage solution opens a browser with some of the fields filled. It would be nice to fill the full bug description, but its limited by the maximum size of an URL, so it fills only an introductory text (something like This bug was filled on Debian BTS #XXX) currently. Let's hope bugzilla 3 get on the GNOME world soon.

Conclusions

Unfortunately, this project hasn't evolved as much as I would like in the second half of the GSoC. As I said elsewhere, a lot of stuff has happened, some good, some bad, but on the end everything I hadn't been able to dedicate myself to anything I should. Even so, I think the project may be considered sucessful.

Maybe the more interesting product of this project, at least currently, is still on python-btsutils. As I said in the middle-term report, it'll be integrated in the debian_bundle soon, and might be powering important stuff like the devscripts in the future.

The bug-triage application has a lot to evolve yet, but is also an interesting tool IMHO. There are a lot of hurdles in the way to get different BTSes integrated, but the effort will for sure pay in the future. I plan to keep developing it (evon though keeping the GSoC pace is impossible) and am anxious to see the results ;)

Finally, as I said before, one of the main things on GSoC for me was the potential to learn cool stuff. The best choice I've made during all of this was to use Python. Even though it's still a new language for me, and i still have to search embarassingly simple things in the tutorial/reference, it was a lot more productive than if I was developing in C. Learning to use git was also a great. This goal was the more fulfilled one, I've really learnt a lot of things which will surely be useful in the future. BTOH, I still can't get used to glade :/

I've just finished releasing new versions of all modules related to my Summer of Code project, the Bug Triage and Forward Tool. If my mentor doesn't find any serious bug, these will be the last releases under the SoC flag. Thanks again, Google, for this great program and the progress it brings to Free Software.

The releases:

bzutils 0.2:

• Announcement
• Source .tar.gz

btsutils 0.3:

• Announcement
• Source .tar.gz

bug-triage 0.2.2:

• Announcement
• Source .tar.gz

Debian packages should hit the archive as soon as they get evaluated and sponsored. Of course, a full status report will also be posted during the evaluation time.

Note to self: It's a bad idea to change a list in the middle of a loop iterating over it. The following doesn't work:

for item in list:
    if foo:
        list.remove(list)

And must be replaced with:

for item in list[:]:
    if foo:
        list.remove(list)

On the wrong way, the loop doesn't execute on items positioned immediately after the removed ones. This has cost me a big amount of debugging time...

Dear lazyweb,

Is there any reasonable way to reorder the tabs of a GtkNotebook in glade-3?

I needed to do it to add a tab on a configuration dialog, and couldn't find any. I worked around it by editing the XML file by hand, but it just feels counter-productive and ugly...

Thanks to the timely efforts of Lo c Minier, a more usable (even though still featureless) bug-triage version (0.1-2) has found its way to the Debian Archive, alongside with the shining new btsutils 0.2-1. Now it does return something on queries...

The development is also progressing, the current version in git can already show the upstream bugs for a given Debian package whose upstream uses bugzilla.

For what can be read on Planet SoC, computer problems seems to be the norm between SoC students... I've got my part of this cake today with a power cut and some strange /dev permissions when my PC got back...

Fortunately it was somewhat easy to solve (I believe it was the udev upgrade which did the trick), so I've been able to release btsutils 0.2. Please take a look on the announcement for more details.

Still on the SoC side, thanks to the great work from the Debian ftpmaster team and Lo c Minier, the packages python-btsutils and python-bzutils have made their way through the NEW queue and are on the Debian official archive.

And finally, on Debian side, Lo c Minier also uploaded libgnomedb3 3.0.0-2, a bug fix release for libgnomedb closing two bugs. Thanks, Lo c.

I. Introduction

Today, the mid-term evaluations of the Google Summer of Code starts. It means it's a good for a comprehensive summary of what I've done so far.

For those who don't remember, My GSoC project is the Bug Triaging and Forwarding Tool, whose aim is to help triaging of Debian bug reports. On my proposal, I had included the following timeline:

• April - May: Study of the resources available for use in the project
  (Debian BTS SOAP/LDAP interface, Bugzilla interaction resources, etc)
  and the design of the tool. Request needed unavailable features,
  if any, with patches if possible


• May - July: Development of initial interface and code to interact with
  Debian BTS


• August: Development of Bugzilla plugin

During the proposals evaluations and the community bounding period, it was pointed to me that there's already a lot of good tools to handle bug reports, so I've changed this a bit and already started to work on BTS-Bugzilla interaction, which seems to be the more novel part of the
tool.

The development has been fragmented in three modules: btsutils, bzutils and bug-triage.

II. btsutils

The btstutils is a python module to interact with the Debian Bug Track System. Currently, it can query the bts by bug number, package, source package, maintainer, submitter and usertag. For each bug, it returns bug number, package, summary, status, severity, submitter and tags. Support for returning usertags is under way.

Ths btsutils can use html or soap to access the bug data, and has also a backend-agnostic module wwith tries the best method and, if that fails, tries the other.

Steffano Zacchiroli has offered me to include the btsutils on the debian-bundle, a bundle of python modules for dealing with Debian infrastrcture. I'm very excited by this idea, and believe that if all or at least some of the Debian tools which interact with BTS end using a standardized module for access, this might end up being the major contribution of my GSoC project for the Debian community.

III. bzutils

The bzutils is similar to the bzutils, but is to interact with bugzillas instead. Currently it can do queries based on GNOME's bugzilla boogle and the more generic boolean charts. The later works (at least) on GNOME, KDE and standard bugzilla (i.e., landfill). For each bug, it returns bug id,
product, component, status, resolution, reporter, assignee, summary, priority and severity.

IV. bug-triage

The bug-triage is the Bug triaging and forwarding tool. It's being developed using python, gtk and glade.

Currently, it can query the BTS using any of the btsutils' supported fields (even though the usertags aren't really accessible through the interface yet). It can also launch one of the returned bug reports in a web browser.

Integration with bugzillas is under development. It already has an interface to establish links between Debian packages and upstream bugzilla's address, product and component. The next steps will be to add an upstream button in the toolbar/menu and make it show the bugs selected through these info, allowing the user to mark the bug as forwarded to one of them.

V. The future

After that, I'll work on these features (in no particular order): filling a new upstream bug using the Debian bug data as a starting point, more advanced filtering options to filter the upstream bugs (such as filtering by words in the bug summary, and maybe through backtraces if available) and more ways to manipulate the BTS through the interface.

To be sincere, it seems this project won't be as advanced as I would like on the end of the GSoC. There are some new, unexpected (and good ^_^) things going on my life, which are taking some of the time I expected to dedicate to GSoC, which means I'll likely keep the current pace until the end of the project (I expected to increase it during July). Even so, I believe the
project is going on a good path, and I hope it'll be very useful for the Debian community.

A small bit of info that can be hard to find: It's possible to call debbugs' SOAP get_usertag function passing only an user as argument, to get all usertags set for this user and all bugs with each of these tags, like this:

>>> import SOAPpy
>>> proxy = SOAPpy.SOAPProxy("http://bugs.debian.org/cgi-bin/soap.cgi", "Debbugs/SOAP/V1")
>>> proxy.get_usertags("bts-link-upstream@lists.alioth.debian.org")

Following the hints by Bastian Venthur, which answered my last post (thanks, Bastian), and some suspicious mails to debian-debbugs, I've finally gotten to know that the debbugs' SOAP get_bugs is finally fixed. Of course, this means that btsutils has just got its query function implemented in SOAP besides the html parsing stuff.

Also great news on the debbugs' SOAP camp is that get_status now works on more than one bug per call. Yay! Debbugs people rocks. I'm not realy in place to ask something more drom these great folks, but it would be great if those cool features they keep adding were announced somewhere.

Now, the downside: Querying through SOAP is still quite slower than parsing the html. According to Python's timeit, getting a list with the bugs on all packages maintained by me takes about 17 seconds, while the same operation through HTML parsing takes about 10 seconds. I believe this differences comes from accessing the server one time on the HTML parsing and two times (get_bugs and get_status) on the SOAP.

A feature I've been wanting to implement on bug-triage for some time now is to have unhandled exceptions shown in a gtk message dialog instead of getting lost on the terminal. And I've finally got it working now.

At first, I had some vague impression of having read about changing the default exception handler on Python. Maybe I've just mixed things up, but I couldn't find anything about that today (maybe it was Java?). Instead, I found out some information about Python decorators.

This is the first time I've seen anything like the decorators, and I must say this is a concept a bit hard to get used to, at least for me. Basically, if is a function that gets a callable (ie, a function or anything else that can be called) and returns another callable to be used in its place -- usually the original one with some nice extra.

I've created a python decorator to wrap the function call it decorates inside a try statement, whose except clause show some info about any exception it gets. The code can be seen on the bug-triage git repository.

On the linked file, errorhandler is the decorator and msg_exception takes an exception as argument and shows a message dialog. Applying the decorator to a function is simple, import it and add an @errorhandler just before its declaration.

Some references on decorators:

• PEP 318 - Decorators for functions and methods


• Python Cookbook: Basic exception handling idiom using decorators


• zephyrfalcon.org's "Decorate This"

PS: I just got really happy to discover that this blogger/blogspot draft autosaving stuff works...

This one was a bit tricker to find, so I'm going to post it here: if you use os.fork() on Python, you will want to use os._exit() on the child process, or it will clean stuff it shouldn't and you will have segfaults with doubled memory free.

That said, python's webbrowser module is quite useful; check it if you ever has to open a browser from a Python script/program. The only downside I see on it is exactly that it doesn't takes care of the forking stuff for you. I'm using it on bug-triage to show a bug report on the user browser.

Talking about GSoC, during the last week the Debian BTS was updated, breaking, between other things, btsutils' 0.1 version. The current development version of btsutils, which uses BeautifulSoup to parse the HTML stuff more realiable, wasn't affected by the update.

That's all for now, time to code.