(Disclaimer: Possibly one of my longest post ever, you may want to
scroll to the bottom and only look at the pictures.)
So, what s new in SD since
last week?
Fetching everything
From
http://bugs.freedesktop.org/, it was possible to download all
the bugs I reported there (that means 2 ), but trying to download
those reported by Julien led to a crash without any trace. Dichotomy
FTW, I came up with a bunch of bugs which would lead to the same
result, which confirmed that the amount of bugs processed at once
wasn t the issue.
Playing around with
strace, it appeared that exchanges with the remote
server were apparently fine, so an issue on the client-side SOAP
machinery got suspected. So let s enable tracing:
use SOAP::Lite +trace => 'all';
Tada! The issue was indeed local: malformed XML got received, leading
to a
die call issued from the XML-RPC layer. Thankfully, one can set
up a fault handler, which I used to display the range of bug IDs which
triggered that bug, so that people can look into it and determine what
to add to the blacklist until the underlying issue s been investigated
(presumably, bugzilla s at fault).
Speeding things up
With that blacklisting of buggy bugs, one can then try to move to
other queries, dealing with more bugs. Some examples follow:
reporter=kibi@d.o
product=xorg&component=Driver/VMWare
product=xorg&component=Driver/nouveau
product=xorg&component=Server/general
product=xorg&component=Driver/intel
Moving from 2 bugs to 10-100 bugs (Julien s or VMWare s) was OK. But
then moving to several hundreds of bugs (
Driver/nouveau = 500+ bugs) led to
noticeable performance issues. Not to mention what happened when one
reaches several thousands of bugs (
Driver/intel = 2500+ bugs). Indeed, even
with network exchanges cached into a local file, processing data was
taking up to several dozens of minutes.
I knew about Perl s
-d:DProf, which helps figuring out where time is
spent, but was pointed to
-d:NYTProf (and its accompanying tool,
nytprofhtml). Some hotspots got noticed:
- There s a huge pile of stuff relying on UUIDs heavily, and a cache
is going to be introduced to avoid later calls once a value s been
computed once. That s going to benefit all replica types, not just
bugzilla.
- I didn t care much about date/time at the beginning, but that
turned out to be a very bad idea: since the format returned by
bugzilla wasn t matching a well-known format, time was spent in
the
DateTime::Format::Natural fallback, leading to a big
performance penalty. Fixed with a trivial regular expression.
Things got better, but not good enough. There are several
Prophet (the engine under the hood) backends,
so one can play with:
PROPHET_REPLICA_TYPE=sqlite # the default for SD
PROPHET_REPLICA_TYPE=prophet
Switching to
prophet was a big win, but still not good
enough. Indeed, many tiny files are written, and most of the time is
spent in I/O. Although I m nothing like a performance guru, I guessed
that running on an average laptop, with
ext3 and its default commit
interval of 5 seconds might not be helping, so I gave a quick try to
-o remount,commit=60, and that seemed to help.
Even though there are probably other tricks to find in that area
(which hopefully won t require
root privileges ), there s already a
patch which landed in
prophet s
master branch, replacing
File::Spec->catfile with an optimized version: that function alone
was eating 10% of runtime
In the
sqlite case, disabling the auto-commit feature helped
reducing the I/O load, but a proper patch is still lacking for now
(running into locked database issues, or into missing tables after
having created them isn t fun, so I postponed debugging that).
Since performance issues looked like they could be solved eventually,
I switched back to implementing missing features.
Handling more than comments
Currently 3 types of stuff are currently fetched from the bugzilla
server:
- Bug status: plenty of properties.
- Bug comments: comments that are linked to bugs.
- Bug history: changes that impacted bugs.
(Yes, that means that attachments are totally ignored for now.)
Until now, only bug comments were considered. The first comment was
used to determine a pseudo-title (using its first line), the reporter,
and the creation date. This approach was chosen to try and get a basic
sync working quickly, so as to get:
- A list of bugs matching the query.
- All comments for each of these bugs.
Now, the algorithm is the following: from the bug status, determine a
set of properties of interest; then walk the history backwards, and
update the properties incrementally until the (presumed) initial
state is reached. Then create the ticket using this initial
state . Adding the incremental property changes to that initial
ticket makes it possible to represent the bug s life as a list of
Prophet::ChangeSet objects.
That s where the fun begins, since properties in the bug status may
not match properties in the bug history, so one needs to establish
property correspondence. Also, some properties can be multivalued for
added fun. I believe that s where most of the time is going to be
spent while developing a new replica type in SD: once one knows how to
get a hand on needed info on the remote server, the main question is
what to do with it. For now, I decided to ignore many fields to make
it possible to do a big sync like
Server/general, property support
will be improved later on.
Screenshots
Instead of pasting lengthy terminal excerpt, let s use some
screenshots instead (sorry, I m not sure how to present such things in
an accessible way, suggestions welcome).
Cloning Julien s bugs, listing all
open tickets, listing
all
tickets, searching using a regular expression:
Displaying bug
42 (that s the local ID):
Now, let s start the embedded web server through
sd server --port
1234 and point the browser there.
List of
RESOLVED bugs:
Status and comments for bug
42:
History for bug
42:
Compared to the
original bugzilla page:
Next time
Some items which need work:
- Tweak properties to address the issues raised above.
- Start fetching attachments as well.
- Support further syncs. Currently, a big sync is done once, and
there s no way to tell
sd to sync new changes since last time, if
any. This will probably lead to rewriting how fetching is currently
done, which is: discover all bugs, then fetch all comments and all
history items, for all of them. Properties like last_change_time
will probably be of some help here.
- Have a look at what happens with other bugzilla instances, like
Gnome s.
My tip of day. When you lose control of a process (for example with a process launched with disown or, another example, when you close your SSH terminal with unsynced mutt), thanks gdb.
As mentioned in 
5 weeks after the 0.7.2.2 "Lovegood" release, I am very happy to announce the major feature release, 0.8 codenamed "Godsend". Major feature highlights of this release are desktop notifications (with full support of actions, icons, updates, append and sound), messaging menu / indicators (as provided by Ubuntu's 
Messaging Menu
Further on, Smuxi comes with the following improvements in its user interface:
After a bit of work, I got the sheeva plug working the way I wanted it to. First of all, I grabbed a spare 1G USB flash disk I had laying around and installed Debian squeeze to it by following tbm s instructions here:
I'm swinging on the front porch shade, with happy wifi bars lit up on my
laptop. A good network makes for a happy home, isn't that how the saying goes?
(Maybe not yet.)
Typically that's easy to achieve -- here it wasn't. The house is solar powered,
the only connection to the outside is a phone line. So I needed a computer
that would work in that environment, using minimal (12 volt) power, and
getting the most out of the limited bandwidth while providing those
happy wifi bars.
At DebConf we learned that the name for these little home servers we have been
building for years is "
Long time no see Not much hacking in the last few months. Not much ranting either (some of you I m sure will appreciate ;-).
Anyway, I recently grew excited to learn that 
For me, the most enjoyable part of traveling is the inspiration that I derive from visiting different places, talking to people, and generally being outside of my normal environment. This bank holiday weekend, when so many Londoners visit faraway lands, my partner and I stayed in London instead, and my sought inspiration closer to home. The city has been delightfully quiet, and in contrast to the preceding week, the weather was mostly pleasant, apart from the sudden downpours the BBC described as squally showers .
I ve been wondering if the following kind of testing management software exists (preferably free software, of course).
It would allow one to specify a number of test cases. For each, one should be able to describe preconditions, testing instructions and expected outcome. Also, file attachments should be supported in case a test case needs a particular data set.
It would publish a web site describing each test case.
A tester (who in the free software world could be anyone) would take a test case, follow the instructions given and observe whatever outcome occurs. The tester would then file a test report with this software, either a terse success report or a more verbose failure report.
The software should maintain testing statistics so that testers could easily choose test cases that have a dearth of reports.
As a bonus, it would be nice if the software could submit a failure report as a bug report .
(Note that this would be useful for handling the sort of tests that cannot be automated. There are many good ways already to run automated test suites.)
It looks a lot as if Facebook is closing up on third party developers. Seems
like they've cherry picked the features they want to copy, and now are no
longer interested in third parties anymore:
OCaml compiles native executables in static mode. It allows to have a minimal set of dependencies when delivering an executable. It has also disadvantages like the size of the executable and problems arising when considering libraries update -- but this is another topic. There is still one strong dependency that you should not forget when you want to deliver a product for most of the Linux distributions: dependency on the glibc version.
Trying to run OASIS compiled with Debian Lenny, on CentOS 5.5:
When I started working on WebKitGTK+ I was a web developer, writing IT applications using Python and Django, and building features for content portals running Plone (argh). Even though I was an Epiphany user ever since it was forked off Galeon, I still had to use Firefox for my work, because I couldn t really live without Firebug.
It should come as no surprise, then, that one of my first patches to WebKitGTK+ was actually 
