Recently, Google started changing the web in some interesting ways. A lot by contributing code. It started with projects such as the Chrome web browser, which quickly achieved a market shares over 10%, which is more than Apple Safari and Opera together.And the good side, Chrome did not actually add "yet another incompatible platform", but by being closely related to Safari (or more precisely WebKit), it is not that different.Other stuff that belongs into this category is the Google Web Toolkit and the closure compiler (which essentially is a JavaScript code checker and optimizer).But recently, Google goes a step further. They started tackling internet file formats. First they proposed a new Video codec known as WebM or VP8. Now they extracted an image format out of this codec, known as WebP.There is some serious doubt on the quality, the images do appear to be a bit blurry and lose details. I concur with JPEG even being a bit better, despite the block artefacts. The blurry results of WebP are not really convincing.The h.264 based image looks best to me, but H.264 is patent-encumbered for all I know, so we might not see this new format in wide use until 2028. This already ruled out other image formats such as JBIG or FIF. Or look at audio formats: we're still using MP3 everywhere, because everyone tries to push another patent-encumbered audio format that the others are not going to adopt. So MP3 remains the only thing widely accepted (Ogg Vorbis is nice, but it looks as if companies such as Apple are not going to adopt it, probably because they have interest in other formats. Maybe the Ogg Vorbis support in Chrome will help here on the long run).In particular, why would you pay royalties for a tiny bit of image quality, given the constantly sinking prices of bandwidth? The H.264 patents may make a difference on video data, they probably won't pay of for still images.Given that WebM is sometimes claimed to be a simplified H.264 (minus the patent issues?), WebP probably is as good as we can get to the H.264 based quality without running into the same patent issues?But enough on the image data. There is one thing, I'd really like to see Google change on the web. The worst thing about the web is called JavaScript (or more accurately, ECMAScript). There is just so much wrong with this language that we really need to replace it ASAP.Some things that are wrong with JavaScript:

• The syntax is about as crazy as Perl, anything could be valid code.
• There a lot of syntax quirks and oddities. Try to test for a variable storing an Array ...
• It lacks any reasonable Standard Library - which is a key strength of modern languages such as Java, Python, C# - and which is why everbody is using one of the thousand JavaScript frameworks such as Dojo, Mojikit, Google Closure Library, Prototype, jQuery, MooTools, Yahoo UI Library, ...
• Due to the lack of standard functionality, the JavaScript development cost is a lot higher than needed, development time is bad for the triviality of most JavaScript tasks, and the file size is even worse than JPEG vs. WebP ...

So my biggest wishlist item of Google to contribute to the Web via Chrome and Firefox is a new web scripting language. As much as I love python, the whitespace syntax of Python doesn't work for the web (we need to be able to store complex statements in HTML attributes without newlines), so we'll have to look for something else. Maybe Ruby will work?

Finally someone is talking about distributed name resolution. As usual, the suitability of the Monkeysphere model is underestimated. Someone tells me that this is because they do not have a three-letter protocol abbreviation.

For some time, I've had this idea brewing for how the 'net would work without DNS. Initially I was mostly thinking of how to get rid of the dark side of the registrar business, but it appears that maybe it could help with security in general too. It may not be feasible at all, or I may have forgotten extremely crucial points. I'm not a DNS guru, so bear with me and consider this as a thought experiment. This is, however, a proposal for how the Internet could be made truly decentralized, or acentric. Made so, because it currently is very centralized on at least two counts: the DNS is an inherently hierarchical structure of authority; and so is the situation with the SSL certificates used for securing websites. Let's start with DNS, if you will. Basically, the idea is to replace the current Domain Name System, which is centralised in nature (even if subdomains can be delegated) with a fully decentralised system with no central authority. The point of current DNS is to map names to IP addresses in a unique way. It currently works by registering a this name is managed by such-and-such server, ask for details at such-and-such IP address message. Of course, since there's only one namespace (or a limited number thereof, if you consider that TLDs are different namespaces), there needs to be a central authority to resolve conflicts, because some names will be more popular than others. I propose to start with a cryptographic key pair (think GPG key). Anyone can create their key pair; we assume that it's extremely unlikely for two people to generate the same keys, and that it's also very hard for two people to generate keys with the same fingerprint. By (being the only one) knowing the secret part of a key, I am the authority on what can be signed with it. I could for instance sign records that say that www points to this set of IP addresses, and smtp points to that other address. Within my area of authority, which is to say within my domain. Which is identified by the fingerprint of my key pair. What do we have so far? A mechanism that allows in a decentralised way to verify who's authoritative on the F5566852EB92BD779CF137190EA756B5144843F5 domain . In DNS terms, more probably F55...3F5.some-tld. Now let's publish the signed records on a decentralised peer-to-peer network. I'm not up to date with the latest trends in that domain, but I think something like Gnutella or Freenet would qualify, since it doesn't need central hubs but only an initial list of nodes, and it does automatic discovery of new nodes. Projects such as the FreedomBox (name likely to be changed) will probably help with that. Anyway, when looking up www.F55...3F5.some-tld, the resolver gets the signed record (and the public part of the key) from the P2P network, checks the signature, and uses the contents of the record to determine the IP address. No central or delegated registry involved at any point. Lesser risk of a single point of failure, which means better resilience against outages. Also, a better resilience against DNS-based censorship, be it promoted by non-democratic states or by greedy corporations. Of course, F5566852EB92BD779CF137190EA756B5144843F5 isn't a terribly easy name to remember or transmit. But the string of hex could be presented to the user in some other form easier to recognize for people without geek super-powers. My favourite would be the bibi-binary representation, but I'm told that the bubble-babble method works equally well. It would still be unreasonable to expect people to remember such names, and especially to type them into their web browsers or e-mail agents. A possible solution to that would be to use a short form, such as 144843F5 (or its bibi-binary equivalent), complemented by a visual representation of the full fingerprint such as the one SSH does. That pair could be displayed on business cards, email signatures, glossy brochures, advertisements and so on, and allow users to just type the strange word or flash the 2D barcode, check that the funny picture matches (in case the strange word returns several results), and get the website they want. Which they can of course bookmark under a name they can remember, probably even the same name that everyone else uses to bookmark that website. And nothing precludes convenient directories mapping names to full URLs (like search engines currently do) as long as users check that the visual representation of the site they go to matches what they've seen elsewhere. That elsewhere could, of course, take advantage of the natural web of trust that emerges in of GPG-using communities. If I'm directed at a website whose key I don't immediately recognize, then I can see who trusts that key to be legitimate (by checking the signatures), and who trusts them, and so on. Ultimately, the decision is mine to make, and mine only. In any case, once I know what URL I want to visit, I can be sure this DNS replacement will give me the right IP address (or addresses). DNS servers replying with advertisement sites instead of correct ones, and DNS-based filtering, will be things of the past. Of course, having the right IP address only solves the attacks on the DNS part. Eavesdropping can also happen by diverting traffic at the IP level. But we have a trustable way to know information related to a domain, so let's use it: the signed record can also contain the fingerprint of cryptographic keys (SSL or SSH or whatever). So we can be sure we're actually talking to the correct server, rather than to some man-in-the-middle spy. With the same guarantees of correctness as the ones for the DNS, backed by the web of trust and out-of-band validation. Again, we get rid of the hierarchy of certification authorities , some of which have been known for helping organizations get at the (supposedly secret) data people exchange over the net. To summarize: with this scheme, if I know the site I want to access is F55...3F5, then I can be certain that a) I get the right IP address for it and b) I'm actually talking to it. And nobody in-between can intercept the traffic, if the site is properly administrated and its private key is kept secure. And from the site admin's point of view, there's no need to trust DNS registrars and SSL certification authorities not to fool around with the data, or blackmail me into paying increasing fees for keeping my URL working or my SSL key certified. There's still the problem of the allocation for the IP addresses themselves, but two major parts of the centralized and hierarchical net can be gotten rid of. There are proposals attempting to fix part of the problems already, but they're incomplete. DNSSEC only solves the authentication of DNS records, not the centralisation problem. Monkeysphere aims at adding a web of trust to the SSL certificates system, but the CA chain problem seems to persist (although I must admit I'm not up to speed with the actual details). This is of course only a rough draft, full of technical considerations yet certainly not exhaustive. I've thought about it though, and I believe there are no theoretical problems with the implementation of such a scheme (apart for the handling of expiration and revocations, probably). The main obstacle to mass-adoption I can envision is the UI part, and the entrenched habit of just typing a keyword into a search engine and implicitly trusting its results, then blindly trusting the website itself; but even if mass-adoption isn't reached, I can't see anything wrong with giving those who want to be careful a more secure Internet.

Being a buildd admin means one gets to sign successful build logs so that the built packages get uploaded from buildds to the archive, but that also includes filing bugs for packages which fail to build from source (Build-Attempted). Once that done, they can be marked as Failed, usually with a reason. A few words plus a bug number is usually convenient, so that people (maintainers, possible NMUers, other buildd admins) browsing https://buildd.debian.org/status/ can just click the bug number to reach the BTS. There were a few packages in Build-Attempted for alpha, for which I reported some bugs yesterday, which explains the crossing lines on the following graph: (That s an edited version of the buildd stats for alpha.) One has to be careful while filing bugs, since the severity depends on the target architecture(s) for which the package is failing, but also on its already being built there. Mostly, that boils down to serious for regressions (no longer builds) on release architectures, important otherwise. The only exception I m applying is packages failing on all architectures due to obviously missing build dependencies, since they can t be built again, even on the single architecture they re available for (and we want to guarantee that doesn t happen, we want to be able to binNMU them). There are currently no packages in Build-Attempted for alpha, kfreebsd-amd64, kfreebsd-i386, and sparc (architectures I take care of), but I m going to have a look at other architectures as well, since having portable software and reported (FTBFS) bugs sounds like a worthwhile goal.

xserver-xorg-video-intel There s a previous episode regarding this driver, but let s try and get a short summary regarding UMS (User Mode Setting, old-fashioned) vs. KMS (Kernel Mode Setting, new but also Linux-specific): upstream moved from UMS+KMS to KMS only (that means a single code base and allegedly fewer bugs). UMS vs. KMS timeline:

2.9	2.10	2.11	2.12	2.12+legacy	2.12+shadow	2.13
UMS+KMS	KMS	KMS	KMS	KMS+UMS	KMS	KMS

In order for UMS introduced back into 2.12+legacy to be useful for the ever-buggy i8xx chips, those were blacklisted for KMS in the i915 module as of linux-2.6 2.6.32-21. The plan didn t quite work out, so we moved to the shadow version, relying on KMS again. At that point, we asked for the removal of the kfreebsd-* binaries, since they were unusable. Kernel timeline:

k 2.6.32-20	2.6.32-21 k 2.6.32-23	2.6.32-24 k
KMS	KMS + blacklist for i8xx	KMS

Quick summary:

• You want to use the latest kernel from sid, with the latest -intel driver from sid as well. (That holds for other drivers as well.)
• You may want to check whether kernel bugs are fixed in experimental s kernel.
• You may want to check whether driver bugs are fixed in experimental s driver package (2.13). Feel free to open a bug upstream if there s none for your issue already, optionally telling us about it (through a bug in the BTS which can be marked as forwarded there).

libdrm Due to nouveau s ABI changes, libdrm s compatibility (for nouveau users) is a funny subject. Nouveau users are expected to have the following in mind (README.Debian is there already in the xserver-xorg-video-nouveau package).

2.4.21-1~squeeze3 (sid)	2.4.22-1 (experimental)
nouveau 0.15	nouveau 0.16 aka 1.x
2.6.32 k 2.6.33	2.6.34 k

We carry a big ABI-revert patch in our 2.4.21 package so that we get the bugfixes of that version while still using the 0.15 interface, which is what we have in the sid kernel for now. If we need more stuff from 2.4.22 for squeeze, we should be able to cherry-pick them, or even adjust the ABI-revert patch quite easily. For now (and in experimental anyway), the 2.13 version of -intel is going to depend on libdrm 2.4.22, but cherry-picking fixes should be easy; making it build against sid s 2.4.21 shouldn t be hard either. xserver-xorg-video-openchrome It s now possible to start X with the openchrome driver again, thanks to Nethanel s feedback on #597379, but that doesn t mean it s usable for real unfortunately, since users still report frequent crashes.

Can you believe it? I still talk to people who don t know how to play blackjack. There must be something in the mechanics or tableau of blackjack that people find intimidating; for some reason, people can be a little cautious about taking the first move.
Photo courtesy of Winquerque on Net via Flickr. Clearly, there s no cause to hesitate. Players find quite quickly that it is actually not hard to figure out, and the strategies aren t even terribly complicated. For sure, if you re used to playing online slots, you are probably going to need to work a little harder to master the nuts and bolts. But it s still nothing like getting to grips with Texas Hold em from the beginning, for example!. Quite frankly, you very likely already know a fair portion of what most players have to know. I am able to give you the essentials in one simple little paragraph: If the card has aparticular number, that s the amount it will be worth. If it doesn t, it s worth 10. The only exception to this rule are Aces, which are worth either 1 or 11 (more about that later). Say hit when you have to get closer to 21, and/or stand when you don t. If you are closest to 21, you win. If you bust (get more than 21), you lose. Getting dealt blackjack (a natural 21 from the first two cards), makes you an instant winner (well, you and the dealer can also tie, or push ). Got it? OK, so it is not completely all you need to know. To begin with, most professional blackjack fans understand all too well that your goal has less to do with trying to get to 21, and just about everything to do with defeating an opponent. The dealer will not have the luxury of actually playing , precisely. Whether you re playing at a land-based casino or online (speaking of which, for additional info on USA casino players, all you have to do is click the link!), the house rules specify each individual play they can make. Mainly, she or he needs to take another card until their cards add up to the house rule limit (generally 17 and over). Of course, one of you will get an ill-fated draw and find themselves busted , but this decides a fewer number of games than you might expect. The tricky aspect of blackjack is the decision that you have to make if the cards add up somewhere around 12 and 16 (approximately, subject to other details). If you only had your own cards to consider, it would just be nothing but chance. Don t get me wrong, dumb luck has quite a lot to do with it. It is a card game, of course, and all card games are based on luck to some extent. But the serious gamer recognizes that luck is essentially a code word for statistics ; if you know what you have, and you possess a clue about the dealer s cards, you may make a knowledgeable judgment about which cards you re more inclined to end up with if you decide to hit another time. The most useful clue that you require to evaluate whether to play risky or conservative (sorry, I meant loose or tight ) is the dealer s upcard. Well, that s all for now, but for more information, you can always take a look at our friends at USA online casinos accepting Webmoney , who will get you up to speed. A bit later, we re going to get down to the serious nitty-gritty: doubling down, splitting, and much much more. Until then !

For those who haven t heard: OpenOffice.org Community announces The Document Foundation

The community of volunteers developing and promoting OpenOffice.org sets up an independent Foundation to drive the further growth of the project The Internet, September 28, 2010 The community of volunteers who develop and promote OpenOffice.org, the leading free office software, announce a major change in the project s structure. After ten years successful growth with Sun Microsystems as founding and principle sponsor, the project launches an independent foundation called The Document Foundation , to fulfil the promise of independence written in the original charter. The Foundation will be the cornerstone of a new ecosystem where individuals and organisations can contribute to and benefit from the availability of a truly free office suite. It will generate increased competition and choice for the benefit of customers and drive innovation in the office suite market. From now on, the OpenOffice.org community will be known as The Document Foundation . Oracle, who acquired OpenOffice.org assets as a result of its acquisition of Sun Microsystems, has been invited to become a member of the new Foundation, and donate the brand the community has grown during the past ten years. Pending this decision, the brand LibreOffice has been chosen for the software going forward. The Document Foundation is the result of a collective effort by leading independent members of the OpenOffice.org community, including several project leads and key members of the Community Council. It will be led initially by a Steering Committee of developers and national language projects managers. The Foundation aims to lower the barrier of adoption for both users and developers, to make LibreOffice the most accessible office suite ever. The Foundation will coordinate and oversee the development of LibreOffice, which is available in beta version at the placeholder site: http://www.libreoffice.org. Developers are invited to join the project and contribute to the code in the new friendly and open environment, to shape the future of office productivity suites alongside contributors who translate, test, document, support, and promote the software. Speaking for the group of volunteers, Sophie Gautier a veteran of the community and the former maintainer of the French speaking language project has declared: We believe that the Foundation is a key step for the evolution of the free office suite, as it liberates the development of the code and the evolution of the project from the constraints represented by the commercial interests of a single company. Free software advocates around the world have the extraordinary opportunity of joining the group of founding members today, to write a completely new chapter in the history of FLOSS . [...]

So welcome The Document Foundation and LibreOffice. I already can see how does the community benefits from this due to the merge with go-oo:

Q: What does this announcement mean to other derivatives of OpenOffice.org?
A: We want The Document Foundation to be open to code contributions from as many people as possible. We are delighted to announce that the enhancements produced by the Go-OOo team will be merged into LibreOffice, effective immediately. We hope that others will follow suit.

(from http://www.documentfoundation.org/faq/) So, will Oracle contribute the OpenOffice.org trademark to the community ? Time will tell
Filed under: LibreOffice, Openoffice.org, Proud to use free software

There are situations in which a given package needs to have different dependencies in Debian and in Ubuntu. Despite this difference it s possible to keep a single source package that will build both variants of the package. Continue reading to discover a step-by-step explanation. 1. When is that needed? While it is possible to have different dependencies depending on the distribution on which the package is built, it should be usually avoided when possible. This infrastructure should only be used as a last resort when there are no better alternatives. Here are some examples of when it might be needed:

• Ubuntu has some packages that Debian does not have (or vice-versa), and the resulting package would benefit from having them installed.
• The package names differ between Ubuntu and Debian and it s on purpose (i.e. the difference is a justified choice and not a mistake because both distributions failed to coordinate).
• The packages are built differently in both distributions, and the run time dependencies are not the same due to this. Maybe some associated patches are only applied in Ubuntu.

2. Using substitution variables in debian/control The dependency that varies between both distributions can t be hardcoded in debian/control, instead you should put a substitution variable (substvar) that will be replaced at build time by dpkg-gencontrol. You can name it $ dist:Depends for example:

[...]
Depends: bzip2, $ shlibs:Depends , $ misc:Depends , $ dist:Depends 
[...]

Note that you typically already have other substitution variables ($ shlibs:Depends comes from dpkg-shlibdeps and $ misc:Depends from debhelper and its dh_* scripts). 3. dpkg-gencontrol needs a -V option The value used to replace this new variable needs to be communicated to dpkg-gencontrol. You can use the -V option for this, the syntax would be something like this:

dpkg-gencontrol [...] -Vdist:Depends="foo (>= 2), bar"

If you use debhelper, you have to pass the option to dh_gencontrol after two dashes (--):

dh_gencontrol -- -Vdist:Depends="foo (>= 2), bar"

If you use CDBS, you can set the DEB_DH_GENCONTROL_ARGS_ALL make variable:

include /usr/share/cdbs/1/rules/debhelper.mk
DEB_DH_GENCONTROL_ARGS_ALL = -Vdist:Depends="foo (>= 2), bar"

The value given to dpkg-gencontrol is static in all those examples, now let s see how we can use give a different value depending on the distribution that we re targetting. 4. Using dpkg-vendor in debian/rules dpkg-vendor is a small tool (provided by the dpkg-dev package) that parses the /etc/dpkg/origins/default file (provided by the base-files package) to know the current distribution and its ancestry. It can be used in debian/rules to adjust the behavior depending on the current distribution. You can check its man-page to learn about the various options supported but we re only going to use --derives-from <vendor> in this sample. With this option the script exits with zero if the current distribution is or derives from the indicated distribution, or with 1 otherwise. Now combining all together, we can use dpkg-vendor to dynamically define the content of the substitution variable in debian/rules. Let s suppose that you want $ dist:Depends to be foo (>= 2) on Ubuntu (and its derivatives) and bar everywhere else. Using debhelper s 7 tiny rules file, this example could be:

ifeq ($(shell dpkg-vendor --derives-from Ubuntu && echo yes),yes)
	SUBSTVARS = -Vdist:Depends="foo (>= 2)"
else
	SUBSTVARS = -Vdist:Depends="bar"
endif
%:
	dh $@
override_dh_gencontrol:
	dh_gencontrol -- $(SUBSTVARS)

If you use CDBS, it could be this:

include /usr/share/cdbs/1/rules/debhelper.mk
ifeq ($(shell dpkg-vendor --derives-from Ubuntu && echo yes),yes)
	DEB_DH_GENCONTROL_ARGS_ALL = -Vdist:Depends="foo (>= 2)"
else
	DEB_DH_GENCONTROL_ARGS_ALL = -Vdist:Depends="bar"
endif

Do you want to read more tutorials like this one? Click here to subscribe to my free newsletter, you can opt to receive future articles by email.

9 comments Support my work

When I was a child, an Xmas tradition in our house was that the whole family would play Monopoly. It was a fun game, even though my father would always win. It seems they played it so much in his childhood that he still remembered all the rules and cards and prices and so on by heart, thirty years later. Me, I wouldn't remember what I did yesterday, if I didn't have my ikiwiki-powered journal. Anyway, in a random discussion with my SO, we thought of some variants to the rules that might make it more interesting to adults. First, introduce a legal system, with lawyers, law suits, judges, etc. For example, if you don't want to pay your bill, sue the landlord for something. This would give some time to go on with other activities. The winner is the person who first throws a double, and the loser pays 1d6 times the original amount. (Exact rules to be determined by playtesting, of course.) Then introduce a political system into the game. Let players or bystanders be elected into a parliament, which gets to invent new rules. And finally, allow politicians and judges to be bribed. Corruption is rampant in the real world, so it should not be missing from Monopoly, either.

I m putting together a list of email server features that you should look for. I m thinking:

Transport Layer Security support -

this successor to SSL means that your connection will be encrypted. You want this so that your email and your login details aren t being intercepted. Unless you trust everything between you and your mailserver completely, this is pretty much a minimum requirement.

A good login method, probably either CRAM-MD5 or APOP -

you want this so that you aren t simply sending your password over the network. Your connection might be encrypted, but this is a belt-and-braces approach.

IMAP rather than POP -

Internet Message Access Protocol usually offers many more features than the antiquated Post Office Protocol. POP usually shoves all your email through your virtual letterbox, junk or not, and dumps it on your computer. IMAP is more like telling a postman which email to deliver where and when. You can create folders on the server (as well as on your local computer), you can delete junk mail before you download its bodies and it s easier to access your email from multiple locations like your desktop and your phone. QMUL has a longer comparison but nearly everyone would be better off with IMAP.

Sieve configuration -

Now I m getting really optimistic. A Sieve file is an instruction file of standardised mail filtering rules. They re a little harder to write, but should be much easier to move between Sieve-compatible mailservers.

So do you agree with those as desirable features, and would you add more?

Another one, just not to forget it, since I'm just starting to play with grsecurity. When building package under pbuilder/cowbuilder and using a grsec kernel, you have some stuff to tune. I built my grsec kernel with the sysctl options enabled, so it's easier to fix. The first thing I needed, not directly related to building packages, is the permission for my user to execute stuff in untrusted folder (since I really need to be able to run stuff from my home). I've configured Trusted Path Execution with:

corsac@hidalgo: sudo sysctl -a grep kernel.grsecurity.tpe
kernel.grsecurity.tpe = 1
kernel.grsecurity.tpe_gid = 500
kernel.grsecurity.tpe_invert = 1
kernel.grsecurity.tpe_restrict_all = 1 So what I need from there is to add my user to that group:

corsac@hidalgo: sudo addgroup --gid 500 grsec-tpe
Adding group grsec-tpe' (GID 500) ...
Done.
corsac@hidalgo: sudo adduser corsac grsec-tpe
Adding user corsac' to group grsec-tpe' ...
Adding user corsac to group grsec-tpe
Done. That works fine for general usage (at the cost of less protection for my user). When trying to build stuff in pbuilder, the first problem I hit was during dependencies installation:

[81903.221359] grsec: From 127.0.0.6: denied chmod +s
of /home/corsac/debian/pbuilder/build/cow.8616/usr/local/share/sgml/stylesheet
by /home/corsac/debian/pbuilder/build/cow.8616/bin/chmod[chmod:10424] uid/euid:0/0 gid/egid:0/0,
parent /home/corsac/debian/pbuilder/build/cow.8616/var/lib/dpkg/info/sgml-base.postinst[sgml-base.posti:10421] uid/euid:0/0 gid/egid:0/0 grsec enforces some more protection when in a chroot, and especially forbids some operations in there. So I add an exception, using sysctl. For that, a convenient /etc/sysctl.conf.d/grsec.conf will help:

# we need to do stuff in chroots for package building
kernel.grsecurity.chroot_deny_chmod=0

# lock grsec sysctl
# kernel.grsecurity.grsec_lock=1 The last one is still in comment since I know I'll have to tune further the sysctl. With this, the build-deps install fine, but when starting the build itself, it fails because I can't execute stuff inside chroot, and especially not debian/rules:

Sep 20 19:43:24 hidalgo kernel: [87339.510137] grsec: From 127.0.0.6: denied untrusted exec of
/home/corsac/debian/pbuilder/build/cow.26657/tmp/buildd/evolution-data-server-2.30.3/debian/rules by
/home/corsac/debian/pbuilder/build/cow.26657/usr/bin/fakeroot-sysv[fakeroot:10916] uid/euid:1234/1234 gid/egid:1234/1234,
parent /home/corsac/debian/pbuilder/build/cow.26657/usr/bin/fakeroot-sysv[fakeroot:10895] uid/euid:1234/1234 gid/egid:1234/1234 That's again because of TPE. Because, inside the chroot, the pbuilder user (uid 1234) doesn't belong to the grsec-tpe group (which doesn't even exist). So the correct fix here is to create a 500 group inside the chroot, and add the pbuilder user:

corsac@hidalgo: sudo cowbuilder --login --save-after-login
-> Copying COW directory
[ ]
root@hidalgo:/# sudo addgroup --gid 500 grsec-tpe
Adding group grsec-tpe' (GID 500) ...
Done.
root@hidalgo:/# sudo adduser pbuilder grsec-tpe
Adding user pbuilder' to group grsec-tpe' ...
Adding user pbuilder to group grsec-tpe
Done. Et voil !

thanks to Lucas' recent archive rebuild & bug filing we have new stuff to work on :) my activities in the last week:

• #567241 libbuffy-perl: "fails after binnmu"
  apply patches from ntyni and upload to DELAYED/2
• #578458 src:libcoro-perl: "libcoro-perl: FTBFS: E: Caught signal 'Terminated': terminating immediately"
  send a mail to -close after it finally builds again (pkg-perl)
• #592620 time: "time: FTBFS in ubuntu due to nonexistent build dependency"
  upload NMU prepared by carnil to DELAYED/5
• #594503 liblunar-1-0-dbg: "liblunar-1-0-dbg: empty package"
  tell dh_strip about the -dbg package and upload to DELAYED/2
• #595473 gobby-0.5: "does not depend on dbus"
  close the bug as #595480 is fixed
• #595836 enscribe: "enscribe: FTBFS in squeeze: Nonexistent build-dependency: 'libgd-dev'"
  switch order of real/virtual package in Build-Depends and uploaded to DELAYED/2
• #595837 libproxy: "libproxy: FTBFS in squeeze: pacrunner_mozjs.c:33:19: error: jsapi.h: No such file or directory"
  tag unreproducible, close later
• #595845 namazu2: "namazu2: FTBFS in squeeze: Nonexistent build-dependency: 'tk8.3'"
  fix build-dep and upload to DELAYED/2
• #595850 gjs: "gjs: FTBFS in squeeze: modules/console.c:53:18: error: glib.h: No such file or directory"
  tag unreproducible, close later
• #595871 freej: "freej: FTBFS in squeeze: make[5]: *** No rule to make target ./Makefile'. Stop."
  tag unreproducible, close later
• #595876 multiboot: "multiboot: FTBFS in squeeze: rm: cannot remove debian/multiboot/usr/share/info/dir': No such file or directory"
  tag unreproducible

Rant follows. I have upgraded my PC to squeeze this weekend. While this generally worked fine, udev (version 160-1) made my day. As usual I had not read the release notes previously and was hit by

Please upgrade your kernel before or while upgrading udev.

AT YOUR OWN RISK, you can force the installation of this version of udev WHICH DOES NOT WORK WITH YOUR RUNNING KERNEL AND WILL BREAK YOUR SYSTEM

dpkg to the rescue:

dpkg -i /var/cache/apt/archives/linux-base_2.6.32-21_all.deb 
dpkg -i /var/cache/apt/archives/linux-image-2.6.32-5-amd64_2.6.32-21_i386.deb  /var/cache/apt/archives/linux-image-2.6-amd64_2.6.32+28_i386.deb 
reboot

Afterwards the upgrade succeded. (For reference: The correct solution would probably have been to run apt-get install linux-image-2.6-amd64 and rebooting, before starting the dist-upgrade.) udev was not finished with me, though, the syntax for udev rules has changed again. ("NAME="%k" is ignored, because it breaks kernel supplied names", "SYSFS = will be removed in a future udev version, please use ATTR =", "BUS= will be removed in a future udev version, please use SUBSYSTEM=", etc.) So I started googling and rewriting my custom rules for USB-sticks and USB cardreader. After many reboots (the only way to test these rules nowadays), thing were still not working. My cardreader is not an exotic but a common one, i.e. it does not report media changes. (The guide for writing udev rules shares this opinion.) Previously I had:

BUS=="scsi", KERNEL=="sd*", SYSFS model =="Compact Flash",SYSFS vendor =="Generic-",NAME all_partitions ="cardreader/cf"

which caused udev to generate /dev/cardreader/cf1, /dev/cardreader/cf2, etc. on bootup (without media present), enabling me to configure /etc/fstab to allow mounting of /dev/cardreader/cf1 without superuser privileges. After googling I ended up with this new rule:

SUBSYSTEMS=="scsi", KERNEL=="sd*", DRIVERS=="sd", ATTRS vendor =="Generic-", ATTRS model =="SD/MMC          ", SYMLINK+="cardreader/sd%n", OPTIONS+="all_partitions"

and while I get ...

argenau:~# ls -l /dev/cardreader/sd
lrwxrwxrwx 1 root root 6 Sep 19 14:42 /dev/cardreader/sd -> ../sdd

... there is still no /dev/cardreader/sd1 (unless I insert a card and run fdisk -l /dev/cardreader/sd as superuser). Looks like OPTIONS+="all_partitions" is ignored. And indeed I find this in udev's changelog:

udev 152
[...]
The option "all_partitions" was removed from udev. This should not be needed for usual hardware. Udev can not safely make assumptions about non-existing partition major/minor numbers, and therefore no longer provide this unreliable and unsafe option.

Joy. Thanks udev for telling me consumer devices have been fixed overnight, USB cardreaders have suddenly started reporting media change events.

Hello fellow developers, The summer is over but I m happy to announce that this year s Summer of Code at Debian has been better than ever! This is indeed the 4th time we had the privilege of participating in the Google Summer of Code and each year has been a little different. This year, 8 of our 10 students succeeded in our (very strict!) final evaluations, but we have reasons to believe that they will translate into more long-term developers than ever, all thank to you. The highlight this year has been getting almost all of our students at DebConf10. Thanks again this year to generous Travel Grants from the Google Open Source Team, we managed to fly in 7 of our students (up from 3!). You certainly saw them, presenting during DebianDay, hacking on the grass of Columbia, hacking^Wcheering our Debian Project Leader throwing the inaugural pitch of a professional baseball game or hacking^Wsun-tanning on the tr s kitsch Coney Island beach. Before I give the keyboard to our Students, I d like to tell you that it will be the pleasure and honor of Obey Arthur Liu (yours truly, as Administrator) and Bastian Venthur (as Mentor) to represent Debian at the Summer of Code 2010 Mentors Summit on 23-24 October 2010, at the Google Headquarters in Mountain View. Like last year, we expect many other DDs to be present under other hats. We will be having 2 days of unconference on GSoC and free software related topics. We all look forward to reporting from California on Planet and soc-coordination@l.a.d.o! All of our students had a wonderful experience, even if they couldn t come to DebConf, that is best shared in their own voice, so without further ado, our successful projects: Multi-Arch support in APT by David Kalnischkies, mentored by Michael Vogt apt-get install MultiArch does mostly work now as most code is already merged in squeeze, but if not complain about us at deity@l.d.o! Still, a lot left on the todo list not only in APT so let us all add MultiArch again to the Release Goals and work hard on squeezing it into wheezy. Debbugs Bug Reporting and Manipulation API by David Wendt Jr., mentored by Bastian Venthur Hello, I m David Wendt, and I went to Debconf10 to learn more about the development side of Debian. Having used it since the 9th grade, I ve been intimately familiar with many of Debian s internals. However, I wanted to see the developers and other Debian users. At DebConf, I was able to see a variety of talks from Debian and Ubuntu developers. I also got to meet with my mentor as well as the maintainer of Debbugs. Content-aware Config Files Upgrading by Krzysztof Tyszecki, mentored by Dominique Dumont Config::Model is now capable of manipulating files using shorter and easier to write models. Thanks to that, packagers may start experiment with creating upgrade models. Further work is needed to support more complicated config files Dominique Dumont is working on DEP-5 parser, I ll shortly start working on a cupsd config file parser.
The best thing about DebConf10 is that every person I talked with knew what I was doing. I had a mission to get some feedback on my project. Everybody liked the idea of making upgrades less cumbersome. On the other side, it was my first visit to United States, so I decided to go on a daytrip on my own (instead of staying inside the building, despite heat warnings). I had a chance to visit many interesting places like Ground Zero, the UN headquarters, Grand Central Terminal, Times square and Rockefeller Center that was a great experience. Hurd port and de-Linux-ization of Debian-Installer by J r mie Koenig, mentored by Samuel Thibault Debconf10 was great! Among other people working on the installer, I met Aur lien Jarno from the Debian/kFreeBSD team and we worked together on a cross-platform busybox package. Besides, the talks were very interesting and I ve filled my TODO-list for the year.
For instance I learned about the Jigsaw project of OpenJDK, and how Debian would be the ideal platform to experiment with it. More generally, some people think Debian could push Java 7 forward and I d like to see this happen. Smart Upload Server for FTP Master by Petr Jasek, mentored by Joerg Jaspert I must say that it was great time for me in NY, I ve met and talked and coded with people from ftp-master team like Torsten Werner who helped me to push the project a bit further and with some other people who were looking forward to release of the tool which I hope they will use quite soon. Everybody interested, everybody excited, really cool place and time. And I can t forget the Coney Island beach and stuff, lot of fun, lot of sun;) Aptitude Qt by Piotr Galiszewski, mentored by Sune Vuorela Currently, development branches support full features searching, viewing extended package s informations, performing cache and packages operations. Code and GUI still require a lot of work which will be continued. Informations about further progress could be found on aptitude mailing list and repository rss channel. Debian-Installer on Neo FreeRunner and Handheld Devices by Thibaut Girka, mentored by Gaudenz Steinlin For me, DebConf 10 started at the airport, where Sylvestre Ledru (whom I didn t know of before) was wearing a GSoC 2007 t-shirt, that is, given the circumstances, almost equivalent to say I m a hacker, I m going to DebConf 10 .
I ve spent my time at the conference attending various talks, hacking, meeting DDs and other hackers (amongst others, my co-mentor Per Andersson, Paul Wise, Julien Cristau, Christian Perrier, Cyril Brulebois, Martin Michlmayr, Colin Watson and Otavio Salvadores who I have to thank for his patience while dealing with my questions), chatting, cross-signing keys, rushing to finish eating before 7pm, getting sunburnt, sightseeing (thanks, Arthur, for the lightning-fast tour of Manhattan!), and so on. Debian Developers and community, we count on you. See you next year! (cross-posted to debian-devel-announce@l.d.o and soc-coordination@l.a.d.o)

I've now enabled IPv6 for both eyrie.org systems and have published corresponding DNS records (although DNS as always may take some time to update). Panix mentioned a while back that they now supported IPv6 (with, as usual, a great admin console interface that made it trivial to turn it on and set up proper reverse DNS), but I knew there were going to be some issues with setting it up so I put it off. Amusingly, it was Debian that finally pushed me into configuring it: I kept getting the AAAA record for security.debian.org first and waiting for the connection to time out before connecting to the IPv4 address, and I got tired of waiting. For the most part, I just had to turn it on and everything simply worked. There were only a couple bits of weirdness:

• The way that iptables handles IPv6 (or, rather, doesn't) is extremely annoying. I assumed I could just add IPv6 addresses to my regular iptables rules and be done, but no. There's a completely separate iptables table for IPv6 that's managed with a different set of tools. So I had to duplicate most of my rules and separate the address rules into two separate rule sets, plus tweak the code to load the rules. This is lame.
• tinydns of course can't handle AAAA records, since tinydns is part DNS server and part exercise in Dan Bernstein's opinions about how DNS should be constructed. Also, and probably more relevantly to this, it's not being updated these days. There's an escape where you can add raw records and a calculator on-line, so it's all set up anyway, but it's kind of a pain. At some point I'm going to care about DNSSEC, and then I'll need to find another DNS server. I still think BIND is a bloated pig, which is why I'm running tinydns. I want a miniscule DNS server that is trivial to set up and just serves out exactly the records that I tell it to, which tinydns is great at. If anyone knows anything else out there like that which they'd recommend, drop me a message.

Like many people I use Apache's mod_proxy to proxy from *:80 to a bunch of servers running upon 127.0.0.1:XX. (I've mentioned this too often; but in short I have a bunch of sites all running with thttpd under their own UID). Why do I use apache soley as a reverse proxy, instead of pound, varnish, nginx, or lighttpd? After all it is pretty heavy-weight. Well the answer to that is that I have a bunch of mod_rewrite rules. So I'm wondering, could I drop apache if I were to hack together a simple network proxy that would listen upon port 80, reading requests, and directing them to local servers? The answer to that is plainly "yes". There are many reverse-proxies around and writing them isn't hard. So what would be the point? Imagine a reverse-proxy that understood mod_rewrite rules. That would rock. In short we'd have to define three things:

• Matching vhost name.
• Destination to proxy to.
• (Optionally) the mod_rewrite rules

Given something like this:

LISTEN 1.2.3.4:80
host example.net or host www.example.net
 
   #redirect traffic here
   proxy_to 127.0.0.1 1011
   RewriteRule /about /cgi-bin/index.cgi
 

When the proxy received an incoming request to http://example.net/about it would actually send the request /cgi-bin/index.cgi to the host 127.0.0.1:1011. That seems neither too complex nor too impossible. The hard part would be emulating mod_rewrite 100%. Especially chained requests. I would be willing to write the trivial version, but I suspect the full emulation would be a job of diminishing returns. Am I right? ObSubject: You crossed the line first, sir. - "The Dark Knight"

While gzip is the standard Unix tool when it comes to compression, there are other tools available and some of them are performing better than gzip in terms of compression ratio. This article will explain where you can make use of them in your Debian packaging work. In the source package A source package is composed of multiple files. The .dsc file is always uncompressed and it s fine since it s a small textual file. The upstream tarballs can be compressed with gzip (orig.tar.gz), bzip2 (orig.tar.bz2), lzma (orig.tar.lzma) or xz (orig.tar.xz), so choose the one that you want if upstream provides the tarball compressed with multiple tools. Put it at the right place and dpkg-source will automatically use it. Note however that packages using source format 1.0 are restricted to gzip, and the main Debian archive currently only allows gzip and bzip2 (xz might be allowed later) even if the source format 3.0 (quilt) supports all of them. The debian packaging files are provided either in a .diff.gz file for source format 1.0 (again only gzip is supported) or in a .debian.tar file for source format 3.0 (quilt) . The latter tarball can be compressed with the tool of your choice, you just have to tell dpkg-source which one to use (see below, note that gzip is the default). In a native package, dpkg-source must generate the main tarball and you can instruct it to use another tool than gzip with the --compression option. That option is usually put in debian/source/options:

# Use bzip2 instead of gzip
compression = "bzip2"
compression-level = 9

For 3.0 (quilt) source packages, this option is not very useful as the debian tarball that gets compressed is usually not very large. But some maintainers like to use the same compression tool for the upstream tarball and the debian tarball, so you can use this option to harmonize both. In native packages, it s much more interesting: for instance the size of dpkg s source package has been reduced of 30% by switching to bzip2, saving 2Mb of disk. In the binary packages .deb files also contain compressed tar archives and by default they use gzip as well:

$ ar t dpkg_1.15.9_i386.deb
debian-binary
control.tar.gz
data.tar.gz

data.tar.gz is the archive that contains all the files to be installed and it s the one that you can compress with another tool if you want. Again this is mostly interesting for (very) large packages where the size difference clearly justifies deviating from the default compression tool. Try it out and see how many megabytes you can shove. Another aspect that you must keep in mind is that those alternative tools might use important amount of memory to do their job, both for compression and decompression. So if your package is meant to be installed on embedded platforms, or if you want to build your package on low-end hardware with few memory, you might want to stick with gzip. Now how do you change the compression tool? Easy, dpkg-deb supports a -Z option, so you just have to pass -Zbzip2 for example. You can also pass -z6 for example to change the compression level to 6 (it s interesting because a lower compression level might require less memory depending on the tool used). The dpkg-deb invocation is typically hidden behind the call to dh_builddeb in your debian/rules so you have to replace that invocation with dh_builddeb -- -Zbzip2 . If you are using a debhelper 7 tiny rules files, you have to add an override like in this example:

%:
	dh $@
override_dh_builddeb:
	dh_builddeb -- -Zbzip2

If you are using CDBS, you have to set the variable DEB_DH_BUILDDEB_ARGS:

include /usr/share/cdbs/1/rules/debhelper.mk
[...]
DEB_DH_BUILDDEB_ARGS = -- -Zbzip2

I hope you found this article helpful. Follow me on identi.ca or on twitter. One comment Support my work

I have always liked learning and understanding history. Since I discovered him, for a couple of years already I always try to catch Javier Garciadiego's program Conversaciones sobre historia, Saturday 9AM in the Horizonte 108 radio station (can be listened to online). This program started by going over the events just before the beginning of the 1910 revolution in Mexico - and along slightly over five years, one hour per week and following different threads, the program has reached the end of the Cristiada, in the early 1930s. Garciadiego has a very nice, followable, amenable way of telling history, and I have recommended his program to many friends. This last June, I spent some days in Guatemala City, for DrupalCamp Centroam rica. I stayed with my good Colombian friend, Dilson, and at his house he had framed a poster of the History of the Civilizations. Of course, I got my nose close to it, guessing as many faces as possible in the lot. And he showed me his last Christmas present: Two books, each of them with 6 CDs. One is Historia de las Civilizaciones, the second one is Historia de las independencias. They are made by Colombia's very well known and well regarded historian Diana Uribe. I copied the CDs in order to listen to them later And wow, was I impressed! Diana Uribe makes a great narrative about topics that to some people would seem boring and dry. As I said, I have always found passion in understanding the human processes that have shaped civilization and brought us all the way to where we stand now. Well, Diana Uribe manages to bring more "normal" people to this passion. While looking for information on her to share in this blog post, I found so many places offering download of her disks, with apparently young people talking about how she has got them all so excited and interested in history... That's, I think, the best "thank you" any academician can get: having non-specialists say how her work has opened up the passion of one of the world's least sexy professions to them. And yes there are so many "thank you" and "I want" commentaries, so much of what I would call "fan mail", that it took me a bit to find an online library carrying both works. And yes, at ~US$50 each, I do intend to buy them. Now, why am I writing this today? Well, yes, because I finished listening to the series today, but besides During this year, most of Latin American countries conmemorate their 200 years of existence. Most of the independentist struggles in the continent started in 1809-1810. And today is the "partying" day in Mexico Says the legend that in the night between September 15 and 16, 1810, a priest who is always painted as old and charismatic called on his small town urging the people to rise and fight for independence, and as a result of that, only 11 years later Mexico was a fully independent country, spanning from Costa Rica to California, and... well, a nice and very idealized myth. A century later, in 1910, after a very long stability and growth period (attained mostly through repression, the same abstract thing named as "the people" rose against the dictator Porfirio D az, who had been Mexico's president for 30 years. The revolution deeply changed the social face of the country, but politically... After ~15 years of fighting, the result was that a 30 year long dictatorship was replaced by a 70 year long one... And our political system still has not evolved beyond that model. Now, comparing what has not improved nor even stayed the same but went backwards... A century ago, the festivities of the hundred years of independence were a time for showing pride, for showing to the guests from more "civilized" countries how ours was by then a modern, thriving country worth believing in, worth investing in: Besides the important, majestic and well built monuments that were erected and still stand today (i.e. the Column of the Independence or Hemiciclo a Ju rez, many institutions that would socially shape the next century even after D az's death, even after he had been declared not the role model we wanted after all were born: The National University (nowadays the most important university in Latin America), the National School for Professors, the Railroad Technical School and many others (see Javier Aranda's note for some more details)... The celebration was well-thought and planned. Of course, it didn't go into some darker corners, the country was as uneven and unfair as it can be for the poorer indigenous population (which back then was a majority), and what not. But this year? Well, we are expecting an impressive show tonight (which I won't see, even though I'd like to, as I no longer have a TV and even if I had wanted to go downtown for the celebration, different government branches are insisting we should just sit and watch it by TV at home as it can be too crowded... so not even that was well thought out Of course not every Mexican can go to the same square and see the same de-facto president do the ritual, but some more redundancy could be thought, spreading acts through all of the city instead of concentrating the festivities all along Reforma. But anyway Leaving aside our current de-facto ruler's inabilities to do anything worthy, which are already well known and documented... I took this opportunity to listen to a great work, and am most happy to do it, and to be able to share it with you.

The Exoclimes conference ended on an interesting perennial problem: what should we call these things we work on? The climate folks pointed out what the astronomers have grown used to: the telephone numbers we use as planet names. The two most studied Hot Jupiters, for example, are HD209458b and HD189733b. Come on, we could do better than that. In fact the naming worries started earlier when Frank Selsis asked people to avoid calling the planets he works on "super-Earths" (or should that be Super-Earths?). Dimitar Sasselov had accidentally kicked up a kerfuffle in the media a few weeks back by calling them "Earth-like" planets, when he meant "Earth-sized" planets. Selsis pointed out non-scientists will interpret "super-Earth" to mean "like Earth, only Better!", when we mean its just a bigger ball of rock and ice than Earth. But what to call them? Well, at least one planet already has a name, sort of: HD 209458b is frequently called 'Osiris'. And there is a published list of possible planet names. But Osiris the Egyptian god is more famous than Osiris the planet, confusing google. And people are going to assume things based on the planet names: are we really going to name a planet Vulcan? One suggestion was that if we found a system with seven planets we could call them after the Seven Dwarfs, but this was ruled out: don't mention the war (but dwarfs could also mean brown dwarfs!) At the end of the conference, there was show of hands: who agreed with naming planets? of about 100 present, all in favour, none against. The delegate to the IAU commision on the matter, which had voted against, duly noted. Perhaps we will name them then. Tags names, exoplanets, astronomy

(Disclaimer: Possibly one of my longest post ever, you may want to scroll to the bottom and only look at the pictures.) So, what s new in SD since last week? Fetching everything From http://bugs.freedesktop.org/, it was possible to download all the bugs I reported there (that means 2 ), but trying to download those reported by Julien led to a crash without any trace. Dichotomy FTW, I came up with a bunch of bugs which would lead to the same result, which confirmed that the amount of bugs processed at once wasn t the issue. Playing around with strace, it appeared that exchanges with the remote server were apparently fine, so an issue on the client-side SOAP machinery got suspected. So let s enable tracing:

use SOAP::Lite +trace => 'all';

Tada! The issue was indeed local: malformed XML got received, leading to a die call issued from the XML-RPC layer. Thankfully, one can set up a fault handler, which I used to display the range of bug IDs which triggered that bug, so that people can look into it and determine what to add to the blacklist until the underlying issue s been investigated (presumably, bugzilla s at fault). Speeding things up With that blacklisting of buggy bugs, one can then try to move to other queries, dealing with more bugs. Some examples follow:

reporter=kibi@d.o
product=xorg&component=Driver/VMWare
product=xorg&component=Driver/nouveau
product=xorg&component=Server/general
product=xorg&component=Driver/intel

Moving from 2 bugs to 10-100 bugs (Julien s or VMWare s) was OK. But then moving to several hundreds of bugs (Driver/nouveau = 500+ bugs) led to noticeable performance issues. Not to mention what happened when one reaches several thousands of bugs (Driver/intel = 2500+ bugs). Indeed, even with network exchanges cached into a local file, processing data was taking up to several dozens of minutes. I knew about Perl s -d:DProf, which helps figuring out where time is spent, but was pointed to -d:NYTProf (and its accompanying tool, nytprofhtml). Some hotspots got noticed:

• There s a huge pile of stuff relying on UUIDs heavily, and a cache is going to be introduced to avoid later calls once a value s been computed once. That s going to benefit all replica types, not just bugzilla.
• I didn t care much about date/time at the beginning, but that turned out to be a very bad idea: since the format returned by bugzilla wasn t matching a well-known format, time was spent in the DateTime::Format::Natural fallback, leading to a big performance penalty. Fixed with a trivial regular expression.

Things got better, but not good enough. There are several Prophet (the engine under the hood) backends, so one can play with:

PROPHET_REPLICA_TYPE=sqlite   # the default for SD
PROPHET_REPLICA_TYPE=prophet

Switching to prophet was a big win, but still not good enough. Indeed, many tiny files are written, and most of the time is spent in I/O. Although I m nothing like a performance guru, I guessed that running on an average laptop, with ext3 and its default commit interval of 5 seconds might not be helping, so I gave a quick try to -o remount,commit=60, and that seemed to help. Even though there are probably other tricks to find in that area (which hopefully won t require root privileges ), there s already a patch which landed in prophet s master branch, replacing File::Spec->catfile with an optimized version: that function alone was eating 10% of runtime In the sqlite case, disabling the auto-commit feature helped reducing the I/O load, but a proper patch is still lacking for now (running into locked database issues, or into missing tables after having created them isn t fun, so I postponed debugging that). Since performance issues looked like they could be solved eventually, I switched back to implementing missing features. Handling more than comments Currently 3 types of stuff are currently fetched from the bugzilla server:

• Bug status: plenty of properties.
• Bug comments: comments that are linked to bugs.
• Bug history: changes that impacted bugs.

(Yes, that means that attachments are totally ignored for now.) Until now, only bug comments were considered. The first comment was used to determine a pseudo-title (using its first line), the reporter, and the creation date. This approach was chosen to try and get a basic sync working quickly, so as to get:

• A list of bugs matching the query.
• All comments for each of these bugs.

Now, the algorithm is the following: from the bug status, determine a set of properties of interest; then walk the history backwards, and update the properties incrementally until the (presumed) initial state is reached. Then create the ticket using this initial state . Adding the incremental property changes to that initial ticket makes it possible to represent the bug s life as a list of Prophet::ChangeSet objects. That s where the fun begins, since properties in the bug status may not match properties in the bug history, so one needs to establish property correspondence. Also, some properties can be multivalued for added fun. I believe that s where most of the time is going to be spent while developing a new replica type in SD: once one knows how to get a hand on needed info on the remote server, the main question is what to do with it. For now, I decided to ignore many fields to make it possible to do a big sync like Server/general, property support will be improved later on. Screenshots Instead of pasting lengthy terminal excerpt, let s use some screenshots instead (sorry, I m not sure how to present such things in an accessible way, suggestions welcome). Cloning Julien s bugs, listing all open tickets, listing all tickets, searching using a regular expression: Displaying bug 42 (that s the local ID): Now, let s start the embedded web server through sd server --port 1234 and point the browser there. List of RESOLVED bugs: Status and comments for bug 42: History for bug 42: Compared to the original bugzilla page: Next time Some items which need work:

• Tweak properties to address the issues raised above.
• Start fetching attachments as well.
• Support further syncs. Currently, a big sync is done once, and there s no way to tell sd to sync new changes since last time, if any. This will probably lead to rewriting how fetching is currently done, which is: discover all bugs, then fetch all comments and all history items, for all of them. Properties like last_change_time will probably be of some help here.
• Have a look at what happens with other bugzilla instances, like Gnome s.