On Wednesday 17th October the majority of my belongings will leave Norwich, to arrive in Castlerock at some point the next week (exact day to be confirmed). I will then start work for 3par as a software developer in Belfast on Monday 29th. It's been 6 years since we moved to Norwich and I'll be sad to leave it, and all the friends I've made here, behind. I'd have stayed a bit longer but 3par were keen for me to get started. I imagine I'll be in the Fat Cat on Saturday 13th if people are around. And I expect visited in NI once we get settled in!

96.43.01 and 71.86.01 legacy packages have been started but I didn't get them done in time for a weekend release. Sorry about that. The plan is for the next two days.

Madduck wondered why a package using revision control needed pristine tarballs at all. Firstly, the example he gave of tagging an upstream release and creating a tarball of it to build the package against is exactly what pristine-tar does. Except it preserves permissions and timestamps and other tar cruft. Sometimes timestamp or permissions issues do cause issues with package builds, so being able to build against the original files from the pristine tarball is potentially useful to catch and fix such problems. As to other uses of pristine tarballs... It's hard to list the advantages to using pristine tarballs, since Debian (rpm too) has had them forever, and lots of people may depend on them in ways that we don't anticipate. For example, I've seen FreeBSD point at ftp.debian.org to get tarballs for their packages, since they don't archive tarballs on their own, and sometimes the tarball isn't available upstream (or ftp,debian.org is the upstream). That's a wacky example, but I've seen it more often than I've seen pristine tarballs used for the often-cited md5sum comparison case. Rather than try to enumerate all the possible reasons for pristine tarballs, and then try to whittle the list down to reasons that actually make sense, and cost-benefit analise those, it seemed better to just try to write a solution to the problem. (Or at least, to most of it -- a hopefully small subset of the set of reasons for pristine tarballs are not satisfied by pristine-tar.) I've also been theoretically interested in this problem of generating pristine tarballs from version controlled source ever since Scott Remnant mentioned at a meeting (140 mb video of meeting here) that he had a tool to do it -- and AFAIK never shared the tool! So the challange factor was another reason to write it. However, pristine-tar has already saved me hundreds of megabytes of disk space, sped up my release process (avoiding bug #225483) and simplified the way I work, so it's already a net win for me, even if it turns out we don't need pristine tarballs for anything at all!

Joey writes about the difficulty of dealing with the need of pristine tarballs and version control systems. He concludes that it's best to save the tarballs in the repository and has written a tool to store binary differences to save space to achieve his goal. This being another piece of software by Joey, it's probably worth checking out. But I wonder why we actually bother about pristine tarballs. Sure, our Debian archive scripts require a tarball, but why does that have to be the pristine upstream tarball? As Debian maintainer, I use git to keep track of the software that I package, and if possible, I track the upstream repository directly (Git makes it very easy to track repositories of the other, popular version control systems). When I decide to upload a new upstream version to the Debian archive, I tag the upstream tree, sign the tag, and generate a tarball from it:

git tag -s -m'preparing release of upstream 1.2.3' upstream/1.2.3
git archive --prefix=foo-1.2.3/ upstream/1.2.3   gzip -9 \
  > ../foo_1.2.3.orig.tar.gz

Then I merge the tag into my Debian branch and prepare a Debian release, which I also tag (debian/1.2.3-1) just before uploading the package to our archive. This is very flexible and allows me to release packages based on software snapshots, as I just did with mdadm. Now, a filename of mdadm_2.6.3+200709292116+4450e59.orig.tar.gz pretty clearly states that this is not mdadm-2.6.3.tar.gz, so people would be aware that this tarball is not "upstream-official". But what about official releases, like mdadm-2.6.3.tar.gz? When preparing the Debian package, it's only slightly more effort for me to download the official tarball and store it as ../mdadm_2.6.3.orig.tar.gz, but why do I need to keep it around after I uploaded Debian package 2.6.3-1? I know I need the orig tarball to prepare the diff.gz for 2.6.3-2, but for that purpose, I can really just git-archive the appropriate tag. The result won't produce the same hash (because the tar format stores timestamps and is thus not really appropriate for the cause), but it provides the same contents. In fact, what I (or someone else) should finally implement is a tool to create the diff.gz directly from the version control data or even better, we should finally get rid of source packages as I've previously suggested, long before Ubuntu came around to propose and blueprint the concept. I am aware of some people's preference to be able to "independently verify Debian's orig tarball using e.g. a detached GPG signature provided by upstream," but I dare to question the point of this verification. Those who care enough to verify that the Debian-provided orig tarball is pristine are hopefully building their own binary packages instead of trusting mine (another beef I've raised in the past), and then they might just as well download the source directly from upstream. I'd say most people (rightfully) don't do this and just use the Debian-provided content, which can be assumed to be pristine if you trust me and Debian: my GPG signature authenticates uploads to the Debian servers, and APT provides authenticity checks downstream up until the installation on an end-user system. and if you trust the version control system, and this actually raises an interesting point for me: how feasible would a man-in-the-middle attack between Neil's (the mdadm upstream) repository and mine be, given that I use the plain-text git transport protocol to fetch new commits. If you have any comments on that, please let me know. I'll leave this issue for another discussion or post. NP: The Beatles: Abbey Road Update: Matej Cepl pointed me to this thread on the fedora-devel mailing list, where they discuss switching from CVS to Git and talk a lot about their needs and how the end result is much the same as what I am proposing. Thanks!

I’ve just started getting a lot of traffic referred by live.com. It seems that my post Porn for Children [1] is the second link returned from a live.com search for “porn” and my post Porn vs Rape [2] is the third link. These results occur in two of the three settings for “safe search” (the most safe one doesn’t return any matches for a query about “porn”). A query for “porn” and “research” (which would reasonably be expected to match a blog post concerning scientific research made my page the 8th listing (behind http://www.news.com.au/404, http://www.news.com.au/couriermail/404, and http://www.theaustralian.news.com.au/404). It seems strange that a query which should match my page gives it a lower ranking than three 404 error pages while a query which shouldn’t match my page (no-one who searches for “porn” on it’s own wants to read about scientific research) gives it a high ranking. One very interesting thing about the live.com search is that it doesn’t filter out some of the least effective ways of gaming search engines. For example the URL http://gra.sdsu.edu/research.php has a huge number of links to porn pages and pages that apparently sell illegal pharmecuticals) that are not visible (view page source to see them). The links that I tested were all broken so it seems that the other sites (including http://www.hcs.harvard.edu/~hraaa, http://base.rutgers.edu/pgforms, http://www.wccs.edu/employees, http://base.rutgers.edu/mirna, http://www.calstatela.edu/faculty/jperezc/students/oalamra, and http://institute.beacon.edu/), were cleaned up long ago. There is probably some money to be made in running a service that downloads all content from a web site and/or has a firewall device that sniffs all content that is sent out and makes sure that it seems to be what is intended (about half the URLs in question appear to relate to content that is illegal under US law). As an aside, I did a few other live.com searches for various sites and the word “porn” and found one Australian university running a forum with somewhat broken HTTP authentication that has some interesting posts about porn etc. I’m not going to provide a link because the content didn’t appear to violate Australian law and you expect some off-topic content on a forum. But to be fair, live.com have significantly improved their service since last time I tested it [3]. Now a search for “bonnie” or “bonnie++” will give me the top two spots which is far better than the previous situation. Although I have to admit that the Google result of not giving me a high ranking for “bonnie” is probably better.

• [1] http://etbe.coker.com.au/2007/08/12/porn-for-children/


• [2] http://etbe.coker.com.au/2007/07/30/porn-vs-rape/


• [3] http://etbe.coker.com.au/2007/04/02/livecom-malware/

Share This

I had one of these devices collecting dust in my basement. I finally got around to reverse engineering a userspace driver for it, which was much easier than I expected. It just shows up as a usb->serial device, and it was mostly a matter of of watching the serial line while hitting buttons. I have started to figure out some of what I need to know to communicate with it in the other direction in order to do stuff like turn on the lights and manipulate the lcd displays. That stuff is not yet, however, functional, but in the spirit of "release early, release often", I've uploaded a 0.0.1 release to the project page on sourceforge. Also included is a python extension which allows you to register callbacks for the various controls on the device. After I get the leds and lcds working, I think the next step will be to write a program to generate synthetic MIDI events when the device is manipulated so I can hook the thing up to something like mixxx. Hopefully someone else has one of these kicking around and might also find this work useful!

I had one of these devices collecting dust in my basement. I finally got around to reverse engineering a userspace driver for it, which was much easier than I expected. It just shows up as a usb->serial device, and it was mostly a matter of of watching the serial line while hitting buttons. I have started to figure out some of what I need to know to communicate with it in the other direction in order to do stuff like turn on the lights and manipulate the lcd displays. That stuff is not yet, however, functional, but in the spirit of "release early, release often", I've uploaded a 0.0.1 release to the project page on sourceforge. Also included is a python extension which allows you to register callbacks for the various controls on the device. After I get the leds and lcds working, I think the next step will be to write a program to generate synthetic MIDI events when the device is manipulated so I can hook the thing up to something like mixxx. Hopefully someone else has one of these kicking around and might also find this work useful!

Keeping pristine upstream tarballs around is a pain. Wouldn't it be nice to be able to keep them in revision control? Except, it would use far too much disk... Here's a solution. It generates an binary delta between the pristine upstream tarball and a tarball created using files checked out of the repository. The delta should be quite small, and any checkout of the repository that has the same file contents as that used to create the delta can be used to regenerate the pristine tarball. Example (which would presumably be more fun and faster if I used git):

joey@kodama:~package/uqm-voice>svn switch svn+ssh://uqm.debian.net/srv/svn/uqm/tags/uqm-voice/upstream_version_0.3
D    debian
A    comm/blackur/black041.ogg
U    comm/shofixt/shofi040.ogg
U    comm/starbas/starb182.ogg
D    comm/slyland/slyla030.ogg
U    comm/chmmr/chmmr035.ogg
U    comm/supox/supox031.ogg
Updated to revision 223.
joey@kodama:~package/uqm-voice> pristine-tar extract ~/uqm-voice.delta ../uqm-voice_0.3.orig.tar.gz

The generated tarball is bit-identical to the 19 MB upstream tarball, though you have to gunzip them both to check this, since the gzip compression differs. The delta file is all of 41k large. The uqm-voice.delta file was created earlier as follows:

joey@kodama:~> pristine-tar stash lib/debian/unstable/uqm-voice_0.3.orig.tar.gz uqm-voice.delta

I've uploaded pristine-tar to incoming.

The tricky part of using a git (or other rcs) archive as distribution source package format is handling pristine upstream tarballs. One approach would be to try to create a git archive that didn't include objects present in the upstream tarball. Then, to unpack the source package, you'd unpack the upstream tarball, convert the files in it into git objects and add them into the .git directory. This way a pristine upstream tarball is available for verification, and it also has the same properties of not needing to re-upload the source for every package release as the current Debian source format. This seems like it might be possible to implement, but you'd need to know quite a lot about git internals to remove the redundant objects from the git repo and regenerate them from the tarball.

---

Another approach would be to keep the pristine upstream tarball in the git archive, and then the source package would consist entirely of the git archive. This doesn't have the same nice minimal bandwidth upload behavior -- unless you can git push your changes to do the upload. Storing a lot of upstream tarballs in git wouldn't be efficient, but I've written pristine-tar to take care of that. :-)

Recently, I added some photo related blogs to my reading list, amongst them strobist, and so I came across a post about the Open Source Wireless Trigger. Being interested in both electronics and free software, I read through most of the posts on the forum as well as the wiki. I also started experimenting a bit with an old Nikon SB-24 flash I had lying around since even if I managed to break it, it wouldn't be that bad. And it's old and robust. So far, I've played around with how to make the flash go off. For the hot shoe, it's just shorting the ground and center pin (I just used a big resistor). For the PC connector, I had to apply a bit of voltage; I used an AA/LR06 battery and that worked splendidly. (Oh, and PC in this context has nothing to do with computers, it's an abbreviation of Prontor-Compur, two shutter manufacturers who decided on the design of the connector back in the 1950s.)

From time to time the question comes up on Debian mailing lists of whether debian directories in upstream tarballs are harmful and what do do about them. I tend hide from such threads, since all upstream tarballs I create contain debian directories. But I realised tonight that the common wisdom from those discussions is wrong, or at least incomplete. Here's why. First, it's important the notice that this is not a problem in the rpm world. Putting an rpm spec file in an upstream tarball is considered good practice. There's no downside; if a packager needs to change how the package is put together, they can replace the spec file. And users can run rpmbuild on the tarball and get an rpm spit out. Handy. So, then, why is the equivilant practice a problem in the debian world? Three intertwined reasons:

1. The tools assume all files in debian/ are good. There's no way to make debhelper not install a debian/init script, or any similar file. Note that this problem only affects helper tools like debhelper; the core deb building tools only care about a few files that are in every debian directory. Effectively, debhelper, and (possibly) cdbs are the only tools with the problem currently.
2. The source package format doesn't support removing files from the upstream tarball. This is a pretty silly limitation that any source format better than the current one should override. If wig and pen was ever implemented, it should support some means of deleting files.
3. Package maintainers fear that they can't get upstream to make changes, or not fast enough, so worry about getting into a mess with files in an upstream debian/ that can't easily be fixed.

I feel that if any of these three problems is solved, an upstream debian directory is ok. #3 can be solved to some extent for some packages, though these solutions will always be the least satisfying. It's tempting to try to solve #1 in debhelper. I could add an --ignore switch, to make debhelper commands not act on a given file in the debian directory. (Update: done in version 5.0.57!) The best place to solve it, though, would be in the source package format. I'd be happy to see wig and pen be implemented, but would be happier to see us moving toward using a distributed revision control system as a source package format. Consider a source format that replaces .diff.gz with .git.tar.gz.

This project that sounded like a joke when a French person announced on April 1st that he plans to review English in some texts of Debian apckages, is now well going on. Contrary to what the status page shows, this is far from being a single-person project. I receive invaluable help from a few people, with a very special mention to Justin B Rye and MJ Ray. Both of them (and a few other people, including the package maintainers) carefully review my rewrites for the considerable amount of Frenglish I introduce. One of the funniest part is reading both of them sarcastically commenting on the "dialect" we're using as we decided to go for US-style spelling and writing style (by far the most used in Debian) while MJ and Justin are definitely citizens of the UK (and very clever in GB English). But both "play the game" ('jouer le jeu' as we way in French) very constructively. I think that the improvement for packages at the end of the process (a very long one as it includes translation updates for debconf templates) is noticeable and nearly always much appreciated by package maintainers. We are indeed only limited by time. I have decided to begin with packages that use debconf (a few hundreds) and I cannot start more than 1 package per day (the initial review is very time-consuming: about 20 minutes for one package and up to 1 hour for some of them). So, of course, new contributors will be welcomed (even those speaking dialects!).

One of the things through this move has been that we've been trying to make sure that whatever place we live in is close enough to "stuff" that Angie doesn't go nuts. The bitch about corporate housing is that there's no point in trying to meet your neighbours - everyone is transient. While I never came home to the walls smeared with f ces but I think she was contemplating it.

So, we're in new corporate housing now in Mountain View. This time we're a block off of Castro and El Camino Real, several neighbours have said hi, there's other families around and a starbucks around the corner. I'm expecting she'll be much happier here.

And I found a store that sells lychee-fruit cups. I haven't had those since I left Vancouver! The salt-water taffy store's products aren't vegan, sadly.

Adopting w3-recs (i.e. experiences in package generation) A week ago or so I decided to adopt w3-recs, a documentation only package aiming to ship recommendations of the W3C in Debian. The previous maintainer (Frankie) step down from w3-recs maintenance since the packaging work was quite boring (add recommendations piecewise as soon as they came out, update them when needed, keep doc-base entries in sync, ...). I didn't want to follow that path and, probably for the first time in my life, the (not so) semantic web has come to the rescue. Starting from the TR automation page, I've found an always up to date index of all Technical Reports published by W3C, enriched with some information about the status of the TRs (draft, candidate recommendation, recommendation, ...) which is in RDF/XML format, and hence machine processable. The final result is that now the w3-recs package is mostly automatically generated via debian/rules target. The get-orig-source target first downloads the RDF index of the technical reports and then downloads all of them via wget (it took me sometime to find the appropriate flags, I've ended up with a award-winning wget command line containing 12 dash switches!). Then, at build time, several derived stuff is generated from the very same RDF index via XSLT: doc-base entries for all the shipped documents (the current grand total is 119 recommendations), a HTML index summarizing the available documents, and other debian/ stuff needed later on in the package build process. The next steps for w3-recs should be automatically split of the package in sub-packages classified according to the corresponding W3C activity, and enrichment of the doc-base entries with abstracts. Unfortunately neither of the 2 kinds of information are provided yet in the RDF index (not so semantic web, after all). Infrastructure deficiencies encountered in the process

• wget is not that good at mirroring: I've been bitten by a lot of undocumented behaviours and bugs (e.g.: undocumented ordering requirements among command line flags, fuzzy timestamping, bugs in a posteriori URL rewriting when a destination base dir is specified, ...). I've found no better alternatives which can basically just mimick what firefox does when you save a "complete" (set of) web page(s) on disk
• dhelp is just completely broken. It is not able to deal with some 100 doc-base entries, it just shows some of them (about 10, do not dare to hope in more) and even trims the title lines. Luckily doc-central is much better. At a times I find myself thinking that I'm the only believer in Debian's doc-base ...
• Oh, and while we are at it: I'm still convinced that having the doc-base hierarchy bound to the menu policy is just plainly dumb

I have no idea how the other major MTAs handle this, but sendmail seems to issue an ident-auth-request on each connect it gets.

As i think this is rather useless, as most hosts don't answer it, and, even if they would, the answer is easily fakeable, i switched it off.

adding

define( confTO_IDENT', 0')dnl

to /etc/mail/sendmail.mc disables this. Sadly this isn't documented, for confTO_IDENT it only states

The timeout waiting for a response to an IDENT query.

thanks goes to http://sial.org/howto/sendmail/tips/

addendum: 'Your Name' in the comments is right. ident is still of some use on multiuser-systems to the local admin. But in the field of Internet-mailservers it is useless.

I just upgraded to Wordpress 2.3. When using Konqueror (my favourite browser) the comment approval is slightly broken (when I tag a comment as spam it usually just turns red and doesn’t disappear from the main Comments tab) and I have to refresh that window more often than usual to make sure I got the result I desired. Also the Sidebar Widget editing is totally broken in Konqueror, I guess I’ll have to login with Firefox to get the new tags feature working. Also I have got a few Wordpress errors about the table “$table_prefix” . “post2cat” not existing. The table in question doesn’t exist in any of my blogs. So far this is the worst Wordpress upgrade experience I’ve had (I started on 2.0).

Share This

(or: Why Everyone Should Learn Some Basic Programming) I am an organisational disaster area. I just can't remember "event oriented" information (like "I need to do $foo" or "I need to do $bar at $time"). I think I've accumulated such a huge collection of technical trivia that it's taken over the event planning part of my brain -- although, since I've never been able to remember event oriented information, even before I became a computer geek, it's more likely that I was just built without that bit of the brain. So I guess I'm the perfect target market for some sort of external brain pack (a la "Ratbert the Consultant"). I've tried paper-based organisers (the mighty Filofax), a few pre-written apps, and I even bought an SL-C3200 Zaurus a while ago (although, to be fair, that was really just for the hack factor, regardless of what justification I may have given to others at the time). None have worked, for various reasons. I can't grep paper, and Other People's Apps bug me because they've got their idea of workflow, and I've got mine, and when they don't match up (and they never do) I just get irritated -- not to mention the fact that the ubiquitous relational database, while wonderful in some ways, does tend to get in the way of ad-hoc faffing with your data. But this article is a success story, not a rant (gasp!), so it's safe for you to assume that I've got something that's working for me. My method is real simple -- plain text files, managed by vim-outliner and a simple shell one-liner I've named 'pda', which just runs vim ~/pda/$1.otl. At the moment I've got a monster todo.otl, which is my world's todo list, and a deeply underused calendar.otl. I'm also using vim-outliner to do some project planning, which has been moderately successful, although I'm a bit freaked out by the idea of trying to manage a large-scale project using it. The really, really cool part of this whole shebang, though, is that because all of my data is (structured) plain text, writing little programs to parse the data and do funky things with it is dead simple. For example, in my project outlines, I put estimated, actual, and remaining times in for all tasks (a la Joel Spolsky's Painless Software Schedules) so I can keep track of my estimates and where they differ from reality (because one day I'd like to not Suck at estimating times, and feedback is the only way I'm going to do that). So, about an hour's worth of Ruby gave me "estimator", a script that reads all that data and does a pile of mathematical gubbins about how far over my estimate I am so far (or under estimate, if that were to ever happen), the percentage of the project complete (based on the estimates and number of tasks completed), and things like that. Similarly, today I wrote a small script (run out of cron in the wee hours of the morning) to read my calendar and send me an e-mail if I've got anything in there for today. Since I practically live in mutt, a morning e-mail is going to be far more effective than my trying to remember to check my calendar every morning (especially since, at present, I'm not using the calendar as often as I should, so there's even less motivation to check it). This, in my opinion, is why everybody (literally, every single person) should learn to program at some fairly simple level, like learning to read and write. By using plain text and small scripts, I've got a workflow that works for me, and it's cost me less time than I'd spend learning some large pre-written app and putting all my data into it. Being able to manipulate data like this is, I think, a fairly important tool in the modern world, and I don't think it should be left to any sort of priesthood of developers -- it should be as universal as most countries try to make literacy and numeracy.

Apparently Scott James Remnant in his article on version control systems confuses arch and git. One can only speculate that his short git usage stems from the pre 1.0 days, where you had to use higher level tools (called porcelain) to happily work on git. A funny anecdote is that Scott back in his dpkg hacking year promoted arch heavily. Ubuntu^Wbzr propaganda spreads speed gains as big bonus of the last major releases. In order to be able to do that you have to start with a terrible baseline. Testing bzr on middle sized repos is no fun at all. The bzr pain inside launchpad must be beyond imagination. Nowadays it is much easier to hack on mdadm than on lvm2. The reason is that later project uses rusty cvs. With git it is really easy to contribute back. Either you mail the patches or publish your repository. git will help you along on each way. The other very big bonus of git is the big community around git. It is a community excited around building and delivering the best version control system. The git development does regular surveys on git usages and incorporates back the wishlists. The most "funny" way to use git is to run it as cvsserver. You may believe me or not i have seen git cvsserver emulation usage in the wild.