Exactly one year ago today, I wrote about Git, Mercurial, and Bzr. I have long been interested in VCS, and looked at the three main DVCS systems back then.

A Quick Review

Mercurial was, and for the moment, remains, my main VCS. Bzr remains really uninteresting; I don't see it offering anything compelling that Mercurial or Git can't do. My Git gripes mainly revolved around its interface and documentation. Also, I do have Windows people using my software, and need a plausible solution for them, even though I personally do no development on that platform.

Ted Tso wrote his own article in reply to mine, noting that the Git community had identified many of the same things I had ans was working on them.

I followed up to Ted with:

... So if Ted's right, and a year from now git is easier to use, better documented, more featureful, and runs well on Windows, it won't be that hard to switch over and preserve history. Ted's the sort of person that usually is right, so maybe I should starting looking at hg2git right now.

So I guess that means it's time to start looking at Git again.

This is rather rambly, I know. It's late and I want to get these thoughts down before going to sleep...

Looking at Git

I started at the Git wikipedia page for an overview of the software. It linked to two Google Tech Talks about Git: one by Linus Torvalds and another by Randal Schwartz. Of the two, I found Linus' more entertaining and Randal's more informative. Linus' point that CVS is fundamentally broken, and that SVN trying to be "a better CVS" (an early goal of svn, at least) means it too is fundamentally broken, strikes me as quite sound.

One other interesting tidbit I picked up is that git can show you where functions have moved from one file to another, thanks to its rename-detection heuristic. That sounds really sweet, and is the best reason I've yet heard for Git's stubborn refusal to track renames.

The Landscape

I've been following Mercurial and Darcs somewhat, and not paying much attention to Git. Mercurial has been adding small features, and is nearing version 1.0. Darcs has completed a major overhaul both of its repository format and internal algorithms and is nearing version 2.0, and appears to have finally killed the doppleganger (aka conflict spinlock) bug for good.

Git, meanwhile, seems to have made strides in usability and documentation in its 1.5.x versions.

One thing particularly interesting to me is: what projects are using the different VCSs. High-profile projects now using Mercurial include OpenSolaris, OpenJDK (Java 7), and Mozilla's projects. Git has, of course, the Linux kernel. It also has just about everything associated with freedesktop.org, including X. Also a ton of Unixy stuff.

Both Mercurial and Git communities are working on TortoiseHg/TortoiseGit types of GUIs for Windows users. Git appears to have a sane Windows port now as well, putting it on pretty much even footing with Mercurial and Darcs there. However, I didn't spot anything with obvious Windows ties in the Git "what projects use git" pages.

The greater speed of Mercurial and Git -- even for pushing and pulling small patches -- likely will keep me away from Darcs for the moment.

Onwards...

As time allows (I do have other things keeping me busy), I plan to install git and work through some tutorials and try to use it in practice as much as possible, to get a good feel for it.

Future

It is beneficial to be using a VCS that is popular, though that is certainly not a major criterion for me. I refuse to use SVN because its lack of distributed functionality makes it too unproductive to be useful. But it looks like Git is gaining a lot of traction these days, especially in Debian circles, which also makes it more interesting.

I notice that Ted did convert e2fsprogs over to git as he said he might, incidentally.

Wanted: a device that will rewind time, turn off greylisting in my mail server, wait for the other side send the mail, and then resume business as before.

Sometimes a bug report is a labyrinth. #348046 is an example of this. It is a horrible mess of at least three different issues with half of the original participants having become unresponsive. I would like to pull the issues apart into different bug reports to be able to deal with them (and their probably unresponsive submitters) individually. Obviously, cloning and renaming is not an option since this copies the mess. So, it would probably be desireable to download the bug mbox and to bounce individual messages to new bugs (that have been created before), but the BTS recognizes the dupes and bins them. Blars has helped me by looking at BTS mail log, so it was clear that removing the X-Loop, X-Debian-PR, X-Spam, Resent- and Received headers from the mbox before loading it into mutt to do the actual bouncing works fine. A command line to do this:

rm -f mboxout; mboxbug formail -d -I Received -I X-Debian-PR -I X-Loop -I X-Spam -I Resent -s >> mboxout

After trying this in “production”, one needs to send one message per BTS pulse, or one will totally mess up the order of the messages. That’s a real pity and an annoyance.

I will see you all at my talk, right? ;)

Yesterday witnessed the most important step forward for the Definition of Free Cultural Works (DFCW) since its adoption and endorsement by the Wikimedia Foundation a year ago. Although I might have wished things otherwise, Creative Commons is not a social movement fighting for essential freedom or the essential freedoms at the core of the DFCW in particular. From the movement's perspective, CC is more like a law and advocacy firm that works for us -- a very sympathetic one. CC writes, hosts, and supports a variety of licenses. Some are free. Some are not. Last year they took steps to explicitly limit the extent of restrictions they are willing to tolerate in their licenses. Yet, while CC has resisted taking a stand in favor of the Definition of Free Cultural Works, they continue to produce some of the best free licenses, tools, and metadata available and they seem honestly interested in helping users interested in social movements based around these definitions organize more effectively. In perhaps its most important move to date in this area, Creative Commons announced yesterday that it was placing a seal on each of its licenses that provide the essential freedoms laid out in the Definition of Free Cultural Works. The seal links to the definition over at freedomdefined.org. In Creative Commons' words:

This seal and approval signals an important delineation between less and more restrictive licenses, one that creators and users of content should be aware of. A very practical reason users should be aware of these distinctions is that some important projects accept only freely (as defined) licensed or public domain content, in particular Wikipedia and Wikimedia sites, which use the Definition of Free Cultural Works in their licensing guidelines.

The seal is currently on two CC licenses that provide for essential freedom (Attribution and Attribution-ShareAlike) and their public domain dedication. Thanks go to Erik Moeller at the Wikimedia Foundation and everyone at Creative Commons to helped make this happen.

After about two weeks I'm finally able to come back to Debian, which feels good. After my last package upload was botched (and gracefully recovered by my teammates) I realized that I needed to step away for a little bit.

I'm currently in my fifth year of my PhD program. For those who don't know how such things work in the US, I have a small committee of people (my thesis committee) who act as advisors and gatekeepers on my work. They're the ones who gauge my progress, and ultimately give me permission to write my thesis and receive my doctorate. I'm supposed to meet with them every 6 months, and it had been over a year since my last meeting. Because I'd like to actually graduate one day, I was rather nervous about having the meeting. I was more nervous than I realized, and the botched upload reflected that.

Anyway, to make a long story short, the meeting went as well as I could have hoped for and my committee seems to be happy with my progress. I'm looking to get a paper out in the spring, and they'd like to meet with me again in three months time (which will absolutely fly by) to assess where that paper is at, and how things are going. At that point, I wouldn't be surprised if they tell me to write my thesis and graduate.

What I do after my dissertation work has been weighing on my mind for years and years, and it's obviously coming to a head. My boss will let me stay on as a post-doc in the lab for a while, which I may do depending on other job opportunities and progress on my projects. Ultimately though I need to make a decision that's familiar to a few others in my position. Do I stay in science or go try and work on free software for a living? I honestly don't have an answer yet, because I love them both.

I see tons of incredible opportunities in my field right now, and while I'm not the only coder interested in synthetic biology, I'm one who's positioned to get in on the ground floor of the science in a very real way and have a significant hand in shaping its future. On the other hand I spend my free time devouring computing literature, not biology literature, as well as producing software and things related to it. I do this because I enjoy it, and I know I enjoy it on a deeper level than the biology. Would I enjoy it this way if I did it for a living? I've got no idea, since I've never coded professionally.

I'm coming up to the fork quite fast now, and I'll have to choose soon. I know I'm lucky because I do have the chance to choose for myself, but that doesn't make it any easier.

For various reasons, I usually carry an USB stick with me that holds a single ext2fs and has grub installed. This blog entry quickly documents how to copy a Debian-Installer to it to be able to quickly install Debian without the need to burn a CD. I had tried this stunt last year already, but without success. I don’t know whether Debian Installer was not able to boot from an USB stick back then or if I was just too stupid to do things right. This procedure was done with today’s d-i daily image, so I guess that it will work this way in the future. I’d like the USB stick to be useable for other things even if d-i is available on it as well. So I would have liked d-i to be confined to a single subdirectory, which is possible with today’s code. The Installation Manual is rather verbose about booting from USB stick. The only challenge is to find the Kernel and the initrd files that are needed to actually boot. The exact path to these two files is not in the manual, but the nice people on #debian-boot confirmed that these are the correct ones. The second challenge was a non-issue for me since grub was already installed and functional on the USB Stick (its installation was also surprisingly straightforward). Next step: Create a directory on the USB stick and copy vmlinuz, initrd.gz and an arbitrary Debian CD image there. I used the daily netinst image. root (hd0,3); kernel /d-i/vmlinuz expert; initrd /d-i/initrd.gz and boot That’s it. The installer boots, finds the CD image, mounts it and pulls its components from there. The rest is a normal, stock Debian installation. Debian Installer rocks. It nearly always does the right thing.

I'm (again) writing a library that is supposed to be cross-platform, but needs sort of an event loop, which I'd rather leave in the main program and just give them a bunch of file descriptors to watch and a timeout. Problem is, file descriptors are platform dependent. There are a few approaches to this. The mighty ASIO simply annexes the entire loop, and just tells people to hook into it if they need anything -- it works, but on the downside you get threads for free when you didn't want or need them. Simply spawning a new thread for everyone isn't a solution either, as it requires me to do lots of synchronised inter-thread communication -- and on Windows, file handles are thread affine, so I cannot even call into the same library from different threads. I will probably go with ASIO for this project, but I'm fairly certain that there are better solutions out there.

For those who are not going to FOSDEM, the DebConf video team aims to provide live streams of all talks in the Debian dev room as an ogg theora stream. I hope it will work out :-)

Currently there is a single server behind that URL. If we see lots of viewers we can add another server with round robin dns easily. Please submit your offer in the teams irc channel or on the mailinglist, if you already operated at streaming server for DebConf7 in Edinburgh that would be best.

And if you do attend FOSDEM please consider helping the video team, there are still about 20 positions to cover (operating the camera, the audio mixer and the video mixer). All those jobs are easy and fun, if you don't have to do them for the whole conference ;-)

One of the interesting things about Google's Android project is the mix of cultures. There are a number of people like me, firmly grounded in the open source world, who are happy to get something that 1) works, 2) looks to have legs in terms of popularity, and 3) will be open source. Think of a lot of your favorite open source projects when they first launched. "Polished" isn't a word you would probably use to describe them, although it usually happens in later releases (Ubuntu looks better and better, for example). There are also a lot of people who come from a 'commercial' background, who expect to see everything ready and waiting on a plate for them, which isn't a mistaken expectation if you're used to using products from big companies who are staking their reputations on them. They want nice, thorough docs for everything, almost all bugs worked out, and lots of easy to use tools. Android lies in a middle ground somewhere between these two camps, which I think at times makes people uneasy. It's not open source yet. You can't hack it as much as you may want. You can't rebuild it from the ground up. You don't get access to the communication between the developers working on it, and cannot participate in that development. However, there is a promise to open source it that would reflect very, very poorly on Google were it not met and is therefore pretty credible. The developers and many people involved with it certainly seem to "get" open source, and are taking good steps to create a community around Android (a few even hang out in the #android IRC channel on Freenode) that will most likely not take too long to start assimilating interested outsiders once the code is open. In terms of quality, it wasn't bad when it came out, but it was very definitely an "early look", put out there to solicit feedback and change according to it, something that's part and parcel of a release early, release often open source model. On the other side, Android looks to be a big effort from a big company to get into a big market in a big way - it's not a "under the radar" type of strategy that the open source world is often constrained to follow for lack of monetary and marketing resources to do it any other way. This has attracted a number of developers who either don't see or don't really get some of the "open source" aspects of what we have so far: it's still changing! The docs aren't complete! The emulator is still ugly in places! And from their point of view, it must seem odd to getting something that's obviously not a finished product, without crystal clear direction and goals and a roadmap, and without actual hardware only available in the distant (well, in internet terms! - 8 months or so) future. That kind of thing probably isn't done very often in their world, or where it's done, is certainly kept quieter and perhaps restricted to registered developers. They came over to have a look at something that was ready to go, and it isn't, yet. This has caused some disappointment for people who expected to see something ready to go out of the box. Perhaps Google needed to manage their initial communications slightly better in order to manage the second group's expectations more carefully. In terms of us open source folks, I think that what we want is more access, as soon as Google is willing to hand it out. More source code access, but also access to the development process and people. "Real" open source isn't just about the code, it's about being more or less equal participants in a community. Luckily, I think the people at Google get this, which is one of the reasons I'm enthusiastic about Android, even if I wish we didn't have to wait so long for the source code.

Watch the youtube video of the n810 with a tilt device.It's pretty cool to see Enigma running on the n810. The tilt device seems to be suprisingly good for playing Enigma, I wouldn't have expected that - Enigma is heavily designed to be played with a mouse.Congrats to the tilt stick makers and the Enigma porters for the n810.There will be a new release of Enigma in a few months, with serious engine improvements (several algorithms have been replaced with more efficient ones, leading to serious speed improvements in certain levels!). The code got a pretty big overhaul by the main developers, and the level API was also redesigned, so this will really be a major thing.

I'm employed at Solarflare Communications in the test group, supporting the work of testers and developers by developing test frameworks. (We actually have very few pure testers and a whole lot of automation.). In a major break from my previous jobs, I've been working mostly in Python on applications for use in-house. More recently I've become involved in maintenance of our network driver for Linux, starting with updating it to use current kernel coding conventions, and submitting for inclusion in the kernel. This has become a bit frustrating because while there are a lot of people prepared to run checkpatch and pick nits, there are not so many who will do a thorough review of a driver that's 20,000-odd lines long. And without someone doing that, it won't go in. Meanwhile some of my colleagues have done some great work to accelerate Linux networking on Xen by having drivers in dom0 and domU cooperate to expose hardware resources directly to domU. Our hardware architecture supports many virtual NICs with their own packet queues and set of buffers, so this does not compromise the security of the physical machine and dom0. The results are very impressive. The original motivation for the virtual NIC architecture was as a basis for a user-level networking stack, which can provide very low latency and reduced CPU usage. This software was originally proprietary but has now been released under GPLv2 as OpenOnload. Google invited two of my senior colleagues to talk about this - here's a video of their talk about user-level networking.

I'm going to admit up front here that I'm pushing my luck, and that I anticipate the chances of success are minimal. But that aside .. There are a lot of people who read my entries, because of syndication, and I'm optimistic that somebody here in the UK will have a copy of the following three books they could send me:

• Flash Gordon vol 3: Crisis on Citiadel II
• Flash Gordon vol 5: Citadels under attack
• Flash Gordon vol 6: Citadels on Earth

(All three are cheap paperback pulp fiction novels from the 1980s written by Alex Raymond.) If you have a copy of any of those three books, and are willing to part with them, then I'd love to hear from you. Either as a comment or via email. I'm certainly expecting to pay for them up to around £5 for each volume.

Backstory: I read the first when I was 10-12, then mostly forgot about it. A while back I remembered enjoying it and bought volumes 1, 2, 3, & 4 from an online store. I got screwed and volume 3 hasn't arrived, but possibly that will be rectified soon. Here in the UK the last two volumes are either extremely rare or extremely in demand. Typically they seem to sell for £15-30 - I'm frustrated to not have the conclusion, but not desperate to spend so much money upon them, (been there, done that).

So if anybody has some or all of these books and can bear to part with them please do let me know. </luck:pushing>

I've two video-related queries, which I'd be greatful if people could help me out with:

Mass Video Uploading

Is there any tool, or service, which will allow me to upload a random movie to multiple video-download sites? Specifically I'm curious to learn whether there is a facility to transcode as necessary a given input file and then upload to youtube, google video, and other sites as a one-step operation.

Mass Video Searching

Relating to that is there a service which will allow me to search for vidoes with given titles/tags/keywords across multiple video-hosting networks?

Regarding the searching I see that YouTube has support for "OpenSearch", but Google's video hosting has neither that nor a sitemap.xml file: Irony Is ...

Although an interim release, there are a number of new features in 0.8.2 that are worth noting.

1. emrecent - possible candidate for a rename to emdput, this script started off just as a review tool for recent builds (hence the current name). Finds recent .changes files in the Emdebian working directory, collates them, provides a viewer for the debc -a $ARCHcontents using whiptail or zenity along with error logging. If the default zenity frontend is installed, you can still select to use whiptail (e.g. when over an SSH connection) using the -w option to emrecent. Further checks are run using the second script:


2. emdebcheck - a local Emdebian version of the debcheck that runs on the Debian archive using edos-debcheck to highlight non-installable packages before an upload breaks the archive. This isn't such a problem in Debian but Emdebian needs to build installer images and similar tools from packages in unstable so an installability check is very useful. Certain packages are expected to be non-installable due to dependencies that are removed in Emdebian so emrecent supports a --skip option to omit the emdebcheck run for such builds.


3. emsource --status now reports errors logged by emrecent or emdebcheck.


4. Unpacking support in emsandbox - EXPERIMENTAL!. debootstrap --foreign puts a higher-than-necessary burden on the host device when running the default --second-stage - not just in CPU workload but in the amount of free space required to copy and unpack all the .deb files. The original debootstrap tarball contains the downloaded .deb archives and only a few packages are actually unpacked manually. Unpacking support in emsandbox adds a handler using dpkg to unpack ALL .deb files in .$BUILDPLACE/var/cache/apt/archives/ using dpkg -e and dpkg -x, creates a suitable status file for dpkg and removes the .deb packages for a smaller tarball. The result is (almost) a foreign filesystem image that can be copied directly onto the host device. The remaining tasks are only:
   • /sbin/ldconfig
   • /usr/bin/dpkg --configure -a
   . Depending on how well this works in real situations, unwrap could become the default handler for debootstrap --second-stage --foreign in emdebian-tools. Unpack support is enabled via the debootstrap suite scripts which define the package set, the first_stage_install and the (much reduced) second_stage, replacing the sequence of install commands with a single function provided by emdebian-tools. See /usr/lib/emdebian-tools/emdebian.gpe for an example.

In common with all previous interim Emdebian releases, 0.8.2 has been uploaded to the Emdebian tools repository already added to your apt sources when installing emdebian-tools so apt will be able to upgrade seamlessly.
(0.8.0 introduced a check in emsetup to ensure that the latest version has actually been installed.)
Once features have stabilised (probably after testing at Fosdem), I'll upload 0.9.0 to Debian unstable, as usual, or maybe one or two more interim releases to Emdebian.

Around the end of 2006 I come up with the idea of setting up something I called font-machine: a collection of scripts / utilities, intended to improve quality of fonts by defining and implementing automatic tests able to catch errors and provide screenshots. Some pioneering work on this area was done a while ago by Miriam who set up a page showing how fonts currently in Debian look like. I did something similar focusing on the fonts used by the Debian graphical installer (will write a separate post to describe the way it works). I'll start describing the tools which I've used more frequently; of course my preference goes to the ones which can be run non-interactively: those that can be invoked inside shell scrips and keep your CPU busy! fntsample is probably one of the programs I've used more frequently during the last months: it was written by Eugeniy Meshcheryakov, who also made it available as a Debian package, and creates really nice and professional PDF / PS charts showing all the glyphs contained in a particular font file. The following code snippet shows how you can use fntsample to create pdf charts for each of the ttf files contained in ttf-dejavu package (will often refer to this package since I maintain it :-) ):

font="http://ftp.de.debian.org/debian/pool/main/t/ttf-dejavu/ttf-dejavu-core_2.23-1_all.deb"
deb=$(basename $ font )
wget $font
dpkg -x $ deb  $ deb/.deb/ 

outdir="out"
mkdir $outdir
for ttf in $(find $ deb/.deb/  -name '*.ttf') ; do
pdf="$(basename $ ttf    sed -e "s \.ttf$ \.pdf ")"
fntsample -f $ ttf  -o $ outdir /$ pdf  -l > $ outdir /$ pdf/.pdf/.outline 
pdfoutline $ outdir /$ pdf  $ outdir /$ pdf/.pdf/.outline  $ outdir /$ pdf 
done

Note that PDF files will have useful outlines (aka bookmarks) courtesy of the pdfoutline tool shipped with the fntsample package. Results are IMHO very imperessive! PS: looking at the above chart for Cyrillic, I've just noticed the glyphs are ordered as to read "SEX"... I did not choose that particular chart snippet on purpose!

This morning, right before leaving for work and somewhat in a hurry, I created the list of keys for the FOSDEM '08 GPG key signing party. You'll find it at the keysigning website. Unfortunately, I fucked up a bit in that I forgot to check my mailbox. Apparently the key server was down for most of tuesday, and I didn't notice. Some people tried to send their keys, but couldn't. Also, the mail wasn't double-checked carefully enough. Grmbl. Next time, I'll try not to forget to do such things when I actually planned them—in the evening. Anyway; if your key has been received in time by the keyserver, you should have received an email this morning, and you should be in the list that I linked to, above. If you weren't then all is not lost; if you really want to participate, print out key fingerprint slips, and bring them to the party. This is not as efficient, and some people may choose not to accept such fingerprint slips, but at least you'll be able to get some signatures. If you didn't make it because the key server was down last night, I apologise (but then again, waiting for the last moment to participate isn't very smart...). Please also note that we only have two hours, and the list contains 202 people. As such, it may be the case that we won't be able to finish the entire key signing party; it may be the case that we'll have to cut it short. In any case, you will be able to sign a significant amount of keys, and a significant amount of people will be able to sign yours, so that should be good enough -- and you can do the rest next year, or so...

The notice for the February meeting was posted on Monday. Late again - it's meant to be a week's notice, according to this. I'm disappointed that the current SPI board still hasn't got basic member communications working, even as the year end approaches. Is a default cron-job mail (like we use at TTLLP ) so difficult to set up? Anyway, one of the few interesting items on the agenda is

"Decision on new board member"

but I've not applied for the post for three reasons. Firstly, on reflection, enough SPI members placed me as last choice (which I think is mostly based on old mud-slinging, but it still happened) in its last election that I don't think being appointed would give me a strong enough mandate for the changes I want (more transparency, consultation and work towards meeting the Better Business Bureau standards). Secondly, I'm finding that my Cooperatives-SW board post is taking more time this year. Finally, I hope to be appointed to a representative post in another organisation soon and it's nearly always better to work where I'm obviously wanted. I've asked if the applications of any candidates for the SPI board post will be posted for member review somewhere (because members are supposed to oversee the board, but how can that happen if we don't see basic info on them?) and for there to be a members Q+A during the meeting, but I don't know whether there's more than one candidate or whether the Q+A will happen. Members are a bit in the dark on this one. Again. I hope to see some other SPI members online about 1900Z tonight (20 Feb). I should actually be online during this one, for a change.