Search Results: "nate"

14 October 2007

Martin F. Krafft: Converting a package to Git

Previously, I demonstrated a Debian packaging workflow using Git and I mentioned the possibility of a follow-up post; well, here it is: you want to use my workflow (or one that's related) for a package that is currently maintained with Subversion on svn.debian.org and you'd like to keep the history during the conversion. Make sure to read the previous post before this one. I am again using the example of mdadm since its Git packaging repository is in a state of shambles and I want to restart to get it right and import the history from the previous Subversion repository. What better way than to write a blog post as I do so? Well, plenty actually. This kind of post isn't really made for a blog, and I have started work on setting up ikiwiki on madduck.net, but it's not yet ready, so I'll stick with the blog for now. I will make sure that links don't break as I move content over, so feel free to bookmark this
Importing the package into Git Thanks to git-svn, the initial step of getting your package imported into Git is a breeze: 
$ git-svn clone --stdlayout --no-metadata \
    svn+ssh://svn.debian.org/svn/pkg-mdadm/mdadm mdadm
Sit back and enjoy. If that command exits prematurely with an error such as the following:
Malformed network data: Malformed network data at /usr/local/bin/git-svn line 1029
then you should upgrade to a newer Git version, or have a look here. If your Git does not know --stdlayout then upgrade as well (or use -T trunk -t tags -b branches instead). Sam Vilain notes that it is important to "get the attribution right with the final SVN import - getting the authors map right. I didn't do that. If you look at the repository resulting from the above command, you'll notice strange commit authors, such as madduck@some-unique-uuid-from-svn . git-svn allows you to map these to real names with real email addresses, which ensures that the attributions are good for the whole world to see. When done, switch to the repository and run git-branch -r. As you'll see, git-svn imported all SVN branches and tags as remote branches. You need those if you want to bidirectionally track the Subversion repository, but we are converting, as you may have guessed by the --no-metadata switch above. Therefore, we resort to the Dinosaur method of converting branches to tags, which I'll simplify for mdadm. We also just delete all remote branches after tagging, since mdadm never used branches in the SVN repository. Your mileage may vary. 
git branch -r   sed -rne 's, *tags/([^@]+)$,\1,p'   while read tag; do
  echo "git tag debian/$tag tags/$ tag ^; git branch -r -d tags/$tag"
done
git branch -r   while read tag; do
  echo "git branch -r -d $tag"
done
If that seems to work alright, then you can execute the commands. Sam Vilain (again) hints me at git-pack-refs and then to edit .git/packed-refs with an editor. This certainly leaves more room for errors but might be significantly faster.
Cleaning up the SVN references Even though we passed --no-metadata to git-svn, it did leave some traces in .git/, which we can now safely remove: 
$ git config --remove-section svn-remote.svn
$ rm -r .git/svn
Setting things straight You can skip this section unless you want to know a bit about how to fix up stuff with Git. There was actually some nasty tagging errors leading up to the 2.5.6-9 release for etch and I could never be bothered to fix those in SVN, but now I can (I love Git!): 
$ git tag -d debian/2.5.6-10            # never existed
$ git tag -f debian/2.5.6-8 2.5.6-8~2   # mistagged
$ git checkout -b maint/etch 2.5.6-8    # this is when we diverged
$ git apply < /tmp/mdadm-2.5.6-8..2.5.6-9.diff
$ git add debian/po/gl.po debian/po/pt.po debian/changelog
$ git commit -s
$ git tag debian/2.5.6-9
Now that that's fixed, there is one other thing to worry about, namely the very last commit to SVN, which obsoletes the repository and points to the Git repository. But that's not all of it. I was also silly enough to include a fix in the same commit. Let's see what Git can do. Since the process of obsoletion involves all but adding a file, we can simply --amend the last commit and provide a new log message: 
$ git checkout master
$ git rm OBSOLETE debian/OBSOLETE
$ git commit --amend
Now the repository is in an acceptable state.
Making ends meet The pkg-mdadm effort on svn.debian.org only maintained the ./debian/ directory, separate from the upstream code, and boy was that a bad idea. Just to give one example: think about what's involved in preparing a Debian-specific patch against the upstream code this has to end, and we can make it end right here; let's import upstream's code (again not using his ADSL line, but the upstream branch of the pkg-mdadm Git repository; see the previous post for details): 
$ git remote add upstream-repo git://git.debian.org/git/pkg-mdadm/mdadm
$ git config remote.upstream-repo.fetch \
    +refs/heads/upstream:refs/remotes/upstream-repo/upstream
$ git fetch upstream-repo
$ git checkout -b upstream upstream-repo/master
Now we have two unconnected ancestries in our repository, and it's time to join them together. The most logical way seems to be to use the last upstream tag for which we have a Debian tag: 2.6.2. For this, we branch off the corresponding Debian tag (2.6.2-1) and merge upstream's 2.6.2 tag into the new branch. This will be a temporary branch Then, we rebase (remember, nothing has been published yet) the master branch on top of this temporary branch, before we end that branch's short life. The Debian tag stays where it is since it describes the state of the repository at time of the release of 2.6.2-1. 
$ git checkout -b tmp/join debian/2.6.2-1
$ git merge mdadm-2.6.2
$ git rebase tmp/join master
$ git branch -d tmp/join
It just so happens that the head of the SVN repository, which is identical to the tip of our master branch, corresponds to Debian release 2.6.2-2, so we tag it: 
$ git tag debian/2.6.2-2
We are now also "born" in the sense that maintenance in Git has started. Let's mark that point in history. There is no real reason I can foresee for this yet, but nonetheless: 
$ git tag -s git-birth
Turning dpatch files into feature branches We want to turn dpatch files into feature branches and we somehow make it "proper". We could branch, apply the patch, delete the patch file, checkout master and delete the patch file there as well, but that appears "improper" to me at least; so instead, we'll cherry-pick: 
$ git checkout -b deb/conffile-location
$ debian/patches/01-mdadm.conf-location.dpatch -apply
$ git rm debian/patches/01-mdadm.conf-location.dpatch
$ git commit -s
$ git commit -s $(git ls-files --others --modified)
I should quickly intervene to make sure you are following. I am making use of Git's index here. Applying the patch makes the changes in the working tree, but we did not tell Git that we want those to be part of the commit just yet. Instead, we delete the dpatch with git-rm, which automatically registers the deletion with the index. Thus, the first git-commit creates a commit which deletes the dpatch, while the second git-commit creates a commit with all the changes from the dpatch, using git-ls-files to identify new and modified files. But for now, let's move on. We have two commits in the deb/conffile-location branch, and one of those is relevant to the master branch, we cherry-pick it: 
$ git cherry-pick deb/conffile-location^
If you're confused, let me explain: our goal is to have a number of feature branches, of which master is the one in which most of ./debian/ is maintained. All the branches later come together in the long-living build branch, so deb/conffile-location will never be merged back into master. However, once we applied the dpatch to the feature branch, we can delete it from there and the master branch. By cherry-picking, we "import" the deletion to the master branch. I repeat the same procedure for deb/docs, merging all the documentation-related dpatches, but I'll spare you the details.
and then Git let me down In the next step, I found I had misunderstood Git merging: I thought Git was smart, but Linus had his reasons for calling Git the "stupid content tracker" (more on that later). Read on as I am obsoleting dpatch files that upstream had merged: 99-*-FIX.dpatch. For consistency, I wanted to cherry-pick each of the appropriate upstream commits into the master branch along with deleting the corresponding dpatch file. Here is one example: 99-monitor-6+10-FIX.dpatch was obsoleted by upstream's commit 66f8bbb; the -x records the original commit ID in the log: 
$ git cherry-pick -x 66f8bbb
$ git rm debian/patches/99-monitor-6+10-FIX.dpatch
$ git commit -s -m"remove dpatch obsoleted by $(git rev-parse --short HEAD)"
I repeated the procedure for the other dpatch files, removed the dpatch infrastructure, and then went on to merge it all into build to build the package. The build branch is a long-living branch off upstream, but which upstream? I'll fast-forward you past a segfault problem with mdadm, which upstream (thought to have) resolved with commit 23dc1ae after 2.6.3, but he had not yet released 2.6.4. Looking at the commits between 23dc1ae and upstream's HEAD at the time, I decided to include them all and snapshot 4450e59: 
$ git fetch upstream-repo
$ git checkout upstream
$ git merge upstream-repo
$ git tag mdadm-2.6.3+200709292116+4450e59 4450e59
$ git checkout master
$ git merge --no-commit mdadm-2.6.3+200709292116+4450e59
$ dch -v mdadm-2.6.3+200709292116+4450e59-1
$ git add debian/changelog
$ git commit -s
And then I called poor-mans-gitbuild, which merges master and then deb/* into build. Here is when stuff blew up. I'll make a long story short (read my description of the problem and Linus' answer if you want to know more): I thought Git was smart to identify merges common to both branches and do the right thing, but it turn out that Git does not care at all about commits, it only worries about content and the end result. In our case, unfortunately (or fortunately), the outcome meant a conflict because the upstream branch introduced a simple change (last hunk) in the lines surrounding the patch we cherry-picked, and Git can't handle it. The solution is not to cherry-pick, to cherry-pick all commits touching the context of the dpatch, or to simply merge upstream into all out feature branches. In our case, the first is the easiest solution and since importing dpatch files is a one-time thing (thank $DEITY), I'll leave it at that. Almost. I have spent two days thinking about this more than I should have. And it was this point Linus made which made me appreciate Git even more:
Conflicts aren't bad - they're good. Trying to aggressively resolve them automatically when two branches have done slightly different things in the same area is stupid and just results in more problems. Instead, git tries to do what I don't think anybody else has done: make the conflicts easy to resolve, by allowing you to work with them in your normal working tree, and still giving you a lot of tools to help you see what's going on.
The end This concludes today's report. Importing the changes from the old Git repo, tagging and merging the branches is all covered in my previous post, or at least you'll find enough information there to complete the exercise. I would like to specifically thank Sam Vilain and Linus Torvalds for their help in preparing this post, as well as the #git/freenode inhabitants, as always. If you are interested in the topic of using version control for distro packaging, I invite you to join the vcs-pkg mailing list and/or the #vcs-pkg/irc.oftc.net IRC channel. Also, if you are interested in Git in general, you can find a list of blog posts on the Git wiki. NP: The Police: Zenyatta Mondatta

Mike Hommey: Adding some VCS information in bash prompt

I don’t spend a lot of time customizing my “working” environment nowadays, like enhancing vim configuration, or tweaking the shell. But when I read MadCoder’s zsh git-enabled prompt, I though it was too convenient to not have something like that. Except I don’t work with git only (sadly, but that’s changing), and I don’t like colours in prompt (and a 2 lines prompt is too much). Anyways, since I have a bunch of directories in my $HOME that contain either svk, svn, mercurial, or git working trees, I thought it would be nice to have some information about all this on my prompt. After a few iterations, here are the sample results:
mh@namakemono:~/dd/packages$
mh@namakemono:(svn)~/dd/packages/iceape[trunk:39972]debian/patches$
mh@namakemono:(svk)~/dd/packages/libxml2[trunk:1308]include/libxml$
mh@namakemono:(hg)~/moz/cvs-trunk-mirror[default]uriloader/exthandler$
mh@namakemono:(git)~/git/webkit[debian]JavaScriptCore/wtf$
The script follows, with a bit of explanation intertwined.
_bold=$(tput bold)
_normal=$(tput sgr0)
tput is a tool I only dicovered recently, and avoids the need to know the escape codes. There are also options for cursor placement, colours, etc. It lies in the ncurses-bin package, if you want to play with it.
__vcs_dir()
  local vcs base_dir sub_dir ref
  sub_dir()
    local sub_dir
    sub_dir=$(readlink -f "$ PWD ")
    sub_dir=$ sub_dir#$1
    echo $ sub_dir#/
  
We declare as much as possible as local (even functions), so that we avoid cluttering the whole environment. sub_dir is going to be used in several places below, which is why we declare it as a function. It outputs the current directory, relative to the directory given as argument.
  git_dir()
    base_dir=$(git-rev-parse --show-cdup 2>/dev/null) return 1
    base_dir=$(readlink -f "$base_dir/..")
    sub_dir=$(git-rev-parse --show-prefix)
    sub_dir=$ sub_dir%/
    ref=$(git-symbolic-ref -q HEAD git-name-rev --name-only HEAD 2>/dev/null)
    ref=$ ref#refs/heads/
    vcs="git"
  
This is the first function to detect a working tree, for git this time. Each of these functions set the 4 variables we declared earlier: vcs, base_dir, sub_dir and ref. They are, respectively, the VCS type, the top-level directory of the working tree, the current directory, relative to base_dir, and the branch, revision or a reference in the repository, depending on the VCS in use. These functions return 1 if the current directory is not in a working tree of the currently considered VCS.
The base directory of a git working tree can be deduced from the result of git-rev-parse --show-cdup, which gives the way up to the top-level directory, relative to the current directory. readlink -f then gives the canonical top-level directory. The current directory, relative to the top-level, is simply given by git-rev-parse --show-prefix.
git-name-rev --name-only HEAD gives a nice reference for the current HEAD, especially if you’re on a detached head. But this can turn out to do a lot of work, introducing a slight lag when you cd for the first time in the git working tree, while most of the time, the HEAD is just a symbolic ref. This is why we first try git-symbolic-ref --name-only HEAD.
  svn_dir()
    [ -d ".svn" ] return 1
    base_dir="."
    while [ -d "$base_dir/../.svn" ]; do base_dir="$base_dir/.."; done
    base_dir=$(readlink -f "$base_dir")
    sub_dir=$(sub_dir "$ base_dir ")
    ref=$(svn info "$base_dir" awk '/^URL/ sub(".*/","",$0); r=$0 /^Revision/ sub("[^0-9]*","",$0); print r":"$0 ')
    vcs="svn"
  
Detecting an svn working tree is easier : it contains a .svn directory, be it top-level or sub directory. We look up the top-level directory by checking the last directory containing a .svn sub directory on the way up. This obviously doesn’t work if you checkout under another svn working tree, but I don’t do such things.
For the ref, I wanted something like the name of the directory that has been checked out at the top-level directory (usually “trunk” or a branch name), followed by the revision number.
  svk_dir()
    [ -f ~/.svk/config ] return 1
    base_dir=$(awk '/: *$/ sub(/^ */,"",$0); sub(/: *$/,"",$0); if (match("'$ PWD '", $0"(/ $)")) print $0; d=1; /depotpath/ && d == 1 sub(".*/","",$0); r=$0 /revision/ && d == 1 print r ":" $2; exit 1 ' ~/.svk/config) && return 1
    ref=$ base_dir##*

    base_dir=$ base_dir%%
*
    sub_dir=$(sub_dir "$ base_dir ")
    vcs="svk"
  
svk doesn’t have repository files in the working tree, so we would have to ask svk itself if the current directory is a working tree. Unfortunately, svk is quite slow at that (not that it takes several seconds, but that induces a noticeable delay to display the prompt), so we have to parse its config file by ourselves. We avoid running awk twice by outputing both the informations we are looking for, separated by a carriage return, and then do some tricks with bash variable expansion.
  hg_dir()
    base_dir="."
    while [ ! -d "$base_dir/.hg" ]; do base_dir="$base_dir/.."; [ $(readlink -f "$ base_dir ") = "/" ] && return 1; done
    base_dir=$(readlink -f "$base_dir")
    sub_dir=$(sub_dir "$ base_dir ")
    ref=$(< "$ base_dir /.hg/branch")
    vcs="hg"
  
I don’t use mercurial much, but I happen to have exactly one working tree (a clone of http://hg.mozilla.org/cvs-trunk-mirror/), so I got some basic information. There is no way we can ask mercurial itself for information, it is too slow for that (main culprit being the python interpreter startup), so we take the informations we can (and since I don’t know much about mercurial, that’s really basic). Note that if you’re deep in the VFS tree, but not in a mercurial working tree, the while loop may be slow. I didn’t bother much looking for a better solution.
  git_dir
  svn_dir
  svk_dir
  hg_dir
  base_dir="$PWD"
Here we just run all these functions one by one, stopping at the first that matches. Adding some more for other VCS would be easy.
  echo "$ vcs:+($vcs) $ _bold $ base_dir/$HOME/~ $ _normal $ vcs:+[$ref]$ _bold $ sub_dir $ _normal "

PS1='$ debian_chroot:+($debian_chroot) \u@\h:$(___vcs_dir)\$ '
Finally, we set up the prompt, so that it looks nice with all the gathered information. Update: made the last lines of the script a little better and factorized.

13 October 2007

Christian Perrier: RWC: One of the worst games of this World Cup

So, A completely non-existing French team lost against a....nearly completely non-existing English team. Too many bad decisions, too much bad kick play, too many lost balls....and even no luck (the English try...damn). The French team couldn't win with such poor play (even when they were leading by 9-8, we both were completely sure they would lose on a penalty kick). What's pretty sad is that England nearly showed nothing too and, therefore, both teams made this the worst match I've seen in that world cup. Hopefully, Argentina and South Africa will show some better play tomorrow. However, when I see this, I nearly regret that France eliminated New Zealand. The All Blacks could never have played as bad as France did and they would certainly have kicked out that boring English team. Definitely, I'll be South African or Argentinian next Sunday (I have a small preference for being Argentinian, indeed).

Julien Valroff: ndiswrapper and WPA

I had been using ndiswrapper with a Netgear WG511 PCMCIA card[1] without any problems for a while, then have begun to get kernel crashes, which were more and more frequent. I have thus backported to Etch the 1.47 version available in Sid, which *does* fix my kernel crashes. However, this newer version has brought new problems: with a “high” network load, the module just crashes silently, and has to be reloaded so that the network connection can be enabled again. I have packaged ndiswrapper 1.48 as the release notes let me think my issues were fixed (”Disassocation with wpa_supplicant is fixed”), but the problem is still there with this latest release! I have finally managed to get things work correctly thanks to this post: I use Debian linux-image, ie. without preemption enabled, and I can now confirm the still-to-be-released 1.49 version fixes this issue. I have uploaded 1.49-rc3 packages to my unofficial repository just in case someone has the same issue.
Linknotes:
  1. the free prism54 unfortunately does not support WPA ↩

Biella Coleman: Decoding Liberation: The Promise of FOSS (and Web 2.0)

Last week I helped Samir Chopra and Scott Dexter kick off their book release party in New York City. The book Decoding Liberation: The Promise of Free and Open Source Software is the first academic book length piece on free software proper that among other things examines the repercussions of such elements as language use (free software vs open source) and licensing (such as non-copyleft licensing). Here are my opening remarks, which don’t give justice to the book but give a small taste of what is in there. Samir and Scott are computer scientists, philosophers, and political thinkers and bring these positions and perspectives to bear in their work. While I tend to avoid the discussion on the differences and divergences between free and open source software and licensing (just because I ask a set of questions that tend not to go into that territory), they spend a hefty about of time on this sort of engagement And what is so useful about their approach is that it is technically detailed and carefully analyzed, clarifying the stakes involved in choosing a certain set of licensing over others, or the political implications of language use. Along with this focus, there are many other threads they unpack and one of my favorites is on the aesthetics of code, which I discuss with some detail in my opening remarks with the help of one of my favorite literary writers, Susan Sontag. The conversation that followed was lively, in part because there were a number of people in the audience who are also very familiar with FOSS (Somewhat unbelievably, there were 3 anthropologists there who study free software, myself, Jelena Karanovic, and Anita Chan). And I think one of the most interesting questions was launched by Anita who asked the authors what they meant more precisely by the promise of FOSS as well as liberation. The conversation that followed was too rich to recount here, but something that I raised and I do think is important is the relationship between the buzz word of the last few years, Web 2.0 and FOSS. Web 2.0 is related to FOSS in so far as Web 2.0 refers to a suite of technologies that allow for the creation of user-generated content and collaboration. FOSS refers to a development methodology that is based on promiscuous sharing of code and collaboration. The similarities, however, end there because much of the Web 2.0 infrastructure is proprietary. FOSS by definition is non-proprietary. But I think that soon we are going to see more Web 2.0-like companies open up their infrastructure entirely or at least important components. One example of a new technology that is Web 2.0-like and is entirely free software is a activist networking tool crabgrass that is pretty impressive (I have used it to coordinate my move and am using it now to coordinate a collaborative grant). It is still under development but once released, it will be a great boon to any group that needs to collaborate and organize and coordinate:
Crabgrass also provides a public advocacy centric view of content so that people can learn more about issues and organizations through social relationships. Blog tools, voter guides, petitions, event organizing tools, and action alerts are being added to the functionality of the platform. Crabgrass integrates wikis, asset repositories, task lists, calendars, polls, and meeting schedulers into one tool which allows groups to manage their internal organizing.
The other technology that I am excited about and that I have already written about is Kaltura. As I mentioned, Kaltura is important because it lowers the bar for collaboration, providing tools to facilitate video editing. But what I find as interesting and as significant is that they are perhaps the first large-scale Web 2.0 company that is actively seeking to enter the territory of FOSS and in this respect, once they do so, they will lead the field, not simply for technical reasons, but because they choose to make and engage with open source technologies. Bringing Web 2.0 within the orbit of FOSS and brining FOSS within the orbit of Web 2.0 can only work to bolster each other, and this is where I think, at least part of the promise of FOSS lies.

12 October 2007

Benjamin Mako Hill: Ubuntu (w/ Special Guests) in Boston

There are quite a few important events related to Ubuntu -- and to free software communities more generally -- in the Boston area in the next few weeks. I plan to participate in many of them. First, this coming Saturday, October 13, there will be an Ubuntu install party hosted at MIT and organized by the Ubuntu Massachusetts local community team. It promises to be a lot of fun and a great opportunity to have a gaggle of geeks install a free OS on your computer for you. If you've been thinking about installing free software but been hesitant (my guess is that this is not the majority of my readers), this is the event for you. I'll probably be doing RockBox installs as well so backup your music and bring an iPod if you're unfortunate enough to have funded Apple through the purchase of one. Next week on October 18, Ubuntu Massachusetts will be hosting a party at the Globe Bar and Cafe to celebrate the (scheduled) release of the Gutsy Gibbon. I am not thrilled about everything in this release -- like Compiz by default -- but I am happy about the progress of the distribution both technically and in reaching out to an ever-wider and ever-larger group of users. On the week of October 29-November 2, Canonical is hosting the Ubuntu Developer Summit in Cambridge. I'll definitely drop by for a least a day or two to make some strategic interjections and to participate in a few specifications that I care about. The summit is just down the street from my office at the Hotel at MIT so I have little excuse to not show up. I'll also being hanging out with friends from Ubuntu during the week. Finally, as part of the Ubuntu conference, Canonical is sponsoring FOSSCamp. It promises to be a Foocamp/Barcamp style "un-conference" with a focus on free software and open source. I'll be there and, if there's demand, will run sessions on Selectricity and a quick Making Debian/Ubuntu Packages for Sysadmins talk -- basically a more polished version of what I did at the Ubucon in New York. All events are open to the public although people who are not Ubuntu developers may be a little bored at the developer summit. I look forward to seeing both old and new faces around the project in the next month.

Rapha&#235;l Hertzog: The DSA dilemma

For once, Clint blogged on something that I can understand. :-) I don’t buy everything he says, but in the case of DSA, the part where he says “you cannot have a functional and respectable subgroup if it maintains autonomy like that” is a real problem. The leadership problem I mentioned is real. And it can theoretically be solved by undelegating one of the problematic side of this DSA-internal dispute. But which one? Given the unwillingness of Joey to discuss the problems, he makes an easy target… which would leave DSA up to Ryan, James and Phil. But is that a desirable thing? If DSA is perceived as being an “autonomous” group which is not involved in Debian’s main discussions and which is somewhat disconnected from Debian’s day-to-day life, it’s largely due to the behavior of James and Ryan. E-mail communication with them is very difficult as they’ll respond only if they really care about something. And despite the setup of the request tracker, they have barely been able to make proper usage of it… the idea was to use RT tickets to track everything that DSA does but they don’t use it as such. For example, James setup a “wikiadm” group and he never reported anything to the related ticket (#194) (I did it myself once I found out). Also there’s an internal ticket about the replacement of ftp.debian.org (that I created because ftp.d.o ran out of space regularly) and AFAIK Jeroen has been in touch with James to setup that replacement, but nothing got reported to the tracker. Ryan promised me once to put his DSA TODO list in the tracker so that other people can jump in and help out. He never did. So while Joey is definitely a pain for DSA, at least he’s a visible participant of the team and he interacts with the community. James and Ryan are not, they interact only through private channels and do not share their opinions or their vision of Debian.
I believe this is a real problem. On the other hand, most of the interesting changes in the last months are the results of James’s work. But he’s also implicitly blocking addition of new members as long as the leadership problem is not solved. I tried to fill the communication void of the DSA team by various means. I follow everything as closely as I can so that I can report changes on other channels, mailing lists when needed. I made efforts to document stuff on the wiki page, etc. But this is not a long term solution, the communication issue must be fixed within the team. The path ouf of this mess is still not very clear, but something is going to change soon. Not quite sure what though. What would you suggest? And if you were DPL, what would you do? Since private discussions and negotiations lead nowhere, it’s tempting to bring the issue in the public area. In theory, they have no way to escape discussions and they’ll have to communicate their grudges against the other side if they want to have some fair judgment between both parties. Unfortunately, given the habits of James and Ryan, they probably won’t participate in any public discussion and either resign or stay where they are waiting for any decision… Comments welcome. Partagez cet article / Share This

11 October 2007

Andrew Pollock: [life] A c-section from a bloke's perspective

I've been meaning to write this for weeks. Hopefully I can still remember everything. So Sarah had a c-section the day before we were scheduled to have our c-section class. We'd both read a bit though, so it wasn't like we were going in totally uninformed and unprepared.. Firstly, Stanford Hospital's operating room area is rather mind-blowing. Sarah was transferred from Lucille Packard Children's Hospital to Stanford because they could get an operating room there sooner. The transfer was trivial, just roll the whole bed down the corridor, through the magic double doors into the operating rooms. The doors in question have a whole heap of video cameras pointing straight down, so they see everyone that gets wheeled in and out, which I found interesting. Good for records of patient movements I guess. Anyway, through those doors is a maze of corridors. There was a light-up board hanging from the ceiling in the middle of the main corridor with lights for all of the ORs. I think there were about 20. The illuminated ones were available. We went down a corridor, and past a few ORs. Outside of each one is a big stainless-steel scrub area. We got to "our" OR, and they transferred Sarah onto the operating table. Because of her heart condition, they wanted to have an arterial line in her for the duration of the procedure. This is some sort of gadget that goes into an artery and monitors your blood pressure beat by beat. The "interesting" thing with the anaesthesiologist was the New Zealand woman that we'd seen when we had an anaesthesiologist months prior, but the appointment was with a different anaesthesiologist, and this New Zealand woman was "observing". At the time, the anaesthesiologist we had the appointment with told us that she was going to be on vacation at the time of the delivery, so she wouldn't be attending. So I don't know if the Kiwi woman got promoted out of "observing" to "doing" in that period, or what, but she reintroduced herself to us in LPCH prior to moving to Stanford as being our anaesthesiologist for the delivery. Anyway, in the OR, they handed her the arterial line, in it's plastic packaging, and the first thing she asked was whether it came with anything else to put it in. At this point I figured she hadn't put one of these in before. (The answer to her question, by the way, was "no") She then proceeded to have a terrible time getting the line into Sarah's arm. Normally intravenous lines go into a vein, but this has to go into an artery. Apparently when you go poking an artery, it spasms. It then becomes very hard to get it into a moving target. She had a few goes on one arm, then gave up, and had to move over to the other arm. Sarah doesn't like needles at the best of times, and getting this honking great big thing in was no picnic. The bruising on both her wrists afterwards was phenomenal. She tends to bruise fairly spectacularly whenever they stick a needle in her. I almost stepped in and asked if it was really necessary that she have it, because they were having so much trouble getting it in, and it was clearly hurting Sarah. I think someone else ended up getting it in, not the anaesthesiologist. Not "that" one anyway. There were heaps of people in the OR. Next, came the epidural. Again, we had the inexperienced anaesthesiologist. This time she said to someone else how she was so nervous because there were "too many eyes". Not really reassuring. They gave Sarah a numbing injection first, and then proceeded to poke her in back with the epidural catheter needle thing. Sarah was flinching when they did it, and they kept yelling at her not to move. Apparently it didn't hurt, but it caused discomfort every time they tried to get it in. Eventually the associate professor of anaesthesiology stepped in and got it in. Later, they said that Sarah had a slightly curved spine or something, and that had made it a bit more complicated. I'm not begrudging the obviously inexperienced anaesthesiologist for anything. Stanford is a teaching hospital, and doctors have to get their experience somehow, but I think that when the patient is conscious, it's important to come across a little more confident. So once they got all of that sorted out, they could get her lying down on the table and get down to business. Apparently one of the side-effects of the epidural is that it can cause your body to think it's got hypothermia. Part way through the procedure, Sarah started shaking uncontrollably. I can't remember whether she complained about feeling cold or not. She also got a really itchy face. Apparently that's a side-effect of morphine. Sarah also doesn't tolerate anaesthetics very well, and started feeling nauseous part way through the procedure. Sure enough, as has happened countless times previously post-operatively, she threw up. At least we had warning and I got a kidney dish in place in time. That was about all the eventful surgical stuff. They had the usual screen up from about her neck area, so I couldn't see what they were doing. After Joshua was delivered, one of the nurses asked me if I wanted to see him, and took me around from her left shoulder where I'd been sitting to around below her right foot, to where they had him in one of those perspex portable crib things. I could see a lot of red out of the corner of my eye, and I could have seen a lot more if I'd bothered to look, but I was more interested in seeing my son at that stage. They let me bring him back around to where I'd been sitting previously, to show Sarah, but it was hard getting him up to a point where she could see him properly, and she was partially out of it by this stage. I think she dozed off for a little bit not long after that. I forget what time we got into the OR, but they started the operation in earnest at about 10:00am, and we were out of there by 10:45am if memory serves. The nurse in charge of the recovery room really didn't want me there, so I went back to LPCH with one of the nurses to spend some time with him while he was weighed and measured, and I took some photos, and held him. This was good, because I got to have a good cry on my own with him. So overall, in my "I'll never be squeezing a watermelon out of my nether regions" blokey perspective, having seen what Sarah went through, and how much discomfort that it looked like she was going through, both in the OR and afterwards, I'm not convinced c-sections are any less painful than a vaginal birth (taking the average case of both). I think you're trading some "short term" (depends on how long your labour is I guess) pain for some "long term" pain while you recover from your incision. Granted, Sarah wasn't really laid up all that badly for all that long, but it still didn't look all that pleasant. I guess really only the women who have done both delivery methods can really comment. All I know is I found it very difficult watching Sarah go through so much pain. So much so that I'm not really looking forward to that again. Maybe, hopefully, after her aorta is repaired, they'll let her delivery vaginally next time. I'm not holding much hope though. The high-risk obstetricians are fairly paranoid. There's also the possibility the baby will make the decision very easy for everyone by being breech. Sarah raised an interesting point today. Why didn't they just knock her out with a general anaesthetic? There was no need for her to be conscious for the delivery. That would have spared her the pain and suffering of getting the arterial line in, and they wouldn't have needed to bother with an epidural. One for her to ask the obstetrician at her follow-up appointment.

10 October 2007

Martin F. Krafft: Packaging with Git

Introduction I gave a joint presentation with Manoj at Debconf7 about using distributed version control for Debian packaging, and I volunteered to do an on-line workshop about using Git for the task, so it's about time that I should know how to use Git for Debian packaging, but it turns out that I don't. Or well, didn't. After I made a pretty good mess out of the mdadm packaging repository (which is not a big problem as it's just ugly history up to the point when I start to get it right), I decided to get down with the topic and figure it out once and for all. I am writing this post as I put the pieces together. It's been cooking for a week, simply so I could gather enough feedback. I am aware that Git is not exactly a showcase of usability, so I took some extra care to not add to the confusion. It may be the first post in a series, because this time, I am just covering the case of mdadm, for which upstream also uses Git and where I am the only maintainer, and I shall pretend that I am importing mdadm to version control for the first time, so there won't be any history juggling. Future posts could well include tracking Subversion repositories with git-svn, and importing packages previously tracked therewith, but this ain't no promise! (well, that last post is already being drafted, but far from finished; you have been warned!) I realise that git-buildpackage exists, but imposes a rather strict branch layout and tagging scheme, which I don't want to adhere to. And gitpkg (Romain blogged about it recently), deserves another look since, according to its author, it does not impose anything on its user. But in any case, before using such tools (and possibly extending them to allow for other layouts), I'd really rather have done it by hand a couple of times to get the hang of it and find out where the culprits lie. Now, enough of the talking, just one last thing: I expect this blog post to change quite a bit as I get feedback. Changes shall be highlighted in bold typeface.
Setting up the infrastructure First, we prepare a shared repository on git.debian.org for later use (using collab-maint for illustration purposes), download the Debian source package we want to import (version 2.6.3+200709292116+4450e59-3 at time of writing, but I pretend it's -2 because we shall create -3 further down ), set up a local repository, and link it to the remote repository. Note that there are other ways to set up the infrastructure, but this happens to be the one I prefer, even though it's slightly more complicated: 
$ ssh alioth
$ cd /git/collab-maint
$ ./setup-repository pkg-mdadm mdadm Debian packaging
$ exit
$ apt-get source --download-only mdadm
$ mkdir mdadm && cd mdadm
$ git init
$ git remote add origin ssh://git.debian.org/git/collab-maint/pkg-mdadm
$ git config branch.master.merge refs/heads/master
Now we can use git-pull and git-push, except the remote repository is empty and we can't pull from there yet. We'll save that for later. Instead, we tell the repository about upstream's Git repository. I am giving you the git.debian.org URL though, simply because I don't want upstream repository (which lives on an ADSL line) hammered in response to this blog post: 
$ git remote add upstream-repo git://git.debian.org/git/pkg-mdadm/mdadm
Since we're using the upstream branch of the pkg-mdadm repository as source (and don't want all the other mess I created in that repository), we'll first limit the set of branches to be fetched (I could have used the -t option in the above git-remote command, but I prefer to make it explicit that we're doing things slightly differently to protect upstream's ADSL line). 
$ git config remote.upstream-repo.fetch \
    +refs/heads/upstream:refs/remotes/upstream-repo/upstream
And now we can pull down upstream's history and create a local branch off it. The "no common commits" warning can be safely ignored since we don't have any commits at all at that point (so there can't be any in common between the local and remote repository), but we know what we're doing, even to the point that we can forcefully give birth to a branch, which is because we do not have a HEAD commit yet (our repository is still empty): 
$ git fetch upstream-repo
warning: no common commits
[ ]
  # in the real world, we'd be branching off upstream-repo/master
$ git checkout -b upstream upstream-repo/upstream
warning: You appear to be on a branch yet to be born.
warning: Forcing checkout of upstream-repo/upstream.
Branch upstream set up to track remote branch
  refs/remotes/upstream-repo/upstream.
$ git branch
* upstream
$ ls   wc -l
77
Importing the Debian package Now it's time to import Debian's diff.gz remember how I pretend to use version control for package maintenance for the first time. Oh, and sorry about the messy file names, but I decided it's best to stick with real data in case you are playing along: Since we're applying the diff against version 2.6.3+200709292116+4450e59, we ought to make sure to have the repository at the same state. Upstream never "released" that version, but I encoded the commit ID of the tip when I snapshotted it: 4450e59, so we branch off there. Since we are actually tracking the git.debian.org pkg-mdadm repository instead of upstream, you can use the tag I made. Otherwise you could consider tagging yourself: 
$ #git tag -s mdadm-2.6.3+200709292116+4450e59 4450e59
$ git checkout -b master mdadm-2.6.3+200709292116+4450e59
$ zcat ../mdadm_2.6.3+200709292116+4450e59-2.diff.gz   git apply
The local tree is now "debianised", but Git does not know about the new and changed files, which you can verify with git-status. We will split the changes made by Debian's diff.gz across several branches.
The idea of feature branches We could just create a debian branch, commit all changes made by the diff.gz there, and be done with it. However, we might want to keep certain aspects of Debianisation separate, and the way to do that is with feature branches (also known as "topic" branches). For the sake of this demonstration, let's create the following four branches in addition to the master branch, which holds the standard Debian files, such as debian/changelog, debian/control, and debian/rules:
  • upstream-patches will includes patches against the upstream code, which I submit for upstream inclusion.
  • deb/conffile-location makes /etc/mdadm/mdadm.conf the default over /etc/mdadm.conf and is Debian-specific (thus the deb/ prefix).
  • deb/initramfs includes the initramfs hook and script, which I want to treat separately but not submit upstream.
  • deb/docs similarly includes Debian-only documentation I add to the package as a service to Debian users.
If you're importing a Debian package using dpatch, you might want to convert every dpatch into a single branch, or at least collect logical units into separate branches. Up to you. For now, our simple example suffices. Keep in mind that it's easy to merge two branch and less trivial to split one into two. Why? Well, good question. As you will see further down, the separation between master and deb/initramfs actually makes things more complicated when you are working on an issue spanning across both. However, feature branches also bring a whole lot of flexibility. For instance, with the above separation, I could easily create mdadm packages without initramfs integration (see #434934), a disk-space-conscious distribution like grml might prefer to leave out the extra documentation, and maybe another derivative doesn't like the fact that the configuration file is in a different place from upstream. With feature branches, all these issues could be easily addressed by leaving out unwanted branches from the merge into the integration/build branch (see further down). Whether you use feature branches, and how many, or whether you'd like to only separate upstream and Debian stuff is entirely up to you. For the purpose of demonstration, I'll go the more complicated way.
Setting up feature branches So let's commit the individual files to the branches. The output of the git-checkout command shows modified files that have not been committed yet (which I trim after the first example); Git keeps these across checkouts/branch changes. Note that the ./debian/ directory does not show up as Git does not know about it yet (git-status will tell you that it's untracked, or rather: contains untracked files since Git does not track directories at all): 
$ git checkout -b upstream-patches mdadm-2.6.3+200709292116+4450e59
M Makefile
M ReadMe.c
M mdadm.8
M mdadm.conf.5
M mdassemble.8
M super1.c
$ git add super1.c     #444682
$ git commit -s
  # i now branch off master, but that's the same as 4450e59 actually
  # i just do it so i can make this point 
$ git checkout -b deb/conffile-location master
$ git add Makefile ReadMe.c mdadm.8 mdadm.conf.5 mdassemble.8
$ git commit -s
$ git checkout -b deb/initramfs master
$ git add debian/initramfs/*
$ git commit -s
$ git checkout -b deb/docs master
$ git add RAID5_versus_RAID10.txt md.txt rootraiddoc.97.html
$ git commit -s
  # and finally, the ./debian/ directory:
$ git checkout master
$ chmod +x debian/rules
$ git add debian
$ git commit -s
$ git branch
  deb/conffile-location
  deb/docs
* master
  upstream
  upstream-patches
At this time, we push our work so it won't get lost if, at this moment, aliens land on the house, or any other completely plausible event of apocalypse descends upon you. We'll push our work to git.debian.org (the origin, which is the default destination and thus needs not be specified) by using git-push --all, which conveniently pushes all local branches, thus including the upstream code; you may not want to push the upstream code, but I prefer it since it makes it easier to work with the repository, and since most of the objects are needed for the other branches anyway after all, we branched off the upstream branch. Specifying --tags instead of --all pushes tags instead of heads (branches); you couldn't have guessed that! See this thread if you (rightfully) think that one should be able to do this in a single command (which is not git push refs/heads/* refs/tags/*) 
$ git push --all
$ git push --tags
Done. Well, almost
Building the package (theory) Let's build the package. There seem to be two (sensible) ways we could do this, considering that we have to integrate (merge) the branches we just created, before we fire off the building scripts:
  1. by using a temporary (or "throw-away") branch off upstream, where we integrate all the branches we have just created, build the package, tag our master branch (it contains debian/changelog), and remove the temporary branch. When a new package needs to be built, we repeat the process.
  2. by using a long-living integration branch off upstream, into which we merge all our branches, tag the branch, and build the package off the tag. When a new package comes around, we re-merge our branches, tag, and build.
Both approaches have a certain appeal to me, but I settled for the second, for two reasons, the first of which leads to the second:
  1. When I upload a package to the Debian archive, I want to create a tag which captures the exact state of the tree from which the package was built, for posterity (I will return to this point later). Since the throw-away branches are not designed to persist and are not uploaded to the archive, tagging the merging commit makes no sense. Thus, the only way to properly identify a source tree across all involved branches would be to run git-tag $branch/$tagname $branch for each branch, which is purely semantic and will get messy sooner or later.
  2. As a result of the above: when Debian makes a new stable release, I would like to create a branch corresponding to the package in the stable archive at the time, for security and other proposed updates. I could rename my throw-away branch, if it still existed, or I could create a new branch and merge all other branches, using the (semantic) tags, but that seems rather unfavourable.
So instead, I use a long-living integration branch, notoriously tag the merge commits which produced the tree from which I built the package I uploaded, and when a certain version ends up in a stable Debian release, I create a maintenance branch off the one, single tag which corresponds to the very version of the package distributed as part of the Debian release. So much for the theory. Let's build, already!
Building the package (practise) So we need a long-living integration branch, and that's easier done than said: 
$ git checkout -b build mdadm-2.6.3+200709292116+4450e59
Now we're ready to build, and the following procedure should really be automated. I thus write it like a script, called poor-mans-gitbuild, which takes as optional argument the name of the (upstream) tag to use, defaulting to upstream (the tip): 
#!/bin/sh
set -eu
git checkout master
debver=$(dpkg-parsechangelog   sed -ne 's,Version: ,,p')
git checkout build
git merge $ 1:-upstream 
git merge upstream-patches
git merge master
for b in $(git for-each-ref --format='%(refname)' refs/heads/deb/*); do
  git merge -- $b
done
git tag -s debian/$debver
debuild   # will ignore .git automatically
git checkout master
Note how we are merging each branch in turn, instead of using the octopus merge strategy (which would create a commit with more than two parents) for reasons outlined in this post. An octopus-merge would actually work in our situation, but it will not always work, so better safe than sorry (although you could still achieve the same result). If you discover during the build that you forgot something, or the build script failed to run, just remove the tag, undo the merges, checkout the branch to which you need to commit to fix the issue, and then repeat the above build process: 
$ git tag -d debian/$debver
$ git checkout build
$ git reset --hard upstream
$ git checkout master
$ editor debian/rules    # or whatever
$ git add debian/rules
$ git commit -s
$ poor-mans-gitbuild
Before you upload, it's a good idea to invoke gitk --all and verify that all goes according to plan:
screenshot of gitk after the above steps
When you're done and the package has been uploaded, push your work to git.debian.org, as before. Instead of using --all and --tags, I now specify exactly which refs to push. This is probably a good habit to get into to prevent publishing unwanted refs: 
$ git push origin build tag debian/2.6.3+200709292116+4450e59-3
Now take your dog for a walk, or play outside, or do something else not involving a computer or entertainment device.
Uploading a new Debian version If you are as lucky as I am, the package you uploaded still has a bug in the upstream code and someone else fixes it before upstream releases a new version, then you might be in the position to release a new Debian version. Or maybe you just need to make some Debian-specific changes against the same upstream version. I'll let the commands speak for themselves: 
$ git checkout upstream-patches
$ git-apply < patch-from-lunar.diff   #444682 again
$ git commit --author 'J r my Bobbio <lunar@debian.org>' -s
  # this should also be automated, see below
$ git checkout master
$ dch -i
$ dpkg-parsechangelog   sed -ne 's,Version: ,,p'
2.6.3+200709292116+4450e59-3
$ git commit -s debian/changelog
$ poor-mans-gitbuild
$ git push
$ git push origin tag debian/2.6.3+200709292116+4450e59-3
That first git-push may require a short explanation: without any arguments, git-push updates only the intersection of local and remote branches, so it would never push a new local branch (such as build above), but it updates all existing ones; thus, you cannot inadvertedly publish a local branch. Tags still need to be published explicitly.
Hacking on the software Imagine: on a rainy Saturday afternoon you get bored and decide to implement a better way to tell mdadm when to start which array. Since you're a genius, it'll take you only a day, but you do make mistakes here and there, so what could be better than to use version control? However, rather than having a branch that will live forever, you are just creating a local branch, which you will not publish. When you are done, you'll feed your work back into the existing branches. Git makes branching really easy and as you may have spotted, the poor-mans-gitbuild script reserves an entire branch namespace for people like you: 
$ git checkout -b tmp/start-arrays-rework master
Unfortunately (or fortunately), fixing this issue will require work on two branches, since the initramfs script and hook are maintained in a separate branch. There are (again) two ways in which we can (sensibly) approach this:
  • create two separate, temporary branches, and switch between them as you work.
  • merge both into the temporary branch and later cherry-pick the commits into the appropriate branches.
I am undecided on this, but maybe the best would be a combination: merge both into a temporary branch and later cherry-pick the commits into two additional, temporary branches until you got it right, and then fast-forward the official branches to their tips: 
$ git merge master deb/initramfs
$ editor debian/mdadm-raid                     #  
$ git commit -s debian/mdadm-raid
$ editor debian/initramfs/script.local-top     #  
$ git commit -s debian/initramfs/script.local-top
[many hours of iteration pass ]
[  until you are done]
$ git checkout -b tmp/start-arrays-rework-init master
  # for each commit $c in tmp/start-arrays-rework
  # applicable to the master branch:
$ git cherry-pick $c
$ git checkout -b tmp/start-arrays-rework-initramfs deb/initramfs
  # for each commit $c in tmp/start-arrays-rework
  # applicable to the deb/initramfs branch:
$ git cherry-pick $c
This is assuming that all your commits are logical units. If you find several commits which would better be bundled together into a single commit, this is the time to do it: 
$ git cherry-pick --no-commit <commit7>
$ git cherry-pick --no-commit <commit4>
$ git cherry-pick --no-commit <commit5>
$ git commit -s
Before we now merge this into the official branches, let me briefly intervene and introduce the concept of a fast-forward. Git will "fast-forward" a branch to a new tip if it decides that no merge is needed. In the above example, we branched a temporary branch (T) off the tip of an official branch (O) and then worked on the temporary one. If we now merge the temporary one into the official one, Git determines that it can actually squash the ancestry into a single line and push the official branch tip to the same ref as the temporary branch tip. In cheap (poor man's), ASCII notation: 
- - - O             >> merge T >>     - - - = - - OT
         - - T      >>  into O >>
This works because no new commits have been made on top of O (if there would be any, we might be able to rebase, but let's not go there quite yet; rebasing is how you shoot yourself in the foot with Git). Thus we can simply do the following: 
$ git checkout deb/initramfs
$ git merge tmp/start-arrays-rework-initramfs
$ git checkout master
$ git merge tmp/start-arrays-rework-init
and test/build/push the result. Or well, since you are not an mdadm maintainer (We^W I have open job positions! Applications welcome!), you'll want to submit your work as patches via email: 
$ git format-patch -s -M origin/master
This will create a number of files in the current directory, one corresponding for each commit you made since origin/master. Assuming each commit is a logical unit, you can now submit these to an email address. The --compose option lets you write an introductory message, which is optional: 
$ git send-email --compose --to your@email.address <file1> <file2> < >
Once you've verified that everything is alright, swap your email address for the bug number (or the pkg-mdadm-devel list address). Thanks (in advance) for your contribution! Of course, you may also be working on a feature that you want to go upstream, in which case you'd probably branch off upstream-patches (if it depends on a patch not yet in upstream's repository), or upstream (if it does not): 
$ git checkout -b tmp/cool-feature upstream
[ ]
when a new upstream version comes around After a while, upstream may have integrated your patches, in addition to various other changes, to give birth to mdadm-2.6.4. We thus first fetch all the new refs and merge them into our upstream branch: 
$ git fetch upstream-repo
$ git checkout upstream
$ git merge upstream-repo/master
we could just as well have executed git-pull, which with the default configuration would have done the same; however, I prefer to separate the process into fetching and merging. Now comes the point when many Git people think about rebasing. And in fact, rebasing is exactly what you should be doing, iff you're still working on an unpublished branch, such as the previous tmp/cool-feature off upstream. By rebasing your branch onto the updated upstream branch, you are making sure that your patch will apply cleanly when upstream tries it, because potential merge conflicts would be handled by you as part of the rebase, rather than by upstream: 
$ git checkout tmp/cool-feature
$ git rebase upstream
What rebasing does is quite simple actually: it takes every commit you made since you branched off the parent branch and records the diff and commit message. Then, for each diff/commit_message pair, it creates a new commit on top of the new parent branch tip, thus rewrites history, and orphans all your original commits. Thus, you should only do this if your branch has never been published or else you would leave people who cloned from your published branch with orphans.
If this still does not make sense, try it out: create a (source) repository, make a commit (with a meaningful commit message), branch B off the tip, make a commit on top of B (with a meaningful message), clone that repository and return to the source repository. There, checkout the master, make a commit (with a ), checkout B, rebase it onto the tip of master, make a commit (with a ), and now git-pull from the clone; use gitk to figure out what's going on.
So you should almost never rebase a published branch, and since all your branches outside of the tmp/* namespace are published on git.debian.org, you should not rebase those. But then again, Pierre actually rebases a published branch in his workflow, and he does so with reason: his patches branch is just a collection of branches to go upstream, from which upstream cherry-picks or which upstream merges, but which no one tracks (or should be tracking). But we can't (or at least will not at this point) do this for our feature branches (though we could treat upstream-patches that way), so we have to merge. At first, it suffices to merge the new upstream into the long-living build branch, and to call poor-mans-gitbuild, but if you run into merge conflicts or find that upstream's changes affect the functionality contained in your feature branches, you need to actually fix those. For instance, let's say that upstream started providing md.txt (which I previously provided in the deb/docs branch), then I need to fix that branch: 
$ git checkout deb/docs
$ git rm md.txt
$ git commit -s
That was easy, since I could evade the conflict. But what if upstream made a change to Makefile, which got in the way with my configuration file location change? Then I'd have to merge upstream into deb/conffile-location, resolve the conflicts, and commit the change: 
$ git checkout deb/conffile-location
$ git merge upstream
CONFLICT!
$ git-mergetool
$ git commit -s
When all conflicts have been resolved, I can prepare a new release, as before: 
$ git checkout master
$ dch -i
$ dpkg-parsechangelog   sed -ne 's,Version: ,,p'
2.6.3+200709292116+4450e59-3
# git commit -s debian/changelog
$ poor-mans-gitbuild
# git push
$ git push origin tag debian/2.6.3+200709292116+4450e59-3
Note that Git often appears smart about commits that percolated upstream: since upstream included the two commits in upstream-patches in his 2.6.4 release, my upstream-patches branch got effectively annihilated, and Git was smart enough to figure that out without a conflict. But before you rejoice, let it be told that this does not always work.
Creating and using a maintenance branch Let's say Debian "lenny" is released with mdadm 2.7.6-1, then: 
$ git checkout -b maint/lenny debian/2.7.6-1
You might do this to celebrate the release, or you may wait until the need arises. We've already left the domain of reality ("lenny" is not yet released), so the following is just theory. Now, assume that a security bug is found in mdadm 2.7.6 after "lenny" was released. Upstream is already on mdadm 2.7.8 and commits deadbeef and c0ffee fix the security issue, then you'd cherry-pick them into the maint/lenny branch: 
$ git checkout upstream
$ git pull
$ git checkout maint/lenny
$ git cherry-pick deadbeef
$ git cherry-pick c0ffee
If there are no merge conflicts (which you'd resolve with git-mergetool), we can just go ahead to prepare the new package: 
$ dch -i
$ dpkg-parsechangelog   sed -ne 's,Version: ,,p'
2.7.6-1lenny1
$ git commit -s debian/changelog
$ poor-mans-gitbuild
$ git push origin maint/lenny
$ git push origin tag debian/2.7.6-1lenny1
Future directions It should be trivial to create the Debian source package directly from the repository, and in fact, in response to a recent blog post of mine on the dispensability of pristine upstream tarballs, two people showed me their scripts to do it. My post also caused Joey Hess to clarify his position on pristine tarballs, before he went out to implement dpkg-source v3. This looks very promising. Yet, as Romain argues, there are benefits with simple patch management systems. Exciting times ahead! In addition to creating source packages from version control, a couple of other ideas have been around for a while:
  • create debian/changelog from commit log summaries when you merge into the build branch.
  • integrate version control with the BTS, bidirectionally:
    • given a bug report, create a temporary branch and apply any patches found in the bug report.
    • upon merging the temporary branch back into the feature branch it modifies, generate a patch, send it to the BTS and tag the bug report + pending patch.
And I am sure there are more. If you have any, I'd be interested to hear about them!
Wrapping up I hope this post was useful. Thank you for reading to the end, this was probably my longest blog post ever. I want to thank Pierre Habouzit, Johannes Schindelin, and all the others on the #git/freenode IRC channel for their tutelage. Thanks also to Manoj Srivastava, whose pioneering work on packaging with GNU arch got me started on most of the concepts I use in the above. And of course, the members of the the vcs-pkg mailing list for the various discussions on this subject, especially those who participated in the thread leading up to this post. Finally, thanks to Linus and Junio for Git and the continuously outstanding high level of support they give. If you are interested in the topic of using version control for distro packaging, I invite you to join the vcs-pkg mailing list and/or the #vcs-pkg/irc.oftc.net IRC channel. NP: Aphex Twin: Selected Ambient Works, Volume 2 (at least when I started writing )

Antti-Juhani Kaijanaho: It s time to fix the ABI

SELinux is entirely correct about disallowing dynamic code generation, as it is a major security risk. Disregarding Just-In-Time compilation, the main legitimate need for dynamic code generation is to support (downward) closures that are ABI-compatible with normal C functions. GCC’s local functions extension of C is one example, and many non-C languages need them badly in their foreign-function interfaces (Haskell is one, Ada I’m told is another). A closure is a function pointer that refers to a function that is local to another function. That function has access to the local variables of the parent function, and this access is provided by having the caller give the callee a static link (a pointer to the parent function’s stack frame) as a hidden parameter. If the call is direct (that is, not through a function pointer), the caller knows the appropriate static link and can just pass it. The trouble comes with function pointers, as we need some way of including the static link in the function pointer. The simplest way is to have function pointers be two words long; one of the words contains the classical function pointer (that is, the entry point address), and the other contains the static link. Unfortunately, the prevalent C ABIs, including the SYSV ABI used by GNU/Linux, mandate that function pointers are one word long. The only way I know to work around this is to dynamically generate, when the function pointer is created, a small snippet of code that loads the correct static link to the appropriate place and jumps to the actual code of the function, and use the address of this snippet (usually called a trampoline) as the function pointer. The snippet is generated on the stack or in the heap, and thus requires an executable stack or executable dynamic memory. It’s time to fix the ABI to allow for proper two-word function pointers.

Axel Beckert: Group packages by origin in aptitude

I always wondered how others recognise non-Debian packages in the aptitude package tree. I also missed the additional priority level in the hierachy well-known from good old dselect. For the last one, I quickly found out that you can set the priority as subsection — it’s straight forward after you’ve read the documentation: Just add ,priority at the end of the default grouping method for package views under “Options → UI Options” in the aptitude menu. Getting the origin as given in the Release file of the repository a package originates from is a little bit more difficult. You need to use the pattern() group function with the appropriate search pattern: pattern(~O) Since already the default default grouping method for package views doesn’t fit into the dialog, I nowadays just edit /etc/apt/apt.conf directly for changes on aptitude’s default grouping method for package views. It now looks like this on several of my machines: 
Aptitude::UI  
  Default-Grouping "filter(missing),status,section(subdir,passthrough),pattern(~O),section(topdir),priority";
 ;
In aptitude this looks like this: 
[...]
  --- text - Text processing utilities
  --\ utils - Various system utilities
    --- Debian
    --- Mowgli
    --- volatile.debian.org
  --\ web - Web browsers, servers, proxies, and other tools
    --- Debian
    --- Opera Software ASA
  --\ x11 - The X window system and related software
    --\ Debian
      --- contrib - Programs which depend on software not in Debian
      --\ main - The main Debian archive
        --- Priority optional
        --- Priority extra
      --- non-free - Programs which are not free software
    --- Mowgli
[...]
Unfortunately this doesn’t work with all non-Debian repositories since a few repository maintainer, e.g. those from Emdebian, arrogate to just keep “Debian” as their packages’ origin. This could be solved, if there’s a possibility to group by e.g. repository URL (host and/or path). Another problem I haven’t solved yet is that grouping by origin does neither work with locally created nor virtual packages nor tasks — probably since all of them lack an origin. Those branches are just empty or don’t even show up anymore with this configuration. I probably have to dig a little bit more in the aptitude documentation to resolve this.

Now playing: E-Rotic — Max don’t have sex with your ex

9 October 2007

David Welton: Languages: worst case vs average

This is a pretty good take on Tim Bray vs Erlang. However, I'd like to focus on something said at the end:
The fact is, all languages optimize for particular problem domains and I don't think that a universal programming language exists that would perform well on all tasks. Bray rapidly found out that Erlang isn't optimized for doing line-oriented I/O and it's regex library sucks. So what? While those problems could be eliminated from Erlang, the fact that Ericsson has deployed large telecom gear without having to fix those issues means that Erlang is ideally suited to its original programming domain.
The problem is that learning new languages is difficult for many people. They're not going to bother starting from 0 with a new language if what they've got is only 10/20% worse than the other guys. For companies, developing expertise (and trying to maintain it) in lots of languages is probably less productive than picking a few (Google only has four: Java, Python, Javascript and C++, and they're not exactly lacking in brainpower or cash). This is why "the right tool for the right job" works for things like saws and hammers, but not for complicated, idiomatic things like programming languages. Therefore, it's important that languages that aim to be widely used do lots of things pretty well, and try to avoid doing anything really badly. This is something I've always admired about Python, even if I don't use it anymore. It's a language that you can do a very wide variety of tasks with, and you can be sure there's probably some code out there already, and that Python will be ok for your task, even if it's not the best thing. Performance is a weak point compared to compiled languages, but even there, get the libraries right and things aren't *so* bad. In any case, my point isn't to bash Erlang (which I like), or put Python on a pillar (I prefer Ruby and Tcl), but to say that a widely used programming language doesn't need to be the best at everything, but it should aim to make people comfortable using it, safe in the knowledge that they can probably do most things *ok* if needs be. That's a far better situation than being able to do one thing realy well, and perform poorly at many other tasks, forcing its users to look elsewhere for solutions to those tasks. Personally, I think that for concurrency, Erlang is in a bit of a race right now - they're pretty far ahead in terms of concurrency (and also fault tolerance, but fewer people care about that), but behind in other areas (less library code). Other languages don't have as good a tools for concurrency, which isn't an easy problem. So, what will happen first - Erlang catches up elsewhere, other languages get "good enough" concurrency solutions, or a new language (like Scala) comes along and takes the lead?

MJ Ray: Cooperatives SW September steering

I was at the September meeting of the Cooperatives SW steering group a couple of weeks ago. We had two new members (Alex Lawrie from Upstart and Alan Bonner from RadCo) and a guest from Co-operative Futures. Unfortunately, we seem to have lost our only member from a Credit Union. Our guest told us about how important it is to put information about co-operative business models into Business Link and offer paths from BL "Information and Diagnostic Brokerage" to co-operative development bodies (CDBs). I agree with that - by and large, BL were hopeless about anything other than a sole trader business when I used them (in 2000 and again in 2002/3) and knowing more about co-operative development would have saved me about a year of confused administration. (However, my then-local CDB was also a competitor, so they weren't much help even once I found them.) There are also some other ideas to connect CDBs to funding better, which the CDBs are going to develop with our blessing and bring back later. Related yet different, we were told that the Regional Infrastructure for Social Enterprise is about to launch a trade association and may exclude all co-operatives from it by mistake. Other than that, it was mostly about getting promotional materials and plans in place for the Co-operatives SW Annual Meeting in Exeter on Monday 22 October ( places available to all co-operative members).

Adrian von Bidder: Microsoft declares Bankruptcy

Intellectually, I mean. According to some Media outlets Ballmer (once again) declares war on the Free Software community. The technical “we are better” argument is apparently completely dropped (honesty, at last) and replaced by the pure “we have more money and patents”, this time not even thinly veiled in marketing speak. But I don't think PJ is right when she sais that “FOSS can't survive in an IP regime such as Ballmer has laid before our eyes.”. We've all seen how successful the sue your own customers business model is. At this time, I think it's safe to say that most big Linux users are also big Microsoft customers because Linux in the enterprise still means primarily Linux on the Server, Windows on the Desktop, so going after the big Linux users will be exactly that. Going after the small Linux users is just not interesting. Going after hobbyist FOSS developers will not work because only using patented works commercially actually violates the patent, as far as I understand patent law. Going against the commercial Linux developers will work to some degree, so let's look at this closer: Software patents are strongest in the U.S., so in Europe it will be much harder for MS for one thing. And, Novell and Red Hat excluded (and IBM, who should be big enough to take on MS, especially since their lawyers by now have had extensive training in what Linux is abaut), the commercial Linux space is still dominated by small companies, so MS would have to overload the judges with thousands of small lawsuits all over the world, and since no two companies do exactly the same thing I doubt the lawsuit templating technique adapted by the RIAA for music downloads will work. And, finally, the nice thing about Open Source Software (using this term here on purpose): it's out in the open. So kill one company, its former employees will found three new ones because the former customers (even if they're scared and would like to get rid of their Free Software) will still need support because they can't all switch (back) to MS based environments immediately. And I haven't even talked about public opinion and what antitrust offices might have to say, because those are extremely hard to predict and their effects are uncertain.

8 October 2007

Steve Kemp: Oh, didnt i, didnt i, didnt I see you cryin?

Curse you Debian! Your programs are too secure... So I was looking over some setgid binaries last night, seeing if there were any obvious security bugs. Up popped omega-rpg - a fun game I've recently been playing. Unfortunately it is mostly OK: The only thing that I can is persuade the game to die with a SIGSEG if I manaully edit a save-game file, then load it. I'm sure with care and patience it could be coerced into running shellcode. In theory this is a security hole. In practise it is hard to take seriously! On the other hand I'm not convinced the game should be setgid(games)..

Theodore Ts'o: Sous Vide, Revisited

In a previous post, I had recommended the 4130 NIST-Traceable Temperature Controller to control the temperature in a slow cooker. Unfortunately, that particular controller has a range that tops out at 60 degrees C / 140 degrees F, which is enough for cooking beef for long periods of time, but not enough for say, cooking duck confit, which for which a sous vide temperature of 80 degrees C is recommended. In addition, the 4130 is pretty expensive; almost $150. It’s possible to add a resistor to change the range of the 4130, but the temperature displayed by the controller is no longer correct, and you have manually create a conversion table between true temperature and the temperature as seen by the controller. I’ve recently come across a cheaper and better possibility, the Ranco ETC-111000-000 Temperature Controller which is only half the price and comes with a much larger working range (-30 to 220 degrees F). The price with the AC cord already wired in is $75; and the version which just has a 120VAC SPDT relay is only $60. A bit more about food safety. There has always been a lot of concern about bacteria growth and botulism, for good reason — and so therefore the recommendations for cooking temperature have a lot of safety margin in them — to the point now that the USDA recommends that steaks be cooked to at least 145 degrees F, which is well within what had traditionally been called “medium”, and chicken to at least 165 degrees F, which is enough to really destroy taste and texture. Sous vide cooking, especially some of the more low temperature variants, have raised a lot of concerns, to the point where a few years ago New York City (temporarily) banned it, causing a great outcry in the foodie community, since many top restaurants use sous vide techniques. First of all, any recommendation about internal temperatures and food safety that doesn’t also factor in time is massively oversimplifying the problem. Here is a table taken from “Food Safety Hazards and Controls for the Home Food Preparer”, published by the Hospitality Institute of Technology in 1994:
Temperature, F Time, 5D kill Time, 6.5D kill
130 86.42 minutes 112.34 minutes
135 27.33 minutes 35.53 minutes
140 8.64 minutes 11.23 minutes
145 2.73 minutes 3.55 minutes
150 51.85 seconds 1.12 minutes
155 16.40 seconds 21.32 seconds
160 5.19 seconds 6.74 seconds
165 1.64 seconds 2.13 seconds
This table lists the time to reduce bacteria concentrations of Salmonella and E. coli from 100,000 to 1 (5D) or 3,162,277 to 1 (6.5D). The FDA and USDA recommend cooking hamburger to 5D destruction. Since it is extremely unlikely for there to be more than 100 Salmonella organisms per gram of meat, a 5D kill will reduce Salmonella concentrations to no more than 1 organism per kilogram. So whether you cook a piece of meat to an internal temperature of 165 degrees for 1.64 seconds, or hold it at 130 degrees for 90 minutes, the effect on Salmonella and E. coli bacteria will be the same.Of course, one concern is that in many forms of cooking, particularly oven roasting and grilling, the temperature of the food as it heats up may not be even; so how do you guarantee that all parts of the food product has been brought up to the requisite 165 degrees? One way that recipe authors, fearful of liability concerns, have done so is to tell people to cook meat to much higher internal temperatures, to provide that extra safety margin at the expense of dessicated, horribly tasting turkey or chicken. But the advantage of sous vide cooking is by immersing the food in a water bath, the excellent heat conductivity of water helps guarantee that the entire body of meat will get raised to desired temperature relatively quickly. (How quickly depends on the thickness of the meat, obviously). So if you hold a roast beef that has been vacuum packed in a Foodsaver bag for five or six hours, there should be no question that all of the common bacteria has been inactivated, and that amount of time at 130 degrees F should be sufficient to inactivate 99.9 percent of all botulism toxin molecules (not there should be any in a fresh piece of meat, of course!) However, sous vide temperatures are not enough to kill bacterial spores, in particular C. botulinum, which is responsible for botulism. This requires temperatures far in excess of boiling water at sea level. For example, home canning protocols recommend holding the food product at 250 degrees F for at least 15 minutes. This is an issue in the restaurant business because very often food would be cooked sous vide, and then stored in the vacuum sealed bags for potentially weeks (yes, you could be eating an extremely expensive meal at a top-end French restaurant that had been cooked several weeks ago, and reheated just before serving; yummy, no?). If the food packages aren’t cooled quickly enough, and then allowed to warm to the danger zone, it’s possible that in the anerobic environment the C. botulinum spores could germinate and then start producing toxin. But, if you are cooking home sous vide where you are serving the food right after it has been cooked, this shouldn’t be a concern.

Thijs Kinkhorst: Utrechts Kampioenschap Programmeren

Last Saturday we organised the Utrecht Programming Championship, which apart from declaring a local champion also selects for the Benelux championship. Team A-Cognito won by solving 6 out of 8 problems, the best score of all cities organising parallel contests. See the results here. We organise this every year, but this year was special as it was a general rehearsal for the North Western European contest that we are also hosting this autumn. This is part of the ACM ICPC worldwide network of contests which culminate in the ICPC World Finals. Luckily everything went great. Of course there were some problems, but they were resolved in a calm and adequate fashion. It gives good hope that NWERC will be a success, at least from the techinical side...

7 October 2007

Christian Perrier: RWC: Argentina reaches semi-finals (19-13)

Logically, Argentina won against Scotland tonight after a.....quite boring match. Scotland, as expected for them unfortunately, wasn't able to put speed in their game, the only way to beat Argentina this year. Even with that poor game (Lamont, their rear player, was a disaster), Scotland finished quite close to ARG, who didn't play brilliantly with many approximations (Hernandez being a bit arrogant, trying to kick too many drop goals) and a bit of luck. So, nothing really impressive in that match. Does that mean that South Africa has better chances to qualify for the final: who knows? Argentina can be unpredictable...but it seems they are a little bit burned out (they don't really have 30 top players and some of their key players appear tired). The semi-finals will feature France-England (about 50-50 chances from what we saw up to now) and South Africa-Argentina (60-40, in my opinion). Next week-end, Stade de France.

Christian Perrier: RWC: No surprise this time...

...but, really, Fiji would have deserved the victory. They did put the Stade Velodrome on fire this afternoon, pushing South-Africa to play....a quite bad rugby with few imagination and many errors. In the RSA team, only Steyn emerged and saved his team from sinking (37-20). RSA scored 5 tries mostly only because they have good four-wheel-driven "trucks". In the Fiji side, Sisa Koyamaibole and Mosese Rauluni lead their team to even the score 20-20 on 65th minute...unfortunately, losing in the last 10 minutes despite the support of the entire French crowd. The Springboks need to improve their play dramatically if they want to have a chance against Argentina (very probably unless Scotland realises a miracle). My conclusion today: the Three Nations from southern hemisphere (RSA, NZL, AUS) should really consider opening their annual tournament to other nations such as Samoa, Tonga, Fiji or Argentina. That would certainly force them to play another game than their usual game. Of course, that would mean being less attractive to players (just count the number of players from Samoa, Tonga and Fiji who play with New Zealand and Australia) but considerably widening rugby in the Pacific area.

6 October 2007

Gustavo Franco: Earth is closer

The best sci-fi series for a while IMNSHO, Battlestar Galactica next and last season will start with a two-hours episode (razor) in november 24! Fortunately I'll be in USA. So say we all. :P

Next.

Previous.