Search Results: "cas"

16 October 2007

Marc 'Zugschlus' Haber: A thousand things I never wanted to know about X

In the last few days, I have replaced the two 20 inch CRT monitors that I have hardly used the last years with two 20 inch TFT displays, and my company (finally) gave me a 19 inch TFT display to accompany my notebook display at work. Maybe I should take that as a hint that they want to see me in the office more frequently rather than in my home office which I generously use these days. At home, I built a “new” computer from mainly used parts to drive the two 20 inchers. I have learned a lot about X in the last days, but spent too much time with it. I have been running my Desktop with older X and two CRTs for quite some time, but I hardly ever used it due to the setup of my now ex-flat. Back then, I learned that the “two monitors, one display, one screen, one desktop” option of X was called Xinerama and statically configured: You write down your display layout in xorg.conf and it uses it. Configuration changes mean restarting the X server. KDE notices Xinerama automatically, designates one monitor the “main monitor” and displays its panel there. You can move windows from one monitor to the other one, but changing virtual desktops always changes what both monitors display. This is not really what I want, since I often have “static” contents (news reader, mail reader, chat windows, help screens etc) on one monitor while I do my normal work on the other monitor, which involves changing virtual desktops. So I was rather happy when I learned upon the arrival of the external TFT display for the notebook that you can work without Xinerama. In that case, the X server comes up with one display and two screens, which KDE uses to display two independent desktops which can be independently configured, but can both be accessed with one mouse and one keyboard. For me, this means that I finally can independently change virtual desktops on both monitors, but pay the price that I cannot display a certain virtual desktop on monitor 1 now, and on monitor 2 in five minutes, and that a window opened on monitor 1 is never going to be shown on monitor 2 without serious interference. This looked ok for me. Additionally, you can use different font sizes on the different monitors as defaults, which comes in handy if both monitors are blatantly different in size (which is the case with the notebook and its external monitor). However, with that setup, KDE seems to be challenged when it comes to saving the configuration. Some things (such as desktop background and panel config) seem to save fine, desktop independent, other things (such as open windows and their position) are lost when the session exitst. My daily visit to Kevin&Kell (which used to be in a browser window that was saved with my session) has badly suffered since then. Then, a few weeks ago, somebody talked me into trying the X server from experimental. Which is when my dual-desktop setup ceased working. The nice people on #debian-x told me that Xinerama is a thing of the past, and is currently in the process of being replaced by Xrandr 1.2. The removal of “my” two monitor, two screens, two desktop setup is collateral damage, and if I still want that behavior, the X.org upstream considers this an issue to be handled by the Window Manager / Desktop Environment now. Gee, thanks. That’s what I call a regression. After spending half a day cursing and ranting, and even considering to go back to lenny’s (or even etch’s) X.org, I finally settled on going with the “new” way as I was going to lose my independent desktops sooner or later anyway. Again, with the friendly help of the people on #debian-x, I found out about a lot of advantages of the latest X servers. In most basic cases, the latest X server does not need a configuration file any more. It autodetects your hardware and chooses whatever modes it finds optimal. It does a pretty decent job in doing so. If one wants to influence how the X system comes up, once can always write an Xorg.conf file - the configfileless server even writes the “virtual config file” it uses to the log as a starting point. Additionally, almost everything regarding screen layout can be reconfigured at run time using the xrandr binary from a command line. Even a hardware change (such a new monitor plugged into the powered on notebook) is correctly detected. Cute. This allows me to reconfigure my notebook depending on where it currently is and which external monitor (or projector) is connected. This can be perfected if I find out whether X offers a hook to plug a reconfiguration script into that is called when the X server starts or when the system is waking up from standby or hibernation with the X server running. A lot to do, and great potential for elegant stuff. However, while I am talking about hibernation, this is a downside of the new driver: When the system wakes up from hibernation, the X server sometimes confuses itself enough to become nearly unuseable: I had it become psychotic when plugging in the external monitor (which can do 1600x1200 points) after waking it up from hibernation at the office. The internal display can only do 1400x1050, and the box thought that the external monitor only had 1050 y axis pixels, displaying bit garbage in the lower part of the big monitor, and showing a useable, but unintelligible panel. When opening any browser or any other application with considerable text amounts in the window, all fonts become unreadable once one scrolls down. Restarting the X server helps and everyting is fine. But I do not use hibernation to force myself to log out of X after waking up. After going back to the “one monitor mode”, the X server now thinks that my virtual desktop is 1600x1200 and only shows 1400x1050, with dead space at the right and lower side of the virtual desktop. xrandr does not allow me to reisize the FB below 1600x1200. No idea what’s going on here. In most of the cases, calling xrandr --auto for each output will fix the issues at run time, but sometimes a session restart is needed. Code needs to stabilize quite a bit before becoming useable in production. There are a few disadvantages that will remain: Both monitors will change their contents when the virtual desktop is changed, and both displays will use the same default font size (which might be inappropriate if both displays are of different size). But I think, when the issues are cleared up, that I can live with these disadvantages. The “new” desktop system that I built has a Matrox G450 dual head graphics card, and unfortunately, the Xrandr 1.2 enabled X server from experiemental has shown to be unuseable: Upon some mode changes, the X server goes into an endless loop and proves itself to be unkillable short of SIGKILL. And if it dies, it sometimes pulls the entire system with it. There is a corresponding bug in the upstream BTS, but it hasn’t received any noticeable attention by the upstream developers. So I might find myself configuring one last Xinerama config before I can finally completly migrate to Xrandr 1.2 on the desktop as well. Then I’ll probably need to bug the KDE upstream people to implement a two-desktops-on-a-single-screen feature in one of their future versions so that I can get my independent-desktop-things back. But that’s a far far away future.

15 October 2007

Martin F. Krafft: CO and the environment

Today is environment blog action day. I don't like memes or mass actions of this sort, but given a recent discussion on the effects of Al Gore's movie An Inconvenient Truth and everyday/everyone environmental protection on a UL-internal discussion list called "opinions", I feel like sharing my views. The original poster suggested everyone to see the movie and was promptly smashed down by two people who responded along the lines of "the movie has no scientific basis," quoting fancy research on how global warming is not as bad as it's hyped to be, and that CO is not as guilty as everyone claims. None of this research is relevant to what I had to say to this, so I won't quote it, but if you're interested, the Intergovernmental Panel on Climate Change (IPCC) is the place to start. One person went further and compared the CO -reduction hype to communism and criticised how we're expecting everyone to take action on the basis of false claims. I was partly disgusted by (all) those replies, not because they weren't accurate, but because they simply miss the point. And because the entire discussion was just another iteration of talk versus action. It may well be true that CO is not that bad, after all. But it's also true that any reasonable attempt to reduce its emission is not going to harm the environment (not to say that it might actually have a net-positive effect). Or as I put it in the thread:
I don't care for politics, I don't even care for research on this matter. I care for my environment, and I don't need to be a scientist to assume that plastic bags and car exhaust and lights left on when out to the pub are not making it any better. And yet, I am not a militant green. I prefer halogen light bulbs to the energy-saving ones and occasionally I'll be at the supermarket without a reusable bag; I don't condemn people using cars instead of a 15 minute walk to the office, and I still haven't managed to talk to a housemate of mine (no, not my Swiss flatmate), who notoriously leaves lights and appliances running when away. But at least I can say that I think of these issues, without scientific basis, but common sense. And I'll say that a thousand people aware of that will make more of a difference than any politician out there.
Not only do I think about these issues, I also act accordingly. But to be able to do that, I had to become aware of them. In as such, I'll stand by the claim in the discussion: awareness is what we ought to strive for, rather than talking about possible ways to start considering how we could potentially one day make leaps towards the solution of the world's problems, of which the environment is one. Awareness and little steps, which everyone can take: turn off lights and appliances, turn down the heater, use reusable shopping bags, prefer local produce to imported products, avoid fancy packaging, and so on In the thread, I asked "What have you done today to help lessen the burden we place on our environment?" I got no replies. I hope that's because people couldn't be bothered to show off their little contributions to the environment. If that were the case, then we are on the right path. NP: Gazpacho: Night

Russell Coker: Blog Copyright Infringement

I have previously written about some of my efforts to counter sploggers [1]. Since then I have had a particularly brazen splogger copy one of my posts entirely and claim to have written it. The only reason I noticed the copyright violation (my blog license is on my About Page [2]) was because the post in question linked to other posts of mine and I saw the links. I was offended by the flagrant violation of all aspects of copyright law (breaking the license and infringing my moral rights by not attributing me as the author) and by the fact that the splog in question was hosted by Dreamhost (who have offended me by refusing a DMCA take-down request). So I decided that merely issuing a DMCA take-down was not enough. I went through the splog and identified content copied from several major journals (including by a journalist I regard as a friend) as well as by one multi-national corporation - and I notified all the relevant people. The splog in question deleted all it’s old content the next day, and immediately started copying new articles from other blogs. I have informed the people who appear to be copyright holders for some of the new articles… I recommend that other people who deal with sploggers also go to the extra effort of notifying other victims. It’s usually quite easy to do you just select a random bit of text from the copied article and paste it into your favourite search engine - usually you get only a single result. Some of the splog posts are edited in small ways so the first search may fail - if so then you merely need to search for a second piece of text. If you only request that your own illegally copied material be taken down then the splogger still has a good business model. They can keep copying content in violation of the license, occasionally take a post down when they get caught, and both the splogger and the ISP continue to make money. If you notify other victims (many of which won’t have the skills to find the content themselves or the background knowledge sufficient to recognise the benefits in having it removed unless you explain it to them) then the splogger loses a lot of content at one go and the ISP will have a more difficult time claiming to be innocent of the process. Also when you notify multi-national corporations you can expect that they have some decent lawyers and a budget assigned to such work. While I would be extremely unlikely to sue an ISP that repeatedly hosts unauthorised copies of my copyright materiel the same can’t be said for a corporation. For more information on splogging see the Wikipedia entry [3].

14 October 2007

Martin F. Krafft: Converting a package to Git

Previously, I demonstrated a Debian packaging workflow using Git and I mentioned the possibility of a follow-up post; well, here it is: you want to use my workflow (or one that's related) for a package that is currently maintained with Subversion on svn.debian.org and you'd like to keep the history during the conversion. Make sure to read the previous post before this one. I am again using the example of mdadm since its Git packaging repository is in a state of shambles and I want to restart to get it right and import the history from the previous Subversion repository. What better way than to write a blog post as I do so? Well, plenty actually. This kind of post isn't really made for a blog, and I have started work on setting up ikiwiki on madduck.net, but it's not yet ready, so I'll stick with the blog for now. I will make sure that links don't break as I move content over, so feel free to bookmark this
Importing the package into Git Thanks to git-svn, the initial step of getting your package imported into Git is a breeze:
$ git-svn clone --stdlayout --no-metadata \
    svn+ssh://svn.debian.org/svn/pkg-mdadm/mdadm mdadm
Sit back and enjoy. If that command exits prematurely with an error such as the following:
Malformed network data: Malformed network data at /usr/local/bin/git-svn line 1029
then you should upgrade to a newer Git version, or have a look here. If your Git does not know --stdlayout then upgrade as well (or use -T trunk -t tags -b branches instead). Sam Vilain notes that it is important to "get the attribution right with the final SVN import - getting the authors map right. I didn't do that. If you look at the repository resulting from the above command, you'll notice strange commit authors, such as madduck@some-unique-uuid-from-svn . git-svn allows you to map these to real names with real email addresses, which ensures that the attributions are good for the whole world to see. When done, switch to the repository and run git-branch -r. As you'll see, git-svn imported all SVN branches and tags as remote branches. You need those if you want to bidirectionally track the Subversion repository, but we are converting, as you may have guessed by the --no-metadata switch above. Therefore, we resort to the Dinosaur method of converting branches to tags, which I'll simplify for mdadm. We also just delete all remote branches after tagging, since mdadm never used branches in the SVN repository. Your mileage may vary.
git branch -r   sed -rne 's, *tags/([^@]+)$,\1,p'   while read tag; do
  echo "git tag debian/$tag tags/$ tag ^; git branch -r -d tags/$tag"
done
git branch -r   while read tag; do
  echo "git branch -r -d $tag"
done
If that seems to work alright, then you can execute the commands. Sam Vilain (again) hints me at git-pack-refs and then to edit .git/packed-refs with an editor. This certainly leaves more room for errors but might be significantly faster.
Cleaning up the SVN references Even though we passed --no-metadata to git-svn, it did leave some traces in .git/, which we can now safely remove:
$ git config --remove-section svn-remote.svn
$ rm -r .git/svn
Setting things straight You can skip this section unless you want to know a bit about how to fix up stuff with Git. There was actually some nasty tagging errors leading up to the 2.5.6-9 release for etch and I could never be bothered to fix those in SVN, but now I can (I love Git!):
$ git tag -d debian/2.5.6-10            # never existed
$ git tag -f debian/2.5.6-8 2.5.6-8~2   # mistagged
$ git checkout -b maint/etch 2.5.6-8    # this is when we diverged
$ git apply < /tmp/mdadm-2.5.6-8..2.5.6-9.diff
$ git add debian/po/gl.po debian/po/pt.po debian/changelog
$ git commit -s
$ git tag debian/2.5.6-9
Now that that's fixed, there is one other thing to worry about, namely the very last commit to SVN, which obsoletes the repository and points to the Git repository. But that's not all of it. I was also silly enough to include a fix in the same commit. Let's see what Git can do. Since the process of obsoletion involves all but adding a file, we can simply --amend the last commit and provide a new log message:
$ git checkout master
$ git rm OBSOLETE debian/OBSOLETE
$ git commit --amend
Now the repository is in an acceptable state.
Making ends meet The pkg-mdadm effort on svn.debian.org only maintained the ./debian/ directory, separate from the upstream code, and boy was that a bad idea. Just to give one example: think about what's involved in preparing a Debian-specific patch against the upstream code this has to end, and we can make it end right here; let's import upstream's code (again not using his ADSL line, but the upstream branch of the pkg-mdadm Git repository; see the previous post for details):
$ git remote add upstream-repo git://git.debian.org/git/pkg-mdadm/mdadm
$ git config remote.upstream-repo.fetch \
    +refs/heads/upstream:refs/remotes/upstream-repo/upstream
$ git fetch upstream-repo
$ git checkout -b upstream upstream-repo/master
Now we have two unconnected ancestries in our repository, and it's time to join them together. The most logical way seems to be to use the last upstream tag for which we have a Debian tag: 2.6.2. For this, we branch off the corresponding Debian tag (2.6.2-1) and merge upstream's 2.6.2 tag into the new branch. This will be a temporary branch Then, we rebase (remember, nothing has been published yet) the master branch on top of this temporary branch, before we end that branch's short life. The Debian tag stays where it is since it describes the state of the repository at time of the release of 2.6.2-1.
$ git checkout -b tmp/join debian/2.6.2-1
$ git merge mdadm-2.6.2
$ git rebase tmp/join master
$ git branch -d tmp/join
It just so happens that the head of the SVN repository, which is identical to the tip of our master branch, corresponds to Debian release 2.6.2-2, so we tag it:
$ git tag debian/2.6.2-2
We are now also "born" in the sense that maintenance in Git has started. Let's mark that point in history. There is no real reason I can foresee for this yet, but nonetheless:
$ git tag -s git-birth
Turning dpatch files into feature branches We want to turn dpatch files into feature branches and we somehow make it "proper". We could branch, apply the patch, delete the patch file, checkout master and delete the patch file there as well, but that appears "improper" to me at least; so instead, we'll cherry-pick:
$ git checkout -b deb/conffile-location
$ debian/patches/01-mdadm.conf-location.dpatch -apply
$ git rm debian/patches/01-mdadm.conf-location.dpatch
$ git commit -s
$ git commit -s $(git ls-files --others --modified)
I should quickly intervene to make sure you are following. I am making use of Git's index here. Applying the patch makes the changes in the working tree, but we did not tell Git that we want those to be part of the commit just yet. Instead, we delete the dpatch with git-rm, which automatically registers the deletion with the index. Thus, the first git-commit creates a commit which deletes the dpatch, while the second git-commit creates a commit with all the changes from the dpatch, using git-ls-files to identify new and modified files. But for now, let's move on. We have two commits in the deb/conffile-location branch, and one of those is relevant to the master branch, we cherry-pick it:
$ git cherry-pick deb/conffile-location^
If you're confused, let me explain: our goal is to have a number of feature branches, of which master is the one in which most of ./debian/ is maintained. All the branches later come together in the long-living build branch, so deb/conffile-location will never be merged back into master. However, once we applied the dpatch to the feature branch, we can delete it from there and the master branch. By cherry-picking, we "import" the deletion to the master branch. I repeat the same procedure for deb/docs, merging all the documentation-related dpatches, but I'll spare you the details.
and then Git let me down In the next step, I found I had misunderstood Git merging: I thought Git was smart, but Linus had his reasons for calling Git the "stupid content tracker" (more on that later). Read on as I am obsoleting dpatch files that upstream had merged: 99-*-FIX.dpatch. For consistency, I wanted to cherry-pick each of the appropriate upstream commits into the master branch along with deleting the corresponding dpatch file. Here is one example: 99-monitor-6+10-FIX.dpatch was obsoleted by upstream's commit 66f8bbb; the -x records the original commit ID in the log:
$ git cherry-pick -x 66f8bbb
$ git rm debian/patches/99-monitor-6+10-FIX.dpatch
$ git commit -s -m"remove dpatch obsoleted by $(git rev-parse --short HEAD)"
I repeated the procedure for the other dpatch files, removed the dpatch infrastructure, and then went on to merge it all into build to build the package. The build branch is a long-living branch off upstream, but which upstream? I'll fast-forward you past a segfault problem with mdadm, which upstream (thought to have) resolved with commit 23dc1ae after 2.6.3, but he had not yet released 2.6.4. Looking at the commits between 23dc1ae and upstream's HEAD at the time, I decided to include them all and snapshot 4450e59:
$ git fetch upstream-repo
$ git checkout upstream
$ git merge upstream-repo
$ git tag mdadm-2.6.3+200709292116+4450e59 4450e59
$ git checkout master
$ git merge --no-commit mdadm-2.6.3+200709292116+4450e59
$ dch -v mdadm-2.6.3+200709292116+4450e59-1
$ git add debian/changelog
$ git commit -s
And then I called poor-mans-gitbuild, which merges master and then deb/* into build. Here is when stuff blew up. I'll make a long story short (read my description of the problem and Linus' answer if you want to know more): I thought Git was smart to identify merges common to both branches and do the right thing, but it turn out that Git does not care at all about commits, it only worries about content and the end result. In our case, unfortunately (or fortunately), the outcome meant a conflict because the upstream branch introduced a simple change (last hunk) in the lines surrounding the patch we cherry-picked, and Git can't handle it. The solution is not to cherry-pick, to cherry-pick all commits touching the context of the dpatch, or to simply merge upstream into all out feature branches. In our case, the first is the easiest solution and since importing dpatch files is a one-time thing (thank $DEITY), I'll leave it at that. Almost. I have spent two days thinking about this more than I should have. And it was this point Linus made which made me appreciate Git even more:
Conflicts aren't bad - they're good. Trying to aggressively resolve them automatically when two branches have done slightly different things in the same area is stupid and just results in more problems. Instead, git tries to do what I don't think anybody else has done: make the conflicts easy to resolve, by allowing you to work with them in your normal working tree, and still giving you a lot of tools to help you see what's going on.
The end This concludes today's report. Importing the changes from the old Git repo, tagging and merging the branches is all covered in my previous post, or at least you'll find enough information there to complete the exercise. I would like to specifically thank Sam Vilain and Linus Torvalds for their help in preparing this post, as well as the #git/freenode inhabitants, as always. If you are interested in the topic of using version control for distro packaging, I invite you to join the vcs-pkg mailing list and/or the #vcs-pkg/irc.oftc.net IRC channel. Also, if you are interested in Git in general, you can find a list of blog posts on the Git wiki. NP: The Police: Zenyatta Mondatta

Marco d'Itri: Bug or feature? s/// and the g option

I recently discovered some perl behaviour that I cannot explain. My goal in this code was to get the matched string in $1. I see how only having a scalar for multiple possible matches (the result of g) could be a problem, but I expected to alyway get something. The effect instead is that $1 may or may not be defined, depending on apparently unrelated changes like setting the i option or changing the regexp in a way which does not alter the matched text. My test case follows.

#!/usr/bin/perl
use warnings;
use strict;
#use re 'debugcolor';
my $text = <<END;
XXWY
XXWZ
END
# $1 is defined only if I remove either of ?, g or i
my $count = $text =~ s#(XXW?Y)##gi;
print "REMOVED: <<$1>>\nCOUNT: $count\n";

John Goerzen: My New Printer

Yesterday, Terah and I were in a store selling used items. I spotted a good deal: a used printer for $25. Here are some of its features: Here are some photos:
img_3705.jpg
img_3707.jpg
img_3706.jpg

MJ Ray: Paris-Tours

Today sees the 101st Paris-Tours race, the last-but-one ProTour race. It’s showing on Eurosport (free-to-air at 19 east), France 3 (FTA at 5 west) 15:15-17:00 CET, Rai Tre (FTA 13e) 15:30-18:10 CET. If you know of other broadcasters showing the race, please leave me a comment. RAI Sport Sat (FTA 13e) repeat Paris-Tours tomorrow at 16:00 CET and show highlghts of the Giro dell’Emilia today at 13:40-14:20, and Gran Premio Beghelli 19:30-20:00.

Christian Perrier: RWC: why did France lose...

...at least, in my opinion (which is obviously biased, let's admit that). There are several reasons for this loss: Some *bad* reasons for the English victory:

13 October 2007

Adam Rosi-Kessel: Opened Pandora

Via Eric Goldman’s recent recommendation, I decided to give Pandora another shot. The short version: Pandora is an intelligent predictive personalized Internet radio service with an arguably sustainable and protectable business model. And by “intelligent,” I mean there are real human brains at work. As Eric explains:
Pandora’s main competitive differentiator is its “Music Genome Project.” 50 trained musicians with at least a college degree in music (called “music analysts”) listen to songs all day long and rate each song on 400 different musical attributes. See the 2005 WSJ article discussing them. By profiling songs this way, the system can predict that a person who likes an artist’s song might like other songs with similar musical attributes. From listening to Pandora for many, many hours, IMO the system isn’t perfect, but it does a pretty good job, and it has definitely hooked me on music that I wouldn’t have listened to otherwise.
They have apparently cataloged approximately half a million songs and the database continues to grow apace. There is also a collaborative-filtering aspect, similar to Netflix and Amazon. I suspect this hybrid between the “wisdom of crowds” and the “wisdom of experts” will be the future of most large content projects (including wikipedia). My first few hours have returned excellent results. I created a for-working “station” called, lazily, “The Bad Plus Radio” (described as “Avant garde and angular funk/jazz, but not so dissonant that you can’t do mind-taxing work while listening”). You are welcome to listen as well. (See also my Pandora Profile.) The “artist seeds” for the channel include the following: Pandora has played several tracks by these artists, but is increasingly mixing in other artists that match up on some axis of preference. I’ve thumbed-up and thumbed-down several tracks from beyond the “seed” set (and will continue to do so), thus driving the predictive engine. I look forward to creating some entirely different channels and publishing the URLs here. Although supposedly the “free” version is ad-supported, I haven’t heard any ads yet. Maybe that is still to come. In any case, it is probably worth the $36/year subscription cost. Beyond the specific content I’m enjoying here, it is nice to see a Web 2.0 (or pick your favorite version) business model that doesn’t require a leap of faith to see how it can work. Of course, they have a Facebook app as well.

Julien Valroff: ndiswrapper and WPA

I had been using ndiswrapper with a Netgear WG511 PCMCIA card[1] without any problems for a while, then have begun to get kernel crashes, which were more and more frequent. I have thus backported to Etch the 1.47 version available in Sid, which *does* fix my kernel crashes. However, this newer version has brought new problems: with a “high” network load, the module just crashes silently, and has to be reloaded so that the network connection can be enabled again. I have packaged ndiswrapper 1.48 as the release notes let me think my issues were fixed (”Disassocation with wpa_supplicant is fixed”), but the problem is still there with this latest release! I have finally managed to get things work correctly thanks to this post: I use Debian linux-image, ie. without preemption enabled, and I can now confirm the still-to-be-released 1.49 version fixes this issue. I have uploaded 1.49-rc3 packages to my unofficial repository just in case someone has the same issue.
Linknotes:
  1. the free prism54 unfortunately does not support WPA ↩

Biella Coleman: Bad Apples .

I am not even sure what to make of this article, written by a professor at a well-known university, on the perils of advising graduate students. Given that I have been sitting at my computer for nearly 12 hours straight and since 4 a.m, I don’t think I am really in a state of mind to think too much of it. Neither do I have much experience doing this (though whatever experience I have had, it has been good so far), so I can’t really comment back via personal experience. I only hope that her/his experience was the exception or that he/she allowed too many boundaries to be crossed, but whatever the case, interesting food for thought. I am sure a graduate student could write an interesting counterpoint, talking about all the questionable things their professors did back to them too and knowing the web, I am sure that will be forthcoming soon.

Daniel Burrows: Monad.Reader

The Reader monad is a simple and easy-to-use structure from the Haskell library that solves a common problem.

Very often, it's the case that you have a lot of functions that all need to use the same values. For instance, you might be laying out some text on a page, and you need to keep the details of the page size around in all your layout algorithms. Typically, you handle this situation by adding a new parameter to all your functions which represents the state. So,


layoutPage :: Text -> Page


becomes


layoutPage :: PageLayoutParams -> Text -> Page


This works just fine, but it can get really ugly. Every single function gets a new parameter, which must be threaded through to every related function that it ever calls. This causes visual clutter, particularly for functions that don't actually use the parameter themselves but just pass it on.

A function written in the Reader monad is augmented with a hidden parameter. For the hypothetical example above, we would write:


import Control.Monad.Reader
type LayoutFunc = Reader PageLayoutParams

layoutPage :: Text -> LayoutFunc Page


Note that the Reader monad is partially applied here. The result, LayoutFunc, is a new monad that adds a parameter of type PageLayoutParams to any code that runs inside it.

So, now that we have this monad, how do we use it? The "ask" operation retrieves the parameter that's hidden in the monad:


ask :: Reader a a


or, specialized to LayoutFunc:


ask :: LayoutFunc PageLayoutParams


In order to remain close to the example in question, I will provide specialized type signatures for the rest of the library functions. You can find the generic type signatures in the Haskell library documentation for Control.Monad.Reader.

Suppose that layoutPage started out like this before it was moved into the monad:


layoutPage st txt =
let width = pageWidth st
...


In the Reader monad, this becomes:


layoutPage txt =
do st - ask
let width = pageWidth st
...


As you can see, we've gone from one line of code to two. As this is clearly unacceptable for a clutter-reducing device, the Reader module provides a convenience function encapsulating the above pattern:


asks :: (PageLayoutParams -> a) -> PageLayoutFunc a


This allows us to simplify the monadic implementation to:


layoutPage txt =
do width - asks pageWidth
...


It's quite common, when multiple functions share context information, to invoke a function in an altered context. For instance, we might have a layoutSubPage routine that changes the page width:


layoutSubPage st newWidth txt =
doLayout (st pageWidth = newWidth txt)


The Reader equivalent of this idiom is the function "local":


local :: (PageLayoutParams -> PageLayoutParams) -> LayoutFunc a -> LayoutFunc a


local executes an operation in a context modified by its first parameter. For instance,


layoutSubPage newWidth txt =
local (\st -> st pageWidth = newWidth ) txt


Finally, to actually invoke a computation in the Reader monad, use runReader:


runReader :: LayoutFunc a -> PageLayoutParams -> a

main = showPage (runReader defaultPageParams someText)


For much more information on the Reader monad, see All About Monads, or the Haskell library documentation.

Biella Coleman: On Networks and Experimental Writing

One of my dissertation thesis advisors, Christopher Kelty, is teaching a superb looking course at Harvard this fall on networks. The only thing I would add to that syllabus right now is a book by a department colleague, Alexander Galloway, who just published a book with Eugene Thacker The Exploit. And while I have not read more than a chapter, what I like about it is its experimental style. They open this book with the following orientation:
It is our intention in this book to avoid the limits of academic writing in favor of a more experimental, speculative approach. To that end, we adopt a two-tier format. Throughout Part I, “Nodes,” you will find a number of condensed, italicized header that are glued together with more standard prose. For quick immersion, we suggest skikking Part I by reading the italicized sections only…. In this sense, we hope you will experience the book not as the step-by-step propositional evolution of a complete theory but as a series of marginal claims, disconnected in a living environment of many thoughts, distributed across as many pages.
The good thing is while the form is experimental, at the sentence level, things are quite clear. I have often had the experience of reading experimental work whose content was the experiment but not the form, and basically I did not understand a thing. In this case, it is the form that achieves their desire to explore and present their marginal claims. Annemarie Mol in The Body Multiple also uses a two-tiered experimental approach that is just fantastic, especially since her writing is especially accessible.

Clint Adams: One of us does not get it

Rapha l, I'm not sure you understood. The solution is to get rid of all of DSA (and maybe anyone that supports them). Removing one side of a power struggle may win you some short term gains, but may be a no-op or even more harmful in the long run. Such a solution is indicative of what I would call an overly-practical mindset, the kind I'd expect from Project Scud. You can say that DSA's inadequacies and ineptitude are due to specific individuals, but it is a team: each member is free to quit at any time (unless there is some kind of sinister blackmail or other means of coercion in progress, in which case I'm sure we would all appreciate if you exposed it), and each member is responsible for the performance of the team. If you accept that each team member is responsible, and you acknowledge that DSA's behavior is unacceptable, then how can you imagine that adding or subtracting members piecemeal will effect anything substantial?

12 October 2007

Eddy Petri&#537;or: SELinux: unusable from a newbie perspective

Thanks Russell for the explanations on the execmem bits.

Now I am trying to go further and set up my system to really work with SELinux enabled because, although the promise of the targeted policy is to allow you to do your job mostly as you did before, that "mostly" has a really wide meaning, more than you'd think you bargained for.

Examples from my laptop: hald does not start by default (various denials), resolvconf is denied some getattr operations on tmpfs, hald-addon-dell-backlight is denied access to some character device I just know it should have access to, etc. Of course, this means that automatic mounting does not work anymore and there is a decrease in usability just because of the "mostly" part.

I am sure that some of the denials are correct (see the execstack stuff or the memexec), but there are cases where this "mostly" is stretched way too much. IMHO, desktop installs should suffer no restrictions when using the targeted policy (and I mean "no restrictions that would make my system less than it was before enabling and enforcing SELinux").


But let me tell a reason why SELinux sucks without any help from others:

The interface sucks big time.

Probably there is are good reasons, but if you need to create and use a new policy based on the denials found in the logs, you need to use no less than 4 (four) different tools with the right incantation (although you can use just 2, if you don't want to customize the rules[1]):
In the end, you still hit the bad interface:

bounty:~# semodule -i local.pp
libsepol.check_assertion_helper: assertion on line 0 violated by allow hald_t memory_device_t:chr_file read ;
libsepol.check_assertions: 1 assertion violations occured
libsemanage.semanage_expand_sandbox: Expand module failed
semodule: Failed!

Now what? That error message doesn't tell me much, except that some assertion failed. But where does the assertion come from? Why is it an assertion? What is bad about that line?
To me that really looks like an internal error or an inconsistency in what the SELinux tools generate.


[1] still, why advertise the longer path as the Fedora FAQ does? I would have done it the other way around.

Rapha&#235;l Hertzog: The DSA dilemma

For once, Clint blogged on something that I can understand. :-) I don’t buy everything he says, but in the case of DSA, the part where he says “you cannot have a functional and respectable subgroup if it maintains autonomy like that” is a real problem. The leadership problem I mentioned is real. And it can theoretically be solved by undelegating one of the problematic side of this DSA-internal dispute. But which one? Given the unwillingness of Joey to discuss the problems, he makes an easy target… which would leave DSA up to Ryan, James and Phil. But is that a desirable thing? If DSA is perceived as being an “autonomous” group which is not involved in Debian’s main discussions and which is somewhat disconnected from Debian’s day-to-day life, it’s largely due to the behavior of James and Ryan. E-mail communication with them is very difficult as they’ll respond only if they really care about something. And despite the setup of the request tracker, they have barely been able to make proper usage of it… the idea was to use RT tickets to track everything that DSA does but they don’t use it as such. For example, James setup a “wikiadm” group and he never reported anything to the related ticket (#194) (I did it myself once I found out). Also there’s an internal ticket about the replacement of ftp.debian.org (that I created because ftp.d.o ran out of space regularly) and AFAIK Jeroen has been in touch with James to setup that replacement, but nothing got reported to the tracker. Ryan promised me once to put his DSA TODO list in the tracker so that other people can jump in and help out. He never did. So while Joey is definitely a pain for DSA, at least he’s a visible participant of the team and he interacts with the community. James and Ryan are not, they interact only through private channels and do not share their opinions or their vision of Debian.
I believe this is a real problem. On the other hand, most of the interesting changes in the last months are the results of James’s work. But he’s also implicitly blocking addition of new members as long as the leadership problem is not solved. I tried to fill the communication void of the DSA team by various means. I follow everything as closely as I can so that I can report changes on other channels, mailing lists when needed. I made efforts to document stuff on the wiki page, etc. But this is not a long term solution, the communication issue must be fixed within the team. The path ouf of this mess is still not very clear, but something is going to change soon. Not quite sure what though. What would you suggest? And if you were DPL, what would you do? Since private discussions and negotiations lead nowhere, it’s tempting to bring the issue in the public area. In theory, they have no way to escape discussions and they’ll have to communicate their grudges against the other side if they want to have some fair judgment between both parties. Unfortunately, given the habits of James and Ryan, they probably won’t participate in any public discussion and either resign or stay where they are waiting for any decision… Comments welcome. Partagez cet article / Share This

11 October 2007

Russell Coker: Getting People into IT

Pia writes about the difficulty in getting young women and young people in general into the computer industry [1]. While I agree that having more women in the computer industry would be a good thing, I have difficulty believing some of the claims that Pia makes. For example the claim that “[girls] are more career focused earlier in their school life“. I chose my career when I was about 11 years old [2] and several of my friends made similar decisions at similar ages. I would be interested to read anecdotal evidence from women in the computer industry about how old they were when they decided on their career and if their friends did the same, a reference to any research on this topic would also be useful. I tend to believe that boys are more career focussed at all stages of their life but have little evidence to support this idea. One fact that seems obvious is that the idea that “if you don’t succeed in your career then you can always marry someone who does” is almost non-existent among boys. It seems likely that such ideas have a statistically relevant affect on the focus on career of boys vs girls. Also the Australian Bureau of Statistics reports that the MEDIAN income for women is significantly lower than for men [4], I find it difficult to imagine that girls could be more career focussed from a younger age and yet get significantly lower pay (note the fact that it’s median not mean income is very significant as it removes the “glass ceiling” effect). Phillip Greenspun writes about why there are so few women doing scientific research [6] and makes some good points about why scientific research is generally not well paid and therefore a university student would choose a career in some other area and suggests that it’s a macho thing that guys enter such competitive fields for relatively low wages. Maybe some women correctly assess the costs and benefits of a career in scientific research and then make the mistake of equating Computer Science to other branches of science. But the median income suggests that although there may be some valid reasons for avoiding science that would only cover a small portion of the problem (the difference in median income can not be explained by misplaced attempts to maximise income). One problem that is significant is the quality of school education for girls. Not only is there the issue that boys may crowd-out girls for some subjects that are supposedly traditionally for boys (such as all science) but even girls schools aren’t as good as they should be. Some time ago I was talking to a teacher at an all-girls school, the school was moderately expensive and parents were paying the extra money presumably to give their daughters educational opportunities that they might miss in a co-ed school. However the school did not teach hard maths (”Maths B” was the official name at the time) and only taught the easy maths (official name “Maths A” and unofficial name “Vegie Maths“) because they didn’t have many girls demanding it (which is probably difficult to measure if you don’t offer it as a reasonable option) and the girls who wanted to study it could always move to a different school. So the choice facing girls at the exclusive school in question was “skip the subject that is most useful for further studies in most science subjects” or “go to a different school and miss most of your friends“, this sort of decision would surely discourage some potential female computer programemers. Also I think that the difference between boys and girls in regard to studying computer science has a lot to do with the fact that given a choice between missing most of their friends and missing out on something related to computers would be a no-brainer for most boys. Paul Graham’s article about Nerds has some interesting points to make in this regard - maybe the problem is that girls aren’t Nerdish enough [5]. Pia also writes about parents and teachers advising children not do study IT because of a perceived lack of jobs. I think that the problem here is not just bad advice, but also a bad tendency to take advice. Someone who wants to study in an established field which changes little over time (law and accounting spring to mind) probably should take careful note of the advice that they are given - things haven’t changed much in the last few decades. But someone who wants to study in a field that changes rapidly and where every year has new and significant developments (of which the best example is the computer industry) should probably be quite skeptical of all advice - most advice about the computer industry concerns how things used to be not how things are. Finally when considering whether to accept advice you should consider who is offering it. For example advice from a hiring manager should be carefully noted (as the manager will tell you precisely what factors influence their own decisions on hiring). Advice from people who are successful in the industry should also be noted. Advice from a school career advisor who gets paid about 1/3 what any 25yo can earn in the computer industry should be entirely ignored. I wonder whether being hesitant to ignore advice is a problem for girls in this regard. When I was in year 11 I had to take a subject related to career planning. It covered some things that were of minor use (such as writing CVs) and had an assignment of writing a fictional CV for yourself a few years after leaving university. I received bad marks for preparing a CV that involved changing jobs as companies went bankrupt or projects failed due to bad management, I was told that if your employer fails in the market it makes you look bad! However my fictional CV did bear some resemblance to what really happened… In terms of what industries have jobs available, the best advice I can give students is to actively do some research of their own. It’s not difficult to get the jobs sections of some newspapers and do a quick scan to see how many positions are open in a field, and it’s even easier to do some searches on online jobs sites (which usually tell you how many positions posted in the last X days match your criteria). For example I just visited jobserve.com.au and found 1724 Engineering jobs and 5622 IT jobs advertised. If you compare this to the university intake (I visited the Swinburne university courses list [3] and found 25 IT courses vs 29 Engineering course) it seems that the ratio of Engineering graduates to jobs is not likely to be as good as that for CS graduates. Of course it may be that all the other universities have hardly any Engineering courses and balance the ratio out (but I doubt it). In any case this would be a good way of injecting some facts into a discussion of the relative merits of different career choices and avoiding it being an issue of parents/teachers not liking computers vs children liking them. Determining the relative pay rates of different industries is a lot more difficult (and requires a significant amount of work), some recruiting agencies publish statistics - but those stats only apply to the positions that they fill (which is a sub-set of the actual positions). Finally as a piece of advice for children, try and find a job that you enjoy. If you earn $30K doing something you enjoy then you’ll probably be happier than if you earn $100K doing something you hate. Also if you enjoy your work then you will probably be able to take the extra steps needed to become successful - often it’s not a choice between having fun or making good money but a choice between fun and good money or the absence of both. If someone tells you to avoid doing what you love and instead do something boring for some unsubstantiated belief that there would be more money in it then be a nerd and tell them that their opinion is not relevant (it does tend to make teachers angry though).

Matthew Palmer: EBay Sez: Linux is for scammers

In what I can only assume is a case of "graphic design spec gone horribly wrong", EBay has decided to portray our lovable penguin friend as the mascot of the online scam artist: Linux is for scammers, says EBay (Click on the image for a full-size version) Personally, I think the scam artists' best friend is a Windows machine, as it provides a rich source of untraceable connectivity, but my guess is that Microsoft has scarier lawyers. I don't know how long the image has been up there for, but I'd imagine it won't last long. Linux users may not be that numerous, but we're a noisy lot. <grin>

Andrew Pollock: [life] A c-section from a bloke's perspective

I've been meaning to write this for weeks. Hopefully I can still remember everything. So Sarah had a c-section the day before we were scheduled to have our c-section class. We'd both read a bit though, so it wasn't like we were going in totally uninformed and unprepared.. Firstly, Stanford Hospital's operating room area is rather mind-blowing. Sarah was transferred from Lucille Packard Children's Hospital to Stanford because they could get an operating room there sooner. The transfer was trivial, just roll the whole bed down the corridor, through the magic double doors into the operating rooms. The doors in question have a whole heap of video cameras pointing straight down, so they see everyone that gets wheeled in and out, which I found interesting. Good for records of patient movements I guess. Anyway, through those doors is a maze of corridors. There was a light-up board hanging from the ceiling in the middle of the main corridor with lights for all of the ORs. I think there were about 20. The illuminated ones were available. We went down a corridor, and past a few ORs. Outside of each one is a big stainless-steel scrub area. We got to "our" OR, and they transferred Sarah onto the operating table. Because of her heart condition, they wanted to have an arterial line in her for the duration of the procedure. This is some sort of gadget that goes into an artery and monitors your blood pressure beat by beat. The "interesting" thing with the anaesthesiologist was the New Zealand woman that we'd seen when we had an anaesthesiologist months prior, but the appointment was with a different anaesthesiologist, and this New Zealand woman was "observing". At the time, the anaesthesiologist we had the appointment with told us that she was going to be on vacation at the time of the delivery, so she wouldn't be attending. So I don't know if the Kiwi woman got promoted out of "observing" to "doing" in that period, or what, but she reintroduced herself to us in LPCH prior to moving to Stanford as being our anaesthesiologist for the delivery. Anyway, in the OR, they handed her the arterial line, in it's plastic packaging, and the first thing she asked was whether it came with anything else to put it in. At this point I figured she hadn't put one of these in before. (The answer to her question, by the way, was "no") She then proceeded to have a terrible time getting the line into Sarah's arm. Normally intravenous lines go into a vein, but this has to go into an artery. Apparently when you go poking an artery, it spasms. It then becomes very hard to get it into a moving target. She had a few goes on one arm, then gave up, and had to move over to the other arm. Sarah doesn't like needles at the best of times, and getting this honking great big thing in was no picnic. The bruising on both her wrists afterwards was phenomenal. She tends to bruise fairly spectacularly whenever they stick a needle in her. I almost stepped in and asked if it was really necessary that she have it, because they were having so much trouble getting it in, and it was clearly hurting Sarah. I think someone else ended up getting it in, not the anaesthesiologist. Not "that" one anyway. There were heaps of people in the OR. Next, came the epidural. Again, we had the inexperienced anaesthesiologist. This time she said to someone else how she was so nervous because there were "too many eyes". Not really reassuring. They gave Sarah a numbing injection first, and then proceeded to poke her in back with the epidural catheter needle thing. Sarah was flinching when they did it, and they kept yelling at her not to move. Apparently it didn't hurt, but it caused discomfort every time they tried to get it in. Eventually the associate professor of anaesthesiology stepped in and got it in. Later, they said that Sarah had a slightly curved spine or something, and that had made it a bit more complicated. I'm not begrudging the obviously inexperienced anaesthesiologist for anything. Stanford is a teaching hospital, and doctors have to get their experience somehow, but I think that when the patient is conscious, it's important to come across a little more confident. So once they got all of that sorted out, they could get her lying down on the table and get down to business. Apparently one of the side-effects of the epidural is that it can cause your body to think it's got hypothermia. Part way through the procedure, Sarah started shaking uncontrollably. I can't remember whether she complained about feeling cold or not. She also got a really itchy face. Apparently that's a side-effect of morphine. Sarah also doesn't tolerate anaesthetics very well, and started feeling nauseous part way through the procedure. Sure enough, as has happened countless times previously post-operatively, she threw up. At least we had warning and I got a kidney dish in place in time. That was about all the eventful surgical stuff. They had the usual screen up from about her neck area, so I couldn't see what they were doing. After Joshua was delivered, one of the nurses asked me if I wanted to see him, and took me around from her left shoulder where I'd been sitting to around below her right foot, to where they had him in one of those perspex portable crib things. I could see a lot of red out of the corner of my eye, and I could have seen a lot more if I'd bothered to look, but I was more interested in seeing my son at that stage. They let me bring him back around to where I'd been sitting previously, to show Sarah, but it was hard getting him up to a point where she could see him properly, and she was partially out of it by this stage. I think she dozed off for a little bit not long after that. I forget what time we got into the OR, but they started the operation in earnest at about 10:00am, and we were out of there by 10:45am if memory serves. The nurse in charge of the recovery room really didn't want me there, so I went back to LPCH with one of the nurses to spend some time with him while he was weighed and measured, and I took some photos, and held him. This was good, because I got to have a good cry on my own with him. So overall, in my "I'll never be squeezing a watermelon out of my nether regions" blokey perspective, having seen what Sarah went through, and how much discomfort that it looked like she was going through, both in the OR and afterwards, I'm not convinced c-sections are any less painful than a vaginal birth (taking the average case of both). I think you're trading some "short term" (depends on how long your labour is I guess) pain for some "long term" pain while you recover from your incision. Granted, Sarah wasn't really laid up all that badly for all that long, but it still didn't look all that pleasant. I guess really only the women who have done both delivery methods can really comment. All I know is I found it very difficult watching Sarah go through so much pain. So much so that I'm not really looking forward to that again. Maybe, hopefully, after her aorta is repaired, they'll let her delivery vaginally next time. I'm not holding much hope though. The high-risk obstetricians are fairly paranoid. There's also the possibility the baby will make the decision very easy for everyone by being breech. Sarah raised an interesting point today. Why didn't they just knock her out with a general anaesthetic? There was no need for her to be conscious for the delivery. That would have spared her the pain and suffering of getting the arterial line in, and they wouldn't have needed to bother with an epidural. One for her to ask the obstetrician at her follow-up appointment.

10 October 2007

Martin F. Krafft: Packaging with Git

Introduction I gave a joint presentation with Manoj at Debconf7 about using distributed version control for Debian packaging, and I volunteered to do an on-line workshop about using Git for the task, so it's about time that I should know how to use Git for Debian packaging, but it turns out that I don't. Or well, didn't. After I made a pretty good mess out of the mdadm packaging repository (which is not a big problem as it's just ugly history up to the point when I start to get it right), I decided to get down with the topic and figure it out once and for all. I am writing this post as I put the pieces together. It's been cooking for a week, simply so I could gather enough feedback. I am aware that Git is not exactly a showcase of usability, so I took some extra care to not add to the confusion. It may be the first post in a series, because this time, I am just covering the case of mdadm, for which upstream also uses Git and where I am the only maintainer, and I shall pretend that I am importing mdadm to version control for the first time, so there won't be any history juggling. Future posts could well include tracking Subversion repositories with git-svn, and importing packages previously tracked therewith, but this ain't no promise! (well, that last post is already being drafted, but far from finished; you have been warned!) I realise that git-buildpackage exists, but imposes a rather strict branch layout and tagging scheme, which I don't want to adhere to. And gitpkg (Romain blogged about it recently), deserves another look since, according to its author, it does not impose anything on its user. But in any case, before using such tools (and possibly extending them to allow for other layouts), I'd really rather have done it by hand a couple of times to get the hang of it and find out where the culprits lie. Now, enough of the talking, just one last thing: I expect this blog post to change quite a bit as I get feedback. Changes shall be highlighted in bold typeface.
Setting up the infrastructure First, we prepare a shared repository on git.debian.org for later use (using collab-maint for illustration purposes), download the Debian source package we want to import (version 2.6.3+200709292116+4450e59-3 at time of writing, but I pretend it's -2 because we shall create -3 further down ), set up a local repository, and link it to the remote repository. Note that there are other ways to set up the infrastructure, but this happens to be the one I prefer, even though it's slightly more complicated:
$ ssh alioth
$ cd /git/collab-maint
$ ./setup-repository pkg-mdadm mdadm Debian packaging
$ exit
$ apt-get source --download-only mdadm
$ mkdir mdadm && cd mdadm
$ git init
$ git remote add origin ssh://git.debian.org/git/collab-maint/pkg-mdadm
$ git config branch.master.merge refs/heads/master
Now we can use git-pull and git-push, except the remote repository is empty and we can't pull from there yet. We'll save that for later. Instead, we tell the repository about upstream's Git repository. I am giving you the git.debian.org URL though, simply because I don't want upstream repository (which lives on an ADSL line) hammered in response to this blog post:
$ git remote add upstream-repo git://git.debian.org/git/pkg-mdadm/mdadm
Since we're using the upstream branch of the pkg-mdadm repository as source (and don't want all the other mess I created in that repository), we'll first limit the set of branches to be fetched (I could have used the -t option in the above git-remote command, but I prefer to make it explicit that we're doing things slightly differently to protect upstream's ADSL line).
$ git config remote.upstream-repo.fetch \
    +refs/heads/upstream:refs/remotes/upstream-repo/upstream
And now we can pull down upstream's history and create a local branch off it. The "no common commits" warning can be safely ignored since we don't have any commits at all at that point (so there can't be any in common between the local and remote repository), but we know what we're doing, even to the point that we can forcefully give birth to a branch, which is because we do not have a HEAD commit yet (our repository is still empty):
$ git fetch upstream-repo
warning: no common commits
[ ]
  # in the real world, we'd be branching off upstream-repo/master
$ git checkout -b upstream upstream-repo/upstream
warning: You appear to be on a branch yet to be born.
warning: Forcing checkout of upstream-repo/upstream.
Branch upstream set up to track remote branch
  refs/remotes/upstream-repo/upstream.
$ git branch
* upstream
$ ls   wc -l
77
Importing the Debian package Now it's time to import Debian's diff.gz remember how I pretend to use version control for package maintenance for the first time. Oh, and sorry about the messy file names, but I decided it's best to stick with real data in case you are playing along: Since we're applying the diff against version 2.6.3+200709292116+4450e59, we ought to make sure to have the repository at the same state. Upstream never "released" that version, but I encoded the commit ID of the tip when I snapshotted it: 4450e59, so we branch off there. Since we are actually tracking the git.debian.org pkg-mdadm repository instead of upstream, you can use the tag I made. Otherwise you could consider tagging yourself:
$ #git tag -s mdadm-2.6.3+200709292116+4450e59 4450e59
$ git checkout -b master mdadm-2.6.3+200709292116+4450e59
$ zcat ../mdadm_2.6.3+200709292116+4450e59-2.diff.gz   git apply
The local tree is now "debianised", but Git does not know about the new and changed files, which you can verify with git-status. We will split the changes made by Debian's diff.gz across several branches.
The idea of feature branches We could just create a debian branch, commit all changes made by the diff.gz there, and be done with it. However, we might want to keep certain aspects of Debianisation separate, and the way to do that is with feature branches (also known as "topic" branches). For the sake of this demonstration, let's create the following four branches in addition to the master branch, which holds the standard Debian files, such as debian/changelog, debian/control, and debian/rules:
  • upstream-patches will includes patches against the upstream code, which I submit for upstream inclusion.
  • deb/conffile-location makes /etc/mdadm/mdadm.conf the default over /etc/mdadm.conf and is Debian-specific (thus the deb/ prefix).
  • deb/initramfs includes the initramfs hook and script, which I want to treat separately but not submit upstream.
  • deb/docs similarly includes Debian-only documentation I add to the package as a service to Debian users.
If you're importing a Debian package using dpatch, you might want to convert every dpatch into a single branch, or at least collect logical units into separate branches. Up to you. For now, our simple example suffices. Keep in mind that it's easy to merge two branch and less trivial to split one into two. Why? Well, good question. As you will see further down, the separation between master and deb/initramfs actually makes things more complicated when you are working on an issue spanning across both. However, feature branches also bring a whole lot of flexibility. For instance, with the above separation, I could easily create mdadm packages without initramfs integration (see #434934), a disk-space-conscious distribution like grml might prefer to leave out the extra documentation, and maybe another derivative doesn't like the fact that the configuration file is in a different place from upstream. With feature branches, all these issues could be easily addressed by leaving out unwanted branches from the merge into the integration/build branch (see further down). Whether you use feature branches, and how many, or whether you'd like to only separate upstream and Debian stuff is entirely up to you. For the purpose of demonstration, I'll go the more complicated way.
Setting up feature branches So let's commit the individual files to the branches. The output of the git-checkout command shows modified files that have not been committed yet (which I trim after the first example); Git keeps these across checkouts/branch changes. Note that the ./debian/ directory does not show up as Git does not know about it yet (git-status will tell you that it's untracked, or rather: contains untracked files since Git does not track directories at all):
$ git checkout -b upstream-patches mdadm-2.6.3+200709292116+4450e59
M Makefile
M ReadMe.c
M mdadm.8
M mdadm.conf.5
M mdassemble.8
M super1.c
$ git add super1.c     #444682
$ git commit -s
  # i now branch off master, but that's the same as 4450e59 actually
  # i just do it so i can make this point 
$ git checkout -b deb/conffile-location master
$ git add Makefile ReadMe.c mdadm.8 mdadm.conf.5 mdassemble.8
$ git commit -s
$ git checkout -b deb/initramfs master
$ git add debian/initramfs/*
$ git commit -s
$ git checkout -b deb/docs master
$ git add RAID5_versus_RAID10.txt md.txt rootraiddoc.97.html
$ git commit -s
  # and finally, the ./debian/ directory:
$ git checkout master
$ chmod +x debian/rules
$ git add debian
$ git commit -s
$ git branch
  deb/conffile-location
  deb/docs
* master
  upstream
  upstream-patches
At this time, we push our work so it won't get lost if, at this moment, aliens land on the house, or any other completely plausible event of apocalypse descends upon you. We'll push our work to git.debian.org (the origin, which is the default destination and thus needs not be specified) by using git-push --all, which conveniently pushes all local branches, thus including the upstream code; you may not want to push the upstream code, but I prefer it since it makes it easier to work with the repository, and since most of the objects are needed for the other branches anyway after all, we branched off the upstream branch. Specifying --tags instead of --all pushes tags instead of heads (branches); you couldn't have guessed that! See this thread if you (rightfully) think that one should be able to do this in a single command (which is not git push refs/heads/* refs/tags/*)
$ git push --all
$ git push --tags
Done. Well, almost
Building the package (theory) Let's build the package. There seem to be two (sensible) ways we could do this, considering that we have to integrate (merge) the branches we just created, before we fire off the building scripts:
  1. by using a temporary (or "throw-away") branch off upstream, where we integrate all the branches we have just created, build the package, tag our master branch (it contains debian/changelog), and remove the temporary branch. When a new package needs to be built, we repeat the process.
  2. by using a long-living integration branch off upstream, into which we merge all our branches, tag the branch, and build the package off the tag. When a new package comes around, we re-merge our branches, tag, and build.
Both approaches have a certain appeal to me, but I settled for the second, for two reasons, the first of which leads to the second:
  1. When I upload a package to the Debian archive, I want to create a tag which captures the exact state of the tree from which the package was built, for posterity (I will return to this point later). Since the throw-away branches are not designed to persist and are not uploaded to the archive, tagging the merging commit makes no sense. Thus, the only way to properly identify a source tree across all involved branches would be to run git-tag $branch/$tagname $branch for each branch, which is purely semantic and will get messy sooner or later.
  2. As a result of the above: when Debian makes a new stable release, I would like to create a branch corresponding to the package in the stable archive at the time, for security and other proposed updates. I could rename my throw-away branch, if it still existed, or I could create a new branch and merge all other branches, using the (semantic) tags, but that seems rather unfavourable.
So instead, I use a long-living integration branch, notoriously tag the merge commits which produced the tree from which I built the package I uploaded, and when a certain version ends up in a stable Debian release, I create a maintenance branch off the one, single tag which corresponds to the very version of the package distributed as part of the Debian release. So much for the theory. Let's build, already!
Building the package (practise) So we need a long-living integration branch, and that's easier done than said:
$ git checkout -b build mdadm-2.6.3+200709292116+4450e59
Now we're ready to build, and the following procedure should really be automated. I thus write it like a script, called poor-mans-gitbuild, which takes as optional argument the name of the (upstream) tag to use, defaulting to upstream (the tip):
#!/bin/sh
set -eu
git checkout master
debver=$(dpkg-parsechangelog   sed -ne 's,Version: ,,p')
git checkout build
git merge $ 1:-upstream 
git merge upstream-patches
git merge master
for b in $(git for-each-ref --format='%(refname)' refs/heads/deb/*); do
  git merge -- $b
done
git tag -s debian/$debver
debuild   # will ignore .git automatically
git checkout master
Note how we are merging each branch in turn, instead of using the octopus merge strategy (which would create a commit with more than two parents) for reasons outlined in this post. An octopus-merge would actually work in our situation, but it will not always work, so better safe than sorry (although you could still achieve the same result). If you discover during the build that you forgot something, or the build script failed to run, just remove the tag, undo the merges, checkout the branch to which you need to commit to fix the issue, and then repeat the above build process:
$ git tag -d debian/$debver
$ git checkout build
$ git reset --hard upstream
$ git checkout master
$ editor debian/rules    # or whatever
$ git add debian/rules
$ git commit -s
$ poor-mans-gitbuild
Before you upload, it's a good idea to invoke gitk --all and verify that all goes according to plan:
screenshot of gitk after the above steps
When you're done and the package has been uploaded, push your work to git.debian.org, as before. Instead of using --all and --tags, I now specify exactly which refs to push. This is probably a good habit to get into to prevent publishing unwanted refs:
$ git push origin build tag debian/2.6.3+200709292116+4450e59-3
Now take your dog for a walk, or play outside, or do something else not involving a computer or entertainment device.
Uploading a new Debian version If you are as lucky as I am, the package you uploaded still has a bug in the upstream code and someone else fixes it before upstream releases a new version, then you might be in the position to release a new Debian version. Or maybe you just need to make some Debian-specific changes against the same upstream version. I'll let the commands speak for themselves:
$ git checkout upstream-patches
$ git-apply < patch-from-lunar.diff   #444682 again
$ git commit --author 'J r my Bobbio <lunar@debian.org>' -s
  # this should also be automated, see below
$ git checkout master
$ dch -i
$ dpkg-parsechangelog   sed -ne 's,Version: ,,p'
2.6.3+200709292116+4450e59-3
$ git commit -s debian/changelog
$ poor-mans-gitbuild
$ git push
$ git push origin tag debian/2.6.3+200709292116+4450e59-3
That first git-push may require a short explanation: without any arguments, git-push updates only the intersection of local and remote branches, so it would never push a new local branch (such as build above), but it updates all existing ones; thus, you cannot inadvertedly publish a local branch. Tags still need to be published explicitly.
Hacking on the software Imagine: on a rainy Saturday afternoon you get bored and decide to implement a better way to tell mdadm when to start which array. Since you're a genius, it'll take you only a day, but you do make mistakes here and there, so what could be better than to use version control? However, rather than having a branch that will live forever, you are just creating a local branch, which you will not publish. When you are done, you'll feed your work back into the existing branches. Git makes branching really easy and as you may have spotted, the poor-mans-gitbuild script reserves an entire branch namespace for people like you:
$ git checkout -b tmp/start-arrays-rework master
Unfortunately (or fortunately), fixing this issue will require work on two branches, since the initramfs script and hook are maintained in a separate branch. There are (again) two ways in which we can (sensibly) approach this:
  • create two separate, temporary branches, and switch between them as you work.
  • merge both into the temporary branch and later cherry-pick the commits into the appropriate branches.
I am undecided on this, but maybe the best would be a combination: merge both into a temporary branch and later cherry-pick the commits into two additional, temporary branches until you got it right, and then fast-forward the official branches to their tips:
$ git merge master deb/initramfs
$ editor debian/mdadm-raid                     #  
$ git commit -s debian/mdadm-raid
$ editor debian/initramfs/script.local-top     #  
$ git commit -s debian/initramfs/script.local-top
[many hours of iteration pass ]
[  until you are done]
$ git checkout -b tmp/start-arrays-rework-init master
  # for each commit $c in tmp/start-arrays-rework
  # applicable to the master branch:
$ git cherry-pick $c
$ git checkout -b tmp/start-arrays-rework-initramfs deb/initramfs
  # for each commit $c in tmp/start-arrays-rework
  # applicable to the deb/initramfs branch:
$ git cherry-pick $c
This is assuming that all your commits are logical units. If you find several commits which would better be bundled together into a single commit, this is the time to do it:
$ git cherry-pick --no-commit <commit7>
$ git cherry-pick --no-commit <commit4>
$ git cherry-pick --no-commit <commit5>
$ git commit -s
Before we now merge this into the official branches, let me briefly intervene and introduce the concept of a fast-forward. Git will "fast-forward" a branch to a new tip if it decides that no merge is needed. In the above example, we branched a temporary branch (T) off the tip of an official branch (O) and then worked on the temporary one. If we now merge the temporary one into the official one, Git determines that it can actually squash the ancestry into a single line and push the official branch tip to the same ref as the temporary branch tip. In cheap (poor man's), ASCII notation:
- - - O             >> merge T >>     - - - = - - OT
         - - T      >>  into O >>
This works because no new commits have been made on top of O (if there would be any, we might be able to rebase, but let's not go there quite yet; rebasing is how you shoot yourself in the foot with Git). Thus we can simply do the following:
$ git checkout deb/initramfs
$ git merge tmp/start-arrays-rework-initramfs
$ git checkout master
$ git merge tmp/start-arrays-rework-init
and test/build/push the result. Or well, since you are not an mdadm maintainer (We^W I have open job positions! Applications welcome!), you'll want to submit your work as patches via email:
$ git format-patch -s -M origin/master
This will create a number of files in the current directory, one corresponding for each commit you made since origin/master. Assuming each commit is a logical unit, you can now submit these to an email address. The --compose option lets you write an introductory message, which is optional:
$ git send-email --compose --to your@email.address <file1> <file2> < >
Once you've verified that everything is alright, swap your email address for the bug number (or the pkg-mdadm-devel list address). Thanks (in advance) for your contribution! Of course, you may also be working on a feature that you want to go upstream, in which case you'd probably branch off upstream-patches (if it depends on a patch not yet in upstream's repository), or upstream (if it does not):
$ git checkout -b tmp/cool-feature upstream
[ ]
when a new upstream version comes around After a while, upstream may have integrated your patches, in addition to various other changes, to give birth to mdadm-2.6.4. We thus first fetch all the new refs and merge them into our upstream branch:
$ git fetch upstream-repo
$ git checkout upstream
$ git merge upstream-repo/master
we could just as well have executed git-pull, which with the default configuration would have done the same; however, I prefer to separate the process into fetching and merging. Now comes the point when many Git people think about rebasing. And in fact, rebasing is exactly what you should be doing, iff you're still working on an unpublished branch, such as the previous tmp/cool-feature off upstream. By rebasing your branch onto the updated upstream branch, you are making sure that your patch will apply cleanly when upstream tries it, because potential merge conflicts would be handled by you as part of the rebase, rather than by upstream:
$ git checkout tmp/cool-feature
$ git rebase upstream
What rebasing does is quite simple actually: it takes every commit you made since you branched off the parent branch and records the diff and commit message. Then, for each diff/commit_message pair, it creates a new commit on top of the new parent branch tip, thus rewrites history, and orphans all your original commits. Thus, you should only do this if your branch has never been published or else you would leave people who cloned from your published branch with orphans.
If this still does not make sense, try it out: create a (source) repository, make a commit (with a meaningful commit message), branch B off the tip, make a commit on top of B (with a meaningful message), clone that repository and return to the source repository. There, checkout the master, make a commit (with a ), checkout B, rebase it onto the tip of master, make a commit (with a ), and now git-pull from the clone; use gitk to figure out what's going on.
So you should almost never rebase a published branch, and since all your branches outside of the tmp/* namespace are published on git.debian.org, you should not rebase those. But then again, Pierre actually rebases a published branch in his workflow, and he does so with reason: his patches branch is just a collection of branches to go upstream, from which upstream cherry-picks or which upstream merges, but which no one tracks (or should be tracking). But we can't (or at least will not at this point) do this for our feature branches (though we could treat upstream-patches that way), so we have to merge. At first, it suffices to merge the new upstream into the long-living build branch, and to call poor-mans-gitbuild, but if you run into merge conflicts or find that upstream's changes affect the functionality contained in your feature branches, you need to actually fix those. For instance, let's say that upstream started providing md.txt (which I previously provided in the deb/docs branch), then I need to fix that branch:
$ git checkout deb/docs
$ git rm md.txt
$ git commit -s
That was easy, since I could evade the conflict. But what if upstream made a change to Makefile, which got in the way with my configuration file location change? Then I'd have to merge upstream into deb/conffile-location, resolve the conflicts, and commit the change:
$ git checkout deb/conffile-location
$ git merge upstream
CONFLICT!
$ git-mergetool
$ git commit -s
When all conflicts have been resolved, I can prepare a new release, as before:
$ git checkout master
$ dch -i
$ dpkg-parsechangelog   sed -ne 's,Version: ,,p'
2.6.3+200709292116+4450e59-3
# git commit -s debian/changelog
$ poor-mans-gitbuild
# git push
$ git push origin tag debian/2.6.3+200709292116+4450e59-3
Note that Git often appears smart about commits that percolated upstream: since upstream included the two commits in upstream-patches in his 2.6.4 release, my upstream-patches branch got effectively annihilated, and Git was smart enough to figure that out without a conflict. But before you rejoice, let it be told that this does not always work.
Creating and using a maintenance branch Let's say Debian "lenny" is released with mdadm 2.7.6-1, then:
$ git checkout -b maint/lenny debian/2.7.6-1
You might do this to celebrate the release, or you may wait until the need arises. We've already left the domain of reality ("lenny" is not yet released), so the following is just theory. Now, assume that a security bug is found in mdadm 2.7.6 after "lenny" was released. Upstream is already on mdadm 2.7.8 and commits deadbeef and c0ffee fix the security issue, then you'd cherry-pick them into the maint/lenny branch:
$ git checkout upstream
$ git pull
$ git checkout maint/lenny
$ git cherry-pick deadbeef
$ git cherry-pick c0ffee
If there are no merge conflicts (which you'd resolve with git-mergetool), we can just go ahead to prepare the new package:
$ dch -i
$ dpkg-parsechangelog   sed -ne 's,Version: ,,p'
2.7.6-1lenny1
$ git commit -s debian/changelog
$ poor-mans-gitbuild
$ git push origin maint/lenny
$ git push origin tag debian/2.7.6-1lenny1
Future directions It should be trivial to create the Debian source package directly from the repository, and in fact, in response to a recent blog post of mine on the dispensability of pristine upstream tarballs, two people showed me their scripts to do it. My post also caused Joey Hess to clarify his position on pristine tarballs, before he went out to implement dpkg-source v3. This looks very promising. Yet, as Romain argues, there are benefits with simple patch management systems. Exciting times ahead! In addition to creating source packages from version control, a couple of other ideas have been around for a while:
  • create debian/changelog from commit log summaries when you merge into the build branch.
  • integrate version control with the BTS, bidirectionally:
    • given a bug report, create a temporary branch and apply any patches found in the bug report.
    • upon merging the temporary branch back into the feature branch it modifies, generate a patch, send it to the BTS and tag the bug report + pending patch.
And I am sure there are more. If you have any, I'd be interested to hear about them!
Wrapping up I hope this post was useful. Thank you for reading to the end, this was probably my longest blog post ever. I want to thank Pierre Habouzit, Johannes Schindelin, and all the others on the #git/freenode IRC channel for their tutelage. Thanks also to Manoj Srivastava, whose pioneering work on packaging with GNU arch got me started on most of the concepts I use in the above. And of course, the members of the the vcs-pkg mailing list for the various discussions on this subject, especially those who participated in the thread leading up to this post. Finally, thanks to Linus and Junio for Git and the continuously outstanding high level of support they give. If you are interested in the topic of using version control for distro packaging, I invite you to join the vcs-pkg mailing list and/or the #vcs-pkg/irc.oftc.net IRC channel. NP: Aphex Twin: Selected Ambient Works, Volume 2 (at least when I started writing )

Next.

Previous.