The Google books ngram datasets are a nice resource for playing with text generation. I've created a program that will, given a starting phrase, blather on by connecting phrases from the datasets. The results are much more varied, and seem to more often make sense than most Markov text generators I've seen before, such as dadadodo. It had better be -- it's working with data derived from 4% of all books ever written! Some examples. Do they tell a story? (The starting phrase I entered is in italics.)

• He is assailed they merge
• To be or not to be so deceived ! She denies abdominal scanning
• Beauty with enormous strides . The hospital was under the control of the communal organization of solitary mammals . In other words the mere failure of an individual .
• Time passes . It is the child's first day in town and at his own level ?
• Time above his left shirt pocket for safe keeping . At first I accepted the old man's work . He settled in that part of Paris .
• Until then no ill Accident would generate hydrogen sulfide
• Random nonsense put together huge boxes with ammunition against it itself stops . Working things out afterwards
• A change in voice and piano . He sat down at last at No . 5 in connection with another or with others ; and that by some strange disaster .
• O God ! What is the condition of morality is that they all hold hands in a chain of propositions . These propositions are the primary bearers of meaning making and the rearing of children

I have cheated slightly by picking where to stop quoting the blather each time. It tends to go on for quite a while otherwise. Although it often does manage some kind of sense at length too:

• Worst Case Scenarios in Search of Reality : The ERA and emendation . I can never recover . She became thoughtful . He looked at me gloomily above the tree tops . The women were asked how they used to behave more competitively . Forecasts should be made

---

This was accomplished using only 99 lines of code. I imported the datasets into Xapian databases using this program. And this program is the text generator. I threw out the metadata, to keep the Xapian databases of reasonable size. By which I mean, only a dozen gigabytes. It might be interesting to include the publication year data in Xapian, so that it could prefer ngrams that were published around the same time as the input text. I'm still importing data. I may put up a web interface later if there's interest. PS:

• What about Spam ? Who caused it ? You have said it so loud that the little lady in a black storm .
• What about Spam ? I think it cannot be any . The Church had the right men . They are our friends and relations who could be depended upon with regard to the UN . Some people paint the portraits of the writers .

PPS:

• Joey Hess was not the real author of these lines .

Let's say you want to clone a git repository that has the debian packaging on master and the upstream source is kept on upstream/master (since it tracks upstream's master branch). You can simply use git clone or you can do a

gbp-clone --upstream-branch=upstream/master --pristine-tar git://git.debian.org/git/pkg-lvm/multipath-tools.git

This will additionally set up master, upstream/master and pristine-tar to track the corresponding remote branches. There's no --debian-branch option in the above example since master is the default. If you want to safely update these branches at a later point invoke

gbp-pull --upstream-branch=upstream/master --pristine-tar

from inside the repository. This will only update the branches if they can be fast-forwarded. If the repository has a debian/gbp.conf describing the branch layout a call to

gbp-pull

is enough.

It seems the current trend is to use short-form dh. Some people have even thought that dh has superseded CDBS. Since I prefer CDBS for my own packaging, I will say that no, CDBS is not being deprecated and in fact has been active. Packaging with CDBS is very simple. I'll try to explain how it works, and how to package with it. This may turn into a blog post series, but I won't promise anything.

CDBS is a set of makefiles that do several tasks that are common while packaging software, so that you don't have to repeat them over and over on each package. The makefiles are classified into 2 groups: classes and rules. Classes implement the rules required for building and installing software. These are classes because they can and do inherit from others. For example, there is the makefile class, and the autotools, qmake and cmake classes inherit from it, and the gnome and kde (which is for kde 3) classes inherits from the autotools class. Rules implement several other general purpose rules that don't depend on the toolchain used to build the package. For example, there is the debhelper rules file, which takes care of creating the debian package using the usual dh_* commands. There are several rules files which do all sorts of useful stuff, from running license checks to downloading new upstream releases. If this post transforms into a series, we may look into some of them. For now, I'll demonstrate how to use CDBS to package some simple software (only the debian/rules file, of course).

The first task while running creating debian/rules is determining which build system the software uses. I will use qutecsound package as a guide. So, first things first, we start using CDBS!

#!/usr/bin/make -f



# Uncomment this to turn on verbose mode.

#export DH_VERBOSE=1



include /usr/share/cdbs/1/rules/debhelper.mk

include /usr/share/cdbs/1/class/qmake.mk

Now this would be all that is required if qutecsound where a standard qmake package where nothing needs to be customized. Alas, this is not the case. First, there are some variables that need to be passed to qmake to appropriately configure the build. Also, the projects .pro file is not standard-named. How do we fix this? The CDBS way of overriding this kind of behavior is through the use of variables. Since we are using the qmake class, we will be overriding variables named DEB_QMAKE_*:

DEB_QMAKE_CONFIG_VAL += build64

DEB_QMAKE_ARGS = qcs.pro

We add a configuration value (because CDBS is setting other config values), and we set the extra arguments to qmake.

Now, there is a problem: the resulting binary is not created with the name we want. How do we change that? We could patch the source to avoid that, but it is far easier to add a rule to the makefile to do that. We will modify the build to rename the file at the end (and then clean it up on clean because the upstream makefile will not spot it):

build/qutecsound::

	[ -f bin/qutecsound ]   mv bin/qutecsound-d bin/qutecsound



clean::

	rm -f bin/qutecsound

So what did we just do? We extended the build and clean rules with extra stuff to do. In CDBS each package listed in debian/control gets a build/package rule (and several others at different stages of the build) so that one can add stuff specific to each package (this will be much more useful when dpkg finally learns about build-arch and build-indep).

For the final touch, we want to do a few more things. First, we want to ensure that we are using QT4's qmake, because the build will fail if QT3's is used. Second, since I created a manpage for the command, I want to install it. Finally, we want to use parallel building when the user has specified it, to build the project faster. How to do that? Again, via variables:

QMAKE = qmake-qt4

DEB_BUILD_PARALLEL = 1

DEB_INSTALL_MANPAGES_qutecsound = debian/qutecsound.1

CDBS creates several variables for each of the packages listed in debian/control, so we can customize each package build. In our case, this variable is passed to dh_installman from the debhelper rules file (CDBS will invoke the debhelper tools once for each binary package).

After all this we now have the complete debian/rules of a relatively simple package. How does it look like? Like this:

#!/usr/bin/make -f



# Uncomment this to turn on verbose mode.

#export DH_VERBOSE=1



include /usr/share/cdbs/1/rules/debhelper.mk

include /usr/share/cdbs/1/class/qmake.mk



QMAKE = qmake-qt4

DEB_QMAKE_CONFIG_VAL += build64

DEB_QMAKE_ARGS = qcs.pro

DEB_BUILD_PARALLEL = 1



DEB_INSTALL_MANPAGES_qutecsound = debian/qutecsound.1



build/qutecsound::

        [ -f bin/qutecsound ]   mv bin/qutecsound-d bin/qutecsound



clean::

        rm -f bin/qutecsound

Tags: cdbs, packaging

We had another meeting of the Chicago R User Group last evening. This was scheduled somewhat belatedly once we learned that Drew Conway would be in town. Drew gave a very nice talk about his brand new infochimps package (for accessing the eponymous infochimps data service and marketplace). Slides are available on Drew's blog. In fact, he had already blogged about his talk before I had even started to write my slides... The user group meetings have a meme of showing how to use R with different editors, UIs, IDEs,... It started with a presentation on Eclipse and its StatET plugin. So a while ago I had offered to present on ESS, the wonderful Emacs mode for R (and as well as SAS, Stata, BUGS, JAGS, ...). And now I owe a big thanks to the ESS Core team for keeping all their documentation, talks, papers etc in their SVN archive, and particularly to Stephen Eglen for putting the source code to Tony Rossini's tutorial from useR! 2006 in Vienna there. This allowed me to quickly whip up a few slides though a good part of the presentation did involve a live demo missing from the slides. Again, big thanks to Tony for the old slides and to Stephen for making them accessible when I mentioned the idea of this talk a while back -- it allowed to put this together on short notice. And for those going to useR! 2011 in Warwick next summer, Stephen will present a full three-hour ESS tutorial which will cover ESS in much more detail.

<iframe frameborder="0" marginheight="0" marginwidth="0" scrolling="no" src="http://rcm.amazon.com/e/cm?lt1=_blank&amp;bc1=000000&amp;IS2=1&amp;bg1=FFFFFF&amp;fc1=000000&amp;lc1=0000FF&amp;t=dedasys-20&amp;o=1&amp;p=8&amp;l=as1&amp;m=amazon&amp;f=ifr&amp;md=10FE9736YVPPT7A0FBG2&amp;asins=0470929839" style="width: 120px; height: 240px;"></iframe> I got this to read on my trip to Oregon. Initially I was a bit dubious, after reading the chapter headings on Amazon. Indeed, if you read a lot of articles about tech startups, a lot of the material is not particularly novel, and are things that you've already read elsewhere. If, on the other hand, you're looking for a quick introduction to some of the common ideas behind web/tech startups, it's a pretty good list, written by a variety of people, with a lot of practical advice. The list of concepts includes "execution, not ideas", "fail fast", get feedback early", "get a co-founder", and a lot of similar advice that permeates discussions of early stage internet companies. Other people have different tastes, but for me, this passage was a strikingly effective advertisement for living in Boulder, Colorado:

With this as the backdrop, it was natural to make cycling a central part of my work with Everlater's founders, Nate Abbott and Natty Zola. Nate, Natty, and I scheduled regular weekly rides (sometimes several times a week) and while we didn't keep a formal agenda, each ride had a number of topics planned out for us to cover. In this way, we worked through many of the early challenges the business faced as it moved from idea to reality in the course of the summer of 2009. Our typical rides lasted from 60 to 90 minutes during which time we were able to focus on the business without interruption. And while getting some fresh air helps bring perspective, it's hard to avoid thinking about problems in a new way when you're out of the office, surrounded by the beauty of the Boulder foothills while your mind is completely clear. Startups, cycling, mountains... sign me up! Another tidbit I found interesting:

We often talk about the Number One startup killer at TechStars making a product for which there is no interesting market. TechStars accepts just 10 of more than 600 startups that apply and presumably they are among the best. But still, we find that at least one-third of those startups are attempting to build a product that they want, or that no one wants, instead of what the market wants But wait, I thought "execution is everything", and "ideas are worthless". It seems like avoiding bad ideas is potentially worth a whole lot of money in wasted time and energy though, no? The bit about ideas being worthless is a load of crap. The reason there is no market for ideas is that they are not "excludable". There was never a market for lighthouses either, the classic example in economics of a public good. So the government built them, and they were valuable to people operating ships. Ideas for startups may not have a lot of value compared to people executing them well, but to say they are worthless is hyperbole. In any case, though, if you're interested in startups, or especially if you happen to be interested in the TechStars program, it's book full of good advice, even if none of it will come as a revalation to anyone who follows that sort of thing regularly.

I ve used Delicious (del.icio.us) for some time now for managing bookmarks. I have never really used its social features, just its management features. With word that it is closing (sigh, just after I had to leave Bloglines, too) I m looking for something else. There are a somewhat bewildering array of new options available, and I m wondering if people have had experience with them. My requirements are:

• Completely reliable syncing between multiple devices, even if each one is being used and actively bookmarking things simultaneously. My browser is Firefox. Back when I chose Delicious, nothing else accomplished this.
• The ability to present bookmarks as a toolbar in Firefox, either using its own system or a different one.
• I prefer tagging to hierarchical organization.
• I can run a service on my own server, IF it is Free Software, trivial to set up, and needs little care and feeding. If it s not part of Debian, that s a strong negative here.
• If it s hosted in the cloud, I am concerned about privacy, security, and long-term stability. I must have a way to export my data, preferably automatically. I will be reading ToS and privacy policies carefully.
• A system that can be trusted to encrypt my bookmarks on the server side is another nice to have.
• Being able to create bookmarks directly from my Android phone is nice but not required.
• Being able to access bookmarks from a regular web browser is a feature I use a few times a year; again, nice but not required.
What should I look into? Incidentally, if you need to export your Delicious data, go here.

This has been bugging me ever since we got our Sony Internet TV, and I think I've finally figured out what's going on. Pandora hasn't been working on the Google TV ever since we got it, it just spits out Unexpected HTTP error occurred code: 417 while StartupAsyncTask and that's the end of that. I have vague recollections seeing this on my phone in the past also, but this doesn't appear to be the case any longer. My phone has version 1.5.2 on it, and the TV has 1.0, so I think I see the problem here. I took a full packet capture while starting up Pandora, and it was a lot more revealing. I could see that it was my Squid proxy (I transparently run all HTTP traffic through Squid) was barfing. That'd explain the errors I was seeing in Squid's cache.log

parseHttpRequest: Unsupported method ''
clientTryParseRequest: FD 153 (172.16.1.5:55001) Invalid Request

Looking at the headers of the HTTP request that the Pandora client is sending out, I see Expect: 100-Continue, and the text that accompanies a HTTP 417 error is "Expectation Failed", so I think this is the culprit. Searching for Expect: 100-Continue squid turns up information for an ignore_expect_100 directive, but this appears to only be available if Squid is compiled with --enable-http-violations, which the Debian Squid package doesn't appear to be compiled with. I may have to adjust my transparent proxying rules to not transparently proxy the TV, but this makes me a bit sad. Update I tried adding ignore_expect_100 on anyway, and Squid took it, and now Pandora isn't barfing. Yay!

So, I've been working on multitouch on and off the past few months (which have included a solid ten weeks of holiday), but have finally posted the third patch series, which I think should be pretty close to final, to the list today.

Touch events look kind of similar to standard Xi 2.1 events (DeviceMotionNotify and friends), but the implementation of grabbing and event delivery is quite different from the traditional X model that we all know and love.

Firstly, touch grabs are not strictly grabs as with other input events. When a synchronous pointer or keyboard grab activates, it 'freezes' the device, stopping delivery of all further events until the grabbing client has decided what to do. The client can either pass on the event to the next client (replay), opt to continue receiving all events with the device freezing again after each event (synchronous mode), or continue receiving all events with the device unfrozen (asynchronous). While this is happening, the grabbing client is the only one receiving events, hence the name.

Touch grabs break both of these assumptions. Firstly, the device never freezes: delivery of all events (pointer, keyboard, touch, whatever) continues at full tilt, and a touch grab never affects pointer or keyboard events, with the possible future exception of pointer events that are emulated from touch events. Secondly, all clients with grabs, as well as the client with the first selection we find<super>0</super>, receive the events. An 'owner' flag is set (or unset) to indicate to the client that it does, or does not, own the touch stream in question.

The owner principle is important: the theory is that all touch streams must be considered as a whole (i.e. from init, through all the motion events, to the touch stream ending), and cannot be separated into component parts. So a client which does not own a touch stream can do gesture recognition or whathaveyou in the background, but must not act upon it until it becomes the owner. We added an extra request, XIAllowTouchEvents, to control ownership: clients can either reject ownership, passing it on to the next in line, or assert their ownership and permanently remove all other clients from the delivery list.

So it's somewhat different, but (hopefully) not without reason. The motivation behind this was to keep latency as low as humanly possible, since people seem to demand lower latencies from direct touch-based devices. Sticking to the traditional grab model would have meant a flurry of round trips between all the clients in the stack and the server, as well as maintaining huge buffers in the server for every touch event, which each client would have to transfer in turn.

This model is a compromise between my original design and Ubuntu's suggested design; both of us had far more simple designs, but oriented towards different usecases. The original Ubuntu usecase was for a global gesture recognition system, which apps would only opt out of in rare cases. My primary usecase was exactly the reverse: apps doing their own gesture recognition, with a very small, if any, global system. Hopefully it's actually the best of both worlds, rather than the worst possible intersection: I guess we'll find out. :)

At the moment, it's all largely theoretical, but some of the Nokia Qt guys are working on a port and have provided really helpful feedback, and hopefully the GTK+ multitouch branch, which was written against a much earlier and, er, speculative, version of the spec, gets updated too.

Finally, some thanks: to both Nokia and my excellent employer Collabora for sponsoring my part of this work; to Chase Douglas from Canonical for his contributions and for not just giving us the finger and vanishing when we rejected the X Gesture Extension, but hanging around and working through it; and to Peter Hutterer, for spec work and review. There were also numerous others involved earlier (including, but not limited to, Henrik Rydberg working on mtdev and others) who deserve credit, so sorry if I forgot anyone.

If you want to get it and play around with it, the patches/repositories are all laid out in my initial mail linked to earlier, as well as the follow-up mail noting the evdev repository location. The spec has more details, and will hopefully be cleaned up to be a lot less clunkily-written.

[0]: The delivery order is unchanged. For grabs, we start at the root window and proceed down the window stack to the innermost child window, activating grabs in that order. For events selected for with XISelectEvents, we start at the innermost child window, work our way up towards the root window, and stop as soon as we find one.

Since I got my amateur radio license back in July, I ve had a lot of fun with it. It s a great hobby for anyone technically-inclined or anyone socially-inclined, and between those categories that includes a lot of people. I ve learned quite a bit over the last few months and really enjoyed it all. I passed my extra class exam back this fall, and thus got my new callsign, KR0L. So long, KD0MJT. I ve enjoyed some contesting, as well as general conversations on the system. I ve also done some work with the keyboard-to-keyboard digital modes on HF. Debian includes a very nice program called fldigi for this. Of late, I have developed an interest in packet radio. Packet radio uses a networking protocol called AX.25 over RF links. AX.25 bears a familial resemblance to TCP/IP, and in fact, you can run TCP/IP over AX.25 and AX.25 over TCP/IP. My learning curve on packet was somewhat steep. It has declined in popularity significantly since the growth of generally-available Internet access, though seems to be once again growing now. So a lot of information about it is 10 years old. As I was learning about packet, I of course was using my Debian system. The Linux kernel has long had AX.25 support integrated as a first-class networking protocol. You can open AX.25 sockets, monitor AX.25 traffic, etc. from the Linux kernel. You can use soundmodem to make a software-defined packet modem (called a TNC), or you can use kissattach to hook up to a traditional TNC via a serial port and a protocol strongly similar to SLIP (which, for those of you with shorter memories, is a predecessor to PPP). Linux can do what you d expect out of a modern networking system: multiplexing with AX.25, handling lots of simultaneous users, etc. So I was a bit surprised and baffled to keep running into systems that only supported 1 user at a time, couldn t easily do some things I was taking for granted, etc. Until I realized that Linux is the only major operating system with integrated AX.25 support in the kernel. Things started to make a bit more sense. I hadn t realized just how awesome a setup I had until I started learning about the hoops some other people went through. It is pretty easy to run a basic client on Windows, but to run the server side of things as I am doing well some of the features just aren t there or are really kludgy. Anyhow, I have decided to start documenting things I learn as I go. Beyond amateur radio, I also have sometimes wanted places to stick bits of information. Things that other people might benefit from if they Google, but that maybe aren t the best blog fodder or website material. So I have set up a wiki, openly editable of course, at http://wiki.complete.org/. To date, only the amateur radio section has much content in it. I m also sending in patches and bug reports to the various projects involved in amateur radio in Linux, and am glad to see development has resumed on several of those.

Update There is an easier method to do all this using gbp-clone as described here. Ah ! Then to build the package, you just need to suggest git-buildpackage where to find the pristin-tar :

git-buildpackage --git-upstream-branch=upstream/master

or you could simply describe (as suggested) the layout in debian/gbp.conf. Easy !!!

---

I've found a lot of different recipes and howtos about git debian packaging, but I failed to find one simple recipe to create a debian package from scratch when upstream is using git. Of course the following is a big patchwork from many different sources. First we need to do a bit of administrative work to setup the repository :

mkdir yourpackage
cd yourpackage
git init --shared

Then, since I'm interested in tracking upstream development branch I'm going to add a remote branch to my repo:

git remote add upstream git://the.url/here.git

at this point I need to fetch upstream and create a branch for it.

git fetch upstream
git checkout -b upstream upstream/master

Now in my repo I have a master branch and an upstream branch. So far, so good. Let's add the debian branch based on master:

git checkout master
git checkout -b debian master

It's in the debian branch where I'm going to keep the debian related files. I'm finally read for hacking git add / git commit / git remove ... When I'm done, I can switch to master, merge the debian branch into it and use git-buildpackage to build the package.

git checkout master
git branch
debian
* master
upstream

git merge debian
git-buildpackage

Suppose I want to put everything on gitourious for example. I'll create an acocunt, set up my ssh pub key and then I've to add an origin ref in my .git/config . Something like :

[remote "origin"]
url = git@gitorious.org:debian-stuff/yourpackage.git
fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
remote = origin
merge = refs/heads/master

The only thing left to do is to push everything on gitourious. the --all is important.

git push --all

People willing to pull your work from girourious have to follow the following script :

$git clone git@gitorious.org:debian-stuff/yourpackage.git
$cd yourpackage
$git branch -a
* master
remotes/origin/HEAD -> origin/master
remotes/origin/debian
remotes/origin/master
remotes/origin/upstream
$git checkout -t origin/debian
$git checkout -t origin/upstream
$git branch -a
debian
master
* upstream
remotes/origin/HEAD -> origin/master
remotes/origin/debian
remotes/origin/master
remotes/origin/upstream
$git checkout master
$git-buildpackage

Maybe there is an easier way to pull all remote branches at once, but I'm not aware of it. Any better way ?

An other week, a few more bugs looked, unfortunately in the mean time the number of bug grew to 875

• #471423 ping ati driver ignore Option Enable false in monitor section, but accept the undocumented Option Disable true .
• #468561 ping & closed vesa: switching resolution with xrandr after suspend-to-ram.
• #447526 ping memory leaks.
• #475534 closed as reported said, oclock seams to work correctly
• #466044 ping xserver-xorg-core: internal screen of laptop with external screen attached not blanked when laptop closed but still in use.
• #482317 ping libxt6: scroll list overflow in gv.
• #425814 ping xserver-xorg-video-ati: Xv broken on Mach64 3D Rage LT Pro AGP-133.

The X Strike Force (still) needs you ! You can have a look at the X Strike Force Bug Closing Procedure and check XSF unstable bugs sorted by date.

Paul Wise just pointed me to a very good explanation of the issue with non-free firmware files:

Updated qlogic 2400 and 2500 firmware to 5.03.13. What does 5.03.13 do? No one knows, except for QLogic, and they're not telling. I asked, and they told me that information was only available under NDA. So, I encourage you to imagine what this firmware does, and the bugs it fixes. While you're at it, imagine a world where vendors release source code for their firmware.

The source? No, not some ideologist, not even someone involved in Debian. It's a comment about an update of a firmware package in Fedora.

One of the bad aspects of the the C Programing language is that it silently allows many bad C programs. Together with the widespread use of an architecture that is very bad at catching errors (i386), this sometimes leads to common ideoms that are only working accidentially. This is bad as they often break on other architectures and can break with every new optimisations or new feature the compiler adds. Take for example a look at the code there (I tried to leave a comment there but did not succeed): If you see something like this:

     char buffer[1000];
     struct thingy *header;
     header = (struct thingy *)buffer;

then it is time to run. I hope you do not depend on this software, because it is a pure accident if this is doing anything at all. While you can cast a char * to a struct, that is only allowed if that memory actually was this struct (or one compatible, like a struct with the same initial part and you are only accessing that part). In this case it is obviously not (it's just an array of char), so you might see bus errors or random values if the compiler does no optimisations and you are on a architecture where alignment matters. Or the compiler might optimize it to whatever it wants to, because the C compiler is allowed to do everything with code like that. The next problem is the one that post was about: You are not allowed to access an array after its end. Something like

 struct thingy  
     int magic;
     char data[4];
  

means you may only access the first 4 bytes of data. If you access more than that it may work now on your machine, but it can stop tomorrow with the next revision of the compiler or on another machine. If you have a struct with a variable length data block, then you can use the new C99 feature of char data[] or the old gcc extension of char data[0]. Or you can use unions. (Or in some case use the behaviour of structs with the same initial parts). If you use C code with undefined semantics then every new compiler might break it with some optimisation. There is often the tempting option of just using a slightly different code that currently works. But in the not too distant future the compiler (or even some processor) might again get some new optimisations and the code will break again. Fixing it properly might be harder but it's less likely it will fail to compile again and it also reduces the chances that it will not fail to compile but simply do something you did not expect.

This guy is very known by people who use to walk around the Paulista Avenue at night. This week Gilberto Kassab mayor of S o Paulo authorized the local police to arrest him and all other artists in order to replace them by one more crap christmas tree playing those well known depressed songs. SAD :_(

The bug webinterface of the Ultimate Debian Database currently knows about the following release critical bugs:

In Total:	655
Affecting Squeeze:	204
Squeeze only:	56
Remaining to be fixed in Squeeze:	148

Of these 148 bugs, the following tags are set:

Pending in Squeeze:	8
Patched in Squeeze:	30
Duplicates in Squeeze:	15
Can be fixed in a security Update:	17
Contrib or non-free in Squeeze:	4
Claimed in Squeeze:	2
Delayed in Squeeze:	5
Otherwise fixed in Squeeze:	21

Ignoring all the above (multiple tags possible) 88 bugs need to be fixed by Debian Contributors to get Debian 6.0 Squeeze released. However, with the view of the Release Managers, 156 need to be dealt with for the release to happen. Please see Interpreting the release critical bug statistics for an explanation of the different numbers.

In the last years, I have seen many people try to use dpkg --build to create Debian packages. Indeed, if you look up dpkg s and dpkg-deb s manual pages, this option seems to be what you have to use:

-b, --build directory [archive directory] Creates a debian archive from the filesystem tree stored in directory. directory must have a DEBIAN subdirectory, which contains the control information files such as the control file itself. This directory will not appear in the binary package s filesystem archive, but instead the files in it will be put in the binary package s control information area.

And indeed, dpkg-deb is what ultimately creates the .deb files (aka binary packages). But it s a low-level tool that you should not call yourself. If you want to properly package a new software, you should rather create a Debian source package that will transform upstream source code into policy-compliant binary packages. Creating a source package also involves preparing a directory tree (but with a debian sub-directory), it s probably more complicated than calling dpkg -b on a manually crafted directory. But the result is much more versatile: the tools used bring value by dynamically analyzing/modifying the files within your package (for example, the dependencies on C libraries that your package needs are automatically inserted). If this is news to you, you might want to check out the New Maintainers Guide and the Debian Policy.

Found it useful? Be sure to not miss other packaging tips (or lessons), click here to subscribe to my free newsletter and get new articles by email.

10 comments Liked this article? Click here. My blog is Flattr-enabled.

Recently gcc (4.5) improved its ability to see the size of various structures. As a result, the FORTIFY protections have suddenly gotten a bit stricter. In the past, you used to be able to do things like this:

struct thingy  
    int magic;
    char data[4];
 
void work(char *input)  
    char buffer[1000];
    int length;
    struct thingy *header;
    header = (struct thingy *)buffer;
    length = strlen(input);
    if (length > sizeof(buffer) - sizeof(*header) - 1) abort();
    strcpy(header->data, input);
    header->magic = 42;
    do_something_fun(header);
 

The problem here is that gcc thinks that header->data is only 4 bytes long. But gcc doesn t know we intentionally overruled this (and even did length checking), so due to -D_FORTIFY_SOURCE=2, the strcpy() checks kick in when input is more than 4 bytes. The fix, in this case, is to use memcpy() instead, since we actually know how long our destination is, we can replace the strcpy(...) line with:

    memcpy(header->data, input, length + 1); /* take 0-term too */

This kind of header and then data stuff is common for protocol handlers. So far, things like Wine, TFTP, and others have been experiencing problems with the change. Please keep an eye out for it when doing testing.

I've recently spent five days debugging a problem in obnam, my backup program. It turned out to be a bug in my B-tree implementation, which I wrote for obnam. The process was pretty long, and it might be instructive to de-brief myself. The bug manifested itself when I ran a backup test that was larger than what I had ran before: one terabyte of live data. The symptom was a key lookup problem in a B-tree. Good: the problem was entirely deterministic, and easy to reproduce. Bad: it took hours to reproduce it. My first step was to fiddle with the obnam parameters to see if I could make it fail faster, but in the same way. I got the time down to half an hour. After this, I started fiddling with the code to see if I can figure out what happens, and perhaps find a quick fix to the bug. After some hours, I realized I was being stupid. I was making essentially random changes to the code, instead of hunting down the actual cause. I had no idea what was going wrong, so trying to fix it was an act of utter stupidty. So I went back to where I had started from. The next step was to run "obnam fsck" on the failing backup store. This did not find any problems. The fsck I've written is a bit simplistic, and merely checks that for every client in the backup store, every generation is found, and within every generation each file's metadata, and data chunks are found. I will improve fsck in the future, but right now it was not helpful. I did not have a tool to verify that the actual B-trees were OK. So I wrote one. It, too, is quite simplistic, but at least it proved that the key was still present in the on-disk version of the B-tree. At this point, I did a code review of the B-tree code. It had been a some months since I'd touched the code, and hence the review brought up a bunch of minor issues: outdated comments, badly formatted code, stupid variable names, etc. One or two buglets, too, but nothing relevant to the case of the missing key. At this point I made a mistake: I concluded that the error was not in the B-tree code, but the obnam code using the B-trees. I spent the next couple of days running the test case, adding more logging, and analyzing the results. Without my mistake, I would not have ignored indicators that would have led me faster into the depths of the B-tree code. I might have saved a day or two of the process. Eventually I narrowed down the problem to a single B-tree insertion call in the obnam code. The call inserted a new value for a given key into one tree, and the value in another tree changed as well. Quite mysterious. My B-trees are an implementation of the Rodeh B-trees. All updates are done using copy-on-write, and this allows efficient cloning of trees. However, as an optimization, a node that is used by only one tree at a time can be modified in-place. This speeds things up a lot when the same tree is updated many times. In obnam, each B-tree represents a backup generation. The specific bug I was hunting was that the key I use for storing the unique identifier for a generation changed in two trees at the same time. Now that I have actually found and fixed the problem, it is clear that I should have realized at once what was happening, but I didn't, so I spent another couple of days adding logging statements into the B-tree insertion code, and tracing where things went wrong. Since B-trees are updated quite heavily by obnam, the log files grew quite big, and the test runs grew quite long. The last run I did resulted in a 94 gigabyte log file, and lasted over 12 hours. less had trouble browsing the log file, probaby due to the rather long lines generated by dumps of the B-trees. I ended up splitting the large file into quite a number of small ones, using csplit. After some more analysis, I figured out the problem: the code that shadowed a B-tree node, i.e., implemented the copy-on-write modification, failed to update the reference counts of the children of a shadowed index node. The Rodeh B-trees use lazy reference counts: when an index node is shadowed, only the immediate children get reference count updates. This turns out to work quite well, and saves quite massive amounts of updates. Unfortunately, my code was too lazy, and did not update any reference counts. After I had come this far, it only took a couple of minutes to write a unit test to catch the problem, fix the code, run the unit tests successfully, and commit. Then I ran the big test case again, without the extra logging I had added, and the code worked fine. At least it worked fine for about 24 hours until it ate up all memory in the VM I was running it on, but that's a separate problem. What can I do to prevent this kind of bug happening in the future? Unit testing didn't catch it, and I have 100% statement coverage with unit tests (not counting 19 excluded statements). Code review by other programmers would surely help. I should perhaps attract others to the project. Please excuse the length of this blog post. It is a distillation of the about ten A4-pages' worth of notes I made during this debugging process. (Always make notes when debugging. Your brain can't remember all details.)

It's been a while since I started working on this project and even longer
since I had the idea. It's therefore a pleasure to finally announce the DACA project.


What is DACA?

Automated Code Analysis helps detect and fix bugs and other issues in source
code. The project aims to give users easy access to a wide variety of tools to
improve quality of software distributed by Debian, while giving the tool's
developers a test bed, more visibility, and more feedback.
This is achieved by running those tools on the complete Debian archive.

http://qa.debian.org/daca/

What is there for everyone?

At the moment there are only partial reports from two tools, but the list of
tools to be evaluated and possibly included goes over twenty.

Current tools: cppcheck, and checkbashisms (at the source package level.)

Limitations

Most of the tools are CPU-bound, limiting considerably the number of tools and
time it takes to check the whole Debian archive. For example, with the typical
sid repository update (i.e. not during the freeze and with a working ftp-
master) it is impossible for the server running cppcheck to keep up with all
the changes.

How can you help?

* First of all you can go and squash bugs!
Please keep in mind what's in the notices at the bottom of the pages. They are
rather static now, but they may change later.

* Second, report false positives, fix bugs, improve the tools
Every report page (at the footer) should mention the version of the tool used
to generate it. They are usually the latest.

* Third, join the DACA project
More hands are needed to evaluate other tools, setup an infrastructure for
running them, and finally generating the web reports.
Discussing tools already available at DACA is also welcome.

There's a project request at Alioth pending its approval, but once accepted
access to the repository and mailing lists will be found at:

http://alioth.debian.org/projects/daca

(the contact email address at the DACA website will be updated accordingly)

* Fourth, donate hardware

If you have equipment with very powerful CPUs or other hardware you (or your
company) can donate, please take a look at the following page:

http://www.debian.org/donations#equipment_donations

Note that this may be handled by people unrelated to the DACA project, and as
such you should mention what you would prefer your hardware to be used for.

* And finally, figure out what I missed, provide feedback, and go back to step
one.


Thanks!

By the way, thanks to the DSA team for tolerating a few hiccups on the host running
cppcheck, and to those who previewed the website and provided feedback.