Debian 6.0 Squeeze to be released with completely free Kernel(s) Today we have announced that, starting with the upcoming release of Squeeze, Debian will be even Free-er. Exceptions to the DFSG for non-free Linux firmware blobs, which have been granted in the past, will no longer be granted. Starting from Squeeze, Debian will be Free the bottom up, no matter where your own definition of software ends. The news has been known for a while within the Debian development community, but we had the pleasure of sharing that with our users only today. According to the reaction on identi.ca, where the news quickly made into the most popular notices, people are enthusiastic about the news and that in turn is very rewarding. If you are as happy and proud about this achievement as I am, please direct your kudos to all the teams who made this possible:

• the Debian kernel team who worked for the past 2 release cycles, together with Linux upstream, to split free-vs-non-free firmware bits and monitor new releases to avoid freedom regressions;
• the Debian CD team who basically "forked" different media production lines, for different user needs wrt firmware;
• the d-i team who added, long ago, support for loading user-provided firmware at installation time;
• whoever else I might have forgotten who contributed to this result (feel free to leave a comment to refresh my memory!).

Having been trol^W asked several times in the past about the "real freeness" of Debian at events, talks, and the like, I consider this episode to be a very nice story to tell. While we have granted exceptions for Linux firmware at the time of Etch and Lenny releases, during this time Debian people have worked, together with upstream, to make this final result possible. All this work has improved the situation not only for Debian users, but also for users of those derivatives which rely upon Debian kernels. Nonetheless, don't expect us to live under a rock. We know that there are users out there who need nasty non-free firmware bits to boot their machines or to access the Internet. That is why separate images with firmware included are being provided, although they are not part of Debian and are properly "discriminated": they should be looked for explicitly by users and can't be supported to the same extent of Free firmware, simply because we (as anybody else other than the hardware manufacturer) do not have access to the corresponding source code. This separation is a perfect example of the pragmatism embodied by the Debian Social contract: Debian consists of 100% Free software; at the same time, Debian enables its users to make their own choices in terms of Freedom, being explicit about what users risk if they cross the boundaries of DFSG. Awareness is the key here and I find it to be a very honest yet effective way of explaining to the world why Free Software is better. This is not the end of the story though, as we need to document this change properly to both increase awareness and avoid leaving users in the dark. A couple of bugs filed today might benefit from the help of some kind doc-writers: Debian bug #607191 (documenting the firmware change in the Squeeze release notes) and Debian bug #607193 (documenting the firmware change on the website).

Being able to rebuild an existing Debian package is a very useful skill. It s a prerequisite for many tasks that an admin might want to perform at some point: enable a feature that is disabled in the official Debian package, rebuild a source package for another suite (for example build a Debian Testing package for use on Debian Stable, we call that backporting), include a bug fix that upstream developers prepared, etc. Discover the 4 steps to rebuild a Debian package. 1. Download the source package The preferred way to download source packages is to use APT. It can download them from the source repositories that you have configured in /etc/apt/sources.list, for example:

deb-src http://ftp.debian.org/debian unstable main contrib non-free
deb-src http://ftp.debian.org/debian testing main contrib non-free
deb-src http://ftp.debian.org/debian stable main contrib non-free

Note that the lines start with deb-src instead of the usual deb . This tells APT that we are interested in the source packages and not in the binary packages. After an apt-get update you can use apt-get source publican to retrieve the latest version of the source package publican . You can also indicate the distribution where the source package must be fetched with the syntax package/distribution . apt-get source publican/testing will grab the source package publican in the testing distribution and extract it in the current directory (with dpkg-source -x, thus you need to have installed the dpkg-dev package).

$ apt-get source publican/testing
Reading package lists... Done
Building dependency tree
Reading state information... Done
NOTICE: 'publican' packaging is maintained in the 'Git' version control system at:
git://git.debian.org/collab-maint/publican.git
Need to get 727 kB of source archives.
Get:1 http://nas/debian/ squeeze/main publican 2.1-2 (dsc) [2253 B]
Get:2 http://nas/debian/ squeeze/main publican 2.1-2 (tar) [720 kB]
Get:3 http://nas/debian/ squeeze/main publican 2.1-2 (diff) [4728 B]
Fetched 727 kB in 0s (2970 kB/s)
dpkg-source: info: extracting publican in publican-2.1
dpkg-source: info: unpacking publican_2.1.orig.tar.gz
dpkg-source: info: unpacking publican_2.1-2.debian.tar.gz
$ ls -dF publican*
publican-2.1/                 publican_2.1-2.dsc
publican_2.1-2.debian.tar.gz  publican_2.1.orig.tar.gz

If you don t want to use APT, or if the source package is not hosted in an APT source repository, you can download a complete source package with dget -u dsc-url where dsc-url is the URL of the .dsc file representing the source package. dget is provided by the devscripts package. Note that the -u option means that the origin of the source package is not verified before extraction. 2. Install the build-dependencies Again APT can do the grunt work for you, you just have to use apt-get build-dep foo to install the build-dependencies for the last version of the source package foo. It supports the same syntactic sugar than apt-get source so that you can run apt-get build-dep publican/testing to install the build-dependencies required to build the testing version of the publican source package. If you can t use APT for this, enter the directory where the source package has been unpacked and run dpkg-checkbuilddeps. It will spit out a list of unmet build dependencies (if there are any, otherwise it will print nothing and you can go ahead safely). With a bit of copy and paste and a apt-get install invocation, you ll install the required packages in a few seconds. 3. Do whatever changes you need I won t detail this step since it depends on your specific goal with the rebuild. You might have to edit debian/rules, or to apply a patch. But one thing is sure, if you have made any change or have recompiled the package in a different environment, you should really change its version number. You can do this with dch --local foo (again from the devscripts package), replace foo by a short name identifying you as the supplier of the updated version. It will update debian/changelog and invite you to write a small entry documenting your change. 4. Build the package The last step is also the simplest one now that everything is in place. You must be in the directory of the unpacked source package.
Now run either debuild -us -uc (recommended, requires the devscripts package) or directly dpkg-buildpackage -us -uc . The -us -uc options avoid the signature step in the build process that would generate a (harmless) failure at the end if you have no GPG key matching the name entered in the top entry of the Debian changelog.

$ cd publican-2.1
$ debuild -us -uc
 dpkg-buildpackage -rfakeroot -D -us -uc
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2
dpkg-buildpackage: export CPPFLAGS from dpkg-buildflags (origin: vendor):
dpkg-buildpackage: export CXXFLAGS from dpkg-buildflags (origin: vendor): -g -O2
dpkg-buildpackage: export FFLAGS from dpkg-buildflags (origin: vendor): -g -O2
dpkg-buildpackage: export LDFLAGS from dpkg-buildflags (origin: vendor):
dpkg-buildpackage: source package publican
dpkg-buildpackage: source version 2.1-2rh1
dpkg-buildpackage: source changed by Rapha l Hertzog <hertzog>
 dpkg-source --before-build publican-2.1
dpkg-buildpackage: host architecture i386
[...]
dpkg-deb: building package  publican' in  ../publican_2.1-2rh1_all.deb'.
 dpkg-genchanges  >../publican_2.1-2rh1_i386.changes
dpkg-genchanges: not including original source code in upload
 dpkg-source --after-build publican-2.1
dpkg-buildpackage: binary and diff upload (original source NOT included)
Now running lintian...
Finished running lintian.

The build is over, the updated source and binary packages have been generated in the parent directory.

$ cd ..
$ ls -dF publican*
publican-2.1/                    publican_2.1-2rh1.dsc
publican_2.1-2.debian.tar.gz     publican_2.1-2rh1_i386.changes
publican_2.1-2.dsc               publican_2.1-2rh1_source.changes
publican_2.1-2rh1_all.deb        publican_2.1.orig.tar.gz
publican_2.1-2rh1.debian.tar.gz

Do you want to read more tutorials like this one? Click here to subscribe to my free newsletter, you can opt to receive future articles by email.

11 comments Liked this article? Click here. My blog is Flattr-enabled.

A new package RcppDE has been uploaded in a first version 0.1.0 to CRAN. It provides differential evolution optimisation---a variant of stochastic optimisation that is similar to genetic algorithms but particularly suitable for the floating-point representations common in numerical optimisation. It builds of on the nice DEoptim package by Ardia et al, but reimplements the algorithm in C++ (rather than C) using a large serving of Rcpp and RcppArmadillo. I worked on this on for a few evenings and weekends in October and November and then spent a few more evenings writing a paper / vignette (which is finished as a very first draft now) about it. This was an interesting and captivating problem as I had worked on genetic algorithms going back quite some time to the beginning and then again the end of graduate school (and traces of that early work are near the bottom of my presentations page). So what got me started? DEoptim is a really nice package, but it is implemented in old-school C. There is nothing wrong with that per se, but at the same time that I was wrestling with GAs, I also taught myself C++ which, to put it simply, offers a few more choices to the programmer. I like having those choices. And with all the work that Romain and I have put into Rcpp, I was curious how far I could push this cart if I were to move it along. I made a bet with myself starting from the old saw shorter, easier, faster: pick any two. Would it be possible to achieve all three of these goals? DEoptim, and I take version 2.0-7 as my reference point here, is pretty efficiently yet verbosely coded. Copying a vector takes a loop with an assignment for each element, copying a matrix does the same using two loops. Replacing that with a single statement in C++ is pretty easy. We also have a few little optimisations behind the scenes here and there in Rcpp: would all that be enough to move the needle in terms of performance? And the same time, DEoptim is also full of the uses of the old R API which we often point to in the Rcpp documentation so fixing readibility should be a relatively low-hanging fruit. To cut a long story short, I was able to reduce code size quite easily by using a combination of C++ and Rcpp idioms. I was also able to get to faster: the paper / vignette demostrates consistent speed improvements on all setups that I tested (three standard functions on three small and three larger parameter vectors). More important speed gains were achieved by allowing use of objective functions that are written in C++ which again is both possible and easy thanks to Rcpp. That leaves easier to prove: adding compiled objective functions is one indication; further proof could be provided by, say, moving the inner loop to parallel execution thanks to Open MP which I may attempt over the next few months. So far I'd like to give myself about half a point here. So not quite yet shorter, easier, faster: pick any three, but working on it. Over the next few days I may try to follow up with a blog post or two contrasting some code examples and maybe showing a chart from the vignette.

I recently decided to try maintaining a Debian package (bibutils) without committing any patches to Git. One of the disadvantages of this approach is that the patches for upstream are not nicely sorted out in ./debian/patches. I decided to write a little tool to sort out which commits should be sent to upstream. I'm not too happy about the length of it, or the name "git-classify", but I'm posting in case someone has some suggestions. Or maybe somebody finds this useful.

#!/usr/bin/perl
use strict;
my $upstreamonly=0;
if ($ARGV[0] eq "-u") 
  $upstreamonly=1;
  shift (@ARGV);
 
open(GIT,"git log -z --format=\"%n%x00%H\" --name-only  @ARGV ");
# throw away blank line at the beginning.
$_=<GIT>;
my $sha="";
LINE: while(<GIT>) 
  chomp();
  next LINE if (m/^\s*$/);
  if (m/^\x0([0-9a-fA-F]+)/) 
    $sha=$1;
    else  
    my $debian=0;
    my $upstream=0;
    foreach my $word  ( split("\x00",$_) )  
      if  ($word=~m@^debian/@)  
    $debian++;
        elsif (length($word)>0)   
    $upstream++;
       
     
    if (!$upstreamonly) 
      print "$sha\t";
      print "MIXED" if ($upstream>0  && $debian>0);
      print "upstream" if ($upstream>0  && $debian==0);
      print "debian" if ($upstream==0  && $debian>0);
      print "\n";
      else  
      print "$sha\n" if ($upstream>0  && $debian==0);
     
   
 
=pod
=head1 Name
git-classify  - Classify commits as upstream, debian, or MIXED
=head1 Synopsis
=over
=item B<git classify> [I<-u>] [I<arguments for git-log>]
=back
=head1 Description
Classify a range of commits (specified as for git-log) as I<upstream>
(touching only files outside ./debian), I<debian> (touching files only
inside ./debian) or I<MIXED>. Presumably these last kind are to be
discouraged.
=head2 Options
=over
=item B<-u> output only the SHA1 hashes of upstream commits (as
      defined above).
=back
=head1 Examples
Generate all likely patches to send upstream
     git classify -u $SHA..HEAD   xargs -L1 git format-patch -1

As a test project, I updated linuxsi.com (in Italian) to Rails 3. So far, I am not terribly impressed. It's nice enough, but the upgrade process was quite painful - there's a lot involved in upgrading even such a relatively simple site. Rails remains the best thing out there, as far as I can tell, in terms of programmer productivity for web projects, but at times it seems like there's a lot of taillight chasing. Upgrade this. Move to this newer, fancier, shinier system. Replace that. Sure, I suppose you can always stick with older versions, but it seems that that is a recipe for simply postponing all of the changes until a later date, when they become even more onerous. And Ruby gems don't really seem to have a mechanism to separate updates with new features from security updates, so for some things staying with older versions might be a security risk. And stuff like plugins mostly just sits there where you dumped it into your project, without any real sense of an upgrade path. I suppose I'm being a bit of a Grumpy Old Man, but at times the changes feel a bit gratuitous, and with all the breakage that Rails 3 causes, I would caution against upgrading to it for existing projects - use it for new projects to learn it. In some ways it seems to be a Ruby web framework that happens to have some compatibility with Rails. But not 99% - more like 80%, so that you'll spend 80% of your time upgrading that 20%. As one particular example, if you look at the original rails screencast, he uses <%= form_tag, with the "=". Subsequent versions of Rails dropped that, so you'd do <% form_tag .... do ... and now, with Rails 3, we're back to <%= form_tag, with the "=" again. Hrmph! Hopefully, with a bit of time, I'll see some of the nicer aspects of Rails 3.

(This is part of a series of posts on Why Linux?) My first PC was an Amstrad PPC640D; an 8088 with twin 720k 3.5" disk drives. It never ran Windows (3.0 was current at that point in time and I don't think it would manage to run off a single floppy), so ran DOS. I moved on to an 8086 desktop machine, complete with 10M full height 5.25" HDD and CGA graphics. It still runs DOS. From there I moved to an 80386DX-40 desktop, with 4M RAM and SVGA graphics. A massive leap forward, and something actually capable of running more than DOS. Except I didn't. I had a Windows for Workgroups 3.11 install, but mostly I still did what I needed to from DOS. The machine was never networked; it had a modem attached but that was used to connect to Fidonet which was well serviced by DOS tools. I put Linux on the box at one point but it was C and TCP/IP and I was Pascal and Fido in those days, so I didn't really know what to make of it. Fast forward a few years and I'm still mostly using DOS, but I'm on a 486 and am running a separate machine as a BBS. It's using RemoteAccess under DOS and it feels like with 486 hardware I should be able to do some of this multitasking lark. I try OS/2 and Windows 95, but both end up dropping modem data when doing other things (I've moved on to ISDN at this point). Maybe I just needed to tweak things more, but I deem multitasking with BBS software a failure and go back to DOS and 2 machines. When I went to university one of my new course mates had brought a machine running Linux, and various of the older students are already running it. There was a wealth of information and interest available to me. So I try again. I've learnt C and TCP/IP networking since last time, and suddenly it all makes more sense and I'm able to do more with it (I'm sure a summer using HP/UX on my desktop at Nortel helped). And, as I finally get to the point, it's efficient. It makes use of the extra memory in the machine that DOS can only touch with kludges. It allows me to multitask in a usable fashion. I don't feel the need for a GUI so I don't have to run one, which no doubt helps, but everything that's running is easily visible and tunable. I start running it as my desktop at university and convert the BBS over to it at some point soon after. It doesn't drop modem data. I rejoice, and don't look back. In those days I was running hardware that was probably at the low end of what the popular multitasking OSes wanted (I remember seeing Win95 on a 4M machine when it first came out, and it crawled). These days my main machines are (I would hope) more than capable of running Windows well. The efficiency angle is still an appeal of Linux though; for example in the server space I don't understand why you'd want to run something with the overhead of an always on GUI (people who leave Linux servers running GDM confuse me). I want to be able to do everything I need on a server remotely, be that via SSH or, in a pinch, a serial console. I don't want to sit at the box and use a GUI. Linux lets me run only what I actually need on the machine.

The two first interviews were dedicated to long-time Debian developers. This time I took the opposite approach, I interviewed David Kalnischkies who is not (yet) a Debian developer. But he s contributing to one of the most important software within Debian the APT package manager since 2009. You can already see him in many places in Debian sharing his APT knowledge when needed. English is not his native language and he s a bit shy, but he accepted the interview nevertheless. I would like to thank him for the efforts involved and I hope his story can inspire some people to take the leap and just start helping My questions are in bold, the rest is by David. Who are you? I am David Kalnischkies, 22 years old, living in the small town Erbach near Wiesbaden in Germany and I m studying computer science at the TU Darmstadt. Furthermore I am for more than half a decade now young group leader of my hometown. I never intended to get into this position, but it has similarities with my career in this internet-thingy here. I don t remember why, but in April 2009 I was at a stage that some simple bugs in APT annoyed me so much that I grabbed the source, and most importantly I don t know why I did it but I published my changes in Mai with #433007, a few more bugs and even a branch on launchpad. And this public branch got me into all this trouble in June: I got a mail from Mr. package managment Michael Vogt regarding this branch A few days later I joined an IRC session with him and closely after that my name appeared for the first time in a changelog entry. It s a strange but also addicting feeling to read your own name in an unfamiliar place. And even now after many IRC discussions, bugfixes and features, three Ubuntu Developer Summits and a Google Summer of Code in Debian, my name still appear in places I have never even thought about e.g. in an interview. What s your biggest achievement within Debian? I would like to answer MultiArch in APT as it was my Google Summer of Code project, but as it has (not much) use for the normal user at this point will hopefully change for wheezy I chose three smaller things in squeeze s APT that many people don t even know yet:

• MMap run out of room , the most famous error related to the APT::Cache-Limit option is gone. The DynamicMMap in APT now deserves its name
• Download and usage of multiple Translations of Descriptions
• crazy stuff like apt-cache show ^apt-.*$/experimental works

If your impression is now that I only do APT stuff: that s completely right, but that s already more than enough for me for now as the toolchain behind the short name APT contains so many tools and use cases that you always have something different. You re an active member of the APT development team. Are there plans for APT in Debian Wheezy? What features can we expect? That s very hard to answer, as the team is too small to be able to really plan something. I mean, you can have fancy plans and everything and half a second later someone arrives on the mailing list with a small question which eats days of development time just for debugging But right now the TODO list contains (in no particular order):

• MultiArch managing packages built for different architectures like i386 on amd64 to ease the usage of packages which only work on specific architectures or cross-compiling.
• rewriting the subsystem responsible for downloading files to e.g. enable tools like apt-file or debtags to reuse the infrastructure by plugging into it.
• splitting long descriptions out of the Packages file to possibly decrease the size to download on an apt-get update . Includes mostly pulling the button only to enable it by the archive masters
• backporting popular features of some frontends to enable all frontends to use them, e.g. apt-get changelog or apt-get download
• support research on resolver improvements by implementing support for a distribution independent upgrade problem format (CUDF).
• an additional single binary for users wrapping apt-get, apt-cache and co.
• getting in contact (again) with various forks like apt-rpm and derivatives like maemo or telesphoreo to join forces instead of splitting them by working on different (and possibily very old) heads of the dragon

We will see what will get real for wheezy and what is postponed, but one thing is sure: more will be done for wheezy if you help! If you could spend all your time on Debian, what would you work on? I would spend it on APT s debbugs count zero would be cool to look at! We make progress in this regard, but with the current velocity we will reach it in ten years or so. Reading more mailing lists would be interesting, as I am kind of an information junky. Maintaining a package could be interesting to share the annoyance of a maintainer with handcrafted dependencies just to notice that APT doesn t get it in the way I intended it to be. Through, to make it feel real I need to train a few new APT contributors before so they can point my mistake out, but this unfortunately doesn t depend so much on time but on victims Maybe I could even be working on getting an official status. Beside that, I would love to be able to apt-get dist-upgrade the increasing mass of systems I and many others carry around in their pockets. In regards to my phone, this is already fixed, but there is much room for improvements. What s the biggest problem of Debian? You need to be lucky. You need to talk at the right time to the right person. That s not really a debian-only problem as such, but in a global project full of volunteers you can see it clearly as there are plenty of opportunities to be unlucky. For example, it s unlikely that an interview would be made with me now if Michael had not contacted me in June 2009. In a big project like Debian, you are basically completely lost without a mentor guiding you, so things like the debian-mentors list are good projects, but I am pretty certain they could benefit from some more helping hands. The other thing which I consider a problem is that and I read from time to time some people don t care for translations. That s bad. Yes, a developer is able to read English, otherwise s/he couldn t write code or participate on the mailinglists. Still, I personally prefer to use a translated application if I have the chance as it s simply easier for me to read in my mother tongue, not only because I am dyslexic, but because my mind still thinks in German and not in English. Yes, I could personally fix that by thinking in English only from now on, but its a quite big problem to convince my family which is not really familiar with tech-stuff to use something if they can t understand what is written on screen. It was hard enough to tell my mother how to write an SMS in a German interface. My phone with English words all over the place would be completely unusable for her despite the fact that my phone is powered by Debian and better for the task from a technical point of view. You are not yet an official Debian developer/maintainer, but you re already perceived in the community as one the most knowledgeable person about APT. It s a great start! What s your advice to other people who want to start contributing to Debian in general, and to APT in particular? It was never a goal in my life to start contributing . My goal was and still is to make my life easier by letting the computer work for me. At some point APT hindered the success of this goal, so it needed to be fixed. I didn t expect to open pandora s box. So, my advice is simple: Just start. Ignore the warning signs telling you that this is not easy. They are just telling you that you do something useful. Only artificial problems are easy. Further more, contribution to APT, dpkg or any other existing package is in no way harder than opening an ITP and working on your own, and it s cooler as you have a similar minded team around you to talk to. APT didn t accept release codenames as target release was one of the first things I fixed. If I had asked someone if that would be a good starting point the answer would have been a clear no , but I didn t search for a good starting point As a kid I can start playing football by just walking on the field and play or I can sit near the field, watching the others play, while analyzing which position would be the best for me to start ruling out one by one as the technical requirements seem too high Oh bicycle kick that sounds complicated I can t do that Julian Andreas Klode is working on a APT replacement, there s also Cupt by Eugene V. Lyubimkin. Both projects started because their authors are not satisfied with APT, they find APT s code difficult to hack partly due to the usage of C++. Do you share their concerns and what s your opinion on those projects? I don t think C++ is a concern in this regard, after all cupt is currently rewritten to C++0x and APT2 started in vala and is now C + glib last time I checked at least. I personally think that something is wrong if we need to advertise an application by saying in which language it is written The major problem for APT is probably that the code is old : APT does its job for more than 12 years now, under different maintainers with an always changing environment around it: so there are lines in APT which date from a time when nobody knew what a Breaks dependency is, that packages can have long descriptions which can be translated or even that package archives can be signed with a gpg key! And yet we take all those for granted today. APT has proven to adapt to these changes in the environment and became in this process very popular. So I don t think the point is near (if it will come at all) that APT can go into retirement as it is completely replaced by something else. The competitors one the other hand have their first 12 years still to go. And it will be interesting to see how they will evolve and what will be the state of the art in 2022 But you asked what I think about the competitors: I prefer the revolution from inside simply because I can see effects faster as more users will profit from it now. Cupt and co. obviously prefer the normal revolution. The goal is the same, creating the best package manager tool, but the chosen way to the goal is different. aptitude and cupt have an interactive resolver for example: that s something I dislike personally, for others that is the ultimate killer feature. cupt reading the same preference file as APT will have a different pinning result, which we should consider each time someone mentions the word drop-in replacement . APT2 isn t much more than the name which I completely dislike currently from a user point of view, so I can t really comment on that. All of them make me sad as each line invested in boilerplate code like configuration file parsing would be in my eyes better be spent in a bugfix or new feature instead, but I am not here to tell anyone what they should do in their free time But frankly, I don t see them really as competitors: I use the tools I use, if other do that too that s good, if not that s their problem. The thing that annoys me really are claims like plan is to remove APT by 2014 as this generates a vi vs. emacs like atmosphere we don t need. If some people really think emacs is a good editor who cares? I really hope we all can drink a beer in 2022 in Milliways, the restaurant at the end of the package universe, remembering the good old 2010 Is there someone in Debian that you admire for his contributions? No, not one, many! Michael Vogt who has nearly the monopole of package manager maintainer by being upstream of APT, synaptics and software center to name only the biggest and still has the time to answer even the dumbest of my questions. Jason Gunthorpe for being one of the initial developers behind deity who I will probably never meet in person beside in old comments and commit logs. Christian Perrier for caring so much about translations. Obey Arthur Liu as a great admin for Debian s participation in Google s Summer of Code. Paul Wise for doing countless reviews on debian-mentors which are a good source of information not only for the maintainer of the package under review. I guess I need to stop here because you asked for just one. So let s end with some big words instead: I am just a little cog in the big debian wheel

---

Thank you to David Kalnischkies for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

3 comments Liked this article? Click here. My blog is Flattr-enabled.

I attended a workshop sponsored by the IAB, W3C, ISOC and MIT on Internet Privacy. The workshop had much more of a web focus than it should have: the web is quite important should certainly cover a majority of the time, but backend issues, network issues, and mobile applications are certainly important too. For me this workshop was an excellent place to think about linkability and correlation of information. When people describe attacks such as using the ordered list of fonts installed in a web browser to distinguish one person from another, it s all too easy to dismiss people who want to solve that attack as the privacy fringe. Who cares if someone knows my IP address or what fonts I use? The problem is that computers are very good at putting data together. If you log into a web site once, and then later come back to that same website, it s relatively easy to fingerprint your browser and determine that it is the same computer. There s enough information that even if you use private browsing mode, clear your cookies and move IP addresses, it s relatively easy to perform this sort of linking. It s important to realize that partially fixing this sort of issue will make it take longer to link two things with certainty, but tends not to actually help in the long-run. Consider the font issue. If your browser returns the set of fonts it has in the order they are installed, then that provides a lot of information. Your fingerprint will look the same as people who took the same OS updates, browser updates and installed the same additional fonts in exactly the same order as you. Let s say that the probability that someone has the same font fingerprint as you is one in a million. For a lot of websites that s enough that you could very quickly be linked. Sorting the list of fonts reduces the information; in that case, let s say your probability of having the same font set as someone else is one in a hundred. The website gets much less information from the fonts. However it can combine that information with timing information etc. It can immediately rule out all the people who have a different font profile. However as all the other people who have the same font fingerprint access the website over time, differences between them and you will continue to rule them out until eventually you are left. Obviously this is at a high level. One important high-level note is that you can t fix these sorts of fingerprinting issues on your own; trying makes things far worse. If you re the only one whose browser doesn t give out a font list at all, then it s really easy to identify you. The big question in my mind now is how much do we care about this linking. Governments have the technology to do a lot with linking. We don t have anything we technical we can do to stop them, so we ll need to handle that with laws. Large companies like Google, Facebook and our ISPs are also in a good position to take significant advantage of linking. Again, though, these companies can be regulated; technology will play a part, especially in telling them what we re comfortable with and what we re not, but most users will not need to physically prevent Google and Facebook from linking their data. However smaller websites are under a lot less supervision than the large companies. Unless you take significant steps, such a website can link all your activities on that website. Also, if any group of websites in that space want to share information, they can link across the websites. I d like to run thought experiments to understand how bad this is. I d like to come up with examples of things that people share with small websites but don t want linked together or alternatively don t want linked back to their identity. Then look at how this information could be linked. However, I m having trouble with these thought experiments because I m just not very privacy minded. I can t think of something that I share on the web that I wouldn t link directly to my primary identity. I certainly can t find anything concrete enough to be able to evaluate how clearly I care to protect it. Helping me out here would be appreciated; if you can think of fairly specific examples. There s lots of important I prefer to keep private like credit card numbers, but there, it s not about linking at all. I can reasonably assume that the person I m giving my credit card number to has a desire to respect my privacy.a

With this weeks lawless governmental attacks on Wikileak and free speech, it has become obvious that PayPal, visa and mastercard can not be trusted to handle money transactions. A blog post from Simon Phipps on bitcoin reminded me about a project that a friend of mine mentioned earlier. I decided to follow Simon's example, and get involved with BitCoin. I got some help from my friend to get it all running, and he even handed me some bitcoins to get started. I even donated a few bitcoins to Simon for helping me remember BitCoin. So, what is bitcoins, you probably wonder? It is a digital crypto-currency, decentralised and handled using peer-to-peer networks. It allows anonymous transactions and prohibits central control over the transactions, making it impossible for governments and companies alike to block donations and other transactions. The source is free software, and while the key dependency wxWidgets 2.9 for the graphical user interface is missing in Debian, the command line client builds just fine. Hopefully Jonas will get the package into Debian soon. Bitcoins can be converted to other currencies, like USD and EUR. There are companies accepting bitcoins when selling services and goods, and there are even currency "stock" markets where the exchange rate is decided. There are not many users so far, but the concept seems promising. If you want to get started and lack a friend with any bitcoins to spare, you can even get some for free (0.05 bitcoin at the time of writing). Use BitcoinWatch to keep an eye on the current exchange rates. As an experiment, I have decided to set up bitcoind on one of my machines. If you want to support my activity, please send Bitcoin donations to the address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b. Thank you!

A few days ago, I was introduces to some students in the robot student assosiation Robotica Osloensis at the University of Oslo where I work, who planned to get their own 3D printer. They wanted to learn from me based on my work in the area. After having a short lunch meeting with them, I offered them to borrow my reprap kit, as I never had time to complete the build and this seem unlike to change any time soon. I look forward to see how this goes. This monday their volunteer driver picked up my kit and drove it to their lab, and tomorrow I am told the last exam is over so they can start work on getting the 3D printer operational. The robotic group have already build several robots on their own, and seem capable of getting the reprap operational. I really look forward to being able to print all the cool 3D designs published on Thingiverse. I even got some 3D scans I got made during Dagen@IFI when one of the groups at the computer science department at the university demonstrated their very cool 3D scanner.

I know ecologists often come with great ideas how to save some random resource. There was a "green" font, which contained holes and was supposed to save ink/toner from printers. It turned out not to be that big saver as the font itself was actually a bit wider than usual... Today I got to know about "new" file format called WWF. Technically it is just a PDF with disabled printing. Well at least it is not a new format, so users of other operating systems (WWF software is available for Mac OS X only right now) are not lost and can use existing tools to view the files (on Linux you can for example use Evince). And if Evince rejects to print this document, just disable these restrictions:

gconftool-2 -s /apps/evince/override_restrictions -t bool 1

Is it really helpful for environment to spend resources on creating yet another useless file format?

Filed under: Life 4 comments Flattr this!

The ProblemMy SE Linux Play Machine has been down for a couple of weeks. I ve changed to a cheaper Internet access plan which also allows me to download a lot more data, but I don t have a static IP address any more and my ISP seems to change the IP a lot more often than I ve experienced in the past (I m used to having a non-static IP address not change for months rather than hours). So I needed to get Dynamic DNS working. Naturally I wasn t going to use one of the free or commercial Dynamic DNS solutions, I prefer to do things myself. So my Play Machine had to remain offline until I fixed this.The Solutiondyn IN NS ns.sws.net.au.
IN NS othello.dycom.com.au.
play IN CNAME play.dyn.coker.com.au.The first thing I did was to create a separate zone file, I put the above records in my main zone file to make play.coker.com.au be a CNAME for play. and dyn.coker.com.au is a dynamic domain. I have SE Linux denying BIND the ability to write to the primary zone file for my domain to make it slightly more difficult for an attacker to insert fake DNS records (they could of course change the memory state of BIND to make it serve bogus data). The dynamic zone file is stored where BIND can write it and therefore a BIND exploit could easily replace it (but such an attack is out of the scope of the Play Machine project so don t get any ideas).Another reason for separating the dynamic data is that BIND journals changes to a dynamic zone and therefore if you want to manually edit it you have to delete the journal, stop BIND, edit the file, and then restart BIND. One of the things that interests me is setting up dynamic DNS for some of my clients, as a constraint is that my client must be able to edit the zone file themself I have to keep the editing process for the main zone file relatively simple.dnssec-keygen -a hmac-md5 -b 128 -n host foo-dyn.keyI used the above command to create the key files. It created Kfoo-dyn.key.+X+Y.key and Kfoo-dyn.key.+X+Y.private where X and Y are replacements for numbers that might be secret.key "foo" algorithm hmac-md5; secret "XXXXXXXX"; ;
zone "dyn.coker.com.au"
type master;
file "/var/cache/bind/dyn.coker.com.au";
allow-update key "foo"; ;
allow-transfer key ns; ;
;I added the above to the BIND configuration to create the dynamic zone and allow it to be updated by this key. The value which I replaced with XXXXXXX in this example came from Kfoo-dyn.key.+X+Y.key. I haven t found any use for the .private file in this mode of operation. Please let me know if I missed something.Finally I used the following shell script to take the IP address from the interface that is specified on the command-line and update the DNS with it. I chose a 120 second timeout because i will sometimes change IP address often and because the system doesn t get enough hits for anyone to care about DNS caching.#!/bin/bash
set -e
IP=$(ip addr list $1 sed -n -e "s/\/.*$//" -e "s/^.*inet //p")
nsupdate -y foo:XXXXXXXX << END
update delete play.dyn.coker.com.au A
update add play.dyn.coker.com.au 120 A $IP
send
ENDTodoIt is supposed to be possible to use the -k option to nsupdate to specify a file containing the key. Reading a key file prevents another process on the same system from from using ps to discover the key which is a good security benefit. But as the security of my dynamic DNS isn t that important to me and I have SE Linux restricting which domains can see all processes in ps output this isn t a big concern for me.I would appreciate any suggestions on how to get -k working.

LWN has a recent article titled The dark side of open source conferences which concerns sexual assault at conferences [1].There are a significant number of comments with attempts to derail the discussion in ways that can really only be interpreted as attacks on Autistic people. The claims seem to be that the problem is not violent sexual assault at conferences, but guys on the Autism Spectrum who hit on girls when they don t want it. Naturally no supporting evidence was made for such claims. But that doesn t stop the discussion which has a logical end-point of excluding people like me from conferences.The IronyI think it s ironic that those who are making claims about what they call empathy disorders have failed to be Empathic by not realising the following things:

• Women who start a discussion about serious sexual assault probably aren t going to be happy if someone starts talking about chat-up attempts. While unwanted chat-up attempts are unpleasant they are in a different category and mixing them seems to be diminishing the significance of violent attack.
• Most members of a minority group (in this case women in the Free Software community) probably don t want discussions of how to help their group diverted by discussions that attack another minority group. What looks a lot like a divide and conquer attack against minority groups isn t going to be appreciated by members of either group and probably members of other minority groups who see what s happening aren t going to like it.
• There is no reason why people should require significant exposure to members of a minority group to treat them in a decent manner. The claim that we need more women at conferences so that men can get used to them and not treat them badly is ridiculous. Among the many stupid aspects of that idea is forgetting the fact that women comprise 52% of the population in first-world countries and we all deal with women every day. Women at IT conferences are not fundamentally different from women in the rest of society.
• When most people interpret your writing in a way other than what you intended it seems to be a reasonable assumption that you failed to explain things clearly. Telling everyone who disagrees to get therapy is unlikely to help convey your point. Telling people to get therapy is particularly likely to get a bad reaction if you are discussing something that actually involves dealing with psychologists.
• When there is an obvious resource on the Internet relating to a topic it s a good idea to read it instead of just making stuff up. Failing to do so will be taken as an indication that you aren t trying to be Empathic. The Wikipedia section on Autism Spectrum Disorders has a lot of useful information [2]. Please read it before making comments about Autism and Empathy.
• Having to be told how other people feel is not inherently a sign of a lack of Empathy. Asking people how they feel because you can t work it out is a sign of Empathy as it indicates an acknowledgement that other people have different emotions and you probably don t understand them all. The people who know that they can t understand other people and listen when told are probably better than average when it comes to Empathy.
• Finally what people say about themselves and their own experiences should be taken seriously.

Some Final PointsInstead of talking about how some other people should be more Empathic it seems that a better idea would be to try and demonstrate Empathy. Set a positive example.I did a Google search for Empathy Disorder and found this interesting article [3]. It s about how Neuro-Typical people (people who aren t on the Autism Spectrum) can learn to be more Empathic, it s probably more relevant to the issue of Empathy in the free software community than discussions of Autism.I think that the experience of a lot of people on the Autism Spectrum is similar to mine. It s not that we can t work things out it s just that it takes a lot longer. For things that can t be worked out in real time we have rules based on past experience. Naturally the rules include don t touch people and don t try to chat up women at conferences .Here is a link to my previous post on this topic [4].Update: I ve clarified some of the writing and added an extra point about having to be told how others feel.

• [1] https://lwn.net/Articles/417952/
• [2] http://en.wikipedia.org/wiki/Empathy#Autism_spectrum_disorders
• [3] http://articles.cnn.com/2008-06-18/living/o.empathy_1_empathy-friend-disorder
• [4] http://etbe.coker.com.au/2010/12/03/aspie-social-skills-free-software/

Richard Tibbetts has called me out for conflating Web 2.0 startups with all startups in my recent blog posting, Google has a problem retaining great engineers? Bullcrap . His complaint was that I was over generalizing from Web 2.0 startups to all startups. He s right, of course. The traditional technology startup by definition does have a large amount technology work that needs to be done, in addition to the business development work. However, things have changed a lot even for technology startups. Consider a company like Sequent Computer Systems, which started in 1983. At the time the founders had a key idea, which was to use multiple commodity intel CPU s to create first SMP, and then later, NUMA minicomputers. But in order to do that, they had to design, build and manufacture a huge mount of hardware, as well as develop a whole new Unix-derived operating system, just to bring that core idea to market. These days, the founder or founders will have a core idea, which they will hopefully patent, to prevent competitors from replicating their work, just as before. However, these days there is a huge selection of open source software so there is much less technology that needs to be re-developed / re-invented in order to bring that idea to market. From Linux and BSD at the operating system level, to databases like MySQL, Apache web servers, etc., there is an awful lot for the startup to chose from. This is all goodness, of course. But it means that most of the technology developed in a typical startup will tend to be focused on supporting the core idea that was developed by the founder. If a company is focused on databases, they probably won t be interested in supporting me to do my file system work. Why should they? There s lots of open source file systems they can use; one of them will probably meet their needs. So while it s obvious that you can do technology at large variety of companies, of different sizes, I don t think it s arrogance to say that there are certain types of technology that can only be done at Google, or maybe at a very small subset of other companies. I m pretty confident, for example, that Google has the world s largest number of computers in its various data centers around the world. That means there are certain things that don t make sense at other companies, but which absolutely makes sense at our scale. In addition, Google s business model means allows us to pursue open source projects such as Chrome and Android that wouldn t make sense at other companies. With Android in particular, it means that multiple firmware updates for any given handset model, if it causes people to use the web more and drive more advertising revenue, makes sense and so it s something we can pursue in comparison to Nokia, which gets its revenue from hardware sales, so a firmware update that extends the life of a handset is actually a bad thing for them; better to force you to upgrade an handset every year or two. So I think Richard misunderstood me if he thought I was trying to make the argument that Google is the only place where you can do interesting technical work. That s obviously not true, of course. But I do think there are many examples of technical work which don t make business sense to do at smaller companies, and startups in particular have to be very much focused on bringing the founder s idea to market, and all else has to be subordinated to that goal. And one of the most interesting developments is how the combination of commoditized and standardized hardware, and comoditized software in the form of open source, has changed the game for startups. For most startups, though, open source software is something that they will use, but not necessarily develop except in fairly small ways. Many of the economic arguments in favor of releasing code as open source, and dedicating a significant fraction of an engineer s time to serve as a OSS project maintainer or kernel subsystem maintainer, are ones that make much more sense at a very large company like Google or IBM. That s not because startups are evil, or deficient in any way; just the economic realities that at a successful startup, everything has to be subordinated to the central goal of proving that they have a sustainable, scalable business model and that they have a good product/market fit. Everything else, and that includes participating in an open source community, is very likely a distraction from that central goal. No related posts.

Richard Tibbetts has called me out for conflating Web 2.0 startups with all startups in my recent blog posting, Google has a problem retaining great engineers? Bullcrap . His complaint was that I was over generalizing from Web 2.0 startups to all startups. He s right, of course. The traditional technology startup by definition does have a large amount technology work that needs to be done, in addition to the business development work. However, things have changed a lot even for technology startups. Consider a company like Sequent Computer Systems, which started in 1983. At the time the founders had a key idea, which was to use multiple commodity intel CPU s to create first SMP, and then later, NUMA minicomputers. But in order to do that, they had to design, build and manufacture a huge mount of hardware, as well as develop a whole new Unix-derived operating system, just to bring that core idea to market. These days, the founder or founders will have a core idea, which they will hopefully patent, to prevent competitors from replicating their work, just as before. However, these days there is a huge selection of open source software so there is much less technology that needs to be re-developed / re-invented in order to bring that idea to market. From Linux and BSD at the operating system level, to databases like MySQL, Apache web servers, etc., there is an awful lot for the startup to chose from. This is all goodness, of course. But it means that most of the technology developed in a typical startup will tend to be focused on supporting the core idea that was developed by the founder. If a company is focused on databases, they probably won t be interested in supporting me to do my file system work. Why should they? There s lots of open source file systems they can use; one of them will probably meet their needs. So while it s obvious that you can do technology at large variety of companies, of different sizes, I don t think it s arrogance to say that there are certain types of technology that can only be done at Google, or maybe at a very small subset of other companies. I m pretty confident, for example, that Google has the world s largest number of computers in its various data centers around the world. That means there are certain things that don t make sense at other companies, but which absolutely makes sense at our scale. In addition, Google s business model means allows us to pursue open source projects such as Chrome and Android that wouldn t make sense at other companies. With Android in particular, it means that multiple firmware updates for any given handset model, if it causes people to use the web more and drive more advertising revenue, makes sense and so it s something we can pursue in comparison to Nokia, which gets its revenue from hardware sales, so a firmware update that extends the life of a handset is actually a bad thing for them; better to force you to upgrade an handset every year or two. So I think Richard misunderstood me if he thought I was trying to make the argument that Google is the only place where you can do interesting technical work. That s obviously not true, of course. But I do think there are many examples of technical work which don t make business sense to do at smaller companies, and startups in particular have to be very much focused on bringing the founder s idea to market, and all else has to be subordinated to that goal. And one of the most interesting developments is how the combination of commoditized and standardized hardware, and comoditized software in the form of open source, has changed the game for startups. For most startups, though, open source software is something that they will use, but not necessarily develop except in fairly small ways. Many of the economic arguments in favor of releasing code as open source, and dedicating a significant fraction of an engineer s time to serve as a OSS project maintainer or kernel subsystem maintainer, are ones that make much more sense at a very large company like Google or IBM. That s not because startups are evil, or deficient in any way; just the economic realities that at a successful startup, everything has to be subordinated to the central goal of proving that they have a sustainable, scalable business model and that they have a good product/market fit. Everything else, and that includes participating in an open source community, is very likely a distraction from that central goal. No related posts.

On a private mailing list there was some discussion, and I had an outburst. Summarising very roughly, someone said bad things about their competitor, without (at first) offering any justification. Here's a version of my outburst, edited to remove references to the actual topic and identifyable people. I thought it might amuse me in fifty years or so to see how full of myself I was.

---

Oh dear. I have no experience with this service. Or the person saying bad things about them. I am entirely ignorant about everything.[0] I am not biased towards or against any of them. There's another point I would like to raise. Or rather, an opinion I would like to express. I feel very strongly about this. There are two reasons to not show respect. The proper reason is because the target has thoroughly proven themselves to not deserve it, by their words and action, or lack of them. The wrong reason is anything related to anyone else than the target: your own low self-esteem, your fears, your evil schemes, your own bad day. Even when the right reason exists, it is a sign of respect to the audience to be respectful towards someone who does not need it: it avoids shoving the discussion down the drain. Showing respect, even when undeserved, indicates strength and moral high ground. (Please note the difference between showing respect to and agreeing with or approving of the other party.) To me, as a total outsider, these remarks about the service give a bad impression, but not of the service. It may well be that they are a scammy corporation that gives lousy service. However, nothing in this discussion has given me a reason to think so. They may be well-known in your circle. They may be well-known for their bad quality and bad service. But the rest of us don't know that. When you denigrate them, you just look like a whiner[1]. Especially someone who wants to promote their own service over that of a competitor, looks bad, because they're saying bad things about them, without giving any reasons. If you're going to be criticising your competition, you need to be very careful of how you do it, or you'll look bad instead. Or also, since it's easy to make everyone in a discussion look bad. If you'd like promote your company you can explain its virtues and benefits and the good deals it provides. You don't need to denigrate anyone else. If you'd like to get people to avoid your competitor, because they are bad or evil, then show why they are bad, or evil, with examples and references. [0] This is a true statement in every context. [1] After decades of experience as a whiner, I have a finely tuned sensor for detecting fellow whiners.

The relationship between Debian and Ubuntu has been the subject of many vigorous debates over the years, ever since Ubuntu s launch in 2004. Six years later, the situation has improved and both projects are communicating better. The Natty Narwhal Ubuntu Developer Summit (UDS) featured like all UDS for more than 2 years a Debian Health Check session where current cooperation issues and projects are discussed. A few days after that session, Lucas Nussbaum gave a talk during the mini-Debconf Paris detailing the relationship between both projects, both at the technical and social level. He also shared some concerns for Debian s future and gave his point of view on how Debian should address them. Both events give valuable insights on the current state of the relationship. Lucas Nussbaum s Debian-Ubuntu talk Lucas started by introducing himself. He s an Ubuntu developer since 2006 and a Debian developer since 2007. He has worked to improve the collaboration between both projects, notably by extending the Debian infrastructure to show Ubuntu-related information. He attended conferences for both projects (Debconf, UDS) and has friends in both communities. For all of these reasons, he believes himself to be qualified to speak on this topic. Collaboration at the technical level He then quickly explained the task of a distribution: taking upstream software, integrating it in standardized ways, doing quality assurance on the whole, delivering the result to users, and assuring some support afterward. He pointed out that in the case of Ubuntu, the distribution has one special upstream: Debian. Indeed Ubuntu gets most of its software from Debian (89%), and only 7% are new packages coming from other upstream projects (the remaining 4% are unknown, they are newer upstream releases of software available in Debian but he was not able to find out whether the Debian packaging had been reused or not). From all the packages imported from Debian, 17% have Ubuntu-specific changes. The reasons for those changes are varied: bugfixes, integration with Launchpad/Ubuntu One/etc., or toolchain changes. The above figures are based on Ubuntu Lucid (10.04) while excluding many Ubuntu-specific packages (language-pack-*, language-support-*, kde-l10n-*, *ubuntu*, *launchpad*). The different agendas and the differences in philosophy (Debian often seeking perfect solutions to problems; Ubuntu accepting temporary suboptimal workarounds) also explain why so many packages are modified on the Ubuntu side. It s simply not possible to always do the work in Debian first. But keeping changes in Ubuntu requires a lot of work since they merge with Debian unstable every 6 months. That s why they have a strong incentive to push changes to upstream and/or to Debian. There are 3 channels that Ubuntu uses to push changes to Debian: they file bug reports (between 250 to 400 during each Ubuntu release cycle), they interact directly with Debian maintainers (often the case when there s a maintenance team), or they do nothing and hope that the Debian maintainer will pick up the patch directly from the Debian Package Tracking System (it relays information provided by patches.ubuntu.com). Lucas pointed out that those changes are not the only thing that Debian should take back. Ubuntu has a huge user base resulting in lots of bug reports sitting in Launchpad, often without anyone taking care of them. Debian maintainers who already have enough bugs on their packages are obviously not interested in even more bugs, but those who are maintaining niche packages, with few reports, might be interested by the user feedback available in Launchpad. Even if some of the reports are Ubuntu-specific, many of them are advance warnings of problems that will affect Debian later on, when the toolchain catches up with Ubuntu s aggressive updates. To make this easier for Debian maintainers, Lucas improved the Debian Package Tracking System so that they can easily get Ubuntu bug reports for their packages even without interacting with Launchpad. Human feelings on both sides Lucas witnessed a big evolution in the perception of Ubuntu on the Debian side. The initial climate was rather negative: there were feelings of its work being stolen, claims of giving back that did not match the observations of the Debian maintainers, and problems with specific Canonical employees that reflected badly on Ubuntu as a whole. These days most Debian developers find something positive in Ubuntu: it brings a lot of new users to Linux, it provides something that works for their friends and family, it brings new developers to Debian, and it serves as a technological playground for Debian. On the Ubuntu side, the culture has changed as well. Debian is no longer so scary for Ubuntu contributors and contributing to Debian is The Right Thing to do. More and more Ubuntu developers are getting involved in Debian as well. But at the package level there s not always much to contribute, as many bugfixes are only temporary workarounds. And while Ubuntu s community follows this philosophy, Canonical is a for-profit company that contributes back mainly when it has compelling reasons to do so. Consequences for Debian In Lucas s eyes, the success of Ubuntu creates new problems. For many new users Linux is a synonym for Ubuntu, and since much innovation happens in Ubuntu first, Debian is overshadowed by its most popular derivative. He goes as far as saying that because of that Debian becomes less relevant . He went on to say that Debian needs to be relevant because the project defends important values that Ubuntu does not. And it needs to stay as an independent partner that filters what comes out of Ubuntu, ensuring that quality prevails in the long term. Fixing this problem is difficult, and the answer should not be to undermine Ubuntu. On the contrary, more cooperation is needed. If Debian developers are involved sooner in Ubuntu s projects, Debian will automatically get more credit. And if Ubuntu does more work in Debian, their work can be showcased sooner in the Debian context as well. The other solution that Lucas proposed is that Debian needs to communicate on why it s better than Ubuntu. Debian might not be better for everybody but there are many reasons why one could prefer Debian over Ubuntu. He listed some of them: Debian has better values since it s a volunteer-based project where decisions are made publicly and it has advocated the free software philosophy since 1993. On the other hand, Ubuntu is under control of Canonical where some decisions are imposed, it advocates some proprietary web services (Ubuntu One), the installer recommends adding proprietary software, and copyright assignments are required to contribute to Canonical projects. Debian is also better in terms of quality because every package has a maintainer who is often an expert in the field of the package. As a derivative, Ubuntu does not have the resources to do the same and instead most packages are maintained on a best effort basis by a limited set of developers who can t know everything about all packages. In conclusion, Lucas explained that Debian can neither ignore Ubuntu nor fight it. Instead it should consider Ubuntu as a chance and should leverage it to get back in the center of the FLOSS ecosystem . The Debian health check UDS session While this session has existed for some time, it s only the second time that a Debian Project Leader was present at UDS to discuss collaboration issues. During UDS-M (the previous summit), this increased involvement from Debian was a nice surprise to many. Stefano Zacchiroli the Debian leader collected and shared the feedback of Debian developers and the session ended up being very productive. Six months later is a good time to look back and verify if decisions made during UDS-M (see blueprint) have been followed through. Progress has been made On the Debian side, Stefano set up a Derivatives Front Desk so that derivative distributions (not just Ubuntu) have a clear point of contact when they are trying to cooperate but don t know where to start. It s also a good place to share experiences among the various derivatives. In parallel, a #debian-ubuntu channel has been started on OFTC (the IRC network used by Debian). With more than 50 regulars coming from both distributions, it s a good place for quick queries when you need advice on how to interact with the distribution that you re not familiar with. Ubuntu has updated its documentation to prominently feature how to cooperate with Debian. For example, the sponsorship process documentation explains how to forward patches both to the upstream developers and to Debian. It also recommends ensuring that the patch is not Ubuntu-specific and gives some explanation on how to do it (which includes checking against a list of common packaging changes made by Ubuntu). The Debian Derivative Front Desk is mentioned as a fallback when the Debian maintainer is unresponsive. While organizing Ubuntu Developer Week, Ubuntu now reaches out to Debian developers and tries to have sessions on working with Debian . Launchpad has also been extended to provide a list of bugs with attached patches and that information has been integrated in the Debian Package Tracking system by Lucas Nussbaum. Still some work to do Some of the work items have not been completed yet: many Debian maintainers would like a simpler way to issue a sync request (a process used to inject a package from Debian into Ubuntu). There s a requestsync command line tool provided by the ubuntu-dev-tools package (which is available in Debian) but it s not yet usable because Launchpad doesn t know the GPG keys of Debian maintainers. Another issue concerns packages which are first introduced in Ubuntu. Most of them have no reason to be Ubuntu-specific and should also end up in Debian. It has thus been suggested that people packaging new software for Ubuntu also upload them to Debian. They could however immediately file a request for adoption (RFA) to find another Debian maintainer if they don t plan to maintain it in the long term. If Ubuntu doesn t make this effort, it can take a long time until someone decides to reintegrate the Ubuntu package into Debian just because nobody knows about it. This represents an important shift in the Ubuntu process and it s not certain that it s going to work out. As with any important policy change, it can take several years until people are used to it. Both issues have been rescheduled for this release cycle, so they re still on the agenda. This time the UDS session was probably less interesting than the previous one. Stefano explained once more what Debian considers good collaboration practices: teams with members from both distributions, and forwarding of bugs if they have been well triaged and are known to apply to Debian. He also invited Ubuntu to discuss big changes with Debian before implementing them. An interesting suggestion that came up was that some Ubuntu developers could participate in Debcamp (one week hack-together before Debconf) to work with some Debian developers, go through Ubuntu patches, and merge the interesting bits. This would nicely complement Ubuntu s increased presence at Debconf: for the first time, community management team member Jorge Castro was at DebConf 10 giving a talk on collaboration between Debian and Ubuntu. There was also some brainstorming on how to identify packages where the collaboration is failing. A growing number of Ubuntu revisions (identified for example by a version like 1.0-1ubuntu62) could indicate that no synchronization was made with Debian, but it would also identify packages which are badly maintained on the Debian side. If Ubuntu consistently has a newer upstream version compared to Debian, it can also indicate a problem: maybe the person maintaining the package for Ubuntu would be better off doing the same work in Debian directly since the maintainer is lagging or not doing their work. Unfortunately this doesn t hold true for all packages since many Gnome packages are newer in Ubuntu but are actively maintained on both sides. Few of those discussions led to concrete decisions. It seems most proponents are reasonably satisfied with the current situation. Of course, one can always do better and Jono Bacon is going to ensure that all Canonical teams working on Ubuntu are aware of how to properly cooperate with Debian. The goal is to avoid heavy package modifications without coordination. Conclusion The Debian-Ubuntu relationships used to be a hot topic, but that s no longer the case thanks to regular efforts made on both sides. Conflicts between individuals still happen, but there are multiple places where they can be reported and discussed (#debian-ubuntu channel, Derivatives Front Desk at derivatives@debian.org on the Debian side or debian@ubuntu.com on the Ubuntu side). Documentation and infrastructure are in place to make it easier for volunteers to do the right thing. Despite all those process improvements, the best results still come out when people build personal relationships by discussing what they are doing. It often leads to tight cooperation, up to commit rights to the source repositories. Regular contacts help build a real sense of cooperation that no automated process can ever hope to achieve.

This article was first published in Linux Weekly News. You can get my monthly summary of the Debian/Ubuntu news, all you have to do is to click here to subscribe to my free newsletter.

5 comments Liked this article? Click here. My blog is Flattr-enabled.

As a first step on the road to evaluating different configuration management systems, I set up two VMs to test out puppet. Puppet is conveniently packaged within Debian so client and server installation was very simple. By default, puppet clients will look for a host named puppet on the local network, so by naming the server sensibly the two started talking to each other. puppetca --sign [client hostname] on the server started the ball rolling. My first test configuration was to add a user to the sudoers file. With CF Engine 2, I would have achieved this by using editfiles to check for a regular expression and insert a line if necessary. I stumbled across some CF Engine-style definitions for puppet in a Debian Administration article (taken originally from puppet's own wiki, which have since been moved/deleted) and achieved it using

append_if_no_such_line   sudo:
  file => "/etc/sudoers",
  line => "jon ALL=(ALL) ALL"
 

I'm a little concerned that this approach to problem solving is no better than CF Engine 2, and in fact worse than how I'd do it with CF Engine 2: if the host had a line with a different quantity of whitespace, such as jon ALL=(ALL) ALL, it won't match exactly and a duplicate line will be appended, whereas with CF Engine 2 I would have checked for something like ^jon \+ALL=(ALL) \+ALL$. I set up my work laptop as another test node, and used puppet to install some packages that I commonly use:

class jon-desktop  
  package   ['icedove', 'git-core', 'vim-gnome', 'vinagre', 'build-essential',
             'devscripts', 'subversion', 'git-buildpackage', 'mutt',
             'offlineimap', 'ascii', 'gitk', 'chromium-browser'
            ]: ensure => installed  
 

This worked fine, but is a little verbose and the package list is awkward to maintain buried inside the puppet syntax. I spent some time trying to see whether I could populate a puppet list based on the contents of a file: then I could list the packages one-per-line in a separate file. I haven't found a way of doing that yet. Using my laptop as a second node also exposed some interesting puppet client behaviour. Once I went off-site, the laptop tried to look up puppet on foreign networks (such as my home network). I would have assumed that, once the strong SSL association had been made, the client would try to connect to the FQDN of the server. This probably isn't a serious problem in practice, as we probably won't need to rely on nodes off-site talking to the configuration server, but I hope that the client doesn't trust any host that happens to answer to puppet... I fixed this by hacking my hosts file to include an entry for puppet. I could maintain this hack using Puppet itself, which would protect me against stale definitions if any renumbering takes place. I could also have added a line

server = puppet.fqdn.example.org

in /etc/puppet/puppet.conf, which is a more robust way of solving the problem.

This week I had to write a little tool that would collect input on different channels, via socket / netcat, via a http server, .... Calls for a parralel Design -- Maybe a good place to write something real in Erlang. While erlang itself was really nice to write -- I do like Prolog as well as the bunch of functional languages -- doing networking in erlang seems a bit special, the interfaces just aren't thin wrappers around the libc stuff. Getting a Socket Text interface What sounds like a easy challenge to accomplish was actually harder than expected. All I found was some way to pass binaries representing erlang code over a socket and evaluating it remotle. While it's nice that such things are as easy to do as they are it doesn't help me with my task of moving simple strings.

start() ->
     ok, Listen  = gen_tcp:listen(51622, [binary,  packet, 0 ,
                                          reuseaddr, true ,
                                          active, true ]),
    spawn(fun () -> accept_loop(Listen) end).
accept_loop(Listen) ->
     ok, Socket  = gen_tcp:accept(Listen),
    accept_loop(Listen),
    handle(Socket)
handle(Socket) ->
    receive
         tcp, Socket, Bin  ->
            io:format("~p~n", binary_to_list(Bin));
         tcp_closed, Socket  ->
            Buffer
    end.

So the socket is now up and receives text as wanted. However, as we are already runnign a parralel program it would be nice to be hable to handle multiple socket connections in parralel right? For that we just need to add a spawn() at the right place. The right place is not the handle(Socket) but the accept_loop(Listen) because the process that called accept will receive all the tcp messages. This last part was quite obvious after finding the documentation of the active, _ properties for the socket. always here means that you'll receive all data from the socket as erlang Messages, once delivers one package and waits until it is activated again and false requires calling a method -- this would have been possible as well when forking handle(Socket). The web server Ok we also want a webserver. We do not want to run some webapplication inside appache or so, just do some post/get and simple pages. Erlang here provides a built-in httpd with some mod_esi that calls some function depending on the URL used. It doesn't do anything fancy like templating or DB backends or stuff, just taking strings and building the http answers. Unfortunately there are no examples around and basically noone seems to be using this combination (apart from some hacks like mine probably). So as I needed to get some additional information into the handler function (a Pid to connect to some service), I, as a novice, just couldn't find a way. Asking on IRC the solution was rather simple: Just using erlang's process registry. For more complex stuf gproc might prove usefull here. Result I guess I've got a huge step farther in programming erlang now. The manpages are easily found by your search engine -- for python I have to either walk through the (well structured) online documentation or search for the right link in the search results, for erlang they're typically first. Also Joe Armstrong's Books as proven usefull. The most dificult part probably is getting around all the nice extras you can do (transfering functions over sockets et al) and find out how to do the thing you need.

This is the second post about distcheck. I want to give a quick overview of the differences between edos-distcheck and the new version. First despite using the same sat solver and encoding of the problem, Distcheck has been re-written from scratch. Dose2 has several architectural problems and not very well documented. Adding new features had become too difficult and error-prone, so this was a natural choice (at least for me). Hopefully Dose3 will survive the Mancoosi project and provide a base for dependency reasoning. The framework is well documented and the architecture pretty modular. It's is written in ocaml, so sadly, I don't expect many people to join the development team, but we'll be very open to it. These are the main differences with edos-debcheck .

Performances distcheck is about two times faster than edos-debcheck (from dose2), but it is a "bit" slower then debcheck (say the original debcheck), that is the tool wrote by Jerome Vouillon and that was then superseded in debian by edos-debcheck. The original debcheck was a all-in-one tool that did the parsing, encoding and solving without converting the problem to any intermediate format. distcheck trades a bit of speed for generality. Since it is based on Cudf, it can handle different formats and can be easily adapted in a range of situation just by changing the encoding of the original problem to cudf. Below there are a couple of test I've performed on my machine (debian unstable). The numbers speak alone.

$time cat tmp/squeeze.packages edos-debcheck -failures > /dev/null
Completing conflicts... * 100.0%
Conflicts and dependencies... * 100.0%
Solving * 100.0%

real 0m19.515s
user 0m19.193s
sys 0m0.276s

$time ./distcheck.native -f deb://tmp/squeeze.packages > /dev/null

real 0m10.859s
user 0m10.669s
sys 0m0.172s

Input The second big difference is about different input format. In fact, at the moment, we have two different tools in debian, one edos-debcheck and the other edos-rpmcheck. Despite using the same underlying library these two tools have different code bases. distcheck basically is a multiplexer that convert different inputs to a common format and then uses it (agnostically) to solve the installation problem. It can be called in different ways (via symlinks) to behave similarly to its predecessors. At the moment we are able to handle 5 different formats

1. deb:// Packages 822 format for debian based distributions
2. hdlist:// a binary format used by rpm based distribution
3. synth:// a simplified format to describe rpm based package repositories
4. eclipse:// a 822 based format that encoded OSGi plugings metadata
5. cudf:// the native cudf format

distcheck handles gz and bz2 compressed file transparently . However if you care about performances, you should decompress your input file first and the parse it with distcheck and it often takes more time to decompress the file on the fly that run the installability test itself. There is also an experimental database backend that is not compiled by default at them moment.

Output Regarding the output, I've already explained the main differences in an old post. As a quick reminder, the old edos-debcheck had two output options. The first is a human readable - unstructured output - that was a handy source of information when running the tool interactively. The second was a xml based format (without a dtd or a schema, I believe) that was used for batch processing. distcheck has only one output type in the YAML format that aims to be human and machine readable. Hopefully this will cater for both needs. Moreover, just recently I've added the output of distcheck a summary of who is breaking what. The output of edos-debcheck was basically a map of packages to the reasons of the breakage. In addition to this information distcheck gives also a maps between reason (a missing dependency or a conflict) to the list of packages that are broken by this problem.This additional info is off by default, but I think it can be nice to know what is the missing dependency that is responsible for the majority of problems in a distribution... For example, calling distcheck with --summary :

$./distcheck.native --summary deb://tests/sid.packages
backgroud-packages: 29589
foreground-packages: 29589
broken-packages: 143
missing-packages: 138
conflict-packages: 5
unique-missing-packages: 52
unique-conflict-packages: 5
summary:
-
missing:
missingdep: libevas-svn-05-engines-x (>= 0.9.9.063)
packages:
-
package: enna-dbg
version: 0.4.0-4
architecture: amd64
source: enna (= 0.4.0-4)
-
package: enna
version: 0.4.0-4
architecture: amd64
source: enna (= 0.4.0-4)
-
missing:
missingdep: libopenscenegraph56 (>= 2.8.1)
packages:
-
package: libosgal1
version: 0.6.1-2+b3
architecture: amd64
source: osgal (= 0.6.1-2)
-
package: libosgal-dev
version: 0.6.1-2+b3
architecture: amd64
source: osgal (= 0.6.1-2)

Below I give a small example of the edos-debcheck output compared to the new yaml based output.

$cat tests/sid.packages edos-debcheck -failures -explain
Completing conflicts... * 100.0%
Conflicts and dependencies... * 100.0%
Solving * 100.0%
zope-zms (= 1:2.11.1-03-1): FAILED
zope-zms (= 1:2.11.1-03-1) depends on missing:
- zope2.10
- zope2.9
zope-tinytableplus (= 0.9-19): FAILED
zope-tinytableplus (= 0.9-19) depends on missing:
- zope2.11
- zope2.10
- zope2.9
...

And an extract from the distcheck output (the order is different. I cut and pasted parts of the output here...)

$./distcheck.native -f -e deb://tests/sid.packages
report:
-
package: zope-zms
version: 1:2.11.1-03-1
architecture: all
source: zope-zms (= 1:2.11.1-03-1)
status: broken
reasons:
-
missing:
pkg:
package: zope-zms
version: 1:2.11.1-03-1
architecture: all
missingdep: zope2.9 zope2.10
-
package: zope-tinytableplus
version: 0.9-19
architecture: all
source: zope-tinytableplus (= 0.9-19)
status: broken
reasons:
-
missing:
pkg:
package: zope-tinytableplus
version: 0.9-19
architecture: all
missingdep: zope2.9 zope2.10 zope2.11
...

Future The roadmap to release version 1.0 of distcheck is as follows:

1. add background and foreground package selection. This feature will allow the use to specify a larger universe (background packages), but check only a subset of this universe (foreground packages). This should allow users to select packages using grep-dctrl and then pipe them to discheck . At the moment we can select individual packages on the command line or we can use expression like bash (<= 2.7) to check all version of bash in the universe with version greater than 2.7.
2. code cleanup and a bit of refactoring between distcheck and buildcheck (that is a frontend for distcheck that allow us to report broken build dependencies)
3. consider essential packages while performing the installation test. Here there are few things we have to understand, but the idea would be to detect possible problems related the implicit presence of essential packages in the distribution. At the moment, distcheck performs the installation test in the empty universe, while ideally, the universe should contain all essential packages.
4. finish the documentation. The effort in underway and we hope to finalize shortly to release the debian package in experimental.