I’ve been remiss in pointing my readers recently to Zachary Schrag’s Institutional Review Blog, which patiently documents the overreach and misapplication of federal regulations regarding the protection of human subjects to the social and behavioral sciences, including research that by federal regulation is exempt from review by IRBs. While Schrag is cautiously optimistic that the new head of the Office of Human Research Protections will be an improvement, the continued domination of the process at most levels by biomedical researchers—along with the general sense that, as Schrag notes, “researchers cannot be trusted to apply the exemptions themselves”—is still troubling to those of us who want to conduct human subjects research, particularly secondary data analysis. Technically speaking (even though I dare say most social scientists observe this requirement in the breach), even the analysis of secondary data collected by others and fully anonymized before we see it (e.g. use of data such as the General Social Survey, American National Election Studies series, the Eurobarometer series, etc.) requires IRB oversight and approval beforehand.

now that ps3-utils is packaged, and it seems spufs didn't take much to get to work, i'd like to turn back to the installation procedure, which was admittedly manual and not exactly as "seamless" as i think it could be. the first step as i see it is to get the bootloader working both generally and also specifically for the installer. by "working", i mean the following:

• that it can be used to start the installer
• that it can be used to boot the system, of course
• that it can be built within debian

the third of these being the most complicated, as will be detailed below. how a linux (kboot) "otheros" bootloader works on the ps3 okay, so i'm not intimately familiar with the powerpc architecture, nor with all the details of cross-compilation, but this is as much as i can tell.

• the file "otheros.bld" which you use to start the installation is a compressed linux image with a built in ramdisk initrd.
• it is a custom generated image made by kboot, which includes specialized versions of busybox and kexec-tools among others.
• it is compiled with a locally bootstrapped compiler and uclibc library.
• the initrd boots to a kboot script/prompt, which will then find and load the "real" or "system installed" kernel off of the disk.
• you can also specify a different kernel to use, different boot params, etc.
• you can also optionally get to a limited shell environment to poke around.

how kboot can be used on the installer including support for this "otheros.bld" bootloader on the installer is fairly easy from a technical point of view, whether it's a usb disk or netinst cdrom image. basically, in either case the media should have a directory called "ps3/otheros", in which the file is placed and named "otheros.bld". when running the PS3 in game-os mode, this is where it looks when you tell it to "scan for otheros". how kboot can be used/maintained, generally speaking beyond a kboot.conf file, nothing really needs to be done on the host os to use kboot. of course, it would be nice to be able to issue updates for an image, which should be possible with a package that makes use of ps3-flash-util from the ps3-utils package. nothing too difficult/complicated there. how kboot can be used/built within debian this is the tricky part. in additional to the standard Free Software guidelines which debian is so (in)famous for following, we also have some basic QA rules that are needed from a distribution release management / security perspective. one of these rules is that generally speaking it is considered taboo to include embedded copies of other software within a package, if that software is already available in debian. for example, i don't think it's very likely that the ftp-masters will look very kindly on some new software package that includes a copy of the entire kernel source tree, the gcc compiler, (uc)libc, busybox, coreutils, udev, etc. it may be that a certain subset will require special patches, in which case some convincing/justification will be in order, but for the big ones (i.e. the kernel) there will almost certainly need to be modifications made. ps3-kboot in ubuntu there are already source packages available for ps3-kboot in ubuntu. apart from the fact that i can't stand working with packages that use cdbs, this package can't be used because the package maintainers' solution to the previously mentioned problem is basically to ignore that it exists. it also seems that there's quite a bit of customization in this source package with respect to the initrd generation, which is probably worth some review... so i will likely still use this package as a base point of reference.

As I wrote previously, I started looking around for a new hosting provider, as Layered Tech is jacking up their prices too much. This isn't something I consider 'fun', as it will involve a lot of time swapping things over, with the potential for problems, but the amount of money disappearing every month is too much to ignore at this point. After some dithering, I finally went ahead and spent some money for a minimal slice at Slicehost. Linode seems pretty competitive too, but I more or less flipped a mental coin. I was actually ready to go ahead the other day, but I stopped to think some more after seeing that Slicehost has been acquired by Rackspace. In terms of seeing hard working people rewarded for their efforts, I am of course happy for the guys at Slicehost, but it also leaves a few questions:

• Sometimes, perhaps not immediately, but after a while, there is less incentive to bust your butt if you cashed in in a big way.
• Rackspace is not my kind of hosting outfit: they don't even list their prices where people can see them, like Slicehost, Layered Tech, and most everyone else does. I've heard they're good, but quite expensive. I don't need 'managed' servers - I can do that myself.
• The FAQ says that Slicehost will continue as a going concern in its own right, but there's always the chance that they were acquired for their tech, and that the FAQ just says that they will continue to operate to keep their current clients from bolting.

Of course, good things could come out of the deal too, so I decided to put my paranoia aside and test the waters. There are a lot of things to like about a system like Slicehost's: it's relatively quick and easy to set up new slices, fiddle with the resources that each slice has, and so on, compared with a physical server. Furthermore, they have a reputation for not 'overbooking' their servers, which is why I wanted to get away from a VPS in the first place, many years ago. Setting up a slice was very quick and easy, but unfortunately got off to a rough start: their "fraud detection" system had a problem with me being in a different place than my billing address. Luckily, they were very fast at correcting the problem. As an aside, I think that fraud detection is probably best left to the credit card companies, although of course I don't really know their business, so maybe that particular check has saved them some money/frustration in the past. Upgrading the Slicehost Ubuntu to Intrepid In any case, once I got it up and running, I wanted to test out the upgrade to Intrepid, as that's going to be out in less than a week, and I might as well upgrade while I have 'fresh' servers to play with, and prior to installing a bunch of stuff. It went very well. I did a do-release-upgrade -d, hit 'yes' to everything, rebooted, and everything was fine! Admittedly, I don't have anything running on the server yet, so that kept things short and sweet, and didn't require the upgrade of various systems which might have had troubles, but so far so good If you'd like to sign up for Slicehost, it looks like they give you a referral code. Mine can be utilized by going here.

For quite a few years now, I've been using a Dell 2001FP 20" LCD. Aside from the fact that this is the monitor that has apparently placed me on Dell's "permanently harass with paper catalogs" list (BTW, USPS has sent them a nasty enforcement letter over that), and the fact that when it needed warranty repair, Dell didn't know how to do it because it wasn't connected to a Dell computer, it's been a good monitor.

With Terah's recent conversion to Linux, she needed a monitor. We decided she could use the 20" LCD, and I'd get a new monitor. I have a nice 24" HP LP2465 at work, 1920x1200, so I thought I'd hop on newegg and see what 24" monitors I could find.

I found a Samsung T240 over there. Apparently Samsung's premium "touch of color" line. I did some research. It got glowing reviews -- people seemed to think the display was brilliant, and the only problem was that its stand wasn't height-adjustable. It got rave reviews on Newegg. It was $400 after rebate. And it arrived here Thursday.

I opened up the box, and I have to tell you that it looks beautiful. The bezel is this reflective black with a hint of dark red at the bottom. I put it together, hooked it up, and...

Well, let's just say that "Touch of Color" isn't all that accurate. "Touch of Crap" is more like it.

As I write this, my Firefox window covers half the screen. The center part of it is this brilliant color. The lower part has the white faded to this sickening blue, and the bottom right corner has almost turned to a black-and-white image. While the top part of my screen simply is more contrasty, except for the very top of the screen, which has half an inch of brighter area than anywhere else.

In short, the viewing angle on here is so bad that sitting directly in front of the screen gives significant color variations. An xterm with a black background maximized to fill the screen has a black background some places and a gray background others. My solid midnightblue background looks more sky blue at some places and more... really dark blue... at others. And of course, moving my head slightly varies the color that I see. Also, on photos, there is very visible horizontal banding in them -- apparently some sort of dithering going on since this is only a 6-bit display. Ugh.

How on earth this monitor got glowing reviews, I have no idea. I learned a bit about LCD panel types. Apparently the T240 is a TN display. The horror. I'll never buy another one of those again.

The HP LP2465 that I have at work, and which is great with solid colors all across it, is an S-PVA. So I'm buying one of those for home. It's more expensive, but -- it's a good display.

I am still trying to figure out exactly who thinks this is a good monitor. I tried games on it. It was responsive, but again, colors were distorted due to viewing angle. And this distortion means it's completely useless for photo editing, where precise colors are important.

I bought it from Newegg, and their policy generally prohibits refunds on LCDs. I called them, explained the situation, and told them that I wanted to buy an HP LP2465 from them to replace it. After being on hold for a few minutes, they agreed to waive their policy -- and even the 15% restocking fee, once I placed the order for the HP monitor. Classy show, Newegg. That's the one bright spot in all this.

So lately I had the bad fortune to type in apt-get install gnome-control-center on my workstation. It pulled in probably a hundred dependencies, but I confirmed installing it, never really looking at that list.

The next day, I had a reason to reboot. When I logged back in, I noticed that my beloved standard X11 cursors had been replaced by some ugly antialiased white cursor theme. I felt as if XP had inched closer to taking over my machine.

I grepped all over $HOME for some indication of what happened. I played with the cursor settings in gnome-control-center's appearance thing, which didn't appear to have any effect. When I logged out, I noticed that the cursor was messed up in kdm of all things, and no amount of restarting it could fix it.

After some grepping in /etc, I realized that I could fix it with this command:

update-alternatives --config x-cursor-theme

And I set it back to /etc/X11/cursors/core.theme. Ahh, happiness restored.

I guess that'll teach me to install bits of gnome on my box. Maybe.

The Debian bug sprint has started! 27 players registered in the hope to win cookies. The list of players and their assignments can be viewed at the usual place. Good luck to everyone. NB: for those who wonder, the assigned bugs were chosen as the 27 oldest RC bugs without a patch and randomized with shuf.

Last Wednesday I went for the first time in many years to FES Iztacala, the UNAM faculty where I worked for four years (1999-2003) and where I have most learnt and advanced in my career so far. I have a very special spot in my heart for Iztacala :-)
But it was not just a nostalgy drive - In no small part, I had not visited Iztacala -despite several invitations- because... It is really far away, in Tlalnepantla, Northern Mexico City (while I live in the South, just by UNAM). It takes me approximately 1.5 hours to get there via public transportation, and I would not venture less than 1 hour travel time by car. So... Having nothing to lose, I decided to go by bike - you can look at my route to get there (OpenStreetMap, SportsTracker) and safely back home (OpenStreetMap, SportsTracker). Some people I have talked with think it was a crazy thing to do - No, I don't feel that, by a long shot. A 26Km ride in slightly under two hours, and back. People insist on thinking that biking in such a large and chaotic city as Mexico is unsafe, dangerous, suicidal... I deeply disagree. Cycling is fun and got me to my destination in almost the same time I would have made by bus. And no, I would not buy four liters of gasoline just to cross my city.
Anyway, I am also happy about the reason that actually got me to go to Iztacala - I submitted a talk+paper I prepared together with Alejandro Miranda to Congreso Internacional Software Libre y Democratizaci n del Conocimiento organized by Universidad Polit cnica Salesiana in Quito, Ecuador. This conference is quite different to those I am used to, as it is quite more formal and academic; it is mainly targetted at social scientists working on understanding our movement. We prepared a talk called Software Libre y la Construcci n Democr tica de la Sociedad - which was accepted, to my amazement.
Neither Alejandro nor I were unable to travel to Ecuador to give the talk, so we arranged to present it via a videoconference call - Which was based on Iztacala. A nice session, although quite different to what I am used to. Our presentation was on a panel setting, under the global Ethical and political dimensions in the Free Software culture, with 20 minutes to present the topic (I am used to preparing one- or two-hour talks), and it was frankly rushed... We "met" with some friends (or were able at least to greet them shortly after the talk) who attended live to the conference, and... Well, all in all, it was one of those good, interesting experiences I would surely repeat. And besides, I have several things pending to show off about my current work to my Iztacalan fellows ;-)

I bought my first Apple hardware a number of years ago -- a 400MHz titanium Powerbook. The thing came with Mac OS X preloaded, but it was so slow that it was unusable. I put Debian on it, and was mostly happy with it -- until the titanium case started coming apart. Oh, and the small detail that the built-in wifi reception was so poor that I had to use a PCMCIA card to get a signal in some parts of my house.

But ever since then, we've always had some piece of Apple hardware in active use in our house.

About 7 years ago, my main workstation was a rapidly-aging 600MHz Alpha with -- gasp -- PCI video. It was OK for programming but really awfully slow at anything graphical. I was wanting to get into video editing, and we bought an 800 MHz iMac. The kind with the inverted-bowl base. A nice machine at the time. This was going to be my main workstation.

When it arrived at home, I unpacked it. Terah walked in and said, "Wow, my new computer is cute!" So it became *her* computer, and has been ever since.

It's rather old and is showing its age. Its wifi card died a couple of years ago. The 800MHz CPU is slow with today's websites.

So yesterday, we clicked on the apple and shutdown for the last time. It left her desk, replaced with a Linux desktop with a 20" 1600x1200 LCD (replacing the 15" 1024x768). I had assembled this Linux desktop out of spare parts in our basement, with a 4-year-old Athlon XP CPU. It's still faster than her Core 2 Duo work laptop running Vista, but that's another story.

I put Debian lenny on it. And I've got to say -- wow -- the utility of Debian as a desktop OS for the non-geeky has taken an incredible turn for the better in the last few years. Terah was fine with switching to Linux. I set it up for her -- Firefox for web, Thunderbird for mail, J-Pilot for syncing to her Palm, OpenOffice, gajim, etc.

Her concerns were: having Thunderbird make a nice "whoosh" sound when it successfully finished transmitting an email, having desktop gadgets (I installed screenlets, which I think she likes a lot), and.... I don't think I remember any others.

So now we both run Linux on our desktops.

Oh, and on her laptop, I poked around in the Vista control panels (I must have "authorized it to display this panel" a dozen times, stupid thing), and eventually found a box where I could optimize for visual effects or optimize for performance. I hit optimize for performance, and it transformed its interface into approximately Windows 2000 after a couple of seconds. I think that change made it a lot faster. It's now about as fast as the 25%-as-fast Linux box. I guess that's good for Vista.

For a minute, I’d like to think about The Announcement in terms of problems and solutions. Problem: We don’t have enough non-developing contributors in Debian
Solution in The Announcement: Give them special, official, statuses in the Debian community, so they are rewarded for their work.
Alternate solution: Are they really looking forward this status? Aren’t we just thinking that they are just as power-hungry as we are? Last time I checked, the Debian community wasn’t really welcoming non-developers. It isn’t a problem of official status. We have to work on ourselves to make Debian a better place to be for non-developing contributors. Also, giving those people a second-class citizenship, that won’t be widely recognized — “Debian Member” instead of “Debian Developer”, isn’t going to help. (or even worse, “Debian Contributor”, which doesn’t give any right except an email address). Problem: We have a trust and security problem. It’s difficult to give upload rights and access to Debian machines to more than 1000 people, with some of them not being active anymore, or not having much experience with security (in the case of non-developing contributors).
Solution in The Announcement: Remove rights based on classes of developers, so the group of people having full access stays within manageable boundaries.
Alternate solution: Have fine-grained control on who can do want. Problem: DMs can’t get access to Debian resources because their keys aren’t managed by the Debian keyring team, but by another team.
Solution in The Announcement: Keyring managers will take care of the Debian Maintainers keyring too. In the past, the keyring managers were the cause for huge delays in the NM process. It has improved a lot recently, but it doesn’t mean that it will always be like that.
Alternate solution: Make the DM keyring team, the keyring managers, and DSA, work together on a solution. it doesn’t sound impossible. If that’s necessary, merge the DM keyring team and the keyring managers. Problem: DM was done without the blessing of most of the members of the loosely defined group of powerful DDs (sometimes described as “cabal”).
Solution in The Announcement: Drop it, replace it with something very similar, but originating from the cabal.
Alternate solution: Why not keep it? Problem: Clueless DDs advocate clueless contributors for DMs. Then those DMs upload crap to the archive. (Note: I don’t necessarily agree with that problem, that’s just a problem that the announcement tries to solve)
Solution in The Announcement: Require answering a few questions before becoming a DM. Remove the right for every DD to decide which DM can upload which package. Give that control only to the NM committee.
Alternate solution: If we can’t trust one DD to take the good decision about advocating someone for DM, we could require that each person needs to be advocated by two different DDs. Or three, four, five. Asking the DM to answer a few questions, whose answers he can probably get by asking around on IRC, is not going to improve the average level of our DMs or DDs.

---

I believe that there are real problems that need to be solved, but that the decisions announced don’t solve the real underlying problem. Here is what I’d like to see:

• Fine-grained access control for each DD or DM or …:
  This would allow people to have specific rights inside Debian, like:

• Right to login on each debian.org host (a per-host switch)
• Right to upload packages for which the person is Maintainer or Uploader
• Right to upload any package
• Right to vote

All rights would default to “NO”. DDs would be allowed to change their rights for everything, while DMs would only be allowed to change specific rights. Of course, we need a secure way to change rights (HTTPS+confirmation using GPG, like the current “sudo password” thing?). But that would help with the security problem (you wouldn’t have access to all Debian hosts without enabling it). In fact, we already have that: many hosts require a switch to be enabled in LDAP before you can login, and we have tons of Unix groups to restrict access to specific areas of our infrastructure. I’m just proposing to extend that to basically everything, and to allow people to grant themselves some rights without going through DSA.

Modified NM process for non-developing contributors, that would still include many P&P and T&S questions, but would give “Debian Developer” status without any upload rights. That way, non-developing contributor would have an official status, and will still be able to write “Debian Developer” in their resume. Giving a different status to those people just because we don’t think that they should have the right to upload any package doesn’t feel right.

Is that enough? Probably not. We need to be more welcoming towards non-developing contributors, which is a social problem, not a technical one. But we are not going to solve that in a GR.

One month with Emacs and counting - Part 1 Straight to the point: since mid September I've been using Emacs, trying to evaluate whether I was willing to switch from Vim to it. Yup, that's true, me (user of Vim since the day I've started using GNU/Linux 10 years ago, (not so) active maintainer in Debian of vim and related packages, author of some popular Vim extensions and of vim-addon-manager) it's considering switching to Emacs. What is worse is that I've de facto already switched. May jamessan forgive me. OK, that was the hardest part to write, now ... This choice will have an impact on me: partly because I'm quite sure several fellow DDs will start making fun of this story , and partly because as many geeks I've always had strong opinions on the editor war. Switching side ain't easy. Hence, I've decided to write a small (2-3) series of blog posts on the issue, to future memory. The post you are reading is about why, since a few months ago, I wasn't willing to give Emacs a try, and how I've changed my mind. Why I wasn't willing to give Emacs a try, but then changed my mind I love knowing tools which can improve my workflow, no matter the task. Editors happen to be at the intersection of many tasks, hence knowing well his own editor and feeling satisfied about it is quite important. This is even more so for free software hackers which, for a reason or another, happen to use the very same editor for a lot of different tasks; in my case: programming, conf file hacking (i.e., playing the sysadm role), mail writing, and everything in between. I've started using vi on Solaris around 1998, then switched to Vim starting from version 4.0, and then followed all its releases up to 7.0. I consider my "Vim karma" to be quite hight and I've delivered several talks about using Vim for LUGs and other free software-related events. Still, I've always been hit bit several minor nuisances and glitches of Vim which couldn't be fixed by simple configuration tuning or add-on implementations (more on this in a future post). In the quest for the perfect workflow, those glitches have made me try several times other editors hoping for more satisfying work environments. Of course Emacs was one of them, which I've tried repeatedly during the years; the last time was circa 2006. For one reason or another, after a few days of experiments, I've always decided to give up with it. Why? Various reasons, listed below, together with an explanation of what (I think) has changed since then.

1. Poor Unicode/multibyte support. Support for whatever Unicode encoding was close to null. It was not available in the legacy editor, and you had to use something called MULE, to be specifically enabled. That was so 60's, and rather hard to use. Now (apparently starting from Emacs 22.1, June 2007) finally you have built-in Unicode support and the needed two variables for setting the input encoding and the file encoding, which is basically all you need. Looks easy once you have it ...
2. Lack of "modern" UI toolkit support. Yes, I do live in a console and invoke my editor from it, but nevertheless for long coding sessions I do also love having a nice GTK+ window containing my editor using settings which smoothly integrated with the graphical settings of my desktop environment. Maybe I'm getting old, maybe GNOME has changed my perceptions, but why the heck I had to tolerate different monospace fonts in my gnome-terminal wrt my GUI editor I never understood. Firing up Emacs was like getting back in time of 15 years, and if a geek having a maximized terminal on most of its workspaces had this feeling, then something was really wrong. This has changed: starting again from Emacs 22.1, the editor has frown support for that "bleeding edge" technology called GTK+. Finally you can use Pango font faces and avoid the time travel feeling when switching from a terminal to your GUI.
3. "Dead upstream" syndrome. As we all know by experience, choosing a free software product is not only a matter of evaluating bugs and features, there are other environmental factors. A notable example is how active upstream is. A few of us will use a product affected by the "dead upstream" syndrome. That's exactly what I was feeling about Emacs. The first argument I can offer to back that feeling is again the Emacs release history (noted that "small" hole between 2001 and 2007? Rest In Peace release early, release often ...). The second argument is the bug tracking system: there was none, bug discussions happened only on the development mailing list and the bug tracker was a text file on CVS! (Yes, Vim doesn't have a proper BTS either, but I was looking for reasons to switch from it and hence looking for something better in several respects.) That has changed completely, as I discovered only at this year DebConf thanks to gismo. 2007 has enjoyed 1 Emacs release and 2008 already two. They do have a BTS now and guess what? is dear old debbugs. According to my first experiences as Emacs bug reported the maintainers are very reactive and also friendly in dealing with bug reporters. Finally, also keeping up with snapshot releases in Debian is entirely trivial, thanks to Romain's repository.
4. Poor performances & memory consumption. This goes in the direction of probably the most popular joke used by Vim zealots in the editor war: Emacs is a nice operating system, but lacks a good editor, or something such. In the past I had the impression that starting Emacs was really slow and that generally the memory footprint was too high to be acceptable. You know what: starting a full-blown gvim with a good deals of add-ons (and I used quite a lot of them, that's why vim-addon-manager was born) is not that faster. And on the contrary, firing up an emacsclient is way faster than starting up a new gvim instance. Regarding memory consumption Emacs is still "gaining" the battle. Still, one has to keep in mind the different workflow: with Vim you are encouraged to enter and exit the editor (not without drawbacks, as I'll discuss in a future post), while with Emacs you always use the same one. It turns out that via emacsclient the resulting feeling is the same as having multiple editors (as you can fire them up as separate windows or in separate consoles), still sharing all user memory. A final comment on memory consumption while looking at my top sorted by decreasing RSS memory: the top process is xulrunner (from the browser) with 356Mb, then Xorg (159M), pidgin (128Mb !!!!), trackerd, gnome-terminal, gnome-panel, ..., Emacs shows up with 51 Mb at the 7th position. Yes, it is not the thinnest editor in the world, but on average machine it doesn't really matter at all.

In the next post: how to switch from Vim to Emacs, without loosing your mental sanity. (Whether I did succeed wrt sanity is up to the reader to judge.)

so i've been looking into the state of things with running debian on my ps3, and while talking to some folks on #cell, it was brought to my attention that there was a problem or two with the libspe2 packages, used for programming SPU using applications. for those not in the loop, the Cell Broadband Engine chip which powers the ps3 is a multi-core processor, with two general purpose PPU's and 6 specialized SPU's available. Without spufs support, the SPU's aren't available, and the real number-crunching capabilities of the ps3 aren't either. more information on the nitty-gritty of spufs is available here anyway, the problems are fixed now still, as of now there's still some one-time setup after installation. namely:

    echo spu    /spu    spufs   defaults    0   0 >> /etc/fstab

(and then mount it or just reboot) there really should be some way to have this setup automatically, either from the installer or through some kind of spufs support package. i'll take it up on the mailing lists and see what the consensus would be for this.

I attended Openmind last week, an interesting conference organized by the Finnish Centre for Open Source Solutions (COSS) to bring together open source professionals, community members and academics in Finland. In the session about business aspects of open source, in which I gave a talk about FOSS Governance, Nina Helander and Mikko R nkk presented the preliminary results of the National Software Industry Survey 2008. I found the results incredibly interesting. In particular, they found that 75% of responding firms use open source software in some way. This figure is up from approximately 13% in the year 2000, which is a major increase in just a few years. The study also found that that there was no statistically significant difference of company size when it came to the use of open source. Finally, companies that have open source components in their offering rate themselves as more innovative. Roberto Galoppini, who moderated the session, commented that Finland is where Gartner predicted that companies would be in 3 years. Indeed, I think everyone in the room was pleasantly surprised by the amazing numbers from the survey. We should not forget that the study was with software companies and that the use of open source will certainly be lower in other industries, but nevertheless the study shows that Finland is a leading country when it comes to the adoption of FOSS. (Originally published on FOSSBazaar)

In my previous post I drafted a rough idea how to add a feature to Debian that would bridge the gap between Debian as a Binary Distribution and any Source Distribution. The feature in question is giving users the chance to build customized Debian packages for specific feature sets. The possible use cases are simple:

Lets say Anthony User needs super-duper-tool with postgresql support. However this is a not so common use case and so the archive only has postgresql support. If he now wants to build a custom package, the process is "as simple" as:

apt-get source super-duper-tool
cd super-duper-tool-*
apt-get build-dep super-duper-tool OR sudo mk-build-deps -r -i
(possibly try out which ./configure options are supported by super-duper-tool)
vi debian/rules
edit debian/rules
dpkg-buildpackage -us -uc
sudo debi

That is, to be honest lot of work. And the biggest problem about it is that you need to know all this, which is not really user knowledge.

The basic idea is that Anthony can do something like
export DEB_BUILD_OPTIONS=nomysql,postgresql
deb-build-tool build super-duper-tool

and is done.
So the question is how could this be achieved?

1. Define a set of common options (options should be consistent through all source packages, otherwise it makes not that much sense) that every package should support, if it is a supported feature by the package in question. This includes flags like mysql, postgresql, ldap and its no-equivalents (nomysql, nopostgresql) to negate it.
2. Enable use of this common options in the package build process and that is really the hardest part. Defining a lot of ifneq..findstring constructs for 2-10 options per source package in any available source package is.... not.... a good approach. Luckily a lot of packages uses autoconf where enabling and disabling features is as easy as adding a --with-foo or --enable-foo option to the ./configure parameters. So we could write a wrapper that would handle these options. The debhelper scripts already have dh_auto_configure: Maybe this could get enhanced.
3. If it does not exist: Write a tool to auto-build packages with DEB_BUILD_OPTIONS set by the user. Should automatically get the source, (optional: setup/update a pbuilder environment) and build it and optional install/upload it (to a user repository for example) .
   

Still, the idea is only a rough draft. There are still a lot of open questions/issues, for example:

• How to handle these feature flags in packages that don't use autoconf
• How to enable these feature flags for packages that don't use debhelpers
• Shall a central place for setting DEB_BUILD_OPTIONS exist (like it does for Gentoo, FreeBSD etc.) and if so: Where?
• How should packages define new build options, if those defined Debian-wide, aren't enough?
• What about support? Do we support such custom builds of packages or will this be a support-on-good-will thing

Is anybody interested in pushing this idea forward? I'm happily interested in it and I know people who like the idea, too. But I can't push this forward on my own all alone. Certainly the work won't affect Lenny anymore, but considerable the time frame that is planned for Lenny+1 by the release team, this could be a feature for Lenny+1.

Gathering
I spent this weekend at a Skolelinux/Debian-edu gathering in Oslo. 8.5 hours by bus each way. I got to meet some of the people shipping a customized Debian for schools. As it was a gathering related to educational software, looking at the KDE4 kdeedu packages seemed most appropriate. I’ve tested step packaging and hopefully it is going to end up on your local debian archive one day, along with some fixes to where kvtml files are packaged, much awaited by Annma and others. Apparantly, in Brazil, they have their own debian based kde distribution and 130 GB of free as in public domain free educational movies. Some people discussed an application to view, tag and search the movies. I quickly thought of a mixup of nepomuk and phonon in a nice userinterface. So if anyone feels like implementing that, you are more than welcome. Gnash
The gathering was together with a Gnash hackathon, and apart from gnash getting more and more usable, the gnash people are also thinking about integration with the desktop environments. For KDE 4 they have plans of a KPart easy embeddable anywhere, including in KHTML. (Maybe another use of the KPartLoader dfaure wrote recently). The gnash team are unfortunately quite short on KDE knowledge and they asked for help. I’m probably not myself the right one to help here, but maybe some of you out there feel like helping. There is already a patch set around that some people claims works, but others haven’t figured out to make it work. Feel free to comment here and I will forward to the Gnash people - or just contact them directly on the gnash-dev@gnu.org maillist. Trolls
I left the gathering a bit early to get to a Troll picnic in a Oslo park and got stuffed with p lse and lompar. (sausages and pancake like bread) I was by bus all night back home and then a shower and straight to work. After work, I slept for 14 hours or so.

Today my 5 years old laptop gave me some filesystem problem after running apt-get update, which resulted in some corrupted packages, among them libc6, with the result of my system becoming non-functional. So I needed a recovery CD to be able to mount read-only root partition. I had around an Ubuntu 8.04 Hardy Heron LTS, so I used it to run its included LiveCD and recover from there. At the same time I was curious about it, as I hadn’t had chance to test it before. One of the things that surprised me most was that Compiz was enabled by default. After a second thinking that this was going to be a problem, making the system unusable (remember, a 5 years old laptop, and I had checked myself compiz some time ago with a catastrophic result), I started to meve windoes, jump from one virtual desktop to other, and minimiza and maximize windows without a problem. The work that had been made amazed me! So, some fsck and apt-get commands after, I managed to reboot on my Debian system, and I installed compiz. The result: my system goes faster using compiz, with a bunch of effects enabled than running plain metacity window manager. I have a also installed some compiz fusion plugins, and they are running without a glitch and with an incredible speed. Obviously, I have not chosen full quality, but it is enough to show effects in an efficient way. I have improved my user experience and system speed at the same time… Once again Linux shows that it can get the best of old hardware. Good work, guys!

I fully agree that working with Gentoo is no fun at all, as Julian points out in his post
My impression, when I tried Gentoo for the first (and last) time is the same. Thats not because it is made bad. In fact things in Gentoo aren't made that bad. There is good documentation, portage is quiet nice (with its USE flags and alike), but in the end I'm not really satisfied.
After all Gentoo uses a copied concept, which itself is good. The concept is derived from the BSD-world and is just the concept of source-based systems. This concept has its advantages over binary distributions, because it allows a flexibility that is not really possible with a binary distribution. That really is the only appreciable advantage of these systems. Users of these systems (including FreeBSD and alike) tend to give other arguments as well: Newest Software, Highest Performance and even Security is a point they give.

Usually the "newest software" argument is brought up with a rant against Debian. I keep hearing statements like "Debian Etch is totally outdated and Testing is broken". That statement in itself is just wrong, but the important thing is: What do you need such new software for? On a server its preferrable not to upgrade to each and every major release, for certain reasons. Its also a bad thing to build on production systems, so you need to take further measures to administer systems like that. On a desktop I can understand the logic, but then again: No. Why would I want a system I have to compile from scratch and on each upgrade (wasting a lot of time, power etc.) which changes often?

The performance argument is the dumbest of all. Binary distributions usually build binaries for various platforms. And while they can't be optimized for a very specific processor or a very specific feature set (=reduces binary size) they usually perform well enough that a difference between the self-compiled Gentoo system and a foreign-compiled Debian is not noticed by the user and sometimes not even measurable. So the time you save during the lifetime of your builds (which is not very long on a Desktop, is it?) because of the enhanced performance is used up a hundred times, by the time you waste for compiling the whole software only for yourself. Even if some of the software performs noticable better (e.g. video processing is said to benefit a lot from an optimized platform).

What about the security argument? It has been said, that you get your updates earlier then users of binary distributions. That is partly true, because in a binary distribution the update needs to be built first, before it gets to the end user (however, everything before is the same for a binary or a source distribution). But do you really follow security issues that close, that you benefit a lot from this? If you do, indeed, you save some hours. If (and only if) both distributions have a solution for a security issue at the same time.

Anyway. The flexibility argument can't get discussed away. Its the argument that makes Ports in BSD-systems attractive. You can have exactly the features you want, with the build-time options you want. In a comfortable manner. That is in contrast to binary distributions where we package developers need to guess which features might be needed (/wanted) by the users with mixed results. Good if they can decide for themselves. Still, I don't see the reason for compiling the whole system, if I need one or two customized components. So I go with Debian and customize packages if I really need/want to. The only difference is that it takes a lot of more effort.

I would love to see something done in Debian to reach a compromise. Making rebuilding of packages with different options very comfortable. We have DEB_BUILD_OPTIONS, maybe we should enhance it to support a lot more options then noopt,nostrip or nodoc. Possibly it would be a good thing to standardize on some use flags (e.g. [no]ssl, [no]ldap, etc.) and support them in the debian/rules file. This way building customized packages would be as easy as setting sensible DEB_BUILD_OPTIONS and run dpkg-buildpackage on the source. This could be eased further by providing a tool to download the source and build a given package (IIRC such a tool already exists). How does that sound?

A few years into this running hobby, I realized that my times were getting better. But I had no feel for by how much, or whether that was a constant rate of improvement etc pp. Long story short, I started to plot some of the data. What seemed natural was to record the date, the distance in miles as well as in a qualitative variable, and finally the average pace. Additionally, I played with groupings into just three categories 'short', 'mid' and 'long'. This leads to a natural 'one-factor' model of pace as a function of race date grouped by race distance. And given how easy it is to do conditional plots in R, I quickly arrived at something that already resembled the following chart: At first, some of the groups had too few data points to actually reliably construct regression lines, let alone non-parametric smoothers. But over time more and more data points were added as I kept running races. Including for example the somewhat disappointing result from last year's Chicago marathon in record heat that resulted in the outlier in the last panel. It actually made the smooth fit turn upwards! Luckily, the subsequent times in New York last fall, London in April, and of course in Berlin last month helped to dampen the effect of the one outlier, resulting in a more normal straight line for marathon performance that is comparable to the other four race lengths. All in all I am now quite happy with the chart. The combination of the non-parametric loess smoother and the robust linear regression (using lrm from the MASS package for R) shows that most groups exhibit very little non-linearity as both regression curves are very close to each other. The curvature in the '10m' group is probably mostly a small-sample effect. And I am obviously happy with the fact that three of the five panels show their respective last race as a PR :) The R script containing the data and code is available here but requires some familiarity with the lattice package for R (as the lattice book would provide).

In a recent blog post, Kai complained about various existing tools for marking up email in HTML. He also asked for pointers to other tools. Since he didn't specify good tools :-), I took the opportunity to promote my work in progress plugin for ikiwiki to do that very thing. When asked about demo sites, I realized that my blog doesn't actually use threaded comments, yet, so made a poor demo. Follow the link and you will find one of the mailboxes from the distribution, mostly a few posts from one of the debian lists. The basic idea is to use the Email::Thread perl module to get a forest of thread trees, and then walk those generating output. I think it would be fairly easy to make a some kind of mutt-like index using the essentially same tree walking code. Not that I'm volunteering immediately mind you, I have to replys to comments on my blog working (which is the main place I use this plugin right now).