There was a discussion today on the LUV-main mailing list about apache virtual hosting. Someone suggested mod_vhost_alias which is OK for a simple solution, but pretty limited. IMO, there are much better ways of doing it. Specifically, generating config fragments for each virtual host. I wrote such a system years ago, and have always thought “I should package it up for debian”. There’s a moderate amount of work involved in doing so, and probably an even larger amount of support work afterwards. Worst of all, I’d have to write - or con someone else into writing - documentation for it (it’s easy enough to use that you can be SHOWN how to use it in 10 minutes, but that isn’t good enough for a distributed package). I guess the point of this blog post is to find out if there’s sufficient interest to make it worth the effort. Feel free to comment or email me about it. If anyone wants to look at the code, i can tar it up easily enough, and even write some simple/minimalist instructions. anyway, here’s (a slightly edited version of) what i posted to LUV-main: Preamble I looked into apache’s mod_vhost stuff years ago, but IMO it’s just too inflexible and limited to be of much use. It’s fine if ALL your vhosts are identical…but not if you need different settings on some or all of them. As soon as you need a custom setting for one vhost, you have to step outside mod_vhost_alias and roll-your-own anyway. which means you now have two different methods of configuring your vhosts. Instead, I’ve used my own vhosting system for years (long before mod_vhost), on dozens of servers…the largest of which had over 500 vhosts on it - the biggest problem was the enormous number of file handles the apache processes needed for log files. After tweaking the kernel file-max etc limits, it actually worked without problem but I always intended to rewrite that bit of it to use a log splitter daemon, perhaps vlogger or similar. Unfortunately, I a) left that job before I had time to implement it, and b) never did figure out a good way to handle the error logs because apache doesn’t prefix the vhost name to ErrorLog output like it does for CustomLog outout (best compromise was to just have one combined error log rather than one per vhost….most web programmers/website operators never look at the error logs anyway. Personally, I find it invaluable, most people don’t even seem to think of looking at it) Synopsis of Operation 1. Creating a new vhost is a 3-step operation:

1. create a user (with adduser), optionally with a postgres or mysql account of the same name.
2. edit /etc/virtuals/virtual-hosts.conf, copy/paste/edit one line
3. run “make”

In step 2, hardly anything needs to be changed from one vhost to another - in the most common case, just the account’s login name, and the domain name. There are a number of different “flags” that can be set for each vhost in this file - ssl, cgi, mason, htdig, webcheck, aliases, group, samba, canonify (use mod_rewrite to redirect back to the site’s canonical name), and more. The Makefile runs perl scripts to generate the apache config fragment (and other config fragments - e.g. for htdig and samba), add/remove any IP addresses if needed (usually not, with name-based vhosts), set up the home directory structure, copy in any sample files (e.g. html forms and images for htdig), and reloads apache. It also checks in any changes to svn (or RCS. easily modified to use other revision control systems). On machines which are part of a load-balanced setup, it even ssh-es into other machines in the array, runs “svn update” and “make” to ensure that the vhost is operational on the other machines too. Finally, it concatenates all the active config fragments into one large file because apache reloads a LOT faster if it doesn’t have to load hundreds of little config files. 2. Removing a vhost is as simple as editing virtual-hosts.conf and commenting out or deleting the line, then running make again. virtual-hosts.conf The virtual-hosts.conf file is extremely useful. It gives me a single authoritative source of all the relevant details about EVERY vhost running on a machine. These details can be used by other scripts (e.g. cron jobs to run webalizer, analog, htdig indexing, webcheck, etc; cgi scripts to display a list of vhosts on the machine with clickable links, and more). This single config file makes the whole system easily extensible: write a new script to parse the config file, and loop through each entry, performing some relevant operation. e.g. run webalizer for every vhost, dumping the output under ~/public_html/LOGS/ (with optional .htaccess file to control access). Adding optional generation of samba shares for each vhost was quite easy. In theory, it wouldn’t be hard to put a GUI or web front-end on to it. Oddly enough, i’ve never bothered. And I’m not inclined to bother now (but wouldn’t object if anyone else did) Database? no, thanks I’ve occasionally thought about putting this config data into a database, but the benefits just aren’t anywhere near good enough to outweigh vi and all the benefits of text file configuration (especially comments!) Packaging? And, of course, one of these days, i should package it all up for debian. I’ve been saying that for years, and just haven’t got around to it. It wouldn’t take a huge amount of work to do so, there are only a few small things i need to do to automate the installation, things that i just do by hand in a few minutes when installing on a new box. Before packaging it, I also want to automate the adduser part of the process, as well as creation of postgres or mysql accounts. Most importantly, I ought to separate the code from the templates in the config fragment generator scripts, so that the local sysadmin can edit them without having to edit the scripts….OK, all up not a tiny job but not particularly big either….just big enough to be ignored on my TODO list for years. I’d definitely have to change the config file format from the current one-line (colon separated fields) to using a .ini style file. I’ve wanted to do that for a while but have inertially-resisted it so far because the one-line format is so convenient…unfortunately, I’ve added so many flags and features over the years that it’s seriously ugly - probably scary to someone who doesn’t know it already (i.e. anyone outside of the set of “me + a handful of other people”). I estimate this would take about a days worth of hacking I’d also take the opportunity to clean up some of the cruft :-) Essential Design Criteria BTW, one crucial design criteria for my system is that it should *NEVER* under any circumstances overwrite any hand-edited changes. This is implemented in a number of ways. First, it takes an md5sum of each vhost’s config fragment when it generates it. If/when the script is run again, it only re-generates the config fragment if the md5sum hasn’t changed. If it has changed, it generates the updated fragment as “.new” for manual merging if required. Secondly, when generating conf fragments it looks for specially named files in a directory for extra stuff to add before the <VirtualHost> line, between the <VirtualHost>…</VirtualHost> lines, and/or after the </VirtualHost> line. This is to provide another location to put custom changes for each vhost while still retaining the ability to have improvements/new features to the script automatically applied to existing vhosts. The reason for this is that I utterly loathe and despise allegedly “easy” configuration tools that blow away custom hand-edited changes. I hate them so much that this was THE major incentive for developing my own vhosting system. In Use I’ve been using my system for over 10 years now. It grew out of earlier bits and pieces, but the current incarnation has been working since 1999, with the occasional new feature or update (e.g. to add support for apache2 as well as apache1, and minor changes to adapt to changes in debian’s apache packaging). There are also a handful of ISPs and several small businesses around Australia running it because they’ve hired me to set up their web servers. Finally… My vhosting system isn’t a system for people who are scared of or dislike command-lines or text-file configs. It’s a system for people who love them, but want an easy way to automate a dull, repetitive task. Such automation is easy for CLI tools, and next to impossible for web/GUI tools. Syndicated from Craig Sanders' Errata: Miscellaneous Thoughts poll: package my vhosting system for debian?

Andrew Binstock interviewed Donald Knuth recently, and one of the more amusing tidbits was this:

I currently use Ubuntu Linux, on a standalone laptop it has no Internet connection. I occasionally carry flash memory drives between this machine and the Macs that I use for network surfing and graphics; but I trust my family jewels only to Linux.

More seriously, I found his comments about about multi-core computers to be very interesting:

I might as well flame a bit about my personal unhappiness with the current trend toward multicore architecture. To me, it looks more or less like the hardware designers have run out of ideas, and that they re trying to pass the blame for the future demise of Moore s Law to the software writers by giving us machines that work faster only on a few key benchmarks! I won t be surprised at all if the whole multithreading idea turns out to be a flop, worse than the “Itanium” approach that was supposed to be so terrific until it turned out that the wished-for compilers were basically impossible to write. Let me put it this way: During the past 50 years, I ve written well over a thousand programs, many of which have substantial size. I can t think of even five of those programs that would have been enhanced noticeably by parallelism or multithreading. Surely, for example, multiple processors are no help to TeX…. I know that important applications for parallelism exist rendering graphics, breaking codes, scanning images, simulating physical and biological processes, etc. But all these applications require dedicated code and special-purpose techniques, which will need to be changed substantially every few years.

This is a very interesting issue, because it raises the question of what next-generation CPU’s need to do in order to be successful. Given that it is no longer possible to just double the clock frequency every 18 months, should CPU architects just start doubling the number of cores every 18 months instead? Or should they try to concentrate a lot more computing power into an individual core, and optimize for a fast and dense interconnect between the CPU’s? The latter is much more difficult, and the advantage of doing the first is that it’s really easy for marketing types to use some cheesy benchmark such as SPECint to help sell the chip, but then people find out that it’s not very useful in real life. Why? Because programmers have proven that they have a huge amount of trouble writing programs that take advantage of these very large multicore computers. Ultimately, I suspect that we will need a radically different way of programming in order to take advantage of these systems, and perhaps a totally new programming language before we will be able to use them. Professor Knuth is highly dubious that the later approach will work, and while I hope he’s wrong (since I suspect the hardware designers are starting to run out of ideas, so it’s time software engineers started doing some innovating), he’s a pretty smart guy, and he may well be right. Of course, another question is whether what would we do with all of that computing power? Whatever happened to the predictions that computers would be able to support voice or visual recognition? And of course, what about the power and cooling issues for these super-high-powered chips? All I can say is, the next couple of years is going to be interesting, as we try to sort out all of these issues.

Lessons learned today:

• Assembling a bunch of kitchen cabinets is no easy feat. While assembling a bit of IKEA furniture is usually done in 30-60 minutes, assembling 4 cabinets took all afternoon. And we're only halfway through.
• Apparently, Gamma will not double-check that the package you receive from them is complete—the drawer set for the 30cm cabinet wasn't.
• Sometimes, being helpful turns out not to be. The Gamma kitchen cabinets come with some of the metal bits already in the wood. Due to this, I had not been able to form a mental picture of what the metal bits look like; and as a result of that, we damaged some of the cabinets (though luckily not critically).

Anyway, the bottom half of the kitchen is now assembled. Sortof. The upper half, we'll probably do tomorrow. Or, er, well, "today", since 5 minutes ago... Other than that, there's still much work to be done: Do something about the holes in one of the walls, treat another one of the walls for water damage, finish the paint job. Next tuesday and wednesday, a plumber will be connecting the kitchen sink and most of the plumbery in the bath room—hence the need for the assembly of the kitchen before that time. After that, I'll probably need to finish off some other things as well... Oh, well, and I'll also need to move, of course. In case you've been wondering, yes, this is why I haven't been very active on the Debian front lately. I've been signing my buildd mails (although not usually as prompt as one would expect), but that's about it. As a result, I got NMU'ed for the first time in my career as a Debian Developer—by two people at the same time, no less. Feels rather strange, I must say.

In this post, Steven Hanley wonders how to keep /var/cache/apt/archives empty on a machine that has a full mirror of debian on it. That’s actually the wrong question. The right question is ‘Why is /var/cache/apt/archives even relevant on a machine with a full mirror of debian?’ /etc/apt/sources.list has always supported file:/ URLs. for example, sources.list on my local debian mirror looks like this:

deb file:/home/ftp/debian unstable main contrib non-free
deb file:/home/ftp/debian-multimedia/ unstable main

With a file URL, apt-get doesn’t need to download any packages, and thus doesn’t need to cache anything in /var/cache/apt/archives. No need to keep it empty, it just is empty. In a related post, Andrew Pollock suggests adding DPkg::Post-Invoke “apt-get clean”; ; to /etc/apt/apt.conf. That would be fine if you ONLY ever ran “apt-get update ; apt-get dist-upgrade”. It’s not fine if you run “apt-get -d” to download packages first, then install a few packages with “apt-get install package”, and then run “apt-get dist-upgrade”. If you try that, then the DPKG::Post-Invoke rule will clean out /var/cache/apt/archives when you install the individual package(s). When you then run the dist-upgrade, apt-get will have to download all the packages again. Better just to get into the habit of running apt-get clean or autoclean occasionally when you want to reclaim some disk space. Syndicated from Craig Sanders' Errata: Miscellaneous Thoughts Keeping apt archives empty…or not.

Following my first ftpmaster blog post, which got some nice replies, lets have the next, as I didn’t stop working on stuff. Not as much as I hoped, but well, something. So, what was done since then?

• Merged a little patch from Christoph Berg fixing usage of a function - which doesn’t exist in global namespace, so it needs the full name.
• Merged another patch from Christoph - now we write the content of the Changed-By field from .changes files into projectb. To cite him for the reason: This will allow to identify NMUs and sponsored uploads more precisely in tools querying projectb. Broke dinstall with this and as a little side-effect also the merkel copy of the projectb database. Ups. Well, fixed it immediately again. :) There is some old data available, which I could import into projectb, so it would have data for all packages. But there is some weirdness with the dataset I got from Christoph and am waiting for him to provide a solution. Anyway, all new uploads already store this data, so it is only old data missing, currently.
• Merged a small patch from Thomas Viehmann into our archive software which enables it to send a copy of all mails to the person sponsoring the upload. This is in affect for all future uploads, not for any package which already hit the queues (like those in NEW right now). We do
  • only sent the extra copy if it is a sourceful upload, to not spam buildd admins,
  • only sent the mails to sponsors who did not disable their @debian.org address.
  For those interested, the process works similar to:
  • We look if the Debian login attached to the key signing the upload has an active @debian.org address.
  • If that is the case we look for all uids on that key that look like an email address and compare them with the Maintainer: and Changed-By: field in the .changes file.
  • If there is no match, ie. none of the “emails” of the key uids is in those fields, the @debian.org mail is added to the set of valid receivers.
• Changed split-done to only move files older than 30 days and let it run by our weekly cronjob. Also run it once, which moved 120079 files into their $YEAR/$MONTH subdirectories. (split-done seperates the subdir where we store every .changes of uploads accepted. That quickly grows fast, and we all know how bad filesystems perform with such a number of files. The above 120079 files are from November/Dezember 2007 til now, so imagine how many we have in total, as we have them back to 2002).
• Merged a patch I wrote 7 days ago that does a little sanity checking in the Provides field. Do not try to version your provide, that is forbidden and actually breaks software. (See this bug about it.)
• Merged a patch, submitted by Marc Brockschmidt some long time ago, that lets cruft-report also display NBS for experimental. Changed the cron script to use that extra feature.

---

And as in my last blog post, lets add some items to the todo list:

• Enable dak to read a Built-By field from the .changes file. Thats the sanest solution to sponsored uploads. And also better for buildds, they then should no longer overwrite the Maintainer field with the -m parameter when building packages. Exact semantics to be defined, but dak probably would go something similar to
  • Built-By same as Changed-By or Maintainer?” - do not send mail there.
  • Built-By differs from Changed-By or Maintainer and its sourceful? Send mail to it too.
  • Built-By differs from Changed-By or Maintainer and binary-only? Only send mail there.
• Track the Uploaders: field in projectb. That would help the BTS to offer another display.

Having written all this - today (AKA Saturday, when writing this) won’t see code merges by me. I think most of my ftpmaster related work today will be fun with the NEW queue. For some unknown reason people keep uploading packages. Tsss. Comments: 3

It seems existing versions of gcc don't warn when you use an assignment as a truth value as the first operand to the conditional operator. For example: Presumably you meant ==, but unfortunately that does not warn with gcc (maybe it will one day). We can use the tree dumping mechanisms (previously mentioned here) to confirm our problem. For the following function gcc creates: As you can see, this isn't the result we wanted. This can be quite nasty if you use the conditional operator in a hidden fashion; for example behind an assert. A common idiom is: You'll currently get no warning you've slipped up and used the assert wrong. Moral of the story: a quick audit of your code might turn up some surprising uses! However, in this case, the Intel compiler picks this one up: Although it's often not trivial to move to another compiler, as a rule I would recommend trying alternative compilers for your code as it's a great sanity check.

Till today I thought make was a very simple concept, but looking at other people's debian/rules files I start to lose that faith. So let's begin with some basics (as I guess many reading this are maintainers for Debian packages, and you might need some of this knowledge): As you will already know, the most important part of a Makefile is a rule. Each rule is there to produce something and has prerequisites, i.e. things that have to be done before. So far so simple. When you think about this way, the first pitfall is already no pitfall anymore:

   build: patch build-indep
   build-indep: build-indep-stamp
   build-indep-stamp:
   	$(MAKE) doc
   	touch build-indep-stamp
   

There are two mistakes in this. First of all, the clean is only called on ./debian/rules build, but not on ./debian/rules build-indep. And then patch is only called by the build target. But what you really want is that the source is patched before you do something, so it is something do be done before build-indep-stamp. The pitfall with this error is that you will not see it most of the time. As make usually processes targets in the order it finds them, it usually runs patch first. Except when you have multiple processors and tell make to make sure of them (and even then there is a chance it might work as the command to run first is fast enough) or if someone trusting on what he learned about make does call ./debian/rules build-indep-stamp build. I guess a reason one sees this so often is the next pitfall. Most likely previous the following was tried:

   build-indep: build-indep-stamp
   build-indep-stamp: patch
   	$(MAKE) doc
   	touch build-indep-stamp
   patch: patch-stamp
   patch-stamp:
   	whatever
   	touch patch-stamp
   

The pitfall with this is some new concept. The problem is that the patch rule is phony. A rule is called phony when it does not produce the target it claims to do. The classical example is a clean target. You do not want that a clean target creates a file clean and the next time it is called says "already everything done". Targets get phony by telling make it is (via .PHONY:) or by just not producing the file it tells to produce or (suprising to many, it seems) by having a phony target as prerequisite. In the above example the patch target gets implicitly phony, as it does not produce a file called patch. Thus after having build the source and calling binary to create packages, this will most likely in some way depend on build-indep-stamp. But when make thus looks at build-indep-stamp to decide if it already there and even though it sees the file produces there with the touch command, it cannot determine if that is up to date. It depends on patch and there is no file called this, thus make must assume it is not up to date, thus build-indep-stamp has become effectively also phony, in the sense of having to be called every time it is depended on. (In case you have not noticed, the fix would have been to make build-indep-stamp depend on patch-stamp instead or to go without patch-stamp and make patch the file to be generated). Thus, you can put as many -stamp files as you want in a row, as long as you have a single phony prerequisite in them, it is all void.

New QA website I modified qa.debian.org’s stylesheet/template, using the PTS’s stylesheet as a basis. It looks a bit better. The content was also updated, so we should stop receiving totally outdated answers to the “What does the QA team do?” question in NM. Now, who is going to do the same thing with www.debian.org? :-) Closing bugs in removed packages When packages are removed from unstable and testing, their bugs are not necessarly marked as closed, so they can’t be archived. A few days ago, there was about 3300 open bugs filed against removed packages. Thanks to the work of Barry deFreese, Marco Rodrigues and Raphael Geissert, we are now down to ~2500 bugs. If you want to help, just drop in #debian-qa and ask about our scripts/process. (There are some tricky details) DEP #1: NMUs With Bas Wijnen, we finally announced the DEP about NMUs we have been working on. Please join the (currently very quiet) discussion!

I'm just old enough to have only discovered the internet at about the time I graduated from high school, and have been using it for about 15 years. Every once in a while though, it comes back and hits me just how cool the whole thing is. Not "yeah, neat", but a serious wow at just how amazing it is to be able to communicate with people the world over. I think the first time that happened was when I had been exploring some of the early nineties internet - gopher and the web with lynx. Which was neat, but not in the "wow" sense. Then I stumbled on to IRC somehow, and since at the time I'd already started taking Italian courses, I thought I'd see if there was an Italian channel. There was, and I think the conversation went something like this:

me: so, where are you guys? someone: Italy, and you? me: Oregon, but, no, seriously, where are you? someone: Italy! me: Seriously?!

It was a very concrete demonstration of the fact, that, thanks to this new thing, I could talk to people all over the world, for free! Of course I've grown used to this, and take for granted that I can call my parents in Oregon via Skype for free, and a lot of other cool things, but once in a while something makes me take a step back and say "cool!". Most recently, a web site I follow that I follow has had an interesting back and forth between Paul Graham and David Heinemeier Hansson, with additional comments by the likes of Paul Buchheit, which, agree or disagree with their modus operandi, is an impressive cast of characters to be able to interact with, and learn from, without moving from my perch up here in the middle of the Tyrolean Alps.

For a university seminar I m currently writing a GUI program to view fractals based on simple iterated function systems (only similaritudes allowed). It supports three different drawing algorithms and you can edit the IFS by dragging squares around on the screen. But this post is not about this program (I might present it later), but how Haskell allowed me to solve a problem very nicely:The first instances of the code had a problem that a lot of GUI programmers know: The drawing of the fractal took quite some time, and during that time the gtk main loop is blocked and the program becomes unresponsive. At first I avoided this problem by manually splitting the drawing function (e.g. by repeatedly increasing the resolution) and kept re-drawing it at higher resolutions in an idle handler, so at least I could interact whenever one resolution has finished drawing. It worked somewhat but it was not easily done for the other, not pixel based, algorithms and it was ugly.So I wanted a way to (a) pause the drawing at any convenient point, to resume it later, and (b) safely abort the drawing if what I m drawing has changed and I need to restart.A common solution to this would be to do the drawing in a separate thread, so (a) is actually not needed, but I did not know if I can safely do (b), and I have heard that threads cause problems with gtk.So I tried to dig deeper for the hidden treasures of advanced haskell programming: I need a monad transformer! I expected the infamous ContT monad transformer to help, but I couldn t figure out how, and I started to create my own monad transformer, called CoroutineT.I tried to figure out what an action of type (CoroutineT IO a) should do, and I came up with this type signature:

pausingAction :: IO (Either (CoroutineT IO a) a)

which means that after the pausingAction is done, it is Either paused (and I get back another pausingAction to run when I want it), or it is done (and I get the result). Note that I m writing IO here, but it can be any monad.The definition of the datatype and the monad instance came mostly from trying to make this type work (yay to haskell s type system, less thinking required), and looks like this:

data CoroutineT m a = CoroutineT  unCoroutineT :: (m (Either (CoroutineT m a) a))  

instance (Monad m) => Monad (CoroutineT m) where
    return v = CoroutineT (return (Right v))
    a >>= b = CoroutineT $ do
          r <- unCoroutineT a
          case r of
            Left  paused   -> return $ Left (paused >>= b)
            Right unpaused -> unCoroutineT (b unpaused)

This translates to english like this: A call to return is not paused. When an action is already paused, further actions should be run after the paused action is resumed. When an action is not paused, further actions can happen now.Like every well behaving monad transfomer, I also need a "runCoroutineT" function to start the coroutine. I probably could have used unCoroutineT directly, but for my use case (GUI drawing) I did not need a return value, so this function is more handy:

runCoroutineT :: Monad m => CoroutineT m () -> m (Maybe (CoroutineT m ()))
runCoroutineT a = either (Just) (const Nothing)  liftM  unCoroutineT a

Nothing surprising here, basically just turning the Either into a Maybe. So it becomes clear how to do (b): We can just throw away the resume action returned by runCoroutineT (if any). The more interesting thing is how we do (a): We need a pause action of type (Coroutine m ()). But how should it work? I did not really try to understand why it works, but by looking at the types, I came up with this:

pause :: Monad m => CoroutineT m ()
pause = CoroutineT (return (Left (CoroutineT (return (Right ())))))

Yes, it sounds like some dance step instructions (read the second line out aloud!), but it works somehow.So here is some example code: I have a pausable IO action that counts from one to ten, pausing after each number. I also have function that resumes an pausable action up to n times:

example n = keepGoingFor n $ do
        liftIO $ putStrLn "This is the coroutine"
        forM_ [1..10] $ \i -> do
            liftIO $ putStrLn $ "Counting to "++ show i ++" while you keep calling it"
            pause
  where --keepGoing :: Monad m => CoroutineT m () -> m ()
      keepGoingFor 0 _   = putStrLn "Here I just abort the run"
      keepGoingFor n cor = do
        resume <- runCoroutineT cor
        case resume of
            Just runAgain -> keepGoingFor (n-1) runAgain
            Nothing       -> putStrLn "Finally stopped"

And here is the output of two different runs:

*CouroutineT> example 5
This is the coroutine
Counting to 1 while you keep calling it
Counting to 2 while you keep calling it
Counting to 3 while you keep calling it
Counting to 4 while you keep calling it
Counting to 5 while you keep calling it
Here I just abort the run
*CouroutineT> example 14
This is the coroutine
Counting to 1 while you keep calling it
Counting to 2 while you keep calling it
Counting to 3 while you keep calling it
Counting to 4 while you keep calling it
Counting to 5 while you keep calling it
Counting to 6 while you keep calling it
Counting to 7 while you keep calling it
Counting to 8 while you keep calling it
Counting to 9 while you keep calling it
Counting to 10 while you keep calling it
Finally stopped

So it does really works fine, and it proved very useful in my GUI drawing problem. For that, I created this nice control structure which works like mapM_, but calls pause every n iterations, and therefore hides the pausing stuff almost completely:

pausingForM_ :: Monad m => Int -> [a] -> (a -> CoroutineT m ()) -> CoroutineT m ()
pausingForM_ period list action = pausing' 0 list
  where pausing' _ []     = return ()
        pausing' n (x:xs) = do action x
                       if n==period then pause >> pausing' 0     xs
                                    else          pausing' (n+1) xs

I have put the complete module (including instances omitted here) in the darcs repository that might later also contain the fractal drawing program.

One week ago, I went to a branch office of Servicio de Administraci n Tributaria, the government office in charge of processing taxes. This year, I plan on doing something quite bold, as my Mexican friends will acknowledge: I will prepare my (quite simple, I hope) tax declaration by myself. I do not want to be held hostage of the accountant guild - So I might end doing some fuckup which in the end costs me money or time. I hope it is not the case.
Anyway... Last week I went to this office, as I needed either a CIECF (Clave de Identificaci n Electr nica Confidencial Fortalecida - Strengthened Confidential Electronic Identification Key) or a FIEL (Firma Electr nica Avanzada - Advanced Electronic Signature). No, please don't believe it is a security token, a card with printed numbers, a one-time-pad or the sort - The CIECF is... A password. Why is it strengthened? Because it has the feature of including a question, in case you forget the key, to allow you to change it. I guess the FIEL is a more reliable device, but I prefer not to even request it.
And as far as the questions go, the emergency questions for CIECF suck. First, I was not even asked the meta-question - I was not told why this information was needed. So imagine the clerk saying: Full name? ... Date of birth? ... RFC (Tax ID)? ... Favorite color? I was there just... Stunned. Why do you need it? Oh, just in case you forget your password. Ok... Don't you have any other questions which I am not prone to answer a different thing, and that are not dead obvious for a casual passer-by? (I guess that at least 1/4 of the public will say blue. Feel like brute-forcing SAT to its knees?) Other questions include your fathers' second family name, your favorite soccer team, your pet's name... It seems they took the first "security dos and don'ts" book off the wall, and started reading backwards.
But anyway, that's the system, and I must play nice with it. So I get back home, and decide to start hacking up my declaration. No, Mr. Policeman, I'm not saying I would try to break into the SAT - I just say it is a complex and non-obvious task to do. Now please release me. Thanks.
And I enter the system. Of course, I tried first with Iceweasel, knowing it would fail (it is documented: MSIE 5.5 recommended). I tried again with Konqueror. I tried, sigh, with MSIE from inside Wine. No luck. Well, even from within qemu's Windows 2000. Wrong password. WTF?! Stranger: It worked with SAT's My portal, although it didn't with the declaration, which is what matters now.
I cannot take the time every day to come to the SAT and move my data - It was a full week until I came back again. I insisted on fully logging in to the system, to be sure the password I entered this time was right. As well as my ber-secret safety question, of course.
And it failed.
Twice.
Until the clerk noticed something strange in the way I typed...
Sir, excuse me..., he muttered, why are you typing such a long password? Well, basically because I value my tax declaration, and I know brute force is a powerful force. (explain it, of course, in simple terms) Oh... No, the password must be eight characters long.
No wonder.
So I entered the first eight characters of my password, which was a true work of prose for their standards, at around 20 characters. And it worked.
Now, for bonus points: What do we gather from the fact that the long password works fine in one system, but in another system it only the short version? Why, but of course! I guess the passwords for every economically active Mexican is stored in their master database in plain text. Isn't it just beautiful?
Anyway, it seems I have a lot of work to do. If all goes as planned, maybe next year I will be for hire as a public accountant? Hmh, does not sound too much like fun, does it?

We received an interesting contribution to Hecl recently: Martin Mainusch put together a Hecl interface to kxml, and also sent in some demo code that fetches and parses weather forecast information. I thought it was pretty cool: Note that it's not compiled into the default build, so you have to change a property from 0 to 1 and recompile, and...being new there might be some issues, but still, it's nice to have that to let people play around with simple "web service" type things that use XML. Of course, another strategy might be to have a 'proxy' server that fetches the actual XML, and returns a tagged Hecl list so that Hecl can parse it directly, but that requires a server and has more overhead.

How to not start a service by default Use case: in my laptop, I sometimes need MySQL, PostgreSQL or Apache in order to test some software that I'm developing, but I do not want them on all the time. The solution is: rm /etc/rc*.d/S*mysql* (thanks to Wouter) update-rc.d will not touch your symlinks as long as there is at least one still around for a package. Also, this leaves the stop symlinks around, so that when I start one of these services for development, it will still be properly stopped on shutdown.

In the last leg of what has been marathon traveling over the last two months, I'm going to be heading back to San Francisco to give two talks at CommunityOne. CommunityOne is a new one-day conference that Sun is putting on -- along side it's massive JavaOne conference -- that focuses on free software, open source, and non-Sun projects. I'm going to be there talking about free software and free culture. I will be giving updated versions of the two talks that I have at the FSF members meetings over first two years. In the first talk, I'll be making the case for a strong free culture movement and in the second I'll be talking about liberating network services. If you will be at the conference, or just in the Bay area, and would like to meet up, I'll be in the area for most of a week and would love to arrange something. Just get in contact.

Sometimes, companies come up with the most stupid restrictions. And unfortunately, other companies enable them to do that.In this particular case, the DHL parcel service and Acrobat teamed up: You can buy and print shipping labels online with DHL. I thought I'd give that a try, especially since there is a 24h dropbox nearby.The label can be printed either using a Java applet or a PDF file. It thought PDF - sounds great. But it isn't. It's a very special PDF file. It has a scriped printing button that will print with the default settings (so it comes out at a completely different printer that I would have wanted to print it). In fact I would have expected the printing dialog to come up, and thought the button had not functioned at all. Also I could probably not have used it if I had opened the PDF file in a different PDF viewer. At my own laptop I might not even have Adobe Reader anymore - I use Evince. It's faster and has a much better UI. But I don't have a printer, so I used a computer at the university which used Acrobat by default, fortunately.And: you may only print the real shipping label once. Only 'sample' labels can be printed arbitrarily often. This is counted on the server side, btw., and also applies to the Java applet.Apparently you also must not rename the PDF file (WTF?) and you need to be online to print the file. So you can't save the PDF file and take it to an offline computer that does have a printer...Dear DHL: have you heard of that ancient technology called 'Photocopiers', sometimes also dubbed 'Xerox'? You know... lots of people still have these. Heck, I figure I could even use Adobe Reader to print the label into another PDF file with the print restrictions removed...So you gain nothing by preventing multiple printouts, but you can seriously annoy users (e.g. when their printer malfunctions)Please stop imposing such stupid restrictions on users.

Brian Aker dropped by and replied to my previous essay by making the following comment:

I believe you are hitting the nail on the organic vs nonorganic open source. I do not believe we have a model for going from one to the other. Linux and Apache both have very different models for contribution but I don t believe either are really optimized at this point. Optimization to me would lead to a system of less priests and more inclusion.

I made an initial reply as comment, and then decided it was so long that I should promote it to a top-level post. I assume that when Brian talks about “organic open source” what he means is what I was calling an “open source development community”. Some googling turned up the following definition from Mozilla Firefox’s organic software page: “Our most well-known product, Firefox, is created by an international movement of thousands, only a small percentage of whom are actual employees.” This puts it in contrast with “non-organic” software, where all or nearly all of the developers are employed by one company. (And anyone who proves talented at adding features to that source base soon gets a job offer by that one company. By that definition we can certainly see projects like Wine, Mysql, Ghostscript (at one time), and others as fitting into that model, and being quite successful. There’s nothing really wrong with the non-organic software model, although many of them have struggled to make enough money when competing with pure proprietary softare competitors, with MySQL perhaps being the exception which proves the rule. In most of these cases, though, the project started more as an organic open source, and then transitioned into the non-organic model when there was a desire to monetize the project and/or when the open source programmers decided that it would be nice if they could turn their avocation into a vocation, and let their hobby put food on the family table. Solaris, of course, is doing something else quite different, though. They are trying to make the transition from a proprietary customer/supplier relationship to trying to develop an Open Source community and what John s candidate statement pointed out is that they weren t really interested in creating an organic open source developer community at all, but they wanted the fruits of an open source community with plenty of application developers, end-users, etc., all participating in that community. We don t have a lot of precedent for projects who try to go in this direction, but I suspect they are skipping a step when they try to go to the end step without bothering to try to make themselves open to outside developers. And by continuing to act like a corporation, they end up shooting themselves in the foot. For example, the OpenSolaris license still prohibits people from publishing benchmarks or comparisons with other operating systems. Very common in closed-source operating systems and databases, but it discourages people from even trying to make things better, both within and outside of the Open Solaris core team. Instead, they respond to posts like David Miller s with Have you ever kissed a girl? . (Thanks, Simon, for that quote; I had seen it before, but not for a while, and it pretty well sums up the sheer arrogance of the Open Solaris development team.) So while Linux may not be completely optimized in terms of less priests and more inclusion, at least over 1200 developers contributed to 2.6.25 during its development cycle. Compared to that, Open Solaris is positively dominated by high priests and with a you may not touch the holy-of-holies attitude; heck, they won t even allow you to compare them to other religions without branding you a heretic and suing you for licensing violations!

Brian Aker dropped by and replied to my previous essay by making the following comment:

I believe you are hitting the nail on the organic vs nonorganic open source. I do not believe we have a model for going from one to the other. Linux and Apache both have very different models for contribution but I don t believe either are really optimized at this point. Optimization to me would lead to a system of less priests and more inclusion.

I made an initial reply as comment, and then decided it was so long that I should promote it to a top-level post. I assume that when Brian talks about “organic open source” what he means is what I was calling an “open source development community”. Some googling turned up the following definition from Mozilla Firefox’s organic software page: “Our most well-known product, Firefox, is created by an international movement of thousands, only a small percentage of whom are actual employees.” This puts it in contrast with “non-organic” software, where all or nearly all of the developers are employed by one company. (And anyone who proves talented at adding features to that source base soon gets a job offer by that one company. By that definition we can certainly see projects like Wine, Mysql, Ghostscript (at one time), and others as fitting into that model, and being quite successful. There’s nothing really wrong with the non-organic software model, although many of them have struggled to make enough money when competing with pure proprietary softare competitors, with MySQL perhaps being the exception which proves the rule. In most of these cases, though, the project started more as an organic open source, and then transitioned into the non-organic model when there was a desire to monetize the project and/or when the open source programmers decided that it would be nice if they could turn their avocation into a vocation, and let their hobby put food on the family table. Solaris, of course, is doing something else quite different, though. They are trying to make the transition from a proprietary customer/supplier relationship to trying to develop an Open Source community and what John s candidate statement pointed out is that they weren t really interested in creating an organic open source developer community at all, but they wanted the fruits of an open source community with plenty of application developers, end-users, etc., all participating in that community. We don t have a lot of precedent for projects who try to go in this direction, but I suspect they are skipping a step when they try to go to the end step without bothering to try to make themselves open to outside developers. And by continuing to act like a corporation, they end up shooting themselves in the foot. For example, the OpenSolaris license still prohibits people from publishing benchmarks or comparisons with other operating systems. Very common in closed-source operating systems and databases, but it discourages people from even trying to make things better, both within and outside of the Open Solaris core team. Instead, they respond to posts like David Miller s with Have you ever kissed a girl? . (Thanks, Simon, for that quote; I had seen it before, but not for a while, and it pretty well sums up the sheer arrogance of the Open Solaris development team.) So while Linux may not be completely optimized in terms of less priests and more inclusion, at least over 1200 developers contributed to 2.6.25 during its development cycle. Compared to that, Open Solaris is positively dominated by high priests and with a you may not touch the holy-of-holies attitude; heck, they won t even allow you to compare them to other religions without branding you a heretic and suing you for licensing violations!

Previously, my journal was managed by Movable Type. Originally, Six Apart seemed like a cool company, but I should have known better than to trust anything that wasn't open source. They started screwing with the license, it was never packaged properly, and then of course it became far too painful to upgrade. I've been wanting to fix this for some time, and have occasionally poked at other blog software. Serendipity looked like a winner for a while, but then I spent two days fighting with it and while I got it to mostly look like what I want, that was demoralizing. I hate spending large quantities of time fighting with someone else's software to get it to do what I want, particularly when I then have to write plugins and additions to get what feels to me like basic functionality. It would be one thing if I had time and inclination to get involved as a developer, but usually what Web 2.0 software developers want to pursue and what I care about are almost entirely at odds. During my recent vacation, I finally realized that I just wanted something that used all the infrastructure I already have. I have a macro language that I like for generating web pages, I have a page design that I like; the journal one isn't bad, but it doesn't match. I don't care about nearly all of the features of blog software. I could use something that's much better-designed (and free software) like ikiwiki, but I don't particularly like wikis either. (That's another story that I should write up sometime.) I really just wanted something that works like my book reviews work and lets me maintain the whole thing in spin. So I just gave up and wrote it. It probably took me more time than it would take to beat some piece of blog software into submission, but now it's done, I know how it works, it knows what I want to do, and I don't have to fight with it again with every upgrade. So this is my first post with my new journal software, with an RSS feed generated by my own code. There will probably be bugs for a while to iron out, but it will also make it far easier for me to maintain multiple RSS feeds with different topics, post reviews to my journal even more automatically, and add automatic posting for other things. The main loss is comments. With this system, my entire web site can be static, which I love. But that means no comment scripts. Since almost no one was commenting on my journal anyway (as opposed to mailing me or discussing posts elsewhere), I think I'm okay with that. Eventually I may mirror my journal somewhere else where there are comments. All of the old posts have been converted over, including old comments. Old reviews with no comments have become redirects to the permanent review pages. Let me know if you see anything weird.

A few news from the campaign of NMU's targeted at bringing po-debconf translation ratio as high as possible for as many languages as possible (and also improve packaging quality quite significantly on many loosely maintained packages).

• Galician translation are going up quickly. Each NMU features updates by Jacobo Tarrio and Galician is now ranked 10th in the translation ratio list. They could even hit the Spanish (Castillian) translation ratio and become the most translated language from Spain..:-)
• Finnish follows the same way. Starting from quite low, Esko Araj rvi bringed it to 13th rank and now above 50%
• Russian is also going up quite fast and could at least reach 70 or even 80%
• Portuguese (non Brazilian) hit the 90% mark and heads up to 100%
• German is very likely to be bringed to 100%. Crossed 97% today.
• French will be bringed to 100%, as it was for Etch. Close to 99% now.

Maintainers are incredibly cooperative (at least those who are active..:-)). l10n NMUs are always welcomed. Some prefer uploading themselves which is of course perfectly OK. I will now also focus my efforts to minimize uncoordinated changes to debconf templates. Those are generally motivated by the will to follow writing style guidelines, which is perfectly OK. I however still suggest maintainers to send call for translation updates *before* uploading packages with modified templates. Will probably soon post that suggestion to -release.

simple case Imagine a very simple debian/rules, that should work for most packages w/o modification:

#!/usr/bin/make -f
build:
    dh build
clean:
    dh clean
binary-arch:
    dh binary-arch
binary-indep:
    dh binary-indep
binary: build binary-arch binary-indep

absurdly simple case

AKA debian/rules zen guru level.
#!/usr/bin/make -f
%:
    dh %@

the complex cases Often you'll want to handle configuring by hand. (And maybe, run automake too.)

build:
    dh build --before configure
    #aclocal
    #autoheader
    #automake --add-missing
    #autoconf
    ./configure --kitchen-sink=yes
    dh build --after configure

Here's how to skip two steps in a row (configure and build):

build:
    dh build --before configure
    ./mondoconfig
    make universe-explode-in-delight GROOVINESS=420
    dh build --after build

Another common case is wanting to run some code manually after a particular debhelper command is run.

binary: build
    dh binary --until dh_fixperms
    # dh_fixperms has run, now override it for one program
    chmod 4755 debian/foo/usr/bin/foo
    # and continue
    dh binary

Suppose you hate dh_strip, and just don't want it to run. (This might be a contrived example; if people frequently don't want to run a few commands in a set, it may be better to have -X exclude them.)

binary: build
    dh binary --before dh_strip
    dh binary --after dh_strip

Maybe you want to run a command out of order. This should be no problem:

binary: build
    dh binary --until dh_fixperms
    dh_install foo usr/bin
    dh binary --after dh_fixperms

A separate install target can be added if you like. And maybe you want to run make install by hand in it. (This is tricky because dh binary needs to avoid running dh_auto_install which was skipped over in the install target.)

install: build
    dh install --before install
    $(MAKE) install PREFIX=debian/tmp
    dh install --after install
binary: install
    dh binary

Once you have an install target, you can do this to run some commands manually after the install target runs, and before the remaining steps are run to build a binary.

binary: install
    dh_strip -X foo
    dh_fixperms -X bar
    dh binary --remaining

If the --remaining was left off, dh binary would only run commands in its list after dh_fixperms. With --remaining, it runs commands that haven't run before, avoiding running dh_strip and dh_fixperms again. A more elaborate way to do the same thing, without needing a separate install target is this.

binary: build
    dh binary --before dh_strip
    dh_strip -X foo
    dh binary --after dh_strip --before dh_fixperms
    dh_fixperms -X bar
    dh binary --after dh_fixperms

the dh_auto_* commands dh_auto_* are thin wrappers, with enough intelligence to handle 90% of cases (autofoo, perl/python packages, etc) automatically. These will rarely be run by hand, instead if they don't do what is needed, they can just be skipped.

• dh_auto_configure: Figures out how to run autofoo, ./configure, Makefile.PL, or whatever.
• dh_auto_build: runs make
• dh_auto_test: runs make test or the like (may have a config file specifying if test failures should fail the build).
• dh_auto_clean: runs "make clean" etc
• dh_auto_install: runs "make install" or similar

the dh command dh runs a set of other debhelper commands in order (and prints their names as it does so). The set that it runs is determined by its parameter. The sets of commands are:

• build: dh_auto_configure, dh_auto_build, dh_auto_test
• clean: dh_testdir, dh_testroot(?), dh_auto_clean, and finally dh_clean.
• install: Everything in the build list, plus dh_installdirs, dh_install, dh_auto_install (and probably dh_install* too)
• binary: Everything in the install list, and all debhelper commands used in binary targets.
• binary-arch: Like binary, but runs commands with -s.
• binary-indep: Like binary, but does not run dhstrip or dhshlibdeps or dh_makeshlibs, and runs commands with -i.

Possibly dh does smart optimisations to avoid running lesser-used commands if they will do nothing. (Or not.) Each debhelper command will record when it's successfully run in debian/package.log.debhelper. (Which dh_clean deletes.) So dh can tell which commands have already been run, for which packages, and skip those. Options passed to dh are passed on to each command it runs, which is useful for common options like -v, -X, and the like. For that matter, you can use it for uncommon options as well, since commands will ignore options that don't apply to them. dh has a few options of its own:

• --until cmd: Run commands until and including cmd, then stop.
• --before cmd: Run commands before cmd, then stop.
• --after cmd: Run commands that are listed after cmd.
• --remaining: Run all listed commands that have yet to be run.

cmd can be a full name of a debhelper command, or a substring. It'll first search for a command exactly matching the name, to avoid any ambiguity. For example, "--before dh_installdeb" does not refer to dh_installdebconf. Otoh, "--before build" is much easier to type, and clearer, than "--before dhautobuild". If there are multiple substring matches, the last in the command list is taken. So "--before install" stops before dh_auto_install, and not before the earlier dh_install. Each time dh is run, it examines the log, and finds the last logged command that is in its list of commands to run. It then continues with the following command in its list. (Or the one after --after, if that was specified. Or if --remaining was specified, start at the first command in the list that is not in the log.) So... If you run debian/rules build twice, and it just has "dh build" in it, the second run will be a no-op. The same as if the build target touched a build-stamp. This is generally considered a feature, so I'll consider it a feature in dh too. (To force a rebuild, run dh_clean to remove the log.) If you run debian/rules binary, and a build hasn't happened yet, it will run the build steps first. Even if the rules file is a bit lame (or over-golfed), and doesn't have the binary target explicitly depend on the build target. Ditto for the install target. If a rules file runs a command out of order, it'll need to use --after to get dh back on track. If you manually run a debhelper command at the command line, running debian/rules binary will skip forward until after that command. Similarly, if debian/rules binary is interrupted (or fails), running it again will resume after the last debhelper command run. This is a bit of a behavior change, and could be considered a bug, but is easy enough to work around by running debian/rules clean binary. (Or by manually deleting the log.)