Aaron Huey gave a disturbing TED talk titled America s native prisoners of war [1]. He says the last chapter in any successful genocide is the one in which the oppressor can remove their hands and say my god, what are these people doing to themselves, they re killing each other, they re killing themselves while we watch them die .Peter Haas gave an interesting TED talk about how the poor engineering work in Haiti contributed to the significant death toll from the earthquake [2]. He advocates training for builders to prevent death and property damage from the next earthquake which is a lot cheaper than cleaning up the mess after buildings have fallen down.Wired has an interesting article on Phylo, a new crowd-sourced science game where you can sequence DNA [3]. The article also has links to other crowd-sourced science games.Brendan Scott gives a good summary of some of the most interesting news articles related to Wikileaks [4].Petter Reinholdtsen published a detailed and informative letter that Peruvian Congressman Edgar Villanueva wrote to Microsoft on the topic of a Peruvian bill to compell the government to use free software [5]. This has a lot of great ideas for anyone who wants to lobby their government for free software related legislation.Melissa McEwan wrote an informative blog post about why she doesn t trust men [6]. I can t do justice to this with a summary so just read it if you are male.Diana Laufenberg gave an interesting and inspiring TED talk about ways of teaching children [7]. Her main point was about embracing failure, having children learn from their mistakes. I think that perhaps embracing failure is only going to work with an exceptional teacher such as Diana, and that the majority of teachers would probably fail if they tried to implement it. She does have some really interesting examples of how she teaches so it s worth watching even if you don t agree with the central point.

• [1] http://www.ted.com/talks/aaron_huey.html
• [2] http://www.ted.com/talks/peter_haas_haiti_s_disaster_of_engineering.html
• [3] http://www.wired.com/wiredscience/2010/11/phylo-game/
• [4] http://brendanscott.wordpress.com/2010/12/04/not-your-fathers-civil-society/
• [5] http://people.skolelinux.org/pere/blog/The_reply_from_Edgar_Villanueva_to_Microsoft_in_Peru.html
• [6] http://shakespearessister.blogspot.com/2009/08/terrible-bargain-we-have-regretfully.html
• [7] http://www.ted.com/talks/diana_laufenberg_3_ways_to_teach.html

The bug webinterface of the Ultimate Debian Database currently knows about the following release critical bugs:

In Total:	608
Affecting Squeeze:	115
Squeeze only:	37
Remaining to be fixed in Squeeze:	78

Of these 78 bugs, the following tags are set:

Pending in Squeeze:	3
Patched in Squeeze:	19
Duplicates in Squeeze:	12
Can be fixed in a security Update:	3
Contrib or non-free in Squeeze:	4
Claimed in Squeeze:	0
Delayed in Squeeze:	3
Otherwise fixed in Squeeze:	11

Ignoring all the above (multiple tags possible) 41 bugs need to be fixed by Debian Contributors to get Debian 6.0 Squeeze released. However, with the view of the Release Managers, 85 need to be dealt with for the release to happen. Please see Interpreting the release critical bug statistics for an explanation of the different numbers.

It's midnight and I really want to be coding. I'm half way or so through adding a git-annex-shell to git-annex, which will enable some nice features with ssh access. I've done the research, and written the documentation[1], laid all the necessary refactoring groundwork, and written much of the code and just have to connect it all up and test it. This is a dangerous point to be at, because I have it all in my head, but I'm tired. If I start coding now, I could make a stupid mistake and find myself stuck in a blind alley at 5 am. I've been there before. Even if I avoid hard stuff, I'd surely look up and an hour or more would have gone by, it'd be past 1 am -- and I know that once I stop coding it can take me hours to spin my thoughts back down to the point where I can go to sleep. Also, I can feel the back of my mind still working on something. Some bits feel like they might not quite mesh up right in my mental model. I don't consciously know what the problem is. I need to sleep on it; it will probably be clearer later. Like yesterday when I lifted a bag of groceries into the car and paused, realizing code I'd written two months ago had a major bug, one I'd never seen, in an edge case that had never came up, but was surely there. I always assumed that when programmers got older and stopped coding late it was because they couldn't take the strain. Nah. We're just coding even as we sleep. :)

---

[1] Documentation is where I do my design. This is why git-annex has quantities of documentation that Linux Weekly News finds surprising.

After trying to compare Ogg Theora to the Digistan definition of a free and open standard, I concluded that this need to be done for more standards and started on a framework for doing this. As a start, I want to get the status for all the standards in the Norwegian reference directory, which include UTF-8, HTML, PDF, ODF, JPEG, PNG, SVG and others. But to be able to complete this in a reasonable time frame, I will need help. If you want to help out with this work, please visit the wiki pages I have set up for this, and let me know that you want to help out. The IRC channel #nuug on irc.freenode.net is a good place to coordinate this for now, as it is the IRC channel for the NUUG association where I have created the framework (I am the leader of the Norwegian Unix User Group). The framework is still forming, and a lot is left to do. Do not be scared by the sketchy form of the current pages. :)

It's after Christmas, and the stores are full of Android tablets ranging in quality from mediocre to competent. Obviously, they're also ranging in terms of GPL compliance from "utter failure" to "pretty good". I've written a summary page here so you have:

• Some idea of whether you're funding the theft of sweets from innocent children
• Some idea of whether there's any realistic chance of you getting further updates once the vendor has decided that last year's devices are, well, last year
• Some idea of just how bad the situation is

The vast majority of Android tablets I've been able to find are shipping without any source being made available, and that includes devices from well-known vendors. It's pretty much a given from the ones you've never heard of.

It's probably also worth noting that many of the devices in this list are probably rebadged versions of identical hardware, and that in some cases the same model may in fact refer to wildly different devices. There's a few cases where I've ended up with model numbers without any idea who the vendor is. If anyone has any corrections or updates, please feel free to let me know.

(Side note: People sometimes ask why Google aren't doing more to prevent infringing devices. For the vast majority of these cases, Google's sole contribution has been to put Android source code on a public website. Red Hat own more of the infringing code than Google do. There's no real reason why Google should be the ones taking the lead role here, and there's fairly sound business reasons why it's not in their interest to do so)

(This is part of a series of posts on Why Linux?) I find Linux more flexible. Maybe that's the familiarity showing, maybe it's about the package management, but it's a powerful reason for me to use it. For example, a couple of years ago I wanted to try out some iSCSI stuff against a SAN. Of course I have test boxes available I can do this on, but this was just to try out a few bits and pieces rather than anything more concrete. So I installed open-iscsi on my desktop and was able to merrily do the tests I wanted with very little additional work. Or I wanted to try out some BitKeeper to git conversion work recently. I wasn't sure how much resource it would take on a build server, and didn't want to tie things up there. So I ran it on my desktop overnight, where I could easily setup the appropriate environment and wouldn't impact on anyone else's resources. Problems talking to dodgy hardware? Linux is much better about giving you some idea what's going on, without needing to install extra software. I had a workmate grappling with an old USB music player recently; hooking it up to her Windows laptop wasn't providing a lot of joy so I attached it to my Linux box and was able to see that it did identify ok, but was disconnecting randomly at times too. Want to script querying an AD server for the current employee list and displaying who's joined and who's left since the last time you did so? I found that easy enough with the common Linux LDAP tools. I'm sure it's doable under Windows too, but I'm not sure it would be quite so simple. For bonus points add graphviz into the mix for automatic organisation charts (modulo accuracy of the AD data). This flexibility is something that helps me do my job. Sure, as I mentioned above I do have access to test boxes that I can use for this, but being able to do it on my desktop can be useful - for example if I'm offline, or on a slow network connection, or just geographically distant from my test machines so network latency is higher than I'd like. (Also, it's something that makes a Linux box a really great test box. I'm lucky in that I have a mix of OSes available to me for testing, but the one that I use most often is the Debian box. Much easier to get and install decent diagnosis tools for it that can give me packet level dumps, or do really odd stuff that turns out to be really useful.)

I just discovered that I'd been hit by #409864 since upgrading to Lenny. A very unfortunate bug that one. I'm surprised it hasn't been fixed in a point release.

... looking for an HOWTO? These days, possibly because it's holiday in several countries around the world, I'm receiving a higher number of "how can I help Debian?" mails than usual. This is good and makes me happy! Those mails show we have a community who cares about giving back to Free Software, in exchange of what they have received via Debian. Routinely, in my replies, I point people to the following list of documentation we have on how to contribute to Debian:

• "How can you help Debian", the official contribution page on www.d.o
• HelpDebian on the Debian wiki
• HelpDebian FAQ, companion FAQ of the above page
• Chapter "Contributing to the Debian Project" of the Debian FAQ

Am I missing something? At the same time, it happens that I reply to people who clearly went through (at least some of) the above resources, but which are still blocked at the "what I do next?" step. I guess this is because some of the day to day practices of Debian life are not described in the documentation we have. For instance, I find scarce references to "cultural" aspects of Debian technical life such as:

• a lot of coordination happens over IRC
• a typical way of starting collaborating on packaging is interacting with the package maintainers via the BTS, by doing triaging, submitting patches, etc.
• lots of packages are team-maintained (in fact, this is mentioned in a couple of places, but unfortunately our team list is not complete. Therefore by saying "choose from the list a team you're interested in and contact them", as I recurrently do, I know I'm leaving someone out)
• how to find a mentor
• ...

And I'm sure I'm missing quite some other cultural/folklore traits, which look obvious to longstanding members of the Debian community, but which are probably not so obvious to external people. So, dear lazyweb, have you started contributing to Debian and remember your experience? If so, I'd like to hear from you. In particular, I'd like to hear your suggestions on how we can improve our documentation on the matter (patches welcome, as usual!). Bonus point. Of the various pieces of documentation we have, it seems that a sufficiently detailed HOWTO-style document, covering both technical and cultural aspects, is missing. If you are into writing for Debian, that might be an experiment worth trying.

Dear Lazyweb, Today I bought me a Canon IXUS 130 and was quite disappointed to see that it couldn t properly connect to my Debian/Sid machine via USB. Apparenty the camera is not recognized as a USB Mass Storage Device but utilizes PTP. I upgraded to libgphoto2 (2.4.10.1-3) from experimental and added a proper udev rule with vendor- and product-id but it did not help. Is there anything more I can try? Update: Accessing the camera actually works, even with ligbphoto2 (2.4.6) from unstable. I can access the camera with gphoto2 and download pictures (but not with KDE s filemanager). My problem was that I had rhythmbox running and the camera was automatically mounted by it as a media device. When KDE s filemanager or gphoto2 tried to access the camera I got an error like: ( Could not lock the device ): Camera is already in use., which is probably also a bug. Closing rhythmbox solved the issue and I was able to download images with gphoto2.

After several months of leisurely development (I've read through everything on the Internet several times, and notified everyone who was wrong), I have finally gotten obnam, my backup application, into a state where it successfully backs up my laptop's hard disk to a USB disk. I therefore declare it to be perfe... no, wait, I don't. It's slow. It's probably buggy in ways that will make you want to abandon computers forever. It's also slow. And buggy. Did I mention it is slow, making backups at only a few megabytes per second? Disclaimers aside, I am happy to have gotten obnam to this stage. It's not nearly ready, but it's getting along nicely. If you've been curious about obnam ever since I first mentioned it in 2007, now might be a good time to try it out. I would be very interested in hearing any feedback. Obnam is implemented using copy-on-write B-trees, and I've also released my B-tree implementation. See the NEWS file in each source tree for information for what's changed since the previous release. Links:

• Obnam
• btree

I have made Debian packages of obnam, btree, and some of my other stuff that they need, for all, amd64, and i386 architectures. The repository is at http://code.liw.fi/debian/, so add this to your sources.list if you want to use that:

deb http://code.liw.fi/debian squeeze main

Note, however, that I do not guarantee that this repository will be in any useful state at any given time. I use it, and rely on it, and will fix things when they bother me, but that's all the guarantee I give. (Do you get the impression I've read too many contracts?) Once obnam is ready, and Debian squeeze is released, I'll upload it to Debian, and then you'll get it for any architecture Debian runs one. If anyone wants to help me maintain the packages for Debian, please e-mail me. If, on the other hand, you'd like to help make obnam faster, or better, the bug tracker has some pointers. Right now speed is perhaps the biggest problem. The B-tree library needs optimizing, as does obnam itself. And the LRU cache I use. The worst missing feature is encryption support. I do my backups to a local hard disk, which is full-disk encrypted, which is why I haven't cared about implementing encryption yet. Also, I need to design a good way to manage keys, etc. Help with that would be quite welcome.

There s this great feature in modern operating systems called the page cache. Simply put, it keeps in memory what normally is stored on disk and helps both read and write performance. While it s all nice for a day to day use, it often gets in the way when one wants to track performance issues with cold cache (when the files you need to access are not in the page cache yet). A commonly used command to flush the Linux page cache is the following:

# echo 3 > /proc/sys/vm/drop_caches

Unfortunately, its effect is broad, and it flushes the whole page cache. When working on cold startup performance of an application like Firefox, what you really want is to have your page cache in a state close to what it was before you started the application. One way to get in a better position than flushing the entire page cache is to reboot: the page cache will be filled with system and desktop environment libraries during the boot process, making the application startup closer to what you want. But it takes time. A whole lot of it. In one of my what if moments, I wondered what happens to the page cache when using posix_fadvise with the POSIX_FADV_DONTNEED hint. Guess what? It actually reliably flushes the page cache for the given range in the given file. At least it does so with Debian Squeeze s Linux kernel. Provided you have a list of files your application loads that aren t already in the page cache, you can flush these files and only these from the page cache. The following source code compiles to a tool to which you give a list of files as arguments, and that flushes these files:

#include <unistd.h>
#include <fcntl.h>
#include <sys/stat.h>

int main(int argc, char *argv[])  
  int i, fd;
  for (i = 1; i < argc; i++)  
    if ((fd = open(argv[i], O_RDONLY)) != -1)  
      struct stat st;
      fstat(fd, &st);
      posix_fadvise(fd, 0, st.st_size, POSIX_FADV_DONTNEED);
      close(fd);
     
   
  return 0;
 

It s actually a pretty scary feature, especially on multi-user environments, because any user can flush any file she can open, repeatedly, possibly hitting system performance. By the way, on systems using lxc (and maybe other containers, I don t know), running the echo 3 > /proc/sys/vm/drop_caches command from a root shell in a container does flush the host page cache, which could be dangerous for VPS hosting services using these solutions. Update: I have to revise my judgment, it appears posix_fadvise(,,,POSIX_FADV_DONTNEED) doesn t flush (parts of) files that are still in use by other processes, which still makes it very useful for my usecase, but also makes it less dangerous than I thought. The drop_cache problem is still real with lxc, though.

It gives me pleasure to announce RDiffFS, a virtual filesystem for browsing backups created by the rdiff-backup tool. Assuming that "/backup" is the location of a series of backups created using rdiff-backup, here is a usage example:

$ rdifffs /backup/dest/ /mnt
$ ls -l /mnt
total 5
drwxr-xr-x 4 jon jon 4096 Feb  7  2010 2010-09-04T16:38:10+01:00
drwxr-xr-x 4 jon jon 4096 Feb  7  2010 2010-12-29T10:51:50Z
drwxr-xr-x 4 jon jon 4096 Feb  7  2010 2010-12-29T10:52:30Z
lr-xr-xr-x 2 jon jon   11 Jan  1  1970 current -> 2010-12-29T10:52:30Z
$ for i in /mnt/*/a; do c=$(cat "$i"); printf "%32s:\t%s\n" "$i" "$c"; done
/mnt/2010-09-04T16:38:10+01:00/a:   foo
     /mnt/2010-12-29T10:51:50Z/a:   bar
     /mnt/2010-12-29T10:52:30Z/a:   baz
                  /mnt/current/a:   baz

I will write more about the creation of rdifffs at a later date: in particular my choice of implementation language, but I don't want to clutter up this announcement with all that. Suffice to say, I started writing a "quick tool" nearly a year ago, and I have become increasingly desperate to announce it. At present, rdifffs should be considered alpha quality software. It can browse my real-world backup, albeit slowly, and with several metadata bugs. This is more than archfs can manage. It can read files from the most recent backup, and most files from incremental backups, but not all.

As XTaran and uschebit asked yesterday at the 27C3 about my nice windowlist in irssi, here is what I use: adv_windowlist.pl. Download it, put it into you ~/.irssi/scripts and load it with /script load adv_windowlist. Now you have a list with all those window names in them, but it s still a bit ugly ;)First of all, get rid of the default Act list of irssi: /statusbar window remove actNow let s customize awl a bit:

• Shorten window names to 10 chars, so we can get more windows in one row:
  /set awl_block = 10
/set awl_sbar_maxlength = ON
• Display a maximum of 5 rows of windows so irssi is still usable on my Milestone with the 122 30 char screen:
  /set awl_maxlines = 5
• Shorten the windowname layout a bit, strip the shortcut display, remove the braces around the window number:
  /set awl_display_key = $N$H$C$S
/set awl_display_nokey = $N$H$C$S

Done! Your irssi should look like this now:

So when I was travveling to my parent's for christmas it looked like I'd have limited computer access. My Netbook is quite OK for reading mail but not really useable for any real hacking. And my trusty Thinkpad (Z61m) was oopsing when X was running so not much useable either. But as some Live CDs placed here were working well I decided that this would be fixed by an reinstall. And as I was reinstalling anyway I decided I could just choose kfreebsd-amd64. Turned out to be a quite entertaining decision with lots of stuff to hack away with wireless Bad news: there's no wireless support on Debian GNU/kFreeBSD at the moment. This problem is tracked as Bug #601803 so for wireless internet you will need a (plain) Freebsd chroot. Haven't tried this myself yet -- busy figuring other stuff out. SBCL Having a FreeBSD chroot I decided to give SBCL on GNU/kFreeBSD another try after having failed to get it working in a VM some time ago. With quite some help on SBCL's IRC channel I managed to build a patch that enables building (you need to force a :os-provides-dlopen to the feature list additionally). There's currently no multi-threading working so I hae a project for the rest of the hoidays (well lots of other stuff to do as well ;)) Audio Some more user-related stuff now. As it is this time of the year I wanted to listen to some 27c3 streams so I needed working audio. However there's no OSS device available. Turned out you just need to kldload the right module (here snd_hda) to get sound working. Volume was rather low although hardware controls of the soundcard where at max. As that's all OSS there's no point looking for alsamixer. Turns out aumix can do that here. IPv6 aiccu stuff Installing aiccu, copying the config in and starting did not work out as well. I already tried to do that from within the FreeBSD chroot already (which doesn't work for some reason) until I discovered just loading the if_tun kernel module solves the aiccu on Debian issue quite well. To get a default route up the last step was finding /lib/freebsd/route again -- /sbin/route is a wrapper around that abstracting differences in BSD route but not supporting IPv6.

Some people claim that AZERTY is a crime against humanity. I claim that if you're going to point and laugh at one layout in defense of another, you should have that other be something actually better, like the dvorak or Turkish F layout, not QWERTY. One additional reason: the mechanical layout of a US QWERTY keyboard (the ANSI layout) is inferior to the mechanical layout of the AZERTY keyboards. This is a mathematical fact the ISO layout has 102 keys, the ANSI one has only 101 of them. In that light, I find it highly disappointing that Thinkgeek.com's Das Keyboard uses the ANSI mechanical layout, and that there is no ISO version of the same thing. If there were, I'd have bought one years ago. (Yes, that's meant as a hint, in case you didn't catch that) Update: apparently it exists, it's just that ThinkGeek doesn't have it. For those who want one, it's available here, amongst others.

ntrack 011 heading for experimental: I added a ntrack-libnl binary package, for the libnl backend so it s in NEW queue for now.

John Goerzen blogs about moving pianos complete with outdoors music. While I don't have a similar story, my dad does. It doesn't involve a pickup (what with that being the standard american car type) but instead a piano which at an angle is stuck through the roof of a 2CV. I wasn't there (my parents hadn't even met yet), but I'm told it was great fun. In the not so distant future, however, I may have another story to tell about moving pianos. Currently, I live right above my office. Due mainly to bureaucratic reasons, this will change in the near future (it's a long story). Now when I moved into the apartment, moving the piano involved several people, a ladderlift, and some extra helping hands from another apartment a few blocks down the road where they just happened to be moving, too, just to lift it 10 cm up and over the window sill. Then, we just tried to avoid dropping it to the floor, about a meter down. Now if me, my three brothers, a friend, and an extra hand were only just enough to lift it for just 10 cm, I'm a bit worried as to how we'll be moving it out again, which would involve moving it up about a meter. To be continued, I'm sure.

Perhaps I am not very imaginative. Or perhaps insufficiently adventurous. But when I woke up today, a day that would actually get a little bit above freezing, I didn t have a thought of riding without a coat in the bed of a pickup. Nor did I expect to be involved with moving a piano. I didn t have any thought of listening to piano music outdoors. And I certainly didn t anticipate the fun to be had doing all of these things simultaneously. Now, let me back up and set the scene. This was our annual Goerzen Christmas today. We already ate and exchanged gifts. Here s the thing about moving pianos, especially the old heavy ones with the cast iron frame inside: it s not something where a group of people just get together, count to 3, and that s that. No, you ve got to start the project out right. You have to break the news that people will be moving a piano gently. And then, of course, you get a room full of people usually mostly men and at this point, before any lifting happens, you ve got to plan. Tape measures must be produced, measurements taken, plans made for how to lift and who should lift where. Differing points of view have to be considered carefully. Then, of course, you have too many people for each one to actually help lift so each person not lifting has to find a job. And then when things actually start going, you realize that not one of the plans fully accounted for the unprecedented weight and maddening stubbornness of the piano, and you have to stop every few feet to plan anew. I don t think I ve ever seen a piano move happen in any other fashion. So, today, my uncle, whose house our gathering was at this year, said, Ahh good, lots of young backs. Just what I need. Mildly concerned glances pass between my brothers, my cousins, and I. One brother looks like he s pretending not to have heard. We have a few things to move. Anticipation. A small couch probably not too bad, right? An upholstered chair maybe a bit unruly but still not too bad. And an old piano is going to a neighbor. Full alert now more glances, a pause for a second, and then acceptance: Oh! Uhm great! My uncle s friend Ken teaches music at the local college, and on the side, Ken also teaches children how to play piano. One of Ken s students didn t have a piano at home, and my uncle and aunt had an old one they weren t using, which they would give to the family of the student just a couple of blocks away. All that remained was moving it. There was the inevitable measuring, planning, re-planning, until we got it out of the house and onto the back of my uncle s pickup. We have about 8 men involved, plus my aunt. Most of us rode in the back of the pickup with the piano. And of course, there came the inevitable request: Ken, play for us! So Ken sat on the edge of the pickup, at the piano, playing as we and the piano went through town in the open air. He started with Joplin s Entertainer, then broke out into some Christmas music We Wish You a Merry Christmas, followed by Nun Ist Sie Erschienen and a few others. It must be said that small towns in Kansas are probably not accustomed to their Christmas carolers having a piano with them, much less being played from the bed of a pickup. But there we were, mostly laughing too hard or enjoying the ride too much to sing. To add excitement, or perhaps embarrassment, my uncle was sounding the horn. Some thought that his intent was to have it in time with the piano music from the back, but I ll give him the benefit of the doubt and say that he meant to sound the horn in that rather random fashion. Eventually we got to the neighbor s house, and my dad and another uncle who hadn t been on the pickup came running up. We hopped down and asked them where they wanted the piano. Upstairs! That got repeated among everyone there, producing laughter each time. More tape measures were produced, as it was a narrow staircase. The conclusion was that the piano could make it with about an inch (2.54cm) to spare. So we got it out of the pickup and onto the porch, then into the house. Now, how to get it upstairs? This called for an extra helping of planning and calculating. Ken, the unofficial piano moving boss, kept reminding us in jest that I grew up on a farm . He said he could be alone at the top of the piano while we pushed. This quickly turned out to not be practical; due to the incline, the weight at the bottom was just too much to get it up over the stairs. Ken then went and picked up a wooden contraption they use to move pianos at the college. We tried that, but its supports underneath got stuck on the steps as well. Various ideas were tossed around at this point, until I pointed out that four of us on the bottom couldn t get it up over a step, so when we get it into the stairway only wide enough for 2 we won t be able to get it up anyhow. That being considered a good point, various other ideas were tossed around. They all seemed to be wanting either for practicality or, perhaps, safety. Though the idea of having a dozen Goerzens stand at the top of the staircase pulling it up by a rope did have some excellent humor value. In the end, perhaps wanting our group of free, highly ambitious, but somewhat less effective piano movers out of their house, the neighbors found a place for it on the ground floor. We all piled back into the pickup for the ride back, which was rather more quiet due to lack of horn and piano music. So there you have it. The most fun I have ever had moving a piano. (Really, the only fun I have ever had moving a piano.)

A few weeks ago I was lamenting the lack of of a reverse proxy that understood Apache's mod_rewrite syntax. On my server I have a bunch of thttpd processes, each running under their own UID, each listening on local ports. For example I have this running:

thttpd -C /etc/thttpd/sites.enabled/blog.steve.org.uk

This is running under the Unix-user "s-blog", with UID 1015, and thus is listening on 127.0.0.1:1015:

steve@steve:~$ id s-blog
uid=1015(s-blog) gid=1016(s-blog) groups=1016(s-blog)
steve@steve:~$ lsof -i :1015
COMMAND   PID   USER   FD   TYPE  DEVICE SIZE NODE NAME
thttpd  26293 s-blog    0u  IPv4 1072632       TCP localhost:1015 (LISTEN)

Anyway in front of these thttpd processes I have Apache. It does only two things:

• Expands mod_rewrite rules.
• Serves as a proxy to the back-end thttpd processes.

Yes other webservers could be run in front of these processes, and yes other webservers have their own rewrite-rule-syntax. I'm going to say "Lala lala can't hear you". Why? Because mod_rewrite is the defacto standard. It is assumed, documented, and included with a gazillion projects from wikipedia to wordpress to ... So this weekend I decided I'd see what I needed to do to put together a simple proxy that only worked for reverse HTTP, and understood Apache's mod_rewrite rules. I figure there are three main parts to such a beast:

Be a network server

Parse configuration file, accept connections, use fork(), libevevent(), or similar such that you ultimately receive HTTP requests..

Process HTTP Requests, rewriting as required

Once you have a parsed HTTP-request you need to test against each rule for the appropriate destination domain. Rewriting the request as appopriate.

Proxy

Send your (potentially modified) request to the back-end, and then send the response back to the client.

Thus far I've written code which takes this:

GET /etc/passwd?root=1 HTTP/1.0
Host: foo.example.com:80
Accept-Language: en-us,en-gb?q=0.5
Refer: http://foo.bar.com/
Keep-Alive: 300
Connection: keep-alive

Then turns it into this:

struct http_request
 
  /*
   * Path being requested. "/index.html", etc.
   */
  char *path;
  /*
   * Query string
   */
  char *qs;
  /**
   * Method being requested "GET/POST/PUT/DELETE/HEAD/etc".
   */
  char *method;
  /**
   * A linked list of headers such as "Referer: foo",
   * "Connection: close"
   */
  struct http_headers *headers;
 ;

There are methods for turning that back to a string, (so that you can send it on to the back-end), finding headers such as "Referer:", and so on. The parser is pretty minimal C and takes only a "char *buffer" to operate on. It has survived a lot of malicious input, as proved by a whole bunch of test cases. My next job is to code up the mod_rewrite rule-processor to apply a bunch of rules to one of these objects - updating the struct as we go. Assuming that I can get that part written cleanly over the next week or two then I'll be happy and can proceed to write the networking parts of the code - both the initial accepting, and the proxying to the back-end. In terms of configuration I'm going to assume something like:

/etc/proxy/global.conf                    : Global directives.
/etc/proxy/steve.org.uk/
/etc/proxy/steve.org.uk/back-end          : Will contain 127.0.0.1:1019
/etc/proxy/steve.org.uk/rewrite.conf      : Will contain the domain-wide rules
/etc/proxy/blog.steve.org.uk/
/etc/proxy/blog.steve.org.uk/back-end     : Will contain 127.0.0.1:1015
/etc/proxy/blog.steve.org.uk/rewrite.conf : Will contain the domain-wide rules
..

That seems both sane & logical to me. ObQuote: "I shall control the fate of the world... " - Day Watch. ObRandom: Coding in C is pleasant again.