Search Results: "xam"

13 January 2011

Dominique Dumont: Debian copyright DEP5 parser/editor/validator/migrator is released

Hello The new version of libconfig-model-perl 1.229 is now available in Debian/Sid. This new version provides a model of Debian s DEP-5 copyright file specification updated according to the CANDIDATE version. (Among other features, like editor/validator for debian/control). The DEP-5 editor must be run in the source package directory (I ll provide an example below) with the following command:
config-edit -application dpkg-copyright
Features: Known issues: Example: The copyright file of libconfig-model-backend-augeas-perl needs to be updated: Here s the original copyright file (slightly edited to protect my mail box and save electrons) :
$ cat debian/copyright
Format-Specification: http://wiki.debian.org/Proposals/CopyrightFormat?action=recall&rev=196 Upstream-Maintainer: Dominique Dumont (ddumont at cpan.org)
Upstream-Source: http://search.cpan.org/dist/Config-Model-Backend-Augeas/
Upstream-Name: Config-Model-Backend-Augeas Copyright: 2008-2009, Dominique Dumont (ddumont at cpan.org)
License: LGPL-2+ Files: debian/*
Copyright: 2009, Dominique Dumont
License: LGPL-2+ License: LGPL-2+
This program is free software; you can redistribute it and/or
[snip]
First, let s just check issues:
$ config-edit -application dpkg-copyright -ui none
Element Upstream-Maintainer of node Debian::Dpkg::Copyright is deprecated
Element Upstream-Source of node Debian::Dpkg::Copyright is deprecated
Element Upstream-Name of node Debian::Dpkg::Copyright is deprecated
Missing Files: specification at top of section number 2. Adding Files: * spec
Then, let s see what s config-edit can fix all by itself:
$ config-edit -application dpkg-copyright -ui none -save
[ same warnings displayed ]
Let s check the result of the migration:
$ config-edit -application dpkg-copyright -ui none
[ good: no more warnings ]
And here s the content of the upgraded file (drum rolls, please):
$ cat debian/copyright
Format: http://dep.debian.net/deps/dep5/
Upstream-Name: Config-Model-Backend-Augeas
Upstream-Contact: Dominique Dumont (ddumont at cpan.org)
Source: http://search.cpan.org/dist/Config-Model-Backend-Augeas/ Files: *
Copyright: 2008-2009, Dominique Dumont (ddumont at cpan.org)
License: LGPL-2+ Files: debian/*
Copyright: 2009, Dominique Dumont
License: LGPL-2+ License: LGPL-2+
This program is free software; you can redistribute it and/or
[snip]
You can see that: Please provide feedback! If you have problem migrating DEP5 files, yell, and we ll fix them. If this migration works fine, don t be shy, please say so. Under the hood:

Most of the migration work is specified in DEP-5 model [2]. Using config-model s GUI [3] to update DEP-5 model, about one hour was spent to: This shows one important feature of Config::Model: maintenance and evolution of a model is not a huge task. Even though copyright files are not configuration files, the same approach can be used to specify model and migration of other configuration files. Now comes the question of the migration as seen by the user. Well, it s your turn now. Try it and tell us how it works! Hopefully, encouraged by your feedback, other people will apply config-model to other specifications or other configuration. All the best (*) E.g, in the text format produced by the model editor GUI: 'Upstream-Contact',

'value_type' => 'string',
'migrate_from' =>
'formula' => '$old_maintainer',
'variables' =>
'old_maintainer' => '- Maintainer'

,
'type' => 'leaf',
'description' => '* Syntax: line based list [snip]'
,

12 January 2011

Mike Hommey: Attempting to track I/O with systemtap

There are several ways a program can hit the disk, and it can be hard to know exactly what s going on, especially when you want to take into account the kernel caches. This includes, but is not limited to: There are various ways to track system calls (e.g. strace) allowing to track what a program is doing with files and directories, but that doesn t tell you if you re actually hitting the disk or not. There are also various ways to track block I/O (e.g. blktrace) allowing to track actual I/O on the block devices, but it is then hard to back-track what part of which files or directories these I/O relate to. To the best of my knowledge, there are unfortunately no tools to do such tracking easily. Systemtap, however, allows to access the kernel s internals and to gather almost any kind of information from any place in a running kernel. The downside is that it means you need to know how the kernel works internally to gather the data you need ; that will limit the focus of this post. I had been playing, in the past, with Taras script, which he used a while ago to track I/O during startup. Unfortunately, it became clear something was missing in the picture, so I had to investigate what s going on in the kernel. Setting up systemtap On Debian systems, you need to install the following packages: That should be enough to pull all the required dependencies. You may want to add yourself to the stapdev and stapusr groups, if you don t want to run systemtap as root. If, like in my case, you don t have enough space left for all the files in /usr/lib/debug, you can trick dpkg into not unpacking files you don t need:
# echo path-exclude /usr/lib/debug/lib/modules/*/kernel/drivers/* > /etc/dpkg/dpkg.cfg.d/kernel-dbg
The file in /etc/dpkg/dpkg.cfg.d can obviously be named as you like, and you can adjust the path-exclude pattern to what you (don t) want. In the above case, kernel drivers debugging symbols will be ignored. Please note that this feature requires dpkg version 1.15.8 or greater. Small digression One of the first problems I had with Taras script is that systemtap would complain that it doesn t know the kernel.function("ext4_get_block") probe. It is due to a very unfortunate misfeature of systemtap, where the kernel.* probes refer to whatever is in the vmlinux image. Modules probes have a separate namespace, namely module("name").*. So for the ext4_get_block() function, this means you need to set a probe for either kernel.function("ext4_get_block") or module("ext4").function("ext4_get_block"), depending how your kernel was compiled. And you can t even use both in your script, because systemtap will complain about either being an unknown probe Tracking the right thing I was very recently pointed to a Red Hat document containing 4 useful systemtap scripts ported from dtrace, which gives me a good occasion to explain the issue at hand with the first of these scripts. This script attempts to track I/O by following read() and write() system calls. Which is not tracking I/O, it is merely tracking some system calls (Taras script had the same kind of problem with read()/write() induced I/O). You could just do the very same with existing tools like strace, and that wouldn t even require some system privileges. To demonstrate the script doesn t actually track the use of storage devices as the document claims, consider the following source code: 
#include <fcntl.h>
#include <sys/stat.h>
#include <unistd.h>
/* Ensure the resulting binary is decently sized */
const char dummy[1024 * 1024] = "a";
int main(int argc, char *argv[])  
  struct stat st;
  char buf[65536];
  int i, fd = open(argv[0], O_RDONLY);
  for (i = 0; i < 100000; i++)  
    read(fd, buf, 65536);
    lseek(fd, 0, SEEK_SET);
    read(fd, buf, 65536);
    lseek(fd, 512 * 1024, SEEK_SET);
   
  close(fd);
  return 0;
All it does is reading some parts of the executable a lot of times (notice the trick to make the executable size at least 1MB). Not a lot of programs will actually do something as bold as reading the same data again and again (though we could probably be surprised), but this easily points the problem. Here is what the output of the systemtap script looks like for this program (stripping other unrelevant processes): 
         process     read   KB tot    write   KB tot
            test   400002 25600001        0        0
Now, do you really believe 25MB have actually been read from the disk? (Note that the read() count seems odd as well, as there should only be around 200000 calls) Read-ahead, page cache and mmap() What the kernel actually does is that a read() on a file is going to check the page cache first. If there is nothing corresponding to the read() request in the page cache, then it goes down to the disk, and fills page cache. But as loading only a few bytes or kilobytes from the disk could be wasteful, the kernel also reads a few more blocks ahead, apparently with some heuristic. But read() and write() aren t the sole way a program may hit the disk. On UNIX systems, a file can be mapped in memory with the mmap() system call, and all accesses in the memory range corresponding to this mapping will be reflected on the file. There are exceptions, depending on how the mapping is established, but let s keep it simple. There is a lot of litterature on the subject if you want to document yourself on mmap(). The way the kernel will read from the file, however, is quite similar to that of read(), and uses page cache and read ahead. The systemtap script debunked above doesn t track these. I ll skip write accesses, because for now, they haven t been in my scope. Tracking some I/O with systemtap What I ve been trying to track so far has been limited to disk reads, which happen to be the only accesses occurring on shared library files. Programs and shared libraries are first being read() from, so that the dynamic linker gets the ELF headers and knows what to load, and then are mmap()ed following PT_LOAD entries in these headers. As far as my investigation in the Linux kernel code goes, fortunately, both accesses, before they end up actually hitting the disk, go through the __do_page_cache_readahead kernel function (this function was tracked in Taras script). Unfortunately, while it is called with an offset and a number of pages to read for a given file, it turns out the last pages in that number are not necessarily being read from the disk. I don t know for sure, because the latter had an effect on my observations, but some might even already be in the page cache. Going further down, we reach the VFS layer, which ends up being file-system specific. But fortunately, a bunch of (common) file-systems actually share page mapping code, and commonly use the mpage_readpage and mpage_readpages functions, both calling do_mpage_readpage to do the actual work. And this function seems to be properly called only once for each page not in the page cache already. If my reading of the kernel source is right, this however is not really where it ends, and do_mpage_readpage doesn t actually hit the disk. It seems to only gather some information (basically, a mapping between storage blocks and memory) that is going to be submitted to the block I/O layer, which itself may do some fancy stuff with it, such as reordering the I/Os depending on other requests it got from other processes, the position of the disk heads, etc. And when I say do_mpage_readpage doesn t actually hit the disk, I m again simplifying the issue, because it actually might, as there might be a need to read some metadata from the disk to know where some blocks are located. But tracking metadata reads is much harder, and I haven t investigated it. Anyways, skipping metadata, going further down after do_mpage_readpage is hard because it s difficult to back-track which block I/O is related to which read-ahead, corresponding to what read at what position in which file. do_mpage_readpage already has part of this problem because it is not called with any reference to the corresponding file. But __do_page_cache_readahead is. So knowing all the above, here is my script, the one I used to get the most recent Firefox startup data you can find in my last posts: 
global targetpid;
global file_path;
probe begin  
  targetpid = target();
 
probe kernel.function("__do_page_cache_readahead")  
  if (targetpid == pid())
    file_path[tid()] = d_path(&$filp->f_path);
 
probe kernel.function("do_mpage_readpage")  
  if (targetpid == pid() && (tid() in file_path))  
    now = gettimeofday_us();
    printf("%d %s %d\n", now, file_path[tid()], $page->index*4096);
   
 
probe kernel.function("__do_page_cache_readahead").return  
  if (targetpid == pid())
    delete file_path[tid()];
This script needs to be used with a command given to systemtap, with the -c option, such as in the following command line:
# stap readpage.stp -c firefox
Each line of output represents a page (i.e. 4,096 bytes) being read, and contains a timestamp, the name of the file being read, and the offset in the file. As discussed above, do_mpage_readpage is not really the place the I/O actually occurs, so the timestamps are not entirely accurate, and the actual read order from disk might be slightly different, but it still is a quite reliable view in that the result should be reproducible with the same files even when they re not located on the same blocks on disk, and provided their page cache status is the same when starting the program. This systemtap script ignores writes, as well as metadata accesses (including but not limited to inodes, dentries, bitmap blocks and indirect blocks). It also doesn t account accesses to files opened with the O_DIRECT flag or similar constructs (raw devices, etc.) Read-ahead in action Back to the small example program, my systemtap script records 102 page accesses, that is, 417,792 bytes, much less than the actual binary size on my system (1,055,747 bytes). We are still far from the 25MB figure from the other systemtap script. But we are also far from the 128KiB the program actively reads (64KiB twice, leaving a hole between the two blocks). At this point, it is important to note that the ELF headers and program code all fit within a single page (4 KiB), and following the 1MiB section corresponding to the dummy variable, there are only 5,219 bytes of other ELF sections, including the .dynamic section. So even counting everything the dynamic linker needs to read, and the program code itself, we re still far from what my systemtap script records. Grouping consecutive blocks with consecutive timestamps, here is what can be observed:
Offset Length
0 16,384
983,040 73,728
16,384 262,144
524,288 65,536
(By now, you should have guessed why I wanted that big hole between the read()s ; if you want to reproduce at home, I suggest you also use my page cache flush helper) As earlier investigations showed, the first accesses by the dynamic loader are to read the ELF headers and .dynamic section. As mentioned above, these are really small. Yet, the kernel actually reads much more: 16KiB at the beginning of the file when reading the ELF headers, and 72KiB at the end when reading the .dynamic section. Subsequent accesses from the dynamic loader are obviously already covered by these accesses. Next accesses are those due to the program itself. The program actively reads 64KiB at the beginning of the file, then 64KiB starting at offset 524,288. For the first read, the kernel already had 16KiB in the page cache, so it didn t read them again, but instead of reading the remainder, it reads 256KiB. For the second read, however, it only reads the requested 64KiB. As you can see, this is far from being something like you wanted n KiB, I ll read that fixed amount now . Further testing with different read patterns (e.g. changing the hole size, read size, or reading from the dummy variable directly instead of read()ing the binary) is left as an exercise to the reader.

8 January 2011

David Bremner: Beamer overlays in highlighted source code

I use a lot of code in my lectures, in many different programming languages. I use highlight to generate HTML (via ikiwiki) for web pages. For class presentations, I mostly use the beamer LaTeX class. In order to simplify generating overlays, I wrote a perl script hl-beamer.pl to preprocess source code. An htmlification of the documention/man-page follows.
NAME hl-beamer - Preprocessor for hightlight to generate beamer overlays. SYNOPSIS hl-beamer -c // InstructiveExample.java highlight -S java -O latex > figure1.tex DESCRIPTION hl-beamer looks for single line comments (with syntax specified by -c) These comments can start with @ followed by some codes to specify beamer overlays or sections (just chunks of text which can be selectively included). OPTIONS CODES EXAMPLE Example input follows. I would probably process this with 
hl-beamer -s 4 -k encodeInner
Sample Input 
 // @( omit
 import java.io.BufferedReader;
 import java.io.FileReader;
 import java.io.IOException;
 import java.io.Serializable;
 import java.util.Scanner;
 // @)
     // @( encoderInner
     private int findRun(int inRow, int startCol) 
         // @<
         int value=bits[inRow][startCol];
         int cursor=startCol;
         // @>
         // @<
         while(cursor<columns && 
               bits[inRow][cursor] == value) 
             //@<
             cursor++;
             //@>
         // @>
         // @<
         return cursor-1;
         // @>
      
     // @)
BUGS AND LIMITATIONS Currently overlaytype and section must consist of upper and lower case letters and or underscores. This is basically pure sloth on the part of the author. Tabs are always expanded to spaces.

7 January 2011

Peter Eisentraut: Git commit mode

Hardly anything ruins a glorious day of coding like fat-fingering the commit message late at night as you doze off, and then pushing it out for the world to see. To prevent that, I have equipped my Emacs configuration with a few little tools now.

First, I found the git-commit-mode, a special mode for Git commit messages. This helps you format the commit messages according to convention, and will use ugly colors if, for example, you write lines that are too long or you do not keep the second line blank. It also allows the use of things like M-q without shredding the whole file template.

Second, I integrated on-the-fly spell checking into the git-commit-mode. It won't stop you from writing nonsense, but it will catch the silly mistakes.

Here's a simple configuration snippet:
(require 'git-commit)
(add-hook 'git-commit-mode-hook 'turn-on-flyspell)
(add-hook 'git-commit-mode-hook (lambda () (toggle-save-place 0)))
The last line is handy if you have save-place on by default. When you make a new commit, it would then normally place the cursor where a previously edited commit message was finished, because to the save-place functionality, it looks as though it's the same file.

Rapha&#235;l Hertzog: How to create custom RSS feeds with WordPress

WordPress has many alternate built-in feeds: per category, per tag, per author, per search-keyword. But in some cases, you want feeds built with some more advanced logic. Let s look at the available options. WordPress advanced built-in feeds You can create feeds for unions or intersections of tags, you just have to use a URL like /tag/foo,bar/feed/ (all articles tagged with foo or bar) or /tag/foo+bar/feed/ (all articles tagged with foo and bar). You can also have feeds excluding a category, although that requires you to know the category identifier (and hardcode it in the URL like this: /?feed=rss2&cat=-123 where 123 is the category id that you want to exclude). But there s no simple way to have a feed that excludes articles with a given tag. The best solution found involves creating a custom feed. I ll show you a variation of this below. Creating a custom feed
  1. First of, install the Feed Wrangler plugin, it will take care of registering our custom feeds with wordpress.
  2. Go to Settings > Feed Wrangler in your WordPress administrative interface and create a new feed, let s call it myfeed .
  3. You should now create a feed-myfeed.php file and put it in your current theme s directory. The initial content of that file should be this: 
    <?php
include('wp-includes/feed-rss2.php');
?>
  4. At this point, you already have a new feed that you can access at /feed/myfeed/ (or /?feed=myfeed). It s a complete feed like the main one.
Now, we re going to look at ways to customize this feed. We re going to do this by changing/overriding the default query that feed-rss2.php s loop will use. A feed excluding articles with a tag If you want to create a feed that excludes the tag foo , you could use this: 
<?php
global $wp_query;
$tag = get_term_by("slug", "foo", "post_tag");
$args = array_merge(
        $wp_query->query,
        array('tag__not_in' => array($tag->term_id))
);
query_posts($args);
include('wp-includes/feed-rss2.php');
?>
That was relatively easy, thanks to the tag__not_in query parameter. Now you can further customize the feed by adding supplementary query parameters to the $args array. The documentation of query_posts details the various parameters that you can use. A feed excluding articles with a custom field (meta-data) I went further because I did not want to use a tag to exclude some posts: that tag would have been public even if it was only meaningful to me. So I decided to use a custom field to mark the posts to exclude from my new feed. I named the field no_syndication and I always give it the value 1 . This time it s not so easy because we have no query parameter that can be used to exclude posts based on custom fields. We re going to use the post__not_in parameter that can be used to exclude a list of posts. But we must first generate the list of posts that we want to exclude. Here we go: 
<?php
global $wp_query;
$excluded = array();
$args_excluded = array(
    'numberposts'     => -1,
    'meta_key'        => 'no_syndication',
    'meta_value'      => 1,
    'post_type'       => 'post',
    'post_status'     => 'published'
);
foreach (get_posts($args_excluded) as $item)  
        $excluded[] = $item->ID;
 
$args = array_merge(
        $wp_query->query,
        array('post__not_in' => $excluded)
);
query_posts($args);
include('wp-includes/feed-rss2.php');
?>
A feed with modified content You might want to add a footer to the articles that are syndicated. I use the Ozh Better Feed plugin for this but it applies to all your feeds. You could do that sort of transformation only in your customized feed by using the WordPress filter named the_content_feed. Here s a simple example: 
<?php
function myfeed_add_footer($content)  
        return $content . "<hr/>My footer here";
 
add_filter('the_content_feed', 'myfeed_add_footer');
include('wp-includes/feed-rss2.php');
?>
I ll stop here but obviously you have lots of options and many ways to tweak all the snippets above. They have been tested with WordPress 3.0.4. Note that in all those examples, I took care to not duplicate the code from feed-rss2.php, instead I used include() to execute it. That way my custom feeds will automatically benefit from all the future enhancements and fixes made by the WordPress developers. But if you have to modify the XML structure of your custom feeds, you can paste the content of feed-rss2.php in your file and change it like you want

One comment Liked this article? Click here. My blog is Flattr-enabled.

6 January 2011

Michael Prokop: Booting ISO images from within GRUB2

You might be aware of GRUB s loopback option for booting an ISO, I wrote about it in Boot an ISO via Grub2 more than a year ago. A few months ago Goswin von Brederlow came up with this idea:
grml functions great as rescue system. So it would be nice to have a boot entry for it in grub instead of having to go look for the CD when needed. To make installing and updating simple it would be great if one could install this as a normal debian package which would register itself in grub (and maybe lilo too).
What a lovely idea: no need for a CD or USB stick as long as the bootloader and the harddisk are still working. Minimal manual intervention needed to keep it up2date and working sounds like the perfect rescue system. 8-) Now since end of 2010 a new stable version of sysadmin s favorite live system (AKA Grml 2010.12) is available and the great news is that we came up with two independent solutions known as grml-rescueboot and grub-imageboot. Having shipped them to one of my customers already I d like to write some words about it and why you should also consider using it on all your systems where GRUB2 is available. grml-rescueboot uses GRUB2 and its loopback feature for booting. Grml as well as Ubuntu provide all what s needed out-of-the-box. If you re interested in the details check out Jordan Uggla s excellent Loopback.cfg webpage in the supergrubdisk wiki. The best about it: you can provide custom bootoptions automatically when booting Grml. For example just set CUSTOM_BOOTOPTIONS= ssh=grml2011 in /etc/default/grml-rescueboot and the Grml rescue system will automatically start the OpenSSH server with the specified argument as password for user grml. Of course you can use all the other nifty bootoptions like scripts/netscript/ to further customize your Grml rescue system. [A note to other distributions like Ubuntu: I'd be interested to establish a mechanism to pass kernel options for loopback boot in a standardized way, drop me a note if you're interested in this.] grub-imageboot uses GRUB2 and syslinux memdisk to boot ISOs and floppy images. It doesn t rely on the loopback feature but instead maps the ISO into the memory directly. This is great for Linux ISOs that can t and won t support the the loopback feature (and have everything inside their initrd so the ISO doesn t have to find itself during booting), like BIOS or firmware updates. Also for example FreeDOS and Alpharev are known to work just fine. But sadly memdisk ISO emulation doesn t work with all Linux systems, as documented in the syslinux wiki. The good news is that Grml supports the memdiskfind/phram/mtdblock approach out-of-the-box. As far as I know therefore Grml is the first Debian based live system supporting memdisk ISO boot, though my patches already went to the Debian-Live project so if you re using live-boot >=2.0.14-1 or >=2.0.12-1+grml.04 your live system should be able to boot via memdisk ISO emulation as well. Summary: If you want to boot a Linux system which supports loopback.cfg use grml-rescueboot as its a more powerful tool to boot Linux live systems like Grml and Ubuntu. If you want to boot non-Linux systems (BIOS-/Firmware-Updates/FreeDOS/ .) use grub-imageboot. Alright how do you deploy this solution? Just grab and install grml-rescueboot and/or grub-imageboot. To deploy grml-rescueboot (adjust $VERSION if you want to use another flavour): 
# choose Grml version:
VERSION=grml64-medium_2010.12
# create directory
mkdir -p /boot/grml
# download and verify ISO
cd /boot/grml
wget download.grml.org/$ VERSION .iso ,.md5 
md5sum -c $ VERSION .iso.md5
To deploy grub-imageboot: 
# choose Grml version:
VERSION=grml64-medium_2010.12
# create directory
mkdir -p /boot/images
# download and verify ISO
cd /boot/images
wget download.grml.org/$ VERSION .iso ,.md5 
md5sum -c $ VERSION .iso.md5
That s it! Now when running update-grub you should get something like: 
# update-grub
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.36-grml64
Found initrd image: /boot/initrd.img-2.6.36-grml64
Found memtest86+ image: /boot/memtest86+.bin
Found memtest86+ multiboot image: /boot/memtest86+_multiboot.bin
  No volume groups found
Found Grml ISO image: /boot/grml/grml64-medium_2010.12.iso
Found memdisk: /boot/memdisk
Found iso image: /boot/images/grml64-medium_2010.12.iso
done
Voil , when rebooting your system you should see something like: Screenshot: grml-rescueboot The Grml Rescue System entry is what grml-rescueboot provides and Bootable ISO Image is what s provided by grub-imageboot. Just select the entry you d like to use, press enter and and you should get the bootsplash of the according ISO. BTW: I ve tested this with Ubuntu 10.10 too, grml-rescueboot works out-of-the-box with the Ubuntu ISO as well, it just doesn t support the memdisk ISO boot by grub-imageboot (yet). Tip: if you want to use grml-rescueboot and grub-imageboot with the same ISOs without having them twice on the disk just point the configuration option IMAGES in /etc/default/grub-imageboot to the according directory (like /boot/grml). Note: if you re using the ext 2,3,4 filesystem on the partition that s being used for /boot please be aware that you need GRUB >=1.98+20100804-12 (newer version is already available in Debian/unstable and hopefully migrates to squeeze in time) or GRUB >=1.99~20101122-1 because of the INDIRECT_BLOCKS support, see #543924 for the details. PS: We think about providing grml-rescueboot and grub-imageboot within official Debian. If you re interested in seeing integration support for other distributions as well please help and drop us a note.

Gunnar Wolf: Back with Ruby on Debian polemics

Once again, a polemic regarding how to properly integrate the Ruby language and libraries with the Debian distribution has been ignited. Similar arguments were presented in November and December 2008 and September 2010 And excuse me for just refering to my own blog, but there are links to some other posts from there... and there will surely be many others I just missed. There were even nice collaboration attempts (such as DebGem, announced in January 2009... But which apparently never stuck on, as it is still listed as in its public Beta period and lists only support for Debian 4.0 and Ubuntu 8.04 and 8.10). Some days ago, while I was on vacation, I received a mail from Lucas Nussbaum expressing the burnout he had been suffering from this situation. Some days after that, he posted a corresponding blog post - Giving up on Ruby packaging. The comments following his post are most interesting The one comment I'd like to highlight (noting I skimmed a whole deal of them) is by Paul Brannan, one of the original RubyGems authors. Yes, possibly the design criteria for Gems included some mutually-conflicting goals, and they cover some of Debian's goals. Unfortunately, this conflicting criteria... Was resolved to the opposite way we would have wanted (specifically, that a sysadmin should be able to install software with the same tools he was already familiar with). And of course, there are deeper disagreements, which are rightly stemmed from different priorities: Debian (+derivatives) is meant to make user's and sysadmin's lifes simpler. Gems are meant to make developers' lifes simpler. And while all developers are some sort of users, and sysadminsthe reverse is (obviously) not so. A developer maintaining several different codebases will surely suffer (well, this particular developer has surely suffered) if he has to maintain them all using coherent versions of system libraries. When a system is programmed, its developers want to use the greatest, latest tools to offer the best experience and functionality. And it's natural for libraries to evolve over time. However, any sysadmin will grind their teeth at the prospect of having many different versions of libraries, as the Gem model proposes. Why? Well, I was bitten by a minor example of it Bug fixes do not get backported. Of course, if simple bugs are not backported, security and stability of the system as a whole suffers. Often, new software versions will not only add functionality, but improve on how things are done. Not only bugs get corrected, but we get better response times, better memory handling, etc. It might sound harsh to say this, and even more, as a developer it feels I am talking against myself But while development time should be minimized during the system design/implementation, once systems are in production, it should be used to allow for friendlier sysadmin relations. I have to wear both hats at my real-life job. So, what is my take on this? Of course, blaming myself for choosing the wrong version is a no-go. When evaluating a library before using it in my projects, I try to look at its history and API-stability. APIs that change often speak about immature libraries which are still trying to get the right way to implement their functionality; just-started projects might be great and revolutionary, but they do not yet show any kind of long-term committment... And can lead (and have often led me) to painful rewrites when the Next Big Thing is reached. As for the users: Even if your favorite language has the best and friendliest distribution method, users just do not care about what language was a particular piece of software implemented in. Users want to be able to install and uninstall with their system's usual tools. Directly using Gems, CPAN, PEAR and whatnot is just unnatural for users, however convenient they are for developers. Distributions offer a non-technical advantage in this regard as well: A human filter. As an example, as I write this there are 19392 Ruby Gems and 19092 Perl modules in CPAN (note that CPAN stores some older versions but discourages authors from keeping too many - So no, they are still very different in size). Debian has around 30,000 packages. Why is that? Because all Debian packages _must_ be human-generated, human-reviewed, human-submitted. This means, a person must think each packaged piece of code is worth packaging, is stable enough and provides value to users as it is, and is fit for being part of a stable release. I am not saying with this that 90% of CPAN and Gems are crap I am saying that they are probably early implementations, to be installed, tested and improved by developers, and still not apt for general public use. Or maybe not interesting enough to be packaged as a service for the non-techie public at large. Oh, and one last point: Ruby is not a new language anymore. It is a mature, powerful language with different implementations, every time more stable... But it is a language deeply affected by the (not so new anymore either) the appearance of Rails. Why do I say this? Because although the language is in no way tied to Web development, many of its strongest uses are Web-oriented. How does this affect the current discussion? Well, because many people argue that users are no longer needed to install the software. Web systems are installed by sysadmins and used via a Web browser, and sysadmins are expected be more skilled than casual users. Still, in Debian (and in other distributions, surely) we try to make sysadmin's lives simpler I have (again, talking out of personal experience) installed several webapps (and system tools, and whatnot) for which I never followed any instructions besides aptitude install foo Using different languages, frameworks, and so on. Can I troubleshoot their installs? Probably, as there is a common logic for how the distribution I have chosen and specialized in works. Can I find causes for bugs in them? Possibly, although there are some languages and frameworks I dare not touch. Can I get help on getting them out of a tight spot? Surely, as there is a central bug tracking system for my distribution And one of the maintainer's tasks is to separate the problems related to the distribution (packaging, installing, simple user questions and misconceptions) to those derived from real bugs upstream. Anyway I am not saying that our way is the best way. No, by a long shot. Again, developers should have an easy, convenient way for installing whatever they want to play with. And to publish it without jumping through hoops. With this post, I'm only trying to express again why Debian works the way it does. And hope for better cooperation in the future. And as for several comments of what I read in Lucas' post, I think that there is interest for this synergy among some of the most committed Ruby people.

Daniel Kahn Gillmor: Liberating Knowledge: A Librarian's Manifesto

A friend just pointed me to Liberating Knowledge: A Librarian's Manifesto for Change by Barbara Fister, an academic librarian who also happens to write mystery novels. She has an excellent perspective on the meaning of libraries, and the tradeoffs involved with the current societal trend toward privatizing knowledge through so-called "intellectual property" regulations. In a great critique of the passivity of academia and libraries in the face of attempts at intellectual enclosure by private corporations, she writes:
This uninformed indifference is laying the groundwork for a new tragedy of the commons: a world in which knowledge is turned into intellectual property, monetizied, and made artificially scarce.
She closes with a six-point manifesto that begins:
Liberation bibliography arises out of outrage at the injustice of the current system. It s not about saving money, it s about the empowering nature of knowledge and the belief that it shouldn t be a luxury good for the few.
The article abounds in examples of heinous arrangements in the current system that seem to be accepted as standard procedure, and clear thinking about what the actual tradeoffs are (and how we, as a society, are making them poorly). If i had one objection, it would be that she neglects to mention increased surveillance as one of the problems that come with privatization of knowledge. Our abilities to read privately and anonymously, and to correspond confidentially are at risk because of these systems of control. Anyway, I'd love to see more open allegiances between librarians and free software folks; the ideals and struggles are very much in parallel. Go talk to your librarian friends about this stuff today!Tags: academia, freedom, intellectual property, libraries

Enrico Zini: Streaming JSON objects with python

Streaming JSON objects with python Dear Lazyweb, Suppose I'd like to create an HTTP service that gives you a looong list of JSON objects. For example, one JSON object for every package in Debian. One way I could do that is to just transfer a big, dozens of megabytes JSON array with one element per package. With the standard json module in python, this requires the client to buffer all the data in memory for decoding, as the JSON result would be something like this: 
[
    info for package 1  ,
    info for package 2  ,
  ...dozens of megabytes...
    info for package 30000  ,
  ...
]
However, it would be reasonable to engineer things so that the client can process packages one at a time, as they are produced. This can dramatically reduce client memory usage, as well as remove a large latency between the start of the request and the start of processing. One trivial idea would be to just send JSON objects one after the other, and sort of call load multiple times. It doesn't work: 
import json
from cStringIO import StringIO
a = StringIO()
json.dump( 1:3 , a)
json.dump( 2:4 , a)
a.reset()
print a.getvalue()
print json.load(a)
gives: 
$ python z1.py 
 "1": 3 "2": 4 
Traceback (most recent call last):
  File "z1.py", line 8, in <module>
    print json.load(a)
  File "/usr/lib/python2.6/json/__init__.py", line 267, in load
    parse_constant=parse_constant, **kw)
  File "/usr/lib/python2.6/json/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.6/json/decoder.py", line 322, in decode
    raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 1 column 8 - line 1 column 16 (char 8 - 16)
And the json module documentation does not provide any way of telling the "stream" decoder to just stop parsing after the first valid object. An alternative can be to use chunked transfer encoding and transfer each JSON object in a separate HTTP chunk. It would add the overhead of a little HTTP header per JSON object, which could be significant when generating lots of small JSON objects, but it would be an implementation that strictly follows HTTP and JSON standards to the letter in a clean way. Except:
  1. How do you stream HTTP chunks in WSGI? The WSGI specification's only mention of chunks is that if you produce a result as a generator of "chunks", they are just sent one after the other (effectively, concatenated) in the response stream.
  2. How do you read and process one chunk at a time with urllib/urllib2? I searched on Google but I only found people in tears.
Another alternative is to wrap the JSON objects in some custom container protocol, but then we are not talking standards anymore. Is this all you can do with the standard python library? Can this only be done by adding a dependency on a C library with two obscure separately maintained bindings, or by writing my own JSON parser? Ideas? Please mail enrico@enricozini.org and I'll add them here. Updates Wouter van Heyst mentions that he heard of ijson, which indeed has my exact use case in the synopsis, but after some research looks like a third binding for yajl. Aigars Mahinovs suggests using YAML instead of JSON. Indeed YAML supports streaming very well (it's even already supported in DDE, but last time I tried to do anything serious with it, PyYaml was slooow. Of course, this being the Python ecosystem, There Are Lots Other Ways To Do It And You Shall Be Patronised For Not Chosing The Proper One, but Debian only has PyYaml. Another itch with YAML is that there is no YAML parser in the standard python library, which requires clients to add an extra dependency. Jyrki Pulliainen has so far mailed my favourite solution: disable formatting in the encoder and separate JSON objects with newlines: 
we built a streaming JSON app as a part of our platform. However, we
didn't stream a single JSON object as that was a bit cumbersome. When
having syntax like:
[
  <json-obj1>,
  <json-obj2>,
  ....
  <json-objN>
]
The reader has some problems. Should it read the whole stream to find
the closing bracket ] even if the stream would me tens of megabytes?
Should it try to split on , and parse single objects? What if the
objects contain , so should there be some regexp magic?
We ended up streaming separate JSON objects separated by newlines
(\n). This way the reader can read until hitting \n or end of stream
and then split the result on newlines. The new stream looked like
this:
<json-obj1>\n
<json-obj2>\n
...
<json-objN>\n
This way we could efficiently stream hundreds of thousands of JSON
objects from server to client without reserving a load of memory. For
the WSGI part how to do this, the best would probably to do a
generator that would yield a single JSON object with the newline (we
didn't use WSGI back then). Reading can be done using any reader
capable of handling streams. We used pycurl for reading, but I can't
really recommend it as it is way too complicated for a simple task
like this :p
Hopefully this helps!
This means that if the client has no access to a JSON reader that supports a stream of concatenated toplevel objects, they can still trivially code it on top of any JSON library. This means that working clients can be written based just on Squeeze's python standard library. That's what I'm going to do; I'll probably implement it as a separate 'jsons' output format for DDE, because producing a JSON response with multiple toplevel obejcts somehow breaks the JSON standard. Thanks to all who provided this excellent information!

5 January 2011

Wouter Verhelst: Substituting gdb debug paths

Gdb is a debugger with many, many, many(1) options. Some of them are rather obscure, others... less so. While it's always had the ability to read source files and do useful stuff with them, recent versions of gdb have gained another interface (which you get by calling "gdb -tui") that makes it a bit easier to watch the source as you step through the code. Additionally (and I don't know how long that feature's been there), you can do some interesting things with source paths. Such as substituting a source path prefix by another prefix -- useful if you've moved sources around. This is accomplished by way of the 'set substitute-path' command: 
(gdb) set substitute-path /home/buildd /home/wouter
Now whenever gdb is told by the debugging info to search for a file /home/buildd/foo/bar/baz.c, it will instead search for /home/wouter/foo/bar/baz.c. Very useful. One problem with the above approach, however, is that you need to know what the original path is the /home/buildd in our example. If you've built the binary yourself, you just need to remember where you built it, and you're all set. But what if you haven't built the binary yourself? I used to use strings for that, and look for any likely candidates, but that isn't very reliable and requires a bit of hunting through many many lines that just happen to look like strings to what really is a fairly naive program. Additionally, it doesn't work when the debugging symbols are in a separate file (as implemented by dh_strip), amongst others. But no worries! There's a simple solution. Rather than using strings, use readelf -w instead, and grep for DW_AT_comp_dir, which will give you the correct source directory to substitute. I'm happily single-stepping through a library now. If that isn't nice. 1 no really, many

Iustin Pop: More Haskell

It's no secret to the people I around me that over the past two years I've become a Haskell fan, to put it mildly. I mean, what's not to like? Just in the past month: The downside is that whenever I feel Ok, now I kind of know my way around Haskell , I learn of another thing that completely resets how I feel about my Haskell understanding. The most recent example is Template Haskell. I mean, there are people who already blame Haskell for its arcane syntax, which I like. But Template Haskell goes a step forward and allows you to use a Haskell library to write Haskell code, with an even more contorted syntax or at least that how it looks for now to my eyes. Gladly this is not easily doable in a recursive way (meta-TH code writing TH code that will write Haskell code), otherwise heads would explode (at least mine). But even though it seems very complicated, I feel that the power that it enables (a macro system that generates code that will be statically checked) is removing one of the last big advantages that (for me) Python had over Haskell: it's easy to do class modification in Python and introspection (e.g. via metaclasses) to automate boiler plate away, but it's of course not type safe (so unittests galore). With TH, you can do the same customization of data types (and not only) and compile-time introspection (called reification) to alter (some) of the properties of the respective type, for example automatic instance derivation, so one less reason to say this particular problem is harder to do in Haskell than in other languages . I haven't yet used TH for my own code and until I actually use it extensively I won't be sure it's actually a good replacement for throw a __metaclass__ in there and automate the boring parts away . But in any case, it looks to be yet another awesome tool in the Haskell programmer's toolbox, which I can't wait to use.

4 January 2011

John Goerzen: Looking back at 2010: reading

A year ago, I posted my reading list for 2010. I listed a few highlights, and a link to my Goodreads page, pointing out that this wasn t necessarily a goal, just a list of things that sounded interesting. I started off with Homer s Iliad, which I tremendously enjoyed and found parallels to modern life surprisingly common in that ancient tale. I enjoyed it so much, in fact, that I quickly jumped to a book that wasn t on my 2010 list: The Odyssey. I made a somewhat controversial post suggesting that the Old Testament of the Bible can be read similar to how we read The Odyssey. Homer turned out to be much more exciting than I d expected. Jordan s Fires of Heaven (WoT #5) was a good read, though it is one of those books that sometimes is action-packed and interesting, and other times slow-moving and almost depressing. I do plan to continue with the series but I m not enjoying it as much as I did at first. War and Peace is something I started late last year. I m about 400 pages into it, which means I ve not even read a third of it yet. It has some moving scenes, and is a fun read overall, but the work it takes to keep all the many characters straight can be a bit frustrating at times. Harvey Cox s The Future of Faith was one of the highlights of the year. A thought-provoking read by someone that embraces both science and religion, and shows a vision of religion that returns to its earlier roots, less concerned about what particular truths a person believes in than it is about more fundamental issues. Marcus Borg s Jesus: Uncovering the Life, Teachings, and Relevance of a Religious Revolutionary began with a surprisingly engaging history lesson on how agriculture caused the formation of domination societies. It also described in a lot of detail how historians analyze ancient texts their drafting, copying, etc. It paints a vivid portrait of Jewish society in the time that Jesus would have lived, and follows the same lines of thought as Cox regarding religion finally moving past the importance of intellectual assent to a set of statements. Among books that weren t on my 2010 list, I also read and here I m not listing all of them, just some highlights: The Cricket on the Hearth in something of a Christmastime tradition of reading one of the shorter Dickens works. I enjoyed it, but not as much as I enjoyed A Christmas Carol last year. Perhaps I made up for that by watching Patrick Stewart as Scrooge instead. How to Disappear Completely was a fun short humorous read, with a very well-developed first-person narrative. Paralleling my interest in amateur radio, I read and studied three books in order to prepare myself for the different exams. In something of a surprise, I laughed a lot at Sh*t My Dad Says, which was more interesting and funny than I expected it to be. All I can say is that Justin s got quite the dad and quite the interesting childhood. I even read two other recent releases: The Politician (about John Edwards) and Game Change (about the 2008 presidential race). Both were interesting, vibrant, and mostly unsourced so hard to know exactly how much to take from them. And finally, reflecting on and travel before my first trip to Europe, Travel as a Political Act, which encourages us to find the fun in my cultural furniture rearranged and my ethnocentric self-assuredness walloped. And that was fun. Now to make up the 2011 list

2 January 2011

Lucas Nussbaum: Giving up on Ruby packaging

I have finally reached a decision regarding my involvement in the Debian Ruby packaging efforts. I have decided to stop. This has been a very hard decision to make. I have invested huge amounts of time in that work over the years. I still love the language, and will continue to use it on a daily basis for my own developments. I still hope that it will succeed. I know that some people will be disappointed by that decision (and that others will think your work was useless anyway, people should use RVM and rubygems ). But I also know that I won t be able to push for all the required changes alone. I just don t have the time, nor the motivation. For the record, here are the changes I would have liked to see in the Ruby community. The core Ruby development community should mature.
The core Ruby development community is still dominated by Japanese developers. While not a bad thing in itself, it is easily explained by the fact that the main development mailing list, where most of the important decisions are taken, is in japanese. ruby-dev@ should be closed, and all the technical discussions should happen on the english-speaking ruby-core@ list instead. The release management process should also improve. Currently, it looks like a total mess. The following Ruby development branches are actively maintained:
ruby_1_8 (106 commits over the last six months)
ruby_1_8_6 (4 commits over the last six months)
ruby_1_8_7 (35 commits over the last six months)
ruby_1_9_1 (4 commits over the last six months)
ruby_1_9_2 (227 commits over the last six months)
trunk (1543 commits over the last six months) While the state of the ruby_1_8_6 and ruby_1_9_1 branches is clear (very important bugfixes only), the state of all of the other branches is rather unclear.
What s the stable Ruby branch? 1.8 or 1.9? If it s 1.9, why are people still actively developing in the ruby_1_8 branch? How long will they continue to be maintained in parallel, dividing the manpower? Is a Ruby 1.8.8 release to be expected? Will it be ABI/API compatible with 1.8.7? Is the ruby_1_8_7 branch really bugfixes-only? How much testing of it has been done? If it s bugfixes-only and regression-free, I should push it to Debian squeeze, due to be released in a few weeks. But would you recommend that? Due to past breakages in the ruby_1_8_7 branch, it s unlikely that we will do it.
Is the ruby_1_9_2 a regression-free, bugfix-only branch? If yes, isn t 227 commits over 6 months a lot? What will be the version of the next release of trunk ? When is it expected? Will it be ABI-compatible with the current ruby_1_9_2 branch? API-compatible?
New releases in the 1.8.7 and 1.9.2 branches were done on december 25th. Why were they no betas or RCs allowing wider testing? How much testing has been done behind the scenes? Most of those questions have no clear answer. The Ruby development community should build a common understanding of the status of the various branches, and of their release expectations. Releasing on december 25th of each year sounds fun, but is releasing when everybody is on vacation really a good idea? It would be fantastic to have something similar to Python Enhancement Proposals in the Ruby community. But having open discussions in english about the major issues would already be great. Ruby is not just the interpreter.
The Ruby development community should clearly define what the Ruby platform is. There are some big players, like Rails, and newer interpreter releases should not be done before ensuring that those big players still work. Also, since we have alternative Ruby interpreters, like JRuby, Rubinius and MacRuby, we need a clear process on how they integrate with the rest of the ecosystem. For example, having each of them rely on their own outdated fork of the whole stdlib is ridiculous, since it s not where they compete. The Ruby community should acknowledge that RVM and Rubygems are not for everybody. People who say so should be laughed at. Of course, RVM and Rubygems are nice tools for some people. But it is completely wrong to believe that compiling from source using RVM should be the standard way of installing Ruby, or that all people interested in installing Redmine should know that Ruby has its own specific packaging system. The Ruby community should work with their target platforms to improve how Ruby is distributed instead of reinventing the wheel. That includes Debian, but also RedHat-based distros, for example. It is likely that it won t be possible to reach a one-size-fits-all situation. But that s real life. Some people in the Ruby community should stop behaving like assholes. As one of the Debian Ruby maintainers, I have been routinely accused of creating crippled packages on purpose (FTR, I don t think that the Debian packages are crippled, despite what the rumors says). Debian is not the only target of that. Just yesterday, someone called for abandonning YARV (the new Ruby VM in Ruby 1.9), calling it Yet Another Random Vailure. This kind of comments is really hurting the people who are investing their free time in Ruby, and is turning away people who consider getting involved. In Debian, we have had a lot of problems getting people to help with Ruby maintenance since they are getting shit from the community all the time. So, what s the future for Ruby in Debian? Update: there s also a number of interesting comments about this post on this site.
Update 2: First, thanks a lot for all the interesting comments. I will make some follow-up posts trying to summarize what was said. It seems that this post also triggered some reactions on ruby-core@, with Charles Olivier explaining the JRuby stdlib fork, and Yui Naruse clarifying that all questions are welcomed on ruby-core@. This is great, really.

1 January 2011

Steve Kemp: It would be nice if we could record which files populate or read

It would be really neat if there were some tool which recorded which dotfiles an application read, used, or created. As an example emacs uses .emacs, but won't create it. However firefox will create and fill ~/.mozilla if it isn't present, and links will create ~/.links2. What would we do with that data? I'm not sure off the top of my head, but I think it is interesting to collect regardless. Perhaps a simple tool such as apt-file to download the data and let you search: 
who-creates ~/.covers
who-creates ~/.dia
Obviously the simple use is to purge user-data when matching packages are removed - e.g. dpkg-posttrigger hook. But that's a potentially dangerous thing to do. Anyway I'm just pondering - I expect that over time applications will start switching to using "centralised" settings such as ~/.gconf2 etc. In the menatime I've started cleaning up ~/ on my own machines - things like ~/.spectemurc, ~/.grip, etc. ObQuote: What a long sword. I like that in a man - Blood of the Samurai (Don't be tempted; awful film.)

31 December 2010

Russell Coker: Links December 2010

Aaron Huey gave a disturbing TED talk titled America s native prisoners of war [1]. He says the last chapter in any successful genocide is the one in which the oppressor can remove their hands and say my god, what are these people doing to themselves, they re killing each other, they re killing themselves while we watch them die .Peter Haas gave an interesting TED talk about how the poor engineering work in Haiti contributed to the significant death toll from the earthquake [2]. He advocates training for builders to prevent death and property damage from the next earthquake which is a lot cheaper than cleaning up the mess after buildings have fallen down.Wired has an interesting article on Phylo, a new crowd-sourced science game where you can sequence DNA [3]. The article also has links to other crowd-sourced science games.Brendan Scott gives a good summary of some of the most interesting news articles related to Wikileaks [4].Petter Reinholdtsen published a detailed and informative letter that Peruvian Congressman Edgar Villanueva wrote to Microsoft on the topic of a Peruvian bill to compell the government to use free software [5]. This has a lot of great ideas for anyone who wants to lobby their government for free software related legislation.Melissa McEwan wrote an informative blog post about why she doesn t trust men [6]. I can t do justice to this with a summary so just read it if you are male.Diana Laufenberg gave an interesting and inspiring TED talk about ways of teaching children [7]. Her main point was about embracing failure, having children learn from their mistakes. I think that perhaps embracing failure is only going to work with an exceptional teacher such as Diana, and that the majority of teachers would probably fail if they tried to implement it. She does have some really interesting examples of how she teaches so it s worth watching even if you don t agree with the central point.

30 December 2010

Jonathan McDowell: Why Linux? (Part 5: Flexibility)

(This is part of a series of posts on Why Linux?) I find Linux more flexible. Maybe that's the familiarity showing, maybe it's about the package management, but it's a powerful reason for me to use it. For example, a couple of years ago I wanted to try out some iSCSI stuff against a SAN. Of course I have test boxes available I can do this on, but this was just to try out a few bits and pieces rather than anything more concrete. So I installed open-iscsi on my desktop and was able to merrily do the tests I wanted with very little additional work. Or I wanted to try out some BitKeeper to git conversion work recently. I wasn't sure how much resource it would take on a build server, and didn't want to tie things up there. So I ran it on my desktop overnight, where I could easily setup the appropriate environment and wouldn't impact on anyone else's resources. Problems talking to dodgy hardware? Linux is much better about giving you some idea what's going on, without needing to install extra software. I had a workmate grappling with an old USB music player recently; hooking it up to her Windows laptop wasn't providing a lot of joy so I attached it to my Linux box and was able to see that it did identify ok, but was disconnecting randomly at times too. Want to script querying an AD server for the current employee list and displaying who's joined and who's left since the last time you did so? I found that easy enough with the common Linux LDAP tools. I'm sure it's doable under Windows too, but I'm not sure it would be quite so simple. For bonus points add graphviz into the mix for automatic organisation charts (modulo accuracy of the AD data). This flexibility is something that helps me do my job. Sure, as I mentioned above I do have access to test boxes that I can use for this, but being able to do it on my desktop can be useful - for example if I'm offline, or on a slow network connection, or just geographically distant from my test machines so network latency is higher than I'd like. (Also, it's something that makes a Linux box a really great test box. I'm lucky in that I have a mix of OSes available to me for testing, but the one that I use most often is the Debian box. Much easier to get and install decent diagnosis tools for it that can give me packet level dumps, or do really odd stuff that turns out to be really useful.)

Andrew Pollock: [debian] "A fine example of shit code in any language"

I just discovered that I'd been hit by #409864 since upgrading to Lenny. A very unfortunate bug that one. I'm surprised it hasn't been fixed in a point release.

29 December 2010

Jon Dowland: announcing RDiffFS

It gives me pleasure to announce RDiffFS, a virtual filesystem for browsing backups created by the rdiff-backup tool. Assuming that "/backup" is the location of a series of backups created using rdiff-backup, here is a usage example: 
$ rdifffs /backup/dest/ /mnt
$ ls -l /mnt
total 5
drwxr-xr-x 4 jon jon 4096 Feb  7  2010 2010-09-04T16:38:10+01:00
drwxr-xr-x 4 jon jon 4096 Feb  7  2010 2010-12-29T10:51:50Z
drwxr-xr-x 4 jon jon 4096 Feb  7  2010 2010-12-29T10:52:30Z
lr-xr-xr-x 2 jon jon   11 Jan  1  1970 current -> 2010-12-29T10:52:30Z
$ for i in /mnt/*/a; do c=$(cat "$i"); printf "%32s:\t%s\n" "$i" "$c"; done
/mnt/2010-09-04T16:38:10+01:00/a:   foo
     /mnt/2010-12-29T10:51:50Z/a:   bar
     /mnt/2010-12-29T10:52:30Z/a:   baz
                  /mnt/current/a:   baz
I will write more about the creation of rdifffs at a later date: in particular my choice of implementation language, but I don't want to clutter up this announcement with all that. Suffice to say, I started writing a "quick tool" nearly a year ago, and I have become increasingly desperate to announce it. At present, rdifffs should be considered alpha quality software. It can browse my real-world backup, albeit slowly, and with several metadata bugs. This is more than archfs can manage. It can read files from the most recent backup, and most files from incremental backups, but not all.

28 December 2010

Steve Kemp: Minor success has been achieved in reimplimenting mod_rewrite

So yesterday I mentioned the mod_rewrite-compatible proxy server. Today I've spent several hours getting to grips with that. I've progressed far enough along that the trivial cases are handled, as the following test case shows: 
void
TestParamteredMatch (CuTest * tc)
 
    /**
     * We pretend this came in via the network.
     */
    char *request = "GET /login/steve.kemp/secret#password HTTP/1.1\n\n";
    /**
     * This is the mod_rewrite rule we're going to test.
     */
    char *rule    = "RewriteRule
                       ^/login/(.*)/(.*)/*$
                         /cgi-bin/index.cgi?mode=login;lname=$1;lpass=$2";
    int res = 0;
    /* Parse the HTTP request */
    struct http_request *req = http_request_new (request);
    CuAssertPtrNotNull (tc, req);
    /* Ensure it looks sane. */
    CuAssertStrEquals(tc, "/login/steve.kemp/secret#password", req->path );
    /* Create the rewrite rule */
    struct rewrite_rule *r = rewrite_rule_new (rule);
    CuAssertPtrNotNull (tc, r);
    /* Assert it contains what we think it should. */
    CuAssertStrEquals(tc, "^/login/(.*)/(.*)/*$", r->pattern );
    /* Apply - expect success (==1) */
    res = rewrite_rule_apply( r, req );
    CuAssertIntEquals (tc, 1, res );
    /* Ensure path is updated. */
    CuAssertStrEquals(tc, "/cgi-bin/index.cgi?mode=login;lname=steve.kemp;lpass=secret#password", req->path );
    free_http_request (req);
    free_rewrite_rule (r);
So all is good? Sadly not. I was expecting to handle a linked list of simple rules, but I've now realised that this isn't sufficient. Consider the following two (real) examples: 
#
#  If the path is /robots.txt and the hostname isn't repository.steve.org.uk
# then redirect to the master one.
#
RewriteCond % http_host  !^repository\.steve\.org\.uk
RewriteRule /robots.txt$  http://repository.steve.org.uk/robots.txt [R=permanent,L]
#
#  Request for :  http://foo.repository.steve.org.uk 
#  becomes:  http://repository.steve.org.uk/cgi-bin/hgwebdir.cgi/foo/file/tip
#
RewriteCond % http_host  .
RewriteCond % http_host  !^repository.steve.org.uk [NC]
RewriteCond % http_host  ^([^.]+)\.repository.steve.org.uk [NC]
RewriteRule ^/$ http://repository.steve.org.uk/cgi-bin/hgwebdir.cgi/%1/file/tip [QSA,L]
So rather than having a simple linked list of rules for each domain I need to have a list of rules - each of which might in turn contain sub-rules. In terms of parsing this is harder than I'd like because it means I need to maintain state to marry up the RewriteCond & RewriteRules. Still the problem isn't insurmountable and I'm pleased with the progress I've made. Currently I can implement enough of mod_rewrite that I could handle all of my existing sites except the single site I have with the complex rule demonstrated above. (In all honesty I guess I could simplify my setup by dropping the wildcard hostname handling for the repository.steve.org.uk name, but I do kinda like it, and it makes for simple canonical mercurial repositories.) ObQuote: - 300

27 December 2010

Steve Kemp: This weekend I have mostly been parsing HTTP

A few weeks ago I was lamenting the lack of of a reverse proxy that understood Apache's mod_rewrite syntax. On my server I have a bunch of thttpd processes, each running under their own UID, each listening on local ports. For example I have this running: 
thttpd -C /etc/thttpd/sites.enabled/blog.steve.org.uk
This is running under the Unix-user "s-blog", with UID 1015, and thus is listening on 127.0.0.1:1015: 
steve@steve:~$ id s-blog
uid=1015(s-blog) gid=1016(s-blog) groups=1016(s-blog)
steve@steve:~$ lsof -i :1015
COMMAND   PID   USER   FD   TYPE  DEVICE SIZE NODE NAME
thttpd  26293 s-blog    0u  IPv4 1072632       TCP localhost:1015 (LISTEN)
Anyway in front of these thttpd processes I have Apache. It does only two things: Yes other webservers could be run in front of these processes, and yes other webservers have their own rewrite-rule-syntax. I'm going to say "Lala lala can't hear you". Why? Because mod_rewrite is the defacto standard. It is assumed, documented, and included with a gazillion projects from wikipedia to wordpress to ... So this weekend I decided I'd see what I needed to do to put together a simple proxy that only worked for reverse HTTP, and understood Apache's mod_rewrite rules. I figure there are three main parts to such a beast:
Be a network server
Parse configuration file, accept connections, use fork(), libevevent(), or similar such that you ultimately receive HTTP requests..
Process HTTP Requests, rewriting as required
Once you have a parsed HTTP-request you need to test against each rule for the appropriate destination domain. Rewriting the request as appopriate.
Proxy
Send your (potentially modified) request to the back-end, and then send the response back to the client.
Thus far I've written code which takes this: 
GET /etc/passwd?root=1 HTTP/1.0
Host: foo.example.com:80
Accept-Language: en-us,en-gb?q=0.5
Refer: http://foo.bar.com/
Keep-Alive: 300
Connection: keep-alive
Then turns it into this: 
struct http_request
 
  /*
   * Path being requested. "/index.html", etc.
   */
  char *path;
  /*
   * Query string
   */
  char *qs;
  /**
   * Method being requested "GET/POST/PUT/DELETE/HEAD/etc".
   */
  char *method;
  /**
   * A linked list of headers such as "Referer: foo",
   * "Connection: close"
   */
  struct http_headers *headers;
 ;
There are methods for turning that back to a string, (so that you can send it on to the back-end), finding headers such as "Referer:", and so on. The parser is pretty minimal C and takes only a "char *buffer" to operate on. It has survived a lot of malicious input, as proved by a whole bunch of test cases. My next job is to code up the mod_rewrite rule-processor to apply a bunch of rules to one of these objects - updating the struct as we go. Assuming that I can get that part written cleanly over the next week or two then I'll be happy and can proceed to write the networking parts of the code - both the initial accepting, and the proxying to the back-end. In terms of configuration I'm going to assume something like: 
/etc/proxy/global.conf                    : Global directives.
/etc/proxy/steve.org.uk/
/etc/proxy/steve.org.uk/back-end          : Will contain 127.0.0.1:1019
/etc/proxy/steve.org.uk/rewrite.conf      : Will contain the domain-wide rules
/etc/proxy/blog.steve.org.uk/
/etc/proxy/blog.steve.org.uk/back-end     : Will contain 127.0.0.1:1015
/etc/proxy/blog.steve.org.uk/rewrite.conf : Will contain the domain-wide rules
..
That seems both sane & logical to me. ObQuote: "I shall control the fate of the world... " - Day Watch. ObRandom: Coding in C is pleasant again.

Next.

Previous.