So we are about 1/2 way done with my hacker class and I have to say it has been a pretty fun ride, especially since many students thought hacking was all about pimply kids engaging in malicious acts of computer violence and many had no knowledge of basic technological concepts like UNIX or source code. We had to first sweep away the cobwebs of misrepresentation and replace with a more solid foundation of facts (however messy the world of hacking still is!) But after building a foundation, you still wonder whether students are learning. Traditionally we gauge progress with exams or essays, which can be effective but let’s face it, at times a little tedious. But today a student sent me the following very short email, which made me realize I had another barometer at my disposal to gauge their progress xkcd:

http://xkcd.com/344/ I wouldn’t have gotten the joke if it wasn’t for you. Thanks.

The email not only made me smile (for it is always nice to know your students are leaning something) but it gave me the idea that in the future I might include a comic based exams composed of 5-10 comic strips (many from xkcd) and ask for an exegesis of them. Why not? Seems like fun to me. And it would be great to include the following as one of my course objectives: “By the end of this class you will be able to read xkcd and actually understand (most) of it.” If they could do that, well, they must have learned at least something update: One of my readers, Florian, sent this nice bit, which I have not read before: “I read your blog via Planet Debian and immediately felt reminded of Jane
Goodall’s Foreword to Gary Larson’s The Far Side Gallery 5 which I’d
like to share with you:”

Recently I was talking with one of the best researcher know, Tim
O’Halloran. He has been able to inspire generations of middle and high
school students to care for the natural world. Tim told me that Gary
Larson has had a major impact on his teaching. Tim uses Far Side
cartoons to introduce topics, to illustrate points, and to “reinforce
the notion that the more we investigate the universe, the richer is our
experience.” When designing exam papers Tim finds the cartoons “ease the
tension and spark the memory.” It all began when, in the fall of 1985,
he was given the task of teaching science to 162 Tulsa ninth-graders who
were convinced that it was absolutely irrelevant to their futures. Tim
put one hundred Far Side cartoons on a large bulleting board, and told
the students to study them. The consensus was that they didn’t
understand the humor - The Far Side was “too weird”. However, Tim wrote
me, “Each time we completed a unit and the students approached the
bulletin board with newly acquired wisdom, I smiled quietly and thanked
the cosmos for Gary’s perspectives as the kids roared with the confident
laughter of the enlightened.” Having taught myself for a while, I can fully and happily relate.

Later in the semester, and for my hacker course, we will be reading Understanding Comics, which will give us a bit of a meta perspective on why comics are so good at conveying a certain type of message. I can’t wait to read it as I have heard it is fantastic.

A lazy morning again, so you'll enjoy another short list of changes in the upcoming awesome 3.1 (first one is still here):

• Text can now be spread across multiple lines, using pango ability to wrap text;
• New full screen layout;
• awful.clients.swap.bydirection function;
• Images get crop_and_scale() method;
• New module: invaders, a space invader game;
• New module: naughty, notification module (like GNOME's notification-daemon);
• More hooks: "clients" for client list changes, "tags" for tag list changes and "tagged" for client's tag list changes;
• Wiboxes's widgets can now be composed of arrays of arrays, specifying a DAG structure, unless of a simple array. A reference is kept to it so it can be retrieved later. Any array can be modified with awesome being notified using a proxy table system.
• Taglist and tasklist widgets are gone from core and are now written in Lua, using the above system.

Not too much changes, I've spent something like a week thinking and writing the 2 last points. I've now a bunch of things to finish and I think we'll go for a RC release in something like a couple of weeks.

Bonfire Onlooker by dominicspics (cc-by) Of all the things I wanted to do yesterday, testing TTLLP’s disaster recovery plan was not one of them. I often suggest customers make disaster recovery plans, but I don’t like having to use them. One of our server farms suffered a serious hardware failure and slowly died during the day. (It didn’t catch fire - the title is a silly reference to lp0 on fire.) The current stinging electricity bills means that there’s an emphasis on power-saving now that wasn’t there 4 or so years ago, when we still built servers ourselves, before WEEE and cheap big-brand systems. Today’s servers aren’t quite as resilient, because each extra piece of hardware in use takes extra power. There are some much cleverer tactics now, but they don’t catch everything before it goes “bang” and we can’t just bring the “spare” hardware into action if it’s not installed yet. Replacement hardware will be installed as soon as possible, as well as some extra hardware that will help protect against a similar failure in future. The slow failure of the server also means that we lost some emails, voicemails and instant messages from 10am to 2pm yesterday. If one was yours, please resend. Most services have been rerouted or temporarily replaced with help from friendly associates. Three important services that haven’t yet are: 1. our financial system (which gets updated in batches anyway), 2. our task tracking (which is newer than our recovery plan and I need reminding about its backups, but at worst all the current data exists in other places) and 3. our newsletter server. I’ve copied the customer newsletter subscriptions to an older listserver, to send out news of our recovery, but the member newsletters (with GPG integration and so on) will wait for the replacement hardware. I’m hoping we can recover some things from the old server filesystem, else I’ll recreate it from the last release (that’ll teach me to hack it on the live server!). In the meantime, I’ve still most of yesterday’s work to do…

You know you've got a bunch of Mennonites together when the pastor spontaneously asks the church to sing "Jesus Loves Me" while the children walk up for their children's story. And, without music, everybody spontaneously breaks into beautiful 4-part harmony. It must be genetic or something.

It was a day of beautiful singing of hymns a week ago for Tabor Mennonite Church's centennial. It's hard to write about that weekend, in fact -- it was such a nice experience.

Sunday ended with opening the box sealed in the church's cornerstone back in 1965 when the current building was built. Rudy Schmidt, the person that sealed it up back then, was on hand to open it back up. They took out the stone and removed the box. Before the box was opened, Rudy told a story:

Back in 1965, I was the one to solder this box shut. While I was doing this, all of a sudden I smelled burning from inside. I figured I must have caught something inside on fire. I asked Jake Koehn (another person on the building committee) what to do. Jake said, "Nobody will ever see it again, just put it in." So let's open it!

I think Rudy has been wondering all these years whether he set it on fire or not.

The box was opened, and other than a few scorch marks on an envelope, was completely intact and in good shape. There was a copy of a local newspaper from 1965 with a headline about Goldwater, copies of programs of dedication for the new building and parsonage, a Bible, copies of the church constitution, and all sorts of other interesting things. To that we will add a copy of our centennial book (which I helped develop), CDs with recordings of our centennial activities, programs from the centennial celebration, and photos of the weekend.

Earlier on Sunday, we had a long but good worship service with sermons from two former pastors. The choir sang, including a song commissioned for the occasion by Larry Nickel. We got applause after that one (which happens occasionally, but not regularly, in our church). All the living former pastors, and the widow of one of them, served communion as well.

The service ended with the congregation singing the "Mennonite Anthem". I think it was the most beautiful singing of that song I've ever heard. We had 352 people in church that day, and it seemed that not one person was going to let the moment go by without singing.

Saturday had started off with outdoor activities in the church park. There was old-fashioned soap making, remembering how church members used to store fat all year, then get together to make soap for use in their homes and to donate to the needy. There was rope making, wool spinning, and rides for the children. Under the tent, we had some great music the local Greenhorns band, and the former pastors all shared some memories and stories about their time here.

On Saturday evening, we had a meal and a drama -- and managed to somehow fit 321 people into the church basement for the meal. We think that's a record! The drama was written for the occasion by one of our members. John Gaeddert, a pastor here in the early 1970s, has become an expert wood carver in his spare time. He had carved a piece for us to celebrate the occasion, and presented it during the drama. That fit right in, because the drama was called "Bring Your Own Hedgepost" -- back in the early days of the church, each member was supposed to do just that so they would have a place to tie their horses.

Friday night was a lecture by James Juhnke about our first paster, P. H. Richert, who was pastor for nearly 40 years. Quite entertaining and interesting.

I think the highlight of the weekend for me was getting to talk to some of the former pastors I never knew well. I had a few minutes to chat with John Gaeddert. I introduced myself, and he said, "Oh! I don't think I've met you, but I've heard your name a few times this weekend. You're on the centennial committee, right?" Yes indeed. I was born after he left this church, but he knew my grandparents -- which means he also knows exactly where we live. John and his wife Mary are both such warm and friendly people that it feels like I've known them much longer than a few days!

I got to chat with Jim Schrag too. He was pastor at Tabor until I was about 5 or 6. He didn't remember me specifically, but he was also interesting to talk with. He was one of the people that worked on a detailed history book in 1983 for the church's 75th anniversary. He told me that he processed most of the black and white photos for that book in a darkroom in his basement. I hadn't known that -- and mentioned I had found all those photos in the church archives and had scanned them all in. Jim requested a copy.

Terah found Lenore Waltner, wife of former pastor James Waltner, who passed away about a year ago. James had started his career as a pastor at Tabor, and ended it at College Mennonite Church in Goshen, IN, where Terah grew up. So she knew him, but I didn't. Terah introduced herself to Lenore, who seemed quite excited to make this sort of family connection!

And Brenda Martin Hurst, another former pastor, found Terah and introduced herself. Terah said that Brenda said something like, "Hi, I'm Brenda, and I don't think I know you!" These pastors all want to get to know everybody, I think!

Digi released a new beta release of the dgap drivers a couple of days ago, notably fixing the build issue (TTY flip buffer) with newer kernels. They now ship two versions of the driver, one suitable for 2.4 and 2.6 kernels up to and including 2.6.26, and one for kernels 2.6.27+. The appropriate version will be picked up automatically when the module is built, of course. APT source line, unchanged: deb http://debian.technologeek.org/ etch non-free The packages are still built for Etch; Sarge users should stay with 1.3.6 (unless running a newer kernel, but then a rebuild of the userspace tools is required). Feedback at the usual address.

One of html's deficiencies is that it doesn't have a tag to mark up a date. If html had such a tag, then browsers could display dates using your time zone, and in whatever format you like. Rather than in some random time zone, that is often not even shown, and some random format. My fav way to see a date is as relative date, such as "two days ago". With a mouseover so I can see the exact, absolute date in the rare case I need it. It's possible to combine some javascript with html, so that, if javascript is available, a nicely formatted relative date is displayed. And with no javascript, an plain old absolute date is shown. Perhaps the cleanest way to do it is using html like this, combined with javascript that fires on page load, parses the date and replaces it with something nicer:

<span class="date">Fri Oct 17 16:17:02 2008 (EDT)</span>

But you might want to avoid using the ugly machine-parseable date, since it will still display in any situation where javascript isn't available. To do that, I added an attribute containing the machine-parseable date. I chose to use the "title" attribute, so as a bonus you can hover over it to get a popup with the raw date.

<span class="date" title="Fri Oct 17 16:17:02 2008 (EDT)">Friday afternoon, October 17th</span>

So, relatively clean html out of the way, now we just need some javascript to munge in the relative dates. There are various examples out there. The best one, requires the prototype framework. I try to stay away from javascript frameworks, since I'm not a javascript programmer, and since I dislike page loads pulling in thousands of lines of irrelevant code. So I came up with my own more bare metal implementation, which handles both types of html above. You can see it in action on my website, or by enabling the relativedate plugin on your own ikiwiki. (This code is in the public domain, but do note that I'm not a javascript programmer, really.)

---

var dateElements;
window.onload = getDates;
function getDates()  
    dateElements = getElementsByClass('date');
    for (var i = 0; i < dateElements.length; i++)  
        var elt = dateElements[i];
        var title = elt.attributes.title;
        var d = new Date(title ? title.value : elt.innerHTML);
        if (! isNaN(d))  
            dateElements[i].date=d;
            elt.title=elt.innerHTML;
         
     
    showDates();
 
function showDates()  
    for (var i = 0; i < dateElements.length; i++)  
        var elt = dateElements[i];
        var d = elt.date;
        if (! isNaN(d))  
            elt.innerHTML=relativeDate(d);
         
     
    setTimeout(showDates,30000); // keep updating every 30s
 
function getElementsByClass(cls, node, tag)  
        if (document.getElementsByClass)
                return document.getElementsByClass(cls, node, tag);
        if (! node) node = document;
        if (! tag) tag = '*';
        var ret = new Array();
        var pattern = new RegExp("(^ \\s)"+cls+"(\\s $)");
        var els = node.getElementsByTagName(tag);
        for (i = 0; i < els.length; i++)  
                if ( pattern.test(els[i].className) )  
                        ret.push(els[i]);
                 
         
        return ret;
 
var timeUnits = new Array;
timeUnits['minute'] = 60;
timeUnits['hour'] = timeUnits['minute'] * 60;
timeUnits['day'] = timeUnits['hour'] * 24;
timeUnits['month'] = timeUnits['day'] * 30;
timeUnits['year'] = timeUnits['day'] * 364;
var timeUnitOrder = ['year', 'month', 'day', 'hour', 'minute'];
function relativeDate(date)  
    var now = new Date();
    var offset = date.getTime() - now.getTime();
    var seconds = Math.round(Math.abs(offset) / 1000);
    var ret = "";
    var shown = 0;
    for (i = 0; i < timeUnitOrder.length; i++)  
        var unit = timeUnitOrder[i];
        if (seconds >= timeUnits[unit])  
            var num = Math.floor(seconds / timeUnits[unit]);
            seconds -= num * timeUnits[unit];
            if (ret)
                ret += "and ";
            ret += num + " " + unit + (num > 1 ? "s" : "") + " ";
            if (++shown == 2)
                break;
         
        else if (shown)
            break;
     
    if (! ret)
        ret = "less than a minute "
    return ret + (offset < 0 ? "ago" : "from now");
 

I mostly stayed under a rock during the last weeks: exams season, some assingments plus one week of much needed [VAC]; ironically in the DebConf 8 secondary hotel. I'm in a very interesting course on operating systems' programming, it's a new optative course, and has generated a lot of expectatives: most of the systems-oriented undergrads in the computer science career are attending. The downside is that it's *intensive*: we're expected to build from scratch a minimal operating system, starting from the bootloader. So far my group has already done a bootloader that understands FAT, and the entry point for the OS, in which the basic stuff is setup: gate A20, interruptions, protected mode, GDT, etc. Also, each group is required to do a presentation of some systems research topic, and we choose Plan 9 from Bell Labs, that mostly unknown OS that was meant to be a better UNIX, but never gained popularity. So I spent many days playing with it, trying to grok quickly what it was about, and creating a small network of virtual hosts (in Qemu) that could show how it all fit togheter. Plan 9 has all the credentials: created in the same lab as UNIX, designed by the same people, which already learned from past mistakes. It was started in the late eighties, and it really feels like what the future was meant to be. Sadly, we're just starting to see a few of it's revolutionary ideas being applied to other systems: UTF-8, /proc, unionfs, fuse, clone(2), separate namespaces; all of these follow concepts first created in Plan 9, UTF-8 was designed by Ken Thompson "in front of my eyes, on a placemat in a New Jersey diner one night in September or so 1992", in Rob Pike's words. Not happy enough with reworking all the core concepts, they also recreated the visual environment. I have to admit, the user-facing ideas are so foreign that I still don't feel very comfortable using it. Surprisingly, the desktop is very mice-centric, it cannot be used at all without the mouse, and it's bothersome with less than three buttons. Anyway, the underlying workings are so attractive, that I cannot stop from wondering if a hybrid system could be built (Debian/kPlan9 anyone?). To illustrate, I'll reproduce here the same example that I showed yesterday and left some jaws hanging. I had booted a CPU server and a diskless terminal, called cpu-0 and term-0 respectively. This is a screenshot taken in the terminal, which is basically a telnet to my Debian box:
Nothing spectacular here, except for the weird syntax, it was seen like this in the Debian box:

martin@abraxas:~$ netstat -nt   grep :22
tcp        0      0 10.0.2.1:22             10.0.2.30:39879         ESTABLISHED

But here comes the fun, I ran an import command, which is more or less a remote filesystem mount, with which I mount the remote machine's /net in the local namespace. It has to be noted that the /net path is by convention the entry point to anything network related, including all the networking API.
And the wornderful thing is that this works as it should (note the remote address):

martin@abraxas:~$ netstat -nt   grep :22
tcp        0      0 10.0.2.1:22             10.0.2.10:38158         ESTABLISHED

So, what happened? telnet wanted to create a TCP connection, and that is accomplished opening /net/tcp/clone, but then it found the imported filesystem. The open(2) request was then transferred to the remote machine, and there the networking code did the actual TCP/IP dance. Things like this make you realise that maybe the BSD sockets API is not that nice an abstraction; in Plan 9 there are no BSD sockets, it's only implemented in the POSIX compatibility libraries, which aren't used by any of the core tools. Neither there exists any threads support, you just call rfork(2) setting the appropiate flags, exactly what glibc does under the covers nowadays. They also decided to live without select(2) or poll(2), there are no non-blocking calls. If you want non-blocking, you rfork(). The only syncronisation primitive is rendezvous(2), and everything else is built on that. At this point, it should not surprise you that to kill a process you can type echo kill > /proc/$pid/ctl; and that there's no need for an equivalent to Xnest, as rio (the graphical interface) provides a modified /dev, you just type rio inside a window and there you go. Yes, I'm hooked. If you're interested, I compiled many links here (some Spanish there, but all the content is in English), as one of the weakest points in Plan 9 is the unorganised and missing documentation. Tags: Plan9, Planet Lugfi, Planet Debian, Facu

I’ve just set up a poll on my site asking: what topics do you want to read here? Since this July, when cycling and most local and political themes were split off into other blogs, this site has been mostly Web Development and GNU/Linux topics, with an SPI article appearing most weeks, and minor themes of Koha news and ideas and general internet/telecoms issues around ThePhoneCoop. Finally, Drupal, OSCommerce and general business topics appear less frequently. By traffic, the SPI series seems by far the most popular (but at least one of those is heavily spam-attacked now, so I’d discount it), followed by other political themes, then the technical posts. I suspect that politics is getting a temporary boost because of the US and Canadian elections, even though I’ve not written about them much. But is there some other theme that people want to read about? The poll box on the right column lets you tick as many or as few as you want. It requires javascript, but if you don’t have javascript, please leave a comment on my site. If you want to suggest a new topic which I should add, please leave that in a comment too.

Screen Fork?

There are times when I think of forking. Mostly sanity returns very quickly, though Still GNU Screen is one program that I use almost constantly, and it seems to work at a glacial pace. The Debian package has a lot of open bugs against it. Some trivial, some annoying, and some with patches. Making the program GNU/Linux only would simplify a lot of things. But then again would that be a legitimate reason to fork it? Me? I'd just like to see some additional primitives.

More QPSMTPD

I've come up with a nice simple qpsmtpd plugin to do spamgourmet-like setup. This means I can have email addresses:

• steve.3.count@steve.org.uk
  • Allows only three mails to this address.
• steve.date.20-10-2008@steve.org.uk
  • Allows only mail to be sent here prior to Oct 20th.

Plugin code will be in the usual place in the next day or two..

ObFilm: xXx

One of the many interesting ideas in Peter F. Hamilton’s Night’s Dawn series [1] is that of Combat Wasps. These are robots used in space combat which may be armed with some combination of projectile weapons, MASERs, thermo-nuclear and anti-matter weapons. In a lot of science fiction the space combat is limited to capital ships, a large source of this problem is technological issues such as the Star Trek process of making models of ships - it’s too expensive and time consuming to make lots of small models. Shows such as Babylon 5 [2] have fighters which make more sense. Sustaining life in space is difficult at the best of times and it seems likely for battles in space to have few if any survivors. So sending out fighters allows the capital ships to have a chance to survive. I suspect that a major motivating factor in the space battles in Babylon 5 was making it fit on a TV screen. Dramatic TV portrayal of small groups of fighters engaging in a battle is an art that has been perfected over the course of 80+ years. It’s about individuals being shown, whether it’s riders on horseback, pilots of biplanes, or space pilots, it’s much the same. But a reasonable analysis of the facts suggests that without some strange religious motive adopted by all parties in a war (as used in Dune [3]) the trend in warfare is to ever greater mechanisation. So while a medium size starship might be able to carry dozens or even hundreds of fighter craft, if using small robotic craft then thousands of fighters could be carried. So the issue is how to effectively use such robots. It seems likely that an effective strategy would involve large numbers of robots performing different tasks, some would detonate thermo-nuclear weapons to remove enemies from an area while others would prepare to advance into the breach. The result would be a battle lasting seconds that involves large numbers of robots (too many to focus on in a group) while each robot matters to little that there’s no interest in following one. Therefore it just wouldn’t work on TV and in a book it’s given a couple of sentences to describe what would have been an epic battle if humans had done anything other than press the launch buttons. One of the many things I would do if I had a lot more spare time would be to write a Combat Wasp simulator. There are already quite a number of computer games based on the idea of writing a program to control a robot and then having the robots do battle. This would be another variation on the theme but based in space. In a comment on my previous post about programming and games for children [4], Don Marti suggests that a RTS game could allow programming the units. It seems to me that the current common settings for controlling units in RTS games (attack particular enemies, attach whichever enemies get in range, patrol, move to location, retreat, and defend other units or strategic positions) are about as complex as you can get without getting to the full programming language stage. Then of course if you have any real programming language for a unit then changing it takes more time than an RTS game allows, and if the programming is good then there won’t be much for a human to do during the game anyway. So I can’t imagine much potential for anything between RTS and fully programmed games. There is some interesting research being conducted by the US military in simulating large numbers of people in combat situations. I think that the techniques in question could be more productively used in determining which of the various science fiction ideas for space combat could be most effectively implemented.

• [1] http://en.wikipedia.org/wiki/Night’s_dawn


• [2] http://en.wikipedia.org/wiki/Babylon_5


• [3] http://en.wikipedia.org/wiki/Dune_universe


• [4] http://etbe.coker.com.au/2008/10/16/programming-games-children/

Share This

It looks like the jack reporting API for ALSA which just got merged into the mainline kernel for inclusion in 2.6.28 already has its first user - code from Matthew Ranostay supporting the jack detection in Sigmatel HDA codecs was just queued for merge in the next merge window. Admittedly, the jack reporting API has been available in ALSA git since July but it does look like it is just impressively fast adoption after the mainline merge. Still, a long way to go before user space can start to rely on knowing if things are plugged in or not.

In my previous post I drafted a rough idea how to add a feature to Debian that would bridge the gap between Debian as a Binary Distribution and any Source Distribution. The feature in question is giving users the chance to build customized Debian packages for specific feature sets. The possible use cases are simple:

Lets say Anthony User needs super-duper-tool with postgresql support. However this is a not so common use case and so the archive only has postgresql support. If he now wants to build a custom package, the process is "as simple" as:

apt-get source super-duper-tool
cd super-duper-tool-*
apt-get build-dep super-duper-tool OR sudo mk-build-deps -r -i
(possibly try out which ./configure options are supported by super-duper-tool)
vi debian/rules
edit debian/rules
dpkg-buildpackage -us -uc
sudo debi

That is, to be honest lot of work. And the biggest problem about it is that you need to know all this, which is not really user knowledge.

The basic idea is that Anthony can do something like
export DEB_BUILD_OPTIONS=nomysql,postgresql
deb-build-tool build super-duper-tool

and is done.
So the question is how could this be achieved?

1. Define a set of common options (options should be consistent through all source packages, otherwise it makes not that much sense) that every package should support, if it is a supported feature by the package in question. This includes flags like mysql, postgresql, ldap and its no-equivalents (nomysql, nopostgresql) to negate it.
2. Enable use of this common options in the package build process and that is really the hardest part. Defining a lot of ifneq..findstring constructs for 2-10 options per source package in any available source package is.... not.... a good approach. Luckily a lot of packages uses autoconf where enabling and disabling features is as easy as adding a --with-foo or --enable-foo option to the ./configure parameters. So we could write a wrapper that would handle these options. The debhelper scripts already have dh_auto_configure: Maybe this could get enhanced.
3. If it does not exist: Write a tool to auto-build packages with DEB_BUILD_OPTIONS set by the user. Should automatically get the source, (optional: setup/update a pbuilder environment) and build it and optional install/upload it (to a user repository for example) .
   

Still, the idea is only a rough draft. There are still a lot of open questions/issues, for example:

• How to handle these feature flags in packages that don't use autoconf
• How to enable these feature flags for packages that don't use debhelpers
• Shall a central place for setting DEB_BUILD_OPTIONS exist (like it does for Gentoo, FreeBSD etc.) and if so: Where?
• How should packages define new build options, if those defined Debian-wide, aren't enough?
• What about support? Do we support such custom builds of packages or will this be a support-on-good-will thing

Is anybody interested in pushing this idea forward? I'm happily interested in it and I know people who like the idea, too. But I can't push this forward on my own all alone. Certainly the work won't affect Lenny anymore, but considerable the time frame that is planned for Lenny+1 by the release team, this could be a feature for Lenny+1.

I just read an interesting paper titled An Analysis of Data Corruption in the Storage Stack [1]. It contains an analysis of the data from 1,530,000 disks running at NetApp customer sites. The amount of corruption is worrying, as is the amount of effort that is needed to detect them. NetApp devices have regular “RAID scrubbing” which involves reading all data on all disks at some quiet time and making sure that the checksums match. They also store checksums of all written data. For “Enterprise” disks each sector stores 520 bytes, which means that a 4K data block is comprised of 8 sectors and has 64 bytes of storage for a checksum. For “Nearline” disks 9 sectors of 512 bytes are used to store a 4K data block and it’s checksum. These 64byte checksum includes the identity of the block in question, the NetApp WAFL filesystem writes a block in a different location every time, this allows the storage of snapshots of old versions and also means that when reading file data if the location that is read has data from a different file (or a different version of the same file) then it is known to be corrupt (sometimes writes don’t make it to disk). Page 3 of the document describes this. Page 13 has an analysis of error location and the fact that some disks are more likely to have errors at certain locations. They suggest configuring RAID stripes to be staggered so that you don’t have an entire stripe covering the bad spots on all disks in the array. One thing that was not directly stated in the article is the connection between the different layers. On a Unix system with software RAID you have a RAID device and a filesystem layer on top of that, and (in Linux at least) there is no way for a filesystem driver to say “you gave me a bad version of that block, please give me a different one”. Block checksum errors at the filesystem level are going to be often caused by corruption that leaves the rest of the RAID array intact, this means that the RAID stripe will have a mismatching checksum. But the RAID driver won’t know which disk has the error. If a filesystem did checksums on metadata (or data) blocks and the chunk size of the RAID was greater than the filesystem block size then when the filesystem detected an error a different version of the block could be generated from the parity. NetApp produced an interesting guest-post on the StorageMojo blog [2]. One point that they make is that Nearline disks try harder to re-read corrupt data from the disk. This means that a bad sector error will result in longer timeouts, but hopefully the data will be returned eventually. This is good if you only have a single disk, but if you have a RAID array it’s often better to just return an error and allow the data to be retrieved quickly from another disk. NetApp also claim that “Given the realities of today’s drives (plus all the trends indicating what we can expect from electro-mechanical storage devices in the near future) - protecting online data only via RAID 5 today verges on professional malpractice“, it’s a strong claim but they provide evidence to support it. Another relevant issue is the size of the RAID device. Here is a post that describes the issue of the Unrecoverable Error Rate (UER) and how it can impact large RAID-5 arrays [3]. The implication is that the larger the array (in GB/TB) the greater the need for RAID-6. It has been regarded for a long time that a larger number of disks in the array drove a greater need for RAID-6, but the idea that larger disks in a RAID array gives a greater need for RAID-6 is a new idea (to me at least). Now I am strongly advising all my clients to use RAID-6. Currently the only servers that I run which don’t have RAID-6 are legacy servers (some of which can be upgraded to RAID-6 - HP hardware RAID is really good in this regard) and small servers with two disks in a RAID-1 array.

• [1] http://www.cs.wisc.edu/adsl/Publications/corruption-fast08.html


• [2] http://storagemojo.com/2007/02/26/netapp-weighs-in-on-disks/


• [3] http://utcc.utoronto.ca/~cks/space/blog/tech/BigRAID5Problem

Share This

Procmail This will be the last time I talk about this, but here's my anti-rubbish procmail filter. It correctly copes with:

• Foreign character sets that I can't read.
• Bounces from joe-jobs.
• Malformed emails.

Obviously edit to suit your tastes. Especially with regard to character sets - it is a wide brush which is tarring a group unfairly. That said in practise it works for me. If you want real antispam filtering then you should probably be looking at externalising it, or having a layered approach.

• Procmail snippets

ObQuote: Citizen Kane

Some brief highlights of the last three months of Debian Eee PC development. Thermal and ACPI breakage resolved in 2.6.26-7 We’re pleased to see that in the upload to Sid of linux-image-2.6.26-1-686 version 2.6.26-7, the pair of 2.6.26 bugs we’ve been tracking that have made it difficult for Eee users to upgrade their systems have been resolved. Since then 2.6.26-8 has been uploaded and is expected to enter Lenny this week due to a freeze exception. Once the new kernel has migrated we will move quickly to build and release a new installer that includes it. Ath5k wifi works on Eee PC in Linux 2.6.27 Jean-Christophe reports that ath5k works in Linux 2.6.27 on the Eee PC 701, and just needs a small patch to work with our eeepc-acpi-scripts package. This is good news for those of us with models 701, 900, 900A and 1000HD who have been wanting to get off of the non-free Madwifi drivers and onto DFSG free drivers. New Eee PC model 701SD wifi support in the works Users of the new Eee PC Model 701SD have just started showing up looking for support in mainstream Linux distros. Martin Filtenborg confirmed using our Eee PC Live image with the GPL’d rtl8187se driver from Realtek that we can at least use it to connect to an unencrypted AP, get an IP address and ping other hosts. Of course, it is one thing to have a working vendor-supplied driver and quite another to have mainstream support. We’ll make do with what we have now, but will be seeking a mainstream solution as soon as possible. We’re seeking more testers and developers to work on this. To date, an ITP has not been filed, as it is not yet clear who is going to carry this work forward. Chasing the 5 second boot An interesting discussion on Arjan van de Ven and Auke Kok’s work to get an Eee 901 to boot in 5 seconds took place this month. While the Debian Eee PC team is not making work on this a priority, we’ll keep an eye on it to see if Debian can incorporate some or all of the techniques they used so that our users can benefit without making radical changes to their systems. Working towards mainstream support for rt2860 Our filing of an ITP for rt2860 (the wifi driver for models 901, 1000 and 1000H) was followed by discussion about how to separate out the GPL’d driver from the embedded non-free firmware so that it can at least go into contrib. Glenn Saberton has been working on rewriting the build system around kbuild and separating out the firmware. Numerous improvements to ACPI scripts Since my last progress report, there have been numerous improvements to the eeepc-acpi-scripts package to deal with all of the various models we now support and make the scripts more robust and flexible. Check out the changelog for details.

I fully agree that working with Gentoo is no fun at all, as Julian points out in his post
My impression, when I tried Gentoo for the first (and last) time is the same. Thats not because it is made bad. In fact things in Gentoo aren't made that bad. There is good documentation, portage is quiet nice (with its USE flags and alike), but in the end I'm not really satisfied.
After all Gentoo uses a copied concept, which itself is good. The concept is derived from the BSD-world and is just the concept of source-based systems. This concept has its advantages over binary distributions, because it allows a flexibility that is not really possible with a binary distribution. That really is the only appreciable advantage of these systems. Users of these systems (including FreeBSD and alike) tend to give other arguments as well: Newest Software, Highest Performance and even Security is a point they give.

Usually the "newest software" argument is brought up with a rant against Debian. I keep hearing statements like "Debian Etch is totally outdated and Testing is broken". That statement in itself is just wrong, but the important thing is: What do you need such new software for? On a server its preferrable not to upgrade to each and every major release, for certain reasons. Its also a bad thing to build on production systems, so you need to take further measures to administer systems like that. On a desktop I can understand the logic, but then again: No. Why would I want a system I have to compile from scratch and on each upgrade (wasting a lot of time, power etc.) which changes often?

The performance argument is the dumbest of all. Binary distributions usually build binaries for various platforms. And while they can't be optimized for a very specific processor or a very specific feature set (=reduces binary size) they usually perform well enough that a difference between the self-compiled Gentoo system and a foreign-compiled Debian is not noticed by the user and sometimes not even measurable. So the time you save during the lifetime of your builds (which is not very long on a Desktop, is it?) because of the enhanced performance is used up a hundred times, by the time you waste for compiling the whole software only for yourself. Even if some of the software performs noticable better (e.g. video processing is said to benefit a lot from an optimized platform).

What about the security argument? It has been said, that you get your updates earlier then users of binary distributions. That is partly true, because in a binary distribution the update needs to be built first, before it gets to the end user (however, everything before is the same for a binary or a source distribution). But do you really follow security issues that close, that you benefit a lot from this? If you do, indeed, you save some hours. If (and only if) both distributions have a solution for a security issue at the same time.

Anyway. The flexibility argument can't get discussed away. Its the argument that makes Ports in BSD-systems attractive. You can have exactly the features you want, with the build-time options you want. In a comfortable manner. That is in contrast to binary distributions where we package developers need to guess which features might be needed (/wanted) by the users with mixed results. Good if they can decide for themselves. Still, I don't see the reason for compiling the whole system, if I need one or two customized components. So I go with Debian and customize packages if I really need/want to. The only difference is that it takes a lot of more effort.

I would love to see something done in Debian to reach a compromise. Making rebuilding of packages with different options very comfortable. We have DEB_BUILD_OPTIONS, maybe we should enhance it to support a lot more options then noopt,nostrip or nodoc. Possibly it would be a good thing to standardize on some use flags (e.g. [no]ssl, [no]ldap, etc.) and support them in the debian/rules file. This way building customized packages would be as easy as setting sensible DEB_BUILD_OPTIONS and run dpkg-buildpackage on the source. This could be eased further by providing a tool to download the source and build a given package (IIRC such a tool already exists). How does that sound?

Not much to say. Just that I am back, now with a mixed English-Spanish blog. I will try to keep entries written in Spanish out of syndicated feed on Planet Debian, but I won’t promise that some entry can slip. It has been long since I had no blog, so it will be a new challenge to try to update it somehow regularly. Regarding my Debian work, I have to admit that I have become mostly a ‘package and upload’ guy. I don’t have enough spare time to devote to other tasks, so I will mostly try to keep my packages as clean as I can.

Not much to say. Just that I am back, now with a mixed English-Spanish blog. I will try to keep entries written in Spanish out of syndicated feed on Planet Debian, but I won’t promise that some entry can slip. It has been long since I had no blog, so it will be a new challenge to try to update it somehow regularly. Regarding my Debian work, I have to admit that I have become mostly a ‘package and upload’ guy. I don’t have enough spare time to devote to other tasks, so I will mostly try to keep my packages as clean as I can.

One thing that concerns me about using any online service is the security. When that service is a virtual server running in another country the risks are greater than average. I’m currently investigating the Amazon EC2 service for some clients, and naturally I’m concerned about the security. Firstly they appear to have implemented a good range of Xen based security mechanisms, their documentation is worth reading by anyone who plans to run a Xen server for multiple users [1]. I think it would be a good thing if other providers would follow their example in documenting the ways that they protect their customers. Next they seem to have done a good job at securing the access to the service. You use public key encryption for all requests to the service and they generate the keypair. While later in this article I identify some areas that could be improved, I want to make it known that overall I think that EC2 is a good service and it seems generally better than average in every way. But it’s a high profile service which deserves a good deal of scrutiny and I’ve found some things that need to be improved. The first problem is when it comes to downloading anything of importance (kernel modules for use in a machine image, utility programs for managing AMIs, etc). All downloads are done via http (not https) and the files are not signed in any way. This is an obvious risk that anyone who controls a router could compromise EC2 instances by causing people to download hostile versions of the tools. The solution to this is to use https for the downloads AND to use GPG to sign the files, https is the most user-friendly way of authenticating the files (although it could be argued that anyone who lacks the skill needed to use GPG will never run a secure server anyway) and GPG allows end to end encryption and would allow me to verify files that a client is using if the signature was downloaded at the same time. More likely problems start when it comes to the machine images that they provide. They have images of Fedora Core 4, Fedora Core 6, and Fedora 8 available. Fedora releases are maintained until one month after the release of two subsequent versions [6], so Fedora 8 will end support one month after the release of Fedora 10 (which will be quite soon) and Fedora Core 6 and Fedora Core 4 have been out of support for a long time. I expect that someone who wanted to 0wn some servers that are well connected could get a list of exploits that work on FC4 or FC6 and try them out on machines running on EC2. While it is theoretically possible for Amazon staff to patch the FC4 images for all security holes that are discovered, it would be a lot of work, and it wouldn’t apply to all the repositories of FC4 software. So making FC4 usable as a secure base for an online service really isn’t a viable option. Amazon’s page on “Tips for Securing Your EC2 Instance” [3] mostly covers setting up ssh, I wonder whether anyone who needs advice on setting up ssh can ever hope to run a secure server on the net. It does have some useful information on managing the EC2 firewall that will be of general interest. One of the services that Amazon offers is to have “shared images” where any Amazon customer can share an image with the world. Amazon has a document about AMI security issues [4], but it seems to only be useful against clueless security mistakes by the person who creates an image not malice. If a hostile party creates a machine image you can expect that you won’t discover the problem by looking for open ports and checking for strange processes. The Amazon web page says “you should treat shared AMIs as you would any foreign code that you might consider deploying in your own data center and perform the appropriate due diligence“, the difference of course is that most foreign code that you might consider deploying comes from companies and is shipped in shrink-wrap packaging. I don’t count the high quality free software available in a typical Linux distribution in the same category as this “foreign code”. While some companies have accidentally shipped viruses on installation media in the past it has been quite rare. But I expect hostile AMIs on EC2 to be considerably more common. Amazon recommends that people know the source of the AMIs that they use. Of course there is a simple way of encouraging this, Amazon could refrain from providing a global directory of AMIs without descriptions (the output of “ec2dim -x all“) and instead force customers to subscribe to channels containing AMIs that have not been approved by Amazon staff (the images that purport to be from Oracle and Red Hat could easily have their sources verified and be listed as trusted images if they are what they appear to be). There seems to be no way of properly tracking the identity of the person who created a machine image within the Amazon service. The ec2dim command only gives an ID number for the creator (and there seems to be no API tool to get information on a user based on their ID). The web interface gives a name and an Amazon account name. The next issue is that of the kernel. Amazon notes that they include “vmsplice root exploit patch” in the 2.6.18 kernel image that they supply [2], however there have been a number of other Linux kernel security problems found since then and plenty of security issues for 2.6.18 were patched before the vmsplice bug was discovered - were they patched as well? The file date stamp on the kernel image and modules files of 20th Feb 2008 indicates that there are a few kernel security issues which are not patched in the Amazon kernel. To fix this the obvious solution is to use a modern distribution image. Of course without knowing what other patches they include (they mention a patch for better network performance) this is going to be difficult. It seems that we need some distribution packages of kernels designed for EC2, they would incorporate the Amazon patches and the Amazon configuration as well as all the latest security updates. I’ve started looking at the Amazon EC2 kernel image to see what I should incorporate from it to make a Debian kernel image. It would be good if we could get such a package included in an update to Debian/Lenny. Also Red Hat is partnering with Amazon to offer RHEL on EC2 [5], I’m sure that they provide good kernels as part of that service - but as the costs for RHEL on EC2 more than double the cost of the cheapest EC2 instance I expect that only the customers that need the the larger instances all the time will use it. The source for the RHEL kernels will of course be good for CentOS (binaries produced from such sources may be in CentOS already, I haven’t checked). This is not an exhaustive list of security issues related to EC2, I may end up writing a series of posts about this.

• [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1697&categoryID=100


• [2] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1350


• [3] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1233&categoryID=100


• [4] http://docs.amazonwebservices.com/AWSEC2/latest/DeveloperGuide/index.html?AESDG-chapter-usingsharedamis.html#usingsharedamis-security


• [5] http://www.redhat.com/solutions/cloud/


• [6] http://fedoraproject.org/wiki/LifeCycle

Share This