A frequent criticism of solar power is that the sun only shines brightly for part of the day, and that many of the times when there is significant electrical load (EG when people get home from work in winter) the sun light will be weak. One interesting solution to this problem is to store the solar power by splitting ammonia into nitrogen and hydrogen gases, storing them separately, and then reacting them to produce super-heated steam for power generation at any time that power is needed. The technology is based on three decades of research at ANU and the Federal government has allocated $7,400,000 for building a power station near Whyalla in South Australia. The down-side to this is that it needs water for the steam part of the electricity generation process which is a minor problem as we are having water shortages in most parts of Australia. But the up-side is that the process of combining nitrogen and hydrogen to produce heat should be something that can be turned on rapidly. So it seems that there is potential for having a wind power plant designed to satisfy all the power requirements on windy days and store hydrogen and nitrogen for times when there the combination of sunlight and wind is not adequate to satisfy the power requirements. Previous plans for the maximum possible use of renewable energy in producing electricity have included gas fired power plants for times when wind and solar power can’t meet the demand. It seems that with technology such as this one it will be practical to have all electricity produced from renewable sources of energy.

Share This

It is nice to see books on free software finally get their day under the sun and today, Samir Chopra and Scott Dexter have announced the release of Decoding Liberation. Because it is a bit on the pricey side, try to get your library or work to order it and then when you get it, enjoy. I know I did and had the pleasure of reading early versions during an informal reading ground held in NYC 2 years ago and final versions more recently. Among other great chapters, the one on the aesthetics of code, is, well beautiful. I can’t wait to re-read it. If you are in the city, make sure to catch one of the book events that will be happening; your very own will help lead a discussion on October 3rd and I might write something up more formal about the book (and of course post here) then.

Why do I get these errors in a real etch system and I don't in a pbuilder login environment? Missing build-deps?


# Add here commands to compile the package.
/usr/bin/make
make[1]: Entering directory /home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1'
/usr/bin/make -C doc
make[2]: Entering directory /home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc'
docbook2man uclean.sgml
Using catalogs: /etc/sgml/catalog
Using stylesheet: /usr/share/docbook-utils/docbook-utils.dsl#print
Working on: /home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml
nsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:1:59:W: cannot generate system identifier for public text "-//OASIS//DTD DocBook V4.1//EN"
nsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:35:0:E: reference to entity "REFENTRY" for which no system identifier could be generated
nsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:1:0: entity was defined here
nsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:35:0:E: DTD did not contain element declaration for document type name
nsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:37:9:E: element "REFENTRY" undefined
nsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:38:15:E: element "REFENTRYINFO" undefined
nsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:39:12:E: element "ADDRESS" undefined
nsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:40:15:E: element "EMAIL" undefined
nsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:42:11:E: element "AUTHOR" undefined
nsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:43:19:E: element "FIRSTNAME" undefined
nsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:44:17:E: element "SURNAME" undefined
nsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:46:14:E: element "COPYRIGHT" undefined
nsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:47:11:E: element "YEAR" undefinednsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:48:13:E: element "HOLDER" undefined
nsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:50:12:E: element "DATE" undefinednsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:52:10:E: element "REFMETA" undefined
nsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:53:17:E: element "REFENTRYTITLE" undefined
nsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:55:15:E: element "MANVOLNUM" undefined
nsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:57:13:E: element "REFNAMEDIV" undefined
nsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:58:12:E: element "REFNAME" undefined
nsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:60:15:E: element "REFPURPOSE" undefined
nsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:62:17:E: element "REFSYNOPSISDIV" undefined
nsgmls:/home/eddy/usr/src/perso/build-area/svn-buildpackage-0.6.22~bpo40+1/doc/uclean.sgml:63:16:E: element "CMDSYNOPSIS" undefined

And no, I don't mean Mao.
How was Thursday 23 for me?
6:30 Wake up, knowing it will be a long day. No, don't get those delicious extra 15 minutes of sleep.
8:00 Arrive to work, ~90min earlier than usual, to finish the Rails app you have to demo at 10:00
10:00 Meeting with the Institute's Director, Academic Secretary and Technic Secretary to show them the system for following the internal inter-tematic colloquium information workflow. Thankfully, they were happy with it.
11:20 (20 minutes after scheduled): As several academics have said they are interested in getting to know Linux, I'm giving a series of introductory sessions on Debian to them. So, I went with some of them through the Etch installation. Of course, mostly everything was fine. What was not? Resizing a Windows partition is still a step that requires several jumps through hoops (i.e. booting Debian LiveCDs, some console-based interactivity which I wanted to avoid...), and... Well, d-i does not time out if the selected mirror is not responding. Will file a bug, yes. Anyway...
14:30 Managed to leave the lab working, with no damages. Went to the bank to check some data and to have a quick lunch.
16:00 Get back to the institute. Jos Luis, from Difusi n y extensi n was already waiting for me to show a set of components for the Comas-based system we use for tracking attendance to the Institute's activities
16:45 The people from AMECIDER arrived so I could show them my proposal for their new website. Can you believe most of the websites I host are still completely static HTML, and, what's even worse, generated by the $ despiseful_adjective Dreamweaver? Yes, I am proposing them to move over to Drupal
17:30 I finally received another person, one of the people that's attending my Thursday lessons, wanting me to help with some Linux questions. What's Linux's killer app for my Institute's researchers? Believe it or not, a not-brain-dead PDF reader. They are amazed that, using Gnome's Evince, they can just copy-paste bits of PDFs into their documents (and the fact that pasting tabular data into OpenOffice's Calc allows for an easy import feature - Yesterday I spent ~1hr with this researcher importing data generated by the International Monetary Fund into his work), and they are even more amazed with the gnome-screenshot they can directly include graphics from such documents without redrawing them.
19:45 (45 minutes behind schedule): Went to the Astronomy Research Institute, to meet with Sergio, H ctor and Eric, to decide on bits of the implementation of the new gear we will be putting in nisamox, our dear and faithful ftp.mx.debian.org mirror. Thanks to everybody! We will soon tell you what we got and what we did with it, it's mostly settled by now.
20:30 Had the first of some... Four calls regarding yet another system I have been working with for the ICME conference, to be held in Mexico in 2008. Several pushing details that need my attention and work, bah :-/ Have to work on them...
21:00 Got home. Did a couple housekeeping activities (i.e. take out the trash, buy some food...). I was too burnt out to start making my bags for going to Vienna (I'm leaving tonight!), work on the pending systems I have to work on, or even stay awake.
Leaving work for a week to go a conference is always fun and great... But man, it's stressing!

Due to climate change and population increases we are having increasing problems with the water supply in Australia. Peter Lieverdink suggests that we have more options for water supply including treated sewage similar to the Netherlands here he grew up. However I believe that in Australia we already do what he proposes (put treated sewage in rivers and then use the same rivers for the water supply). It’s just removing the middle step (of having the treated sewage in a river) that is controversial. I recently examined a water bill for my house. Among other things it said that my water use was slightly above the average for houses that use water efficiently and significantly below the average for typical water use. The funny thing was that most of the expenses on the bill were not actually related to the amount of water used. My bill for a quarter was $54 service charges for sewerage and water (of which $39 was sewerage), $30 for sewage disposal, $26 for water, and $14 for drainage. So the cost of disposing of sewage was greater than the cost of getting fresh water and the sewerage service charge was the largest single item on the bill. Australia supposedly has a capitalist economic system in place (except for the communist policies related to farming and the protection of some industries such as car manufacture and coal mining). Surely the right thing to do is to make water expensive enough that people have a financial incentive to conserve it! The first thing that should be done is to make the majority of the value of a water bill received by a typical household be determined by the amount of water used. This could be achieved by reducing the fixed components of the bill to something small (EG $20 per quarter instead of $98 for me) and then increasing the cost per kilo-litre (EG from $0.85 to $2.50). The end result would be that people who use average amounts of water would pay about the same amount as they do now and people who use less than the average amount would get smaller bills, while government revenue would remain the same. The next thing to do is to have different tarriffs for different quantities of water used. The first 50L per day could be free (water for drinking and cooking is a right), between 50 and 450L per day (451L per day being regarded as efficient water use for a house with four occupants and a medium garden) could be at slightly more than current prices, and usage greater than 450L per day could be significantly more expensive. The current situation is that everyone is going to pay through taxes for desalination plants and other expensive methods of producing new fresh water. Having people pay for what they use is the capitalist way.

Share This

Russel Coker blogs about securing a daemon using a chroot, and offsets it against SE Linux. He argues that SE Linux provides better security; no doubt. One downside about SE Linux, however, is that it's far more difficult to configure correctly than a chroot. Setting up a chroot involves creating a directory, copying or bind mounting stuff in there, and then just using the chroot system call (either from a shell script or from a daemon). Setting up a non-standard daemon using SE Linux involves a very fine-grained process of allowing access to files and system calls that many people inexperienced with SE Linux will find too hard to do. OTOH, he's right that it's possible to break out of a chroot, and thus a chroot system isn't totally safe. This is why the FreeBSD developers implemented the jail system call since FreeBSD 4: basically a chroot on steroids, it implements a basic form of virtualization -- your "chroot" gets an IP address assigned, and jailed processes cannot communicate with processes outside the jail other than through TCP/IP or some other form of networking. Processes outside the jail can modify stuff inside it, of course (it remains a simple directory). Of course, Linux doesn't have anything like the jail system call, but it's easy to set up a similar level of security using virtualization, in a way that is far easier for the uninitiated than when using SE Linux. That's not to say that jails or virtualization will give you the same level of security that SE Linux can offer (e.g., with jails or virtualization a user can still exploit a bug in a network-facing daemon to turn your machine into a zombie; SE Linux can make this impossible), but it's a different option.

When I was about 11 years old I decided that I wanted a career related to computers. My first computer was the TEC-1 single-board Z80 based kit computer from Talking Electronics magazine (see the photo below). I think that I built this when I was 10. The computer is 16cm high and 25cm wide. The six seven segment displays are the only built-in output device (there were optional kits for other output devices). The keypad has the hexadecimal number keys, an “ad” button for entering addresses, a “go” button for executing programs, and “+” and “-” keys for incrementing and decrementing the address. Below the reset button (labelled “R“) you will see the optional function key (of which I can’t remember the purpose). Programming this computer required entering the hexadecimal code on the keypad with the “+” and “-” keys being the main method of editing (the “ad” key was used to jump to a different section of RAM). In editing mode the first four seven-segment displays showed the address (the Z80 could only address 64K of RAM) and the other two showed the memory contents (the word size was one byte). In terms of user-friendlyness it was probably about equal to punched cards - apart from the lack of non-volatile storage (unless you built the optional NVRAM kit). My TEC-1 has 2K of RAM (the 83251R chip is equivalent to an Intel 16kilo-bit 6116 static RAM chip) and 2K of ROM (the chip with the orange sticker labeled Mon1 is a 2716 EPROM - 16kilo-bit). Not long after that my parents bought the first serious computer for the family, a Microbee Z80 based system with a tape drive that used a monochrome monitor of resolution approximately equal to CGA and which had either 16K or 32K of RAM (I can’t recall). The next family computer was a Microbee Premium series 128K which is probably the same model as the one depicted on the Microbee Wikipedia page (a serious omission of the Wikipedia page is that it has no picture of the box containing the PSU and the floppy drives for the Premium Series). My first published article in a computer magazine was when I was about 15 years old and I wrote a long email on a Fidonet echo (mailing list) reviewing a 3rd party update to the CP/M system for the Premium Series Microbee and was surprised by having it published in the Microbee club magazine (in those days we didn’t bother much about copyright so no-one asked for my permission before publishing). I wonder if starting with computers at such an age is typical for people who now contribute to free software development. I think it would be interesting to see some blog posts from other people in the community about how old they were when they started with computers and what type of computer they started with. I also wonder about the correlation between the age of starting with computers and career success in the computer industry. One significant benefit of starting early was that I could learn things that would be useful for my career in later decades while other children were wasting time studying what teachers told them to study. It also meant that in later years of high-school I could relax knowing that I could get straight B’s without effort which was more than was required to enter a CS degree program at that time. Until half-way through year 12 I tried to avoid ever doing home-work at home - home-time was computer time! Do you think that the age at which you chose your career significantly affected your success? If so in what way? If you were asked for advice by parents as to when their child should be given it’s first computer what age would you suggest? Unfortunately I usually get asked for advice about such things by people who have children aged 16+ (which is way too late IMHO).

Share This

emacs-snapshot is now taken from the CVS trunk rather than from the Emacs 22 release branch. You may want to put your current version on hold, or use the emacs22 packages from Debian. However, if you feel adventurous and choose to upgrade, you will have:

• SVG support,
• Tramp 2.1,
• isearch in minibuffer history (addictive),
• M-x kill-matching-buffers (but you should use Ibuffer anyway),
• mb-depth.el,
• other minor things.

It's been a while since I've posted here; I've had a busy two weeks, and posting on my blog hasn't been a priority. Something I should do more often. Of course, as time backs up it seems harder and harder to actually get around to posting. I wanted to bring the attention of my friends and colleagues, though, to a notice that Maj put on the Wikitravel shared logbook. As of 15 August 2007, Maj is no longer working on Wikitravel. In addition, my time has been curtailed quite a bit, and I'm concentrating only on community and content issues -- no more programming or sysadmin work. This is all coming about for several reasons. The main one is that Internet Brands, the company that now owns wikitravel.org, wanted to have their own staff do the programming and sysadmin work. The end of our service contract seemed like an ideal time to effect this change. On our side, it's been challenging for us to work with a large organization not specialized in Open Content and wiki sites. We've gotten along really well with the IB people, but we haven't always seen eye-to-eye on the best way to grow and develop the site. As the decision was made to remove us from our technical roles, Maj decided it was a good time to opt out entirely. I'm going to miss her tremendously. As most people reading this already know, we started the Wikitravel project together, and over the last 4 years it's been a constant important part of our family life. Maj's information-architecture skills and dedication to travel have been crucial to the development of the site. She's been a strategizer, a spokesperson, a content organizer and a content builder. Probably most important for the future of the project, she developed almost single-handedly the Wikitravel Extra personal experience component of the site. We've got a lot to keep us busy, though. We recently launched Keiki, a project to create an Open Content parenting guide. For myself, I've also started Vinismo, an Open Content wine guide (sensing a pattern here...?). And we'll be working with our partner Jani P. on Wikitravel Press, an independent project to publish printed travel guidebooks based on Wikitravel articles. Despite the shakeups, I anticipate that the future of Wikitravel will be good. We have an incredibly strong and smart community, dedicated to an important and noble purpose. I'll remain involved on a part-time basis. We've got a lot of traction in public opinion and great Web traffic. I don't think that IB is going to make any bad decisions about the site, but if they do there's a local and global community of people that take Wikitravel and Open Content travel guides extremely seriously. I think things are probably going to be OK. tags: wiki wikitravel opencontent maj work ib

On the subject of wiki The South by Southwest 2008 Panel Picker app is up for public review right now. SXSW Interactive, in Austin, is one of the best tech conferences of the year. I've got a panel proposal up, on Wiki 2.0 -- I really want to talk about some of the interesting new wiki apps, new wikis, and new wiki business models coming up. There are a couple of other wiki-related panels proposed -- including one on commercial wikis, which I presented about last year (Talks/SXSW07). Folks who are interested in wikis, or in SXSW, should take some time to give their feedback on the proposed panels and improved the level of discussion for 2008. tags: sxsw wiki vote panel sxsw08

[Long post, sorry. If you're short on time skip to the end for the juicy parts.]

It all started about a week ago when I decided to find out how many files were unique in the whole Debian archive, and how many were duplicated in the same package or in other packages. I had done some work on duplication detection before, and I knew that the process involves getting some kind of hash value of every document, then finding duplicate values in the list of hashes.

I already had a local Debian mirror so the raw data was all there. I basically had two options: either compute the hash value of every file myself from the packages in my mirror, or find some other (ideally faster) way to determine if a file is unique. I quickly decided that the best option was to use the md5sums embedded in most Debian packages (in the DEBIAN/md5sums control file). That would give me MD5 hashes of all regular files, excluding conffiles.

So the first step was to check how many packages have embedded md5sums, and a simple script showed that less than 3% don't have them. This first check exposed a bug in python-debian, which was duly reported. Along the way, my post prompted a discussion on debian-devel about the state of md5sums, and I set up a daily check to keep track of things.

The next step was to make sure that all md5sum-enabled packages had usable information. It turned out that python-debian choked on 103 packages because of embedded spaces in filenames, and that it also had a blocker bug when used in Python 2.4. I filed two more bugs.

All that remained was the easiest part: write the program to find duplicate files. I did that, and I now have all kinds of funny statistics:

• The dataset consists of 20170 packages with md5sums, shipping a total of 2069830 files. That gives an average ratio of 102 files per package, excluding conffiles.
• There are 113732 duplicates in the archive. 1556 files are duplicated more than 10 times, and 14 files are duplicated more than 300 times.
• The empty file is present 8325 times in the archive, spread over 874 packages. This isn't surprising since it's used for all kinds of purposes like Python's init.py files, Perl's .bs files, etc. I also learned (among other oddities) that the python2.4-doc package ships a few zero-byte .png files. Uh?
• Also popular is the file with just one newline character in it: 343 occurrences. In the same vein, we have 461 occurrences of the "deny from all\n" file.
• Most of the hits are for Doxygen images in -dev and -doc packages, namely doxygen.png, tab_b.gif, tab_l.gif, etc (about 350 hits each). In the same category, gjdoc CSS files (149 hits).
• The partlibrary package is our worst offender. It ships a total of 9680 non-empty files, and only 4833 of them are unique. 6 files are duplicated more than 400 times each in the package.

If you're still reading so far, thanks! You can see the report file, and the program itself. If you want to run it, note that it's optimized for speed rather than memory efficiency; it runs in under 3 minutes but uses up to 1.5GB of memory (my home desktop has 4GB).

The grand conclusion is that all things considered, there is very little file duplication in the archive: the #1 duplicate represents less than 0.4% of the two million analyzed files, and it doesn't actually use any space since it's an empty file... :)

Watching the popular Inbox Zero video would probally improve my email handling skills better. But this took less time and I wanted to play around with Net::XMPP;

#!/usr/bin/perl -l
use warnings;
use strict;
use Config::Simple;
use Linux::Inotify2;
use Email::Abstract;
use Net::XMPP; # aka Jabber and Gtalk
use POSIX;
my $DEBUG = 0;
our %config;
Config::Simple->import_from("$ENV HOME /.mailnotifyrc", %config);
deamonize() unless $DEBUG;
my $inotify = new Linux::Inotify2
     or die "Unable to create new inotify object: $!";
# Different Maildir implementations triggers different events"
#   IN_MOVED_TO: rename("tmp/file","new/file")
#   IN_CREATE:   link("tmp/file", "new/file") && unlink("tmp/file")
#
# At this point the file should be written and closed.
$inotify->watch("$config Maildir /new", IN_MOVED_TO IN_CREATE, sub  
        my $e = shift;
        my ($fh, $xmpp, $email, $message);
        debug('Got event for ' . $e->fullname);
        open $fh, "<", $e->fullname;
        $email = new Email::Abstract join("", <$fh>);
        close $fh;
        $message = 'Mail from ' . $email->get_header('From') .
                   ' concerning "' . $email->get_header('Subject') .'"';
        debug("Getting ready to send [$message]");
        my $sender   = new Net::XMPP::JID ($config Sender );
        my $receiver = new Net::XMPP::JID ($config Receiver );
        $xmpp = new Net::XMPP::Client();
        $xmpp->Connect( hostname => $sender->GetServer,
                        port     => $config port    5222,
                        tls      => $config usetls    0,
                      );
        $xmpp->AuthSend( username => $sender->GetUserID,
                         password => $config password ,
                         resource => $sender->GetResource   'mailnotify',
                       );
        $xmpp->MessageSend( to   => $receiver,
                            type => $config MessageType    'chat',
                            body => $message,
                          );
        $xmpp->Process(1);
        $xmpp->Disconnect;
        debug("Event done");
 );
1 while $inotify->poll;
sub deamonize  
        my $pid = fork();
        if ($pid)  
                exit 0;
         
        ### close all input/output and separate
        ### from the parent process group
        open STDIN,  '</dev/null' or die "Can't open STDIN from /dev/null: [$!]n";
        open STDOUT, '>/dev/null' or die "Can't open STDOUT to /dev/null: [$!]n";
        open STDERR, '>&STDOUT'   or die "Can't open STDERR to STDOUT: [$!]n";
        ### Change to root dir to avoid locking a mounted file system
        ### does this mean to be chroot ?
        chdir '/'                 or die "Can't chdir to \"/\": [$!]";
        ### Turn process into session leader, and ensure no controlling terminal
        POSIX::setsid();
 
sub debug   return unless $DEBUG; print STDERR for @_;  
__END__

All the version control systems I am familiar with run scripts on checkout and commit to take additional site specific actions, and arch is no different. Well, actually, arch is perhaps different in the sense that arch runs a script on almost all actions, namely, ~/.arch-params/hook script. Enough information is passed in to make this mechanism one of the most flexible I have had the pleasure to work with. In my hook script, I do the following things:

• On a commit, or an initial import
  • For my publicly replicated repositories (and only for my public repositories), the script creates a full source tree in the repository for every 20th commit. This can speed up subsequent calls to get for that and subsequent revisions, since users do not have to get the base version and all patches.
  • For the public repositories, the script removes older cached versions, keeping two cached versions in place. I assume there is not much demand for versions more than 40 patches out of date; and so having to download a few extra patches in that uncommon case is not a big issue.
  • If it is an ikiwiki commit, the script makes sure that it updates the checked out sources of the wiki on the webserver, and rebuilds the wiki.
  • If this is a commit to an archive for which I have a corresponding -MIRROR defined, the script updates the mirror now, and logs an informational message to the screen.
  • There is special handling for my Debian packages.
    • If the category matches one of my packages, the script looks to see if any bugs have been closed in this commit, and, if so, sends the log to the bug, and tags it fixed.
    • If the category being checked in is one that corresponds to one of my Debian packages, or to the ./debian directory that belongs to one of my packages, then the script sends a cleaned up change log by mail to the packages.qa.debian.org. People can subscribe to the mailing list setup for each package to get commit logs, if they so desire.
    • Arch has the concept of a grab file, and people can get all the components of a software package by just feeding arch either the grab file (either locally, or via a http URL). The script makes sure that a arch config file is created , as well as a grab file (using the script arch_create_config), and uploads the grab file to to a public location (using the script arch_upload_grab) mentioned in ./debian/control for all my packages.
    • For commits to the Debian policy package, the script also sends mail to the policy list with full commit logs. This is a group maintained package, so changes to this are disseminated slightly more volubly.
    • Whenever a new category, branch, or version is added to the repository corresponding to the Debian policy package, the script sends mail to the policy list. Again, changes to the Policy repository are fairly visible.
  • The scripts send myself mail, for archival purposes, whenever a new category or branch is created in any of my repositories (but not for every revision).
  • Additional action is taken to ensure that versions are cached in the local revision library. I am no longer sure if this is strictly needed.

I d be happy to hear about what other people add to their commit scripts, to see if I have missed out on anything.

One problem with the blog space is that there is a lot of negativity. Many people seem to think that if they don’t like a blog post then the thing to do is to write a post complaining about it - or even worse a complaint that lacks specific details to such an extent that the subject of the complaint would be unable to change their writing in response. The absolute worst thing to do is to post a complaint in a forum that the blog author is unlikely to read - which would be a pointless whinge that benefits no-one. Of course an alternate way for the recipient to takeg such complaints as suggested by Paul Graham is “you’re on the right track when people complain that you’re unqualified, or that you’ve done something inappropriate” and “if they’re driven to such empty forms of complaint, that means you’ve probably done something good” (Paul was talking about writing essays not blogs, but I’m pretty sure that he intended it to apply to blogs too). If you want to actually get a blog author (or probably any author) to make a change in their material in response to your comments then trying to avoid empty complaints is a good idea. Another useful point Paul makes in the same essay is ““Inappropriate” is the null criticism. It’s merely the adjective form of “I don’t like it.”” - something that’s worth considering given the common criticism of particular blog content as being “inappropriate” for an aggregation feed that is syndicating it. Before criticising blog posts you should consider that badly written criticism may result in more of whatever it is that you object to. If you find some specific objective problem in the content or presentation of a blog the first thing to do is to determine the correct way of notifying the author. I believe that it’s a good idea for the author to have an about page which either has a mailto URL or a web form for sending feedback, I have a mailto on my about page - (here’s the link). Another possible method of contact is a comment on a blog post, if it’s an issue for multiple posts on the blog then writing a comment on the most recent post will do (unless of course it’s a comment about the comment system being broken). For those who are new to blogging, the blog author has full control over what happens to comments. If they decide that your comment about the blog color scheme doesn’t belong on a post about C programming then they can respond to the comment in the way that they think best (making a change or not and maybe sending you an email about it) and then delete the comment if they wish. If there is an issue that occurs on multiple blogs then a good option is to write a post about the general concept as I did in the case of column width in blogs where I wrote about one blog as an example of a problem that affects many blogs. I also described how I fixed my own blog in this regard (in sufficient detail to allow others to do the same). Note that most blogs have some degree of support for Linkback so any time you link to someone else’s blog post they will usually get notified in some way. On my blog I have a page for future posts where I invite comments from readers as to what I plan to write about next. Someone who prefers that I not write about topic A could write a comment requesting that I write about topic B instead. Wordpress supports pages as a separate type of item to posts. A post is a dated entry while pages are not sorted in date order and in most themes are displayed prominently on the front page (mine are displayed at the top). I suggest that other bloggers consider doing something comparable. One thing I considered is running a wiki page for the future posts. One of the problems with a wiki page is that I would need to maintain my own private list which is separate, while a page with comments allows only me to edit the page in response to comments and then use the page as my own to-do list. I may experiment with such a wiki page at some future time. One possibility that might be worth considering is a wiki for post requests for any blog that is syndicated by a Planet. For example a wiki related to Planet Debian might request a post about running Debian on the latest SPARC systems, the first blogger to write a post on this topic could then remove the entry from the wish-list (maybe adding the URL to a list of satisfied requests). If the person who made the original request wanted a more detailed post covering some specific area they could then add such a request to the wish-list page. If I get positive feedback on this idea I’ll create the wiki pages and add a few requests for articles that would interest me to start it up. Finally to encourage the production of content that you enjoy reading I suggest publicly thanking people who write posts that you consider to be particularly good. One way of thanking people is to cite their posts in articles on your own blog (taking care to include a link to at least one page to increase their Technorati rank) or web site. Another is to include a periodic (I suggest monthly at most) links post that contains URLs of blog posts you like along with brief descriptions of the content. If you really like a post then thank the author by not only giving a links with a description (to encourage other people to read it) but also describe why you think it’s a great post. Also if recommending a blog make sure you give a feed URL so that anyone who wants to subscribe can do it as easily as possible (particularly for the blogs with a bad HTML layout). Here are some recent blog posts that I particularly liked:

• Security Anti-Pattern: Path based access control from Joshua Brindle (feed). A fairly strong criticism of path based access control (as used in AppArmor). I’d have written something about this myself but Joshua did it so well that there’s no need. His post about Status Quo Encapsulation is also a good read. It’s a pity that Joshua doesn’t write more than two posts a month.


• Federico’s post about concrete for housing. Some technical data and some good pictures. (feed)


• Tim Connor’s blog posts about Australian politics (feed), particularly this post about interest rates (especially his latest comment). It’s a pity that Livejournal seems to offer no good options for searching and the comment in question has broken links. I would be happy to offer advice if Tim wanted to switch to a self-hosted Wordpress installation (as would some other bloggers who are members of our local LUG).

Here are some blogs that I read regularly:

• Problogger (feed), I don’t think that I’ll be a full-time blogger in the forseeable future, but his posts have lots of good ideas for anyone who wants to blog effectively. I particulaly appreciate the short posts with simple suggestions.


• Mega Tokyo (feed) - A manga comic on the web. The amusing portrayal of computer gaming fanatics will probably remind most people in the computer industry of some of their friends.


• Defence and the National Interest (feed). The most interesting part of this (and the only reason I regularly read it) is the blog of William S. Lind (titled On War. William writes some very insightful posts about military strategy and tactics but some things about politics will offend most people who aren’t white Christian conservatives.
  
  It’s a pity that there is not a more traditional blog feed for the data, the individual archives contain all posts and there seems to be no possibility of viewing the posts for the last month (for people who read it regularly in a browser and don’t use an RSS feed) and no search functionality built in.


• WorseThanFailure.com (was TheDailyWTF.com) (feed) subtitled Curious Perversions in Information Technology. Many amusing anecdotes that illustrate how IT projects can go wrong. This is useful for education, amusement, and as a threat (if you do THAT then we could submit to WorseThanFailure.com).


• XKCD - a stick-figure web comic, often criticised for the drawing quality by people who just don’t get it, some people read comics for amusement and insightful commentry not drawings. It’s yet another example of content beating presentation when there’s a level playing field.

Finally I don’t read it myself, but CuteOverload.com is a good site to refer people to when they claim that the Internet is too nasty for children - the Internet has lots of pictures of cute animals!

Share This

Hey'all! We're still in transit on our great trans-continental adventure! Sorry if you've emailed us and we've not gotten back, Internet access is spotty at best on this trip, and who wants to spend their vacation reading email anyway? =)

We just finished a good time in the kitchens of Opus and ConCentric. Angie, Sera, Chris and I all worked well together in a cohesive team, with a minimum of drama. It was really nice to work with such a solid group of people that we could say things like "Dude, I need a nap, can you cover for me?" and there was no problem with the rest of the team trading off.

Leif was an absolute hit at the con. We were able to participate in a couple worships where him singing along wasn't a distraction at all, and he totally loved being passed between people. It does mean that he caught his first cold, but he's doing alright.

There are always the people that I'll miss when I'm gone from there. Alexis was in the kitchen on a regular basis saying hi, Liz was always nearby, Karina with her star maps, lizzie the brit with her piles of energy, the frying-tofu master Geoff seemed to be in the kitchen often. In the kitchen, though, we got to see pretty much everyone so it was a nice chance to connect with people.

The staff was also a very neat set of people. I don't think there was anyone in the Opus staff that I couldn't see actively working hard to pull of great programming. There's always the curmudgeonly part of me that sits in the back corner and can nit pick things to death, but in doing so I wouldn't want to hide that by almost every measure this was a successful Opus. In particular, I was impressed that the staff faced head-on how to handle walk-ins at the staff meeting just before the con started. It was interesting to watch a consensus process work with a group of people who came in from wildly different personal perspectives and to watch people change from varying viewpoints to agree on the compromise that best suited the group.

So what's next? We've spent the night with Andrew from Koolu and will be heading down to Buffalo next to get an van and start on the adventure. We don't know where we're going yet, or how we'll get there, but the idea is to head down to the Creation Museum and then west towards Vegas and the Grand Canyon. On the way, we might see if we can visit Stephanie in Ohio, Sera in Albuquerque, and Karina in Flagstaff.

Peace, Love, Faith and Blessings to all.

I had a pretty good week last week in terms of doing some open source work. I had the opportunity to do some consulting regarding Hecl, which was pretty cool. Some of the work was fed back into Hecl itself, including the beginnings of a "media" API (video streams, taking pictures with the camera), and some other fixes and updates. I've also been working on it a bit on my own time, adding a Canvas demo to the MIDP2.0 demo application (works in the emulator, haven't tried it on a phone yet), and today I'm going to look at updating some of the documentation that's been lagging the code. In the course of my work, I also happened on a cool idea to speed up the launch of Hecl applications: by creating the widgets as regular widgets in Java, right away, in the startApp MIDlet method, it's possible to get them on the screen quickly. At that point, you can start loading Hecl, and create some Hecl objects to hold the already created Java widgets. It goes something like this:

Form mainform = new Form("Initial Form");
TextField search = new TextField("search:", "", 30, 0);
... load script ...
interp.setVar("mainform", ObjectThing.create(mainform));
interp.setVar("searchwidget", ObjectThing.create(search));
... evaluate script ...

This is a good tradeoff, because it gets something in front of the user right away, and it's not a throwaway like a splash screen. On another front, I've updated some of the code in DedaWiki, to enable email notifications in the comment system, which has percolated into the Linux Incompatibility List, and, with a bit of tweaking, into Squeezed Books. I don't think the code is quite ready for a real release, but by running the incompatibility list on it, it's a great way to give it a thorough shakedown, and I'll attempt to release it sooner or later. Of course, in the meantime, interested parties can get the code out of subversion on rubyforge.

For a while I used the Item Link Clicks feature in Feedburner. For those who aren’t aware Feedburner is a service that proxies access to an RSS feed (you need to either publish the Feedburner URL as the syndication link or use a HTTP redirect to send the requests there - I use a HTTP redirect). Then when people download the feed they get it from Feedburner which is fast and reliable (unlike my blog on a bad day) and which also tracks some statistics which can be interesting. The Item Link Clicks feature rewrites the guid URLs to point to a Feedburner URL that will redirect back to the original post (and track clicks along the way). The down-side of doing this is that some people who read blogs via Planet installations and just copy the link from the Planet page when citing a blog post instead of actually visiting the blog in question. This causes a potential problem for the person citing the post in that they won’t know whether the URL is valid unless they visit it. So when (not if) people have misconfigured blogs that are widely syndicated the people who cite them without verifying the links could end up linking to invalid URLs. The problem for the person who is cited is that such Feedburner redirects don’t seem to be counted as part of the Technorati ranking (which is a count of the number of links to a blog in the last 6 months which give some rough approximation of how important the blog is). The Technorati rating can sometimes be used in negotiations with an advertiser and is often used when boasting about how popular a blog is. To increase my Technorati ranking I have stopped using the Feedburner URL rewriting feature. For people who view my blog directly or through a Planet installation this will not give any difference that you would notice. The problem is for people who use a service that syndicates RSS feeds and then forwards them on by email, such people received two copies of the last 10 items as the URL (GUID) change means that the posts are seen as new (Planet solves this by deleting the posts which are seen as unavailable and then creating new posts with the new URLs and no change is visible to the user). Based on this experience I suggest not using URL rewriting services. They will hurt your technorati ranking, give little benefit (IMHO) and annoy the small number of RSS to email readers. Particularly don’t change your mind about whether to use such a feature or not. Changing the setting regularly would be really annoying. Also this means that if you use such a service you should take care not to have you Feedburner redirection ever get disabled. A minor Apache configuration error corrected a day later could end up in sending all the posts in the current feed an extra two times.

Share This

Various newspapers report that the health product giant Johnson & Johnson is suing the Red Cross for using the red cross as a logo. Johnson & Johnson registered the trademark in 1887, six years after the us-american Red Cross started to use it. Needless to say, that a red cross has been in use for much longer and that european Red Cross associations also exist for quite some more years.

Training the Discriminators It has been a while since I posted on this category actually, it has been a long while since my last blog. When I last left you, I had mail (mbox format) folders called ham and/or junk, which were ready to be used for training either CRM114 or Spamassassin or both. Setting up Spamassassin This post lays the groundwork for the training, and details how things are set up. The first part is setting up Spamassassin. One of the things that bothered me about the default settings for Spamassassin was how swiftly Bayes information was expired; indeed, it seems really eager to dumb the Bayes information (don t they trust their engine?). I have spent some effort building a large corpus, and keeping ti clean, but Spamassassin would discard most of the information from the DB after training over my corpus, and the decrease in accuracy was palpable. To prevent this information from leeching away, I firstly increased the size of the database, and turned off automatic expiration, by putting the following lines into ~/.spamassassin/user_prefs:

bayes_expiry_max_db_size  4000000
bayes_auto_expire         0

I also have regularly updated spam rules from the spamassassin rules emporium to improve the efficiency of the rules; my current user_prefs is available as an example. Initial training I keep my Spam/Ham corpus under the directory /backup/classify/Done, in the subdirectories Ham and Spam. At the time of writing, I have approximately 20,000 mails in each of these subdirectories, for a total of 41,000+ emails. I have created a couple of scripts to train the discriminators from scratch using the extant Spam corpus; and these scripts are also used for re-learning, for instance, when I moved from a 32-bit machine to a 64-bit one, or when I change CRM114 discrimators. I generally run them from ~/.spamassassin/ and ~/var/lib/crm114 (which contains my CRM114 setup) directories. I have found that training Spamassassin works best if you alternate Spam and Ham message chunks; and this Spamassassin learning script delivers chunks of 50 messages for learning. With CRM114, I have discovered that it is not a good idea to stop learning based on the number of times the corpus has been gone over; since stopping before all messages i the Corpus are correctly handled is also disastrous. So I set the repeat count to a ridiculously high number, and tell mailtrainer to continue training until a streak larger than the sum of Spam and Ham messages has occurred. This CRM114 trainer script does the hob nicely; running it under screen is highly recommend. Routine updates Coming back to where we left off, we had mail (mbox format) folders called ham and/or junk sitting in the local mail delivery directory, which were ready to be used for training either CRM114 or Spamassassin or both. There are two scripts that help me automate the training. The first script, called mail-process, does most of the heavy listing. This processes a bunch of mail folders, which are supposed to contain mail which is either all ham or all spam, indicated by the command line arguments. We go looking though every mail, and any mail where either the CRM114 or the Spamassassin judgement was not what we expected, we strip out mail gathering headers, and then we save the mail, one to a file, and we train the approprite filter. This ensures that we only train on error, and it does not matter if we accidentally try to train on correctly classified mail, since that would be a no-op (apart from increasing the size of the corpus). The second script, called mproc is a convenience front-end; it just calls mail-process with the proper command line arguments, and feeds them the ham and junk in sequence; and takes no arguments. So, after human classification, just calling mproc does the classification. This pretty much finishes the series of posts I had in mind about spam filtering, I hope it has been useful.