And so there went a day of work. Minutes done, come back tomorrow

---

Okok, more text: We did lots of stuff. There was breakfast, soon after we had Lunch, and guess, Dinner also. Now, would you please come back tomorrow?

---

Anyways, we did some things. One of those was to have PostgreSQL 9.0 and to enable the replication to the machine any DD can access. Which is, from today on, ries.debian.org. And for the first time ever (since that time ftp-master got closed to the public), DDs can now see the live status of the archive, literally a second after a package got accepted a dak ls on ries will show you that fact. Torsten worked on fixing up old remnants of code and database entries around contents and also tackled a problem between our usage of SQLalchemy and the python multiprocessing module. He also started working on the source contents and was weird enough to get us standardized logging into a dak tool. Mark worked with DSA on the Postgres 9.0 upgrade to fix what was to fix in out database and did some tests with our environment in squeeze and coordinated changes in python-apt needed for multiarch and xz. He then started working on some larger database changes around the built-using feature. I myself started off with getting generate-packages-sources to behave when there is an error instead of silently ignoring that. I also did a bit of groundwork to enable us to do NEW processing adding overrides for all affected binary packages, not just those from the architecture currently uploaded. That will allow stuff like linux-2.6 to pass NEW just once, not once per architecture. That involves a new field in our dsc files and the status of that is tracked in Bug #619131.

I've just made a couple of software releases:

• larch version 0.19. This used to be known as btree, but others are using that name, so I changed it. This is my copy-on-write B-tree library for Python.
  • see http://liw.fi/larch/ for more information
  • see http://liw.fi/larch/NEWS/ for changes
• obnam version 0.15, and then the obligatory brown paper bag version 0.15.1 to fix a build issue under pbuilder. Obnam is my backup program.
  • see http://braawi.org/obnam/ for more information

This is the first release of Obnam this year. Below is a summary of the changes. My apt repository has binary packages for amd64, i386, and armel. I've started mentoring someone to help me maintain the Debian packages, and hope to start uploading Obnam and its dependencies to Debian within a few weeks. Obnam is still in ALPHA, but the only missing thing for moving to BETA is encryption support. There's a bunch of bugs that need to be fixed, but I'm not going to block the BETA on them. I'd be very interested to hear any feedback on this version. Bugs fixed:

• Manual page GPL copyright blurb is now properly marked up as a comment. (Thanks, Joey Hess.)
• README now links to python-lru correctly. (Thanks, Erik Johansson.)

Improvements and other changes:

• Filenames and directories are backed up in sorted order. This should make it easier to know how far obnam's gotten.
• The location where backups are stored is now called the repository, instead of the store. Suggested by Joey Hess.
• The repository and the target directory for restored data are now both created by Obnam, if they don't already exist. Suggested by Joey Hess.
• Better control of logging, using the new --trace option.
• Manual page now explains making backups a little better.
• Default value for --lru-size reduced to 500, for great improvement in memory used, without, it seems, much decrease in speed.
• obnam verify now reports success explicitly. Based on question from Joey Hess.
• obnam verify now accepts both non-option arguments and the --root option. Suggested by Joey Hess.
• obnam forget now accepts "generation specifiers", not just numeric generation ids. This means that obnam forget latest works.
• I/O statistics are logged more systematically.
• obnam force-lock introduced, to allow breaking a lock left behind if obnam crashes. But it never does, of course. (Well, except if there's a bug, like when a file changes at the wrong moment.)
• obnam genids introduced, to list generation ids without any other data. The old command obnam generations still works, and lists other info about each generation as well, but that's sometimes bad for scripting.
• The --dump-memory-profile option now accepts the value simple, for reporting basic memory use. It has such a small impact that it's the default.
• Obnam now stores the version of the on-disk format in the repository. This should allow it to handle repositories created by a different version and act suitably (hopefully without wiping all your backups).

I have been working in the last weeks in the removal of the last pieces of KDE 3 from the Debian archive and I have found there are a lot of packages that is sad having to remove. If you are looking for a good idea/excuse to learn and improve your KDE 4 / C++ /Qt4 skills, have to do a small application for school or you just feel like some useful coding, here is the list:

• creox real-time guitar effects
• kbarcode barcode and label printing application for KDE
• kbiff KDE mail notification utility
• KKBSwitch keyboard layout indicator for KDE (see this post also)
• kmyfirewall iptables based firewall configuration tool for KDE
• kpogre a graphical administrator tool for PostgreSQL
• kredentials KDE taskbar applet to update kerberos/AFS credentials
• ksociograma technical educational software to make sociograms
• ktechlab circuit simulator for microcontrollers and electronics
• qalculate-kde Powerful and easy to use desktop calculator KDE version
• score-reading-trainer trainer for reading music notes

Note that I do not list here software whose port in KDE 4 is being developed although it is still not finished and therefore it is not packaged in Debian. By the way, even if all those packages are being removed from Debian testing and unstable, they will remain in Squeeze, so you can keep using them or install it from there.

Now that I sent an update about my running activities, I have to report about what I'm doing these weeks in Debian to try keeping a fair balance for those of you who read this blog through Planet Debian. Most January was spent in preparing the release of squeeze. Of course, I didn't work directly on the release, not being involved in the release team. However, last bits of preparation for Debian Installer as well as rushing out the last remaining bits of localization work took most of my time. After the release of squeeze, time came to resume back some activities. First of all, on the l10n front, I resumed the work on "l10n NMUs", trying to get old l10n bugs fixed. I restarted proposing NMUs or, more precisely, help to perform translation updates rounds to these maintainers. Most of them had "old" l10n bugs because of the freeze. So, I indeed did not end up in doing many NMUs...but got several l10n bugs closed anyway. Work being resumed in unstable also lead to many changes in packages' debconf templates (new packages using debconf, packages introducing new templates, etc.). As these are the trigger for Smith reviews (reviews of English in debconf templates and packages' description, this activity, which was very low during the freeze, resumed again (we'll soon celebrate the 4th anniversary of this project). I also decided to invest more time in D-I again. The D-I team is currently more active than it was last years. Thanks to a few individual, work resumed in some areas and bug triaging has been very active. I recently resyned a nice improvement which had been contributed by a user to iso-scan, then polished by Frans Pop. This will allow to choose among several different ISO images when installing Debian from a USB stick while, previously, only the first found was being used. I also worked with a new translator, to prepare the activation of Uyghur translation in D-I. Some work happend in the pkg-fonts team. I had a few fonts adopted by the team, as part of the general effort to get rid of defoma. I sent patches for a few others, but many remain. If you maintain font packages, please help us in getting rid of defoma. Changes are easy to do; ask the team if you need help (pkg-fonts-devel@lists.alioth.debian.org). Samba packages also ate a significant part of my time. First of all, DSA-2175 lead to prepare fixes for samba in squeeze and lenny. Then, upstream released samba 3.5.8 which I uploaded to unstable. I briefly considered trying to get the Stable Release managers approval to push that version in squeeze, but the number of fixes made by upstream was discouraging. So, I chose to examine them slowly and cherry-pick those I consider important enough to warrant an upload for squeeze. More git wizardry learned doing so (thanks to Julien Cristau for his help). And I now have a pending proposal for three of these fixes (that represent 14 atomic patches from upstream) to be included in squeeze.

I posted a brief snippet of code on Friday which was my initial stab at a reverse HTTP proxy in Javascript (using node.js). Over the past couple of days I've tidied it up, added a command line parser, and made it flexible enough that it works for me. My node reverse HTTP proxy is now both documented ( a little ) and available for further eyeballs. Usage is pretty much:

$ node ./node-reverse-proxy.js --config ./path/to/config.file.js

The configuration file defines lists of virtual hosts along with the destination back-ends to proxy to - which is usually going to be a server running upon a high port on the loopback adapter, but might not be. In addition to that we can perform rewrites such as:

/**
  * Handler for wildcard host: *.repository.steve.org.uk
  *
  */
'([^.]*).repository.steve.org.uk':
     
        /**
         * Rewrites for static files - these will be handled via a
         * separate virtual host.
         */
        'rules':  
            '^/robots.txt':  'http://repository.steve.org.uk/robots.txt',
            '^/favicon.ico': 'http://repository.steve.org.uk/favicon.ico',
         ,
      ,

That says requests for http://chronicle.repository.steve.org.uk/robots.txt will be redirected to http://repository.steve.org.uk/robots.txt. Alternatively we can invoke javascript for each request matching a pattern:</p.>

    /**
     * static.steve.org.uk will mostly proxy to 127.0.0.1:1008
     * but files beneath /private/ have an IP-based ACL.
     */
    'static.steve.org.uk':
     
        host: 'localhost',
        port: '1008',
        'functions':  
            '/private': (function(orig_host, vhost,req,res)  
                var remote = req.connection.remoteAddress;;
                if ( ( remote != "80.68.85.46" ) &&
                     ( remote != "82.41.51.252" ) &&
                     ( remote != "89.16.161.34") &&
                     ( remote != "89.16.161.98" ) )
                 
                    res.writeHead(403);
                    res.write( "Denied access to " + req.url  + " from " + remote );
                    res.end();
                 
            ),
         
     ,

Fun stuff. It was live for my server, replacing apache, for a few hours today. I need to add some trivial HTTP Basic-Auth handling then it will go back. Otherwise I hope it is vaguely useful to others, and that the provided examples explain things neatly. ObQuote: "Only one thing alive with less than four legs can hear this frequency" - Superman.

AltOS 0.9.2 Minor update to ground station software Bdale and I are pleased to announce the release of AltOS version 0.9.2. AltOS is the core of the software for all of the Altus Metrum products. It consists of cc1111-based microcontroller firmware and Java-based ground station software. AltosUI Just a few bug fixes AltOS version 0.9.2 is a minor release with just a couple of fixes:

• Mac OS X graphing repaired. The 0.9 release was missing a file required to support the graphing feature (jcommon.jar). This has been added.
• Fixed multiple flight log downloading. Attempts to download more than one recorded flight log would fail in weird ways. The failures have been addressed and additional diagnostics are presented when things go wrong.
• You can see the AltOS version number in the Configure AltosUI dialog box now. This should make it easier to tell what you ve got installed.
• Post-flight graphing tool. This lets you explore the behaviour of your rocket after flight with a scroll-able and zoom-able chart showing the altitude, speed and acceleration of the airframe along with events recorded by the flight computer. You can export graphs to PNG files, or print them directly.

No Firmware Update AltOS version 0.9.2 does not include any firmware changes. We re still busy testing the Kalman filter code; that ll be in the next major release.

pommed v1.38 is out and adds support for the Early 2011 MacBook Pro machines:

• MacBookPro8,1: 13
• MacBookPro8,2: 15
• MacBookPro8,3: 17

These machines have a new keyboard/trackpad assembly, the WellSpring 5. You ll need a patched kernel or a kernel recent enough that it has the USB HID quirks for the new assembly; that s 2.6.39-rc material. This release also adds a new config item that allows disabling the audio support in pommed entirely. The audio hotkeys are fairly well supported in desktop environments nowadays so you ll probably want that if you re using one.

I found the following quote from Stefano's DPL platform interesting:

When faced with the dilemma, I've favored ditching some DPL tasks and communicating or taking notes about the others, instead of the other way around.

It takes someone who really knows Debian to realize that sometimes communicating about what's being done is more important than doing more.

For the users of Debian Squeeze we, the KVIrc maintainers, plan to upload the soon-to-be-released upstream bug fix release 4.0.4, packages of RC2 are available. RC2 is most likely going to be identical to the final 4.0.4 release. The update will be, if the SRMs agree (still need to ask them, but I wanted to wait for the final version before I do that), uploaded to stable-proposed-updates, maybe squeeze-updates and fix a few nasty bugs, which some users might experience, including graphical glitches when used under Gnome (upstream bug #1010), some crashes (e.g. upstream bugs #878, #879, #1093, #1098 and some without bug numbers like r4796), a regression in the Perl scripting support and a lot of translation updates (btw: if you use KVIrc and the translation for your mothertounge is lacking, consider to help! Coordinate your efforts on IRC.) Apart from that I just uploaded a new SVN snapshot of 4.1 into Sid, the next will follow in about ten days unless you report bugs, which make an earlier upload necessary. ;-)

Recently there has been some major developments in DebConf11 organization. In short, I will list some of the most important.

Website

Official DC11 website has finally been completed and now contains a great deal of information for those interested in attending this years conference. Some of the most notable changes include:

Navigation

Frontpage now contains a small menu with links to most important parts of website, like About, Registration, Contact Sidebar menu is now divided in 3 categories:

• The Conference Links to pages which are directly connected with the Conference like About DC, Debian Day, Registration, Call for Papers, Talk Schedules, Payments Most of these links are hidden at the moment, and will be until the registration is open, to avoid any unnecessary confusion.
• More Information Basic information for attendees like Important Dates, Accommodations, Venue, Visa Everything you need to know about your stay in Banja Luka.
• Practical While these pages have almost nothing to do with the Conference itself, they do provide a vast information about city, getting to Bosnia, some practical information, as well as the Map of the whole city made my me :)

New Pages added

• How Can I Help This page contains some basic information for anyone who wants to help out. Weather by volunteering, sponsoring, or just by adding a banner on their website
  DC11: How Can I Help?
• Contact Instead of just sending visitors to mailto, we replaced it by a page which contains ALL contact information, emails AND phone numbers (which will be added just before DebCamp starts).
  DC11: Contact
• Sponsorship Just like with the contact page, we made a page to be more user friendly to any interested parties or individuals, and contains most important info, such as sponsorship levels and contact address.
  DC11: Sponsorship
• About DebConf11 Actually it s just a page with a content copied from frontpage (which is now used for something else).
  DC11: About DebConf11

New features:

• Documentation Is now used to store all important documents and records for public use and sharing, like brochures, official government documents, important public mails, as well as the recordings of all IRC meeting logs. Can be found at:
  DC11: Documentation
• Frontpage: Navigation To ensure the main goal of website, information availability, frontpage has been redesigned to provide maximum accessibility as well as easy access to any part of the website. DC11: Frontpage
• Frontpage: Latest News To keep the public and attendees well informed of any important events and developments, we decided to add latest news section. Note however, that if you want to stay completely informed of any changes and updates, we still recommend subscribing to our mailing list, which can be found under contact page.
  DC11: Frontpage
• Map of Banja Luka Because of lack of data on Google Maps and Open Street Map, as well as some other minor local maps, we made a small map of almost whole city and added around 60 or 70 locations which we consider to be of importance for our attendees. Such locations include hotels, parking spots, restaurants, bars, clubs, petrol stations, taxi stations, etc. This page uses simple JavaScript and is not dragable . Therefore we added Index Location to auto-locate the needed locations, as well as the legend to clarify the category of locations. Pointing your mouse over a certain location will open a small bubble with some info on that location/object. We should also point out that some of this data may not be 100% correct (ie working hours may be off ). Map is available at:
  DC11: Banja Luka City Map

Bosnian language

Site is now available on Bosnian language for native speaking attendees, should they have problems with English version. Every page is available on both languages, which can be changed by clicking on ba/en flag in upper left corner, or DC11: Bosnian Version.

Design

As you may have noticed, the official page received a small face lift. New header with the Government building, a small token of appreciation for all the help the Government is providing; as well as some minor changes in CSS and color scheme. While talking about design, we d like to thank Leandro G mez for his help with the header.

Sponsorship Sponsorship Levels

As agreed on the meeting, which took place on 22. of February, [Meeting logs], following sponsorship levels will be used for this years conference:

• Steel < 1.000
• Bronze > 1.000
• Silver > 5.000
• Gold > 10.000
• Platinum > 20.000

Main reason for lowering amounts is due to last years results of high increase in the same. As for the benefits of these levels, the only thing changed is t-shirts / bags places; we believe t-shirts are much more noticeable than bags, and have therefore been promoted to Silver, while bags have been downgraded to Bronze. For a full list of requirements and benefits, please visit: DC11: Sponsorship Sponsorship Brochure

Sponsorship brochures have finally been completed and are available for a public distribution. The brochures are available in 3 qualities: low, high and original. Low and high quality brochures, ~6Mb and ~8Mb respectively, are for normal distribution to our sponsors, while original quality, at around 240Mb, is intended for maximum quality printing. Just as with the website, each quality is available in both Bosnian and English. Current version of the brochure is 1.1 and is most likely the final one. These brochures are available at Documentation page: DC11: Documentation >> Documents made by Local Team Or download them directly:

• Sponsorship Brochure LQ [en]
• Sponsorship Brochure HQ [en]
• Sponsorship Brochure OQ [en]
• Sponsorship Brochure LQ [ba]
• Sponsorship Brochure HQ [ba]
• Sponsorship Brochure OQ [ba]

Also, I d like to thank Mirosal Remetic for making a great template and Pablo Duboue for some very valuable suggestions regarding the structure of the brochure.

Visas and Customs Rules

Many questions have been raised regarding Bosnian visa regime. We got in contact with the embassy and they clarified who can enter our country with(out) Visa. The list can be found on DC11 Wiki: Visa regime. It also states, which countries have a privilidge of ID-only entrance. If your country is not listed, that means you will need visa. If you want to know what exactly you need to get one, please visit the this official Government page and chose your country from a drop-down menu. As for the visa itself, as well as the list of Customs Rules, please read DC11: Visa.

Also, thanks to Darjan Prtic for getting the list from the Government officials.

So, what s next?

If you think, that is it for now, you are dead wrong. At this very moment following is being organized and developed:

• Registration Registration Team is currently hacking penta and modifying it for this years needs and conditions. We expect to have registrations open by the end of March / beginning of April.
• Travel and getting to Banja Luka Being one of the greatest obstacle, Travel Team is working hard on finding best routes for attendees and mapping the city
• Sponsorship With final decision and agreement for sponsorship levels and brochures ready for public distribution, Sponsorship Team is now actively looking for new sponsors, global and local, preparing various promotional material, as well as planning our advertising strategy.

Final word

Well, these are pretty much the highlights of some recent and some future developments of DebConf11 organizational team. Mind you, there s a lot more going on, but I tried to give a small summary from the inside. Should you be interested in attending, helping out, sponsoring or just have some questions, feel free to ask them on our mailing list at: debconf11-localteam@lists.debconf.org. I should inform you that this is a public mailing list and its archives are available for a broader public. If you do not wish for your mail to be made public, please direct any enquiry directly to me trough the Contact Form or any other member of Core Local Team, Adnan Hodzic or Velimir Iveljic.

I ve just written some small scripts to help me manage spam emergencies on a mail server. I ve been doing this thing with a bit of manual effort for a while, but after having done it once from a phone I want to optimise it a bit to reduce painful typing.My observation is that when a system I run is used as an outbound mail relay for spamming I will notice this reasonably quickly by the queues getting big. Some portion of mail that is queued is delayed due to general network issues and anti-spam measures that neither accept nor reject mail cause a typical spam message to be more likely to be queued than a typical non-spam message. So I look for accounts that send large amounts of mail.The biggest mail server I run (and the only one to have an outbound spam problem) is configured to not allow users to fake their sender address (IE use an address from gmail.com or another server) unless they pay extra. So the spam I deal with tends to have a valid origin email address that I can use.Queue CountingThe following command provides a sorted list of the accounts that have the most mail in the queue. This usually means a maximum of 2 or 3 spammers at the end of the list:mailq grep ^[A-F0-9] cut -c 42-80 sort uniq -c sort -n tailThe following function declaration makes the command get-top-sender find the sender with the largest number of queue entries (which has been a spammer every time I ve checked such things) and assign their email address to the environment variable $TOP_SENDER. It has to be a shell function so that the environment variable will be set in the context of the shell and available to child processes.function get-top-sender() TOP_SENDER_LINE=$(mailq grep ^[A-F0-9] cut -c 42-80 sort uniq -c sort -n tail -1) ; export TOP_SENDER=$(echo $TOP_SENDER_LINE cut -f2 -d\ ) ; TOP_COUNT=$(echo $TOP_SENDER_LINE cut -f1 -d\ ); echo "Top Sender is $TOP_SENDER with $TOP_COUNT messages" ; Viewing the QueueThe following script uses the first parameter or the environment variable $TOP_SENDER to specify the user who s mail should be read. In the normal course of events user email won t be read by the sysadmin, but if we are to determine whether a slew of email is spam or not there s no other way. So far I haven t seen a large number of queued messages not be spam, but I expect it ll happen eventually and the user will be happy that I checked it instead of locking their account and deleting all the queued mail. For readers who don t do sysadmin work, this is why we always have entries in the Terms of Service about the possibility of your email being read to fix technical problems or to investigate possible breaches of the ToS from your end.#!/bin/bash
set -e
mkdir -p ~/tmp/$$
cd ~/tmp/$$
if [ "$1" != "" ]; then
TOP_SENDER="$1"
fi
if [ "$TOP_SENDER" = "" ]; then
echo "Specify the sender on the command-line or in \$TOP_SENDER"
exit 1
fi
for n in $(mailq grep ^[A-F0-9].*$TOP_SENDER cut -c 1-10) ; do
FILE=$(echo $n sed -e "s/^\(.\)/\/var\/spool\/postfix\/deferred\/\1\/\1/" -e "s/ .*$//")
if [ -f "$FILE" ]; then
postcat "$FILE" > $n
else
echo "File $FILE missing, probably active"
fi
done
less *
rm -rf ~/tmp/$$Note that this needs to be done with a single less command so that I can terminate it by pressing q and easily go to the next file with :n . It does waste some server resources by running postcat on all queue files relating to the user in question, but that doesn t matter, servers are supposed to be powerful enough to cope with some inefficiency in processing uncommon operations.The Final CommandAfter that I have a script that connects by ssh to all outbound mail relays and deletes bad messages from the queues and then connects to the database server to lock the account that was used for sending the mail. I m not publishing it because it s specific to the servers I run.So when NAGIOS reports that the queue is too large I have one script to find the most likely culprit, one script to view the queued email from that account, and a final script to lock the account and purge the spam.I ve also added a reminder of the command names to /etc/motd as I don t want to be running ls and set on a phone to discover command names that I ve forgotten.Any suggestions for improvements to this will be welcome.

A while back I was working on a mod_rewrite compatible proxy server written in C. The reason for this is that my current webhost uses Apache2 in front ofa number of thttpd processes, and I'd like to remove apache and use something smaller/faster/neater. Dividing things up I'm running about ten domains, and only around half of them use mod_rewrite rules - small enough perhaps to port the rules, large enough to make it annoying. Upon reflection I think the thing to do is to replace apache with javascript - via node.js. Writing proxies with node.js is almost ridiculously simple - in fact doing anything HTTP-like is very very simple thanks to the bundled libraries. Time will tell whether this is a waste of time or not, but I'm confident I could listen upon *:80 and route requests to localhost:N for a few domains. As a trivial toy I wrote a simple transforming proxy last night which performed simple rewrites as it passed traffic about. Another fun, and possibly useful, thing I put together was a node.js port of the simple httpstat.us website. That server would have been easy to write in any number of languages, but in node it was almost too easy. Update - here is my rewriting node.js-based reverse-proxy. ObQuote: "I just need you to stop being nice to me unless you're gonna marry me. " - He's just not that into you.

The results of the Grml User Survey 2011 are available. I strongly recommend any open source project to run such a survey. Besides gathering really interesting feedback it s motivating for developers to read what people think about your product. Interesting facts for Planet Debian readers: With a leading 86% our users are Debian users. Further quoting:

Also just a short overview but it seems that the most important reasons for using Grml are:
* Based on Debian
* Command-line interface
* Ships all necessary command-line tools
* Zsh

I personally consider the survey a big success for the Grml project. Thanks to all participants for your feedback.

Since I resumed active status in Debian, I ve been thinking about how to bridge the gap between Debian and its derivatives*. I ve spoken at length with Zack, the attendees of the Derivatives BoF at DebConf 10, and the fine folks at the Derivatives Front Desk about the technical and social issues affecting derivative projects, and could probably write a very thorough series of blog posts on the subject. Instead, Zack and I decided to try doing something about it: we have begun a project to test out a new approach to the problem. Introducing DEX DEX is all about action: merging patches, fixing bugs, crunching data, whatever is necessary to get changes from derivatives into Debian proper. DEX doesn t try to change the way any existing project works, but adds a fast path for getting code from one place to another. DEX is a joint task force where developers from Debian and its derivatives work together on this common goal. As a pilot project, we ve established an Ubuntu DEX Team focused on merging code from Ubuntu into Debian. With members from both projects, we hope to be able to resolve blockage anywhere in the pipeline. Whatever needs to get done in order to merge an Ubuntu patch, someone in the Ubuntu DEX team will know what to do. If we get good results with Ubuntu, we hope that other derivatives will follow. With thanks to David Paleino, we re excited that the Utnubu project is merging into DEX as it aligns well with their goals. I m very grateful to have Colin Watson and James Westby signed up to contribute as well. Our first project is simple: turn this list green. This is an archive of quite old patches from Ubuntu, most of which have probably been merged already or made obsolete, but they pre-date any kind of tracking system so they need to be verified. Once that s done, we ll move on to a new project with a new todo list. If you want to see Debian benefit from technical work done in derivatives, DEX is a chance for you to act together to make it happen. If you work on a derivative and want to carry a smaller delta, come and join us. I m sure we ll learn a lot from this experience. * There are many instances of great cooperation between Debian and derivative distributions, including joint package maintenance teams, and some derivatives are even part of the Debian project. Nonetheless, there are areas were most people I ve spoken to agree that we need to do better. This is what I ve referred to as the gap .

I spent part of today looking at packaging Peter Gutmann's CryptLib for debian. My conclusion: it may not be worth my time. One of the main reasons i wanted to package it for debian is because i would like to see more OpenPGP-compliant free software available to debian users and developers, particularly if the code is GPL-compatible. Cryptlib makes it quite clear that it intends to be GPL-compatible:

cryptlib is distributed under a dual license that allows free, open-source use under a GPL-compatible license and closed-source use under a standard commercial license.

However, a significant portion of the cryptlib codebase (particularly within the bn/ and crypt/ directories) appears to derive directly from OpenSSL, and it retains Eric Young's copyright and licensing. This licensing retains the so-called "advertising clause" that is generally acknowledged to be deliberately incompatible with the GPL. (A common counterargument for this incompatibility is that OpenSSL should be considered a "System Library" for GPL'ed tools that link against it; whether or not you believe this for tools linked against OpenSSL, this counterargument clearly does not hold for a project that embeds and ships OpenSSL code directly, as CryptLib does) This does not mean that CryptLib is not free software (it is!), nor does it mean that you cannot link it against GPL'ed code (you can!). However, you probably can't distribute the results of linking CryptLib against any GPL'ed code, because the GPL is incompatible with the OpenSSL license. The un-distributability of derived GPL-covered works makes the CryptLib package much less appealing to me, since i want to be able to write distributable code that links against useful libraries, and many of the libaries i care about happen to be under the GPL. I also note that Peter Gutmann and CryptLib Security Software (apparently some sort of company set up to distribute CryptLib) may be in violation of Eric Young's License, which states:

 * 3. All advertising materials mentioning features or use of this software
 *    must display the following acknowledgement:
 *    "This product includes cryptographic software written by
 *     Eric Young (eay@cryptsoft.com)"
 *    The word 'cryptographic' can be left out if the rouines from the library
 *    being used are not cryptographic related :-).

A Google search for Eric Young's name on cryptlib.com yields no hits. I do find his name on page 340 of the Cryptlib manual, but that hardly seems like it covers "All advertising materials mentioning features or use of this software". I suppose it's also possible that CryptLib has negotiated an alternate license with Eric Young that I simply don't know about. In summary: i think CryptLib could go into debian, but its incorporation of OpenSSL-licensed code makes it less useful than i was hoping it would be. I have a few other concerns about the package after looking it over in more detail. I suspect these are fixable one way or another, but i haven't sorted them out yet. I'm not sure i'll spend the time to do so based on the licensing issues i sorted out above. But in case anyone feels inspired, the other concerns i see are:

embedded code copies

In addition to embedded (and slightly-modified) copies of bn/ and crypt/ from OpenSSL, CryptLib contains an embedded copy of zlib. Debian has good reasons for wanting to avoid embedded code copies. I've got it CryptLib building against the system copy of zlib, but the bits copied from OpenSSL seem to be more heavily patched, which might complicate relying on the system version.

executable stack

lintian reports shlib-with-executable-stack for the library after it is built. I don't fully understand what this means or how to fix it, but i suspect it derives from the following set of object files that also have an executable stack:

0 dkg@pip:~/src/cryptlib/cryptlib$ for foo in $(find . -iname '*.o'); do
>  if ! readelf -S "$foo"   grep -q .note.GNU-stack ; then
>   echo "$foo"
>  fi
> done
./shared-obj/desenc.o
./shared-obj/rc5enc.o
./shared-obj/md5asm.o
./shared-obj/castenc.o
./shared-obj/bfenc.o
./shared-obj/rmdasm.o
./shared-obj/sha1asm.o
./shared-obj/bn_asm.o
./shared-obj/rc4enc.o
0 dkg@pip:~/src/cryptlib/cryptlib$ 

They probably all need to be fixed, unless there's a good reason for them to be that way (in which case they need to be documented).

non-PIC code

lintian reports shlib-with-non-pic-code on the library after it is built. I don't fully understand what this means, or how to resolve it cleanly, but i imagine this is just a question of sorting out the details.

Anyone interested in using my packaging attempts as a jumping off point can start with:

git clone git://lair.fifthhorseman.net/~dkg/cryptlib

Oh, and please correct me about anything in this post; I'd be especially happy to hear that my analysis of the licensing issues above is wrong :) Tags: cryptlib, gpl, licensing, openssl

A few weeks ago, at work, we were looking for a solution to a tricky printing problem: how to manage, in a centralized infrastructure, a large number of locations, worstations and printers? One of the consultants working for us came up with a great idea. With only a 20-line patch to CUPS, workstations would be able to find which printers are on the same location. 20 lines of code, instead of a complex virtualisation solution? This is exactly the kind of reasons why we use free software: when there s something wrong, you can fix it. When you need something more, you can code it. Now, many others could benefit of such an improvement, and we don t want to maintain a forked version of CUPS, so we forwarded it upstream, who looked interested. But upstream now being Apple, they requested a stupid copyright assignment agreement. I will leave to the reader s imagination the complexity of getting such a document signed in a Fortune 500 company with no business with Apple. This will, of course, not happen - and if the decision was mine, the answer would have been a clear No. No, because I want to improve free software, not to contribute to Apple s proprietary version. No, because copyleft is about giving as much as you take. How many contributions are being left out of CUPS because of this stupid copyright assignment? It looks to me that such software is doomed to remain crippled as long as companies like Apple are in charge of their maintenance. There is free software. And there is free software by Apple. And Oracle. And Canonical.

Last fall, Moonshot was steaming forward. We ran into some non-technical obstacles and progress on the implementation was disturbingly quite from the end of October through February. That changed: the code was released February 25. Since then, the project has picked up the momentum of last fall. There s a new developers corner with helpful links for participating in the project, obtaining the code, and preparing for our upcoming Second Moonshot Meeting. Standards work in the ABFAB working group has been making steady progress the entire time. The jabber chat room has been quite active. Developers have been working in three time zones. Whenever In get up there s likely to be interesting progress awaiting me and new things to work on in the chat logs. Today was no exception. Luke moonshooted jabber. This is exciting: it s the first tim our code has been used to authenticate some real application instead of a test service. Other discussion from the chat room not reflected in e-mail is equally exciting. He has Moonshot working with OpenSSH in controlled environments. It appears to require some updates to the OpenSSH GSS-API support. Now is a really great time to get involved in Moonshot. We hope to see you on our lists and in our chat. With last night s news, we need to think towards eating our own dogfood and using Moonshot to authenticate to our own Jabber server and to authenticate to our repository for commits. Right now, there are some security issues with the code (lack of EAP channel binding) that might make that undesirable. However in a very small number of weeks or months I expect we will be there!

I spent most of last week working on Ubuntu bug 693671 ("wubi install will not boot - phase 2 stops with: Try (hd0,0): NTFS5"), which was quite a challenge to debug since it involved digging into parts of the Wubi boot process I'd never really touched before. Since I don't think much of this is very well-documented, I'd like to spend a bit of time explaining what was involved, in the hope that it will help other developers in the future. Wubi is a system for installing Ubuntu into a file in a Windows filesystem, so that it doesn't require separate partitions and can be uninstalled like any other Windows application. The purpose of this is to make it easy for Windows users to try out Ubuntu without the need to worry about repartitioning, before they commit to a full installation. Wubi started out as an external project, and initially patched the installer on the fly to do all the rather unconventional things it needed to do; we integrated it into Ubuntu 8.04 LTS, which involved turning these patches into proper installer facilities that could be accessed using preseeding, so that Wubi only needs to handle the Windows user interface and other Windows-specific tasks. Anyone familiar with a GNU/Linux system's boot process will immediately see that this isn't as simple as it sounds. Of course, ntfs-3g is a pretty solid piece of software so we can handle the Windows filesystem without too much trouble, and loopback mounts are well-understood so we can just have the initramfs loop-mount the root filesystem. Where are you going to get the kernel and initramfs from, though? Well, we used to copy them out to the NTFS filesystem so that GRUB could read them, but this was overly complicated and error-prone. When we switched to GRUB 2, we could instead use its built-in loopback facilities, and we were able to simplify this. So all was more or less well, except for the elephant in the room. How are you going to load GRUB? In a Wubi installation, NTLDR (or BOOTMGR in Windows Vista and newer) still owns the boot process. Ubuntu is added as a boot menu option using BCDEdit. You might then think that you can just have the Windows boot loader chain-load GRUB. Unfortunately, NTLDR only loads 16 sectors - 8192 bytes - from disk. GRUB won't fit in that: the smallest core.img you can generate at the moment is over 18 kilobytes. Thus, you need something that is small enough to be loaded by NTLDR, but that is intelligent enough to understand NTFS to the point where it can find a particular file in the root directory of a filesystem, load boot loader code from it, and jump to that. The answer for this was GRUB4DOS. Most of GRUB4DOS is based on GRUB Legacy, which is not of much interest to us any more, but it includes an assembly-language program called GRLDR that supports doing this very thing for FAT, NTFS, and ext2. In Wubi, we build GRLDR as wubildr.mbr, and build a specially-configured GRUB core image as wubildr. Now, the messages shown in the bug report suggested a failure either within GRLDR or very early in GRUB. The first thing I did was to remember that GRLDR has been integrated into the grub-extras ntldr-img module suitable for use with GRUB 2, so I tried building wubildr.mbr from that; no change, but this gave me a modern baseline to work on. OK; now to try QEMU (you can use tricks like qemu -hda /dev/sda if you're very careful not to do anything that might involve writing to the host filesystem from within the guest, such as recursively booting your host OS ... [update: Tollef Fog Heen and Zygmunt Krynicki both point out that you can use the -snapshot option to make this safer]). No go; it hung somewhere in the middle of NTLDR. Still, I could at least insert debug statements, copy the built wubildr.mbr over to my test machine, and reboot for each test, although it would be slow and tedious. Couldn't I? Well, yes, I mostly could, but that 8192-byte limit came back to bite me, along with an internal 2048-byte limit that GRLDR allocates for its NTFS bootstrap code. There were only a few spare bytes. Something like this would more or less fit, to print a single mark character at various points so that I could see how far it was getting:

	pushal
	xorw	%bx, %bx	/* video page 0 */
	movw	$0x0e4d, %ax	/* print 'M' */
	int	$0x10
	popal

In a few places, if I removed some code I didn't need on my test machine (say, CHS compatibility), I could even fit in cheap and nasty code to print a single register in hex (as long as you didn't mind 'A' to 'F' actually being ':' to '?' in ASCII; and note that this is real-mode code, so the loop counter is %cx not %ecx):

	/* print %edx in dumbed-down hex */
	pushal
	xorw	%bx, %bx
	movb	$0xe, %ah
	movw	$8, %cx
1:
	roll	$4, %edx
	movb	%dl, %al
	andb	$0xf, %al
	int	$0x10
	loop	1b
	popal

After a considerable amount of work tracking down problems by bisection like this, I also observed that GRLDR's NTFS code bears quite a bit of resemblance in its logical flow to GRUB 2's NTFS module, and indeed the same person wrote much of both. Since I knew that the latter worked, I could use it to relieve my brain of trying to understand assembly code logic directly, and could compare the two to look for discrepancies. I did find a few of these, and corrected a simple one. Testing at this point suggested that the boot process was getting as far as GRUB but still wasn't printing anything. I removed some Ubuntu patches which quieten down GRUB's startup: still nothing - so I switched my attentions to grub-core/kern/i386/pc/startup.S, which contains the first code executed from GRUB's core image. Code before the first call to real_to_prot (which switches the processor into protected mode) succeeded, while code after that point failed. Even more mysteriously, code added to real_to_prot before the actual switch to protected mode failed too. Now I was clearly getting somewhere interesting, but what was going on? What I really wanted was to be able to single-step, or at least see what was at the memory location it was supposed to be jumping to. Around this point I was venting on IRC, and somebody asked if it was reproducible in QEMU. Although I'd tried that already, I went back and tried again. Ubuntu's qemu is actually built from qemu-kvm, and if I used qemu -no-kvm then it worked much better. Excellent! Now I could use GDB:

(gdb) target remote   qemu -gdb stdio -no-kvm -hda /dev/sda

This let me run until the point when NTLDR was about to hand over control, then interrupt and set a breakpoint at 0x8200 (the entry point of startup.S). This revealed that the address that should have been real_to_prot was in fact garbage. I set a breakpoint at 0x7c00 (GRLDR's entry point) and stepped all the way through to ensure it was doing the right thing. In the process it was helpful to know that GDB and QEMU don't handle real mode very well between them. Useful tricks here were:

• Use set architecture i8086 before disassembling real-mode code (and set architecture i386 to switch back).
• GDB prints addresses relative to the current segment base, but if you want to enter an address then you need to calculate a linear address yourself. For example, breakpoints must be set at (CS << 4) + IP, rather than just at IP.

Single-stepping showed that GRLDR was loading the entirety of wubildr correctly and jumping to it. The first instruction it jumped to wasn't in startup.S, though, and then I remembered that we prefix the core image with grub-core/boot/i386/pc/lnxboot.S. Stepping through this required a clear head since it copies itself around and changes segment registers a few times. The interesting part was at real_code_2, where it copies a sector of the kernel to the target load address, and then checks a known offset to find out whether the "kernel" is in fact GRUB rather than a Linux kernel. I checked that offset by hand, and there was the smoking gun. GRUB recently acquired Reed-Solomon error correction on its core image, to allow it to recover from other software writing over sectors in the boot track. This moved the magic number lnxboot.S was checking somewhat further into the core image, after the first sector. lnxboot.S couldn't find it because it hadn't copied it yet! A bit of adjustment and all was well again. The lesson for me from all of this has been to try hard to get an interactive debugger working. Really hard. It's worth quite a bit of up-front effort if it saves you from killing neurons stepping through pages of code by hand. I think the real-mode debugging tricks I picked up should be useful for working on GRUB in the future.

Around the first bunch of postings in my Useful but Unknown Unix Tools, Tobias Klauser of inotail and Symlink fame came up with the idea of making a Planet (i.e. a blog aggregator) of all the comandline blogs and blog categories out there. A first Planet Venus running prototype based on the template and style sheets of Planet Symlink was quickly up and running. I just couldn t decide if I should use an amber or phosphor green style for this new planet. Marius Rieder finally had the right idea to solve this dilemma: Offer both, an amber and a phosphor green style. Christian Herzog pointed me to the right piece of code at A List Apart. So here is it, available in you favourite screen colors:

Planet Commandline For a beginning, the following feeds are included:

• Myon s Unix blog category
• Evgeni Golov s Desktop in a Shell column
• My Useful but Unknown Unix Tools column
• My Shell blog category
• The GRML Development Blog
• Chr tertee (Swiss-German)
• Commandline Magic s identi.ca feed

Which leads us to the discussion what kind of feeds should be included in Planet Commandline. Of course, all blogs or blog categories which (nearly) solely post neat tips and tricks about the command line in English are welcome. Microblogging feeds containing (only) small but useful command line tips are welcome, too, if they neither permanently contain dozens of posts per day nor have a low signal-to-noise ratio. Unfortunately most identi.ca groups do, so they re not suitable for such a planet. What I m though unsure about are non-English feeds. Yes, there s one in already, but I noticed this only after including Beat s Chr tertee and his FreeBSD command line tips are really good. So if it doesn t go overboard, I think it s ok. If there are too many non-English feeds, I ll probably split Planet Commandline off into at least three Planets: One with all feeds, one with English only and one with all non-English feeds or maybe even one feed per language. But for now that s still a long way off. Another thing I m unsure about are more propgram specific blogs like the impressive Mastering Emacs blog about mastering the world s best text editor . *g* (Yeah, I didn t include that one yet. But as soon someone shows me the vi-equivalent of that blog, I ll include both. Anyone thinks, spf13 s vim category is up to that?) Oh, and sure, any shell-specific (zsh, tcsh, bash, mksh, busybox) tips & tricks blogs don t count as program-specific blogs like some $EDITOR, $BROWSER, or $VCS specific blogs do. :-) Of course I m happy about further suggestions for feeds to include in Planet Commandline. Just remember that the feed should provide (at least nearly) exclusively command line tips, tricks or howtos. Suggestions for links to other commandline related planets are welcome, too.

Another week, though this one wasn't as fruitful as the last one. My excuse here is that I was overwhelmed with private stuff like acquiring a house and starting with cleaning it up so it can become a home. This is the list for my second week of my stable RC bug squashing:

• 603846: Update LSB header for hal D-Bus activision, not relevant for squeeze.
• 604299: please use KDE 4 port, not relevant for squeeze.
• 608017: XLIB package is not available even after installation, only appears in relation with clisp 2.49.

I know three isn't much, and actually it doesn't impact the list of stable RC bugs not much, we are at 172 open RC bugs against squeeze now. I can only attribute it to that new bugs were filed since, because I am aware that I'm not the only one working on this front, I've been contacted by at least two other people in the last week that are investing some time into this, too. Mostly for self-reference, the highest reported squeeze RC bugs in the list is 618295. This should help me to get a number of newly reported issues (ignoring severity-bump in lower bug numbers). Read you next week!

/debian permanent link Comments: 0 Flattr this