Modern daemon implementations can be run in a variety of ways, in a range of contexts. The daemon software itself can be a useful tool in environments where the associated traditional system service is neither needed nor desired. Unfortunately, common debian packaging practice has a tendency to conflate the two ideas, leading to some potentially nasty problems where only the tool itself is needed, but the system service is set up anyway. How would i fix it? i'd suggest that we make a distinction between packages that provide tools and packages that provide system services. A package that provides the system service foo would need to depend on a package that provides the tool foo. But the tool foo should be available through the package manager without setting up a system service automatically. Bad Examples Here are some examples of this class of problem i've seen recently:

akonadi-server depends on mysql-server

akonadi is a project to provide extensible, cross-desktop storage for PIM data. It is a dependency of many pieces of the modern KDE4 desktop. Its current implementation relies on a private instantiation of mysqld, executed directly by the end user whose desktop is running. This means that a sysadmin who installs a graphical calendar application suddenly now has (in addition to the user-instantiated local mysqld running as the akonadi backend) a full-blown system RDBMS service running and potentially consuming resources on her machine.Wouldn't it be better if the /usr/sbin/mysqld tool itself was distinct from the system service?

puppetmaster depends on puppet

Puppet is a powerful framework for configuration management. A managed host installs the puppet package, which invokes a puppetd service to reconfigure the managed host by talking to a centralized server on the network. The central host installs the the puppetmaster package, which sets up a system puppetmasterd service. puppetmaster depends on puppet to make use of some of the functionality available in the package. But this means that the central host now has puppetd running, and is being configured through the system itself! While some people may prefer to configure their all-powerful central host through the same configuration management system, this presents a nasty potential failure mode: if the configuration management goes awry and makes the managed nodes inaccessible, it could potentially take itself out too. Shouldn't the puppet tools be distinct from the puppetd system service?

monkeysphere Build-Depends: openssh-server

The Monkeysphere is a framework for managing SSH authentication through the OpenPGP Web-of-Trust (i'm one of the authors). To ensure that the package interacts properly with the OpenSSH implementation, the monkeysphere source ships with a series of test suites that exercise both sshd and ssh.This means that anyone trying to build the monkeysphere package must pull in openssh-server to satisfy the build-depends, thereby inadvertently starting up a potentially powerful network service on their build machine and maybe exposing it to remote access that they didn't intend. Wouldn't it be better if the /usr/sbin/sshd tool was available without starting up the ssh system service?

Good ExamplesHere are some examples of debian packaging that already understand and implement this distinction in some way:

apache2.2-bin is distinct from apache2-mpm-foo

Debian's apache packaging recently transitioned to split the apache tool into a separate package (apache2.2-bin) from the packages that provide an apache system service (apache2-mpm-foo). So apache can now be run by a regular user, for example as part of gnome-user-share.

git-core is distinct from git-daemon-run

git-core provides the git daemon subcommand, which is a tool capable of providing network access to a git repo. However, it does not set up a system service by default. The git-daemon-run package provides a way for an admin to quickly set up a "typical" system service to offer networked git access.

vblade is distinct from vblade-persist

vblade offers a simple, powerful utility to export a single file or block device as an AoE device. vblade-persist (disclosure: i wrote vblade-persist) provides a system service to configure exported devices, supervise them, keep them in service across system reboots, etc.

Tools are not Services: a ProposalLet's consider an existing foo package which currently provides:

• the tool itself (say, /usr/sbin/foo) and
• sets up a system service running foo, including stock foo daemon configuration in /etc/foo/*, startup scripts at /etc/init.d/foo, service configs at /etc/default/foo, etc.

I suggest that this should be split into two packages. foo-bin would contain the tool itself, and foo (which Depends: foo-bin) would contain the service configuration information, postinst scripts to set it up and start it, etc. This would mean that every instance of apt-get install foo in someone's notes would retain identical semantics to the past, but packages which need the tool (but not the service) can now depend on foo-bin instead, leaving the system with fewer resources consumed, and fewer unmanaged services running. For brand new packages (which don't have to account for a legacy of documentation which says apt-get install foo), i prefer the naming convention that the foo package includes the tool itself (after all, it does provide /usr/sbin/foo), and foo-service sets up a standard system service configured and running (and depends on foo). Side Effects This proposal would fix the problems noted above, but it would also offer some nice additional benefits. For example, it would make it easier to introduce an alternate system initialization process by creating alternate service packages (while leaving the tool package alone). Things like runit-services could become actual contenders for managing the running services, without colliding with the "stock" services installed by the bundled tool+service package. The ability to create alternate service packages would also mean that maintainers who prefer radically different configuration defaults could offer their service instantiations as distinct packages. For example, one system-wide foo daemon (foo-service) versus a separate instance of the foo daemon per user (foo-peruser-service) or triggering a daemon via inetd (foo-inetd-service). One negative side effect is that it adds some level of increased load on the package maintainers if the service package and the tool package both come from the same source package, then some work needs to be done to figure out how to split them. If the tool and service packages have separate sources (like vblade and vblade-persist) then some coordinating footwork needs to be done between the two packages when any incompatible changes happen. Questions? Disagreements? Next Steps? Do you disagree with this proposal? If so, why? Unfortunately (to my mind), Debian has a long history of packages which conflate tools with services. Policy section 9.3.2 can even be read as deliberately blurring the line (though i'd argue that a cautious reading suggests that my proposal is not in opposition to policy):

Packages that include daemons for system services should place scripts in /etc/init.d to start or stop services at boot time or during a change of runlevel.

I feel like this particular conflict must have been hashed out before at some level are there links to definitive discussion that i'm missing? Is there any reason we shouldn't push in general for this kind of distinction in debian daemon packages? Tags: daemons, packaging, policy

Now there is yet another package with a weird name in Debian - douf00 - a lightweight, slim and straight forward presentation tool. Written by some guys from a hacking and security team, which is mainly based in Vienna - guess that explains the l33t name (if you're scared now, yes, I've reviewed the code...). At the FrOSCon one of the authors asked me if I'd be willing to package it for Debian.I'm sure you'll love it if you're giving presentations and don't like to mess with Openoffice.org and friends. Below I've cited some parts of the README - make sure you read the linked file if you want to know more, or just install the package and give it a try. Douf00 is a lightweight, slim and straight forward Presentation Tool. It assists novice as well as experienced speakers when giving lectures and business meetings. With its simple presenters Screen that includes current slide - next slide (preview) as well as timers it is designed to assist those of us, that are willing to step up their lectures to the next level. The main reason why we created douf00 is because we (nice name crew) travel to conferences very often. As a matter of fact i always carried my macbook pro to every event. I always loved keynote and how the presenters screen works. Not having to worry about your beamer configuration and presenters screen is really comforting. However since i love to pack small and play big (on it stuff) i really felt we had to do something about it. Given the current situation presenters using linux have to deal with, we decided to simplify it. Of corse as a geek i would love to travel just with my netback saving valuable space in my backpack. In reality i was always caring my heavy macbook pro 15" everywhere just to lecture. Not only is it bad to give a presentation standing behind your machine because your presenters remote doesn't work. It is also highly annoying to always have to fumble around with your graphics card. Given the large verity of tools for creating your slides. (i.e.: powerpoint, open office impress, apple's keynote, latech-beamer,....) We felt there was no room for yet another slide creation tool. Nonetheless i noticed my desire to jump onstage, give a really good presentation without the usual worries. To be honest i use Keynote from apple on my home computer as i find it the most comforting tool to create slides. But now that we have created doufoo i save up on baggage space, and i finally managed to get rid of all the worries beforehand. But what is doufoo? - The perfect choice for every presenter that has more important stuff todo than care about beamer configuration more than once. Its fast, simple and lightweight. We think that caring less about your gear and focusing on the presentation itself is the best way to keep your audience enthusiastic and entertained.

[So much for at-least-one-post-a-week : I'm now back online, courtesy of a new router]. In the this week is another announcement from the CoRoT group: CoRoT-7 is the first star (outside our own) known to host two Super-Earths. Two planets of 1.6 and (max.) 21 times Earths' mass, and the smaller one at least is rocky (the outer planet has not been seen in transit, so we can't tell yet). Its not the first Earth-sized planets to be discovered: as my supervisor points out, <http:>PSR 1257+12 holds that record, as well as the one for the first exoplanet to be found. As usual, Exoplanet.eu gives the details: CoRoT-7b has a flurry of papers to its credit already. CoRoT-7b transits its star and is the first Super-Earth with a clear radius measurement. Together with a mass calculated from a radial-velocity measurement by the HARPS spectrometer, they calculate a density of 5.5 g/cm3, and so its shown to be a rocky world. CsoRoT is a hot A-class star, and CoRoT-7b orbits only 2.5 million km from it. Because of this, it is unlikely that it has retained an atmosphere (it'll be way too hot), and is expected to be tidally locked: with one face permanently pointing at the star, one in permanent darkness, the two sides will be both too hot and too cold for life. (Is there a Goldilocks, twilight zone? unlikely: the tidal forces will be too high, and with no atmosphere to moderate matters, there can be little hope for life). Still, while it may not support life, it could help answer important questions, such as how many terrestrial planets are there out there. Most of the planets discovered to date have been gas giants, which are easier to discover. But are there lots of rocky planets hiding between the giants ? One way of answering this might involve the disks from which the planets evolved. One trend we've seen is that (gas giant) planets are more likely to be found around stars with high metallicity (astronomers are strange in calling all elements heavier than helium a metal). But our Sun is strange in being comparatively depleted in 'refractory' elements such as iron or silicon. Is this important for the story of terrestrial planets? Melenendez et al. have a paper on arxiv.org on this topic. They find that most stars similar to our sun found not to have giant planets have elemental abundances similar to our Sun: perhaps they have undiscovered rocky planets instead ? Tags astronomy, exoplanets, planet-of-the-week

Two organizations I care deeply about are having elections this month. The first is the Wikimedia Foundation who is electing three community representatives to their board of directors. The second is Ubuntu who will soon be electing a new Community Council. The Wikimedia Foundation is perhaps the most important organization working on issues related to free culture. Wikimedia elections are currently ongoing and will close on August 10th. Editors who have more than 600 edits to their name across all Wikimedia wikis and 50 edits in 2009 made before July 1st are eligible to vote. The vast majority of eligible contributors to Wikimedia projects have not voted in previous elections. Ubuntu will be electing all members in a new --- larger --- Community Council. I have been a member of the council since it was created and I will be standing for election once again --- the last time I plan to do so. Work in setting up the election is being finalized and all Ubuntu Members will be able to vote in the election. Both Wikimedia and Ubuntu are struggling to find the right relationship between the communities who produce most of the value at the heart of their projects and the organizations and leadership structures that try to support and, from time to time, direct it. Ubuntu and Wikimedia are very different. What's at state at these elections is different too. But both elections are are extremely important and at pivotal times in their communities' growth. Both elections will have an important impact on the process of creating new organizational forms. My message in regards to both elections is also the same: If you are eligible to vote, please do. No governance system I've seen has as healthy and close relationship to the community it serves as it should. Wikimedia and Ubuntu are not exceptions. The result is a strange, and unhealthy, relationship between governance systems and the organizations at the heart of our community-driven projects and the communities themselves. We can all do better. There are not enough opportunities for community members to help push this balance in a better direction. These elections are one. I urge everyone who can to vote and to become involved.

Over the last few years, I've been pretty immersed in the world of wikis. A wiki is like a collaborative web site; it lets several people view and edit a web page and link to other pages, maybe new pages that they themselves create. Wikis let tech-savvy people all over the world collaborate on a single web page, a single document. That's a pretty powerful idea, really. The obvious thing to do once an idea like wikis appears is to attempt to cram all of humanity's knowledge into it. Well that's just crazy enough to work. Wikipedia is the gold standard of wikis. It's a collaborative encyclopedia with lots and lots of articles. As the gold standard, it influences people's thinking about wikis in ways that might not be quite true. If Wikipedia were the only wiki you'd ever heard of, you might think that wikis are used to create encyclopedias. If that were true, you'd really only need one of them. If Wikipedia were the only wiki you'd ever heard of, you might be under the false impression that all wikis everywhere are editable by anyone. So if you create a wiki page, it's no longer yours. Other people can scribble on it. Well, some kinds of documents perhaps just aren't suitable for being edited by anyone, and the good news is that you can build a wiki for you and your classmates, or you and your coworkers, and you can decide who gets to edit what. I want to think about a few distinct uses for wikis in different contexts. The most common use is of course to create a knowledge repository; a collection of information written by people who know what they're talking about. That's what Wikipedia is, mostly. That's what the Haskell Prime wiki (which I run) is. And the excellent wiki about the game of Go, Sensei's Library, is also a knowledge repository. But there are other uses for wikis that are pretty cool. I often have to work on proposals with my coworkers. One (horrible) way to do this is to write the first draft in MS Word, email it to everyone, who might edit it and email it back. Of course, only one person can actually edit it at a time, otherwise I have to figure out what everyone did and merge their changes by hand. That's what we, in the computer industry, call teh suck. A better way to do this is to use a version control system, which lets multiple people work on a document at one time. Of course, if you're editing evil document formats like Word, still only one person can edit it at a time, but at least you cut out email, so things are a bit better organized. But wikis are actually a really great way to work on a document with another person. Two or more people can each collaborate on the document using the wiki software over the web. What becomes of the document at that point depends on its ultimate use. Maybe you have to convert it to MS Word or PDF at the end. That's a bit of a bummer. So there are two uses of wikis that I've mentioned: building a knowledge repository, and collaborative writing. Google started a service for collaborating on documents called Google Docs & Spreadsheets, which is probably useful for collaborative writing, but not knowledge repositories. They also acquired a wiki startup a few days ago, so they are clearly trying to get all over this space. There's another very interesting use of wikis that I'll call situation awareness. You can get a really vivid picture of this use of wikis if you take a look at the wikipedia article on the July 2005 London bombings. By the way, I didn't discover this excellent example of situation awareness using wikis. I saw it in the slides for a talk, but I don't know the original source at this point. If you go to that link, you'll see an excellent and up-to-date encyclopedia article about the London bombings. It's not a news article. Go to Google News and try to find a news article from a major news source which is this detailed, up-to-date, and well organized. Most articles you'll find are a snapshot, taken at a particular point in time (when the article was published) that most likely contains an account of the latest developments, and usually brief background in case you missed previous stories. The wikipedia article isn't a snapshot, it's an article that explains things in a logical order, which isn't necessarily chronological. Each wikipedia article contains a link to its own history. You can look back over time and see the article evolve. If you look at the early edit history of that London bombing article, you'll notice something fascinating. The article was created shortly after the bombing, with an account of the information that was known at that time. Within an hour, many people had edited the article, adding information they know about. If you were refreshing that wikipedia entry over & over (or if you were subscribed to it using RSS), you'd see not an account of just the latest developments, but an evolving, logical understanding of the attacks. The article started evolving right away, and it continues to evolve to this moment. That's what I'm calling situation awareness. Don't get me wrong, the mainstream news media is quite good at situation awareness. They do it a bit differently, and their articles or TV stories don't become integrated into a knowledge repository like wikipedia. I've enumerated three somewhat overlapping uses for wikis that I think are pretty interesting to examine:

Knowledge Repositories, which are a lot like encyclopedias,

Collaborative writing, which is a bit like a more elegant version of emailing word documents, and

Situation awareness, which isn't so different from the news; it draws from the news as a source, it can also be authored by the eye-witnesses themselves, and each story becomes an integral part of a knowledge repository. I don't claim that these are an exhaustive, nor are they a partition: Wikis are also used for bug tracking and as web discussion forums, for instance. What other categories can you think of? Email me if you have ideas.

---

*whew* I did a bunch of things yesterday. We took our kinderg rtner to her first Friday at her new school (and were about 10 minutes tardy. oops). We then took our toddler to a nearby playground with swings and slides and let her expend some energy. After she had been sufficiently exercised, we walked back home, stopping at a coffee shop on the way. The baristo (you call male baristas baristos, right? :) ) recognized my MC Frontalot shirt and asked whether I had caught him the previous weekend at PAX. Unfortunately, I have not attended PAX since 2006, but I *did* purchase the tee directly from The Front himself ;) When we got home, I worked a bit on an English Language parser implementation and then went to the University of Washington to meet with Emily Bender about getting in to the Professional Master s program in Computational Linguistics. It all looks good, and I even got the good news that the GRE is no longer required! After the meeting, I headed home and poked at the parser for a little while longer. I then picked Scarlet up from after-school care and brought her home. I then hopped in the car and drove toward Bellevue to meet up with Monty while he s in town. I over-estimated the amount of time traffic would steal on my way to Bellevue, and had an extra hour to blow. So I dropped by building 41 and shot the IronPython bull with Dino. It turns out he s got an android phone, too. I told him it was possible to put a debian chroot on it and that he should even be able to apt-get install ironpython to his phone soon ;) We talked briefly about the CodePlex Foundation and Sam Ramji s departure from The Evil Empire. Dino seems skeptical about the project. I don t have enough information to have much of an opinion. However, it sounds like some folks I trust are involved, so I m hopeful. I left MS just in time to make it to the wrong address at the specified time. My phone had just enough juice to call Monty to get the right address and then use the navigation system to find my way there. I wasn t able to make reservations at the place we intended to go for dinner until 8:15, so we went to the Barnes & Noble for a bit. They only had one NLP book in stock and the examples are all in Python. I should learn that language one of these days As we were leaving the Pacific Place, Monty mentioned to me that he is on the advisory board for the CodePlex Foundation, and that they have been responsive enough to his input that they changed the Mission statement, at his recommendation, just one day before the Foundation was publicized. He feels that this is a very good direction for Microsoft to be heading. My brother Chris was kind enough to watch the kids while we went out to dinner. Quick note: he recently graduated from UW with a BA in Electrical Engineering and is looking for work using his acquired knowledge, in case anyone needs one of those ;) We met up with my wife, Hannah and our friends, Mike & Cynthia at our place. Monty graciously avoided mentioning the terrible state in which our apartment has recently found itself. The kids were super cute and polite and said hi/bye. Over dinner we discussed building an android app (Monty has one, too ;) ) to automate the process of creating bounties for apps and getting folks to implement them. We also talked about MySQL and MariaDB, of course. Hannah and I recalled my time working for MySQL, Inc. on the MaxDB project and some subtle cultural differences we noticed while traveling. It was interesting getting the inside scoop about the Sun acquisition and some of the recent goings-on in the MySQL/Sun/Oracle world. I wasn t aware, for instance, that the EU is balking on the merger because of monopoly concerns.

It's been quite some time (almost two years) since my previous "release" of debtree, but now version 0.7.3 is available. And it still generates very nice graphs :-) The changes are relatively minor: a few nice fixes for corner cases that were not handled correctly, and an update of the default lists of "skip" and "end" packages which help to limit the size of graphs for a fair number of packages I tried (including konqueror and openoffice.org).

---

Reason to revisit debtree was a recent nice mail from a debtree user, but also the current discussions about udev and the FHS. I'm on the side of "let's please keep /usr mountable separately". Mostly because I like a (small) encrypted root with a separate (large) unencrypted /usr'. I'm also increasingly unhappy with the default size of Debian's desktop installs, especially now that it looks as if Squeeze will see installation of Recommends by default by tasksel (and thus Debian Installer). For comparison, the size of a default Gnome desktop install for Etch was 1360MB; for Lenny it is 1830MB; for Squeeze it looks like it will be well over 3000MB! Remember that for Sarge we installed both Gnome and KDE from CD1 with both together taking 1390MB? Sure, some of that is real functionality, but a lot is also (IMO) redundant visual effects that only serve to slow the desktop down and junk needed to do stuff automagically. And a heck of a lot is duplicated functionality. One of the main reasons I switched to Linux was because it gave me back control over my systems, but with KDE4 and pervasive stuff like hal and all the various "kits" Linux is on a fast track that's giving priority to flashiness over real functionality and eroding that control. Here's a fairly default dependency graph for hal (click for full image). Looks reasonable, right? But that's only because most major dependencies, such as dbus, policykit and pm-utils have been pruned. Here's a complete graph, with only libc6 omitted (full image is 1.5MB). Truly a tangled web. Scary. One can also look at it from the other side. Today I upgraded my sid chroot and found I suddenly needed to install libavahi-client3, libavahi-common3, libavahi-common-data and libdbus-1-3. Why? Reason turned out to be libcups2, so I checked if I really needed that. And here's why I do. Most of these dependencies of libgtk2.0-0 I can understand, but isn't gtk supposed to be a graphical toolkit library? Couldn't printing support be implemented in some more specialized Gnome printing toolkit library? But I'm probably missing something. Anyway, now that I have a bit more perl experience through my recent work on debmirror, maybe I should finally port debtree from shell script to perl...

---

See the debtree home page for a full overview of how to read the graphs, but here's a quick intro. Purple arrows are Pre-Depends, blue are Depends and black are Recommends; green connections show Provides. The green packages are currently installed in my sid chroot, while the white ones are not. The diamonds show where the graph has been pruned: dependencies for these packages are not shown.

I think this was my first contribution to the Debian operating system:

grace  (5.0.3-1) unstable; urgency=low

* added Replaces: xmgr, closes #35010, #42115, the only conflicting
  file is the grconvert binary which is identical in both packages
* changed editor to sensible-editor
* redebianized using dh_make and debhelper
* new upstream, closes: #41146, #27308, #28944, #29705, #38214
* NMU

-- Torsten Werner   Mon, 6 Sep 1999 22:19:04 +0200

That was 10 years ago. By the way the package is still in the distribution but orphaned now. Please note that our bug numbers had 5 digits only!

I don't want to count the packages that I am involved today but I am concentrating on Java related stuff. We got OpenJDK into Lenny and since DebConf9 it is our default JDK. Thanks to the recent work of Ludovic Claude we can use Maven for building official Debian packages. That will hopefully make packaging easier for many packages where upstream is using Maven as the build tool. I recommend joining the debian-java list if you are interested in those topics.

And I am a member of the FTP team today which is a great honour. It is quite
some extra work but we are now closer to the 1 week waiting time for NEW
processing (on average). Processing NEW is another place where I could see that
times are changing: this week I accepted a package (c) Microsoft Corporation
into Debian main what seemed to be impossible ten years ago.

Since Petter summarized what's happening to Debian's boot process, I'm pondering "aptitude install upstart". I've already been among the pionneers for parallel and dependency-based boot stuff....and never had any problem with it. So, really, moving one step forward is tempting. Anyone to discourage me to do so? :-)

Something that has been annoying me recently with my bank has been that their website tells me that they will never ask for my password over the phone. And then their call centre asks me for my password. Over the phone. Of course the call centre doesn't mean my website password - they mean the special 'ultra-secure 5ekr1t code phrase', but they don't have a good, universally understood word to use for that. Hopefully they'll work one out, but they appear to have got the message anyway. This got me to thinking about how these phrases are used, and how insecure they are in reality. After all when I store a website password I go to significant lengths to ensure that the same password is not represented by the same string of characters in my database. How vulnerable are our secrets in the databases of organisations we do business with?
<!--break--> Simple Password Storage Surprisingly often people do store passwords in databases in plain text, so that should their website get hacked someone would quite possibly be able to download the whole password database. Please feel free to name and shame these organisations in the comments below. My own pet hate in this regard is the 'Mailman' mailing list software: by default on 'mailman day' - the first day of each month - it sends me my password. In plain text. Of course many developers recognise this flaw, and work around it by using a one-way hash to obscure the password. Usually they choose md5 for their hashing algorithm though, and they often fail to use a 'salt' to randomise the plaintext prior to hashing. This means that even though a password might seem obscure like 'Supercalifragilisticexpialidocious!', and no doubt it will hash to something that seems obscure like 'a7290d426b6a1764af6fd7fba5db214e', but you can often find it straighforwardly by looking it up through one of the friendly reverse hash lookup websites. There's even a Digest::MD5::Reverse perl module on CPAN to interface to a bunch of these in a more automated way! Oh dear. One way to go beyond this is using a 1-way hashing algorithm, with a random salt included into the plaintext before the hashing, so that if (god forbid) two users had 'password' for their password I might see two rows in my database like:

davical=# select username, password from usr;
  username                       password                          
-------------+------------------------------------------------
 user1          SSHA qCctCH5dirYCf29uxJiE68LvmLRDdnBkbldiWlE=
 user2          SSHA y8yOzjoh9fSkVwLaXGoVtWdiIYxmU2FCb2dOZXc=
(2 rows)

When the user wants to log in I apply the same transformation to their incoming password (appending the same salt) and compare against my stored hash. If they match then it must be the same password they used previously. Storing passwords in this way secures them from casual, or even reasonably determined access, although naturally they can still be logged at the beginning and end of the communication - or even in the middle, if we didn't encrypt that bit! The PHP function I use to salt and hash the password is as follows:

/**
* Make a salted SHA1 string, given a string and (possibly) a salt.  PHP5 only (although it
* could be made to work on PHP4 (@see http://www.openldap.org/faq/data/cache/347.html). The
* algorithm used here is compatible with OpenLDAP so passwords generated through this
* function should be able to be migrated to OpenLDAP.
*
* If no salt is supplied we will generate a random one.
*
* @param string $instr The string to be salted and SHA1'd
* @param string $salt Some salt to sprinkle into the string to be SHA1'd so we don't
*                     get the same PW always hashing to the same value.
* @return string  SSHA  followed by a base64 encoded SHA1 of the salted string.
*/
function session_salted_sha1( $instr, $salt = "" )  
  if ( $salt == "" )  
    $salt = substr( base64_encode(sha1(rand(100000,9999999),true))), 2, 9);
   
  return ( sprintf(" SSHA %s", base64_encode(sha1($instr.$salt, true) . $salt)));
 

What about Secret Code Phrases? The problem with these secret code phrases, apart from all of the forgetability and guessability problems that have repeatedly been identified elsewhere, is that they are much less likely to be stored in a one-way hash. Are you going to ask your call-centre staff to type in the customer's secret code phrase? Didn't think so. And if you did it's going to add pronouncability issues to the whole mix. So this means that those organisations who have our secret code phrases in their database will, in all likelihood, have them stored directly as plain text, displaying them to the random call-centre staffer along with all of our other account details, and especially making them vulnerable to accidental disclosure. Disclosure of a sort that doesn't necessarily involve knowing they have been disclosed. Proliferation of Use These things provide the appearance of security - 'Security Theatre' as Bruce Schneier terms it - and because of that they're taken up in a kind of a cargo cult of security: "if the banks do it that way it must be a really good form of security". This makes the problem much worse, because now I have to remember a secret code phrases not only for banks, but for ISPs, phone companies, online auction websites, and so on. How many mother's maiden names, favourite teachers and friend's phone numbers do I have? I'm sure I'm at well up whatever curve it is that measures the number of passwords a person has, because five years ago I had so many I started to store them all in an encrypted database - protected by a yet another password, of course. Now in order to get my story straight I have to store my 'secret code phrases' in there too. If I didn't store my secret code phrases in that database, I'd obviously be re-using them everywhere, from a very small set - perhaps the same one that everyone around me overhears, every time I have to ring my bank to authorise another payment from my account. Because the proliferation of use is not just the breadth of wannabe thespians hoping to climb on the stage of this latest play, but the way they want to use it all the time, too. In fact the only conversation I've had recently with my bank where they didn't want it was when they rang me. Obviously I was the only person who could answer a phone in my house, right? It isn't just security theatre: I think we can see that this analogy belongs much deeper into the sub-genre of 'Security Farce'. Is there a solution? I don't have any easy answers - other than the ones to my security questions, of course - but some improvements are possible. Other banks have quizzed me about stuff like recent expenditure or credit card limits from time to time, but I've usually passed those tests by reading my last bank statement - or failed them by not having it to hand! I don't really think that the answers can lie in that direction because the information is only quite loosely tied to my identity. For some parts of the call-centre handling of secret code phrases there are changes that could make them more secure, but in the fairly short term these organisations need to find a different way to perform these out of band identity checks. For the actual storage of the code phrases it would be a marginal improvement if the database did not contain the actual phrase. Perhaps it could be encrypted with some application-known key, so that it can be unencrypted when it needs to be displayed, but never stored in the clear. Of course there's still the problem with that key... Verification of the secret code phrase could be done by someone not involved in the transaction, so that the call could be temporarily passed into a 'verification stream' where a different person performed the verification step without the context of the account details or enquiry. Though this sort of complexity seems unlikely with call centres seemingly being outsourced to the cheapest supplier. One thing does seem likely to become increasingly true: there is less and less private data in our lives, and every time we share one of these little nuggets with our bank, or our electricity company, or our on-line associates-we-call-friends, is one more chance that it escapes into the hands of the foaming-at-the-mouth-terrorist-cracker-communist-nazi-right-wing-religious-fruitcake hordes. The highest bar for personal verification which any of my banks currently sets for me is a random choice from a set of personally entered questions, with a set of personally entered answers, for which I have to enter two randomly selected characters using my mouse. That's not bad for safe verification, and I'd have to be really impressed with their security if that was stored in the database by a passphrase-protected encryption key. With that bank I don't know what they do over the phone - I assume they've concluded they can trust my logged on persona enough that I can do what I want online, consequently I haven't had to call them and share those secrets with everyone in earshot. Maybe paranoid freaks like me will go back to a chequebook and close down on-line access to their bank accounts entirely when they find themselves having to supply a skin scraping in order to authorise their next $500 payment. "Please insert your finger in the AccuYou(tm) BloodSucker(tm) to proceed with this payment" - well, I guess it might cut down my spending! In any case, biometrics need to be understood before they can be used effectively and appropriately - and remotely over the phone is probably not one of the ways that they can be trusted to work.

Voice-controlled waypoints I have it in my TODO list to implement taking waypoints when pressing the headset button of the openmoko, but that is not done yet. In the meantime, I did some experiments with audio mapping, and since I did not manage to enter waypoints while recording them, I was looking for a way to make use of them anyway. Enter findvoice:

$ ./findvoice  --help
Usage: findvoice [options] wavfile
Find the times in the wav file when there is clear voice among the noise
Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -v, --verbose         verbose mode
  -p NUM, --percentile=NUM
            percentile to use to discriminate noise from voice
            (default: 90)
  -t, --timestamps      print timestamps instead of human readable information

You give it a wav file, and it will output a list of timestamps corresponding to where it things that you were talking clearly and near the FreeRunner / voice recorder instead of leaving the recorder dangling to pick up background noise. It does not use sound statistics because I have no background whatsoever in audio processing, but it basically finds those parts of the audio file where the variance of the samples is above a given percentile: the higher the percentile, the less timestamps you get; the lower the percentile, the more likely it is that it picks a period of louder noise. For example, you can automatically extract waypoints out of an audio file by using it together with gpxinterpolate:

./findvoice -t today.wav   ./gpxinterpolate today.gpx > today-waypoints.gpx

On the timestamps that it outputs, it will add the modification time of the .wav. This means that if you touch -d "@timestamp" the .wav file with the date you started recording, then you can get decently accurate waypoints provided your system clock is decently synchronized. For example:

START= date +%s 
arecord file.wav
^C
touch -d "@$START" file.wav
./findvoice -t file.wav   ./gpxinterpolate today.gpx > today-waypoints.gpx

In record-audio-on-freerunner I even set the .wav mtime using the GPS time at the start of the recording.

Today I attended RubyCamp (schedule available), organized by my friends at Instituto de F sica, at my University. I presented three plugins I made and regularly use. The organizers followed a short, more informal scheme than most conferences I am used to Talk slots range between 5 and 15 minutes, so we attended a whole day of semi-lightning talks. Of course, many people have run late, and although there was quite a bit of free space in the schedule, it has been practically non-stop I was thinking on also giving a talk on encodings (as many people really still don't understand what is UTF-8, what is Latin1, why all that mess People, please learn at least The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)), but the schedule is full by now. Interesting and successful experiment, I'd say. The talks are being taped, and the organizers say they will be soon made available online. There are many cultural details to note here. First of all, yes, the Rails Fanboys _are_ a cult/sect. I think we have ~90% of MacBooks... I still fail to understand why a coder feels at ease on MacOS. I deeply despise it! Also, most of this community have bitten the Twitter plague. This is also a community very much into businessspeak, speaking a word in English for each two words in Spanish (I try to be consistent, not mixing languages at least). Some conferences have been quite business- and enterpeneur-oriented. Although I should not complain too much about this, as it is an important aspect for many people But I still don't feel at ease having talks on how to run a business if we were asked for presentations on technical aspects! Anyway - I was quite happy to be here. This is the first real technical, code-oriented conference I have attended in a long time in Mexico. And we need more like this! We have too many entry-level, evangelization-oriented conferences, but very few like this one. [update]: Group photo!

An old friend is ranting about localization. I first considered not really reaction to Joey's post, mostly because he has very true points...and particularly when he has about the same reaction than mine about a recent post in ou mailing lists. So, I will stick to react to one part of Joey's post...the one I consider really inaccurate. "everyone uses the software in English anyway" Really, Joey, really, that's completely untrue. Of course, I do live in a country that very well known (or at least that has the reputation) for being deeply ignorant in English language. But, really....*all* computers I see around me, both in my personal life *and in my professionnal life* are nearly completely localized. It's probably now well know that I spend most of my professionnal time....providing desktop systems (or managing the people who provide them) to users in ONERA, "the French Aerospace Lab". That means about 2000 users in a scientific environment where the use of English is much more developed than the average environment. Most of these users speak a fairly decent English and are perfectly able to understand that language in some way. About 80% of these systems are Windows-based and 20% are Linux-based. Probably more than 95% of them are localized in French. Even on Windows systems, we mostly integrate free software. And for *each* of them, the most important feature we need, besides software being free, is it to be localized in French. I bet our friends from Extremadura can say the same. The success of Linux-based free systems there also happened because the software is available in Spanish. And, of course, there are tons of similar stories and counter-examples (M nchen, Bhutan, Venezuela, etc.). So, really, please don't spread out that idea. Oh, yes, geeks *do* use their software in English (and, even here, I need to be convinced if I remember well what I saw at conferences in Germany or Spain, for instance). And, yes, it's very well widespread among geeks that using software in English is "better" or "easier". Which I indeed always fight and will fight until I die..:-). We certainly agree that these l10n needs put a high load on development and sometimes slow it down. and, for sure, we sometimes need to make a balance and I really thank you for giving me the credit to try doing that balance. Anyway, all this (including your rant) had to be said and it's interesting to have that small discussion (I just should not start writing down a blog entry at 23:00...:-)) See you soon, hopefully, Joey !

Whilst working on update-manager I have been wondering whether I should use gettext for localizing debug output and error messages sent to stderr.
As for debug output itself I basically do not see the need for providing a localized version for each and every message sent to stderr, but as far as error messages are concerned I am uncertain. The point is that update-manager (apart from its experimental text interface) is usually not launched from a terminal at all and so most users won t even see these messages ever. Also, I believe that every developer s English skills are good enough so that he or she is able to understand simple messages.
Error messages however might be useful to all users when they experience a problem with the software, but localizing those could make handling bug reports a bit harder, possibly having to translate the error message back to English before being able to see what has gone wrong. So basically I am asking you: What do you think? Is it worth localizing these messages? What is your experience with localized or non-localized error and debug messages? I would be glad if I could get some input from you, either as a comment to this article, via email to debian(dot)sp(dot)or(dot)at or through the update-manager-devel mailing list.

TurboGears 2 packaging - status update Brief status update about my TurboGears 2 packaging efforts, after some evening work of today:

• repoze.what- pylons,quickstart,plugins.sql (ITP): in experimental/NEW
• WebFlash (ITP): in experimental/NEW
• sqlalchemy_migrate: false positive of yesterday's quick estimate, it is already packaged in Debian as python-migrate

Here is my ex-virgin /usr/local/lib/python2.5/site-packages/ as it looks now:

    zack@usha:~$ ls /usr/local/lib/python2.5/site-packages/ -1
    AddOns-0.6-py2.5.egg
    BytecodeAssembler-0.3-py2.5.egg
    Catwalk-2.0.2-py2.5.egg
    easy-install.pth
    Extremes-1.1-py2.5.egg
    PEAK_Rules-0.5a1.dev_r2582-py2.5.egg
    prioritized_methods-0.2.1-py2.5.egg
    sprox-0.5.5-py2.5.egg
    SymbolType-1.0-py2.5.egg
    tg.devtools-2.0-py2.5.egg
    tgext.admin-0.2.4-py2.5.egg
    tgext.crud-0.2.4-py2.5.egg
    TurboGears2-2.0-py2.5.egg
    tw.forms-0.9.3-py2.5.egg
    zope.sqlalchemy-0.4-py2.5.egg

At this point, it should be already possible to install the 2 main eggs composing TurboGears 2 (tg.devtools and TurboGears2 itself) relying only on Debian-package dependencies. ... modulo some "details":

• the two eggs themselves aren't properly packaged yet
• I haven't tested this
• you will be missing a lot of stuff that is part of the usual TG workflow, such as Catwalk (that requires a good deal of dependencies of its own)

This evening's motto: ci siamo quasi. Next week I'll be pretty busy, but I guess some night work (or some of you, fellow readers ) will come to the rescue.

TurboGears 2 in debian ... soon ! After a long incubation, a few days ago TurboGears 2 has been released. Historically, I've been preferring TurboGears over Django for being closer to the open source philosophy of reusing existing components. Since the long 2.0 release was marking a gap with Django, I was eager to test the 2.0, and I was delighted to find it in Debian. Unfortunately, it doesn't seem to be in good shape yet. In particular it lacks several dependencies before it can even be used to quickstart a project, in spite of its presence in unstable. To give an idea of the needed work, after having installed it manually (via easy_install), my virgin /usr/local/lib/python2.5/site-packages/ has been polluted by 26 egg-thingies. I decided to spend some week-end time to start closing the gap, because we cannot lack such an important web framework in Debian (and ... erm, yes, also because I need it ). Here is the current status of what has been ITP-ed / packaged already (by yours truly):

• ToscaWidgets (ITP): in experimental/NEW, packaging resurrected from a never-uploaded version
• repoze.tm2 (ITP): in experimental/NEW
• transaction (no ITP) : being packaged by the Zope team (temporary unofficial package available)
• repoze.who (ITP): in experimental/NEW
• repoze.who- friendlyforms,plugins.sa,testutil (ITP): in experimental/NEW
• repoze.what (ITP): in experimental/NEW
• repoze.what- pylons,quickstart,plugins.sql : just ITP-ed

Where to Considering, foolishly, the last package as being already done, the way to go is still long:

    zack@usha:~$ ls /usr/local/lib/python2.5/site-packages/ grep -v repoze.what
    AddOns-0.6-py2.5.egg
    BytecodeAssembler-0.3-py2.5.egg
    Catwalk-2.0.2-py2.5.egg
    easy-install.pth
    Extremes-1.1-py2.5.egg
    PEAK_Rules-0.5a1.dev_r2582-py2.5.egg
    prioritized_methods-0.2.1-py2.5.egg
    sprox-0.5.5-py2.5.egg
    sqlalchemy_migrate-0.5.2-py2.5.egg
    SymbolType-1.0-py2.5.egg
    tg.devtools-2.0-py2.5.egg
    tgext.admin-0.2.4-py2.5.egg
    tgext.crud-0.2.4-py2.5.egg
    TurboGears2-2.0-py2.5.egg
    tw.forms-0.9.3-py2.5.egg
    WebFlash-0.1a9-py2.5.egg
    zope.sqlalchemy-0.4-py2.5.egg

Possibly, some of them are already in Debian hidden somewhere, but sure there is still work to be done. If you want to help, you are more then welcome. The rules are simple: all packages will be maintained under the umbrella of the Python Modules Team, but you should be willing to take responsibility as the primary maintainer. Please get in touch with if you are interested, as I'm in turn already in touch with some other very kind volunteers (thanks Enrico and Federico2!) to coordinate who-is-doing-what. Preview packages available What has already been packaged, including a temporary workaround for python-transaction, will be available in experimental after NEW processing. The idea is that nothing will go to unstable until TurboGears 2 will be (proven to be) fully functional. In the meantime, packages are available from my personal APT repository (signed by my key), here are the friendly /etc/apt/sources.list :

    deb http://people.debian.org/~zack/debian zack-unstable/
    deb-src http://people.debian.org/~zack/debian zack-unstable/

Versions are tilde-friendly and shouldn't get in your way when the official packages will hit unstable. Packaging multiple-egg / multiple-upstream packages In all this, I've faced an interesting problem with the python-repoze. who,what -plugins packages. They correspond to a handful of plugins, each of which is about 20/30 Kb. I didn't consider appropriate to prepare 5 different packages, due to archive bloat potential. Hence, you get the usual problems of multiple-upstream packages. To counter some of them, as well as some egg-specific packaging annoyances with multiple upstream, I wrote a couple of very simple helpers:

• get-orig-source.mk (live version): a debian/rules snippet which relies on several debian/*.watch files which are fed to uscan to implement (not yet completely) the get-orig-source target described in policy
• setup.py (live version): a multiplexer for sub-directories shipping distutils-enabled Python modules. That way you can use /usr/share/cdbs/1/class/python-distutils.mk as usual, and setup.py will take care of the "multiple upstream" problem at build time

I'm no Python-packaging-guru so, lazyweb, if you spot in this choice something I utterly overlooked, or if you have improvements to suggest, please let me know. Repacking eggs An interesting problem will be faced in trying to integrate the above approach with Python modules that are shipped only as .egg files (i.e., no tarballs, ... yes there are such horrifying things out there). To preserve uniformity, we would need uscan to support --repack ing of .egg files as they were simple .zip archives. Since eggs are .zip archives ... why not?

On Thursday it was my birthday, and I fulfilled a promise I made to myself many years ago. I've always been a car nut, and when I was about 6 years old and had posters of Lamborghinis on my bedroom walls I promised myself that by the time I reached 35 I would have one. That would be long enough in the future that it might happen, but I'd still be young enough to still be able to appreciate it. Well, as it got to the beginning of this year I realised I was running out of time and my bank balance didn't quite look large enough to buy a Lamborghini. So, I cheated a little and I hired one for the day on Thursday. More pictures here. I had one car booked a few weeks ago, but unfortunately the company involved had to let me down as the car developed problems a couple of days before I was due to have it and needed to go away for repairs. I'm not going to complain too much about them, as it wasn't their fault and they did their level best to make things right: offering a different car, a refund or a later replacement hire. But dammit! I wanted the Lamborghini and I wanted it on my birthday! I quickly rang around on Wednesday to find somebody else, and thankfully the folks at Signature Car Hire were very helpful, sorting out a car with just a few hours' notice. Yay! When they guys turned up on Thursday morning to deliver the car to my office, I couldn't stop myself grinning like the 6-year old boy whose promise I was keeping. Here was a gorgeous car, and it was mine for the day!!! I planned a route for the day to take in some nice driving roads and, importantly, some tunnels. I've watched Top Gear on the TV and I read Autocar every week, and they had both said that taking a fast sports car through a tunnel with the roof down was a good thing to do. I now know that's an understatement! As a whole, the car was awesome: the acceleration, the handling, the brakes were all wonderful. But, as I found out on that day, the noise from the engine was just astonishing. I went through several tunnels that day, heading down past Baldock to Hatfield and back, and at full throttle I could not help but laugh out loud at the music coming from the 5-litre V10 orchestra just inches behind me. It's an experience I'm never going to forget. grin. I also gave short rides to quite a few of the people at work; oddly I had lots of friends that day... :-) I'd also planned other things for the day. I bought myself a new camera in advance to make sure I got some nice photos of the day. Then at the end of the day I went out to the pub with a huge group of friends and I was forced to drink lots of beer. It was horrible, horrible I tell you.

While I've only skimmed the available protocol draft and whitepapers, I do have a couple thoughts about wave. Firstly, I can imagine something like this supplanting not just email and instant messaging, but also web forums, blog comments, mailing lists, wikis, bug tracking systems, twitter, and probably various other forms of communication. (But not, apparently, version control systems.) While one might have an attachment to a few of those things, I think most of us would be glad to see most of them collectively die (especially twitter and web forums of course) and be replaced by a data-driven standard, with multiple clients, rather than their general ad-hoc, server-side generated nastiness. (The Project Xanadu guys would be probably be proud.) Secondly, while Google are making all the right noises about open protocols, multiple providers, etc, and probably only plan to lock in about five nines percent of users to wave.google.com before competing implementations arrive, wave is not a fully distributed system. Spec says:

The operational transform used for concurrency control in wave is [currently] based on mutation of a shared object owned by a central master. The wave server where the wave is created (IE that has the first operation) is considered the authoritative source of operations for that wave.

If the master server goes away, it sounds like a wave will become frozen and uneditable. There seems to be no way to change the master server after a wave is created, without ending up with an entirely different wave. (Update: The demo, around minute 70, seems to show a private modification to a wave being made on another server -- so I may be missing a piece of the puzzle.)

The concurrency control protocol will be distributed at some point in the future

How many centralized systems have been successfully converted into distributed systems? In a similar vein, after reading this whitepaper, I'm left unsure about whether it will be possible to respond to (edit) a wave while offline. Maybe, as clients can cache changes, but how well will conflict resolution work? It seems it would have to be done in the client, and there are some disturbing references here to clients throwing their local state away if they get out of sync with the server.