Steve Jobs: “You don t want your phone to be an open platform.” More Steve Jobs: “These are devices that need to work, and you can t do that if you load any software on them. That doesn t mean there s not going to be software to buy that you can load on them coming from us. It doesn t mean we have to write it all, but it means it has to be more of a controlled environment.” It’s widely understood that Microsoft beat Apple in the PC game because Apple was a tightly integrated, closed platform, whereas Microsoft was an open ecosystem where anyone could play (albeit at their own peril). Clearly, the strategy has not adapted well to digital media players. Why? Because people are tired of products that don’t work. Anyone who has ever had to glue a PC back together for a family member knows what I mean. Does “open” have to mean “poorly integrated” though? Clearly, I don’t think so, given how much I’ve invested in Linux over the years and how much I’m investing in the LSB now. And the job the distributions do in the Linux world is nothing short of amazing, if you think about it—taking thousands of moving parts written by thousands of different people and crafting them into what appears, for the most part, to be an orchestrated whole. Personal experience bears this out too—it’s not that the entire Windows Media ecosystem produces bad products (iTunes pales in comparison to Rhapsody), it’s that the integration points are extremely fragile (e.g., I tried Rhapsody To Go with a Plays For Sure device, which was an awful experience). In other words, just like a chain, an open ecosystem is only as strong as its weakest link, because it’s the links where things break. Sure, there are bad products. I’ve owned no less than two MP3 players in the last few years, and they were pretty terrible. It’s the very, very simple things too. For example, I listen to a lot of podcasts. There’s (gasp!) a Podcasts menu on the iPod, whereas on every other device I’ve owned, podcasts are interspersed with the music, making them difficult to find and, adding insult to injury, mixing gobbledygook into my otherwise well organized collection. Furthermore (gasp!), the iPod keeps track of which podcasts I listen to, and in concert with iTunes, automatically deletes the ones I’m done with. Them’s lasers! Ah, iTunes. Despite the fact that anyone in the world can write a podcatcher (it’s just RSS with enclosures, fer godsakes), the only one I’ve ever used that works worth a damn is iTunes. Everything else is just “integrated” with bailing wire—create a Windows Media playlist, drop it in this directory, etc. The situation on Linux is a bit better, but guess what—Rhythmbox, Banshee, etc. didn’t work with either of the open media players I owned, even though the integration point for one of them was the file system. No, the players on Linux support the iPod best because that’s what everyone has, and if I’m going to pay extra for the iPod, aren’t I primarily buying the integrated experience? (On a related note, I’ve never understood why people pay top dollar for Macs and install Linux on them. But I digress..) Once you taste the fruit of the integrated experience, it’s easy to get sucked in all the way. I eventually tired of all the wasted time and money trying to use an open media player and broke down and bought an iPod. Not too long after, I canceled my Rhapsody subscription. Why? Because I couldn’t use it with my iPod. My overall music experience is diminished (I’m a big believer in the celestial jukebox), but I want one music platform, not two. And the iPod is the one with critical mass. Anything else is swimming upstream. Are open platforms doomed then? I think that’s, as they say, throwing the baby out with the bath water. The key is to have open interfaces, and the key to having open interfaces that work is cooperation between the vendors that implement them. Perhaps surprisingly, the iPod is a good example of how it can be done. The biggest reason for my iPod purchase: I can hook it directly into my car stereo instead of using an FM transmitter. Apple isn’t getting into the car business—it’s decided to cooperate with other companies to make this work. And you don’t hear too much about people calling their technophile relatives to come glue their cars back together. (And talk about the importance of not crashing..) The lesson here, it seems to me, is that an ecosystem is never the sum of its parts—it’s either a whole lot more or a whole lot less. Furthermore, it’s the responsibility of the vendors (and community) that make up that ecosystem to ensure the result comes up on the right side of the equation in the final analysis.

I had trouble with tomcat. It started fine, but shutdown took a very long time (enough for eclipse to suggest killing it, for example). Some people suggested I try the tarball from Apache, instead of using the Debian package, but that didn't help either. By not running tomcat from eclipse and waiting a long time for the shutdown, I finally managed to get some actual error messages: This was usually the last message I was seeing:

INFO: Pausing Coyote HTTP/1.1 on http-8080

After some timeout, I now got these messages:

Protocol handler pause failed
java.net.NoRouteToHostException: No route to host

No route to localhost? WTF? I resolved it by putting all hostnames it might try (localhost.localdomain, although I use that nowhere, as well as the FQDN I'm using) into my /etc/hosts and forcing them to point to 127.0.0.1 - and voila, tomcat actually shuts down instead of pausing and then failing to notice it has successfully paused...

Or, Cleanliness is next to godliness The last time when I blogged about Spam fighting Mail Filtering With CRM114 Part 1, I left y'all with visions of non-converging learning, various ingenious ways of working around a unclean corpus, and perhaps a sinking feeling that this whole thing was more fragile than it ought to be. During this eriod of flailing around, trying to get mailtrainer to learn the full corpus correctly, I upgraded to an unpackaged version of crm114. Thanks to the excellent packaging effort by the maintianers, this was dead easy: get the debian sources using apt-get source crm114, download tghe new tarball from crm114 upstream, cp the debian dir over, and just edit the changelog file to reflect the new version. I am currently running my own, statically linked 20061103-Blame Dalkey.src-1. Cleaning the corpus made a major difference to the quality of discrimination. As mentioned earlier, I examined every mail that was initially incorrectly classified during learning. Now, there are two ways this can happen: That the piece of mail was correctly placed in the corpus, but had a feature that was different from those learned before; or that it was wrongly classified by me. When I started the chances were almost equally likely; I have now hopefully eliminated most of the misclassifications. When mailtrainer goes into cycles, retraining on a couple of emails round after round, you almost certainly are trying to train in conflicting ways. Cyclic retraining is almost always a human's error in classification. Some of the errors discovered were not just misclassifications: some where things that were inappropriate mail, but not Spam; for instance there was the whole conversation where someone one subscribed debian-devel to another mailing list, there was the challenge, the subscription notice, the un-subscription, challenge, and notice -- all of which were inappropriate, and interrupted the flow, and contributed to the noise -- but were not really Spam. I had, in a fit of pique, labelled them as Spam; but they were really like any other mailing list subscription conversations, which I certainly want to see for my subscriptions. crm114 did register the conversations as Spam and non-Spam, as requested, but that increased the similarity between Spam and non-Spam features -- and probably decreased the accuracy. I've since decided to train only on Spam, not on inappropriate mails; and let Gnus keep inappropriate mails from my eyes. I've also waffled over time about whether or not to treat newsletters from Dr. Dobbs Journal or USENIX as Spam or not -- now my rule of thumb is that since I signed up for them at some point, they are not Spam -- though I don't feel guilty about letting mailagent squirrel them away mostly out of sight. A few tips about using mailtrainer:

• Make sure you have a clean corpus
• Try and make it so that your have roughly equal numbers of Ham and Spam
• Don't let mailtrainer quit after a set number of rounds. I use a ridiculous repeat count of about 100 -- never expecting to reach anywhere close to that. Instead, I set the streak to a number = number of Spam + number of Ham + 10. This means that mailtrainer does not quit until it has processed every email in the corpus correctly without needing to retrain. Letting mailtrainer quit after it repeated the corpus twice, but before it got a streak of correct classifications left me with a filter with horrible accuracy.
• Make sure you have a clean corpus (yes, really)
• While the ratio of similarities between .css files to the differences between .css files is a fun metric, and I use it as an informal, rough benchmark while training, it is not really correlated to accuracy, so don't get hung up on it, like I was. Usually, the value I get when training from scratch on my corpus is somewhere around 7.5; but over time the ratio degrades (falling to about 5), while the filter accuracy keeps increasing.
• Train only on errors. The good news is that both mailtrainer and mailreaver already do so, so this is easy rule to follow. The reason is that you should only be doing minimal training, in case you have to modify/change the underlying rule in the future. So your filter should be trained to correctly classify the mails you get, but don't shoot for overwhelming scores.
• Use mailreaver. It has the nice behaviour of asking for retraining when it is unsure of some mail, and it caches mail for some time which really helps in training and rebuilding .css files. The reason mailreaver learns faster than mailtrainer is just this feature, I think.
• Stash mails you train on in your corpus, and don't hesitate to re-run mailtrainer over your corpus again after you have been training the unsure emails. When you train single emails, the changed filter may no longer correctly classify some email in the corpus you train with. Running mailtrainer over the corpus adjusts the filter to correctly classify every mail again. I generally retrain about once a week, though I no longer retrain from scratch, unless I change the classification algorithm.

• echo 1 > /proc/sys/vm/block_dump
  
  The above command sets a sysctl to cause the kernel to log all disk writes. Below is a sample of the output from it. Beware that there is a lot of data.
  
  Jan 10 09:05:53 aeon kernel: kjournald(1048): WRITE block XXX152 on dm-6
  Jan 10 09:05:53 aeon kernel: kjournald(1048): WRITE block XXX160 on dm-6
  Jan 10 09:05:53 aeon kernel: kjournald(1048): WRITE block XXX168 on dm-6
  Jan 10 09:05:54 aeon kernel: kpowersave(5671): READ block XXX384 on dm-7
  Jan 10 09:05:54 aeon kernel: kpowersave(5671): READ block XXX400 on dm-7
  Jan 10 09:05:54 aeon kernel: kpowersave(5671): READ block XXX408 on dm-7
  Jan 10 09:05:54 aeon kernel: bash(5803): dirtied inode XXXXXX1943 (block_dump) on proc
  


• Prefixing a bash command with ' ' will prevent a ! operator from running it. For example if you had just entered the command " ls -al /" then "!l" would not repeat it but would instead match the preceeding command that started with a 'l'. On SLES-10 a preceeding space also makes the command not appear in
  the history while on Debian/etch it does (both run Bash 3.1).
  


• LD_PRELOAD=/lib/libmemusage.so ls > /dev/null
  
  The above LD_PRELOAD will cause a dump to stderr of data about all memory allocations performed by the program in question. Below is a sample of the output.
  
  

  Memory usage summary: heap total: 28543, heap peak: 20135, stack peak: 9844
           total calls   total memory   failed calls
   malloc          85          28543              0
  realloc          11              0              0   (in place: 11, dec: 11)
   calloc           0              0              0
     free          21          12107
  Histogram for block sizes:
      0-15             29  30% ==================================================
     16-31              5   5% ========
     32-47             10  10% =================
     48-63             14  14% ========================
     64-79              4   4% ======
     80-95              1   1% =
     96-111            20  20% ==================================
    112-127             2   2% ===
    208-223             1   1% =
    352-367             4   4% ======
    384-399             1   1% =
    480-495             1   1% =
   1536-1551            1   1% =
   4096-4111            1   1% =
   4112-4127            1   1% =
  12800-12815           1   1% =
  

Russel Coker wrote in etbe: some random Linux tips: Prefixing a bash command with ‘ ‘ will prevent a ! operator from running it. For example if you had just entered the command ” ls -al /” then “!l” would not repeat it but would instead match the preceeding command that started with a ‘l’. On SLES-10 a preceeding space also makes the command not appear in the history while on Debian/etch it does (both run Bash 3.1). It’s pretty easy to get the Bash shell to ignore lines starting with a space in its history: Just add

export HISTCONTROL=ignorespace

to your ~/.bashrc. Alternatively, ignoredups ignores duplicates and ignoreboth ignores both. It’s also possible to tell Bash to ignore lines which match a specific pattern. For example,

export HISTIGNORE=" *:~/*"

will ignore any command starting with a space (emulation the ignorespace option to HISTCONTROL) or starting with ~/ (any program/script in or below your directory). Personally, I just use “HISTCONTROL=ignoreboth“.

Justification. Currently there is a huge mess of files and folders that start with a “.” in any users home folder. There is no structure or policy on how applications should choose file and folder names for data that needs to be stored in users home directory. Additionally there is no established consistency between Gnome, KDE and most other applications. Gnome application have part of their configuration information in gconf folder and other part in a gnome subfolder. KDE applications have a complex structure under .kde/. And most other applications either have one file directly in users home folder or have their own dot-folder there. Problems. There are two major problems with current ad-hoc approach:

1. Namespace pollution of users home space. In any desktop system “ls -al” in the home folder would be a very lengthy affair. Moreeven this namespace is folded - KDE and Gnome apps live in their own subfolders.
2. Lack of structure in folders makes it much harder to properly understand their purpose. For example, if a user has misconfigured an application and would like to reset it to default settings, it currently could be hard to figure out how to do that. Especially without losing his data in that application.

Proposal. I propose to implement a policy on where an application can store data and configuration information in users home folder. Such specification would be proposed as an extension for FHS, first implemented in Debian and then promoted for inclusion upstream and in other distributions. Structure. I propose a following structure for a users home folder:

• data/appname/ - each application, that stores user data in application-specific format, will have to store this data into this folder. Only configuration information that is essential to load this data would be permitted into this folder (in hidden dot files). For example, Evolution could store emails and email access information here.
• .conf/appname/ - each application that stores any configuration information whould have to store all of it in this folder. GConf will have to also store applications configuration info here. If this folder would be deleted, application must revert to default configuration, but still be able to fully access user data in data/appname/ or other, generic folders (below). For example, if .conf/evolution/ is erased, then on next start-up Evoution should still be able to access all email that is stored in data/evolution/ without any configuration wizards.
• .tmp/appname/ - each application that would need to create temporary files that survive reboot, would put them here. These folders can be safely deleted at any time with no data or configuration loss. For example, Firefox cache information and Nautilus thumbnail cache would have to be here.
• Desktop/ - the desktop folder, just like it is now
• Media folders - (Documents/, Photos/, Videos/, Music/, …) a suggestion of basic document structure for user to use or ignore. Applications dealing with particular media types would default to use files from such folders, but should also cope with working with any other folders on users choice. For example, a screensaver with slideshow function would by default use photos from Photos/, but will have to have a configuration option for user to change that setting.
• Additionally I would propose having a .mounts/ folder where applications could mount user specific media and other mounts, such as FUSE. For example, FUSE could be used for automatic archive introspection.

Benefits The main benefit for such system for me is twofold:

• SBackup can be configured to exclude .tmp/ and .mounts/ by default thus saving significant space in backups.
• A tool can be made to restore settings and/or data for specific applications in an uniform fashion.

So, any suggestions on the idea and on where this should be best discussed?

Lilypond is a great piece of software, but its documentation is sucky as hell. I've been trying to tweak the output of a particular piece of music which turned out to have just one system on the final page using the default settings, so that this final staff would end up on the first page, too. The documentation on how to do this, however, is less than helpful. If there is a mistake that could've been made by the people who wrote the manual, be sure they made it:

• In the "lilypond-internals" manual, there is mention of the fact that a particular variable needs to be a "moment", but no mention of how to do it; in fact, the default setting that is listed there for that variable is listed as #<Mom 1 4>, but trying to use that will give you a syntax error. It took me 15 minutes of trying, and finally more or less accidentally stumbling upon an actual example somewhere which sets a moment variable, to figure out how to do it; the right syntax is #(ly:make-moment 1 4)
• In the section on horizontal spacing, there is no single mention of the fact that settings on horizontal spacing apply to the score as a whole, and cannot be changed on a per-measure basis, or halfway the score in other ways. A pretty important detail, in my opinion.
• Yet, even if settings on horizontal spacing can only be made on a per-score basis, you still have to modify them by using \override commands somewhere in between your notes. Logic? Not really.
• In the section on horizontal spacing in the 2.10 version of the manual, there is also no reference to the new \newSpacingSection command, which allows one to split the score up in multiple sections in which the horizontal spacing can be changed anyway. Way to go.
• Even after reading and understanding most stuff in both manuals about horizontal spacing, my attempts still resulted in no visible changes on the page, and I have no clue why. There is no message, no way to be sure whether I did the wrong thing, or whether I did do stuff correctly but that it failed because of unrelated reasons.

Really, I would be very happy if the lilypond authors were to just find someone who actually knows how to write decent documentation.

Our brunch yesterday went super-well. We had about 15 people over, which was far too many for our supposedly mellow evening. But it was great -- we got to catch up with a lot of folks I've been missing over the holidays and haven't gotten to see. I hadn't seen Lucinda Catchlove much for a few months, and Heather Kelley and Damian DiFede made an appearance, too. We got to see John Usher, who's off to do weird things to people's ears at the Consumer Electronics Show in Las Vegas. Niko came with friend Johanie, who made some great black-bean brownies; DJ Ghostbeard was still feeling ill from NYE; and Meg showed off her deep purple hockey bruises. It was also nice to have some new friends come, too. I've been wanting to hang out with MC Turgeon, one of the most experienced bloggers in Quebec, founder of me dia and toiles-du-Web, since we met last year at South by Southwest. She and her partner Chris came over with a nice bottle of wine and got to know everyone there. Actually, she knew a lot of the people already -- she and Anne-Marie had met at Mutek, for example. When Patti came in, she said, "Oh, yeah, I know M-C. We do a lot of panels together." Ah, the panel circuit. Altogether a really good time. Probably the only downside is that our oven was really dirty, and while I was making quiches and roasted potatoes I smoked up the entire house. But it cleared pretty quickly, and with the nice weather we're having, I could even have the kitchen door open to clear the smoke. Unheard of! tags: bruncher mcturgeon montreal cooking

Making a point Speaking of unseasonably warm weather, the Green Party of Canada held a picnic and ice cream social on Mount Royal last Saturday, which would have a couple of feet of snow on it in a regular year. The subject of discussion? Climate change, of course. I dunno -- clever move, but do they have to rub it in? tags: green party picnic climate change

Shameless plugs My friend Allegra has started a great new blog, Sucking-stones. She's got a really good voice and it's now in circulation as one of the blogs I read each morning. Also, my friend Anne-Marie Bergeron has her first travel TV show on 33mag tomorrow. She's got a four-part series on her trip to Iceland which they're going to show. It's in French, but her husband Jeff only cusses in English, so that's probably what you'll hear when they're climbing glaciers and lava floes. Did that get everyone? I hope so. You know what they say: You can plug your friends, and you can plug your ears, but you can't plug your friends' ears. Or something. tags: allegra sucking-stones annemarie jeff 33mag

Chix I was just reading Angela Beesley's end-of-year blog post and realized that I'd missed the establishment of WikiChix. It's a new organization for female wiki participants, in the great tradition *Chix tradition of Linux Chix or Debian Women. All the more galling is that two other blogs I regularly read also covered the new group. Joseph Reagle made a note about gendered spaces and the role they play in any movement (like Open Content and wikis). Biella blogged about his bloggage and put some social-science twist that was a little hard for me to follow. In any event, good luck to WikiChix. I find clubs extremely fun and useful and by nature exclusionary, and nothing to get worked up about. Of course, as one of the two founding members of the Maldekstrulaj Vegeteranaj Esperantistaj Linuxamantoj (Left-handed Vegetarian Esperantist Linux-users Club), I'd probably be pretty hypocritical calling anyone else exclusionary. tags: wikichix wiki angela beesley joseph reagle biella coleman clubs

Querying Lbdb from Vim In spite of my attempts for replacing Mutt with Evolution I gave up for several reasons, mainly in my experience Evolution isn't stable enough with huge mailboxes: it crashes often and is painfully slow. Now I'm back at using Mutt, but at least I've learned to use the Evolution address book which is nice, integrated with other Gnome apps (gaim for example), and strictly more powerful than the Mutt address book. I'm using lbdb to query the Evolution address book from Mutt via the external address queries mechanism. But that sucks in various ways:

1. names should be expanded one by one hitting CTRL-T, changing the headers in Mutt one by one (first To:, then Cc:, then ...)
2. I like to edit headers while composing the mail in Vim, but names written there in the various recipient lines (To:, Cc:, ...) are not handled properly. For example if I fill the To: line with To: name surname when exiting Vim mutt will collapse it in a header To: namesurname which can no longer be passed successfully to lbdb

My solution is lbdbq.vim a plugin for Vim which enables expansion of contact names directly withing Vim (normal/insert/visual modes). It supports multiple matches asking interactively the user to choose among them and can be used to replace all contacts in one or more recipient lines at once. It's new, so it's probably bugged. If you sit like me at the intersection Mutt/Vim/Evo-address-book please test it and give me some feedback. As a side note, this is the first time I program in vimscript after the release of Vim 7.0. I have to admit that is really becoming more pleasant than what it used to be. Lists and Dictionaries really (try to) bridge the gap between pre-7.0 vimscript and programming Vim with external languages (python/perl/...). Update #1: lbdbq.vim 0.2 released with an important bug fix for RFC822 fields continuation line and other minor improvements. Update #2: Vimscript #388 provides similar features to lbdbq.vim and is older. Still lbdbq.vim is more feature-complete, have a look at the Discussion page fore more info.

I am deeply missing a real wiki.debian.org/Hardware page. The current one is a joke. It would be nice to have something in the style of the Gentoo hardware page with nice pages which show what works and what doesn't.

I know about the Debian GNU/Linux device driver check page which is a good start, but:

• in most places (installation reports in and out of Debian) you will not see lspci -n output
• there is more to it than lspci (think lsusb, think stuff behind new/strange/uncommon buses - like the sound chips on some Apple ppc based machines)
• you need a running linux system (a live CD can be used for the test), but:
• Debian's live CD is in its infancy (I know, Knoppix should do fine)
• most live CD-s are i386 only (it doesn't matter that much, since amd64 machine should work with i386 live CDs)
  
• most places where you can buy hardware from:
• either they don't have the hardware in stock,
• either want an answer immediately,
• or they don't allow you to access the machine
• (I think) the database is outdated and people are not adding stuff to it
  
• there is no way to make a search based on the product name of the whole system and/or the component

Current situation (bad stuff)

wiki.debian.org/Hardware is a joke. For example, I was expecting to see a Network compatibility list, but instead I see some useless info.

What annoys me the most in this matter are dead links or links to dead/inactive projects.

Information is scattered all over the place (redirects and directories can help), is mixed/ungroupped, or is outdated.

Current situation (good stuff)

The information partly exists. Probably, in time, the wiki will be cleaned up, but until then you have to dig for the information.

There are plenty of sites and pages with information, but, still, you need to weed out the old/irrelevant stuff.

What to do about it

Before you say anything, no, there is no need for yet another Linux compatibility list project.

Ideas:

• contribute to the wiki
• add a wiki page for every machine you installed under Hardware/Compatibility/ Laptop,Desktop,Misc /$PRODUCTNAME (would it make sense to have it under Hardware/Compatibility/ Laptop,Desktop,Misc /$PRODUCTNAME/$DIST ?); link your page to the installation-report (you did fill an installation report for every installation you did, didn't you?)
  
• reorganize pages with directories (as proposed above)
  
• remove dead llinks
• add redirects for pages that just say "See foo"
  
• unify pages that contain almost the same information
  
• automatically make a wiki page for each (successful and unsuccessful) Debian installation-report based on the machine type (would need some logic to weed out the home made systems) - but only select the success/fail rate based on strictly hardware issues
• help Kenshi with his efforts for better hardware support

Update: Justin told me about the Ubuntu Laptop Testing Team project which is kind of nice. There is also a scraper script which was supposed to help making queries like "info about laptop X" and "which laptops have these features", but that script didn't work for me.

Via the Trademark Blog, My Cage — digitally generated silence for cellphone:

My Cage (Silence for Cellphone) dispenses with performer, piano and auditorium, instead utilizing a continuous stream of silence produced on a computer, and compressed to ringtone format. While noting that Mr. Keats doesn’t have a cellphone of his own, and may be less-than-qualified to make global pronouncements about them, the CEO of the company that distributes the ringtone believes that “My Cage” may be a platinum hit. “People want a respite,” he says, “and not everybody has the time or money to go to a spa. The virtues of silence are unsung.” Nevertheless, Mr. Keats is careful not to take credit for silence in general, and hopes that people will bootleg his creation, just as he was inspired by John Cage. Mr. Cage, who died in 1992, could not be reached for comment.

I hope the IP professors in the crowd are preparing a hypothetical for the next round of exams.

I have been asked for advice about long-term storage of documents. I decided to blog about it because my thoughts may be useful to others, and because if I get something wrong then surely people will correct me. ;)

Many organizations are looking at using computers for storing all documents. This gives savings on the costs of storing paper - the promise of the paperless office is being fulfilled. There are some potential issues about whether a signature in a PDF file that's scanned from paper is valid - but I guess it's the same as a signature on a FAX and everyone seems to accept FAXed contracts.

The technical problem is how to reliably store data long-term. The problem is that all modern methods of storing data will degrade over time. Anything less than engraving a message in stone, gold, or platinum and burying it will have some data loss eventually.

If the documents that need to be archived have no special requirements and if you have a good backup system in place (testing backup media, off-site storage in case of disaster, multiple sets of hardware that can read the backup media in case of hardware failure, etc) then you might be able to just store the documents on a server and include them in the backup plan. The regular backups should cater for replacing media over the long term. If however there is a significant amount of data or the data has confidentiality requirements that preclude having it all online all the time then you need a separate infrastructure for such storage.

Regular backup systems have to deal with files being deleted from storage and files that have their contents changed. For a document archiving system no file will ever be changed once it has been created and no file will be deleted. This allows some simplifications to the backup strategy. For example if you have multiple terabytes of documents backed up by tape and stored off-site you could use CD-ROMs or other media for storing recent changes. It would be very easy for an employee to grab a couple of CDs before rushing out of a burning building, but grabbing a set of tapes (or the correct tape from a large set) may not be possible.

It would be possible to use a tape library system as the primary storage for documents. If a large organization was going to implement this a few years ago that might be a good option. Nowadays storage is getting increasingly large and cheap. Terabytes are available in desktop PCs and hundreds of terabytes are available for server storage. So having the primary document store on a server with a decent amount of space and then making tape backups for storage in secure locations seems viable.

One thing to note about such document storage is that having everything on a server allows a much larger amount of data to be accessed and copied more easily than on paper. Sorting through a billion paper documents and copying the thousand most useful ones would be a difficult task for someone who was involved in industrial espionage. Finding the most useful files when they are indexed on a server should be quite easy and copying a few thousand is also easy (one thousand scanned documents of medium size should fit on a USB memory stick - much smaller than a few reams of copied documents).

Finally documents have to be archived in publicly documented file formats that can be easily read in the future. The PDF specification is well known and there are multiple programs that can display data in such files, another good option for scanned documents is JPEG. Proprietary formats such as MS-Word should never be used, you never know whether you will be able to read them in four years, let alone the seven years for which many documents must be retained or the 20-30 years that some documents must be retained.

Marc, this is not a problem with the Debian BTS, but with BTS in general. What you describe also happens with Mozilla and Gnome BTS, speaking of those I know. The Mozilla Bugzilla is indeed an example of what NOT to try to replicate, it’s probably even worse than the Debian BTS.

In response to my rant about Debian's BTS, I got some mails about other project's handling of bug tracking system - I'm still looking through those.

More interestingly, Joey Hess blogged about it and defended keeping old bug reports - which I also think. I explicitly talked about bug reports with no activity and no maintainer reaction. I took the time to look at some of the bugs Joey submitted and found some good examples for bugs and packages that obviously need help.

230485 is now in the ripe age of nearly three years. Almost three years is also the age of the last mail to that bug report, which was one maintainer setting the severity to important, without a comment. Since then, there has been no note if this bug is going to be fixed, how, when, why and who is going to do it. A user might get the impression that the maintainer simply doesn't care about this (or other) bugs. That's not a good thing, I believe.

271752 is also a good example. The bug seemed to be unreproducible, now someone found a way to reproduce it and ... nothing happened.

206605 is also weird - a patch is there (and we're talking about little grammar change here!), but no upload actually included it. Without a comment why it is not happening...

Keeping old bug reports to inform users of existing problems is OK. But not fixing bugs (or even trying to fix them) is a problem!

Marc, I disagree. Solving every issue and whim that every user files against a popular package, while not letting the package degenerate into a mess, is hard. Taking some time to gather opinions and produce a well-designed fix that strikes to the core of an issue, rather than treating symptoms, is a good thing for many classes of bugs. (Other classes should instead be fixed immediately.) The existence of many open bugs against a package in Debian is not an indication that it's of lower quality than the corresponding package in Ubuntu -- which in fact probably has 99% of the same bugs anyway. As a user, I also regard a BTS as perhaps the best form of documentation there is about real-life problems that I might encounter while using the package, the general quality of the code in the package, etc. It's the first thing I look at when installing some new peice of software. It's also a good way to gauge how well the package is maintained -- not by counting bugs, but by reading how the maintainer responds to bugs. What I look for in a BTS is an indication that the maintainers care about a bug. The fact that bug #725 took ten years to get fixed doesn't mean that X's maintainers suck, it means that for ten years, we were able to document a minor problem to the few users who cared about it, and that we were able to keep track of it until someone finally figured out how to fix it. That's quite an accomplishment. Another example, to keep picking on X, the fact that bug #216933 is marked wontfix is not an indication that the bug is worthless; in fact the bug documents a quite subtle issue in the Crusoe's code morphing software, and has been an invaluable reference to a lot of Crusoe users. I'm much more concerned about all our users who don't know about the BTS and are unwilling to bother to file bugs than I am about the total number of unfixed bugs in the BTS. Getting 100% of our users to file good bugs on every issue they run into would do much more to increase the overall quality of Debian than fixing every > 1 year old bug in the BTS would.

This is sort of a New Year's resolution... In no particular order:

1. Write a book.
2. Learn at least one new programming language.
3. Donate money to (or become member of) either the EFF, the FSF, the CCC, or another similarly important organization.
4. Make use of my rights as a citizen and write letters to elected politicians, urging them to promote certain topics/issues/laws (privacy, democracy, transparency) and to combat others (software patents, voting computers, data retention, mass surveillance and lots more comes to mind).
5. Work on and support selected Free Software projects in my spare time, especially projects which are of a greater importance to the Free Software movement (or the Free Culture movement; or freedom; or privacy; or anonymity; or democracy) than the 158th IRC client or the 276th tetris clone. Some examples: LinuxBIOS, Nouveau, Tor, Gnash, and Democracy Player to name just a few projects. General motto: Choose your battles!

Oh, and one more thing: Do the most important duty as a citizen of any democratic country — help to save democracy by killing voting computers. Yeah, so that makes six things I plan to do in 2007. Sue me.

I need to reconsider my current way to do IRC.

• My main computer is a laptop, so I can’t leave my IRC client open all the time.
• More and more, I tend to ask stuff on IRC, then I have to leave, so I never get the chance to see the answer. Being able to stay connected all the time would be much easier.
• I’m currently using XChat, and would like to continue using it (ie: I’d prefer to avoid going back to irssi).
• Most of the time, I’m connected to two different networks (OFTC and Freenode).

• I tried to set up an IRC bouncer, but I had the impression that it was complex, and not very user-friendly. (Maybe I’m wrong, or maybe I should have tried a different bouncer (which one?))
• I’ve considered running two clients: XChat for my “main client”, and irssi just for logging everything. But that’s not really optimal.

Which solution are you using? Should I really go back to using irssi? Any good example irssi configs somewhere? Update:

• Comments are now open (they were closed because of a broken WP plugin)
• I used to use screen+irssi, but then, I switched to xchat. I really like xchat, have written some custom scripts for it, and would prefer to continue to use it…
• The possible suggestions I already got are:
  • use xchat with irssi_proxy
  • use xchat with ctrlproxy (looks very nice, but RC buggy and not in etch)
  • use xchat with shroudbnc (or psybnc, or dircproxy)
  Update 2: Ok, due to all the praise of the screen+irssi solution, I’ll give it another try…!

I've set up some simple RDF export (gzip only, please add caching yourself!) for the Debtags data. The export is using the DOAP ("Description of a project") vocabulary, though it isn't optimal (we're not talking about separate projects, but multiple packages may belong to the same). The format of the RDF file may (and will!) still change, for example I'd like to have some explicit URIs for the packages instead of just storing it in the name tag. Suggestions for a matching vocabulary / serialization are welcome. But if you want to play around with some real-life data, just grab a copy. There is a couple of interesting things you can do with it, such as my folding tag cloud. So go ahead, and play around with it a bit.

After some comments on this post by Javier Candeira I've read more carefully Coloriuris license and I've revoked the license because I've some doubts that I should clear before anything. Also some statement are not as accurate as should be. I should not post anything when I've to study. Quick posts can be very dangerous :-/. Also I've closed comments because I've noticed that spam was hitting my blog. Sorry, I hope to fix this after exams. ... and I'm not talking about colors. Coloriuris is an organization that provides free licenses with the same spirit of Creative Commons. This well-known licenses are only applicable in the Common Law, countries like Spain, Portugal and the whole Latin America are based on the Roman Law where Coloriuris licenses should be used instead CC. For this blog I've choosen Coloriuris blue which allows comercial use and derivated works keeping the same license.