In the last few days we (as in DebConf admins Steve Gran, Mark Hymers and myself) have been doing some DebConf maintenance work. Some time ago one of DebConf’s permanent sponsors, ByteMark sponsored a new machine, krabappel, which we use as a replacement for the older machine we have from them, cmburns. It was neccessary, as good old cmburns had a lot of stuff running and so was always pretty loaded. Imagine it running most of our websites, main mail and dns server and then also having 4 vserver instances on it. Those vserver itself where also running things that do use some resources, like our wiki and the main DebConf7 site, the PentaBarf test-host and our LedgerSMB instance. While it was long decided to move stuff our - we just had to find time to do it. This weekend seems to be it, everything is migrated except cmburns itself, as that needs a fix in the reverse dns entries for the new /28 we got. We are using xen-tools to make installation of new images easy. For that I wrote a script that is used as a role script, automating all that various setup tasks that one can automate easily. Which means that it takes about 5 minutes to get a complete new XEN domain up and running. (Add some minutes to add it to nagios2, munin, our mail setup and to sync the userdatabase.) Mark and Steve also did another migration: gallery.debconf.org is now using gallery2. Still needs a nice layout designed looking similar to our normal DebConf website, but that shouldn’t be too hard to get. While they have been on it they also integrated gallery.debian.net, so that it is back up again. Feel free to use both of them! gallery.debconf.org offers space for all DebConf related pictures, be it the main yearly DebConf (and related) or regional DebConfs, like the one planned for Panama soon. gallery.debian.net is a place where pictures from all Debian related events/parties/whatever can be put. Basically - whenever people working on Debian meet (like a BSP, a work meeting, etc) - feel free to put pictures up there. Just put them into a folder matching the year it happened in… Basic goal is to get a huge set of pictures from people involved in Debian.

mod_wsgi 1.0 Yesterday the official 1.0 version of mod_wsgi was released and uploaded to unstable (the binary package you want to install is called libapache2-mod-wsgi, in case you were wondering), so I think it's a good time to blog about it. Mod_wsgi is an Apache module which can host any Python application which supports the WSGI interface (the reference is also available as package). Major applications and frameworks which are known to work well with mod_wsgi include CherryPy, Django, Karrigell, Pylons, TurboGears, web.py, MoinMoin, PyBlosxom and Trac. In fact every application which provides a WSGI compatible interface should be able to be hosted with mod_wsgi. The module is very simple to use, is very well configurable, documented, has a very active and responsive upstream author and is suitable for use in hosting high performance production web sites, as well as your average personal site. There are two common ways of operation:

• Embedded mode - working in a way similar to mod-python: The Python code is executed within the context of the Apache child process.
• Daemon mode - similar to FASTCGI/SCGI solutions, but with the important difference that a separate infrastructure is not needed when implementing the WSGI application and everything is handled automatically by mod_wsgi. Of course this mode has a smaller impact on the normal Apache child processes and also allows to run applications under different users.

If you're running python web applications which provide a WSGI interface: go and give mod_wsgi a try! ~~DISCUSSION~~

Finally we (Frank Lichtenheld, myself) took the time and upgraded the host for packages.d.o to etch. This was absolutely needed, as pdo was always running with a very high load for the machine, and the new code needed some etch-specific packages/package versions. So, today we went, turned off old pdo and upgraded it. It got a new kernel and lots of new package (versions) from etch, including apache2-mpm-worker. It is now accessible for everyone again, and its running the new code. Which does look nice, is way better and also much faster. Uses libapache2-mod-perl2, database files (instead of textfiles and grep) and a nice new layout. And possibly much more changes, but Frank will send a mail about it later. (At the time of writing this blog pdo is still generating its initial database files, so not everything working atm, but when that first run is done it will be MUCH faster in the future…) Say thanks to Frank for his work on pdo! Update: Thanks to the new and fast code (a full update now only takes real 40m26.415s, user 16m36.538s and sys 0m50.055s instead of hours), packages.d.o will now update twice a day, with a push directly after the mirror update has happened. Which should greatly help to reduce the “packages.d.o is out of date” symptom…

The Belgian Navy recently acquired a new frigate, baptized the Leopold I; and it's leaving soon on a training trip. The TV news just had an interview with the captain of this vessel, who declared that the function of a frigate is to protect "other vessels"; one of the examples he gave was "a carrier vessel". What he neglected to mention is that the Belgian Navy, in all her glory, doesn't have any carriers. In fact, it doesn't have any ships that are larger than frigates. But hey, at least this frigate has a helicopter pad, so maybe it qualifies. Then again, maybe not.

This is the first post of a series on componentization, starting with a brief explanation of the problem(s) we are trying to solve. I expect some familiarity with programming in general, but no specific knowledge of any particular system; the terms "DLL" and "shared library" are used interchangeably. When you are writing a DLL, you are essentially committing to support whatever interface you are exporting forever. So it better be good on the first try. DLL hell is what happens if it was not good, and you end up making an incompatible change. The obvious way an interface can be "not good" is when you need to touch the header files that define it in order to implement some new functionality. You can get away with a few things, like the addition of new functions, if you can somehow make sure that users will always have a recent enough version of your DLL around, but in general any change to the interface is going to be incompatible. The traditional Unix way of solving the issue has been renaming the library. When you link a program against a shared library on a system that uses the ELF format, the linker looks inside the library for the SONAME field, which tells it what name the library wants to be known under, and the program will then attempt to load this file at startup (which usually is a symlink to the latest compatible version). This "official" name of the library usually contains a number (sometimes more than one), and when there is an incompatible change, you simply change that number, and the new version of the library can be installed in parallel with the old. There are additional mechanisms, like symbol versioning support, that allow a single library to provide different "versions" of exported functions, and have the program specify the desired version for each symbol. To run a system like this, one generally needs to keep track of programs' dependencies; this generally happens in the package manager with more (Debian) or less (Red Hat) support from package build scripts. If you built your system from scratch, you end up doing this work yourself. That's why you had highly paid system administrators in the old days, and very short shell scripts to replace them (with various degrees of success) in the current age. In the Windows world, things have been a bit different, because except in large installations, you generally do not have a clueful, benevolent system administrator who will take care of these things; rather, you have users that expect your software to work without having an insight into the inner workings. In general, software developers have not been taking chances and simply shipped all the DLLs they needed along with the main program. When they installed these DLLs into the same directory as the program, things were fine, if they installed them into the WINDOWS directory, not so much, because when you had two conflicting versions of the same DLL, you had to decide which one to keep, and a program that would only work with the one that you threw away would simply fail in curious and interesting ways. This is the original definition of DLL hell — you cannot do it right. To make matters worse, this is not the only way an interface could change. When you define a platform, you write up an "Application Binary Interface" (ABI) document that details conventions that should be followed; for example, the ARM processor does not have a dedicated stack pointer register, so the ABI document for Linux on ARM says that R13 is to be used as a stack pointer, functions may thus expect it to point to memory that is readable and writeable, and memory with an address lower than the stack pointer is considered "free". When the function is done, it is supposed to jump to the address in R14, which happens to be the register the ARM's "Branch with Link" instruction sets to the address of the instruction following it. If the function takes arguments, the ABI says where to find them, and if it returns a value, it also tells where to place that. For everything not in the ABI document, the compiler author gets to choose. The ABI document usually attempts to be rather minimal, so it defines how C functions are to be called, and calls it a day. Anything not in the C standard is not guaranteed to work between different compilers (or even different versions of the same compiler). That includes C language extensions such as the "long double" data type, language features in C++ and Java such as namespaces, nested types and exceptions, and the data structures used by the C library's malloc() function. The latter makes life difficult even for pure C programmers, as it disallows releasing memory from a different module than the one it was allocated in, because the different modules may link to different C libraries. In essence, this means you are not supposed to export anything from a DLL that is not a C function, and even then, there are additional requirements. You could in theory export data objects, but that is discouraged, as it exposes an implementation detail. Now, we're in a bit of a dilemma here. There is lots of interesting code out there that is not necessarily written in C, we'd like to reuse it and we don't want stuff to break. The Linux approach so far has been "we simply pretend the problem does not exist", and this has been working to a large extent, simply because everyone uses gcc, and when the ABI extensions the gcc people had made changed, recompiling everything that uses these extensions makes the problem go away. This has several drawbacks. For one, it mandates that the tools used to rebuild everything and keep the installation consistent be written in C or handle the transition gracefully (for the curious who have a Debian system handy: look at libapt-pkg's SONAME), and it also requires that we have the full source code for everything not just available but buildable in an automated way. Within the context of a Linux distribution, both of these are true, so we have survived the last major ABI transitions without too many losses. So we need an extension of the ABI document that specifies how modules written by different people and compiled with different compilers can interact. These are commonly called component models, and, to quote Andrew Tanenbaum, "[t]he good thing about standards is that there are so many to choose from". Next: "componentization tutorial #2: CORBA and COM"

A couple of people have pointed me to the "Skype DoS exploit code" that has been published. I had seen that, but I'm not convinced it works as simple as that. Some of the information around it doesn't make completely sense (such as using the term 'server', when they're referring to super nodes I guess, and since supernodes are just regular user machines annexed by the Skype network, they supposedly run the same software, don't they? So why doesn't it take down the client the exploit is run on?Also I'd bet that someone has tried feeding the Skype client long URIs before; that is one of the most popular ways of seeing if some software can break. You know, Buffer Overflow [wikipedia] is probably the most common class of security issues (maybe second only to PHP programming errors or SQL injection by now, though, with so many people with too little expertise writing webapps in PHP)Others probably are wondering why I'm writing so much "against" Skype.There are numerous reasons:

• The whole P2P thing isn't necessary, they could use real servers
• Skype is a pain for every network admin (and thus a users nightmare, since the admin might decide to just block any traffic that could be Skype, and enforce the use of HTTP proxies etc. and thus limiting other applications as well)
• Skype uses all kinds of shady coding techniques in their client to obfuscate what their application is actually doing
• Skype is a security risk
• Skype is a memory hog (it uses 10 times as much memory as my other IM client, who does ICQ, MSN, Yahoo, Google Talk and tons of others!)
• It's a resource hog (it wakes up 200 times as second for nothing, thus preventing my CPU from using power saving states efficiently)
• It's a closed protocol and network, while there are open industry standards such as SIP [wikipedia] and H.323 [wikipedia] that can do much more than Skype
• It's UI is crap (especially Linux version 1.4 is a serious degradation vs. version 1.3), contrary to any usability best practises
• Their API is crap. I'd call that "raping" the DBus API what they're doing (basically they're offering a DBus interface that is just a transport wrapper for a text-based 'telnet-like' API. You know, DBus interfaces are meant to have meaningful functionality (like 'make a phone call') and not meant to be just "send data to the skype application")
• They don't tell the truth. Like e.g. what has really been happening these days. Or what their software really does (see 'obfuscation' above and search for "Silver Needle In The Skype")

And, honest, there is nothing in Skype that other apps wouldn't offer, or had been offering before except being really aggressive at getting through firewalls without any user intervention.

Joey complains about users filing aggressive or otherwise inappropriate bug reports. Well, in this case I must say I bit the bullet.
Yesterday, somebody complained about the maintenance statuse of libapache2-mod-perl2. Usually, I would have sadly nodded and said that, yes, since the API change between mod_perl for Apache 1.x and mod_perl2 for Apache 2.x (which was a long time ago - with perfect timing not to be accepted in the last weeks of Sarge's hard-freeze period, causing many of us to have to maintain two or even three versions of our code depending on the API used - But that's a different story I won't go into details now). But this time, the bug report was sent with Severity: critical.
I replied to this report, intending just to lower its priority and get the attention of the maintainers by sending it to a couple of Debian lists - But soon afterwards, it became obvious that I was probably the person most interested in this package in Debian - At least, me and some other pkg-perl fellows who -as foolishly as myself- volunteered to step forward when needed.
Well, in short: I am now the proud owner of one of the packages most vital for many of the systems I've written for my work. It's also more complex than most of the other packages I maintain. From an undermaintained and quite full of warnings build process and resulting package set, I managed to make it linda- and lintian- clean, and... Well, now we have to dig through its open bugs. Its build system is far from orthodox from a Perl point of view, and that took me most of the day, but hey - TIMTOWTDI, right? :)

Making progress here - courtesy of apt-rdepends and Simon Law. The Depends handling in libapt-pkg-perl is fraught with difficulty. Self-referencing arrays mean that I can't use my normal use Data::Dumper; print Dumper($foo); because Dumper quickly descends into an infinite loop and if you don't catch it fast, you lose X.
Hopefully, using a simpler interface will help develop other scripts to use the apt cache directly.

Introducing debcheckout Cute little tiny teeny new addition to devscripts: debcheckout (not yet uploaded though, in the mean time you can get it from here. It checks out the versioning repository used to maintain a given package. Sample usage:

$ debcheckout devscripts
declared svn repository at svn://svn.debian.org/devscripts/trunk
svn co svn://svn.debian.org/devscripts/trunk ...
A    trunk/debian
A    trunk/debian/control
A    trunk/debian/links
A    trunk/debian/dirs
A    trunk/debian/compat
<snip>
 U   trunk
Checked out revision 749.
$

The information about where to find a repository is extracted parsing (in a rather dumb way actually, but I really can't stand libapt-pkg API!) Vcs-XXX fields. Intended usages:

1. NMU scenarios: when you're NMUing, please commit your patches (if possible of course: directly to the repository if it has already adhered to the open your VCS campaign I'm sponsoring, or somewhere else if you're using a distributed VCS); with debcheckout the first step it's easy
2. ease the creation of patches: isn't it better to checkout a repository, fiddle around, and then just invoke svn (or whatever) diff instead of remembering (I always forgot that!) to first create a .orig copy of the debianized source tree?
3. retrieving the bleeding edge version of a package which includes the patch for a pending bug you have been waiting for ages

RFC Let me know what you think of debcheckout, feature requests, whatever. In particular let me know if I did something wrong using some VCS, since I'm not proficient in all VCSs supported by debcheckout; for example: I'm quite sure the Arch part is not working ... help is appreciated! Vcs-Cvs proposed convention debcheckout can also give the ground for standardizing the VCS-specific meanings of the various Vcs-XXX fields. In writing it I've noticed that almost all VCSs have de facto standards about what to put in the field, with the notable exception of CVS. That's probably because all modern VCSs rely on some URL-like identifier for a repository location, while CVS does not. It needs a pair URL/module to be checkout. The format I'm currently supporting in debcheckout is a pair of space separated values "CVSROOT MODULENAME"; later on I'm using those values as in cvs -d CVSROOT checkout MODULENAME. Also please note that if you do not put the heading ":pserver:" string in the CVSROOT, users won't be able to checkout the repository without providing a password. Tip of the day: Pod::Usage Perl's Pod::Usage module is cool. Finally I can write the usage string only once instead of duplicating it in the manpage and in the string to be printed upon --help. Ruby had something similar, but the output on console was so horrible that I preferred duplicating stuff for Ruby scripts. Update: I've changed the link to debcheckout.pl so that it points to the "live" version in the devscripts repository, since some patches are already flowing in ...

Dearest Lazyweb, I want to write a program that I can point at a standard repository, and an in-house one, so I can track version differences between common packages. I wanted to write this in Python, so python-apt sprang to mind, however the documentation is somewhat lacking, and none of the examples include retrieving a package list from anything other than the local APT cache. The documentation in libapt-pkg-doc is also rather lacking. So before I go giving up on using a "proper" library for the task, I thought I'd reach out. Love, and sloppy kisses, Andrew

Having taken time out to go to the other side of the world to attend DebConf 7 in Edinburgh, and then after spending some time with my family, I have been somewhat distracted from things CalDAV for a wee while now. There are signs that I will be able to put a few more evenings and weekends into it in coming months. Current Status With the release of 0.8.0 I believe it is quite a usable application for shared group calendars. We use it at Catalyst for about 80 staff and, while we're not heavy calendar users, it manages without any noticeable performance problems. There have been an impressive number of downloads from Sourceforge, as well as substantial downloads from my Debian repository, and I believe that the program is in tens, if not hundreds, of installations. Future Plans In the immediate future I want to get the basic framework for querying properties about the owner of a calendar or a set of calendars. This will be used by Mozilla soon during their configuration of a new CalDAV source, and I think that Apple iCal probably uses them already. This is partly done, but it's implementation has been slowed by the necessity to refactor the handling of the REPORT method in general to make the code more modular and readable, and to support queries and properties in a more general manner. The refactoring of the REPORT method handling is in place now, so moving on to the handling of DAV:principal objects should become fairly straightforward. I may end up with a fair bit of quiet, undistracted time over the coming weekend so if I can get in the 'zone', it may get done, and we can start rolling towards a release 0.9. Release 0.9 will have a new name. One from the cast of thousands that have been submitted to me so far, or one that I will hastily think up in between, but the next released version will not be called RSCDS any longer. The main goals for further into the future are:

• Maintain interoperability with clients
• Improve specification compliance
• Extend coverage to related specifications
• Documentation improvements
• Configuration improvements
• Installation improvements

Interoperability I think that RSCDS does fairly well on the interoperability front. Unfortunately I have no access to the preview of OS 10.5, with it's native CalDAV client (though I have been told that it works with RSCDS). Likewise, other commercial CalDAV clients aren't available to me for testing either, so I have concentrated my efforts on supporting the featureset used by the various free CalDAV clients. My initial work focused around supporting Evolution, in particular, but with the recent effort put in by the Mozilla Calendar team the primary interoperability challenge is to enhance RSCDS to support their areas of development. If I could see how the OS 10.5 client behaves, too, I believe that would help identify a few points to extend, as well as suggesting some future directions. The best place to review interoperability would probably be at the CalConnect Roundtable 10 but I think that without some sponsorship for airfares I probably won't be able to make that. Maybe next time. Specifications Compliance As I read, and reread, the CalDAV (and DAV, and HTTP, and ...) RFCs the compliance with the various specifications improves. Recently I have put a lot of work into the calendar-query report, in particular, which means it now supports querying against arbitrary properties much more comprehensively. This is important because some of the client software is starting to do arbitrary queries also, and it is inevitable that queries for the classification of events, or the percentage completion of todo items will become more common. And now I notice that there is a new version of DAV in RFC4918, though not too many changes as far as I can see. The intent of this release is mostly to clarify, and slightly to extend, primarily in order to improve client interoperability. Coverage of Related Specifications In terms of general calendaring activities, the biggest hole in the CalDAV specification at present is standardisation around the location of other people's schedules. The current draft of the Scheduling Extensions to CalDAV appears to address that, and a lot more besides. I expect to start working on parts of this at some point, but perhaps not sooner than I see a client which starts using it, or when it is ratified, whichever occurs soonest. Documentation The documentation is OK. The wiki, in particular, is a reasonable resource for resolving problems (I believe), but it would be good to have some documentation of other things. One area that could do with enhancement is some documentation of how to check out a copy of the code, and how to provide patches back to me. Even better if other people can maintain their own trees which I could pull patches from, since we're using a distributed version control system (Git) to start with. Configuration The administration interface for RSCDS is OK, but could do with a few improvements to make it more usable. Really there are not a lot of reasons to be in the administrative interface, so these improvements should be relatively few, and relatively easy. This would be a good area for someone interested in helping out to become involved, because it does not entail reading large reams of RFCs, or watching tcpdumps scroll past for the debugging. Installation One thing that I have done right, I believe, was to provide the software from the beginning in a packaged form to make the installation easier. Although the software is packaged for Debian, however, it has not yet been uploaded to the archive because I have been waiting for the webapps-common package to enter Debian so I can use that for configuration. I think I have waited long enough though, so I will upload the next version to Debian for use by a wider audience. Security Security has always been a focus for me and I have had the code reviewed for any obvious security flaws. If any particular vulnerabilities are found they will be addressed with appropriate haste. Retrospective I've been working on this for a little over a year now, and I think I have largely met my goals for the project. It has certainly been an educational experience for me, and the effect of this work on other projects that I also participate in has also been valuable. Any new open-source project is always at risk of the developer losing energy and interest, but I believe this one has made it to a state where the bulk of the infrastructure effort has been done, and there are enough interesting challenges in the future to continue to keep me (at least) working on it into the future. The scope of this project is good, in that it is not at risk of expanding to take over the world. The intention behind this is simply to provide a back-end service for client software which should be very much Someone Else's Problem. In reality there have been times when I have had to get to grips with some of that client software in order to resolve problems, but those have by and large been good experiences.

We had a great weekend this last weekend here in Montreal. Yesterday, June 24, was Saint-Jean-Baptiste Day here in Quebec -- the national holiday of this cool and fun nation. St. Jean Baptiste is important to me in a lot of ways. Of course, it's cool that it's the national holiday of our adopted new home. But John the Baptist, the patron saint of Quebec, is also our family's name saint. ''Prodromos'' means "forerunner" in Greek, and it's one of the titles of St. John. Most of all, it's the day that Maj and I moved to Montreal. We came in on St. Jean Baptiste Day, 2003, and we've been here ever since. We drove my beautiful 1972 Citro n DS from San Francisco, where my niece Elena had just been born, all the way out here. We arrives on June 24th and wandered around the McGill Ghetto looking for our sight-unseen sublet. It's hard to believe it's only been 4 years -- it seems like so much longer. We had a good day yesterday. The three of us went to Piknic in the afternoon and hung out with our friends Jeremy and Lina and their two great kids Clea and Sylvie. We had a lot of hummus and pasta salad and bread and cheese and generally had a great time. Meg and Catherine came and hung out for a while, and also our friend Kristen, who's 7 months pregnant and has a 10-month-old. Good luck on that one. Most of all a pretty day enjoying the outdoors in our new home town, as well as fun dancing to house under a big Calder statue. How great is that? tags: montreal quebec stjeanbaptiste prodromou piknic jeremy lina

72 hours from Flickr I forgot to mention that we had a really nice time on Friday evening going to the 24 Hours of Flickr party at the SAT. Heather Champ, community leader at Flickr and welcomer to partiers, put it well: "We have three requirements for Flickr parties: the food must be good, the drinks must be copious, and the schwag must be ample." And that's exactly what happened. All the usual Montreal suspects were out in force: YULblog types, Web entrepreneurs, Web pranksters. It was great seeing m-c and c-c, who wore their awesome Flickr Socks -- although they each had a matching pair. I also talked to Allen Morris, which was really fun. He's a developer living now in Saranac Lake, one of the towns in the Adirondacks I find strangely compelling. We exchanged GPG key signatures and talked a lot about the Bay Area (California), where we both once lived. Probably my best talk was with Heather Champ and Derek Powazek. We talked about their joys of creation of JPG Magazine, the despair of leaving it, and looking forward to the future. We also chatted a lot about print on demand technologies, which we're evaluating for Wikitravel. Heather took the photo of my friend Ben Cerveny at Burning Man that's become his avatar icon on about every social software platform I know of. It must be good to capture someone in the way they want to think of themselves. I took my own self-concept picture by putting the camera on the dashboard of my Citroen. Heh. All in all a good party. One thing I was interested to find out was that there's a Yahoo! Quebec, which launched last month and I guess just slipped by me. Sadly, if you click on the "English" button, it takes you to... Yahoo! Canada. Hey, Yahoo! There are actually a lot of anglophones here! Just in case you were wondering. tags: yahooquebec 24heuresflickrmontreal 24hoursflickrmontreal flickr montreal party

Montreal Web Development Book Club Montreal Web scene stalwart Heri Rakotomalala -- the man with the magic name, originally from Madagascar, along with Montevideo and Marfa, one of the M-places on my travel wishlist -- recently started a Montreal Web Development Book Club. It's a meeting for people interested in the practical side of the Web to talk about the books they read and let that drift into talk about the work we do and ideas we have. I think the book club is an awesome idea. Book clubs are a great grass-roots intellectual meetup that we don't often celebrate in the post-flash mob BarCamp-ized world. The great thing about a book club, for Web people, is that it filters out the bullshitters and dilettantes. If you're going to a book club, you're going to have to know what you're talking about, and you're going to have to care about the subject. The book gives focus that keeps the conversation from drifting to tangential topics. Flakes and hangers-on just don't have the stamina for it. Also, we met at La ka, so it was nice to have some beers in a good place on a hot night. Why not? We managed to talk about a lot of good books, too. Jean-Fran ois <something> (I didn't get his last name) talked about In Search of Stupidity, a look at the last 25 years of marketing and business blunders in the high-tech realm. Our industry is subject to a lot of decisions that seem absolutely self-defeating in hindsight, and the book goes in-depth on 15 case studies -- like Wordstar, Ashton-Tate and Borland. It sounds like a great cautionary tale for Web 2.0 startup mavens. Heri talked about the book Information Architecture for the World Wide Web -- the infamous "Polar Bear Book". It sounds like a really good book for doing initial planning for a Web site, but I'm really interested in tools for profiling an IA effort. We get about 2 million unique visitors a month at Wikitravel, for example, which is a huge amount of usability data I could be mining. But I don't really have the tools for it, so I don't do it much. I'd love to know, for example, about people who land on a Wikitravel page from a search engine and then leave. What were they looking for? Why couldn't they find it? Also, users who browse through our pages clicking on links, then resort to a full-text search. What did they think they were clicking on, and what were they looking for instead? Did they find it? How can we put that in front of them in the first place? Most of all, I want to know how and why people start editing Wikitravel pages, and why they stop. Of people who click the "edit" button, only 1 in 17 click "save". (That is one bit of clickpath analysis I've done.) Why? I've often been tempted to put a survey up when someone navigates away from our edit page. "Why didn't you save? Tell us so we can fix it for other people." I talked about the book I just finished reading a few weeks ago, Everything is Miscellaneous. It's the latest book by Dave Weinberger about how we impose order on information to make it easy to understand and retrieve. It's an awesome book -- really fun -- especially for someone married to a knowledge management expert. I don't think I gave a good enough description of the book to ignite the club's interest, but I hope they check it out, because it's really readable and good. At the end of the meeting Heri took a poll and decided that we'll continue doing the book club on the last Monday of the month. I've already picked out my book for next month: RESTful Web Services. I've heard good things about this book (admittedly mostly from the co-author Leonard Richardson) and I want to give it a look-see. So book-club fodder it is. ''Update: Heri has his own book club report here.'' tags: bookclub montreal heri webdev montrealwebdevbookclub

Those of you that look at photos on my blog may have noticed that I recently switched from a self-hosted Gallery2 installation to the popular photo-sharing site Flickr.

I initially decided to try Flickr for a simple reason: digikam's image upload to Gallery2 has never worked for me, and I hoped that its Flickr upload would work (it did).

I had heard good things about Flickr from friends and colleagues, so I gave it a try.

First Impressions

Almost the first thing I noticed about Flickr was that there are people there that are interested in the same things I'm interested in. I discovered the Anabaptist History group. I've been contacted by a person in North Dakota that is interested German-speaking Russians, as my ancestors were. There are groups about Debian, classic-style black & white photography on digital cameras, and really just about any topic. These groups contain both discussions and a photo "pool" to which any group member can post.

In one sense, Flickr is social networking via photography.

Flickr also seems to be a company with a clue. It has tons of open APIs, permitting everything from flickrfs (mount flickr as a FUSE filesystem) to countless uploaders, mashups, scripts, etc. Despite a couple of recent high-publicity controversies, they seem to be laid back overall and haven't been turned into a corporate drone by Yahoo. There's an RSS feed for just about everything, too.

What is Flickr?

Depends on what you want it to be. It could be that place where you post your cell phone's snapshots, or a place to share photos with family in private, or just a place to find other's work.

I found sharing and talking with others to be infectious. I went from never getting comments on photos on my Gallery2 site to wondering what people will think -- and looking at their photos too.

Let's look at its different aspects in more detail.

Uploading Photos

You can upload photos manually with a HTTP form, or you can use any of the numerous uploading tools. Most of these will let you scale down your images on the fly, should you wish to. Most will also let you assign tags and write descriptions as they're being posted, as well as define who can view the photos. Uploaders are available for every common platform.

Flickr has a AJAX tool called the Organizr that helps you assign tags, add photos to groups, put them in sets, etc. It's fairly nice.

Sharing Photos

Making photos available to the public is easy. You can just them them a link to your photostream. You can also tag photos and give people a link to photos with a specific tag. And you can create sets -- photos around a specific theme.

Security

Each photo can be made visible to: just you, just your contacts marked "family", just your contacts marked "friend", any of your contacts, or anyone at all. This can be defined per photo.

On a global basis, you can define whether you make EXIF information visible on the site, default security for new uploads, who can post comments, who can post photo notes or add tags, and who can use the "all sizes" button.

You can also control how much of your profile to make public. Settings range from making everything public to concealing even your email address.

For sets, you can also send out a "Guest Pass" (GP). A GP lets someone that is not a Flickr member see non-public photos. I use GPs to share family photos, so that family doesn't have to create a Flickr account to see them. GPs can be tied only to sets.

Groups

Flickr groups appear to have been designed for discussions, but I find them most attractive because each group has its own dedicated photo pool. Members of the group can add their photos to the pool. Some groups have tens of thousands of photos, and others a few dozen. It's an interesting way to find images. There are a lot of group management tools as well.

Free vs. Paid Accounts

Free account holders can upload up to 100MB of photos per month. There is no cap on bandwidth consumed by views of photos or management of them. Free users can create up to 3 sets and put photos in up to 10 groups.

The photostream for free account holders will show only their 200 most recent photos. Additionally, the "All Sizes" button will max out at the Large size, which generally will be 1024 pixels along the longest side.

Paid accounts are $25/year and include unlimited upload bandwidth (10MB max per photo though), unlimited photostream size, unlimited sets, and removal of all ads on the site. Additionally, the original upload is made available to you. If you have enabled the All Sizes button, it's also made available to the public. One nice thing is that with a free account, all that data is saved, even if it isn't available. So if you later upgrade to Pro, it's all there.

Prints

Flickr sells prints of your photos for $0.15 each. As with viewing, you can designate who can purchase prints of your photos. I ordered a test batch and they turned out very nice -- approximately the same quality as Shutterfly, which is one of the better consumer-quality labs. They are a bit pricier than Shutterfly when buying in bulk, though the convenience of all the Flickr uploaders may make up for that.

Annoyances

There are a few things that bug me.

One is that the "All Sizes" button is either on or off. Once I upgraded to pro, suddenly my original size images were available. There is no way with a pro account to limit image availability to the 1024xwhatever size like there is with free -- either you give people the medium size only (500xwhatever) or you give them everything. Those that want to have the originals on Flickr generally upload twice: once in original size, kept private; and another resized to 1024xwhatever, made public.

It is also complex for some people to join Flickr. Several of our family and friends had trouble with the signup process, since you first have to create a Yahoo account, then a Flickr one, which can use a different username. It's a nice flexibility, but overly complex for people that just want to sign up quickly. Guest Passes have received rave reviews however -- with the one caveat that the guest pass URLs aren't bookmarkable (they set a cookie and then redirect to the real target).

There is also no built-in way to see all the new photos in the groups you're a member of. However, each group's pool has an RSS feed so I have just subscribed to them with bloglines.

Overall

Overall I am happy with Flickr. I will be migrating my Gallery2 install to it, and will not miss having to maintain Yet Another PHP Script on my server. The best part about Flickr is the community, which has already taught me about mpix.com, which I'll write about tomorrow.

Hmm, it's funny to be asked to blog about something. By mupltiple people. I could talk about an old church filled with a 100+ geeks, an organ played by Keith Packard, wine and cheese from all over the world, etc, but ok, I'll bite: Now that I got it working, Enrico's ept-cache is pretty neat. It found a lot of odd cruft on my laptop, along with some packages you probably don't know about that I'd be happy to chat about. termnet - Simple Telnet replacement for termnetd
grunt - Secure remote execution via UUCP or e-mail using GPG
longrun - Transmeta Crusoe LongRun control utility
iconc - Compiler for Icon, a high-level programming language
dbacl - digramic Bayesian text classifier
icont - Interpreter for Icon, a high-level programming language
bogl-bterm - Ben's Own Graphics Library - graphical terminal
intercal - an INTERCAL de-obfuscator
geotoad - geocaching query tool
libnet-openid-consumer-perl - library for consumers of OpenID identities
libapm-dev - Library for interacting with APM driver in kernel
libtextwrap-dev - text-wrapping library with i18n - development files
libklibc-dev - kernel headers used during the build of klibc
fuzz - stress-test programs by giving them random input
libdebconfclient0-dev - Development files for cdebconf
libslang2-pic - The S-Lang programming library, shared library subset kit
libnewt-pic - Not Erik's Windowing Toolkit, shared library subset kit
gnats-user - The GNU problem report management system (client tools)
libdebian-installer4-dev - Library of common debian-installer functions
mailfilter - A program that filters your incoming e-mail to help remove spam
buici-clock - attractive desktop clock)
atari800 - Atari emulator for X/curses/SDL
hercules - System/370, ESA/390 and z/Architecture Emulator This does make for fun conversations..

I received this week my new bridge camera: a Sony DSC-H9. It is meant to replace my Konica Minolta Z5, which had some defaults for the use I have. Along with its 8.1MP CCD and its 15x optical zoom, the H9 has some nice features, such as a big 3″ screen, a NightShot infrared system, a high sensibility mode (up to 3200 ISO with some expected noise) etc. Following to my first tests, I particularly like the menus, which are clear and allow very quick fine setup. The AF is very reactive, the built-in flash quite effective. The camera is delivered with an infrared remote control, as well as a lenscap. The only drawback for the moment is the usb cable which is not easy to connect to the camera. I guess I should use an external smart card reader in the future.

I am considering switching from SVN to a distributed SCM for my personal stuff. I had a look at git and mercurial, but neither really support branching a sub-directory: Often, I am working on a big private project, and, while working on a sub-project (stored inside the project’s repository), I’d like to share that sub-project with others. So there are actually two problems:

• being able to checkout/branch/clone a sub-directory
• possibility to control access on a per-directory basis

SVN only partially meets my needs with that (it’s possible to checkout a sub-directory directly, for example with svn co svn://svn.debian.org/svn/pkg-ruby-extras/tools/ruby-pkg-tools). I think that it’s possible to do fine-grained access control
using libapache2-svn, but I haven’t tried yet. It seems that mercurial can do that, using the forest extension. But you have to convert the specific directory into a repository, with a complex step to keep the history. Amongst the distributed SCM, is there one that supports that ? (at least the sub-directory branching part)

Before I start with my report I need to fulfill my promise for those who stumble here from kushaldas.in: The URL of the web interface for the package description translation framework is for now: ddtp.debian.net (It's easy: Debian Description translation pproject). There might be some changes after DebConf, our annual conference, which will take place this month. So keep an eye on your language's team mailing list. But more about Kushal Das later. The first part of my report will cover a bit of our DebianDay and of course especially my talk ;) More will follow later. Friday was DebianDay and I attended Holger's talk about debian-community. It's a quite interesting project, trying to solve the problem, that there's not much between real DDs and the rest of the world. It's more low level than the Debian maintainers idea floating around for a while: Look around, you'll see much contributors to Debian, who are not maintaining any package. The idea is basically to have kind of bonuses for contributing to Debian, as a more direct way to say Thanks!. Have look at the website, and think how you could help to get things rolling. After his talk it was my turn. I did a Debian package building for beginners talk... Well, actually it's a workshop stripped down by the practical part leaving only the slides of the introduction. But who cares ;) Again I was surprised how many people where interested in that topic. I always thought it's kind of special and not that interesting, but the room was quite stuffed. Some people where even sitting on the floor. I neither counted how many people attended my talk, nor how many seats where available. I guess I had something between 80 and 120? Perhaps Wolfgang Borgert, who moderated DebianDay, can correct me, if I'm wrong. Since I stripped down a workshop to a talk, I needed to take special care about the timing. Well... I didn't work perfectly. I took a bit too long while answering questions, but I think all in all it was quite right. As usual I took gnujump as example; easy package, works without much tweaking of the templates created by dh_make, and if you have some time left at the end, you can show some additional stuff, like splitting of a -data package. After the talk I got some quite interesting questions; the three most interesting ones were the following:

• Non English license texts: One guy asked me about non English licence texts (in his case: A Japanese license text for some special printer driver).
  I asked him, to seek help by a Japanese DD, which might be okay to let the package pass ftpmaster. Sorry, but THIS IS NOT ENOUGH! I just asked J rg Jaspert, one of the ftpmasters, and the Debian Project need's (of course) a translated version of that licence, as well as a statement of the upstream author, that the translation is okay. Otherwise the translated text has no legal binding and is therefore useless.
• Installation packages: Packaging non-free stuff, which you need to download yourself, is a) sadly sometimes needed, b) useful for some people who need it and c) some kind of tricky. I could help much here; didn't did anything similar, yet. So I answered to take a look at either , msttcorefonts or java-package. Question to the others: Is there some kind of common infrastructure to build an installer package upon?
• The Joomla! problem: Again a thing I have no experience myself in; packaging web applications. According to the Joomla! guy I talked, too, there is a special problem with that (please correct me, if I understood something wrong; as said: webapps aren't my speciality).
  There is demand for Joomla! packages, but so fare none exist. Major problem: While Joomla! is capable of running at multiple aliases, it is not (yet) capable of handling them to serve different content to them. So he wanted to be able to install Debian-Packages to different directories, where he would add different configurations of them. Short term solution could be to install it to /usr/share/ or so, and create a script, creating a symlink farm... long term solution should be to fix Joomla! ;)

So much for now; I'll write some more about the event in general, a goodie from an old friend from Treuchtlingen (rather short; the german entry is longer), and a small talk I had with an other J rg later... I'm kind of tired right now.

Disjointed: Busy weekend: A brief trip to Glasgow, a lot of rat-lovin and a few minutes to hack on CFEngine package automation. I discovered insane performance problems with running Ruby on Rails applications under libapache2-mod-fastcgi so I switched to Apache2, mod_proxy & mongrel. The process wasn t as painless as I d expected so I might write it up shortly. In other rails-news: I can t get typo running on Etch. Migration from wordpress 2 fails in strange ways. Help?! In other news xmms on my AMD64 machine, reading /home/mp3 over an NFS share locks up every couple of days. Not sure if this is an AMD64 bit problem, or if the NFS usage is responsible. I should investigate. (The GUI is visible and responsible; but hitting next , play , etc, result in no action. I usually have to kill it then restart.)

Does anyone have a (partial?) policy for Tomcat on SELinux "strict"? It's definitely something I'd like to confine, however Java needs tons of permissions. Multiple ports, execmem, log files, fifos, on-demand extracted webapps and so on. :-(Has anyone started on writing a policy for this beast? If there is any way to confine it actually, given that it's pretty much a system on it's own.[Update: I'd like to point out that running tomcat on a SELinux 'targeted' system works fine by running tomcat in an 'unconfined' domain. Since tomcat usually isn't run with root privileges anyway, it's not that risky]

First of all, sorry for the delay. Leaving just as the discussion gets started is bad, yes... But I'm only now reading Erich's and Sven's follow-ups. Both (as well as some comments in my blog) ask why not integrating the startup links in each of the packages - Well, basically because I don't think that most maintainers will take care to do this, and we will end up having a situation very close to what we have today: If I'm not interested in supporting your favorite init system in my packages, I just won't bother to make the scripts.
Note: I'm going into braindump mode. Verbose blabber and some stupidity might follow ;-)
Think on the webapp scene - Most webapps ship with an Apache-like snippet so that http://yourserver/thisapp just works(tm). I love that, and it's one of the little details that make Debian shine - Things usually work with the least administrator burden possible. But it happens that there are other web servers around there - They just become somehow second class citizens (I happen to sponsor/comaintain Cherokee, for instance), as nobody cares to include the equivalent snippets for them. Apache is the standard, and is good enough.
The same goes for sysv-rc: It just rules the world. Who will work all the needed patches to support all the different init systems? As the maintainer for a simple package which requires to be started up, I probably won't care to even understand all the intimacies of every init system, at least until they all have a decent user base. But by having one package per (server,init-scheme) pair, any maintainer can come up with the needed initialization.
Of course, this degrades quickly. First of all, as a user: if mydaemon is not correctly starting up, I will probably file bugs at mydaemon, not at mydaemon-initscheme. If mydaemon changes its parameters, we will witness transition of mydaemon-*.
Further, remember this is Debian, and volunteer-work has its downsides. If the mydaemon maintainer is a primadonna (or just does not give a flying crap about the runit init scheme), he will just trash reports regarding a nonimportant init scheme. Bad. And, as Erich points out, we have ~1000 packages including /etc/init.d/something - It will mean 3000 or so packages for a decent (not complete!) coverage of the different schemes. And, of course, a very uncoordinated way of working. But back to my line of thought: If I'm promoting an init scheme, I cannot just push it down each maintainer's throat. I must include at least the most important init scripts somewhere. Maybe we could just group daemons by task, and then have -say- a webservers-runit package providing the init scripts for each webserver for runit? This could cut down from 3000 to some 100 packages, most probably team-maintained... But it still faces many scalability problems, and the bug-filled-somewhere-else problem seems unavoidable.
I think something interesting could come off Sven's idea of providing several independent scripts instead of today's complete init scripts - This would make it easier to adapt startup/shutdown and similar events to different world views, and if not specifically needed, most init scripts could even be autogenerated calling the right bits here and there. That would rock - except in the corner cases (I predict no less than 10% of the packages will become corner cases ;-) ) where it will crumble apart. But maybe if a package declares it should be autostarted and provides the separate bits, the sysv-rc, upstart or runit helper can come up with an autobuilt initscript (or equivalent) - And if it does not work, it can always be overriden by a maintainer- (or user-) supplied, explicitly built script. Humh...
The topic surely calls for a Debcamp session, as Joachim says in comments in two of our posts and Erich acknowledges. Erich, as the main instigator of this blog series, I hope you can at least join via Ekiga or such, as it can be quite interesting - But, yes, none of the people involved so far participates in any of the inits' maintenance... Anyway, please keep the ideas flowing. I want to sketch something up, as I feel this can be useful - and not only for initscripts, but for many of the areas where Debian provides several ways to do the same thing. And, once again, that's one of the best points of Debian for me.