(This rant was originally part of the live streaming series, but I moved it to a separate article because people got stuck on it instead of responding to the actual subject of the article.) Before I get flamed for attacking Pulseaudio (PA), let's just settle this: I don't like it. I think PA is over-engineered and tries to do too many things at once. I used to (until just now) systematically purge pulseaudio-related packages from my system, mainly because PA has this awful tendency of automatically starting and staying around eternally, which wouldn't be so bad except PA has also the bad habit of hogging the audio device exclusively, which makes regular programs like mplayer, ogg123 fail to simply play audio, unless they go through the PA straight-jacket. (Update: this seems to be a bit better in newer versions of PA, where the audio device is released when sound is not being played. Thanks Philipp Kern for the correction.) In the GNU/Linux audio stack, we already have another system that supports multiple accesses: ALSA itself, so I believe that PA should at least behave nicely to other apps, and it seems I am always having problems with that. (The fact that ALSA itself is over-engineered is also telling of the poor state of the audio stack in GNU/Linux.) The other beef I have with PA is the Not Invented Here syndrome: instead of extending existing tools like ALSA or Jack, people just figured they could do everything better and start from scratch. So now we just have one more "standard" way of playing audio, good job Lennart.

The previous article was introducing basic streaming principles based on Icecast. The issue with this, of course, is lag and overhead of HTTP-based connexions. In this article we introduce RTP-based streaming system, (unfortunately) based on Pulseaudio and multicast. (Update: there was a rant here about Pulseaudio (PA) that I have moved out of this post because it's not what I was aiming to talk about. Those wanting to answer that troll are welcome to join the flamewar here.)

A word on multicast I took the opportunity of trying out Multicast (it's actually IP multicast, to be more precise), since I wanted to have only minimal configuration to perform on the client side. Plus, since the RTP frames are raw 44khz 16 bit audio, the traffic is actually quite heavy, 1.4mbps, to be more precise (punch in 44kHz * 16bit/Hz * 2 in the excellent Qalculate to see why), so I didn't want bandwidth to explode as more listeners joined in. As a comparison, also note that the Icecast Vorbis stream is around 64 or 128kbps depending on quality. Multicast allows packets to be sent only once for multiple listeners on the same network. It's actually quite cool, but doesn't work very well over the wider internet, for reasons that are still unclear to me. I presume that the fact there is virtual monopoly over traditional broadcasting systems (e.g. television and radio) means that multicast hasn't really kicked in yet... I hope this will make sense over the Montreal mesh, although the quality would probably need to be dialed down, or the audio streamed encoded instead of be sent as raw samples.

So why PA? VLC/pipe failures That aside, it seems that to stream through RTP, I have no choice. I have tried to use VLC as a pipe to stream data out of MPD as RTP, but somehow this config snippet just failed:

audio_output  
   type            "pipe"
   name            "RTP"
   command         "cvlc -q - --sout '#rtp dst=239.72.72.144,port=5004 '"
   encoder         "vorbis"                # optional, vorbis or lame
   format          "44100:16:2"
 

I don't know why - the multicast port wouldn't open properly and MPD would even hang when restarting. It seems the pipe plugin is somewhat broken. Note that the vlc command above actually works fine to stream audio content out through multicast, which is a good way to test if multicast works in the first place...

Shoving PA down MPD's throat Instead, I had to go to the clunky PA-way (also know as "my way or the highway"). The first step was to configure the default.pa to enable RTP streaming. I copied over the default /etc/pulse/default.pa file into ~mpd/.pulse/default.pa and uncommented the following lines:

# just to get the PA announced
load-module module-zeroconf-publish
# we need a null sink, no clue why, but without this glue, it fails
load-module module-null-sink sink_name=rtp format=s16be channels=2 rate=44100 sink_properties="device.description='RTP Multicast Sink'"
# port is specified otherwise it is random (?!)
# loop=1 is to broadcast locally too
# other useful setting: destination (defaults to 224.0.0.56)
load-module module-rtp-send source=rtp.monitor port=5004 loop=1

See the rtp-send module documentation for more information here. Then we need MPD to connect to that sink:

audio_output  
    type            "pulse"
    name            "Pulsaudio RTP"
    sink            "rtp"
 

Also, now that we got into the PA kitchen sink, we are kind of forced to use it for the local sound card output too, so we'll add this as a sink to:

audio_output  
    type            "pulse"
    name            "Pulseaudio sound card"
    sink            "alsa_output.pci-0000_00_1b.0.analog-stereo"
 

(Take a moment to admire the nice sink name. Yours may be different, have fun finding it in the output of pacmd list-sinks, but that works only once PA is started!) So now you need to restart MPD for those changes to take effect. Note that if the mpd user has already started PA, you're in luck: it won't restart it by itself, and you need to:

service mpd stop
killall pulseaudio
service mpd start

... yep. Basically because pulseaudio just detaches completely from the parent process when spawned automatically. Lots of fun.

Testing and listening Of course, you can listen to the stream using... Pulseaudio. I assume this would magically show up on the network thanks to avahi, if you run Pulseaudio on (say) your laptop. Since I still want to stay away from it, I would rather use a straight commandline tool to listen to the stream. According to this page, there aren't that many clients that could listen to such a funky stream. I chose VLC, since its commandline options actually make sense. Here's the magic mantra to repeat every night before you go to bed:

cvlc rtp://@224.0.0.56:5004

Yes, that's all. Drop the c if you want a GUI. With mplayer, it's more complicated - you need to specify a bitmask using a hexadecimal string (fun stuff):

mplayer -demuxer rawaudio -rawaudio format=0x20776172 rtp://224.0.0.56:5004

Yes, good old 0x20776172...

Results! We want results! All this work reduced the lag from 15 seconds to around 1 second. It's still present, and confusing to listen to, but maybe more acceptable. It seems most of the lag is in the vlc client buffering, when I enable debugging, I see:

[0x963c7bc] main input debug: Stream buffering done (1001 ms in 976 ms)

I was able to reduce that delay significantly by using the --rtp-caching option of vlc, as such:

cvlc --rtp-caching=48  rtp://@224.0.0.56:5004

The value is in miliseconds. Anything below 48ms introduces audible artifacts (probably buffer underruns). With that setting, there's still a noticeable delay, especially with isolated sounds that are naturally more noticeable. With more harmonious music, it sounds more like an echo. I have also tried to fiddle with similar settings with mplayer and simply failed to get any results. --no-cache simply drops samples like crazy. --cache 64 is the lowest I could push it down to, and it still has noticeable delay, which I couldn't measure, but I would assume would be around 400ms at least. Also note that contrarily to the previous setup I had, based on Icecast, VLC will automatically resume streaming from the RTP stream as it is stopped or started, which is also very nice - as it allows me to disable remote RTP streams easily and harmlessly.

Troubled areas Speaking of insane nonsense, here are a few pitfalls to avoid. Take care to disable the ALSA output, if it's not yet commented out, because MPD will freak out with the following error message:

Feb 03 20:25 : output: "Sound card" [alsa] failed to play: Resource temporarily unavailable

I have not been able to find a way to make PA let go of the audio sound card. I would have prefered MPD to open ALSA directly if PA wasn't needed (I need it only for RTP streaming), but it seems that PA assumes you'll want to play stuff locally. I have tried commenting out the following in default.pa:

load-module module-udev-detect

Seems like it's not enough. Also note that PA is designed with "dumb users" in mind (read: you need a GUI to operate it), so in the context of MPD, it's really annoying to debug things unless you can actually fire up a X11 session as the MPD user (which strikes me as a crazy idea) to configure PA properly with paprefs and all the graphical gizmos. I just gave up and went with PA all the way, and just added the RTP sink through the commandlines. Other guides can show you screenshots to clikety your way through inane graphical interfaces to do the above, if you are into that kind of stuff. With PA, you will want to get familiar with the pacmd and pactl commands. Why there are two different commands to basically do the same thing is completely beyond me.

Reference and more reading I got a lot of information in the Pulseaudio MPD wiki page and the Pulseaudio user documentation. The VLC manual about RTP streaming (server) (client) was also useful for debugging. The Pulseaudio Network documentation has a set of recipes for using pulseaudio for everything. This Ask Ubuntu question explains how to clikety your way through paprefs to essentially do the above.

I have been using MPD for a good while as my media player, in conjunction with gmpc as a graphical interface (since MPD is just a server). It's really cute, works reasonably well, and is one of the only software that can handle my huge music collection gracefully. The cool thing with the setup is that, because MPD is a network-transparent server, I can control (and listen to!) my music remotely.

Basic HTTP / Icecast streaming For a while, I have also been streaming my music through Icecast so that I can basically listen to my music from anywhere there is an internet connection with sufficient bandwidth. This configuration is simple enough:

audio_output  
    type            "shout"
    name            "Currently listening to..."
    host            "localhost"
    port            "8000"
    mount           "/radio.ogg"
    password        "hackme"
    quality         "4.0"
    format          "44100:16:2"
    description     "My playlist of the day. High quality (4) OGG/Vorbis stream."
 

This will make a stream available at http://radio.example.com:8000/radio.ogg, provided Icecast is configured properly. That setup is a bit more complicated than necessary, because it needs an icecast server to be installed (which is just apt-get install icecast on Debian, but still). So an alternative, simpler setup, but which may not scale as well to many listeners, is:

audio_output  
   type            "httpd"
   name            "My HTTP Stream"
   encoder         "vorbis"                # optional, vorbis or lame
   port            "8000"
   quality         "5.0"                   # do not define if bitrate is defined
   bitrate         "128"                   # do not define if quality is defined
   format          "44100:16:1"
 

I haven't played around too much with that plugin, so I don't know how well it works - I am using the Icecast plugin instead.

Always on radio Another cool thing is that I have actually two MPD daemons running at the same time - one is my regular player, but the other is the "shuffle" player. The basic idea is that when the regular player stops, the shuffle starts, and vice versa, so there is always something playing on the stream. The shuffle, by default, doesn't play on the local sound card so that even though the radio stream is up, it doesn't blast random music through my living room speakers all day (and night!). This trick is accomplished through a hook in the Icecast daemon:

<mount>
    <mount-name>/radio.ogg</mount-name>
    <fallback-mount>/random.ogg</fallback-mount>
    <!-- when I listen to my radio, I turn off the random MPD stream to save CPU -->
    <on-connect>/home/anarcat/bin/mpd-stop-random</on-connect>
    <!-- ... and when I stop listening, the other stream just kicks in -->
    <on-disconnect>/home/anarcat/bin/mpd-start-random</on-disconnect>
    <!-- i put those here to try to display something on the stats page when there's no client, but it's not working -->
    <stream-name>Currently listening to...</stream-name>
    <stream-description>My playlist of the day. High quality (5) OGG/Vorbis stream. If I'm not listening to anything, you'll be redirected to the random playlist.</stream-description>
</mount>
<!-- final fallback: the random radio, which is started when the above playlist disconnects -->
<mount>
    <mount-name>/random.ogg</mount-name>
    <stream-name>Random!</stream-name>
    <stream-description>Random songs from a 33 days playlist</stream-description>
</mount>

The mpd-start-random script is rather idiotic:

#!/bin/sh
export MPD_HOST=hackme@radio.anarcat.ath.cx
export MPD_PORT=6601
mpc play

You can guess what the mpd-stop-random script does (hint: the opposite of play is pause). So that's the old way of doing things. The one problem I found with this setup is when I want to setup another listener in the same house - the problem is that the Icecast stream has significant delay: a good 14 seconds here, with ogg123 as a client. Here's the special configuration for the mpd-slave daemon that runs the random playlist:

# same as main player
music_directory         "/var/lib/mpd/music"
playlist_directory      "/var/lib/mpd/playlists"
# those need to be different
db_file                 "/var/lib/mpd-slave/tag_cache"
log_file                "/var/log/mpd/mpd-slave.log"
pid_file                "/var/run/mpd/slave.pid"
state_file         "/var/lib/mpd-slave/state"
# avoids having to update the database by hand all the time
auto_update    "yes"
# disabled by hand after daemon is started    
audio_output  
    type            "alsa"
    name            "Sound card"
 
# main icecast stream
audio_output  
    type            "shout"
    name            "Random!"
    host            "localhost"
    port            "8000"
    mount           "/random.ogg"
    password        "hackme"
    quality         "4.0"
    format          "44100:16:2"
    description     "Random songs from a 33 days playlist"
 
# another player for low bandwidth situations, disabled
audio_output  
    type            "shout"
    name            "Random! (lowfi)"
    host            "localhost"
    port            "8000"
    mount           "/random.ogg"
    password        "hackme"
    quality         "0.0"
    format          "44100:16:2"
    description     "Random songs from a 33 days playlist"
 
filesystem_charset "UTF-8"
user               "mpd"
default_permissions     "read"
port    "6601"
mixer_type                      "software"
volume_normalization            "yes"

The random playlist is done by simply adding the whole library to the playlist. MPD chokes a little the first time you do that, but it's surprisingly fast. I also went through the playlist to remove spoken words and other stuff I don't want to stumble upon, and after I shuffled the playlist.

Final notes Otherwise, this is a fairly standard MPD setup, although there are some special configuration that vary from the upstream default:

# this needs to be commented out for the server to be accessible from everywhere
#bind_to_address                "localhost"
# setup a password for remote access
password                "hackme@read,add,control"
# make sure the mixer controls don't affect the local hardware mixer
mixer_type                      "software"
# we are beyond the ASCII world, but MPD doesn't know
filesystem_charset              "UTF-8"
id3v1_encoding                  "UTF-8"

This is the first part of a series of articles I intend to write about my MPD configuration, next ones detail the configuration of a dumb player and RTP streaming.

A short announcement to mark the release of the first stable OpenID provider for Drupal. Started by James Walker around 4 years ago, I have since then got commit access and taken care of the module for the last 2 years. And while it was not an easy task, with the help of the community, this module was able to mature to a fully featured, flexible and standard implementation during that time. One has to wonder however, is it too late for OpenID?

A bit of history Drupal was one of the pioneers in OpenID support. Back in 2006 when the protocol was still in its infancy (less than a year old!), Drupal was adding OpenID support to its 4.7 release through a contrib module and two years later, the Drupal 6.0 release was sporting OpenID in core, although not without significant interoperability issues. Still, we could say that rapid adoption by such a popular CMS as Drupal really boosted OpenID's adoption. At about the same time, Yahoo, Sourceforge, Myspace, Microsoft and Google would all join the OpenID bandwagon. On the OpenID "provider" side of things, things were not so great however. Back when I started experimenting with this, it wasn't actually that easy to host your own OpenID provider, to allow your users to login to other OpenID-enabled sites. While there are a lot of different libraries to provide OpenID support in your application there was not (and still isn't) any software that gives you an OpenID provider and is dedicated to that. No official reference implementation either, other than the code from the one of the founding companies of OpenID, JanRain. There was an OpenID provider module in Drupal, but there was no official release done until a two beta releases in 2010.

Koumbit's community involvement It is about that time I started getting involved in the project. Koumbit needed an easy way for our workers to log into multiple sites without too much trouble, so I started looking at OpenID provider implementations. The lack of dedicated software somehow made the Drupal implementation a natural fit, and I started experimenting with the module, filing bugs and reviewing patches, as there were a lot of interoperability and design issues with the provider. After a while, walkah was kind enough to give me full access to the project and I started doing regular releases (every few months), culminating in today's official 1.0 release for both Drupal 6 and Drupal 7, which I am very proud and happy about. While I am here, I want to send many thanks to the people behind the Drupal 7 port: paranojik, mikecar, xamanu and testing from neizod. A lot of issues were also patched, reviewed and tested by numerous contributors that are unfortunately difficult to track down other than reading the commit log. So we at Koumbit have been running our own OpenID provider for a good while now, and it has the key advantage of being connected to our LDAP directory (with the D6 LDAP integration module so it is, in a sense, a central point for authentication at Koumbit. Very often people don't clearly understand what their LDAP password does, and I can easily remind them "it's the OpenID thing", or "try your password in the OpenID site" if they think they have forgotten it. So much goes on in the web browser these days that it makes everything much easier.

The problem with Drupal's OpenID provider However, it is kind of embarrassing that it took so long. More than 6 years after the OpenID protocol was written, we have a stable OpenID provider. Now, this may be because I am overly conservative about stable releases, but I don't think so: the OpenID provider module had, for a long time, serious interoperability issues with OpenID sites. There was also a clear lack of features on the OpenID provider, a gap that various third party modules, now documented on the OpenID provider homepage are now trying to fill. There is also serious security issues with the protocol itself. The complexity and lack of clarity of the OpenID specification have rendered my work quite difficult at time, as the protocol has significant ambiguities and no clear reference implementation. OpenID is unfortunately renowned to be vulnerable to phishing and while Koumbit's use of the HTTPS protocol alleviates some of those issues (especially the redirect session hijack), tools like sslstrip can bypass those protections easily. The privacy aspect is also problematic, as the user is usually not free to choose which information from his or her profile is divulged to the site they are logging into. Of course those issues may be overlooked if the security trade off is not worth it. In the case of Koumbit, we use OpenID on a lot of client sites, but usually nothing critical, and we train our users to pay attention to the HTTPS icon when login in their OpenID account.

Are there alternatives? But unfortuantely, the reality is that there are not really any alternatives to OpenID out there. Mozilla's new BrowserID initiative (now rebranded Persona, boy those guys sure love rebranding..) is interesting, but actually has even more flaws than OpenID itself. Not only is it not resolving the phishing problem, it actually creates new problems, including my personal favorite, trusting server-side Javascript code, which OpenID is not using that much. I should also mention the Shibboleth module which has a surprising number of installs recorded on Drupal.org, but that depends on the much more complicated Shibboleth federation, for which there is no server implementation for Drupal, from what I understand. Oauth, on the other end, is even more widely used, and has server-side components like Oauth Login Provider and Services 3.x. We could also simply give up and use our souls stored at Facebook, Google and Twitter for authentication everywhere. Maybe OpenID opened the door for those corporations to see that there is a real opportunity in leveraging user's profile to track their online activity. It is not something that OpenID was necessarily designed for but it's an idea that certainly permeates Facebook connect and "like" buttons or Twitter and Google online business strategies, where you are the product sold to advertisers. I personally believe we should keep the struggle for distributed and standard identity mechanisms for the web that respect users' privacy. With Drupal's OpenID Provider, it is possible to create a secure, privacy-aware identity provider, although I still have to perform that damn security audit. Experiments with those third party modules should also be encouraged and I welcome any feedback on those modules. While I do not plan to attend OpenID conferences or do significant development on the OpenID provider right now, I will certainly make sure Koumbit keeps its commitment to maintaining stewardship on the OpenID provider module in Drupal. I encourage people to try it out, submit issues and patches to make it even better. Long life to OpenID! PS: if people are interested in funding that security audit, I would be happy to perform this through my regular work at Koumbit.org, just file a request at services@koumbit.org.

This is a long story. I wanted to document which tools I used on my desktop, but then realized it was difficult to do this without an historical context. This therefore, became a history of my computer usage and programming habits.

Before the desktop: vic 20 and apple My first contacts with computers were when my uncle showed me his Apple computers at his home. He was working at UQAM and had access to all sorts of crazy computers. My memory is vague, but I believe he showed me a turtle moving around the screen, which I suspect was Logo, and some rabbit game which I was quite fond of. Later, I hooked up with a Vic-20 (similar to a Commodore 64) and typed in my first programs. Back then, you would painstakingly transcribe the program from a book into the screen, type RUN at the end and really hope that you didn't make a typo. You could also read and write data off audio tape with an external tape drive, which was really amazing. To put things into perspective, the Vic-20 would plug into your television for a display, and the computer was "in the keyboard". Old school shit. The language was BASIC, which was awful. Later, we got luxurious and had the privilege of a Macintosh Plus computer. 3000$ plus the printer, that was quite something. Still, that modern system could run Word, had games, and a 9" black and white screen. But the best part, and where I really started to learn to program, is with Hypercard, the crazy software written by Bill Atkinson that became the inspiration for wikis, the Myst computer game and, to a certain extent, popularized the concept of hyperlinking, which is the basic principle of the World Wide Web.

The early years: afterstep and FreeBSD But my real computer life all started with FreeBSD 2.2.x (probably 2.2.5), around 1997, when I installed my first FLOSS operating system. This was just amazing: source code for everything, down to the compilers themselves, I was amazed, since I was coming from a Mac (still had that Mac Plus lying around) and Windows (95!) world. Back then, X11 (XFree86, it was called) was a fantasy. Something I would compile with the FreeBSD ports system and sometimes would be able to run. Window managers were some weird things I wasn't sure how to use, but I figured out twm (yuck!) but was still amazed. At the university, we had SGI workstations (remember the Indigo machines?) on which I started setting up the Afterstep window manager, which I used for a long time. The web browser was Netscape Communicator 4, and lo and behold, it also did email, although I mostly used Pine over a modem terminal to the university. Internet access at home was spotty and was dependent on students compiling discreetly copies of slirp which were allowing us, in a really clunky way, to run a PPP session over those terminal lines. That actually got my account suspended because I messed up my sendmail configuration and was sending emails that appeared to be coming as , which freaked out the sysadmins that be over there. The hardware was a Pentium 166Mhz with 32MB of ram and 20GB of disk space, which was a huge improvement over the 1MB of ram and 20MB of (external!) hard disk that the Mac Plus was proud of. By then, I started programming in more languages: I learned the basics of shell scripting, C programming and I had learned Pascal in a college class.

The real freedom: Debian and... enlightenment? I then switched to Debian, thanks to my friend scyrma, which showed me the power of dselect which, for a FreeBSD user, was really impressive. Imagine that: you could just install software and all dependencies without compiling anything! ;) I don't exactly remember when that was, but it was probably Woody. I probably still used afterstep, but at that time, changing window managers was like changing t-shirts and we would waste countless hours fiddling around with borders and background images and other silly things. I remember running Enlightenment and writing software for it, for example, but not much else. Back then, email was still over pine, but a new web browser was rearing its ugly head, Mozilla, and was quickly trimmed down to Phoenix, shortly lived as Mozilla Firebird, only to be finally renamed Mozilla Firefox, and was unleashed as a stable release to an unsuspecting Internet Explorer and Microsoft-dominated world. Throughout this chaos, I also used various Mozilla derivatives like Galeon since all of this was pretty unstable and heavy for my poor computers. This was probably around 2002 to 2005. By then, I had formal classes on programming languages, was forced to learn Java, discovered C++, and learned more C as the university assumed that you knew that language in the data structures and algorithm class, which some students really didn't find funny at all. I also learned some Prolog and Haskell back then. Through my experiences with free software, I discovered various eclectic programming languages like TCL, and kept an eye out for other languages like lisp (thanks to Emacs), Forth (thanks to the FreeBSD boot loader), HTML/CSS/Javascript (no thanks to the web). There are probably others I am forgetting here, voluntarily or not.

The years of the desktops: rox, sawfish and gnome Another stable increment on my desktop was the use of the sawfish window manager and the rox desktop, mostly because of the awesome Rox file manager that had no equal in the rest of the free software world. Gnome was an atrocity and KDE even more so: bloated and unstable. Eventually, however, KDE and Gnome not only matured, but simply took over the whole of the graphical environment psyche of the free world. People nowadays don't even know what a window manager is and assume that you now have a "desktop environment", which, for me, is a real desk with a bunch of papers, a radio transceiver and maybe a computer with a screen lying around. So I did use KDE, and then Gnome. I switched from KDE to Gnome because I felt that Gnome left me more space to run my own stuff without starting crazy daemons to preload everything. At this point, Firefox was the new thing and basically stable enough for daily use. This lasted until around 2009, I guess. On the programming side, I had picked up Perl because I had to learn it to work at Ericsson (don't ask) and learned PHP to pay my rent, which I consider one of the biggest professional mistakes I have made so far in that story. It is around that time that Koumbit was founded by me and other friends.

The tiles fall on the gnomes: awesome desktop and sysadmin programming At some point, friends (mostly comar and millette) introduced me to wmii as the best thing since sliced bread. I banged my head against it with great pains, but I really enjoyed the hands off approach to window management. Tiling window managers really got a hold on me there, and in fact we can say on the whole desktop, with recent Gnome 3 taking up some tiling concepts itself. I have since then switched to the Awesome Window Manager which is probably the worst name you can use for software (try googling for "Awesome crash" to get an idea). I have switched from Firefox to Chromium, after having repeated crashes from Firefox on a site that Chrome would load fine. I was expecting Chrome to crash only one tab, because of its design, and it turned out to not crash at all, and I have since given up on Firefox totally. I have also given up on desktop environments. While I have given them a chance in quick setups and still use them when forced to, I find them too bloated and single-minded for my needs. I want control over my desktop, and those tools work very hard to remove control and hide it behind user interfaces that treat me like a brain-dead dog. Finally, I have kept on programming in PHP for work, because that's what we do over there (unfortunately). Mostly because of Drupal and Aegir, in which I was and still am deeply involved. But I am quite fond of Python for my personnal project and still use shell scripting to get myself out of trouble or automate data processing from time to time. I have a rule though: once a program hits 100 lines of shell scripting, it needs to be rewritten, usually in Perl because the syntax is closer. During the years 2009 to 2012, I have started to work more and more on system administration however. In 2010, I became a Debian Developer and a lot of my work at Koumbit revolves around system administration, optimisation and maintenance. I work oncall and sometimes get called during the night to nurse servers (which is why I am up writing this right now).

The future of programming I have become seriously disgruntled with programming. I have seen the emergence of the web, where programmers felt it was relevant to rewrite 30 years of programs into the web framework (webmail? wtf?), and now we are seeing the same thing happen with mobile devices, the tracking devices they call phones and "pads", we should rather say. And again they are trying to make people pay for trivial software. I am tired of seeing our community reinvent the wheel every 10 years because of infatuated egos and marketing sprints. We are wasting great minds on this, a lot of energy and precious time. I would rather maintain 5 year old imperfect software than coordinate a 5 year project to rewrite that software into another imperfect software. That is what is happening right now, and it's going nowhere. So here's the current state of affairs, some perls, some tiles, but mostly system administration and less programming. Through all this time, the common pattern is that I have learned to solve problems and document my work, which are key concepts of the job of a system administrator, which is what I really am now. Somehow, that popped into my life like a natural fit. Maybe, one day, I will retire and become a shepherd in far away mountains, but I will always have in my mind the strong desire to fix shit up and make things right with the world. Maybe get involved in a project like Open Source Ecology and especially their Global Village Construction Set to make sure humankind survives all that stupidity.

It seems I am still having problems with spam on this site now. Instead of getting tons of spam comments posted, I get false positive, that is legitimate content marked as spam. First off, I wish to apologize for the people that haven't seen their comments published immediately. Next, I will work on improving this and explaining the current situation. Here are the statistics that the spam module gives me:

• prevented comment spam: 100 (27% aka positives)
• marked comment as spam: 242 (65% aka positives)
• manually marked comment as spam: 3 (0.8%, aka false negatives)
• marked comment as not spam: 17 (5% negatives)
• manually marked comment as not spam: 8 (2.2% false positives)
• total: 370 comments

Another way to put it:

• spam: 93%, 99% of which is automatically detected
• ham: 7%, 32% of which is marked as spam

In other words, the filter works very well at marking spam, so well in fact that it also marks non-spammy comments as spam... Which is a shame because it breaks the interactivity of the site. Basically, you have one chance out of 3 of not seeing your content posted online, which leads people to try to submit it three times, which actually gets them blocked even more. And this is only the spam module filters: the CAPTCHA module tells me it's blocked 26535 attempts (26 thousand) - amazing. So overall, the solutions I put in place a year ago work, somehow, but are too aggressive. I have therefore bumped the threshold from 65% to 80%, hopefully that will help with this. I have also updated the Honeypot module since they have done lots of releases since 1.5, fixed the issue I reported and it's now in use on Drupal.org after some tweaking by our venerable webmaster killes. If this fails, I will look again at other solutions, like the blogspam plugin that relies on the communnity-run blogspam.net site which powers, amongst other things, Debian Administration and Ikiwiki sites. Back then, I also was considering the Bad Behavior and Block Anonymous links modules. (Note to self: I have also removed around 150 comments sitting in the spam module moderation queue, that it didn't seem to remove as it should have done.)

It's actually quite simple, and this can be used to also track any package that was installed on your system from a different "origin". The "origin" field is defined in the mirror's Release file and represents where the packages come from, be it Debian, Ubuntu, or deb-multimedia (example: here is Debian Wheezy's official Release file). You can check for non official packages in Aptitude using this syntax in the search field, triggered by the / ("slash") key:

!?origin(debian)

en lire plus

I got in late last night, and I've been trying to get caught up on all the various aspects of my life. I spent some time on the plane writing up my RecentChangesCamp 2007 notes, which aren't yet done. But I hit the ground running today. All charged up from a weekend of wiki talk, I decided to exercise some DoOcracy and organize this month's Wiki Wednesday here in Montreal. It wasn't really that hard -- I just made sure that the great meeting space at La Bande-passante was available, then updated the Montreal Wiki Wednesday page and sent out a link to the RoCoCo discussion list for good measure. I'm pretty excited to start discussions about RoCoCo, the Montreal RecentChangesCamp coming up May 18-20. We're about 3 months out, and although Anne and Antoine have done a ton of prep work, we still have a lot to do to get ready. tags: wiki wednesday wiki recentchangescamp rcc07 montreal bande-passante rococo

Wiki Wednesday Portland Speaking of WW, I find it hard to believe that Portland, the so-called "Wiki City", doesn't have its own Wiki Wednesday event. I'm sure everyone there is burnt out right now, but it'd still be really useful. tags: portland wiki wiki wednesday

OpenID and CardSpace So, it seems like everybody and their dog is nattering about Bill Gates's announcement that Microsoft will be supporting the OpenID single sign-on solution in some future version of CardSpace. This is probably quite good for people who are using Microsoft desktop platforms, and obviously very good for those of us who already have OpenID deployed (like Wikitravel does). But I wonder if Windows Live ID users will be able to fully participate in the already growing world of OpenID-enabled sites. It would be cool if a Live Spaces URL, like http://evanprodromou.spaces.live.com/, it's possible someone smart is going to create a Live-ID-to-OpenID bridge, like idproxy is for Yahoo IDs. tags: openid microsoft live id cardspace idproxy bill gates live spaces

Hey, somebody used a photo I took of the Papineau mural in Papineau metro station in a discussion of the prison Pied-du-Courant. They even gave me proper attribution, per the license under which the photo was published. As an American I have a shamefully bad understanding of Canadian history. Example: when I first moved to Montreal, I had to get my local land-line phone service set up. When I was asked for my address (in English -- I didn't even try doing this stuff in French back then), I answered: "Avenue Papineau. That's P-A-P-I-N--" the operator cut me off. "I know perfectly well how to spell 'Papineau', sir," he said, icily. Oops. I'm trying to get caught up. I've been reading a ton of wp:Pierre Berton books, but I still don't have the whole story as integrated into my own mental processes the way I know, say, American history almost instinctively. Like, even though I know the story, I continue to boggle that wp:Louis Riel keeps popping up in various apparently unconnected historical contexts and era. It's like, "Wha? Louis Riel again? Didn't they shoot him four or five chapters ago? Why is he still here?" I guess it just takes time. Or maybe it's something I'm never going to understand, like why there were two teams called "Roughriders" in the CFL for so long. I dunno. tags: papineau flickr louis riel bell canada me

Talk about your busy days Busy, busy day today. Alain Desilets, a researcher at the National Research Council and a really smart person when it comes to wiki, came to Montreal today for Wiki Wednesday. We had lunch with Alain and fellow researcher Julie at nearby Ty Breiz restaurant, a neighborhood landmark that makes fine crepes and cider and good garlicky salads. We talked about improving wiki interfaces to make it easier to translate between multiple language versions of a page (like you see on Wikipedia and Wikitravel), and came up with some interesting solutions. Then it was off to Wiki Wednesday proper, at which Alain presented his translation projects and also some collaborative fiction ideas for children. Very cool stuff! It was a real celebrity event: Sunir Shah of MeatballWiki, Sebastien Paquet of SocialText, Marc Laporte of Tiki CMS/Groupware, Anne and Antoine, organizing the RoCoCo 2007 event, and lots of other folks too. Anne and Antoine also presented their brilliant image-only wiki, Wikigraphe. It lets people upload images as a replacement for pages, then build "hotspots" on the images that link to other pages -- which are in themselves images. It's a destruction of the tyranny of text in the wiki world -- and fun to watch. Sylvain Carle compared it to drawball, but I think they're different. Much followup talk on RoCoCo 2007. It's going to be an exciting event -- so glad to have wiki friends around the world coming here, to my favorite city. Lots to do between now and May, but I think we can get it done and have a fantastic wiki (and wiki-like) event. Wiki Wednesdays are such excellent events -- kudos to Seb Paquet for organizing them in Montreal, and also to SocialText for such a good idea, and Robin Millette and gang for hosting. tags: wiki wednesday alain desilets ty breiz sunir shah seb paquet sylvain carle anarcat marc laporte rococo

CC Salon Montreal On top of that, we had the Creative Commons Salon Montreal tonight at Am re de boire on rue St. Denis. The Salon is developing internationally as a way for CC-related people to interact, chat, and have fun in cities around the world, and Montreal has too many CC folks to be left out. Tina Pipers, co-chair of Creative Commons Canada, is on the faculty at McGill, so we're lucky to have such a heavy hitter in the area to chat with us. (Also, I found out tonight she's Greek! Good for us.) Tonight was expecially great since Maj and I talked about Wikitravel and our new project, Crossroads, to create Open Content travel opinion, stories, blogs, photos, etc. It's just getting off the ground, but it was great to share our vision with some people who really understand the idea of content sharing and an information commons. The discussion after our talk was great. I couldn't drag myself to YULblog, though, after 6 hours of talking... I had wanted to go to get people interested in BarCampMontreal, but Sylvain (one of the event organizers) was already going, so I felt justified in sitting it out. But when am I going to get to another Yulblog? I just don't know. tags: cc cc salon cc salon montreal tina pipers wikitravel crossroads travel barcampmontreal yulblog

Searls ha ha Searls's 4th Law: No matter what car you want to rent, what you'll get is a Chevy Cavalier. Man, I love the wp:Chevrolet Cavalier. It's a fun car to drive, and it just says "rental car". Which mean, y'know, "vacation" and "distant lands" and "irresponsibility". How can you not love that car? Not to mention that someone has to straighten out Doc Searls on Elements of Style Rule #1: Form the possessive singular of nouns with 's. This includes singular names that end with s... like "Searls". There are a few exceptions for ancient names ending in -is/-us/-es, but those are few. To paraphrase: I knew Achilles; I served with Achilles; Doc Searls, you are no Achilles. tags: searls doc searls chevy cavalier elements of style

UFO Beliefs Hey! I'm back on Technorati. Rockin'! After like 3 months complaining to their customer service Web form and getting 0 (= zero) response, I finally canceled my blog claim and started a new one. The software picked up on my site right away. Problem solved! It would have been nice to hear, you know, one tiny peep out of Technorati about my 8-10 bug reports, but I guess they're pretty busy. tags: technorati listing bug reports