When I started looking for a lightweight solution of serving a music library over LAN I did not expect so many complications. I expected it not to be a unique need to have something running on a SheevaPlug straight from the Debian repository. Apparently it kind of was. Debian used to have mt-daapd (popcon: 165), but now it is available from oldstable only and upstream is dead. There is tangerine (popcon: 98) with its Mono dependencies and GUI which seemed to me overkill and more like a demo of a networked application written in Mono than a music library server. The most promising candidate was forked-daapd (popcon: 220) but it was far from being a true winner. First, it had a series of dead upstreams. At the beginning it was forked from mt-daapd (hence the name) by Julien Blache who also served as the prior Debian maintainer. Then the code base was forked and converted to use Grand Central Dispatch. Then the GCD fork died off slowly as well a few years ago. When I found the package it had been unmaintained for a few years and was based on the GCD branch which prevented building it on many architectures and the server itself was crashing or quitting occasionally. Luckily there still existed a fork thanks to Espen J rgensen which was well maintained and could serve as a way out but examining it closely it turned out that it had switched to libevent from GCD but to a version (1.4) which is present only in oldstable! And some say Debian s software versions are ancient ;-). Moreover it was not simply libevent 1.4-based, but it included some heavily patched parts of it. Espen also liked the idea of packaging his version in Debian and we extracted the patches to libevent and slowly got them accepted to libevent s master. Forked-daapd s master works best with libevent 2.1.4-alpha, but thanks to Espen the development branch now also works with libevent 2.0.x giving up some performance and a little feature. This was a long journey, but finally Espen s forked-daapd became ready for being used as a new upstream of the Debian package thus please welcome 20.0+git20140530+gc740e6e-1, the first version of forked-daapd building on all architectures for a very long time and a prime candidate for being the music library server in Jessie (and wheezy-backports, soon)! Testing, bug reports are always welcome! From the package description:

 forked-daapd is an iTunes-compatible media server, originally intended
 as a rewrite of Firefly Media Server (also known as mt-daapd).
 It supports a wide range of audio formats, can stream video to iTunes,
 FrontRow and other compatible clients, has support for Apple's Remote
 iPhone/iPod application and can stream music to AirTunes devices like
 the AirPort Express.
 It also features RSP support for Roku's SoundBridge devices.
 Built-in, on-the-fly decoding support enables serving popular free music
 formats like FLAC, Ogg Vorbis or Musepack to those clients that do not
 otherwise support them.

With this release come two long-awaited improvements:

• database speedups;
• fix for iTunes timing out after 30 minutes.

Release 0.18 is a small update to forked-daapd, adding a config knob for the ALSA mixer channel to use for volume setting; this is required in some setups. Starting with this release, the forked-daapd logfile needs to be owned by the user forked-daapd runs as. If you use logrotate, you want to add the create option to the config stanza for forked-daapd. The GCD codebase received a build fix for FreeBSD and I m told it works nicely there too. Tarballs available at http://alioth.debian.org/~jblache/forked-daapd/ or in the forked-daapd project on Alioth; GPG signatures made with my (new) Debian key FA1E5292. I m trying to transition the git repository to the forked-daapd project too, but I can t get it to show up under the SCM tab in the web interface nor does it show up in gitweb. Still waiting for a resolution on that.

After months working on getting the cluster up and running via proof-of-concept, demos, tests, meetings, negotiations and the like, the press release is now out! We ve got that big thingy running and it s doing pretty well. Been there, done that, waiting for the t-shirt any moment now (hint, hint) :)

just how many Space Shuttle Programs can you fund with the insane amounts of money Europe is about to spend to pull Greece out of the hole they ve dug for themselves? Same for the countries that will undoubtedly follow the same path in the coming weeks. Yeah, fuck that. Point is: there are so many better ways to spend that money (that we don t have anyway).

Well, today was the day and I ve just watched the last ever Space Shuttle launch. Let s just say there s something very, very wrong with this world and leave it at that. Hats off to all the men and women who dedicated their lives or part thereof to these magnificent birds over the past 30+ years, from the original design studies back in the days to the transition & retirement operations that will last for a year still. We ll most probably never get to see any other machine pushing boundaries like the Shuttle has done, and this is a gigantic loss for us all. One day people will realize If you aren t familiar with the challenges faced by the Shuttle, with the advances in science and countless engineering disciplines brought by Shuttle and more generally with Shuttle and facts about the Shuttle, go read Wings in Orbit; a book by the people who built and operated the Space Shuttle, available freely in PDF format from NASA. Did you know that the Space Shuttle Main Engines (SSME, RS-25) are the most powerful, most efficient engines ever built? They were designed 40 years ago. I won t make a list, but this is true of a number of other components. What the hell have we been doing all these years?

forked-daapd v0.17 is out; the main purpose of this release is to add support for libav 0.7. There are also a few bugfixes that made it into this release, but nothing earth-shattering. One More Thing You ll notice an extra tarball and tag in this release, both marked 0.17gcd. With this release, the GCD (Grand Central Dispatch / libdispatch) codebase is making its debut. libdispatch replaces libevent and allows for greater concurrency inside forked-daapd. This makes forked-daapd snappier, which shows when using Remote or a SoundBridge. The filescanner also benefits from this; it now scans several directories concurrently, reducing the time needed for a rescan by making better use of the machine resources. Thanks to Mark Heily s efforts on libdispatch/libkqueue/libpthread_workqueue over the past year, we now have a Linux version of libdispatch that works well enough to support forked-daapd. This codebase will become the primary codebase for forked-daapd once the database performance issue will be fixed. After this point, the current libevent 1.4 codebase will be legacy and won t see any further development (we ll discuss bugfixes in due time). This new codebase comes with new requirements:

• the Clang compiler from the LLVM project is required, due to the use of the C Blocks extension that isn t supported by GCC;
• libdispatch, of course, is required;
• libtre is a new dependency for concurrent regexps;
• glibc 2.13+ is required, or any glibc version containing a fix for glibc BZ#10563.

libdispatch and its dependencies are available in Debian unstable; it s best and easier to use whatever is in unstable, instead of mixing and matching the libdispatch/libkqueue/libpthread_workqueue versions until they all work together. Due to platform support in Clang, the GCD codebase can only be built for and will only run on i386 and amd64, as far as Linux is concerned. Other architectures will have to wait until support for them is added to Clang. FreeBSD is untested and will probably require some modifications to work; recent enough versions of FreeBSD have a native pthread_workqueue implementation and do not need libpthread_workqueue. Packages are available in experimental for Debian users; they ll end up in unstable in a few weeks, before the libevent2 migration happens there. Tarballs available at http://alioth.debian.org/~jblache/forked-daapd/ ; GPG signatures made with my Debian key F5D65169.

I ve just released pommed v1.39 ; this is a maintenance release for Linux 2.6.39, following the renaming of the mbp_backlight driver to apple_backlight.

The Alioth admins have been hard at work over the past few days to migrate Alioth to a new configuration split over a pair of servers. As a consequence, personal git repositories aren t working anymore for the time being. So, don t panic, the git repository will be back soon, one way or another.

forked-daapd v0.16 is a bugfix release, mainly addressing a bad regression introduced in v0.15. Fixes:

• DAAP songlist off-by-one with sort tags
• handling of non-UTF-8 tags under some circumstances
• file size in HTTP streaming

The DAAP bug is what causes an incorrect display and sorting in iTunes, the tags issue could cause a segfault in some cases, and finally fixing the file size in the HTTP streaming code may or may not fix some issues but was a bad bug anyway. Tarballs available at http://alioth.debian.org/~jblache/forked-daapd/ ; GPG signatures made with my Debian key F5D65169.

forked-daapd v0.15 is here! The database speedups did not make it into this release as they still need some more work. Now let s talk about what did make it into this release: AppleTV metadata support. When streaming to an AppleTV, you ll now get the cover art, artist, album, title and playback position displayed on your TV. This was made possible by a forked-daapd user who offered me one of the new AppleTVs, so thanks much to him! Although we haven t got the database speedups this time around, we ve got some other speedups:

• the code generating the song list was slightly reworked;
• hash tables of static data were replaced with static hash tables generated by gperf;
• the artwork code can now send out both PNG and JPEG to clients.

The first two items bring in a new build-dependency: gperf. I promise it s not some Java-based monstrosity ;) It s small and C++, so no issue getting that running on whatever your platform is. Pre-generated files are not provided for this. The last item means if your cover art is mostly JPEG, forked-daapd won t be spending time turning it into PNG before sending it out to the client. That s some CPU time that can be put to better use. Next release: when it s ready! For Squeeze users, I am providing backports on backports.debian.org; due to the update policy, they are lagging behind the releases. Tarballs available at http://alioth.debian.org/~jblache/forked-daapd/ ; GPG signatures made with my Debian key F5D65169.

Due to cockups and/or willful sabotage by the telcos my ISP relies on for its DSL services, I am unreachable by email starting early this morning (UTC) and until further notice. My primary MX is off the net, mail is spooling on my backup MX so nothing should be lost. <expletive deleted> Update 2011-04-05: stupid telco finally rebuilt the line properly. I m back, no mail lost at first glance and I m slowly digging into that pile of mail. If you don t receive a reply by tomorrow, please resend.

forked-daapd v0.14 is available. This release includes a number of improvements to the sort headers/sort fields handling by Kai that got entangled into a larger database revamp at the end of last year. That will be the subject of the next release. Also in this release:

• better handling of tags for TV shows
• better handling of DRM-afflicted files
• configurable IPv6 support
• fixes to the URL file scanner
• misc fixes

This release performs a heavy database upgrade process that dumps and reloads the files table; this can take some time on big libraries, so please be patient while this process takes place. You should backup your database file before upgrading, just in case. See you in about two weeks for 0.15 with some database speedups that will finally see the light :-) Tarballs available at http://alioth.debian.org/~jblache/forked-daapd/ ; GPG signatures made with my Debian key F5D65169.

pommed v1.38 is out and adds support for the Early 2011 MacBook Pro machines:

• MacBookPro8,1: 13
• MacBookPro8,2: 15
• MacBookPro8,3: 17

These machines have a new keyboard/trackpad assembly, the WellSpring 5. You ll need a patched kernel or a kernel recent enough that it has the USB HID quirks for the new assembly; that s 2.6.39-rc material. This release also adds a new config item that allows disabling the audio support in pommed entirely. The audio hotkeys are fairly well supported in desktop environments nowadays so you ll probably want that if you re using one.

forked-daapd v0.13 is available! This release has been in the making for a long time, got delayed for a lot of reasons and had its content changed a few times along the way. Nonetheless, it s here now and if all goes to plan at least a couple releases should appear in the coming months, say before the summer. In this release:

• support for Remote 2.0
• support for per-speaker volume control
• support for RAOP retransmission
• improved compatibility with the 802.11g AirPort Express
• support for sort field metadata
• support for JPEG artwork and more alternative filenames
• fix for Murmur hash on 32bit platforms
• improved mDNS address resolution, making IPv4 usable again
• support for FFmpeg 0.6 (moving away from deprecated APIs)
• DAAP session expiration has been disabled
• misc fixes

Thanks to all the contributors that sent bugs and patches for this release! I know development doesn t go quite as fast as some would like, but keep in mind that I m doing this in whatever free time I have and that the public git tree doesn t tell the whole story. There are several parallel feature branches you don t get to see, for instance. Hopefully 0.14 will appear in a few weeks. Tarballs available at http://alioth.debian.org/~jblache/forked-daapd/ ; GPG signatures made with my Debian key F5D65169.

Long time no see! I ve just packaged Digi s latest dgap beta release, dgap 1.3-21, for Squeeze. The packages are available at the usual address, using this APT source: deb http://debian.technologeek.org squeeze non-free I have made some changes to the packaging as far as udev and firmware loading is concerned, but this is nothing major. Firmware files moved out of /usr, along with udev-related utilities. Should you notice any issue with the packages or the APT repository, please let me know. The previous packages, version 1.3-15, are still available for Lenny, APT source unchanged: deb http://debian.technologeek.org lenny non-free

A few weeks ago, I ve had to look into getting a QLogic QLE7342 InfiniBand adapter working on a Debian Squeeze system, with OFED 1.5.2. This post will probably save quite some time to anyone trying to do the same; it applies to all the QLogic adapters supported by the ib_qib kernel module. Note: on the ibverbs side of things, the adapters are supported by the ipath plugin, just like older QLogic adapters (that use the ib_ipath kernel module). First of all: grab a recent version of the ofa-kernel package. The ib_qib module in ofa-kernel from OFED 1.5.2 did not work at all for me. I used the ofa-kernel snapshot from 20110203. There are two things to know about the QLogic adapters:

• they don t autodetect the fabric speed by default, a dedicated utility has to be used to set the speed(s)
• the driver exports a sysfs-like pseudo-filesystem, ipathfs, that needs to be mounted

QLogic offers a complete IB stack based on OFED and dubbed QLogic OFED+ . The complete package weighs in at over 500 MB and contains the QLogic-blessed drivers, OFED stack, libraries and, of utmost importance to us, the QLogic-specific utilities. Unfortunately, as simple as the QLogic utilities are, they don t come with source. The package also only exists for RedHat 4 and 5. So, go to the QLogic support website, select your hardware and download the QLogic OFED+ Host Software for RHEL 5 package. In the tarball, one directory contains the OFED stack and another contains all the utilities (QLogic-Tools.*). For setting the adapter speed, you ll want the iba_portconfig utility along with the initscript (shipped as iba_portconfig.sh). The desired adapter speed is set in the initscript by choosing the proper arguments to iba_portconfig (-s 1 for SDR, -s 2 for DDR, -s 4 for QDR or any combination thereof). The iba_portconfig initscript must run after the drivers have been loaded. Which brings us to loading the driver and mounting the ipathfs. This is all handled by the openibd initscript provided in ofa-kernel (under ofed_scripts/) and its companion script dedicated to QLogic adapters, truescale.cmds. This initscript will load the OFED stack and drivers; if a QLogic adapter is present, it ll mount the ipathfs in the right place. Voila, once this is done, the adapter should happily report itself ACTIVE/LinkUp.

I ve got a few mails lately enquiring about the lack of visible activity on forked-daapd in recent times. Well, for one, I ve been busy with other things, but there s also been some non-public activity. So, here s what should happen in the next few months, in no particular order:

• overhaul of the database schema and code;
• metadata handling will (finally, probably) move away from ffmpeg;
• release of the libdispatch-based forked-daapd code.

The database overhaul will bring a much-needed speedup on small devices and big libraries. It s being worked on, but it ground to a halt with new year and all. Hopefully it ll resume soon. I m also giving up on ffmpeg for metadata extraction. It doesn t work well at all (as everybody knows by now) and it s actually getting worse and worse. With the ffmpeg takeover/fork (call it whatever you want) it s unclear what s going to happen on this front. So I m giving up and the metadata scanner will probably switch to using TagLib (one more dependency, yay). Last, but definitely not least, the libdispatch codebase should finally see the light at the end of the tunnel in the coming months. I ve been working on this since July last year, moving away from libevent. If you never heard about libdispatch, or Grand Central Dispatch, read my previous post on the subject. I ll talk about it in more details at a later date. Finally, as far as Squeeze is concerned, I plan to provide forked-daapd backports as long as it ll be practical to do so. This is it for now. I don t have a one more thing in my sleeve, sorry. Time is at a premium right now, so big things have to wait.

pommed v1.36 is out and brings support for the MacBook7,1, along with a few fixes and updates. A typo in the pmac keyboard backlight has been fixed, so fading now works there too. Product IDs for the ANSI and JIS keyboard variants for pmac machines were also added. It s been like forever since I last touched that code; not a lot of people still use those (great) machines. This release also contains a small update that will make it compatible with future kernel releases as applesmc will no longer have a fixed a name and location in sysfs. Update: I just noticed a bug in 1.36, so 1.37 is out now. It also contains two patches that were submitted through Alioth.

As part of my work for EDF, I ve had to package and integrate a set of tools for performance measurement and profiling of HPC code. This toolbox comprises tools for analysis of both sequential and parallel codes, MPI communications profiling and, of course, visualization frontends. Without going into too much details, here s the list:

• OpenSpeedShop: a complete performance and profiling workbench, including I/O and MPI
• PerfSuite: a relatively simple and easy to use performance analysis toolkit
• TAU: a complete performance analysis framework including automatic instrumentation with PDT
• Scalasca: a tool for performance optimization of parallel codes, including MPI communications
• a number of dependencies: slog2, Open Trace Format libraries, VampirTrace, dyninst, monitor, perfctr, pfmon, PAPI,
• visualization frontends: paraprof, jumpshot4, cube3,

If you ve been anywhere near HPC code, the names probably ring a bell; they re the best tools out there in their category, developed and used by the top laboratories. Over the past year, all the tools have received some level of testing, meaning we know they do work at the very least to some extent. As you can imagine, testing such tools is no easy task and takes an insane amount of time and resources of all kinds. Testing is all the more important that I had to produce a number of patches to integrate the tools properly in the distribution and, in some cases, to even get them to build. Now, we would like to share this work with the HPC community in and around Debian. How exactly we are going to do that isn t clear just yet; most probably, we ll end up building a team with other interested parties and offer our packages as a base to build upon. There are a number of challenges with these tools: they re not easy to build, they re not easy to maintain, they re not easy to use, they re not easy to understand. Basically, nothing is easy. Some of those tools were never meant to be packaged and integrated in a distribution and no sane amount of patching will fix that, so we have to live with packages that aren t quite as polished as we like them to be. And then, there are licenses. Some tools are non-free due to usage restrictions. Others rely on non-free dependencies. Although I ve been looking at the licenses, a thorough license check will be required and decisions will need to be made. It s not for the faint of heart! If you are interested in these tools and in bringing them to Debian, please get in touch. I ve also had to package the Apache Derby database (Java); if someone out there cares about Derby, I d be more than happy to provide my packages as a start base for getting Derby into Debian. The packages need some work by someone who knows a thing or two about Derby and who can test and enhance the packaging of the server part.