I am going to Debian Women MiniDebConf, BCN 2014.

Recently I've been pondering how to do service discovery. Pretend you have a load balancer which accepts traffic and routes incoming requests to different back-ends. The loadbalancer might be pound, varnish, haproxy, nginx, or similar. The back-ends might be node applications, apache, or similar. The typical configuration of the load-balancer will read:

# forward
# backends
backend web1    .host = "10.0.0.4";  
backend web2    .host = "10.0.0.6";  
backend web3    .host = "10.0.0.5";  
#  afterword

I've seen this same setup in many situations, and while it can easily be imagined that there might be "random HTTP servers" on your (V)LAN which shouldn't receive connections it seems like a pain to keep updating the backends. Using UDP/multicast broadcasts it is trivial to announce "Hey I'm a HTTP-server with the name 'foo'", and it seems to me that this should allow seamless HTTP load-balancing. To be more explicit - this is normal:

• The load-balancer listens for HTTP requests, and forwards them to back-ends.
• When back-ends go away they stop receiving traffic.

What I'd like to propose is another step:

• When a new back-end advertises itself with the tag "foo" it should be automatically added and start to receive traffic.

i.e. This allows backends to be removed from service when they go offline but also to be added when they come online. Without the load-balancer needing its configuration to be updated. This means you'd not give a static list of back-ends to your load-balancer, instead you'd say "Route traffic to any service that adfvertises itself with the tag 'foo'.". VLANS, firewalls, multicast, udp, all come into play, but in theory this strikes me as being useful, obvious, and simple. (Failure cases? Well if the "announcer" dies then the backend won't get traffic routed to it. Just like if the backend were offline. And clearly if a backend is announced, but not receiving HTTP-requests it would be dropped as normal.) If I get the time this evening I'll sit down and look at some load-balancer source code to see if any are written in such a way that I could add this "broadcast discovery" as a plugin/minor change.

I am going to Debian Women MiniDebConf, BCN 2014.

A couple of weeks ago, Gunnar Wolf mentioned on IRC that his CuBox-i4 had arrived. This resulted in various jealous noises from me; having heard about this device making the rounds at the Kernel Summit, I ordered one for myself back in December, as part of the long-delayed HDification of our home entertainment system and coinciding with the purchase of a new Samsung SmartTV. We've been running an Intel Coppermine Celeron for a decade as a MythTV frontend and encoder (hardware-assisted with a PVR-250), which is fine for SD video, but really doesn't cut it for anything HD. So after finally getting a TV that would showcase HD in all its glory, I figured it was time to upgrade from an S-Video-out, barely-limping-along tower machine to something more modern with HDMI out, eSATA, hardware video decoding, and whose biggest problem is it's so small that it threatens to get lost in the wiring! Since placing the order, I've been bemused to find that the SmartTV is so smart that it has had a dramatic impact on how we consume media; between that and our decision to not be a boiled frog in the face of DISH Network's annual price increase, the MythTV frontend has become a much less important part of our entertainment center, well before I ever got a chance to lay hands on the intended replacement hardware. But that's a topic for another day. Anyway, the CuBox-i4 finally arrived in the mail on Friday, so of course I immediately needed to start hacking on it! Like Gunnar, who wrote last week about his own experience getting a "proper" Debian install on the box, I'm not content with running a binary distribution image prepared by some third party; I expect my hardware toys to run official distro packages assembled using official distro tools and, if at all possible, distributed on official distro images for a minimum of hassle. Whereas Gunnar was willing to settle for using third-party binaries for the bootloader and kernel, however, I'm not inclined to do any such thing. And between my stint at Linaro a few years ago and the recent work on Ubuntu for phones, I do have a little knowledge of Linux on ARM (probably just enough to be dangerous), so I set to work trying to get the CuBox-i4 bootable with stock Debian unstable. Being such a cutting-edge piece of hardware, that does pose some challenges. Support for the i.MX6 chip is in the process of being upstreamed to U-Boot, but the support for the CuBox-i devices isn't there yet, nor is the support for SPL on i.MX6 (which allows booting the variants of the CuBox-i with a single U-Boot build, instead of requiring a different bootloader build for each flavor). The CuBox-i U-Boot that SolidRun makes available (with source at github) is based on U-Boot 2013.10-rc4, so more than a full release behind Debian unstable, and the patches there don't apply to U-Boot 2014.01 without a bit of effort. But if it's worth doing, it's worth doing right, so I've taken the time to rebase the CuBox-i patches on top of 2014.01, publishing the results of the rebase to my own github repository and submitting a bug to the Debian U-Boot maintainers requesting its inclusion. The next step is to get a Debian kernel that not only works, but fully supports the hardware out of the box (a 3.13 generic arm kernel will boot on the machine, but little things like ethernet and hdmi don't work yet). I've created a page in the Debian wiki for tracking the status of this work.

Zoe slept through the night for the first time in recent memory, and was up at around 6am. It was lovely. I'm thinking the slightly lower temperature helped. I'm definitely going to explore air conditioning for the bedrooms before next Summer. I'd picked up some egg cups on the weekend, so we did soft boiled eggs in the Thermomix for breakfast. They turned out lovely this time, and Zoe seemed to enjoy eating hers. She dressed herself while they were cooking. Her Kindergarten t-shirts arrived yesterday, so now she looks like all the other kids. It's very cute. It rained on and off in the morning, and I was in two minds about whether to chance it on the bike or not. As it happened we ended up messing around a bit before leaving, so we drove, and it was a good call, because it started raining again as we pulled out of the driveway. Drop off was super smooth. Megan had been away yesterday, but was back today, so Zoe was very happy to see her. My financial advisor came out for a chat about investments mid-morning, and then I got stuck into my US taxes. I will not miss having to do these tax returns. The 2013 one is going to be particularly complicated by the fact we sold our home and were only in the country for like 80 days. I biked over to Kindergarten to pick up Zoe, as the weather had improved. Surprisingly, she was awake, but her teacher said she'd just woken up from a nap. We managed to clear out pretty quickly. I stopped at the health food shop on the way home to get some peanut oil for the dinner I was making. Due to a miscommunication, Zoe had missed her Saturday swim class. I explained to the swim school the error, and they were kind enough to allow Zoe to have a make up class anyway, so we went to that after Kindergarten. Zoe wanted to do some more practice riding her bike. She's got a little 10 inch bike with no training wheels and a gyrobike front wheel. We haven't had a lot of success learning to ride to date, due to a combination of her height and just time, so it was interesting that she decided today she wanted to do some more practice. She wanted to "ride" her bike from home to the park on the way to swim class, but I managed to convince her to walk the bike to the park. It turns out even walking the bike is a bit of a challenge for her, so I ended up carrying it most of the way. It's heavy. We got to the park, and had a bit of a run with the bike. I'm glad I decided to take her knee pads and elbow pads, because she did stack it fairly decently, but managed to survive mostly unscathed. To her credit, she even got back on the bike again. She's a tough kid. Once we made it to the playground, she lost all interest in the bike, and we had about 15 minutes to kill before we needed to continue to the pool. We walked/carried the bike from the park to the pool, and the make up class Zoe ended up in was Megan's regular class, so the girls had a great time together. After swim class, we walked back home. Zoe's Nana was coming over for dinner, and I wanted to make a satay chicken dish. I'd bought some peanuts from the Peanut Wagon on Sunday, when I was in Nanango for my uncle's 90th birthday party, and I wanted to try making peanut butter from scratch in the Thermomix, and use that to make the satay sauce. Long story short, dinner wasn't super great. I'd made a satay chicken in the Thermomix previously, but I couldn't find the recipe I used last time. I used this recipe tonight, but didn't really follow it properly because I was trying to cut corners due to the time. I tried cooking wild rice in the microwave and it was a total flop. The satay chicken itself turned out okay, but it was a shame about the rice. It was cool making my own peanut butter. Zoe had a great time playing with her Nana, and after we dropped her off at the train station went to bed without a peep.

Recently I purchased a Tenvis IP391W-HD camera. I would be unlikely to recommend it. The price is certainly right and the picture quality is quite good. The Android and iPhone apps do work to watch the stream live. However, the interface is terrible and almost useless without Internet Explorer. There is a RTSP stream (rtsp://admin:password@ip) which VLC can seem to handle, but not mplayer. The recording format (.h264) is not viewable by VLC or mplayer and all I could find is a Windows .exe to convert them to an .avi. The motion detection gets troubled by the dark. It would really only be useful for something permanently well-lit. It did send me emails via gmail. I have got it recording to a NFS server, but I don't have a lot of confidence in the reliability of it. I think I have it configured to record in 3600-second blocks (given the interface, it's hard to tell if I've set it up to the network, or to the internal flash, etc), but it seems to intersperse 60 minute recordings with random small recordings. Given the whole idea of a security camera is to record the unexpected, you want a lot of confidence you're actually recording, which you don't get with this. You can see below it recorded 3 hour blocks, then started going a little crazy...

-rw-r--r-- 1 nobody nogroup  69M Mar 11 01:25 0-003035.v264
-rw-r--r-- 1 nobody nogroup  69M Mar 11 02:25 0-013049.v264
-rw-r--r-- 1 nobody nogroup  69M Mar 11 03:26 0-023103.v264
-rw-r--r-- 1 nobody nogroup 5.9M Mar 11 03:31 0-033117.v264
-rw-r--r-- 1 nobody nogroup 1.5M Mar 11 03:40 0-034350.v264
-rw-r--r-- 1 nobody nogroup  17M Mar 11 04:02 0-035259.v264
-rw-r--r-- 1 nobody nogroup 306K Mar 11 04:10 0-041548.v264
-rw-r--r-- 1 nobody nogroup 4.9M Mar 11 04:23 0-042457.v264

There is a support forum, where I found the following files scattered in various posts. From what I can tell, they are the latest as of this writing. I can confirm they work with my IP391W-HD, which the system tells me is GM8126 hardware and came with firmware 1.2.8.3.

• 1.3.3.3.pk2 - firmware (b56f211a569fb03a37d13b706c660dcb)
• web.pk2 - a UI update that includes dropbox support. This is really for the model that has pan and tilt, so those buttons don't work. (0e42e42bd6f8034e87dcd443dcc3594d)
• V264ToAVIen.exe - converts the output to an AVI file that mplayer will play (with some complaints) (9c5a858aa454fed4a0186cf244c0d234)

www.modern.ie offers free limited-time Windows VM's which will work to upload this firmware. Just make sure you use a bridged network in the VM; I'm guessing the firmware ActiveX control tells the camera to TFTP the data from it, which doesn't work via NAT. Somewhat worryingly, you can telnet to it and get a login prompt (TASTECH login). So it has a built-in backdoor you can't disable. There have been some efforts to hack the device. leecher@dose.0wnz.at did an excellent job reverse engineering the .pk2 format and writing tenvis_pack.c (no license, I'm generously assuming public domain). I used this to recreate the firmware above with a telnet daemon listening with a shell on port 2525 (no password, just telnet to it)

• 1.3.3.3-telnet.pk2 (19cf63ca8f0928af8ef5a9eb8326aae6)

It's interesting to poke around, but it seems like the whole thing is really driven by a binary called ipc8126

/ # ipc8126 --help
*** TAS-Tech IPCAM/DVS
*** Version: 1.3.3.3
*** Release date: 2013-08-05 15:48:32

In general, I'd say hackability is quite low. Warning : any of the above might turn your camera into a paperweight. It worked for me, but that's all I can say...

I've spent a good part of the weekend baking and taking pictures of cookies, because of reasons.





And now I'm forced to eat them, I can't let them go stale O:- )

(Actually, I plan to share at least some of them at @LIFO this night.)

Each year the 7DRL competion gives me an opportunity to abandon developing a roguelike game part way through. This year I am going to strive mightily to actually complete a playable game though owing to family commitments I have already gotten off to a rocky start which does not bode well. But enough negativity... So this year 7DRL coincides with the first ever Procedural Death Jam. This is a similar type of thing except the genre is Procedural Death Labyrinth. Procedural Death Labyrinths are a subset of Roguelikes (or is it the other way round?) If you have taken advantage of the wonderful offer Valve has given to Debian developers, you can find a number of examples of Procedural Death Labyrinths on Steam. The most important part of any software development project is of course choosing a cool name. My game is called "The Girl Who Played With The Dragons Nest" (Can you guess what I've been reading?) It is written in C++ with the Curses toolkit. It will of course be open source. I intend to blog my progress here if you care to follow along.

I got up not as early as I'd have liked, and did the first 10km run I've done for a while, with my new shoes and the orthotics in them. It was a bit cooler, and despite not being particularly well hydrated, I lasted the distance, and managed to complete it in under an hour. I'm a long way off my best time at the moment, but I'm hoping as it cools down my running fitness will improve. I ended up doing a bunch of startup-related stuff today, instead of Debian stuff. It felt good to make some vaguely forward progress, even if it was just business operations planning stuff. I picked up Zoe from Kindergarten by bike. We'd both confirmed with her several times since Friday as to which mode of transport she wanted to use to come home, and she'd indicated the bike. Zoe was asleep as usual, but woke up pretty easily. The Kindergarten has indicated that they want everyone out of the centre by 2:30pm, which makes it a bit of a challenge to rouse a deeply sleeping child and exit the centre in 10 minutes. I managed to sunscreen her while she was still horizontal, so that sped things up, and it helped greatly that she didn't have a post-nap meltdown. One of her class mates and her mother were hanging around waiting for her big brother to finish school, and the bike trailer tends to be a good conversation starter, so we chatted for a little bit before leaving. It was nice to properly meet another parent, and I'm glad to see Zoe's started learning some of her class mates names. I had a kale, carrot and apple juice for lunch today, and as I hate wasting the pulp out of the juicer, I'd previously found a nice muffin recipe that I could use the kale and carrot pulp in. I'd previously baked a batch of mini muffins for Zoe's lunches, and was pleased that she liked them. We were getting a bit low, so I thought we'd bake another batch this afternoon, so that's what we did after we got home. Once they were in the oven, Zoe went off to watch TV and I made a start on dinner. Bedtime went nice and smoothly tonight.

I've spent a while trying to make GPU switching work more reliably on Apple hardware, and got to the point where my Retina MBP now comes up with X running on Intel regardless of what the firmware wanted to do (test patches here. But in the process I'd introduced an additional GPU switch which added another layer of flicker to the boot process. I spent some time staring at driver code and poking registers trying to figure out how I could let i915 probe EDID from the panel without switching the display over and made no progress at all.

And then I realised that nouveau already has all the information that i915 wants, and maybe we could just have the Switcheroo code hand that over instead of forcing i915 to probe again. Sigh.

comments

As a followup for my previous blog post, I decided to port QAntenna to Qt 5. Here's my experience.

First of all, I ran Qt 5's qmake:

qmake -qt5

Then I just ran make:

make

The first error that appeared was that QFileDialog does not has a setFilters() method. I couldn't find this on on the C++ API change page but looking at the class' documentation I found it was renamed to setSectionsMovable(). Fine, let's just change that and continue compiling.

The next errors came all from the same class QHeaderView. This time the changes are documented in the above linked page, so it was a matter to replace setMovable() with setSectionsMovable() and setResizeMode() with setSectionResizeMode(). Fixed, let's continue.

The following error turned out to be the last one: toAscii() has been deprecated in favor of toLatin1(). Once this was fixed voil , QAntenna is running with Qt 5.

Easy, wasn't it?

I have just pushed the new version to Debian unstable.

As you might know, Qt 4 is now in bug-fixes-only maintenance mode. This means that only bug fixes are allowed to be pushed to Qt 4's repository, but no new features.

On the other hand, Qt 5 is already here. Porting existing apps from Qt 4 to Qt 5 is actually easier than it was to port Qt 3 stuff to Qt 4. Take a look at this Qt project's wiki page for more info. Also pay attention at the links at the bottom of it under "Recommended Reading".

Some time ago I blogged about the 5.2.0 status in Debian experimental. We currently have it in testing with a much better status:


And we also have 5.2.1 in experimental with an even better status:

Note: qttools is FTBFS in armhf just because of some symbols changes, it will be fixed on the next upload.

Don't be afraid of getting it from experimental! Take a look at this blog post to know how to compile with Qt 5 in Debian.

BonusI have also uploaded Qt Creator built against Qt 5 to experimental. I plan to push it to unstable with Qt 5.2.1.

There's a German joke about Germans:

Q: How can you find Germans abroad?
A: You stand outside a bakery and wait until someone starts cussing.

I am constantly reminded of this joke, abroad and back home, as we do tend to take our bread seriously... As an additional data point supporting this, the German local team just spent 20 minutes discussing bread. I somehow doubt you could make an off-hand comment about bread and garner much interest in most communities; in a German one, this works. In mildly related news, this is what bits.debian.org will pick up soon: During the DebConf committee meeting, it has been decided that the 16th annual Debian Conference, DebConf15, will be held in Germany next year. Thanks to the Belgian and Swedish teams; we are looking forward to their renewed bids for future DebConfs! Specifics as to location and date are still being nailed down and we will keep Debian as a whole informed about our progress. A dedicated (English-language) mailing list has been created for the organization and we welcome interested people to subscribe and join the discussion.

If you re the sort that cares about federation, the MediaGoblin project is for you!

MediaGoblin is a media hosting platform, where you can post all sorts of things, like video, images or even 3d models. It s a nice replacement for things like Flickr, and YouTube, and super easy to set up.

If I wasn t such a lazy person, I d have uploaded it to Debian, but alas. Soon. Soon!

It s an official GNU project, and it s maintainer, Chris is a totally awesome guy, and MediaGoblin is really important work.



If you feel the urge to support federation on the web, Support MediaGoblin! Help us take back the net!

Yesterday I played with LXC a bit, and I liked it, as LXC provides a very lightweight isolation of processes, much like enchanced chroot. However, I realised, I don t always need the chroot-like part of LXC, sometimes what I need is just to make sure the process is unable to see the other processes and talk to them in any other way except the filesystem, but I don t want a whole separate root file system for that. LXC provides a simple utility, lxc-unshare, which uses Linux-specific clone(2) call to run a process with new PID, IPC and other namespaces. However, this utility can t be used for running forking daemons, as the container is destroyed when its PID 1 exits. That s why I decided to add the support for process isolation to start-stop-daemon. I was careful enough to introduce minimal changes to it to make sure they can be merged back to the version as shipped with dpkg, and at the same time be buildable separately from the rest of it. Speaking of building, by the way, this version of start-stop-daemon uses mk-configure, a light-weight autotools replacement, as its build system. What I added, from a user s perspective, is a single --isolate option. When run with this option, start-stop-daemon calls clone(2) just before configuring the environment of a process it s going to run. The cloned process runs in a new namespace, and immediately remounts /proc, /dev/shm and /dev/mqueue. Then it forks, and a forked process executes the daemon. The parent process, which is PID 1 in the new PID namespace, uses waitpid(2) to monitor its child processes. As soon as its very first child exits, it talks via a pipe to the process outside of the container, so that it knows the container has a forked daemon and it needs to detach. In any case, the container s PID 1 does waitpid(2) in a loop, checking the process list every time a child terminates. As soon as it s left alone or a SIGTERM is received, it exits. The code is published at Bitbucket.

Or, "Why 2014 will NOT be the year of Linux in the desktop". So, it happens that my mum (66 yo) has been a Debian user for over a year now. With highs and lows, she manages to do what she needs; sometimes I need to intervene. Today I thought I could send her a quick email explaining how to download using BitTorrent, because of reasons. So, as I was writing, I realised that in many torrent sites, you only get a magnet link these days. No problem! Click on the magnet link, at it should work automagically. Then I remembered: it works on my computer, because I've spent a couple of hours some time ago researching how to make FireFox work with magnet links, creating a custom script, etc. I hoped that by now this must have been solved, at least in Debian unstable. Wrong again. I created a new user in my computer, launched IceWeasel/FireFox and boom: I get a dialog asking me to select a program, not from a list of desktop applications, taken from one of the gazillion sources where applications are defined, but just from any place on the file system! (At least, now you don't need to go tweaking with the hidden FireFox configuration editor). I was very angry at the brainiac at Mozilla who thought it was a great idea to ignore the host system and do their own MIMEtype handling. And then, tried Chromium to see what would happen... And I get first a scary message telling me that it is going to use the super-obscure xdg-open program to open my link, and that it could harm my computer! It was followed by another very helpful dialog telling me something like:

Unable to detect the URI-scheme of "magnet:?xt=urn:btih:diePh6iengei4quaep4shai8ahshahnae9 oolahtetheir2bohmu1eelaChui1ohdahruegh4wief6PusahDae4ho oshahjoogai7bae9shuvei9shufeX4boog8neichi3OoDee5ei9Uori c6aingairepon9gok8Mee7uRahphah4EucoopheiYin4xe4lahn0goh"

Then the real fun started... I starting looking around to understand how this is supposed to work, I wanted to provide a patch! So, it turns out that if you add some values to GConf this should work. So, try to find where would that be. Read about GConf schemas, default and mandatory values, and their 10 possible locations. Find that Azureus provides an schema, use that to create one for Transmission. Then find that in fact, Transmission was providing defaults, which are not the same but work the same, and that they had an error there: yes, problem found! (#741069) No! It turns out that the Gnome desktop does not use that any more, and now they scan the .desktop (who knows in which of the 100 directory tress where .destop files are present) files for MIME handlers, and the transmission-gtk.desktop file had that correctly. So why does it not work? Well, it turns out that if I used gvfs-open instead of xdg-open, it did work! The thing is, I am running XFCE here, which is GTK based, but it is not Gnome: instead of gvfs-open, I was getting exo-open, which is it's brain-dead cousin, and can't do anything but files, email and web. It is fecking 2014, and we still don't have a sensible, unified way to select preferred applications. We still have incompatible, duplicated, incomplete, competing implementations. We have FreeDesktop doing one thing to try and unify criteria, which is then ignored or mis-implemented up and down by some desktops and applications. Some days I get really angry at the Free Software world. PS: I guess I will tell mum to copy&paste the links from the browser to the torrent client, but not today. I have already lost 4 hours of sleep on this.

Brief Glamor Hacks Eric Anholt started writing Glamor a few years ago. The goal was to provide credible 2D acceleration based solely on the OpenGL API, in particular, to implement the X drawing primitives, both core and Render extension, without any GPU-specific code. When he started, the thinking was that fixed-function devices were still relevant, so that original code didn t insist upon modern OpenGL features like GLSL shaders. That made the code less efficient and hard to write. Glamor used to be a side-project within the X world; seen as something that really wasn t very useful; something that any credible 2D driver would replace with custom highly-optimized GPU-specific code. Eric and I both hoped that Glamor would turn into something credible and that we d be able to eliminate all of the horror-show GPU-specific code in every driver for drawing X text, rectangles and composited images. That hadn t happened though, until now. Fast forward to the last six months. Eric has spent a bunch of time cleaning up Glamor internals, and in fact he s had it merged into the core X server for version 1.16 which will be coming up this July. Within the Glamor code base, he s been cleaning some internal structures up and making life more tolerable for Glamor developers. Using libepoxy A big part of the cleanup was a transition all of the extension function calls to use his other new project, libepoxy, which provides a sane, consistent and performant API to OpenGL extensions for Linux, Mac OS and Windows. That library is awesome, and you should use it for everything you do with OpenGL because not using it is like pounding nails into your head. Or eating non-tasty pie. Using VBOs in Glamor One thing he recently cleaned up was how to deal with VBOs during X operations. VBOs are absolutely essential to modern OpenGL applications; they re really the only way to efficiently pass vertex data from application to the GPU. And, every other mechanism is deprecated by the ARB as not a part of the blessed core context . Glamor provides a simple way of getting some VBO space, dumping data into it, and then using it through two wrapping calls which you use along with glVertexAttribPointer as follows:

pointer = glamor_get_vbo_space(screen, size, &offset);
glVertexAttribPointer(attribute_location, count, type,
              GL_FALSE, stride, offset);
memcpy(pointer, data, size);
glamor_put_vbo_space(screen);

glamor_get_vbo_space allocates the specified amount of VBO space and returns a pointer to that along with an offset , which is suitable to pass to glVertexAttribPointer. You dump your data into the returned pointer, call glamor_put_vbo_space and you re all done. Actually Drawing Stuff At the same time, Eric has been optimizing some of the existing rendering code. But, all of it is still frankly terrible. Our dream of credible 2D graphics through OpenGL just wasn t being realized at all. On Monday, I decided that I should go play in Glamor for a few days, both to hack up some simple rendering paths and to familiarize myself with the insides of Glamor as I m getting asked to review piles of patches for it, and not understanding a code base is a great way to help introduce serious bugs during review. I started with the core text operations. Not because they re particularly relevant these days as most applications draw text with the Render extension to provide better looking results, but instead because they re often one of the hardest things to do efficiently with a heavy weight GPU interface, and OpenGL can be amazingly heavy weight if you let it. Eric spent a bunch of time optimizing the existing text code to try and make it faster, but at the bottom, it actually draws each lit pixel as a tiny GL_POINT object by sending a separate x/y vertex value to the GPU (using the above VBO interface). This code walks the array of bits in the font and checking each one to see if it is lit, then checking if the lit pixel is within the clip region and only then adding the coordinates of the lit pixel to the VBO. The amazing thing is that even with all of this CPU and GPU work, the venerable 6x13 font is drawn at an astonishing 3.2 million glyphs per second. Of course, pure software draws text at 9.3 million glyphs per second. I suspected that a more efficient implementation might be able to draw text a bit faster, so I decided to just start from scratch with a new GL-based core X text drawing function. The plan was pretty simple:

1. Dump all glyphs in the font into a texture. Store them in 1bpp format to minimize memory consumption.
2. Place raw (integer) glyph coordinates into the VBO. Place four coordinates for each and draw a GL_QUAD for each glyph.
3. Transform the glyph coordinates into the usual GL range (-1..1) in the vertex shader.
4. Fetch a suitable byte from the glyph texture, extract a single bit and then either draw a solid color or discard the fragment.

This makes the X server code surprisingly simple; it computes integer coordinates for the glyph destination and glyph image source and writes those to the VBO. When all of the glyphs are in the VBO, it just calls glDrawArrays(GL_QUADS, 0, 4 * count). The results were encouraging :

1: fb-text.perf
2: glamor-text.perf
3: keith-text.perf
       1                 2                           3                 Operation
------------   -------------------------   -------------------------   -------------------------
   9300000.0      3160000.0 (     0.340)     18000000.0 (     1.935)   Char in 80-char line (6x13) 
   8700000.0      2850000.0 (     0.328)     16500000.0 (     1.897)   Char in 70-char line (8x13) 
   6560000.0      2380000.0 (     0.363)     11900000.0 (     1.814)   Char in 60-char line (9x15) 
   2150000.0       700000.0 (     0.326)      7710000.0 (     3.586)   Char16 in 40-char line (k14) 
    894000.0       283000.0 (     0.317)      4500000.0 (     5.034)   Char16 in 23-char line (k24) 
   9170000.0      4400000.0 (     0.480)     17300000.0 (     1.887)   Char in 80-char line (TR 10) 
   3080000.0      1090000.0 (     0.354)      7810000.0 (     2.536)   Char in 30-char line (TR 24) 
   6690000.0      2640000.0 (     0.395)      5180000.0 (     0.774)   Char in 20/40/20 line (6x13, TR 10) 
   1160000.0       351000.0 (     0.303)      2080000.0 (     1.793)   Char16 in 7/14/7 line (k14, k24) 
   8310000.0      2880000.0 (     0.347)     15600000.0 (     1.877)   Char in 80-char image line (6x13) 
   7510000.0      2550000.0 (     0.340)     12900000.0 (     1.718)   Char in 70-char image line (8x13) 
   5650000.0      2090000.0 (     0.370)     11400000.0 (     2.018)   Char in 60-char image line (9x15) 
   2000000.0       670000.0 (     0.335)      7780000.0 (     3.890)   Char16 in 40-char image line (k14) 
    823000.0       270000.0 (     0.328)      4470000.0 (     5.431)   Char16 in 23-char image line (k24) 
   8500000.0      3710000.0 (     0.436)      8250000.0 (     0.971)   Char in 80-char image line (TR 10) 
   2620000.0       983000.0 (     0.375)      3650000.0 (     1.393)   Char in 30-char image line (TR 24)

This is our old friend x11perfcomp, but slightly adjusted for a modern reality where you really do end up drawing billions of objects (hence the wider columns). This table lists the performance for drawing a range of different fonts in both poly text and image text variants. The first column is for Xephyr using software (fb) rendering, the second is for the existing Glamor GL_POINT based code and the third is the latest GL_QUAD based code. As you can see, drawing points for every lit pixel in a glyph is surprisingly fast, but only about 1/3 the speed of software for essentially any size glyph. By minimizing the use of the CPU and pushing piles of work into the GPU, we manage to increase the speed of most of the operations, with larger glyphs improving significantly more than smaller glyphs. Now, you ask how much code this involved. And, I can truthfully say that it was a very small amount to write:

 Makefile.am             2 
 glamor.c                5 
 glamor_core.c           8 
 glamor_font.c         181 ++++++++++++++++++++
 glamor_font.h          50 +++++
 glamor_priv.h          26 ++
 glamor_text.c         472 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 glamor_transform.c      2 
 8 files changed, 741 insertions(+), 5 deletions(-)

Let s Start At The Very Beginning The results of optimizing text encouraged me to start at the top of x11perf and see what progress I could make. In particular, looking at the current Glamor code, I noticed that it did all of the vertex transformation with the CPU. That makes no sense at all for any GPU built in the last decade; they ve got massive numbers of transistors dedicated to performing precisely this kind of operation. So, I decided to see what I could do with PolyPoint. PolyPoint is absolutely brutal on any GPU; you have to pass it two coordinates for each pixel, and so the very best you can do is send it 32 bits, or precisely the same amount of data needed to actually draw a pixel on the frame buffer. With this in mind, one expects that about the best you can do compared with software is tie. Of course, the CPU version is actually computing an address and clipping, but those are all easily buried in the cost of actually storing a pixel. In any case, the results of this little exercise are pretty close to a tie the CPU draws 190,000,000 dots per second and the GPU draws 189,000,000 dots per second. Looking at the vertex and fragment shaders generated by the compiler, it s clear that there s room for improvement. The fragment shader is simply pulling the constant pixel color from a uniform and assigning it to the fragment color in this the simplest of all possible shaders:

uniform vec4 color;
void main()
 
       gl_FragColor = color;
 ;

This generates five instructions:

Native code for point fragment shader 7 (SIMD8 dispatch):
   START B0
   FB write target 0
0x00000000: mov(8)          g113<1>F        g2<0,1,0>F                        align1 WE_normal 1Q  ;
0x00000010: mov(8)          g114<1>F        g2.1<0,1,0>F                      align1 WE_normal 1Q  ;
0x00000020: mov(8)          g115<1>F        g2.2<0,1,0>F                      align1 WE_normal 1Q  ;
0x00000030: mov(8)          g116<1>F        g2.3<0,1,0>F                      align1 WE_normal 1Q  ;
0x00000040: sendc(8)        null            g113<8,8,1>F
                render ( RT write, 0, 4, 12) mlen 4 rlen 0        align1 WE_normal 1Q EOT  ;
   END B0

As this pattern is actually pretty common, it turns out there s a single instruction that can replace all four of the moves. That should actually make a significant difference in the run time of this shader, and this shader runs once for every single pixel. The vertex shader has some similar optimization opportunities, but it only runs once for every 8 pixels with the SIMD format flipped around, the vertex shader can compute 8 vertices in parallel, so it ends up executing 8 times less often. It s got some redundant moves, which could be optimized by improving the copy propagation analysis code in the compiler. Of course, improving the compiler to make these cases run faster will probably make a lot of other applications run faster too, so it s probably worth doing at some point. Again, the amount of code necessary to add this path was tiny:

 Makefile.am             1 
 glamor.c                2 
 glamor_polyops.c      116 ++++++++++++++++++++++++++++++++++++++++++++++++++--
 glamor_priv.h           8 +++
 glamor_transform.c    118 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 glamor_transform.h     51 ++++++++++++++++++++++
 6 files changed, 292 insertions(+), 4 deletions(-)

Discussion of Results These two cases, text and points, are probably the hardest operations to accelerate with a GPU and yet a small amount of OpenGL code was able to meet or beat software easily. The advantage of this work over traditional GPU 2D acceleration implementations should be pretty clear this same code should work well on any GPU which offers a reasonable OpenGL implementation. That means everyone shares the benefits of this code, and everyone can contribute to making 2D operations faster. All of these measurements were actually done using Xephyr, which offers a testing environment unlike any I ve ever had build and test hardware acceleration code within a nested X server, debugging it in a windowed environment on a single machine. Here s how I m running it:

$ ./Xephyr  -glamor :1 -schedMax 2000 -screen 1024x768 -retro

The one bit of magic here is the -schedMax 2000 flag, which causes Xephyr to update the screen less often when applications are very busy and serves to reduce the overhead of screen updates while running x11perf. Future Work Having managed to accelerate 17 of the 392 operations in x11perf, it s pretty clear that I could spend a bunch of time just stepping through each of the remaining ones and working on them. Before doing that, we want to try and work out some general principles about how to handle core X fill styles. Moving all of the stipple and tile computation to the GPU will help reduce the amount of code necessary to fill rectangles and spans, along with improving performance, assuming the above exercise generalizes to other primitives. Getting and Testing the Code Most of the changes here are from Eric s glamor-server branch:

git://people.freedesktop.org/~anholt/xserver glamor-server

The two patches shown above, along with a pair of simple clean up patches that I ve written this week are available here:

git://people.freedesktop.org/~keithp/xserver glamor-server

Of course, as this now uses libepoxy, you ll need to fetch, build and install that before trying to compile this X server. Because you can try all of this out in Xephyr, it s easy to download and build this X server and then run it right on top of your current driver stack inside of X. I d really like to hear from people with Radeon or nVidia hardware to know whether the code works, and how it compares with fb on the same machine, which you get when you elide the -glamor argument from the example Xephyr command line above.

While in the process of polishing up a paper, I had to draw figures using a well-known proprietary program for drawing chemical formula whose name I will not write down, out of decency (but you know which one it is anyway). I'm using the mac version (yes, no version for Linux) which has a neat glitch in that the "Save As" button does not work at all: no way to export your chemical formula as a SVG/PDF/PNG picture or whatever ! So I resorted to the neat Print as PDF feature of macintosh. I had to edit the resulting PDF files by hand as some of the colors were wrong (great !), and when I tried to include the PDF, I realized they had a white background... No way !So I dug in ImageMagick's convert documentation, and came up with the following code that converts the PDF file into a PNG with a transparent background:

convert -density 600 figure.pdf -channel alpha \
  -fx '((r == 1 && g == 1 && b == 1) ? 0 : 1)' figure.png

There are still some few points that look white but are probably not that white (due to antialiased rendering of the PDF file ?). There are many more things that can be done using the -fx operator, this page was helpful to me !Edit: while the alpha channel seems to be on by default for PDF files, it is not necessarily the case for all images. If the above doesn't work, try adding -alpha Set before the -fx bit.

This week I received a logitech squeezebox radio, which is basically an expensive toy that allows you to listen to either "internet radio", or music streamed from your own PC via a portable device that accesses the network wirelessly. The main goal of this purchase was to allow us to listen to media stored on a local computer in the bedroom, or living-room. The hardware scans your network looking for a media server, so the first step is to install that:

• The slimdevices wiki has decent information.
  • Or you can find the code on github.com.

The media-server has a couple of open ports; one for streaming the media, and one for a user-browsable HTML interface. Interestingly the radio-device shows up in the web-interface, so you can mess around with the currently loaded playlist from your office, while your wife is casually listening to music in the bedroom. (I'm not sure if that's a feature or not yet ;) Although I didn't find any alternative server-implementations I did find a software-client which you can use to play music from the central server - slimp3slave - and again you can push playlists, media, etc, to this. My impressions are pretty positive; the device was too expensive, certainly I wouldn't buy two, but it is functional. The user-interface is decent, and the software being available and open is a big win. Downsides? No remote-control for the player, because paying an additional 70 is never going to happen, but otherwise I can't think of anything. (Shame the squeezebox product line seems to have been cancelled (?)) Procmail Alternatives?

Although I did start hacking a C & Lua alternative, it looks like there are enough implementations out there that I don't feel so strongly any more. I'm working in a different way to most people, rather than sort mails at delivery time I'm going to write a trivial daemon that will just watch ~/Maildir/.Incoming, and move mails out of there. That means that no errors will cause mail to be lost at SMTP/delivery time. I'm going to base my work on Email::Filter since it offers 90% of the primitives I want. The only missing thing is the ability to filter mails via external commands which has now been reported as a bug/omission.