Just back in from an amazing concert at the Chicago Symphony Center. Vocalists Angelique Kidjo, Dianne Reeves and Lizz Wright supported by all-star band of Geri Allen on piano, Romero Lubambo on electric and acoustic guitar, James Genus on electric bass guitar, Munyungo Jackson on percussion and Terri Lyne Carrington on drums (and musical director). The concert alternates between solos and joint pieces with more joy and soul than I heard in a long time. Great evening.

I finally manage to create an image of the Debian Installer automatically, with no key or password transiting on the Cloud or the network, and without using the ec2-api-tools, which are non-Free. Thanks to Eucalyptus for implementing the attribute InstanceInitiatedShutdownBehavior in euca2ools 2.1.2. In brief, I start a micro-instance with an additional volume, and I pass it a script for cloud-init, which will download the Installer on the volume and shut down the instance. Then, I register the volume as a machine image. Those machine images of the Installer are to be preseeded via instance metadata to install Debian on another volume. I am still struggling with the pre-configuration. More details will follow in the next article. In the meantime, here are the commands I a using. On the local computer, start an instance on the Cloud. Until #696595 is fixed and cloud-init is installed by default, I use a Ubuntu image.

HELPER_AMI=ami-7609bb77 # Ubuntu 12.04 LTS Precise amd64 EBS (Asia North-East)

Start the instance with an extra volume of 1 GiB, which will persist after termination (/dev/sdb=:1:false). Pass the script debigen-install-installer-cloud (see below).

HELPER_INSTANCE=$( euca-run-instances \
        --instance-initiated-shutdown-behavior terminate \
        --instance-type t1.micro \
        --block-device-mapping /dev/sdb=:1:false \
        --user-data-file debigen-install-installer-cloud \
        $HELPER_AMI  
        tee /dev/stderr   grep INSTANCE   awk ' print $2 ')

Wait that the boot finished.

while [ ! $(euca-describe-instances $HELPER_INSTANCE   grep INSTANCE   cut -f 6   tee /dev/stderr) = "running" ]
    do sleep 30
done

Get the volume's identifier

TARGET_VOLUME=$( euca-describe-volumes  
        grep $HELPER_INSTANCE   grep '/dev/sdb'  
        tee /dev/stderr   awk ' print $2 ')

Wait that the scripts shuts down the instance.

while [ ! $(euca-describe-instances $HELPER_INSTANCE   grep INSTANCE   cut -f 6   tee /dev/stderr) = "terminated" ]
    do sleep 30
done

Snapshot the volume and register it as a machine image.

PV_KERNEL=aki-44992845 # Boot PV-Grub on hd0 (Asia North-East)
TARGET_SNAPSHOT=$( euca-create-snapshot $TARGET_VOLUME  
        tee /dev/stderr   awk ' print $2 ')
while euca-describe-snapshots $TARGET_SNAPSHOT   grep -q pending ; do sleep 30 ; done
euca-register \
    --name test_di \
    --description test_of_debian_installer \
    --snapshot $TARGET_SNAPSHOT \
    --kernel $PV_KERNEL \
    --architecture x86_64

Here is the script called debigen-install-installer-cloud, used above.

#!/bin/sh -ex    
mke2fs -L debian-installer /dev/xvdb -F
mount LABEL=debian-installer /mnt/
cd /mnt    
ARCH=amd64
DIST=squeeze
DI_VERSION=20110106+squeeze4+b2
MIRROR=jp
BASEURL=http://ftp.$MIRROR.debian.org/debian/dists/$DIST/main/installer-$ARCH/$DI_VERSION/images/netboot/xen
wget $BASEURL/initrd.gz $BASEURL/vmlinuz    
mkdir -p boot/grub    
cat > boot/grub/menu.lst <<__END__
default 0
timeout 3
title  Debian Installer ($DI_VERSION $ARCH)
root   (hd0)
kernel /vmlinuz root=LABEL=debian-installer ro console=hvc0 auto=true priority=critical url=http://169.254.169.254/latest/user-data DEBIAN_FRONTEND=text
initrd /initrd.gz
__END__
sleep 30     
halt

What is Valgrind? Valgrind is a framework for building dynamic analysis tools. Several useful tools are included in the Valgrind distribution including tools to check dynamic memory usage (memcheck), a cache profiler (cachegrind), heap profiler (massif) and thread debugger (helgrind) among others. Valgrind also provides a framework which can be used to build other tools. The Valgrind tool which I find most useful and the one which I have most experience with is memcheck. This tool can detect all manner of memory management problems, including use after free, using uninitialized data, memory leaks, double free. Between them these can result in savings of many hours of staring a core dumps and gdb backtraces. How Does memcheck Work? At its core Valgrind is a dynamic binary translation engine, which is used to instrument the code at runtime. In the case of memcheck this is used to track properties of the memory in a process, including aspects such as whether each bit (yes, bit) of memory has been initialized since it was allocated. It also tracks this information as data is loaded into registers, so it can know if a given register is currently tainted with uninitialized data or not. As well as instrumentation through binary translation Valgrind also includes instrumented versions of the C dynamic memory allocation functions which are used to track whether a each bit of memory is currently considered free or allocated, as well as tainting registers when they contain a pointer to memory which has been freed. A large part of memcheck's functionality is built upon Valgrind's ability to determine when memory has been initialized. However although binary translation can be used to instrument when the application itself has initialized some memory it is not currently possible to instrument the behaviour of system calls, in other words Valgrind cannot automatically determine when memory has been initialised e.g. after a read(2) system call. For this reason Valgrind has baked in knowledge about the behaviour of system calls on various platforms. For example lets consider the read(2) system call. The prototype of read(2) is:

    ssize_t read(int fd, void *buf, size_t count);

This system call reads up to count bytes from the file descriptor fd into the memory pointed to by buf and returns the number of bytes read. Here is the code within Valgrind which handles this system call (found in coregrind/m_syswrap/syswrap-generic.c):

PRE(sys_read)
 
   *flags  = SfMayBlock;
   PRINT("sys_read ( %ld, %#lx, %llu )", ARG1, ARG2, (ULong)ARG3);
   PRE_REG_READ3(ssize_t, "read",
                 unsigned int, fd, char *, buf, vki_size_t, count);
   if (!ML_(fd_allowed)(ARG1, "read", tid, False))
      SET_STATUS_Failure( VKI_EBADF );
   else
      PRE_MEM_WRITE( "read(buf)", ARG2, ARG3 );
 
POST(sys_read)
 
   vg_assert(SUCCESS);
   POST_MEM_WRITE( ARG2, RES );
 

These two functions are called before and after the system call respectively. The main piece of functionality is the calls to PRE_MEM_WRITE and POST_MEM_WRITE. In this case PRE_MEM_WRITE is used to check that the memory starting at ARG2 (recall that this is the supplied buffer) and extending for ARG3 bytes (the system calls count argument) consists of allocated memory. After the system call in complete the post function calls POST_MEM_WRITE to indicate to Valgrind that RES bytes of the buffer have know been initialized. The above is a relatively simple case, but the above two hooks must be supplied for each system call on each platform which Valgrind wishes to support. One of the most problematic system calls is the ioctl(2) system call. The ioctl call takes a file descriptor, a request number and a pointer to a per-ioctl argument structure and implements per-device control requests. There are literally dozens of devices many with their own specific ioctls. For this reason a sizable portion of Valgrind code is dedicate to decoding the behaviour of the myriad of potential ioctls and providing pre- and post-call instrumentation for them. Interacting with the Hypervisor from Userspace Under Xen the toolstack is a normal userspace process running in the control domain (typically domain 0). As part of its operation the toolstack needs to communicate with the hypervisor and to do this it uses hypercalls. However a userspace process cannot simply make hypercalls on its own, instead it must request that the OS kernel make the hypercall for it. This prevents just any userspace process making a hypercall, since the kernel will first check the privilege of the process making the request. Under Linux a userspace process makes hypercalls by using the ioctl(2) system call on a special "Privileged Command" (privcmd) device. In this case the toolstack uses the IOCTL_PRIVCMD_HYPERCALL ioctl request which takes a hypercall number and the hypercall arguments as its argument. Other OSes which can be used as a control domain posses similar interfaces. Making memcheck Work With Xen Toolstacks? As might be expected the majority of the work to allow Valgrind to work on Xen toolstack processes was teaching it about the IOCTL_PRIVCMD_HYPERCALL ioctl. Fortunately it was not necessary to teach Valgrind about every hypercall but only about those which are used by toolstacks. This is a smallish subset of all hypercalls including the domctl interfaces and a handful of others. The initial batch of patches cover all of the system calls needed to start new domains, shut them down and migrate them using the XL toolstack. New hypercalls will be added as they are discovered to be missing. Getting and Using This Functionality The initial set of patches were accepted into Valgrind's Subversion repository trunk revision 13081 in October 2012 and are expected to be part of the 3.9.0 release. Until 3.9.0 is released it will be necessary to get this functionality from SVN. Fortunately this is relatively simple and follows the usual autofoo dance. Although the usual caveats regarding running unreleased software apply.

  $ svn co svn://svn.valgrind.org/valgrind/trunk valgrind
  $ cd valgrind
  $ ./autogen.sh
  $ ./configure
  $ make
  $ sudo make install

Once Valgrind is built and installed simply call it passing the command to be debugged as an option:

  # valgrind xl list
  ==16186== Memcheck, a memory error detector
  ==16186== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
  ==16186== Using Valgrind-3.9.0.SVN and LibVEX; rerun with -h for copyright info
  ==16186== Command: xl list
  ==16186== 
  Name                                        ID   Mem VCPUs    State   Time(s)
  Domain-0                                     0   512     4     r-----     332.5
  ==16186== 
  ==16186== HEAP SUMMARY:
  ==16186==     in use at exit: 0 bytes in 0 blocks
  ==16186==   total heap usage: 16 allocs, 16 frees, 93,529 bytes allocated
  ==16186== 
  ==16186== All heap blocks were freed -- no leaks are possible
  ==16186== 
  ==16186== For counts of detected and suppressed errors, rerun with: -v
  ==16186== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 39 from 6)

Here we see that xl list has no leaks! However if I comment out the call to libxl_dominfo_list_free in tools/libxl/xl_cmdimpl.c:main_list() then instead Valgrind reports:

[...]
  ==16203== HEAP SUMMARY:
  ==16203==     in use at exit: 90,112 bytes in 1 blocks
  ==16203==   total heap usage: 16 allocs, 15 frees, 93,529 bytes allocated
  ==16203== 
  ==16203== LEAK SUMMARY:
  ==16203==    definitely lost: 0 bytes in 0 blocks
  ==16203==    indirectly lost: 0 bytes in 0 blocks
  ==16203==      possibly lost: 90,112 bytes in 1 blocks
  ==16203==    still reachable: 0 bytes in 0 blocks
  ==16203==         suppressed: 0 bytes in 0 blocks
  ==16203== Rerun with --leak-check=full to see details of leaked memory
  ==16203== 
  ==16203== For counts of detected and suppressed errors, rerun with: -v
  ==16203== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 39 from 6)
[...]

So it has caught the memory leak! Rerunning with the suggests --leak-check=full goes even further and tells me where the leaked memory was allocated:

[...]
  ==16204== 90,112 bytes in 1 blocks are possibly lost in loss record 1 of 1
  ==16204==    at 0x4024480: calloc (vg_replace_malloc.c:593)
  ==16204==    by 0x4050CE4: libxl_list_domain (libxl.c:548)
  ==16204==    by 0x805EEC7: main_list (xl_cmdimpl.c:3931)
  ==16204==    by 0x804D6DD: main (xl.c:285)
[...]

To ease debugging of domain creation I find it useful to use the -F option to xl create to stop xl from forking and daemonising. Most xl subcommands do not daemonize and so need no special treatment. For more information on running Valgrind see the Valgrind Documentation. Conclusion Valgrind is a powerful tool in the arsenal of the C programmer, and can be used to catch a large number of hard to debug and common issues before they happen can save many hours of tedious debugging. The ability to use tools such as this on Xen toolstack processes is of enormous benefit and has already found several bugs in the XL toolstack.

Some times I try to figure out which Iceweasel browser plugin to install to get support for a given MIME type. Thanks to specifications done by Ubuntu and Mozilla, it is possible to do this in Debian. Unfortunately, not very many packages provide the needed meta information, Anyway, here is a small script to look up all browser plugin packages announcing ther MIME support using this specification:

#!/usr/bin/python
import sys
import apt
def pkgs_handling_mimetype(mimetype):
    cache = apt.Cache()
    cache.open(None)
    thepkgs = []
    for pkg in cache:
        version = pkg.candidate
        if version is None:
            version = pkg.installed
        if version is None:
            continue
        record = version.record
        if not record.has_key('Npp-MimeType'):
            continue
        mime_types = record['Npp-MimeType'].split(',')
        for t in mime_types:
            t = t.rstrip().strip()
            if t == mimetype:
                thepkgs.append(pkg.name)
    return thepkgs
mimetype = "audio/ogg"
if 1 < len(sys.argv):
    mimetype = sys.argv[1]
print "Browser plugin packages supporting %s:" % mimetype
for pkg in pkgs_handling_mimetype(mimetype):
    print "  %s" %pkg

It can be used like this to look up a given MIME type:

% ./apt-find-browserplug-for-mimetype 
Browser plugin packages supporting audio/ogg:
  gecko-mediaplayer
% ./apt-find-browserplug-for-mimetype application/x-shockwave-flash
Browser plugin packages supporting application/x-shockwave-flash:
  browser-plugin-gnash
%

In Ubuntu this mechanism is combined with support in the browser itself to query for plugins and propose to install the needed packages. It would be great if Debian supported such feature too. Is anyone working on adding it? Update 2013-01-18 14:20: The Debian BTS request for icweasel support for this feature is #484010 from 2008 (and #698426 from today). Lack of manpower and wish for a different design is the reason thus feature is not yet in iceweasel from Debian.

gitano is not entirely unlike the non-web, server side of github. It allows you to create and manage users and their SSH keys, groups and repositories from the command line. Repositories have ACLs associated with them. Those can be complex ("allow user X to push to master in the doc/ subtree) or trivial ("admin can do anything"). Gitano is written by Daniel Silverstone, and I'd like to thank him both for writing it and for holding my hand as I went stumbling through my initial gitano setup. Getting started with Gitano can be a bit tricky, as it's not yet packaged and fairly undocumented. Until it is packaged, it's install from source time. You need luxio, lace, supple, clod, gall and gitano itself. luxio needs a make install LOCAL=1, the others will be installed to /usr/local with just make install. Once that is installed, create a user to hold the instance. I've named mine git, but you're free to name it whatever you would like. As that user, run gitano-setup and answer the prompts. I'll use git.example.com as the host name and john as the user I'm setting this up for. To create users, run ssh git@git.example.com user add john john@example.com John Doe, then add their SSH key with ssh git@git.example.com as john sshkey add workstation < /tmp/john_id_rsa.pub. To create a repository, run ssh git@git.example.com repo create myrepo. Out of the box, this only allows the owner (typically "admin", unless overridden) to do anything with it. To change ACLs, you'll want to grab the refs/gitano/admin branch. This lives outside of the space git usually use for branches, so you can't just check it out. The easiest way to check it out is to use git-admin-clone. Run it as git-admin-clone git@git.example.com:myrepo ~/myrepo-admin and then edit in ~/myrepo-admin. Use git to add, commit and push as normal from there. To change ACLs for a given repo, you'll want to edit the rules/main.lace file. A real-world example can be found in the NetSurf repository and the lace syntax might be useful. A lace file consists of four types of lines:

• Comments, start with -- or #
• defines, look like define name conditions
• allows, look like allow "reason" definition [definition ]
• denials, look like deny "reason" definition [definition ]

Rules are processed one by one, from the top and terminate whenever a matching allow or deny is found. Conditions can either be matches to an update, such as ref refs/heads/master to match updates to the master branch. To create groupings, you can use the anyof or allof verbs in a definition. Allows and denials are checked against all the definitions listed and if all of them match, the appropriate action is taken. Pay some attention to what conditions you group together, since a basic operation (is_basic_op, aka op_read and op_write) happens before git is even involved and you don't have a tree at that point, so rules like:

define is_master ref refs/heads/master
allow "Devs can push" op_is_basic is_master

simply won't work. You'll want to use a group and check on that for basic operations and then have a separate rule to restrict refs.

I got some feedback that the analogy I used in my last post was a bit confusing, and indeed I blew the phrasing of the analogy (also now corrected). So let me try this again, since I think there's a subtlety here that may be missed. I should note for the record that my understanding of what Swartz did that started the process is apparently somewhat based on the description from the prosecution, so it may not be the complete or accurate facts. Since there will now be no trial, we may never find out what the defense was, and whether those facts would be challenged. So it may be best to think of this as a hypothetical. We never established, or will establish, in court exactly what happened. Swartz was, generally speaking, charged with two things that I consider quite distinct, at least from an ethical perspective. Most of the focus is on the copyright part: downloading JSTOR articles with an intent (never acted upon) to distribute them to the world. There are a bunch of reasons why this may or may not be justified, which are tied into the origin of those articles (many of them were publicly funded) and the legitimacy of copyright licensing agreements. I think there is significant room to hold a variety of opinions on this, although I don't believe that "crime worth 35 years in prison" is even remotely close to justifiable under any interpretation. However, there is a second part of what he was charged with, and that's primarily what I was commenting on, not the copyright part. He allegedly hooked up a laptop in an unlocked MIT wiring closet without permission and then used MIT's network and JSTOR license to download information. This is, to me, a subtlely but entirely distinct act from the question of whether taking JSTOR's data was ethically or legally wrong. Whether or not one believes that JSTOR's copyrights are not legitimate, it's still not okay to use someone else's network and license or to trespass in their wiring closets without permission. (I work in central IT for a university, so this strikes closer to home.) And this is where the analogy came in, which I flubbed. I had originally said that this was akin to "traipsing into someone's barn or backyard shed without their permission and taking some of their tools because you want to use them." The word "taking" was wrong; it falsely implies that you weren't going to return the tools. I had been thinking "taking and then returning" when I wrote that, and the important second part didn't make it into the post. So let's try this again: Swartz's actions at MIT, as I understand them (and many things could change this, such as a revelation that he had MIT's prior permission), are akin to going onto someone's property and into their barn without permission, borrowing their hedge trimmer for a while because you want to use it, and returning the hedge trimmer without any damage when you're done (and without them noticing it was gone). I'm quite fond of this analogy, since I think it clearly establishes two things:

1. Most people are going to be unhappy about this happening and will intuitively feel like it should probably be illegal. Not everyone; there are folks who don't believe in personal property, or at least wouldn't extend it to tools in a barn. But most people will feel that someone should ask first before they come borrow your tools. Even if they don't damage them, even if they return them before you notice they're gone, you might have wanted to use the tool at the same time or they might have damaged them without intending to, and they should just ask first. It's common politeness, and depending on the circumstances, someone who doesn't ask and is covert about borrowing tools might be worth calling the police over.
2. There is absolutely no way in any reasonable moral system that doing this should result in 35 years in prison. Or even 10, or even 1. Yes, most people would consider this a crime, but most people would consider it a minor crime. It's the sort of thing where you might have to impose some consequences just to make sure the message of "knock it off" is delivered firmly, but someone doing this is rude and inconsiderate, not evil.

The JSTOR copyright stuff is more complex to analyze and is more politically divisive, but for me the key points are (a) Swartz never released the data, and (b) JSTOR declined to press charges. To me, that means the deeper copyright questions, which are quite interesting, were never actually reached in this particular case. The crime that did apparently happen was the trespass at MIT, for which I think the above analogy is the right way to think about it. The point I do want people to take away from this is that one should not overlook the trespass at MIT even if one wants to celebrate the undermining of the copyright regime and doesn't believe JSTOR's data should be considered their private property. Social activism and political disobedience are important and often valuable things, but performing your social activism using other people's stuff is just rude. I think it can be a forgivable rudeness; people can get caught up in the moment and not realize what they're doing. But it's still rude, and it's still not the way to go about civil disobedience. For both ethical and tactical reasons, involving bystanders in your act of social activism without their consent is a bad approach. ETA: The problem, of course, with discussing all of this is that while it's relevant and possibly even somewhat important in the broader sense of how our community acts going forward, it also doesn't capture the fact that this was only one incident in a remarkable life. One of the worst problems with the abusive prosecution of Swartz is that it blew this incident completely out of proportion. A moment of arguable judgement should not dominate one's life or cast a shadow over all of one's other accomplishments; the prosecution tried to make it do just that. That's part of what I'm arguing, but ironically that partly feeds into the lack of proportionality in the discussion. I've gotten kind of far afield here, so let me go back and say explicitly: Swartz was a remarkable person who did much to admire and respect, and the world is a worse place without him.

The Debian package of the Linux kernel has some additional features not present in mainline Linux 3.2. This continues from part 1 and part 2. Real-time scheduling The 'PREEMPT_RT' aka 'linux-rt' patchset provides true real-time scheduling under Linux. This can allow applications to respond to I/O with very low latency and jitter, although the results depend on application design and on the hardware and firmware used. Real-time scheduling is commonly used in industrial control, live audio/video applications, and high-frequency trading systems; it can also be useful for tightly-coupled cluster computing. However, most applications will run more slowly under these kernels, and may not be any more responsive. The Linux 3.2-rt stable series maintained by Steven Rostedt is used to build alternate kernels for amd64 and i386. You can install these using the metapackage linux-image-rt-amd64 or linux-image-rt-686-pae. Hyper-V guest drivers Like most VM hypervisors, Microsoft Hyper-V supports paravirtualised drivers which can provide much better performance than emulation of old hardware devices. Linux 3.2 had quite early versions of these drivers that were relegated to the 'staging' area, but we've updated them to the much improved versions found in Linux 3.4. I hoped we would also be able to package the userland daemon that supports management of Hyper-V guests, but this also needed many fixes and wasn't ready in time.

While almost everyone has already worked with cryptographic signatures, they are usually only used as black boxes, without taking a closer look. This article intends to shed some lights behind the scenes. Let's take a look at some signature. In ascii-armoured form or behind a clearsigned message one often does only see something like this:

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
iQIcBAABAgAGBQJQ8qxQAAoJEH8RcgMj+wLc1QwP+gLQFEvNSwVonSwSCq/Dn2Zy
fHofviINC1z2d/voYea3YFENNqFE+Vw/KMEBw+l4kIdJ7rii1DqRegsWQ2ftpno4
BFhXo74vzkFkTVjo1s05Hmj+kGy+v9aofnX7CA9D/x4RRImzkYzqWKQPLrAEUxpa
xWIije/XlD/INuhmx71xdj954MHjDSCI+9yqfl64xK00+8NFUqEh5oYmOC24NjO1
qqyMXvUO1Thkt6pLKYUtDrnA2GurttK2maodWpNBUHfx9MIMGwOa66U7CbMHReY8
nkLa/1SMp0fHCjpzjvOs95LJv2nlS3xhgw+40LtxJBW6xI3JvMbrNYlVrMhC/p6U
AL+ZcJprcUlVi/LCVWuSYLvUdNQOhv/Z+ZYLDGNROmuciKnvqHb7n/Jai9D89HM7
NUXu4CLdpEEwpzclMG1qwHuywLpDLAgfAGp6+0OJS5hUYCAZiE0Gst0sEvg2OyL5
dq/ggUS6GDxI0qUJisBpR2Wct64r7fyvEoT2Asb8zQ+0gQvOvikBxPej2WhwWxqC
FBYLuz+ToVxdVBgCvIfMi/2JEE3x8MaGzqnBicxNPycTZqIXjiPAGkODkiQ6lMbK
bXnR+mPGInAAbelQKmfsNQQN5DZ5fLu+kQRd1HJ7zNyUmzutpjqJ7nynHr7OAeqa
ybdIb5QeGDP+CTyNbsPa
=kHtn
-----END PGP SIGNATURE-----

This is actually only a form of base64 encoded data stream. It can be translated to the actual byte stream using gpg's --enarmor and --dearmour commands (Can be quite useful if some tool only expects one BEGIN SIGNATURE/END SIGNATURE block but you want to include multiple signatures but cannot generate them with a single gpg invocation because the keys are stored too securely in different places). Reading byte streams manually is not much fun, so I wrote gpg2txt some years ago, which can give you some more information. Above signature looks like the following:

89 02 1C -- packet type 2 (signature) length 540
        04 00 -- version 4 sigclass 0
        01 -- pubkey 1 (RSA)
        02 -- digest 2 (SHA1)
        00 06 -- hashed data of 6 bytes
                05 02 -- subpacket type 2 (signature creation time) length 4
                        50 F2 AC 50 -- created 1358081104 (2013-01-13 12:45:04)
        00 0A -- unhashed data of 10 bytes
                09 10 -- subpacket type 16 (issuer key ID) length 8
                        7F 11 72 03 23 FB 02 DC -- issuer 7F11720323FB02DC
        D5 0C -- digeststart 213,12
        0F FA -- integer with 4090 bits
                02 D0 [....]

Now, what does this mean. First all gpg data (signatures, keyrings, ...) is stored as a series of blocks (which makes it trivial to concatenate public keys, keyrings or signatures). Each block has a type and a length. A single signature is a single block. If you create multiple signatures at once (by giving multiple -u to gpg) there are simple multiple blocks one after the other. Then there is a version and a signature class. Version 4 is the current format, some really old stuff (or things wanting to be compatible with very old stuff) sometimes still have version 3. The signature class means what kind of signature it is. There are roughly two signature classes: A verbatim signature (like this one), or a signature of a clearsigned signature. With a clearsigned signature not the file itself is hashed, but instead a normalized form that is supposed to be invariant under usual modifications by mailers. (This is done so people can still read the text of a mail but the recipient can still verify it even if there were some slight distortions on the way.) Then the type of the key used and the digest algorithm used for creating this signature. The digest algorithm (together with the signclass, see above) describes which hashing algorithm is used. (You never sign a message, you only sign a hashsum. (Otherwise your signature would be as big as your message and it would take ages to create a signature, as asymetric keys are necessarily very slow)). This example uses SHA1, which is no longer recommended: As SHA1 has shown some weaknesses, it may get broken in the not too distant future. And then it might be possible to take this signature and claim it is the signature of something else. (If your signatures are still using SHA1, you might want to edit your key preferences and/or set a digest algorithm to use in your ~/.gnupg/gpg.conf. Then there are some more information about this signature: the time it was generated on and the key it was generated with. Then, after the first 2 bytes of the message digest (I suppose it was added in cleartext to allow checking if the message is OK before starting with expensive cryptograhic stuff, but it might not checked anywhere at all), there is the actual signature. Format-wise the signature itself is the most boring stuff. It's simply one big number for RSA or two smaller numbers for DSA. Some little detail is still missing: What is this "hashed data" and "unhashed data" about? If the signed digest would only be a digest of the message text, then having a timestamp in the signature would not make much sense, as anyone could edit it without making the signature invalid. That's why the digest is not only signed message, but also parts of the information about the signature (those are the hashed parts) but not everything (not the unhashed parts).

Review: Asimov's Science Fiction, April/May 2011

Editor:	Sheila Williams
Issue:	Volume 35, No. 4 & 5
ISSN:	1065-2698
Pages:	192

Williams's editorial this issue is about the tendency of SF to take a rose-colored view of the world, which on the surface seems odd given the tendency of recent SF towards dystopia. But she makes a good point that the portrayal of the past is rose-colored, linking that into the current steampunk trend. She doesn't take the argument quite as far as I'd like, but I'm glad to see editorials raising points like this. I'm inclined to think that a lot of the rose-colored frame of the past is because few of us want to read about real historic conditions at any length, even for edification, because the stench and discomfort isn't fun to read about. Silverberg's column is another discussion of programmatic plot generators, which mostly makes the point that plot ideas are the easy part of writing. James Gunn contributes an extended biography of Isaac Asimov that probably won't be new to long-time genre readers but may fill in some details (although it politely sticks to mostly flattering material). Spinrad's book review column is one of his better ones; it looks at two novels by China Mi ville and two by Ian McDonald and explores differences in world-building. Spinrad predictably makes the case in favor of science fiction with rules and against the New Weird, but the discussion along the way was worth reading. "The Day the Wires Came Down" by Alexander Jablokov: Speaking of steampunk, here's an example. There is even an airship, although the primary technological focus is suspended street cars. Jablokov postulates a city-wide transportation network of suspended carriages called telpher cars, along with a city built around the telpher cables: stores on roofs, windows displaying merchandise to passing cars, and even a history of heated competition and dirty tricks between competing telpher networks. The story is set, as the title would indicate, on the last day of the network. It's being shut down for cost, with some hints that progress is destroying something precious. There is a plot here, revolving around some mysteries of the history of the telpher network and the roles of several people in that history. But the story is primarily a celebration of old technology. It's a rail fan's story recast with a steampunk technology, featuring the same mix of fascination with mechanics and a sense that the intricate details are falling out of common knowledge (and perhaps usefulness). As a story, it's a bit slow-moving, but I enjoyed the elegiac tone. (7) "An Empty House with Many Doors" by Michael Swanwick: This is a very short story, more of an emotional profile, involving a man's reaction to the death of his wife. Oh, and parallel universes. It's sort of the inverse of Niven's classic "All the Myriad Ways." Similar to Niven's story, I found the idea vaguely interesting but the conclusion and emotional reaction unbelievable and alien. (5) "The Homecoming" by Mike Resnick: Resnick tends to yank on the heart-strings rather sharply in his stories, so I knew roughly what to expect when a father comes home to find his son is visiting. A son who, rather against his father's wishes, has been significantly altered to be able to live with aliens. Throw in a mother with serious dementia, and you can probably predict what Resnick does with this. Still, most of the story is a two-sided conversation, and I thought he succeeded in doing justice to both sides, even though one of them was destined to lose. (6) "North Shore Friday" by Nick Mamatas: Illegal Greek immigrants, a family-run system for getting them married off before the INS catch them, government psi probes and eavesdropping on thoughts, joint projects between computer and religion departments, secret government experiments, and even ghosts... this story is a complex mess, with numerous thoughts stuck into small boxes and scattered through the surface story. It's one of those stories where figuring out what's going on, and even how to read the story in a sensible way, is much of the fun. If you find that fun, that is; if not, it will probably be frustrating. I wished there was a bit more plot, but there's something delightful about how much stuff Mamatas packs into it. (6) "Clockworks" by William Preston: This is a prequel to Preston's earlier "Helping Them Take the Old Man Down". Like that story, it's primarily a pulp adventure, but layered with another level of analysis and thoughtfulness that tries to embed the pulp adventure in our understanding of human behavior and the nature of the world, although this one stays a bit more pulp than its predecessor. As with Preston's other story, we don't get directly in the head of the Old Man (here, just called the man, but identifiable from clues in both stories as Doc Savage); instead, the protagonist is a former villain named Simon Lukic who the man hopes to have fixed by operating on his brain. The undercurrent that lies beneath a more typical pulp adventure is the question of whether Lukic is actually healed. I think there was a bit too much daring-do and human perfection, but it's a perfectly servicable pulp story with some depth. (6) "The Fnoor Hen" by Rudy Rucker: If you've read any of Rucker's work before, you probably know what to expect: a mind-boggling blizzard of mathematically-inspired technobabble that turns into vaguely coherent surrealism. (You can probably tell that I'm not much of a fan, although the clear good humor in these stories makes it hard to dislike them too much.) There's a mutated chicken and some sort of alternate mathematical space and then something that seems like magic... I'd be lying if I said that I followed this story. If you like Rucker, this seems like the sort of thing that you'd like. (4) "Smoke City" by Christopher Barzak: At the start of this story, I thought it was going to be an emotional parable about immigration. The progatonist lives two lives: one in our world, and one in the Smoke City of industry, a world of hard labor, pollution, and little reward, with families in both. But nearly all of the story is set within Smoke City, and the parable turns out to be a caustic indictment of industry and its exploitation of labor. I kind of wish Barzak hadn't used rape as a metaphor, but when the captains of industry show up, I can't argue with how deeply and accurately the story shoves in the knife. There isn't much subtlety here, but it's still one of the better stories in this issue. (7) "A Response from EST17" by Tom Purdom: I'm very happy to see Purdom's writing appearing regularly. His stories are always quiet and matter-of-fact, and at first seem to miss emotional zest, but they almost always grow on me. He lets the reader fill in their own emotional reactions to events, and there's always a lot going on. This story is a first-contact story, except that the "humans" here are not human at all. They're automated probes sent by two separate human civilizations, with different programming and different governance algorithms, and they quickly start competing negotiations. The aliens they've discovered similarly have factions, who start talking to the different probes in an elaborate dance of gathering information without giving too much away. The twist is that this pattern has replayed itself many times in the past, and information itself can be a weapon. I enjoyed this one from start to finish. (7) "The One That Got Away" by Esther M. Friesner: Friesner is best known, at least to me, for humorous fantasy, and this story is advertised as such from early on. The first-person protagonist is a prostitute in a seaside town. She's bemused to finally be invited over by a sailor who's been eyeing her all evening, but that sailor has something else in mind than normal business. For much of this story, the fantasy element is unclear; when it finally comes, it was an amusing twist. (7) "The Flow and Dream" by Jack Skillingstead: This is a mildly interesting variation on the old SF story of hibernating humans (on a generation ship or elsewhere) waking up to a transformed world. Here, it's not a ship, it's a planet, and the hiberation was to wait for terraforming rather than for transit. The twist comes from an excessively literal computer and the fun of putting together the pieces. Sadly, the story trails off at the end without much new to say. (5) "Becalmed" by Kristine Kathryn Rusch: "Becalmed" takes place immediately before "Becoming One with the Ghosts" and explains the incident that created the situation explored in that story. The first-person protagonist of "Becalmed" is a linguist, an expert in learning alien languages so that the Fleet can understand the civilizations that it runs across. But something went horribly wrong at their last stop, something that she's largely suppressed, and now she's confined to quarters and possibly in deep trouble. As is the ship; they're in foldspace, and they have been for days. "Becalmed" is structed like a mystery, centered around recovering the protagonist's memories. It's also a bit of a legal procedural; the ship is trying to determine what to do with her and to what degree she's responsible. But the heart of the story is a linguistic and cultural puzzle. This is another great SF story from Rusch, whose name on a cover will make me eager to start reading a new magazine. I love both angles on the universe she's built, but I think I like the Fleet even better than the divers. The Fleet captures some of the magic of the original Star Trek, but with much more mature characters, more believable situations, and a more sensible and nuanced version of the Prime Directive. Rusch writes substantial, interesting plots that hold my interest. I'd love to see more like this. (8) Rating: 7 out of 10

CL-JSON provides an encoder for Lisp data structures and objects to JSON format. Unfortunately, in some case, its default encoding mechanism for CLOS objects isn't exactly doing the right thing. I'll show you how Common Lisp makes it easy to change that. Identifying the problem CL-JSON & CLOS CL-JSON mechanism encoding CLOS object is really neat. Let's see how it works for a simple case:
will produce:
Still using CL-JSON, we can also decode the JSON object to a CLOS object:
That code will return "black". Note that it's also possible to specify which class should be used when decoding objects, but that's beyond the purpose of this article. Postmodern Now, let's introduce Postmodern, a wonderful Common Lisp system providing access to the wonderful PostgreSQL database. It also provides a simple system to map rows in a database to CLOS classes, called DAO for Database access objects. With this, we can easily store our kitten into a table.
If we try to encode this to JSON, it will produce the exact same result seen previously. The problem is what happens when one of our column has a NULL value. Postmodern encodes this using the :null symbol. So this code:
will return:
Fail! The fact that the column is NULL is represented by the :null symbol. And CL-JSON encodes all symbols as string. This is not at all what we want here! Overriding encode-json CL-JSON provides and uses the encode-json method to encode all kind of object. It is defined as a generic function, and a lot of different methods are implemented to handle the different standard Common Lisp types. The one used for standard-object is defined liked that:
All we need to do here, is to create a new method for our kitten objects, that handles correctly the :null case.
With that new method, as soon as we encounter a :null symbol as a value for an object's slot, we replace it by nil. Now if we try to encode another kitten, we'll get:
which is far better for our JavaScript data consumers! In the end, I think that this kind of trick is feasible that easily because of the way CLOS provides its generic method implementation. The fact that methods don't belong to any class makes the extension of every program, library and class so much easier. Doing this in another language like Java would likely by impossible, and in Python it would unlikely be as clean as it is done in Common Lisp. The ability to teach any library about how it should handle your class just by defining a new method is really handy!

Writing recursive descent parsers is easy, especially in Python (just like everything is easy in Python). But Python defines a low limit on the number of recursive calls that can be made, and recursive descent parsers are prone to exceed this limit. We should thus write our parser without real recursion. Fortunately, Python offers us a way out: Coroutines, introduced in Python 2.5 as per PEP 342. Using coroutines and a simple trampoline function, we can convert every mutually recursive set of functions into a set of coroutines that require constant stack space. Let s say our parser looks like this (tokens being an iterator over characters):

def parse(tokens):
    token = next(tokens)
    if token.isalnum():
        return token
    elif token == '(':
         result = parse(tokens)
         if next(tokens) != ')': raise ...
         return result
    else:
         raise ...

We now apply the following transformations:
return X => yield (X)
parse() => (yield parse()) That is, we yield the value we want to return, and we yield a generator whenever we want to call (using a yield expression). Our rewritten parser reads:

def parse(tokens):
    token = next(tokens)
    if token.isalnum():
        yield token
    elif token == '(':
         result = yield parse(tokens)
         if next(tokens) != ')': raise ...
         return result
    else:
         raise ...

We obviously cannot call that generator like the previous function. What we need to introduce is a trampoline. A trampoline is a function that manages that yielded generators, calls them, and passes their result upwards. It looks like this:

def trampoline(generator):
    """Execute the generator using trampolining. """
    queue = collections.deque()
    def schedule(coroutine, stack=(), value=None):
        def resume():
            if 0:
                global prev
                now = stack_len(stack)
                if now < prev:
                    print("Length", now)
                    prev = -1
                elif prev != -1:
                    prev = now
            result = coroutine.send(value)
            if isinstance(result, types.GeneratorType):     # Normal call
                schedule(result, (coroutine, stack))
            elif isinstance(result, Tail):                  # Tail call (if you want to)
                schedule(result.value, stack)
            elif stack:                                     # Return to parent
                schedule(stack[0], stack[1], result)
            else:                                           # Final Return
                return result
        queue.append(resume)
    schedule(generator)
    result = None
    while queue:
        func = queue.popleft()
        result = func()
    return result

This function is based on the code in PEP 342, the difference being that

• we do not correctly propagate exceptions through the stack, but directly unwind to the caller of the parser (we don t handle exceptions inside our parser generators anyway)
• the code actually compiles (code in PEP used value = coroutine.send(value) which does not work)
• the code returns a value (code in PEP was missing a return in schedule)
• we don t use a class, and allow only one function to be scheduled at once (yes, we could get rid of the queue)
• we allow tail calls [where Tail = namedtuple("Tail", ["result"])] to save some more memory.

For a more generic version of that, you might want to re-add exception passing, but the exceptions will then have long tracebacks, so I m not sure how well they will work if you have deep recursion. Now, the advantage of that is that our parser now requires constant stack space, because the actual real stack is stored in the heap using tuples which are used like singly-linked lists in scheme here. So, the only recursion limit is available memory. A disadvantage of that transformation is that the code will run slightly slower for small inputs that could be handled using a normally recursive parser.
Filed under: Python

As there is still some development ongoing in Weblate, it's time to share information about some of the features you can expect in upcoming release. Recently, Weblate gained ability to display changes which caused string to be fuzzy. This currently only works with Gettext PO files as no other format provides this information. If this is available Weblate shows differences between current and previous string, what allows translator to easily spot what has been changed. Example can be seen on Uzbek translation of phpMyAdmin (until somebody translates that). While working on this feature, I've realized, that Django does not include this information in PO files. Fortunately it was pretty easy to fix and I've created a pull request to implement this. Looking at this file made me fix another thing which annoyed me since I've used translations in Django for first time - it generates POT file for every language, making the updates really slow. Fortunately it was also easy to fix and pull request is here. So if you feel makemessages is too slow, this is patch for you. But back to Weblate as there are more changes. For example automatic locking now works more fine grained and uses AJAX to update lock period. This means translation will be unlocked much faster if you close browser with Weblate. Also searching for messages has been updated to include substring search as well as it is sometimes more useful than fulltext search.

Filed under: Django English Weblate 0 comments Flattr this!

Sorry, a post about MacOS again, but this is a problem my boss has been faced with, and judging by what google searches gave us as clues to go on, I thought it would be useful for others to know about this.So here it is: for a couple of weeks, my boss's Mac had been slowing down very painfully, to the point that he was just sitting in front of it waiting for web pages to show up or applications to start... We tried a whole bunch of things, such as disabling spotlight, dropbox, and even the time machine, but nothing worked. Finally, he had a look at Disk Utility that said that the disk was gone beyond repair. Fortunately, a new colleague suggested to restore from the Time Machine, holding Command-R during startup. It took ages (over a slow network), but now the machine is up again, without any slowing down.

A friend wrote me a simple question today. My response turned out to be longer than i expected, but i hope it's useful (and maybe other people will have better suggestions) so i thought i'd share it here too: Angela Starita wrote:

I'd like to save my work in a location where I can access it from any computer. I'm wary of using the mechanisms provided by Google and Apple. Can you suggest another service?

Here's my reply: I think you're right to be wary of the big cloud providers, who have a tendency to inspect your data to profile you, to participate in arbitrary surveillance regimes, and to try to sell your eyeballs to advertisers. Caveat: You have to trust the client machine too But it's also worth remembering that the network service provider is not the only source of risk. If you really mean "accessing your data from any computer", that means the computer you're using to access the data can do whatever it wants with it. That is, you need to trust both the operator of these "cloud" services, *and* the administrator/operating system of the client computer you're using to access your data. For example, if you log into any "secure" account from a terminal in a web caf , that leaves you vulnerable to the admins of the web caf (and, in the rather-common case of sloppily-administered web terminals, vulnerable to the previous user(s) of the terminal as well). Option 0: Portable physical storage One way to have your data so that you can access it from "any computer" is to not rely on the network at all, but rather to carry a high-capacity MicroSD card (and USB adapter) around with you (you'll probably want to format the card with a widely-understood filesystem like FAT32 instead of NTFS or HFS+ or ext4, which are only understood by some of the major operating systems, but not all). Here is some example hardware:

• MicroSD USB adapter ($5)
• 16GB class 10 MicroSD card ($20)

Almost every computer these days has either a microSD slot or a USB port, while some computers are not connected to the network. This also means that you don't have to rely on someone else to manage servers that keep your data available all the time. Note that going the microSD route doesn't remove the caveat about needing to trust the client workstation you're using, and it has another consideration: You'd be responsible for your own backup in the case of hardware failure. You're responsible for your own backup in the case of online storage too, of course -- but the better online companies are probably better equipped than most of us to deal with hardware failure. OTOH, they're also susceptible to some data loss scenarios that we aren't as individual humans (e.g. the company might go bankrupt, or get bought by a competitor who wants to terminate the service, or have a malicious employee who decides to take revenge). Backup of a MicroSD card isn't particularly hard, though: just get a USB stick that's the same size, and regularly duplicate the contents of the MicroSD card to the USB stick. One last consideration is storage size -- MicroSD cards are currently limited to 32GB or 64GB. If you have significantly more data than that, this approach might not be possible, or you might need to switch to a USB hard disk, which would limit your ability to use the data on computers that don't have a USB port (such as some smartphones). Option 1: Proprietary service providers If you don't think this portable physical storage option is the right choice for you, here are a couple proprietary service providers who offer some flavor of "cloud" storage while claiming to not look at the contents of your data:

• wuala
• spideroak

I'm not particularly happy with either of those, though, in part because the local client software they want you to run is proprietary, so there's no way to verify that they are actually unable to access the contents of your data. But i'd be a lot happier with either wuala or spideroak than i would be with google drive, dropbox, or iCloud. Option 2: What i really want I'm much more excited about the network-accessible, free-software, privacy-sensitive network-based storage tool known as git-annex assistant. The project is spearheaded by Joey Hess, who is one of the most skilled and thoughtful software developers i know of. "assistant" (and git-annex, from which it derives) has the advantage of being pretty agnostic about the backend service (many plugins for many different cloud providers) and allows you to encrypt your data locally before sending it to the remote provider. This also means you can put your encrypted data in more than one provider, so that if one of the providers fails for some reason, you can be relatively sure that you have another copy available. But "assistant" won't be ready for Windows or Android for several months (builds are available for Linux and Mac OS X now), so i don't know if it meets the criterion for "accessible from any computer". And, of course, even with the encryption capabilities, the old caveat about needing to trust the local client machine still applies.

This is my monthly summary of my free software related activities. If you re among the people who made a donation to support my work (836.78 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Debian Packaging I uploaded Zim 0.58, WordPress 3.5 (and I had to file a ticket about the availability of sources of minified javascript files again) and another security update for python-django (696535). Speaking of python-django, I forwarded one bug report of Anders Kaseorg concerning Django s bash completion (#695811). I also sponsored the upload of ledgersmb 1.3.25-1. Other Debian work I contributed some patches to improve debian-installer support in live-build. I also added a work-around for bug #652946 in live-installer and committed a fix for this same bug in a jessie branch of partman-target. I also prepared a bugfix for a counter-productive behavior of choose-mirror (#695261, it was not possible to override the codename of the release to install via preseed if you install from a CD with a full base system). I discovered an oddity in the Packages.diff/Index file for the architectures armhf and s390x, I reported it to ftpmasters in #696792. (All those issues were discovered while working for a customer) Debian France I spent quite some time on Debian France this month again. I started by setting up an internal gitolite to manage our accounting/administrative documents. Then I updated galette, the web application that we are using to manage our database of members. In the process, I filed two bugs that we discovered. I immediately tested Galette by registering 4 members that joined during the former mini-Debconf in Paris. We have plans to automate the membership renewal process so we have opened a Paypal account. This month we also cleared the last steps so that I and Sylvestre Ledru have full control on the Debian France bank account. I also registered the new officers at the Tribunal d instance de Sarreguemines . Salt bug reports During the last mini-debconf, I discovered Salt (thanks to Julien Cristau!) and since I had to switch some servers of mine, I took this opportunity to upgrade to wheezy and try out salt at the same time. It took much more time than expected but the result is pleasant. The configuration of all my servers is now well documented/specified in a central Git repository, and moving services is much easier than before. In the process, I filed quite some bugs (#2865, #2851, #2866 and #2875), most of them have been fixed in the 0.11.1 release that just happened. Thanks I wish you a happy new year and all the best for 2013! See you next month for a new summary of my activities.

4 comments Liked this article? Click here. My blog is Flattr-enabled.

This issue is about a bit specific situation in Germany and it's (mainly tax-financed) public service broadcasting. If you think you are not interested, please skip this entry. :) Just beginning from scratch... In Germany we have (like most european countries) a public service broadcasting. Starting with 2013 the system collecting the funding of the public service broadcasting changed from GEZ to Beitragsservice.
To make it short (and a bit simplified) ... it changed from "most 'families' have to pay" to "every menage has to pay". This means, even if you don't have any device to use the service of the german public service broadcasting but are a menage, you have to pay for it. This sitiation is even more worse than pay-TV. Don't get me wrong ... I'm neither against public service broadcasting nor against paying for it in principle. In the rare cases I'm watching TV, this is most likely a channel of the public service broadcasting. Same counts for the rest of my small family. Mostly we are watching news and documentations ... also some movies are watched. The kids are watching KiKA, which we like cause it's Ads-free and it's not just some cartoons with hidden violence. So we most likely benefit from public service broadcasting. Anyways ... in common we are not using "TV" as most of the others may do. We still like some sort of the broadcast, but we are missing it cause various reasons. For our luck there is a service called 'Mediathek' provided by the most german public service broadcasting channels. If you missed a broadcast or are prevented for any reason, you may watch the broadcast there. Unfortunately the timeframe you can access the broadcast is very short, in most cases 7 days. This is manifested in the so called Rundfunkstaatsvertrag, more precisely in the 12. Rundfunk nderungsstaatsvertrag. The 'Tagesschau' has a good explanation about the issue. This means, in simple words, the tax payed broadcasts needs to be deleted after a (in my eyes) short time. WTF?!? Why do we wipe work we all payed for it? Guess what ... it was implemented by lobbying from commercial print media and private (e.g. ads-financed) TV-channels. A comment about a related problem, which I agree, can be found here. Another challenge ... How do you know there is a broadcasting your are interested in? How do you know when it is broadcasted? I remember, my parents was subscribed to a daily newspaper, which included an additional TV paper, my parents-in-law bought also one.
As we are neither subscribed to a daily newspaper nor buying a TV paper, we have to solve that differently (beside that a printed paper won't help us I think). When we use the opportunity to watch TV at the evening, I just use an electronical TV paper (-app). If we find something which fits, we are fine .. if not, we do other things. This sounds a bit like an old school behaviour and it is not, how I would like to use a media today. I like using a feedreader. You can subscribe differnt 'channels' of information feeds and it is a way I would love to use TV. In the past I used a tool called 'Mediathek', which is a tool for Mac OSX. With this tool it was easily possible to find broadcasts, you could subscribe to so called 'channels', which means broadcasts of a same series for example. You could also mark them as "unread" und "read" .. something like a feedreader.
On the other way, it was also possible to list recent broadcast per genres and popular broadcast of the recent and the last week. This was very comfortable as this was working for several mediathek sites for german public service broadcasting channels in one central place. For every broadcast you could just view it or download it to view it later. You could even download subscribed channels automatically. After using the Mediathek-App and being satisfied by the Mediathek sites of the german public service broadcasting channels for several years, this week the server(s) behind the App was shut down. What the f....?!? On Twitter the author stated that the shutdown was cause by problems with the copyright law. Das Erste, ZDF and WDR denied that they has anything todo with this recent issue. The question is ... what is the direct cause for shutting down such a good service. This smells a bit like the PirateBay issue, but in this case the content which is Mediathek providing access to isn't pirated, but legal hosted. So where is the real issue? In my mind, the german citizen has payed to produce this content and should have the right to access the content is an usable way. The Mediathek App did provide such a way and it is really pissing me off that this access isn't available anymore, at least for this reason. Actually I don't see any solution to access our payed content in a decent way. Footnote: I think it's obscenely, that publishing houses and commercial broadcaster trying to gain their turnover at the expense of the german citizen by forcing the german legislature to implement Depublication,.

I've been reading "The Millionaire Next Door" and have so far found it to be a pleasant book with a good message: don't waste your money on silly things and appearance (fancy suits, fancy cars, expensive boats, etc...), save what you do earn consistently and constantly, invest wisely, and so on. Wikipedia has a good summary:
http://en.wikipedia.org/wiki/The_Millionaire_Next_Door One of the things I like about it is that it focuses on "ordinary" wealthy people, those with a million or more in the bank, but not the Warren Buffets or Bill Gates types that are extreme statistical outliers. There are plenty of people in the US who have done well by themselves by slowly but surely putting together enough money to be financially independent, without, however, being in the spotlight. As the book says, these are the kind of people who maybe own a local chain of businesses doing something fairly ordinary, but doing it well enough to succeed. They may very well not live in a fancy house, nor drive an expensive car, or otherwise outwardly draw much attention to themselves. The world of software does not revolve around "dressing for success" (you noticed?), but we do tend to focus on the "big winners". Gates, Jobs, Zuckerberg, Larry & Sergey, Larry Ellison, and so on are the stars of the show. Of course, the economics of software being what they are, instances of winner-take-all markets with one big fish and a lot of also-rans are not uncommon. However, that is not the only story, and I think it'd be interesting to know more about those in our industry who have accumulated significant wealth, yet are not the guys with more money than they could possibly ever spend on things that aren't, say, country-sized chunks of real-estate. I'm guessing they'd fall into these categories:

• Highly paid workers who have consistently saved over the years. There are examples in the aforementioned book about people with relatively low salaries who happened to be very frugal and invest well (and have had some luck in their investments too). These people would probably tend to be older, as it takes a while to save up that kind of money, and since this industry is so young with so much turnover, I would not think there would be a lot of people out there like this, but who knows, maybe there are a bunch of IBMers with this kind of story.
• Those who got in on the right IPO, like Google or Facebook or something like that. These events not only generate billions for those at the top of the heap, but for the right person at the right place at the right time, can mean significant wealth even without being in the upper echelons of the company. My suspicion is that this kind of IPO, where everyone cashes out, is not common enough to have a lot of people in this category, but who knows, maybe it adds up over the years.
• Those who own or started software firms that do something that's not very visible, but nonetheless dominates some particular niche. This is where I'd guess most of them would be, but I certainly have no data or even anecdotes to back this up.

It'd be very interesting to gather some actual data on this, although I'm not in a position to do so myself - I wouldn't even really know where to start. As I age, I think the third category has begun to seem appealing in many ways - I'm simply not cut out for the Big Company life, and I'm not interested in living in Silicon Valley and going "all in" on the latest startup - I already did that, and while it was fun and I don't regret it, it's not the kind of thing I'd want to do now that I'm married and have kids. Incidentally, this more relaxed, under the radar approach is exactly what is expoused in one of my favorite books of the past few years, Start Small, Stay Small. Edit : I finished reading the book and reviewed it here: http://davids-book-reviews.blogspot.com/

If you rip your DVD's with a unix-y name like longish-show-name-season-01-01.m4v then the iPad and iPhone video player only shows the prefix of the file-name to it's string limit, so you only ever see longish-show-na and can never tell what episode is what in the overview list. You can edit the files in iTunes to add meta-data information, but that's annoying and non-unix-y. It is of course easy, but I went down several dead-ends with tools that didn't recognise file-extensions, corrupted files and plain didn't work. The package I had most success with is mp4v2-utils and the mp4tags utility. The help is fairly self-explanatory, but something like

mp4tags -show "Show Name" -season <season> -episode <episode> -type "tvshow"  file.m4v

will mean that when you import your file into iTunes it displays correctly as a TV show (and not a movie), is grouped correctly into series and has a name you can see. Unfortunately, it doesn't seem my Nexus 7 tablet picks up these tags. And both players interfaces remain terrible at indicating which episodes have been watched and which have not. But this has allowed me to re-visit my old box-set of Yes Minister which, despite a few odd references to Quango's still seems as relevant as ever!

For the year of 2012, I finished and reviewed 60 books, the same number as 2011. Given how stressful and chaotic much of the year was, I count this a major triumph. The hardest part was not the reading but the review writing; the end of the year required a significant push to finish writing reviews of all the books I'd read that year, and at one point I was reviewing things I'd read more than two months earlier. But I can enter 2013 entirely caught-up. This continues to feel like about the right pace, striking a balance between reading enough that I can pursue multiple reading goals at the same time, while leaving enough time for other projects and video games. I gave three novels a 10 out of 10 this year, but the stand-out even among that group was Elizabeth Wein's Code Name Verity. Not only was it my favorite book of the year, but it was one of the best books I've ever read. The other two 10-rated books are also highly recommended, of course: Matt Ruff's fascinating novel of multiple personalities, Set This House in Order; and C.J. Cherryh's SF classic, Cyteen. The latter was a re-read in advance of reading the sequel, Regenesis, which I also recommend if you've read and liked the last third of Cyteen. Other fiction highlights of the year were China Mi ville's Embassytown and Suzanne Collins's The Hunger Games triology, particularly the third book, Mockingjay. Embassytown continues the trend from Mi ville's The City & The City of tighter, faster-moving novels while moving into space opera and first contact territory. It keeps Mi ville in the top rank of current SFF writers, despite having some suspension of disbelief problems. Collins's world-building in The Hunger Games also has suspension of disbelief problems, but I understood why the series was so popular. I was caught by surprise by its look at violence and its after-effects and was particularly impressed by Mockingjay and Collins's chosen ending. I think this is a popular series that lives up to the hype. There were no 10 ratings in non-fiction this year, but several books nonetheless stood out. John Kenneth Galbraith's The Affluent Society is a thought-provoking book that questions the foundation of a consumer-oriented economy while perceptively pointing out how it distorts choices away from public goods. Susan Cain's Quiet is a passionate defense of introversion in an extroverted world. And, finally, Joshua Bloch's Effective Java and Damian Conway's Perl Best Practices are both insightful looks at the good and bad of their respective languages and taught me a great deal as a practicing programmer. One final highlight to mention: Eclipse Phase by Posthuman Studios is an excellent RPG sourcebook, at least from the perspective of interesting world-building. I haven't played it, but I thoroughly enjoyed reading it (and have acquired nearly all of the supplements). The full analysis includes some additional personal reading statistics, probably only of interest to me.