Today in #debian-mentors, a brief discussion about screencasts came up. Some people felt that a series of short screencasts might be a nice addition to the existing Debian Developer documentation. After seeing the popularity of of Ubuntu's various video series over the years, I tend to agree that video documentation can be a nice way to supplement traditional text-based documentation. No, this is not designed to replace the existing documentation; it is merely designed to provide an alternative method for people to learn the material. From past experience, I know that if I were to start making screencasts by myself, I would end up making one or two before getting bored and moving on to a new project. That is why I am posing a challenge to the Debian community: I will make one screencast for every screencast made by someone in the Debian community. These screencasts can and should be short (only a couple of minutes in length) and focus on accomplishing one small task or explaining a single concept related to Debian Development. I discovered a Debian Screencasts page on the Debian wiki that links to Ubuntu's guide to creating screencasts. While I have done some traditional documentation work in the past, I have never really tried doing screencasts. As a result, this will be a bit of a learning experience for me at first. If you are interested in taking me up on this challenge, simply send me an email (nhandler [at] ubuntu [dot] com) or poke me on IRC (nhandler@ freenode,oftc ). This will allow us to coordinate and plan the screencasts. As I mentioned earlier, I have no experience creating screencasts, so if you are familar with the Debian Development processes and think this challenge sounds interesting, feel free to contact me.

I've written rather a lot about how we're implementing our Secure Boot support, but there's been a range of subtle changes over time. Here's an overview of how it all fits together.Secure Boot and signingSecure Boot is part of the UEFI specification. When enabled, systems will refuse to boot any UEFI executables (such as an OS bootloader) unless they've been signed by a trusted key. The only trusted key effectively guaranteed to be present is Microsoft's, and Microsoft will sign binaries if you have access to the Windows hardware dev center. You gain access to this by purchasing a certificate from Verisign that verifies your identity and then use that to create a developer account at the Sysdev website. Once you've agreed to the legal contracts, you can submit a UEFI executable and Microsoft will then send you back a signed one. The signature is added to the end of the binary, so it's trivial to verify that your executable hasn't otherwise been modified in the process.
Avoiding signing round tripsWe didn't want to have to get binaries re-signed every time we updated our bootloader or kernel, so we came up with a compromise approach. We implemented a shim bootloader (cunningly called "Shim") which is signed by Microsoft and includes a copy of a public key that's under our control. Shim will launch any binary signed with either a key installed in the system firmware or the public key built into it. This allows us to build and sign all other binaries ourselves.
Preventing Secure Boot circumventionSecure Boot is intended to prevent scenarios where untrusted code can be run before an OS kernel. Locking down the boot environment avoids the simple case of using a modified bootloader, and requiring that the kernel also be signed avoids simply booting arbitrary attack code that looks roughly like a kernel but does nothing to prevent the more complex case of using a running kernel to launch another kernel. To do that we've had to restrict the behaviour of the kernel in Secure Boot environments. Direct hardware access from userspace has been blocked, and modules must be signed by a key the kernel trusts.
Providing user control over trustMicrosoft requires that it be possible to modify the key database on all Windows-certified x86 systems, but the UI to do so may be inconsistent and awkward. Suse developed a plan to avoid this, involving adding an additional key database to the system. The user enrols a key from the running OS and enters a password. On the next reboot, shim detects that a key is awaiting enrolment and drops to a prompt. If the user does nothing or just hits enter, the system continues to boot. If the user chooses to enrol a new key, the user is asked to re-enter the password in order to confirm that they're the same user that initiated the original request. If successful, the key is then copied into a boot services variable which cannot be directly modified from the OS. Anything signed with the user's key will then be trusted.
Providing user control over signature verificationSome users (such as kernel developers) may be hampered by Secure Boot - forcing them to sign every new kernel they build is time consuming and unhelpful. Windows-certified x86 hardware will be required to provide an option to disable Secure Boot, but again that UI may be inconsistent. We've added an option where a user can generate a request to disable signature validation, again requiring a password. On next boot the user must explicitly choose the "Toggle signature validation" menu item and will then be requested to enter three randomly chosen characters from their password. After confirmation, signature validation in Shim will then be disabled.
Supporting other distributionsNot all distributions will want (or be able) to sign via Microsoft. They'll be able to take any other signed version of Shim and put it on their boot media. If their second stage bootloader is signed by a key that Shim doesn't trust, Shim will pop up a menu permitting the user to choose to enrol a new key off the install media. The user can select this, navigate a file browser and select the key. After confirmation, the key will be added to Shim's trusted key list. The user can then install the distribution.
Third party modulesThere's two approaches here. The first is for the third party to provide a module key that the user can install into Shim's database. The kernel will import this at boot time and trust any modules signed with it. The second is for the third party to provide a key signed by Microsoft. David Howells has come up with with the idea of utilising this approach of embedding a public key in a pe binary, with the kernel then having support for importing this key, noticing that it's signed by a trusted key and adding it to the kernel keyring.
How about ARM?We're not planning on supporting Secure Boot on ARM, for two reasons. The first is that Microsoft require that it be impossible to disable Secure Boot on ARM, and we don't want to support systems unless it's guaranteed that the user will be able to run what they want. The second is that there's no strong expectation that the signing key used for signing third party UEFI binaries will be present on ARM systems.
What are other distributions doing?As far as I know, Suse and Fedora will be shipping the same code. Ubuntu is shipping an older version of Shim but should pick up the local key management code in the next release. The only significant difference is that Ubuntu doesn't require that kernel modules be signed.

comments

It's not difficult to write a UEFI application under Linux. You'll need three things:

• An installed copy of the gnu-efi library and headers
  
• A copy of the UEFI specification from here
  
• Something running UEFI (OVMF is a good option if you don't have any UEFI hardware)

So, let's write a trivial UEFI application. Here's one from the gnu-efi source tree:

#include <efi.h>
#include <efilib.h>
EFI_STATUS
efi_main (EFI_HANDLE image, EFI_SYSTEM_TABLE *systab)
 
        SIMPLE_TEXT_OUTPUT_INTERFACE *conout;
        conout = systab->ConOut;
        InitializeLib(image, systab);
        uefi_call_wrapper(conout->OutputString, 2, conout, L"Hello World!\n\r");
        return EFI_SUCCESS;
 

The includes are fairly obvious. efi.h gives you UEFI functions defined by the specification, and efilib.h gives you functions provided by gnu-efi. The runtime will call efi_main() rather than main(), so that's our main function. It returns an EFI_STATUS which will in turn be returned to whatever executed the binary. The EFI_HANDLE image is a pointer to the firmware's context for this application, which is used in certain calls. The EFI_SYSTEM_TABLE *systab is a pointer to the UEFI system table, which in turn points to various other tables.

So far so simple. Now things get interesting. The heart of the UEFI programming model is a set of protocols that provide interfaces. Each of these interfaces is typically a table of function pointers. In this case, we're going to use the simple text protocol to print some text. First, we get a pointer to the simple text output table. This is always referenced from the system table, so we can simply assign it. InitializeLib() just initialises some internal gnu-efi functions - it's not technically required if we're purely calling native UEFI functions as we are here, but it's good style since forgetting it results in weird crashes.

Anyway. We now have conout, a pointer to a SIMPLE_TEXT_OUTPUT_INTERFACE structure. This is described in section 11.4 of the UEFI spec, but looks like this:

typedef struct _EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL  
    EFI_TEXT_RESET Reset;
    EFI_TEXT_STRING OutputString;
    EFI_TEXT_TEST_STRING TestString;
    EFI_TEXT_QUERY_MODE QueryMode;
    EFI_TEXT_SET_MODE SetMode;
    EFI_TEXT_SET_ATTRIBUTE SetAttribute;
    EFI_TEXT_CLEAR_SCREEN ClearScreen;
    EFI_TEXT_SET_CURSOR_POSITION SetCursorPosition;
    EFI_TEXT_ENABLE_CURSOR EnableCursor;
    SIMPLE_TEXT_OUTPUT_MODE *Mode;
  EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL;

The majority of these are functions - the only exception is the SIMPLE_TEXT_OUTPUT_MODE pointer, which points to the current mode. In an ideal world you'd be able to just call these directly, but sadly UEFI and Linux calling conventions are different and we need some thunking. That's where the uefi_call_function() call comes in. This takes a UEFI function as its first argument (in this case, conout->OutputString), the number of arguments (2, in this case) and then the arguments. Checking the spec for OutputString, we see this:

typedef
EFI_STATUS
(EFIAPI *EFI_TEXT_STRING) (
    IN EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL *This,
    IN CHAR16 *String
);

The first argument is a pointer to the specific instance of the simple text output protocol. This permits multiple instances of the same protocol to exist on a single machine, without having to duplicate all of the code. The second argument is simply a UCS-2 string. Our conout pointer is a pointer to the protocol, so we pass that as the first argument. For the second argument, we pass L"Hello World!\n", with the L indicating that this is a wide string rather than a simple 8-bit string. uefi_call_function() then rearranges these arguments into the appropriate calling convention and calls the UEFI function. The firmware then prints "Hello World!" on the screen. Success.

(Warning: uefi_call_function() does no type checking on its arguments, so if you pass in a struct instead of a pointer it'll build without complaint and then give you a bizarre error at runtime)

Finally, we clean up by simply returning EFI_SUCCESS. Nothing more for us to worry about. That's the simple case. What about a more complicated one? Let's do something with that EFI_HANDLE that got passed into efi_main().

#include <efi.h>
#include <efilib.h>
EFI_STATUS
efi_main (EFI_HANDLE image, EFI_SYSTEM_TABLE *systab)
 
        EFI_LOADED_IMAGE *loaded_image = NULL;
        EFI_GUID loaded_image_protocol = LOADED_IMAGE_PROTOCOL;
        EFI_STATUS status;
        InitializeLib(image, systab);
        status = uefi_call_wrapper(systab->BootServices->HandleProtocol,
                                3,
                                image, 
                                &loaded_image_protocol, 
                                (void **) &loaded_image);
        if (EFI_ERROR(status))  
                Print(L"handleprotocol: %r\n", status);
         
        Print(L"Image base        : %lx\n", loaded_image->ImageBase);
        Print(L"Image size        : %lx\n", loaded_image->ImageSize);
        Print(L"Image file        : %s\n", DevicePathToStr(loaded_image->FilePath));
        return EFI_SUCCESS;
 

UEFI handles can have multiple protocols attached to them. A handle may represent a piece of physical hardware, or (as in this case) it can represent a software object. In this case we're going to get a pointer to the loaded image protocol that's associated with the binary we're running. To do this we call the HandleProtocol() function from the Boot Services table. Boot services are documented in section 6 of the UEFI specification and are the interface to the majority of UEFI functionality. HandleProtocol takes an image and a protocol identifier, and hands back a pointer to the protocol. UEFI protocol identifiers are all GUIDs and defined in the specification. If you call HandleProtocol on a handle that doesn't implement the protocol you request, you'll simply get an error back in the status argument. No big deal.

The loaded image protocol is fairly different to the simple text output protocol in that it's almost entirely data rather than function pointers. In this case we're printing the address that our executable was loaded at and the size of our relocated executable. Finally, we print the path of the file. UEFI device paths are documented in section 9.2 of the UEFI specification. They're a binary representation of a file path, which may include the path of the device that the file is on. DevicePathToStr is a gnu-efi helper function that converts the objects it understands into a textual representation. It doesn't cover the full set of specified UEFI device types, so in some corner cases it may print "Unknown()".

That covers how to use protocols attached to the image handle. How about protocols that are attached to other handles? In that case we can do something like this:

#include <efi.h>
#include <efilib.h>
EFI_STATUS
efi_main (EFI_HANDLE image_handle, EFI_SYSTEM_TABLE *systab)
 
        EFI_STATUS status;
        EFI_GRAPHICS_OUTPUT_PROTOCOL *gop;
        EFI_GUID gop_guid = EFI_GRAPHICS_OUTPUT_PROTOCOL_GUID;
        InitializeLib(image_handle, systab);
        status = uefi_call_wrapper(systab->BootServices->LocateProtocol,
                                   3,
                                   &gop_guid,
                                   NULL,
                                   &gop);
        Print(L"Framebuffer base is at %lx\n", gop->Mode->FrameBufferBase);
        return EFI_SUCCESS;
 

This will find the first (where first is an arbitrary platform ordering) instance of a protocol and return a pointer to it. If there may be multiple instances of a protocol (as there often are with the graphics output protocol) you should use LocateHandleBuffer() instead. This returns an array of handles that implement the protocol you asked for - you can then use HandleProtocol() to give you the instance on the specific handle. There's various helpers in gnu-efi like LibLocateHandle() that make these boilerplate tasks easier. You can find the full list in /usr/include/efi/efilib.h.

That's a very brief outline of how to use basic UEFI functionality. The spec documents the full set of UEFI functions available to you, including things like direct block access, filesystems and networking. The process is much the same in all cases - you locate the handle that you want to interact with, open the protocol that's installed on it and then make calls using that protocol.Building these examplesBuilding these examples is made awkward due to UEFI wanting PE-COFF binaries and Linux toolchains building ELF binaries. You'll want a makefile something like this:

ARCH            = $(shell uname -m   sed s,i[3456789]86,ia32,)
LIB_PATH        = /usr/lib64
EFI_INCLUDE     = /usr/include/efi
EFI_INCLUDES    = -nostdinc -I$(EFI_INCLUDE) -I$(EFI_INCLUDE)/$(ARCH) -I$(EFI_INCLUDE)/protocol
EFI_PATH        = /usr/lib64/gnuefi
EFI_CRT_OBJS    = $(EFI_PATH)/crt0-efi-$(ARCH).o
EFI_LDS         = $(EFI_PATH)/elf_$(ARCH)_efi.lds
CFLAGS          = -fno-stack-protector -fpic -fshort-wchar -mno-red-zone $(EFI_INCLUDES)
ifeq ($(ARCH),x86_64)
        CFLAGS  += -DEFI_FUNCTION_WRAPPER
endif
LDFLAGS         = -nostdlib -znocombreloc -T $(EFI_LDS) -shared -Bsymbolic -L$(EFI_PATH) -L$(LIB_PATH) \
                  $(EFI_CRT_OBJS) -lefi -lgnuefi
TARGET  = test.efi
OBJS    = test.o
SOURCES = test.c
all: $(TARGET)
test.so: $(OBJS)
       $(LD) -o $@ $(LDFLAGS) $^ $(EFI_LIBS)
%.efi: %.so
        objcopy -j .text -j .sdata -j .data \
                -j .dynamic -j .dynsym  -j .rel \
                -j .rela -j .reloc -j .eh_frame \
                --target=efi-app-$(ARCH) $^ $@

There's rather a lot going on here, but the important stuff is the CFLAGS line and everything after that. First, we need to disable the stack protection code - there's nothing in the EFI runtime for it to call into, so we'll build a binary with unresolved symbols otherwise. Second, we need to build position independent code, since UEFI may relocate us anywhere. short-wchar is needed to indicate that strings like L"Hi!" should be 16 bits per character, rather than gcc's default of 32 bits per character. no-red-zone tells the compiler not to assume that there's 256 bytes of stack available to it. Without this, the firmware may modify stack that the binary was depending on.

The linker arguments are less interesting. We simply tell it not to link against the standard libraries, not to merge relocation sections, and to link in the gnu-efi runtime code. Nothing very exciting. What's more interesting is that we build a shared library. The reasoning here is that we want to perform all our linking, but we don't want to build an executable - the Linux executable runtime code would be completely pointless in a UEFI binary. Once we have our .so file, we use objcopy to pull out the various sections and rebuild them into a PE-COFF binary that UEFI will execute.

comments

A friend of mine recently reminded me of the OpenPGP smartcard v2, and told me that it was perfectly able to handle 4096 bit RSA keys (provided you have GnuPG v2.0.18+). I had the opportunity to play with one a little, and notice it was super easy to use it for ssh authentication, especially since I already use gpg-agent as my ssh-agent (it should be easy to use a purely software authentication key as ssh key with GnuPG 2.1). So I decided to buy two of them and try to switch my main key (0x71ef0ba8) to it. The cards arrived this weekend, and I was able to play with it a little. I didn't log every command I typed, but it was pretty easy, in the end. What I decided to do was to use one smartcard for every day usage, and one only for key signing. So basically, I would generate three (signing, encryption, authentication) subkeys, put them on smartcard 1, then put the primary key on smartcard 2. Then erase the private parts, and only keep them on smartcards. In case it interests people, here the somehow detailed steps. Note that everywhere 'gpg' means 'gpg2' on Debian, we really need GnuPG v2 for correct smartcard handling. You'd better use gpg-agent too, although it doesn't seem mandatory.

1. make a backup! As we're gonna play with private parts (!), it's always a good idea to have backups. And it'll be useful to have one later, in case there's a problem with the smartcards. You can do a copy of your complete ~/.gnupg folder, but I simply did:

   corsac@scapa: umask 066
   corsac@scapa: gpg -o 71ef0ba8.gpg --export-secret-keys 71ef0ba8
   

2. Add three subkeys. Skip this is you already have subkeys (you usually already have an encryption subkey, but I wanted to switch to a new one too) --expert is needed in order to chose capabilities.

   corsac@scapa: gpg --expert --edit-key 71ef0ba8
   gpg> addkey
   Please select what kind of key you want:
      (3) DSA (sign only)
      (4) RSA (sign only)
      (5) Elgamal (encrypt only)
      (6) RSA (encrypt only)
      (7) DSA (set your own capabilities)
      (8) RSA (set your own capabilities)
   Your selection? 8
   Possible actions for a RSA key: Sign Encrypt Authenticate 
   Current allowed actions: Sign Encrypt 
      (S) Toggle the sign capability
      (E) Toggle the encrypt capability
      (A) Toggle the authenticate capability
      (Q) Finished
   Your selection? e
   Possible actions for a RSA key: Sign Encrypt Authenticate 
   Current allowed actions: Sign 
      (S) Toggle the sign capability
      (E) Toggle the encrypt capability
      (A) Toggle the authenticate capability
      (Q) Finished
   Your selection? Q
   RSA keys may be between 1024 and 4096 bits long.
   What keysize do you want? (2048) 
   Requested keysize is 2048 bits
   Please specify how long the key should be valid.
            0 = key does not expire
           = key expires in n days
         w = key expires in n weeks
         m = key expires in n months
         y = key expires in n years
   Key is valid for? (0) 1y
   Key expires at dim. 27 oct. 2013 20:38:44 CET
   Is this correct? (y/N) y
   Really create? (y/N) y
   We need to generate a lot of random bytes. It is a good idea to perform
   some other action (type on the keyboard, move the mouse, utilize the
   disks) during the prime generation; this gives the random number
   generator a better chance to gain enough entropy.
   

   Repeat this for encryption and authentication subkeys. Then save and send the key to keyservers

   gpg> save
   corsac@scapa: gpg --send-keys 71ef0ba8
   

3. Next, we'll switch to the smartcard part. I use a Gemalto PC ExpressCard reader which is perfectly recognized under Debian. You just need few tools:

   root@scapa: ~# apt-get install pcscd scdaemon
   

   Plug the reader, insert the card, make sure it's detected:

   corsac@scapa: gpg --card-status
   Application ID ...: D2760001240102000005000016A10000
   Version ..........: 2.0
   Manufacturer .....: ZeitControl
   ...
   

   You can edit various parameter (name etc.) and change the PINs using gpg:

   corsac@scapa: gpg --change-pin
   corsac@scapa: gpg --card-edit
   

4. Then we'll put the subkeys in the first smartcard. It might be a good idea to export again the private keys for backups.

   corsac@scapa: gpg -o 71ef0ba8.gpg --export-secret-keys 71ef0ba8
   

5. We'll now use the keytocard gpg command to move the private parts on the smartcard:

   corsac@scapa: gpg --edit-key 71ef0ba8
   gpg> key 1 # select encryption subkey
   gpg> keytocard
   gpg> key 2 # select signature subkey
   gpg> keytocard
   gpg> key 3 # select authentication subkey
   gpg> keytocard
   gpg> save
   

   A quick check on the card now reveals that it's populated:

   corsac@scapa: gpg --card-status
   Application ID ...: D2760001240102000005000016A10000
   Version ..........: 2.0
   Manufacturer .....: ZeitControl
   Serial number ....: 000016A1
   Name of cardholder: Yves-Alexis Perez
   Language prefs ...: fr
   Sex ..............: unspecified
   URL of public key : http://www.corsac.net/71ef0ba8.asc
   Login data .......: corsac
   Signature PIN ....: forced
   Key attributes ...: 2048R 2048R 2048R
   Max. PIN lengths .: 32 32 32
   PIN retry counter : 3 3 3
   Signature counter : 7
   Signature key ....: 9745 B022 7323 81FE 9E7E  AFF5 6DDB 53F2 A675 C0A5
         created ....: 2012-10-27 11:24:07
   Encryption key....: F7E0 078F EA1A 5F23 92E0  20B3 A83A D136 D98D 0D9F
         created ....: 2012-10-27 11:27:01
   Authentication key: 8CFD D478 AB4A 16F8 F0EC  CD33 24E2 3B5C CC0E 273D
         created ....: 2012-10-17 14:29:18
   General key info..: pub  2048R/A675C0A5 2012-10-27 Yves-Alexis Perez 
   sec>  4096R/71EF0BA8  created: 2009-05-06  expires: never     
                         card-no: 0005 000016A2
   ssb   4096g/36E31BD8  created: 2009-05-06  expires: never     
   ssb>  2048R/CC0E273D  created: 2012-10-17  expires: 2013-10-27
                         card-no: 0005 000016A1
   ssb>  2048R/A675C0A5  created: 2012-10-27  expires: 2013-10-27
                         card-no: 0005 000016A1
   ssb>  2048R/D98D0D9F  created: 2012-10-27  expires: 2013-10-27
                         card-no: 0005 000016A1
   

6. At that point, the private part is replaced by a stub in the secret keyring, so when you export them, you only export stubs which you can then use anywhere without actually giving your private key. So now is a good idea to export the subkeys so you can import them on other boxes:

   corsac@scapa: gpg -o 71ef0ba8-subkeys.gpg --export-secret-subkeys 71ef0ba8
   

   Note that only the subkeys private parts have been moved to the card, not the primary one, so you're still able to sign keys. Here, you have multiple choices. You can simply erase the private key (and later re-import the stubs) and use the offline copy made above when you need to sign another key.
7. What I did is something else. I've put the primary key on my second OpenPGP smartcard. That way, I won't lose it, it'll be kept safely in my house, but still be on a hardware token where it won't come out. The procedure for doing is so is exactly the same as above. First take a backup (in case you didn't do it first, do it now since after the keytocard command you won't have a backup of your primary key and there'll be no way to extract it from the smartcard. Then put the new smartcard in the reader, edit the key (don't select a subkey) and run the keytocard command. After that, running gpg --export-secret-keys will export the stub and not the private part of your primary key.

In the end, it seems that everything is running fine. Only issue is that scdaemon is sometime not behaving nicely (especially after a card change or or suspend/resume cycle). I didn't yet report a bug but you might want to kill it in case it's stuck. You can also use the authentication subkey for ssh logins. When the card is inserted, the authentication subkey appears automatically (through the magic of gpg-agent):

corsac@scapa: ssh-add -L
ssh-rsa AAAAB3NzaC1yc2EA... cardno:0005000016A1 And now you can add it to your various authorized_keys and use the smartcard for SSH.

(This post contains some discussion of rape and sexual assault but does not go into any specifics)

There was a brief controversy at Linux.conf.au back in 2011. The final keynote speaker gave a compelling presentation on online privacy, including some slides containing sexualised imagery. This was against the terms of the conference policies, and resulted in an apology from the conference organisers and the speaker. The situation was unfortunate but well handled, and that should have been the end of it.

Afterwards, there was some pushback on the conference mailing list. Concerns were raised about the policy being overly restrictive and the potential for it to be used to stifle speech that influential groups disagreed with. I don't agree with these arguments, but discussion of why policies have been implemented is completely natural and provides an opportunity for a community to determine what its expected standards are.

And then Ted Ts'o effectively called rape victims liars[1]. At first I assumed that this was just some sort of horrific failure to understand the implications of what he was saying, so I emailed him to check. The reply I got drew a pretty clear distinction between the case of a drunk college student raping another drunk college student in their room and the case of knifepoint rape in a dark park. You know, the difference between accidental rape and rape rape. The difference between the one any of us might have done and the one that only bad people do. Legitimate rape and the "rape" that those feminists talk about. The distinction that lets rapists convince themselves that they didn't really rape anyone because they weren't holding a knife at the time.

Ted Ts'o argues that only a small percentage of rape really counts as what people think of as rape. Ted Ts'o is a rape apologist.

There's an ongoing scandal in the UK at the moment. A well known DJ, Jimmy Savile, died last year. He grew up in a working class family, but through hard work and natural talent was one of the most significant figures in promoting pop music in the UK in the 50s and 60s, and worked in various parts of the BBC for the best part of 30 years. He spent significant amounts of time raising money for charity, and it's estimated that he raised over 40 million for various causes. Since his death, around 300 people have accused him of sexually abusing them. The BBC is desperately trying to explain why it cancelled an expose shortly before it aired. Multiple people who worked there at the time claim that everyone knew he was involved in indecent activities, but saying anything would risk both their career and the charities that depended on his fundraising. Nobody said anything, and he was allegedly free to continue his abuse.

Ted Ts'o is a significant figure in the Linux kernel community. He has expressed abhorrent beliefs that damage that community. Condemnation was limited to a mailing list with limited readership, meaning, effectively, that nobody said anything. Last week the Ada Initiative published a blog post pointing out the damage that did, and I realised that my effective silence was not only helping to alienate 50% of the population from involving themselves with Linux, it was also implicitly supporting my community leadership. I was giving the impression that I was basically fine with our community leaders telling people that it wasn't really rape if you were both drunk enough. I was increasing the chances of members of our community being sexually assaulted. Silence is endorsement. Saying nothing is not ok.

In the absence of an apology and explanation from Ted, I'll be interacting with him to the bare minimum that I'm compelled to as a result of my job. I won't be attending any Linux Foundation events he's involved in organising. If I'm running any events, I won't be inviting him. At a time when we're finally making progress in making our community more open and supportive, we don't need leaders who undermine that work. Support organisations who encourage that progress, not the people who help drag us back.

Footnotes

[1]The original archive has vanished. I've put up a copy of the relevant thread here. Throughout, Ted states that he's actually arguing against the idea that women need to be frightened of sexual assault, and not against the definition of rape. Except saying things like This one does a pretty good job of taking apart the Koss / Ms. Magazine study, which is the source for the "1 in 4" number. For example, it points out that over half of those cases were ones where undergraduates were plied with alcohol, and did not otherwise involve using physical force or other forms of coercion is difficult to read in any way other than "Half of the people you're counting as having been raped haven't really been raped", and favourably referring to an article that asserts that the rate of false rape reports is probably close to 50% is pretty strong support for the idea that many rape victims are liars.

(Update 2012/10/30: Adam Williamson suggests in this comment that this mail is a better example of Ted's behaviour - there's some explicit victim blaming and a lot of "Is that rape" questioning with the obvious implication that the answer should be "no". Ted Ts'o is a victim blaming rape apologist.)

(Update 2012/11/05: It's been suggested that I haven't been sufficiently clear about which of Ted's statements justify my claims. So, here we go.

In this mail, Ted links to and endorses this article. He explicitly links to it because of its treatment of rape statistics. Quoting directly from that article:
the rate of false reports is at least 9 percent and probably closer to 50 percent
Ted explicitly endorses an article that claims that a significant percentage of reported rapes are false. The study that generated that figure is held in poor regard by other researchers in the field - Australian police figures indicate that 2.1% of rape accusations were classified as false. Ted asserts that he was trying to argue against poor use of statistics, so it's a fair assumption that he agrees with the alternative statistics that he's citing. Ted believes that many rape victims are making false accusations. Ted believes that many rape victims are liars.

Again in this mail, Ted argues against a claimed figure that 1 in 4 women have been sexually assaulted. One of his arguments is that Also found in the Koss study, although not widely reported, was the statistic that of the women whom she classified as being raped (although 73% refused to self-classify the event as rape), 46% of them had subsequent sex with the reported assailant. Ted disagrees with a statistic because some rape victims subsequently have sex with the reported assailant. This means that Ted believes that this indicates that they were not really raped. Ted is a rape apologist.)

comments

I am looking for a replacement for IlohaMail, since I just noticed it stopped working reliably:

[Mon Oct 29 18:57:18 2012] [error] [client 195.144.91.202] PHP Warning:  strcasecmp() expects parameter 1 to be string, array given in /usr/share/IlohaMail/include/mime.inc on line 87, referer: [...]

This happens when trying to view an email. When that happens, the headers are parsed correctly, but the body isn't shown, and one has to instead click on the "view source" link to see anything at all. There's actually a patch for this bug gathering dust in the BTS; that, plus the fact that it has not seen any upstream activity since 2005, and that the last maintainer upload into Debian was in 2008, makes me conclude that IlohaMail is probably dead. Therefore, I'm looking for a replacement. Specifically, I'm looking for:

• Something simple and lightweight. The server on which this will be running is a QNAP 419U, which is powerful enough to do some useful things, but not to the extent that I can just waste resources. Additionally, the webmail is only used as an emergency thing, when I don't have my laptop immediately with me.
• If it's not written in PHP, that'd be a plus.
• It must not require MySQL, and preferably no database at all (though I can live with PostgreSQL if needs be).
• It should try to get at the mail either through IMAP or through reading maildirs

Does something like that exist? If so, pointers would be greatly appreciated.

I'm the maintainer of MySQL Connector/Python Debian package, and it's shipped with several unittests, but they require a MySQL instance to be run against, so up to know I wasn't able to run them at build-time.

But I want to change that, so I wanted to find a way to spawn a MySQL instance just for testing purposes. I've searched a lot, and then found a package solving the same problem: php5.

The package has a setup-mysql.sh script that takes care of starting/stopping the instances, and debian/rules file sets up instance parameters and then it starts the instance before the tests and then stops it after them.

It's a very interesting solution, so I thought of sharing + taking notes to self, just in case I need it in other situations.

Almost a year after my previous post, I felt inclined to spend another Sunday (this one was chilly rather than rainy) working on my script to integrate the drumkit of Guitar Hero World Tour for Wii within a MIDI environment. And wiigh2midi got, if not a rewrite, then at least a few enhancements since I last mentioned it here.

• The buttons are now handled. The cross-stick , the A and B buttons, + and -, Home, 1 and 2. And if the Wiimote is plugged into a drumkit, you get even more buttons: another set of + and -, plus another mini-joystick. Thunder! Martians! Bubbles! Yay for sound effects!
• A side-effect: the drumkit is no longer even necessary. Run several instances with different configurations, give a wiimote each to a bunch of kids, and get them to perform a batucada by pressing the big round button in time. I tried and failed, but it was due to the age and over-excitedness of the bunch of kids in question rather than the technical side of things.
• Yeah, configurations. It's all in a nice .ini file now, and you can define several kits (input-to-MIDI-note mappings) in different sections, and select which kit to use when running.
• The configuration can also define which MIDI device to send events to, and which channel to use. My use case is that sometimes I want to use my GHWT drumkit as a set of extra pads and send the MIDI stuff to my real electronic drumkit (over an USB adaptor), and sometimes I just want to send the MIDI events to a software synthesiser such as Hydrogen, to have a second, cheaper, lighter drumkit.
• Speaking of Hydrogen: since the MIDI-to-instrument mappings are somewhat strange in there, I provided a template Hydrogen file with a reasonable drumkit that matches the example configuration if wiigh2midi.
• Hey, you know, three toms, two cymbals and one pedal are really a small kit, even for beginners. And what about open/closed hi-hats? Well, we could, you know, split some of the pads so they can produce different sounds depending on context. Like, maybe if you hit the left pad real soft, you get a cross-sticks sound, but you get the standard snare drum sound if you hit it a bit harder (or very hard). And the open hi-hat could be obtained by hitting the yellow cymbal harder, with normal hits still sounding like closed hi-hats. And since the crash cymbal is usually hit markedly, while the ride is usually more of a soft-to-medium hit, you guess what we could do with the orange cymbal. Well, scratch the could there, that's exactly what we do now. So my cheap drumkit has hi-hats that open, a cross-stick sound on the snare drum, ride and crash cymbals, a bass drum and two toms. Not so bad.

It's still not something that I'd put in everyone's hands, but it's coming to be seriously usable. I wonder if there'd be any interest in me packaging that and uploading it to Debian? It would need some cleanup first (and a more generic name, since it's far from restricted to Guitar Hero controllers or drumkits), but I guess it could be useful. Ping me if you're interested.

The 12.10 release is the first version of Ubuntu that supports Secure Boot out of the box. In what is largely an accident of release timing, from what I can tell (and please correct me if I'm wrong), this actually makes Ubuntu 12.10 the first general release of any OS to support Secure Boot. (Windows 8 of course is also now available; and I'm sure Matthew Garrett, who has been a welcome collaborator throughout this process, has everything in good order for the upcoming Fedora 18 release.) That's certainly something of a bittersweet achievement. I'm proud of the work we've done to ensure Ubuntu will continue to work out of the box on the consumer hardware of the future; in spite of the predictable accusations on the blogowebs that we've sold out, I sleep well at night knowing that this was the pragmatic decision to make, maximizing users' freedom to use their hardware. All the same, I worry about what the landscape is going to look like in a few years' time. The Ubuntu first-stage EFI bootloader is signed by Microsoft, but the key that is used for signing is one that's recommended by Microsoft, not one that's required by the Windows 8 certification requirements. Will all hardware include this key in practice? The Windows 8 requirements also say that every machine must allow the user to disable Secure Boot. Will manufacturers get this right, and will users be able to make use of it in the event the manufacture didn't include the Microsoft-recommended key? Only time will tell. But I do think the Linux community is going to have to remain engaged on this for some time to come, and possibly hold OEMs' feet to the fire for shipping hardware that will only work with Windows 8. But that's for the future. For now, we have a technical solution in 12.10 that solves the parts of the problem that we can solve.

• The first-stage UEFI bootloader on 12.10 install media is a build of Matthew Garrett's shim code, embedding a Canonical UEFI CA and signed by Microsoft. This is pretty much the same as the Fedora solution, just with a different key and a binary built on the Ubuntu build infrastructure rather than Fedora's. If Secure Boot is not enabled, the second-stage bootloader is booted without applying any checks. If Secure Boot is enabled, the signature is checked on the second stage before passing control, as expected.
• The second-stage bootloader is GRUB 2. Readers may remember that there were earlier concerns about GPLv3 in the mix for some Ubuntu use cases, but these have been ironed out now in discussion with the FSF. This enables us to continue to provide a consistent boot experience across the different ways Ubuntu may be booted.
• We are providing signed kernel images in the Ubuntu archive and using them by default. When a signed kernel is present, this allows the bootloader to pass control to the kernel without first calling ExitBootServices(), letting the kernel apply any EFI quirks it might need to (such as for bug #1065263.
• Unlike Fedora, however, we are not enforcing kernel signature checking. If the kernel is unsigned and the system has Secure Boot turned on, the Ubuntu grub will fall back to calling ExitBootServices() first, and then booting the kernel. This way, users still have the freedom to boot their own kernels on Secure Boot-enabled hardware, possibly with a slightly degraded boot experience, while the firmware remains protected from untrusted code.
• We are not enforcing module signing. This allows external modules, such as dkms packages, to continue to work with SB turned on. While there's potential value in being able to verify the source of kernel modules at load time, that's not something implemented for this round, and we would want such enforcement to be optional regardless.

This first release gives us preliminary support for booting on Secure Boot, but there's more work to be done to provide a full solution that's sustainable over the long term. We'll be discussing some of that this week at UDS in Copenhagen.

• The 12.10 solution does not include support for SuSE's MOK (machine owner key) approach. Supporting this is important to us, as this helps to ensure users have freedom and control over their machines instead of being forced to choose between disabling Secure Boot and only running vendor-provided kernels. Among other things, we want to make sure people have the freedom to continue doing kernel and bootloader development, without having to muddle their way through impossible firmware menus!
• We will be enhancing the tooling around db/dbx updates, to ensure that Ubuntu users of Secure Boot receive the same protection against malware that Windows users do. This also addresses the need for publishing revocations in the event of bugs in our grub or kernel implementations that would compromise the security of Secure Boot.
• Netboot support is currently nonexistent. The major blocker seems to be that unlike in BIOS booting we can't rely on the firmware's PXE stack, and in practice GRUB2's tftp support doesn't seem to be very robust yet. Once we have a working GRUB2 image that successfully reads files over tftp, we can evaluate signing it for Secure Boot as well.

And as part of our committment to enabling new hardware on the current LTS release, we will be backporting this work for inclusion in 12.04.2. It remains to be decided how Debian will approach the Secure Boot question. At DebConf 12, many people seemed to consider it a foregone conclusion that Debian would never agree to include binaries in main signed by third-party keys. I don't think that should be a given; I think allowing third-party signatures in main for hardware compatibility is consistent with Debian's principles, and refusing to make Debian compatible with this hardware out of the box does nothing to advance user freedom. I hope to see frank discussion post-wheezy about keeping Debian relevant on consumer hardware of the future.

The summer came finally to an end and (yes in Berlin we still had 20 C end of October), unfortunately, so did GSoC with it. This has been the second time for SHOGUN to be in GSoC. For those unfamiliar with SHOGUN - it is a very versatile machine learning toolbox that enables unified large-scale learning for a broad range of feature types and learning settings, like classification, regression, or explorative data analysis. I again played the role of an org admin and co-mentor this year and would like to take the opportunity to summarize enhancements to the toolbox and my GSoC experience: In contrast to last year, we required code-contributions in the application phase of GSoC already, i.e., a (small) patch was mandatory for your application to be considered. This reduced the number of applications we received: 48 proposals from 38 students instead of 70 proposals from about 60 students last year but also increased the overall quality of the applications. In the end we were very happy to get 8 very talented students and have the opportunity of boosting the project thanks to their hard and awesome work. Thanks to google for sponsoring three more students compared to last GSoC. Still we gave one slot back to the pool for good to the octave project (They used it very wisely and octave will have a just-in-time compiler now, which will benefit us all!). SHOGUN 2.0.0 is the new release of the toolbox including of course all the new features that the students have implemented in their projects. On the one hand, modules that were already in SHOGUN have been extended or improved. For example, Jacob Walker has implemented Gaussian Processes (GPs) improving the usability of SHOGUN for regression problems. A framework for multiclass learning by Chiyuan Zhang including state-of-the-art methods in this area such as Error-Correcting Output Coding (ECOC) and ShareBoost, among others. In addition, Evgeniy Andreev has made very important improvements w.r.t. the accessibility of SHOGUN. Thanks to his work with SWIG director classes, now it is possible to use python for prototyping and make use of that code with the same flexibility as if it had been written in the C++ core of the project. On the other hand, completely new frameworks and other functionalities have been added to the project as well. This is the case of multitask learning and domain adaptation algorithms written by Sergey Lisitsyn and the kernel two-sample or dependence test by Heiko Strathmann. Viktor Gal has introduced latent SVMs to SHOGUN and, finally, two students have worked in the new structured output learning framework. Fernando Iglesias made the design of this framework introducing the structured output machines into SHOGUN while Michal Uricar has implemented several bundle methods to solve the optimization problem of the structured output SVM. It has been very fun and interesting how the work done in different projects has been put together very early, even during the GSoC period. Only to show an example of this dealing with the generic structured output framework and the improvements in the accessibility. It is possible to make use of the SWIG directors to implement the application specific mechanisms of a structured learning problem instance in python and then use the rest of the framework (written in C++) to solve this new problem. Students! You all did a great job and I am more than amazed what you all have achieved. Thank you very much and I hope some of you will stick around. Besides all these improvements it has been particularly challenging for me as org admin to scale the project. While I could still be deeply involved in each and every part of the project last GSoC, this was no longer possible this year. Learning to trust that your mentors are doing the job is something that didn't come easy to me. Having had about monthly all-hands meetings did help and so did monitoring the happiness of the students. I am glad that it all worked out nicely this year too. Again, I would like to mention that SHOGUN improved a lot code-base/code-quality wise. Students gave very constructive feedback about our (lack) of proper Vector/Matrix/String/Sparse Matrix types. We now have all these implemented doing automagic memory garbage collection behind scenes. We have started to transition to use Eigen3 as our matrix library of choice, which made quite a number of algorithms much easier to implement. We generalized the Label framework (CLabels) to be tractable for not just classification and regression but multitask and structured output learning. Finally, we have had quite a number of infrastructure improvements. Thanks to GSoC money we have a dedicated server for running the buildbot/buildslaves and website. The ML Group at TU Berlin does sponsor virtual machines for building SHOGUN on Debian and Cygwin. Viktor Gal stepped up providing buildslaves for Ubuntu and FreeBSD. Gunnar Raetschs group is supporting redhat based build tests. We have Travis CI running testing pull requests for breakage even before merges. Code quality is now monitored utilizing LLVMs scan-build. Bernard Hernandez appeared and wrote a fancy new website for SHOGUN. A more detailed description of the achievements of each of the students follows:

• Kernel Two-sample/Dependence test
  • Student: Heiko Strathmann
  • Mentor: Arthur Gretton, Soeren Sonnenburg
  Heiko Strathmann, mentored by Arthur Gretton, worked on a framework for kernel-based statistical hypothesis testing. Statistical tests to determine whether two random variables are are equal/different or are statistically independent are an important tool in data-analysis. However, when data are high-dimensional or in non-numerical form (documents, graphs), classical methods fail. Heiko implemented recently developed kernel-based generalisations of classical tests which overcome these issues by representing distributions in high dimensional so-called reproducing kernel Hilbert spaces. By doing so, theoretically any pair samples can be distinguished. Implemented methods include two-sample testing with the Maximum Mean Discrepancy (MMD) and independence testing using the Hilbert Schmidt Independence Criterion (HSIC). Both methods come in different flavours regarding computational costs and test constructions. For two-sample testing with the MMD, a linear time streaming version is included that can handle arbitrary amounts of data. All methods are integrated into a newly written flexible framework for statistical testing which will be extended in the future. A book-style tutorial with descriptions of algorithms and instructions how to use them is also included.
• Implement multitask and domain adaptation algorithms
  • Student: Sergey Lisityn
  • Mentor: Christian Widmer
  Like Heiko, Sergey Lisitsyn did participate in the GSoC programme for the second time. This year he focused on implementing multitask learning algorithms. Multitask learning is a modern approach to machine learning that learns a problem together with other related problems at the same time using a shared representation. This approach often leads to a better model for the main task, because it allows the learner to use the commonality among the tasks. During the summer Sergey has ported a few algorithms from the SLEP and MALSAR libraries with further extensions and improvements. Namely, L12 group tree and L1q group multitask logistic regression and least squares regression, trace-norm multitask logistic regression, clustered multitask logistic regression, basic group and group tree lasso logistic regression. All the implemented algorithms use COFFIN framework for flexible and efficient learning and some of the algorithms were implemented efficiently utilizing the Eigen3 library.
• Implementation of / Integration via existing GPL code of latent SVMs.
  • Student: Viktor Gal
  • Mentor: Alexander Binder
  A generic latent SVM and additionally a latent structured output SVM has been implemented. This machine learning algorithm is widely used in computer vision, namely in object detection. Other useful application fields are: motif finding in DNA sequences, noun phrase coreference, i.e. provide a clustering of the nouns such that each cluster refers to a single object. It is based on defining a general latent feature Psi(x,y,h) depending on input variable x, output variable y and latent variable h. Deriving a class from the base class allows the user to implement additional structural knowledge for efficient maximization of the latent variable or alternative ways of computation or on-the-fly loading of latent features Psi as a function of the input, output and latent variables.
• Bundle method solver for structured output learning
  • Student: Michal Uricar
  • Mentor: Vojtech Franc
  We have implemented two generic solvers for supervised learning of structured output (SO) classifiers. First, we implemented the current state-of-the-art Bundle Method for Regularized Risk Minimization (BMRM) [Teo et al. 2010]. Second, we implemented a novel variant of the classical Bundle Method (BM) [Lamarechal 1978] which achieves the same precise solution as the BMRM but in time up to an order of magnitude shorter [Uricar et al. 2012]. Among the main practical benefits of the implemented solvers belong their modularity and proven convergence guarantees. For training a particular SO classifier it suffices to provide the solver with a routine evaluating the application specific risk function. This feature is invaluable for designers who can concentrate on tuning the classification model instead of spending time on developing new optimization solvers. The convergence guarantees remove the uncertainty inherent in use of on-line approximate solvers. The implemented solvers have been integrated to the structured output framework of the SHOGUN toolbox and they have been tested on real-life data.
• Built generic structured output learning framework
  • Student: Fernando Jose Iglesias Garcia
  • Mentor: Nico Goernitz
  During GSoC 2012 Fernando implemented a generic framework for structured output (SO) problems. Structured output learning deals with problems where the prediction made is represented by an object with complex structure, e.g. a graph, a tree or a sequence. SHOGUN's SO framework is flexible and easy to extend [1]. Fernando implemented a na ve cutting plane algorithm for the SVM approach to SO [2]. In addition, he coded a case of use of the framework for labelled sequence learning, the so-called Hidden Markov SVM. The HM-SVM can be applied to solve problems in different fields such as gene prediction in bioinformatics or speech to text in pattern recognition. [1] Class diagram of the SO framework.
  [2] Support Vector Machine Learning for Interdependent and Structured Output Spaces.
• Improving accessibility to shogun
  • Student: Evgeniy Andreev
  • Mentor: Soeren Sonnenburg
  During the latest google summer of code Evgeniy has improved the Python modular interface. He has added new SWIG-based feature - director classes, enabling users to extend SHOGUN classes with python code and made SHOGUN python 3 ready. Evgeniy has also added python's protocols for most usable arrays (like vectors, matrices, features) which makes possible to work with Shogun data structures just like with numpy arrays with no copy at all. For example one can now modify SHOGUN's RealFeatures in place or use.
• Implement Gaussian Processes and regression techniques
  • Student: Jacob Walker
  • Mentor: Oliver Stegle
  Jacob implemented Gaussian Process Regression in the Shogun Machine Learning Toolbox. He wrote a complete implementation of basic GPR as well as approximation methods such as the FITC (First Independent Training Conditional) method and the Laplacian approximation method. Users can utilize this feature to analyze large scale data in a variety of fields. Scientists can also build on this implementation to conduct research extending the capability and applicability of Gaussian Process Regression.
• Build generic multiclass learning framework
  • Student: Chiyuan Zhang
  • Mentor: Cheng Soon Ong
  Chiyuan defined a new multiclass learning framework within SHOGUN. He re-organized previous multiclass learning components and refactored the CMachine hierarchy in SHOGUN. Then he generalized the existing one-vs-one and one-vs-rest multiclass learning scheme to general ECOC encoding and decoding strategies. Beside this, several specific multiclass learning algorithms are added, including ShareBoost with feature selection ability, Conditional Probability Tree with online learning ability, and Relaxed Tree.

Review: Devil Take the Hindmost, by Edward Chancellor

Publisher:	Plume
Copyright:	1999
Printing:	June 2000
ISBN:	0-452-28180-6
Format:	Trade paperback
Pages:	349

Subtitled A History of Financial Speculation, Devil Take the Hindmost is exactly that: a history, with chapters on notable speculative bubbles or surges going back to the 1630s. The first significant bubble described is, of course, the notorious tulip speculation in Holland, but Chancellor is more thorough than any previous history I've read. There are chapters on the early stock market schemes, the South Sea bubble, 1820s emerging markets, the railway mania of 1845, the Gilded Age, the late 1970s and early 1980s, and the Japanese real estate bubble of the 1980s, as well as the mandatory look at the stock market crash of 1929. Each is thorough and detailed; indeed, perhaps slightly too detailed, as the clear narrative picture can occasionally groan under the weight of names and capsule biographies. The first thing one should realize about this book is that, despite the apparent currency of the topic, it was written in 1999. That means it predates both the subprime real estate collapse and the dot.com boom, as well as some more minor but possibly relevant events such as the Enron electricity market manipulation. Japan is the most recent bubble that Chancellor can cover; one can only speculate as to what he would make of the 2008 financial crash. The other focus to be aware of is the omission of sovereign debt. Chancellor is exclusively focused on private markets, usually stock markets, and (apart from the chapter on Japan) Western markets, mostly in the US and the UK. Governments play a role in some of the early speculative bubbles (most notably the South Sea bubble), but Chancellor's topic is private speculative frenzies, not the risks and effects of government borrowing. Interrogation of the role of government is mostly limited to cursory discussions of the impact of regulation, mostly in the negative. More on that in a moment. This is a history as opposed to either a polemic or a prescription. Chancellor does not set out to explain how to curb financial speculation. Insofar as he expresses an opinion on it, he presents speculation as a force of unbridled freedom in natural tension with regulation and financial conservatism and sees a cyclical pattern of speculative frenzy, collapse, retrenchment, and then renewed speculation. Speculation, as presented here, is more akin to a force of nature than, as seen in most other books on the topic, a phenomenon of greed that should be limited or prevented. Indeed, Chancellor seems skeptical of the entire project of preventing or limiting speculation, and presents considerable reason for the reader to become skeptical as well. The history he presents is one of speculative manias adapting to and circumventing every control that governments and markets have attempted to place on them. When certain practices are outlawed (and some modern complex financial practices have surprisingly long histories), people just find loopholes or alternative approaches through which to accomplish the same thing. Or the speculative mania takes an unexpected new form. Nor do the manias seem to be decreasing in severity as more regulation, custom, and restrictions attempt to blunt them. Even without adding the financial collapse of 2008, the impacts of the speculative frenzies documented here seem, if anything, to be growing. This is, therefore, a relatively straightforward history, but it's an entertaining one, filling in details that I'd not previously been aware of. The complexity of financial schemes as early as the 1600s was particularly eye-opening. There's a tendency to believe that complex derivative schemes are an artifact of modern financial markets, but they've been a part of speculation for as long as Chancellor goes back. Computers may have added layers to the mathematics, but they haven't added as much to the legal strategems and levels of indirection as one might think. Chancellor emphasizes that throughout: speculation involves a lot of complexity, not necessarily to deceive but to construct situations in which it's in the interest (at least short term) of all involved parties to continue and escalate the speculation. While Chancellor doesn't provide much encouragement to the reader hoping to find ways to curb speculation, he does show some commonalities that provide a bit of a handle on how speculation starts. The one that lept out at me was leverage. Every speculative bubble documented in Devil Take the Hindmost involves borrowing money to reinvest. The mechanism keeps changing, from straightforward loans during the tulip craze to much more complex constructions in modern finance, primarily in response to regulation that attempted to reduce leverage but only banned one specific technique. But whatever method is used, at the bottom is always some scheme for purchasing an asset using debt, magnifying both positive and negative price swings. Some variation of buying stocks on margin is particularly common; Chancellor documents how it was reinvented despite the limits on straight margin purchases instituted after the Great Depression. There is no recommendation here for any regulatory or legal action, but it's obvious from the repetition of history that limiting leverage would be the place to focus attention. I enjoyed this history, but it can be dry, and at times is a bit overstuffed with names, unimportant details, and microbiographies. It's also unfortunate that Chancellor missed two of the most significant (and two very different) speculative bubbles through the timing of his work. It would have been interesting to see them through the lens of his history. What we have is still worth reading, but also somewhat frustrating in Chancellor's refusal to draw or at least underline general conclusions. Devil Take the Hindmost is long on history but short on analysis, and while I appreciate the historian's focus in a source of data, it left me somewhat unsatisfied. Rating: 6 out of 10

I recently worked on the following RC bugs:

• #691125 ejabberd: package installation creates /root/.erlang.cookie
  I tracked down the cause of this and provided a patch. Now I also know how piuparts works in detail :-).
• #691180 connman: Connman won't run due to missing libxtables.so.7
  Since this bug does not affect testing and it doesn t look like iptables 1.4.16.3-1 (a new upstream release) will go into wheezy (no freeze unblock filed), I tagged this with wheezy-ignore. I hope this was right, if not, feel free to correct me.
• #684645 liblockfile1: Order of fcntl and dotlock in maillock
  I identified the piece of code which does locking the wrong way. Since I m not familiar enough with either sendmail or NFS-safe locking, I did not send a patch. In case you have experience with this, please have a look!

Work on slaughter 2.x is going rather well. The scripting hasn't changed, and no primitives have been altered to break backward compatibility, but it is probably best to release this as "slaughter2" - because the way to specify the source from which to pull scripts has changed. Previously we'd specify two arguments (or have them in a configuration file):

• --server=example.com
• --prefix=/slaughter/

That would result in policies being downloaded from:

  http://example.com/slaughter/

Now the rework is complete we use "transports" and "prefixes". The new way to specify the old default is to run with:

--transport=http --prefix=http://example.com/slaughter/

I've implemented four transports thus far:

• GIT
• http
• Mercurial
• rsync

The code has been made considerably neater, the test-cases are complete, and the POD/inline documentation is almost 100% complete. Adding additional revision-controlled transports would be trivial at this point - but I suspect I'd be wasting my time if I were to add CVS support! Life is good. Though I've still got a fair bit more documentation, prettification and updates to make before I'm ready to release it. Play along at home if you wish: via the repository.

I ve been a wannabe GTD afficionado for some years. I ve wanted to do it, but managing lists has always been something that has too much friction, overhead, or whatever. Finally, though, I think I might have found a way to manage lists that works. My use-case isn t unique, although I will concede I m perhaps being more dogmatic than most. I want something that:

• Is electronic (yes, the hipster PDA is a cool idea, but my handwriting is beyond woeful and I already carry enough crap in my pockets as it is);
• Works offline (because I often do);
• Will work on both my phone and laptop (because I want to have my lists with me when I m not at my computer, but long-form data entry or manipulation on a phone is painful);
• Makes it easy to add, browse, modify, and above all remove items; and
• Is mine (no central databases I don t control, proprietary apps on my phone, etc).

My previous attempt was a tool I called tagnote it was a vim-outliner file full of hierarchically organised outliner entries, with tags inlined. It was a neat idea, but it wasn t smooth to add/browse/delete items, and didn t work with my phone at all (trying to use vim for any length of time on a bottom-of-the-range Android phone would kill me). The current iteration, as the title of this post suggests, is a list manager that entirely uses e-mail. It really is a perfect symbiosis:

• I want lists of text items with titles (Subject:), potentially other metadata (X-Whatever headers), and possibly some notes (the body of the e-mail);
• I need to be able to browse and remove completed items (that s what e-mail clients are for);
• Getting new items into the system is trivial (Anything I send to my PDA s e-mail address goes straight into the INBOX, which I can then process as time permits); and
• Syncing between my laptop and PDA is as simple as offlineimap and K-9 mail.

So what have I got, exactly? It s fairly straightforward:

• An IMAP account on my existing mail server;
• A slightly tweaked copy of mutt (different colours so I don t confuse myself, and a different layout of the index page to get rid of unnecessary columns);
• Another offlineimap account;
• Another K-9 mail account;
• Note2self (a neat little app to take a typed or voice-transcribed note and e-mail it to a pre-set address) on the phone, pre-programmed to e-mail any notes I write to the PDA s inbox; and
• A small shell script to make it trivial to create new lists (which has to be done on the IMAP server for reasons of offlineimap), add new items to a list, sync my lists (in other words, run offlineimap ), display my lists (in other words, run mutt ), and process my tickler file.

That last point is the one I m really happy I achieved. I ve always been a fan of hide it until you need it , but my previous system didn t let me do that. Now, though, I have a separate list called tickler, and all the items in there have an X-Tickle header, which specifies the date I want to see them. Each night a cronjob runs through the tickler and moves anything for today into the INBOX. An X-Tickle-Repeat header lets me have things that repeat over and over again. So in short, using entirely open-source tools and a couple of hours of my time doing things I enjoy anyway (shell scripts! woo!), I ve now got a list manager that doesn t get in my way more than it absolutely has to. We ll see how long I last this time before I feel the urge to improve my lists again.

Review: Fables: Legends in Exile, by Bill Willingham, et al.

Series:	Fables #1
Penciller:	Lan Medina
Inker:	Steve Leialoha
Inker:	Craig Hamilton
Colorist:	Sherilyn van Valkenburgh
Letterer:	Todd Klein
Publisher:	Vertigo
Copyright:	2002
ISBN:	1-56389-942-6
Format:	Graphic novel
Pages:	127

The characters of human fairy tales are all real. Real, but hidden: they have been driven from their ancestral homelands by some force referred to here only as the Adversary, and have fled to the mundane world to escape his notice. Specifically, New York, where those characters who can pass as human blend into the streets of New York City, and those who can't (referred to only in passing in this story arc apart from one pig) stay at a farm upstate. That's the brilliant premise of Fables, an ongoing comic series from Vertigo that's up to 121 issues plus numerous spinoffs as of this writing. This is the first trade paperback, collecting issues 1 through 5 plus a prose story. Legends in Exile is, in underlying form, a murder mystery. It opens with Jack reporting a murder to Bigby Wolf, who is what passes for law enforcement among the Fables. Rose Red's apartment has been discovered covered in blood, and her body is missing. "No more happily ever after" is written on the wall in blood. Jack, despite being the one to report the crime, is Rose's lover and an obvious suspect. So is Snow White, the effective leader of the Fables (King Cole is a figurehead) and Rose's estranged sister. And, perhaps coincidentally, Prince Charming, Snow White's philandering and stunningly narcissistic ex-husband, has just turned up in town again. The basic plot plays out like a typical murder mystery, complete with a grand reveal in the last chapter and a reconstruction of events. It's diverting but nothing special in itself; it's primarily a hook on which to hang an introduction to the major characters and an exploration of Fables politics and history. And that part is excellent. Like most fractured fairy tales, the characters are more complex and nuanced than they are allowed to be in the canonical stories; unlike most fractured fairy tales, nearly all of that development is allowed to happen after the original stories. The original stories happened, largely as we all remember them, but then the characters kept living and changing. Wolf and his relationship with one of the three pigs is one of the highlights, but I think the odious Price Charming is the best bit of characterization. Willingham takes the idea of a charming seducer and layers in the rest of the womanizing associations that come to mind, while retaining a slick surface charm that is hard not to like at some level. I'm not the best at critiquing art. I read graphic novels for the story, with the art as a nice bonus if it's fairly good. But I can say that Medina's pencils tell the story well, with a few larger panels that were quite impressive. I particularly liked Snow White's office, early in the collection. Detail is reasonably good throughout, there aren't too many panels featuring only talking heads, and Medina does a wonderful job with the pig. (Significant credit probably also goes to Steve Leialoha and Craig Hamilton, the inkers. I'm not a good enough art critic to distinguish between the quality of the pencil work and what was added by subsequent inking.) The largest problem for me with Legends in Exile is my standard problem with graphic novels and the reason I don't buy many of them: the cover price is $10 for what is effectively a short story that builds an interesting world. If one really loves art in stories, graphic novels may be a good value. If, like me, one mostly cares about the story and would be almost as happy with straight prose, the value becomes questionable given the brief length of the material. That said, Legends in Exile does add a quite good prose short story by Willingham, "A Wolf in the Fold," which in several ways is better than the main story of the collection. It tells the story of the original exile and the covenant that settled all past grudges (mostly) from the perspective of the wolf. Willingham does a great job here both striking a more mythic tone than the main story but still tying it back to the sarcastic and practical tone of the rest of the collection. Although the main story itself is not particularly memorable, the background and concept make this one of the better graphic novels I've read. I can see why it's won so many awards, and I'll probably keep reading, despite the high graphic novel price tag. Followed by Fables: Animal Farm. Rating: 7 out of 10

During the Squeeze release cycle, Alexander Reichle-Schmehl used to blog every week about this. Let s try to continue the tradition. The UDD bugs interface currently knows about the following release critical bugs:

• In Total: 916
  • Affecting wheezy: 464
    That s the number we need to get down to zero before the release. They can be split in two big categories:
    • Affecting wheezy and unstable: 327
      Those need someone to find a fix, or to finish the work to upload a fix to unstable:
      • 77 bugs are tagged patch .
        Please help by reviewing the patches, and (if you are a DD) by uploading them.
      • 17 bugs are marked as done, but still affect unstable.
        This can happen due to missing builds on some architectures, for example. Help investigate!
      • 235 bugs are neither tagged patch, nor marked done.
        Help make a first step towards resolution!
    • Affecting wheezy only: 137
      Those are already fixed in unstable, but the fix still needs to migrate to wheezy. You can help by submitting unblock requests for fixed packages, by investigating why packages do not migrate, or by reviewing submitted unblock requests.

Note: I m looking for someone to send this posts on a regular basis. All you need is a blog syndicated on Planet Debian (there s a secret UDD page that generates the HTML code for you ;) ). Contact me if you are interested. taken!

Review: Passion Play, by Sean Stewart

Publisher:	Ace
Copyright:	1992
Printing:	December 1993
ISBN:	0-441-65241-7
Format:	Mass market
Pages:	194

Diane is a shaper: an empath, a sort of telepath, who sees emotions and the actions of others in the form of patterns and stories that she is nearly compelled to follow. She uses that skill as an investigator, a semi-official adjunct to the police who can read crime scenes and motives and hunt down criminals with uncanny ability. That makes her almost socially acceptable, but a shaper is still not something it's safe to be, something best kept hidden. Diane's world is one of theocracy, of dominance by aggressive religion, and even shapers working with the police are not exactly okay. This is Sean Stewart's first novel. He's since gone on to write many other books, one of which I've read and quite enjoyed (Nobody's Son) and another (Galveston) that won several awards, although he's probably most famous for his work on the ilovebees ARG. Passion Play won a minor award itself: the Aurora for the best Canadian SF novel of the year. I found it surprising as a first novel, since it's sharp, focused, and very short. Usually first novels are stuffed to the gills with ideas and plot, as if material had built up for years and exploded in the first book. Passion Play doesn't have that problem; if anything, it errs on the side of telling too little. This is, at its heart, a mystery. A famous actor died in the middle of a play. It was possibly suicide, possibly an accident, and possibly murder; the events of the death are quite ambiguous. But the actor is very important in Redemptionist politics, which warrants an investigation, and then Diane's shaper instincts start finding a pattern and a story in the death. The rest of the cast and crew form the obvious set of suspects, and have the normal mystery novel variety of flaws, foibles, strong personalities, and possible motives. I'm not much of a mystery novel reader, and to be honest I had some trouble tracking all of the characters and remembering Diane's suspicions about each one. But the details of the mystery are less important to this book than Diane herself. Shapers are not exactly stable. The patterns they see are all-consuming, the experience of others' strong emotions acts like a sort of drug, and shapers can burn themselves out. They can fall into a spiral of seeking stronger and stronger emotions until they can't feel anything at all. That's the core conflict of the book: Diane is slowly losing herself. She has a tenuous existence at the outskirts of a society that has inculcated her with an inflexible set of religious beliefs and an uncompromising attitude towards law and justice, but it's not a stable existence, she knows it, and she doesn't know what to do about it. She knows that the pattern of this particular death is deep and powerful, but she can't stay away from it. This is an unusual, and sometimes disturbing, focus for a story. It's risky, unconventional, and doesn't quite work. I think some of the failure is because of first novel problems, but most of it is due to character problems. In the first novel problems category, Passion Play is a bit too choppy and a bit too disjointed. I think Stewart was trying hard to keep the story focused, fast-moving, and tight, but the introduction of the world background is not particularly smooth, and Diane's first-person account of her emotional state is a bit too labored. The plot wants to build in a smooth arc towards its climax, but the writing and scene-setting jerks and jumps instead of flowing. But the larger problem for me is that there aren't enough interesting characters in this book; specifically, there aren't enough to show all sides of Diane's character. Stewart tries, by introducing Jim early in the book to serve essentially as Diane's friend (and she's desperately in need of one), but he's too much of a non-entity. He serves some plot purposes, but he doesn't have a strong enough voice and character to hold his ground against Diane and balance her obsessive internal monologue. And, other than Jim, all the other characters in the book (with the possible exception of the murder victim) are playing bit roles. They're not bad characters, but they don't feel fully real to either Diane or to the reader. It's very easy to put them into boxes that they never break out of. This does support the feeling of narrative inevitability about some parts of the book, but I think it would have been stronger with more conflict, more open challenging of Diane's perspective. I would characterize Passion Play as interesting but flawed, and ultimately a minor work. I'm not sorry I read it, but neither would I have missed much if I'd gone through life without reading it. It has a few interesting ideas, and a bravely unconventional narrative resolution, but it felt choppy and off-balance. Unless you love Stewart's writing and want to read everything he wrote, or unless you (like me) has a quirk about reading all award winners, this is probably best skipped over in favor of Stewart's later work. Rating: 6 out of 10

I've had a few weeks with very little time spent on Debian. I did set aside some time to try to find and fix an RC bug, to help the release. Steven C commented to let me know about the UDD Developer's Dashboard which is a great way to find RC bugs relevant to a team. I sat down to try and fix #648272 in 'Abuse' but sadly I didn't manage, instead finding another possible RC bug #689457 on the way. I was curious as to how long the version 1 web services API for musicbrainz will be kept around, in case there was a strong likelyhood it would be turned off during the lifetime of wheezy. If that were the case, then it might have been worth requesting freeze exceptions to patch the last few WS1 clients to use WS2 instead. However, the musicbrainz developers have no plans to turn off WS1 as long as there are people still using it. This is a bit of a chicken and egg situation for wheezy, but probably not strong enough grounds to persue it any further. So last night was my first "Debian Day" in a while, and by chance I've had a number of enquiries for software and packages which I haven't touched in a while. For the prboom package, we rebuild the upstream tarball to remove some material we don't have the right to distribute. The code that does the repackaging lives in a script called debian/fix_upstream.sh. Prboom upstream release very infrequently, so the script hasn't been run in a while. Fabian Greffrath noticed some bashisms which meant it would not work reliably with a non-bash /bin/sh. He also noticed that we couldn't build the binary lump data/prboom.wad using only free software since some changes that were made to freedoom. One possible fix was to expand the deutex package to recognise freedoom.wad, which I've now applied. Fabian has fixed the bashism issue, but I suspect a more robust solution long-term would be to always rebuild data/prboom.wad during package build, so that code gets exercised more often. The prboom changes are in the Git repository but I haven't release them yet. I need to decide whether it's worth requesting a freeze exception for the orig tarball build issue. It's not an FTBFS but it's similar: you could not rebuild a binary distributed in the source tarball using software only in Debian, at least not without some workarounds. We've had another request to package Prboom+. I've never been particularly interested in doing this myself, but it would probably be very little work, you could more-or-less take the prboom packaging as it stands. One thing I thought would be interesting to try would be to set up a git repository for the packaging and import the upstream Subversion repository in its entirety via git svn, rather than just import the released tarballs. Sadly their repository layout is not quite ideal for this. game-data-packager continues to grow. Following hot on the heels of Hexen 2 support, Markus Koschany has written preliminary support for LGeneral, a game which has been in Debian, at least in part, on and off. Hopefully we'll have that merged soon, and finish of Hexen 2 support whilst we're at it. On that note, here's a lazyweb question: what's the most convenient way to determine the length (at microsecond resolution) of a PCM WAV file from the command-line? Finally, we've been building Debian-based virtual machines at work, which meant dusting off some old preseed files and tweaking them to have unattended installs. This took a lot longer than I thought. It's certainly a lot more painful than kickstart and Redhat-based systems, and it's very unlikely that a preseed file that worked for one version of Debian would fully work for another. I'm not certain but I think even different releases of d-i targetting the same release "break" assumptions you might make, particularly about which questions you need to answer via the kernel command line, rather than in the preseed file. That's one thing we really need to sort. It's crazy having to specify keyboard layouts in the command line for VMs which won't ever have a keyboard. The disparity between what the user is asked, what value is written to the debconf database at that point in the install process, and what ends up in the debconf database at the end of the process is also a real pain: answering a question about keyboard layout sets half a dozen different debconf values in a cascade that is probably sensitive to other things like priority or whether d-i is in expert mode, too. So working backwards from the debconf database of an installed machine is an exercise in frustration, and figuring out which questions are asked prior to the preseed file being read for any given d-i or Debian release is a slow trial and error process. I'm going to get in touch with the d-i folks and see what kind of continuous integration systems they have in place. It would be great to have commit-level notification that you've broken preseed builds. If not for the d-i folks, then certainly at work.

I get a lot of spam. Most of it, thankfully is blocked by dspam but occasionally i get some through the filter. One that particularly caught my eye was interesting not so much what it was advertising (I don t read that part of the email) but where it came from and goes to. Normally there are two service providers involved in spam. The email comes from (or via) one and then the spamvertised website is another. The interesting thing is for this spam both of these were the same service provider. The email came from 174.34.168.85 and the spamvertised website was 70.32.40.194. Both of these addresses are owned by nobistech.net. I punted the email to spamcop and it said that they re not interested in spam reports. A few google queries shows that these guys seem quite happy to have spam sources and destinations and have been doing it for years. They either appear as nobistech.net or unbiquity servers but they are one and the same organisation, or at least related. I won t bother to send anything to them, it seems this has been done many many times by others with no results. Instead some CIDR blocks will be put into my blacklist.

A few days ago, Samsung released the source code of Synaps, an implementation of the Amazon Web Service CloudWatch API for OpenStack. Being a developer on the Ceilometer project, I've been curious to look on this project and how it could overlap with Ceilometer or other projects like Heat. What is CloudWatch? CloudWatch is a monitoring system provided by Amazon on its Web Services platform to monitor services. This allows you get notifications and trigger an action on certain threshold. For example, this can be used to scale your architecture by monitoring the number of requests you get on it and its general load by starting new servers. Synaps Synaps is written in around 7k lines of Python (with 28 % of which are comments), reuses at least one common module of OpenStack (openstack.common.cfg) and copy some modules from Nova. One thing that strikes me, is that there seems to be only a few unit tests compared to most OpenStack projects. Also, many parts of the code and documentation contains text written in korean, which won't be very helpful for most people! :-) It uses some external technologies, like Storm, Cassandra to store its persistent data and Pandas to do data analysis. The API server provides an EC2 compatible API only: no OpenStack specific API. This is probably not a bad thing for now, since I am not aware of any work in this direction. The API access directly the Cassandra back-end for read operation, but relies on RPC to do writes. This way, a set of daemon handles the write using the Storm part of Synaps and do data aggregation. The authentication only supports LDAP, but it should still be possible to add a driver for Keystone. A Java and a Python SDK are provided to record metrics into Synaps, but there's not enough documentation for it to be useful. Overlap with Heat For now, there's not a lot of overlap with Heat, because Heat does not implement completely the CloudWatch API. Heat actually still misses a lot of the CloudWatch functions. But as soon as it will implement the CloudWatch API completely, the overlap will be complete with Synaps in this regard. One divergence point however, is that Heat uses RPC to access data from the storage back-end via its engine (the central daemon), whereas Synaps directly connects to Cassandra. Also, Heat relies on SQLAlchemy, like most OpenStack projects needing a database. Overlap with Ceilometer One of the goal of Ceilometer is to provide data probes and pollsters for all OpenStack components (Nova, Swift, Quantum ) whereas Synaps let the OpenStack users to put any kind of metric inside it, and therefore doesn't provide anything for now. But the storage of metrics is the main common point between Synaps and Ceilometer. Synaps chose only one technology, Cassandra, to store its metrics, whereas Ceilometer took care of building an abstraction layer for the storage engine. Ceilometer currently allows an operator to use SQL or MongoDB, but Cassandra could likely be added. Data metric consolidation is done by Synaps. This makes sense, since Synaps don't need to have the full data history to trigger alarms. On the opposite, Ceilometer needs to have a full history to allow things like billing, and don't do any aggregation on data. Also, in Synaps, the data analysis is done using Pandas. This means the data used are retrieved from the Cassandra back-end, and then transformed by Pandas inside Synaps in something else. It's likely that in such a case, Synaps should use CQL to achieve that. Ceilometer manipulates the data near their storage: it means that the computation are done by back-end to be efficient (SQL, mapreduce ). Conclusion Considering Samsung open-sourced Synaps late in the development process, I don't feel like they aimed to have it becoming a core component. This is always sad, because the effort put into this implementation are big and it would have probably little to add some abstraction layers to follow what other OpenStack projects do. But this takes time and energy, and it's understandable that Samsung didn't want to achieve this in a short time frame. There's a part of the code and architecture that overlaps with Ceilometer and Heat. Ceilometer is becoming a specialized point to store data metrics from any source: so it's sad, but understandable, that Synaps did not tried to reuse it. Fortunately, Heat is working with Ceilometer to achieve exactly that. This means OpenStack would have only one metrics storage point, used for billing, for monitoring and alarming. Therefore, I think Synaps is an implementation of CloudWatch that should be looked at as an inspiration for Heat and Ceilometer to build a better and more integrated solution!