There s been some media coverage at the start of this week about blocking child porn. Except it s not about child porn that s a trojan horse. People who want to access pornography that is already illegal (Protection of Children Act 1978) are probably already using security tools to hide their downloading and will be unaffected by this unless they re pretty stupid. And the announcement, about the same time as the predicted birth of a royal baby, third in line to the throne, seems like a cynical attempt to bury bad news taken straight from the Blair Government. That would almost be enough reason to oppose it: they don t want the media to look at this too closely for some reason. So what s this actually about? It looks like a way to force through widespread acceptance of the ability to censor most UK internet users by shouting won t somebody think of the children? If you doubt it, take a look at the list of filtered topics:

• Dating
• Drugs, Alcohol and Tobacco
• File Sharing Sites
• Gambling
• Games
• Pornography
• Social Networking
• Suicide and Self-Harm
• Weapons and Violence

So if they get away with this censorship, you won t be able to use Twitter or contact the Samaritans until you deactivate it. Except I suspect you will because they re pretty big and the Cameron Government won t want to pick a fight with them: it ll be the next Twitter and the next Samaritans, currently much smaller and unable to defend themselves, who get shut out of UK homes. So what can we do, besides explaining this and writing to our MPs? Are we better off joining parties who oppose this censorship, like the Pirate Party, or joining existing parties and trying to overturn their stupid support for it?

Implementing Vblank Synchronization in the Present Extension This is mostly a status update on how the Present extension is doing; the big news this week is that I ve finished implementing vblank synchronized blts and flips, and things seem to be working quite well. Vblank Synchronized Blts The goal here is to have the hardware executing the blt operation in such a way as to avoid any tearing artifacts. In current drivers, there are essentially two different ways to make this happen:

1. Insert a command into the ring which blocks execution until a suitable time immediately preceding the blt operation.
2. Queue the blt operation at vblank time so that it executes before the scanout starts.

Option 1. provides the fewest artifacts; if the hardware can blt faster than scanout, there shouldn t ever be anything untoward visible on the screen. However, it also blocks future command execution within the same context. For example, if two vblank synchronized blts are queued at the same time, its possible for the second blt to be delayed by yet another frame, causing both applications to run at half of the frame rate. Option 2. avoids blocking the hardware, allowing for ongoing operations to proceed on the hardware without waiting for the synchronized blt to complete. However, it can cause artifacts if the delay from the vblank event to the eventual execution of the blt command is too long. Queuing the blt right when it needs to execute means that we also have the opportunity to skip some blts; if the application presents two buffers within the same frame time, the blt of the first buffer can be skipped, saving memory bandwidth and time. Present uses Option 2, which may occasionally cause a tearing artifact, but avoids slowing down applications while allowing the X server to discard overlapping blt operations when possible. Queuing the Blt at Vblank There are several options for getting the blt queued and executed when the vblank occurs:

1. Queue the blt from the interrupt handler
2. Queue the blt from a kernel thread running in response to the interrupt
3. Send an event up to user space and have the X server construct the blt command.

These are listed in increasing maximum latency, but also in decreasing complexity. Option 1. is made more complicated as much of the work necessary to get a command queued to the hardware cannot be done from interrupt context. One can imagine having the desired command already present in the ring buffer and have the interrupt handler simply move the ring tail pointer value. Future operations to be queued before the vblank operation could then re-write the ring as necessary. A queued operation could also be adjusted by the X server as necessary to keep it correct across changes to the window system state. Option 2. is similar, but the kernel implementation should be quite a bit simpler as the queuing operation is done in process context and can use the existing driver infrastructure. For the X server, this is the same as Option 1, requiring that it construct a queued blt operation and deliver that to the kernel, and then revoke and re-queue if the X server state changed before the operation was completed. Option 3 is the simplest of all, requiring no changes within the kernel and few within the X server. The X server waits to receive a vblank notification event for the appropriate frame and then simply invokes existing mechanisms to construct and queue the blt operation to the kernel. Oddly, Present currently uses Option 3. If that proves to generate too many display artifacts, we can come back and change the code to try something more complicated. Flipping the Frame Buffer Taking advantage of the hardware s ability to quickly shift scanout from one chunk of memory to another is critical to providing efficient buffer presentation within the X server. It is slightly more complicated to implement than simply copying data to the current scanout buffer for a few reasons:

1. The presented pixmap is owned by the application, and so it shouldn t be used except when the presented window covers the whole screen. When the window gets reconfigured, we end up copying the window s pixmap to the regular screen pixmap.
2. The kernel flipping API is asynchronous, and doesn t provide any abort mechanism. This isn t usually much of an issue; we simply delay reporting the actual time of flip until the kernel sends the notification event to the X server. However, if the window is reconfigured or destroyed while the flip is still pending, cleaning that up must wait until the flip has finished.
3. The applications buffer remains busy until it is no longer being used for scanout; that means that applications will have to be aware of this and ensure that they don t deadlock waiting for the current scanout buffer to become idle before switching to a new scanout buffer.

Present is different from DRI2 in using application-allocated buffers for this operation. For DRI2, when flipping to a window buffer, that buffer becomes the screen pixmap the driver flips the new buffer object into the screen pixmap and releases the previous buffer object for other use. For Present, as the buffer is owned by the application, I figured it would be better to switch back to the real screen buffer when necessary. This also means that applications aren t left holding a handle to the frame buffer, which seems like it might be a nice feature. The hardest part of this work was dealing with client and server shutdown, dealing with objects getting deleted in random orders while other data structures retained references. (The kernel DRM drivers use the term page flipping to mean an atomic flip between one frame buffer and another, generally implemented by simply switching the address used for the scanout buffer. I d like to avoid using the word page in this context as we re not flipping memory pages individually, but rather a huge group of memory that forms an entire frame buffer. We could use plane flipping (as intel docs do), frame buffer flipping (but that s a mouthful), display flipping or almost anything but page flipping ). Overall DRI3000 Status At this point, the DRI3 extension is complete and the Present extension is largely complete, except for redirection for compositors. The few piglit tests for GLX_OML_sync_control all pass now, which is at least better than DRI2 does. I think I ve effectively replicated the essential features of DRI2 while offering room to implement a couple of new GL extensions:

• GLX_EXT_swap_control_tear. This will provide applications with the ability to avoid dropping frames when pushing the hardware just over the frame rate limit.
• EGL_EXT_buffer_age. (I assume we ll probably want a GLX version as well?) This will allow compositors to more efficiently perform partial updates in a flipping environment, and is enabled by having all of the buffer management within the GL library.

The code for this stuff has all been pushed to a number of repositories:

• git://people.freedesktop.org/~keithp/dri3proto.git master. DRI3 protocol specification and X server headers.
• git://people.freedesktop.org/~keithp/presentproto.git master. Present protocol specification and X server headers.
• git://people.freedesktop.org/~keithp/xcb/proto.git dri3. XCB protocol defines for both DRI3 and Present.
• git://people.freedesktop.org/~keithp/xcb/libxcb.git dri3. XCB library changes for file descriptor passing.
• git://people.freedesktop.org/~keithp/xserver.git dri3. X server with file descriptor passing, DRI3 and Present support.
• git://people.freedesktop.org/~keithp/mesa.git dri3 Mesa with DRI3/Present support for GLX.
• git://people.freedesktop.org/~keithp/drm.git dri3. DRM library with defines for async flipping.
• git://people.freedesktop.org/~keithp/xf86-video-intel.git dri3. Intel driver with DRI3, Present and async flipping support.
• git://git.kernel.org/pub/scm/linux/kernel/git/keithp/linux-2.6 dri3. Kernel with async flipping.

Asynchronous Display Flipping for Linux Using display (or plane or page) flipping is a nice optimization for screen updates as it avoids copying the application render buffer to the separate scanout buffer. However, in the current DRM interface, the flip operation is not performed immediately. Rather, the flip operation must be presented to the kernel before the vertical blank interval preceding the desired display time. If the flip operation is even slightly delayed, the new scanout image will not be visible until the following frame. With many applications pushing the performance of the graphics hardware right to the edge, it s not at all uncommon for an occasional frame to just miss and cause an ugly jerk on the screen. The GLX_EXT_swap_control_tear extension provides a way for applications to request that when the swap request is delivered too late, instead of delaying for the next frame, the contents should be presented as soon as possible, even if that might cause some tearing artifacts on the screen. This gives the application developer the choice between the two visual artifacts (tearing vs judder). A Linux interface for Async Flipping This part was pretty easy to implement. First, a new capability value, DRM_CAP_ASYNC_PAGE_FLIP was added so that applications could discover whether the driver offered this feature or not. Second, a new flag value, DRM_MODE_PAGE_FLIP_ASYNC was added to the DRM_IOCTL_MODE_PAGE_FLIP ioctl. To get the new flag down into the driver required adding the flag parameter to the page_flip crtc function. Async Display Flipping support in Intel Hardware Recent Intel display hardware has support for asynchronous flipping, either through the ring using the MI_DISPLAY_FLIP instruction or directly through the registers. There are some limitations however:

• The pitch of the current and new frame buffers must match
• Both current and new frame buffers must use X tiling

I ve also only implemented flipping on Sandybridge and Ivybridge hardware as that s what I ve got for testing. Using Async Display Flipping with Present I m using async display flipping when the kernel supports it in the current Present implementation. This offers the ability to short-circuit multiple presentations targeting the same scanout frame. I haven t exposed the ability to select this to applications over the Present protocol yet, but that s not a big change. Implementing the GLX_EXT_swap_control_tear extension I haven t even looked at implementing this extension; I m hoping that once I ve gotten the functionality exposed over the X protocol that someone will at least help me figure out how to hook this up. Getting the Bits The interesting pieces here are the new kernel bits; the DRM library changes simply add new defines to existing APIs.

• Kernel bits are in my kernel.org repository git://git.kernel.org/pub/scm/linux/kernel/git/keithp/linux-2.6 dri3
• DRM library bits are in my freedesktop.org drm repository git://people.freedesktop.org/home/keithp/drm.git dri3

Jon Masters, Chief ARM Architect at Red Hat, recently posted a description of his expectations for baseline arm64 servers. The quick summary is that systems should implement UEFI and ACPI, and any more traditional ARM boot mechanisms should be ignored. This is an interesting departure from the status quo in the ARM world, and it's worth thinking about the benefits and drawbacks of this approach.

It's very easy to build a generic kernel for most x86 systems, since the PC platform is fairly well defined even if not terribly well specified. Where system hardware does vary, it's almost always exposed on an enumerable bus (such as PCI or USB) which allows the OS to bind appropriate drivers. Things are different in the ARM world. Even once you're past the point of different SoC vendors requiring different kernel setup code and drivers, you still have to cope with the fact that system vendors can wire these SoCs up very differently. Hardware is often attached via GPIO lines without any means to enumerate them. The end result is that you've traditionally needed a different kernel for every ARM board. This is viable if you're selling the OS and hardware as a single product, but less viable if there's any desire to run a generic OS on the hardware.

The solution that's been adopted for this in the Linux world is called Device Tree. Device Tree actually has significant history, having been used as the device descriptor format in Open Firmware. Since there was already support for it in the Linux kernel, adapting it for use in ARM devices was straightforward. Device Tree aware devices can pass a descriptor blob to the kernel at startup[1], and devices without that knowledge can have a blob build into the kernel.

So, if this problem is already solved, why the push to move to UEFI and ACPI? This push didn't actually originate in the Linux world - Microsoft mandate that Windows RT devices implement UEFI and ACPI, and were they to launch a Windows ARM server product would probably carry that over. That makes sense for Microsoft, since recent versions of Windows have been x86 only and so have grown all kinds of support for ACPI and UEFI. Supporting Device Tree would require Microsoft to rewrite large parts of Windows, whereas mandating UEFI and ACPI allowed them to reuse most of their existing Windows boot and driver code. As a result, largely at Microsoft's behest, ACPI 5 has grown a range of additional features for describing things like GPIO pinouts and I2C connections. Whatever your weird device layout, you can probably express it via ACPI.

This argument works less well for Linux. Linux already supports Device Tree, whereas it currently doesn't support ACPI or UEFI on ARM[2]. Hardware vendors are already used to working with Device Tree. Moving to UEFI and ACPI has the potential to uncover a range of exciting new kernel issues and vendor bugs. It's not obviously an engineering win.

So how about users? There's an argument that since server vendors are now mostly shipping ACPI and UEFI systems, having ARM support these technologies makes it easier for customers to replace x86 systems with ARM systems. This really doesn't fly for ACPI, which is entirely invisible to the user. There are no standard ACPI entry points for system configuration, and the bits of ACPI that are generically useful (such as configuring system wakeup times) are already abstracted away to a standard interface by the kernel. It's somewhat more compelling for UEFI. UEFI supports a platform-independent bytecode language (EFI Byte Code, or EBC), which means that customers can write their own system management utilities, build them for EBC and then deploy them to their servers without caring about whether they're x86 or ARM. Want a bootloader that'll hit an internal HTTP server in order to determine which system image to deploy, and which works on both x86 and ARM? Straightforward.

Arnd Bergmann has a interesting counterargument. In a nutshell, ARM servers aren't currently aiming for the same market as x86 servers, and as a result customers are unlikely to gain any significant benefit from shared functionality between the two.

So if there's no real benefit to users, and if there's no benefit to kernel developers, what's the point? The main one that springs to mind is that there is a benefit to distributions. Moving to UEFI means that there's a standard mechanism for distributions to interact with the firmware and configure the bootloader. The traditional ARM approach has been for vendors to ship their own version of u-boot. If that's in flash then it's not much of a problem[3], but if it's on disk then you have to ship a range of different bootloaders and know which one to install (and let's not even talk about initial bootstrapping).

This seems like the most compelling argument. UEFI provides a genuine benefit for distributions, and long term it probably provides some benefit to customers. The question is whether that benefit is worth the flux. The same distribution benefit could be gained by simply mandating a minimum set of u-boot functionality, which would seem much more straightforward. The customer benefit is currently unclear.

In the end it'll probably be a market decision. If Red Hat produce an ARM product that has these requirements, and if Suse produce an ARM product that will work with u-boot and Device Tree, it'll be up to vendors to decide whether the additional work to support UEFI/ACPI is worth it in order to be able to sell to customers who want Red Hat. I expect that large vendors like HP and Dell will probably do it, but the smaller ones may not. The customer demand issue is also going to be unclear until we learn whether using UEFI is something that customers actually care about, rather than a theoretical benefit.

Overall, I'm on the fence as to whether a UEFI requirement is going to stick, and I suspect that the ACPI requirement is tilting at windmills. There's nothing stopping vendors from providing a Device Tree blob from UEFI, and I can't think of any benefits they gain from using ACPI instead. Vendor interest in the generic parts of the ACPI spec has been tepid even in the x86 world (the vast majority of ACPI spec updates come from Microsoft and Intel, not any of the system vendors), and I don't see that changing with the introduction of a range of ARM vendors who are already happy with Device Tree.

We'll see. Linux is going to need to gain the support for UEFI and ACPI on ARM in any case, since there's already hardware shipping in that configuration. But with ARM vendors still getting to grips with Device Tree, forcing them to go through another change in how they do things is going to be hard work. Red Hat may be successful in enforcing these requirements at the cost of some vendor unhappiness, or Red Hat may find that their product doesn't boot on most of the available hardware. It's an aggressive gamble, and while it'll be interesting to see how it plays out, I'm not that optimistic.

[1] The blob could be pulled from the firmware, but it's not uncommon for it to be built into u-boot instead. This does mean that you have a device-specific u-boot even if you have a generic kernel, but that's typically true anyway.
[2] Patches have been posted for ARM UEFI support. They're not mergeable in their current form, but they should be in the near future. ACPI support is in development.
[3] Although not all u-boots are created equal - some vendors ship versions that will only boot off FAT, some vendors ship versions that will only boot off ext2. Having to special case this stuff in your installer is a pain.

comments

These are some very rough/sketchy notes that I took while doing the exercises on the introductory track of the Ruby language at codeacademy. They were written from the point-of-view of an amateur programmer in C and Python (read: "me"). Since this is my first time with Ruby and I only have about 1 week of experience with it, these notes are likely to contain many mistakes, many simplifications, many omissions or simply gross errors. I would love to be informed of such facts so that I can fix them and not write code that is burdensome for others to maintain. Operators

• Math operators: +, -, *, /, %, ** (just like Python)
• Relational operators: ==, <, >, <=, >= (just like other languages)
• Boolean operators: &&, , ! (just like C/C++/Perl)
• The combined comparison operator ("spaceship operator", like Perl): <=> (follows the spirit of what strcmp in C does)
• Conditional assignment operator ( =) only assigns a value to a variable if that variable is bound to nil.

Numbers

• You can write number literals embedding an underscore (_) for readability purposes.

Basic I/O

• Output: print and puts. Both print strings to stdout, but puts also appends a new line to the end.
• Input: gets accepts an input from stdin and returns a string.
• To convert a string foo to an integer, use the Integer(foo) constructor. It throws an exception if it cannot convert the value.

Strings

• Strings have at least some methods: .length, .reverse, .upcase, .downcase, .capitalize, .chomp (like Perl, removes the trailing endline)
• .upcase seems to have problems with non-ASCII characters, like .
• Interpolation is performed by putting the name name of the variable inside a string and delimited as # name at the place we want the interpolation to occur.
• Methods like .capitalize don't change the string (they return a new string capitalized). To change the string, you call the method .capitalize! (with a ! at the end of the name of the method).
• Other useful methods: .gsub for globally substituting a regular expression (and its variant .gsub! for inplace modification), and .include? to verify if the first string includes another.
• Another method: .split(" "), to split on the string specified as argument.

Control flow If

• The syntax of a if is:

    if expression1
      block1
    elsif expression2
      block2
    else
      block3
    end
  

  note that it ends with the word end, there are no need for parentheses, no braces, no colons, and the keyword for "else if" is elsif (different from Python's elif).
• The "if" can also be started with the keyword unless instead, which checks if the expression is false rather than true (as is the case of if).

While

• The syntax of the while is similar to other languages:

    while condition
      block
    end
  

• A variation of the while is the until, with the same syntax, except for the until in place of the while, and that it iterates when the condition is false, instead of true.

For

• The simple case of iterating over a range of numbers is:

    for i in 1...10
      block
    end
  

  Like ranges in Python, the 10 is not included (the variable i iterates from 1 to 9). For an inclusive range, we use two dots (1..10), instead of three dots (1...10).

Loop

• To iterate a block, we can use the loop keyword, as in:

    loop   block   # with braces
  

  or as in

    loop do # note the pair do/end in place of braces
      block
    end
  

• It is useful to use break when using loop to break out of the infinite iterations.
• Similarly, we can use next as in other languages (the same as continue in C or C++).

Arrays

• The equivalent to Python's lists are arrays and they are declared in the same fashion:

    my_array = [1, 2, 3, 4, 5]
  

• Arrays in Ruby are 0-based.
• Every array has an iterator called the .each method. The basic syntax is:

    object.each    item  block   # do something
  

  which can be written as:

    object.each do  item  block end
  

  where the name between is bound to (a copy of?) each element of object in turn. The equivalent in Python would be:

    for item in object:
        block
  

• Arrays can be sorted with the .sort method (or its sibling .sort!). When passing a parameter, it is useful to use the spaceship operator (<=>) (like Perl) to specify a custom order (similar in spirit to what we do with Python's key argument to the sort method of a list). Example:

     my_array.sort!  a, b  a <=> b  # Ascending order
     my_array.sort!  a, b  b <=> a  # Descending order
  

• The .push method appends an element to an Array (similar to Python's .append).
• The empty Array is denoted by [], like an empty list in Python.
• An iterator (like (1..10)) can be converted to an array with the .to_a method.

Hashes

• A literal hash is defined by the syntax:

    my_hash =  
        "key1" => "abc",
        "key2" => 2,
        "key3" => false
     
  

  Quite similar to the version in Python:

    my_hash =  
        "key1": "abc",
        "key2": 2,
        "key3": False
     
  

  In fact, in Ruby 1.9, you can write, instead:

    my_hash =  
        key1: "abc",
        key2: 2,
        key3: false
     
  

  which is very similar to Python or JSON.
• The empty hash, OTOH, is defined by calling the constructor .new of the class Hash:

    my_hash = Hash.new
  

  The syntax used by Python can also be used: my_hash =
• Iterating over a hash is done the same way as done for an array, but the .each method gives both the key and the value, as in:

    my_hash.each do  k, v 
      puts "# k : # v "
    end
  

• To iterate only on keys, we use the method .each_key. To iterate only on values, we use the method .each_value.
• A hash can be sorted with the .sort_by method (to sort by keys, values or any expression involving them) and since it can be ordered, we can reverse (in-place) the order of its elements with the .reverse! method. This makes Ruby's hashes to "taste" like Python's OrderedDict's.
• When one initializes a hash, one can specify a default value to the .new method (e.g., foo = Hash.new(0)), which makes it convenient to do the equivalent of Python's defaultdict or Counter.
• You can select part of a hash by using the .select method and giving it a block. For example:

    good_movies = movie_ratings.select k, v  v > 3 
  

Functions/methods

• Functions in Ruby are defined with the word def (as in Python) and end with the word end, as other composite constructs. Example:

    def square(n)
      return n*n
    end
  

• Parentheses are optional in Ruby, unless they create ambiguity. So, to call the function above, one can use any of the following:

    square(2)
    square 2
  

• A function can take a block as a parameter (and this is what .each does, for instance).
• Functions can take a variable number of elements with the splash operator (denoted by *), which puts the extra parameters in an Array (like Python).
• Functions can have default parameters, which are set in the "prototype", just like in Python or C++.
• If the return keyword is omitted from the method, Ruby uses an implicit return, where the value of the last executed expression is returned to the caller of the method.

Symbols

• Any instances of a given symbol share the same ID. This is due to the fact that they are immutable.
• Symbols start with a colon (:) and they can contain any character that is valid for a variable name.
• It is possible to convert between a symbol and a string with the methods .to_s and .to_sym.

Comments

• Single line comments use the hash symbol (#).
• Ruby also has multiline comments, which are delimited by a line beginning with =begin and a line beginning with =end.

Lambda functions

• A lambda function in Ruby is defined, as expected, as, for example:

    foo = lambda    x  x.to_sym  
  

• The basic differences between a lambda and a proc is that:
  • A lambda checks the number of elements that it receives, while a proc doesn't and assigns nil to parameters that it didn't receive (a proc seems to ignore extra parameter that it wasn't expecting).
  • When a lambda returns, the control goes back to the function that called the lambda, while when a proc returns, the control is returned also from the function that called the proc.

Classes

• "Everything is an object in Ruby." (At least one exception are blocks).
• To test if an object is of a given class, you use the method .is_a? on the object and the type as argument (e.g., foo.is_a? Integer)
• The "null" value in Ruby is nil. It is equivalent to None in Python.
• The only "falsy" values in Ruby are false and nil. The others are "truthy", differently from C/C++/Perl/Python.
• For a given object, the method .object_id returns what is the "address" of that object.
• Classes in Ruby are created with the syntax

    class ClassExample
        # body
    end
  

  (That is for a class that has is not derived from other regular classes).
• Constructors in Ruby are called initialize. An instance of a class is created by calling the method .new on the class name (and passing parameters as needed for the initialize method).
• Instance variables are named with 1 at sign (@) as a prefix. Class variables are named with 2 at signs (@@) as a prefix. Global variables are named with a dollar sign ($) as a prefix.
• Class methods are declared inside a class prefixed with self. and they are called with the class name suffixed with the name of the method. For example, for a class MyClass, with a class method class_method, we call it as MyClass.class_method.
• When overriding a method in a subclass, one can call the method from the parent class with the keyword super, as in super(arg1, arg2).
• Ruby doesn't support multiple inheritance. In place of multiple inheritance, Ruby offers mixins.
• To access one attribute, say @my_attrib, in Ruby classes, one usually creates two methods to manipulate it: a method with the name my_attrib, that returns the value of @my_attrib (a getter) and a method with the name my_attrib= that accepts a parameter to set @my_attrib to (a setter). There is syntatic sugar for creating both getters or setters. For a getter, you declare attr_reader :my_attrib (the member name as a symbol). For a setter, you declare attr_writer :my_attrib. If you need both a getter and a setter, then you can abbreviate them with attr_accessor.
• All methods, by default, are public in Ruby. To explicitly mark something as public, you use the keyword public. To mark something as private, you use the keyword private.

Modules

• You can import a module with the keyword require (as in Perl) and the name of the module in quotes.
• One example of a module is the benchmark module, which provides the Benchmark class, which has the method .realtime, which takes a block and reports how much time it used.
• To create a module, use a syntax similar to that of a class, but substituting the keyword class with module.
• The scope resolution operator in Ruby is the double colon (::), like in Perl.
• To create a method inside a module, you have to prefix it by the name of the module with a dot or with self., if you want to create a method to be called outside the module. The same also holds for creating class methods (as opposed to instance methods): they can be simply named self.foo to get a class method called foo.
• While some modules are already "preloaded" by the interpreter (e.g., Math), others need to be explicitly loaded. We do this by using require 'foo' to load the module (in the file foo?).

Mixins

• A mixin is the use of a module to "enrich" a class, adding "instance" methods of the module to the class. It is, in a sense, a way to have some of multiple inheritance in Ruby (without its headaches?)
• Using the word include Foo imports the items of the module Foo in the current namespace for instances of a class.
• Using the word extend Foo imports the items of the module Foo in the current namespace as class methods/attributes.

Conventions

• Local variables are named in lower case, with parts separated by underscores, as in my_name.
• Methods that return boolean values have a question mark (?) at the end of their name.
• Methods that have destructive side-effects (like changing their input) have an exclamation mark (!) at the end of their name.
• Code is indented by 2 spaces.
• Like other languages, constants are written in all caps, with words separated by underscores. For example, MAX_VALUE.
• Global variables are prefixed with a dollar sign (e.g., $Foo).
• Classes are named in camel case (no initial lowercase).
• The "poetic mode" in Ruby recommends one to omit the parentheses on function calls (whenever unambiguous) and, in essence, to write as little "bureaucracy" (like the explicit return keyword in methods).

About two months ago I was writing about a quirk I found in GNU Make related to the $(patsubst ) function.

I have just tried this on my Debian Wheezy laptop which has make 3.81, but I wasn't able to reproduce the issue with the version from Debian (3.81-8.2).

The makefile looks like this:

PATH := ../some/prefixCPU12suf/include
CPUINC := $(patsubst ../some/prefix%,%,$(PATH))
CPU := $(patsubst %/include,%,$(CPUINC))

default:
@echo "PATH = $(PATH)"
@echo "CPUINC = $(CPUINC)"
@echo "CPU = $(CPU)"

And the result was correct:

0 eddy@heidi /tmp $ make
PATH = ../some/prefixCPU12/include
CPUINC = CPU12/include
CPU = CPU12
0 eddy@heidi /tmp $ make --version
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for x86_64-pc-linux-gnu

The worst part is that I know I tested this issue on 3.82 on Cygwin and on Linux with the 3.82 version and it failed, but I wasn't able to remember how I did it. I started searching through the directory where I knew there could be the test makefile, I wasn't able to find it, until I remembered what I was trying to achieve.

From a path like ../some/prefixCPU12suf/include I wanted to use % to remove the parts 'some/prefix' and 'suf/include' because in the directory ../CPU12 there were some files that needed to be processed.

The actual issue is that GNU Make's '%' is not analogous to shell's '*', so that means code like this does not work as I assumed anf the 'pref' part is not an anchor:

PATH := ../some/prefCPU12suf/include
CPUINC := $(patsubst pref%,%,$(PATH))
CPU := $(patsubst %suf/include,%,$(CPUINC))

default:
@echo "PATH = $(PATH)"
@echo "CPUINC = $(CPUINC)"
@echo "CPU = $(CPU)"

Which leads to these results, no matter the version:

0 eddy@heidi ~/usr/src/make/make-profiler/make-3.82 $ ./make -f /tmp/makefile
PATH = ../some/prefCPU12suf/include
CPUINC = ../some/prefCPU12suf/include
CPU = ../some/prefCPU12
0 eddy@heidi ~/usr/src/make/make-profiler/make-3.82 $ ./make --version
GNU Make 3.82
Built for x86_64-unknown-linux-gnu
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http: gnu.org="" gpl.html="" licenses="">
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
0 eddy@heidi ~/usr/src/make/make-profiler/make-3.82 $ make -f /tmp/makefile
PATH = ../some/prefCPU12suf/include
CPUINC = ../some/prefCPU12suf/include
CPU = ../some/prefCPU12
0 eddy@heidi ~/usr/src/make/make-profiler/make-3.82 $ make --version
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for x86_64-pc-linux-gnu</http:>

Not sure if this could be qualified as a true bug, or a if the way I expected is a nice to have feature, but, in any case, the behaviour is consistent, unlike my brain which failed to initially identify the inconsistency in my code:

0 eddy@heidi ~/usr/src/make/make-profiler/make-3.82 $ grep patsubst /tmp/makefile
CPUINC := $(patsubst pref%,%,$(PATH))
CPU := $(patsubst %suf,%,$(CPUINC))
0 eddy@heidi ~/usr/src/make/make-profiler/make-3.82 $ make -f /tmp/makefile
PATH = ../some/prefCPU12suf/include
CPUINC = ../some/prefCPU12suf/include
CPU = ../some/prefCPU12suf/include

Note that this behaviour of patsubst is asymtric to how subst works, as explained in the updated old post.

This took some extra effort to remember what was the actual issue, and shows why logs are important when reporting an issue, and why reporting issues as soon as they were encountered: because human brains are faulty. (Yes, yours, too!)

This week's project is a quick t-shirt reconstruction with the reverse appliqu technique which basically means layering two different fabrics, sew them togheter, and then cut away part of the top one to reveal the one underneath.
Ready to start? What you will need

• a plain t-shirt in your size
• some fabric for the patch
• fusible interfacing
• fabric shears
• small and sharp scissors
• pins
• needle and thread (or, if you prefer, a sewing machine)
• pencil

A guitar, Infinite Jest, The Illuminatus Trilogy and The Year of the Flood, while suggested for your personal enjoyment, are not strictly necessary for this project. Step 1 Cut a square of red fabric of the desired dimensions, I've used the short sleeve of a ruined red t-shirt.
On the wrong side of your patch, apply the fusible interfacing. Step 2 Draw a swirl on the interfacing.
IMPORTANT: remember to draw the swirl as mirrored, because the interfacing is the wrong side of our patch.
So, if usually the Debian swirl has the back to the left and the front to the right - like a "(" - when you draw it on the interfacing it will need to have the back to the right and front to the side - like a ")".
I do realize that this explanation isn't really clear: just look at the picture above. Step 3 Turn your t-shirt inside out and pin the patch (with the interfacing up) in the place you want to put it: I decided to put mine on the lower left corner, approximately 10 cm above the bottom hem.
(In the picture is lower, I changed idea on the position just before sewing it)
Now you can sew it, following closely the drawn line.
Because I'm lazy, I decided to do it by hand, but if you're not afraid of sewing curved shapes, doing it with the sewing machine is definitely the right (and quick) way to do it. Step 4 Now, you can cut the extra fabric around your swirl. Step 5 Turn your shirt right side out.
Make a snip through the knit in the fabric on the inside part of your stitches and cut off the extra fabric, to reveal the red swirl underneath.
Be careful to not cut the red fabric! Step 6 If you like it, you can end embroidering a dashed line (with running stitches) in a contrasting thread all around the swirl. And you're finished: now you can enjoy your new Debian t-shirt! :) For any questions feel free to contact me.

I'm sure you all will be a bit disappointed (I know I am) that there are no ascii screenshots in this weeks report. But let me make it up to you by telling you a story. I was at the FOSDEM in the year 2012 and I went to a nice workshop on sunday afternoon in the Virtualization Devroom. Renzo Davoli was the host of this workshop and he started with a little introductory talk to get everyone to agree on a common terminology first. He began by asking the question, what virtualization meant in the most general way. He defined the ability to virtualize a resource as the ability to freely and transparently control how someone (say a process) is interacting with said resource. So for example if you LD_PRELOAD a library to impersonate fopen(3) and friends, you have virtualized the filesystem (for some small values of virtualized). He then went on to discuss various methods of virtualization that are commonly available on Linux, not only full system virtualization solutions, but also all kinds of methods allowing a more fine-grained control over what resources are virtualized. Of course every method had its strengths and its weaknesses. Having seen Samuels talk about all the awesome things Hurd can do for virtualization I approached him with a question in the free discussion part of his workshop. I asked whether he would agree that once a resource is moved from the kernel to the userspace, the problem of virtualizing that resource is almost trivially solved, and he agreed. So I said that this was awesome, because then the trivial and elegant solution for all his virtualization (and tracing) needs are micro kernel operating systems, and he also agreed to that but (as far as I can recall) he mentioned that there is none that would meet his needs (I believe he is a computer science professor and heavily relies on virtualization techniques for his lab and for teaching purposes). And even though a microkernel operating systems has a cost (message passing instead of function calls for example) it also has its merits such as scaling better from a development point of view. Also, many cool features can emerge just from the design. For example think about the container support in Linux and how painfully long it took to make all the resources namespace aware (with one of the most critical, the user namespace being the most difficult one). On Hurd you get the same functionality for free. If you are curious, read Samuels awesome slides. So what have I done this week?

• procfs should ignore --no dev,exec,suid : http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00076.html
• libnetfs does not respond properly to file_get_translator requests on nodes with a passive translator record: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00148.html
• As the mtab translator is a translator of its own, there needs to be a way to bind it to /proc/mounts. I figured the easiest way would be to just serve a node with an appropriate passive translator record: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00149.html
• There was no clean way to start the console-client. I added daemonizing support to the console-client and created an appropriate initscript: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00165.html
• I also worked a lot on my mtab translator prototype: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00183.html

Next week I'm going to work on the two remaining sysvinit related issues. These are the only ones preventing me to come up with a clean patch series against the sysvinit package and I figured that it would be nice to propose such a series rather sooner than later so that I will have plenty of time to discuss any issues with the sysvinit maintainers. See you next week :)

Michael Langguth and Scalaris AG asked me to publish the mksh/Win32 Beta 14 source and binary archive, and it is with joy I m doing this. Checksums and Hashes

• RMD160 (ports/mksh-w32-beta14.zip) = 0dc8ef6e95592bd132f701ca77c4e0a3afe46f24
• TIGER (ports/mksh-w32-beta14.zip) = 966e548f9e9c1d5b137ae3ec48e60db4a57c9a0ed15720fb
• 1181543005 517402 /MirOS/dist/mir/mksh/ports/mksh-w32-beta14.zip
• MD5 (ports/mksh-w32-beta14.zip) = b57367b0710bf76a972b493562e2b6b5

Just a few words on it (more in the README.1st file included): this is a port of The MirBSD Korn Shell R39 to the native WinAPI; it s not quite got the full Unix feel (especially as it targets the Weihenstephan unxutils instead of a full Interix or Cygwin environment) but doesn t need a full POSIX emulation layer either. It s intended to replace MKS ksh and the MKS Toolkit. Source for the compatibility library is also included under The MirOS Licence; we aim at publishing it as OSI Certified Open Source Software like mksh itself. (There is a situation with dlmalloc/nedmalloc being resolved, and the icon is derived from the BSD d mon which is a protected unregistered trademark, but we re not Mozilla and allow distro packages to keep using it ) Rebasing it on a newer mksh(1) followed by (partial) integration into the main source code is a goal. Have fun trying it out and hacking on it. It s currently built with -DMKSH_NOPROSPECTOFWORK (so coprocesses and a few other minor things won t work), but a SIGCHLD emulation is being worked on but if you want to help out, I m sure it s welcome, just come on IRC or post on the mailing list, and I ll forward things to Michael as needed. Reports on testing with other toolchain and OS versions are also welcome.

There are three main ways to distribute a command-line daemon for OS X:

1. Distributing source code and instructions on how to compile it.
2. Using a third-party package manager, like Homebrew.
3. Providing an installer package.

Homebrew Homebrew is a popular package management system. It works like the BSDs ports collections by downloading, compiling and installing the requested software, while also installing any required dependencies automatically. Creating a new package is quite easy and there are a lot of examples available. However, there are some limitations:

• You don t really build a package but execute a recipe to locally install the software.
• You need to install development tools, either a whole Xcode installation¹ or the command line version.
• If you need to execute some steps as root, you will have to explain them to the user for her to execute by hand. This includes the creation of a system user or the installation of a daemon through launchd.

If you can t live with those limitations, you may need to build an installer package.

Building an installer package OS X comes with a graphical and a command-line installer. The graphical one is run by opening a package from the Finder. Then, the user experiences some familiar wizard allowing her to install the software in a matter of a few seconds. The documentation around how to build such a package is somewhat clumsy. You may find outdated information or pieces of information that only apply to projects using Xcode. I will try to provide here accurate bits in the following context:

1. You are packaging a pure command-line tool.
2. You are using Autoconf and Automake as a build system.
3. You want to support several architectures.
4. You want to support older versions of OS X.

Creating a package Building such a package was previously done with some graphical tool called PackageMaker. This tool is not available any more and developers are asked to switch to pkgbuild and productbuild. There is a very neat Stackoverflow article on how those tools work. A package is built in two steps:

1. Build component packages.
2. Combine them into a product archive.

A component package contains a set of files and a set of scripts to execute at various steps of the installation. You can have several component packages, for example a package for the daemon and a package for the client. They are built with pkgbuild. A product archive contains the previously created component packages as well as a file describing various options of the installer (required and optional components, license, welcome text, ). To create a component package, we need to install the needed files in some directory:

$ ./configure --prefix=/usr --sysconfdir=/private/etc
$ make
$ make install DESTDIR=$PWD/osx-pkg

We will now put the content of osx-pkg into a component package with pkgbuild:

$ mkdir pkg1
$ pkgbuild --root osx-pkg \
>    --identifier org.someid.daemon \
>    --version 0.47 \
>    --ownership recommended
>    pkg1/output.pkg
pkgbuild: Inferring bundle components from contents of osx-pkg
pkgbuild: Wrote package to output.pkg

You need to be careful with the identifier. It needs to be unique and identify your software as well as the specific component. Then, you need to create an XML file describing the installer. Let s call it distribution.xml:

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<installer-gui-script minSpecVersion="1">
    <title>Some daemon</title>
    <organization>org.someid</organization>
    <domains enable_localSystem="true"/>
    <options customize="never" require-scripts="true" rootVolumeOnly="true" />
    <!-- Define documents displayed at various steps -->
    <welcome    file="welcome.html"    mime-type="text/html" />
    <license    file="license.html"    mime-type="text/html" />
    <conclusion file="conclusion.html" mime-type="text/html" />
    <!-- List all component packages -->
    <pkg-ref id="org.someid.daemon"
             version="0"
             auth="root">output.pkg</pkg-ref>
    <!-- List them again here. They can now be organized
         as a hierarchy if you want. -->
    <choices-outline>
        <line choice="org.someid.daemon"/>
    </choices-outline>
    <!-- Define each choice above -->
    <choice
        id="org.someid.daemon"
        visible="false"
        title="some daemon"
        description="The daemon"
        start_selected="true">
      <pkg-ref id="org.someid.daemon"/>
    </choice>
</installer-gui-script>

Fortunately, this file is documented on Apple Developer Library. Since we only describe one package, we pass customize="never" as an option to skip the choice part. You can however remove this attribute when you have several component packages. The attribute rootVolumeOnly="true" explains that this daemon can only be installed system-wide. It is marked as deprecated but still works. Its replacement (domains tag) displays an unusual and buggy pane which Apple doesn t use in any of its packages. You need to put the HTML² documents into a resources directory. It is also possible to choose the background by specifying a <background/> tag. The following command will generate the product archive:

$ productbuild --distribution distribution.xml \
>              --resources resources \
>              --package-path pkg1 \
>              --version 0.47 \
>              ../final.pkg
productbuild: Wrote product to ../final.pkg

Scripts The installer can execute some scripts during installation. For example, suppose we would like to register our daemon with launchd for it to be run at system start. You first need to write some org.someid.plist file and ensure it will get installed in /Library/LaunchDaemons:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>org.someid</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/sbin/mydaemon</string>
    <string>-d</string>
  </array>
  <key>RunAtLoad</key><true/>
  <key>KeepAlive</key><true/>
</dict>
</plist>

Then create a scripts/postinstall file with the following content:

#!/bin/sh
set -e
/bin/launchctl load "/Library/LaunchDaemons/org.someid.plist"

You also need a scripts/preinstall file which will allow your program to be smoothly upgraded:

#!/bin/bash
set -e
if /bin/launchctl list "org.someid" &> /dev/null; then
    /bin/launchctl unload "/Library/LaunchDaemons/org.someid.plist"
fi

Ensure that those scripts are executable and add --scripts scripts to pkgbuild invocation. If you need to add a system user, use dscl.

Dependencies There is no dependency management built into OS X installer. Therefore, you should ensure that everything is present in the package or in the base system. For example, lldpd relies on libevent, an event notification library. This library is not provided with OS X. When it is not found, the build system will use an embedded copy. However, if you have installed libevent with Homebrew, you will get binaries linked to this local libevent installation. Your package won t work on another host. The output of otool -L can detect unwanted dependencies:

$ otool -L build/usr/sbin/lldpd
build/usr/sbin/lldpd:
        /usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
        /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 744.19.0)
        /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 945.18.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

Compilation for older versions of OS X The package as built above may only work on the same version of OS X that you use to compile. If you want a package that will work on OS X 10.6, you will need to download the appropriate SDK³. Then, you need to tell the compiler the target version of OS X and the SDK location. You can do this by setting CFLAGS and LDFLAGS (and CXXFLAGS if you use C++):

$ SDK=/Developer/SDKs/MacOSX10.6.sdk
$ ./configure --prefix=/usr --sysconfdir=/private/etc \
>         CFLAGS="-mmacosx-version-min=10.6 -isysroot $SDK" \
>         LDFLAGS="-mmacosx-version-min=10.6 -isysroot $SDK"

The -mmacosx-version-min flag will be used by various macros in the headers to mark some functions as available or not, depending on the version of OS X you target. It will also be used by the linker. The -isysroot will tell the location of the SDK. Headers and libraries will be looked up in this location first.

Universal binaries Mac OS X 10.6 was available for both IA-32 and x86-64 architectures. If you want to support both installations with a single package, you need to build universal binaries. The Mach object file format, used by OS X, allows several versions of the executable in the same file. The operating system will select the most appropriate one. An easy way to generate such files is to pass -arch i386 -arch x86_64 to the compiler:

$ ./configure --prefix=/usr --sysconfdir=/private/etc \
>     CC="gcc -arch i386 -arch x86_64" \
>     CPP="gcc -E"

However, this is a dangerous option. Suppose that your ./configure script tries to determine some architecture-dependent parameter, like the size of an integer (with AC_CHECK_SIZEOF); it will compute it for the host architecture. The binary generated for the other architecture will therefore use a wrong value and may crash at some point. The correct way to generate an universal binary is to execute two separate compilations and to build the universal binary with lipo:

$ for arch in i386 x86_64; do
>   mkdir $arch ; cd $arch
>   ../configure --prefix=/usr --sysconfdir=/private/etc \
>       CC="gcc -arch $ARCH" \
>       CPP="gcc -E"
>   make
>   make install DESTDIR=$PWD/../target-$ARCH
>   cd ..
> done
[...]
$ lipo -create -output daemon i386/usr/sbin/daemon x86_64/usr/sbin/daemon

Since lipo only works on file, I have written a Python script applying lipo recursively to several directories.

Putting everything together Now, we need to automate a bit. You could provide some nifty script. I propose an appropriate Makefile.am instead. This way, we can use the output variables, like @VERSION@, from ./configure to generate some of the files, like distribution.xml. You need to add that in your configure.ac:

AC_CONFIG_FILES([osx/Makefile osx/distribution.xml osx/im.bernat.lldpd.plist])
AC_CONFIG_FILES([osx/scripts/preinstall],  [chmod +x osx/scripts/preinstall])
AC_CONFIG_FILES([osx/scripts/postinstall], [chmod +x osx/scripts/postinstall])
AC_SUBST([CONFIGURE_ARGS], [$ac_configure_args])

Let s have a look at osx/Makefile.am. First, we define some variables:

PKG_NAME=@PACKAGE@-@VERSION@.pkg
PKG_TITLE=@PACKAGE@ @VERSION@
PKG_DIR=@PACKAGE@-@VERSION@
ARCHS=@host_cpu@

If we want to build for several architectures, we will type make ARCHS="x86_64 i386". Otherwise it defaults to the current host s architecture. We use install-data-local target to install files specific to OS X:

install-data-local:
    install -m 0755 -d $(DESTDIR)/Library/LaunchDaemons
    install -m 0644 im.bernat.@PACKAGE@.plist $(DESTDIR)/Library/LaunchDaemons
uninstall-local:
    rm -f $(DESTDIR)/Library/LaunchDaemons/im.bernat.@PACKAGE@.plist

The main target is the product archive, built with productbuild:

../$(PKG_NAME): pkg.1/$(PKG_NAME) distribution.xml resources
    $(PRODUCTBUILD) \
        --distribution distribution.xml \
        --resources resources \
        --package-path pkg.1 \
        --version @VERSION@ \
        $@

Its main dependency is the component package:

pkg.1/$(PKG_NAME): $(PKG_DIR) scripts
    [ -d pkg.1 ]   mkdir pkg.1
    $(PKGBUILD) \
        --root $(PKG_DIR) \
        --identifier im.bernat.@PACKAGE@.daemon \
        --version @VERSION@ \
        --ownership recommended \
        --scripts scripts \
        $@

Now, we need to build $(PKG_DIR):

$(PKG_DIR): stamp-$(PKG_DIR)
stamp-$(PKG_DIR): $(ARCHS:%=%/$(PKG_DIR))
    $(srcdir)/lipo $(PKG_DIR) $^
    touch $@

The $(ARCHS:%=%/$(PKG_DIR)) will expand to x86_64/$(PKG_DIR) i386/$(PKG_DIR). This is a substitution reference⁴. Before applying our lipo script, we need to be able to build the architecture-dependent package directories:

pkg_curarch = $(@:stamp-%=%)
$(ARCHS:%=%/$(PKG_DIR)): %/$(PKG_DIR): stamp-%
$(ARCHS:%=stamp-%): stamp-%: im.bernat.lldpd.plist
    [ -d $(pkg_curarch) ]   mkdir -p $(pkg_curarch)
    (cd $(pkg_curarch) && \
        $(abs_top_srcdir)/configure @CONFIGURE_ARGS@ \
            CC="@CC@ -arch $(pkg_curarch)" \
            CPP="@CPP@")
    (cd $(pkg_curarch) && \
        $(MAKE) install DESTDIR=$(abs_builddir)/$(pkg_curarch)/$(PKG_DIR))
    touch $@

This may seem a bit obscure, but this is really like what we have described previously. Note that we use @CONFIGURE_ARGS@ which is a variable we have defined in configure.ac. Here is how a user would create its own OS X package:

$ SDK=/Developer/SDKs/MacOSX10.6.sdk
$ mkdir build && cd build
$ ../configure --prefix=/usr --sysconfdir=/private/etc --with-embedded-libevent \
   CFLAGS="-mmacosx-version-min=10.6 -isysroot $SDK" \
   LDFLAGS="-mmacosx-version-min=10.6 -isysroot $SDK"
[...]
$ make -C osx pkg ARCHS="i386 x86_64"
[...]
productbuild: Wrote product to ../lldpd-0.7.5-21-g5b90c4f-dirty.pkg
The package has been built in ../lldpd-0.7.5-21-g5b90c4f-dirty.pkg.

---

1. Even if Xcode is free, you will need to register to Apple and show a valid credit card number. That s quite an interesting requirement.
2. It is possible to use other formats (like RTF or just plain text) for documents but HTML seems the best choice if you want some formatting without fiddling with RTF.
3. Unfortunately, Apple makes it difficult to find them. The easiest way to get the SDK for Mac OS X 10.6 is to download Xcode 4.3.3, mount the image and copy the SDK to /Developer/SDKs.
4. This is equivalent to the use of patsubst function. However, this function is specific to GNU make. The substitution reference pattern was accepted recently in POSIX and is coined as pattern macro expansion.

Even though the backend of Rosenberg NMS uses rrdtool RRD files, the front end graphs are created using jqPlot. The idea is to have a set of templates for the different types of graphs and just apply them to the various data sets. It makes things a lot simpler for new graphs because you just select which one you want; unless you want something a lot different which would involve a new graph template. In any case, anyone that looks enough at the standard rrdtool graphs will know they are a series of steps. While it depends on the RRA, usually they are 5 minute steps, so a graph showing an increasing rate might show 5 minutes of 2 Mbps and then the next 5 minutes of 11 Mbps. jqPlot graphs as I ve currently got them draws a line between two data points, so there would be a sloping line starting at 2.5 minutes (half of the first 5 minute interval) and 2Mbps sloping up to 11 Mbps at 7.5 minutes. At first I thought this was wrong and spent some time attempting to fix the graph by making it look like a rrdtool graph more. Someway through that process I stopped and wondered, what IS the right way? The answer like a lot of other things, is it depends . For a graph showing a rate, such as the output bits per second on an interface, the way this is done is at regular intervals a counter is measured. So if at time 0 the counter is 140 and time 60 the counter is 200 and finally at time 120 the counter is at 800 there has been an average rate of 1 [(200-140)/60] and 100 [(800-200)/60)]. rrdtool would show a horizontal bar at 1 bps and then another horizontal bar at 100 at the next time interval. jqPlot would show a sloping line going from 30,1 up to 90,100. Two graphs looking very different from the same data, what gives? Each graph is right, or rather is showing an estimation of different things, hence their differences. rrdtool is showing the average rate for that time period. It is in some ways accurate because leaving aside missed wrap-arounds and resets that many bits did pass through that interface for the time specified. However often the graphs are interpreted incorrectly as the real rate and not an average. We can be reasonably sure for a rate that it would not be 1 for exactly a minute and then immediately jump to 100 for another minute. This isn t rrdtool s fault, it is just how the graphs can be interpreted. jqplot will show more realistic graphs, with a curve sloping up. However this too makes assumptions that the rate increase is linear which often it is not. It just gives the illusion that the graph knows more about the data than it really does. In the end, both graphs are at the same time accurate and misleading. It s important when looking at any graph in general (not just these two types) that you understand its limits and assumptions. To give one example of the problems that can be missed, traffic interfaces may have microbursts (large amount of traffic in short amount of time) which, due to the averaging that goes on in graphing are invisible to graphs and give an incorrect account of what is going on.

Today I switched to my new laptop. I've previously written about the problems I had with my new Thinkpad X230, which was delivered with an 180 GB Intel SSD disk with Lenovo firmware that did not handle sustained writes. My hardware supplier have been very forthcoming in trying to find a solution, and after first trying with another identical 180 GB disks they decided to send me a 256 GB Samsung SSD disk instead to fix it once and for all. The Samsung disk survived the installation of Debian with encrypted disks (filling the disk with random data during installation killed the first two), and I thus decided to trust it with my data. I have installed it as a Debian Edu Wheezy roaming workstation hooked up with my Debian Edu Squeeze main server at home using Kerberos and LDAP, and will use it as my work station from now on. As this is a solid state disk with no moving parts, I believe the Debian Wheezy default installation need to be tuned a bit to increase performance and increase life time of the disk. The Linux kernel and user space applications do not yet adjust automatically to such environment. To make it easier for my self, I created a draft Debian package ssd-setup to handle this tuning. The source for the ssd-setup package is available from collab-maint, and it is set up to adjust the setup of the machine by just installing the package. If there is any non-SSD disk in the machine, the package will refuse to install, as I did not try to write any logic to sort file systems in SSD and non-SSD file systems. I consider the package a draft, as I am a bit unsure how to best set up Debian Wheezy with an SSD. It is adjusted to my use case, where I set up the machine with one large encrypted partition (in addition to /boot), put LVM on top of this and set up partitions on top of this again. See the README file in the package source for the references I used to pick the settings. At the moment these parameters are tuned:

• Set up cryptsetup to pass TRIM commands to the physical disk (adding discard to /etc/crypttab)
• Set up LVM to pass on TRIM commands to the underlying device (in this case a cryptsetup partition) by changing issue_discards from 0 to 1 in /etc/lvm/lvm.conf.
• Set relatime as a file system option for ext3 and ext4 file systems.
• Tell swap to use TRIM commands by adding 'discard' to /etc/fstab.
• Change I/O scheduler from cfq to deadline using a udev rule.
• Run fstrim on every ext3 and ext4 file system every night (from cron.daily).
• Adjust sysctl values vm.swappiness to 1 and vm.vfs_cache_pressure to 50 to reduce the kernel eagerness to swap out processes.

During installation, I cancelled the part where the installer fill the disk with random data, as this would kill the SSD performance for little gain. My goal with the encrypted file system is to ensure those stealing my laptop end up with a brick and not a working computer. I have no hope in keeping the really resourceful people from getting the data on the disk (see XKCD #538 for an explanation why). Thus I concluded that adding the discard option to crypttab is the right thing to do. I considered using the noop I/O scheduler, as several recommended it for SSD, but others recommended deadline and a benchmark I found indicated that deadline might be better for interactive use. I also considered using the 'discard' file system option for ext3 and ext4, but read that it would give a performance hit ever time a file is removed, and thought it best to that that slowdown once a day instead of during my work. My package do not set up tmpfs on /var/run, /var/lock and /tmp, as this is already done by Debian Edu. I have not yet started on the user space tuning. I expect iceweasel need some tuning, and perhaps other applications too, but have not yet had time to investigate those parts. The package should work on Ubuntu too, but I have not yet tested it there. As for the answer to the question in the title of this blog post, as far as I know, the only solution I know about is to replace the disk. It might be possible to flash it with Intel firmware instead of the Lenovo firmware. But I have not tried and did not want to do so without approval from Lenovo as I wanted to keep the warranty on the disk until a solution was found and they wanted the broken disks back.

I have a word of warning against improper use of local in shell functions. If you are using shell functions, you might want to declare some variables local to the shell function. That is good. The basic syntax for that is

local a b c

In some shells, you can also combine the local declaration and assignment, like this:

local foo=$1
local bar=$2

(The Debian policy even explicitly allows it.) This is somewhat dangerous. Bare shell assignment like

foo=$bar

does not perform word splitting, so the above is safe even if there are spaces in $bar. But the local command does perform word splitting (because it can take multiple arguments, as in the first example), so the seemingly similar

local foo=$bar

is not safe. This can be really confusing when you add local to existing code and it starts breaking. You can avoid this, of course, by always quoting everything to like

local foo="$bar"

but overquoting isn't always desirable, because it can make code less readable when commands are nested, like

local foo="$(otherfunc "other arg")"

(Nesting is legal and works fine in this case, however.) I suggest using local only for declaring variables, and using separate assignment statements. That way, all assignments are parsed in the same way.

Today I held a lightning talk at the Karlsruhe Functional Programmer s Group, where I presented Real World Haskell applications, i.e. programs that are written in Haskell, are somewhat mature and well-known and have non-programmers as their target group. Here is the (surely subjective) list that I have come up (also see the slides):

• darcs (since 2002, 35 000 loc): Distributed version control system with an innovative focus on changes instead of states.
  
• xmonad (since 2007, 30000 loc): Well known tiling window manager with a huge library of layout and other plugins. Made it into the list despite its configuration file being a Haskell file.
• hledger (since 2007, 9000 loc): Text-file based double-ledger accounting tool, a clone of ledger.
• Raincat (since 2008, 2000 loc): Platform game with a cat that does not want to get wet.
• arbtt (since 2009, 2000 loc): My automatic rule-based time tracker. Made it into the list as a shameless plug; probably not that popular. It has now a proper web page contributed by Waldir Pimenta.
• detexify (since 2010, 500 loc): The back end of the very useful LaTeX character command finder is written in Haskell.
• git-annex (since 2010, 28 000 loc): Manages your files and their location, a mixture of dropbox and git. Written by famous Joey Hess, who made a living from it via kickstarter He is currently running a second round of funding!
• Nikki and the Robots (since 2010, 18 000 loc): Platform game with Nikki and, well, his robots. It was produced as a commercial independent game and sold via a pay-what-you-like scheme, but the company unfortunately closed down.
• hoodle (since 2011, 13 000 loc): A note-taking and PDF annotation software like xournal.
  
• Chordify (since 2012, ? loc): Analyses music, e.g. from a YouTube video, and calculates the corresponding guitar chords. Closed software, but supposedly written in Haskell.

There is also an impressive list of industrial users that use Haskell without talking about it much, but that does not work well to show off that you can do any kind of application in Haskell. If I missed anything important, let me know!

The blog Math with Bad Drawings recently featured an article about Ultimate Tic Tac Toe, a variant of Tic Tac Toe where each of the nine fields is a separate game of Tic Tac Toe. To mark a field of the large game as your, you have to win the small game therein. If you chose a field in the small game, this position determines the small field that the other play may play next. See the linked article for a full explanation. As far as I know, the question of who wins this game was open; at least nothing definite was known on Hacker News or on the Board Games StackExchange site. We discussed this a bit in our office, and my coworker Denis Lohner came up with what seems to be a winning strategy. Update: Not surprising, but with these variants of the rule, the winning strategy was already known.
Strategy Assume Denis ( ) plays against me ( ). Like most suggestions for a winning strategy for the first player, Denis (X) starts with the middle:

Now I have to put my in the center field of the center game. No matter where I place it, Denis will send me back ot the middle, until one field of the center game is free. Doing this eight times inevitably puts us in a position like this:

The only way I can influence the game is by chosing which I place last; this determines where Denis goes now. But (and please verify that carefully) it will not matter: The only thing required from that field is that there is a second field that, together with the center, forms a row (or column or diagonal); all fields satisfy that. Assume I placed the top-left last, and Denis has to go there. He will send me to that field:

Now the game of the first field is repeated: Whereever I send him, he will send me back. This works great for all fields but the middle field. The middle field is special: When I send him there, he has the free choice. He will pick the bottom-right game.

• If I send him to the center field before I send him to bottom-right, he will put the in the top-left corner of that field, sending me back. Once I send him to bottom-right then, he will put the in the bottom-right corner of that field.
• If I send him to the center field after I sent him to bottom-right, he will put the in the bottom-right corner of that field.

In any case we will end up in this situation: I won the center game; I likely have a few in the top-left game. To be precise: I have a there if and only if the he has a in the top-left corner of the corresponding game. He also has the diagonal of the lower-right game. For example:

I have to put my mark in the lower-right game now. From now on, whereever I go, he will send me to either to the top-left or bottom-right game. I can do nothing about it (I cannot send him to the diagonal any more, and whereever I send him there is at least one of the top-left or bottom-right fields fee), so he wil easily win all the other games by getting the diagonal. Eventually, he wins the whole game with the bottom row or the right column:

This is not a formal proof yet, but hopefully close enough to convince you, or alternatively allow you to precisely describe how you can prevent losing against Denis strategy.

Having reached the end of my Kickstarter funded year working on the git-annex assistant, I've decided to try one more crowdfunding campaign, to see if I can get funded to work on it a little while longer. I went back and forth on this for a while. The Kickstarter funded development was extremely successful (one of the most productive years of my life). I certianly want to work on git-annex more, and have lots more stuff to do, particularly around security and encryption. On the other hand, it's hard to frame ongoing development as a normal Kickstarter campaign to start something new. Anyway, I've decided to go ahead and try it, and not do it through Kickstarter this time. So I have my own website set up and accepting donations, and hopefully I'll make enough to spend a few more months working on git-annex. campaign.joeyh.name ... And if not, I'll probably spend a few months working on git-annex part time, while looking for paying work with the rest of my time. By the way, I'm taking payments in both US Dollars (via Paypal) and Bitcoin (via Coinbase). Can't wait to see how this works out!

As Americans we might like to believe that the US legal system is intended to protect all of our citizens. Unfortunately, it doesn t protect us all equally, and in fact disproportionately fails to protect the most vulnerable. We re surrounded by instances of injustice related to gender, race and other axes of social privilege, and the machinations of law are not exempt. The state of Florida has recently provided an especially stark example in the application of its self-defense laws in two cases: Marissa Alexander and George Zimmerman. This example is notable because although there were many similarities between the cases, the outcomes were very different. Alexander s case was tried in May 2012 , Zimmerman s in July 2013, both prosecuted by Florida state s attorney Angela Corey. Both cases involved the use of firearms which were legally purchased and carried, and their owners were trained in their use. Both prosecutions cast the defendant as the aggressor, who could have avoided the confrontation. Both of the encounters were with unarmed persons. Both defenses were based on Florida self-defense laws, which include stand your ground laws justifying the use of deadly force without the obligation to retreat. Both shooters admitted to firing a single shot with the intent of defending themselves. Beyond those similarities, each case had its own unique circumstances. The events of Alexander s case took place in her home. Her altercation was with her husband, Rico Gray Sr., who was under a restraining order following a conviction for domestic battery which put Alexander in the hospital. After Gray threatened to kill her, Alexander retrieved a handgun from her car, returned to confront him, firing once. She was arrested and charged the same day. She had had no prior criminal record. A jury deliberated for just 12 minutes before convicting her. A judge sentenced her to 20 years in prison, in accord with mandatory minimums specified by law. Gray, previously sentenced to probation for his earlier conviction, remains free. Zimmerman s shot was fired in his neighborhood, in an altercation with a teenager, Trayvon Martin, who was a guest in the community and walking by himself. The two were not acquainted. Zimmerman called police from his car, claiming that Martin appeared suspicious, and began to follow him. Some of the facts of their encounter remain in dispute, but that Zimmerman fired his gun is not in question. Afterward, Zimmerman was detained by police, questioned and released the same night without being arrested or charged. Following a public outcry, a new investigation was launched and two months later he was arrested and charged. He had been previously arrested and charged with assaulting a police officer, but the charges were later dropped. After 16 hours of deliberation, the jury found Zimmerman not guilty, and he is free today. The most striking difference between the two cases is where each defendant aimed their gun: George Zimmerman shot Trayvon Martin in the chest and killed him, while Marissa Alexander fired at a wall and injured no one. Alexander, a black woman, is in prison for scaring her abusive husband away, while Zimmerman, who killed a young black man, walks free. Alexander and Martin s families have lost a mother and a son. The outcomes for the people involved in these cases could not be more different. Regardless of the merits of the relevant laws themselves, their radically unequal application is deeply troubling. What does this tell us about the relative value of these human lives, as weighed by the judicial system?

The Florida criminal justice system has sent two clear messages today. One is that if women who are victims of domestic violence try to protect themselves, the Stand Your Ground Law will not apply to them. [...] The second message is that if you are black, the system will treat you differently. - U.S. Rep. Corrine Brown

References

• http://en.wikipedia.org/wiki/Shooting_of_Trayvon_Martin
• http://www.thefloridastar.com/new-trial-denied-battered-wife/
• http://www.huffingtonpost.com/2012/05/19/marissa-alexander-gets-20_n_1530035.html
• http://www.huffingtonpost.com/josh-horwitz/protecting-marissa-at-the_b_1551998.html
• http://justiceformarissa.blogspot.com/
• http://www.documentcloud.org/documents/326700-full-transcript-zimmerman.html