Intentionally helping someone poke a hole through their firewall to allow any users on a remote machine access to their computer is generally a bad idea... unless they want you to help them with their computer. In that case, I find it really useful. Toward this end, I've setup a dedicated user account (jamie-share@chavez.mayfirst.org in this example). I granted myself shell access to this user and generated an ssh key pair:

ssh-keygen -t rsa

Then, as a convenience I move my public key into the home directory:

cp ~/.ssh/id_rsa.pub ~/

Next, I give the user I want to help ssh access to this shell account on an Internet connected computer (in this example: jamie-share@chavez.mayfirst.org). Either I share the password with them or if they have an ssh key or monkeysphere identity I use that. Then, I ask them to grant teh jamie-share user access to their user account by running (which downloads the key I just created):

mkdir -p ~/.ssh
ssh jamie-share@chavez.mayfirst.org "cat id_rsa.pub" >> ~/.ssh/authorized_keys

Next, ask them to install openssh-server on their local computer:

sudo apt-get install openssh-server screen

Next, they run the following command on their local computer:

ssh -R 2222:localhost:22 jamie-share@chavez.mayfirst.org

This command says: forward port 2222 on chavez.mayfirst.org to port 22 on your local computer. Lastly, I log in to jamie-share@chavez.mayfirst.org and run, replacing with their local username.

ssh <their-username>localhost -p 2222

You can do whatever you want now, but running screen is a good way to share a session so the person you are working with can see what you are doing as you do it. p.s. Thanks Ross for the tips!

These are some very rough/sketchy notes that I took while doing the exercises on the introductory track of the Ruby language at codeacademy. They were written from the point-of-view of an amateur programmer in C and Python (read: "me"). Since this is my first time with Ruby and I only have about 1 week of experience with it, these notes are likely to contain many mistakes, many simplifications, many omissions or simply gross errors. I would love to be informed of such facts so that I can fix them and not write code that is burdensome for others to maintain. Operators

• Math operators: +, -, *, /, %, ** (just like Python)
• Relational operators: ==, <, >, <=, >= (just like other languages)
• Boolean operators: &&, , ! (just like C/C++/Perl)
• The combined comparison operator ("spaceship operator", like Perl): <=> (follows the spirit of what strcmp in C does)
• Conditional assignment operator ( =) only assigns a value to a variable if that variable is bound to nil.

Numbers

• You can write number literals embedding an underscore (_) for readability purposes.

Basic I/O

• Output: print and puts. Both print strings to stdout, but puts also appends a new line to the end.
• Input: gets accepts an input from stdin and returns a string.
• To convert a string foo to an integer, use the Integer(foo) constructor. It throws an exception if it cannot convert the value.

Strings

• Strings have at least some methods: .length, .reverse, .upcase, .downcase, .capitalize, .chomp (like Perl, removes the trailing endline)
• .upcase seems to have problems with non-ASCII characters, like .
• Interpolation is performed by putting the name name of the variable inside a string and delimited as # name at the place we want the interpolation to occur.
• Methods like .capitalize don't change the string (they return a new string capitalized). To change the string, you call the method .capitalize! (with a ! at the end of the name of the method).
• Other useful methods: .gsub for globally substituting a regular expression (and its variant .gsub! for inplace modification), and .include? to verify if the first string includes another.
• Another method: .split(" "), to split on the string specified as argument.

Control flow If

• The syntax of a if is:

    if expression1
      block1
    elsif expression2
      block2
    else
      block3
    end
  

  note that it ends with the word end, there are no need for parentheses, no braces, no colons, and the keyword for "else if" is elsif (different from Python's elif).
• The "if" can also be started with the keyword unless instead, which checks if the expression is false rather than true (as is the case of if).

While

• The syntax of the while is similar to other languages:

    while condition
      block
    end
  

• A variation of the while is the until, with the same syntax, except for the until in place of the while, and that it iterates when the condition is false, instead of true.

For

• The simple case of iterating over a range of numbers is:

    for i in 1...10
      block
    end
  

  Like ranges in Python, the 10 is not included (the variable i iterates from 1 to 9). For an inclusive range, we use two dots (1..10), instead of three dots (1...10).

Loop

• To iterate a block, we can use the loop keyword, as in:

    loop   block   # with braces
  

  or as in

    loop do # note the pair do/end in place of braces
      block
    end
  

• It is useful to use break when using loop to break out of the infinite iterations.
• Similarly, we can use next as in other languages (the same as continue in C or C++).

Arrays

• The equivalent to Python's lists are arrays and they are declared in the same fashion:

    my_array = [1, 2, 3, 4, 5]
  

• Arrays in Ruby are 0-based.
• Every array has an iterator called the .each method. The basic syntax is:

    object.each    item  block   # do something
  

  which can be written as:

    object.each do  item  block end
  

  where the name between is bound to (a copy of?) each element of object in turn. The equivalent in Python would be:

    for item in object:
        block
  

• Arrays can be sorted with the .sort method (or its sibling .sort!). When passing a parameter, it is useful to use the spaceship operator (<=>) (like Perl) to specify a custom order (similar in spirit to what we do with Python's key argument to the sort method of a list). Example:

     my_array.sort!  a, b  a <=> b  # Ascending order
     my_array.sort!  a, b  b <=> a  # Descending order
  

• The .push method appends an element to an Array (similar to Python's .append).
• The empty Array is denoted by [], like an empty list in Python.
• An iterator (like (1..10)) can be converted to an array with the .to_a method.

Hashes

• A literal hash is defined by the syntax:

    my_hash =  
        "key1" => "abc",
        "key2" => 2,
        "key3" => false
     
  

  Quite similar to the version in Python:

    my_hash =  
        "key1": "abc",
        "key2": 2,
        "key3": False
     
  

  In fact, in Ruby 1.9, you can write, instead:

    my_hash =  
        key1: "abc",
        key2: 2,
        key3: false
     
  

  which is very similar to Python or JSON.
• The empty hash, OTOH, is defined by calling the constructor .new of the class Hash:

    my_hash = Hash.new
  

  The syntax used by Python can also be used: my_hash =
• Iterating over a hash is done the same way as done for an array, but the .each method gives both the key and the value, as in:

    my_hash.each do  k, v 
      puts "# k : # v "
    end
  

• To iterate only on keys, we use the method .each_key. To iterate only on values, we use the method .each_value.
• A hash can be sorted with the .sort_by method (to sort by keys, values or any expression involving them) and since it can be ordered, we can reverse (in-place) the order of its elements with the .reverse! method. This makes Ruby's hashes to "taste" like Python's OrderedDict's.
• When one initializes a hash, one can specify a default value to the .new method (e.g., foo = Hash.new(0)), which makes it convenient to do the equivalent of Python's defaultdict or Counter.
• You can select part of a hash by using the .select method and giving it a block. For example:

    good_movies = movie_ratings.select k, v  v > 3 
  

Functions/methods

• Functions in Ruby are defined with the word def (as in Python) and end with the word end, as other composite constructs. Example:

    def square(n)
      return n*n
    end
  

• Parentheses are optional in Ruby, unless they create ambiguity. So, to call the function above, one can use any of the following:

    square(2)
    square 2
  

• A function can take a block as a parameter (and this is what .each does, for instance).
• Functions can take a variable number of elements with the splash operator (denoted by *), which puts the extra parameters in an Array (like Python).
• Functions can have default parameters, which are set in the "prototype", just like in Python or C++.
• If the return keyword is omitted from the method, Ruby uses an implicit return, where the value of the last executed expression is returned to the caller of the method.

Symbols

• Any instances of a given symbol share the same ID. This is due to the fact that they are immutable.
• Symbols start with a colon (:) and they can contain any character that is valid for a variable name.
• It is possible to convert between a symbol and a string with the methods .to_s and .to_sym.

Comments

• Single line comments use the hash symbol (#).
• Ruby also has multiline comments, which are delimited by a line beginning with =begin and a line beginning with =end.

Lambda functions

• A lambda function in Ruby is defined, as expected, as, for example:

    foo = lambda    x  x.to_sym  
  

• The basic differences between a lambda and a proc is that:
  • A lambda checks the number of elements that it receives, while a proc doesn't and assigns nil to parameters that it didn't receive (a proc seems to ignore extra parameter that it wasn't expecting).
  • When a lambda returns, the control goes back to the function that called the lambda, while when a proc returns, the control is returned also from the function that called the proc.

Classes

• "Everything is an object in Ruby." (At least one exception are blocks).
• To test if an object is of a given class, you use the method .is_a? on the object and the type as argument (e.g., foo.is_a? Integer)
• The "null" value in Ruby is nil. It is equivalent to None in Python.
• The only "falsy" values in Ruby are false and nil. The others are "truthy", differently from C/C++/Perl/Python.
• For a given object, the method .object_id returns what is the "address" of that object.
• Classes in Ruby are created with the syntax

    class ClassExample
        # body
    end
  

  (That is for a class that has is not derived from other regular classes).
• Constructors in Ruby are called initialize. An instance of a class is created by calling the method .new on the class name (and passing parameters as needed for the initialize method).
• Instance variables are named with 1 at sign (@) as a prefix. Class variables are named with 2 at signs (@@) as a prefix. Global variables are named with a dollar sign ($) as a prefix.
• Class methods are declared inside a class prefixed with self. and they are called with the class name suffixed with the name of the method. For example, for a class MyClass, with a class method class_method, we call it as MyClass.class_method.
• When overriding a method in a subclass, one can call the method from the parent class with the keyword super, as in super(arg1, arg2).
• Ruby doesn't support multiple inheritance. In place of multiple inheritance, Ruby offers mixins.
• To access one attribute, say @my_attrib, in Ruby classes, one usually creates two methods to manipulate it: a method with the name my_attrib, that returns the value of @my_attrib (a getter) and a method with the name my_attrib= that accepts a parameter to set @my_attrib to (a setter). There is syntatic sugar for creating both getters or setters. For a getter, you declare attr_reader :my_attrib (the member name as a symbol). For a setter, you declare attr_writer :my_attrib. If you need both a getter and a setter, then you can abbreviate them with attr_accessor.
• All methods, by default, are public in Ruby. To explicitly mark something as public, you use the keyword public. To mark something as private, you use the keyword private.

Modules

• You can import a module with the keyword require (as in Perl) and the name of the module in quotes.
• One example of a module is the benchmark module, which provides the Benchmark class, which has the method .realtime, which takes a block and reports how much time it used.
• To create a module, use a syntax similar to that of a class, but substituting the keyword class with module.
• The scope resolution operator in Ruby is the double colon (::), like in Perl.
• To create a method inside a module, you have to prefix it by the name of the module with a dot or with self., if you want to create a method to be called outside the module. The same also holds for creating class methods (as opposed to instance methods): they can be simply named self.foo to get a class method called foo.
• While some modules are already "preloaded" by the interpreter (e.g., Math), others need to be explicitly loaded. We do this by using require 'foo' to load the module (in the file foo?).

Mixins

• A mixin is the use of a module to "enrich" a class, adding "instance" methods of the module to the class. It is, in a sense, a way to have some of multiple inheritance in Ruby (without its headaches?)
• Using the word include Foo imports the items of the module Foo in the current namespace for instances of a class.
• Using the word extend Foo imports the items of the module Foo in the current namespace as class methods/attributes.

Conventions

• Local variables are named in lower case, with parts separated by underscores, as in my_name.
• Methods that return boolean values have a question mark (?) at the end of their name.
• Methods that have destructive side-effects (like changing their input) have an exclamation mark (!) at the end of their name.
• Code is indented by 2 spaces.
• Like other languages, constants are written in all caps, with words separated by underscores. For example, MAX_VALUE.
• Global variables are prefixed with a dollar sign (e.g., $Foo).
• Classes are named in camel case (no initial lowercase).
• The "poetic mode" in Ruby recommends one to omit the parentheses on function calls (whenever unambiguous) and, in essence, to write as little "bureaucracy" (like the explicit return keyword in methods).

I'm sure you all will be a bit disappointed (I know I am) that there are no ascii screenshots in this weeks report. But let me make it up to you by telling you a story. I was at the FOSDEM in the year 2012 and I went to a nice workshop on sunday afternoon in the Virtualization Devroom. Renzo Davoli was the host of this workshop and he started with a little introductory talk to get everyone to agree on a common terminology first. He began by asking the question, what virtualization meant in the most general way. He defined the ability to virtualize a resource as the ability to freely and transparently control how someone (say a process) is interacting with said resource. So for example if you LD_PRELOAD a library to impersonate fopen(3) and friends, you have virtualized the filesystem (for some small values of virtualized). He then went on to discuss various methods of virtualization that are commonly available on Linux, not only full system virtualization solutions, but also all kinds of methods allowing a more fine-grained control over what resources are virtualized. Of course every method had its strengths and its weaknesses. Having seen Samuels talk about all the awesome things Hurd can do for virtualization I approached him with a question in the free discussion part of his workshop. I asked whether he would agree that once a resource is moved from the kernel to the userspace, the problem of virtualizing that resource is almost trivially solved, and he agreed. So I said that this was awesome, because then the trivial and elegant solution for all his virtualization (and tracing) needs are micro kernel operating systems, and he also agreed to that but (as far as I can recall) he mentioned that there is none that would meet his needs (I believe he is a computer science professor and heavily relies on virtualization techniques for his lab and for teaching purposes). And even though a microkernel operating systems has a cost (message passing instead of function calls for example) it also has its merits such as scaling better from a development point of view. Also, many cool features can emerge just from the design. For example think about the container support in Linux and how painfully long it took to make all the resources namespace aware (with one of the most critical, the user namespace being the most difficult one). On Hurd you get the same functionality for free. If you are curious, read Samuels awesome slides. So what have I done this week?

• procfs should ignore --no dev,exec,suid : http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00076.html
• libnetfs does not respond properly to file_get_translator requests on nodes with a passive translator record: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00148.html
• As the mtab translator is a translator of its own, there needs to be a way to bind it to /proc/mounts. I figured the easiest way would be to just serve a node with an appropriate passive translator record: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00149.html
• There was no clean way to start the console-client. I added daemonizing support to the console-client and created an appropriate initscript: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00165.html
• I also worked a lot on my mtab translator prototype: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00183.html

Next week I'm going to work on the two remaining sysvinit related issues. These are the only ones preventing me to come up with a clean patch series against the sysvinit package and I figured that it would be nice to propose such a series rather sooner than later so that I will have plenty of time to discuss any issues with the sysvinit maintainers. See you next week :)

Yesterday, I've spent some time trying to improve speed of translation import in Weblate. It got annoyingly slow while adding new features and I felt there has to be some area for improvements even with all the features. I was testing on development server with debug stuff enabled (what slows down things quite a lot already) and first import took about 30 seconds. Quick look revealed that for linked repositories, the link lookup happens too often, so some caching could be used. This has saved about 3 seconds... With offloaded indexing (what is anyway only useful setup for production use), every reindex request was checking whether there is already one before creating new one. In most cases there is none before, so it's simply faster to try to create one and ignore error in case it is already there. We're now 5 seconds faster. Another big resource consumer are quality checks. They provide useful feedback to translators, but it means that quite a lot of code needs to be executed for every single string. First thing I've realized is that unit check state is recalculated everytime regardless the fact whether there was any change, what was easy to fix. The other speed issue is in consistency check - it does too complex SQL query, which usually does not match anything. Converting this to much simpler one (which will in most cases still have no results) improves performance for usual path while having minimal impact for the real checking. Another 4 seconds down on my test import. Last thing which I changed is way how per translation statistics are updated. In past this required quite a lot of queries, but yesterday I've realized this can be much reduced. During test suite run on PostgreSQL, I saw that it's not that easy - PostgreSQL does not allow to sum boolean fields, but fortunately this can be fixed as well. This has saved about 2 seconds. In the end the tested import took 16 seconds instead of original 30 and issued 2500 SQL queries instead of 5000 at the beginning. I hope you will enjoy regained speed of imports :-).

Filed under: English Weblate 0 comments Flattr this!

There are three main ways to distribute a command-line daemon for OS X:

1. Distributing source code and instructions on how to compile it.
2. Using a third-party package manager, like Homebrew.
3. Providing an installer package.

Homebrew Homebrew is a popular package management system. It works like the BSDs ports collections by downloading, compiling and installing the requested software, while also installing any required dependencies automatically. Creating a new package is quite easy and there are a lot of examples available. However, there are some limitations:

• You don t really build a package but execute a recipe to locally install the software.
• You need to install development tools, either a whole Xcode installation¹ or the command line version.
• If you need to execute some steps as root, you will have to explain them to the user for her to execute by hand. This includes the creation of a system user or the installation of a daemon through launchd.

If you can t live with those limitations, you may need to build an installer package.

Building an installer package OS X comes with a graphical and a command-line installer. The graphical one is run by opening a package from the Finder. Then, the user experiences some familiar wizard allowing her to install the software in a matter of a few seconds. The documentation around how to build such a package is somewhat clumsy. You may find outdated information or pieces of information that only apply to projects using Xcode. I will try to provide here accurate bits in the following context:

1. You are packaging a pure command-line tool.
2. You are using Autoconf and Automake as a build system.
3. You want to support several architectures.
4. You want to support older versions of OS X.

Creating a package Building such a package was previously done with some graphical tool called PackageMaker. This tool is not available any more and developers are asked to switch to pkgbuild and productbuild. There is a very neat Stackoverflow article on how those tools work. A package is built in two steps:

1. Build component packages.
2. Combine them into a product archive.

A component package contains a set of files and a set of scripts to execute at various steps of the installation. You can have several component packages, for example a package for the daemon and a package for the client. They are built with pkgbuild. A product archive contains the previously created component packages as well as a file describing various options of the installer (required and optional components, license, welcome text, ). To create a component package, we need to install the needed files in some directory:

$ ./configure --prefix=/usr --sysconfdir=/private/etc
$ make
$ make install DESTDIR=$PWD/osx-pkg

We will now put the content of osx-pkg into a component package with pkgbuild:

$ mkdir pkg1
$ pkgbuild --root osx-pkg \
>    --identifier org.someid.daemon \
>    --version 0.47 \
>    --ownership recommended
>    pkg1/output.pkg
pkgbuild: Inferring bundle components from contents of osx-pkg
pkgbuild: Wrote package to output.pkg

You need to be careful with the identifier. It needs to be unique and identify your software as well as the specific component. Then, you need to create an XML file describing the installer. Let s call it distribution.xml:

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<installer-gui-script minSpecVersion="1">
    <title>Some daemon</title>
    <organization>org.someid</organization>
    <domains enable_localSystem="true"/>
    <options customize="never" require-scripts="true" rootVolumeOnly="true" />
    <!-- Define documents displayed at various steps -->
    <welcome    file="welcome.html"    mime-type="text/html" />
    <license    file="license.html"    mime-type="text/html" />
    <conclusion file="conclusion.html" mime-type="text/html" />
    <!-- List all component packages -->
    <pkg-ref id="org.someid.daemon"
             version="0"
             auth="root">output.pkg</pkg-ref>
    <!-- List them again here. They can now be organized
         as a hierarchy if you want. -->
    <choices-outline>
        <line choice="org.someid.daemon"/>
    </choices-outline>
    <!-- Define each choice above -->
    <choice
        id="org.someid.daemon"
        visible="false"
        title="some daemon"
        description="The daemon"
        start_selected="true">
      <pkg-ref id="org.someid.daemon"/>
    </choice>
</installer-gui-script>

Fortunately, this file is documented on Apple Developer Library. Since we only describe one package, we pass customize="never" as an option to skip the choice part. You can however remove this attribute when you have several component packages. The attribute rootVolumeOnly="true" explains that this daemon can only be installed system-wide. It is marked as deprecated but still works. Its replacement (domains tag) displays an unusual and buggy pane which Apple doesn t use in any of its packages. You need to put the HTML² documents into a resources directory. It is also possible to choose the background by specifying a <background/> tag. The following command will generate the product archive:

$ productbuild --distribution distribution.xml \
>              --resources resources \
>              --package-path pkg1 \
>              --version 0.47 \
>              ../final.pkg
productbuild: Wrote product to ../final.pkg

Scripts The installer can execute some scripts during installation. For example, suppose we would like to register our daemon with launchd for it to be run at system start. You first need to write some org.someid.plist file and ensure it will get installed in /Library/LaunchDaemons:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>org.someid</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/sbin/mydaemon</string>
    <string>-d</string>
  </array>
  <key>RunAtLoad</key><true/>
  <key>KeepAlive</key><true/>
</dict>
</plist>

Then create a scripts/postinstall file with the following content:

#!/bin/sh
set -e
/bin/launchctl load "/Library/LaunchDaemons/org.someid.plist"

You also need a scripts/preinstall file which will allow your program to be smoothly upgraded:

#!/bin/bash
set -e
if /bin/launchctl list "org.someid" &> /dev/null; then
    /bin/launchctl unload "/Library/LaunchDaemons/org.someid.plist"
fi

Ensure that those scripts are executable and add --scripts scripts to pkgbuild invocation. If you need to add a system user, use dscl.

Dependencies There is no dependency management built into OS X installer. Therefore, you should ensure that everything is present in the package or in the base system. For example, lldpd relies on libevent, an event notification library. This library is not provided with OS X. When it is not found, the build system will use an embedded copy. However, if you have installed libevent with Homebrew, you will get binaries linked to this local libevent installation. Your package won t work on another host. The output of otool -L can detect unwanted dependencies:

$ otool -L build/usr/sbin/lldpd
build/usr/sbin/lldpd:
        /usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
        /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 744.19.0)
        /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 945.18.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

Compilation for older versions of OS X The package as built above may only work on the same version of OS X that you use to compile. If you want a package that will work on OS X 10.6, you will need to download the appropriate SDK³. Then, you need to tell the compiler the target version of OS X and the SDK location. You can do this by setting CFLAGS and LDFLAGS (and CXXFLAGS if you use C++):

$ SDK=/Developer/SDKs/MacOSX10.6.sdk
$ ./configure --prefix=/usr --sysconfdir=/private/etc \
>         CFLAGS="-mmacosx-version-min=10.6 -isysroot $SDK" \
>         LDFLAGS="-mmacosx-version-min=10.6 -isysroot $SDK"

The -mmacosx-version-min flag will be used by various macros in the headers to mark some functions as available or not, depending on the version of OS X you target. It will also be used by the linker. The -isysroot will tell the location of the SDK. Headers and libraries will be looked up in this location first.

Universal binaries Mac OS X 10.6 was available for both IA-32 and x86-64 architectures. If you want to support both installations with a single package, you need to build universal binaries. The Mach object file format, used by OS X, allows several versions of the executable in the same file. The operating system will select the most appropriate one. An easy way to generate such files is to pass -arch i386 -arch x86_64 to the compiler:

$ ./configure --prefix=/usr --sysconfdir=/private/etc \
>     CC="gcc -arch i386 -arch x86_64" \
>     CPP="gcc -E"

However, this is a dangerous option. Suppose that your ./configure script tries to determine some architecture-dependent parameter, like the size of an integer (with AC_CHECK_SIZEOF); it will compute it for the host architecture. The binary generated for the other architecture will therefore use a wrong value and may crash at some point. The correct way to generate an universal binary is to execute two separate compilations and to build the universal binary with lipo:

$ for arch in i386 x86_64; do
>   mkdir $arch ; cd $arch
>   ../configure --prefix=/usr --sysconfdir=/private/etc \
>       CC="gcc -arch $ARCH" \
>       CPP="gcc -E"
>   make
>   make install DESTDIR=$PWD/../target-$ARCH
>   cd ..
> done
[...]
$ lipo -create -output daemon i386/usr/sbin/daemon x86_64/usr/sbin/daemon

Since lipo only works on file, I have written a Python script applying lipo recursively to several directories.

Putting everything together Now, we need to automate a bit. You could provide some nifty script. I propose an appropriate Makefile.am instead. This way, we can use the output variables, like @VERSION@, from ./configure to generate some of the files, like distribution.xml. You need to add that in your configure.ac:

AC_CONFIG_FILES([osx/Makefile osx/distribution.xml osx/im.bernat.lldpd.plist])
AC_CONFIG_FILES([osx/scripts/preinstall],  [chmod +x osx/scripts/preinstall])
AC_CONFIG_FILES([osx/scripts/postinstall], [chmod +x osx/scripts/postinstall])
AC_SUBST([CONFIGURE_ARGS], [$ac_configure_args])

Let s have a look at osx/Makefile.am. First, we define some variables:

PKG_NAME=@PACKAGE@-@VERSION@.pkg
PKG_TITLE=@PACKAGE@ @VERSION@
PKG_DIR=@PACKAGE@-@VERSION@
ARCHS=@host_cpu@

If we want to build for several architectures, we will type make ARCHS="x86_64 i386". Otherwise it defaults to the current host s architecture. We use install-data-local target to install files specific to OS X:

install-data-local:
    install -m 0755 -d $(DESTDIR)/Library/LaunchDaemons
    install -m 0644 im.bernat.@PACKAGE@.plist $(DESTDIR)/Library/LaunchDaemons
uninstall-local:
    rm -f $(DESTDIR)/Library/LaunchDaemons/im.bernat.@PACKAGE@.plist

The main target is the product archive, built with productbuild:

../$(PKG_NAME): pkg.1/$(PKG_NAME) distribution.xml resources
    $(PRODUCTBUILD) \
        --distribution distribution.xml \
        --resources resources \
        --package-path pkg.1 \
        --version @VERSION@ \
        $@

Its main dependency is the component package:

pkg.1/$(PKG_NAME): $(PKG_DIR) scripts
    [ -d pkg.1 ]   mkdir pkg.1
    $(PKGBUILD) \
        --root $(PKG_DIR) \
        --identifier im.bernat.@PACKAGE@.daemon \
        --version @VERSION@ \
        --ownership recommended \
        --scripts scripts \
        $@

Now, we need to build $(PKG_DIR):

$(PKG_DIR): stamp-$(PKG_DIR)
stamp-$(PKG_DIR): $(ARCHS:%=%/$(PKG_DIR))
    $(srcdir)/lipo $(PKG_DIR) $^
    touch $@

The $(ARCHS:%=%/$(PKG_DIR)) will expand to x86_64/$(PKG_DIR) i386/$(PKG_DIR). This is a substitution reference⁴. Before applying our lipo script, we need to be able to build the architecture-dependent package directories:

pkg_curarch = $(@:stamp-%=%)
$(ARCHS:%=%/$(PKG_DIR)): %/$(PKG_DIR): stamp-%
$(ARCHS:%=stamp-%): stamp-%: im.bernat.lldpd.plist
    [ -d $(pkg_curarch) ]   mkdir -p $(pkg_curarch)
    (cd $(pkg_curarch) && \
        $(abs_top_srcdir)/configure @CONFIGURE_ARGS@ \
            CC="@CC@ -arch $(pkg_curarch)" \
            CPP="@CPP@")
    (cd $(pkg_curarch) && \
        $(MAKE) install DESTDIR=$(abs_builddir)/$(pkg_curarch)/$(PKG_DIR))
    touch $@

This may seem a bit obscure, but this is really like what we have described previously. Note that we use @CONFIGURE_ARGS@ which is a variable we have defined in configure.ac. Here is how a user would create its own OS X package:

$ SDK=/Developer/SDKs/MacOSX10.6.sdk
$ mkdir build && cd build
$ ../configure --prefix=/usr --sysconfdir=/private/etc --with-embedded-libevent \
   CFLAGS="-mmacosx-version-min=10.6 -isysroot $SDK" \
   LDFLAGS="-mmacosx-version-min=10.6 -isysroot $SDK"
[...]
$ make -C osx pkg ARCHS="i386 x86_64"
[...]
productbuild: Wrote product to ../lldpd-0.7.5-21-g5b90c4f-dirty.pkg
The package has been built in ../lldpd-0.7.5-21-g5b90c4f-dirty.pkg.

---

1. Even if Xcode is free, you will need to register to Apple and show a valid credit card number. That s quite an interesting requirement.
2. It is possible to use other formats (like RTF or just plain text) for documents but HTML seems the best choice if you want some formatting without fiddling with RTF.
3. Unfortunately, Apple makes it difficult to find them. The easiest way to get the SDK for Mac OS X 10.6 is to download Xcode 4.3.3, mount the image and copy the SDK to /Developer/SDKs.
4. This is equivalent to the use of patsubst function. However, this function is specific to GNU make. The substitution reference pattern was accepted recently in POSIX and is coined as pattern macro expansion.

this week(end) I was more in the "RC bug fix flow" than in the last weeks, & here are the results:

• #665518 src:clipit: "clipit: Including individual glib headers no longer supported"
  add patch from Michael Biebl, upload to DELAYED/2 (NMU originally prepared by Sebastian Ramacher)
• #701406 src:nagios-plugins: "nagios-plugins: ftbfs with eglibc-2.17"
  add patch from Ubuntu / Robie Basak, upload to DELAYED/2, then cancelled on maintainer's request, who uploaded the fixed package himself
• #707393 src:gorm.app: "gorm.app: FTBFS: GormCore/./obj/libGormCore.so: undefined reference to class_pose_as'"
  add patch from Ubuntu / Julian Taylor, upload to DELAYED/2
• #710768 ganglia-webfrontend: "ganglia-webfrontend: please update libgd dependencies"
  drop dependency on libgd2*, upload to DELAYED/2
• #711800 src:python-poppler: "python-poppler: FTBFS: /usr/bin/make: unrecognized option '--prefix --exec-prefix --includes --libs --cflags --ldflags --extension-suffix --help --configdir'"
  apply changes from Ubuntu / Matthias Klose, upload to DELAYED/2
• #713204 src:voms-api-java: "voms-api-java: FTBFS: cp: cannot stat 'target/site/javadoc/apidocs': No such file or directory"
  upload NMU with BouncyCastle patch prepared by Emmanuel Bourg to DELAYED/2
• #713360 src:regina-normal: "regina-normal: FTBFS: CMake Error: The following variables are used in this project, but they are set to NOTFOUND. PYTHON_LIBRARY"
  apply improved patch from Peter Green, upload to DELAYED/2
• #713450 src:oregano: "oregano: FTBFS: ld: sheet/libsheet.a(grid.o): undefined reference to symbol 'floor@@GLIBC_2.2.5'"
  add patch from Petter Reinholdtsen, upload to DELAYED/2
• #713627 src:rox: "rox: FTBFS: ld: support.o: undefined reference to symbol 'floor@@GLIBC_2.2.5'"
  add patch to link against math lib, upload to DELAYED/2
• #713649 src:gdis: "gdis: FTBFS: gdis.h:269:18: fatal error: glib.h: No such file or directory"
  propose a patch

The Present Extension I ve finally gotten an initial implementation of the Present extension written and running and thought I should write up the current design and status. Present Design The current Present extension consists of two requests:

1. PresentRegion. Puts new bits from a pixmap in a window.
2. PresentSelectInput. Asks for Present events to be delivered.

PresentRegion This request takes a pile of arguments:

 
    PresentRegion
    window: WINDOW
    pixmap: PIXMAP
    valid-area: REGION or None
    update-area: REGION or None
    x-off, y-off: INT16
    target-msc: CARD64
    divisor: CARD64
    remainder: CARD64
    idle-fence: FENCE
 
Errors: Drawable, Pixmap, Match

Provides new content for the specified window, to be made visible at the specified time (defined by target-msc , divisor and remainder ). update-area defines the subset of the window to be updated, or None if the whole window is to be updated. valid-area defines the portion of pixmap which contains valid window contents, or None if the pixmap contains valid contents for the whole window. PresentRegion may use any region of pixmap which contains update-area and which is contained by valid-area . In other words, areas inside update-area will be presented from pixmap , areas outside valid-area will not be presented from pixmap and areas inside valid-area but outside update-area may or may not be presented at the discretion of the X server. x-off and y-off define the location in the window where the 0,0 location of the pixmap will be presented. valid-area and update-area are relative to the pixmap. If target-msc is greater than the current msc for window , the presentation will occur at (or after) the target-msc field. Otherwise, the presentation will occur after the next field where msc % divisor == remainder . If divisor is zero, then the presentation will occur after the current field. idle-fence is triggered when pixmap is no longer in use. This may be at any time following the PresentRegion request, the contents may be immediately copied to another buffer, copied just in time for the vblank interrupt or the pixmap may be used directly for display, in which case it will be busy until some future PresentRegion operation. If window is destroyed before the presentation occurs, then the presentation action will not be completed. PresentRegion holds a reference to pixmap until the presentation occurs, so pixmap may be immediately freed after the request executes, even if that is before the presentation occurs. If idle-fence is destroyed before the presentation occurs, then idle-fence will not be signaled but the presentation will occur normally. PresentSelectInput

 
    PresentSelectInput
    event-id: PRESENTEVENTID
    window: WINDOW
    eventMask: SETofPRESENTEVENT
 
Errors: Window, Value, Match, IDchoice, Access

Selects the set of Present events to be delivered for the specified window and event context. PresentSelectInput can create, modify or delete event contexts. An event context is associated with a specific window; using an existing event context with a different window generates a Match error. If eventContext specifies an existing event context, then if eventMask is empty, PresentSelectInput deletes the specified context, otherwise the specified event context is changed to select a different set of events. If eventContext is an unused XID, then if eventMask is empty no operation is performed. Otherwise, a new event context is created selecting the specified events. Present Extension Events There are three different events for the Present extension:

1. PresentConfigureNotify
2. PresentCompleteNotify
3. PresentRedirectNotify

PresentConfigureNotify This event is moving from the DRI3 extension where it doesn t belong.

 
    PresentConfigureNotify
    type: CARD8         XGE event type (35)
    extension: CARD8        Present extension request number
    length: CARD16          2
    evtype: CARD16          Present_ConfigureNotify
    eventID: PRESENTEVENTID
    window: WINDOW
    x: INT16
    y: INT16
    width: CARD16
    height: CARD16
    off_x: INT16
    off_y: INT16
    pixmap_width: CARD16
    pixmap_height: CARD16
    pixmap_flags: CARD32
 

PresentConfigureNotify events are sent when the window configuration changes if PresentSelectInput has requested it. PresentConfigureNotify events are XGE events and so do not have a unique event type. x and y are the parent-relative location of window . PresentCompleteNotify

 
    PresentCompleteNotify
    type: CARD8         XGE event type (35)
    extension: CARD8        Present extension request number
    length: CARD16          2
    evtype: CARD16          Present_CompleteNotify
    eventID: PRESENTEVENTID
    window: WINDOW
    ust: CARD64
    msc: CARD64
    sbc: CARD64
 

Notify events are delivered when a PresentRegion operation has completed and the specified contents are being displayed. sbc, msc and ust indicate the swap count, frame count and system time of the related PresentRegion request. PresentRedirectNotify This one is not specified yet, but the intent is for it to contain sufficient information for the compositing manager to be able to construct a suitable screen update that includes an application window update. Finishing GLXOMLsync_control At this point, Mesa only exposes the old swapinterval configuration value, it doesn t provide any of the GLXOMLsynccontrol APIs. However, the Present protocol does have the bits necessary to support glXSwapBuffersMscOML in the PresentRegion request. Let s see how the remaining APIs in this GL extension will be supported. glXGetSyncValuesOML This one is easy; it only needs to return the UST/MSC/SBC values from the most recent SwapBuffers request. Those are returned in the PresentCompleteNotify event, so Mesa just needs to capture that event and save the values away for return to the application. We don t need any new Present protocol for this. glXGetMscRateOML This returns the refresh rate of the monitor associated with the specified drawable. RandR exposes all of the necessary data for each monitor, but the monitor each window is going to be synchronized against isn t exposed anywhere, and is effectively implementation-dependent. So, I think the easy thing to do here is to have a Present request which reports which RandR output a window will be synchronized with. glXSwapBuffersMscOML This is the one API which is already directly supported by the Present extension. glXWaitForMscOML This is effectively the same as glXSwapBuffersMscOML, except that it doesn t actually perform a swap. For this, I think we want a new request that generates an X event when the target values are reached, and then have the client block until that event is received. This will avoid blocking other threads using the same X connection. glXWaitForSbcOML Like glXWaitForMscOML, this just needs to trigger an event to be delivered at the right time. Presentation Redirection I m focusing on finishing the above stuff before I start writing the redirection spec and code, but I ve been thinking a bit about it. What we want is for applications that are presenting a new frame within a redirected window to have that presentation delivered directly to the compositing manager instead of having the bits copied to the redirected buffer and have the compositing manager only learn about this when the damage event is received from copying the bits. Once the compositing manager receives notification that there are new bits available for a portion of a window, it can then get those bits onto the screen in one of two ways:

1. Copy them to the back buffer for the whole screen and then present that back buffer to the screen.
2. Present them directly to the screen, bypassing the compositing manager s back buffer entirely.

The first option is what you d do if there were more updates than just a single window; it would update the whole screen at the same time. The second option is what you d do if there was only the one window update to act on, and this would automatically take advantage of page flipping to avoid copying data at all. No need to un-redirect windows for the page flip to work. In both cases, we need to inform the original application both when its pixmap is idle and when the swap actually hits the screen. And we need to make sure the final swap happens when the application requested. For the pixmap idle notification, the current PresentRegion idlefence argument should suffice; we just need to pass the idlefence XID along in the redirection event. Simple. For completion notification, we need to make sure the SBC value for the original window gets incremented and that a PresentCompleteNotify event is delivered. I think that means we want to append a chunk of data to the PresentRegion request so that suitable events can be delivered when the compositing manager PresentRegion occur. I think that just needs to be the window of that original window. To make sure the swap happens at the right time, we just need to have the targetmsc value provided to the compositing manager. I m hoping that having this all event driven is that when a later PresentRegion is redirected that has an *earlier* targetmsc, we can simply queue that as well and things should just work . Current Status I ve got the protocol definitions (both X server and XCB) done, and libxcb supporting the extension. I ve got the X server bits working, but the actual updates are not synchronized to the monitor; they re using OS timers and CopyArea for now, mostly so I can test things without also hacking drivers. Mesa is using the extension, but it only provides the swapinterval value and is not yet supporting the full GLXOMLsynccontrol extension. I ve also got a simple 2D core X application using the extension which is in the shmfd repository in my home directory on freedesktop.org. Availability As always, these bits are already published in my home directory on freedesktop.org in various git repositories.

Dear all, we are excited that the first Shogun workshop, July 12-14 in Berlin, is getting closer. Thanks to all the people that signed up -- we are sure it will be a packed and inspiring weekend! We have finalized the schedule for Friday, July 12, taking place at the C-Base (see description below [1]). After an intro where everyone gets to know each other and where we introduce ourselves, Shogun, and Machine Learning in general, there will be some tutorials by Shogun developers. In addition, we will have discussions about various topics and various coffee breaks and lunch. Finally, we will enjoy a summer's evening in Berlin. On Saturday and Sunday, July 13-14 there will be hands on sessions at the Technical University Berlin [2], where developers are around for more close up discussions and practical guidance. Bring your Laptop if you want to try things. See the final schedule for more details [1]. We plan to do video recordings of all lectures and will have a live stream [3]. See you there! The Shogun-Team

• [0] c-base, Rungestra e 20, 10179 Berlin, Germany
• [1] Workshop Program
• [2] TU-Berlin Room MAR 0.001
• [3] Live Stream

After setting an automatic code coverage tool, I just plugged an automatic scan-build on the LLVM toolchain trunk: http://buildd-clang.debian.net/scan-build/ Note that:

• polly is currently not analyzed. I had to disable the build until the new version of cloog is available.
• compiler-rt is not analyzed because it does not respect the CC/CXX argument to use the clang built locally [1]. Not sure it is a bug here.

This work is done through the Debian/LLVM jenkins instance.
The report is updated twice a day.
The scan-build/clang used to produce the report is the one published on http://llvm.org/apt/. That means that a new feature/fix done on scan-build will appear only about a day after on report. As a side effect, it tests automatically the packages distributed on llvm.org/apt/. For a project of this size, the number of reported errors / warnings is pretty low. At time of writing, only 329 errors are reported and many of them are in generated code (AttrDump.inc and LLDBWrapPython.cpp for example) or in unit tests. [1]
/tmp/buildd/llvm-toolchain-snapshot-3.4~svn185728/build-llvm/Release/bin/clang -fno-exceptions -fPIC -funwind-tables -I/tmp/buildd/llvm-toolchain-snapshot-3.4~svn185728/projects/compiler-rt/lib -I/tmp/buildd/llvm-toolchain-snapshot-3.4~svn185728/projects/compiler-rt/include -Wall -Werror -O3 -fomit-frame-pointer -m64 -fPIE -fno-builtin -gline-tables-only -fno-rtti -DASAN_FLEXIBLE_MAPPING_AND_OFFSET=1 -c -o /tmp/buildd/llvm-toolchain-snapshot-3.4~svn185728/build-llvm/tools/clang/runtime/compiler-rt/clang_linux/asan-x86_64/x86_64/SubDir.lib__sanitizer_common/sanitizer_symbolizer_itanium.o /tmp/buildd/llvm-toolchain-snapshot-3.4~svn185728/projects/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_itanium.cc

Having migrated my websites away from my ssh/mail box I'm going to upgrade that this weekend. I've got my new mail client, lumail, working well enough to use exclusively, so the upgrade should be nice and simple. I spent a few hours last night removing packages from my ssh/mail box to trim it down. Removing a bunch of Perl modules I used in my CGI coding, removing services such as nfs-common, portmapper, etc. Today I'll have a stab at the upgrade. The only thing I have to be careful of is my backported/tweaked qpsmptd packages. I'll try the native wheezy version now it has caught up. (I don't use any of the standard qpsmtpd plugins at all. Instead I have a separate tree of my own custom anti-spam and virtual-hosting aware plugins. They all work in a unified fashion. Using these plugins against a new version of qpsmtpd should be just fine. But obviously I need to test that.) Work on Lumail is probably going to slow down now it is genuinely in use, but I'll keep an eye out for feature requests and missing primitives. Annoyingly I wasted 30 minutes just now implementing a plugin I'd already written: lumail issue #51. I also need to step-back this weekend and reassess my hosting. When I was tweaking my slaughter setup I recently realized I have more hosts than I thought:

• Cluster running for Debian-Administration.org:
  • 4 x web nodes
  • 2 x database nodes
  • 2 x "planet" nodes
  • 1 x "misc" node.
• Personal stuff
  • 1 x ssh/mail host. (ssh.steve.org.uk/mail.steve.org.uk)
  • 1 x web host. (www.steve.org.uk, www.lumail.org, etc, etc.)
  • 1 x blogspam.net server (www.blogspam.net)
  • 1 x builder node - Runs buildd for producing my packages.

Total: 13 virtual machines. (+ one kvm-host)

Okay, so imagine this: you just generated an SSH RSA key and threw its public part on system B into ~foo/.ssh/authorized_keys and its private part on system A into ~bar/.ssh/id_rsa but can t login. Why? Automated processes (Jenkins *cough*) often need you to ssh(1) manually once, to accept the remote host s server key. Do that. The id_rsa file on system A must be owned by the user bar and chmod 0600 or 0400 (similarily, the .ssh directory has strict permission checks, and everything in the path until there). Check those. And, the most surprising one of the day: if there s an id_rsa.pub it will be used for offering a key to the remote host (B) even if it does not match the secret key. Deleting A:~bar/.ssh/id_rsa.pub apparently makes OpenSSH generate the public part from the secret key each time (or just put the correct pubkey there), but if one s there, it seems to like to use them. (That was the only part of this post that was news to even me, of course ) And, as bottom line: hello to Planet Debian from mirabilos at work , too. I ll occasionally tag posts so they show up here, if I think they re of interest, since I m doing Debian work at the dayjob, too.

Jacob, Oliver, and I were driving back home, about an 8-hour drive. At one point I heard Jacob say from the back seat, Oliver, you can t have my book. It has a bookmark in it! Aside from the apparent anti-sharing properties of bookmarks, I sort of smiled at Jacob inventing a bookmark. An hour later, I heard Oliver saying, Jacob, you can t have MY book. It has a bookmark too! And again I thought that sounded rather cute. The thought of traveling boys and they did travel very well bookmarking the books I brought for them was a nice one. And then I realized: I hadn t brought bookmarks. Or, as far as I could recall, bookmark-making material. I really wanted to say, Uhm, Jacob, just what did you make that bookmark from? But I didn t. I figured everyone would be happier not having to deal with that question.

The fourth wheezy based alpha release of Debian Edu was wrapped up today. This is the release announcement: New features for Debian Edu 7.1+edu0~alpha3 released 2013-07-03 These are the release notes for for Debian Edu / Skolelinux 7.1+edu0~alpha3, based on Debian with codename "Wheezy". About Debian Edu and Skolelinux Debian Edu, also known as Skolelinux, is a Linux distribution based on Debian providing an out-of-the box environment of a completely configured school network. Immediately after installation a school server running all services needed for a school network is set up just waiting for users and machines being added via GOsa , a comfortable Web-UI. A netbooting environment is prepared using PXE, so after initial installation of the main server from CD, DVD or USB stick all other machines can be installed via the network. The provided school server provides LDAP database and Kerberos authentication service, centralized home directories, DHCP server, web proxy and many other services. The desktop contains more than 60 educational software packages and more are available from the Debian archive, and schools can choose between KDE, Gnome, LXDE and Xfce desktop environment. This is the fourth test release based on Debian Wheezy. Basically this is an updated and slightly improved version compared to the Squeeze release. Software updates

• Dropped ispell dictionaries from our default installation.
• Dropped menu-xdg from the KDE desktop option, to drop the Debian submenu. It was not included with Gnome, LXDE or Xfce, so this brings KDE in line with the others.
• Dropped xdrawchem, xjig and xsok from our default installation as they don't have a desktop menu entry and thus won't show up in the menu now that menu-xdg was removed.
• Removed the killer system to kill left behind processes on multi-user machines, as it was no longer able to understand when a X display was in use and killed the processes of the active users too.
• Dropped the golearn (from goplay) package as the debtags in wheezy are too few to make the package useful.

Other changes

• Updated artwork matching http://wiki.debian.org/DebianArt/Themes/Joy
• Multi-arch i386/amd64 USB stick ISO available.
• Got rid of ispell/wordlist related debconf questions that showed up for some language options.
• Switched to using http.debian.net as APT source by default.
• Fixed proxy configuration on Main Server installations.
• Changed LTSP setup to ask dpkg to use force-unsafe-io the same way d-i is doing it.
• Made sure root and user passwords were not left behind in the debconf database after installation on Main Server installations.
• Made Roaming Workstation dynamic setup more robust and added draft script setup-ad-client to hook a Roaming Workstation up to a Active Directory server instead of a Debian Edu Main Server.
• Update system to install needed firmware packages during installation, to work properly in Wheezy.
• Update system to handle hardware quirks (debian-edu-hwsetup).
• Corrected PXE installation setup to properly pass selected desktop and keymap settings to PXE installation clients.
• LTSP diskless workstations use sshfs by default, allowing them to work without adding them to DNS and NIS netgroups for NFS access.

Known issues

• No mass import of user account data in GOsa (ldif or csv) available yet (698840).
• Artwork not enabled for all desktops.

Where to get it To download the multiarch netinstall CD release you can use

• ftp://ftp.skolelinux.org/skolelinux-cd/wheezy/debian-edu-7.1+edu0~a3-CD.iso
• http://ftp.skolelinux.org/skolelinux-cd/wheezy/debian-edu-7.1+edu0~a3-CD.iso
• rsync -avzP ftp.skolelinux.org::skolelinux-cd/wheezy/debian-edu-7.1+edu0~a3-CD.iso .

The MD5SUM of this image is: 2b161a99d2a848c376d8d04e3854e30c
The SHA1SUM of this image is: 498922e9c508c0a7ee9dbe1dfe5bf830d779c3c8 To download the multiarch USB stick ISO release you can use

• ftp://ftp.skolelinux.org/skolelinux-cd/wheezy/debian-edu-7.1+edu0~a3-USB.iso
• http://ftp.skolelinux.org/skolelinux-cd/wheezy/debian-edu-7.1+edu0~a3-USB.iso
• rsync -avzP ftp.skolelinux.org::skolelinux-cd/wheezy/debian-edu-7.1+edu0~a3-USB.iso .

The MD5SUM of this image is: 25e808e403a4c15dbef1d13c37d572ac
The SHA1SUM of this image is: 15ecfc93eb6b4f453b7eb0bc04b6a279262d9721 How to report bugs http://wiki.debian.org/DebianEdu/HowTo/ReportBugs

I have attended the SciPy 2013 conference in Austin, Texas. Here are my impressions.

Number one is the fact that the IPython notebook was used by pretty much everyone. I use it a lot myself, but I didn't realize how ubiquitous it has become. It is quickly becoming the standard now. The IPython notebook is using Markdown and in fact it is better than Rest. The way to remember the "[]()" syntax for links is that in regular text you put links into () parentheses, so you do the same in Markdown, and append [] for the text of the link. The other way to remember is that [] feel more serious and thus are used for the text of the link. I stressed several times to +Fernando Perez and +Brian Granger how awesome it would be to have interactive widgets in the notebook. Fortunately that was pretty much preaching to the choir, as that's one of the first things they plan to implement good foundations for and I just can't wait to use that.

It is now clear, that the IPython notebook is the way to store computations that I want to share with other people, or to use it as a "lab notebook" for myself, so that I can remember what exactly I did to obtain the results (for example how exactly I obtained some figures from raw data). In other words --- instead of having sets of scripts and manual bash commands that have to be executed in particular order to do what I want, just use IPython notebook and put everything in there.
Number two is that how big the conference has become since the last time I attended (couple years ago), yet it still has the friendly feeling. Unfortunately, I had to miss a lot of talks, due to scheduling conflicts (there were three parallel sessions), so I look forward to seeing them on video.

+Aaron Meurer and I have done the SymPy tutorial (see the link for videos and other tutorial materials). It's been nice to finally meet +Matthew Rocklin (very active SymPy contributor) in person. He also had an interesting presentation
about symbolic matrices + Lapack code generation. +Jason Moore presented PyDy.
It's been a great pleasure for us to invite +David Li (still a high school student) to attend the conference and give a presentation about his work on sympygamma.com and live.sympy.org.

It was nice to meet the Julia guys, +Jeff Bezanson and +Stefan Karpinski. I contributed the Fortran benchmarks on the Julia's website some time ago, but I had the feeling that a lot of them are quite artificial and not very meaningful. I think Jeff and Stefan confirmed my feeling. Julia seems to have quite interesting type system and multiple dispatch, that SymPy should learn from.

I met the VTK guys +Matthew McCormick and +Pat Marion. One of the keynotes was given by +Will Schroeder from Kitware about publishing. I remember him stressing to manage dependencies well as well as to use BSD like license (as opposed to viral licenses like GPL or LGPL). That opensource has pretty much won (i.e. it is now clear that that is the way to go).

I had great discussions with +Francesc Alted, +Andy Terrel, +Brett Murphy, +Jonathan Rocher, +Eric Jones, +Travis Oliphant, +Mark Wiebe, +Ilan Schnell, +St fan van der Walt, +David Cournapeau, +Anthony Scopatz, +Paul Ivanov, +Michael Droettboom, +Wes McKinney, +Jake Vanderplas, +Kurt Smith, +Aron Ahmadia, +Kyle Mandli, +Benjamin Root and others.


It's also been nice to have a chat with +Jason Vertrees and other guys from Schr dinger.

One other thing that I realized last week at the conference is that pretty much everyone agreed on the fact that NumPy should act as the default way to represent memory (no matter if the array was created in Fortran or other code) and allow manipulations on it. Faster libraries like Blaze or ODIN should then hook themselves up into NumPy using multiple dispatch. Also SymPy would then hook itself up so that it can be used with array operations natively. Currently SymPy does work with NumPy (see our tests for some examples what works), but the solution is a bit fragile (it is not possible to override NumPy behavior, but because NumPy supports general objects, we simply give it SymPy objects and things mostly work).

Similar to this, I would like to create multiple dispatch in SymPy core itself, so that other (faster) libraries for symbolic manipulation can hook themselves up, so that their own (faster) multiplication, expansion or series expansion would get called instead of the SymPy default one implemented in pure Python.

Other blog posts from the conference:

• Aaron's post
• Fernando's post

Julien LIE pointed out a rendering problem with one of my pages, and some further investigation revealed that the changes that I made to faq2html when moving away from Subversion caused all sorts of problems. I started testing a new version of it, and then found a bunch of other problems with pod2thread. One I already knew about, but others I'd not seen before. Since I was working on this anyway, I decided to fix all the problems I knew about. Pod::Thread 1.01 now correctly encodes its output for UTF-8, which will fix some long-standing problems with ISO 8859-1 characters used to render non-breaking spaces. It also fixes some minor formatting issues and adds a workaround for the inability to format formatting codes in headings when generating an artificial table of contents. faq2html 1.32 now supports multiple subheadings at the top of a page provided that the start of the second subheading is still a centered line. I use this format in a bunch of my package README files, all of which had been misformatted due to not having this support. I did a large test generation of my web pages, and I don't think I added any new problems. (faq2html is inherently tricky, since it's converting plain text to HTML and has to do a bunch of markup guessing. I suppose one of these days I should just give up and switch over to writing README files in Markdown or something.) While I was at it, I also modernized Pod::Thread's module packaging and switched it to Module::Build, which is the direction I'm (slowly) going for all of my Perl module work. The generic parts of the test suite also got a refresh to the latest versions of my generic test suite code (which I'm currently maintaining in rra-c-util for lack of a better location). That had one interesting wrinkle. Pod::Thread was the one Perl distribution where I'd played with using a module filter program (something ExtUtils::MakeMaker supports) to set the module $VERSION automatically during the build. This had the advantage of letting me maintain the distribution version in one place (Makefile.PL) no matter how many Perl modules there are. However, Module::Build doesn't support that. So, this afternoon I wrote Module::Build::SetVersion, which overrides the processing of *.pm files to set the $VERSION string to whatever one sets dist_version to in Build.PL. This lets you just put placeholder strings in all of the modules, and the process of building them into blib will substitute in the current distribution version. Currently, this module is only in this package and subclasses Module::Build (it really should be a mixin of some kind, but I didn't work out a good way of doing that). I'm going to experiment with copying it into my various module packages, but I suppose at some point I should wrap it up in its own distribution, add a test suite, and release it properly. (If you would find this useful, drop me email and I'll prioritize releasing it properly on CPAN somewhat higher.) Note that you can do the same thing with the existing Module::Build::PM_Filter module, as well as a bunch of other things, but that requires you write and maintain a Perl script at the top level of the distribution that does the desired transformations and seemed unnecessarily complex for what I was trying to do. I really wanted something I could just drop in and forget about. You can get the latest releases of Pod::Thread and faq2html from my web tools distribution page.

A few months ago, I was looking for a piece of software to spawn long-running tasks on behalf of some daemon with the benefit of the tasks not being interrupted when this daemon is restarted. Here are the specifications:

• No special privilege is needed to submit a task.
• Submitted tasks are not known beforehand.
• Task output is redirect to a log file.
• Submitted tasks are identified by a provided name.
• With only the name as a reference, a task can be checked for existence or killed.
• Tasks should not be interrupted by an unrelated event, like a configuration change or a software upgrade.

The last requirement explains why tasks are not spawned directly by the daemon requesting them: its complexity or the way it needs to be operated may need frequent restarts. Even if it is possible to re-execute a daemon while keeping its children, like stateful re-exec support in Upstart, this is quite difficult: internal state should be serialized and restored. Part of this state can be contained into a third-party library. While Upstart or systemd seemed to be good candidates for this purpose, I didn t find a straightforward way to run arbitrary tasks without any privilege¹. Here comes lan o². It is a very simple task launcher. It can run any tasks, stop them and check if they are still running. It leverages cgroups in recent Linux kernels and avoids the use of any daemon.

Quick tour Before looking at how lan o works, let s have a look at how it can be used. To avoid usage conflicts, each task is run in the context of a namespace that needs to be initialized:

$ sudo lanco testns init -u $(id -un) -g $(id -gn)

This is the only command that needs to be run as root. Subsequent ones can be run as a normal user. Let s run some task:

$ lanco testns run first-task openssl speed aes
$ lanco testns check first-task && echo "Still running"
Still running
$ lanco testns ls
testns
   first-task
      28456 openssl speed aes 
  

The output of the task is logged into a file:

$ head -3 /var/log/lanco-testns/task-first-task.log
Doing aes-128 cbc for 3s on 16 size blocks: 8678442 aes-128 cbc's in 2.85s
Doing aes-128 cbc for 3s on 64 size blocks: 2478283 aes-128 cbc's in 2.99s
Doing aes-128 cbc for 3s on 256 size blocks: 628105 aes-128 cbc's in 3.00s

If the task is too long, we can kill it:

$ lanco testns run first-task openssl speed aes
$ lanco testns stop first-task

You cannot run a task that already exists or kill a task that does not exist:

$ lanco testns run first-task openssl speed aes
2013-06-09T22:50:34 [WARN/run] task first-task is already running
$ lanco testns stop second-task
2013-06-09T22:50:45 [WARN/stop] task second-task is not running

Thanks to the use of cgroups, lan o is able to track multiple processes even when they are forking away³:

$ lanco testns run second-task sh -c \
>  "while true; do (sleep 30 &)& sleep 1; done"
$ lanco testns ls
testns
   first-task
      28456 openssl speed aes 
   second-task
      29572 sh -c while true; do (sleep 30 &)& sleep 1; done 
      29575 sleep 30 
      29593 sleep 30 
      29596 sleep 30 
      29599 sleep 30 
      29622 sleep 30 
      29644 sleep 1 
      29645 sleep 30 
  
$ lanco testns stop second-task
$ lanco testns check second-task   echo "Killed!"
Killed!

Also, there is a top-like command (lanco testns top):

Using cgroups Control groups (cgroups) is a mechanism to partition a set of tasks and their future children into hierarchical groups with a set of parameters.

Hierarchy Let s start with the hierarchical stuff first. To create a new hierarchy, you have to mount the cgroup filesystem on some empty directory. Usually, cgroups hierarchies are setup in /sys/fs/cgroups which is a tmpfs filesystem:

# mount -t tmpfs tmpfs /sys/fs/cgroup -o nosuid,nodev,noexec,relatime,mode=755

Now, we can create our first hierarchy:

# cd /sys/fs/cgroup
# mkdir my-first-hierarchy
# mount -t cgroup cgroup my-first-hierarchy -o name=my-first-hierarchy,none
# ls -1 my-first-hierarchy
cgroup.clone_children
cgroup.event_control
cgroup.procs
notify_on_release
release_agent
tasks

We ll see later the purpose of none. The most interesting file is tasks. It contains the PIDs of all processes in our group. Since there are currently no sub-group, all processes are part of it. Let s create a sub-group and attach a process to it:

# mkdir first-child
# cd first-child
# ls -1
cgroup.clone_children
cgroup.event_control
cgroup.procs
notify_on_release
tasks
# echo $$ > tasks
# cat tasks
23184
23311
# cat /proc/$$/cgroup 
9:name=my-first-hierarchy:/first-child
8:perf_event:/
7:blkio:/
6:net_cls:/
5:freezer:/
4:devices:/
3:cpuacct,cpu:/
2:cpuset:/
1:name=systemd:/user/bernat/1

We have added our shell to the new cgroup. Moreover, all its children will also be part of this group. This explains why we have two tasks: the shell and cat. The last command is quite interesting: for each hierarchy, it shows which cgroup the task belongs to. For a given hierarchy, each task is exactly in one group. The most useful features of lan o are done using just this: a namespace is a named hierarchy and each task is enclosed into a cgroup so it can be tracked properly.

Subsystems A subsystem makes use of the task grouping provided by cgroups to assign resources to a set of tasks. Each available subsystem can only be in one hierarchy, so the usual way to setup things is to assign a hierarchy for each subsystem⁴. Let s have a look at the cpuset subsystem: it allows to assign tasks to specific CPUs and memory nodes (for NUMA systems). Let s suppose we have 4 cores and we want to assign the first core to common system tasks and the three remaining ones to nginx:

# cd /sys/fs/cgroup
# mkdir cpuset
# mount -t cgroup cgroup cpuset -o cpuset
# echo 0-3 > cpuset/cpuset.cpus
# echo 0 > cpuset/cpuset.mems
# mkdir cpuset/system
# echo 0 > cpuset/system/cpuset.cpus
# echo 0 > cpuset/system/cpuset.mems
# for task in $(cat cpuset/tasks); do
>    echo $task > cpuset/system/tasks
> done
# mkdir cpuset/nginx
# echo 1-3 > cpuset/nginx/cpuset.cpus
# echo 0 > cpuset/nginx/cpuset.mems
# for task in $(pidof nginx); do
>    echo $task > cpuset/nginx/tasks
> done

The first mount is not needed if something else (like systemd) has setup the appropriate subsystem. Now, even if some system process is going crazy, it won t affect the performance of your webserver. The kernel comes with a more complete documentation if you need additional details on cgroups. There are high level tools to manipulate them, like the tools provided by libcg, but they are currently quite buggy and invasive.

Use in lan o Here are the features used by lan o:

• For each namespace, a specific named hierarchy is created.
• By setting the appropriate permissions on the hierarchy, an unprivileged user can create subgroups.
• Each submitted task has its own sub-group for tracking purpose.
• Actions can be executed when a task terminates by using the release agent mechanism⁵.
• The cpuacct subsystem is used to track CPU usage: a group is created in this subsystem for this purpose.

---

1. Both systemd and Upstart are difficult to run without being PID 1 and without being root. They both support user sessions but this requires to use them as PID 1 as well. I discovered later than runit would have been a good fit: it does not require to be PID 1, it does not need to be run as root and the service directory can be specified by an environment variable. It could have been tweaked to meet the above requirements.
2. Lan o means launcher in esperanto. The unusual diacritic on the letter c is not present in many web fonts, so the rendering may be a little odd.
3. Linux cgroups do not provide any facility to kill a whole group. We need to iterate through the processes and kill them one by one. lan o does not try to freeze processes before killing them and therefore may be inefficient against violent fork bombs.
4. The cpu and cpuacct subsystems are the notable exception. There seems to be little use to keep them in separate hierarchies, so they are part of the same hierarchy.
5. Unfortunately, only one global release agent per hierarchy can be used. Since lan o provides with the ability to execute an arbitrary command when a task terminates, the command needs to be stored on the filesystem for the release agent of the namespace to execute it.

It s been two years since my latest blog post about my Linux desktop and many things have changed since then. I completely moved all my machines to GNOME 3, switched my main editor from nano to vim and my terminal multiplexer from screen to tmux. What didn t change at all except for a tweaks on the theme is my Irssi setup. Switching from nano to vim has been a pain at first, nano is really a straightforward editor, it does what you actually need from a CLI editor but while it works just fine when modifying configuration or text files, it s a bit limiting when it comes to programming. Vim on the other hand is highly customizable in every single part also thanks to its huge amount of plugins. Honestly I admit I spent several hours watching videos, reading documentation, trying out key bindings and I m not completely used to vim to be as much productive I would like to be with it. What I found to be the common error of vim s newcomers is their willingness to look around the web for a complete vimrc configuration file, full of key bindings, custom settings and personalizations. That s definitely something you should avoid when learning to use vim, the perfect vimrc doesn t objectively exist, each of us should spend some time investigating what is the best configuration for your needs and build a vimrc accordingly time by time. It will probably take months to have a complex vimrc file matching your needs completely, until then you won t be able to define your vimrc to be ultimate . And that s actually why wgetting someone else s vimrc and copying it to your home folder won t make you an expert of vim, it ll probably make your life harder when trying a specific action with the stock settings will result in something you wouldn t have expected thanks to a particular key binding on the vimrc you downloaded. The other tool that definitely improved my productivity is Tmux given the huge amount of open terminals I had every day during my sysadmin s duties at GNOME. Each day usually started with one or two open terminals mainly meant for random maintenance issues, after a few hours the amount of open terminals jumped to around 30. Switching between tabs became a pain and the usual amount of time spent finding out which terminal had the the specific remote connection I needed was surprisingly sticking around 10 seconds. I looked around for a possible solution and found Tmux, it has been a joy finding out how easy it was to set it up and get used to it. I m now able to divide my 27 inches monitor into several panels and switch between them with a little keybinding, in addition you can set up as many windows (and consequently as many panels) as you wish, that way I can divide my programming windows from the ones specifically meant as remote consoles. From there I can detach / attach my previous Tmux session with one command and keep working on the same files and connections I had open the last time I used Tmux. In addition to the above tools I ve kept being an huge fan of Irssi and some of its plugins that have improved my overall productivity when daily chatting with Fedora, Debian and GNOME contributors. What actually changed on it since two years ago is the theme which is currently a mix between the Solarized and the XChat themes. Please don t hesitate to ask me any question if you are curious about learning something more about my current working setup! :-)

NSCA is a tool used to submit passive check results to nagios. Unfortunately, an incompatibility was recently introduced between wheezy clients and old servers. Since I don't want to upgrade my server, this caused some problems and I decided to just get rid of NSCA completely. The server side of NSCA is pretty trivial, it basically just adds a timestamp and a command name to the data sent by the client, then changes tabs into semicolons and stuffs all of that down Nagios' command pipe. The script I came up with was:

#! /usr/bin/python
# -* coding: utf-8 -*-
import time
import sys
# format is:
# [TIMESTAMP] COMMAND_NAME;argument1;argument2; ;argumentN
#
# For passive checks, we want PROCESS_SERVICE_CHECK_RESULT with the
# format:
#
# PROCESS_SERVICE_CHECK_RESULT;<host_name>;<service_description>;<return_code>;<plugin_output>
#
# return code is 0=OK, 1=WARNING, 2=CRITICAL, 3=UNKNOWN
#
# Read lines from stdin with the format:
# $HOSTNAME\t$SERVICE_NAME\t$RETURN_CODE\t$TEXT_OUTPUT
if len(sys.argv) != 2:
    print "Usage:  0  HOSTNAME".format(sys.argv[0])
    sys.exit(1)
HOSTNAME = sys.argv[1]
timestamp = int(time.time())
nagios_cmd = file("/var/lib/nagios3/rw/nagios.cmd", "w")
for line in sys.stdin:
    (_, service, return_code, text) = line.split("\t", 3)
    nagios_cmd.write(u"[ timestamp ] PROCESS_SERVICE_CHECK_RESULT; hostname ; service ; return_code ; text \n".format
                     (timestamp = timestamp,
                      hostname = HOSTNAME,
                      service = service,
                      return_code = return_code,
                      text = text))

The reason for the hostname in the line (even though it's overridden) is to be compatible with send_nsca's input format. Machines submit check results over SSH using its excellent ForceCommand capabilities, the Chef template for the authorized_keys file looks like:

<% for host in @nodes %>
command="/usr/local/lib/nagios/nagios-passive-check-result <%= host[:hostname] %>",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa <%= host[:keys][:ssh][:host_rsa_public] %> <%= host[:hostname] %>
<% end %>

The actual chef recipe looks like:

nodes = []
search(:node, "*:*") do  n 
  # Ignore not-yet-configured nodes                                                                       
  next unless n[:hostname]
  next unless n[:nagios]
  next if n[:nagios].has_key?(:ignore)
  nodes << n
end
nodes.sort!    a,b  a[:hostname] <=> b[:hostname]  
print nodes
template "/etc/ssh/userkeys/nagios" do
  source "authorized_keys.erb"
  mode 0400
  variables( 
              :nodes => nodes
             )
end
cookbook_file "/usr/local/lib/nagios/nagios-passive-check-result" do
  mode 0555
end
user "nagios" do
  action :manage
  shell "/bin/sh"
end

To submit a check, hosts do:

printf "$HOSTNAME\t$SERVICE_NAME\t$RET\t$TEXT\n"   ssh -i /etc/ssh/ssh_host_rsa_key -o BatchMode=yes -o StrictHostKeyChecking=no -T nagios@$NAGIOS_SERVER