Search Results: "zed"

26 September 2011

John Goerzen: Five

Dad, will you and mom stay up all night decorating the house for me?
That was Jacob s question to me at bedtime the evening before his fifth birthday. Jacob had already had his birthday party a few weeks ago. When scheduling means that parties happen that far away from the boys real birthdays, they get a smaller celebration with just Terah and me where we open their presents from us. A low-key thing, so we weren t planning to decorate the house. I said, Well, I don t think we ll be doing all that since you already had your birthday party. And the look of eager anticipation on his face turned to a very sad and disappointed look, and made me feel really bad. Uhoh. So while Jacob was sleeping, I mentioned it to Terah. We decided we should improvise something simple, so she found some old streamers and we taped them up, running them through several rooms in the house and across his door. It took a few minutes using supplies we already had, but the joy the next morning was priceless. Oh dad, you said you wouldn t stay up all night, BUT YOU DID! Oh I am SO HAPPY! YAY YAY YAY! And he ran through the house to discover what else was set up. Then he ran to find Oliver and gave him a tour of everything. Then we sat down to open his presents. Here he is, holding a present from Terah and me: That s a copy of The Lorax. My friend Jonathan had brought a copy along during our road trip in Mexico, and Jacob was really excited about it. And didn t really want to give it up, because You can only get The Lorax in Mexico. He (and Oliver) really enjoyed all his presents he also got a train book from us (which he said, Oh, yay, it s the book they have at preschool! ), and a game and some other presents from the distant relatives. But the highlight was something of an impulse buy. I was at the RadioShack in Derby a place I ve written about before, It is what a RadioShack should (and used to) be. It has a large amateur radio section, sells all sorts of coax by the foot, and provides astonishingly good post-sale service. Well, I was there with a radio question, and Mark (the owner) who is an excellent salesman in a positive way pointed me to a display of snap kits. I noticed their Electronics 101 Snap-Kit (a rebranded Snap Circuits Jr. SC-100). A ham in Pennsylvania had suggested them to me once, and as Jacob s birthday was coming up, I gave it some thought. The kit said ages 8 and up. I asked Mark what he thought about a boy just turning 5. He said, Well, probably not normally. But knowing you, if you re there to work on them with him, I think he d enjoy it. But I wouldn t have him work on it by himself. I agreed and we bought it. I pondered how to explain the concept of this thing to Jacob. Eventually I decided I would call it a toy-building kit. He understood that. Jacob and I spent hours together working with it. He would flip through the book, either picking circuits that looked interesting or telling me what kind of circuit he wanted to build. Then I would tell him what to put where, and he d snap them together and play with them. He only played with each finished product a few minutes before he was ready to try another. Once I got him very excited with my offer to show him how to hook up two switches in parallel for the fan he built (and later introduced the parallel vs. series concept by hooking them up in series instead.) Here we are working on it together. Jacob repeatedly called Terah over to look at the things he built. He was very excited that he assembled it himself. Eventually, Oliver (age 2) came over wanting to help. So he sat on my lap, and handed parts to Jacob, then Jacob put them on the grid. Oliver really enjoyed being involved in this way, even though I had to keep him from doing things like ripping the capacitor off its mount. We tend to be modest in terms of the number of things we give the boys and their cost, reasoning that we, like many, already have too many toys in our house, and that greater cost doesn t necessarily equate with a better experience for the boys. I particularly look for things with lasting value and unique experiences for them, and I think we succeeded this year. But I realized quickly that the greatest value of this kit wasn t electronics. It was having a great way for me to spend a lot of time doing things with the boys which all of us enjoyed. Those hours building things together were as much a present for me as for Jacob, I m quite sure. Jacob s real party was a few weeks ago at the Great Plains Transportation Museum in Wichita. They let people rent a historic caboose to use for a birthday party for children. So we did that for Jacob this year. That was a huge hit for the boys. Jacob got to help his grandpa make some pie (instead of cake) for the party. He enjoyed eating it, of course. He enjoyed opening his presents high up on the observation chair in the caboose. And the boys got to play on all the other equipment in the museum. Jacob enjoyed playing tour guide for family since most of them hadn t been there. He also enjoyed watching freight trains pass on the other side of the fence from the museum preferably while sitting in one of the museum s engines. Oliver certainly didn t get left out. Train-watching is serious business, after all. Jacob has long talked about going to the train museum and the airplane museum (Kansas Aviation Museum) on the same day, so one of his birthday surprises was that we went to the airplane museum after his party. His favorite item there is a retired FedEx 727. Here he is walking down the rear of the plane. And, of course, they played captain and co-pilot in several different planes.

24 September 2011

Sylvain Beucler: Smooth lighting

phong After some restless hours of debugging, here's Phong lighting :)
http://en.wikibooks.org/wiki/GLSL_Programming/GLUT/Smooth_Specular_Highlights
The code can be optimized, but should give you a clear understanding on how to make smooth OpenGL lighting. And you can admire the stunning results with a mere basic material and one lamp!

23 September 2011

Matthew Garrett: Supporting UEFI secure boot on Linux: the details

An obvious question is why Linux doesn't support UEFI secure booting. Let's ignore the issues of key distribution and the GPL and all of those things, and instead just focus on what would be required. There's two components - the signed binary and the authenticated variables.

The UEFI 2.3.1 spec describes the modification to the binary format required to produce a signed binary. It's not especially difficult - you add an extra entry to the image directory, generate a hash of the entire binary other than the checksum, the certificate directory entry and the signatures themselves, encrypt that hash with your key and embed the encrypted hash in the binary. The problem has been that there was a disagreement between Microsoft and Intel over whether this signature was supposed to include the PKCS header or not, and until earlier this week the only widely available developer firmware (Intel's) was incompatible with the only widely available signed OS (Microsoft's). There's further hilarity in that the specification lists six supported hash algorithms, but the implementations will only accept two. So pretty normal, really. Developing towards a poorly defined target is a pain. Now that there's more clarity we'll probably have a signing tool before too long.

Authenticated variables are the other part of the puzzle. If a variable requires authentication, the operating system's attempt to write it will fail unless the new data is appropriately signed. The key databases (white and blacklists) are examples of authenticated variables. The signing actually takes place in userspace, and the handoff between the kernel and firmware is identical for both this case and the unauthenticated case. The only problem in Linux's support here is that our EFI variable support was written to a pre-1.0 version of the EFI specification which stated that variables had a maximum size of 1024 bytes, and this limitation ended up exposed to userspace. So all we really need to do there is add a new interface to let arbitrary sized variables be written.

Summary: We don't really support secure boot right now, but that's ok because you can't buy any hardware that supports it yet. Adding support is probably about a week's worth of effort at most.

comment count unavailable comments

22 September 2011

Christian Perrier: 3rd update about Debian Installer localization

Yet another progress report about D-I l10n. Full stats are here Status for D-I level 1 (core D-I files): New complete languages since last update: Esperanto, Spanish, Tamil, Simplified Chinese. Status for D-I level 2 (packages that have localized material that may appear during default installs, such as iso-codes, tasksel, etc.): Status for D-I level 3 (packages that have localized material that may appear during non-default installs, such as win32-loader)

21 September 2011

Ian Wienand: Exploring $ORIGIN

Anyone who works with building something with a few libraries will quickly become familiar with rpath's; that is the ability to store a path inside a binary for finding dependencies at runtime. Slightly less well known is that the dynamic linker provides some options for this path specification; one in particular is a path with the special variable $ORIGIN. ld.so man page has the following to say about the an $ORIGIN appearing in the rpath:
  ld.so understands the string $ORIGIN (or equivalently $ ORIGIN )
  in  an  rpath specification to mean the directory containing the
  application  executable.  Thus,  an   application   located   in
  somedir/app   could   be  compiled  with  gcc  -Wl,-rpath,'$ORI 
  GIN/../lib' so that it finds an  associated  shared  library  in
  somedir/lib  no matter where somedir is located in the directory
  hierarchy.
  
As a way of a small example, consider the following:
$ cat Makefile
main: main.c lib/libfoo.so
	gcc -g -Wall -o main -Wl,-rpath='$$ORIGIN/lib' -L ./lib -lfoo main.c
lib/libfoo.so: lib/foo.c
	gcc -g -Wall -fPIC -shared -o lib/libfoo.so lib/foo.c
$ cat main.c
void function(void);
int main(void)
 
   function();
   return 0;
 
$ cat lib/foo.c
#include <stdio.h>
void function(void)
 
   printf("called!\n");
 
$ make
gcc -g -Wall -fPIC -shared -o lib/libfoo.so lib/foo.c
gcc -g -Wall -o main -Wl,-rpath='$ORIGIN/lib' -L ./lib -lfoo main.c
$ ./main
called!
Now, wherever you should choose to locate main, as long as the library is in the right relative location it will be found correctly. If you are wondering how this works, examining the back-trace from the linker function that expands it is helpful:
$ gdb ./main
(gdb) break _dl_get_origin
Function "_dl_get_origin" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (_dl_get_origin) pending.
(gdb) r
Starting program: /tmp/test-origin/main 
Breakpoint 1, _dl_get_origin () at ../sysdeps/unix/sysv/linux/dl-origin.c:37
37	../sysdeps/unix/sysv/linux/dl-origin.c: No such file or directory.
	in ../sysdeps/unix/sysv/linux/dl-origin.c
(gdb) bt
#0  _dl_get_origin () at ../sysdeps/unix/sysv/linux/dl-origin.c:37
#1  0x00007ffff7de61f4 in expand_dynamic_string_token (l=0x7ffff7ffe128, 
    s=<value optimized out>) at dl-load.c:331
#2  0x00007ffff7de62ea in decompose_rpath (sps=0x7ffff7ffe440, 
    rpath=0xffffffc0 <Address 0xffffffc0 out of bounds>, l=0x0, 
    what=0x7ffff7df780a "RPATH") at dl-load.c:554
#3  0x00007ffff7de7051 in _dl_init_paths (llp=0x0) at dl-load.c:722
#4  0x00007ffff7de2124 in dl_main (phdr=0x7ffff7ffe6b8, 
    phnum=<value optimized out>, user_entry=0x7ffff7ffeb28) at rtld.c:1391
#5  0x00007ffff7df3647 in _dl_sysdep_start (
    start_argptr=<value optimized out>, dl_main=0x7ffff7de14e0 <dl_main>)
    at ../elf/dl-sysdep.c:243
#6  0x00007ffff7de0423 in _dl_start_final (arg=0x7fffffffe7c0) at rtld.c:338
#7  _dl_start (arg=0x7fffffffe7c0) at rtld.c:564
#8  0x00007ffff7ddfaf8 in _start () from /lib64/ld-linux-x86-64.so.2
Just from a first look we can divine that this has found the RPATH header in the dynamic section and has decided to expand the string $ORIGIN.
$ readelf --dynamic ./main
Dynamic section at offset 0x7d8 contains 23 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libfoo.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/lib]
 0x000000000000000c (INIT)               0x4004d8
 0x000000000000000d (FINI)               0x4006f8
At this point it proceeds to do a readlink on /proc/self/exe to get the path to the currently running binary, effectively does a basename on it and replaces the value of $ORIGIN. Interestingly, if this should fail, it will fall back on the environment variable LD_ORIGIN_PATH for unknown reasons. This might be useful if you were in a bad situation where /proc was not mounted and you still had to run your binary, although you could probably achieve the same thing via the use of LD_LIBRARY_PATH just as well. This feature should be used with some caution to avoid turning your application in a mess that can't be packaged in any standard manner. One common example of use is probably your java virtual machine to find its implementation libraries.
$ readelf --dynamic /usr/lib/jvm/java-6-sun/bin/java
Dynamic section at offset 0x9a28 contains 25 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libjli.so]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [lib.so]
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/../lib/amd64/jli:$ORIGIN/../jre/lib/amd64/jli]
 ...
There is a small but interesting security issue. If an attacker can hardlink to your binary that is using $ORIGIN then any path expansion is now relative to the attackers directory. Hence it is possible to make the linker read in arbitrary libraries and hence run arbitrary code. Clearly if your program is running setuid as someone else, such as root, you've just given up the keys to the house! Luckily, the dynamic linker will not let you shoot yourself in the foot like this, and prevents expansion of the $ORIGIN field if the program is running setuid.
$ sudo chown root ./main
$ sudo chmod +s ./main
$ ./main
./main: error while loading shared libraries: libfoo.so: cannot open shared object file: No such file or directory
However, this is a good example of why you might want to keep users home and temp directories on separate file systems away from where binaries live. There is also a similar variable $PLATFORM, which describes the current running system. This is gained via the auxiliary vector, which I have written about here and here. Setting LD_SHOW_AUXV=1 will allow you to examine this value.
$ LD_SHOW_AUXV=1 ls
...
AT_PLATFORM:     x86_64
...

19 September 2011

Joey Hess: happy haskell hacker

There are certian things haskell is very good at, and I had the pleasure of using it for one such thing yesterday. I wanted to support expressions like find(1) does, in git-annex. Something like:
git-annex drop --not --exclude '*.mp3' --and \
    -\( --in usbdrive --or --in archive -\) --and \
    --not --copies 3
So, parens and booleans and some kind of domain-specific operations. It's easy to build a data structure in haskell that can contain this sort of expression.
 - A Token can either be a single word, or an Operation of an arbitrary type. - 
data Token op = Token String   Operation op
        deriving (Show, Eq)
data Matcher op = Any
          And (Matcher op) (Matcher op)
          Or (Matcher op) (Matcher op)
          Not (Matcher op)
          Op op
        deriving (Show, Eq)
(The op could just be a String, but is parameterized for reasons we'll see later.) As command-line options, the expression is already tokenised, so all I needed to do to parse it is consume a list of the Tokens. The only mildly tricky thing is handling the parens right -- I chose to not make it worry if there were too many, or too few closing parens.
generate :: [Token op] -> Matcher op
generate ts = generate' Any ts
generate' :: Matcher op -> [Token op] -> Matcher op
generate' m [] = m
generate' m ts = uncurry generate' $ consume m ts
consume :: Matcher op -> [Token op] -> (Matcher op, [Token op])
consume m [] = (m, [])
consume m ((Operation o):ts) = (m  And  Op o, ts)
consume m ((Token t):ts)
          t == "and" = cont $ m  And  next
          t == "or" = cont $ m  Or  next
          t == "not" = cont $ m  And  (Not next)
          t == "(" = let (n, r) = consume next rest in (m  And  n, r)
          t == ")" = (m, ts)
          otherwise = error $ "unknown token " ++ t
        where
                (next, rest) = consume Any ts
                cont v = (v, rest)
Once a Matcher is built, it can be used to check if things match the expression the user supplied. This next bit of code almost writes itself.
 - Checks if a Matcher matches, using a supplied function to check
 - the value of Operations. - 
match :: (op -> v -> Bool) -> Matcher op -> v -> Bool
match a m v = go m
        where
                go Any = True
                go (And m1 m2) = go m1 && go m2
                go (Or m1 m2) = go m1   go m2
                go (Not m1) = not (go m1)
                go (Op o) = a o v
And that's it! This is all nearly completly generic and could be used for a great many things that need support for this sort of expression, as long as they can be checked in pure code. A trivial example:
*Utility.Matcher> let m = generate [Operation True, Token "and", Token "(", Operation False, Token "or", Token, "not", Operation False, Token ")"]
*Utility.Matcher> match (const . id) m undefined 
True
For my case though, I needed to run some IO actions to check if expressions about files were true. This is where I was very pleased to see a monadic version of match could easily be built.
 - Runs a monadic Matcher, where Operations are actions in the monad. - 
matchM :: Monad m => Matcher (v -> m Bool) -> v -> m Bool
matchM m v = go m
        where
                go Any = return True
                go (And m1 m2) = liftM2 (&&) (go m1) (go m2)
                go (Or m1 m2) =  liftM2 ( ) (go m1) (go m2)
                go (Not m1) = liftM not (go m1)
                go (Op o) = o v
With this and about 100 lines of code to implement specific tests like --copies and --in, git-annex now supports the example at the top. Just for comparison, find(1) has thousands of lines of C code to build a similar parse tree from the command line parameters and run it. Although I was surprised to see that it optimises expressions by eg, reordering cheaper tests first. The last time I wrote this kind of thing was in perl, and there the natural way was to carefully translate the expression into perl code, which was then evaled. Meaning the code was susceptible to security holes. Anyway, this is nothing that has not been done a hundred times in haskell before, but it's very nice that it makes it so clean, easy, and generic.

Keith Packard: MacBook-Air-2

Fixing the Sandbrige MacBook Air display initialization We left our hero with Debian installed on the MacBook Air, but with the display getting scrambled as soon as the i915 driver loaded. As was reported to Matthew, the problem is as simple as a lack of the right mode for the eDP panel in the machine. This mode is supposed to come from the panel EDID data, but for some reason the driver wasn t able to query the EDID data, and so it decided to try some random panel timings it dug out of the VBT tables, which are generally supposed to be used by LVDS panels. Apple helpfully stuck valid data there, but for some other panel one that is 1280x800 pixels instead of the 1366x768 pixel panel in the MacBook Air. I heard rumors that some machines would get a black screen when the i915 driver loaded. I was fortunate my machine simply displayed a 1366x768 subset of the programmed 1280x800 mode. A bit of garbage on the right side, and a few scanlines missing at the bottom. Quite workable, especially after I ran fbset -yres 768 to keep the console in the visible portion of the screen. DDC failure Looking through the kernel logs, the Intel driver tries to access the EDID data and times out, as if DDC is just completely broken. This is rather unexpected; the eDP spec says that the panel is required to support DDC and provide EDID. Now, we ve seen a lot of panels which don t quite live up to the rigorous eDP specifications, but it s a bit surprising from Apple, who generally do VESA stuff pretty well. We ve heard reports about panels reporting invalid EDID data, or EDID data which didn t actually match the panel (causing us to prefer the VBT data on LVDS machines). But I ve not heard of an eDP panel which didn t have anything hanging off of the DDC channel. But X works fine? During early debugging, I happened to start X up. Much to my surprise, X came up at the native 1366x768 mode. Digging through the kernel logs after that, I discovered that EDID was successfully fetched from the eDP panel while X started up. At this point, I knew it was all downhill the EDID data was present, it just wasn t getting picked up during the early part of the driver initialization when the console mode is initialized. eDP power management The CPU is given complete control over the power management of the eDP panel; sequencing through various power states and waiting appropriate amounts of time when things change. Given the goal of keeping power usage as low as possible, this makes a huge amount of sense. The eDP spec is quite clear though, without power, the panel will not respond to anything over the aux channel, and that includes EDID data. The eDP panel power hardware in the Sandybridge chip has a special mode for dealing with this requirement. If the panel is not displaying data, you can supply power for the aux channel stuff by setting a magic bit in the panel power registers. When initializing the frame buffer, the kernel driver turns off the panel completely so that it has all of the hardware in a known state (yeah, this is not optimal, but that s another bug). When X started, the panel was already running with the console mode. Given the difference between these two states EDID querying with the panel off failed, while EDID querying with the panel on worked, it seemed pretty clear that the panel power wasn t getting managed correctly. So, it seemed pretty clear that the magic power the panel bit wasn t getting turned on at the right times. Getting the power turned on. I stuck a check inside all of the aux channel communication functions to see where things were broken. This pointed out several places missing the panel power calls. This wasn t quite sufficient to get EDID data flowing. The remaining problem was that the code wasn t waiting long enough after turning the panel power on before starting the aux channel communication. A few msleep calls and huzzah! EDID at boot time and the console had the right mode. Making it faster However, it turns out that the driver does this a lot, and the msleeps required were fairly long the eDP panel wants a 500ms delay from turning the panel power off before you can turn it back on. I fix this by simply delaying the panel power off until things were idle for a long time. Now mode setting goes zipping through, and a few seconds later, the bit to force panel power on gets turned off. Getting these bits for yourself I ve pushed out the code to my (temporary) kernel repository git://people.freedesktop.org/~keithp/linux in the fix-edp-vdd-power branch. I d love to hear if you ve tried this on either a MacBook Air or any other eDP machine from Ironlake onwards.

17 September 2011

Patrick Schoenfeld: Struggling with Advanced Format during a LVM to RAID migration

Recently I decided to invest in another harddisk for my atom system. That system, I built up almost two years ago, has become the central system in my home network, serving as a fileserver to host my personal data, some git repositories etc., streaming server and since I switched to a cable internet connection it also serves as a router/firewall.
Originally, I bought that disk to backup some data, of the systems in the network, but I realized that all data on this system were hosted on a single 320GB 2,5" disk and it became clear to me, that, in absense of a proper backup strategy, I at least should provide some redundancy.


So I decided, once the disk was in place, that the whole system should move to a RAID1 over the two disks. Basically this is not that hard as it may seem at a first glance, but I had some problems due to a new sector size in some recent harddisks, which is called Advanced Format.


But lets begin from the start. The basic idea of such a migration is:
  1. Install mdadm with apt-get. Make sure to answer 'all' to the question which devices need to be activated in order to boot the system.
  2. Partition the new disk (almost) identical.
    Because the new drive is somewhat bigger that wouldn't make sense, but at least the two partitions which should be mirrored on the second disk, need to be identical.
    Usually this is achieved easily by using
    sfdisk -d /dev/sda sfdisk /dev/second/sdb

    In this case, it wasn't that easy. But I will come to that in a minute.
  3. Change the type of the partitions to 'FD' (Linux RAID autodetect) with fdisk
  4. Erase evidence of an eventual old RAID from the partitions, which is probably pointless on a brand-new disk, but we want to be sure:
    mdadm --zero-superblock /dev/sdb1
    mdadm --zero-superblock /dev/sdb2
  5. Create two DEGRADED raid1 arrays from the partitions:
    mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 missing
    mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdb2 missing
  6. Create filesystem on the first raid device, which will become /boot.
  7. Mount that filesystem somewhere temporary and move the contents of /boot to it:
    mount /dev/md0 /mnt/somewhere
  8. Unmount /boot, edit fstab to mount /boot from /dev/md0 and re-mount /boot (from md0)
  9. Create mdadm configuration with mdadm and append it to /etc/mdadm/mdadm.conf:
    mdadm --examine --scan >> /etc/mdadm/mdadm.conf
  10. Update the initramfs and grub (no manual modification needed with grub2 on my system)
    and install grub into the MBR of the second disk.

    update-initramfs -u
    update-grub
    grub-install /dev/sdb
  11. The first point to pray: Reboot the system to verify it can boot from the new /boot.
  12. Create a physical volume on /dev/md1:
    pvcreate /dev/md1
  13. Extend the volume group to contain that device:
    vgextend <volgroup_name> /dev/md1
  14. Move the whole volume group physically from the first disk to the degraded RAID:
    vgmove <volgroup_name> /dev/md1

    (Wait for it to complete... takes some time ;)
  15. Reduce first disk from the VG:
    vgreduce <volgroup_name> /dev/sda2
  16. Prepare it for addition to the RAID (see step 3 and 4) and add it:
    mdadm --add /dev/md0 /dev/sda1
    mdadm --add /dev/md1 /dev/sda2
  17. Hooray! Watch into /proc/mdstat. You should see that the RAID is recovering.
  18. When recovery is finished pray another time and hope that system is still booting with it running from the RAID entirely. If it does: Finished :-)

Now to the problem with the advanced format:
There is some action taking place with the hardware vendors to move to a new sector size. Physically my new device has a size of 4096 bytes per sector. Somewhat different to the 512 bytes disks used to have the last decade.


Logically it still has 512 bytes per sector. As far as I understand this is achieved by placing 8 logical sectors into one physical sector, so when partitioning a new disk the alignment of the disk has to be so that partitions start in a sector which is a multiple of 8.

That, obviously, wasn't the case with the old partitioning on my first disk. So I had to manually create partitions by specifying start points manually and making sure they are dividable by 8.

Otherwise fdisk would complain about the layout on the disk.
This does not work with cfdisk, because it does not accept manual alignment parameters and unfortunately the partitions it creates do have a wrong alignment. So good old fdisk and some calculations how many sectors are needed and where to start, to the rescue.

So the layout is now:
Device Boot Start End Blocks Id System
/dev/sdb1 2048 291154 144553+ fd Linux raid autodetect
/dev/sdb2 291160 625139334 312424087+ fd Linux raid autodetect


Benjamin Mako Hill: Software Freedom Day Boston 2011

This year, Software Freedom Day in Boston is being organized by Asheesh and Deb and OpenHatch which means a focus on increasing involvement in free software communities. If you are all interested in getting involved in the free software community in any way and at any level -- or interested in hearing about how that might happen someday -- this is a great event to attend. For my part, I'll be giving a short talk on getting involved in Debian. The event will be held on Saturday, September 17 at Cambridge College -- between Harvard and Central squares -- with an after party at Tommy Doyle's in Harvard.

16 September 2011

John Goerzen: Mexico Part 5: Food, Restaurants, and Dueling Karaoke Guys

The fifth in a series; see also parts 1, 2, 3, and 4. One piece of advice we got in Mexico went like this: the nicer a place looks, the worse the food and prices will be. Roadside taco stands will be great, and nice-looking restaurants not so much. That seemed to be accurate. We only tried one real nice-looking restaurant and it was very good (though pricy), but it may have been sort of an exception. But perhaps the most interesting bit about eating in Mexico wasn t the food. It s the adventure. We ate one day at Guadalajara s San Juan de Dios market. In that huge labyrinth somewhere was a set of restaurants. They d tend to have a small cooking area, usually just open, and a few tables. We chose one. And at this point, I have to take brief detour and explain something. There are a lot of people in Mexico that do things for tips, and quite often without being asked. Some other examples might be washing a car s windshield at a stoplight. So anyhow, we had ordered our food, and before long, a guy wanders down the aisle and plonks down a boombox. And turns it on. And then he pulled out a microphone, which we quite soon realized was connected to the boombox. (I guess making it more of a karaoke box.) Anyhow, he started singing a song decently and seemed to be enjoying it. About 45 seconds into it, a competing boombox man plonked down a competing boombox 25 feet away, turned it on, and yes, you guessed it pulled out a microphone and started singing a different song. Worse than the first person but louder. Eventually the boombox people left and our lunchtime conversation could resume. But pretty soon a drum guy showed up. He had a bunch of drums on a strap so he could just walk around and play them. He apparently decided that an excellent place to play them would be directly behind my head. I did not entirely agree with his decision, but hey, it beat the competing karaoke guys. Eventually the drum guy left, and somehow between the time I looked down to get out money to pay our bill and the time I had it counted out, a clown had shown up and made several balloon animals for our boys. I tipped him, we paid, and then headed on. You might think from this story that this would be an annoying series of events. And honestly, if it had happened in a big mall in the USA, it would probably have been both annoying and creepy. But really I enjoyed it. The fact that dueling karaoke happened, despite sounding really awful, was pretty funny and really seeing this whole parade of people was interesting too. It made American restaurants seem a little boring. You always know what s going to happen here (and if something surprising does happen, the place probably gets a bad review on Yelp.) Interesting things sometimes happen at mealtimes in Mexico and I like it that way. I had a torta ahogada (drowned sandwich) at that restaurant. And at this point, another brief aside. I m the kind of person that can go to an average American restaurant, see items on the menu helpfully indicated as spicy, order one, and genuinely wonder if other people would find them spicy, because I either don t notice spiciness at all, or maybe notice a tiny bit if I concentrate really hard. Others, meanwhile, might take a bite and lunge for the water. Having said that, I know people that lived in Thailand for awhile and I have nowhere near their tolerance for spiciness. So, having been in Mexico a whole 24 hours or so, I decided not to follow Jonathan s wise lead in ordering a torta with the spicy sauce on the side. I figured I hadn t had anything spicy yet, so maybe this was would be nice and mild for me. Via Jonathan s translation, I ordered it with the spicy sauce. I believe the phrase I heard him use was con chile . The waitress looked at me, gave me an amused the American is ordering it con chile? Hahaha . sort of smile, and went off. Pretty soon our food arrived. (The food always seems to arrive pretty soon in Mexico, by the way.) Oliver was having a bit of a culture shock that day, and mostly refusing to eat (once hunger got the best of him later, he really enjoyed Mexican food.) But the rest of us dug in, including me. I enjoyed my torta. It was spicy, but not too bad. I took some big bites (it was, after all, a thick sandwich) and was really enjoying it. For about a minute. Slowest-acting spiciness I ve had in awhile. Then it hit me. Spiciness, and lots of it. I took a big gulp of my horchata (a creamy sweet rice drink that I found at many restaurants). That helped. A little. I really liked the torta and ate it, but it wouldn t surprise me if the waitress noticed how extremely quickly a drank my horchata Another interesting experience was in Guanajuato. It was raining as we walked towards the Guanajuato market. Their market was large and similar in concept to the Guadalajara one, though a lot smaller. The restaurants were all in a row, in a side of the building that was open to the outside. Most were on the ground level but it looked like a restaurant or two were upstairs. As we approached, all of a sudden people were yelling at us. First it was a guy on the second story, then pretty soon people at the restaurants on the first floor did so as well. They were yelling rapidly in Spanish, waving their menus around in the air. I m imagining they were naming foods they sold or reasons to eat there, but I don t know enough Spanish to know. As we walked down the long row of restaurants, the ones we left behind would quiet down in disgust and other hopeful restaurant owners would take up the yelling and waving cause. I imagine if we did some time-lapse videography and walked up and down that row, we could produce an effect not unlike the sound of a dot-matrix printer going back and forth on the page. Anyhow, we selected one of the quieter restaurants pretty much random. The others then quieted down until another person chanced to walk past at which point it would get loud again. The lunch there was good but I think I mainly will remember it for the selection process! On our way into Guanajuato, we stopped at a wonderful roadside taco place. In typical fashion, they had a large vertical pork thing (I don t know the proper word for it) from which they would carve off meat on the spot anytime someone ordered something with pastor. We found a table. And we ordered a few tacos and such. They were usually a few pesos each (working out to less than a dollar), small round things on a soft tortilla, with meat, cilantro, and onion on top. And typically delicious. They had very little in common with an American taco . We d often order a few, and if we wanted more, just order more. They were made quickly enough for that. Tacos were very similar from one restaurant to the next. My favorite flavors were pastor (pork), chorizo (sausage), and bistec (beef steak). A restaurant in Guadalajara sadly I ve forgotten its name, since we kept calling it the potato place had what I might call a Mexican version of the loaded baked potato, with a meat, queso (cheese), a delicious sauce with a flavor unlike anything I d had before, and some garnish. But really my favorite thing from that restaurant was their amazing juices. I am not much of a juice drinker normally, but in Mexico I went for them whenever they were offered. What passes as fruit juice in the USA has about as much resemblance to a real Mexican fruit juice as Taco Bell has to a real Mexican taco stand. (Very little, in case that wasn t crystal clear.) That particular restaurant offered three types of juices, which were, if I m remembering right, aguas, frescas, and jugos. I has a jugo verde (green juice) on the first visit there. It was good, but the one I can still remember was called, I think, the fresa fresca (fresh strawberry juice). And it was incredible. I m not sure how to describe it, other than real. One observation before I end. It seemed a common thread at some Mexican roadside taco stands to not have soap in their restrooms. Instead there would be a plastic cup holding I kid you not powder-form Tide laundry detergent. It was amusing anyhow. My hands left those places extremely soft and smelling like laundry. One of the last restaurants we visited on our trip was in Ajijic, near the Chapala lake. It was actually right on the lake and served seafood. This was the only restaurant with prices as high as I d be used to in the United States. I ordered a stew served in a stone bowl. It came out sizzling, and since the very thick stone bowl retains heat well, it kept sizzling the entire time I was eating. It was excellent as usual. Coming up in part 6: some thoughts on returning to the United States, our decision to visit, communication, and tips for anyone else considering a first visit to Mexico.

14 September 2011

John Goerzen: Mexico Part 4: Street Scenes and Architecture

The fourth in a series; see also parts 1, 2, and 3. This post is going to be more a photolog than a narrative, and I apologize in advance for it being a bit disjointed. I ve already touched on these themes a bit in the other post, but now it s time to focus on them. Immediately after leaving the airport, it s quite clear that things are a little different. Trees are square. People ride around in the backs of pickups sometimes on top of piles of debris. Left turns are made in front of other lanes of traffic going the same way. But those are just the things obvious from the road. It s a lot of fun to enjoy the differences. First, the ubiquitous square trees. They look pretty, and are found all over. I also found carefully-manicured trees in cone shapes, more cylindrical shapes, etc. It seems that tree care is taken seriously in Mexico. It was also not uncommon to see the bottom few feet of a tree painted white. A park in Guanajuato had a whole bunch of trees carefully trimmed. And from up on the mountain, it still looked impressive (the green area behind the dome). Driving in Mexico was interesting for a lot of reasons. The highways there aren t quite as limited access as the freeways in the USA. It was quite common to see bicyclists, walkers, a mule, or some cattle ambling along the side of the road. Roadside taco stands don t require taking an exit. You just pull off the road because it s right there. Some sights were a bit surprising. Cattle in a pickup, with rope, for instance. Or cattle crossing the highway on the overpass. Street vendors were everywhere. Stop at a red light and someone might spring from the side of the road and suddenly start washing your windshield (expecting a tip); try to sell you flowers, juice, or bug zappers; or even throw business card-sized advertisements for adult websites into any open windows they can find. One night we saw an incredible fire juggler. I would have tipped him well but he was too far away to do so before the light turned green. Mexico s history stretches back into prehistoric times, and we saw the Teuchitlan ruins at Guachimontones one day. It was truly a remarkable feeling to be able to walk down the middle of the ancient ball court, or to climb up one pyramid and see the other from it. It s not exactly architecture, but Jacob and Oliver sure enjoyed visiting the hot springs at Bosque de la Primavera. Jacob still remembers that where the steam is, the water is 200 degrees, and we CAN T TOUCH IT THERE! Back in Guadalajara, here s a photo from the inside of the grand old cathedral. Compared to the cathedrals we saw in Europe, this was of a similar general size and design, and perhaps only slightly newer. But one big difference: worshipers outnumbered tourists at every Mexican cathedral I saw, whether in the center of Guadalajara or at a rainy intersection in Guanajuato or a plaza in Tlaquepaque. It made them feel more alive, and perhaps more sacred as well. One surprise was seeing people sitting on the steps of the cathedral in downtown Guadalajara selling trinkets such as beads. I think the only other place I had seen something like that was in New Orleans. All of Guadalajara s Centro was beautiful. Much of it survives from colonial days; I think a person could spend days exploring its museums and buildings. Way too many of my 900 photos were taken in Centro to post on the blog, but just for flavor, here s one of the less than historic scenes. Yes, that is a bus shaped like a tequila bottle. Fountains were beautiful and common across Mexico. A few of them were easily reachable by boys, and ours sure loved those. There was a lot of public art, including this interesting chair/skeleton/I m not sure what it is: And, just for good measure while walking around Centro, they tossed in an apparent Redundant Array of Inexpensive Typewriters. I don t know what they were doing, but there were about a dozen guys sitting out in the sun typing on their manual typewriters on their identical tables. And who can leave Guadalajara without seeing one of North America s most impressive traffic circles. I ve got to hand it to the Mexicans for making something that is normally really boring into an interesting work of art. Over in Guanajuato, a lot of driving takes place in the city s vast underground tunnel system. Here s a scene emerging from one of them. Guanajuato was already getting decorated for Mexican independence day festivities (Sept. 15-16) while we were there. Here s a typical Guanajuato street scene. Many of the streets were closed to traffic and perhaps not wide enough to handle vehicles anyway. Those streets had a wonderful peaceful and slow feel to them. I feel that I ve barely done the trip justice with this post. The feelings of walking down a beautiful Guanajuato street, or stepping into a Spanish cathedral, or even seeing a bunch of guys with typewriters, just can t be replicated. It s brimming with history and character, and shouldn t be missed.

13 September 2011

Vincent Sanders: Electricity is really just organized lightning.

I have recently been working on a project that requires a 12V supply. Ordinarily this is no problem my selection of bench supplies are generally more than a match for anything I throw at them.

My TS3022S Bench SupplyThis project however needed a little more "oomph" than usual, specifically 200W more. Funnily enough a precision variable output bench supply capable of supplying 20A are rare and *very* expensive beasties.

So we turn to a fixed output supply, after all I will want to run my project without hogging my bench supplies anyway. These can be bought from various electronics suppliers like Farnell from around the 50 mark and Chinese imports from Ebay sellers start around the 20 mark.

All very well and good but that is money I was not planning on spending and possibly a month of waiting for an already badly delayed project. So I decided to Convert an old ATX PSU into a 12V source. This is not a new idea and a quick search revealed many suitable guides online. I had a quick skim, decided I did understand the general idea and ploughed ahead.

Wikipedia has a very useful page on the ATX standard complete with pinout diagrams and colour codes. The pile of grey box ATX supplies available on my shelf was examined and one was helpfully labelled with a sticker proclaiming 22A@12V and we had a winner.

Opening the case of the donor 450W CIT branded supply revealed a mostly empty enclosure with the usual basic switching arrangement. I removed most of the wire loom aside from two of each output voltage (3.3V, 5V and 12V i figured the other voltages might be useful in future) and three commons, the 3.3V and 5V sense lines were also kept. Each of these pairs were cut to length and leads were wired to 4mm sockets.

The "PWR_EN" line was wired via a toggle switch to ground so the output can be switched on and off easily. The 5V standby and a 5V output line were wired to a green/red bi-colour LED (via 270 current limit resistors) to give indication that mains is present and when the output is on.

Holes were drilled for four 4mm sockets an indicator LED and a switch. The connectors and switches were all mounted in the PSU casework. I plugged it all in, put an 8.2 load resistor on the 5V line with an ammeter in line and a voltmeter across the 12V rail.

ATX bench power supply I turned the mains on and the LED lit up green (5V standby worked) and when I flicked the output switch the LED turned orange, the 12V line went to 12V and the expected 0.6A flowed through the load resistor.

Basically, Success!

I have since loaded the supply up to the 200W operating load and nothing unexpected has happened so I am happy. Seems converting an ATX PSU is a perfectly good way of getting a 200W 12V supply and I can recommend it for anyone as cheap as me willing to put an hour or so into such a project.

Christian Perrier: 10 years being Debian Developer - part 4: NM process

So, this story begins in January 2001, when I applied in the Debian New Maintainer Process. I wanted to maintain Geneweb, which I was using to publish my genealogy research results. And I wanted to keep it up-to-date while upstream was doing a quite fast development. Moreover, I quickly noticed that Geneweb had big trouble in respecting the way data is organized on a Debian system, and respect the FHS. How to have weveral users able to publish their data on the same server without compromizing the overall system security, etc. Upstream development didn't really care about that. Daniel de Rauglaudre is an excellent genealogist and developer but he was not interested in making Geneweb clean "the Debian way". For instance, at that time, it wasn't easy to setup a server, where genealogy databases could be published without having to manually launch a daemon in user mode at each reboot. Arranging this was indeed my first contribution. I contributed a few patches to make it easier to turn Geneweb into something FHS-compliant...and I developed init scripts and an organization allowing one to have shared databases after system reboots. That was rapidly a great introduction to Debian maintainer scripts and even security-related challenges. And all this....because I needed it. Basically, in 2001, the geneweb package adopted the organization it still has in Debian and Ubuntu, 10 years later: While doing so, I was going through the New Maintainer Process. The Debian developer who had signed my key was Sam Tardieu, of of those longstanding "Freenix" dudes I was sometimes hanging around. I don't remember who did write my advocacy. Michael Mattice was my first Application Manager. Interestingly, here are the three first questions he asked my as part of the Policy and Procedures check : Mike actually asked the questions on February 17th and I probably answered immediately. Unfortunately it seems that I lost my answers. Maybe they are in some archive somewhere. Would be interesting to see how I ended giving an answer about the TeX exception.:-) Immediately, Mike moved to the Tasks and Skills step and checked my geneweb and lifelines packages (I already prepared a package for that other genealogy software). From my records, he found a few lintian warnings, which I probably made my best to fix, then later on, on March 12th, he answered me that thigns were OK and he was marking the T&S step as passed...handing things over to the NM database, asking me what username I wanted on debian.org (obviously, you know what I answered to this!). Then nothing happened..:-) I was not that impatient, but finally, around June 27th, I politely mailed NM Frontdesk (I guess) to get news...and I got a very quick answer by Martin Michlmayr. Martin was indeed worried about the situation (he told me I was on his radar....fun as, many years later, as many DDs, Martin probably went on mine because of pending localization work..:-)) Finally, Martin decided he was not entirely happy with the (maybe too short) processing for P&P and proposed me (very politely, you know how Martin is!) to do it again. So, some more questions...:-) Here, I have the answers. Really fun ro read. Hang on, pure bubulle style, very verbose! What a bunch of questions..:-). Reading my answers ten years after, I see that some answers are somehow naive but, after all, they're not so bad. I really remember spending hours in writing them (my English was somehow more shaky at that time) The last question was: "Please tell me about about yourself and what you intend to do for Debian". Here, Martin opened a big can of worms..:-) So, you'll learn about this in the next episode..:-)

12 September 2011

Wouter Verhelst: "Yoe" or "wouter"?

For the longest time, I went by the nick "Yoe" when I went on IRC. When I chose this nick for myself, I was sixteen, and the Internet computer at the local public library had an IRC client installed that I tried out. Obviously, my first choice of nick -- "wouter", a fairly common first name in Dutch-speaking regions -- was already chosen, so I had to come up with something else. If I'd realized that I'd still be using this nick today, seventeen years later, I might have chosen that nick with a bit more care... Over the years, some people have asked me what the nickname meant, where it came from. I've never really told anyone the entire story, mainly because it was somewhat embarrasing. But hey, we've all been 16, right? In the early nineties, there was a show on TV called "Het huis van wantrouwen". On this show, they had a running joke involving a short clip of Belgian members of parliament informally voting on a subject by raising hands (literally); the joke was that this clip was dubbed with a bunch of people saying "Yoe, de manne" -- which is dutch for something along the lines of "Yo, guys". As a young geeky teenager, I found this joke way more funny than it actually was; and as such, a few years later, during one of the more weird episodes of my life, I started pranking people by sneaking up to them from behind and shouting "Yoe, de manne" right next to their ears. I even managed to do this to one of our teachers who was walking on the playground. This was just around the time when I first tried out this "IRC" thing. It didn't take me too long to realize I was the only person who thought this "joke" was funny; but somehow the nick stuck, and here we are, 17 years later, I'm still using it. Or, well, I was. As of today, if you want to catch me on IRC, you'll find me going by "wouter" instead of "Yoe". Of course someone had registered that nick several years ago, but then he hadn't been using it in years, and so after a quick request to the OFTC admins they made it available for me. Whee.

John Goerzen: Mexico Part 2: Lodging & Family

As I wrote in part 1, my family and I were in Mexico recently. Today I ll write about the places we stayed. We spent most of the time in a room we rented in a private home in Guadalajara. My friend Jonathan had found it for us, and it was not too far from his home. The owner was a grandmother, and across the courtyard was more family, including a granddaughter close in age to our boys. They enjoyed playing together. It was really a perfect arrangement for us. There s no better way to be a part of local life when traveling than to stay in someone s house. Here s our bedroom: Notice the glass slats in the window it s a nifty, though not airtight, alternative to our regular windows. More on that later. More of the inside: We had a bit of a language barrier while in Mexico, though never anything significant. My Spanish vocabulary started with almost nothing and I reached maybe a few dozen words by the time we left. Terah knew some Spanish from high school and college, and my friend was fluent. Our hostess also knew a little English. But we all communicated well enough. Terah or Jonathan would help translate when needed. As I d seen before, the children would say things to each other, but never seemed to be bothered that their playmates didn t understand what was being said. They just had a great time anyhow. On Sunday afternoon, when we came back from our activities, there was a buzz of activity. Children everywhere outside, running and playing. Adults too, chatting. We didn t know exactly what was happening, but sent Jacob and Oliver out to play anyhow (which they were eager to do). The yard was enclosed by a wall, so children could pretty much run around without lots of supervision. Eventually they were invited to have some birthday cake (ah ha!) it was one of the children s birthday. Jacob and Oliver actually were served the first two pieces of cake (as the amigos ). Everyone seemed so friendly, warm, and welcoming. Each morning started with breakfast at the house, followed by a scene like this: That s Jacob and Oliver, looking to see if Jonathan had arrived for the day yet. I often noticed in Mexico that I was unsure if I was inside or outside. Here in Kansas, we can have a string of summer days that each exceed 110F (44C) or a few weeks in winter that never get above 15F (-9C), even in daytime. And then we have some pleasant days like right now, too or rain blowing sideways at 60MPH. In general, we spend a lot of effort keeping the outside, well, out. It is quite clear that this isn t a problem in the Guadalajara area. Some restaurants could have been described as buildings with large, open windows so you feel a lot of breeze while inside. Or perhaps as a simple shade roof with a few supports on the edges, so you re never really inside at all (sort of like going under a small shade tent outdoors). To the extent that windows could close, many of them couldn t be made airtight. It was clear that in that area, people spend more energy finding ways to invite the outdoors in rather than to keep it out, thanks to the year-round moderate climate. Perhaps the best example of this surprised me one evening. We were arriving in Guanajuato, an old silver mining town in the mountains, and were going to spend the night at Hotel Socav n, which had been recommended to us by a local friend of Jonathan s. From the street, the hotel looked tiny. But walk in, and you get in this old-looking (and feeling) entry tunnel: That s the front desk, apparently cut out of it there on the right. We asked to see some of the rooms before buying apparently a normal request around there. The innkeeper agreed, and gave us keys and directions to find them on the third floor. It included going up 2 flights of stairs, passing through a courtyard, and going up another flight. It was after dark, and the hotel was dimly lit something I was fine with. I thought we were stepping out into a beautiful atrium with some potted plants in the center of the building something fairly common in some nicer hotels. Until I felt rain on my head. Then I realized that the courtyard, which began two floors up from the street, was open to the sky. Beautiful! Here was the view from out room door: And down the hall : After getting home, a Google happened to turn up some reviews of this hotel. I was so annoyed at what some people wrote! One person gave them only 3 stars because they didn t have air conditioning, had poor water pressure, and lots of steps . Someone else complained of the dark entry tunnel something I couldn t help but smiling about the moment I entered. My review, which should hopefully get posted soon, is certainly different. I gave them 5 stars, because if I wanted a Super 8 with generic fluorescent lighting and the same layout as thousands of other hotels, I would have gone to Nebraska instead of Mexico. Most homes and local hotels in the region don t have air conditioning because they don t need it, and that s just how water pressure is in Mexico (due to needing to pump it from municipal supplies to private storage tanks overhead). And who doesn t appreciate entering a hotel through a brick tunnel? Ah, sigh This should give you some idea of the kind of travel we like: part of the point of traveling is enjoying the differences from home, and I think it is a huge mistake to be annoyed at everything that is different. Enjoy the differences! Finally, here s a photo of the staircase in the home we stayed in, which I thought was fascinating:

11 September 2011

John Goerzen: Mexico Part 1: Genesis and Travel

My family and I (including our boys) are just back from a great visit to Mexico. It was my first time there, and also the first time our boys have been outside the USA. I ll be writing about all the fun stuff in the posts to come, so you ll have to bear with me on this one as I describe why we did something that appears to horrify a segment of Americans. About a year ago, I wrote a review of Travel as a Political Act by Rick Steves. Rick s point wasn t actually directly political in the conventional sense, but summed up as:
I ve taught people how to travel. I focus mostly on the logistics: finding the right hotel, avoiding long lines But that s not why we travel. We travel to have enlightening experiences, to meet inspirational people, to be stimulated, to learn, and to grow. Travel has taught me the fun in having my cultural furniture rearranged and my ethnocentric self-assuredness walloped.
He speaks of giving onesself permission to have a conversation with someone that doesn t speak a language you know, for instance. I got an email this spring from my longtime friend Jonathan Hall, who had moved to Mexico a couple of years ago. He invited my family and me to go to Mexico, generously offered to host us and show us around, and specifically mentioned my review of Rick Steves as a reason to do so, this was immediately intriguing. Due to his other plans, it was also something of a limited-time offer. Those of you that don t live in the USA may not know what sort of stories we get about Mexico over here. Most of them involve either illegal immigration to the USA or the Mexican drug war. Occasionally there is some sort of drug-related violence on the US side of the border as well, which always makes the news. There are a lot of people that have the perception that Mexico is a dangerous place to be. Terah even knows some people that grew up in Mexico and are too afraid to return. So we did some research, asked some questions, and it became pretty clear that yes, some parts of Mexico are dangerous, but many parts are really quite safe, and Jonathan had invited us to one of those. The reactions we got from Americans when we told them of our plan ranged from excitement that we would get to visit a beautiful country to concern and worry about our safety. Besides that, I knew almost no Spanish and Terah had a few high school and college classes years ago to go on. And, we knew that Mexico would, in many ways, be more different from the USA than Germany was. We concluded that this would get us out of our comfort zone in a significant way, have lots of great things to do, be a good experience for the boys, and something that we wouldn t do without Jonathan. So we bought the tickets for it! As you might notice, I m quite glad we did. I ll follow up with the details in the next stories, but for today I ll end with the story of getting there. It involves two cute boys, so of course it ll be interesting. Jacob is almost 5 and Oliver is 2, and neither one of them had flown for over a year. Despite leaving home at 5AM to drive to the airport (about an hour away), they were both awake and alert. Jacob was jabbering away the whole way there. He enjoyed the security process and found it interesting I had to explain that they were checking to make sure everyone was following the airplane rules. Then as we walked to our gate, he pressed his face against every window, looking at the parked airplanes in the dark. Oliver would see him doing this and run over to join him. With a bit of help, he noticed some airplanes had United Express painted on them, and others had American with an AA on them. He would talk about American AA and United Express airplanes for the rest of the week whenever he heard one in the sky. I sat between the boys on our flight to Dallas, and Terah was in the row ahead of us. This is how Jacob spent most of the flight. And Oliver, who had the aisle seat, enjoyed paging through the inflight magazines, safety brochure, wifi instruction card, and airsickness bag. The real highlight came at the Dallas/Ft. Worth (DFW) airport, where we had a connection. And the reason: we needed to take a train to get to our connecting gate. The DFW Skylink system really works very well but the boys cared most that it s a TRAIN. It was difficult to get them both away from it when we got off. They wanted to see it leave, stay and watch the next one come, etc. Jacob was only happy when he realized he could see the Skylink trains running high above the window at our departure gate for Guadalajara. The 2.5-hour flight to Guadalajara got a little long for them, but they managed OK. We struck up several conversations with friendly people that knew English as we waited in various lines. It seems to be something of a rarity for American families with young children and no Mexican heritage to visit Mexico. People went out of their way to be friendly and welcoming even the customs officials. It felt like a great start to the visit. One Mexican man who was chatting with me encouraged me to learn some Spanish. I said my wife knows some, and that I had tried to learn some German back when I took foreign language classes. Laughing, he said, Why would you do that? Not as an insult; it just genuinely didn t seem useful to him. I think they were happy and proud that someone wanted to visit Mexico and was excited about it! More to come.

10 September 2011

Biella Coleman: How to secure revolution in Silicon Valley

This is the best response to this (simplistic) claim about net access and revolt in Egypt featured in the New York Times: [m] In a widely circulated American Political Science Association conference paper, Yale scholar Navid Hassanpour argues that shutting down the internet made things difficult for sustaining a centralized revolutionary movement in Egypt. But, he adds, the shutdown actually encouraged the development of smaller revolutionary uprisings at local levels where the face-to-face interaction between activists was more intense and the mobilization of inactive lukewarm
[m] maybe we ll have some more fiber optic cable cuts in silicon valley and then the revolution will happen

9 September 2011

Rapha&#235;l Hertzog: People behind Debian: Enrico Zini, member of the New Maintainer Frontdesk

Enrico ZiniEven though Enrico is not smiling on this picture, he s one of the friendliest Debian person that I know. I always enjoy his presentations because he can t refrain from inserting jokes or other funny tricks. :-) That said he s serious too, there s lots of good stuff that he has developed over the years (starting with Debtags) and he has put a lot of effort in reforming the New Maintainer process. Read on to learn more about his various projects. Raphael: Who are you? Enrico: Hi, I m Enrico Zini, a DD from Italy. I m 35 and I work as a freelance Free Software developer. One of my historical roles in Debian is taking care of Debtags, but that is not all I do: my paid work led me to write and maintain some weather forecast related software in Debian, and recently I gained a Front Desk hat, and then a DAM hat. Raphael: How did you start contributing to Debian? Enrico: It was 2001, I was at uni, I was using Debian. At some point I wanted to learn packaging so I read through the whole Policy from top to bottom. Then I thought: why package only for myself? . There were many DDs at my uni, and it only seemed natural to me to join Debian as well. Evidently this was also natural for Zack [Note from editor: Stefano Zacchiroli], who had become DD 6 months earlier and didn t hesitate to advocate me. I found the Policy and the Developer s Reference to be very interesting things to read, and I welcomed my AM s questions as an excuse to learn more. I completely understand those people who have fun trying to answer all the questions in the NM templates while they wait for an AM. With my super DAM powers I can see that my AM report was submitted on October 16, 2001 by my AM Martin Michlmayr, and that James Troup created my account 9 days later, on October 25. Raphael: You have a special interest in the New Maintainer (NM) Process since you are a Debian Account Manager (DAM) and a member of the NM Frontdesk. Thanks to your work the process is much less academic/bureaucratic than it used to be. Can you remember us the main changes? Enrico: One of the first things I noticed when I become a Front Desk member is that there was a tendency to advocate people too early, thinking by the time they ll get an AM, they ll know enough . Unfortunately, this didn t always work, and once the real NM process started it would turn into a very long and demotivating experience both for the applicant and for the AM. So we tried raising the bar on advocacies, and that seems to have helped a lot. If people join NM when they are ready, it means that NM is quick and painless both for them and their AMs, who are therefore able to process more applicants. We also did a rather radical cleanup of the NM templates , which are a repository of questions that Application Managers can ask to their applicants. We realized that AMs were just sending the whole templates to their applicants, so we moved all non-essential questions to separate files, to drastically reduce the amounts of questions that are asked by default. Other improvements in the NM process came from other parts of Debian: nowadays there are lots of ways to learn and gradually gain experience and reputation inside the project before joining NM, which means that we get many candidates who we can process quickly. For example, packages can now be uploaded via sponsors, and the Mentors project helps new contributors to find sponsors and get their first packages reviewed. Then one can now become Debian Maintainer and take full responsibility of their own packages, gaining experience and reputation. The idea of working in teams also helped: big teams like the Perl, Python, KDE, Ocaml, Haskell teams (and many more) are excellent entry points for people who have something to package. But Debian is not just for packagers, and one could join teams like the Website team, the Press and Publicity team, the Events and Merchandise team or their local translation team. Becoming DDs the non-uploading way is not just for non-technical people: one could enjoy programming but not packaging. An interesting way to get involved in that way is to help writing or maintaining some of the many Debian services. Note that I m not suggesting this as a way to learn how to program, but as a way to get involved in Debian by writing code. Finally, we started to appreciate the importance of having people activities in Debian explicitly visible, which means that the more obviously good work one has done in Debian, the less questions we need to ask. Jan Dittberner s DDPortfolio is an excellent resource for AMs and Front Desk, and I m maintaining a service called minechangelogs that for people who have done lots of work in Debian is able to fully replace the Tasks&Skills parts of the NM process. Raphael: What are your plans for Debian Wheezy? Enrico: For Wheezy I hope to be able to streamline and simplify Debtags a bit more. At Debconf11 I had a conversation with FTP-masters on how to make some tags more official, and I now have to work a bit more on that. I d also like to considerably downsize the codebase behind the debtags package, now that its job is quite clear and I don t need to experiment with fancy features the way I did in the past. I have to say that I enjoy programming more than I enjoy packaging, so most of my plans in Debian are not tied to releases. For example, I d like to finish and deploy the new NM website codebase soon: it would mean to have a codebase that s much easier to maintain, and in which it s much easier to implement new features. I d also like to design a way to allow maintainers to review the tag submission to their own packages instead of having to wait for me or David Paleino to do a regular review of all the submissions. Finally, I d like to promote the usage of apt-xapian-index in more cutting-edge packaging applications. And to design a way to maintain up to date popcon information in one s local index. And improve and promote those services that I maintain, and I tend to often have ideas for new ones. Raphael: If you could spend all your time on Debian, what would you work on? Enrico: If I could spend all my time on Debian, I would do a lot of software development: I love doing software development, but most of my development energy is spent on my paid work. I guess I would start my all your time in Debian by taking better care of the things that I m already doing, and by promoting them better so that I wouldn t end up being the only person maintaining them. After that, however, I reckon that I do have a tendency of noticing new, interesting problems in need(?) of a solution, and I guess I would end up wildly experimenting new ideas in Debian much like a victorian mad scientist. Which reminds me that I most definitely need minions! Where can I find minions? Raphael: You re the author of the Debian Community Guidelines. I have always felt that this document should be more broadly advertised. For example by integrating it in the Developer s Reference. What do you think? Enrico: The DCG was really a collection of tips to improve one s online communication, based on ideas and feedback that I collected from pestering many experienced and well-respected people for some time. Like every repository of common sense, I think it should be widely promoted but never ever turned into law. It wouldn t be a bad idea to mention it in the Developer s Reference, or to package it as a separate file in the developers-reference package. The reasons I haven t actively been pushing for that to happen are two: there isn t much in the DCG that is specific to Debian, and I don t have the resources to do a proper job maintaining it. It d be great if somebody could take over its maintenance and make it become some proper, widespread, easy-to-quote online reference document, like one of those HOWTOs that all serious people have read at some point in their lives. Raphael: What s the biggest problem of Debian? Enrico: It s sometimes hard to get feedback or help if you work on something unusual. That is partly to be expected, and partly probably due to me not having yet learnt how to get people involved in what I do. Raphael: What motivates you to continue to contribute year after year? Enrico: Debian keeps evolving, so there is always something to learn. And Debian is real, so everything I do is constantly measured against reality. What more intellectual stimulation could one possibly want? Raphael: Is there someone in Debian that you admire for their contributions? Enrico: I don t think I could reasonably list everyone I admire in Debian: pretty much in every corner of the project there is someone, sometimes not very well known, who is putting a lot of Quality in what they do. Someone who decided that X should work well in Debian or that Debian should work well for Y or that Z is something Debian people can rely on and makes sure that it is so. Those are the people who make sure Debian is and will be not just a hobby, but a base upon which I can rely for my personal and working life.
Thank you to Enrico for the time spent answering my questions. I hope you enjoyed reading his answers as I did.

Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

5 comments Liked this article? Click here. My blog is Flattr-enabled.

8 September 2011

Dirk Eddelbuettel: Faster (recursive) function calls: Another quick Rcpp case study

There was another question recently on StackOverflow that I had meant to discuss in a follow-up post here. User deltanovember asked about slow recursive functions and used the very classic Fibonacci number as an example. To recap, Fibonacci number are defined with two initial values F(0) = 0 and F(1) = 1; thereafter the Fibonacci number F(n) is defined as the sum of the two preceding numbers: F(n) = F(n-2) + F(n-1). This leads to very straightforward implementations using recursion:
## R implementation of recursive Fibonacci sequence
fibR <- function(n)  
    if (n == 0) return(0)
    if (n == 1) return(1)
    return (fibR(n - 1) + fibR(n - 2))
 
Unfortunately, this elegant implementation which remain close to the abtract formulation of the recurrence algorithm performs very poorly in R as there is noticeable overhead in function calls which becomes dominant in a recursion. This lead to the original question on StackOverflow, and the accepted answer uses a trick presented by Pat Burns in his lovely R Inferno: rewrite the solution using a computer science trick called memoization:
fibonacci <- local( 
    memo <- c(1, 1, rep(NA, 100))
    f <- function(x)  
        if(x == 0) return(0)
        if(x < 0) return(NA)
        if(x > length(memo))
        stop("'x' too big for implementation")
        if(!is.na(memo[x])) return(memo[x])
        ans <- f(x-2) + f(x-1)
        memo[x] <<- ans
        ans
     
 )
That is a fair answer, and even more was suggested with a link to a terrific analysis calling the Fibonacci recurrence the worst algorithm in the world. That is also fair, but all the basic research into better algorithms exploiting some structure of the problem to advance performance (and of course understanding) is overlooking one crucial part: algorithm analysis is essentially independent of the language. So whatever improvements we obtain by thinking really hard about a problem are then available for other implementations too. So with a tip of the hat to the old Larry Wall quote about Lazyness, Impatience and Hubris, I would like to suggest what I consider a much simpler route to much better performance: recode it in C++ using both Rcpp (for the R/C++ integration) and inline for the on-the-fly compilation, linking and loading of C++ code into R.
## inline to compile, load and link the C++ code
require(inline)
## we need a pure C/C++ function as the generated function
## will have a random identifier at the C++ level preventing
## us from direct recursive calls
incltxt <- '
  int fibonacci(const int x)  
     if (x == 0) return(0);
     if (x == 1) return(1);
     return (fibonacci(x - 1)) + fibonacci(x - 2);
   '
## now use the snippet above as well as one argument conversion
## in as well as out to provide Fibonacci numbers via C++
fibRcpp <- cxxfunction(signature(xs="int"),
                       plugin="Rcpp",
                       incl=incltxt,
                       body
                       ='
   int x = Rcpp::as<int>(xs);
   return Rcpp::wrap( fibonacci(x) );
')
This single R function call cxxfunction() takes the code embedded in the arguments to the body variable (for the core function) and the incltxt variable for the helper function we need to call. This helper function is needed for the recursion as cxxfunction() will use an randomized internal identifier for the function called from R preventing us from calling this (unknown) indentifier. But the rest of the algorithm is simple, and as beautiful as the initial recurrence. Three lines, three statements, and three cases for F(0), F(1) and the general case F(n) solved by recursive calls. This also illustrates how easy it is to get an integer from R to C++ and back: the as and wrap simply do the right thing converting to and from the SEXP types used internally by the C API of R. A performance comparison of the basic R version fibR, its byte-compiled variant fibRC and and the C++ version fibRcpp shown above is very compelling. We have added a file fibonacci.r to the large and still growing set of examples included with Rcpp, and we can just execute that script with Rscript or (as here) r from the littler package:
edd@max:~/svn/rcpp/pkg/Rcpp/inst/examples/Misc$ r fibonacci.r
Loading required package: inline
Loading required package: methods
Loading required package: compiler
        test replications elapsed relative user.self sys.self
3 fibRcpp(N)            1   0.092   1.0000      0.09     0.00
2   fibRC(N)            1  61.480 668.2609     61.47     0.00
1    fibR(N)            1  61.877 672.5761     61.83     0.02
edd@max:~/svn/rcpp/pkg/Rcpp/inst/examples/Misc$ 
So the recursion for the original argument of N=35 takes just over a minute at about 61.5 and 61.9 seconds, respectively, for the R version and its byte-compiled variant (as per the column titled elapsed). So byte-compilation essentially offers no help for the bottleneck of slow function calls. The C++ versions relying on Rcpp which created in a few lines of code and a single call to cxxfunction however takes just 92 milliseconds---or a relative gain of well over six-hundred times. That provides another nice demonstration of what Rcpp can do. Improved algorithms for well-understood problems are surely one way to accelerate solutions. But there are (many ?) times when we do not have the luxury of being able to think through to a new and improved approach. Or worse, such an approach may even introduce new errors or inaccurracies if we get it wrong on a first try. With Rcpp, we are able to the express the problem as written in its original statement: a simple recursion. The gain relative to a slow R implementation is noteworthy---and could of course be improved further if we really needed to by relying on better algorithms like memoization. But for day to day tasks, I gladly take speedups of (up to) a few hundred times thanks to Rcpp without having to do hard algorithmic work. Before closing, a quick reminder that I will be giving two classes on Rcpp in a few weeks. These will be in New York on September 24, and San Franciso on October 8, see this blog post as well as this page at Revolution Analytics (who are a co-organiser of the classes) for details and registration information.

David Welton: Zero to Profitable - LiberWriter Lessons Learned

Here's how I have created the beginnings of a profitable Kindle formatting and conversion service, LiberWriter, from scratch. Last Winter, I read one of the best business books I have read in a long time, Rob Walling's Start Small, Stay Small: <iframe frameborder="0" marginheight="0" marginwidth="0" scrolling="no" src="http://rcm.amazon.com/e/cm?lt1=_blank&amp;bc1=000000&amp;IS2=1&amp;bg1=FFFFFF&amp;fc1=000000&amp;lc1=0000FF&amp;t=dedasys-20&amp;o=1&amp;p=8&amp;l=as4&amp;m=amazon&amp;f=ifr&amp;ref=ss_til&amp;asins=B003YH9MMI" style="width: 120px; height: 240px;"></iframe> Rather than being a "big idea" book that includes one big idea and lots of stories to support it, it's a very practical book on how to go about creating a small online business. I've always been someone who loves to build things, from various open source projects, to diverse web sites that people have found valuable. However, I am not a "business guy". I'm not that good at making money, so the simple approach in the book is really better for people like me: make something and sell it. Advertising can certainly work, but it's so much harder to figure out the numbers compared with a product, where you can compare the cost of providing the product to the price people will pay for it. And that's just one thing; the book has tons of ideas about practical ways of going about putting together a profitable small online business. It's not a "get rich quick" book - part of the idea is that if you are looking at a niche that is fairly small, it will simply not be of interest to larger companies, so you aren't going to get outspent and outcompeted by them. In any event, though, at about the same time I read the book, I was poking around looking at Amazon's KDP, which allows you to self publish for the Kindle, and noticing how difficult the process is. The native format of Amazon's books is HTML with several XML files thrown in for good measure. Now, for someone technical, that's certainly not an insurmountable obstacle, but for many authors, it's a very unpleasant prospect. Granted, you can also manipulate a Word file to make it suitable for the KDP, but even that is something that many people dont' find to be a productive use of their time. They're authors, and they want to write. Having spotted a niche that looked interesting - I love to read, and I like the idea of helping out authors - I discarded the book idea I was looking at myself and did some writing of my own: source code for a project that would automate the process of converting and formatting Kindle books. The core of the system was a fun evenings/weekends project, and soon it was able to produce pretty good books. After "inflicting" the beta version of the system on several people, such as Rich Bowen (thanks again, Rich!), I got it beat into shape and started accepting the first paying customers. It's very exciting to have those first few people pay for something you made - it's a great feeling! Initially, my thought was to sell it as a way to write books directly for the Kindle, however, it soon became evident that 99% of people have already written their books with Microsoft Word, so what they were really interested in was a conversion and formatting system that would take a Word document, poke and prod, stitch and stretch it some, and output a nice looking book for the Kindle. One of the lessons from Rob's book was that stuff doesn't have to be perfect if it works. Initially, I had people putting their word documents into LiberWriter not with the smooth(ish) upload process we have now, but by cutting and pasting in their documents! As a computer guy, doing things in a hacky way is sometimes very unpleasant, but you know what? It worked, and it was easy for people to handle, so I ran with it until a better system was in place. Writing code was the easy part, and definitely within my "comfort zone". Some of the other things I've had to do have been much more difficult, and as a consequence, more of a learning experience. For instance, I realized early on that the process would never be 100% automatic, so I'd have to get some people to work on the books, with the idea being to provide them tools to make the process as simple, quick, and easy as possible. I'm always on the lookout for new things to script within the system. However, need people it does, and finding them and engaging them has been a difficult process. As per the book's suggestion, I've primarily used oDesk to find contractors, and have grardually begun to find some people who work really well. It has taken time, and has not been easy, though. I'm not an introvert, but in some ways I'm happiest when doing my own thing. Being the 'boss' is new. One of the tricky aspects of oDesk is that pretty much any job posting gets tons of responses, some of them from people who appear to have barely read the posting. One trick I learned is to request that the potential contractor creates an account with LiberWriter. Very simple and not a waste of time, and yet a majority of applicants ignored the request. Through most of the spring and summer, LiberWriter remained something strictly for the evenings and weekends, as I was busy with client work during the days, and with my daughter during the afternoons, but as August rolled around, I found some more time on my hands, and really started investing more energy in LiberWriter. It has been very rewarding to see a corresponding increase in the number of customers. We (at this point, between myself and the contractors, LiberWriter is a group, and no longer "I") also raised prices - another valuable piece of advice from the book, and... pretty much anyone who writes for people jumping into online businesses. As tech people, we often tend to aim low in terms of prices. Indeed, soonish, prices will likely go up again. Another challenge has been turning it as much as possible into a "productized service". Since Kindle books are fairly simple in what you can do with them, as long as you don't get too fancy, it's possible to automate *most* things, but not everything, so it's not just a product that people buy and it works and that's that; there's a human element involved, which means more communication, and more potential for things to go off the rails. I've been investing time in automating the workflow of the system, but have a lot more to do on that front. Also, one of the things our customers love about the service is being able to deal with "real people", something that is more difficult at Amazon. That's good for us in one way, but in the future, it will be critical to document things and make them as simple as possible to avoid generating unneccessary support requests, as it can easily take up a lot of time. Indeed, because LiberWriter does attract customers who are often not all that technical, making the user experience as easy and straightforward for them as possible (and then some) is extremely important. It can be exasperating once in a while, but I do my best to channel any frustrated energy on my part into making the site ever easier and ever more straightforward. In other words, instead of "how the heck do they not get that?!" it's "what can I do to make sure the tools and information they need are even easier for them to get?". There's lots to do on that front, but on the whole it's actually part of the project that I've grown to like a lot. Rather than dealing with other techies like myself, our customers are people who really benefit from what we offer because we can save them so much time and frustration. Marketing the system is fun, but another area where I have a lot to learn: I'm not a natural when it comes to marketing or selling. One thing that has worked well so far is finding forums where your target users hang out, and then.... spam them as often as possible? No way! Be yourself, and provide genuine help and value to people in the forums. By being on the level, and not trying to sell! sell! sell! people are friendlier and more receptive to your message. It can be time consuming, but I suppose it's the sort of thing where taking the wrong short cut could be very detrimental over the long haul. Without disclosing too much, I don't think I'll ever get rich from LiberWriter, but it is incredibly satisfying to have created and marketed a product of my own that goes beyond just selling my own time. LiberWriter doesn't bring in as much as consulting, yet, but it feels like much more of an investment, something that will have some momentum of its own, and of course it has been an incredible learning experience. I like programming, but being outside of my "comfort zone", doing stuff that's new and challenging is also a very positive part of the project. If things continue to grow the way they have been, I could envision depending on LiberWriter as my primary source of income in the very near future. That's a bit of a jump into the unknown in some ways, but I'd love it. I have worked very hard on it lately, and one of the things I am enjoying is always having new ideas about how to improve every aspect of the system, from customer service to the mechanics of the conversion process, to tracking the finances. In some ways it's tiring because I'm thinking about it *all* the time, but it's that "good kind of tired". It's been a tumultous summer, and in the past few weeks we (well, my wife much more than me!) had a new baby boy, and bought a house here in Padova. To keep things interesting for the future, I applied to the Startup Chile program with LiberWriter. I suspect they're more interested in startups with high growth potential, but I feel strongly that the approach outlined in Rob's book is one that people outside of Silicon Valley, without easy access to venture capital and the huge talent pool that area has, should consider, rather than trying to copy what makes Silicon Valley work like it does. Whether I get accepted or not, LiberWriter will continue to grow, and, I sincerely hope, thrive. A big thanks to all our customers, and to my wife Ilenia, and children, Helen and Daniel for tollerating all the time I spend with the computer. I wrestled a bit with writing this post, as I don't want to "jinx things", and I've only just got a small taste of "success", but I've always enjoyed reading about how "ordinary people" have managed to create their own small businesses, people like those behind Balsamiq, "Bingo Card Creator", and the like, and I figure that writing up my own experiences is a way of "paying it forward".

Next.

Previous.