I'll start here a small series of posts about ganeti, xen and puppet. For my work I run few servers sitting on xen and it has always been a bit of a pain to create a new instance and keep it up to date. Up to now I've used the excellent xen-create-image tool to create my VMs, but I wanted to try something new and more sexy... Last week I finally found some time to learn (and a spare box to run my experiments) how to use ganeti. Ganeti is the only tool I tried out, but it seems to fit the bill for my use and it seems polished and mature project to me... Moreover I've seen a presentation about it in every FLOSS conference I've attended in the last few years and I thought it was time to give it a try. Installing and configuring ganeti is fairly easy, there is a lot of documentation available and this post is not going to be about installing it, but rather how to create a new bare instance with ganeti-deboostrap-instance. There is also a way to create a new instance from an image, but I didn't go that way yet. This first post is about the first problem I've encountered, that is, how to automatically assign a network address and a name to each new instance created by gnt-instance add. Since all my instances should be able to communicate together on the same subnet, I've decided to configure xen to create a NATted private network and add each new instance to this network. The first step is to create an interface in /etc/network/interfaces . This is the standard debian way but since xen uses a different naming convention (here I'm using ganeti naming convention xenbr0 vs xen-br0), I need to convince tell xen what I intend to do by adding these lines in /etc/xen/xend.config : Next I have to connect my real network interface to the private network using few iptables rules in /etc/rc.local (probably there is a better place to do this...): The xen setup is complete and every new image should have a vif connected to the subbet 10.0.0.0. The xen setup corresponds to the physical wiring of the network. The next step is to configure each instance so to allow them to communicate on this subnet. Since I build my VMs using ganeti-debootstrap-instance, and by default debootstrap does not configure the network, we need to add a new hook in the directory /etc/ganeti/instance-debootstrap/hooks. This hook will do two things. First it will configure the interfaces of the new instance to get configured using dhcp, second, it will add an entry to the dnsmasq configuration file to make this instance known to the world. This basically boils down to add a file in /etc/dnsmasq.d/ with the mac address of the new instance and its designated name. Dnsmasq will then provide an ip address for this instance and add it to the dns. dhcp-host=aa:00:00:24:6c:8a,node1 Configuring dnsmasq is pretty easy as well. First I want it to answer dhcp queries only on the internal network, second I want to configure my clients passing 10.0.0.1 as nameserver and gataway. You can just add the following lines in /etc/dnsmasq.d/general to get it going. To create your new instance you can just run the following command : If you are running your dom0 on debian squeeze before running this command you should configure ganeti to pass the right xen parameters to the newly created instance : I use --no-ip-check and --no-name-check to skip ip and dns check performed by ganeti and to avoid a sort of chicken-egg problem, where the name and address of this new instance is yet unknown to dnsmasq and that node1 is the name that will be used by the hook to add an entry in the dnsmasq configuration. debootstrap+unstable is a variant of the default configuration and you need to add it to the list of variants used by ganeti-deboostrap-instance. That should be it. The new instance should come up with a dynamically assigned ip address, able to talk to the outside world and automatically known by all the other machine on the subnet via dns. Next post will be about how to add a swap hook for ganete-debootstrap-istance.

I have a Seagate GoFlex Net with two 2TB harddrives attached to it via SATA. The device itself is connected to my PC via its Gigabit Ethernet connection. It houses a Marvell Kirkwood at 1.2GHz and 128MB. I am booting Debian from a USB stick connected to its USB 2.0 port. The specs are pretty neat so I planned it as my NAS with 4TB of storage being attached to it. The most common use case is the transfer of big files (1-10 GB) between my laptop and the device. Now what are the common ways to achieve this? scp:

scp /local/path user@goflex:/remote/path

rsync:

rsync -Ph /local/path user@goflex:/remote/path

sshfs:

sshfs -o user@goflex:/remote/path /mnt
cp /local/path /mnt

ssh:

ssh user@goflex "cat > /remote/path" < /local/path

I then did some benchmarks to see how they perform: scp: 5.90 MB/s rsync: 5.16 MB/s sshfs: 5.05 MB/s ssh: 5.42 MB/s Since they all use ssh for transmission, the similarity of the result does not come as a surprise and 5.90 MB/s are also not too shabby for a plain scp. It means that I can transfer 1 GB in a bit under three minutes. I could live with that. Even for 10 GB files I would only have to wait for half an hour which is mostly okay since it is mostly known well in advance that a file is needed. But lets see if we can somehow get faster than this. Lets analyze where the bottleneck is. Lets have a look at the effective TCP transfer rate with netcat:

ssh user@goflex "netcat -l -p 8000 > /dev/null"
dd if=/dev/zero bs=10M count=1000   netcat goflex 8000

79.3 MB/s wow! Can we get more? Lets try increasing the buffer size on both ends. This can be done using nc6 with the -x argument on both sides.

ssh user@goflex "netcat -x -l -p 8000 > /dev/null"
dd if=/dev/zero bs=10M count=1000   netcat -x gloflex 8000

103 MB/s okay this is definitely NOT the bottleneck here. Lets see how fast I can read from my harddrive:

hdparm -tT /dev/sda

114.86 MB/s.. hmm... and writing to it?

ssh user@goflex "time sh -c 'dd if=/dev/zero of=/remote/path bs=10M count=100; sync'"

42.93 MB/s Those values are far faster than my puny 5.90 MB/s I get with scp. A look at the CPU usage during transfer shows, that the ssh process is at 100% CPU usage the whole time. It seems the bottleneck was found to be ssh and the encryption/decryption involved. I'm transferring directly from my laptop to the device. Not even a switch is in the middle so encryption seems to be quite pointless here. Even authentication doesnt seem to be necessary in this setup. So how to make the transfer unencrypted? The ssh protocol specifies a null cipher for not-encrypted connections. OpenSSH doesnt support this. Supposedly, adding

  "none", SSH_CIPHER_NONE, 8, 0, 0, EVP_enc_null  

to cipher.c adds a null cipher but I didnt want to patch around in my installation. So lets see how a plain netcat performs.

ssh user@goflex "netcat -l -p 8000 > /remote/path"
netcat goflex 8000 < /local/path

32.9 MB/s This is far better! Lets try a bigger buffer:

ssh user@goflex "netcat -x -l -p 8000 > /remote/path"
netcat -x goflex 8000 < /local/path

37.8 MB/s now this is far better! My Gigabyte will now take under half a minute and my 10 GB file under five minutes. But it is tedious to copy multiple files or even a whole directory structure with netcat. There are far better tools for this. An obvious candidate that doesnt encrypt is rsync when being used with the rsync protocol.

rsync -Ph /local/path user@goflex::module/remote/path

30.96 MB/s which is already much better! I used the following line to have the rsync daemon being started by inetd:

rsync stream tcp nowait root /usr/bin/rsync rsyncd --daemon

But it is slower than pure netcat. If we want directory trees, then how about netcatting a tarball?

ssh user@goflex "netcat -x -l -p 8000   tar -C /remote/path -x"
tar -c /local/path   netcat goflex 8000

26.2 MB/s so tar seems to add quite the overhead. How about ftp then? For this test I installed vsftpd and achieved a speed of 30.13 MB/s. This compares well with rsync. I also tried out nfs. Not surprisingly, its transfer rate is up in par with rsync and ftp at 31.5 MB/s. So what did I learn? Lets make a table: </dr></dr></dr></dr></dr></dr></dr></dr></dr></dr>

method	speed in MB/s
scp	5.90
rsync+ssh	5.16
sshfs	5.05
ssh	5.42
netcat	32.9
netcat -x	37.8
netcat -x tar	26.2
rsync	30.96
ftp	30.13
nfs	31.5

For transfer of a directory structure or many small files, unencrypted rsync seems the way to go. It outperforms a copy over ssh more than five-fold. When the convenience of having the remote data mounted locally is needed, nfs outperforms sshfs at speeds similar to rsync and ftp. As rsync and nfs already provide good performance, I didnt look into a more convenient solution using ftp. My policy will now be to use rsync for partial file transfers and mount my remote files with nfs. For transfer of one huge file, netcat is faster. Especially with increased buffer sizes it is a quarter faster than without. But copying a file with netcat is tedious and hence I wrote a script that simplifies the whole remote-login, listen, send process to one command. First argument is the local file, second argument is the remote name and path just as in scp.

#!/bin/sh -e
HOST=$ 2%%:* 
USER=$ HOST%%@* 
if [ "$HOST" = "$2" -o "$USER" = "$HOST" ]; then
        echo "second argument is not of form user@host:path" >&2
        exit 1
fi
HOST=$ HOST#*@ 
LPATH=$1
LNAME= basename "$1" 
RPATH= printf %q $ 2#*: /$LNAME 
ssh "$USER@$HOST" "nc6 -x -l -p 8000 > $RPATH" &
sleep 1.5
pv "$LPATH"   nc6 -x "$HOST" 8000
wait $!
ssh "$USER@$HOST" "md5sum $RPATH" &
md5sum "$LPATH"
wait $!

I use pv to get a status of the transfer on my local machine and ssh to login to the remote machine and start netcat in listening mode. After the transfer I check the md5sum to be sure that everything went fine. This step can of course be left out but during testing it was useful. Escaping of the arguments is done with printf %q. Problems with the above are the sleep, which can not be avoided but must be there to give the remote some time to start netcat and listen. This is unclean. A next problem with the above is, that one has to specify a username. Another is, that in scp, one has to double-escape the argument while above this is not necessary. The host that it netcats to is the same as the host it ssh's to. This is not necessarily the case as one can specify an alias in ~/.ssh/config. Last but not least this only transfers from the local machine to the remote host. Doing it the other way round is of course possible in the same manner but then one must be able to tell how the local machine is reachable for the remote host. Due to all those inconveniences I decided not to expand on the above script. Plus, rsync and nfs seem to perform well enough for day to day use.

tldr: counting without cumulative timing errors Sometimes I want just a small counter, incrementing an integer each second running somewhere in a terminal. Maybe it is because my wristwatch is in the bathroom or because I want to do more rewarding things than counting seconds manually. Maybe I want not only to know how long something takes but also for how long it already ran in the middle of its execution? There are many reason why I would want some script that does nothing else than simply counting upward or downward with some specific frequency. Some bonuses:

• the period should be possible to give as a floating point number and especially periods of a fraction of a second would be nice
• it should be able to execute an arbitrary program after each period
• it should not matter how long the execution of this program takes for the overall counting

Now this can not be hard, right? One would probably write this line and be done with it:

while sleep 1; do echo $i; i=$((i+1)); done

or to count for a certain number of steps:

for i in  seq 1 100 ; do echo $i; sleep 1; done

This would roughly do the job but in each iteration some small offset would be added and though small, this offset would quickly accumulate. Sure that cumulative error is tiny but given that this task seems to be so damn trivial I couldn't bear anymore with running any of the above but started looking into a solution. Sure I could just quickly hack a small C script that would check gettimeofday(2) at each iteration and would adjust the time to usleep(3) accordinly but there HAD to be people before me with the same problem who already came up with a solution. And there was! The solution is the sleepenh(1) program which, when given the timestamp of its last invocation and the sleep time in floating point seconds, will sleep for just the right amount to keep the overall frequency stable. The author suggests, that sleepenh is to be used in shell scripts that need to repeat an action in a regular time interval and that is just what I did. The result is trivial and simple but does just what I want:

• the interval will stay the same on average and the counter will not "fall behind"
• count upward or downward
• specify interval length as a floating point number of seconds including fractions of one second
• begin to count at given integer and count for a specific number of times or until infinity
• execute a program at every step, optionally by forking it from the script for programs possibly running longer than the given interval

You can check it out and read how to use and what to do with it on github: https://github.com/josch/periodic Now lets compare the periodic script with the second example from above:

$ time sh -c 'for i in  seq 1 1000 ; do echo $i; sleep 1; done'
0.08s user 0.12s system 0% cpu 16:41.55 total

So after only 1000 iterations, the counter is already off by 1.55 seconds. This means that instead of having run with a frequency of 1.0 Hz, the actual frequency was 1.00155 Hz. Is it too much to not want this 0.155% of error?

$ time ./periodic -c 1000
0.32s user 0.00s system 0% cpu 16:40.00 total

1000 iterations took exactly 1000 seconds. Cool.

Just set up Synology Audio Station for music playback at home. I originally thought about some more open solution, but this was the easiest choice which might still change. Main motivation was that I got really cheaply Synology Remote, what is wireless remote and USB sound card and I was too lazy to set it up with something else. It works quite nicely but now I'd like to control it from my N950 phone. Synology provide app for iPhone and Android, but others are left in the dark (there is also no documentation about the protocol). The options for me are to write native client or emulate some well know protocol, where I could use existing clients. Quick googling have shown me python-mpd-server, what looks exactly as half of program I need, so I would have to implement only bindings to (not documented) Audio Station protocol (there is open-syno, which seems to implement this protocol, so at least there will be some hints).

Filed under: English Life Linux 0 comments Flattr this!

I'm blogging about this small configuration issue as it took me some time to figure out how to configure cupt and smart to solve this problem. The reason I'm playing with cupt and smartpm is that I'm working to compare again a number of package managers in debian against the state of the art cudf solvers using mpm, and I'm suffering quite a bit to configure my virtual environment. Last year I promised to revise and fix our results. I didn't forget my promise, but it seems it took longer then expected. Anyway, back to the main topic. The release-problem arises because the key used to sign sarge (that I'm using as baseline for my experiments) is long expired. If you try to retrieve sarge from archive.debian.org you will find that sarge is signed with the key A70DAF536070D3A1 and apt-get will complain loudly if you try to use an archive signed with an expired key.

cupt For cupt this is documented in the man page and there are a number of options to add either to /etc/apt/apt.conf to the the cupt own conf file. Then cupt will happily accept the sarge Packages file and let you run update.

cupt::cache::release-file-expiration::ignore "true";
cupt::update::check-release-files "false";
cupt::update::keep-bad-signatures "true";

smartpm For smart I could not found this information anywhere, but reading the source code (tnx good it's python !). To cut it short, you need to set the key trustdb to an empty value for a specific channel. On the command line you get something like : smart channel --set aptsync-614482cb2c7e08d5722af3498232ba52 keyring= --config-file=/root/var/lib/smart/config where aptsync-614482cb2c7e08d5722af3498232ba52 is the channel name corresponding to sarge in my conf. Since I'm using a simulated environment, I save the result of this option in a non-default config file in my chroot.

Dear Lazyweb,

is there a simple way to block some users who login with SSH to read /proc/<pid>/cmdline of processes they don't own? Or better yet: don't see these pids at all?

I know that there are PID namespaces, but they seem to require a special PID 1. Seems hard to get for a simple SSH login. (I wouldn't mind changing a user's shell. But brittle shell startup scripts wouldn't cut it.) systemd-nspawn wants to boot a full Linux distribution in a container and even then I'd be unsure how to wire it up so that it cannot be skipped. I wouldn't mind a read-only bind mount of the outermost Linux installation into a chroot environment, as long as the parent SSH process can get the user jailed into it securely. (No need for someone to be root in the chroot.)

I know that there are RBAC frameworks, but they're cumbersome to use. I don't need file labelling or path-based access control, as I do trust the Linux file permissions for this. I think SMACK wouldn't help here, AppArmor isn't really useable in Debian testing and TOMOYO is a tad crazy to use with its domain transitions through process invocations.

I bet that grsecurity would have something for me up its sleeve. But it's not in a Debian kernel. Even though I know how to compile my own kernel I'd only do that if everything else fails.

Thanks in advance for your help.

UPDATE: That was quick, thanks to everyone who participated! Vasiliy Kulikov came up with a kernel patch to my problem (a hidepid mount option for procfs) that landed in 3.3. I tested it with the kernel in experimental and it works just fine and as expected. With hidepid set to 1, it will still leak the process count and their euids and egids. With hidepid set to 2, you only see your own processes, unless you're root. For ps there's no visible distinction between the two. So to test it you can just invoke this as root on a host running 3.3+:

mount -o remount,hidepid=1 /proc

There's even a backport request in the Debian BTS to get the feature into the wheezy kernel (3.2).

At the recent Ubuntu Developer Summit, I managed to convince a few people (after assurances that there would be no permanent damage) to plug a USB stick into their machines so we could watch Xorg crash and wedge their console. What was this evil thing, you ask? It was an AVR microprocessor connected to USB, acting as a USB HID Keyboard, with the product name set to %n . Recently a Chrome OS developer discovered that renaming his Bluetooth Keyboard to %n would crash Xorg. The flaw was in the logging stack, triggering glibc to abort the process due to format string protections. At first glance, it looks like this isn t a big deal since one would have to have already done a Bluetooth pairing with the keyboard, but it would be a problem for any input device, not just Bluetooth. I wanted to see this in action for a normal (USB) keyboard. I borrowed a Maximus USB AVR from a friend, and then ultimately bought a Minimus. It will let you put anything you want on the USB bus. I added a rule for it to udev:

SUBSYSTEM=="usb", ACTION=="add", ATTR idVendor =="03eb", ATTR idProduct =="*", GROUP="plugdev"

installed the AVR tools:

sudo apt-get install dfu-programmer gcc-avr avr-libc

and pulled down the excellent LUFA USB tree:

git clone git://github.com/abcminiuser/lufa-lib.git

After applying a patch to the LUFA USB keyboard demo, I had my handy USB-AVR-as-Keyboard stick ready to crash Xorg:

-       .VendorID               = 0x03EB,
-       .ProductID              = 0x2042,
+       .VendorID               = 0x045e,
+       .ProductID              = 0x000b,
...
-       .UnicodeString          = L"LUFA Keyboard Demo"
+       .UnicodeString          = L"Keyboard (%n%n%n%n)"

In fact, it was so successfully that after I got the code right and programmed it, Xorg immediately crashed on my development machine. :)

make dfu

After a reboot, I switched it back to programming mode by pressing and holding the H button, press/releasing the R button, and releasing H . The fix to Xorg is winding its way through upstream, and should land in your distros soon. In the meantime, you can disable your external USB ports, as Marc Deslauriers demonstrated for me:

echo "0" > /sys/bus/usb/devices/usb1/authorized
echo "0" > /sys/bus/usb/devices/usb1/authorized_default

Be careful of shared internal/external ports, and having two buses on one port, etc.

2012, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.

This year, the annual Debian conference will be held in Managua, Nicaragua. And I'll be lucky enough to spend two weeks visiting the country after Debconf, along with Elisabeth. Yes, I'll arrive in Nicarague on July 2nd, spend nearly the entire Debcamp, then Debconf, then we'll spend 16 days around the western part of Nicaragua, trying to discover the magic of this country. So, this post is about sharing our plans with my readers. Of course, I do not know the country so we may have made mistakes and bad choices. We'll see. Immediately after Debconf, Elisabeth will join in Managua. She'll be landing on July 15th. We'll then spend a night in an hotel near the airport and immediately leave the day after for Matagalpa, in North Nicaragua. We rented a car for the entire trip indeed, and will be on our own on wild Nicaragua roads..:-) We'll spend two nights in Matagalpa. We plan to visit some coffee or cigar plants, probably have a trip to Lake Apanas and Jinotega. Then, we'll have a short road trip to Esteli where we spend again two nights. We'll be visiting a coffee growing place (beneficio seco de caf ). A full day visit is planned at Miraflor natural reserve to enjoy te beauties of hundreds orchids and some local natural marvels. The next move will be to Leon, where we'll spend 4 nights, visiting a cigar factory (tabacaleras de puros?) on the way, as well as San Jacinto, a place with hydrothermal sources and "Hervideros" (geysers). Four nights in Leon leaves plenty of time for several activities *and* enjoying the colonial city. We'll have a full day at Juan Venado Island reserve with boat trip from Las Penitas (on a fisherman's boat from what the travel agency mentioned), then another full day climbing on the Cerro Negro volcano. Indeed, I was originally considering climbing the Momotombo, but our travel agency warned about the high difficulty. I would have loved that myself but maybe not the two of us...and this is a trip for both of us! So, we played the safe option..:-) After these 4 nights in Leon, we'll move to Granada for 3 nights, through Leon Viejo (the former site of Leon). From Granada, one day will there be used for a visit in the Masaya National Park and see the beauties of Masaya volcano (this is indeed something that could be done for Debconf day trip, IMHO, as it doesn't seem that far from Managua). Another day will be spent to Las Isletas on lake Nicaragua and others visiting the colonial city of Granada. Or, of course, whatever things we don't even known about now..:-) Then we'll move to what I personnally consider the peak of the trip: 3 nights on Ometepe island on lake Nicaragua. Just check Wikipedia to see why Ometepe is, in my opinion, THE place to go in Nicaragua. Here, I'll have my volcano..:-). Indeed, Elizabeth "authorized" me to book a local guide and then climb Concepcion Volcan, if the weather allows for it. 1600m height, that doesn' seem to be a big issue....except when starting from a little bit above sea level and are climbing a volcano that looks like s postcard volcano : nearly a perfect cone shape. So, let's cross fingers for having good weather that day. I promise myself I'll record the GPS track of that one and, even if I'll probably be walking most of the climbing (except if I have a very trained guide...), I'll add it to my run tracks! We might also be going to climb Ometepe's other volcano (Maderas) the day after so that Elizabeth also enjoys these beauties. There also seems to be great places around Maderas such as San Ramon Cascade, Finca el Porvenir, etc. Then, at the end of all this, it will be time to come back to Managua in the final day and fly back to Paris in the early morning of July 31st. All over, I'll be in Nicaragua from July 2nd until July 31st! Full month away, yay! Hurrah for the crazy number of holidays those lazy French people have..:-) During this trip, we might find it interesting some local geeks (not too many as Elizabeth is not that deeply interested in beersigning!) and share a few nice things in local places which are only known by locals. In case you're interested, out (very clever) travel agency is named Nicaragua Adventures and they're definitely worth contacting if you want to travel around .ni, particularly if we prefer booking things in advance as we do. They speak Spanish (of course!), English and French. They're very responsive to e-mail as well.

The first Beta of the upcoming PostgreSQL 9.2 was released yesterday (see announcement). Your humble maintainer has now created packages for you to test. Please give them a whirl, and report any problems/regressions that you may see to the PostgreSQL developers, so that we can have a rock solid 9.2 release. Remember, with the postgresql-common infrastructure you can use pg_upgradecluster to create a 9.2 cluster from your existing 8.4/9.1 cluster and run them both in parallel without endangering your data. For Debian the package is currently waiting in the NEW queue, I expect them to go into experimental in a day or two. For Ubuntu 12.04 LTS you can get packages from my usual PostgreSQL backports PPA. Note that you need at least postgresql-common version 0.130, which is available in Debian unstable and the PPA now. I (or rather, the postgresql-common test suite) found one regression: Upgrades do not keep the current value of sequences, but reset them to their default value. I reported this upstream and will provide updated packages as soon as this is fixed.

You can get inspired from the most unlikely of places, and this time it was a twitter background image. I came across the twitter profile of @Nitish today, a fellow GSoC student from India. One thing that held my attention, apart from the fact that he tweets crazy, was the background picture in his profile. Yes I had not seen something like that before and I absolutely loved the concept. The image was large grid and each tile carried the profile picture of one of his twitter followers. It was beautiful. The background image was generated by twilk, a very popular application that I had never heard of. I wanted to try it, but I figured my follower count might be a little too low for the application to work. Twilk works only with twitter and I wanted it for Facebook. Instead of looking for something similar, I decided to write it myself. I had worked with the Python Imaging Library some time back and knew it would not be very difficult task. The first thing I needed was the profile pic thumbnails of my friends. Facebook API supports a call which returns a small 50x50 pixel profile pic of a user, if we send a valid user ID. I wrote a quick script that returned the user ID s of my friends on Facebook as JSON data. I grepped the user ID s from the JSON and wrote a small wget script to download the profile pics of all my friends to a directory. Once the download was in progress, I started my work with the imaging library. I knew the image count beforehand and after some quick math, I fixed the dimensions of the output image that I wanted. Once the dimensions were fixed, the work was fairly easy. I knew before hand how many images will be there in each row and how many rows will be there in total. All I had to do was to randomly pick one of the images from the directory, and arrange it on the output image canvas. Few lines of code later I get the result that I wanted and it is on my twitter profile. Yay! :) I have uploaded the python code in here. If you are running it, make sure you change the values according to your requirements. The image below is a cropped out from the output image to fit this div. Click on it to see the actual output. When I showed this to @Jaseeemabid, he asked me to put this up on Google App Engine and start a service so that even non geeks could use this. I liked the idea but I am slightly busy(lazy) now. I ll do it when I am free or when I see that there is some demand for this! :) Update As few Redditors pointed out, this may not be the best way to do it. This was just a quick hack to get something done. The JSON could be processed in Python itself and files can be downloaded as well. I modified an old PHP script to get the JSON data and since the structure was relatively simple, I grepped the relevant information and fed it to wget as I prefer wget over any other tool for mass downloads. Update 2 Redditor Bio Lazarus improved upon this and made an end to end version that does all of the above in one single python script and the code is available here. And if you like this post, you should follow me on twitter :)

For years, no decades, I've typed tar tzf something, tar xzf something. Except when someone annoying sent an uncompressed tar file and I had to then go and take out the z in the middle. Then came bzip2, and we learned tar tjf, tar xjf. OK, I could live with that. One emerging problem was that the tab completion now worked the wrong way around conceptually, because you had to pick and type the right letter first in order to see the appropriate set of files to unpack offered for completion. Then came lzma, which was (quick, guess?), tar tJf, tar xJf. And then there was lzop, which was too boring to get its own letter, so you had to type out tar -x --lzop -f. But lzma was short-lived, because then came xz, which was also J, because lzma was now too boring as well to get its own letter. Oh, and there is also the old compress, which is Z, and lzip, which I'd never heard of. But stop that. Now there is

 -a, --auto-compress
            use archive suffix to determine the compression program

This handles all the above compression programs, and no compression. So from now on, I always use tar taf and tar xaf. Awesome. The finger movements will be almost the same on QWERTY and AZERTY, and easier than before on QWERTZ. Actually, this option is already four years old in GNU tar. Funny I'd never heard of it until recently.

When I set an alarm, my clock, now running on the computer in my pocket, is smart enough to tell me how much time will pass until the alarm is scheduled to sound. This has eliminated the old problem of sleeping past meetings before being surprised by an alarm precisely half a day after I had originally planned to wake. The price has been having to know exactly how little I will sleep: a usually depressing fact that had previously been obscured by my difficulty doing time arithmetic in my most somnolent moments.

Sparkleshare is a file sharing utility that keeps folders in sync on multiple computers in a similar fashion to the commercial and proprietary Dropbox. Sparkleshare recently released a Windows client, finally allowing me to start switching my co-workers at the Progressive Technology Project away from Dropbox. Overall, I'm very impressed. In particular, I appreciate the Sparkleshare authors' decision to build on top of existing tools (git for storage and revision control, ssh for transport, and ssh public key infrastructure for authorization and authentication). That means I don't have to learn new tools and protocols to debug and it means Sparkleshare can focus on the file sharing pieces. Despite my overall enthusiasm, I do have some serious concerns. Protecting your credentials A researcher found a startling security flaw in Dropbox - if you can copy a particular file from a user's computer to your own computer, you can impersonate them flawlessly, accessing all their Dropbox files without having to know the user's password. Unfortunately, Sparkleshare suffers from this same vulnerability if you use the default configuration. When you first install Sparkleshare, it creates a password-less ssh private/public key pair and then makes the public part easily accessible to you so you can add it to your server (or gitorious account, etc). Very convenient. But it also means that all an attacker needs to do is copy your private key (and your Sparkleshare configuration file) and they get complete control over your files. This problem is easily avoided. If you are running Linux or Mac OS X and you have your own key loaded in your ssh agent, Sparkleshare will happily use that key. So - simply by using your existing (presumably password-protected key) on your remote Sparkleshare servers, you can mitigate this problem. Sparkleshare will still load your Sparkleshare generated key, but if you don't provide that key with access to anything, no harm is done. Confirming each use of your key However... that leads to a new problem. If you are like me, your ssh agent is configured to ask for a confirmation every time your key is used. And, Sparkleshare regularly polls the remote git repository for changes. At best repeatedly clicking to confirm is tedious. At worst, it prevents you from intelligently rejecting malicious requests, thus defeating the whole purpose of the check. It is possible to launch Sparkleshare via ssh-agent in an environment in which you are not requiring the confirmation when Sparkleshare uses your key, but still requiring it for all other uses. However, given the trade offs, I've decided to add a password to my Sparkleshare provided ssh key rather than using my existing key: ssh-keygen -p -f ~/.config/sparkleshare/sparkleshare.jamie@progressivetech.org.key Now, I am prompted to enter my passphrase when I start Sparkleshare and don't have to confirm every use of the key. And, I continue to confirm each use of my main key. git was designed to store code, not documents Just because something is designed for one purpose doesn't mean it can't be used for another. However, there are a few limitations. Large files are one. git can handle files over 100MB, but may run into memory problems. I had to alter the git windowMemory setting, raising it higher than the size of the largest file. Another problem is disk space. Since git keeps full revision history on every machine, you have to download more data than just the files that are checked out. The more edits you make to a repository, the more disk space beyond what is checked out is needed (and deleting files won't help). Lastly, you can't use file modification times. With git, the file modification time will depend on when you checkout the files. With some fancy hook writing, you could tweak things so that the file modification date is the same as the commit date, but that still won't help you if you add an existing directory to Sparkleshare because all the files will have the same commit time. This "bug" has been reported to git and it has been rejected because mucking with the modification time of files can have bad results when you are using make to compile code. As Linus colorfully put it:

I'm sorry. If you don't see how it's WRONG to set a datestamp back to something that will make a simple "make" miscompile your source tree, 
I don't know what definition of "wrong" you are talking about.
It's WRONG.
It's STUPID.
And it's totally INFEASIBLE to implement.

Well, did I mention that git was designed for source code? No server validation Of all the problems I encountered, this one is by far the most serious. Perusing ~/.config/sparkleshare/debug.log is very informative. On a default installation, after you have added a project, you'll see:

19:32:27 [Fetcher][/home/jamie/SparkleShare/.tmp/bar] Fetching folder: ssh://foo@bar.org/foo/bar
19:32:27 [Fetcher] Disabled host key checking for bar.org

Wah. Once the project is added, whatever key fingerprint is given is stuffed into your ~/.ssh/known_hosts file. Host key checking is only disabled when you initially create the project, so if you connect to the proper server the first time, you are protected from subsequent man-in-the-middle attacks. However... if the initial fingerprint is wrong, your personal ssh configuration is now poisoned. This seems like a very bad idea. I've opened a debian bug to address it.

It has been a few busy weeks for me, but I am finally back to publish another interview with the people behind Debian Edu and Skolelinux. This time it is one of our German developers, who have helped out over the years to make sure both a lot of major but also a lot of the minor details get right before release. Who are you, and how do you spend your days? My name is J rgen Leibner, I'm 49 years old and living in Bielefeld, a town in northern Germany. I worked nearly 20 years as certified engineer in the department for plant design and layout of an international company for machinery and equipment. Since 2011 I'm a certified technical writer (tekom e.V.) and doing technical documentations for a steam turbine manufacturer. From April this year I will manage the department of technical documentation at a manufacturer of automation and assembly line engineering. My first contact with linux was around 1993. Since that time I used it at work and at home repeatedly but not exclusively as I do now at home since 2006. How did you get in contact with the Skolelinux/Debian Edu project? Once a day in the early year of 2001 when I wanted to fetch my daughter from primary school, there was a teacher sitting in the middle of 20 old computers trying to boot them and he failed. I helped him to get them booting. That was seen by the school director and she asked me if I would like to manage that the school gets all that old computers in use. I answered: "Yes". Some weeks later every of the 10 classrooms had one computer running Windows98. I began to collect old computers and equipment as gifts and installed the first computer room with a peer-to-peer network. I did my work at school without being payed in my spare time and with a lot of fun. About one year later the school was connected to Internet and a local area network was installed in the school building. That was the time to have a server and I knew it must be a Linux server to be able to fulfil all the wishes of the teachers and being able to do this in a transparent and economic way, without extra costs for things like licence and software. So I searched for a school server system running under Linux and I found a couple of people nearby who founded 'skolelinux.de'. It was the Skolelinux prerelease 32 I first tried out for being used at the school. I managed the IT of that school until the municipal authority took over the IT management and centralised the services for all schools in Bielefeld in December of 2006. What do you see as the advantages of Skolelinux/Debian Edu? When I'm looking back to the beginning, there were other advantages for me as today. In the past there were advantages like:

• I don't need to buy it so it generates no costs to the school as they had little money to spent for computers and software.
• It has a licence which grands all rights to use it without cost.
• It was more able to fit all requirements of a server system for schools than a Microsoft server system, even if there are only Windows clients because of it's preconfigured overall concept of being a infrastructure solution and community for schools, not only a server
• I was able to configure the server to the needs of the school.

Today some of the advantages has been lost, changed or new ones came up in this way:

• Most schools here do have money to buy hardware and software now.
• They are today mostly managed from central IT departments which have own concepts which often do not fit to Debian Edu concepts because they are to close to Microsoft ideology.
• With the Squeeze version of Debian Edu which now uses GOsa for management I feel more able to manage the daily tasks than with the interfaces used in the past.
• It is more modular than in the past and fits even better to the different needs.
• The documentation is usable and gets better every day.
• More people than ever before are using Debian Edu all over the world and so the community, which is an very important part I think, is sharing knowledge and minds.
• Most, maybe all, of the technical requirements for schools are solved today by Debian Edu.

What do you see as the disadvantages of Skolelinux/Debian Edu?

• There are too few IT companies able to integrate Debian Edu into their product portfolio for serving schools with concepts or even whole municipality areas.
• Debian Edu has beside other free and open software projects not enough lobbyists which promote free and open software to politicians.
• Technically there are no disadvantages I'm aware of.

Which free software do you use daily? I use Debian stable on my home server and on my little desktop computer. On my laptop I use Debian testing/sid. The applications I use on my laptop and my desktop are Open/Libre-office, Iceweasel, KMail, DigiKam, Amarok, Dolphin, okular and all the other programs I need from the KDE environment. On console I use newsbeuter, mutt, screen, irssi and all the other famous and useful tools. My home server provides mail services with exim, dovecot, roundcube and mutt over ssh on the console, file services with samba, NFS, rsync, web services with apache, moinmoin-wiki, multimedia services with gallery2 and mediatomb and database services with MySQL for me and the whole family. I probably forgot something. Which strategy do you believe is the right one to use to get schools to use free software? I believe, we should provide concepts for IT companies to integrate Debian Edu into their product portfolio with use cases for different countries and areas all over the world.

Generally I haven t had a positive attitude towards the NBN. It doesn t seem likely to fulfill the claims of commercial success and would be a really bad thing to privatise anyway. Also it hasn t seemed to offer any great benefits either. The claim that it will enable lots of new technical developments which we can t even imagine yet that aren t possible with 25Mb/s ADSL but which also don t require more than the 100Mb/s speed of the NBN never convinced me. But one thing it could really do well is to give better Internet access in remote areas. Ideally with static or near-static IPv6 addresses (because we have already run out of IPv4 addresses). Currently 3G networks do all sorts of nasty NAT things to deal with the lack of IPv4 addresses which causes a lot of needless pain if you have a server connected via 3G. One of the NBN plans is for wireless net access to remote homes, with some sanity among the people designing the network such NBN connections would all have static IPv6 subnets as long as they don t move. I m currently working on a project that involves servers on 3G links. I don t have a lot of options on implementation due to hardware and software constraints. So if the ISPs using the NBN and the NBN itself (for the wireless part) could just give us all IPv6 static ranges then lots of problems would be solved. Of course I don t have high hopes for this. One of the many ways that the NBN has been messed up is in allowing the provision of lower speed connections. As having an ADSL2+ speed NBN connection is the cheapest option a lot of people will choose it. Therefore the organisations providing services will have to do so with the expectation that most NBN customers have ADSL2+ speed and thus they won t provide services to take advantage of higher speeds. Related posts:

1. RPC and SE Linux One ongoing problem with TCP networking is the combination of...
2. A New Strategy for Xen MAC Allocation When installing Xen servers one issue that arises is how...
3. New Net Connections On Thursday my new InterNode ADSL2+ service was connected [1]....

I ve send a mail today with a summary of some of my experiences in Managua and tiny bits of Nicaragua late April / early May 2012. If you are considering going to DebConf12 (which you obviously should do as you are reading this!) go and read this mail, as it includes a thematically sorted list of impressions and advice written by Gunnar, Norman, Leo, Felix and myself, which should give you some better idea about DebConf12 in Managua! While I spent 14 days in Nicaragua in total, for the last three days I was joined by Gunnar Wolf, to make sure my impressions about the local team, the status of the preparations and everything were correct. Our visit also seems to have been very motivating to local team members, whom I ve seen really enthusiastic to welcome a huge Debian crowd for the first DebConf in Central America in just a few weeks! At the last DebConf12 IRC meeting I summarized it like this: in very short: my two weeks were great, I had a great vacation, met many members of the localteam, which is way bigger then it seems on irc, saw and liked UCA (Universidad Centroamericana, the DebConf12 venue), had zillions of reunions (meetings, see schedule wiki page) and met many nice people and a very hot & interesting country+capital with great countryside. The deadline for sponsored accomodation ends in three days - so if you haven t registered yet, hurry up, DebCamp starts in 49 days! :-) (After that registration is still possible, but you will need to pay for your costs yourself.) We re looking forward to see you there and have a great DebConf together! Hopefully we sweat together while Wheezy is frozen :-D

Last year I was able to switch tasksel to using metapackages, instead of the weird non-package task things that had been used before Debian supported Recommends fields well. An unanticipated result of the new task packages is that I have this nice popcon data available for them, so can get graphs like these. For new installs of testing, KDE and Xfce are neck and neck. With Gnome being the default, it's hard to say which desktop users really prefer. My feeling is that it's probably nearly evenly split now. (I installed Xfce on my sister's laptop last week, and anticipate moving all my family to it, rather than Gnome 3.) The above graph also shows a surprisingly large number of ssh server task installs. In fact, it's the most often manually installed task. Probably many of those are server machines, and so I'm considering having tasksel automatically select ssh on systems where it doesn't automatically select a desktop. Language data is also available. Taskel uses language tasks internally, without exposing an interface, so this will be almost entirely users who did an install of testing localised to their language. Interesting data can be teased out of this too. For example there seem more installs in Catalan than Chinese ... and at least 10 Esperanto users. (As with any popcon number, this is a lower bound, to be multiplied by the scaling guesstimate of your choice.)

---

By the way, I've got a new vanity domain for my blog and wiki: http://joeyh.name/ The old http://kitenet.net/~joey/ will continue to work, like it has since 1997. But the new is easier to type. And it let me move my site to Branchable, at last.

• Content: explanations about cryptography, SSL and PGP, then signing-party
• Location: EPN la Bourdonnais, 105 avenue de la Bourdonnais, 75007 Paris
• Date: 2012-05-21 18:45+02:00
• Duration: 02:15

Monday 21st during at 18:45, in Paris, there will be a conference organized by Parinux, where I will explain the principles of cryptography and their application in the SSL and PGP systems. This conference will be followed at 20:30 by a signing-party PGP et CAcert. For the signing-party, I will ask participants to:

1. generate a key pair if you do not already have one;
2. send me you public key and register;
3. print some copies of your key fingerprint;
4. print the list of participants I will send you;
5. come with all that stuff and one or two identity documents.

This is a partial translation of the full article I wrote in French, in case foreigners could attend. Sorry for the very late notice

Do you wish, like I do, you could edit collaboratively in Vim? This feature is number 10 on the Vim voting page, so it seems I m not alone. How about Pentadactyl coupled with any of the existing web-based collaborative editors, such as Etherpad? OK, so it s not quite Vim, and there are some rough edges to this particular pairing, but I m finding it s good enough for my needs. It even gives me a Vim-like editing experience while other participants use the default Etherpad editor. Yes, I know about whiteboard.debian.net, but for the past three years I have been using a single instance of Etherpad with my family to maintain a shopping list to which we all make contributions. First of all, that s not a Debian activity, so to make the switch, I d need to make a personal clone of the service for our personal use. But more importantly, we find Etherpad features such as colours for different participants and the timeline are just too useful to give up on. On the other hand, the less the web editor interferes with your web browser s default textarea behaviour, the easier time Pentadactyl is going to have. Indeed, I asked on #pentadactyl @ irc.oftc.net about some problems I was having and I was told flat out that Pentadactyl does not work with graphical web editors. So, you may wish to use another web-based collaborative editor for this reason. That being said, I did learn a few things about helping Pentadactyl get along better with Etherpad, so if you would like to try it yourself, read on. The key to getting started was to enter text edit mode within the textarea with <C-t>. For the most part, this behaves like Vim normal mode . I am still learning, but many basic motion and editing keys behave just as they would in Vim. Fantastic! However, the moment I tried to :undo I hit my first problem. Using the latest release version of Pentadactyl (1.0rc1 at time of writing), pressing "u" to :undo produced no visible result. I tried the latest daily build as well, and only saw a marginally more helpful "Node not found" error message displayed in the status area. But it turns out you can use passthrough mode to use the textarea s own undo. Just :tmap u <C-v><C-z> and we re back in business again. I m still experimenting with this setup, so the jury s still out on whether I ll stick with it, or whether the remaining incompatibilities between Pentadactyl and Etherpad will drive me nuts. But it looks promising. Clearly, judicious use of :autocmd to always start in text edit mode and bind that undo key whenever I enter the site will help make the experience even better. If you try it out yourself, I d love to hear how things went for you. Or if you have found an even better solution that works for you, do share.