Search Results: "xam"

11 October 2011

Simon Josefsson: Unattended SSH with Smartcard

I have several backup servers that run the excellent rsnapshot software, which uses Secure Shell (SSH) for remote access. The SSH private key of the backup server can be a weak link in the overall security. To see how it can be a problem, consider if someone breaks into your backup server and manages to copy your SSH private key, they will now have the ability to login to all machines that you take backups off (and that should be all of your machines, right?). The traditional way to mitigate SSH private key theft is by password protecting the private key. This works poorly in an unattended server environment because either the decryption password needs to be stored in disk (where the attacker can read it) or the decrypted private key has to be available in decrypted form in memory (where attacker can read it). A better way to deal with the problem is to move the SSH private key to a smartcard. The idea is that the private key cannot be copied by an attacker who roots your backup server. (Careful readers may have spotted a flaw here, and I need to explain one weakness with my solution: an attacker will still be able to login to all your systems by going through your backup server, however it will require an open inbound network connection to your backup server and the attacker will never know what your private key is. What this does is to allow you to more easily do damage control by removing the smartcard from the backup server.) In this writeup, I ll explain how to accomplish all this on a Debian/Ubuntu-system using a OpenPGP smartcard, a Gemalto USB Shell Token v2 with gpg-agent/scdaemon from GnuPG together with OpenSSH.


First we need to install some packages. The goal is to configure OpenSSH to talk to the gpg-agent which will start and talk to scdaemon which in turn talks to pcscd which talks to the smart card reader. For some strange reason, the scdaemon binary is shipped with GnuPG s S/MIME interface in the gpgsm package. 
# apt-get install pcscd gnupg-agent gpgsm
The above command should install and start pcscd and if all works well, you should be able to check the status of the smartcard using GnuPG. 
# gpg --card-status
You need to initialize the smartcard and generate a private key on it, again using GnuPG. If you trust GnuPG more than the smartcard to generate a good private key, you may generate the private key using GnuPG and then move it onto the smartcard (hint: use the keytocard command). Make sure you don t leave a copy of the private key on the same machine! 
# gpg --card-edit
gpg: detected reader  Gemalto GemPC Key 00 00'
...
gpg/card> admin
Admin commands are allowed
gpg/card> name
Cardholder's surname:
Cardholder's given name: host.example.org
gpg: 3 Admin PIN attempts remaining before card is permanently locked
Please enter the Admin PIN
gpg: gpg-agent is not available in this session
gpg/card> lang
Language preferences: en
gpg/card> generate
Make off-card backup of encryption key? (Y/n) n
Please enter the PIN
What keysize do you want for the Signature key? (2048)
What keysize do you want for the Encryption key? (2048)
What keysize do you want for the Authentication key? (2048)
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y
You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"
Real name: host.example.org
Email address:
Comment:
You selected this USER-ID:
    "host.example.org"
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
gpg: existing key will be replaced
gpg: please wait while key is being generated ...
gpg: key generation completed (33 seconds)
gpg: signatures created so far: 0
gpg: existing key will be replaced
gpg: please wait while key is being generated ...
gpg: key generation completed (18 seconds)
gpg: signatures created so far: 1
gpg: signatures created so far: 2
gpg: existing key will be replaced
gpg: please wait while key is being generated ...
gpg: key generation completed (23 seconds)
gpg: signatures created so far: 3
gpg: signatures created so far: 4
gpg: key 12345678 marked as ultimately trusted
public and secret key created and signed.
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   2048R/12345678 2011-09-19
      Key fingerprint = 1234 5678 1234 5678 1234  5678 1234 5678 1234 5678
uid                  host.example.org
sub   2048R/23456789 2011-09-19
sub   2048R/34567890 2011-09-19
gpg/card> quit
#
Now for the interesting part. OpenSSH talks to an agent for private key handling, and GnuPG s gpg-agent supports this protocol when the --enable-ssh-support parameter is given. During startup, gpg-agent will print some environment variables that needs to be set when ssh is run. Normally gpg-agent is invoked by the Xsession.d login scripts, so that the environment variables are inherited by all your processes. However, for an unattended machine without any normal login process, we need to write a script to start gpg-agent. First do these manual steps, to confirm that everything works. 
# gpg-agent --daemon --enable-ssh-support > /var/run/gpg-agent-info.env
# . /var/run/gpg-agent-info.env
# ssh-add -L
ssh-rsa AAAAB3N... cardno:000500000BD8
#
If the final step printed a SSH public id, the (sometimes) tricky part in getting the hardware to work is (hopefully) complete. What remains is to script things so that gpg-agent is started on boot and to make sure that your backup scripts has the proper environment variables before launching whatever processes will launch ssh. Further, since we will be running unattended, we need a mechanism to unlock the smartcard using a PIN interactively once on each boot of the machine. I prefer manually entering the PIN on every boot over having the PIN stored in a file on the disk. I will use the /etc/rc.local mechanism to start gpg-agent, like this: 
# cat> /etc/rc.local
#!/bin/sh -e
exec gpg-agent --daemon --enable-ssh-support \
    --pinentry-program /usr/local/sbin/pinentry-unattended \
    --write-env-file /var/run/gpg-agent-info.env
^D
The astute reader will now ask what /usr/local/sbin/pinentry-unattended is and why it is needed. Now here is the situation. scdaemon will normally query the user for a PIN using a tool called pinentry which reads and write to the user s TTY directly. This won t work in unattended mode, so we want the scdaemon to signal failure here unless we are actually unlocking the smartcard manually. Here is the entire script: 
#!/bin/sh
# /usr/local/sbin/pinentry-unattended -- by Simon Josefsson
if test x"$PINENTRY_USER_DATA" = xinteractive; then
    exec pinentry "$@"
fi
exit 1
What remains is a script to unlock the smartcard by providing the PIN. This is typically invoked manually if the server has restarted for some reason. Don t worry, any ssh sessions invoked by cron until you have managed to unlock the smartcard will fail with an authentication error it won t hang waiting for a PIN to be entered. 
#!/bin/sh
# /usr/local/sbin/unlock-smartcard -- by Simon Josefsson.
. /var/run/gpg-agent-info.env; export GPG_AGENT_INFO SSH_AUTH_SOCK SSH_AGENT_PID
gpg-connect-agent 'scd killscd' /bye > /dev/null
while ! gpg-connect-agent 'scd serialno' /bye   grep -q SERIALNO; do
    sleep 1
done
PINENTRY_USER_DATA=interactive
export PINENTRY_USER_DATA
checkpin
And the script checkpin is as follows: 
#!/bin/sh
# /usr/local/sbin/checkpin -- by Simon Josefsson.
id= gpg-connect-agent 'scd serialno' /bye   head -1   cut -d\  -f3 
gpg-connect-agent "scd checkpin $id" /bye   grep -q OK
At this point, you should have everything configured and installed. Don t forget to chmod +x the scripts. The typical use-pattern is as follows. After the machine has been started, gpg-agent is running but the smartcard is not unlocked with the PIN. You need to manually login to the machine and run unlock-smartcard and enter the PIN. In the script that runs the backup jobs, invoked via cron, make sure that the first line of the scripts reads (assuming Bourne shell script syntax): 
. /var/run/gpg-agent-info.env; export GPG_AGENT_INFO SSH_AUTH_SOCK SSH_AGENT_PID
To avoid needlessly attempting ssh connections if the smartcard is not unlocked, your backup script can also call the checkpin code and abort if it doesn t return true. 
checkpin   exit 1
Some final words about debugging. A basic command to run to check that the GnuPG side is working is gpg --card-status, it should print some information about the smartcard if successful. To check that the SSH agent part is working, use ssh-add -L. If you get error messages, try killing the scdaemon process by running killall -9 scdaemon and let gpg-agent respawn a new scdaemon process. That s it! If you like my writeup, please flattr it. :)

flattr this!

10 October 2011

Vincent Bernat: SSL termination: stunnel, nginx & stud, round 2

A month ago, I published an article on the compared performances of stunnel, nginx and stud as SSL terminators. The conclusion was to use stud on a 64-bit system, with session caching and AES. stunnel was unable to scale properly and nginx exhibited important latency issues. I got constructive comments on many aspects. Therefore, here is the second round. The protagonists are the same but both the conditions and the conclusions have changed a bit.

Benchmarks The hardware platform remains unchanged: We focus on the following environment:
  • Ubuntu Lucid, featuring a patched version of OpenSSL 0.9.8k with AES-NI builtin and enabled by default
  • 2.6.39 kernel (250 HZ)
  • 64-bit
  • 1 core for HAProxy, 2 cores for network cards1, 4 cores for SSL, 1 core for the system
  • system limit of 100,000 file descriptors per process
On the SSL side, the main components are:
  • 2048-bit certificates
  • AES128-SHA cipher suite
  • 8 HTTP/1.0 requests per client
  • HTTP response payload of 1024 bytes
If you are interested in different conditions (32-bit, number of cores, 1024-bit certificates, proportion of session reuses), have a look at the previous article whose results on the matter are still valid. I only consider 2048-bit certificates because most certificates issued today are 2048-bit. In SP800-57, NIST advises that 1024-bit RSA keys will no longer be viable after 2010 and advises moving to 2048-bit RSA keys:
If information is initially signed in 2009 and needs to remain secure for a maximum of ten years (i.e., from 2009 to 2019), a 1024-bit RSA key would not provide sufficient protection between 2011 and 2019 and, therefore, it is not recommended that 1024-bit RSA be used in this case.
And the protagonists are: UPDATED: Configuration of nginx has been updated to switch to off both multi_accept and accept_mutex thanks to the feedback of Maxim Dounin on latency issues that were present in the previous article (and still present in the early version of this article). HAProxy (configuration) is used for stunnel and stud to add load-balancing. For each test, we want to maximize the number of HTTP transactions per second (TPS) while minimizing latency and unsucessful transactions. The maximum number of allowed failed transactions is set at 1 transaction out of 1000. The maximum allowed average latency is 100 ms2.

Control case Below is the first comparison between stud, nginx and stunnel on plain Ubuntu Lucid. Like in previous benchs, stunnel is not able to scale, with a maximum of 400 TPS. However, early latency issues of nginx has been solved, thanks to the change in configuration. stud and nginx gets similar performance but, as you can see on the bottom plot (whose scale is logarithmic) stud starts with an initial latency of 40 ms. This is pretty annoying. I have tried some micro optimization to solve this issue but without luck. If latency matters for you, stick with stunnel or nginx (or find where the problem lies in stud). UPDATED: After reporting this issue, a fix has been commited. The culprit was Nagle s algorithm which improves bandwidth at the expense of latency. With the fix, performances are mostly the same but there is no latency penalty any more. stud, nginx, stunnel on Ubuntu Lucid

OpenSSL 1.0.0e When I published the first round of benchmarks, Micha Trojnara was dubious about the bad results of stunnel. He told me on Twitter:
I solved it! You compiled stunnel against a 4 years old OpenSSL with MT-unsafe SSL_accept. Upgrade OpenSSL and rebuild stunnel. This bug was fixed in OpenSSL 1.0.0b. I implemented a (slow) workaround, so stunnel doesn t crash with earlier versions of OpenSSL.
Therefore, I backported Ubuntu Oneiric OpenSSL 1.0.0e to Ubuntu Lucid3 and, as you can see yourself on the plot below, there is a huge performance boost for stunnel: about 400%! stud features a slight improvement too (about 8%) but still starts with a latency of 40 ms while nginx does not improve. stud, nginx, stunnel with OpenSSL 1.0.0e

Intel Accelerator Engine Most Linux distributions are still using OpenSSL 0.9.8. For example, Debian Squeeze ships with OpenSSL 0.9.8o, Ubuntu Lucid with 0.9.8k and RHEL 5.7 with 0.9.8e. Therefore, they are not eligible to recent performance improvements. To solve this, Intel released Intel Accelerator Engine, which also brings, among other things, AES-NI support:
This is essentially a collection of assembler modules from development OpenSSL branch, packed together as a standalone engine. Standalone means that it can be compiled outside OpenSSL source tree without patching the latter. Idea is to provide a way to utilize new code in already released OpenSSL versions constrained by support and release policies limiting their evolvement to genuine bug fixes.
Alas, in my tests, there was no improvement compared to OpenSSL 1.0.0e (already patched with AES-NI support). stunnel comes with out-of-the-box support for additional engines. Just add something like this to your stunnel.conf: 
engine=dynamic
engineCtrl=SO_PATH:/WOO/intel-accel-1.4/libintel-accel.so
engineCtrl=ID:intel-accel
engineCtrl=LIST_ADD:1
engineCtrl=LOAD
engineCtrl=INIT
For stud, you should grab support for engine selection from GitHub and for nginx, just specify the engine with ssl_engine.

Intel Performance Primitives In his great article Accelerated SSL , Simon Kuhn shows huge performance improvements by using Intel s IPP library, a library aimed at leveraging power of multi-core processors and including some cryptographic primitives and a patch to make use of them in OpenSSL. Unfortunately, this library is not free, not even as in free beer . The patch provided by Intel only applies on OpenSSL 0.9.8j. Simon updated it for OpenSSL 1.0.0d. It also applies with a slight modification to OpenSSL 1.0.0e. Look at Simon s article for instructions on how to compile OpenSSL with IPP. Simon measured 220% improvement on RSA 1024-bit private key operations by using this library. Unfortunately, we don t get such a performance burst with stud which exhibits a 15% improvement (but remember that private key operations only happen once every 8 transactions thanks to session reuse). Worst, with stunnel, performances are reduced by one third and nginx just segfaults. I think such a library is risky to use in production. stud and stunnel with IPP

Hyperthreading During the previous tests, only half the capacity of each core were used. For stud, adding more workers per core only slightly improves this situation (less than 10% with 4 workers per core). Therefore, I thought using hyperthreading could rectify the situation here. In this case, stud, stunnel and nginx are allocated 8 virtual cores instead of 4. stud only got a small improvement of 5%, nginx a moderate improvement of 11% while stunnel got a significant improvement of 26%. stud, stunnel and nginx with hyperthreading

Conclusion Here is a quick recap of the maximum TPS reached on each tests (while keeping average response time below 100 ms). All tests were run on 4 physical cores with 2048-bit certificates, AES128-SHA cipher suite. Each client makes 8 HTTP requests and reuse the first SSL session.
Context nginx 1.1.4 stunnel 4.44 stud @e207dbc5a3
OpenSSL 0.9.8k 1489 TPS 400 TPS 1492 TPS
OpenSSL 1.0.0e 1501 TPS 1561 TPS 1609 TPS
OpenSSL 1.0.0e + IPP - 1101 TPS 1857 TPS
OpenSSL 1.0.0e + HT 1718 TPS 1973 TPS 1694 TPS
I also would like to stress that below 1500 TPS, stunnel adds an overhead of a few milliseconds while stud adds 40 milliseconds. With OpenSSL 1.0.0e and hyperthreading enabled, stunnel has both good performance and low latency on low loads to moderate loads. You may just want to limit the number of allowed connections to avoid collapsing on high loads. While in the previous round, stud was the champion and stunnel lagged behind, the situation is now reversed : stunnel is on top! UPDATED: As said earlier, the latency issue with stud has been solved. Performances remain mostly the same.
  1. HAProxy and network cards don t need dedicated core for this kind of load. However, if you also want to handle non SSL traffic, those dedicated cores can get busy.
  2. The real condition is more complex. Each point is considered valid if the corresponding average latency is less than 100 ms or if the average on the next 8 points is less than 100 ms. Temporary latency bursts are therefore allowed. We are not so forgiveful with failed transactions.
  3. This is pretty straighforward: I just dropped multiarch support. I chose to use Ubuntu version of OpenSSL instead of upstream because there are some interesting patches, like automatic support of AES-NI.

7 October 2011

Kai Wasserb ch: MPlayer: and no black bars

After a little pause, I thought I add another entry to my ongoing tips series. Today we'll get rid of the black bars you can most likely observe on a widescreen display, when playing a video DVD with MPlayer. This little post assumes that you've already inserted the DVD you'd like to watch, your MPlayer setup works with dvd:// style addresses, you know which title from the DVD you want to watch (the examples will use the first title) and you know your monitor's aspect ratio (the examples uses 16:9). Now all that is needed are the following two steps:
  1. Detect the cropping parameters: 
    mplayer -vf cropdetect -monitoraspect 16:9 dvd://1
    The previous line is followed by the usual MPlayer header and output. The film should start playing and after a few seconds you can press q (just wait until the detected values stop changing). Now there should be a few lines looking like: 
    [CROP] Crop area: X: 0..719  Y: 76..499  (-vf crop=720:416:0:80).
A:   1.5 V:   1.5 A-V: -0.000 ct: -0.018  38/ 38  5%  7%  1.0% 0 0
[CROP] Crop area: X: 0..719  Y: 75..500  (-vf crop=720:416:0:80).
A:   1.6 V:   1.6 A-V: -0.000 ct: -0.018  39/ 39  5%  7%  1.1% 0 0
[CROP] Crop area: X: 0..719  Y: 75..500  (-vf crop=720:416:0:80).
A:   1.6 V:   1.6 A-V: -0.000 ct: -0.018  40/ 40  5%  7%  1.1% 0 0
  2. Play the film with the detected crop values and the correct aspect for the video material (4:3 for PAL and 3:2 for NTSC): 
    mplayer -vf crop=720:416:0:80 -monitoraspect 16:9 -aspect 4:3 dvd://1
Of course you can add any other options to the invocation as usual, though you only need to add those to the second invocation (e.g. select the correct audio stream or start in fullscreen mode). Also, you might want to set the monitor aspect permantly in your MPlayer configuration (just add monitoraspect = "16:9"), as the monitor rarely changes for the average machine /etc/mplayer/mplayer.conf.local is probably the best place (make sure your /etc/mplayer/mplayer.conf includes that file at the end). Now you should be able to enjoy all video DVDs on your widescreen display without black bars at the top and bottom!

6 October 2011

Colin Watson: Top ideas on Ubuntu Brainstorm (August 2011)

The Ubuntu Technical Board conducts a regular review of the most popular Ubuntu Brainstorm ideas (previous reviews conducted by Matt Zimmerman and Martin Pitt). This time it was my turn. Apologies for the late arrival of this review. Contact lens in the Unity Dash (#27584) Unity supports Lenses, which provide a consistent way for users to quickly search for information via the Dash. Current lenses include Applications, Files, and Music, but a number of people have asked for contacts to be accessible using the same interface. While Canonical's DX team isn't currently working on this for Ubuntu 11.10 or 12.04, we'd love somebody who's interested in this to get involved. Allison Randal explains how to get started, including some skeleton example code and several useful links. Displaying Ubuntu version information (#27460) Several people have asked for it to be more obvious what Ubuntu version they're running, as well as other general information about their system. John Lea, user experience architect on the Unity team, responds that in Ubuntu 11.10 the new LightDM greeter shows the Ubuntu version number, making that basic information very easily visible. For more detail, System Settings -> System Info provides a simple summary. Volume adjustments for headphone use (#27275) People often find that they need to adjust their sound volume when plugging in or removing headphones. It seems as though the computer ought to be able to remember this kind of thing and do it automatically; after all, a major goal of Ubuntu is to make the desktop Just Work. David Henningson, a member of Canonical's OEM Services group and an Ubuntu audio developer, responds on his blog with a summary of how PulseAudio jack detection has improved matters in Ubuntu 11.10, and what's left to do:
The good news: in the upcoming Ubuntu Oneiric (11.10), this is actually working. The bad news: it isn't working for everyone.
Making it easier to find software to handle a file (#28148) Ubuntu is not always as helpful as it could be when you don't have the right software installed to handle a particular file. Michael Vogt, one of the developers of the Ubuntu Software Center, responded to this. It seems that most of the pieces to make this work nicely are in place, but there are a few more bits of glue required:
Thanks a lot for this suggestion. I like the idea and it's something that software-center itself supports now. In the coming version 5.0 we will offer to "sort by top-rated" (based on the ratings&reviews data). It's also possible to search for an application based on its mime data. To search for a mime-type, you can enter "mime:text/html" or "mime:audio/ogg" into the search field. What is needed however is better integration into the file manager nautilus. I will make sure this gets attention at the next developer meeting and filed bug #860536 about it. In nautilus, there is now a button called "Find applications online" available as an option when opening an unknown file or when the user selects "open with...other application" in the context menu. But that will not use the data from software-center.
Show pop-up alert on low battery (#28037) Some users have reported on Brainstorm that they are not alerted frequently enough when their laptop's battery is low, as they clearly ought to be. This is an odd one, because there are already several power alert levels and this has been working well for us for some time. Nevertheless, enough people have voted for this idea that there must be something behind it, perhaps a bug that only affects certain systems. Martin Pitt, technical lead of the Ubuntu desktop team, has responded directly to the Brainstorm idea with a description of the current system and how to file a bug when it does not work as intended.

Vincent Sanders: Introduction to printing in another dimension

Mankind, it would appear, owe a great deal of their evolutionary advantage to using tools. This ability appears to have been massively amplified by our creation of machine tools.

A machine tool is widely defined to be a machine where the movement of the tool (the tool path) is not directly controlled by a human. One of the first known examples is a late 15th century lathe used to cut screw threads . The Industrial revolution was intimately interconnected with the creation of new machine tools and arguably by the mid 19th century all the distinct subtractive machine tool types had been discovered.

I ought to explain the word subtractive in this context, it is a pretty simple and rather arbitrary distinction (but important for this discussion). Traditional machining removes or subtracts material to obtain a finished item akin to a sculptor revealing the statute from within a block of stone by using a chisel and hammer. The corollary to this is, unsurprisingly, the additive process where material is added to create the finished item.

The machine tools from the 19th centuary were primarily single use devices controlled by gears and link mechanisms. Although the Jacquard loom was well known, because of the physical engineering difficulties, combining the concept with a machine tool to create a programmable tool path was not fully realised until the opening of the 20th century.

In the late 1940s electrical motors and punch cards/tape made machine tools Numerically Controlled (NC) and when computers arrived in the 60s we gained Computer Numerical Control (CNC) and the opportunity to completely screw things up with software became available.

With the advent of CNC additive systems became practical and by the late 1980s these machines were being widely used used for Rapid Prototyping.

The first additive systems generally used was the simple pen plotter which added ink on top of paper and became popular in draughting offices for producing blueprints etc. Though more generally thought of as computer printing technique plotters owe their heritage to CNC machines.

Next came prototyping systems based on layered object manufacture which cut shapes in a thin flat material (paper or plastic) and glued them together. These systems were expensive compared to casting processes (use a subtractive machine to make a mould and cast the part), extremely wasteful of source material and the results can be of variable quality. Systems based on this process are still manufactured and used.

Then came the stereolithography approach which scans a focused UV laser to cure resin and build up an object. There are several commercial machines available and even some home built systems but the costs of the resin have not yet made this approach generally cost effective.

Currently the most common commercial rapid prototyping additive systems are selective sintering processes where either an electron beam or a high power laser melt a layer of powdered material on a bed, the bed is lowered, more powder added and the process repeated. This process can use many different types of material and is very flexible as the power used can be plastic or metals. The quality is very high and high resolutions are available. Unfortunately these machines are expensive and generally start around 20,000 which puts them out of most individuals reach.

If anyone is still reading here is the summary of what we have covered so far:
  • Humans have used tools since they stopped being monkeys.
  • More than a century back we figured out how to make machines control the tools.
  • Fifty years back we made computers control the tools, before this all tools were subtractive.
  • In the last twenty years we have discovered several expensive ways to make objects with additive methods.
Now we get to the promise of the title, in the last few years Fused Filament Fabrication has become a viable option for a hobbyist. This method extrudes a thermoplastic through a nozzle and constructs an object one layer at a time from the bottom up.

The RepRap project at Bath university helped kickstart development of a plethora of practical operational 3D printers that can be built or bought. These machines are relatively inexpensive (starting from 400 if you build it yourself) and the feedstock is also reasonably inexpensive.

In another post I will discuss the actual practicalities of building and running one of these devices and looking at their software.



5 October 2011

Niels Thykier: Numbers and lintian

Lintian uses some small scripts to collect data from packages. In daily talk, they are usually referred to as collection scripts. Lintian uses files to track the status of collection scripts between runs. Consider the following directory listing on lintian.d.o: 
$ ls -al laboratory/binary/eclipse/.*-*
-rw-r--r-- 1 user group  45 Jul 18 01:21 laboratory/binary/eclipse/.ar-info-1
-rw-r--r-- 1 user group  45 Jul 18 01:21 laboratory/binary/eclipse/.bin-pkg-control-1
-rw-r--r-- 1 user group  45 Jul 18 01:21 laboratory/binary/eclipse/.changelog-file-1
-rw-r--r-- 1 user group  45 Jul 18 01:21 laboratory/binary/eclipse/.copyright-file-1
-rw-r--r-- 1 user group  45 Jul 18 01:21 laboratory/binary/eclipse/.debian-readme-1
-rw-r--r-- 1 user group  45 Jul 18 01:21 laboratory/binary/eclipse/.doc-base-files-1
-rw-r--r-- 1 user group  45 Jul 18 01:21 laboratory/binary/eclipse/.fields-1
[...]
This example shows that a bunch of collections scripts have been run for the eclipse binary package. Each of these files contain the version of Lintian that created them (or last wrote them) and a timestamp. Why the interest in these files? Let us go a bit back in time to the 15th of June 2011. That was the day where Lintian 2.5.1 was uploaded to unstable. That version of Lintian had 17 collection scripts[1] for binary packages. So every binary package would have 17 files and there are over 35 000 binary packages in the Debian archive. Ouch, so that makes about 595 000 files if we use this on all binary packages on the archive. The size of each of those files are about 45 bytes, so that is a total of 25.5 MB for all of these files[2]. So other than the inode abuse , this is not too bad. A little du -h should confirm this 
$ du -h laboratory/binary/eclipse/.*-*
4.0K    laboratory/binary/eclipse/.ar-info-1
4.0K    laboratory/binary/eclipse/.bin-pkg-control-1
4.0K    laboratory/binary/eclipse/.changelog-file-1
4.0K    laboratory/binary/eclipse/.copyright-file-1
4.0K    laboratory/binary/eclipse/.debian-readme-1
4.0K    laboratory/binary/eclipse/.doc-base-files-1
4.0K    laboratory/binary/eclipse/.fields-1
[...]
Whaaa oh the file system uses a block-size of 4K bytes, so I guess we have to pay a full block for these files. Let s see what that gives, 595 000 times 4 kB is 2.27 GB Oops! That was in the (not too distant) past. About a month later (12th of July), the code creating these files are refactored into: 
sub _mark_coll_finished  
    my ($self, $collname, $collver) = @_;
    # In the "old days" we would also write the Lintian version and the time
    # stamp in these files, but since we never read them it seems like overkill.
    #  - for the timestamp we could use the mtime of the file anyway
    return touch_file "$self-> base_dir /.$collname-$collver";
This turns out to be a very space-saving change if we ask du -h: 
$ du -h laboratory/binary/lintian/.*-*
0       laboratory/binary/lintian/.ar-info-1
0       laboratory/binary/lintian/.bin-pkg-control-1
0       laboratory/binary/lintian/.changelog-file-1
0       laboratory/binary/lintian/.copyright-file-1
0       laboratory/binary/lintian/.debian-readme-1
0       laboratory/binary/lintian/.doc-base-files-1
[...]
Existing files are not emptied, so we still have some old non-empty files left on lintian.d.o. Nevertheless that was the "nice story" about "the side effect of being lazy sometimes reduces space waste"[3]. But we have another "number" problem. If you grep for "Too many links" in the lintian.log from lintian.d.o you should see: 
$ grep -i "too many links" logs/lintian.log
mkdir: cannot create directory  /srv/lintian.debian.org/laboratory/binary/aces3': Too many links
mkdir: cannot create directory  /srv/lintian.debian.org/laboratory/binary/boinc-cgi-stripchart': Too many links
mkdir: cannot create directory  /srv/lintian.debian.org/laboratory/binary/libbrailleutils-java': Too many links
mkdir: cannot create directory  /srv/lintian.debian.org/laboratory/binary/libbrailleutils-java-doc': Too many links
mkdir: cannot create directory  /srv/lintian.debian.org/laboratory/binary/libclippoly0': Too many links
mkdir: cannot create directory  /srv/lintian.debian.org/laboratory/binary/libclippoly-dev': Too many links
mkdir: cannot create directory  /srv/lintian.debian.org/laboratory/binary/xul-ext-cookie-monster': Too many links
mkdir: cannot create directory  /srv/lintian.debian.org/laboratory/binary/cucumber': Too many links
[...]
$ grep -i "too many links" logs/lintian.log   wc -l
3274
In the Lintian Laboratory every package is unpacked in a directory based on its type and name (as hinted in the output above). The problem is that ext3 has a limit of the amount sub-directories a directory can have[4]. This limit is just shy of 32 000 and (as a reminder) Debian has over 35 000 binary packages. So a lot of packages are currently not checked on lintian.d.o (#641468). The current feature branch for #641468 already solves the directory limit issue (by using a "mirror-like" pool layout). It also removes the need for "1 file per completed collection" by writing this information in the existing "per entry" status file.
I hope we can get the branch merged into master within 2-3 weeks, though there are still a few issues that needs to be worked out before then. Footnotes: [1] Actually it had 18 collections, but one of them are "auto-removed" so it is better not to include it in this case. [2] Content alone - metadata like permissions and such is ignored. [3] I admit it was more luck than intention! [4] The limit comes from the "32 000 hardlink per inode" limitation.

Jonathan McDowell: Thanks for the offer, but...

I was due for another Google interview mail it seems. I have to say I wasn't expecting it, but this week I had a follow up to my polite mail from 6 months ago that said "No thanks, I'm not looking" asking if it was still the case. Normally I welcome this little bit of ego stroking; it's always nice to be wanted. Except that's not really the case, is it? It's an invitation to interview for something, not any indication that you've done more than tick some initial boxes. Google mails inevitably ask me if I'd like to work in SRE. It's always SRE. No one ever emails and asks if I want to work on self-replicating nanobots that will roam Mars searching for the perfect spot to build a beach house. And that's where things fall down. If someone currently has a job, then emailing them out of the blue to ask if they want to come and interview for something vague is hoping that they're either looking, and just haven't come to you yet, or not looking but unhappy enough with their current role that all they needed to start was an email asking them to submit a recent CV. For the former, if you're Google, do you really think that person doesn't know where to find you? For the latter, you're being quite presumptuous, aren't you? The act of updating my CV my be some effort (actually it's usually not, because the stuff that's not on it is the stuff I can't talk about because it's not released yet, or stuff that's specific and thus wouldn't go on a CV for a vague job spec). Even if it's not the act of interviewing is potentially a waste of time for both of us, if the role isn't clear. One argument used is that people will be placed according to the skills they show during the interview process. That's fine from the employers point of view, but if you're actively trying to get some interest from people who are gainfully employed you really need to grab their attention somehow. I can't remember the last time I had an unsolicited email interview offer that actually wowed me, or indeed even showed more than a passing sign of tailoring a spec to my profile. When I was running Black Cat I made a point of always replying to unsolicited CVs. How polite I was depended on how the covering emails were worded (a Word document with nothing else was likely to get short shrift, something well targeted in a Linux friendly format would normally get some comment about how we weren't hiring and were unlikely to be, but if that changed it would be mentioned on the website), but I felt people deserved a reply - I have been disappointed by not receiving responses myself to what I considered well targeted job inquiries. So far I've so far taken the same approach with mails from corporate recruiters (less so with recruiters that are associated with recruitment firms, rather than directly with the companies they are hiring for), but I'm starting to feel like changing that stance. Candidates are told to tailor CVs to the role being applied for, provide a decent cover letter, and in general make companies want to talk to them. Companies who are sending out recruitment emails should be held to the same standard. Even assuming you do a basic phone screen first, I can probably expect to need to take a day off work assuming that goes well. You need to convince me I can justify that before I'm going to feel like engaging at all. (And if I'm honest, based on what I've seen so far, it's unlikely to happen. All of the things I've considered have come from conversations with people I know directly about companies they own or work for, never some random contact via email. I try hard not to think of recruitment mail as spam, but I can how that line of thought follows through.) I should apologize to Google here. They got mentioned as an example, but I don't think they're particularly bad. I did interview with them at one point, and made the decision not to continue that process after deciding a different, more certain, path was better for me. So I've displayed interest. And in response to my reply today of "I know where to find you, so please assume I'll do so if I change my mind." they've said they'll make a note on their records.

4 October 2011

Michael Banck: 4 Oct 2011

Woodchuck and FrOSCon

At the end of August, I attended FrOSCon in Bonn again, after skipping it last year. The evening before FrOSCon however, I visited Neal Walfield, his wife Isabel and their little son Noam in D sseldorf. Besides having a great time and a lovely dinner, I was most impressed by their collection of Maemo devices (they had at least two N770s, an N900 and, to my jealousy, an N950) which Neal is doing research on these days. He works on woodchuck, which is a project investigating how to improve data availability on mobile devices and our conversation prompted him to implement ATP Woodchuck, which makes smarter decisions when to run APT upgrade on your Maemo device then the standard updater. As part of the research, they also run a user behaviour study which I joined, where one installs a client which records various data off your N900 and sends them anonymized (he seems to be doing a good job at that) to figure out how people use their mobile devices and hopefully enhance the experience. So if you have a N900, you should consider joining the study so they get better data.
The next day, I picked up Martin Michlmayr nearby and we headed for FrOSCon. I was quite impressed by the Makerbot at the Tarent booth, but I still don't know what they are really doing and why they had it on display... In the afternoon, I attended a couple of talks in the PostgreSQL developer room and a talk about a big OpenVPN deployment, before ending the day with the excellent as always social event barbeque. On Sunday, I went to quite a few talks, but I thought that two of them were particularly interesting:
Michael "Monty" Widenius of MySQL gave a talk titled "Why going open source will improve your product" about starting businesses on an open source project, or how business can/should open-source their product. Besides a detailed discussion about the various forms of Open Source licenses and the Open-Core model, he proposed the idea of "Business Source" (see slide 20 of his presentation), where a startup would distribute the source code under a non-commercial (but otherwise open-source) license with the explicit guarantee that the license would be changed to a true FLOSS license at some defined point in the future, giving the company a head start to develop and nurture their project. I asked whether this has been already implemented in practise and how the community could be sure that e.g. lawyers after a hostile takeover would not just remove that part of the copyright notice, as long as a true distribution under a FLOSS license has not happened yet. Monty wasn't aware of any real-word cases, and he did not seem to be concerned about this and said the original intent would be clear in a possible court case. This was the first time I heard about this approach, I wonder how other people think about it, whether it would work in practise and be a useful thing to have?
Second, I attended a talk by Gregor Geiermann, a Ph.D. student in linguistics on "Perceptions of rudeness in Free Software communities". He conducted an online survey about the perceived rudeness of several forum thread posts on Ubuntu Forums. Survey participants were first asked a couple of generic questions about their gender, nationality etc. and were then presented with a series of posts. For each post, they were asked to rate how rude they thought it was on a scale of 1 to 5 and they also had the possibility to highlight the parts of the post they considered rude as well as add comments. He presented a neat web application for analyzing the results, which makes it possible to select different groups (he did male vs. female and Americans vs. Germans in the talk) and have their overall rudeness ratings as well as the highlighted texts visualized as different shades of blue. Comments can be easily accessed. There were quite a few interesting differences e.g. in how Germans perceived rudeness compared to Americans (RTFM comments were considered less rude by Germans for example, IIRC). In response to my question, he said he intended to release the web application as open source and this might be an interesting tool for FLOSS projects to analyze how their public communication channels are perceived by various groups. Unfortunately, I cannot find any other resources about this on the web as of today, so I should try to contact him about it at some point.

3 October 2011

Tanguy Ortolo: Viewers should stay in foreground

Traditionally, when you start a program to view a document, for instance xpdf resume.pdf, it stays in foreground, and only returns when it finishes, when you are done with the document. Of course you can still run it in background if you want. Nowadays, several programs behave in a different way, for instance Evince. When you first start it, it stays in foreground as usual. However, if you launch it a second time while the first instance it still running, it will reuse this existing instance, asking it to open your document in a new window, and immediately return. Other examples are Firefox and LibreOffice.This behaviour is certainly more efficient in terms of memory usage and speed, but it is non-standard and causes subtle problems. When you open the attachment of an email message, or when you open a document from the Web without saving it permanently, the mailer or the browser saves the document to a temporary file, and then run a viewer with that file. As this is a temporary file, so they cannot leave it forever in /tmp; in fact they should delete it when you are done viewing it. I think the usual way to implement that is to delete it when the viewer returns: with a program that returns immediately, this means deleting it even before it has had a chance to load it. So, if you can no longer directly open a PDF attachment from Mutt, or if you are unable to view the compressed documentation of the LaTeX class Beamer with texdoc (which uncompresses it to a temporary file and calls a PDF viewer on it), you can blame your modern PDF viewer that returns before you are finished and start hacking your mailcap file. And if you are the author of such a program, please consider implementing a way to keep a lightweight process that does not return before the main process has closed the document or to disable process reuse.

Timo Jyrinki: From MeeGo to Tizen, Debian, ...?

The MeeGo community is frustrated with the news of the MeeGo brand being abandoned. Some are understandably angry or otherwise not happy about how Linux Foundation, Intel handled the Tizen announcement and community in general - or more like how they didn't handle it at all.Last week Openmind 2011 happened to be arranged in Tampere on the very same day as Tizen announcement came alive. It was good in the way that it lead to the fact that Nomovok's CEO Pasi Nieminen was able to initiate the "Reigniting MeeGo" session not just by talking vague things about future, but actually about the process which led to Tizen and the unfortunately brief initial PR about it. Pasi is intense on emphasizing the quality and role of Qt in Tizen as well, even though officially Tizen is all about HTML5 and apparently from Samsung's part at least EFL is provided as a native toolkit. However, the promise of Tizen compared to MeeGo is reportedly that the toolkit is not specified in compliancy documents, so HTML5 with WAC is the main/only "3rd party apps" layer whereas others can be offered case-by-case. This means that unlike before, the underlying system can be built on top of practically any distribution (theoretically) and using whatever toolkits and other techniques wanted. Obviously the "Nordic System Integrators" are probably all very keen of using Qt to produce more of Nokia N9 quality user experiences in various products.

Taking the corporate hat off, I as a community member am also puzzled. The only reason I was not completely blown by the news was that I didn't yet manage to get involved in MeeGo community on a daily basis, since I'm involved with a dozen communities already. Instead I've been more like scratching the surface with MeeGo Network Finland meetings, IRC activity, OBS usage for building a few apps for MeeGo Harmattan and MeeGo proper etc. But I can somewhat understand how people like Jarkko Moilanen from meego-fi feel. They have given a _lot_ to the MeeGo community and brand, all taken away without hearing or pre-notice.

So where to now for MeeGo community? Tizen is one obvious choice. However, for all the talks that even I started this post with, Tizen is still vaporware today, and the dislike of how community is being treated might make it easy to consider other options. Also, if Tizen's reference implementation has lesser meaning, it might also mean less to actually be "in" the Tizen community than in MeeGo. I met Jos Poortvliet at Openmind, and he invited people to openSUSE. There is a lot of common ground with MeeGo and openSUSE - strong OBS usage, RPM packaging, community side focused on KDE and therefore Qt.

I would like to now point similarly to Debian! If one is tired about corporate interests and not listening to community, there is no match for Debian's 15+ years history, purely volunteer based, trust based organization, and first of all scope. While openSUSE has traditionally focused on desktop (even though like Jos pointed out they are open to all new contributions and projects), Debian has always had the "universal" scope, ie. no boundaries besides producing free software operating system for various purposes. There are over 10 architectures maintained at the moment, including the ARM (different ports for ARMv4 and hard-float ARMv7) and x86 from MeeGo world. There are even alternative kernels to Linux, mainly the GNU/kFreeBSD port. There are multiple relevant plans and projects like the Smartphones wiki area, most noticeably Debian on Neo FreeRunner. I have run Debian on my primary mobile phone for over 2.5 years, although now in the recent months I've had dual-SIM in my Nokia N950 as well (Debian not yet running on Nokia N950 or Nokia N9 - but it can and will be done!).

What Debian may lack in both good and bad is corporate funding, if you don't count the still quite respectful contributions from Ubuntu to Debian (it's in Ubuntu's interests to contribute as much possible back to Debian, so that the delta remains small). For each and every aspect, it needs a volunteer - there are a thousand volunteer Debian Developers, and at least a double of that of people without the official DD status but who still maintain a package or two among the 25000+ packages in Debian. That means also that one my find it more lucrative to join a project that has paid people to do some of the "boring parts", more of fancy web tools, including for bug handling and build systems like the OBS (which I do love by the way). On the other hand, there is no other project in my opinion where what you do really matters as much.

To find out more about Debian from MeeGo perspective, please see the recent mailing list post Mobile UXes - From the DebConf11 BoF to the stars where I wrote most of the MeeGo (CE) part when I was asked to and known of my MeeGo involvement.

Last but not certainly least, there is the Mer project - originally "maemo reconstructed", ie. making Nokia's "not really distro" into a real distro by filling in the void places. Now it's obviously MeeGo reconstructed, and they aim to be the MeeGo they always wanted MeeGo to be! Read the post for details from Carsten Munk and other key Mer people. They share the love for Qt, and want the core to be as lean as possible. They also aim to incorporate the most community like aspect from MeeGo - MeeGo CE - as the reference vendor in Mer. They also aim to be Tizen compliant - and when Tizen comes alive, I wouldn't see why the Tizen reference implementation couldn't be used for saving resources. Maybe Nomovok and/or others could offer the Qt maintaining part.

So, it might be that Tizen itself is enough for most people's needs. The key point however in this post is not to fall in agony if one corporate based project takes big turns - it has happened before, it will happen in the future. There are always enough political and business reasons from some points of view to do Big Changes. But the wider community is out there, always, and it's bigger than you think. You should consider where you want to contribute by asking yourself why you are/were part of for example the MeeGo community. Aaron Seigo from KDE asked us all this question in the Openmind MeeGo Reignited session, and I think it's good to repeat.

2 October 2011

Neil Williams: Uses of Emdebian - special purpose computers

A continuation of my intermittent series on Emdebian and prompted by a query from Paul Wise, I thought I'd cover some of the uses to which a device running Emdebian can be put. It is a bit long this one...

So, yes, there are general purpose computers which could run Emdebian but there are common features amongst general purpose computers:
  1. Lots of different tasks must be supported - lots of different packages installed, each user could quite easily use a distinct set of packages from another user.

  2. Multi-user support - even if any one machine is only used by one person, a general purpose computer must have support for adding another user - not just a system user, a real user with a login, shell, possibly bookmarks and browser history etc.

  3. Connectivity - general purpose computers (and that includes mobile phones) must connect to a variety of other computers to be at all useful.

  4. Multi-modal data input - there needs to be support for a mouse and a keyboard or touchscreen and just as likely, software support for accessibility interfaces like an on screen keyboard and switch inputs.

  5. Storage for user data - the system admin can't know how much data users are going to add and if it is general purpose, it's probably going to support media players, image viewers, even office software. The data files managed and written by such software tend to be large and tend to accumulate into large collections. This means that a general purpose machine cannot anticipate the full storage requirements, so has to allow plenty of room for user data storage. If there's so much storage space available, why use an OS which is primarily designed to take up less space? Just use the full version, Debian.


Now compare with the common features of special purpose or single-purpose computers:
  1. Single task only - a splash screen (without support for removal) covers all the boot information, the single task starts as a daemon and if the user quits the task, the task initiates a shutdown, not quit. Every effort is made to prevent the task from exposing the underlying OS, including auto-restarts if it does crash.

  2. Single-user support - indeed, user is not the typical user. user in this case is a non-login user without a shell, created by the postinst of the task package purely to support running the task without root privileges (because if the task doesn't need privileges, it shouldn't retain privileges).

  3. Single mode input - don't expect these machines to support 105 key keyboards. You'll be lucky to have 5 keys which are really just buttons. The button won't just enter 'J' into the machine, it will fire off a command to charge your bank or start a drill or shutdown a malfunctioning service. (Each button will often be multi-functional as well, depending on context). Some devices will replace the mouse with a touchscreen but many are fitted in places where touchscreens simply won't survive, where keys have to be able to withstand being submerged in water (or oil) for hours. Many machines simply won't have mouse support at all. Many will not have any typical external connectors. Indeed, units which go into dirty or wet environments certainly won't have standard external connectors of any kind. Think boats and factories. Internal battery packs with custom hot-fit replacements, buttons encased in 4mm of clear plastic, screens viewable only through protective mesh / filters.

  4. Restricted connectivity - if the task doesn't need networking, no networking hardware is fitted. If the task doesn't need USB, no USB hardware is fitted. If the task doesn't need serial connections, there will probably be an internal serial connector but also a Warranty void if removed sticker. Don't expect an RS232 inside the case either, this will be a flat cable connector which you'll need to attach to a breakout board which correctly connects the cable wires to the RS232 port pins. That mapping is not standardised (why would it? only service engineers are approved to use it.)

  5. Constrained user data - the task is in complete control and determines what the user can view, store and access. Storage can be expensive and a power drain, so if the task doesn't need Gb of storage, don't fit the machine with Gb of storage. Profit.


So what are these mysterious single-use computers running Emdebian? It's hard to tell because the task has no particular reason to tell the user anything about the OS. Perhaps a different way to phrase the same question would be: Why use a general purpose OS for a special purpose computer?.


This is why Emdebian Grip is popular for these machines - because Emdebian Grip is binary compatible with the equivalent Debian suite and when a bug appears in the high level user interface, it is much easier to debug that on the desktop than on device. Debian stable is a fantastic OS for commercial development - the stability of the OS makes detection of interface bugs much easier. It is rare that developers have to investigate whether bugs in their UI come down to a bug in the underlying OS. That saves time.

The machines themselves? I'm sure others can come up with other examples but these are some which come to my mind (no particular order):


Anywhere where the device simply needs to DoTheRightThing in a variety of unpredictable circumstances, you're going to need a general purpose OS to gather the data and produce the right result. Anywhere where the device has only a single task, you're going to want to not provide access to the rest of the system. There's no point providing "general purpose access" when the device doesn't have "general purpose hardware". There's no point fitting general purpose hardware if the device does not use it. That is a waste of money, a waste of resources and causes unnecessary delays if one component or other becomes unobtainable or changes behaviour. Limit your exposure to hardware bugs and get the product out to the user more quickly and more reliably, everyone is happy. Fit non-standard hardware and you can fit custom hardware which better matches the profile of the device usage - why use standard speakers capable of CD quality across a vast frequency range when the device only makes very very loud alert noises which can be done at lower sample rates and with narrow range, high amplitude, speakers.

There are all sorts of estimates for the number of computers now in the average Western home. It's worth noting that the vast majority of those computers are not PCs or laptops or even game consoles and routers - all instances of general purpose machines with general purpose access / connectivity. A lot of the computers in your home are embedded in machines like white goods, media controllers (set top boxes and the like). Some of these will still use low level firmware written entirely in assembly or COBOL. More complex ones already have a general purpose OS inside yet constrain that OS with a simple, user-friendly interface which is tailored to the hardware actually fitted. It might be cool to connect to your set top box over Bluetooth but really, is that actually going to sell more units? Only if there are other high-end services written for the device which the average customer won't want to see pushing up the price of the base unit. Some machines (particularly in automotive uses) will need to be using Real Time operating systems and Safety Critical operating systems but even some of those are based on general purpose systems - just old versions which have been thoroughly tested.

Finally, consider that for a lot of these devices, the customer is not you. The customer is another business who then put the device into something else which is installed into a factory production line which produces something which goes into creating a consumer product you find in Tesco or on Amazon or bundled in with another service like your TV/broadband contract. There is absolutely no reason for these devices to provide general purpose computing to you, even if the device itself is part of something else which can provide such services.

1 October 2011

Dirk Eddelbuettel: Reminder: One week til Rcpp class in San Francisco

Just a quick note to remind everyone that the Rcpp class in San Francisco, which I am holding together with Revolution Analytics, will take place a week from today. We are happy to report that the number of registrations has met our initial targets. But as a number of open slots remain, we have decided to offer a few places at discounts of 25% for academics (with code acad1) and 50% for students (with code student). Course details are at the Revolution course page, registration is at the Eventbrite page. And just for completeness, here is what I wrote in the previous announcement:
The format will follow the workshop Romain and I gave during the tutorial day preceding this year's R/Finance conference. The style will once again be hands-on, with copious concrete examples and solid coverage of most aspects of Rcpp and related packages such as RInside, RcppArmadillo and others. The eight-hour schedule contains about six hours of instruction, split into four sessions of around ninety minutes. This leaves ample time for both lunch and coffee breaks, and for informal discussions and Q+A. The one-day class will be offered in San Franciso on Saturday, October 8, 2011. Please see the official course page for more details, concrete location info and maps as well as registration details.
Feel free to contact me at the usual email address with questions. Or with suggestions for the after-party in San Francisco :)

30 September 2011

Dirk Eddelbuettel: Rcpp 0.9.7

A fresh maintenance release version 0.9.7 of Rcpp went onto CRAN and into Debian earlier today. This release contains two contributed fixes. The first, suggested by Darren Cook via the rcpp-devel mailing list, corrects how we had set up exceptions specifications, reflecting a bit of Java-think on our part. The idiom is generally discouraged in C++, and we now conform. The second came in two excellent patches by R Core member Martyn Plummer which finally get us compilation on Solaris. This is much appreciated as our hands were tied here for lack of access to such a box. Otherwise, two new examples and a new unit test were added. The complete NEWS entry is below; more details are in the ChangeLog file in the package and on the Rcpp Changelog page. 
0.9.7   2011-09-29
    o   Applied two patches kindly provided by Martyn Plummer which provide
        support for compilation on Solaris using the SunPro compiler
    o   Minor code reorganisation in which exception specifiers are removed;
        this effectively only implements a run-time (rather than compile-time) 
        check and is generally seen as a somewhat depreated C++ idiom. Thanks
        to Darren Cook for alerting us to this issue.
    o   New example 'OpenMPandInline.r' in the OpenMP/ directory, showing how
        easily use OpenMP by modifying the RcppPlugin output
    o   New example 'ifelseLooped.r' showing Rcpp can accelerate loops that may
        be difficult to vectorise due to dependencies
    o   New example directory examples/Misc/ regrouping the new example as
        well as the fibonacci example added in Rcpp 0.9.6
    o   New Rcpp-FAQ example warning of lossy conversion from 64-bit long
        integer types into a 53-bit mantissa which has no clear fix yet.
    o   New unit test for accessing a non-exported function from a namespace
Thanks to CRANberries, you can also look at a diff to the previous release 0.9.6. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

28 September 2011

Erich Schubert: Privacy in the public opinion

Many people in the united states seem to have the opinion, that the "public is willing to give up most of their privacy" in particular when dealing with online services such as Facebook. I believe in his keynote at ECML-PKDD, Albert-L szl Barab si of Harvard University expessed such a view, that this data will just become more and more available. I'm not sure if it was him or someone else (I believe it was someone else) that essentially claimed "privacy is irrelevant". Another popular opinion is that "it's only old people caring for privacy".
However, just like politics, these things tend to oscillate from one extreme to another. For example, the recent years in Europe, conservative parties were winning one election after another. Now in France, the socialist parties have just won the senate, the conservative parties in Germany are losing in one state after the other and so on. And this will change back again, too. Democracy also lives from changing roles in government, as this drives both progress and fights corruption.
We might be seeing the one extreme in the united states right now, where people are readily giving away their location and interests for free access to a web site. This can swing back any time.
In Germany, one of the government parties - the liberal democrats, FDP - just dropped out of the Berlin city government, down to 1.8% of voters. Yes, this is the party the German foreign minister, Guido Westerwelle is from. The pirate party [en.wikipedia.org] - much of their program is about privacy, civil rights, copyright reforms and the internet - which didn't even participate in the previous elections since they were was founded just 5 years ago jumped to 8.9%, scoring higher than the liberal democrats did in the previous elections. In 2009 they scored a surprising high 2% in the federal elections - current polls see them anywhere from 4% to 7% at the federal level, so they will probably get seats in parliament in 2013. (There are also other reasons why the liberal democrats have been losing voters so badly, though! Their current numbers indicate they might drop out of parliament in 2013.)
The Greens in Germany, which are also very much oriented towards privacy and civil rights, are also on the rise, and in march just became the second strongest party and senior partner in the governing coalition of Baden-W rttemberg, which historically was a heartland of the conservatives.
So don't assume that privacy is irrelevant nowadays. The public opinion can swing quickly. In particular in democratic systems that have room for more than two parties - so probably not in the united states - such topics can actually influence elections a lot. Within 30 years, the Greens now frequently reach values of 20% in federal polls and up to 30% in some states. It doesn't look as if they are going to go away soon.
Also don't assume that it's just old people caring about privacy - in Germany, in particular the pirate party and the Greens are very much favored by the young people. The typical voter for the pirates is less than 30 years old, male, has a higher education and works in the media or internet business.
In Germany, much of the protest for more privacy - and against the too readily data collection by companies such as Facebook and Google - is driven by the young internet-users and -workers. I believe this will be similar in other parts of Europe - there are other pirate parties all over Europe. And this can happen to the united states any time, too.
Electronic freedom - e.g. pushed by the Electronic Frontier Foundation, but also the open source movement - does have quite a history in the united states. But in particular open source has made such a huge progress the last decade, these movements in the US could just be a bit out of breath right now. I'm sure they will come back with a strong push against the privacy invasions we're seeing right now. And that can likely take down a giant like Facebook, too. So don't bet on people continuing to give up their privacy!

23 September 2011

Matthew Garrett: Supporting UEFI secure boot on Linux: the details

An obvious question is why Linux doesn't support UEFI secure booting. Let's ignore the issues of key distribution and the GPL and all of those things, and instead just focus on what would be required. There's two components - the signed binary and the authenticated variables.

The UEFI 2.3.1 spec describes the modification to the binary format required to produce a signed binary. It's not especially difficult - you add an extra entry to the image directory, generate a hash of the entire binary other than the checksum, the certificate directory entry and the signatures themselves, encrypt that hash with your key and embed the encrypted hash in the binary. The problem has been that there was a disagreement between Microsoft and Intel over whether this signature was supposed to include the PKCS header or not, and until earlier this week the only widely available developer firmware (Intel's) was incompatible with the only widely available signed OS (Microsoft's). There's further hilarity in that the specification lists six supported hash algorithms, but the implementations will only accept two. So pretty normal, really. Developing towards a poorly defined target is a pain. Now that there's more clarity we'll probably have a signing tool before too long.

Authenticated variables are the other part of the puzzle. If a variable requires authentication, the operating system's attempt to write it will fail unless the new data is appropriately signed. The key databases (white and blacklists) are examples of authenticated variables. The signing actually takes place in userspace, and the handoff between the kernel and firmware is identical for both this case and the unauthenticated case. The only problem in Linux's support here is that our EFI variable support was written to a pre-1.0 version of the EFI specification which stated that variables had a maximum size of 1024 bytes, and this limitation ended up exposed to userspace. So all we really need to do there is add a new interface to let arbitrary sized variables be written.

Summary: We don't really support secure boot right now, but that's ok because you can't buy any hardware that supports it yet. Adding support is probably about a week's worth of effort at most.

comment count unavailable comments

22 September 2011

Axel Beckert: Emacs Macros: Repeat on Steroids

vi users have their . (dot) redo command for repeating the last command. The article Repeating Commands in Emacs in Mickey Petersen s blog Mastering Emacs explained Emacs equivalent for that, namely the command repeat, by default bound to C-x z. I though seldomly use it as I mostly have to repeat a chain of commands. What I use are so called Keyboard Macros. For example for the CVE-2011-3192 vulnerability in Apache I added a line like Include /etc/apache2/sites-common/CVE-2011-3192.conf to all VirtualHosts. So I started Emacs with all the relevant files: grep CVE-2011-3192 -l /etc/apache2/sites-available/*[^~] xargs emacs & To remove those Include lines again M-x flush-lines is probably the easiest way in Emacs. So for every file I had to call flush-lines with always the same parameter, save the buffer and then close the file or in Emacsish kill the buffer. So while working on the first file I recorded my doing as a keyboard macro:
C-x (
Start recording
M-x flush-lines<Enter>CVE-2011-3192<Enter>
flush all lines which contain the string CVE-2011-3192
C-x C-s
save the current buffer
C-x C-k<Enter>
kill the current buffer, i.e. close the file
C-x )
Stop recording
Then I just had to call the saved macro with C-x e. It flushed all lines, saved the changes and switched to the next remaining file by closing the current file with three key-strokes. And to make it even easier, from the second occasion on I only had to press e to call the macro directly again. So I just pressed e for a bunch of time and had all files edited. (In this case I used git diff afterwards to check that I didn t wreck anything by half-automating my editing. :-) Of course there are other ways to do this, too, e.g. use sed or so, but I still think it s a neat example for showing the power of keyboard macros in Emacs. More things you can do with Emacs Keyboard Macros are described in the EmacsWiki entry Keyboard Macros. And if you still miss vi s . command in Emacs, you can use the dot-mode, an Emacs mode currently maintained by Robert Wyrick which more or less automatically defines keyboard macros and lets you call them with C-..

21 September 2011

Ian Wienand: Exploring $ORIGIN

Anyone who works with building something with a few libraries will quickly become familiar with rpath's; that is the ability to store a path inside a binary for finding dependencies at runtime. Slightly less well known is that the dynamic linker provides some options for this path specification; one in particular is a path with the special variable $ORIGIN. ld.so man page has the following to say about the an $ORIGIN appearing in the rpath: 
  ld.so understands the string $ORIGIN (or equivalently $ ORIGIN )
  in  an  rpath specification to mean the directory containing the
  application  executable.  Thus,  an   application   located   in
  somedir/app   could   be  compiled  with  gcc  -Wl,-rpath,'$ORI 
  GIN/../lib' so that it finds an  associated  shared  library  in
  somedir/lib  no matter where somedir is located in the directory
  hierarchy.
As a way of a small example, consider the following: 
$ cat Makefile
main: main.c lib/libfoo.so
	gcc -g -Wall -o main -Wl,-rpath='$$ORIGIN/lib' -L ./lib -lfoo main.c
lib/libfoo.so: lib/foo.c
	gcc -g -Wall -fPIC -shared -o lib/libfoo.so lib/foo.c
$ cat main.c
void function(void);
int main(void)
 
   function();
   return 0;
 
$ cat lib/foo.c
#include <stdio.h>
void function(void)
 
   printf("called!\n");
 
$ make
gcc -g -Wall -fPIC -shared -o lib/libfoo.so lib/foo.c
gcc -g -Wall -o main -Wl,-rpath='$ORIGIN/lib' -L ./lib -lfoo main.c
$ ./main
called!
Now, wherever you should choose to locate main, as long as the library is in the right relative location it will be found correctly. If you are wondering how this works, examining the back-trace from the linker function that expands it is helpful: 
$ gdb ./main
(gdb) break _dl_get_origin
Function "_dl_get_origin" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (_dl_get_origin) pending.
(gdb) r
Starting program: /tmp/test-origin/main 
Breakpoint 1, _dl_get_origin () at ../sysdeps/unix/sysv/linux/dl-origin.c:37
37	../sysdeps/unix/sysv/linux/dl-origin.c: No such file or directory.
	in ../sysdeps/unix/sysv/linux/dl-origin.c
(gdb) bt
#0  _dl_get_origin () at ../sysdeps/unix/sysv/linux/dl-origin.c:37
#1  0x00007ffff7de61f4 in expand_dynamic_string_token (l=0x7ffff7ffe128, 
    s=<value optimized out>) at dl-load.c:331
#2  0x00007ffff7de62ea in decompose_rpath (sps=0x7ffff7ffe440, 
    rpath=0xffffffc0 <Address 0xffffffc0 out of bounds>, l=0x0, 
    what=0x7ffff7df780a "RPATH") at dl-load.c:554
#3  0x00007ffff7de7051 in _dl_init_paths (llp=0x0) at dl-load.c:722
#4  0x00007ffff7de2124 in dl_main (phdr=0x7ffff7ffe6b8, 
    phnum=<value optimized out>, user_entry=0x7ffff7ffeb28) at rtld.c:1391
#5  0x00007ffff7df3647 in _dl_sysdep_start (
    start_argptr=<value optimized out>, dl_main=0x7ffff7de14e0 <dl_main>)
    at ../elf/dl-sysdep.c:243
#6  0x00007ffff7de0423 in _dl_start_final (arg=0x7fffffffe7c0) at rtld.c:338
#7  _dl_start (arg=0x7fffffffe7c0) at rtld.c:564
#8  0x00007ffff7ddfaf8 in _start () from /lib64/ld-linux-x86-64.so.2
Just from a first look we can divine that this has found the RPATH header in the dynamic section and has decided to expand the string $ORIGIN. 
$ readelf --dynamic ./main
Dynamic section at offset 0x7d8 contains 23 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libfoo.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/lib]
 0x000000000000000c (INIT)               0x4004d8
 0x000000000000000d (FINI)               0x4006f8
At this point it proceeds to do a readlink on /proc/self/exe to get the path to the currently running binary, effectively does a basename on it and replaces the value of $ORIGIN. Interestingly, if this should fail, it will fall back on the environment variable LD_ORIGIN_PATH for unknown reasons. This might be useful if you were in a bad situation where /proc was not mounted and you still had to run your binary, although you could probably achieve the same thing via the use of LD_LIBRARY_PATH just as well. This feature should be used with some caution to avoid turning your application in a mess that can't be packaged in any standard manner. One common example of use is probably your java virtual machine to find its implementation libraries. 
$ readelf --dynamic /usr/lib/jvm/java-6-sun/bin/java
Dynamic section at offset 0x9a28 contains 25 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libjli.so]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [lib.so]
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/../lib/amd64/jli:$ORIGIN/../jre/lib/amd64/jli]
 ...
There is a small but interesting security issue. If an attacker can hardlink to your binary that is using $ORIGIN then any path expansion is now relative to the attackers directory. Hence it is possible to make the linker read in arbitrary libraries and hence run arbitrary code. Clearly if your program is running setuid as someone else, such as root, you've just given up the keys to the house! Luckily, the dynamic linker will not let you shoot yourself in the foot like this, and prevents expansion of the $ORIGIN field if the program is running setuid. 
$ sudo chown root ./main
$ sudo chmod +s ./main
$ ./main
./main: error while loading shared libraries: libfoo.so: cannot open shared object file: No such file or directory
However, this is a good example of why you might want to keep users home and temp directories on separate file systems away from where binaries live. There is also a similar variable $PLATFORM, which describes the current running system. This is gained via the auxiliary vector, which I have written about here and here. Setting LD_SHOW_AUXV=1 will allow you to examine this value. 
$ LD_SHOW_AUXV=1 ls
...
AT_PLATFORM:     x86_64
...

19 September 2011

Joey Hess: happy haskell hacker

There are certian things haskell is very good at, and I had the pleasure of using it for one such thing yesterday. I wanted to support expressions like find(1) does, in git-annex. Something like: 
git-annex drop --not --exclude '*.mp3' --and \
    -\( --in usbdrive --or --in archive -\) --and \
    --not --copies 3
So, parens and booleans and some kind of domain-specific operations. It's easy to build a data structure in haskell that can contain this sort of expression. 
 - A Token can either be a single word, or an Operation of an arbitrary type. - 
data Token op = Token String   Operation op
        deriving (Show, Eq)
data Matcher op = Any
          And (Matcher op) (Matcher op)
          Or (Matcher op) (Matcher op)
          Not (Matcher op)
          Op op
        deriving (Show, Eq)
(The op could just be a String, but is parameterized for reasons we'll see later.) As command-line options, the expression is already tokenised, so all I needed to do to parse it is consume a list of the Tokens. The only mildly tricky thing is handling the parens right -- I chose to not make it worry if there were too many, or too few closing parens. 
generate :: [Token op] -> Matcher op
generate ts = generate' Any ts
generate' :: Matcher op -> [Token op] -> Matcher op
generate' m [] = m
generate' m ts = uncurry generate' $ consume m ts
consume :: Matcher op -> [Token op] -> (Matcher op, [Token op])
consume m [] = (m, [])
consume m ((Operation o):ts) = (m  And  Op o, ts)
consume m ((Token t):ts)
          t == "and" = cont $ m  And  next
          t == "or" = cont $ m  Or  next
          t == "not" = cont $ m  And  (Not next)
          t == "(" = let (n, r) = consume next rest in (m  And  n, r)
          t == ")" = (m, ts)
          otherwise = error $ "unknown token " ++ t
        where
                (next, rest) = consume Any ts
                cont v = (v, rest)
Once a Matcher is built, it can be used to check if things match the expression the user supplied. This next bit of code almost writes itself. 
 - Checks if a Matcher matches, using a supplied function to check
 - the value of Operations. - 
match :: (op -> v -> Bool) -> Matcher op -> v -> Bool
match a m v = go m
        where
                go Any = True
                go (And m1 m2) = go m1 && go m2
                go (Or m1 m2) = go m1   go m2
                go (Not m1) = not (go m1)
                go (Op o) = a o v
And that's it! This is all nearly completly generic and could be used for a great many things that need support for this sort of expression, as long as they can be checked in pure code. A trivial example: 
*Utility.Matcher> let m = generate [Operation True, Token "and", Token "(", Operation False, Token "or", Token, "not", Operation False, Token ")"]
*Utility.Matcher> match (const . id) m undefined 
True
For my case though, I needed to run some IO actions to check if expressions about files were true. This is where I was very pleased to see a monadic version of match could easily be built. 
 - Runs a monadic Matcher, where Operations are actions in the monad. - 
matchM :: Monad m => Matcher (v -> m Bool) -> v -> m Bool
matchM m v = go m
        where
                go Any = return True
                go (And m1 m2) = liftM2 (&&) (go m1) (go m2)
                go (Or m1 m2) =  liftM2 ( ) (go m1) (go m2)
                go (Not m1) = liftM not (go m1)
                go (Op o) = o v
With this and about 100 lines of code to implement specific tests like --copies and --in, git-annex now supports the example at the top. Just for comparison, find(1) has thousands of lines of C code to build a similar parse tree from the command line parameters and run it. Although I was surprised to see that it optimises expressions by eg, reordering cheaper tests first. The last time I wrote this kind of thing was in perl, and there the natural way was to carefully translate the expression into perl code, which was then evaled. Meaning the code was susceptible to security holes. Anyway, this is nothing that has not been done a hundred times in haskell before, but it's very nice that it makes it so clean, easy, and generic.

Sylvain Beucler: Building Android NDK samples

I searched forever for a way to just build the examples in the Android NDK, preferably from the command line. Here's how. First step: build the C++ code: 
cd /usr/src/android-ndk-r6/samples/san-angeles/
/usr/src/android-ndk-r6/ndk-build
Simple enough. Second step:
You will then need to generate your final .apk through the usual means. -- Android NDK Dev Guide
Thanks a ****ing lot! In addition the documentation is very much Eclipse-centered - and I can't be bothered to install and configure all the Eclipse stack for a mere Java wrapper for the above-generated .so file XD We first need to use the 'android' wrapper and download the "SDK Platform Android 2.2, API 8, revision 3" - aka "android-8", the first environment to support OpenGL ES 2. 
/usr/src/android-sdk-linux_x86/tools/android
It appears we first need to "manage the project", more precisely import from an existing project: http://developer.android.com/guide/developing/projects/projects-cmdline.html So we create a project by importing the sources from the current directory: 
/usr/src/android-sdk-linux_x86/tools/android update project \
  --name san-angeles --path . --target "android-8"
Then we use ant to build the thing: http://developer.android.com/guide/developing/building/building-cmdline.html 
apt-get install ant openjdk-6-jdk
ant debug  # avoid messing with signing
We now have a bin/san-angeles-debug.apk and we can run it in the emulator. Create a new "Virtual device" using the "android-8" platform we downloaded above, and launch it. Given the time needed to boot the OS, I suggest enabling snapshots. Scaling to display resulted in a very tiny, dark and slow display, in a first step I suggest using a non-scaled one. 
/usr/src/android-sdk-linux_x86/tools/android
At last we can run it! Install it in the emulator: 
/usr/src/android-sdk-linux_x86/platform-tools/adb install -r bin/san-angeles-debug.apk
You'll find a DemoActivity in your apps directory :)
Note: I tried to compile the "hello-gl2", showcasing OpenGL ES 2.0, but sadly it crashes:
Sorry! The application GL2JNI (process com.android.gl2jni) has stopped unexpectedly. Please try again.
The emulator apparently doesn't advertise EGL_OPENGL_ES2_BIT (checked in in GL2JNIView.java). If you try to by-pass it, you'll get loads of:
E/libEGL ( 333): called unimplemented OpenGL ES API
It's worth noting the OpenGLES20 sample from the "API Demos" application uses a fall-back OpenGL ES 1 renderer if v2 is not available - it made me mistakenly believe that OpenGL ES 2 was supported only from Java >( OpenGL ES 2 is not supported in the emulator as of 2011-09-17 - period.

17 September 2011

Patrick Schoenfeld: Struggling with Advanced Format during a LVM to RAID migration

Recently I decided to invest in another harddisk for my atom system. That system, I built up almost two years ago, has become the central system in my home network, serving as a fileserver to host my personal data, some git repositories etc., streaming server and since I switched to a cable internet connection it also serves as a router/firewall.
Originally, I bought that disk to backup some data, of the systems in the network, but I realized that all data on this system were hosted on a single 320GB 2,5" disk and it became clear to me, that, in absense of a proper backup strategy, I at least should provide some redundancy.


So I decided, once the disk was in place, that the whole system should move to a RAID1 over the two disks. Basically this is not that hard as it may seem at a first glance, but I had some problems due to a new sector size in some recent harddisks, which is called Advanced Format.


But lets begin from the start. The basic idea of such a migration is:
  1. Install mdadm with apt-get. Make sure to answer 'all' to the question which devices need to be activated in order to boot the system.
  2. Partition the new disk (almost) identical.
    Because the new drive is somewhat bigger that wouldn't make sense, but at least the two partitions which should be mirrored on the second disk, need to be identical.
    Usually this is achieved easily by using
    sfdisk -d /dev/sda sfdisk /dev/second/sdb

    In this case, it wasn't that easy. But I will come to that in a minute.
  3. Change the type of the partitions to 'FD' (Linux RAID autodetect) with fdisk
  4. Erase evidence of an eventual old RAID from the partitions, which is probably pointless on a brand-new disk, but we want to be sure:
    mdadm --zero-superblock /dev/sdb1
    mdadm --zero-superblock /dev/sdb2
  5. Create two DEGRADED raid1 arrays from the partitions:
    mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 missing
    mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdb2 missing
  6. Create filesystem on the first raid device, which will become /boot.
  7. Mount that filesystem somewhere temporary and move the contents of /boot to it:
    mount /dev/md0 /mnt/somewhere
  8. Unmount /boot, edit fstab to mount /boot from /dev/md0 and re-mount /boot (from md0)
  9. Create mdadm configuration with mdadm and append it to /etc/mdadm/mdadm.conf:
    mdadm --examine --scan >> /etc/mdadm/mdadm.conf
  10. Update the initramfs and grub (no manual modification needed with grub2 on my system)
    and install grub into the MBR of the second disk.

    update-initramfs -u
    update-grub
    grub-install /dev/sdb
  11. The first point to pray: Reboot the system to verify it can boot from the new /boot.
  12. Create a physical volume on /dev/md1:
    pvcreate /dev/md1
  13. Extend the volume group to contain that device:
    vgextend <volgroup_name> /dev/md1
  14. Move the whole volume group physically from the first disk to the degraded RAID:
    vgmove <volgroup_name> /dev/md1

    (Wait for it to complete... takes some time ;)
  15. Reduce first disk from the VG:
    vgreduce <volgroup_name> /dev/sda2
  16. Prepare it for addition to the RAID (see step 3 and 4) and add it:
    mdadm --add /dev/md0 /dev/sda1
    mdadm --add /dev/md1 /dev/sda2
  17. Hooray! Watch into /proc/mdstat. You should see that the RAID is recovering.
  18. When recovery is finished pray another time and hope that system is still booting with it running from the RAID entirely. If it does: Finished :-)

Now to the problem with the advanced format:
There is some action taking place with the hardware vendors to move to a new sector size. Physically my new device has a size of 4096 bytes per sector. Somewhat different to the 512 bytes disks used to have the last decade.


Logically it still has 512 bytes per sector. As far as I understand this is achieved by placing 8 logical sectors into one physical sector, so when partitioning a new disk the alignment of the disk has to be so that partitions start in a sector which is a multiple of 8.

That, obviously, wasn't the case with the old partitioning on my first disk. So I had to manually create partitions by specifying start points manually and making sure they are dividable by 8.

Otherwise fdisk would complain about the layout on the disk.
This does not work with cfdisk, because it does not accept manual alignment parameters and unfortunately the partitions it creates do have a wrong alignment. So good old fdisk and some calculations how many sectors are needed and where to start, to the rescue.

So the layout is now:
Device Boot Start End Blocks Id System
/dev/sdb1 2048 291154 144553+ fd Linux raid autodetect
/dev/sdb2 291160 625139334 312424087+ fd Linux raid autodetect


Next.

Previous.