Introduction As much as my provider may have gold standards on how to do things (they don't, there are some very bad practises in the default installation, like putting their SSH key into root's authorized_keys file), I wouldn't trust an installation done by a third party. Also, I wanted to have all my data securely encrypted. I know this is not perfect, and there are possible attacks. But I think it is a good barrier to have to deter entities without big budgets from getting my data. I have done this twice on my servers, and today I was reviewing each step as he was doing the same thing (with some slight differences) on his brand new server, so I think this is all mostly correct. Please, tell me if you find a bug in this guide. This was done on my 12 /month Kimsufi dedicated server, sold by OVH (see my previous post on why I chose it), and some things are specific to them. But you can do the same thing with any dedicated server that has a rescue netboot image. The process is to boot into the rescue image (this is of course a weak link, as the image could have a keylogger, but we have to stop the paranoia at some point), manually partition the disk, set-up encryption, and LVM; and then install a Debian system with debootstrap. To be able to unlock the encrypted disks, you will have to ssh into the server after a reboot and enter the passphrase (this is done inside the initrd phase). Once unlocked, the normal boot process continues. If anything fails, you end up with an unreachable system: it might or might have not booted, the disk might or might not be unlocked, etc. You can always go back into the rescue netboot image, but that does not allow you to see the boot process. Some providers will give you real remote console access, OVH charges you silly money for that. They used to offer a "virtual KVM", which was a bit of a kludge, but it worked: another netboot image that started a QEMU connected to a VNC server, so by connecting to the VNC server, you would be able to interact with the emulated boot process, but with a fake BIOS and a virtual network. For some unspecified reason they've stopped offering this, but there is a workaround available. The bottom line is, if you have some kind of rescue netboot image, you can just download and run QEMU on it and do the same trick.

The gritty details Start by netbooting into your rescue image. For OVH, you'd go to the control panel, in the Services/Netboot section and select "rescue pro". Then reboot your server. OVH will mail you a temporary password when it finishes rebooting. Connect to it, without saving the temporary SSH key:

$ ssh -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no root@$ IP 

For the rest of the text, I am assuming you have one hard drive called /dev/sda. We start by partitioning it:

# fdisk /dev/sda

Start a new partition table with o, and then create two primary partitions: a small one for /boot at the beginning (100 to 300 MB would do), and a second one with the remaining space. Set both as type 83 (Linux), and don't forget to activate the first one, as this servers refuse to boot from the hard drive without that. Create the file system for /boot, and the encrypted device:

# mkfs.ext4 /dev/sda1
# cryptsetup -s 512 -c aes-xts-plain64 luksFormat /dev/sda2

The encryption parameters are the same as the ones used by the Debian Installer by default, so don't change them unless you really know what you are doing. You will need to type a passphrase for the encrypted device, be sure not to forget it! This passphrase can later be changed (or secondary passphrases added) with the cryptsetup tool. Look up the crypt device's UUID, and save it for later:

# cryptsetup luksDump /dev/sda2   grep UUID:
UUID:           xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Open the encrypted device (type the passphrase again), and set up the LVM volume group:

# cryptsetup luksOpen /dev/sda2 sda2_crypt
# pvcreate /dev/mapper/sda2_crypt
# vgcreate vg0 /dev/mapper/sda2_crypt

Create the logical volumes, this is of course a matter of personal taste and there are many possible variations. This is my current layout, note that I put most of the "big data" in /srv.

# lvcreate -L 500m -n root vg0
# lvcreate -L 1.5g -n usr vg0
# lvcreate -L 3g -n var vg0
# lvcreate -L 1g -n home vg0
# lvcreate -L 10g -n srv vg0
# lvcreate -L 500m -n swap vg0
# lvcreate -L 100m -n tmp vg0

Some possible variations:

• You can decide to use a ramdisk for /tmp, so instead of creating a logical volume, you would add RAMTMP=yes to /etc/default/tmpfs.
• You can merge / and /usr in one same partition, as neither of them change much.
• You can avoid having swap if you prefer.
• You can put /home in /srv, and bind mount it later.

Now, create the file systems, swap space, and mount them in /target. Note that I like to use human-readable labels.

# for i in home root srv tmp usr var; do 
  mkfs.ext4 -L $i /dev/mapper/vg-$i; done
# mkswap -L swap /dev/mapper/vg0-swap
# mkdir /target
# mount /dev/mapper/vg0-root /target
# mkdir /target/ boot,home,srv,tmp,usr,var 
# mount /dev/sda1 /target/boot
# for i in home srv tmp usr var; do
  mount /dev/mapper/vg-$i /target/$i; done
# swapon /dev/mapper/vg-swap

Don't forget to set the right permissions for /tmp.

# chmod 1777 /target/tmp

If you want to do the /home on /srv, you'll need to do this (and then copy to /etc/fstab):

# mkdir /target/srv/home
# mount -o bind /target/srv/home /target/home

The disk is ready now. We will use debootstrap to install the base system. The OVH image carries it, otherwise consult the relevant section in the Install manual for details. It is important that at this point you check that you have a good GPG keyring for debootstrap to verify the installation source, by comparing it to a good one (for example, the one in your machine):

# gpg /usr/share/keyrings/debian-archive-keyring.gpg
pub  4096R/B98321F9 2010-08-07 Squeeze Stable Release Key <debian-release@lists.debian.org>
pub  4096R/473041FA 2010-08-27 Debian Archive Automatic Signing Key (6.0/squeeze) <ftpmaster@debian.org>
pub  4096R/65FFB764 2012-05-08 Wheezy Stable Release Key <debian-release@lists.debian.org>
pub  4096R/46925553 2012-04-27 Debian Archive Automatic Signing Key (7.0/wheezy) <ftpmaster@debian.org>

Now, for the actual installation. You can use any Debian mirror, OVH has their own in the local network. In OVH's case it is critical to specify the architecture, as the rescue image is i386. I didn't notice that and had to painfully switch architectures in place (which was absolutely not possible a couple of years ago).

# debootstrap --arch amd64 wheezy /target http://debian.mirrors.ovh.net/debian

After a few minutes downloading and installing stuff, you almost have a Debian system ready to go. Since this is not D-I, we still need to tighten a few screws manually. Let's mount some needed file systems, and enter the brand new system with chroot:

# mount -o bind /dev /target/dev
# mount -t proc proc /target/proc
# mount -t sysfs sys /target/sys
# XTERM=xterm-color LANG=C.UTF-8 chroot /target /bin/bash

The most critical parts now are to correctly save the parameters for the encrypted device, and the partitions and logical volumes. You'll need the UUID saved before:

# echo 'sda2_crypt UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx none luks' \
  > /etc/crypttab

Create the file systems table in /etc/fstab. Here I use labels to identify the devices:

# file system   mount point type    options             dump    pass
LABEL=root      /           ext4    errors=remount-ro   0       1
LABEL=tmp       /tmp        ext4    rw,nosuid,nodev     0       2
LABEL=var       /var        ext4    rw                  0       2
LABEL=usr       /usr        ext4    rw,nodev            0       2
LABEL=home      /home       ext4    rw,nosuid,nodev     0       2
# Alternative home in /srv:
#/srv/home      /home       auto    bind                0       0
LABEL=srv       /srv        ext4    rw,nosuid,nodev     0       2
LABEL=boot      /boot       ext4    rw,nosuid,nodev     0       2
LABEL=swap      none        swap    sw                  0       0

You can also just use the device mapper's names (/dev/mapper/<volume_group>-<logical_volume>), be sure not to use the /dev/<volume_group>/<logical_volume> naming, as there are some initrd-tools that choked on them.

# file system           mount point type    options             dump    pass
/dev/mapper/vg0-root    /           ext4    errors=remount-ro   0       1
/dev/mapper/vg0-tmp     /tmp        ext4    rw,nosuid,nodev     0       2
/dev/mapper/vg0-var     /var        ext4    rw                  0       2
/dev/mapper/vg0-usr     /usr        ext4    rw,nodev            0       2
/dev/mapper/vg0-home    /home       ext4    rw,nosuid,nodev     0       2
# Alternative home in /srv:
#/srv/home              /home       auto    bind                0   0
/dev/mapper/vg0-srv     /srv        ext4    rw,nosuid,nodev     0   2
/dev/sda1               /boot       ext4    rw,nosuid,nodev     0   2
/dev/mapper/vg0-swap    none        swap    sw                  0   0

Some tools depend on /etc/mtab, which now is just a symbolic link:

# ln -sf /proc/mounts /etc/mtab

Now configure the network. You most surely can use DHCP, but you might prefer static configuration, that's personal choice. For DHCP, it is very straightforward:

# cat >> /etc/network/interfaces
auto eth0
iface eth0 inet dhcp

For static configuration, first find the current valid addresses and routes as obtained by DHCP:

# ip address
# ip route

And then store them:

# cat >> /etc/network/interfaces
auto eth0
iface eth0 inet static
    address AAA.BBB.CCC.DDD/24
    gateway AAA.BBB.CCC.254
    pre-up /sbin/ip addr flush dev eth0   true

Note that pre-up command I added; that is to remove the configuration that is to be done by the kernel during boot (more on that later), otherwise ifupdown will complain about existing addresses. If your provider does IPv6, add it too. For OVH, the IPv6 set-up is a bit weird, so you need to add the routes in post-up. Your default gateway is going to be your /64 prefix, with the last byte replaced by ff, and then followed by :ff:ff:ff:ff. As you can see, that gateway is not in your network segment, so you need to add an explicit route to it. They have some information, but it is completely unreadable. If your IPv6 address is 2001:41D0:1234:5678::1/64, you will add:

iface eth0 inet6 static
    address 2001:41D0:1234:5678::1/64
    post-up /sbin/ip -6 route add 2001:41D0:1234:56ff:ff:ff:ff:ff dev eth0
    post-up /sbin/ip -6 route add default via 2001:41D0:1234:56ff:ff:ff:ff:ff

You probably don't want the auto-configured IPv6 addresses, so disable them via sysctl:

# cat >> /etc/sysctl.conf
# Disable IPv6 autoconf 
net.ipv6.conf.all.autoconf = 0
net.ipv6.conf.default.autoconf = 0
net.ipv6.conf.eth0.autoconf = 0
net.ipv6.conf.all.accept_ra = 0
net.ipv6.conf.default.accept_ra = 0
net.ipv6.conf.eth0.accept_ra = 0

To have a working DNS resolver, we can use the local server (OVH in this case):

# cat > /etc/resolv.conf 
search $DOMAIN
nameserver 213.186.33.99

The most important part of a new install: choose a host name (and make the system use it).

# echo $HOSTNAME > /etc/hostname
# hostname $HOSTNAME
# echo "127.0.1.1 $HOSTNAME.$DOMAIN $HOSTNAME" >> /etc/hosts

If we want to speficy the BIOS clock to use UTC:

# echo -e '0.0 0 0.0\n0\nUTC' > /etc/adjtime

Set up your time zone:

# dpkg-reconfigure tzdata

Configure APT with your preferred mirrors. I also prevent APT from installing recommends by default.

# echo deb http://ftp2.fr.debian.org/debian wheezy main contrib non-free \
  >> /etc/apt/sources.list
# echo deb http://ftp2.fr.debian.net/debian wheezy-updates main contrib non-free \
  >> /etc/apt/sources.list
# echo deb http://security.debian.org/ wheezy/updates main contrib non-free \
  >> /etc/apt/sources.list
# echo 'APT::Install-Recommends "False";' > /etc/apt/apt.conf.d/02recommends
# apt-get update

Before installing any package, let's make sure that the initial ram disk (initrd) that is going to be created will allow us to connect. There will be no chance of using the root password during boot. Your public key is usually found in $HOME/.ssh/id_rsa.pub.

# mkdir -p /etc/initramfs-tools/root/.ssh/
# echo $(YOUR_PUB_RSA_KEY) > /etc/initramfs-tools/root/.ssh/authorized_keys

If you change this, or the host key stored at /etc/dropbear/dropbear_*_host_key, the /etc/crypttab, or any other critical piece of information for the booting process, you need to run update-initramfs -u. Now we can install the missing pieces:

# apt-get install makedev cryptsetup lvm2 ssh dropbear busybox ssh \
  initramfs-tools locales linux-image-amd64 grub-pc kbd console-setup

During the installation you will have to choose where to install grub, I recommend directly on /dev/sda. Also, the magic initrd will be created. We want to double check that it has all the important pieces for a successful boot:

# zcat /boot/initrd.img-3.2.0-4-amd64   cpio -t conf/conf.d/cryptroot \
  etc/lvm/lvm.conf etc/dropbear/\* root/.ssh/authorized_keys sbin/dropbear
etc/lvm/lvm.conf
etc/dropbear/dropbear_dss_host_key
etc/dropbear/dropbear_rsa_host_key
sbin/dropbear
root/.ssh/authorized_keys
conf/conf.d/cryptroot

All these files need to be there. Most critically, we need to check that the cryptroot file has the right information to access the root file system:

# zcat /boot/initrd.img-*   cpio -i --to-stdout conf/conf.d/cryptroot
target=sda2_crypt,source=UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx,key=none,rootdev,lvm=vg0-root

If all that was correct, now we need to tell the kernel to configure the network as soon as possible so we can connect to the initrd and unlock the disks. This is done by passing a command-line option though grub. This should match what was done in /etc/network/interfaces: either DHCP or static configuration. For DHCP, this line should be changed in /etc/default/grub:

GRUB_CMDLINE_LINUX="ip=:::::eth0:dhcp"

For static configuration:

GRUB_CMDLINE_LINUX="ip=MY_IP_ADDR::MY_DEFAULT_GW:MY_NETMASK::eth0:none"

It is also a good idea to disable the quiet boot and graphical boot splash, in case we need to use QEMU to fix some booting issue:

GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_TERMINAL=console

And make the changes effective:

# update-grub2

Having fsck fix problems automatically can be a life-saver too:

# echo FSCKFIX=yes >> /etc/default/rcS

Get some very useful packages:

# apt-get install vim less ntpdate sudo

Create an user for yourself, possibly make it an administrator:

# adduser tincho
# adduser tincho sudo
# adduser tincho adm

This is mostly done, exit the chroot, and unmount everything.

# exit  # the chroot.
# umount /target/ dev,proc,sys,boot,home,srv,tmp,usr,var 
# umount /target
# swapoff -a
# lvchange -an /dev/mapper/vg0-*
# cryptsetup luksClose sda2_crypt

Disable the netboot option from your administration panel, reboot, and hope it all goes well. If you followed every step carefully, a few minutes later you should be able to ping your server. Use this snippet to enter the password remotely:

$ stty -echo; ssh -o UserKnownHostsFile=$HOME/.ssh/known_hosts.initramfs \
  -o BatchMode=yes root@"$HOST" 'cat > /lib/cryptsetup/passfifo'; \
  stty echo

It is very important that you close the pipe (with control-D twice) without typing enter. For my servers, I have a script that reads the passphrase from a GPG-encrypted file and pipes it directly into the remote server. That way, I only type the GPG passphrase locally:

$ cat unlock.sh 
#!/bin/sh
BASE="$(dirname "$0")"
HOST="$1"
gpg --decrypt "$BASE"/key-"$HOST".gpg   \
    ssh -o UserKnownHostsFile="$BASE"/known_hosts.initramfs -o BatchMode=yes \
        root@"$HOST" 'cat > /lib/cryptsetup/passfifo'

It might be a good idea to create a long, impossible to guess passphrase that you can use in the GPG-encrypted file, and that you can also print and store in somewhere safe. See the luksAddKey function in the cryptsetup(8) man page. Once again, if everything went right, a few seconds later the openSSH server will replace the tiny dropbear and you will be able to access your server normally (and with the real SSH host key). Hope you find this article helpful! I would love to hear your feedback.

• There's a repository on git.debian.org, which piggy-backs on the collab-maint group, so most Debian people have commit access to it.
• git-annex is used to upload files to that repository, as an incoming queue.
• A git-annex-shell annex-content hook is run whenever someone uploads a file to there. It moves all annexed content over to annex.debconf.org for publication. This involves some ugly but safe stuff to do with publically readable restricted use ssh private keys. It was the hardest piece to get working, and is only necessary because we don't want to bloat git.debian.org with this stuff and it's not practical to give everyone logins to annex.debconf.org.
• As an additional guard against accidental bloat, the git.debian.org repository will refuse to accept uploads when there is less than 5 gb free disk.
• annex.debconf.org was set up following the git-annex public repository on a web site howto.

$ exec /bin/bash
$ type -t yes
file
$ yes()   while :; do echo $ 1:-y ; done;  
$ type -t yes
function

$ run-parts()   
    if [ "$1" = "--test" ]; then
        shift;
        simulate=true;
    else
        simulate=false;
    fi;
    for f in "$1"/*; do
        [ -x "$f" ] && [ -f "$f" ]   continue;
        case "$(basename "$f")" in 
            *[!a-zA-Z0-9_/-]*)
                :
            ;;
            *)
                if $simulate; then
                    echo $f;
                else
                    $f;
                fi
            ;;
        esac;
    done
 
$ type -t run-parts
function

function @   "$@" > /dev/null ;  

• [1] http://www.kogan.com/au/km/
• [2] https://www.aldimobile.com.au/

--
-- Prove keybindings work.
--
keymap['global'][' '] = 'msg("UTF-8 rocks!")'

--
-- Pretend the user typed ":msg ...\n"
--
stuff( ":msg('  is pie')\n" );

• A nice menu for viewing/opening attachments: view_attachments().
• A nice system for marking messages as read based upon their subjects: mark_read_by_pattern().

• Show folders with unread mail.
• Quickly read it.

* - I do not have the black death, but I'm not well.

• We borrowed the given/when/then structure from BDD, and the idea of using that structure so that tests are specified using lightly structured text in a way that allowsnon-developers to participate.
• This borrowed structure gives a way to express sequences of things that happen, including setup and checks. We call these sequences test scenarios, and thus this approach to testing is called scenario testing.
• We embed the test scenarios, and the code that implements them, as code blocks in Markdown documents. This makes it easy to explain what is going on in the tests, when that's needed, right next to the tests. The Markdown files can easily be formatted as web pages or PDFs (among other formats), making it easy to share and discuss them with people who might not be comfortable accessing a git repository.

SCENARIO Browsing backups with FUSE plugin
ASSUMING user is in group fuse
GIVEN a live data directory
AND a 0 byte file called empty
AND a 1 byte file called one
AND a 4096 byte file called 4k
AND a 10485760 byte file called 10meg
AND a hardlink to 4k called 4k-hardlink
AND a symlink to 4k called 4k-symlink
AND a directory called some-dir
AND a symlink to ../4k called some-dir/other-symlink
WHEN live data is backed up
AND repository is fuse mounted
THEN latest generation can be copied correctly from fuse mount
FINALLY unmount repository

• SCENARIO is the name of the scenario.
• ASSUMING is a condition for the entire scenario: the FUSE plugin can only be tested when the user can use FUSE.
• GIVEN is a setup step, WHEN is the step that does the thing that is being tested, and THEN checks that the right thing happened during WHEN.
• AND is an alias for the keyword on the previous line; it makes the test scenario seem more like normal English text
• FINALLY is for cleaning up afterwards, whether the scenario passed or not

IMPLEMENTS GIVEN a (\d+) byte file called (\S+)
dd if=/dev/zero of="$DATADIR/live/$MATCH_2" bs=1 count="$MATCH_1"