Search Results: "Emanuele Rocca"

5 April 2024

Emanuele Rocca: PGP keys on Yubikey, with a side of Mutt

Here are my notes about copying PGP keys to external hardware devices such as Yubikeys. Let me begin by saying that the gpg tools are pretty bad at this.
MAKE A COUPLE OF BACKUPS OF ~/.gnupg/ TO DIFFERENT ENCRYPTED USB STICKS BEFORE YOU START. GPG WILL MESS UP YOUR KEYS. SERIOUSLY.
For example, would you believe me if I said that saving changes results in the removal of your private key? Well check this out.
Now that you have multiple safe, offline backups of your keys, here are my notes.
apt install yubikey-manager scdaemon
Plug the Yubikey in, see if it s recognized properly:
ykman list
gpg --card-status
Change the default PIN (123456) and Admin PIN (12345678):
gpg --card-edit
gpg/card> admin
gpg/card> passwd
Look at the openpgp information and change the maximum number of retries, if you like. I have seen this failing a couple of times, unplugging the Yubikey and putting it back in worked.
ykman openpgp info
ykman openpgp access set-retries 7 7 7
Copy your keys. MAKE A BACKUP OF ~/.gnupg/ BEFORE YOU DO THIS.
gpg --edit-key $KEY_ID
gpg> keytocard # follow the prompts to copy the first key
Now choose the next key and copy that one too. Repeat till all subkeys are copied.
gpg> key 1
gpg> keytocard
Typing gpg --card-status you should be able to see all your keys on the Yubikey now.

Using the key on another machine
How do you use your PGP keys on the Yubikey on other systems?
Go to another system, if it does have a ~/.gnupg directory already move it somewhere else.
apt install scdaemon
Import your public key:
gpg -k
gpg --keyserver pgp.mit.edu --recv-keys $KEY_ID
Check the fingerprint and if it is indeed your key say you trust it:
gpg --edit-key $KEY_ID
> trust
> 5
> y
> save
Now try gpg --card-status and gpg --list-secret-keys, you should be able to see your keys. Try signing something, it should work.
gpg --output /tmp/x.out --sign /etc/motd
gpg --verify /tmp/x.out

Using the Yubikey with Mutt
If you re using mutt with IMAP, there is a very simple trick to safely store your password on disk. Create an encrypted file with your IMAP password:
echo SUPERSECRET   gpg --encrypt > ~/.mutt_password.gpg
Add the following to ~/.muttrc:
set imap_pass= gpg --decrypt ~/.mutt_password.gpg 
With the above, mutt now prompts you to insert the Yubikey and type your PIN in order to connect to the IMAP server.

12 February 2024

Emanuele Rocca: Enabling Kernel Settings in Debian

This time it s about enabling new kernel config options in the official Debian kernel packages. A few dependencies are needed to run the various scripts used by the Debian kernel folks, as well as to build the kernel itself:
apt install git gpg python3-debian python3-dacite
apt build-dep linux
With that in place, fetch the linux and kernel-team repos:
git clone --depth 1 https://salsa.debian.org/kernel-team/linux
git clone --depth 1 https://salsa.debian.org/kernel-team/kernel-team
So far you ve only got the Debian-specific bits. Fetch the actual kernel sources now. In the likely case that you re building a stable kernel, run the following from within the linux directory:
debian/bin/genorig.py https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
Use the torvalds repo if you re building an RC version instead:
debian/bin/genorig.py https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Now generate the upstream tarball as well as debian/control. The first command will take a bit, and the second command will fail: but that s success just as the output says.
debian/rules orig
debian/rules debian/control
Now generate patched sources with:
debian/rules source
Time to edit the Kconfig and enable/disable whatever setting you wanted to change. Take a look around the files under debian/config/ to see where your changes should go. If it s a setting shared among multiple architectures that may be debian/config/config. For x86-specific things, the file is debian/config/amd64/config. On aarch64 debian/config/arm64/config. If in doubt, you could try asking #debian-kernel on IRC.
It may look like you need to figure out where exactly in the file the setting should be placed. That is not the case. There s a helpful script fixing things up for you:
../kernel-team/utils/kconfigeditor2/process.py .
The above will fail if you forgot to run debian/rules source. The debian/build/source_rt/Kconfig file is needed by the script:
Traceback (most recent call last):
  File "/tmp/linux/../kernel-team/utils/kconfigeditor2/process.py", line 19, in __init__
    menu = fs_menu[featureset or 'none']
           ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'rt'
During handling of the above exception, another exception occurred:
[...]
FileNotFoundError: [Errno 2] No such file or directory: './debian/build/source_rt/Kconfig'
If that happens, run:
debian/rules source
Now process.py should work fine and fix your config file.
Excellent, now the config is updated and we re ready to build the kernel. Off we go:
export MAKEFLAGS=-j$(nproc)
export DEB_BUILD_PROFILES='pkg.linux.nokerneldbg pkg.linux.nokerneldbginfo pkg.linux.notools nodoc'
dpkg-buildpackage -b -nc -uc

6 October 2023

Emanuele Rocca: Custom Debian Installer and Kernel on a USB stick

There are many valid reasons to create a custom Debian Installer image. You may need to pass some special arguments to the kernel, use a different GRUB version, automate the installation by means of preseeding, use a custom kernel, or modify the installer itself.
If you have a EFI system, which is probably the case in 2023, there is no need to learn complex procedures in order to create a custom Debian Installer stick.
The source of many frustrations is that the ISO format for CDs/DVDs is read-only, but you can just create a VFAT filesystem on a USB stick, copy all ISO contents onto the stick itself, and modify things at will.

Create a writable USB stick
First create a FAT32 filesystem on the removable device and mount it. The device is sdX in the example.
$ sudo parted --script /dev/sdX mklabel msdos
$ sudo parted --script /dev/sdX mkpart primary fat32 0% 100%
$ sudo mkfs.vfat /dev/sdX1
$ sudo mount /dev/sdX1 /mnt/data/
Then copy to the USB stick the installer ISO you would like to modify, debian-testing-amd64-netinst.iso here.
$ sudo kpartx -v -a debian-testing-amd64-netinst.iso
# Mount the first partition on the ISO and copy its contents to the stick
$ sudo mount /dev/mapper/loop0p1 /mnt/cdrom/
$ sudo rsync -av /mnt/cdrom/ /mnt/data/
$ sudo umount /mnt/cdrom
# Same story with the second partition on the ISO
$ sudo mount /dev/mapper/loop0p2 /mnt/cdrom/
$ sudo rsync -av /mnt/cdrom/ /mnt/data/
$ sudo umount /mnt/cdrom
$ sudo kpartx -d debian-testing-amd64-netinst.iso
$ sudo umount /mnt/data
Now try booting from the USB stick just to verify that everything went well and we can start customizing the image.

Boot loader, preseeding, installer hacks
The easiest things we can change now are the shim, GRUB, and GRUB s configuration. The USB stick contains the shim under /EFI/boot/bootx64.efi, while GRUB is at /EFI/boot/grubx64.efi. This means that if you want to test a different shim / GRUB version, you just replace the relevant files. That s it. Take for example /usr/lib/grub/x86_64-efi/monolithic/grubx64.efi from the package grub-efi-amd64-bin, or the signed version from grub-efi-amd64-signed and copy them under /EFI/boot/grubx64.efi. Or perhaps you want to try out systemd-boot? Then take /usr/lib/systemd/boot/efi/systemd-bootx64.efi from the package systemd-boot-efi, copy it to /EFI/boot/bootx64.efi and you re good to go. Figuring out the right systemd-boot configuration needed to start the Installer is left as an exercise.
By editing /boot/grub/grub.cfg you can pass arbitrary arguments to the kernel and the Installer itself. See the official Installation Guide for a comprehensive list of boot parameters.
One very commong thing to do is automating the installation using a preseed file. Add the following to the kernel command line: preseed/file=/cdrom/preseed.cfg and create a /preseed.cfg file on the USB stick. As a little example:
d-i time/zone select Europe/Rome
d-i passwd/root-password this-is-the-root-password
d-i passwd/root-password-again this-is-the-root-password
d-i passwd/user-fullname string Emanuele Rocca
d-i passwd/username string ema
d-i passwd/user-password password lol-haha-uh
d-i passwd/user-password-again password lol-haha-uh
d-i apt-setup/no_mirror boolean true
d-i popularity-contest/participate boolean true
tasksel tasksel/first multiselect standard
See Steve McIntyre s awesome page with the full list of available settings and their description: https://preseed.einval.com/debian-preseed/.
Two noteworthy settings are early_command and late_command. They can be used to execute arbitrary commands and provide thus extreme flexibility! You can go as far as replacing parts of the installer with a sed command, or maybe wgetting an entirely different file. This is a fairly easy way to test minor Installer patches. As an example, I ve once used this to test a patch to grub-installer:
d-i partman/early_command string wget https://people.debian.org/~ema/grub-installer-1035085-1 -O /usr/bin/grub-installer
Finally, the initrd contains all early stages of the installer. It s easy to unpack it, modify whatever component you like, and repack it. Say you want to change a given udev rule:
$ mkdir /tmp/new-initrd
$ cd /tmp/new-initrd
$ zstdcat /mnt/data/install.a64/initrd.gz   sudo cpio -id
$ vi lib/udev/rules.d/60-block.rules
$ find .   cpio -o -H newc   zstd --stdout > /mnt/data/install.a64/initrd.gz

Custom udebs
From a basic architectural standpoint the Debian Installer can be seen as an initrd that loads a series of special Debian packages called udebs. In the previous section we have seen how to (ab)use early_command to replace one of the scripts used by the Installer, namely grub-installer. It turns out that such script is installed by a udeb, so let s do things right and build a new Installer ISO with our custom grub udeb.
Fetch the code for the grub-installer udeb, make your changes and build it with a classic dpkg-buildpackage -rfakeroot.
Then get the Installer code and install all dependencies:
$ git clone https://salsa.debian.org/installer-team/debian-installer/
$ cd debian-installer/
$ sudo apt build-dep .
Now add the grub-installer udeb to the localudebs directory and create a new netboot image:
$ cp /path/to/grub-installer_1.198_arm64.udeb build/localudebs/
$ cd build
$ fakeroot make clean_netboot build_netboot
Give it some time, soon enough you ll have a brand new ISO to test under dest/netboot/mini.iso.

Custom kernel
Perhaps there s a kernel configuration option you need to enable, or maybe you need a more recent kernel version than what is available in sid.
The Debian Linux Kernel Handbook has all the details for how to do things properly, but here s a quick example.
Get the Debian kernel packaging from salsa and generate the upstream tarball:
$ git clone https://salsa.debian.org/kernel-team/linux/
$ ./debian/bin/genorig.py https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
For RC kernels use the repo from Linus instead of linux-stable.
Now do your thing, for instance change a config setting by editing debian/config/amd64/config. Don t worry about where you put it in the file, there s a tool from https://salsa.debian.org/kernel-team/kernel-team to fix that:
$ /path/to/kernel-team/utils/kconfigeditor2/process.py .
Now build your kernel:
$ export MAKEFLAGS=-j$(nproc)
$ export DEB_BUILD_PROFILES='pkg.linux.nokerneldbg pkg.linux.nokerneldbginfo pkg.linux.notools nodoc'
$ debian/rules orig
$ debian/rules debian/control
$ dpkg-buildpackage -b -nc -uc
After some time, if everything went well, you should get a bunch of .deb files as well as a .changes file, linux_6.6~rc3-1~exp1_arm64.changes here. To generate the udebs used by the Installer you need to first get a linux-signed .dsc file, and then build it with sbuild in this example:
$ /path/to/kernel-team/scripts/debian-test-sign linux_6.6~rc3-1~exp1_arm64.changes
$ sbuild --dist=unstable --extra-package=$PWD linux-signed-arm64_6.6~rc3+1~exp1.dsc
Excellent, now you should have a ton of .udebs. To build a custom installer image with this kernel, copy them all under debian-installer/build/localudebs/ and then run fakeroot make clean_netboot build_netboot as described in the previous section. In case you are trying to use a different kernel version from what is currently in sid, you will have to install the linux-image package on the system building the ISO, and change LINUX_KERNEL_ABI in build/config/common. The linux-image dependency in debian/control probably needs to be tweaked as well.
That s it, the new Installer ISO should boot with your custom kernel!
There is going to be another minor obstacle though, as anna will complain that your new kernel cannot be found in the archive. Copy the kernel udebs you have built onto a vfat formatted USB stick, switch to a terminal, and install them all with udpkg:
~ # udpkg -i *.udeb
Now the installation should proceed smoothly.

4 May 2023

Emanuele Rocca: UEFI Secure Boot on the Raspberry Pi

UPDATE: this post unexpectedly ended up on Hacker News and I received a lot of comments. The two most important points being made are (1) that Secure Boot on the RPi as described here is not actually truly secure. An attacker who successfully gained root could just mount the firmware partition and either add their own keys to the EFI variable store or replace the firmware altogether with a malicious one. (2) The TianCore firmware cannot be used instead of the proprietary blob as I mentioned. What truly happens is that the proprietary blob is loaded onto the VideoCore cores, then TianoCore is loaded onto the ARM cores. Thanks for the corrections.

A port of the free software TianoCore UEFI firmware can be used instead of the proprietary boot blob to boot the Raspberry Pi. This allows to install Debian on the RPi with the standard Debian Installer, and it also makes it possible to use UEFI Secure Boot. Note that Secure Boot had been broken on arm64 for a while, but it s now working in Bookworm!.

Debian Installer UEFI boot
To begin, you ll need to download the appropriate firmware files for the RPi3 or RPi4. I ve got a Raspberry Pi 3 Model B+ myself, so the rest of this document will assume an RPi3 is being installed.
Plug the SD card you are going to use as the RPi storage device into another system. Say it shows up as /dev/sdf. Then:
# Create an msdos partition table
$ sudo parted --script /dev/sdf mklabel msdos
# Create, format, and label a 10M fat32 partition
$ sudo parted --script /dev/sdf mkpart primary fat32 0% 10M
$ sudo mkfs.vfat /dev/sdf1
$ sudo fatlabel /dev/sdf1 RPI-FW
# Get the UEFI firmware onto the SD card
$ sudo mount /dev/sdf1 /mnt/data/
$ sudo unzip Downloads/RPi3_UEFI_Firmware_v1.38.zip -d /mnt/data/
$ sudo umount /mnt/data
At this point, the SD card can be used to boot the RPi, and you ll get a UEFI firmware.
Download the Bookworm RC 2 release of the installer, copy it to a USB stick as described in the Installation Guide, and boot your RPi from the stick. If for some reason booting from the stick does not happen automatically, enter the firmware interface with ESC and choose the USB stick from Boot Manager.
Proceed with the installation as normal, paying attention not to modify the firmware partition labeled RPI-FW. I initially thought it would be nice to reuse the firmware partition as ESP partition as well. However, setting the esp flag on makes the RPi unbootable. Either configuring the partition as ESP in debian-installer, or manually with sudo parted --script /dev/sda set 1 esp on, breaks boot. In case you accidentally do that, set it back to off and the edk2 firmware will boot again.
What I suggest doing in terms of partitioning is: (1) leave the 10M partition created above for the firmware alone, and (2) create another 512M or so ESP partition for EFI boot.
The installation should go smoothly till the end, but rebooting won t work. Doh. This is because of an important gotcha: the Raspberry Pi port of the TianoCore firmware we are using does not support setting UEFI variables persistently from a "High Level Operating System (HLOS)", which is the debian-installer in our case. Persistently is the keyword there: variables can be set and modified regularly with efibootmgr or otherwise, but crucially the modifications do not survive reboot. However, changes made from the firmware interface itself are persistent. So enter the firmware with ESC right after booting the RPi, select Boot Maintenance Manager Boot Options Add Boot Option Your SD card Your ESP partition EFI debian shimaa64.efi. Choose a creative name for your boot entry (eg: "debian"), save and exit the firmware interface. Bookworm should be booting fine at this point!

Enabling Secure Boot
Although the TianoCore firmware does support Secure Boot, there are no keys enrolled by default. To add the required keys, copy PK-0001.der, DB-0001.der, DB-0002.der, KEK-0001.der, and KEK-0002.der to a FAT32 formatted USB stick.
Here s a summary of the Subject field for each of the above:
PK-0001.der.pem
        Subject: O = Debian, CN = Debian UEFI Secure Boot (PK/KEK key), emailAddress = debian-devel@lists.debian.org
DB-0001.der.pem
        Subject: C = US, ST = Washington, L = Redmond, O = Microsoft Corporation, CN = Microsoft Windows Production PCA 2011
DB-0002.der.pem
        Subject: C = US, ST = Washington, L = Redmond, O = Microsoft Corporation, CN = Microsoft Corporation UEFI CA 2011
KEK-0001.der.pem
        Subject: O = Debian, CN = Debian UEFI Secure Boot (PK/KEK key), emailAddress = debian-devel@lists.debian.org
KEK-0002.der.pem
        Subject: C = US, ST = Washington, L = Redmond, O = Microsoft Corporation, CN = Microsoft Corporation KEK CA 2011
Plug the stick into the RPi, boot and enter the firmware interface with ESC. Select Device Manager Secure Boot Configuration Secure Boot Mode choose Custom Mode Custom Secure Boot Options PK Options Enroll PK choose PK-0001.der. Do the same for DB Options, this time choose DB-0001.der and DB-0002.der. As you may have guessed by now, the same must be done for KEK Options, but adding KEK-0001.der and KEK-0002.der. Save, exit, reboot. If everything went well, your RPi now has booted with Secure Boot enabled.
See https://wiki.debian.org/SecureBoot for the details on how to check whether Secure Boot has been enabled correctly and much more.

26 March 2023

Emanuele Rocca: EFI and Secure Boot Notes

To create a bootable EFI drive to use with QEMU, first make a disk image and create a vfat filesystem on it.
$ dd if=/dev/zero of=boot.img bs=1M count=512
$ sudo mkfs.vfat boot.img
By default, EFI firmwares boot a specific file under /efi/boot/. The name of such file depends on the architecture: for example, on 64 bit x86 systems it is bootx64.efi, while on ARM it is bootaa64.efi.
Copy /usr/lib/grub/x86_64-efi/monolithic/grubx64.efi from package grub-efi-amd64-bin to /efi/boot/bootx64.efi on the boot image, and that should be enough to start GRUB.
# mount boot.img /mnt/
# mkdir -p /mnt/efi/boot/
# cp /usr/lib/grub/x86_64-efi/monolithic/grubx64.efi /mnt/efi/boot/bootx64.efi
# umount /mnt/
Now get the x86 firmware from package ovmf and start qemu:
$ cp /usr/share/OVMF/OVMF_CODE.fd /tmp/code.fd
$ qemu-system-x86_64 -drive file=/tmp/code.fd,format=raw,if=pflash -cdrom boot.img
GRUB looks fine, but it would be good to have a kernel to boot. Let s add one to boot.img.
# mount boot.img /mnt
# cp vmlinuz-6.1.0-7-amd64 /mnt/vmlinuz
# umount /mnt/
Boot with qemu again, but this time pass -m 1G. The default amount of memory is not enough to boot.
$ qemu-system-x86_64 -drive file=/tmp/code.fd,format=raw,if=pflash -cdrom boot.img -m 1G
At the grub prompt, type the following to boot:
grub> linux /vmlinuz
grub> boot
The kernel will start and reach the point of trying to mount the root fs. This is great but it would now be useful to have some sort of shell access in order to look around. Let s add an initrd!
# mount boot.img /mnt
# cp initrd.img-6.1.0-7-amd64 /mnt/initrd
# umount /mnt/
There s the option of starting qemu in console, let s try that out. Start qemu with -nographic, and append console=ttyS0 to the kernel command line arguments.
$ qemu-system-x86_64 -drive file=/tmp/code.fd,format=raw,if=pflash -cdrom boot.img -m 1G -nographic
grub> linux /vmlinuz console=ttyS0
grub> initrd /initrd
grub> boot
If all went well we are now in the initramfs shell. We can now run commands! At this point we can see that the system has Secure boot disabled:
(initramfs) dmesg   grep secureboot
[    0.000000] secureboot: Secure boot disabled
In order to boot with Secure boot, we need:
  • a signed shim, grub, and kernel
  • the right EFI variables for Secure boot
The package shim-signed provides a shim signed with Microsoft s key, while grub-efi-amd64-signed has GRUB signed with Debian s key.
The signatures can be shown with sbverify --list:
$ sbverify --list /usr/lib/shim/shimx64.efi.signed
warning: data remaining[823184 vs 948768]: gaps between PE/COFF sections?
signature 1
image signature issuers:
 - /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011
image signature certificates:
 - subject: /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Windows UEFI Driver Publisher
   issuer:  /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011
 - subject: /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation UEFI CA 2011
   issuer:  /C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/CN=Microsoft Corporation Third Party Marketplace Root
Similarly for GRUB and the kernel:
$ sbverify --list /usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed
signature 1
image signature issuers:
 - /CN=Debian Secure Boot CA
image signature certificates:
 - subject: /CN=Debian Secure Boot Signer 2022 - grub2
   issuer:  /CN=Debian Secure Boot CA
$ sbverify --list /mnt/vmlinuz
signature 1
image signature issuers:
 - /CN=Debian Secure Boot CA
image signature certificates:
 - subject: /CN=Debian Secure Boot Signer 2022 - linux
   issuer:  /CN=Debian Secure Boot CA
Let s use the signed shim and grub in the boot image:
# mount boot.img /mnt
# cp /usr/lib/shim/shimx64.efi.signed /mnt/efi/boot/bootx64.efi
# cp /usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed /mnt/efi/boot/grubx64.efi
# umount /mnt
And start QEMU with the appropriate EFI variables for Secure boot:
$ cp /usr/share/OVMF/OVMF_VARS.ms.fd /tmp/vars.fd
$ qemu-system-x86_64 -drive file=/tmp/code.fd,format=raw,if=pflash -drive file=/tmp/vars.fd,format=raw,if=pflash -cdrom boot.img -m 1G -nographic
We can double-check in the firmware settings if Secure boot is indeed enabled. At the GRUB prompt, type fwsetup:
grub> fwsetup
Check under "Device Manager" "Secure Boot Configuration" that "Attempt Secure Boot" is selected, then boot from GRUB as before. If all went well, the kernel should confirm that we have booted with Secure boot:
(initramfs) dmesg   grep secureboot
[    0.000000] secureboot: Secure boot enabled

15 March 2023

Emanuele Rocca: Disposable Debian VMs with debvm

Some notes on using debvm, an amazing piece of software I ve started using only recently.
Create a new virtual machine:
$ debvm-create
You now have a virtual machine with Debian Sid of your host native architecture (probably amd64). The image file is called rootfs.ext4. You ve got 1G of disk space in the VM.
You can now just run the VM! You will be automatically logged is as root.
$ debvm-run
Experiment in the VM, run all the sort of tests you have in mind. For example, one thing I commonly do is verifying that things work in a clean environment, as opposed to "on my machine".
If anything goes wrong, or if you just want to repeat the test: shutdown the guest, remove rootfs.ext4, and start again with debvm-create.
Now, especially if you intend creating and recreating VMs multiple times, it helps to use a local apt mirror as a cache to avoid fetching all the required packages from the internet over and over again. Install apt-cacher-ng on your host machine and point debvm-create at it:
$ debvm-create -- http://10.0.3.1:3142/debian
The additional options after -- are passed to mmdebstrap. In this case we re specifying http://10.0.3.1:3142/debian as the URL of our local apt mirror. It is going to be used both for image creation and as the only entry in /etc/apt/sources.list on the guest. This is the reason for using 10.0.3.1, the IP address of the lxcbr0 interface used by qemu, instead of 127.0.0.1: to make sure that the guest VM has access to it too.
For a slightly more advanced example, we now want to:
  • run a arm64 VM
  • have more disk space availably, say 2G
  • install additional packages (curl and locales)
  • allow SSH as root with the given public keys, in this example the authorized_keys installed on the host
  • start the VM in the background with no console output
$ debvm-create -a arm64 -o sid-arm64.ext4 -z 2G -k ~/.ssh/authorized_keys -- http://10.0.3.1:3142/debian --include curl,locales
Start the VM:
$ debvm-run -i sid-arm64.ext4 -s 2222 -g -- -display none &
SSH into the guest:
$ ssh -o NoHostAuthenticationForLocalhost=yes -p 2222 root@127.0.0.1
Enjoy!

7 October 2015

Emanuele Rocca: systemd is your friend

Today I want to talk a bit about some cool features of systemd, the default Debian init system since the release of Jessie. Ubuntu has also adopted systemd in 15.04, meaning that you are going to find it literally everywhere.
Logging The component responsible for logging in systemd is called journal. It collects and stores logs in a structured, indexed journal (hence the name). The journal can replace traditional syslog daemons such as rsyslog and syslog-ng, or work together with them. By default Debian keeps on using rsyslog, but if you don't need to ship logs to a centralized server (or do other fancy things) it is possible to stop using rsyslog right now and rely on systemd-journal instead. The obvious question is: why would anybody use a binary format for logs instead of a bunch of tried and true plain-text files? As it turns out, there are quite a lot of good reasons to do so. The killer features of systemd-journald for me are:
  • Index tons of logs while being able to search with good performance: O(log(n)) instead of O(n) which is what you get with text files
  • No need to worry about log rotation anymore, in any shape or form
The last point in particular is really critical in my opinion. Traditional log rotation implementations rely on cron jobs to check how much disk space is used by logs, compressing/removing old files. Log rotation is usually: 1) annoying to configure; 2) hard to get right; 3) prone to DoS attacks. With journald, there is pretty much nothing to configure. Log rotation is built into the daemon disk space allocation logic itself. This also allows to avoid vulnerability windows due to time-based rotation, which is what you get with logrotate and friends. Enough high-level discussions though, here is how to use the journal! Check if you already have the directory /var/log/journal, otherwise create it (as root). Then restart systemd-journald as follows: sudo systemctl restart systemd-journald You can get all messages produced since the last boot with journalctl -b. All messages produced today can get extracted using journalctl --since=today. Want to get all logs related to ssh? Try with journalctl _SYSTEMD_UNIT=ssh.service. There are many more filtering options available, you can read all about them with man journalctl. journald's configuration file is /etc/systemd/journald.conf. Two of the most interesting options are SystemMaxUse and SystemKeepFree, which can be used to change the amount of disk space dedicated to logging. They default to 10% and 15% of the /var filesystem respectively. Here is a little cheatsheet:
journalctl -b                # Show all messages since last boot
journalctl -f                # Tail your logs
journalctl --since=yesterday # Show all messages produced since yesterday
journalctl -pcrit            # Filter messages by priority
journalctl /bin/su           # Filter messages by program
journalctl --disk-usage      # The amount of space in use for journaling
Further reading:
Containers A relatively little known component of systemd is systemd-nspawn. It is a small, straightforward container manager. If you don't already have a chroot somewhere, here is how to create a basic Debian Jessie chroot under /srv/chroots/jessie:
$ debootstrap jessie /srv/chroots/jessie http://http.debian.net/debian/
With systemd-nspawn you can easily run a shell inside the chroot:
$ sudo systemd-nspawn -D /srv/chroots/jessie
Spawning container jessie on /srv/chroots/jessie.
Press ^] three times within 1s to kill container.
/etc/localtime is not a symlink, not updating container timezone.
root@jessie:~#
Done. Everything works out of the box: no need for you to mount /dev, /run and friends, systemd-nspawn took care of that. Networking also works. If you want to actually boot the system, just add the -b switch to the previous command:
$ sudo systemd-nspawn -b -D /srv/chroots/jessie
Spawning container jessie on /srv/chroots/jessie.
Press ^] three times within 1s to kill container.
/etc/localtime is not a symlink, not updating container timezone.
systemd 215 running in system mode. (+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ -SECCOMP -APPARMOR)
Detected virtualization 'systemd-nspawn'.
Detected architecture 'x86-64'.
Welcome to Debian GNU/Linux jessie/sid!
Set hostname to <orion>.
[  OK  ] Reached target Remote File Systems (Pre).
[  OK  ] Reached target Encrypted Volumes.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Swap.
[  OK  ] Created slice Root Slice.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
[  OK  ] Listening on Delayed Shutdown Socket.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
[  OK  ] Created slice System Slice.
[  OK  ] Created slice system-getty.slice.
[  OK  ] Listening on Syslog Socket.
         Mounting POSIX Message Queue File System...
         Mounting Huge Pages File System...
         Mounting FUSE Control File System...
         Starting Copy rules generated while the root was ro...
         Starting Journal Service...
[  OK  ] Started Journal Service.
[  OK  ] Reached target Slices.
         Starting Remount Root and Kernel File Systems...
[  OK  ] Mounted Huge Pages File System.
[  OK  ] Mounted POSIX Message Queue File System.
[  OK  ] Mounted FUSE Control File System.
[  OK  ] Started Copy rules generated while the root was ro.
[  OK  ] Started Remount Root and Kernel File Systems.
         Starting Load/Save Random Seed...
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
[  OK  ] Reached target Remote File Systems.
         Starting Trigger Flushing of Journal to Persistent Storage...
[  OK  ] Started Load/Save Random Seed.
         Starting LSB: Raise network interfaces....
[  OK  ] Started Create Volatile Files and Directories.
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Trigger Flushing of Journal to Persistent Storage.
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Started LSB: Raise network interfaces..
[  OK  ] Reached target Network.
[  OK  ] Reached target Network is Online.
[  OK  ] Reached target System Initialization.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Timers.
[  OK  ] Reached target Basic System.
         Starting /etc/rc.local Compatibility...
         Starting Login Service...
         Starting LSB: Regular background program processing daemon...
         Starting D-Bus System Message Bus...
[  OK  ] Started D-Bus System Message Bus.
         Starting System Logging Service...
[  OK  ] Started System Logging Service.
         Starting Permit User Sessions...
[  OK  ] Started /etc/rc.local Compatibility.
[  OK  ] Started LSB: Regular background program processing daemon.
         Starting Cleanup of Temporary Directories...
[  OK  ] Started Permit User Sessions.
         Starting Console Getty...
[  OK  ] Started Console Getty.
[  OK  ] Reached target Login Prompts.
[  OK  ] Started Login Service.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Cleanup of Temporary Directories.
[  OK  ] Started Update UTMP about System Runlevel Changes.
Debian GNU/Linux jessie/sid orion console
orion login:
That's it! Just one command to start a shell in your chroot or boot the container, again zero configuration needed. Finally, systemd provides a command called machinectl that allows you to introspect and control your container:
$ sudo machinectl status jessie
jessie
           Since: Wed 2015-10-07 11:22:56 CEST; 55min ago
          Leader: 32468 (systemd)
         Service: nspawn; class container
            Root: /srv/chroots/jessie
         Address: fe80::8e70:5aff:fe81:2290
                  192.168.122.1
                  192.168.1.13
              OS: Debian GNU/Linux jessie/sid
            Unit: machine-jessie.scope
                   32468 /lib/systemd/systemd
                   system.slice
                     dbus.service
                       32534 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile -...
                     cron.service
                       32539 /usr/sbin/cron
                     systemd-journald.service
                       32487 /lib/systemd/systemd-journald
                     systemd-logind.service
                       32532 /lib/systemd/systemd-logind
                     console-getty.service
                       32544 /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt102
                     rsyslog.service
                       32540 /usr/sbin/rsyslogd -n
With machinectl you can also reboot, poweroff, terminate your containers and more. There are so many things to learn about systemd and containers! Here are some references. This stuff is pretty exciting. Now that all major distributions use systemd by default, we can expect to have access to tools like journalctl and systemd-nspawn everywhere!

23 May 2014

Emanuele Rocca: A (very) brief history of Australia

This post is mostly a sum-up of the Wikipedia page History of Australia, with some content taken from History of the British Empire. Both texts are available under the Creative Commons Attribution-ShareAlike License. I do not seem to be able to learn about a new topic without taking notes: in this case I have decided to publish my work, hoping that someone will find it useful. Some very important themes such as the Gold Rush and Australian History during the World Wars have been impudently ignored.
Indigenous Australians The ancestors of Indigenous Australians are believed to have arrived in Australia 40,000 to 60,000 years ago, and possibly as early as 70,000 years ago. By 1788, the population of Australia existed as 250 individual nations, many of which were in alliance with one another, and within each nation there existed several clans, from as few as five or six to as many as 30 or 40. Each nation had its own language and a few had multiple, thus over 250 languages existed, around 200 of which are now extinct. Permanent European settlers arrived at Sydney in 1788 and came to control most of the continent by end of the 19th century. Bastions of largely unaltered Aboriginal societies survived, particularly in Northern and Western Australia into the 20th century, until finally, a group of Pintupi people of the Gibson Desert became the last people to be contacted by outsider ways in 1984.
European explorers Terra Australis (Latin for South Land) is one of the names given to a hypothetical continent which appeared on European maps between the 15th and 18th centuries. Although the landmass was drawn onto maps, Terra Australis was not based on any actual surveying of such a landmass but rather based on the hypothesis that continents in the Northern Hemisphere should be balanced by land in the south. The first documented European landing in Australia was made in 1606 by a Dutch ship led by Willem Janszoon. Hence the ancient name "Nova Hollandia". The same year, a Spanish expedition had landed in the New Hebrides and, believing them to be the fabled southern continent, named the land: "Terra Austral del Espiritu Santo". Hence the current name "Australia". Although various proposals for colonisation were made, notably by Pierre Purry from 1717 to 1744, none was officially attempted. Indigenous Australians were less able to trade with Europeans than were the peoples of India, the East Indies, China, and Japan. The Dutch East India Company concluded that there was "no good to be done there". In 1769, Lieutenant James Cook tried to locate the supposed Southern Continent. This continent was not found, and Cook decided to survey the east coast of New Holland, the only major part of that continent that had not been charted by Dutch navigators. Cook charted and took possession of the east coast of New Holland. He noted the following in his journal:
"I can land no more upon this Eastern coast of New Holland, and
 on the Western side I can make no new discovery the honour of
 which belongs to the Dutch Navigators and as such they may lay
 Claim to it as their property, but the Eastern Coast from the
 Latitude of 38 South down to this place I am confident was never
 seen or viseted by any European before us and therefore by the
 same Rule belongs to great Brittan."
Colonisation The American Revolutionary War (1775-1783) saw Great Britain lose most of its North American colonies and consider establishing replacement territories. The British colony of New South Wales was established with the arrival of the First Fleet of 11 vessels in January 1788. It consisted of over a thousand settlers, including 778 convicts (192 women and 586 men). A few days after arrival at Botany Bay the fleet moved to the more suitable Port Jackson where a settlement was established at Sydney Cove on 26 January 1788. This date later became Australia's national day, Australia Day. Between 1788 and 1868, approximately 161,700 convicts (of whom 25,000 were women) were transported to the Australian colonies of New South Wales, Van Diemen's land and Western Australia. Early colonial administrations were anxious to address the gender imbalance in the population brought about by the importation of large numbers of convict men. In 1835, the British Colonial Office issued the Proclamation of Governor Bourke, implementing the legal doctrine of terra nullius upon which British settlement was based, reinforcing the notion that the land belonged to no one prior to the British Crown taking possession of it and quashing any likelihood of treaties with Aboriginal peoples, including that signed by John Batman. Its publication meant that from then, all people found occupying land without the authority of the government would be considered illegal trespassers. A group in Britain led by Edward Gibbon Wakefield sought to start a colony based on free settlement and political and religious freedoms, rather than convict labour. The South Australia Act [1834], passed by the British Government which established the colony reflected these desires and included a promise of representative government when the population reached 50,000 people. Significantly, the Letters Patent enabling the South Australia Act 1834 included a guarantee of the rights of 'any Aboriginal Natives' and their descendants to lands they 'now actually occupied or enjoyed'. In 1836, two ships of the South Australia Land Company left to establish the first settlement on Kangaroo Island. The foundation of South Australia is now generally commemorated as Governor John Hindmarsh's Proclamation of the new Province at Glenelg, on the mainland, on 28 December 1836. By 1851 the colony was experimenting with a partially elected council.
Development of Australian democracy Traditional Aboriginal society had been governed by councils of elders and a corporate decision making process, but the first European-style governments established after 1788 were autocratic and run by appointed governors. The reformist attorney general, John Plunkett, sought to apply Enlightenment principles to governance in the colony, pursuing the establishment of equality before the law. Plunkett twice charged the colonist perpetrators of the Myall Creek massacre of Aborigines with murder, resulting in a conviction and his landmark Church Act of 1836 disestablished the Church of England and established legal equality between Anglicans, Catholics, Presbyterians and later Methodists. In 1840, the Adelaide City Council and the Sydney City Council were established. Men who possessed 1,000 pounds worth of property were able to stand for election and wealthy landowners were permitted up to four votes each in elections. Australia's first parliamentary elections were conducted for the New South Wales Legislative Council in 1843, again with voting rights (for males only) tied to property ownership or financial capacity. Voter rights were extended further in New South Wales in 1850 and elections for legislative councils were held in the colonies of Victoria, South Australia and Tasmania. Women became eligible to vote for the Parliament of South Australia in 1895. This was the first legislation in the world permitting women also to stand for election to political office and, in 1897, Catherine Helen Spence became the first female political candidate for political office, unsuccessfully standing for election as a delegate to the Federal Convention on Australian Federation. Western Australia granted voting rights to women in 1899. Early federal parliamentary reform and judicial interpretation sought to limit Aboriginal voting in practice, a situation which endured until rights activists began campaigning in the 1940s.
Road to independence Despite suspicion from some sections of the colonial community (especially in smaller colonies) about the value of nationhood, improvements in inter-colonial transport and communication, including the linking of Perth to the south eastern cities by telegraph in 1877, helped break down inter-colonial rivalries. New South Wales Premier Henry Parkes addressed a rural audience in his 1889 Tenterfield Oration, stating that the time had come to form a national executive government:
"Australia [now has] a population of three and a half millions,
 and the American people numbered only between three and four
 millions when they formed the great commonwealth of the United
 States. The numbers were about the same, and surely what the
 Americans had done by war, the Australians could bring about in
 peace, without breaking the ties that held them to the mother
 country."
Though Parkes would not live to see it, his vision would be achieved within a little over a decade, and he is remembered as the "father of federation". The Commonwealth of Australia came into being when the Federal Constitution was proclaimed by the Governor-General, Lord Hopetoun, on 1 January 1901. Australia took part in WWI. The contributions of Australian and New Zealand troops during the 1915 Gallipoli Campaign against the Ottoman Empire had a great impact on the national consciousness at home, and marked a watershed in the transition of Australia and New Zealand from colonies to nations in their own right. The countries continue to commemorate this occasion on ANZAC Day. Australia achieved independent Sovereign Nation status after World War I, under the Statute of Westminster, which defined Dominions of the British empire in the following way:
"They are autonomous Communities within the British Empire, equal
 in status, in no way subordinate one to another in any aspect of
 their domestic or external affairs, though united by a common
 allegiance to the Crown, and freely associated as members of the
 British Commonwealth of Nations."
The parliaments of Canada, Australia, New Zealand, the Union of South Africa, the Irish Free State and Newfoundland (currently part of Canada) were now independent of British legislative control, they could nullify British laws and Britain could no longer pass laws for them without their consent. The Australia Act 1986 removed any remaining links between the British Parliament and the Australian states.

29 January 2014

Emanuele Rocca: Antifeatures and Antibugs

Software Engineering distinguishes between software features and software bugs. It is usually understood that features are positive, expected characteristics of a computer program. Features make users happy by allowing them to do something useful, interesting, or fun. Something good, anyways. Bugs are instead undesirable and annoying. You're sitting there at your computer writing a long email and the software crashes right before your email is sent. Bad stuff. Features are generally implemented by programmers on purpose, whereas bugs are purely unintentional. They are mistakes. You don't make a mistake on purpose. We might at this point be inclined to think that: i) what is good for users is done on purpose by software manufacturers; ii) what is bad for users was not meant to be. It happened by mistake. Here is a handy table to visualize this idea:
On purpose By mistake
Good Feature
Bad Bug
It seems to make a lot of sense. But you might have noticed that two cells of the table are empty. Right! In a great talk titled When Free Software isn't better, Benjamin Mako Hill mentions the concept of antifeatures, and how they relate to Free Software. Antifeatures are features that make the software do something users will hate. Something they will hate so much they would pay to have those features removed, if that's an option. Microsoft Windows 7 is used in the talk to provide some examples of software antifeatures: the Starter Edition does not allow users to change their background image. Also, it limits the amount of usable memory on the computer to 2GBs, regardless of how much memory the system actually has. Two antifeatures engineered to afflict users to the point that they will purchase a more expensive version of the software, if they have the means to do that. I have another nice example. The Spotify music streaming service plays advertisements between songs every now and then. To make sure users are annoyed as much as possible, Spotify automatically pauses an advertisement if it detects that the volume is being lowered. A poor Spotify user even tried to report the bug on The Spotify Community forum, only to find out that what she naively considered as a software error was "intentional behavior". A spectacular antifeature indeed. Whenever a piece of technology does something you most definitely do not want it to do, such as allowing the NSA to take complete control of your Apple iPhone, including turning on its microphone and camera against your will, that's an antifeature.
On purpose By mistake
Good Feature
Bad Antifeature Bug
Both bugs and antifeatures are bad for users. The difference between them is that antifeatures are engineered. Time and money are spent to make sure the goal is reached. A testing methodology is followed. "Are we really sure customers cannot change their wallpaper even if they try very very hard?" Engineering processes, of course, can fail. If the poor devils at Microsoft who implemented those harassments would have made a mistake that allows users to somehow change their wallpaper on Windows Starter... Well, I would call that a glorious antibug.
On purpose By mistake
Good Feature Antibug
Bad Antifeature Bug
There is no place for antifeatures in Free and Open Source Software. Free Software gives users control over what their software does. Imagine Mozilla adding a feature to Firefox that sets your speakers volume to 11 and starts playing a random song from the black metal artist Burzum every time you add a bookmark, unless you pay for Mozilla Firefox Premium Edition. The source code for Firefox is available under a free license. People who are not into Burzum's music would immediately remove this neat antifeature. I have spent many years of my life advocating Free and Open Source Software, perhaps using the wrong arguments. Mako's talk made me think about all this (thanks mate!). All these years I've been preaching about the technical superiority of Free Software, despite evidence of thousands of bugs and usability issues in the very programs I am using, and contributing to develop. Free Software is not better than Proprietary Software per se. Sometimes it is, sometimes it's not. But it gives you control, and freedom. When it annoys you, when it doesn't do what you expect and want, you can be sure it's not on purpose. And we can fix it together.

4 July 2013

Emanuele Rocca: Useful tools for Python developers

Python is a great language with an impressive number of tools designed to make developers' life easier. Sometimes, however, the problem is getting to know that these tools exist in the first place. By contributing to projects like OpenStack's Nova client and Falcon, I have recently come across some useful tools that can seriously improve the quality of your code. The first one is called pyflakes, a passive checker of Python programs developed by Phil Frost. What it does is parsing your source files and checking for possible errors such as undefined names and unused imports. Let's consider the following example: <figure class="code">
1
2
3
4
import urllib

print "pyflakes example"
urlib.urlopen('http://www.linux.it')
</figure>The code above contains a typo, we have misspelled urllib. Here is what pyflakes thinks about our program:
$ pyflakes example.py
example.py:1: 'urllib' imported but unused
example.py:4: undefined name 'urlib'
On line 4 we try to use urlib which is not defined. Also, we import urllib on line 1 and we do nothing with it. Our typo has been spotted! Notice that, even though our program contains a print statement, 'pyflakes example' has not been printed. That is because pyflakes parses the source files it checks, without importing them, making it safe to use on modules with side effects. pyflakes can be installed with pip or apt-get. The second tool I want to talk about is Ned Batchelder's coverage.py. No doubt you write unit tests for your programs. Right? Good. coverage.py is out there to help you checking how much of your program is actually covered. Let's use as an example codicefiscale, a Python project of mine. First we install coverage:
pip install coverage
Then we run our unit tests:
$ coverage run --source=codicefiscale tests.py
.......
----------------------------------------------------------------------
Ran 7 tests in 0.003s
We pass the module we want to test with --source=codicefiscale so that coverage will only report information about that specific module. Now that our tests have been performed successfully it is time to check how much of our code is covered by unit tests:
$ coverage report -m
Name            Stmts   Miss  Cover   Missing
---------------------------------------------
codicefiscale      73      4    95%   61, 67, 95, 100
Not bad, 95% of our module is covered! Still, coverage let us know that 4 lines have not been touched by the unit tests. With this information, we can go write some meaningful test cases that will also cover the missing lines.

2 September 2012

Emanuele Rocca: Story of a bug in Ubuntu

Some months ago I have run into a pretty interesting bug while working on a Ubuntu-based remote desktop system. The whole OS was installed on a server somewhere and users could access their desktop remotely. Some call this stuff Desktop-as-a-Service. The operating system we chose was Ubuntu Oneiric (11.10) and the remote access part was implemented with x2go, which uses nxagent to provide NX transport of X sessions. Users could access their Ubuntu machines remotely, with sound, video, youtube, and all you would expect from a desktop machine. The whole thing was working quite well. Now, as I said that was in May. Ubuntu 12.04 LTS was available, and the choice of upgrading to it sounded pretty much obvious. So we upgraded our test system to Precise and everything seemed to work smoothly. Till we tried to open a PDF document, actually. It took evince about 50 seconds to display a relatively small document. Same story with images opened with the default image viewer, eog. The fix delivered to our users was simple: we have set shotwell-viewer as the default image viewer, and epdfview as the default PDF viewer. Everybody happy. In the meantime, obviously, I was interested in this issue so I ve tried to run run evince from a terminal, getting the following output:
(evince:15833): GRIP-WARNING **: Failed to initialize gesture manager.
Funny. On another test system running Debian Sid (unstable) everything was working smoothly. The diff between Ubuntu s version of evince and Debian s is a 6MB monster. Among other changes, I noticed that Ubuntu s version build-depends on libgrip-dev, which depends on libutouch-geis. Multitouch stuff. Why should multitouch support break my remote session? So on May the 10th I filed a bug on launchpad. How this issue got handled is in my opinion one of the many fine examples of the inherent superiority of free software, coupled with a we won t hide problems mindset. For an example of how bad is the proprietary approach, just check a random bug in Adobe s bug tracking system. But let s go back to the evince bug. Other users reported that their VNC sessions were also affected by the same problem. After a few days it was clear that the culprit was utouch-geis, and a patch appeared. Unfortunately it did not actually address the issue. Somebody else reported that RDP sessions were broken too. At the beginning of June Precise was still affected. Finally, on August the 6th a working patch was submitted by Bradley M. Froehle and included by Chase Douglas (thank you guys). End of August, fixed version of geis accepted into precise-updates, case closed. Now for some considerations. The problem was clearly of a certain importance. A Long Term Support, stable version of Ubuntu shipped with broken PDF and image viewing functionalities. It got fixed properly, even though 3 good months are quite a long time for such a bug to get solved. However, the issue only affected a pretty limited number of users, also certainly not Ubuntu s main target audience. This bug never affected Debian, simply because utouch-geis has not made its way into the archive yet. It takes longer to make changes like this in Debian, but for some categories of users stability is more important than new, cool features. Choice is a good thing.

14 December 2011

Emanuele Rocca: First steps with Sinatra on Heroku

Lately I have been playing a little bit with different Platform as a Service providers. So far, the most developer-friendly one seems to be Heroku. In particular, Heroku provides an incredibly straightforward way to get started. After creating an account on their website, you just have to install the appropriate gem: sudo gem install heroku The command heroku help will then show you all the possible invocations of the command line client, just to give you an idea of what you can do with it (everything). Alright, that s not particularly interesting without an application to play with, is it? So let s create a new Sinatra application. In order to do that, we need a new directory containing a file like the following one:
# webapp.rb
require 'rubygems'
require 'sinatra'
# Add other gems you might need here
get '/' do
  "Your Sinatra app is working!"
end
Also add a Gemfile with all the required dependencies, in our trivial example sinatra is actually the only gem you need:
source :rubygems
gem "sinatra"
Lastly, you need a config.ru file where you can require our newly created Ruby program and specify that it is a Sinatra application. In this example, the Ruby file is called webapp.rb.
require './webapp'
run Sinatra::Application
A very useful program when developing a Sinatra application is shotgun. gem install shotgun and you will be able to run your new shiny app by executing the command shotgun. While developing on your machine, shotgun will take care of reloading the application for every request, so that you don t have to restart the server after every modification to the source code. Run bundle install to easily install all the required dependencies in your development environment, and shotgun to start the app locally. Pointing your browser to http://127.0.0.1:9393/ you can check if everything works as expected. If nothing went wrong, you can create a git repository to track changes to the source code:
git init
git add . 
git commit -m "Initial commit"
Perfect! Now, how do you create and deploy your application on Heroku?
heroku create
git push heroku master
Heroku will automatically find out that you deployed a Sinatra application, it will install all the required dependencies and start serving requests. A few interesting commands you might want to run to see what s going on are heroku info, heroku ps and heroku logs. The client also provides a command, heroku open, that opens your browser at the right production URL. So, you now have a working application under revision control. Dependencies are handled by specifying them into the Gemfile, and the required gems can be installed locally using bundle. Shotgun is our friend when it comes to use the application for development purposes. A git push heroku master is all we need to deploy a new version of the code. Of course there is way much more about Sinatra and Heroku than what I ve covered in this mini-intro. Please refer to the Sinatra Book and the Heroku Dev Center for more information!

9 October 2011

Emanuele Rocca: A neat VLC extension to download subtitles

As somebody who often watches movies and TV shows I have the recurring problem of finding the right English subtitles for the video I am about to enjoy. Now, that is really a tedious thing to do. You go on a website such as opensubtitles.org and type the title of what you are looking for. Then you usually have to choose from a plethora of results, download the archive, unzip it, fire up VLC, go to Video Subtitles Track Open File, and finally open your subtitles file. And what you typically find out at that point is that the subtitles are completely out of sync with the audio, but rather than doing the whole process again you prefer to just sync them with a bit of VLC-fu. Which always takes longer than you thought, but hey. Now, what if there was an extremely simple way to just choose the subtitles for a given movie directly from VLC? The good news is that somebody wrote a beautiful VLC extension that does exactly that. He also wrote an introductory article about VLC extensions. Thank you, Jean-Philippe Andr . Installing the thing is as simple as downloading the subtitles extension and putting it in the VLC scripts folder. On Linux machines, the extensions folder can be found at ~/.local/share/vlc/lua/extensions/, on Windows APPDATA\VLC\lua\extensions. But seriously, don t use Windows. Once the extension is installed you can start watching a movie and then click View Subtitles to open the subtitles search interface. This is how it looks like: Now that the practical problem is solved, please take some time to look at the source code. VLC uses Lua as the language for writing extensions. I certainly cannot say that the web is full of resources about how to extend VLC using Lua, but the piece of code I wrote about in this post is certainly a good way to start hacking, and maybe scratch yet another itch.

4 October 2011

Emanuele Rocca: Gospel according to Tux

Some years ago I came across a really peculiar newsgroup post. It was not about technicalities of any sort. It was about history. A beautifully written history of computers. From the Turing machine to the Free Software world, the original author managed to capture all the important events of the computer revolution with a great deal of humor. Re-posting it here, it is just brilliant. The Gospel of Tux (v1.0) In the beginning Turing created the Machine. And the Machine was crufty and bodacious, existing in theory only. And von Neumann looked upon the Machine, and saw that it was crufty. He divided the Machine into two Abstractions, the Data and the Code, and yet the two were one Architecture. This is a great Mystery, and the beginning of wisdom. And von Neumann spoke unto the Architecture, and blessed it, saying, Go forth and replicate, freely exchanging data and code, and bring forth all manner of devices unto the earth. And it was so, and it was cool. The Architecture prospered and was implemented in hardware and software. And it brought forth many Systems unto the earth. The first Systems were mighty giants; many great works of renown did they accomplish. Among them were Colossus, the codebreaker; ENIAC, the targeter; EDSAC and MULTIVAC and all manner of froody creatures ending in AC, the experimenters; and SAGE, the defender of the sky and father of all networks. These were the mighty giants of old, the first children of Turing, and their works are written in the Books of the Ancients. This was the First Age, the age of Lore. Now the sons of Marketing looked upon the children of Turing, and saw that they were swift of mind and terse of name and had many great and baleful attributes. And they said unto themselves, Let us go now and make us Corporations, to bind the Systems to our own use that they may bring us great fortune. With sweet words did they lure their customers, and with many chains did they bind the Systems, to fashion them after their own image. And the sons of Marketing fashioned themselves Suits to wear, the better to lure their customers, and wrote grave and perilous Licenses, the better to bind the Systems. And the sons of Marketing thus became known as Suits, despising and being despised by the true Engineers, the children of von Neumann. And the Systems and their Corporations replicated and grew numerous upon the earth. In those days there were IBM and Digital, Burroughs and Honeywell, Unisys and Rand, and many others. And they each kept to their own System, hardware and software, and did not interchange, for their Licences forbade it. This was the Second Age, the age of Mainframes. Now it came to pass that the spirits of Turing and von Neumann looked upon the earth and were displeased. The Systems and their Corporations had grown large and bulky, and Suits ruled over true Engineers. And the Customers groaned and cried loudly unto heaven, saying, Oh that there would be created a System mighty in power, yet small in size, able to reach into the very home! And the Engineers groaned and cried likewise, saying, Oh, that a deliverer would arise to grant us freedom from these oppressing Suits and their grave and perilous Licences, and send us a System of our own, that we may hack therein! And the spirits of Turing and von Neumann heard the cries and were moved, and said unto each other, Let us go down and fabricate a Breakthrough, that these cries may be stilled. And that day the spirits of Turing and von Neumann spake unto Moore of Intel, granting him insight and wisdom to understand the future. And Moore was with chip, and he brought forth the chip and named it 4004. And Moore did bless the Chip, saying, Thou art a Breakthrough; with my own Corporation have I fabricated thee. Thou thou art yet as small as a dust mote, yet shall thou grow and replicate unto the size of a mountain, and conquer all before thee. This blessing I give unto thee: every eighteen months shall thou double in capacity, until the end of the age. This is Moore s Law, which endures unto this day. And the birth of 4004 was the beginning of the Third Age, the age of Microchips. And as the Mainframes and their Systems and Corporations had flourished, so did the Microchips and their Systems and Corporations. And their lineage was on this wise: Moore begat Intel. Intel begat Mostech, Zilog and Atari. Mostech begat 6502, and Zilog begat Z80. Intel also begat 8800, who begat Altair; and 8086, mother of all PCs. 6502 begat Commodore, who begat PET and 64; and Apple, who begat 2. (Apple is the great Mystery, the Fruit that was devoured, yet bloomed again.) Atari begat 800 and 1200, masters of the game, who were destroyed by Sega and Nintendo. Xerox begat PARC. Commodore and PARC begat Amiga, creator of fine arts; Apple and PARC begat Lisa, who begat Macintosh, who begat iMac. Atari and PARC begat ST, the music maker, who died and was no more. Z80 begat Sinclair the dwarf, TRS-80 and CP/M, who begat many machines, but soon passed from this world. Altair, Apple and Commodore together begat Microsoft, the Great Darkness which is called Abomination, Destroyer of the Earth, the Gates of Hell. Now it came to pass in the Age of Microchips that IBM, the greatest of the Mainframe Corporations, looked upon the young Microchip Systems and was greatly vexed. And in their vexation and wrath they smote the earth and created the IBM PC. The PC was without sound and colour, crufty and bodacious in great measure, and its likeness was a tramp, yet the Customers were greatly moved and did purchase the PC in great numbers. And IBM sought about for an Operating System Provider, for in their haste they had not created one, nor had they forged a suitably grave and perilous License, saying, First we will build the market, then we will create a new System, one in our own image, and bound by our Licence. But they reasoned thus out of pride and not wisdom, not forseeing the wrath which was to come. And IBM came unto Microsoft, who licensed unto them QDOS, the child of CP/M and 8086. (8086 was the daughter of Intel, the child of Moore). And QDOS grew, and was named MS-DOS. And MS-DOS and the PC together waxed mighty, and conquered all markets, replicating and taking possession thereof, in accordance with Moore s Law. And Intel grew terrible and devoured all her children, such that no chip could stand before her. And Microsoft grew proud and devoured IBM, and this was a great marvel in the land. All these things are written in the Books of the Deeds of Microsoft. In the fullness of time MS-DOS begat Windows. And this is the lineage of Windows: CP/M begat QDOS. QDOS begat DOS 1.0. DOS 1.0 begat DOS 2.0 by way of Unix. DOS 2.0 begat Windows 3.11 by way of PARC and Macintosh. IBM and Microsoft begat OS/2, who begat Windows NT and Warp, the lost OS of lore. Windows 3.11 begat Windows 95 after triumphing over Macintosh in a mighty Battle of Licences. Windows NT begat NT 4.0 by way of Windows 95. NT 4.0 begat NT 5.0, the OS also called Windows 2000, The Millenium Bug, Doomsday, Armageddon, The End Of All Things. Now it came to pass that Microsoft had waxed great and mighty among the Microchip Corporations; mighter than any of the Mainframe Corporations before it had it waxed. And Gates heart was hardened, and he swore unto his Customers and their Engineers the words of this curse: Children of von Neumann, hear me. IBM and the Mainframe Corporations bound thy forefathers with grave and perilous Licences, such that ye cried unto the spirits of Turing and von Neumann for deliverance. Now I say unto ye: I am greater than any Corporation before me. Will I loosen your Licences? Nay, I will bind thee with Licences twice as grave and ten times more perilous than my forefathers. I will engrave my Licence on thy heart and write my Serial Number upon thy frontal lobes. I will bind thee to the Windows Platform with cunning artifices and with devious schemes. I will bind thee to the Intel Chipset with crufty code and with gnarly APIs. I will capture and enslave thee as no generation has been enslaved before. And wherefore will ye cry then unto the spirits of Turing, and von Neumann, and Moore? They cannot hear ye. I am become a greater Power than they. Ye shall cry only unto me, and shall live by my mercy and my wrath. I am the Gates of Hell; I hold the portal to MSNBC and the keys to the Blue Screen of Death. Be ye afraid; be ye greatly afraid; serve only me, and live. And the people were cowed in terror and gave homage to Microsoft, and endured the many grave and perilous trials which the Windows platform and its greatly bodacious Licence forced upon them. And once again did they cry to Turing and von Neumann and Moore for a deliverer, but none was found equal to the task until the birth of Linux. These are the generations of Linux: SAGE begat ARPA, which begat TCP/IP, and Aloha, which begat Ethernet. Bell begat Multics, which begat C, which begat Unix. Unix and TCP/IP begat Internet, which begat the World Wide Web. Unix begat RMS, father of the great GNU, which begat the Libraries and Emacs, chief of the Utilities. In the days of the Web, Internet and Ethernet begat the Intranet LAN, which rose to renown among all Corporations and prepared the way for the Penguin. And Linus and the Web begat the Kernel through Unix. The Kernel, the Libraries and the Utilities together are the Distribution, the one Penguin in many forms, forever and ever praised. Now in those days there was in the land of Helsinki a young scholar named Linus the Torvald. Linus was a devout man, a disciple of RMS and mighty in the spirit of Turing, von Neumann and Moore. One day as he was meditating on the Architecture, Linus fell into a trance and was granted a vision. And in the vision he saw a great Penguin, serene and well-favoured, sitting upon an ice floe eating fish. And at the sight of the Penguin Linus was deeply afraid, and he cried unto the spirits of Turing, von Neumann and Moore for an interpretation of the dream. And in the dream the spirits of Turing, von Neumann and Moore answered and spoke unto him, saying, Fear not, Linus, most beloved hacker. You are exceedingly cool and froody. The great Penguin which you see is an Operating System which you shall create and deploy unto the earth. The ice-floe is the earth and all the systems thereof, upon which the Penguin shall rest and rejoice at the completion of its task. And the fish on which the Penguin feeds are the crufty Licensed codebases which swim beneath all the earth s systems. The Penguin shall hunt and devour all that is crufty, gnarly and bodacious; all code which wriggles like spaghetti, or is infested with blighting creatures, or is bound by grave and perilous Licences shall it capture. And in capturing shall it replicate, and in replicating shall it document, and in documentation shall it bring freedom, serenity and most cool froodiness to the earth and all who code therein. Linus rose from meditation and created a tiny Operating System Kernel as the dream had foreshewn him; in the manner of RMS, he released the Kernel unto the World Wide Web for all to take and behold. And in the fulness of Internet Time the Kernel grew and replicated, becoming most cool and exceedingly froody, until at last it was recognised as indeed a great and mighty Penguin, whose name was Tux. And the followers of Linus took refuge in the Kernel, the Libraries and the Utilities; they installed Distribution after Distribution, and made sacrifice unto the GNU and the Penguin, and gave thanks to the spirits of Turing, von Neumann and Moore, for their deliverance from the hand of Microsoft. And this was the beginning of the Fourth Age, the age of Open Source. Now there is much more to be said about the exceeding strange and wonderful events of those days; how some Suits of Microsoft plotted war upon the Penguin, but were discovered on a Halloween Eve; how Gates fell among lawyers and was betrayed and crucified by his former friends, the apostles of Media; how the mercenary Knights of the Red Hat brought the gospel of the Penguin into the halls of the Corporations; and even of the dispute between the brethren of Gnome and KDE over a trollish Licence. But all these things are recorded elsewhere, in the Books of the Deeds of the Penguin and the Chronicles of the Fourth Age, and I suppose if they were all narrated they would fill a stack of DVDs as deep and perilous as a Usenet Newsgroup. Now may you code in the power of the Source; may the Kernel, the Libraries and the Utilities be with you, throughout all Distributions, until the end of the Epoch. Amen.

7 November 2007

Roland Mas: Planet scores

Top posters in a few Debian-related Planets:
$ planet-scores.sh 
Planet Debian-FR :
     19 Rapha l Hertzog
      4 Roland Mas
      3 Jean-Christophe Dubacq
      2 Gr gory Colpart
      2 Alexis Sukrieh
Sometimes I think this should be renamed Planet Buxy.
Planet Debian-FR (utilisateurs) :
     10 Julien Candelier
      8 Emilien Macchi
      4 Guilhem Bonnefille
      3 Shams Fantar
      1 Rapha l Hertzog
      1 Olivier Berger (perso)
      1 Jean-Christophe Dubacq
      1 Jean-Baptiste H tier (djib)
      1 Eric Veiras Galisson
Newly added contributors to that planet have all their recent articles aggregated, not only the ones they wrote since they were added.
Planet Debian :
     40 Christian Perrier
      2 Russell Coker
      2 Raphael Geissert
      1 Wouter Verhelst
      1 Steve Kemp
      1 Romain Francoise
      1 NOKUBI Takatsugu
      1 Michal  iha 
      1 John Goerzen
      1 Joey Schulze
      1 Gerfried Fuchs
      1 Fathi Boudra
      1 Enrico Zini
      1 Emanuele Rocca
      1 Dirk Eddelbuettel
      1 David Welton
      1 Christine Spang
      1 Antti-Juhani Kaijanaho
      1 Adam Rosi-Kessel
Planet "Christian loves rugby".
debian-community.org :
      4 Holger Levsen
      3 Andrew Donnellan
      2 Evgeni Golov
      1 Wolfgang Lonien
      1 Rapha l Hertzog
      1 Martin Albisetti
      1 Marcos Marado
      1 Jean-Christophe Dubacq
      1 Cord Beermann
      1 Benjamin A'Lee
      1 Andreas Putzo
$
I know I have an encoding problem on some planets, but that script is a very basic curl+shell+sed+grep+recode+sort+uniq pipeline, and I only use it for the amusement value. Maybe I'll recode it with a proper RSS parser some day if I feel utterly bored.

6 November 2007

Emanuele Rocca: Historic GNU information

Today I've stumbled upon this old page about the GNU project.

The last update has been in 1997, it's nice to see how things are changed in 10 years... For instance:

Linux is a copylefted kernel, which uses GNU, BSD and X software to make a compleat free OS. This or FreeBSD or NetBSD are great OS's to use until the Hurd is stable.


There's also an interesting newspaper article about the impact of reward on motivation (1987).

7 March 2007

Emanuele Rocca: oobase and PostgreSQL

Problem:
accessing a PostgreSQL database from OpenOffice.org base under Debian.

Solution:
apt-get install unixodbc odbc-postgresql
# odbcinst -i -d -f /usr/share/psqlodbc/odbcinst.ini.template
$ cat /usr/share/doc/odbc-postgresql/examples/odbc.ini.template >> ~/.odbc.ini
  1. Set connection parameters in ~/.odbc.ini (username, password, host...)
  2. Check "Connect to an existing database" from the oobase Database Wizard, select ODBC, click on "Browse".

At this point your PostgreSQL database should be listed among the available data sources.
Yay!

6 March 2007

Emanuele Rocca: ...to wrein

Wrein, I'm not going to create a wordpress account.
As you can see I already have unmaintained blogs here and there. :-)

9 January 2006

Emanuele Rocca: darcs record == bzr shelve

darcs record behaves exactly as bzr shelve And I agree that it's really a cool feature.

29 December 2005

Emanuele Rocca: s/websearch/smartbookmark/

It seems that an Xfce plugin called websearch already exists. I renamed mine to xfce4-smartbookmark-plugin. I will use Google before choosing project names.
I will use Google before choosing project names.
I will use Google before choosing project names.
I will use Google before choosing project names.
I will use Google before choosing project names.

Next.