Search Results: "Sergio Talens-Oliag"

22 May 2022

Sergio Talens-Oliag: New Blog

Welcome to my new Blog for Technical Stuff. For a long time I was planning to start publishing technical articles again but to do it I wanted to replace my old blog based on ikiwiki by something more modern. I ve used Jekyll with GitLab Pages to build the Intranet of the ITI and to generate internal documentation sites on Agile Content, but, as happened with ikiwiki, I felt that things were kind of slow and not as easy to maintain as I would like. So on Kyso (the Company I work for right now) I switched to Hugo as the Static Site Generator (I still use GitLab Pages to automate the deployment, though), but the contents are written using the Markdown format, while my personal preference is the Asciidoc format. One thing I liked about Jekyll was that it was possible to use Asciidoctor to generate the HTML simply by using the Jekyll Asciidoc plugin (I even configured my site to generate PDF documents from .adoc files using the Asciidoctor PDF converter) and, luckily for me, that is also possible with Hugo, so that is what I plan to use on this blog, in fact this post is written in .adoc. My plan is to start publishing articles about things I m working on to keep them documented for myself and maybe be useful to someone else. The general intention is to write about Container Orchestration (mainly Kubernetes), CI/CD tools (currently I m using GitLab CE for that), System Administration (with Debian GNU/Linux as my preferred OS) and that sort of things. My next post will be about how I build, publish and update the Blog, but probably I will not finish it until next week, once the site is fully operational and the publishing system is tested.
Spoiler Alert: This is a personal site, so I m using Gitea to host the code instead of GitLab. To handle the deployment I ve configured json2file-go to save the data sent by the hook calls and process it asynchronously using inotify-tools. When a new file is detected a script parses the JSON file using jq and builds and updates the site if appropriate.

15 February 2015

Sergio Talens-Oliag: Retooling

I haven't blogged for a long time, but I've decided that I'm going to try to write again, at least about technical stuff. My plan was to blog about the projects I've been working on lately, the main one being the setup of the latest version of Kolab with the systems we already have at work, but I'll do that on the next days. Today I'm just going to make a list of the tools I use on a daily basis and my plans to start using additional ones in the near future. Shells, Terminals and Text Editors I do almost all my work on Z Shell sessions running inside tmux; for terminal emulation I use gnome-terminal on X, VX ConnectBot on Android systems and iTerm2 on Mac OS X. For text editing I've been using Vim for a long time (even on Mobile devices) and while I'm aware I don't know half of the things it can do, what I know is good enough for my day to day needs. In the past I also used Emacs as a programming editor and my main tool to write HTML, SGML and XML, but since I haven't really needed an IDE for a long time and I mainly use Lightweight Markup Languages I haven't used it for a long time (I briefly tried to use Org mode, but for some reason I ended up leaving it). Documentation formats and tools Since a long time ago I've been an advocate of Lightweight Markup Languages; I started to use LaTeX and Lout, then moved to SGML/XML formats (LinuxDoc and DocBook) and finally moved to plain text based formats. I started using Wiki formats (parsewiki) and soon moved to reStructuredText; I also use other markup languages like Markdown (for this blog, aka ikiwiki) and tried MultiMarkdown to replace reStructuredText for general use, but as I never liked Markdown syntax I didn't liked an extended version of it. While I've been using ReStructuredText for a long time, I recently found Asciidoctor and the Asciidoc format and I guess I'll be using it instead of rst whenever I can (I still need to try the slide backends and conversions to ODT, but if that works I guess I'll write all my new documents using Asciidoc). Programming languages I'm not a developer, but I read and patch a lot of free software code written on a lot of different programming languages (I wouldn't be able to write whole programs on most of them, but thanks to Stack Overflow I'm usually able to fix what I need). Anyway, I'm able to program in some languages; I write a lot of shell scripts and I go for Python and C when I need something more complicated. On the near future I plan to read about javascript programming and nodejs (I'll probably need it at work) and I already started looking at Haskell (I guess it was time to learn about functional programming and after reading about it, it looks like haskell is the way to go for me). Version Control For a long time I've been a Subversion user, at least for my own projects, but seems that everything has moved to git now and I finally started to use it (I even opened a github account) and plan to move all my personal subversion repositories at home and at work to git, including the move of all my debian packages from svn-buildpackage to git-buildpackage. Further Reading With the previous plans in mind, I've started reading a couple of interesting books: Now I just need to get enough time to finish reading them ... ;)

1 October 2011

Sergio Talens-Oliag: Static website generators

The last month I was supposed to work on a OpenStack related project, but for administrative reasons it has been delayed and I've tried to do small tasks to be able to finish them quickly and start the work on the main project when the issues get solved. As the delay has been longer than expected last Wednesday I've realized than on the last weeks I did a lot of small system administration tasks: With all the changes I did I noticed that I had to do something with our Intranet server; it is just a reverse proxy for a lot of different web services and its main page was one static HTML page with links to them, nothing else. In the long term maybe we will replace it with something based on Drupal or Lifeay, but for now I just wanted something to be able to organize the links and provide some information about the services for the new users without having to write HTML (I really like Agile Documentation Tools that let me focus on the content and forget about the markup), and started to look at some of them. My first idea was to use ikiwiki, as it has all the features I was looking for: I can use Markdown or reStructuredText to write the contents, the source pages are easily handled on a Version Control System, it supports the use of templates for the HTML, etc., but it seemed to me that using ikiwiki was like killing flies with a cannon (that's a Spanish say, I guess it's easy to understand it in English, no?) and I decided to review other tools to build static web sites. To make a long story short, I selected some tools that met my requirements and looked nice on their demo sites; after my first review I thought that Hyde was going to be my bet, as it uses technologies I'm already familiar with, but after trying it I saw that I was going to have a problem with documentation (the current Hyde version lacks it) and it was going to be more complicated that using ikiwiki. Before giving up I decided to review simpler tools, just in case, and after looking some of them I ended up using poole, a simple python script (the source is just one file and it only requires python-markdown to work). Before moving to the content I tried to adapt a couple of free themes to be used by the tool, but I didn't liked the result, so I went back to the plain style provided by the tool and added a logo and a background. With that simple look and feel I started to work with the content, splitting it into eight markdown files and a python macro to include a file that has all the links used on the site. While trying to make the main page look good I noticed how little I know about CSS, but using search engines I was able to build a two column block into the main page and publish the contents and with the help of some CSS enabled co-workers I changed the look and feel of the site in about 30 minutes. In summary, if you want a really simple website, you know a little bit of python and don't want to spend much time learning how to use a website generator then Poole is a good option. If you want something more complex I still think that ikiwiki is a good option, but YMMV.

30 November 2010

Sergio Talens-Oliag: freebsd-utils-8.1-2 to 2.4.patch

Sergio Talens-Oliag: The FreakyWall (Part 3: Packages)

In this post I'll describe the changes made to the kernel and some of the Squeeze packages for the Freaky Wall. The plan is to submit whishlist bugs to the BTS on the hope of having all what is needed for this project available on Debian after the Squeeze release, as my feeling is that a freeze is not the right time to push this changes... ;) I'm giving access here to all the changes made to the source packages, but if anyone wants the binary packages (amd64 only) send me an email and I'll give you the URL of an apt repository that contains all the modified packages (it's the one at work, that contains other modified packages) or, if there is interest, I can put them on people.debian.org. Kernel To be able to build the firewall we need a kFreeBSD kernel with some options not compiled on the version distributed with Debian. To compile the kernel I've followed the procedure described on the following debian-bsd mailing list post: http://lists.debian.org/debian-bsd/2010/09/msg00023.html Basically I've done the following:
    apt-get build-dep kfreebsd-8
    apt-get source kfreebsd-8
    cd kfreebsd-8-8.1
    cat >> debian/arch/amd64/amd64.config << EOF
    # Add pflog, pfsync, ALTQ and CARP support
    # ----------------------------------------
    # http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/firewalls-pf.html
    device      pf
    device      pflog
    device      pfsync
    options         ALTQ
    options         ALTQ_CBQ        # Class Bases Queuing (CBQ)
    options         ALTQ_RED        # Random Early Detection (RED)
    options         ALTQ_RIO        # RED In/Out
    options         ALTQ_HFSC       # Hierarchical Packet Scheduler (HFSC)
    options         ALTQ_PRIQ       # Priority Queuing (PRIQ)
    options         ALTQ_NOPCC      # Required for SMP build
    # http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/carp.html
    device      carp
    EOF
    vi debian/changelog 
    dpkg-buildpackage -B -uc
Once the package was built I installed the new kernel package and rebooted the machine. Utilities To be able to use some utilities related to pf I have built patched versions of three packages: On the next post I'll describe how I've configured the system, the network interfaces and the different utilities patched and compiled on this post.

Sergio Talens-Oliag: libpcap-1.1.1-2 to 2.1.patch

Sergio Talens-Oliag: tcpdump 4.1.1-1 to 1.1.patch

24 November 2010

Sergio Talens-Oliag: The Freaky Wall (Part 2: Initial Installation)

For the Freaky Wall I have installed a Debian GNU/kFreeBSD system using the installer with ZFS support announced on: http://robertmh.wordpress.com/2010/09/06/debian-installer-with-zfs/ I used the mini.iso found on: http://people.debian.org/~rmh/zfs/kfreebsd-amd64/monolithic/mini.iso the 12th of October of 2010; as I had some problems and reported them to Robert is possible that the current image solves part of them. Installation plan I did a standard installation on a machine with two hard disks, but only used the first one from the installer. The plan was to use ZFS with RAID-1, but current versions of grub do not support booting from a ZFS + RAID file system, so I had to use the same technique used for Linux for a long time; three partitions: a swap partition, a small /boot partition and a big partition for /; / and /boot were formated to use ZFS. First reboot After the installation the system failed to boot because of a bug when building the /boot/grub/grub.cfg (some routes were missing a //@ prefix); to be able to boot Iwe edited the config on the grub prompt and later fixed the file:
    --- grub.cfg.orig       2010-10-13 16:40:39.000000000 +0200
    +++ grub.cfg    2010-10-13 18:38:47.535436766 +0200
    @@ -64,7 +64,7 @@
            set root='(hd0,1)'
            search --no-floppy --fs-uuid --set a371979bb836d1fe
            echo                    'Loading kernel of FreeBSD 8.1-1-amd64 ...'
    -       kfreebsd                /kfreebsd-8.1-1-amd64.gz
    +       kfreebsd                //@/kfreebsd-8.1-1-amd64.gz
            insmod part_msdos
            insmod zfs
            set root='(hd0,3)'
    @@ -75,7 +75,7 @@
            insmod zfs
            set root='(hd0,1)'
            search --no-floppy --fs-uuid --set a371979bb836d1fe
    -       kfreebsd_module         /zfs/zpool.cache type=/boot/zfs/zpool.cache
    +       kfreebsd_module         //@/zfs/zpool.cache type=/boot/zfs/zpool.cache
            set kFreeBSD.vfs.root.mountfrom=zfs:dkfbf1-ad4s3
            set kFreeBSD.vfs.root.mountfrom.options=rw
      
I haven't tested the installer since that day, but I believe that the current ZFS installer was fixed by Robert to deal with that problem. Once the system was booted I had to fix a couple of things: Adjusting ZFS to do RAID-1 On the second disk I created the same partitions as the ones on the first disk using parted; the final result was:
    # parted -l     
    Model: ST3250620NS/3BKS (ide)
    Disk /dev/ad6: 250GB
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos
    Number  Start   End     Size    Type     File system  Flags
     1      1049kB  256MB   255MB   primary
     2      256MB   4256MB  4000MB  primary
     3      4256MB  250GB   246GB   primary
    Model: ST3250620NS/3BKS (ide)
    Disk /dev/ad4: 250GB
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos
    Number  Start   End     Size    Type     File system     Flags
      1      1049kB  256MB   255MB   primary
      2      256MB   4256MB  4000MB  primary  linux-swap(v1)
      3      4256MB  250GB   246GB   primary
To use the second partition of both disks as swap I added the following to /etc/fstab:
    /dev/ad4s2      none            swap    sw              0       0
    /dev/ad6s2      none            swap    sw              0       0
To configure the mirroring for the root file system I did the following:
    zpool attach dkfbf1-ad4s3 ad4s3 ad6s3
As the /boot can't work as a replica I adjusted it to make two copies of every file:
    zfs   set copies=2     dkfbf1-ad4s1
Leaving the second disk copy alone, although my plan is to configure it to hold a copy of the /boot partition synchronized with rsync each night. After all those changes the system didn't boot, as the grub-pc generates a buggy /boot/grub/grub.cfg; the problem is on the /etc/grub.d/10_kfreebsd section:
    ### BEGIN /etc/grub.d/10_kfreebsd ###
    menuentry 'Debian GNU/kFreeBSD, with kFreeBSD 8.1-1-amd64' --class debian \
      --class gnu-kfreebsd --class gnu --class os  
          insmod part_msdos
          insmod zfs
          set root='(hd0,1)'
          search --no-floppy --fs-uuid --set a371979bb836d1fe
          echo 'Loading kernel of FreeBSD 8.1-1-amd64 ...'
          kfreebsd /kfreebsd-8.1-1-amd64.gz
          set kFreeBSD.vfs.root.mountfrom=unknown:/dev/ad4s3
          set kFreeBSD.vfs.root.mountfrom.options=rw
     
    ### END /etc/grub.d/10_kfreebsd ###
To fix it there has to be a copy of the modules for ZFS on the boot partition (in my case I moved the /lib/modules directory to /boot and created a link on the root partition to the new directory):
    cd /boot
    mkdir lib
    mv /lib/modules lib
    cd /lib
    ln -s ../boot/lib/modules
And instead of fixing the /etc/grub.d/10_kfreebsd code I wrote a new script (/etc/grub.d/09_zfs_kfreebsd) that creates the right config for my current configuration on the grub.cfg file:
    #!/bin/sh
    prefix=/usr
    exec_prefix=$ prefix 
    bindir=$ exec_prefix /bin
    libdir=$ exec_prefix /lib
    . $ libdir /grub/grub-mkconfig_lib
    prepare_boot_cache="$(prepare_grub_to_access_device $ GRUB_DEVICE_BOOT    sed -e "s/^/\t/")"
    kfreebsd_versions="$(ls /lib/modules/)"
    zfs_root_device="$(zfs list   awk '/\/$/   print $1  '   head -1)"
    for kversion in $kfreebsd_versions; do
      cat << EOF
    # Entry when using ZFS (we have issues with /etc/grub.d/10_kfreebsd)
    menuentry 'Debian GNU/kFreeBSD, with kFreeBSD $kversion and ZFS' --class debian --class gnu-kfreebsd --class gnu --class os  
    $ prepare_boot_cache 
          echo                    'Loading kernel of FreeBSD $kversion ...'
          kfreebsd                //@/kfreebsd-$kversion.gz
          kfreebsd_module_elf     //@/lib/modules/$kversion/opensolaris.ko
          kfreebsd_module_elf     //@/lib/modules/$kversion/zfs.ko
          kfreebsd_module         //@/zfs/zpool.cache type=/boot/zfs/zpool.cache
          set kFreeBSD.vfs.root.mountfrom=zfs:$zfs_root_device
          set kFreeBSD.vfs.root.mountfrom.options=rw
     
    EOF
    done
I solved the problem this way to have a working solution that does not break with squeeze upgrades, assuming that a future grub-pc package will deal well with my config and I'll be able to remove this script, but I guess I'll have to install it from backports. The entry generated by the script when called from update-grub will be similar to:
    ### BEGIN /etc/grub.d/09_zfs-kfreebsd ###
    # Entry when using ZFS (we have issues with /etc/grub.d/10_kfreebsd)
    menuentry 'Debian GNU/kFreeBSD, with kFreeBSD 8.1-1-amd64 @ ITI' --class debian --class gnu-kfreebsd --class gnu --class os  
          insmod part_msdos
          insmod zfs
          set root='(hd0,1)'
          search --no-floppy --fs-uuid --set a371979bb836d1fe
          echo                    'Loading kernel of FreeBSD 8.1-1-amd64 ...'
          kfreebsd                //@/kfreebsd-8.1-1-amd64.gz
          kfreebsd_module_elf     //@/lib/modules/8.1-1-amd64/opensolaris.ko
          kfreebsd_module_elf     //@/lib/modules/8.1-1-amd64/zfs.ko
          kfreebsd_module         //@/zfs/zpool.cache type=/boot/zfs/zpool.cache
          set kFreeBSD.vfs.root.mountfrom=zfs:dkfbf1-ad4s3
          set kFreeBSD.vfs.root.mountfrom.options=rw
     
    ### END /etc/grub.d/10_iti-kfreebsd ###
And after rebooting the machine with this new configuration the system boots OK. On my next post I'll continue explaining how to compile a kernel that supports the use of the OpenBSD Packet Filter and related technologies (CARP, pflog, etc.).

22 November 2010

Sergio Talens-Oliag: The Freaky Wall (Part 1: Why?)

This post and the next to come are about a project I'm doing at work that I've called The Freaky Wall. The project has its origins on the idea that using multiples technologies is better for security; almost all the servers I use are running Debian GNU/Linux and use iptables locally, so when I decided that we had to build new firewalls at work I thought it was a good idea to look at different technologies, that is, a different kernel and firewalling tools. As I wanted to avoid iptables and the Linux kernel my first idea was to go after the free BSD systems (FreeBSD, OpenBSD or NetBSD), and soon realized that pf (the OpenBSD Packet Filter) was the way to go; it has a clean syntax and includes advanced features like CARP and pfsync that allow me to build redundant firewalls. Before going after the standard systems I looked at pfSense a firewall appliance built on top of FreeBSD that uses a PHP interface to do everything. At first it seemed that it was going to be a good option, but soon I felt that I wasn't in control of what the system was doing and I had to change the PHP code to do trivial things (I wanted to configure IP aliases on a CARP interface and it was not possible with the web interface, while it is trivial to do using the standard system configuration files), so I left the idea of using it. The second option was to use OpenBSD directly, as it is the system were pf has been developed. Soon I saw that I was going to be able to do what I wanted with the system, but I missed the Debian's way of installing and upgrading the system and the list of packages available. For different reasons the firewall project was left in a limbo for a little while and when I went back to it I already had to upgrade my test systems to a new OpenBSD release; after reading a little bit about how to upgrade and not liking the idea of doing it I remembered that jordi suggested that if we only want the kernel and the firewall tools the Debian GNU/kFreeBSD port could be an option instead of OpenBSD or FreeBSD. Before trying to install the Debian GNU/kFreeBSD system I saw Robert's post about a Debian installer with ZFS support and I decided to start with it, as the use of ZFS will allow us to use software RAID-1 and snapshots, something we have on almost all our Linux servers (we use software RAID for redundancy and LVM snapshots to be able to do our backups at any time of the day with consistent data, but that is for another post). On my next post I'll explain how I did the initial installation with ZFS, and after that I'll explain the changes I did to the kernel and some of the packages to be able to build a firewall as described on the Firewalling with PF document (that is, I needed pfctl, pflogd, a tcpdump with pflog support, the pf's ftp-proxy, etc.) and on the last document I'll explain how I've configured the firewalls The Debian Way .

26 October 2010

Sergio Talens-Oliag: Debian Squeeze, PowerPC and the Linux Containers

Two kids, their really busy mother and my paid job leave me without much time to blog or do Debian related work lately (well, at least on my free time, I do Debian related things at work, but mostly as a user, not as a developer). Anyway, a couple of weeks ago I decided it was time to upgrade my home servers to Squeeze and I did it, but it was harder than expected. At home I'm using two old laptops as servers, an old Aluminium PowerBook and an Asus EeePC; the Asus was installed to replace an older PowerBook (a really old one, BTW) that I was using as home server since my father gave it to me. The plan was to use OpenVZ on the Asus to move all the PowerPC services to a couple of Virtual Environments, but as I wanted to migrate and change almost all the services I never got enough free time to finish the job and when the old PowerBook hardware failed I replaced it with another PowerBook that I wasn't using anymore, but instead of reinstalling the machine I did a clean Lenny install using a kernel with support for linux-vserver (OpenVZ does not work on PowerPC) and transformed the old machine installation (it was an Etch installation at the time) into a Virtual Private Server that run on the new hardware. Having both systems running I upgraded the VPS to Lenny and, as usually happens, left the things as they were without consolidating the services into only one machine, as I initially planned. With this state of affairs I upgraded the Asus to Squeeze without much trouble (in fact I installed a kernel without OpenVZ support, as the services I use from this laptop were running on the host and not on a VE) and did the same with the PowerPC host, but to my surprise the linux-vserver VPS failed to start with a message that seemed to imply that the VServer support was not enabled. I should have filled a bug on the BTS then, but as I looked into how to solve the issue I found bugs saying that the meaning of the message was that I had no support for linux-vserver and I needed to start the VPS ASAP, as it was the machine that runs my SMTP server. Before doing a restore of my last backup I did some digging and found a lot of messages recommending to move OpenVZ and Linux-VServer virtual machines to LXC and decided to give it a try. First I built a container on the Asus and it worked OK, after that I did the same on the PowerPC, but the script failed; luckily the patch was trivial, the problem was on the /usr/lib/lxc/templates/lxc-debian script; it uses arch to get the Debian architecture, but for powerpc it gives ppc instead of powerpc, so it needs to be fixed on the script (Note to self: I have to submit bug + patch to the lxc package to fix it). After creating this container and trying it I tried to boot my old VPS with a LXC configuration: After a couple of tries I noticed that the system was not booting because it was missing the devices files needed; to fix it I copied the /dev directory of my first LXC test and using a chroot I also removed the udev packages from the container. After that last changes the machine booted as expected and all services were running OK. To summarize, I decided to do the move to LXC and fixed the configuration to boot the virtual machines on each restart: I know that LXC is still missing some functionality (I hate the way the container stop function kills everything instead of doing a run-level change, I guess I'll be using hacks until I move to a newer kernel with the proper support enters into Debian), but having the code on the mainline kernel is a great bonus and the user level utilities are good enough for my home needs... and I hope they'll arrive to a point where we'll be able to migrate the OpenVZ containers at work (we are using Proxmox and the support of the OpenVZ patchset is starting to worry us). On my next post: The Freak Firewall or The Story of a HA Firewall based on OpenBSD's pf running on Debian GNU/kFreeBSD hosts.

16 September 2009

Sergio Talens-Oliag: Hugo meets Marc

As promised a photo of the first meeting between Hugo and Marc. Hugo meets Marc With luck this afternoon the whole family will be at home.

Sergio Talens-Oliag: Redmine

I've been using Subversion and Trac for some years now, and I have encouraged its use at work since the last couple of years, with the undesired effect of having to maintain four different Trac installations with different database systems (SQLite3 and PostgreSQL), plugins (more than 15 on the big servers), authentication systems (htpass files, LDAP and a database based system) and tons of projects published (two internal servers have 64 and 16 projects, one of the client system has 33 projects and there is only one single project installation, but it is living at a client's system). Yesterday night, while reading Planet Debian I found a post from John Goerzen about tools to replace Trac, including the option to use Git as the project VCS. In the post he talks about different options, mainly projects that I would categorize as issue tracking systems (mantis, roundup, etc.), but it also talks about Redmine, a project management system implemented using the Ruby on Rails framework that is similar to Trac. As it looked interesting I downloaded, installed and executed an instance in about 15 minutes (I love the systems that support sqlite3 for this quick tests, not having to touch real database servers speeds up simple tests a lot). I played a little bit with the system and I believe that I will spend some more time testing it at work next week, as it looks quite promising; the standard version has almost all the features I'm interested in without the need to install additional plugins and it can do most of the things I was missing from Trac to do lightweight project management. I evaluated ]project-open[ to use it together with Trac for our internal project management tasks, mainly because we miss important features from Trac, like having clean systems to view the tasks of a user in all projects or a clean way to do the project planning using tickets and gantt charts. Of course there are ways to do it, but the plugins I've tried are not as good and simple as I would like. The problem with the use of ]project-open[ is that I don't really like it for us, as it has tons of features that I feel we don't need nor will use and, on a first try, the system seemed difficult to deploy and maintain, probably because my lack of knowledge about OpenACS and TCL. In fact we still don't have ]po[ running at work because I was unable to to integrate the authentication system with our LDAP server on my first tries and have had no time to investigate further since then. The good thing about trying Redmine is that if we don't end up using it at least I can take the most of this opportunity by looking at Ruby on Rails and the Ruby Programming Language, at least from the administration side, as I have never looked at it seriously.

Sergio Talens-Oliag: Encrypting a Debian GNU/Linux installation on a MacBook

A couple of weeks ago I updated my Debian Sid setup on the MacBook to use disk encryption; this post is to document what I did for later reference. The system was configured for dual booting Debian or Mac OS X using refit and grub2 as documented on the Debian Wiki; I don't use the Mac OS X system much, but I left it there to be able to test things and be able to answer questions of Mac OS X users when I have to. The Debian installation was done using two primary partitions, one for swap (I used a partition to be able to suspend to disk without troubles) and an ext3 file system used as the root file system. The plan was to use the Debian Installer to do the disk setup and recover the Sid installation from a backup once the encrypted setup was working OK. Backup for later recovery My first step was to install all the needed packages on the original system; basically I verified that I had the lvm2 and cryptsetup packages installed. The second step was to backup the root file system; to do it I changed to run level 1 and copied the files to an external USB disk using rsync. My third step was to boot into Mac OS X to reduce the space assigned to it; I had a lot of free space that I didn't plan to use with Mac OS X and I thought that this was the best occasion to reassign it to the Debian file system. Encrypted Lenny installation Now the machine was ready for the installer. As I formatted the system a couple of weeks ago I used a daily build of the Lenny Debian Installer, now that Lenny is out I would have used the official version. I booted the installer and on the partition disk step I selected the manual method; I left sda1 and sda2 as they were (the Mac OS X installation uses them) and set up sda3 and sda4 as follows: Note that I decided to put /boot on a plain ext3 partition to be able to use grub2 as the boot loader (if we put the kernel on an LVM logical volume we need to use lilo as the boot loader). Once sda4 was adjusted as LVM I entered on the LVM setup and created a LVM Volume Group (VG) with the name debian, using sda4 as the physical volume. Once the VG was defined I created a couple of Logical Volumes (LV): I left some space unallocated to be able to create LVM snapshots (I use them to do backups, I'll post about it on the next days). Once the LV were ready I finished with the LVM setup and went back to the partitioner to configure the Logical Volumes: Once both encrypted volumes were ready I entered on the Configure the encrypted volumes menu and the installer formatted the volumes for encryption and asked for the debian-root pass phrase. Back on the main partitioning menu I set up the debian-root_crypt encrypted volume: I didn't need to touch the debian-swap_crypt, it was configured automatically as swap because I choose a random encryption key. At this point I was finished with the partitioning; to finish I installed a minimal system and rebooted to try the system. As I had changed the disk layout I had to re-sync the partition tables from refit; once that was done I was able to boot from the newly installed system. Setting up suspend to disk I was using s2disk to suspend the system; to test if it still worked with the new setup I installed the uswsusp package and adjusted the resume device on the /etc/uswsusp.conf to /dev/mapper/debian-swap_crypt. After my first try I noticed that the resume step failed with the encrypted swap partition because it was using a random key, which means that the swap contents are unrecoverable after a reboot. Looking at the cryptsetup documentation I found that the solution was to use a derived key for the swap partition instead of a random one. The command sequence was as follows:
# disable swap
swapoff -a
# close encrypted volume
cryptsetup luksClose debian-swap_crypt
# change the swap partition setup on the /etc/crypttab file
sed -e -i 's%^debian-swap.*%debian-swap_crypt /dev/mapper/debian-swap debian-root_crypt cipher=aes-cbc-essiv:sha256,size=256,swap,hash=sha256,keyscript=/lib/cryptsetup/scripts/decrypt_derived,swap%' /etc/crypttab
# open the encrypted volumes with the new setup
/etc/init.d/cryptdisks start
# enable swap
swapon -a
# update the initrd image
update-initramfs -u
After executing all those commands the suspend to disk system worked as expected. Recovering the original system If I were going to reinstall the system completely I would have finished here, but in my case I wanted to recover my original system setup (except the minimal changes required to use the encrypted passions, of course). To recover my old installation I backed up some files (/etc/fstab, /etc/crypttab, /etc/uswsusp.conf and the current /boot contents to be able to boot in case of failure with my old kernel) from the current installation, after that I recovered all the files from the initial backup (except the ones just saved) using rsync again and regenerated the initrd images of my old kernels:
update-initramfs -u -k all
After that I rebooted and everything worked as on my original installation (except for the disk encryption, of course).

Sergio Talens-Oliag: Encrypting a Debian GNU/Linux installation (followup)

Yesterday I received a mail message from a Debian user called Ekrem Erdem about my previous post, proposing a different partitioning schema that I found interesting. The basic idea is to swap the order of the technologies, that is, use LVM on top of an encrypted partition instead of encrypting logical volumes. I never thought about this schema because I always use LVM on servers and that is one of the fist things I setup (just after software RAID-1, if the machine has two hard drives); when I was evaluating how to setup my system for encryption I started with the LVM setup and never looked back. The advantage of this setup is that there is only one pass phrase (the one used to unlock the encrypted partition, sda4 in my case), eliminating the need of derived keys (i. e. my swap setup) or key files (I use them to mount snapshots of the encrypted partition non interactively). On the negative side I believe that this setup looses some flexibility: Anyway if the plan is to encrypt all the file systems on a laptop the proposed setup is simpler and, IMHO, as safe as my configuration (remember that my keys are related). I'm not going to change my setup now (it works great), but I'll probably try this one in the future if I need an encrypted setup on a different machine.

Sergio Talens-Oliag: Encrypting a Debian GNU/Linux installation (take 3)

After my followup to the Tuesday post I've received some additional comments and I'm writing this entry to close the subject... ;) One of the comments was from Gunnar to tell me that the followup setup was the same provided by the automatic partitioner of the Debian Installer since 2007. I was unaware of that because until some weeks ago I never tried to install a system with encryption support and when I did it on my laptop I used the manual setup because I wanted to keep the MacOS X partitions. Anyway my followup blog entry made sense anyway, as I just wanted to comment my thoughts about the advantages and disadvantages of each partitioning schema. I also received a couple of messages proposing the use of three layers to keep the flexibility of the original setup and the simplicity of the second; the setup is as follows: With the LVM at the lower level you get the advantages of my setup (mix encrypted and unencrypted partitions, the crypted volume can use multiple physical volumes, etc.) and the advantages of the second setup (only one key for all the encrypted file systems). I believe that this setup is a little too much for a laptop, but can be a good option if you need encrypted file systems on a server.

9 September 2009

Sergio Talens-Oliag: Marc

At 17:55 of the 21th of August of 2008 (four days late from the expected schedule) passenger Marc Talens i Blasco landed at La F Hospital in Valencia. Marc On my next post... Hugo meets Marc!

Sergio Talens-Oliag: Free Software Summer

On recent years I've only been in one or two free software related conferences per year, usually because I've been invited to go for a day or two and give a talk. The truth is that my day to day activities don't leave me enough time to contribute or participate a little bit more in free software projects, but this year I decided (and negotiated) that I had to go to Debconf, as it is the best opportunity to go to a Debian conference that I'm going to have in the near future. So this summer it is going to be a free software summer, the 3rd of July I'll be giving a talk at the Jornades de Programari Lliure in Barcelona and from the 23th to the 30th or 31th I'll be at Debconf 9 in C ceres. In Barcelona I will talk about building IT infrastructures using free software, explaining which programs I use, why I've choosen them and how I configure things depending on my needs and in C ceres I will be just listening and exchanging gpg keys.

18 September 2008

Sergio Talens-Oliag: My first nginx module

Yesterday I sent a message to the Nginx list to announce the availability of my first module for that web server. The module is used to do HTTP Basic Authentication agains PAM instead of using an htpasswd file; I wrote it because I wanted to authenticate against OpenLDAP and PostgreSQL and PAM already has support for that ;) The code is available here and on the readme there are instructions on how to build a patched debian package with the module included.

14 August 2008

Sergio Talens-Oliag: Summertime, change times

After less than a week of vacation I've decided it was time to write a short blog entry, as I plan to write at least another one in some days, as we are waiting the arrival of Marc, our second son, for this week or the next one. In the last months I haven't done what I said in my last posts, I'm quite busy with the rest of my life and blogging or keeping my home computing infrastructure is not on the top list. Anyway I still have managed to do some things like giving a talk about virtualization on the VII Jornades de Programari Lliure, not going to Debconf8 (next year should be the one, the conference is in Spain and I have enough time to prepare it, including a possible trip with all the family) or do a partial server migration at home, leaving two machines to do the work of one. My plan for the migration has changed and if time permits I'll try to do it in the next couple of weeks; now I plan to move my current servers to an ASUS EeePC with 2GB of RAM and an external USB disk (it is a lot smaller and the hardware is still faster than my old server) and I'll use OpenVZ instead of Linux-Vserver for virtualization (OpenVZ enabled kernels are available for Lenny).

27 February 2008

Sergio Talens-Oliag: Tips &#x26; Tricks: plone, nginx and path rewriting

The problem On a couple of Debian Etch systems we have a plone-site that is published using a backport of the nginx web server. The Zope instance is running on the standard port and serves the Plone contents under the /plone path. Initially we were publishing the site to the external world using an https site served by nginx using the following entry on the configuration:
  location /plone/  
    proxy_pass http://plone:9673;
    include    /etc/nginx/proxy.conf;
   
The proxy.conf contents are quite standard:
  # proxy.conf
  proxy_redirect                  off;
  proxy_set_header                Host $host;
  proxy_set_header                X-Real-IP $remote_addr;
  proxy_set_header                X-Forwarded-For $proxy_add_x_forwarded_for;
  client_max_body_size            0;
  client_body_buffer_size         128k;
  proxy_connect_timeout           90;
  proxy_send_timeout              90;
  proxy_read_timeout              90;
  proxy_buffer_size               4k;
  proxy_buffers                   4 32k;
  proxy_busy_buffers_size         64k;
  proxy_temp_file_write_size      64k;
With this settings we see the /plone contents using the same path that is used by the Zope instance, but after testing we have decided to change the /plone path and server the contents under the /web path. The Wrong Solution The fist option I though about was quite simple, rename the Zope's plone object to web. Seems reasonable and simple for someone without Zope experience (I don't administer the internals of the Zope/Plone site), but now I know that it is a very big mistake, because renaming objects in Zope in not cheap, as it implies that the server has to modify all the contents of the renamed object and the operation can take a very long time. With my ignorance I tried to rename the plone object using the Zope administrative interface and after a minute or so I cancelled the page loading that was running on my browser, thinking that I had cancelled the rename operation. To make a long story short I'll tell you that the operation was still running and after several hours the folder was renamed (in fact I noticed when the good solution broke, as I had already solved the problem using the next method), but something went wrong and part of the site functionality was broken... the final solution to the debacle has been to recover a backup of the Zope instance older than the rename operation and continue from that copy. The Right Solution (TM) It seems that Zope has a couple of systems to do Virtual Hosting and the best option is the use of the product called Virtual Host Monster, a weird and confusing system (IMHO, of course), that does the job once the right configuration settings are in place. The best solution to our problem was to modify the requests done by the reverse proxy without touching anything on the Plone site (the original one already had a Virtual Host Monster object installed and that was the only thing that we needed to add). The nginx configuration for the new /web path is the following:
  location /web/  
    proxy_pass http://plone:9673/plone/VirtualHostRoot/_vh_web/;
    include    /etc/nginx/proxy.conf;
   
With this change, when the user asks for anything under the /web/ path the Zope server gets the contents traversing the /plone object and adding to it the elements that appear after the VirtualHostRoot component, ignoring components that start with the _vh_ prefix (the protocol and host name of the requests are not modified, as we did not touched that). Once the object is found, the server rewrites the URLs included on the HTML files using the path components that appear after the VirtualHostRoot one, including the suffix of the components that start with the prefix _vh_. For example, when the Zope server receives a request for an URL like:
  http://plone:9673/plone/VirtualHostRoot/_vh_web/home
it publishes the content found on:
  http://plone:9673/plone/home
but the HTML files returned assume that their base URL is:
  http://plone:9673/web/home

Next.

Previous.