A bash script to backup mysql databases, with separate schema and plain-text dump files (INSERT commands) for each database. Makes it easy to restore individual databases or copy them into dev/test servers. Keeps 30 days worth of backups in separate YYYY-MM-DD directories. Also dumps the mysql grants to a separate file mysql-grants.txt , and creates a sometimes conveniently useful mysql-create-databases.txt containing CREATE DATABASE commands. Based on my similar backup script for postgresql databases.

#! /bin/bash
# mysql backup script
#
# $Id: backup-mysql.sh,v 1.8 2013/06/21 02:20:39 cas Exp $
#
# by Craig Sanders <cas@taz.net.au>
# this script is public domain.  do whatever you want with it.
# set user/password here, or leave undefined to rely on ~/.my.cnf
#MYSQL_USER="root"
#MYSQL_PWD="SECRET"
[ -n "$MYSQL_USER" ] && ARGS="$ARGS -u$MYSQL_USER"
[ -n "$MYSQL_PWD" ] && ARGS="$ARGS -p$MYSQL_PWD"
DATABASES=$(mysql $ARGS -D mysql --skip-column-names -B -e 'show databases;'   egrep -v 'information_schema' );
BACKUPDIR=/var/backups/mysql
YEAR=$(date +"%Y")
MONTH=$(date +"%m")
DAY=$(date +"%d")
DATE="$YEAR-$MONTH/$YEAR-$MONTH-$DAY"
# make sure that perms on all created files/dirs are rwxr-x---
umask 0027
mkdir -p "$BACKUPDIR/$DATE"
cd "$BACKUPDIR/$DATE"
mysql $ARGS -Bse "SELECT CONCAT('SHOW GRANTS FOR \'', user ,'\'@\'', host, '\';') FROM mysql.user"   \
    mysql $ARGS -Bs   \
    sed 's/$/;/g'  > mysql-grants.txt
# "create database" lines, for easy cut-and-paste
> mysql-create-databases.txt
for i in $DATABASES ; do
  echo "CREATE DATABASE $i;" >> mysql-create-databases.txt
done
ARGS2="--skip-opt --no-create-db --no-create-info --single-transaction --flush-logs"
for db in $DATABASES ; do 
    echo -n "backing up $db: schema..."
    mysqldump $ARGS --no-data $db > $db.schema.sql
    echo -n "extended..."
    mysqldump $ARGS $ARGS2 --extended-insert $db > $db.data.extended.sql
    # uncomment if you want complete-insert dumps too.
    #echo -n "data...complete..."
    #mysqldump $ARGS $ARGS2 --complete-insert $db > $db.data.complete.sql
    echo -n "compressing..."
    gzip -9fq $db.schema.sql $db.data.extended.sql #$db.data.complete.sql
    echo "done."
done
echo deleting backups older than 30 days:
find "$BACKUPDIR" -mindepth 1 -a -type d -mtime +30 -print0   xargs -0r rm -rfv

backup-mysql.sh is a post from: Errata

Due to boredom and needing something to do to keep my brain from atrophying, I ve decided to polish some of the rough edges off some of my scripts so that they re suitable for publication. here s the first of them, a perl script to list kernels and other grub menu entries, with numeric indexing suitable for use with grub-set-default and grub-reboot output looks like this:

# grub-list-kernels.pl
   0    Debian GNU/Linux    
   1    Advanced options for Debian GNU/Linux   
 1>0    Debian GNU/Linux, with Linux 3.9-1-amd64    
 1>1    Debian GNU/Linux, with Linux 3.9-1-amd64 (recovery mode)    
 1>2    Debian GNU/Linux, with Linux 3.8-2-amd64    
 1>3    Debian GNU/Linux, with Linux 3.8-2-amd64 (recovery mode)    
 1>4    Debian GNU/Linux, with Linux 3.8-1-amd64    
 1>5    Debian GNU/Linux, with Linux 3.8-1-amd64 (recovery mode)    
 1>6    Debian GNU/Linux, with Linux 3.7-trunk-amd64    
 1>7    Debian GNU/Linux, with Linux 3.7-trunk-amd64 (recovery mode)    
 1>8    Debian GNU/Linux, with Linux 3.2.0-4-amd64  
 1>9    Debian GNU/Linux, with Linux 3.2.0-4-amd64 (recovery mode)  
   2    Network boot (iPXE) 
   3    Bootable floppy: LSI    
   4    Bootable floppy: freedos-bare   
  Default: 0  "Debian GNU/Linux"

(this example tells me i need to uninstall some old kernels) and here s the script:

#! /usr/bin/perl 
# parse and list the boot entries in the grub boot menu (grub.cfg)
#
# $Id: grub-list-kernels.pl,v 1.8 2013/06/20 10:36:47 cas Exp $
# Copyright (C) 2007, 2008, 2009, 2010, 2011, 2012, 2013 Craig Sanders <cas@taz.net.au>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.
# Partial Revision History:
# 2013-06-20 - better handling of non-numeric default saved entries, code still uglyish but better commented.
# 2013-05-13 - first stab at support for submenus...works but uglified the code
# 2010-11-03 - rewrote for grub2 (grub.cfg rather than menu.lst)
use strict;
my $cfg = shift   '/boot/grub/grub.cfg';
my $grubenv='/boot/grub/grubenv';
# hash of kernels (menu entry index -> menu entry description) 
# e.g. '4' => 'Debian GNU/Linux, with Linux 3.8-1-amd64'
my %K = ();
# hash of menuentry_id_option -> menuindex
# e.g. 'gnulinux-3.8-1-amd64-advanced-6bb6d228-0581-49ae-9d49-dd148c273ecc' => 4
my %M = ();
# read in the menu entries.
open(MENU,"<$cfg")   die "couldn't open $cfg for read: $!\n" ;
my $insubmenu = 0;
my $menuitem=0;
my $submenuitem=0;
my $prefix='';
my $menuindex='';
my $kernel='';
my $menuidoption='';
while (<MENU>)  
  #next unless /^\s*menuentry\s/io;
  chomp;
  if ($insubmenu ne 0)  
    next unless /^\s*(menuentry submenu)\s ^ $/io;
    if (m/^ $/)   $insubmenu = 0 ; $prefix='' ; next  ;
    $menuindex = $prefix . $submenuitem++;
    else  
    next unless /^\s*(menuentry submenu)\s/io;
    if (m/submenu/)  
        $insubmenu=1;
        $prefix = $menuitem . '>';
        $submenuitem = 0;
        $menuindex = $submenuitem;
     ;
    $menuindex = $menuitem++;
   
  # sometimes the default entry saved in grubenv is the menu_id_option
  # rather than numeric.  The %M hash links menu_id_option to menuindex
  # numbers.
  # e.g. 'gnulinux-3.8-1-amd64-advanced-6bb6d228-0581-49ae-9d49-dd148c273ecc' => 4
  $menuidoption='';
  my $line = $_;
  if ($line =~ m/menuentry_id_option/)   
      $line =~ s/.*menuentry_id_option '//;
      $line =~ s/'.*//;
      $menuidoption=$line;
      $M $menuidoption  = $menuindex;
   ;
  s/[^'"]*["']([^'"]*)['"].*/$1/;
  $kernel = $_;
  $K $menuindex  = $kernel;
  #printf "%4s\t%s\t%s\n", $menuindex, $_, $menuidoption;
  printf "%4s\t%s\t\n", $menuindex, $_;
  ;
# now get the default kernel and boot-once kernel if any
my $saved_entry='';
my $prev_saved_entry='';
my $defk = 0;
my $once = 0;
open(GRUBENV, "<$grubenv")   die "couldn't open $grubenv for read: $!\n" ;
while (<GRUBENV>)  
  chomp;
  next unless /saved_entry/;
  if (/^saved_entry/)  
    (undef,$saved_entry) = split /=/;
    elsif (/^prev_saved_entry/)  
    (undef,$prev_saved_entry) = split /=/;
   
 
close(GRUBENV);
$saved_entry = 0 if ($saved_entry eq '');
if ($prev_saved_entry ne '')  
  $defk = $prev_saved_entry;
  $once = $saved_entry;
  else  
  $defk = $saved_entry;
 
# if $defk is non-numeric, look up index in %M
if ($defk !~ /^[0-9]+$/)  
  $defk = $M $defk ;
 
# if $once is non-numeric, look up index in %M
if ($once !~ /^[0-9]+$/)  
  $once = $M $once ;
 
print "\n  Default: $defk";
print "  \"", $K $defk , "\"\n";
if ($prev_saved_entry ne '')  
  print "Boot Once: $once";
  print "  \"", $K $once , "\"\n";
 

grub-list-kernels.pl is a post from: Errata

The French version is available on linuxembedded.fr : Cr e un CD d installation d une debian sp cialis e The goal is to build a debian install CD suitable for the distribution of a complete system including the operating system and applications. Debian already has a tool for that purpose: simple-cdd. Simple-cdd is a set of scripts wrapping debian-cd which is the tool used to build official CDs. In our case, we will include some non-free packages (firmwares for instance) and application specific packages in the system. Using simple-cdd Simple-cdd gets some of its configuration from your host computer, so it s recommended to work on a machine similar to your target (in particular the machine should use the same architecture: i386 or amd64). All the following actions should be done in a working directory that will contain the downloads and configurations for our system. We will start by creting a configuration file my-cdd.conf stating the mirror and components to use:

debian_mirror="http://ftp2.fr.debian.org/debian/"
mirror_components="main contrib non-free"

Then we will launch the build-simple-cdd tool once. This will build a local mirror of the needed packages that we will reused later. So be patient this can be a bit long.

build-simple-cdd --conf ./my-cdd.conf

Once done, you should get a CD iso in the images subfolder. Customizing the image To customize our image we will create a profile my-profile that will define which package to install and which extra package to include on the CD (dev packages for instance). Create a profiles folder:

mkdir profiles

Selecting the packages If you ve got an already installed machine (the one you used to test your application) you can ask dpkg to list the installed packages from that machine and use that as a basis.

dpkg --get-selections >package-list

In this file only the names of packages in the install state are needed:

grep -e '\<install$' package-list   awk ' print $1 ;' >profiles/my-profile.downloads

The file my-profile.downloads is a list of packages to include on the CD-ROM. The list of program to install is my-profile.packages . The tool will resolve dependencies, so we don t need to track down all dependencies add add them to these files. Specific packages We want to add some custom packages that we built ourselves to the CD, those packages not present in the debian archive. Let s create a local_pkg folder and fill it with all of our packages. We now can update the configuration to use all those files and make the profile my-profile the default profile.

debian_mirror="http://ftp2.fr.debian.org/debian/"
mirror_components="main contrib non-free"
simple_cdd_dir=$(pwd)
profiles="my-profile"
auto_profiles="my-profile"
local_packages="$simple_cdd_dir/local_pkg/"

If you now run the tool again, you ll get a full custom install CD!

build-simple-cdd --conf ./my-cdd.conf

However you might have missed some specific packages or their dependencies, in that case they will be missing on the CD. To find this out, check the messages for a line like:

WARNING: missing optional packages from profile my-profile: libevent-2.0-5 [...]

In that case libevent from squeeze-backports was missing. Simple-cdd is not able to download packages from backports, so I just added the packages to local_pkg . Customizing the installer Simple-cdd automatically uses a default profile. This profile might not suite your needs. To override it you just have to place modified file in the profiles folder. To do so, first copy the desired file from /usr/share/simple-cdd/profiles/ and update it. For instance the default.pressed file:

cp /usr/share/simple-cdd/profiles/default.pressed profiles

Then update the wanted options in this file. For example the partitioning options. Also as we enabled contrib and non-free you should uncomment the following lines:

d-i apt-setup/non-free  boolean true
d-i apt-setup/contrib boolean true

Adding extra files to the CD Finally we want to add the documentation, sources and other files to the CD. Simple-cdd can add files to the simple-cdd folder but there is no way to add a complete directory structure. However, as we can override the profiles, we can also override some scripts, especially tools/build/debian-cd that builds the CD.

mkdir -p tools/build
cp /usr/share/simple-cdd/tools/build/debian-cd tools/build

Then add the following lines after the $extras_base_dir cleanup:

if [ -d "$cd_extras" ]; then
  mkdir -p "$extras_base_dir"
  cp -a "$cd_extras/." "$extras_base_dir"
fi

Finally let s add the cd_extras variable to our configuration file :

cd_extras="$simple_cdd_dir/local_extras/"

Be careful to not create files conflicting with the CD content, nothing would warn you about that! Conclusion We now have a way of building a debian install CD including any files we might want to deliver with our system: sources and documentation for example.

Recently, there's been discussions on IRC and the debian-devel mailing list about how to notify users, typically from a cron script or a system daemon needing to tell the user their hard drive is about to expire. The current way is generally "send email to root" and for some bits "pop up a notification bubble, hoping the user will see it". Emailing me means I get far too many notifications. They're often not actionable (apt-get update failed two days ago) and they're not aggregated. I think we need a system that at its core has level and edge triggers and some way of doing flap detection. Level interrupts means "tell me if a disk is full right now". Edge means "tell me if the checksums have changed, even if they now look ok". Flap detection means "tell me if the nightly apt-get update fails more often than once a week". It would be useful if it could extrapolate some notifications too, so it could tell me "your disk is going to be full in $period unless you add more space". The system needs to be able to take in input in a variety of formats: syslog, unstructured output from cron scripts (including their exit codes), snmp, nagios notifications, sockets and fifos and so on. Based on those inputs and any correlations it can pull out of it, it should try to reason about what's happening on the system. If the conclusion there is "something is broken", it should see if it's something that it can reasonably fix by itself. If so, fix it and record it (so it can be used for notification if appropriate: I want to be told if you restart apache every two minutes). If it can't fix it, notify the admin. It should also group similar messages so a single important message doesn't drown in a million unimportant ones. Ideally, this should be cross-host aggregation. The notifications should be possible to escalate if they're not handled within some time period. I'm not aware of such a tool. Maybe one could be rigged together by careful application of logstash, nagios, munin/ganglia/something and sentry. If anybody knows of such a tool, let me know, or if you're working on one, also please let me know.

Sorry Bryan,
I can show you plenty of hardware that is perfectly 64 bit capable but probably never will run Ubuntu and/or Unity. First, what is 64 bit for you? Looking at ubuntu.com/download and getting images from there, one gets the impression, that 64 bit is amd64 (also called x86_64). If one digs deeper to cdimage.ubuntu.com, one will find non-Intel images too: PowerPC and amrhf. As the PowerPC images are said to boot on G3 and G4 PowerPCs, these are 32 bit. Armhf is 32 bit too (arm64/aarch64 support in Linux is just evolving). So yes, if 64 bit means amd64, I do have hardware that can run Unity. But you asked if I have hardware that is 64 bit capable and can run Ubuntu/Unity, so may I apply my definiton of 64 bit here? I have an old Sun Netra T1-200 (500MHz UltraSPARC IIe) running Debian s sparc port, which has a 64 bit kernel and 32 bit userland. Unity? No wai. I do not own any ia64 or s390/s390x machines, but I am sure people do. And guess what, no Unity there either :) Sorry for ranting like this, but 64 bit really just means that the CPU can handle 64 bit big addresses etc. End even then, it not always will do so ;)

0 comment(s) this blog is flattr enabled

I found the following news item; if you can read Spanish, you will most probably prefer the original version in the Proceso magazine's site. The subject? The federal police (PGR) and army arrest 17 artisans for making money out of Spiderman. The following translation is mine. Done past midnight, and being quite tired, and translated so this news item can reach a broader audience. All errors are mine (except those carried out by the security forces, that is).

June 13, 2013 Cuernavaca, Morelos. Policement from the General Republic Attorney (Procuradur a General de la Rep blica, PGR) and the Army entered and searched the "3 de mayo" neighbourhood, in the municipality of Emiliano Zapata, detaining 17 ceramist artisans that sold candies, dolls and pi atas shaped like Spiderman. This search was done on the evening of last Wednesday, around 16:00. Federal ministerial policement and army soldiers closed a street with several informal stores and detained workers taht were selling this Marvel Comics character, following said company's denounce. As a result for this operation, 17 artisants were detained, although the same day five of them were freed. The policemen also seized 12 bags of candies, pi atas, ceramics and wooden figures of the superhero. PGR closed down 11 stores where ceramics with this same figure was being sold, accusing the detainees of plagiarizing Spiderman's image, protected under the copyright law. The 12 that remained under detention were put at the Federal Justice's disposal, which prompted that this Thursday, around 10AM, hundreds of sellers of "3 de mayo" went out to PGR's building to demand their friends' freedom, who are facing a bail of up to 200,000 pesos (~USD$18,000). Outraged because they said they were treated as if they were part of a drug ring, hundreds of artisans closed intermitently Avenida Cuauhn huac, where the PGR representation in Morelos state is located. The artisans' pressure helped for the amount of the bail to be lowered from MX$200,000 to MX$16,000, and so they were set free. Francisco Fern ndez Flores, president of the Ceramists Association, criticized the operation because, he said, it was as strong as if they were "drug dealers". The artisans explained that they don't even make the Spiderman figures, they are made by the interns of the Centro Estatal de Reinserci n Social de Atlacholoaya (prision), located in the Xochitepec municipality, who offered them to the ceramists so they could be sold. "The Atlacholoaya inmates do them, we buy them to support them, and turns out we are the delinquents now", said Miriam Monroy, sister of one of the detainees. This information was contradicted by Jes s Valencia Valencia, responsible for Morelos' state prision system, who assured that in said prision no ceramics are done. Fern ndez Flores insisted though that from within the prision they are being offered pi atas, candies and "piggy banks" with Spiderman's shape. Jos Luis Pozo, vicepresident of the Ceramists Union, said that to avoid more such federal operations for copyright breaches, they have committed not to produce or commercialize Marvel superhero figures, and any other characters the authority demands. "We do commit to, from now on, those products singled out to us will not be commercialized", he said. Pozo said that the PGR operation caused losses not just to the detained producers and salesmen, but to over 200 ceramists that had to close their stores in solidarity with their friends. Acording to the artisans, the products were a success until the PGR came, seized the products and detained the salesmen.

And yes, the copyright insanity does not stop. Spiderman is by today a clear part of popular culture. Marvel brilliantly succeeded in creating such a popular icon that everybody recognizes, that everybody identifies with And that everybody should be able to recreate. We are not talking about brand protection. Marvel does not, and will never, commercialize pi atas, ceramics or wooden toys. And even if they were plastic-cast While Spiderman is still under the protection of copyright, as the Berne Convention defines it (and of course, as the much stricter Mexican laws agree), that does not mean that any and every product resembling a Spiderman should be protected. Many ceramists and pi ata makers will create unique pieces of art Ok, handicraft. But reading the copyright law more strictly, Spiderman is more treated as a trademark than as a copyright. And it is a trademark that should be declared as having passed on to the public domain.

If all goes well, a new release of dput-ng will hit the archive this week (depending on when I can find time to close the last 2 bigger bugs) - anyone interested in getting a new feature into dput-ng should email me (or file a reportbug bug at dput-ng) Small list of changes so far: * Add per-host loading of command files by partially parsing command line arguments. * Add disallowed_distributions. * Add codename groupping support. * Add codename groups to vital targets, limit security uploads to the right servers (Closes: #708575). * Implement dcut commands for Deb-o-Matic. * Add an override and a unset option to dput which overrules any profile key. This is an experimental option. * Be more precise about our configuration file inheritance in dput(5). * Fix dcut: manpage and help talk about -U and upload by replacing those parts in the text by the upload command (Closes: #699812) * Make the login name determination more portable (Closes: #709831) * Fix Allow to give out dm permissions without using local keyring by * documenting the force option in the man page (Closes: #711057) * Allow uploads to mentors to target every distribution * Add a space in debug s help message. * Fixed a typo (DEBNAME DEBFULLNAME) in the dcut(1) manpage. Thanks, y all!

... backups? did someone talk about backups? I'm sure I heard someone mention backups here somewhere. Backups! BACKUPS! BACKUPS ARE AWESOME!

That's a direct quote from my recent IRC history. I find backups quite interesting, particularly from an implementation point of view, and I may sometimes obsess about them a little bit. This is why I've written my own backup software. I'm unusual: most people find backups boring at best, and tedious most of the time. When I talk with people about backups, the usual reaction is "um, I know I should". There are a lot of reasons for this. One is that backups are a lot like insurance: you have to spend time, effort, money, up front, to have any use for them. Another is that the whole topic is scary: you have to think about when things go wrong, and that puts people off. A third reason is that while there are lots of backup tools and methods, it's not always easy. After all, backups are about answering the question, "what can I do to keep my data safe, whatever happens in the future?". I've spent a fair bit of the past several years thinking about backups. This is the first in a series of blog posts about backups, where I share my thinking about the topic. Perhaps it can be of some use to others. At least people will poke holes in my delusions. In this post, I'll define some terminology. I am not a backup scholar, and I may have invented some of these terms myself, and they may be different from what real sysadmins use. I'll define the words I use, so we can understand each other. Live data is the data you work with or keep. It's the files on your hard drive: the documents you write, the photos you save, the unfinished novels you wish you'd finish. A backup is a spare copy of your live data. If you lose some or all of your live data, you can get it back ("restore") from your backup. The backup copy is, by practical necessity, older than your live data, but if you made the backup recently enough, you won't lose much. Sometimes it's useful to have more than one old backup copy of your live data. You can have a sequence of backups, made at different times, giving you a backup history. Each copy of your live data in your backup history is a generation. This lets you retrieve a file you deleted a long time ago, but didn't realise you needed until now. If you only keep one backup version, you can't get it back, but if you keep, say, a daily backup for a month, you have a month to realise you need it, before it's lost forever. The place your backups are stored is the backup repository. You can use many kinds of backup media for backup storage: hard drives, tapes, optical disks (DVD-R, DVD-RW, etc), USB flash drives, online storage, etc. Each type of medium has different characteristics: size, speed, convenicence, reliability, price, which you'll need to balance for a backup solution that's reasonable for you. You may need multiple backup repositories or media, with one of them located off-site, away from where your computers normally live. Otherwise, if you house burns down, you'll lose all your backups too. You need to verify that your backups work. It would be awkward to go to the effort and expense of making backups and then not be able to restore your data when you need to. You may even want to test your disaster recovery by pretending that all your computer stuff is gone, except for the backup media. Can you still recover? You'll want to do this periodically, to make sure your backup system keeps working. Most live data is precious in that you'll be upset if you lose it. Some live data is not precious: your web browser cache probably isn't, for example. This distinction can let you limit the amount of data you need to back up, which can significantly reduce your backup costs. There is a very large variety of backup tools. They can be very simple and manual: you can copy files to a USB drive using your file manager, once a blue moon. They can also be very complex: enterprise backup products that cost huge amounts of money and come with a multi-day training package for your sysadmin team, and which require that team to function properly. You'll need to define a backup strategy to tie everything together: what live data to back up, to what medium, using what tools, what kind of backup history to keep, and how to verify that they work. That's the groundwork. In the next episode, I'll blather about what is a backup, and what isn't.

Another maintenance release of inline is now on CRAN and in already included in Debian. This release was triggered by a change in the development version of R which removed an argument to package.skeleton(). The complete NEWS entry is below.

Changes in inline version 0.3.12 (2013-06-12)

• Align package.skeleton with the R-devel version of the function which no longer has a NAMESPACE argument.
• Remove copy of LGPL in source archive as R CMD check now complains about it. License status continues to be specified in file DESCRIPTION.

Courtesy of CRANberries, there is also a diffstat report for the most recent release. A few more details are available at the R-Forge page.

Hello Config::Model::Dpkg project (a Debian source package model based on Config::Model) is partly based on a ParseRec::Descent grammar. This grammar is used to parse the dependency of a Debian source package. This article will show how such a grammar is written, its limitation regarding error handling and how to improve the situation. Debian package main data is described in debian/control file. This file can feature a list of dependencies, i.e. a list of package that must be installed for the package to work. These dependencies are declared in fields like Build-Depends , or Depends as a list of package. For Dpkg model purpose, I needed only to parse one item of a dependency list at a time. This dependency item can be a simple package name: foo or a package name with a version requirement: foo ( > 1.24 ) or a package name with architectures restrictions: foo [alpha amd64 hurd-arm linux-armeb] or both: foo ( > 1.24 ) [alpha amd64 hurd-arm linux-armeb] or a list of alternate choices combining the possibilities above: foo ( > 1.24 ) bar [ linux-any] baz ( << 3.14 ) [ ! hurd-armel !hurd-armeb ] or a variable that is replaced during package build: $ perl-depends Writing a Parse::RecDescent grammar to parse this is relatively straightforward. The first production handles alternate dependencies separated by and raises an error if some text was not consumed by the dependencies:

dependency_item: depend(s /\ /) eofile  

A dependency as explained above is expressed as:

depend: pkg_dep   variable

A variable like $ foo or $ bar -1.24~ is parsed with:

variable: /\$ [\w:\-]+ [\w\.\-~+]*/

This rule handles a package name with optional version or arch restriction:

pkg_dep: pkg_name dep_version(?) arch_restriction(?) 
pkg_name: /[a-z0-9][a-z0-9\+\-\.]+/

The remaining rules are quite simple:

dep_version: '(' oper version ')' 
oper: '<<'   '<='   '='   '>='   '>>'
version: variable   /[\w\.\-~:+]+/
arch_restriction: '[' arch(s) ']'
arch:  /!?[\w-]+/
eofile: /^\Z/

The grammar above works well to parse the dependency. You can test it with this small Perl script:

#!/usr/bin/perl
use strict;
use warnings;
use 5.010 ;
use Parse::RecDescent ;
my $parser = Parse::RecDescent->new(join('',));
my $dep = shift ;
say "parsing '$dep'";
my $ret = $parser->dependency_item($dep) ;
say "result is ", $ret if $ref ;
__DATA__
# insert grammar here !!!

Unfortunately, any error in the optional parts (i.e version requirements and arch restriction) leads to an error message which is not very helpful. The error message only mention that some text could not be parsed:

parsing 'foo ( != 1.24 )   bar'
       ERROR (line 1): Invalid dependency item: Was expecting /\ / but found
                       "( != 1.24 )   bar" instead

or

parsing 'foo [ arm & armel]   bar'
       ERROR (line 1): Invalid dependency item: Was expecting /\ / but found
                       "[ arm & armel]   bar" instead

The problem comes from the fact that version requirements or arch restrictions are optional. For instance if a version requirement has a syntax error, Parse::RecDescent will try to parse it as an arch restriction. This arch restriction rule will also fail and the last terminal ( eofile ) will fail. So the error message does not hint at the actual syntax problem. To generate better error messages, I improved the suggestion made in Parse::RecDescent FAQ. Instead of calling a plain subroutine, I use a sub reference that will store the error messages in a closure. This sub ref is declared in a start-up action. Note that the sub ref explicitly returns undef. I ll explain why later.

 
    my @dep_errors ;
    my $add_error = sub  
        my ($err, $txt) = @_ ;
        push @dep_errors, "$err: '$txt'" ;
        return ;
      ;
 

The following production always fails while ensuring that the error list is reset. This production is always run at the beginning of the dependency parsing:

dependency:   @dep_errors = ();  

Here s the actual dependency production that is run when dependency method is called on the parser. It will return an array ref containing (1, data) if the dependency is valid or (0, errors) otherwise:

dependency: depend(s /\ /) eofile
   
    $return = [ 1 , @ $item[1]  ] ;
   
   
   
    push( @dep_errors, "Cannot parse: '$text'" ) unless @dep_errors ;
    $return =  [ 0, @dep_errors ];
   

The following productions don t change much:

depend: pkg_dep   variable
variable: /\$ [\w:\-]+ [\w\.\-~+]*/
pkg_dep: pkg_name dep_version(?) arch_restriction(?) 
dep_version: '(' oper version ')'

The first rule of this production parses the package name which must be followed by a space, end of string ( or ['. A positive look-ahead assertion is used so only the package name is consumed. If the first rule fails, the second rule provides a meaningful error message. The second rule will match anything which is not a space and create an error message. Since $add_error returns undef, the second rule returns undef and the production fails. So the text stored in the error message is not consumed:

pkg_name: /[a-z0-9][a-z0-9\+\-\.]+(?=\s \Z \( \[)/
      /\S+/   $add_error->("bad package name", $item[1]) ; 

The same trick is used with these productions:

oper: '<<'   '='   '>>'
      /\S+/   $add_error->("bad dependency version operator", $item[1]) ; 
version: variable   /[\w\.\-~:+]+(?=\s \) \Z)/
      /\S+/   $add_error->("bad dependency version", $item[1]) ; 

The action of this production is a little bit more tricky. The action ensures that '!' are either added before all arch or not at all. Otherwise an error message is generated and added to the list of errors:

arch_restriction: '[' osarch(s) ']'
     
        my $mismatch = 0;
        # $ref contains ['!',os,arch] or ['',os,arch]
        my $ref = $item[2] ;
        for (my $i = 0; $i < $#$ref -1 ; $i++ )  
            $mismatch  = ($ref->[$i][0] xor $ref->[$i+1][0]) ;
         
        my @a = map   ($_->[0]   '') . ($_->[1]   '') . $_->[2]   @$ref ;
        if ($mismatch)  
            $add_error->("some names are prepended with '!' while others aren't.", "@a") ;
         
        else  
            $return = 1 ;
         
     

The check above is possible only if the "osarch" production returns an array ref containing something like ('!','linux','any') for "!linux-any' or ('','linux','any') for "linux-any":

osarch: not(?) os(?) arch
     
        $return =  [ $item[1][0], $item[2][0], $item[3] ];
     
      /.?(?=\s \] \Z)/   $add_error->("bad arch specification: ", $item[1]) ;  
not: '!'

Here's the remaining of the grammar:

os: /(any uclibc-linux linux kfreebsd knetbsd etc...)-/
     /\w+/ '-'   $add_error->("bad os in architecture specification", $item[1]) ; 
arch: / (any  alpha amd64  arm\b  arm64  etc... )
        (?=(\]  ))
      /x
        /\w+/   $add_error->("bad arch in architecture specification", $item[1]) ; 
eofile: /^\Z/

That's all for grammar 2.0... Before someone yells: "Show me the message ! ", here are some example of bad dependencies and their error message generated by the parser:

parse 'foo ( != 1.24 )   bar'
result is: 0 bad dependency version operator: '!='
parsing 'foo [ arm & armel]   bar'
result is: 0 bad arch specification: : '&'
parsing 'foo [ arm armel ]   bar [!moo]'
result is: 0 bad arch specification: : ']' bad arch in architecture specification: 'moo'

The 2 first error messages are spot on the actual error. The second one has a false positive (']' is correct) but correctly highlights the wrong arch name ('moo'). Mission accomplished. In order to keep this post (relatively) simple, I've removed the part that actually store parsed data. They don't really matter for error handling. Nevertheless, you may see the whole grammar in Config::Model::Dpkg::Dependency module. All the best
Tagged: Config::Model, debian, dpkg, package, Parse::RecDescent, Perl

Migrating a Debian installation between architectures has always been difficult. The recommended way to crossgrade an i386 Debian to amd64 Debian was to reinstall the system, move over data and configuration. For the more brave, in-place crossgrades usually involved chroots, rescue CDs, a lot of ar p tar xf - data.tar.gz and luck. I have never been brave when it comes to system administration, have done a lot of architecture migrations with reinstallation, and have always taken the opportunity to clear out the contamination that accumulates itself when a system is running for a long time. I would even recommend doing this to most people even now. However, I have a few very ugly systems in place that are still on i386 because I didn t dare going the reinstallation path. Doing in-place crossgrades has become a lot easier since wheezy s release, since once now can have both i386 and amd64 libraries installed in parallel, which allows to replace foo:i386 with foo:amd64 without influencing the other parts of the system. The process is still full of pitfalls:

• apt removes a package before it reinstalls its new counterpart. This results in apt calling dpkg to remove dpkg, and then calling dpkg again to install dpkg. Guess which operation fails and the state of the system after this failure. Same applies to coreutils, which leaves the system without rm, which in turn dpkg of either architecture doesn t like. Using apt-get --download-only install to resolve dependencies and downloading the debs, followed by a traditional dpkg --install solves this issue since multiarch dpkg will replace a package with another one without deinstalling the first one first.
• At least for the process, you need a kernel that can run both 32bit and 64bit binaries for the i386 architecture. AFAIR, setting CONFIG_64BIT, CONFIG_X86_64 and CONFIG_IA32_EMULATION in the kernel configuration takes care of this.
• During the process, apt wants to remove the better part of your system. It is important to not let it do this, as it wants to deinstall essential packages as well.
• It looks like the process is not always exactly reproducible. During the first tries, I found myself without an initrd at all, with an initrd that lacked the ext[234].ko kernel modules, without working e2fstools and in a number of other undesireable states of the system.

I have only tried this yet with a freshly installed minimal wheezy server system. I do not, however, expect surprises when it comes to using this process with real life systems. I will document other pitfalls I have fallen into here at a later time. My minimal wheezy system was running in a KVM VM with its virtual disk as a LVM LV in the host system. I took a snapshot before beginning and used lvconvert --merge numerous time to return my LV to the original state. Be aware that lvconvert --merge removes the snapshot after merging it, so you ll need to re-create the snapshot before trying again. During the process, I discussed things with Paul Tagliamonte, who has done this before, but on a live system and with a slightly more invasive approach. He has blogged about this. Thank you very much, your hints were very helpful. Here is a commented typescript of what I have done. Be warned: This was a lab setting with a rather minimal system. If you re going to try this with a production system, have a backup, or, if you re on snapshottable infrastructure, take snapshots and be prepared to roll back if anything goes wrong. First let s see what we have and save a list of installed packages.

mh@swivel:~$ ssh wheezyarch.zugschlus.de
$ uname -a
Linux wheezyarch 3.9.4-zgsrv2008064 #2 SMP PREEMPT Wed Jun 5 12:57:51 UTC 2013 x86_64 GNU/Linux
$ dpkg --print-architecture
i386
$ dpkg --print-foreign-architectures
$ dpkg --list   grep i386   wc -l
175
$ dpkg --list   grep amd64   wc -l
0
$ sudo dpkg --get-selections   grep install   awk  print $1    sed  s/:.*//;s/$/:amd64
install/;  > get-selections.pre

So we have an i386 system running an x86_64 kernel. Next, we add amd64 as an additional architecture which allows us to install amd64 packages. We download the amd64 packages files. Note that dpkg still considers this an i386 system with amd64 as a foreign arch.

$ sudo dpkg --add-architecture amd64
$ sudo apt-get update
Get:1 http://security.debian.org wheezy/updates Release.gpg [836 B]
( )
Fetched 6.155 kB in 4s (1.400 kB/s)
Reading package lists... Done
$ dpkg --print-architecture
i386
$ dpkg --print-foreign-architectures
amd64
$ dpkg --list   grep i386   wc -l
175
$ dpkg --list   grep amd64   wc -l
0

Next, download everything that is necessary to migrate dpkg and apt from i386 and amd64. We cannot directly apt things as the remove-install cycle used by apt will leave us without working dpkg. One could work around this by un-aring the dpkg.deb and directly forcing the amd64 files upon the system, but I think this approach is much cleaner. apt-get download will not resolve dependices while apt-get --download install does. After the download, dpkg --install the packages. dpkg is not particularly smart about installation order, so Pre-Depends may fail or not. If they fail, dpkg will give a list of deb files that were affected, which is an easy help to finish their installation. One could also choose the easy way (which I did here) and repeat the dpkg --install *.deb command. Since the Pre-Depends will already be installed, the second install run will succeed.

$ sudo apt-get --download-only install dpkg:amd64 apt:amd64
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  gcc-4.7-base:amd64 libapt-pkg4.12:amd64 libbz2-1.0:amd64 libc6:amd64
  libgcc1:amd64 liblzma5:amd64 libselinux1:amd64 libstdc++6:amd64
  zlib1g:amd64
Suggested packages:
  aptitude:amd64 synaptic:amd64 wajig:amd64 dpkg-dev:amd64 apt-doc:amd64
  python-apt:amd64 glibc-doc:amd64 locales:amd64
The following packages will be REMOVED:
  apt cron-apt dpkg
The following NEW packages will be installed:
  apt:amd64 dpkg:amd64 gcc-4.7-base:amd64 libapt-pkg4.12:amd64
  libbz2-1.0:amd64 libc6:amd64 libgcc1:amd64 liblzma5:amd64 libselinux1:amd64
  libstdc++6:amd64 zlib1g:amd64
0 upgraded, 11 newly installed, 3 to remove and 0 not upgraded.
Need to get 10,0 MB of archives.
After this operation, 14,9 MB of additional disk space will be used.
Do you want to continue [Y/n]? 
Get:1 http://debian.debian.zugschlus.de/debian/ wheezy/main gcc-4.7-base amd64 4.7.2-5 [144 kB]
( )
Fetched 10,0 MB in 2s (3.535 kB/s)
Download complete and in download only mode
$ sudo dpkg --install /var/cache/apt/archives/*.deb
(Reading database ... 16586 files and directories currently installed.)
Preparing to replace apt 0.9.7.8 (using .../archives/apt_0.9.7.8_amd64.deb) ...
Unpacking replacement apt ...
dpkg: regarding .../dpkg_1.16.10_amd64.deb containing dpkg, pre-dependency problem:
 dpkg pre-depends on libbz2-1.0
dpkg: error processing /var/cache/apt/archives/dpkg_1.16.10_amd64.deb (--install):
 pre-dependency problem - not installing dpkg
Selecting previously unselected package gcc-4.7-base:amd64.
( )
Errors were encountered while processing:
 /var/cache/apt/archives/dpkg_1.16.10_amd64.deb
$ sudo dpkg --install /var/cache/apt/archives/*.deb
(Reading database ... 16905 files and directories currently installed.)
Preparing to replace apt 0.9.7.8 (using .../archives/apt_0.9.7.8_amd64.deb) ...
Unpacking replacement apt ...
( )
$

From dpkg s point of view, the system is now already amd64, with i386 being a foreign arch. However, the majory of packages is still i386.

$ dpkg --print-architecture
amd64
$ dpkg --print-foreign-architectures
i386
$ dpkg --list   grep i386   wc -l
173
$ dpkg --list   grep amd64   wc -l
11
$ 

The apt resolver is currently in a very bad broken state since the system is missing most essential packages in its native architecture. This is not easily solved by apt-get -f install, as this would zap most of the existing system . We let apt-get --download-only -f install download the packages that it wants to replace with its amd64 counterparts and their dependencies, and, again, use dpkg --install *.deb to install them.

$ sudo apt-get --download-only -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  cpio gettext-base:i386 klibc-utils libapt-inst1.5 libasprintf0c2:i386
  libcurl3-gnutls:i386 libfreetype6:i386 libfuse2:i386 libklibc
  libreadline5:i386
Use  apt-get autoremove  to remove them.
The following extra packages will be installed:
  acpid aide bsd-mailx cpio cron curl daemon dctrl-tools exim4-base
  exim4-daemon-light initscripts insserv kbd klibc-utils less libbsd0
  libcomerr2 libcurl3 libdb5.1 libexpat1 libgcrypt11 libgdbm3 libgnutls26
  libgpg-error0 libgssapi-krb5-2 libidn11 libk5crypto3 libkeyutils1 libklibc
  libkmod2 libkrb5-3 libkrb5support0 libldap-2.4-2 liblockfile1 libncursesw5
  libnewt0.52 libp11-kit0 libpam-modules libpam0g libpcre3 libpopt0
  libreadline6 librtmp0 libsasl2-2 libslang2 libsqlite3-0 libssh2-1
  libssl1.0.0 libtasn1-3 libterm-readkey-perl libtinfo5 openssl perl
  perl-base python2.7 python2.7-minimal sysvinit-utils whiptail
Suggested packages:
  libarchive1 anacron logrotate checksecurity debtags mail-reader eximon4
  exim4-doc-html exim4-doc-info spf-tools-perl swaks bootchart2 rng-tools
  krb5-doc krb5-user libpam-doc perl-doc make python2.7-doc binutils
  binfmt-support bootlogd sash
Recommended packages:
  exim4 postfix mail-transport-agent psmisc mailx ca-certificates
  krb5-locales libgpm2 libfribidi0 libsasl2-modules libpng12-0
The following packages will be REMOVED:
  acpid:i386 aide:i386 aptitude:i386 bsd-mailx:i386 cpio:i386 cron:i386
  curl:i386 daemon:i386 dctrl-tools:i386 debsecan dmsetup:i386 e2fsprogs:i386
  exim4-base:i386 exim4-daemon-light:i386 git:i386 grub-common:i386
  grub-pc:i386 grub-pc-bin:i386 grub2-common:i386 ifupdown:i386
  initramfs-tools initscripts:i386 insserv:i386 ippl:i386 jed:i386 kbd:i386
  klibc-utils:i386 less:i386 libdevmapper-event1.02.1:i386
  libdevmapper1.02.1:i386 libklibc:i386 libnewt0.52:i386
  libparted0debian1:i386 libterm-readkey-perl:i386 lsof:i386 lvm2:i386
  molly-guard ntp:i386 openssh-server:i386 openssl:i386 parted:i386 perl:i386
  perl-base:i386 procps:i386 python-apt:i386 python2.7:i386
  python2.7-minimal:i386 rsyslog:i386 sysvinit:i386 sysvinit-utils:i386
  udev:i386 util-linux:i386 whiptail:i386
The following NEW packages will be installed:
  acpid aide bsd-mailx cpio cron curl daemon dctrl-tools exim4-base
  exim4-daemon-light initscripts insserv kbd klibc-utils less libbsd0
  libcomerr2 libcurl3 libdb5.1 libexpat1 libgcrypt11 libgdbm3 libgnutls26
  libgpg-error0 libgssapi-krb5-2 libidn11 libk5crypto3 libkeyutils1 libklibc
  libkmod2 libkrb5-3 libkrb5support0 libldap-2.4-2 liblockfile1 libncursesw5
  libnewt0.52 libp11-kit0 libpam-modules libpam0g libpcre3 libpopt0
  libreadline6 librtmp0 libsasl2-2 libslang2 libsqlite3-0 libssh2-1
  libssl1.0.0 libtasn1-3 libterm-readkey-perl libtinfo5 openssl perl
  perl-base python2.7 python2.7-minimal sysvinit-utils whiptail
0 upgraded, 58 newly installed, 53 to remove and 0 not upgraded.
Need to get 23,4 MB of archives.
After this operation, 17,5 MB disk space will be freed.
Do you want to continue [Y/n]? Y
Get:1 http://security.debian.org/debian-security/ wheezy/updates/main libgnutls26 amd64 2.12.20-7 [619 kB]
( )
Fetched 23,4 MB in 11s (2.038 kB/s)                                           
Download complete and in download only mode
$ sudo dpkg --install /var/cache/apt/archives/*.deb
(Reading database ... 16905 files and directories currently installed.)
Preparing to replace acpid 1:2.0.16-1+deb7u1 (using .../acpid_1%3a2.0.16-1+deb7u1_amd64.deb) ...
( )
Selecting previously unselected package libpam-modules:amd64.
dpkg: regarding .../libpam-modules_1.1.3-7.1_amd64.deb containing libpam-modules:amd64, pre-dependency problem:
 libpam-modules pre-depends on libdb5.1
  libdb5.1:amd64 is unpacked, but has never been configured.
dpkg: error processing /var/cache/apt/archives/libpam-modules_1.1.3-7.1_amd64.deb (--install):
 pre-dependency problem - not installing libpam-modules:amd64
Selecting previously unselected package libpcre3:amd64.
Unpacking libpcre3:amd64 (from .../libpcre3_1%3a8.30-5_amd64.deb) ...
( )
Errors were encountered while processing:
 /var/cache/apt/archives/libpam-modules_1.1.3-7.1_amd64.deb
$ sudo dpkg --install /var/cache/apt/archives/libpam-modules_1.1.3-7.1_amd64.deb
(Reading database ... 17007 files and directories currently installed.)
Unpacking libpam-modules:amd64 (from .../libpam-modules_1.1.3-7.1_amd64.deb) ...
Setting up libpam-modules:amd64 (1.1.3-7.1) ...
$ sudo apt-get clean
$ dpkg --print-architecture
amd64
$ dpkg --print-foreign-architectures
i386
$ dpkg --list   grep i386   wc -l
151
$ dpkg --list   grep amd64   wc -l
66
$ 

The concluding apt-get clean prevents packages from being reinstalled again in the next dpkg --install *.deb run. Now, the apt resolver is kind of fixed, but apt will still want to remove most of the system. I haven t found a way to get out of this mess short of allowing apt to de-install those packages and to manually install their amd64 counterparts later. First, we see what apt wants to do, and paste the The following packages will be removed part of apt-get s output into a sed expression which fixes the architecture strings for us so that we can use the resulting file as input for the following installation procedure. After that, we let apt-get do what itself considers a bad thing and requires us to type a consent sentence. The key is that this deinstallation of essential packages breaks the system, but not apt and dpkg, which allows us to un-break the system afterwards.

$ sudo apt-get -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  gettext-base:i386 libapt-inst1.5 libasprintf0c2:i386 libcurl3-gnutls:i386
  libfreetype6:i386 libfuse2:i386 libreadline5:i386
Use  apt-get autoremove  to remove them.
The following extra packages will be installed:
  sysvinit-utils:i386
Suggested packages:
  bootlogd:i386 sash:i386
The following packages will be REMOVED:
  adduser aide-common aptitude:i386 bsd-mailx console-common console-data
  console-log cron curl debian-goodies debsecan dmsetup:i386 e2fsprogs:i386
  exim4-base exim4-config exim4-daemon-light git:i386 grub-common:i386
  grub-pc:i386 grub-pc-bin:i386 grub2-common:i386 ifupdown:i386
  initramfs-tools initscripts ippl:i386 jed:i386 libcurl3
  libdevmapper-event1.02.1:i386 libdevmapper1.02.1:i386 liberror-perl
  libfile-find-rule-perl libnumber-compare-perl libparted0debian1:i386
  libswitch-perl libterm-readkey-perl libterm-readline-perl-perl
  libtext-glob-perl libtimedate-perl locales lsof:i386 lvm2:i386 molly-guard
  ntp:i386 openssh-client:i386 openssh-server:i386 openssl openssl-blacklist
  openssl-blacklist-extra parted:i386 perl perl-modules procps:i386
  python-apt:i386 rsyslog:i386 stow sysv-rc sysvinit:i386 sysvinit-utils
  tzdata ucf udev:i386 util-linux:i386
The following NEW packages will be installed:
  sysvinit-utils:i386
WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
  e2fsprogs:i386 util-linux:i386 (due to e2fsprogs:i386) sysvinit:i386
  sysvinit-utils tzdata (due to util-linux:i386)
0 upgraded, 1 newly installed, 62 to remove and 0 not upgraded.
Need to get 97,1 kB of archives.
After this operation, 131 MB disk space will be freed.
You are about to do something potentially harmful.
To continue type in the phrase  Yes, do as I say!  ? (abort with Ctrl-C)
$ sed  s/:i386//g  > removedpkg (paste  The following packages will be REMOVED:  stanza here and
type Ctrl-D afterwards)
  adduser aide-common aptitude:i386 bsd-mailx console-common console-data
  console-log cron curl debian-goodies debsecan dmsetup:i386 e2fsprogs:i386
  exim4-base exim4-config exim4-daemon-light git:i386 grub-common:i386
  grub-pc:i386 grub-pc-bin:i386 grub2-common:i386 ifupdown:i386
  initramfs-tools initscripts ippl:i386 jed:i386 libcurl3
  libdevmapper-event1.02.1:i386 libdevmapper1.02.1:i386 liberror-perl
  libfile-find-rule-perl libnumber-compare-perl libparted0debian1:i386
  libswitch-perl libterm-readkey-perl libterm-readline-perl-perl
  libtext-glob-perl libtimedate-perl locales lsof:i386 lvm2:i386 molly-guard
  ntp:i386 openssh-client:i386 openssh-server:i386 openssl openssl-blacklist
  openssl-blacklist-extra parted:i386 perl perl-modules procps:i386
  python-apt:i386 rsyslog:i386 stow sysv-rc sysvinit:i386 sysvinit-utils
  tzdata ucf udev:i386 util-linux:i386
$ cat removedpkg 
  adduser aide-common aptitude bsd-mailx console-common console-data
  console-log cron curl debian-goodies debsecan dmsetup e2fsprogs
  exim4-base exim4-config exim4-daemon-light git grub-common
  grub-pc grub-pc-bin grub2-common ifupdown
  initramfs-tools initscripts ippl jed libcurl3
  libdevmapper-event1.02.1 libdevmapper1.02.1 liberror-perl
  libfile-find-rule-perl libnumber-compare-perl libparted0debian1
  libswitch-perl libterm-readkey-perl libterm-readline-perl-perl
  libtext-glob-perl libtimedate-perl locales lsof lvm2 molly-guard
  ntp openssh-client openssh-server openssl openssl-blacklist
  openssl-blacklist-extra parted perl perl-modules procps
  python-apt rsyslog stow sysv-rc sysvinit sysvinit-utils
  tzdata ucf udev util-linux
$ sudo apt-get -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  gettext-base:i386 libapt-inst1.5 libasprintf0c2:i386 libcurl3-gnutls:i386
  libfreetype6:i386 libfuse2:i386 libreadline5:i386
Use  apt-get autoremove  to remove them.
The following extra packages will be installed:
  sysvinit-utils:i386
Suggested packages:
  bootlogd:i386 sash:i386
The following packages will be REMOVED:
  adduser aide-common aptitude:i386 bsd-mailx console-common console-data
  console-log cron curl debian-goodies debsecan dmsetup:i386 e2fsprogs:i386
  exim4-base exim4-config exim4-daemon-light git:i386 grub-common:i386
  grub-pc:i386 grub-pc-bin:i386 grub2-common:i386 ifupdown:i386
  initramfs-tools initscripts ippl:i386 jed:i386 libcurl3
  libdevmapper-event1.02.1:i386 libdevmapper1.02.1:i386 liberror-perl
  libfile-find-rule-perl libnumber-compare-perl libparted0debian1:i386
  libswitch-perl libterm-readkey-perl libterm-readline-perl-perl
  libtext-glob-perl libtimedate-perl locales lsof:i386 lvm2:i386 molly-guard
  ntp:i386 openssh-client:i386 openssh-server:i386 openssl openssl-blacklist
  openssl-blacklist-extra parted:i386 perl perl-modules procps:i386
  python-apt:i386 rsyslog:i386 stow sysv-rc sysvinit:i386 sysvinit-utils
  tzdata ucf udev:i386 util-linux:i386
The following NEW packages will be installed:
  sysvinit-utils:i386
WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
  e2fsprogs:i386 util-linux:i386 (due to e2fsprogs:i386) sysvinit:i386
  sysvinit-utils tzdata (due to util-linux:i386)
0 upgraded, 1 newly installed, 62 to remove and 0 not upgraded.
Need to get 97,1 kB of archives.
After this operation, 131 MB disk space will be freed.
You are about to do something potentially harmful.
To continue type in the phrase  Yes, do as I say! 
 ?] Yes, do as I say! (actually type this!)
Get:1 http://debian.debian.zugschlus.de/debian/ wheezy/main sysvinit-utils i386 2.88dsf-41 [97,1 kB]
Fetched 97,1 kB in 0s (687 kB/s)    
(Reading database ... 17050 files and directories currently installed.)
Removing aide-common ...
( )
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
Removing e2fsprogs ...
Removing sysv-rc ...
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
Removing sysvinit ...
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
Removing sysvinit-utils ...
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
Removing ucf ...
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
Removing libdevmapper1.02.1:i386 ...
Removing libswitch-perl ...
Removing perl ...
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
Removing dmsetup ...
Removing perl-modules ...
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
Removing util-linux ...
Removing tzdata ...
Processing triggers for mime-support ...
Selecting previously unselected package sysvinit-utils.
(Reading database ... 9802 files and directories currently installed.)
Unpacking sysvinit-utils (from .../sysvinit-utils_2.88dsf-41_i386.deb) ...
Setting up sysvinit-utils (2.88dsf-41) ...
$ sudo apt-get install $(cat removedpkg)
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libasprintf0c2:i386 libcurl3-gnutls:i386 libfreetype6:i386 libfuse2:i386
  libreadline5:i386
Use  apt-get autoremove  to remove them.
The following extra packages will be installed:
  e2fslibs gettext-base libapt-inst1.5 libasprintf0c2 libattr1 libblkid1
  libboost-iostreams1.49.0 libcap2 libcurl3-gnutls libcwidget3 libedit2
  libept1.4.12 libfreetype6 libfuse2 libgpm2 libncurses5 libopts25 libprocps0
  libreadline5 libsepol1 libsigc++-2.0-0c2a libss2 libudev0 libuuid1 libwrap0
  libxapian22 logrotate
Suggested packages:
  liblocale-gettext-perl tasksel debtags unicode-data anacron checksecurity
  popularity-contest xdg-utils zenity gpart e2fsck-static mail-reader eximon4
  exim4-doc-html exim4-doc-info spf-tools-perl swaks git-daemon-run
  git-daemon-sysvinit git-doc git-el git-arch git-cvs git-svn git-email
  git-gui gitk gitweb multiboot-doc grub-emu xorriso desktop-base
  isc-dhcp-client dhcp-client ppp rdnssd net-tools gpm libcwidget-dev fuse
  libparted0-dev libparted0-i18n xapian-tools ntp-doc ssh-askpass libpam-ssh
  keychain monkeysphere rssh ufw parted-doc perl-doc make libpod-plainer-perl
  python-apt-dbg python-gtk2 python-vte python-apt-doc rsyslog-mysql
  rsyslog-pgsql rsyslog-doc rsyslog-gnutls rsyslog-gssapi rsyslog-relp
  doc-base sysv-rc-conf bum bootlogd sash util-linux-locales dosfstools
Recommended packages:
  aptitude-doc-en aptitude-doc apt-xapian-index libparse-debianchangelog-perl
  exim4 postfix mail-transport-agent psmisc mailx patch rsync ssh-client
  os-prober busybox busybox-initramfs busybox-static ca-certificates
  uuid-runtime xauth ncurses-term iso-codes usbutils
The following packages will be REMOVED:
  gettext-base:i386 libboost-iostreams1.49.0:i386 libcwidget3:i386
  libept1.4.12:i386 libopts25:i386 libxapian22:i386 logrotate:i386
  sysvinit-utils:i386
The following NEW packages will be installed:
  adduser aide-common aptitude bsd-mailx console-common console-data
  console-log cron curl debian-goodies debsecan dmsetup e2fslibs e2fsprogs
  exim4-base exim4-config exim4-daemon-light gettext-base git grub-common
  grub-pc grub-pc-bin grub2-common ifupdown initramfs-tools initscripts ippl
  jed libapt-inst1.5 libasprintf0c2 libattr1 libblkid1
  libboost-iostreams1.49.0 libcap2 libcurl3 libcurl3-gnutls libcwidget3
  libdevmapper-event1.02.1 libdevmapper1.02.1 libedit2 libept1.4.12
  liberror-perl libfile-find-rule-perl libfreetype6 libfuse2 libgpm2
  libncurses5 libnumber-compare-perl libopts25 libparted0debian1 libprocps0
  libreadline5 libsepol1 libsigc++-2.0-0c2a libss2 libswitch-perl
  libterm-readkey-perl libterm-readline-perl-perl libtext-glob-perl
  libtimedate-perl libudev0 libuuid1 libwrap0 libxapian22 locales logrotate
  lsof lvm2 molly-guard ntp openssh-client openssh-server openssl
  openssl-blacklist openssl-blacklist-extra parted perl perl-modules procps
  python-apt rsyslog stow sysv-rc sysvinit sysvinit-utils tzdata ucf udev
  util-linux
WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
  sysvinit-utils:i386
0 upgraded, 89 newly installed, 8 to remove and 0 not upgraded.
Need to get 55.3 MB of archives.
After this operation, 134 MB of additional disk space will be used.
You are about to do something potentially harmful.
To continue type in the phrase  Yes, do as I say! 
 ?] Yes, do as I say! (actually type this!)
Get:1 http://debian.debian.zugschlus.de/debian/ wheezy/main e2fslibs amd64 1.42.5-1.1 [197 kB]
( )
Fetched 55.3 MB in 20s (2648 kB/s)                                            
perl: warning: Setting locale failed.
( )

This process may re-ask some debconf questions that you have already answered during initial system setup. I find it comforting to actually see that my answers were preserved, but you can always export DEBIAN_FRONTEND=noninteractive if you want installation to be silent. If you know how this step can be done more elegantly, please comment. We now, finally, have the resolver in a consistent state.

$ sudo apt-get -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libasprintf0c2:i386 libcurl3-gnutls:i386 libfreetype6:i386 libfuse2:i386
  libreadline5:i386
Use  apt-get autoremove  to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
$ dpkg --print-architecture
amd64
$ dpkg --print-foreign-architectures
i386
$ dpkg --list   grep i386   wc -l
121
$ dpkg --list   grep amd64   wc -l
119
$ 

Now we can force-crossgrade all i386 packages that are still left on the system. Again, we use the apt-get --download install, dpkg --install stunt as coreutils gets crossgraded at this step and dpkg is less than happy if it finds itself without /bin/rm. Remember, with --download, nothing is actually removed during this step even if apt-get claims to remove things. Apt will also complain that many of these packages are already installed. Those are libs that were pulled in previously. The first dpkg --install run in this step will complain about gazillions of dependency problems, which will all be resolved in concluding apt-get -f install after the second dpkg --install run, this time a manual one.

$ sudo apt-get --download-only install $(dpkg --list   awk  if( $1 ==  ii  && $4 ==
 i386  )   print $2     sed  s/:i386// )
Reading package lists... Done
Building dependency tree       
Reading state information... Done
( )
The following packages were automatically installed and are no longer required:
  libasprintf0c2:i386 libcurl3-gnutls:i386 libfreetype6:i386 libfuse2:i386
  libreadline5:i386
Use  apt-get autoremove  to remove them.
Suggested packages:
  powermgmt-base bash-doc bzip2-doc diffutils-doc wdiff mlocate locate
  gnupg-doc xloadimage imagemagick eog libpcsclite1 iproute-doc resolvconf
  avahi-autoipd nfs-common ncompress indent
Recommended packages:
  bsdmainutils gnupg-curl libatm1
The following packages will be REMOVED:
  anacron:i386 apt-utils:i386 base-files:i386 base-passwd:i386 bash:i386
( )
The following NEW packages will be installed:
  anacron apt-utils base-files base-passwd bash bsdutils busybox bzip2
( )
0 upgraded, 56 newly installed, 49 to remove and 0 not upgraded.
Need to get 24,1 MB of archives.
After this operation, 2.030 kB of additional disk space will be used.
Do you want to continue [Y/n]? y
Get:1 http://debian.debian.zugschlus.de/debian/ wheezy/main kmod amd64 9-3 [60,5 kB]
( )
Fetched 24,1 MB in 11s (2.120 kB/s)                                           
Download complete and in download only mode
$ sudo dpkg --install /var/cache/apt/archives/*.deb
(Reading database ... 17098 files and directories currently installed.)
Preparing to replace adduser 3.113+nmu3 (using .../adduser_3.113+nmu3_all.deb) ...
Unpacking replacement adduser ...
( )
Errors were encountered while processing:
 /var/cache/apt/archives/bash_4.2+dfsg-0.1_amd64.deb
 /var/cache/apt/archives/coreutils_8.13-3.5_amd64.deb
 initscripts
 lvm2
 procps
 rsyslog
 sysvinit
 sysv-rc
 udev
 util-linux
 aide-common
 dmsetup
 e2fsprogs
 ifupdown
 initramfs-tools
 libdevmapper1.02.1:amd64
 libdevmapper-event1.02.1:amd64
 libparted0debian1:amd64
 molly-guard
 openssh-server
 parted
 grub-common
 grub-pc
 grub-pc-bin
 grub2-common
$ sudo dpkg --install /var/cache/apt/archives/bash_4.2+dfsg-0.1_amd64.deb
/var/cache/apt/archives/coreutils_8.13-3.5_amd64.deb
(Reading database ... 17129 files and directories currently installed.)
Preparing to replace bash 4.2+dfsg-0.1 (using .../bash_4.2+dfsg-0.1_amd64.deb) ...
Unpacking replacement bash ...
Preparing to replace coreutils 8.13-3.5 (using .../coreutils_8.13-3.5_amd64.deb) ...
Unpacking replacement coreutils ...
Setting up bash (4.2+dfsg-0.1) ...
update-alternatives: using /usr/share/man/man7/bash-builtins.7.gz to provide /usr/share/man/man7/builtins.7.gz
(builtins.7.gz) in auto mode
Setting up coreutils (8.13-3.5) ...
$ sudo apt-get -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  libasprintf0c2:i386 libcurl3-gnutls:i386 libfreetype6:i386 libfuse2:i386
  libreadline5:i386
Use  apt-get autoremove  to remove them.
The following extra packages will be installed:
  sysvinit-utils
Suggested packages:
  bootlogd sash
The following packages will be REMOVED:
  libslang2-modules:i386 sysvinit-utils:i386
The following NEW packages will be installed:
  sysvinit-utils
WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
  sysvinit-utils:i386
0 upgraded, 1 newly installed, 2 to remove and 0 not upgraded.
23 not fully installed or removed.
Need to get 0 B/99,5 kB of archives.
After this operation, 328 kB disk space will be freed.
You are about to do something potentially harmful.
To continue type in the phrase  Yes, do as I say! 
 ?] Yes, do as I say! (actually type this!)
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
(Reading database ... 17129 files and directories currently installed.)
Removing sysvinit-utils ...
Selecting previously unselected package sysvinit-utils.
(Reading database ... 17105 files and directories currently installed.)
Unpacking sysvinit-utils (from .../sysvinit-utils_2.88dsf-41_amd64.deb) ...
Setting up sysvinit-utils (2.88dsf-41) ...
Setting up sysv-rc (2.88dsf-41) ...
Setting up initscripts (2.88dsf-41) ...
Setting up util-linux (2.20.1-5.3) ...
Setting up e2fsprogs (1.42.5-1.1) ...
Setting up sysvinit (2.88dsf-41) ...
sysvinit: restarting... done.
(Reading database ... 17129 files and directories currently installed.)
Removing libslang2-modules:i386 ...
Setting up ifupdown (0.7.8) ...
Setting up procps (1:3.3.3-3) ...
[ ok ] Setting kernel variables ... /etc/sysctl.conf...done.
Setting up udev (175-7.2) ...
[ ok ] Stopping the hotplug events dispatcher: udevd.
[ ok ] Starting the hotplug events dispatcher: udevd.
update-initramfs: deferring update (trigger activated)
Setting up rsyslog (5.8.11-3) ...
[ ok ] Stopping enhanced syslogd: rsyslogd.
[ ok ] Starting enhanced syslogd: rsyslogd.
Setting up aide-common (0.15.1-8) ...
Setting up initramfs-tools (0.109.1) ...
update-initramfs: deferring update (trigger activated)
Setting up openssh-server (1:6.0p1-4) ...
[ ok ] Restarting OpenBSD Secure Shell server: sshd.
Setting up molly-guard (0.4.5-1) ...
Setting up libdevmapper1.02.1:amd64 (2:1.02.74-7) ...
Setting up libdevmapper-event1.02.1:amd64 (2:1.02.74-7) ...
Setting up libparted0debian1:amd64 (2.3-12) ...
Setting up grub-common (1.99-27+deb7u1) ...
Setting up grub2-common (1.99-27+deb7u1) ...
Setting up grub-pc-bin (1.99-27+deb7u1) ...
Setting up grub-pc (1.99-27+deb7u1) ...
Installation finished. No error reported.
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-3.9.4-zgsrv2008064
Found initrd image: /boot/initrd.img-3.9.4-zgsrv2008064
done
Setting up parted (2.3-12) ...
Setting up dmsetup (2:1.02.74-7) ...
update-initramfs: deferring update (trigger activated)
Setting up lvm2 (2.02.95-7) ...
[ ok ] Setting up LVM Volume Groups...done.
update-initramfs: deferring update (trigger activated)
Processing triggers for initramfs-tools ...
update-initramfs: Generating /boot/initrd.img-3.9.4-zgsrv2008064
Found 12 processes using old versions of upgraded files
( )
$ sudo apt-get -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libasprintf0c2:i386 libcurl3-gnutls:i386 libfreetype6:i386 libfuse2:i386
  libreadline5:i386
Use  apt-get autoremove  to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
$ dpkg --print-architecture
amd64
$ dpkg --print-foreign-architectures
i386
$ dpkg --list   grep i386   wc -l
72
$ dpkg --list   grep amd64   wc -l
175
$ 

We have completed the actual crossgrade. All non-multiarch packages are now amd64. The resolver is now fine as well. At this current state, all i386 packages that are still installed should be unneeded libs. We can safely nuke them. Careful minds may inspect the list of packages to be removed for important or non-lib items. After completion of the command, a second call of the same command verifies that we were successful.

$ sudo apt-get remove $(dpkg --list   awk  if( $1 ==  ii  && $4 ==  i386  )   print $2
  )
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  e2fslibs:i386 gcc-4.7-base:i386 libacl1:i386 libapt-inst1.5:i386
  libapt-pkg4.12:i386 libasprintf0c2:i386 libattr1:i386 libblkid1:i386
  libbsd0:i386 libbz2-1.0:i386 libc6:i386 libcap2:i386 libcomerr2:i386
  libcurl3:i386 libcurl3-gnutls:i386 libdb5.1:i386 libedit2:i386
  libexpat1:i386 libfreetype6:i386 libfuse2:i386 libgcc1:i386
  libgcrypt11:i386 libgdbm3:i386 libgnutls26:i386 libgpg-error0:i386
  libgpm2:i386 libgssapi-krb5-2:i386 libidn11:i386 libk5crypto3:i386
  libkeyutils1:i386 libkmod2:i386 libkrb5-3:i386 libkrb5support0:i386
  libldap-2.4-2:i386 liblockfile1:i386 liblzma5:i386 libmagic1:i386
  libncurses5:i386 libncursesw5:i386 libp11-kit0:i386 libpam-modules:i386
  libpam0g:i386 libpci3:i386 libpcre3:i386 libpng12-0:i386 libpopt0:i386
  libprocps0:i386 libreadline5:i386 libreadline6:i386 librtmp0:i386
  libsasl2-2:i386 libselinux1:i386 libsemanage1:i386 libsepol1:i386
  libsigc++-2.0-0c2a:i386 libslang2:i386 libsqlite3-0:i386 libss2:i386
  libssh2-1:i386 libssl1.0.0:i386 libstdc++6:i386 libtasn1-3:i386
  libtinfo5:i386 libudev0:i386 libusb-0.1-4:i386 libustr-1.0-1:i386
  libuuid1:i386 libwrap0:i386 linux-image-3.9.4-zgsrv2008064:i386 zlib1g:i386
0 upgraded, 0 newly installed, 70 to remove and 0 not upgraded.
After this operation, 67,5 MB disk space will be freed.
Do you want to continue [Y/n]? y
(Reading database ... 17111 files and directories currently installed.)
Removing e2fslibs:i386 ...
( )
$ sudo apt-get remove $(dpkg --list   awk  if( $1 ==  ii  && $4 ==  i386  )   print $2
  )
Reading package lists... Done
Building dependency tree       
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
$ dpkg --print-architecture
amd64
$ dpkg --print-foreign-architectures
i386
$ dpkg --list   grep i386   wc -l
70
$ dpkg --list   grep amd64   wc -l
175
$ 

To finally make the packages really vanish from the system, we purge all packages that dpkg --list still reports in state rc , which means removed, configuration still present . Thankfully, dpkg seems to handle the case where a dpkg-conffile belongs to a transitioned package gracefully.

$ sudo dpkg --purge $(dpkg --list   awk  if ($1 ==  rc )   print $2  )
(Reading database ... 15868 files and directories currently installed.)
Removing e2fslibs:i386 ...
Purging configuration files for e2fslibs:i386 ...
( )
$ dpkg --print-architecture
amd64
$ dpkg --print-foreign-architectures
i386
$ dpkg --list   grep i386   wc -l
0
$ dpkg --list   grep amd64   wc -l
175
$ 

We now verify that all packages we have are either arch all or arch amd64.

$ dpkg --list   grep -v  \(amd64 all\) 
Desired=Unknown/Install/Remove/Purge/Hold
  Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
 / Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
 / Name                            Version                   Architecture Description
+++-===============================-=========================-============-========================================================================

As I found myself suddenly without e2fsprogs during one of my experiments, I check for e2fsprogs being present. If this package is missing, you ll only find out after rebooting.

$ dpkg --list e2fsprogs
Desired=Unknown/Install/Remove/Purge/Hold
  Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
 / Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
 / Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  e2fsprogs      1.42.5-1.1   amd64        ext2/ext3/ext4 file system utilit

My locally built amd64 kernel with 32 bit support was an arch i386 package as well, so it got zapped in the migration process. Replace it with a real amd64 kernel. This step will vary if you use Debian kernels. Do not forget the firmwares.

$ sudo apt-get -t wheezy-zg-experimental install kernel-image-zgserver linux-firmware-image
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  cpio initramfs-tools kernel-image-zgsrv20080 klibc-utils libklibc
  linux-image-3.9.4-zgsrv20080
Suggested packages:
  libarchive1
The following NEW packages will be installed:
  cpio initramfs-tools kernel-image-zgserver kernel-image-zgsrv20080
  klibc-utils libklibc linux-firmware-image linux-image-3.9.4-zgsrv20080
0 upgraded, 8 newly installed, 0 to remove and 0 not upgraded.
Need to get 11,2 MB of archives.
After this operation, 30,5 MB of additional disk space will be used.
Do you want to continue [Y/n]? 
Get:1 http://zg20110.debian.zugschlus.de/ wheezy-zg-experimental/main linux-image-3.9.4-zgsrv20080 amd64
3.9.4.20130605.0 [10,4 MB]
( )

I now make sure that the initrd I have built contains all file system modules and is of the right architecture:

$ cd /tmp
$ < /boot/initrd.img-3.9.4-zgsrv20080 gunzip   cpio -i
32053 blocks
$ find . -name  *ext*.ko 
./lib/modules/3.9.4-zgsrv20080/kernel/fs/ext3/ext3.ko
./lib/modules/3.9.4-zgsrv20080/kernel/fs/ext2/ext2.ko
./lib/modules/3.9.4-zgsrv20080/kernel/fs/ext4/ext4.ko
$ file ./lib/modules/3.9.4-zgsrv20080/kernel/fs/ext4/ext4.ko
./lib/modules/3.9.4-zgsrv20080/kernel/fs/ext4/ext4.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV),
BuildID[sha1]=0x259a642448805c9597932ee25c62c8e6a00a32ad, not stripped
$ cd

As the last step, remove i386 as an architecture and reboot into the newly migrated system. We verify that the system considers itself amd64 with no foreign architectures.

$ sudo dpkg --remove-architecture i386
$ dpkg --print-architecture
amd64
$ 
$ dpkg --print-foreign-architectures
$ dpkg --list   grep i386   wc -l
0
$ dpkg --list   grep amd64   wc -l
181
$ sudo shutdown -r now
Broadcast message from root@wheezyarch (pts/0) (Wed Jun 12 16:09:24 2013):
The system is going down for reboot NOW!
$ Connection to wheezyarch.zugschlus.de closed by remote host.
Connection to wheezyarch.zugschlus.de closed.
$ ssh wheezyarch.zugschlus.de
Linux wheezyarch 3.9.4-zgsrv20080 #2 SMP PREEMPT Wed Jun 5 13:12:04 UTC 2013 x86_64
$ dpkg --print-architecture
amd64
$

We now check whether we have lost any packages in this process:

$ sudo dpkg --get-selections   grep install   awk  print $1    sed  s/:.*//;s/$/:amd64
install/;  > get-selections.post
$ wc -l get-selections.*
  234 get-selections.post
  226 get-selections.pre
$ diff --unified=0 get-selections.pre get-selections.post 
--- get-selections.pre  2013-06-12 19:47:40.278176000 +0200
+++ get-selections.post 2013-06-12 20:28:51.389874000 +0200
@@ -22,0 +23 @@
+cpio:amd64 install
@@ -58,0 +60 @@
+initramfs-tools:amd64 install
@@ -68,0 +71,3 @@
+kernel-image-zgserver:amd64 install
+kernel-image-zgsrv20080:amd64 install
+klibc-utils:amd64 install
@@ -108,0 +114 @@
+libklibc:amd64 install
@@ -163,0 +170,2 @@
+linux-firmware-image:amd64 install
+linux-image-3.9.4-zgsrv20080:amd64 install

So we have not actually lost things, but we have gained the pure amd64 kernel and the packages that are needed to build a proper initrd. These were missing from the initial install of the i386 test VM. The downside of the process is that we have lost most of the automatically installed markers in aptitude which make sure to remove a library once nothing depends on it any more. If you make a point in having those markers correctly set, as I do, this is tedious work to set it on every package in the aptitude curses interface, making aptitude suggest removing the entire system, and then again remove it from the packages you actually intend to keep. This should not result in anything being removed, otherwise you have goofed previously. All Done! I am now off perfecting my upgrade process so that the i386 lenny and squeeze machines can first be moved to wheezy and then to amd64.

There is a certain cross-over between the Debian Edu / Skolelinux project and the Edubuntu project, and for example the LTSP packages in Debian are a joint effort between the projects. One person with a foot in both camps is Jonathan Carter, which I am now happy to present to you. Who are you, and how do you spend your days? I'm a South-African free software geek who lives in Cape Town. My days vary quite a bit since I'm involved in too many things. As I'm getting older I'm learning how to focus a bit more :) I'm also an Edubuntu contributor and I love when there are opportunities for the Edubuntu and Debian Edu projects to benefit from each other. How did you get in contact with the Skolelinux / Debian Edu project? I've been somewhat familiar with the project before, but I think my first direct exposure to the project was when I met Petter [Reinholdtsen] and Knut [Yrvin] at the Edubuntu summit in 2005 in London. They provided great feedback that helped the bootstrapping of Edubuntu. Back then Edubuntu (and even Ubuntu) was still very new and it was great getting input from people who have been around longer. I was also still very excitable and said yes to everything and to this day I have a big todo list backlog that I'm catching up with. I think over the years the relationship between Edubuntu and Debian-Edu has been gradually improving, although I think there's a lot that we could still improve on in terms of working together on packages. I'm sure we'll get there one day. What do you see as the advantages of Skolelinux / Debian Edu? Debian itself already has so many advantages. I could go on about it for pages, but in essence I love that it's a very honest project that puts its users first with no hidden agendas and also produces very high quality work. I think the advantage of Debian Edu is that it makes many common set-up tasks simpler so that administrators can get up and running with a lot less effort and frustration. At the same time I think it helps to standardise installations in schools so that it's easier for community members and commercial suppliers to support. What do you see as the disadvantages of Skolelinux / Debian Edu? I had to re-type this one a few times because I'm trying to separate "disadvantages" from "areas that need improvement" (which is what I originally rambled on about) The biggest disadvantage I can think of is lack of manpower. The project could do so much more if there were more good contributors. I think some of the problems are external too. Free software and free content in education is a no-brainer but it takes some time to catch on. When you've been working with the same proprietary eco-system for years and have gotten used to it, it can be hard to adjust to some concepts in the free software world. It would be nice if there were more Debian Edu consultants across the world. I'd love to be one myself but I'm already so over-committed that it's just not possible currently. I think the best short-term solution to that large-scale problem is for schools to be pro-active and share their experiences and grow their skills in-house. I'm often saddened to see how much money educational institutions spend on 3rd party solutions that they don't have access to after the service has ended and they could've gotten so much more value otherwise by being more self-sustainable and autonomous. Which free software do you use daily? My main laptop dual-boots between Debian and Windows 7. I was Windows free for years but started dual-booting again last year for some games which help me focus and relax (Starcraft II in particular). Gaming support on Linux is improving in leaps and bounds so I suppose I'll soon be able to regain that disk space :) Besides that I rely on Icedove, Chromium, Terminator, Byobu, irssi, git, Tomboy, KVM, VLC and LibreOffice. Recently I've been torn on which desktop environment I like and I'm taking some refuge in Xfce while I figure that out. I like tools that keep things simple. I enjoy Python and shell scripting. I went to an Arduino workshop recently and it was awesome seeing how easy and simple the IDE software was to get up and running in Debian compared to the users running Windows and OS X. I also use mc which some people frown upon slightly. I got used to using Norton Commander in the early 90's and it stuck (I think the people who sneer at it is just jealous that they don't know how to use it :p) Which strategy do you believe is the right one to use to get schools to use free software? I think trying to force it is unproductive. I also think that in many cases it's appropriate for schools to use non-free systems and I don't think that there's any particular moral or ethical problem with that. I do think though that free software can already solve so so many problems in educational institutions and it's just a shame not taking advantage of that. I also think that some curricula need serious review. For example, some areas of the world rely heavily on very specific versions of MS Office, teaching students to parrot menu items instead of learning the general concepts. I think that's very unproductive because firstly, MS Office's interface changes drastically every few years and on top of that it also locks in a generation to a product that might not be the best solution for them. To answer your question, I believe that the right strategy is to educate and inform, giving someone the information they require to make a decision that would work for them.

So I'd previously talked about migrating machines. From having one virtual machine running "mail" + "web" + "stuff" I've now got three hosts:

ssh.steve.org.uk

This is supposed to be used solely for shell access, email reading, IRC. Sadly it still hosts one website, the web interface to my Mercurial repositories. This can't be moved without moving the repositories which is a step too far. Although I don't particularly want people browsing my code/changes I do want them to be able to clone them. If I could get anonymous-SSH checkouts working, sanely, then I'd be happy, but I don't see how to do that.

skx-web

This hosts all my websites except for two. The two that are excluded are my mercurial repositories, which still lives on ssh.steve.org.uk, and my blogspam service.

blogspam

This runs my blogspam.net service. I wish I could retired this, since it uses cruftly XML::RPC. I'd rather see a RESTful application sending/receiving JSON. Sadly I can't kill it without annoying a lot of people. So it must remain.

I chose to add the blogspam guest, because that service does really take over an IP, and it just seemed simplest to move it to one machine. As a quick hack both http://repository.steve.org.uk & http://blogspam.net/ run under Apache. Although the other websites run under their own UID with thttpd + my proxy. Now time to change the subject entirely. I've recently joined gym. Which isn't as horrible as I thought it might be, though as a matter of policy I refuse to go on any of those fancy running/jogging machines. The first three weeks I just alternated between the cycling machines and the rowing machines. Initially ten minutes on each, then twenty, then thirty. Now I'm being all brave and adventurous, using new machines and pieces of equipment. Writing this I've got a dull ache in both my arms, after doing seated-dips with 100Lbs. So I guess I'm starting to make progress. No specific goals in mind, but I've been paying slightly more attention to my diet over the past month and I think if I'm a "little fitter" and have "slightly nicer arms" then I'll be happy enough. I've no desire to go all anal and count calories, or give up chocolate and beer. So it is almost hard to explain why I'm going, but .. it is fun, and watching the numbers change is fun too. I'll probably post more about this in the future.

Over my time working with Debian packages, I've always been concerned that I have been missing catchable mistakes by not running all the static checking tools I could run. As a result, I've been interested in writing some code that automates this process, a place where I can push a package and come back a few hours later to check on the results. This is great, since it provides a slightly less scary interface to new packagers, and helps avoid thinking they've just been told off by a Developer. I've spent the time to actually write this code, and I've called it debuild.me. The code itself is in its fourth iteration, and is built up from a few core components. The client / server code (lucy and ethel) are quite interconnected, but firehose works great on it's own, and is a single, unified (and sane!) spec that is easy to hack with (or even on!). Hopefully, this means that our wrappers will be usable outside of debuild.me, which is a win for everyone. Backend Design The backend (lucy) was the first part I wanted to design. I made the decision (very early on) that everything was going to be 100% Python 3.3+. This lets me use some of the (frankly, sweet) tools in the stdlib. Since I've written this type of thing before (i've tried to write this tool many, many, many, many times before), so I had a rough sense of how I wanted to design the backend. Past iterations had suffered from and overly complex server half, so I decided to go ultra minimal with the design of debuild.me. <aside class="left"> You can find the code for the server (lucy) on my GitHub </aside> The backend watches a directory (using a simple inotify script) and processes .changes files as they come in. If the package is a source package, a set of jobs are triggered (such as lintian, build and desktop-file-validate), as well as a different set for binary packages (such as lintian, piuparts and adequite). Only people may upload source packages (without any debs) and only builders can upload binary packages (without source). The client and server talk using XML-RPC with BASIC HTTP auth. I'm going to (eventually) SSL secure the transport layer, but for now, this will work as a proof of concept. Since I tend to like to keep my codebase simple and straightforward, I've used MongoDB as Lucy's DB. This lets me move between documents in Mongo to Python objects without any trouble. In addition, I evaluated some of the queue code out there (ZMQ, etc), and they all seemed like overkill for my problem, and had a hard time keeping track of jobs that (must never!) get lost. As a result, I wrote my own (very simple) job queue in Mongo, which has no sense of scheduling (at all), but can do it's job (and do it well). Jobs describe what's to be built with a link to the package document that the job relates to, and it's arch and suite (don't worry about the rest just yet). Jobs get assigned via natural sort on it's UUID based _id, and assigned to the first builder that can process it's arch / suite. Source packages are considered arch:all / suite:unstable (so they always get the most up-to-date linters on any arch that comes along). Lucy also allows for uploads to be given an X-Lucy-Group tag to manage which set of packages they're a part of. This comes in handy for doing partial archive rebuilds, or eventually using it to manage what jobs should be run on which uploads. This will allow me to run much more time-consuming tools for packages I want to review versus rebuilding to ensure packages don't FTBFS or aren't adequite. Client Design The buildd client (ethel) talks with lucy via XML-RPC to get assigned new jobs, release old jobs, close finished jobs, and upload package report data. When the etheld requests a new job, it also passes along what suites it knows of, which arches it can build, as well as what types it can run (stuff like lintian, build or cppcheck.) Lucy then assigns the builder to that job (so that we don't allocate the same job twice), and what time it was assigned at. <aside class="right"> You can find the code for the client (ethel) on my GitHub </aside> Ethel then takes the result of the job (in the form of a firehose.model tree) and transmits it over the line back to the Lucy server as a report (which also contains information on if the build failed or not), at which point lucy hands back a location (on the lucy host) that the daemon can write the log to. If the job was a binary build, the etheld process will dput the package to the server, with a special X-Lucy-Job tag to signal which job that build relates to, so that future lint runs can fetch the deb files that the build produced. Tooling Ethel runs a set of static checkers on the source code, which are basically fancy wrappers around the tools we all know and love (like lintian, desktop-file-validate, or piuparts) which output Firehose in place of home-grown stdout. This allows us to programmatically deal with the output of these tools in a normal and consistent way. <aside class="left"> You can read more about Firehose over in the Firehose README.rst </aside> Some of the more complex runners are made of 3 parts - a runner, wrapper and command. The server invokes the command routine, which invokes the runner (the command just provides a unified interface to all the runners), who's output gets parsed by the wrapper to turn it into a Firehose model tree. The goal here is that tons of very quick-running tools get run over a distributed network, and machine-readable reports get filed in a central location to aid in reviewing packages. Ricky In addition to the actual code to run builds, I've worked on a few tools to aid with using debuild.me for my DD related life. I have some uncommon use-cases that are nice to support. One such use-case is the ability to rebuild packages from the archive (unmodified) to check that they rebuild OK against the target. This is handy for things like arch:all packages that get uploaded (since they never get rebuilt on the buildd machines, and FTBFSs are sadly common) or packages that have had a Build-Dependency change on them. Ricky is able to create a .dsc url to your friendly local mirror, and fetch that exact version of the package. Ricky can then also use the .dsc (in a monumental hack) to forge a package_version_source.changes file, and sign it with an autobuild key and upload it to the debuild.me instance. Since it can also modify the .changes's target distribution, you can also use this to test if a package will build on stable or testing, unmodified. Fred Fred is a wrapper around Ricky, to help with fetching packages that may not exist yet. Fred also contains an email scraper to read off such lists as debian-devel-changes, and add an entry to fetch that upload when it becomes available on the local mirror, pass it to ricky, and allow debuild.me to rebuild new packages that match a set of criteria. I'm currently playing around with the idea of rebuilding all incoming Python packages to ensure they don't FTBFS in a clean chroot. Loofah Loofah is also another wrapper around Ricky, but for use manually. Loofah is able to sync down the apt Sources list, and place it in Mongo for fast queries. This than allows me to manually run rebuilds on any Source package that fits a set of critera (written in the form of a Mongo query), which get pulled and uploaded by Ricky. An example script to rebuild any packages that Build-Depend on python3-all-dev in Debian unstable / main would look like: <aside class="right"> You can find more queries in the Loofah examples </aside> Or, a script to rebuild any package that depends on CDBS: You can use anything that exists in the Sources.gz file to query off of ( including Maintainer!) Future Work The future work on debuild.me will be centered around making it easier for buildd nodes to be added to the network, with more and more automation in that process (likely in the form of debs). I also want to add better control over the jobs, so that packages I upload only go to my personal servers. I'd also very much like to get better EC2 / Virtualization support integrated into the network, so that the buildd count grows with the queue size. This is a slightly hard problem that I'm keen to fix. I'm also considering moving the log parsing code out of the workers, so that the parsing code can be fixed without upgrading all the workers. This would also drop the Firehose dep on the client code, which would be nice. Migration from a debuild.me build into a local reprepro repo is something that would be fairly easy to do as well, likely to be done remotely via the XML-RPC interface, which calls a couple of reprepro commands (such as includedsc and includedeb) and publishes it to the user's repo. This is a nice use of the debs that get built, and could also allow debuild.me to be used like a PPA system, but this allows the user to not migrate packages that may contain piuparts issues.

This blog post is the first of a series of posts dealing with the results of the Debian systemd survey. I intend to give a presentation at DebConf 2013, too, so you could either read my posts, or watch the talk, or both :-). The top concern shared by most people is:

systemd has too many dependencies, or it is bloated, or it does too many things, or is too complex

Now this concern actually has a lot of different facets, and I am trying to share my opinion on each of them. systemd has too many dependencies First, let s start with too many dependencies , because that is easy to check and reason about. I have created a document which lists all dependencies of the systemd binary itself (pid 1) and all the binaries which are currently shipped by the systemd Debian package. If you don t want to take my word for granted, please read that document. Have you read the document? Very nice! As you can see, the systemd binary itself has 10 dependencies (excluding libc). Now, the question is, what is bad about dependencies? Why do people list dependencies as a top concern?

1. Cyclic dependencies. When you hear that your init system depends on DBus, you might argue that there is a cyclic dependency here, because DBus needs to be started by the init system. However, systemd does NOT depend on dbus-daemon (!) to boot your machine. Instead of using the system bus, it uses a private UNIX socket. Therefore, systemd uses DBus merely as a serialization format for IPC between its different processes. Only when you want to access systemd via its API as a user (non-root), you actually use the system bus. Since we are talking about DBus: DBus provides a well-tested serialization format and IPC mechanism so that systemd doesn t have to reinvent the wheel and instead benefits from wide support within languages.
2. Complicated code. I feel like there is the implicit assumption that lots of dependencies correlate with complicated code that is easy to break. I encourage you to have a look at systemd s source code: look for the places where specific libraries are used, e.g. enforce_user which uses libcap. You ll notice that the code is not complex and usage of the libraries is clear.
3. Software dragging in lots of library packages. The libraries which systemd uses are already in widespread use (e.g. DBus, udev, selinux, libcap, pcre, ). On a typical Debian installation, only very few of them will be dragged in by systemd, if at all. As an example, on a fresh Debian Wheezy installation, less than 10 packages will end up on your machine when running apt-get install systemd .
4. More memory use. The Linux kernel maps libraries into memory only once, no matter how many processes use them. As stated in the dependency list, on machines where the libraries are not already loaded, systemd brings in about 500 KiB of additional memory-mapped libraries in the worst case. On the machines we have these days, this is a reasonable cost to pay for all the benefits systemd gets us. This holds true on embedded systems with only a few MiB of RAM and especially on typical workstations with 8+ GiB of RAM.

systemd is bloated Now, let s talk about bloat. Again, this is a point which has many facets. I d like to quote the Wikipedia definition of software bloat:

Software bloat is a process whereby successive versions of a computer program become perceptibly slower, use more memory or processing power, or have higher hardware requirements than the previous version whilst making only dubious user-perceptible improvements.

The first part of the definition certainly does not match systemd it is measurably faster than sysvinit. As for memory usage: systemd s RSS is 1.8 MiB, whereas sysvinit uses 0.8 MiB. As I argued on the More memory use point in the dependencies section, I think the additional resource cost is well worth the benefits. Also note that systemd s features are NOT all implemented in the binary which is PID 1. As explained in the dependency list, systemd consists of many cleanly separated binaries. So if a new version of systemd gathers an additional feature, this does not mean that your PID 1 will be bigger. While systemd runs on any hardware, it has an indirect hardware requirement: it requires some Linux kernel features (which are all enabled in Debian kernels). That might rule out usage of systemd on really old embedded hardware where you don t have a chance to update the kernel. While it is sad that those machines cannot profit from systemd, switching to systemd as a default has no downside either: Debian continues to support sysvinit for quite some time, so these machines will continue to work even with upcoming Debian versions. systemd does too many things The Wikipedia definition continues:

[ ] perceived bloat can occur from the software servicing a large, diverse marketplace with many differing requirements. Most end users will feel they only need some limited subset of the available functions and will regard the others as unnecessary bloat, even if people with different requirements do use them.

I think the last part of the Wikipedia definition applies to systemd: it does service a large and diverse marketplace . That marketplace is the entirety of existing software which is started by an init system. Also, systemd can be used on a wide range of hardware (embedded devices, tablets, phones, notebooks, desktops, servers) which requires different features. As an example: on a desktop system you typically don t care strongly about a watchdog feature, but on embedded or servers that feature is very handy. Similarly, on a tablet, forward secure sealing of logfiles is not as important as on a server. Therefore, I can understand if you feel that you don t need many of the features systemd provides. But please think of other users and maintainers who are very happy with systemd s benefits. Also note that while systemd supports many things (in separate binaries!), you don t have to use them all. It still makes sense to ship them all in the same package. Take coreutils as another example in that area. The binaries belong together, even though you most likely haven t used all of them (e.g. od, pr, ptx, :-)). systemd is too complex The remaining concern is that systemd is too complex. In my experience, complexity is often inherent to a specific area and one cannot simply make it go away. Instead, there are different models of how that complexity is represented. Think of the monolithic Linux kernel versus the MINIX microkernel. The latter has a very small amount of lines of source in the kernel, but puts the complexity into userspace. The former uses a different approach with more source in the kernel. The arguments between both camps show that neither is clearly right or clearly wrong. In a way, sysvinit represents the MINIX model: it has a small core (the init binary itself), but a lot of complexity in shell scripts and external programs. The fact that solutions are copied from one init script to another leads to lots of subtle errors and makes code reuse really hard. systemd however has more source code in the binaries, but requires only very simple, descriptive, textual configuration instead of complex init scripts. To me, it seems preferable to have the complexity in a single place instead of distributed across lots of people and projects. Conclusion In a way, you are right. systemd centralizes complexity from tons of init scripts into a single place. However, it therefore makes it very easy for maintainers to write service files (equivalent of an init script) and provides a consistent and reliable interface for service management. Furthermore, it is different than sysvinit, and different solutions often seem complex at first. While systemd consumes more resources than sysvinit, it uses them to make more information available about services; its finer-grained service management requires more state-keeping, but in turn offers you more control over your services.

Yes! It is possible! I ve done it! On my production linode! It was a bit hard, and I made quite a few mistakes, but I m convinced that a minimal system can be sanely crossgraded. I did it and only had an hour or so of server downtime In the process. However, most of the stack gets stripped down and back out, so I fear what happens with the X stack (nevermind GNOME Or anything that depends on it) A more detailed post to follow! Thanks to the dpkg team for making this happen!