Yes, You Should Not discard cached ssh host keys without looking. An unexpected change of an ssh host key is always a reason to step back from the keyboard and think. However, there are situations when you know that a systems ssh host key has changed, for example when the system reachable under this host name has been redeployed, which happens increasingly often proportionally to the devopsness of your environment, or for example in test environments. Later versions of ssh offer you the ssh-keygen -R command line to paste from the error message, so that you can abort the connection attempt, paste the command and reconnect again. This will still ask for confirmation of the new host key though. Almost every sysadmin has an alias or wrapper to make handling of this situation easier. Solutions range from using StrictHostKeyChecking no and/or UserKnownHostsFile /dev/null , turning off this layer of securit altogether either globally or usually too broadly, to more-or-less sophisticated solutions that involve turning off know-host file hashing, parsing client output and/or grep-sed-awk magic. grml even comes with an insecssh script that is rather neat and that I used until I developed my own. Later ssh versions (not including the 6.7 in Debian jessie, but the 7.4 in debian sid) do offer a -G command line option which will give access to configuration options after the complete client configurtion was processed. This will allow you to neatly get access to the actual hostname the ssh client will connect to. This can, in turn be used to obtain the IP address so that all traces of the host and its IP addresses can be removed from the known_hosts file before the actual connection is done. The following script is the insecssh that I currently use. It only uses documented interfaces to the ssh client which I consider rather neat.

#!/bin/bash
HOSTNAME=$(ssh -G  $@    grep  ^hostname '   awk  print $2 )
CNAME= 1 
while [ -n  $CNAME  ]; do
  CNAME=$(dig +short +search -t CNAME  $HOSTNAME )
  if [ -n  $CNAME  ]; then
    HOSTNAME= $CNAME 
  fi
done
IP4ADDRESS=$(dig +short +search -t A  $HOSTNAME )
IP6ADDRESS=$(dig +short +search -t AAAA  $HOSTNAME )
echo  HOSTNAME $HOSTNAME 
echo  IP4ADDRESS $IP4ADDRESS 
echo  IP6ADDRESS $IP6ADDRESS 
[ -n  $HOSTNAME  ] && ssh-keygen -R  $HOSTNAME 
[ -n  $IP4ADDRESS  ] && ssh-keygen -R  $IP4ADDRESS 
[ -n  $IP6ADDRESS  ] && ssh-keygen -R  $IP6ADDRESS 
ssh -oStrictHostKeyChecking=no -oVisualHostKey=yes  $@ 

Ugliness results from the necessity of following CNAMEs manually since there is - to my knowlegde (educate me!) - no command line utility that has the output simplicity and selection powers of dig which can automatically follow CNAMEs without relying on the recursor having the CNAME cached or not. Use this script if you know that a host has changed its host key. It will first zap all knowledge of the previous host key from known_hosts and then invoke the ssh client with the given arguments from its command line, with options added so that you ll see the new host key in random art and that you don t need to manually confirm the new host key.

Starting with OpenSSH 7.2, a new restrict option for authorized_keys lines has become available. It sets all available restrictions that the current OpenSSH version can do (like no-agent-forwarding, no-x11-forwarding etc). One can individually turn on those features again by corresponding new options. This saves one from sorrows when a new capability of OpenSSH is introduced through an update which is enabled by default, since one has to remember that restricted authorized_keys lines are in unse and then to manually add the restrictions. On the downside, Debian jessie and CentOS 7 don t have a recent enough OpenSSH. So we ll have to continue worrying about new features being inadvertendly enabled for a while. P.S.: Yes, I haven t blogged about Linux and Debian in English in a while.

First of all, good news, MirBSD is not vulnerable to The Heartbleed Bug due to my deliberate choice to stick to an older OpenSSL version. My inquiry (in various places) as to what precisely could leak when a vulnerable client connected to a n n-vulnerable server has yet to be answered, though we can assume private key material is safe. Now the bad news: while the CA I use and a CA I don t use offer free rekeying (in general), a CA I also use occasionally refuses to do that. The ugly: they will not even revoke the certificates, so any attacker who gained your key, for example when you have been using a certificate of theirs on a Debian system, will be able to use it (e.g. to MITM your visitors traffic) unless you shell over lots of unreasonable money per certificate. (Someone wrote they got the fee waived, but others don t, nor do I. (There s also a great Twitter discussion-thingy about this involving Zugschlus, but I won t link Twitter because they are not accessible to Lynx users like me and other Planet Debian authors.) Now I ve got a dilemma here. I ve created a CA myself, to use with MirBSD infrastructure and things like that X.509 certificates for my hosts (especially so I can use them for SMTP) and possibly personal friends (whose PGP key I ve signed with maximum trust after the usual verification) but am using a StartSSL certificate for www.mirbsd.org as my GoDaddy wildcard certificate expires in a week or so (due to the aforementioned monetary issues), and I d rather not pay for a limited certificate only supporting a single vhost. There is absolutely no issue with that certificate and key (only ever generated and used on MirBSD, only using it in Apache mod_ssl). Then, there s this soon-to-be tax-exempt non-profit society of public utility I m working with, whose server runs Debian, and which is affected, but has been using a StartSSL certificate for a while. Neither the society nor I can afford to pay for revocation, and we do not see any possible justification for this especially in the face of CVE-2014-0160. I expect a rekey keeping the current validity end date, and would accept a revocation even if I were unable to get a new certificate, since even were we to get a certificate for the society s domain from someplace else, an attacker could still MITM us with the previous one from Startcom. The problem here is: I d really love to see (all of!) Startcom dropped from the global list of trustworthy CAs, but then I d not know from where to get a cert for MirBSD; Globalsign is not an option because I will not limit SSL compatibility to a level needed to pass their quality test possibly GoDaddy, ISTR they offer a free year to Open Source projects no idea about one for the society but it would solve the problem of not getting the certificates revoked. For everyone. I am giving Startcom time until Friday after $dayjob (for me); after that, I ll be kicking them off MirBSD s CA bundle and will be lobbying for Debian and Mozilla to do the same. Any other ideas of how to deal with that? I d probably pay 5 for a usable certificate accepted by people (including old systems, such as MSIE 5.0 on Win2k and the likes) without questioning most of the time, I only serve public content anyway and just use SSL to make the NSA s job more difficult (and even when not I m not dealing with any payment information, just the occasional login protected area). By the way, is there any way to access the information that is behind a current-day link to groups.google.com with Lynx or Pine? I can t help but praise GMane for their NNTP interface. ObFunfact: just when I was finished writing this wlog entry, I got a new eMail Special offer just for you. from GoDaddy. Sadly, no offer for a 5 SSL certificate, just the usual 20-35% off coupon code.

I have just uploaded PowerDNS 3.3-1~exp1 to Debian experimental. This new upstream version has introduced its own include directive, so Debian was able to drop its patch. Hence, our conffiles had to grow a .conf extension, which most of them didn t have in previous version. If anybody wants to test updates from PowerDNS 3.1 to the new 3.3-1~exp1 in Debian unstable, please go ahead and report bugs in the Debian BTS. The package is known to not offer seamless DNSSEC, I ll work on that before I upload to unstable. While we re at it: I would appreciate help with the PostgreSQL backend. Myself, I use mainly MySQL and am not too proficient in PostgreSQL. I ll accept both patches and more formal co-maintenance.

Migrating a Debian installation between architectures has always been difficult. The recommended way to crossgrade an i386 Debian to amd64 Debian was to reinstall the system, move over data and configuration. For the more brave, in-place crossgrades usually involved chroots, rescue CDs, a lot of ar p tar xf - data.tar.gz and luck. I have never been brave when it comes to system administration, have done a lot of architecture migrations with reinstallation, and have always taken the opportunity to clear out the contamination that accumulates itself when a system is running for a long time. I would even recommend doing this to most people even now. However, I have a few very ugly systems in place that are still on i386 because I didn t dare going the reinstallation path. Doing in-place crossgrades has become a lot easier since wheezy s release, since once now can have both i386 and amd64 libraries installed in parallel, which allows to replace foo:i386 with foo:amd64 without influencing the other parts of the system. The process is still full of pitfalls:

• apt removes a package before it reinstalls its new counterpart. This results in apt calling dpkg to remove dpkg, and then calling dpkg again to install dpkg. Guess which operation fails and the state of the system after this failure. Same applies to coreutils, which leaves the system without rm, which in turn dpkg of either architecture doesn t like. Using apt-get --download-only install to resolve dependencies and downloading the debs, followed by a traditional dpkg --install solves this issue since multiarch dpkg will replace a package with another one without deinstalling the first one first.
• At least for the process, you need a kernel that can run both 32bit and 64bit binaries for the i386 architecture. AFAIR, setting CONFIG_64BIT, CONFIG_X86_64 and CONFIG_IA32_EMULATION in the kernel configuration takes care of this.
• During the process, apt wants to remove the better part of your system. It is important to not let it do this, as it wants to deinstall essential packages as well.
• It looks like the process is not always exactly reproducible. During the first tries, I found myself without an initrd at all, with an initrd that lacked the ext[234].ko kernel modules, without working e2fstools and in a number of other undesireable states of the system.

I have only tried this yet with a freshly installed minimal wheezy server system. I do not, however, expect surprises when it comes to using this process with real life systems. I will document other pitfalls I have fallen into here at a later time. My minimal wheezy system was running in a KVM VM with its virtual disk as a LVM LV in the host system. I took a snapshot before beginning and used lvconvert --merge numerous time to return my LV to the original state. Be aware that lvconvert --merge removes the snapshot after merging it, so you ll need to re-create the snapshot before trying again. During the process, I discussed things with Paul Tagliamonte, who has done this before, but on a live system and with a slightly more invasive approach. He has blogged about this. Thank you very much, your hints were very helpful. Here is a commented typescript of what I have done. Be warned: This was a lab setting with a rather minimal system. If you re going to try this with a production system, have a backup, or, if you re on snapshottable infrastructure, take snapshots and be prepared to roll back if anything goes wrong. First let s see what we have and save a list of installed packages.

mh@swivel:~$ ssh wheezyarch.zugschlus.de
$ uname -a
Linux wheezyarch 3.9.4-zgsrv2008064 #2 SMP PREEMPT Wed Jun 5 12:57:51 UTC 2013 x86_64 GNU/Linux
$ dpkg --print-architecture
i386
$ dpkg --print-foreign-architectures
$ dpkg --list   grep i386   wc -l
175
$ dpkg --list   grep amd64   wc -l
0
$ sudo dpkg --get-selections   grep install   awk  print $1    sed  s/:.*//;s/$/:amd64
install/;  > get-selections.pre

So we have an i386 system running an x86_64 kernel. Next, we add amd64 as an additional architecture which allows us to install amd64 packages. We download the amd64 packages files. Note that dpkg still considers this an i386 system with amd64 as a foreign arch.

$ sudo dpkg --add-architecture amd64
$ sudo apt-get update
Get:1 http://security.debian.org wheezy/updates Release.gpg [836 B]
( )
Fetched 6.155 kB in 4s (1.400 kB/s)
Reading package lists... Done
$ dpkg --print-architecture
i386
$ dpkg --print-foreign-architectures
amd64
$ dpkg --list   grep i386   wc -l
175
$ dpkg --list   grep amd64   wc -l
0

Next, download everything that is necessary to migrate dpkg and apt from i386 and amd64. We cannot directly apt things as the remove-install cycle used by apt will leave us without working dpkg. One could work around this by un-aring the dpkg.deb and directly forcing the amd64 files upon the system, but I think this approach is much cleaner. apt-get download will not resolve dependices while apt-get --download install does. After the download, dpkg --install the packages. dpkg is not particularly smart about installation order, so Pre-Depends may fail or not. If they fail, dpkg will give a list of deb files that were affected, which is an easy help to finish their installation. One could also choose the easy way (which I did here) and repeat the dpkg --install *.deb command. Since the Pre-Depends will already be installed, the second install run will succeed.

$ sudo apt-get --download-only install dpkg:amd64 apt:amd64
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  gcc-4.7-base:amd64 libapt-pkg4.12:amd64 libbz2-1.0:amd64 libc6:amd64
  libgcc1:amd64 liblzma5:amd64 libselinux1:amd64 libstdc++6:amd64
  zlib1g:amd64
Suggested packages:
  aptitude:amd64 synaptic:amd64 wajig:amd64 dpkg-dev:amd64 apt-doc:amd64
  python-apt:amd64 glibc-doc:amd64 locales:amd64
The following packages will be REMOVED:
  apt cron-apt dpkg
The following NEW packages will be installed:
  apt:amd64 dpkg:amd64 gcc-4.7-base:amd64 libapt-pkg4.12:amd64
  libbz2-1.0:amd64 libc6:amd64 libgcc1:amd64 liblzma5:amd64 libselinux1:amd64
  libstdc++6:amd64 zlib1g:amd64
0 upgraded, 11 newly installed, 3 to remove and 0 not upgraded.
Need to get 10,0 MB of archives.
After this operation, 14,9 MB of additional disk space will be used.
Do you want to continue [Y/n]? 
Get:1 http://debian.debian.zugschlus.de/debian/ wheezy/main gcc-4.7-base amd64 4.7.2-5 [144 kB]
( )
Fetched 10,0 MB in 2s (3.535 kB/s)
Download complete and in download only mode
$ sudo dpkg --install /var/cache/apt/archives/*.deb
(Reading database ... 16586 files and directories currently installed.)
Preparing to replace apt 0.9.7.8 (using .../archives/apt_0.9.7.8_amd64.deb) ...
Unpacking replacement apt ...
dpkg: regarding .../dpkg_1.16.10_amd64.deb containing dpkg, pre-dependency problem:
 dpkg pre-depends on libbz2-1.0
dpkg: error processing /var/cache/apt/archives/dpkg_1.16.10_amd64.deb (--install):
 pre-dependency problem - not installing dpkg
Selecting previously unselected package gcc-4.7-base:amd64.
( )
Errors were encountered while processing:
 /var/cache/apt/archives/dpkg_1.16.10_amd64.deb
$ sudo dpkg --install /var/cache/apt/archives/*.deb
(Reading database ... 16905 files and directories currently installed.)
Preparing to replace apt 0.9.7.8 (using .../archives/apt_0.9.7.8_amd64.deb) ...
Unpacking replacement apt ...
( )
$

From dpkg s point of view, the system is now already amd64, with i386 being a foreign arch. However, the majory of packages is still i386.

$ dpkg --print-architecture
amd64
$ dpkg --print-foreign-architectures
i386
$ dpkg --list   grep i386   wc -l
173
$ dpkg --list   grep amd64   wc -l
11
$ 

The apt resolver is currently in a very bad broken state since the system is missing most essential packages in its native architecture. This is not easily solved by apt-get -f install, as this would zap most of the existing system . We let apt-get --download-only -f install download the packages that it wants to replace with its amd64 counterparts and their dependencies, and, again, use dpkg --install *.deb to install them.

$ sudo apt-get --download-only -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  cpio gettext-base:i386 klibc-utils libapt-inst1.5 libasprintf0c2:i386
  libcurl3-gnutls:i386 libfreetype6:i386 libfuse2:i386 libklibc
  libreadline5:i386
Use  apt-get autoremove  to remove them.
The following extra packages will be installed:
  acpid aide bsd-mailx cpio cron curl daemon dctrl-tools exim4-base
  exim4-daemon-light initscripts insserv kbd klibc-utils less libbsd0
  libcomerr2 libcurl3 libdb5.1 libexpat1 libgcrypt11 libgdbm3 libgnutls26
  libgpg-error0 libgssapi-krb5-2 libidn11 libk5crypto3 libkeyutils1 libklibc
  libkmod2 libkrb5-3 libkrb5support0 libldap-2.4-2 liblockfile1 libncursesw5
  libnewt0.52 libp11-kit0 libpam-modules libpam0g libpcre3 libpopt0
  libreadline6 librtmp0 libsasl2-2 libslang2 libsqlite3-0 libssh2-1
  libssl1.0.0 libtasn1-3 libterm-readkey-perl libtinfo5 openssl perl
  perl-base python2.7 python2.7-minimal sysvinit-utils whiptail
Suggested packages:
  libarchive1 anacron logrotate checksecurity debtags mail-reader eximon4
  exim4-doc-html exim4-doc-info spf-tools-perl swaks bootchart2 rng-tools
  krb5-doc krb5-user libpam-doc perl-doc make python2.7-doc binutils
  binfmt-support bootlogd sash
Recommended packages:
  exim4 postfix mail-transport-agent psmisc mailx ca-certificates
  krb5-locales libgpm2 libfribidi0 libsasl2-modules libpng12-0
The following packages will be REMOVED:
  acpid:i386 aide:i386 aptitude:i386 bsd-mailx:i386 cpio:i386 cron:i386
  curl:i386 daemon:i386 dctrl-tools:i386 debsecan dmsetup:i386 e2fsprogs:i386
  exim4-base:i386 exim4-daemon-light:i386 git:i386 grub-common:i386
  grub-pc:i386 grub-pc-bin:i386 grub2-common:i386 ifupdown:i386
  initramfs-tools initscripts:i386 insserv:i386 ippl:i386 jed:i386 kbd:i386
  klibc-utils:i386 less:i386 libdevmapper-event1.02.1:i386
  libdevmapper1.02.1:i386 libklibc:i386 libnewt0.52:i386
  libparted0debian1:i386 libterm-readkey-perl:i386 lsof:i386 lvm2:i386
  molly-guard ntp:i386 openssh-server:i386 openssl:i386 parted:i386 perl:i386
  perl-base:i386 procps:i386 python-apt:i386 python2.7:i386
  python2.7-minimal:i386 rsyslog:i386 sysvinit:i386 sysvinit-utils:i386
  udev:i386 util-linux:i386 whiptail:i386
The following NEW packages will be installed:
  acpid aide bsd-mailx cpio cron curl daemon dctrl-tools exim4-base
  exim4-daemon-light initscripts insserv kbd klibc-utils less libbsd0
  libcomerr2 libcurl3 libdb5.1 libexpat1 libgcrypt11 libgdbm3 libgnutls26
  libgpg-error0 libgssapi-krb5-2 libidn11 libk5crypto3 libkeyutils1 libklibc
  libkmod2 libkrb5-3 libkrb5support0 libldap-2.4-2 liblockfile1 libncursesw5
  libnewt0.52 libp11-kit0 libpam-modules libpam0g libpcre3 libpopt0
  libreadline6 librtmp0 libsasl2-2 libslang2 libsqlite3-0 libssh2-1
  libssl1.0.0 libtasn1-3 libterm-readkey-perl libtinfo5 openssl perl
  perl-base python2.7 python2.7-minimal sysvinit-utils whiptail
0 upgraded, 58 newly installed, 53 to remove and 0 not upgraded.
Need to get 23,4 MB of archives.
After this operation, 17,5 MB disk space will be freed.
Do you want to continue [Y/n]? Y
Get:1 http://security.debian.org/debian-security/ wheezy/updates/main libgnutls26 amd64 2.12.20-7 [619 kB]
( )
Fetched 23,4 MB in 11s (2.038 kB/s)                                           
Download complete and in download only mode
$ sudo dpkg --install /var/cache/apt/archives/*.deb
(Reading database ... 16905 files and directories currently installed.)
Preparing to replace acpid 1:2.0.16-1+deb7u1 (using .../acpid_1%3a2.0.16-1+deb7u1_amd64.deb) ...
( )
Selecting previously unselected package libpam-modules:amd64.
dpkg: regarding .../libpam-modules_1.1.3-7.1_amd64.deb containing libpam-modules:amd64, pre-dependency problem:
 libpam-modules pre-depends on libdb5.1
  libdb5.1:amd64 is unpacked, but has never been configured.
dpkg: error processing /var/cache/apt/archives/libpam-modules_1.1.3-7.1_amd64.deb (--install):
 pre-dependency problem - not installing libpam-modules:amd64
Selecting previously unselected package libpcre3:amd64.
Unpacking libpcre3:amd64 (from .../libpcre3_1%3a8.30-5_amd64.deb) ...
( )
Errors were encountered while processing:
 /var/cache/apt/archives/libpam-modules_1.1.3-7.1_amd64.deb
$ sudo dpkg --install /var/cache/apt/archives/libpam-modules_1.1.3-7.1_amd64.deb
(Reading database ... 17007 files and directories currently installed.)
Unpacking libpam-modules:amd64 (from .../libpam-modules_1.1.3-7.1_amd64.deb) ...
Setting up libpam-modules:amd64 (1.1.3-7.1) ...
$ sudo apt-get clean
$ dpkg --print-architecture
amd64
$ dpkg --print-foreign-architectures
i386
$ dpkg --list   grep i386   wc -l
151
$ dpkg --list   grep amd64   wc -l
66
$ 

The concluding apt-get clean prevents packages from being reinstalled again in the next dpkg --install *.deb run. Now, the apt resolver is kind of fixed, but apt will still want to remove most of the system. I haven t found a way to get out of this mess short of allowing apt to de-install those packages and to manually install their amd64 counterparts later. First, we see what apt wants to do, and paste the The following packages will be removed part of apt-get s output into a sed expression which fixes the architecture strings for us so that we can use the resulting file as input for the following installation procedure. After that, we let apt-get do what itself considers a bad thing and requires us to type a consent sentence. The key is that this deinstallation of essential packages breaks the system, but not apt and dpkg, which allows us to un-break the system afterwards.

$ sudo apt-get -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  gettext-base:i386 libapt-inst1.5 libasprintf0c2:i386 libcurl3-gnutls:i386
  libfreetype6:i386 libfuse2:i386 libreadline5:i386
Use  apt-get autoremove  to remove them.
The following extra packages will be installed:
  sysvinit-utils:i386
Suggested packages:
  bootlogd:i386 sash:i386
The following packages will be REMOVED:
  adduser aide-common aptitude:i386 bsd-mailx console-common console-data
  console-log cron curl debian-goodies debsecan dmsetup:i386 e2fsprogs:i386
  exim4-base exim4-config exim4-daemon-light git:i386 grub-common:i386
  grub-pc:i386 grub-pc-bin:i386 grub2-common:i386 ifupdown:i386
  initramfs-tools initscripts ippl:i386 jed:i386 libcurl3
  libdevmapper-event1.02.1:i386 libdevmapper1.02.1:i386 liberror-perl
  libfile-find-rule-perl libnumber-compare-perl libparted0debian1:i386
  libswitch-perl libterm-readkey-perl libterm-readline-perl-perl
  libtext-glob-perl libtimedate-perl locales lsof:i386 lvm2:i386 molly-guard
  ntp:i386 openssh-client:i386 openssh-server:i386 openssl openssl-blacklist
  openssl-blacklist-extra parted:i386 perl perl-modules procps:i386
  python-apt:i386 rsyslog:i386 stow sysv-rc sysvinit:i386 sysvinit-utils
  tzdata ucf udev:i386 util-linux:i386
The following NEW packages will be installed:
  sysvinit-utils:i386
WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
  e2fsprogs:i386 util-linux:i386 (due to e2fsprogs:i386) sysvinit:i386
  sysvinit-utils tzdata (due to util-linux:i386)
0 upgraded, 1 newly installed, 62 to remove and 0 not upgraded.
Need to get 97,1 kB of archives.
After this operation, 131 MB disk space will be freed.
You are about to do something potentially harmful.
To continue type in the phrase  Yes, do as I say!  ? (abort with Ctrl-C)
$ sed  s/:i386//g  > removedpkg (paste  The following packages will be REMOVED:  stanza here and
type Ctrl-D afterwards)
  adduser aide-common aptitude:i386 bsd-mailx console-common console-data
  console-log cron curl debian-goodies debsecan dmsetup:i386 e2fsprogs:i386
  exim4-base exim4-config exim4-daemon-light git:i386 grub-common:i386
  grub-pc:i386 grub-pc-bin:i386 grub2-common:i386 ifupdown:i386
  initramfs-tools initscripts ippl:i386 jed:i386 libcurl3
  libdevmapper-event1.02.1:i386 libdevmapper1.02.1:i386 liberror-perl
  libfile-find-rule-perl libnumber-compare-perl libparted0debian1:i386
  libswitch-perl libterm-readkey-perl libterm-readline-perl-perl
  libtext-glob-perl libtimedate-perl locales lsof:i386 lvm2:i386 molly-guard
  ntp:i386 openssh-client:i386 openssh-server:i386 openssl openssl-blacklist
  openssl-blacklist-extra parted:i386 perl perl-modules procps:i386
  python-apt:i386 rsyslog:i386 stow sysv-rc sysvinit:i386 sysvinit-utils
  tzdata ucf udev:i386 util-linux:i386
$ cat removedpkg 
  adduser aide-common aptitude bsd-mailx console-common console-data
  console-log cron curl debian-goodies debsecan dmsetup e2fsprogs
  exim4-base exim4-config exim4-daemon-light git grub-common
  grub-pc grub-pc-bin grub2-common ifupdown
  initramfs-tools initscripts ippl jed libcurl3
  libdevmapper-event1.02.1 libdevmapper1.02.1 liberror-perl
  libfile-find-rule-perl libnumber-compare-perl libparted0debian1
  libswitch-perl libterm-readkey-perl libterm-readline-perl-perl
  libtext-glob-perl libtimedate-perl locales lsof lvm2 molly-guard
  ntp openssh-client openssh-server openssl openssl-blacklist
  openssl-blacklist-extra parted perl perl-modules procps
  python-apt rsyslog stow sysv-rc sysvinit sysvinit-utils
  tzdata ucf udev util-linux
$ sudo apt-get -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  gettext-base:i386 libapt-inst1.5 libasprintf0c2:i386 libcurl3-gnutls:i386
  libfreetype6:i386 libfuse2:i386 libreadline5:i386
Use  apt-get autoremove  to remove them.
The following extra packages will be installed:
  sysvinit-utils:i386
Suggested packages:
  bootlogd:i386 sash:i386
The following packages will be REMOVED:
  adduser aide-common aptitude:i386 bsd-mailx console-common console-data
  console-log cron curl debian-goodies debsecan dmsetup:i386 e2fsprogs:i386
  exim4-base exim4-config exim4-daemon-light git:i386 grub-common:i386
  grub-pc:i386 grub-pc-bin:i386 grub2-common:i386 ifupdown:i386
  initramfs-tools initscripts ippl:i386 jed:i386 libcurl3
  libdevmapper-event1.02.1:i386 libdevmapper1.02.1:i386 liberror-perl
  libfile-find-rule-perl libnumber-compare-perl libparted0debian1:i386
  libswitch-perl libterm-readkey-perl libterm-readline-perl-perl
  libtext-glob-perl libtimedate-perl locales lsof:i386 lvm2:i386 molly-guard
  ntp:i386 openssh-client:i386 openssh-server:i386 openssl openssl-blacklist
  openssl-blacklist-extra parted:i386 perl perl-modules procps:i386
  python-apt:i386 rsyslog:i386 stow sysv-rc sysvinit:i386 sysvinit-utils
  tzdata ucf udev:i386 util-linux:i386
The following NEW packages will be installed:
  sysvinit-utils:i386
WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
  e2fsprogs:i386 util-linux:i386 (due to e2fsprogs:i386) sysvinit:i386
  sysvinit-utils tzdata (due to util-linux:i386)
0 upgraded, 1 newly installed, 62 to remove and 0 not upgraded.
Need to get 97,1 kB of archives.
After this operation, 131 MB disk space will be freed.
You are about to do something potentially harmful.
To continue type in the phrase  Yes, do as I say! 
 ?] Yes, do as I say! (actually type this!)
Get:1 http://debian.debian.zugschlus.de/debian/ wheezy/main sysvinit-utils i386 2.88dsf-41 [97,1 kB]
Fetched 97,1 kB in 0s (687 kB/s)    
(Reading database ... 17050 files and directories currently installed.)
Removing aide-common ...
( )
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
Removing e2fsprogs ...
Removing sysv-rc ...
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
Removing sysvinit ...
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
Removing sysvinit-utils ...
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
Removing ucf ...
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
Removing libdevmapper1.02.1:i386 ...
Removing libswitch-perl ...
Removing perl ...
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
Removing dmsetup ...
Removing perl-modules ...
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
Removing util-linux ...
Removing tzdata ...
Processing triggers for mime-support ...
Selecting previously unselected package sysvinit-utils.
(Reading database ... 9802 files and directories currently installed.)
Unpacking sysvinit-utils (from .../sysvinit-utils_2.88dsf-41_i386.deb) ...
Setting up sysvinit-utils (2.88dsf-41) ...
$ sudo apt-get install $(cat removedpkg)
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libasprintf0c2:i386 libcurl3-gnutls:i386 libfreetype6:i386 libfuse2:i386
  libreadline5:i386
Use  apt-get autoremove  to remove them.
The following extra packages will be installed:
  e2fslibs gettext-base libapt-inst1.5 libasprintf0c2 libattr1 libblkid1
  libboost-iostreams1.49.0 libcap2 libcurl3-gnutls libcwidget3 libedit2
  libept1.4.12 libfreetype6 libfuse2 libgpm2 libncurses5 libopts25 libprocps0
  libreadline5 libsepol1 libsigc++-2.0-0c2a libss2 libudev0 libuuid1 libwrap0
  libxapian22 logrotate
Suggested packages:
  liblocale-gettext-perl tasksel debtags unicode-data anacron checksecurity
  popularity-contest xdg-utils zenity gpart e2fsck-static mail-reader eximon4
  exim4-doc-html exim4-doc-info spf-tools-perl swaks git-daemon-run
  git-daemon-sysvinit git-doc git-el git-arch git-cvs git-svn git-email
  git-gui gitk gitweb multiboot-doc grub-emu xorriso desktop-base
  isc-dhcp-client dhcp-client ppp rdnssd net-tools gpm libcwidget-dev fuse
  libparted0-dev libparted0-i18n xapian-tools ntp-doc ssh-askpass libpam-ssh
  keychain monkeysphere rssh ufw parted-doc perl-doc make libpod-plainer-perl
  python-apt-dbg python-gtk2 python-vte python-apt-doc rsyslog-mysql
  rsyslog-pgsql rsyslog-doc rsyslog-gnutls rsyslog-gssapi rsyslog-relp
  doc-base sysv-rc-conf bum bootlogd sash util-linux-locales dosfstools
Recommended packages:
  aptitude-doc-en aptitude-doc apt-xapian-index libparse-debianchangelog-perl
  exim4 postfix mail-transport-agent psmisc mailx patch rsync ssh-client
  os-prober busybox busybox-initramfs busybox-static ca-certificates
  uuid-runtime xauth ncurses-term iso-codes usbutils
The following packages will be REMOVED:
  gettext-base:i386 libboost-iostreams1.49.0:i386 libcwidget3:i386
  libept1.4.12:i386 libopts25:i386 libxapian22:i386 logrotate:i386
  sysvinit-utils:i386
The following NEW packages will be installed:
  adduser aide-common aptitude bsd-mailx console-common console-data
  console-log cron curl debian-goodies debsecan dmsetup e2fslibs e2fsprogs
  exim4-base exim4-config exim4-daemon-light gettext-base git grub-common
  grub-pc grub-pc-bin grub2-common ifupdown initramfs-tools initscripts ippl
  jed libapt-inst1.5 libasprintf0c2 libattr1 libblkid1
  libboost-iostreams1.49.0 libcap2 libcurl3 libcurl3-gnutls libcwidget3
  libdevmapper-event1.02.1 libdevmapper1.02.1 libedit2 libept1.4.12
  liberror-perl libfile-find-rule-perl libfreetype6 libfuse2 libgpm2
  libncurses5 libnumber-compare-perl libopts25 libparted0debian1 libprocps0
  libreadline5 libsepol1 libsigc++-2.0-0c2a libss2 libswitch-perl
  libterm-readkey-perl libterm-readline-perl-perl libtext-glob-perl
  libtimedate-perl libudev0 libuuid1 libwrap0 libxapian22 locales logrotate
  lsof lvm2 molly-guard ntp openssh-client openssh-server openssl
  openssl-blacklist openssl-blacklist-extra parted perl perl-modules procps
  python-apt rsyslog stow sysv-rc sysvinit sysvinit-utils tzdata ucf udev
  util-linux
WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
  sysvinit-utils:i386
0 upgraded, 89 newly installed, 8 to remove and 0 not upgraded.
Need to get 55.3 MB of archives.
After this operation, 134 MB of additional disk space will be used.
You are about to do something potentially harmful.
To continue type in the phrase  Yes, do as I say! 
 ?] Yes, do as I say! (actually type this!)
Get:1 http://debian.debian.zugschlus.de/debian/ wheezy/main e2fslibs amd64 1.42.5-1.1 [197 kB]
( )
Fetched 55.3 MB in 20s (2648 kB/s)                                            
perl: warning: Setting locale failed.
( )

This process may re-ask some debconf questions that you have already answered during initial system setup. I find it comforting to actually see that my answers were preserved, but you can always export DEBIAN_FRONTEND=noninteractive if you want installation to be silent. If you know how this step can be done more elegantly, please comment. We now, finally, have the resolver in a consistent state.

$ sudo apt-get -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libasprintf0c2:i386 libcurl3-gnutls:i386 libfreetype6:i386 libfuse2:i386
  libreadline5:i386
Use  apt-get autoremove  to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
$ dpkg --print-architecture
amd64
$ dpkg --print-foreign-architectures
i386
$ dpkg --list   grep i386   wc -l
121
$ dpkg --list   grep amd64   wc -l
119
$ 

Now we can force-crossgrade all i386 packages that are still left on the system. Again, we use the apt-get --download install, dpkg --install stunt as coreutils gets crossgraded at this step and dpkg is less than happy if it finds itself without /bin/rm. Remember, with --download, nothing is actually removed during this step even if apt-get claims to remove things. Apt will also complain that many of these packages are already installed. Those are libs that were pulled in previously. The first dpkg --install run in this step will complain about gazillions of dependency problems, which will all be resolved in concluding apt-get -f install after the second dpkg --install run, this time a manual one.

$ sudo apt-get --download-only install $(dpkg --list   awk  if( $1 ==  ii  && $4 ==
 i386  )   print $2     sed  s/:i386// )
Reading package lists... Done
Building dependency tree       
Reading state information... Done
( )
The following packages were automatically installed and are no longer required:
  libasprintf0c2:i386 libcurl3-gnutls:i386 libfreetype6:i386 libfuse2:i386
  libreadline5:i386
Use  apt-get autoremove  to remove them.
Suggested packages:
  powermgmt-base bash-doc bzip2-doc diffutils-doc wdiff mlocate locate
  gnupg-doc xloadimage imagemagick eog libpcsclite1 iproute-doc resolvconf
  avahi-autoipd nfs-common ncompress indent
Recommended packages:
  bsdmainutils gnupg-curl libatm1
The following packages will be REMOVED:
  anacron:i386 apt-utils:i386 base-files:i386 base-passwd:i386 bash:i386
( )
The following NEW packages will be installed:
  anacron apt-utils base-files base-passwd bash bsdutils busybox bzip2
( )
0 upgraded, 56 newly installed, 49 to remove and 0 not upgraded.
Need to get 24,1 MB of archives.
After this operation, 2.030 kB of additional disk space will be used.
Do you want to continue [Y/n]? y
Get:1 http://debian.debian.zugschlus.de/debian/ wheezy/main kmod amd64 9-3 [60,5 kB]
( )
Fetched 24,1 MB in 11s (2.120 kB/s)                                           
Download complete and in download only mode
$ sudo dpkg --install /var/cache/apt/archives/*.deb
(Reading database ... 17098 files and directories currently installed.)
Preparing to replace adduser 3.113+nmu3 (using .../adduser_3.113+nmu3_all.deb) ...
Unpacking replacement adduser ...
( )
Errors were encountered while processing:
 /var/cache/apt/archives/bash_4.2+dfsg-0.1_amd64.deb
 /var/cache/apt/archives/coreutils_8.13-3.5_amd64.deb
 initscripts
 lvm2
 procps
 rsyslog
 sysvinit
 sysv-rc
 udev
 util-linux
 aide-common
 dmsetup
 e2fsprogs
 ifupdown
 initramfs-tools
 libdevmapper1.02.1:amd64
 libdevmapper-event1.02.1:amd64
 libparted0debian1:amd64
 molly-guard
 openssh-server
 parted
 grub-common
 grub-pc
 grub-pc-bin
 grub2-common
$ sudo dpkg --install /var/cache/apt/archives/bash_4.2+dfsg-0.1_amd64.deb
/var/cache/apt/archives/coreutils_8.13-3.5_amd64.deb
(Reading database ... 17129 files and directories currently installed.)
Preparing to replace bash 4.2+dfsg-0.1 (using .../bash_4.2+dfsg-0.1_amd64.deb) ...
Unpacking replacement bash ...
Preparing to replace coreutils 8.13-3.5 (using .../coreutils_8.13-3.5_amd64.deb) ...
Unpacking replacement coreutils ...
Setting up bash (4.2+dfsg-0.1) ...
update-alternatives: using /usr/share/man/man7/bash-builtins.7.gz to provide /usr/share/man/man7/builtins.7.gz
(builtins.7.gz) in auto mode
Setting up coreutils (8.13-3.5) ...
$ sudo apt-get -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  libasprintf0c2:i386 libcurl3-gnutls:i386 libfreetype6:i386 libfuse2:i386
  libreadline5:i386
Use  apt-get autoremove  to remove them.
The following extra packages will be installed:
  sysvinit-utils
Suggested packages:
  bootlogd sash
The following packages will be REMOVED:
  libslang2-modules:i386 sysvinit-utils:i386
The following NEW packages will be installed:
  sysvinit-utils
WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
  sysvinit-utils:i386
0 upgraded, 1 newly installed, 2 to remove and 0 not upgraded.
23 not fully installed or removed.
Need to get 0 B/99,5 kB of archives.
After this operation, 328 kB disk space will be freed.
You are about to do something potentially harmful.
To continue type in the phrase  Yes, do as I say! 
 ?] Yes, do as I say! (actually type this!)
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
(Reading database ... 17129 files and directories currently installed.)
Removing sysvinit-utils ...
Selecting previously unselected package sysvinit-utils.
(Reading database ... 17105 files and directories currently installed.)
Unpacking sysvinit-utils (from .../sysvinit-utils_2.88dsf-41_amd64.deb) ...
Setting up sysvinit-utils (2.88dsf-41) ...
Setting up sysv-rc (2.88dsf-41) ...
Setting up initscripts (2.88dsf-41) ...
Setting up util-linux (2.20.1-5.3) ...
Setting up e2fsprogs (1.42.5-1.1) ...
Setting up sysvinit (2.88dsf-41) ...
sysvinit: restarting... done.
(Reading database ... 17129 files and directories currently installed.)
Removing libslang2-modules:i386 ...
Setting up ifupdown (0.7.8) ...
Setting up procps (1:3.3.3-3) ...
[ ok ] Setting kernel variables ... /etc/sysctl.conf...done.
Setting up udev (175-7.2) ...
[ ok ] Stopping the hotplug events dispatcher: udevd.
[ ok ] Starting the hotplug events dispatcher: udevd.
update-initramfs: deferring update (trigger activated)
Setting up rsyslog (5.8.11-3) ...
[ ok ] Stopping enhanced syslogd: rsyslogd.
[ ok ] Starting enhanced syslogd: rsyslogd.
Setting up aide-common (0.15.1-8) ...
Setting up initramfs-tools (0.109.1) ...
update-initramfs: deferring update (trigger activated)
Setting up openssh-server (1:6.0p1-4) ...
[ ok ] Restarting OpenBSD Secure Shell server: sshd.
Setting up molly-guard (0.4.5-1) ...
Setting up libdevmapper1.02.1:amd64 (2:1.02.74-7) ...
Setting up libdevmapper-event1.02.1:amd64 (2:1.02.74-7) ...
Setting up libparted0debian1:amd64 (2.3-12) ...
Setting up grub-common (1.99-27+deb7u1) ...
Setting up grub2-common (1.99-27+deb7u1) ...
Setting up grub-pc-bin (1.99-27+deb7u1) ...
Setting up grub-pc (1.99-27+deb7u1) ...
Installation finished. No error reported.
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-3.9.4-zgsrv2008064
Found initrd image: /boot/initrd.img-3.9.4-zgsrv2008064
done
Setting up parted (2.3-12) ...
Setting up dmsetup (2:1.02.74-7) ...
update-initramfs: deferring update (trigger activated)
Setting up lvm2 (2.02.95-7) ...
[ ok ] Setting up LVM Volume Groups...done.
update-initramfs: deferring update (trigger activated)
Processing triggers for initramfs-tools ...
update-initramfs: Generating /boot/initrd.img-3.9.4-zgsrv2008064
Found 12 processes using old versions of upgraded files
( )
$ sudo apt-get -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libasprintf0c2:i386 libcurl3-gnutls:i386 libfreetype6:i386 libfuse2:i386
  libreadline5:i386
Use  apt-get autoremove  to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
$ dpkg --print-architecture
amd64
$ dpkg --print-foreign-architectures
i386
$ dpkg --list   grep i386   wc -l
72
$ dpkg --list   grep amd64   wc -l
175
$ 

We have completed the actual crossgrade. All non-multiarch packages are now amd64. The resolver is now fine as well. At this current state, all i386 packages that are still installed should be unneeded libs. We can safely nuke them. Careful minds may inspect the list of packages to be removed for important or non-lib items. After completion of the command, a second call of the same command verifies that we were successful.

$ sudo apt-get remove $(dpkg --list   awk  if( $1 ==  ii  && $4 ==  i386  )   print $2
  )
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  e2fslibs:i386 gcc-4.7-base:i386 libacl1:i386 libapt-inst1.5:i386
  libapt-pkg4.12:i386 libasprintf0c2:i386 libattr1:i386 libblkid1:i386
  libbsd0:i386 libbz2-1.0:i386 libc6:i386 libcap2:i386 libcomerr2:i386
  libcurl3:i386 libcurl3-gnutls:i386 libdb5.1:i386 libedit2:i386
  libexpat1:i386 libfreetype6:i386 libfuse2:i386 libgcc1:i386
  libgcrypt11:i386 libgdbm3:i386 libgnutls26:i386 libgpg-error0:i386
  libgpm2:i386 libgssapi-krb5-2:i386 libidn11:i386 libk5crypto3:i386
  libkeyutils1:i386 libkmod2:i386 libkrb5-3:i386 libkrb5support0:i386
  libldap-2.4-2:i386 liblockfile1:i386 liblzma5:i386 libmagic1:i386
  libncurses5:i386 libncursesw5:i386 libp11-kit0:i386 libpam-modules:i386
  libpam0g:i386 libpci3:i386 libpcre3:i386 libpng12-0:i386 libpopt0:i386
  libprocps0:i386 libreadline5:i386 libreadline6:i386 librtmp0:i386
  libsasl2-2:i386 libselinux1:i386 libsemanage1:i386 libsepol1:i386
  libsigc++-2.0-0c2a:i386 libslang2:i386 libsqlite3-0:i386 libss2:i386
  libssh2-1:i386 libssl1.0.0:i386 libstdc++6:i386 libtasn1-3:i386
  libtinfo5:i386 libudev0:i386 libusb-0.1-4:i386 libustr-1.0-1:i386
  libuuid1:i386 libwrap0:i386 linux-image-3.9.4-zgsrv2008064:i386 zlib1g:i386
0 upgraded, 0 newly installed, 70 to remove and 0 not upgraded.
After this operation, 67,5 MB disk space will be freed.
Do you want to continue [Y/n]? y
(Reading database ... 17111 files and directories currently installed.)
Removing e2fslibs:i386 ...
( )
$ sudo apt-get remove $(dpkg --list   awk  if( $1 ==  ii  && $4 ==  i386  )   print $2
  )
Reading package lists... Done
Building dependency tree       
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
$ dpkg --print-architecture
amd64
$ dpkg --print-foreign-architectures
i386
$ dpkg --list   grep i386   wc -l
70
$ dpkg --list   grep amd64   wc -l
175
$ 

To finally make the packages really vanish from the system, we purge all packages that dpkg --list still reports in state rc , which means removed, configuration still present . Thankfully, dpkg seems to handle the case where a dpkg-conffile belongs to a transitioned package gracefully.

$ sudo dpkg --purge $(dpkg --list   awk  if ($1 ==  rc )   print $2  )
(Reading database ... 15868 files and directories currently installed.)
Removing e2fslibs:i386 ...
Purging configuration files for e2fslibs:i386 ...
( )
$ dpkg --print-architecture
amd64
$ dpkg --print-foreign-architectures
i386
$ dpkg --list   grep i386   wc -l
0
$ dpkg --list   grep amd64   wc -l
175
$ 

We now verify that all packages we have are either arch all or arch amd64.

$ dpkg --list   grep -v  \(amd64 all\) 
Desired=Unknown/Install/Remove/Purge/Hold
  Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
 / Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
 / Name                            Version                   Architecture Description
+++-===============================-=========================-============-========================================================================

As I found myself suddenly without e2fsprogs during one of my experiments, I check for e2fsprogs being present. If this package is missing, you ll only find out after rebooting.

$ dpkg --list e2fsprogs
Desired=Unknown/Install/Remove/Purge/Hold
  Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
 / Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
 / Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  e2fsprogs      1.42.5-1.1   amd64        ext2/ext3/ext4 file system utilit

My locally built amd64 kernel with 32 bit support was an arch i386 package as well, so it got zapped in the migration process. Replace it with a real amd64 kernel. This step will vary if you use Debian kernels. Do not forget the firmwares.

$ sudo apt-get -t wheezy-zg-experimental install kernel-image-zgserver linux-firmware-image
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  cpio initramfs-tools kernel-image-zgsrv20080 klibc-utils libklibc
  linux-image-3.9.4-zgsrv20080
Suggested packages:
  libarchive1
The following NEW packages will be installed:
  cpio initramfs-tools kernel-image-zgserver kernel-image-zgsrv20080
  klibc-utils libklibc linux-firmware-image linux-image-3.9.4-zgsrv20080
0 upgraded, 8 newly installed, 0 to remove and 0 not upgraded.
Need to get 11,2 MB of archives.
After this operation, 30,5 MB of additional disk space will be used.
Do you want to continue [Y/n]? 
Get:1 http://zg20110.debian.zugschlus.de/ wheezy-zg-experimental/main linux-image-3.9.4-zgsrv20080 amd64
3.9.4.20130605.0 [10,4 MB]
( )

I now make sure that the initrd I have built contains all file system modules and is of the right architecture:

$ cd /tmp
$ < /boot/initrd.img-3.9.4-zgsrv20080 gunzip   cpio -i
32053 blocks
$ find . -name  *ext*.ko 
./lib/modules/3.9.4-zgsrv20080/kernel/fs/ext3/ext3.ko
./lib/modules/3.9.4-zgsrv20080/kernel/fs/ext2/ext2.ko
./lib/modules/3.9.4-zgsrv20080/kernel/fs/ext4/ext4.ko
$ file ./lib/modules/3.9.4-zgsrv20080/kernel/fs/ext4/ext4.ko
./lib/modules/3.9.4-zgsrv20080/kernel/fs/ext4/ext4.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV),
BuildID[sha1]=0x259a642448805c9597932ee25c62c8e6a00a32ad, not stripped
$ cd

As the last step, remove i386 as an architecture and reboot into the newly migrated system. We verify that the system considers itself amd64 with no foreign architectures.

$ sudo dpkg --remove-architecture i386
$ dpkg --print-architecture
amd64
$ 
$ dpkg --print-foreign-architectures
$ dpkg --list   grep i386   wc -l
0
$ dpkg --list   grep amd64   wc -l
181
$ sudo shutdown -r now
Broadcast message from root@wheezyarch (pts/0) (Wed Jun 12 16:09:24 2013):
The system is going down for reboot NOW!
$ Connection to wheezyarch.zugschlus.de closed by remote host.
Connection to wheezyarch.zugschlus.de closed.
$ ssh wheezyarch.zugschlus.de
Linux wheezyarch 3.9.4-zgsrv20080 #2 SMP PREEMPT Wed Jun 5 13:12:04 UTC 2013 x86_64
$ dpkg --print-architecture
amd64
$

We now check whether we have lost any packages in this process:

$ sudo dpkg --get-selections   grep install   awk  print $1    sed  s/:.*//;s/$/:amd64
install/;  > get-selections.post
$ wc -l get-selections.*
  234 get-selections.post
  226 get-selections.pre
$ diff --unified=0 get-selections.pre get-selections.post 
--- get-selections.pre  2013-06-12 19:47:40.278176000 +0200
+++ get-selections.post 2013-06-12 20:28:51.389874000 +0200
@@ -22,0 +23 @@
+cpio:amd64 install
@@ -58,0 +60 @@
+initramfs-tools:amd64 install
@@ -68,0 +71,3 @@
+kernel-image-zgserver:amd64 install
+kernel-image-zgsrv20080:amd64 install
+klibc-utils:amd64 install
@@ -108,0 +114 @@
+libklibc:amd64 install
@@ -163,0 +170,2 @@
+linux-firmware-image:amd64 install
+linux-image-3.9.4-zgsrv20080:amd64 install

So we have not actually lost things, but we have gained the pure amd64 kernel and the packages that are needed to build a proper initrd. These were missing from the initial install of the i386 test VM. The downside of the process is that we have lost most of the automatically installed markers in aptitude which make sure to remove a library once nothing depends on it any more. If you make a point in having those markers correctly set, as I do, this is tedious work to set it on every package in the aptitude curses interface, making aptitude suggest removing the entire system, and then again remove it from the packages you actually intend to keep. This should not result in anything being removed, otherwise you have goofed previously. All Done! I am now off perfecting my upgrade process so that the i386 lenny and squeeze machines can first be moved to wheezy and then to amd64.

I have published PowerDNS version 3.1-1.0 on https://ivanova.notwork.de/~mh/debian/pdns/ This is a preliminary package and a release candidate to be 3.1-2 in Debian. If you re interested in PowerDNS on Debian, please test this package. I plan to upload next week. This package will vanish from the web server once the package is visible in Debian.

Eight days ago, I uploaded atop 1.26-1 to DELAYED/8, listing me as new maintainer. This means that the package has in the mean time appeared in unstable, and I hope that it ll swiftly migrate to testing.

As grub is now working on the loongson 2e boxes as well (thanks to phcoder and Colin Watson), it is time to move the buildds running at my home to a data center (previously we couldn't remote manage the kernels / boot flags without VGA console, which means no data center usage). Also one of the boxes could be converted to a porter machine, so that we could get an mipsel porter box again. The machines are delivered only with 256M of ram, which is a bit too less for usage. Thanks to Zugschlus (Marc Haber) I got 1g ram for both machines going to Vienna (one buildd, one porter box), and thanks to Robert Grimm a 160g harddisk to replace the build-in 40g in the porter box (the buildd can cope with 40g fine). The additional ram modules and hard disk are visible on the following picture.
eysler.debian.org is now DSAed, online and building packages (including autosigning). I will shutdown monteverdi.ayous.org which is still at my place in the next days, reinstall the system and send it to Darmstadt as another buildd for data center hosting there. After that happens all mipsel buildds are DSAed as it should be, and are running in a data center and not via some DSL line. (In case you're looking for hardware at your place, there are a couple of loongson 2f-systems available to buy. 2f is the successor cpu of 2e. Some 2f systems get delivered with Debian installed on it, see e.g. http://www.aliexpress.com/product-gs/290632002-Yeeloong-8101-Laptop-10-inch-Debian-EU-Adaptor-version--wholesalers.html. However, for buildd usage, the 2e are fine as well, and we got them sponsored some time ago.)

Dear Lazyweb, for a long time I have been using iproute2 s label feature to assign arbitrary labels to IP addresses configured on Interfaces:

40: int152@dotqa: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 00:25:b3:01:e5:6c brd ff:ff:ff:ff:ff:ff
inet 10.1.152.254/24 brd 10.1.152.255 scope global int152:98fe8

Recently, this has shown to at least confuse both isc-dhcp-relay (#617258) and dhcp-helper (#617264). As I have never seen interfaces labels used outside my firewalls (which happen to use ifupdown-scripts-zg2 (Debian PTS)) and ifupdown s rather twisted handling of multiple IP addresses per interface (using Alias Interfaces), I d like to know whether my usage is a legitimate one and whether there are other uses for interface labels. At the moment, I m tempted to remove label support from ifupdown-scripts-zg2 in the next release, or to make it optional. Please comment if you have an opinion.

It is well known that apt has an issue when it comes to resolving circular dependencies. Therefore, Debian bug reporters have set out to eradicate circular dependencies from the archive. This does, however, add significant bloat to the actual packages, and I am questioning why this is really necessary. For example, aide. aide has an aide-common package, depending on aide aide-binary. aide-binary is a virtual package by aide, aide-dynamic, and aide-xen. All three providers of aide-binary depend on aide-common again, which is a circular dependency. If I now follow the request posed in #545852 and remove aide-common s dependency on aide aide-binary, I ll have to cater for the case of the aide-binary bein absent in each and every helper script included in aide-common, which would drive the script s complexity up to a new level. I really hate that idea. Is it really necessary to work around apt s shortcomings by eliminating the case that apt doesn t handle well from the archive, and driving up other packages complexity (and the possibility of bugs in this added complexity)?

In the last few days, I found the time to spend some with KVM and libvirt. Unfortunately, there is a subject that I haven t yet found a satisfying solution: Naming of block devices in guest instances. This is surely a common issue, but solutions are rare. Neither an article on Usenet (in German) nor the German version of this blog article has found solutions for the main question. I should have written this in English in the first place and am thus translating from German to english, hoping that there will be some answers and suggestions. KVM is quite inflexible when it coms to configure block devices. It is possible to define on the host, which files or whole devices from the host should be visible in the guest. The documentation suggests that devices should be brought into the guest with the virtio model, which needs suppport in the guest kernel. Importing a device as emulated ATA or SCSI device brings a performance penalty. The devices brought into the guest via virtio appear in the guest s dev as /dev/vd<x> and do also have their corresponding entries in /dev/disk/by-uuid and /dev/disk/by-path. The vd<x> node is simply numbered in consecutive order as hd<x> and sd<x>. /dev/disk/by-uuid is the correct UUID of the file system found on the device, at least if it s a block device partitioned inside the guest and formatted with ext3 (I didn t try anything else yet). The terminology of the /dev/disk/by-path node is not yet understood, and I am somewhat reluctant to assume the PCI paths of emulated hardware as stable. Looks like I am forced to configure inside the guest when I have changed the host s mass storage system (for example, after migration to a different file system or after new block devices have been added) to accommodate for the new order of the /dev/vd<x> or to have the UUID correctly configured. This is like calling for configuration errors. This is a reincarnation of the same issue that has been killed by LVM on Linuxes running on bare metal : The block device itself has a mnemonic name which is constant even in migration and copy actions. This works without file-system specific stuff like UUID or label, and wouldn t even be possible with a UUID (which won t be unique in this case). The mnemonic names of LVs are also available when the data is directly written to the raw device such as a CNFS buffer of a news server. I would love to have something like a paravirtualized device mapper interface which would allow to say in the host s configuration which of the host s LVs should be visible in the guest with which name in /dev/mapper. That way, the guest s configuration could remain stable during data wrangling operations on the host. One solution is to have a single LV for each guest and import this LV as /dev/vda into the guest. /dev/vda would then be partitioned like a real disk and have its own LVM installation. This would however need kpartx if one wants to access the data from the host, and loses flexibility when resizing file systems. These issues appear in every installation of KVM virtualization, and I would expect that there are gazillions of other possible solutions. I am interested in knowing how other people have tackled this issue and whether there are more possiblity that I haven t thought of before. Maybe there is a solution that doesn t leave me with the feeling of having implemented something ugly. Does the interface between the host s KVM and the guest s device mapper that I have been dreaming of maybe exist. Or is there any other possibility of configuring the device node s name in the guest Linux on the host side?

Steinar H. Gunderson, sesse, has written an interesting article about TCP performance. I didn t find your blog s comment function, so I am commenting with a trackback. (note: which didn t work either, The auto-discovered trackback URI does not match our target URI ) I frequently use mobile internet, using various of the German GSM/UMTS network operators, out of a moving train. As you have written, this frequently causes packet loss which is not only not caused by congestion, but sends the congestion avoidance algorithms on a false path. For example, when the train passes through the 3575 m long Distelrasentunnel between Frankfurt and Fulda, my network link is broken for like two minutes. Passing through other parts of Germany sometimes gives me a ping response of hundreds of thousands of microseconds by virtue of the rather huge send buffer the UMTS equipment has. In these circumstances, ssh sessions frequently take tens of minutes to notice that the network is back before the session is useable again. Frequently, it doesn t come up again before an hour has passed. And I have not found a way to work myself around this. Can you explain what s happening here, and do you have any ideas to solve the issue?

Dear Lazyweb, I have just found out that ksynaptics has stopped working against the X in unstable, and that ksynaptics is not even in lenny, let alone in current testing and/or unstable. This currently leaves me with an unconfigured touchpad, which is a major nuisance since I have gotten accustomed to tap-dragging and touchpad border scrolling. xserver-xorg-input-synaptics README.Debian dates back to 2004, so I suspect that the information given there in does not any more apply to today s configfile-less X. So, dear lazyweb, how do I get my touchpad back into the more intelligent mode? Clickable configuration preferred.

During this past weekend I tried to get Windows XP, Windows 7, OSX, Debian, and Ubuntu all on one machine. Due to some mishaps, driver oddities with Windows 7, and other things, I had to install certain operating systems multiple times. Now, the problem is, each operating system has it's own peculiar way of loading. Linux uses GRUB, Windows XP has NTLDR, 7 has the Windows Boot Manager, etc. Most of these operating systems also decide it would be helpful for the end-user that when his computer reboots, it can actually load the new OS. That's great, but I already have a bootloader.

I have a partition at /dev/sda5 which contains a GRUB installation. That's supposed to handle every OS's own method of booting, including the Linux installations which have their own GRUBs on their own partitions. But when you decide to overwrite the MBR, I don't get your nice and shiny OS. A good number of OSes use an MBR that won't boot an extended partition anyway. So instead of seeing your bright and shiny new OS, I see "NTLDR is missing". Or the ever-so-helpful "boot0: error". I'm -lucky- if I even get an OS at all, much less one that can load the grub shell so I can fix what you broke.

I understand having 3 GRUB installations plus a whole bunch of other bootloaders isn't exactly a standard use case. Most people don't go beyond dual-boot. But please, for the love of all that is holy, please just put a little checkmark somewhere,"Do not replace MBR". Ubuntu and Debian are the only OSes that actually give you that option in some form.

Or better yet, let's all just standardize on GRUB. Sure, it's not perfect, but it's as flexible as you can get.

This is a rant. A rant which goes to the grub maintainers, and one that could go nearly identically to many people in the KDE environment or many other open source projects. I really like grub. I really like grub 0.97 despite that it s been unmaintained for years and not booting on two of my important machines. I should like grub 2 because its configuration looks more straightforward and for its better features - direct booting of .iso images, from LVM and RAID. But actually, I have learned to hate grub 2 since it is not finished and badly documented, and that its existence is already being used as an excuse for grub 0 s development having stopped years ago (and it being renamed to grub-legacy to clearly show that it s the unloved child) - and things looks like this is not going to change any time soon. grub 0.97 is the de facto standard in booting Linux, its most prevalent advantages probably being its capability of reading filesystems and its command shell, which allows booting kernels and other software that one wasn t planning to boot when the system was shut down. There is a small fraction of people who still use lilo despite its disadvantages, for example on achitectures that are not supported by grub, if one wants to have the boot FS on LVM, if one just prefers lilo for non-technical reasons or on systems that grub doesn t work on. The last three systems, by the way, that I still run lilo on are Pyramid pizzaboxes with supermicro-made mainboards from 2002, which just refuse to boot if grub is used and the vendor just says that grub 0.97 is still beta and that they do not support grub on their systems: Use Lilo. Grub s last upstream release, 0.97, dates back to 2005, and upstream has stopped developing this grub release years ago. grub 2, the new way to go, had its first release in 2003, and the last stable release that people are supposed to use dates back to februar 2008. grub 2 is clearly a step in the right direction, but there are still bugs, and the most prevalent show stopper against grub 2 is the almost complete absence of any documentation. I have, for example, not yet found good documentation about the new config file format, which is a completely different deal from what we are used to with grub 0. The commands manual pages do little more than list the (obvious!) command line options without giving much more information about what the actually do, let alone usage examples. The documentation (which used to be really excellent for grub 0), has been moved to a wiki, which has some useful content but cannot keep up with the excellent documentation that grub 0 used to have. Additionally, grub2 does not seem to have a grub shell which can be called from within the host system, which makes installing grub 2 in exotic situations much harder than it used to be with grub 0. I am also quite disappointed that despite the grub upstream web pages claiming that grub 2 is under active development, the last release is more than fifteen months in the past while there are still so many shortcomings. The Debian maintainers of grub 2 are regularly packaging grub 2 subversion snapshots, which is a good idea in face of upstream s release policy which is quite different from what s being told in the wiki. I am, however, reluctant to rely on a development snapshot of a software to boot productive systems. Guys, you cannot declare a widely used, well-documented software legacy when the replacement is still in a nearly unuseable shape and the docs are hidden in the helpful files named .c and .h. This is simply unacceptable. It s a drama that the docs are still in such a pitiful shape fifteen months after your last release. But, grub is not the only free software project that so blatantly scores developer fun over the needs of the end user. I know that it is a boring task to fix bugs and to work around design shortcomings in the existing version of software, and that it is much more fun to hack away on new designs and new code, but dropping old versions before the new version is up to par with the old one is annoying the users and driving people away to other software packages. This applies to grub, a quite low-level program, but also to larger and more complex projects like KDE who has recently pulled the same stunt by dropping development and support for KDE 3.5 while KDE 4 is still missing many many features and KDE 3.5 s stablity, thus driving people (back) to GNOME or alternative desktop environments. This we give a damn about people who actually use our software, our compilers run without users as well attitude is a show of disrespect for the user base and makes me very sad as this does indeed show that free software is in no way superior to its commercial counterparts. Sadly, I do not have a solution for this issue since I know that free software developers are usually volunteers who do not have to report to anybody, and that they do not need to care for their user base. But I hope that people begin to reconsider their attitude and think about what kind of horrible damage is being done to the user base and to free software s reputation.

Linux policy routing is still incredibly painful if one wants to have more sophisticated routing than just take source and destination IP address for the routing decision . The mechanisms that have been in use seven years ago still work though, and I didn t find any possibility to do it any easier. In this article, I ll try to explain the old mechanisms and hope that somebody from lazyweb will comment and say it can be done so much easier . This is a translation of the Usenet article <gu48cs$rul$1@news1.tnib.de> in de.comp.os.unix.networking.misc in the hope that the english-speaking blogosphere can give additional insights. Given a Linux-based router with one internal network (int0), one perimeter network (per0) and two Internet connections (ext0, ext1) with one IP address each. We need to do source NAT to deliver Internet to the internal and perimeter networks. The internet connection on ext0 will be used for http and https, while all other traffic needs to go out on ext1. The perimeter network is reachable from the outside via destination NAT.

1: ext0:  mtu 1500 qdisc noqueue
    inet 10.81.221.145/29 brd 10.1.221.151 scope global ext0
2: ext1:  mtu 1500 qdisc noqueue
    inet 10.82.83.225/29 brd 10.82.83.231 scope global ext1
3: per0:  mtu 1500 qdisc noqueue
    inet 172.16.1.254/24 brd 172.16.1.255 scope global per0
4: int0:  mtu 1500 qdisc noqueue
    inet 192.168.8.254/24 brd 192.168.8.255 scope global int0

We have the following routing rules:

0:      from all lookup 255
10:     from all lookup main
32:     from all fwmark 0x12e lookup to_ext0
32:     from all fwmark 0x12f lookup to_ext1
32000:  from all lookup defaultroute
32000:  from all lookup defaultroute
32766:  from all lookup main
32767:  from all lookup default

Table main contains all rules for the directly connected networks, but no default route. Both to_ tables have one default route each, pointing to the respective ISP, and the table defaultroute has once more a default gateway pointing to ext1 s gateway. The PREROUTING chain of the mangle table has these rules:

iptables --protocol tcp --dport 80 --in-int int+ --set-mark 0x12e
iptables --protocol tcp --dport 443 --in-int int+ --set-mark 0x12e
iptables --in-int int+ --set-mark 0x12f

and the POSTROUTING chain of the nat table has these:

iptables --match mark --mark 0x12e --out-int ext0 --jump SNAT --to-source 10.81.221.145
iptables --match mark --mark 0x12f --out-int ext1 --jump SNAT --to-source 10.82.83.225

If now somebody from the internal network accesses a web server on the Internet. The outgoig packet will first be marked in the PREROUTING chain, then the routing rules will use the firewall mark to consult the correct routing table and pass the packet back to the packet filter, where the POSTROUTING chain will use the outgoing interface and the firewall mark to SNAT the packet to the correct source IP so that the answer will actually reach us. Has a less ugly way available way to do this become available in the last five years? Or is it still necessary to have the different parts of the networking subsystems play ping-pong with the packet? I particularly dislike that someone not familiar with policy routing will not see any default route in the routing table printed by route or ip route , which will probably be very confusing. Is there any possibility to interact with the routing code without firewall marks, maybe by directly setting a gateway from a firewall rule?

I haven t been using ATM on Linux for some six years now. I neither have access to an ATM network any more nor do I have ATM hardware any more. Therefore, I plan to remove ATM support from ifupdown-scripts-zg2 in the next release which will be done in the next few weeks. If anybody does still use ATM on Linux in conjunction with my scripts, you might want to offer help with the package if you want to have continued ATM support in ifdown-scripts-zg2. I cannot test the code any more and therefore cannot maintain it in the future.

I just wanted to make an USB stick bootable and wondered why mkdiskimage -4 /dev/sda 0 32 64 complained about the disk having too many cylinders. After a few moments, it ocurred to me that since libata, the system hard disk has become sda and that the stick was sdb or sdc. One ctrl-C later, fdisk confirmed both counts: That I accidentally started mkdiskimaging my main system hard disk and that the partition table was already gone. A few hours later, the notebook is back in business without too much data loss. Lucky me. My notebook has an alibi installation of Windows XP in its first primary partition sized 4 GB. Since I aborted the mkdiskimage process early enough, I hoped that only the windows installation was hosed. The Linux system is entirely in LVM (with crypted LVs), and LVM hadn t noticed yet that its PV was gone. I reckoned that I still had a chance to rescue my data if I didn t reboot. So I hastily commandeered an USB hard disk (thanks, $CUSTOMER, for quickly supplying one), fdisked, pvcreated and vgextended it and started the pvmove. While this was happening, I theoretically was able to continue working, which I didn t because I was afraid of losing. About half an hour into the copy process, the USB disk chose to deregister itself and the pvmove aborted. Rebooting the disk made it reappear as sde (it was sdd before), and miraculously, a new pvmove call just worked. However, a good fraction of actions on the shell resulted in a lot of input/output errors on the screen. I guessed that this was the first sign of my data finally dying, but I let the pvmove continue. After the pvmove was finished (which was rather fast since the move of the crypted LV means that the crypted data is moved and not de- and recrypted), I had to recreate a /dev/sda2 partition with type 8e before the LVM tools allowed me to vgreduce the VG. I then zeroed out the first 10 GB of the internal disk and spent a few hours reinstalling and patching XP from scratch before I finally went through the pvcreate, vgextend, pvmove routine in the other direction. Then the big moment arrived: fsck of all LVs. I guess that the reason for the input/output errors was that /usr was completely hosed (no single file outside lost+found), but /home and all other file systems holding data where the way back to the (a week old) backup would have been quite painful were ok. I pulled /, /usr and /var from the backup to be consistent again, updated to current sid (avoiding KDE 4 for the time being) and could start working again. The only real loss was a few hours of work that were stored in /usr/local which died along with /usr, so the result of my stupidity was not very catastrophic but still a good lesson. And I have learned that LVM is a robust and resilient beast. Good work. Now I d like to know what happened to my /usr - I always had the impression that pvmove should not break data even when the target disk suddenly goes away...

In the last few weeks, I spent quite some time wondering about how to arrange the hard disk layout of my productive systems in the future. This article outlines my thoughts and would like to ask the lazyweb for comments. I try to keep my Debian servers as identically as possible, making it possible to talk non-linux persons remotely through the system without having to worry about this particular box configuration. I have been especially worrying about:

• (h1) root FS location (/etc/fstab, grub/menu.lst)
• (h2) LVM volume naming (/etc/fstab)

On my current systems, I usually have the root file system on /dev/hda1, with /dev/hda2 being a PV which is the only PV of a VG vg0, which in turn has LVs named home, var and usr. This setup has a bunch of disadvantages.

• It is necessary to derive from the standard setup, when RAID or crypto is used. In these cases, the root fs needs to be in LVM as well, and hda1 is /boot.
• LVM setup breaks in recovery and/or migration scenarios, when the disks from one server are connected to another one due to two VGs named vg0 being present. These situations are solveable via lvrename per UUID though.
• Migration to libata is painful since the dreaded hda string lurks in half a dozen places, and Debian grub does not really cleanly support having different kernels that need different root= clauses on their command line.

Thankfully, there is a number of possible solutions:

• (l1) Mount the root fs with UUID
  • - UUID needs to be adapted in /etc/fstab and grub/menu.lst if file system is rebuilt.
  • - mount-per-UUID is a function of the Debian initrd, mechanism nonportable, initrd needed
  • + no conflicts when a server can access disks of another system, as UUID are unique.
• (l2) Mount root fs with label (all root fsses have the same label)
  • - Conflicts when disks are moved between servers, making it possible to boot the wrong system.
  li>
• (l3) Mount root fs with a host-specific label
  • - When the system (and the root fs) is renamed, etc/fstab and grub/menu.lst need to be adapted.
  • + no conflicts when disks of more than one server are plugged in.
• (l4) VG is named identically on all servers
  • - Manual intervention is necessary (see above) if another server s disks are placed into one server.
  • + Backup and other disk related scripts can be configured identically in all systems.
• (l5) VG is called like the server
  • - Attention needed when renaming a system (/etc/fstab needs to be adjusted).
  • - Backup and other disk related scripts need to be configured differently on each system, no standard config possible.
  • + no conflicts in the more than one set of disks in a server case.

I am pondering about a solution, but suspect that this is totally overengineered.

• (l1) Root fs is mounted via label, so that /boot/grub/menu.lst only needs to be edited once during syste installation. After migration to grub 2, a hook script (if grub2 s update-grub finally supports hooks), the root FS label can be automatically generated. molly-guard refuses to boot the system when the root FS label from the boot manager configuration is not identical to the currently mounted root fs label, thus forcing to boot with the current root fs or not to boot at all. Do I need to call blkid -g in that case?
• (l2) /etc/fstab is identical on all systems and has entries like
  • /dev/disk/localhost/usr /usr ext3 defaults
  The symlinks in /dev/disk/localhost are generated by a run level S (27) init script (or can this be accomplished by udev?) from /dev/mapper/$(uname -n)--usr, so that the VG is always named like the host. Renaming the VG needs a rescue system anyway, so that s not a real loss.

Did I miss something? Is this a realistic solution? How do you handle this issue? Comments to this article will be open for quite a while.

A few months, I blogged about the pains I had with my nVidia FX 5200 graphics card, Debian and current kernels. I have solved the issue in the mean time and would like to document what I did. Currently, the package nvidia-graphics-drivers has its version 173.14.09 in Testing and Unstable; and 177.82 in Experimental. There is also a package nvidia-graphics-drivers-legacy from 2006 in stable. 177.82 does build with recent kernels, but doesn t support my old FX card. 173.14.09 supports my FX card, but doesn t build with recent kernels. I didn t try the legacy stuff because of its old age. The actual solution was easy: The current legacy beta driver that can be downloaded from the nVidia ftp server has 173.14.15, which happens to build with a recent kernel and supports my FX card, as kju pointed out in a comment to my original blog entry. I then took the 173.14.09 diff from Debian unstable, put it over the pkg0.run and pkg2.run archives from nvidia (i386 and amd64), modified debian/upstream_info and built. This first failed because some Debian patches didn t apply correctly, but since the file name patches/xen.patch suggested that it was only needed for xen systems, I dared to remove it. Leaving the patches/ directory empty made the build system topple, so I touched an empty file into patches and the build went fine. Out of the resulting nvidia-kernel-source, I was able to compile kernel modules against 2.6.28, and am now back in business even with a recent kernel.