
What started with
a little host naming theme
riddle, and spread as
a meme
across various planets, has come full circle today, as I
installed a new machine,
khyber.madduck.net in the
racks hosted at the
LayerOne
colocation site in Z rich.
Init Seven, my favourite Swiss
ISP, are sponsoring the rack space and connectivity: a routed
/28 IPv4 subnet and a
/64 IPv6
subnet.
The machine is a
Transtec Calleo 331 1U server with two quad-core AMD Opteron
2354 processors, 16Gb of RAM, and 4Tb of disk space, configured
into a 2Tb RAID1 with LVM on top. It will be used to host a couple
of
KVM
instances for a number of projects I currently operate off my main,
personal server,
pulse.madduck.net.
Unfortunately,
Transtec (as
well as most other vendors I checked out) uses nVidia s
MCP55Pro chipset, which had me sceptical long before I
sent the order.
nVidia is known to suck at thinking open-source, and the Linux
kernel driver for their network chips,
forcedeth, had
to be reverse-engineered, because nVidia failed to cooperate. Apart
from that, nVidia s chips are also often simply crap. Many of their
motherboards come with broken APICs and the network chips often
wreak havoc on subnets.
I find it curious that a search for MCP55 or MCP55Pro on the
nVidia website yields no results.
Given these expectations, I insisted on a try-and-buy contract,
which are not default in Switzerland (as opposed to the EU). My
fears proved right, it wasn t long until I encountered the first
problems with nVidia s crap:
the network chip is unable to deal
with segmentation offloading for larger packets, e.g. those
with IPv6 payload in a bridging context, and generic segmentation
offloading has to be turned off on the interface:
ethtool -K eth0 gso off
This fixes the issue, and so I stuffed it into
/etc/network/interfaces and decided to keep the
machine, especially after having invested the time into the server
to figure out how to work the watchdog and the IPMI card. The folks
behind Transtec s support hotline are helpful, but it took me a bit
of time to get the information I wanted out of them. At least they
deemed it perfectly natural that I wanted to run
Debian, and never questioned that.
Let s document some of the more internal specs here for
posterity:
The BIOS gives the impression of two separate watchdogs, one on
the
IPMI card (a
SIMSO+)
and one on the mainboard itself (a
SuperMicro H8DMU+). Transtec told me that SuperMicro disabled
the IPMI watchdog (called BMC watchdog in the BIOS) with firmware
1.29, leaving only the one on the motherboard. It took a bit of
time to find out that this was a W83627THF and supported by the
w83627hf_wdt driver, which I configured with module
options
timeout=120 nowayout=1. I left the BIOS
setting at disabled since that would otherwise interfere with
booting e.g.
the awesome grml live
system by rebooting the machine after the set amount of
time.
It also took a bit of time to get
SOL working
with the IPMI card. In the end, I found it to work best by
configuring the console redirection in the BIOS to use
COM2 with
38400/n/8, using hardware flow
control, redirection-after-POST always, and
vt100
emulation
ANSI and
vt-utf8 emulation
were horrific. I instructed the bootloader to pass
console=tty0 console=ttyS1,38400n8 to the kernel
(order matters), and configured
agetty in
/etc/inittab like this:
T0:2345:respawn:/sbin/getty -8hL ttyS1 38400n8 vt100
To guard against the worst, I also left a
grml CD
in the drive, to be able to boot a rescue system via SOL in case
everything else was broken. The following kernel boot parameters
enable me to use the rescue system via IPMI:
vga=normal noeject noprompt console=tty0 console=ttyS1,38400n8 forensic
Michael Prokop of the grml
project tried to create
a customised ISO image
with these options for me, but I kept getting medium errors
trying to read it, independent of the burner, CD-writing-software,
operating system used to burn, medium, and drive used to read, so
it s possible that the ISO image is faulty. Michael used the
following command line to create it (supporting files are
here):
grml-live -t ~/templates-madduck -c GRMLBASE,GRML_MEDIUM,AMD64 -o \
/grml-live/grml64-medium_2008.11-madduck -a amd64 -s sid -v \
2008.11-madduck -g grml64-medium -r Schluchtenscheisser -V -b
Until we figure out the problem with this, the
grml64
0.2 CD in the drive will do.
The machine is now operational, but the IPv6 prefix I have still
arrives at
vera.madduck.net, a Xen-Instance hosted by
Init Seven, which I will no longer
need, once I find the time to migrate all the services it
hosts.
NP:
This Will Destroy You:
Young Mountain