Now that we have KMS, it happens things go bad enough that when X goes
down, one can no longer access any console to see what
happened. Mainly, one can distinguish between two cases.
- The good one: the machine is still accessible over the
network. Unfortunately, that requires having a spare machine to ssh
into the first machine, but that means one can run
dmesg, look
into X logs, possibly run X inside gdb and see where it breaks
(because glibc s backtrace isn t exactly an excellent tool to
get the idea of what s going on and that s the tool used in X to
print a backtrace in the log through a signal handler).
- The bad one: the machine is no longer accessible over the
network. Likely, the kernel is in such a bad shape that one is
never going to know what happened. Usually, writing logs to disc
didn t happen, so logs won t magically show up at next reboot.
I m going to discuss the second case. There s a nice tool in the
kernel which makes it possible to send kernel logs over the
network. It s called
netconsole. As far as limitations are
concerned, one shall note that it s UDP only, and over Ethernet (in
other words: no wireless). The good news is that it can usually make
the last crucial lines available.
Let s call the crashy machine a
patient and the logging machine a
doctor.
One should keep in mind that if something goes wrong while configuring
the
netconsole module, it can be unloaded at anytime through:
sudo modprobe -r netconsole
Easy case: on a local network
Here s an example: Patient s IP is
192.168.0.1, doctor's IP is
192.168.0.2.
Doctor setup
What one needs on the receiving side: only
netcat. A tiny warning,
there are several implementations. Here s the appropriate syntax:
# netcat-traditional:
nc -l -u -p 6666 tee ~/netconsole.log
# netcat-openbsd:
nc -l -u 6666 tee ~/netconsole.log
The
nc command is handled through the alternatives system, one can
use
update-alternative --config nc to pick the appropriate one if
several
netcat-* packages are installed. To list the current one, an
easy way is
readlink -f $(which nc).
Using
tee means that incoming messages are going to be printed, both
to the standard output and to the specified file.
Running as non-privileged user is sufficient.
Patient setup
Now, to have the patient send stuff to the doctor, a simple
modprobe
is needed:
sudo modprobe netconsole netconsole=@/,6666@192.168.0.2/
What happens here? One requests the
netconsole module to be loaded,
and one specifies the parameters. Details can be read in the Linux
kernel documentation (
Documentation/networking/netconsole.txt), but
concentrating on the points of interest here:
- 192.168.0.2: The doctor s IP.
- 6666: The UDP port to which packets are sent. 6666 is the default
and can be omitted.
That s all! On the doctor side, one should see something along those
lines:
[2920549.188090] console [netcon0] enabled
[2920549.188102] netconsole: network logging started
Slightly harder case: over internet
Because one might not have a second machine handy, it s also possible
to go through a router and send stuff across the internet.
Here s an example: Patient s IP is
192.168.0.1, connecting to
internet through a router, which IP is
192.168.0.254. The doctor,
available over the internet, is
192.0.32.10.
Doctor setup
Exactly the same as in the previous case. The main issue one can run
into is firewalls at this point. That s why one may want to switch
from the default 6666 port to another one which wouldn t be
filtered. One can think of 53 (DNS), but that means running
nc as
root since it s a privileged port (below 1024).
Patient setup
Routing is also possible, but that requires an extra parameter: the
MAC address of the router. To obtain it, one can use the
arp
command:
/usr/sbin/arp 192.168.0.254
Supposing it returned the
01:02:03:04:05:06 address, loading the
module becomes:
sudo modprobe netconsole netconsole=@/,6666@192.0.32.10/01:02:03:04:05:06
Now, if you re running into firewall-related issues, you can change
the source for the UDP packets. The default is 6665, but assuming you
want to send from an unfiltered 1234 port, that becomes:
sudo modprobe netconsole netconsole=1234@/,6666@192.0.32.10/01:02:03:04:05:06
You probably all know du from 
I'm going to be listing the more inept of the commercial spammers
that I'm annoyed by, so I can hopefully help give them a more accurate
reputation on Google searches.
Spammer Number 1: CM Crossroads via Streamsend
Guilty of:
Developers are not usually the right people to write documentation, but in an effort to encourage more Debian folk to consider 
Most packages contain files that you don t need: for example translations in languages that you don t understand, or documentation that you don t read. Wouldn t it be nice if you could get rid of them and save a few megabytes? Good news: since dpkg 1.15.8 you can!
dpkg has two options 
This brief announcement was published in the 
The other options in the poll were 
In my professional role as Ubuntu CTO, I take on a number of different perspectives, which sometimes compete for my attention, including:
I miss the days when Debian was about making software work well together.
These days the user mailing lists are full of posts from users asking for help on Ubuntu [*1*], people suggesting that we copy what Ubuntu has done, and people asking for "howtos" because documentation is too scary or too absent for them to read.
Yesterday the whole