If you want a reliable network then you need to determine an appropriate level of redundancy. When servers were small and there was no well accepted virtual machine technology there were always many points at which redundancy could be employed. A common example is a large mail server. You might have MX servers to receive mail from the Internet, front-end servers to send mail to the Internet, database or LDAP servers (of which there is one server for accepting writes and redundant slave servers for allowing clients to read data), and some back-end storage. The back-end storage is generally going to lack redundancy to some degree (all the common options involve mail being stored in one location). So the redundancy would start with the routers which direct traffic to redundant servers (typically a pair of routers in a failover configuration - I would use OpenBSD boxes running CARP if I was given a choice in how to implement this [1], in the past I’ve used Cisco devices). The next obvious place for redundancy is for the MX servers (it seems that most ISPs have machines with names such as mx01.example.net to receive mail from the Internet). The way that MX records are used in the DNS means that there is no need for a router to direct traffic to a pair of servers, and even a pair of redundant routers is another point of failure so it’s best to avoid them where possible. A smaller ISP might have two MX machines that are used for both sending outbound mail from their users (which needs to go through a load-balancing router) as well as inbound mail. A larger ISP will have two or more machines dedicated to receiving mail and two or more machines dedicated to sending mail (when you scan for viruses on both sent and received mail it can take a lot of compute power). Now the database or LDAP servers used for storing user account data is another possible place for redundancy. While some database and LDAP servers support multi-master operation a more common configuration is to have a single master and multiple slaves which are read-only. This means that you want to have more slaves than are really required so that you can lose one without impacting the service. There are several ways of losing a server. The most obvious is a hardware failure. While server class machines will have redundant PSUs, RAID, ECC RAM, and a general high quality of hardware design and manufacture, they still have hardware problems from time to time. Then there are a variety of software related ways of losing a server, most of which stem from operator error and bugs in software. Of course the problem with the operator errors and software bugs is that they can easily take out all redundant machines. If an operator mistakenly decides that a certain command needs to be run on all machines they will often run it on all machines before realising that it causes things to go horribly wrong. A software bug will usually be triggered by the same thing on all machines (EG I’ve had bad data written to a master LDAP server cause all slaves to crash and had a mail loop between two big ISPs take out all front-end mail servers). Now if you have a mail server running on a virtual platform such that the MX servers, the mail store, and the database servers all run on the same hardware then redundancy is very unlikely to alleviate hardware problems. It’s difficult to imagine a situation where a hardware failure takes out one DomU while leaving others running. It seems to me that if you are running on a single virtual server there is no benefit in having redundancy. However there is benefit in having an infrastructure which supports redundancy. For example if you are going to install new software on one of the servers there is a possibility that the software will fail. Doing upgrades and then having to roll them back is one of the least pleasant parts of sys-admin work, not only is it difficult but it’s also unreliable (new software writes different data to shared files and you have to hope that the old version can cope with them). To implement this you need to have a Dom0 that can direct traffic to multiple redundant servers for services which only have a single server. Then when you need to upgrade (be it the application or the OS) you can configure a server on the designated secondary address, get it running, and then disable traffic to the primary server. If there are any problems you can direct traffic back to the primary server (which can be done much more quickly than downgrading software). Also if configured correctly you could have the secondary server be accessible from certain IP addresses only. So you could test the new version of the software using employees as test users while customers use the old version. One advantage a virtual machine environment for load balancing is that you can have as many virtual Ethernet devices as you desire and you can configure them using software (without changing cables in the server room). A limitation on the use of load-balancing routers is that traffic needs to go through the router in both directions. This is easy for the path from the Internet to the server room and the path from the server room to the customer network. But when going between servers in the server room it’s a problem (which is not insurmountable, merely painful and expensive). Of course there will be a cost in CPU time for all the extra routing. If instead of having a single virtual ethernet device for all redundant nodes you have a virtual ethernet device for every type of server and use the Dom0 as a router you will end up doubling the CPU requirements for networking without even considering the potential overhead of the load balancing router functionality. Finally there is a significant benefit in virtual machines for reliability of services. That is the ability to perform snapshot backups. If you have sufficient disk space and IO capacity you could have a snapshot of your server taken every day and store several old snapshots. Of course doing this effectively would require some minor changes to the configuration of machines to avoid unnecessary writes, this would include not compressing old log files and using a ram disk for /tmp and any other filesystem with transient data. When you have snapshots you can then run filesystem analysis tools on the snapshots to detect any silent corruption that may be occurring and give the potential benefit of discovering corruption before it gets severe (but I have yet to see a confirmed report of this saving anyone). Of course similar snapshot facilities are available on almost every SAN and on many NAS devices, but there are many sites that don’t have the budget to use such equipment.

• [1] http://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol

Share This

I was looking for a Python lexer and parser, and found out that antlr can emit Python code with only minor twiddles of the input grammar.

The antlr site has a "Hello World" type of grammar for testing it out in Java: http://www.antlr.org/article/cutpaste/index.html

What follows is that page ported to Python and Debian/Ubuntu. On a Debian-derived system:

$ sudo apt-get install antlr

Create a file called "t.g" with the following contents:

options  
	language=Python;
 
class P extends Parser;
startRule
    :   n:NAME
         print "Hi there, "+n.getText() 
    ;
class L extends Lexer;
// one-or-more letters followed by a newline
NAME:   ( 'a'..'z' 'A'..'Z' )+ NEWLINE
    ;
NEWLINE
    :   '\r' '\n'   // DOS
        '\n'        // UNIX
    ;

Because antlr isn't in the CLASSPATH by default, the maintainer has included a "runantlr" script, which handles this for you. So,

$ runantlr t.g

This will create L.py, P.py and PTokenTypes.txt.

Now create main.py:

#!/usr/bin/python
import sys
import L
import P
l = L.Lexer(sys.stdin)
p = P.Parser(l)
p.startRule()

And run the resulting file:

$ python main.py
Jeff
Hi there, Jeff
$

Or theming PS1 I am not fanatical about my command prompt. No sirree, not me. It is just that I spend half my life either staring at an Emacs window, or at an xterm command prompt; so even a marginal boost in productivity goes a long way. And I am often logged in over ssh to machines half a continent away, and am still comfortable enough on the Linux VT to spend time there, and I often do not have the GUI gew-gaws feeding me data. Hmm. Data. inpuuut. No. Must focus. With all these open xterms and Emacs terminal mode frames floating around, it is easy to lose track of where I am on each terminal, and what the working directory is. So I want my command prompt to help me keep track of where I am. If the terminal is an xterm, the title can be setup like

 "user@host:../shotened/path/to/current/working/dir"

I want to know what machine, id (am I root?), and directory I am in. If I am deep down in the labyrinths of some work related directory tree, I want the path to be pruned, from the left, one component at a time. However, this does not help me on the console; so I also want the path to be in the command prompt; but it should not take up too much of the command line; and ideally, should just go away as I type a longish command. Gawd, I love zsh. I once did a analysis of the command history. The most often used command sequences were

 cd some-place, ls

So. pwd and ls. I really really want to see the directory listing when I change directory to a new one. As one grows old, memorizing the directory contents for dozens of machines all the time taxes the gray cells a bit. So, I figure, why not let the command prompt handle all that? Having the current working dir always visible cuts down a heck of a lot on the pwd commands, and so all that s left is to insist that the command prompt thingy always run ls after a change directory. Simple enough. Saves on typing. And time. And this is not just some crazy talk. I want help with noticing whether the previous command exited with an error status (useful for commands that normally do not create an output). If I am logged in to a machine on battery power, I want to know that. I also like visual cues to the amount of power remaining (good for my laptop on the long flights). I don t want to have to know f the machine uses APM or ACPI, I just want my prompt color to change as the power fades. I want my command prompt to let me know if I am in a directory which is under version control, and if so, what branch I am on. (I occasionally have to come in contact with arch, bzr, git, subversion, svk, and mercurial). If I am in a version controlled project, where I am relative to the root of the checked out tree is often more important than the absolute path, so I want to see relative paths, not absolute paths. I want to know if there are uncommitted files in the working directory. Visually. I want to be reminded if I am in the middle of an ongoing

 rebase -i .

This is not asking for too much, is it? So, here is a screen-shot of this in action: I start from my home directory, go to a directory not under version control, go to a project under git, then a different project with uncommitted files, and then finally to a subversion checkout. All with an angry fruit salad of colors warrantied to make Martin Krafft want to claw his eyes out . If you use zsh, then just grab hold of this, and then do:

 autoload -U promptinit
 promptinit
 prompt manoj

This might be a bit of a hit on slow machines, but even my laptop is a core 2 duo, so I do not find it noticeable. Oh, and if you liked this article, you might also like Theming Emacs, and Theming XTerms.

Ever the sucker for punishment, I decided to pick three difficult things and stick them all together: LDAP, SSL and replication. Here's how to make it go on Debian and Ubuntu. The problem You want LDAP replication to happen over the internet, and you want it to happen securely. The caveat I'm not going to tell you how to set up your LDAP from scratch here: I'm assuming you've reached a solution you're happy with and want to replicate it. The solution We're going to set up a replicating slave LDAP server, which communicates with the master over the internet via an SSL-protected connection. Enabling replication First up, the master LDAP server needs to be configured to permit replication.  The key lines to add to your slapd.conf include:

moduleload syncprov
index entryCSN,entryUUID eq
overlay syncprov
syncprov-checkpoint 100 10
syncprov-sessionlog 200

These load up the synchronization module, add indices which make sync go faster, and enable sync. For more detail see the OpenLDAP site. Next you need to add a replicator user to your LDAP database, give your replicator user access to passwords as well as general read access. To create the replicator user, I made this simple LDIF file and fed it to ldapadd.

dn: cn=replicator,dc=mydomain,dc=com
objectClass: simpleSecurityObject
objectClass: organizationalRole
cn: replicator
description: LDAP replicator
userPassword: TOPSEKRIT

Once this user is in your LDAP database, you should give it read access to passwords (I assume you've already given read access to authenticated users.) I have this in my slapd.conf:

access to attrs=userPassword,sambaNTPassword,sambaLMPassword
...
   by dn="cn=replicator,dc=mydomain,dc=com" read

To check that this works, try using ldapsearch to check that the passwords are returned:

ldapsearch -x -D cn=replicator,dc=mydomain,dc=com \
  -W   grep -i password

Enter the replicator password when prompted, and you should see the encrypted passwords from your LDAP database. Securing access Now you've got replication enabled on the master, you will want to ensure it is available on the internet only via TLS or SSL. Here's what I added to slapd.conf to enable this:

TLSCertificateFile      /etc/ssl/certs/ldapserver_crt.pem
TLSCertificateKeyFile   /etc/ssl/private/ldapserver_key.pem
TLSCACertificateFile    /etc/ssl/certs/myCA.pem
TLSVerifyClient         demand

As you will guess from the configuration, the first two lines set the SSL key and certificate the master uses (see "A little twist" below for an important note on key permissions.) The third line tells slapd where to find my site-local certificate authority (CA), and the fourth line says slapd must require any connecting client to have a valid SSL certificate signed by the site-local CA. This is important, as it provides a second layer of access control: a replicating client must connect using a certificate you signed, plus the replicator password. Before this enables TLS access, we must tell slapd which network interfaces to listen on. To do this, edit the SLAPD_SERVICES variable in /etc/default/slapd. Here's my configuration:

SLAPD_SERVICES="ldap://127.0.0.1/ ldap://192.168.0.1/ ldaps:///"

This enables regular LDAP on the loopback and intranet network interfaces, and LDAP/SSL on all interfaces, including the public internet. So, with slapd restarted we are at this situation: connections are now possible from the internet, as long as they are made over SSL with a certificate signed by our site-local CA. (In fact, you can make much finer-grained access restrictions in your configuration than I have done. Using LDAPS rather than TLS over regular LDAP is a rather broad precaution. As explained on the OpenLDAP site, the ssf= parameter can be used to require a certain level of secure connectivity on a per-user or client basis.) Setting up the replicating server
Your slave server should have the same configuration as the master, except you can leave out the bits enabling replication. Firstly, you'll need add to slapd.conf the replication configuration:

syncrepl rid=123
        provider=ldaps://ldapmaster.mydomain.com/
        type=refreshAndPersist
        searchbase="dc=mydomain,dc=com"
        filter="(objectClass=*)"
        scope=sub
        attrs="*"
        schemachecking=off
        bindmethod=simple
        binddn="cn=replicator,dc=mydomain,dc=com"
        credentials=TOPSEKRIT

Most of this I took as boilerplate from the OpenLDAP documentation. Items to note include:

• the rid is a unique 3-digit integer per slave, used to maintain sync state
• the credentials should be the password you gave the replicator user
• the type can either be refreshAndPersist, or refresh. The latter institutes a simple polling replication, whose interval you can vary with the interval parameter. In our case, we do a poll and then keep the replication search open: our client gets notified immediately when there's any new data matching the replicating search.
• the searchbase is an LDAP search matching the data we wish to be replicated.

And here's the /etc/default/slapd configuration:

SLAPD_SERVICES="ldap://127.0.0.1/"

The slave slapd exists only in this case to serve the local machine. Finally, there's the tricky bit! You need to configure slapd to connect to the master server using a certificate. I'll assume you've created and signed a key and certificate pair for your slave server (see my post Low-tech SSL certificate maintenance for more on this.) Awkwardly, the TLS configuration in slapd.conf is for the server only. Replication works as a client, and thus needs separate configuration. Furthermore, you cannot configure this globally on your machine, as the SSL certificate is a per-user only parameter (see man ldap.conf for more information on this.) Instead, we must set it in slapd's environment. Add these two lines to the end of /etc/default/slapd:

export LDAPTLS_CERT=/etc/ssl/certs/slapd.crt
export LDAPTLS_KEY=/etc/ssl/private/slapd.key

This file is sourced as a shell script by slapd's init script. Amend the path to your certificate and keys as required. Use /etc/init.d/slapd restart and you should be good to go. Finally, we want the slave server to be certain it's talking to the real master. So we also configure client connections to verify the SSL certificate of the peer, in ldap.conf again:

TLS_CACERT      /etc/ssl/certs/myCA.crt
TLS_REQCERT     demand

A little twist One gotcha to notice with both client and server is that slapd runs as the openldap user by default on Debian. Also by default SSL keys are readable only by the ssl-cert group. You'll need add the openldap user to this group, otherwise it won't be able to access /etc/ssl/private. Related articles on this site:

• Turn your world LDAP-tastic

Join the conversation about this post

It’s a common practice when hosting email or web space for large numbers of users to group the accounts by the first letter. This is due to performance problems on some filesystems with large directories and due to the fact that often a 16bit signed integer is used for the hard link count so that it is impossible to have more than 32767 subdirectories. I’ve just looked at a system I run (Bluebottle anti-spam email service [1]) which has about half a million accounts and counted the incidence of each first letter. It seems that S is the most common at almost 10% and M and A aren’t far behind. Most of the clients have English as their first language, naturally the distribution of letters would be different for other languages. Now if you were to have a server with less than 300,000 accounts then you could probably split them based on the first letter. If there were more than 300,000 accounts then you would face the risk of having there be too many account names starting with S. See the table below for the incidences of all the first letters. The two letter prefix MA comprised 3.01% of the accounts. So if faced with a limit of 32767 sub-directories then if you split by two letters then you might expect to have no problems until you approached 1,000,000 accounts. There were a number of other common two-letter prefixes which also had more than 1.5% of the total number of accounts. Next I looked at the three character prefixes and found that MAR comprised 1.06% of all accounts. This indicates that splitting on the first three characters will only save you from the 32767 limit if you have 3,000,000 users or less. Finally I observed that the four character prefix JOHN (which incidentally is my middle name) comprised 0.44% of the user base. That indicates that if you have more than 6,400,000 users then splitting them up among four character prefixes is not necessarily going to avoid the 32767 limit. It seems to me that the benefits of splitting accounts by the first characters is not nearly as great as you might expect. Having directories for each combination of the first two letters is practical I’ve seen directory names such as J/O/JOHN or JO/JOHN (or use J/O/HN or JO/HN if you want to save directory space). But it becomes inconvenient to have J/O/H/N and the form JOH/N will have as many as 17,576 subdirectories for the first three letters which may be bad for performance. This issue is only academic as far as most sys-admins won’t ever touch a system with more than a million users. But in terms of how you would provision so many users, in the past the limits of server hardware were approached long before these issues. For example in 2003 I was running some mail servers on 2RU rack mounted systems with four disks in a RAID-5 array (plus one hot-spare) - each server had approximately 200,000 mailboxes. The accounts were split based on the first two letters, but even if it had been split on only one letter it would probably have worked. Since then performance has improved in all aspects of hardware. Instead of a 2RU server having five 3.5″ disks it will have eight 2.5″ disks - and as a rule of thumb increasing the number of disks tends to increase performance. Also the CPU performance of servers has dramatically increased, instead of having two single-core 32bit CPUs in a 2RU server you will often have two quad-core 64bit CPUs - more than four times the CPU performance. 4RU machines can have 16 internal disks as well as four CPUs and therefore could probably serve mail for close to 1,000,000 users. While for reliability it’s not the best idea to have all the data for 1,000,000 users on internal disks on a single server (which could be the topic of an entire series of blog posts), I am noting that it’s conceivable to do so and provide adequate performance. Also of course if you use one of the storage devices that supports redundant operation (exporting data over NFS, iSCSI, or Fiber Channel) then if things are configured correctly then you can achieve considerably more performance and therefore have a greater incentive to have the data for a larger number of users in one filesystem. Hashing directory names is one possible way of alleviating these problems. But this would be a little inconvenient for sys-admin tasks as you would have to hash the account name to discover where it was stored. But I guess you could have a shell script or alias to do this. Here is the list of frequency of first letters in account names:

First Letter	Percentage
a	7.65
b	5.86
c	5.97
d	5.93
e	2.97
f	2.85
g	3.57
h	3.19
i	2.21
j	6.09
k	3.92
l	3.91
m	8.27
n	3.15
o	1.44
p	4.82
q	0.44
r	5.04
s	9.85
t	5.2
u	0.85
v	1.9
w	2.4
x	0.63
y	0.97
z	0.95

• [1] http://www.bluebottle.com/

Share This

This new version of DebTorrent finalizes the work I previously uploaded to experimental. This new version is the first in unstable to use unique piece numbers to create long-lived torrents for testing and unstable. It also comes with a new version of the apt-transport-debtorrent package, which can now tell APT not to cache any package files downloaded with DebTorrent, which should save some disk space for some people. The locations of the long-lived torrent files and piece files that DebTorrent uses has also changed, as they are now hosted on my merkel page and should be more up to date. However, I was not able to get the ssh trigger for the sync from ftp-master setup, nor am I able to access the projectb database through pygresql. If anyone knows how I can get those working, please email me. Here is the changelog for debtorrent:

• Display the torrent identifier on the local status page rather than the info hash (Closes: #465339)
• Add support for the new No Content message to apt-transport-debtorrent so that apt treats debtorrent sources as local
• Make the pieces and unique piece number locations config options, and use the new ones at merkel.debian.org/~camrdale/.
• Decompress the files needed to create torrents so they are created and started faster (this and the previous Closes: #463676)

And for apt-transport-debtorrent:

• Add a config option to make debtorrent sources local (Closes: #477383)
• Upgrade the recommends on debtorrent to 0.1.7.
• Add a config file for APT to load, and a manpage describing the options.

Lucas Nussbaum recently posted some graphics showing the number of packages that are grabbed from Debian and kept untouched, modified, out of sync, etc, etc.

But, "statistics are tricky":

---

$ wget http://patches.ubuntu.com/p/php5/php5_5.2.6-1ubuntu3.patch
--2008-06-19 22:25:03-- http://patches.ubuntu.com/p/php5/php5_5.2.6-1ubuntu3.patch
Resolving patches.ubuntu.com... 91.189.90.149
Connecting to patches.ubuntu.com 91.189.90.149 :80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 441869 (432K) [text/plain]
Saving to: php5_5.2.6-1ubuntu3.patch'

100%[======================================================================>] 441,869 62.3K/s in 7.3s

2008-06-19 22:25:11 (59.4 KB/s) - php5_5.2.6-1ubuntu3.patch' saved [441869/441869]

$ lsdiff -i '*out' php5_5.2.6-1ubuntu3.patch wc -l
44

I believe people familiar with test suites know what that means...

Update: it means that the test suite files (outputs, expected results, and so on) are on the .diff.gz, thus contaminating it

Oh, and:

$ out=$(mktemp); filterdiff -i '*tests*' php5_5.2.6-1ubuntu3.patch > $out
$ du -h php5_5.2.6-1ubuntu3.patch $out
436K php5_5.2.6-1ubuntu3.patch
372K /tmp/tmp.eAxRv20298

...what a waste of space

Even though I ve dealt with IPv6 for almost a decade, have delivered presentations, and given multi-day courses on IPv6 security aspects, I ve never actually added IPv6 to my own server/home network infrastructure because it seemed that Linux and/or Debian just weren t ready for it. This seems to have changed (although there are still a number of problems) and in less than a day, I put a few of my machines online. In the following, I d like to share with you how I did it.

Kernel versions and stateful connection tracking Unfortunately, I have to start off with some bad news: even though Debian etch, our current stable release, which uses a Linux kernel version 2.6.18, speaks IPv6, I cannot recommend it for deployment, as the 2.6.18 kernel does not support proper stateful connection tracking for IPv6, and thus makes it impossible to firewall hosts in a sensible manner (I always add local packet filters to all my hosts, and if only to guard against the situation when a user installs a malicious programme to listen on a high port). Of course, it is possible to configure a packet filter statelessly in an acceptable manner once you know the use case, so do with this information what you wish; I prefer to stay general for now. For me, a remedy is almost around the corner: the 2.6.24 kernel seems to support stateful connection tracking for IPv6, and it s even available for etch as it will be included in the upcoming etch-and-a-half release. I simply ended up using the kernel packages pre-release, and so far have not had a problem with it. To do so, add the following line to your /etc/apt/sources.list, making sure to use a close archive mirror:

deb http://ftp.xx.debian.org/debian etch-proposed-updates main

I then just upgraded the system and pulled in all proposed updates. As that may have let in software that won t be part of etch-and-a-half, or even lenny, you may want to pin the archive and only upgrade the kernel packages, by adding to /etc/apt/preferences (replacing amd64 with your architecture):

Package: *
Pin: release a=proposed-updates
Pin-Priority: -1
Package: linux-image-2.6.24-etchnhalf.1-amd64
Pin: release a=proposed-updates
Pin-Priority: 600

Alternatively, you could use the 2.6.24 linux kernel packages on backports.org.

Xen and IPv6 One drawback of switching to 2.6.24 is that you cannot run a dom0 on that machine any longer, so by practical extension, you cannot connect it to the IPv6 network with a packet filter in place. Supposedly, running 2.6.24 instances on a 2.6.18 dom0 is reported to work, however.

Configuring the packet filter The first thing I did was to configure the packet filter on each host appropriately. Unfortunately, this is harder than it should be, because to quote one of the netfilter developers when ip6tables was conceived, someone had a big bad brainfart : rather than adding IPv6 rules to your existing iptables ruleset, you have to create a new ruleset, duplicate all chains, networks, hosts, and individual rules, and maintain the two in parallel. Even though there are efforts of unification on the way, I speculate it ll take another couple of years until PF_INET6 will be fused into PF_INET and one will be able to do sensible cross-address-family packet filtering with Linux. Since I ve recently started to look (again) at pyroman, maybe the most logical way forward would be to extend it to write both, IPv4 and IPv6 rulesets from its knowledge about the hosts and networks you configured. Anyway, we want to get stuff working now! Thus, let s configure ourselves a packet filter. (Almost) all IPv6-related filtering must be configured via ip6tables (read on further down about IPv6 in IPv4 tunneling, the reason I said almost ). The following is a simple default ruleset to start with, which I put into /etc/network/ip6tables to load with ip6tables-restore:

*filter
:INPUT REJECT [0:0]
:FORWARD REJECT [0:0]
:OUTPUT ACCEPT [0:0]
:in-new - [0:0]
### INPUT chain
# allow all loopback traffic
-A INPUT -i lo -j ACCEPT
# RT0 processing is disabled since 2.6.20.9
#-A INPUT -m rt --rt-type 0 -j REJECT
# allow all ICMP traffic
-A INPUT -p icmpv6 -j ACCEPT
# packets belonging to an establish connection or related to one can pass
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
# packets that are out-of-sequence are silently dropped
-A INPUT -m state --state INVALID -j DROP
# new connections unknown to the kernel are handled in a separate chain
-A INPUT -m state --state NEW -j in-new
# pass SYN packets for SSH
-A in-new -p tcp -m tcp --dport 22 --syn -j ACCEPT
# log everything else
-A INPUT -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[INPUT6]: "
### OUTPUT chain
# RT0 processing is disabled since 2.6.20.9
#-A OUTPUT -m rt --rt-type 0 -j REJECT
# allow outgoing traffic, explicitly (despite chain policy)
-A OUTPUT -j ACCEPT
### FORWARD chain
# RT0 processing is disabled since 2.6.20.9
#-A FORWARD -m rt --rt-type 0 -j REJECT
# disallow forwarded traffic, explicitly (despite chain policy)
-A FORWARD -j REJECT
COMMIT

Note that this recipe is pretty much unusable on pre-2.6.20 kernels due to their broken implementation of stateful connection tracking. The ruleset should be fairly obvious, but you might wonder about my use of REJECT and allowing all ICMP after all, you ve heard for the past 30 years that ICMP is a bad hacker protocol , and Internet security is no domain for being nice to people, so to prevent any information disclosure, you should DROP connections, not let people know that they re simply not allowed. Well, to hell with all that! I don t see a single reason or attack vector that is foiled by DROP or disallowing ICMP. In fact, it s just security by obscurity, and might inconvenient at the same time. ICMP is also much more important with IPv6 than with IPv4 (it replaces ARP, for instance), and it s actually useful to be able to ping hosts, or get back informational messages on why something failed. Finally, rejecting traffic rather than dropping it doesn t suggest to a hacker that something s hidden here. Then there is RFC 4890, which almost made me puke. This document is part of the reason why I say: let s fix problems in the kernel, rather than shielding them with unreadable and unmanageable rulesets!

Getting connected If you already have an IPv6 address, you re basically ready to go, but may want to read further down on how to connect your local network to the IPv6 Internet as well. If you are searching for a provider, have a look at the list of providers with native IPv6 connectivity over at sixxs.net. If you are reading up to here, I assume you are connected to the Net with IPv4. There are two ways for you to move towards IPv6: 6to4 or by way of a tunnel provider. A Kiwi website explains how to setting up 6to4 connectivity, and thus I will concentrate only on the tunnel approach. Even though everyone can set up 6to4 in a breeze without any accounts or waiting, there are a number of security considerations, it s pretty ugly to debug (due in part to asymmetric routing), and makes your life unnecessarily difficult when all you have is a dynamic IP that changes from time to time. If you are stuck behind a NAT gateway, you cannot use 6to4 either. Thus, I prefer the tunnel approach. With the tunnel approach, IPv6 packets are wrapped up in IPv4 packets on your host, and sent to the IPv4 address of your tunnel provider, who has native IPv6 connectivity. The tunnel provider unwraps your packet and shoves the contained IPv6 packet onto the backbone. The IPv6 address you used as source address is routed to the tunnel provider, so any replies arrive at their machines, where they re again wrapped into IPv4 packets and sent to your (publicly-accessible) IPv4 address. Those IPv4 packets specify payload type 41 ( ipv6 ), which is why we need those -p ipv6 -j ACCEPT rules in the iptables ruleset. There are a few tunnel providers out there. I chose SixXS and have not regretted my choice. I shall thus assume that you do the same: sign up for an account right now, so that you have it by the time you finished reading this document! SixXS works on a credit system: tunnels and subnets cost credits, which you can accumulate by maintaining your tunnels properly. This ensures that everyone can play around, but to do more advanced stuff, you need to first display competence with the basic concepts. Your first step with SixXS will be to request a tunnel. SixXS offers three types of tunnels:

• static tunnels, for those with static IPs,
• heartbeat tunnels, for those with dynamic IPs, and
• AYIYA tunnels, for those behind NAT gateways.

Each of these tunnels have advantages and disadvantages, as everything does: the first two types of tunnels use IP protocol 41 packets to encapsulate the IPv6 packets. As such, there are security considerations involving the impersonation by spoofing, and all upstream firewalls must let protocol 41 pass. AYIYA addresses these problems by using signed packets, but that solution comes with extra computation overhead and smaller MTUs. I suggest to use the first type of tunnel that fits your situation. Debian s aiccu package can take care of heartbeat and AYIYA tunnels for you, and it can even set up static ones. During registration, you will also need to choose a PoP , which stands for Point of Presence . If your country only has a single PoP, that s the one you will end up using (unless you have a good reason for another one), but if there are more options, I strongly suggest that you go through the list of PoPs and select the one with the best roundtrip time and lowest latency from your location! Note that you must answer ping requests (ICMP echo-request) from the PoP you chose, or else the tunnel will not be created. Once your tunnel request gets approved, you ll get a /64 prefix, in which you only use two addresses: the PoP will configure the :1 address and you need to configure your host to use the :2 address on the tunnel interface. You ll also be told the IPv4 address of your PoP endpoint . Joey Hess taught me that aiccu can set up the interface for you, using the data it queries from the SixXS registration (TIC) server. I tried it, and it works. However, I prefer the pure ifupdown approach, as it makes things explicit and allows me to use the hooks for stuff like loading the packet filter. So in my /etc/network/interfaces, you can find:

auto sixxs
iface sixxs inet6 v4tunnel
  endpoint 194.1.163.40
  address 2001:41e0:ff00:3b::2
  netmask 64
  gateway 2001:41e0:ff00:3b::1
  ttl 64
  pre-up ip6tables-restore < /etc/network/ip6tables
  up ip link set mtu 1480 dev $IFACE
  up invoke-rc.d aiccu start
  down invoke-rc.d aiccu stop

Make sure to read about MTU values of the tunnel and adjust the 1480 value in the above to your tunnel settings and ISP connectivity. Also set ipv6_interface sixxs in /etc/aiccu.conf, if you are using aiccu, or else aiccu will bring up a duplicate/additional interface. If you tell it to use the same interface, it will actually execute all the same commands (which will fail), but won t report any errors. A future version will have a switch to prevent it from configuring the interface. Unfortunately, this will probably not work. The reason is that your regular IP packet filter (iptables, without the 6) doesn t let those encapsulating IPv4 packets pass, unless we tell it to; we probably want to do this early on in the chain, and also limit it to our tunnel peer, so:

iptables -I INPUT -p ipv6 -s 194.1.163.40/32 -j ACCEPT

For AYIYA, you need to open port 5072, either for UDP, TCP, or SCTP, depending on how you configured it. Also have a look at this FAQ entry on what a firewall needs to pass. If it still doesn t work, you have an upstream packet filter that needs some of those holes poked. Good luck. In most situations, the FORWARD chain does not get such a rule, since the tunnel terminates at the gateway, which routes to a native IPv6 network, or another tunnel. Allowing tunnels through a gateway is almost always a bad thing, as it would allow undetected and untraceable traffic from compromised boxes in the local network. The OUTPUT chain also does not need such a rule, if you have configured stateful filtering properly. Now bring up the interface and verify the connection:

# ifup sixxs
# ping6 -nc1 2001:41e0:ff00:3b::1
PING 2001:41e0:ff00:3b::1(2001:41e0:ff00:3b::1) 56 data bytes
64 bytes from 2001:41e0:ff00:3b::1: icmp_seq=1 ttl=64 time=74.0 ms
[...]
# ping6 -nc1 ipv6.aerasec.de
PING ipv6.aerasec.de(2001:a60:9002:1::184:1) 56 data bytes
64 bytes from 2001:a60:9002:1::184:1: icmp_seq=1 ttl=55 time=91.5 ms
[...]

Welcome to the Internet of the future!

Setting up an IPv6-capable gateway Your IPv6 connection works, but it s limited to a single address, and you do not get to specify the reverse DNS PTR record for it. Since the concept of NAT is mostly absent from IPv6 (thanks! thanks! thanks!), you will not be able to connect any other hosts to the IPv6 network. If your local network has a few hosts behind a gateway, you will need to request a subnet from SixXS and configure your gateway (which has the tunnel connection) appropriately. Don t worry, this is not really very difficult. First, request a subnet for your tunnel from your PoP via your SixXS homepage. Once approved, you will get a /48 prefix for your own use: 2^80 1.2 heptillion addresses which are yours to assign to every dust particle in your office or home, if you so desire. The way I set it up is to add the first of these addresses to your internal interface on the gateway, by adding the following two lines to the interface s stanza in /etc/network/interfaces; you will need the iproute package installed (which you should be using for everything network-related anyway):

up ip -6 addr add 2001:41e0:ff12::1/64 dev $IFACE
down ip -6 addr del 2001:41e0:ff12::1/64 dev $IFACE

Instead of bringing the interface down and up, just run ip -6 addr add 2001:41e0:ff12::1/64 dev eth0. Note the use of the /64 prefix instead of the /48 that got assigned, leaving only 20 pentillion addresses. Oh no! The reason for this is buried in the specs: basically, /48 is a site prefix, but individual networks should not be larger than /64, which is the prefix length of links in the IPv6 domain. Now is also a good time to enable IPv6 forwarding, e.g. like so:

# echo net.ipv6.conf.all.forwarding = 1 >> /etc/sysctl.conf
# sysctl -p /etc/sysctl.conf

Obviously, you will also need to change the policy on the ip6tables FORWARD chain. For now, let s just set it to accept. You should later create a proper ruleset, though!

# ip6tables -I FORWARD -j ACCEPT

Bringing IPv6 to your local network The final step is to spread the love to your local network. Refrain from selecting addresses from your subnet and assigning them to the local hosts, or wondering how to configure the DHCP server, because IPv6 does it differently: your gateway will advertise its routes (which includes a default route) to your network, and each host will pick an address based on its MAC address (unless it already has an EUI-64 address assigned. This all happens automagically, at least with current Debian and Windows machines. On the gateway, you need to install radvd and simply tell it which prefix to use on which interface. My /etc/radvd looks like this, and I won t explain it:

interface eth0
 
  AdvSendAdvert on;
  prefix 2001:41e0:ff12::/64
   
   ;
 ;

Note again how we advertise a /64 network rather than the /48 we own . You cannot advertise smaller networks if you want automatic configuration to work, and you should not use networks larger than /64 in any case. If 2^64 addresses are not enough for you, I trust you ll be able to figure out how to advertise another of your 65536 /64 prefixes in the /48 subnet to appropriate hosts. Restart radvd and run over to another host to witness how it automagically gets connected to the IPv6 network by scanning /var/log/kern.log and watching the output of ip -6 addr and ip -6 route. Try ping6ing from there! Find the dancing turtle! It should all work. If you don t like the automagic aspect of this, look into stateful configuration, using DHCPv6, as provided by dibbler-server and ?wide-dhcpv6-server.

Resolving names Take note of the IPv6 address of each host. There s a way to determine it given the host s MAC address, but this is easier (ipv6calc is also useful). You might want to let your local DNS server know by adding AAAA records in parallel to the existing A records, and possibly even adding PTR entries. If you re serious about IPv6, you can tell SixXS to delegate reverse lookups for the IPv6 addresses to your DNS servers, but you ought to refrain from polluting the DNS namespace. Note that bind9-host provides an improved host tool, which fetches all kinds of information about names, not just the one single information configured as default:

% host pulse.madduck.net
pulse.madduck.net has address 130.60.75.74
pulse.madduck.net has IPv6 address 2001:41e0:ff1a::1
pulse.madduck.net mail is handled by 99 b.mx.madduck.net.
pulse.madduck.net mail is handled by 10 a.mx.madduck.net.
% host 2001:41e0:ff1a::1
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.a.1.f.f.0.e.1.4.1.0.0.2.ip6.arpa
domain name pointer pulse.madduck.net.

Oh, and if you re really that curious about how IPv6 addresses are computed from MAC addresses, read RFC 2464. Basically, given a prefix 2001:41e0:ff1a:: and a MAC address aa:bb:cc:dd:ee:ff, the resulting IPv6 address is obtained by:

1. inserting ff:fe into the middle of the MAC address to yield aa:bb:cc:ff:fe:dd:ee:ff;
2. flipping the second lowest bit of the first octet to yield a8:bb:cc:ff:fe:dd:ee:ff;
3. removing the odd colons to yield a8bb:ccff:fedd:eeff, the EUI-64;
4. concatenating the prefix and this result to get 2001:41e0:ff1a::a8bb:ccff:fedd:eeff.

If you find your (Windows) IPv6 addresses changing all the time, you might be faced by privacy features .

Remaining issues Even though my IPv6 connectivity works, I have two remaining issues.

Sending larger amounts of data to the network I am experiencing a curious issue where outgoing ssh IPv6 connections time out and outgoing data transfers hiccup. I have yet to find out what s going on.

Mapping names to laptops Laptops generally have two interfaces, one with a cable, and the other wireless. Both of these interfaces will have separate MAC addresses, and by extension, the laptop will have different IPv6 addresses depending on how it is connected to the local network. I want to be able to connect to laptops without knowing the medium they use to connect to the network. Unfortunately, there seems to be no feasible way. The solutions I see are:

• override the MAC address of one interface with that of the other, which is going to cause bgi problems in the case when the laptop (accidentally) gets connected to the same network twice;
• add both IPv6 addresses as AAAA records to the laptop s DNS name, which will cause random delays when connecting as the resolver may return the currently inactive address first;
• set up mobile IPv6, e.g. by following this Mobile IPv6 how-to, which would allow accessing the laptop uniformly, no matter where in the world it is. Unfortunately, Debian s support for Mobile IPv6 is severly lacking at time of writing. Also, Yves-Alexis Perez notes that this how-to is horribly outdated and promised to tend to it Real Soon Now .

The second solution works for me for now, but I am interested in the third. In response to this document, Andreas Henriksson has suggested the replace the stateless configuration (radvd) with stateful configuration, using DHCPv6. I have yet to investigate this option. Jeroen Massar suggests to unite cable and wireless into a bridged interface, which seems like a very good idea.

Credits Thanks to Bernhard Schmidt, William Boughton, and Jeroen Massar, and everyone on #ipv6/irc.freenode.org for their help over the past few weeks, and all those who fed back comments in response to this document!

Because I found it a tad non-obvious here a little tutorial. First we enable serial access to grub. For this add the following to your /boot/grub/menu.lst (tested with Grub legacy):

serial --unit=0 --speed=19200
terminal --timeout=5 serial console

I chose such a low baud rate because it worked and I didn't want to spend more time on it and because the highest baud rate (115200) had some timing problems with that specific configuration. Then we need to adjust Xen to send the hypervisor output to the serial console as well (again in /boot/grub/menu.lst, this probably applies this way to Debian only):

## Xen hypervisor options to use with the default Xen boot option
# xenhopt=com1=19200,8n1 console=com1,vga sync_console

Debian's update-grub uses the comments in the file to generate appropriate menu entries, even if the kernel and hypervisor versions change. Thus editing a comment works, but you must remember to call update-grub after your changes, otherwise they won't be active! The last thing to change is the kernel's console. You might specify multiple consoles on the kernel's command line but only the last one will be the primary where the boot messages from init and the boot scripts are sent to. Kernel messages will end up on both, however. The edited comment looks like this (without Xen you would change kopt instead):

## Xen Linux kernel options to use with the default Xen boot option
# xenkopt=console=tty0 console=ttyS0,19200n8

The key change which enabled me to use the serial console was passing sync_console to Xen. Sadly it complains now on boot that this option should not be used on production systems because it ensures that all messages reach the serial console and wait if necessary. But in my humble opinion it should only hurt on output and there should not be that much kernel and hypervisor output anyway, except if there is something wrong with the machine. Serial output from Linux only was also messed up when Xen hadn't this option activated. To access the serial console just invoke GNU screen as a user in group dialout or as root like this to get the serial console as screen's first window:

# screen /dev/ttyS0 19200

Now the only thing that's missing is an affordable remotely controllable power line to trigger power cycles... Update: Liberie Cunha-Neto kindly pointed me to the Xen hypervisor command line, from which one could reboot the machine. It is invoked by pressing Ctrl-a three times on the serial console (it does not work on the real one I told). In screen you need Ctrl-a, a instead, as Ctrl-a is its command prefix. Then you can get help by pressing h and reboot the machine by pressing R.

So I've been inexcusably tardy in announcing this, but Obey Arthur Liu is working on aptitude for Google's Summer of Code program. His goal is to write a third interface to the program, using GTK+ to produce the UI; as the abstract for his project states:

I will create a GTK+ GUI for Aptitude that will work alongside improved current ncurses and command-line interfaces. This will offer an alternative to Synaptic with an interface design geared toward usability and advanced functionality.

I'm looking forward to seeing what he comes up with! I think there are a lot of interesting possibilities in the design space of GUI package managers; hopefully he'll have time to explore at least a few of them. At the moment he's focusing on figuring out the care and feeding of the apt library backend and learning some of the quirks you have to deal with when writing a frontend against it. Arthur's blog can be found at http://www.milliways.fr and on Planet Debian. The Mercurial repository for his work can currently be found at http://dev.graffit.net/aptitude/hg/.

Or configuring LS COLORS After successfully Theming Emacs, I decoded to turn my efforts to providing more copacetic colors in other aspects of my working environment. I ll write more about my effort to write my own color theme for fvwm when it is closer to being done. For now, I ll concentrate on ANSI Colors in XTerms, and how one may theme those. The old Tektronics (or ANSI) terminals had 8 colors, namely, black, red, green, yellow, blue, magenta, cyan, and white. These colors had a normal, and a bright mode. The default set of colors is OK, but is not optimal, since they have to work on both dark and light backgrounds; however, I use mostly dark background colors. Given that, I set to design my own colors, using fully laturated colors, with a base luminosity of 70, and equally spaced in chroma and used the same saturation and chroma, but a luminosity of 80, for the bright set. This results in a pleasant pastely set of colors, which are e4asy on the eyes, when used on very dark backgrounds. The configuration can be seen in XTerm. I set the colors for rxvt as well as xterm. Now, the most noticeable effect of these changes is if using the color option for ls, therefore the next thing to do was to configure the colors used by ls using a utility dedicated for that task, namely, dircolors. By trial and error, I modified the default configuration provided by dircolors and ended up with dir colors, with the results as you see.

The Linux Plumbers Conference[1] now has a call for papers, with registration also being open. Why is this important? Because it's the first conference aimed at getting together developers from all walks of the Linux world and fixing the core infrastructural problems. Ever looked at a kernel interface and wondered how the christ any userspace programmer was supposed to use it? Ever looked at a desktop application and wondered what kind of $2 crack they were on when they wrote it? Here's your chance to find out or get it fixed.

I'm running the power management microconf. We're looking at discussing what sort of interfaces the kernel needs to export to userspace and how userspace needs to make use of them. Want to be part of engineering the first OS to have high quality power management all the way from embedded to servers? Be in Portland in September. Ideally at the conference rather than a bar, though there'll be plenty of time for that as well.

So, if you have opinions on power management (or any of the other topics), sign up now. Otherwise we'll ignore them and you'll be unhappy.

[1] Plumbers? Plumbers'? Plumbers's? Almost certainly not Plumber's.

On the testing and unstable systems that i run, i occasionally get a disturbing "there are N newly obsolete packages" message. For example

[0 root@squeak ~]# aptitude update
Hit http://cmrg.fifthhorseman.net unstable Release.gpg
 [...]
Get:42 http://ftp.us.debian.org unstable/main [2077kB]                          
Fetched 15.2MB in 57s (262kB/s)                                                 
Reading package lists... Done
Updating debtags database...... Done
Current status: 3 updates [+3], 1136 new [-21212].
There are 1804 newly obsolete packages.
[255 root@squeak ~]# 

Usually, i can fix this with another aptitude update. Does this mean i've hit a mirror at an inopportune time? Or do i have apt mis-configured somehow? Other people have documented this before, and casual conversation with friends lets me know it's not just me. If this is due to some inconsistency at the mirrors (particularly for suites with rolling updates?), is there some way to engineer the mirror transfers so that this doesn't happen? For example (i have no idea how this stuff is done, so i'm just making this up):

1. rsync the pool/ without --delete to all mirrors,
2. rsync the Packages and Release, and DiffIndex files to all mirrors,
3. rsync the pool/ with --delete

Or should apt itself deal with these circumstances differently? It seems like signed apt ought to be able to detect when something like this is amiss. I'm reluctant to consider the situation acceptable, because i'd like to be able to trust apt in general. In particular, i can imagine apt eventually becoming more assertive about suggesting removal of newly obsolete and unsupported packages. If it is wrong about what's obsolete and unsupported, this could be dangerous.

The newspapers brought good news bits in the last two days. P blico reports on Paco Rivi re's ongoing quest to get a refund for the extra money he had to pay for a Windows licence when he bought a laptop. Paco is a well known member of the Ubuntu Catalan community and has been battling for this common-sense right for 3 years. The trial took place last Monday, and hopefully he'll be able to report some good news soon. In totally unrelated news, the Valencian caveman Juan Garc a Sentandreu, leader of the right-wing Coalici n Valenciana party, was arrested yesterday, for still not too clear reasons. Being one of the biggest enemies of my language, and having a long record of violent attacks to cultural entities and other political parties in Val ncia, I can't say I pitty him at all. I hope he had fun sleeping with the yonkis in the central police station last night. :)

I maintain a bunch of SSL certificates, mostly signed by my own site authority. Too many not to automate, but not enough to warrant heavy machinery. Here's how I do it. The configuration files Each certificate needs a config to describe what's in it. I create each of these and name it with a .cnf suffix. Here's an example:

[ req ]
prompt                  = no
distinguished_name      = server_distinguished_name
[ server_distinguished_name ]
commonName              = server.usefulinc.com
stateOrProvinceName     = England
countryName             = GB
emailAddress            = edd@usefulinc.com
organizationName        = Useful Information Company
organizationalUnitName  = Hosting
[ req_extensions ]
subjectAltName=edd@usefulinc.com
issuerAltName=issuer:copy
nsCertType            = server
[ x509_extensions ]
subjectAltName=edd@usefulinc.com
issuerAltName=issuer:copy
nsCertType            = server

Let's say this config is server.cnf. I then just type make server.pem to generate the corresponding certificate and key, signed by my local certificate authority. As I don't want to attend the startup of every service, I ensure the key is password-less. The Makefile rules Here are the makefile steps I use to generate and sign keys.

.SUFFIXES: .pem .cnf
.cnf.pem:
        OPENSSL_CONF=$< openssl req -newkey rsa:1024 -keyout tempkey.pem -keyform PEM -out tempreq.pem -outform PEM
        openssl rsa <tempkey.pem >  basename $< .cnf _key.pem
        chmod 400  basename $< .cnf _key.pem
        OPENSSL_CONF=./usefulCA/openssl.cnf openssl ca -in tempreq.pem -out  basename $< .cnf _crt.pem
        rm -f tempkey.pem tempreq.pem
        cat  basename $< .cnf _key.pem  basename $< .cnf _crt.pem > $@
        chmod 400 $@
        ln -sf $@  openssl x509 -noout -hash < $@ .0

The resultant files are:

• server.pem — contains both certificate and key in one file
  
• server_crt.pem — certificate file
  
• server_key.pem — key file
  

Some notes on these steps: my site-local certificate authority is in the directory usefulCA, along with an OpenSSL config which describes my preferences. This config was created by copying and making appropriate adjustments to the default /etc/ssl/openssl.cnf which ships with Debian. For generating certificate signing requests to ship to a commercial certificate authority, it's a bit simpler. I save the config files with a .reqcnf suffix instead, and use this rule:

.SUFFIXES: .pem .cnf .reqcnf .csr
.reqcnf.csr:
        OPENSSL_CONF=$< openssl req -newkey rsa:1024 -keyout  basename $< .reqcnf .key -keyform PEM -out  basename $< .reqcnf .csr -outform PEM

And finally, a rule I use to sign incoming certificate requests from other systems:

.csr.pem:
        OPENSSL_CONF=./usefulCA/openssl.cnf openssl ca -in $< -out  basename $< .csr _crt.pem

I offer these without warranty in the hope they might be useful to somebody. They're not much more than a transcription of a how-to into a makefile, but it's just enough technology to ensure creating certificates isn't a big nuisance. Further reading

• A more complete guide to setting up a site-local certificate authority
  

Notes Why do I bother with a site-local CA, rather than just self-sign? It lets me bypass the annoyance of SSL warnings on clients once I've installed my own CA certificate, and gives me a coarse grained level of access control: for instance, only clients with certificates signed by my CA are allowed to access the site's LDAP server. My personal next step with this is to integrate the certificate production process with my emerging Puppet recipes for managing local infrastructure.Join the conversation about this post

Joys of schroot Schroot is good! Try schroot. You have LVM set up and some free extents in a vg? Try this: Then apt-get install schroot and put this in your /etc/schroot/schroot.conf: Now, schroot -c sid will make a LVM snapshot of /dev/YOUR-VG-NAME/schroot-sid, mount it, set it up as a nice working system, and chroot into it running a shell. Have a look around: resolv.conf is set up, your home directory is bind mounted, /proc and /sys are mounted. When you exit the chroot, the LVM snapshot disappears, and /dev/YOUR-VG-NAME/schroot-sid is unchanged. If you want to modify /dev/YOUR-VG-NAME/schroot-sid, you just schroot -c sid-source. In my laptop, I did this: At this point, to build a debian package like if I were using pbuilder, I do this: You can repeat this whole process with lenny and etch, for example, and if you are on an amd64 system, you can also do sid32, lenny32 and etch32. If you don't want to redownload packages all the time, install approx and configure the -source chroots to use it. This also allows you to easily switch all systems from one mirror to another if, for example, you are traveling abroad. If you want to do the same as pbuilder update, you can do this: You can use schroot -c sid-source for all sorts of permanent maintenance works. You can also create persistent named snapshots: This will only disappear when you do: And you can enter the schroot as many time as you want with: You can also enter a temporary snapshot as many times as you want, by using the random name that schroot generated for it (you can see it, for example, when running df or mount). You fear that lvm snapshots are a little fragile and a little slow at the moment? It's not a problem: your home is bind-mounted inside the chroot, so you can do all the work in your normal home filesystem, without stressing the snapshot. This is good not only to build packages. For example, now that svk is broken in sid and lenny, I just did schroot -b -c etch -n svk, installed svk on it, and then every time I need svk, I can do schroot -r -c svk and have it ready.

Passing values to turbogears widgets at display time (the general case) Last time I dug this up I was not clear enough in documenting my findings, so I had to find them again. Here is the second attempt. In Turbogears, in order to pass parameters to arbitrary widgets in a compound widget, the syntax is: And if you have more complex nested widgets and would like to know what goes on, this monkey patch is good for inspecting the params lookup functions: The code for the lookup itself is, as the name suggests, in the retrieve_params_by_path function in the file widgets/forms.py in the Turbogears source code.

OK, so I've been hacking on and testing my shiny new One A110 mini-laptop during the last few days and I must say I'm very happy with it. I'll write up some more details later (check the wiki if you're impatient), but today I want to highlight a very nice feature of this laptop (compared to, for instance, the Eee PC): The VIA C7-M ULV CPU in the laptop has VIA Padlock support. VIA Padlock is a hardware feature in recent VIA CPUs which provides hardware-accelerated AES and SHA-1/SHA-256 support, among other things. This can be used in Linux (with the proper drivers and patches) to improve performance of dm-crypt, OpenSSL (and all programs using it), scp, sha1sum, OpenVPN, etc. etc. I have written a quite extensive VIA Padlock HOWTO and benchmarks in the A110 wiki (but all of this will work on other systems which have VIA Padlock, too). To summarize, here are the most important benchmarks: dm-crypt (256bit AES, cbc-essiv:sha256) Without VIA Padlock support:

$ hdparm -tT /dev/mapper/hdc2_crypt
/dev/mapper/hdc2_crypt:
 Timing cached reads:   448 MB in  2.00 seconds = 223.47 MB/sec
 Timing buffered disk reads:   22 MB in  3.07 seconds =   7.17 MB/sec

With VIA Padlock support:

$ hdparm -tT /dev/mapper/hdc2_crypt
/dev/mapper/hdc2_crypt:
 Timing cached reads:   502 MB in  2.00 seconds = 250.41 MB/sec
 Timing buffered disk reads:   90 MB in  3.07 seconds =  29.36 MB/sec

The native speed of the SSD in the laptop is 31.01 MB/sec, so there is almost no performance penalty when using VIA Padlock. OpenSSL OpenSSL speed benchmark, first line without Padlock, second line with Padlock enabled:

$ openssl speed -evp aes-256-cbc [-engine padlock]
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc       9187.18k    10572.28k    11054.32k    11179.36k    11218.02k
aes-256-cbc      47955.92k   150619.73k   325730.73k   458320.11k   520520.79k

ssh/scp Without VIA Padlock support:

$ scp -c aes256-cbc bigfile.dat localhost:/dev/null
bigfile.dat                100%  159MB   5.9MB/s   00:27

With VIA Padlock support:

$ scp -c aes256-cbc bigfile.dat localhost:/dev/null
bigfile.dat                100%  159MB  14.5MB/s   00:11

OpenVPN A real speed benchmark is pending (not measurable easily on 100MBit LAN, will try on a slower link), but as OpenVPN uses OpenSSL is should have roughly the same speedup iff you tell OpenVPN to use AES (it uses Blowfish per default). Also, there's a measurable difference in CPU load while tranferring large files over OpenVPN: 8% CPU load with VIA Padlock (vs. 20% CPU load without VIA Padlock). sha1sum / phe_sum phe_sum is a small C program which can be used as drop-in replacement for sha1sum (which doesn't support VIA Padlock yet). Quick benchmark: sha1sum, without VIA Padlock:

$ time sha1sum bigfile.dat
real    0m6.511s
user    0m5.864s
sys     0m0.412s

phe_sum (with VIA Padlock support):

$ time ./phe_sum bigfile.dat
real    0m1.149s
user    0m0.704s
sys     0m0.424s

All in all VIA Padlock gives you a pretty impressive speedup for many crypto-using applications on Linux, which is especially useful on the A110 mini-laptop (think OpenVPN or scp for mobile usage, and dm-crypt for an encrypted SSD, of course).

Monkeytail is a neat little tool which allows web developers and sysadmins to easily watch new additions to their log files by tailing them (i.e. tail -f) and providing helpful vertical spacing after detecting pauses in the output.

What makes monkeytail so useful is its ability to easily tail multiple files together, even when it involves connecting to these machines over ssh and running sudo on them. For example, you could tail all of your load-balanced servers by doing:

mtail @webservers

where webservers is an alias defined in ~/.mtailrc:

<group webservers>
    sudo yes
    <file>
        filename /var/log/apache2/error.log
        prefix "server1: "
        host server1.example.com
    </file>
    <file>
        filename /var/log/apache2/error.log
        prefix "server2: "
        host server2.example.com
    </file>
</group>

It is available in Debian unstable and from its official homepage.

Kudos to Martyn Smith for turning this idea into a release!