I've always been bothered by those "Property Of Blank University" t-shirts that used to actually be the loaned (or stolen) property of college athletic departments but have now become popular enough that you can find them, for sale, in nearly any university store or gift shop in the US. Few people would assume that somebody with a "Property of" shirt had stolen their clothing. In fact, it's often impossible to find the shirts except on sale anymore -- and rarely from universities themselves. Here's my response. For those that don't know (and that's certainly many), Pierre-Joseph Proudhon is the nineteenth century French anarchist and mutualist most famous for saying, "La propri t , c'est le vol!" In English: "Property is theft!" You can buy my t-shifts (red on black, where possible), in my Printfection store. Source SVG is here. Please share variations in a comment.

I spent Saturday at BarCamp Belfast. There were more people than I expected; I'm no good at judging exact numbers but the wiki lists 62. As it turned out not everyone ended up talking; this worked out well for me because I don't think I'm Web2.0 enough (is that a bit ironic given I'm writing this on a blog?). I don't really do user interfaces or website stuff and that seemed to be the primary focus of a lot of the attendees. Nice to know there's a fairly active crowd in
Belfast though; I discovered the existence of Belfast Open Coffee which I'll have to try and make at some point.

I went to at least the following talks::

Gerard McBreen on Digital Circle & Adobe Flex: Quite a cute demonstration of the Flex framework for Flash. I'd have been more convinced if Flash could be considered truly cross platform; Adobe can't even manage a 64 bit client yet and Gnash is far from ready.

Daniel McLaughlin on Nagios: Not really anything I didn't already know, but always interesting to see how other people are using these things to see if there's any nice tricks you've missed.

Andrew Gribbon from Exile Coffee on speciality coffee: This was really interesting. A complete tangent from IT but enjoyable and I must buy some roasted beans from him (I've been meaning to place a Roast and Post order for ages anyway).

Brian O'Neill on freelancing: Tailored towards the web market in my opinion, which is fair enough but things like aiming to never employ anyone and stay a one man operation doesn't really work if you're doing something that a customer wants supported around the clock. Or even just during the working day if you're crossing time zones. Interesting to hear that Brian had run an agency in the past but preferred working on his own these days. I kinda feel I've gone a bit the other way. :)

Linux Hardware workshop: Not really a lot of interest here; a clutch of EEEs, an N800 and a Debian box or two. However I had a chat with David Newman about his MythTV setup which convinced me I really should give it another try. After I get someone to install a bigger aerial in the hope of stabilising our Freeview reception.

Brian Cleland on Linux Trends: Talking about lots of Linux takeup by big names and general market adoption eg the recent selection of cheap subnotebooks. Main thing of interest was OpenPandora which I hadn't heard of before; OMAP3 based handheld device running Linux with a 800x480 4.3" screen. Aimed as a gaming device, but has a keyboard and bluetooth/wifi so has potential as a PDA as well. Hopefully the build quality will be better than the GP2X I saw.

Andy McMillan on coffee shop working: I've thought about this in the past myself; I worked from Cafe Java in Norwich a few times while doing Black Cat, but in general didn't find it as conducive to work as I'd hoped. He mentioned Coworking Belfast which is an idea Tristam has talked about in the past, in terms of people who work from home getting to have some form of social contact with like minded individuals, without having to go to the expense of a whole office. It'll be interesting to see if it takes off.

Darragh Doyle and Grannymar on blogging: I think everyone in the audience already had a blog of some description. I don't really consider my blog a social forum; it's not something I expect to have a long conversation on, just ask a quick question or discuss something
I've done that I think might interest people or give a general "this is what's happening at present" post. Blog commenting systems are all a bit crap to keep up anything sustained. Someone needs to write a blog<>nntp gateway that correctly cross posts comments as well. I suppose the issue with that is then everyone needs to run a news server, or all agree on
using the same one. Fine for groups of people who know each other but not if you want to read a dozen blogs from different areas in a single client that'll let you properly follow conversations. Mind you, people seem to cope with forums and I don't really understand how. Supporters of news and mailing lists seem to be dwindling. :(

Anyway, that was my first BarCamp. I wasn't quite sure what to expect, it was a bit more Web2.0 than I'd have liked (which is fair enough, but I'd have liked to have seen some talks about low level bits really) but I'd go again and will keep an eye out for the next one. And hopefully I'll see some of the people I met there at the BLUG meeting this Wednesday (Tap House in Lower Crescent from 6pm for anyone who's not on the mailing list).

If you want a reliable network then you need to determine an appropriate level of redundancy. When servers were small and there was no well accepted virtual machine technology there were always many points at which redundancy could be employed. A common example is a large mail server. You might have MX servers to receive mail from the Internet, front-end servers to send mail to the Internet, database or LDAP servers (of which there is one server for accepting writes and redundant slave servers for allowing clients to read data), and some back-end storage. The back-end storage is generally going to lack redundancy to some degree (all the common options involve mail being stored in one location). So the redundancy would start with the routers which direct traffic to redundant servers (typically a pair of routers in a failover configuration - I would use OpenBSD boxes running CARP if I was given a choice in how to implement this [1], in the past I’ve used Cisco devices). The next obvious place for redundancy is for the MX servers (it seems that most ISPs have machines with names such as mx01.example.net to receive mail from the Internet). The way that MX records are used in the DNS means that there is no need for a router to direct traffic to a pair of servers, and even a pair of redundant routers is another point of failure so it’s best to avoid them where possible. A smaller ISP might have two MX machines that are used for both sending outbound mail from their users (which needs to go through a load-balancing router) as well as inbound mail. A larger ISP will have two or more machines dedicated to receiving mail and two or more machines dedicated to sending mail (when you scan for viruses on both sent and received mail it can take a lot of compute power). Now the database or LDAP servers used for storing user account data is another possible place for redundancy. While some database and LDAP servers support multi-master operation a more common configuration is to have a single master and multiple slaves which are read-only. This means that you want to have more slaves than are really required so that you can lose one without impacting the service. There are several ways of losing a server. The most obvious is a hardware failure. While server class machines will have redundant PSUs, RAID, ECC RAM, and a general high quality of hardware design and manufacture, they still have hardware problems from time to time. Then there are a variety of software related ways of losing a server, most of which stem from operator error and bugs in software. Of course the problem with the operator errors and software bugs is that they can easily take out all redundant machines. If an operator mistakenly decides that a certain command needs to be run on all machines they will often run it on all machines before realising that it causes things to go horribly wrong. A software bug will usually be triggered by the same thing on all machines (EG I’ve had bad data written to a master LDAP server cause all slaves to crash and had a mail loop between two big ISPs take out all front-end mail servers). Now if you have a mail server running on a virtual platform such that the MX servers, the mail store, and the database servers all run on the same hardware then redundancy is very unlikely to alleviate hardware problems. It’s difficult to imagine a situation where a hardware failure takes out one DomU while leaving others running. It seems to me that if you are running on a single virtual server there is no benefit in having redundancy. However there is benefit in having an infrastructure which supports redundancy. For example if you are going to install new software on one of the servers there is a possibility that the software will fail. Doing upgrades and then having to roll them back is one of the least pleasant parts of sys-admin work, not only is it difficult but it’s also unreliable (new software writes different data to shared files and you have to hope that the old version can cope with them). To implement this you need to have a Dom0 that can direct traffic to multiple redundant servers for services which only have a single server. Then when you need to upgrade (be it the application or the OS) you can configure a server on the designated secondary address, get it running, and then disable traffic to the primary server. If there are any problems you can direct traffic back to the primary server (which can be done much more quickly than downgrading software). Also if configured correctly you could have the secondary server be accessible from certain IP addresses only. So you could test the new version of the software using employees as test users while customers use the old version. One advantage a virtual machine environment for load balancing is that you can have as many virtual Ethernet devices as you desire and you can configure them using software (without changing cables in the server room). A limitation on the use of load-balancing routers is that traffic needs to go through the router in both directions. This is easy for the path from the Internet to the server room and the path from the server room to the customer network. But when going between servers in the server room it’s a problem (which is not insurmountable, merely painful and expensive). Of course there will be a cost in CPU time for all the extra routing. If instead of having a single virtual ethernet device for all redundant nodes you have a virtual ethernet device for every type of server and use the Dom0 as a router you will end up doubling the CPU requirements for networking without even considering the potential overhead of the load balancing router functionality. Finally there is a significant benefit in virtual machines for reliability of services. That is the ability to perform snapshot backups. If you have sufficient disk space and IO capacity you could have a snapshot of your server taken every day and store several old snapshots. Of course doing this effectively would require some minor changes to the configuration of machines to avoid unnecessary writes, this would include not compressing old log files and using a ram disk for /tmp and any other filesystem with transient data. When you have snapshots you can then run filesystem analysis tools on the snapshots to detect any silent corruption that may be occurring and give the potential benefit of discovering corruption before it gets severe (but I have yet to see a confirmed report of this saving anyone). Of course similar snapshot facilities are available on almost every SAN and on many NAS devices, but there are many sites that don’t have the budget to use such equipment.

• [1] http://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol

Share This

Part of the joy of spending 3 months stoned is that my Inbox had managed to fill up quite nicely with all sorts of crap. In a desperate attempt to prune things, I've taken everything from LiveJournal and Facebook and archived them.

The LJ things are often bits where people have wished me well about the surgery and the recovery. But at over 80 conversations (some of which were multiple people), I'll just *never* get to them. I love you all for the well wishes, and I hope that I've managed to connect with all of you after them at some point.

I've also done the same with Facebook emails on the theory that Wall posts are probably stale by now (or already replied to), and the Inbox there will tell me what I haven't replied to (something like 77 posts)

I started the evening at 1350 messages, I'm now down to 497. Hopefully I'll get it down to zero over the course of this week.

Bed time!

It happened a number of times now that my inbox would shrink in message count without my explicit doing. Generally, your inbox automatically emptying should be conceived a good thing, but it isn t always. It took me a while to put together the pieces:

• left over messages are all new or old, meaning not read ,
• offlineimap logs suggested that I deleted those messages on one of my workstations, before offlineimap synchronised those changes to the other machines, and
• it seems to coincide with instances of use of Mozilla Thunderbird and maybe other IMAP clients.

I think I unvealed the mystery: some IMAP clients automatically mark read messages as deleted. Don t ask me why, I did not configure it, and even though I told Thunderbird specifically not to do it, I have no other explanation than to assume that it doesn t care about what I want, but marks them for deletion anyway. Firefox decides to block cookies several times a day, despite my explicit requests to store them, and the two are from the same project, so it seems plausible. Once marked for deletion (by way of an IMAP flag), offlineimap propagates the flag to all clients. Since I set delete=yes for mutt, if I then open and close a mailbox with such messages without noticing them, the messages are purged. I gave up fighting and solved the problem at a different point, namely mutt (which was doing the deleting anyway):

folder-hook . push '<undelete-pattern>~D<enter>'

Since mutt deletes mail marked for deletion when I close a mailbox, finding those messages at time of mailbox opening must mean that they have been marked outside of the mailer I use mutt for everything, exclusively. So let s undelete them. I can t see any negative consequences of the above hook. NP: Oceansize: Efflorescence

Or theming PS1 I am not fanatical about my command prompt. No sirree, not me. It is just that I spend half my life either staring at an Emacs window, or at an xterm command prompt; so even a marginal boost in productivity goes a long way. And I am often logged in over ssh to machines half a continent away, and am still comfortable enough on the Linux VT to spend time there, and I often do not have the GUI gew-gaws feeding me data. Hmm. Data. inpuuut. No. Must focus. With all these open xterms and Emacs terminal mode frames floating around, it is easy to lose track of where I am on each terminal, and what the working directory is. So I want my command prompt to help me keep track of where I am. If the terminal is an xterm, the title can be setup like

 "user@host:../shotened/path/to/current/working/dir"

I want to know what machine, id (am I root?), and directory I am in. If I am deep down in the labyrinths of some work related directory tree, I want the path to be pruned, from the left, one component at a time. However, this does not help me on the console; so I also want the path to be in the command prompt; but it should not take up too much of the command line; and ideally, should just go away as I type a longish command. Gawd, I love zsh. I once did a analysis of the command history. The most often used command sequences were

 cd some-place, ls

So. pwd and ls. I really really want to see the directory listing when I change directory to a new one. As one grows old, memorizing the directory contents for dozens of machines all the time taxes the gray cells a bit. So, I figure, why not let the command prompt handle all that? Having the current working dir always visible cuts down a heck of a lot on the pwd commands, and so all that s left is to insist that the command prompt thingy always run ls after a change directory. Simple enough. Saves on typing. And time. And this is not just some crazy talk. I want help with noticing whether the previous command exited with an error status (useful for commands that normally do not create an output). If I am logged in to a machine on battery power, I want to know that. I also like visual cues to the amount of power remaining (good for my laptop on the long flights). I don t want to have to know f the machine uses APM or ACPI, I just want my prompt color to change as the power fades. I want my command prompt to let me know if I am in a directory which is under version control, and if so, what branch I am on. (I occasionally have to come in contact with arch, bzr, git, subversion, svk, and mercurial). If I am in a version controlled project, where I am relative to the root of the checked out tree is often more important than the absolute path, so I want to see relative paths, not absolute paths. I want to know if there are uncommitted files in the working directory. Visually. I want to be reminded if I am in the middle of an ongoing

 rebase -i .

This is not asking for too much, is it? So, here is a screen-shot of this in action: I start from my home directory, go to a directory not under version control, go to a project under git, then a different project with uncommitted files, and then finally to a subversion checkout. All with an angry fruit salad of colors warrantied to make Martin Krafft want to claw his eyes out . If you use zsh, then just grab hold of this, and then do:

 autoload -U promptinit
 promptinit
 prompt manoj

This might be a bit of a hit on slow machines, but even my laptop is a core 2 duo, so I do not find it noticeable. Oh, and if you liked this article, you might also like Theming Emacs, and Theming XTerms.

Can't find any documentation about this on the web, and it's a PITA, so here goes. Note: No pictures. I was too scared to take time out to take pictures. Disassembly is not hard at first. Three screws hidden over the battery pack allow removing the cover over the power button and wireless switch, which then frees 4 tabs that lock the top of the keyboard down. My keyboard was also taped down with double-sided tape, and glued down. This is the part I hate, both because I still haven't accepted that the common way to put together consumer electorinics is tape and glue, and because it's never possible to get it apart undamaged. I pried the keyboard up from the top, pushing under to untape/glue, and taking care of the cable. There was also a flap of plastic stuck down over the cable, which had to be peeled away. Many clips on the bottom of the keyboard hold it in place and have to be slipped out. I ended up slightly bending the arrow keys of the keyboard in yanking it out, though I think they'd still be ok-ish. Luckily I was putting in a new one. Removing the keyboard cable was strange. Its socket is not the typical kind you can life to unlock. The socket is one piece and does not move at all. I eventually just pulled the cable out of it. This is where I started to get very nervous. Replacing the cable was a PITA. You have to bend its end at a 90 degree angle, and then just line it up and press it down into the socket. Hard. And rub along it and do whatever you can to get enough force exerted down the thin cable strip to make it slip far enough into the socket to work. I booted up the laptop several times while doing this, to test if I had managed to get it working. Eventually I had. Then I very carefully reseated the keyboard, and replaced the cover and screws. Unfortunatly, the tape and glue was doing something. The new keyboard is very springy in the middle, where that held it down before. So I will have to take it apart again and put on some new double-sided tape or something eventually. Conclusions:

• Fujitsu sucks for not springing for an extra 0.01 cents for a locking cable socket.
• Laptop manufacturers in general suck for using tape and glue. (Or for building weaghty mehemoths that use better fastening systems.)
• When it's been disassembled, a lot of the magic of a laptop seeps out for me and never really comes back. Maybe next time I should not do my own repairs, just to preserve that heady comsumer magic of not knowing what's inside?
• I'm still using Caps Lock for Esc, so why did I even bother?!
• New keyboards feel strange.

Even though I ve dealt with IPv6 for almost a decade, have delivered presentations, and given multi-day courses on IPv6 security aspects, I ve never actually added IPv6 to my own server/home network infrastructure because it seemed that Linux and/or Debian just weren t ready for it. This seems to have changed (although there are still a number of problems) and in less than a day, I put a few of my machines online. In the following, I d like to share with you how I did it.

Kernel versions and stateful connection tracking Unfortunately, I have to start off with some bad news: even though Debian etch, our current stable release, which uses a Linux kernel version 2.6.18, speaks IPv6, I cannot recommend it for deployment, as the 2.6.18 kernel does not support proper stateful connection tracking for IPv6, and thus makes it impossible to firewall hosts in a sensible manner (I always add local packet filters to all my hosts, and if only to guard against the situation when a user installs a malicious programme to listen on a high port). Of course, it is possible to configure a packet filter statelessly in an acceptable manner once you know the use case, so do with this information what you wish; I prefer to stay general for now. For me, a remedy is almost around the corner: the 2.6.24 kernel seems to support stateful connection tracking for IPv6, and it s even available for etch as it will be included in the upcoming etch-and-a-half release. I simply ended up using the kernel packages pre-release, and so far have not had a problem with it. To do so, add the following line to your /etc/apt/sources.list, making sure to use a close archive mirror:

deb http://ftp.xx.debian.org/debian etch-proposed-updates main

I then just upgraded the system and pulled in all proposed updates. As that may have let in software that won t be part of etch-and-a-half, or even lenny, you may want to pin the archive and only upgrade the kernel packages, by adding to /etc/apt/preferences (replacing amd64 with your architecture):

Package: *
Pin: release a=proposed-updates
Pin-Priority: -1
Package: linux-image-2.6.24-etchnhalf.1-amd64
Pin: release a=proposed-updates
Pin-Priority: 600

Alternatively, you could use the 2.6.24 linux kernel packages on backports.org.

Xen and IPv6 One drawback of switching to 2.6.24 is that you cannot run a dom0 on that machine any longer, so by practical extension, you cannot connect it to the IPv6 network with a packet filter in place. Supposedly, running 2.6.24 instances on a 2.6.18 dom0 is reported to work, however.

Configuring the packet filter The first thing I did was to configure the packet filter on each host appropriately. Unfortunately, this is harder than it should be, because to quote one of the netfilter developers when ip6tables was conceived, someone had a big bad brainfart : rather than adding IPv6 rules to your existing iptables ruleset, you have to create a new ruleset, duplicate all chains, networks, hosts, and individual rules, and maintain the two in parallel. Even though there are efforts of unification on the way, I speculate it ll take another couple of years until PF_INET6 will be fused into PF_INET and one will be able to do sensible cross-address-family packet filtering with Linux. Since I ve recently started to look (again) at pyroman, maybe the most logical way forward would be to extend it to write both, IPv4 and IPv6 rulesets from its knowledge about the hosts and networks you configured. Anyway, we want to get stuff working now! Thus, let s configure ourselves a packet filter. (Almost) all IPv6-related filtering must be configured via ip6tables (read on further down about IPv6 in IPv4 tunneling, the reason I said almost ). The following is a simple default ruleset to start with, which I put into /etc/network/ip6tables to load with ip6tables-restore:

*filter
:INPUT REJECT [0:0]
:FORWARD REJECT [0:0]
:OUTPUT ACCEPT [0:0]
:in-new - [0:0]
### INPUT chain
# allow all loopback traffic
-A INPUT -i lo -j ACCEPT
# RT0 processing is disabled since 2.6.20.9
#-A INPUT -m rt --rt-type 0 -j REJECT
# allow all ICMP traffic
-A INPUT -p icmpv6 -j ACCEPT
# packets belonging to an establish connection or related to one can pass
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
# packets that are out-of-sequence are silently dropped
-A INPUT -m state --state INVALID -j DROP
# new connections unknown to the kernel are handled in a separate chain
-A INPUT -m state --state NEW -j in-new
# pass SYN packets for SSH
-A in-new -p tcp -m tcp --dport 22 --syn -j ACCEPT
# log everything else
-A INPUT -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[INPUT6]: "
### OUTPUT chain
# RT0 processing is disabled since 2.6.20.9
#-A OUTPUT -m rt --rt-type 0 -j REJECT
# allow outgoing traffic, explicitly (despite chain policy)
-A OUTPUT -j ACCEPT
### FORWARD chain
# RT0 processing is disabled since 2.6.20.9
#-A FORWARD -m rt --rt-type 0 -j REJECT
# disallow forwarded traffic, explicitly (despite chain policy)
-A FORWARD -j REJECT
COMMIT

Note that this recipe is pretty much unusable on pre-2.6.20 kernels due to their broken implementation of stateful connection tracking. The ruleset should be fairly obvious, but you might wonder about my use of REJECT and allowing all ICMP after all, you ve heard for the past 30 years that ICMP is a bad hacker protocol , and Internet security is no domain for being nice to people, so to prevent any information disclosure, you should DROP connections, not let people know that they re simply not allowed. Well, to hell with all that! I don t see a single reason or attack vector that is foiled by DROP or disallowing ICMP. In fact, it s just security by obscurity, and might inconvenient at the same time. ICMP is also much more important with IPv6 than with IPv4 (it replaces ARP, for instance), and it s actually useful to be able to ping hosts, or get back informational messages on why something failed. Finally, rejecting traffic rather than dropping it doesn t suggest to a hacker that something s hidden here. Then there is RFC 4890, which almost made me puke. This document is part of the reason why I say: let s fix problems in the kernel, rather than shielding them with unreadable and unmanageable rulesets!

Getting connected If you already have an IPv6 address, you re basically ready to go, but may want to read further down on how to connect your local network to the IPv6 Internet as well. If you are searching for a provider, have a look at the list of providers with native IPv6 connectivity over at sixxs.net. If you are reading up to here, I assume you are connected to the Net with IPv4. There are two ways for you to move towards IPv6: 6to4 or by way of a tunnel provider. A Kiwi website explains how to setting up 6to4 connectivity, and thus I will concentrate only on the tunnel approach. Even though everyone can set up 6to4 in a breeze without any accounts or waiting, there are a number of security considerations, it s pretty ugly to debug (due in part to asymmetric routing), and makes your life unnecessarily difficult when all you have is a dynamic IP that changes from time to time. If you are stuck behind a NAT gateway, you cannot use 6to4 either. Thus, I prefer the tunnel approach. With the tunnel approach, IPv6 packets are wrapped up in IPv4 packets on your host, and sent to the IPv4 address of your tunnel provider, who has native IPv6 connectivity. The tunnel provider unwraps your packet and shoves the contained IPv6 packet onto the backbone. The IPv6 address you used as source address is routed to the tunnel provider, so any replies arrive at their machines, where they re again wrapped into IPv4 packets and sent to your (publicly-accessible) IPv4 address. Those IPv4 packets specify payload type 41 ( ipv6 ), which is why we need those -p ipv6 -j ACCEPT rules in the iptables ruleset. There are a few tunnel providers out there. I chose SixXS and have not regretted my choice. I shall thus assume that you do the same: sign up for an account right now, so that you have it by the time you finished reading this document! SixXS works on a credit system: tunnels and subnets cost credits, which you can accumulate by maintaining your tunnels properly. This ensures that everyone can play around, but to do more advanced stuff, you need to first display competence with the basic concepts. Your first step with SixXS will be to request a tunnel. SixXS offers three types of tunnels:

• static tunnels, for those with static IPs,
• heartbeat tunnels, for those with dynamic IPs, and
• AYIYA tunnels, for those behind NAT gateways.

Each of these tunnels have advantages and disadvantages, as everything does: the first two types of tunnels use IP protocol 41 packets to encapsulate the IPv6 packets. As such, there are security considerations involving the impersonation by spoofing, and all upstream firewalls must let protocol 41 pass. AYIYA addresses these problems by using signed packets, but that solution comes with extra computation overhead and smaller MTUs. I suggest to use the first type of tunnel that fits your situation. Debian s aiccu package can take care of heartbeat and AYIYA tunnels for you, and it can even set up static ones. During registration, you will also need to choose a PoP , which stands for Point of Presence . If your country only has a single PoP, that s the one you will end up using (unless you have a good reason for another one), but if there are more options, I strongly suggest that you go through the list of PoPs and select the one with the best roundtrip time and lowest latency from your location! Note that you must answer ping requests (ICMP echo-request) from the PoP you chose, or else the tunnel will not be created. Once your tunnel request gets approved, you ll get a /64 prefix, in which you only use two addresses: the PoP will configure the :1 address and you need to configure your host to use the :2 address on the tunnel interface. You ll also be told the IPv4 address of your PoP endpoint . Joey Hess taught me that aiccu can set up the interface for you, using the data it queries from the SixXS registration (TIC) server. I tried it, and it works. However, I prefer the pure ifupdown approach, as it makes things explicit and allows me to use the hooks for stuff like loading the packet filter. So in my /etc/network/interfaces, you can find:

auto sixxs
iface sixxs inet6 v4tunnel
  endpoint 194.1.163.40
  address 2001:41e0:ff00:3b::2
  netmask 64
  gateway 2001:41e0:ff00:3b::1
  ttl 64
  pre-up ip6tables-restore < /etc/network/ip6tables
  up ip link set mtu 1480 dev $IFACE
  up invoke-rc.d aiccu start
  down invoke-rc.d aiccu stop

Make sure to read about MTU values of the tunnel and adjust the 1480 value in the above to your tunnel settings and ISP connectivity. Also set ipv6_interface sixxs in /etc/aiccu.conf, if you are using aiccu, or else aiccu will bring up a duplicate/additional interface. If you tell it to use the same interface, it will actually execute all the same commands (which will fail), but won t report any errors. A future version will have a switch to prevent it from configuring the interface. Unfortunately, this will probably not work. The reason is that your regular IP packet filter (iptables, without the 6) doesn t let those encapsulating IPv4 packets pass, unless we tell it to; we probably want to do this early on in the chain, and also limit it to our tunnel peer, so:

iptables -I INPUT -p ipv6 -s 194.1.163.40/32 -j ACCEPT

For AYIYA, you need to open port 5072, either for UDP, TCP, or SCTP, depending on how you configured it. Also have a look at this FAQ entry on what a firewall needs to pass. If it still doesn t work, you have an upstream packet filter that needs some of those holes poked. Good luck. In most situations, the FORWARD chain does not get such a rule, since the tunnel terminates at the gateway, which routes to a native IPv6 network, or another tunnel. Allowing tunnels through a gateway is almost always a bad thing, as it would allow undetected and untraceable traffic from compromised boxes in the local network. The OUTPUT chain also does not need such a rule, if you have configured stateful filtering properly. Now bring up the interface and verify the connection:

# ifup sixxs
# ping6 -nc1 2001:41e0:ff00:3b::1
PING 2001:41e0:ff00:3b::1(2001:41e0:ff00:3b::1) 56 data bytes
64 bytes from 2001:41e0:ff00:3b::1: icmp_seq=1 ttl=64 time=74.0 ms
[...]
# ping6 -nc1 ipv6.aerasec.de
PING ipv6.aerasec.de(2001:a60:9002:1::184:1) 56 data bytes
64 bytes from 2001:a60:9002:1::184:1: icmp_seq=1 ttl=55 time=91.5 ms
[...]

Welcome to the Internet of the future!

Setting up an IPv6-capable gateway Your IPv6 connection works, but it s limited to a single address, and you do not get to specify the reverse DNS PTR record for it. Since the concept of NAT is mostly absent from IPv6 (thanks! thanks! thanks!), you will not be able to connect any other hosts to the IPv6 network. If your local network has a few hosts behind a gateway, you will need to request a subnet from SixXS and configure your gateway (which has the tunnel connection) appropriately. Don t worry, this is not really very difficult. First, request a subnet for your tunnel from your PoP via your SixXS homepage. Once approved, you will get a /48 prefix for your own use: 2^80 1.2 heptillion addresses which are yours to assign to every dust particle in your office or home, if you so desire. The way I set it up is to add the first of these addresses to your internal interface on the gateway, by adding the following two lines to the interface s stanza in /etc/network/interfaces; you will need the iproute package installed (which you should be using for everything network-related anyway):

up ip -6 addr add 2001:41e0:ff12::1/64 dev $IFACE
down ip -6 addr del 2001:41e0:ff12::1/64 dev $IFACE

Instead of bringing the interface down and up, just run ip -6 addr add 2001:41e0:ff12::1/64 dev eth0. Note the use of the /64 prefix instead of the /48 that got assigned, leaving only 20 pentillion addresses. Oh no! The reason for this is buried in the specs: basically, /48 is a site prefix, but individual networks should not be larger than /64, which is the prefix length of links in the IPv6 domain. Now is also a good time to enable IPv6 forwarding, e.g. like so:

# echo net.ipv6.conf.all.forwarding = 1 >> /etc/sysctl.conf
# sysctl -p /etc/sysctl.conf

Obviously, you will also need to change the policy on the ip6tables FORWARD chain. For now, let s just set it to accept. You should later create a proper ruleset, though!

# ip6tables -I FORWARD -j ACCEPT

Bringing IPv6 to your local network The final step is to spread the love to your local network. Refrain from selecting addresses from your subnet and assigning them to the local hosts, or wondering how to configure the DHCP server, because IPv6 does it differently: your gateway will advertise its routes (which includes a default route) to your network, and each host will pick an address based on its MAC address (unless it already has an EUI-64 address assigned. This all happens automagically, at least with current Debian and Windows machines. On the gateway, you need to install radvd and simply tell it which prefix to use on which interface. My /etc/radvd looks like this, and I won t explain it:

interface eth0
 
  AdvSendAdvert on;
  prefix 2001:41e0:ff12::/64
   
   ;
 ;

Note again how we advertise a /64 network rather than the /48 we own . You cannot advertise smaller networks if you want automatic configuration to work, and you should not use networks larger than /64 in any case. If 2^64 addresses are not enough for you, I trust you ll be able to figure out how to advertise another of your 65536 /64 prefixes in the /48 subnet to appropriate hosts. Restart radvd and run over to another host to witness how it automagically gets connected to the IPv6 network by scanning /var/log/kern.log and watching the output of ip -6 addr and ip -6 route. Try ping6ing from there! Find the dancing turtle! It should all work. If you don t like the automagic aspect of this, look into stateful configuration, using DHCPv6, as provided by dibbler-server and ?wide-dhcpv6-server.

Resolving names Take note of the IPv6 address of each host. There s a way to determine it given the host s MAC address, but this is easier (ipv6calc is also useful). You might want to let your local DNS server know by adding AAAA records in parallel to the existing A records, and possibly even adding PTR entries. If you re serious about IPv6, you can tell SixXS to delegate reverse lookups for the IPv6 addresses to your DNS servers, but you ought to refrain from polluting the DNS namespace. Note that bind9-host provides an improved host tool, which fetches all kinds of information about names, not just the one single information configured as default:

% host pulse.madduck.net
pulse.madduck.net has address 130.60.75.74
pulse.madduck.net has IPv6 address 2001:41e0:ff1a::1
pulse.madduck.net mail is handled by 99 b.mx.madduck.net.
pulse.madduck.net mail is handled by 10 a.mx.madduck.net.
% host 2001:41e0:ff1a::1
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.a.1.f.f.0.e.1.4.1.0.0.2.ip6.arpa
domain name pointer pulse.madduck.net.

Oh, and if you re really that curious about how IPv6 addresses are computed from MAC addresses, read RFC 2464. Basically, given a prefix 2001:41e0:ff1a:: and a MAC address aa:bb:cc:dd:ee:ff, the resulting IPv6 address is obtained by:

1. inserting ff:fe into the middle of the MAC address to yield aa:bb:cc:ff:fe:dd:ee:ff;
2. flipping the second lowest bit of the first octet to yield a8:bb:cc:ff:fe:dd:ee:ff;
3. removing the odd colons to yield a8bb:ccff:fedd:eeff, the EUI-64;
4. concatenating the prefix and this result to get 2001:41e0:ff1a::a8bb:ccff:fedd:eeff.

If you find your (Windows) IPv6 addresses changing all the time, you might be faced by privacy features .

Remaining issues Even though my IPv6 connectivity works, I have two remaining issues.

Sending larger amounts of data to the network I am experiencing a curious issue where outgoing ssh IPv6 connections time out and outgoing data transfers hiccup. I have yet to find out what s going on.

Mapping names to laptops Laptops generally have two interfaces, one with a cable, and the other wireless. Both of these interfaces will have separate MAC addresses, and by extension, the laptop will have different IPv6 addresses depending on how it is connected to the local network. I want to be able to connect to laptops without knowing the medium they use to connect to the network. Unfortunately, there seems to be no feasible way. The solutions I see are:

• override the MAC address of one interface with that of the other, which is going to cause bgi problems in the case when the laptop (accidentally) gets connected to the same network twice;
• add both IPv6 addresses as AAAA records to the laptop s DNS name, which will cause random delays when connecting as the resolver may return the currently inactive address first;
• set up mobile IPv6, e.g. by following this Mobile IPv6 how-to, which would allow accessing the laptop uniformly, no matter where in the world it is. Unfortunately, Debian s support for Mobile IPv6 is severly lacking at time of writing. Also, Yves-Alexis Perez notes that this how-to is horribly outdated and promised to tend to it Real Soon Now .

The second solution works for me for now, but I am interested in the third. In response to this document, Andreas Henriksson has suggested the replace the stateless configuration (radvd) with stateful configuration, using DHCPv6. I have yet to investigate this option. Jeroen Massar suggests to unite cable and wireless into a bridged interface, which seems like a very good idea.

Credits Thanks to Bernhard Schmidt, William Boughton, and Jeroen Massar, and everyone on #ipv6/irc.freenode.org for their help over the past few weeks, and all those who fed back comments in response to this document!

So I've been inexcusably tardy in announcing this, but Obey Arthur Liu is working on aptitude for Google's Summer of Code program. His goal is to write a third interface to the program, using GTK+ to produce the UI; as the abstract for his project states:

I will create a GTK+ GUI for Aptitude that will work alongside improved current ncurses and command-line interfaces. This will offer an alternative to Synaptic with an interface design geared toward usability and advanced functionality.

I'm looking forward to seeing what he comes up with! I think there are a lot of interesting possibilities in the design space of GUI package managers; hopefully he'll have time to explore at least a few of them. At the moment he's focusing on figuring out the care and feeding of the apt library backend and learning some of the quirks you have to deal with when writing a frontend against it. Arthur's blog can be found at http://www.milliways.fr and on Planet Debian. The Mercurial repository for his work can currently be found at http://dev.graffit.net/aptitude/hg/.

The Linux Plumbers Conference[1] now has a call for papers, with registration also being open. Why is this important? Because it's the first conference aimed at getting together developers from all walks of the Linux world and fixing the core infrastructural problems. Ever looked at a kernel interface and wondered how the christ any userspace programmer was supposed to use it? Ever looked at a desktop application and wondered what kind of $2 crack they were on when they wrote it? Here's your chance to find out or get it fixed.

I'm running the power management microconf. We're looking at discussing what sort of interfaces the kernel needs to export to userspace and how userspace needs to make use of them. Want to be part of engineering the first OS to have high quality power management all the way from embedded to servers? Be in Portland in September. Ideally at the conference rather than a bar, though there'll be plenty of time for that as well.

So, if you have opinions on power management (or any of the other topics), sign up now. Otherwise we'll ignore them and you'll be unhappy.

[1] Plumbers? Plumbers'? Plumbers's? Almost certainly not Plumber's.

Well I'm back and ready to do some fun work. In the meantime it seems that at least one of my crash-fixes, from the prior public bugfixing, has been uploaded:

• yelp: segfault while loading info documentation

I'm still a little bit frustrated that some of the other patches I made (to different packages) were ignored, but I guess I shouldn't be too worried. They'll get fixed sooner or later regardless of whether it was "my" fix. In other news I've been stalling a little on the Debian Administration website. There are a couple of reasonable articles in the submissions queue - but nothing really special. I can't help thinking that the next article being a nice round number of 600 deserves something good/special/unique? hard to quantify, but definitely something I'm thinking. I guess I leave it while the weekend and if nothing presents itself I'll just go dequeue the pending pieces. In other news I've managed to migrate the mail scanning service into a nicely split architecture - with minimal downtime. I'm pleased that:

• The architecture was changed massively from a single-machine orientated service to a trivially scalable one - and that this was essentially seamless.
• My test cases really worked.
• I've switched from being "toy" to being "small".
• I've even pulled in a couple of new users.

Probably more changes to come once I've had a rest (but I guess I write about that elsewhere; because otherwise people get bored!). The most obvious change to consider is to allow almost "instant-activation". I dislike having to manually approve and setup new domains, even if it does boil down to clicking a button on a webpage - so I'm thinking I should have a system in place such that you can sign up, add your domain, and be good to go without manual involvement. (Once DNS has propogated, obviously!) Anyway enough writing. Ice-cream calls, and then I must see if more bugs have been reported against my packages... ObQuote: Run Lola Run.

The newspapers brought good news bits in the last two days. P blico reports on Paco Rivi re's ongoing quest to get a refund for the extra money he had to pay for a Windows licence when he bought a laptop. Paco is a well known member of the Ubuntu Catalan community and has been battling for this common-sense right for 3 years. The trial took place last Monday, and hopefully he'll be able to report some good news soon. In totally unrelated news, the Valencian caveman Juan Garc a Sentandreu, leader of the right-wing Coalici n Valenciana party, was arrested yesterday, for still not too clear reasons. Being one of the biggest enemies of my language, and having a long record of violent attacks to cultural entities and other political parties in Val ncia, I can't say I pitty him at all. I hope he had fun sleeping with the yonkis in the central police station last night. :)

I maintain a bunch of SSL certificates, mostly signed by my own site authority. Too many not to automate, but not enough to warrant heavy machinery. Here's how I do it. The configuration files Each certificate needs a config to describe what's in it. I create each of these and name it with a .cnf suffix. Here's an example:

[ req ]
prompt                  = no
distinguished_name      = server_distinguished_name
[ server_distinguished_name ]
commonName              = server.usefulinc.com
stateOrProvinceName     = England
countryName             = GB
emailAddress            = edd@usefulinc.com
organizationName        = Useful Information Company
organizationalUnitName  = Hosting
[ req_extensions ]
subjectAltName=edd@usefulinc.com
issuerAltName=issuer:copy
nsCertType            = server
[ x509_extensions ]
subjectAltName=edd@usefulinc.com
issuerAltName=issuer:copy
nsCertType            = server

Let's say this config is server.cnf. I then just type make server.pem to generate the corresponding certificate and key, signed by my local certificate authority. As I don't want to attend the startup of every service, I ensure the key is password-less. The Makefile rules Here are the makefile steps I use to generate and sign keys.

.SUFFIXES: .pem .cnf
.cnf.pem:
        OPENSSL_CONF=$< openssl req -newkey rsa:1024 -keyout tempkey.pem -keyform PEM -out tempreq.pem -outform PEM
        openssl rsa <tempkey.pem >  basename $< .cnf _key.pem
        chmod 400  basename $< .cnf _key.pem
        OPENSSL_CONF=./usefulCA/openssl.cnf openssl ca -in tempreq.pem -out  basename $< .cnf _crt.pem
        rm -f tempkey.pem tempreq.pem
        cat  basename $< .cnf _key.pem  basename $< .cnf _crt.pem > $@
        chmod 400 $@
        ln -sf $@  openssl x509 -noout -hash < $@ .0

The resultant files are:

• server.pem — contains both certificate and key in one file
  
• server_crt.pem — certificate file
  
• server_key.pem — key file
  

Some notes on these steps: my site-local certificate authority is in the directory usefulCA, along with an OpenSSL config which describes my preferences. This config was created by copying and making appropriate adjustments to the default /etc/ssl/openssl.cnf which ships with Debian. For generating certificate signing requests to ship to a commercial certificate authority, it's a bit simpler. I save the config files with a .reqcnf suffix instead, and use this rule:

.SUFFIXES: .pem .cnf .reqcnf .csr
.reqcnf.csr:
        OPENSSL_CONF=$< openssl req -newkey rsa:1024 -keyout  basename $< .reqcnf .key -keyform PEM -out  basename $< .reqcnf .csr -outform PEM

And finally, a rule I use to sign incoming certificate requests from other systems:

.csr.pem:
        OPENSSL_CONF=./usefulCA/openssl.cnf openssl ca -in $< -out  basename $< .csr _crt.pem

I offer these without warranty in the hope they might be useful to somebody. They're not much more than a transcription of a how-to into a makefile, but it's just enough technology to ensure creating certificates isn't a big nuisance. Further reading

• A more complete guide to setting up a site-local certificate authority
  

Notes Why do I bother with a site-local CA, rather than just self-sign? It lets me bypass the annoyance of SSL warnings on clients once I've installed my own CA certificate, and gives me a coarse grained level of access control: for instance, only clients with certificates signed by my CA are allowed to access the site's LDAP server. My personal next step with this is to integrate the certificate production process with my emerging Puppet recipes for managing local infrastructure.Join the conversation about this post

I've posted yesterday an overview of awesome 3 development state on the awesome mailing list. To sum up, everything's fine. I hope this summer will see a good hacker-friendly window manager coming out.

(hint: D-I=Debian Installer, the installation system for your favourite distro) Commenting on the blog entry I posted yesterday about D-I releases... I want to make it clear, in case it could be misunderstood, that writing that the next D-I release will certainly be called a Release Candidate and be the final release, is purely personal assumptions. I am not the D-I release manager. Otavio Salvador is. So, my assumption is only based on the small experience I got from a little bit less than 5 years in the D-I team and nearly 3 entire release cycles for Debian and D-I. Pure speculation, then. I could even say "science-fiction". I still believe that there will (should) be only one D-I release until the release of Lenny, but that decision is up to the D-I release manager(s?) and you're free to not believe me and just ask on #debian-boot or the debian-boot mailing list. Still, when it comes at i18n, I still believe that translators should move their asses if they want to make it for the release. And that involves /me saying them that the release is bloody near no matter if it's called a Release Candidate or Yet Another Beta.

Okay, I will again miss DebConf. This time it is not because of time, but because of money. The flight to Buenos Aires is far too expensive to go there just for DebConf and I didn't manage to plan some more travelling around South America... Hopefully it will get better next year, at least DebConf will be much closer :-). PS: Just after deadline for confirmation and my decision for not going there, I got an email that my sponsorship for travel was approved. Unfortunately too late to change my decision.

Ten years ago, most of us wouldn't have dreamt we'd be managing terabits of storage, tens of megabits of bandwidth, arrays of network-distributed services. The height of a programmer's worry would likely be choice of UI toolkit or finding the right way to indent code, and the height of consumer concern deciding which room to put the new computer in. Now the problems associated with managing large networks are becoming real for everyone, right down to the consumer level. Stupendously large amounts of computing resource are available at an instant. Your household probably has more than a terabyte of storage already. Issues such as single sign-on are going to hit home over the next year, as networked computing and entertainment devices profilerate. Features such as Apple's Time Machine will be increasingly vital — software that makes traditionally gnarly sysadmin tasks consumer-friendly. The rebranding of .Mac into "Mobile Me" is also a step in this direction. The impact on developers As software developers, we also have to cope with the effects of this resource-richness. For small sums of money we can get access to large computing clusters, geographically redundant hosting services. Our programs have left the desktop and found their new home on the web. System administration issues loom large upon us, security concerns lurk auspiciously in the corners of our minds. Although the cost of infrastructure has dropped radically, other costs remain high and are going to stay that way. System administrators are not only grumpy, they demand high wages. Commercial software license fees spiral out of control: traditional per-CPU licensing models make little sense when you can quickly bring up tens of machines. The cost in power is already troubling large companies, and there's no reason to suspect the problems won't ripple down. Help is at hand from a variety of technologies. If they don't yet make massive resource management trivial, they at least make it possible. Some of these also inhabit the weird territory of being both the source of a problem and a solution at the same time: virtualization, for example. Distributed revision control systems Distributed revision control is a technology whose time has finally come in popular circles, thanks in part to Linus Torvald's Git system. DRCS has several important impacts on today's developer:

• Branching and, importantly, merging become much cheaper, allowing agile and flexible iterations of development.
• Loosely connected and geographically diverse development becomes much easier. Even within a single organization it is not uncommon to find teams spanning countries, time zones. Complex multi-site VPN setups aren't necessary when a few SSH keys can do the job.
• Revision control becomes packaging on the cheap. Like it or not, the mere tagging of a source tree has now become a valid option for releasing software.

All these trends lower the barrier to entry, increase collaboration and agility of development. You can the value of this as more software tools become free. Selling such tools is rapidly becoming a thing of the past, the advantages of sharing enable the developers at the sharp end to get their jobs done quicker. However, such increased agility and, well, messiness leave other problems to solve, which the next two technologies address. Virtualization Hardware-as-a-service, infrastructure-as-a-service, call it what you will. The ability to create what we used to call entire machines, pick them up and move them around the network is revolutionary, and it's something that will have a real impact on regular developers. The benefits are at several levels.

• Agile infrastructure — a ready supply of new machines makes it a lot easier and cheaper to test different scenarios, architectures, and to separate concerns. If things go wrong, throw away the image and start over. It's all about cutting the administration load.
• A packaging solution — the new macroeconomics of software distribution mean that distributing entire machine images which communicate exclusively via the network is now a feasible way to distribute your software. We must adjust to the notion of distributing appliances, not code. We may mourn the lost crafts of creating RPMs or installers, but let's face it, it's now a waste of time.
• New business models — your application can now be delivered as a black-box appliance, circumventing compatibility issues, or as a service, with virtualization part of the solution to scaling.

Configuration management Computing is a zero-sum game, and despite our increased ability to create and distribute software, problems still exist. We just pushed them to the next level. In good part, this next level is the problem of configuration management. We now have networks and clusters of (virtual) machines, software so agile we need six decimal places to describe its revision levels, and network and authentication paths that are starting to tangle. How do we manage that? One thing developers crave is repeatability. That's why we love our makefiles, autoconf, Ant, rake and so on. It's the one time even the most imperative-minded programmer writes declarative code. We like to say "let the world be like this." Our new sprawling world lacks this feature, and the best of our old toolkits — .debs, RPMs — address things only at the level of packages in a single environment. So developers must look to the world of operations, a territory we probably thought we needn't enter. In this world the new "make" is called Puppet. You write recipes to describe how things ought to be, and Puppet will make it so. I've been spending some time digging into Puppet, and feel excited by the confidence it's giving me. Now my applications exceed single source trees, and single machines, it gives me the means to tie the whole together. This article was going to be solely about Puppet, but that will have to wait now for another time. It's likely you'll have played with virtual machines and distributed revision control, but have you tried Puppet yet? Give it a spin, and let your mind wander over the benefits for your organization and development approaches. Conclusions
For developers and users alike, our world is changing. Hardware, connectivity and increasingly software is becoming cheap or free. The solidity of the old things we put value on — real things you can touch like disks — is eroding. What really matters is our data, our creations, and their communication. If they don't quite yet exist in a universal "cloud" yet, they're certainly getting frisky. As vendors provide solutions for consumers to manage their new domestic infrastructure, developers must look to network-aware toolkits and operations techniques to manage and get the best from their emergent infrastructures. Also on this topic:

• In search of agile infrastructure for web applications (June 2006)
  

Join the conversation about this post

With firefox3 entering unstable, it's now possible to upload firefox3 extensions. Vimperator just entered unstable tonight, just enjoy it. For those not knowing about it, it's an excellent extension that gives to your firefox the vim look and feel. People that are already hooked to git will love it. Tiled window manager users usually love it. The best feature it has is the so called quick hints feature. Type f in command mode, to have nice labels next to every link in the page. You can type some letter, and only links that contains those letters will have the hints, and you'll see something like this (I added 'mo' after having asked for quick hints like you can see in the ruler):