So I attended the conference last week in Chicago, IL. I was just been procrastinating to write about this.

As a (almost) complete outsider of the Perl community, I have countered thoughts about the whole event:

• I was actually expecting a deeper technical content on some of the talks.
• Some speakers were just very bad at presenting, they were either boring with a very exciting topic or lacked the spark to get the audience attentive.
• Some people on the IRC channel kept criticizing what some (not very known in the Perl circle) speakers were talking about. For instance, the usual stupid argument of "why to do this, if we can already make it with...", or "yeah, but we can do that with Foo::Bar and ZMOFGML". After all, just as Andy Lester once wrote, let's see people's work, let's see their new ideas or new ways to implement things. Heck, TIMTOFWTDI.
• Wireless connectivity was very poor, apparently due to ad-hoc nodes and malware on the cloud.
• Since I'm used to DebConfs, it was nice to see, though, a job fair and some circles discussing new ways to do business with Perl development.
• Around 40% of the attendees were attending their actual first YAPC. Apparently, there are a lot of newcomers to Perl conferences, but they usually don't come back that often or get a bit scared.
• Being a community conference, makes it, after all, a bit special. YAPC was mostly, as Jos Castro stated, just about people and beer.
• Organizers gave their best to make this happen successfully, you could see, feel that.
• I met some guys that after introducing us, I was confused were they were from, which was mainly because of my huge ignorance of US geography. For instance, people would tell they are from South Benton, from Omaha or from Columbus, between others. At the moment I felt it'd be tacky to ask on what states those cities are on, but now I know ;) Indiana, Nebraska and Ohio. Heh.
  
• Chicago is a beautiful city and stuffed pizza kicks ass.
• What I liked from the Perl community: They usually get things done and don't fool around too much with senseless shit.
  
• What I didn't like: If you are not a regular into the circle, it's quite difficult to feel involved or part of it. To this, add being from another country :P
• What I liked the most on talks: Ricardo SIGNES's talk about email which was awesome and the lightning talks (specially the improvised ones).
  

After all, it was quite a nice and new experience. I'm not totally sure I'll attend YAPC::NA 2009 in Pittsburgh, PA, we'll see.

Paul Graham has recently published an essay titled How To Disagree [1]. One form that he didn’t mention is to claim that a disagreement is a matter of opinion. Describing a disagreement about an issue which can be proved as a matter of opinion is a commonly used method of avoiding the need to offer any facts or analysis. Sam Varghese published an article about the Debian OpenSSL issue and quoted me [2]. The Basic AI Drives [3] is an interesting papar about what might motivate an AI and how AIs might modify themselves to better achieve their goals. It also has some insights into addiction and other vulnerabilities in human motivation. It seems that BeOS [4] is not entirely dead. The Haiku OS project aims to develop an open source OS for desktop computing based on BeOS [5]. It’s not nearly usable for end-users yet, but they have vmware snapshots that can be used for development. On my Document Blog I have described how to debug POP problems with the telnet command [6]. Some users might read this and help me fix their email problems faster. I know that most users won’t be able to read this, but the number of people who can use it will surely be a lot greater than the number of people who can read the RFCs… Singularity tales is an amusing collection of short stories [7] about the Technological Singularity [8]. A summary of the banana situation [9]. Briefly describes how “banana republics” work and the fact that a new variety of the Panama disease is spreading through banana producing countries. Given the links between despotic regimes and banana production it’s surprising that no-one is trying to spread the disease faster. Maybe Panama disease could do for South America what the Boll weevil did for the south of the US [10]. Jeff Dean gives an interesting talk about the Google server architecture [11]. One thing I wonder about is whether they have experimented with increasing the chunk size over the years. It seems that the contiguous IO performance of disks has been steadily increasing while the seek performance has stayed much the same, and the dramatic increases in the amount of RAM you can get for any given amount of money over the last few years have been amazing. So it seems that now it’s possible to read larger chunks of data in the same amount of time and more easily store such large chunks in memory.

• [1] http://www.paulgraham.com/disagree.html


• [2] http://www.itwire.com/content/view/18376/1090/1/0/


• [3] http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf


• [4] http://en.wikipedia.org/wiki/BeOS


• [5] http://www.haiku-os.org/


• [6] http://doc.coker.com.au/internet/how-to-debug-pop/


• [7] http://www.vivtek.com/fiction/singularity_tales/


• [8] http://en.wikipedia.org/wiki/Technological_singularity


• [9] http://www.nytimes.com/2008/06/18/opinion/18koeppel.html?ei=5087&em=&en=acf4d20d4c12e559&ex=1214020800&pagewanted=print


• [10] http://en.wikipedia.org/wiki/Boll_Weevil_Monument


• [11] http://www.youtube.com/v/qsan-GQaeyk

Share This

I've spent a few hours recently looking at building RPM packages of GNU/Linux kernels, which has been a frustrating process. There are many many online guides which give the impression that this is actually a pretty complex process. For example How To Compile A Kernel - The CentOS Way guide. (Did I mention how bad most of the howtoforge guides are recently?) So, after fiddling around for an afternoon and getting lost I decided to abandon the process. Here is a tested process for building a binary RPM kernel package:

cd linux-2.6.24.7/
make rpm

Yes this works just fine upon a Centos 5.x machine - I'm used to using make-kpkg to make a Debian kernel package, but it seems that if you just visit kernel.org and download the latest version you can build a RPM without any extra effort thanks to native support. Cool. Now I need to work out how to create, host, and update a YUM repository. That looks fiddly and annoying too. XML. Eww. Any guides are most welcome - ultimately I need to package and host a "recent" kernel for Centos 4.x, Centos 5.x and Fedora Core 6-9 - each for i386 + amd64. ObQuote: Spiderman

One of the traditional problems with power management has been balancing power savings against reduced functionality. The two traditional approaches to this have been to have sane defaults or to let the user tweak them, with a hybrid approach being to have sane defaults and an "advanced" box that lets users set their own parameters.

Both of these options suck.

The problem with sane defaults is that power management is an area where the "sane default" genuinely does differ between users. Usage patterns will determine the appropriate spindown period for a hard drive. AHCI's ALPM will disable hotswap detection, which is fine for home users but a pain if people want to rip out a failed RAID element. ALPM also introduces a small amount of latency (on the order of a seek in the most aggressive mode), which is entirely irrelevant for almost everyone but is a pain if you're trying to offer i/o latency guarantees. PCI-E link power management adds small quantities of latency. Letting the CPU run to full speed whenever it wants to is good for power management but potentially bad for thermal management. Screensaver timeouts vary depending on whether you're watching a movie or not.

The problem with leaving everything tweakable is that you're asking users to make choices about things but not giving them the information they need to make those choices. Whether you get a power saving from hard drive spindown depends on whether the drive is idle for long enough to save the power you'll spend spinning it back up. Get it wrong and you'll be putting your drive under extra load, reducing performance and consuming more power than you were to begin with. If you limit the maximum speed of your CPU to keep it cool, you'll be wasting power and performance even when it's well below the thermal limits. Disabling ALPM because you might want to change a RAID device at some stage means you're consuming power despite generally not needing to use the additional functionality.

But while these options both suck in different ways, they share one fundamental failure. They both imply static configuration. The correct power management policy depends on the way that a machine is being used at a specific instant in time, and no matter how many power management profiles you add there's no way for a user to switch between them fast enough to obtain a decent power/performance ratio.

Adaptive power management adopts a different strategy. Rather than define a static policy (or several static policies), the hardware is monitored to determine the best power management settings. An example of this is the ondemand CPUfreq governor. Rather than attempt to provide a static configuration, the processor frequency is automatically changed to meet demand while also keeping the overall power consumption as low as possible. This patch adds a thermal component to its policy, preventing the machine from overheating. Result? No requirement to provide tunables to userspace.

Something similar is achievable with hard-drive spindown. By simply monitoring how often reads or writes actually hit disk, it's possible to automatically adjust the spindown period to obtain decent power savings while reducing the number of spinups. This approach is imperfect - in an ideal world we'd know to spin the drive up before it's needed, but that simply isn't possible. However, the user is almost certainly unable to make a better guess. It's not realistic for a user to be able to predict whether or not a given page is still in cache or has been evicted. Nothing is lost by providing a dynamic policy, and another unnecessary tunable has vanished.

The difficult part of an adaptive policy comes when roles are switched in ways that don't correspond to changse in hardware status. One of the most common cases is that of screensaver settings when the user is watching a video. It's not practical for the kernel or a system daemon to predict this case, so we have two choices:

• Get the user to switch to a "media" power policy and then remember to switch back afterwards
• Have the media player deactivate the screensaver at startup and reactivate it when the video isn't playing

.One of these choices involves the user having to explicitly state something that's obvious - playing a video automatically implies the desire to adopt a different power management policy, and so asking the user to manually adjust that is ridiculous.

The loss of hotswap notifications when ALPM is enabled? Simply reenable it if SMART starts complaining or the RAID set becomes degraded or a filesystem is unmounted or any of the other things that will have to be done before a drive can be safely removed. Latency an issue? Let userspace tell you what its latency requirements are. When we get down to it, the only settings that we can't infer are things like idle-time-to-suspend, screensaver settings and default brightnesses. Which, curiously, are about the only things that Mac OS exposes[1].

The problem we face is that we don't currently have interfaces to let applications tell us what their requirements are. This is theoretically straightforward, but getting it right is important. Beyond latency and screensaver issues, what constraints do applications want to impose? How can we expose an interface to those constraints? This is the sort of thing we'll be looking at at the Linux Plumbers Conference, so if you have ideas about what they'd need to look like then sign up. Or, even better, submit a paper and be part of the process that gets Linux's power consumption as close to the theoretical minimum as possible.

[1] My Vista system gives me 25 options which can be set for each of AC and battery.

As a followup to yesterday's post I profiled the speed of various ways of executing shell scripts on both my laptop using Bash and Dash, and on the Zoom OMAP3430 development board. The results are fairly predictable although the relatively small difference between . foo and ( . foo ) is useful to know. We all knew that Dash was faster than Bash, but this microbenchmark demonstrates it very clearly. It also demonstrates how much faster my laptop is than the (relatively powerful) Zoom development board. Overall conclusion: don't use Bash unless you needs the extra power, and source when possible. NP: Aerial, 2562

So I've had a hectic few days, and I'm getting close to having caught up with the things that I've been sitting on whilst I've been away.

ObRandom: Several people, independantly, have told me within the past few days that "whilst" is not a real word. it is. End of ..

Some interesting things I've been working upon recently include a fun little firewall tool. Once upon a time I wrote a firewall script which worked like this:

firewall/
 -- incoming.d
     -- smtp
     -- ssh
     -- www
 -- outgoing.d
     -- ssh
     -- smtp
     -- dns
     -- icmp

When you executed the magic firewall script it would scan the incoming.d directory, and for each file it found lookup the relevant port in /etc/services. These port numbers would then be opened. And at the end you'd just have a "-j DROP". After a long phone conversation to a colleague on Thursday/Friday of last week I've now reworked this idea anew. There is still the notion of filenames referring to what is allowed for a pair of directories (incoming.d/ + outgoing.d/) but even more flexability and no hardwired use of /etc/servvices. I guess some ideas are just too simple to give up ..? Anyway there are a plethora of different firewall applications of varying sophistication and complexity in the world. I don't really want to go out of my way to promote this one - but at the same time it might be a useful idea for somebody? The next (work) job I have is determining how to make a "kernel" + "kernel-dev" RPM package based on Debian sources. Joy. Actually the more I look around the more fiddly, annoying, and troublesome I suspect this is going to be. Sigh. ObQuote: The Grudgy

I spent Saturday at BarCamp Belfast. There were more people than I expected; I'm no good at judging exact numbers but the wiki lists 62. As it turned out not everyone ended up talking; this worked out well for me because I don't think I'm Web2.0 enough (is that a bit ironic given I'm writing this on a blog?). I don't really do user interfaces or website stuff and that seemed to be the primary focus of a lot of the attendees. Nice to know there's a fairly active crowd in
Belfast though; I discovered the existence of Belfast Open Coffee which I'll have to try and make at some point.

I went to at least the following talks::

Gerard McBreen on Digital Circle & Adobe Flex: Quite a cute demonstration of the Flex framework for Flash. I'd have been more convinced if Flash could be considered truly cross platform; Adobe can't even manage a 64 bit client yet and Gnash is far from ready.

Daniel McLaughlin on Nagios: Not really anything I didn't already know, but always interesting to see how other people are using these things to see if there's any nice tricks you've missed.

Andrew Gribbon from Exile Coffee on speciality coffee: This was really interesting. A complete tangent from IT but enjoyable and I must buy some roasted beans from him (I've been meaning to place a Roast and Post order for ages anyway).

Brian O'Neill on freelancing: Tailored towards the web market in my opinion, which is fair enough but things like aiming to never employ anyone and stay a one man operation doesn't really work if you're doing something that a customer wants supported around the clock. Or even just during the working day if you're crossing time zones. Interesting to hear that Brian had run an agency in the past but preferred working on his own these days. I kinda feel I've gone a bit the other way. :)

Linux Hardware workshop: Not really a lot of interest here; a clutch of EEEs, an N800 and a Debian box or two. However I had a chat with David Newman about his MythTV setup which convinced me I really should give it another try. After I get someone to install a bigger aerial in the hope of stabilising our Freeview reception.

Brian Cleland on Linux Trends: Talking about lots of Linux takeup by big names and general market adoption eg the recent selection of cheap subnotebooks. Main thing of interest was OpenPandora which I hadn't heard of before; OMAP3 based handheld device running Linux with a 800x480 4.3" screen. Aimed as a gaming device, but has a keyboard and bluetooth/wifi so has potential as a PDA as well. Hopefully the build quality will be better than the GP2X I saw.

Andy McMillan on coffee shop working: I've thought about this in the past myself; I worked from Cafe Java in Norwich a few times while doing Black Cat, but in general didn't find it as conducive to work as I'd hoped. He mentioned Coworking Belfast which is an idea Tristam has talked about in the past, in terms of people who work from home getting to have some form of social contact with like minded individuals, without having to go to the expense of a whole office. It'll be interesting to see if it takes off.

Darragh Doyle and Grannymar on blogging: I think everyone in the audience already had a blog of some description. I don't really consider my blog a social forum; it's not something I expect to have a long conversation on, just ask a quick question or discuss something
I've done that I think might interest people or give a general "this is what's happening at present" post. Blog commenting systems are all a bit crap to keep up anything sustained. Someone needs to write a blog<>nntp gateway that correctly cross posts comments as well. I suppose the issue with that is then everyone needs to run a news server, or all agree on
using the same one. Fine for groups of people who know each other but not if you want to read a dozen blogs from different areas in a single client that'll let you properly follow conversations. Mind you, people seem to cope with forums and I don't really understand how. Supporters of news and mailing lists seem to be dwindling. :(

Anyway, that was my first BarCamp. I wasn't quite sure what to expect, it was a bit more Web2.0 than I'd have liked (which is fair enough, but I'd have liked to have seen some talks about low level bits really) but I'd go again and will keep an eye out for the next one. And hopefully I'll see some of the people I met there at the BLUG meeting this Wednesday (Tap House in Lower Crescent from 6pm for anyone who's not on the mailing list).

If you want a reliable network then you need to determine an appropriate level of redundancy. When servers were small and there was no well accepted virtual machine technology there were always many points at which redundancy could be employed. A common example is a large mail server. You might have MX servers to receive mail from the Internet, front-end servers to send mail to the Internet, database or LDAP servers (of which there is one server for accepting writes and redundant slave servers for allowing clients to read data), and some back-end storage. The back-end storage is generally going to lack redundancy to some degree (all the common options involve mail being stored in one location). So the redundancy would start with the routers which direct traffic to redundant servers (typically a pair of routers in a failover configuration - I would use OpenBSD boxes running CARP if I was given a choice in how to implement this [1], in the past I’ve used Cisco devices). The next obvious place for redundancy is for the MX servers (it seems that most ISPs have machines with names such as mx01.example.net to receive mail from the Internet). The way that MX records are used in the DNS means that there is no need for a router to direct traffic to a pair of servers, and even a pair of redundant routers is another point of failure so it’s best to avoid them where possible. A smaller ISP might have two MX machines that are used for both sending outbound mail from their users (which needs to go through a load-balancing router) as well as inbound mail. A larger ISP will have two or more machines dedicated to receiving mail and two or more machines dedicated to sending mail (when you scan for viruses on both sent and received mail it can take a lot of compute power). Now the database or LDAP servers used for storing user account data is another possible place for redundancy. While some database and LDAP servers support multi-master operation a more common configuration is to have a single master and multiple slaves which are read-only. This means that you want to have more slaves than are really required so that you can lose one without impacting the service. There are several ways of losing a server. The most obvious is a hardware failure. While server class machines will have redundant PSUs, RAID, ECC RAM, and a general high quality of hardware design and manufacture, they still have hardware problems from time to time. Then there are a variety of software related ways of losing a server, most of which stem from operator error and bugs in software. Of course the problem with the operator errors and software bugs is that they can easily take out all redundant machines. If an operator mistakenly decides that a certain command needs to be run on all machines they will often run it on all machines before realising that it causes things to go horribly wrong. A software bug will usually be triggered by the same thing on all machines (EG I’ve had bad data written to a master LDAP server cause all slaves to crash and had a mail loop between two big ISPs take out all front-end mail servers). Now if you have a mail server running on a virtual platform such that the MX servers, the mail store, and the database servers all run on the same hardware then redundancy is very unlikely to alleviate hardware problems. It’s difficult to imagine a situation where a hardware failure takes out one DomU while leaving others running. It seems to me that if you are running on a single virtual server there is no benefit in having redundancy. However there is benefit in having an infrastructure which supports redundancy. For example if you are going to install new software on one of the servers there is a possibility that the software will fail. Doing upgrades and then having to roll them back is one of the least pleasant parts of sys-admin work, not only is it difficult but it’s also unreliable (new software writes different data to shared files and you have to hope that the old version can cope with them). To implement this you need to have a Dom0 that can direct traffic to multiple redundant servers for services which only have a single server. Then when you need to upgrade (be it the application or the OS) you can configure a server on the designated secondary address, get it running, and then disable traffic to the primary server. If there are any problems you can direct traffic back to the primary server (which can be done much more quickly than downgrading software). Also if configured correctly you could have the secondary server be accessible from certain IP addresses only. So you could test the new version of the software using employees as test users while customers use the old version. One advantage a virtual machine environment for load balancing is that you can have as many virtual Ethernet devices as you desire and you can configure them using software (without changing cables in the server room). A limitation on the use of load-balancing routers is that traffic needs to go through the router in both directions. This is easy for the path from the Internet to the server room and the path from the server room to the customer network. But when going between servers in the server room it’s a problem (which is not insurmountable, merely painful and expensive). Of course there will be a cost in CPU time for all the extra routing. If instead of having a single virtual ethernet device for all redundant nodes you have a virtual ethernet device for every type of server and use the Dom0 as a router you will end up doubling the CPU requirements for networking without even considering the potential overhead of the load balancing router functionality. Finally there is a significant benefit in virtual machines for reliability of services. That is the ability to perform snapshot backups. If you have sufficient disk space and IO capacity you could have a snapshot of your server taken every day and store several old snapshots. Of course doing this effectively would require some minor changes to the configuration of machines to avoid unnecessary writes, this would include not compressing old log files and using a ram disk for /tmp and any other filesystem with transient data. When you have snapshots you can then run filesystem analysis tools on the snapshots to detect any silent corruption that may be occurring and give the potential benefit of discovering corruption before it gets severe (but I have yet to see a confirmed report of this saving anyone). Of course similar snapshot facilities are available on almost every SAN and on many NAS devices, but there are many sites that don’t have the budget to use such equipment.

• [1] http://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol

Share This

And the packages that drove me nuts today are....root-system and mxallowd. Congratulations, people, you succeeded in turning me mad the same day by introducing debconf templates without any prior call for translations. So, another move back for translation teams. As a reward for that, please expect an *immediate* l10n NMU proposal by /me. Maybe, some day, people will *learn*. I can dream... Besides that, these packages are packages indeed coming out of NEW. I should maybe talk with the ftpmasters and see if we can enforce maintainers who introduce new packages with debconf stuff to actually:

• ask for a templates review on d-l-english
• send a call for translations

Or theming PS1 I am not fanatical about my command prompt. No sirree, not me. It is just that I spend half my life either staring at an Emacs window, or at an xterm command prompt; so even a marginal boost in productivity goes a long way. And I am often logged in over ssh to machines half a continent away, and am still comfortable enough on the Linux VT to spend time there, and I often do not have the GUI gew-gaws feeding me data. Hmm. Data. inpuuut. No. Must focus. With all these open xterms and Emacs terminal mode frames floating around, it is easy to lose track of where I am on each terminal, and what the working directory is. So I want my command prompt to help me keep track of where I am. If the terminal is an xterm, the title can be setup like

 "user@host:../shotened/path/to/current/working/dir"

I want to know what machine, id (am I root?), and directory I am in. If I am deep down in the labyrinths of some work related directory tree, I want the path to be pruned, from the left, one component at a time. However, this does not help me on the console; so I also want the path to be in the command prompt; but it should not take up too much of the command line; and ideally, should just go away as I type a longish command. Gawd, I love zsh. I once did a analysis of the command history. The most often used command sequences were

 cd some-place, ls

So. pwd and ls. I really really want to see the directory listing when I change directory to a new one. As one grows old, memorizing the directory contents for dozens of machines all the time taxes the gray cells a bit. So, I figure, why not let the command prompt handle all that? Having the current working dir always visible cuts down a heck of a lot on the pwd commands, and so all that s left is to insist that the command prompt thingy always run ls after a change directory. Simple enough. Saves on typing. And time. And this is not just some crazy talk. I want help with noticing whether the previous command exited with an error status (useful for commands that normally do not create an output). If I am logged in to a machine on battery power, I want to know that. I also like visual cues to the amount of power remaining (good for my laptop on the long flights). I don t want to have to know f the machine uses APM or ACPI, I just want my prompt color to change as the power fades. I want my command prompt to let me know if I am in a directory which is under version control, and if so, what branch I am on. (I occasionally have to come in contact with arch, bzr, git, subversion, svk, and mercurial). If I am in a version controlled project, where I am relative to the root of the checked out tree is often more important than the absolute path, so I want to see relative paths, not absolute paths. I want to know if there are uncommitted files in the working directory. Visually. I want to be reminded if I am in the middle of an ongoing

 rebase -i .

This is not asking for too much, is it? So, here is a screen-shot of this in action: I start from my home directory, go to a directory not under version control, go to a project under git, then a different project with uncommitted files, and then finally to a subversion checkout. All with an angry fruit salad of colors warrantied to make Martin Krafft want to claw his eyes out . If you use zsh, then just grab hold of this, and then do:

 autoload -U promptinit
 promptinit
 prompt manoj

This might be a bit of a hit on slow machines, but even my laptop is a core 2 duo, so I do not find it noticeable. Oh, and if you liked this article, you might also like Theming Emacs, and Theming XTerms.

Ever the sucker for punishment, I decided to pick three difficult things and stick them all together: LDAP, SSL and replication. Here's how to make it go on Debian and Ubuntu. The problem You want LDAP replication to happen over the internet, and you want it to happen securely. The caveat I'm not going to tell you how to set up your LDAP from scratch here: I'm assuming you've reached a solution you're happy with and want to replicate it. The solution We're going to set up a replicating slave LDAP server, which communicates with the master over the internet via an SSL-protected connection. Enabling replication First up, the master LDAP server needs to be configured to permit replication.  The key lines to add to your slapd.conf include:

moduleload syncprov
index entryCSN,entryUUID eq
overlay syncprov
syncprov-checkpoint 100 10
syncprov-sessionlog 200

These load up the synchronization module, add indices which make sync go faster, and enable sync. For more detail see the OpenLDAP site. Next you need to add a replicator user to your LDAP database, give your replicator user access to passwords as well as general read access. To create the replicator user, I made this simple LDIF file and fed it to ldapadd.

dn: cn=replicator,dc=mydomain,dc=com
objectClass: simpleSecurityObject
objectClass: organizationalRole
cn: replicator
description: LDAP replicator
userPassword: TOPSEKRIT

Once this user is in your LDAP database, you should give it read access to passwords (I assume you've already given read access to authenticated users.) I have this in my slapd.conf:

access to attrs=userPassword,sambaNTPassword,sambaLMPassword
...
   by dn="cn=replicator,dc=mydomain,dc=com" read

To check that this works, try using ldapsearch to check that the passwords are returned:

ldapsearch -x -D cn=replicator,dc=mydomain,dc=com \
  -W   grep -i password

Enter the replicator password when prompted, and you should see the encrypted passwords from your LDAP database. Securing access Now you've got replication enabled on the master, you will want to ensure it is available on the internet only via TLS or SSL. Here's what I added to slapd.conf to enable this:

TLSCertificateFile      /etc/ssl/certs/ldapserver_crt.pem
TLSCertificateKeyFile   /etc/ssl/private/ldapserver_key.pem
TLSCACertificateFile    /etc/ssl/certs/myCA.pem
TLSVerifyClient         demand

As you will guess from the configuration, the first two lines set the SSL key and certificate the master uses (see "A little twist" below for an important note on key permissions.) The third line tells slapd where to find my site-local certificate authority (CA), and the fourth line says slapd must require any connecting client to have a valid SSL certificate signed by the site-local CA. This is important, as it provides a second layer of access control: a replicating client must connect using a certificate you signed, plus the replicator password. Before this enables TLS access, we must tell slapd which network interfaces to listen on. To do this, edit the SLAPD_SERVICES variable in /etc/default/slapd. Here's my configuration:

SLAPD_SERVICES="ldap://127.0.0.1/ ldap://192.168.0.1/ ldaps:///"

This enables regular LDAP on the loopback and intranet network interfaces, and LDAP/SSL on all interfaces, including the public internet. So, with slapd restarted we are at this situation: connections are now possible from the internet, as long as they are made over SSL with a certificate signed by our site-local CA. (In fact, you can make much finer-grained access restrictions in your configuration than I have done. Using LDAPS rather than TLS over regular LDAP is a rather broad precaution. As explained on the OpenLDAP site, the ssf= parameter can be used to require a certain level of secure connectivity on a per-user or client basis.) Setting up the replicating server
Your slave server should have the same configuration as the master, except you can leave out the bits enabling replication. Firstly, you'll need add to slapd.conf the replication configuration:

syncrepl rid=123
        provider=ldaps://ldapmaster.mydomain.com/
        type=refreshAndPersist
        searchbase="dc=mydomain,dc=com"
        filter="(objectClass=*)"
        scope=sub
        attrs="*"
        schemachecking=off
        bindmethod=simple
        binddn="cn=replicator,dc=mydomain,dc=com"
        credentials=TOPSEKRIT

Most of this I took as boilerplate from the OpenLDAP documentation. Items to note include:

• the rid is a unique 3-digit integer per slave, used to maintain sync state
• the credentials should be the password you gave the replicator user
• the type can either be refreshAndPersist, or refresh. The latter institutes a simple polling replication, whose interval you can vary with the interval parameter. In our case, we do a poll and then keep the replication search open: our client gets notified immediately when there's any new data matching the replicating search.
• the searchbase is an LDAP search matching the data we wish to be replicated.

And here's the /etc/default/slapd configuration:

SLAPD_SERVICES="ldap://127.0.0.1/"

The slave slapd exists only in this case to serve the local machine. Finally, there's the tricky bit! You need to configure slapd to connect to the master server using a certificate. I'll assume you've created and signed a key and certificate pair for your slave server (see my post Low-tech SSL certificate maintenance for more on this.) Awkwardly, the TLS configuration in slapd.conf is for the server only. Replication works as a client, and thus needs separate configuration. Furthermore, you cannot configure this globally on your machine, as the SSL certificate is a per-user only parameter (see man ldap.conf for more information on this.) Instead, we must set it in slapd's environment. Add these two lines to the end of /etc/default/slapd:

export LDAPTLS_CERT=/etc/ssl/certs/slapd.crt
export LDAPTLS_KEY=/etc/ssl/private/slapd.key

This file is sourced as a shell script by slapd's init script. Amend the path to your certificate and keys as required. Use /etc/init.d/slapd restart and you should be good to go. Finally, we want the slave server to be certain it's talking to the real master. So we also configure client connections to verify the SSL certificate of the peer, in ldap.conf again:

TLS_CACERT      /etc/ssl/certs/myCA.crt
TLS_REQCERT     demand

A little twist One gotcha to notice with both client and server is that slapd runs as the openldap user by default on Debian. Also by default SSL keys are readable only by the ssl-cert group. You'll need add the openldap user to this group, otherwise it won't be able to access /etc/ssl/private. Related articles on this site:

• Turn your world LDAP-tastic

Join the conversation about this post

This past week has been a really good ride on the medical front. As of yesterday, I've dropped the 1g of Tylenol I was taking first thing in the morning. Pills-wise, that leaves me with one Ultrum (Tramadol) at night before bed. Sleeping on my surgery site without pills still causes me to wake up in the middle of the night. Next week's goal is to drop the prescription drugs completely, except for an occasional one on the weekend when I'm doing a lot with Leif. As far as I know, it's okay to take Tylenol and have an occasional glass of wine - I'm going to check with the pharmacist - but there's a winery tour that I'm signed up for late next week...

I'm also back to Physical Therapy and have appointments booked for twice a week over the next few months. My PT doesn't seem to think that I'll need to stay that long, which would be awesome, but my hopes aren't high on that. We keep discovering new things like "Hey, if you had triceps, you could do this exercise that I want you to. But you don't, so do tricep pull downs for now." From not doing a whole lot, I find I have to ask other people to do things like open jars, or open heavy doors still. We'll see how it goes. My lifting/bending/twisting restrictions won't be lifted until August and then the use-more-than-4kg-weights exercises can finally start.

I'm biking around the campus at work a lot now. I'm getting braver with going out on the streets, and think that I'll probably be ready to try taking my road bike for a trip between home and work early next week. I'd originally hoped for this week, but I'm just not there yet.

In other news, I'm hoping that someone can point me to a freer replacement for this: I've wanted Angie to see the movie Wargames for a bit, so I plugged the Windows laptop into the TV and the stereo, rented it off of iTunes, made it full screen and we enjoyed the movie. I'm quite fussy about DRM when it comes to music or movies that I've purchased, but I'm less fussy where I've contracted to watch something for a day. Netflix has an on-demand movie service like iTunes as well, but neither of them seem to run under Linux or Wine. I suspect I'll need to find the person who was hacking wine to work with iTunes and see if (s)he likes occasional shipments of beer. =)

Instead of trying to set a world record in the most stupid world record ever category, finally bringing software development to the same level as sausages, you can do something useful for your computer: download the latest epiphany package in unstable. Thanks to glandium s work, it fixes two notable regressions introduced by gecko 1.9, including the removal of the incredibly unusable certificate error page, also known as I want to be as cryptic as IE. As a side note, after spending a whole sunday trying to fix #393837, without anything remotely looking like success, I m very happy that the Epiphany developers decided to abandon Gecko. Maybe we will finally have a browser that is usable without needing a herd of maniac cultists to fix a bug. Something they won t do anyway they re too busy downloading the longest sausage in the world.

I've posted yesterday an overview of awesome 3 development state on the awesome mailing list. To sum up, everything's fine. I hope this summer will see a good hacker-friendly window manager coming out.

LUG of Glastonbury meets at Tor Leisure in Glastonbury at 7pm tonight (Monday). It will be a general planning meeting, maybe with some GPG-key-signing and other tasks. If you want the LUG to show you something in particular, this will be a good event to attend. This week is BikeWeek 2008 and there's a free cyclists breakfast at the Victorian Cafe on the Weston-super-Mare seafront about 8am Wednesday morning. For events in other areas, stick a partial postcode into the BikeWeek event search. Someone from The Doon Of May was at Hacktionlab 2008 @ Highbury Farm this last weekend, as were Bristol Wireless, who were running the wifi. I've not seen an official announcement, but SPI's board meeting will be on Wednesday at 8pm UK time (1900 UTC), according to my last meeting report. I've heard through BBLUG that the notorious Shevek is co-organising an event called "An Adventure in Technology" at Trinity Community Arts in Bristol on 28 June 2008. It's a follow-up event to the 2003 Bristol Linux and will be an all-inclusive event where everybody is encouraged to bring something along, talk about it, swap ideas, and build things on site. It doesn't have to be Linux-based, but a lot of things will be. The event web site is http://www.techadventure.org/ and you should post there if you have an idea or want to run a session. There will also be a list for people who decide on the day that they want to give a talk.

As the title says - the Ftpmaster dak repository is now using git instead of bzr. I never did like bzr, so I recently asked what the rest of the team thinks of switching to git. Luckily I got no objections, so here we are, with a VCS thats not dead-slow. If you want to do dak development and submit patches you want to read my post to the debian-dak list and then clone the git repo. Sorry, there is no git-daemon support, you need to use the http way if you do not have access to ries. Unfortunately debian-admin thinks git-daemon isn’t good enough to run on ftp-master and that “http is enough”. Bah. (No, I won’t work around it (git-daemon uses high-port, so could be run without admin help, but NO WAY), and I don’t want to put it on alioth). The actual conversion of the repository was done by Dato, with me commenting and asking for stuff. Like the fixup of the old revisions committer names, so all the old CVS revisions now correctly tell the full name/email of the committer, not just the unix login. Or fixing all the bzr commits done by “ajt@ries”, “joerg@ries” and other broken combinations of “login@machine”. I also use his git hook to send commit mails to get commits mailed to the debian-dak lists. [Update] Nice, with the setup we use we found a bug in git. Ha. (a stupid issue with git shared repositories, and reflogs enabled at the same time that basically means you cant commit as the logs got the wrong permissions set whenever someone else did an action, like git gc.) Comments: 2

Ten years ago, most of us wouldn't have dreamt we'd be managing terabits of storage, tens of megabits of bandwidth, arrays of network-distributed services. The height of a programmer's worry would likely be choice of UI toolkit or finding the right way to indent code, and the height of consumer concern deciding which room to put the new computer in. Now the problems associated with managing large networks are becoming real for everyone, right down to the consumer level. Stupendously large amounts of computing resource are available at an instant. Your household probably has more than a terabyte of storage already. Issues such as single sign-on are going to hit home over the next year, as networked computing and entertainment devices profilerate. Features such as Apple's Time Machine will be increasingly vital — software that makes traditionally gnarly sysadmin tasks consumer-friendly. The rebranding of .Mac into "Mobile Me" is also a step in this direction. The impact on developers As software developers, we also have to cope with the effects of this resource-richness. For small sums of money we can get access to large computing clusters, geographically redundant hosting services. Our programs have left the desktop and found their new home on the web. System administration issues loom large upon us, security concerns lurk auspiciously in the corners of our minds. Although the cost of infrastructure has dropped radically, other costs remain high and are going to stay that way. System administrators are not only grumpy, they demand high wages. Commercial software license fees spiral out of control: traditional per-CPU licensing models make little sense when you can quickly bring up tens of machines. The cost in power is already troubling large companies, and there's no reason to suspect the problems won't ripple down. Help is at hand from a variety of technologies. If they don't yet make massive resource management trivial, they at least make it possible. Some of these also inhabit the weird territory of being both the source of a problem and a solution at the same time: virtualization, for example. Distributed revision control systems Distributed revision control is a technology whose time has finally come in popular circles, thanks in part to Linus Torvald's Git system. DRCS has several important impacts on today's developer:

• Branching and, importantly, merging become much cheaper, allowing agile and flexible iterations of development.
• Loosely connected and geographically diverse development becomes much easier. Even within a single organization it is not uncommon to find teams spanning countries, time zones. Complex multi-site VPN setups aren't necessary when a few SSH keys can do the job.
• Revision control becomes packaging on the cheap. Like it or not, the mere tagging of a source tree has now become a valid option for releasing software.

All these trends lower the barrier to entry, increase collaboration and agility of development. You can the value of this as more software tools become free. Selling such tools is rapidly becoming a thing of the past, the advantages of sharing enable the developers at the sharp end to get their jobs done quicker. However, such increased agility and, well, messiness leave other problems to solve, which the next two technologies address. Virtualization Hardware-as-a-service, infrastructure-as-a-service, call it what you will. The ability to create what we used to call entire machines, pick them up and move them around the network is revolutionary, and it's something that will have a real impact on regular developers. The benefits are at several levels.

• Agile infrastructure — a ready supply of new machines makes it a lot easier and cheaper to test different scenarios, architectures, and to separate concerns. If things go wrong, throw away the image and start over. It's all about cutting the administration load.
• A packaging solution — the new macroeconomics of software distribution mean that distributing entire machine images which communicate exclusively via the network is now a feasible way to distribute your software. We must adjust to the notion of distributing appliances, not code. We may mourn the lost crafts of creating RPMs or installers, but let's face it, it's now a waste of time.
• New business models — your application can now be delivered as a black-box appliance, circumventing compatibility issues, or as a service, with virtualization part of the solution to scaling.

Configuration management Computing is a zero-sum game, and despite our increased ability to create and distribute software, problems still exist. We just pushed them to the next level. In good part, this next level is the problem of configuration management. We now have networks and clusters of (virtual) machines, software so agile we need six decimal places to describe its revision levels, and network and authentication paths that are starting to tangle. How do we manage that? One thing developers crave is repeatability. That's why we love our makefiles, autoconf, Ant, rake and so on. It's the one time even the most imperative-minded programmer writes declarative code. We like to say "let the world be like this." Our new sprawling world lacks this feature, and the best of our old toolkits — .debs, RPMs — address things only at the level of packages in a single environment. So developers must look to the world of operations, a territory we probably thought we needn't enter. In this world the new "make" is called Puppet. You write recipes to describe how things ought to be, and Puppet will make it so. I've been spending some time digging into Puppet, and feel excited by the confidence it's giving me. Now my applications exceed single source trees, and single machines, it gives me the means to tie the whole together. This article was going to be solely about Puppet, but that will have to wait now for another time. It's likely you'll have played with virtual machines and distributed revision control, but have you tried Puppet yet? Give it a spin, and let your mind wander over the benefits for your organization and development approaches. Conclusions
For developers and users alike, our world is changing. Hardware, connectivity and increasingly software is becoming cheap or free. The solidity of the old things we put value on — real things you can touch like disks — is eroding. What really matters is our data, our creations, and their communication. If they don't quite yet exist in a universal "cloud" yet, they're certainly getting frisky. As vendors provide solutions for consumers to manage their new domestic infrastructure, developers must look to network-aware toolkits and operations techniques to manage and get the best from their emergent infrastructures. Also on this topic:

• In search of agile infrastructure for web applications (June 2006)
  

Join the conversation about this post

So my Esc key broke today. Maybe due to high humidity last night -- the laptop was generating a lot of spurious lid open events earlier. New keyboard ordered, etc, nothing too unusual given the crummy nature of laptop keyboards (Fujitsu's in specific). The interesting thing though, is that even though I've remapped Caps Lock to generate Escape for now, this threw my typing off to an amazing extent. I don't just have trouble typing ^]:wq in vim. I have trouble typing arbitrary sentences in programs that don't use escape at all. Why? I don't touch type the way people are taught to in school, with fingers on the home keys. I touch type the way you learn to by typing a lot, ad-hoc, until your hands learn exactly how to type the common things you throw at them. This includes being able to type ^]:wq as fast as I can type "joey". My home keys are, approximatly:

Tab Q S X Space       Down . ; ' Enter

(Space and Down are covered by my thumbs.) This might seem a bit weird, but as a programmer, I hit tab a lot, and, enter lots of semicolons, periods, and quotes. As a human on the WWW, I scroll down a lot. As a command-line user, Enter is a key I hit a lot. That's my guesses why my fingers go there. This wasn't planned, it evolved. In fact, I have other home key locations, that are used for other activities too. Now that I'm typing in a lot of text here, my WPM has increased a lot, because my hands have moved closer to the center of the keyboard. Something like this.

      W E F V    J I O  

Although every time I've checked, it's been a slightly different set of keys. My hands know where they are, and seem to make pretty good guesses about where I'll want them in the next word and pick a different "home row", depending. Also, if I'm scrolling around a lot, my left hand switches to:

                      Left Up Right

Point being that this is all very dynamic, and unconcious, and very adapted to this exact keyboard. Learning a new keyboard will take me days of fumbling, that are reminiscent of someone describing switching from QWERTY to DVORAK. And, it turns out that even moving a single key like Esc is close to switching to an entire new keyboard. Needing to reach down to Caps Lock for Esc threw everything else off subtly, even when I wasn't typing that key. Clearly this is not the best way to type. It's just my way. :-/

postfwd 1.10pre7c-1 has been accepted and did hit experimental yesterday.
If you using policyd-weight you might want to give it a try, since its active development stoped. If you are annoyed by too much spam, it maybe a good idea as well.
Please have a look on postfwd.cf, there is no default config provided, since the configuration depends heavily on your environment. But maybe you have a look at /usr/share/doc/postfwd/examples, there is what you expect.
If you have any comments, send a mail to the maintainer or if there are bugs I didn’t spotted yet, please fill one!
In case I don’t respond quit fast, maybe I’m in the delivery room or anywhere else in the hospital or something like that, so please give me some extra time.
In case there are no big issues, I plan to upload the package to unstable in a reasonable time.

I have been tracking the Debian release goals progress for a while now, through bug-squashing parties and using various iterations of wiki pages. It's time to write down some thoughts.

First of all, I think the concept of release goals is a fantastic idea. It may be one of the most important conceptual moves in Debian of late. It allows developers to implement distribution-wide changes without having to negotiate with every package maintainer individually and without having to go through the vicious no-change-without-policy/no-policy-without-established-practice maze. It moves the focus back on operating system development instead of package hoarding.

In the future, I think we should come up with some "sexier" release goal descriptions. If you look over the current list, many of these tasks are implementation-level, without any obvious benefit to users: double-build support? no unmet recommends? Or they have incomprehensible descriptions. The pretty much only release goal that has a real user benefit, namely the piuparts cleanness, ensuring clean distribution upgrades, is both the most cryptic and actually the most neglected one in terms of development attention. Don't get me wrong, all these goals are very valid. For better or worse, the press has started to pick up these release goals to put in their various Debian release candidate timeline pre-announcement articles, and next time around we should have something cool in there. If armel support or KDE 4 integration had been release goals, the marketing effect would be better.

The next thing is tracking bugs. If you have some time on your hands, how do you find a release goal bug that you can help with? I had worked up this wiki page to help myself, but it is still too weird. The fact that bts.turmzimmer.net exists shows that the BTS is lacking features or is too hard to use or both. I'm thinking, we should regularly copy the whole BTS into an SQL database and make queries from there. Think aggregation, grouping, full-text search! I'm not familiar with the BTS internals or the LDAP gateway, but I know my SQL. So if anyone knows the other end of this deal, I'd love to chat about setting something like this up.

So then, what happens to the release goals after the release? Has this been thought through?

Obviously, some release goals become technically self-enforcing. Python 2.5 and GCC 4.3 are now the default, and anything that doesn't work with these will have permanent trouble in the future.

Release goals that implement a distribution-wide feature, such as the LSB-based init system enhancements, will really need to become part of policy after the release. Otherwise they would revert to being normal bugs without any NMU powers and the other special attention given to release goals. So over time, the support for the former release goals across packages would again decline. To keep it up, you'd either have to repeat the release goal indefinitely, which would be silly, especially when there are no problems right now, or you'd have to make it a part of policy to enforce it in the future. Essentially, you'd have to consider most release goals trial policy changes, which become permanent if successful.

Near-examples of the above are the eternal release goals of IPv6 support and LFS support. We really have these under control quite well now -- although I still wouldn't dare to call Debian or any general-purpose Linux distribution fully IPv6-ready -- but nowadays these problems are almost all upstream code problems that Debian can't really fix very well on the packaging side. So under the current approach these release goals will exist forever, or worse, will be dropped from release goal status.

And then of course, some release goals fail to be reached. These should reapply for the next release, if interest persists. The dash goal comes to mind.

By the way, we had 13 release goals. 3 are completely done, 5 are pretty much done (less than 10 bugs), 1 more looks like it could get pretty much done before the release, and 4 will most likely fail to be accomplished.