After some time and various patches there's a new version of xscreensaver that fixes a lot of issues with 5.05 and XineRama. Thanks to all that submitted patches: Bas Zoetekouw, Vincent Crevot, Lionel Elie Mamane and Daniel Dickinson for the desktop file. The deb files are available here along with the dsc, changes, diff.gz and orig.tar.gz files. If you don't use i386 then do:

dget --build \
http://debian.rivco.info/xscreensaver/5.05-3/xscreensaver_5.05-3.dsc

One of the traditional problems with power management has been balancing power savings against reduced functionality. The two traditional approaches to this have been to have sane defaults or to let the user tweak them, with a hybrid approach being to have sane defaults and an "advanced" box that lets users set their own parameters.

Both of these options suck.

The problem with sane defaults is that power management is an area where the "sane default" genuinely does differ between users. Usage patterns will determine the appropriate spindown period for a hard drive. AHCI's ALPM will disable hotswap detection, which is fine for home users but a pain if people want to rip out a failed RAID element. ALPM also introduces a small amount of latency (on the order of a seek in the most aggressive mode), which is entirely irrelevant for almost everyone but is a pain if you're trying to offer i/o latency guarantees. PCI-E link power management adds small quantities of latency. Letting the CPU run to full speed whenever it wants to is good for power management but potentially bad for thermal management. Screensaver timeouts vary depending on whether you're watching a movie or not.

The problem with leaving everything tweakable is that you're asking users to make choices about things but not giving them the information they need to make those choices. Whether you get a power saving from hard drive spindown depends on whether the drive is idle for long enough to save the power you'll spend spinning it back up. Get it wrong and you'll be putting your drive under extra load, reducing performance and consuming more power than you were to begin with. If you limit the maximum speed of your CPU to keep it cool, you'll be wasting power and performance even when it's well below the thermal limits. Disabling ALPM because you might want to change a RAID device at some stage means you're consuming power despite generally not needing to use the additional functionality.

But while these options both suck in different ways, they share one fundamental failure. They both imply static configuration. The correct power management policy depends on the way that a machine is being used at a specific instant in time, and no matter how many power management profiles you add there's no way for a user to switch between them fast enough to obtain a decent power/performance ratio.

Adaptive power management adopts a different strategy. Rather than define a static policy (or several static policies), the hardware is monitored to determine the best power management settings. An example of this is the ondemand CPUfreq governor. Rather than attempt to provide a static configuration, the processor frequency is automatically changed to meet demand while also keeping the overall power consumption as low as possible. This patch adds a thermal component to its policy, preventing the machine from overheating. Result? No requirement to provide tunables to userspace.

Something similar is achievable with hard-drive spindown. By simply monitoring how often reads or writes actually hit disk, it's possible to automatically adjust the spindown period to obtain decent power savings while reducing the number of spinups. This approach is imperfect - in an ideal world we'd know to spin the drive up before it's needed, but that simply isn't possible. However, the user is almost certainly unable to make a better guess. It's not realistic for a user to be able to predict whether or not a given page is still in cache or has been evicted. Nothing is lost by providing a dynamic policy, and another unnecessary tunable has vanished.

The difficult part of an adaptive policy comes when roles are switched in ways that don't correspond to changse in hardware status. One of the most common cases is that of screensaver settings when the user is watching a video. It's not practical for the kernel or a system daemon to predict this case, so we have two choices:

• Get the user to switch to a "media" power policy and then remember to switch back afterwards
• Have the media player deactivate the screensaver at startup and reactivate it when the video isn't playing

.One of these choices involves the user having to explicitly state something that's obvious - playing a video automatically implies the desire to adopt a different power management policy, and so asking the user to manually adjust that is ridiculous.

The loss of hotswap notifications when ALPM is enabled? Simply reenable it if SMART starts complaining or the RAID set becomes degraded or a filesystem is unmounted or any of the other things that will have to be done before a drive can be safely removed. Latency an issue? Let userspace tell you what its latency requirements are. When we get down to it, the only settings that we can't infer are things like idle-time-to-suspend, screensaver settings and default brightnesses. Which, curiously, are about the only things that Mac OS exposes[1].

The problem we face is that we don't currently have interfaces to let applications tell us what their requirements are. This is theoretically straightforward, but getting it right is important. Beyond latency and screensaver issues, what constraints do applications want to impose? How can we expose an interface to those constraints? This is the sort of thing we'll be looking at at the Linux Plumbers Conference, so if you have ideas about what they'd need to look like then sign up. Or, even better, submit a paper and be part of the process that gets Linux's power consumption as close to the theoretical minimum as possible.

[1] My Vista system gives me 25 options which can be set for each of AC and battery.

The plot..
Some years ago, when I started using linux, I felt the need for video chat, and hence intended to buy a webcam..
Unfortunately, while most of usb hardware is standard, such as the usb mass storage class, this is not at all the case for video, although it is starting to get better with the usb video device class which is more and more used by modern webcams.
So, at this time, each webcam would have its own communication protocol, and each driver had to implement a different way (...)

I recently joined the community based around the TED conference [1]. The TED conference is expensive ($6000US) and has a long waiting list (the 2009 conference is sold out) so it seems quite unlikely that I will ever attend one. But signing up to the web site is easy and might offer some benefit. One thing that interested me was that part of the sign-up process requests that you select up to 10 words from the list above to describe yourself. Some of the words seem almost mandatory for anyone who is interested in what TED has to offer (I find it difficult to imagine someone declaring that they are not an “activist” or a “change agent” while wanting to be involved with TED in any way). The range of words also seems quite strange, there are some professions mixed with educational status, marital status, and religion. The way it is laid out would tend to encourage people to make a decision as to which aspects of their life are more important, is career, marital status, or religion more important? Given the nature of TED I’m wondering whether the intentionally did a bad job of that part of the site design to encourage people to think about these issues. It seems to me that a better way of doing this would be to provide a few suggestions and allow people to fill in text fields with their own values. Even defining marital status can require many choices and there is no limit to the number of religions and careers. If you try to make a comprehensive list then you will end up doing what British Airways did with their frequent flyer membership application page [2]. Even disregarding the choices of spelling (EG Admiral vs Admiraal and Brig Gen vs Brig General vs Brigadier General) the British Airways list is unreasonably long, and I doubt that anyone who deserves the title “Her Magesty” or “His Holyness” is going to be interested in frequent flyer points. Also I wonder which of the entries in the TED list would be most commonly accepted by the free software community. It seems that activist and technologist would be quite popular. Here is the list in text form for those who can’t get the picture above:

Activist, Agnostic, Architect, Artist, Atheist, Athlete, Blogger, Brainstormer, Buddhist, Business adviser, Business leader, Business mentor, Care taker, Change Agent, Christian, Clergy, Concerned citizen, Connector, Consultant, Designer, Doctor, Educator/Teacher, Engineer, Entrepreneur, Environmentalist, Event planner, Explorer, Filmmaker, Finance professional, Foodie, Gay, Global soul, Hindu, Idea generator, Industrialist, Inventor, Investor, Jewish, Job-seeker, Journalist, Lawyer, Life mentor, Marketer, Musician, Muslim, Parent, Performer, Philanthropist, Photographer, Policy maker, Potential employer, Producer, Project manager, Promoter, Public servant, Real Estate professional, Sales specialist, Scientist, Single, Social entrepreneur, Startup, Student, Technologist, Web guru, World traveler, and Writer/Editor.

• [1] http://www.ted.com/index.php/pages/view/id/112


• [2] http://www.britishairways.com/travel/inet/public/en_gb

Share This

If you want a reliable network then you need to determine an appropriate level of redundancy. When servers were small and there was no well accepted virtual machine technology there were always many points at which redundancy could be employed. A common example is a large mail server. You might have MX servers to receive mail from the Internet, front-end servers to send mail to the Internet, database or LDAP servers (of which there is one server for accepting writes and redundant slave servers for allowing clients to read data), and some back-end storage. The back-end storage is generally going to lack redundancy to some degree (all the common options involve mail being stored in one location). So the redundancy would start with the routers which direct traffic to redundant servers (typically a pair of routers in a failover configuration - I would use OpenBSD boxes running CARP if I was given a choice in how to implement this [1], in the past I’ve used Cisco devices). The next obvious place for redundancy is for the MX servers (it seems that most ISPs have machines with names such as mx01.example.net to receive mail from the Internet). The way that MX records are used in the DNS means that there is no need for a router to direct traffic to a pair of servers, and even a pair of redundant routers is another point of failure so it’s best to avoid them where possible. A smaller ISP might have two MX machines that are used for both sending outbound mail from their users (which needs to go through a load-balancing router) as well as inbound mail. A larger ISP will have two or more machines dedicated to receiving mail and two or more machines dedicated to sending mail (when you scan for viruses on both sent and received mail it can take a lot of compute power). Now the database or LDAP servers used for storing user account data is another possible place for redundancy. While some database and LDAP servers support multi-master operation a more common configuration is to have a single master and multiple slaves which are read-only. This means that you want to have more slaves than are really required so that you can lose one without impacting the service. There are several ways of losing a server. The most obvious is a hardware failure. While server class machines will have redundant PSUs, RAID, ECC RAM, and a general high quality of hardware design and manufacture, they still have hardware problems from time to time. Then there are a variety of software related ways of losing a server, most of which stem from operator error and bugs in software. Of course the problem with the operator errors and software bugs is that they can easily take out all redundant machines. If an operator mistakenly decides that a certain command needs to be run on all machines they will often run it on all machines before realising that it causes things to go horribly wrong. A software bug will usually be triggered by the same thing on all machines (EG I’ve had bad data written to a master LDAP server cause all slaves to crash and had a mail loop between two big ISPs take out all front-end mail servers). Now if you have a mail server running on a virtual platform such that the MX servers, the mail store, and the database servers all run on the same hardware then redundancy is very unlikely to alleviate hardware problems. It’s difficult to imagine a situation where a hardware failure takes out one DomU while leaving others running. It seems to me that if you are running on a single virtual server there is no benefit in having redundancy. However there is benefit in having an infrastructure which supports redundancy. For example if you are going to install new software on one of the servers there is a possibility that the software will fail. Doing upgrades and then having to roll them back is one of the least pleasant parts of sys-admin work, not only is it difficult but it’s also unreliable (new software writes different data to shared files and you have to hope that the old version can cope with them). To implement this you need to have a Dom0 that can direct traffic to multiple redundant servers for services which only have a single server. Then when you need to upgrade (be it the application or the OS) you can configure a server on the designated secondary address, get it running, and then disable traffic to the primary server. If there are any problems you can direct traffic back to the primary server (which can be done much more quickly than downgrading software). Also if configured correctly you could have the secondary server be accessible from certain IP addresses only. So you could test the new version of the software using employees as test users while customers use the old version. One advantage a virtual machine environment for load balancing is that you can have as many virtual Ethernet devices as you desire and you can configure them using software (without changing cables in the server room). A limitation on the use of load-balancing routers is that traffic needs to go through the router in both directions. This is easy for the path from the Internet to the server room and the path from the server room to the customer network. But when going between servers in the server room it’s a problem (which is not insurmountable, merely painful and expensive). Of course there will be a cost in CPU time for all the extra routing. If instead of having a single virtual ethernet device for all redundant nodes you have a virtual ethernet device for every type of server and use the Dom0 as a router you will end up doubling the CPU requirements for networking without even considering the potential overhead of the load balancing router functionality. Finally there is a significant benefit in virtual machines for reliability of services. That is the ability to perform snapshot backups. If you have sufficient disk space and IO capacity you could have a snapshot of your server taken every day and store several old snapshots. Of course doing this effectively would require some minor changes to the configuration of machines to avoid unnecessary writes, this would include not compressing old log files and using a ram disk for /tmp and any other filesystem with transient data. When you have snapshots you can then run filesystem analysis tools on the snapshots to detect any silent corruption that may be occurring and give the potential benefit of discovering corruption before it gets severe (but I have yet to see a confirmed report of this saving anyone). Of course similar snapshot facilities are available on almost every SAN and on many NAS devices, but there are many sites that don’t have the budget to use such equipment.

• [1] http://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol

Share This

To my horror, I recently saw this online SSH key generator. I hope nobody reading this needs to be told why this is a bad idea. However, in case you do, here are a few reasons:

• Every SSH implementation I know of - certainly all the major ones - that support public key authentication also provide a key generation utility. Even aside from all the good reasons not to, there is simply no reason why you should need to use a web-based tool in the first place.
• How can you trust the person running this site? Without implying that I know he or she is untrustworthy (I don't), and with the best will in the world, it's a big Internet with a lot of nasty people on it. Do you really want somebody you don't know in a position to keep a copy of all your private keys?
• Even if the person is trustworthy, the server running sshkeygen.com is now a giant blinking target. If lots of people use it, there is every incentive in the world for the bad guys to try to take control of it so that they can keep a copy of all your private keys. (Or, as we know from recent bitter experience, they can just give out keys from a limited set and it will probably take a couple of years before anyone notices ...)
• The front page of sshkeygen.com says that the keys are escrowed. The plain English meaning of this would be that the operator of that site keeps a copy of the private key, to be held in trust in case (presumably) you lose it and need to retrieve it. Normally this sort of thing depends on a legal trust relationship, perhaps linked to a contract. What does it mean here? Is it just a buzzword? If it isn't, then this just makes sshkeygen.com even more of a target.
• sshkeygen.com delivers keys to you over unencrypted HTTP. Yes, this is on its to-do list. That isn't really an excuse.
• Even if keys were delivered over HTTPS, that still relies on people diligently checking the authenticity of the certificate. A self-signature (as suggested as an alternative in the to-do list) would be impossible to check with any reliability; and will people who have trouble with non-web-based key generation software really be able or inclined to confirm the signature chain? Browsers typically don't enforce this very strictly, or if they do they provide fairly simple ways to bypass the enforcement, simply because so many sites have broken or poorly-signed SSL certificates, and keeping up with all the CAs is pretty hard work too.
• Furthermore, delivering private keys over HTTPS makes that SSL certificate a single giant blinking target. Might it be compromised? How would you tell? What servers would need to be compromised in order to get a copy of the private SSL key?
• Sure, Debian is in an awkward position here given the recent OpenSSL random number generation vulnerability. However, how do you know that sshkeygen.com is running on a system that doesn't suffer from this? (As it happens, I have checked, and it doesn't appear to suffer from this vulnerability - but most people won't check and won't know how to check.)

I think this is probably being done in innocent seriousness (although I kind of hope it's a joke in poor taste), and have e-mailed the contact address offering to explain why it's a bad idea.

I just got Guitar Hero for the Wii. And I played it for 5 straight hours last night. Oh dear.

Part of the joy of spending 3 months stoned is that my Inbox had managed to fill up quite nicely with all sorts of crap. In a desperate attempt to prune things, I've taken everything from LiveJournal and Facebook and archived them.

The LJ things are often bits where people have wished me well about the surgery and the recovery. But at over 80 conversations (some of which were multiple people), I'll just *never* get to them. I love you all for the well wishes, and I hope that I've managed to connect with all of you after them at some point.

I've also done the same with Facebook emails on the theory that Wall posts are probably stale by now (or already replied to), and the Inbox there will tell me what I haven't replied to (something like 77 posts)

I started the evening at 1350 messages, I'm now down to 497. Hopefully I'll get it down to zero over the course of this week.

Bed time!

As one of the maintainers of debian.o-hand.com I use the always wonderful pbuilder and cowbuilder to rebuild packages originally build for Debian Sid for Debian Etch, Ubuntu Gutsy, Hardy, and so on. Continually typing the commands to update the cowbuilders can get tiresome fast so last week I scratched the itch and produced Zebu. As of version 0.1 it is barely functional but it does let you update or login to a cowbuilder. It requires that the cowbuilders are named /var/cache/pbuilder/*.cow and doesn't support "traditional" pbuilder rootstraps yet, but that is planned. Anyway, cowbuilders are the future. If anyone else thinks this could be useful there is a tarball and a Bazaar repository. I must also thank the wonderful Ulisse Perusin for the rocking icon he created. NP: Cosmos, Murcof

I have just uploaded an important update to apt-p2p that is highly recommended for all users. At some point over the last 48 hours bittorrent nodes have started to infiltrate the apt-p2p DHT (you may have noticed many ValueError tracebacks in your log file). The (mainline) bittorrent DHT is very similar in protocol to apt-p2p, and so the nodes are able to partially communicate and pollute each others routing table. I didn't think this would ever happen, as there wouldn't seem to be a reason for them to ever come into contact, but somehow it did. I've made some minor changes that exploit the differences between the two protocols to exclude bittorrent nodes from the apt-p2p routing table, and to drop any requests from bittorrent nodes (which should prevent apt-p2p nodes from polluting the bittorrent DHT). However, it is very important that all users upgrade to this new version to prevent any further mixing of the DHTs.

I was looking for a Python lexer and parser, and found out that antlr can emit Python code with only minor twiddles of the input grammar.

The antlr site has a "Hello World" type of grammar for testing it out in Java: http://www.antlr.org/article/cutpaste/index.html

What follows is that page ported to Python and Debian/Ubuntu. On a Debian-derived system:

$ sudo apt-get install antlr

Create a file called "t.g" with the following contents:

options  
	language=Python;
 
class P extends Parser;
startRule
    :   n:NAME
         print "Hi there, "+n.getText() 
    ;
class L extends Lexer;
// one-or-more letters followed by a newline
NAME:   ( 'a'..'z' 'A'..'Z' )+ NEWLINE
    ;
NEWLINE
    :   '\r' '\n'   // DOS
        '\n'        // UNIX
    ;

Because antlr isn't in the CLASSPATH by default, the maintainer has included a "runantlr" script, which handles this for you. So,

$ runantlr t.g

This will create L.py, P.py and PTokenTypes.txt.

Now create main.py:

#!/usr/bin/python
import sys
import L
import P
l = L.Lexer(sys.stdin)
p = P.Parser(l)
p.startRule()

And run the resulting file:

$ python main.py
Jeff
Hi there, Jeff
$

I promised to post some updates about how the Aptitude project would be going so here it is. Here’s what aptitude-gtk currently looks like : Well, nothing very special right now. It’s really only a testing interface. The final GUI will probably be different. Now you may be interesting by what’s going on behind it ? Where to start ? Aptitude is written in C++ with varying dosages of OO coding depending on the age of the code :). For the most part, the code is neat, clean and easy to understand. Sometimes, it’s overengineered: the code that implements the progress status of packages downloads and actions is exhaustive enough to cover the setup phases of a hydraulic dam, although sometimes you can still see progress bars mysteriously going backwards. Sometimes, Daniel Burrows will be happy to provide you with complimentary barf bags while explaining some ungodly oddities that have existed for years because it’s easier to remember the ritual dances to do than to make stuff behave logically. Currently Aptitude-gtk is able to execute the libapt-pkg initialization phase, load up repository lists and update lists (think “aptitude update”). The next big task will be to load up the installed packages list and display it. It involves quite a lot of code because of all the things that connect to it : incremental package searches, sorts, filters and so on. It’s been very interesting so far. By the way, the minesweeper feature is already implemented, courtesy of gnomine

Or theming PS1 I am not fanatical about my command prompt. No sirree, not me. It is just that I spend half my life either staring at an Emacs window, or at an xterm command prompt; so even a marginal boost in productivity goes a long way. And I am often logged in over ssh to machines half a continent away, and am still comfortable enough on the Linux VT to spend time there, and I often do not have the GUI gew-gaws feeding me data. Hmm. Data. inpuuut. No. Must focus. With all these open xterms and Emacs terminal mode frames floating around, it is easy to lose track of where I am on each terminal, and what the working directory is. So I want my command prompt to help me keep track of where I am. If the terminal is an xterm, the title can be setup like

 "user@host:../shotened/path/to/current/working/dir"

I want to know what machine, id (am I root?), and directory I am in. If I am deep down in the labyrinths of some work related directory tree, I want the path to be pruned, from the left, one component at a time. However, this does not help me on the console; so I also want the path to be in the command prompt; but it should not take up too much of the command line; and ideally, should just go away as I type a longish command. Gawd, I love zsh. I once did a analysis of the command history. The most often used command sequences were

 cd some-place, ls

So. pwd and ls. I really really want to see the directory listing when I change directory to a new one. As one grows old, memorizing the directory contents for dozens of machines all the time taxes the gray cells a bit. So, I figure, why not let the command prompt handle all that? Having the current working dir always visible cuts down a heck of a lot on the pwd commands, and so all that s left is to insist that the command prompt thingy always run ls after a change directory. Simple enough. Saves on typing. And time. And this is not just some crazy talk. I want help with noticing whether the previous command exited with an error status (useful for commands that normally do not create an output). If I am logged in to a machine on battery power, I want to know that. I also like visual cues to the amount of power remaining (good for my laptop on the long flights). I don t want to have to know f the machine uses APM or ACPI, I just want my prompt color to change as the power fades. I want my command prompt to let me know if I am in a directory which is under version control, and if so, what branch I am on. (I occasionally have to come in contact with arch, bzr, git, subversion, svk, and mercurial). If I am in a version controlled project, where I am relative to the root of the checked out tree is often more important than the absolute path, so I want to see relative paths, not absolute paths. I want to know if there are uncommitted files in the working directory. Visually. I want to be reminded if I am in the middle of an ongoing

 rebase -i .

This is not asking for too much, is it? So, here is a screen-shot of this in action: I start from my home directory, go to a directory not under version control, go to a project under git, then a different project with uncommitted files, and then finally to a subversion checkout. All with an angry fruit salad of colors warrantied to make Martin Krafft want to claw his eyes out . If you use zsh, then just grab hold of this, and then do:

 autoload -U promptinit
 promptinit
 prompt manoj

This might be a bit of a hit on slow machines, but even my laptop is a core 2 duo, so I do not find it noticeable. Oh, and if you liked this article, you might also like Theming Emacs, and Theming XTerms.

Ever the sucker for punishment, I decided to pick three difficult things and stick them all together: LDAP, SSL and replication. Here's how to make it go on Debian and Ubuntu. The problem You want LDAP replication to happen over the internet, and you want it to happen securely. The caveat I'm not going to tell you how to set up your LDAP from scratch here: I'm assuming you've reached a solution you're happy with and want to replicate it. The solution We're going to set up a replicating slave LDAP server, which communicates with the master over the internet via an SSL-protected connection. Enabling replication First up, the master LDAP server needs to be configured to permit replication.  The key lines to add to your slapd.conf include:

moduleload syncprov
index entryCSN,entryUUID eq
overlay syncprov
syncprov-checkpoint 100 10
syncprov-sessionlog 200

These load up the synchronization module, add indices which make sync go faster, and enable sync. For more detail see the OpenLDAP site. Next you need to add a replicator user to your LDAP database, give your replicator user access to passwords as well as general read access. To create the replicator user, I made this simple LDIF file and fed it to ldapadd.

dn: cn=replicator,dc=mydomain,dc=com
objectClass: simpleSecurityObject
objectClass: organizationalRole
cn: replicator
description: LDAP replicator
userPassword: TOPSEKRIT

Once this user is in your LDAP database, you should give it read access to passwords (I assume you've already given read access to authenticated users.) I have this in my slapd.conf:

access to attrs=userPassword,sambaNTPassword,sambaLMPassword
...
   by dn="cn=replicator,dc=mydomain,dc=com" read

To check that this works, try using ldapsearch to check that the passwords are returned:

ldapsearch -x -D cn=replicator,dc=mydomain,dc=com \
  -W   grep -i password

Enter the replicator password when prompted, and you should see the encrypted passwords from your LDAP database. Securing access Now you've got replication enabled on the master, you will want to ensure it is available on the internet only via TLS or SSL. Here's what I added to slapd.conf to enable this:

TLSCertificateFile      /etc/ssl/certs/ldapserver_crt.pem
TLSCertificateKeyFile   /etc/ssl/private/ldapserver_key.pem
TLSCACertificateFile    /etc/ssl/certs/myCA.pem
TLSVerifyClient         demand

As you will guess from the configuration, the first two lines set the SSL key and certificate the master uses (see "A little twist" below for an important note on key permissions.) The third line tells slapd where to find my site-local certificate authority (CA), and the fourth line says slapd must require any connecting client to have a valid SSL certificate signed by the site-local CA. This is important, as it provides a second layer of access control: a replicating client must connect using a certificate you signed, plus the replicator password. Before this enables TLS access, we must tell slapd which network interfaces to listen on. To do this, edit the SLAPD_SERVICES variable in /etc/default/slapd. Here's my configuration:

SLAPD_SERVICES="ldap://127.0.0.1/ ldap://192.168.0.1/ ldaps:///"

This enables regular LDAP on the loopback and intranet network interfaces, and LDAP/SSL on all interfaces, including the public internet. So, with slapd restarted we are at this situation: connections are now possible from the internet, as long as they are made over SSL with a certificate signed by our site-local CA. (In fact, you can make much finer-grained access restrictions in your configuration than I have done. Using LDAPS rather than TLS over regular LDAP is a rather broad precaution. As explained on the OpenLDAP site, the ssf= parameter can be used to require a certain level of secure connectivity on a per-user or client basis.) Setting up the replicating server
Your slave server should have the same configuration as the master, except you can leave out the bits enabling replication. Firstly, you'll need add to slapd.conf the replication configuration:

syncrepl rid=123
        provider=ldaps://ldapmaster.mydomain.com/
        type=refreshAndPersist
        searchbase="dc=mydomain,dc=com"
        filter="(objectClass=*)"
        scope=sub
        attrs="*"
        schemachecking=off
        bindmethod=simple
        binddn="cn=replicator,dc=mydomain,dc=com"
        credentials=TOPSEKRIT

Most of this I took as boilerplate from the OpenLDAP documentation. Items to note include:

• the rid is a unique 3-digit integer per slave, used to maintain sync state
• the credentials should be the password you gave the replicator user
• the type can either be refreshAndPersist, or refresh. The latter institutes a simple polling replication, whose interval you can vary with the interval parameter. In our case, we do a poll and then keep the replication search open: our client gets notified immediately when there's any new data matching the replicating search.
• the searchbase is an LDAP search matching the data we wish to be replicated.

And here's the /etc/default/slapd configuration:

SLAPD_SERVICES="ldap://127.0.0.1/"

The slave slapd exists only in this case to serve the local machine. Finally, there's the tricky bit! You need to configure slapd to connect to the master server using a certificate. I'll assume you've created and signed a key and certificate pair for your slave server (see my post Low-tech SSL certificate maintenance for more on this.) Awkwardly, the TLS configuration in slapd.conf is for the server only. Replication works as a client, and thus needs separate configuration. Furthermore, you cannot configure this globally on your machine, as the SSL certificate is a per-user only parameter (see man ldap.conf for more information on this.) Instead, we must set it in slapd's environment. Add these two lines to the end of /etc/default/slapd:

export LDAPTLS_CERT=/etc/ssl/certs/slapd.crt
export LDAPTLS_KEY=/etc/ssl/private/slapd.key

This file is sourced as a shell script by slapd's init script. Amend the path to your certificate and keys as required. Use /etc/init.d/slapd restart and you should be good to go. Finally, we want the slave server to be certain it's talking to the real master. So we also configure client connections to verify the SSL certificate of the peer, in ldap.conf again:

TLS_CACERT      /etc/ssl/certs/myCA.crt
TLS_REQCERT     demand

A little twist One gotcha to notice with both client and server is that slapd runs as the openldap user by default on Debian. Also by default SSL keys are readable only by the ssl-cert group. You'll need add the openldap user to this group, otherwise it won't be able to access /etc/ssl/private. Related articles on this site:

• Turn your world LDAP-tastic

Join the conversation about this post

It’s a common practice when hosting email or web space for large numbers of users to group the accounts by the first letter. This is due to performance problems on some filesystems with large directories and due to the fact that often a 16bit signed integer is used for the hard link count so that it is impossible to have more than 32767 subdirectories. I’ve just looked at a system I run (Bluebottle anti-spam email service [1]) which has about half a million accounts and counted the incidence of each first letter. It seems that S is the most common at almost 10% and M and A aren’t far behind. Most of the clients have English as their first language, naturally the distribution of letters would be different for other languages. Now if you were to have a server with less than 300,000 accounts then you could probably split them based on the first letter. If there were more than 300,000 accounts then you would face the risk of having there be too many account names starting with S. See the table below for the incidences of all the first letters. The two letter prefix MA comprised 3.01% of the accounts. So if faced with a limit of 32767 sub-directories then if you split by two letters then you might expect to have no problems until you approached 1,000,000 accounts. There were a number of other common two-letter prefixes which also had more than 1.5% of the total number of accounts. Next I looked at the three character prefixes and found that MAR comprised 1.06% of all accounts. This indicates that splitting on the first three characters will only save you from the 32767 limit if you have 3,000,000 users or less. Finally I observed that the four character prefix JOHN (which incidentally is my middle name) comprised 0.44% of the user base. That indicates that if you have more than 6,400,000 users then splitting them up among four character prefixes is not necessarily going to avoid the 32767 limit. It seems to me that the benefits of splitting accounts by the first characters is not nearly as great as you might expect. Having directories for each combination of the first two letters is practical I’ve seen directory names such as J/O/JOHN or JO/JOHN (or use J/O/HN or JO/HN if you want to save directory space). But it becomes inconvenient to have J/O/H/N and the form JOH/N will have as many as 17,576 subdirectories for the first three letters which may be bad for performance. This issue is only academic as far as most sys-admins won’t ever touch a system with more than a million users. But in terms of how you would provision so many users, in the past the limits of server hardware were approached long before these issues. For example in 2003 I was running some mail servers on 2RU rack mounted systems with four disks in a RAID-5 array (plus one hot-spare) - each server had approximately 200,000 mailboxes. The accounts were split based on the first two letters, but even if it had been split on only one letter it would probably have worked. Since then performance has improved in all aspects of hardware. Instead of a 2RU server having five 3.5″ disks it will have eight 2.5″ disks - and as a rule of thumb increasing the number of disks tends to increase performance. Also the CPU performance of servers has dramatically increased, instead of having two single-core 32bit CPUs in a 2RU server you will often have two quad-core 64bit CPUs - more than four times the CPU performance. 4RU machines can have 16 internal disks as well as four CPUs and therefore could probably serve mail for close to 1,000,000 users. While for reliability it’s not the best idea to have all the data for 1,000,000 users on internal disks on a single server (which could be the topic of an entire series of blog posts), I am noting that it’s conceivable to do so and provide adequate performance. Also of course if you use one of the storage devices that supports redundant operation (exporting data over NFS, iSCSI, or Fiber Channel) then if things are configured correctly then you can achieve considerably more performance and therefore have a greater incentive to have the data for a larger number of users in one filesystem. Hashing directory names is one possible way of alleviating these problems. But this would be a little inconvenient for sys-admin tasks as you would have to hash the account name to discover where it was stored. But I guess you could have a shell script or alias to do this. Here is the list of frequency of first letters in account names:

First Letter	Percentage
a	7.65
b	5.86
c	5.97
d	5.93
e	2.97
f	2.85
g	3.57
h	3.19
i	2.21
j	6.09
k	3.92
l	3.91
m	8.27
n	3.15
o	1.44
p	4.82
q	0.44
r	5.04
s	9.85
t	5.2
u	0.85
v	1.9
w	2.4
x	0.63
y	0.97
z	0.95

• [1] http://www.bluebottle.com/

Share This

As he’s blogged, on Monday the wonderful John Carr of Conduit and very cool bzr-mirror.gnome.org fame will be starting work at Codethink. He’s initially going to be working on our top secret project and looking at bringing Conduit to mobile devices. We’re also moving offices to the beautiful Peak District village of Hope, in the Hope Valley. I’ll post some pictures when we’ve got it all set up. Apparently the name ‘Hope’ derives from a celtic word for valley, so we’ll be in Valley valley

It happens from time to time that bug reports I file receive attention, but I don t notice because our bug tracking system still does not auto-subscribe bug submitters to their own bugs (see bug #37078 and bug #351856). I thus decided to take the matter into my own hands. Just in time, before I started hacking this up myself, I found Justin Pryzby s procmail recipies, which are installed to /usr/share/doc/devscripts/examples/bts_autosubscription.procmail by the devscripts package. The result is available in my mailfilter git repository. So far, I only auto-subscribe in response to bugs I file; Justin also auto-subscribes to bugs he manipulates via the control bot. For completeness, I also wanted to subscribe to all bugs that I have submitted. This turned out to be easier than I thought, thanks to bts in the devscripts package; note the sleep instruction in the loop to prevent hammering the system:

bts select submitter:madduck@debian.org \
    while read bugno; do
      echo X-debbugs-autosubscribe: madduck \
          sendmail -f madduck@debian.org $ bugno -subscribe@bugs.debian.org
      echo subscription to \#$ bugno  sent
      sleep 30
    done

NP: The Dukes of Leisure: The Dukes of Leisure

This new version of DebTorrent finalizes the work I previously uploaded to experimental. This new version is the first in unstable to use unique piece numbers to create long-lived torrents for testing and unstable. It also comes with a new version of the apt-transport-debtorrent package, which can now tell APT not to cache any package files downloaded with DebTorrent, which should save some disk space for some people. The locations of the long-lived torrent files and piece files that DebTorrent uses has also changed, as they are now hosted on my merkel page and should be more up to date. However, I was not able to get the ssh trigger for the sync from ftp-master setup, nor am I able to access the projectb database through pygresql. If anyone knows how I can get those working, please email me. Here is the changelog for debtorrent:

• Display the torrent identifier on the local status page rather than the info hash (Closes: #465339)
• Add support for the new No Content message to apt-transport-debtorrent so that apt treats debtorrent sources as local
• Make the pieces and unique piece number locations config options, and use the new ones at merkel.debian.org/~camrdale/.
• Decompress the files needed to create torrents so they are created and started faster (this and the previous Closes: #463676)

And for apt-transport-debtorrent:

• Add a config option to make debtorrent sources local (Closes: #477383)
• Upgrade the recommends on debtorrent to 0.1.7.
• Add a config file for APT to load, and a manpage describing the options.

Lucas Nussbaum recently posted some graphics showing the number of packages that are grabbed from Debian and kept untouched, modified, out of sync, etc, etc.

But, "statistics are tricky":

---

$ wget http://patches.ubuntu.com/p/php5/php5_5.2.6-1ubuntu3.patch
--2008-06-19 22:25:03-- http://patches.ubuntu.com/p/php5/php5_5.2.6-1ubuntu3.patch
Resolving patches.ubuntu.com... 91.189.90.149
Connecting to patches.ubuntu.com 91.189.90.149 :80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 441869 (432K) [text/plain]
Saving to: php5_5.2.6-1ubuntu3.patch'

100%[======================================================================>] 441,869 62.3K/s in 7.3s

2008-06-19 22:25:11 (59.4 KB/s) - php5_5.2.6-1ubuntu3.patch' saved [441869/441869]

$ lsdiff -i '*out' php5_5.2.6-1ubuntu3.patch wc -l
44

I believe people familiar with test suites know what that means...

Update: it means that the test suite files (outputs, expected results, and so on) are on the .diff.gz, thus contaminating it

Oh, and:

$ out=$(mktemp); filterdiff -i '*tests*' php5_5.2.6-1ubuntu3.patch > $out
$ du -h php5_5.2.6-1ubuntu3.patch $out
436K php5_5.2.6-1ubuntu3.patch
372K /tmp/tmp.eAxRv20298

...what a waste of space

Can't find any documentation about this on the web, and it's a PITA, so here goes. Note: No pictures. I was too scared to take time out to take pictures. Disassembly is not hard at first. Three screws hidden over the battery pack allow removing the cover over the power button and wireless switch, which then frees 4 tabs that lock the top of the keyboard down. My keyboard was also taped down with double-sided tape, and glued down. This is the part I hate, both because I still haven't accepted that the common way to put together consumer electorinics is tape and glue, and because it's never possible to get it apart undamaged. I pried the keyboard up from the top, pushing under to untape/glue, and taking care of the cable. There was also a flap of plastic stuck down over the cable, which had to be peeled away. Many clips on the bottom of the keyboard hold it in place and have to be slipped out. I ended up slightly bending the arrow keys of the keyboard in yanking it out, though I think they'd still be ok-ish. Luckily I was putting in a new one. Removing the keyboard cable was strange. Its socket is not the typical kind you can life to unlock. The socket is one piece and does not move at all. I eventually just pulled the cable out of it. This is where I started to get very nervous. Replacing the cable was a PITA. You have to bend its end at a 90 degree angle, and then just line it up and press it down into the socket. Hard. And rub along it and do whatever you can to get enough force exerted down the thin cable strip to make it slip far enough into the socket to work. I booted up the laptop several times while doing this, to test if I had managed to get it working. Eventually I had. Then I very carefully reseated the keyboard, and replaced the cover and screws. Unfortunatly, the tape and glue was doing something. The new keyboard is very springy in the middle, where that held it down before. So I will have to take it apart again and put on some new double-sided tape or something eventually. Conclusions:

• Fujitsu sucks for not springing for an extra 0.01 cents for a locking cable socket.
• Laptop manufacturers in general suck for using tape and glue. (Or for building weaghty mehemoths that use better fastening systems.)
• When it's been disassembled, a lot of the magic of a laptop seeps out for me and never really comes back. Maybe next time I should not do my own repairs, just to preserve that heady comsumer magic of not knowing what's inside?
• I'm still using Caps Lock for Esc, so why did I even bother?!
• New keyboards feel strange.