I've been working hard on a new project for the last couple of months, which I'm glad to be able to share with you now. For the impatient: today my company Control Yourself, Inc. is launching Identi.ca, a new microblogging service. (There's a press release on the Control Yourself site). Users can post short messages about themselves to Identi.ca, which are then broadcast to friends in their social network using instant messages (IM), RSS feeds, and the Web. The difference between Identi.ca and other services like Twitter, Jaiku, or Pownce is that it is Open Source software. I tried to meet the requirements of the Open Service Definition -- Free Software and Free Data. In the same spirit, Identi.ca supports open standards like OpenID and OpenMicroBlogging, to integrate with other Web sites and services. A little background: in March of this year, I was privileged to take part in a summit at the Free Software Foundation about open network services. The summit brought together a number of people involved in Open Source and Open Content to discuss problems with "software as a service" and how to solve them. More and more, people are depending on Web sites and other network services "in the cloud" to do much of their computing tasks. Examples: using Yahoo! Maps instead of a desktop mapping application, or using Google Docs instead of Microsoft Word. Software-as-a-service can be extremely convenient -- data on remote servers stays up-to-date, software gets updated regularly, and you can use different computers or mobile devices to get at the same data. The sacrifice, however, is user autonomy. If you decide that Google Docs doesn't work the way you want, you can't tinker with the software and fix it. If you want to share a map on your Web site, you need Yahoo!'s permission. If you want to use a new social networking site, you have to re-enter all your personal data and re-invite all your friends. The data and code belong to someone else, and they're hidden behind servers that you, the user, aren't allowed to touch. There aren't a lot of clear answers to this. But our loosely-organized summit group is moving forward to promote and support Free/Open Network Services. These are Web sites (or other services) that use Free/Open Source software (with the source available) and provide Open Content data (except for data that users mark as private). There are a lot of them popping up -- like OpenStreetMap and Wikipedia. Others are close on the software side, although the data side is a little less clear -- like Wordpress.com, Reddit, or LiveJournal. After the summit, I wanted to try to see what I could do personally to further this mission. I think several of the sites I work on now -- like Vinismo and Keiki -- would qualify. But I also wanted to break some new ground and challenge people's assumptions about software services. And the services I was using, and getting invited to, the most were microblogging sites. Clearly that was the area to dig into further. To do it, I started Laconica, the software underlying Identi.ca. It's AGPL'd PHP using MySQL as a backend -- probably the most accessible platform on the planet right now. I used lots of existing libraries to make development easier, and I hired a great designer (Montreal's Marie-Claude Doyon) to give the site a professional look. Almost as important as the Open Source and Open Content is the distributed nature of Identi.ca. I designed an open protocol called OpenMicroBlogging, based on OAuth, to make it easy for users on one Laconica server to subscribe to notices by users on another server. I hope this will make the service more robust, develop a rich and diverse social network, and stimulate innovation. Because in the end, that's the Internet that we all want to be part of. As a society we rejected "walled gardens" like AOL, Compuserve, and MSN in the early 90s in favor of the distributed, open nature of the World Wide Web. And I think that the new walled gardens like Facebook and Google, need to be replaced with open protocols and standards. Will Open Network Services be the basis of "Web 3.0", the next stage of Web culture? I truly think so, but that's still an open question. People have been using Identi.ca for a few weeks now, although I've asked that folks wait until the software was more mature before posting public links. Today I'm lifting that embargo and asking friends and colleagues to join and participate in the conversation. I hope that people who are sympathetic with the ideals of Identi.ca will take the time to blog about it, invite their friends, and start using the site regularly. And of course I'd love to get feedback on bugs, requested features, and ideas for the future. Please also subscribe to me on the site -- see http://identi.ca/evan -- so I can see who's out there. There are still so many standard services on the Web that I'd like to see open sourced, open content, and distributed -- advertising, storage, image and file sharing, comments, portal pages, social news, social bookmarks, search engines. There's a real business potential for taking the lead in these areas, and I hope more people jump in.

New python-debian feature: Dependency Parsing Since my merge commit of this afternoon python-debian has grown dependency parsing support. (But first things first: you know about python-debian, don't you? If you don't, and you always wanted to program with Debian-related files with Python, then shame on you!) Thus far, playing with Packages-like files using dictionary-like objects was already as simple as (quoting from /usr/share/doc/python-debian/examples/deb822/grep-maintainer):

for pkg in deb822.Packages.iter_paragraphs(file('/var/lib/dpkg/status')):
    if pkg.has_key('Maintainer') and maint_RE.search(pkg['maintainer']):
        print pkg['package']

However, it was a bit unfortunate that Depends-like fields were only returned as strings. Now each package (as iterated upon in the above snippet) is equipped with a .relations property returning a dictionary of inter-package relationship fields. Looking up keys like "depends" you will get back a conjunctive normal form (CNF) representation of dependencies, together with parsed constraints on version or architectures, if any. A simple example is shipped as /usr/share/doc/python-debian/examples/deb822/depgraph, which outputs a labeled Graphviz script of all inter-package dependencies extracted from a Packages file. Here is its most relevant snippet:

name = pkg['package']
rels = pkg.relations
for deps in rels['depends']:
    if len(deps) == 1:
        emit_arc(name, deps[0]['name'])
    else:   # output an OR node
        or_node = get_id()
        emit_arc(name, or_node)
        emit_node(or_node, 'OR')
        for dep in deps:
            emit_arc(or_node, dep['name'].lower())

The appropriate sub-classes of Deb822 have been customized with the knowledge of which inter-package relationship fields are supported in their stanzas (e.g. Build-Depends are supported by Sources, but not by Packages; the other way around for Recommends). Testing from the git repo is more than welcome.

Problems with hmrc.gov.uk - the new non-PDF format of the tax return is a complete mess and very misleading. I wanted a PDF so that I could compare the generated return with an actual return from previous years, so I sent this feedback:

"The Help and FAQ claim that a colour PDF is available and I'm
assuming that the PDF will be a closer match to the paper based form
which I need to be able to compare this return against last year. The
option to obtain a PDF has disappeared. (I am able to process PDF
without needing any software from Adobe as that is the main point of a
Portable Document Format so if this is because the site is checking for
Adobe software on my computer, it should stop doing so and simply
provide a link to the PDF file.) "

The response shows that nobody actually reads the comment:

"Thank you for your online enquiry.

To print your return in PDF format you are required to have Adobe Acrobat Reader 4.0 or above. If you do not have this on your computer please logon http://www.adobe.com/products/acrobat/readstep.html to download a copy. Please be aware that this download can take a long time."

i.e. Her Majesty's Government requires that you allow them to interrogate your computer to find a specific executable file or registry ID and if this is not found, certain services will be refused to you without further comment.

Instead of just generating the PDF on the server and providing a link to it.

The worst part is, the "secure gateway" used by HMRC doesn't even allow replies!

The only option is delete!!

More bias from government departments against free software. I refuse to use anything under an Adobe licence when there are perfectly capable PDF clients available under a free licence. (Let alone the discriminatory download licence). (I'm a British national but I still object to a licence that would prevent someone with dual British-Iraqi nationality, for example, obtaining a service from HMRC.) Besides which, I somehow doubt that the site would be able to correctly identify the Unix version of Adobe Acrobat Reader.

I receive notification of all my taxable income via PDF so why is the taxman different??

This is the third installment of my article about the Serial Console Server for the Poor. First installment here, Second installment here. The first part of the article having covered the hardware and the udev part creating the device nodes, and the second part explaining how to solve the software part using ser2net, this part explains why ser2net was ditched in favor of cereal and how the console server operates with cereal now. Cereal (which I was pointed to on IRC as a reaction to article II) is a nifty combination of runit’s runsv and screen. One of the less-known features of screen is that screen can also connect to a serial port instead of a “real” console. cereal makes use of this feature, putting up with the fact that screen can only set a baud rate. I have not seen a serial application with other parameters than 8n1 (8 data bits, no parity, 1 stop bit) in a decade, so I cannot comment whether this is a real disadvantage. I guess that it might be possible to set the serial port to the desired set of parameters before starting up screen, but I didn’t actually try it since I do not need it. Cereal uses a set of configuration parameters and processes called a “session” to associate a serial port with a baud rate and a set of user/group settings to control who is able to start, stop, attach or detach a session. It doesn’t have a “real” configuration file. Instead, it has an admin program, cereal_admin, which takes the parameters associated with a session and writes them to an (intentionally? undocumented?) set of config files in /var/lib/cereal. Only a single user is able to attach to a session read/write, while an entire group can attach to a session read-only and can thus access the logs. A session is automatically started when the sysem boots and can be stopped and started using cereal_admin as well. Once a session is running, a specially configured screen process runs attached to the serial port, and users simply attach and detach to and from it as one would to a normally running screen. That way, one can scroll back and examine what was shown on the serial port while nobody was attached. This is quite handy when the device connected to the port prints out warnings, errors and status information to its serial port. The screen configuration that is used is rather sophisticated. So, a permanent log file is written, which is also used for read-only access to the serial session. Also, a very informative status line is configured. Unfortunately, one of screen’s biggest disadvantages applies here as well: User X cannot attach to a session started by User Y unless Y has access privileges for X’s pty (or vice versa, I never can remember), which can be a security risk. Additionally, it is not possible to su or sudo to X and then to attach to the session - one needs to have the original pty opened by the correct user, which usually means that one needs to do a “real” log in with the correct account (via /bin/login or sshing in to the box). If one has the privileges necessary, one can use the cereal executeable to attach to the screen (which gives read-write access) or to tail -F the log file (giving timestamped read-only access). That way, one never knows that one is actually working with an appropriately configured screen that is kept running by runit. The restrictions coming with cereal suggest a setup similiar to what I did with ser2net previously: Each session has its own user account, which forces an attach to the appropriately named cereal shell on login:

my-device-name:x:1801:1801:my-device-name,,,:/home/my-device-name:/usr/local/sbin/cerealshell

$ cat /usr/local/sbin/cerealshell
#!/bin/bash
set -eu
cereal attach $(id --user --name)
$

You can see that I decided to name the sessions after the actual device name, allowing the unterlying ttyUSB device name to change without change in the user interface. The actual users and sessions were created with a small shell script:

#!/bin/bash
createconsole()  
  NAME=“$1”
  PORT=“$2”
  BAUD=“$3”
  MUID=“$ PORT/USBserial/180 ”
  echo sudo addgroup --force-badname --gid $MUID $NAME
  echo sudo adduser --force-badname --gecos “$NAME,,,” --ingroup $NAME --uid $MUID --disabled-password
$NAME
  echo sudo adduser $NAME dialout
  echo sudo mkdir -p /home/$NAME/.ssh
  echo sudo chmod 755 /home/$NAME/.ssh
  echo sudo touch /home/$NAME/.ssh/authorized_keys
  echo sudo chmod 644 /home/$NAME/.ssh/authorized_keys
  echo sudo chown $NAME:$NAME /home/$NAME/.ssh /home/$NAME/.ssh/authorized_keys
  echo sudo cereal-admin create $1 /dev/$2 $3 $1 $1
  echo sleep 2
    echo sudo cereal-admin start $1
  echo echo -----
 
createconsole my-device-1 USBserial1 9600
createconsole my-device-2 USBserial3 115200

Note that the script also uses a hardcoded UID range which might need adaption to your local needs (and breaks for more than ten serial ports), and that it also takes care of creating an empty authorized_keys file with correct permissions. That way one can simply ssh to my-devlce-1@consoleserver and find oneself immediately connected to the serial console, seeing the last things that happened on the serial port, and immediately being able to work with the device. That’s fine, comfortable and useable. I actually like that better than what I built with ser2net previously.

I read 7 Ways Religion is Detrimental to Science, and thought it would be an interesting read. It was, but I don't think it really made sense. Let's look at the 7 ways they highlighted:

1. Faith and the Scientific Method are Opposites

The article states:

Faith is a belief in an idea regardless of the evidence for or against it.

Actually, that's not true at all, at least for traditional Christianity. (Note that traditional Christianity is not the same as fundamentalism). Marcus J. Borg describes faith in four ways:

Fiducia -- trusting in God. Borg says, "Faith . . . [means] we trust in God as the one upon whom we rely, as our support and foundation and ground, as our safe place."

Fidelitas -- loyalty, or "the comment of the self at its deepest level, the commitment of the 'heart'. Faith as fidelitas does not mean faithfulness to statements about God. . . Fidelitas refers to a radical centering in God."

Visio -- a way of seeing -- "a way of seeing the whole, a way of seeing 'what is'. . . the ability to love and to be present to the moment. It generates a 'willingness to spend and be spent' for the sake of a vision that goes beyond ourselves."

Assensus -- perhaps the closest to what the author meant, is "faith as belief." Borg goes on to add:

The notion that Christian faith is primarily . . . about belief, about a "head" matter, is recent. . . For many, Christian faith began to mean believing questionable things to be true. . . this is the most widespread contemporary understanding of "belief".

This is very different from what faith as assensus meant. . . A deep but humble (and therefore imprecise) understanding of Christian faith as assensus, as involving affirmation of the centrality of God as known in the Bible and Jesus, is very close to faith as vision. It is a way of seeing reality.

As a Christian, I do not find the scientific method to be a problem. I find it to be enlightening in all sorts of matters, including even the history of religion in some instances. Christian faith is not about believing in certain ideas (such as the world being created in 6 literal 24-hour days), though there are those that distort it to be so. Rather, Christian faith is about living your life a certain way, about the meaning of life, about our duties to make the world a better place.

2. People Vote Base on Religious Ideas

The author says that "Stem Cells weren t the first time a body of research didn t get proper funding because some religious wack-jobs." Well, I agree that that was a problem. Non-religious people can vote in odd ways too. I argue that the rejection of stem-cell research goes against Christian teaching; after all, we are to help the least among us, and do not people with Alzheimer's qualify?

I think the author's point should have been "exremists have odd views." Extremists can be atheist, Christian, Muslim, Buddhist, or other religions. Religion does not have a monopoly on them.

3. Religion Removes the Need For Science

The author says:

When people are content to believe in something that explains why they are here, even if it is wrong, they become less interested in other ideas. Religion often leads people to believe that they have all the answers. Science is self-correcting in that nobody assumes they are absolutely correct.

That is incorrect on several levels. First, science cannot explain why we are here. It can explain some of the mechanics. As an example, let us take for granted that modern scientific thought on the origin of humanity is correct: that there was a Big Bang, that single-celled creatures evolved on earth, and that human life eventually evolved through a complex process of evolution and natural selection. Fine; this does not contradict religion in any way. Science and religion answer different questions. Science answers "how". Religion answers "why". Science cannot answer the question "Why was there a big bang?" or "Did the evolution of humanity serve a larger purpose?"

Moreover, religion should not assume that they have all the answers, either. The canonical Christian Bible was mostly fixed within a few centuries AD. Life in 200AD was a lot different from life today. Part of the reason there are many different groups of Christians is the complexity of applying the stories in the Bible to modern life. The ideas in any given denomination evolve over time, too. I think it would be the height of hubris for anyone, religious or not, to claim that he or she had all the answers. Again, I know that some religious people act that way, but then so do some atheists or agnostics.

4. People Lean on Religion, When They Could Benefit From Science

The author of "7 Ways" cites the quite rare case where a child dies of a curable disease while his/her parents pray, refusing medical care. This is an extreme position that is not shared by the vast majority of religious people. Most religious people are perfectly content to use the latest medical care.

Meditation or prayer does not replace medical care; it supplements it. Scientific studies have even demonstrated its effectiveness.

5. The Church Takes Up Natural Resources

The original author states that "The land that churches take up around the world could be used to build schools, homes, recreational buildings and commercial operations." This is perhaps the most frivolous of arguments. Putting aside the fact that many churches operate schools, churches are often one of the few ways that modern city dwellers have to form a sense of community. They are not just places to engage in religion. They are places to meditate, to get away from it all, to meet your neighbors, to vote.

6. The Church Takes Up Monetary Resources

The original author says "if people donated to scientific advancement like they did to the church, imagine where we would be today." It's not a pleasant picture. Religion, and institutions supported primarily because of the teaching of religion, are the people that feed the hungry after natural disasters, that operate food pantries (our church operates the only one in our community, and it's open to anyone without any questions or talk about religion), that operate schools in disadvantaged areas, that have spread the whole idea of fair trade for third-world artisans, etc.

It is true that acts of evil have been committed in the name of the church, too. It is also unquestionably true that some church groups spend money more carefully than others. As with any donation, people should be careful where they give. Government-operated research studies are not necessarily a good use of money, either.

7. Religion is A Strong Meme

The author of the original story seems to be responding to a particular brand of Christianity: what Borg calls "literal-factual" religion. There are quite a few people that take that stand. They are sometimes inaccurately referred to as "evangelicals" or the "religious right"; while there is some overlap between the groups, they are not one and the same. The people with the literal-factual view are not representative of the whole.

More interestingly, Borg points out that the literal-factual view was actually a response to the development of the scientific method during the Enlightenment. As the modern idea of truth moved to literal, factual, provable truth, some Christians grew defensive about their faith, and started to look for "scientific" ways to prove that the world was created in 6 days, etc. in an attempt to show the world that Christianity fit their new notion of truth.

That makes a compelling argument that the scientific method is the stronger meme in today's Western world -- so strong that the author of the rant against religion has apparently forgotten the more prevalant -- both throughout history and today -- view of Christianity that is "more than" Science, not at odds with science.

The Age has an interesting article about proposed DNS changes [1]. Apparently ICANN is going to sell top level DNS names and a prediction has been made that they will cost more than $100,000 each. A suggestion for a potential use of this would be to have cities as top level names (a .paris TLD was given as an example). The problem with this is that they are not unique. Countries that were colonised in recent times (such as the US and Australia) have many names copied from Europe. It will be interesting to see how they plan to determine which of the cities registers names, for the .paris example I’m sure that the council of Paris Illinois [2] would love to register it. Does the oldest city win an international trademark dispute over a TLD? The current situation is that French law unambiguously determines who gets to register paris.fr and someone who sees the URL will have no confusion as to what it means (providing that they know that fr is the ISO country code for France). As well as city names there are region names which are used for products. Australian vineyards produce a lot of sparkling wine that they like to call Champagne and a lot of fortified wine that they like to call Port. There are ongoing battles about how these names can be used and it seems likely to me that the Australian wine industry will change to other terms. But in the mean-time it would be interesting if .champagne and .port were registered by Australian companies. The fuss that would surely cause would probably give enough free publicity to the Australian wine industry to justify an investment of $200,000 on TLDs. The concern that is cited by business people (including the client who forwarded me the URL and requested my comments) is that of the expense of protecting a brand. Currently if you have a company named “Example” you can register example.com, example.net, and example.org if you are feeling enthusiastic. Then if you have a significant presence in any country you could register your name in the DNS hierarchy for that country (large companies try to register their name in every country - for a multinational registering ~200 domains is not really difficult or expensive). But if anyone can create a new TLD (and therefore if new ones are liable to be created at any time) it becomes much more difficult. For example if a new TLD was created every day then a multi-national corporation would need to assign an employee to work full-time on investigating the new TLDs and deciding which ones to use. A small company that has an international presence (IE an Internet company) would just lose a significant amount of control over their name. I don’t believe that this is as much of a concern as some people (such as my client) do. Currently I could register a phone line with a listed name indicating that it belongs to the Melbourne branch of a multi-national corporation. I don’t expect that Telstra would stop me, but the benefit from doing this would be minimal (probably someone who attempted fraud using such means would not gain much and would get shut down quickly). I don’t think that a DNS name registered under a .melbourne TLD would cause much more harm than a phone number listed in the Melbourne phone book. Incidentally for readers from the US, I’m thinking of Melbourne in Australia not a city of the same name in the US - yet another example of a name conflict. Now I believe that it would be better if small companies didn’t use .com domains. The use of country specific names relevant to where they work are more appropriate and technically easier to implement. I don’t regret registering coker.com.au instead of some name in another country or in the .com hierarchy. Things would probably be working better right now if a .com domain name had always cost $100,000 and there were only a few dozen companies that had registered them. But we have to go with the flow sometimes, so I have registered RussellCoker.com. Now when considering the merit of an idea we should consider who benefits and who (if anyone) loses. Ideally we would choose options that provide benefits for many people and losses for few (or none). In this case it seems that the suggested changes would be a loss for corporations that want to protect their brand, a loss for end-users who just want to find something without confusion, and provide more benefits for domain-squatters than anyone else. Maybe I should register icann.port and icann.champagne if those TLDs are registered in Australia and impersonate ICANN. ;)

• [1] http://www.theage.com.au/news/web/rule-changes-to-spark-domain-name-goldrush/2008/06/27/1214472719636.html


• [2] http://parisillinois.org/

Share This

Today I received a Dell PowerEDGE T105 for use by a client. My client had some servers for development and testing hosted in a server room at significant expense. They also needed an offsite backup of critical data. So I suggested that they buy a cheap server-class machine, put it on a fast ADSL connection at their home, and use Xen DomU’s on that for development, testing, and backup. My client liked the concept but didn’t like the idea of having a server in his home. So I’m going to run the server from my home. I selected a Dell PowerEDGE tower system because it’s the cheapest server-class machine that can be purchased new. I have a slight preference for HP hardware but HP gear is probably more expensive and they are not a customer focussed company (they couldn’t even give me a price). So exactly a week after placing my order I received my shiny new Dell system, and it didn’t work. I booted a CentOS CD and ran “memtest” and the machine performed a hard reset. When it booted again it informed me that the event log had a message, and the message was “Uncorrectable ECC Error” with extra data of “DIMM 2,2“. While it sucks quite badly to receive a new machine that doesn’t work, that’s about the best result you can hope for when you have a serious error on the motherboard or the RAM. A machine without ECC memory would probably just randomly crash every so often and maybe lose data (see my previous post on the relative merits of ECC RAM and RAID [1]). So I phoned up Dell (it’s a pity that their “Packing Slip” was a low quality photocopy which didn’t allow me to read their phone number and that the shipping box also didn’t include the number so I had to look them up on the web) to get technical support. Once we had established that by removing the DIMMs and reinserting them I had proved that there was a hardware fault they agreed to send out a technician with a replacement motherboard and RAM. I’m now glad that I bought the RAM from Dell. Dell’s business model seems to revolve around low base prices for hardware and then extremely high prices for extras, for example Dell sells 1TB SATA disks for $818.40 while MSY [1] has them for $215 or $233 depending on brand. When I get the machine working I will buy two 1TB disks from MSY (or another company with similar prices). Not only does that save some money but it also means that I can get different brands of disk. I believe that having different brands of hard disk in a RAID-1 array will decrease the probability of having them both fail at the same time. One interesting thing about the PowerEdge T105 is that Dell will only sell two disks for it, but it has four SATA connectors on the motherboard, one is used for a SATA DVD player so it would be easy to support three disks. Four disks could be installed if a PCIe SATA controller was used (one in the space for a FDD and another in the space for a second CD/DVD drive), and if you were prepared to go without a CD/DVD drive then five internal disks could probably work. But without any special hardware the space for a second CD/DVD drive is just begging to be used for a third hard disk, most servers only use the primary CD/DVD drive for installing the OS and I expect that the demand for two CD/DVD drives in a server is extremely low. Personally I would prefer it if servers shipped with external USB DVD drives for installing the OS. Then when I install a server room I could leave one or two drives there in case a system recovery is needed and use the rest for desktop machines. One thing that they seem to have messed up is the lack of a filter for the air intake fan at the front of the case. The Opteron CPU has a fan that’s about 11cm wide which sucks in air from the front of the machine, in front of that fan there is a 4cm gap which would nicely fit a little sponge filter. Either they messed up the design or somehow my air filter got lost in transit. Incidentally if you want to buy from Dell in Australia then you need to configure your OS to not use ECN (Explicit Congestion Notification [2] as the Dell web servers used for sales rejects all connections from hosts with ECN enabled. It’s interesting that the web servers used for providing promotional information work fine with ECN and it’s only if you want to buy that it bites you. But in spite of these issues, I am still happy with Dell overall. Their machine was DOA, that happens sometimes and the next day service is good (NB I didn’t pay extra for better service). I expect that they will fix it tomorrow and I’ll buy more of their gear in future. Update: I forgot to mention that Dell shipped the machine with two power cables. While two power cables is a good thing for the more expensive servers that have redundant PSUs, for a machine with only one PSU it’s a bit of a waste. For some time I’ve been collecting computer power cables faster than I’ve been using them (due to machines dying and due to clients who want machines but already have spare power cables). So I’ve started giving them away at meetings of my local LUG. At the last meeting I gave away a bag of power cables and I plan to keep taking cables to the meetings until people stop accepting them.

• [0] http://etbe.coker.com.au/2008/06/13/ecc-ram-vs-raid/


• [1] http://www.msy.com.au/


• [2] http://www.ietf.org/rfc/rfc3168.txt

Share This

Yes Stefano, I too believe that the SSL certificate thingy in Iceweasel 3 is just a bad joke. I mean WHAT THE FUCK WERE THEY THINKING ? At least in the RCs it improved, and the logo isn't the same as 404, it took me 10 refresh the first time I saw it to understand that it was not a 404 but an untrusted self-signed certificate. And the whole "exception" thing is absurd. NO I DON'T WANT TO ADD AN EXCEPTION you morons. Here would be a good UI, that I never saw implemented in any browser but would make sense:

• I want to browse the damn site, that you remember for me the current certificate, and browse without pretending it's safe (this way users do not believe to be protected when they are not).
• Next time I go to that website again, if the untrusted certificate changed, then and only then I want to be bugged about the fact that this website changed its certificate.
• I don't want any popup, anything, no "yellow" background in the URL (as the certificate is not trusted) no nothing, just browse it as if it was plain HTTP wrt the UI.

It's enough for 99.9% of the users. Then, some of us really care about importing some untrusted certificates (for our own webmails e.g.). Then well, a small button that allows advanced users to check, import, and mark an untrusted certificate as trusted is just what one needs. One can even hide that in some menu, it would be fine. I for example, have only checked the SSL certificate from my webmail and something like 3 or 4 websites. I do NOT want to be bugged for the other. But no, instead we have this completely delirious interface that no-one can sanely call User Interface. They're on crack. Anyways, here is a way to make your pain easier, either in about:config or in your $ HOME /.mozilla/firefox/$ profile-with-name-on-crack-id /user.prefs set the following values this way^[1]:

 user_pref("browser.ssl_override_behavior", 2);
 user_pref("browser.xul.error_pages.expert_bad_cert", true);

It makes validating a certificate two clicks away (one of the settings shows the 'add exception' buttons instead of showing a link to make them visible, the other settings makes firefox download the certificate for you so that you don't have to ASK HIM to do so^[2]. And you know the worst thing ? I'm sure the guy(s) who wrote this fucking dialog is(are) very proud of it.

Gunnar, have a look at this: . So your example using the Penta website was just a bad one, and I personally agree with the "INVALID" resolution of your bug report. In fact, showing a partially encrypted connection as an insecure one is an overall advantage for (l)user securities, as with the browser interfaces we are used to use, we don't know which parts are actually sent encrypted and which are not. As you can see in the example above a fully encrypted connection is properly reported as such by Firefox 3.0. FWIW: I'm overall satisfied with Firefox 3.0, especially about the new URL bar Wouther is disliking. Yes it is a bit slower than before, but even on my 4-year old laptop it is perfectly usable. The HTML title shown there? Well, it is rather handy when I start typing bugs.debian.org and then I've to discriminate among tons of almost identical bug report URLs ... Is the about:config hand-holding dialog worst than the pop-up the first time you "have provided unencrypted information" in a random form? It is not, we have probably just forgot about the first time we unchecked the corresponding checkbox. All in all, the only "feature" of the new Firefox I found incredibly stupid is the SSL certificate exception mechanism. Installing the untrusted certificate is what? ... 5 click-away from the initial page? That is just dumb. (And as it is well-known that habits form anyhow, getting the habits harder to form is just a way to piss-off users.)

I have just watched an interesting lecture by Steven Levitt about car safety for children in the 2-6 age range [1]. The evidence he presents shows that the benefits for children in that age range are at best insignificant and that in some corner cases (EG rear impacts) the child seat may give a worse result than an adult seat belt! He advocates a 5-point harness [2] for children in the 2-6 age range that is based on a standard adult seat and seems to be advocating a child “booster seat” integrated into the adult seat (which approximates the booster seats offered by some recent cars such as the VW Passat). He has a picture of a child in a child-sized 5-point harness to illustrate his point. But one thing that should be considered is the benefit of a 5 point harness for adults. Race car drivers use 5 point harnesses, I wonder how the probability of a race car dying during the course of their employment compares with the probability of an average adult dying while doing regular driving. I also wonder how a 5 point harness compares to a three point harness with a pre-tensioner, it seems quite possible that a 5 point harness would be cheaper and safer than the 3 point harness with pre-tensioner that is found in all the most expensive cars manufactured in the last few years. He believes (based on tests with crash-test dummies) that part of the problem is that the child seat will move in an accident (it’s attached to a soft seat). It seems that one potential solution to this is to have child seats that firmly attach to some solid part of a vehicle. I had previously suggested that child seats which replace existing seats as an option from the manufacturer would be a good idea [3]. But there is a good option for making better child seats for existing vehicles. It is becoming common in the “people mover” market segment to design vehicles with removable seats. For example the Kia Carnival has three seats in the middle row which are removable and which attach to four steel bars in the floor. It should not be difficult to design a child seat which attaches to those bars and could therefore be plugged in to a Carnival in a matter of minutes. The Carnival is designed to have the mid row seats installed or removed easily and safely by someone who is untrained, while for comparison it is recommended that a regular child seat should only be installed by a trained professional (IE your regular mechanic can’t do it).

• [1] http://www.ted.com/talks/view/id/30


• [2] http://en.wikipedia.org/wiki/Seat_belt#Five-point_harnesses


• [3] http://etbe.coker.com.au/2007/12/22/a-better-design-for-child-seats/

Share This

When it comes to research and writing, anthropologists are notorious for being slow as molasses. It takes, on average, 9 years to finish grad school and then another 3-6 years before you write and publish your first book. Such is the sometimes frustrating pace of academia. But sometimes it is worth the wait as this new book, Two Bits, the Cultural Significance of Free Software by the anthropologist of science and technology Chris Kelty, makes clear. I am pretty intimate with the details of the book as ck was one of my dissertation committee members. I have read earlier versions and was fortunate enough to teach the final version to a small group of graduate students last spring. While there have been previous books on the subject of open source or peer-to-peer production more generally, this is the first book to delve deep, analytically and historically, into the cultural significance of free software. And the best part is that ck managed (I am not sure how but he did) to convince Duke University Press to release the whole book under a CC license. So download it to your heart’s delight and if you find it useful, do order a treeware copy (as you can’t beat that for easy and comfortable reading). For the developers out there, I would suggest starting with the introduction, which provides an initial overview of some of the main themes he develops and then I would dig into part two where the historical chapters reside and then loop back to the theoretical chapters in part one or continue to part three. Trust me, these will fascinate and surprise even those geeks with a relatively deep historical sense of this world. In fact, my very favorite academic piece on free software exists in these pages: and is the chapter on the history of the actual conflict between James Gosling and Richard Stallman that led to the creation of the GNU General Public License. Here is a small excerpt to whet your appetite:

In brief the controversy was this: in 1983 James Gosling decided to sell his version of EMACS a version written in C for UNIX called GOSMACS to a commercial software vendor called Unipress. GOSMACS, the second most famous implementation of EMACS (after Stallman s itself ), was written when Gosling was a graduate student at Carnegie Mellon University. For years, Gosling had distributed GOSMACS by himself and had run a mailing list on Usenet, on which he answered queries and discussed extensions. Gosling had explicitly asked people not to redistribute the program, but to come back to him (or send interested parties to him directly) for new versions, making GOSMACS more of a benevolent dictatorship than a commune. Gosling maintained his authority, but graciously accepted revisions and bug-fixes and extensions from users, incorporating them into new releases. Stallman s system, by contrast, allowed users to distribute their extensions themselves, as well as have them included in the official EMACS. By 1983, Gosling had decided he was unable to effectively maintain and support GOSMACS a task he considered the proper role of a corporation.
[ . . .]
Indeed, the general irony of this complicated situation was certainly not as evident as it might have been given the emotional tone of the debates: Stallman was using code from Gosling based on permission Gosling had given to Labalme, but Labalme had written code for Gosling which Gosling had commercialized without telling Labalme conceivably, but not likely, the same code. Furthermore, all of them were creating software that had been originally conceived in large part by Stallman (but based on ideas and work on TECO, an editor written twenty years before EMACS), who was now busy rewriting the very software Gosling had rewritten for UNIX. The once proud hacker ethic that Labalme mentions would thus amount not so much to an explicit belief in sharing so much as a fast-and-loose practice of making contributions and fixes without documenting them, giving oral permission to use and reuse, and losing records that may or may not have existed hardly a noble enterprise.
[. . .]

The story is a lively one, punctuated with some great e-mail “moments,” and gives you a play-by-play sense of how it is that free software become so entangled with the law and how the unwritten norms that guided sharing were violated to then become transformed, explicit, and written. The other chapter that will also fascinate those who want a deeper sense of this history is on UNIX and its role in creating the technical and collaborative conditions necessary to secure the explosion of free software years later. The story of the norms of sharing source code is, not by accident, writes Kelty, also the history of the UNIX operating system. This connection has been made by others but not shown in any historical detail as it is done in this book. For those interested in more anthropological issues, the first few chapters is where this type of more conceptual theorizing is at. Chris Kelty introduces the master trope of his book: the recursive public which is an idea that feeds into a longer academic debate, inaugurated by Jurgen Habermas, on the role of publicity, dialogue, and circulation of ideas and texts in cementing a democratic sphere. Many others, such as Charles Taylor, Michael Warner, and Nancy Fraser have since refigured and rehashed the meaning publics and public debate but there have been few ethnographies on the topic. It is here where Chris Kelty intervenes most forcefully to conceptualize publics in explicitly ethnographic terms and in an arena technology and the Internet which is often left undertheorized by cultural theorists. There is a lot more in the book (the third section examines some of the various political modulations of free software, such as Connexions and Creative Commons) but instead of saying any more, I will just leave you with his definition of a recursive public and again urge you to check out the book for yourself!

A recursive public is a public that is vitally concerned with the material and practical maintenance and modification of the technical, legal, practical, and conceptual means of its own existence as a public; it is a collective independent of other forms of constituted power and is capable of speaking to existing forms of power through the production of actually existing alternatives.

I've spent a few hours recently looking at building RPM packages of GNU/Linux kernels, which has been a frustrating process. There are many many online guides which give the impression that this is actually a pretty complex process. For example How To Compile A Kernel - The CentOS Way guide. (Did I mention how bad most of the howtoforge guides are recently?) So, after fiddling around for an afternoon and getting lost I decided to abandon the process. Here is a tested process for building a binary RPM kernel package:

cd linux-2.6.24.7/
make rpm

Yes this works just fine upon a Centos 5.x machine - I'm used to using make-kpkg to make a Debian kernel package, but it seems that if you just visit kernel.org and download the latest version you can build a RPM without any extra effort thanks to native support. Cool. Now I need to work out how to create, host, and update a YUM repository. That looks fiddly and annoying too. XML. Eww. Any guides are most welcome - ultimately I need to package and host a "recent" kernel for Centos 4.x, Centos 5.x and Fedora Core 6-9 - each for i386 + amd64. ObQuote: Spiderman

One of the traditional problems with power management has been balancing power savings against reduced functionality. The two traditional approaches to this have been to have sane defaults or to let the user tweak them, with a hybrid approach being to have sane defaults and an "advanced" box that lets users set their own parameters.

Both of these options suck.

The problem with sane defaults is that power management is an area where the "sane default" genuinely does differ between users. Usage patterns will determine the appropriate spindown period for a hard drive. AHCI's ALPM will disable hotswap detection, which is fine for home users but a pain if people want to rip out a failed RAID element. ALPM also introduces a small amount of latency (on the order of a seek in the most aggressive mode), which is entirely irrelevant for almost everyone but is a pain if you're trying to offer i/o latency guarantees. PCI-E link power management adds small quantities of latency. Letting the CPU run to full speed whenever it wants to is good for power management but potentially bad for thermal management. Screensaver timeouts vary depending on whether you're watching a movie or not.

The problem with leaving everything tweakable is that you're asking users to make choices about things but not giving them the information they need to make those choices. Whether you get a power saving from hard drive spindown depends on whether the drive is idle for long enough to save the power you'll spend spinning it back up. Get it wrong and you'll be putting your drive under extra load, reducing performance and consuming more power than you were to begin with. If you limit the maximum speed of your CPU to keep it cool, you'll be wasting power and performance even when it's well below the thermal limits. Disabling ALPM because you might want to change a RAID device at some stage means you're consuming power despite generally not needing to use the additional functionality.

But while these options both suck in different ways, they share one fundamental failure. They both imply static configuration. The correct power management policy depends on the way that a machine is being used at a specific instant in time, and no matter how many power management profiles you add there's no way for a user to switch between them fast enough to obtain a decent power/performance ratio.

Adaptive power management adopts a different strategy. Rather than define a static policy (or several static policies), the hardware is monitored to determine the best power management settings. An example of this is the ondemand CPUfreq governor. Rather than attempt to provide a static configuration, the processor frequency is automatically changed to meet demand while also keeping the overall power consumption as low as possible. This patch adds a thermal component to its policy, preventing the machine from overheating. Result? No requirement to provide tunables to userspace.

Something similar is achievable with hard-drive spindown. By simply monitoring how often reads or writes actually hit disk, it's possible to automatically adjust the spindown period to obtain decent power savings while reducing the number of spinups. This approach is imperfect - in an ideal world we'd know to spin the drive up before it's needed, but that simply isn't possible. However, the user is almost certainly unable to make a better guess. It's not realistic for a user to be able to predict whether or not a given page is still in cache or has been evicted. Nothing is lost by providing a dynamic policy, and another unnecessary tunable has vanished.

The difficult part of an adaptive policy comes when roles are switched in ways that don't correspond to changse in hardware status. One of the most common cases is that of screensaver settings when the user is watching a video. It's not practical for the kernel or a system daemon to predict this case, so we have two choices:

• Get the user to switch to a "media" power policy and then remember to switch back afterwards
• Have the media player deactivate the screensaver at startup and reactivate it when the video isn't playing

.One of these choices involves the user having to explicitly state something that's obvious - playing a video automatically implies the desire to adopt a different power management policy, and so asking the user to manually adjust that is ridiculous.

The loss of hotswap notifications when ALPM is enabled? Simply reenable it if SMART starts complaining or the RAID set becomes degraded or a filesystem is unmounted or any of the other things that will have to be done before a drive can be safely removed. Latency an issue? Let userspace tell you what its latency requirements are. When we get down to it, the only settings that we can't infer are things like idle-time-to-suspend, screensaver settings and default brightnesses. Which, curiously, are about the only things that Mac OS exposes[1].

The problem we face is that we don't currently have interfaces to let applications tell us what their requirements are. This is theoretically straightforward, but getting it right is important. Beyond latency and screensaver issues, what constraints do applications want to impose? How can we expose an interface to those constraints? This is the sort of thing we'll be looking at at the Linux Plumbers Conference, so if you have ideas about what they'd need to look like then sign up. Or, even better, submit a paper and be part of the process that gets Linux's power consumption as close to the theoretical minimum as possible.

[1] My Vista system gives me 25 options which can be set for each of AC and battery.

If you want a reliable network then you need to determine an appropriate level of redundancy. When servers were small and there was no well accepted virtual machine technology there were always many points at which redundancy could be employed. A common example is a large mail server. You might have MX servers to receive mail from the Internet, front-end servers to send mail to the Internet, database or LDAP servers (of which there is one server for accepting writes and redundant slave servers for allowing clients to read data), and some back-end storage. The back-end storage is generally going to lack redundancy to some degree (all the common options involve mail being stored in one location). So the redundancy would start with the routers which direct traffic to redundant servers (typically a pair of routers in a failover configuration - I would use OpenBSD boxes running CARP if I was given a choice in how to implement this [1], in the past I’ve used Cisco devices). The next obvious place for redundancy is for the MX servers (it seems that most ISPs have machines with names such as mx01.example.net to receive mail from the Internet). The way that MX records are used in the DNS means that there is no need for a router to direct traffic to a pair of servers, and even a pair of redundant routers is another point of failure so it’s best to avoid them where possible. A smaller ISP might have two MX machines that are used for both sending outbound mail from their users (which needs to go through a load-balancing router) as well as inbound mail. A larger ISP will have two or more machines dedicated to receiving mail and two or more machines dedicated to sending mail (when you scan for viruses on both sent and received mail it can take a lot of compute power). Now the database or LDAP servers used for storing user account data is another possible place for redundancy. While some database and LDAP servers support multi-master operation a more common configuration is to have a single master and multiple slaves which are read-only. This means that you want to have more slaves than are really required so that you can lose one without impacting the service. There are several ways of losing a server. The most obvious is a hardware failure. While server class machines will have redundant PSUs, RAID, ECC RAM, and a general high quality of hardware design and manufacture, they still have hardware problems from time to time. Then there are a variety of software related ways of losing a server, most of which stem from operator error and bugs in software. Of course the problem with the operator errors and software bugs is that they can easily take out all redundant machines. If an operator mistakenly decides that a certain command needs to be run on all machines they will often run it on all machines before realising that it causes things to go horribly wrong. A software bug will usually be triggered by the same thing on all machines (EG I’ve had bad data written to a master LDAP server cause all slaves to crash and had a mail loop between two big ISPs take out all front-end mail servers). Now if you have a mail server running on a virtual platform such that the MX servers, the mail store, and the database servers all run on the same hardware then redundancy is very unlikely to alleviate hardware problems. It’s difficult to imagine a situation where a hardware failure takes out one DomU while leaving others running. It seems to me that if you are running on a single virtual server there is no benefit in having redundancy. However there is benefit in having an infrastructure which supports redundancy. For example if you are going to install new software on one of the servers there is a possibility that the software will fail. Doing upgrades and then having to roll them back is one of the least pleasant parts of sys-admin work, not only is it difficult but it’s also unreliable (new software writes different data to shared files and you have to hope that the old version can cope with them). To implement this you need to have a Dom0 that can direct traffic to multiple redundant servers for services which only have a single server. Then when you need to upgrade (be it the application or the OS) you can configure a server on the designated secondary address, get it running, and then disable traffic to the primary server. If there are any problems you can direct traffic back to the primary server (which can be done much more quickly than downgrading software). Also if configured correctly you could have the secondary server be accessible from certain IP addresses only. So you could test the new version of the software using employees as test users while customers use the old version. One advantage a virtual machine environment for load balancing is that you can have as many virtual Ethernet devices as you desire and you can configure them using software (without changing cables in the server room). A limitation on the use of load-balancing routers is that traffic needs to go through the router in both directions. This is easy for the path from the Internet to the server room and the path from the server room to the customer network. But when going between servers in the server room it’s a problem (which is not insurmountable, merely painful and expensive). Of course there will be a cost in CPU time for all the extra routing. If instead of having a single virtual ethernet device for all redundant nodes you have a virtual ethernet device for every type of server and use the Dom0 as a router you will end up doubling the CPU requirements for networking without even considering the potential overhead of the load balancing router functionality. Finally there is a significant benefit in virtual machines for reliability of services. That is the ability to perform snapshot backups. If you have sufficient disk space and IO capacity you could have a snapshot of your server taken every day and store several old snapshots. Of course doing this effectively would require some minor changes to the configuration of machines to avoid unnecessary writes, this would include not compressing old log files and using a ram disk for /tmp and any other filesystem with transient data. When you have snapshots you can then run filesystem analysis tools on the snapshots to detect any silent corruption that may be occurring and give the potential benefit of discovering corruption before it gets severe (but I have yet to see a confirmed report of this saving anyone). Of course similar snapshot facilities are available on almost every SAN and on many NAS devices, but there are many sites that don’t have the budget to use such equipment.

• [1] http://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol

Share This

Thanks to Oliver Grawert and Michael Haas for this idea!

1. Check your LTSP server's sources.list Ensure that the server has the relevant APT sources to fetch the latest updates:

   deb http://security.ubuntu.com/ubuntu hardy-security main universe restricted multiverse
   #deb-src http://security.ubuntu.com/ubuntu hardy-security main universe restricted multiverse
   deb http://fi.archive.ubuntu.com/ubuntu/ hardy-updates main universe restricted multiverse
   #deb-src http://fi.archive.ubuntu.com/ubuntu/ hardy-updates main universe restricted multiverse
   deb http://fi.archive.ubuntu.com/ubuntu/ hardy main universe restricted multiverse
   #deb-src http://fi.archive.ubuntu.com/ubuntu/ hardy main universe restricted multiverse
   deb http://ppa.launchpad.net/q-funk/ubuntu hardy main
   #deb-src http://ppa.launchpad.net/q-funk/ubuntu hardy main

   In this example, the Finnish Ubuntu mirror for the Hardy suite is used. Change this as necessary. Notice that all source package lines are purposely commented out.
2. Build a boot image using that sources.list Build the image using the following command:

   ltsp-build-client --copy-sourceslist --accept-unsigned-packages

   This builds the image using the repositories found in the server's sources.list. Accepting unsigned packages is necessary, because the PPA is not digitally signed.

The above method is a good way to check what would happen on a clean install, if any particular package were updated to a newer proposed release that is currently sitting on someone's PPA. Enjoy!

Previously, I wrote:

Seriously: the browser looks like a big improvement from Firefox 2, but there are so many niggles with this download day idea...

In reply to Open Sesame Did you download Firefox 3?, I answer "Yes". It was a major upgrade for me, requiring new versions of Cairo and GTK+2, and installation of DBus-GLib on my GoboLinux computer, which brought in new versions of Xorg and so required a recompile of my GNUstep desktop applications. Once that was done, Firefox compiled unattended. As noted by Adam Sampson in the comments on my last post, even after building from source, you still get all the obnoxious click-through EULA and when you type about:config into the address bar, you get a "no user-servicable parts" sort of notice, which really sucks. I notice that MozCorp don't call it "100% Open Source", preferring instead Firefox: 100% Organic Software (because we need another marketing campaign for free software, right?), so I expect I need to winkle out the restrictively-licensed parts again - GNUzilla, there's still demand for your good work! After day 3 with Firefox 3, what do I think of it? Well, it seems a lot faster and a lot less RAM-hungry, and I'm quite impressed that all of the fancier bits of Koha and Wordpress seem to be working nicely but while I'm not annoyed enough to switch browsers yet (unlike FF3 and Safari - DrBacchus' Journal), there are still a hell of a lot of niggles and interface bugs. Some of the problems may have been introduced in Firefox 2, but I didn't actually use that enough to notice. My day-to-day browsing for the last year or so has been on a customised Firefox 1.5. The FF3 user interface has some big steps backwards from FF1.5: in particular, I've lost the "force pages that try to open new windows into the same window" option (or whatever it was called... I can't find the FF1.5 manual online anymore); some keyboard shortcuts have changed - for no good reason that I can see (JavaScript has switched from Alt-E n Alt-S to Alt-E n Alt-J, for example); what on earth is the history drop down doing next to the "Go Forward" arrow?; and the button to close a tab is on each tab, so I need to be careful to miss it when trying to switch to a tab and my pointer makes a pointless detour to the top-right when I want to close a tab. It's not all bad on the interface. The new RSS feed and bookmark links in the location bar are much better than in previous versions. The bookmark tagging and auto-generated folders could be a great idea once I've used it for a while. I'm pretty annoyed that Firefox 3 seems to come with some spyware enabled as default. I usually have cookies either switched off or set to "ask me every time" so I was surprised to be offered a cookie from safebrowsing.google.com! I know it's for a noble goal, but what's this doing enabled without asking first? Untick the "tell me if the site I'm visiting is ..." options in Edit: Preferences: Security if you don't want details of your browsing to be sent to the USA. Another thing which really annoys me is that the Firefox support site requires javascript and seems unhappy with my cookie settings. Not cool. Other than that, the main problems with Firefox 3 are omissions rather than bugs. For example, Microformats [Alex Faaborg] support was one of the long-trumpeted new features in Firefox 3, but they're really not obviously included, as noted by others in posts like Firefox 3 is here - where's the microformats? And finally, searching mozilla.com for firefox returns 0 hits, which is a bit strange... are they ashamed of it?

It’s a common practice when hosting email or web space for large numbers of users to group the accounts by the first letter. This is due to performance problems on some filesystems with large directories and due to the fact that often a 16bit signed integer is used for the hard link count so that it is impossible to have more than 32767 subdirectories. I’ve just looked at a system I run (Bluebottle anti-spam email service [1]) which has about half a million accounts and counted the incidence of each first letter. It seems that S is the most common at almost 10% and M and A aren’t far behind. Most of the clients have English as their first language, naturally the distribution of letters would be different for other languages. Now if you were to have a server with less than 300,000 accounts then you could probably split them based on the first letter. If there were more than 300,000 accounts then you would face the risk of having there be too many account names starting with S. See the table below for the incidences of all the first letters. The two letter prefix MA comprised 3.01% of the accounts. So if faced with a limit of 32767 sub-directories then if you split by two letters then you might expect to have no problems until you approached 1,000,000 accounts. There were a number of other common two-letter prefixes which also had more than 1.5% of the total number of accounts. Next I looked at the three character prefixes and found that MAR comprised 1.06% of all accounts. This indicates that splitting on the first three characters will only save you from the 32767 limit if you have 3,000,000 users or less. Finally I observed that the four character prefix JOHN (which incidentally is my middle name) comprised 0.44% of the user base. That indicates that if you have more than 6,400,000 users then splitting them up among four character prefixes is not necessarily going to avoid the 32767 limit. It seems to me that the benefits of splitting accounts by the first characters is not nearly as great as you might expect. Having directories for each combination of the first two letters is practical I’ve seen directory names such as J/O/JOHN or JO/JOHN (or use J/O/HN or JO/HN if you want to save directory space). But it becomes inconvenient to have J/O/H/N and the form JOH/N will have as many as 17,576 subdirectories for the first three letters which may be bad for performance. This issue is only academic as far as most sys-admins won’t ever touch a system with more than a million users. But in terms of how you would provision so many users, in the past the limits of server hardware were approached long before these issues. For example in 2003 I was running some mail servers on 2RU rack mounted systems with four disks in a RAID-5 array (plus one hot-spare) - each server had approximately 200,000 mailboxes. The accounts were split based on the first two letters, but even if it had been split on only one letter it would probably have worked. Since then performance has improved in all aspects of hardware. Instead of a 2RU server having five 3.5″ disks it will have eight 2.5″ disks - and as a rule of thumb increasing the number of disks tends to increase performance. Also the CPU performance of servers has dramatically increased, instead of having two single-core 32bit CPUs in a 2RU server you will often have two quad-core 64bit CPUs - more than four times the CPU performance. 4RU machines can have 16 internal disks as well as four CPUs and therefore could probably serve mail for close to 1,000,000 users. While for reliability it’s not the best idea to have all the data for 1,000,000 users on internal disks on a single server (which could be the topic of an entire series of blog posts), I am noting that it’s conceivable to do so and provide adequate performance. Also of course if you use one of the storage devices that supports redundant operation (exporting data over NFS, iSCSI, or Fiber Channel) then if things are configured correctly then you can achieve considerably more performance and therefore have a greater incentive to have the data for a larger number of users in one filesystem. Hashing directory names is one possible way of alleviating these problems. But this would be a little inconvenient for sys-admin tasks as you would have to hash the account name to discover where it was stored. But I guess you could have a shell script or alias to do this. Here is the list of frequency of first letters in account names:

First Letter	Percentage
a	7.65
b	5.86
c	5.97
d	5.93
e	2.97
f	2.85
g	3.57
h	3.19
i	2.21
j	6.09
k	3.92
l	3.91
m	8.27
n	3.15
o	1.44
p	4.82
q	0.44
r	5.04
s	9.85
t	5.2
u	0.85
v	1.9
w	2.4
x	0.63
y	0.97
z	0.95

• [1] http://www.bluebottle.com/

Share This

It happens from time to time that bug reports I file receive attention, but I don t notice because our bug tracking system still does not auto-subscribe bug submitters to their own bugs (see bug #37078 and bug #351856). I thus decided to take the matter into my own hands. Just in time, before I started hacking this up myself, I found Justin Pryzby s procmail recipies, which are installed to /usr/share/doc/devscripts/examples/bts_autosubscription.procmail by the devscripts package. The result is available in my mailfilter git repository. So far, I only auto-subscribe in response to bugs I file; Justin also auto-subscribes to bugs he manipulates via the control bot. For completeness, I also wanted to subscribe to all bugs that I have submitted. This turned out to be easier than I thought, thanks to bts in the devscripts package; note the sleep instruction in the loop to prevent hammering the system:

bts select submitter:madduck@debian.org \
    while read bugno; do
      echo X-debbugs-autosubscribe: madduck \
          sendmail -f madduck@debian.org $ bugno -subscribe@bugs.debian.org
      echo subscription to \#$ bugno  sent
      sleep 30
    done

NP: The Dukes of Leisure: The Dukes of Leisure

On the testing and unstable systems that i run, i occasionally get a disturbing "there are N newly obsolete packages" message. For example

[0 root@squeak ~]# aptitude update
Hit http://cmrg.fifthhorseman.net unstable Release.gpg
 [...]
Get:42 http://ftp.us.debian.org unstable/main [2077kB]                          
Fetched 15.2MB in 57s (262kB/s)                                                 
Reading package lists... Done
Updating debtags database...... Done
Current status: 3 updates [+3], 1136 new [-21212].
There are 1804 newly obsolete packages.
[255 root@squeak ~]# 

Usually, i can fix this with another aptitude update. Does this mean i've hit a mirror at an inopportune time? Or do i have apt mis-configured somehow? Other people have documented this before, and casual conversation with friends lets me know it's not just me. If this is due to some inconsistency at the mirrors (particularly for suites with rolling updates?), is there some way to engineer the mirror transfers so that this doesn't happen? For example (i have no idea how this stuff is done, so i'm just making this up):

1. rsync the pool/ without --delete to all mirrors,
2. rsync the Packages and Release, and DiffIndex files to all mirrors,
3. rsync the pool/ with --delete

Or should apt itself deal with these circumstances differently? It seems like signed apt ought to be able to detect when something like this is amiss. I'm reluctant to consider the situation acceptable, because i'd like to be able to trust apt in general. In particular, i can imagine apt eventually becoming more assertive about suggesting removal of newly obsolete and unsupported packages. If it is wrong about what's obsolete and unsupported, this could be dangerous.

I maintain a bunch of SSL certificates, mostly signed by my own site authority. Too many not to automate, but not enough to warrant heavy machinery. Here's how I do it. The configuration files Each certificate needs a config to describe what's in it. I create each of these and name it with a .cnf suffix. Here's an example:

[ req ]
prompt                  = no
distinguished_name      = server_distinguished_name
[ server_distinguished_name ]
commonName              = server.usefulinc.com
stateOrProvinceName     = England
countryName             = GB
emailAddress            = edd@usefulinc.com
organizationName        = Useful Information Company
organizationalUnitName  = Hosting
[ req_extensions ]
subjectAltName=edd@usefulinc.com
issuerAltName=issuer:copy
nsCertType            = server
[ x509_extensions ]
subjectAltName=edd@usefulinc.com
issuerAltName=issuer:copy
nsCertType            = server

Let's say this config is server.cnf. I then just type make server.pem to generate the corresponding certificate and key, signed by my local certificate authority. As I don't want to attend the startup of every service, I ensure the key is password-less. The Makefile rules Here are the makefile steps I use to generate and sign keys.

.SUFFIXES: .pem .cnf
.cnf.pem:
        OPENSSL_CONF=$< openssl req -newkey rsa:1024 -keyout tempkey.pem -keyform PEM -out tempreq.pem -outform PEM
        openssl rsa <tempkey.pem >  basename $< .cnf _key.pem
        chmod 400  basename $< .cnf _key.pem
        OPENSSL_CONF=./usefulCA/openssl.cnf openssl ca -in tempreq.pem -out  basename $< .cnf _crt.pem
        rm -f tempkey.pem tempreq.pem
        cat  basename $< .cnf _key.pem  basename $< .cnf _crt.pem > $@
        chmod 400 $@
        ln -sf $@  openssl x509 -noout -hash < $@ .0

The resultant files are:

• server.pem — contains both certificate and key in one file
  
• server_crt.pem — certificate file
  
• server_key.pem — key file
  

Some notes on these steps: my site-local certificate authority is in the directory usefulCA, along with an OpenSSL config which describes my preferences. This config was created by copying and making appropriate adjustments to the default /etc/ssl/openssl.cnf which ships with Debian. For generating certificate signing requests to ship to a commercial certificate authority, it's a bit simpler. I save the config files with a .reqcnf suffix instead, and use this rule:

.SUFFIXES: .pem .cnf .reqcnf .csr
.reqcnf.csr:
        OPENSSL_CONF=$< openssl req -newkey rsa:1024 -keyout  basename $< .reqcnf .key -keyform PEM -out  basename $< .reqcnf .csr -outform PEM

And finally, a rule I use to sign incoming certificate requests from other systems:

.csr.pem:
        OPENSSL_CONF=./usefulCA/openssl.cnf openssl ca -in $< -out  basename $< .csr _crt.pem

I offer these without warranty in the hope they might be useful to somebody. They're not much more than a transcription of a how-to into a makefile, but it's just enough technology to ensure creating certificates isn't a big nuisance. Further reading

• A more complete guide to setting up a site-local certificate authority
  

Notes Why do I bother with a site-local CA, rather than just self-sign? It lets me bypass the annoyance of SSL warnings on clients once I've installed my own CA certificate, and gives me a coarse grained level of access control: for instance, only clients with certificates signed by my CA are allowed to access the site's LDAP server. My personal next step with this is to integrate the certificate production process with my emerging Puppet recipes for managing local infrastructure.Join the conversation about this post