Hey! I have my 3-month update with my surgeon on Tuesday. We'll be hopefully reviewing my movement restrictions and getting the final all-clear to return to work at the beginning of next month. I don't expect that I'll be able to lift Leif, but even if I can twist and bend a little bit and reach out with both my arms at once it would be a huge improvement for me.

Do any of you have any questions? I'm going to ask for some clarification on the photo that I posted before (and hopefully will get another one). But if there's anything that other people want to know, I'll ask as much as I can.

Yesterday my wife's (Angie) father's (Ernie) wife's (Janet) mother (Lydia) died from a heart attack. I've only met her a few times, so I'm not really shaken by it, but she was a really nice lady. Something I think about is that with Janet marrying into the family after Angie and her siblings had all moved out means that there's a large chunk of Leif's family now that Angie and I can't really teach him about. So, I worry about Henry, and I'm thinking a lot about Janet. Since we're going up to Vancouver at the end of the week, we should hopefully be in town for any memorial service.

It's crazy to think back over the past 3 months. Various family members health problems; Leif learning to walk, and getting his first words; Leif moving to his own room and his own bed; Going from where I needed a walker and a trip to the mailbox was far, to where a ~1km to a park pushing a stroller makes me tired, but I can do the trip back easily enough; Going through the work to change to a new work visa; almost completing one of my master's courses; and importantly: I've got 30 of the songs finished on "Hard" in Guitar Hero 2.

It'll be fun to get back to work and regular life, though. =)

How to view the fingerprint of the ssh host key This way, ready to copy and paste:

ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key

Background: It already takes a lot of resources to recall that to see the host key fingerprint you need to run something called 'keygen'. Then ssh-keygen doesn't support --help: it will try to generate a new key instead. We're in 2008. There should be a law against this sort of behaviour. To figure out how to see the host key, you need to dig through a long manpage with no examples section. ssh-keygen does have commandline help, but does not implement any switch to invoke it (check the getopt invocation in the source code if you don't believe me). It will however show commandline help when given an unrecognised option, so it will mutter but at least give you love if you ask for it:

$ ssh-keygen - 
ssh-keygen: illegal option -- 
Usage: ssh-keygen [options]
Options:
  [...]

After figuring out that it's -l -f, you still have to go and fish the file wherever it is. And luckily we had the recent Debian openssh problems, so now I can get the fingerprint of the RSA file only and be done with it. But thanks to this blog entry, no more of that, at last.

There are obviously some things we need to remind if we don t want something like the OpenSSL debacle to happen again. It doesn t mean we need to throw stones nor to rush into changing our processes without thinking. However, there are already some things that should be obvious but unfortunately are not.

1. Shipping a giant diff.gz that contains all changes in one, putting security fixes, policy fixes, bug fixes, cosmetic changes and autotools files at the same level, is not something we should accept anymore. Improvements in the dpkg-source format are much welcome in this direction, but they are useless if maintainers don t use them. Neither a VCS nor a build tool will be able to know which line of the changes is related to which bug. Only the maintainer can.
2. Core packages should all have co-maintainers. This is pretty much stating the obvious, and is much easier said than done. The OpenSSL case is one of the best examples here: Kurt is not one of those who refuses help, but frankly, would you want to maintain that package? Having already maintained packages with messy code, upstreams not understanding at all the needs of a distributor, avalanches of security alerts and randomly-changing ABIs, I can tell you this is no fun like it can be to hack on a desktop environment or a device driver. The only sane reason to do this is that you need the package to work. The only visible result you get from your work is that programs are not randomly crashing.
   I have no magic recipe to propose so that more people help with such packages, and that s where we need to be really innovative. Cross-distribution teams, mandatory co-maintainership on a core package for each DD these (and all ideas I have not heard of) are the experiments we should start now.
3. Patching bad code leads to unpredictable results. What maintainer of a complex package has not introduced a new bug while trying to fix another one? Even when a piece of code is maintained by uncooperative developers, is not commented, uses arcane variable names or is impossible to understand without having contributed 3 winning entries to the IOCCC, it needs to evolve. And in these cases, it is only a matter of time until such things happen.
   Don t get me wrong: I m not trying to put the blame on upstream here. They have contributed very valuable code to the community and their work helped in the considerable widespread of cryptography. It s just that their code is not enough for our needs. If we can t patch it safely (and I m now convinced we can t), maybe we need to focus on alternatives and help them getting used by crypto-related packages. The code in GnuTLS and NSS is not necessarily better, but most (if not all) patches Debian needs to apply to them are build and portability fixes.
4. Unless Debian-specific, 1 patch = 1 bug in the upstream tracker. This should be obvious, but given the number of patches that are never forwarded, it doesn t seem so. You should not only give a chance for upstreams to review the patch, but you need them to track it, and you must give them the chance to review it anytime someone else stomps on a similar issue. If upstream does not have a bug tracker, they probably think their software has no bugs. Which means they are not trustworthy, and we go back to point 3.
5. We need to give more priority to security. Issues in the security team seem now fixed for good and they have been doing an awesome work. There isn t much left to do so that packages are all built with security-hardening features, but it still needs to be done. And there is much more to do so that we can provide out of the box a decent SELinux setup, or, if it turns out unrealistic to do, a decent system hardening setup using another framework. I know the SELinux zealots will jump on their high horses to explain that their framework is better, but the current situation where it is impossible for the average system engineer to setup a Debian-based MAC system is much worse than having a suboptimal setup that already works.

All in all, this incident has a great impact on Debian s image. If we don t react accordingly, adapting our processes and our system to match what our users expect from us and they expect the best they will turn away from us. With very good reasons to do so. Update : It seems OpenSSL does have a bug tracker. Thanks Kurt for pointing me to it.

Cowdancer is activly developed in debian, that's very nice. it's really a cool tool to build packages for unstable but also for backporting efforts. I'm using it on a daily basis to build debian etch packages of lustre. However three days ago cowdancer stopped working with my etch chroot after an upgrade of my unstable system:

dpkg-source: info: building lustre in lustre_1.6.5~rc3-1.dsc
dpkg-genchanges -S >../lustre_1.6.5~rc3-1_source.changes
dpkg-genchanges: including full source code in upload
dpkg-buildpackage: source only upload (original source is included)
chroot: cannot run command  cowdancer-ilistcreate': No such file or directory
W: cowdancer-ilistcreate failed to run within chroot, falling back to old method
 -> Running in no-targz mode
I: using fakeroot in build.
Current time: Sat May 17 12:18:30 CEST 2008
pbuilder-time-stamp: 1211019510
 -> copying local configuration
 -> mounting /proc filesystem
 -> mounting /dev/pts filesystem
 -> policy-rc.d already exists
Obtaining the cached apt archive contents
Installing the build-deps
cowdancer: .ilist size unexpected
cowdancer: .ilist size unexpected
Can't open perl script "/usr/bin/dpkg-architecture": Cannot allocate memory
 -> Attempting to satisfy build-dependencies
 -> Creating pbuilder-satisfydepends-dummy package
cowdancer: .ilist size unexpected
cowdancer: .ilist size unexpected
cowdancer: .ilist size unexpected
cowdancer: .ilist size unexpected
cowdancer: .ilist size unexpected
sh: /tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy/DEBIAN/control: Cannot allocate memory
E: pbuilder-satisfydepends failed.
Copying back the cached apt archive contents
 -> unmounting dev/pts filesystem
 -> unmounting proc filesystem
 -> Copying COW directory
 -> Invoking pbuilder
 -> Cleaning COW directory
Command /bin/sh -c pdebuild "--pbuilder" "cowbuilder" "--" "--basepath" "/var/cache/pbuilder/etch.cow" failed

After some research I found that a change of cowdancer (something related to .ilist) is responible for this issue. After that I asked on #debian-devel if someone else had this problem and were told that removing an re-creating the chroot will help. But that results in the same error msg. After that I decided to wrote a bugreport against cowdancer, since this is in my eyes a major defect in this software to do not support the latest stable release. I filled this report as RC Bug and got very fast a answer:

severity 481344 wishlist
retitle 481344  provide cowdancer etch backport
[ ... ]

I think this is the wrong way to fix a problem in unstable with a backport of a software. Providing a fallback for older debian releases would be the better way. Is anybody else having this problems?

I was reading about fiber to the home, but got interrupted by the fiber installation man. BTES got here less than 24 hours after I signed up, so I got to see a fiber optic splice done in the field this morning, 1st time for me. I think they're using Active Ethernet. I will so not miss my crappy cable modem.

This post is to some extent a fall-out of the recent discovery of a serious flaw in Debian's openssl and openssh packages. However, as Raghavan will confirm, a weaker version of what is said below was part of our discussion two weeks ago. Moreover, there are earlier posts on this topic in this blog.There are times when I wonder why I am so involved with computers ... this is clearly one of those times.

कर्मण्यैवाधिकरस्ते मा फलेशु कदाचनkarmaNyaivaadhikaraste maa faleshu kadaachana

This is a phrase from the भगवद गीत (bhagavad giit) which has been stuck in my head for the last 30 odd years. It roughly paraphrases into

Do something because you think it is worth doing not because of what you hope to achieve by doing it.¹

The above maxim is a good one but is sometimes a cop-out. Moreover, it provides no basis for actually making ethical choices.Ethics comes from one's interactions with the communities one is a part of. There are (roughly) two communities that I am a part of in the context of computers:

• the FOSS community; specifically, the Debian community.
• the IMSc computer community

I take these in turn below.

Debian and FOSS communitiesBy its own standards, the Debian community has suffered a massive failure ... and by these same standards it has reacted extremely well to this failure.I feel shame and blame. Why have I been ignoring RFH#332498 all these days when it shows up in the output of wnpp-alert? Here I am, a mathematician with some understanding of the issues, not helping out! Three years ago I even gave a short course of lectures on implementations of crypto; the source of openssl and openssh were used as examples. Excuses like, "I don't know anything about library packaging" and "I need more time!" (who doesn't) seem too weak now.At the same time, I feel a sense of solidarity with the Debian (and more widely FOSS) community as it tries to pull out of the resulting mess. The resilience that allows us to laugh wryly at ourselves is IMHO admirable.As Steve Kemp wrote: "[When we look back we will see that] we did good".

IMSc computer communityThe IMSc computer setup was built by volunteers and was genuinely a community when I joined this institute in 1996. It has since then broken into users, system administrators and the computer committee. As Indira Gandhi would have said: "This is a world-wide phenomenon", and as was the case when she said it, my response is: "That doesn't make it a good thing!"

When I speak about this fractured IMSc community below it is in generalities. There are certainly individuals who rise above the shards.

I have been promoting the use of FOSS and more specifically Debian at IMSc ever since I got here. While explaining the pragmatic aspects like cost and security, I have also tried to emphasize the freedom and community aspects of FOSS use. When the latter are not understood or accepted, the former are easily blown away.This year I made an attempt to get the Computer Committee to invite users to choose² their own computers and the software that ran on it, but it turned out that no one really wanted this. Users just wanted to buy "fancy toys", the administrators just wanted to make their life simple and CC members just wanted the power to dictate what people bought.³I also made an attempt to get our users to educate each other on the use of computers for their work --- first through the establishment of a wiki and then through the "No-Excuse" mailing list.⁴Unfortunately, the fractured IMSc computer community sees computers and software as expensive commodities --- with some combination of fear, greed and irritation. In any case, there is no feeling of being part of a larger community that is trying to solve (certain types of) problems.

A wise man once said:

With great freedom comes even greater responsibility.

This may explain why we prefer being dictated to by proprietary vendors and computer committees that "buy stuff for us". We are afraid of the responsibility that comes with freedom.Another much quoted quote is:

If you are not part of the solution then you are part of the problem.

The FOSS community (actively) invites people to join-in in solving problems. This participation (which can be at a level of one's own choosing) is the source of one's freedom in free software. The IMSc community just wants मा-बाप सर्कार (maa-baap sarkaar) to fix their toys.

A decisionThe IMSc computer community was tied to Debian and FOSS after the break-ins into our system in the early 2000's. At that point, I was instrumental in installing a security infrastructure based mainly on Debian. This led to my greater involvement with FOSS and Debian and also to the greater "infiltration" of Debian and FOSS into IMSc computers.From the "commodity" point of view at IMSc we have come a full circle since most users will have to "do a lot" so that the IMSc computer LAN emerges unscathed from the crisis created by the Debian openssl flaw.From the "community" point of view at Debian we have spiralled out and even this major whirlpool will not drag us back in.It is no longer possible (for me) to straddle the circle and the spiral in an attempt to widen the former; I'm taking the "outward radial vector"!

---

1. To this sentiment I have often added "do it because it's fun". ↩
2. ... and be willing to justify their choices of course.[WGFCEGR] ↩
3. OK! I'm exaggerating a bit! ↩
4. No-Excuse is an acronym for Novice and Expert Computer Users. The blurb says:

   Now that this list exists there is "no-excuse" for novices to remain in-experienced or for experienced users to claim that something is "too hard to explain to a novice"!

   ↩

I believe that the Red Hat process which has Fedora for home users (with a rapid release cycle and new versions of software but support for only about one year) and Enterprise Linux (with a ~18 month release cycle, seven years of support, and not always having the latest versions) gives significant benefits for the users. The longer freeze times of Enterprise Linux (AKA RHEL) mean that it often has older versions of software than a Fedora release occurring at about the same time. In practice the only time I ever notice users complaining about this is in terms of OpenOffice (which is always being updated for compatability with the latest MS changes). As an aside, a version of RHEL or CentOS with a back-port of the latest OpenOffice would probably get a lot of interest. RHEL also has a significantly smaller package set than Fedora, there is a lot of software out there that you wouldn’t want to support for seven years, a lot of software that you might want to support if you had more resources, and plenty of software that is not really of interest to enterprise customers (EG games). Now there are some down-sides to the Red Hat plan. The way that they run Fedora is to have new releases of software instead of back-porting fixes. This means that bugs can be fixed with less effort (simply compiling a new version is a lot less effort than back-porting a fix), and that newer versions of the upstream code get tested. With some things this isn’t a problem, but in the past I have had problems with the Fedora kernel. One example was when I upgraded the kernel on a bunch of remote Fedora machines only to find that the new kernel didn’t support the network card, so I had to talk the users through selecting the older kernel at the GRUB menu (this caused pain and down-time). A problem with RHEL (which I see regularly on the CentOS machines I run) is that it doesn’t have the community support that Fedora does, and therefore finding binary packages for RHEL can be difficult - and often the packages are outdated. I believe that in Debian we could provide benefits for some of our users by copying some ideas from Red Hat. There is currently some work in progress on releasing packages that are half-way between Etch and Lenny (Etch is the current release, Lenny will be the next one). The term Etch and a half refers to the work to make Etch run on newer hardware [1]. It’s a good project, but I don’t think that it goes far enough. It certainly won’t fulfill the requirements of people who want something like Fedora. I think that if we had half-way releases of Debian (essentially taking a snap-shot of Testing and then fixing the worst of the bugs) then we could accommodate user demand for newer versions (making available a release which is on average half as old). Users who want really solid systems would run the full releases (which have more testing pre-release and more attention paid to bug fixes), but users who need the new features could run a half-way release. Currently there are people working on providing security support for Testing so that people who need the more recent versions of software can use Testing, I believe that making a half-way release would provide better benefits to most users while also possibly taking less resources from the developers. This would not preclude the current “Etch and a half” work of back-porting drivers, in the Red Hat model such driver back-ports are done in the first few years of RHEL support. If we were to really follow Red Hat in this regard the “Etch and a half” work would operate in tandem with similar work for Sarge (version 3.1 of Debian which was released in 2005)! In summary, the Red Hat approach is to have Fedora releases aimed at every 6 months, but in practice coming out every 9 months or so and to have Enterprise Linux releases aimed at every year, but in practice coming out every 18 months. This means among other things that there can be some uncertainty as to the release order of future Fedora and RHEL releases. I believe that a good option for Debian would be to have alternate “Enterprise” (for want of a better word) and half-way releases (comparable to RHEL and Fedora). The Enterprise releases could be frozen in coordination with Red Hat, Ubuntu, and other distributions (Mark Shuttleworth now refers to this as being a “pulse” in the free software community [], while the half-way releases would come out either when it’s about half-way between releases, or when there is a significant set of updates that would encourage users to switch. One of the many benefits to having synchronised releases is that if the work in back-porting support for new hardware lagged in Debian then users would have a reasonable chance of taking the code from CentOS. If nothing else I think that making kernels from other distributions available for easy install is a good thing. There is a wide combination of kernel patches that may be selected by distribution maintainers, and sometimes choices have to be made between mutually exclusive options. If the Debian kernel doesn’t work best for a user then it would be good to provide them with a kernel compiled from the RHEL kernel source package and possibly other kernels. Mark also makes the interesting suggestion of having different waves of code freeze, the first for the kernel, GCC, and glibc, and possibly server programs such as Apache. The second for major applications and desktop environments. The third for distributions. One implication of this is that not all distributions will follow the second wave. If a distribution follows the kernel, GCC, and glibc wave but not the applications wave it will still save some significant amounts of effort for the users. It will mean that the distributions in question will all have the same hardware support and kernel features, and that they will be able to run each others’ applications (except when the applications in question use system libraries from later waves). Also let’s not forget the possibility of running a kernel from distribution A on distribution B, it’s something I’ve done on many occasions, but it does rely on the kernels in question being reasonably similar in terms of features.

• [1] http://wiki.debian.org/EtchAndAHalf


• [2] http://www.markshuttleworth.com/archives/150

Share This

One of the proudest moments I had as a Canadian was sitting in the Senate visitor's gallery watching the debates on same-sex marriage that led to the full legalisation in Canada. In coming down to the US for work, I was sad to come to a place that not only didn't allow it, but had actually had a number of ballots in states voting specifically that the marriage referred to "One Man, One Woman".

I found a transcription of the Hansard of Prime Minister Paul Martin's introduction of Bill C-38 (The Civil Marriage Act). The logic in there for why Canada needed to legalise same-sex marriage is quite specific to Canada, but he talks about why it's important for the issue not to come to a vote:

The Charter was enshrined to ensure that the rights of minorities are not subjected, are never subjected, to the will of the majority. The rights of Canadians who belong to a minority group must always be protected by virtue of their status as citizens, regardless of their numbers. These rights must never be left vulnerable to the impulses of the majority.

I don't really understand the US system of rights. I hope that the protection of the right to love another consenting adult and make a commitment to them becomes the law of the land throughout the United States.

I love being married. I get the joy of being able to come home to my belle and share my day, spend time, and know that to the best of our ability, we're going to try to grow old together. I'm so happy that this ruling has come down and that in California the debate is now over.

For a project I was looking for a way to use GNU Mailman for mailinglist management (especially its powerful web based moderation, bounce handling and attachment scrubbing features) but storing the mailinglist member information in a SQL database. That's useful when you want to relate mailinglist members to extra information already in a database and functionality on an interactive website, and it's possible on a per-mailinglist basis. This turned out to be not very difficult but just not that clearly documented. What I used:

• Mailman. I used the 2.1.x version packaged in Debian etch: an "open" system is just one apt-get away!
• The MySQL Member Adaptor available from rezo.net. There are several versions of MySQLMemberAdaptor around the web, but this one seems the most recent.

This explanation assumes you know how to work with Mailman in a regular setup, and things like creating databases and users under MySQL. Take the following steps:

1. Install and set up mailman as you would otherwise.
2. Put the MysqlMemberships.py file into the dir that is Mailman's base, this normally already has files like MemberAdaptor.py in it. For Debian this is /usr/lib/mailman/Mailman. You only need to add MysqlMemberships.py, the other files from rezo.net are not needed.
3. The create table query in MysqlMemberships.py does not work with MySQL 5, I had to apply this patch which I've sent to MySQLMembership's author in the meantime. This patch has been applied by Fil.
4. Create a database and a MySQL user, and add that information into your mm_cfg.py, like this:

   MYSQL_MEMBER_DB_NAME = "mailman"
   MYSQL_MEMBER_DB_USER = "mailman"
   MYSQL_MEMBER_DB_PASS = "somethingrandom"
   MYSQL_MEMBER_DB_HOST = "localhost"
   MYSQL_MEMBER_TABLE_TYPE = "wide"        # wide means one table per list, flat = one large table

5. Create a mailinglist like your normally would (e.g. via newlist). To enable the MySQL backend for just that list, create a file extend.py under /var/lib/mailman/lists/listname with the following content:

   from Mailman.MysqlMemberships import MysqlMemberships
   def extend(list):
           list._memberadaptor = MysqlMemberships(list)
   

   (whitespace is significant in Python).
6. It should work now! Upon receiving the first request for your list, the database table will be created automatically, which you can then populate. You may need to restart the mailman qrunner after you make further changes to your setup.

And today, I finally turn 30. I've been grumpy about this day getting closer and closer for the last three or four years, which have passed in front of my eyes with me nearly not noticing. The last year has had more downs than ups, and at times has been quite dark. I feel things are slowly getting better, and I spend more time looking forward than back, which certainly should help. Tomorrow I'll hold a small party at home with some friends, but the big and proper event will be in September, when five or six people in our colla, born in 1978, will celebrate our 30^th birthday, in a massive, weekend-long party already dubbed La festa dels excessos. You shouldn't miss this one! Thanks to the many people who have phoned, texted or emailed me already. It reminds me that I'm surrounded by people who love me and were there when I needed them.

How does one convert between SSH keys and SSL keys?

Let us first divide the problem into two parts.

Private Key availableThe simpler case is when you have the private key. In this case, SSH private key is in fact the same as an SSL key. If the key was generated using ssh-keygen -t rsa then it is a 2048-bit RSA private key for SSL.It follows that it rather trivial to extract an SSH public key from an SSL private key. You just run

ssh-keygen -f private_key_file -y > ssh.pub

This will prompt you for a password if your key is kept encrypted as it should be. The ssh.pub file contains the SSH public key.It is similarly easy to extract the SSL public key from an SSH private key.

openssl rsa -in private_key_file -pubout -out ssl.pub

will create ssl.pub containing the SSL public key.

Private Key un-availableThe more difficult case is when you only have the SSH public key or an SSL certificate. In this case one should be able to extract the SSL public key from the SSL certificate and interchange between the SSL and SSH public keys.The extraction of the public key from the SSL certificate is achieved by the command¹

openssl x509 -in cert_file -pubkey -noout > ssl.pub

which extracts the public key from the SSL certificate.

The MessThe second part, i. e. converting between SSH and SSL public keys, has a rather tortuous route given that the data contained in the public keys is the same!First of all one needs some somewhat un-common utilities (which are currently only in sid on Debian)

apt-get install lsh-utils 

This pulls in the nettle-bin package which also contains some useful programs. So one can run

ssh-conv < ssh.pub   sexp-conv 

To get a look at the "real stuff" inside the SSH public key.If ssl.pub and ssh.pub refer to the same key then you will see that the output of

ssh-conv < ssh.pub   sexp-conv -s hex

and

openssl rsa -in ssl.pub -pubin -noout -text

are entirely similar and clearly inter-convertible. The same applies to DSA keys with the openssl rsa replaced by openssl dsa in the command above.One could easily write a script that would massage one of these text formats into the other. Luckily, someone did it for us. So we run

sed -e'1d' -e'$d' < /tmp/test.pub   \
 pkcs1-conv --public-key-info --base-64   \
 sexp-conv   \
 sed -e'1s/(rsa/(rsa-pkcs1-sha1/'   \
 sexp-conv -s transport

The output of this is identical to the output of

ssh-conv < ssh.pub   sexp-conv -s transport

Hence, we have reached the point of convertibility! The full pipeline is

sed -e'1d' -e'$d' \< ssl.pub   \
 pkcs1-conv --public-key-info --base-64   \
 sexp-conv   \
 sed -e'1s/(rsa/(rsa-pkcs1-sha1/'   \
 sexp-conv -s transport   \
     lsh-export-key --openssh > newssh.pub

ConclusionHaving got this far the reader may be wondering what the purpose of this (lack of) exercise is --- other than writing complex pipelines instead of real code.

Answer:

Since you can now convert certificates into SSH public keys you can check these certificates against the SSH public keys that became bad as a consequence of the recent fiasco.Moreover, to do this you do not need to have access to the private keys of the certificates.

So help out a friend by checking whether their SSL certificate is based on a weak key. However, note that the above procedure doesn't work for DSA keys. In addition, the publicly available tests for bad keys do not work with 1024-bit RSA keys. If your friend has been using the letter type of keys on a Debian "etch" system then you should advise them to change keys anyway (and switch to 2048-bit RSA in the process).

---

1. Locating this command took a bit of effort since the -pubkey option to x509 is not documented on its man-page. Rather it is documented on the rsautl man-page. (See: Debian Bug #291095). ↩

If you're a French person and were guessing what I have to do with the most famous French union, you've just been tricked. CGT stands for California Guitar Trio, a band of three virtuose guitar players. We attended their French gig yesterday night at 'Le New Morning', a famous jazz club in Paris. They came and played with Tony Levin who is, for the skilled music amateurs, one of the top world's bass player (*the* top for me). That was an amazing concert and great experience. Just try to imagine Pink Floyd's 'Echoes' played by 3 (electrified) acoustic guitar players, just after a 'Prelude and Fugue' by J.S. Bach, played cyclically (one note per player in round-robin order....a playing method invented by Robert Fripp and the League of Crafty Guitarists). That plus Bohemian Rhapsody, several pieces of their own and a stunning composition by Tony Levin where he proved that he *is* really a virtuose bass player..:-) I was just stunned, indeed....and, of course, also enjoyed a lot the presence in a very small club of Uncle Tony who I saw many times until now, but only in big arenas in all Peter Gabriel concerts I've attended since about 1980. CGT will play tonight (15th) on Milan, tomorrow in Roma, on 27th in Toronto, 31th in Rouyn-Noranda (Quebec) and a few times with friends during the Quebec Summer Festival. Then, they'll play in many places in California and all over USA. If you love eclectic music, just go listening to them. So, yes, CGT rocks!

So, Charter will be spying on every web page their customers view, and collating and using this information in a Big Brotherly fashion. But it's for advertising, so it's a good thing, right? Yeah, right. I signed up for fiber to the home from my local ISP today. $charter_customers--. Oddly, besides not being a privacy nightmare, my local ISP is cheaper too. Who'd have thought. Anyone using http://kitenet.net/wifi wireless or on the network here: You should assume Charter is spying on you for now. (BTES will install their connection on Monday.)

When an emergency occurs in a critical piece of your infrastructure, problems are to be expected. I don't think anyone can argue with that. If you do have a piece of infrastructure so critical that the use of your services all over the country will be affected by it not being there anymore, you'll want to have emergency procedures in place; one of these procedures could be having a backup copy of your infrastructure standing by, allowing it to take over should the primary break down. If you have multiple instances of almost-but-not-quite the same thing, allowing another instance of said thing to take over your functioning should you break down might be a good idea, too. If the emergency that sparks the problem in the first place turns out to be "fire", it might be a good idea to investigate your infrastructure for fire safety; you wouldn't want the same thing to happen all over again in te near future on a different site. And if things go really bad, you will want to inform your users as much as humanly possible: in today's day and age, that would include posting something on your website, even if that's only reference to a phone number which people can call for more information. Welcome to the real world. Long live the NMBS... Not. Last year or so, some critical piece of infrastructure in the Brussels South trainstation burned down. As a result, trains were delayed and often cancelled for months afterwards. This clearly shows they did not have any emergency procedures in place; if they did, train services would have returned to normal after, at most, a few days. Granted, the piece of infrastructure that burned down last year was a power transfer station, which might be hard to replace, but still. Today, another fire burned down another piece of infrastructure; this time, it was the signalling station in Brussels North. The fire itself occurred around 16:30, and from 17:30 onwards, not a single train could enter or leave Brussels anymore. Right now, more than 7 hours later, it is still impossible to get from or to Brussels. Even here, in Mechelen, it seems impossible to get to Antwerp—the opposite site, but because almost all trains to Antwerp come from Brussels, well... And of course, the NMBS website mentions none of this. The only thing it mentions is the fact that there are "extra trains to the coast", because of the hot weather. As if anyone would be interested in taking the train right now... Huge companies. A PITA, that's what they are.

Let me just point out, that the consequences affect all users of SSH. Therefore IMHO all other Linux and BSD distributions need to release a security update to OpenSSH as well, to prevent the use of insecure (too common) keys, because it threatens the security of their systems as well!Apparently, there are only about 2^15 different keys generated by the SSH versions shipped with Debian for 2 years. It's really surprising that noone noticed this earler. This is just about 32767 different keys. (For each type, size and endianess, but that still makes this number much much much too low) The weakness is caused by a bad random number generator in the Debian package.Hackers have already generated all these 32767 different keys, for two key lengths and types. In a few hours, they'll also have generated all the 4096 bit keys that could have been generated. Other key lengths are uncommon and sometimes might even be unsupported. Most people use keys with length 1024 or 2048.So we now have about 32767 keys which are used by lots of Debian and Ubuntu users. That's not very much. Now you have to realize how the keys are used:The key is used to log into a system without a password. Sometimes a key is protected with a passphrase (you really should do that), but this doesn't help here, because an unencrypted clone of the key was already generated.Sometimes (or let me even claim 'often') one such key is also used to login as root into a server. This is equivalent to just 32767 different passwords being used as root passwords. So with about this number of tries, an attacker might be able to log into your server as 'root'!Now the weakness is 'distributed' by the users, it's not just a server-side vulnerability. If your server is running e.g. RedHat, it doesn't mean it is secure!.In fact, if your server is running Debian and you installed the Debian security update for openssh, it will be much more secure than the RedHat server. Because the Debian server has a blacklist of keys that are too common. The other-Linux server who doesn't have this blacklist doesn't know that a certain 'weak' key is not trustworthy.Fixing the bad key-generation is just half of the deal. "Recalling" all the keys in use out there is the big challenge, that affects all systems using SSH (and to a different extend, SSL). The most reliable way is if all other distributions would release a security update as well, which refuses to accept the keys that were generated by vulnerable Debian systems.Let me just repeat it in other words: Any Linux/Unix/*BSD system is vulnerable that grants access to a key that was generated on an affected Debian or Ubuntu system. (Until the system has a reliable detection method of such weak keys.) Keys are usually generated on the users workstation, so if any of your users is or was potentially running Debian or Ubuntu ... you get the idea.Note that if you are not careful, you might lock yourself out from your server. If you don't have or remember the password, installing the security update might disable your login key. So if your key is bad, make sure to generate a new, secure key and distribute it ASAP. Also remove any vulnerable key ASAP; remember that hackers now have a list of all possible keys and could use that to brute-force login.P.S. Since some people still don't seem to get the consequences in full: The bigger problem is to remove are the weak keys, not to fix the broken library. The weak keys (especially in the form of public keys!) can live on tons of other systems, not just on Debian and Ubuntu. This is why TOR also released a security update and e.g. CACert urges non-Debian distributors to also ship and use the blacklists of known weak keys. Also note that not all keys that can be considered compromised can be detected this easily. If you've been using a DSA key on an affected system - even when it was created on a different system - it is to be considered compromised.

After recent not so funny thing with OpenSSL in Debian, I realized that I will have to regenerate most of keys and certificates, because last big changes I did in networking/vpn/ssh setup which involved generating keys are not older than broken OpenSSL appeared in archives. First obvious thing was SSH keys and cleanup of ~/.ssh/authorized_keys on all hosts. While doing that, I realized that I still have there several keys, which are more or less gone (not that I'd lost them, but I simply stopped to use them). So it was good opportunity to do cleanup here. While I was at these changes, cleaning up ~/.ssh/known_hosts was also good idea, because I still had there lot of hosts I collected during some of my previous jobs and I definitely won't (and can not) access these machines anymore. So good, big cleanup in SSH configuration was forced :-). Next and harder step was to found out where else I use certificates generated by vulnerable OpenSSL. Server certificates for sure were also generated by OpenSSL, so let's regenerate web and email certificates and hope I did not miss anything. All this happened yesterday, but today I realized that I missed other even more important thing - OpenVPN certificates. While regenerating certificates, I also found some machine keys which are not really used anymore, so I again could drop some of them. So that was task for this evening and now I'm hopefully really done with this issue and I really hope that this won't happen again in near future, I don't need to cleanup that often ;-).

Anyone seen this before? Today, I went to an event about the Sub-National Review Consultation (as a substitute for someone else AIUI). I'd not heard about this before, but if you're in England and you've any interest in our regional planning system (which I think you should, if you have your main home here or run a business here), you have six weeks left to comment on the UK Government's suggested changes. As I understand it, it will move the second-highest tier of planning control from democratically-accountable regional bodies to the business-led Regional Development Agencies, with some oversight by MPs and the very- indirectly-accountable council leaders. I've posted more detail on Co-opNet. When I asked about local involvement and cooperatives, I was directed towards Local Strategic Partnerships, but I'm pessimistic about how easy it will be to influence regional planning through those: a few weeks ago, I was at the launch of the North Somerset Partnership Sustainable Community Strategy for 2008-2026. It's a 72-page A4 glossy book which I've still not found time to read properly. I think the size says something about its sustainability. I've posted a little more detail on WsMForum. I'll try to answer questions about either of them on this blog or those forums...

I did this three years ago, too. I guess now's the time to go again: I just went through my referer logs—specifically, keywords people entered in search engines right before they ended up on my site—, and will try to answer those questions people seem to have when they end up on my site. Here goes:

• voorwoord eindwerk: Eh. How about you make your homework yourself? (to my english readers: "voorwoord eindwerk" is dutch for "foreword dissertation"... and a few months ago, I noticed that I accidentally had my brother's dissertation on my website. It's now removed, and the search hits seem to have stopped, too, but it's still in my not-so-recent referer logs)
• php accept language: Yes, I did write that. By the number of people searching for it, you'd think it's the most important contribution I made to open source. I sure hope not, but anyway.
• wouter verhelst: Yes, you found him, all 1.4% of you. Perhaps you meant that other wouter verhelst, however.
• digikam vs f-spot/f-spot vs digikam: I eventually picked digikam, not f-spot. Make sure you pick the right one from the beginning, though, since switching currently requires you to re-tag everything.
• nicknames: My IRC nickname is 'Yoe'. But as I've explained before, I prefer not being called that elsewhere.
• vim vs emacs: Not sure why you end up on my site for that. I use both, depending on what I want or need to do. (emacs is more powerful in some areas, but takes far longer to start...)
• duck with whisky:yes, yes, yes, very funny.
• nbdmagic: The NBD protocol uses a magic number to mark the start of a request packet; this defends against the client and server getting out of sync (which may happen in case of a bug somewhere). If that happens, the server will kill the connection with a message "Not enough magic." Please report the bug, or (even better) fix it :-)

Moving further down, it's fascinating to see what people were looking for when they ended up on my site. Some of the below, I have absolutely no clue what happened...:

• belgium driver licence—go to your local municipal office, please!
• sockets in brainfuck—now, really.
• wouter's mum—look at your own, you dammit you!
• what to do with debian—I should know, how?
• several things related to "android"—I'm not involved in that
• list of programming languages to learn—that's easy: the list of programming languages you don't know, yet.
• kernel panic not syncing aiee killing interrupt handler 2.6.24 (and several other literal panic messages)
• verhelst -meaning—In other words: "anything that has verhelst, but no meaning". Thanks. I guess.

I guess blogging about technical things for several years has made me "interesting" enough for some people. Grin.

I wasn't going to comment on the recent openssl security update, because too many people have already done so. Personally I thought that Aigars Mahinovs made the best writeup I've seen so far. However I would like to say that having 20+ people all mailing security[at]debian.org to say the webpage we referenced in the security advisory is currently blank is not useful, or ask for details already released in the advisory they replied to, or ask for even more details is not so much fun. Having people immediately start mailing questions like "Huh? What can I do" is only natural, but you can't expect a response when things are as hectic as they have been recently. Ideally people would sit on their hands and bite their tongues. Realistically that isn't going to happen, and realistically this post will make no difference either... Had the issue not leaked to unstable so quickly (and inappropriately IMHO) then we'd have had a little more time. But once an issue is reported you need to coordinate with other distributions, and etc. Handling something as severe as this is not fun, and random mails from users are a distraction, and a resource-hog. I should say I was not in any way involved in the discovery, the reporting, the preparation of the fix(es), or the releasing of the update. I knew it was coming, but everybody else seemed to have it well in hand. When there are mails going back and forth for 5+ days with ever-growing Cc: lists, and mailing lists being involved I figure one more cook wouldn't be useful. So in conclusion:

a. Bad hole. b. Fixing this will take years, probably. c. 50+ mails to the security team within an hour of the advisory going public complaining of missing information is not helpful, not useful, and quite irritating. (Albeit understandable). d. People who don't know the details of an attack, or issue, shouldn't speculate and start panic, fear, and confusion. Esp. when details are a little vague. e. I still like pies.

Once again thanks to everybody who was involved and put in an insane amount of work. Yes this is only the start - our users have to suffer the pain of regenerating everything - but we did good. Really. Debian did good. It might not look like it right now, but it could have been so much worse, and Debian did do good. ObQuote: X-Men: The Last Stand