Yesterday I stumbled about files and dead symlinks left in /usr/lib/python2.3/site-packages/ on my Sid box. These files/symlinks seem to have been shipped/created(?) by:

python-ldap python-cairo python-crypto kiki python-foomatic python-mysqldb
python-logilab-common python-egenix-mxtools python-numarray python-pygresql
python-imaging-sane python-imaging python-xml python-reportlab

Deleting /usr/lib/python2.3 (dpkg -S didn’t show any package relationship nor did I find something in /var/lib/dpkg/info/) and reinstalling the above mentioned packages did not recreate the files/symlinks. So it seems the directory can be safely removed. Maybe I missed some announcement or one (or more) packages need to be fixed. No time to examine it atm. Update This is known as Debian bug #409390. Thanks to Josselin Mouette for the information.

This post is to some extent a fall-out of the recent discovery of a serious flaw in Debian's openssl and openssh packages. However, as Raghavan will confirm, a weaker version of what is said below was part of our discussion two weeks ago. Moreover, there are earlier posts on this topic in this blog.There are times when I wonder why I am so involved with computers ... this is clearly one of those times.

कर्मण्यैवाधिकरस्ते मा फलेशु कदाचनkarmaNyaivaadhikaraste maa faleshu kadaachana

This is a phrase from the भगवद गीत (bhagavad giit) which has been stuck in my head for the last 30 odd years. It roughly paraphrases into

Do something because you think it is worth doing not because of what you hope to achieve by doing it.¹

The above maxim is a good one but is sometimes a cop-out. Moreover, it provides no basis for actually making ethical choices.Ethics comes from one's interactions with the communities one is a part of. There are (roughly) two communities that I am a part of in the context of computers:

• the FOSS community; specifically, the Debian community.
• the IMSc computer community

I take these in turn below.

Debian and FOSS communitiesBy its own standards, the Debian community has suffered a massive failure ... and by these same standards it has reacted extremely well to this failure.I feel shame and blame. Why have I been ignoring RFH#332498 all these days when it shows up in the output of wnpp-alert? Here I am, a mathematician with some understanding of the issues, not helping out! Three years ago I even gave a short course of lectures on implementations of crypto; the source of openssl and openssh were used as examples. Excuses like, "I don't know anything about library packaging" and "I need more time!" (who doesn't) seem too weak now.At the same time, I feel a sense of solidarity with the Debian (and more widely FOSS) community as it tries to pull out of the resulting mess. The resilience that allows us to laugh wryly at ourselves is IMHO admirable.As Steve Kemp wrote: "[When we look back we will see that] we did good".

IMSc computer communityThe IMSc computer setup was built by volunteers and was genuinely a community when I joined this institute in 1996. It has since then broken into users, system administrators and the computer committee. As Indira Gandhi would have said: "This is a world-wide phenomenon", and as was the case when she said it, my response is: "That doesn't make it a good thing!"

When I speak about this fractured IMSc community below it is in generalities. There are certainly individuals who rise above the shards.

I have been promoting the use of FOSS and more specifically Debian at IMSc ever since I got here. While explaining the pragmatic aspects like cost and security, I have also tried to emphasize the freedom and community aspects of FOSS use. When the latter are not understood or accepted, the former are easily blown away.This year I made an attempt to get the Computer Committee to invite users to choose² their own computers and the software that ran on it, but it turned out that no one really wanted this. Users just wanted to buy "fancy toys", the administrators just wanted to make their life simple and CC members just wanted the power to dictate what people bought.³I also made an attempt to get our users to educate each other on the use of computers for their work --- first through the establishment of a wiki and then through the "No-Excuse" mailing list.⁴Unfortunately, the fractured IMSc computer community sees computers and software as expensive commodities --- with some combination of fear, greed and irritation. In any case, there is no feeling of being part of a larger community that is trying to solve (certain types of) problems.

A wise man once said:

With great freedom comes even greater responsibility.

This may explain why we prefer being dictated to by proprietary vendors and computer committees that "buy stuff for us". We are afraid of the responsibility that comes with freedom.Another much quoted quote is:

If you are not part of the solution then you are part of the problem.

The FOSS community (actively) invites people to join-in in solving problems. This participation (which can be at a level of one's own choosing) is the source of one's freedom in free software. The IMSc community just wants मा-बाप सर्कार (maa-baap sarkaar) to fix their toys.

A decisionThe IMSc computer community was tied to Debian and FOSS after the break-ins into our system in the early 2000's. At that point, I was instrumental in installing a security infrastructure based mainly on Debian. This led to my greater involvement with FOSS and Debian and also to the greater "infiltration" of Debian and FOSS into IMSc computers.From the "commodity" point of view at IMSc we have come a full circle since most users will have to "do a lot" so that the IMSc computer LAN emerges unscathed from the crisis created by the Debian openssl flaw.From the "community" point of view at Debian we have spiralled out and even this major whirlpool will not drag us back in.It is no longer possible (for me) to straddle the circle and the spiral in an attempt to widen the former; I'm taking the "outward radial vector"!

---

1. To this sentiment I have often added "do it because it's fun". ↩
2. ... and be willing to justify their choices of course.[WGFCEGR] ↩
3. OK! I'm exaggerating a bit! ↩
4. No-Excuse is an acronym for Novice and Expert Computer Users. The blurb says:

   Now that this list exists there is "no-excuse" for novices to remain in-experienced or for experienced users to claim that something is "too hard to explain to a novice"!

   ↩

I believe that the Red Hat process which has Fedora for home users (with a rapid release cycle and new versions of software but support for only about one year) and Enterprise Linux (with a ~18 month release cycle, seven years of support, and not always having the latest versions) gives significant benefits for the users. The longer freeze times of Enterprise Linux (AKA RHEL) mean that it often has older versions of software than a Fedora release occurring at about the same time. In practice the only time I ever notice users complaining about this is in terms of OpenOffice (which is always being updated for compatability with the latest MS changes). As an aside, a version of RHEL or CentOS with a back-port of the latest OpenOffice would probably get a lot of interest. RHEL also has a significantly smaller package set than Fedora, there is a lot of software out there that you wouldn’t want to support for seven years, a lot of software that you might want to support if you had more resources, and plenty of software that is not really of interest to enterprise customers (EG games). Now there are some down-sides to the Red Hat plan. The way that they run Fedora is to have new releases of software instead of back-porting fixes. This means that bugs can be fixed with less effort (simply compiling a new version is a lot less effort than back-porting a fix), and that newer versions of the upstream code get tested. With some things this isn’t a problem, but in the past I have had problems with the Fedora kernel. One example was when I upgraded the kernel on a bunch of remote Fedora machines only to find that the new kernel didn’t support the network card, so I had to talk the users through selecting the older kernel at the GRUB menu (this caused pain and down-time). A problem with RHEL (which I see regularly on the CentOS machines I run) is that it doesn’t have the community support that Fedora does, and therefore finding binary packages for RHEL can be difficult - and often the packages are outdated. I believe that in Debian we could provide benefits for some of our users by copying some ideas from Red Hat. There is currently some work in progress on releasing packages that are half-way between Etch and Lenny (Etch is the current release, Lenny will be the next one). The term Etch and a half refers to the work to make Etch run on newer hardware [1]. It’s a good project, but I don’t think that it goes far enough. It certainly won’t fulfill the requirements of people who want something like Fedora. I think that if we had half-way releases of Debian (essentially taking a snap-shot of Testing and then fixing the worst of the bugs) then we could accommodate user demand for newer versions (making available a release which is on average half as old). Users who want really solid systems would run the full releases (which have more testing pre-release and more attention paid to bug fixes), but users who need the new features could run a half-way release. Currently there are people working on providing security support for Testing so that people who need the more recent versions of software can use Testing, I believe that making a half-way release would provide better benefits to most users while also possibly taking less resources from the developers. This would not preclude the current “Etch and a half” work of back-porting drivers, in the Red Hat model such driver back-ports are done in the first few years of RHEL support. If we were to really follow Red Hat in this regard the “Etch and a half” work would operate in tandem with similar work for Sarge (version 3.1 of Debian which was released in 2005)! In summary, the Red Hat approach is to have Fedora releases aimed at every 6 months, but in practice coming out every 9 months or so and to have Enterprise Linux releases aimed at every year, but in practice coming out every 18 months. This means among other things that there can be some uncertainty as to the release order of future Fedora and RHEL releases. I believe that a good option for Debian would be to have alternate “Enterprise” (for want of a better word) and half-way releases (comparable to RHEL and Fedora). The Enterprise releases could be frozen in coordination with Red Hat, Ubuntu, and other distributions (Mark Shuttleworth now refers to this as being a “pulse” in the free software community [], while the half-way releases would come out either when it’s about half-way between releases, or when there is a significant set of updates that would encourage users to switch. One of the many benefits to having synchronised releases is that if the work in back-porting support for new hardware lagged in Debian then users would have a reasonable chance of taking the code from CentOS. If nothing else I think that making kernels from other distributions available for easy install is a good thing. There is a wide combination of kernel patches that may be selected by distribution maintainers, and sometimes choices have to be made between mutually exclusive options. If the Debian kernel doesn’t work best for a user then it would be good to provide them with a kernel compiled from the RHEL kernel source package and possibly other kernels. Mark also makes the interesting suggestion of having different waves of code freeze, the first for the kernel, GCC, and glibc, and possibly server programs such as Apache. The second for major applications and desktop environments. The third for distributions. One implication of this is that not all distributions will follow the second wave. If a distribution follows the kernel, GCC, and glibc wave but not the applications wave it will still save some significant amounts of effort for the users. It will mean that the distributions in question will all have the same hardware support and kernel features, and that they will be able to run each others’ applications (except when the applications in question use system libraries from later waves). Also let’s not forget the possibility of running a kernel from distribution A on distribution B, it’s something I’ve done on many occasions, but it does rely on the kernels in question being reasonably similar in terms of features.

• [1] http://wiki.debian.org/EtchAndAHalf


• [2] http://www.markshuttleworth.com/archives/150

Share This

"Here are three examples of rules that I think it's time to abandon. These particular examples are all about email. 1/ Top Posting [...] 2/ HTML Email [...] 3/ Reply-To On Mailing Lists [...] So, yes, the barbarians are at the gate. The lunatics have taken over the asylum. Good ideas have been crushed by the number of people who don't understand them. But there's no point in complaining about it. You just have to accept it and move on."

-- Pointless Battles For Geeks, By Dave Cross Unsurprisingly, given the above links to my site, I disagree with Dave Cross's conclusion, but I do agree with two aspects: battling is generally pointless and using hard rules about these things is unnecessary. I have those pages on my website so that I can point to them when their broken emails aren't handled as expected. I use some aspects of them as scoring inputs in my mail filters. I don't use them as rules and I try not to complain about them too often. Nevertheless, I still believe sending properly-trimmed plain text emails from a list-friendly email client is clearly best practice, to be recommended when someone asks why their email bad habits are causing them problems.

There's something about certian very visually information dense console apps that makes certian bits of my neural network that have been quietly developing for years very happy. rtorrent is the best example of this ever (excluding the unix shell as a whole). It starts out a utter incomprehensible mess, and I can just feel my brain reconfiguring over several hours so it can understand it at a glance.

                 Chunks seen: [C/A/D 39/10/2.98]   X downloaded  X missing  X qu
Peer list          0 7465767364 6546566456 7576655464 3676833757 5654376467
                  50 5565765774 4785647665 4565464565 5777557464 4865465737
Info             100 4435567665 4547636475 6756567736 5665345373 6356566756
                 150 5454855548 6766756846 4476585655 5566676764 5665754652
File list        200 3674466375 7474666747 8766756866 5646667647 6677676353
                 250 5557754755 6755756775 4476686458 6544656687 5777557555
Tracker list     300 5567666653 7675564686 6263555577 6467766443 8635577643
                 350 7586655656 7765373543 4454677664 6654665666 4344666556
Chunks seen      400 6565665554 7656544673 7636765776 3567556727 6466735257
                 450 6344246566 7364539664 5366744446 6577564574 3563656833
Transfer list    500 5765366475 6685663564 7466756575 5567636754 4425467555

This is the same part of my brain that is happy out in the woods following along faint game trails. It's a pity that we've been taught to think of this kind of UI as "hard" and thus "bad".

                 12.xxx.75.xx    0.3    0.0    10.2   l/ci/ci  0/0    21Azureus
                 67.xxx.58.xxx   0.0    0.0    0.0    l/cn/ci  0/0   100uTorrent
               * 67.xxx.222.xxx  0.0    0.1    214.2  r/Ui/ci  0/2    88   617Az
                 24.xxx.89.xx    0.2    0.1    26.5   R/Ui/ui  0/2    30   695uT
                 65.xx.113.xxx   0.0    0.0    0.0    l/cn/ci  0/0   100Azureus
                 24.xxx.38.xxx   0.0    0.0    0.0    L/cn/ci  0/0   100Azureus

Does this make your neural networks light up and say "more plz"?

                 Transfer list: [Size 13]
Peer list        620 [P: 2 F: 0]AAAAaAAAAbbBBBbbbbaC...DdCc.....
                  59 [P: 2 F: 0]EEEEEeeFff......................
Info             141 [P: 2 F: 0]GGGGGGGGgGGg.GG.................
                 618 [P: 2 F: 0]HHHHHHHHHHHHHHHHHHHHHhhhh.......
File list        696 [P: 2 F: 0]IIIIIIIIIIIIIIIIIIIIIJJIIIIIijjI
                  26 [P: 2 F: 0]VVii............................
Tracker list      56 [P: 2 F: 0]WWWWWWWWWWWWWWWWWWWWWWWWWIWwIwIw
                  93 [P: 2 F: 0]NNNNNLLLLNNNLNNNNUNNNtXXPPtUUYYP
Chunks seen      313 [P: 2 F: 0]zzww............................
                  96 [P: 2 F: 0]NzZZUNNXXNNNRRPPZZZZPxXNNnZZXzXP
Transfer list    633 [P: 2 F: 0]ZZZz..Zz........................
                 694 [P: 2 F: 0]YYYYYyyy........................
                 109 [P: 2 F: 0]pzzppxxpznnnppuuRRuuxzzrrnn.....

So one of the amazing things about the net is not so much how it makes lots of data available (actually that can be the nightmare of the net) but how it makes it available. And one of the more powerful forms of availability is in the form of visual mapping. So this new tool theNNDB Mapper is just one such visualization tool that I have to say, is pretty niftfy, though still a little green and beta at least in terms of its data. Take for example, the hacker map, which is pretty sparse right now but the great thing about it is 1) one can add data 2) and it generates useful data and links, for example, to profiles, such as this this one of Kevin Mitnick. Apparently, he is not linked to anyone, which is ironic (and just wrong) because he it is more correct to say was linked like to an entire generation of hackers, especially those who participated in the Free Kevin movement (and that could perhaps be another field?)..

Eric, I cann’t claim to 100% understand the situation but after glancing trough the logs of the discussions and of the patches the conclusion I came to was this - OpenSSL used supposed randomness of the uninitialized memory as an added source of entropy (interesting hack, but not an example of good coding as such). Valgring caught that problem and the Debian maintainer during a cleanup fixed it. Making such a fix can be considered a preventive step against possible attack vectors by poisoning the uninitialized memory. He took it up to upstream, they did not raise red flags, but did not quite merge the ‘clean up’ patch either. It fell through the cracks. The problem is that in the same file, in another function all other sources of entropy were being merged into the pool of randomness using exactly the same code line as the one code line flagged by Valgrind. The maintainer assumed that the second code line has a similar function to the first and commented that one as well. AFAIK that also did not show up in the emails to the upstream list. So we have:

• Upstream using clever hacks that rely on uninitialized memory having some randomness to it
• Upstream using same code and same variable names to describe different things
• Upstream having no comments in the code explaining the two things above
• Maintainer slightly over-generalizing a change
• A bug slipping trough the cracks in the review processes
• Another Debian Developer discovering the bug and recognizing its significance despite all of the above
• Debian project coming out and admitting all of the above and scrambling to get fixes out to its users ASAP

I am impressed by the swift action of the people involved in fixing this. And while I think everyone can find some lesson be learned here, I think this is another good example of free software in action. And I hope that in the aftermath of this we will find ways to prevent this from happening in the future without stifling our progress.

Helping a user run GNU/Linux programs on a proprietary system when one knows that the user has no intention of switching to the use of a free system is generally a bad idea. It is sometimes called "enabling" and is generally considered a waste of one's time.All the same, there are situations when one is faced with a Mac¹ with strict instructions not to reboot or otherwise "disturb" the existing system.The news in brief is:

Debian Etch runs with Xorg (vesa driver) under Parallels.
It installs using "mini.iso" for network based install.

Using this one can run all the GNU/Linux programs one needs. It "disturbs" the existing system less than things like "fink" or "MacPorts".Another alternative is "VirtualBox" from Sun which is free for educational use (and has an "Open Source Edition" too!).QEMU offers yet another alternative. Since it emulates all the hardware it is significantly slower. However, on current hardware, the speed will be more than adequate if you just want to use standard desktop tools from GNU/Linux.

Pros

1. Applications run in the environment they were designed for.
2. Speed is "native" for running computational code as there
   is no emulation going on for this part.
3. X applications work in full screen mode.
4. Full networking support.

Cons

1. Can only run in 32-bit mode.
2. Currently runs X in "vesa" modes so you don't have
   accelerated graphics.
3. Some keys like F10 and Ctrl+Alt Alt+Enter are trapped by
   MacOS so you may have to remap these keys if your application
   uses them. (For example, Ctrl+T instead of F10 for aptitude).
4. Possible delay in disk access since parallels provides an
   "emulated disk".

Not yet tried

1. Sharing files (Samba/NFS/SSHfs?)

---

1. Also known as Macintrash. ↩

One of the biggest questions which has to be answered when one starts a blog is what is the blog engine going to be? (Using a readymade hosted solution like LiveJournal or blogspot is for wimps.) Since the blog is going to be on IMSc's "people" server, there are a lot of choices. The scripting language for the engine is usually a choice between PHP, Perl, Ruby and Python (though with nanoblogger, even a POSIX shell is possible!). The gem looked tempting for a while. Eventually, I realised that the engine is far less important than the entries. Start writing the entries with some engine. It should be "relatively easy" to migrate the entries to a different engine. In some sense, having put essays on the web in the past, a change of engine (from wetware to software) is what I am doing anyway!So the temporary choice is the snake --- think of this is as the snake in the monkey's shadow. Specifically, this blog is currently powered by pyblosxom.

Configuring pyblosxom

StructureSince everything on this blog is meant to be public, all of its data and configuration information is kept in public_html. As far as I can see this causes no security issues --- but if some turn up then it may change!Current directory structure is

blog/ ->
    config.py   Configuration file
    drafts/     -> draft entries
    sources/    -> sources where required
    entries/    -> format of entries is "string"-"date"
    flavours/   -> not tutti-fruity!
    plugins/    -> actual plugins used
    style/      -> stylesheets and images
    index.cgi   -> pyblosxom.cgi renamed
    .htaccess   -> for apache2

Any and all of these files (except drafts) can be examined by extending the URL http://www.imsc.res.in/~kapil/blog by the file or directory name.The development of my web pages is done on my user machine and these are synchronised with the "real" web pages using unison. The local copy of my web pages is managed using git.The plugins were downloaded from the pyblosxom web site.The flavour was downloaded from snarfed.org.Since everything about this blog is visible, there is probably no need to document things further.

Minor cribsI couldn't find anyone who had documented how they had setup their blog using pyblosxom. This meant that it took a while for me to set things up.The apache2 web server behaved rather differently from mathopd. In particular, the latter does not need or use .htaccess files and just declaring that index.cgi was the a CGI script and was the "index file" for the blog/ directory was enough for the blog to start working. However, I had to struggle with apache2 as it needed "Rewrite*" entries in the .htaccess file in order to pick up the style sheets and the blog pages.

Following Martin's post on the subject, since I created my prompt, I've updated quite a lot. It only showed the branch before, now it does so even when I'm on a detached head or something, and also shows when I'm in the middle of a rebase or a merge. You can see the zsh-fu for this. For example, when I'm in the middle of a rebase on my paid work repository, it looks like:

 (10:34) <~/dev/mmsx master <rebase -i>> 
 [artemis]                                                 (~/dev/mmsx/Build/)

With nice colors I cannot really show without a screenshot that I'm too lazy to do :) This is a recent addition that I shamelessly took from the contrib bash prompt in git-core package. And to be frank it's really needed, because it's cheap tests (basically looking for magic file names) and that it can tell you if you forgot to end a rebase or a merge, which can happen if you have been disturbed in the middle of it by a colleague for example. I liked the '*' idea from Martin to show if the tree is dirty. Sadly it's not an option. Martin, to do that, you can do:

 unclean=
 git diff-files --quiet && git diff-index --cached --quiet HEAD --   unclean='*'

But this is a very expensive operation. On the glibc git repository, it takes seconds with cold cache (and it's not very surprising because it basically has to stat(3) a lot of stuff). And not having a shell for seconds is a bit extreme. PS: I know my prompt only supports git, but:

1. I barely care about other VCSes as I only use git and sometimes svn for packaging ;
2. when I have to use svn it's for cheap stuff where I don't really need the prompt help.

I was excited by Pierre s idea to add Git branch information to the Zsh prompt and even more so when I saw Mike implement support for multiple VCSs. Unfortunately, Mike s a Bash user, and so I took it upon myself to port the idea to Zsh. The file 60vcsprompt is sourced from my .zshrc, which sets psvar[1] through psvar[3]. Those are then used in 80prompt (also sourced from .zshrc) when setting $PS1. My prompt follows the same principle as Mike s and puts the branch name at the repository root location in the repository path. In the following example, ~, ~/code, and ~/code/netconf/netconf are three separate Git repositories, while ~/code/unionfs-fuse and ~/code/unperish are maintained with Mercurial and Bazaar respectively:

lapse:~ master % cd code
lapse:~/code master % cd netconf 
lapse:~/code master netconf% cd netconf
lapse:..e/netconf/netconf master % cd src
lapse:..etconf/netconf master src% git checkout no-threads
Switched to branch "no-threads"
lapse:..nf/netconf no-threads src% cd ../../../unionfs-fuse 
lapse:../unionfs-fuse hg:default % cd ../unperish
lapse:..unperish bzr:unperish@159 %

You ll notice that unlike Mike s prompt, mine s limited to a maximum length of 25 characters. However, the repository root path is kept at least 10 characters long, so the prompt might get longer than 25 characters if you descend deep into a repository s subdirectories. I couldn t easily figure out how to add support for other version control systems, so if you do, please feed back the patches! And the same goes for suggestions and improvements. One of the next things I am planning to implement is an indicator for when your working tree contains uncommitted changes, e.g.:

lapse:..etconf/netconf master src% touch foo
lapse:..tconf/netconf master* src%

So watch those files. NP: Gazpacho: Bravo

Those of you who know me might be aware that I have a fondness for cool and neato hardware which is possibly a tiny bit above that of the average geek. I have more nodes on my ThinNet than on my TP network, for example. Back in February, I had the random idea to call SGI Norway and ask them if they had anything they were about to throw out. I knew that they were working hard on moving from MIPS to Itanium, so perhaps some hardware was disappearing along the way. Oslo didn’t have anything, but when I called Bergen I found out that they were about to throw out a large amount of hardware, including some really nice boxes like an Onyx2 deskside workstation, and some Origin systems. For those of you who aren’t into SGI machines, I can sum it up by saying that these are really cool computers: Big, noisy as hell, heavy, and in terms of performance, nearly on par with a normal desktop nowadays. In short, fantastic educational toys. So, this was in February, and we were making arrangements for transport to Oslo, until one day I stopped receiving mail. I didn’t think much of it; email is asynchronous, and I figured he was busy doing something else. I was certainly very eager to get at the machines, but didn’t feel like nagging at him either. Finally, recently I sent him a polite reminder, and I get the rather rude response: Diagnostic code: smtp;550 cuda_nsu 5.7.1 <****@sgi.com>: Recipient address rejected: THIS USER IS NO LONGER WITH SGI. CONTACT help@sgi.com IF YOU NEED INFO. I tried calling the Bergen office, but the number was out of service. So I call the Oslo office - and I am told the Bergen office has been shut down, and all the equipment has been thrown out. This is the bummer of the century, for two main reasons:

1. I’m not getting cool hardware that I was totally looking forward to getting,
2. Great hardware has now been ground up for use in car tires.

For some odd reason, the second reason is infinitely more grating than the first. I feel like a museum curator being forced to watch someone feed a Picasso painting to a paper shredder. This really, really screwed up my day.

Tonight, I converted the XML-Security-C package from Stanford's internal Subversion repository to Git and packaged 1.4.0, but unfortunately I can't upload yet because Xalan needs to be updated to build against the latest Xerces-C. (I'd NMU, but I should not be taking on more things right now.) This is the first of the three current Shibboleth packages, and I hope to tackle OpenSAML and Shibboleth SP tomorrow or Wednesday. An Alioth project for Shibboleth packaging now exists and the pkg-shibboleth-devel mailing list is being created. I've requested Git space on git.debian.org, and as soon as that's set up, I'll push the three existing repositories. Shibboleth 2.0 packaging will add at least three more packages (OpenSAML 2.0, Shibboleth SP 2.0, and another XML support library), possibly four if we have to package log4shib, but I still hope to use the standard log4cpp. And then there will be another couple of packages once the group packages the Shibboleth IdP and WAYF, which I hope someone with more Java experience can tackle. One of the main goals for getting this set up is so that people other than me can do more of the work so that I can work on other things. Like, for example, krb5 packaging. Bastian Blank started using the Kerberos server packages and posted a whole flurry of bugs, a couple of which are (somewhat arguably) RC, and I'm trying to absorb that and fix the ones I can ASAP. OpenLDAP is in desperate need of attention, but I don't feel as much responsibility for that. I said going in that I'd only have time to work on it in fits and starts, and just because no one else has any time to work on it either doesn't change that. However, I do need to find time to post an RFH, since it really needs an active maintainer, and I'll at least find time to upload the upcoming 2.4.9 release before lenny freezes more, since the current version has a ton of bugs. Frank Lichtenheld has been doing a lot of the work on Lintian lately, which is wonderful, but I still have at least 10 bugs and patches that I want to deal with as soon as possible for it and get another upload out. We're well over my magic threshold of 100 bugs, and a lot of the stuff that's in the BTS just needs to be reviewed and committed. And then there's Policy, which is just about ready for an upload, and where I owe several responses on bugs and discussion threads. It's in the best shape right now of the stuff on my hot list, but there really should be a new upload in May, and preferrably in the first half. The current goal is to get the two other RC bugs against the Shibboleth packages dealt with in the next couple of days, then do a krb5 upload before the end of the week, and then Policy and Lintian are back at the top of the list, insofar as work lets me concentrate on anything else. BTW, I have to say, the more I use Git, the more I'm becoming a convert. I still think I'd rather use quilt if I have complex merges. Ironic, given that that's supposed to be Git's strength, but I have fifteen years of experience merging patches and I really understand how quilt works. That may mean I'll change my mind when I become even more familiar with Git. But for the typical package with a small number of divergences from upstream, I'm getting the hang of managing those on branches and I think I like it. I've yet to see a good reason to use rebase instead of merge, though, at least as part of a Debian packaging workflow. So either I'm still not getting something, or many of the people using Git are way more obsessive than I am about their branch merge graphs (which is weird, because I tend to be remarkably obsessive about things like that). I suppose if I ever get to using git bisect, that may change my mind.

Apparently it is a political suicide in USA to make people think. Obama’s ex-preacher Jeremiah Wright is the most sane US preacher that I’ve ever heard. If you read or listen to his sermons in full length and go beyond the Fox News presented soundbites, he is 99% right. The US has long been doing extreme violence to countries around the world, so it is no wonder that such violence came back to US. Russia had the same problem with Afghanistan and Chechnya. If you try to put yourself up as ‘world police’, some people will disagree. Then he damned America’s _government_ for letting Americans down by not caring enough about the people. The only thing that I would smile and shake my head at was the statement that American government invented HIV. I doubt even they have such resources. Read this transcript for example, the sermon is just a great sermon that urges the people think for themselves. And what people put this sermon down for? Using a word ‘Negro’? Having an issue with some specific politicians? He is a pastor, pastors are meant to have a strong opinion a vision and to project that for the congregation to see. “I’m not going to stop thinking just because I love Jesus” is the summary of that sermon if you ask me. And that is the most positive religious message I’ve heard from that side of the pond for a long time. You can agree or disagree with rev. Wright, but you can still go to his sermons and enjoy it because he makes you think. I would talk to most racist bastard, serial killer or even Hitler himself if he could tell me something that would provoke my mind to think and thus improve myself. You don’t have to agree with people to benefit from speaking to them or from listening to what they have to say. Even if someone is 100% wrong and misguided, if his speeches can make you think of some good idea, it is worth it. And rev. Jeremiah Wright is not 100% wrong, from what I’ve heard he is on the money most of the time and only some of his statements I would disagree with. And he is a great speaker. If I would have not been an Obama supporter, I would become one because of Wright. Or because of the very accurate observation by Obama that in hard times people in rural USA cling to trusted values such as guns and religion. There is nothing wrong with that. I am amazed how such true statements can be so turned by US press and can actually reduce Obamas rating. I was starting to believe that not all is lost in the US, but if Obama does not get elected, well … it is all lost in the US. PS. How this related to Debian? Or software is the best choice for the society in the long term. However, that is only true if we assume that people behave rationally and use it, improve it and contribute back improvements for the public good. If the general population is made weary of rational thinking and starts thinking in soundbites and slogans, then there is a ceiling to free software expansion and the whole free software movement can eventually be crushed in the interests of homeland security with strong corporate commercial software backing. (Only government approved software is allowed to run with mandatory government approved back-doors and hardware verification to prevent cyber-terrorism or something, thus also forbidding any compilers and interpreters outside government controls.)

JROBI, a chess blogger, on energy policy:

A large study in Europe concluded that it takes more gas and oil to produce a bottle of bio-fuel than it does to produce a bottle of gas. What does this mean? It means that Bio-Fuel is more damaging to the environment in the long run, and on top of that it is driving up the cost of basic food supplies. Millions and millions around the world in a number of countries are unable to afford the rising food costs for basic staples like Corn, and for what? If Bio-Fuel is not better for the environment, why are politicians and environmentalists getting behind this growing industry? I think it’s because it seems to be the “trendy” thing to do, and we all know what happens when the media promotes a new trend. We get tons of media coverage telling us why it’s a good thing, and hardly any coverage of the negative impacts. Already people from the Bio-Fuel industry are getting on television shouting out that there are many factors contributing to rising food prices, trying to deflect the fact that their destruction of food to fuel vehicles is the main culprit.

Actually, I suspect the emphasis on biofuels in the USA and Europe has to do with the fact that it’s the only alternative to fossil-based motor fuels proven to be sustainable and scalable:

The success of FFFVs, together with the mandatory use of E25 blend of gasoline throughout the country, allowed Brazil to get more than 40% of its automobile fuels from sugar cane-based ethanol in 2007.

I see no link to the European study in question, but previous studies have suffered from various faults; for example, the assumption that trucks transporting fuel cannot themselves shift to biofuels. I’m sure better analysis of the study is on its way. But that’s not the most interesting thing, to me. More interesting: my general impression that a lot of the climate-change hysteria is just that. If we hear what science seems to be telling us about the environment, and we think that something needs to be done, then we should do things that will actually work. One thing that really works is conservation: use less of the bad stuff we’re using. But we’ve done quite a bit on that front, only to hear that much, much more is required to make a difference. I’m not sure there’s much, much more benefit for us to realize in conservation, at least in the short term. So, to make a real difference, we have to make more radical changes. Can we change our motor fuel? Sure; starting with something that pollutes less, and that even absorbs some of that same pollutant in its production, sounds like a winner. JROBI, again:

It makes no sense whatsoever to create Bio-Fuel when there are much better options on the table - for instance Hydrogen vehicles. When was the last time you heard someone on the news talk about Hydrogen initiatives?

I hear it every so often. But most talk, today, focuses on the very real problems with hydrogen as a motor fuel. There are many; just look at the discussions of hydrogen fuel tank technology for a sample. But one of the biggest problems is that of developing an infrastructure for delivering fuel to the customer. No one talks about the problems of setting up an ethanol infrastructure. We already have it. Brazil has demonstrated that the current gasoline infrastructure can easily be adapted to deliver ethanol instead, and that there is a viable migration plan for gradually moving people off fossil fuels. Now, this isn’t to say that the world of ethanol is hunky-dory. It’s arguable that, while ethanol may be sustainable, the corn-based system the USA has adopted isn’t. Some people are talking about sensible tweaks that may solve the food problems while continuing to support biofuels–removing our silly tariff on Brazilian ethanol, for example, or developing alternative feedstocks for ethanol production. The problem is that hysteria seems to be breeding hysteria. Global warming is so severe, we are told, that we need solutions, and we need them immediately. So we develop solutions we can use immediately. But no! These solutions cost; we need something else, and we need it immediately, and we need it cost-free. Practically, this kind of insistence on perfection–that we deploy solutions with no drawbacks, only benefits–has the effect of dampening our enthusiasm for environmental solutions. We tried, our leaders will tell us, but nothing was good enough, so we gave up. And so, rather than do something that helps, or even something that lays the foundation for helping, we continue our use of fossil fuels. Perhaps ethanol is the wrong solution. But if it is, we should resign ourselves to the inevitability of the future, as foretold by science, or fervently hope that the global warming deniers are right, because other solutions will arrive too late to do much good.

I made an emergency release of the chronicle blog compiler yesterday, after noticing that it was truncating titles containing periods. That was a bit of a mea-culpea moment, but I guess mistakes happen. The new release is in perfect shape for Lenny, and now includes two new scripts installed into the examples/ directory:

• The RSS-feed importer I mentioned prviously.
• A quick hack to detect duplicate post/entry titles.

The latter was applied to my own blog, and I discovered several duplicates. I guess my film quotes having only a limited source collection to work from could also include duplicates - so I've updated my Makefile to only build and rysnc my blog if there are none detected. (In many ways that films site is the precursor to this blog; it uses a collection of text files, one per film, and generates a cross-linked HTML output of film entries. Sadly it is out of date, because entering titles is a real pain..) Chronicle Comments

I'm pleased with the comment process now though, the CGI comment submission script simply archives each submitted comment into a "comments/" directory on the webserver. There a cron-job passes each one through a bayasian filter and moves the file(s) to either "comments/good/", "comments/bad/" or "comments/unsure/". When I come to rebuild the blog I rsync the "comments/good" directory to my local machine, rebuild and then rsync the output back to my remote webserver. (On a single machine this would be much simpler process!) I've imported my blog source into a mercurial repository, so the client-side is consistent. I have a bad habit of making new postings from wherever I happen to be and having a central repository will make that less prone to diaster. Just running "make steve" against the Makefile is sufficient to rebuild everything and sync it to my live system.

ObQuote: Kalifornia

This Thursday, we did the largest part of the final phase of Stanford's Kerberos v5 migration. Our account services database system has now moved from Solaris to Debian, from Oracle to MySQL, and from Kerberos v4 to Kerberos v5. This has a lot to do with why I've been exhausted this week (I've worked over 50 hours and will probably do some more work this weekend). We're at the very end of a project that's been ten years in the making. Stanford first deployed Kerberos v4 in 1992 using the Transarc AFS kaserver as the KDC, with the full intention that it would be temporary until we moved to some Kerberos v5 implementation, probably Transarc DFS. Sixteen years later, we're finally turning off the OpenAFS kaservers and going to Kerberos v5 exclusively on May 15th. Various people at Stanford have been working on this project fairly continuously since the late 1990s, particularly Booker Bense and Roland Schemers (Roland has now moved on to other things). I started taking responsibility for it about five years ago and it became my major project three years ago when we finally got funding to finish it. It's the biggest project of my career to date. When we finally got funding and I was able to start working on it for a substantial percentage of my time, I went into the project with the intention to solve the large problem and do it right whenever possible. That meant trying to enhance the software where it was deficient rather than working around it, solving the large problem, and trying to improve the world and not just Stanford. Stanford University has been extremely supportive of that goal, and as a result, what's come out of the last three years is a bit staggering to look back on. I was feeling rather down earlier tonight, mostly because I'm exhausted. It's been a very intense past six months, a very tiring week, and I've not had the energy and attention for other things that are important to me. But I started thinking about everything that we've been able to do as part of this project and realizing just how big this is and managed to cheer myself up a lot. Five years from now, ten years from now, thirty years from now, this is still something that I'm going to be proud of. That's a great feeling. I was going to write up more specific details of how to tackle a project like this and do it right, listing some of the concrete results as examples, but yeah. Exhausted and low energy. Maybe I'll just do it over the next two weeks as a lead-up to May 15th. Immediately after that I'm going to the AFS and Kerberos Best Practices Workshop in part to talk about many of the things that we've written. And then, for the first time in three years, all of my major projects will be cleared and I'll have several months to catch up on everything else, recover, and gather my thoughts before deciding what the next big project will be.

Lets go on your nerves again with a ftpmaster post about work I’ve done in the recent past.

• The already mentioned move of britney. And it seems that this did help, so something that should have been done way earlier already… Oh well.
• Enabled a few triggers for the qa user on merkel. While I already mentioned them in my first post, there is a third one now. The qa user now gets notified at the beginning of the dinstall run, so the pages can show why packages might not be visible in the archive and in incoming right now.
• Split up the insanely huge mbox files we had as the ftpmaster mail archive. They got that big because the monthly cronjob rotating them got disabled when the ftp-master archive went away from raff, and noone bothered to reenable that job. Done now, so we are back to a manageable size. (Even if 90% of the content is spam, no matter how much greylisting and blacklists are activated :( ).
• Just committed a largish change thats nothing more than a cleanup. Someone in the past did love to not care about tab versus spaces for the indentation, so dak code ended up with a mess of tab/space mix for indentation. Which is not nice, one style is way better. Fixed that with the help of reindent.py, which one can find in Pythons examples.
• Got the code on security.debian.org to use the latest dak revision from ftpmaster. It was a few commits behind, but worse - it did have one dak tool changed in a way that it only worked on security, no longer on ftpmaster. Also some files that clearly have been copied by scp -r or rsync, but not using the VCS that should be used for it. Args. Brrrr. The “fix” was a simple check if that value the code wants to use from the config file actually exists, instead of blindly assuming it will always be there.
• Generally looked around on security.debian.org, to get more familiar with the setup there, so I can work on a request I got from some members of the security teams: Run a queue daemon that manages the upload of the packages from the security archive to the normal archive (so they end up in proposed-updates). Currently it is done with some lftp calls within the script that does the security install, and that seems to break every now and then. Which means security uploads to proposed-updates randomly go MIA. Using a queue daemon and then only locally copy files into its dir should make that way more reliable.
• Used a tool from Dato and enabled bzr commit mails for the ftpmaster dak tree, they are now going to the dak list.
• Modified the cronjobs that are expected to have output to log that into files in a new logdir. Also modified the cron.daily AKA dinstall cronjob to be more verbose about the actions done. The directory is synced to merkel too (log/cron in the ftp.debian.org tree), so every DD who wants to can look at them now.
• I also merged a small patch from Thomas, fixing/unifying some wording in reject messages.

Comments: 3

Recently I had to deal with POST requests on mod_perl 2 on a custom XML-RPC webservice server we have been developing and maintaining over the last year (we are actually moving it to REST some time in the middle future). It was time to move some of our base modules from a couple of CGIs to the power and muscle of mod_perl. So, while dealing with POST requests (like XML-RPC is actually specified) I faced the issue that I wasn t being able to get the POST data on the PerlResponseHandler phase. I was probably not reading it correctly, I got deeply into it and even read some libapreq source code. All I was getting was some nifty message like not supported . After reading some documentation (and, to my own shame, the Apache2::Request front page pod) I found out that the usual:

    use Apache2::Request;
    $req = Apache2::Request->new($r);
    @foo = $req->param("foo");
    $bar = $req->args("bar");

will not work on your POST if the content type is different than application/x-www-form-urlencoded or multipart/form-data. For example, say, XML-RPC! So, in case you actually need to read POST data on your mod_perl 2 handler where it s not a form or a file upload, you can read the data manually (I didn t find a simpler way to read the data, actually):

sub handler  
my $r = shift;

return Apache2::Const::HTTP_BAD_REQUEST unless
		$r->method eq "POST" and
		$r->headers_in-> 'Content-Type'  eq 'text/xml' and
		$r->headers_in-> 'Content-Length' ;

my $length = $r->headers_in-> 'Content-Length' ;

my $xml;

while ($length)  
		$r->read($xml, ($length [less than symbol] 2048) ? $length : 2048);
		$length -= length($xml);
 

	...

And you are ready for booze.

John writes about PDFs and the problem of embedded fonts that are missing due to restrictive licencing. Font designers usually distinguish between "embedding" and "using" a font, prohibiting the former in their licence agreement. The difference is that when a font is "used", all information about the text is lost, and only rendering instructions remain (usually a set of Bezier curves). PDF allows you to annotate a rendered page in various ways, among other things, it allows you to specify that a certain region should be interpreted as a specific text. For example, OCR tools working with PDF files start with a file containing just the image data, and add the textual information, so you end up with a file that looks like the scan, but has selectable text. While that is certainly not legal advice, so far it has been considered acceptable to ship a PDF file with pre-rendered text and an annotation that facilitates copy&paste (which you also need with embedded fonts to allow selections to see ligatures like " " as two distinct characters). These files may be slightly larger than they would be with an embedded font, but have the advantage of looking exactly the same everywhere and at the same time, being trivial to render as the rasterizer does not need to understand anything about text layout. tl;dr: if you have a document that people should be able to see exactly the way you intended it, please include the text curves if you may not embed the font directly, and provide selection hints.