At The MIT Kerberos Consortium s 2010 conference, Josh Howlett and Sam Hartman delivered a talk on Moonshot. Slides should be up in a day or so. We reported on status and gave a brief overview. The new material was apropos for the venue. At the bar BOF back in March at IETF 77, we received several comments on Moonshot s limitations. It doesn t work well for services that require rapid authentications for multiple requests. There s not a good story for use when a Moonshot service needs to contact another service. There isn t a good standardized mechanism for mapping in domain-specific policy. We presented a proposal that Luke and Sam developed to optionally provide a Kerberos ticket as part of moonshot authentication. This scales from a service that simply generates its own service tickets all the way through resource domains that have many services and complex policy and provide the client a TGT. Clients can implement the feature in order to achieve better performance. Server can implement the feature in order to get delegation support within a resource domain and to get policy mapping. Luke has prototyped a version of this service involving a service ticket. We plan on briefly mentioning a desire to have extensible fast reauthentication support at the ABFAB meeting in IETF 79. However in the interest of getting the working group off to a good start we re going to focus on the well understand parts of the system and formally propose this extension after IETF 79.

Three thousand lines of code is not a huge program, but it is enough to get a pretty good feel for a language. Now that I've completed my first real Haskell program I feel that I've gotten over several of the humps in the learning curve and am starting to get a good feel for it. Actually, I've written closer to five thousand lines, since there were several big refactorings. One was when I stopped manually threading my program state around and added a StateT monad. I did know from the beginning I would need one, but it seemed easier and a better learning exercise to let the program start out with a vesigial tail and gills before growing up into a modern Haskell program. (I suppose it's still written in baby-Haskell, really..) Another refactoring came when I realized I needed to use a custom data type, not String, to represent keys. That was a great experience in type-based refactoring. Being able to keep typing ':make' and landing on the next bit of code that needed fixing was great, and simply adding that one type exposed several non-obvious bugs. I found myself writing code that is much more solid and reusable than normally comes easily. And yet it's also very malleable. Actually, pulling out better data types and abstractions can get a bit addictive. When I realized that I had a similar three-stage control flow being used for each of git-annex's subcommands, and factored that control flow out into a function that used the 3 data types below, I felt I'd gone down that rabbit hole perhaps far enough for now.

type SubCmdStart = String -> Annex (Maybe SubCmdPerform)
type SubCmdPerform = Annex (Maybe SubCmdCleanup)
type SubCmdCleanup = Annex Bool

(That will allow for some nice parallelism later though, and removed dozens of lines of code, so was worth it.) Since git-annex is a very Real World Haskell type program, there is a lot of impure code in it. I could probably do better at factoring out more pure code. I count 117 impure functions, and only 37 pure. Anyhow, from my perspective of a long-time perl programmer, some other random impressions..

• ghc --make is handy, but every time it spits out a new 13 mb executable I can feel my laptop's SSD groan!
• It was surpisingly easy to get into nasty situations with recursive dependencies between the 19 haskell modules I wrote. Sometimes solving them was really messy. I lost hours to this. More time than I've lost to the problem in all other languages combined over 15 years. It's not clear to me if it was due to the overall design of my program, or if Haskell's types tend to encourage this problem. Or if there's some simple "please let me have recursive dependencies" switch to ghc that I missed..
• I'm used to being able to use man to get at mutiple books worth of detailed documentation for perl, and work easily offline or with limited bandwidth. With Haskell, I spend much more time searching online for documentation than I an comfortable with (although Hoogle is pretty neat). And the haddock-produced documentation is often pretty sketchy. The saving grace is that the source to any library function is a click away, and tends to be very readable.
• I'm used to being able to use pretty much any Unix syscall by name from perl: mkdir, chmod, rename, etc. In Haskell, there is a Windows smell to the names, like createDirectoryIfMissing and setPermissions. And there are pointless distinctions like renameFile vs renameDirectory. These long names are not memorable and I have to look them up every time. Most of POSIX is available, but it's scattered amoung many disparate libraries, and I can't find an interface for sysconf(3) at all. There is a certian temptation, that I am so far resisting, to make a library for C/perl refugees that exports the sane Unix names for everything.
• Anything involving the IO monad, or probably most monads, has a certian level of syntactic clumsiness about it. Compare:

  if ($flag foo  && length $l = <>)  
  

  vs

  foo <- getFlag "foo"
  l <- getLine
  if (foo && not $ null l)
      then do
  

  When writing lots of impure code, that got old, and while I could use ifM, or make up some other similar thing, its syntax would also be somewhat clumsy.
• The fixity levels for a lot of stuff seems a bit off. I too often found myself writing error $ "foo: " ++ (show bar) or return $ Just $ ... (Still a lot better than Scheme thanks to $!)
• I've leveled up a couple times now, but this particular video game seems to have more levels going up and up, forever. Can't even see the top from here!

Like you most likely know, I've been at the openSUSE Conference last week. I've been representing Debian through a talk about Debian - The Project and its Resources, covering services and resources that are usable and useful for other distributions too. The invitation to submit talks were sent to various projects and distributions because of the set topic on Collaboration across Borders for the conference. My impression was that very few distributions have followed the invitation, I am only aware of some Fedora people that have attended. Vincent Untz from GNOME gave a combined keynote with Cornelius Schumacher from KDE. The abstract of my talk and the slides are available from the conference website. The talk was well received and interested people also asked about some of the services like how our buildd network works, or whether the screenshots service has support for localized images too, or if it would be expandable for more general usage of non-Debian based distributions. One topic raised was with respect to the whohas tool: A mapping layer/tool/database for package names. We divert from upstream naming schemes for consistency reasons, like with our library, Perl and Python naming schemes. Other distributions have similar approaches but a different naming scheme. To make inter-distribution tools really useful it would require to have some layer that is able to map the package name from one distribution to another one. One idea for it that was thrown in was to use upstream homepage, but not all packages do have proper homepages, some only live in some git repository, others would either like to link to sourceforge's project page instead of the project's homepage, and other tricky issues. Something unique is clearly called for here and needed to get stored in the package's metadata. Apart from my talk and the occasional mentioning of openSUSE is on my desktop, but I run Debian on my server I noticed that the beverage of openSUSE is their own beer. Taste is a tough topic, personally I rather prefer our Debian wine. And I missed the chance to seduce them into playing Mao, I forgot to bring playing cards along. I hope to get a second chance at anoter event.

/debian permanent link Comments: 3

As requested by our beloved leader, here s my report on this year s Google Summer of Code 2010 Mentor Summit. The summit took place in Google s HQ in Mountain View, California and was what they call an unconference . In the beginning of the unconference the attendees had 10 minutes to write down proposals for a session and put them on a huge white board. After that, every attendee voted for the sessions in which he was interested by putting a small sticker on the proposal. This way we could evaluate the interest on a specific topic and schedule the sessions to the differently sized rooms according to their interest. Neat trick! The the rest of the summit was basically like a conference: we had 11 tracks filled with 5-6 sessions per day, which ran from 10:00 to 17:30. Google provided shuttle buses from our wonderful hotel to the googleplex in the morning and back to the hotel in the evening. The hotel was very nice, it was equipped a swimming pool and a hot tub (both outside), which was extensively used in the morning and of course during the night. Speaking of the night, Google sponsored Thai food in the first night and ye olde Pizza and Beer in the second. Some of the attendees brought their keyboard, guitar, bass and electric clarinet, and where playing some sweet ass Jazz and Pop (on special request ). The sessions itself where not quite what I expected. I expected some proper talks about specific topics, but it was more like and open discussion round where everyone was invited to participate. For some topics it worked pretty well, for others not. It was still totally worth coming, since you meet a lot of like minded people and there is plenty stuff to talk about. Of course I was very exited to see the googleplex. And the rumors are true: the googleplex seems to be a very nice place to work. They provide free breakfast, lunch and dinner for their employees. They have micro kitchens everywhere where you can grab free snacks and beverages. There are lot s of toys lying around, they have bicycles everywhere on the campus which you can just grab to move from one building to another. There is a dinosaur on the yard, a spaceship over the stairwell and lots of other things (testing on the toilet, anyone?), and we have probably only seen a small fraction, since we were not allowed to leave the restricted areas for the unconference. Despite the horrible yet lag (leaving Germany on Friday morning and leaving San Francisco on Sunday night), I m glad I was able to attend to the summit and can only recommend aspiring mentors and students to participate to the next summer of code!

Email. We get a lot of it. We filter out the SPAM. Then if we're well-organized we file it away into folders as it arrives. (To be fair some people use priority settings such that all mail stays in their INBOX until they're "done" with it. I've never had the patience for that kind of behaviour.) One problem which I often encounter is wanting to have email be delivered, archived, and stored, but I don't want to read it. Yet when I see a folder in my mail client which has unread mail in it I cannot unsee. In my case I deliver mail to folders as it arrives via either Exim's filter language, or procmail. Procmail blows goats on the whole, but it is common and available everywhere - I just don't trust my own DSL enough to rely upon it (which is a dangerous sign). So, how do you deliver a new mail to a Maildir folder, and mark it read at the same time? Well you can be na ve and do what I did which is to invoke formail to add "Status: ro" to the header. Unfortunately that's insufficient to mark a mail as read when viewed in mutt. When an email is stored in a Maildir folder its status is encoded in part of its filename - which is why you'll have files like:

new/1288039894.28406_3.steve.org.uk
cur/1283711971.11157_3.skx.xen-hosting.net:2,S

The latter file has been Seen. So to mark a message as not-new you need to do two things:

• Save it to ./cur/ not ./new/.
• Append the appropriate flags to the filename (generally :2,S).

I've seen some horrific shell + procmail code to do the job, but the simpler version is:

:0
*^(To: From:).*root@
*^X-added-header:.*debian-administration.org
  ~/bin/read-to-maildir .machines.debian-administration.org/

Similarly you can use Exim's filter language to do the same job:

# Exim filter
if $h_to: contains "hostmaster@" then
    pipe "/home/skx/bin/read-to-maildir /home/steve/Maildir/.hostmaster/"
finish

Cute. Obvious too? Maybe not to me. ObSubject: Why did you murder someone, Raymond? - In Bruges

For people who do not follow the Merkaartor mailing list: Regulary updated development snapshots are available in experimental again. Please report bugs, either in the Debian BTS or in the upstream bugtracker. Latest addition to Merkaartor is a plugin to support the French Cadastre peoject.

Sunday. A good day to take some time to myself, and get some much needed excercise. And the pseudo-meditation that comes with it: That's one of the factors I love the most about cycling. It's possibly my favorite way of just idling, of allowing for the time to pass without me noticing, just watching the movie of life, watching the world go around me with very little effort. I just love cycling. I have posted here several mentions of the Ciclot n: Every month, the last Sunday, our city government closes a ~32Km circuit to make it for cycling (and skating, and walking, and jogging, and other human-powered locomotion) use. Whenever I am in the city and have the possibility, I do the Ciclot n. Two months ago, I even did it twice in a row, for a total of almost 70Km. But this Sunday was not the last Sunday of the month! So, why am I writing this? Well, because when I left home today at ~10AM, I thought it was. Only when I reached Circuito Interior and had a this is full of cars moment, I remembered October has 31 days. And next Sunday I'll very probably be in Oaxaca city (for the Hackmitin 04h4ck4 2010). And I didn't want to miss the fun of a nice Sunday ride! I don't care if streets are closed just for bikes I am used to riding in traffic. Of course, this meant I would not take the central lanes in Circuito Interior on the way back, taking the ~10 bridges (which are quite tiring, to be honest!). It also meant I was free to make my own route. So, I thought, a nice small ~20-30Km ride will do. I'll just get to Eje 2 Sur or so, and back by another route (i.e. Plutarco E. Calles or so). But as I got closer to Reforma, I realized that every Sunday (independent of a Ciclot n), Reforma is closed for the "Paseos Dominicales" - Shorter than a Ciclot n, but still a nice way to share the streets with other cyclists. So I headed for Reforma. Part of Reforma (from the ngel roundabout westward) was closed due to a precisely cyclist race. But from there eastwards we had the street for us. Good, lets head downtown. The surprise came when I saw that it was not only towards downtown, but passing Av. Ju rez, Reforma continued to be closed just for cyclist use northwards. Great! Lets see how far does it go. The first surprise there: A couple of days ago, it was the "Noche de los Alebrijes". The alebrijes are a beautiful and very elaborated Mexican typical handicraft, depicting psychodelic dragons or dragonic representations of animals. Of course, alebrijes are usually ~30cm long... But here we had huge ones! And verrrry elaborated! How much so? You tell me. And as I started off... A nice thing about cycling is that you get to meditate. My body continued cycling, taking care not to run over the kids that are not so proficient, stoping at each red light... At some point, my mind entered back the body, and I realized we were passing by Tlatelolco (ok, to be honest, the picture was taken when crossing southwards, on my way back), about to cross Circuito Interior on the North, in that symbolically very strange place where Paseo de la Reforma (a long, beautiful avenue named after the war that marked the complete separation between church and Mexican state, in 1860) becomes two streets... Calzada de los Misterios and Calzada de Guadalupe. And, yes, it is among the greatest peregrination routes in the world. On December 12, the Holiday of the Guadalupe Virgin, up to 7 million people go visit her temple. And also around here, when my mind was busy inside my body, I noticed a pair from the Un paseo a ciegas (roughly translated, a blind ride) program was riding just by me. I knew about this program, but had never seen them before So I took the opportunity to get to know this program. Basically: Two people ride a tandem bike. The one in front, as always, drives and steers. The one in the back is a blind person, who gets the amazing sensation of freedom and enjoyment that only a bike gives you. Un paseo a ciegas is held every Sunday at 9AM except for the last Sunday every month. And I definitively want to be part of it! They told me the program has been adopted as well in Puebla, and if I recall correctly, it will soon be started in Quer taro and Guadalajara. Anyway, by then I realized the ride was probably going to reach the Bas lica, a very impressive church I had just visited for the first time ever a couple of weeks ago. And yes we went all the way up Guadalupe until the place where the street gets impossible to bike on, then left along a small street, and back to the South by Misterios. Then, all the way along Reforma again, saying goodbye to Cuitlahuac, Jos de San Mart n, Sim n Bol var, El Caballito, Cristobal Col n, Cuauhtemoc, La Palmera, El ngel... And back home. Of course, not without the sight of some bizarre signs we enjoy so much to find, point at and laugh. Have I ever told you, my nameless reader, that I love this city? I love getting surprised with so many great things. I love the distances I can safely ride on my bike without ever being far from a bike repair shop (which are only needed when they are far away). I love being surrounded by people breathing their freedom to take the streets. I love the safety of our wide, long streets (and no, I'm not joking here Mexico City is among the safest places for cyclists I have been to). I love its weather, that allows me to go out and enjoy the street at any time of year. And I love biking in it, one of the best ways to get to know so many bits of the city I would have never ventured into otherwise.

While I have spent countless hours working on the new source format known as 3.0 (quilt) , I ve just realized that I have never blogged about its features and the reasons that lead me to work on it. Let s fix this. The good old 1.0 format Up to 2008, dpkg-source was only able to cope with a single source format (now named 1.0 ). That format was used since the inception of the project. While it worked fine for most cases, it suffered from a number of limitations mainly because it stored the Debian packaging files as a patch to apply on top of the upstream source tarball. This patch can have two functions: creating the required files in the debian sub-directory and applying changes to the upstream sources. Over time, if the maintainer made several modifications to the upstream source code, they would end up entangled (and undocumented) in this single patch. In order to solve this problem, patch systems were created (dpatch, quilt, simple-patchsys, dbs, ) and many maintainers started using them. Each implementation is slightly different but the basic principle is always the same: store the upstream changes as multiple patches in the debian/patches/ directory and apply them at build-time (and remove them during cleanup). Design goals for the new formats When I started working on the new source package format, I set out to get rid of all the known limitations and to integrate a patch system in dpkg-source. I wanted to clear up the situation so that learning packaging only requires to learn one patch system and would not require modifying debian/rules to use it. I picked quilt because it was popular, came with a large set of features, and was not suffering from NIH syndrome. This lead to the 3.0 (quilt) source format. I also created 3.0 (native) as a distinct format. 1.0 was able to generate two types of source packages (native and non-native) but I did not want to continue with this mistake of mixing both in a single format. The KISS principle dictated that the user should pick the format of his choice, put it in debian/source/format and be done with it. Now the build can rightfully fail when the requirements are not met instead of doing something unexpected as a fallback. Features of 3.0 (quilt) This is the format that replaces the non-native variant of the 1.0 source format. The features below are specific to the new format and differentiate it from its ancestor:

• Supports compression formats other than gzip: bzip2, lzma, xz.
• Can use multiple upstream tarballs.
• Can include binary files in the debian packaging.
• Automatically replaces the debian directory present in the upstream tarball (no repacking required).
• Creates a new quilt-managed patch in debian/patches/ when it finds changes to the upstream files.

Features of 3.0 (native) This format is very similar to the native variant of the 1.0 source format except for two things:

• it supports compression formats other than gzip: bzip2, lzma, xz.
• it excludes by default a bunch of files that should usually not be part of the tarball (VCS specific files, vim backup files, etc.)

Timeline Looking back at the history is interesting. This project already spans multiple years and is not really over until a majority of packages have switched to the new formats.

• January 2008: the discussion how to cope with patches sanely rages on debian-devel@lists.debian.org. My initial decisions are the result of this discussion.
• March 2008: I have implemented the new formats and I request feedback. dpkg 1.14.17 (uploaded to experimental) is the first release supporting them.
• April 2008: I ask ftpmasters to support the new source packages in #457345.
• June 2008: Lenny freeze. dpkg is not supposed to change anymore. Several changes concerning the new source formats are still accepted in the following months given that this code is not yet used in production and must only be present so that lenny can cope with new source packages once squeeze starts using them.
• February 2009: Lenny release.
• March 2009: Work on squeeze has started, ftpmasters have done nothing to support new source formats, I submit a patch in #457345 to speed things up. I start a wiki page to track the project s progress and to answer common questions of maintainers.
• November 2009: After an ftpmaster sprint, it s now possible to upload new source packages in unstable. This draws massive attention to the new format and some people start complaining about some design decisions. The implementation of 3.0 (quilt) changes a lot during this month. dpkg in lenny is even updated to keep up with those changes.
• March 2010: Up to now, I was planning to let dpkg-source build new source packages by default at some point in the future. After several rounds of discussions, I agree that it s not the best course of action and decides instead to make debian/source/format mandatory. The maintainer must be explicit about the source format that s/he wants to use.
• October 2010: The new source formats are relatively popular, a third of the source packages have already switched: see the graph. The squeeze freeze in August clearly stopped the trend, hopefully it will continue once squeeze is released.
• June 2013: Project is finished?

As you can see this project is not over yet, although the most difficult part is already behind me. For my part, the biggest lesson is that you won t ever get enough review until your work is used within unstable. So if you have a Debian project that impacts a lot of people, make sure to organize an official review process from the start. And specifying your project through a Debian Enhancement Proposal is probably the best way to achieve this. If you appreciate the work that I put into this project, feel free to join Flattr and to flattr dpkg from time to time. Or check out my page Support my work .

12 comments Support my work

I recently uploaded new upstream versions of two neat small X tools to Debian Experimental:

• xrootconsole displays on a transparent or shaded layer on the root window what it gets as input on STDIN, from a FIFO or from a file, and
• keynav, a way to control your mouse cursor efficiently with the keyboard.

Both packages introduce several new features and I d be happy if users of these packages in Debian Sid or Debian Sid users curious about them could test the versions in Debian Experimental. xrootconsole 0.6 + patches xrootconsole saw no love since 2006 with the last maintainer upload having been in 2002. Nevertheless it never got kicked out of Debian just because of this. The package had been forcibly orphaned just less than a year ago. So it s no big wonder that this new upstream version I packaged was released already back in 2004. :-) But besides packaging a new upstream version, bumping Standards-Version and debhelper compatibility, fixing tons of lintian warnings and some bugs, I also added two patches which add new features not (yet) available upstream:

• UTF-8 support: Upstream xrootconsole just drops all bytes where the 8th bit is set which only allows ASCII. Miroslav Jezbera submitted a patch to allow at least displaying localized text containing single byte 8 bit characters as used in all ISO-Latin encodings. Inspired by the patch to add UTF-8 support to ratmenu, I wrote a patch to add UTF-8 support for xrootconsole, too.
• ANSI color support: Last year, Julien Viard de Galbert, who is also active in Debian, posted a patch in his blog to support ANSI colors as produced by many log colorizers like e.g. loco (RIP), colortail, lwatch, or ccze. (Didn t get colortail and lwatch to work with xrootconsole yet, though.) I included this patch and made it compatible with my UTF-8 support patch. The patch raises the memory consumption per displayed character by one byte, but effectively I just saw an overall memory usage increase of about 25% which seems acceptable.

Of course I informed upstream about these feature patches, but I haven t got any feedback yet. But I got feedback from Julien Viard de Galbert, and he ll join me in packaging xrootconsole as co-maintainer. keynav 0.20101014.3067 I adopted the Debian package of keynav recently, subscribed to the keynav mailing list (well, it s a Google group), got a nice welcome mail from the very friendly upstream developer Jordan Sissel who offered to help with any keynav issues. I told him what features I d like to see in keynav to fit better the setup where I m using it. And just a few days later there was a new upstream release including both features I suggested, some more neat new features, and one bug fix. Jordan Sissel writes in the upstream changelog (emphasis and italic text by me):

• Added restart command. Makes keynav restart. Useful for binding a key to reload the config.
• Added loadconfig command. This lets you include additional config files to load on the command line or in one of the default keynavrc files. (requested by Axel Beckert) and already in use here.
• keynav will now restart if it receives SIGHUP or SIGUSR1
• Map Enter by default to warp,click 1,end (requested by Axel Beckert)
• Fix a bug causing the point under the mouse cursor to not click through the keynav window in certain conditions. Reported via mailing list by Eric Van Dewoestine and Krister Svanlund.

Both packages will be reuploaded to Debian Sid (Unstable) after the release of Debian Squeeze (currently testing ).

The big advantage of Unison on Windows is that it allows quite easily to synchronize between Windows and Linux. For those who need to work on Windows with the same set of files as on Linux, this is a big plus. Other tools do it as well, but the 2-way sync of Unison is quite nice. When you need to compile a software on Linux and Windows, you can modify both side at the same time and (almost) don't have problems. On Windows, the .unison and unison.log are located into your %HOMEPATH%, which the upper directory of the classic Documents folder. In the directory .unison, you will find the .prf files that describe your unison profiles. As usual, default.prf in this directory is the default profile. Basic The basic tips are:

• use fastcheck = true in your default.prf
• disable directory indexing

You can also disable virus live scan -- if you think it is safe!!!! SSH Using ssh under Windows is always a challenge. As a matter of fact, this tool doesn't match Windows context and it is not as integrated as in Linux/BSD. There is Putty which can help you. It has a good support for remote shell but it is not very easy to setup with Unison. Putty and OpenSSH doesn't have precisely the same set of options and Unison relies on some not available in Putty. There is a script called ssh2plink.bat that can help you using Putty's plink as with Unison. I used it for a while, but this didn't give the expected throughput. The best option is to use the ssh command provided by Cygwin. In this case you have at the same time good throughput and unison integration. I explain here how to configure you Cygwin's ssh to use a SSH key. You can bypass the following steps, if you wish to use a password or if you have already setup your ssh to connect to the target computer. Launch Cygwin's setup.exe and select openssh for installation. To add a SSH key, launch the cygwin shell:

$ ssh-keygen -t rsa
Generating public/private rsa key pair.
[...]

Copy the file .ssh/id_rsa.pub to your target computer's .ssh/authorized_keys. You should be aware that the file format can be Windows EOL style (in this case use dos2unix to convert the file) and if you copy/paste from a dos box, some end of lines are added and you should remove them from the authorized_keys, to have a single line key. Once, you have installed your ssh key into target computer, you should try to connect directly from the cygwin shell.

$ ssh XXX

Now, you can configure sshcmd = c:\cygwin\bin\ssh.exe to your default.prf. Using Cygwin's ssh allow you to get ~2MB/s (or more) when you only get ~100KB/s using ssh2plink.bat. If you have any other tips to improve Unison on Windows, I will be happy to test them and post it here.

I like to think that in the Ubuntu project, we re pragmatic about technology. This means keeping an open mind, considering alternatives, and evaluating them objectively. It means bearing in mind the needs of the user, and measuring ourselves based on how well we solve their problems (not merely our own). It is in this spirit that I have been thinking about Qt recently. We want to make it fast, easy and painless to develop applications for Ubuntu, and Qt is an option worth exploring for application developers. In thinking about this, I ve realized that there is quite a bit of commonality between the strengths of Qt and some of the new directions in Ubuntu:

• Qt has a long history of use on ARM as well as x86, by virtue of being popular on embedded devices. Consumer products have been built using Qt on ARM for over 10 years. We ve been making Ubuntu products available for ARM for nearly two years now, and 10.10 supports more ARM boards than ever, including reference boards from Freescale, Marvell and TI. Qt is adding ARMv7 optimizations to benefit the latest ARM chips. We do this in order to offer OEMs a choice of hardware, without sacrificing software choice. Qt preserves this same choice for application developers.
• Qt is a cross-platform application framework, with official ports for Windows, MacOS and more, and experimental community ports to Android, the iPhone and WebOS. Strong cross-platform support was one of the original principles of Qt, and it shows in the maturity of the official ports. With Ubuntu Light being installed on computers with Windows, and Ubuntu One landing on Android and the iPhone, we need interoperability with other platforms. There is also a large population of developers who already know how to target Windows, who can reach Ubuntu users as well by choosing Qt.
• Qt has a fairly mature touch input system, which now has support for multi-touch and gestures (including QML), though it s only complete on Windows 7 and Mac OS X 10.6. Meanwhile, Canonical has been working with the community to develop a low-level multi-touch framework for Linux and X11, for the benefit of Qt and other toolkits. These efforts will eventually meet in the middle.

Overall, I think Qt has a lot to offer people who want to develop applications for (and on) Ubuntu, particularly now. It already powers popular cross-platform applications like VLC, not to mention the entire Kubuntu distribution. I missed it when this happened last year, but Qt is now available under either the LGPL 2.1 or the GPL 3.0, which should make it suitable for virtually any Ubuntu application. It has strong commercial backing as well as a large developer community. No single solution will meet all developers needs, of course, and Ubuntu supports multiple toolkits and frameworks for this reason, but Qt seems like a great tool to have in our toolbox for the road ahead.

If you re running a 64bit system, and you ve got users with access to a video device (/dev/video*), then be sure you update your kernels for CVE-2010-2963. I ve been slowly making my way through auditing the many uses in the Linux kernel of the copy_from_user() function, and ran into this vulnerability. Here s the kernel code from drivers/media/video/v4l2-compat-ioctl32.c:

static int get_microcode32(struct video_code *kp, struct video_code32 __user *up)
 
        if (!access_ok(VERIFY_READ, up, sizeof(struct video_code32))  
                copy_from_user(kp->loadwhat, up->loadwhat, sizeof(up->loadwhat))  
                get_user(kp->datasize, &up->datasize)  
                copy_from_user(kp->data, up->data, up->datasize))
                        return -EFAULT;
        return 0;
 

Note that kp->data is being used as the target for up->data in the final copy_from_user() without actually verifying that kp->data is pointing anywhere safe. Here s the caller of get_microcode32:

static long do_video_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
 
        union  
                struct video_tuner vt;
                struct video_code vc;
...
          karg;
        void __user *up = compat_ptr(arg);
...
        switch (cmd)  
...
        case VIDIOCSMICROCODE:
                err = get_microcode32(&karg.vc, up);
...

So, the contents of up are totally under control of the caller, and the contents of karg (in our case, the video_code structure) are not initialized at all. So, it seems like a call for VIDIOCSMICROCODE would write video_code->datasize bytes from video_code->data into some random kernel address, just causing an Oops, since we don t control what is on the kernel s stack. But wait, who says we can t control the contents of the kernel s stack? In fact, this compat function makes it extremely easy. Let s look back at the union. Notice the struct video_tuner? That gets populated from the caller s up memory via this case of the switch (cmd) statement:

...
        case VIDIOCSTUNER:
        case VIDIOCGTUNER:
                err = get_video_tuner32(&karg.vt, up);
...

So, to control the kernel stack, we just need to call this ioctl twice in a row: once to populate the stack via VIDIOCSTUNER with the contents we want (including the future address for video_code->data, which starts at the same location as video_tuner->name[20]), and then again with VIDIOCSMICROCODE. Tricks involved here are: the definition of the VIDIOCSMICROCODE case in the kernel is wrong, and calling the ioctls without any preparation can trigger other kernel work (memory faults, etc) that may destroy the stack contents. First, we need the real value for the desired case statement. This turns out to be 0 4020761b. Next, we just repeatedly call the setup ioctl in an attempt to get incidental kernel work out of the way so that our last ioctl doing the stack preparation will stick, and then we call the buggy ioctl to trigger the vulnerability. Since the ioctl already does a multi-byte copy, we can now copy arbitrary lengths of bytes into kernel memory. One method of turning an arbitrary kernel memory write into a privilege escalation is to overwrite a kernel function pointer, and trigger that function. Based on the exploit for CVE-2010-3081, I opted to overwrite the security_ops function pointer table. Their use of msg_queue_msgctl wasn t very good for the general case since it s near the end of the table and its offset would depend on kernel versions. Initially I opted for getcap, but in the end used ptrace_traceme, both of which are very near the top the security_ops structure. (Though I need share credit here with Dan Rosenberg as we were working together on improving the reliability of the security_ops overwrite method. He used the same approach for his excellent RDS exploit.) Here are the steps for one way of taking an arbitrary kernel memory write and turning it into a root escalation:

• overwrite security_ops with default_security_ops, which will revert the LSM back to the capabilities-only security operations. This, however, means we can calculate where cap_ptrace_traceme is.
• overwrite default_security_ops->ptrace_traceme to point to our supplied function that will actually perform the privilege escalation (thanks to Brad Spengler for his code from Enlightenment).
• trigger the function (in this case, call ptrace(PTRACE_TRACEME, 0, NULL, NULL)).
• restore default_security_ops->ptrace_traceme to point to cap_ptrace_traceme so the next caller doesn t Oops the system (since userspace memory will be remapped).

Here s the source for Vyakarana as seen running in Enlightenment using cap_getcap (which is pretty unstable, so you might want to switch it to use ptrace_traceme), and as a stand-alone memory writer. Conclusions: Keep auditing the kernel for more arbitrary writes; I think there are still many left. Reduce the exploitation surface within the kernel itself (which PaX and grsecurity have been doing for a while now), specifically:

• Block userspace memory access while in kernel mode. This would stop the ability to make the kernel start executing functions that live in userspace a clear privilege violation. This protection would stop the current exploit above, but the exploit could be adjusted to use kernel memory instead.
• Keep function pointers read-only. There is no reason for these function pointer tables (fops, IDT, security_ops, etc) to be writable. These should all be marked correctly, with inline code exceptions being made for updating the global pointers to those tables, leaving the pointer read-only after it gets set. This would stop this particular exploit above, but there are still plenty more targets.
• Randomize the kernel stack location on a per-syscall basis. This will stop exploits that depend on a stable kernel stack location (as this exploit does).

Buying a ThinkPad? Don't buy one with this wireless chipset. "ThinkPad wireless" is apparently code for "poorly supported RealTek chip". If you do end up with one of these chips (perhaps you failed to notice the addition of a new option on the website before the time you spec'ed out your new laptop and the time you ordered it; perhaps you assumed it was iwlagn by default; who can say?), you may be treated to such issues as a lack of rfkill interface support, a driver that needs to be hard reset about once an hour to continue to be able to see APs, a kernel panic if you try to run powertop... or maybe a mysterious bug that prevents the wireless card from transmitting packets to a VoIP server, with 100% reliability. And after you wear yourself out from laughing at the Lenovo sales rep's suggestion that you could pay a 15% restocking fee and reorder another laptop that costs $40 more to get the Intel Centrino option when that same Centrino card can be found on-line for $40 by itself, you might decide to order an Intel Centrino Ultimate-N 6300 from a third party, the same card that's advertised on Lenovo's website as an option for the X201, and install it yourself. And that's when the fun would start. Did you know that it's common practice for laptop OEMs to hard-code a whitelist of authorized wifi (and 3G) cards in the BIOS, and refuse to boot with an unrecognized card - nominally to ensure no one can combine a wireless adapter with their hardware antennas to create a transmission device that violates FCC regulations? And that these same OEMs may have different PCI subsystem IDs for the cards they ship than are used for the ones you can buy retail from third parties? I did not know that. Apparently some people knew that because they had tried this before, but I missed that memo, having never before needed to combine a laptop and a wifi card from different vendors. Here, gentle reader, is a memo. You're welcome. Hacking the ThinkPad BIOS So although this feature was news to me, it was not news to the Internet at large. A quick search on the BIOS error message quickly turned up a relevant description of the problem, as well as pointers to some options for fixing (in the veterinary sense) the whitelist checking in the ThinkPad BIOS. It's great to be part of such a large community of ThinkPad-using Free Software hackers looking out for each other (and not taking "no" for an answer from their BIOS)! But wait, none of the methods described on that page for modifying your BIOS (or bypassing the check by moving the card to a different slot, or taping over one of the pins, or...) work on the X201! As Matthew summarized it, things are more difficult for "anything with three digits or starting with a six". Fortunately, another hint from Matthew took me to the website of a guy who's been providing fixes for this BIOS issue for the past couple of years. Hmm, but the X201 is a bleeding-edge new model and isn't on his list of models that he's patched the BIOS for. Well, fortunately there's a step-by-step howto here that I can follow here to do it myself... grab the latest BIOS image from the vendor's website; decompress the image with phcomp.exe (oh Wine, you pair well with more than cheese); unpack the individual BIOS modules with phnxsplit; run phnxpatch to apply the binary patches, finding that the only one that applies is "wwan1a" when the issue is not with a WWAN card at all, so hand-hack the whitelist in the binary instead; use the fi.exe and fp.exe tools to add the necessary header back onto the BIOS module (after deciphering the example batch script accompanying the BIOS update and translating it into a reusable script in a real scripting language); use phnxmod to inject the newly-packaged BIOS module into the original image; call phnxcksm to update the BIOS's internal checksum; ... Uh? Phnxcksum fails, saying that there's no extended checksum field. Just how deep is this rabbit hole? How much was that restocking fee again, and how much is my time worth? But no; now it's a matter of principle. Besides, I'd rather spend my time fighting with a BIOS than having to reinstall the OS on another new machine. Which is to say that I'd rather beg the local reverse engineering experts for their time fighting with a BIOS. So now we have a BIOS image whose code we're confident has been correctly patched to bypass the whitelist, but we can't find the checksum field that our patching tools expect. Maybe we can flash this to the BIOS as is? No such luck; sure enough, we get a checksum error. Ah, but WinPhlash gives us a logfile, and shows that the failure is a per-module checksum mismatch. So where is that checksum? Oh, there it is; it's the byte labeled "data_checksum" in the module header description in phnxsplit.h, which phnxmod doesn't copy when it splices an updated module into the BIOS image. Well, could it really be as easy as updating a single byte? Yes, the BIOS utility takes it and flashes it without complaint... and the system boots afterwards! Thanks to Paul and Matthew for their great work documenting this problem in general; Zender, for tools that provide a 99% solution; and Kees and Jesse for a dazzling display of impromptu reverse-engineering. Now, to figure out how to get these tools packaged and into Debian and Ubuntu. Posted from my Centrino-wireless-using Lenovo X201 laptop.

This is definitely a "captain obvious" observation, but it's interesting to see just how much bad weather here in the Veneto is correlated with traffic to my real-time weather site, http://www.meteo-veneto.net The precipitation and insolation numbers for the month of September - as well as the radar images for the site itself - come from http://www.arpa.veneto.it/ The numbers are all normalized so as to make them easy to compare. The cloudy/dark numbers are the inverse of the insolation for the month, and could probably be fiddled with to be a bit more accurate - it's obvious that the number increases as the month goes on, with less sunlight.

A little while back, as part of a discussion on debian-devel, I spent some time bad-mouthing pkg-config, and then later realized (with the help of an IRC conversation) that I was doing so unfairly. So as penance here is some documentation for how to add a pkg-config configuration file to a typical Automake project. This information is also in my build system coding style and will be maintained there (so if you see this some time after I wrote it, check that page for more up-to-date information). pkg-config is a tool to provide the flags required to compile or link against a shared library. It's a replacement for complex Autoconf probes and for all of the incompatible and occasionally buggy *-config scripts. To install a pkg-config configuration file, provide a template pkg-config configuration in a file named after the library but with a .pc.in extension. Here is an example:

    prefix=@prefix@
    exec_prefix=@exec_prefix@
    includedir=@includedir@
    libdir=@libdir@
    Name: <library>
    Description: <short description>
    URL: <upstream url>
    Version: @PACKAGE_VERSION@
    Cflags: -I$ includedir 
    Libs: -L$ libdir  -l<library>
    Libs.private: @KRB5_LDFLAGS@ @KRB5_LIBS@

Replace <library> with the official name of the library (which isn't necessarily the library name with lib removed). This is only informational, may contain spaces, and isn't used to locate the file. Its filename is used for that. Replace <short description> with a short description of the purpose of the library. Often the short description of the package from the start of the README file works well here. Replace <upstream url> with the URL of the package's distribution page. Normally, all Cflags needs to contain is what's indicated above, which lets the compiler find the header files if installed in a non-standard location. For Libs, note that pkg-config distinguishes between the shared libraries that must be linked against on a system with proper transitive dependency support and the libraries that must be linked for a static build or on a system with poor support for shared libraries. For a well-designed shared library, one should rarely need anything in Libs other than what's listed above. Libs.private, on the other hand, needs all the LDFLAGS and LIBS of everything the library itself links with external to this package. In this example, the shared library is linked with the Kerberos libraries and therefore needs the Kerberos LDFLAGS and LIBS; modify as needed for your example. If the dependencies of the shared library also use pkg-config, just list the other pkg-config file names in Requires or Requires.private instead of using Libs or Libs.private for them. Finally, add the Automake code to build and install the pkg-config configuration file. It should look like this:

    pkgconfigdir = $(libdir)/pkgconfig
    nodist_pkgconfig_DATA = lib/libfoo.pc
    lib/libfoo.pc: $(srcdir)/lib/libfoo.pc.in
            sed -e 's![@]prefix[@]!$(prefix)!g' \
                -e 's![@]exec_prefix[@]!$(exec_prefix)!g' \
                -e 's![@]includedir[@]!$(includedir)!g' \
                -e 's![@]libdir[@]!$(libdir)!g' \
                -e 's![@]PACKAGE_VERSION[@]!$(PACKAGE_VERSION)!g' \
                -e 's![@]KRB5_LDFLAGS[@]!$(KRB5_LDFLAGS)!g' \
                -e 's![@]KRB5_LIBS[@]!$(KRB5_LIBS)!g' \
                $(srcdir)/lib/libfoo.pc.in > $@

Replace libfoo with the actual name of the library and change its path in the source package as needed. Note the pattern above and use it for any Autoconf substitution variables that you need to replace in the pkg-config template. The above example adds KRB5_LDFLAGS and KRB5_LIBS to the variables; you may have other ones you need, particularly to construct Libs.private. The above installs the pkg-config configuration file under $(libdir)/pkgconfig. $(datadir)/pkgconfig is appropriate instead if one is certain that the flags required to link with this library will never vary by architecture, but that's not the case for most libraries due to Libs.private. Note that the Automake documentation recommends using comma as the sed delimiter for doing this sort of trick, but LDFLAGS frequently contains a comma since it's the separator for parts of a linker-specific option. I use an exclamation point instead.

The recent CVE-2010-2961 mountall vulnerability got a nice write-up by xorl today. I ve seen a few public exploits for it, but those that I ve seen, including the one in xorl s post, miss a rather important point: udev events can be triggered by regular users without any hardware fiddling. While the bug that kept udev from running inotify correctly on the /dev/.udev/rules.d directory during initial boot kept this vulnerability exposure pretty well minimized, the fact that udev events can be triggered at will made it pretty bad too. If udev had already been restarted, an attacker didn t have to wait at all, nor have physical access to the system. While it is generally understood that udev events are related to hardware, it s important to keep in mind that it also sends events on module loads, and module loads can happen on demand from unprivileged users. For example, say you want to send an X.25 packet, when you call socket(AF_X25, SOCK_STREAM), the kernel will go load net-pf-9, which modules.alias lists as the x25 module. And once loaded, udev sends a module event. (Which, by the way, should serve as a reminder to people to block module loading if you can.) So, as I mentioned, here s yet another exploit for the mountall vulnerability: mountall-CVE-2010-2961.py. It writes to the vulnerable udev rule file and then attempts to trigger udev immediately by walking a list of possible socket() AF_* types.

If you re using Debian, you know that this distribution is built entirely by volunteers that form a very diverse community. And you could be part of it. But why should you do that? I can t tell for you but I can share my own experience. It s been 12 years since I joined Debian and I m going to tell you what keeps me on board. 1. Technical excellence When facing a new challenge, Debian will strive to do the right thing. This pays off in the long term. In many cases, it means that we will take more time to implement our solution compared to other distributions out there, but this is also the reason why our packaging infrastructure allows us to offer painless upgrades and consistency across all packages. Debian is committed to quality and builds up on his experience thanks to the Debian Policy. My time is precious, I like to spend it on something useful in the long term. 2. Inspiring goals In its social contract, Debian has set out to create a 100% free as in freedom operating system. The criteria defining what constitutes a free work are listed in the Debian Free Software Guidelines (DFSG). While the above is relatively ambitious in itself, it doesn t inspire me much. What makes the difference to me is the emphasis given by the social contract on our users and free software. We don t build a free operating system in the void, we build it for people. Debian s motto the universal operating system can also be interpreted in multiple ways: universal as in for everybody on the world , on all kinds of computers , or for every possible usage . 3. High impact work Knowing that my work is useful to people is important, but it s even better when I know that it will benefit to lots of people. With Ubuntu and the hundreds of other derivatives, there are nowadays literally millions of users impacted by my work. Even an insignificant one second improvement experienced by 10 millions of people represent 115 days of time saved for something else, you get the idea 4. Working with great people Debian has the chance to have lots of smart people on board. There s always someone sharing valuable advice when you read Debian s mailing lists. When I joined in 1998, I was a real newbie and I learned a lot of things by reading and interacting with people more knowledgeable than me. You can still experience the same thing nowadays but there s one caveat: you must cope with various kinds of mailing list contributors including the smart but uncivilized (don t be offended too quickly!) and the occasional troll (best ignore it, don t feed it!). 5. Recognition of work When you contribute to Debian, people get to know you through your contributions. It s very rewarding to be thanked by your peers and by Debian s users. Check out thanks.debian.net to convince you that many people are grateful for the work we put into Debian. So that s it for me. But what about you? What motivates you to contribute year after year, or to start contributing if you re a prospective contributor?

Subscribe to this blog (it s free!) by RSS, by email, or on Facebook.

21 comments Support my work

This essay is a summary of my presentation at the workshop Inlaws and Outlaws, held on August 19-20, 2010 in Split, Croatia. The workshop brought together advocates of piracy with participants in the free culture and free software movements.

In Why Software Should Not Have Owners, Richard Stallman explains that, if a friend asks you for a piece of software and the license of the software bars you from sharing, you will have to choose between being a bad friend or violating the license of the software. Stallman suggests that users will have to choose between the lesser of two evils and will choose to violate the license. He emphasizes that it's unfair to ask a user to make such a choice. Over the past few years, pirate parties have grown across much of the developed world. Of course, piracy remains the primary means of distributing media across most of the rest. Advocates of access to information have gathered and organized under the "pirate" banner, representing the choice of sharing with friends over compliance with license terms. The free/libre and open source software (FLOSS) and free culture movements seem to have a confused and conflicted reaction to all this. On one hand, major proponents of several pirate parties are FLOSS and free culture stalwarts and several pirate parties have made FLOSS advocacy a component of their political platforms. Pirate Parties' clear opposition to software patents and DRM resonates with both FLOSS and free culture communities. On the other hand, FLOSS leaders, including Stallman, have warned us about "pirate" anti-copyright policies. Free culture leaders, like Lawrence Lessig, have repeatedly and vociferously denounced piracy, treated even the intimation of an association with piracy as an affront, and systematically distanced themselves and their work from piracy. Should FLOSS and free culture advocates embrace pirates as comrades in arms or condemn them? Must we choose between being either with the pirates or against them? Our communities seem to have no clearly and consistently articulated consensus. I believe that, unintuitively, if you take a strong principled position in favor of information freedom and distinguish between principles and tactics, a more nuanced "middle ground" response to piracy is possible. In light of a principled belief that users should be able to share information, we can conclude there is nothing ethically wrong with piracy. Licenses have the power of the law but they are protected by unjust "intellectual property" laws. That said, principles are not the only reason activists choose to do things. Many political stunts are bad ideas not because they are wrong, but because they won't work and have negative side effects. Tactics matter too. Even though there might not be anything ethically wrong with piracy from the perspective of free software or free culture, it might still be a bad idea. There are at least three such tactical reasons that might motivate free software and culture to not support piracy or participate in pro-piracy movements and politics. First, a systematic disrespect for copyright undermines respect for all licenses which have been of a huge tactical benefit to free software and a increasingly important factor in the success of free culture. Copyleft licenses like the GPL or CC BY-SA have power only because copyright does. As Stallman has suggested, anti-copyright actions are anti-copyleft. That needn't be an argument against attempts to limit copyright. Indeed, I think we must limit and reduce copyright. But we must tread carefully. In the current copyright climate, I believe that copyleft offers a net advantage. Why should others respect our licenses if we don't respect theirs? Looking at the long term, we must weigh the benefits of promoting the systematic violation of proprietary licenses with the benefits of adherence to free software and free culture. Second, piracy is fundamentally reactionary. Part of its resonance as a political symbol comes from the fact that the piracy represents a way that consumers of media can fight back against a set of companies which have attacked them -- with lawsuits, DRM systems, and demonization in propaganda -- for sharing in ways that most consumers think are natural and socially positive. But piracy focuses on reaction rather the fundamental importance of sharing that drives it. As a result, most pirates do not support, or are even familiar with, a principled approach to access to information. As a result, many piracy advocates who speak out against DRM on DVDs will be as happy to use NetFlix to stream DRMed movies for $5 a month as they were to download for free. The best rallying cries do not always translate into be the most robust movements. Third, through its focus on a reaction, a dialog about piracy avoids engagement with the tough questions of what we will replace the current broken copyright system with. A principled position suggests that it is our ethical prerogative to create alternative models. The free software movement has succeeded because it created such a prerogative and then, slowly over time, provided examples of workable alternatives. A principled position on free software did not require that one provide working new systems immediately, but it makes the development of creative, sustainable approaches a priority. Attacking the system without even trying to speak about alternative modes of production is unsustainable. Free software and free culture call for a revolution. Piracy only calls for a riot. Piracy, in these three senses, can be seen as tactically unwise, without necessarily being unethical. By taking a principled position, one can go build on, and go beyond, RMS's comment. On free culture and free software's terms, we can suggest that piracy is not ethically wrong, but that it is an unwise way to try to promote sharing. Without being hypocritical, we can say: "I don't think piracy is unethical. But I also do not support it."

Two years ago, I wrote about Elvera Voth and the power of love. Back then I wrote about her memories of seeing of service workers, who would be away from their home for 7 years at a time. Elvera remembered a gathering of people at the train station to see them off, and how they sang some German hymns at the occasion. Elvera s done a lot since, including starting an arts in prison program. And today, she hosted a hymn sing at the pump organ in the Friesen House parlor. The Friesen House is nearly 100 years old, and was on my parents property before it got moved to the Mennonite Heritage and Agricultural Museum. We were among maybe 20 people that showed up for the event. We walked in a few minutes early and Elvera was sitting at the organ already, along with a few other early arrivals. Many of the people there were over 70, and the moment we walked in, she said, Oh good! Another generation! And, of course, asked about us, where we live, who we are related to, etc. The announcement in our church bulletin said to bring a copy of Gesangbuch mit Noten if you have one. That was a common hymn book in Mennonite churches in Kansas (songbook with notes and yes, there was one without notes that had only words.) I pulled out my copy, and just opened it up. I saw my grandma s beautiful handwriting saying it belonged to Mr. and Mrs. R. A. Klassen . This I brought along, but was perhaps the only one. Elvera recognized it and was pleased to hear where it came from. We had a great time, and it turned out to feel like the kind of afternoon people around here used to have: time with friends and family visiting and maybe singing on occasion. After some introduction and some stories, we got to the singing and started with hymn #1 in the 1969 (English) Mennonite Hymnal. That song had German words below as well, which we sang: Grosser Gott, wir loben dich! (Holy God, We Praise Thy Name) Elvera played the organ while we all sang. And then we turned to #556, O Have You Not Heard. We read through the German lyrics (to help those that don t speak German):

Ich weiss einen Strom, dessen herrliche Flut
fliesst wunderbar stille durchs Land,
doch strahlet und gl nzt er wie feurige Glut,
wem ist dieses W sser bekannt? O Seele, ich bitte dich: Komm!
Und such diesen herrlichen Strom!
Sein Wasser fliesst frei und m chtiglich,
o glaub s, es fliesset f r dich!

Elvera translated this herself, a much more powerful translation than we usually get in English. And then we sang, this time a cappella. Elvera told the story about the train station to this group, and then, of course, we sang O Power of Love, one of the songs sung on that occasion. For that, she asked us to stand up and form a circle around the organ, and again we sang mostly a cappella. Singing that song with Elvera left few dry eyes in the room for sure. Someone brought up Nun Ist Sie Erschienen, and so of course we sang that. It wasn t in any hymnal, but I suspect we were the only ones there that didn t have it completely memorized. Elvera told us the story of the museum s pump organ; it had been in her family, and she had paid to have it restored and eventually donated it to the museum. It had been used every evening at twilight for singing and devotions in the family. She also told us that the singing at the Newton train station the same one our family uses a few times a year was what inspired her to a career of choral music. In the end, we probably spent more time visiting than singing, but that was just fine. Next the group went over to the Preparatory School, an old schoolhouse also on the museum grounds. There we had a traditional Faspa, an afternoon meal with coffee, Zwieback and jam, cheese, and various cookies. We all visited for awhile longer and then went on our way. It was a wonderful afternoon, and I hope to have a chance to do that sort of thing again.

I should probably have researched the batteries in this old solar-powered house before I installed a charge controller. I had assumed that the 24 6-volt batteries in the house's battery bay were standard lead-acid deep cycle batteries. It wasn't until I stumbled over a receipt from 1997 that I learned that the batteries are really NiCads. Specifically Saft model STM5-180. Thirty of these were purchased in 1997 from TVA in Chatanooga, for $350 total. An amazing price, since a single deep cycle battery is in that ballpark, new. These batteries were used, they had been in a "bus" -- perhaps it was a 1995 Chrysler TEVan, or maybe one of the pilot electric buses running there in the late 90's. They were probably not in this '79 VW TVA Bus. All mentions of these batteries I can find involve electric vehicles -- it's unusual for them to be used to power a house. Being vented NiCads probably accounts for these batteries' long useful life -- surely at least 15 years. Still, with only twenty-four good ones left, they are probably toward the end of their lifespan and need to be taken care of in order to last. Once I realized they were NiCads, I knew the charge controller was charging them wrong. NiCads like to be charged at a higher voltage than the 13 or so volts used for lead acid. I would have liked to charge them at 16 volts, but that would feed back through the house wiring, and could fry 12 volt stuff. Checking ratings, 15 volts seemed the highest voltage I could risk. Coming back a week later, I found the batteries charged up to 13.4 volts. And they are now working great. Through several cloudy days, we had all the power we needed. And when the sun was out fully, I sometimes saw the solar panels charging the batteries at 125 watts -- fully half of the panels' rated capacity, and much better than before. The batteries start each evening at 13.4 volts, and only drop to 13.3 by morning. We started using electric lights more, and then just leaving them on all evening, and the nslu2 online all night, and the batteries remained at the magic 13.3 in the morning. This is because NiCad batteries have a near-constant voltage until they are perhaps 30% discharged. In other words, I had been charging them less than half full before. I have figured out how to combine the two banks the batteries are in into one large bank, and once I get the cables to do that, I hope to have battery capacity to get through up to a week of solid clouds in midwinter.

battery box and solar panel