I did not realize how much I was missing this sort of heads-down programming time in my life until the last two days. I think the last time that I had this much fun with my job was when I was learning Java, and even that wasn't quite this much fun since it was a lot of experimentation rather than being directly productive in a language I know well. Here's a short list of modern Perl, Perl techniques, and interesting modules that I'd not had a reason to use before today:

• Profiling and Devel::NYTProf. What a lovely piece of software. I'm starting to like this trend towards useful developer tools dumping their output in HTML, particularly when many of them are this pretty. (Devel::Cover by way of Test::Strict also does this, and I've been making heavy use of it.)
• Memoize. I've known it existed for years, but I've never had a reason to use it. Memoizing str2time when doing log parsing cut the time by a quarter. (I'll probably write a hand-rolled date parser for the specific date format eventually, since str2time is really slow since it's so general, but wrapping Memoize around it takes five seconds.)
• Module::Build. I was staying away from this because of all the problems Debian had with embedded code copies, but it appears to not be doing that any more (or at least I can use it in a way that it doesn't). And it really is so much nicer than ExtUtils::MakeMaker. I will probably convert all of my packages over time, although I have to see what it's like for XS modules. ETA 2013-01-24: Gregor pointed out that I was confusing Module::Build with Module::Install. I feel silly now; I've been avoiding Module::Build solely because I've been confusing it with something else entirely!
• HTTP::BrowserDetect. This isn't a general piece of software, of course, but once again it was lovely to discover that CPAN already contained exactly the piece of software that I needed. I'm trying to produce a summary of what browsers our authenticated users use as part of a general software inventory, and this turned the nonsense that browsers send into exactly the sort of management-consumable classifications I needed.
• IO::Compress and IO::Uncompress. Yes, this is much nicer than what I used to have to do with Compress::Zlib. It's like people kept working on Perl while I've been using all the things I learned five years ago!

I still have tomorrow morning to continue focusing on this and will hopefully finish up the work to feed long time periods of logs into my analysis tool. And, even better, I have a log parsing framework that I really like and that I think will make any number of things easier in the long run. (Also, it has a remarkably thorough test suite. I love test-driven development. Although I haven't written comprehensive tests for the very final front-end piece, since generating sanitized log input to trigger all the cases is too much effort right at the moment.)

With the arrival earlier today of the stochvol package onto the CRAN network for R, our Rcpp project reached a new milestone: 100 packages have either a Depends:, Imports: or LinkingTo: statement on it. The full list will always be at the bottom of the CRAN page for Rcpp; I also manually edit a list on my Rcpp page. But for the record as of today, here is the current list as produced by a little helper script I keep:

 acer                apcluster           auteur             
 bcp                 bfa                 bfp                
 bifactorial         blockcluster        ccaPP              
 cda                 classify            clusteval          
 ConConPiWiFun       EpiContactTrace     fastGHQuad         
 fdaMixed            forecast            fugeR              
 GeneticTools        gMWT                gof                
 gRbase              gRim                growcurves         
 GUTS                jaatha              KernSmoothIRT      
 LaF                 maxent              mets               
 minqa               mirt                mRMRe              
 multmod             mvabund             MVB                
 NetworkAnalysis     ngspatial           oem                
 openair             orQA                parser             
 pbdBASE             pbdDMAT             phom               
 phylobase           planar              psgp               
 quadrupen           Rchemcpp            Rclusterpp         
 RcppArmadillo       RcppBDT             rcppbugs           
 RcppClassic         RcppClassicExamples RcppCNPy           
 RcppDE              RcppEigen           RcppExamples       
 RcppGSL             RcppOctave          RcppRoll           
 RcppSMC             RcppXts             rforensicbatwing   
 rgam                RInside             Rmalschains        
 Rmixmod             robustgam           robustHD           
 rococo              RProtoBuf           RQuantLib          
 RSNNS               RSofia              rugarch            
 RVowpalWabbit       SBSA                sdcMicro           
 sdcTable            simFrame            spacodiR           
 sparseHessianFD     sparseLTSEigen      SpatialTools       
 stochvol            surveillance        survSNP            
 termstrc            tmg                 transmission       
 trustOptim          unmarked            VIM                
 waffect             WideLM              wordcloud          
 zic                

And not to be forgotten is BioConductor which has another 10:

 ddgraph            GeneNetworkBuilder GOSemSim          
 GRENITS            mosaics            mzR               
 pcaMethods         Rdisop             Risa              
 rTANDEM  

As developers of Rcpp, we are both proud and also a little humbled. The packages using Rcpp span everything from bringing new libraries to R, to implementing faster ways of doing things we have before to doing completely new things. It is an exciting time to be using R, and to be connecting R to C++, especially with so many exciting things happening in C++ right now. Follow the Rcpp links for more, and come join us on the Rcpp-devel mailing list to discuss and learn.

Weblate 1.4 has been released today. It comes with lot of improvements, especially in configurability, admin interface and usability. Full list of changes for 1.4:

• Fixed deleting of checks/comments on unit deletion.
• Added option to disable automatic propagation of translations.
• Added option to subscribe for merge failures.
• Correctly import on projects which needs custom ttkit loader.
• Added sitemaps to allow easier access by crawlers.
• Provide direct links to string in notification emails or feeds.
• Various improvements to admin interface.
• Provide hints for production setup in admin interface.
• Added per language widgets and engage page.
• Improved translation locking handling.
• Show code snippets for widgets in more variants.
• Indicate failing checks or fuzzy strings in progressbars.
• More options for formatting commit message.
• Fixed error handling with machine translation services.
• Improved automatic translation locking behaviour.
• Support for showing changes from previous source string.
• Added support for substring search.
• Various quality checks improvements.
• Support for per project ACL.
• Basic unit tests coverage.

You can find more information about Weblate on it's website, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Ready to run appliances will be soon available in SUSE Studio Gallery. Weblate is also being used https://l10n.cihar.com/ as official translating service for phpMyAdmin, Gammu, Weblate itself and others. If you are free software project which would like to use Weblate, I'm happy to help you with set up or even host Weblate for you (this will be decided case by case as my hosting space is limited).

Filed under: English Phpmyadmin Suse Weblate 0 comments Flattr this!

In this article, I'm going to explain you how you can extend Swift, the OpenStack Object Storage project, so it performs extra action on files at upload or at download time. We're going to build an anti-virus filter inside Swift. The goal is to refuse uploaded data if they contain a virus. To help us with virus analyses, we'll use ClamAV. WSGI, paste and middleware To do our content analysis, the best place to hook in the Swift architecture is at the beginning of every request, on swift-proxy, before the file is actually stored on the cluster. Swift proxy uses, like many other OpenStack projects, paste to build his HTTP architecture. Paste uses WSGI and provides an architecture based on a pipeline. The pipeline is composed of a succession of middleware, ending with one application. Each middleware has the chance to look at the request or at the response, can modify it, and then pass it to the following middleware. The latest component of the pipeline is the real application, and in this case, the Swift proxy server. If you've already deployed Swift, you encountered a default pipeline in the swift-proxy.conf configuration file:
This is a really basic pipeline with a few middleware. The first one catches error, the second one is in charge to return 200 OK response if you send a GET /healthcheck request on your proxy server. The third one is in charge of caching, the fourth one is used for rate limiting, the fifth for authentication, the sixth one for logging, and the final one is the actual proxy server, in charge of proxying the request to the account, container, or object servers (the others components of Swift). Of course, we could remove or add any of the middleware here at our convenience. Be aware that the order matters: for example, if you put healthcheck after tempauth, you won't be able to access the /healthcheck URL without being authenticated! ClamAV If you don't know ClamAV, it's an antivirus engine designed for detecting trojans, viruses, malware and other malicious threats. Wwe're going to use it to scan every incoming file. To build the middleware, we'll use the Python binding pyclamd. The API is quite simple, see:
Anatomy of a WSGI middleware Your WSGI middleware should consist of a callable object. Usually this is done with a class implementing the __call__ method. Here's a basic boilerplate:
I'm not going to expand more on why this is built this way, but if you want to have more info on this kind of filter middleware, you can read their documentation on Paste. This middleware will just do nothing as it is. It's going to simply pass all requests it receives to the final application, and returns the result. Testing our basic middleware Now is a really good time to add unit tests. I hope you didn't think we were going to write code without some tests, right? It's really easy to test a middleware, as we're going to use WebOb for that.
We create a FakeApp class, that represents a fake WSGI application. You could also use a real application, or write a fake application looking like the one you want to test. It'll require more time, but your tests will be closer to the reality. Here we write the simplest test we can for our middleware. We're just sending a GET / request to it, so it passes the request to the final application and returns the result. It is transparent, it does nothing. Now, with that solid base we'll able to add more features and test these features incrementally. Plugging ClamAV in With our base ready, we can start thinking about how to plug ClamAV in. What we want to check here, is the content of the file when it's uploaded. If we refer to the OpenStack object storage API , a file upload is done via a PUT request, so we're going to limit the check to that kind of requests. Obviously, more checks could be added, but we'll keep things simple here for the sake of comprehensibility. With WSGI, the content of the request is available in env['wsgi.input'] as an object implementing a file interface. We'll scan that stream with ClamAV to check for viruses.
That's it. We only check for PUT requests and if there's a virus in the file, we return a 403 Forbidden error with the name of the detected virus, bypassing entirely the rest of the middleware chain and the application handling. Then, we can simply test it.
The first test test_put_empty simulates an empty PUT request. The second one, test_put_no_virus simulates a regular PUT request but with a simple file containing no virus. Finally, the third and last test simulates the upload of a virus using the EICAR test file. This is a special test file that is recognized as a virus, even if it's not real one. Very handy for testing virus detection software! Configuring Swift proxy Our middleware is ready! We can configure Swift's proxy server to use it. We need to add the following lines to our swift-proxy.conf to teach it how to load the filter:
We'll assume that our Python modules is named swiftclamava here. Now that we've defined our filter and how to load it, we can use it in our pipeline:
Just before reaching the proxy-server, and after the user being authenticated, the content will be scanned for viruses. It's important here to put this after authentication for example, because otherwise we may scan content that will get rejected by the authtemp module, thus scanning for nothing! Beyond scanning And voil , we now have a simple middleware testing uploaded content and refusing infected files. We could enhance it with various other things, like configuration handling, but I'll let that as an exercise for the interested readers. We didn't exploited it here, but note that you can also manipulate request headers and modify them if needed. For example, we could have added a header X-Object-Meta-Scanned-By: ClamAV to indicates that the file has been scanned by ClamAV. You should now be able to build your own middleware doing whatever you want with uploaded data. Happy hacking!

Today in the USA is Martin Luther King, Jr., day. But sometimes these holidays get confusing for a 6-year-old. I asked Jacob the other day if he knew what holiday was coming up. He thought about it for a second, then declared it would be St. Patrick s Day. He was excited because St. Patrick s Day is green. When he realized that it was really MLK Day, he was disappointed. That day isn t green. So I said, Jacob, how about we celebrate pretend St. Patrick s Day on Monday? His face lit up, he got a huge smile, and said, Oh yes! Great idea, dad! Oliver got all excited about it too. I was already planning on us doing some cooking, and thankfully had green food coloring already. So I sort of discarded my plans so each meal could have something green in it. When the boys woke up this morning, I wished each of them Happy Pretend St. Patrick s Day! We all wore green. Jacob put his shirt on backwards so the side with more green would be facing front. For breakfast, our green dish was green crepes with a succotash (based on baby lima beans and corn) filling. The boys were excited to discover that the crepes could be green on one side, and green and a little brown on the other. Jacob was unsure of the succotash idea, but after having a few bites, declared it excellent . After breakfast, we made bread. They loved watching the green food coloring disperse in the water. We checked on how green the dough looked periodically. We watched how it was rising and whether it was staying green. And we checked in on it backing, as the crust turned from green to brown. We discussed green bread over and over. Important questions were asked and answers were attempted. And then, of course, the moment of truth removing the loaf from the pan. The boys jabbered excitedly that there was some green peeking out. While we waited for it to cool, we went out to the creek. The creek is dry this year, so we got to walk in it. Jacob used his stick to make a line behind him. I asked him, Is that a line so we can find our way back? Oh! Uhm. Yes! And then he added arrows so we d know which way to go. Jacob stopped every 20 or so feet to collect pretend train tickets from Oliver and me. Oliver eventually grew tired of this, so Jacob started collecting Oliver s ticket from me. They climbed on some trees, managed to find some mud, drew outlines of train cars in the dirt, and then followed Jacob s line back down the creek bed. They pointed out any green things they saw. Then we went back to the house, took off our warm coats, and cut into the bread. Can you imagine the excitement? I hadn t realized green is a flavor, but it must have been somehow, because those boys absolutely loved this green bread. When we got out the jam, Jacob realized that it was red on the green bread, and that now his bread was Christmas-colored. All sorts of green bread discoveries were made, but the best among them was that if you hold a slice of green bread up to the bright sun, the sun makes it glow green and it looks like a stained-glass window. Sometimes a few drops of food coloring can add a ton of excitement to a day.

when it s a prediction . In the 4th January edition of the Guardian Weekly , the front page story, entitled Meet the world s new boomers ¹ contained this little gem:

Back in 2006, [PricewaterhouseCoopers] made some forecasts about what the global economy might look like in 2050, and it has now updated the predictions in the light of the financial crisis and its aftermath.

Delightful. They made some forecasts about what the global economy might look like. Given that they clearly didn t include any impact of the GFC in their forecasts, it clearly wasn t a particularly accurate forecast. Y know what an inaccurate prediction is called? Guesswork. Let s call a spade a spade here. I see this all the time, and it s starting to shit me. People making predictions and forecasts and projections hither and yon, and they re almost always complete bollocks, and they never get called on it. I read the Greater Fool blog now and then, and that blog is chock full of examples of people making predictions which have very little chance of being in any way accurate. While Dr Ben Goldacre and others are making inroads into requiring full disclosure in clinical trials, I m not aware of anyone taking a similar stand against charlatans making dodgy-as-hell predictions over and over again, with the sole purpose of getting attention, without any responsibility for the accuracy of those predictions. Is anyone aware of anyone doing work in this area, or do I need to register badpredictions.net and start calling out dodginess?

Review: Fantasy & Science Fiction, March/April 2011

Editor:	Gordon van Gelder
Issue:	Volume 120, No. 3 & 4
ISSN:	1095-8258
Pages:	258

Charles de Lint's book review column sticks with the sorts of things he normally reviews: urban and contemporary fantasy and young adult. Predictably, I didn't find that much of interest. But I was happy to see that not all the reviews were positive, and he talked some about how a few books didn't work. I do prefer seeing a mix of positive and negative (or at least critical) reviews. James Sallis's review column focuses entirely on Henry Kuttner and C.L. Moore (by way of reviewing a collection). I'm always happy to see this sort of review. But between that and de Lint's normal subject matter, this issue of F&SF was left without any current science fiction reviews, which was disappointing. Lucius Shepard's movie review column features stunning amounts of whining, even by Shepard's standards. The topic du jour is how indie films aren't indie enough, mixed with large amounts of cane-shaking and decrying of all popular art. I find it entertaining that the F&SF film review column regularly contains exactly the sort of analysis that one expects from literary gatekeepers who are reviewing science fiction and fantasy. Perhaps David Langford should consider adding an "As We See Others" feature to Ansible cataloging the things genre fiction fans say about popular movies. "Scatter My Ashes" by Albert E. Cowdrey: The protagonist of this story is an itinerant author who has been contracted to write a family history (for $100K, which I suspect is a bit of tongue-in-cheek wish fulfillment) and has promptly tumbled into bed with his employer. But he is somewhat serious about the writing as well, and is poking around in family archives and asking relatives about past details. There is a murder (maybe) in the family history, not to mention some supernatural connections. Those familiar with Cowdrey's writing will recognize the mix of historical drama, investigation, and the supernatural. Puzzles are, of course, untangled, not without a bit of physical danger. Experienced fantasy readers will probably guess at some of the explanation long before the protagonist does. Like most Cowdrey, it's reliably entertaining, but I found it a bit thin. (6) "A Pocketful of Faces" by Paul Di Filippo: Here's a bit of science fiction, and another mystery, this time following the basic model of a police procedural. The police in this case are enforcing laws around acceptable use of "faces" in a future world where one can clone someone's appearance from their DNA and then mount it on a programmable android. As you might expect from that setup, the possibilities are lurid, occasionally disgusting, and inclined to give the police nightmares. After some scene-setting, the story kicks in with the discovery of the face of a dead woman who, at least on the surface, no one should have any motive to clone. There were a few elements of the story that were a bit too disgusting for me, but the basic mystery plot was satisfying. I thought the ending was a let-down, however. Di Filippo tries to complicate the story and, I thought, went just a little too far, leaving motives and intent more confusing than illuminating. (6) "The Paper Menagerie" by Ken Liu: Back to fantasy, this time using a small bit of magic to illustrate the emotional conflicts and difficulties of allegiance for second-generation immigrants. Jack is the son of an American farther and a Chinese mother who was a mail-order bride. He's young at the start of the story and struggling with the embarassment and humiliation that he feels at his mother's history and the difficulties he has blending in with other kids, leading to the sort of breathtaking cruelty that comes so easily from teenagers who are too self-focused and haven't yet developed adult empathy. I found this very hard to read. The magic is beautiful, personal, and very badly damaged by the cruelty in ways that can never really be fixed. It's a sharp reminder of the importance of being open-hearted, but it's also a devastating reminder that the lesson is normally learned too late. Not the story to read if you're prone to worrying about how you might have hurt other people. (6) "The Evening and the Morning" by Sheila Finch: This long novella is about a third of the issue and is, for once, straight science fiction, a somewhat rare beast in F&SF these days. It's set in the far future, among humans who are members of the Guild of Xenolinguists and among aliens called the Venatixi, and it's about an expedition back to the long-abandoned planet of Earth. I had a lot of suspension of disbelief problems with the setup here. While Earth has mostly dropped out of memory, there's a startling lack of curiosity about its current condition among the humans. Finch plays some with transportation systems and leaves humanity largely dependent on other races to explain the failure to return to Earth, but I never quite bought it. It was necessary to set up the plot, which is an exploration story with touches of first contact set on an Earth that's become alien to the characters, but it seemed remarkably artificial to me. But, putting that aside, I did get pulled into the story. Its emotional focus is one of decline and senescence, a growing sense of futility, that's halted by exploration, mystery, and analysis. The question of what's happened on Earth is inherently interesting and engaging, and the slow movement of the story provides opportunities to build up to some eerie moments. The problem, continuing a theme for this issue, is the ending. Some of the reader's questions are answered, but most of the answers are old, well-worn paths in science fiction. The emotional arc of the story is decidedly unsatisfying, at least to me. I think I see what Finch was trying to do: there's an attempted undermining of the normal conclusion of this sort of investigation to make a broader point about how to stay engaged in the world. But it lacked triumph and catharsis for me, partly because the revelations that we get are too pedestrian for the build-up they received. It's still an interesting story, but I don't think it entirely worked. (6) "Night Gauntlet" by Walter C. DeBill, Jr., et al.: The full list of authors for this story (Walter C. DeBill, Jr., Richard Gavin, Robert M. Price, W.H. Pugmire, Jeffrey Thomas, and Don Webb) provides one with the first clue that it's gone off the rails. Collaborative storytelling, where each author tries to riff off the work of the previous author while spinning the story in a different direction, is something that I think works much better orally, particularly if you can watch facial expressions while the authors try to stump each other. In written form, it's a recipe for a poorly-told story. That's indeed what we get here. The setup is typical Cthulhu mythos stuff: a strange scientist obsessed with conspiracy theories goes insane, leaving behind an office with a map of linkages between apparently unrelated places. The characters in the story also start going insane for similar reasons, leading up to a typical confrontation with things man was not meant to know, or at least pay attention to. If you like that sort of thing, you may like this story better than I did, but I thought it was shallow and predictable. (3) "Happy Ending 2.0" by James Patrick Kelly: More fantasy, this time of the time travel variety. (I call it fantasy since there's no scientific explanation for the time travel and it plays a pure fantasy role in the story.) That's about as much as I can say without giving away the plot entirely (it's rather short). I can see what Kelly was going for, and I think he was largely successful, but I'm not sure how to react to it. The story felt like it reinforced some rather uncomfortable stereotypes about romantic relationships, and the so-called happy ending struck me as the sort of situation that was going to turn very nasty and very uncomfortable about five or ten pages past where Kelly ended the story. (5) "The Second Kalandar's Tale" by Francis Marion Soty: The main question I have about this story is one that I can't answer without doing more research than I feel like doing right now: how much of this is original to Soty and how much if it is straight from Burton's translation of One Thousand and One Nights. Burton is credited for the story, so I suspect quite a lot of this is from the original. Whether one would be better off just reading the original, or if Soty's retelling adds anything substantial, are good questions that I don't have the background to answer. Taken as a stand-alone story, it's not a bad one. It's a twisting magical adventure involving a djinn, a captive woman, some rather predictable fighting over the woman, and then a subsequent adventure involving physical transformation and a magical battle reminiscent of T.H. White. (Although I have quite likely reversed the order of inspiration if as much of this is straight from Burton as I suspect.) Gender roles, however, are kind of appalling, despite the presence of a stunningly powerful princess, due to the amount of self-sacrifice expected from every woman in the story. Personally, I don't think any of the men in the story are worth anywhere near the amount of loyalty and bravery that the women show. Still, it was reasonably entertaining throughout, in exactly the way that I would expect a One Thousand and One Nights tale to be. Whether there's any point in reading it instead of the original is a question I'll leave to others. (7) "Bodyguard" by Karl Bunker: This is probably the best science fiction of the issue. The first person protagonist is an explorer living with an alien race, partly in order to flee the post-singularity world of uploaded minds and apparent stagnation that Earth has become. It's a personal story that uses his analysis of alien mindsets (and his interaction with his assigned alien bodyguard) to flesh out his own desires, emotional background, and reactions to the world. There are some neat linguistic bits here that I quite enjoyed, although I wish they'd been developed at even more length. (The alien language is more realistic than it might sound; there are some human languages that construct sentences in a vaguely similar way.) It's a sad, elegiac story, but it grew on me. (7) "Botanical Exercises for Curious Girls" by Kali Wallace: One has to count this story as science fiction as well, although for me it had a fantasy tone because the scientific world seems to play by fantasy rules from the perspective of the protagonist. Unpacking that perspective is part of the enjoyment of the story. At the start, she seems to be a disabled girl who is being cared for by a strange succession of nurses who vary by the time of day, but as the story progresses, it becomes clear that something much stranger is going on. There are moments that capture a sense of wonder, reinforced by the persistantly curious and happy narrative voice, but both are undercut by a pervasive sense of danger and dread. This is a light story written about rather dark actions. My biggest complaint with the story is that it doesn't so much end as wander off into the sunset. It set up conflict within a claustrophobic frame, so I can understand the thematic intent of breaking free of that frame, but in the process I felt like the story walked away from all of the questions and structure that it created and ended in a place that felt less alive with potential than formless and oddly pointless. I think I wanted it to stay involved and engaged with the environment it had created. (6) "Ping" by Dixon Wragg: I probably should just skip this, since despite the table of contents billing and the full title introduction, it's not a story. It's a two-line joke. But it's such a bad two-line joke that I had to complain about it. I have no idea why F&SF bothered to reprint it. (1) "The Ifs of Time" by James Stoddard: This certainly fits with the Arabian Nights story in this issue. The timekeeper of a vast and rambling castle (think Gormenghast taken to the extreme) wanders into a story-telling session in a distant part of the castle. The reader gets to listen to four (fairly good) short stories about time, knowledge, and memory, told in four very different genres. All of this does relate to why the timekeeper is there, and the frame story is resolved by the end, but the embedded stories are the best part; each of them is interesting in a different way, and none of them outlast their welcome. This was probably the strongest story of this issue. (7) Rating: 6 out of 10

It's been a long time since I've posted any photographs here (about 15 months, it looks like), so have some clouds. This also serves as a test for whether the scripts I use to post photographs have been properly converted to Git. (And a test to see if the occasional amateur photograph eating up screen real estate will provoke annoyed mutters from the direction of Planet Debian, although so far no complaints over my even-longer book reviews.) I'm still taking photographs occasionally, although not as much as I was at one point. Too many hobbies and not enough time. Part of why I've been slow in posting them is that I was bad about keeping notes about where I took them and bad about keeping up with sorting through them. So I have large collections of photographs from some days where I don't remember exactly where I was, and where I've not already singled out the ones that are worth sharing. One of the things my hobbies always seem to generate is yet another pile of stuff that's not organized quite to my liking. Maybe some evening I'll feel inspired to do a bit more sorting. This is a long weekend in the US for some of us, myself included, so I get another day in this weekend. Which is good, since I've only gotten to a fraction of what I wanted to get to, although at least I found several hours to play video games today. I have finally converted PGP::Sign to Git, so I have no more personal Subversion repositories, but I forgot how much code I'd written to handle its build-time configuration. All of that desperately needs to be reworked, so it's going to take a bit longer than I expected before I can push out another release.

In December, I wrote a post on the progress of the Wheezy release. Back then, we had 348 RC bugs in testing and today there are about 249 left. A bit of simple math put us at 99 RC bugs fixed since last and an average of about 2.4 RC bugs fixed per day (up from 1.8). Assuming a constant rate, we would be able to release in about 100 days or 3 months (plus 1 week or two) from now. If you want the release earlier than that, fix RC bugs even faster! Though, the data we are looking might be inaccurate. We filed a bug against UDD, which claims that #639407 affects Wheezy while it is in fact fixed as far as the BTS is concerned. If UDD is only wrong in this direction, we can hope for a positive surprise when the UDD bug is fixed. On the other hand, we may also be unpleasently surprised if not. While talking about the UDD, I would like to mention a new feature its bug search script. It is now possible to filter based on whether or not a package is unblocked. If you take this into account (and assume that all unblocked packages will migrate in a timely manner), our RC bug count drops to about 219. It also gives us a much better view of how many packages that could actually use an unblock. which is now down to about 61. In case you have been wondering what is up with the removal of RC buggy leaf packages . We have still been looking at those, but for the last couple of times there have been at most one candidate for a given run (obviously not the same package each time). It felt a bit excessive to send an email to debian-devel for just one RC bug.

What is Valgrind? Valgrind is a framework for building dynamic analysis tools. Several useful tools are included in the Valgrind distribution including tools to check dynamic memory usage (memcheck), a cache profiler (cachegrind), heap profiler (massif) and thread debugger (helgrind) among others. Valgrind also provides a framework which can be used to build other tools. The Valgrind tool which I find most useful and the one which I have most experience with is memcheck. This tool can detect all manner of memory management problems, including use after free, using uninitialized data, memory leaks, double free. Between them these can result in savings of many hours of staring a core dumps and gdb backtraces. How Does memcheck Work? At its core Valgrind is a dynamic binary translation engine, which is used to instrument the code at runtime. In the case of memcheck this is used to track properties of the memory in a process, including aspects such as whether each bit (yes, bit) of memory has been initialized since it was allocated. It also tracks this information as data is loaded into registers, so it can know if a given register is currently tainted with uninitialized data or not. As well as instrumentation through binary translation Valgrind also includes instrumented versions of the C dynamic memory allocation functions which are used to track whether a each bit of memory is currently considered free or allocated, as well as tainting registers when they contain a pointer to memory which has been freed. A large part of memcheck's functionality is built upon Valgrind's ability to determine when memory has been initialized. However although binary translation can be used to instrument when the application itself has initialized some memory it is not currently possible to instrument the behaviour of system calls, in other words Valgrind cannot automatically determine when memory has been initialised e.g. after a read(2) system call. For this reason Valgrind has baked in knowledge about the behaviour of system calls on various platforms. For example lets consider the read(2) system call. The prototype of read(2) is:

    ssize_t read(int fd, void *buf, size_t count);

This system call reads up to count bytes from the file descriptor fd into the memory pointed to by buf and returns the number of bytes read. Here is the code within Valgrind which handles this system call (found in coregrind/m_syswrap/syswrap-generic.c):

PRE(sys_read)
 
   *flags  = SfMayBlock;
   PRINT("sys_read ( %ld, %#lx, %llu )", ARG1, ARG2, (ULong)ARG3);
   PRE_REG_READ3(ssize_t, "read",
                 unsigned int, fd, char *, buf, vki_size_t, count);
   if (!ML_(fd_allowed)(ARG1, "read", tid, False))
      SET_STATUS_Failure( VKI_EBADF );
   else
      PRE_MEM_WRITE( "read(buf)", ARG2, ARG3 );
 
POST(sys_read)
 
   vg_assert(SUCCESS);
   POST_MEM_WRITE( ARG2, RES );
 

These two functions are called before and after the system call respectively. The main piece of functionality is the calls to PRE_MEM_WRITE and POST_MEM_WRITE. In this case PRE_MEM_WRITE is used to check that the memory starting at ARG2 (recall that this is the supplied buffer) and extending for ARG3 bytes (the system calls count argument) consists of allocated memory. After the system call in complete the post function calls POST_MEM_WRITE to indicate to Valgrind that RES bytes of the buffer have know been initialized. The above is a relatively simple case, but the above two hooks must be supplied for each system call on each platform which Valgrind wishes to support. One of the most problematic system calls is the ioctl(2) system call. The ioctl call takes a file descriptor, a request number and a pointer to a per-ioctl argument structure and implements per-device control requests. There are literally dozens of devices many with their own specific ioctls. For this reason a sizable portion of Valgrind code is dedicate to decoding the behaviour of the myriad of potential ioctls and providing pre- and post-call instrumentation for them. Interacting with the Hypervisor from Userspace Under Xen the toolstack is a normal userspace process running in the control domain (typically domain 0). As part of its operation the toolstack needs to communicate with the hypervisor and to do this it uses hypercalls. However a userspace process cannot simply make hypercalls on its own, instead it must request that the OS kernel make the hypercall for it. This prevents just any userspace process making a hypercall, since the kernel will first check the privilege of the process making the request. Under Linux a userspace process makes hypercalls by using the ioctl(2) system call on a special "Privileged Command" (privcmd) device. In this case the toolstack uses the IOCTL_PRIVCMD_HYPERCALL ioctl request which takes a hypercall number and the hypercall arguments as its argument. Other OSes which can be used as a control domain posses similar interfaces. Making memcheck Work With Xen Toolstacks? As might be expected the majority of the work to allow Valgrind to work on Xen toolstack processes was teaching it about the IOCTL_PRIVCMD_HYPERCALL ioctl. Fortunately it was not necessary to teach Valgrind about every hypercall but only about those which are used by toolstacks. This is a smallish subset of all hypercalls including the domctl interfaces and a handful of others. The initial batch of patches cover all of the system calls needed to start new domains, shut them down and migrate them using the XL toolstack. New hypercalls will be added as they are discovered to be missing. Getting and Using This Functionality The initial set of patches were accepted into Valgrind's Subversion repository trunk revision 13081 in October 2012 and are expected to be part of the 3.9.0 release. Until 3.9.0 is released it will be necessary to get this functionality from SVN. Fortunately this is relatively simple and follows the usual autofoo dance. Although the usual caveats regarding running unreleased software apply.

  $ svn co svn://svn.valgrind.org/valgrind/trunk valgrind
  $ cd valgrind
  $ ./autogen.sh
  $ ./configure
  $ make
  $ sudo make install

Once Valgrind is built and installed simply call it passing the command to be debugged as an option:

  # valgrind xl list
  ==16186== Memcheck, a memory error detector
  ==16186== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
  ==16186== Using Valgrind-3.9.0.SVN and LibVEX; rerun with -h for copyright info
  ==16186== Command: xl list
  ==16186== 
  Name                                        ID   Mem VCPUs    State   Time(s)
  Domain-0                                     0   512     4     r-----     332.5
  ==16186== 
  ==16186== HEAP SUMMARY:
  ==16186==     in use at exit: 0 bytes in 0 blocks
  ==16186==   total heap usage: 16 allocs, 16 frees, 93,529 bytes allocated
  ==16186== 
  ==16186== All heap blocks were freed -- no leaks are possible
  ==16186== 
  ==16186== For counts of detected and suppressed errors, rerun with: -v
  ==16186== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 39 from 6)

Here we see that xl list has no leaks! However if I comment out the call to libxl_dominfo_list_free in tools/libxl/xl_cmdimpl.c:main_list() then instead Valgrind reports:

[...]
  ==16203== HEAP SUMMARY:
  ==16203==     in use at exit: 90,112 bytes in 1 blocks
  ==16203==   total heap usage: 16 allocs, 15 frees, 93,529 bytes allocated
  ==16203== 
  ==16203== LEAK SUMMARY:
  ==16203==    definitely lost: 0 bytes in 0 blocks
  ==16203==    indirectly lost: 0 bytes in 0 blocks
  ==16203==      possibly lost: 90,112 bytes in 1 blocks
  ==16203==    still reachable: 0 bytes in 0 blocks
  ==16203==         suppressed: 0 bytes in 0 blocks
  ==16203== Rerun with --leak-check=full to see details of leaked memory
  ==16203== 
  ==16203== For counts of detected and suppressed errors, rerun with: -v
  ==16203== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 39 from 6)
[...]

So it has caught the memory leak! Rerunning with the suggests --leak-check=full goes even further and tells me where the leaked memory was allocated:

[...]
  ==16204== 90,112 bytes in 1 blocks are possibly lost in loss record 1 of 1
  ==16204==    at 0x4024480: calloc (vg_replace_malloc.c:593)
  ==16204==    by 0x4050CE4: libxl_list_domain (libxl.c:548)
  ==16204==    by 0x805EEC7: main_list (xl_cmdimpl.c:3931)
  ==16204==    by 0x804D6DD: main (xl.c:285)
[...]

To ease debugging of domain creation I find it useful to use the -F option to xl create to stop xl from forking and daemonising. Most xl subcommands do not daemonize and so need no special treatment. For more information on running Valgrind see the Valgrind Documentation. Conclusion Valgrind is a powerful tool in the arsenal of the C programmer, and can be used to catch a large number of hard to debug and common issues before they happen can save many hours of tedious debugging. The ability to use tools such as this on Xen toolstack processes is of enormous benefit and has already found several bugs in the XL toolstack.

What I have been up to over the last couple of years

I have now worked at credativ for four years. A sizable part of that time I spent working with the Munich City Council's LiMux project. The focus of our work is helping them with their LDAP/web-based GOsa user and system installation(based on FAI)/configuration/management system. Personally, I mostly did project management and QA besides occasional coding, bug fixing and on-site consulting.

Contrary to a lot of our other customers, the Munich city council openly acknowledges the work by external companies and lets us talk about it as well. I wrote a lengthy blog post (now available in english) last month about our work. It explains how the LiMux project is using GOsa and FAI and work we did for them.

The project took a long time, but managed to pull off the mass migration over the last two years and reached their goal of 12000 migrated workstation last November. It is now widely regarded as a "success story" by german news sites, and I am glad to have had a small part in it. I really think this was (and still is) an important project, especially after several others backpedalled and Munich remained the last high-profile public sector migration in the german speaking countries.

Besides that, the LiMux project members are really nice people I get along with well both on working hours and during the Munich Debconf11 bid and organizing various BSPs at their office. It is a bit of a shame that not more of their work and customizations are available in Debian, but hopefully this well happen one day. At least our changes to GOsa are up on github.

Review: Soulless, by Gail Carriger

Series:	Parasol Protectorate #1
Publisher:	Orbit
Copyright:	October 2009
ISBN:	0-316-07165-X
Format:	Kindle
Pages:	357

Alexia Tarabotti is a spinster in London society in the time of Queen Victoria. She is possessed of a hopelessly overwrought and society-focused mother, two half-sisters whose only thoughts are for fashion and gossip, a mixed Italian heritage, an unpopular body shape and complexion, and a secret. In a world in which vampire hives, werewolf packs, and the occasional ghost are out of hiding and openly participating in society, and where the Bureau of Unnatural Registry (headed by a particularly intimidating werewolf who is also an earl) keeps track of the undead, Alexia is something still rarer, and not entirely acceptable. She is preternatural, able to draw the supernatural out of the undead with a touch. She has no soul. In Carriger's Victorian urban fantasy, laced with a gentle helping of steampunk, vampires, werewolves, and ghosts all have similar conditions. All three are undead; even the change for werewolves happens after normal death. But the change doesn't happen to everyone killed by the corresponding type of undead. The requirements are quite exacting and include the quantity of soul in the person being changed. Only those with a surfeit of soul, too much to stay dead, can be successfully changed. Vampires and werewolves both are surrounded by hopeful humans, called drones and clavigers respectively, who hope to be transformed and gain immortality. But no one can tell in advance how much soul someone has. Alexia is the precise opposite of the undead. Rather than a surfeit of soul, she has none at all, a condition inherited from her father but unknown to the rest of her family. Her kind, the preternatural, make up the vampire and werewolf hunters of legend, called soul-suckers, curse-breakers, and exorcists by the supernatural set. She's viewed with great suspicion by most, but with fascination by some for her ability to revert them to humanity with physical contact. Not that this has been helpful in finding a husband, or, really, for much of anything else in her life. Soulless opens with Alexia being accousted by a rogue vampire in the library, where she went for refuge from a private ball. Not only does he not know who she is, despite every hive in the area telling all of their new vampires about her, but he doesn't seem to even know what a preternatural is, and she has to kill him with her parasol. That is the first unusual occurance in an escalating mystery involving new vampires appearing from nowhere, loner vampires and werewolves disappearing, and a truly prodigious use of chloroform. Not to mention quite a few encounters with Lord Maccon, Earl of Woolsey, the Alpha werewolf who runs the Bureau of Unnatural Registry. The start of Soulless feels a bit uneven and stilted until the story gets moving and the reader gets used to Carriger's voice. It's written in the style of a comedy of manners and takes delight in long words and a semi-formal style. For example:

The vampire recovered his equanimity quickly enough. He reared away from Alexia, knocking over a nearby tea trolley. Physical contact broken, his fangs reappeared. Clearly not the sharpest of prongs, he then darted forward from the neck like a serpent, diving in for another chomp. "I say!" said Alexia to the vampire. "We have not even been introduced!"

Sometimes this works. Sometimes it doesn't. The frequency with which it works seemed to me to increase as the book went along. The best thing this book has going for it is the character of Alexia Tarabotti. Rather than the typical kick-ass female protagonist of a modern urban fantasy novel, Alexia is not cynical, battle-hardened, or courageously sarcastic. Her attitude for most of the book is something closer to bemusement crossed with exasperation. She sees herself as an outsider, has largely given up on fitting into any sort of proper society, and has little use for the expectations of her family. But she's still embedded within her culture, still shocked (at least momentarily) by breaches in etiquette, and quite able to fit in, if a bit uncomfortably. Her delightfully logical tone and constant open-minded self-analysis is what made this book for me. The plot is fairly straightforward (and has a bit too much anti-science flavor for me). The romance angle, which is reasonably obvious from early on in the book, was entertaining mostly in Alexia's combination of thoughtful analysis (based in part on her father's extensive and not entirely acceptable library) and runaway physical reactions, although I thought the latter was slightly overdone. But all the elements of the story improve noticably towards the end of the book. I was unenthused by the halfway point, but by the time I finished Soulless, I decided I would probably read the sequel. This is more in the enjoyably fun reading pile than a book that I would go to great effort to acquire, and it has a somewhat rocky start, but I enjoyed it and would mildly recommend it. Followed by Changeless. Rating: 7 out of 10

I've been lucky in life. I've known several people who've killed themselves, but I've been friendly with them rather than friends with them. Their deaths have saddened me, but haven't left giant gaping holes that I have trouble imagining filling. I've also been lucky in that I'm not especially predisposed towards depression, and even luckier that the one serious episode I had ended well. That luck means I'm not qualified to speak authoritatively on the topic, but this seems like a good time to share. Most of 2002 wasn't great for me. Many of my friends had graduated and moved away. My PhD, which was what I'd spent the past few years of my life working towards, was going horribly wrong. The girl I liked didn't like me. Lots of individual small things that didn't fundamentally matter, but which build up into overall feelings of isolation and failure at life. I was spending an increasing number of days not leaving bed and not talking to people. It wasn't that I couldn't have fun - socialising was still pleasurable, and I didn't actively avoid people, but it was always tempered with the knowledge that this was temporary and I'd be returning to the unhappiness afterwards. I couldn't see any sequence of events that would turn my life around and restore my happiness. This was how life was, and this was how life was going to be. I'd irrevocably fucked up, and this was my future. Looking back, what strikes me is how reasonable this seemed. I could point at specific things that were making me unhappy, and nobody could argue that it was unreasonable to be unhappy about them. There wasn't any point in speaking to people about it, because what would they be able to do except agree that I was justifiably unhappy? I thought about suicide. Not seriously, because overall life still seemed worthwhile, but I could conceive of a level of further unhappiness that would make it seem like the best choice. I don't think it would ever have occured to me to speak to someone about it first. It seemed like it would be the same argument - I'm justifiably unhappy, I already feel like I'm letting my friends down, what could they do other than tell me that my feelings are wrong or make me feel even more guilty? So, when I saw exhortions for people to speak to someone if they felt suicidal, it seemed like they were talking to people who hadn't thought this through as well as me. It felt like I'd thought this all through carefully and rationally and come to a completely logical decision. If changes in circumstances and further consideration made it seem like suicide was the better choice, that would be because it was the better choice. Maybe other people weren't thinking about this as logically as I was. Maybe they'd have their minds changed by speaking to a friend or a professional. I wouldn't. Of course, with hindsight I was rationalising the way I already felt rather than making entirely rational decisions. I could have rationalised myself to death even though there were (in my case) straightforward ways to make my life better. In any case, I've no idea how close I ever got to that point. Things were at their worst in August - by September I had a new job and new house, and things just got better from there. In the end, the friends I was convinced could do nothing for me ended up giving me the opportunity to find gainful employment and made sure I had somewhere to live. Without them, things might have been different. As it was, I spent less than nine months depressed and it was still the most hellish experience of my life. The thought of returning to that state is terrifying. I was lucky. I might not be again. There's no terribly good moral here. If I'd known more about depression beforehand, I might have been able to identify what was happening to me and seek professional help. Other than that, I didn't learn anything about how to avoid or deal with depression. The experience didn't make me a better person. I've no advice for people in the same situation. The only thing I gained from it all was the realisation that if I'd felt any worse and knew that this was what I faced for the rest of my life, death might not have been the worst choice I had. Depression is a huge social problem and we deal with it badly. We refuse to talk about it, and when we do talk about it we mostly limit ourselves to platitudes about how things will get better or placing the blame on depressed people for not wanting to talk to those around them. Sometimes it doesn't get better. Sometimes talking to those around you will make things worse. People need to be aware of what the effects of depression are and get better at identifying it in others, rather than assuming that they'll be able to ask for help themselves. Society as a whole needs to be better at ensuring that professional support is there for people who need it. And, unless we can make massive improvements in our understanding of the causes of depression and effective mechanisms for countering it, we need to accept that it will cost us friends. Let's redirect the anger we feel at their choice to avoid a lifetime of misery into anger at the society that still hasn't done everything it can to help them.

comments

Daniel Kahn Gillmor has a very good blog post. You should read it. It includes a thoughtful rebuttal to some of my earlier thoughts about activism. I think I'm developing a richer understanding of where I see boundaries here, but after my last post, I also realized that by focusing on the specific details of what should have been a minor alleged crime, I'm derailing. Swartz did so much else. I made a note to come back to the more theoretical discussion in six months; now isn't the time. Now is the time to celebrate open content and all of the things Swartz achieved. (But thank you very much to the multiple people who have pointed out flaws in my reasoning and attempted approach.) Hopefully, it's also an opportunity to keep the pressure on for a saner and less abusive judicial system that doesn't threaten people with ridiculous and disproportional punishment in order to terrify them into unwarranted plea bargains. The petition I mentioned has reached nearly 40,000 signatures and passed the threshold (at the time it was posted) for forcing a White House response. Probably more importantly, it also seems to be creating the feedback cycle that I was hoping to see: the popularity of the petition is causing this story to stay in the news cycle and continue to be written about, which in turn drives more signatures to the petition. I'm not particularly hopeful that the Obama administration cares about the vast and deep problems with our criminal justice system, but I'm somewhat more hopeful that they, like most politicians, hate news cycles that they don't control. The longer this goes on, the stronger the incentive to find some way to make it go away, which could lead to real disciplinary action. A key committee in the US House of Representatives is starting a formal investigation. One of my local representatives has proposed modifying the US federal law on computer fraud and abuse to remove violations of terms of service from the definition of the crime. (I don't have much hope that this will pass when proposed by the minority party in a fairly hostile House, but the mere act of proposing it keeps the news focus on.) Glenn Greenwald has a (typically long-winded) round-up of news in the Guardian. Note that both Greenwald and Declan McCullagh link directly to the petition in articles in mainstream news outlets. One thing that slacktivism can do is perpetuate a news cycle until it gets more uncomfortable for people in power. It's still nowhere near as effective as the types of activism that Swartz was so good at, but in this specific case I think one gets a reasonable return on one's five-minute investment of effort. I'm going to stop talking about this now, since other people are a lot better at this sort of post than I am. But one last link: the medical community has a related open content problem, and theirs is also killing people. Possibly people you know. If all of this has inspired you, as it has me, to care even more about open content, be watching the push for open access to clinical trial data. More background is in Ben Goldacre's TED talk.

According to the bureau of meteorology today is 39C. But mad dogs and Ingressmen go out in the midday sun, so I took advantage of some spare time to capture a couple of portals. After that my phone battery was apparently at 46C and my phone refused to charge. It seems that in addition to the range of hardened phone cases we need some cooling cases for phones. A case that contained a substance with a melting point of 39C wouldn t melt from body heat but would set an upper limit on the phone temperature. A peltier device probably wouldn t work as it would take too much power (and the batteries supplying the power would produce more heat). I think that the phones with an aluminium back are the best design. Aluminium is light, reflective (unlike the black plastic which is so common), and conducts heat better than most things. A phone shell made of copper probably isn t viable due to copper being dense and soft. Another problem is the need for third party cases to protect against damage. If the phone companies designed phones to be solid, rubbery at the edges (to bounce not break) and so that the screen didn t touch the surface when the phone is face down then we could avoid phone cases which also act as thermal insulation. I am a bit disappointed in Samsung. I could understand Nokia making phones that don t survive the heat well, but I don t think that Korea is that much cooler than Australia. A phone that works well on the hottest day of summer in Seoul should do better than my Galaxy S3.

I was upset to learn about Aaron Swartz's death last week. I continue to be upset about his loss, and about our loss. He didn't just show promise of great things to come in the future -- he had already done more work for the public good than many of us will ever do. I'd only met him IRL a couple times, but (like many others) i had encountered him on the 'net in many places. He was a good person, someone i didn't need to always agree with to respect. I read Russ Allbery's posts about Aaron and "slacktivism" with much appreciation. I had been ambivalent about signing the whitehouse.gov petition asking for the removal of the prosecutor for overreach, because I generally distrust the effectiveness of online petitions (and offline petitions, for that matter). But Russ's analysis convinced me to go ahead and sign it. The petition is concrete, clear (despite wanting a grammatical proofread), and actionable. For people willing to go beyond petition signing to civil disobedience, The Aaron Swartz Memorial JSTOR Liberator is an option. It makes it straightforward to potentially violate the onerous JSTOR terms of service by re-publishing a public-domain article from JSTOR to archive.org, where it will be accessible to anyone directly. As someone who builds and maintains information/communications infrastructure, i have very mixed feelings about most online civil disobedience, since it often takes the form of a Distributed Denial of Service (DDoS) attack of some sort. DDoS attacks of public services are notoriously difficult to defend against without having huge resources to throw at the problem, so encouraging participation in a DDoS often feels a little bit like handing out cans of gasoline when you know that everyone is living in a house of straw. However, the JSTOR Liberator is not a DDoS at all -- it's simply a facilitation of people breaking the JSTOR Terms of Service (ToS), some of the same terms that Aaron was facing charges for violating. So it is a well-targeted way to demonstrate that the prosecutions were overreaching. I wanted to take issue with one of Russ' statements, though. In his second post about the situation, Russ wrote:

Social activism and political disobedience are important and often valuable things, but performing your social activism using other people's stuff is just rude. I think it can be a forgivable rudeness; people can get caught up in the moment and not realize what they're doing. But it's still rude, and it's still not the way to go about civil disobedience.

While i generally agree with Russ' thoughtful consideration of consent, I have to take issue with this elevation of some sort of hyper-extended property right over the moral agency that drives civil disobedience. To use someone else's property for the sake of a just cause without damaging the property or depriving the owner of its use is not "forgivable rudeness" -- it's forgivable, laudable even, because it is just. And the person using the property doesn't need to be "caught up in the moment and not realize what they're doing" for it to be acceptable. Civil disobedience often involves putting some level of inconvenience or discomfort on other people, including innocent people. It might be the friends and family of the activist who have to deal with the jail time; it might be the drivers stuck in a traffic jam caused by a demonstration; it might be the people forced to shop elsewhere because the store's doors are barricaded by protestors. All of these people could be troubled by the civil disobedience more than MIT's network users and admins were troubled by Aaron's protest, and that doesn't make the protests described worse or "not the way to go about civil disobedience." The trouble highlights a more significant injustice, and in its troubling way does what it can to help right it. Aaron was a troublemaker, and a good one. He will be missed. Tags: aaronsw

I got some feedback that the analogy I used in my last post was a bit confusing, and indeed I blew the phrasing of the analogy (also now corrected). So let me try this again, since I think there's a subtlety here that may be missed. I should note for the record that my understanding of what Swartz did that started the process is apparently somewhat based on the description from the prosecution, so it may not be the complete or accurate facts. Since there will now be no trial, we may never find out what the defense was, and whether those facts would be challenged. So it may be best to think of this as a hypothetical. We never established, or will establish, in court exactly what happened. Swartz was, generally speaking, charged with two things that I consider quite distinct, at least from an ethical perspective. Most of the focus is on the copyright part: downloading JSTOR articles with an intent (never acted upon) to distribute them to the world. There are a bunch of reasons why this may or may not be justified, which are tied into the origin of those articles (many of them were publicly funded) and the legitimacy of copyright licensing agreements. I think there is significant room to hold a variety of opinions on this, although I don't believe that "crime worth 35 years in prison" is even remotely close to justifiable under any interpretation. However, there is a second part of what he was charged with, and that's primarily what I was commenting on, not the copyright part. He allegedly hooked up a laptop in an unlocked MIT wiring closet without permission and then used MIT's network and JSTOR license to download information. This is, to me, a subtlely but entirely distinct act from the question of whether taking JSTOR's data was ethically or legally wrong. Whether or not one believes that JSTOR's copyrights are not legitimate, it's still not okay to use someone else's network and license or to trespass in their wiring closets without permission. (I work in central IT for a university, so this strikes closer to home.) And this is where the analogy came in, which I flubbed. I had originally said that this was akin to "traipsing into someone's barn or backyard shed without their permission and taking some of their tools because you want to use them." The word "taking" was wrong; it falsely implies that you weren't going to return the tools. I had been thinking "taking and then returning" when I wrote that, and the important second part didn't make it into the post. So let's try this again: Swartz's actions at MIT, as I understand them (and many things could change this, such as a revelation that he had MIT's prior permission), are akin to going onto someone's property and into their barn without permission, borrowing their hedge trimmer for a while because you want to use it, and returning the hedge trimmer without any damage when you're done (and without them noticing it was gone). I'm quite fond of this analogy, since I think it clearly establishes two things:

1. Most people are going to be unhappy about this happening and will intuitively feel like it should probably be illegal. Not everyone; there are folks who don't believe in personal property, or at least wouldn't extend it to tools in a barn. But most people will feel that someone should ask first before they come borrow your tools. Even if they don't damage them, even if they return them before you notice they're gone, you might have wanted to use the tool at the same time or they might have damaged them without intending to, and they should just ask first. It's common politeness, and depending on the circumstances, someone who doesn't ask and is covert about borrowing tools might be worth calling the police over.
2. There is absolutely no way in any reasonable moral system that doing this should result in 35 years in prison. Or even 10, or even 1. Yes, most people would consider this a crime, but most people would consider it a minor crime. It's the sort of thing where you might have to impose some consequences just to make sure the message of "knock it off" is delivered firmly, but someone doing this is rude and inconsiderate, not evil.

The JSTOR copyright stuff is more complex to analyze and is more politically divisive, but for me the key points are (a) Swartz never released the data, and (b) JSTOR declined to press charges. To me, that means the deeper copyright questions, which are quite interesting, were never actually reached in this particular case. The crime that did apparently happen was the trespass at MIT, for which I think the above analogy is the right way to think about it. The point I do want people to take away from this is that one should not overlook the trespass at MIT even if one wants to celebrate the undermining of the copyright regime and doesn't believe JSTOR's data should be considered their private property. Social activism and political disobedience are important and often valuable things, but performing your social activism using other people's stuff is just rude. I think it can be a forgivable rudeness; people can get caught up in the moment and not realize what they're doing. But it's still rude, and it's still not the way to go about civil disobedience. For both ethical and tactical reasons, involving bystanders in your act of social activism without their consent is a bad approach. ETA: The problem, of course, with discussing all of this is that while it's relevant and possibly even somewhat important in the broader sense of how our community acts going forward, it also doesn't capture the fact that this was only one incident in a remarkable life. One of the worst problems with the abusive prosecution of Swartz is that it blew this incident completely out of proportion. A moment of arguable judgement should not dominate one's life or cast a shadow over all of one's other accomplishments; the prosecution tried to make it do just that. That's part of what I'm arguing, but ironically that partly feeds into the lack of proportionality in the discussion. I've gotten kind of far afield here, so let me go back and say explicitly: Swartz was a remarkable person who did much to admire and respect, and the world is a worse place without him.

While looking into how to look up Debian packages based on hardware information, to find the packages that support a given piece of hardware, I refreshed my memory regarding modalias values, and decided to document the details. Here are my findings so far, also available in the Debian Edu subversion repository: Modalias decoded This document try to explain what the different types of modalias values stands for. It is in part based on information from <URL: https://wiki.archlinux.org/index.php/Modalias >, <URL: http://unix.stackexchange.com/questions/26132/how-to-assign-usb-driver-to-device >, <URL: http://code.metager.de/source/history/linux/stable/scripts/mod/file2alias.c > and <URL: http://cvs.savannah.gnu.org/viewvc/dmidecode/dmidecode.c?root=dmidecode&view=markup >. The modalias entries for a given Linux machine can be found using this shell script:

find /sys -name modalias -print0   xargs -0 cat   sort -u

The supported modalias globs for a given kernel module can be found using modinfo:

% /sbin/modinfo psmouse   grep alias:
alias:          serio:ty05pr*id*ex*
alias:          serio:ty01pr*id*ex*
%

PCI subtype A typical PCI entry can look like this. This is an Intel Host Bridge memory controller:

pci:v00008086d00002770sv00001028sd000001ADbc06sc00i00

This represent these values:

 v   00008086  (vendor)
 d   00002770  (device)
 sv  00001028  (subvendor)
 sd  000001AD  (subdevice)
 bc  06        (bus class)
 sc  00        (bus subclass)
 i   00        (interface)

The vendor/device values are the same values outputted from 'lspci -n' as 8086:2770. The bus class/subclass is also shown by lspci as 0600. The 0600 class is a host bridge. Other useful bus values are 0300 (VGA compatible card) and 0200 (Ethernet controller). Not sure how to figure out the interface value, nor what it means. USB subtype Some typical USB entries can look like this. This is an internal USB hub in a laptop:

usb:v1D6Bp0001d0206dc09dsc00dp00ic09isc00ip00

Here is the values included in this alias:

 v    1D6B  (device vendor)
 p    0001  (device product)
 d    0206  (bcddevice)
 dc     09  (device class)
 dsc    00  (device subclass)
 dp     00  (device protocol)
 ic     09  (interface class)
 isc    00  (interface subclass)
 ip     00  (interface protocol)

The 0900 device class/subclass means hub. Some times the relevant class is in the interface class section. For a simple USB web camera, these alias entries show up:

usb:v0AC8p3420d5000dcEFdsc02dp01ic01isc01ip00
usb:v0AC8p3420d5000dcEFdsc02dp01ic01isc02ip00
usb:v0AC8p3420d5000dcEFdsc02dp01ic0Eisc01ip00
usb:v0AC8p3420d5000dcEFdsc02dp01ic0Eisc02ip00

Interface class 0E01 is video control, 0E02 is video streaming (aka camera), 0101 is audio control device and 0102 is audio streaming (aka microphone). Thus this is a camera with microphone included. ACPI subtype The ACPI type is used for several non-PCI/USB stuff. This is an IR receiver in a Thinkpad X40:

acpi:IBM0071:PNP0511:

The values between the colons are IDs. DMI subtype The DMI table contain lots of information about the computer case and model. This is an entry for a IBM Thinkpad X40, fetched from /sys/devices/virtual/dmi/id/modalias:

dmi:bvnIBM:bvr1UETB6WW(1.66):bd06/15/2005:svnIBM:pn2371H4G:pvrThinkPadX40:rvnIBM:rn2371H4G:rvrNotAvailable:cvnIBM:ct10:cvrNotAvailable:

The values present are

 bvn  IBM            (BIOS vendor)
 bvr  1UETB6WW(1.66) (BIOS version)
 bd   06/15/2005     (BIOS date)
 svn  IBM            (system vendor)
 pn   2371H4G        (product name)
 pvr  ThinkPadX40    (product version)
 rvn  IBM            (board vendor)
 rn   2371H4G        (board name)
 rvr  NotAvailable   (board version)
 cvn  IBM            (chassis vendor)
 ct   10             (chassis type)
 cvr  NotAvailable   (chassis version)

The chassis type 10 is Notebook. Other interesting values can be found in the dmidecode source:

  3 Desktop
  4 Low Profile Desktop
  5 Pizza Box
  6 Mini Tower
  7 Tower
  8 Portable
  9 Laptop
 10 Notebook
 11 Hand Held
 12 Docking Station
 13 All In One
 14 Sub Notebook
 15 Space-saving
 16 Lunch Box
 17 Main Server Chassis
 18 Expansion Chassis
 19 Sub Chassis
 20 Bus Expansion Chassis
 21 Peripheral Chassis
 22 RAID Chassis
 23 Rack Mount Chassis
 24 Sealed-case PC
 25 Multi-system
 26 CompactPCI
 27 AdvancedTCA
 28 Blade
 29 Blade Enclosing

The chassis type values are not always accurately set in the DMI table. For example my home server is a tower, but the DMI modalias claim it is a desktop. SerIO subtype This type is used for PS/2 mouse plugs. One example is from my test machine:

serio:ty01pr00id00ex00

The values present are

  ty  01  (type)
  pr  00  (prototype)
  id  00  (id)
  ex  00  (extra)

This type is supported by the psmouse driver. I am not sure what the valid values are. Other subtypes There are heaps of other modalias subtypes according to file2alias.c. There is the rest of the list from that source: amba, ap, bcma, ccw, css, eisa, hid, i2c, ieee1394, input, ipack, isapnp, mdio, of, parisc, pcmcia, platform, scsi, sdio, spi, ssb, vio, virtio, vmbus, x86cpu and zorro. I did not spend time documenting all of these, as they do not seem relevant for my intended use with mapping hardware to packages when new stuff is inserted during run time. Looking up kernel modules using modalias values To check which kernel modules provide support for a given modalias, one can use the following shell script:

  for id in $(find /sys -name modalias -print0   xargs -0 cat   sort -u); do \
    echo "$id" ; \
    /sbin/modprobe --show-depends "$id" sed 's/^/  /' ; \
  done

The output can look like this (only the first few entries as the list is very long on my test machine):

  acpi:ACPI0003:
    insmod /lib/modules/2.6.32-5-686/kernel/drivers/acpi/ac.ko 
  acpi:device:
  FATAL: Module acpi:device: not found.
  acpi:IBM0068:
    insmod /lib/modules/2.6.32-5-686/kernel/drivers/char/nvram.ko 
    insmod /lib/modules/2.6.32-5-686/kernel/drivers/leds/led-class.ko 
    insmod /lib/modules/2.6.32-5-686/kernel/net/rfkill/rfkill.ko 
    insmod /lib/modules/2.6.32-5-686/kernel/drivers/platform/x86/thinkpad_acpi.ko 
  acpi:IBM0071:PNP0511:
    insmod /lib/modules/2.6.32-5-686/kernel/lib/crc-ccitt.ko 
    insmod /lib/modules/2.6.32-5-686/kernel/net/irda/irda.ko 
    insmod /lib/modules/2.6.32-5-686/kernel/drivers/net/irda/nsc-ircc.ko 
  [...]

If you want to help implementing a system to let us propose what packages to install when new hardware is plugged into a Debian machine, please send me an email or talk to me on #debian-devel. Update 2013-01-15: Rewrite "cat $(find ...)" to "find ... -print0 xargs -0 cat" to make sure it handle directories in /sys/ with space in them.

I recently added a formal interface to testrepository to enable cross-machine scaling of test runs. As testrepository is still a static scheduler, this isn t perfect, but its quite a minimal interface, which makes it easy to implement. I will likely evolve it in reaction to feedback and experience. In the long term I d love to have a super generic tool that matches that interface, so the project VCS copy of .testr.conf can just call out to it. However I don t yet have that, but I do have a simple by-hand implementation that I use to run nova s tests across my personal laptop, desktop and work laptop. Testr models this by assuming each test running process can be mapped to a single instance id (which could be a chroot, vm, cloud instances, ) and then running one or more commands in the instance, before disposing of it. This by hand implementation consists of 4 things:

1. A tiny script to rsync my source directory to the relevant places before I run tests. (This takes <2seconds on my home wifi).
2. A script to allocate instance ids (I just use ints)
3. A script to discard them
4. And a script to copy tempfiles onto the target machine and run a given command.

I do my testing in lxc containers, because I like my primary environment to be free of project-specific quirks and workarounds. lxc is not needed though, if you don t want it. So, to set this up for yourself:

1. on each host, make an lxc container (e.g. following) http://wiki.openstack.org/DependsOnUbuntu
2. start them all (lxc-start -n nova -d)
3. Make SSH config entries for the lxc containers, so you can get at them remotely. (make sure your host * rules are at the end of the file otherwise the master overrides won t work [and you might not notice for some time...]):

   Host desktop-nova.lxc
   # lxc addresses may be present on localhost too, so namespace the control
   # path to avoid connecting to the wrong container.
     ControlPath ~/.ssh/master-lxc-%r@%h:%p
     hostname 10.0.3.19
     ProxyCommand ssh 192.168.1.106 nc -q0 %h %p
   Host hplaptop-nova.lxc
   # lxc addresses may be present on localhost too, so namespace the control
   # path to avoid connecting to the wrong container.
     ControlPath ~/.ssh/master-lxc-%r@%h:%p
     hostname 10.0.3.244
     ProxyCommand ssh 192.168.1.116 nc -q0 %h %p

4. make a script to copy your nova source tree to each test location. I called mine sync

   #!/bin/bash           
   cd $(dirname $0)
   echo syncing in $(pwd) 
   (rsync -a . desktop-nova.lxc:source/openstack/nova --delete-after && echo dell done) &
   (rsync -a . hplaptop-nova.lxc:source/openstack/nova --delete-after && echo hp done)

5. Make sure you have the base directory on each location

   ssh desktop-nova.lxc mkdir -p source/openstack
   ssh hplaptop-nova.lxc mkdir -p source/openstack

6. Sync your code over.

   ./sync

7. And check tests run by running a few.

   ssh hplaptop-nova.lxc "cd source/openstack/nova && ./run_tests.sh compute"
   ssh hplaptop-nova.lxc "cd source/openstack/nova && ./run_tests.sh compute"

   This will check the test environment: we re not going to be running tests on each node via run-tests or even testr (because it gets immediately meta), but if this fails, later attempts won t work. Your test virtualenv is inside the source tree, so it is copied implicitly by the sync.
8. Decide what concurrency you want. For me, I picked 12: I have a desktop i7 with 4 cores, and two laptops with 2 cores each, and hyperthreads are on on all of them I m going to set a concurrency figure of 12 between the cores (8) and threads (16) counts, and possibly balance it more in future. A higher number assumes less contention between ALU s and other elements of the core pipeline, and I expect quite some contention because most of nova s unittests are CPU bound not I/O. If the test servers are not busy, I can always raise it later.
9. Create scripts to create / dispose / execute logical worker threads.
10. Creation. I call this instance-provision and all it does is find the lowest ints not currently allocated and return them.

    #!/usr/bin/env python
    import os.path
    import sys
    if not os.path.isdir('.instances'):
        os.mkdir('.instances')
    running_ids = os.listdir('.instances')
    count = int(sys.argv[1])
    top = count + len(running_ids)
    ids = [str(i) for i in range(top)]
    new = set(ids) - set(running_ids)
    for id in new:
        file('.instances/%s' % id, 'w').close()
    print(' '.join(new))

11. Disposal is easy: remove the file marking the instance as in-use.

    #!/bin/bash
    echo freeing $@
    cd .instances
    rm $@

12. Execution is a little trickier. We need to run some commands locally, and other ones by copying in temp files that testr has setup to the machine sshing to the remote machine, cd ing to the right directory, sourcing the virtual env, and finally running the command.

    #!/bin/bash
    instance="$(($1 % 4))"
    case $instance in
    [0]) node=
         local="true"
         ;;
    [1]) node=hplaptop-nova.lxc
         local=""
         ;;
    [2-3]) node=desktop-nova.lxc
         local=""
         ;;
    *)   echo "Unknown instance $instance" >&2
         exit 1
         ;;
    esac
    shift
    files=
    # accumulate files to copy
    while [ "--" != "$1" ]; do 
    files="$files $1"
    shift ; done 
    shift   
    if [ -n "$files" -a -z "$local" ]; then
        echo copying $files to node.
        for f in $files; do
            rsync $f $node:$(dirname $f) ;
        done
    fi  
    if [ -n "$local" ]; then
        eval $@
    else
        echo ssh to $node
        ssh $node "cd source/openstack/nova && . .venv/bin/activate && $@"
    fi

13. Finally, tell testr how to use this. (Don t commit this change to nova, as it would break other people). Add this to your .testr.conf.

    test_run_concurrency=echo 12
    instance_provision=./instance-provision $INSTANCE_COUNT
    instance_execute=./instance-execute $INSTANCE_ID $FILES -- $COMMAND
    instance_dispose=./instance-dispose $INSTANCE_IDS

Now, when you run testr run parallel, it will run across your machines. Just do a ./sync before running tests to get the code out there. It is possible to wrap all of this up via automation (or to include just-in-time provisioned cloud instances), but I like the results of still rough scripts here it strikes a good balance between effort, reliability and performance.

Normally, I'm not that much of a fan of the slacktivist trend of signing pre-canned petitions, signing things that political organizations send in the mail, and so forth. It takes very little effort and therefore carries very little weight. A lot of those efforts are more exercises in helping the signers feel better about themselves. But there is the occasional exception. Most of you probably already know about the death of Aaron Swartz. For those who aren't familiar, see Larry Lessig's article. Swartz suffered from depression (and, seriously, fuck depression it's an awful, horrible disease), but there's little doubt that the ongoing federal prosecution using the full weight of the US district attorney's office to hound him for what amounts to political trespassing was part of what led to his suicide. As it happens, I personally believe that Swartz committed a crime, and probably should have paid some consequence for it (on the order of a fine or some community service). Hooking your devices up to someone else's network without their permission and messing around in their wiring closets (locked or not) is, for me, akin to traipsing into someone's barn or backyard shed without their permission and using some of their tools because you want to use them. And whether or not one likes the current copyright regime (and I don't like it at all), MIT is still in the awkward position of having to work with it. Abusing their license for your political goals is effectively recruiting them into your activism without their permission, and as I've mentioned before, I have a mild obsession with consent. It was a crime. However, it was a minor crime, and that's where this whole situation went completely off the rails. One aspect of a justice system is fairness: uniform application of the laws to everyone. However, another aspect of a justice system is proportionality: punishments that fit the crime. Without ever having been convicted, Swartz was already punished completely out of proportion to what he actually did, both financially and emotionally, by prosecution by the US attorney that went far, far beyond zealous into actively abusive. This despite the fact that the owner of the academic papers that he was downloading as an act of open access activism stated they did not want the case to proceed and asked the US government to drop the charges. I don't think what Swartz did was right, or legal. However, the correct reaction was "look, involuntarily recruiting MIT as an accomplice to your act of civil disobedience is not okay don't ever do that again." Not "you are evil and should be locked up in prison for 35 years." And I'm completely fed up with the disproportionality of our justice system and the practice of ridiculous over-charging of crimes in an attempt to terrify people into bad plea bargains. Which brings me to the slacktivism. This is exactly the sort of situation where popular opinion matters. US attorneys who have lost the faith and support of the population they serve won't keep their jobs. And no one in government particularly wants this case to be splashed across the front pages, or to have to answer questions about the appropriateness or proportionality of the prosecution while people are mourning a dead young man. If we make it clear enough to the Obama administration that people are watching, that this matters, and that we're angry about it, not only is it quite likely there will be consequences for this prosecutor, but it may serve as a deterrant for other prosecutors in the future. There is a petition on WhiteHouse.Gov to remove the district attorney for prosecutorial overreach, and for once taking five minutes to create an account and clicking on a petition may be both useful and appropriate. Taking disciplinary action here is, unlike with a lot of petitions, something that the Obama administration can actually do, directly, without involving the rest of the dysfunctional US government, and without making new law. If you are a US citizen, please consider going to this site and signing it to say that this matters to you and you believe this prosecution was excessive and inappropriate. Please note: you do not have to believe that Swartz died solely or even primarily because of this prosecution to do this. We'll never really know the complex factors behind his death. But this was a burden that he shouldn't have had to deal with. Please also note that you do not have to think he was justified in his behavior to sign this petition. This is not a question of whether what he did should be legal or was ethical. Rather, even assuming it was illegal, it's a question of appropriate punishment and proportionality of response. Whether or not you agree with his political cause, there simply was not enough damage done, to anyone, to warrant this kind of aggressive prosecution. And the direct victims agreed, which was the point at which the district attorney should have scaled way back on their actions or dropped the matter entirely. Not doing so was an abuse of office and position, and that should have consequences. ETA: Corrected "taking" to "using" in the analogy about tools. I was actually thinking "taking and then returning" when I wrote that, but only the first word made it into the post, and ended up creating a confusing parallel with theft that wasn't intended.