i have a colleague who is forced by work situations to use Windows. Somehow, I'm the idiot^W^W^W^W^Wfriendly guy who gets tapped to fix it when things break. Well, this time, the power supply broke. As in, dead, no lights, no fan, no nothing. No problem, though, the disk is still good, and i've got a spare machine lying around; and the spare is actually superior hardware to the old machine so it'll be an upgrade in addition to a fix. Nice! So i transplant the disk and fire up the new chassis. But WinXP fails to boot with a lovely "0x0000007b" BSOD. The internet tells me that this might mean it can't find its own disk. OK, pop into the new chassis' BIOS, tell it to run the SATA ports in "legacy IDE" mode, and try again. Now we get a "0x0000007e" BSOD. Some digging on the 'net makes me think it's now complaining now about the graphics driver. Hmm. Well, i figure i can probably work around that by installing new drivers from Safe Mode. So i reboot into Safe Mode. Success! It boots to the login screen in Safe Mode. And, mirabile dictu, i happen to know the Administrator password. I put it in, and get a message that this Windows installation isn't "activated" yet -- presumably because the hardware has changed out from under it. And by the way, i'm not allowed to log in via safe mode until it's activated. So please reboot to "normal" Windows and activate it first. Except, of course, the whole reason i'm booting into safe mode was because normal Windows gives a BSOD. Grrrr. Who thought up this particular lovely catch-22? OK, change tactics. Scavenging the scrap bin turns up a machine with a failed mainboard, but a power supply with all the right leads. It's rated for about 80W less than the old machine's failed supply, but i figure if i rip out the DVD-burner and the floppy drive maybe it will hold. Oh, and the replacement power supply doesn't physically fit the old chassis, but it hangs halfway out the back and sort of rattles around a bit. I sacrifice the rest of the scrap machine, rip out its power supply, stuff the power supply into the old chassis, swap the original disk back in, and ... it boots successfully, finally. That was the shorter version of the story :P So now my colleague has a horrible mess of a frankencomputer which is more likely to fail again in the future, instead of a nice shiny upgrade. Why? Because Microsoft's need to control the flow of software takes priority over the needs of their users. This is what you get when you let Marketing and BizDev drive your technical decisions. Do i still need to explain why i prefer free software? Tags: griping, windows

Thanks to all the people at LCA2013 in Canberra; it was a blast! So good to see old friends and chat freely on what s hot and happening. Radia (known for STP, TRILL), Sir Tim (the web) and old friend Bdale (Debian, SPI, Freedom Box) were inspiring. As was Robert Llewellyn (Kryten, Red Dwarf), who was a complete pleasure he wandered back and talked for a while with the volunteer video crew. Hats off to Pia for organising the TBL tour, to Mary Gardner for being awarded the Rusty Wrench, and to the team from PLUG (Euan, Jason, Leon, Luke) who stepped up to help with the video team and to Paul who graciously accepted the help. Next up LCA2014 Perth! Y all come back now.. it s been a decade.

The Debian PTS now speaks the Turtle representation format for the export of RDF meta-data about Debian source packages. Alongside HTML pages for humans, and the RDF/XML that had already been added to it this means that a new flavour of RDF is now available. The Turtle format offers the benefits of both machine-readable meta-data, and a somehow human readable textual format too. For instance, you may check the apache2 Turtle meta-data from the command-line with :
$ curl -L -s -H "Accept: text/turtle" http://packages.qa.debian.org/apache2 Here s a link to a colorized HTML preview of http://packages.qa.debian.org/a/apache2.ttl. Under the hood, the XSLT stylesheets of the PTS have been reworked to produce the Turtle format by default, and later convert them to RDF/XML. Every Debian source package then has a reference URI in the Linked Data word, in the form http://packages.qa.debian.org/PACKAGE_NAME, that redirects, through proper content-negociation (the HTTP Accept header) to the HTML, RDF/XML or Turtle documents. For apache2, these are, resp. at http://packages.qa.debian.org/a/apache2.html, http://packages.qa.debian.org/a/apache2.rdf and http://packages.qa.debian.org/a/apache2.ttl. The meta-data uses the model of the ADMS.SW ontology (1.0), and the content has also been slightly updated to make it more conformant to the ADMS.SW specifications (checks done with the ADMS.SW validator). Let s hope this makes RDF more familiar to Debian folks, and allows more Linked Data interlinking with other resources about FLOSS packages.

Following my last rant about configuration management, I've had a closer look at Salt, both for my personal use, as well as for a client who would like to deploy something that is not Puppet. Salt has some very good ideas, for instance:

• Configuration is just data in YAML format, which makes it really easy to whack up stuff and also to generate it using other means;
• There's a lot of plugability: runners, renderers, states, modules, returners, outputters, all of which can be replaced with custom implementations;
• It builds on remote execution. In fact, in the docs, they talk first about remote execution and then about configuration management. Not only does this makes a lot of sense, it also gives the admin the additional benefit of having quick, remote access to all hosts in the infrastructure. I like that as it means I don't also have to deploy a parallel-execution solution.
• Salt supports both, a push and a pull model for configuration management. In fact, the push model is part of its remote execution base and it's implemented through persistent connections from the clients to the master. This neatly solves all problems relating to firewalling and NAT, and it also means that the master always has a pretty good understanding of who's around. It feels good to me. And there is no NIH going on with respect to pulling: there is no daemon, you have to use cron. Good.

But there are also a couple of downsides to Salt:

• The default templating engine is Jinja. While that may be an excellent choice for applications in inherently insecure environments, such as the Web, they are quite painful in the context of Salt: no multi-line logic, whitespace control is a nightmare, and a completely unfamiliar set of conditionals and filters, rather than access to e.g. Python. This is only the default, and mako is an alternative, but it seems like a poor choice and that does not instill confidence.
• I don't think that the method of targeting hosts that Salt uses is scalable. I don't want to intersperse my configuration with if-then-else statements referencing hostnames (what would that mean if a hostname changed or a new host was added?). Instead, I want the configuration to be fully parametrised and assemble the parameters from a node database, ideally a hierarchical one. I have a solution for that which works with Salt, called reclass.
• I don't think much of "pillars", which are Salt way to provide random data (parameters) to nodes, again because data needs to be targeted at hosts, rather than defined for a host. This is also solved with reclass.
• Logging and error reporting in Salt are abysmal, inconsistent, and useless. This is offset a little bit by the ability to use pdb, but it's not ideal.
• Exceptions are not properly handled, the file descriptors aren't closed when daemonising, children are not reaped as they should, and the exit code will almost always be zero, even on failure.
• State requirements are not enforced where they are required, but only come into play when a higher order function is called.
• The code reinvents module loading and does that in very complex ways, while putting tight limitations on what's possible. State functions and modules cannot be namespaced, for instance, nor can they easily make use of each other. There is also no way to quickly define macros without resorting to the templating language (and thus a completely different syntax and paradigm), and the concept of having modular, self-contained states (e.g. one for sudo, one for SSH, etc.) with all their dependencies in one hierarchy is not really part of the design. Also, it's not trivial for a state definition to export an interface for other states to use.
• While we're on the topic of code and design, there's a lot of duplicate stuff in need of refactoring (e.g. saltutil._sync, and it seems that this even causes a lot of crypto work to be done over and over again. It also doesn't help that salt-call, which is used to execute commands on the clients (e.g. pull configuration), works completely independently of the daemon process running on each client (although the same code base is being used).
• And why aren't state and execution modules implemented as classes? Instead, the module is itself treated as an entity, and a home-cooked "virtual" system is used to determine which platforms support which modules. Object-oriented programming principles gives you all of that, and more.
• Fortunately, the Salt authors didn't cave in and write the entire communication layer themselves. Instead, they employed ZeroMQ, but it's still quite common for a client to become unreachable (permanently), e.g. due to an IP address change, or networking problems. What's worse is that the master does not keep track of who's listening.
• While we're on the subject of standing on the shoulders of giants, Python's msgpack apparently cannot handle Unicode. Yay.

Those are the big issues. There are many small issues two, but those won't be around for too long as the project is moving along quickly and the community is vibrant. This is surely an important point that speaks for Salt. However, the above issues seem to hint at design choices that might well turn out to stand in the way later. Following a day of frustration, I now feel the overpowering urge to write my own configuration management system, because of course I feel that I could do it better than everyone else. Does this sound familiar to you? Let's just say hypothetically that I would, then I'd want to reuse as much existing functionality as possible. For instance, I'd want the entire remote execution framework to be independent from any configuration management implemented on top. So what does this mean? What would such a remote execution framework need? Here are some thoughts:

• I'd want to keep it the way of Salt, i.e. the clients maintain persistent connections to the master, and the master regularly pings the clients, for housekeeping. And the clients know to expect such pings at regular intervals, so can reset themselves in case they don't hear from the master. Or scream.
• Of course, authentication and encryption need to be part of this. Ideally, key and roster management are already available in the tool that's being reused. I don't want to have to have a new PKI for this
• The protocol would probably be something like XMLRPC, with an extensible list of modules on the client to do the work. Data would be standardised to JSON-format.
• Asynchronous execution would be a plus. I am even tempted to say that it should be all built asynchronously, even though I don't really know a use case for this.
• The clients should be able to feed back information to the master, where data can be accumulated, allowing for cross-node configuration (cf. http://comments.gmane.org/gmane.comp.sysutils.salt.user/2098). This could be implemented using master-side polls to keep the protocol easier.

Doesn't this sound like a Unix botnet to you? ;) I could imagine whacking this up with a bit of Python, some shell glue, socat and SSH: the server would have an authorized_keys file with forced commands connecting the client to the server process via sockets. Or I could imagine using twisted for that. But I would prefer if something like this already existed. Anyone? Comments are broken on my blog, and I cannot be bothered to work on them. If you have any input, please write to me. I will (eventually) condense all feedback into a new article. NP: Mouse on Mars: Parastrophics

Executive summary: Most things work fine.

Things we know are broken:

• Some Samsung laptops. The samsung-laptop driver is a slightly weird thing. By 2010 (when it first appeared) most vendors had moved over to using some level of firmware abstraction, either using ACPI or WMI. Samsung still seemed to be stuck around a decade earlier - they were providing a region of memory at a known address, and you'd read that address to find a bunch of offsets. Then you'd write magic values based on those offsets to magic system IO ports based on those offsets and something would happen. Those writes were triggering System Management Mode, a special x86 CPU mode where the processor executes code from memory that the OS can't see, without telling the OS that it's doing so. There's nothing especially new in this (SMM first appeared in the 386sl back in 1990), but it also means that you depend on the system vendor not changing the interface without telling you. Turns out that Samsung apparently changed their platform interface when they moved to UEFI, but didn't actually do anything to prevent old drivers from breaking things - performing exactly the same series of accesses on some modern Samsung laptops gives an uncorrectable machine check exception (in the best case) or destroys your firmware (in the worst case). Given that the driver was written to Samsung's specifications, this is pretty obviously Samsung's fault, but that's probably little consolation to anyone who ended up with a dead laptop. Although, given Samsung's track record, this may not be surprising.
  
  On the bright side, some of the machines that are affected by this predate Secure Boot, so at least it's not a Secure Boot bug.
  
• Some Toshibas won't boot Linux. This turns out to be some staggering incompetence on the part of Toshiba (or, more likely, their third-party vendor) - they managed to leave the signing key out of the database that's used to validate binaries, and managed to leave the signature database signing key out of the database that's used to provide whitelist or blacklist updates. The good news is that this is a blatant violation of Microsoft's Windows 8 certification guidelines, and that seems to have encouraged Toshiba to actually fix their BIOS. The bad news is that any of the affected machines that are currently available are still broken, and Toshiba don't seem to be willing to actually give you the firmware update yet.
  
• Some Lenovos will only boot Windows or Red Hat Enterprise Linux. I recommend drinking, because as far as I know they haven't actually got around to doing anything useful about this yet.

Not an amazingly positive list, but as far as I know that's about it - other than some Samsungs, one range of Toshibas and one range of Lenovo desktops, Linux should be fine. If you have any other UEFI system that's unable to install Fedora 18, let me know and we'll do our best to work out what's going on.

comments

Brianna Laugher wrote a blog post about the speaking stack used in the free software activism BoF at LCA 2013 [1]. Occupy Wall St uses what they call a progressive speaking stack this means that white men step back in the queue and people from marginalised groups step forward [2]. During the free software activism BoF the speaking queue that was used was that people who hadn t already spoken had priority over those who had spoken before. This was a really good idea and could be used a lot more in LCA and other conferences. It is fairly common that a small number of delegates take up the vast majority of question time. I suggest that all white men watch the questions and observe how many are asked by white men and how many are asked by everyone else. Also note the way that questions are asked, who shouts a question, who wins when two delegates ask at the same time, and who waits until the end of the talk. The Reasons for a Speaking Stack In Occupy Wall St there is a real benefit in giving priority to members of minority groups. The political needs of white men are generally reasonably well publicised due to disparities in media coverage. As the aim of the occupy movement is not to replace one group of white men with another there is an obvious need to get opinions from members of minority groups. Bugs in software generally affect members of all groups equally (with the exception of bugs related to accessibility features). But even so I think it is important to encourage diversity among people who ask questions. When someone is in the audience sees that no-one who is in their minority group is asking questions they will get the impression that they are just watching someone else s conference. We should aim to have a conference for everyone. How to Implement it When taking questions for one of my talks I generally try to give priority to people who find it difficult to be heard. But doing that requires some concentration and I often don t have any to spare when giving a demanding technical talk. I think that this needs to be managed by the moderator/MC/microphone holder. Someone who doesn t need to think much about the content of the talk can concentrate on choosing the best people to ask questions. Also a significant issue is questions that are called out during a talk. Some speakers insist that questions are only asked at the end of their talk. But I prefer some degree of interaction with the audience so my talks often end up being more about having a conversation with the audience than reading from a script. The difficulty with an interactive talk is that it strongly favors those who are prepared to shout a question over those who wait their turn. I think I ll try to make a strict policy of having people raise their hand to ask a question in future to address this issue, but I will need assistance from someone who s not concentrating on the technical issues. For a conference I think it would make sense for the people who hold the microphones to keep a mental list of who s asked questions. If someone asks their share of questions on the first day of the conference then they would deserve a lower priority for questions on later days. This would also encourage delegates to consider whether their question is really worth asking during the lecture or whether they should save their question quota and talk to the speaker afterwards. Also we could ask delegates to exercise restraint. One suggestion I heard was that people should set themselves a quota of 3 questions per conference or 1 per day. In a conference with ~600 delegates and ~33 sessions per day if everyone asked a question each day that would be about 18 questions per session more than is typical. So it seems that anyone who asks a single question per day is still likely to be asking more than 1/600 of all questions. Exceptions There are occasions when multiple questions and comments make sense. One example is where a member of the audience has significant expertise in the topic in question. Another is when a speaker completes significantly before the end of their allotted time and some questions from the MC or an experienced member of the audience can help them spend all their time educating the audience. But I think there needs to be a compelling reason that has a clear benefit for the audience. General Benefits How many of the repeat questions are useful to the audience? It seems to me that there is a correlation between multiple questions and questions that are more about the person asking than about clarifying issues that are likely to matter to the audience. Would such limits improve the quality of the discussion even for people who don t care about diversity? Also I have asked a disproportionate number of questions in the past. I am reducing the number of questions that I ask although I think I asked more than 3 at this conference.

• [1] http://tinyurl.com/aens6vr
• [2] http://tinyurl.com/3s8exu4

WebAuth is the site-wide web authentication software that we use at Stanford. We're about to start another round of intensive development on multifactor authentication support, so I wanted to get a minor release out with the other things that have accumulated in the past month and a half. The main feature change in this release is to add support for another WebLogin configuration callback. This one is run whenever WebLogin is attempting to establish the user's identity and can return any identity it wishes, so it's available as a generic callback to determine identity information from the environment. The most likely use will be to inspect the results of client-side certificate authentication (which Apache puts in various environment variables) and interpret them in a site-specific way. Also in this release I refactored the WebLogin scripts so that they use FastCGI more correctly. The application objects are now instantiated only once and then reused for the lifetime of the FastCGI script, rather than torn down and set up for each new request. This is primarily of interest to those using the new features that require a memcached connection, since that connection will no longer be constantly set up and torn down, but it should help with speed in other situations. This is the riskiest change, since if I didn't get it exactly right it would be possible for information to leak from one request to the next. I've audited it fairly carefully, though, and think I've gotten everything. Finally, besides some documentation updates, I've switched the Perl build to Module::Build from ExtUtils::MakeMaker. This should mostly be transparent, but it means some additional Perl modules from CPAN will be required to build the distribution with --enable-webkdc for versions of Perl older than 5.10. (5.10 is fairly old, so I doubt this will be a serious issue.) You can get the latest release from the official WebAuth distribution site or from my WebAuth distribution pages.

Earlier today a new package BH arrived on CRAN. Over the years, Jay Emerson, Michael Kane and I had numerous discussions about a basic Boost infrastructure package providing Boost headers for other CRAN packages (and yes, we are talking packages using C++ here). JJ and Romain chipped in as well, and Jay finally took the lead by first creating a repo on R-Forge. And now the package is out, so I just put together a quick demo post over at the Rcpp Gallery. As that post notes, BH is still pretty new and rough, and we probably missed some other useful Boost packages. If so, let one of us know.

After recovering from the shock of seeing consensus spelled correctly for a change, I thought I should respond to some claims of healthiness.

1. Package fiefdoms waste people's time quite frequently: MIA processing, duplicated effort in DELAYED NMUs, conspiracy and infighting, trying to get an absentee maintainer to respond, the ridiculous things that happen when a maintainer is still around but not doing his job for 3 consecutive years, and so on. Having all packages maintained by "Debian Developers" is a great idea.
2. The current hierarchy might work better than some lesser evils, but that doesn't mean it's good at all. We should strive toward ZERO hierarchy, not some magical fiction where people imagine themselves wiser than everyone else because their drinking buddies said so.

The anti-consensus features in Debian would better be replaced by a real solution to the problem pervading nearly all free software projects: ego.

Earlier this week, things were looking different, and I did have to cancel a Haskell talk in Freiburg on Monday due to illness. But I m back on track and will be travelling to Brussels tomorrow. I ll be holding a talk on how we package Haskell in Debian, on Suday at 15:30. I hope it will be useful to Debian users (who will better understand the packaging), other Debian Developers (who ll learn about the peculiarities of Haskell and the implications for the Debian infrastructure), other distro s maintainers (to compare best practices) and Haskell developers (to learn about the needs and worries of downstream packages). The talk will be based on my DebConf 11 talk on the same topic. I m also happy to answer questions about Haskell, Haskell in Debian or any other topic that you want to hear my opinion about, so just talk to me during FOSDEM.
In related news: GHC 7.6.2 was uploaded to Debian experimental the day it was released; the rebuilding of all libraries is still in progress (~370 of ~570 done).

* Somehow, I felt that there should be post for atleast to abuse Planet Debian in the new year of 2013. So, what s going on? 1. Finished Full Marathon aka Standard Charted Mumbai Marathon with pathetic timing of 5h55m35s (and felt good at the end). 2. Good thing: Visited family twice in last 2 months. 3. N number of Idlis were consumed. Where, N >= 300. Overall, I m big fan of South Indian food which is more suitable for my stomach. Thanks to running, my junkfood consumption is lesser than I can imagine but you can find me in those places during some weekends to fullfill calories lost in LSD run! 4. Last Sunday, I delivered lecture on Debian Ecosystem , mostly same presentation with additional updates and details, at SVIT, Doddaballapur (technically, some outskirt of BLR) as part of FSMK s 5 days event. It was well received from the applause I got. Photos are still lying in memory card and probably will not come out until next usage of camera. 5. I m very silent on Debian packaging and someone will hit me for pending tasks, but enjoying packaging and fixing tlsdate/torbirdy for Debian lately to get motivation. Other than that, few uploads were done.

As most people reading this have probably seen, xkcd found a particularly entertaining Wikipedia argument. (It is all true. There are at least 40,000 words arguing over whether to capitalize an "i" in a page title. I went and looked last night and spent several hours following the conversation.) This is, of course, funny, and I'm sure lots of people went away smiling at an excellent example of bikeshed painting on the Internet. But I think it's rather more serious than it appears to be. In fact, I would go so far as to say that it is a visible manifestation of a deeply-rooted problem that is the core of why I personally refuse to contribute to Wikipedia in any way that requires me to invest non-trivial time or become a member of that community. To explain that statement requires a bit of background. One of the very first services I found when I joined the Internet in 1993 was Usenet. Originally, I talked about comics collecting and then Magic: The Gathering, but I eventually ended up involved in Usenet governance. I spent more than ten years deeply invested in Usenet hierarchy administration; roughly, the process of agreeing on names for and sometimes moderators for global Usenet discussion groups, as well as all the supporting infrastructure for Usenet news server administrators to agree upon and share articles in those groups. Usenet newsgroup creation was, throughout that period, a consensus-driven process. There was a voting system that was used to measure consensus, but the voting system was largely designed to ensure that rough consensus existed (it was heavily skewed towards not changing anything). We took great pride in that fact. I'm sure that one could find many articles from me proudly championing the concept of a consensus-based decision-making process against many alternatives. That means I was also there when the toxic flaws at the heart of a consensus-based process completely destroyed it and, as a side effect, destroyed the community of people who had formed around it. I will never again invest my time and energy in a community that operates solely via a consensus process unless that community is small enough that consensus can actually work (which means not much more than twenty people). Consensus decision-making in large groups destroys communities and hurts people. These sorts of extended arguments on Wikipedia are not rare anomolies. They happen with some frequency; just by going through the debate xkcd references and looking at pointers, you can easily find others (Sega Genesis, cat ownership vs. companionship, endless notability debates... there are many). They look funny from outside, but from inside they're hugely demoralizing. Even when they don't get heated and spill over into personal animosity (and if they don't do that, you're lucky), they consume astonishingly vast quantities of time exactly not the resource that you want to be squandering in volunteer projects. And they do so without giving back anything emotionally. I know these arguments very well. One of the most contentious and angry arguments in the history of Usenet newsgroup creation was the argument over whether, when breaking up a group into multiple more focused groups, to rename the original group to end in *.misc or leave its existing name. This argument went on for years and led to deep, lingering hatred between the participants. It's the same sort of thing. These arguments happen in the way that they happen because there is toxic waste at the center of Wikipedia's governance process. And that toxic waste hurts people, wastes their time, and destroys their willingness to volunteer. It undermines community, which is sad and ironic because the whole point of consensus decision-making is to create, support, and empower community. But it doesn't work. It doesn't work for two very simple but very closely-related reasons: it's impossible to ever end an argument, and there is no incentive for anyone to shut up. As a result, it is nearly impossible to get the decision-making to cohere into something that is timely, consistent, unambiguous, and final. These are not optional requirements of a decision-making process. Without them, people will lose faith in the decision-making process itself, and some people will learn that, as long as they never stop talking about something, they can prolong the debate indefinitely and possibly eventually get their way. Now, of course, I'm deeply involved and invested in Debian. And on the surface, it may look like Debian also uses a consensus-based decision-making process. But while Debian decision-making has some problems, it's much healthier for a few reasons that are useful to examine:

1. Individual packages in Debian are not managed by consensus, but rather fall under the authority of the package maintainer, who is either a single person or a small team. (Consensus works in small teams; it's great if the number of people involved is five or less.) This drastically limits the number of decisions that are subject to a large-scale consensus process. Nearly all decisions in Debian are made directly by the person doing the work, and only in exceptional circumstances can some larger process be invoked. This, on the surface, sounds like it could be similar to Wikipedia, but note that appeals to a larger process are much rarer in Debian than they are in Wikipedia. There is a strong bias in favor of letting the person working on the package do their job however they want to do it. This is more social than structural, but it's a core value (and we should be very careful about changing it). As a result, Debian operates less by mass consensus and more as a federation of semi-autonomous fiefdoms. This sounds worse: it sounds authoritarian and non-democratic. But it's significantly more functional, and it doesn't waste people's time.
2. On appeal, and for broader questions that necessarily cut across multiple packages, there are several people in Debian who are perceived as having clear authority to make timely and relatively final decisions, who are almost never overruled, and who can therefore effectively end arguments. These include core teams like ftp-master or the release team, formal appeal bodies like the Technical Committee, and the Debian Project Leader. Not all of this ability is written in black and white in the governance process, but that's not the important part; the important part is that it works in practice to provide timely, consistent, unambiguous, and final decisions most of the time. (Some parts work better than others, and we do struggle with timely, but I've seen much, much worse.)
3. There is a well-established ultimate appeal (a General Resolution), the ultimate appeal process is very specific and concrete, there is little or no ambiguity or human interpretation involved in analyzing the results, it is time-limited, and it's almost universally respected. Everyone abides by the result of a GR if matters get to that point, and while it's theoretically possible to propose another GR immediately to overturn the first one, this doesn't happen in practice. There's also substantial pushback against invoking the GR process for anything that isn't really important.

We still occasionally devolve into endless discussion, usually on some technical topic that cuts across the distribution and usually when the options haven't cohered enough for any authority to make a useful decision (often because it's too early for it to be possible to make a final decision). Those cases, such as the recurring systemd vs. upstart discussion, are still quite frustrating. But they're also relatively rare and largely ignorable because they have to be about just the right sort of issue. Wikipedia has these sorts of arguments all the time, and they're much less ignorable, because consensus is invoked for every minor dispute on any page in the entire project. You have to participate in endless consensus process or risk having your work undone, reverted, or removed. We would all like all decisions to be made by consensus, since we don't want anyone in a volunteer project to lose (and because we all love to convince other people we're right). And for small teams, consensus is wonderful. But that's precisely because governance in small teams is very easy; humans are naturally adapted for decision-making in small teams and have a huge variety of natural tools, tendencies, and abilities devoted to making those sorts of decisions. You rarely need any formal decision-making process; most decisions will just happen. You need a decision-making process when enough people are involved that mutual social pressure can't bring a dispute to resolution, and when you're in that situation, consensus is a horrible process. Due to my past experiences, I'm almost certainly at the extreme in dislike of consensus processes. Most people are not going to refuse to participate in communities solely on that basis. But be aware that 40,000 words of discussion about something that an outsider will not consider particularly important is not an accidental phenomenon. It's not something that just happens randomly on the Internet. It's an effect, and symptom, of a governance process, or lack thereof, and it's something that you can do something about if you want to.

A new Armadillo version 3.6.2 came out yesterday, and the corresponding RcppArmadillo version is now on CRAN. Changes are mostky incremental:

Changes in RcppArmadillo version 0.3.6.2 (2013-01-29)

• Upgraded to Armadillo release Version 3.6.2
  • faster determinant for matrices marked as diagonal or triangular
  • more fine-grained handling of 64 bit integers
• Added a new example of a Kalman filter implementation in R, and C++ using Armadillo via RcppArmadillo, complete with timing comparison

Courtesy of CRANberries, there is also a diffstat report for the most recent release As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

At the end of my talk on Monday about the status of SE Linux [1] I described some of the things that I want to do with SE Linux in Debian (and general SE Linux stuff). Here is a brief summary of some of them: One thing I ve wanted to do for years is to get X Access Controls working in Debian. This means that two X applications could have windows on the same desktop but be unable to communicate with each other by any of the X methods (this includes screen capture and clipboard). It seems that the Fedora people are moving to sandbox processes with Xephyr for X access (see Dan Walsh s blog post about sandbox -X [2]). But XAce will take a lot of work and time is always an issue. An ongoing problem with SE Linux (and most security systems) is the difficulty in running applications with minimum privilege. One example of this is utility programs which can be run by multiple programs, if a utility is usually run by a process that is privileged then we probably won t notice that it requires excess privileges until it s run in a different context. This is a particular problem when trying to restrict programs that may be run as part of a user session. A common example is programs that open files read-write when they only need to read them, if the program then aborts when it can t open the file in question then we will have a problem when it s run from a context that doesn t grant it write access. To deal with such latent problems I am considering ways of analysing the operation of systems to try and determine which programs request more access than they really need. During my talk I discussed the possibility of using a shared object to log file open/read/write to find such latent problems. A member of the audience suggested static code analysis which seems useful for some languages but doesn t seem likely to cover all necessary languages. Of course the benefit of static code analysis is that it will catch operations that the program doesn t perform in a test environment error handling is one particularly important corner case in this regard.

• [1] http://etbe.coker.com.au/2013/01/28/selinux-status-lca2013/
• [2] http://danwalsh.livejournal.com/31146.html

Like many people I'll be going to FOSDEM again this year. I'll be giving a talk on Saturday at 1800 in the Virtualisation devroom on Evolving Xen Paravirtualization. I'll also be manning the Xen.org booth at least some of the time but hopefully I'll get to see some of the other talks too. Hope to see everyone there!

Dear reader, I know you're wondering what I'm getting up to these days. Or at least, I guess there's a possibility that you're wondering. So since there's no single bit of news that would be worthy of a post here by itself, here's one summarizing things, in the hope that the accumulation makes a tiny bit of a difference. On the rock'n'roll front: Eleven did its first true gig on our own in a pub last Saturday. And when I say on our own , it could almost be taken literally, for we must have had a grand total of about 10 people. A bit of a disappointment, to say the least, especially since the pub was about 120 kilometers from home, 60 of which I rode on my motorbike at 2 AM (and close to 0 C). However, the landlord seemed to like us and hinted at further gigs during seasons where people are more likely to go out for drinks and music than to stay warm at home. In a related note: we got ourselves a small set of stage lights (LED-powered, of course), and they have, in addition to the power cord and various switches, two sockets at the back for plugging XLR-3 cables. On investigation, it seems that this means they can be controlled by a protocol known as DMX 512, which opens up a lot of possibilities for someone who likes to control various things from a computer. I read a few web pages to get an idea of how this is supposed to work, it seems rather simple and straightforward, but the required software isn't in Debian yet. So I guess that if/when I get the necessary hardware, I'll have a new hobby and new toys to play with. Maybe our next gig will have bursts of lights on the big accented beats, triggered by strong enough hits on my drum cymbals. This allows me to link to a new minor release of Wiimidi. The only addition is a new configuration section mapping to the default drumkit provided by Hydrogen. And finally, to stay in the small bits of software , I was prompted to give my simple GPG-encrypted password store, a.k.a. SGEPS, its own page, with a proper release and so on. So there, 0.1 is out, with minimal Debian packaging included.

In response to my own post from 7 years ago Bachelorhood, I would like to announce that I am a much better married man than Simon. My bachelorhood days are truly over, my laundry goes in a laundry basket, my clothes are always put away (whether I like it or not). My fridge is plugged in, has real food in it; I have a cell phone (actually numerous cell phones); I have an awesome custom made table; lastly I have a TV in my house. Bachelorhood officially ended last year, 30 years was a good run :)

I like App Engine, my wedding website runs on App Engine as well as a couple recent commercial projects. However, there are some annoyances that make a lot of the useful things I learned at Google not applicable.

1. Chunked responses
   App Engine doesn't have a way of doing chunked responses.
   1.5. Processing after sending the response My annoyance here is that I can't send an early response and wait for some resource. Say I want to do 5 url fetches that I cache for a short period of time. If I want to reply in 250ms, I want to use any of the results I've gotten and reply to the user. Then once the rest of the fetches come in, I want to put them in the cache, so they will come back quickly the next time.
2. Counters
   App Engine has crappy support for counters. You can shard your counters into the datastore: https://developers.google.com/appengine/articles/sharding_counters#counter_python or you can use taskqueues: http://blog.appenginefan.com/2009/10/non-sharded-counters-part-2-using-task.html . Now, how do you graph these counters over time, well, that is left as an exercise for the reader. Google has special datastores for stores and retrieving counters.
3. Backends
   Backends in appengine are versioned separately from frontends. This sounds like a good idea, but to run the devserver with backends, the backends need to be in the same source tree. Thus backends share the same url space. Backends also share the same app.yaml, so again the url space is shared. Thus, if you make a change that you want to coordinate between the frontend and the backend, you need to change the application version in app.yaml, then rename all the backends in backends.yaml. If backends are supposed to be in the same source tree as frontends, then they should be prefixed with the application version. If backends should be versioned separately from frontends, then they shouldn't share url handlers from app.yaml.
   3.5 Uploading backends is slow
   Each backend is uploaded through appcfg.py separately. So if you are crazy and have one backend per backend level, then you need to upload 4 copies of the source code each time the backends get updated.
4. Logging
   I want structured logs. Instead what you have to do is use the python logging module to write out human readable logs, then parse them out into the structure you want.
   http://code.google.com/p/google-app-engine-samples/source/browse/trunk/logparser/logparser.py
   

As a preemptive counter, 1.5 could be dealt with using backends and a background thread, except backends don't scale to the number of queries, so my use case above isn't fixed.

It's not too obvious how to test email receive handlers in Appengine. The important observation is that the handlers take HTTP POSTs with multipart/form-data encoded data. In python you can build an email to be handled with the following code:

from email.message import Message 

def test_email(self):
  body = Message()
  body.add_header('to', 'test-unknown@other-app.com')
  body.add_header('from', 'test@app.com')
  body.add_header('Content-Type', 'multipart/alternative', boundary=self.boundary)
  text = Message()
  text['content-type'] = 'text/plain'
  text.set_payload('I am I! Don Quixote!  The man of La Mancha!')
  body.attach(text)

  post(payload=body.as_string())

I have two annoyances with Appengine's NDB OR queries. One is that the order must be by key, the other is that OR queries are done in series, so ndb.OR(a == 1, a == 2) are done as query(a == 1) then query(a == 2). To work around this, I created a new cursor class that allows sorting by any attribute.

Using this cursor does come with it's own caveats, one is that the order between the queries is not maintained (fixable) and two the limit attribute is per query not for the results of all queries (fixable).

import base64
import json

from google.appengine.ext import ndb

class MultiCursor(object):
@staticmethod
def from_websafe_string(value):
values = None
try:
value = base64.urlsafe_b64decode(str(value))
values = json.loads(value)
except Exception as e:
logging.error('Invalid Cursor %s - %s', e, value)
if not values:
return MultiCursor()

cursors = MultiCursor()
for key, cursor in values.iteritems():
try:
if isinstance(cursor, basestring):
cursors.set(key, ndb.Cursor.from_websafe_string(cursor))
else:
cursors.set(key, cursor)
except:
logging.error('MultiCursor.from_string - bad cursor (%s) for %s in %s',
cursor, key, values)
return cursors

def to_websafe_string(self):
values =
for key, value in self.values:
if isinstance(value, ndb.Cursor):
values[key] = value.to_websafe_string()
else:
values[key] = value
return base64.urlsafe_b64encode(json.dumps(values))

def __init__(self, values=None):
self.values = values or

def get(self, key):
return self.values.get(key), self.values.get(key + _'done')

def set(self, key, cursor, more):
self.values[key] = cursor
self.values[key + _'done'] = not more

def __len__(self):
return len(self.values)

To use the cursor the code should look like:

@ndb.tasklet
def _query_A_by_value_async(value, cursor, limit=100):
start_cursor, done = cursor.get(value)
if not done:
query = Foo.query(Foo.A == value)
query = Foo.order(-Foo.last_access_time)
foos, end_cursor, more = yield query.fetch_page_async(
limit, start_cursor=start_cursor)
cursor.set(value, end_cursor, more)
raise ndb.Return(foos, end_cursor, more)
raise ndb.Return([], None, False)