Last week, I visited London for the January Docker meetup, which was the first time I'd attended this group. It was a talk-oriented format, with around 200 attendees packed into Shoreditch Village Hall; free pizza and beer was provided thanks to the sponsors, which was awesome (and makes logistics easier when you're travelling there from work). There were three talks. First, Andrew Martin from British Gas spoke about how they use Docker for testing and continuous deployment of their Node.js microservices - buzzword bingo! But it's helpful to see how companies approach these things. Second, Johan Euphrosine from Google gave a short demo of Google Cloud Platform for running Docker containers (mostly around Container Engine, but also briefly App Engine). This was relevant to my interests, but I'd already seen this sort of talk online. Third, Dan Williams presented his holiday photos featuring a journey on a container ship, which wins points from me for liberal interpretation of the meetup topic, and was genuinely very entertaining/interesting - I just regret having to leave to catch a train halfway through. In summary, this was worth attending, but as someone just getting started with containers I'd love some sort of smaller meetings with opportunities for interaction/activity. There's such a variety of people/use cases for Docker that I'm not sure how much everyone had in common with each other; it would be interesting to find out.

I moved back from the California Bay Area to Belfast a while back and for various reasons it looks like I m going to be here a while, so it made sense to have my belongings shipped over here. They haven t quite arrived yet, and I ll do another post about that process once they have, but I ve been doing various tweets prefixed with [shipping] during the process. Various people I ve spoken to (some who should know me better) thought this was happening manually. It wasn t. If you care about how it was done, read on. I d been given details of the ship carrying my container, and searching for that turned up the excellent MarineTraffic which let me see the current location of the ship. Turns out ships broadcast their location using AIS and anyone with a receiver can see the info. Very cool, and I spent some time having a look at various bits of shipping around the UK out of interest. I also found the ship s itinerary which give me some idea of where it would be calling and when. Step one was to start recording this data; it was time sensitive and I wanted to be able to see historical data. I took the easy route and set up a cron job to poll the location and itinerary on an hourly basis, and store the results. That meant I had the data over time, if my parsing turned out to miss something I could easily correct it, and that I wasn t hammering Marine Traffic while writing the parsing code. Next I wanted to parse the results, store them in a more manageable format than the HTML, and alert me when the ship docked somewhere or set off again. I ve been trying to learn more Python rather than doing my default action of turning to Perl for these things, and this seemed like a simple enough project to try out. Beautiful Soup seemed to turn up top for HTML parsing in Python, so that formed the basis. Throwing the info into a database so I could do queries felt like the right move so I used SQLite - if this had been more than a one off I d have considered looking at PostgreSQL and its GIS support. Finally Tweepy made it very easy to tweet from Python in about 4 lines of code. The whole thing weighed in at only 175 lines of code, mostly around pulling the info out of the HTML and then a little to deal with checking for state changes against the current status and the last piece of info in the database. The pieces of information I chose to store were the time of the update (i.e. when the ship sent it, not when my script ran), reported area, reported state, the position + course, reported origin, reported destination and eta. The fact this is all in a database makes it very easy to do a few queries on the data. How fast did the ship go?

sqlite> SELECT MAX(speed) FROM status;
MAX(speed)
21.9

What areas did it report?

sqlite> SELECT area FROM status GROUP BY area;
area
-
Atlantic North
California
Caribbean Sea
Celtic Sea
English Channel
Hudson River
Pacific North
Panama Canal

What statuses did we see?

sqlite> SELECT status FROM status GROUP BY status;
status
At Anchor
Moored
Stopped
Underway
Underway using Engine

Finally having hourly data lets me draw a map of where the ship went. The data isn t complete, because the free AIS info depends on the ship being close enough to a receiving station. That means there were a few days in the North Atlantic without updates, for example. However there s enough to give a good idea of just how well traveled my belongings are, and it gave me an excuse to play with OpenLayers. (Apologies if the zoom buttons aren t working for you here; I think I need to kick the CSS in some manner I haven t quite figured out yet.)

Normally, this doesn t work as one might naively expect:

program > firstfile > secondfile

The second redirection will override the first one. You d have to use an external tool to make this work, maybe something like:

program   tee firstfile secondfile

But with zsh, this type of thing actually works. It will duplicate the output and write it to multiple files. This feature also works with a combination of redirections and pipes. For example

ls > foo   grep bar

will write the complete directory listing into file foo and print out files matching bar to the terminal. That s great, but this feature pops up in unexpected places. I have a shell function that checks whether a given command produces any output on stderr:

! myprog "$arg" 2>&1 >/dev/null   grep .

The effect of this is:

• If no stderr is produced, the exit code is 0.
• If stderr is produced, the exit code is 1 and the stderr is shown.

(Note the ordering of 2>&1 >/dev/null to redirect stderr to stdout and silence the original stdout, as opposed to the more common incantation of >/dev/null 2>&1, which silences both stderr and stdout.) The reason for this is that myprog has a bug that causes it to print errors but not produce a proper exit status in some cases. Now how will my little shell function snippet behave under zsh? Well, it s quite confusing at first, but the following happens. If there is stderr output, then only stderr is printed. If there is no stderr output, then stdout is passed through instead. But that s not what I wanted. This can be reproduced simply:

ls --bogus 2>&1 >/dev/null   grep .

prints an error message, as expected, but

ls 2>&1 >/dev/null   grep .

prints a directory listing. That s because zsh redirects stdout to both /dev/null and the pipe, which makes the redirection to /dev/null pointless. Note that in bash, the second command prints nothing. This behavior can be changed by turning off the MULTIOS option (see zshmisc man page), and my first instinct was to do that, but options are not lexically scoped (I think), so this would break again if the option was somehow changed somewhere else. Also, I think I kind of like that option for interactive use. My workaround is to use a subshell:

! ( myprog "$arg" 2>&1 >/dev/null )   grep .

The long-term fix will probably be to write an external shell script in bash or plain POSIX shell.

About a month ago, I blogged about extremon. As a reminder, ExtreMon is a monitoring tool that allows you to view things as they are happening, rather than with the ~5 minute delay that munin gives you, and also avoiding the quad-state limitation of Nagios' "good", "bad", "ugly", and "unknown" states. No, they're not really called that. Yes, I know you knew that. Anyway. In my blog post, I explained how you can set up ExtreMon, and I also set up a fairly limited demo version on my own server. But I have since realized that while it is functional, it doesn't actually show why ExtreMon is so great. In an effort to remedy that, I present you an example of what ExtreMon can do. Let's start with a screenshot of the ExtreMon console at the customer for which I spent time trying to figure out how to get it up and running: Click for full sized version. You'll note that even in that full-sized version, many things are unreadable. This is because the ExtreMon console allows one to move around (right mouse button drag for zoom; left mouse button drag for moving around; control+RMB for rotate; center mouse button to reset to default); so what matters is that everything fits on the screen, not whether it is all readable (if you need to read, you zoom). The image shows 18 rectangles. Each rectangle represents a single machine in this particular customer's HPC cluster. The top three rectangles are the cluster's file servers; the rest are its high performance nodes. You'll note that the left fileserver has 8 processor cores (top row), 8 network cards (bottom row, left part), and it also shows information on its memory usage (bottom row, small rectangle in the middle) as well as its NFS client and server procedure calls (bottom row, slightly larger rectangles to the right). This file server is the one on which I installed ZFS a while back; hence the large amount of disks visible in the middle row. The leftmost disk is the root filesystem (which is an ext4 off a hardware RAID1); the two rightmost "disks" are the PCIe-attached SSDs which are used for the ZFS L2ARC and write log. The other disks in this file server nicely show how ZFS does write load balancing over all its disks. The second file server has a hardware RAID1 on which it stores all its data; as such, there is only one disk graph there. It is also somewhat more limited in network, as it has only two NICs. It does, however, also have 8 cores. The last file server has no more than four processor cores; in addition, it also does not have a hardware RAID controller, so it must use software RAID over its four hard disks. This server is used for archival purposes, mostly, since it is insufficient for most anything else. As said, the other nodes are the "compute nodes", where the hard work is done. Most of these compute nodes have 16 cores each; two have 12 instead. When this particular screenshot was taken, four of the nodes (the ones showing red in their processor graphs) were hard at work; the others seem to have been mostly idling. In addition to the familiar memory, NFS (client only), network, and processor graphs, these nodes also show a "swap space" graph (just below the memory one), which seems fine for most nodes, except for the bottom left one (which shows a few bars that are coloured yellow rather than green). The green/yellow/red stuff is supposed to represent the "ok", "warning", "bad" states that would be familiar from Nagios. In this particular case, however, where "processor is busy all the time" is actually a wanted state, a low amount of idleness on the part of the processor isn't actually a problem, on the contrary. I did consider, therefore, to modify the ExtreMon configuration so that the processor graphs would not show red when the system was under high load; however, I found that differences in colour like this actually makes it more easy to see, at a glance, which machines are busy -- and that's one of the main reasons why we wanted to set this up. If you look carefully, you can find a particular processor core in the graph which shows 100% usage for "idle", "system", and "softirq", at the same time. Obviously that can't be the case, so there's a bug somewhere. Frank seems to believe it is a bug in CollectD; I haven't verified that. At any rate, though, this isn't usually a problem, due to the high update frequency of ExtreMon. The amount of data that's flowing through ExtreMon is amazing:

• 22 values for NFS (times two for the file servers) per server: 22x2x3+22x15
• 4 values for memory: 4x18
• 3 values for swap: 3x15
• 8 values per CPU core: 8x8x2+8x4+8x12x2+8x16x13
• 2 values per disk: 2x25+2+2x4
• 2 values per NIC: 2x8x12+2x2x2+2x4x4

Which renders a grand total of 2887 data points that are shown in this particular screenshot; and then I'm not even counting all the intermediate values, some of which also pass through ExtreMon. Nor am I counting the extra bits which have since been added (this screenshot is a few days old, now, and I'm still finetuning things). Yet even so, ExtreMon manages to update those values once every few seconds, in the worst case. As a result, the display isn't static for a moment, constantly moving and updating data so that what you see is never out of date for more than a second or two. Awesome.

LCA 2015 is next week so it seems like a good time to offer some suggestions for other delegates based on observations of past LCAs. There s nothing LCA specific about the advice, but everything is based on events that happened at past LCAs. Don t Oppose a Lecture Question time at the end of a lecture isn t the time to demonstrate that you oppose everything about the lecture. Discussion time between talks at a mini-conf isn t a time to demonstrate that you oppose the entire mini-conf. If you think a lecture or mini-conf is entirely wrong then you shouldn t attend. The conference organisers decide which lectures and mini-confs are worthy of inclusion and the large number of people who attend the conference are signalling their support for the judgement of the conference organisers. The people who attend the lectures and mini-confs in question want to learn about the topics in question and people who object should be silent. If someone gives a lecture about technology which appears to have a flaw then it might be OK to ask one single question about how that issue is resolved, apart from that the lecture hall is for the lecturer to describe their vision. The worst example of this was between talks at the Haecksen mini-conf last year when an elderly man tried at great length to convince me that everything about feminism is wrong. I m not sure to what degree the Haecksen mini-conf is supposed to be a feminist event, but I think it s quite obviously connected to feminism which is of course was why he wanted to pull that stunt. After he discovered that I was not going to be convinced and that I wasn t at all interested in the discussion he went to the front of the room to make a sexist joke and left. Consider Your Share of Conference Resources I ve previously written about the length of conference questions [1]. Question time after a lecture is a resource that is shared among all delegates. Consider whether you are asking more questions than the other delegates and whether the questions are adding benefit to other people. If not then send email to the speaker or talk to them after their lecture. Note that good questions can add significant value to the experience of most delegates. For example when a lecturer appears to be having difficulty in describing their ideas to the audience then good questions can make a real difference, but it takes significant skill to ask such questions. Dorm Walls Are Thin LCA is one of many conferences that is typically held at a university with dorm rooms offered for delegates. Dorm rooms tend to have thinner walls than hotel rooms so it s good to avoid needless noise at night. If one of your devices is going to make sounds at night please check the volume settings before you start it. At one LCA I was startled at about 2AM but the sound of a very loud porn video from a nearby dorm room, the volume was reduced within a few seconds, but it s difficult to get to sleep quickly after that sort of surprise. If you set an alarm then try to avoid waking other people. If you set an early alarm and then just get up then other people will get back to sleep, but pressing snooze repeatedly for several hours (as has been done in the past) is anti-social. Generally I think that an alarm should be at a low volume unless it is set for less than an hour before the first lecture in which case waking people in other dorm rooms might be doing them a favor. Phones in Lectures Do I need to write about this? Apparently I do because people keep doing it! Phones can be easily turned to vibrate mode, most people who I ve observed taking calls in LCA lectures have managed this but it s worth noting for those who don t. There are very few good reasons for actually taking a call when in a lecture. If the hospital calls to tell you that they have found a matching organ donor then it s a good reason to take the call, but I can t think of any other good example. Many LCA delegates do system administration work and get calls at all times of the day and night when servers have problems. But that isn t an excuse for having a conversation in the middle of the lecture hall while the lecture is in progress (as has been done). If you press the green button on a phone you can then walk out of the lecture hall before talking, it s expected that mobile phone calls sometimes have signal problems at the start of the call so no-one is going to be particularly surprised if it takes 10 seconds before you say hello. As an aside, I think that the requirement for not disturbing other people depends on the number of people who are there to be disturbed. In tutorials there are fewer people and the requirements for avoiding phone calls are less strict. In BoFs the requirements are less strict again. But the above is based on behaviour I ve witnessed in mini-confs and main lectures. Smoking It is the responsibility of people who consume substances to ensure that their actions don t affect others. For smokers that means smoking far enough away from lecture halls that it s possible for other delegates to attend the lecture without breathing in smoke. Don t smoke in the lecture halls or near the doorways. Also using an e-cigarette is still smoking, don t do it in a lecture hall. Photography Unwanted photography can be harassment. I don t think there s a need to ask for permission to photograp people who harass others or break the law. But photographing people who break the social agreement as to what should be done in a lecture probably isn t. At a previous LCA a man wanted to ask so many questions at a keynote lecture that he had a page of written notes (seriously), that was obviously outside the expected range of behaviour but probably didn t justify the many people who photographed him. A Final Note I don t think that LCA is in any way different from other conferences in this regard. Also I don t think that there s much that conference organisers can or should do about such things.

• [1] http://etbe.coker.com.au/2014/01/06/length-conference-questions/

All I want for 2015 is a Free/Open Source Software social network which is:

• easy to register on (no reCaptcha disability-discriminator or similar, a simple openID, activation emails that actually arrive);
• has an email help address or online support or phone number or something other than the website which can be used if the registration system causes a problem;
• can email when things happen that I might be interested in;
• can email me summaries of what s happened last week/month in case they don t know what they re interested in;
• doesn t email me too much (but this is rare);
• interacts well with other websites (allows long-term members to post links, sends trackbacks or pingbacks to let the remote site know we re talking about them, makes it easy for us to dent/tweet/link to the forum nicely, and so on);
• isn t full of spam (has limits on link-posting, moderators are contactable/accountable and so on, and the software gives them decent anti-spam tools);
• lets me back up my data;
• is friendly and welcoming and trolls are kept in check.

Is this too much to ask for? Does it exist already?

I promised to write about this a long time, ooops... :-) Another ARM port in Debian - yay! arm64 is officially a release architecture for Jessie, aka Debian version 8. That's taken a lot of manual porting and development effort over the last couple of years, and it's also taken a lot of CPU time - there are ~21,000 source packages in Debian Jessie! As is often the case for a brand new architecture like arm64 (or AArch64, to use ARM's own terminology), hardware can be really difficult to get hold of. In time this will cease to be an issue as hardware becomes more commoditised, but in Debian we really struggled to get hold of equipment for a very long time during the early part of the port. First bring-up in Debian Ports To start with, we could use ARM's own AArch64 software models to build the first few packages. This worked, but only very slowly. Then Chen Baozi and the folks running the Tianhe-2 supercomputer project in Guangzhou, China contacted us to offer access to some arm64 hardware, and this is what Wookey used for bootstrapping the new port in the unofficial Debian Ports archive. This has now become the normal way for new architectures to get into Debian. We got most of the archive built in debian-ports this way, and we could then use those results to seed the initial core set of packages in the main Debian archive. Second bring-up - moving into the main Debian archive By the time that first Debian bring-up was done, ARM was starting to produce its own "Juno" development boards, and with the help of my boss^4 James McNiven we managed to acquire a couple of those machines for use as official Debian build machines. The existing machines in China were faster, but for various reasons quite difficult to maintain as official Debian machines. So I set up the Junos as buildds just before going to DebConf in August 2014. They ran very well, and (for dev boards!) were very fast and stable. They built a large chunk of the Debian archive, but as the release freeze for Jessie grew close we weren't quite there. There was a small but persistent backlog of un-built packages that were causing us issues, plus the Juno machines are/were not quite suitable as porter boxes for Debian developers all over the world to use for debugging their packages on the new architecture. More horsepower - Linaro machines This is where Linaro came to our aid. Linaro's goal is to help improve Free and Open Source Software on ARM, and one of the more recent projects in Linaro is a cluster of servers that are made available for software developers to use to get early access to ARMv8 (arm64) hardware. It's a great way for people who are interested in this new architecture to try things out, port their software or indeed just help with the general porting effort. As Debian is seen as such an important part of the FLOSS ecosystem, we managed to negotiate dedicated access to three of the machines in that cluster for Debian's use and we set those up in October, shortly before the freeze for Jessie. Andy Doan spent a lot of his time getting these machines going for us, and then I set up two of them as build machines and one as the porter box we were still needing. With these extra machines available, we quickly caught up with the ever-busy "Needs-Build" queue and we've got sufficient build power now to keep things going for the Jessie release. We were officially added to the list of release architectures at the Cambridge mini-Debconf in November, and all is looking good now! And in the future? I've organised the loan of another arm64 machine from AMD for Debian to use for further porting and/or building. We're also expecting that more and more machines will be coming out soon as vendors move on from prototyping to producing real customer equipment. Once that's happened, more kit will be available and everybody will be able to have arm64-powered computers in the server room, on their desk and even inside their laptop! Mine will be running Debian Jessie... :-) Thanks! There's been a lot of people involved in the Debian arm64 bootstrapping at various stages, so many that I couldn't possibly credit them all! I'll highlight some, though. :-) First of all, Wookey's life has revolved around this port for the last few years, tirelessly porting, fixing and hacking out package builds to get us going. We've had loads of help from other teams in Debian, particularly the massive patience of the DSA folks with getting early machines up and running and the prodding of the ftpmaster, buildd and release teams when we've been grinding our way through ever more package builds and dependency loops. We've also had really good support from toolchain folks in Debian and ARM, fixing bugs as we've found them by stressing new code and new machines. We've had a number of other people helping by filing bugs and posting patches to help us get things built and working. And (last but not least!) thanks to all the folks who've helped us beg and borrow the hardware to make the Debian arm64 port a reality. Rumours of even more ARM ports coming soon are entirely scurrilous... *grin*

Review: Code Complete, Second Edition, by Steve McConnell

Publisher:	Microsoft
Copyright:	June 2004
ISBN:	0-7356-1967-0
Format:	Kindle
Pages:	960

As mentioned in the title, this is a review of the second edition of Code Complete, published in 2004. There doesn't appear to be a later edition at the time of this writing. I should say, as a prefix to this review, that I'm the sort of person who really likes style guides. When learning a language, a style guide is usually the second or third document I read. I enjoy debates over the clearest way to express a concept in code, trying to keep all the code in a large project consistent, and discussing the subtle trade-offs that sit on the boundary between mechanical style issues and the expressiveness of programming. I try to spend some time reading good code and getting better at expressing myself in code. Presumably, therefore, I'm the target audience for this book. It sounded good from the descriptions, so I picked it up during one of the Microsoft Press sales. The stated goal of Code Complete is to collect in one place as much as possible of the oral tradition and lore of the programming field, to try to document and communicate the techniques and approaches that make someone a good programmer. The table of contents sounds like a style guide, with entire sections on variables and statements in addition to topics like how to improve existing code and how to design a new program. If you're starting to think that a 960 page style guide sounds like a bad idea, you're wiser than I. (In my defense, I grabbed this as an ebook and didn't realize how large it was before I bought it.) I have not actually finished this book. I hate to do this: I don't like reviewing books I haven't finished (this will be the first), and I hate starting books and not finishing them. (This is probably not particularly wise, since some books aren't worth finishing, but I've gotten into a rhythm of reading and reviewing that works for me, so I try not to mess with it.) But I've been trying to finish this book off and on for about a year, I don't think it's worth the time investment, and I think I've gotten far enough in it to provide some warnings by others who are deceived by the very high ratings that it gets on Amazon and other places. The primarily problem with Code Complete is its sheer, mind-numbing comprehensiveness. It tries to provide a set of guidelines and a checklist to think about at each level of writing code. This is one of those ideas that might sound good on paper, but which completely doesn't work for me. There is no way I'm going to keep this many rules in my head, in the form of rules, while programming. Much of good style has to be done by feel, and the book I'm looking for is one that improves my feel and my sense of taste for code. What Code Complete seems to provide instead is a compilation of every thought that McConnell has ever had about programming. There's a lot of basic material, a few thoughtful statements, a ton of style advice, an endless compilation of trade-offs and concepts that one should keep in mind, and just a massive, overwhelming pile of stuff. Each chapter (and there are a lot of chapters) ends in a checklist of things that you should think about when doing a particular programming task. To give you a feel for the overwhelming level of trivia here, this is the checklist at the end of the chapter where I stopped reading, on quality assurance in software. This is one picked at random; a lot of them are longer than this.

• Have you identified specific quality characteristics that are important to your project?
• Have you made others aware of the project's quality objectives?
• Have you differentiated between external and internal quality characteristics?
• Have you thought about the ways in which some characteristics might compete with or complement others?
• Does your project call for the use of several different error-detection techniques suited to finding several different kinds of errors?
• Does your project include a plan to take steps to assure software quality during each stage of software development?
• Is the quality measured in some way so that you can tell whether it's improving or degrading?
• Does management understand that quality assurance incurs additional costs up front in order to save costs later?

I'm not saying those are bad things to think about with quality assurance, but you may notice a few issues immediately. They're very general and vague, they're not phrased in a particularly compelling or memorable way, and there are a lot of them. This falls between two stools: it's too much for the programmer who is thinking about quality as part of an overall project but not focusing on it (particularly when you consider that the book is full of checklists like this for everything from variable naming to how to structuring if statements to program debugging), but it's not nearly specific or actionable enough for someone who is focusing on quality assurance. It's not that the information isn't organized: there's a lot of structure here. And there are bits and pieces here that are occasionally interesting. McConnell is very data-driven and tries to back up recommendations with research on error rates and similar concrete measurements. It's just insufficiently filtered and without elegant or memorable summary. There is far too much here, an overwhelming quantity, and hopelessly mixed between useful tidbits and obvious observations that anyone who has been programming for a while would pick up, all presented in the same earnest but dry tone. It didn't help that there's a lot here I didn't agree with. Some of that is to be expected: I've never agreed completely with any style guide. But McConnell kept advocating variable and function naming conventions that I find rather ugly and tedious, and the general style of code he advocates feels very "bureaucratic" to me. It's not exactly wrong, but one of the things that I look for in style discussions is to be inspired by the elegant and simple way someone finds to phrase something in code. A lot of the code in this book just felt mind-numbing. It's functional, but uninteresting; perfectly adequate for a large project, but not the sort of discussion that inspires me to improve the quality of my craft. So, I didn't finish this. I gave up about halfway through. It's frustrating, since I was occasionally finding an interesting nugget of information. But they were too few and far between, and the rest of the book was mostly... boring. It's possible that I just know too much about programming to be the person for whom that McConnell was writing this book. It's certainly true that the book has not aged particularly well; it's focused on fairly old-school languages (C, C++, Java, and Visual Basic) and says almost nothing about modern language techniques, although it does have a bit about extreme programming. But whatever the reason is, it didn't work for me at all. I would rate it as one of the worst books about programming I've tried to read. And that's notably different enough from its reviews that it seems worth throwing this out there as a warning. I'm quite disappointed, since I'd heard nothing but praise for this book before picking it up. But it's not for me, and I'm now dubious of its value for any programmer outside of a fairly narrow, large-team, waterfall development process involving large numbers of people writing very large quantities of code in languages that aren't very expressive. And, well, in that situation I think one would get more benefit from changing that environment than reading this book. Rating: unfinished

This was my first month working on Debian LTS. My first project at Codethink was winding down, so Freexian's Debian LTS initiative was able to hire me via Codethink. I spent all of the assigned 11.5 hours working on an update to the kernel package (linux-2.6, version 2.6.32-48squeeze9). We had stopped following the upstream stable branch maintained by Willy Tarreau after 2.6.32.60 (released October 2012). Since then, we have only applied specific security fixes and other critical fixes. Rapha l Hertzog and Holger Levsen started to rebase our package on 2.6.32.64 (released November 2014), bringing in a few security fixes we didn't yet have and a larger number of fixes for functional and performance issues. I spent most of my time reviewing the several hundred changes from the upstream stable branch. I found a number of mistakes that would have caused regressions. Those should all be fixed in the update to linux-2.6, though I did not have nearly enough time for a thorough regression test. I sent my fixes to Willy for inclusion in 2.6.32.65. I also reviewed and applied fixes for several security flaws in the kernel entry and exit paths. Andy Lutomirski identified and fixed a number of problems upstream, the most serious of which was CVE-2014-9322 (though this is not listed in the changelog because the details weren't yet public). Willy found and backported the upstream fixes for inclusion in 2.6.32.65. I checked that these make sense (so far as I understand this code) and verified that Andy's test cases now have the expected results when run on the new kernel version. Updated: Added references to Codethink and Freexian.

Review: Ancillary Sword, by Ann Leckie

Series:	Imperial Radch #2
Publisher:	Orbit
Copyright:	October 2014
ISBN:	0-316-24665-4
Format:	Trade paperback
Pages:	354

This is the second book in the Imperial Radch series and a direct sequel to Ancillary Justice. You don't want to read this book out of order, since the previous book sets up the background of everything that happens here. Besides, Ancillary Justice is an amazing book. It's going to be challenging to review Ancillary Sword without spoiling the previous book. If you're planning on reading Ancillary Justice but haven't gotten to it, you may want to stop here and come back to this review after you've read it. Or, even better, just read both books. They're some of the best science fiction I've read. Ancillary Justice started small, with one person and their quixotic search for revenge, and grew large, to encompass conflicts and confrontations that would shake the Radch. Ancillary Sword returns to a smaller scale and stays there. This means that much of what was left unresolved at the end of the previous book is still unresolved; Leckie does not continue escalating into large-scale conflict. It also means that we see a lot more of Breq making personal choices and trying to work out her own sense of morality, plus semi-adopting a couple more injured people along the way. One of my favorite types of stories is where I get to watch someone who is very good at something do the thing that they're very good at. Breq's unique background and experience makes her a wildcard outsider with vast experience in her new role. (Not to mention the special advantages she has from her implants.) Her long experience with people, similarly from a unique perspective, lets her use her power to effectively navigate political situations while keeping people slightly off-balance. And now she has some real power, made more potent for being somewhat ill-defined. In short, this is a story of political agency, given to someone who hasn't had it before but who is very good at using it. It's immensely satisfying, in part because it's not a simple wish fulfillment. Breq can't just reshape the world to her preferences; in fact, she can't do much about one of the social conflicts she runs into, except treat the people involved with unexpected respect. But she can occasionally do something, and she can always upset existing power structures in subtle ways, and the way Leckie writes this makes it so much fun to read. I think one of the reasons why I enjoyed this so much is that Breq is not relentlessly introspective. She just acts. Usually this sort of book involves lots of soul-searching and analysis, and the lack is refreshing. The other people in the story analyze Breq much more than she analyzes herself, sometimes incorrectly, and Breq finds the whole thing faintly amusing. Not only does this keep the story from bogging down in too much internal drama, it means that Breq frequently surprises the reader, usually in ways that had me grinning. And, despite not mulling things over incessantly, she is growing and developing, finding her own sense of morality and and ethics in a way that's sometimes only apparent in retrospect. The one caveat I will mention is that this is a book that concerns itself a great deal with colonialism and racial slavery, but it's a fantasy of political agency focused on someone who's part of the dominant culture. While it's not quite accurate to say that Breq is this world's equivalent of white, she can pass, and she's Radchaai. I thought the book handles the issues reasonably well, but it is still using oppressed cultures to focus on the agency and power of someone who is, comparatively, privileged. This didn't bother me while I was reading the story, but it started to bother me a little afterwards once it was pointed out. There's nothing inherently wrong with that story, but it's a rather common pattern, and I'm afraid Ancillary Sword doesn't do much to broaden the pattern. That said, it's a caveat rather than a fatal flaw, at least for me. Ancillary Sword is obviously the middle book of a trilogy, and normally the lack of forward progress on the overarching story and the sense of filling in background and setting the scene would undermine the book. But Breq and the other characters in this world are so fascinating that I didn't mind. The ending was not quite what I expected, but worked better the more that I thought about it. I'm really looking forward to the next book. Followed by Ancillary Mercy. Rating: 9 out of 10

I finally decided it was about time to get my whole-house sound project off the ground. As an added bonus, I d like to be able to stream music from my house to my Android phone. Some Background It was about 2.5 years ago that I last revisited the music-listening picture on Linux. I used Spotify for awhile, but the buggy nature of its support for local music eventually drove me up the wall. I have a large collection of music that will never be on Spotify (local choirs, for instance) and this was an important feature. When Google Play Music added the feature of uploading your local collection, I used that; it let me stream music from my phone in my car (using the Bluetooh link to the car). I could also listen at home on a PC, or plug my phone into various devices to play. But that was a hassle, and didn t let me have music throughout the house. Google Play is reasonable for that, but it has a number of really glaring issues. One is that it often gets album artwork wrong; it doesn t use the ID3 tags embedded in the files, but rather tries to guess . Another is that the sync is only add . Move files to another place in your collection, or re-rip them to FLAC and replace your old MP3s, and suddenly they re in Google Play twice. It won t ever see updated metadata, either quite a hassle for someone that uses Musicbrainz and carefully curates metadata. My Hardware I already have an oldish PC set up as a entertainment box running MythTV. It is a diskless system (boots PAE over the network and has NFS root) and very quiet. It has video output to my TV, and audio output via S/PDIF to my receiver. It is one logical audio frontend. My workstation in my office is another obvious place. My kitchen has a radio with a line in jack, and I also have a small portable speaker with a line in jack to make the last two options. I also have a Raspberry Pi model B that I bought awhile ago and was looking for a purpose, so I thought this should be cheap and easy, right? Well. Cheap, yes. Easy, not so much. First attempt: Ampache The Ampache project produces quite a nice piece of software. Ampache has matured significantly in the last 2.5 years, and the usability of its web-interface with HTML5 and Flash players as options is quite impressive. It is easily as usable as Google s, though its learning curve is rather more steep. There are multiple Android apps for Ampache to stream remotely. And, while most are terribly buggy and broken, there is at least one that seems to work well. Ampache can also output m3u/pls files for a standalone player. It does on-the-fly transcoding. There are some chinks in the armor, however. The set of codecs that are transcoded or passed through is a global setting, not a per-device setting. The bitrates are per-account, so you can t easily have it transcode FLAC into 320Kbps mp3 for streaming on your LAN and 128Kbps MP3 for streaming to your phone. (There are some hacks involving IP address ranges and multiple accounts, but they are poorly documented and cumbersome.) Ampache also has a feature called localplay in which it drives local players instead of remote ones. I tried to use this in combination with mpd to drive music to the whole house. Ampache s mpd interface is a bit odd; it actually loads things up into mpd s queue. Sadly it shares the same global configs as the rest. Even though mpd is perfectly capable of handling FLAC audio, the Ampache web player isn t, so you have to either make it transcode mp3 for everything or forego the web player (or use a second account that has transcode everything set). Frustratingly, not one of the Android clients for Ampache is even remotely compatible with Localplay, and some will fail in surprising ways if you have been using Localplay on the web client. So let s see how this mpd thing worked out. Ampache with mpd for whole-house audio The primary method here is to use mpd s pulseaudio driver. I configured it like so:
audio_output
type "pulse"
name "MPD stream"
#server "remote_server" # optional
sink "rtp" # optional
mixer_type "software"

Then in /etc/pulse/system.pa:
load-module module-null-sink sink_name=rtp format=s16be channels=2 rate=44100 sink_properties="device.description='RTP Multicast Sink'"
load-module module-rtp-send source=rtp.monitor
This tells Pulse to use multicast streaming to the LAN for the audio packets. Pulse is supposed to have latency synchronization to achieve perfect audio everywhere. In practice, this works somewhat poorly. Plus I have to install pulse everywhere, which inserts its tentacles way too deeply into the ALSA stack for my taste. (alsamixer suddenly turns useless by default, for instance.) But I gave it a try. After much fiddling Pulse is rather poorly documented and the interactions of configuration tools with it even more so I found a working configuration. On my MythTV box, I added:
load-module module-rtp-recv
load-module module-native-protocol-tcp auth-ip-acl=127.0.0.1;x.x.x.x
The real annoyance was getting it to set the output through the S/PDIF digital port. Finally I figured out the magic potion:
set-card-profile 0 output:iec958-stereo+input:analog-stereo
This worked reasonably well. The Raspberry Pi was a much bigger challenge, however. I put Raspbian on it easily enough, and installed Pulse. But apparently it is well-known in the Pulse community that Pulse s RTP does not work well with wifi. Multicast itself works poorly with wifi in general, and Pulse won t do unicast RTP. However, Pulse assumes very low latency and just won t work well with wifi at all. Morever, pulseaudio on the rpi is something of a difficult beast to tame. It has crackling audio, etc. I eventually got it working decently with: daemon.conf:
realtime-scheduling = yes
realtime-priority = 5
resample-method = src-sinc-fastest
default-sample-rate = 44100
default-fragments = 4
default-fragment-size-msec=20
and in system.pa, commented out the module-udev-detect and module-detect, and added:
load-module module-alsa-card device_id=0 tsched=0 fragments=10 fragment_size=640 tsched_buffer_size=4194384 tsched_buffer_watermark=262144
plus the rtp-recv and tcp protocol lines as before. This got it working with decent quality, but it was always out of sync by at least a few tenths of a second, if not a whole second, with the other rooms. Brief diversion: mplayer Many players besides pulseaudio can play the RTP streams that pulseaudio generates. mplayer, for instance, can. I found this worked on the Raspberry Pi:
mplayer -really-quiet -cache 64 -cache-min 95 -demuxer rawaudio -rawaudio format=0x20776172 rtp://224.0.0.56:46510
But this produced even worse sync. It became clear that Ampache was not going to be the right solution. Logitech Media Server I have looked at this a few times over the years, but I ve somehow skipped it. But I looked at it again now. It is an open source (GPL) server, and was originally designed to work with Logitech hardware. There are now all sorts of software clients out there. There is a helpful wiki about it, although Logitech has rebranded the thing so many times, it s not even consistent internally on what the heck it s called (is it LMS? or Squeezebox Server? or Slimserver? The answer is: yes.) LMS does no web-based streaming. At all. But I thought I d give it a try. Installation is a little weird (its .deb packages up binary perl modules for things that are already in Debian, for half a dozen architectures and many Perl versions.) I had some odd issues but eventually it worked. It scanned my media collection. I installed squeezelite on my workstation as my first player, and things worked reasonably well out of the box. I proceeded to install squeezelite on the Raspberry Pi and MythTV server (where I had to compile libsoxr for wheezy, due to an obscure error I couldn t find the cause of until I used strace). Log in to the LMS web interface, tell it to enable synchronization, and bam! Perfect synchronization! Incredible. But I ran into two issues: one was that it used a lot of CPU even on my workstation (50% or more, even when idle). Strangely, it used far less CPU on the pi than did pulseaudio. Secondly, I d get annoying clicks from it from time to time. Some debugging and investigation revealed that they were both somewhat related; it was getting out of sync with the pi and correcting. Initially I assumed this to be an issue with the pi, but tracked it down to something else. I told it to upsample to 48kHz and this made the problem go away. My command on PC hardware is:
squeezelite -a 200:10 -o dmix -u hLX -z
And on the Raspberry Pi:
squeezelite -a 800:10:16:0 -n kitchen -z
It is working perfectly now. LMS even has an Android remote control app that can control all devices together, queue up playlists, etc. Very slick. The Raspberry Pi The Raspberry Pi, even the model B+, apparently has notoriously bad audio output. I noticed this first as very low volume on the output. I found a $11 USB DAC on Amazon that seems to do the trick, so we will see. I am also using the $10 Edimax USB Wifi and it seems to work well. Other Options The big-name other option here is Sonos. They have perfect sync audio already working. But they re pricy, closed, and proprietary; you have to buy Sonos speakers, transmitters, etc. for the entire house. Their receivers start at $200, and it s at least $350 to integrate an existing stereo into the system. I d be shelling out over $1000 to use Sonos for this setup. As it is, I ve bought less than $100 of equipment (a second Pi and its accessories) and getting output that, while no doubt not quite as pristine, it still quite nice and acceptable. Quite nice. Update: LMS with Apache It turns out that Logitech Media Server can automatically save created playlists to a spot where Ampache can find them. So an Ampache streaming player can be used to access the same collection LMS uses, with full features. Nice!

Nice, crisp and pretty cold morning for the 2015 New Year's 5k. I had run this before with friends: 2003, 2005, 2006 and 2007. Needless to say, I am a little slower now: A hand-stopped 23:13 for a 7:27 min/mile pace is fine by me given the weather, lack of speedwork in the last few months, and generally reduced running---but also almost a minute slower than the last time I ran this. In case you're a fellow running geek and into this, Strava has my result nicely aggregated.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Each software tool exists to solve some problem. For each problem, there are many possible solutions. Even when different programs basically do the same thing, they can have quite different shapes. As an example, this morning I was wondering if it would be possible for me to use notmuch to index my entire mail archive. For that, I needed to convert a number of mbox folders to Maildir format. That's a resonably easy problem, given access to suitable programming libraries, but there's an existing tool for that, called mb2md. Unfortunately, it has the wrong shape for my needs. mb2md doesn't just convert one mbox to one maildir. It's designed to for a mail admin converting all server-side mbox folders for a user into a corresponding structure of Maildir folders. This seems to be necessary when switching IMAP servers. That's a fairly specialised problem, and the program has been written to make it easy for a mail admin to do that. What I need is part of the problem solved by mb2md and indeed it can do just that part. However, the overall shape of mb2md is such that my part is hard to do. The incantation is quite unintuitive and requires careful reading of the documentation. The shape of a solution matters. mb2md could easily have been written in a way that provides a simple tool for the single folder conversion, and then a more complex tool for the mail admin's more complicated problem. This would have resulted in a much more general tool, and that would make it easier for more people to use it without much effort. Mail folder format conversions are a fairly esoteric thing to do. However, the lack of generality is a frequent issue with how programs are designed. It is easy to fall into the trap of writing a highly specialised tool, instead of taking a step back and making a more general purpose tool. The specialised tool will help a small number of people. The general tool will help many people. Examples of this are fairly common. Debian has a set of tools for making Debian live CDs; they are not quite able to make a bootable hard disk image as well (thus, vmdebootstrap). There's programs for computing cyclomatic complexity, which produce HTML reports, rather than something that can be processed by other programs without too much effort. There's tools for managing address books that are limited to specific cultures, e.g., by hardcoding assumptions of what a person's name looks like (thus, clab). One of my favourite examples is xargs, which by default does the wrong thing by assuming its input is whitespace delimited. Any whitespace, not just newlines. Any sensible use requires adding the -0 option, which makes xargs that much more tedious to use. Furthermore, I've often found that the more general tool is simpler. It's functional specification is simpler; it's implementation is simpler, and has fewer special cases; it's user experience is simpler. That's not always true, but often it is. Sometimes the general solution shape is not worth it. But it's always worth considering whether it might be. One of the parts of the Unix culture I really like is the preference for general tools that are easy to combine together.

Review: Three Parts Dead, by Max Gladstone

Series:	Craft #1
Publisher:	Tor
Copyright:	October 2012
ISBN:	1-4668-0203-0
Format:	Kindle
Pages:	336

Tara Abernathy was a student in the Hidden Schools, learning Craft, until she was expelled. Literally expelled: thrown from the floating schools to crash painfully to earth in the Badlands, left to return to her family and village and a life of small workings of Craft and contracts on behalf of local farmers. She had largely resigned herself to that life until raiders started killing people. Tara is not the sort of person who could stand by and watch that, or someone to refrain from using Craft to fix the world. The result was undead guardians for the town, perhaps unwisely formed from the town's risen dead, and only a job offer saves Tara from the ungrateful attention of her neighbors. That's how Tara finds herself employed by the firm of Kelethras, Albrecht, and Ao, in the person of partner Elayne Kevarian. Provisionally, depending on her performance on their job: the investigation of the death of a god. It's possible to call Three Parts Dead urban fantasy if you squint at it the right way. It is fantasy that takes place largely in cities, it features the investigation of a crime (and, before long, several crimes), and Tara's attitude is reminscent of an urban fantasy heroine. But this is considerably different from the normal fare of supernatural creatures. In this world, magic, called Craft, is an occupation that requires a great deal of precision and careful construction. Small workings are described similar to magic, although with an emphasis on metaphor. Larger workings more often come in the form of energy flows, contracts, and careful hedging, and the large Craft firms bear more resemblence to mergers and acquisitions specialists than to schools of wizards. This means that the murder investigation of the god of Alt Coulumb involves a compelling mix of danger, magic, highly unusual library investigations, forensic accounting, hidden Craft machinery, unexpected political alliances, and an inhuman police force. Rather than the typical urban fantasy approach of being beaten up until the resolution of the mystery becomes obvious, Tara and her companions do quite a lot of footwork and uncover a more complex political situation than they were expecting. And, in keeping with this take on magic, the story culminates in a courtroom drama (of a sort). I really enjoyed this. It combines the stylistic elements of urban fantasy that I like with some complex and original world-building and a great take on magical contracts. I prefer worlds like this one, where any source of power people have lived with for a long time is surrounded by the controls, formal analysis, and politics that humans create around anything of value. Tara is also a great protagonist. This is a coming of age story in a sense, and Tara is sometimes unsure of her abilities, but it's refreshingly devoid of worry or angst over new-found abilities. Tara enjoys her work, and approaches it with a well-written mix of uncertainty, impulsiveness, and self-confidence (sometimes warranted, sometimes not). I've read some good stories where the protagonist gets dragged into the story against their will, and some of them are quite good, but it's refreshing to read a book about someone who takes to the story like a duck to water. This is a believable protrayal of a character with a lot of native ability and intelligence, not much wisdom (yet), but a lot of thoughtful enthusiasm. I was disappointed to learn that she isn't the protagonist of the next book in the series. The biggest flaw I found in this book is that Gladstone doesn't stick reliably to his world conception. At times, Craft collapses into something more like typical fantasy magical battles, instead of legal procedure and contract made concrete. I suppose this makes parts of the book more exciting, but I would have preferred a plot resolution that involved less combat and more argument. This isn't helped by the utterly hissable villain. There's a lot of complexity in understanding what happened and who was going to benefit (and how), but there is absolutely no doubt who the enemy is, and he's essentially without redeeming qualities. I would have preferred more nuance, given how satisfyingly complex the rest of the world-building is. Three Parts Dead also occasionally suffers from the typical first novel problem of being a bit overstuffed. The world-building comes fast and thick, and nearly everything Tara does involves introducing new concepts. But the world does have a coherent history, and quite a lot of it. It used to be a more typical fantasy world ruled by gods, each with their own territory and worshippers (and Alt Coulumb is a throwback to this era), but an epic war between gods and Craft is in Tara's past, leading to the defeat or destruction of many of the gods. She lives in a time of uneasy truce between human and inhuman powers, featuring some very complex political and economic alliances. There's a lot of material here for an ongoing series. This is a great first novel. It's not without its flaws, but I enjoyed it from beginning to end, and will definitely keep reading the series. Recommended. Followed by Two Serpents Rise. Rating: 8 out of 10

I just watched a CCC talk in which the speaker claims Perl is horribly broken. Watching it was fairly annoying however, since I had to restrain myself from throwing things at the screen. If you're going to complain about the language, better make sure you actually understand the language first. I won't deny that there are a few weird constructions in there, but hey. The talk boils down to a claim that perl is horrible, because the list "data type" is "broken". First of all, Netanel, in Perl, lists are not arrays. Yes, that's confusing if you haven't done more than a few hours of Perl, but hear me out. In Perl, a list is an enumeration of values. A variable with an '@' sigil is an array; a construct consisting of an opening bracket ('(') followed by a number of comma- or arrow-separated values (',' or '=>'), followed by a closing bracket, is a list. Whenever you assign more than one value to an array or a hash, you need to use a list to enumerate the values. Subroutines in perl also use lists as arguments or return values. Yes, that last bit may have been a mistake. Perl has a concept of "scalar context" and "list context". A scalar context is what a sub is in when you assign the return value of your sub to a scalar; a list context is when you assign the return value of your sub to an array or a hash, or when you use the list construct (the thing with brackets and commas) with sub calls (instead of hardcoded values or variables) as the individual values. This works as follows:

sub magic  
    if (wantarray())  
        print "You're asking for a list!";
        return ('a', 'b', 'c');
      else  
        print "You're asking for a scalar!";
        return 'a';
     
 
print ("list: ", magic(), "\n");
print "scalar: " . magic() . "\n";

The above example will produce the following output:

You're asking for a list!
list: abc
You're asking for a scalar!
scalar: a

What happens here? The first print line creates a list (because things are separated by commas); the second one does not (the '.' is perl's string concatenation operator; as you can only concatenate scalars, the result is that you call the magic() sub in scalar context). Yes, seen as how arrays are not lists, the name of the wantarray() sub is horribly chosen. Anyway. It is documented that lists cannot be nested. Lists can only be one-dimensional constructs. If you create a list, and add another list as an element (or something that can be converted to a list, like an array or a hash), then the result is that you get a flattened list. If you don't want a flattened list, you need to use a reference instead. A reference is a scalar value that, very much like a pointer in C, contains a reference to another variable. This other variable can be an array, a hash, or a scalar. But it cannot be a list, because it must be a variable -- and lists cannot be variables. If you need to create multi-dimensional constructs, you need to use references. Taking a reference is done by prepending a backslash to whatever it is you're trying to take a reference of; or, in the case of arrays of hashes, one can create an anonymous array or hash with [] resp . E.g., if you want to add a non-flattened array to a list, you instead create a reference to an array, like so:

$arrayref = [ 'this', 'is', 'an', 'anonymous', 'array'];

you can now create a multi-dimensional construct:

@multiarray = ('elem1', $arrayref);

Or you can do that in one go:

@multiarray = ('elem1', [ 'this', 'is', 'an', 'anonymous', 'array']);

Alternatively, you can create a non-anonymous array first:

@onedimarray = ('this', 'is', 'not', 'an', 'anonymous', 'array');
@multiarray = ('elem1', \@onedimarray);

In perl, curly brackets can be used to create a reference to anonymous hashes, whereas square brackets can be used to create a reference to anonymous arrays. This is all a basic part of the language; if you don't understand that, you simply don't understand Perl. In other words, whenever you see someone doing this:

%hash =  'a' => 'b' ;

or

@array = [ '1', '2' ];

you can say that they don't understand the language. For reference, the assignment to %hash will result in an (unusable) hash with a single key that is a reference to an anonymous hash (which cannot be accessed anymore) and a value of undef; the assignment to @array will result in a two-dimensional array with one element in the first dimension, and two elements in the second. The CGI.pm fix which Natanel dismisses in the Q&A part of the talk as a "warning" which won't help (because it would be too late) is actually a proper fix, which should warn people in all cases. That is, if you do this:

%hash =   'name' => $name, 'password' => $cgi->param('password')  ;

then CGI.pm's param() sub will notice that it's being called in list context, and issue a warning -- regardless of whether the user is passing one or two password query-parameters. It uses the wantarray() sub, and produces a warning if that returns true. In short, Perl is not the horribly broken construct that Natanel claims it to be. Yes, there are a few surprises (most of which exist for historical reasons), and yes, those should be fixed. This is why the Perl community has redone much of perl for Perl 6. But the fact that there are a few surprises doesn't mean the whole language is broken. There are surprises in most languages; that is a fact of life. Yes, the difference between arrays and hashes on the one hand, and lists on the other hand, is fairly confusing; it took me a while to understand this. But once you get the hang of it, it's not all that difficult. And then these two issues that Natanel found (which I suppose could be described as bugs in the core modules) aren't all that surprising anymore. So, in short:

• Don't stop using Perl. However, do make sure that whenever you use a language, you understand the language, first, so you don't get bitten by its historical baggage. This is true for any language, not just Perl.
• Don't assume that just because you found issues with core modules, the whole language is suddenly broken.

What I do agree with is that if you want to use a language, you should understand its features. Unfortunately, this single line in the final slide of Natanel's talk is just about the only thing in the whole talk that sortof made sense to me. Ah well.

Here is a slightly overdue status on Jessie.

• We are not ready to set a date on a Jessie release yet. Even if we were, it would be unlikely that said date would be in January. Accordingly, it is safe to assume that the rapid automatic removals clause of the freeze policy will be applied before our release. If your package (or a package you use) depend on any package in this list, then they are at risk.
• It is unclear to me whether our CD1s are able to contain all the necessary packages. This was a major issue for Wheezy. So far I have only done some minor prodding here. However, I really want this item done soon as it can easily take a month or more to resolve this.
• While the release notes have been improved quite a bit, I am certain that there are more cases we can cover. As an example, I am looking for input on #773386.
• Despite the declining number of RC bugs, we still have some particular unhealthy ones left. E.g. #759530. There is also a view of some of the known unfixed Jessie blockers.
• We have had a number of issues with APT and dpkg (e.g. trigger cycles, APT breaking under puppet) that caused upgrade failures or were a severe regression. Most of these have been resolved now. There are some trigger issues left and I have pushed for a fixed dpkg at the cost of possibly removing some packages. See #772866 (among others).

Stricter freeze policy per January 5th The next timed change of the freeze policy will apply per January 5th. After that date, we will only accept RC bugs fixes. Which means it is final chance for translation updates. More on RC bugs In absolute numbers, the RC bugs have declined quite well. We are below 150 now. We lost quite a bit of traction in December compared to November. However, November was an extremely efficient month. However, we still need the final push here. Debian installer release pending Yesterday, we received a list of packages that needed to be unblocked for d-i with a remark that a release of d-i might follow. Based on what we have unblocked previously, it will likely contain some (improved?) UEFI support. Pending Debian 7.8 release While not directly relevant to Jessie, we also got a pending Wheezy release planned for the 10th of January. The window for getting changes into the 7.8 release closes this weekend. Want to help?

• File bugs against release-notes (source at [RN source]) and installation-guide for missing or outdated documentation. Patches and drafts especially welcome.
• Fix RC bugs especially the known Jessie blockers.
• Test upgrade paths and installation media. Although in both cases, you may want to wait a bit (for the new dpkg to migrate and for the new debian-installer release respectively).
• Consider offering your help to teams such as the CD team or debian-installer team.

Thank you, [RN source]: https://anonscm.debian.org/viewvc/ddp/manuals/trunk/release-notes/ svn co https://anonscm.debian.org/viewvc/ddp/manuals/trunk/release-notes/ Git Repo: http://anonscm.debian.org/cgit/users/jcristau/release-notes.git/
Filed under: Debian, Release-Team

Here is a slightly overdue status on Jessie.

• We are not ready to set a date on a Jessie release yet. Even if we were, it would be unlikely that said date would be in January. Accordingly, it is safe to assume that the rapid automatic removals clause of the freeze policy will be applied before our release. If your package (or a package you use) depend on any package in this list, then they are at risk.
• It is unclear to me whether our CD1s are able to contain all the necessary packages. This was a major issue for Wheezy. So far I have only done some minor prodding here. However, I really want this item done soon as it can easily take a month or more to resolve this.
• While the release notes have been improved quite a bit, I am certain that there are more cases we can cover. As an example, I am looking for input on #773386.
• Despite the declining number of RC bugs, we still have some particular unhealthy ones left. E.g. #759530. There is also a view of some of the known unfixed Jessie blockers.
• We have had a number of issues with APT and dpkg (e.g. trigger cycles, APT breaking under puppet) that caused upgrade failures or were a severe regression. Most of these have been resolved now. There are some trigger issues left and I have pushed for a fixed dpkg at the cost of possibly removing some packages. See #772866 (among others).

Stricter freeze policy per January 5th The next timed change of the freeze policy will apply per January 5th. After that date, we will only accept RC bugs fixes. Which means it is final chance for translation updates. More on RC bugs In absolute numbers, the RC bugs have declined quite well. We are below 150 now. We lost quite a bit of traction in December compared to November. However, November was an extremely efficient month. However, we still need the final push here. Debian installer release pending Yesterday, we received a list of packages that needed to be unblocked for d-i with a remark that a release of d-i might follow. Based on what we have unblocked previously, it will likely contain some (improved?) UEFI support. Pending Debian 7.8 release While not directly relevant to Jessie, we also got a pending Wheezy release planned for the 10th of January. The window for getting changes into the 7.8 release closes this weekend. Want to help?

• File bugs against release-notes (source at [RN source]) and installation-guide for missing or outdated documentation. Patches and drafts especially welcome.
• Fix RC bugs especially the known Jessie blockers.
• Test upgrade paths and installation media. Although in both cases, you may want to wait a bit (for the new dpkg to migrate and for the new debian-installer release respectively).
• Consider offering your help to teams such as the CD team or debian-installer team.

Thank you, [RN source]: https://anonscm.debian.org/viewvc/ddp/manuals/trunk/release-notes/ svn co https://anonscm.debian.org/viewvc/ddp/manuals/trunk/release-notes/ Git Repo: http://anonscm.debian.org/cgit/users/jcristau/release-notes.git/
Filed under: Debian, Release-Team

Conrad produced another minor release 4.600 of Armadillo. As before, I had created a GitHub-only pre-release(s) of his pre-release(s), and tested a pre-release as well as the actual release against the now over one hundred CRAN dependents of our RcppArmadillo package. The tests passed fine as usual with less than a handful of checks not passing, all for known cases -- and results are as always in the rcpp-logs repository. Changes are summarized below based on the NEWS.Rd file.

Changes in RcppArmadillo version 0.4.600.0 (2014-12-27)

• Upgraded to Armadillo release Version 4.600 ("Off The Reservation")
  • added .head() and .tail() to submatrix views
  • faster matrix transposes within compound expressions
  • faster accu() and norm() when compiling with -O3 -ffast-math -march=native (gcc and clang)
  • workaround for a bug in GCC 4.4

Courtesy of CRANberries, there is also a diffstat report for the most recent release. As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

I have been hard at work on the shell-monad ever since it was born on Christmas Eve. It's now up to 820 lines of code, and has nearly comprehensive coverage of all shell features. Time to use it for something interesting! Let's make a shell script and a haskell program that both speak a simple protocol. This kind of thing could be used by propellor when it's deploying itself to a new host. The haskell program can ssh to a remote host and run the shell program, and talk back and forth over stdio with it, using the protocol they both speak. abstract beginnings First, we'll write a data type for the commands in the protocol.

data Proto
    = Foo String
      Bar
      Baz Integer
    deriving (Show)

Now, let's go type class crazy!

class Monad t => OutputsProto t where
    output :: Proto -> t ()
instance OutputsProto IO where
    output = putStrLn . fromProto

So far, nothing interesting; this makes the IO monad an instance of the OutputsProto type class, and gives a simple implementation to output a line of the protocol.

instance OutputsProto Script where
    output = cmd "echo" . fromProto

Now it gets interesting. The Script monad is now also a member of the OutputsProto. To output a line of the protocol, it just uses echo. Yeah -- shell code is a member of a haskell type class. Awesome -- most abstract shell code evar! Similarly, we can add another type class for monads that can input the protocol:

class Monad t => InputsProto t p where
    input :: t p
instance InputsProto IO Proto where
    input = toProto <$> readLn
instance InputsProto Script Var where
    input = do
        v <- newVar ()
        readVar v
        return v

While the IO version reads and deserializes a line back to a Proto, the shell script version of this returns a Var, which has the newly read line in it, not yet deserialized. Why the difference? Well, Haskell has data types, and shell does not ... speaking the protocol Now we have enough groundwork to write haskell code in the IO monad that speaks the protocol in arbitrary ways. For example:

protoExchangeIO :: Proto -> IO Proto
protoExchangeIO p = do
    output p
    input
fooIO :: IO ()
fooIO = do
    resp <- protoExchangeIO (Foo "starting up")
    -- etc

But that's trivial and uninteresting. Anyone who has read to here certianly knows how to write haskell code in the IO monad. The interesting part is making the shell program speak the protocol, including doing various things when it receives the commands.

foo :: Script ()
foo = do
    stopOnFailure True
    handler <- func (NamedLike "handler") $
        handleProto =<< input
    output (Foo "starting up")
    handler
    output Bar
    handler
handleFoo :: Var -> Script ()
handleFoo v = toStderr $ cmd "echo" "yay, I got a Foo" v
handleBar :: Script ()
handleBar = toStderr $ cmd "echo" "yay, I got a Bar"
handleBaz :: Var -> Script ()
handleBaz num = forCmd (cmd "seq" (Val (1 :: Int)) num) $
    toStderr . cmd "echo" "yay, I got a Baz"

serialization I've left out a few serialization functions. fromProto is used in both instances of OutputsProto. The haskell program and the script will both use this to serialize Proto.

fromProto :: Proto -> String
fromProto (Foo s) = pFOO ++ " " ++ s
fromProto Bar = pBAR ++ " "
fromProto (Baz i) = pBAZ ++ " " ++ show i
pFOO, pBAR, pBAZ :: String
(pFOO, pBAR, pBAZ) = ("FOO", "BAR", "BAZ")

And here's the haskell function to convert the other direction, which was also used earlier.

toProto :: String -> Proto
toProto s = case break (== ' ') s of
    (w, ' ':rest)
          w == pFOO -> Foo rest
          w == pBAR && null rest -> Bar
          w == pBAZ -> Baz (read rest)
          otherwise -> error $ "unknown protocol command: " ++ w
    (_, _) -> error "protocol splitting error"

We also need a version of that written in the Script monad. Here it is. Compare and contrast the function below with the one above. They're really quite similar. (Sadly, not so similar to allow refactoring out a common function..)

handleProto :: Var -> Script ()
handleProto v = do
    w <- getProtoCommand v
    let rest = getProtoRest v
    caseOf w
        [ (quote (T.pack pFOO), handleFoo =<< rest)
        , (quote (T.pack pBAR), handleBar)
        , (quote (T.pack pBAZ), handleBaz =<< rest)
        , (glob "*", do
            toStderr $ cmd "echo" "unknown protocol command" w
            cmd "false"
          )
        ]

Both toProto and handleProto split the incoming line apart into the first word, and the rest of the line, then match the first word against the commands in the protocol, and dispatches to appropriate actions. So, how do we split a variable apart like that in the Shell monad? Like this...

getProtoCommand :: Var -> Script Var
getProtoCommand v = trimVar LongestMatch FromEnd v (glob " *")
getProtoRest :: Var -> Script Var
getProtoRest v = trimVar ShortestMatch FromBeginning v (glob "[! ]*[ ]")

(This could probably be improved by using a DSL to generate the globs too..) conclusion And finally, here's a main to generate the shell script!

main :: IO ()
main = T.writeFile "protocol.sh" $ script foo

The pretty-printed shell script that produces is not very interesting, but I'll include it at the end for completeness. More interestingly for the purposes of sshing to a host and running the command there, we can use linearScript to generate a version of the script that's all contained on a single line. Also included below. I could easily have written the pretty-printed version of the shell script in twice the time that it took to write the haskell program that generates it and also speaks the protocol itself. I would certianly have had to test the hand-written shell script repeatedly. Code like for _x in $(seq 1 "$ _v#[!\ ]*[\ ] ") doesn't just write and debug itself. (Until now!) But, the generated scrpt worked 100% on the first try! Well, it worked as soon as I got the Haskell program to compile... But the best part is that the Haskell program and the shell script don't just speak the same protocol. They both rely on the same definition of Proto. So this is fairly close to the kind of type-safe protocol serialization that Fay provides, when compiling Haskell to javascript. I'm getting the feeling that I won't be writing too many nontrivial shell scripts by hand anymore! :) the complete haskell program Is here, all 99 lines of it. the pretty-printed shell program

#!/bin/sh
set -x
_handler ()   :
    _v=
    read _v
    case "$ _v%%\ * " in FOO) :
        echo 'yay, I got a Foo' "$ _v#[!\ ]*[\ ] " >&2
    : ;; BAR) :
        echo 'yay, I got a Bar' >&2
    : ;; BAZ) :
        for _x in $(seq 1 "$ _v#[!\ ]*[\ ] ")
        do :
            echo 'yay, I got a Baz' "$_x" >&2
        done
    : ;; *) :
        echo 'unknown protocol command' "$ _v%%\ * " >&2
        false
    : ;; esac
 
echo 'FOO starting up'
_handler
echo 'BAR '
_handler

the one-liner shell program

set -p; _handler ()   :;    _v=;    read _v;    case "$ _v%%\ * " in FOO) :;        echo 'yay, I got a Foo' "$ _v#[!\ ]*[\ ] " >&2;     : ;; BAR) :;        echo 'yay, I got a Bar' >&2;    : ;; BAZ) :;        for _x in $(seq 1 "$ _v#[!\ ]*[\ ] ");      do :;           echo 'yay, I got a Baz' "$_x" >&2;      done;   : ;; *) :;      echo 'unknown protocol command' "$ _v%%\ * " >&2;       false;  : ;; esac;  ; echo 'FOO starting up'; _handler; echo 'BAR '; _handler

It Will Never Work in Theory is a web site that blogs, though slowly, of important research and findings about software development. It's one of the most interesting sites I've found recently, possibly for a long time. I disagree with the term "software engineering" to describe the software development that happens today. I don't think it's accurate, and indeed I think the concept's too much of a fantasy for the term to be used seriously about practicing developers do. For software development to be an engineering discipline, it needs a strong foundation based on actual research. In short, we need to know what works, what doesn't work, and preferably why in both cases. We don't have much of that. This website is one example of how that's now changing, and that's good. As a practicing software developer, I want to know, for example, whether code review actually helps improve software quality, the speed of software development, and the total cost of a software project, and also under what the limits of code review are, how it should be done well, and what kind of review doesn't work. Once I know that, I can decide whether and how to do reviews in my development teams. The software development field is full of anecdotal evidence about these things. It's also full of people who've done something once, and then want to sell books, seminars, and lectures about it. That's not been working too well: it makes research be mostly about fads, and that's no way to build a strong foundation. Now I just need the time to read everything, and the brain to understand big words.