Note: This post will probably not be on the liking of those who think free software is always preferable to closed source software, so if you are such a person, please take this article as an invitation to implement better open source alternatives that can realistically compete with the closed source applications I am mentioning here. I am not going to mention here where the open source alternatives are not up to the same level as the commercial tools, I'll leave that for the readers or for another article.



Semanticmerge is a merge tool that attempts to do the right thing when it comes to merging source code. It is language aware and it currently supports Java and C#. Just today the creators of the software have started working on the support for C.

Recently they added Debian packages, so I installed it on my system. For open source development Codice Software, the creators of Semanticmerge, offers free licenses, so I decided to ask for one today, and, although is Sunday, I received an answer and I will get my license on Monday.

When a method is moved from one place to another and changed in a conflicting way in two parallel development lines, Semanticmerge can isolate the offending method and can pass all its incarnations (base, source and destination or, if you prefer, base, mine and theirs) to a text based merge tool to allow the developer to decide how to resolve the merge. On Linux, the Semanticmerge samples are using kdiff3 as the text-based merge tool, which is nice, but I don't use kdiff3, I use Meld, another open source visual tool for merges and comparisons.


OTOH, Beyond Compare is a merge and compare tool made by Scooter Software which provides a very good text based 3-way merge with a 3 sources + 1 result pane, and can compare both files and directories. Two of its killer features is having the ability split differences into important and non-important ones according to the syntax of the compared/merged files, and the ability to easily change or add to the syntax rules in a very user-friendly way. This allows to easily ignore changes in comments, but also basic refactoring such as variable renaming, or other trivial code-wide changes, which allows the developer to focus on the important changes/differences during merges or code reviews.

Syntax support for usual file formats like C, Java, shell, Perl etc. is built in (but can be modified, which is a good thing) and new file types with their syntaxes can be added via the GUI from scratch or based on existing rules.

I evaluated Beyond Compare at my workplace and we decided it would be a good investment to purchases licenses for it for the people in our department.


Having these two software separate is good, but having them integrated with each other would be even better. So I decided I would try to see how it can be done. I installed Beyond compare on my system, too and looked through the examples


The first thing I discovered is that the main assumption of Semanticmerge developers was that the application would be called via the SCM when merges are to be done, so passing lots of parameters would not be problem. I realised that when I saw how one of the samples' starting script invoked semantic merge:

semanticmergetool -s=$sm_dir/src.java -b=$sm_dir/base.java -d=$sm_dir/dst.java -r=/tmp/semanticmergetoolresult.java -edt="kdiff3 \"#sourcefile\" \"#destinationfile\"" -emt="kdiff3 \"#basefile\" \"#sourcefile\" \"#destinationfile\" --L1 \"#basesymbolic\" --L2 \"#sourcesymbolic\" --L3 \"#destinationsymbolic\" -o \"#output\"" -e2mt="kdiff3 \"#sourcefile\" \"#destinationfile\" -o \"#output\""

Can you see the problem? It seems Semanticmerge has no persistent knowledge of the user preferences with regards to the text-based merge tool and exports the issue to the SCM, at the price of overcomplicating the command line. I already mentioned this issue in my license request mail and added the issue and my fix suggestion in their voting system of features to be implemented.

The upside was that by comparing the command line for kdiff3 invocations, the kdiff3 documentation and, by comparison, the Beyond Compare SCM integration information, I could deduce what is the command line necessary for Semanticmerge to use Beyond Compare as an external merge and diff tool.

The -edt, -emt and -e2mt options are the ones which specify how the external diff tool, external 3-way merge tool and external 2-way merge tool is to be called. Once I understood that, I split the problem in its obvious parts, each invocation had to be mapped, from kdiff3 options to beyond compare options, adding the occasional bell and whistle, if possible.

The parts to figure out, ordered by compexity, were:

1. -edt="kdiff3 \"#sourcefile\" \"#destinationfile\"


2. -e2mt="kdiff3 \"#sourcefile\" \"#destinationfile\" -o \"#output\""


3. -emt="kdiff3 \"#basefile\" \"#sourcefile\" \"#destinationfile\" --L1 \"#basesymbolic\" --L2 \"#sourcesymbolic\" --L3 \"#destinationsymbolic\" -o \"#output\""

Semantic merge integrates with kdiff3 in diff mode via the -edt option. This was easy to map to Beyond Compare, I just replaced kdiff3 with bcompare:

-edt="bcompare \"#sourcefile\" \"#destinationfile\""

Integration for 2-way merges was also quite easy, the mapping to Beyond Compare was:

-e2mt="bcompare \"#sourcefile\" \"#destinationfile\" -savetarget=\"#output\""

For the 3-way merge I was a little confused because the Beyond Compare documentation and options were inconsistent between Windows and Linux. On Windows, for some of the SCMs, the options that set the titles for the panes are '/title1', '/title2', '/title3' and '/title4' (way too descriptive for my taste /sarcasm), but for some others are '/lefttitle', '/centertitle', '/righttitle', '/outputtitle', while on Linux the options are the more explicit kind, but with a '-' instead of a '/'.

The basic things were easy, ordering the parameters in the 'source, destination, base, output' instead of kdiff3's 'base, source, destination, -o ouptut', so I wanted to add the bells and whistles, since it really makes more sense for the developer to see something like 'Destination: [method] readOptions' instead of '/tmp/tmp4327687242.tmp', and because that's exactly what is necessary for Semanticmerge when merging methods, since on conflicts the various versions of the functions are placed in temporary files which don't mean anything.

So, after some digging into the examples from Beyond Compare and kdiff3 documentation, I ended up with:

-emt="bcompare \"#sourcefile\" \"#destinationfile\" \"#basefile\" \"#output\" -lefttitle='#sourcesymbolic' -righttitle='#destinationsymbolic' -centertitle='#basesymbolic' -outputtitle='merge result'"

Sadly, I wasn't able to identify the symbolic name for the output, so I added the hard coded 'merge result'. If Codice people would like to help with with this information (or if it exists), I would be more than willing to do update the information and do the necessary changes.

Then I added the bells and whistles for the -edt and -e2mt options, so I ended up with an even more complicated command line. The end result was this monstrosity:

semanticmergetool -s=$sm_dir/src.java -b=$sm_dir/base.java -d=$sm_dir/dst.java -r=/tmp/semanticmergetoolresult.java -edt="bcompare \"#sourcefile\" \"#destinationfile\" -lefttitle='#sourcesymbolic' -righttitle='#destinationsymbolic'" -emt="bcompare \"#sourcefile\" \"#destinationfile\" \"#basefile\" \"#output\" -lefttitle='#sourcesymbolic' -righttitle='#destinationsymbolic' -centertitle='#basesymbolic' -outputtitle='merge result'" -e2mt="bcompare \"#sourcefile\" \"#destinationfile\" -savetarget=\"#output\" -lefttitle='#sourcesymbolic' -righttitle='#destinationsymbolic'"

So when I 3-way merge a function I get something like this (sorry for high resolution, lower resolutions don't do justice to the tools):



I don't expect this post to remain relevant for too much time, because after sending my feedback to Codice, they were open to my suggestion to have persistent settings for the external tool integration, so, in the future, the command line could probably be as simple as:

semanticmergetool -s=$sm_dir/src.java -b=$sm_dir/base.java -d=$sm_dir/dst.java -r=/tmp/semanticmergetoolresult.java

And the integration could be done via the GUI, while the command line can become a way to override the defaults.

This is my monthly summary of my free software related activities. If you re among the people who made a donation to support my work (47.50 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Package Tracking System There are only 2-3 weeks left in the summer of code project dedicated to rewrite the package tracking system. We have come a long way during August check it out yourself in pts.debian.net. The rewrite doesn t have all the features of the old PTS yet, but I opted to keep some of the easy and less interesting features for others to re-implement. Instead I asked Marko to work in the coming weeks on new features that will bring more value, like the possibility to have user accounts with the possibility to easily review and tweak all your subscriptions on the web, and like the possibility to subscribe to groups of packages (i.e. those managed by a team). Our main problem right now is that exim has a pretty poor default behavior of forking hundreds of processes if you get hundreds of mails (in a batch) to an address that delivers via a pipe (postfix is saner, it serializes the deliveries on pipes). The new PTS is much more modular and its memory footprint is bigger (about 3 times more for the process that delivers mails, 30Mb instead of 10Mb), and in such a situation we managed to run out of memory for now we worked around the situation with an exim setting that queues mails once the load gets too high but it s a poor workaround IMO. We could obviously implement our own queue and a daemon but I d like to avoid this. So who knows how to tell exim to behave? On the positive side, Marko has gotten some feedback from people who like the new PTS and are using it daily already. And several persons have expressed their interest to work on the new codebase already. On my side, I created a package so that it s easy to deploy for derivatives. In this process, I revamped the way we manage the Django settings (for development and for production). The package is not finished yet, but it s mostly usable already. But I still want to do some cleanup/refactoring in the models before others start deploying it. We must also enable South to make it possible to upgrade easily afterwards. DebConf 13 in Vaumarcus From August 10th to 17th, I was attending DebConf 13. It matched the only week of vacation that my wife had this summer so we went there with the whole family (that is with a 3 years old son, and 6 months old one). Thus I could not immerse myself in Debconf and missed all the nice things that happen outside of the talk rooms. I picked 3-4 interesting talks per day and I spent the rest with my family. On the positive side, I was pleased that my wife could meet (or at least see) some other Debian people. She knows quite a few (of you) by name because I have been telling her Debian stories for years now Debian France Debian France sold quite some merchandise during Debconf but I didn t take care of that. It was supervised by Sylvestre Ledru but fortunately he got the help of multiple persons, both to bring everything there, to sell it, and to bring back the rest. The good news of the month is that the upstream author of galette published a new version with all the features that we ordered him a few months ago. We send now automatic reminders to members who must renew their subscription, we have automatic update of our accounting books (in a ledger file in a git repository) when we people donate or pay their subscription via the paypal form on our website. I was so pleased to finally have this that I took some hours to finalize the packaging of galette, so that it could be uploaded to Debian. It s now waiting in the NEW queue. I also spent multiple hours to write the python script that is executed by galette and that updates the accounting files. Misc Debian stuff Debian Packaging. I did two uploads of logidee-tools to fix bugs #718671 and #718836. I created a package for Dolibarr a PHP-based CRM and ERP software (it doesn t do accounting however), it s sitting in the NEW queue for almost a month already. I forwarded #719000 to the upstream Publican developers. I filed #720393 to request a new upstream version of libphp-mailer. git-multimail. After its deployment on Alioth last month, Niels Thykier reported me a case where it lead to bounces, I filed this as a new upstream ticket and in fact I fixed it myself a few days after. I got the fixed version installed on Alioth. dpkg. I investigated why the the automatic builds of dpkg were no longer happening and asked Michael Prokop if he could install a newer version of gettext in the build chroot. He told me that he would need a backport for that so I asked Santiago Vila if he was willing to provide it and he kindly accepted. A few days after, the package was in backports and I m now again running the latest dpkg out of git thanks to the nice service provided by Michael. Misc discussions. The thread about user planets drifted into a discussion of how to avoid promotional posts on such planets and in that context someone again brought up the Debian Machine Usage Policy as a way to shut down any kind of (self-)promotional content on planet if there s money involved. This always irritates me and this time I opted to ask James Troup about the origin of that clause in the DMUP. So who is willing to work with DSA to fix the DMUP so that people stop abusing it in contexts where it doesn t make sense? I also participated in some discussions concerning dgit. I like the ideas behind the tool, but I m saddened by the behavior of Ian Jackson. I helped him to fill his gap of knowledge about new sources formats but he keeps on bashing about the 3.0 (quilt) source format both in the manual page and in the output of the program. He believes that dgit is no longer an experiment but the truth is that it s still a poorly commented Perl script doing lots of hackish things. Kali Linux Between Debconf and all, I haven t done much for Kali except a couple of fixes. There s a nice story of how I tracked a bug in live-installer on the Kali blog. That fix has been committed to Debian. I also improved live-build to include xfsprogs/jfsutils on the ISO image when you include the debian-installer (so that you don t end up in problems when you pick JFS or XFS as file systems for your installation). Thanks See you next month for a new summary of my activities.

One comment Liked this article? Click here. My blog is Flattr-enabled.

AddThis:

Categories:

• Debian-Blog
• Tools

Keywords:

• laptop-mode-tools

Review: An Artificial Night, by Seanan McGuire

Series:	October Daye #3
Publisher:	DAW
Copyright:	September 2010
ISBN:	1-101-44276-X
Format:	Kindle
Pages:	368

This is the third book of the October Daye series, following A Local Habitation. It could be read out of order, as each chapter of Toby's story so far is somewhat standalone, but I wouldn't recommend it. You'd miss the character development and the background behind several friendships. As with the previous book review, a caveat: I am reviewing this book four months after finishing it, rather than the normal couple of weeks, so apologies in advance for any thinness or inaccuracies. The first sign that something is about to go seriously wrong in Toby's life this time is a visitor. She says her name is May Daye, and apparently she's Toby's Fetch: a duplicate in nearly every respect except personality, manifested to guide someone to their death. Their imminent death. The second sign is that her best friend's children are kidnapped. The children who treat her like an aunt. And the cause of that kidnapping is far, far beyond the level of power that Toby could be expected to deal with. I have profound mixed feelings about this book. On one hand, McGuire further expands Toby's web of connections and shows us more of Toby's friends and their background, and that's becoming my favorite part of this series. She introduces May, who I think is a great character: at first apparently much different than Toby, but with a similarity beneath surface impressions that one notices more over time. And I thought the climax of the story, once we finally got there, was very moving and well-done. Also, I just love the way that McGuire uses children's rhymes and doggerel throughout this series as a magical element, and that's particularly well-handled here. Due to the nature of the antagonist, McGuire has a good excuse to bring that front and center, and the process of familiar rhymes expanding into deep magical rules was a fun intellectual experience. Best of all, the Luidaeg plays a significant role in the story again. The Luidaeg is, hands down, my favorite character in McGuire's constructed universe, and she becomes even more so over the course of this story. Her complex relationship with Toby is clearly blossoming into a friendship that's surprising both of them deeply, but a friendship that has to play by an odd set of rules (like so much else in Toby's life). I loved her here. Seeing someone that difficult and abrupt show unexpected emotional depths is one of the story elements I enjoy the most. The problem, though, is that quite a lot of An Artificial Night is very, very dark. And by that, I don't mean that there is a lot of tension, or that Toby is struggling against monsterous things, although both of those statements are also true. Rather, Toby is not only out of her league but ends up in some viciously nasty situations, and they persist for rather more of this book than I would have preferred. Putting one's protagonist through hell has a long tradition in urban fantasy, but I found the level of sheer helplessness that Toby struggled through to be a bit more than I wanted to read about. It was too reminiscent of events from Toby's background (as discussed in Rosemary and Rue) that I still don't like to think about. There is, of course, an eventual payoff, as dark as the interim events seem. One knows that, given the genre and given that this is an ongoing series, and even just given the nature of stories. But as much as I like the growing cast of this series, and adored the scenes with the Luidaeg, I found parts of this journey to be deeply disturbing, and not in an enjoyable dramatic tension sort of way. I don't mind protagonists being seriously hurt (although it's not my favorite plot), or even some amount of despair, but this went beyond that. I think McGuire was going for a story about sacrifice and self-definition, about trying even when one can't possibly win, and I can appreciate the narrative arc. But parts of the book were simply awful to read for me. I'm therefore not quite sure what to say in terms of a recommendation. I'm still reading the series, and will definitely read the next book. I think my reaction is somewhat idiosyncratic; the events here will probably not bother others as much as they bothered me. And I can't disagree with their dramatic effectiveness, or with the way that McGuire builds conflict out of the rules of fairy and is building Toby into a stronger and more capable person than she ever thought she could be. But there were still spots where I wished I were reading a different book. Hopefully the next book in the series will be that book I wanted to read. Followed by Late Eclipses. Rating: 6 out of 10

Review: Redshirts, by John Scalzi

Publisher:	Tor
Copyright:	June 2012
ISBN:	1-4299-6360-3
Format:	Kindle
Pages:	317

The next few reviews deserve a small apology in advance. Normally, I try to write reviews within a few weeks, or at most a month, after reading a book, while it's still fresh in my mind. This is particularly important since I read a lot and sometimes don't have a good memory for specific detail. But this summer has made it completely impossible to keep to any sort of regular writing schedule. As a result, this book, and the subjects of the next few reviews that follow, are books that I read four months ago and am reviewing from memory. Hopefully, I'll still get the details right. Ensign Andrew Dahl is one of the newest crew members of the Universal Union Capital Ship Intrepid. It's his first posting, just out of the Academy. He's one of five new ensigns, expecting a normal tour of duty (albeit with the pride of being on a capital ship of the fleet). But very shortly after arriving on the ship, Andrew and and the other new ensigns discover that something is very... strange. The casualty rate on the ship is surprisingly high (as foreshadowed in a prologue). The other crew members avoid the bridge crew, and particularly away missions, with startling effectiveness (and desperation). And things regularly happen on the ship that, for lack of a better description, simply make no sense. Redshirts starts off as a very obvious Star Trek parody. So obvious, in fact, that it's eyeroll-inducing. The prologue reads exactly like a section of a Star Trek episode written by a third-rate hack, leavened with some of the humor that readers of Scalzi's blog have come to expect. But it's obvious that Scalzi is doing this intentionally, and it doesn't take long before the subject of the book shifts. It's a Star Trek parody that functions much more like a Star Trek parody than any universe should, to the degree that it becomes obvious to the characters. And the characters, despite being straight from central casting, are clearly not expecting their universe to act that way and find this extremely strange. By far the best part of this book is the second quarter: the characters have been introduced, they're starting to figure out what's going on, and they have to do so while dodging away-team duty (or trying to survive it) and participating in the ridiculous plot contrivances that keep occurring. Scalzi mixes a feeling of serious consequences with wry humor and handles the mix deftly. I was both laughing and eagerly reading to see what comes next. The last half of the book, though, has problems. How many problems the reader perceives will, I think, depend on how hard one thinks about the situation Scalzi has set up, or how much one has thought about similar constructs in the past. But I'm afraid the setup does not survive serious scrutiny in ways that, for me at least, seriously damaged my ability to engage with the story. I'm going to avoid saying explicitly where Scalzi goes with this plot, since it is a spoiler (although it's commonly mentioned in discussions of this book on the Internet). But it happens to be an idea that some friends of mine and I pounded on for years in some of our role-playing scenarios, poking at the corners and implications, and subsequently gave up on because it doesn't work for storytelling. The idea is compelling, and on first glance seems like a great story seed. But it's too large. The problem with really big ideas with huge implications is that they can change the nature of a story so much that there's no longer any point of commonality for the reader and the entire nature of story falls apart. And I think that's what happens here. Scalzi handles the deeper implications of the concept by, essentially, ignoring them. He focuses on one narrow set of implications for one specific set of people, largely skips over any discussion of anything larger, and tries a pivot from wry humor (and a bit of slapstick) into serious emotional depth. For me, in part because I'm so familiar with the concept, none of this really worked. The implications of the discovery that drives this story are so much broader and so much more significant than the characters seem to admit that it threw me right out of the story. The ethical implications are absolutely paralyzing, to the point that I don't think you can write a coherent story about them, but ignoring them doesn't work either. It's the kind of discovery that requires action of some kind beyond the attempt by the characters to go back to what they consider a normal life, even if I have a hard time imagining what that action could be. To be fair, he does try to salvage this in the first epilogue, which is all about the implications of his idea and its effects. But it's still too small. That epilogue just barely touches on the implications on a single person, but if one thinks a bit longer, one starts coming up with more and more implications for all of society, for how we talk about story, for how we raise children, for nearly every form of creativity. Implications that involve nasty concepts like slavery and mind control and murder and the nature of consent in ways that are very difficult to avoid. I think Scalzi's attempts at serious emotional connection, including in the other two attached epilogues (which are perspective-shifting close focuses on some of the characters), also suffer from his world setup, but in a different way. The characters are simply too bland to sustain the level of emotional connection he's striving for. They're bland for a very specific reason that is entirely justified by the story that Scalzi is handling, and the implications of that for the primary protagonist were, I thought, one of the best-handled moments of the book. But they're still bland, which undermines the attempted pivot. All that being said, this is not a bad book. Parts of it, particularly the second quarter, are excellent. And even when I wasn't engaging with the characters, Scalzi's writing on a paragraph level is snappy and often quite funny. The first epilogue in particular is hilarious, even if I think it was too limited in the scope of implications that it considers and a bit too meta in where it goes with those implications. And the story is a fun romp, with drama and humor and tension and a fairly satisfying ending. Everyone's a bit too bland and earnest when Scalzi isn't being funny, but he's funny enough that I didn't mind that much. Despite the relatively low rating, I actually recommend this book. If you don't dive into the metaphysical implications, it's a lot of fun, and if you've thought a lot about this particular set of metaphysical implications, it's somewhat satisfying to argue with (and to internally point out all of the hard questions that Scalzi is skipping over). It's particularly fun if you've watched a lot of Star Trek, since Scalzi gets some of the absurdities note-perfect. But be aware that the character reactions to Scalzi's concept don't hold up to much scrutiny. Rating: 7 out of 10

While I usually don't announce releases of tioga, as it is now mostly in maintenance mode, with not a lot of new features, today is a different story. I finally got around to try and build it on windows, and after a bit of a fight, I got it to compile and work there (I got hit by the classical fopen(file, "w") for binary files, that don't make any difference on Linux and Mac, but that do on Windows...). The release is already available on the rubyforge download page. Gems will be available later on today.To actually make it work on windows, here is what you need to install:

• Ruby, using the one-click installer. The 1.9.3 version works just fine. Don't forget to click the box to add the executables to the PATH
• the Ruby DevKit, at the same location. Make sure you take the one matching the Ruby version you installed (there's an explicative text on the side). You need to unpack it in the directory where Ruby got installed
• you need to get a LaTeX distribution, try out proTeXt
• at this point, you may need to reboot
• then, just open a windows terminal (Execute cmd.exe) and run:
  

  gem install tioga
  

And that's it ! That also means that ctioga2 works on windows, if you run:

gem install ctioga2

For me, DebConf13 in Switzerland was a perfect DebConf, and generally an great developer meeting and conference. I've attended several past DebConfs in Canada, Norway, Brazil, Finland, Mexico, Scotland, Spain, NYC, Bosnia, and Nicaragua. I have special memories of each. DebConf13 was ideal in every way. Let's start with the camping. Only a minority of us did it, and my campsite was located in a small field behind a bar that was often noisy late into the night, when I was trying to rest and preserve my European jetlag remedy of inverted sleeping schedule. Still, I love sleeping cocooned in a tent, and going through the routines of camping and spending time alone there was a good counterpoint to DebConf. I generally find myself retreating for a day or two in the middle of a week-long conference, burnt out with social interactions. That didn't happen this time. I need to find way to camp at future DebConfs. (Also special shout out to wonderful people who lent me a warmer sleeping bag, and bought me a pillow while on a trip to town! Upgraded my first backpacking in Europe experience's comfort level a lot!) The next perfect thing for me were the talks I gave at DebConf. I always enjoy presenting at DebConf, and barely remember when I'd get nervous doing it. But I really felt comfortable giving these talks in a way I've not before. I'm pleased with how the Debian Cosmology talk turned out; I'd worried it would be a downer talk, or too light, or not well received, and none of those problems surfaced. I was especially happy with my dh_busfactor session, which I prepared for all of 10 minutes and was structured entirely to benefit me by talking over my thoughts and getting feedback, and in which I probably drove the camerapeople crazy running my own mic into the audience. Despite all that, I think it succeeded being interesting for the participants as well as for me. Hope to follow up on it soon.. But beyond the two talks I submitted, there were a series of ad-hoc scheduled talks. It seemed that every day I spent at least an hour discussing something I'm deeply involved in. This started with the Ikiwiki bof, and then there was an epic series of (unrecorded) BoFs about git stuff, which basically turned into an unofficial git track at DebConf. Much, much discussion of git-annex and other things I'm involved in. BTW, I really like that DebConf, as well as being a formal conference on top, is really an un-conference underneath, with new BoFs and things popping up on the schedule all week long. You can even have completely unscheduled meetings that somehow get just the right mix of people to make something awesome happen.
But it wasn't all about talks. Indeed, I only attended 2 or 3 sessions a day. The DebConf13 venue was controversial, but I found it excellent for getting things done. Rather than one or two massive rooms that get too noisy to think in, this venue had many little nooks and crannies suited for different types of work. There were so many I only discovered some while walking down the hill as I was leaving the venue! Anchoring it all was the excellent view, and I particularly enjoyed the "scenic hacklab". I was particularly impressed with the flexability of the venue's management. Someone asked if we could have cheese for breakfast, and every breakfast thereafter had not one but often three excellent cheeses. One day we'd be complaining that there were not enough vegetables in the food, and the next we'd be eating an excellent vegetarian Swiss lunch. They even apparently relaxed stringent Swiss rules on not allowing tents in sightlines visible from people further up the hill. They were clearly happy to welcome our crazy gathering to Le Camp. They let a DD borrow and use a chain saw and build a bonfire! The venue also encouraged getting out and exploring the surrounding countryside. Too often I end up spending the whole DebConf without seeing much of the area around. This time I at least got down the hill to see the castle in Vaumarcus a few times, and enjoyed many evening walks amoung the grape fields. The conference dinner (on a boat!) tied into this well too. And the daytrip to CERN was a great addition. We got to poke around the control center for a massive particle accellerator! My only regret is we didn't get a chance to actually go inside the LHC tunnels. Back to DebConf, another thing I really liked about this one was that its location allowed several upstream authors of software in Debian (other than the usual suspects who are both upstreams and Debian Developers) to attend. I don't know if Lennart Poettering changed any minds about systemd with his talk, but I was happy for the opportunity to buy him a beer. I didn't get to Monty's talk, but it was excellent to see him attend as well. Finally, I got to meet old friends I'd not seen in years, and new ones I've not met in real life before. We're still not sure where this whole DebConf thing is going -- will it be all about the cheese and wine party and greybeards in 20 years time? But one thing we know is we enjoy our little slice of the year where kilts and coding are equally unremarkable, where problems that seem intractable online somehow solve themselves in person, where we come away with ideas for an entire year and memories for a lifetime.

This week I have mostly:

Knocked up a forum ("gathering")

It is a simple clone of hacker news, storing things in Redis.None of the aging though, entries are first->last and my tagging support is basic. I'm in two minds about releasing it. I'm in three minds about deploying it at the location I'd written it for - forums.lumail.org - since there's nothing worse than a forum with no posts, unless it is a forum that used to be popular and is now a wasteland, or circlejerk. (c.f. slashdot. ahem.)

Updated Slaugther

I found a hidden dependency when installing a new slaughter controlled host.A new release is imminent.

Setup a remote host for backups

Using BigV I configured a system with LUKS-encrypted disks to act as a "remote dropbox" via git-annex, and rsync.An encrypted volume is manually mounted post-boot. It stays mounted (oops that's bad) but provides security when the guest is offline, or has been retired.

Wrestled with Graphite

Because damn that software is hard to install.

Turned down two weddings

Because I will never shoot a wedding again. And if I did I'd not do it for free for "exposure".

Merged my (digital) music archive with that belonging to my partner

Which took more discussion than a) moving in together, or b) opening a shared bank account.

Secretly decided I do like my kindle

Even though I bought two books from a charity shop tonight I'm almost certainly going to file them away and look for the epub online instead.

He fought with the goblins! He battled the trolls! He riddled with Gollum! The magic ring he stole! This weekend I'll be mostly offline. Saturating my home broadband while I sync backups.

During DebConf, Asheesh presented the idea of using git instead of the file system for storing the contents of Debian Code Search. The hope was that it would lead to fewer disk seeks and less data due to gits delta-encoding. Maybe the reduction would be big enough that enough data could be held in RAM to allow for fast retrieval. Joey Hess helped me out with a couple of details: he revealed that using git repack -a -d will lead to a single packfile that optimally contains HEAD, not caring about the history. Also, he showed me how to use git cat-file. We did a small-scale experiment and the results were promising. I told him I ll do the experiment of just committing the entire unpacked source mirror to git and promised to follow up with the results, so here goes:

stapelberg@couper ~/unpacked master $ time cat <<EOT   git cat-file --batch >/dev/null
:linux_3.2.32-1/sound/pci/ice1712/aureon.c
:linux_3.2.32-1/sound/soc/codecs/sgtl5000.c
:linux_3.2.32-1/sound/pci/hda/patch_conexant.c
EOT
cat <<<''  0,00s user 0,00s system 62% cpu 0,006 total
git cat-file --batch > /dev/null  4,38s user 1,08s system 99% cpu 5,477 total
stapelberg@couper ~/unpacked master $ time cat linux_3.2.32-1/sound/pci/ice1712/aureon.c linux_3.2.32-1/sound/soc/codecs/sgtl5000.c linux_3.2.32-1/sound/pci/hda/patch_conexant.c >/dev/null
cat linux_3.2.32-1/sound/pci/ice1712/aureon.c   > /dev/null  0,00s user 0,00s system 69% cpu 0,006 total
stapelberg@couper ~/unpacked master $ time cat <<EOT   git cat-file --batch >/dev/null
:i3-wm_4.6-1/libi3/font.c
EOT
cat <<<':i3-wm_4.6-1/libi3/font.c'  0,00s user 0,00s system 51% cpu 0,008 total
git cat-file --batch > /dev/null  4,30s user 1,18s system 99% cpu 5,533 total
stapelberg@couper ~/unpacked master $ time cat i3-wm_4.6-1/libi3/font.c >/dev/null
cat i3-wm_4.6-1/libi3/font.c > /dev/null  0,00s user 0,00s system 0% cpu 0,004 total

Even after repeating those tests a couple of times to get a warm page cache, the result stays the same: git takes about 5 seconds to resolve the deltas in a large repository. Even if this was 5 seconds startup time and a very small amount of additional time per file, it would not be acceptable for our use case. The conclusion is that git is clearly not suitable for this kind of usage, which is not surprising after having heard a couple of times that git does not scale :-). For the curious, the .git/objects directory is 29 GiB for roughly 140 GiB of source code, so the delta encoding is quite impressive in terms of saving space. However, keep in mind that the compressed (!) Debian source archive is about 35 GiB, so the savings are not that huge.

With the ifupdown fixes that I published last week I actually reached my initial goal, that is to make Debian/Hurd boot using sysvinit and the initscripts provided by Debian. So on Monday we were discussing in #hurd what I could do next. Michael Banck suggested that I should port Upstart, but we agreed to do something different instead for two reasons:

1. Upstart and systemd are somewhat competing to be the default init system for Debian, and we felt it might be inappropriate to get involved with this question as porting Upstart to Hurd would probably also enable it to be used on FreeBSD. The Upstart folks could then point out that Upstart is more portable because it runs on all kernels used by Debian.
2. Upstart uses ptrace(2) to track child processes of servers it monitors. Obviously this is kind of a hack, and it was conjectured that Upstart would eventually use cgroups to do that. Also, the Hurd lacks support for ptrace(2) (that is most likely by choice by the way, ptrace(2) is not a nice interface and the Hurd (Mach actually) has much nicer interfaces to implement a debugger).

So we decided that no matter how the struggle between Upstart and systemd turns out, the Hurd would eventually need to support cgroups. So I started to write a cgroupfs translator, it is in its early stages but it already looks and acts a lot like Linux' cgroups:

% settrans -ac cg ./cgroupfs --release-agent=foobar
% ls cg
release_agent  tasks
% tail -n3 cg/tasks
11395
12869
1266
% mkdir cg/foo
% echo 1266 >> cg/foo/tasks
% tail -n3 cg/tasks cg/foo/tasks
==> cg/tasks <==
215
11395
12869
==> cg/foo/tasks <==
1266

To make this fully functional I will have to modify /hurd/proc and most likely also GNU Mach, but on the bright side this will help make subhurds (Hurds native, by-design-for-free-and-without-overhead container like functionality) work better and more securely (among other things this could enable non-root users to start subhurds). I will also look into porting libcg (I have a hacky patch series ready) so that we can actually test the cgroupfs translator. All current users of the cgroup interface are very Linux specific (surprise!), and libcg looks like the easiest one to port. And they do have a test suite that could help me improve the cgroupfs translator.

E: Release file for http://snapshot.debian.org/archive/debian/20111207T220456Z/dists/unstable/InRelease is expired (invalid since 624d 17u 27min 15s). Updates for this repository will not be applied.

sigh.

Testing uncovered additional bugs in the WebLogin and mod_webkdc components of WebAuth (our site-wide web authentication system), so this is another bug-fix release. It only affects the central login server; the application module hasn't changed and doesn't need to be updated. The biggest changes are around multifactor authentication, and particularly around error reporting and handling. The handling of errors such as invalid codes or rejections by the validation service was somewhat broken in 4.5.3 and earlier and even more broken in 4.5.4 (although in different ways). This release now avoids sending the user to dead-end error pages in more circumstances, displays better and more specific errors, and remembers if the user has already been sent an SMS message and doesn't force them to send another if they enter the code incorrectly. (This last change will require changes to the local multifactor template.) This release also fixes replay detection, which had been broken by a partially-made change in the memcached key format, and applies rate limiting and replay detection to second factor logins as well as passwords. As a side effect, this corrects a UI issue where people can send themselves additional SMS messages by accident when using the back button on the browser. You can get the latest release from the official WebAuth distribution site or from my WebAuth distribution pages.

I ve noticed that my post from three months ago ( Bye bye non-free PHP JSON extension ) was quoted quite often (thanks (: ). But I also noticed that most people didn t read the post fully, and didn t see the replacement package. Hopefully the title of this post will correct that impression. I d like to indicate the existence of the php5-json package. It s part of Debian unstable since June 2013 and testing (jessie) since August 2013. Users who wish to build the free extension on their own are welcome to download it from PECL at http://pecl.php.net/package/jsonc . It s a good chance to thank Remi Collet for his work on this replacement. I m happy that the main Linux distributions have already adopted it.
Filed under: Uncategorized

I would probably not yet worthy of the admiration Emerson was alluding to but I do like to make things. As anyone who has read previous posts knows I have pretty much embraced the "do things, tell people" idea.

One small wrinkle is doing things needs somewhere to work. Since moving myself and the family to rented accommodation in Cambridge (swampy 3,500 year old English city, not the one in Massachusetts) I have been lacking space to do practical projects.

To fix this lack I have joined the cambridge makespace which, in addition to somewhere I can work gives me access to some tools I was previously unable to afford. The space gives practical training on the more complex machines (any tool can be dangerous if you do not use it correctly) which recently allowed my induction on the CNC router.

My instructor , Mark Mellors, who was good enough to give up some of his valuable making time to train me (and accidentally get his car stuck in a car park overnight by staying late) suggested that it was a good idea to have a design to try.

I decided to use this opportunity to actually create something useful (though now I re-read Marks message it did suggest a simple design...oopsy). I had been working at the electronics bench in previous weeks and been uncomfortable using the existing stools and chairs as they were either a bit high or unable to be adjusted high enough for the 80cm tall benches. I decided to design a 60cm tall stool for use at this workbench.

My initial idea was for a simple three leg stool, round top, three legs, how hard could it be? Initial research showed that showed that above 30cm the legs needed to be braced to each other. This is because the leg to seat joints simply cannot handle the stress caused by leverage which longer legs introduce.

I looked at the structure of several stools online and was initially drawn towards creating something like the IKEA Dalfred bar stool. It was discovered that the design would be easier to realise if it were made from sheet material which give a smaller challenge to a naive operator of a CNC router. Because of this simpler designs were researched and I eventually found a simple design I liked.

The design could not however be used directly as it was for imperial sized material which is not available in europe. I selected the QCAD open source CAD package and recaptured the design adjusting for the available 12mm plywood sheet material. This resulted in an imperial measurement design for metric materials.

Mark helped me transfer the DXF into the CAM software (vcarve pro 7, after I discovered the demo version of this software generates files which cannot be imported into the full edition!) and generate toolpaths for our machine. Once the toolpaths are saved to a USB stick (no modern conveniences like direct upload here) the job can be run on the machine.

Here I ran into reality, turns out that tolerances in imperial combined with lack of understanding how plywood reacts resulted in excessive play in the joints. This resulted in an unusable stool, which simply tried to rotate around its central axis and become flatpack. I had successfully turned 10 worth of plywood into some sawdust and a selection of useless shapes. On the other hand I did become competent with the router workflow so it was not a complete failure.

I was determined to make the design work so I decided to start from scratch with a similar design but entirely in metric. I performed some material research both online and practically (why yes i did spend an informative hour in several cambridge DIY shops with digital calipers, why do you ask?) it turns out that generally available 12mm thick plywood actually ranges between 11.9 and 12mm thick.

I did some test slot cuts of varying width and determined that the available stock can be "persuaded" to fit into a 11.8mm wide slot. This Interference fit joint is strong and removes the need for adhesive in most cases.

The sheet plywood material is readily available with a width of 1220mm and a height of 606mm (a full sheet is 2440mm long which is cut into four with a 4mm wide saw blade) so making the legs fit within a sheet and be close to the 60cm target should be possible.

I selected a suitable seat radius (175mm) and from that determined the minimal gap to the base ring with a 6mm end mill tool (20mm for two toolpaths and some separation) and hence the minimum suitable width of the base ring (50mm) giving a total radius of 245mm.

For the legs allowance was made for two joints of 30mm with 30mm separation between them the legs come out at 90mm width. If a 6mm space and a 6mm toolpath top and bottom of the sheet is accounted for a 582mm height (606 - 24) is available in which to fit the legs. The top of the leg which fits into the seat is 12mm tall leaving 570mm total leg height.

At this point I selected some arbitrary values for the leg positioning and angle (30mm from seat centre and 15 ) using a right angle triangle triganometry this produced a stool with a base of around 550mm or 100mm outside the radius of the seat. This seemed a pleasing shape and when the values for the holding ring were calculated it is at 104mm height which also seems to work out well.

This design was cut on the machine, lightly finished with some sandpaper and assembled. The legs slotted into the ring first and then the legs eased into the seat slots, the whole thing flipped and the seat hammered home onto the legs.

Success! It physically fit together and if you were brave enough you could sit on it. Unfortunately the plywood seemed to flex around the central axis of rotation in a rather alarming way, fine if you are under 70 kilos but not giving the impression of security most people want from their seating.

Ok, lets try with five legs instead of three (at least we can reuse the existing three legs)...nope still not good enough and another 10 gone.

The others in the space suggested a few ideas to improve matters and the one I selected was to use 18mm plywood instead of 12mm, this should improve rigidity. There was a brief pause in proceedings to discover 18mm sheet is actually 17.7mm and needs a 17.5mm slot to make the interference fit work.

A swift redesign later altering the seat radius, gap, ring width and leg height to accommodate the new material and we have version 4 and it works without caveat. Tested up to 150Kg load without trouble, there is a small amount of flex still but nothing that feels worrying.

I finished the stool by rounding the seat top edge with a ball bearing rounding bit in a manual router and applying a couple of coats of gloss acrylic varnish. Finished stool is now doing service at the space.

In conclusion the final design allows someone with a CNC router to create a useful 580mm high fixed stool for 7.50 in timber plus cutter wear and varnish so maybe 8.50 total.

You can actually get five legs and a seat/ring out of a 1220x606 sheet and with intelligent arrangement a single 1220x2440 sheet will probably yield five or possibly six stools in total

I am making the design files of the proven 18mm version available (heck they are all there...but you have been warned, none of the other solutions produce a satisfactory result) under the MIT licence so anyone can reproduce. More pretty pictures are also available.

This is part 5 in a series on organizational design and growth. What it means

Each of us has a job to do, probably more than one, and our teammates should know what they are. Why it s important

Roles are a kind of standing commitment we make to each other. They re a way of dividing up work which is easy to understand and simple to apply. Utilizing this tool will make it easier for us to coordinate our day to day work, and manage the changes and growth we re going through. Old Status Quo

Roles are vague or nonexistent. Management roles in particular are probably not well understood. Many people juggle multiple roles, all of which are implicit. Moving to a new team means learning from scratch what other people do. People take responsibility for tasks and decisions largely on a case-by-case basis, or based on implicit knowledge of what someone does (or doesn t do). In many cases, there is only one person in the company who knows how to perform a certain function. When someone leaves or goes on vacation, gaps are left behind. New Status Quo

Each individual has a clear understanding of the scope of their job. We have a handful of well defined roles, which are used by multiple teams and have written definitions. People who are new or transfer between teams have a relatively easy time understanding what the people around them are doing. Many day to day responsibilities are defined by roles, and more than one person can fill that role. When someone leaves a team, another person can cover their critical roles. Behaviors that help

• Define project roles: when starting something new, make it explicit who will be working on it. Often this will be more than one person, often from different teams. For example, customer-facing product changes should have at least a product owner and an engineering owner. This makes it easy to tell if too many concurrent projects are dependent on a single person, which is a recipe for blockage.

• Define team roles: Most recurring tasks should fall within a defined role. An owner of a technical service is an example of a role. An on-call engineer is an example of a time-limited role. There are many others which will depend on the team and its scope.

• Define job roles: Have a conversation with your teammates and manager about what the scope of your job is, which responsibilities are shared with other members of the team and which are yours alone.

Obstacles that stand in our way

• Getting hung up on titles as ego gratification. Roles are tools, not masters.

• Fear that a role limits your options, locks you into doing one thing forever. Roles can be as flexible as we want them to be.

Confused yet? The popular browser extension AdBlock for Chrome/Chromium and Safari has launched a new crowdsourced campaign - to buy up internet advertising space inviting people to download AdBlock, so they can block ads. Additionally, if they raise enough, they'd like to get additional space for a billboard at Times Square, a full page spread in the New York Times, and even a TV slot at the Superbowl (although the target for this is $4.2M, so I'm not entirely confident they'll make it.) However, this does raise an interesting question, which has been under debate for some time. How do you make money on the internet without adverts? Wikipedia has a few interesting suggestions. Firstly, affiliate programmes. These mean that any products or services that are linked or searched via the original site earn a little bit of commission for that site. Founded in the mid-90s, and most famously used by Amazon, it's a possible method for getting money. However, it should be deployed with careful thought. If a blogger or reviewer is recommending a product, is it simply because they are being paid to do so, or is it a genuine endorsement? The BBC ban this practice, and the Society of Professional Journalists also has strong guidelines on taking gifts. Interestingly, a few search engines now offer this to browsers for them sending their default search traffic to them. Secondly, you could engage in (what I consider to be) the shady practice of data monetisation. People visit a website, and they leave data about their visit. By visiting here, you've told me that you're probably interested in technology issues, what country and city you're from, and where you came from. Other programmes and sites will also be able to wrap up what other sites you've visited, and can build up quite a profile of your browsing habits. Add together a rather unscrupulous vendor, and they can also know your contact details. This metadata can then be sold on. Thirdly, you can essentially ask your readers for money. Donations, via a micropayment link (which I use) or larger donation requests. Another method is to have a membership system, where content is locked up behind a paywall. This does, however, risk making your content less valuable as people will just head elsewhere for news and opinion. To counter this, a couple of sites (LWN, Slashdot) simply delay the latest news for a little while. Alternatively, you can just ask like the fantastic C G P Grey has done. Interestingly, he also has a great video on why advertising can be a bad thing for content creators. Lastly, you can sell things. This could be your authorship services on another site as a guest post, writing a book, offering consultancy services, or simple merchandise. In summary though, which is more honest? An advert may well bend your view on what sort of content is being created, but is at least a lot more open than potentially hiding real motives behind affiliate links, and a lot more open than selling data. Asking for money is fairly straightforward, though could easily backfire if it's viewed as begging. Google's advertising income totalled $43.7 billion in 2012 - so it's quite clear that advertising is big business. Websites cost money to run, can the web survive in an ad-free environment? If so, it certainly won't be the same web as we know today.

TL;DR: db.collection.update( _id: 'x' , $pull: embedded_doc: _id: 'y' ) Queries are wrote for the mongo shell Dot based access If you have been using MongoDB for some time, then there are big chances that you will know the "dot-based access" on the find queries. This is the easier way to write those queries and will probably want to keep using that. Like:

db.collection.find( _id: 'x', 'embedded_doc._id': 'y' )

The thing is that does not work with everything, like: Arrays of embedded documents! Little example schema Let's suppose the following little schema (written for Mongoose): A = new Schema( w: String, x: String ); Document = new Schema( embedded_doc: [A] ); The collection of Document is called documents. Updating arrays If the arrays are not embedded documents I can actually just use a regular update like with any other field combined with a $set and change all the array like:

db.documents.update( _id: 'x',  $set:  array: [
   a: 1 , 
   a: 2 , 
   a: 3 
] )

And using conveniently $pull and $addToSet. But when it comes to embedded documents the history is very different. You can't make a simple $set because it will give you an error related to ModId which you can't change. So you really need to use $push, $addToSet, $pull and it's friends. So all those you can do it as in the manual, but what if you want to make a $pull over an array of objects and w and x are not unique? Well, that's why you probably made them an embedded document! Then you have the _id and you try to use the dot notation like:

db.documents.update( _id: 'x' ,  $pull:  "embedded_doc._id": 'y' )

And it looks good, except that it wont work. You have to use the complete syntax like:

db.documents.update( _id: 'x' ,  $pull:  embedded_doc:  _id: 'y' )

Had an afternoon of steak and science fiction. Elysium is only so-so, but look what we found in a bookstore that was half religious materials and half SF, local books, and carefully hidden romance novels:

Star Wars, Star Trek and the 21st Century Christians

Best part was at the end, when I finally found one of the local asian markets Tomoko tells us about when she casually pulls out the good stuff at family gatherings. I will be back for whole ducks, fresh fish, squid, 50 lb bags of rice, tamarind paste, fresh ginger that has not sat on the shelf for 2 months because only I buy it, etc. Only an hour from home in the woods! Between the garlic scapes, bean sprouts, enoki mushrooms, etc that I got for $10 today and this week's CSA surprise of 18 inch snake beans and smoked pork knuckles, I have the epic stir fry potential..

I ve been really quiet on the blogging front the last few months. There s been a lot happening in my life. In short, I moved back to South Africa and started working for another company. I have around 10 blog drafts that I will probably never finish, so I m just declaring post bankruptcy and starting from scratch (who wants read my old notes from Ubuntu VUDS from March anyway?) For what it s worth, I m still alive and doing very well, possibly better than ever. Over the next few months I want to focus my free software time on the Edubuntu project, my Debian ITPs (which I ve neglected so badly they ve been turned into RFPs) and Flashback. Once I ve caught up there s plenty of other projects where I d like to focus some energy as well (LTSP, Debian-Edu, just to name a few). Thanks for reading! More updates coming soon.