Background R relies on package repositories for initial installation of a package via install.packages(). A crucial second step is update.packages(): For all currently installed packages, a list of available updates is constructed or offered for either one-by-one or bulk updates. This keeps the local packages in sync with upstream, and provides for a very convenient way to obtain new features, bug fixes and other improvements. So by installing from a repository, we automatically have the ability to track the repository for updates.

Enter drat Fairly recently, the drat package was added to the R ecosystem. It makes both aspects of package distribution easy: providing a package (if you are an author) as well as installing it (if you are a user). Now, because drat is at the same time source code (as it is also a package providing the functionality), and a repository (using what drat provides ib features), the "namespace" becomes a little cluttered. But because a key feature of drat is the "one variable" unique identification via the GitHub, I opted to create a drat repository in the name of a new organisation: ghrr. This is a simple acronym for GitHub Hosted R Repository.

Use cases We can outline several use case for packages in ghrr:

• packages not published in a repo by their authors: I already use two like that:
  • fasttime, an impeccably fast parser for ISO datetimes by Simon Urbanek which was however never released into a repo by Simon, and
  • RcppR6, a very nice extension to both R6 (by Winston) and Rcpp, by Rich FitzJohn; similarly never released beyond GitHub;
• packages possibly unsuitable for mainline repos:
  • Rblpapi is a great package by Whit Armstong and John Laing to which I have been contributing quite a bit of late. As it requires a free-to-use but not open source library and headers from Bloomberg, it will never make it to the mainline repository for R, but hosting it in ghrr is perfect as I can easily update several machines at work once I cut a new development release;
  • winsorize is a small package I needed a few weeks ago; it is spun out of robustHD but does not yet contain new code so Andreas and I are content to keep it in this drat for now;
• packages in pre-relase mode:
  • RcppArmadillo where I announced both a release candidate before Armadillo 5.000 came out, as well as the actual RcppArmadillo 0.500.0.0 which is not (yet) on the mainline repository as two affected packages need a small update first. Users, however, can get RcppArmadillo already from the sibling Rcpp drat repo.
  • RcppToml is a new package I am currently working on implementing a toml parser based on cpptoml. It works, but it not quite ready for public announcements yet, and hence perfect for ghrr.

Going forward ghrr is meant to be open. While anybody can open a drat repository, particularly on GitHub, it may be beneficial to somehow group packages. This is however not something that can be planned ex-ante: it may just happen if others who see similar benefits in this can in fact contribute. In that spirit, I strongly encourage pull requests. Early on, I made my commit messages conform to a pattern of package version sha1 repourl to make code provenance of every commit very clear. Ideally, subsequent commits would conform to such a scheme, or replace it with a better one.

Some Resources A few links to learn more about drat and ghrr:

• ghrr repo view (to see both content and commits) and ghrr web page
• drat repo view, drat code page, and drat web page

Comments and questions via email or issue tickets are more than welcome. We hope that others find ghrr to be a useful tool for easy repository management and use via GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

I m a big fan of org-mode (see previous posts), and I ve started maintaining (sic) my professional webpage(s) with it. But I ve also recently tried and publish some more Semantic/Linked Data aware documents too (again, previous posts). Ideally, I think my preferred workflow for publishing articles or documents of some importance, would be to author them in org-mode, and then publish them as HTML5 including RDFa meta-data and annotations. Instead, I ve more frequently been doing conversions of org-mode to LaTeX, in order to submit a printable version, and later-on decided to convert the LaTeX to HTML5+RDFa But one of the issues is how to properly embed the RDF meta-data inside the org-mode documents, so that the syntax is both compact and expressive enough. I doubt there s a universal solution, given that RDF tends to be complex, and graphs may not project easilly along a mainly linear structure of an org-mode document, but anyway, there seems to be possible middle grounds that are practically good enough. I ve tried and implement a solution, which reuses the principles set by John Kitchin in Extending the org-mode link syntax with attributes, i.e. implementing an HTML exporter for a particular custom link type, which will convert the plist-like syntax to some RDFa constructs. Here s a description of the whole solution : http://www-public.telecom-sudparis.eu/~berger_o/test-org-publishing-rdfa.html The nice thing about org-mode, and its litterate programming babel environment, is that it allows to embed the code of the links exporter inside the org document, avoiding to dissociate the converter from the document s source, making it auto-complete. Next step will probably be to author a paper (or convert back a preprint of mines) with org-mode, in order to provide Linked Research meta-data. Stay tuned for more details, and in the meantime, I welcome any improvement to the org/babel/elisp setup. Edit: I ve recorded a webcast to provide a bit more details, available on YouTube : https://youtu.be/OyI3DVqllx4

Erlang lets you write applications supporting zero downtime by switching one live system to another running a different application version converting the application s state on the fly to the new representation. Debian packages however can have only one installed version on a system which prevents using Erlang s hot-upgrade feature easily. Engineers at Yakaz (Jean-S bastien P dron) came up with a nice solution by creating separate directories for each application release and creating .deb packages for managing the transitions. I had to solve the same problem recently and found that the erlsvc Perl application they created needed a few patches to be usable with latest Erlang and other packages and with the changes it worked perfectly. Yakaz was not interested in accepting the patches and developing it further, but let me continue the maintenance. Please find the updated erlsvc application under my GitHub account and feel free to submit patches if you find something to fix in it. I have also packaged erlsvc as an official Debian package and it is waiting in the NEW queue for being accepted. When it enters unstable you will have to make very little effort to make your applications hot-upgradeable on Debian!

About a decade ago, I played in the (now defunct) "Jozef Pauly ensemble", a flute choir connected to the musical academy where I was taught to play the flute. At the time, this ensemble had the habit of goin on summer trips every year; sometimes these trips were large international concert tours (like our 2001 trip to Australia), but that wasn't always the case; there have also been smaller trips, like the 2002 one to the French Ardennes. While there, we went on a day trip to the city of Reims. As a city close to the front in the first world war, it has a museum dedicated to that subject that I remembered going to. But the fondest memory of that day was going to a park where a podium was set up, with a few stacks of fold-up chairs standing nearby. I took one and listened to the music. That was the day when I realized that I kindof like jazz. I had come into contact with Jazz before, but it had always been something to be used as a kind of musical wallpaper; something you put on, but don't consciously listen to. Watching this woman sing, however, was a different kind of experience altogether. I'm still very fond of her rendition of "Besame Mucho". After having listened to the concert for about two hours, they called it quits, but did tell us that there was a record which you could buy. Of course, after having enjoyed the afternoon so much, I couldn't imagine not buying it, so that happened. Fast forward several years, in the move from my apartment above my then-office to my current apartment (just around the corner), the record got put into the wrong box, and when I unpacked things again it got lost; permanently, I thought. Since I also hadn't digitized it yet at the time, I haven't listened to it anymore in quite a while. But that time came to an end today. The record which I thought I'd lost wasn't, it was just in a weird place, and while cleaning yesterday, I found it sitting among a bunch of old stuff that I was going to throw out. Putting on the record today made me realize again how good it really is, and I thought that I might want to see if she was still active, and if she might perhaps have made another album. It was great to find out that not only had she made six more albums since the one I bought, she'd also become a lot more known in the Jazz world (which I must admit I don't really follow all that well), and won a number of awards. At the time, Youn Sun Nah was just a (fairly) recent graduate from a particular Jazz school in Paris. Today, she appears to be so much more...

Note: In this post I mention some problems and ask questions (to myself, like thinking aloud ). The goal is not to get answers to those questions (I suppose that I will find them soon or later in the internet, manuals and so), but to show the kind of problems and questions that arise in my selfhosting adventures, which I suppose are common to other people trying to administer a home server with some web services. Am I an userop? Well I m something in the middle of (GNU/Linux) user and sysadmin: I have studied computer technical engineering but most of my experience has been in helpdesk, providing support for Windows users. I m running Debian in some LAMP boxes at work (without GUI) since 2008 or so, and in my desktops (with GUI) since 2010. I don t code nor package, but I don t mind trying to read code and understand it (or not). I know a bit of C, a bit of Python, of PHP, and enough Perl to open a Perl file and close it after two minutes, understanding that it s great, but too much for me :) I translate software, so I m not scared to clone a repository, edit files, commit or submit a patch. I m not scared of compiling a program (except if it s an Android app: I try to avoid setting up the development environment just to try some translation that I made but I built my Puma before it was the binary available for download or in F-Droid). In conclusion, I feel more like a GNU/Linux power user than a sysadmin . Sometimes just a user or even a newbie (for example, I don t know very well the Unix/Linux folder tree where are the wallpapers stored? Does it depend on the desktop that I use?). Anyway. I won t stop my free software + free networks digital life because I don t know many things. I bought a small server for home last September, and I wanted to try to selfhost some services, for me and for my family. I want to be a home sysadmin or something like that, so I joined the userops mailing list :) Here you have my experiences on selfhosting/being an userop until now. Mail I even didn t try to setup my mail server, because many people say it s a pain (although nice articles were published about how to do it, for example this series in ArsTechnica) and I need a static IP which is 14 /month more to my ISP, and Gandi, the place where I rented my domain name, provides mail, and they use Debian and Roundcube, and sponsor Debian too, so I decided to trust on them. So this is my strategy now, to try to keep mail under my control:

• Trust my domain provider.
• Backup my mail and keep local copies, removing sensible stuff from the server.
• Use and spread the word about GPG encryption.
• Try not to send photos or videos by mail, just send the link to my MediaGoblin instance (see below).

MediaGoblin I ve setup two MediaGoblin instances (yes, two!). I managed to do it in Debian 7 stable (I think NodeJS npm was not needed then), but soon later I upgraded to Jessie so now it s even better. I installed Nginx and PostgreSQL via apt, to use them for both instances (and probably some more software later). One instance is public, and I use a Debian user, a PostgreSQL database, and it s running in http://media.larjona.net
I have requested an SSL cert to Gandi but I still didn t deployed it (lazy LArjona!!). The other instance is private, for family photos. I didn t know very well how much of my existing setup could reuse and how to keep both instances in case of downtimes or attack I know more or less the concept or chroot but I don t know how to deploy it in my machine. So I decided to use another Debian user, another PostgreSQL database, deploy MediaGoblin in a different folder, and create another virtual server in my Nginx to serve it. I managed to setup that virtual server to http-authenticate and to serve content via a different port, and use a self-signed SSL certificate (it s only for family, so it does not matter). I created another (unprivileged) Debian user with a password for the nginx authentication, and gave my family the URL in the form https://mediaprivate.larjona.net:PortNumber and the user and password (mediaprivate is a string, and PortNumber is a number). I think they don t use the instance too much, but at least I upload photos there from time to time and email the link instead of emailing the photos themselves (they don t use GPG either ). Upgrades I upgraded MediaGoblin from 0.7.1 to 0.8.0 successfully, I sent a report about how I did it to the mailing list. First I upgraded the public instance, when I figure out the process, I upgraded the second instance to test my instructions, and then, I sent the report with the instructions to the mailing list. Static site and LimeSurvey: the power of free software (with instructions) I wanted to act as a mirror of floss2013.libresoft.es and surveys.libresoft.es since they suffer a downtime and I participated in that project (not in the sysadmin part, but in the research and content creation). The static site floss2013.libresoft.es offered a zip with the whole website tree (since the website was licensed as AGPL), and I had access to the git repo holding the development copy of the website. So I just cloned the repository and setup another nginx virtual server in my machine, and tuned my DNS zone in Gandi website to serve floss2013.larjona.net from home. 10 minutes setup YAY! #inGitWeTrust #FreeSoftwareFTW :) For surveys.larjona.net I had to install a LimeSurvey instance. I knew how to do it because we use LimeSurvey at work, but at home I had Nginx instead of Apache, and PostgreSQL instead of MySQL. And no PHP I searched about how to install PHP in Nginx (I can use apt-get, nice!) and how to install LimeSurvey with Nginx and PostgreSQL (I had documentation about that, so I followed, and it worked). For making available the data (one survey and its results, so people can login as visitor to query and get statistics), I downloaded the LimeSurvey export dataset that we were providing in the static website, followed the replication instructions (hey, I wrote them!), and they worked #oleole! (And here, dear researchers, gets demonstrated that free software and free culture really empower your research and help spreading your results). Etherpad: not so easy, it seems! I m trying to install Etherpad-Lite, but I m suffering a bit. I think I did everything ok according to some guides but I get Bad Gateway and these kind of errors when trying to browse with Lynx in the host:

[error] 3615#0: *24 upstream timed out (110: Connection timed out) 
while reading response header from upstream, 
client: 127.0.0.1, 
server: pad.larjona.net, 
request: "GET / HTTP/1.0", 
upstream: "http://127.0.0.1:9001/", 
host: "pad.larjona.net"
2015/04/17 20:52:56 [error] 3615#0: *24 connect() failed 
(111: Connection refused) while connecting to upstream, 
client: 127.0.0.1, 
server: pad.larjona.net, 
request: "GET / HTTP/1.0", 
upstream: "http://[::1]:9001/", 
host: "pad.larjona.net"

I m not sure if I need to open some port in iptables, my router, or change my nginx configuration because the guides assume you re only serving one website in the port 80 (and I have several of them, now ), or what I ve spent three chunks of time (maybe ~2h each?) on this, in different days, and couldn t figure it out, so I decided to round-robin in my TODO list. Userops thoughts Debian brings peace of mind (for me) On one side, maintaining a Debian box it s quite easy, and the more software that it s packaged, the less time that I spend installing or upgrading. I like being in stable, I m in Jessie now (I migrated when it was frozen), but I ll stay in stable as much as I can. I like that I can use the software that I installed via apt-get for several services (nginx, PostgreSQL ). About the software that is not packaged (MediaGoblin, LimeSurvey, EtherPad, maybe others later), I wonder how dependencies and updates are handled. And maybe (probably) I have installed some components several times, one for each service (this sounds like a Windows box #grr). For example MediaGoblin uses PyPump. PyPump 0.5 is packaged in Debian Jessie. MediaGoblin uses PyPump 0.7+. What if PyPump 0.7+ gets, let s say, into Jessie-backports? Can I benefit from that? I know that MediaGoblin upgrade instructions includes upgrading the dependencies, but what about some security patch in one dependency? Should I upgrade the pip modules periodically? How to know if some upgrade is recommended because patches a vulnerability, or it s just new features (and maybe breaking my setup)? This kind of things are the peace of mind that Debian packaging brings to me: when some piece of software is packaged, I know maybe I need to care about proper setup and configuration, but later, it s kind-of-easy to maintain (since the Debian maintainers care about the rest). I don t mind about cloning a repo and compiling, I mind about later, or coexistance with other program/services. I trust in the MediaGoblin community and I m an active member (I m not developer, but hang on IRC, follow the mailing list, etc) but for example I don t know anything about the EtherPad project. And I don t feel like joining the community (I m already an active member in Debian, MediaGoblin, F-Droid, Pump.io, translator of LimeSurvey and many other small apps that I use, and in the future will use more services, like OwnCloud, XMPP ), joining the community of each software that I use is becoming not sustainable :s Free software is more than software I follow the userop mailing list, and it s becoming very technical. I mostly understand the problems (which are similar to the problems that I face: how to isolate different services, how to easily-configure them, how to make them installable by average user ) But I don t understand most of the solutions proposed, and I think that probably we need technical solutions, but in the meanwhile, some issues can be addressed not with software, but with other means: good documentation, community support, translations, beta-testers This is my conclusion until now. When a project is well documented, I think I can find my way to selfhost no matter if the software is packaged (or contained ) or not. MediaGoblin, and LimeSurvey, are well documented, and the user support channels are very responsive. I find lots of instructions that assume that you will use a whole machine for their service (and not for other things). And lots of documentation for the LAMP stack, but not for Nginx + PostgreSQL and Node instead of PHP So, for each particularity of my setup, I search the internet and try to pick good sources to help me do what I wanted to do. I m kind of privileged Some elements, not software related, to take into account as pre-requisites for succeed selfhosting services:

• I knew what to search.
• I knew which sites to visit from the results (arch wiki, debian wiki, stack overflow, etc: some of them were not the Top1 in the results).
• I had time to read several sources and make my mind about what to do and how.
• I can read, understand, and write in English.
• I have no fear about my broken English.
• I have no impostor syndrome.
• I felt welcome in the FLOSS communities where I hanged out.

These aspects are not present in a lot of people. If I look around to the computer users that I know (mostly Windows+Android, some GNU/Linux users, some Mac OSX users, some iOS users), I find that they search things like X does not work or they cannot write a proper search query in English. Or they trust some random person writing a recipe in their blog, without trying first to understand what the recipe does. Other people just say I m not a professional sysadmin, I ll just do what everybody does (aka use Google services or whatever). What if I try and I don t succeed? . Things like that. We may need some technical solutions (and hackers are thinking about that, and working on that). But I feel that we need (more) a huge group of beta-testers, dogfooding people, aventurers that try the half-cooked solutions and provide successful and unsuccessful experiences, to guide the research and make software technologies advance. I m not sure if I am an userop, but I feel part of that vanguard force , I want to be part of the future of free software and free networks. Comments? You can comment about this post on this pump.io thread.
Filed under: My experiences and opinion Tagged: Communities, Contributing to libre software, Debian, Developer motivations, English, free networks, Free Software, Freedom, innovation, MediaGoblin, Moving into free software, Project Management, selfhosting, sysadmin

only one week left until the jessie release. yay! in the last weeks I didn't find many RC bugs that I could fix; still, here's the short list; nice feature: I mostly help others or could build an work done by others.

• #669735 dpkg-www: "dpkg-www: transition towards Apache 2.4"
  sponsor QA upload from Jean-Michel Nirgal Vourg re
• #669777 yocto-reader: "yocto-reader: transition towards Apache 2.4"
  sponsor NMU from Jean-Michel Nirgal Vourg re, upload to DELAYED/2
• #669796 w3c-linkchecker: "w3c-linkchecker: transition towards Apache 2.4"
  cherry-pick periapt's commit from 2012 (pkg-perl)
• #764284 testdisk: "[testdisk] after ntfs-3g upgrade, testdisk cannot be installed (Depends: error)"
  tag + sid to get it out of jessie RC bugs
• #780629 libibverbs1: "libibverbs1: please add Breaks: libopenmpi1.3"
  upload NMU prepared by Andreas Beckmann, adding a Breaks, upload to DELAYED/2
• #780729 pbuilder: "pbuilder must define PATH as in debian-policy (and as used on buildds)"
  downgrade
• #782160 src:chrony: "chrony: Multiple issues: CVE-2015-1821 CVE-2015-1822 CVE-2015-1853"
  sponsor maintainer upload by Joachim Wiedorn

Review: The Long Way to a Small, Angry Planet, by Becky Chambers

Publisher:	CreateSpace
Copyright:	2014
ISBN:	1-5004-5330-7
Format:	Kindle
Pages:	503

The Wayfarer is a tunneling ship: one of the small, unremarked construction ships that help build the wormhole network used for interstellar transport. It's a working ship with a crew of eight (although most people would count seven and not count the AI). They don't all like each other particularly not the algaeist, who is remarkably unlikeable but they're used to each other. It's not a bad life, although a more professional attention to paperwork and procedure might help them land higher-paying jobs. That's where Rosemary Harper comes in. At the start of the book, she's joining the ship as their clerk: nervous, hopeful, uncertain, and not very experienced. But this is a way to get entirely away from her old life and, unbeknownst to the ship she's joining, her real name, identity, and anyone who would know her. Given that introduction, I was expecting this book to be primarily about Rosemary. What is she fleeing? Why did she change her identity? How will that past come to haunt her and the crew that she joined? But that's just the first place that Chambers surprised me. This isn't that book at all. It's something much quieter, more human, more expansive, and more joyful. For one, Chambers doesn't stick with Rosemary as a viewpoint character, either narratively or with the focus of the plot. The book may open with Rosemary and the captain, Ashby, as focal points, but that focus expands to include every member of the crew of the Wayfarer. We see each through others' eyes first, and then usually through their own, either in dialogue or directly. This is a true ensemble cast. Normally, for me, that's a drawback: large viewpoint casts tend to be either jarring or too sprawling, mixing people I want to read about with people I don't particularly care about. But Chambers avoids that almost entirely. I was occasionally a touch disappointed when the narrative focus shifted, but then I found myself engrossed in the backstory, hopes, and dreams of the next crew member, and the complex ways they interweave. Rosemary isn't the center of this story, but only because there's no single center. It's very hard to capture in a review what makes this book so special. The closest that I can come is that I like these people. They're individual, quirky, human (even the aliens; this is from more the Star Trek tradition of alien worldbuilding), complicated, and interesting, and it's very easy to care about them. Even characters I never expected to like. The Long Way to a Small, Angry Planet does have a plot, but it's not a fast-moving or completely coherent one. The ship tends to wander, even when the mission that gives rise to the title turns up. And there are a lot of coincidences here, which may bother you if you're reading for plot. At multiple points, the ship ends up in exactly the right place to trigger some revelation about the backstory of one of the crew members, even if the coincidence strains credulity. Similar to the algae-driven fuel system, some things one just has to shrug about and move past. On other fronts, though, I found The Long Way to be refreshingly willing to take a hard look at SF assumptions. This is not the typical space opera: humans are a relatively minor species in this galaxy, one that made rather a mess of their planet and are now refugees. They are treated with sympathy or pity; they're not somehow more flexible, adaptable, or interesting than the rest of the galaxy. More fascinatingly to me, humans are mostly pacifists, a cultural reaction to the dire path through history that brought them to their current exile. This is set against a backdrop of a vibrant variety of alien species, several of whom are present onboard the Wayfinder. The history and background of the other species are not, sadly, as well fleshed out as the humans, but each with at least a few twists that add interest to the story. But the true magic of this book, the thing that it has in overwhelming abundance, is heart. Not everyone in this book is a good person, but most of them are trying. I've rarely read a book full of so much empathy and willingness to reach out to others with open hands. And, even better, they're all nice in different ways. They bring their own unique personalities and approaches to their relationships, particularly the complex web of relationships that connects the crew. When bad things happen, and, despite the overall light tone, a few very bad things happen, the crew rallies like friends, or like chosen family. I have to say it again: I like these people. Usually, that's not a good sign for a book, since wholly likeable people don't generate enough drama. But this is one of the better-executed "protagonist versus nature" plots I've read. It successfully casts the difficulties of making a living at a hard and lonely and political job as the "nature" that provides the conflict. This is a rather unusual book. It's probably best classified as space opera, but it doesn't fit the normal pattern of space opera and it doesn't have enough drama. It's not a book about changing the universe; at the end of the book, the universe is in pretty much the same shape as we found it. It's not even about the character introduced in the first pages, or really that much about her dilemma. And it's certainly not a book about winning a cunning victory against your enemies. What it is, rather, is a book about friendships, about chosen families and how they form, about being on someone else's side, about banding together while still being yourself. It's about people making a living in a hard universe, together. It's full of heart, and I loved it. I'm unsurprised that The Long Way to a Small, Angry Planet had to be self-published via a Kickstarter campaign to find its audience. I'm also unsurprised that, once it got out there, it proved very popular and has now been picked up by a regular publisher. It's that sort of book. I believe it's currently out of print, at least in the US, as its new publisher spins up that process, but it should be back in print by late 2015. When that happens, I recommend it to your attention. It was the most emotionally satisfying book I've read so far this year. Rating: 9 out of 10

Sometimes one stumbles upon stuff that touches one deeply. Granted, the topic of the first video from the artist I want to present you now did touch me naturally. But it made me take a closer look. This is about HollySiz. Yes, yet another French singer, but fortunately (for me) she sings mostly in English. :) So here are the songs:

• The Light: At first I wasn't even aware it's a music video. And the story is strong. I'm uncertain on the story of Nils Pickert did inspire the video, but it's lovely to see people getting it right. The parents job is to support their kid in finding their own identity instead of defining it for them.
• Better Than Yesterday: In the light of The Light everything else looks antique. So what's better as a video that actually does look antique. ;)
• Tricky Game (feat. Sianna): I somehow like this version of the song better because it contains rap. But that might be just me. A catchy beat anyway.

Like always, enjoy! And take good care of your kids if you happen to have some.

/music permanent link Comments: 0 Flattr this

It was a surprise to me to learn that project to create a complete computer system for schools I've involved in, Debian Edu / Skolelinux, was being used in India. But apparently it is, and I managed to get an interview with one of the friends of the project there, Shirish Agarwal. Who are you, and how do you spend your days? My name is Shirish Agarwal. Based out of the educational and historical city of Pune, from the western state of Maharashtra, India. My bread comes from giving training, giving policy tips, installations on free software to mom and pop shops in different fields from Desktop publishing to retail shops as well as work with few software start-ups as well. How did you get in contact with the Skolelinux / Debian Edu project? It started innocently enough. I have been using Debian for a few years and in one local minidebconf / debutsav I was asked if there was anything for schools or education. I had worked / played with free educational softwares such as Gcompris and Stellarium for my many nieces and nephews so researched and found Debian Edu or Skolelinux as it was known then. Since then I have started using the various education meta-packages provided by the project. What do you see as the advantages of Skolelinux / Debian Edu? It's closest I have seen where a package full of educational software are packed, which are free and open (both literally and figuratively). Even if I take the simplest software which is gcompris, the number of activities therein are amazing. Another one of the softwares that I have liked for a long time is stellarium. Even pysycache is cool except for couple of issues I encountered #781841 and #781842. I prefer software installed on the system over web based solutions, as a web site can disappear any time but the software on disk has the possibility of a larger life span. Of course with both it's more a question if it has enough users who make it fun or sustainable or both for the developer per-se. What do you see as the disadvantages of Skolelinux / Debian Edu? I do see that the Debian Edu team seems to be short-handed and I think more efforts should be made to make it popular and ask and take help from people and the larger community wherever possible. I don't see any disadvantage to use Skolelinux apart from the fact that most apps. are generic which is good or bad how you see it. However, saying that I do acknowledge the fact that the canvas is pretty big and there are lot of interesting ideas that could be done but for reasons not known not done or if done I don't know about them. Let me share some of the ideas (these are more upstream based but still) I have had for a long time : 1. Classical maths question of two trains in opposing directions each running @x kmph/mph at y distance, when they will meet and how far would each travel and similar questions like these. The computer is a fantastic system where questions like these can be drawn, animated and the methodology and answers teased out in interactive manner. While sites such as the Ask Dr. Math FAQ on The Two Trains problem (as an example or point of inspiration) can be used there is lot more that can be done. I dunno if there is a free software which does something like this. The idea being a blend of objects + animation + interaction which does this. The whole interaction could be gamified with points or sounds or colourful celebration whenever the user gets even part of the question or/and methodology right. That would help reinforce good behaviour. This understanding could be used to share/showcase everything from how the first wheel came to be, to evolution to how astronomy started, psychics and everything in-between. One specific idea in the train part was having the Linux mascot on one train and the BSD or GNU mascot on the other train and they meeting somewhere in-between. Characters from blender movies could also be used. 2. Loads of crossword-puzzles with reference to subjects: We have enormous data sets in Wikipedia and Wikitionary. I don't think it should be a big job to design crossword puzzles. Using categories and sub-categories it should be doable to have Q&A single word answers from the existing data-sets. What would make it easy or hard could be the length of the word + existence of many or few vowels depending on the user's input. 3. Jigsaw puzzles - We already have a great software called palapeli with number of slicers making it pretty interesting. What needs to be done is to download large number of public domain and copyleft images, tease and use IPTC tags to categorise them into nature, history etc. and let it loose. This could turn to be really huge collection of images. One source could be taken from commons.wikimedia.org, others could be huge collection of royalty-free stock photos. Potential is immense. Apart from this, free software suffers in two directions, we lag both in development (of using new features per-se) and maintenance a lot. This is more so in educational software as these applications need to be timely and the opportunity cost of missing deadlines is immense. If we are able to solve issues of funding for development and maintenance of such software I don't see any big difficulties. I know of few start-ups in and around India who would love to develop and maintain such software if funding issues could be solved. Which free software do you use daily? That would be huge list. Some of the softwares are obviously apt, aptitude, debdelta, leafpad, the shell of course (zsh nowadays), quassel for IRC. In games I use shisen-sho while card-games are evenly between kpat and Aiselriot. In desktops it's a tie between gnome-flashback and mate. Which strategy do you believe is the right one to use to get schools to use free software? I think it should first start with using specific FOSS apps. in whatever environment they are. If it's MS-Windows or Mac so be it. Once they are habitual with the apps. and there is buy-in from the school management then it could be installed anywhere. Most of the people now understand the concept of a repository because of the various online stores so it isn't hard to convince on that front. What is harder is having enough people with technical skills and passion to service them. If you get buy-in from one or two teachers then ideas like above could also be asked to be done as a project as well. I think where we fall short more than anything is in marketing. For instance, Debian has this whole range of fonts in its archive but there isn't even a page where all those different fonts in the La Ipsum format could be tried out for newcomers. One of the issues faced constantly in installations is with updates and upgrades. People have this myth that each update and upgrade means the user interface will / has to change. I have seen this innumerable times. That perhaps is one of the reasons which browsers like Iceweasel / Firefox change user interfaces so much, not because it might be needed or be functional but because people believe that changed user interfaces are better. This, can easily be pointed with the user interfaces changed with almost every MS-Windows and Mac OS releases. The problems with Debian Edu for deployment are many. The biggest is the huge gap between what is taught in schools and what Debian Edu is aimed at. Me and my friends did teach on week-ends in a government school for around 2 years, and gathered some experience there. Some of the things we learnt/discovered there was :

1. Most of the teachers are very territorial about their subjects and they do not want you to teach anything out of the portion/syllabus given.
2. They want any activity on the system in accordance to whatever is in the syllabus.
3. There are huge barriers both with the English language and at times with objects or whatever. An example, let's say in gcompris you have objects falling down and you have to name them and let's say the falling object is a hat or a fedora hat, this would not be as recognizable as say a Puneri Pagdi so there is need to inject local objects, words wherever possible. Especially for word-games there are so many hindi words which have become part of english vocabulary (for instance in parley), those could be made into a hinglish collection or something but that is something for upstream to do.

Like each month, here comes a report about the work of paid contributors to Debian LTS. Individual reports In February, 61 work hours have been equally split among 4 paid contributors. Their reports are available:

• Ben Hutchings did only 11.25h.
• Holger Levsen did only 4.5h during which he worked on the security tracker JSON output (see #761859) and on DLA-169-1 (fixing 2 CVE on axis).
• Rapha l Hertzog
• Thorsten Alteholz

The remaining hours of Ben and Holger have been redispatched to other contributors for April (during which Mike Gabriel joins the set of paid contributors). BTW, if you want to join the team of paid contributors, read this and apply! Evolution of the situation April has seen no change in terms of sponsored hours but we have two new sponsors in the pipe and May should hopefully have a few more sponsored hours. For the need of a LTS presentation I gave during the Mini-DebConf Lyon I prepared a small graph showing the evolution of the hours sponsored through Freexian:
The growth is rather slow and it will take years to reach our goal of funding the equivalent a full time position (176 hours per month). Even the intermediary goal of funding the equivalent of a half-time position (88h/month) is more than 6 months away given the current growth rate. But the perspective of Wheezy-LTS should help us to convince more organizations and hopefully we will reach that goal sooner. If you want to sponsor the project, check out this page. In terms of security updates waiting to be handled, the situation looks similar to last month: the dla-needed.txt file lists 40 packages awaiting an update (exactly like last month), the list of open vulnerabilities in Squeeze shows about 56 affected packages in total (2 less than last month). Thanks to our sponsors The new sponsors of the month are in bold (none this month).

• Gold sponsors:
• The Positive Internet
• Silver sponsors:
• AD&D David Ayers IntarS Austria
• Blablacar
• Domeneshop AS
• Evolix
• Gandi SAS
• Trollweb Solutions
• Universit Lille 3
• Bronze sponsors:
• Bitfolk LTD
• Daevel SARL
• FOSSter
• Freeside Internet Service
• Gree, Inc.
• Greenbone Networks GmbH
• Intevation GmbH
• Linuxhotel GmbH
• Megaspace Internet Services GmbH
• MyTux
• NUMLOG
• Nantes M tropole
• Offensive Security
• Seznam.cz, a.s.
• WinGo AG

No comment Liked this article? Click here. My blog is Flattr-enabled.

After using haproxy at work for some time I realized that it can be configured for a lot of things, for example: it knows about SNI (on ssl is the method we use to know what host the client is trying to reach so that we know what certificate to present and thus we can multiplex several virtual hosts on the same ssl IP:port) and it also knows how to make transparent proxy connections (the connections go through haproxy but the ending server will think they are arriving directly from the client, as it will see the client's IP as the source IP of the packages).With this two little features, which are available on haproxy 1.5 (Jessie's version has them all), I thought I could give it a try to substitute sslh with haproxy giving me a lot of possibilities that sslh cannot do.Having this in mind I thought I could multiplex several ssl services, not only https but also openvpn or similar, on the 443 port and also allow this services to arrive transparently to the final server. Thus what I wanted was not to mimic sslh (which can be done with haproxy) but to get the semantic I needed, which is similar to sslh but with more power and with a little different behaviour, cause I liked it that way.There is however one caveat that I don't like about this setup and it is that to achieve the transparency one has to run haproxy as root, which is not really something one likes :-( so, having transparency is great, but we'll be taking some risks here which I personally don't like, to me it isn't worth it.Anyway, here is the setup, it basically consists of a setup on haproxy but if we want transparency we'll have to add to it a routing and iptables setup, I'll describe here the whole setupHere is what you need to define on /etc/haproxy/haproxy.cfg:frontend ft_ssl bind 192.168.0.1:443 mode tcp option tcplog tcp-request inspect-delay 5s tcp-request content accept if req_ssl_hello_type 1 acl sslvpn req_ssl_sni -i vpn.example.net use_backend bk_sslvpn if sslvpn use_backend bk_web if req_ssl_sni -m found default_backend bk_ssh backend bk_sslvpn mode tcp source 0.0.0.0 usesrc clientip server srvvpn vpnserver:1194 backend bk_web mode tcp source 0.0.0.0 usesrc clientip server srvhttps webserver:443 backend bk_ssh mode tcp source 0.0.0.0 usesrc clientip server srvssh sshserver:22 An example of a transparent setup can be found here but lacks some details, for example, if you need to redirect the traffic to the local haproxy you'll want to use the xt_TPROXY, there is a better doc for that at squid's wiki. Anyway, if you are playing just with your own machine, like we typically do with sslh, you won't need the TPROXY power, as packets will come straight to your 443, so haproxy will be able to get the without any problem. The problem will come if you are using transparency (source 0.0.0.0 usesrc clientip) because then packets coming out of haproxy will be carrying the ip of the real client, and thus the answers of the backend will go to that client (but with different ports and other tcp data), so it will not work. We'll have to get those packets back to haproxy, for that what we'll do is mark the packages with iptables and then route them to the loopback interface using advanced routing. This is where all the examples will tell you to use iptables' mangle table with rules marking on PREROUTING but that won't work out if you are having all the setup (frontend and backends) in just one box, instead you'll have to write those rules to work on the OUTPUT chain of the mangle table, having something like this:*mangle :PREROUTING ACCEPT :INPUT ACCEPT :FORWARD ACCEPT :OUTPUT ACCEPT :POSTROUTING ACCEPT :DIVERT - -A OUTPUT -s public_ip -p tcp --sport 22 -o public_iface -j DIVERT -A OUTPUT -s public_ip -p tcp --sport 443 -o public_iface -j DIVERT -A OUTPUT -s public_ip -p tcp --sport 1194 -o public_iface -j DIVERT -A DIVERT -j MARK --set-mark 1 -A DIVERT -j ACCEPT COMMIT Take that just as an example, better suggestions on how to know what traffic to send to DIVERT are welcome. The point here is that if you are sending the service to some other box you can do it on PREROUTIING, but if you are sending the service to the very same box of haproxy you'll have to mark the packages on the OUTPUT chain.Once we have the packets marked we just need to route them, something like this will work out perfectly:ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 And that's all for this crazy setup. Of course, if, like me, you don't like the root implication of the transparent setup, you can remove the "source 0.0.0.0 usesrc clientip" lines on the backends and forget about transparency (connections to the backend will come from your local IP), but you'll be able to run haproxy with dropped privileges and you'll just need the plain haproxy.cfg setup and not the weird iptables and advanced routing setup.Hope you like the article, btw, I'd like to point out the main difference of this setup vs sslh, it is that I'm only sending the packages to the ssl providers if the client is sending SNI info, otherwise I'm sending them to the ssh server, while sslh will send ssl clients without SNI also to the ssl provider. If your setup mimics sslh and you want to comment on it, feel free to do it.

Review: Zero Sum Game, by S.L. Huang

Series:	Russell's Attic #1
Publisher:	S.L. Huang
Copyright:	2014
ISBN:	0-9960700-1-X
Format:	Kindle
Pages:	326

Cas Russell finds things for people, often involving some strategic violence. She belongs to that genre of action novel protagonists who have a rough code of ethics, but who don't exactly follow the law. Her friend Rio is much worse: a functioning sociopath with his own code of ethics, the derivation of which is a key plot point. When the book opens, Cas is on a mission to find a person and rescue them from a drug cartel not her normal mission, but her contact said she was referred by Rio. Oh, and Rio is currently hitting her in the face. This setup matches any number of present-day thrillers. The SFnal twist is that Cas is very good at numbers, in a completely unrealistic action hero way. (I found it unsurprising that Huang was inspired by the superhero genre; that's the right model to have in mind when reading about Cas.) She can calculate where bullets are going to go, knows just the right angle and velocity with which to throw things (and, even more notably, can get her body to do that), and, in one particularly memorable scene, sets up an eavesdropping sound concentrator by changing the angles of available random surfaces in the neighborhood, like trash cans. This ability is not without drawbacks. When she's not in a middle of a job, with something to focus on, she usually ends up drinking herself into a stupor to get her brain to stop working. But it's an extremely useful ability that requires the villains of this book go to great efforts to try to kill her. The plot starts out as fairly typical thriller material, involving threats and dire consequences to those Cas loves (or at least likes a lot) and an unfolding sense that this retrieval of a kidnapped woman is the tip of a very deep iceberg. The expected counterpart, a private investigator with a less casual attitude towards killing than Cas, shows up early on. (Rio does not play that role in the story. His role is much more complex.) But the superhero inspirations show up in the villains as well, in a twist that many on-line summaries spoil, but which I will leave unmentioned. Mostly, Zero Sum Game is a fast-moving story with lots of violence, lots of guns, shadowy conspiracies, and a hypercompetent protagonist. (Female, refreshingly, particularly since she doesn't fall in love with any of the other characters in the book.) It's a recipe for enjoyable brain candy, and I think that's the best attitude to bring to it. However, a couple of things set it apart for me. First, Cas spends quite a bit of time really thinking about her life and questioning her decisions, rather than just blithely enjoying her world of stress and violence. There's more introspection here than in the typical thriller plot, but she stops short of wallowing in angst and stays decisive. I liked that balance: a bit of inner discomfort, and a few hard ethical decisions, but not to the point of paralyzing her. Second, her relationship with Rio is something special. Rio himself is a character type that I've seen before in books like this, but I don't think I've seen the dynamic with a character like that handled this well before. I particularly liked that the focus of the book stayed on Cas, not on Rio, and the reader was encouraged to see that relationship as a reflection on Cas and her sense of internal ethics. Seeing Rio through Cas's eyes, and then seeing other characters react to him and react to their relationship, touched some chords that I really enjoyed reading. Unfortunately, the villains weren't as successful, at least for me. Partly this is a personal quirk: the nature of the threat posed (not revealed for about half the book) is a kind that I dislike reading about. It makes my skin crawl in a way that I don't enjoy. But, even putting that aside, the story ends on a very odd and disturbing anti-climax. It's clearly the first book of an ongoing series, and I hope later books will salvage this. (I certainly liked it well enough to read on.) But the ending left me unsettled and rather irritated at the author. Huang plays fair, and the ending is consistent with what we know by the end of the book, but I read this sort of action-thriller story for catharsis and the glory of competent people doing what they do well. I got deeply engrossed in this book and had a hard time putting it down. Both Cas and Rio are great characters, as are most of the supporting cast. I wish the ending wasn't quite as much of a letdown so that I could recommend it more strongly. But it's still a fun superhero thriller. If you're looking for something with unrealistic superpowers, a large helping of competence, and a high body count, this may be worth picking up. (And no, I don't know what the series title means. I know what the series title refers to, but I haven't yet figured out what connection it or the Axiom of Choice has to the plot.) Followed by Half Life. Rating: 7 out of 10

Recently I was in Seoul in the middle of three weeks of travel and my laptop died on me. Since I had some work that needed doing fairly urgently I took myself over to Yongsan Electronics Market and got myself a cheap replacement to tide myself over. What I ended up with was an Acer Aspire E11. There s a bunch of different models all with very similar plastics, I got one which has a N2940 SoC, 2G of RAM (upgraded to 4G in store), a 500G hard disk and no fans for just over 200000 Korean Won, or about $200. As you d expect at that price it s got shortcomings but overall I ve been extremely happy with it, it s worth looking at if you need something cheap. The keyboard in particular is probably the nicest I ve used 0n a laptop in a long time with a good, definite but not excessive click feel as you press. Battery life is about 5 hours as advertised which is not wonderful but basically fine for me most of the time, and while not exactly Retina it s clear with good viewing angles and generally pleasant to look at. Everything is plastic but feels very solid and robust, better than a lot of more expensive devices I ve used, and there s not much bezel around the screen which means it s the first laptop I ve had which has been comfortable to use in a standard economy seat on a plane. The biggest drawback is performance it s a little slow opening applications sometimes and kernel builds crawl with an x86 allmodconfig taking about one and three quarter hours. For e-mail and web browsing there s no problem at all, I did have to move from offlineimap to mbsync to get my mail to sync in a reasonable time but that s more to do with the performance of offlineimap than that of the system. Overall in use it feels like the Dell I was using from about 2008-2011 or so, comfortable in use outside of builds, and I do appreciate having a system with no fans. There were a couple of small tricks getting Debian installed this is the first system I ve seen with secure boot enabled by default which took me a few moments to work out (but is really good to see). Once that was disabled the install was smooth other than being bitten by Debian bug#778810 which meant I needed a manual fixup to actually get it to boot from the disk. It s also got a Broadcom WiFi module which means it doesn t work at all with mainline but it looked like that was on a standard mini PCI Express module so easily replaceable (I happened to have a USB dongle handy so haven t bothered) and the wired ethernet just worked. Like I say I ve been very happy with it, there s a bunch of other models with different specs for everything except the case (some touchscreen, some with small 32G eMMC drives) as well. Were it not for my need to do kernel builds I d probably be keeping it as my primary laptop.

The mini-Debconf Lyon 2015, in addition to being a great meeting to meet both friendly and new faces, has been the occasion for me to update and enrich the GNOME for system administrators course.

For those who couldn t be here, as well as those who were here and disappointed to see me cut through the last third for timing reasons, here are the slides:

This post is almost a carbon copy of the Kallithea 0.2 release notes. Kallithea project have just released Kallithea 0.2. Kallithea is a Python-based GPLv3 source code management software for web-based hosting of Mercurial and Git repositories. This release brings many changes since 0.1. Notably, pull requests system have been improved, making contributing changes more robust. The visual appearance has also been refined: modern font-based symbolic icons from FontAwesome and GitHub Octicons have replaced the previously used bitmap icons, and revision graphs are now drawn with HiDPI display support. Kallithea now supports Mercurial 3.3 and Dulwich 0.9.9. Several fixes in the database code boosted performance significantly. We have also updated our Javascript libraries: jQuery, CodeMirror and Mergely. Javascript and CSS code have been cleaned up, with less and less code depending on Yahoo UI library. Since 0.1 we have discovered two security issues, so all users are strongly recommended to upgrade. For more details on these issues, please see our Security Notices page For more information, see https://kallithea-scm.org/ or http://docs.kallithea-scm.org/. The summary of the changes since 0.1 release is below. Bug fixes:

• forms: add CSRF protection to all forms CVE-2015-0276
• api: don t send internal data unless asked for it CVE-2015-0260
• middleware: fix force_tls typo in force_https backward compatibility code (Issue #44)
• rebranddb: update user extern_type and _name to internal instead of kallithea (Issue #38)
• git: do not fail if git is not installed and has been removed from __init__.py backends
• git: fix 'name' must be bytestring, not unicode error on browsing changesets
• git: close SubprocessIOChunker inputstream used for git (Issue #32)
• git: introduce hack for handling git failure on --depth cloning (Issue #33)
• git: fix version detection with unexpected version string (Issue #71)
• git: preserve line endings when calling out to git
• hg: fix clone from svn+http urls using hg-svn (Issue #72)
• migrate: add missing import (Issue #29)
• pullrequests: handle pull requests to empty repos (Issue #27)
• pullrequests: when creating PRs, fix handling of new ajax requests while other ajax requests are pending
• pullrequests: fix updates for PRs between different repositories
• pullrequests: preserve query parameters in pull request overview paging links
• compare: workaround unexpected Mercurial behaviour when finding ancestor of null rev
• javascript: remove trailing comma to please IE8 (Issue #39)
• download: fix for zip file downloads not being valid zips (Issue #35)
• helpers: fix crash on new users without email (Issue #28)
• middleware: change middleware ordering so we don t buffer hgweb output in redirect
• bin: give kallithea_config.py #!/usr/bin/env python so it can be executed directly
• bin: reintroduce cleanup-repos paster command
• bin: fix update_repoinfo command
• urlify: don t include trailing punctuation in markup
• urlify: markup of hashes in ()
• styling: don t loop on trying to load kallithea-logo.png after kallithea-logo.svg failed
• error: don t crash on /error/document urls
• error: don t crash on response without status
• admin: prevent deletion of users that are owners of a repository/user group (Issue #64)
• admin: make settings hook name field smaller so it is less likely to overlap with the value field
• admin: when scanning for repos to add, ignore removed groups
• auth_crowd: fix Admin Group Membership when using the Atlassian Crowd plugin
• auth: enable selecting PAM authentication module
• auth_ldap: fix user automatically activated with LDAP authentication (Issue #78)
• db: fix beaker cache key for user groups don t collide with users
• db: fix handling of (invalid) unicode email addresses
• db: don t use sql in on empty sets in compare avoid warning of bad performance
• db: to the extent it makes sense to have a one byte size limit, make it 255 instead of 256 to please MySQL (Issue #96)
• files: use current revision as default for show at (Issue #31)
• files: fix HTML injection via file names
• files: use forward slash as path separator for consistency
• files: don t use HTML encoding where not needed (Issue #74).
• feed: urlify and escape the commit description
• diff: fix diff of renamed files with whitespace in their names
• diff: don t split lines on bare CR as python splitlines do
• mergely: update Mergely to v3.3.9 (Issue #83)
• jquery: upgrade to 1.11.1
• codemirror: cleanup of integration and update to version 4.7
• setup: constrain dulwich to version 0.9.9, fixes CVE-2015-0838
• gists: add missing formencode.htmlfill import
• ini: Make celeryd.log.level default values uppercase lowercase might not be recognized and cause failure
• notification: handle unicode translations (Issue #95)
• repos: fix redirect after repo creation (and user/group add focus) (Issue #98)
• changeset: don t crash on malformed whitespace parameter return 400 Bad Request
• stats: fix display when no data ready yet
• stats: fix show more link

Features and improvements:

• ui: use font based icons FontAwesome and Git Hub Octicons from fontello
• ui: HiDPI / Retina support in graphs and other places
• paster: add install-iis command to automate IIS handler generation and update documentation
• email templates: send text/plain part as well
• docs: lots of improvements of structure and content and language
• translations: updated using WebLate
• admin: show links to added repos after repo scan
• admin: show links to created users and groups
• admin: when scanning for repos without remove , report the repos that could/would be removed
• setup: support Mercurial 3.3
• db: drop most lazy joins significant performance improvement
• db: add missing indices significant performance improvement
• db: don t commit session if cache invalidation didn t change anything improves performance
• comment: show full username in emails for PR/changeset comments
• changelog: added Settings to repo context menu
• utils: better display of ages >1 year when in short form
• remotes: add support to clone from Mercurial repositories over ssh
• hg: reimplement branch head listings more efficiently
• pullrequests: use same code and table ui for Repo Pull Requests and My Pull Requests
• pullrequests: add delete button to My Pull Requests overview
• pullrequests: on my pullrequests show the user s own vote as second column next to latest vote
• pullrequests: add option for adding repo owner as reviewer if none assigned
• pullrequests: make top pane one big form with meta data and one save button and preview changes in pane below more like when creating the PR
• pullrequests: make update functionality a separate submit button in the big form, automatically enabled when an update changeset is chosen
• pullrequests: show graph next to the list of changesets that are available for update
• pullrequests: for available changesets, also show changesets that not are descendants and thus not directly available but will be merged in
• pullrequests: show unrelated branch heads in the graph too show the problem instead of trying to explain it
• pullrequests: show pending reviewer changes with (not saved)
• pullrequests: show comment count in top of summary box
• diff: improved parsing, be more strict and fail with good error messages
• diff: handle GIT delta binary patches
• diff: show renamed files as that
• diff: make trailing spaces more visible make vertical line less transparent
• diff: show CR (\r) in diffs
• comment: warn before leaving page with open comments
• comment: clarify shown comment count (inline vs general)
• comment: extend next/previous links to global comments (Issue #91)

Changes (might influence workflows):

• i18n: en translation has been removed make sure to update your .ini files to have lang = (nothing)
• ini: turn show_revision_number off by default
• pullrequests: don t give changeset unreviewed status just because it is added to a PR
• pullrequests: tweak readable titles just append /_/ and branch name
• pullrequests: only author and admin can close
• pullrequests: don t add repo owner as reviewer in all PRs
• pullrequests: PRs between different repos should default to same destination/source branch
• pullrequests: use -- as separator in the automatic text when updating pull requests and is it to put latest updates first
• diff: don t group diffs based on the kind of change
• admin: set current user as owner when running repo scan
• changelog: change default view to 100 changesets, max to 2000
• date representation: use ISO8601 rather than a specific locale
• user groups: drop add/remove all buttons for member selection they are too dangerous and multi-select can do the same
• hg: drop support for hg web.hidden
• template: link the last revision in the file browser to the changeset

Cleanup:

• code: Fix typos and language in comments and code
• ui: Correct capitalization and improved English text in the UI
• javascript: use strict mode
• tags: rename old non-Kallithea tags and prefix them with 0.0 to show that Kallithea is superior
• javascript: more jQuery, less YUI, less clutter
• templates: better structure, less cut n paste coding
• html: various improvements
• css: various improvements
• exceptions: avoid silently catching exceptions especially avoid catch all
• tests: stability fixes
• tests: remove large unused test fixture
• whitespacecleanup.sh: run it regularly to ensure consistent whitespace and avoid spurious changes
• controllers: consistently use formfill.render with force_defaults=False
• gists: use allow_extra_fields and filter_extra_fields in gist forms like in most other forms
• json: always use kallithea.lib.compat for json and nothing else
• json: drop simplejson fallback and just use stdlib
• db: check for None instead of boolean false if that is what we mean

Experiments Last week I discovered The Fan Club s Experiments page. It reminds me of the Debian Experiments community on Google+. I like the concept of trying out all kinds of different things and reporting back to the community on how it works. I ve been meaning to do more of that myself so this is me jumping in and reporting back on something I ve been poking at this weekend. Introducing Squashfs Squashfs is a read-only compressed filesystem commonly used on embedded devices, Linux installation media and remote file systems (as is done in LTSP). Typically, a system like tmpfs, unionfs or aufs is mounted over this read-only system to make it usable as a root filesystem. It has plenty of other use cases too but for the purposes of this entry we ll stick with those use cases in mind. It supports gzip, lzo and xz(lzma) as compression back-ends. It also supports block sizes from 4K up to 1M. Compression technique as well as block size can have major effects on both performance and file size. In most cases the defaults will probably be sufficient, but if you want to find a good balance between performance and space saving, then you ll need some more insight. My Experiment: Test effects of squashfs compression methods and block sizes I m not the first person to have done some tests on squashfs performance and reported on it. Bernhard Wiedemann and the Squashfs LZMA project have posted some results before, and while very useful I want more information (especially compress/uncompress times). I was surprised to not find a more complete table elsewhere either. Even if such a table existed, I probably wouldn t be satisfied with it. Each squashfs is different and it makes a big difference whether it contains already compressed information or largely uncompressed information like clear text. I d rather be able to gather compression ratio/times for a specific image rather than for one that was used for testing purposes once-off. So, I put together a quick script that takes a squashfs image, extracts it to tmpfs, and then re-compressing it using it all the specified compression techniques and block sizes and then uncompressing those same images for their read speeds. My Testing Environment For this post, I will try out my script on the Ubuntu Desktop 14.04.2 LTS squashfs image. It s a complex image that contains a large mix of different kinds of files. I m extracting it to RAM since I want to avoid having disk performance as a significant factor. I m compressing the data back to SSD and extracting from there for read speed tests. The SSD seems fast enough not to have any significant effect on the tests. If you have a slow storage, the results of the larger images (with smaller block sizes) may be skewed unfavourably. As Bernhard mentioned on his post, testing the speed of your memory can also be useful, especially when testing on different kinds of systems and comparing the results:

# dd if=/dev/zero of=/dev/null bs=1M count=100000
104857600000 bytes (105 GB) copied, 4.90059 s, 21.4 GB/s

CPU is likely to be your biggest bottleneck by far when compressing. mksquashfs is SMP aware and will use all available cores by default. I m testing this on a dual core Core i7 laptop with hyper-threading (so squashfs will use 4 threads) and with 16GB RAM apparently transferring around 21GB/s. The results of the squashfs testing script will differ greatly based on the CPU cores, core speed, memory speed and storage speed of the computer you re running it on, so it shouldn t come as a surprise if you get different results than I did. If you don t have any significant bottleneck (like slow disks, slow CPU, running out of RAM, etc) then your results should more or less correspond in scale to mine for the same image. How to Run It Create a directory and place the filesystem you d like to test as filesystem.squashfs, then:

$ apt-get install squashfs-tools 
$ wget https://raw.githubusercontent.com/highvoltage/squashfs-experiments/master/test-mksquashfs.sh
$ bash test-mksquashfs.sh

With the default values in that file, you ll end up with 18 squashfs images taking up about 18GB of disk space. I keep all the results for inspection, but I ll probably adapt/fix the script to be more friendly to disk space usage some time. You should see output that look something like this, with all the resulting data in the ./results directory.

* Setting up...
- Testing gzip
  * Running a squashfs using compression gzip, blocksize 4096
  * Running a squashfs using compression gzip, blocksize 8192
  * Running a squashfs using compression gzip, blocksize 16384
  ...
- Testing lzo
  * Running a squashfs using compression lzo, blocksize 4096
  * Running a squashfs using compression lzo, blocksize 8192
  * Running a squashfs using compression lzo, blocksize 16384
  ...
- Testing xz
  * Running a squashfs using compression xz, blocksize 4096
  * Running a squashfs using compression xz, blocksize 8192
  * Running a squashfs using compression xz, blocksize 16384
  ...
* Testing uncompressing times...
  * Reading results/squashfs-gzip-131072.squashfs...
  * Reading results/squashfs-gzip-16384.squashfs...
  * Reading results/squashfs-gzip-32768.squashfs...
  ...
* Cleaning up...

On to the Results The report script will output the results into CSV. Here s the table with my results. Ratio is percentage of the size of the original uncompressed data, CTIME and UTIME is compression time and uncompress time for the entire image.

Filename	Size	Ratio	CTIME	UTIME
squashfs-gzip-4096.squashfs	1137016	39.66%	0m46.167s	0m37.220s
squashfs-gzip-8192.squashfs	1079596	37.67%	0m53.155s	0m35.508s
squashfs-gzip-16384.squashfs	1039076	36.27%	1m9.558s	0m26.988s
squashfs-gzip-32768.squashfs	1008268	35.20%	1m30.056s	0m30.599s
squashfs-gzip-65536.squashfs	987024	34.46%	1m51.281s	0m35.223s
squashfs-gzip-131072.squashfs	975708	34.07%	1m59.663s	0m22.878s
squashfs-gzip-262144.squashfs	970280	33.88%	2m13.246s	0m23.321s
squashfs-gzip-524288.squashfs	967704	33.79%	2m11.515s	0m24.865s
squashfs-gzip-1048576.squashfs	966580	33.75%	2m14.558s	0m28.029s
squashfs-lzo-4096.squashfs	1286776	44.88%	1m36.025s	0m22.179s
squashfs-lzo-8192.squashfs	1221920	42.64%	1m49.862s	0m21.690s
squashfs-lzo-16384.squashfs	1170636	40.86%	2m5.008s	0m20.831s
squashfs-lzo-32768.squashfs	1127432	39.36%	2m23.616s	0m20.597s
squashfs-lzo-65536.squashfs	1092788	38.15%	2m48.817s	0m21.164s
squashfs-lzo-131072.squashfs	1072208	37.43%	3m4.990s	0m20.563s
squashfs-lzo-262144.squashfs	1062544	37.10%	3m26.816s	0m15.708s
squashfs-lzo-524288.squashfs	1057780	36.93%	3m32.189s	0m16.166s
squashfs-lzo-1048576.squashfs	1055532	36.85%	3m42.566s	0m17.507s
squashfs-xz-4096.squashfs	1094880	38.19%	5m28.104s	2m21.373s
squashfs-xz-8192.squashfs	1002876	34.99%	5m15.148s	2m1.780s
squashfs-xz-16384.squashfs	937748	32.73%	5m11.683s	1m47.878s
squashfs-xz-32768.squashfs	888908	31.03%	5m17.207s	1m43.399s
squashfs-xz-65536.squashfs	852048	29.75%	5m27.819s	1m38.211s
squashfs-xz-131072.squashfs	823216	28.74%	5m42.993s	1m29.708s
squashfs-xz-262144.squashfs	799336	27.91%	6m30.575s	1m16.502s
squashfs-xz-524288.squashfs	778140	27.17%	6m58.455s	1m20.234s
squashfs-xz-1048576.squashfs	759244	26.51%	7m19.205s	1m28.721s

Some notes:

• Even though images with larger block sizes uncompress faster as a whole, they may introduce more latency on live media since a whole block will need to be uncompressed even if you re just reading just 1 byte from a file.
• Ubuntu uses gzip with a block size of 131072 bytes on it s official images. If you re doing a custom spin, you can get improved performance on live media by using a 16384 block size with a sacrifice of around 3% more squashfs image space.
• I didn t experiment with Xdict-size (dictionary size) for xz compression yet, might be worth while sacrificing some memory for better performance / compression ratio.
• I also want stats for random byte reads on a squashfs, and typical per-block decompression for compressed and uncompressed files. That will give better insights on what might work best on embedded devices, live environments and netboot images (the above table is more useful for large complete reads, which is useful for installer images but not much else), but that will have to wait for another day.
• In the meantime, please experiment on your own images and feel free to submit patches.

This was my fourth month working on Debian LTS. I was assigned 14.5 hours by Freexian's Debian LTS initiative, but I only worked 11.5 as I had a week's holiday and then was ill for part of this week. eglibc My first task was to complete the eglibc update that I began at the end of February. There were a few unexpected failures in the regression test suite, but on inspection these turned out to be quirks of the build environment:

• A test of the nice(2) function failed because I ran the build itself with nice(1). I should probably use a cgroup to reduce its priority instead; this time I re-ran without using either.
• Several tests of cancellation failed because they rely on a socket write blocking due to filling up the buffer, while the kernel's minimum socket buffer size has increased such that the write isn't big enough to block. (The host kernel was Linux 3.16 from jessie.) I found and applied a fix for this from the (glibc) upstream repository.

With those changes, the regression test results are the same as for previous package versions. (There is still one 'unexpected' failure, but I didn't investigate because it is not a regression within squeeze.) I reviewed the backported patches again to check as well as I could that they did not depend on other upstream changes, and they did not seem to. I also received positive feedback from further testing - I can't find the message now, but I think it was that the Univention application test suite passed with the updated eglibc package installed. With that confirmation and no regressions reported, I uploaded eglibc 2.11.3-4+deb6u5 and issued DLA 165-1. freetype A large number of vulnerabilities in font file parsing in Freetype were reported by Mateusz Jurczyk. I think these were less critical in squeeze than in wheezy, because while current web browsers use Freetype to render untrusted 'web fonts' we don't support any web browsers in squeeze LTS. But it seemed like they ought to be fixed anyway, and it was not too hard to backport the patches included in the wheezy security update. Freetype doesn't have a regression test suite and I didn't have samples of the broken fonts, so I came up with some ad hoc tests for regressions. I viewed several of each of the affected font formats with the ftview demo application. I then wrote a script to match up the dependencies of a package (in this case, libfreetype6) with those used by Freexian's customers; the top results were fontconfig, xfonts-utils and imagemagick. So I ran some basic tests against each of the affected formats with each of these (fc-list, mkfontscale, and the simple text label recipe for ImageMagick) and found no regression. Uploaded freetype 2.4.2-2.1+squeeze5 and issued DLA 185-1.