I've had a half-broken temperature monitoring setup at home for quite some time. It started out with a Atom-based NAS, a USB-serial adapter and a passive 1-wire adapter. It sometimes worked, then stopped working, then started when poked with a stick. Later, the NAS was moved under the stairs and I put a Beaglebone Black in its old place. The temperature monitoring thereafter never really worked, but I didn't have the time to fix it. Over the last few days, I've managed to get it working again, of course by replacing nearly all the existing components. I'm using the DS18B20 sensors. They're about USD 1 a piece on Ebay (when buying small quantities) and seems to work quite ok. My first task was to address the reliability problems: Dropouts and really poor performance. I thought the passive adapter was problematic, in particular with the wire lengths I'm using and I therefore wanted to replace it with something else. The BBB has GPIO support, and various blog posts suggested using that. However, I'm running Debian on my BBB which doesn't have support for DTB overrides, so I needed to patch the kernel DTB. (Apparently, DTB overrides are landing upstream, but obviously not in time for Jessie.) I've never even looked at Device Tree before, but the structure was reasonably simple and with a sample override from bonebrews it was easy enough to come up with my patch. This uses pin 11 (yes, 11, not 13, read the bonebrews article for explanation on the numbering) on the P8 block. This needs to be compiled into a .dtb. I found the easiest way was just to drop the patched .dts into an unpacked kernel tree and then running make dtbs. Once this works, you need to compile the w1-gpio kernel module, since Debian hasn't yet enabled that. Run make menuconfig, find it under "Device drivers", "1-wire", "1-wire bus master", build it as a module. I then had to build a full kernel to get the symversions right, then build the modules. I think there is or should be an easier way to do that, but as I cross-built it on a fast AMD64 machine, I didn't investigate too much. Insmod-ing w1-gpio then works, but for me, it failed to detect any sensors. Reading the data sheet, it looked like a pull-up resistor on the data line was needed. I had enabled the internal pull-up, but apparently that wasn't enough, so I added a 4.7kOhm resistor between pin 3 (VDD_3V3) on P9 and pin (GPIO_45) on P8. With that in place, my sensors showed up in /sys/bus/w1/devices and you can read the values using cat. In my case, I wanted the data to go into collectd and then to graphite. I first tried using an Exec plugin, but never got it to work properly. Using a [python plugin] worked much better and my graphite installation is now showing me temperatures. Now I just need to add more probes around the house. The most useful references were

• Bonebrews article
• Cross compiling official kernel packages
• Verifying pin usage. This is useful for double checking that your DTB actually exports the pins you expect it to.

In addition, various searches for DS18B20 pinout and similar, of course.

For once you have tasted flight,
You will walk the earth with your eyes turned skyward;
For there you have been,
And there you long to return. Leonardo da Vinci

There is something of a magic to flight, to piloting. I remember the first flight I ever took, after years of dreaming of flying in a plane: my grandma had bought me a plane ticket. In one of the early morning flights, I witnessed a sunrise above cumulus clouds. Although I was just 10 or so at the time, that still is a most beautiful image seared into my memory. I have become meh about commercial flight over the years. The drive to the airport, the security lines, the lack of scenery at 35,000 feet. And yet, there is much more to flight than that. When I purchased what was essentially a flying camera, I saw a whole new dimension of the earth s amazing beauty. All the photos in this post, in fact, are ones I took. I then got a RC airplane, because flying the quadcopter was really way too easy.

It s wonderful to climb the liquid mountains of the sky.
Behind me and before me is God, and I have no fears. Helen Keller

Start talking to pilots, and you notice a remarkable thing: this group of people that tends to be cool and logical, methodical and precise, suddenly finds themselves using language almost spiritual. Many have told me that being a pilot brings home how much all humans have in common, the unifying fact of sharing this beautiful planet together. Many volunteer with organizations such as Angel Flight. And having been up in small planes a few times, I start to glimpse this. Flying over my home at 1000 up, or from lake to lake in Seattle with a better view than the Space Needle, seeing places familiar and new, but from a new perspective, drives home again and again the beauty of our world, the sheer goodness of it, and the wonderful color of the humanity that inhabits it.

The air up there in the clouds is very pure and fine, bracing and delicious. And why shouldn t it be? It is the same the angels breathe. Mark Twain

The view from 1000 feet, or 3000, is often so much more spectacular than the view from 35,000 ft as you get on a commercial flight. The flexibility is too; there are airports all over the country that smaller planes can use which the airlines never touch. Here is one incredible video from a guy that is slightly crazy but does ground-skimming, flying just a few feet off the ground: (try skipping to 9:36) So what comes next is something I blame slightly on my dad and younger brother. My dad helped get me interested in photography as a child, and that interest has stuck. It s what caused me to get into quadcopters ( a flying camera for less than the price of a nice lens! ). And my younger brother started mentioning airplanes to me last year for some reason, as if he was just trying to get me interested. Eventually, it worked. I started talking to the pilots I know (I know quite a few; there seems to be a substantial overlap between amateur radio and pilots). I started researching planes, flight, and especially safety the most important factor. And eventually I decided I wanted to be a pilot. I ve been studying feverishly, carrying around textbooks and notebooks in the car, around the house, and even on a plane. There is a lot to learn. And today, I took my first flight with a flight instructor. Today I actually flew a plane for awhile. Wow! There is nothing quite like that experience. Seeing a part of the world I am familiar with from a new perspective, and then actually controlling this amazing machine I really fail to find the words to describe it. I have put in many hours of study already, and there will be many more studying and flying, but it is absolutely worth it. Here is one final video about one of the most unique places you can fly to in Kansas. And a blog with lots of photos of a flight to Beaumont called Horse on the runway .

A few months ago, I wrote a long post about what I called back then the "Gnocchi experiment". Time passed and we me and the rest of the Gnocchi team continued to work on that project, finalizing it. It's with a great pleasure that we are going to release our first 1.0 version this month, roughly at the same time that the integrated OpenStack projects release their Kilo milestone. The first release candidate numbered 1.0.0rc1 has been released this morning! The problem to solve Before I dive into Gnocchi details, it's important to have a good view of what problems Gnocchi is trying to solve. Most of the IT infrastructures out there consists of a set of resources. These resources have properties: some of them are simple attributes whereas others might be measurable quantities (also known as metrics). And in this context, the cloud infrastructures make no exception. We talk about instances, volumes, networks which are all different kind of resources. The problems that are arising with the cloud trend is the scalability of storing all this data and being able to request them later, for whatever usage. What Gnocchi provides is a REST API that allows the user to manipulate resources (CRUD) and their attributes, while preserving the history of those resources and their attributes. Gnocchi is fully documented and the documentation is available online. We are the first OpenStack project to require patches to integrate the documentation. We want to raise the bar, so we took a stand on that. That's part of our policy, the same way it's part of the OpenStack policy to require unit tests. I'm not going to paraphrase the whole Gnocchi documentation, which covers things like installation (super easy), but I'll guide you through some basics of the features provided by the REST API. I will show you some example so you can have a better understanding of what you could leverage using Gnocchi! Handling metrics Gnocchi provides a full REST API to manipulate time-series that are called metrics. You can easily create a metric using a simple HTTP request:
The archive_policy_name parameter defines how the measures that are being sent are going to be aggregated. You can also define archive policies using the API and specify what kind of aggregation period and granularity you want. In that case , the low archive policy keeps 1 hour of data aggregated over 1 second and 1 day of data aggregated to 30 minutes. The functions used for aggregations are the mathematical functions standard deviation, minimum, maximum, and even 95th percentile. All of that is obviously customizable and you can create your own archive policies. If you don't want to specify the archive policy manually for each metric, you can also create archive policy rule, that will apply a specific archive policy based on the metric name, e.g. metrics matching disk.* will be high resolution metrics so they will use the high archive policy. It's also worth noting Gnocchi is precise up to the nanosecond and is not tied to the current time. You can manipulate and inject measures that are years old and precise to the nanosecond. You can also inject points with old timestamps (i.e. old compared to the most recent one in the timeseries) with an archive policy allowing it (see back_window parameter). It's then possible to send measures to this metric:

These measures are synchronously aggregated and stored into the configured storage backend. Our most scalable storage drivers for now are either based on Swift or Ceph which are both scalable storage objects systems. It's then possible to retrieve these values:
As older Ceilometer users might notice here, metrics are only storing points and values, nothing fancy such as metadata anymore. By default, values eagerly aggregated using mean are returned for all supported granularities. You can obviously specify a time range or a different aggregation function using the aggregation, start and stop query parameter. Gnocchi also supports doing aggregation across aggregated metrics:
This computes the mean of mean for the metric 65071775-52a8-4d2e-abb3-1377c2fe5c55 and 9ccdd0d6-f56a-4bba-93dc-154980b6e69a starting on 6th October 2014 at 14:34 UTC. Indexing your resources Another object and concept that Gnocchi provides is the ability to manipulate resources. There is a basic type of resource, called generic, which has very few attributes. You can extend this type to specialize it, and that's what Gnocchi does by default by providing resource types known for OpenStack such as instance, volume, network or even image.
The resource is created with the UUID provided by the user. Gnocchi handles the history of the resource, and that's what the revision_start and revision_end fields are for. They indicates the lifetime of this revision of the resource. The ETag and Last-Modified headers are also unique to this resource revision and can be used in a subsequent request using If-Match or If-Not-Match header, for example:
Which is useful to synchronize and update any view of the resources you might have in your application. You can use the PATCH HTTP method to modify properties of the resource, which will create a new revision of the resource. The history of the resources are available via the REST API obviously. The metrics properties of the resource allow you to link metrics to a resource. You can link existing metrics or create new ones dynamically:
Haystack, needle? Find! With such a system, it becomes very easy to index all your resources, meter them and retrieve this data. What's even more interesting is to query the system to find and list the resources you are interested in! You can search for a resource based on any field, for example:
That query will return a list of all resources owned by the user_id bd3a1e52-1c62-44cb-bf04-660bd88cd74d. You can do fancier queries such as retrieving all the instances started by a user this month:
And you can even do fancier queries than the fancier ones (still following?). What if we wanted to retrieve all the instances that were on host foobar the 15th April and who had already 30 minutes of uptime? Let's ask Gnocchi to look in the history!
I could also mention the fact that you can search for value in metrics. One feature that I will very likely include in Gnocchi 1.1 is the ability to search for resource whose specific metrics matches some value. For example, having the ability to search for instances whose CPU consumption was over 80% during a month. Cherries on the cake While Gnocchi is well integrated and based on common OpenStack technology, please do note that it is completely able to function without any other OpenStack component and is pretty straight-forward to deploy. Gnocchi also implements a full RBAC system based on the OpenStack standard oslo.policy and which allows pretty fine grained control of permissions.

There is also some work ongoing to have HTML rendering when browsing the API using a Web browser. While still simple, we'd like to have a minimal Web interface served on top of the API for the same price! Ceilometer alarm subsystem supports Gnocchi with the Kilo release, meaning you can use it to trigger actions when a metric value crosses some threshold. And OpenStack Heat also supports auto-scaling your instances based on Ceilometer+Gnocchi alarms. And there are a few more API calls that I didn't talk about here, so don't hesitate to take a peek at the full documentation! Towards Gnocchi 1.1! Gnocchi is a different beast in the OpenStack community. It is under the umbrella of the Ceilometer program, but it's one of the first projects that is not part of the (old) integrated release. Therefore we decided to have a release schedule not directly linked to the OpenStack and we'll release more often that the rest of the old OpenStack components probably once every 2 months or the like. What's coming next is a close integration with Ceilometer (e.g. moving the dispatcher code from Gnocchi to Ceilometer) and probably more features as we have more requests from our users. We are also exploring different backends such as InfluxDB (storage) or MongoDB (indexer). Stay tuned, and happy hacking!

Background R relies on package repositories for initial installation of a package via install.packages(). A crucial second step is update.packages(): For all currently installed packages, a list of available updates is constructed or offered for either one-by-one or bulk updates. This keeps the local packages in sync with upstream, and provides for a very convenient way to obtain new features, bug fixes and other improvements. So by installing from a repository, we automatically have the ability to track the repository for updates.

Enter drat Fairly recently, the drat package was added to the R ecosystem. It makes both aspects of package distribution easy: providing a package (if you are an author) as well as installing it (if you are a user). Now, because drat is at the same time source code (as it is also a package providing the functionality), and a repository (using what drat provides ib features), the "namespace" becomes a little cluttered. But because a key feature of drat is the "one variable" unique identification via the GitHub, I opted to create a drat repository in the name of a new organisation: ghrr. This is a simple acronym for GitHub Hosted R Repository.

Use cases We can outline several use case for packages in ghrr:

• packages not published in a repo by their authors: I already use two like that:
  • fasttime, an impeccably fast parser for ISO datetimes by Simon Urbanek which was however never released into a repo by Simon, and
  • RcppR6, a very nice extension to both R6 (by Winston) and Rcpp, by Rich FitzJohn; similarly never released beyond GitHub;
• packages possibly unsuitable for mainline repos:
  • Rblpapi is a great package by Whit Armstong and John Laing to which I have been contributing quite a bit of late. As it requires a free-to-use but not open source library and headers from Bloomberg, it will never make it to the mainline repository for R, but hosting it in ghrr is perfect as I can easily update several machines at work once I cut a new development release;
  • winsorize is a small package I needed a few weeks ago; it is spun out of robustHD but does not yet contain new code so Andreas and I are content to keep it in this drat for now;
• packages in pre-relase mode:
  • RcppArmadillo where I announced both a release candidate before Armadillo 5.000 came out, as well as the actual RcppArmadillo 0.500.0.0 which is not (yet) on the mainline repository as two affected packages need a small update first. Users, however, can get RcppArmadillo already from the sibling Rcpp drat repo.
  • RcppToml is a new package I am currently working on implementing a toml parser based on cpptoml. It works, but it not quite ready for public announcements yet, and hence perfect for ghrr.

Going forward ghrr is meant to be open. While anybody can open a drat repository, particularly on GitHub, it may be beneficial to somehow group packages. This is however not something that can be planned ex-ante: it may just happen if others who see similar benefits in this can in fact contribute. In that spirit, I strongly encourage pull requests. Early on, I made my commit messages conform to a pattern of package version sha1 repourl to make code provenance of every commit very clear. Ideally, subsequent commits would conform to such a scheme, or replace it with a better one.

Some Resources A few links to learn more about drat and ghrr:

• ghrr repo view (to see both content and commits) and ghrr web page
• drat repo view, drat code page, and drat web page

Comments and questions via email or issue tickets are more than welcome. We hope that others find ghrr to be a useful tool for easy repository management and use via GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

I m a big fan of org-mode (see previous posts), and I ve started maintaining (sic) my professional webpage(s) with it. But I ve also recently tried and publish some more Semantic/Linked Data aware documents too (again, previous posts). Ideally, I think my preferred workflow for publishing articles or documents of some importance, would be to author them in org-mode, and then publish them as HTML5 including RDFa meta-data and annotations. Instead, I ve more frequently been doing conversions of org-mode to LaTeX, in order to submit a printable version, and later-on decided to convert the LaTeX to HTML5+RDFa But one of the issues is how to properly embed the RDF meta-data inside the org-mode documents, so that the syntax is both compact and expressive enough. I doubt there s a universal solution, given that RDF tends to be complex, and graphs may not project easilly along a mainly linear structure of an org-mode document, but anyway, there seems to be possible middle grounds that are practically good enough. I ve tried and implement a solution, which reuses the principles set by John Kitchin in Extending the org-mode link syntax with attributes, i.e. implementing an HTML exporter for a particular custom link type, which will convert the plist-like syntax to some RDFa constructs. Here s a description of the whole solution : http://www-public.telecom-sudparis.eu/~berger_o/test-org-publishing-rdfa.html The nice thing about org-mode, and its litterate programming babel environment, is that it allows to embed the code of the links exporter inside the org document, avoiding to dissociate the converter from the document s source, making it auto-complete. Next step will probably be to author a paper (or convert back a preprint of mines) with org-mode, in order to provide Linked Research meta-data. Stay tuned for more details, and in the meantime, I welcome any improvement to the org/babel/elisp setup. Edit: I ve recorded a webcast to provide a bit more details, available on YouTube : https://youtu.be/OyI3DVqllx4

In my previous post I mentioned my involvement with the OpenRISC or1k port. It was the technical activity in which I spent most time during 2014 (Debian and otherwise, day job aside). I thought that it would be nice to talk a bit about the port for people who don't know about it, and give an update for those who do know and care. So this post explains a bit how it came to be, details about its development, and finally the current status. It is going to be written as a rather personal account, for that matter, since I did not get involved enough in the OpenRISC community at large to learn much about its internal workings and aspects that I was not directly involved with. There is not much information about all of this elsewhere, only bits and pieces scattered here and there, but specially not much public information at all about the development of the Debian port. There is an OpenRISC entry in the Debian wiki, but it does not contain much information yet. Hopefully, this piece will help a bit to preserve history and give an insight for future porters. First Things First I imagine that most people reading this post will be familiar with the terminology, but just in case, to create a new Debian port means to get a Debian system (GNU/Linux variant, in this case) to run in the OpenRISC or1k computer architecture. Setting to one side all differences between hardware and software, and as described in their site:

The aim of the OpenRISC project is to create free and open source computing platforms

It is therefore a good match for the purposes of Debian and Free Software world in general. The processor has not been produced in silicon, or not available for the masses in any case. People with the necessary know-how can download the hardware description (Verilog) and synthesise it in a FPGA, or otherwise use simulators. It is not some piece of hardware that people can purchase yet, and there are no plans to mass-produce it in the near future either. The two people (including me) involved in this Debian port did not have the hardware, so we created the port entirely through cross-compiling from other architectures, and then compiling inside Qemu. In a sense, we were creating a Debian port for hardware that "does not [phisically] exist". The software that we built was tested by people who had hardware available in FPGA, though, so it was at least usable. I understand that people working in the arm64 port had to work similarly in the initial phases, working in the dark without access to real hardware to compile or test. The Spark The first time that I heard about the initiative to create the port was in late February of 2014, in a post which appeared in Linux Weekly News (sent by Paul Wise) and Slashdot. The original post announcing it was actually from late January, from Christian Svensson (blueCmd):

Some people know that I've been working on porting Glibc and doing some toolchain work. My evil master plan was to make a Debian port, and today I'm a happy hacker indeed! Below is a link to a screencast of me installing Debian for OpenRISC, installing python2.7 via apt-get (which you shouldn't do in or1ksim, it takes ages! (but it works!)) and running a small Python script. http://asciinema.org/a/7362

So, now, what can a Debian Hacker do when reading this? (Even if one's Hackery Level is not that high, as it is my case). And well, How Hard Can It Be? I mean, Really? Well, in my own defence, I knew that the answer to the last two questions would be a resounding Very . But for some reason the idea grabbed me and I couldn't help but think that it would be a Really Exciting Project, and that somehow I would like to get involved. So I wrote to Christian offering my help after considering it for a few days, around mid March, and he welcomed me aboard. The Ball Was Already Rolling Christian had already been in contact with the people behind DebianBootstrap, and he had already created the repository http://openrisc.debian.net/ with many packages of the base system and beyond (read: packages name_version_or1k.deb available to download and install). Still nowadays the packages are not signed with proper keys, though, so use your judgement if you want to try them. After a few weeks, I got up to speed with the status of the project and got my system working with the necessary tools. This meant basically sbuild/schroot to compile new packages, with the base system that Christian already got working, installed in a chroot, probably with the help of debootstrap, and qemu-system-or1k to simulate the system. Only a few of the packages were different from the version in Debian, like gcc, binutils or glibc -- they had not been upstreamed yet. sbuild ran through qemu-system-or1k, so the compilation of new packages could happen "natively" (running inside Qemu) rather than cross-compiling the packages, pulling _or1k.deb packages for dependencies from the repository that he had prepared, and _all.deb packages from snapshots.debian.org. I started by trying to get the packages that I [co-]maintain in Debian compiled for this architecture, creating the corresponding _or1k.deb. For most of them, though, I needed many dependencies compiled before I could even compile my packages. The GNU autotools / autoreconf Problem Since very early, many of the packages failed to build with messages such as:

Invalid configuration 'or1k-linux-gnu': machine 'or1k' not recognized
configure: error: /bin/bash ../config.sub or1k-linux-gnu failed

This means that software packages based on GNU autotools and using configure scripts need recent versions of the files config.sub and config.guess that they ship in their root directory, to be able to detect the architecture and generate the code accordingly. This is counter-intuitive, having into account that GNU autotools were designed to help with portability. Yet, in the case of creating new Debian ports, it meant that unless upstream had very recent versions of config. guess,sub , it would prevent the package to compile straight away in the new architectures -- even if invoking gcc without ado would have worked without problems in most cases for native compilation. Of course this did not only affect or1k, and there was already the autoreconf effort underway as a way to update these files automatically when building Debian packages, pushed by people porting Debian to the new architectures added in 2013/2014 (mips64el, arm64, ppc64el), which encountered the same roadblock. This affected around a thousand source packages in unstable. A Royal Pain. Also, all of their reverse dependencies (packages that depended on those to be built) could not be compiled straight away. The bugs were not Release Critical, though (none of these architectures were officially accepted at the time), so for people not concerned with the new ports there was no big incentive to get them fixed. This problem, which conceptually is easily solvable, prevented new ports to even attempt compile vast portions of the archive straight away (cleanly, without modifications to the package or to the host system), for weeks or months. The GNU autotools / autoreconf Solution We tackled this problem mainly in two ways. First, more useful for Debian in general, was to do as other porters were doing and submit bug reports and patches to Debian packages requesting them to use autoreconf, and NMUing packages (uploading changes to the archive without the official maintainers' intervention). A few NMUs were made for packages which had bug reports with patches available for a while, that were in the critical path to get many other packages compiled, and that were orphan or had almost no maintainer activity. The people working in the other new ports, and mainly Ubuntu people which helped with some of those ports and wanted to support them, had submitted a large amount of requests since late 2013, so there was no shortage of NMUs to be made. Some porters, not being Debian Developers, could not easily get the changes applied; so I also helped a bit the porters of other architectures, specially later on before the freeze of Jessie, to get as many packages compiled in those architectures as possible. The second way was to create dpkg-buildpackage hooks that updated unconditionally config. guess,sub before attempting to build the package in the local build system. This local and temporary solution allowed us in the or1k port to get many _or1k.deb packages in the experimental repository, which in turn would allow many more packages to compile. After a few weeks, I set up many sbuilds in a multi-core machine attempting to build uninterruptedly packages that were not previously built and which had their dependencies available. Every now and then (typically several times per day in peak times) I pushed the resulting _or1k.deb files to the repository, so more packages would have the necessary dependencies ready to attempt to build. Christian was doing something similar, and by April at peak times, among the two of us, we were compiling some days more than a hundred packages -- a huge amount of packages did not need any change other than up-to-date config. guess,sub files. At some point, late April, Christian set up wanna-build in a few hosts to do this more elegantly and smartly than my method, and more effectively as well. Ugly Hacks, Bugs and Shortcomings in the Toolchain and Qemu Some packages are extremely important because many other packages need them to compile (like cmake, Qt or GTK+), and they are themselves very complex and have dependency loops. They had deeper problems than the autoreconf issue and needed some seriously dirty hacking to get them built. To try to get as many packages compiled as possible, we sometimes compiled these important packages with some functionality disabled, disabling some binary packages (e.g. Java bindings) or specially disabling documentation (using DEB_BUILD_OPTIONS=nodoc when possible, and more aggressively when needed by removing chunks of debian/rules). I tried to use the more aggressive methods in as few packages as possible, though, about a dozen in total. We also used DEB_BUILD_OPTIONS=nocheck for speeding up compilation and avoiding build failures -- many packages' tests failed due to qemu-system-or1k not supporting multi-threading, which we could do nothing about at the time, but otherwise the packages mostly passed tests fine. Due to bugs and shortcomings in Qemu and the toolchain --like the compiler lacking atomics, missing functionality in glibc, Qemu entering in endless loops, or programs segfaulting (especially gettext, used by many packages and causing the packages failing to build)--, we had to resort to some very creative ways or time-consuming dull work to edit debian/rules, or to create wrappers of the real programs avoiding or forcing certain options (like gcc -O0, since -O2 made buggy binaries too often). To avoid having a mix of cleanly compiled and hacked packages in the same repository, Christian set up a two-tiered repository system -- the clean one and the dirty one. In the dirty one we dumped all of the packages that we got built, no matter how. The packages in the clean one could use packages from the dirty one to build, but they themselves were compiled without any hackery. Of course this was not a completely airtight solution, since they could contain code injected at build time from the "dirty repository" (e.g. by static linking), and perhaps other quirks. We hoped to get rid of these problems later by rebuilding all packages against clean builds of all their dependencies. In addition, Christian also spent significant amounts of time working inside the OpenRISC community, debugging problems, testing and recompiling special versions of the toolchain that we could use to advance in our compilation of the whole archive. There were other people in the OpenRISC community implementing the necessary bits in the toolchain, but I don't know the details. Good Progress Olof Kindgren wrote the OpenRISC health report April 2014 (actually posted in May), explaining the status at the time of projects in the broad OpenRISC community, and talking about the software side, Debian port included. Sadly, I think that there have been no more "health reports" since then. There was also a new post in Slashdot entitled OpenRISC Gains Atomic Operations and Multicore Support shortly thereafter. In the side of the Debian port, from time to time new versions of packages entered unstable and we started to use those newer versions. Some of them had nice fixes, like the autoreconf updates, so they did not require local modifications. In other cases, the new versions failed to build when old ones had worked (e.g. because the newer versions added support and dependencies of new versions of gnutls, systemd or other packages not yet available for or1k), and we had to repeat or create more nasty hacks to get the packages built again. But in general, progress was very good. There were about 10k arch-dependent packages in Debian at the time, and we got about half of them compiled by the beginning of May, counting clean and dirty. And, if I recall correctly, there were around the same number of arch=all (which can be installed in any architecture, after the package is built in one of them). Counting both, it meant that systems using or1k got about 15k packages available, or 75% of the whole Debian archive (at least "main", we excluded "contrib" and "non-free"). Not bad. By the middle to end of May, we had about 6k arch-dependent packages compiled, and 4k to go. The count of packages eventually reached ~6.6k at its peak (I think that in June/July). Many had been built with hacks and not rebuilt cleanly yet, but everything was fine until the amount of packages built plateaued. Plateauing There were multiple reasons for that. One of them is that after having fixed the autoreconf issue locally in some packages, new versions were uploaded to Debian without fixing that problem (in many cases there was no bug report or patch yet, so it was understandable; in other cases the requests were ignored). The wanna-build for the clean repository set up by Christian rightly considered the package out-of-date and prepared to build the more recent version, that failed. Then, other packages entering the unstable archive and build-depending on newer versions of those could not be built ("BD-Uninstallable"), until we built the newer versions of the dependencies in the dirty repository with local hacks. Consequently, the count of cleanly built packages went back-and-forth, when not backwards. More challenging was the fact that when creating a new port, language compilers which are written in that same language need to be built for that architecture first. Sometimes it is not the compiler, but compile-time or run-time support for modules of a language are not ported yet. Obviously, as with other dependencies, large amounts of packages written in those languages are bound to remain uncompiled for a long time. As Colin Watson explained in porting Haskell's GHC to arm64 and ppc64el, untangling some of the chicken-and-egg problems of language compilers for new ports is extremely challenging. Perl and Python are pretty much a pre-requisite of the base Debian system, and Christian got them working early on. But for example in May, 247 packages depended on r-base-dev (GNU R) for building, and 736 on ghc, and we did not have these dependencies compiled. Just counting those two, 1k source packages of the remaining 4k to 5k to be compiled for the new architecture would have to wait for a long time. Then there was Java, Mono, etc... Even more worrying problems were the pending issues with the toolchain, like atomics in glibc, or make check failing for some packages in the clean repository built with wanna-build. Christian continued to work on the toolchain and liasing with the rest of the OpenRISC community, I continued to request more changes to the Debian archive through a few requests to use autoreconf, and pushing a few more NMUs. Though many requests were attended, I soon got negative replies/reactions and backed off a bit. In the meantime, the porters of other new architectures at the time were mostly submitting requests to support them and not NMUing much either. Upstreaming Things continued more or less in the same state until the end of the summer. The new ports needed as many packages built as possible before the evaluation of which official ports to accept (in early September, I think, the final decision around the time of the freeze). Porters of the other new architectures (and maintainers, and other helpful Debian Developers) were by then more active in pushing for changes, specially remaining autoreconf issues, many of which benefited or1k. As I said before, I also kept pushing NMUs now and then, specially during summer, for packages which were not of immediate benefit for our port but helped the others (e.g. ppc64el needed updates to ltmain.sh for libtool which were not necessary for or1k, in addition to config. guess,sub ). In parallel in the or1k camp, there were patches that needed changes to be sent upstream, like for example Python's NumPy, that I submitted in May to the Debian package and upstream, and was uploaded to Debian in September with a new upstream release. Similar paths were followed between May and September for packages such as jemalloc, ocaml, gstreamer0.10, libgc, mesa, X.org's cf module and cmake (patch created by Christian). In April, Christian had reached the amazing milestone of tracking and getting all of the contributors of the port of GNU binutils to assign copyright to the Free Software Foundation (FSF), all of the work was refreshed and upstreamed. In July or August, he started to gather information about the contributors of the GCC port, which had started more than a decade ago. After that, nothing much happened (from the outside) until the end of the year, when Christian sent a message about the status of upstreaming GCC to the OpenRISC community. There was only one remaining person to assign the copyright to the FSF, but it was a blocker. In addition, there was the need to find one or more maintainers to liaise with upstream, review the patches, fix the remaining failures in the test suite and keeping the port in good shape. A few months after that and from what I could gather, the status remains the same. Current Status, and The Future? In terms of the Debian port, there have not been huge visible changes since the end of the summer, and not only because of the Jessie freeze. It seems that for this effort to keep going forward and be sustainable, sorting out the issues with GCC and Glibc is essential. That means having the toolchain completely pushed upstream and in good shape, and particularly completing the copyright assignment. Debian will not accept private forks of those essential packages without a very good reason even in unofficially supported ports; and from the point of view of porters, working in the remaining not-yet-built packages with continuing problems in the toolchain is very frustrating and time-consuming. Other than that, there is already a significant amount of software available that could run in an or1k system, so I think that overall the project has achieved a significant amount of success. Granted, KDE and LibreOffice are not available yet, neither are the tools depending on Haskell or Java. But a lot of software is available (including things high in the stack, like XFCE), and in many aspects it should provide a much more functional system that the one available in Linux (or other free software) systems in the late 1990s. If the blocking issues are sorted out in the near future, the effort needed to get a very functional port, on par with the unofficial Debian ports, should not be that big. In my opinion, and looking at the big picture, not bad at all for an architecture whose hardware implementation is not easy to come by, and in which the port was created almost solely with simulators. That it has been possible to get this far with such meagre resources, it's an amazing feat of Free Software and Debian in particular. As for the future, time will tell, as usual. I will try to keep you posted if there is any significant change in the future.

Review: The Girls from Alcyone, by Cary Caffrey

Series:	Girls from Alcyone #1
Publisher:	Tealy
Copyright:	2011
ISBN:	1-105-33727-8
Format:	Kindle
Pages:	315

Sigrid is a very special genetic match born to not particularly special parents, deeply in debt in the slums of Earth. That's how she finds herself being purchased by a mercenary corporation at the age of nine, destined for a secret training program involving everything from physical conditioning to computer implants, designed to make her a weapon. Sigrid, her friend Suko, and the rest of their class are a special project of the leader of the Kimura corporation, one that's controversial even among the corporate board, and when the other mercenary companies unite against Kimura's plans, they become wanted contraband. This sounds like it could be a tense SF thriller, but I'll make my confession at the start of the review: I had great difficulty taking this book seriously. Initially, it had me wondering what horrible alterations and mind control Kimura was going to impose on the girls, but it very quickly turned into, well, boarding school drama, with little of the menace I was expecting. Not that bullying, or the adults who ignore it to see how the girls will handle it themselves, are light-hearted material, but it was very predictable. As was the teenage crush that grows into something deeper, the revenge on the nastiest bully that the protagonist manages to not be responsible for, and the conflict between unexpectedly competent girls and an invasion of hostile mercenaries. I'm not particularly well-read or informed about the genre, so I'm not the best person to make this comparison, but the main thing The Girls from Alcyone reminded me of was anime or manga. The mix of boarding-school interpersonal relationships, crushes and passionate love, and hypercompetent female action heroes who wear high heels and have constant narrative attention on their beauty had that feel to it. Add in the lesbian romance and the mechs (of sorts) that show up near the end of the story, and it's hard to shake the feeling that one is reading SF yuri as imagined by a North American author. The other reason why I had a hard time taking this seriously is that it's over-the-top action sequences (it's the Empire Strikes Back rescue scene!) mixed with rather superficial characterization, with one amusing twist: female characters almost always end up being on the side of the angels. Lady Kimura, when she appears, turns into exactly the sort of mentor figure that one would expect given the rest of the story (and the immediate deference she got felt like it was lifted from anime). The villains, meanwhile, are hissable and motivated by greed or control. While there's a board showdown, there's no subtle political maneuvering, just a variety of more or less effective temper tantrums. I found The Girls from Alcyon amusing, and even fun to read in places, but that was mostly from analyzing how closely it matched anime and laughing at how reliably it delivered characteristic tropes. It thoroughly embraces its action-hero story full of beautiful, deadly women, but it felt more like a novelization of a B-grade sci-fi TV show than serious drama. It's just not well-written or deep enough for me to enjoy it as a novel. None of the characters were particularly engaging, partly because they were so predictable. And the deeper we got into the politics behind the plot, the less believable I found any of it. I picked this up, along with several other SFF lesbian romances, because sometimes it's nice to read a story with SFF trappings, a positive ending, and a lack of traditional gender roles. The Girls from Alcyone does have most of those things (the gender roles are tweaked but still involve a lot of men looking at beautiful women). But unless you really love anime-style high-tech mercenary boarding-school yuri, want to read it in book form, and don't mind a lot of cliches, I can't recommend it. Followed by The Machines of Bellatrix. Rating: 3 out of 10

Note: In this post I mention some problems and ask questions (to myself, like thinking aloud ). The goal is not to get answers to those questions (I suppose that I will find them soon or later in the internet, manuals and so), but to show the kind of problems and questions that arise in my selfhosting adventures, which I suppose are common to other people trying to administer a home server with some web services. Am I an userop? Well I m something in the middle of (GNU/Linux) user and sysadmin: I have studied computer technical engineering but most of my experience has been in helpdesk, providing support for Windows users. I m running Debian in some LAMP boxes at work (without GUI) since 2008 or so, and in my desktops (with GUI) since 2010. I don t code nor package, but I don t mind trying to read code and understand it (or not). I know a bit of C, a bit of Python, of PHP, and enough Perl to open a Perl file and close it after two minutes, understanding that it s great, but too much for me :) I translate software, so I m not scared to clone a repository, edit files, commit or submit a patch. I m not scared of compiling a program (except if it s an Android app: I try to avoid setting up the development environment just to try some translation that I made but I built my Puma before it was the binary available for download or in F-Droid). In conclusion, I feel more like a GNU/Linux power user than a sysadmin . Sometimes just a user or even a newbie (for example, I don t know very well the Unix/Linux folder tree where are the wallpapers stored? Does it depend on the desktop that I use?). Anyway. I won t stop my free software + free networks digital life because I don t know many things. I bought a small server for home last September, and I wanted to try to selfhost some services, for me and for my family. I want to be a home sysadmin or something like that, so I joined the userops mailing list :) Here you have my experiences on selfhosting/being an userop until now. Mail I even didn t try to setup my mail server, because many people say it s a pain (although nice articles were published about how to do it, for example this series in ArsTechnica) and I need a static IP which is 14 /month more to my ISP, and Gandi, the place where I rented my domain name, provides mail, and they use Debian and Roundcube, and sponsor Debian too, so I decided to trust on them. So this is my strategy now, to try to keep mail under my control:

• Trust my domain provider.
• Backup my mail and keep local copies, removing sensible stuff from the server.
• Use and spread the word about GPG encryption.
• Try not to send photos or videos by mail, just send the link to my MediaGoblin instance (see below).

MediaGoblin I ve setup two MediaGoblin instances (yes, two!). I managed to do it in Debian 7 stable (I think NodeJS npm was not needed then), but soon later I upgraded to Jessie so now it s even better. I installed Nginx and PostgreSQL via apt, to use them for both instances (and probably some more software later). One instance is public, and I use a Debian user, a PostgreSQL database, and it s running in http://media.larjona.net
I have requested an SSL cert to Gandi but I still didn t deployed it (lazy LArjona!!). The other instance is private, for family photos. I didn t know very well how much of my existing setup could reuse and how to keep both instances in case of downtimes or attack I know more or less the concept or chroot but I don t know how to deploy it in my machine. So I decided to use another Debian user, another PostgreSQL database, deploy MediaGoblin in a different folder, and create another virtual server in my Nginx to serve it. I managed to setup that virtual server to http-authenticate and to serve content via a different port, and use a self-signed SSL certificate (it s only for family, so it does not matter). I created another (unprivileged) Debian user with a password for the nginx authentication, and gave my family the URL in the form https://mediaprivate.larjona.net:PortNumber and the user and password (mediaprivate is a string, and PortNumber is a number). I think they don t use the instance too much, but at least I upload photos there from time to time and email the link instead of emailing the photos themselves (they don t use GPG either ). Upgrades I upgraded MediaGoblin from 0.7.1 to 0.8.0 successfully, I sent a report about how I did it to the mailing list. First I upgraded the public instance, when I figure out the process, I upgraded the second instance to test my instructions, and then, I sent the report with the instructions to the mailing list. Static site and LimeSurvey: the power of free software (with instructions) I wanted to act as a mirror of floss2013.libresoft.es and surveys.libresoft.es since they suffer a downtime and I participated in that project (not in the sysadmin part, but in the research and content creation). The static site floss2013.libresoft.es offered a zip with the whole website tree (since the website was licensed as AGPL), and I had access to the git repo holding the development copy of the website. So I just cloned the repository and setup another nginx virtual server in my machine, and tuned my DNS zone in Gandi website to serve floss2013.larjona.net from home. 10 minutes setup YAY! #inGitWeTrust #FreeSoftwareFTW :) For surveys.larjona.net I had to install a LimeSurvey instance. I knew how to do it because we use LimeSurvey at work, but at home I had Nginx instead of Apache, and PostgreSQL instead of MySQL. And no PHP I searched about how to install PHP in Nginx (I can use apt-get, nice!) and how to install LimeSurvey with Nginx and PostgreSQL (I had documentation about that, so I followed, and it worked). For making available the data (one survey and its results, so people can login as visitor to query and get statistics), I downloaded the LimeSurvey export dataset that we were providing in the static website, followed the replication instructions (hey, I wrote them!), and they worked #oleole! (And here, dear researchers, gets demonstrated that free software and free culture really empower your research and help spreading your results). Etherpad: not so easy, it seems! I m trying to install Etherpad-Lite, but I m suffering a bit. I think I did everything ok according to some guides but I get Bad Gateway and these kind of errors when trying to browse with Lynx in the host:

[error] 3615#0: *24 upstream timed out (110: Connection timed out) 
while reading response header from upstream, 
client: 127.0.0.1, 
server: pad.larjona.net, 
request: "GET / HTTP/1.0", 
upstream: "http://127.0.0.1:9001/", 
host: "pad.larjona.net"
2015/04/17 20:52:56 [error] 3615#0: *24 connect() failed 
(111: Connection refused) while connecting to upstream, 
client: 127.0.0.1, 
server: pad.larjona.net, 
request: "GET / HTTP/1.0", 
upstream: "http://[::1]:9001/", 
host: "pad.larjona.net"

I m not sure if I need to open some port in iptables, my router, or change my nginx configuration because the guides assume you re only serving one website in the port 80 (and I have several of them, now ), or what I ve spent three chunks of time (maybe ~2h each?) on this, in different days, and couldn t figure it out, so I decided to round-robin in my TODO list. Userops thoughts Debian brings peace of mind (for me) On one side, maintaining a Debian box it s quite easy, and the more software that it s packaged, the less time that I spend installing or upgrading. I like being in stable, I m in Jessie now (I migrated when it was frozen), but I ll stay in stable as much as I can. I like that I can use the software that I installed via apt-get for several services (nginx, PostgreSQL ). About the software that is not packaged (MediaGoblin, LimeSurvey, EtherPad, maybe others later), I wonder how dependencies and updates are handled. And maybe (probably) I have installed some components several times, one for each service (this sounds like a Windows box #grr). For example MediaGoblin uses PyPump. PyPump 0.5 is packaged in Debian Jessie. MediaGoblin uses PyPump 0.7+. What if PyPump 0.7+ gets, let s say, into Jessie-backports? Can I benefit from that? I know that MediaGoblin upgrade instructions includes upgrading the dependencies, but what about some security patch in one dependency? Should I upgrade the pip modules periodically? How to know if some upgrade is recommended because patches a vulnerability, or it s just new features (and maybe breaking my setup)? This kind of things are the peace of mind that Debian packaging brings to me: when some piece of software is packaged, I know maybe I need to care about proper setup and configuration, but later, it s kind-of-easy to maintain (since the Debian maintainers care about the rest). I don t mind about cloning a repo and compiling, I mind about later, or coexistance with other program/services. I trust in the MediaGoblin community and I m an active member (I m not developer, but hang on IRC, follow the mailing list, etc) but for example I don t know anything about the EtherPad project. And I don t feel like joining the community (I m already an active member in Debian, MediaGoblin, F-Droid, Pump.io, translator of LimeSurvey and many other small apps that I use, and in the future will use more services, like OwnCloud, XMPP ), joining the community of each software that I use is becoming not sustainable :s Free software is more than software I follow the userop mailing list, and it s becoming very technical. I mostly understand the problems (which are similar to the problems that I face: how to isolate different services, how to easily-configure them, how to make them installable by average user ) But I don t understand most of the solutions proposed, and I think that probably we need technical solutions, but in the meanwhile, some issues can be addressed not with software, but with other means: good documentation, community support, translations, beta-testers This is my conclusion until now. When a project is well documented, I think I can find my way to selfhost no matter if the software is packaged (or contained ) or not. MediaGoblin, and LimeSurvey, are well documented, and the user support channels are very responsive. I find lots of instructions that assume that you will use a whole machine for their service (and not for other things). And lots of documentation for the LAMP stack, but not for Nginx + PostgreSQL and Node instead of PHP So, for each particularity of my setup, I search the internet and try to pick good sources to help me do what I wanted to do. I m kind of privileged Some elements, not software related, to take into account as pre-requisites for succeed selfhosting services:

• I knew what to search.
• I knew which sites to visit from the results (arch wiki, debian wiki, stack overflow, etc: some of them were not the Top1 in the results).
• I had time to read several sources and make my mind about what to do and how.
• I can read, understand, and write in English.
• I have no fear about my broken English.
• I have no impostor syndrome.
• I felt welcome in the FLOSS communities where I hanged out.

These aspects are not present in a lot of people. If I look around to the computer users that I know (mostly Windows+Android, some GNU/Linux users, some Mac OSX users, some iOS users), I find that they search things like X does not work or they cannot write a proper search query in English. Or they trust some random person writing a recipe in their blog, without trying first to understand what the recipe does. Other people just say I m not a professional sysadmin, I ll just do what everybody does (aka use Google services or whatever). What if I try and I don t succeed? . Things like that. We may need some technical solutions (and hackers are thinking about that, and working on that). But I feel that we need (more) a huge group of beta-testers, dogfooding people, aventurers that try the half-cooked solutions and provide successful and unsuccessful experiences, to guide the research and make software technologies advance. I m not sure if I am an userop, but I feel part of that vanguard force , I want to be part of the future of free software and free networks. Comments? You can comment about this post on this pump.io thread.
Filed under: My experiences and opinion Tagged: Communities, Contributing to libre software, Debian, Developer motivations, English, free networks, Free Software, Freedom, innovation, MediaGoblin, Moving into free software, Project Management, selfhosting, sysadmin

Yesterday, my first term started as the Debian Project Leader. There s been a large number of emails congratulating me, and thanks to everyone who sent those. I d also like to thank Mehdi Dogguy and Gergely Nagy for running, and of course Lucas Nussbaum for his service over the past two years. Lucas also did a great handover, and so (I hope!) I m aware of most of the issues that are ongoing. As started previously, I ll keep my daily log of activities in /srv/leader/news/ on master.debian.org.

Review: The Long Way to a Small, Angry Planet, by Becky Chambers

Publisher:	CreateSpace
Copyright:	2014
ISBN:	1-5004-5330-7
Format:	Kindle
Pages:	503

The Wayfarer is a tunneling ship: one of the small, unremarked construction ships that help build the wormhole network used for interstellar transport. It's a working ship with a crew of eight (although most people would count seven and not count the AI). They don't all like each other particularly not the algaeist, who is remarkably unlikeable but they're used to each other. It's not a bad life, although a more professional attention to paperwork and procedure might help them land higher-paying jobs. That's where Rosemary Harper comes in. At the start of the book, she's joining the ship as their clerk: nervous, hopeful, uncertain, and not very experienced. But this is a way to get entirely away from her old life and, unbeknownst to the ship she's joining, her real name, identity, and anyone who would know her. Given that introduction, I was expecting this book to be primarily about Rosemary. What is she fleeing? Why did she change her identity? How will that past come to haunt her and the crew that she joined? But that's just the first place that Chambers surprised me. This isn't that book at all. It's something much quieter, more human, more expansive, and more joyful. For one, Chambers doesn't stick with Rosemary as a viewpoint character, either narratively or with the focus of the plot. The book may open with Rosemary and the captain, Ashby, as focal points, but that focus expands to include every member of the crew of the Wayfarer. We see each through others' eyes first, and then usually through their own, either in dialogue or directly. This is a true ensemble cast. Normally, for me, that's a drawback: large viewpoint casts tend to be either jarring or too sprawling, mixing people I want to read about with people I don't particularly care about. But Chambers avoids that almost entirely. I was occasionally a touch disappointed when the narrative focus shifted, but then I found myself engrossed in the backstory, hopes, and dreams of the next crew member, and the complex ways they interweave. Rosemary isn't the center of this story, but only because there's no single center. It's very hard to capture in a review what makes this book so special. The closest that I can come is that I like these people. They're individual, quirky, human (even the aliens; this is from more the Star Trek tradition of alien worldbuilding), complicated, and interesting, and it's very easy to care about them. Even characters I never expected to like. The Long Way to a Small, Angry Planet does have a plot, but it's not a fast-moving or completely coherent one. The ship tends to wander, even when the mission that gives rise to the title turns up. And there are a lot of coincidences here, which may bother you if you're reading for plot. At multiple points, the ship ends up in exactly the right place to trigger some revelation about the backstory of one of the crew members, even if the coincidence strains credulity. Similar to the algae-driven fuel system, some things one just has to shrug about and move past. On other fronts, though, I found The Long Way to be refreshingly willing to take a hard look at SF assumptions. This is not the typical space opera: humans are a relatively minor species in this galaxy, one that made rather a mess of their planet and are now refugees. They are treated with sympathy or pity; they're not somehow more flexible, adaptable, or interesting than the rest of the galaxy. More fascinatingly to me, humans are mostly pacifists, a cultural reaction to the dire path through history that brought them to their current exile. This is set against a backdrop of a vibrant variety of alien species, several of whom are present onboard the Wayfinder. The history and background of the other species are not, sadly, as well fleshed out as the humans, but each with at least a few twists that add interest to the story. But the true magic of this book, the thing that it has in overwhelming abundance, is heart. Not everyone in this book is a good person, but most of them are trying. I've rarely read a book full of so much empathy and willingness to reach out to others with open hands. And, even better, they're all nice in different ways. They bring their own unique personalities and approaches to their relationships, particularly the complex web of relationships that connects the crew. When bad things happen, and, despite the overall light tone, a few very bad things happen, the crew rallies like friends, or like chosen family. I have to say it again: I like these people. Usually, that's not a good sign for a book, since wholly likeable people don't generate enough drama. But this is one of the better-executed "protagonist versus nature" plots I've read. It successfully casts the difficulties of making a living at a hard and lonely and political job as the "nature" that provides the conflict. This is a rather unusual book. It's probably best classified as space opera, but it doesn't fit the normal pattern of space opera and it doesn't have enough drama. It's not a book about changing the universe; at the end of the book, the universe is in pretty much the same shape as we found it. It's not even about the character introduced in the first pages, or really that much about her dilemma. And it's certainly not a book about winning a cunning victory against your enemies. What it is, rather, is a book about friendships, about chosen families and how they form, about being on someone else's side, about banding together while still being yourself. It's about people making a living in a hard universe, together. It's full of heart, and I loved it. I'm unsurprised that The Long Way to a Small, Angry Planet had to be self-published via a Kickstarter campaign to find its audience. I'm also unsurprised that, once it got out there, it proved very popular and has now been picked up by a regular publisher. It's that sort of book. I believe it's currently out of print, at least in the US, as its new publisher spins up that process, but it should be back in print by late 2015. When that happens, I recommend it to your attention. It was the most emotionally satisfying book I've read so far this year. Rating: 9 out of 10

Today I upgraded my main web-host to the Jessie release of Debian GNU/Linux. I performed the upgraded by changing wheezy to jessie in the sources.list file, then ran:

apt-get update
apt-get dist-upgrade

For some reason this didn't upgrade my kernel, which remained the 3.2.x version. That failed to boot, due to some udev/systemd issues (lots of "waiting for job: udev /dev/vda", etc, etc). To fix this I logged into my KVM-host, chrooted into the disk image (which I mounted via the use of kpartx), and installed the 3.16.x kernel, before rebooting into that. All my websites seemed to be OK, but I made some changes regardless. (This was mostly for "neatness", using Debian packages instead of gems, and installing the attic package rather than keeping the source-install I'd made to /opt/attic.) The only surprise was the significant upgrade of the Net::DNS perl-module. Nothing that a few minutes work didn't fix. Now that I've upgraded the SSL-issue I had with redirections is no longer present. So it was a worthwhile thing to do.

For these who similarily suffer from having to use Googlemail at work. If anyone else has more of these, please do share. Deactivate the spamfilter The site admins can do that. Otherwise, you will have work-relevant eMails, for example from your own OTRS system, end up in Spam (where you don t see it, as their IMAP sucks) and deleted without asking 30 days later. (AIUI, the only way to get eMails actually deleted from Google ) Do not use their SMTP service Use your own outgoing MTA. This brings back the, well, not feature but should-have-been-granted-but-Google-doesn t-do-it-anyway that, when you write to a mailing list, you also get your own messages into your own INBOX. Calendars I have no solutions for this. I stopped using the Googlemail calendars because they didn t think it a problem that, when I accept an invitation in Kontact (KDEPIM as packaged in Debian sid), the organiser of the calendar item in the sender s calendar (for which I do not have write permissions) changes to me (so the actual meeting organiser cannot change anything afterwards) and/or calendar items get doubled. I now run a local uw-imapd (forward-ported to sid by means of a binNMU) for sent-mail folders etc. and a local iCalendar directory for calendars.

Like each month, here comes a report about the work of paid contributors to Debian LTS. Individual reports In February, 61 work hours have been equally split among 4 paid contributors. Their reports are available:

• Ben Hutchings did only 11.25h.
• Holger Levsen did only 4.5h during which he worked on the security tracker JSON output (see #761859) and on DLA-169-1 (fixing 2 CVE on axis).
• Rapha l Hertzog
• Thorsten Alteholz

The remaining hours of Ben and Holger have been redispatched to other contributors for April (during which Mike Gabriel joins the set of paid contributors). BTW, if you want to join the team of paid contributors, read this and apply! Evolution of the situation April has seen no change in terms of sponsored hours but we have two new sponsors in the pipe and May should hopefully have a few more sponsored hours. For the need of a LTS presentation I gave during the Mini-DebConf Lyon I prepared a small graph showing the evolution of the hours sponsored through Freexian:
The growth is rather slow and it will take years to reach our goal of funding the equivalent a full time position (176 hours per month). Even the intermediary goal of funding the equivalent of a half-time position (88h/month) is more than 6 months away given the current growth rate. But the perspective of Wheezy-LTS should help us to convince more organizations and hopefully we will reach that goal sooner. If you want to sponsor the project, check out this page. In terms of security updates waiting to be handled, the situation looks similar to last month: the dla-needed.txt file lists 40 packages awaiting an update (exactly like last month), the list of open vulnerabilities in Squeeze shows about 56 affected packages in total (2 less than last month). Thanks to our sponsors The new sponsors of the month are in bold (none this month).

• Gold sponsors:
• The Positive Internet
• Silver sponsors:
• AD&D David Ayers IntarS Austria
• Blablacar
• Domeneshop AS
• Evolix
• Gandi SAS
• Trollweb Solutions
• Universit Lille 3
• Bronze sponsors:
• Bitfolk LTD
• Daevel SARL
• FOSSter
• Freeside Internet Service
• Gree, Inc.
• Greenbone Networks GmbH
• Intevation GmbH
• Linuxhotel GmbH
• Megaspace Internet Services GmbH
• MyTux
• NUMLOG
• Nantes M tropole
• Offensive Security
• Seznam.cz, a.s.
• WinGo AG

No comment Liked this article? Click here. My blog is Flattr-enabled.

After using haproxy at work for some time I realized that it can be configured for a lot of things, for example: it knows about SNI (on ssl is the method we use to know what host the client is trying to reach so that we know what certificate to present and thus we can multiplex several virtual hosts on the same ssl IP:port) and it also knows how to make transparent proxy connections (the connections go through haproxy but the ending server will think they are arriving directly from the client, as it will see the client's IP as the source IP of the packages).With this two little features, which are available on haproxy 1.5 (Jessie's version has them all), I thought I could give it a try to substitute sslh with haproxy giving me a lot of possibilities that sslh cannot do.Having this in mind I thought I could multiplex several ssl services, not only https but also openvpn or similar, on the 443 port and also allow this services to arrive transparently to the final server. Thus what I wanted was not to mimic sslh (which can be done with haproxy) but to get the semantic I needed, which is similar to sslh but with more power and with a little different behaviour, cause I liked it that way.There is however one caveat that I don't like about this setup and it is that to achieve the transparency one has to run haproxy as root, which is not really something one likes :-( so, having transparency is great, but we'll be taking some risks here which I personally don't like, to me it isn't worth it.Anyway, here is the setup, it basically consists of a setup on haproxy but if we want transparency we'll have to add to it a routing and iptables setup, I'll describe here the whole setupHere is what you need to define on /etc/haproxy/haproxy.cfg:frontend ft_ssl bind 192.168.0.1:443 mode tcp option tcplog tcp-request inspect-delay 5s tcp-request content accept if req_ssl_hello_type 1 acl sslvpn req_ssl_sni -i vpn.example.net use_backend bk_sslvpn if sslvpn use_backend bk_web if req_ssl_sni -m found default_backend bk_ssh backend bk_sslvpn mode tcp source 0.0.0.0 usesrc clientip server srvvpn vpnserver:1194 backend bk_web mode tcp source 0.0.0.0 usesrc clientip server srvhttps webserver:443 backend bk_ssh mode tcp source 0.0.0.0 usesrc clientip server srvssh sshserver:22 An example of a transparent setup can be found here but lacks some details, for example, if you need to redirect the traffic to the local haproxy you'll want to use the xt_TPROXY, there is a better doc for that at squid's wiki. Anyway, if you are playing just with your own machine, like we typically do with sslh, you won't need the TPROXY power, as packets will come straight to your 443, so haproxy will be able to get the without any problem. The problem will come if you are using transparency (source 0.0.0.0 usesrc clientip) because then packets coming out of haproxy will be carrying the ip of the real client, and thus the answers of the backend will go to that client (but with different ports and other tcp data), so it will not work. We'll have to get those packets back to haproxy, for that what we'll do is mark the packages with iptables and then route them to the loopback interface using advanced routing. This is where all the examples will tell you to use iptables' mangle table with rules marking on PREROUTING but that won't work out if you are having all the setup (frontend and backends) in just one box, instead you'll have to write those rules to work on the OUTPUT chain of the mangle table, having something like this:*mangle :PREROUTING ACCEPT :INPUT ACCEPT :FORWARD ACCEPT :OUTPUT ACCEPT :POSTROUTING ACCEPT :DIVERT - -A OUTPUT -s public_ip -p tcp --sport 22 -o public_iface -j DIVERT -A OUTPUT -s public_ip -p tcp --sport 443 -o public_iface -j DIVERT -A OUTPUT -s public_ip -p tcp --sport 1194 -o public_iface -j DIVERT -A DIVERT -j MARK --set-mark 1 -A DIVERT -j ACCEPT COMMIT Take that just as an example, better suggestions on how to know what traffic to send to DIVERT are welcome. The point here is that if you are sending the service to some other box you can do it on PREROUTIING, but if you are sending the service to the very same box of haproxy you'll have to mark the packages on the OUTPUT chain.Once we have the packets marked we just need to route them, something like this will work out perfectly:ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 And that's all for this crazy setup. Of course, if, like me, you don't like the root implication of the transparent setup, you can remove the "source 0.0.0.0 usesrc clientip" lines on the backends and forget about transparency (connections to the backend will come from your local IP), but you'll be able to run haproxy with dropped privileges and you'll just need the plain haproxy.cfg setup and not the weird iptables and advanced routing setup.Hope you like the article, btw, I'd like to point out the main difference of this setup vs sslh, it is that I'm only sending the packages to the ssl providers if the client is sending SNI info, otherwise I'm sending them to the ssh server, while sslh will send ssl clients without SNI also to the ssl provider. If your setup mimics sslh and you want to comment on it, feel free to do it.

Review: Zero Sum Game, by S.L. Huang

Series:	Russell's Attic #1
Publisher:	S.L. Huang
Copyright:	2014
ISBN:	0-9960700-1-X
Format:	Kindle
Pages:	326

Cas Russell finds things for people, often involving some strategic violence. She belongs to that genre of action novel protagonists who have a rough code of ethics, but who don't exactly follow the law. Her friend Rio is much worse: a functioning sociopath with his own code of ethics, the derivation of which is a key plot point. When the book opens, Cas is on a mission to find a person and rescue them from a drug cartel not her normal mission, but her contact said she was referred by Rio. Oh, and Rio is currently hitting her in the face. This setup matches any number of present-day thrillers. The SFnal twist is that Cas is very good at numbers, in a completely unrealistic action hero way. (I found it unsurprising that Huang was inspired by the superhero genre; that's the right model to have in mind when reading about Cas.) She can calculate where bullets are going to go, knows just the right angle and velocity with which to throw things (and, even more notably, can get her body to do that), and, in one particularly memorable scene, sets up an eavesdropping sound concentrator by changing the angles of available random surfaces in the neighborhood, like trash cans. This ability is not without drawbacks. When she's not in a middle of a job, with something to focus on, she usually ends up drinking herself into a stupor to get her brain to stop working. But it's an extremely useful ability that requires the villains of this book go to great efforts to try to kill her. The plot starts out as fairly typical thriller material, involving threats and dire consequences to those Cas loves (or at least likes a lot) and an unfolding sense that this retrieval of a kidnapped woman is the tip of a very deep iceberg. The expected counterpart, a private investigator with a less casual attitude towards killing than Cas, shows up early on. (Rio does not play that role in the story. His role is much more complex.) But the superhero inspirations show up in the villains as well, in a twist that many on-line summaries spoil, but which I will leave unmentioned. Mostly, Zero Sum Game is a fast-moving story with lots of violence, lots of guns, shadowy conspiracies, and a hypercompetent protagonist. (Female, refreshingly, particularly since she doesn't fall in love with any of the other characters in the book.) It's a recipe for enjoyable brain candy, and I think that's the best attitude to bring to it. However, a couple of things set it apart for me. First, Cas spends quite a bit of time really thinking about her life and questioning her decisions, rather than just blithely enjoying her world of stress and violence. There's more introspection here than in the typical thriller plot, but she stops short of wallowing in angst and stays decisive. I liked that balance: a bit of inner discomfort, and a few hard ethical decisions, but not to the point of paralyzing her. Second, her relationship with Rio is something special. Rio himself is a character type that I've seen before in books like this, but I don't think I've seen the dynamic with a character like that handled this well before. I particularly liked that the focus of the book stayed on Cas, not on Rio, and the reader was encouraged to see that relationship as a reflection on Cas and her sense of internal ethics. Seeing Rio through Cas's eyes, and then seeing other characters react to him and react to their relationship, touched some chords that I really enjoyed reading. Unfortunately, the villains weren't as successful, at least for me. Partly this is a personal quirk: the nature of the threat posed (not revealed for about half the book) is a kind that I dislike reading about. It makes my skin crawl in a way that I don't enjoy. But, even putting that aside, the story ends on a very odd and disturbing anti-climax. It's clearly the first book of an ongoing series, and I hope later books will salvage this. (I certainly liked it well enough to read on.) But the ending left me unsettled and rather irritated at the author. Huang plays fair, and the ending is consistent with what we know by the end of the book, but I read this sort of action-thriller story for catharsis and the glory of competent people doing what they do well. I got deeply engrossed in this book and had a hard time putting it down. Both Cas and Rio are great characters, as are most of the supporting cast. I wish the ending wasn't quite as much of a letdown so that I could recommend it more strongly. But it's still a fun superhero thriller. If you're looking for something with unrealistic superpowers, a large helping of competence, and a high body count, this may be worth picking up. (And no, I don't know what the series title means. I know what the series title refers to, but I haven't yet figured out what connection it or the Axiom of Choice has to the plot.) Followed by Half Life. Rating: 7 out of 10

This post is almost a carbon copy of the Kallithea 0.2 release notes. Kallithea project have just released Kallithea 0.2. Kallithea is a Python-based GPLv3 source code management software for web-based hosting of Mercurial and Git repositories. This release brings many changes since 0.1. Notably, pull requests system have been improved, making contributing changes more robust. The visual appearance has also been refined: modern font-based symbolic icons from FontAwesome and GitHub Octicons have replaced the previously used bitmap icons, and revision graphs are now drawn with HiDPI display support. Kallithea now supports Mercurial 3.3 and Dulwich 0.9.9. Several fixes in the database code boosted performance significantly. We have also updated our Javascript libraries: jQuery, CodeMirror and Mergely. Javascript and CSS code have been cleaned up, with less and less code depending on Yahoo UI library. Since 0.1 we have discovered two security issues, so all users are strongly recommended to upgrade. For more details on these issues, please see our Security Notices page For more information, see https://kallithea-scm.org/ or http://docs.kallithea-scm.org/. The summary of the changes since 0.1 release is below. Bug fixes:

• forms: add CSRF protection to all forms CVE-2015-0276
• api: don t send internal data unless asked for it CVE-2015-0260
• middleware: fix force_tls typo in force_https backward compatibility code (Issue #44)
• rebranddb: update user extern_type and _name to internal instead of kallithea (Issue #38)
• git: do not fail if git is not installed and has been removed from __init__.py backends
• git: fix 'name' must be bytestring, not unicode error on browsing changesets
• git: close SubprocessIOChunker inputstream used for git (Issue #32)
• git: introduce hack for handling git failure on --depth cloning (Issue #33)
• git: fix version detection with unexpected version string (Issue #71)
• git: preserve line endings when calling out to git
• hg: fix clone from svn+http urls using hg-svn (Issue #72)
• migrate: add missing import (Issue #29)
• pullrequests: handle pull requests to empty repos (Issue #27)
• pullrequests: when creating PRs, fix handling of new ajax requests while other ajax requests are pending
• pullrequests: fix updates for PRs between different repositories
• pullrequests: preserve query parameters in pull request overview paging links
• compare: workaround unexpected Mercurial behaviour when finding ancestor of null rev
• javascript: remove trailing comma to please IE8 (Issue #39)
• download: fix for zip file downloads not being valid zips (Issue #35)
• helpers: fix crash on new users without email (Issue #28)
• middleware: change middleware ordering so we don t buffer hgweb output in redirect
• bin: give kallithea_config.py #!/usr/bin/env python so it can be executed directly
• bin: reintroduce cleanup-repos paster command
• bin: fix update_repoinfo command
• urlify: don t include trailing punctuation in markup
• urlify: markup of hashes in ()
• styling: don t loop on trying to load kallithea-logo.png after kallithea-logo.svg failed
• error: don t crash on /error/document urls
• error: don t crash on response without status
• admin: prevent deletion of users that are owners of a repository/user group (Issue #64)
• admin: make settings hook name field smaller so it is less likely to overlap with the value field
• admin: when scanning for repos to add, ignore removed groups
• auth_crowd: fix Admin Group Membership when using the Atlassian Crowd plugin
• auth: enable selecting PAM authentication module
• auth_ldap: fix user automatically activated with LDAP authentication (Issue #78)
• db: fix beaker cache key for user groups don t collide with users
• db: fix handling of (invalid) unicode email addresses
• db: don t use sql in on empty sets in compare avoid warning of bad performance
• db: to the extent it makes sense to have a one byte size limit, make it 255 instead of 256 to please MySQL (Issue #96)
• files: use current revision as default for show at (Issue #31)
• files: fix HTML injection via file names
• files: use forward slash as path separator for consistency
• files: don t use HTML encoding where not needed (Issue #74).
• feed: urlify and escape the commit description
• diff: fix diff of renamed files with whitespace in their names
• diff: don t split lines on bare CR as python splitlines do
• mergely: update Mergely to v3.3.9 (Issue #83)
• jquery: upgrade to 1.11.1
• codemirror: cleanup of integration and update to version 4.7
• setup: constrain dulwich to version 0.9.9, fixes CVE-2015-0838
• gists: add missing formencode.htmlfill import
• ini: Make celeryd.log.level default values uppercase lowercase might not be recognized and cause failure
• notification: handle unicode translations (Issue #95)
• repos: fix redirect after repo creation (and user/group add focus) (Issue #98)
• changeset: don t crash on malformed whitespace parameter return 400 Bad Request
• stats: fix display when no data ready yet
• stats: fix show more link

Features and improvements:

• ui: use font based icons FontAwesome and Git Hub Octicons from fontello
• ui: HiDPI / Retina support in graphs and other places
• paster: add install-iis command to automate IIS handler generation and update documentation
• email templates: send text/plain part as well
• docs: lots of improvements of structure and content and language
• translations: updated using WebLate
• admin: show links to added repos after repo scan
• admin: show links to created users and groups
• admin: when scanning for repos without remove , report the repos that could/would be removed
• setup: support Mercurial 3.3
• db: drop most lazy joins significant performance improvement
• db: add missing indices significant performance improvement
• db: don t commit session if cache invalidation didn t change anything improves performance
• comment: show full username in emails for PR/changeset comments
• changelog: added Settings to repo context menu
• utils: better display of ages >1 year when in short form
• remotes: add support to clone from Mercurial repositories over ssh
• hg: reimplement branch head listings more efficiently
• pullrequests: use same code and table ui for Repo Pull Requests and My Pull Requests
• pullrequests: add delete button to My Pull Requests overview
• pullrequests: on my pullrequests show the user s own vote as second column next to latest vote
• pullrequests: add option for adding repo owner as reviewer if none assigned
• pullrequests: make top pane one big form with meta data and one save button and preview changes in pane below more like when creating the PR
• pullrequests: make update functionality a separate submit button in the big form, automatically enabled when an update changeset is chosen
• pullrequests: show graph next to the list of changesets that are available for update
• pullrequests: for available changesets, also show changesets that not are descendants and thus not directly available but will be merged in
• pullrequests: show unrelated branch heads in the graph too show the problem instead of trying to explain it
• pullrequests: show pending reviewer changes with (not saved)
• pullrequests: show comment count in top of summary box
• diff: improved parsing, be more strict and fail with good error messages
• diff: handle GIT delta binary patches
• diff: show renamed files as that
• diff: make trailing spaces more visible make vertical line less transparent
• diff: show CR (\r) in diffs
• comment: warn before leaving page with open comments
• comment: clarify shown comment count (inline vs general)
• comment: extend next/previous links to global comments (Issue #91)

Changes (might influence workflows):

• i18n: en translation has been removed make sure to update your .ini files to have lang = (nothing)
• ini: turn show_revision_number off by default
• pullrequests: don t give changeset unreviewed status just because it is added to a PR
• pullrequests: tweak readable titles just append /_/ and branch name
• pullrequests: only author and admin can close
• pullrequests: don t add repo owner as reviewer in all PRs
• pullrequests: PRs between different repos should default to same destination/source branch
• pullrequests: use -- as separator in the automatic text when updating pull requests and is it to put latest updates first
• diff: don t group diffs based on the kind of change
• admin: set current user as owner when running repo scan
• changelog: change default view to 100 changesets, max to 2000
• date representation: use ISO8601 rather than a specific locale
• user groups: drop add/remove all buttons for member selection they are too dangerous and multi-select can do the same
• hg: drop support for hg web.hidden
• template: link the last revision in the file browser to the changeset

Cleanup:

• code: Fix typos and language in comments and code
• ui: Correct capitalization and improved English text in the UI
• javascript: use strict mode
• tags: rename old non-Kallithea tags and prefix them with 0.0 to show that Kallithea is superior
• javascript: more jQuery, less YUI, less clutter
• templates: better structure, less cut n paste coding
• html: various improvements
• css: various improvements
• exceptions: avoid silently catching exceptions especially avoid catch all
• tests: stability fixes
• tests: remove large unused test fixture
• whitespacecleanup.sh: run it regularly to ensure consistent whitespace and avoid spurious changes
• controllers: consistently use formfill.render with force_defaults=False
• gists: use allow_extra_fields and filter_extra_fields in gist forms like in most other forms
• json: always use kallithea.lib.compat for json and nothing else
• json: drop simplejson fallback and just use stdlib
• db: check for None instead of boolean false if that is what we mean

I am a console user. I really just started to use X11 again about two weeks ago, to sometimes test an a Qt application I am developing. I am not using Firefox or anything similar, all my daily work happens in shells and inside of emacs, in a console, not in X11. BRLTTY runs all the time, translating the screen content to something that my braille display can understand, sent out via USB. So the most important programs to me, are really emacs, and brltty. This is my desktop, that is up since 179 days. I am sorry, but this doesn't look right, not at all. I am not even beginning to talk about dbus-daemon and systemd. Why the HECK does packagekitd (which I definitely don't use actively) use up more then two hours of plain CPU time? What did it do, talk to NSA via an asymmetric cipher, or what?! I play music via sshfs, sometimes FLAC files. That barely consumed more CPU time then brltty, which is probably the most active daemon on my system, erm, it should be. I don't want to chime into any flamewars. I have accepted that we have systemd. But this does not look right! I remember, back in the good old days, emacs and brltty were my top CPU users.

I intend to make the extra effort of republishing my own research papers as Linked Research, i.e. in a form readable by humans (HTML5), but also embedding meta-data (as RDF) for machine processing. I ve started with Authoritative Linked Data descriptions of Debian source packages using ADMS.SW (a good candidate, as it deals with Linked Data ;). You ll notice the menu which helps select different style sheets for preparing clean printable versions, not far from the LaTeX output usually converted to PDF. I hope this will pave the way to more Linked Research, and less opaque publications. The only hassle at the moment is the conversion from LaTeX to HTML5 which I m doing manually, in Emacs + nxml-mode. Update: Check the preprint links in my publications page, for more papers.

Experiments Last week I discovered The Fan Club s Experiments page. It reminds me of the Debian Experiments community on Google+. I like the concept of trying out all kinds of different things and reporting back to the community on how it works. I ve been meaning to do more of that myself so this is me jumping in and reporting back on something I ve been poking at this weekend. Introducing Squashfs Squashfs is a read-only compressed filesystem commonly used on embedded devices, Linux installation media and remote file systems (as is done in LTSP). Typically, a system like tmpfs, unionfs or aufs is mounted over this read-only system to make it usable as a root filesystem. It has plenty of other use cases too but for the purposes of this entry we ll stick with those use cases in mind. It supports gzip, lzo and xz(lzma) as compression back-ends. It also supports block sizes from 4K up to 1M. Compression technique as well as block size can have major effects on both performance and file size. In most cases the defaults will probably be sufficient, but if you want to find a good balance between performance and space saving, then you ll need some more insight. My Experiment: Test effects of squashfs compression methods and block sizes I m not the first person to have done some tests on squashfs performance and reported on it. Bernhard Wiedemann and the Squashfs LZMA project have posted some results before, and while very useful I want more information (especially compress/uncompress times). I was surprised to not find a more complete table elsewhere either. Even if such a table existed, I probably wouldn t be satisfied with it. Each squashfs is different and it makes a big difference whether it contains already compressed information or largely uncompressed information like clear text. I d rather be able to gather compression ratio/times for a specific image rather than for one that was used for testing purposes once-off. So, I put together a quick script that takes a squashfs image, extracts it to tmpfs, and then re-compressing it using it all the specified compression techniques and block sizes and then uncompressing those same images for their read speeds. My Testing Environment For this post, I will try out my script on the Ubuntu Desktop 14.04.2 LTS squashfs image. It s a complex image that contains a large mix of different kinds of files. I m extracting it to RAM since I want to avoid having disk performance as a significant factor. I m compressing the data back to SSD and extracting from there for read speed tests. The SSD seems fast enough not to have any significant effect on the tests. If you have a slow storage, the results of the larger images (with smaller block sizes) may be skewed unfavourably. As Bernhard mentioned on his post, testing the speed of your memory can also be useful, especially when testing on different kinds of systems and comparing the results:

# dd if=/dev/zero of=/dev/null bs=1M count=100000
104857600000 bytes (105 GB) copied, 4.90059 s, 21.4 GB/s

CPU is likely to be your biggest bottleneck by far when compressing. mksquashfs is SMP aware and will use all available cores by default. I m testing this on a dual core Core i7 laptop with hyper-threading (so squashfs will use 4 threads) and with 16GB RAM apparently transferring around 21GB/s. The results of the squashfs testing script will differ greatly based on the CPU cores, core speed, memory speed and storage speed of the computer you re running it on, so it shouldn t come as a surprise if you get different results than I did. If you don t have any significant bottleneck (like slow disks, slow CPU, running out of RAM, etc) then your results should more or less correspond in scale to mine for the same image. How to Run It Create a directory and place the filesystem you d like to test as filesystem.squashfs, then:

$ apt-get install squashfs-tools 
$ wget https://raw.githubusercontent.com/highvoltage/squashfs-experiments/master/test-mksquashfs.sh
$ bash test-mksquashfs.sh

With the default values in that file, you ll end up with 18 squashfs images taking up about 18GB of disk space. I keep all the results for inspection, but I ll probably adapt/fix the script to be more friendly to disk space usage some time. You should see output that look something like this, with all the resulting data in the ./results directory.

* Setting up...
- Testing gzip
  * Running a squashfs using compression gzip, blocksize 4096
  * Running a squashfs using compression gzip, blocksize 8192
  * Running a squashfs using compression gzip, blocksize 16384
  ...
- Testing lzo
  * Running a squashfs using compression lzo, blocksize 4096
  * Running a squashfs using compression lzo, blocksize 8192
  * Running a squashfs using compression lzo, blocksize 16384
  ...
- Testing xz
  * Running a squashfs using compression xz, blocksize 4096
  * Running a squashfs using compression xz, blocksize 8192
  * Running a squashfs using compression xz, blocksize 16384
  ...
* Testing uncompressing times...
  * Reading results/squashfs-gzip-131072.squashfs...
  * Reading results/squashfs-gzip-16384.squashfs...
  * Reading results/squashfs-gzip-32768.squashfs...
  ...
* Cleaning up...

On to the Results The report script will output the results into CSV. Here s the table with my results. Ratio is percentage of the size of the original uncompressed data, CTIME and UTIME is compression time and uncompress time for the entire image.

Filename	Size	Ratio	CTIME	UTIME
squashfs-gzip-4096.squashfs	1137016	39.66%	0m46.167s	0m37.220s
squashfs-gzip-8192.squashfs	1079596	37.67%	0m53.155s	0m35.508s
squashfs-gzip-16384.squashfs	1039076	36.27%	1m9.558s	0m26.988s
squashfs-gzip-32768.squashfs	1008268	35.20%	1m30.056s	0m30.599s
squashfs-gzip-65536.squashfs	987024	34.46%	1m51.281s	0m35.223s
squashfs-gzip-131072.squashfs	975708	34.07%	1m59.663s	0m22.878s
squashfs-gzip-262144.squashfs	970280	33.88%	2m13.246s	0m23.321s
squashfs-gzip-524288.squashfs	967704	33.79%	2m11.515s	0m24.865s
squashfs-gzip-1048576.squashfs	966580	33.75%	2m14.558s	0m28.029s
squashfs-lzo-4096.squashfs	1286776	44.88%	1m36.025s	0m22.179s
squashfs-lzo-8192.squashfs	1221920	42.64%	1m49.862s	0m21.690s
squashfs-lzo-16384.squashfs	1170636	40.86%	2m5.008s	0m20.831s
squashfs-lzo-32768.squashfs	1127432	39.36%	2m23.616s	0m20.597s
squashfs-lzo-65536.squashfs	1092788	38.15%	2m48.817s	0m21.164s
squashfs-lzo-131072.squashfs	1072208	37.43%	3m4.990s	0m20.563s
squashfs-lzo-262144.squashfs	1062544	37.10%	3m26.816s	0m15.708s
squashfs-lzo-524288.squashfs	1057780	36.93%	3m32.189s	0m16.166s
squashfs-lzo-1048576.squashfs	1055532	36.85%	3m42.566s	0m17.507s
squashfs-xz-4096.squashfs	1094880	38.19%	5m28.104s	2m21.373s
squashfs-xz-8192.squashfs	1002876	34.99%	5m15.148s	2m1.780s
squashfs-xz-16384.squashfs	937748	32.73%	5m11.683s	1m47.878s
squashfs-xz-32768.squashfs	888908	31.03%	5m17.207s	1m43.399s
squashfs-xz-65536.squashfs	852048	29.75%	5m27.819s	1m38.211s
squashfs-xz-131072.squashfs	823216	28.74%	5m42.993s	1m29.708s
squashfs-xz-262144.squashfs	799336	27.91%	6m30.575s	1m16.502s
squashfs-xz-524288.squashfs	778140	27.17%	6m58.455s	1m20.234s
squashfs-xz-1048576.squashfs	759244	26.51%	7m19.205s	1m28.721s

Some notes:

• Even though images with larger block sizes uncompress faster as a whole, they may introduce more latency on live media since a whole block will need to be uncompressed even if you re just reading just 1 byte from a file.
• Ubuntu uses gzip with a block size of 131072 bytes on it s official images. If you re doing a custom spin, you can get improved performance on live media by using a 16384 block size with a sacrifice of around 3% more squashfs image space.
• I didn t experiment with Xdict-size (dictionary size) for xz compression yet, might be worth while sacrificing some memory for better performance / compression ratio.
• I also want stats for random byte reads on a squashfs, and typical per-block decompression for compressed and uncompressed files. That will give better insights on what might work best on embedded devices, live environments and netboot images (the above table is more useful for large complete reads, which is useful for installer images but not much else), but that will have to wait for another day.
• In the meantime, please experiment on your own images and feel free to submit patches.

FTP assistant Recently the NEW queue grew due to lots of uploads of new KDE software and several smaller node-packages. The KDE-stuff will be processed one after another, but the node-stuff seems to be rather strange. After the last discussion I was told that all those small packages can be accumulated into bigger chunks. I hope this discussion doesn t need to be repeated again Anyway, this month I marked 117 packages for accept and rejected 51 packages. Squeeze LTS This was my ninth month that I did some work for the Squeeze LTS initiative, started by Raphael Hertzog at Freexian. This month I got assigned a workload of 15.25h and I spent these hours to upload new versions of:

• [DLA 163-1] bind9 security update
• [DLA 166-1] libarchive security update
• [DLA 167-1] redcloth security update
• [DLA 170-1] mod-gnutls security update
• [DLA 171-1] libssh2 security update
• [DLA 181-1] xerces-c security update
• [DLA 182-1] batik security update
• [DLA 183-1] libxfont security update
• [DLA 184-1] binutils security update

Finally I was also able to upload the binutils package. Up to now, I got no complaints that something is not working anymore, so yeah, I seem to make it. The next big adventure will be a new upload of PHP. I already started with some patches, but it is still a good piece of work. I also uploaded update for DLA 164-1] unace security update, [DLA 168-1] konversation security update and [DLA 172-1] libextlib-ruby security update although no LTS sponsor indicated any interest. Other packages This month the severity of one bug in greylistd had been raised from normal to severe and such I had to upload a new version. Thanks to Andreas Beckmann for raising and for providing a patch. I also uploaded a new version of dict-elements and closed a bug related to reproducible builds. As I am the maintainer of libkeepalive, I got an email from Andreas Florath. He wanted to persuade me to create a package for his library libdontdie, which is rather similar to libkeepalive but has some improvements. As I promised to do some more packaging work, he didn t have to argue much and voila, there now is a new package libdontdie available. As the cooperation with him is really pleasant, I also created a package for his other project: pipexec. Donations Thanks alot to all donors, this month I got 30 in total. I really appreciate this and hope that everybody is pleased with my commitment. Don t hesitate to make suggestions for improvements.