A new major version 5.000 of Armadillo was released by Conrad a couple of days ago. Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab.
This version brings several new functions for sparse matrices, and automagically switches to 64-bit matrix indices in C++11 mode. See below for a short description of all the major changes based on the NEWS.Rd file.
This version is now on CRAN, as well as in Debian. The integration into CRAN was delayed by a few days as my testing had a shortcoming. We run full reverse-dependency checks against all 115 CRAN package depending in RcppArmadillo, and we even made full pre-release 0.4.999.1 and 0.5.000.0 shipped to the drat repo of the RcppCore GitHub organization (which was described in the previous release post). But a minor flaw in my setup made it miss how the change in indexing affected packages dfcomb and dfmta. My thanks to its maintainer Marie-Karelle Riviere for providing updates to her packages permitting this release to get onto CRAN. The testing process has been tightened and this should not happen again.
Changes in RcppArmadillo version 0.5.000.0 (2015-04-12)
Upgraded to Armadillo release Version 5.000 ("Ankle Biter")
added spsolve() for solving sparse systems of linear equations
added svds() for singular value decomposition of sparse matrices
added nonzeros() for extracting non-zero values from matrices
added handling of diagonal views by sparse matrices
expanded repmat() to handle sparse matrices
expanded join_rows() and join_cols() to handle sparse matrices
sort_index() and stable_sort_index() have been placed in the delayed operations framework for increased efficiency
use of 64 bit integers is automatically enabled when using a C++11 compiler
workaround for a bug in recent releases of Apple Xcode
workaround for a bug in LAPACK 3.5
Changes in RcppArmadillo version 0.4.999.1.0 (2015-04-04)
A few months ago, I wrote a long post about what I called back then the
"Gnocchi experiment".
Time passed and we me and the rest of the Gnocchi team continued to work on
that project, finalizing it.
It's with a great pleasure that we are going to release our first 1.0 version
this month, roughly at the same time that the integrated
OpenStack projects release their Kilo milestone. The
first release candidate numbered 1.0.0rc1
has been released this morning!
The problem to solve
Before I dive into Gnocchi details, it's important to have a good view of what
problems Gnocchi is trying to solve.
Most of the IT infrastructures out there consists of a set of resources. These
resources have properties: some of them are simple attributes whereas others
might be measurable quantities (also known as metrics).
And in this context, the cloud infrastructures make no exception. We talk about
instances, volumes, networks which are all different kind of resources. The
problems that are arising with the cloud trend is the scalability of storing
all this data and being able to request them later, for whatever usage.
What Gnocchi provides is a REST API that allows the user to manipulate
resources (CRUD) and their attributes, while preserving the history of those
resources and their attributes.
Gnocchi is fully documented and the
documentation is available online.
We are the first OpenStack project to require patches to integrate the
documentation. We want to raise the bar, so we took a stand on that. That's
part of our policy, the same way it's part of the OpenStack policy to require
unit tests.
I'm not going to paraphrase the whole Gnocchi documentation, which covers
things like installation (super easy), but I'll guide you through some basics
of the features provided by the REST API. I will show you some example so you
can have a better understanding of what you could leverage using Gnocchi!
Handling metrics
Gnocchi provides a full REST API to manipulate time-series that are called
metrics. You can easily create a metric using a simple HTTP request:
The archive_policy_name parameter defines how the measures that are being
sent are going to be aggregated. You can also define archive policies using the
API and specify what kind of aggregation period and granularity you want. In
that case , the low archive policy keeps 1 hour of data aggregated over 1
second and 1 day of data aggregated to 30 minutes. The functions used for
aggregations are the mathematical functions standard deviation, minimum,
maximum, and even 95th percentile. All of that is obviously customizable and
you can create your own archive policies.
If you don't want to specify the archive policy manually for each metric, you
can also create archive policy rule, that will apply a specific archive
policy based on the metric name, e.g. metrics matching disk.* will be high
resolution metrics so they will use the high archive policy.
It's also worth noting Gnocchi is precise up to the nanosecond and is not tied
to the current time. You can manipulate and inject measures that are years old
and precise to the nanosecond. You can also inject points with old timestamps
(i.e. old compared to the most recent one in the timeseries) with an archive
policy allowing it (see back_window parameter).
It's then possible to send measures to this metric:
These measures are synchronously aggregated and stored into the configured
storage backend. Our most scalable storage drivers for now are either based on
Swift or Ceph which are both
scalable storage objects systems.
It's then possible to retrieve these values:
As older Ceilometer users might notice here, metrics are only storing points
and values, nothing fancy such as metadata anymore.
By default, values eagerly aggregated using mean are returned for all supported
granularities. You can obviously specify a time range or a different
aggregation function using the aggregation, start and stop query
parameter.
Gnocchi also supports doing aggregation across aggregated metrics:
This computes the mean of mean for the metric
65071775-52a8-4d2e-abb3-1377c2fe5c55 and
9ccdd0d6-f56a-4bba-93dc-154980b6e69a starting on 6th October 2014 at 14:34
UTC.
Indexing your resources
Another object and concept that Gnocchi provides is the ability to manipulate
resources. There is a basic type of resource, called generic, which has very
few attributes. You can extend this type to specialize it, and that's what
Gnocchi does by default by providing resource types known for OpenStack such as
instance, volume, network or even image.
The resource is created with the UUID provided by the user. Gnocchi handles the
history of the resource, and that's what the revision_start and
revision_end fields are for. They indicates the lifetime of this revision of
the resource. The ETag and Last-Modified headers are also unique to this
resource revision and can be used in a subsequent request using If-Match or
If-Not-Match header, for example:
Which is useful to synchronize and update any view of the resources you might
have in your application.
You can use the PATCH HTTP method to modify properties of the resource, which
will create a new revision of the resource. The history of the resources are
available via the REST API obviously.
The metrics properties of the resource allow you to link metrics to a
resource. You can link existing metrics or create new ones dynamically:
Haystack, needle? Find!
With such a system, it becomes very easy to index all your resources, meter
them and retrieve this data. What's even more interesting is to query the
system to find and list the resources you are interested in!
You can search for a resource based on any field, for example:
That query will return a list of all resources owned by the user_idbd3a1e52-1c62-44cb-bf04-660bd88cd74d.
You can do fancier queries such as retrieving all the instances started by a
user this month:
And you can even do fancier queries than the fancier ones (still following?).
What if we wanted to retrieve all the instances that were on host foobar the
15th April and who had already 30 minutes of uptime? Let's ask Gnocchi to look
in the history!
I could also mention the fact that you can
search for value in metrics.
One feature that I will very likely include in Gnocchi 1.1 is the ability to
search for resource whose specific metrics matches some value. For example,
having the ability to search for instances whose CPU consumption was over 80%
during a month.
Cherries on the cake
While Gnocchi is well integrated and based on common OpenStack technology,
please do note that it is completely able to function without any other
OpenStack component and is pretty straight-forward to deploy.
Gnocchi also implements a full RBAC system based on the
OpenStack standard oslo.policy
and which allows pretty fine grained control of permissions.
There is also some work ongoing to have HTML rendering when browsing the API
using a Web browser. While still simple, we'd like to have a minimal Web
interface served on top of the API for the same price!
Ceilometer alarm subsystem supports Gnocchi with the Kilo release, meaning you
can use it to trigger actions when a metric value crosses some threshold. And
OpenStack Heat also supports auto-scaling your
instances based on Ceilometer+Gnocchi alarms.
And there are a few more API calls that I didn't talk about here, so don't
hesitate to take a peek at the
full documentation!
Towards Gnocchi 1.1!
Gnocchi is a different beast in the OpenStack community. It is under the
umbrella of the Ceilometer program, but it's one of the first projects that is
not part of the (old) integrated release. Therefore we decided to have a
release schedule not directly linked to the OpenStack and we'll release more
often that the rest of the old OpenStack components probably once every 2
months or the like.
What's coming next is a close integration with Ceilometer (e.g. moving the
dispatcher code from Gnocchi to Ceilometer) and probably more features as we
have more requests from our users. We are also exploring different backends
such as InfluxDB (storage) or MongoDB (indexer).
Stay tuned, and happy hacking!
Background
R relies on package repositories for initial installation of a package via install.packages(). A crucial second step is update.packages(): For all currently installed packages, a list of available updates is constructed or offered for either one-by-one or bulk updates. This keeps the local packages in sync with upstream, and provides for a very convenient way to obtain new features, bug fixes and other improvements. So by installing from a repository, we automatically have the ability to track the repository for updates.
Enter drat
Fairly recently, the drat package was added to the R ecosystem. It makes both aspects of package distribution easy: providing a package (if you are an author) as well as installing it (if you are a user). Now, because drat is at the same time source code (as it is also a package providing the functionality), and a repository (using what drat provides ib features), the "namespace" becomes a little cluttered.
But because a key feature of drat is the "one variable" unique identification via the GitHub, I opted to create a drat repository in the name of a new organisation: ghrr. This is a simple acronym for GitHub Hosted R Repository.
Use cases
We can outline several use case for packages in ghrr:
packages not published in a repo by their authors: I already use two like that:
fasttime, an impeccably fast parser for ISO datetimes by Simon Urbanek which was however never released into a repo by Simon, and
RcppR6, a very nice extension to both R6 (by Winston) and Rcpp, by Rich FitzJohn; similarly never released beyond GitHub;
packages possibly unsuitable for mainline repos:
Rblpapi is a great package by Whit Armstong and John Laing to which I have been contributing quite a bit of late. As it requires a free-to-use but not open source library and headers from Bloomberg, it will never make it to the mainline repository for R, but hosting it in ghrr is perfect as I can easily update several machines at work once I cut a new development release;
winsorize is a small package I needed a few weeks ago; it is spun out of robustHD but does not yet contain new code so Andreas and I are content to keep it in this drat for now;
packages in pre-relase mode:
RcppArmadillo where I announced both a release candidate before Armadillo 5.000 came out, as well as the actual RcppArmadillo 0.500.0.0 which is not (yet) on the mainline repository as two affected packages need a small update first. Users, however, can get RcppArmadillo already from the sibling Rcpp drat repo.
RcppToml is a new package I am currently working on implementing a toml parser based on cpptoml. It works, but it not quite ready for public announcements yet, and hence perfect for ghrr.
Going forward
ghrr is meant to be open. While anybody can open a drat repository, particularly on GitHub, it may be beneficial to somehow group packages. This is however not something that can be planned ex-ante: it may just happen if others who see similar benefits in this can in fact contribute. In that spirit, I strongly encourage pull requests.
Early on, I made my commit messages conform to a pattern of package version sha1 repourl to make code provenance of every commit very clear. Ideally, subsequent commits would conform to such a scheme, or replace it with a better one.
Some Resources
A few links to learn more about drat and ghrr:
Comments and questions via email or issue tickets are more than welcome. We hope that others find ghrr to be a useful tool for easy repository management and use via GitHub.
Since I just had cause to do this and it was harder than it needed to be due to bitrot in the public documentation I could find I thought I d write up how to get a modern bootloader onto older Atmel boards. These instructions are written for the AT91SAM9G20-EK though they should also apply to other Atmel boards of a similar generation.
These instructions are for booting from NAND since it s the default thing for the board, for this J34 should be fitted to enable the chip select and J33 disconnected to disable the dataflash. If there is something broken programmed into flash then booting while holding down BP4 should cause the second stage bootloader to trash itself and ensure the ROM bootloader puts itself into recovery mode, or just removing both J33 and J34 during power on will also ensure no second stage bootloader is found.
There is a ROM bootloader but it just loads a small region from the boot media and jumps into it which isn t enough for u-boot so there is a second stage bootloader called AT91Bootstrap. Download sources for current versions from github. If it (or a more sensibly written equivalent) is not yet merged upstream you ll need to apply this patch to get it to build with a modern compiler, or you could use an old toolchain (which you ll need in the next step anyway):
make at91sam9g20eknf_uboot_defconfig
make CROSS_COMPILE=arm-linux-gnueabihf-
producing binaries/at91sam9g20ek-nandflashboot-uboot-$ VERSION .bin. This configuration will look for u-boot at 0x40000 in the flash so we need a u-boot binary. Unfortunately modern compilers seem to produce binaries that fail with no output. This is normally a sign that they need the ABI specifying more clearly as above but I got fed up trying to spot what was missing so I used an old CodeSourcery 2013.05 release instead, hopefully future versions of u-boot will be able to build for this target with older toolchains. Grab a recent release (I used 2015.01) and build with:
cd $ UBOOT
make at91sam9g20ek_nandflash_defconfig
make CROSS_COMPILE=arm-linux-gnueabihf-
to get u-boot.bin.
These can then be flashed using the Atmel flashing tool SAM-BA. Start it and connect to the target (there is a Linux version, though it appears to rely on old versions of TCL/TK so if you get trouble starting it the easiest thing is to use the sacrificial Windows laptop you ve obtained in order to run the entertaining flashing tools companies sometimes provide without risking a real system, or in my case your shiny new laptop that you ve not yet installed Linux on). Start it then:
Connect SAM-BA to the device following the dialog on start.
Make sure you ve selected NandFlash in the memory type tabs in the center of the window.
Run the Enable NandFlash script.
Run the Erase All script.
Run the Send Boot File script and provide the at91bootstrap binary.
Set Send File Name to be the u-boot binary you built earlier and Address to be 0x40000.
Click Send File
Press the reset button
which should result in at91bootstrap output followed by u-boot output on the serial console. A similar process works for the AT91SAM9263, there the jumper you need is J19 (sadly u-boot does not flash pictures of cute animals or forested shorelines on the screen as the default Basic LCD Project 1.4 firmware does, I m not sure this full operating system thing is really delivering improved functionality).
only one week left until the jessie release. yay!
in the last weeks I didn't find many RC bugs that I could fix; still, here's
the short list; nice feature: I mostly help others or could build an work
done by others.
#669735 dpkg-www: "dpkg-www: transition towards Apache 2.4" sponsor QA upload from Jean-Michel Nirgal Vourg re
#669777 yocto-reader: "yocto-reader: transition towards Apache 2.4" sponsor NMU from Jean-Michel Nirgal Vourg re, upload to DELAYED/2
#669796 w3c-linkchecker: "w3c-linkchecker: transition towards Apache 2.4" cherry-pick periapt's commit from 2012 (pkg-perl)
#764284 testdisk: "[testdisk] after ntfs-3g upgrade, testdisk cannot be installed (Depends: error)" tag + sid to get it out of jessie RC bugs
#780629 libibverbs1: "libibverbs1: please add Breaks: libopenmpi1.3" upload NMU prepared by Andreas Beckmann, adding a Breaks, upload to DELAYED/2
#780729 pbuilder: "pbuilder must define PATH as in debian-policy (and as used on buildds)" downgrade
#782160 src:chrony: "chrony: Multiple issues: CVE-2015-1821 CVE-2015-1822 CVE-2015-1853" sponsor maintainer upload by Joachim Wiedorn
Review: The Long Way to a Small, Angry Planet, by Becky Chambers
Publisher:
CreateSpace
Copyright:
2014
ISBN:
1-5004-5330-7
Format:
Kindle
Pages:
503
The Wayfarer is a tunneling ship: one of the small, unremarked
construction ships that help build the wormhole network used for
interstellar transport. It's a working ship with a crew of eight
(although most people would count seven and not count the AI). They don't
all like each other particularly not the algaeist, who is remarkably
unlikeable but they're used to each other. It's not a bad life,
although a more professional attention to paperwork and procedure might
help them land higher-paying jobs.
That's where Rosemary Harper comes in. At the start of the book, she's
joining the ship as their clerk: nervous, hopeful, uncertain, and not very
experienced. But this is a way to get entirely away from her old life
and, unbeknownst to the ship she's joining, her real name, identity, and
anyone who would know her.
Given that introduction, I was expecting this book to be primarily about
Rosemary. What is she fleeing? Why did she change her identity? How
will that past come to haunt her and the crew that she joined? But that's
just the first place that Chambers surprised me. This isn't that book at
all. It's something much quieter, more human, more expansive, and more
joyful.
For one, Chambers doesn't stick with Rosemary as a viewpoint character,
either narratively or with the focus of the plot. The book may open with
Rosemary and the captain, Ashby, as focal points, but that focus expands
to include every member of the crew of the Wayfarer. We see each
through others' eyes first, and then usually through their own, either in
dialogue or directly. This is a true ensemble cast. Normally, for me,
that's a drawback: large viewpoint casts tend to be either jarring or too
sprawling, mixing people I want to read about with people I don't
particularly care about. But Chambers avoids that almost entirely. I was
occasionally a touch disappointed when the narrative focus shifted, but
then I found myself engrossed in the backstory, hopes, and dreams of the
next crew member, and the complex ways they interweave. Rosemary isn't
the center of this story, but only because there's no single center.
It's very hard to capture in a review what makes this book so special.
The closest that I can come is that I like these people. They're
individual, quirky, human (even the aliens; this is from more the
Star Trek tradition of alien worldbuilding), complicated, and
interesting, and it's very easy to care about them. Even characters I
never expected to like.
The Long Way to a Small, Angry Planet does have a plot, but it's
not a fast-moving or completely coherent one. The ship tends to wander,
even when the mission that gives rise to the title turns up. And there
are a lot of coincidences here, which may bother you if you're reading for
plot. At multiple points, the ship ends up in exactly the right place to
trigger some revelation about the backstory of one of the crew members,
even if the coincidence strains credulity. Similar to the algae-driven
fuel system, some things one just has to shrug about and move past.
On other fronts, though, I found The Long Way to be refreshingly
willing to take a hard look at SF assumptions. This is not the typical
space opera: humans are a relatively minor species in this galaxy, one
that made rather a mess of their planet and are now refugees. They are
treated with sympathy or pity; they're not somehow more flexible,
adaptable, or interesting than the rest of the galaxy. More fascinatingly
to me, humans are mostly pacifists, a cultural reaction to the dire path
through history that brought them to their current exile. This is set
against a backdrop of a vibrant variety of alien species, several of whom
are present onboard the Wayfinder. The history and background of
the other species are not, sadly, as well fleshed out as the humans, but
each with at least a few twists that add interest to the story.
But the true magic of this book, the thing that it has in overwhelming
abundance, is heart. Not everyone in this book is a good person, but most
of them are trying. I've rarely read a book full of so much empathy and
willingness to reach out to others with open hands. And, even better,
they're all nice in different ways. They bring their own unique
personalities and approaches to their relationships, particularly the
complex web of relationships that connects the crew. When bad things
happen, and, despite the overall light tone, a few very bad things happen,
the crew rallies like friends, or like chosen family. I have to say it
again: I like these people. Usually, that's not a good sign for a
book, since wholly likeable people don't generate enough drama. But this
is one of the better-executed "protagonist versus nature" plots I've read.
It successfully casts the difficulties of making a living at a hard and
lonely and political job as the "nature" that provides the conflict.
This is a rather unusual book. It's probably best classified as space
opera, but it doesn't fit the normal pattern of space opera and it doesn't
have enough drama. It's not a book about changing the universe; at the
end of the book, the universe is in pretty much the same shape as we found
it. It's not even about the character introduced in the first pages, or
really that much about her dilemma. And it's certainly not a book about
winning a cunning victory against your enemies.
What it is, rather, is a book about friendships, about chosen families and
how they form, about being on someone else's side, about banding together
while still being yourself. It's about people making a living in a hard
universe, together. It's full of heart, and I loved it.
I'm unsurprised that The Long Way to a Small, Angry Planet had to
be self-published via a Kickstarter campaign to find its audience. I'm
also unsurprised that, once it got out there, it proved very popular and
has now been picked up by a regular publisher. It's that sort of book. I
believe it's currently out of print, at least in the US, as its new
publisher spins up that process, but it should be back in print by late
2015. When that happens, I recommend it to your attention. It was the
most emotionally satisfying book I've read so far this year.
Rating: 9 out of 10
Today I upgraded my main web-host to the Jessie release of Debian GNU/Linux.
I performed the upgraded by changing wheezy to jessie in the sources.list file, then ran:
apt-get update
apt-get dist-upgrade
For some reason this didn't upgrade my kernel, which remained the 3.2.x version. That failed to boot, due to some udev/systemd issues (lots of "waiting for job: udev /dev/vda", etc, etc). To fix this I logged into my KVM-host, chrooted into the disk image (which I mounted via the use of kpartx), and installed the 3.16.x kernel, before rebooting into that.
All my websites seemed to be OK, but I made some changes regardless.
(This was mostly for "neatness", using Debian packages instead of gems,
and installing the attic
package rather than keeping the source-install I'd made to /opt/attic.)
The only surprise was the significant upgrade of the Net::DNS perl-module. Nothing that a few minutes work didn't fix.
Now that I've upgraded the SSL-issue I had with redirections is no longer present. So it was a worthwhile thing to do.
This year's DPL campaign is over and voting period is also almost over. Many did not vote yet and they really should consider doing so. This is meant as a reminder for them. If you didn't have time to dig into debian-vote's archives and read questions/answers, here is the list with links to candidates' replies:
What is your perception of DebConf and its organisation? (mehdi, neilm, algernon)
What is your perception of fundraising in and around Debian? (neilm, algernon). I didn't answer to this question explicitly as I mainly agree with algernon and a similar topic has been discussed/explained elsewhere (platforms, other questions, ...)
Also, as a reminder, you can find candidates platforms here.
Compared to past years, we had a comparable number of questions. All questions did not start big threads as it used to be the case sometimes in the past :-) The good side of this is that we are trolling DPL candidates less than we used to do :-P
Now, if you still didn't vote, it is really time to do so. The voting period ends on Tuesday, April 14th, 23:59:59 UTC, 2015. You have only a few hours left!
The inline package facilitates writing code in-line in simple string expressions or short files. The package is mature and in maintenance mode: Rcpp used it greatly for several years but then moved on to Rcpp Attributes so we have a much limited need for extensions to inline.
But we now have a new inline version 0.3.14. It brings both a few minor code updates since the last release in 2013, but also new extensions to both support Fortran better (for several flavours including f95) and to make working with dynamic library files easier. These were contributed by long-time R author Karline Soetaert who thereby became a package co-author. Also, the package moved to GitHub sometime last year and now lives in this repo.
See below for a detailed list of changes extracted from the NEWS file.
Changes in inline version 0.3.14 (2015-04-11)
Removed call to Rcpp::RcppLdFlags() which is no longer needed
With move of repository to GitHub, added a .travis.yml file and corresponding entry in .Rbuildignore
Replaced calls to require() with calls to requireNamespace(); also updated one call
Much improved support for Fortran and Fortran95 thanks to Karline Soetaert who became a package co-author
New helper functions writeDynLib and readDynLib as well as new methods print and code (also by Karline)
We're about three months away from relocating from Edinburgh to Newcastle and some of the immediate panic has worn off.
We've sold our sofa, our spare sofa, etc, etc. We've bought a used dining-table, chairs, and a small sofa, etc. We need to populate the second-bedroom as an actual bedroom, do some painting, & etc, but things are slowly getting done.
I've registered myself as a landlord with the city council, so that I can rent the flat out without getting into trouble, and I'm in the process of discussing the income possabilities with a couple of agencies.
We're still unsure of precisely which hospital, from the many choices, in Newcastle my wife will be stationed at. That's frustrating because she could be in the city proper, or outside it. So we need to know before we can find a place to rent there.
Anyway moving? It'll be annoying, but we're making progress. Plus, how hard can it be?
VLAN Expansion
I previously had a /28 assigned for my own use, now I've doubled that to a /27 which gives me the ability to create more virtual machines and run some SSL on some websites.
Using SNI I've actually got the ability to run SSL almost all sites. So I configured myself as a CA and generated a bunch of certificates for myself. (Annoyingly few tutorials on running a CA mentioned SNI so it took a few attempts to get the SAN working. But once I got the hang of it it was simple enough.)
So if you have my certificate authority file installed you can browse many, many of my interesting websites over SSL.
SSL
I run a number of servers behind a reverse-proxy. At the moment the back-end is lighttpd. Now that I have SSL setup the incoming requests hit the proxy, get routed to lighttpd and all is well. Mostly.
However redirections break. A request for:
https://lumail.org/docs
Gets rewritten to:
http://lumail.org/docs/
That is because lighttpd generates the redirection and it only sees the HTTP connection. It seems there is mod_extforward which should allow the server to be aware of the SSL - but it doesn't do so in a useful fashion.
So right now most of my sites are SSL-enabled, but sometimes they'll flip to naked and unprotected. Annoying.
I don't yet have a solution..
This post is almost a carbon copy of the Kallithea 0.2 release notes.
Kallithea project have just released Kallithea 0.2. Kallithea is a Python-based GPLv3 source
code management software for web-based hosting of Mercurial and Git repositories.
This release brings many changes since 0.1. Notably, pull requests system
have been improved, making contributing changes more robust. The visual
appearance has also been refined: modern font-based symbolic icons from
FontAwesome and GitHub Octicons have replaced the previously used bitmap
icons, and revision graphs are now drawn with HiDPI display support.
Kallithea now supports Mercurial 3.3 and Dulwich 0.9.9. Several fixes in
the database code boosted performance significantly.
We have also updated our Javascript libraries: jQuery, CodeMirror and Mergely.
Javascript and CSS code have been cleaned up, with less and less code depending
on Yahoo UI library.
Since 0.1 we have discovered two security issues, so all users are strongly
recommended to upgrade. For more details on these issues, please see our
Security Notices page
For more information, see https://kallithea-scm.org/ or
http://docs.kallithea-scm.org/.
The summary of the changes since 0.1 release is below.
Bug fixes:
forms: add CSRF protection to all forms CVE-2015-0276
api: don t send internal data unless asked for it CVE-2015-0260
Several weeks ago we introduced the drat package. Its name stands for drat R Archive Template, and it helps with easy-to-create and easy-to-use repositories for R packages. Two early blog posts describe drat: First Steps Towards Lightweight Repositories, and Publishing a Package, and since the previous release, a a guest post on drat was also added to the blog.
Several people have started to use drat to publish their packages, and this is a very encouraging sign. I also created a new repository and may have more to say about this in another post once I get time to write something up.
Today version 0.0.3 arrived on CRAN. It rounds out functionality, adds some fixes and brings more documentation:
git support via git2r is improved; it is still optional (ie commit=TRUE is needed when adding packages) but plan to expand it
several small bugs got fixed, including issues #9 and #7,
The work on the vignettes is clearly in progress as Colin's guest post isn't really finished yet, and I am not too impressed with how the markdown is rendered at CRAN so some may still become pdf files instead.
Courtesy of CRANberries, there is a comparison to the previous release. More detailed information is on the drat page.
As it is nearly time to release Debian 8 (codename jessie), I've
uploaded a new version of the Linux kernel to unstable which I
hope will be the version to go into the initial release (8.0).
The changes from the current version in testing are mostly bug
fixes:
Please test this new version (which should be on mirrors within the
next 24 hours) and report any regressions you spot.
It's now too late to add new hardware support for Debian 8.0, but
we'll probably be able to improve it in subsequent point releases.
So, please also report driver changes that should be backported from
later kernel versions to improve hardware support, with severity of
'important'. If you can provide precise information about which
upstream commits are needed, that makes things easier for us, and
you should add the 'patch' tag.
Winter time is a good time for some nature observation. Yesterday I had a woodpecker (picture) in front of my kitchen window. During the recent weeks there were long-tailed tits, a wren and other rarely seen birds. So I thought, it might be a good idea to capture some of these events :) I still own a Logitech C910 USB camera which allows HD video capturing up to 1080p. So I checked the web for some software that would begin video capturing in the case of motion detection and found motion, already available for Debian users. So I gave it a try. I tested all available resolutions of the camera together with the capturing results. I found that the resulting framerate of both the live stream and the captured video is highly depending on the resolution and some few configuration options. Below is a summary of my tests and the results I've achieved so far.
Logitech C910 HD cameraJust a bit of data regarding the camera. AFAIK it allows for fluent video streams up to 720p.
$ dmesg [..] usb 7-3: new high-speed USB device number 5 using ehci-pci usb 7-3: New USB device found, idVendor=046d, idProduct=0821 usb 7-3: New USB device strings: Mfr=0, Product=0, SerialNumber=1 usb 7-3: SerialNumber: 91CF80A0 usb 7-3: current rate 0 is different from the runtime rate 16000 usb 7-3: current rate 0 is different from the runtime rate 32000 uvcvideo: Found UVC 1.00 device (046d:0821) input: UVC Camera (046d:0821) as /devices/pci0000:00/0000:00:1a.7/usb7/7-3/7-3:1.2/input/input17
$ lsusb [..] Bus 007 Device 005: ID 046d:0821 Logitech, Inc. HD Webcam C910 [..]
$ v4l2-ctl -V -d /dev/video1 Format Video Capture: Width/Height : 1280/720 Pixel Format : 'YUYV' Field : None Bytes per Line: 2560 Size Image : 1843200 Colorspace : SRGB
Also the uvcvideo kernel module is loaded and the user in question is part of the video group. Installation and startInstallation of the software is as easy as always:
apt-get install motion
It is possible to run the software as a service. But for testing, I copied /etc/motion/motion.conf to ~/.motion/motion.conf, fixed its permissions (you cannot copy the file as user - it's not world readable) and disabled the daemon mode.
daemon off
Note that in my case the correct device is /dev/video1 because the laptop has a built-in camera, that is /dev/video0. Also the target directory should be writeable by my user:
videodevice /dev/video1 target_dir ~/Videos
Then running motion from the command line ...
$ motion [..] [0] [NTC] [ALL] motion_startup: Motion 3.2.12+git20140228 Started [..] [1] [NTC] [ALL] motion_init: Thread 1 started , motion detection Enabled [0] [NTC] [ALL] main: Thread 1 is device: /dev/video1 input -1 [1] [NTC] [VID] v4l2_get_capability: ------------------------ cap.driver: "uvcvideo" cap.card: "UVC Camera (046d:0821)" cap.bus_info: "usb-0000:00:1a.7-1" cap.capabilities=0x84000001 ------------------------ [1] [NTC] [VID] v4l2_get_capability: - VIDEO_CAPTURE [1] [NTC] [VID] v4l2_get_capability: - STREAMING [1] [NTC] [VID] v4l2_select_input: name = "Camera 1", type 0x00000002, status 00000000 [1] [NTC] [VID] v4l2_select_input: - CAMERA [..] [1] [NTC] [ALL] image_ring_resize: Resizing pre_capture buffer to 1 items
... will begin to capture motion detection events and also output a live stream. CTRL+C will stop it again. Live streamThe live stream is available by pointing the browser to localhost:8081. However, the stream seems to run at 1 fps (frames per second) and indeed does. The stream gets more quality by this configuration:
stream_motion on stream_maxrate 100
The first option is responsible, that the stream only runs at one fps if there is no motion detection event. Otherwise the framerate increases to its maximum value, which is either the one given by stream_maxrate or the camera limit. The quality of the stream picture can be increased a bit further too by increasing the stream_quality value. Because I neither need the stream nor the control feed I disabled both:
stream_port 0 webcontrol_port 0
Picture capturingBy default there is video and picture capturing if a motion event is detected. I'm not interested in these pictures, so I turned them off:
output_pictures off
FYI: If you want a good picture quality, then the value of quality should very probably be increased. Video capturingThis is the really interesting part :) Of course if I will "shoot" some birds (with the camera), then a small image of say 320x240 pixels is not enough. The camera allows for a capture resolution up to 1920x1080 pixels (1080p). It is advertised for fluent video streams up to 720p (1280x720 pixels). So I tried the following resolutions: 320x240, 640x480, 800x600, 640x360 (360p), 1280x720 (720p) and 1920x1080 (1080p). These are easily configured by the width and height variables. For example the following configures motion for 1280x720 pixels (720p):
width 1280 height 720
The result was really disappointing. No event is captured with more then 20 fps. At higher resolutions the framerate drops even further and at the highest resolution of 1920x1080 pixels, the framerate is only two(!) fps. Also every created video runs much too fast and even faster by increasing the framerate variable. Of course its default value of 2 (fps) is not enough for fluent videos. AFAIK the C910 can run with 30 fps at 1280x720 pixels. So increasing the value of framerate, the maximum framerate recorded, is a must-do. (If you wanna test yourself, check the log output for the value following event_new_video FPS.)The solution to the issue, that videos are running too fast, however is to increase the pre_capture value, the number of pre-captured (buffered) pictures from before motion was detected. Even small values like 3..5 result in a distinctive improvement of the situation. Though increasing the value further didn't have any effect. So the values below should almost get the most out of the camera and result in videos in normal speed.
framerate 100 pre_capture 5
Videos in 1280x720 pixels are still captured with 10 fps and I don't know why. Running guvcview, the same camera allows for 30 fps in this resolution (even 60 fps in lower resolutions). However, even if the framerate could be higher, the resulting video runs fluently. Still the quality is just moderate (or to be honest, still disappointing). It looks "pixelated". Only static pictures are sharp. It took me a while to fix this too, because I first thought, the reason is the camera or missing hardware support. It is not :) The reason is, that ffmpeg is configured to produce a moderate(?)-quality video. The relevant variables are ffmpeg_bps and ffmpeg_variable_bitrate. I got the best results just changing the latter:
ffmpeg_variable_bitrate 2
Finally the resulting video quality is promising. I'll start with this configuration setting up an observation camera for the bird feeding ground.There is one more tweak for me. I got even better results by enabling the auto_brightness feature.
auto_brightness on
Complete configurationSo the complete configuration looks like this (only those options changed to the original config file)
daemon off videodevice /dev/video1 width 1280 height 720 framerate 100 auto_brightness on ffmpeg_variable_bitrate 2 target_dir /home/user/Videos stream_port 0 #8081 stream_motion on stream_maxrate 100 webcontrol_port 0 #8080
Hello
In my opinion, creating and maintaining Debian copyright file is the most boring task required to create a Debian package. Unfortunately, this file is also one of the most important of a package: it specifies some legal aspect regarding the use of the software.
Debian copyright file is scrutinized by ftp masters gatekeepers when accepting a new package in Debian project: this file must accurately describe the copyright and licenses of all files of a source package, preferably using a specific syntax. (Kudos to the ftp-masters team: reading copyright files must be even more boring than writing them).
The content of the copyright file must reflect accurately the license of all files. This license is often specified in the comments of a source files. The licencecheck command is able to scan sources files and reports the copyright and licenses declared in there. But it does not summarize this information: a copyright line is generated for each file of a package.
licensecheck2dep5 (provided by cdbs package as /usr/lib/cdbs/licensecheck2dep5) does better: the output of licensecheck is consolidated and translated in Debian copyright format. The result is better, but must be heavily edited to be reviewable by ftp-masters team.
The new update subcommand of cme (available with libconfig-model-dpkg-perl 2.061 currently in experimental) goes further than licensecheck2deb:
copyright are coalesced when possible (i.e. 2001,2002,2003-2005 is changed to 2001-2005)
file entries same copyright owner and license are grouped, group of files may be represented with a wild card ( * )
license text is filled with actual text for the most popular licenses
For instance, here s the (slightly edited) output of cme run for pan package starting without debian/copyright file:
$ cme update dpkg-copyright -quiet
Adding dummy license text for license public-domain
for path pan/general/sorted-vector.h
Adding dummy license text for license BSD-2-clause
for path pan/usenet-utils/MersenneTwister.h
$ cat debian/copyright
Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Files: *
Copyright: 1994-2001, Frank Pilhofer. The author may
License: GPL-2+
Files: pan/*
Copyright: 2002-2007, Charles Kerr
License: GPL-2
Files: pan/data/cert-store.cc
pan/data/cert-store.h
Copyright: 2011, Heinrich M ller
2002-2006, Charles Kerr
License: GPL-2
Files: pan/general/e-util.cc
pan/general/e-util.h
Copyright: 2000, 2001, Ximian, Inc
License: LGPL-2
Files: pan/general/locking.h
pan/general/worker-pool.cc
pan/general/worker-pool.h
Copyright: 2007, Calin Culianu
2002-2007, Charles Kerr
License: LGPL-2+
Files: pan/general/sorted-vector.h
Copyright: 2002, Martin Holzherr (holzherr@infobrain.com).
License: public-domain
Please fill license public-domain from header
of pan/general/sorted-vector.h
[ about 100 more lines including license text for Zlib and several
GPL licenses ]
This is a good start, but some modifications must be applied to get a correct license file:
add missing upstream information (Upstream-Name, Upstream-Contact and Source items)
remove irrelevant text from some copyright owner (e.g. remove The author may from Files: * entry).
add some missing license text (e.g. include text from sorted-vector.h comments to specify upstream author s version of public-domain)
These modifications can be done:
directly after update by running cme update dpkg-copyright --edit
after update by running cme edit dpkg-copyright
with your favorite editor
This post has mentioned creation of Debian copyright file, but does not address the problem of updating an existing copyright file when packaging a new version of a software. This will be the subject of a next post.
I hope this new feature of cme will save hours of work for Debian packagers. As usual comments and suggestions are welcome
All the best
This was my fourth month working on Debian LTS. I was assigned 14.5
hours
by Freexian's
Debian LTS initiative, but I only worked 11.5 as I had a week's
holiday and then was ill for part of this week.
eglibc
My first task was to complete the eglibc update that I began at the
end of February. There were a few unexpected failures in the
regression test suite, but on inspection these turned out to be
quirks of the build environment:
A test of
the nice(2)
function failed because I ran the build itself
with nice(1).
I should probably use a cgroup to reduce its priority instead;
this time I re-ran without using either.
Several tests of cancellation failed because they rely on a socket
write blocking due to filling up the buffer, while the kernel's
minimum socket buffer size has increased such that the write isn't
big enough to block. (The host kernel was Linux 3.16 from
jessie.) I found and applied a fix for this from the (glibc)
upstream repository.
With those changes, the regression test results are the same as
for previous package versions. (There is still one 'unexpected'
failure, but I didn't investigate because it is not a regression
within squeeze.)
I reviewed the backported patches again to check as well as I could
that they did not depend on other upstream changes, and they did not
seem to. I also received positive feedback from further testing - I
can't find the message now, but I think it was that the Univention
application test suite passed with the updated eglibc package
installed.
With that confirmation and no regressions reported, I uploaded
eglibc 2.11.3-4+deb6u5 and
issued DLA
165-1.
freetype
A large number of vulnerabilities in font file parsing in Freetype
were reported by
Mateusz Jurczyk. I think
these were less critical in squeeze than in wheezy, because while
current web browsers use Freetype to render untrusted 'web fonts'
we don't support any web browsers in squeeze LTS. But it seemed
like they ought to be fixed anyway, and it was not too hard to
backport the patches included in the wheezy security update.
Freetype doesn't have a regression test suite and I didn't have
samples of the broken fonts, so I came up with some ad hoc tests for
regressions. I viewed several of each of the affected font formats
with the ftview demo application. I then wrote a script to match up
the dependencies of a package (in this case, libfreetype6) with
those used by Freexian's customers; the top results were fontconfig,
xfonts-utils and imagemagick. So I ran some basic tests against
each of the affected formats with each of these (fc-list,
mkfontscale, and
the simple
text label recipe for ImageMagick) and found no regression.
Uploaded freetype 2.4.2-2.1+squeeze5 and
issued DLA
185-1.
This seems as good a day as any to mention that I am a founding member of
ArchiveTeam.
Way back, when Geocities was closing down, I was one of a small
rag-tag group who saved a copy of most of it. That snapshot has since
generated more publicity than most other projects I've worked on.
I've heard many heartwarning stories of it being the only remaining
copy of baby pictures and writings of deceased friends, and so on.
It's even been the subject of
serious academic study as outlined in this talk,
which is pretty awesome.
I'm happy to let this guy be the public face of ArchiveTeam in
internet meme-land. It's a 0.1% project for me, and has grown into a
well-oiled machine, albeit one that shouldn't need to exist. I only get
involved these days when there's another crazy internet silo fire drill
and/or I'm bored.
(Rumors of me being the hand model for ArchiveTeam are, however,
unsubstantiated.)
Relevant XKCD
It appears that this year's MiniDebconf in Lyon, France is going to have a very special guest. After enjoying so much the Q&A held at Debconf14, Linus Torvalds has decided to make a quick detour from a trip to Europe to bring the opportunity for another session with the Debian community.
The event has yet to appear in the MiniDebconf agenda. Rumor has it that the hurds of fans that tend to attend his events pose a logistics and safety challenge, reason why the event might only appear in the agenda a couple of days before the event. Lyon being such an easy place to get by train, flight, car, and even by boat, it is understandable and should be expected that a great number of people attend the minidebconf only for his very session.
If you have not yet registered, I'd recommend you to do so now and to book your travel and accommodation ASAP, before everything is overpriced due to high demand.
My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me.
Debian LTS
This month I have been paid to work 15.25 hours on Debian LTS. In that time I did the following:
CVE triage: I pushed 37 commits to the security tracker and contacted 20 maintainers about security issues affecting their packages.
I started a small helper script based on the new JSON output of the security tracker (see #761859 for details). It s not ready yet but will make it easier to detect issues where the LTS team lags behind the security team, and other divergences like this and will speed up future CVE triage work (once done).
I sent DLA-174-1 (tcpdump update fixing 3 CVE) after having received a debdiff from the Romain Fran oise.
In my last Freexian LTS report, I mentioned briefly that it would be nice to have a logo for the LTS project. Shortly after I got a first logo prepared by Damien Escoffier and a few more followed: they are available on a wiki page (and the logo you see above is from him!). Following a suggestion of Paul Wise, I registered the logo request on another wiki page dedicated to artwork requests. That kind of collaboration is awesome! Thanks to all the artists involved in Debian.
Debian packaging
Django. This month has seen no less than 3 upstream point releases packaged for Debian (1.7.5, 1.7.6 and 1.7.7) and they have been accepted by the release team into Jessie. I m pleased with this tolerance as I have argued the case for it multiple times in the past given the sane upstream release policy (bugfix only in a given released branch).
Python code analysis. I discovered a few months ago a tool combining the power of multiple Python code analysis tools: it s prospector. I just filed a Request for Package for it (see #781165) and someone already volunteered to package it, yay \o/
update-rc.d and systemd. While working on a Kali version based on Jessie, I got hit by what boils down to a poor interaction between systemd and update-rc.d (see #746580) and after some exchanges with other affected users I raised the severity to serious as we really ought to do something about it before release. I also opened #781155 on openbsd-inetd as its usage of inetd.service instead of openbsd-inetd.service (which is only provided as a symlink to the former) leads to multiple small issues.
Misc
Debian France. The general assembly is over and the new board elected its new president: it s now official, I m no longer Debian France s president. Good luck to Nicolas Dandrimont who took on this responsibility.
Salt s openssh formula. I improved salt s openssh formula to make it possible to manage the /etc/ssh/ssh_known_hosts file referencing the public SSH keys of other managed minions.
Tendenci.com. I was looking for a free software solution to handle membership management of a large NPO and I discovered Tendenci. It looked very interesting feature wise and written with a language/framework that I enjoy (Python/Django). But while it s free software, there s no community at all. The company that wrote it released it under a free software license and it really looks like that they did intend to build a community but they failed at it. When I looked their development forums were web-based and mostly empty with only initial discussion of the current developers and no reply from anybody there s also no mention of an IRC channel or a mailing list. I sent them a mail to see what kind of collaboration we could expect if we opted for their software and got no reply. A pity, really.
What free software membership management solution would you use when you have more than 10000 members to handle and when you want to use the underlying database to offer SSO authentication to multiple external services?
Thanks
See you next month for a new summary of my activities.
Pictures from the CDSW sessions in Spring 2014
After two successful rounds in 2014, I m helping put on another round of the Community Data Science Workshops. Last year, our 40+ volunteer mentorss taught more than 150 absolute beginners the basics of programming in Python, data collection from web APIs, and tools for data analysis and visualization and we re still in the process of improving our curriculum and scaling up.
Once again, the workshops will be totally free of charge and open to anybody. Once again, they will be possible through the generous participation of a small army of volunteer mentors.
We ll be meeting for four sessions over three weekends:
A new major version 5.000 of Armadillo was released by Conrad a couple of days ago. Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab.
This version brings several new functions for sparse matrices, and automagically switches to 64-bit matrix indices in C++11 mode. See below for a short description of all the major changes based on the 
Gnocchi provides a full REST API to manipulate time-series that are called
metrics. You can easily create a metric using a simple HTTP request:
Another object and concept that Gnocchi provides is the ability to manipulate
resources. There is a basic type of resource, called generic, which has very
few attributes. You can extend this type to specialize it, and that's what
Gnocchi does by default by providing resource types known for OpenStack such as
instance, volume, network or even image.
only one week left until the jessie release. yay!
in the last weeks I didn't find many RC bugs that I could fix; still, here's
the short list; nice feature: I mostly help others or could build an work
done by others.
Today I upgraded my main web-host to the Jessie release of Debian GNU/Linux.
I performed the upgraded by changing wheezy to jessie in the sources.list file, then ran:
This year's DPL campaign is over and voting period is also almost over. Many did not vote yet and they really should consider doing so. This is meant as a reminder for them. If you didn't have time to dig into 
As it is nearly time to release Debian 8 (codename jessie), I've
uploaded a new version of the Linux kernel to unstable which I
hope will be the version to go into the initial release (8.0).
The changes from the current version in testing are mostly bug
fixes:
This seems as good a day as any to mention that I am a founding member of
ArchiveTeam.
I'm happy to let this guy be the public face of ArchiveTeam in
internet meme-land. It's a 0.1% project for me, and has grown into a
well-oiled machine, albeit one that shouldn't need to exist. I only get
involved these days when there's another crazy internet silo fire drill
and/or I'm bored.
(Rumors of me being the hand model for ArchiveTeam are, however,
unsubstantiated.)
My monthly report covers a large part of what I have been doing in the free software world. I write it for 
In my last 
