This is an updated version of an earlier post in March. A number of talks about Rcpp are scheduled over the next couple of months:

• On Wednesday May 14 I will be speaking at the Chicago R Users Group meeting charmingly titled Warm Up for R/Finance, along with fellow R/Finance organizer Josh Ulrich, fellow R/Finance regular Kent Hosxey as well as Cliff Clive. I will talk about Redis and my rather recent nascent RcppRedis package.
• Our sixth international conference R/Finance will take place May 16 and 17, 2014, at UIC in Chicago. See the current agenda for details of the program---which once again will be preceded by one-hour workshops. And as in the past, I will offer a one-hour gentle introduction to Rcpp. This workshop requires registration and a fee, and registraion will close shortly. I will also present a short lightning talk about the RcppRedis package as part of the conference proper.
• A few days later, on May 20, 2014, I will present a one-hour webinar which is part of a series organized by Ray DiGiacomo and the Orange County R Users Group. See this page for details and registration information; the talk itself will be free of charge.
• Later in the month, on May 30, 2014, I will give a talk / guest lecture about Rcpp and RcppArmadillo and over in Economics at the University of Chicago. I reckon that this event is not open to anybody not part of U of C.
• Another three-hour tutorial is scheduled for the tutorial day preceding useR! 2014 at UCLA in early July. This tutorial will also require registration as part of the annual useR! conference. I am very honoured to have been asked to give one of the invited keynotes at useR! 2014. The talk may (or may not) cover at least some aspects of Rcpp. As my talk proposal about the recent RcppZiggurat package for faster Normal RNGs was accepted, I will also give a regular conference talk.

So the month of June remains pretty open. If someone wants me to talk somewhere else, preferably on a Saturday, feel free to get in touch.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

For some years LVM (the Linux Logical Volume Manager) has been used in most Linux systems. LVM allows one or more storage devices (either disks, partitions, or RAID sets) to be assigned to a Volume Group (VG) some of which can then allocated to a Logical Volume (LVs) which are equivalent to any other block device, a VG can have many LVs. One of the significant features of LVM is that you can create snapshots of a LV. One common use is to have multiple snapshots of a LV for online backups and another is to make a snapshot of a filesystem before making a backup to external storage, the snapshot is unchanging so there s no problem of inconsistencies due to backing up a changing data set. When you create a snapshot it will have the same filesystem label and UUID so you should always mount a LVM device by it s name (which will be /dev/$VGNAME/$LVNAME). One of the problems with the ReiserFS filesystem was that there was no way to know whether a block of storage was a data block, a metadata block, or unused. A reiserfsck --rebuild-tree would find any blocks that appeared to be metadata and treat them as such, deleted files would reappear and file contents which matched metadata (such as a file containing an image of a ReiserFS filesystem) would be treated as metadata. One of the impacts of this was that a hostile user could create a file which would create a SUID root program if the sysadmin ran a --rebuild-tree operation. BTRFS solves the problem of filesystem images by using a filesystem specific UUID in every metadata block. One impact of this is that if you want to duplicate a BTRFS filesystem image and use both copies on the same system you need to regenerate all the checksums of metadata blocks with the new UUID. The way BTRFS works is that filesystems are identified by UUID so having multiple block devices with the same UUID causes the kernel to get confused. Making an LVM snapshot really isn t a good idea in this situation. It s possible to change BTRFS kernel code to avoid some of the problems of duplicate block devices and it s most likely that something will be done about it in future. But it still seems like a bad idea to use LVM with BTRFS. The most common use of LVM is to divide the storage of a single disk or RAID array for the use of multiple filesystems. Each filesystem can be enlarged (through extending the LV and making the filesystem use the space) and snapshots can be taken. With BTRFS you can use subvolumes for the snapshots and the best use of BTRFS (IMHO) is to give it all the storage that s available so there is no need to enlarge a filesystem in typical use. BTRFS supports quotas on subvolumes which aren t really usable yet but in the future will remove the need to create multiple filesystems to control disk space use. An important but less common use of LVM is to migrate a live filesystem to a new disk or RAID array, but this can be done by BTRFS too by adding a new partition or disk to a filesystem and then removing the old one. It doesn t seem that LVM offers any benefits when you use BTRFS. When I first experimented with BTRFS I used LVM but I didn t find any benefit in using LVM and it was only a matter of luck that I didn t use a snapshot and break things. Snapshots of BTRFS Filesystems One reason for creating a snapshot of a filesystem (as opposed to a snapshot of a subvolume) is for making backups of virtual machines without support from inside the virtual machine (EG running an old RHEL5 virtual machine that doesn t have the BTRFS utilities). Another is for running training on virtual servers where you want to create one copy of the filesystem for each student. To solve both these problems I am currently using files in a BTRFS subvolume. The BTRFS kernel code won t touch those files unless I create a loop device so I can only create a loop device for one file at a time. One tip for doing this, don t use names such as /xenstore/vm1 for the files containing filesystem images, use names such as /xenstore/vm1-root. If you try to create a virtual machine named vm1 then Xen will look for a file named vm1 in the current directory before looking in /etc/xen and tries to use a filesystem image as a Xen configuration file. It would be nice if there was a path for Xen configuration files that either didn t include the current directory or included it at the end of the list. Including the current directory in the path is a DOS mistake that should have gone away a long time ago. Psychology and Block Devices ZFS has a similar design to BTRFS in many ways and has some similar issues. But one benefit for ZFS is that it manages block devices in a zpool , first you create a zpool with the block devices and after that you can create ZFS filesystems or ZVOL block devices. I think that most sysadmins would regard a zpool as something similar to LVM (which may or may not be correct depending on how you look at it) and immediately rule out the possibility of running a zpool on LVM. BTRFS looks like a regular Unix filesystem in many ways, you can have a single block device that you mount with the usual mount command. The fact that BTRFS can support multiple block devices in a RAID configuration isn t so obvious and the fact that it implements equivalents to most LVM functionality probably isn t known to most people when they start using it. The most obvious way to start using BTRFS is to use it just like an Ext3/4 filesystem on an LV, and to use LVM snapshots to backup data, this is made even more likely by the fact that there is a program to convert a ext2/3/4 filesystem to BTRFS. This seems likely to cause data loss.

Oracle won their appeal regarding whether APIs are copyrightable. There'll be ongoing argument about whether Google's use of those APIs is fair use or not, and perhaps an appeal to the Supreme Court, but that's the new status quo. This will doubtless result in arguments over whether Oracle's implementation of Linux APIs in Solaris 10 was a violation of copyright or not (and presumably Google are currently checking whether they own any code that Oracle reimplemented), but that's not what I'm going to talk about today.

Oracle own some code called DTrace (Wikipedia has a good overview here - what it actually does isn't especially relevant) that was originally written as part of Solaris. When Solaris was released under the CDDL, so was DTrace. The CDDL is a file-level copyleft license with some restrictions not present in the GPL - as a result, combining GPLed code with CDDLed code will (in the absence of additional permission grants) result in a work that is under an inconsistent license and cannot legally be distributed.

Oracle wanted to make DTrace available for Linux as part of their Unbreakable Linux product. Integrating it directly into the kernel would obviously cause legal issues, so instead they implemented it as a kernel module. The copyright status of kernel modules is somewhat unclear. The GPL covers derivative works, but the definition of derivative works is a function of copyright law and judges. Making use of explicitly exported API may not be sufficient to constitute a derivative work - on the other hand, it might. This is largely untested in court. Oracle appear to believe that they're legitimate, and so have added just enough in-kernel code (and GPLed) to support DTrace, while keeping the CDDLed core of DTrace separate.

The kernel actually has two levels of exposed (non-userspace) API - those exported via EXPORT_SYMBOL() and those exported via EXPORT_SYMBOL_GPL(). Symbols exported via EXPORT_SYMBOL_GPL() may only be used by modules that claim to be GPLed, with the kernel refusing to load them otherwise. There is no technical limitation on the use of symbols exported via EXPORT_SYMBOL().

(Aside: this should not be interpreted as meaning that modules that only use symbols exported via EXPORT_SYMBOL() will not be considered derivative works. Anything exported via EXPORT_SYMBOL_GPL() is considered by the author to be so fundamental to the kernel that using it would be impossible without creating a derivative work. Using something exported via EXPORT_SYMBOL() may result in the creation of a derivative work. Consult lawyers before attempting to release a non-GPLed Linux kernel module)

DTrace integrates very tightly with the host kernel, and one of the things it needs access to is a high-resolution timer that is guaranteed to monotonically increase. Linux provides one in the form of ktime_get(). Unfortunately for Oracle, ktime_get() is only exported via EXPORT_SYMBOL_GPL(). Attempting to call it directly from the DTrace module would fail.

Oracle work around this in their (GPLed) kernel abstraction code. A function called dtrace_gethrtimer() simply returns the value of ktime_get(). dtrace_gethrtimer() is exported via EXPORT_SYMBOL() and therefore can be called from the DTrace module.

So, in the face of a technical mechanism designed to enforce the author's beliefs about the copyright status of callers of this function, Oracle deliberately circumvent that technical mechanism by simply re-exporting the same function under a new name. It should be emphasised that calling an EXPORT_SYMBOL_GPL() function does not inherently cause the caller to become a derivative work of the kernel - it only represents the original author's opinion of whether it would. You'd still need a court case to find out for sure. But if it turns out that the use of ktime_get() does cause a work to become derivative, Oracle would find it fairly difficult to argue that their infringement was accidental.

Of course, as copyright holders of DTrace, Oracle could solve the problem by dual-licensing DTrace under the GPL as well as the CDDL. The fact that they haven't implies that they think there's enough value in keeping it under an incompatible license to risk losing a copyright infringement suit. This might be just the kind of recklessness that Oracle accused Google of back in their last case.

comments

Debian has recently received a donation of 8 build machines from Marvell. The new machines come with Quad core MV78460 Armada XP CPU's, DDR3 DIMM slot so we can plug in more memory, and speedy sata ports. They replace the well served Marvell MV78200 based builders - ones that have been building debian armel since 2009. We are planning a more detailed announcement, but I'll provide a quick summary: The speed increase provided by MV78460 can viewed by comparing build times on selected builds since early april: Qemu build times. We can now build Qemu in 2h instead of 16h -8x faster than before! Certainly a substantial improvement, so impressive kit from Marvell! But not all packages gain this amount of speedup: webkitgtk build times. This example, webkitgtk, builds barely 3x faster. The explanation is found from debian/rules of webkitgkt:

# Parallel builds are unstable, see #714072 and #722520
# ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
#   NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
#   MAKEARGUMENTS += -j$(NUMJOBS)
# endif

The old builders are single-core[1], so the regardless of parallel building, you can easily max out the cpu. New builders will use only 1 of 4 cores without parallel build support in debian/rules. During this buildd cpu usage graph, we see most time only one CPU is consumed. So for fast package build times.. make sure your packages supports parallel building. For developers, abel.debian.org is porter machine with Armada XP. It has schroot's for both armel and armhf. set "DEB_BUILD_OPTIONS=parallel=4" and off you go. Finally I'd like to thank Thomas Petazzoni, Maen Suleiman, Hector Oron, Steve McIntyre, Adam Conrad and Jon Ward for making the upgrade happen. Meanwhile, we have unrelated trouble - a bunch of disks have broken within a few days apart. I take the warranty just run out... [1] only from Linux's point of view. - mv78200 has actually 2 cores, just not SMP or coherent. You could run an RTOS on the other core while you run Linux on the other.

A bug-fix release 0.3.2.1.2 is of RcppEigen is now on CRAN and in Debian. It contains a short upstream patch by Gaek Guennebaud for the UBSAN warnings that were flagged at CRAN (from a rather harmless cpu-id comparison). The NEWS entry follows.

Changes in RcppEigen version 0.3.2.1.2 (2014-05-05)

• Applied upstream patch to not trigger g++ UBSAN warning on the cpu id comparison, with thanks to Gael Guennebaud for the patch

Courtesy of CRANberries, there are diffstat reports for the most recent release. Questions, comments etc about RcppEigen should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Weblate 1.9 has been released today. It comes with lot of improvements and bug fixes and with experimental Zen mode for editing translations. Full list of changes for 1.9:

• Django 1.6 compatibility.
• No longer maintained compatibility with Django 1.4.
• Management commands for locking/unlocking translations.
• Improved support for Qt TS files.
• Users can now delete their account.
• Avatars can be disabled.
• Merged first and last name attributes.
• Avatars are now fetched and cached server side.
• Added support for shields.io badge.

You can find more information about Weblate on http://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Ready to run appliances will be soon available in SUSE Studio Gallery. Weblate is also being used https://l10n.cihar.com/ as official translating service for phpMyAdmin, Gammu, Weblate itself and others. If you are free software project which would like to use Weblate, I'm happy to help you with set up or even host Weblate for you. Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far!

Filed under: English phpMyAdmin SUSE Weblate 0 comments Flattr this!

A new release 4.300 of Armadillo, the templated C++ library for linear algebra, was released by Conrad a few days ago. As usual, this was rolled up in a new RcppArmadillo release 0.4.300.0 which is now on CRAN and in Debian. This release brings nice new upstream changes such as faster find() as well as corresponding functions to find elements which are either finite or non-finite, based on a suggestion we made. This may need an RcppArmadillo specific extension as R extends the IEEE logic to supported NA, NaN and Inf on floating point variables whereas standard C++ (and hence Armadillo) only supports the latter two. We hope to work on this shortly. Kevin also put in some code to make sure unsigned int vectors and matrices are converted correctly, and I updated the short configure script checking for complete LAPACK support to account for the R 3.0.3 or later case.

Changes in RcppArmadillo version 0.4.300.0 (2014-05-04)

• Upgraded to Armadillo release Version 4.300 (Medieval Cornea Scraper)
  • faster find()
  • added find_finite() and find_nonfinite() for finding indices of finite and non-finite elements
  • expressions X=inv(A)*B*C and X=A.i()*B*C are automatically converted to X=solve(A,B*C)
• Corrected conversion to unsigned int vectors and matrices
• Configure script now checks for R version 3.0.3 or newer to enable complex divide-and-conquer SVD in case of R-supplied LAPACK

Courtesy of CRANberries, there is also a diffstat report for the most recent release As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Dashkiosk is a solution to manage dashboards on multiple displays. It comes in four parts:

1. A server will manage the screens by sending URL to be displayed. A web interface enables an administrator to configure groups of dashboards and attach them to a set of displays.
2. A receiver runs in a browser attached to each screen. On start, it contacts the server and waits for the URL to display.
3. An Android application provides a simple fullscreen webview to display the receiver.
4. A Chromecast custom receiver which will run the regular receiver to display dashboards using Google Chromecast devices. The server is able to drive Chromecast devices through nodecastor, a reimplementation of the sender API.

For a demo, have a look at the following video (it is also available as an Ogg Theora video).

Thanks to the work of the Alioth admins and the DSA team, single sign-on services are back online. Everyone will again be able to register for DebConf14 according to the steps described on this page.

An issue with the integration between alioth.debian.org and sso.debian.org is currently preventing people other than Debian developers from logging in to the DebConf website for registration. We apologize for the inconvenience. Administrators are working to resolve the issue as quickly as possible; we will update this blog once the problem has been resolved. In the meantime, Debian developers can continue to register using their debian.org SSO credentials.

If you're looking to get started at packaging free software for Debian, you should start with the excellent New Maintainers' Guide or the Introduction to Debian Packaging on the Debian wiki. Once you know the basics, or if you prefer to learn by example, you may be interested in the full walkthrough which follows. We will look at the contents of three simple packages.

node-libravatar This package is a node.js library for the Libravatar service. Version 2.0.0-3 of that package contains the following files in its debian/ directory:

• changelog
• compat
• control
• copyright
• docs
• node-libravatar.install
• rules
• source/format
• watch

debian/control

Source: node-libravatar
Priority: extra
Maintainer: Francois Marier <francois@debian.org>
Build-Depends: debhelper (>= 9)
Standards-Version: 3.9.4
Section: web
Homepage: https://github.com/fmarier/node-libravatar
Vcs-Git: git://git.debian.org/collab-maint/node-libravatar.git
Vcs-Browser: http://git.debian.org/?p=collab-maint/node-libravatar.git;a=summary
Package: node-libravatar
Architecture: all
Depends: $ shlibs:Depends , $ misc:Depends , nodejs
Description: libravatar library for NodeJS
 This library allows web application authors to make use of the free Libravatar
 service (https://www.libravatar.org). This service hosts avatar images for
 users and allows other sites to look them up using email addresses.
 .
 node-libravatar includes full support for federated avatar servers.

This is probably the most important file since it contains the bulk of the metadata about this package. Maintainer is a required field listing the maintainer of that package, which can be a person or a team. It only contains a single value though, any co-maintainers will be listed under the optional Uploaders field. Build-Depends lists the packages which are needed to build the package (e.g. a compiler), as opposed to those which are needed to install the binary package (e.g. a library it uses). Standards-Version refers to the version of the Debian Policy that this package complies with. The Homepage field refers to the upstream homepage, whereas the Vcs-* fields point to the repository where the packaging is stored. If you take a look at the node-libravatar packaging repository you will see that it contains three branches:

• upstream is the source as it was in the tarball downloaded from upstream.
• master is the upstream branch along with all of the Debian customizations.
• pristine-tar is unrelated to the other two branches and is used by the pristine-tar tool to reconstitute the original upstream tarball as needed.

After these fields comes a new section which starts with a Package field. This is the definition of a binary package, not to be confused with the Source field at the top of this file, which refers to the name of the source package. In this particular example, they are both the same and there is only one of each, however this is not always the case, as we'll see later. Inside that binary package definition, lives the Architecture field which is normally one of these two:

• all for a binary package that will work on all architectures but only needs to be built once
• any for a binary package that will work everywhere but that will need to be built separately for each architecture

Finally, the last field worth pointing out is the Depends field which lists all of the runtime dependencies that the binary package has. This is what will be pulled in by apt-get when you apt-get install node-libravatar. The two variables will be substituted later by debhelper.

debian/changelog

node-libravatar (2.0.0-3) unstable; urgency=low
  * debian/watch: poll github directly
  * Bump Standards-Version up to 3.9.4
 -- Francois Marier <francois@debian.org>  Mon, 20 May 2013 12:07:49 +1200
node-libravatar (2.0.0-2) unstable; urgency=low
  * More precise license tag and upstream contact in debian/copyright
 -- Francois Marier <francois@debian.org>  Tue, 29 May 2012 22:51:03 +1200
node-libravatar (2.0.0-1) unstable; urgency=low
  * New upstream release
    - new non-backward-compatible API
 -- Francois Marier <francois@debian.org>  Mon, 07 May 2012 14:54:19 +1200
node-libravatar (1.1.1-1) unstable; urgency=low
  * Initial release (Closes: #661771)
 -- Francois Marier <francois@debian.org>  Fri, 02 Mar 2012 15:29:57 +1300

This may seem at first like a mundane file, but it is very important since it is the canonical source of the package version (2.0.0-3 in this case). This is the only place where you need to bump the package version when uploading a new package to the Debian archive. The first line also includes the distribution where the package will be uploaded. It is usually one of these values:

• unstable for the vast majority of uploads
• stable for uploads that have been approved by the release maintainers and fix serious bugs in the stable version of Debian
• stable-security for security fixes to the stable version of Debian that cannot wait until the next stable point release and have been approved by the security team

Packages uploaded to unstable will migrate automatically to testing provided that a few conditions are met (e.g. no release-critical bugs were introduced). The length of time before that migration is influenced by the urgency field (low, medium or high) in the changelog entry. Another thing worth noting is that the first upload normally needs to close an ITP (Intent to Package) bug.

debian/rules

#!/usr/bin/make -f
# -*- makefile -*-
%:
    dh $@ 
override_dh_auto_test:

As can be gathered from the first two lines of this file, this is a Makefile. This is what controls how the package is built. There's not much to see and that's because most of its content is automatically added by debhelper. So let's look at it in action by building the package:

$ git buildpackage -us -uc

and then looking at parts of the build log (../node-libravatar_2.0.0-3_amd64.build):

 fakeroot debian/rules clean
dh clean 
   dh_testdir
   dh_auto_clean
   dh_clean

One of the first things we see is the debian/rules file being run with the clean target. To find out what that does, have a look at the dh_auto_clean which states that it will attempt to delete build residues and run something like make clean using the upstream Makefile.

 debian/rules build
dh build 
   dh_testdir
   dh_auto_configure
   dh_auto_build

Next we see the build target being invoked and looking at dh_auto_configure we see that this will essentially run ./configure and its equivalents. The dh_auto_build helper script then takes care of running make (or equivalent) on the upstream code. This should be familiar to anybody who has ever built a piece of free software from scratch and has encountered the usual method for building from source:

./configure
make
make install

Finally, we get to actually build the .deb:

 fakeroot debian/rules binary
dh binary 
   dh_testroot
   dh_prep
   dh_installdirs
   dh_auto_install
   dh_install
...
   dh_md5sums
   dh_builddeb
dpkg-deb: building package  node-libravatar' in  ../node-libravatar_2.0.0-3_all.deb'.

Here we see a number of helpers, including dh_auto_install which takes care of running make install. Going back to the debian/rules, we notice that there is manually defined target at the bottom of the file:

override_dh_auto_test:

which essentially disables dh_auto_test by replacing it with an empty set of commands. The reason for this becomes clear when we take a look at the test target of the upstream Makefile and the dependencies it has: tap, a node.js library that is not yet available in Debian. In other words, we can't run the test suite on the build machines so we need to disable it here.

debian/compat

9

This file simply specifies the version of debhelper that is required by the various helpers used in debian/rules. Version 9 is the latest at the moment.

debian/copyright

Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: node-libravatar
Upstream-Contact: Francois Marier <francois@libravatar.org>
Source: https://github.com/fmarier/node-libravatar
Files: *
Copyright: 2011 Francois Marier <francois@libravatar.org>
License: Expat
Files: debian/*
Copyright: 2012 Francois Marier <francois@debian.org>
License: Expat
License: Expat
 Permission is hereby granted, free of charge, to any person obtaining a copy of this
 software and associated documentation files (the "Software"), to deal in the Software
 without restriction, including without limitation the rights to use, copy, modify,
 merge, publish, distribute, sublicense, and/or sell copies of the Software, and to
 permit persons to whom the Software is furnished to do so, subject to the following
 conditions:
 .
 The above copyright notice and this permission notice shall be included in all copies
 or substantial portions of the Software.
 .
 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
 INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
 PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
 HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
 CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE
 OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

This machine-readable file lists all of the different licenses encountered in this package. It requires that the maintainer audits the upstream code for any copyright statements that might be present in addition to the license of the package as a whole.

debian/docs

README.md

This file contains a list of upstream files that will be copied into the /usr/share/doc/node-libravatar/ directory by dh_installdocs.

debian/node-libravatar.install

lib/*    usr/lib/nodejs/

The install file is used by dh_install to supplement the work done by dh_auto_install which, as we have seen earlier, essentially just runs make install on the upstream Makefile. Looking at that upstream Makefile, it becomes clear that the files will need to be installed manually by the Debian package since that Makefile doesn't have an install target.

debian/watch

version=3
https://github.com/fmarier/node-libravatar/tags /fmarier/node-libravatar/archive/node-libravatar-([0-9.]+)\.tar\.gz

This is the file that allows Debian tools like the Package Tracking System to automatically detect that a new upstream version is available. What it does is simply visit the upstream page which contains all of the release tarballs and look for links which have an href matching the above regular expression. Running uscan --report --verbose will show us all of the tarballs that can be automatically discovered using this watch file:

-- Scanning for watchfiles in .
-- Found watchfile in ./debian
-- In debian/watch, processing watchfile line:
   https://github.com/fmarier/node-libravatar/tags /fmarier/node-libravatar/archive/node-libravatar-([0-9.]+)\.tar\.gz
-- Found the following matching hrefs:
     /fmarier/node-libravatar/archive/node-libravatar-2.0.0.tar.gz
     /fmarier/node-libravatar/archive/node-libravatar-1.1.1.tar.gz
     /fmarier/node-libravatar/archive/node-libravatar-1.1.0.tar.gz
     /fmarier/node-libravatar/archive/node-libravatar-1.0.1.tar.gz
     /fmarier/node-libravatar/archive/node-libravatar-1.0.0.tar.gz
Newest version on remote site is 2.0.0, local version is 2.0.0
 => Package is up to date
-- Scan finished

pylibravatar This second package is the equivalent Python library for the Libravatar service. Version 1.6-2 of that package contains similar files in its debian/ directory, but let's look at two in particular:

• control
• upstream/signing-key.asc

debian/control

Source: pylibravatar
Section: python
Priority: optional
Maintainer: Francois Marier <francois@debian.org>
Build-Depends: debhelper (>= 9), python-all, python3-all
Standards-Version: 3.9.5
Homepage: https://launchpad.net/pyLibravatar
...
Package: python-libravatar
Architecture: all
Depends: $ misc:Depends , $ python:Depends , python-dns, python
Description: Libravatar module for Python 2
 Module to make use of the federated Libravatar.org avatar hosting service
 from within Python applications.
...
Package: python3-libravatar
Architecture: all
Depends: $ misc:Depends , $ python3:Depends , python3-dns, python3
Description: Libravatar module for Python 3
 Module to make use of the federated Libravatar.org avatar hosting service
 from within Python applications.
...

Here is an example of a source package (pylibravatar) which builds two separate binary packages: python-libravatar and python3-libravatar. This highlights the fact that a given upstream source can be split into several binary packages in the archive when it makes sense. In this case, there is no point in Python 2 applications pulling in the Python 3 files, so the two separate packages make sense. Another common example is the use of a -doc package to separate the documentation from the rest of a package so that it doesn't need to be installed on production servers for example.

debian/upstream/signing-key.asc

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1
mQINBEpQYz4BEAC7REQD1za69RUnkt6nRCFhSJmmoeJc+yEiWTKc9GOIMAwJDme1
+CMYgVn4Xzf1VQYwD/lE+mfWgyeMomLQjDM1mxx/LOM2a1WWPOk9+PvQwKfRJy92
...
UxDtZm/4yUmU6KvHvOGiDCMuIiB+MqhqJJ5wf80wXhzu8nmC+fyGt6nvu0ggMle8
sAMgXt/aQUTZE5zNCQ==
=RkTO
-----END PGP PUBLIC KEY BLOCK-----

This is simply the OpenPGP key that the upstream developer uses to sign release tarballs. Since PGP signatures are available on the upstream download page, it's possible to instruct uscan to check signatures before downloading tarballs. The way to do that is to use the pgpsigurlmange option in debian/watch:

version=3
opts=pgpsigurlmangle=s/$/.asc/ https://pypi.python.org/pypi/pyLibravatar https://pypi.python.org/packages/source/p/pyLibravatar/pyLibravatar-(.*)\.tar\.gz

which is simply a regular expression replacement string which takes the tarball URL and converts it to the URL of the matching PGP signature.

fcheck The last package we will look at is a file integrity checker. It essentially goes through all of the files in /usr/bin/ and /usr/lib/ and stores a hash of them in its database. When one of these files changes, you get an email. In particular, we will look at the following files in the debian/ directory of version 2.7.59-18:

• dirs
• fcheck.cron.d
• fcheck.postrm
• fcheck.postinst
• patches/
• README.Debian
• rules
• source/format

debian/patches This directory contains ten patches as well as a file called series which lists the patches that should be applied to the upstream source and in which order. Should you need to temporarily disable a patch, simply remove it from this file and it will no longer be applied at build time. Let's have a look at patches/04_cfg_sha256.patch:

Description: Switch to sha256 hash algorithm
Forwarded: not needed
Author: Francois Marier <francois@debian.org>
Last-Update: 2009-03-15
--- a/fcheck.cfg
+++ b/fcheck.cfg
@@ -149,8 +149,7 @@ TimeZone        = EST5EDT
 #$Signature      = /usr/bin/sum
 #$Signature      = /usr/bin/cksum
 #$Signature      = /usr/bin/md5sum
-$Signature      = /bin/cksum
-
+$Signature      = /usr/bin/sha256sum
 # Include an optional configuration file.

This is a very simple patch which changes the default configuration of fcheck to promote the use of a stronger hash function. At the top of the file is a bunch of metadata in the DEP-3 format. Why does this package contain so many customizations to the upstream code when Debian's policy is to push fixes upstream and work towards reduce the delta between upstream and Debian's code? The answer can be found in debian/control:

Homepage: http://web.archive.org/web/20050415074059/www.geocities.com/fcheck2000/

This package no longer has an upstream maintainer and its original source is gone. In other words, the Debian package is where all of the new bug fixes get done.

debian/source/format

3.0 (quilt)

This file contains what is called the source package format. What it basically says is that the patches found in debian/patches/ will be applied to the upstream source using the quilt tool at build time.

debian/fcheck.postrm

#!/bin/sh
# postrm script for fcheck
#
# see: dh_installdeb(1)
set -e
# summary of how this script can be called:
#        * <postrm>  remove'
#        * <postrm>  purge'
#        * <old-postrm>  upgrade' <new-version>
#        * <new-postrm>  failed-upgrade' <old-version>
#        * <new-postrm>  abort-install'
#        * <new-postrm>  abort-install' <old-version>
#        * <new-postrm>  abort-upgrade' <old-version>
#        * <disappearer's-postrm>  disappear' <overwriter>
#          <overwriter-version>
# for details, see http://www.debian.org/doc/debian-policy/ or
# the debian-policy package
case "$1" in
    remove upgrade failed-upgrade abort-install abort-upgrade disappear)
    ;;
    purge)
      if [ -e /var/lib/fcheck/fcheck.dbf ]; then
        echo "Purging old database file ..."
        rm -f /var/lib/fcheck/fcheck.dbf
      fi
      rm -rf /var/lib/fcheck
      rm -rf /var/log/fcheck
      rm -rf /etc/fcheck
    ;;
    *)
        echo "postrm called with unknown argument \ $1'" >&2
        exit 1
    ;;
esac
# dh_installdeb will replace this with shell code automatically
# generated by other debhelper scripts.
#DEBHELPER#
exit 0

This script is one of the many possible maintainer scripts that a package can provide if needed. This particular one, as the name suggests, will be run after the package is removed (apt-get remove fcheck) or purged (apt-get remove --purge fcheck). Looking at the case statement above, it doesn't do anything extra in the remove case, but it deletes a few files and directories when called with the purge argument.

debian/README.Debian This optional README file contains Debian-specific instructions that might be useful to users. It supplements the upstream README which is often more generic and cannot assume a particular system configuration.

debian/rules

#!/usr/bin/make -f
# -*- makefile -*-
# Sample debian/rules that uses debhelper.
# This file was originally written by Joey Hess and Craig Small.
# As a special exception, when this file is copied by dh-make into a
# dh-make output file, you may use that output file without restriction.
# This special exception was added by Craig Small in version 0.37 of dh-make.
# Uncomment this to turn on verbose mode.
#export DH_VERBOSE=1
build-arch:
build-indep:
build: build-stamp
build-stamp:
    dh_testdir
    pod2man --section=8 $(CURDIR)/debian/fcheck.pod > $(CURDIR)/fcheck.8
    touch build-stamp
clean:
    dh_testdir
    dh_testroot
    rm -f build-stamp 
    rm -f $(CURDIR)/fcheck.8
    dh_clean
install: build
    dh_testdir
    dh_testroot
    dh_prep
    dh_installdirs
    cp $(CURDIR)/fcheck $(CURDIR)/debian/fcheck/usr/sbin/fcheck
    cp $(CURDIR)/fcheck.cfg $(CURDIR)/debian/fcheck/etc/fcheck/fcheck.cfg
# Build architecture-independent files here.
binary-arch: build install
# Build architecture-independent files here.
binary-indep: build install
    dh_testdir
    dh_testroot
    dh_installdocs
    dh_installcron
    dh_installman fcheck.8
    dh_installchangelogs
    dh_installexamples
    dh_installlogcheck
    dh_link
    dh_strip
    dh_compress
    dh_fixperms
    dh_installdeb
    dh_shlibdeps
    dh_gencontrol
    dh_md5sums
    dh_builddeb
binary: binary-indep binary-arch
.PHONY: build clean binary-indep binary-arch binary install

This is an example of a old-style debian/rules file which you still encounter in packages which haven't yet upgraded to the latest version of debhelper 9, as can be shown by the contents of debian/compat:

8

It does essentially the same thing that what we've seen in the build log, but in a more verbose way.

debian/dirs

usr/sbin
etc/fcheck

This file contains a list of directories that dh_installdirs will create in the build directory. The reason why these directories need to be created is that files are copied into these directories in the install target of the debian/rules file. Note that this is different from directories which are created at the time of installation of the package. In that case, the directory (e.g. /var/log/fcheck/) must be created in the postinst script and removed in the postrm script.

debian/fcheck.cron.d

#
# Regular cron job for the fcheck package
#
30 */2  * * *   root    test -x /usr/sbin/fcheck && if ! nice ionice -c3 /usr/sbin/fcheck -asxrf /etc/fcheck/fcheck.cfg >/var/run/fcheck.out 2>&1; then mailx -s "ALERT: [fcheck]  hostname --fqdn " root </var/run/fcheck.out ; /usr/sbin/fcheck -cadsxlf /etc/fcheck/fcheck.cfg ; fi ; rm -f /var/run/fcheck.out

This file is the cronjob which drives the checks performed by this package. It will be copied to /etc/cron.d/fcheck by dh_installcron.

It took some time, but now it s finally done: KDE has translations for AppStream upstream metadata! AppStream is a Freedesktop project to extend metadata about the software projects which is available in distributions, especially regarding applications. Distributions compile a metadata file from data collected from packages, .desktop files and possibly other information sources, and create an AppStream XML file from it, which is then directly or via a Xapian cache read by software-center-like applications such as GNOME-Software or KDEs Apper. Since the metadata available from current sources is not standardized and rather poor, upstream projects can ship small XML files, AppStream upstream metadata or AppData in short. These files contain additional information about a project, such as a long description and links to screenshots. They also provide hints about public interfaces a software provides, for example binaries and libraries, making it possible for distributors to give users exactly the right package name in case they are missing a software component. So, in order to represent graphical KDE applications like they deserve it in the new software centers making use of AppStream, we need to ship AppData files, with long descriptions, screenshots and a few URLs. But how can you create these metadata files? In case you want your graphical KDE app to ship an AppData file, there is now a help page on the Techbase Wiki which provides all information needed to get started! For non-visual stuff or software which just wants to publish it s provided interfaces with AppStream metadata, there is a dedicated page for that as well. Shipping metadata for non-GUI apps will help programmers to satisfy depedencies in order to compile new software, enhance bash-completion for missing binaries and provides some other neat stuff (take a look at this blogpost to get a taste of it). And if you want to read a FAQ about the metadata stuff and get the bigger picture, just go to the Techbase Wiki page about AppStream metadata as well. The pages are not 100% final, so if you have questions, please write me a mail and I ll update the pages, or simply correct/refine it by yourself (it s a wiki afterall). And now to the best thing: As soon as you ship an AppStream upstream metadata file (*.appdata.xml* for apps or *.metainfo.xml* for other stuff), the KDE l10n-script (Scripty!) will automatically start translating it, just like we already do with .desktop files. No further actions are necessary. I already have a large amount of metadata files here, partially auto-generated, which show that we have about 160+ applications in KDE which could get an AppData file, not counting any frameworks or other non-GUI stuff yet. Since that is a bit much to submit via Reviewboard (which I originally planned to do), I hope I can commit the changes directly to the respective repositories, where the maintainers can take a look at it and adjust it to their liking. If that idea does not receive approval, I will just publish a set of data at some place for the KDE app maintainers to take as a reference (the auto-generated stuff needs some fixup to be commit-ready (which I d do in case I can just commit changes)). Either way, it is safe now to write and ship AppData files in KDE projects! In order to get your stuff translated, it is necessary that you follow the AppStream 0.6 metadata specification, and not one of ther older revisions. You can easily detect 0.6 metadata by the <component> root node, instead of <application>, or by it having a metadata_license tag. We don t support the older versions simply because it s not necessary, as there were only two KDE projects shipping AppData before, which are now using 0.6 data as well. Since 0.6, the metadata XML format is guaranteed to be stable, and the only reason which could make me change it in an incompatible way is to prevent something as bad as the end of the world from happening (== won t happen ). You can find the full specification (upstream and distro data) here. All parsers are able to handle 0.6 data now, and the existing tools are almost all migrated already (might take a few months to hit the distributions though). So, happy metadata-writing! Thanks to all people who helped with making this happen, and especially Burkhard L ck and Albert Astals Cid for their patch-review and help with integrating the necessary changes into the KDE l10n-script.

The very useful Valgrind tool had flagged an actual error in the package which the CRAN maintainers asked us to address. This was followed by a minor brown-bad oversight of a missing delete, also tagged by Valgrind. Both are pretty ancient bugs which we probably should have found aeons ago. Releases 0.1.3 and 0.1.4 made it to CRAN yesterday in short succession. To recap, RcppSMC combines the SMCTC template classes for Sequential Monte Carlo and Particle Filters (Johansen, 2009, JSS) with the Rcpp package for R/C++ Integration (Eddelbuettel and Francois, 2011, JSS) and thereby allows for easier and more direct access from R to the computational core of the Sequential Monte Carlo algorithm. The two NEWS entries are below:

Changes in RcppSMC version 0.1.4 (2014-05-02)

• Added missing delete operator to destructor in sampler

Changes in RcppSMC version 0.1.3 (2014-05-01)

• Bugfix in Sampler.iterate() for memory overrun detected by valgrind

Courtesy of CRANberries, there are also diffstat reports for 0.1.3. and 0.1.4. As always, more detailed information is on the RcppSMC page,

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

url_for is normally used to avoid hard coding of URL in Flask based webapps. I've been using it in SILPA port written using Flask by. But this was written long back and written as POC to show for Santhosh author of SILPA app which even managed to go into production :-). Recently I started organizing the code and started using flask.Blueprint which was previously written using flask.views.MethodView class and here I started facing problem with old templates. I'm just gonna explain what is the difference when using url_for with MethodView and Blueprint. Lets have some sample code for MethodView first, this code is from Flask documentation. Here we have a view which is derived from MethodView which has logic on how to handle the requests. We use add_url_rule to register rule to handle /users/ url end point and we pass the class as view function which is done by as_view class method and we can refer this method in our Jinja2 templates using name users. So a statement like in our templates will be converted to /users/ URL when page is rendered to client by Flask. Now when I replaced the MethodView in favor of Blueprint I started getting up werkzeug.routing.BuildError thrown on my face and I had no clue why!. Yeah I know I'm bad at reading documentation but even after reading documentation I was still thinking that should return me a proper URL and I was wondering why its failing. Finally after re-reading documentation for url_for it was becoming clear to me, url_for definition looks like below Here endpoint is actually a function which is supposed to be serving the URL and **values is the arguments for this function. The URL in question should be defined in python code using decorator. In my case following is the new function serving the web pages for SILPA. You can ignore function code but just note the decorators, here I'm registering the function serve_pages with page as argument for URL patterns / and /<page>, _BASE_URL here is mount point of application it can be just / or /mountpoint depending on that URL registered changes. Now I need to modify code for all url_for in my template to look like below The . in front of function is for referring current Blueprint, in my case the Flask will consider it as frontend.serve_pages as function name and generates appropriate URL at run time. So my fault was misunderstanding endpoint argument as URL endpoint but where as its actually function name supposed to serve the page. But when using MethodViews I can simply convert class to a function with my preferred name just like UserAPI.as_view('/') so url_for('/') just works.

Since the MiniDebConf Jonas and I have been travelling in Spain, France and finally staying in Belgium for a week, getting some work done. It's been harder than imagined to work during travel. I haven't exercised either, and regained at least three of four kilos I spent much time and effort getting rid in the year preceding. I thrive in my home and find it hard to keep my own time and focus when I am deprived of my own space. It was challenging to give a talk, "Why aren't more designers using Debian or working for Debian", my first public talk. I've been working to recapture my points in writing, to make a stronger statement, but I seem to blur my own views with conflicting ones, and I'm loosing momentum every day. One of my reasons for speaking up was to do it even though I'm not at trained speaker and have "nothing" to contribute but my opinions from the angle of a user that happens to be a designer. Not claiming to be a superior designer, but one that would like to contribute if it was easier to figure out how. And since the community wants to encourage designers to contribute to the Debian project, I figured it to be a good idea to talk about how this has been challenging to me as a dedicated user and completely out of the question for any other designer I know - or knew before the minidebconf. No reseach, no scientific proofs, just my wiew from my "dumb user" and designer's perspective. I saw one single attendant rolling his eyes during my talk. I didn't care at that time, but I've given that look more consideration than the people approaching me after the talk, saying thank you for voicing their opinions and thoughts. I think that's absolutely astonishing and at the same time it's just typically me. It makes me angry, first with myself for not speaking to this man's perception of things, then with myself for not just letting go of that image. I'm really glad that so many seemed to listen with curiosity and interest. What if one more - or half of the auditorium - had rolled their eyes? I don't like to feel that vulnerable. The truth is, though, that I'm really not. I gave the talk against my fear of failure and public humiliation and I'm convinced that my thoughts and actions matter, just as anybody's does, if we dare to say what's on our minds and to take action. I believe it's in anybody's power to "make a difference" and even "change the world" - at least in a small way. I guess that's one of the underlying reasons to be a designer in the first place. That is quite a strong position to take. I've created the wikipage http://wiki.debian.org/Design - well knowing that design is a word with many meanings. Everything is design. Since the talk I've been in doubt about that page. About the project, my aim with it, what to do about it, how to move on with just a tiny babystep, and I realise that I'm simply afraid to be disturbing someone's peace, making people angry or roll their eyes at my fumbling attempts to figure out in public what can be done to make a thriving community of designers collaborating with coders to make better, more usable and attractive software in the free, wide world. I'm starting a design process, not presenting a perfect, finished solution. Now, having put these thoughts into words, perhaps, my mind will be somewhat appeaced and let me move on with my intended tasks of cultivating that acclaimed space in the Debian information jungle into a friendly and welcoming place with info that makes it easier to be a contributing designer in Debian.

So the Kelly report of the independent review into the events leading to the Co-operative Bank s capital shortfall was published yesterday. During the day, I was putting odd bits from it out in 140 characters with the hashtags #coops #kellylessons. Here they are in one more permanent place. How many of these lessons has your organisation whether a co-op or not learned?

1. Running a full-service bank is a complex business Bank failed to understand the limits of its own capability
2. The most important task for any board is to put in place the right Executive leadership for the business
3. Ownership of a regulated bank requires a clearly articulated statement addressing mgmt & gov nance relationship
4. Failures in board oversight are inevitable if the criteria used to elect do not require the necessary skills
5. A bank board must include sufficient numbers of technically competent directors
6. Boards need good m gmt info and to demand it if it is not forthcoming. Failure to obtain explains failings
7. A bank should develop&implement robust risk gov nce&oversight and an appropriate control framework
8. IT transformation keep as simple as poss , phase delivery.., deploy the right resources, plan for contingencies
9. Bank should have paid closer attention and responded with greater urgency to what the Regulator told it
10. Pay careful attention to the advice of external advisors. The Group ignored well-founded inconvenient advice
11. Postponing dealing with problems is almost never a sustainable solution.
12. Values need to be translated into meaningful guidance The Bank s ethical positioning should be more apparent
13. Mantras about scale and ethics are no substitute for strategies grounded in a real understanding
14. Talent management is critical Lack of capability driven by weaknesses in its recruitment&subsequent m gmt
15. Tolerating culture of underperformance, weak transparency and a lack of accountability, constrains an organisation

Are there other lessons that you would add?

Alan How:

There's an Italian restaurant somewhere in the Lake District in the north of England. And if you can imagine, there are hills, lots of rain, lakes, and so on and so forth. And inside this Italian restaurant is a huge picture a wall-sized mural of Marlon Brando. Now, I asked my students once: Why Marlon Brando? Well he's The Godfather! I don't think he was a Godfather, you know... But he was in that film, wasn't he? It was called The Godfather. Ah, that film from 40 years ago, virtually.. 1970-something? Yes, yes, but he was The Godfather then, wasn't he? You mean if was a mafiosi? He was a leading figure in gangsterland in New York? Well, yes, he was The Godfather, they keep saying, ... and he was Italian. Well, no, actually he was American... So you get this regular kind of placing and misplacing of the real, ie. a very large American actor who played in a film called The Godfather about American gangsters, or Italian-American gangsters, nothing to do with godfathers in Christianity, in the north of England, stuck on a huge wall in a restaurant. All of which tells us that it's an Italian restaurant. And we don't demure, we don't say "no, that's all untrue", we just take it for granted: "Yeah, that's what it means - it's an Italian restaurant". And it's that kind of slippage between the real and the imagined or the true and the fictional which Baudrillard thinks is becoming inceasingly characteristic of how our society goes on.