Search Results: "rul"

11 May 2014

Matthew Garrett: Oracle continue to circumvent EXPORT_SYMBOL_GPL()

Oracle won their appeal regarding whether APIs are copyrightable. There'll be ongoing argument about whether Google's use of those APIs is fair use or not, and perhaps an appeal to the Supreme Court, but that's the new status quo. This will doubtless result in arguments over whether Oracle's implementation of Linux APIs in Solaris 10 was a violation of copyright or not (and presumably Google are currently checking whether they own any code that Oracle reimplemented), but that's not what I'm going to talk about today.

Oracle own some code called DTrace (Wikipedia has a good overview here - what it actually does isn't especially relevant) that was originally written as part of Solaris. When Solaris was released under the CDDL, so was DTrace. The CDDL is a file-level copyleft license with some restrictions not present in the GPL - as a result, combining GPLed code with CDDLed code will (in the absence of additional permission grants) result in a work that is under an inconsistent license and cannot legally be distributed.

Oracle wanted to make DTrace available for Linux as part of their Unbreakable Linux product. Integrating it directly into the kernel would obviously cause legal issues, so instead they implemented it as a kernel module. The copyright status of kernel modules is somewhat unclear. The GPL covers derivative works, but the definition of derivative works is a function of copyright law and judges. Making use of explicitly exported API may not be sufficient to constitute a derivative work - on the other hand, it might. This is largely untested in court. Oracle appear to believe that they're legitimate, and so have added just enough in-kernel code (and GPLed) to support DTrace, while keeping the CDDLed core of DTrace separate.

The kernel actually has two levels of exposed (non-userspace) API - those exported via EXPORT_SYMBOL() and those exported via EXPORT_SYMBOL_GPL(). Symbols exported via EXPORT_SYMBOL_GPL() may only be used by modules that claim to be GPLed, with the kernel refusing to load them otherwise. There is no technical limitation on the use of symbols exported via EXPORT_SYMBOL().

(Aside: this should not be interpreted as meaning that modules that only use symbols exported via EXPORT_SYMBOL() will not be considered derivative works. Anything exported via EXPORT_SYMBOL_GPL() is considered by the author to be so fundamental to the kernel that using it would be impossible without creating a derivative work. Using something exported via EXPORT_SYMBOL() may result in the creation of a derivative work. Consult lawyers before attempting to release a non-GPLed Linux kernel module)

DTrace integrates very tightly with the host kernel, and one of the things it needs access to is a high-resolution timer that is guaranteed to monotonically increase. Linux provides one in the form of ktime_get(). Unfortunately for Oracle, ktime_get() is only exported via EXPORT_SYMBOL_GPL(). Attempting to call it directly from the DTrace module would fail.

Oracle work around this in their (GPLed) kernel abstraction code. A function called dtrace_gethrtimer() simply returns the value of ktime_get(). dtrace_gethrtimer() is exported via EXPORT_SYMBOL() and therefore can be called from the DTrace module.

So, in the face of a technical mechanism designed to enforce the author's beliefs about the copyright status of callers of this function, Oracle deliberately circumvent that technical mechanism by simply re-exporting the same function under a new name. It should be emphasised that calling an EXPORT_SYMBOL_GPL() function does not inherently cause the caller to become a derivative work of the kernel - it only represents the original author's opinion of whether it would. You'd still need a court case to find out for sure. But if it turns out that the use of ktime_get() does cause a work to become derivative, Oracle would find it fairly difficult to argue that their infringement was accidental.

Of course, as copyright holders of DTrace, Oracle could solve the problem by dual-licensing DTrace under the GPL as well as the CDDL. The fact that they haven't implies that they think there's enough value in keeping it under an incompatible license to risk losing a copyright infringement suit. This might be just the kind of recklessness that Oracle accused Google of back in their last case.

comment count unavailable comments

Daniel Pocock: Is Uber on your side?

Crowdsourcing ventures with disruptive business models are a regular point of contention these days. In London, taxi drivers are threatening to create gridlock as part of an anti-Uber protest. In Melbourne, Uber drivers have been issued with $1,700 fines for operating without a taxi license. San Francisco city officials, despite being the birthplace of many of these ventures, are debating whether AirBNB should be regulated. An orderly society or an old-school protection racket? Just what exactly is it that established players in these industries are trying to achieve through their protests and lobbying efforts? In the case of apartment rentals, many people have sympathy for respecting the wishes of neighbourhoods over those of individual landlords. In the case of car pooling schemes, the arguments tend to come not from motorists at large but from those who are afraid of competition. Without competition, could things be any worse? Melbourne actually provides the perfect backdrop for this debate. Only a couple of years before Uber came on the scene, the government had made a detailed study into the taxi industry. One of Australia's most prominent economic experts, a former chairman of the Australian Competition and Consumer Commission spent 18 months studying the industry. One of the highlights of the incumbent system (and the reason I suggest Melbourne is the perfect backdrop for this debate) is the way licenses are issued to taxi drivers. There are a fixed number of licenses issued by the government. The licenses are traded on the open market so prices can go up and down just like real-estate. Under the rules of Australia's pension scheme, people have even been able to use money from their pension fund to purchase a taxi license as an investment. It goes without saying that this has helped rampant speculation and the price of a license is now comparable to the price of a house. The end result is that no real taxi driver can afford a license: most of them have to rent their license from one of the speculators who bought the license. These fixed rental fees have to be paid every month whether the driver uses their car or not. Consequently, taxi drivers have cut back on other expenses, they are often criticised for failing to keep their cars clean and the industry as a whole is criticised due to the poor quality of drivers who don't even know their way around the city. The reason, of course, is simple: by the time some newly arrived immigrant has learnt his way around Melbourne he has also figured out that the economics of driving a taxi are not in his favor. Realizing there is no way to break even, they take other jobs instead. It was originally speculated that the government review would dramatically reduce or abolish these speculative practices but ultimately lower license charges have only been used for the issue of 60 new licenses, barely 1% of the taxi fleet in the city today. Furthermore, the new licenses were only available to existing players in the industry. Uber to the rescue? Uber drove into the perfect storm as they launched their service in Melbourne in 2013. Uber drivers get a significant benefit over their competitors in traditional taxis. In particular, as they don't have the fixed monthly payment to rent a taxi license, they don't have to work every day and can even take holidays or take time to clean the cars. These things may simultaneously benefit road safety and passenger comfort. Meanwhile, those people who speculated on the old taxi licenses have tried hunger strikes and all kinds of other desperate tactics to defer the inevitable loss of their "investment". The reality is that crowdsourcing is here to stay. Even if Uber is stopped by bullying and intimidation, the inefficiency of Melbourne's taxi system is plain for all to see and both customers and drivers will continue looking for alternatives. Other car-pooling apps based on barter or cost sharing will continue to find ways to operate even if the Uber model is prohibited. It is interesting to note that the last great reform of Melbourne taxis, under Premier Jeff Kennett in the 1990s, simply resulted in a change of paint with the aim of making them look like those in New York City. Disruptive services like Uber (with their numerous technology-powered innovations to save time and money) appear to be doing far more to improve the lives of passengers and drivers. The hidden cost That said, large scale schemes like Uber do also have a down side for customer privacy. Hailing cabs in the street leaves no records of your movements. This new model, however, is leaving a very detailed trail of breadcrumbs that can be used for both marketing purposes or extracted (lawfully or otherwise) by some third party who wishes to monitor a particular customer's past or future movements. This is the trade-off that arises when we benefit from the efficiencies of any cloud-based service.

9 May 2014

Rhonda D'Vine: Rhonda's Free Hugging Guidelines

Recently I was wearing my Free Hugs shirt to different Free Software meetings, and I came up with the idea if we are advocating the Free in Software specificly, why not come up with Free Hugging Guidelines, too. So here they are, from now on considered to be named the RFHG.
  1. Free Redistribution Your hugs may not restrict any party from passing on the hugs they received from you.
  2. Source Code The hugs must be possible to be perceived and understandable in complete. You are not allowed to use any special techniques that can not be perceived.
  3. Derived Works Your hugs must be allowed to be modified, and must allow the modified forms of your hugs to be distributed under the same terms as they received them.
  4. Integrity of The Author's Source Code While you are allowed to pass on the hugs in modified form, you are not allowed to modify the DNA of the original person you received the hugs from. Genetic modification is out of the scope of the RFHG.
  5. No Discrimination Against Persons or Groups While we acknowledge that you might not feel willing to hug everyone, you must apply rules that do not distinquish by rules which would violate The Universal Declaration of Human Rights. Furthermore, you have the right to not hug a person if you are not in the mood for it. Please refrain from wearing any Free Hugs markers at those times though.
  6. No Discrimination Against Fields of Endeavor The hugs can not be restricted to be used in a specific field of endeavor. For example, you may not restrict the hugs from being passed on only in times of sorrow.
  7. Distribution of Hugs The rights attached to the hugs must apply to all to whom the hugs are redistributed without the need for execution of an additional license by those parties.
  8. Hugs Must Not Be Specific to Rhonda The rights attached to the hugs must not depend on the hugs being related to Rhonda. If the hugs are extracted from Rhonda and used or distributed without Rhonda nearby but otherwise within the terms of the hug's permissions, all parties to whom the hugs are distributed should have the same rights as those that are granted in conjunction with Rhonda.
  9. Hugs Must Not Contaminate Other People If you are contagious (e.g. got the flu, or worse) you have to apply appropriate counter measures to not transfer your illness with your hugs.
  10. Example Hugs This Group Hug modification and of course the Free Hugs Campaign are great examples of what we consider Free.
One last note: If you feel like it you don't have to wait until I wear my Free Hugs shirt again. I am fine with receiving (or giving) hugs like almost always. Surprise me. I at least know then that you read the RFHG. :) Like always, enjoy!

/personal permanent link Comments: 3 Flattr this

8 May 2014

Riku Voipio: Arm builder updates

Debian has recently received a donation of 8 build machines from Marvell. The new machines come with Quad core MV78460 Armada XP CPU's, DDR3 DIMM slot so we can plug in more memory, and speedy sata ports. They replace the well served Marvell MV78200 based builders - ones that have been building debian armel since 2009. We are planning a more detailed announcement, but I'll provide a quick summary: The speed increase provided by MV78460 can viewed by comparing build times on selected builds since early april: Qemu build times. We can now build Qemu in 2h instead of 16h -8x faster than before! Certainly a substantial improvement, so impressive kit from Marvell! But not all packages gain this amount of speedup: webkitgtk build times. This example, webkitgtk, builds barely 3x faster. The explanation is found from debian/rules of webkitgkt:

# Parallel builds are unstable, see #714072 and #722520
# ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
# NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
# MAKEARGUMENTS += -j$(NUMJOBS)
# endif
The old builders are single-core[1], so the regardless of parallel building, you can easily max out the cpu. New builders will use only 1 of 4 cores without parallel build support in debian/rules. During this buildd cpu usage graph, we see most time only one CPU is consumed. So for fast package build times.. make sure your packages supports parallel building. For developers, abel.debian.org is porter machine with Armada XP. It has schroot's for both armel and armhf. set "DEB_BUILD_OPTIONS=parallel=4" and off you go. Finally I'd like to thank Thomas Petazzoni, Maen Suleiman, Hector Oron, Steve McIntyre, Adam Conrad and Jon Ward for making the upgrade happen. Meanwhile, we have unrelated trouble - a bunch of disks have broken within a few days apart. I take the warranty just run out... [1] only from Linux's point of view. - mv78200 has actually 2 cores, just not SMP or coherent. You could run an RTOS on the other core while you run Linux on the other.

4 May 2014

Jonathan Wiltshire: iptables-persistent overhaul

A couple of weeks ago I finally got round to doing some major surgery on iptables-persistent. First of all it is principally now called netfilter-persistent (although the source package hasn t been renamed) and has a plugin architecture so that it can be extended by other packages. One of those packages is iptables-persistent; others may follow. This opens the way to fixing #662743 and #697088 (patches always welcome). There s also a new binary to handle loading/unloading of rules, instead of having all the logic in an init script. I was therefore able to add systemd support as a first-class unit, and I d appreciate patches for an Upstart service (as I m largely unfamiliar with it). Plugins are simply dropped into /usr/share/netfilter-persistent/plugins.d and must follow certain minimum conventions, detailed in netfilter-persistent(1). They can be any executable, so compiled or interpreted binaries are acceptable. This release finally gets the magic 1.0 identifier. It reaches Jessie today, and is already in Ubuntu Utopic. Flattr
iptables-persistent overhaul is a post from: jwiltshire.org.uk Flattr

Francois Marier: What's in a debian/ directory?

If you're looking to get started at packaging free software for Debian, you should start with the excellent New Maintainers' Guide or the Introduction to Debian Packaging on the Debian wiki. Once you know the basics, or if you prefer to learn by example, you may be interested in the full walkthrough which follows. We will look at the contents of three simple packages.

node-libravatar This package is a node.js library for the Libravatar service. Version 2.0.0-3 of that package contains the following files in its debian/ directory:
  • changelog
  • compat
  • control
  • copyright
  • docs
  • node-libravatar.install
  • rules
  • source/format
  • watch

debian/control
Source: node-libravatar
Priority: extra
Maintainer: Francois Marier <francois@debian.org>
Build-Depends: debhelper (>= 9)
Standards-Version: 3.9.4
Section: web
Homepage: https://github.com/fmarier/node-libravatar
Vcs-Git: git://git.debian.org/collab-maint/node-libravatar.git
Vcs-Browser: http://git.debian.org/?p=collab-maint/node-libravatar.git;a=summary
Package: node-libravatar
Architecture: all
Depends: $ shlibs:Depends , $ misc:Depends , nodejs
Description: libravatar library for NodeJS
 This library allows web application authors to make use of the free Libravatar
 service (https://www.libravatar.org). This service hosts avatar images for
 users and allows other sites to look them up using email addresses.
 .
 node-libravatar includes full support for federated avatar servers.
This is probably the most important file since it contains the bulk of the metadata about this package. Maintainer is a required field listing the maintainer of that package, which can be a person or a team. It only contains a single value though, any co-maintainers will be listed under the optional Uploaders field. Build-Depends lists the packages which are needed to build the package (e.g. a compiler), as opposed to those which are needed to install the binary package (e.g. a library it uses). Standards-Version refers to the version of the Debian Policy that this package complies with. The Homepage field refers to the upstream homepage, whereas the Vcs-* fields point to the repository where the packaging is stored. If you take a look at the node-libravatar packaging repository you will see that it contains three branches:
  • upstream is the source as it was in the tarball downloaded from upstream.
  • master is the upstream branch along with all of the Debian customizations.
  • pristine-tar is unrelated to the other two branches and is used by the pristine-tar tool to reconstitute the original upstream tarball as needed.
After these fields comes a new section which starts with a Package field. This is the definition of a binary package, not to be confused with the Source field at the top of this file, which refers to the name of the source package. In this particular example, they are both the same and there is only one of each, however this is not always the case, as we'll see later. Inside that binary package definition, lives the Architecture field which is normally one of these two:
  • all for a binary package that will work on all architectures but only needs to be built once
  • any for a binary package that will work everywhere but that will need to be built separately for each architecture
Finally, the last field worth pointing out is the Depends field which lists all of the runtime dependencies that the binary package has. This is what will be pulled in by apt-get when you apt-get install node-libravatar. The two variables will be substituted later by debhelper.

debian/changelog
node-libravatar (2.0.0-3) unstable; urgency=low
  * debian/watch: poll github directly
  * Bump Standards-Version up to 3.9.4
 -- Francois Marier <francois@debian.org>  Mon, 20 May 2013 12:07:49 +1200
node-libravatar (2.0.0-2) unstable; urgency=low
  * More precise license tag and upstream contact in debian/copyright
 -- Francois Marier <francois@debian.org>  Tue, 29 May 2012 22:51:03 +1200
node-libravatar (2.0.0-1) unstable; urgency=low
  * New upstream release
    - new non-backward-compatible API
 -- Francois Marier <francois@debian.org>  Mon, 07 May 2012 14:54:19 +1200
node-libravatar (1.1.1-1) unstable; urgency=low
  * Initial release (Closes: #661771)
 -- Francois Marier <francois@debian.org>  Fri, 02 Mar 2012 15:29:57 +1300
This may seem at first like a mundane file, but it is very important since it is the canonical source of the package version (2.0.0-3 in this case). This is the only place where you need to bump the package version when uploading a new package to the Debian archive. The first line also includes the distribution where the package will be uploaded. It is usually one of these values:
  • unstable for the vast majority of uploads
  • stable for uploads that have been approved by the release maintainers and fix serious bugs in the stable version of Debian
  • stable-security for security fixes to the stable version of Debian that cannot wait until the next stable point release and have been approved by the security team
Packages uploaded to unstable will migrate automatically to testing provided that a few conditions are met (e.g. no release-critical bugs were introduced). The length of time before that migration is influenced by the urgency field (low, medium or high) in the changelog entry. Another thing worth noting is that the first upload normally needs to close an ITP (Intent to Package) bug.

debian/rules
#!/usr/bin/make -f
# -*- makefile -*-
%:
    dh $@ 
override_dh_auto_test:
As can be gathered from the first two lines of this file, this is a Makefile. This is what controls how the package is built. There's not much to see and that's because most of its content is automatically added by debhelper. So let's look at it in action by building the package:
$ git buildpackage -us -uc
and then looking at parts of the build log (../node-libravatar_2.0.0-3_amd64.build):
 fakeroot debian/rules clean
dh clean 
   dh_testdir
   dh_auto_clean
   dh_clean
One of the first things we see is the debian/rules file being run with the clean target. To find out what that does, have a look at the dh_auto_clean which states that it will attempt to delete build residues and run something like make clean using the upstream Makefile.
 debian/rules build
dh build 
   dh_testdir
   dh_auto_configure
   dh_auto_build
Next we see the build target being invoked and looking at dh_auto_configure we see that this will essentially run ./configure and its equivalents. The dh_auto_build helper script then takes care of running make (or equivalent) on the upstream code. This should be familiar to anybody who has ever built a piece of free software from scratch and has encountered the usual method for building from source:
./configure
make
make install
Finally, we get to actually build the .deb:
 fakeroot debian/rules binary
dh binary 
   dh_testroot
   dh_prep
   dh_installdirs
   dh_auto_install
   dh_install
...
   dh_md5sums
   dh_builddeb
dpkg-deb: building package  node-libravatar' in  ../node-libravatar_2.0.0-3_all.deb'.
Here we see a number of helpers, including dh_auto_install which takes care of running make install. Going back to the debian/rules, we notice that there is manually defined target at the bottom of the file:
override_dh_auto_test:
which essentially disables dh_auto_test by replacing it with an empty set of commands. The reason for this becomes clear when we take a look at the test target of the upstream Makefile and the dependencies it has: tap, a node.js library that is not yet available in Debian. In other words, we can't run the test suite on the build machines so we need to disable it here.

debian/compat
9
This file simply specifies the version of debhelper that is required by the various helpers used in debian/rules. Version 9 is the latest at the moment.

debian/copyright
Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: node-libravatar
Upstream-Contact: Francois Marier <francois@libravatar.org>
Source: https://github.com/fmarier/node-libravatar
Files: *
Copyright: 2011 Francois Marier <francois@libravatar.org>
License: Expat
Files: debian/*
Copyright: 2012 Francois Marier <francois@debian.org>
License: Expat
License: Expat
 Permission is hereby granted, free of charge, to any person obtaining a copy of this
 software and associated documentation files (the "Software"), to deal in the Software
 without restriction, including without limitation the rights to use, copy, modify,
 merge, publish, distribute, sublicense, and/or sell copies of the Software, and to
 permit persons to whom the Software is furnished to do so, subject to the following
 conditions:
 .
 The above copyright notice and this permission notice shall be included in all copies
 or substantial portions of the Software.
 .
 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
 INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
 PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
 HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
 CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE
 OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
This machine-readable file lists all of the different licenses encountered in this package. It requires that the maintainer audits the upstream code for any copyright statements that might be present in addition to the license of the package as a whole.

debian/docs
README.md
This file contains a list of upstream files that will be copied into the /usr/share/doc/node-libravatar/ directory by dh_installdocs.

debian/node-libravatar.install
lib/*    usr/lib/nodejs/
The install file is used by dh_install to supplement the work done by dh_auto_install which, as we have seen earlier, essentially just runs make install on the upstream Makefile. Looking at that upstream Makefile, it becomes clear that the files will need to be installed manually by the Debian package since that Makefile doesn't have an install target.

debian/watch
version=3
https://github.com/fmarier/node-libravatar/tags /fmarier/node-libravatar/archive/node-libravatar-([0-9.]+)\.tar\.gz
This is the file that allows Debian tools like the Package Tracking System to automatically detect that a new upstream version is available. What it does is simply visit the upstream page which contains all of the release tarballs and look for links which have an href matching the above regular expression. Running uscan --report --verbose will show us all of the tarballs that can be automatically discovered using this watch file:
-- Scanning for watchfiles in .
-- Found watchfile in ./debian
-- In debian/watch, processing watchfile line:
   https://github.com/fmarier/node-libravatar/tags /fmarier/node-libravatar/archive/node-libravatar-([0-9.]+)\.tar\.gz
-- Found the following matching hrefs:
     /fmarier/node-libravatar/archive/node-libravatar-2.0.0.tar.gz
     /fmarier/node-libravatar/archive/node-libravatar-1.1.1.tar.gz
     /fmarier/node-libravatar/archive/node-libravatar-1.1.0.tar.gz
     /fmarier/node-libravatar/archive/node-libravatar-1.0.1.tar.gz
     /fmarier/node-libravatar/archive/node-libravatar-1.0.0.tar.gz
Newest version on remote site is 2.0.0, local version is 2.0.0
 => Package is up to date
-- Scan finished

pylibravatar This second package is the equivalent Python library for the Libravatar service. Version 1.6-2 of that package contains similar files in its debian/ directory, but let's look at two in particular:
  • control
  • upstream/signing-key.asc

debian/control
Source: pylibravatar
Section: python
Priority: optional
Maintainer: Francois Marier <francois@debian.org>
Build-Depends: debhelper (>= 9), python-all, python3-all
Standards-Version: 3.9.5
Homepage: https://launchpad.net/pyLibravatar
...
Package: python-libravatar
Architecture: all
Depends: $ misc:Depends , $ python:Depends , python-dns, python
Description: Libravatar module for Python 2
 Module to make use of the federated Libravatar.org avatar hosting service
 from within Python applications.
...
Package: python3-libravatar
Architecture: all
Depends: $ misc:Depends , $ python3:Depends , python3-dns, python3
Description: Libravatar module for Python 3
 Module to make use of the federated Libravatar.org avatar hosting service
 from within Python applications.
...
Here is an example of a source package (pylibravatar) which builds two separate binary packages: python-libravatar and python3-libravatar. This highlights the fact that a given upstream source can be split into several binary packages in the archive when it makes sense. In this case, there is no point in Python 2 applications pulling in the Python 3 files, so the two separate packages make sense. Another common example is the use of a -doc package to separate the documentation from the rest of a package so that it doesn't need to be installed on production servers for example.

debian/upstream/signing-key.asc
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1
mQINBEpQYz4BEAC7REQD1za69RUnkt6nRCFhSJmmoeJc+yEiWTKc9GOIMAwJDme1
+CMYgVn4Xzf1VQYwD/lE+mfWgyeMomLQjDM1mxx/LOM2a1WWPOk9+PvQwKfRJy92
...
UxDtZm/4yUmU6KvHvOGiDCMuIiB+MqhqJJ5wf80wXhzu8nmC+fyGt6nvu0ggMle8
sAMgXt/aQUTZE5zNCQ==
=RkTO
-----END PGP PUBLIC KEY BLOCK-----
This is simply the OpenPGP key that the upstream developer uses to sign release tarballs. Since PGP signatures are available on the upstream download page, it's possible to instruct uscan to check signatures before downloading tarballs. The way to do that is to use the pgpsigurlmange option in debian/watch:
version=3
opts=pgpsigurlmangle=s/$/.asc/ https://pypi.python.org/pypi/pyLibravatar https://pypi.python.org/packages/source/p/pyLibravatar/pyLibravatar-(.*)\.tar\.gz
which is simply a regular expression replacement string which takes the tarball URL and converts it to the URL of the matching PGP signature.

fcheck The last package we will look at is a file integrity checker. It essentially goes through all of the files in /usr/bin/ and /usr/lib/ and stores a hash of them in its database. When one of these files changes, you get an email. In particular, we will look at the following files in the debian/ directory of version 2.7.59-18:
  • dirs
  • fcheck.cron.d
  • fcheck.postrm
  • fcheck.postinst
  • patches/
  • README.Debian
  • rules
  • source/format

debian/patches This directory contains ten patches as well as a file called series which lists the patches that should be applied to the upstream source and in which order. Should you need to temporarily disable a patch, simply remove it from this file and it will no longer be applied at build time. Let's have a look at patches/04_cfg_sha256.patch:
Description: Switch to sha256 hash algorithm
Forwarded: not needed
Author: Francois Marier <francois@debian.org>
Last-Update: 2009-03-15
--- a/fcheck.cfg
+++ b/fcheck.cfg
@@ -149,8 +149,7 @@ TimeZone        = EST5EDT
 #$Signature      = /usr/bin/sum
 #$Signature      = /usr/bin/cksum
 #$Signature      = /usr/bin/md5sum
-$Signature      = /bin/cksum
-
+$Signature      = /usr/bin/sha256sum
 # Include an optional configuration file.
This is a very simple patch which changes the default configuration of fcheck to promote the use of a stronger hash function. At the top of the file is a bunch of metadata in the DEP-3 format. Why does this package contain so many customizations to the upstream code when Debian's policy is to push fixes upstream and work towards reduce the delta between upstream and Debian's code? The answer can be found in debian/control:
Homepage: http://web.archive.org/web/20050415074059/www.geocities.com/fcheck2000/
This package no longer has an upstream maintainer and its original source is gone. In other words, the Debian package is where all of the new bug fixes get done.

debian/source/format
3.0 (quilt)
This file contains what is called the source package format. What it basically says is that the patches found in debian/patches/ will be applied to the upstream source using the quilt tool at build time.

debian/fcheck.postrm
#!/bin/sh
# postrm script for fcheck
#
# see: dh_installdeb(1)
set -e
# summary of how this script can be called:
#        * <postrm>  remove'
#        * <postrm>  purge'
#        * <old-postrm>  upgrade' <new-version>
#        * <new-postrm>  failed-upgrade' <old-version>
#        * <new-postrm>  abort-install'
#        * <new-postrm>  abort-install' <old-version>
#        * <new-postrm>  abort-upgrade' <old-version>
#        * <disappearer's-postrm>  disappear' <overwriter>
#          <overwriter-version>
# for details, see http://www.debian.org/doc/debian-policy/ or
# the debian-policy package
case "$1" in
    remove upgrade failed-upgrade abort-install abort-upgrade disappear)
    ;;
    purge)
      if [ -e /var/lib/fcheck/fcheck.dbf ]; then
        echo "Purging old database file ..."
        rm -f /var/lib/fcheck/fcheck.dbf
      fi
      rm -rf /var/lib/fcheck
      rm -rf /var/log/fcheck
      rm -rf /etc/fcheck
    ;;
    *)
        echo "postrm called with unknown argument \ $1'" >&2
        exit 1
    ;;
esac
# dh_installdeb will replace this with shell code automatically
# generated by other debhelper scripts.
#DEBHELPER#
exit 0
This script is one of the many possible maintainer scripts that a package can provide if needed. This particular one, as the name suggests, will be run after the package is removed (apt-get remove fcheck) or purged (apt-get remove --purge fcheck). Looking at the case statement above, it doesn't do anything extra in the remove case, but it deletes a few files and directories when called with the purge argument.

debian/README.Debian This optional README file contains Debian-specific instructions that might be useful to users. It supplements the upstream README which is often more generic and cannot assume a particular system configuration.

debian/rules
#!/usr/bin/make -f
# -*- makefile -*-
# Sample debian/rules that uses debhelper.
# This file was originally written by Joey Hess and Craig Small.
# As a special exception, when this file is copied by dh-make into a
# dh-make output file, you may use that output file without restriction.
# This special exception was added by Craig Small in version 0.37 of dh-make.
# Uncomment this to turn on verbose mode.
#export DH_VERBOSE=1
build-arch:
build-indep:
build: build-stamp
build-stamp:
    dh_testdir
    pod2man --section=8 $(CURDIR)/debian/fcheck.pod > $(CURDIR)/fcheck.8
    touch build-stamp
clean:
    dh_testdir
    dh_testroot
    rm -f build-stamp 
    rm -f $(CURDIR)/fcheck.8
    dh_clean
install: build
    dh_testdir
    dh_testroot
    dh_prep
    dh_installdirs
    cp $(CURDIR)/fcheck $(CURDIR)/debian/fcheck/usr/sbin/fcheck
    cp $(CURDIR)/fcheck.cfg $(CURDIR)/debian/fcheck/etc/fcheck/fcheck.cfg
# Build architecture-independent files here.
binary-arch: build install
# Build architecture-independent files here.
binary-indep: build install
    dh_testdir
    dh_testroot
    dh_installdocs
    dh_installcron
    dh_installman fcheck.8
    dh_installchangelogs
    dh_installexamples
    dh_installlogcheck
    dh_link
    dh_strip
    dh_compress
    dh_fixperms
    dh_installdeb
    dh_shlibdeps
    dh_gencontrol
    dh_md5sums
    dh_builddeb
binary: binary-indep binary-arch
.PHONY: build clean binary-indep binary-arch binary install
This is an example of a old-style debian/rules file which you still encounter in packages which haven't yet upgraded to the latest version of debhelper 9, as can be shown by the contents of debian/compat:
8
It does essentially the same thing that what we've seen in the build log, but in a more verbose way.

debian/dirs
usr/sbin
etc/fcheck
This file contains a list of directories that dh_installdirs will create in the build directory. The reason why these directories need to be created is that files are copied into these directories in the install target of the debian/rules file. Note that this is different from directories which are created at the time of installation of the package. In that case, the directory (e.g. /var/log/fcheck/) must be created in the postinst script and removed in the postrm script.

debian/fcheck.cron.d
#
# Regular cron job for the fcheck package
#
30 */2  * * *   root    test -x /usr/sbin/fcheck && if ! nice ionice -c3 /usr/sbin/fcheck -asxrf /etc/fcheck/fcheck.cfg >/var/run/fcheck.out 2>&1; then mailx -s "ALERT: [fcheck]  hostname --fqdn " root </var/run/fcheck.out ; /usr/sbin/fcheck -cadsxlf /etc/fcheck/fcheck.cfg ; fi ; rm -f /var/run/fcheck.out
This file is the cronjob which drives the checks performed by this package. It will be copied to /etc/cron.d/fcheck by dh_installcron.

2 May 2014

Vasudev Kamath: Note to Self: How to use url_for in Flask application

url_for is normally used to avoid hard coding of URL in Flask based webapps. I've been using it in SILPA port written using Flask by. But this was written long back and written as POC to show for Santhosh author of SILPA app which even managed to go into production :-). Recently I started organizing the code and started using flask.Blueprint which was previously written using flask.views.MethodView class and here I started facing problem with old templates. I'm just gonna explain what is the difference when using url_for with MethodView and Blueprint. Lets have some sample code for MethodView first, this code is from Flask documentation.
from flask.views import MethodView
class UserAPI(MethodView):
       def get(self):
           users = User.query.all()
           ...
       def post(self):
           user = User.from_form_data(request.form)
           ...
app.add_url_rule('/users/', view_func=UserAPI.as_view('users'))
Here we have a view which is derived from MethodView which has logic on how to handle the requests. We use add_url_rule to register rule to handle /users/ url end point and we pass the class as view function which is done by as_view class method and we can refer this method in our Jinja2 templates using name users. So a statement like
url_for('users')
in our templates will be converted to /users/ URL when page is rendered to client by Flask. Now when I replaced the MethodView in favor of Blueprint I started getting up werkzeug.routing.BuildError thrown on my face and I had no clue why!. Yeah I know I'm bad at reading documentation but even after reading documentation I was still thinking that
url_for('/')
should return me a proper URL and I was wondering why its failing. Finally after re-reading documentation for url_for it was becoming clear to me, url_for definition looks like below
flask.url_for(endpoint, **values)
Here endpoint is actually a function which is supposed to be serving the URL and **values is the arguments for this function. The URL in question should be defined in python code using decorator. In my case following is the new function serving the web pages for SILPA.
bp = Blueprint('frontend', __name__)
@bp.route(_BASE_URL, defaults= 'page': 'index.html' )
@bp.route(_BASE_URL + '<page>')
def serve_pages(page):
       if page == "index.html":
          return render_template('index.html', title='SILPA',
                               main_page=_BASE_URL,
                               modules=_modulename_to_display)
       elif page == "License":
          return render_template('license.html', title='SILPA License',
                                main_page=_BASE_URL,
                                modules=_modulename_to_display)
       elif page == "Credits":
           return render_template('credits.html', title='Credits',
                                main_page=_BASE_URL,
                                modules=_modulename_to_display)
       elif page == "Contact":
           return render_template('contact.html', title='Contact SILPA Team',
                                main_page=_BASE_URL,
                                modules=_modulename_to_display)
       else:
           # modules requested!.
           if page in _display_module_map:
               return render_template(_display_module_map[page] + '.html',
                                    title=page, main_page=_BASE_URL,
                                    modules=_modulename_to_display)
           else:
               # Did we encounter something which is not registered by us?
               return abort(404)
You can ignore function code but just note the decorators, here I'm registering the function serve_pages with page as argument for URL patterns / and /<page>, _BASE_URL here is mount point of application it can be just / or /mountpoint depending on that URL registered changes. Now I need to modify code for all url_for in my template to look like below
url_for('.serve_pages', page='/License') # for /License
url_for('.serve_pages') # which will turn in to /index.html
The . in front of function is for referring current Blueprint, in my case the Flask will consider it as frontend.serve_pages as function name and generates appropriate URL at run time. So my fault was misunderstanding endpoint argument as URL endpoint but where as its actually function name supposed to serve the page. But when using MethodViews I can simply convert class to a function with my preferred name just like UserAPI.as_view('/') so url_for('/') just works.

28 April 2014

Evgeni Golov: Debian Bug Squashing Party Salzburg 2014

bsp2014_small This weekend, Bernd Zeimetz organized a BSP at the offices of conova in Salzburg, Austria. Three days of discussions, bugfixes, sparc removals and a lot of fun and laughter. We squashed a total of 87 bugs: 66 bugs affecting Jessie/Sid were closed, 9 downgraded and 8 closed via removals. As people tend to care about (old)stable, 3 bugs were fixed in Wheezy and one in Squeeze. These numbers might be not totaly correct, as were kinda creative at counting Marga promised a talk about an introduction to properly counting bugs using the Haus vom Nikolaus algorithm to the base of 7 . IMG_20140427_182902 Speaking of numbers, I touched the following bugs (not all RC): A couple of (non-free) pictures are available at Uwe s salzburg-cityguide.at. Thanks again to Bernd for organizing and conova and credativ for sponsoring!

27 April 2014

Vincent Bernat: Local corporate APT repositories

Distributing software efficiently accross your platform can be difficult. Every distribution comes with a package manager which is usually suited for this task. APT can be relied upon on when using Debian or a derivative. Unfortunately, the official repositories may not contain everything you need. When you require unpackaged software or more recent versions, it is possible to setup your own local repository. Most of what is presented here was setup for Dailymotion and was greatly inspired by the work done by Rapha l Pinson at Orange.

Setting up your repositories There are three kinds of repositories you may want to setup:
  1. A distribution mirror. Such a mirror will save bandwidth, provide faster downloads and permanent access, even when someone searches Google on Google.
  2. A local repository for your own packages with the ability to have a staging zone to test packages on some servers before putting them in production.
  3. Mirrors for unofficial repositories, like Ubuntu PPA. To avoid unexpected changes, such a repository will also get a staging and a production zone.
Before going further, it is quite important to understand what a repository is. Let s illustrate with the following line from my /etc/apt/sources.list:
deb http://ftp.debian.org/debian/ unstable main contrib non-free
In this example, http://ftp.debian.org/debian/ is the repository and unstable is the distribution. A distribution is subdivided into components. We have three components: main, contrib and non-free. To setup repositories, we will use reprepro. This is not the only solution but it has a good balance between versatility and simplicity. reprepro can only handle one repository. So, the first choice is about how you will split your packages in repositories, distributions and components. Here is what matters:
  • A repository cannot contain two identical packages (same name, same version, same architecture).
  • Inside a component, you can only have one version of a package.
  • Usually, a distribution is a subset of the versions while a component is a subset of the packages. For example, in Debian, with the distribution unstable, you choose to get the most recent versions while with the component main, you choose to get DFSG-free software only.
If you go for several repositories, you will have to handle several reprepro instances and won t be able to easily copy packages from one place to another. At Dailymotion, we put everything in the same repository but it would also be perfectly valid to have three repositories:
  • one to mirror the distribution,
  • one for your local packages, and
  • one to mirror unofficial repositories.
Here is our target setup: Local APT repository

Initial setup First, create a system user to work with the repositories:
$ adduser --system --disabled-password --disabled-login \
>         --home /srv/packages \
>         --group reprepro
All operations should be done with this user only. If you want to setup several repositories, create a directory for each of them. Each repository has those subdirectories:
  • conf/ contains the configuration files,
  • gpg/ contains the GPG stuff to sign the repository1,
  • logs/ contains the logs,
  • www/ contains the repository that should be exported by the web server.
Here is the content of conf/options:
outdir +b/www
logdir +b/logs
gnupghome +b/gpg
Then, you need to create the GPG key to sign the repository:
$ GNUPGHOME=gpg gpg --gen-key
Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 10y
Key expires at mer. 08 nov. 2023 22:30:58 CET
Is this correct? (y/N) y
Real name: Dailymotion Archive Automatic Signing Key
Email address: the-it-operations@dailymotion.com
Comment: 
[...]
By setting an empty password, you allow reprepro to run unattended. You will have to distribute the public key of your new repository to let APT check the archive signature. An easy way is to ship it in some package.

Local mirror of an official distribution Let s start by mirroring a distribution. We want a local mirror of Ubuntu Precise. For this, we need to do two things:
  1. Setup a new distribution in conf/distributions.
  2. Configure the update sources in conf/updates.
Let s add this block to conf/distributions:
# Ubuntu Precise
Origin: Ubuntu
Label: Ubuntu
Suite: precise
Version: 12.04
Codename: precise
Architectures: i386 amd64
Components: main restricted universe multiverse
UDebComponents: main restricted universe multiverse
Description: Ubuntu Precise 12.04 (with updates and security)
Contents: .gz .bz2
UDebIndices: Packages Release . .gz
Tracking: minimal
Update: - ubuntu-precise ubuntu-precise-updates ubuntu-precise-security
SignWith: yes
This defines the precise distribution in our repository. It contains four components: main, restricted, universe and multiverse (like the regular distribution in official repositories). The Update line starts with a dash. This means reprepro will mark everything as deleted before updating with the provided sources. Old packages will not be kept when they are removed from Ubuntu. In conf/updates, we define the sources:
# Ubuntu Precise
Name: ubuntu-precise
Method: http://fr.archive.ubuntu.com/ubuntu
Fallback: http://de.archive.ubuntu.com/ubuntu
Suite: precise
Components: main main multiverse restricted universe
UDebComponents: main restricted universe multiverse
Architectures: amd64 i386
VerifyRelease: 437D05B5
GetInRelease: no
# Ubuntu Precise Updates
Name: ubuntu-precise-updates
Method: http://fr.archive.ubuntu.com/ubuntu
Fallback: http://de.archive.ubuntu.com/ubuntu
Suite: precise-updates
Components: main restricted universe multiverse
UDebComponents: main restricted universe multiverse
Architectures: amd64 i386
VerifyRelease: 437D05B5
GetInRelease: no
# Ubuntu Precise Security
Name: ubuntu-precise-security
Method: http://fr.archive.ubuntu.com/ubuntu
Fallback: http://de.archive.ubuntu.com/ubuntu
Suite: precise-security
Components: main restricted universe multiverse
UDebComponents: main restricted universe multiverse
Architectures: amd64 i386
VerifyRelease: 437D05B5
GetInRelease: no
The VerifyRelease lines are GPG key fingerprint to use to check the remote repository. The key needs to be imported in the local keyring:
$ gpg --keyring /usr/share/keyrings/ubuntu-archive-keyring.gpg \
>     --export 437D05B5   GNUPGHOME=gpg gpg --import
Another important point is that we merge three distributions (precise, precise-updates and precise-security) into a single distribution (precise) in our local repository. This may cause some difficulties with tools expecting the three distributions to be available (like the Debian Installer2). Next, you can run reprepro and ask it to update your local mirror:
$ reprepro update
This will take some time on the first run. You can execute this command every night. reprepro is not the fastest mirror solution but it is easy to setup, flexible and reliable.

Repository for local packages Let s configure the repository to accept local packages. For each official distribution (like precise), we will configure two distributions:
  • precise-staging contains packages that have not been fully tested and not ready to go to production.
  • precise-prod contains production packages copied from precise-staging.
In our workflow, packages are introduced in precise-staging where they can be tested and will be copied to precise-prod when we want them to be available for production. You can adopt a more complex workflow if you need. The reprepro part is quite easy. We add the following blocks into conf/distributions:
# Dailymotion Precise packages (staging)
Origin: Dailymotion #  
Label: dm-staging   #  
Suite: precise-staging
Codename: precise-staging
Architectures: i386 amd64 source
Components: main role/dns role/database role/web #  
Description: Dailymotion Precise staging repository
Contents: .gz .bz2
Tracking: keep
SignWith: yes
NotAutomatic: yes #  
Log: packages.dm-precise-staging.log
 --type=dsc email-changes
# Dailymotion Precise packages (prod)
Origin: Dailymotion #  
Label: dm-prod      #  
Suite: precise-prod
Codename: precise-prod
Architectures: i386 amd64 source
Components: main role/dns role/database role/web #  
Description: Dailymotion Precise prod repository
Contents: .gz .bz2
Tracking: keep
SignWith: yes
Log: packages.dm-precise-prod.log
First notice we use several components (in ):
  • main will contain packages that are not specific to a subset of the platform. If you put a package in main, it should work correctly on any host.
  • role/* are components dedicated to a subset of the platform. For example, in role/dns, we ship a custom version of BIND.
The staging distribution has the NotAutomatic flag (in ) which disallows the package manager to install those packages except if the user explicitely requests it. Just below, when a new dsc file is uploaded, the hook email-changes will be executed. It should be in the conf/ directory. The Origin and Label lines (in ) are quite important to be able to define an explicit policy of which packages should be installed. Let s say we use the following /etc/apt/sources.list file:
# Ubuntu packages
deb http://packages.dm.gg/dailymotion precise main restricted universe multiverse
# Dailymotion packages
deb http://packages.dm.gg/dailymotion precise-prod    main role/dns
deb http://packages.dm.gg/dailymotion precise-staging main role/dns
All servers have the precise-staging distribution. We must ensure we won t install those packages by mistake. The NotAutomatic flag is one possible safe-guard. We also use a tailored /etc/apt/preferences:
Explanation: Dailymotion packages of a specific component should be more preferred
Package: *
Pin: release o=Dailymotion, l=dm-prod, c=role/*
Pin-Priority: 950
Explanation: Dailymotion packages should be preferred
Package: *
Pin: release o=Dailymotion, l=dm-prod
Pin-Priority: 900
Explanation: staging should never be preferred
Package: *
Pin: release o=Dailymotion, l=dm-staging
Pin-Priority: -100
By default, packages will have a priority of 500. By setting a priority of -100 to the staging distribution, we ensure the packages cannot be installed at all. This is stronger than NotAutomatic which sets the priority to 1. When a package exists in Ubuntu and in our local repository, we ensure that, if this is a production package, we will use ours by using a priority of 900 (or 950 if we match a specific role component). Have a look at the How APT Interprets Priorities section of apt_preferences(5) manual page for additional information. Keep in mind that version matters only when the priority is the same. To check if everything works as you expect, use apt-cache policy:
$ apt-cache policy php5-memcache
  Installed: 3.0.8-1~precise2~dm1
  Candidate: 3.0.8-1~precise2~dm1
  Version table:
 *** 3.0.8-1~precise2~dm1 0
        950 http://packages.dm.gg/dailymotion/ precise-prod/role/web amd64 Packages
        100 /var/lib/dpkg/status
     3.0.8-1~precise1~dm4 0
        900 http://packages.dm.gg/dailymotion/ precise-prod/main amd64 Packages
       -100 http://packages.dm.gg/dailymotion/ precise-staging/main amd64 Packages
     3.0.6-1 0
        500 http://packages.dm.gg/dailymotion/ precise/universe amd64 Packages
If we want to install a package from the staging distribution, we can use apt-get with the -t precise-staging option to raise the priority of this distribution to 990. Once you have tested your package, you can copy it from the staging distribution to the production distribution:
$ reprepro -C main copysrc precise-prod precise-staging wackadoodle

Local mirror of third-party repositories Sometimes, you want a software published on some third-party repository without to repackage it yourself. A common example is the repositories edited by hardware vendors. Like for an Ubuntu mirror, there are two steps: defining the distribution and defining the source. We chose to put such mirrors into the same distributions as our local packages but with a dedicated component for each mirror. This way, those third-party packages will share the same workflow as our local packages: they will appear in the staging distribution, we validate them and copy them to the production distribution. The first step is to add the components and an appropriate Update line to conf/distributions:
Origin: Dailymotion
Label: dm-staging
Suite: precise-staging
Components: main role/dns role/database role/web vendor/hp
Update: hp
# [...]
Origin: Dailymotion
Label: dm-prod
Suite: precise-prod
Components: main role/dns role/database role/web vendor/hp
# [...]
We added the vendor/hp component to both the staging and the production distributions. However, only the staging distribution gets an Update line (remember, packages will be copied manually into the production distribution). We declare the source in conf/updates:
# HP repository
Name: hp
Method: http://downloads.linux.hp.com/SDR/downloads/ManagementComponentPack/
Suite: precise/current
Components: non-free>vendor/hp
Architectures: i386 amd64
VerifyRelease: 2689B887
GetInRelease: no
Don t forget to add the GPG key to your local keyring. Notice an interesting feature of reprepro: we copy the remote non-free component to our local vendor/hp component. Then, you can synchronize the mirror with reprepro update. Once the packages have been tested, you will have to copy them in the production distribution.

Building Debian packages Our reprepro setup seems complete, but how do we put packages into the staging distribution? You have several options to build Debian packages for your local repository. It really depends on how much time you want to invest in this activity:
  1. Build packages from source by adding a debian/ directory. This is the classic way of building Debian packages. You can start from scratch or use an existing package as a base. In the latest case, the package can be from the official archive but for a more recent distribution or a backport or from an unofficial repository.
  2. Use a tool that will create a binary package from a directory, like fpm. Such a tool will try to guess a lot of things to minimize your work. It can even download everything for you.
There is no universal solution. If you don t have the time budget for building packages from source, have a look at fpm. I would advise you to use the first approach when possible because you will get those perks for free:
  • You keep the sources in your repository. Whenever you need to rebuild something to fix an emergency bug, you won t have to hunt the sources which may be unavailable when you need them the most. Of course, this only works if you build packages that don t download stuff directly from the Internet.
  • You also keep the recipe3 to build the package in your repository. If someone enables some option and rebuild the package, you won t accidently drop this option on the next build. Those changes can be documented in debian/changelog. Moreover, you can use a version control software for the whole debian/ directory.
  • You can propose your package for inclusion into Debian. This will help many people once the package hits the archive.

Builders We chose pbuilder as a builder4. Its setup is quite straightforward. Here is our /etc/pbuilderrc:
DISTRIBUTION=$DIST
NAME="$DIST-$ARCH"
MIRRORSITE=http://packages.dm.gg/dailymotion
COMPONENTS=("main" "restricted" "universe" "multiverse")
OTHERMIRROR="deb http://packages.dm.gg/dailymotion $ DIST -staging main"
HOOKDIR=/etc/pbuilder/hooks.d
BASE=/var/cache/pbuilder/dailymotion
BASETGZ=$BASE/$NAME/base.tgz
BUILDRESULT=$BASE/$NAME/results/
APTCACHE=$BASE/$NAME/aptcache/
DEBBUILDOPTS="-sa"
KEYRING="/usr/share/keyrings/dailymotion-archive.keyring.gpg"
DEBOOTSTRAPOPTS=("--arch" "$ARCH" "--variant=buildd" "$ DEBOOTSTRAPOPTS[@] " "--keyring=$KEYRING")
APTKEYRINGS=("$KEYRING")
EXTRAPACKAGES=("dailymotion-archive-keyring")
pbuilder is expected to be invoked with DIST, ARCH and optionally ROLE environment variables. Building the initial bases can be done like this:
for ARCH in i386 amd64; do
  for DIST in precise; do
    export ARCH
    export DIST
    pbuilder --create
  done
done
We don t create a base for each role. Instead, we use a D hook to add the appropriate source:
#!/bin/bash
[ -z "$ROLE" ]    
  cat >> /etc/apt/sources.list <<EOF
deb http://packages.dm.gg/dailymotion $ DIST -staging role/$ ROLE 
EOF
 
apt-get update
We ensure packages from our staging distribution are preferred over other packages by adding an /etc/apt/preferences file in a E hook:
#!/bin/bash
cat > /etc/apt/preferences <<EOF
Explanation: Dailymotion packages are of higher priority
Package: *
Pin: release o=Dailymotion
Pin-Priority: 900
EOF
We also use a C hook to get a shell in case there is an error. This is convenient to debug a problem:
#!/bin/bash
apt-get install -y --force-yes vim less
cd /tmp/buildd/*/debian/..
/bin/bash < /dev/tty > /dev/tty 2> /dev/tty
A manual build can be run with:
$ ARCH=amd64 DIST=precise ROLE=web pbuilder \
>         --build somepackage.dsc

Version numbering To avoid to apply complex rules to chose a version number for a package, we chose to treat everything as a backport, even in-house software. We use the following scheme: X-Y~preciseZ+dmW.
  • X is the upstream version5.
  • Y is the Debian version. If there is no Debian version, use 0.
  • Z is the Ubuntu backport version. Again, if such a version doesn t exist, use 0.
  • W is our version of the package. We increment it when we make a change to the packaging. This is the only number we are allowed to control. All the others are set by an upstream entity, unless it doesn t exist and in this case, you use 0.
Let s suppose you need to backport wackadoodle. It is available in a more recent version of Ubuntu as 1.4-3. Your first backport will be 1.4-3~precise0+dm1. After a change to the packaging, the version will be 1.4-3~precise0+dm2. A new upstream version 1.5 is available and you need it. You will use 1.5-0~precise0+dm1. Later, this new upstream version will be available in some version of Ubuntu as 1.5-3ubuntu1. You will rebase your changes on this version and get 1.5-3ubuntu1~precise0+dm1. When using Debian instead of Ubuntu, a compatible convention could be : X-Y~bpo70+Z~dm+W.

Uploading To upload a package, a common setup is the following workflow:
  1. Upload the source package to an incoming directory.
  2. reprepro will notice the source package, check its correctness (signature, distribution) and put it in the archive.
  3. The builder will notice a new package needs to be built and build it.
  4. Once the package is built, the builder will upload the result to the incoming directory.
  5. reprepro will notice again the new binary package and integrate it in the archive.
This workflow has the disadvantage to have many moving pieces and to leave the user in the dark while the compilation is in progress. As an alternative, a simple script can be used to execute each step synchronously. The user can follow on their terminal that everything works as expected. Once we have the .changes file, the build script just issues the appropriate command to include the result in the archive:
$ reprepro -C main include precise-staging \
>      wackadoodle_1.4-3~precise0+dm4_amd64.changes
Happy hacking!

  1. The gpg/ directory could be shared by several repositories.
  2. We teached Debian Installer to work with our setup with an appropriate preseed file.
  3. fpm-cookery is a convenient tool to write recipes for fpm, similar to Homebrew or a BSD port tree. It could be used to achieve the same goal.
  4. sbuild is an alternative to pbuilder and is the official builder for both Debian and Ubuntu. Historically, pbuilder was more focused on developers needs.
  5. For a Git snapshot, we use something like 1.4-git20130905+1-ae42dc1 which is a snapshot made after version 1.4 (use 0.0 if no version has ever been released) at the given date. The following 1 is to be able to package different snapshots at the same date while the hash is here in case you need to retrieve the exact snapshot.

24 April 2014

Daniel Pocock: Android betrays tethering data

When I upgraded an Android device the other day, I found that tethering completely stopped working. The updated CyanogenMod had inherited a new bug from Android, informing the carrier that I was tethering. The carrier, Vodafone Italy, had decided to make my life miserable by blocking that traffic. I had a closer look and managed to find a workaround. There is absolutely no difference, from a technical perspective, between data transmitted from a mobile device on-screen application and data transmitted from tethering. Revealing the use of tethering to the carrier is a massive breach of privacy - yet comments in the Google bug tracker suggest it is a feature rather than a bug. This little table helps put that logic in perspective:
Product Person who carries handset
User Mobile network who wants to discriminate against some types of network traffic to squeeze more money out of the Product
Feature Revealing private information about the way the Product uses his/her Internet so the real User can profit.
It is also bad news for the environment: many people are being tricked into buying un-needed USB dongle modems that will end up in the rubbish in 1-2 years time when their contract expires and the company pushes them to upgrade to the next best thing. Behind the scenes What does it really mean in practice, how does Android tell your carrier which data is from tethering? As my device is rooted and as it is my device and I will do what I want with it, I decided to have a look inside. The ip command revealed that there are now two network devices, rmmnet_usb0 and rmmnet_usb1. The basic ip route command reveals that traffic from different source addresses is handled differently and split over the different network devices:
shell@android:/ # ip route
0.0.0.0/1 dev tun0  scope link
default via 100.66.150.89 dev rmnet_usb0
83.224.66.138 via 100.87.31.214 dev rmnet_usb1
83.224.70.94 via 100.87.31.214 dev rmnet_usb1
100.66.150.88/30 dev rmnet_usb0  proto kernel  scope link  src 100.66.150.90
100.66.150.89 dev rmnet_usb0  scope link
100.87.31.212/30 dev rmnet_usb1  proto kernel  scope link  src 100.87.31.213
100.87.31.214 dev rmnet_usb1  scope link
128.0.0.0/1 dev tun0  scope link
192.168.42.0/24 dev rndis0  proto kernel  scope link  src 192.168.42.129
I then looked more closely and found that there is also an extra routing table, it can be found with ip rule
shell@android:/ # ip rule show
0:      from all lookup local
32765:  from 192.168.42.0/24 lookup 60
32766:  from all lookup main
32767:  from all lookup default
shell@android:/ # ip route show table 60
default via 100.87.51.57 dev rmnet_usb1
100.87.51.57 dev rmnet_usb1
192.168.42.0/24 dev rndis0  scope link
In this routing table, it is obvious that data from the tethering subnet (192.168.42.0/24) is sent over the extra device rmnet_usb1. Manually cleaning it up If the phone is rooted, it is possible to very quickly change the routing table to get all the tethering traffic going through the normal rmnet_usb0 interface again. It is necessary to get rid of that alternative routing table first:
ip rule del pref 32765
and then update the iptables entries that refer to interface names:
iptables -t nat -I natctrl_nat_POSTROUTING -s 192.168.0.0/16 -o rmnet_usb0 -j MASQUERADE
iptables -I natctrl_FORWARD -i rmmnet_usb0 -j RETURN
This immediately resolved the problem for me on the Vodafone network in Italy. Conclusion If Google can be bullied into accepting this kind of discriminatory routing in the stock Android builds and it can even propagate into CyanogenMod, then I'm just glad I'm not running one of those Android builds that has been explicitly "enhanced" by a carrier. It raises big questions about who really is the owner and user of the device and who is receiving the value when a person pays money to "buy" a device.

16 April 2014

Richard Hartmann: secure password storage

Dear lazyweb, for obvious reaons I am in the process of cycling out a lot of passwords. For the last decade or so, I have been using openssl.vim to store less-frequently-used passwords and it's still working fine. Yet, it requires some manual work, not least of which manually adding random garbage at the start of the plain text (and in other places) every time I save my passwords. In the context of changing a lot of passwords at once, this has started to become tedious. Plus, I am not sure if a tool of the complexity and feature-set of Vim is the best choice for security-critical work on encrypted files. Long story short, I am looking for alternatives. I did some research but couldn't come up with anything I truly liked; as there's bound to be tools which fit the requirements of like-minded people, I decided to ask around a bit. My personal short-list of requirements is: Any and all feedback appreciated. Depending on the level of feedback, I may summarize my own findings and suggestions into a follow-up post.

David Pashley: Bad Password Policies

After the whole Heartbleed fiasco, I ve decided to continue my march towards improving my online security. I d already begun the process of using LastPass to store my passwords and generate random passwords for each site, but I hadn t completed the process, with some sites still using the same passwords, and some having less than ideal strength passwords, so I spent some time today improving my password position. Here s some of the bad examples of password policy I ve discovered today. First up we have Live.com. A maximum of 16 characters from the Microsoft auth service. Seems to accept any character though. Screenshot from 2014-04-15 21:36:57 This excellent example is from creditexpert.co.uk, one of the credit agencies here in the UK. They not only restrict to 20 characters, they restrict you to @, ., _ or . So much for teaching people how to protect themselves online. Screenshot from 2014-04-15 17:38:28 Here s Tesco.com after attempting to change my password to QvHn#9#kDD%cdPAQ4&b&ACb4x%48#b . If you can figure out how this violates their rules, I d love to know. And before you ask, I tried without numbers and that still failed so it can t be the three and only three thing. The only other idea might be that they meant i.e. rather than e.g. , but I didn t test that. Screenshot from 2014-04-15 16:20:17 Edit: Here is a response from Tesco on Twitter: Screenshot from 2014-04-16 07:47:58 Here s a poor choice from ft.com, refusing to accept non-alphanumeric characters. On the plus side they did allow the full 30 characters in the password. Screenshot from 2014-04-15 15:22:08 The finest example of a poor security policy is a company who will remain nameless due to their utter lack of security. Not only did they not use HTTPS, they accepted a 30 character password and silently truncated it to 20 characters. The reason I know this is because when I logged out and tried to log in again and then used the forgot my password option, they emailed me the password in plain text. I have also been setting up two-factor authentication where possible. Most sites use the Google Authenticator application on your mobile to give you a 6 digit code to type in in addition to your password. I highly recommend you set it up too. There s a useful list of sites that implement 2FA and links to their documentation at http://twofactorauth.org/. I realise that my choice LastPass requires me to trust them, but I think the advantages outweigh the disadvantages of having many sites using the same passwords and/or low strength passwords. I know various people cleverer than me have looked into their system and failed to find any obvious flaws. Remember people, when you implement a password, allow the following things: If you are going to place restrictions, please make sure the documentation matches the implementation, provide a client-side implementation to match and provide quick feedback to the user, and make sure you explicitly say what is wrong with the password, rather than referring back to the incorrect documentation. There are also many JS password strength meters available to show how secure the inputted passwords are. They are possibly a better way of providing feedback about security than having arbitrary policies that actually harm your security. As someone said to me on twitter, it s not like password is too strong was ever a bad thing. The post Bad Password Policies appeared first on David Pashley.com.

15 April 2014

B lint R czey: Proposing amd64-hardened architecture for Debian

Facing last week s Heartbleed bug the need for improving the security of our systems became more apparent than usually. In Debian there are widely used methods for Hardening packages at build time and guidelines for improving the default installations security. Employing such methods usually come at an expense, for example slower code execution of binaries due to additional checks or additional configuration steps when setting up a system. Balancing between usability and security Debian chose an approach which would satisfy the most users by using C/C++ features which only slightly decrease execution speed of built binaries and by using reasonable defaults in package installations. All the architectures supported by Debian aims using the same methods for enhancing security but it does not have to stay the same way. Amd64 is the most widely used architecture of Debian according to popcon and amd64 hardware comes with powerful CPU-s. I think there would be a significant amount of people (being one of them :-)) who would happily use a version of Debian with more security features enabled by default sacrificing some CPU power and installing and setting up additional packages. My proposal for serving those security-focused users is introducing a new architecture targeting amd64 hardware, but with more security related C/C++ features turned on for every package (currently hardening has to be enabled by the maintainers in some way) through compiler flags as a start. Introducing the new architecture would also let package maintainers enabling additional dependencies and build rules selectively for the new architecture improving the security further. On the users side the advantage of having a separate security enhanced architecture instead of a Debian derivative is the potential of installing a set of security enhanced packages using multiarch. You could have a fast amd64 installation as a base and run Apache or any other sensitive server from the amd64-hardened packages! I have sent the proposal for discussion to debian-dev, too. Please join the discussion there or leave a comment here. Update: Many of you wondered if amd64-hardened could have prevented the exploitation of the Heartbleed vulnerability. I have posted a proof of concept to show that using -fsanitize=address and disabling custom freelist would have protected systems against stealing data using the exploits. Disabling the custom freelist-like solutions and enabling-fsanitize=address would be part of amd64-hardened to make memory protection techniques work effectively thus I think if we had this architecture ready at the beginning of April, it would have been immune to Heartbleed.

Colin Watson: Porting GHC: A Tale of Two Architectures

We had some requests to get GHC (the Glasgow Haskell Compiler) up and running on two new Ubuntu architectures: arm64, added in 13.10, and ppc64el, added in 14.04. This has been something of a saga, and has involved rather more late-night hacking than is probably good for me. Book the First: Recalled to a life of strange build systems You might not know it from the sheer bulk of uploads I do sometimes, but I actually don't speak a word of Haskell and it's not very high up my list of things to learn. But I am a pretty experienced build engineer, and I enjoy porting things to new architectures: I'm firmly of the belief that breadth of architecture support is a good way to shake out certain categories of issues in code, that it's worth doing aggressively across an entire distribution, and that, even if you don't think you need something now, new requirements have a habit of coming along when you least expect them and you might as well be prepared in advance. Furthermore, it annoys me when we have excessive noise in our build failure and proposed-migration output and I often put bits and pieces of spare time into gardening miscellaneous problems there, and at one point there was a lot of Haskell stuff on the list and it got a bit annoying to have to keep sending patches rather than just fixing things myself, and ... well, I ended up as probably the only non-Haskell-programmer on the Debian Haskell team and found myself fixing problems there in my free time. Life is a bit weird sometimes. Bootstrapping packages on a new architecture is a bit of a black art that only a fairly small number of relatively bitter and twisted people know very much about. Doing it in Ubuntu is specifically painful because we've always forbidden direct binary uploads: all binaries have to come from a build daemon. Compilers in particular often tend to be written in the language they compile, and it's not uncommon for them to build-depend on themselves: that is, you need a previous version of the compiler to build the compiler, stretching back to the dawn of time where somebody put things together with a big magnet or something. So how do you get started on a new architecture? Well, what we do in this case is we construct a binary somehow (usually involving cross-compilation) and insert it as a build-dependency for a proper build in Launchpad. The ability to do this is restricted to a small group of Canonical employees, partly because it's very easy to make mistakes and partly because things like the classic "Reflections on Trusting Trust" are in the backs of our minds somewhere. We have an iron rule for our own sanity that the injected build-dependencies must themselves have been built from the unmodified source package in Ubuntu, although there can be source modifications further back in the chain. Fortunately, we don't need to do this very often, but it does mean that as somebody who can do it I feel an obligation to try and unblock other people where I can. As far as constructing those build-dependencies goes, sometimes we look for binaries built by other distributions (particularly Debian), and that's pretty straightforward. In this case, though, these two architectures are pretty new and the Debian ports are only just getting going, and as far as I can tell none of the other distributions with active arm64 or ppc64el ports (or trivial name variants) has got as far as porting GHC yet. Well, OK. This was somewhere around the Christmas holidays and I had some time. Muggins here cracks his knuckles and decides to have a go at bootstrapping it from scratch. It can't be that hard, right? Not to mention that it was a blocker for over 600 entries on that build failure list I mentioned, which is definitely enough to make me sit up and take notice; we'd even had the odd customer request for it. Several attempts later and I was starting to doubt my sanity, not least for trying in the first place. We ship GHC 7.6, and upgrading to 7.8 is not a project I'd like to tackle until the much more experienced Haskell folks in Debian have switched to it in unstable. The porting documentation for 7.6 has bitrotted more or less beyond usability, and the corresponding documentation for 7.8 really isn't backportable to 7.6. I tried building 7.8 for ppc64el anyway, picking that on the basis that we had quicker hardware for it and didn't seem likely to be particularly more arduous than arm64 (ho ho), and I even got to the point of having a cross-built stage2 compiler (stage1, in the cross-building case, is a GHC binary that runs on your starting architecture and generates code for your target architecture) that I could copy over to a ppc64el box and try to use as the base for a fully-native build, but it segfaulted incomprehensibly just after spawning any child process. Compilers tend to do rather a lot, especially when they're built to use GCC to generate object code, so this was a pretty serious problem, and it resisted analysis. I poked at it for a while but didn't get anywhere, and I had other things to do so declared it a write-off and gave up. Book the Second: The golden thread of progress In March, another mailing list conversation prodded me into finding a blog entry by Karel Gardas on building GHC for arm64. This was enough to be worth another look, and indeed it turned out that (with some help from Karel in private mail) I was able to cross-build a compiler that actually worked and could be used to run a fully-native build that also worked. Of course this was 7.8, since as I mentioned cross-building 7.6 is unrealistically difficult unless you're considerably more of an expert on GHC's labyrinthine build system than I am. OK, no problem, right? Getting a GHC at all is the hard bit, and 7.8 must be at least as capable as 7.6, so it should be able to build 7.6 easily enough ... Not so much. What I'd missed here was that compiler engineers generally only care very much about building the compiler with older versions of itself, and if the language in question has any kind of deprecation cycle then the compiler itself is likely to be behind on various things compared to more typical code since it has to be buildable with older versions. This means that the removal of some deprecated interfaces from 7.8 posed a problem, as did some changes in certain primops that had gained an associated compatibility layer in 7.8 but nobody had gone back to put the corresponding compatibility layer into 7.6. GHC supports running Haskell code through the C preprocessor, and there's a __GLASGOW_HASKELL__ definition with the compiler's version number, so this was just a slog tracking down changes in git and adding #ifdef-guarded code that coped with the newer compiler (remembering that stage1 will be built with 7.8 and stage2 with stage1, i.e. 7.6, from the same source tree). More inscrutably, GHC has its own packaging system called Cabal which is also used by the compiler build process to determine which subpackages to build and how to link them against each other, and some crucial subpackages weren't being built: it looked like it was stuck on picking versions from "stage0" (i.e. the initial compiler used as an input to the whole process) when it should have been building its own. Eventually I figured out that this was because GHC's use of its packaging system hadn't anticipated this case, and was selecting the higher version of the ghc package itself from stage0 rather than the version it was about to build for itself, and thus never actually tried to build most of the compiler. Editing ghc_stage1_DEPS in ghc/stage1/package-data.mk after its initial generation sorted this out. One late night building round and round in circles for a while until I had something stable, and a Debian source upload to add basic support for the architecture name (and other changes which were a bit over the top in retrospect: I didn't need to touch the embedded copy of libffi, as we build with the system one), and I was able to feed this all into Launchpad and watch the builders munch away very satisfyingly at the Haskell library stack for a while. This was all interesting, and finally all that work was actually paying off in terms of getting to watch a slew of several hundred build failures vanish from arm64 (the final count was something like 640, I think). The fly in the ointment was that ppc64el was still blocked, as the problem there wasn't building 7.6, it was getting a working 7.8. But now I really did have other much more urgent things to do, so I figured I just wouldn't get to this by release time and stuck it on the figurative shelf. Book the Third: The track of a bug Then, last Friday, I cleared out my urgent pile and thought I'd have another quick look. (I get a bit obsessive about things like this that smell of "interesting intellectual puzzle".) slyfox on the #ghc IRC channel gave me some general debugging advice and, particularly usefully, a reduced example program that I could use to debug just the process-spawning problem without having to wade through noise from running the rest of the compiler. I reproduced the same problem there, and then found that the program crashed earlier (in stg_ap_0_fast, part of the run-time system) if I compiled it with +RTS -Da -RTS. I nailed it down to a small enough region of assembly that I could see all of the assembly, the source code, and an intermediate representation or two from the compiler, and then started meditating on what makes ppc64el special. You see, the vast majority of porting bugs come down to what I might call gross properties of the architecture. You have things like whether it's 32-bit or 64-bit, big-endian or little-endian, whether char is signed or unsigned, that sort of thing. There's a big table on the Debian wiki that handily summarises most of the important ones. Sometimes you have to deal with distribution-specific things like whether GL or GLES is used; often, especially for new variants of existing architectures, you have to cope with foolish configure scripts that think they can guess certain things from the architecture name and get it wrong (assuming that powerpc* means big-endian, for instance). We often have to update config.guess and config.sub, and on ppc64el we have the additional hassle of updating libtool macros too. But I've done a lot of this stuff and I'd accounted for everything I could think of. ppc64el is actually a lot like amd64 in terms of many of these porting-relevant properties, and not even that far off arm64 which I'd just successfully ported GHC to, so I couldn't be dealing with anything particularly obvious. There was some hand-written assembly which certainly could have been problematic, but I'd carefully checked that this wasn't being used by the "unregisterised" (no specialised machine dependencies, so relatively easy to port but not well-optimised) build I was using. A problem around spawning processes suggested a problem with SIGCHLD handling, but I ruled that out by slowing down the first child process that it spawned and using strace to confirm that SIGSEGV was the first signal received. What on earth was the problem? From some painstaking gdb work, one thing I eventually noticed was that stg_ap_0_fast's local stack appeared to be being corrupted by a function call, specifically a call to the colourfully-named debugBelch. Now, when IBM's toolchain engineers were putting together ppc64el based on ppc64, they took the opportunity to fix a number of problems with their ABI: there's an OpenJDK bug with a handy list of references. One of the things I noticed there was that there were some stack allocation optimisations in the new ABI, which affected functions that don't call any vararg functions and don't call any functions that take enough parameters that some of them have to be passed on the stack rather than in registers. debugBelch takes varargs: hmm. Now, the calling code isn't quite in C as such, but in a related dialect called "Cmm", a variant of C-- (yes, minus), that GHC uses to help bridge the gap between the functional world and its code generation, and which is compiled down to C by GHC. When importing C functions into Cmm, GHC generates prototypes for them, but it doesn't do enough parsing to work out the true prototype; instead, they all just get something like extern StgFunPtr f(void);. In most architectures you can get away with this, because the arguments get passed in the usual calling convention anyway and it all works out, but on ppc64el this means that the caller doesn't generate enough stack space and then the callee tries to save its varargs onto the stack in an area that in fact belongs to the caller, and suddenly everything goes south. Things were starting to make sense. Now, debugBelch is only used in optional debugging code; but runInteractiveProcess (the function associated with the initial round of failures) takes no fewer than twelve arguments, plenty to force some of them onto the stack. I poked around the GCC patch for this ABI change a bit and determined that it only optimised away the stack allocation if it had a full prototype for all the callees, so I guessed that changing those prototypes to extern StgFunPtr f(); might work: it's still technically wrong, not least because omitting the parameter list is an obsolescent feature in C11, but it's at least just omitting information about the parameter list rather than actively lying about it. I tweaked that and ran the cross-build from scratch again. Lo and behold, suddenly I had a working compiler, and I could go through the same build-7.6-using-7.8 procedure as with arm64, much more quickly this time now that I knew what I was doing. One upstream bug, one Debian upload, and several bootstrapping builds later, and GHC was up and running on another architecture in Launchpad. Success! Epilogue There's still more to do. I gather there may be a Google Summer of Code project in Linaro to write proper native code generation for GHC on arm64: this would make things a good deal faster, but also enable GHCi (the interpreter) and Template Haskell, and thus clear quite a few more build failures. Since there's already native code generation for ppc64 in GHC, getting it going for ppc64el would probably only be a couple of days' work at this point. But these are niceties by comparison, and I'm more than happy with what I got working for 14.04. The upshot of all of this is that I may be the first non-Haskell-programmer to ever port GHC to two entirely new architectures. I'm not sure if I gain much from that personally aside from a lot of lost sleep and being considered extremely strange. It has, however, been by far the most challenging set of packages I've ported, and a fascinating trip through some odd corners of build systems and undefined behaviour that I don't normally need to touch.

14 April 2014

Steve Kemp: Is lumail a stepping stone?

I'm pondering a rewrite of my console-based mail-client. While it is "popular" it is not popular. I suspect "console-based" is the killer. I like console, and I ssh to a remote server to use it, but having different front-ends would be neat. In the world of mailpipe, etc, is there room for a graphic console client? Possibly. The limiting factor would be the lack of POP3/IMAP. Reworking things such that there is a daemon to which a GUI, or a console client, could connect seems simple. The hard part would obviously be working the IPC and writing the GUI. Any toolkit selected would rule out 40% of the audience. In other news I'm stalling on replying to emails. Irony.

Christine Spang: PyCon 2014 retrospective

PyCon 2014 happened. (Sprints are still happening.) This was my 3rd PyCon, but my first year as a serious contributor to the event, which led to an incredibly different feel. I also came as a person running a company building a complex system in Python, and I loved having the overarching mission of what I'm building driving my approach to what I chose to do. PyCon is one of the few conferences I go to where the feeling of acceptance and at-homeness mitigates the introvert overwhelm at nonstop social interaction. It's truly a special event and community. Here are some highlights: I didn't get to go to a lot of talks in person this year since my personal schedule was so full, but the PyCon video team is amazing as usual, so I'm looking forward to checking out the archive. It really is a gift to get the videos up while energy from the conference is still so high and people want to check out things they missed and share the talks they loved. Thanks to everyone, hugs, peace out, et cetera!

Chris Lamb: Race report: Cambridge Duathlon 2014

(This is my first race of the 2014 season.)


I had entered this race in 2013 and found it was effective for focusing winter training. As triathlons do not typically start until May in the UK, scheduling earlier races can be motivating in the colder winter months. I didn't have any clear goals for the race except to blow out the cobwebs and improve on my 2013 time. I couldn't set reasonable or reliable target times after considerable "long & slow" training in the off-season but I did want to test some new equipment and stategies, especially race pacing with a power meter, but also a new wheelset, crankset and helmet. Preparation was both accidentally and deliberately compromised: I did very little race-specific training as my season is based around an entirely different intensity of race, but compounding this I was confined to bed the weekend before. Sleep was acceptable in the preceding days and I felt moderately fresh on race morning. Nutrition-wise, I had porridge and bread with jam for breakfast, a PowerGel before the race, 750ml of PowerBar Perform on the bike along with a "Hydro" PowerGel with caffeine at approximately 30km.
Run 1 (7.5km) A few minutes before the start my race number belt the only truly untested equipment that day refused to tighten. However, I decided that once the race began I would either ignore it or even discard it, risking disqualification. Despite letting everyone go up the road, my first km was still too fast so I dialed down the effort, settling into a "10k" pace and began overtaking other runners. The Fen winds and drag-strip uphill from 3km provided a bit of pacing challenge for someone used to shelter and shorter hills but I kept a metered effort through into transition.
Time
33:01 (4:24/km, T1: 00:47) Last year: 37:47 (5:02/km)
Bike (40km) Although my 2014 bike setup features a power meter, I had not yet had the chance to perform an FTP test outdoors. I was thus was not able to calculate a definitive target power for the bike leg. However, data from my road bike suggested I set a power ceiling of 250W on the longer hills. This was extremely effective in avoiding going "into the red" and compromising the second run. This lends yet more weight to the idea that a power meter in multisport events is "almost like cheating". I was not entirely comfortable with my bike position: not only were my thin sunglasses making me raise my head more than I needed to, I found myself creeping forward onto the nose of my saddle. This is sub-optimal, even if only considering that I am not training in that position. Overall, the bike was uneventful with the only memorable moment provided by a wasp that got stuck between my head and a helmet vent. Coming into transition I didn't feel like I had really pushed myself that hard probably a good sign but the time difference from last year's bike leg (1:16:11) was a little underwhelming.
Time
1:10:45 (T2: 00:58)
Run 2 (7.5km) After leaving transition, my legs were extremely uncooperative and I had great difficulty in pacing myself in the first kilometer. Concentrating hard on reducing my cadence as well as using my rehearsed mental cue, I managed to settle down. The following 4 kilometers were a mental struggle rather than a physical one, modulo having to force a few burps to ease some discomfort, possibly from drinking too much or too fast on the bike. I had planned to "unload" as soon as I reached 6km but I didn't really have it in me. Whilst I am physiologically faster compared to last year, I suspect the lack of threshold-level running over the winter meant the mental component required for digging deep will require some coaxing to return. However, it is said that you have successfully paced a duathlon if the second run faster than the first. On this criterion, this was a success, but it would have been a bonus to have really felt completely completely drained at the end of the day, if only from a neo-Calvinist perspective.
Time
32:46 (4:22/km) / Last year: 38:10 (5:05/km)

Overall
Total time
2:18:19
A race that goes almost entirely to plan is a bit of a paradox there's certainly satisfaction in setting goals and hitting them without issue, but this is a gratification of slow-burning fire rather than the jubilation of a fireworks display. However, it was nice to learn that I managed to finish 5th in my age group despite this race attracting an extremely strong field: as an indicator, the age-group athlete finishing immediately before me was seven minutes faster and the overall winner finished in 1:54:53 (!). The race identified the following areas to work on:
  • Perform an outdoors FTP on my time-trial bike outdoors to develop an optimum power plan.
  • Do a few more brick runs, at least to re-acclimatise the feeling.
  • Schedule another bike fit.
Although not strictly race-related, I also need to find techniques to ensure transporting a bike on public transport is less stressful. (Full results & full 2014 race schedule)

13 April 2014

C.J. Adams-Collier: When was the last time you upgraded from squeeze to wheezy?

Wow. 3G delta. I haven t booted this laptop for a while I think I m finally ready to make the move from gnome2 to gnome3. There are bits that still annoy me, but I think it s off to a good start. Upgrading perl from 5.10 to 5.14.
cjac@calcifer:~$ sudo apt-get dist-upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages will be REMOVED:
  at-spi capplets-data compiz compiz-gnome compiz-gtk defoma deskbar-applet g++-4.3 gcc-4.3 gcj-4.4-base gcj-4.4-jre gcj-4.4-jre-headless gcj-4.4-jre-lib
  gdm3 gir1.0-clutter-1.0 gir1.0-freedesktop gir1.0-glib-2.0 gir1.0-gstreamer-0.10 gir1.0-gtk-2.0 gir1.0-json-glib-1.0 glade-gnome gnome-about
  gnome-accessibility gnome-applets gnome-core gnome-panel gnome-utils-common lib32readline5-dev libbrasero-media0 libclass-mop-perl libdb4.7-java
  libdb4.8-dev libdevhelp-1-1 libdigest-sha1-perl libdirectfb-dev libebook1.2-9 libecal1.2-7 libedata-book1.2-2 libedata-cal1.2-7 libedataserverui1.2-8
  libepc-1.0-2 libepc-ui-1.0-2 libept1 libgcj10 libgcj10-awt libgd2-noxpm libgstfarsight0.10-0 libgtkhtml-editor0 libjpeg62-dev libmetacity-private0
  libmono-accessibility1.0-cil libmono-bytefx0.7.6.1-cil libmono-cairo1.0-cil libmono-cil-dev libmono-corlib1.0-cil libmono-cscompmgd7.0-cil
  libmono-data-tds1.0-cil libmono-data1.0-cil libmono-debugger-soft0.0-cil libmono-getoptions1.0-cil libmono-i18n-west1.0-cil libmono-i18n1.0-cil
  libmono-ldap1.0-cil libmono-microsoft7.0-cil libmono-npgsql1.0-cil libmono-oracle1.0-cil libmono-peapi1.0-cil libmono-posix1.0-cil
  libmono-relaxng1.0-cil libmono-security1.0-cil libmono-sharpzip0.6-cil libmono-sharpzip0.84-cil libmono-sqlite1.0-cil libmono-system-data1.0-cil
  libmono-system-ldap1.0-cil libmono-system-messaging1.0-cil libmono-system-runtime1.0-cil libmono-system-web1.0-cil libmono-system1.0-cil
  libmono-webbrowser0.5-cil libmono-winforms1.0-cil libmono1.0-cil libmtp8 libnautilus-extension1 libpango1.0-common libperl5.10 libpolkit-gtk-1-0
  libpulse-browse0 librpm1 librpmbuild1 libsdl1.2-dev libsdl1.2debian-pulseaudio libseed0 libstdc++6-4.3-dev libtelepathy-farsight0 libupnp3 libvlccore4
  libxmlrpc-c3 linphone-nox linux-headers-2.6.32-5-amd64 linux-sound-base metacity mono-2.0-devel mono-devel mysql-client-5.1 mysql-query-browser
  mysql-server-5.1 mysql-server-core-5.1 openoffice.org-base-core openoffice.org-core openoffice.org-gcj openoffice.org-report-builder-bin
  openoffice.org-style-andromeda php5-suhosin portmap python-beagle python-brasero python-docky python-encutils python-evince python-gnomeapplet
  python-gtop python-mediaprofiles python-metacity python-totem-plparser seahorse-plugins smbfs speedbar totem-coherence tqsllib1c2a unixcw vlc
  xserver-xorg-video-nv
The following NEW packages will be installed:
  accountsservice acl aisleriot apg aptdaemon-data aptitude-common asterisk-core-sounds-en asterisk-modules asterisk-moh-opsound-gsm at-spi2-core
  ax25-node bluez btrfs-tools caribou caribou-antler chromium chromium-inspector colord console-setup console-setup-linux cpp-4.6 cpp-4.7 crda
  cryptsetup-bin cups-filters db-util db5.1-util dconf-gsettings-backend dconf-service dconf-tools distro-info-data docutils-common docutils-doc enchant
  extlinux finger folks-common fonts-cantarell fonts-droid fonts-freefont-ttf fonts-horai-umefont fonts-lg-aboriginal fonts-liberation fonts-lyx
  fonts-opensymbol fonts-sil-gentium fonts-sil-gentium-basic fonts-sipa-arundina fonts-stix fonts-takao fonts-takao-gothic fonts-takao-mincho
  fonts-thai-tlwg fonts-tlwg-garuda fonts-tlwg-kinnari fonts-tlwg-loma fonts-tlwg-mono fonts-tlwg-norasi fonts-tlwg-purisa fonts-tlwg-sawasdee
  fonts-tlwg-typewriter fonts-tlwg-typist fonts-tlwg-typo fonts-tlwg-umpush fonts-tlwg-waree fonts-umeplus fuse g++-4.7 g++-4.7-multilib gcc-4.6
  gcc-4.6-base gcc-4.7 gcc-4.7-base gcc-4.7-multilib gcj-4.7-base gcj-4.7-jre gcj-4.7-jre-headless gcj-4.7-jre-lib gconf-service gcr
  gir1.2-accountsservice-1.0 gir1.2-atk-1.0 gir1.2-atspi-2.0 gir1.2-caribou-1.0 gir1.2-clutter-1.0 gir1.2-clutter-gst-1.0 gir1.2-cogl-1.0
  gir1.2-coglpango-1.0 gir1.2-evince-3.0 gir1.2-folks-0.6 gir1.2-freedesktop gir1.2-gck-1 gir1.2-gconf-2.0 gir1.2-gcr-3 gir1.2-gdesktopenums-3.0
  gir1.2-gdkpixbuf-2.0 gir1.2-gee-1.0 gir1.2-gkbd-3.0 gir1.2-glib-2.0 gir1.2-gmenu-3.0 gir1.2-gnomebluetooth-1.0 gir1.2-gnomekeyring-1.0
  gir1.2-gst-plugins-base-0.10 gir1.2-gstreamer-0.10 gir1.2-gtk-3.0 gir1.2-gtkclutter-1.0 gir1.2-gtksource-3.0 gir1.2-gtop-2.0 gir1.2-gucharmap-2.90
  gir1.2-javascriptcoregtk-3.0 gir1.2-json-1.0 gir1.2-mutter-3.0 gir1.2-networkmanager-1.0 gir1.2-notify-0.7 gir1.2-panelapplet-4.0 gir1.2-pango-1.0
  gir1.2-peas-1.0 gir1.2-polkit-1.0 gir1.2-rb-3.0 gir1.2-soup-2.4 gir1.2-telepathyglib-0.12 gir1.2-telepathylogger-0.2 gir1.2-totem-1.0
  gir1.2-totem-plparser-1.0 gir1.2-upowerglib-1.0 gir1.2-vte-2.90 gir1.2-webkit-3.0 gir1.2-wnck-3.0 gir1.2-xkl-1.0 git-man gjs gkbd-capplet glchess
  glib-networking glib-networking-common glib-networking-services glines gnect gnibbles gnobots2 gnome-bluetooth gnome-contacts gnome-control-center-data
  gnome-desktop3-data gnome-font-viewer gnome-icon-theme-extras gnome-icon-theme-symbolic gnome-online-accounts gnome-packagekit gnome-packagekit-data
  gnome-shell gnome-shell-common gnome-sudoku gnome-sushi gnome-themes-standard gnome-themes-standard-data gnome-user-share gnome-video-effects gnomine
  gnotravex gnotski gnuplot gnuplot-nox grilo-plugins-0.1 groff growisofs gsettings-desktop-schemas gstreamer0.10-gconf gtali guile-2.0-libs gvfs-common
  gvfs-daemons gvfs-libs hardening-includes hwdata iagno ienglish-common imagemagick-common ioquake3 ioquake3-server iputils-tracepath ipxe-qemu iw
  keyutils kmod krb5-locales lib32itm1 lib32quadmath0 lib32tinfo-dev lib32tinfo5 libaacplus2 libaacs0 libabiword-2.9 libaccountsservice0 libamd2.2.0
  libapache-pom-java libapol4 libapt-inst1.5 libapt-pkg4.12 libaqbanking-plugins-libgwenhywfar60 libaqbanking34 libaqbanking34-plugins libaqhbci20
  libaqofxconnect7 libarchive12 libasprintf0c2 libassuan0 libatk-adaptor libatk-adaptor-data libatk-bridge2.0-0 libatkmm-1.6-1 libatkmm-1.6-dev
  libatspi2.0-0 libaudiofile1 libavahi-ui-gtk3-0 libavcodec53 libavcodec54 libavformat53 libavformat54 libavutil51 libbabl-0.1-0 libbind9-80 libbison-dev
  libblas3 libbluray1 libboost-iostreams1.49.0 libboost-program-options1.49.0 libboost-python1.49.0 libboost-serialization1.49.0 libboost-thread1.49.0
  libbrasero-media3-1 libcairo-gobject2 libcairo-script-interpreter2 libcamel-1.2-33 libcanberra-dev libcanberra-gtk3-0 libcanberra-gtk3-module
  libcanberra-pulse libcapi20-3 libcaribou-common libcaribou-gtk-module libcaribou-gtk3-module libcaribou0 libccrtp0 libcdio-cdda1 libcdio-paranoia1
  libcdio13 libcfg4 libchamplain-0.12-0 libchamplain-gtk-0.12-0 libcheese-gtk21 libcheese3 libclass-factory-util-perl libclass-isa-perl libclass-load-perl
  libclass-load-xs-perl libclutter-1.0-common libclutter-gst-1.0-0 libclutter-gtk-1.0-0 libclutter-imcontext-0.1-0 libclutter-imcontext-0.1-bin
  libcluttergesture-0.0.2-0 libcmis-0.2-0 libcogl-common libcogl-pango0 libcogl9 libcolord1 libcommons-parent-java libconfdb4 libcoroipcc4 libcoroipcs4
  libcpg4 libcryptsetup4 libcrystalhd3 libcupsfilters1 libcw3 libdata-alias-perl libdatetime-format-builder-perl libdatetime-format-iso8601-perl
  libdb-java libdb5.1 libdb5.1-dev libdb5.1-java libdb5.1-java-jni libdbus-c++-1-0 libdbus-glib1.0-cil libdbus1.0-cil libdconf0 libdee-1.0-4
  libdevel-partialdump-perl libdevhelp-3-0 libdevmapper-event1.02.1 libdistro-info-perl libdmapsharing-3.0-2 libdns88 libdotconf1.0 libdvbpsi7
  libebackend-1.2-2 libebml3 libebook-1.2-13 libecal-1.2-11 libecore1 libedata-book-1.2-13 libedata-cal-1.2-15 libedataserver-1.2-16
  libedataserverui-3.0-1 libeina1 libemail-valid-perl libencode-locale-perl libepc-1.0-3 libepc-ui-1.0-3 libept1.4.12 libescpr1 libev4
  libeval-closure-perl libevdocument3-4 libevent-2.0-5 libevent-perl libevs4 libevview3-3 libexiv2-12 libexosip2-7 libexporter-lite-perl
  libexttextcat-data libexttextcat0 libfakechroot libfarstream-0.1-0 libfdk-aac0 libfdt1 libfile-basedir-perl libfile-desktopentry-perl
  libfile-fcntllock-perl libfile-listing-perl libfile-mimeinfo-perl libfltk-images1.3 libfltk1.3 libfolks-eds25 libfolks-telepathy25 libfolks25
  libfont-afm-perl libgail-3-0 libgcj13 libgcj13-awt libgck-1-0 libgconf-2-4 libgconf2-doc libgcr-3-1 libgcr-3-common libgd2-xpm libgdata13
  libgdata2.1-cil libgdict-common libgdk-pixbuf2.0-0 libgdk-pixbuf2.0-common libgdk-pixbuf2.0-dev libgegl-0.2-0 libgeocode-glib0 libgettextpo0 libgexiv2-1
  libgirepository-1.0-1 libgjs0b libgkeyfile1.0-cil libgladeui-2-0 libgladeui-common libglapi-mesa libglew1.7 libglib2.0-bin libgmime-2.6-0
  libgmime2.6-cil libgmp10 libgnome-bluetooth10 libgnome-desktop-3-2 libgnome-keyring-common libgnome-media-profiles-3.0-0 libgnome-menu-3-0 libgnomekbd7
  libgnutls-openssl27 libgnutlsxx27 libgoa-1.0-0 libgoa-1.0-common libgphoto2-l10n libgraphite2-2.0.0 libgrilo-0.1-0 libgs9 libgs9-common libgssdp-1.0-3
  libgstreamer-plugins-bad0.10-0 libgtk-3-0 libgtk-3-bin libgtk-3-common libgtk-3-dev libgtk-3-doc libgtk-sharp-beans-cil libgtk-vnc-2.0-0
  libgtkhtml-4.0-0 libgtkhtml-4.0-common libgtkhtml-editor-4.0-0 libgtkmm-3.0-1 libgtksourceview-3.0-0 libgtksourceview-3.0-common libgucharmap-2-90-7
  libgudev1.0-cil libgupnp-1.0-4 libgupnp-av-1.0-2 libgupnp-igd-1.0-4 libgusb2 libgvnc-1.0-0 libgweather-3-0 libgwenhywfar-data libgwenhywfar60 libgxps2
  libhcrypto4-heimdal libheimbase1-heimdal libhtml-form-perl libhtml-format-perl libhttp-cookies-perl libhttp-daemon-perl libhttp-date-perl
  libhttp-message-perl libhttp-negotiate-perl libhunspell-1.3-0 libicu48 libimobiledevice2 libio-aio-perl libisc84 libisccc80 libisccfg82 libiscsi1
  libiso9660-8 libisoburn1 libitm1 libjavascriptcoregtk-1.0-0 libjavascriptcoregtk-3.0-0 libjbig0 libjs-sphinxdoc libjs-underscore libjson0 libjte1
  libkadm5clnt-mit8 libkadm5srv-mit8 libkarma0 libkdb5-6 libkmod2 libkpathsea6 liblapack3 liblavfile-2.0-0 liblavjpeg-2.0-0 liblavplay-2.0-0 liblcms2-2
  liblensfun-data liblensfun0 liblinear-tools liblinear1 liblinphone4 liblockfile-bin liblogsys4 liblvm2app2.2 liblwp-mediatypes-perl
  liblwp-protocol-https-perl liblwres80 liblzma5 libmaa3 libmagick++5 libmagickcore5 libmagickcore5-extra libmagickwand5 libmath-bigint-perl
  libmath-round-perl libmatroska5 libmediastreamer1 libmhash2 libminiupnpc5 libmission-control-plugins0 libmjpegutils-2.0-0 libmodule-implementation-perl
  libmodule-runtime-perl libmono-2.0-1 libmono-2.0-dev libmono-accessibility4.0-cil libmono-cairo4.0-cil libmono-codecontracts4.0-cil
  libmono-compilerservices-symbolwriter4.0-cil libmono-corlib4.0-cil libmono-csharp4.0-cil libmono-custommarshalers4.0-cil libmono-data-tds4.0-cil
  libmono-debugger-soft2.0-cil libmono-debugger-soft4.0-cil libmono-http4.0-cil libmono-i18n-cjk4.0-cil libmono-i18n-mideast4.0-cil
  libmono-i18n-other4.0-cil libmono-i18n-rare4.0-cil libmono-i18n-west4.0-cil libmono-i18n4.0-all libmono-i18n4.0-cil libmono-ldap4.0-cil
  libmono-management4.0-cil libmono-messaging-rabbitmq4.0-cil libmono-messaging4.0-cil libmono-microsoft-build-engine4.0-cil
  libmono-microsoft-build-framework4.0-cil libmono-microsoft-build-tasks-v4.0-4.0-cil libmono-microsoft-build-utilities-v4.0-4.0-cil
  libmono-microsoft-csharp4.0-cil libmono-microsoft-visualc10.0-cil libmono-microsoft-web-infrastructure1.0-cil libmono-npgsql4.0-cil
  libmono-opensystem-c4.0-cil libmono-oracle4.0-cil libmono-peapi4.0-cil libmono-posix4.0-cil libmono-rabbitmq4.0-cil libmono-relaxng4.0-cil
  libmono-security4.0-cil libmono-sharpzip4.84-cil libmono-simd4.0-cil libmono-sqlite4.0-cil libmono-system-componentmodel-composition4.0-cil
  libmono-system-componentmodel-dataannotations4.0-cil libmono-system-configuration-install4.0-cil libmono-system-configuration4.0-cil
  libmono-system-core4.0-cil libmono-system-data-datasetextensions4.0-cil libmono-system-data-linq4.0-cil libmono-system-data-services-client4.0-cil
  libmono-system-data-services4.0-cil libmono-system-data4.0-cil libmono-system-design4.0-cil libmono-system-drawing-design4.0-cil
  libmono-system-drawing4.0-cil libmono-system-dynamic4.0-cil libmono-system-enterpriseservices4.0-cil libmono-system-identitymodel-selectors4.0-cil
  libmono-system-identitymodel4.0-cil libmono-system-ldap4.0-cil libmono-system-management4.0-cil libmono-system-messaging4.0-cil
  libmono-system-net4.0-cil libmono-system-numerics4.0-cil libmono-system-runtime-caching4.0-cil libmono-system-runtime-durableinstancing4.0-cil
  libmono-system-runtime-serialization-formatters-soap4.0-cil libmono-system-runtime-serialization4.0-cil libmono-system-runtime4.0-cil
  libmono-system-security4.0-cil libmono-system-servicemodel-discovery4.0-cil libmono-system-servicemodel-routing4.0-cil
  libmono-system-servicemodel-web4.0-cil libmono-system-servicemodel4.0-cil libmono-system-serviceprocess4.0-cil libmono-system-transactions4.0-cil
  libmono-system-web-abstractions4.0-cil libmono-system-web-applicationservices4.0-cil libmono-system-web-dynamicdata4.0-cil
  libmono-system-web-extensions-design4.0-cil libmono-system-web-extensions4.0-cil libmono-system-web-routing4.0-cil libmono-system-web-services4.0-cil
  libmono-system-web4.0-cil libmono-system-windows-forms-datavisualization4.0-cil libmono-system-windows-forms4.0-cil libmono-system-xaml4.0-cil
  libmono-system-xml-linq4.0-cil libmono-system-xml4.0-cil libmono-system4.0-cil libmono-tasklets4.0-cil libmono-web4.0-cil libmono-webbrowser2.0-cil
  libmono-webbrowser4.0-cil libmono-webmatrix-data4.0-cil libmono-windowsbase4.0-cil libmount1 libmozjs10d libmozjs17d libmozjs185-1.0 libmpeg2encpp-2.0-0
  libmplex2-2.0-0 libmtdev1 libmtp-common libmtp-runtime libmtp9 libmupen64plus2 libmusicbrainz-discid-perl libmusicbrainz5-0 libmutter0 libmx-1.0-2
  libmx-bin libmx-common libmysqlclient18 libnatpmp1 libnautilus-extension1a libnet-domain-tld-perl libnet-http-perl libnet-ip-minimal-perl libnetcf1
  libnetfilter-conntrack3 libnettle4 libnewtonsoft-json4.5-cil libnice10 libnl-3-200 libnl-genl-3-200 libnl-route-3-200 libnm-glib4 libnm-gtk-common
  libnm-gtk0 libnm-util2 libnotify4 libnspr4 libnss-winbind libnss3 libnuma1 libnunit2.6-cil liboauth0 libodbc1 liboobs-1-5 libopal3.10.4 libopenal-data
  libopus0 libosip2-7 libp11-2 libp11-kit-dev libp11-kit0 libpackage-stash-xs-perl libpackagekit-glib2-14 libpam-cap libpam-modules-bin libpam-winbind
  libpanel-applet-4-0 libparams-classify-perl libpcre3-dev libpcrecpp0 libpeas-1.0-0 libpeas-common libperl5.14 libpipeline1 libpload4 libpodofo0.9.0
  libpoe-component-resolver-perl libpoppler-glib8 libpoppler19 libportsmf0 libpostproc52 libprocps0 libpst4 libpt2.10.4 libptexenc1 libpython2.7
  libqt4-declarative libqtassistantclient4 libqtdbus4 libqtwebkit4 libquadmath0 libquicktime2 libquorum4 libquvi-scripts libquvi7 libraptor2-0 librasqal3
  libraw5 libregexp-reggrp-perl libreoffice libreoffice-base libreoffice-base-core libreoffice-calc libreoffice-common libreoffice-core libreoffice-draw
  libreoffice-emailmerge libreoffice-evolution libreoffice-filter-binfilter libreoffice-filter-mobiledev libreoffice-gnome libreoffice-gtk
  libreoffice-help-en-us libreoffice-impress libreoffice-java-common libreoffice-math libreoffice-officebean libreoffice-report-builder-bin
  libreoffice-style-galaxy libreoffice-style-tango libreoffice-writer libresid-builder0c2a librest-0.7-0 librest-extras-0.7-0 librhythmbox-core6 librpm3
  librpmbuild3 librpmio3 librpmsign1 libruby1.9.1 libsaamf3 libsackpt3 libsaclm3 libsaevt3 libsalck3 libsam4 libsamsg4 libsane-common
  libsane-extras-common libsatmr3 libsbsms10 libseed-gtk3-0 libsidplay2 libsigsegv2 libsocialweb-client2 libsocialweb-common libsocialweb-service
  libsocialweb0 libsocket-getaddrinfo-perl libsocket-perl libsonic0 libsoundtouch0 libsox2 libspeechd2 libspice-client-glib-2.0-1
  libspice-client-gtk-2.0-1 libspice-server1 libssl-doc libssl1.0.0 libstdc++6-4.7-dev libsvm-tools libswitch-perl libswscale2 libsystemd-daemon0
  libsystemd-login0 libtagc0 libtelepathy-farstream2 libtelepathy-logger2 libtest-warn-perl libtinfo-dev libtinfo5 libtirpc1 libtokyocabinet9 libtotem-pg4
  libtotem0 libtqsllib1 libtracker-sparql-0.14-0 libtree-dagnode-perl libts-dev libucommon5 libumfpack5.4.0 libunique-3.0-0 libupnp6 libusbredirhost1
  libusbredirparser0 libv4lconvert0 libverto-libev1 libverto1 libvisio-0.0-0 libvlccore5 libvo-aacenc0 libvo-amrwbenc0 libvorbisidec1 libvotequorum4
  libvpx1 libvte-2.90-9 libvte-2.90-common libwacom-common libwacom2 libwebkitgtk-1.0-0 libwebkitgtk-1.0-common libwebkitgtk-3.0-0 libwebkitgtk-3.0-common
  libwebp2 libwebrtc-audio-processing-0 libwildmidi-config libwireshark-data libwireshark2 libwiretap2 libwnck-3-0 libwnck-3-common libwpd-0.9-9
  libwpg-0.2-2 libwps-0.2-2 libwsutil2 libwv-1.2-4 libwww-robotrules-perl libx11-doc libx11-protocol-perl libx264-123 libx264-124 libx264-130 libx264-132
  libxalan2-java libxcb-composite0 libxcb-glx0 libxcb-shape0 libxcb-shm0-dev libxcb-util0 libxen-4.1 libxml-commons-external-java
  libxml-commons-resolver1.1-java libxml-sax-base-perl libxmlrpc-c++4 libxmlrpc-core-c3 libxz-java libyajl2 libyaml-0-2 libyaml-perl libyelp0 libzrtpcpp2
  libzvbi-common libzvbi0 lightsoff linphone-nogtk linux-headers-3.2.0-4-amd64 linux-headers-3.2.0-4-common linux-headers-amd64 linux-image-3.2.0-4-amd64
  linux-image-amd64 linux-kbuild-3.2 live-boot-doc live-config-doc live-manual-html mahjongg memtest86+ minissdpd mono-4.0-gac mono-dmcs mscompress
  multiarch-support mupen64plus-audio-all mupen64plus-audio-sdl mupen64plus-data mupen64plus-input-all mupen64plus-input-sdl mupen64plus-rsp-all
  mupen64plus-rsp-hle mupen64plus-rsp-z64 mupen64plus-ui-console mupen64plus-video-all mupen64plus-video-arachnoid mupen64plus-video-glide64
  mupen64plus-video-rice mupen64plus-video-z64 mutter-common mysql-client-5.5 mysql-server-5.5 mysql-server-core-5.5 mythes-en-us openarena-081-maps
  openarena-081-misc openarena-081-players openarena-081-players-mature openarena-081-textures openarena-085-data openarena-088-data packagekit
  packagekit-backend-aptcc packagekit-tools planner-data planner-doc poppler-data printer-driver-all printer-driver-c2050 printer-driver-c2esp
  printer-driver-cjet printer-driver-escpr printer-driver-foo2zjs printer-driver-gutenprint printer-driver-hpcups printer-driver-hpijs
  printer-driver-m2300w printer-driver-min12xxw printer-driver-pnm2ppa printer-driver-postscript-hp printer-driver-ptouch printer-driver-pxljr
  printer-driver-sag-gdi printer-driver-splix psutils python-aptdaemon.gtk3widgets python-aptdaemon.gtkwidgets python-bzrlib python-dbus-dev
  python-debianbts python-defer python-dnspython python-fpconst python-gi python-gi-cairo python-gi-dev python-gobject-2 python-gobject-2-dev
  python-keyring python-launchpadlib python-lazr.restfulclient python-lazr.uri python-liblarch python-liblarch-gtk python-magic python-oauth
  python-packagekit python-pyatspi2 python-pyparsing python-repoze.lru python-routes python-setools python-simplejson python-soappy python-speechd
  python-spice-client-gtk python-wadllib python-webob python-zeitgeist python2.7 python2.7-dev python2.7-minimal qdbus quadrapassel remmina-common
  rhythmbox-data rpcbind rtkit ruby ruby1.9.1 shotwell-common smartdimmer software-properties-common sound-theme-freedesktop speech-dispatcher
  sphinx-common sphinx-doc swell-foop syslinux-themes-debian syslinux-themes-debian-wheezy tdb-tools telepathy-haze telepathy-logger telepathy-rakia
  tex-gyre ttf-marvosym wireless-regdb xbrlapi xorg-sgml-doctools xorriso xserver-xorg-input-mouse xserver-xorg-input-vmmouse xulrunner-17.0 yelp-xsl
  zeitgeist-core zenity-common
The following packages have been kept back:
  acroread-debian-files db4.8-util hibernate ia32-libs ia32-libs-gtk libboost-dev libboost-serialization-dev opensc wine
The following packages will be upgraded:
  abcde abiword abiword-common abiword-plugin-grammar abiword-plugin-mathview acpi acpi-fakekey acpi-support acpi-support-base acpid acroread-data
  acroread-dictionary-en acroread-l10n-en adduser alacarte alsa-base alsa-utils amb-plugins anacron analog ant ant-optional apache2 apache2-doc
  apache2-mpm-prefork apache2-utils apache2.2-bin apache2.2-common app-install-data apt apt-file apt-utils apt-xapian-index aptdaemon aptitude
  aqbanking-tools aspell aspell-en asterisk asterisk-config asterisk-core-sounds-en-gsm asterisk-doc asterisk-voicemail astyle at audacity audacity-data
  augeas-lenses augeas-tools autoconf autoconf-doc automake automake1.9 autopoint autotools-dev avahi-autoipd avahi-daemon avidemux avidemux-common
  avidemux-plugins aview ax25-tools banshee baobab base-files base-passwd bash bash-completion bc bind9-doc bind9-host bind9utils binfmt-support binutils
  bison bluez-cups bogofilter bogofilter-bdb bogofilter-common brasero brasero-common bridge-utils browser-plugin-gnash bsd-mailx bsdmainutils bsdutils
  busybox buzztard buzztard-data bwidget bzip2 bzr bzrtools ca-certificates calibre calibre-bin ccache cd-discid cdebootstrap cdparanoia cdrdao
  checkpolicy cheese cheese-common chromium-browser chromium-browser-inspector cifs-utils cl-asdf cli-common clisp comerr-dev common-lisp-controller
  console-common console-data console-tools consolekit coreutils cowbuilder cowdancer cpio cpp cpp-4.4 cpufrequtils cracklib-runtime crawl-common
  crawl-tiles cron cryptsetup cups cups-bsd cups-client cups-common cups-driver-gutenprint cups-pk-helper cups-ppdc cupsddk curl curlftpfs cvs cw dash
  dasher dasher-data dbus dbus-x11 dc dcraw dctrl-tools debconf debconf-i18n debhelper debian-archive-keyring debian-faq debian-keyring debianutils debirf
  debootstrap desktop-base desktop-file-utils devhelp devhelp-common devscripts dialog dict dictionaries-common diffstat diffutils djtools dkms dmidecode
  dmsetup dnsmasq-base dnsutils doc-debian docbook docbook-dsssl docbook-to-man docbook-utils docbook-xml docbook-xsl docbook-xsl-doc-html docky dosemu
  dosfstools dpatch dpkg dpkg-dev dput dvd+rw-tools dvi2ps dynagen dynamips e2fslibs e2fsprogs ebtables ed eject ekiga emacs23-bin-common emacs23-common
  emacs23-nox emacsen-common emdebian-archive-keyring empathy empathy-common eog epiphany-browser epiphany-browser-data epiphany-extensions esound-common
  espeak espeak-data ethtool evince evince-common evolution evolution-common evolution-data-server evolution-data-server-common evolution-exchange
  evolution-plugins evolution-webcal exif exiftags exim4 exim4-base exim4-config exim4-daemon-light exiv2 f-spot fakechroot fakeroot fancontrol fceu
  fcrackzip fdupes feynmf file file-roller finch findutils firmware-iwlwifi firmware-linux-free firmware-linux-nonfree flac flashrom fldigi flex
  fontconfig fontconfig-config foo2zjs foomatic-db foomatic-db-engine foomatic-db-gutenprint foomatic-filters fping freedesktop-sound-theme freeglut3
  freetds-common ftp fuse-utils g++ g++-4.4 g++-4.4-multilib g++-multilib gawk gcalctool gcc gcc-4.4 gcc-4.4-base gcc-4.4-doc gcc-4.4-multilib
  gcc-doc-base gcc-multilib gcj-jre gcj-jre-headless gconf-defaults-service gconf-editor gconf2 gconf2-common gddrescue gdebi gdebi-core gedit
  gedit-common gedit-plugins genisoimage geoclue geoclue-hostip geoclue-localnet geoclue-manual geoclue-yahoo geoip-database gettext gettext-base
  ghostscript ghostscript-cups gimp gimp-data git git-buildpackage git-core git-svn gitk gksu glade gnash gnash-common gnash-opengl
  gnome-accessibility-themes gnome-applets-data gnome-backgrounds gnome-cards-data gnome-common gnome-control-center gnome-control-center-dev
  gnome-desktop-data gnome-dictionary gnome-disk-utility gnome-do gnome-do-plugins gnome-doc-utils gnome-games gnome-games-data gnome-games-extra-data
  gnome-icon-theme gnome-js-common gnome-keyring gnome-mag gnome-media gnome-menus gnome-nettool gnome-orca gnome-panel-data gnome-pkg-tools
  gnome-power-manager gnome-rdp gnome-screensaver gnome-screenshot gnome-search-tool gnome-session gnome-session-bin gnome-session-canberra
  gnome-session-common gnome-settings-daemon gnome-settings-daemon-dev gnome-system-log gnome-system-monitor gnome-system-tools gnome-terminal
  gnome-terminal-data gnome-user-guide gnomint gnu-fdisk gnucash-docs gnuchess gnumeric gnumeric-common gnupg gnupg-agent gocr google-talkplugin gparted
  gpgv gpredict gpscorrelate grep groff-base grub-common grub-legacy gsfonts-x11 gsmartcontrol gstreamer0.10-alsa gstreamer0.10-buzztard
  gstreamer0.10-buzztard-doc gstreamer0.10-doc gstreamer0.10-ffmpeg gstreamer0.10-ffmpeg-dbg gstreamer0.10-fluendo-mp3 gstreamer0.10-gnonlin
  gstreamer0.10-gnonlin-dbg gstreamer0.10-gnonlin-doc gstreamer0.10-nice gstreamer0.10-plugins-bad gstreamer0.10-plugins-bad-dbg
  gstreamer0.10-plugins-bad-doc gstreamer0.10-plugins-base gstreamer0.10-plugins-base-apps gstreamer0.10-plugins-base-dbg gstreamer0.10-plugins-base-doc
  gstreamer0.10-plugins-good gstreamer0.10-plugins-good-dbg gstreamer0.10-plugins-good-doc gstreamer0.10-plugins-ugly gstreamer0.10-plugins-ugly-dbg
  gstreamer0.10-plugins-ugly-doc gstreamer0.10-pulseaudio gstreamer0.10-tools gstreamer0.10-x gtg gthumb gthumb-data gtk2-engines gtk2-engines-pixbuf
  gucharmap guile-1.6 guile-1.6-libs guile-1.8-libs gvfs gvfs-backends gvfs-bin gzip hal hamster-applet hardinfo hddtemp hdparm hfsprogs hostname hp-ppd
  hpijs hplip hplip-cups hplip-data htmldoc htmldoc-common iamerican ibritish iceweasel ifupdown ijsgutenprint imagemagick imagemagick-doc info
  initramfs-tools initscripts inkscape insserv install-info installation-report intltool iotop iproute ipsec-tools iptables iptraf iputils-ping
  ircd-hybrid irssi isc-dhcp-client isc-dhcp-common isc-dhcp-server iscsitarget-dkms iso-codes ispell jack jadetex java-common jigdo-file keyanalyze
  keyboard-configuration keychain klibc-utils kpartx krb5-admin-server krb5-auth-dialog krb5-config krb5-doc krb5-kdc krb5-kdc-ldap krb5-multidev
  krb5-pkinit krb5-user lacheck lame latex-beamer latex-xcolor less lesstif2 lesstif2-dev lib32asound2 lib32bz2-1.0 lib32gcc1 lib32gomp1 lib32ncurses5
  lib32ncurses5-dev lib32nss-mdns lib32readline5 lib32stdc++6 lib32v4l-0 lib32z1 lib32z1-dev liba52-0.7.4 libaa1 libaa1-dev libacl1 libaften0
  libaiksaurus-1.2-0c2a libaiksaurus-1.2-data libaiksaurusgtk-1.2-0c2a libaio1 libalgorithm-diff-xs-perl libany-moose-perl libanyevent-perl libao-common
  libao4 libapache-dbi-perl libapache2-mod-apreq2 libapache2-mod-dnssd libapache2-mod-perl2 libapache2-mod-php5 libapache2-mod-python
  libapache2-request-perl libappconfig-perl libapr1 libapreq2 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap libapt-pkg-perl libaqbanking-data
  libarchive-zip-perl libart-2.0-2 libart-2.0-dev libart2.0-cil libasn1-8-heimdal libasound2 libasound2-dev libasound2-plugins libaspell15 libass4
  libasync-interrupt-perl libasyncns0 libatasmart4 libatk1.0-0 libatk1.0-data libatk1.0-dev libatk1.0-doc libatspi1.0-0 libattr1 libaudio-dev libaudio2
  libaudiofile-dev libaudit0 libaugeas0 libavahi-client-dev libavahi-client3 libavahi-common-data libavahi-common-dev libavahi-common3 libavahi-core7
  libavahi-glib-dev libavahi-glib1 libavahi-gobject0 libavahi-ui0 libavc1394-0 libax25 libb-hooks-endofscope-perl libb-keywords-perl libbind9-60
  libblas3gf libblkid1 libbluetooth3 libbml0 libboo2.0.9-cil libbrlapi0.5 libbs2b0 libbsd0 libburn4 libbusiness-paypal-api-perl
  libbusiness-tax-vat-validation-perl libbuzztard0 libbz2-1.0 libc-ares2 libc-bin libc-dev-bin libc6 libc6-dev libc6-dev-i386 libc6-i386 libcaca-dev
  libcaca0 libcache-fastmmap-perl libcairo-perl libcairo2 libcairo2-dev libcairomm-1.0-1 libcairomm-1.0-dev libcanberra-gtk0 libcanberra0 libcap-ng0
  libcap2 libcap2-bin libcapture-tiny-perl libccid libcdaudio1 libcddb-get-perl libcddb2 libcdparanoia0 libcdt4 libchm-bin libchm1 libck-connector0
  libclass-c3-perl libclass-c3-xs-perl libclass-insideout-perl libclass-inspector-perl libclass-method-modifiers-perl libclass-methodmaker-perl
  libclone-perl libclutter-1.0-0 libcolamd2.7.1 libcolor-calc-perl libcomedi0 libcomerr2 libcommon-sense-perl libcommons-beanutils-java
  libcommons-collections3-java libcommons-compress-java libcommons-digester-java libcommons-logging-java libconfig-inifiles-perl libconfig-json-perl
  libconfig-tiny-perl libconsole libcontextual-return-perl libconvert-asn1-perl libcoro-perl libcorosync4 libcpufreq-dev libcpufreq0 libcrack2 libcroco3
  libcrypt-openssl-bignum-perl libcrypt-openssl-random-perl libcrypt-openssl-rsa-perl libcrypt-passwdmd5-perl libcrypt-ssleay-perl libcss-minifier-xs-perl
  libcss-packer-perl libcups2 libcupscgi1 libcupsdriver1 libcupsimage2 libcupsmime1 libcupsppdc1 libcurl3 libcurl3-gnutls libcurses-perl libcwidget3
  libdata-optlist-perl libdata-structure-util-perl libdata-visitor-perl libdatetime-format-http-perl libdatetime-perl libdatetime-set-perl
  libdatetime-timezone-perl libdatrie1 libdb-dev libdb-je-java libdbd-mysql-perl libdbi-perl libdbus-1-3 libdbus-1-dev libdbus-glib-1-2 libdbus-glib-1-dev
  libdc1394-22 libdca0 libdebian-installer-extra4 libdebian-installer4 libdevel-globaldestruction-perl libdevel-size-perl libdevel-stacktrace-perl
  libdevel-symdump-perl libdevmapper1.02.1 libdigest-hmac-perl libdirac-decoder0 libdirac-encoder0 libdirectfb-1.2-9 libdirectfb-extra libdiscid0
  libdjvulibre-text libdjvulibre21 libdns69 libdpkg-perl libdrm-dev libdrm-intel1 libdrm-nouveau1a libdrm-radeon1 libdrm2 libdv4 libdvdcss2 libdvdnav4
  libdvdread4 libedit2 libelf1 libelfg0 libemail-address-perl libenca0 libenchant1c2a libengine-pkcs11-openssl libepc-common libesd0 libesd0-dev
  libespeak1 libevolution libexception-class-perl libexempi3 libexif12 libexpat1 libexpat1-dev libexpect-perl libfaac0 libfaad2 libfcgi-perl libfcgi0ldbl
  libffi-dev libffi5 libfftw3-3 libfile-homedir-perl libfile-libmagic-perl libfile-mmagic-perl libfile-slurp-perl libfile-which-perl libfilter-perl
  libfinance-quote-perl libflac++6 libflac8 libflickrnet2.2-cil libflite1 libfltk1.1 libfluidsynth1 libfontconfig1 libfontconfig1-dev libfontenc1
  libfreetype6 libfreetype6-dev libfribidi0 libfs6 libftdi-dev libftdi1 libfuse2 libgail-common libgail-dev libgail18 libgc1c2 libgcc1 libgcj-bc
  libgcj-common libgconf2-4 libgconf2-dev libgconf2.0-cil libgcrypt11 libgcrypt11-dev libgd-gd2-noxpm-perl libgdata-common libgdbm3 libgdict-1.0-6
  libgdiplus libgdome2-0 libgdome2-cpp-smart0c2a libgdu-gtk0 libgdu0 libgee2 libgeoclue0 libgeoip1 libgfortran3 libgif4 libgimp2.0 libgio-cil libgksu2-0
  libgl1-mesa-dev libgl1-mesa-dri libgl1-mesa-glx libglade2.0-cil libgladeui-1-9 libglib-perl libglib2.0-0 libglib2.0-cil libglib2.0-data libglib2.0-dev
  libglib2.0-doc libglibmm-2.4-1c2a libglibmm-2.4-dev libglu1-mesa libglu1-mesa-dev libgnome-desktop-2-17 libgnome-desktop-dev libgnome-keyring-dev
  libgnome-keyring0 libgnome-keyring1.0-cil libgnome-mag2 libgnome-menu2 libgnome-speech7 libgnome-vfs2.0-cil libgnome2-0 libgnome2-canvas-perl
  libgnome2-common libgnome2-dev libgnome2-perl libgnome2-vfs-perl libgnome2.24-cil libgnomecanvas2-0 libgnomecanvas2-common libgnomecanvas2-dev
  libgnomedesktop2.20-cil libgnomekbd-common libgnomeui-0 libgnomeui-common libgnomeui-dev libgnomevfs2-0 libgnomevfs2-common libgnomevfs2-dev
  libgnomevfs2-extra libgnupg-interface-perl libgnutls-dev libgnutls26 libgoffice-0.8-8 libgoffice-0.8-8-common libgomp1 libgpg-error-dev libgpg-error0
  libgpgme11 libgphoto2-2 libgphoto2-port0 libgpm2 libgpod-common libgpod4 libgraph4 libgsf-1-114 libgsf-1-common libgsl0ldbl libgsm0710-0 libgsm1
  libgssapi-krb5-2 libgssglue1 libgssrpc4 libgstbuzztard0 libgstreamer-plugins-base0.10-0 libgstreamer-plugins-base0.10-dev libgstreamer0.10-0
  libgstreamer0.10-0-dbg libgstreamer0.10-dev libgtk-vnc-1.0-0 libgtk2-perl libgtk2.0-0 libgtk2.0-bin libgtk2.0-cil libgtk2.0-common libgtk2.0-dev
  libgtk2.0-doc libgtkglext1 libgtkhtml3.14-19 libgtkimageview0 libgtkmathview0c2a libgtkmm-2.4-1c2a libgtkmm-2.4-dev libgtop2-7 libgtop2-common
  libgtop2-dev libguard-perl libgudev-1.0-0 libguile-ltdl-1 libgutenprint2 libgvc5 libgweather-common libhal-dev libhal-storage1 libhal1 libhamlib2
  libhpmud0 libhsqldb-java libhtml-packer-perl libhtml-parser-perl libhtml-tableextract-perl libhtml-tagcloud-perl libhtml-template-expr-perl
  libhtml-template-perl libhtml-tree-perl libhtml-treebuilder-xpath-perl libhttp-server-simple-perl libhx509-5-heimdal libhyphen0 libical0 libice-dev
  libice6 libicu44 libicu4j-java libidl-dev libidl0 libidn11 libidn11-dev libieee1284-3 libijs-0.35 libilmbase6 libimage-exif-perl libimage-exiftool-perl
  libio-pty-perl libio-socket-inet6-perl libio-socket-ssl-perl libio-stringy-perl libio-stty-perl libipc-run-perl libiptcdata0 libisc62 libisccc60
  libisccfg62 libisofs6 libiw30 libjack0 libjasper1 libjavascript-minifier-xs-perl libjavascript-packer-perl libjaxp1.3-java libjaxp1.3-java-gcj
  libjbig2dec0 libjline-java libjpeg-progs libjpeg62 libjpeg8 libjs-jquery libjs-yui libjson-any-perl libjson-glib-1.0-0 libjson-perl libjson-xs-perl
  libjtidy-java libk5crypto3 libkadm5clnt-mit7 libkadm5srv-mit7 libkate1 libkdb5-4 libkeyutils1 libklibc libkms1 libkrb5-26-heimdal libkrb5-3
  libkrb5support0 libktoblzcheck1c2a liblapack3gf liblcms1 libldap-2.4-2 liblink-grammar4 liblircclient0 liblist-moreutils-perl liblocale-gettext-perl
  liblocales-perl liblockfile1 liblog-dispatch-perl liblog4c3 liblog4cxx10 libloudmouth1-0 liblouis-data liblouis2 liblqr-1-0 libltdl-dev libltdl7
  liblua5.1-0 liblua5.1-0-dev liblucene2-java liblwres60 liblzo2-2 libmad0 libmagic1 libmagick++3 libmagickcore3 libmagickcore3-extra libmagickwand3
  libmailtools-perl libmeanwhile1 libmime-tools-perl libmime-types-perl libmimic0 libmms0 libmng1 libmodplug1 libmodule-find-perl libmodule-starter-perl
  libmono-accessibility2.0-cil libmono-addins-gui0.2-cil libmono-addins0.2-cil libmono-c5-1.1-cil libmono-cairo2.0-cil libmono-cecil-private-cil
  libmono-corlib2.0-cil libmono-cscompmgd8.0-cil libmono-data-tds2.0-cil libmono-db2-1.0-cil libmono-i18n-west2.0-cil libmono-i18n2.0-cil
  libmono-ldap2.0-cil libmono-management2.0-cil libmono-messaging-rabbitmq2.0-cil libmono-messaging2.0-cil libmono-microsoft-build2.0-cil
  libmono-microsoft8.0-cil libmono-npgsql2.0-cil libmono-oracle2.0-cil libmono-peapi2.0-cil libmono-posix2.0-cil libmono-rabbitmq2.0-cil
  libmono-relaxng2.0-cil libmono-security2.0-cil libmono-sharpzip2.6-cil libmono-sharpzip2.84-cil libmono-simd2.0-cil libmono-sqlite2.0-cil
  libmono-system-data-linq2.0-cil libmono-system-data2.0-cil libmono-system-ldap2.0-cil libmono-system-messaging2.0-cil libmono-system-runtime2.0-cil
  libmono-system-web-mvc1.0-cil libmono-system-web-mvc2.0-cil libmono-system-web2.0-cil libmono-system2.0-cil libmono-tasklets2.0-cil libmono-wcf3.0-cil
  libmono-windowsbase3.0-cil libmono-winforms2.0-cil libmono-zeroconf1.0-cil libmono2.0-cil libmoose-perl libmouse-perl libmp3lame0 libmpc2 libmpcdec6
  libmpfr4 libmpg123-0 libmusicbrainz3-6 libmysqlclient-dev libmysqlclient16 libmythes-1.2-0 libnamespace-autoclean-perl libnamespace-clean-perl
  libncurses5 libncurses5-dev libncursesw5 libncursesw5-dev libndesk-dbus-glib1.0-cil libndesk-dbus1.0-cil libneon27 libneon27-gnutls libnet-daemon-perl
  libnet-dbus-perl libnet-dns-perl libnet-ip-perl libnet-ldap-perl libnet-libidn-perl libnet-netmask-perl libnet-oauth-perl libnet-snmp-perl
  libnet-ssleay-perl libnet1 libnet1-dev libnet6-1.3-0 libnetaddr-ip-perl libnetpbm10 libnewt0.52 libnfnetlink0 libnfsidmap2 libnl1 libnm-glib-dev
  libnm-glib-vpn-dev libnm-glib-vpn1 libnm-util-dev libnotify-dev libnotify0.4-cil libnspr4-0d libnss-mdns libnss3-1d libnunit-cil-dev libofa0 libogg0
  liboobs-1-dev libopenais3 libopenal1 libopencore-amrnb0 libopencore-amrwb0 libopenct1 libopenexr6 libopenjpeg2 libopenraw1 libopenrawgnome1 libopts25
  liborbit2 liborbit2-dev liborc-0.4-0 libortp8 libosp5 libossp-uuid-perl libossp-uuid16 libostyle1c2 libotr2 libots0 libpackage-deprecationmanager-perl
  libpackage-stash-perl libpam-cracklib libpam-gnome-keyring libpam-ldap libpam-modules libpam-p11 libpam-runtime libpam0g libpam0g-dev libpango-perl
  libpango1.0-0 libpango1.0-dev libpango1.0-doc libpangomm-1.4-1 libpangomm-1.4-dev libpaper-utils libpaper1 libparams-util-perl libparams-validate-perl
  libparse-debcontrol-perl libparse-debianchangelog-perl libparse-recdescent-perl libparted0debian1 libpath-class-perl libpathplan4 libpcap0.8
  libpcap0.8-dev libpci3 libpciaccess-dev libpciaccess0 libpcre3 libpcsc-perl libpcsclite-dev libpcsclite1 libperl-critic-perl libperlio-eol-perl
  libphonon4 libpixman-1-0 libpixman-1-dev libpkcs11-helper1 libplist1 libplot2c2 libpng12-0 libpng12-dev libpod-coverage-perl libpoe-api-peek-perl
  libpoe-component-client-http-perl libpoe-component-client-keepalive-perl libpoe-component-ikc-perl libpoe-perl libpolkit-agent-1-0 libpolkit-backend-1-0
  libpolkit-gobject-1-0 libpolkit-gobject-1-dev libpoppler-glib4 libpoppler5 libpopt-dev libpopt0 libportaudio2 libppi-perl libppix-regexp-perl
  libppix-utilities-perl libpq5 libproxy0 libpstoedit0c2a libpthread-stubs0 libpthread-stubs0-dev libpulse-dev libpulse-mainloop-glib0 libpulse0
  libpurple0 libpython2.6 libqdbm14 libqpol1 libqt4-assistant libqt4-core libqt4-dbus libqt4-designer libqt4-gui libqt4-help libqt4-network libqt4-opengl
  libqt4-qt3support libqt4-script libqt4-scripttools libqt4-sql libqt4-sql-mysql libqt4-svg libqt4-test libqt4-webkit libqt4-xml libqt4-xmlpatterns
  libqtcore4 libqtgui4 libraptor1 libraw1394-11 librdf0 libreadline-dev libreadline5 libreadline6 libreadline6-dev libreadonly-perl libreadonly-xs-perl
  librecode0 libregexp-assemble-perl libregexp-common-perl libregexp-java libresample1 libroken18-heimdal librpc-xml-perl librpcsecgss3 librsvg2-2
  librsvg2-2.18-cil librsvg2-common librtmp0 libruby1.8 libsamplerate0 libsane libsane-extras libsane-hpaio libsasl2-2 libsasl2-modules
  libschroedinger-1.0-0 libsctp1 libsdl-image1.2 libsdl-ttf2.0-0 libsdl1.2debian libselinux1 libselinux1-dev libsemanage-common libsemanage1
  libsensors-applet-plugin0 libsensors4 libsepol1 libsepol1-dev libservlet2.5-java libsetools-tcl libsgutils2-2 libshout3 libsigc++-2.0-0c2a
  libsigc++-2.0-dev libslang2 libslang2-dev libslp1 libslv2-9 libsm-dev libsm6 libsmbclient libsmi2ldbl libsndfile1 libsnmp-base libsnmp15
  libsoap-lite-perl libsocket6-perl libsofia-sip-ua-glib3 libsofia-sip-ua0 libsoup-gnome2.4-1 libsoup-gnome2.4-dev libsoup2.4-1 libsoup2.4-dev
  libsox-fmt-all libsox-fmt-alsa libsox-fmt-ao libsox-fmt-base libsox-fmt-ffmpeg libsox-fmt-mp3 libsox-fmt-oss libsox-fmt-pulse libsp1c2 libspandsp2
  libspectre1 libspeex1 libspeexdsp1 libsqlite0 libsqlite3-0 libsqlite3-dev libsrtp0 libss2 libssh-4 libssh2-1 libssl-dev libstartup-notification0
  libstartup-notification0-dev libstdc++6 libstdc++6-4.4-dev libstrongswan libsub-exporter-perl libsub-identify-perl libsub-install-perl libsub-name-perl
  libsub-uplevel-perl libsvga1 libsvga1-dev libsvn-perl libsvn1 libsybdb5 libsysfs-dev libsysfs2 libt1-5 libtag1-vanilla libtag1c2a libtaglib2.0-cil
  libtalloc2 libtar libtasn1-3 libtasn1-3-dev libtdb1 libtelepathy-glib0 libtemplate-perl libterm-readkey-perl libterm-size-perl
  libtest-checkmanifest-perl libtest-class-perl libtest-deep-perl libtest-exception-perl libtest-mockobject-perl libtest-pod-perl libtext-aspell-perl
  libtext-charwidth-perl libtext-csv-perl libtext-csv-xs-perl libtext-iconv-perl libtext-template-perl libthai-data libthai0 libtheora0 libtidy-0.99-0
  libtie-cphash-perl libtie-toobject-perl libtiff4 libtime-format-perl libtool libtotem-plparser17 libtry-tiny-perl libts-0.0-0 libtwolame0 libudev-dev
  libudev0 libuniconf4.6 libunique-1.0-0 libunistring0 libuniversal-can-perl libuniversal-isa-perl libupower-glib-dev libupower-glib1 liburi-perl
  libusb-0.1-4 libusb-1.0-0 libusb-1.0-0-dev libusb-dev libusbmuxd1 libustr-1.0-1 libutempter0 libuuid-perl libuuid1 libv4l-0 libva-x11-1 libva1
  libvamp-hostsdk3 libvariable-magic-perl libvcdinfo0 libvde0 libvdeplug2 libvirt-bin libvirt0 libvisual-0.4-0 libvlc5 libvorbis0a libvorbisenc2
  libvorbisfile3 libvpb0 libvte-common libvte0.16-cil libvte9 libwant-perl libwavpack1 libwbclient0 libwebkit1.1-cil libwildmidi1 libwind0-heimdal
  libwmf0.2-7 libwnck-common libwnck-dev libwnck2.20-cil libwnck22 libwrap0 libwvstreams4.6-base libwvstreams4.6-extras libwww-mechanize-perl libwww-perl
  libwxbase2.8-0 libwxgtk2.8-0 libx11-6 libx11-data libx11-dev libx11-xcb1 libx86-1 libxapian22 libxau-dev libxau6 libxaw7 libxcb-dri2-0 libxcb-keysyms1
  libxcb-randr0 libxcb-render-util0 libxcb-render-util0-dev libxcb-render0 libxcb-render0-dev libxcb-shm0 libxcb-xv0 libxcb1 libxcb1-dev libxcomposite-dev
  libxcomposite1 libxcursor-dev libxcursor1 libxdamage-dev libxdamage1 libxdg-basedir1 libxdmcp-dev libxdmcp6 libxdot4 libxenstore3.0 libxerces2-java
  libxerces2-java-gcj libxext-dev libxext6 libxfixes-dev libxfixes3 libxfont1 libxft-dev libxft2 libxi-dev libxi6 libxinerama-dev libxinerama1
  libxkbfile-dev libxkbfile1 libxklavier-dev libxklavier16 libxml-feedpp-perl libxml-libxml-perl libxml-parser-perl libxml-regexp-perl
  libxml-sax-expat-perl libxml-sax-perl libxml-simple-perl libxml-twig-perl libxml-xpathengine-perl libxml2 libxml2-dev libxml2-doc libxml2-utils libxmu6
  libxmuu1 libxp-dev libxp6 libxpm4 libxrandr-dev libxrandr2 libxrender-dev libxrender1 libxres-dev libxres1 libxslt1-dev libxslt1.1 libxss1 libxt-dev
  libxt6 libxtst6 libxv1 libxvidcore4 libxvmc1 libxxf86dga1 libxxf86vm-dev libxxf86vm1 libyaml-syck-perl libzbar0 libzephyr4 liferea liferea-data
  link-grammar-dictionaries-en links linphone linphone-common lintian linux-base linux-headers-2.6-amd64 linux-headers-2.6.32-5-common
  linux-image-2.6-amd64 linux-image-2.6.32-5-amd64 linux-libc-dev linux-source-2.6.32 live-build lm-sensors lmodern locales lockfile-progs login logjam
  logrotate lsb-base lsb-release lsof luatex lvm2 lwresd lzma m4 make make-doc makedev makepasswd man-db manpages manpages-dev mawk mdadm
  media-player-info mencoder menu mercurial mercurial-common mesa-common-dev mesa-utils metacity-common mic2 mime-support mingw32-binutils mjpegtools
  mktemp mlocate mobile-broadband-provider-info modemmanager module-init-tools mono-2.0-gac mono-csharp-shell mono-gac mono-gmcs mono-mcs mono-runtime
  mono-xbuild mount mousetweaks mozilla-plugin-gnash mpg123 mtd-utils mtools mupen64plus mutt myspell-en-us mysql-client mysql-common mysql-server nano
  nautilus nautilus-data nautilus-sendto nautilus-sendto-empathy nbd-client ncftp ncurses-base ncurses-bin ncurses-term ndisc6 net-tools netatalk netbase
  netcat-openbsd netcat-traditional netenv netpbm network-manager network-manager-dev network-manager-gnome network-manager-openvpn
  network-manager-openvpn-gnome network-manager-vpnc network-manager-vpnc-gnome nfs-common nfs-kernel-server nmap node normalize-audio notification-daemon
  ntp ntpdate nvclock obex-data-server obexd-client odbcinst odbcinst1debian2 open-iscsi openarena openarena-data openarena-server openbios-ppc
  openbios-sparc openbsd-inetd openhackware openjade openocd openoffice.org openoffice.org-base openoffice.org-calc openoffice.org-common
  openoffice.org-draw openoffice.org-emailmerge openoffice.org-evolution openoffice.org-filter-binfilter openoffice.org-filter-mobiledev
  openoffice.org-gnome openoffice.org-gtk openoffice.org-help-en-us openoffice.org-impress openoffice.org-java-common openoffice.org-math
  openoffice.org-officebean openoffice.org-style-tango openoffice.org-thesaurus-en-us openoffice.org-writer openprinting-ppds openssh-blacklist
  openssh-blacklist-extra openssh-client openssh-server openssl openssl-blacklist openvpn openvpn-blacklist orbit2 org-mode os-prober oss-compat p7zip
  p7zip-full parted passwd patch patchutils pavucontrol pavumeter pbuilder pbzip2 pciutils pcmciautils pcsc-tools perl perl-base perl-doc perl-modules
  perlmagick perltidy pgf php-pear php-services-json php5-cli php5-common php5-dev pidgin pidgin-data pidgin-otr pidgin-sipe pinentry-gtk2 pkg-config
  planner pm-utils po-debconf po4a policycoreutils policykit-1 policykit-1-gnome poppler-utils popularity-contest powertop ppp ppp-dev pristine-tar
  procmail procps ps2eps psmisc pstoedit pulseaudio pulseaudio-esound-compat pulseaudio-module-x11 pulseaudio-utils purifyeps pwgen python python-apt
  python-apt-common python-aptdaemon python-aptdaemon-gtk python-axiom python-beautifulsoup python-brlapi python-cairo python-cddb python-central
  python-chardet python-cherrypy3 python-chm python-clientform python-coherence python-configobj python-crypto python-cssutils python-cups
  python-cupshelpers python-dateutil python-dbus python-debian python-demjson python-dev python-django python-django-tagging python-docutils
  python-evolution python-eyed3 python-feedparser python-gconf python-gdata python-gdbm python-glade2 python-gmenu python-gnome2 python-gnome2-desktop-dev
  python-gnome2-dev python-gnome2-doc python-gnomedesktop python-gnomekeyring python-gobject python-gobject-dev python-gpgme python-gst0.10 python-gtk-vnc
  python-gtk2 python-gtk2-dev python-gtk2-doc python-gtkglext1 python-gtksourceview2 python-html5lib python-httplib2 python-imaging python-iniparse
  python-ipy python-jinja2 python-libvirt python-libxml2 python-louis python-lxml python-mako python-markdown python-markupsafe python-mechanize
  python-minimal python-nevow python-notify python-numpy python-ogg python-old-doctools python-opengl python-openssl python-pam python-paramiko
  python-pexpect python-pkg-resources python-pyasn1 python-pyatspi python-pycurl python-pygments python-pykickstart python-pyorbit python-pypdf
  python-pysqlite2 python-pyvorbis python-qt4 python-rdflib python-renderpm python-reportbug python-reportlab python-reportlab-accel python-roman
  python-rpm python-rsvg python-selinux python-semanage python-sepolgen python-serial python-sip python-software-properties python-sphinx python-sqlite
  python-sqlitecachec python-support python-tagpy python-twisted-bin python-twisted-conch python-twisted-core python-twisted-web python-uno
  python-utidylib python-vte python-webkit python-wnck python-xapian python-xdg python-zope.interface python2.6 python2.6-dev python2.6-minimal
  qemu-keymaps qemu-kvm qemu-system qemu-user-static qemu-utils qt4-qtconfig quagga quagga-doc quilt radeontool rdesktop readline-common realpath recode
  remmina reportbug resolvconf rhythmbox rhythmbox-plugins rinse ripit rpm rpm-common rpm2cpio rsync rsyslog samba samba-common samba-common-bin samba-doc
  sane-utils scons screen seabios seahorse sed selinux-policy-default sensible-utils sensors-applet setools sflphone-daemon sflphone-data sflphone-gnome
  sgml-base sgml-data shared-mime-info sharutils shorewall-core shorewall6 shotwell siege signing-party simple-scan slapd smartmontools smbclient smistrip
  snd snd-gtk-pulse snmp software-center software-properties-gtk sound-juicer soundmodem sox sp spidermonkey-bin squashfs-tools ssh-krb5 sshfs ssl-cert
  strace strongswan strongswan-ikev1 strongswan-ikev2 strongswan-starter subversion sudo svn-buildpackage swat synaptic synergy syslinux syslinux-common
  system-config-printer system-config-printer-udev system-tools-backends system-tools-backends-dev sysv-rc sysvinit sysvinit-utils tar tasksel
  tasksel-data tcl tcl8.4 tcl8.5 tcpd tcpdump telepathy-gabble telepathy-mission-control-5 telepathy-salut telepathy-sofiasip tex-common texinfo
  texlive-base texlive-binaries texlive-common texlive-doc-base texlive-extra-utils texlive-font-utils texlive-fonts-recommended
  texlive-fonts-recommended-doc texlive-generic-recommended texlive-latex-base texlive-latex-base-doc texlive-latex-recommended
  texlive-latex-recommended-doc texlive-luatex texlive-metapost texlive-metapost-doc texlive-pstricks texlive-pstricks-doc texlive-xetex tidy time tinymce
  tipa tk tk8.4 tk8.5 tofrodos tomboy toshset totem totem-common totem-mozilla totem-plugins traceroute transfig transmission-cli transmission-common
  transmission-gtk trustedqsl tsconf ttf-ancient-fonts ttf-dejavu ttf-dejavu-core ttf-dejavu-extra ttf-freefont ttf-lg-aboriginal ttf-liberation ttf-lyx
  ttf-opensymbol ttf-sil-gentium ttf-sil-gentium-basic ttf-takao ttf-takao-gothic ttf-takao-mincho ttf-thai-arundina ttf-thai-tlwg ttf-umefont ttf-umeplus
  ttf-unifont twm twolame tzdata ucf udev udisks ufraw-batch unattended-upgrades unetbootin unetbootin-translations unifont unixodbc uno-libs3 unp unrar
  unzip update-inetd update-manager-core update-manager-gnome update-notifier update-notifier-common upower ure usbmuxd usbutils util-linux vde2 vflib3
  vgabios vim-common vim-tiny vino virt-manager virt-viewer virtinst vlc-data vlc-nox vlc-plugin-notify vlc-plugin-pulse vpnc vzctl w3m wamerican wdiff
  wget whiptail whois winbind wireless-tools wireshark wireshark-common wordnet wordnet-base wordnet-gui wpasupplicant wvdial wwwconfig-common x11-apps
  x11-common x11-session-utils x11-utils x11-xfs-utils x11-xkb-utils x11-xserver-utils x11proto-composite-dev x11proto-core-dev x11proto-damage-dev
  x11proto-dri2-dev x11proto-fixes-dev x11proto-fonts-dev x11proto-gl-dev x11proto-input-dev x11proto-kb-dev x11proto-print-dev x11proto-randr-dev
  x11proto-render-dev x11proto-resource-dev x11proto-video-dev x11proto-xext-dev x11proto-xf86dri-dev x11proto-xf86vidmode-dev x11proto-xinerama-dev xauth
  xbase-clients xbitmaps xca xclip xdemorse xdg-user-dirs xdg-user-dirs-gtk xdg-utils xen-tools xen-utils-common xenstore-utils xfonts-100dpi
  xfonts-100dpi-transcoded xfonts-75dpi xfonts-75dpi-transcoded xfonts-a12k12 xfonts-ayu xfonts-baekmuk xfonts-base xfonts-bitmap-mule
  xfonts-biznet-100dpi xfonts-biznet-75dpi xfonts-biznet-base xfonts-cyrillic xfonts-efont-unicode xfonts-efont-unicode-ib xfonts-encodings
  xfonts-jisx0213 xfonts-kaname xfonts-kapl xfonts-mathml xfonts-mona xfonts-naga10 xfonts-scalable xfonts-terminus xfonts-terminus-dos
  xfonts-terminus-oblique xfonts-thai xfonts-thai-etl xfonts-thai-manop xfonts-thai-nectec xfonts-thai-poonlap xfonts-thai-vor xfonts-tipa xfonts-unifont
  xfonts-utils xfonts-wqy xindy xindy-rules xinit xkb-data xml-core xorg xorg-docs-core xoscope xsane xsane-common xserver-common xserver-xephyr
  xserver-xorg xserver-xorg-core xserver-xorg-dev xserver-xorg-input-all xserver-xorg-input-evdev xserver-xorg-input-synaptics xserver-xorg-input-wacom
  xserver-xorg-video-apm xserver-xorg-video-ark xserver-xorg-video-ati xserver-xorg-video-chips xserver-xorg-video-cirrus xserver-xorg-video-fbdev
  xserver-xorg-video-i128 xserver-xorg-video-intel xserver-xorg-video-mach64 xserver-xorg-video-mga xserver-xorg-video-neomagic
  xserver-xorg-video-openchrome xserver-xorg-video-r128 xserver-xorg-video-radeon xserver-xorg-video-rendition xserver-xorg-video-s3
  xserver-xorg-video-s3virge xserver-xorg-video-savage xserver-xorg-video-siliconmotion xserver-xorg-video-sis xserver-xorg-video-sisusb
  xserver-xorg-video-tdfx xserver-xorg-video-trident xserver-xorg-video-tseng xserver-xorg-video-vesa xserver-xorg-video-vmware xserver-xorg-video-voodoo
  xsltproc xterm xtightvncviewer xtrans-dev xutils-dev xz-utils yelp yum zenity zip zlib1g zlib1g-dev
2160 upgraded, 944 newly installed, 133 to remove and 9 not upgraded.
Need to get 90.5 MB/2,928 MB of archives.
After this operation, 1,287 MB of additional disk space will be used.
Do you want to continue [Y/n]? 

9 April 2014

Petter Reinholdtsen: S3QL, a locally mounted cloud file system - nice free software

For a while now, I have been looking for a sensible offsite backup solution for use at home. My requirements are simple, it must be cheap and locally encrypted (in other words, I keep the encryption keys, the storage provider do not have access to my private files). One idea me and my friends had many years ago, before the cloud storage providers showed up, was to use Google mail as storage, writing a Linux block device storing blocks as emails in the mail service provided by Google, and thus get heaps of free space. On top of this one can add encryption, RAID and volume management to have lots of (fairly slow, I admit that) cheap and encrypted storage. But I never found time to implement such system. But the last few weeks I have looked at a system called S3QL, a locally mounted network backed file system with the features I need. S3QL is a fuse file system with a local cache and cloud storage, handling several different storage providers, any with Amazon S3, Google Drive or OpenStack API. There are heaps of such storage providers. S3QL can also use a local directory as storage, which combined with sshfs allow for file storage on any ssh server. S3QL include support for encryption, compression, de-duplication, snapshots and immutable file systems, allowing me to mount the remote storage as a local mount point, look at and use the files as if they were local, while the content is stored in the cloud as well. This allow me to have a backup that should survive fire. The file system can not be shared between several machines at the same time, as only one can mount it at the time, but any machine with the encryption key and access to the storage service can mount it if it is unmounted. It is simple to use. I'm using it on Debian Wheezy, where the package is included already. So to get started, run apt-get install s3ql. Next, pick a storage provider. I ended up picking Greenqloud, after reading their nice recipe on how to use S3QL with their Amazon S3 service, because I trust the laws in Iceland more than those in USA when it come to keeping my personal data safe and private, and thus would rather spend money on a company in Iceland. Another nice recipe is available from the article S3QL Filesystem for HPC Storage by Jeff Layton in the HPC section of Admin magazine. When the provider is picked, figure out how to get the API key needed to connect to the storage API. With Greencloud, the key did not show up until I had added payment details to my account. Armed with the API access details, it is time to create the file system. First, create a new bucket in the cloud. This bucket is the file system storage area. I picked a bucket name reflecting the machine that was going to store data there, but any name will do. I'll refer to it as bucket-name below. In addition, one need the API login and password, and a locally created password. Store it all in ~root/.s3ql/authinfo2 like this:
[s3c]
storage-url: s3c://s.greenqloud.com:443/bucket-name
backend-login: API-login
backend-password: API-password
fs-passphrase: local-password
I create my local passphrase using pwget 50 or similar, but any sensible way to create a fairly random password should do it. Armed with these details, it is now time to run mkfs, entering the API details and password to create it:
# mkdir -m 700 /var/lib/s3ql-cache
# mkfs.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
  --ssl s3c://s.greenqloud.com:443/bucket-name
Enter backend login: 
Enter backend password: 
Before using S3QL, make sure to read the user's guide, especially
the 'Important Rules to Avoid Loosing Data' section.
Enter encryption password: 
Confirm encryption password: 
Generating random encryption key...
Creating metadata tables...
Dumping metadata...
..objects..
..blocks..
..inodes..
..inode_blocks..
..symlink_targets..
..names..
..contents..
..ext_attributes..
Compressing and uploading metadata...
Wrote 0.00 MB of compressed metadata.
# 
The next step is mounting the file system to make the storage available.
# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
  --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
Using 4 upload threads.
Downloading and decompressing metadata...
Reading metadata...
..objects..
..blocks..
..inodes..
..inode_blocks..
..symlink_targets..
..names..
..contents..
..ext_attributes..
Mounting filesystem...
# df -h /s3ql
Filesystem                              Size  Used Avail Use% Mounted on
s3c://s.greenqloud.com:443/bucket-name  1.0T     0  1.0T   0% /s3ql
#
The file system is now ready for use. I use rsync to store my backups in it, and as the metadata used by rsync is downloaded at mount time, no network traffic (and storage cost) is triggered by running rsync. To unmount, one should not use the normal umount command, as this will not flush the cache to the cloud storage, but instead running the umount.s3ql command like this:
# umount.s3ql /s3ql
# 
There is a fsck command available to check the file system and correct any problems detected. This can be used if the local server crashes while the file system is mounted, to reset the "already mounted" flag. This is what it look like when processing a working file system:
# fsck.s3ql --force --ssl s3c://s.greenqloud.com:443/bucket-name
Using cached metadata.
File system seems clean, checking anyway.
Checking DB integrity...
Creating temporary extra indices...
Checking lost+found...
Checking cached objects...
Checking names (refcounts)...
Checking contents (names)...
Checking contents (inodes)...
Checking contents (parent inodes)...
Checking objects (reference counts)...
Checking objects (backend)...
..processed 5000 objects so far..
..processed 10000 objects so far..
..processed 15000 objects so far..
Checking objects (sizes)...
Checking blocks (referenced objects)...
Checking blocks (refcounts)...
Checking inode-block mapping (blocks)...
Checking inode-block mapping (inodes)...
Checking inodes (refcounts)...
Checking inodes (sizes)...
Checking extended attributes (names)...
Checking extended attributes (inodes)...
Checking symlinks (inodes)...
Checking directory reachability...
Checking unix conventions...
Checking referential integrity...
Dropping temporary indices...
Backing up old metadata...
Dumping metadata...
..objects..
..blocks..
..inodes..
..inode_blocks..
..symlink_targets..
..names..
..contents..
..ext_attributes..
Compressing and uploading metadata...
Wrote 0.89 MB of compressed metadata.
# 
Thanks to the cache, working on files that fit in the cache is very quick, about the same speed as local file access. Uploading large amount of data is to me limited by the bandwidth out of and into my house. Uploading 685 MiB with a 100 MiB cache gave me 305 kiB/s, which is very close to my upload speed, and downloading the same Debian installation ISO gave me 610 kiB/s, close to my download speed. Both were measured using dd. So for me, the bottleneck is my network, not the file system code. I do not know what a good cache size would be, but suspect that the cache should e larger than your working set. I mentioned that only one machine can mount the file system at the time. If another machine try, it is told that the file system is busy:
# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
  --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
Using 8 upload threads.
Backend reports that fs is still mounted elsewhere, aborting.
#
The file content is uploaded when the cache is full, while the metadata is uploaded once every 24 hour by default. To ensure the file system content is flushed to the cloud, one can either umount the file system, or ask S3QL to flush the cache and metadata using s3qlctrl:
# s3qlctrl upload-meta /s3ql
# s3qlctrl flushcache /s3ql
# 
If you are curious about how much space your data uses in the cloud, and how much compression and deduplication cut down on the storage usage, you can use s3qlstat on the mounted file system to get a report:
# s3qlstat /s3ql
Directory entries:    9141
Inodes:               9143
Data blocks:          8851
Total data size:      22049.38 MB
After de-duplication: 21955.46 MB (99.57% of total)
After compression:    21877.28 MB (99.22% of total, 99.64% of de-duplicated)
Database size:        2.39 MB (uncompressed)
(some values do not take into account not-yet-uploaded dirty blocks in cache)
#
I mentioned earlier that there are several possible suppliers of storage. I did not try to locate them all, but am aware of at least Greenqloud, Google Drive, Amazon S3 web serivces, Rackspace and Crowncloud. The latter even accept payment in Bitcoin. Pick one that suit your need. Some of them provide several GiB of free storage, but the prize models are quite different and you will have to figure out what suits you best. While researching this blog post, I had a look at research papers and posters discussing the S3QL file system. There are several, which told me that the file system is getting a critical check by the science community and increased my confidence in using it. One nice poster is titled "An Innovative Parallel Cloud Storage System using OpenStack s SwiftObject Store and Transformative Parallel I/O Approach" by Hsing-Bung Chen, Benjamin McClelland, David Sherrill, Alfred Torrez, Parks Fields and Pamela Smith. Please have a look. Given my problems with different file systems earlier, I decided to check out the mounted S3QL file system to see if it would be usable as a home directory (in other word, that it provided POSIX semantics when it come to locking and umask handling etc). Running my test code to check file system semantics, I was happy to discover that no error was found. So the file system can be used for home directories, if one chooses to do so. If you do not want a locally file system, and want something that work without the Linux fuse file system, I would like to mention the Tarsnap service, which also provide locally encrypted backup using a command line client. It have a nicer access control system, where one can split out read and write access, allowing some systems to write to the backup and others to only read from it. As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

7 April 2014

Andrew Pollock: [life] Day 69: Walk to King Island, a picnic at Wellington Point, the long slow acquisition of some linseed and a split lip

Today was a really good day, right up until the end, when it wasn't so good, but could have been a whole lot worse, so I'm grateful for that. I've been wanting to walk out to King Island at low tide with Zoe for a while, but it's taken about a month to get the right combination of availability, weather and low tide timing to make it possible. Today, there was a low tide at about 10:27am, which I thought would work out pretty well. I wasn't sure if the tide needed to be dead low to get to King Island, so I thought we could get there a bit early and possibly follow the tide out. I invited Megan and Jason to join us for the day and make a picnic of it. It turned out that we didn't need a really low tide, the sand bar connecting King Island to Wellington Point was well and truly accessible well before low tide was reached, so we headed out as soon as we arrived. I'd brought Zoe's water shoes, but from looking at it, thought it would be walkable in bare feet. We got about 10 metres out on the sand and Zoe started freaking out about crabs. I think that incident with the mud crab on Coochiemudlo Island has left her slightly phobic of crabs. So I went back to Jason's car and got her water shoes. I tried to allay her fears a bit by sticking my finger in some of the small holes in the sand, and even got her to do it too. I'm actually glad that I did get her water shoes, because the shell grit got a bit sharp and spiky towards King Island, so I probably would have needed to carry her more than I did otherwise. Along the way to the island we spotted a tiny baby mud crab, and Zoe was brave enough to hold it briefly, so that was good. We walked all the way out and partially around the island and then across it before heading back. The walk back was much slower because where was a massive headwind. Zoe ran out of steam about half way back. She didn't like the sand getting whipped up and stinging her legs, and the wind was forcing the brim of her hat down, so I gave her a ride on my shoulders for the rest of the way back. We had some lunch after we got back to Wellington Point, and Zoe found her second wind chasing seagulls around the picnic area. After an ice cream, we went over to the playground and the girls had a great time playing. It was a pretty good park. There was this huge tree with a really big, thick, horizontal branch only about a metre or two off the ground. All the kids were climbing on it and then shimmying along the branch to the trunk. Zoe's had a few climbs in trees and seems not afraid of it, so she got up and had a go. She did really well and did a combination of scooting along, straddling the branch and doing a Brazilian Jiu-Jitsu-style "bear crawl" along the branch. It was funny seeing different kids' limits. Zoe was totally unfazed by climbing the tree. Megan was totally freaking out. But when it came to walking in bare feet in an inch of sea water, Zoe wanted to climb up my leg like a rat up a rope, in case there were crabs. Each to their own. Zoe wanted to have a swim in the ocean, so I put her into her swimsuit, but had left the water shoes back in the car. Once again, she freaked out about crabs as soon as we got ankle deep in the water, and was freaking out Megan as well, so the girls elected to go back to playing in the park. After a good play in the park, we headed back home. We'd carpooled in Jason's truck, with both girls in the back. I'd half expected Zoe to fall asleep on the way back, but the girls were very hyped up and had a great time playing games and generally being silly in the back. When we got back to our place, Jason was in need of a coffee, so we walked to the Hawthorne Garage and had coffee and babyccinos, before Megan and Jason went home. It was about 3:30pm at this point, and I wanted to make a start on dinner. I was making a wholemeal pumpkin quiche, which I've made a few times before, and I discovered we were low on linseed. I thought I'd push things and see if Zoe was up for a scooter ride to the health food shop to get some more and kill some time. She was up for it, but ran out of steam part way across Hawthorne Park. Fortunately she was okay with walking and didn't want me to carry her and the scooter. It took us about an hour to get to the health food shop. Zoe immediately remembered the place from the previous week where we'd had to stop for a post-meltdown pit stop and declared she needed to go to the toilet again. We finally made it out of the shop. I wasn't looking forward to the long walk back home, but there were a few people waiting for a bus at the bus stop near the health food shop, and on checking the timetable, the bus was due in a couple of minutes, so we just waited for the bus. That drastically shortened the trip back. Zoe managed to drop the container of linseed on the way home from the bus stop, but miraculously the way it landed didn't result in the loss of too much of the contents, it just split the container. So I carefully carried the container home the rest of the way. By this stage it was quite a bit later than I had really wanted to be starting dinner, but we got it made, and Zoe really liked the pumpkin quiche, and ate a pretty good dinner. It was after dinner when things took a turn for the worse. Zoe was eating an ice block for dessert, and for whatever reason, she'd decided to sit in the corner of the kitchen next to the dishwasher, while I was loading it. I was carrying over one of the plates, and the knife managed to fall off the plate, bounce off the open dishwasher door and hit her in the mouth, splitting her lip. Zoe was understandably upset, and I was appalled that the whole thing had happened. She never sits on the kitchen floor, let alone in the corner where the dishwasher is. And this knife came so close to her eye. Fortunately the lip didn't look too bad. It stopped bleeding quickly, and we kept some ice on it and the swelling went down. I hate it when accidents happen on my watch. I feel like I'm fighting the stigma of the incompetent single Dad, or the abusive single Dad, so when Zoe sustains an injury to the face like a fat lip, which could be misinterpreted, I, well, really hate it. This was such a freak accident, and it could have gone so much worse. I'm just so glad she's okay. Zoe recovered pretty well from it, and I was able to brush her teeth without aggravating her lip. She went to bed well, and I suspect she's going to sleep really well. It's a bit cooler tonight, so I'm half-expecting a sleep in in the morning with any luck.

Next.

Previous.