Search Results: "spa"

8 April 2016

Colin Watson: No more Hash Sum Mismatch errors

The Debian repository format was designed a long time ago. The oldest versions of it were produced with the help of tools such as dpkg-scanpackages and consumed by dselect access methods such as dpkg-ftp. The access methods just fetched a Packages file (perhaps compressed) and used it as an index of which packages were available; each package had an MD5 checksum to defend against transport errors, but being from a more innocent age there was no repository signing or other protection against man-in-the-middle attacks. An important and intentional feature of the early format was that, apart from the top-level Packages file, all other files were static in the sense that, once published, their content would never change without also changing the file name. This means that repositories can be efficiently copied around using rsync without having to tell it to re-checksum all files, and it avoids network races when fetching updates: the repository you re updating from might change in the middle of your update, but as long as the repository maintenance software keeps superseded packages around for a suitable grace period, you ll still be able to fetch them. The repository format evolved rather organically over time as different needs arose, by what one might call distributed consensus among the maintainers of the various client tools that consumed it. Of course all sorts of fields were added to the index files themselves, which have an extensible format so that this kind of thing is usually easy to do. At some point a Sources index for source packages was added, which worked pretty much the same way as Packages except for having a different set of fields. But by far the most significant change to the repository structure was the package pools project. The original repository layout put the packages themselves under the dists/ tree along with the index files. The dists/ tree is organised by suite (modern examples of which would be stable , stable-updates , testing , unstable , xenial , xenial-updates , and so on). This meant that making a release of Debian tended to involve copying lots of data around, and implementing the testing suite would have been very costly. Package pools solved this problem by moving individual package files out of dists/ and into a new pool/ tree, allowing those files to be shared between multiple suites with only a negligible cost in disk space and mirror bandwidth. From a database design perspective this is obviously much more sensible. As part of this project, the original Debian dinstall repository maintenance scripts were replaced by da-katie or dak , which among other things used a new apt-ftparchive program to build the index files; this replaced dpkg-scanpackages and dpkg-scansources, and included its own database cache which made a big difference to performance at the scale of a distribution. A few months after the initial implementation of package pools, Release files were added. These formed a sort of meta-index for each suite, telling APT which index files were available (main/binary-i386/Packages, non-free/source/Sources, and so on) and what their checksums were. Detached signatures were added alongside that (Release.gpg) so that it was now possible to fetch packages securely given a public key for the repository, and client-side verification support for this eventually made its way into Debian and Ubuntu. The repository structure stayed more or less like this for several years. At some point along the way, those of us by now involved in repository maintenance realised that an important property had been lost. I mentioned earlier that the original format allowed race-free updates, but this was no longer true with the introduction of the Release file. A client now had to fetch Release and then fetch whichever other index files such as Packages they wanted, typically in separate HTTP transactions. If a client was unlucky, these transactions would fall on either side of a mirror update and they d get a Hash Sum Mismatch error from APT. Worse, if a mirror was unlucky and also didn t go to special lengths to verify index integrity (most don t), its own updates could span an update of its upstream mirror and then all its clients would see mismatches until the next mirror update. This was compounded by using detached signatures, so Release and Release.gpg were fetched separately and could be out of sync. Fixing this has been a long road (the first time I remember talking about this was in late 2007!), and we ve had to take care to maintain client/server compatibility along the way. The first step was to add inline-signed versions of the Release file, called InRelease, so that there would no longer be a race between fetching Release and fetching its signature. APT has had this for a while, Debian s repository supports it as of stretch, and we finally implemented it for Ubuntu six months ago. Dealing with the other index files is more complicated, though; it isn t sensible to inline them, as clients usually only need to fetch a small fraction of all the indexes available for a given suite. The solution we ve ended up with, thanks to Michael Vogt s work implementing it in APT, is called by-hash and should be familiar in concept to people who ve used git: with the exception of the top-level InRelease file, index files for suites that support the by-hash mechanism may now be fetched using a URL based on one of their hashes listed in InRelease. This means that clients can now operate like this:

Fetch dists/xenial/InRelease
Fetch dists/xenial/main/binary-amd64/by-hash/SHA256/46316a202cdae76a73b555414741b11d08c66620b76c470a1623cedcc8a14740 (and so on)
Fetch individual package files

This is now enabled by default in Ubuntu. It s only there as of xenial (16.04), since earlier versions of Ubuntu don t have the necessary support in APT. With this, hash mismatches on updates should be a thing of the past. There will still be some people who won t yet benefit from this. debmirror doesn t support by-hash yet; apt-cacher-ng only supports it as of xenial, although there s an easy configuration workaround. Full archive mirrors must make sure that they put new by-hash files in place before new InRelease files (I just fixed our recommended two-stage sync script to do this; ubumirror still needs some work; Debian s ftpsync is almost correct but needs a tweak for its handling of translation files, which I ve sent to its maintainers). Other mirrors and proxies that have specific handling of the repository format may need similar changes. Please let me know if you see strange things happening as a result of this change. It s useful to check the output of

apt -o
Debug::Acquire::http=true update

to see exactly what requests are being issued.

7 April 2016

Dirk Eddelbuettel: RcppArmadillo 0.6.700.3.0

A new Armadillo release 6.700.3 is out, and we uploaded RcppArmadillo 0.6.700.3.0 to CRAN and Debian. This followed the usual thorough reverse-dependecy checking of by now 216 packages using. Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. Changes in this release are as follows:

Changes in RcppArmadillo version 0.6.700.3.0 (2016-04-05)

Upgraded to Armadillo 6.700.3 (Catabolic Amalgamator Deluxe)

added logmat() for calcuating the matrix logarithm

added regspace() for generating vectors with regularly spaced elements

added logspace() for generating vectors with logarithmically spaced elements

added approx_equal() for determining approximate equality

added trapz() for numerical integration

expanded .save() and .load() with hdf5_binary_trans file type, to save/load data with columns transposed to rows

Courtesy of CRANberries, there is also a diffstat report for this release. As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

5 April 2016

Ian Wienand: Image building in OpenStack CI

Also titled minimal images - maximal effort! The OpenStack Infrastructure Team manages a large continuous-integration system that provides the broad range of testing the OpenStack project requires. Tests are run thousands of times a day across every project, on multiple platforms and on multiple cloud-providers. There are essentially no manual steps in any part of the process, with every component being automated via scripting, a few home-grown tools and liberal doses of Puppet and Ansible. More importantly, every component resides in the public git trees right alongside every other OpenStack project, with contributions actively encouraged. As with any large system, technical debt can build up and start to affect stability and long-term maintainability. OpenStack Infrastructure can see some of this debt accumulating as more testing environments across more cloud-providers are being added to support ever-growing testing demands. Thus a strong focus of recent work has been consolidating testing platforms to be smaller, better defined and more maintainable. This post illustrates some of the background to the issues and describes how these new platforms are more reliable and maintainable.

OpenStack CI Overview Before getting into details, it's a good idea to get a basic big-picture conceptual model of how OpenStack CI testing works. If you look at the following diagram and follow the numbers with the explanation below, hopefully you'll have all the context you need. Overview of OpenStack CI

The developer uploads their code to gerrit via the git-review tool. There is no further action required on their behalf and the developer simply waits for results.
Gerrit provides a JSON-encoded "firehose" output of everything happening to it. New reviews, votes, updates and more all get sent out over this pipe. Zuul is the overall scheduler that subscribes itself to this information and is responsible for managing the CI jobs appropriate for each change.
Zuul has a configuration that tells it what jobs to run for what projects. Zuul can do lots of interesting things, but for the purposes of this discussion we just consider that it puts the jobs it wants run into gearman for a Jenkins master to consume. gearman is a job-server; as they explain it "[gearman] provides a generic application framework to farm out work to other machines or processes that are better suited to do the work". Zuul puts into gearman basically a tuple (job-name, node-type) for each job it wants run, specifying the unique job name to run and what type of node it should be run on.
A group of Jenkins masters are subscribed to gearman as workers. It is these Jenkins masters that will consume the job requests from the queue and actually get the tests running. However, Jenkins needs two things to be able to run a job a job definition (what to actually do) and a slave node (somewhere to do it). The first part what to do is provided by job-definitions stored in external YAML files and processed by Jenkins Job Builder (jjb) in to job configurations for Jenkins. Each Jenkins master gets these definitions pushed to it constantly by Puppet, thus each Jenkins master instance knows about all the jobs it can run automatically. Zuul also knows about these job definitions; this is the job-name part of the tuple we said it put into gearman. The second part somewhere to run the test takes some more explaining. To the next point...
Several cloud companies donate capacity in their clouds for OpenStack to run CI tests. Overall, this capacity is managed by a customised orchestration tool called nodepool. Nodepool watches the gearman queue and sees what requests are coming out of Zuul. It looks at node-type of jobs in the queue and decides what types of nodes need to start and which cloud providers have capacity to satisfy demand. Nodepool will monitor the start-up of the virtual-machines and register the new nodes to the Jenkins master instances.
At this point, the Jenkins master has what it needs to actually get jobs started. When nodepool registers a host to a Jenkins master as a slave, the Jenkins master can now advertise its ability to consume jobs. For example, if a ubuntu-trusty node is provided to the Jenkins master instance by nodepool, Jenkins can now consume from gearman any job it knows about that is intended to run on an ubuntu-trusty slave. Jekins will run the job as defined in the job-definition on that host ssh-ing in, running scripts, copying the logs and waiting for the result. (It is a gross oversimplification, but for the purposes of OpenStack CI, Jenkins is pretty much used as a glorified ssh/scp wrapper. Zuul Version 3, under development, is working to remove the need for Jenkins to be involved at all).
Eventually, the test will finish. The Jenkins master will put the result back into gearman, which Zuul will consume. The slave will be released back to nodepool, which destroys it and starts all over again (slaves are not reused and also have no sensitive details on them, as they are essentially publicly accessible). Zuul will wait for the results of all jobs for the change and post the result back to Gerrit; it either gives a positive vote or the dreaded negative vote if required jobs failed (it also handles merges to git, but we'll ignore that bit for now).

In a nutshell, that is the CI work-flow that happens thousands-upon-thousands of times a day keeping OpenStack humming along.

Image builds So far we have glossed over how nodepool actually creates the images that it hands out for testing. Image creation, illustrated in step 8 above, contains a lot of important details. Firstly, what are these images and why build them at all? These images are where the "rubber hits the road" they are instantiated into the virtual-machines that will run DevStack, unit-testing or whatever else someone might want to test. The main goal is to provide a stable and consistent environment in which to run a wide-range of tests. A full OpenStack deployment results in hundreds of libraries and millions of lines of code all being exercised at once. The testing-images are right at the bottom of all this, so any instability or inconsistency affects everyone; leading to constant fire-firefighting and major inconvenience as all forward-progress stops when CI fails. We want to support a wide number of platforms interesting to developers such as Ubuntu, Debian, CentOS and Fedora, and we also want to and make it easy to handle new releases and add other platforms. We want to ensure this can be maintained without too much day-to-day hands-on. Caching is a big part of the role of these images. With thousands of jobs going on every day, an occasional network blip is not a minor annoyance, but creates constant and difficult to debug failures. We want jobs to rely on as few external resources as possible so tests are consistent and stable. This means caching things like the git trees tests might use (OpenStack just broke the 1000 repository mark), VM images, packages and other common bits and pieces. Obviously a cache is only as useful as the data in it, so we build these images up every day to keep them fresh.

Snapshot images If you log into almost any cloud-provider's interface, they almost certainly have a range of pre-canned images of common distributions for you to use. At first, the base images for OpenStack CI testing came from what the cloud-providers had as their public image types. However, over time, there are a number of issues that emerge:

No two images, even for the same distribution or platform, are the same. Every provider seems to do something "helpful" to the images which requires some sort of workaround.
Providers rarely leave these images alone. One day you would boot the image to find a bunch of Python libraries pip-installed, or a mount-point moved, or base packages removed (all happened).
Even if the changes are helpful, it does not make for consistent and reproducible testing if every time you run, you're on a slightly different base system.
Providers don't have some images you want (like a latest Fedora), or have different versions, or different point releases. All update asynchronously whenever they get around to it.

So the original incarnations of OpenStack CI images were based on these public images. Nodepool would start one of these provider images and then run a series of scripts on it these scripts would firstly try to work-around any quirks to make the images look as similar as possible across providers, and then do the caching, setup things like authorized keys and finish other configuration tasks. Nodepool would then snapshot this prepared image and start instantiating VM's from these snapshots into the pool for testing. If you hear someone talking about a "snapshot image" in OpenStack CI context, that's likely what they are referring to. Apart from the stability of the underlying images, the other issue you hit with this approach is that the number of images being built starts to explode when you take into account multiple providers and multiple regions. Even with just Rackspace and the (now defunct) HP Cloud we would end up creating snapshot images for 4 or 5 platforms across a total of about 8 regions meaning anywhere up to 40 separate image builds happening daily (you can see how ridiculous it was getting in the logging configuration used at the time). It was almost a fait accompli that some of these would fail every day nodepool can deal with this by reusing old snapshots but this leads to a inconsistent and heterogeneous testing environment. Naturally there was a desire for something more consistent a single image that could run across multiple providers in a much more tightly controlled manner.

Upstream-based builds Upstream distributions do provide "cloud-images", which are usually pre-canned .qcow2 format files suitable for uploading to your average cloud. So the diskimage-builder tool was put into use creating images for nodepool, based on these upstream-provided images. In essence, diskimage-builder uses a series of elements (each, as the name suggests, designed to do one thing) that allow you to build a completely customised image. It handles all the messy bits of laying out the image file, tries to be smart about caching large downloads and final things like conversion to qcow2 or vhd. nodepool has used diskimage-builder to create customised images based upon the upstream releases for some time. These are better, but still have some issues for the CI environment:

You still really have no control over what does or does not go into the upstream base images. You don't notice a change until you deploy a new image based on an updated version and things break.
The images still start with a fair amount of "stuff" on them. For example cloud-init is a rather large Python program and has a fair few dependencies. These dependencies can both conflict with parts of OpenStack or end up tacitly hiding real test requirements (the test doesn't specify it, but the package is there as part of another base dependency. Things then break when the base dependencies change). The whole idea of the CI is that (as much as possible) you're not making any assumptions about what is required to run your tests you want everything explicitly included.
An image that "works everywhere" across multiple cloud-providers is quite a chore. cloud-init hasn't always had support for config-drive and Rackspace's DHCP-less environment, for example. Providers all have their various different networking schemes or configuration methods which needs to be handled consistently.

If you were starting this whole thing again, things like LXC/Docker to keep "systems within systems" might come into play and help alleviate some of the packaging conflicts. Indeed they may play a role in the future. But don't forget that DevStack, the major CI deployment mechanism, was started before Docker existed. And there's tricky stuff with networking and Neutron going on. And things like iSCSI kernel drivers that containers don't support well. And you need to support Ubuntu, Debian, CentOS and Fedora. And you have hundreds of developers already relying on what's there. So change happens incrementally, and in the mean time, there is a clear need for a stable, consistent environment.

Minimal builds To this end, diskimage-builder now has a serial of "minimal" builds that are really that systems with essentially nothing on them. For Debian and Ubuntu this is achieved via debootstrap, for Fedora and CentOS we replicate this with manual installs of base packages into a clean chroot environment. We add on a range of important elements that make the image useful; for example, for networking, we have simple-init which brings up the network consistently across all our providers but has no dependencies to mess with the base system. If you check the elements provided by project-config you can see a range of specific elements that OpenStack Infra runs at each image build (these are actually specified by in arguments to nodepool, see the config file, particularly diskimages section). These custom elements do things like caching, using puppet to install the right authorized_keys files and setup a few needed things to connect to the host. In general, you can see the logs of an image build provided by nodepool for each daily build. So now, each day at 14:14 UTC nodepool builds the daily images that will be used for CI testing. We have one image of each type that (theoretically) works across all our providers. After it finishes building, nodepool uploads the image to all providers (p.s. the process of doing this is so insanely terrible it spawned shade; this deserves many posts of its own) at which point it will start being used for CI jobs. If you wish to replicate this entire process, the build-image.sh script, run on an Ubuntu Trusty host in a virtualenv with diskimage-builder will get you pretty close (let us know of any issues!).

DevStack and bare nodes There are two major ways OpenStack projects test their changes:

Running with DevStack, which brings up a small, but fully-functional, OpenStack cloud with the change-under-test applied. Generally tempest is then used to ensure the big-picture things like creating VM's, networks and storage are all working.
Unit-testing within the project; i.e. what you do when you type tox -e py27 in basically any OpenStack project.

To support this testing, OpenStack CI ended up with the concept of bare nodes and devstack nodes.

A bare node was made for unit-testing. While tox has plenty of information about installing required Python packages into the virtualenv for testing, it doesn't know anything about the system packages required to build those Python packages. This means things like gcc and library -devel packages which many Python packages use to build bindings. Thus the bare nodes had an ever-growing and not well-defined list of packages that were pre-installed during the image-build to support unit-testing. Worse still, projects didn't really know their dependencies but just relied on their testing working with this global list that was pre-installed on the image.
In contrast to this, DevStack has always been able to bootstrap itself from a blank system to a working OpenStack deployment by ensuring it has the right dependencies installed. We don't want any packages pre-installed here because it hides actual dependencies that we want explicitly defined within DevStack otherwise, when a user goes to deploy DevStack for their development work, things break because their environment differs slightly to the CI one. If you look at all the job definitions in OpenStack, by convention any job running DevStack has a dsvm in the job name this referred to running on a "DevStack Virtual Machine" or a devstack node. As the CI environment has grown, we have more and more testing that isn't DevStack based (puppet apply tests, for example) that rather confusingly want to run on a devstack node because they do not want dependencies installed. While it's just a name, it can be difficult to explain!

Thus we ended up maintaining two node-types, where the difference between them is what was pre-installed on the host and yes, the bare node had more installed than a devstack node, so it wasn't that bare at all!

Specifying Dependencies Clearly it is useful to unify these node types, but we still need to provide a way for the unit-test environments to have their dependencies installed. This is where a tool called bindep comes in. This tool gives project authors a way to specify their system requirements in a similar manner to the way their Python requirements are kept. For example, OpenStack has the concept of global requirements those Python dependencies that are common across all projects so version skew becomes somewhat manageable. This project now has some extra information in the other-requirements.txt file, which lists the system packages required to build the Python packages in the global-requirements list. bindep knows how to look at these lists provided by projects and get the right packages for the platform it is running on. As part of the image-build, we have a cache-bindep element that can go through every project and build a list of the packages it requires. We can thus pre-cache all of these packages onto the images, knowing that they are required by jobs. This both reduces the dependency on external mirrors and improves job performance (as the packages are locally cached) but doesn't pollute the system by having everything pre-installed. Package installation can now happen via the way we really should be doing it as part of the CI job. There is a job-macro called install-distro-packages which a test can use to call bindep to install the packages specified by the project before the run. You might notice the script has a "fallback" list of packages if the project does not specify it's own dependencies this essentially replicates the environment of a bare node as we transition to projects more strictly specifying their system requirements. We can now start with a blank image and all the dependencies to run the job can be expressed by and within the project leading to a consistent and reproducible environment without any hidden dependencies. Several things have broken as part of removing bare nodes this is actually a good thing because it means we have revealed areas where we were making assumptions in jobs about what the underlying platform provides. There's a few other job-macros that can do things like provide MySQL/Postgres instances for testing or setup other common job requirements. By splitting these types of things out from base-images we also improve the performance of jobs who don't waste time doing things like setting up databases for jobs that don't need it. As of this writing, the bindep work is new and still a work-in-progress. But the end result is that we have no more need for a separate bare node type to run unit-tests. This essentially halves the number of image-builds required and brings us to the goal of a single image for each platform running all CI.

Conclusion While dealing with multiple providers, image-types and dependency chains has been a great effort for the infra team, to everyone's credit I don't think the project has really noticed much going on underneath. OpenStack CI has transitioned to a situation where there is a single image type for each platform we test that deploys unmodified across all our providers and runs all testing environments equally. We have better insight into our dependencies and better tools to manage them. This leads to greatly decreased maintenance burden, better consistency and better performance; all great things to bring to OpenStack CI!

Matthew Garrett: There's more than one way to exploit the commons

There's a piece of software called XScreenSaver. It attempts to fill two somewhat disparate roles:

Provide a functioning screen lock on systems using the X11 windowing system, a job made incredibly difficult due to a variety of design misfeatures in said windowing system[1]
Provide cute graphical output while the screen is locked

XScreenSaver does an excellent job of the second of these[2] and is pretty good at the first, which is to say that it only suffers from a disastrous security flaw once very few years and as such is certainly not appreciably worse than any other piece of software.

Debian ships an operating system that prides itself on stability. The Debian definition of stability is a very specific one - rather than referring to how often the software crashes or misbehaves, it refers to how often the software changes behaviour. Debian is very reluctant to upgrade software that is part of a stable release, to the extent that developers will attempt to backport individual security fixes to the version they shipped rather than upgrading to a release that contains all those security fixes but also adds a new feature. The argument here is that the new release may also introduce new bugs, and Debian's users desire stability (in the "things don't change" sense) more than new features. Backporting security fixes keeps them safe without compromising the reason they're running Debian in the first place.

This all makes plenty of sense at a theoretical level, but reality is sometimes less convenient. The first problem is that security bugs are typically also, well, bugs. They may make your software crash or misbehave in annoying but apparently harmless ways. And when you fix that bug you've also fixed a security bug, but the ability to determine whether a bug is a security bug or not is one that involves deep magic and a fanatical devotion to the cause so given the choice between maybe asking for a CVE and dealing with embargoes and all that crap when perhaps you've actually only fixed a bug that makes the letter "E" appear in places it shouldn't and not one that allows the complete destruction of your intergalactic invasion fleet means people will tend to err on the side of "Eh fuckit" and go drinking instead. So new versions of software will often fix security vulnerabilities without there being any indication that they do so[3], and running old versions probably means you have a bunch of security issues that nobody will ever do anything about.

But that's broadly a technical problem and one we can apply various metrics to, and if somebody wanted to spend enough time performing careful analysis of software we could have actual numbers to figure out whether the better security approach is to upgrade or to backport fixes. Conversations become boring once we introduce too many numbers, so let's ignore that problem and go onto the second, which is far more handwavy and social and so significantly more interesting.

The second problem is that upstream developers remain associated with the software shipped by Debian. Even though Debian includes a tool for reporting bugs against packages included in Debian, some users will ignore that and go straight to the upstream developers. Those upstream developers then have to spend at least 15 or so seconds telling the user that the bug they're seeing has been fixed for some time, and then figure out how to explain that no sorry they can't make Debian include a fixed version because that's not how things work. Worst case, the stable release of Debian ends up including a bug that makes software just basically not work at all and everybody who uses it assumes that the upstream author is brutally incompetent, and they end up quitting the software industry and I don't know running a nightclub or something.

From the Debian side of things, the straightforward solution is to make it more obvious that users should file bugs with Debian and not bother the upstream authors. This doesn't solve the problem of damaged reputation, and nor does it entirely solve the problem of users contacting upstream developers. If a bug is filed with Debian and doesn't get fixed in a timely manner, it's hardly surprising that users will end up going upstream. The Debian bugs list for XScreenSaver does not make terribly attractive reading.

So, coming back to the title for this entry. The most obvious failure of the commons is where a basically malicious actor consumes while giving nothing back, but if an actor with good intentions ends up consuming more than they contribute that may still be a problem. An upstream author releases a piece of software under a free license. Debian distributes this to users. Debian's policies result in the upstream author having to do more work. What does the upstream author get out of this exchange? In an ideal world, plenty. The author's software is made available to more people. A larger set of developers is willing to work on making improvements to the software. In a less ideal world, rather less. The author has to deal with bug mail about already fixed bugs. The author's reputation may be harmed by user exposure to said fixed bugs. The author may get less in the way of useful bug fixes or features because people are running old versions rather than fixing new ones. If the balance tips towards the latter, the author's decision to release their software under a free license has made their life more difficult.

Most discussions about Debian's policies entirely ignore the latter scenario, focusing more on the fact that the author chose to release their software under a free license to begin with. If the author is unwilling to handle the consequences of that, goes the argument, why did they do it in the first place? The unfortunate logical conclusion to that argument is that the author realises that they made a huge mistake and never does so again, and woo uh oops.

The irony here is that one of Debian's foundational documents, the Debian Free Software Guidelines, makes allowances for this. Section 4 allows for distribution of software in Debian even if the author insists that modified versions[4] are renamed. This allows for an author to make a choice - allow themselves to be associated with the Debian version of their work and increase (a) their userbase and (b) their support load, or try to distinguish what Debian ship from their identity. But that document was ratified in 1997 and people haven't really spent much time since then thinking about why it says what it does, and so this tradeoff is rarely considered.

Free software doesn't benefit from distributions antagonising their upstreams, even if said upstream is a cranky nightclub owner. Debian's users are Debian's highest priority, but those users are going to suffer if developers decide that not using free licenses improves their quality of life. Kneejerk reactions around specific instances aren't helpful, but now is probably a good time to start thinking about what value Debian bring to its upstream authors and how that can be increased. Failing to do so doesn't serve users, Debian itself or the free software community as a whole.

[1] The X server has no fundamental concept of a screen lock. This is implemented by an application asking that the X server send all keyboard and mouse input to it rather than to any other application, and then that application creating a window that fills the screen. Due to some hilarious design decisions, opening a pop-up menu in an application prevents any other application from being able to grab input and so it is impossible for the screensaver to activate if you open a menu and then walk away from your computer. This is merely the most obvious problem - there are others that are more subtle and more infuriating. The only fix in this case is to nuke the site from orbit.

[2] There's screenshots here. My favourites are the one that emulate the electrical characteristics of an old CRT in order to present a more realistic depiction of the output of an Apple 2 and the one that includes a complete 6502 emulator.

[3] And obviously new versions of software will often also introduce new security vulnerabilities without there being any indication that they do so, because who would ever put that in their changelog. But the less ethically challenged members of the security community are more likely to be looking at new versions of software than ones released three years ago, so you're probably still tending towards winning overall

[4] There's a perfectly reasonable argument that all packages distributed by Debian are modified in some way

comments

4 April 2016

Raphaël Hertzog: My Free Software Activities in February and March 2016

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. I skipped my monthly report last time so this one will cover two months. I will try to list only the most important things to not make it too long.

The Debian Handbook I worked with Ryuunosuke Ayanokouzi to prepare a paperback version of the Japanese translation of my book. Thanks to the efforts of everybody, it s now available. Unfortunately, Lulu declined to take it in distribution program so it won t be available on traditional bookstores (like Amazon, etc.). The reason is that they do not support non-latin character sets in the meta-data. I tried to cheat a little bit by inputting the description in English (still explaining that the book was in Japanese) but they rejected it nevertheless because the English title could mislead people. So the paperback is only available on lulu.com. Fortunately, the shipping costs are reasonable if you pick the most economic offer. Following this I invited the Italian, Spanish and Brazilian Portuguese translators to complete the work (they were close will all the strings already translated, mainly missing translated screenshots and some backcover content) so that we can also release paperback versions in those languages. It s getting close to completion for them. Hopefully we will have those available until next month. Distro Tracker In early February, I tweaked the configuration to send (by email) exceptions generated by incoming mails and by routine task. Before this they were logged but I did not take the time to look into them. This quickly brought a few issues into light and I fixed them as they appeared: for instance the bounce handling code was getting confused when the character case was not respected, and it appears that some emails come back to us after having been lowercased. Also the code was broken when the References field used more than one line on incoming control emails. This brought into light a whole class of problems with the database storing twice the same email with only differing case. So I did further work to merge all those duplicate entries behind a single email entry. Later, the experimental Sources files changed and I had to tweak the code to work with the removal of the Files field (relying instead on Checksums-* to find out the various files part of the entry). At some point, I also fixed the login form to not generate an exception when the user submits an empty form. I also decided that I no longer wanted to support Django 1.7 in distro tracker as Django 1.8 is the current LTS version. I asked the Debian system administrators to update the package on tracker.debian.org with the version in jessie-backports. This allowed me to fix a few deprecation warnings that I kept triggering because I wanted the code to work with Django 1.7. One of those warnings was generated by django-jsonfield though and I could not fix it immediately. Instead I prepared a pull request that I submitted to the upstream author. Oh, and a last thing, I tweaked the CSS to densify the layout on the package page. This was one of the most requested changes from the people who were still preferring packages.qa.debian.org over tracker.debian.org. Kali and new pkg-security team As part of my Kali work, I have been fixing RC bugs in Debian packages that we use in Kali. But in many cases, I stumbled upon packages whose maintainers were really missing in action (MIA). Up to now, we were only doing non-maintainers upload (NMU) but I want to be able to maintain those packages more effectively so we created a new pkg-security team (we re only two right now and we have no documentation yet, but if you want to join, you re welcome, in particular if you maintain a package which is useful in the security field). arm64 work. The first 3 packages that we took over (ssldump, sucrack, xprobe) are actually packages that were missing arm64 builds. We just started our arm64 port on Kali and we fixed them for that architecture. Since they were no longer properly maintained, in most cases it was just a matter of using dh_autoreconf to get up-to-date config. sub,guess files. We still miss a few packages on arm64: vboot-utils (that we will likely take over soon since it s offered for adoption), ruby-libv8 and ruby-therubyracer, ntopng (we have to wait a new luajit which is only in experimental right now). We also noticed that dh-make-golang was not available on arm64, after some discussion on #debian-buildd, I filed two bugs for this: #819472 on dh-make-golang and #819473 on dh-golang. RC bug fixing. hdparm was affected by multiple RC bugs and the release managers were trying to get rid of it from testing. This removed multiple packages that were used by Kali and its users. So I investigated the situation of that package, convinced the current maintainers to orphan it, asked for new maintainers on debian-devel, reviewed multiple updates prepared by the new volunteers and sponsored their work. Now hdparm is again RC-bug free and has the latest upstream version. We also updated jsonpickle to 0.9.3-1 to fix RC bug #812114 (that I forwarded upstream first). Systemd presets support in init-system-helpers. I tried to find someone (to hire) to implement the system preset feature I requested in #772555 but I failed. Still Andreas Henriksson was kind enough to give it a try and sent a first patch. I tried it and found some issues so I continued to improve it and simplify it I submitted an updated patch and pinged Martin Pitt. He pointed me to the DEP-8 test failures that my patch was creating. I quickly fixed those afterwards. This patch is in use in Kali and lets us disable network services by default. I would like to see it merged in Debian so that everybody can setup systemd preset file and have their desire respected at installation time. Misc bug reports. I filed #813801 to request a new upstream release of kismet. Same for masscan in #816644 and for wkhtmltopdf in #816714. We packaged (before Debian) a new upstream release of ruby-msgpack and found out that it was not building on armel/armhf so we filed two upstream tickets (with a suggested fix). In #814805, we asked the pyscard maintainer to reinstate python-pyscard that was dropped (keeping only the Python3 version) as we use the Python 2 version in Kali. And there s more: I filed #816553 (segfault) and #816554 against cdebootstrap. I asked for dh-python to have a better behaviour after having being bitten by the fact that dh with python3 was not doing what I expected it to do (see #818175). And I reported #818907 against live-build since it is failing to handle a package whose name contains an upper case character (it s not policy compliant but dpkg supports them). Misc packaging I uploaded Django 1.9.2 to unstable and 1.8.9 to jessie-backports. I provided the supplementary information that Julien Cristau asked me in #807654 but despite this, this jessie update has been ignored for the second point release in a row. It is now outdated until I update it to include the security fixes that have been released in the mean time but I m not yet sure that I will do it the lack of cooperation of the release team for that kind of request is discouraging. I sponsored multiple uploads of dolibarr (on security update notably) and tcpdf (to fix one RC bug). Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Ian Wienand: Image building for OpenStack CI -- Minimal images, big effort

A large part of OpenStack Infrastructure teams recent efforts has been focused on moving towards more stable and maintainable CI environments for testing.

The developer uploads their code to gerrit via the git-review tool. They wait.
Gerrit provides a JSON-encoded "firehose" output of everything happening to it. New reviews, votes, updates and more all get sent out over this pipe. Zuul is the overall scheduler that subscribes itself to this information and is responsible for managing the CI jobs appropriate for each change.
Zuul has a configuration that tells it what jobs to run for what projects. Zuul can do lots of interesting things, but for the purposes of this discussion we just consider that it puts the jobs it wants run into gearman for a Jenkins host to consume. gearman is a job-server; as they explain it "[gearman] provides a generic application framework to farm out work to other machines or processes that are better suited to do the work".
A group of Jenkins hosts are subscribed to gearman as workers. It is these Jenkins hosts that will consume the job requests from the queue and actually get the tests running. Jenkins needs two things to be able to run a job -- a job definition (what to actually do) and a slave node (somewhere to do it). The first part -- what to do -- is provided by job-definitions stored in external YAML files and processed by Jenkins Job Builder (jjb) in to job configurations for Jenkins. Thus each Jenkins instance knows about all the jobs it might need to run. Zuul also knows about these job definitions, so you can see how we now have a mapping where Zuul can put a job into gearman saying "run test foo-bar-baz" and a Jenkins host can consume that request and know what to do. The second part -- somewhere to run the test -- takes some more explaining. To the next point...
Several cloud companies donate capacity in their clouds for OpenStack to run CI tests. Overall, this capacity is managed by nodepool -- a customised orchestration tool. Nodepool watches the gearman queue and sees what requests are coming out of Zuul, and decides what type of capacity to provide and in what clouds to satisfy the outstanding job queue. Nodepool will start-up virtual-machines as required, and register those nodes to the Jenkins instances.
At this point, Jenkins has what it needs to actually get jobs started. When nodepool registers a host to Jenkins as a slave, the Jenkins host can now advertise its ability to consume jobs. For example, if a ubuntu-trusty node is provided to the Jenkins instance by nodepool, Jenkins can now consume a job from the queue intended to run on an ubuntu-trusty host. It will run the job as defined in the job-definition -- ssh-ing into the host, running scripts, copying the logs and waiting for the result. (It is a gross oversimplification, but but Jenkins is pretty much a glorified ssh/scp wrapper to OpenStack CI. Zuul Version 3, under development, is working to remove the need for Jenkins to be involved at all).
Eventually, the test will finish. Jenkins will put the result back into gearman, which Zuul will consume. The slave will be released back to nodepool, which destroys it and starts all over again (slaves are not reused and also have no sensitive details on them, as they are essentially publicly accessible). Zuul will wait for the results of all jobs and post the result back to Gerrit and give either a positive vote or the dreaded negative vote if required jobs failed (it also handles merges to git, but we'll ignore that bit for now).

In a nutshell, that is the CI work-flow that happens thousands-upon-thousands of times a day keeping OpenStack humming along.

Image builds There is, however, another more asynchronous part of the process that hides a lot of details the rest of the system relies on. Illustrated in step 8 above, this is the management of the images that tests are being run upon. Above we we said that a test runs on a ubuntu-trusty, centos, fedora or some other type of node, but glossed over where these images come from. Firstly, what are these images, and why build them at all? These images are where the "rubber hits the road" -- where DevStack, functional testing or whatever else someone might want to test is actually run for real. Caching is a big part of the role of these images. With thousands of jobs going on every day, an occasional network blip is not a minor annoyance, but creates constant and difficult to debug CI failures. We want the images that CI runs on to rely on as few external resources as possible so test runs are as stable as possible. This means caching all the git trees tests might use, VM images consumed by various tests and other common bits and pieces. Obviously a cache is only as useful as the data in it, so we build these images up every day to keep them fresh.

Provider images If you log into almost any cloud-provider's interface, they almost certainly have a range of pre-canned images of common distributions for you to use. At first, the base images for OpenStack CI testing came from what the cloud-providers had as their public image types. However, over time, there are a number of issues that emerge:

No two images, even for the same distribution or platform, are the same. Every provider seems to do something "helpful" to the images which requires some sort of workaround.
Providers rarely leave these images alone. One day you would boot the image to find a bunch of Python libraries pip-installed, or a mount-point moved, or base packages removed (all happened).
Even if the changes are helpful, it does not make for consistent and reproducible testing if every time you run, you're on a slightly different base system.
Providers don't have some images you want (like a latest Fedora), or have different versions, or different point releases. All update asynchronously whenever they get around to it.

So the original incarnations of building images was that nodepool would start one of these provider images, run a bunch of scripts on it to make a base-image (do the caching, setup keys, etc), snapshot it and then start putting VM's based on these images into the pool for testing (if you hear someone talking about a "snapshot image" in OpenStack CI context, that's what it means). The first problem you hit here is that the number of images being built starts to explode when you take into account multiple providers and multiple regions. Even with just Rackspace and the (now defunct) HP Cloud we would end up snapshot images for 4 or 5 platforms across a total of about 8 regions -- meaning anywhere up to 40 separate image builds happening. It was almost a fait accompli that some images would fail every day -- nodepool can deal with this by reusing old snapshots; but this leads to a inconsistent and heterogeneous testing environment. OpenStack is like a gigantic Jenga tower, with a full DevStack deployment resulting in hundreds of libraries and millions of lines of code all being exercised at once. The testing images are right at the bottom of all this, and it doesn't take much to make the whole thing fall over (see points about providers not leaving images alone). This leads to constant fire-firefighting and everyone annoyed as all CI stops. Naturally there was a desire for something much more consistent -- a single image that could run across multiple providers in a much more tightly controlled manner.

Upstream-based builds Upstream distributions do provide their "cloud-images", which are usually pre-canned .qcow2 format files suitable for uploading to your average cloud. So the diskimage-builder tool was put into use creating images for nodepool, based on these upstream-provided images. In essence, diskimage-builder uses a series of elements (each, as the name suggests, designed to do one thing) that allow you to build a completely customised image. It handles all the messy bits of laying out the image file, tries to be smart about caching large downloads and final things like conversion to qcow2 or vhd or whatever your cloud requires. nodepool has used diskimage-builder to create customised images based upon the upstream releases for some time. These are better, but still have some issues for the CI environment:

You still really have no control over what does or does not go into the upstream base images. You don't notice a change until you deploy a new image based on an updated version and things break.
The images still have a fair amount of "stuff" on them. For example cloud-init is a rather large Python program and has a fair few dependencies. These dependencies can both conflict with parts of OpenStack or end up tacitly hiding real test requirements (the test doesn't specify it, but the package is there as part of another base dependency -- things break when the base dependencies change). The whole idea of the CI is that (as much as possible) you're not making any assumptions about what is required to run your tests -- you want everything explicitly included. (If you were starting this whole thing again, things like Docker to keep "systems within systems" might come into play. Indeed they may be in the future. But don't forget that DevStack, the major CI deployment mechanism, was started before Docker existed. And there's tricky stuff with networking and Neutron etc going on).
An image that "works everywhere" across multiple cloud-providers is quite a chore. cloud-init hasn't always had support for config-drive and Rackspace's DHCP-less environment, for example. Providers all have their various different networking schemes or configuration methods which needs to be handled consistently.

Minimal builds To this end, diskimage-builder now has a serial of "minimal" builds that are really that -- systems with essentially nothing on them. For Debian and Ubuntu, this is achieved via debootstrap, for Fedora and CentOS we replicate this with manual installs of base packages into a clean chroot environment. We add-on a range of important elements that make the image useful; for example, for networking, we have simple-init which brings up the network consistently across all our providers but has no dependencies to mess with the base system. If you check the elements provided by project-config you can see a range of specific elements that OpenStack Infra runs at each image build (these are actually specified by in arguments to nodepool, see the config file, particularly diskimages section). These custom elements do things like caching, using puppet to install the right authorized_keys files and setup a few needed things to connect to the host. In general, you can see the logs of an image build provided by nodepool for each daily build. So now, each day at 14:00 UTC nodepool builds the daily images that will be used for CI testing. We have one image of each type that (theoretically) works across all our providers. After it finishes building, nodepool uploads the image to all providers (p.s. the process of doing this is so insanely terrible it spawned shade; this deserves many posts of its own) at which point it will start being used for CI jobs. If you wish to replicate this entire process, the build-image.sh script, run on an Ubuntu Trusty host in a virtualenv with diskimage-builder will get you pretty close (let us know of any issues!).

Dependencies But guess what, there's more! Along the way, OpenStack CI ended up with the concept of bare nodes and devstack nodes. A bare node was one that was used for functional testing; i.e. what you do when you type tox -e py27 in basically any OpenStack project. The problem here is that tox has plenty of information about installing required Python packages into the virtualenv for testing; but it doesn't know anything about the system packages required to build the Python libraries. This means things like gcc and -devel packages which many Python libraries use to build library bindings. In contrast to this, DevStack has always been able to bootstrap itself from a blank system to a working OpenStack deployment, ensuring it has the right libraries, etc, installed to get everything working. If you look at all the job definitions, anything running DevStack has a dsvm in the job name; which referred to to "DevStack Virtual Machine"; or basically specifying what was installed on the host (and yes, the bare node had more installed, so it wasn't that bare at all!). Thus we don't want packages pre-installed for DevStack, because it hides actual devstack dependencies that we want explicitly defined. But the bare nodes, used for functional testing, were different -- there was an every-growing and not well-defined list of packages that were pre-installed on those nodes to make sure functional testing worked. In general, you don't want jobs relying on something like this; we want to be sure if jobs have a dependency, they require it explicitly. This is where a tool called bindep comes in. OpenStack has the concept of global requirements -- those Python dependencies that are common across all projects so version skew becomes somewhat manageable. This now has some extra information in the other-requirements.txt file, which lists the system-packages required to build the Python-packages in the requirements list. bindep knows how to look at this and get the right packages for the platform it is running on. Indeed -- remember how it was previously mentioned we want to minimise dependencies on external resources at runtime? Well we can pre-cache all of these packages onto the images, knowing that they are likely to be required by packages. How do we get the packages installed? The way we really should be doing it -- as part of the CI job. There is a macro called install-distro-packages which uses bindep to install those packages as required by the global-requirements list. The result -- no more need for the bare node type to run functional tests! In all cases we can start with essentially a blank image and all the dependencies to run the job are expressed by and within the job -- leading to a consistent and reproducible environment. Several things have broken as part of removing bare nodes -- this is actually a good thing because it means we have revealed areas where we were making assumptions in jobs about what the underlying platform provides; issues that get fixed by thinking about and ensuring we have correct dependencies bringing up jobs. There's a few other macros there that do things like provide MySQL/Postgres instances or setup other common job requirements. By splitting these out we also improve the performance of jobs who now only bring in the dependencies they need -- we don't waste time doing things like setting up databases for jobs that don't need it.

Conclusion While dealing with multiple providers, image-types and dependency chains has been a great effort for the infra team, to everyone's credit I don't think the project has really noticed much going on underneath. OpenStack CI has transitioned to a situation where there is a single image type for each platform we test that deploys unmodified across all our providers. We have better insight into our dependencies and better tools to manage them. This leads to greatly decreased maintenance burden, better consistency and better performance; all great things to bring to OpenStack CI!

2 April 2016

Thorsten Alteholz: My Debian Activities in March 2016

FTP assistant This month I marked 226 packages for accept and rejected 22. I also sent 5 emails to maintainers asking questions. It seems to be that a rather quiet month is behind us. As I have seen some packages with strange debian/copyright in binNEW, I wonder whether also the archive should be checked regularly. Maybe it is time to file some bugs Debian LTS This was my twenty-first month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. Due to outstanding hours that were redistributed, my all in all workload had been 14.25h. As Wheezy LTS didn t start yet and I am not able to do normal security uploads, I sent debdiffs to the security team. Btw. this can be done by everybody and the way to go is described in chapter 5.8.5 of the Debian Developer s Reference. Altogether I sent the following debdiffs for

extplorer to fix CVE-2015-0896
inspircd to fix CVE-2015-8702
libmatroska to fix CVE-2015-8792
libstruts1.2-java to fix CVE-2015-0899
fuseiso to fix two temporary issues
minissdpd to fix CVE-2016-3178 and CVE-2016-3179
tlslite to fix CVE-2015-3220

As the security team wants to update Wheezy and Jessie with only one DSA, whenever applicable I created debdiffs for both releases. Up to now the results can be seen in DSA 3526-1, DSA 3527-1 and DSA 3536-1. As tlslite has been removed from Wheezy during today s point release, I am afraid that was a wasted effort. Other stuff My node activities this month involved uploads of: node-component-consoler, node-generator-supported, node-xmlhttprequest-ssl, node-co, node-uid-umber, node-url-join, node-uri-path, node-read-file, node-nth-check, node-base62, node-require-dir, node-for-in, node-obj-util, node-normalize-it-url, node-delve, node-function-bind, node-seq, node-json-localizer, node-through, node-addressparser, node-ansi-regex, node-crypto-cacerts, node-decamelize, node-array-find-index, node-require-main-filename, node-invert-kv, node-starttls. To fix one or the other bug I also uploaded: node-connect, node-mysql. I also forwarded bug #809252, which is tagged as security relevant in the BTS, to the Node Security Project. I even got one answer stating that the report arrived. We will see what happens next. At least after 45 days another email might arrive

1 April 2016

Francois Marier: How Safe Browsing works in Firefox

Firefox has had support for Google's Safe Browsing since 2005 when it started as a stand-alone Firefox extension. At first it was only available in the USA, but it was opened up to the rest of the world in 2006 and moved to the Google Toolbar. It then got integrated directly into Firefox 2.0 before the public launch of the service in 2007. Many people seem confused by this phishing and malware protection system and while there is a pretty good explanation of how it works on our support site, it doesn't go into technical details. This will hopefully be of interest to those who have more questions about it.

Browsing Protection The main part of the Safe Browsing system is the one that watches for bad URLs as you're browsing. Browsing protection currently protects users from:

malware sites,

deceptive sites (including phishing and social engineering sites), and

sites hosting potentially unwanted software.

If a Firefox user attempts to visit one of these sites, a warning page will show up instead, which you can see for yourself here:

fake malware page

fake unwanted software page

fake phishing page

The first two warnings can be toggled using the `browser.safebrowsing.malware.enabled` preference (in `about:config`) whereas the last one is controlled by `browser.safebrowsing.enabled`.

List updates It would be too slow (and privacy-invasive) to contact a trusted server every time the browser wants to establish a connection with a web server. Instead, Firefox downloads a list of bad URLs every 30 minutes from the server (`browser.safebrowsing.provider.google.updateURL`) and does a lookup against its local database before displaying a page to the user. Downloading the entire list of sites flagged by Safe Browsing would be impractical due to its size so the following transformations are applied:

each URL on the list is canonicalized,

then hashed,

of which only the first 32 bits of the hash are kept.

The lists that are requested from the Safe Browsing server and used to flag pages as malware/unwanted or phishing can be found in `urlclassifier.malwareTable` and `urlclassifier.phishTable` respectively. If you want to see some debugging information in your terminal while Firefox is downloading updated lists, turn on `browser.safebrowsing.debug`. Once downloaded, the lists can be found in the cache directory:

`~/.cache/mozilla/firefox/XXXX/safebrowsing/` on Linux

`~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/` on Mac

`C:\Users\XXXX\AppData\Local\mozilla\firefox\profiles\XXXX\safebrowsing\` on Windows

Resolving partial hash conflicts Because the Safe Browsing database only contains partial hashes, it is possible for a safe page to share the same 32-bit hash prefix as a bad page. Therefore when a URL matches the local list, the browser needs to know whether or not the rest of the hash matches the entry on the Safe Browsing list. In order resolve such conflicts, Firefox requests from the Safe Browsing server (`browser.safebrowsing.provider.mozilla.gethashURL`) all of the hashes that start with the affected 32-bit prefix and adds these full-length hashes to its local database. Turn on `browser.safebrowsing.debug` to see some debugging information on the terminal while these "completion" requests are made. If the current URL doesn't match any of these full hashes, the load proceeds as normal. If it does match one of them, a warning interstitial page is shown and the load is canceled.

Download Protection The second part of the Safe Browsing system protects users against malicious downloads. It was launched in 2011, implemented in Firefox 31 on Windows and enabled in Firefox 39 on Mac and Linux. It roughly works like this:

Download the file.

Check the main URL, referrer and redirect chain against a local blocklist (`urlclassifier.downloadBlockTable`) and block the download in case of a match.

On Windows, if the binary is signed, check the signature against a local whitelist (`urlclassifier.downloadAllowTable`) of known good publishers and release the download if a match is found.

If the file is not a binary file then release the download.

Otherwise, send the binary file's metadata to the remote application reputation server (`browser.safebrowsing.downloads.remote.url`) and block the download if the server indicates that the file isn't safe.

Blocked downloads can be unblocked by right-clicking on them in the download manager and selecting "Unblock". While the download protection feature is automatically disabled when malware protection (`browser.safebrowsing.malware.enabled`) is turned off, it can also be disabled independently via the `browser.safebrowsing.downloads.enabled` preference. Note that Step 5 is the only point at which any information about the download is shared with Google. That remote lookup can be suppressed via the `browser.safebrowsing.downloads.remote.enabled` preference for those users concerned about sending that metadata to a third party.

Types of malware The original application reputation service would protect users against "dangerous" downloads, but it has recently been expanded to also warn users about unwanted software as well as software that's not commonly downloaded. These various warnings can be turned on and off in Firefox through the following preferences:

`browser.safebrowsing.downloads.remote.block_dangerous`

`browser.safebrowsing.downloads.remote.block_dangerous_host`

`browser.safebrowsing.downloads.remote.block_potentially_unwanted`

`browser.safebrowsing.downloads.remote.block_uncommon`

and tested using Google's test page. If you want to see how often each "verdict" is returned by the server, you can have a look at the telemetry results for Firefox Beta.

Privacy One of the most persistent misunderstandings about Safe Browsing is the idea that the browser needs to send all visited URLs to Google in order to verify whether or not they are safe. While this was an option in version 1 of the Safe Browsing protocol (as disclosed in their privacy policy at the time), support for this "enhanced mode" was removed in Firefox 3 and the version 1 server was decommissioned in late 2011 in favor of version 2 of the Safe Browsing API which doesn't offer this type of real-time lookup. Google explicitly states that the information collected as part of operating the Safe Browsing service "is only used to flag malicious activity and is never used anywhere else at Google" and that "Safe Browsing requests won't be associated with your Google Account". In addition, Firefox adds a few privacy protections:

Query string parameters are stripped from URLs we check as part of the download protection feature.

Cookies set by the Safe Browsing servers to protect the service from abuse are stored in a separate cookie jar so that they are not mixed with regular browsing/session cookies.

When requesting complete hashes for a 32-bit prefix, Firefox throws in a number of extra "noise" entries to obfuscate the original URL further.

On balance, we believe that most users will want to keep Safe Browsing enabled, but we also make it easy for users with particular needs to turn it off.

Learn More If you want to learn more about how Safe Browsing works in Firefox, you can find all of the technical details on the Safe Browsing and Application Reputation pages of the Mozilla wiki or you can ask questions on our mailing list. Google provides some interesting statistics about what their systems detect in their transparency report and offers a tool to find out why a particular page has been blocked. Some information on how phishing sites are detected is also available on the Google Security blog, but for more detailed information about all parts of the Safe Browsing system, see the following papers:

All Your IFrames Are Belong to Us

Content-Agnostic Malware Protection

Ghost in the Browser

31 March 2016

Antoine Beaupr : My free software activities, march 2016

Debian Long Term Support (LTS) This is my 4th month working on Debian LTS, started by Raphael Hertzog at Freexian. I spent half of the month away on a vacation so little work was done, especially since I tried to tackle rather large uploads like NSS and Xen. I also worked on the frontdesk shift last week.

Frontdesk That work mainly consisted of figuring out how to best help the security team with the last uploads to the Wheezy release. For those who don't know, Debian 7 Wheezy, or "oldstable", is going to be unsupported by the security team starting end of april, and the Debian 6 Squeeze (the previous LTS) is now unsupported. The PGP signatures on the archived release have started yielding expiration errors which can be ignored but that are really a strong reminder that it is really time to upgrade. So the LTS team is now working towards backporting a few security issues from squeeze to wheezy, and this is what I focused on during triage work. I have identified the following high priority packages I will work on after I complete my work on the Xen and NSS packages (detailed below):

libidn

icu

phpmyadmin

tomcat6

optipng

srtp

dhcpcd

python-tornado

Updates to NSS and Xen I have spent a lot of time testing and building packages for NSS and Xen. To be fair, Brian May did most of the work on the Xen packages, and I merely did some work to test the packages on Koumbit's infrastructure, something which I will continue doing in the next month. For NSS, wheezy and jessie are in this weird state where patches were provided to the security team all the way back in November yet were never tested. Since then, yet more issues came up and I worked hard to review and port patches for those new security issues to wheezy. I'll followup on both packages in the following month.

Other free software work

Android TL;DR: there's an even longer version of this with the step-by-step procedures and that I will update as time goes on in my wiki. I somehow inherited an Android phone recently, on a loan from a friend because the phone broke one too many times and she got a new one from her provider. This phone is a HTC One S "Ville", which is fairly old, but good enough to play with and give me a mobile computing platform to listen to podcasts, play music, access maps and create GPS traces. I was previously doing this with my N900, but that device is really showing its age: very little development is happening on it, the SDK is closed source and the device itself is fairly big, compared to the "Ville". Plus, the SIM card actually works on the Ville so, even though I do not have an actual contract with a cell phone provider (too expensive, too invasive on my privacy), I can still make emergency phone calls (911)! Plus, since there is good wifi on the machine, I can use it to connect to the phone system with the built-in SIP client, send text messages through SMS (thanks to VoIP.ms SMS support) or Jabber. I have also played around with LibreSignal, the free software replacement for Signal, which uses proprietary google services. Yes, the VoIP.ms SMS app also uses GCM, but hopefully that can be fixed. (While I was writing this, another Debian Developer wrote a good review of Signal so I am happy to skip that step. Go read that.)
See also my apps list for a more complete list of the apps I have installed on the phone. I welcome recommendations on cool free software apps I should use!
I have replaced the stock firmware on the phone with Cyanogenmod 12.1, which was a fairly painful experience, partly because of the difficult ambiance on the #cyanogenmod channel on Freenode, where I had extreme experiences: a brave soul helped me through the first flashing process for around 2 hours, nicely holding my hand at every step. Other times, I have seen flames and obtuse comments from people being vulgar, brutal, obnoxious, if not sometimes downright homophobic and sexist from other users. It is clearly a community that needs to fix their attitude. I have documented everything I could in details in this wiki page, in case others want to resuscitate their old phones, but also because I ended up reinstalling the freaking phone about 4 times, and was getting tired of forgetting how to do it every time. I am somewhat fascinated by Android: here is the Linux-based device that should save us all from the proprietary Apple nightmares of fenced in gardens and censorship. Yet, public Android downloads are hidden behind license agreements, even though the code itself is free, which has led fellow Debian developers to work on making libre rebuilds of Androids to workaround this insanity. But worse: all phones are basically proprietary devices down to the core. You need custom firmware to be loaded on the machine for it to boot at all, from the bootloader all the way down to the GSM baseband and Wifi drivers. It is a minefield of closed source software, and trying to run free software on there is a bit of a delusion, especially since the baseband has so much power over the phone. Still, I think it is really interesting to run free software on those machines, and help people that are stuck with cell phones get familiar with software freedom. It seems especially important to me to make Android developers aware of software freedom considering how many apps are available for free yet on which it is not possible to contribute significantly because the source code is not published at all, or published only on the Google Store, instead of the more open and public F-Droid repository which publishes only free software. So I did contribute. This month, I am happy to announce that I contributed to the following free software projects on Android:

listmyapps: feature request, patch

Locker: bug report, added to f-droid

Sensorium: bug report

LibreSignal: feature request and testing

I have also reviewed the literature surrounding Telegram, a popular messaging app rival to Signal and Whatsapp. Oddly enough, my contributions to Wikipedia on that subject were promptly reverted which made me bring up the subject on the page's Talk page. This lead to an interesting response from the article's main editors which at least added that the "its security features have been contested by security researchers and cryptography experts". Considering the history of Telegram, I would keep people away from it and direct people to use Signal instead, even though Signal has similar metadata issues, mostly because Telegram's lack of response to the security issues that were outlined by fellow security researchers. Both systems suffer from a lack of federation as well, which is a shame in this era of increasing centralization. I am not sure I will put much more work in developing for Android for now. It seems like a fairly hostile platform to work on, and unless I have specific pain points I want to fix, it feels so much better to work on my own stuff in Debian. Which brings me to my usual plethora of free software projects I got stuck in this month.

IRC projects irssi-plugin-otr had a bunch of idiotic bugs lying around, and I had a patch that I hadn't submitted upstream from the Debian package, which needed a rebuild because the irssi version changed, which is a major annoyance. The version in sid is now a snapshot because upstream needs to make a new release but at least it should fix things for my friends running unstable and testing. Hopefully those silly uploads won't be necessary in the future. That's for the client side. On the server side, I have worked on updating the Atheme-services package to the latest version, which actually failed because the upstream libmowgli is missing release tags, which means the Debian package for it is not up-to-date either. Still, it is nice to have a somewhat newer version, even though it is not the latest and some bugs were fixed. I have also looked at making atheme reproducible but was surprised at the hostility of the upstream. In the end, it looks like they are still interested in patches, but they will be a little harder to deploy than for Charybdis, so this could take some time. Hopefully I will find time in the coming weeks to test the new atheme services daemon on the IRC network I operate.

Syncthing I have also re-discovered Syncthing, a file synchronization software. Amazingly, I was having trouble transferring a single file between two phones. I couldn't use Bluetooth (not sure why), the "Wifi sharing" app was available only on one phone (and is proprietary, and has a limit of 4MB files), and everything else requires an account, the cloud, or cabling. So. Just heading to f-droid, install syncthing, flash a few qr-codes around and voil : files are copied over! Pretty amazing: the files were actually copied over the local network, using IPv6 link-local addresses, encryption and the DHT. Which is a real geeky way of saying it's completely fast, secure and fast. Now, I found a few usability issues, so much that I wrote a whole usability story for the developers, which were really appreciative of my review. Some of the issues were already fixed, others were pretty minor. Syncthing has a great community, and it seems like a great project I encourage everyone to get familiar with.

Battery stats The battery-status project I mentionned previously has been merged with the battery-stats project (yes, the names are almost the same, which is confusing) and so I had to do some work to fix my Python graph script, which was accepted upstream and will be part of Debian officially from now on, which is cool. The previous package was unofficial. I have also noticed that my battery has a significantly than when I wrote the script. Whereas it was basically full back then, it seems now it has lost almost 15% of its capacity in about 6 months. According to the calculations of the script:
this battery will reach end of life (5%) in 935 days, 19:07:58.336480, on 2018-10-23 12:06:07.270290
Which is, to be fair, a good life: it will somewhat work for about three more years.

Playlist, git-annex and MPD in Python On top of my previously mentioned photos-import script, I have worked on two more small programs. One is called get-playlist and is an extension to git-annex to easily copy to the local git-annex repository all files present in a given M3U playlist. This is useful for me because my phone cannot possibly fit my whole MP3 collection, and I use playlists in GMPC to tag certain files, particularly the `Favorites` list which is populated by the "star" button in the UI. I had a lot of fun writing this script. I started using elpy as an IDE in Emacs. (Notice how Emacs got a new webpage, which is a huge improvement was had been basically unchanged since the original version, now almost 20 years old, and probably written by RMS himself.) I wonder how I managed to stay away from Elpy for so long, as it glues together key components of Emacs in an elegant and functional bundle:

Company: the "vim-like" completion mode i had been waiting for forever

Jedi: context-sensitive autocompletion for Python

Flymake: to outline style and syntax errors (unfortunately not the more modern Flycheck)

inline documentation...

In short, it's amazing and makes everything so much easier to work with that I wrote another script. The first program wouldn't work very well because some songs in the playlists had been moved, so I made another program, this time to repair playlists which refer to missing files. The script is simply called fix-playlists, and can operate transparently on multiple playlists. It has a bunch of heuristics to find files and uses a MPD server as a directory to search into. It can edit files in place or just act as a filter.

Useful snippets Writing so many scripts, so often, I figured I needed to stop wasting time always writing the same boilerplate stuff on top of every file, so I started publishing Yasnippet-compatible file snippets, in my snippets repository. For example, this report is based on the humble lts snippet. I also have a base license snippet which slaps the AGPLv3 license on top of a Python file. But the most interesting snippet, for me, is this simple script snippet which is a basic scaffolding for a commandline script that includes argument processing, logging and filtering of files, something which I was always copy-pasting around.

Other projects And finally, a list of interesting issues en vrac:

there's a new Bootstrap 4 theme for Ikiwiki. It was delivering content over CDNs, which is bad for privacy issues, so I filed an issue which was almost immediately fixed by the author!

I found out about BitHub, a tool to make payments with Bitcoins for patches submitted on OWS projects. It wasn't clear to me where to find the demo, but when I did, I quickly filed a PR to fix the documentation. Given the number of open PRs there and the lack of activity, I wonder if the model is working at all...

a fellow Debian Developer shared his photos workflow, which was interesting to me because I have my own peculiar script, which I never mentionned here, but which I ended up sharing with the author

Chris Lamb: Free software activities in March 2016

Here is my monthly update covering a large part of what I have been doing in the free software world (previously):

Submitted a number of pull requests to the Ansible sever configuration tool:
- Add the ability to specify the --allow-unauthenticated to APT. (#2023)
- Ignore EPIPE when flushing standard input and output to avoid unnecessary and ugly tracebacks. (#14774)
- Add an option for ansible-playbook to exit with an error if no plays were matched. (#14742)
Various improvements to django-slack, a library to easily post messages to the Slack group-messaging utility from projects using the Django web development framework:
- Merged a patch from Joshua Blum to update the logging integration documentation. (#39)
- Fixed an issue when rendering various exceptions would in themselves raise an exception. (#40)
- Improved and merged a patch from Patrick Cloke to actually allow settings to be overriden using the @override_settings decorator. (#37)
Updated local-debian-mirror, a package that attempts to make it easy to maintain a partial local mirror of the Debian archive:
- Added an option to exclude DEP-11 files. (ee03b21)
- Added an option to exclude Translation-* files. (28bbf34)
- Corrected the phrasing of the debconf prompt to not ask a direct question to appease Lintian. (13fccc7b)
Updated django-keyerror a library to post exceptions to the KeyError.com error tracking service to support pushing tracebacks from the Celery queue processor using the task_failure signal. (#1)
Submitted a number of pull requests for django-zebra, a collection of utilities that make it easier to integrate the Stripe payment processor and the Django web development framework:
- Add the ability to test an incoming Stripe webhook. (#44)
- Prefer @require_POST over an explicit check in the view. (#43)
- Move away from the deprecated django.conf.urls.patterns method. (#45)
Updated django-template-tests, a tool to perform simple static analysis on Django templates prior to production to improve the robustness of dynamic test generation. (e98f9fc)
Fixed django-force-logout a library to allow administrators to log user's out of Django projects to not require a custom JSON serialiser. (c0e5d64)
Corrected the documentation for travis.debian.net my hosted script to easily test and build Debian packages on the Travis CI continuous integration platform to reflect the actual default mirror. (4c862f0)
Blogged about generating dynamic Python tests using metaclasses.

Debian

Presented Reproducible Builds - fulfilling the original promise of free software at FOSSASIA '16.
Uploaded libfiu (0.94-4), adding a patch from Logan Rose to fix a FTBFS with ld --as-needed.

My work in the Reproducible Builds project was also covered in more depth in Lunar's weekly reports (#44, #45, #46, #47).

LTS

This month I have been paid to work 7 hours on Debian Long Term Support (LTS). Whilst the LTS team will take over support from the Security Team on April 26, 2016, in the meantime I did the following:

Archived the squeeze distribution (via the FTPteam).
Assisted in preparing updates for python-django.
Helping end-users migrate to wheezy now that squeeze LTS has reached end-of-life.

Bugs filed

debmirror: "Won't mirror without dists/experimental/main/binary-amd64/Packages.gz" due to .gz migration in experimental

I also filed 71 FTBFS bugs against ada-reference-manual, arachne-pnr, cdrkit, civicrm, cura-engine, dopewars, eclipse-cdt, fail2ban, fcitx-table-other, fleet, flint, flycheck, git-build-recipe, gitmagic, gnome-audio, golang-ginkgo, golang-github-armon-gomdb, golang-github-aws-aws-sdk-go, golang-github-gorilla-handlers, golang-github-hashicorp-raft-mdb, golang-github-jinzhu-gorm, golang-github-miekg-pkcs11, golang-goyaml, gstreamer0.10, gstreamer1.0, haskell-http2, hidrd, hplip, httpfs2, jedit, kodi-pvr-dvbviewer, kodi-pvr-mythtv, kodi-pvr-vuplus, kodi-pvr-wmc, libraw1394, libtest-file-contents-perl, metview, mpi4py, ndoutils, opensp, phpunit, python-glanceclient, python-oslo.messaging, python-phabricator, rrdtool, ruby-actionpack-xml-parser, ruby-acts-as-list, ruby-acts-as-list, ruby-acts-as-list, ruby-acts-as-list, ruby-amqp, ruby-buff-config, ruby-buff-extensions, ruby-clockwork, ruby-compass, ruby-em-synchrony, ruby-numru-misc, ruby-rabl, ruby-rails-deprecated-sanitizer, ruby-riddle, ruby-safe-yaml, ruby-solve, ruby-toml, ruby-twitter, ruby-typhoeus, ruby-varia-model, shared-mime-info, siproxd, stylish, sup-mail & uima-as.

FTP Team

As a Debian FTP assistant I ACCEPTed 143 packages: acme-tiny, berkshelf-api, circlator, cloud-utils, corsix-th, cronic, diaspora-installer, dub, dumb-init, firehol, firetools, flask-bcrypt, flask-oldsessions, flycheck, ganeti, geany-plugins, git-build-recipe, git-phab, gnome-shell-extension-caffeine, gnome-shell-extension-mediaplayer, golang-github-cheggaaa-pb, golang-github-coreos-ioprogress, golang-github-cyberdelia-go-metrics-graphite, golang-github-cznic-ql, golang-github-elazarl-goproxy, golang-github-hashicorp-hil, golang-github-mitchellh-go-wordwrap, golang-github-mvdan-xurls, golang-github-paulrosania-go-charset, golang-github-xeipuuv-gojsonreference, golang-github-xeipuuv-gojsonschema, grilo-plugins, gtk3-nocsd, herisvm, identity4c, lemonldap-ng, libisal, libmath-gsl-perl, libmemcached-libmemcached-perl, libplack-middleware-logany-perl, libplack-middleware-logwarn-perl, libpng1.6, libqmi, librdf-generator-http-perl, libtime-moment-perl, libvirt-php, libxml-compile-soap-perl, libxml-compile-wsdl11-perl, linux, linux-tools, mdk-doc, mesa, mpdecimal, msi-keyboard, nauty, node-addressparser, node-ansi-regex, node-argparse, node-array-find-index, node-base62, node-co, node-component-consoler, node-crypto-cacerts, node-decamelize, node-delve, node-for-in, node-function-bind, node-generator-supported, node-invert-kv, node-json-localizer, node-normalize-git-url, node-nth-check, node-obj-util, node-read-file, node-require-dir, node-require-main-filename, node-seq, node-starttls, node-through, node-uid-number, node-uri-path, node-url-join, node-xmlhttprequest-ssl, ocrmypdf, octave-netcdf, open-infrastructure-container-tools, osmose-emulator, pdal, pep8, pg-backup-ctl, php-guzzle, printrun, pydocstyle, pysynphot, python-antlr3, python-biom-format, python-brainstorm, python-django-adminsortable, python-feather-format, python-gevent, python-lxc, python-mongoengine, python-nameparser, python-pdal, python-pefile, python-phabricator, python-pika-pool, python-pynlpl, python-qtawesome, python-requests-unixsocket, python-saharaclient, python-stringtemplate3, r-cran-adegraphics, r-cran-assertthat, r-cran-bold, r-cran-curl, r-cran-data.table, r-cran-htmltools, r-cran-httr, r-cran-lazyeval, r-cran-mcmc, r-cran-openssl, r-cran-pbdzmq, r-cran-rncl, r-cran-uuid, rawtran, reel, ruby-certificate-authority, ruby-rspec-pending-for, ruby-ruby-engine, ruby-ruby-version, scribus-ng, specutils, symfony, tandem-mass, tdb, thrift, udfclient, vala, why3, wmaker, xdg-app & xiccd.

30 March 2016

Ian Wienand: Durable photo workflow

Ever since my kids were born I have accumulated thousands of digital happy-snaps and I have finally gotten to a point where I'm quite happy with my work-flow. I have always been extremely dubious of using any sort of external all-in-one solution to managing my photos; so many things seem to shut-down, cease development or disappear, all leaving you to have to figure out how to migrate to the next latest thing (e.g. Picasa shutting down). So while there is nothing complicated or even generic about them, there are a few things in my photo-scripts repo that might help others who like to keep a self-contained archive. Firstly I have a simple script to copy the latest photos from the SD card (i.e. those new since the last copy -- this is obviously very camera specific). I then split by date so I have a simple flat directory layout with each week's photos in it. With the price of SD cards and my rate of filling them up, I don't even bother wiping them at this point, but just keep them in the safe as a backup. For some reason I have a bit of a thing about geotagging all the photos so I know where I took them. Certainly some cameras do this today, but mine does not. I have a two-progned approach; I have a geotag script and then a small website easygeotag.info which quickly lets met translate a point on Google maps to exiv2 command-line syntax. Since I take a lot of photos in the same place, the script can store points by name in a small file sourced by the script. Adding comments to the photos is done with perhaps the lesser-known cousin of EXIF -- IPTC. Some time ago I wrote python bindings for libiptcdata and it has been working just fine ever since. Debian's python-iptcdata comes with a inbuilt script to set title and caption, which is easily wrapped. What I like about this is that my photos are in a simple directory layout, with all metadata embedded within the actual image files in very standarised formats that should be readable by anywhere I choose to host them. For sharing, I then upload to Flickr. I used to have a command-line script for this, but have found the web uploader works even better these days. It reads the IPTC data for titles and comments, and gets the geotag info for nice map displays. I manually coralle them into albums, and the Flickr "guest pass" is perfect for then sharing albums to friends and family without making them jump through hoops to register on a site to get access to the photos, or worse, host them myself. I consider Flickr a cache, because (even though I pay) I expect it to shut-down or turn evil at any time. Interestingly, their AI tagging is often quite accurate, and I imagine will only get better. This is nice extra metadata that you don't have to spend time on yourself. The last piece has always been the "hit by a bus" component of all this. Can anyone figure out access to all these photos if I suddenly disappear? I've tried many things here -- at one point I was using rdiff-backup to sync encrypted bundles up to AWS for example; but I very clearly found the problem in that when I forgot the keep the key safe and couldn't unencrypt any of my backups (let alone anyone else figuring all this out). Finally Google Nearline seems to be just what I want. It's off-site, redundant and the price is right; but more importantly I can very easily give access to the backup bucket to anyone with a Google address, who can then just hit a website to download the originals from the bucket (I left the link with my other "hit by a bus" bits and pieces). Of course what they then do with this data is their problem, but at least I feel like they have a chance. This even has an rsync like interface in the client, so I can quickly upload the new stuff from my home NAS (where I keep the photos in a RAID0). I've been doing this now for 350 weeks and have worked through some 25,000 photos. I used to get an album up every week, but as the kids get older and we're closer to family I now do it in batches about once a month. I do wonder if my kids will ever be interested in tagged and commented photos with pretty much their exact location from their childhood ... I doubt it, but it's nice to feel like I have a good chance of still having them if they do.

Mike Gabriel: Pushing X.org Git repos to Github et al.

TL;DR; If you want to know why cloning several of the X.org repositories to Github or GitLab instances fails and how this can be worked around, you may want to continue reading. Why we stumbled over this issue... As a joint effort of the Arctica Project, TheQVD and X2Go, Ulrich Sibiller and I are currently preparing a build workflow for the nxagent X-server (version 3) [1] that allows building nxagent against the modular X.org 7.0 (using autoconf and automake) rather than the monolithic build workflow of X.org 6.9 (using ancient imake). Our goal is to rewind all X.org components required for building nxagent back to a state where nxagent successfully builds and runs. Then we will go through various, (probably) monthly cycles of

bumping commit level of required X.org components
rebuild and test nxagent
fix FTBFS, if any
fix broken functionality, if any

Our first hurdle... After Ulrich now has a functioning nxagent-against-X.org-7.0-workflow locally, we want to get everything into the Arctica Project's Github namespace. Which fails...

[mike@minobo xorg.upstream]$ git clone https://anongit.freedesktop.org/git/xorg/lib/libX11.git
Cloning into 'libX11'...
remote: Counting objects: 18520, done.
remote: Compressing objects: 100% (3441/3441), done.
remote: Total 18520 (delta 15094), reused 18325 (delta 14954)
Receiving objects: 100% (18520/18520), 5.98 MiB   549.00 KiB/s, done.
Resolving deltas: 100% (15094/15094), done.
Checking connectivity... done.
[mike@minobo xorg.upstream]$ cd libX11/
[mike@minobo libX11 (master)]$ git remote rename origin upstream 
[mike@minobo libX11 (master)]$ git remote add origin git@github.com:ArcticaProject/libX11.git
[mike@minobo libX11 (master)]$ git push --mirror origin 
Counting objects: 18520, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3301/3301), done.
remote: error: object 70d5e4d45dd7bf1e05b099cb5a4dd529344084f0: missingSpaceBeforeDate: invalid author/committer line - missing space before date
remote: fatal: Error in object
error: pack-objects died of signal 13
error: failed to push some refs to 'git@github.com:ArcticaProject/libX11.git'

This issue should be a known issue at X.org / Mesa upstream [2]. The underlying problem occurs with Git 2.5 and above... If you run git-fsck from Git (>= 2.5) against several of the X.org Git repositories, you will stumble over the above issue. For libX11.git, we see these errors reported when running git-fsck from current Debian unstable:

mike@sid:~/xorg/libX11.git$ git fsck 
Checking object directories: 100% (256/256), done.
error in tag 70d5e4d45dd7bf1e05b099cb5a4dd529344084f0: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag a3bfe698090a5d41f1e9acb1b57a049085d6b04e: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag c1fc8c9aec1bc92d03d08d5e986dd40b194a7a3e: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag d4c27f7e7d8e2ea32418f341ad85c33f3b76862d: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 79780379aee21bf4d4bcb046a3b54774893ea6b1: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag a638de80ec20dc1de625cd323ed19a6646fecf2e: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag f6a6414e4e52a971a35fd118c350567e2383d034: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag b5c6810e21be2e7ccfac8f0539d46bf75dbe50a0: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 86cec9d20428cc190ec7278a0abe481b180288ac: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag ac912988091574790c7cffbbc2c60b8c59f8fcef: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 2d01674092951cd3284b3b099978b2436ea468ad: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 922d0534154668918eaa09a36cc5cd53fc6b71b7: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 8ef91b9cc0a4a7b0361f6b40b3f735221c8272cb: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 1516173727005a87aa501e1ad708d72e6ae6e753: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 77c9e63a3abff1593ccbed249b2c3ae7f68e1833: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 58044c7087137ce2b521297dbb31934ccaedc94e: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 2e0d061c6830891dcc856a04aac900e5bb6d0779: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 2f58c7cc4cbf3d30927f6301039b6bf018ed38fd: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 77fc2e1e26019c3ba92275d249fe1979570255d2: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 9b2e06fa262f23dbcbbccd42be6477e08a452268: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 7f99410bb9cc9f7cbbc2d43d8ad044771a6eec0a: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 99b59b83f126b4abb9da89f577a5b8147d543c25: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 97170c33f99761acd932f968e21d6d7664e7ba80: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag b39966d452127c992987d082f69321f950ae4c39: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 05e20c3bd8ac6b7e873607db73894486a4ac0519: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag f478497b5a3e9d6b92237e0f408c9f0f18108f52: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag c5ed936f28ce70c8932e33d67a6e06d140e35ba8: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 2b6beba295a6ff997dfe15a732c95ae6577fa735: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 32edbb1e9c5b7d1c6f8639dab85e5368d303df7f: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 490fbe2afb9ffc7057c0eb150f7cdd4d4fa8686c: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag fc605ac48bd7d675d6d04162a0f3bfebe27f2037: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 6289ecc26d1d505f395c59da744a9354d7aad2a2: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 8c9c30dac72aebd1a2c31a55a47c0ac11bc328db: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 3b7aa57994924be0dad693e837559d9ee900299c: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag b02ad9a8ec96e3c1027ca71289aeef5b8748e17e: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 4402ab1a86226a9cd0ce84fb8147b1c4958c684d: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag bb02b4fbde316644c771d28d5edeeb9b213a6d6a: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 95b6462c70213b562c9d2b8726eda706edc9c456: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag a99fea43ad53907623f269942c55237c238645e0: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 0c417ce98a6cf927ce7a1359b198b2bb746c707e: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag ce3ff6b102f360fc9dcd3a85361f44da054de0b1: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 2fb7b16b38a365c5009bd170d4463634c1f3de26: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 95570f484bfde5fc2ed8548cb17b87d8114a2866: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag d6e5895c104cfe0d135494a605013dd2910a93a1: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 49be7144523c4a56afcb389a9aa021d167710d2c: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 020353d1b982ac938d229039ca13bfea5793fcd9: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 991792d4f8477ab65ad4d8a0245c9a3bbb1f0294: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag e8037555efe299a7143370b485f754f6b08ffac7: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 4fdc8e36a5022f3e32d5567305d8e9cde4011f1a: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 9c93a19a76163328e30a7306bb0babe3175fb9a2: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 9f90bf4424250f80340b3e2803d343c74d2adb0b: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 2e7a8c7974777e3e869de9013025dc3f61633e1e: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 59b7c06da2a64d72668394e9239e3dfe8a5cff59: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 0263c4f684faa9a5126a0db524d5d35be7277e52: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 1514e654c1a9b346cbfca5a81f6d331f106242aa: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 6aa83ec3d1d82ee332cedcf4e99d3cbd56e73edd: missingSpaceBeforeDate: invalid author/committer line - missing space before date
Checking objects: 100% (18523/18523), done.

What actually is going wrong here... When looking at one of those tags (luckily, only tags are affected), we see that something's not ok with the date string of the tag (I am not a Git plumber, so sorry for being superficial here):

mike@sid:~/xorg/libX11.git$ git show 2fb7b16b38a365c5009bd170d4463634c1f3de26
tag XORG-RELEASE-1-TM-MERGE
Tagger: Alan Coopersmith 
Date:   Thu Jan 1 00:00:00 1970 +0000
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
commit 84adc60cae8e944777563501dd633622a28bdb3b
Author: Alan Coopersmith 
Date:   Sat Mar 27 07:08:16 2004 +0000
    Fix typo (IsModiferKey should be IsModifierKey)
[... not quoting the diff for the above commit ...]

How to work around that issue... We then got the idea that we could redo those tags with the date string derived from the commit that this tag points at. This can be scripted (credits go to an unnamed person in Ulrich's surrounding!!!):

mike@sid:~/xorg/libX11$ cat repair.sh
#!/bin/bash
set -x
git fsck 2>&1   tee git-fsck.log   grep "error in tag"   sed -r -e 's/error in tag ([0-9a-f]+):.*/\1/p'   while read broken; do
         echo $broken
         commit=$(git rev-parse $ broken ^ commit )
         tag=$(git cat-file tag $broken   sed -n 's/^tag //p')
         tag_msg=$(git cat-file tag $broken   sed -n '/^$/,$p'   tail -n +2)
         export GIT_COMMITTER_NAME="$(git log -1 --format='%cn' $commit)"
         export GIT_COMMITTER_EMAIL="$(git log -1 --format='%ce' $commit)"
         export GIT_COMMITTER_DATE="$(git log -1 --format='%cD' $commit)"
         git tag -a -f -m "$tag_msg" $tag $commit
done
# drop all old content...
git reflog expire --expire=all --all
git gc --prune=all
# no more errors should be reported...
git fsck

Copy this repair.sh into the Git repo itself and let the script walk over your tags. After this script has been applied you can push the X.org Git repo with broken tags redone to Git hosters like Github or GitLab. light+love,
Mike [1] https://github.com/ArcticaProject/nx-libs
[2] https://lists.freedesktop.org/archives/mesa-dev/2016-February/106268.html

Colin Watson: Re-signing PPAs

Julian has written about their efforts to strengthen security in APT, and shortly before that notified us that Launchpad s signatures on PPAs use weak SHA-1 digests. Unfortunately we hadn t noticed that before; GnuPG s defaults tend to result in weak digests unless carefully tweaked, which is a shame. I started on the necessary fixes for this immediately we heard of the problem, but it s taken a little while to get everything in place, and I thought I d explain why since some of the problems uncovered are interesting in their own right. Firstly, there was the relatively trivial matter of using SHA-512 digests on new signatures. This was mostly a matter of adjusting our configuration, although writing the test was a bit tricky since PyGPGME isn t as helpful as it could be. (Simpler repository implementations that call gpg from the command line should probably just add the --digest-algo SHA512 option instead of imitating this.) After getting that in place, any change to a suite in a PPA will result in it being re-signed with SHA-512, which is good as far as it goes, but we also want to re-sign PPAs that haven t been modified. Launchpad hosts more than 50000 active PPAs, though, a significant percentage of which include packages for sufficiently recent Ubuntu releases that we d want to re-sign them for this. We can t expect everyone to push new uploads, and we need to run this through at least some part of our usual publication machinery rather than just writing a hacky shell script to do the job (which would have no idea which keys to sign with, to start with); but forcing full reprocessing of all those PPAs would take a prohibitively long time, and at the moment we need to interrupt normal PPA publication to do this kind of work. I therefore had to spend some quality time working out how to make things go fast enough. The first couple of changes (1, 2) were to add options to our publisher script to let us run just the one step we need in careful mode: that is, forcibly re-run the Release file processing step even if it thinks nothing has changed, and entirely disable the other steps such as generating Packages and Sources files. Then last week I finally got around to timing things on one of our staging systems so that we could estimate how long a full run would take. It was taking a little over two seconds per archive, which meant that if we were to re-sign all published PPAs then that would take more than 33 hours! Obviously this wasn t viable; even just re-signing xenial would be prohibitively slow. The next question was where all that time was going. I thought perhaps that the actual signing might be slow for some reason, but it was taking about half a second per archive: not great, but not enough to account for most of the slowness. The main part of the delay was in fact when we committed the database transaction after processing each archive, but not in the actual PostgreSQL commit, rather in the ORM invalidate method called to prepare for a commit. Launchpad uses the excellent Storm for all of its database interactions. One property of this ORM (and possibly of others; I ll cheerfully admit to not having spent much time with other ORMs) is that it uses a WeakValueDictionary to keep track of the objects it s populated with database results. Before it commits a transaction, it iterates over all those alive objects to note that if they re used in future then information needs to be reloaded from the database first. Usually this is a very good thing: it saves us from having to think too hard about data consistency at the application layer. But in this case, one of the things we did at the start of the publisher script was:

def getPPAs(self, distribution):
    """Find private package archives for the selected distribution."""
    if (self.isCareful(self.options.careful_publishing) or
            self.options.include_non_pending):
        return distribution.getAllPPAs()
    else:
        return distribution.getPendingPublicationPPAs()
def getTargetArchives(self, distribution):
    """Find the archive(s) selected by the script's options."""
    if self.options.partner:
        return [distribution.getArchiveByComponent('partner')]
    elif self.options.ppa:
        return filter(is_ppa_public, self.getPPAs(distribution))
    elif self.options.private_ppa:
        return filter(is_ppa_private, self.getPPAs(distribution))
    elif self.options.copy_archive:
        return self.getCopyArchives(distribution)
    else:
        return [distribution.main_archive]

That innocuous-looking filter means that we do all the public/private filtering of PPAs up-front and return a list of all the PPAs we intend to operate on. This means that all those objects are alive as far as Storm is concerned and need to be considered for invalidation on every commit, and the time required for that stacks up when many thousands of objects are involved: this is essentially accidentally quadratic behaviour, because all archives are considered when committing changes to each archive in turn. Normally this isn t too bad because only a few hundred PPAs need to be processed in any given run; but if we re running in a mode where we re processing all PPAs rather than just ones that are pending publication, then suddenly this balloons to the point where it takes a couple of seconds. The fix is very simple, using an iterator instead so that we don t need to keep all the objects alive:

from itertools import ifilter
def getTargetArchives(self, distribution):
    """Find the archive(s) selected by the script's options."""
    if self.options.partner:
        return [distribution.getArchiveByComponent('partner')]
    elif self.options.ppa:
        return ifilter(is_ppa_public, self.getPPAs(distribution))
    elif self.options.private_ppa:
        return ifilter(is_ppa_private, self.getPPAs(distribution))
    elif self.options.copy_archive:
        return self.getCopyArchives(distribution)
    else:
        return [distribution.main_archive]

After that, I turned to that half a second for signing. A good chunk of that was accounted for by the signContent method taking a fingerprint rather than a key, despite the fact that we normally already had the key in hand; this caused us to have to ask GPGME to reload the key, which requires two subprocess calls. Converting this to take a key rather than a fingerprint gets the per-archive time down to about a quarter of a second on our staging system, about eight times faster than where we started. Using this, we ve now re-signed all xenial Release files in PPAs using SHA-512 digests. On production, this took about 80 minutes to iterate over around 70000 archives, of which 1761 were modified. Most of the time appears to have been spent skipping over unmodified archives; even a few hundredths of a second per archive adds up quickly there. The remaining time comes out to around 0.4 seconds per modified archive. There s certainly still room for speeding this up a bit. We wouldn t want to do this procedure every day, but it s acceptable for occasional tasks like this. I expect that we ll similarly re-sign wily, vivid, and trusty Release files soon in the same way.

NOKUBI Takatsugu: GAME-ON the historical videogame exhibition

I attended GAME-ON last weedkend at Miraikan, The National Museum of Emerging Science and Innovation in Tokyo. There are many historical videogame hardware and software, and the bland-new PSVR exibition.

I can see the real DEC PDP-1 hardware. Unfortunately it is only haredware, not worked. And also PONG is exibited.

SPACEWAR description

There are many old hardware Atari 2600, Commodore 64, Spectrum, and so on.

Almost exhibition with English description, and officials can talk English, if you feel interested, go Miraikan until 30th May.

28 March 2016

Rhonda D'Vine: Ich bin was ich bin

As my readers probably are well aware, I wrote my transgender coming out poem Mermaids over 10 years ago, to make it clear to people how I define, what I am and how I would hope they could accept me. I did put it publicly into my blog so I could point people to it. And I still do so regularly. It still comes from the bottom of my heart. And I am very happy that I got the chance to present it in a Poetry Slam last year, it was even recorded and uploaded to YouTube. There is just one thing that I was also told over the time every now and then by some people that I would have liked to understand what's going on: Why is it in English, my English isn't that good. My usual response was along the lines of that the events that triggered me writing it were in an international context and I wanted to make sure that they understood what I wrote. At that time I didn't realize that I am cutting out a different group of people from being able to understand what's going on inside me. So this year there was a similar event: the Flawless Poetry Slam which touched the topics of Feminist? Queer? Gender? Rolemodels? - Let's talk about it. I took that as motivation to finally write another text on the topic, and this time in German. Unfortunately though I wasn't able to present it that evening, I wasn't drawn for the lineup. But, I was told that there was another slam going on just last wednesday, so I went there ... and made it onto the stage! And this is the text that I presented there. I am uncertain how well online translators work for you, but I hope you get the core points if you don't understand German:

Ich bin was ich bin
F nf Worte mit wahrem Sinn:
Ich bin was ich bin Du denkst: "Mann im Rock?
Das ist ja wohl l cherlich,
der ist sicher schwul." "Fingernagellack?
Na da schau ich nicht mehr hin,
wer will das schon seh'n." Jedoch liegst du falsch,
Mit all deinen Punkten, denn:
Ich bin was ich bin. Ich bin Transgender
Und erlebe mich selber,
ich bin eine Frau. "Haha, eine Frau?
Wem willst du das weismachen?
Heb mal den Rock hoch!" Und wie ist's bei dir?
Was ist zwischen den Beinen?
Geht mich das nichts an? Warum fragst du mich?
Da ist's dann in Ordnung?
Oder vielleicht nicht? Ich bin was ich bin
F nf Worte mit ernstem Sinn:
Ich bin was ich bin Ich steh weiblich hier
Und das hier ist mein K rper
Mein Geschlecht ist's auch Oberfl chlichkeit
Das ist mein gr tes Problem
Schl gt mir entgegen Wenn ich mich ffne
Verst ndnis fast berall
Es wird akzeptiert Doch gelegentlich
und das schmerzt mich am meisten
sagt doch mal wer "er" Von Fremden? Egal
Doch hab ich mich ge ffnet
Ist es eine Qual "Ich seh dich als Mann"
Da ist, was es transportiert
Akzeptanz? Dahin Meine Pronomen
Wenn ihr ber mich redet
sind sie, ihr, ihres Ich leb was ich leb
F nf Worte mit tiefem Sinn:
Ich bin was ich bin "Doch, wie der erst spricht!
Ich meinte, wie sie denn spricht!
Das ist nicht normal." Ich schreib hier Haikus:
Japanische Gedichtsform
Mit fixem Versmars Sind f nf, sieben, f nf
Silben in jeder Zeile
Haikus sind simpel Probier es mal aus
Transportier eine Message
Es macht auch viel Spa Wortwahl ist wichtig
Ein guter Thesaurus hilft
Sei kurz und pr gnant Ich sag was ich sag
F nf Worte mit klugem Sinn:
Ich bin was ich bin Doch ich schweife ab
Verst ndnis fast berall?
Wird es akzeptiert? Erstaunlicherweise
Doch ich bin auch was and'res
Und hier geht's bergab Eine Sache gibt's
Die erw h'n ich besser nicht
f r die steck ich ein "Deshalb bin ich hier"
So der Titel eines Lieds
verfasst von Thomas D "Wenn ich erkl re
warum ich mich wie ern hr"
So weit komm ich nicht Man erw hnt Vegan
Die Intoleranz ist da
Man ist unten durch "Mangelerscheinung!"
"Das Essen meines Essens!"
Akzeptanz ade Hab 'ne Theorie:
Vegan sein: 'ne Entscheidung
Transgender sein nicht Mensch f hlt sich dann schlecht
dass bei sich selbst die Kraft fehlt
und greift damit an "Ich k nnte das nicht"
Ich verurteile dich nicht
Iss doch was du willst Ich zwing es nicht auf
Aber R cksicht w r schon fein
Statt nur Hohn und Schm h Ich ess was ich ess
F nf Worte zum nachdenken:
Ich bin was ich bin

Hope you get the idea. The audience definitely liked it, the jury wasn't so much on board but that's fine, it's five random people and it's mostly for fun anyway. Later that night though some things happened that didn't make me feel so comfortable anymore. I went to the loo, waiting in line with the other ladies, a bit later the waitress came along telling me "the men's room is over there". I told her that I'm aware of that and thanked her, which got her confused and said something along the lines of "so you are both, or what?" but went away after that. Her tone and response wasn't really giving me much comfort, though none of the other ladies in the line did look strangely.
But the most disturbing event after that was to find out about North Carolina signed the bathroom bill making it illegal for trans people to use the bathroom for their gender and insisting on using the one for the gender they were assigned at birth. So men like James Sheffield are now forced to go to the lady's restroom, or face getting arrested. Brave new world. :/ So, enjoy the text and don't get too wound up by stupid laws and hope for time to fix people's discriminatory minds for fixing issues that already are regulated: Assaults are assaults and are already banned. Arguing with people might get assaulted and thus discriminating trans people is totally missing the point, by miles.

/personal permanent link Comments: 2 Flattr this

Joey Hess: type safe multi-OS Propellor

Propellor was recently ported to FreeBSD, by Evan Cofsky. This new feature led me down a two week long rabbit hole to make it type safe. In particular, Propellor needed to be taught that some properties work on Debian, others on FreeBSD, and others on both. The user shouldn't need to worry about making a mistake like this; the type checker should tell them they're asking for something that can't fly.

-- Is this a Debian or a FreeBSD host? I can't remember, let's use both package managers!
host "example.com" $ props
    & aptUpgraded
    & pkgUpgraded

As of propellor 3.0.0 (in git now; to be released soon), the type checker will catch such mistakes. Also, it's really easy to combine two OS-specific properties into a property that supports both OS's:

upgraded = aptUpgraded  pickOS  pkgUpgraded

type level lists and functions The magick making this work is type-level lists. A property has a metatypes list as part of its type. (So called because it's additional types describing the type, and I couldn't find a better name.) This list can contain one or more OS's targeted by the property:

aptUpgraded :: Property (MetaTypes '[ 'Targeting 'OSDebian, 'Targeting 'OSBuntish ])
pkgUpgraded :: Property (MetaTypes '[ 'Targeting 'OSFreeBSD ])

In Haskell type-level lists and other DataKinds are indicated by the ' if you have not seen that before. There are some convenience aliases and type operators, which let the same types be expressed more cleanly:

aptUpgraded :: Property (Debian + Buntish)
pkgUpgraded :: Property FreeBSD

Whenever two properties are combined, their metatypes are combined using a type-level function. Combining aptUpgraded and pkgUpgraded will yield a metatypes that targets no OS's, since they have none in common. So will fail to type check. My implementation of the metatypes lists is hundreds of lines of code, consisting entirely of types and type families. It includes a basic implementation of singletons, and is portable back to ghc 7.6 to support Debian stable. While it takes some contortions to support such an old version of ghc, it's pretty awesome that the ghc in Debian stable supports this stuff. extending beyond targeted OS's Before this change, Propellor's Property type had already been slightly refined, tagging them with HasInfo or NoInfo, as described in making propellor safer with GADTs and type families. I needed to keep that HasInfo in the type of properties. But, it seemed unnecessary verbose to have types like Property NoInfo Debian. Especially if I want to add even more information to Property types later. Property NoInfo Debian NoPortsOpen would be a real mouthful to need to write for every property. Luckily I now have this handy type-level list. So, I can shove more types into it, so Property (HasInfo + Debian) is used where necessary, and Property Debian can be used everywhere else. Since I can add more types to the type-level list, without affecting other properties, I expect to be able to implement type-level port conflict detection next. Should be fairly easy to do without changing the API except for properties that use ports. singletons As shown here, pickOS makes a property that decides which of two properties to use based on the host's OS.

aptUpgraded :: Property DebianLike
aptUpgraded = property "apt upgraded" (apt "upgrade"  requires  apt "update")
pkgUpgraded :: Property FreeBSD
pkgUpgraded = property "pkg upgraded" (pkg "upgrade")
    
upgraded :: Property UnixLike
upgraded = (aptUpgraded  pickOS  pkgUpgraded)
     describe  "OS upgraded"

Any number of OS's can be chained this way, to build a property that is super-portable out of simple little non-portable properties. This is a sweet combinator! Singletons are types that are inhabited by a single value. This lets the value be inferred from the type, which came in handy in building the pickOS property combinator. Its implementation needs to be able to look at each of the properties at runtime, to compare the OS's they target with the actial OS of the host. That's done by stashing a target list value inside a property. The target list value is inferred from the type of the property, thanks to singletons, and so does not need to be passed in to property. That saves keyboard time and avoids mistakes. is it worth it? It's important to consider whether more complicated types are a net benefit. Of course, opinions vary widely on that question in general! But let's consider it in light of my main goals for Propellor:

Help save the user from pushing a broken configuration to their machines at a time when they're down in the trenches dealing with some urgent problem at 3 am.
Advance the state of the art in configuration management by taking advantage of the state of the art in strongly typed haskell.

This change definitely meets both criteria. But there is a tradeoff; it got a little bit harder to write new propellor properties. Not only do new properties need to have their type set to target appropriate systems, but the more polymorphic code is, the more likely the type checker can't figure out all the types without some help. A simple example of this problem is as follows.

foo :: Property UnixLike
foo = p  requires  bar
  where
    p = property "foo" $ do
        ...

The type checker will complain that "The type variable metatypes1 is ambiguous". Problem is that it can't infer the type of p because many different types could be combined with the bar property and all would yield a Property UnixLike. The solution is simply to add a type signature like p :: Property UnixLike Since this only affects creating new properties, and not combining existing properties (which have known types), it seems like a reasonable tradeoff. things to improve later There are a few warts that I'm willing to live with for now... Currently, Property (HasInfo + Debian) is different than

Property (Debian +
HasInfo)

, but they should really be considered to be the same type. That is, I need type-level sets, not lists. While there's a type level sets library for hackage, it still seems to require a specific order of the set items when writing down a type signature. Also, using ensureProperty, which runs one property inside the action of another property, got complicated by the need to pass it a type witness.

foo = Property Debian
foo = property' $ \witness -> do
    ensureProperty witness (aptInstall "foo")

That witness is used to type check that the inner property targets every OS that the outer property targets. I think it might be possible to store the witness in the monad, and have ensureProperty read it, but it might complicate the type of the monad too much, since it would have to be parameterized on the type of the witness. Oh no, I mentioned monads. While type level lists and type functions and generally bending the type checker to my will is all well and good, I know most readers stop reading at "monad". So, I'll stop writing. ;) thanks Thanks to David Miani who answered my first tentative question with a big hunk of example code that got me on the right track. Also to many other people who answered increasingly esoteric Haskell type system questions. Also thanks to the Shuttleworth foundation, which funded this work by way of a Flash Grant.

27 March 2016

Lunar: Reproducible builds: week 48 in Stretch cycle

What happened in the reproducible builds effort between March 20th and March 26th:

Toolchain fixes

Sebastian Ramacher uploaded breathe/4.2.0-1 which makes its output deterministic. Original patch by Chris Lamb, merged uptream.

Rafael Laboissiere uploaded octave/4.0.1-1 which allows packages to be built in place and avoid unreproducible builds due to temporary build directories appearing in the `.oct` files.

Daniel Kahn Gillmor worked on removing build path from build symbols submitting a patch adding `-fdebug-prefix-map` to clang to match GCC, another patch against gcc-5 to backport the removal of `-fdebug-prefix-map` from `DW_AT_producer`, and finally by proposing the addition of a `normalizedebugpath` to the `reproducible` feature set of `dpkg-buildflags` that would use `-fdebug-prefix-map` to replace the current directory with `.` using `-fdebug-prefix-map`. As succesful result of lobbying at LibrePlanet 2016, the `--clamp-mtime` option will be featured in the next Tar release. This option is likely to be used by `dpkg-deb` to implement deterministic mtimes for packaged files.

Packages fixed The following packages have become reproducible due to changes in their build dependencies: augeas, gmtkbabel, ktikz, octave-control, octave-general, octave-image, octave-ltfat, octave-miscellaneous, octave-mpi, octave-nurbs, octave-octcdf, octave-sockets, octave-strings, openlayers, python-structlog, signond. The following packages became reproducible after getting fixed:

atheme-services/7.0.7-1 by Antoine Beaupr .

boa-constructor/0.6.1-16 by Reiner Herrmann.

calculix-ccx/2.10-1 by Wolfgang F tterer, fixed upstream.

deap/1.0.2.post2-2 by Daniel Stender.

debian-keyring/2016.03.22 uploaded by Jonathan McDowell, fix by Niels Thykier, obsolete patch by Satyam Zode.

firefox-kwallet5/1.0-2 by Sandro Knau .

fox1.6/1.6.51-1 by Joachim Wiedorn.

gdnsd/2.2.3-1 by Faidon Liambotis.

glib2.0/2.48.0-1 uploaded by Iain Lane, original patch by Lunar, merged upstream.

jwm/2.3.4-1 uploaded by Samuel Henrique Oltramari Pinto, fix by Reiner Herrmann.

libasm4-java/5.0.4-2 by Emmanuel Bourg.

libjna-java/4.2.2-1 by Emmanuel Bourg.

libsynthesis/3.4.0.47.5+syncevolution-1.5.1-1 uploaded by Tino Mettler, patch by Reiner Herrmann.

logback/1:1.1.6-1 by Emmanuel Bourg.

nethack/3.6.0-2 uploaded by James Cowgill, patch by Reiner Herrmann.

php-htmlpurifier/4.7.0-2 by David Pr vot.

psocksxx/1.1.0-1 by Gianfranco Costamagna.

sbmltoolbox/4.1.0-3 uploaded by Afif Elghraoui, original patch by Reiner Herrmann.

spades/3.7.1+dfsg-2 by Sascha Steinbiss.

sprng/2.0a-10 uploaded by Dirk Eddelbuettel, original patch by Reiner Herrmann.

Some uploads fixed some reproducibility issues, but not all of them:

gle-graphics/4.2.5-2 by Christian T. Steigies.

kexec-tools/1:2.0.10-2 uploaded by Khalid Aziz, original patch by Lunar.

pdal/1.1.0-4 by Bas Couwenberg.

tcl8.5/8.5.19-2 uploaded by Sergei Golovan, original patch.

tcl8.6/8.6.5+dfsg-2 uploaded by Sergei Golovan, original patch.

Patches submitted which have not made their way to the archive yet:

#818742 on milkytracker by Reiner Herrmann: sorts the list of source files.

#818752 on tcl8.4 by Reiner Herrmann: sort source files using C locale.

#818753 on tk8.6 by Reiner Herrmann: sort source files using C locale.

#818754 on tk8.5 by Reiner Herrmann: sort source files using C locale.

#818755 on tk8.4 by Reiner Herrmann: sort source files using C locale.

#818952 on marionnet by ceridwen: dummy out build date and uname to make build reproducible.

#819334 on avahi by Reiner Herrmann: ship upstream changelog instead of the one generated by gettextize (although duplicate of #804141 by Santiago Vila).

tests.reproducible-builds.org `i386` build nodes have been setup by converting 2 of the 4 `amd64` nodes to `i386`. (h01ger)

Package reviews 92 reviews have been removed, 66 added and 31 updated in the previous week. New issues: timestamps_generated_by_xbean_spring, timestamps_generated_by_mangosdk_spiprocessor. Chris Lamb filed 7 FTBFS bugs.

Misc. On March 20th, Chris Lamb gave a talk at FOSSASIA 2016 in Singapore. The very same day, but a few timezones apart, h01ger did a presentation at LibrePlanet 2016 in Cambridge, Massachusetts. Seven GSoC/Outreachy applications were made by potential interns to work on various aspects of the reproducible builds effort. On top of interacting with several applicants, prospective mentors gathered to review the applications. Huge thanks to Linda Naeun Lee for the new hackergotchi visible on Planet Debian.

Chris Lamb: Generating dynamic Python tests using metaclasses

One common testing anti-pattern is a single testcase that loops over multiple yet independent inputs:

from django.test import TestCase
class MyTests(TestCase):
    def test_all(self):
        for x in (
            'foo',
            'bar',
            'baz',
            # etc.
        ):
            self.assertEqual(test(x), None)

Whilst this code style typically occurs in an honourable attempt to avoid a DRY violation, these tests:

Can only report the first failure
Prevent individual inputs from being tested independently
Are typically slower than their neighbours, or a performance hotspot generally
Do not allow for parallel computation, a feature recently added to Django 1.9

(Note that whilst foo, bar, etc. are defined statically above for simplicity, the values could be determined dynamically by, for example, iterating over the filesystem.)

If you have such tests, consider splitting them out using a metaclass like so:

class MyTestsMeta(type):
    def __new__(cls, name, bases, attrs):
        for x in (
            'foo',
            'bar',
            'baz',
        ):
            attrs['test_%s' % x] = cls.gen(x)
        return super(MyTestsMeta, cls).__new__(cls, name, bases, attrs)
    @classmethod
    def gen(cls, x):
        # Return a testcase that tests  x .
        def fn(self):
            self.assertEqual(test(x), None)
        return fn
class MyTests(TestCase):
    __metaclass__ = MyTestsMeta

This has the effect of replacing the single testcase with individual test_foo, test_bar & test_baz testcases. Each test can then be run separately:

$ ./manage.py test myproject.myapp.tests.MyTests.test_baz
Creating test database for alias 'default'...
.
----------------------------------------------------------------------
Ran 1 test in 0.039s
OK
Destroying test database for alias 'default'...

... or we can test them all in parallel:

$ ./manage.py test myproject.myapp.tests.MyTests --parallel
Creating test database for alias 'default'...
.
----------------------------------------------------------------------
Ran 3 tests in 0.065s
OK
Destroying test database for alias 'default'...

You must ensure that the tests have unique names to avoid cases masking each other. In the above example we could simply use the input string itself, but if you have no obvious candidate you could try using Python's enumerate method to generate a unique, if somewhat opaque, suffix:

for idx, x in enumerate((
    'foo',
    'bar',
    'baz',
)):
    attrs['test_%d' % idx] = cls.gen(x)

One alternative approach to a metaclass is to generate the test methods and use setattr, to bind them to the TestCase class. However, using a metaclass:

Is cleaner and/or more "Pythonic"
Avoids a number of subtle pitfalls with parameter binding
Prevents your TestCase class from being polluted with loop variables, etc.
Can be composed or abstracted into reusable testing components

Note that you can still use setUp and all the other unittest.TestCase and django.test.TestCase methods as before:

from .utils import MyBaseTestCase # custom superclass
class MyTestsMeta(type):
    # <snip>
    @classmethod
    def gen(cls, x):
        def fn(self):
            self.assertRedirects(...)
        return fn
class MyTests(MyBaseTestCase):
    __metaclass__ = MyTestsMeta
    def setUp(self):
        super(MyTests, self).setUp()
        # Code here is run before every test
    def test_other(self):
        # Test some other functionality here

UPDATE: Stu Cox asked: Do you even need the metaclass, or could you do this with __new__ straight on the TestCase? Curiously, unittest does not initialise classes in the typical way if you do not explicitly define at least one test_ method.

26 March 2016

Gunnar Wolf: Yes! I can confirm that...

I am very very (very very very!) happy to confirm that... This year, and after many years of not being able to, I will cross the Atlantic. To do this, I will take my favorite excuse: Attending DebConf! So, yes, this image I am pasting here is as far as you can imagine from official promotional material. But, having bought my plane tickets, I have to start bragging about it ;-) In case it is of use to others (at least, to people from my general geographic roundabouts), I searched for plane tickets straight from Mexico. I was accepting my lack of luck, facing an over-36-hour trip(!!) and at very high prices. Most routes were Mexico- central_europe -Arab Emirates-South Africa... Great for collecting frequent-flier miles, but terrible for anything else. Of course, requesting a more logical route (say, via Sao Paulo in Brazil) resulted in a price hike to over US$3500. Not good. I found out that Mexico-Argentina tickets for that season were quite agreeable at US$800, so I booked our family vacation to visit the relatives, and will fly from there at US$1400. So, yes, in a 48-hr timespan I will do MEX-GRU-ROS, then (by land) Rosario to Buenos Aires, then AEP-GRU-JNB-CPT. But while I am at DebConf, Regina and the kids will be at home with the grandparents and family and friends. In the end, win-win with just an extra bit of jetlag for me ;-) I *really* expect flights to be saner for USians, Europeans, and those coming from further far away. But we have grown to have many Latin Americans, and I hope we can all meet in CPT for the most intense weeks of the year! See you all in South Africa!

25 March 2016

Guido G nther: More sandboxing

More sandboxing When working on untrusted code or data it's impossible to predict what happens when one does a:

bundle install --path=vendor

npm install

Does this phone out your private SSH and GPG keys? Does a

evince Downloads/justdownloaded.pdf

try to exploit the PDF viewer? While you can run stuff in separate virtual machines this can get cumbersome. libvirt-sandbox to the rescue! It allows to sandbox applications using libvirt's virtualization drivers. It took us a couple of years (The ITP is from 2012) but we finally have it in Debian's NEW queue. When libvirt-sandbox creates a sandbox it uses your root filesystem mounted read only by default so you have access to all installed programs (this can be changed with the --root option though). It can use either libvirt's QEMU or LXC drivers. We're using the later in the examples below: So in order to make sure the above bundler call has no access to your $HOME you can use:

sudo virt-sandbox \
   -m ram:/tmp=10M \
   -m ram:$HOME=10M \
   -m ram:/var/run/screen=1M \
   -m host-bind:/path/to/your/ruby-stuff=/path/to/your/ruby-stuff \
   -c lxc:/// \
   -S $USER \
   -n rubydev-sandbox \
   -N dhcp,source=default \
   /bin/bash

This will make your $HOME unaccessible by mounting a tmpfs over it and using separate network, ipc, mount, pid and utc namespaces allowing you to invoke bundler with less worries. /path/to/your/ruby-stuff is bind mounted read-write into the sandbox so you can change files there. Bundler can fetch new gems using libvirt's default network connection. And for the PDF case:

sudo virt-sandbox \
  -m ram:$HOME=10M \
  -m ram:/dev/shm=10M \
  -m host-bind:$HOME/Downloads=$HOME/Downloads \
  -c lxc:/// \
  -S $USER \
  -n evince-sandbox \
  --env="DISPLAY=:0" \
  --env="XAUTHORITY=$XAUTHORITY" \
  /usr/bin/evince Downloads/justdownloaded.pdf

Note that the above example shares /tmp with the sandbox in order to give it access to the X11 socket. A better isolation can probably be achieved using xpra or xvnc but I haven't looked into this yet. Besides the command line program virt-sandbox there's also the library libvirt-sandbox which makes it simpler to build new sandboxing applications. We're not yet shipping virt-sandbox-service (a tool to provision sandboxed system services) in the Debian packages since it's RPM distro specific. Help on porting this to Debian is greatly appreciated.

Next.

Previous.