Search Results: "xam"

6 July 2013

Sylvestre Ledru: Automatic scan-build on the LLVM toolchain

After setting an automatic code coverage tool, I just plugged an automatic scan-build on the LLVM toolchain trunk: http://buildd-clang.debian.net/scan-build/ Note that: This work is done through the Debian/LLVM jenkins instance.
The report is updated twice a day.
The scan-build/clang used to produce the report is the one published on http://llvm.org/apt/. That means that a new feature/fix done on scan-build will appear only about a day after on report. As a side effect, it tests automatically the packages distributed on llvm.org/apt/. For a project of this size, the number of reported errors / warnings is pretty low. At time of writing, only 329 errors are reported and many of them are in generated code (AttrDump.inc and LLDBWrapPython.cpp for example) or in unit tests. [1]
/tmp/buildd/llvm-toolchain-snapshot-3.4~svn185728/build-llvm/Release/bin/clang -fno-exceptions -fPIC -funwind-tables -I/tmp/buildd/llvm-toolchain-snapshot-3.4~svn185728/projects/compiler-rt/lib -I/tmp/buildd/llvm-toolchain-snapshot-3.4~svn185728/projects/compiler-rt/include -Wall -Werror -O3 -fomit-frame-pointer -m64 -fPIE -fno-builtin -gline-tables-only -fno-rtti -DASAN_FLEXIBLE_MAPPING_AND_OFFSET=1 -c -o /tmp/buildd/llvm-toolchain-snapshot-3.4~svn185728/build-llvm/tools/clang/runtime/compiler-rt/clang_linux/asan-x86_64/x86_64/SubDir.lib__sanitizer_common/sanitizer_symbolizer_itanium.o /tmp/buildd/llvm-toolchain-snapshot-3.4~svn185728/projects/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_itanium.cc
Original post blogged on b2evolution.

4 July 2013

Emanuele Rocca: Useful tools for Python developers

Python is a great language with an impressive number of tools designed to make developers' life easier. Sometimes, however, the problem is getting to know that these tools exist in the first place. By contributing to projects like OpenStack's Nova client and Falcon, I have recently come across some useful tools that can seriously improve the quality of your code. The first one is called pyflakes, a passive checker of Python programs developed by Phil Frost. What it does is parsing your source files and checking for possible errors such as undefined names and unused imports. Let's consider the following example: <figure class="code">
1
2
3
4

import urllib

print "pyflakes example"
urlib.urlopen('http://www.linux.it')
</figure>The code above contains a typo, we have misspelled urllib. Here is what pyflakes thinks about our program: 
$ pyflakes example.py
example.py:1: 'urllib' imported but unused
example.py:4: undefined name 'urlib'
On line 4 we try to use urlib which is not defined. Also, we import urllib on line 1 and we do nothing with it. Our typo has been spotted! Notice that, even though our program contains a print statement, 'pyflakes example' has not been printed. That is because pyflakes parses the source files it checks, without importing them, making it safe to use on modules with side effects. pyflakes can be installed with pip or apt-get. The second tool I want to talk about is Ned Batchelder's coverage.py. No doubt you write unit tests for your programs. Right? Good. coverage.py is out there to help you checking how much of your program is actually covered. Let's use as an example codicefiscale, a Python project of mine. First we install coverage: 
pip install coverage
Then we run our unit tests: 
$ coverage run --source=codicefiscale tests.py
.......
----------------------------------------------------------------------
Ran 7 tests in 0.003s
We pass the module we want to test with --source=codicefiscale so that coverage will only report information about that specific module. Now that our tests have been performed successfully it is time to check how much of our code is covered by unit tests: 
$ coverage report -m
Name            Stmts   Miss  Cover   Missing
---------------------------------------------
codicefiscale      73      4    95%   61, 67, 95, 100
Not bad, 95% of our module is covered! Still, coverage let us know that 4 lines have not been touched by the unit tests. With this information, we can go write some meaningful test cases that will also cover the missing lines.

Daniel Pocock: My Linux server IPv6 deployment approach

I previously discussed the ease of deploying IPv6 for Linux servers. Whether it is Debian, Fedora or another distribution the IPv6 stack should "just work" these days. However, for maintaining a production network with minimum risk of interruption, there are a few extra things to be aware of during IPv6 deployment. Here I present a rough set of steps that can be followed, usually with minimal or no downtime, to deploy IPv6. The steps are in the order that they should be used in practice Plenty of other web sites already explain the theory of IPv6 networks or practical configuration detail (like how to set a static IPv6 address in Debian). This page ignores all of that detail and looks at the overall project strategy. Borrow the last decimal octet (or host bits) of IPv4 addresses As mentioned in my earlier blog:
  • Of the 128 bits in IPv6 addresses, most sites now use a fixed 64 bit netmask for all their subnets. This is the practice encouraged by RFC 4291.
  • Therefore, there are always more bits in the 64-bit "host" portion of the address than in the host portion of any 32-bit IPv4 address.
  • Consequently, it is possible to just borrow the host portion (typically the last octet for a class-C network) and use it as the host portion of an IPv6 address
IPv4 address Host portion of address IPv6 address
192.168.1.5/24 5 2001:1234:567::5
Gotcha: remember that the IPv4 addresses are usually written in decimal while IPv6 addresses, including the example above, are hex. This doesn't really matter in practice unless the addresses are being manipulated by code that is sensitive to the host bits. For all other intents and purposes, it just makes it easy to read the addresses and associate them with IPv4. A completely valid alternative is to use the full IPv4 address as the host portion of the IPv6 address. For example, if the IPv4 address = 192.168.1.5, the IPv6 address may be 2001:1234:567::192.168.1.5. This approach is slighly more verbose and is completely valid. Which approach you use is at your discretion. Create reverse mappings (PTR records) in DNS Just as IPv4 has the in-addr.arpa zone and telephone numbers have the e164.arpa zone (for ENUM), IPv6 addresses have a reverse mapping zone too: ip6.arpa The recommended way to proceed is to create reverse mappings for all hosts that have IPv4 reverse mappings. Some services may be using reverse lookups for authentication (e.g. MySQL will check this for any user who has a "Host" restriction defined). This is why it is recommended to create all the reverse mappings very early in the IPv6 program, before any hosts start making connections over IPv6. Create extra A records in DNS It's possible that some applications will be more stubborn about IPv6 adoption than others. For example, if you have a host called wolf.example.org and it hosts a mail server, web server, DNS server and XMPP server all using the name wolf then all those server processes will have to support IPv6 connectivity from the moment you add an AAAA record. A good practice is to create independent DNS A records (such as ns1.example.org and mail.example.org) for each application (not just CNAME records). Existing CNAME records can be converted to A records and converted back to CNAME records after the IPv6 deployment is 100% complete. Duplicate firewall entries Duplicate all IPv4 firewall entries to create an IPv6 firewall For many types of network, this is a trivial (although possibly tedious) task. There are some small gotchas:
  • There is no NAT. Any firewall entries for NAT need to be reviewed: for source NAT, it is necessary to use a mechanism like connection tracking to protect workstations that previously had asymmetrical access to the public Internet. For destination NAT, see the TPROXY feature in ip6tables
  • IPsec works slightly differently. Rules for AH and ESP packets can't simply be duplicated for IPv6, they need further tweaks or warnings will be generated.
  • IPv6 makes some ICMP packages mandatory. If IPv4 ICMP is completely blocked, you still need to enable the IPv6 ICMP (or at least the subset of ICMP packets that are mandatory).
For more comments about firewalling IPv6, please see my earlier blog on the subject Put the IPv6 addresses on the hosts Add the necessary records to the appropriate place, for example, the /etc/network/interfaces files on Debian/Ubuntu style systems. For hosts that have multiple addresses on a single interface, it is often desirable to ensure that outbound connections always appear to come from just one of the addresses. This can be achieved using the preferred_lft option when configuring the non-default addresses. For example, if configuring three addresses on eth0: 
ip addr add dev eth0 preferred_lft 0 2001:1234:567::10/64
ip addr add dev eth0  2001:1234:567::11/64
ip addr add dev eth0 preferred_lft 0 2001:1234:567::12/64
In the example above, all outgoing IPv6 connections will appear to come from 2001:1234:567::11/64 Important: this is only a very shallow discussion of the issues around source addresses. If it doesn't provide a valid solution for you, there are many dedicated articles about this specific topic, madduck's blog looks at it for firewalls and tunnels but the concepts are valid for any type of multi-homed host Review listening processes with netstat Now that hosts have IPv6 addresses, it is worthwhile looking through the output of 
netstat -nlp
to see which processes have bound to both IPv4 and IPv6 addresses and which processes remain stuck on IPv4 only. For processes that are only listening on IPv4 or only bound to a specific address (rather than 0.0.0.0), it is a good idea to investigate the way they are configured and find out if they will support IPv6. Review the monitoring infrastructure (Nagios, Ganglia, etc) Now is a good time to review the monitoring infrastructure. Make sure that tools like Nagios are testing the services using their dedicated DNS names and not the host names. For example, if the host wolf.example.org runs an LDAP server accessible as ldap1.example.org, make sure Nagios is configured to poll ldap1.example.org. Look at the check_v46 wrapper for Nagios and similar solutions to make sure that Nagios is testing both the IPv4 and IPv6 version of each service. Just as developers use unit testing as part of test-driven development workflows, administrators deploying IPv6 can use the monitoring framework to validate the progress of their IPv6 program. Check for ACLs Do any server processes use IP-based ACLs to control access? For example, an Apache web server may be using an IP ACL to restrict access to some pages. MySQL or Postgres databases may have IP ACLs defined for some users or databases. When hosts start using IPv6 to connect to services, their IPv6 source addresses won't pass the ACL and services may be inaccessible. Therefore, it is worthwhile checking for such ACLs early on. Prepare DNS servers first Preparing the DNS servers first is a good step. The DNS servers should be configured to accept queries over IPv6. Once the DNS servers are listening on IPv6 addresses, go ahead and do some of the following:
  • Insert IPv6 AAAA records for the name servers in their own zone files. As mentioned in the previous tip, the name server records should have their own independent A records already with names such as ns1.example.org. Consequently, no other application should notice when the AAAA record is created.
  • Create IPv6 glue records for any name servers that have IPv4 glue records
Configure secondary name servers to poll over IPv6 Configure the secondary nameservers to poll the primary nameserver using it's new IPv6 address. Update resolv.conf on all hosts Now that DNS servers are listening on IPv6, it is possible to put IPv6 nameserver addresses in /etc/resolv.conf on all the other hosts. Configure and test other low-level services Although services such as LDAP and MySQL may not be exposed to public Internet users, they are an essential foundation for many other applications to run. Make sure all these low level services are accessible over IPv6. Test them. After testing each of these services, it is OK to go ahead and create the AAAA records for the service. Configure and test other services Continue testing each service is accessible over it's IPv6 address and then add the necessary AAAA record. This includes all the high level services now, such as mail and web servers. Configure and test hostnames Finally, start adding AAAA records associated with hostnames to the DNS zone. Now it should be possible to ssh or ping6 the hostnames using IPv6. You may also want to add the IPv6 address of each host into it's own /etc/hosts file at this stage.

Julien Danjou: OpenStack meets Lisp: cl-openstack-client

A month ago, a mail hit the OpenStack mailing list entitled "The OpenStack Community Welcomes Developers in All Programming Languages". You may know that OpenStack is essentially built using Python, and therefore it is the reference language for the client libraries implementations. As a Lisp and OpenStack practitioner, I used this excuse to build a challenge for myself: let's prove this point by bringing Lisp into OpenStack!
Welcome cl-openstack-client, the OpenStack client library for Common Lisp! The project is hosted on the classic OpenStack infrastructure for third party project, StackForge. It provides the continuous integration system based on Jenkins and the Gerrit infrastructure used to review contributions. How the tests works OpenStack projects ran a fabulous contribution workflow, which I already talked about, based on tools like Gerrit and Jenkins. OpenStack Python projects are used to run tox, to build a virtual environment and run test inside. We don't have such thing in Common Lisp as far as I know, so I had to build it myself. Fortunately, using Quicklisp, the fabulous equivalent of Python's PyPI, it has been a breeze to set this up. cl-openstack-client just includes a basic shell script that does the following: I just run the test using SBCL, though adding more compiler on the table would be a really good plan in the future, and should be straightforward. You can admire a log from a successful test run done when I proposed a patch via Gerrit, to check what it looks like. Implementation status For the curious, here's an example of how it works: 
* (require 'cl-openstack-client)
* (use-package 'cl-keystone-client)
* (defvar k (make-instance 'connection-v2 :username "demo" :password "somepassword" :tenant-name "demo" :url "http://devstack:5000"))
 
K
 
* (authenticate k)
 
((:ISSUED--AT . "2013-07-04T05:59:55.454226")
 (:EXPIRES . "2013-07-05T05:59:55Z")
 (:ID
  . "wNFQwNzo1OTo1NS40NTQyMthisisaverylongtokenwNFQwNzo1OTo1NS40NTQyM")
 (:TENANT (:DESCRIPTION) (:ENABLED . T)
  (:ID . "1774fd545df4400380eb2b4f4985b3be") (:NAME . "demo")))
 
* (connection-token-id k)
 
"wNFQwNzo1OTo1NS40NTQyMthisisaverylongtokenwNFQwNzo1OTo1NS40NTQyM"

Unfortunately, the implementation is far from being complete. It only implements for now the Keystone token retrieval. I've actually started this project to build an already working starting point. With this, future potential contributors will be able to spend efforts on writing code, and not on setting up the basic continuous integration system or module infrastructure. If you wish to help me and contribute, just follow the OpenStack Gerrit workflow howto or feel free to come by me and ask any question (I'm hanging out on #lisp on Freenode too). See you soon, hopping to bring more Lisp into OpenStack!

Julien Danjou: OpenStack meets Lisp: cl-openstack-client

A month ago, a mail hit the OpenStack mailing list entitled "The OpenStack Community Welcomes Developers in All Programming Languages". You may know that OpenStack is essentially built using Python, and therefore it is the reference language for the client libraries implementations. As a Lisp and OpenStack practitioner, I used this excuse to build a challenge for myself: let's prove this point by bringing Lisp into OpenStack!
Welcome cl-openstack-client, the OpenStack client library for Common Lisp! The project is hosted on the classic OpenStack infrastructure for third party project, StackForge. It provides the continuous integration system based on Jenkins and the Gerrit infrastructure used to review contributions. How the tests works OpenStack projects ran a fabulous contribution workflow, which I already talked about, based on tools like Gerrit and Jenkins. OpenStack Python projects are used to run tox, to build a virtual environment and run test inside. We don't have such thing in Common Lisp as far as I know, so I had to build it myself. Fortunately, using Quicklisp, the fabulous equivalent of Python's PyPI, it has been a breeze to set this up. cl-openstack-client just includes a basic shell script that does the following: I just run the test using SBCL, though adding more compiler on the table would be a really good plan in the future, and should be straightforward. You can admire a log from a successful test run done when I proposed a patch via Gerrit, to check what it looks like. Implementation status For the curious, here's an example of how it works: 
* (require 'cl-openstack-client)
* (use-package 'cl-keystone-client)
* (defvar k (make-instance 'connection-v2 :username "demo" :password "somepassword" :tenant-name "demo" :url "http://devstack:5000"))
 
K
 
* (authenticate k)
 
((:ISSUED--AT . "2013-07-04T05:59:55.454226")
 (:EXPIRES . "2013-07-05T05:59:55Z")
 (:ID
  . "wNFQwNzo1OTo1NS40NTQyMthisisaverylongtokenwNFQwNzo1OTo1NS40NTQyM")
 (:TENANT (:DESCRIPTION) (:ENABLED . T)
  (:ID . "1774fd545df4400380eb2b4f4985b3be") (:NAME . "demo")))
 
* (connection-token-id k)
 
"wNFQwNzo1OTo1NS40NTQyMthisisaverylongtokenwNFQwNzo1OTo1NS40NTQyM"

Unfortunately, the implementation is far from being complete. It only implements for now the Keystone token retrieval. I've actually started this project to build an already working starting point. With this, future potential contributors will be able to spend efforts on writing code, and not on setting up the basic continuous integration system or module infrastructure. If you wish to help me and contribute, just follow the OpenStack Gerrit workflow howto or feel free to come by me and ask any question (I'm hanging out on #lisp on Freenode too). See you soon, hopping to bring more Lisp into OpenStack!

2 July 2013

Ond&#345;ej &#268;ert&iacute;k: My impressions from the SciPy 2013 conference

I have attended the SciPy 2013 conference in Austin, Texas. Here are my impressions.

Number one is the fact that the IPython notebook was used by pretty much everyone. I use it a lot myself, but I didn't realize how ubiquitous it has become. It is quickly becoming the standard now. The IPython notebook is using Markdown and in fact it is better than Rest. The way to remember the "[]()" syntax for links is that in regular text you put links into () parentheses, so you do the same in Markdown, and append [] for the text of the link. The other way to remember is that [] feel more serious and thus are used for the text of the link. I stressed several times to +Fernando Perez and +Brian Granger how awesome it would be to have interactive widgets in the notebook. Fortunately that was pretty much preaching to the choir, as that's one of the first things they plan to implement good foundations for and I just can't wait to use that.

It is now clear, that the IPython notebook is the way to store computations that I want to share with other people, or to use it as a "lab notebook" for myself, so that I can remember what exactly I did to obtain the results (for example how exactly I obtained some figures from raw data). In other words --- instead of having sets of scripts and manual bash commands that have to be executed in particular order to do what I want, just use IPython notebook and put everything in there.

Number two is that how big the conference has become since the last time I attended (couple years ago), yet it still has the friendly feeling. Unfortunately, I had to miss a lot of talks, due to scheduling conflicts (there were three parallel sessions), so I look forward to seeing them on video.

+Aaron Meurer and I have done the SymPy tutorial (see the link for videos and other tutorial materials). It's been nice to finally meet +Matthew Rocklin (very active SymPy contributor) in person. He also had an interesting presentation
about symbolic matrices + Lapack code generation. +Jason Moore presented PyDy.
It's been a great pleasure for us to invite +David Li (still a high school student) to attend the conference and give a presentation about his work on sympygamma.com and live.sympy.org.

It was nice to meet the Julia guys, +Jeff Bezanson and +Stefan Karpinski. I contributed the Fortran benchmarks on the Julia's website some time ago, but I had the feeling that a lot of them are quite artificial and not very meaningful. I think Jeff and Stefan confirmed my feeling. Julia seems to have quite interesting type system and multiple dispatch, that SymPy should learn from.

I met the VTK guys +Matthew McCormick and +Pat Marion. One of the keynotes was given by +Will Schroeder from Kitware about publishing. I remember him stressing to manage dependencies well as well as to use BSD like license (as opposed to viral licenses like GPL or LGPL). That opensource has pretty much won (i.e. it is now clear that that is the way to go).

I had great discussions with +Francesc Alted, +Andy Terrel, +Brett Murphy, +Jonathan Rocher, +Eric Jones, +Travis Oliphant, +Mark Wiebe, +Ilan Schnell, +St fan van der Walt, +David Cournapeau, +Anthony Scopatz, +Paul Ivanov, +Michael Droettboom, +Wes McKinney, +Jake Vanderplas, +Kurt Smith, +Aron Ahmadia, +Kyle Mandli, +Benjamin Root and others.


It's also been nice to have a chat with +Jason Vertrees and other guys from Schr dinger.

One other thing that I realized last week at the conference is that pretty much everyone agreed on the fact that NumPy should act as the default way to represent memory (no matter if the array was created in Fortran or other code) and allow manipulations on it. Faster libraries like Blaze or ODIN should then hook themselves up into NumPy using multiple dispatch. Also SymPy would then hook itself up so that it can be used with array operations natively. Currently SymPy does work with NumPy (see our tests for some examples what works), but the solution is a bit fragile (it is not possible to override NumPy behavior, but because NumPy supports general objects, we simply give it SymPy objects and things mostly work).

Similar to this, I would like to create multiple dispatch in SymPy core itself, so that other (faster) libraries for symbolic manipulation can hook themselves up, so that their own (faster) multiplication, expansion or series expansion would get called instead of the SymPy default one implemented in pure Python.

Other blog posts from the conference:

Bits from Debian: all Debian source are belong to us

This is a verbatim repost from Stefano Zacchiroli's post TL;DR: go to http://sources.debian.net and enjoy.
Debsources is a new toy I've been working on at IRILL together with Matthieu Caneill. In essence, debsources is a simple web application that allows to publish an unpacked Debian source mirror on the Web. You can deploy Debsources where you please, but there is a main instance at http://sources.debian.net (sources.d.n for short) that you will probably find interesting. sources.d.n follows closely the Debian archive in two ways:
  1. it is updated 4 times a day to reflect the content of the Debian archive
  2. it contains sources coming from official Debian suites: the usual ones (from oldstable to experimental), *-updates (ex volatile), *-proposed-updates, and *-backports (from Wheezy on)
Via sources.d.n you can therefore browse the content of Debian source packages with usual code viewing features like syntax highlighting. More interestingly, you can search through the source code (of unstable only, though) via integration with http://codesearch.debian.net. You can also use sources.d.n programmatically to query available versions or link to specific lines, with the possibility of adding contextual pop-up messages (example). In fact, you might have stumbled upon sources.d.n already in the past few days, via other popular Debian services where it has already been integrated. In particular: codesearch.d.n now defaults to show results via sources.d.n, and the PTS has grown new "browse source code" hyperlinks that point to it. If you've ideas of other Debian services where sources.d.n should be integrated, please let me know. I find Debsources and sources.d.n already quite useful but, as it often happens, there is still a lot TODO. Obviously, it is all Free Software (released under GNU AGPLv3). Do not hesitate to report new bugs and, better, to submit patches for the outstanding ones. Acknowledgements PS in case you were wondering: at present sources.d.n requires ~381 GB of disk space to hold all uncompressed source packages, plus ~83 GB for the local (compressed) source mirror

Stefano Zacchiroli: introducing sources.debian.net

all Debian source are belong to us TL;DR: go to http://sources.debian.net and enjoy.
Debsources is a new toy I've been working on at IRILL together with Matthieu Caneill. In essence, debsources is a simple web application that allows to publish an unpacked Debian source mirror on the Web. You can deploy Debsources where you please, but there is a main instance at http://sources.debian.net (sources.d.n for short) that you will probably find interesting. sources.d.n follows closely the Debian archive in two ways:
  1. it is updated 4 times a day to reflect the content of the Debian archive
  2. it contains sources coming from official Debian suites: the usual ones (from oldstable to experimental), *-updates (ex volatile), *-proposed-updates, and *-backports (from Wheezy on)
Via sources.d.n you can therefore browse the content of Debian source packages with usual code viewing features like syntax highlighting. More interestingly, you can search through the source code (of unstable only, though) via integration with http://codesearch.debian.net. You can also use sources.d.n programmatically to query available versions or link to specific lines, with the possibility of adding contextual pop-up messages (example). In fact, you might have stumbled upon sources.d.n already in the past few days, via other popular Debian services where it has already been integrated. In particular: codesearch.d.n now defaults to show results via sources.d.n, and the PTS has grown new "browse source code" hyperlinks that point to it. If you've ideas of other Debian services where sources.d.n should be integrated, please let me know. I find Debsources and sources.d.n already quite useful but, as it often happens, there is still a lot TODO. Obviously, it is all Free Software (released under GNU AGPLv3). Do not hesitate to report new bugs and, better, to submit patches for the outstanding ones. Acknowledgements PS in case you were wondering: at present sources.d.n requires ~381 GB of disk space to hold all uncompressed source packages, plus ~83 GB for the local (compressed) source mirror

Lars Wirzenius: Consistency in backups

A backup run takes time. During a backup run the live data may change. This can mean that the snapshot of the live data may not be internally consistent. For example, suppose you're writing an e-mail when the backup starts automatically. If the timing goes wrong in exactly the right way, the backup might capture both the draft e-mail you were writing when the backup started, and the sent e-mail, when you finished the draft and sent the mail before the backup ended. Or, worse, the backup might miss both the draft and the sent e-mail. The next backup run will, certainly, get the sent e-mail, but what if you have a disaster before that happens? Internal consistency of data can be quite difficult to achive. Some programs, such as MySQL, don't do a good job of keeping the data consistent on-disk, making it difficult for the backup program to get a proper setup. (In that case, a standard trick is to "dump" or "export" the data in a consistent form, and back that up instead.) In an ideal situation, when the backup starts, the system would have a way to tell all running programs to store all their state on disk in a recoverable manner (save all open documents, dump data base content, etc), and once that is done, the backup program creates a snapshot of the filesystem. Unfortunately, I don't know of any operating system that can request consistent state from running applications. Snapshotting, however, is easy-peasy. It can be achieved by the filesystem (e.g., btrfs), or using LVM. Doing the snapshots requires root privileges, but that can probably be arranged. However, snapshotting like this, without getting all programs to commit their state to disk first, is not a guarantee of consistency. A thorough way to be consistent is to shut down the computer, and then reboot it into a special backup mode (similar to single-user mode), where no services or applications are running. However, this is, for most people, much too heavyweight a procedure to be worth it. Does consistency really matter? It matters a lot when the live data is being changed actively and consistency of the data is critical. For example, a bank's database of the amount of money in each account is quite critical: if data is not backed up consistently, money can be lost or created spontaneously, and that's probably a bad thing. Most people, however, don't need to worry a lot, and merely need to be aware of the issue, and perhaps mitigate it by backing up when the computer is mostly idle, and run backups often enough. Oh dear. This blog post contains too few jokes. I seem to have run out of jokes to make, so that will conclude this series of blog posts. I hope it was useful to some.

1 July 2013

Steve Kemp: So wheezy is fun..

Running a pristine operating system is fun. I keep going to run programs and finding they're not installed! For example I'd planned to re-deploy the Debian Administration code-base this evening, but couldn't because fabric is not contained in Wheezy (#714421). Thankfully backporting the package was trivial, but it was a minor stumbling block. I've hit a few of those recently. Still in happier new my lumail mail client can now handle both adding attachments to outgoing mails and extracting them from incoming mails. There are missing features which some people might expect and rely upon, such as: But that said the scripting primitives allow interesting things to be done and I'm enjoying the experience of writing something so "major". True I've got ~20 "stars" on github which isn't a great sign of popularity, but I have had some fun feedback and the client works for me. I'm going to have to spend a few days working on TAB-completion code that plays-nice with curses, because that's a major irritation (and you can't mix/match curses & readline, annoyingly). But otherwise I think we're getting close to being complete enough I'll slow down.

28 June 2013

Lars Wirzenius: Who knows what evil lurks in the data of men? Security aspects of backups

You're not the only one who cares about your data. A variety of governments, corporations, criminals, and overly curious snoopers are probably also interested. (It's sometimes hard to tell them apart.) They might be interested in it to data in order to find evidence against you, blackmail you, or just curious about what you're talking about with your other friends. They might be interested in your data from a statistical point of view, and don't particularly care about your specifically. Or they might be interested only in you. Instead of reading your files and e-mail, or looking at your photos and videos, they might be interested in preventing your access to them, or to destroy your data. They might even want to corrupt your data, perhaps by planting child porn in your photo archive. You protect your computer as well as you can to prevent these and other bad things from happening. You need to protect your backups with equal care. If you back up to a USB drive, you should probably make the drive be encrypted. Likewise, if you back up to online storage. There are many forms of encryption, and I'm unqualified to give advice on this, but any of the common, modern ones should suffice except for quite determined attackers. Instead of, or in addition to, encryption, you could ensure the physical security of your backup storage. Keep the USB drive in a safe, perhaps, or a safe deposit box. The multiple backups you need to protect yourself against earthquakes, floods, and roving gangs of tricycle-riding clowns, are also useful against attackers. They might corrupt your live data, and the backups at your home, but probably won't be able to touch the USB drive encased in concrete and buried in the ground at a secret place only you know about. The other side of the coin is that you might want to, or need to, ensure others do have access to your backed up data. For example, if the clown gang kidnaps you, your spouse might need access to you backups to be able to contact your MI6 handler to ask them to rescue you. Arranging safe access to (some) backups is an interesting problem to which there are various solutions. You could give your spouse the encryption passphrase, or give the passphrase to a trusted friend or your lawyer. You could also use something like libgfshare to escrow encryption keys more safely.

Matt Zimmerman: Scaling Human Systems: Making and keeping commitments

This is part 3 in a series on organizational design and growth. What it means

Taking responsibility for action, and following through. Every time. Why it s important

Commitments are a cornerstone of collaborative teamwork. Knowing what to expect from others makes it possible to plan beyond your own work. If you aren t sure whether a task will be completed, you carry that uncertainty as low-grade anxiety, which accumulates and creates cognitive load. You may even spend extra time checking back to make sure the need has been met. Old status quo

Most work happens without an explicit commitment at all. Consistency is highly variable from one individual, team or circumstance to another. Tasks fall through the cracks, and we may not even realize it until much later. New status quo

When something needs to get done, someone agrees to take responsibility for it. When this happens, everyone involved can trust that it will get done. When exceptions happen, we acknowledge them, and seek to understand what happened so that we can do better in the future. Behaviors that help

Obstacles that stand in our way Further reading

27 June 2013

Russ Allbery: Review: The Odd Clauses

Review: The Odd Clauses, by Jay Wexler
Publisher: Beacon
Copyright: 2011
ISBN: 0-8070-0089-2
Format: Trade paperback
Pages: 193
The subtitle of this book summarizes it nicely: Understanding the Constitution through Ten of Its Most Curious Provisions. (The constitution in question is the United States Constitution.) It's a sort of popularization of US constitutional law, but rather than look at the most well-known and controversial parts, it takes a look at some of the odd side points, obscure clauses, or clauses that one rarely thinks about because they rarely appear in the news. This is partly a tour of curiosity aimed at those who like legal trivia, but it also provides, indirectly, a look at how constitutional law is made and debated, and at some of the burning concerns at the time of the writing that have since faded away. Sometimes why the concern has faded is as interesting as the original problem. This is the sort of book whose table of contents provides a lot of insight into what you're getting. It has ten chapters, each devoted to a particular clause or amendment: incompatibility, weights and measures, recess appointments (one of the rare ones that has made the news), original jurisdiction, natural-born citizen (odd, but sadly now not at all obscure), letters of marque and reprisal, titles of nobility, bills of attainder, and the 3rd and 21st amendments. (For those who haven't memorized all the amendments to the US Constitution, the 3rd is the prohibition on quartering soldiers in private homes and the 21st is Prohibition, which here is examined not for its core intent but for the implications for federalism.) As you can see, this is quite a selection, and while the clauses are often obscure, the issues raised by them are still generally quite active. There are some that were intensely controversial at the time of the original debate that are little more than amusing trivia at this point (the prohibition on officers of the United States taking titles of nobility), or have become so much a part of the structure of the government that people rarely given them a second thought (such as the incompatibility clause, which prohibits members of Congress from holding positions in the executive branch). But most of these clauses establish ground rules for the US government that are still quite current today, even if the legal terminology used at the time is no longer familiar. A good example is the clause prohibiting bills of attainder, something that the writers of the US Constitution saw no need to define. A bill of attainder is an act of a legislature declaring a person guilty of a crime and assigning some punishment without trial. Framed that way, it seems like an obvious extension of the separation of powers system that the US government is constructed around, but unlikely to provoke much modern controversy. However, as Wexler explains quite lucidly, there's a reason why this clause is right next to the prohibition on ex post facto laws. While no one would stand for the US Congress declaring guilt on a matter of criminal law, it has written some startlingly specific laws that seem aimed solely at specific individuals or organizations, which is exactly the sort of behavior that this prohibition, interpreted broadly, was intended to prevent. Similarly, one might think that Congress's power to issue letters of marque and reprisal is little more than ground for an interesting history lesson. But that clause is deeply entangled with the checkered history of US declarations of war and controversies over Congressional oversight, and Wexler uses it as a jumping-off point to take a look at the implications for the modern habit of undeclared wars fought by military contractors. This is a short book, so these treatments are necessarily brief. Some of them are also fairly minor; for example, the discussion of the weights and measures clause is mostly an extended rant on the US refusal to adopt the metric system. That's a sentiment with which I entirely agree, but I'm not sure it has much to do with constitutional law. But Wexler is an entertaining and concise writer; even when the discussion is relatively shallow, I enjoyed reading his analysis. (I will warn, though, that he takes a few potshots from a liberal perspective, so one's enjoyment of his humor may to some extent depend on your personal political allegiances.) I wouldn't rush out to find this book, but I found it mildly diverting and occasionally quite informative. The chapter on original jurisdiction, for instance, explained how the Supreme Court handles certain classes of cases that must be filed directly with it, despite not being a trial court. This was a question that I didn't even know I had, and a side system of judicial procedure that turned out to be surprisingly interesting. I suspect The Odd Clauses will be the most interesting to people who have only a news media understanding of the US Constitution. I went through a personal libertarian period, during which I read the constitution in detail and sought out some discussions of its specifics, so quite a bit of this book was familiar ground to me. Others without that study background may find more novelty. But even with that background, I was entertained, and don't regret buying it. One note, though: I bought this book on the recommendation of Kevin Underhill, author Lowering the Bar, which is a masterpiece of dry humor. If you're looking for legal humor, parts of The Odd Clauses are mildly funny, but Lowering the Bar is the sort of site that will have you staying up late reading just one more page of archives and reading bits aloud to anyone nearby. Rating: 7 out of 10

Matthew Garrett: Automating HP server BIOS setup

One of the goals of our work at Nebula is making it as easy as possible for someone to set up a private cloud. In an ideal world that would involve being able to just plug in the controller hardware, wire up a rack of servers and turn everything on, but right now there are cases where the default firmware configuration on the servers doesn't match our desired configuration. Plugging a console into individual servers just to set some BIOS options is (based on personal experience) about as much fun as writing a doctoral thesis on the experience of watching paint dry, so it seemed worth trying to find a way to avoid people having to deal with that. Thankfully, it turns out that the industry has come to a similar set of conclusions. Recent Dell hardware lets you use WS-MAN, which makes it easy to do things like enable security features as long as you have an authenticated connection to the iDRAC management system. This actually works out wonderfully - the first time a Dell node sends a PXE request to the Cloud Controller, it can push back some configuration changes, reboot the system and then boot it in a known good configuration. Thanks, Dell![1].

HP's slightly trickier. As far as I've been able to work out, the remotely-available programmatic interfaces only provide configuration interfaces to the iLO device itself, along with a range of chassis-level monitoring. Instead, HP provide a couple of utilities that allow the OS to change values. The first of these is called conrep. Conrep is able to modify BIOS settings, but needs an XML file which tells it which configuration values are at which addresses. That's a bit of a pain. Thankfully it's being replaced with a new tool called hprcu which is able to directly query the firmware in order to figure out which configuration options are available and which values have to be stored where in order to set them. You run hprcu once and it spits out a file containing your existing settings. You modify that, feed it back into hprcu, reboot and you're set.

Sounds pretty ideal. What's the problem? The first is that hprcu is currently only shipped as a 32-bit binary, and right now we don't deploy any 32-bit support code on our nodes. It'd be irritating to have to do so just to configure the firmware. The second is that it works by doing raw port IO, and that's not going to be possible from userspace once we've moved to a UEFI Secure Boot setup.

So, obviously, I've been working on reimplementing it. The first step was to figure out what it was actually doing. The first step was to strace it to see if it was using the kernel IPMI interface. strace showed no accesses to /dev/ipmi*. My next thought was that it could be using some sort of shared memory segment with the iLO hardware, so I used MMIOTrace to dump the memory accesses it made. Turned out there weren't many, and certainly not enough to do anything interesting. Then I went back to the strace logs and saw that it was calling iopl() a bunch, and then I got sad.

iopl() allows a userspace application to change its io privilege levels. The Linux manpage for the iopl() call is amazingly helpful, informing us with a straight face that "This call is necessary to allow 8514-compatible X servers to run under Linux." What it actually does is grant userspace access to the full range of io ports.

So, what's an io port? The x86 architecture (and some others) provide two ways to communicate with hardware. The one that's mostly used these days is to map the hardware into the same address space as memory (memory-mapped io), but x86 has an entirely separate address range that can be used. This is significantly more limited, as only 65,536 addresses are available for the entire system. Further, you can only read or write a maximum of 4 bytes in a single transaction. However, while slower and less generally useful than memory-mapped io, port io has the advantage that it's much simpler to implement. As a result, a lot of the original PC hardware was intended to be accessed via port io, and a bunch of that's still present. Want to reprogram your real-time clock? Port io. Want to read from the legacy keyboard controller? Port io. For low-bandwidth transactions, it's a completely reasonable way to implement hardware.

Applications[2] perform port io by executing in and out instructions. Since these instructions allow you to do things like, say, hit the keyboard controller directly[3], userspace is generally forbidden from calling them. Sufficiently privileged applications can call iopl() to raise their privileges and gain access to the full set of io ports. But, since in and out are CPU-level instructions, the actual port io accesses won't show up in strace. So I had to take another approach.

The correct way of doing this would be to LD_PRELOAD something that intercepted iopl(), didn't call it, let the application perform an in or out instruction, caught the fault, looked at the stack frame, called iopl(), performed the access, dumped debug details, dropped iopl() again and restored state. Because that seemed like a bunch of work, I took advantage of the fact that hprcu had separate in and out functions and used gdb. I told gdb to break on every call to in or out, dump the register state and then continue. Then I hacked up a script to parse the register dumps and tell me where the reads and writes were going. Astonishingly, it worked. And then I read the results and got really sad.

PCI setup is hard, if you're a BIOS. You've got PCI devices with large memory windows and you've got to arrange them all somehow and look you've only got 640K of RAM you can access while you're doing this and come on, seriously? So PC-type systems define a port IO interface to performing PCI configuration. Each PCI device has an address made up of a bus, a device and a function. You take a 32-bit integer, set the top bit to indicate that you want PCI access, put the bus in bits 16-23, the device in bits 11-15, the function in bits 8-10 and the configuration register on that device you want to access in bits 0-7. Then you write that to io port 0xcf8. Reads or writes to io port 0xcfc then read or write from that configuration register on that device. This lets the BIOS tell each PCI device where it's going to live in mmio address space without having to actually perform any mmio. Good work, BIOS developers.

This works fine in the BIOS, because nothing else is running while the BIOS is. It even works fine in the kernel, which uses this approach on older hardware that doesn't provide a memory-mapped mechanism to access PCI configuration registers. But it works really badly if you're doing it in userspace, because it shares a problem with all other indexed accesses. You write an address to 0xcf8. You write a value to 0xcfc. What happens if the kernel writes to 0xcf8 before you write to 0xcfc? Your access goes to some other register on some other device and suddenly you've just mapped a PCI device over the top of RAM and well I sure hope you saved all your work.

So, I wasn't thrilled to discover that hprcu was communicating with the iLO by using 0xcf8/0xcfc port io accesses. Not only was it going to stop working once we started deploying UEFI Secure Boot, it had the potential to cause really annoyingly hard to debug problems. Working out how to reimplement it became something of a priority.

By looking at the XML files it generated, and by following the port IO accesses it was performing, I could figure out pretty much how it worked. Some configuration values are stored in the real-time clock CMOS. These were just done in the standard way - write the address to port 0x70, read or write the value to port 0x71[4]. Other configuration values were stored in NVRAM, with accesses going via PCI. This was a little trickier to figure out, because sometimes different NVRAM addresses went to the same PCI address. I finally figured it out - there's a 48 byte window into NVRAM via the PCI configuration registers, and another register which chooses which set of NVRAM is visible. So, take the NVRAM address, divide by 48, write that value to register 0xa6, take the modulus, add it to 0x80 and write the desired value to that address.

So now I knew enough to be able to take an existing XML file and deploy it. Definitely progress, and I could even add support to the kernel. But I still wanted to know how hprcu generated that XML file in the first place. Running strings over the binary showed a bunch of debug output that it never actually printed, but immediately after the help text it also printed SsLlFf:HhAaTtOoDd. I've lost enough years of my life to this kind of thing to be able to identity that as a getopt format string, and there was that tantalising D option at the end that went entirely unmentioned in the help text. Re-running hprcu with the -D argument gave me huge piles of debug output, including references to a DMI entry and an $RBS table.

DMI is a standard for exposing system information, and one aspect of it is providing data from the firmware to the operating system. The OS can look for a defined signature in a fixed area of memory and then pull out a bunch of data about the hardware, including things like the vendor name, model, serial number, BIOS version and more. Some of these information tables are standardised and some are vendor-specific. hprcu was looking for a vendor-specific table and then scanning for it looking for a known signature. That table turned out to to contain a set of signatures and addresses. The address corresponding to the signature hprcu was contained a bunch of data starting with "$RBS". It also contained a huge quantity of ASCII that looked awfully like the strings in the XML file that hprcu had written. Success!

So, the rest of today was spent on working out the format of this table. This was only marginally less tedious than setting up BIOS settings on 20 servers by hand, but I've made enough progress to be able to figure out how to write a kernel-level driver for this. That's about half-done now - the parsing code is all implemented, I just need to add the sysfs glue and I am nowhere near drunk enough for that right now[5], so for now you're just getting a format description.

To start with, look for a DMI table with a type of 0xe5. This contains several entries of 4 bytes of signature, 8 bytes of address and 4 bytes of length. Find the entry with a signature of "$CRQ" and map the corresponding address. The first four bytes of the mapped region should be "$RBS". The first record is 21 bytes into the file. Each record starts with a single byte representing the type and two bytes representing the record length. Records of type 1 contain an ascii string that starts 8 bytes into the record, and have a further type number 6 bytes into the record that tells you what type of string it is. 0x05 indicates that it's a "Feature string", which defines the overall classification of the following features. 0x06 is an "Option string", which provides a human readable explanation of what this configuration value corresponds to. 0x60 is an "Optional warning string", which tells the user which further configuration may be required before the configuration change takes effect.

All other types appear to be configuration options. Bytes 3 to 6 provide the little-endian name of the configuration option. Bytes 7 and 8 provide a unique numberical identifier for the configuration option. Byte 13 indicates the type. 0x03 is stored in CMOS. 0x04 is stored in NVRAM. 0x05 appears to be some different kind of CMOS store. I haven't figured out the others. There then follows a set of choices. These vary in length depending on the type - 0x03 are 14 bytes long, 0x04 are 6 bytes long, 0x05 are 5 bytes long. Type 0x03 choices have the choice id at byte 0, the CMOS offset at byte 2, the mask at byte 3 and the value at byte 4. Type 0x04 choices have the choice id at byte 0, the nvram offset at bytes 1 and 2, the mask at byte 3 and the value at byte 4. Type 0x05 choices have the choice id at byte 0, the cmos offset at byte 1, the mask at byte 2 and the value at byte 3. There are optional flags following each choice - if the final byte of the choice isn't 0, there then follow 6 bytes of flags. The first four bytes provide the name of a configuration option (little-endian, again), the fifth byte refers to a choice on that option and the sixth byte indicates what kind of flag. 0x1 appears to indicate that the option is mutually exclusive with that other option. Examples include the embedded serial port and virtual serial port options, where it's impossible to map both to the same address at once. A choice can have multiple flags.

After a configuration option, there'll be a set of option string records. In turn, these correspond to the previous choices - if a configuration option had three choices, there'll be three option string records. I haven't figured out whether there's a stronger way of binding the option string records to the configuration choice yet. The other thing I haven't entirely figured out are the details of configuring the platform to perform a cold reboot on the next power cycle, but the io traces for that don't look too bad.

The aim is to provide a kernel driver that exposes all these configuration options via sysfs, including indicating the current value, the available values and letting the user set a new value. In an ideal world we'd have a wonderfully generic interface to this kind of functionality, but I'm (sadly) not sure that that's possible.

And that's how I spent the past two days.

[1] (Thell)
[2] Or, indeed, kernels
[3] And, hence, read your keystrokes
[4] Which has exactly the same problems as the PCI access, in that if you happen to ask the clock what the time is while hprcu is accessing CMOS, at least one of you is going to get very confused
[5] Obviously, I would never write kernel code while drunk. You're certainly not running any of it in your enterprise kernel.

comment count unavailable comments

25 June 2013

Daniel Pocock: Configuring strongSwan on Debian, RHEL and Fedora with the Android client

In my earlier blog post about VPNs, I looked at a range of VPN options. The strongSwan wiki documentation is generally quite good but it doesn't describe the exact procedure for an Android user anywhere. This blog aims to fill that gap. Here I am sharing some working examples of how to configure and use RSA certificate authentication between a Linux box and an Android phone using strongSwan. There are several commands to execute for each step, but they can be easily scripted. Here are the versions in play: Certificate creation: private certificate authority "example.org CA root" For production, I would highly recommend creating your own root CA on a smart card and managing it from a clean machine without any network connection. To start quickly for testing, just create a CA using the steps from the strongSwan wiki, in particular: 
umask 0177
ipsec pki --gen > caKey.der
ipsec pki --self --in caKey.der \
   --dn "C=CH, O=Acme Ltd, CN=example.org CA root" \
   --ca > caCert.der
Certificate creation: VPN gateway machine vpn.example.org Now you need to create a key for the server. The key thing to observe here is the use of the "--san" argument: the Android client expects a subjectAltName in the certificate from the server. The VPN gateway has the hostname vpn.example.org: 
ipsec pki --gen > vpnKey.der
ipsec pki --pub --in vpnKey.der   \
   ipsec pki --issue --cacert caCert.der --cakey caKey.der \
  --dn 'CN=vpn.example.org' \
  --san vpn.example.org > vpnCert.der 
cp vpnKey.der /etc/ipsec.d/private/
cp vpnCert.der /etc/ipsec.d/certs/
Certificate creation: mobile user For the Android client, it is necessary to create a private key and certificate and then embed them together with a copy of the root certificate in a PKCS#12 file. We will need OpenSSL or GnuTLS for some of these steps. The pkcs12 command will prompt for a password: make up a long, obscure password, you will only use it once when importing the certificate into the phone. Here is a script you can use for each client certificate you want to create: 
#!/bin/bash -e
umask 0177
TARGET_HOST=$1
DOMAIN=example.org
IPSEC=/usr/sbin/ipsec
KEY_FILE="$ TARGET_HOST Key.der"
CRT_FILE="$ TARGET_HOST Cert.der"
if [ -f $ KEY_FILE  ];
then
  echo "$KEY_FILE exists"
  exit 1
fi
$ IPSEC  pki --gen > $ KEY_FILE  
$ IPSEC  pki --pub --in  $ KEY_FILE    \
  $ IPSEC  pki --issue \
  --cacert caCert.der --cakey caKey.der \
  --dn "CN=$ TARGET_HOST .$ DOMAIN " > $ CRT_FILE 
[ -f caCert.pem ]   \
   openssl x509 -in caCert.der -inform der -out caCert.pem -outform pem
openssl rsa \
   -in $ KEY_FILE  -out $ TARGET_HOST Key.pem -inform der
openssl x509 \
   -in $ CRT_FILE  -inform der \
   -out $ TARGET_HOST Cert.pem -outform pem
openssl pkcs12 \
   -export \
   -in $ TARGET_HOST Cert.pem \
   -inkey $ TARGET_HOST Key.pem \
   -certfile caCert.pem \
   -out $ TARGET_HOST .p12
# delete the key files, leaving only the key in the p12 file:
rm $ KEY_FILE  $ TARGET_HOST Key.pem
Assuming you call that script make-client-creds, you would run it once for each Android device: 
$ ./make-client-creds phone1
$ ./make-client-creds phone2
...
Loading the PKCS#12 file into the phone Copy the cert bundle (the p12 file only) to the SD card in the phone. You can mount the phone over USB or using ADB, e.g. 
$ adb push phone1.p12 /sdcard/
Now go to the Android settings, select Security and Install from SD card. Follow the prompts. Setting up the VPN gateway Add the following line to /etc/ipsec.secrets 
: RSA vpnKey.der
and add a stanza like the following to /etc/ipsec.conf 
conn private-droid
	left=vpn.example.org
	leftsubnet=insert gateway IP here/32
	leftcert=vpnCert.der
	leftid=@vpn.example.org
	leftfirewall=no
	lefthostaccess=no
	right=%any
	rightsourceip=192.168.10.0/24
	auto=add
Finally, restart ipsec and you should be ready. 
# service ipsec restart
Install the app on the phone Finally, install the app on the phone. This bit should be quite easy. Once installed, press the option ADD VPN, put in the DNS name of the VPN gateway (must match the name in the certificate, e.g. vpn.example.org from the example above) and tell it to use IKEv2 Certificate as the authentication type. Press to select a certificate and highlight the certificate that you installed earlier. Checking for problems You should see a message on the phone telling you it is connected to the VPN. If it fails, it will let you see the log messages. If you see the error Constraint check failed it often means that the subjectAltName was not included in the VPN gateway's certificate (see the steps above) or the DNS name doesn't match the name in the certificate. On the server side, this command shows who is connected: 
# ipsec status
Making it secure with iptables Once the packets are decrypted by strongSwan on the VPN gateway, the kernel will route them appropriately just like any other packet. Use standard Linux netfilter/iptables rules to ensure that routing is restricted appropriately so that the clients can access the services they need and nothing else. Support and discussion Please feel free to post comments below or discuss on the strongSwan Users mailing list

Dirk Eddelbuettel: Rcpp 0.10.4

A new version of Rcpp is now on the CRAN network for GNU R; binaries for Debian have been uploaded as well. This release brings a fairly large number of fixes and improvements across a number of Rcpp features, see below for the detailed list. We are also announcing with this release that we plan to phase out the RCPP_FUNCTION_* macros. Not only have they been superceded by Rcpp Modules and Rcpp Atributes (each of which has its own pdf vignette in the Rcpp package), but they also appear to be at best lightly used. We are for example not aware of any CRAN packages deploying them. To provide a smooth transition, we are aiming to keep them around for another twelve months, but plan to remove them with the first release after that time window has passed. As before, we tested the release rather rigorously by checking against all packages I could (relatively easily) built on my server: this time it covered 91 of the 124 CRAN packages depending on Rcpp. As all of these 91 packages passed their checks, we do not expect any issues with dependent packages. The complete NEWS entry for 0.10.4 is below; more details are in the ChangeLog file in the package and on the Rcpp Changelog page.
Changes in Rcpp version 0.10.4 (2013-06-23)
  • Changes in R code: None beyond those detailed for Rcpp Attributes
  • Changes in Rcpp attributes:
    • Fixed problem whereby the interaction between the gc and the RNGScope destructor could cause a crash.
    • Don't include package header file in generated C++ interface header files.
    • Lookup plugins in inline package if they aren't found within the Rcpp package.
    • Disallow compilation for files that don't have extensions supported by R CMD SHLIB
  • Changes in Rcpp API:
    • The DataFrame::create set of functions has been reworked to just use List::create and feed to the DataFrame constructor
    • The operator-() semantics for Date and Datetime are now more inline with standard C++ behaviour; with thanks to Robin Girard for the report.
    • RNGScope counter now uses unsigned long rather than int.
    • Vector<*>::erase(iterator, iterator) was fixed. Now it does not remove the element pointed by last (similar to what is done on stl types and what was intended initially). Reported on Rcpp-devel by Toni Giorgino.
    • Added equality operator between elements of CharacterVectors.
  • Changes in Rcpp sugar:
  • Changes in Rcpp build tools:
    • Fix by Martyn Plummer for Solaris in handling of SingleLogicalResult.
    • The src/Makevars file can now optionally override the path for /usr/bin/install_name_tool which is used on OS X.
    • Vignettes are trying harder not to be built in parallel.
  • Changes in Rcpp documentation:
    • Updated the bibliography in Rcpp.bib (which is also sourced by packages using Rcpp).
    • Updated the THANKS file.
  • Planned Deprecation of RCPP_FUNCTION_*:
    • The set of macros RCPP_FUNCTION_ etc ... from the preprocessor_generated.h file will be deprecated in the next version of Rcpp, i.e they will still be available but will generate some warning in addition to their expected behavior.
    • In the first release that is at least 12 months after this announcement, the macros will be removed from Rcpp.
    • Users of these macros (if there are any) should start replacing them with more up to date code, such as using Rcpp attributes or Rcpp modules.
Thanks to CRANberries, you can also look at a diff to the previous release 0.10.3. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

24 June 2013

Vincent Sanders: A picture is worth a thousand words

When Sir Tim made a band the first image on the web back in 1992, I do not imagine for a moment he understood the full scope of what was to follow. I also do not imagine Marc Andreessen understood the technical debt he was about to introduce and that fateful day in 1993 when he proposed the img tag allowing inline images.

Many will argue of course that the old adage of my title may not apply to the web, where if it were true, every page view would be like reading a novel! and many of those novels would involve cats or one pixel tall colour gradients.

Leaving aside the philosophical arguments for a moment it takes web browser author a great deal of effort to quickly and efficiently put those cat pictures in front of your eyeballs.
NavigatingImages are navigated by a user in a selection of ways, including:
Whatever the method, the resulting object is subject to the same caching operations as any content object within a browser. These caching and storage operations are not specific to images however images are one of the most resource intensive objects a browser must regularly deal with (though javascript and stylesheet sources are starting to rival it at times) because they are relatively large and numerous.
Pre render processing
When the image object fetch is completed the browser must process the image for rendering which is where this gets even more complicated. I shall use how this works in NetSurf as I know that browsers internals best, but operation is pretty similar in many browsers.

The content fetch will have performed basic content sniffing to determine the received objects mime type. This is necessary because a great number of servers are misconfigured and a disturbingly large number of images served as png are really jpegs etc. Indeed sometimes you even get files served which are not images!

Upon receipt of enough data to decode the image header, for the detected mime type, the images metadata is extracted. This metadata usually includes things like size and colour depth.

If the img tag in the original source document omitted the width or height of the image the entire document render may have to be reflowed at this point to allow the correct spacing. The reflow process is often unsightly and should be avoided. Additionally at this stage if the image is too large to handle or an unhandled format the object will be replaced with the "broken" image icon.

Often that will be everything that is done with the image, when I added "lazy" image conversion to NetSurf we performed extensive profiling and discovered well over 40% of images on visited pages are simply never viewed by the user but that small (100 pixel on a side) images were almost always displayed.

This odd distribution comes down to how images are used in modern web pages, they broadly fall into two categories of "decoration" and "content" for example all the background gradients and sidebar images etc. are generally small images used as decoration whereas the cat picture above is part of the "content". A user may not scroll down a page to see content but almost always gets to "view" the decoration.
Rendering
Created by Andrea R used under CC Attribution-NonCommercial-ShareAlike 2.0 licence
The exact browser heuristics used differ as to when the render operation is performed but they all have a similar job to perform. When i say render here this may be possibly as an "off screen" view if they are actually on another tab etc. Regardless the image data must be converted from the source data (a PNG, JPEG etc.) into a format suitable for the browsers display plotting routines.

The browser will create a render bitmap in whatever format the plotting routines require (for example the GTK plotters use a Cairo image surface) , use an image library to unpack the source image data (PNG) into the render bitmap (possibly performing transforms such as scaling and rotation) and then use that bitmap to update the pixels on screen.

The most common transform at render time is that of scaling, this can be problematic as not all image libraries have output scaling capabilities which results in having to decode the entire source image and then scaling from that bitmap.

This is especially egregious if the source image is large (perhaps a multi megabyte jpeg) but the width and height are set to produce a thumbnail. The effect is amplified if the user has set the image cache size limit to a small value like 8 Megabytes (yes some users do this apparently their machines have 32MB of RAM and they browse the web)

In addition the image may well require tiling (for background gradients) and quite complex transforms (including rotation) thanks to CSS 3. Add in that javascript can alter the css style and hence the transform and you can imagine quite how complex the renderers can become.
CachingThe keen reader might spot that repeated renderings of the source image (e.g. because window is scrolled or clipped) result in this computationally expensive operation also being repeated. We solve this by interposing a render image cache between the source data and the render bitmaps.

By keeping the data in the preferred format, image rendering performance can be greatly increased. It should be noted that this cache is completely distinct from the source object cache and relates only to the rendered images.

Originally NetSurf used to perform the render conversion for every image as it was received without exception, rather than at render time, resulting in a great deal of unnecessary processing and memory usage. This was originally done for simplicity and optimising for "decoration" images.

The rules for determining what gets cached and for how long are somewhat involved and the majority of the code within the current implementation NetSurf uses is metrics and statistic generation to produce better decisions.

There comes a time at which this cache is no longer sufficient and rendering performance becomes unacceptable. The NetSurf renderer errs on the side of reducing resource usage (clearing the cache) at the expense of increased render times. Other browsers make different compromises based on the expected resources of the user base.
FinallyHopefully that gives a reasonable overview to the operations a browser performs just to put that cat picture in front of your eyeballs.

And maybe next time your browser is guzzling RAM to plot thousands of images you might have a bit more appreciation to exactly what it is up to.

Daniel Pocock: Private WANs may be less secure than VPNs

The latest round of Snowden revelations concern a British GCHQ program dubbed "Mastering the Internet (MTI)". The program involves, among other things, tapping the world's under-sea fibre-optic cables and systematically monitoring all communications. One of the most significant facts that appears is the benefit of using IPsec and VPNs over the public Internet instead of trusting un-encrypted traffic to pass over private WAN connections. The definition of a private WAN connection is quite broad: many ISPs in the UK, for example, use the British Telecom OpenReach platform to tunnel DSL connections back to their data centers and offer virtual WANs to their business customers. Can any wifi be trusted? It should be assumed that all Wifi networks can be compromised at-will by a sophisticated attacker. This is another situation where VPNs may be part of the solution: even WPA2 secured wifi segments should be treated as untrusted public Internet networks and not bridged or routed onto any private subnets. Users with devices on a wifi network should be forced to use a VPN over wifi connection, even when the wifi is within the same building as the private network. Tor may be compromised by an attacker with this much data As I've mentioned in previous blogs with a focus on real-time communications there is no such thing as perfect privacy. Simply encrypting data with IPsec does not prevent the attacker knowing the volume of data transmitted or the time it was transmitted. Using a service like Tor for onion-routing provides some benefit. However, an attacker with sufficient infiltration of many points in the network may be able to deduce the real origins of a data transfer over Tor using some combination of packet sizes and timestamps. "Mastering the Internet (MTI)" may have sufficient coverage to achieve that. Tor could overcome this by putting dummy data into the network and adding fragmentation, but such schemes may add latency, reduce throughput and require significant effort to upgrade existing users. Gaining intelligence from encrypted data VoIP signals are typically compressed using some type of codec. These are dedicated, lossy compression schemes optimised for voice signals. Some codecs use a fixed compression ratio. Other codecs use a variable compression ratio (VBR): a simple example involves sending shorter packets to indicate a period of silence. Encrypting these packets prevents an attacker seeing what is inside them but the attacker can easily detect the lengths of the packets. Researchers have demonstrated that statistical techniques can be applied to VBR dataflows to identify syllables used in speech and then make a hypothesis about what has been said. (This would be an interesting hack to try and reproduce by collecting data with tcpdump and analysing it with R) Identifying phrases in encrypted VoIP using statistical analysis In many cases like those described, it is necessary to go beyond default encryption settings and put extra dummy data or frame padding into the data flow to gain a more fool-proof level of privacy. Simply connecting VoIP phones to an IPsec network doesn't automatically make them secure. The rise of the free-software VPN Even if the default configurations are not always universally resilient to every type of attack, many free software prodcuts provide an excellent building block for VPNs. Having the source code and specifications on hand, free software developers can then innovate to mix-and-match different technologies or add random data flows to mask the real activity on the network. In a recent hosting migration project, I successfully used the rather trivial VPN package tinc to bridge the ethernet segments between virtual machines in the old and new networks. This allowed virtual machines to be moved one-by-one over a period of a week. This is a remarkably simple solution that can protect the integrity of communications on a private subnet. A more comprehensive solution is strongSwan which provides support for industry-standard IPsec in various topologies. They now offer a full IPsec client for Android phones (hopefully it will be made available on f-droid: using a Google account to access Google Play may result in Google accidentally helping you "back up" your settings, including credentials, to their cloud, undermining the privacy of the system) An IPsec VPN can be deployed to all hosts on a network: then they can use opportunistic encryption for any type of communication without needing to worry about the underlying network. Furthermore, it is possible to add smart cards to further boost VPN security while using free software for the whole solution. For those interested in looking beyond Linux solutions, the *BSD range of operating systems, particularly OpenBSD provide a compelling alternative. This paper about OpenBSD's iked, which is also available on other platforms as OpenIKEd is very recent and provides a thorough overview of it's current status. One notable limitation is that their MOBIKE support is not yet complete. Many older VPNs likely to need an ugprade When I first started deploying Linux VPNs in the 90s the market was favourable for any type of Linux VPN. In extreme cases (which were surprisingly common at the time) companies accustomed to paying $20,000 per month for a leased line would happily pay a flat fee of $100,000 to purchase a VPN that was cobbled together using less than $10,000 of commodity hardware and some free software. Many of the first generation of these VPNs were built using RSA key lengths and algorithms no longer considered secure, such as DES. Some of these solutions (or products derived from them) may still remain in use today: managers may be slightly embarrassed to put aside their $100,000 VPN and replace it with a $50 router that is more secure. While that is the more extreme example, there are many intermediate solutions out there in the field. There are clear opportunities here to upgrade users to free software solutions.

23 June 2013

Ben Armstrong: Tutorial: wheezy live iso-hybrid with persistence on USB

As of wheezy, Debian no longer offers for download any prebuilt hdd (formerly known as usb-hdd) live images, which, in squeeze, were necessary if you wanted to use persistence. The good news is, since the new wheezy iso-hybrid images use xorisso, you can copy an image to a USB key and repartition it to add a persistence partition. Unfortunately, the filesystem itself is read-only, so you have to manually append the persistence boot parameter each time you boot the image. If that doesn t bother you, you can skip this whole tutorial and just follow the instructions in live-manual for adding a persistence partition and you re all set. This tutorial explains how to extract the contents of a wheezy iso-hybrid live image to a USB key and modify it to boot with this parameter permanently enabled. I am writing it solely as a workaround and do not endorse this as a means to customize Debian Live images in general. I would love to help make a more convenient, supported way to accomplish the same thing for a future release. Meanwhile, the supported and recommended way to make any change to a Debian Live image is to build it from scratch using live-build, as described in live-manual. A simplified Debian Live Images Autobuilder web service is available that works well for many common customizations, if you prefer. Please use one of these methods if you want to do this, or other customizations in the supported way. Otherwise, read on. Prerequisites Identify the USB key device Caution: Always double-check the device you are writing to is the correct one to avoid losing precious data. There are two things to do to protect yourself. The first is, don t write the USB key as root. The second is, first use ls -l /dev/disk/by-id to identify which device is the target USB key. 
lrwxrwxrwx 1 root root  9 Jun 22 11:58 
usb-SanDisk_Cruzer_Contour_0000184CA87406BC-0:0 -> ../../sdb
As you can see, my SanDisk Cruzer Countour is /dev/sdb. For the rest of the tutorial, I ll be referring to this device as /dev/sdX to avoid unfortunate cut-and-paste disasters. When you follow along, make sure you substitute the correct device for your USB key on your system. Initialize the USB key Start by plugging in the key and unmounting any automounted partitions to ensure nothing is accessing the device, e.g. 
$ umount /dev/sdX1
Now make changes to the partition table. Use parted because it may be run as an ordinary user and has some advanced capabilities beyond more basic tools like fdisk. 
$ /sbin/parted
WARNING: You are not superuser.  Watch out for permissions.
GNU Parted 2.3
Using /dev/sdX
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print devices
/dev/sdX (8221MB)
(parted) select                                                           
New device?  [/dev/sdX]?                                                  
Using /dev/sdX
(parted)
Note that when run as an ordinary user, if you only have one removable drive (which may include mp3 players, cameras, e-readers) plugged in, parted immediately selects that device. But just to be sure, use print devices to list the devices and select to if you need to select a different one, as shown above. Repartition the USB key Once you ve verified you have the correct device, start repartitioning. Write a new partition table with mklabel msdos to wipe out all of the old partitions on it. You may skip this step and adapt the remaining instructions if there are partitions you need to keep. This is your last chance to bail if the device is wrong, so make sure it is the correct one before proceeding. 
(parted) mklabel msdos
Warning: The existing disk label on /dev/sdX will be destroyed and all data
on this disk will be lost. Do you want to continue?
Yes/No? y
Error: Partition(s) 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16 on
/dev/sdX have been written, but we have been unable to inform the kernel of
the change, probably because it/they are in use.  As a result, the old
partition(s) will remain in use.  You should reboot now before making further
changes.
Ignore/Cancel? i
(parted)
The error is normal, and appears when you run parted as an ordinary user, so it is OK each time it appears to press i to ignore it. For the rest of this tutorial I will omit these errors from the output for brevity. Make three partitions, a fat32 partition for the image itself that is 5% larger than the size of the ISO, a 1G ext4 partition for persistence, and an extra fat32 partition for the rest of the space. Put the extra partition first so that other OSes you might use the key with will be able to use it. Use relative positioning from the end of the device to make it easy to put all of the rest of the space in the first partition. Finally, flag the live image partition bootable. 
(parted) mkpart primary fat32 1 -1897M
(parted) mkpart primary ext4 -1897M -897M
(parted) mkpart primary fat32 -897M -0
(parted) set 3 boot on
(parted) quit
Information: You may need to update /etc/fstab.
Unplug the USB key and plug it in again so that the kernel will re-read the updated partition table. If there were old filesystems on the key that were automounted, unmount them again. Now we re ready to make the new filesystems. 
$ umount /dev/sdX1
$ /sbin/mkdosfs -nEXTRA /dev/sdX1
mkdosfs 3.0.16 (01 Mar 2013)
$ /sbin/mkfs.ext4 -q -Lpersistence /dev/sdX2
$ /sbin/mkdosfs -nLXDE /dev/sdX3
mkdosfs 3.0.16 (01 Mar 2013)
Make the USB key bootable Use install-mbr to install an MBR on the key and syslinux to install the bootloader to boot into your live image partition. 
$ /sbin/install-mbr /dev/sdX
$ syslinux -i /dev/sdX3
Mount the partitions Unplug the key and plug it in again so the new partitions will be automounted. Observe where the partitions are mounted with df. 
$ df
...
/dev/sdX1        6204568         4   6204564   1% /media/EXTRA
/dev/sdX3         831888         4    831884   1% /media/LXDE
/dev/sdX2         944120      1204    894124   1% /media/persistence
Extract the ISO contents to the key To continue preparing this key as an unprivileged user, use p7zip to extract the ISO. You may alternatively mount the ISO loopback and extract it that way, but you must be root to do that. 
$ cd /media/LXDE
$ 7z x ~/debian-live-7.0.0-amd64-lxde-desktop.iso
...
Extracting  live/filesystem.packages-remove
...
When live/filesystem.squashfs is extracted (right after live/filesystem.packages-remove) it will take the longest time of all of the files to extract, as it contains the whole live filesystem. Be patient and eventually it and all other files on the image will finish extracting. 
...
Everything is Ok
Folders: 245
Files: 370
Size:       892183367
Compressed: 895483904
Modify the bootloader configuration files The syslinux bootloader configuration directory and files within it are named isolinux when installed on an ISO. You need to rename the directory and two files changing isolinux to syslinux so the bootloader will find them on your fat32 live image partition. 
$ mv isolinux syslinux
$ mv syslinux/isolinux.cfg syslinux/syslinux.cfg
$ mv syslinux/isolinux.bin syslinux/syslinux.bin
Enable full persistence Next, append persistence to the live boot parameters and turn on full persistence by putting / union in a persistence.conf file in the persistence partition. 
$ sed -i 's/\(append boot=.*\)$/\1 persistence/' syslinux/live.cfg
$ cd /media/persistence
$ echo / union > persistence.conf
Reboot into the live system Now your USB key is ready to boot with full persistence enabled. Have fun!

22 June 2013

Justus Winter: My first week and some promising though hacky results

Hi Planet Debian :) my name is Justus Winter, I'm a computer science student at the University of Hamburg. I'm participating in the GSOC 2013, my task is to make Debian/Hurd systems boot using the same initialization scripts Debian/Linux uses. I spent the community bonding period with some gsoc related administrative tasks, updating my development machine and getting reacquainted with the Hurd system. It was nice to see how far Debian/Hurd has come since my last "serious" encounter with it. I've setup a continuous build solution on one of my Hurd VMs and prepared a package repository. I'll post details once I have some packages to share. I built the hurd, sysvinit and ifupdown packages and dived into the interesting world of literate programming. I'm going to write about that when I start looking into the ifupdown related issues. But most importantly I started writing a tool to automate all the tasks I would do during my work that could possibly be automated. As I'm working on the boot process, it makes sense to use some kind of virtualization software so I can work in the comfort of my usual development environment. My virtualization solution of choice is qemu and I actually wrote a qemu based black-box test solution for a former employer once, so I already was familiar with qemu and wrapping my own code around it. Without further ado, here it is: https://bitbucket.org/teythoon/hurdtest Provided that you have golang installed, you can compile it easily: 
teythoon@europa /tmp % export GOPATH= pwd 
teythoon@europa /tmp % go get bitbucket.org/teythoon/hurdtest
teythoon@europa /tmp % bin/hurdtest
Usage: hurdtest [options] <command> [<arguments...>]
Currently it has the following subcommands:
test
Starts a VM, logs in as root, tests the interaction with the VM, verifies that certain VFS locations are writable, tests the network connectivity, the DNS resolution and installs and runs the hello package.
update
Executes aptitude update && aptitude full-upgrade, reboots the system and runs the test subcommand. If successful, it commits the changes to the virtual disk and creates a snapshot.
overlay
Takes as argument the path to a directory that is copied to the VM, reboots the machine and runs the test subcommand.
The update command is nice because it allows me to keep the VM up to date while ensuring that the image remains usable, with respect to what the test command considers a usable state of a Debian system. I'd really like to get some suggestions on how to test the "health" of a Debian system. You can look at how the tests are implemented in test.go, it's quite easy to add more and more interesting tests, and you don't even have to clean up after them or worry about any side-effects as the changes are not committed to the virtual disk. This is hurdtest doing an update: 
teythoon@europa ~/repos/hurd/hurd-tests % bin/hurdtest update
2013/06/22 19:59:45 starting kvm [-drives file=image.qcow2,cache=writeback,snapshot=on -serial unix:/tmp/285813369/serial -monitor unix:/tmp/285813369/monitor -nographic -m 512]
2013/06/22 19:59:45 Waiting for grub...
2013/06/22 19:59:46 Waiting for login prompt...
2013/06/22 20:00:05 Updating system
2013/06/22 20:02:52 Get: 1 http://www.local unstable InRelease [8279 B]
Get: 2 http://http.debian.net sid InRelease [205 kB]
Get: 3 http://ftp.debian-ports.org unreleased InRelease [23.9 kB]
[...]
Setting up perl (5.14.2-21) ...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Reading task descriptions...
2013/06/22 20:02:52 Update successful. Snapshotting system...
2013/06/22 20:02:53 Rebooting system...
2013/06/22 20:02:54 Waiting for grub...
2013/06/22 20:02:59 Waiting for login prompt...
2013/06/22 20:03:16 Running main.testTrue...
2013/06/22 20:03:16 Running main.testFalse...
2013/06/22 20:03:17 Running main.testClean...
[...]
2013/06/22 20:03:18 Running main.testNetwork...
2013/06/22 20:03:18 Running main.testResolver...
2013/06/22 20:03:20 Running main.testAptitudeInstall...
2013/06/22 20:03:46 Test successful. Reverting system...
2013/06/22 20:03:48 Halting system...
2013/06/22 20:03:53 Committing changes...
2013/06/22 20:03:57 Creating snapshot...
2013/06/22 20:04:04 All done.
bin/hurdtest update  157.31s user 69.71s system 87% cpu 4:18.94 total
The overlay command implements my work flow. I can edit scripts or drop hacked up executables onto my VM and see the results without any manual intervention. There's the -interactive flag that does this in a loop, avoiding the VM startup time using qemu snapshots. With this I can apply an overlay and see the results of the test suite in about a minute on my system. From early experiments I knew that simply running /etc/rcS.d/S* prevented the system from booting correctly, so I figured my first task should be to figure out what went wrong there. Using my tool (and developing it in that process) and many iterations later I came up with a surprisingly small diff of scripts in the initscripts and utilities shipped with the hurd package. For your convenience here is a tarball that you can extract to the root of your Debian/Hurd installation to reproduce my results (of course I invite you to use my tool for that purpose): 
% bin/hurdtest overlay test
2013/06/22 21:35:31 starting kvm [-drive file=image.qcow2,cache=writeback,snapshot=on -serial unix:/tmp/345188134/serial -monitor unix:/tmp/345188134/monitor -nographic -m 512]
2013/06/22 21:35:31 Waiting for grub...
2013/06/22 21:35:31 Waiting for login prompt...
2013/06/22 21:35:48 Creating tarball...
./
./etc/
./etc/hurd/
./etc/hurd/rc
./etc/init.d/
./etc/init.d/checkroot.sh
./etc/init.d/mountall.sh
./runme
./lib/
./lib/init/
./lib/init/mount-functions.sh
./sbin/
./sbin/swapon
./bin/
./bin/mount
2013/06/22 21:35:48 Extracting overlay...
2013/06/22 21:35:49 Rebooting system...
2013/06/22 21:35:49 Waiting for grub...
2013/06/22 21:35:55 Writing log to /tmp/bootlog-802006344
2013/06/22 21:35:55 Waiting for login prompt...
2013/06/22 21:36:20 Booting successful.
2013/06/22 21:36:20 Running main.testTrue...
2013/06/22 21:36:21 Running main.testFalse...
2013/06/22 21:36:21 Running main.testClean...
2013/06/22 21:36:21 Running anonymous function...
2013/06/22 21:36:21 Running anonymous function...
2013/06/22 21:36:21 Running anonymous function...
2013/06/22 21:36:21 Running anonymous function...
2013/06/22 21:36:21 Running anonymous function...
2013/06/22 21:36:22 Running anonymous function...
2013/06/22 21:36:22 Running main.testUname...
2013/06/22 21:36:22 Running main.testNetwork...
2013/06/22 21:36:22 Running main.testResolver...
2013/06/22 21:36:23 Running main.testAptitudeInstall...
bin/hurdtest overlay ../hurd-tests-old/test  44.20s user 20.01s system 82% cpu 1:17.94 total
And here is the boot log: 
Loading GNU Mach ...
Loading the Hurd ...
GNU Mach 1.3.99-486
AT386 boot: physical memory map from 0x0 to 0x9f400
AT386 boot: physical memory map from 0x100000 to 0x1fffe000
AT386 boot: physical memory from 0x0 to 0x1fffe000
Enabling FXSR
pcibios_init : BIOS32 Service Directory structure at 0xfcff0
pcibios_init : BIOS32 Service Directory entry at 0xfc7ba
pcibios_init : PCI BIOS revision 2.10 entry at 0xfc78c
Probing PCI hardware.
ide: Intel 82371 PIIX3 (dual FIFO) DMA Bus Mastering IDE
    Controller on PCI bus 0 function 9
ide: BM-DMA feature is not enabled (BIOS), enabling
    ide0: BM-DMA at 0xc100-0xc107
    ide1: BM-DMA at 0xc108-0xc10f
hd0: got CHS=762/128/63 CTL=c8 from BIOS
hd0: QEMU HARDDISK, 3001MB w/256kB Cache, CHS=762/128/63, DMA
hd2: QEMU DVD-ROM, ATAPI CDROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
Floppy drive(s): fd0 is 1.44M
FDC 0 is a S82078B
probing scsi 0/22: advansys probing scsi 1/22: BusLogic probing scsi 2/22: UltraStor 14F/34F rev. 4.33.00  probing scsi 3/22: UltraStor 14F/24F/34F probing scsi 4/22: Adaptec 152x SCSI driver; $Revision: 1.1 $ probing scsi 5/22: Adaptec 1542 Invalid address for shpnt with 1542.
Invalid address for shpnt with 1542.
probing scsi 6/22: Adaptec 174x (EISA) probing scsi 7/22:  probing scsi 8/22: Always IN2000 probing scsi 9/22:  probing scsi 10/22:  probing scsi 11/22:  probing scsi 12/22: Trantor T128/T128F/T228 probing scsi 13/22: DTC 3180/3280  probing scsi 14/22: Tekram DC390(T) V1.11 Feb-05-1997 probing scsi 15/22:  probing scsi 16/22: ncr53c8xx - revision 2.5f.1 probing scsi 17/22: EATA (Extended Attachment) PIO driver probing scsi 18/22: Western Digital WD-7000 Failed initialization of WD-7000 SCSI card!
probing scsi 19/22: EATA/DMA 2.0x rev. 4.33.00  probing eata on 1f0EATA0: address 0x1f0 in use, skipping probe.
probing eata on 1c88probing eata on 2c88probing eata on 3c88probing eata on 4c88probing eata on 5c88probing eata on 6c88probing eata on 7c88probing eata on 8c88probing eata on 9c88probing eata on ac88probing eata on bc88probing eata on cc88probing eata on dc88probing eata on ec88probing eata on fc88probing eata on 170EATA0: address 0x170 in use, skipping probe.
probing eata on 230probing eata on 330probing scsi 20/22: AM53C974 probing scsi 21/22: Iomega parport ZIP drive ppa: Version 1.42
ppa: Probing port 03bc
ppa: Probing port 0378
ppa:     SPP port present
ppa:     PS/2 bidirectional port present
ppa: Probing port 0278
done
scsi : 0 hosts.
scsi : detected total.
Partition check (DOS partitions):
 hd0: hd0s1 hd0s2 < hd0s5 >
lpr0: at atbus0, port = 378, spl = 6, pic = 7.
module 0: ext2fs --readonly --multiboot-command-line=$ kernel-command-line  --host-priv-port=$ host-port  --device-master-port=$ device-port  --exec-server-task=$ exec-task  -T typed $ root  $(task-create) $(task-resume) module 1: exec /hurd/exec $(exec-task=task-create)                                                                                                                                                                           2 multiboot modules                                                                                                                                                                                                                    task loaded: ext2fs --readonly --multiboot-command-line=root=device:hd0s1 console=com0 --host-priv-port=1 --device-master-port=2 --exec-server-task=3 -T typed device:hd0s1
task loaded: exec /hurd/exec
start ext2fs: Hurd server bootstrap: ext2fs[device:hd0s1] exec init proc auth
Using makefile-style concurrent boot in runlevel S.
ERROR: could not open /proc/stat: No such file or directory
Activating swap...done.
Checking root file system...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
ext2fs_check_if_mount: Can't check if filesystem is mounted due to missing mtab file while determining whether /dev/hd0s1 is mounted.
/dev/hd0s1: clean, 44328/181056 files, 288266/723200 blocks
done.
Cleaning up temporary files... /tmp.
mount: /run: Unknown device or filesystem
mount: /run/lock: Unknown device or filesystem
mount: /proc: Unknown device or filesystem
mount: /run/shm: Unknown device or filesystem
/etc/rcS.d/S06mtab.sh: 48: /etc/rcS.d/S06mtab.sh: cannot open /proc/mounts: No such file
Activating lvm and md swap...(default pager): Already paging to partition hd0s5!
done.
Checking file systems...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
done.
Mounting local filesystems...mount: invalid option -- 'O'
Try  mount --help' or  mount --usage' for more information.
failed.
Activating swapfile swap...(default pager): Already paging to partition hd0s5!
done.
df: Warning: cannot read table of mounted file systems: No such file or directory
Cleaning up temporary files....
Configuring network interfaces...inetutils-ifconfig: invalid arguments
ifup: failed to open pid file /run/network/ifup-/dev/eth0.pid: No such file or directory
Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
Listening on Socket//dev/eth0
Sending on   Socket//dev/eth0
DHCPDISCOVER on /dev/eth0 to 255.255.255.255 port 67 interval 5
DHCPREQUEST on /dev/eth0 to 255.255.255.255 port 67
DHCPOFFER from 10.0.2.2
DHCPACK from 10.0.2.2
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
bound to 10.0.2.15 -- renewal in 36884 seconds.
done.
Cleaning up temporary files....
Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix.
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Starting deferred execution scheduler: atd.
Starting periodic command scheduler: cron.
Starting system message bus: dbusFailed to set socket option"/var/run/dbus/system_bus_socket": Protocol not available.
Starting OpenBSD Secure Shell server: sshd.
GNU 0.3 (debian) (console)
login:
I know it isn't that pretty, but it yields a working Debian/Hurd system that is more like a Debian/Linux system. For example /dev/shm and /run ,/* are connected to a tmpfs translator. I think this is a rather encouraging first result. I added my findings to the list of open issues. Another nice thing about hurdtest is that it is not even Hurd specific (only a tiny bit GNU Mach specific). I'm going to make it work nicely with Debian/Linux and Debian/kFreeBSD, since this is a very handy tool for anyone hacking on the Debian boot process (and with some changes for any OS). If anyone tries it out, I'd love to get feedback and patches. The tool requires the guest to be configured to use the first serial console for grub, the kernel and the userspace. The password for the user root must be empty. The guest has to have a working network connection setup at boot time. I will provide a suitable image as soon as I devise a good way to zero-fill the free space on the virtual disk. Is there any better way than to fill the disk with a huge file containing zeros? Without doing that the image is too large even compressed with xz. So this was my first week working for Debian. Next week I will focus on getting patches for the hurd and initscripts package in shape, I need to add some functionality to the mount utility shipped with the hurd package. It provides a mount-like experience on the Hurd, on which binding a translator (e. g. /hurd/ext2fs) to a filesystem node is an operation that does not involve the kernel (GNU Mach) at all, and is in fact a operation that any unprivileged user may perform. That is one of the nice features the Hurd has, it empowers its users.

Next.

Previous.