I have been using Replicant on the Samsung SIII I9300 for over two years. I have written before on taking a backup of the phone using rsync but recently I automated my setup as described below. This work was prompted by a screen accident with my phone that caused it to die, and I noticed that I hadn t taken regular backups. I did not lose any data this time, since typically all content I create on the device is immediately synchronized to my clouds. Photos are uploaded by the ownCloud app, SMS Backup+ saves SMS and call logs to my IMAP server, and I use DAVDroid for synchronizing contacts, calendar and task lists with my instance of ownCloud. Still, I strongly believe in regular backups of everything, so it was time to automate this. For my use-case, taking backups of the phone whenever I connect it to one of my laptops is sufficient. I typically connect it to my laptops for charging at least every other day. My laptops are all running Debian, but this should be applicable to most modern GNU/Linux system. This is not Replicant-specific, although you need a rooted phone. I thought that automating this would be simple, but I got to learn the ins and outs of systemd and udev in the process and this ended up taking the better part of an evening. I started out adding an udev rule and a small script, thinking I could invoke the backup process from the udev rule. However rsync would magically die after running a few seconds. After an embarrassing long debugging session, finally I found someone with a similar problem which led me to a nice writeup on the topic of running long-running services on udev events. I created a file /etc/udev/rules.d/99-android-backup.rules with the following content:

ACTION=="add", SUBSYSTEMS=="usb", ENV ID_SERIAL_SHORT =="323048a5ae82918b", TAG+="systemd", ENV SYSTEMD_WANTS +="android-backup@$env ID_SERIAL_SHORT .service"
ACTION=="add", SUBSYSTEMS=="usb", ENV ID_SERIAL_SHORT =="4df9e09c25e75f63", TAG+="systemd", ENV SYSTEMD_WANTS +="android-backup@$env ID_SERIAL_SHORT .service"

The serial numbers correspond to the device serial numbers of the two devices I wish to backup. The adb devices command will print them for you, and you need to replace my values with the values from your phones. Next I created a systemd service to describe a oneshot service. The file /etc/systemd/system/android-backup@.service have the following content:

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/android-backup %I

The at-sign ( @ ) in the service filename signal that this is a service that takes a parameter. I m not enough of an udev/systemd person to explain these two files using the proper terminology, but at least you can pattern-match and follow the basic idea of them: the udev rule matches the devices that I m interested in (I don t want this to happen to all random Android devices I attach, hence matching against known serial numbers), and it causes a systemd service with a parameter to be started. The systemd service file describe the script to run, and passes on the parameter. Now for the juicy part, the script. I have /usr/local/sbin/android-backup with the following content.

#!/bin/bash
DIRBASE=/var/backups/android
export ANDROID_SERIAL="$1"
exec 2>&1   logger
if ! test -d "$DIRBASE-$ANDROID_SERIAL"; then
    echo "could not find directory: $DIRBASE-$ANDROID_SERIAL"
    exit 1
fi
set -x
adb wait-for-device
adb root
adb wait-for-device
adb shell printf "address 127.0.0.1\nuid = root\ngid = root\n[root]\n\tpath = /\n" \> /mnt/secure/rsyncd.conf
adb shell rsync --daemon --no-detach --config=/mnt/secure/rsyncd.conf &
adb forward tcp:6010 tcp:873
sleep 2
rsync -av --delete --exclude /dev --exclude /acct --exclude /sys --exclude /proc rsync://localhost:6010/root/ $DIRBASE-$ANDROID_SERIAL/
: rc $?
adb forward --remove tcp:6010
adb shell rm -f /mnt/secure/rsyncd.conf

This script warrant more detailed explanation. Backups are placed under, e.g., /var/backups/android-323048a5ae82918b/ for later off-site backup (you do backup your laptop, right?). You have to manually create this directory, as a safety catch to not wildly rsync data into non-existing directories. The script logs everything using syslog, so run a tail -F /var/log/syslog& when setting this up. You may want to reduce verbosity of rsync if you prefer (replace rsync -av with rsync -a). The script runs adb wait-for-device which you rightly guessed will wait for the device to settle. Next adb root is invoked to get root on the device (reading all files from the system naturally requires root). It takes some time to switch, so another wait-for-device call is needed. Next a small rsyncd configuration file is created in /mnt/secure/rsyncd.conf on the phone. The file tells rsync do listen on localhost, run as root, and use / as the path. By default, rsyncd is read-only so the host will not be able to upload any data over rsync, just read data out. Next rsync is started on the phone. The adb forward command forwards port 6010 on the laptop to port 873 on the phone (873 is the default rsyncd port). Unfortunately, setting up the TCP forward appears to take some time, and adb wait-for-device will not wait for that to complete, hence an ugly sleep 2 at this point. Next is the rsync invocation itself, which just pulls in everything from the phone to the laptop, excluding some usual suspects. The somewhat cryptic : rc $? merely logs the exit code of the rsync process into syslog. Finally we clean up the TCP forward and remove the rsyncd.conf file that was temporarily created. This setup appears stable to me. I can plug in a phone and a backup will be taken. I can even plug in both my devices at the same time, and they will run at the same time. If I unplug a device, the script or rsync will error out and systemd cleans up. If anyone has ideas on how to avoid the ugly temporary rsyncd.conf file or the ugly sleep 2, I m interested. It would also be nice to not have to do the adb root dance, and instead have the phone start the rsync daemon when connecting to my laptop somehow. TCP forwarding might be troublesome on a multi-user system, but my laptops aren t. Killing rsync on the phone is probably a good idea too. If you have ideas on how to fix any of this, other feedback, or questions, please let me know!

Finally, bzed.de is back online and I m planning to start blogging again! Part of the reason why I became inactive was the usage of ikiwiki, which is great, but at end unnecessarily complicated. So I ve migrated by page to gohugo.io - a static website generator, written in go. Hugo has an active community and it is easy to create themes for it or to enhance it. Also it is using plain Markdown syntax instead of special ikiwiki syntax mixed into it - should make it easy to migrate away again if necessary. In case somebody else would like to convert from ikiwiki to Hugo, here is the script I ve hacked together to migrate my old blog posts.

#!/bin/bash
find . -type f -name '*.mdwn'   while read i; do
    tmp= mktemp 
     
        echo '+++'
        slug="$(echo $i   sed 's,.*/,,;s,\.mdwn$,,')"
        echo "slug = \"$ slug \""
        echo "title = \"$(echo $i   sed 's,.*/,,;s,\.mdwn$,,;s,_, ,g;s/\b\(.\)/\u\1/;s,debian,Debian,g')\""
        if grep -q 'meta updated' $i; then
            echo -n 'date = '
            sed '/meta updated/!d;/.*meta updated.*/s,.*=",,;s,".*,,;s,^,",;s,$,",' $i
        else
            echo -n 'date = '
            git log --diff-filter=A --follow --format='"%aI"' -1 -- $i
        fi
        if grep -q '\[\[!tag' $i; then
            echo -n 'tags ='
            sed '/\[\[!tag/!d;s,[^ ]*tag ,,;s,\]\],,;s,\([^ ]*\),"\1",g;s/ /,/g;s,^,[,;s,$,],' $i
        fi
        echo 'categories = ["linux"]'
        echo 'draft = false'
        echo '+++'
        echo ''
        sed -e '/\[\[!tag/d' \
            -e '/meta updated/d' \
            -e '/\[\[!plusone *\]\]/d' \
            -e 's,\[\[!img files[0-9/]*/\([^ ]*\) alt="\([^"]*\).*,![\2](../\1),g' \
            -e 's,\[\([^]]*\)\](\([^)]*\)),[\1](\2),g' \
            -e 's,\[\[\([^ ]*\) \([^]]*\)\]\],[\1](\2),g' \
            $i
      > $tmp
    #cat $tmp; rm $tmp 
    mv $tmp  echo $i   sed 's,\.mdwn,.md,g' 
done

For the planet Debian readers - only linux related posts will show up on the planet. If you are interested in my mountain activities and other things I post, please follow my blog on bzed.de directly.

This is a burst of notes that I wrote in an e-mail in June when asked about it, and I'm not going to have any better steps since I don't remember even that amount as back then. I figured it's better to have it out than not.

So... if you want to use LUKS In-Place Conversion Tool, the notes below on converting a shipped-with-Ubuntu Dell XPS 13 Developer Edition (2015 Intel Broadwell model) may help you. There were a couple of small learnings to be had...

The page http://www.johannes-bauer.com/linux/luksipc/ itself is good and without errors, although funnily uses reiserfs as an example. It was only a bit unclear why I did save the initial_keyfile.bin since it was then removed in the next step (I guess it's for the case you want to have a recovery file hidden somewhere in case you forget the passphrase).

For using the tool I booted from a 14.04.2 LTS USB live image and operated there, including downloading and compiling luksipc in the live session. The exact reason of resizing before luksipc was a bit unclear to me at first so I simply indeed resized the main rootfs partition and left unallocated space in the partition table.

Then finally I ran ./luksipc -d /dev/sda4 etc.

I realized I want /boot to be on an unencrypted partition to be able to load the kernel + initrd from grub before entering into LUKS unlocking. I couldn't resize the luks partition anymore since it was encrypted... So I resized what I think was the empty small DIAGS partition (maybe used for some system diagnostic or something, I don't know), or possibly the next one that is the actual recovery partition one can reinstall the pre-installed Ubuntu from. And naturally I had some problems because it seems vfatresize tool didn't do what I wanted it to do and gparted simply crashed when I tried to use it first to do the same. Anyway, when done with getting some extra free space somewhere, I used the remaining 350MB for /boot where I copied the rootfs's /boot contents to.

After adding the passphrase in luks I had everything encrypted etc and decryptable, but obviously I could only access it from a live session by manual cryptsetup luksOpen + mount /dev/mapper/myroot commands. I needed to configure GRUB, and I needed to do it with the grub-efi-amd64 which was a bit unfamiliar to me. There's also grub-efi-amd64-signed I have installed now but I'm not sure if it was required for the configuration. Secure boot is not enabled by default in BIOS so maybe it isn't needed.

I did GRUB installation I think inside rootfs chroot where I also mounted /dev/sda6 as /boot (inside the rootfs chroot), ie mounted dev, sys with -o bind to under the chroot (from outside chroot) and mount -t proc proc proc too. I did a lot of trial and effort so I surely also tried from outside the chroot, in the live session, using some parameters to point to the mounted rootfs's directories...

I needed to definitely install cryptsetup etc inside the encrypted rootfs with apt, and I remember debugging for some time if they went to the initrd correctly after I executed mkinitramfs/update-initramfs inside the chroot.

At the end I had grub asking for the password correctly at bootup. Obviously I had edited the rootfs's /etc/fstab to include the new /boot partition, I changed / to be "UUID=/dev/mapper/myroot / ext4 errors=remount-ro 0 ", kept /boot/efi as coming from the /dev/sda1 and so on. I had also added "myroot /dev/sda4 none luks" to /etc/crypttab. I seem to also have GRUB_CMDLINE_LINUX="cryptdevice=/dev/sda4:myroot root=/dev/mapper/myroot" in /etc/default/grub.

Writing programs is fun, but making them fast can be a pain. Python programs are no exception to that, but the basic profiling toolchain is actually not that complicated to use. Here, I would like to show you how you can quickly profile and analyze your Python code to find what part of the code you should optimize. What's profiling? Profiling a Python program is doing a dynamic analysis that measures the execution time of the program and everything that compose it. That means measuring the time spent in each of its functions. This will give you data about where your program is spending time, and what area might be worth optimizing. It's a very interesting exercise. Many people focus on local optimizations, such as determining e.g. which of the Python functions range or xrange is going to be faster. It turns out that knowing which one is faster may never be an issue in your program, and that the time gained by one of the functions above might not be worth the time you spend researching that, or arguing about it with your colleague. Trying to blindly optimize a program without measuring where it is actually spending its time is a useless exercise. Following your guts alone is not always sufficient. There are many types of profiling, as there are many things you can measure. In this exercise, we'll focus on CPU utilization profiling, meaning the time spent by each function executing instructions. Obviously, we could do many more kind of profiling and optimizations, such as memory profiling which would measure the memory used by each piece of code something I talk about in The Hacker's Guide to Python. cProfile Since Python 2.5, Python provides a C module called cProfile which has a reasonable overhead and offers a good enough feature set. The basic usage goes down to:
Though you can also run a script with it, which turns out to be handy:
This prints out all the function called, with the time spend in each and the number of times they have been called. Advanced visualization with KCacheGrind While being useful, the output format is very basic and does not make easy to grab knowledge for complete programs. For more advanced visualization, I leverage KCacheGrind. If you did any C programming and profiling these last years, you may have used it as it is primarily designed as front-end for Valgrind generated call-graphs. In order to use, you need to generate a cProfile result file, then convert it to KCacheGrind format. To do that, I use pyprof2calltree.
And the KCacheGrind window magically appears!

Concrete case: Carbonara optimization I was curious about the performances of Carbonara, the small timeserie library I wrote for Gnocchi. I decided to do some basic profiling to see if there was any obvious optimization to do. In order to profile a program, you need to run it. But running the whole program in profiling mode can generate a lot of data that you don't care about, and adds noise to what you're trying to understand. Since Gnocchi has thousands of unit tests and a few for Carbonara itself, I decided to profile the code used by these unit tests, as it's a good reflection of basic features of the library. Note that this is a good strategy for a curious and naive first-pass profiling. There's no way that you can make sure that the hotspots you will see in the unit tests are the actual hotspots you will encounter in production. Therefore, a profiling in conditions and with a scenario that mimics what's seen in production is often a necessity if you need to push your program optimization further and want to achieve perceivable and valuable gain. I activated cProfile using the method described above, creating a cProfile.Profile object around my tests (I actually started to implement that in testtools). I then run KCacheGrind as described above. Using KCacheGrind, I generated the following figures.

The test I profiled here is called test_fetch and is pretty easy to understand: it puts data in a timeserie object, and then fetch the aggregated result. The above list shows that 88 % of the ticks are spent in set_values (44 ticks over 50). This function is used to insert values into the timeserie, not to fetch the values. That means that it's really slow to insert data, and pretty fast to actually retrieve them. Reading the rest of the list indicates that several functions share the rest of the ticks, update, _first_block_timestamp, _truncate, _resample, etc. Some of the functions in the list are not part of Carbonara, so there's no point in looking to optimize them. The only thing that can be optimized is, sometimes, the number of times they're called.

The call graph gives me a bit more insight about what's going on here. Using my knowledge about how Carbonara works, I don't think that the whole stack on the left for _first_block_timestamp makes much sense. This function is supposed to find the first timestamp for an aggregate, e.g. with a timestamp of 13:34:45 and a period of 5 minutes, the function should return 13:30:00. The way it works currently is by calling the resample function from Pandas on a timeserie with only one element, but that seems to be very slow. Indeed, currently this function represents 25 % of the time spent by set_values (11 ticks on 44). Fortunately, I recently added a small function called _round_timestamp that does exactly what _first_block_timestamp needs that without calling any Pandas function, so no resample. So I ended up rewriting that function this way:
And then I re-run the exact same test to compare the output of cProfile.

The list of function seems quite different this time. The number of time spend used by set_values dropped from 88 % to 71 %.

The call stack for set_values shows that pretty well: we can't even see the _first_block_timestamp function as it is so fast that it totally disappeared from the display. It's now being considered insignificant by the profiler. So we just speed up the whole insertion process of values into Carbonara by a nice 25 % in a few minutes. Not that bad for a first naive pass, right?

On the final semester of my MSc program at Columbia SEAS, I was lucky enough to be able to attend a seminar course taught by Alfio Gliozzo entitled Q&A with IBM Watson. A significant part of the course is dedicated to learning how to leverage the services and resources available on the Watson Developer Cloud. This post describes the course project my team developed, the PaperTrail application.

Project Proposal Create an application to assist in the development of future academic papers. Based on a paper s initial proposal, Paper Trail predicts publications to be used as references or acknowledgement of prior art and provides a trend analysis of major topics and methods. The objective is to speed the discovery of relevant papers early in the research process, and allow for early assessment of the depth of prior research concerning the initial proposal.

Meet the Team Wesley Bruning, Software Engineer, MSc. in Computer Science Xavier Gonzalez, Industrial Engineer, MSc. in Data Science Juliana Louback, Software Engineer, MSc. in Computer Science Aaron Zakem, Patent Attorney, MSc. in Computer Science

Prior Art A significant amount of attention has been given to this topic over the past few decades. The table below shows the work the team deemed most relevant due to recency, accuracy and similarity of functionality. The variation in accuracy displayed is a result of experimentation with different dataset sizes and algorithm variations. More information and details can be found in the prior art report. The main differential of PaperTrail is providing a form of access to the citation prediciton and trend analysis algorithm. With the exception of the project by McNee et al., these algorithmns aren t currently available for general use. The application on researchindex.net is open to use but its objective is to rank publications and authors for given topics.

Algorithm Citation Prediction: PaperTrail builds on the work done by Wolski s team in Fall 2014. This algorithmn builds a reference graph used to define research communities, with an associated vector of topic scores generated by an LDA model. The papers in each research community are then ranked by importance within the community with a custom ranking algorithm. When a target document is given to algorithm as input, the LDA model is used to generate a vector of topics that are present in the document. The communities with the most similar topic vectors are selected and the publications within these communities with highest rank and greatest similarity to the input document are recommended as references. A more detailed description can be found here. Trend Analysis: Initially, the idea was to use the AlchemyData News API to obtain statistics pertaining to the amount of publications on a given topic over time. However, with the exception of buzz-words (i.e. big data ), many more specialized topics appeared very infrequently in news articles, if at all. This isn t entirely surprising given the target audience of PaperTrail. As a work around, we use the Alchemy Language API to extract keywords from the abstracts in the dataset, in addition to relevance scores. The PaperTrail database could then be queried for entry counts for a given year and keyword to provide an indication of publication trends in academia. Note that the Alchemy Language API extracts multiple-word keywords as well as single words.

Data To maintain consistency with Wolski s project, we are using the DBLP data as made available on aminer.org. The DBLP-Citation-network V5 dataset contains 1,572,277 entries; we are limited to the use of entries that contain both abstracts and citations, bringing the dataset size down to 265,865 entries.

Architecture A high-level visualization of the project architecture is displayed below. Before launching PaperTrail, it s necessary to train Wolski s algorithm offline. Currently any documentation with regard to the performance of said algorithm is unavailable; the PaperTrail project will include an evaluation phase and report the findings made. The PaperTrail app and database will be hosted on the Bluemix Platform.

Status Report Phases completed:

• Project design
• Prior art research
• Data cleansing
• Development and deployment of an alpha version of the PaperTrail app

Phases under development:

• Algorithm training and evaluation
• Keyword extraction
• MapReduce of publication frequency by year and topic
• Data visualization component

Firefox 42, which was released last week, introduced a new feature in its Private Browsing mode: tracking protection. If you are interested in how this list is put together and then used in Firefox, this post is for you.

Safe Browsing lists There are many possible ways to download URL lists to the browser and check against that list before loading anything. One of those is already implemented as part of our malware and phishing protection. It uses the Safe Browsing v2.2 protocol. In a nutshell, the way that this works is that each URL on the block list is hashed (using SHA-256) and then that list of hashes is downloaded by Firefox and stored into a data structure on disk:

• ~/.cache/mozilla/firefox/XXXX/safebrowsing/mozstd-track* on Linux
• ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/mozstd-track* on Mac
• C:\Users\XXXX\AppData\Local\mozilla\firefox\profiles\XXXX\safebrowsing\mozstd-track* on Windows

This sbdbdump script can be used to extract the hashes contained in these files and will output something like this:

$ ~/sbdbdump/dump.py -v .
- Reading sbstore: mozstd-track-digest256
[mozstd-track-digest256] magic 1231AF3B Version 3 NumAddChunk: 1 NumSubChunk: 0 NumAddPrefix: 0 NumSubPrefix: 0 NumAddComplete: 1696 NumSubComplete: 0
[mozstd-track-digest256] AddChunks: 1445465225
[mozstd-track-digest256] SubChunks:
...
[mozstd-track-digest256] addComplete[chunk:1445465225] e48768b0ce59561e5bc141a52061dd45524e75b66cad7d59dd92e4307625bdc5
...
[mozstd-track-digest256] MD5: 81a8becb0903de19351427b24921a772

The name of the blocklist being dumped here (mozstd-track-digest256) is set in the urlclassifier.trackingTable preference which you can find in about:config. The most important part of the output shown above is the addComplete line which contains a hash that we will see again in a later section.

List lookups Once it's time to load a resource, Firefox hashes the URL, as well as a few variations of it, and then looks for it in the local lists. If there's no match, then the load proceeds. If there's a match, then we do an additional check against a pairwise allowlist. The pairwise allowlist (hardcoded in the urlclassifier.trackingWhitelistTable pref) is designed to encode what we call "entity relationships". The list groups related domains together for the purpose of checking whether a load is first or third party (e.g. twitter.com and twimg.com both belong to the same entity). Entries on this list (named mozstd-trackwhite-digest256) look like this:

twitter.com/?resource=twimg.com

which translates to "if you're on the twitter.com site, then don't block resources from twimg.com. If there's a match on the second list, we don't block the load. It's only when we get a match on the first list and not the second one that we go ahead and cancel the network load. If you visit our test page, you will see tracking protection in action with a shield icon in the URL bar. Opening the developer tool console will expose the URL of the resource that was blocked:

The resource at "https://trackertest.org/tracker.js" was blocked because tracking protection is enabled.

Creating the lists The blocklist is created by Disconnect according to their definition of tracking. The Disconnect list is on their Github page, but the copy we use in Firefox is the copy we have in our own repository. Similarly the Disconnect entity list is from here but our copy is in our repository. Should you wish to be notified of any changes to the lists, you can simply subscribe to this Atom feed. To convert this JSON-formatted list into the binary format needed by the Safe Browsing code, we run a custom list generation script whenever the list changes on GitHub. If you run that script locally using the same configuration as our server stack, you can see the conversion from the original list to the binary hashes. Here's a sample entry from the mozstd-track-digest256.log file:

[m] twimg.com >> twimg.com/
[canonicalized] twimg.com/
[hash] e48768b0ce59561e5bc141a52061dd45524e75b66cad7d59dd92e4307625bdc5

and one from mozstd-trackwhite-digest256.log:

[entity] Twitter >> (canonicalized) twitter.com/?resource=twimg.com, hash a8e9e3456f46dbe49551c7da3860f64393d8f9d96f42b5ae86927722467577df

This in combination with the sbdbdump script mentioned earlier, will allow you to audit the contents of the local lists.

Serving the lists The way that the binary lists are served to Firefox is through a custom server component written by Mozilla: shavar. Every hour, Firefox requests updates from shavar.services.mozilla.com. If new data is available, then the whole list is downloaded again. Otherwise, all it receives in return is an empty 204 response. Should you want to play with it and run your own server, follow the installation instructions and then go into about:config to change these preferences to point to your own instance:

browser.trackingprotection.gethashURL
browser.trackingprotection.updateURL

Note that on Firefox 43 and later, these prefs have been renamed to:

browser.safebrowsing.provider.mozilla.gethashURL
browser.safebrowsing.provider.mozilla.updateURL

Learn more If you want to learn more about how tracking protection works in Firefox, you can find all of the technical details on the Mozilla wiki or you can ask questions on our mailing list. Thanks to Tanvi Vyas for reviewing a draft of this post.

db2twitter 0.2 was just released! If you missed my last post about it, db2twitter automatically extracts fields from your database, use them to feed a template of tweet and send the tweet. db2twitter is developed by and run for LinuxJobs.fr, the job board of th french-speaking Free Software and Opensource community.

• Github of db2twitter (star it if you like ;)
• Official documentation on readthedocs

The main new feature allows to send tweets only during authorized days/hours with the following content in your db2twitter.ini file :

[timer]
days=mon-fri,
hours=0-11,14-17,

• The line days=mon-fri specifies db2twitter can send tweets only from monday to friday.
• The line hours=0-11,14-17 specifies db2twitter can send tweets from midnight to 11AM (included) and from 2PM to 17PM (included)

Given the fact db2twitter will mostly be launched via cron, it allows sending tweets detected since the last db2twitter run during user-defined hours. And that s cool! db2twitter is coded in Python 3.4, uses SQlAlchemy (see supported database types) and Tweepy. The official documentation is available on readthedocs.

I m excited to see other people thinking about userspace-to-kernel attack surface reduction ideas. Theo de Raadt recently published slides describing Pledge. This uses the same ideas that seccomp implements, but with less granularity. While seccomp works at the individual syscall level and in addition to killing processes, it allows for signaling, tracing, and errno spoofing. As de Raadt mentions, Pledge could be implemented with seccomp very easily: libseccomp would just categorize syscalls. I don t really understand the presentation s mention of Optional Security , though. Pledge, like seccomp, is an opt-in feature. Nothing in the kernel refuses to run unpledged programs. I assume his point was that when it gets ubiquitously built into programs (like stack protector), it s effectively not optional (which is alluded to later as comprehensive applicability ~= mandatory mitigation ). Regardless, this sensible (though optional) design gets me back to his slide on seccomp, which seems to have a number of misunderstandings:

• A Turing complete eBPF program watches your program Strictly speaking, seccomp is implemented using a subset of BPF, not eBPF. And since BPF (and eBPF) programs are guaranteed to halt, it makes seccomp filters not Turing complete.
• Who watches the watcher? I don t even understand this. It s in the kernel. The kernel watches your program. Just like always. If this is a question of BPF program verification, there is literally a program verifier that checks various properties of the BPF program.
• seccomp program is stored elsewhere This, with the next statement, is just totally misunderstood. Programs using seccomp define their program in their own code. It s used the same way as the Pledge examples are shown doing.
• Easy to get desyncronized either program is updated As above, this just isn t the case. The only place where this might be true is when using seccomp on programs that were not written natively with seccomp. In that case, yes, desync is possible. But that s one of the advantages of seccomp s design: a program launcher (like minijail or systemd) can declare a seccomp filter for a program that hasn t yet been ported to use one natively.
• eBPF watcher has no real idea what the program under observation is doing I don t understand this statement. I don t see how Pledge would have a real idea either: they re both doing filtering. If we get AI out of our syscall filters, we re in serious trouble. :)

OpenBSD has some interesting advantages in the syscall filtering department, especially around sockets. Right now, it s hard for Linux syscall filtering to understand why a given socket is being used. Something like SOCK_DNS seems like it could be quite handy. Another nice feature of Pledge is the path whitelist feature. As it s still under development, I hope they expand this to include more things than just paths. Argument inspection is a weak point for seccomp, but under Linux, most of the arguments are ultimately exposed to the LSM layer. Last year I experimented with creating a seccomp LSM for path matching where programs could declare whitelists, similar to standard LSMs. So, yes, Linux could match this API on seccomp . It d just take some extensions to libseccomp to implement pledge(), as I described at the top. With OpenBSD doing a bunch of analysis work on common programs, it d be excellent to see this usable on Linux too. So far on Linux, only a few programs (e.g. Chrome, vsftpd) have bothered to do this using seccomp, and it could be argued that this is ultimately due to how fine grained it is.

2015, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.

Categories:

• Debian-Blog
• Tools

Keywords:

• apt-offline
• Offline APT Package Manager

Images:

Like:

It s now official: I m a pilot. This has been one of the most challenging, and also most rewarding, journeys I ve been on. It had its moments of struggle, moments of joy, moments of poetry. I wrote about the poetry of flying at night recently. Here is the story of my first landing on a grass runway, a few months ago. Grass

Where the air is so pure, the zephyrs so free,
The breezes so balmy and light,
That I would not exchange my home on the range
For all of the cities so bright. John A. Lomax

We are used to seeing planes in these massive palaces of infrastructure we call airports. We have huge parking garages, giant terminals, security lines hundreds of people deep, baggage carts, jetways, video screens, restaurants, and miles and miles of concrete. But most of the world s airports are not like that. A pilot of a small plane gets to see the big airports, sure, but we also get to use the smaller airports hidden in plain sight to most. Have you ever taken off from a strip of grass? As I told my flight instructor when I tried it for the first time, I know this will work, but somehow I will still be amazed if it actually does. I took off from a strip of grass not long ago. The airport there had one paved runway, and the rest were grass. Short runways, narrow runways, grass runways. No lights. No paint. No signs. No trucks, no jetways, nothing massive. In fact, no people. Just a mowed path and a couple of yellow or white markers. I taxied down the grass runway, being careful to never let the plane s wheels stop moving lest the nose gear get stuck in a pothole. I felt all the bumps in the ground as we moved. End of runway. Turn the plane around. A little bit of flap for more lift, full throttle, mind the centerline imaginary centerline, this time. It starts picking up speed, slower than usual, bump bump bump. Those buildings at the end of the runway are staring me down. More speed, and suddenly the runway feels smooth; it has enough lift to keep from falling into every bump. Then we lift off just a touch; I carefully keep the plane down until we pick up enough speed to ascend farther, then up we go. I keep a watchful eye on those buildings straight ahead and that water tower just slightly off to the one side. We climb over a lake as I watch that water tower pass safely below and to the side of the plane. It had worked, and I had a smile of amazement. With a half mile of grass, you really can go anywhere. Many times I had driven within half a mile of that runway, but never seen it. Never wondered where people go after using it. Never realizing that, although it s a 45-minute drive from my house, it s really pretty close. Never understanding that where people go after taking off from that runway is everywhere .

Ok, I have been an iPhone user since I moved to Japan and got my first smart-phone ever. First a 3s, then a 4s, then a 5s that I dropped into the toilet, so I switched the SIM back into the 4s to have a working phone. Furthermore, I am a heavy music listener and used iTunes Radio for 3 months. Since I am planning to change phones to an Android phone (being fed up with Apple s super-closed environment), I tried out Google Music Plus for about 3 months, too. Here is my verict Google Music is a big pain, far from iTunes in comfort and user friendliness. My move to Android is not very much in danger. General description of the service In principle, iTunes Radio and Google Music Subscription do the same things:

• allow you to have your own music in the cloud
• stream any music from the respective market place to your device
• provide radio stations, pre-curated or based on artist/genre/etc

There are slight differences, that often create confusion, especially with Apple s services: If you are only signed in, you can listen to music stations and can skip some music (limited), but you cannot listen to arbitrary songs from the whole Apple Music library. Then there is another service from Apple, called iTunes Match, that only allows you to upload your music library to the cloud, but other than that again only listening to normal radio stations. Google Music is much simpler, there are only two options: By default it is free to have your music (up to 25000 songs) in the cloud, but if you want to listen to radio stations or any other music in the Google music store, you need a subscription. Prizes and features overview:

	Apple			Google
	No subscription	iTunes Match	iTunes Radio	No subscription	With subscription
Prize	0	25$/year	10$/month	0	10$/month (early access 8$)
Cloud space	0	100000		25000
Extra service	Radio stations	Radio stations	Radio/all music	Nothing	Radio/all music

Music on the go the application Let us first consider the applications provided to listen to music on your smart phone: Apple Music can of course only be used on Apple devices, and uses the built-in Music application. Start up time is about a few seconds after a cold boot (all on my 4s), and music streaming starts with hardly any delay. Responsiveness is good, and the user interface is clear and easy. Google s application is available on iOS and Android. I have tested the latest version on iOS, but it is a pain in the butt:

• Starting the application, even after cold boot, is successful at a rate of 1/10. Most of the time the application crashes right away. This might be a problem due to my low end device (iPhone 4s with 64Gb), but not due to space problems (half empty) nor internet connectivity (wlan).
• Responsiveness is abysmal
• Access to additional content (radio, songs) and your own library is well done, similar to iMusic.

Managing your library Here iTunes is the way to go, bit of a pain when using Linux, but there are other reasons I have a Windows installation in parallel, so I don t mind to boot now and then into Windows. iTunes gives you very powerful tools to change all kind of data. Google gives you a few options: Use your iTunes library in this case all your playlists and ratings (but see later) are also uploaded. I am not sure what happens if I retag a song in iTunes, or change anything else in there. I guess the song will not be re-uploaded, but who knows for sure. Furthermore, you can edit your library via the web interface, but this is rather poor. Searching your library Over times your library grows and there are hundreds if not thousands of artists. So you want to search them. The natural way would be to scroll to the first letter of the artist you are searching, and then look it up. Well, that works perfectly in both Apple s and Google s application unless you are having artists written in some strange script like Japanese or Korean. iTunes allows you to set a field called something like Artist name for sorting , which allows me to put for example or into the artist field, and into the sort field Tomokawa Kazuki and Kim Doo Soo . This way I find the artists in the correct place. Google on the other hand uses simple Unicode order sorting how could you do that? This is simply plain wrong, and everyone should know that by now. Japanese people will never be able to find anything in their list. And in contrast to Goole Contacts Google Music does not support phonetic name fields (similar to the order sort artist) for artists. What I had to do now is to rename all the artists to include first the phonetic name, followed by the proper name, like in Tomokawa Kazuki ( ) . Something I strongly detest! Radio stations I might have a slightly peculiar music taste, but the radio stations mixed by Google are simply a pain. The reason is easy to explain: I live in Japan, and so what Google does is mixing about 80% of J-Pop into the radio, all those happy yodeling girlies I really dislike (see my Anti J-Pop campaign for alternatives yes, they do exist also in Japan!). iTunes radio is here more relaxed it seems. I appreciate Google s trial to cater to local (dis)taste, but besides voting down each and every song I hear I don t see any other option. And honestly, I cannot go through all this voting down without dying from pain inflicted by J-Pop. Other than this, the two radio stations are probably more or less the same but as I said, due to the local colorit it is hard to compare. Rating Ohhh, what a dire point. So there you have you well curated iTunes library with 5-star ratings. I used the ratings in a way that those songs I like get 1 star, those I even like more get 2 stars, and my absolute favorites got 5 stars. I didn t do any negative ratings. iTunes/iOS Music app allows you to easily adjust rating, and they are synced between devices. All as you would expect. Now for Google they did have a 5-star rating system at some time, but:

Thank you for your feedback. We ve decided to remove the 5-star Rating lab. This decision wasn t made lightly, but Thumbs Up/Down is integral for the future vision for Play Music and will be a central design point for our future releases. Please note that we ll continue to store your ratings that you ve set via the star lab or via iTunes, and we ll translate them to thumbs up/down (1-2 stars = thumbs down, 4-5 stars = thumbs up).

Here we go with the move to a new design they threw away the 5-star system, without any reason but integral for future vision rubbish sales speak. Besides being a very very poor rating system to have only an up and down (good-bad-don t care), the translation from my rating system to the thumbs up-down is just plain wrong. <rant>
Why on earth is the current movement to reduce functionality and rob users of control? Gnome 3 is the prime example of how we stupiditize users by taken any freedom from them in the name of simple design. Google now does the same. I am so sick of getting patronized this way, so Gnome3 was completely banished from my computer and replaced by Cinnamon, which uses the same (good) underlying technique, but takes users seriously!
</rant> Overall verdict As much as I would like to see Google Music a valid alternative to iTunes, by now it makes the impression of a quickly hacked together rotten piece of code that tries to get a share of the market without providing equivalent service. Google is using its market presence and convenience to convert people, not features and quality. I can only hope that this changes in the future. That still leaves me with the question move to Android or remain with iOS

This weekend is organized the Aaron Swartz Day across the world. There are events organized in many cities and video streams available. It is important that we remember Aaron's projects and fights. If you want to know more about Aaron Swartz, you may start by watching the excellent documentary The Internet's Own Boy : The Story of Aaron Swartz. His work was very inspirational and should not be forgotten!

Last week, Senator Omar Fayad presented one of the prime examples of a poorly redacted law that, if enacted, will make basically any way of computer use illegal. And yes, even if he states this is merely a draft, it has so many factual and conceptual errors that there is no way to trust sanity can be regained at any point. Oh, and before I continue with this rant: If the topic interests you, I suggest you to read the 10 key points about Ley Fayad, the worst Internet initiative in history, published by r3d.mx. [update] An English equivalent of the work at r3d, at revolution-news.com: #LeyFayad: The Worst Bill in Internet History The full text (in Spanish, of course) for the law initiative is available at the Senate webpage; the law will be called Ley Federal para Prevenir y Sancionar los Delitos Inform ticos (Federal law to prevent and punish informatic felonies<) A bad name to start with, as there are many laws already in that contested area. I started reading with the preamble (Exposici n de motivos), which already shows bad signs of imprecise redaction and is plagued with factual errors (i.e. asserting that the real danger stems from the Web migrating to the Web 2.0, from which stems that this migration and not any previous one. Or by stating that (quoting+translating a full paragraph):

Activities such as electronic commerce, digital periodism, publicity and the opinions, messages or elements written in social networks can lead to patrimonial, reputation, honor or professional activity losses for people.

He continues by stating that only 16% of the countries have some kind of cybersecurity strategy (and, of course, Mexico doesn't). That... Well, is very hard to believe, as Mexico has two separate policial groups devoted to cybersecurity, and laws regulating from electronic signatures, commerce, identity, privacy, use and abuse, and a long list. Of course, as most law proposals go, it quickly decays into a dry, boring document... And I must admit I didn't fully read it, but picked here and there. I won't copy in full the note I mentioned at the beginning at r3d.mx, but will continue with some strange points, such as:

Article 16

Every person that, without the corresponding authorization or exceeding the authorization confered, accesses, intercepts, interfers or uses an information system, will be punished by one to eight years of prision and fined by 800 to 1000 days of minimum wage

So, yes, borrowing your computer without getting explicit permission, or playing around with the options in kiosks, or tons of whatever we curious people do with systems we encounter are basis for jail. (And yes, fines in this country are expressed in "days of minimum wage", which goes at ~MX$70 per day, which is ~US$4). But it gets funkier quickly:

Article 17

Whoever fraudulently destroys, disables, damages or in any way alters the working of an informatic system or any of its components, will be punished by fice to fifteen years of jail and fined by up to a thousand minimum wage days The same punishment will be given to whoever, without authorization, destroys, damages, modifies, divulges, transfers or disables information contained in any Informatic System or any of its components. The punishment will be ten to twenty years in prision and a fine of up to a thousand days of minimum wage if the effects here mentioned are done by the creation, introduction or fraudulent transmission, by any means, of an informatic weapon or malicious code

This law is meant to protect against cyberfelonies, if such a thing exists. However, here we are putting at risk people even for accidental equipment destructions. I dropped your portable hard disk with my elbow off the table? Accuse me of acting fraudulently, and I'm up for a serious jail time. And yes, laws are meant to be interpreted... And I don't want to be at the receiving end of this one! In this last article, Fayad mentions informatic weapons, which are defined in the preamble as any informatic program, informatic system, or in general, any device or material created or designed with the purpose of committing an informatic crime. So the very next article makes me, as it should make all of my fellow students and researchers, very uneasy:

Article 18

Whoever uses informatic weapons or malicious code will be imprisioned by two to six years, and fined with 200 to 500 days of minimum wage.

Article 19

Whoever builds, distributes, commerces with informatic weapons or malicious codes will be punished by three to seven years of prision and 200 to 500 days of minimum wage.

If we need to analyze malware for our classes (or for paid work, or as a hobby), we clearly fall in article 18. If we write something that can be classified as malware (without even releasing it, as an academic excercise only!), we are covered by article 19. If I give my students code that's known to be malicious (which could be as inofensive as linking to a well-known Web comic), I'm also covered by article 19. I'll jump all the way to article 31 (reproduced only partially):

Article 31

Whoever, by any means, creates, captures, records, copies, alters, duplicates, clones or deletes the information contained in a credit or debit card (...) will be punished by 8 to 14 years of prision and 300 to 500 days of minimum wage. (...)

This clearly disincentivates any way of e-commerce. When I try to buy anything online, I have to capture+copy my (rightfully owned) credit card data. The services provider has to copy, process and then delete said information. Any e-transaction is punished by jail! Well... But thinking about this again, maybe I shouldn't be so worried about the malware distribution issue at my classes. There are clearer and more contundent articles. Say...

Article 35

Whoever convenes, organizes, is part of, or executes a cibernetic attack, will be punished by 20 to 30 years of prision and fined with 100 to 1000 days of minimum wage

Of course we have convened, organized, been part of and executed cibernetic attacks at the computer security lab at ESIME. Why would there be such a lab otherwise? Then, there are clear indications that the Senator didn't understand the topic his team was working on:

Article 37

Who manipulates the digital seals used by command of the public authority will be punished with 240 days of community work

Now... What is a digital seal? It's not a phisical one that does not allow opening the doors to a business found at fault, but something that just proves a document is legitimate and pristine. How can I manipulate them? Of course, if the seals are MD5-based, I can easily forge them (and SHA1-based, it seems they will be broken enough soon to be considered no longer trustable)... But that's about it! And there is more, lots more. I'm swamped with work, and have to get back to it. But chapters the following chapters have a lot of potential for finding holes. PS - And yes, the only use I do of Twitter is via the headlines in my blog ;-) [update] Ley Fayad is dead, yay! \o/ The senator withdrew the proposal.

Finally, Gnocchi 1.3.0 is out. This is our final release, more or less matching the OpenStack 6 months schedule, that concludes the Liberty development cycle.

This release was supposed to be released a few weeks earlier, but our integration test got completely blocked for several days just the week before the OpenStack Mitaka summit. New website We build a new dedicated website for Gnocchi at gnocchi.xyz. We want to promote Gnocchi outside of the OpenStack bubble, as it a useful timeseries database on its own that can work without the rest of the stack. We'll try to improve the documentation. If you're curious, feel free to check it out and report anything you miss! The speed bump Obviously, if it was a bug in Gnocchi that we have hit, it would have been quick to fix. However, we found a nasty bug in Swift caused by the evil monkey-patching of Eventlet (once again) blended with a mixed usage of native threads and Eventlet threads in Swift. Shake all of that, and you got yourself pretty race conditions when using the Keystone middleware authentication. In the meantime, we disabled Swift multi-threading by using mod_wsgi instead of Eventlet in devstack. New features So what's new in this new shiny release? A few interesting things:

• Metric deletion is now asynchronous. That's not the most used feature in the REST API weirdly people do not often delete metrics but it's now way faster and reliable by being asynchronous. Metricd is now in charge of cleaning up things up.
• Speed improvement. We are now confident to be even more faster than in the latest benchmarks I run (around 1.5-2 faster), which makes Gnocchi really fast with its native storage back-ends. We profiled and optimized Carbonara and the REST API data validation.
• Improve metricd status report. It now reports the size of the backlog of the whole cluster both in its log and via the REST API. Easy monitoring!
• Ceph drivers enhancement. We had people testing the Ceph drivers in production, so we made a few changes and fixes to it to make it more solid.

And that's all we did in the last couple of months. We have a lot of things on the roadmap that are pretty exciting, and I'll sure talk about them in the next weeks.

What happened in the reproducible builds effort this week: Toolchain fixes

• Robert Luberda uploaded ispell/3.4.00-4 which fixes another issue with uninitialized memory in ispell hashes. Original patch by Valentin Lorentz.
• Robert Luberda uploaded man2html/1.6g-8 which adds support for SOURCE_DATE_EPOCH. Original patch by akira.
• gregor herrmann uploaded libdebian-copyright-perl/0.2-3 which sorts copyright files for deterministic ordering. Original patch by Reiner Herrmann.
• Rafael Laboissiere uploaded octave-pkg-dev/1.3.2 which normalizes the name of the temporary build directory. This doesn't seem to be enough to make Octave packages reproducible just yet, though.
• Nicolas Boulenguez uploaded gprbuild/2015-1 which sets a deterministic date to the generated source.

Packages fixed The following packages became reproducible due to changes in their build dependencies: maven-plugin-tools, norwegian, ocaml-melt, python-biom-format, rivet. The following packages became reproducible after getting fixed:

• apache2/2.4.17-1 by Stefan Fritsch.
• autogen/1:5.18.6-3 by Andreas Metzler.
• debian-timeline/37 by Chris Lamb.
• fonty-rg/0.6 uploaded by Radovan Garab k, original patch by Chris Lamb.
• foxeye/0.10.2-2 by Andriy Grytsenko.
• hsqldb/2.3.3+dfsg2-1 by Markus Koschany.
• jasperreports/6.1.1+dfsg-1 uploaded by Markus Koschany, fix by Emmanuel Bourg.
• libmath-base-convert-perl/0.11-2 uploaded by Salvatore Bonaccorso, original patch by Reiner Herrmann.
• libsdl2-gfx/1.0.1+dfsg-2 uploaded by Manuel A. Fernandez Montecelo, original patch by Reiner Herrmann.
• libsdl2/2.0.2+dfsg1-8 uploaded by Gianfranco Costamagna, patch by Reiner Herrmann.
• linux/4.2.5-1 by Ben Hutchings.
• litl/0.1.7+dfsg-1 by Samuel Thibault, original patch by Chris Lamb.
• python-keystoneclient/1:1.7.1-4 by Thomas Goirand.
• sphinxbase/0.8+5prealpha-1 by Samuel Thibault.
• tatan/1.0.dfsg1-6 uploaded by Markus Koschany, original patch by Reiner Herrmann.
• v4l2loopback/0.9.1-4 uploaded by IOhannes m zm lnig, original patch.
• yadifa/2.1.4-2 uploaded by Markus Schade, original patch by Reiner Herrmann.

Some uploads fixed some reproducibility issues but not all of them:

• lcov/1.12-2 by Alastair McKinstry, original patch by Reiner Herrmann.
• libsdl1.2/1.2.15-12 uploaded by Manuel A. Fernandez Montecelo, original patch by Reiner Herrmann.
• metview/4.5.7-1 by Alastair McKinstry.
• mumble/1.2.10-2 by Christopher Knadle.

The following package is currently failing to build from source but should now be reproducible:

• p4vasp/0.3.29+dfsg-2 uploaded by Graham Inggs, original patch by Reiner Herrmann.

Patches submitted which have not made their way to the archive yet:

• #803501 on fdroidserver by Reiner Herrmann: add support for SOURCE_DATE_EPOCH to docs/gendocs.sh and normalizes tarball permissions. Sent upstream.
• #803547 on bbswitch by Reiner Herrmann: tell tar to normalize the permissions.
• #803583 on ndiswrapper by Reiner Herrmann: tell tar to normalize the permissions.
• #803601 on xtables-addons by Reiner Herrmann: tell tar to normalize the permissions.
• #803603 on usb-modeswitch-data by Reiner Herrmann: tell tar to normalize the permissions.

reproducible.debian.net A quick update on current statistics: testing is at 85% of packages tested reproducible with our modified packages, unstable on armhf caught up with amd64 with 80%. The schroot name used for running diffoscope when testing OpenWrt, NetBSD, Coreboot, and Arch Linux has been fixed. (h01ger, Mattia Rizzolo) Documentation update Paul Gevers documented timestamps in unit files created by the Free Pascal Compiler. reproducible-builds.org is now live. It contains a comprehensive documentation on all aspects that have been identified so far of what we call reproducible builds . It makes room for pointers to projects working on reproducible builds, news, dedicated tools, and community events. Package reviews 206 reviews have been removed, 171 added and 196 updated this week. Chris Lamb reported 28 failing to build from source issues. New issues identified this week: timestamps_in_pdf_content, different_encoding_in_html_by_docbook_xsl, timestamps_in_ppu_generated_by_fpc, method_may_never_be_called_in_documentation_generated_by_javadoc. Misc. Andrei Borzenkov has proposed a fix for uninitialized memory in GRUB's mkimage. Uninitialized memory is one source of hard to track down reproducibility errors. Holger Levsen presented the efforts on reproduible builds at Festival de Software Libre in Puerto Vallarta, Mexico.

USENIX lately started a new journal called JESA to tackle the issue of education for Systemadministrators. For the first issue Tom Limoncelli wrote an open letter which tries to summarize the current situation the industry faces. For me it's kind of a problem statement one can use to start thinking about solutions. Currently I don't see something like a formal education to call yourself Systemadministrator or Systemengineer anywhere near. And I don't think it's required. But still the expectations I see on both ends - employer and employee - often differ a lot in all kind of directions. In Germany we've a very organized (some call it bureaucratic) system of non academic education, organized as an apprenticeship. And like Tom wrote in the open letter mentioned above many IT departments do not follow best practise, and even more do so unintentionally because they never got that far. But what kind of people can you expect from this system when they got formed for three years in a rather sloppy environment? So there is a lot to fix, but as usual I've doubts when I think about possible solutions. Do I expect too much from the education system and/or the people? Do I look at the wrong people? Is this education system the right system to educate this kind of people I'd like to work with?

Italian Fattura Elettronica with OpenSSL I have had some success signing an Italian fattura elettronica with OpenSSL. I am amazed to realise that the software they gave me to do the job is of such bad quality that I felt like spending a few hours trying to do the same thing with OpenSSL instead.

Following up on Then and Now ... In quiet moments at ICFP last August, I finished teaching Propellor to generate disk images. With an emphasis on doing a whole lot with very little new code and extreme amount of code reuse. For example, let's make a disk image with nethack on it. First, we need to define a chroot. Disk image creation reuses propellor's chroot support, described back in propelling containers. Any propellor properties can be assigned to the chroot, so it's easy to describe the system we want. Now to make an image from that chroot, we just have to tell propellor where to put the image file, some partitioning information, and to make it boot using grub. The disk image partitions default to being sized to fit exactly the files from the chroot that go into each partition, so, the disk image is as small as possible by default. There's a little DSL to configure the partitions. To give control over the partition size, it has some functions, like addFreeSpace and setSize. Other functions like setFlag and extended can further adjust the partitions. I think that worked out rather well; the partition specification is compact and avoids unecessary hardcoded sizes, while providing plenty of control. By the end of ICFP, I had Propellor building complete disk images, but no boot loader installed on them.

---

Fast forward to today. After stuggling with some strange grub behavior, I found a working method to install grub onto a disk image. The whole disk image feature weighs in at: 203 lines to interface with parted
88 lines to format and mount partitions
90 lines for the partition table specification DSL and partition sizing
196 lines to generate disk images
75 lines to install grub on a disk image
652 lines of code total Which is about half the size of vmdebootstrap 1/4th the size of partman-base (probably 1/100th the size of total partman), and 1/13th the size of live-build. All of which do similar things, in ways that seem to me to be much less flexible than Propellor.

---

One thing I'm considering doing is extending this so Propellor can use qemu-user-static to create disk images for eg, arm. Add some u-boot setup, and this could create bootable images for arm boards. A library of configs for various arm boards could then be included in Propellor. This would be a lot easier than running the Debian Installer on an arm board. Oh! I only just now realized that if you have a propellor host configured, like this example for my dialup gateway, leech -- -- The host's properties can be extracted from it, using eg hostProperties leech and reused to create a disk image with the same properties as the host! So, when my dialup gateway gets struck by lightning again, I could use this to build a disk image for its replacement: This also means you can start with a manually built system, write down the properties it has, and iteratively run Propellor against it until you think you have a full specification of it, and then use that to generate a new, clean disk image. Nice way to transition from sysadmin days of yore to a clean declaratively specified system.

Review: What If?, by Randall Munroe

Publisher:	Houghton Mifflin Harcourt
Copyright:	2014
ISBN:	0-544-27299-4
Format:	Hardcover
Pages:	295

This is another one of those reviews that's somewhat pointless to write, at least beyond telling people who for some strange reason aren't xkcd readers that this is a thing that exists in the world. What If? is a collection of essays from that feature on the xkcd web site and new essays in the same vein. (Over half are new to this collection.) If you've read them, you know what to expect; if you haven't, and have any liking at all for odd scientific facts or stick figures, you're in for a treat. So, short review: The subtitle is Serious Scientific Answers to Absurd Hypothetical Questions, and it's exactly what it says on the tin, except that "serious" includes a healthy dose of trademark xkcd humor. Go read what-if.xkcd.com for numerous samples of Munroe's essay style. If you like what you see, this is a whole book of that: a nice, high-quality hardcover (at least the edition I bought), featuring the same mix of text and cartoon commentary, and with new (and in some cases somewhat longer) material. You probably now have all the information necessary to make a purchasing decision. If you need more motivation, particularly to buy a physical copy, the inside of the dust jacket of the hardcover is a detailed, labeled map of the world after a drain in the Marianas Trench has emptied most of the oceans onto Mars. And the book inside the dust jacket is embossed with what happens after the dinosaur on the cover is lowered into, or at least towards, the Great Pit of Carkoon. This made me particularly happy, since too often hardcovers inside the dust jacket look just like every other hardcover except for the spine lettering. Very few of them have embossed Star Wars references. Personally, I think that's a great reason to buy the hardcover even if, like me, you've been following What If? on the web religiously since it started. But of course the real draw is the new material. There's enough of it that I won't try any sort of comprehensive list, but rest assured that it's of equal or better quality than the web-published essays we know and love. My favorite of the new pieces is the answer to the question "what would happen if you made a periodic table out of cube-shaped bricks, where each brick was made of the corresponding element?" As with so many What If? questions, it starts with killing everyone in the vicinity, and then things get weird. Another nice touch in this collection is what I'd call "rejected questions": questions that people submitted but that didn't inspire an essay. Most of these (I wish all) get a single cartoon of reaction to the question itself, which include some of the funniest (and most touching) panels in the book. Ebook formatting has gotten much better, so there's some hope that at least some platforms could do justice to this book with its embedded cartoons. Putting the footnotes properly at the bottom of each page (thank you!) might be a challenge, though. Writing mixed with art is one of the things I think benefits greatly from a physical copy, and the hardcover is a satisfying and beautiful artifact. (I see there's also an audio book, but I'm sure how well that could work; so much of the joy of What If? is the illustrations, and I'm dubious that one could adequately describe them.) Prior web readers will be relieved to know that the mouse-over text is preserved as italic captions under the cartoons, although sadly most cartoons are missing captions. (As I recall, that's also the case for the early web What If? essays, but later essays have mouse-over text for nearly every cartoon.) Anyway, this is a thing that exists. If you follow xkcd, you probably knew that already, given that the book was published last year and I only now got around to reading it. (My current backlog is... impressive.) If you were not previously aware of What If? or of xkcd itself, now you are, and I envy you the joy of discovery. A short bit of reading will tell you for certain whether this is something you want to purchase. If your relationship to physics is at all similar to mine, I suspect the answer will be yes. A small personal note: I just now realized how much the style of What If? resembles the mixed text and illustrations of One Two Three... Infinity. Given how foundational that book was to my love of obscure physics facts, my love of What If? is even less surprising. Rating: 10 out of 10