Search Results: "rul"

31 October 2015

Joey Hess: a tiling region manager for the console

Building on top of concurrent-output, and some related work Joachim Breitner did earlier, I now have a kind of equivilant to a tiling window manager, except it's managing regions of the console for different parts of a single program. Here's a really silly demo, in an animated gif:

Not bad for 23 lines of code, is that? Seems much less tedious to do things this way than using ncurses. Even with its panels, ncurses requires you to think about layout of various things on the screen, and many low-level details. This, by contrast, is compositional, just add another region and a thread to update it, and away it goes. So, here's an apt-like download progress display, in 30 lines of code.

Not only does it have regions which are individual lines of the screen, but those can have sub-regions within them as seen here (and so on). And, log-type messages automatically scroll up above the regions. External programs run by createProcessConcurrent will automatically get their output/errors displayed there, too. What I'm working on now is support for multiline regions, which automatically grow/shrink to fit what's placed in them. The hard part, which I'm putting the finishing touches on, is to accurately work out how large a region is before displaying it, in order to lay it out. Requires parsing ANSI codes amoung other things. STM rules There's so much concurrency, with complicated interrelated data being updated by different threads, that I couldn't have possibly built this without Software Transactional Memory. Rather than a nightmare of locks behind locks behind locks, the result is so well behaved that I'm confident that anyone who needs more control over the region layout, or wants to do funky things can dive into to the STM interface and update the data structures, and nothing will ever deadlock or be inconsistent, and as soon as an update completes, it'll display on-screen. An example of how powerful and beuatiful STM is, here's how the main display thread determines when it needs to refresh the display:

data DisplayChange
        = BufferChange [(StdHandle, OutputBuffer)]
          RegionChange RegionSnapshot
          TerminalResize (Maybe Width)
          EndSignal ()
    ...
                change <- atomically $
                        (RegionChange <$> regionWaiter origsnapshot)
                                 orElse 
                        (RegionChange <$> regionListWaiter origsnapshot)
                                 orElse 
                        (BufferChange <$> outputBufferWaiterSTM waitCompleteLines)
                                 orElse 
                        (TerminalResize <$> waitwidthchange)
                                 orElse 
                        (EndSignal <$> waitTSem endsignal)
                case change of
                        RegionChange snapshot -> do
                ...
                        BufferChange buffers -> do
                ...
                        TerminalResize width -> do
                ...

So, it composes all these STM actions that can wait on various kinds of changes, to get one big action, that waits for all of the above, and builds up a nice sum type to represent what's changed. Another example is that the whole support for sub-regions only involved adding 30 lines of code, all of it using STM, and it worked 100% the first time.

Available in concurrent-output 1.1.0.

29 October 2015

Mart n Ferrari: Tales from the SRE trenches: SREs are not firefighters

This is the third part in a series of articles about SRE, based on the talk I gave in the Romanian Association for Better Software. If you haven't already, you might want to read part 1 and part 2 first. In this post, I talk about some strategies to avoid drowning SRE people in operational work.

SREs are not firefighters As I said before, it is very important that there is trust and respect between Dev and Ops. If Ops is seen as just an army of firefighters that will gladly put away fires at any time of the night, there are less incentives to make good software. Conversely, if Dev is shielded from the realities of the production environment, Ops will tend to think of them as delusional and untrustworthy.

Common staffing pool for SRE and SWE The first tool to combat this at Google -and possibly a difficult one to implement in small organisations- is to have single headcount budgets for Dev and Ops. That means that the more SREs you need to support your service, the less developers you have to write it. Combined with this, Google offers the possibility to SREs to move freely between teams, or even to transfer out to SWE. Because of this, a service that is painful to support will see the most senior SREs leaving and will only be able to afford less experienced hires. All this is a strong incentive to write good quality software, to work closely and to listen to the Ops people.

Share 5% of operational work with the SWE team On the other hand, it is necessary that developers see first hand how the service works in production, understand the problems, and share the pain of things failing. To this end, SWEs are expected to take on a small fraction of the operational work from SRE: handling tickets, being on-call, performing deployments, or managing capacity. This results in better communication among the teams and a common understanding of priorities.

Cap operational load at 50% One very uncommon rule from SRE at Google, is that SREs are not supposed to spend more than half of their time on "operational work". SREs are supposed to be spending their time on automation, monitoring, forecasting growth... Not on repeatedly fixing manually issues that stem from bad systems.

Excess operational work overflows to SWE If an SRE team is found to be spending too much time on operational work, that extra load is automatically reassigned to the development team, so SRE can keep doing their non-operational duties. On extreme cases, a service might be deemed too unstable to maintain, and SRE support is completely removed: it means the development team now has to carry pagers and do all the operational work themselves. It is a nuclear option, but the threat of it happening is a strong incentive to keep things sane.
The next post will be less about how to avoid unwanted work and more about the things that SRE actually do, and how these make things better. Comment

28 October 2015

Daniel Pocock: Free Real-Time Communications workshop in Manchester, 2 November

Manchester Free Software and MadLab are hosting a workshop on Free and Federated Real-Time Communications with Free Software next Monday night, 2 November from 7pm. Add this to your calendar (iCalendar link) Users and developers are invited to come and discover the opportunities for truly free and secure chat, voice and video with WebRTC, mobile VoIP and the free desktop. This is an interactive workshop with practical, hands-on opportunities to explore this topic, see what works and what doesn't and learn about troubleshooting techniques to help developers like myself improve the software. This is also a great opportunity for any Java developers who would like to get some exposure to the Android platform. People wishing to fully participate in all the activities are encouraged to bring a laptop with the Wireshark software and Android tools (especially adb, the command line utility) or if you prefer to focus on WebRTC, have the latest version of Firefox and Chrome installed. If you are keen to run the DruCall module for WebRTC or your own SIP or XMPP server, you can try setting it up as described in the RTC Quick Start Guide and come along to the workshop with any questions you have. A workshop near you? Manchester has a history of great technological innovation, including the first stored program computer and it is a real thrill for me to offer this workshop there. FSFE Manchester ran a workshop evaluating the performance of free software softphones back in 2012. Over the coming months, I would like to be able to hold further workshops in other locations to get feedback from people who are trying this technology, including the Lumicall app, JSCommunicator and DruCall. If you are interested in helping organize such an event or you have any other feedback about this topic, please come and discuss it on the Free RTC mailing list.

27 October 2015

Marco d'Itri: Per-process netfilter rules

This article documents how the traffic of specific Linux processes can be subjected to a custom firewall or routing configuration, thanks to the magic of cgroups. We will use the Network classifier cgroup, which allows tagging the packets sent by specific processes. To create the cgroup which will be used to identify the processes I added something like this to /etc/rc.local:

mkdir /sys/fs/cgroup/net_cls/unlocator
/bin/echo 42 > /sys/fs/cgroup/net_cls/unlocator/net_cls.classid
chown md: /sys/fs/cgroup/net_cls/unlocator/tasks

The tasks file, which controls the membership of processes in a cgroup, is made writeable by my user: this way I can add new processes without becoming root. 42 is the arbitrary class identifier that the kernel will associate with the packets generated by the member processes. A command like systemd-cgls /sys/fs/cgroup/net_cls/ can be used to explore which processes are in which cgroup. I use a simple shell wrapper to start a shell or a new program as members of this cgroup:

#!/bin/sh -e
CGROUP_NAME=unlocator
if [ ! -d /sys/fs/cgroup/net_cls/$CGROUP_NAME/ ]; then
  echo "The $CGROUP_NAME net_cls cgroup does not exist!" >&2
  exit 1
fi
/bin/echo $$ > /sys/fs/cgroup/net_cls/$CGROUP_NAME/tasks
if [ $# = 0 ]; then
  exec $ SHELL:-/bin/sh 
fi
exec "$@"

My first goal is to use a special name server for the DNS queries of some processes, thanks to a second dnsmasq process which acts as a caching forwarder. /etc/dnsmasq2.conf:

port=5354
listen-address=127.0.0.1
bind-interfaces
no-dhcp-interface=*
no-hosts
no-resolv
server=185.37.37.37
server=185.37.37.185

/etc/systemd/system/dnsmasq2.service:

[Unit]
Description=dnsmasq - Second instance
Requires=network.target
[Service]
ExecStartPre=/usr/sbin/dnsmasq --test
ExecStart=/usr/sbin/dnsmasq --keep-in-foreground --conf-file=/etc/dnsmasq2.conf
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/run/dnsmasq/dnsmasq.pid
[Install]
WantedBy=multi-user.target

Do not forget to enable the new service:

systemctl enable dnsmasq2
systemctl start dnsmasq2

Since the cgroup match extension is not yet available in a released version of iptables, you will first need to build and install it manually:

git clone git://git.netfilter.org/iptables.git
cd iptables
./autogen.sh
./configure
make -k
sudo cp extensions/libxt_cgroup.so /lib/xtables/
sudo chmod -x /lib/xtables/libxt_cgroup.so

The netfilter configuration required is very simple: all DNS traffic from the marked processes is redirected to the port of the local dnsmasq2:

iptables -t nat -A OUTPUT -m cgroup --cgroup 42 -p udp --dport 53 -j REDIRECT --to-ports 5354
iptables -t nat -A OUTPUT -m cgroup --cgroup 42 -p tcp --dport 53 -j REDIRECT --to-ports 5354

For related reasons, I also need to disable IPv6 for these processes:

ip6tables -A OUTPUT -m cgroup --cgroup 42 -j REJECT

I use a different cgroup to force some programs to use my office VPN by first setting a netfilter packet mark on their traffic:

iptables -t mangle -A OUTPUT -m cgroup --cgroup 43 -j MARK --set-mark 43

The packet mark is then used to policy-route this traffic using a dedicate VRF, i.e. routing table 43:

ip rule add fwmark 43 table 43

This VPN VRF just contains a default route for the VPN interface:

ip route add default dev tun0 table 43

Depending on your local configuration it may be a good idea to also add to the VPN VRF the routes of your local interfaces:

ip route show scope link proto kernel \
    xargs -I ROUTE ip route add ROUTE table 43

Since the source address selection happens before the traffic is diverted to the VPN, we also need to source-NAT to the VPN address the marked packets:

iptables -t nat -A POSTROUTING -m mark --mark 43 --out-interface tun0 -j MASQUERADE

25 October 2015

Johannes Schauer: unshare without superuser privileges

TLDR: With the help of Helmut Grohne I finally figured out most of the bits necessary to unshare everything without becoming root (though one might say that this is still cheated because the suid root tools newuidmap and newgidmap are used). I wrote a Perl script which documents how this is done in practice. This script is nearly equivalent to using the existing commands

lxc-usernsexec
[opts] -- unshare [opts] -- COMMAND

except that these two together cannot be used to mount a new proc. Apart from this problem, this Perl script might also be useful by itself because it is architecture independent and easily inspectable for the curious mind without resorting to sources.debian.net (it is heavily documented at nearly 2 lines of comments per line of code on average). It can be retrieved here at https://gitlab.mister-muffin.de/josch/user-unshare/blob/master/user-unshare Long story: Nearly two years after my last last rant about everything needing superuser privileges in Linux, I'm still interested in techniques that let me do more things without becoming root. Helmut Grohne had told me for a while about unshare(), or user namespaces as the right way to have things like chroot without root. There are also reports of LXC containers working without root privileges but they are hard to come by. A couple of days ago I had some time again, so Helmut helped me to get through the major blockers that were so far stopping me from using unshare in a meaningful way without executing everything with sudo. My main motivation at that point was to let dpkg-buildpackage when executed by sbuild be run with an unshared network namespace and thus without network access (except for the loopback interface) because like pbuilder I wanted sbuild to enforce the rule not to access any remote resources during the build. After several evenings of investigating and doctoring at the Perl script I mentioned initially, I came to the conclusion that the only place that can unshare the network namespace without disrupting anything is schroot itself. This is because unsharing inside the chroot will fail because dpkg-buildpackage is run with non-root privileges and thus the user namespace has to be unshared. But this then will destroy all ownership information. But even if that wasn't the case, the chroot itself is unlikely to have (and also should not) tools like ip or newuidmap and newgidmap installed. Unsharing the schroot call itself also will not work. Again we first need to unshare the user namespace and then schroot will complain about wrong ownership of its configuration file /etc/schroot/schroot.conf. Luckily, when contacting Roger Leigh about this wishlist feature in bug#802849 I was told that this was already implemented in its git master \o/. So this particular problem seems to be taken care of and once the next schroot release happens, sbuild will make use of it and have unshare --net capabilities just like pbuilder already had since last year. With the sbuild case taken care of, the rest of this post will introduce the Perl script I wrote. The name user-unshare is really arbitrary. I just needed some identifier for the git repository and a filename. The most important discovery I made was, that Debian disables unprivileged user namespaces by default with the patch add-sysctl-to-disallow-unprivileged-CLONE_NEWUSER-by-default.patch to the Linux kernel. To enable it, one has to first either do

echo 1   sudo tee /proc/sys/kernel/unprivileged_userns_clone > /dev/null

sudo sysctl -w kernel.unprivileged_userns_clone=1

The tool tries to be like unshare(1) but with the power of lxc-usernsexec(1) to map more than one id into the new user namespace by using the programs newgidmap and newuidmap. Or in other words: This tool tries to be like lxc-usernsexec(1) but with the power of unshare(1) to unshare more than just the user and mount namespaces. It is nearly equal to calling:

lxc-usernsexec [opts] -- unshare [opts] -- COMMAND

Its main reason of existence are:

as a project for me to learn how unprivileged namespaces work
written in Perl which means:
- architecture independent (same executable on any architecture)
- easily inspectable by other curious minds
tons of code comments to let others understand how things work
no need to install the lxc package in a minimal environment (perl itself might not be called minimal either but is present in every Debian installation)
not suffering from being unable to mount proc

I hoped that systemd-nspawn could do what I wanted but it seems that its requirement for being run as root will not change any time soon Another tool in Debian that offers to do chroot without superuser privileges is linux-user-chroot but that one cheats by being suid root. Had I found lxc-usernsexec earlier I would've probably not written this. But after I found it I happily used it to get an even better understanding of the matter and further improve the comments in my code. I started writing my own tool in Perl because that's the language sbuild was written in and as mentioned initially, I intended to use this script with sbuild. Now that the sbuild problem is taken care of, this is not so important anymore but I like if I can read the code of simple programs I run directly from /usr/bin without having to retrieve the source code first or use sources.debian.net. The only thing I wasn't able to figure out is how to properly mount proc into my new mount namespace. I found a workaround that works by first mounting a new proc to /proc and then bind-mounting /proc to whatever new location for proc is requested. I didn't figure out how to do this without mounting to /proc first partly also because this doesn't work at all when using lxc-usernsexec and unshare together. In this respect, this perl script is a bit more powerful than those two tools together. I suppose that the reason is that unshare wasn't written with having being called without superuser privileges in mind. If you have an idea what could be wrong, the code has a big FIXME about this issue. Finally, here a demonstration of what my script can do. Because of the /proc bug, lxc-usernsexec and unshare together are not able to do this but it might also be that I'm just not using these tools in the right way. The following will give you an interactive shell in an environment created from one of my sbuild chroot tarballs:

$ mkdir -p /tmp/buildroot/proc
$ ./user-unshare --mount-proc=/tmp/buildroot/proc --ipc --pid --net \
    --uts --mount --fork -- sh -c 'ip link set lo up && ip addr && \
    hostname hoothoot-chroot && \
    tar -C /tmp/buildroot -xf /srv/chroot/unstable-amd64.tar.gz; \
    /usr/sbin/chroot /tmp/buildroot /sbin/runuser -s /bin/bash - josch && \
    umount /tmp/buildroot/proc && rm -rf /tmp/buildroot'
(unstable-amd64-sbuild)josch@hoothoot-chroot:/$ whoami
josch
(unstable-amd64-sbuild)josch@hoothoot-chroot:/$ hostname
hoothoot-chroot
(unstable-amd64-sbuild)josch@hoothoot-chroot:/$ ls -lha /proc   head
total 0
dr-xr-xr-x 218 nobody nogroup    0 Oct 25 19:06 .
drwxr-xr-x  22 root   root     440 Oct  1 08:42 ..
dr-xr-xr-x   9 root   root       0 Oct 25 19:06 1
dr-xr-xr-x   9 josch  josch      0 Oct 25 19:06 15
dr-xr-xr-x   9 josch  josch      0 Oct 25 19:06 16
dr-xr-xr-x   9 root   root       0 Oct 25 19:06 7
dr-xr-xr-x   9 josch  josch      0 Oct 25 19:06 8
dr-xr-xr-x   4 nobody nogroup    0 Oct 25 19:06 acpi
dr-xr-xr-x   6 nobody nogroup    0 Oct 25 19:06 asound

Of course instead of running this long command we can also instead write a small shell script and execute that instead. The following does the same things as the long command above but adds some comments for further explanation:

#!/bin/sh

set -exu

# I'm using /tmp because I have it mounted as a tmpfs
rootdir="/tmp/buildroot"

# bring the loopback interface up
ip link set lo up

# show that the loopback interface is really up
ip addr

# make use of the UTS namespace being unshared
hostname hoothoot-chroot

# extract the chroot tarball. This must be done inside the user namespace for
# the file permissions to be correct.
#
# tar will fail to call mknod and to change the permissions of /proc but we are
# ignoring that
tar -C "$rootdir" -xf /srv/chroot/unstable-amd64.tar.gz   true

# run chroot and inside, immediately drop permissions to the user "josch" and
# start an interactive shell
/usr/sbin/chroot "$rootdir" /sbin/runuser -s /bin/bash - josch

# unmount /proc and remove the temporary directory
umount "$rootdir/proc"
rm -rf "$rootdir"

and then:

$ mkdir -p /tmp/buildroot/proc
$ ./user-unshare --mount-proc=/tmp/buildroot/proc --ipc --pid --net --uts --mount --fork -- ./chroot.sh

As mentioned in the beginning, the tool is nearly equivalent to calling lxc-usernsexec [opts] -- unshare [opts] -- COMMAND but because of the problem with mounting proc (mentioned earlier), lxc-usernsexec and unshare cannot be used with above example. If one tries anyways one will only get:

$ lxc-usernsexec -m b:0:1000:1 -m b:1:558752:1 -- unshare --mount-proc=/tmp/buildroot/proc --ipc --pid --net --uts --mount --fork -- ./chroot.sh
unshare: mount /tmp/buildroot/proc failed: Invalid argument

I'd be interested in finding out why that is and how to fix it.

24 October 2015

Mart n Ferrari: Tales from the SRE trenches - Part 1

A few weeks ago, I was offered the opportunity to give a guest talk in the Romanian Association for Better Software. RABS is a group of people interested in improving the trade, and regularly hold events where invited speakers give presentations on a wide array of topics. The speakers are usually pretty high-profile, so this is quite a responsibility! To make things more interesting, much of the target audience works on enterprise software, under Windows platforms. Definitely outside my comfort zone! Considering all this, we decided the topic was going to be about Site Reliability Engineering (SRE), concentrating on some aspects of it which I believe could be useful independently of the kind of company you are working for. I finally gave the talk last Monday, and the audience seemed to enjoy it, so I am going to post here my notes, hopefully some other people will like it too.

Why should I care? I prepared this thinking of an audience of software engineers, so why would anyone want to hear about this idea that only seems to be about making the life of the operations people better? The thing is, having your work as a development team supported by an SRE team will also benefit you. This is not about empowering Ops to hit you harder when things blow apart, but to have a team that is your partner. A partner that will help you grow, handle the complexities of a production environment so you can concentrate on cool features, and that will get out of the way when things are running fine. A development team may seem to only care about adding features that will drive more and more users to your service. But an unreliable service is a service that loses users, so you should care about reliability. And what better to have a team has Reliability on their name?

What is SRE? SRE means Site Reliability Engineering, Reliability Engineering applied to "sites". Wikipedia defines Reliability Engineering as:
[..] engineering that emphasizes dependability in the life-cycle management of a product.
This is, historically, a branch of engineering that made possible to build devices that will work as expected even when their components were inherently unreliable. It focused on improving component reliability, establishing minimum requirements and expectations, and a heavy usage of statistics to predict failures and understand underlying problems. SRE started as a concept at Google about 12 years ago, when Ben Treynor joined the company and created the SRE team from a group of 7 production engineers. There is no good definition of what Site Reliability Engineering means; while the term and some of its ideas are clearly inspired in the more traditional RE, he defines SRE with these words1:
Fundamentally, it's what happens when you ask a software engineer to design an operations function.

Only hire coders After reading that quote it is not surprising that the first item in the SRE checklist2, is to only hire people who can code properly for SRE roles. Writing software is a key part of being SRE. But this does not mean that there is no separation between development and operations, nor that SRE is a fancy(er) name for DevOps3. It means treating operations as a software engineering problem, using software to solve problems that used to be solved by hand, implementing rigorous testing and code reviewing, and taking decisions based on data, not just hunches. It also implies that SREs can understand the product they are supporting, and that there is a common ground and respect between SREs and software engineers (SWEs). There are many things that make SRE what it is, some of these only make sense within a special kind of company like Google: many different development and operations teams, service growth that can't be matched by hiring, and more importantly, firm commitment from top management to implement these drastic rules. Therefore, my focus here is not to preach on how everybody should adopt SRE, but to extract some of the most useful ideas that can be applied in a wider array of situations. Nevertheless, I will first try to give an overview of how SRE works at Google.
That's it for today. In the next post I will talk about how to end the war between developers and SysAdmins. Stay tuned!

http://www.site-reliability-engineering.info/2014/04/what-is-site-reliability-engineering.html

SRE checklist extracted from Treynor's talk at SREcon14: https://www.usenix.org/conference/srecon14/technical-sessions/presentation/keys-sre

By the way, I am still not sure what DevOps mean, it seems that everyone has a different definition for it.

Comment

Vincent Sanders: It takes courage to sit on a jury. How many of us want to decide the fate of another person's life or freedom?

I think Regina Brett has a point although having now experienced being a juror in a British crown court I have a much better understanding of both the process and effectiveness of the jury system.

The actual process of becoming a juror on a case is something I had not been aware of previously. You simply receive a letter telling you to be at the court on a specific date and that you are required to be available for at least ten days, possibly more. The only qualification to receive the letter is to be on the electoral roll and it is an invitation with few options to refuse without serious repercussions.

When you arrive at the court you are directed to the Jury lounge (practical hint: take a book) where you notice there are over forty people, which would seem odd until you realise there are three courtrooms and they each need a jury, even then there are an excess of people which is because of the selection process.

The process of jury selection is fairly simple, an usher for a court comes and calls fourteen names which forms the jury in waiting. The group is taken up to the court waiting room (this room gets terribly familiar over the forthcoming weeks) and then twelve names are called.

As each person is called they enter the jury box in order which persists for the entire trial (practical hint:remember your juror number). Before each person is sworn or affirmed there is the possibility they will be found unsuitable and will be replaced by one of the previously unselected jurors. Any unselected jurors are then sent back to the jury lounge and become available for forming another jury in waiting.

Anyone unselected at the end of the process has to remain available to return to the court to form a jury in waiting when a previous trial ends until they have exhausted their duty. There were a few of these unfortunate people who were kept in a state of limbo for several days and I am relieved this did not happen to me.

Being a juror, from a purely practical perspective, felt like working an office job for ten days. Duties consisting of attending a series of meetings with strange rules and the typically understated British approach to mentioning the result of breaking them.

I participated in two cases, both of which were (almost by definition) unpleasant happenings, these were a case of Grievous bodily harm and an offence under section 7 of the Sexual Offences Act 2003.

Both cases were challenging in their own ways the first because of the way the case was presented and the second because of its subject matter. One of the most important rules is "Do not discuss anything with anyone as it might be perjury" so going along with that I will not be discussing any details. Because I cannot be specific this post has become a little impersonal, you will have to forgive me as I found I had to remove a great deal of more content which was not appropriate.

An important thing to note is that the trials bore no resemblance to TV courtroom drama. The trials proceed in a professional manner with very little theatrics. The prosecution barrister commences outlining the case against the accused, calling witnesses and reading into evidence uncontested material. The defence then gets to present their case, again calling witnesses and placing documents into evidence.

One of the striking things about this process is that if the barristers do not call a witness or present evidence that would seem to be pertinent, the jury must not draw inference from that omission, which is especially bizarre when a central witness referred to by almost everyone involved with the case is not called.

Once the case is presented the jury is sequestered in a room and must come to a unanimous decision on each of the charges. This was, for me, the most challenging part of the whole process. Twelve people with unique views on the information presented have to attempt to discuss the evidence and not simply go with their first impressions based on their preconceptions.

The jury is allowed to ask for some evidence to be repeated and if deliberations take some time the jury may be instructed that a majority of 10 to 2 may be accepted. I imagine at some point the jury will run out of time to make a decision and something else will happen but I did not experience this.

Overall the experience was enlightening if not enjoyable, I understand the process a lot more and am happy to have discharged my duty and am equally glad the responsibility will not come around again for at least a few years.

20 October 2015

Dirk Eddelbuettel: Rblpapi 0.3.1

The first update to the Rblpapi package since the initial CRAN upload in August is now available. Rblpapi connects R to the Bloomberg system, giving access to a truly vast array of time series data and custom calculations. This release brings a new beqs() functions to access the results from Bloomberg EQS queries, improves the build system a correct a bug in the retrieval of multiple tick series. The changes are detailed below.

Changes in Rblpapi version 0.3.1 (2015-10-19)

Added beqs() to run Bloomberg Equity Screen Search, based on initial PR #79 by Rademeyer Vermaak, reworked in PRs #83 and #84 by Dirk; closes tickets #63 and #80.

Bloomberg header and library files are now cached locally instead of being re-downloaded for every build (PR #78 by Dirk addressing issue #65).

For multiple ticks, time-stamps are unique-yfied before merging (PR #77 by Dirk addressing issue #76).

A compiler warning under new g++ versions is now suppressed (PR #69 by John, fixing #68).

Courtesy of CRANberries, there is also a diffstat report for the this release. As always, more detailed information is on the Rblpapi. Questions, comments etc should go to the issue tickets system at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

14 October 2015

Norbert Preining: On Lars Wirzenius, Mart n Ferrari, and Debian

Now that the tsunami that followed my recent blog post has passed, I can read through all the nice comments and blogs of other people. I didn t expect that just writing up some links to posts on LKML, plus adding my personal opinion on what has happened and about my interpretation concerning Debian could gather so much attention. Anyway, let us look at the blog posts written in the aftermath. debian-newspeak-coc

In particular do I want to answer to two blogs: Lars Wirzenius blog and Mart n Ferrari s blog, both of which are excellent examples of insults. I also want to mention that although the previous two are bad examples, there are fortunately also those who are able to resist primitive insults, in particular Gunnar Wolf (direct link to the blog does not work), and Miriam Ruiz blog, which contradict my blog and propose a different view, without pulling the knife. Lars Wirzenius Let us start with Lars (ah sorry, I have to call him Wirzenius here, strange custom has suddenly popped up in Debian!):

I think Preining mis-characterises what happens on the Linux kernel mailing list, and in Debian, in free software development in general, and what Sarah Sharp has said and done. I continue to think he is wrong about everything on this topic.

I am very surprised that Lars^WWirzenius is in possession of a crystal ball that allows him to see and evaluate my attitude towards all these items, considering that I have presented links to posts on the LKML, and yes, my interpretation of the matter. Mind that the whole text amounts to about 130 lines on my screen, while my personal opinion was stated in two paragraphs of total 9 lines plus some interspersed comments. And in these maybe 20 lines I didn t make general statements about open source, kernel development etc etc. Lars, I know you have closed your blog for comments so I couldn t ask you but please, can you send me one of your crystal balls if you have more of them? At least he has managed to keep a bit of proper writing and disagreed with my statement on Debian (whether there is fun or not in Debian). He is of course free to do that, but please, don t rob me the right to state that I think Debian has changed. All this dispute centers around people not being capable to distinguish two things: One, being against the Code of Conduct due to the inclusion of administrative actions without clear definitions, and Two, being pro offensive behavior and and insults. Now, dear Lars^WWirzenius, please listen: I never advocated abusive behavior or insults, nor do I defend it. (Did you hear that!) I simply opposed the Code of Conduct as ruling instrument. And what kind of emails I got due to my opposition was far outside the Code of Conduct you are so strongly defending. So please, stay at the facts, and stop insulting me. Thanks. Mart n Ferrari Concerning Mart n I don t have much to say but please, stop spreading lies. You stated:

Once again, he s complaining about how the fun from Debian has been lost because making sexist jokes, or treating other people like shit is not allowed any more.

Could you please come up with a reference to this? Or are you just interpreting? I am very disappointed about this level of discussion between Debian Developers. You not even cared to answer my comment on your blog. Should I say something clear here you should be happy that this has not been written on a Debian mailing list, otherwise I hope the Code of Conduct hammer would hit you.

So yes, it seems that at least these two blogs underline exactly what my opinion is: Communication culture in Debian has changed to be more company-like. Probably due to the ever increasing amount of developers that are paid for their work on Debian (Ubuntu), the pressure to follow US company codes has taken a firm grip, so firm that even stating my diverging opinion is already enough to get branded. Good Future!

Russ Allbery: Review: Magic's Price

Review: Magic's Price, by Mercedes Lackey

Series:	Last Herald Mage #3
Publisher:	DAW
Copyright:	July 1990
ISBN:	0-88677-426-8
Format:	Mass market
Pages:	351

Magic's Price is the conclusion of the Last Herald Mage trilogy (Lackey is from an era of fantasy in which trilogies stopped at three books), but those books are widely spaced and mostly stand alone. You might miss some context around Vanyel's personal life and the politics of the kingdom, but there's a lot of somewhat-awkward in-line summary, so you could read these books out of order if you wanted to. As with the previous two, this is a re-read of a book I've read several times before, but not for more than fifteen years. This is the book that finally introduces Stefan. On this re-read, that surprised me I thought of him as a large part of Vanyel's overall arc but he's only in the third book. He is, of course, key to the grand conclusion of Vanyel's story, beginning as the Bard who has the ability to blunt the pain of the badly-suffering king of Valdemar and then becoming more than that to Van. It's also a book about Vanyel trying desperately to hold things together as the country falls on hard times and far too much weight is put on his shoulders. And a book about a climactic fight against an arch-villain (awkwardly and retroactively inserted). And Vanyel coming to terms with his own foresight. It's a book about rather a lot of things, and therein lies a problem. I thought Magic's Promise had a better story than Magic's Pawn, but both had reasonably coherent stories and some internal flow. Magic's Price has three or four interwoven stories and some odd stops and starts. This is still early in Lackey's writing career (three years after her first novel), and I don't think she was entirely successful in what she was trying to do. The climaxes of the different stories don't quite line up and don't get enough separate attention, leaving a book that lacks clear dramatic shape, an engrossing build-up, or a satisfying climax. The biggest problem I had is that the main threat is not adequately supported. The story is building towards a rather drastic conclusion, one that has been foreshadowed from early in the series, but the threat that creates that conclusion comes out of nowhere. Lackey tries to tie it back to events from earlier in the series, and maybe it was planned all along, but there's little concrete support and no build-up. It feels like a retcon. And the key character in that threat receives almost no characterization whatsoever. I'm a bit grateful for that, since Lackey tends towards total evil in her villains, but to have the cause of all the drama in this story be a complete cipher takes a lot out of the story. (Also, I truly could have done without the rape.) The other irritation is that Vanyel spends quite a lot of the book being a jackass. This is another occasional problem with Lackey's books: her love stories, at least in her early books, involve a bit too much of people being stupid at each other. But I hadn't remembered just how much Van jerked Stef around and treated him miserably, straight to the end of the book. Love is a bit too ordained and imbued with magical powers in Valdemar, which is lucky for Vanyel: extensive plot support is required to explain why Stef didn't find someone else who was less of an ass. In the real world, when someone insists they have no emotional space for you, the correct reaction is "gotcha, have a nice life" and finding someone else to spend time with. This gets partly sorted out in the end, but I didn't find the reconciliation entirely believable, and certainly not fair to Stef. (And the age difference is just a little creepy, although thankfully Lackey doesn't put much focus on that.) On my first read of the Valdemar corpus, this was the book during which I started getting interested in the lore and politics of this universe. There's much more of that to come, thankfully, as I think it's my favorite part of the series. At this point in her writing career, Lackey is getting better at avoiding infodumps and making her magic system interesting. The politics is still a bit black and white, but the Tayledras are a lot of fun (and increasingly less like elves, even if that's the obvious initial inspiration), and the areas beyond Valdemar are developing some more coherence. The conclusion loses a lot from the lack of build-up and emotional support, but it still adds an interesting bit of magical history that will be important later. I wish I'd enjoyed Magic's Price more on re-read, but it was much spottier and more exasperating than I had remembered. That said, it certainly has its moments, particularly around the lore. This is the first book where one starts to realize just how powerful the Companions are, and just how much they're capable of that they're not showing. When I first read it, I loved it. Now, the flaws are too obvious, and Vanyel's behavior is too irritating, for me to unreservedly recommend it, but Magic's Price, and the whole trilogy, is important for building the lore of later books. When I get to Elspeth's part of the series, the politics and magical system will be developed at much greater length. Next, though, my favorite stories in this universe: Tarma and Kethry. Rating: 6 out of 10

12 October 2015

Joachim Breitner: Incredible Proof Machine put to the test

I m currently on the way home, returning from a four-day workshop for high school students; ages 13 to 20, grade 9 to 13. My goal was to teach them something about proofs and logic, in a ambitiously formal and abstract way. For this workshop, I created a computer program called The Incredible Proof Machine that allows the kids to explore creating a proof, by placing blocks on a canvas and drawing connections between them (see my previous blog post for an introduction). Subtracting time spent on on breaks, organizational stuff, the time the student needed to prepare a final presentation, I think we were working for a total of 14 hours, during which we covered propositional logic. I generally let the students explore and try to solve the tasks of one session mostly on their own, followed by a common discussion of what they have just done, what it means why it makes sense etc. The sessions were: assumptions and conclusions in general, with conjunction; implication; disjunction; falsum and tertium non datur. We also briefly discussed paper proofs with the student: how they look, and how they relate to the Proof Machine proofs. We had some lecture notes that we handed out pice-wise after each session. The sections were mildly¹ password-protected to avoid that the quicker students would work ahead, thus keeping the group together. One or two of the 13 students were doing very well and eventually, I gave them the passwords to the predicate logic section and let them work them on them on their own. The quickest managed to solve almost all of these as well, but (as far I as I can tell) without a deeper understanding of the quantifiers, and more a mechanical intuition. As expected, the students were able to solve most of the exercises, even when a proper understanding of the logical semantics was not yet fully developed. This was by design: I believe that this way it was more motivating and encouraging, as they could make it work , compared to a more traditional approach of first throwing a lot of theory at them and then expecting them to apply it. This was confirmed by their feedback after the workshop. I was happy with my implementation. The students immediately could work with it with very few hick-ups, and only one minor bug occurred², which I could fix on the spot. Having everything run on in the browser was a big plus, given that we had no stable internet connection for everyone: Once the Incredible Proof Machine was loaded, the student could continue to work offline. Personally, I find that the UI is occasionally too sluggish, especially on the weaker laptops, but it seems that the students did not seem to mind. Some students tried to connect outputs with outputs or inputs with inputs and the visualization did make it clearly visible that such a link is not actually connected to the block. The predicate logic part is a bit less convincing, with e.g. scoping of local constants not easily understood. I would say that this part would work better if some explanation is given before the students start working on the more involved rules. Our room was equipped with a smartboard, and I was delighted when I found out, mostly by accident, that I could actually use my finger to drag the blocks of the proof and to draw connections. This not only gave me a Minority Report -like feeling, but also meant that it was much easier for the students to follow my explanations when they could just watch my hand, instead of trying to locate the mouse pointer on a regular projector. I m generally doubtful whether such fancy technological toys are useful in the class room, but in this case at least I liked it. The ability to scribble anywhere on the screen was occasionally a plus as well. All in all I believe the Proof Machine was a useful tool, and I am sure that without it, it would have been tricky to have students voluntarily spend 14 hours on such a relatively dry and abstract topic. Given the amount of work that went into the development, I hope that this will not be the last occasion where it is put to good use. So if you have to teach formal logic and natural deduction-style proofs, you are welcome to use the Incredible Proof Machine to get your students excited. It is purely static, i.e. needs no special server-side infrastructure, and you can define your own logic (i.e. proof rules), sessions and tasks. Also, there are a few interesting way in which the Proof Machine could be extended. In particular, I d like it to be able to generate a normal , natural-language proof from a given proof even if it will sound a bit mechanical and then use hover-highlight effects to relate the formulas and sentences in the text proof to the connections and blocks in the graphical proof. Contributions are welcome!

JavaScript based with no crypto or even obfuscation in the code.
Due to differences between the task specification as given in the code, and the task specification when normalizing the syntax of the terms, the proofs that the students created were lost when they re-loaded the page.

7 October 2015

Emanuele Rocca: systemd is your friend

Today I want to talk a bit about some cool features of systemd, the default Debian init system since the release of Jessie. Ubuntu has also adopted systemd in 15.04, meaning that you are going to find it literally everywhere.

Logging The component responsible for logging in systemd is called journal. It collects and stores logs in a structured, indexed journal (hence the name). The journal can replace traditional syslog daemons such as rsyslog and syslog-ng, or work together with them. By default Debian keeps on using rsyslog, but if you don't need to ship logs to a centralized server (or do other fancy things) it is possible to stop using rsyslog right now and rely on systemd-journal instead. The obvious question is: why would anybody use a binary format for logs instead of a bunch of tried and true plain-text files? As it turns out, there are quite a lot of good reasons to do so. The killer features of systemd-journald for me are:

Index tons of logs while being able to search with good performance: O(log(n)) instead of O(n) which is what you get with text files
No need to worry about log rotation anymore, in any shape or form

The last point in particular is really critical in my opinion. Traditional log rotation implementations rely on cron jobs to check how much disk space is used by logs, compressing/removing old files. Log rotation is usually: 1) annoying to configure; 2) hard to get right; 3) prone to DoS attacks. With journald, there is pretty much nothing to configure. Log rotation is built into the daemon disk space allocation logic itself. This also allows to avoid vulnerability windows due to time-based rotation, which is what you get with logrotate and friends. Enough high-level discussions though, here is how to use the journal! Check if you already have the directory /var/log/journal, otherwise create it (as root). Then restart systemd-journald as follows: sudo systemctl restart systemd-journald You can get all messages produced since the last boot with journalctl -b. All messages produced today can get extracted using journalctl --since=today. Want to get all logs related to ssh? Try with journalctl _SYSTEMD_UNIT=ssh.service. There are many more filtering options available, you can read all about them with man journalctl. journald's configuration file is /etc/systemd/journald.conf. Two of the most interesting options are SystemMaxUse and SystemKeepFree, which can be used to change the amount of disk space dedicated to logging. They default to 10% and 15% of the /var filesystem respectively. Here is a little cheatsheet:

journalctl -b                # Show all messages since last boot
journalctl -f                # Tail your logs
journalctl --since=yesterday # Show all messages produced since yesterday
journalctl -pcrit            # Filter messages by priority
journalctl /bin/su           # Filter messages by program
journalctl --disk-usage      # The amount of space in use for journaling

4 October 2015

Lunar: Reproducible builds: week 23 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes

Barry Warsaw uploaded wheel/0.26.0-1 which now uses SOURCE_DATE_EPOCH instead of WHEEL_FORCE_TIMESTAMP and uses time.gmtime() to avoid timezone issues. Patches by Chris Lamb and Reiner Herrmann.

Andreas Metzler uploaded autogen/1:5.18.6-1 in experimental with several patches for reproducibility issues written by Valentin Lorentz. Groovy upstream has merged a change proposed by Emmanuel Bourg to remove timestamps generated by groovydoc. Ben Hutchings submitted a patch to add support for SOURCE_DATE_EPOCH in linux-kbuild as an alternate way to specify the build timestamp. Reiner Herrman has sent a patch adding support for SOURCE_DATE_EPOCH in docbook-utils. Packages fixed The following packages became reproducible due to changes in their build dependencies: commons-csv. fest-reflect, sunxi-tools, xfce4-terminal, The following packages became reproducible after getting fixed:

httpcomponents-client/4.5.1-1 by Emmanuel Bourg.
jhead/1:3.00-2 by Ludovic Rousseau.
libvigraimpex/1.10.0+dfsg-10 by Daniel Stender.
linux/4.2-1~exp1 by Ben Hutchings.
maelstrom/1.4.3-L3.0.6+main-7 by Santiago Vila.
nedit/1:5.6a-3 by Paul Gevers.
pitivi/0.94-4 by Sebastian Dr ge, reported by Scott Kitterman.
procenv/0.40-2 by James Hunt.
seyon/2.20c-32 by Santiago Vila.
slib/3b1-5 by Santiago Vila.
spock/0.7-groovy-2.0-1 by Emmanuel Bourg.
u-boot/2015.10~rc4+dfsg1-1 by Vagrant Cascadian.
vdr-plugin-remote/0.7.0-1 by Tobias Grimm.

Some uploads fixed some reproducibility issues but not all of them:

dutch/1:2.10-4 uploaded by Thijs Kinkhorst, original patch by Chris Lamb.

Patches submitted which have not made their way to the archive yet:

#800776 on cluster-glue: exports SOURCE_DATE_EPOCH in debian/rules.

Tomasz Rybak uploaded pycuda/2015.1.3-1 which should fix reproducibility issues. The package has not been tested as it is in contrib. akira found an embedded code copy of texi2html in fftw. reproducible.debian.net Email notifications are now only sent once a day per package, instead of on each status change. (h01ger) disorderfs has been temporarily disabled to see if it had any impact on the disk space issues. (h01ger) When running out of disk space, build nodes will now automatically detect the problem. This means test results will not be recorded as FTBFS and the problem will be reported to Jenkins maintainers. (h01ger) The navigation menu of package pages has been improved. (h01ger) The two amd64 builders now use two different kernel versions: 3.16 from stable and 4.1 from backports on the other. (h01ger) We now graph the number of packages which needs to be fixed. (h01ger) Munin now creates graphs on how many builds were performed by build nodes (example). (h01ger) A migration plan has been agreed with DSA on how to turn Jenkins into an official Debian service. A backport of jenkins-job-builder for Jessie is currently missing. (h01ger) Package reviews 119 reviews have been removed, 103 added and 45 updated this week. 16 fail to build from source issues were reported by Chris Lamb and Mattia Rizzolo. New issue this week: timestamps_in_manpages_generated_by_docbook_utils. Misc. Allan McRae has submitted a patch to make ArchLinux pacman record a .BUILDINFO file.

27 September 2015

Niels Thykier: There is nothing like (missing) iptables (rules) to make you use tor

I have been fiddling with setting up both iptables and tor on my local machine. Most of it was fairly easy to do, once I dedicated the time to actually do it. Configuring both at the same time also made things easier for me, but YMMV. Regardless, it did take quite a while researching, tweaking and testing most of that time was spent on the iptables front for me. I ended up doing this incrementally. The major 5 steps I went through were:

Created a basic incoming (INPUT) firewall enforcing
Installed tor + torsocks and aliased a few commands to run with torsocks
Created a basic outgoing (OUTPUT) firewall permissive
Make the outgoing firewall enforcing
Migrate the majority of programs and services to use tor.

Some of these overlapped time-wise and I certainly revisited the configuration a couple of times. A couple of things, that I learned:

You probably want to have a look at netstat --listen -put --numeric when you write your INPUT firewall.
The tor developers have tried a lot to make things easy. It is scary how often torsocks program [args] just works(tm).
- That said, it does not always work.
Tor and iptables (OUTPUT) can have a synergy effect on each other.
- Notably, when it is easier to just torsocks a program than adding the necessary iptables rules.
Writing iptables rules become a lot easier once:
- You learn how to iptables s LOG rule
- You use sensible-editor + iptables-restore or something like puppet s firewall module

Filed under: Debian

24 September 2015

Joachim Breitner: The Incredible Proof Machine

In a few weeks, I will have the opportunity to offer a weekend workshop to selected and motivated high school students¹ to a topic of my choice. My idea is to tell them something about logic, proofs, and the joy of searching and finding proofs, and the gratification of irrevocable truths. While proving things on paper is already quite nice, it is much more fun to use an interactive theorem prover, such as Isabelle, Coq or Agda: You get immediate feedback, you can experiment and play around if you are stuck, and you get lots of small successes. Someone² once called interactive theorem proving the worlds most geekiest videogame . Unfortunately, I don t think one can get high school students without any prior knowledge in logic, or programming, or fancy mathematical symbols, to do something meaningful with a system like Isabelle, so I need something that is (much) easier to use. I always had this idea in the back of my head that proving is not so much about writing text (as in normally written proofs) or programs (as in Agda) or labeled statements (as in Hilbert-style proofs), but rather something involving facts that I have proven so far floating around freely, and way to combine these facts to new facts, without the need to name them, or put them in a particular order or sequence. In a way, I m looking for labVIEW wrestled through the Curry-Horward-isomorphism. Something like this:

A proof of implication currying

So I set out, rounded up a few contributors (Thanks!), implemented this, and now I proudly present: The Incredible Proof Machine ³ This interactive theorem prover allows you to do perform proofs purely by dragging blocks (representing proof steps) onto the paper and connecting them properly. There is no need to learn syntax, and hence no frustration about getting that wrong. Furthermore, it comes with a number of example tasks to experiment with, so you can simply see it as a challenging computer came and work through them one by one, learning something about the logical connectives and how they work as you go. For the actual workshop, my plan is to let the students first try to solve the tasks of one session on their own, let them draw their own conclusions and come up with an idea of what they just did, and then deliver an explanation of the logical meaning of what they did. The implementation is heavily influenced by Isabelle: The software does not know anything about, say, conjunction ( ) and implication ( ). To the core, everything is but an untyped lambda expression, and when two blocks are connected, it does unification⁴ of the proposition present on either side. This general framework is then instantiated by specifying the basic rules (or axioms) in a descriptive manner. It is quite feasible to implement other logics or formal systems on top of this as well. Another influence of Isabelle is the non-linear editing: You neither have to create the proof in a particular order nor have to manually manage a proof focus . Instead, you can edit any bit of the proof at any time, and the system checks all of it continuously. As always, I am keen on feedback. Also, if you want to use this for your own teaching or experimenting needs, let me know. We have a mailing list for the project, the code is on GitHub, where you can also file bug reports and feature requests. Contributions are welcome! All aspects of the logic are implemented in Haskell and compiled to JavaScript using GHCJS, the UI is plain hand-written and messy JavaScript code, using JointJS to handle the graph interaction. Obviously, there is still plenty that can be done to improve the machine. In particular, the ability to create your own proof blocks, such as proof by contradiction, prove them to be valid and then use them in further proofs, is currently being worked on. And while the page will store your current progress, including all proofs you create, in your browser, it needs better ways to save, load and share tasks, blocks and proofs. Also, we d like to add some gamification, i.e. achievements ( First proof by contradiction , 50 theorems proven ), statistics, maybe a share theorem on twitter button. As the UI becomes more complicated, I d like to investigating moving more of it into Haskell world and use Functional Reactive Programming, i.e. Ryan Trickle s reflex, to stay sane. Customers who liked The Incredible Proof Machine might also like these artifacts, that I found while looking whether something like this exists:

Easyprove, an interactive tool to create textual proofs by clicking on rules.
Domino On Acid represents natural deduction rules in propositional logic with and as a game of dominoes.
Proofscape visualizes the dependencies between proofs as graphs, i.e. it operates on a higher level than The Incredible Proof Machine.
Proofmood is a nice interactive interface to conduct proofs in Fitch-style.
Proof-Game represents proofs trees in a sequent calculus with boxes with different shapes that have to match.
JAPE is an editor for proofs in a number of traditional proof styles. (Thanks to Alfio Martini for the pointer.)
Logitext, written by Edward Z. Yang, is an online tool to create proof trees in sequent style, with a slick interface, and is even backed by Coq! (Thanks to Lev Lamberov for the pointer.)
Carnap is similar in implementation to The Incredible Proof Machine (logical core in Haskell, generic unification-based solver). It currently lets you edit proof trees, but there are plans to create something more visual.
Clickable Proofs is a (non-free) iOS app that incorporates quite a few of the ideas that are behind The Incredible Proof Machine. It came out of a Bachelor s thesis of Tim Selier and covers propositional logic.
Euclid the game by Kasper Peulen is a nice game to play with geometric constructions.

Students with migration background supported by the START scholarship
Does anyone know the reference?
We almost named it Proofcraft , which would be a name our current Minecraft-wild youth would appreciate, but it is alreay taken by Gerwin Kleins blog. Also, the irony of a theorem prover being in-credible is worth something.
Luckily, two decades ago, Tobias Nipkow published a nice implementation of higher order pattern unification as ML code, which I transliterated to Haskell for this project.

22 September 2015

Clint Adams: 404 No forwarding address for Ryan Hockert-Lotz in Fall River, MA

[As You Like It, Act III, Scene V] Hey Beautiful I thought of you on the Fourth of July; which does not, in and of itself, distinguish the day from any other since I met you. It was remarkable only in that it was justifiable, given our conversation about fireworks displays. Of course, I'm happy to take the flimsiest of displays as an excuse to mark you. I'm home alone right now, after watching The Bad Seed. Truth be told, the shadows keep spooking me. I can't seem to stop myself from imagining precocious blonde murderers in them. It's a manageable silliness, but made a little less so by the fact that I forgot to lock the door. Less troublesome than the night I spent after Ringu (the original Japanese version of The Ring). I finished watching it in the wee hours of the morning, and I wanted to go to sleep. I was in half a stupor, but the incessant inner critic in me kept imagining all the changes that could have been made to make the movie more truly unsettling until visions of Obake were swimming around me. Ordinarily I doubt I'd be bothered particularly by a 50's classic, but went back up to Boston this afternoon. 's at a conference in Finland, so I invited him down to visit while I have the apartment all to myself. It was strange to have a visitor actually in my home for whom I didn't have to play at being contented. At any rate, being around for three days essentially meant carrying on a three-day-long conversaiton, and the abrupt drop of sociability makes me feel my isolation a little more acutely. We watched The Way We Were together, and agreed that it should be remade with casting that actually works. I hadn't seen it before, and was surprised to find it unusually nuanced and substantial, yet still not good. It was nice to have someone around who would dissect it with me afterward. It's been a while since I actually discussed a film with someone. Partly out of my own fault; I don't always enjoy verbalizing my opinions of movies immediately after watching them if I've found them in the least bit moving. I guess I consider the aftertaste part of the experience. In this case, we both felt the film had missed its emotional mark so it wasn't so much of an issue. On the other hand, I don't find most movies moving. I find them frustratingly flawed, and by the time they end I'm raring to rant about their petty contradictions and failures of logic. I think it might give people the impression that I don't actually make any effort to tease out the messages filmmakers weave into their work. Or maybe I'm just making excuses for having uninteresting friends. Either way, it was pleasant to be in the company of someone eager to tolerate the convolutions of my thought process. On Wednesday night, I have a date to meet up with some former co-workers/friends that I've been passively avoiding for several years now. Every time I fail to carefully manage my visibility, people seem to come flooding back into my life. This time the culprit was a day spent logged into instant messaging without stringent privacy settings. I should feel lucky for that, I suppose. I'm not sure how I actually feel. One of the formers is a woman I was very close to, as far as most of the world her included could discern. The other is a Boston boy I admired for the touch of golden child in the air that hung about him. The main theme of his life was (and I suspect still is) getting to drinks with friends at one of his regular bars at the end of every evening. Which did not at all stop him from being productive, interested in the world, and bright. If he had been a girl I would have been hateful with envy. Instead he's always stood out in my memory as the only person I've had a bit of a crush on despite not finding him particularly intellectually stimulating. A month or so ago he sold his company to google. Now he spends a lot of time out of town giving lectures. I suspect I may be generally happy for him, and I'm not quite sure what to do with that. Thursday I'm leaving for a few days in Denver. I wish I hadn't scheduled it for a time when I could have been the sole occupant of my domicile, but other than that I'm looking forward to it. I've no idea what I'll do there, but at least that means I really am going someplace that wouldn't occur to me outside of a peculiar set of constraints. I think it would be advisable to work out the transportation system before I depart, though. I hope this letter finds you relatively satisfied, at a minimum. I don't actually need to tell you how much I miss talking to you, do I? You're wonderful. Affectionally, as always,

21 September 2015

Lunar: Reproducible builds: week 21 in Stretch cycle

If you see someone on the Debian ReproducibleBuilds project, buy him/her a beer. This work is awesome.

What happened in the reproducible builds effort this week: Media coverage Nathan Willis covered our DebConf15 status update in Linux Weekly News. Access to non-LWN subscribers will be given on Thursday 24th. Linux Journal published a more general piece last Tuesday. Unexpected praise for reproducible builds appeared this week in the form of several iOS applications identified as including spyware. The malware was undetected by Apple screening. This actually happened because application developers had simply downloaded a trojaned version of XCode through an unofficial source. While reproducible builds can't really help users of non-free software, this is exactly the kind of attacks that we are trying to prevent in our systems. Toolchain fixes

Mathieu Malaterre uploaded abi-compliance-checker/1.99.11-1 which drops the timestamps from the generated HTML reports and makes the generated .abi.tar.gz files reproducible. Original patches by Chris Lamb.

Niko Tyni wrote and uploaded a better patch for the source order problem in libmodule-build-perl. Tristan Seligmann identified how the code generated by python-cffi could be emitted in random order in some cases. Upstream has already fixed the problem. Packages fixed The following 24 packages became reproducible due to changes in their build dependencies: apache-curator, checkbox-ng, gant, gnome-clocks, hawtjni, jackrabbit, jersey1, libjsr305-java, mathjax-docs, mlpy, moap, octave-geometry, paste, pdf.js, pyinotify, pytango, python-asyncssh, python-mock, python-openid, python-repoze.who, shadow, swift, tcpwatch-httpproxy, transfig. The following packages became reproducible after getting fixed:

apparmor/2.10-2 uploaded by intrigeri, fixed upstream by Christian Boltz, with the same change suggested by Reiner Herrmann.
ardour/1:4.2~dfsg-2 by IOhannes m zm lnig.
dcmtk/3.6.1~20150629-1 uploaded by Andreas Tille, original patch by akira.
deap/1.0.1-4 by Daniel Stender.
firebird2.5/2.5.4.26856.ds4-2 by Damyan Ivanov.
gamera/3.4.2+svn1437-1 by Daniel Stender.
genometools/1.5.7-1 by Sascha Steinbiss.
golang-github-go-xorm-core/0.4.4-1 by Alexandre Viau.
klibc/2.0.4-4 by Ben Hutchings.
libgtk2-perl/2:1.2496-3 by intrigeri.
lsof/4.89+dfsg-0.1 uploaded by Laurent Bigonville, original patch by Lunar.
monotone/1.1-6 by Markus Wanner.
ndisc6/1.0.1-4 by Santiago Vila.
privoxy/3.0.23-4 by Roland Rosenfeld.
ruby-flexmock/2.0.0~rc1-1 by Antonio Terceiro.
ruby-html2haml/2.0.0-1 by Lunar.
tunnelx/20140102-3 uploaded by Wookey, original patch by Chris Lamb.
wtforms/2.0.2-1 by Orestis Ioannou, original patch by Juan Picca.

Some uploads fixed some reproducibility issues but not all of them:

maxima/5.37-1 by Camm Maguire, report by akira.

Patches submitted which have not made their way to the archive yet:

#783152 on kmod by Lunar: export SOURCE_DATE_EPOCH in debian/rules.
#799010 on 389-ds-base by Chris Lamb: use SOURCE_DATE_EPOCH value as the build date.
#799206 on python-sqlalchemy-utils by Chris Lamb: sort the list of extra requirement.
#799330 on cappuccino by Chris Lamb: pass a fixed seed to polygen.
#799410 on segment by Chris Lamb: use date of the latest debian/changelog entry as build date.

reproducible.debian.net Tests for Coreboot, OpenWrt, NetBSD, and FreeBSD now runs weekly (instead of monthly). diffoscope development Python 3 offers new features (namely yield from and concurrent.futures) that could help implement parallel processing. The clear separation of bytes and unicode strings is also likely to reduce encoding related issues. Mattia Rizolo thus kicked the effort of porting diffoscope to Python 3. tlsh was the only dependency missing a Python 3 module. This got quickly fixed by a new upload. The rest of the code has been moved to the point where only incompatibilities between Python 2.7 and Pyhon 3.4 had to be changed. The commit stream still require some cleanups but all tests are now passing under Python 3. Documentation update The documentation on how to assemble the weekly reports has been updated. (Lunar) The example on how to use SOURCE_DATE_EPOCH with CMake has been improved. (Ben Beockel, Daniel Kahn Gillmor) The solution for timestamps in man pages generated by Sphinx now uses SOURCE_DATE_EPOCH. (Mattia Rizzolo) Package reviews 45 reviews have been removed, 141 added and 62 updated this week. 67 new FTBFS reports have been filled by Chris Lamb, Niko Tyni, and Lisandro Dami n Nicanor P rez Meyer. New issues added this week: randomness_in_r_rdb_rds_databases, python-ply_compiled_parse_tables. Misc. The prebuilder script is now properly testing umask variations again. Santiago Villa started a discussion on debian-devel on how binNMUs would work for reproducible builds.

19 September 2015

Russ Allbery: Review: Magic's Pawn

Review: Magic's Pawn, by Mercedes Lackey

Series:	Last Herald Mage #1
Publisher:	DAW
Copyright:	June 1989
ISBN:	0-88677-352-0
Format:	Mass market
Pages:	349

Continuing my (very slow) re-read of Mercedes Lackey's Valdemar series. I've read all these books before (this series several times, in fact), but long before I started writing reviews. Vanyel is a legendary hero by the time of Talia (the hero of the first Valdemar series, starting with Arrows of the Queen). Talia was reading stories about him at the start of her story: the mage Herald who defended Valdemar against its enemies in the distant past. This is the first book of the trilogy that tells his story, starting (as Lackey's books often do) from rather inauspicious beginnings. Vanyel is the son of a border lord, and about as poorly suited for it as possible. He's small, pretty, entirely uninterested in the hammer-and-tongs sword fighting the arms master wants to teach him, and also gay, not that he has any idea that's even something that exists and has a name. His only ally was his sister, who is now off serving in Valdemar's military. Most of his life is spent hiding, feeling utterly lost and out of place, and wishing he could be a Bard. This is a Valdemar story, so if you're familiar with the series, you know what's coming next. What might come as a surprise is that "next" is quite some time into the story. Vanyel does not get rescued from his situation by a Companion. Instead, his father decides to exile him to the capital under the care of his aunt, who he's only met once and who didn't think highly of him. He expects to be even more miserable, and shuts emotionally down in anticipation. This is one of those books that I remembered as being better than it actually was, and one of the reasons why I enjoyed it less than I expected is that far too much of this book is devoted to describing Vanyel's mental state. This usually involves various elaborate emo analogies (which can be a failing of this series in general), and it's quite hard to maintain sympathy for Vanyel. Lackey gets us to feel for him at the start of the book, when he's really in a bad situation. But once he gets to the capital, he's his own worst enemy, and it's hard to avoid the desire to shake him. I also didn't remember just how much I disliked Tylendel. I'm sure it will come as a surprise to no one that Vanyel eventually meets someone who draws him out of his shell and gives him a reason to want to live. Unfortunately, that person, despite a positive surface impression, is self-obsessed, unstable, and not above manipulating Vanyel into actions that are so obviously catastrophic that it makes one want to yell at the book. I disliked this part of the book so much that even a Companion's choosing was overshadowed. The book does get a bit better after that truly awful middle, but it never hits the emotional stride that other Valdemar books hit. Lackey does introduce the Tayledras, which will be hugely important in later books in this series (and who are some of my favorite characters), but there too I prefer their later appearances. Vanyel's inability to see a good thing when it hits him in the face, punctuated by being occasionally cruel to the people who try to help him, makes it quite hard to enjoy his slow path to becoming a better person. I remember really liking this trilogy, so I think it gets better. But I also remembered liking Vanyel's claiming, and it didn't do much for me on a re-read. I can only recommend this one as a necessary preface to the later books in the trilogy, and expect to need a high tolerance for constant woe-is-me despair. Followed by Magic's Promise. Rating: 5 out of 10

17 September 2015

Julien Danjou: My interview in le Journal du Hacker

A few days ago, the French equivalent of Hacker News, called "Le Journal du Hacker", interviewed me about my work on OpenStack, my job at Red Hat and my self-published book The Hacker's Guide to Python. I've spent some time translating it into English so you can read it if you don't understand French! I hope you'll enjoy it.

Hi Julien, and thanks for participating in this interview for the Journal du Hacker. For our readers who don't know you, can you introduce you briefly?

You're welcome! My name is Julien, I'm 31 years old, and I live in Paris. I now have been developing free software for around fifteen years. I had the pleasure to work (among other things) on Debian, Emacs and awesome these last years, and more recently on OpenStack. Since a few months now, I work at Red Hat, as a Principal Software Engineer on OpenStack. I am in charge of doing upstream development for that cloud-computing platform, mainly around the Ceilometer, Aodh and Gnocchi projects.

Being myself a system architect, I follow your work in OpenStack since a while. It's uncommon to have the point of view of someone as implied as you are. Can you give us a summary of the state of the project, and then detail your activities in this project?

The OpenStack project has grown and changed a lot since I started 4 years ago. It started as a few projects providing the basics, like Nova (compute), Swift (object storage), Cinder (volume), Keystone (identity) or even Neutron (network) who are basis for a cloud-computing platform, and finally became composed of a lot more projects. For a while, the inclusion of projects was the subject of a strict review from the technical committee. But since a few months, the rules have been relaxed, and we see a lot more projects connected to cloud-computing joining us. As far as I'm concerned, I've started with a few others people the Ceilometer project in 2012, devoted to handling metrics of OpenStack platforms. Our goal is to be able to collect all the metrics and record them to analyze them later. We also have a module providing the ability to trigger actions on threshold crossing (alarm). The project grew in a monolithic way, and in a linear way for the number of contributors, during the first two years. I was the PTL (Project Technical Leader) for a year. This leader position asks for a lot of time for bureaucratic things and people management, so I decided to leave my spot in order to be able to spend more time solving the technical challenges that Ceilometer offered. I've started the Gnocchi project in 2014. The first stable version (1.0.0) was released a few months ago. It's a timeseries database offering a REST API and a strong ability to scale. It was a necessary development to solve the problems tied to the large amount of metrics created by a cloud-computing platform, where tens of thousands of virtual machines have to be metered as often as possible. This project works as a standalone deployment or with the rest of OpenStack. More recently, I've started Aodh, the result of moving out the code and features of Ceilometer related to threshold action triggering (alarming). That's the logical suite to what we started with Gnocchi. It means Ceilometer is to be split into independent modules that can work together with or without OpenStack. It seems to me that the features provided by Ceilometer, Aodh and Gnocchi can also be interesting for operators running more classical infrastructures. That's why I've pushed the projects into that direction, and also to have a more service-oriented architecture (SOA)

I'd like to stop for a moment on Ceilometer. I think that this solution was very expected, especially by the cloud-computing providers using OpenStack for billing resources sold to their customers. I remember reading a blog post where you were talking about the high-speed construction of this brick, and features that were not supposed to be there. Nowadays, with Gnocchi and Aodh, what is the quality of the brick Ceilometer and the programs it relies on?

Indeed, one of the first use-case for Ceilometer was tied to the ability to get metrics to feed a billing tool. That's now a reached goal since we have billing tools for OpenStack using Ceilometer, such as CloudKitty. However, other use-cases appeared rapidly, such as the ability to trigger alarms. This feature was necessary, for example, to implement the auto-scaling feature that Heat needed. At the time, for technical and political reasons, it was not possible to implement this feature in a new project, and the functionality ended up in Ceilometer, since it was using the metrics collected and stored by Ceilometer itself. Though, like I said, this feature is now in its own project, Aodh. The alarm feature is used since a few cycles in production, and the Aodh project brings new features on the table. It allows to trigger threshold actions and is one of the few solutions able to work at high scale with several thousands of alarms. It's impossible to make Nagios run with millions of instances to fetch metrics and triggers alarms. Ceilometer and Aodh can do that easily on a few tens of nodes automatically. On the other side, Ceilometer has been for a long time painted as slow and complicated to use, because its metrics storage system was by default using MongoDB. Clearly, the data structure model picked was not optimal for what the users were doing with the data. That's why I started Gnocchi last year, which is perfectly designed for this use case. It allows linear access time to metrics (O(1) complexity) and fast access time to the resources data via an index. Today, with 3 projects having their own perimeter of features defined and which can work together Ceilometer, Aodh and Gnocchi finally erased the biggest problems and defects of the initial project.

To end with OpenStack, one last question. You're a Python developer for a long time and a fervent user of software testing and test-driven development. Several of your blogs posts point how important their usage are. Can you tell us more about the usage of tests in OpenStack, and the test prerequisites to contribute to OpenStack?

I don't know any project that is as tested on every layer as OpenStack is. At the start of the project, there was a vague test coverage, made of a few unit tests. For each release, a bunch of new features were provided, and you had to keep your fingers crossed to have them working. That's already almost unacceptable. But the big issue was that there was also a lot of regressions, et things that were working were not anymore. It was often corner cases that developers forgot about that stopped working. Then the project decided to change its policy and started to refuse all patches new features or bug fix that would not implement a minimal set of unit tests, proving the patch would work. Quickly, regressions were history, and the number of bugs largely reduced months after months. Then came the functional tests, with the Tempest project, which runs a test battery on a complete OpenStack deployment. OpenStack now possesses a complete test infrastructure, with operators hired full-time to maintain them. The developers have to write the test, and the operators maintain an architecture based on Gerrit, Zuul, and Jenkins, which runs the test battery of each project for each patch sent. Indeed, for each version of a patch sent, a full OpenStack is deployed into a virtual machine, and a battery of thousands of unit and functional tests is run to check that no regressions are possible. To contribute to OpenStack, you need to know how to write a unit test the policy on functional tests is laxer. The tools used are standard Python tools, unittest for the framework and tox to run a virtual environment (venv) and run them. It's also possible to use DevStack to deploy an OpenStack platform on a virtual machine and run functional tests. However, since the project infrastructure also do that when a patch is submitted, it's not mandatory to do that yourself locally.

The tools and tests you write for OpenStack are written in Python, a language which is very popular today. You seem to like it more than you have to, since you wrote a book about it, The Hacker's Guide to Python, that I really enjoyed. Can you explain what brought you to Python, the main strong points you attribute to this language (quickly) and how you went from developer to author?

I stumbled upon Python by chance, around 2005. I don't remember how I hear about it, but I bought a first book to discover it and started toying with that language. At that time, I didn't find any project to contribute to or to start. My first project with Python was rebuildd for Debian in 2007, a bit later. I like Python for its simplicity, its object orientation rather clean, its easiness to be deployed and its rich open source ecosystem. Once you get the basics, it's very easy to evolve and to use it for anything, because the ecosystem makes it easy to find libraries to solve any kind of problem. I became an author by chance, writing blog posts from time to time about Python. I finally realized that after a few years studying Python internals (CPython), I learned a lot of things. While writing a post about the differences between method types in Python which is still one of the most read post on my blog I realized that a lot of things that seemed obvious to me where not for other developers. I wrote that initial post after thousands of hours spent doing code reviews on OpenStack. I, therefore, decided to note all the developers pain points and to write a book about that. A compilation of what years of experience taught me and taught to the other developers I decided to interview in the book.

I've been very interested by the publication of your book, for the subject itself, but also the process you chose. You self-published the book, which seems very relevant nowadays. Is that a choice from the start? Did you look for an editor? Can you tell use more about that?

I've been lucky to find out about others self-published authors, such as Nathan Barry who even wrote a book on that subject, called Authority. That's what convinced me it was possible and gave me hints for that project. I've started to write in August 2013, and I ran the firs interviews with other developers at that time. I started to write the table of contents and then filled the pages with what I knew and what I wanted to share. I manage to finish the book around January 2014. The proof-reading took more time than I expected, so the book was only released in March 2014. I wrote a complete report on that on my blog, where I explain the full process in detail, from writing to launching. I did not look for editors though I've been proposed some. The idea of self-publishing really convince me, so I decided to go on my own, and I have no regret. It's true that you have to wear two hats at the same time and handle a lot more things, but with a minimal audience and some help from the Internet, anything's possible! I've been reached by two editors since then, a Chinese and Korean one. I gave them rights to translate and publish the books in their countries, so you can buy the Chinese and Korean version of the first edition of the book out there. Seeing how successful it was, I decided to launch a second edition in Mai 2015, and it's likely that a third edition will be released in 2016.

Nowadays, you work for Red Hat, a company that represents the success of using Free Software as a commercial business model. This company fascinates a lot in our community. What can you say about your employer from your point of view?

It only has been a year since I joined Red Hat (when they bought eNovance), so my experience is quite recent. Though, Red Hat is really a special company on every level. It's hard to see from the outside how open it is, and how it works. It's really close to and it really looks like an open source project. For more details, you should read The Open Organization, a book wrote by Jim Whitehurst (CEO of Red Hat), which he just published. It describes perfectly how Red Hat works. To summarize, meritocracy and the lack of organization in silos is what makes Red Hat a strong organization and puts them as one of the most innovative company. In the end, I'm lucky enough to be autonomous for the project I work on with my team around OpenStack, and I can spend 100% working upstream and enhance the Python ecosystem.

16 September 2015

Russ Allbery: Review: The Goblin Emperor

Review: The Goblin Emperor, by Katherine Addison

Publisher:	Tor
Copyright:	April 2014
ISBN:	1-4299-4640-7
Format:	Kindle
Pages:	446

Maia is the child of the current emperor of the Elflands, but not a favored one. He's the half-goblin son of a political marriage to a goblin princess, quickly orphaned away from the capital and kept in seclusion well out of the way of his father and court politics. And in the care of his cousin Setheris: cruel and abusive companionship. Then the airship Wisdom of Choharo crashes with no survivors, killing everyone in the line of succession. Except Maia. Maia is in no way prepared to be emperor, but the rules of succession are clear: he is now emperor. We all know from medieval history, both real and fantastic, that clear rules rarely got in the way of aggressive political maneuvering, but Maia's short-term survival is helped considerably by complex factions at play at court (of which he is almost wholly ignorant), and a certain lack of bloodthirstiness in elven politics. Rather than disposing of the unexpected heir, people instead start maneuvering to manipulate and control him. The current Lord Chancellor is the first, beginning his play in the message sent informing Maia of his father's death, and his cousin isn't far behind, but they're only the first two. Meanwhile, Maia, who had never been trained for any sort of court life, has to learn nearly everything about his new life: how to interact with servants, the rituals of court, his new duties, and all of the factions in play. One might expect that Maia would quickly be reduced to a pawn of one faction or another, but he has two secret weapons. One is that he is a legitimately nice person with a strongly developed sense of empathy. Early on, he lucks into some opportunities to show this and wins some popularity and some allies of his own, particularly among those people not used to being treated with respect. And two, he's had enough of his cousin controlling and abusing him. He's determined to make his own decisions and not take orders from someone else, and now has enough nominal power to possibly pull this off. The Goblin Emperor is a fantasy of political agency, and a story about political maneuvering, tricky decisions, navigating alliances, and learning politics. There isn't a lot of action here: most of the plot happens in conversation. The tension is high Maia's situation is precarious but it mostly involves second-guessing his decisions and actions, and hoping that his weaknesses and ignorance don't get him into too much trouble. That is an extremely tricky book to write. Addison had to walk a tightrope between letting Maia's new responsibilities come unrealistically easily, making him seem like a Chosen One with special plot powers, and having readers get too frustrated with realistically bad decisions and lose their desire to read about him. Addison walks that tightrope magnificently. I liked Maia throughout this book, and that's the strength of it. He never felt like the typical fantasy coming of age hero who is magically protected by the plot. He's just a genuine, honest person, reacting to his own past as best he can, making the best decisions he knows how to make. He makes mistakes, but Addison wisely avoids making those mistakes cringe-worthy or too obvious. Instead, they're like bad moves in a chess match: they create problems but also some opportunities, and they're realistically recoverable. And it's a delight to see Maia bring empathy and a disinterested outside perspective to the hothouse mess of court politics and come up with some very creative solutions. Obviously, the situation that allows Maia to succeed is artificial. This is a fantasy: Maia happens to be just the right person the empire needs, and that's clearly an authorial construction. It may be impossible to write a political story about a well-meaning, naive outsider that isn't a tragedy without a bit of cheating. But Addison hides the cheating well. Maia may be very lucky in the empire he inherits, but it feels coherent and believable. By the end of the book, Maia has dealt with everything from international diplomacy to the incredibly complex negotiations around further heirs to the empire. And (realistically) he doesn't stay lucky throughout the book: eventually, no matter how careful he tries to be, he angers some powerful people. I enjoyed every step along the way, and would have happily read more about Maia. In my personal ranking, this was second only to Ancillary Sword in the nominees for the 2015 Hugo, and I wouldn't disagree too strongly with someone who argued it was better. It's not a typical sort of fantasy, but it's one I enjoyed very much. Recommended. One warning: consider going to the back of the book and reading the glossary at the start. There may be some mild spoilers in there, but it also explains the title conventions in the empire, and some of those titles look very much like proper names. If, like me, you don't read this until the end of the book, you may find yourself horribly confused by several apparently similarly-named characters. Rating: 8 out of 10

Next.

Previous.