Search Results: "reg"

30 October 2015

Dirk Eddelbuettel: littler 0.3.0 -- on CRAN !!

max-heap image A new major release of littler is now available. And for the first time in the nine years since 2006 when Jeff started the effort (which I joined not long after) we are now a CRAN package. This required a rewrite of the build system, foregoing the calls to aclocal, autoheader, automake and leaving just a simpler autoconf layer creating a configure script and a simple src/Makevars.in. Not only does R provide such a robust and well-understood build system (which I got to understand reasonably well given all my R packages, but being on CRAN and leveraging its mechanism for installation and upgrades is clearly worth the change. There may be a moment or two of transition. While we can create a binary in an R package, we cannot (generally) copy to /usr/bin or /usr/local/bin as part of the build process (for lack of write-rights to those directories). So if you do not have r in the $PATH and load the package, it makes a suggestion (which needs a linebreak which I added here): 
R> library(littler)
The littler package provides 'r' as a binary.
You could link to the 'r' binary installed in
'/usr/local/lib/R/site-library/littler/bin/r' from
'/usr/local/bin' in order to use 'r' for scripting.
R>
Similarly, you could copy (or softlink) r to ~/bin if that is in your $PATH. The Debian (and Ubuntu) packages will continue to provide /usr/bin/r as before. Note thah these packages will now be called r-cran-littler to match all other CRAN packages. The NEWS file entry is below.
Changes in littler version 0.3.0 (2015-10-29)
  • Changes in build system
    • First CRAN Release as R package following nine years of source releases
    • Script configure, src/Makevars.in and remainder of build system rewritten to take advantage of the R package build infrastructure
    • Reproducible builds are better supported as the (changing) compilation timestamps etc are only inserted for 'verbose builds' directly off the git repo, but not for Debian (or CRAN) builds off the release tarballs
  • Changes in littler functionality
    • Also source $R_HOME/etc/Rprofile.site and ~/.Rprofile if present
  • Changes in littler documentation
    • Added new vignette with examples
Full details for the littler release are provided as usual at the ChangeLog page. The code is available via the GitHub repo, from tarballs off my littler page and the local directory here -- and now of course all from its CRAN page and via install.packages("littler"). A fresh package has gone to the incoming queue at Debian where it will a few days as the binary packages was renamed from littler to r-cran-littler matching all other CRAN packages. Michael Rutter will probably have new Ubuntu binaries at CRAN once the source package gets into Debian proper. Comments and suggestions are welcome at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

27 October 2015

Dirk Eddelbuettel: Rcpp now used by over 500 CRAN packages

500 Rcpp packages This morning, Rcpp reached another round milestone: 501 packages on CRAN now depend on it (as measured by Depends, Imports and LinkingTo declarations, and even excluding one or two packages with an accidental declaration that do not use it). The graph is on the left depicts the growth of Rcpp usage over time. And there are a full seventy more on BioConductor in its development branch (but BioConductor is not included in the chart). Rcpp cleared 300 packages less than a year ago. It passed 400 packages in June when I only tweeted about it this June (while traveling for Rcpp training at U Zuerich, the R Summit at CBS, and the fabulous useR! 2015 at U Aalborg; so no blog post). The first and less detailed part uses manually saved entries, the second half of the data set was generated semi-automatically via a short script appending updates to a small file-based backend. A list of user package is kept on this page. Also displayed in the graph is the relative proportion of CRAN packages using Rcpp. The four per-cent hurdle was cleared just before useR! 2014 where I showed a similar graph (as two distinct graphs) in my invited talk. We passed five percent in December of last year, six percent this July and now stand at 6.77 percent, or about one in fourteen R packages. 500 user packages is very humbling, a staggering number and a big responsibility. We will out best try to keep Rcpp as performant and reliable as it has been so that the next set of packages can rely on it---just like these 500 do. So with that a very big Thank You! to all users and contributors of Rcpp for help, suggestions, bug reports, documentation or, of course, code.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Chris Lamb: ImportError: cannot import name add_to_builtins under Django 1.9

Whilst upgrading various projects to Django 1.9, I found myself repeatedly searching for the following code snippet so I offer it below as a more permanent note for myself and to aid others. If you used django.template.base.add_to_builtins to avoid tedious and unsightly % load module % blocks in your template files, under Django 1.9 you will get the following traceback: 

Traceback (most recent call last):
  File "django/core/management/__init__.py", line 324, in execute
    django.setup()
  File "django/__init__.py", line 18, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "django/apps/registry.py", line 108, in populate
    app_config.import_models(all_models)
  File "django/apps/config.py", line 202, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "myproject/myproject/utils/models.py", line 1, in <module>
    from django.template.base import add_to_builtins
ImportError: cannot import name add_to_builtins
The solution is to move to defining settings.TEMPLATES instead of calling add_to_builtins. This replaces a number of your existing settings, including TEMPLATE_CONTEXT_PROCESSORS, TEMPLATE_DIRS, TEMPLATE_LOADERS, etc. For example: 
TEMPLATES = [ 
    'BACKEND': 'django.template.backends.django.DjangoTemplates',
    'DIRS': [
        os.path.join(BASE_DIR, 'templates'),
    ],
    'APP_DIRS': True,
    'OPTIONS':  
        'context_processors': [
            'django.template.context_processors.debug',
            'django.template.context_processors.request',
            'django.contrib.auth.context_processors.auth',
            'django.contrib.messages.context_processors.messages',
            'myproject.utils.context_processors.settings_context',
        ],
        'builtins': [
            'django.contrib.staticfiles.templatetags.staticfiles',
        ],
     ,
 ]
Simply add the modules you previously loaded with add_to_builtins to the builtins key under OPTIONS. (You can read more in the release notes for Django 1.9, as well as read about settings.TEMPLATES generally.)

Marco d'Itri: Per-process netfilter rules

This article documents how the traffic of specific Linux processes can be subjected to a custom firewall or routing configuration, thanks to the magic of cgroups. We will use the Network classifier cgroup, which allows tagging the packets sent by specific processes. To create the cgroup which will be used to identify the processes I added something like this to /etc/rc.local: 
mkdir /sys/fs/cgroup/net_cls/unlocator
/bin/echo 42 > /sys/fs/cgroup/net_cls/unlocator/net_cls.classid
chown md: /sys/fs/cgroup/net_cls/unlocator/tasks
The tasks file, which controls the membership of processes in a cgroup, is made writeable by my user: this way I can add new processes without becoming root. 42 is the arbitrary class identifier that the kernel will associate with the packets generated by the member processes. A command like systemd-cgls /sys/fs/cgroup/net_cls/ can be used to explore which processes are in which cgroup. I use a simple shell wrapper to start a shell or a new program as members of this cgroup: 
#!/bin/sh -e
CGROUP_NAME=unlocator
if [ ! -d /sys/fs/cgroup/net_cls/$CGROUP_NAME/ ]; then
  echo "The $CGROUP_NAME net_cls cgroup does not exist!" >&2
  exit 1
fi
/bin/echo $$ > /sys/fs/cgroup/net_cls/$CGROUP_NAME/tasks
if [ $# = 0 ]; then
  exec $ SHELL:-/bin/sh 
fi
exec "$@"
My first goal is to use a special name server for the DNS queries of some processes, thanks to a second dnsmasq process which acts as a caching forwarder. /etc/dnsmasq2.conf: 
port=5354
listen-address=127.0.0.1
bind-interfaces
no-dhcp-interface=*
no-hosts
no-resolv
server=185.37.37.37
server=185.37.37.185
/etc/systemd/system/dnsmasq2.service: 
[Unit]
Description=dnsmasq - Second instance
Requires=network.target
[Service]
ExecStartPre=/usr/sbin/dnsmasq --test
ExecStart=/usr/sbin/dnsmasq --keep-in-foreground --conf-file=/etc/dnsmasq2.conf
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/run/dnsmasq/dnsmasq.pid
[Install]
WantedBy=multi-user.target
Do not forget to enable the new service: 
systemctl enable dnsmasq2
systemctl start dnsmasq2
Since the cgroup match extension is not yet available in a released version of iptables, you will first need to build and install it manually: 
git clone git://git.netfilter.org/iptables.git
cd iptables
./autogen.sh
./configure
make -k
sudo cp extensions/libxt_cgroup.so /lib/xtables/
sudo chmod -x /lib/xtables/libxt_cgroup.so
The netfilter configuration required is very simple: all DNS traffic from the marked processes is redirected to the port of the local dnsmasq2: 
iptables -t nat -A OUTPUT -m cgroup --cgroup 42 -p udp --dport 53 -j REDIRECT --to-ports 5354
iptables -t nat -A OUTPUT -m cgroup --cgroup 42 -p tcp --dport 53 -j REDIRECT --to-ports 5354
For related reasons, I also need to disable IPv6 for these processes: 
ip6tables -A OUTPUT -m cgroup --cgroup 42 -j REJECT
I use a different cgroup to force some programs to use my office VPN by first setting a netfilter packet mark on their traffic: 
iptables -t mangle -A OUTPUT -m cgroup --cgroup 43 -j MARK --set-mark 43
The packet mark is then used to policy-route this traffic using a dedicate VRF, i.e. routing table 43: 
ip rule add fwmark 43 table 43
This VPN VRF just contains a default route for the VPN interface: 
ip route add default dev tun0 table 43
Depending on your local configuration it may be a good idea to also add to the VPN VRF the routes of your local interfaces: 
ip route show scope link proto kernel \
    xargs -I ROUTE ip route add ROUTE table 43
Since the source address selection happens before the traffic is diverted to the VPN, we also need to source-NAT to the VPN address the marked packets: 
iptables -t nat -A POSTROUTING -m mark --mark 43 --out-interface tun0 -j MASQUERADE

26 October 2015

Steve Kemp: It begins - a new mail client, with lua scripting

Once upon a time I wrote a mail-client, which worked in the console directly via Maildir manipulation. My mail client was written in C++, and used Lua for scripting unlike clients such as mutt, alpine, and similar alternatives which don't have complete scripting support. I've pondered several times whether to restart this project, but I think it is the right thing to do. The original lumail client has a rich API, but it is very ad-hoc and random. Functions were added where they seemed like a good idea, but with no real planning, and although there are grouped functions that operate similarly there isn't a lot of consistency. The implementation is clean in places, elegant in others, and horrid in yet more parts. This time round everything is an object, accessible to Lua, with Lua, and for Lua. This time round all the drawing-magic is will be written in Lua. So to display a list of Maildirs I create a bunch of objects, one for each Maildir, and then the Lua function Maildir.to_string is called. That function looks like this: 
--
-- This method returns the text which is displayed when a maildir is
-- to be show in maildir-mode.
--
function Maildir.to_string(self)
   local total  = self:total_messages()
   local unread = self:unread_messages()
   local path   = self:path()
   local output = string.format( "[%05d / %05d] - %s", unread, total, path );
   if ( unread > 0 ) then
      output = "$[RED]" .. output
   end
   if ( string.find( output, "Automated." ) ) then
      output = strip_colour( output )
      output = "$[YELLOW]" .. output
   end
   return output
end
The end result is something that looks like this:
[00001 / 00010 ] - Amazon.de
[00000 / 00023 ] - Automated.root
The formatting can thus be adjusted clearly, easily, and without hacking the core of the client. Providing I implement the apporpriate methods to the Maildir object. It's still work in progress. You can view maildirs, view indexes, and view messages. You cannot reply, forward, or scroll properly. That said the hard part is done now, and I'm reasonably happy with it. The sample configuration file is a bit verbose, but a good demonstration regardless. See the code, if you wish, online here:

Russ Allbery: Review: Hawk

Review: Hawk, by Steven Brust
Series: Vlad Taltos #14
Publisher: Tor
Copyright: October 2014
ISBN: 0-7653-2444-X
Format: Hardcover
Pages: 320
This is the fourteenth book in the Vlad Taltos series (not counting the various associated books and other series), builds directly on the long-term plot arc of the series (finally!), and is deeply entangled with Vlad's friends and former life as a Jhereg boss. As you might imagine from that introduction, this is absolutely not the place to start with this series. For the past few books, Brust has been following a pattern of advancing the series plot in one book and then taking the next book to fill in past history or tell some side story. That means, following Tiassa, we were due some series advancement, and that's exactly what we get. We also, finally, get some more details about Lady Teldra. Nothing all that revelatory, but certainly intriguing, and more than just additional questions (at last). When Brust finally takes this gun off the wall and fires it, the resulting bits of world-building might be even better than Issola. At its heart, though, Hawk is a caper novel. If you're like me, you're thinking "it's about time." I think this is the sort of story Brust excels at, particularly with Vlad as his protagonist. Even better, unlike some of the other multi-part novels, this is a book-length caper focused on a very important goal, and with the potential to get rid of some annoyances in Vlad's life that have lingered for rather too long. We see many of Vlad's Dragaeran friends, but (apart from Daymar) mostly in glimpses. This is Vlad's book, with heavy helpings of Loiosh. The caper is also a nicely twisty one, involving everything from different types of magic to the inner workings of the Jhereg organization. As is typical for Vlad's schemes, there are several false fronts and fake goals, numerous unexpected twists, and a rather fun guest appearance. Oh, and lots and lots of snark, of course. I think my favorite part of the book was the interaction between Vlad and Kragar, which added a lot of emotional depth both to this story and to some of the previous stories of Vlad's life as a Jhereg. And I'm hoping that where Brust leaves things at the end of this book implies a Vlad who is more free to act, to see his friends, and to get entangled in Imperial politics, since I think that leads to the best stories. Of course, if Brust holds to pattern, the next book will be backfill or side stories and we'll have to wait longer for a continuation of the main story. As much as I like those side stories, I'm hoping Brust will break pattern. I'm increasingly eager to see where this story will go. The all-too-brief interaction with Sethra in this book promises so much for the future. If you like the Vlad Taltos books overall, you'll probably like this one. It's a return to the old scheming Vlad, but tempered by more experience and different stakes. There's a bit of lore, a bit of world-building, and a lot of Vlad being tricky. This series is still going strong fourteen books in. Rating: 8 out of 10

25 October 2015

Johannes Schauer: unshare without superuser privileges

TLDR: With the help of Helmut Grohne I finally figured out most of the bits necessary to unshare everything without becoming root (though one might say that this is still cheated because the suid root tools newuidmap and newgidmap are used). I wrote a Perl script which documents how this is done in practice. This script is nearly equivalent to using the existing commands lxc-usernsexec [opts] -- unshare [opts] -- COMMAND except that these two together cannot be used to mount a new proc. Apart from this problem, this Perl script might also be useful by itself because it is architecture independent and easily inspectable for the curious mind without resorting to sources.debian.net (it is heavily documented at nearly 2 lines of comments per line of code on average). It can be retrieved here at https://gitlab.mister-muffin.de/josch/user-unshare/blob/master/user-unshare Long story: Nearly two years after my last last rant about everything needing superuser privileges in Linux, I'm still interested in techniques that let me do more things without becoming root. Helmut Grohne had told me for a while about unshare(), or user namespaces as the right way to have things like chroot without root. There are also reports of LXC containers working without root privileges but they are hard to come by. A couple of days ago I had some time again, so Helmut helped me to get through the major blockers that were so far stopping me from using unshare in a meaningful way without executing everything with sudo. My main motivation at that point was to let dpkg-buildpackage when executed by sbuild be run with an unshared network namespace and thus without network access (except for the loopback interface) because like pbuilder I wanted sbuild to enforce the rule not to access any remote resources during the build. After several evenings of investigating and doctoring at the Perl script I mentioned initially, I came to the conclusion that the only place that can unshare the network namespace without disrupting anything is schroot itself. This is because unsharing inside the chroot will fail because dpkg-buildpackage is run with non-root privileges and thus the user namespace has to be unshared. But this then will destroy all ownership information. But even if that wasn't the case, the chroot itself is unlikely to have (and also should not) tools like ip or newuidmap and newgidmap installed. Unsharing the schroot call itself also will not work. Again we first need to unshare the user namespace and then schroot will complain about wrong ownership of its configuration file /etc/schroot/schroot.conf. Luckily, when contacting Roger Leigh about this wishlist feature in bug#802849 I was told that this was already implemented in its git master \o/. So this particular problem seems to be taken care of and once the next schroot release happens, sbuild will make use of it and have unshare --net capabilities just like pbuilder already had since last year. With the sbuild case taken care of, the rest of this post will introduce the Perl script I wrote. The name user-unshare is really arbitrary. I just needed some identifier for the git repository and a filename. The most important discovery I made was, that Debian disables unprivileged user namespaces by default with the patch add-sysctl-to-disallow-unprivileged-CLONE_NEWUSER-by-default.patch to the Linux kernel. To enable it, one has to first either do 
echo 1   sudo tee /proc/sys/kernel/unprivileged_userns_clone > /dev/null
or 
sudo sysctl -w kernel.unprivileged_userns_clone=1
The tool tries to be like unshare(1) but with the power of lxc-usernsexec(1) to map more than one id into the new user namespace by using the programs newgidmap and newuidmap. Or in other words: This tool tries to be like lxc-usernsexec(1) but with the power of unshare(1) to unshare more than just the user and mount namespaces. It is nearly equal to calling: 
lxc-usernsexec [opts] -- unshare [opts] -- COMMAND
Its main reason of existence are: I hoped that systemd-nspawn could do what I wanted but it seems that its requirement for being run as root will not change any time soon Another tool in Debian that offers to do chroot without superuser privileges is linux-user-chroot but that one cheats by being suid root. Had I found lxc-usernsexec earlier I would've probably not written this. But after I found it I happily used it to get an even better understanding of the matter and further improve the comments in my code. I started writing my own tool in Perl because that's the language sbuild was written in and as mentioned initially, I intended to use this script with sbuild. Now that the sbuild problem is taken care of, this is not so important anymore but I like if I can read the code of simple programs I run directly from /usr/bin without having to retrieve the source code first or use sources.debian.net. The only thing I wasn't able to figure out is how to properly mount proc into my new mount namespace. I found a workaround that works by first mounting a new proc to /proc and then bind-mounting /proc to whatever new location for proc is requested. I didn't figure out how to do this without mounting to /proc first partly also because this doesn't work at all when using lxc-usernsexec and unshare together. In this respect, this perl script is a bit more powerful than those two tools together. I suppose that the reason is that unshare wasn't written with having being called without superuser privileges in mind. If you have an idea what could be wrong, the code has a big FIXME about this issue. Finally, here a demonstration of what my script can do. Because of the /proc bug, lxc-usernsexec and unshare together are not able to do this but it might also be that I'm just not using these tools in the right way. The following will give you an interactive shell in an environment created from one of my sbuild chroot tarballs: 
$ mkdir -p /tmp/buildroot/proc
$ ./user-unshare --mount-proc=/tmp/buildroot/proc --ipc --pid --net \
    --uts --mount --fork -- sh -c 'ip link set lo up && ip addr && \
    hostname hoothoot-chroot && \
    tar -C /tmp/buildroot -xf /srv/chroot/unstable-amd64.tar.gz; \
    /usr/sbin/chroot /tmp/buildroot /sbin/runuser -s /bin/bash - josch && \
    umount /tmp/buildroot/proc && rm -rf /tmp/buildroot'
(unstable-amd64-sbuild)josch@hoothoot-chroot:/$ whoami
josch
(unstable-amd64-sbuild)josch@hoothoot-chroot:/$ hostname
hoothoot-chroot
(unstable-amd64-sbuild)josch@hoothoot-chroot:/$ ls -lha /proc   head
total 0
dr-xr-xr-x 218 nobody nogroup    0 Oct 25 19:06 .
drwxr-xr-x  22 root   root     440 Oct  1 08:42 ..
dr-xr-xr-x   9 root   root       0 Oct 25 19:06 1
dr-xr-xr-x   9 josch  josch      0 Oct 25 19:06 15
dr-xr-xr-x   9 josch  josch      0 Oct 25 19:06 16
dr-xr-xr-x   9 root   root       0 Oct 25 19:06 7
dr-xr-xr-x   9 josch  josch      0 Oct 25 19:06 8
dr-xr-xr-x   4 nobody nogroup    0 Oct 25 19:06 acpi
dr-xr-xr-x   6 nobody nogroup    0 Oct 25 19:06 asound
Of course instead of running this long command we can also instead write a small shell script and execute that instead. The following does the same things as the long command above but adds some comments for further explanation: 
#!/bin/sh

set -exu

# I'm using /tmp because I have it mounted as a tmpfs
rootdir="/tmp/buildroot"

# bring the loopback interface up
ip link set lo up

# show that the loopback interface is really up
ip addr

# make use of the UTS namespace being unshared
hostname hoothoot-chroot

# extract the chroot tarball. This must be done inside the user namespace for
# the file permissions to be correct.
#
# tar will fail to call mknod and to change the permissions of /proc but we are
# ignoring that
tar -C "$rootdir" -xf /srv/chroot/unstable-amd64.tar.gz   true

# run chroot and inside, immediately drop permissions to the user "josch" and
# start an interactive shell
/usr/sbin/chroot "$rootdir" /sbin/runuser -s /bin/bash - josch

# unmount /proc and remove the temporary directory
umount "$rootdir/proc"
rm -rf "$rootdir"
and then: 
$ mkdir -p /tmp/buildroot/proc
$ ./user-unshare --mount-proc=/tmp/buildroot/proc --ipc --pid --net --uts --mount --fork -- ./chroot.sh
As mentioned in the beginning, the tool is nearly equivalent to calling lxc-usernsexec [opts] -- unshare [opts] -- COMMAND but because of the problem with mounting proc (mentioned earlier), lxc-usernsexec and unshare cannot be used with above example. If one tries anyways one will only get: 
$ lxc-usernsexec -m b:0:1000:1 -m b:1:558752:1 -- unshare --mount-proc=/tmp/buildroot/proc --ipc --pid --net --uts --mount --fork -- ./chroot.sh
unshare: mount /tmp/buildroot/proc failed: Invalid argument
I'd be interested in finding out why that is and how to fix it.

Russ Allbery: Review: The Thrilling Adventures of Lovelace and Babbage

Review: The Thrilling Adventures of Lovelace and Babbage, by Sydney Padua
Publisher: Pantheon
Copyright: 2015
ISBN: 0-307-90827-5
Format: Hardcover
Pages: 317
I seem to be writing a lot of reviews lately where any reader can easily do their own research and see if this is something they'd like. This is another of those, but perhaps a bit more obscure. As Padua explains in the preface, the first Lovelace and Babbage strip was intended to be a one-off joke. A friend suggested that she draw a comic telling the (short and rather tragic) life of Ada Lovelace, widely considered the first computer programmer even though the computer for which she wrote programs was never built: Charles Babbage's Analytic Engine. Padua found the story of Babbage and Lovelace's collaboration fascinating but too grim, and so wrote an alternate ending involving a pocket universe in which Babbage and Lovelace lived on to fight crime! Or at least fought against street organs and poetry. Padua meant this as a joke. The Internet took it as a teaser. Quite to her surprise, she found herself writing occasional additional episodes and getting lost in fascinating research about Lovelace and Babbage. The result is all at 2dgoggles.com. This book is a curated collection of those comics, including much of their (extensive) footnotes and research notes, in a very attractive and well-constructed book form. The best review, therefore, would be to go read some of the comics yourself, easily accessible via the Comics tab on the web site, and see if this is the sort of thing you'd enjoy. There is some material in the book that isn't on the web site, but most of it is there (including one full story that didn't make it into the book). There are a few reasons to buy a collection like this of material that previously appeared on the Internet. One, of course, is out of gratitude to support the author, which is the main reason I bought it. If you want to do that, though, you probably already know you do. Another reason is for additional unpublished material, but that's not really the case here. But a third is that, despite all the technology of the web, books can sometimes provide a more elegant and beautiful presentation. And that's very much the case for The Thrilling Adventures of Lovelace and Babbage. The hardcover is a beautiful work of art. It has what you would expect from a hardcover graphic novel, of course: sturdy paper, lovely story headings in the form of Victorian-style posters, and very nice artwork beneath the dust jacket. (I've been so happy to see that in the last few hardcovers I've reviewed.) But, best of all, it presents the primary research notes on each page underneath comic panels. Those who have been reading Lovelace and Babbage as it appears on the web know Padua's copious research notes. But they're normally presented at the end of each complete part, and I found myself often not reading them in detail. This might not be the case for everyone, but at least for me this presentation works so much better. Historical notes are at the bottom of (nearly) every page, in context. This might sound like it would distract from and break up the flow of the story, but at least for me it does the opposite. The comics feel so much richer when intermixed with the actual history (which in some cases is surprisingly similar to what seemed to be wild, invented scenarios). The historical notes also have end notes, and quite substantial ones. I found that organization less successful (readers of my reviews will know I have a long-standing antipathy towards end notes for anything other than cross-references), but I have to admit I have no idea how Padua would have fit those on the same page as footnotes. They're also written well enough, and with sufficient detail, that I could usually read a chapter and then read all the end notes and mostly remember what they referred to. Padua also takes advantage of the format to play a few neat games with frame breaks: characters commenting on the notes, stabbing things into them, or otherwise affecting them. I have to admit to some mild frustration where this makes the notes unreadable, since they're nearly as fascinating as the comics, but the overall effect is still worth it. (And, of course, the full notes are available on the web if one wishes to look them up.) Rounding out the book version are a few of the best original source documents that Padua found, thankfully excerpted and well-edited to not outlive their welcome for those of us who don't like poring over Victorian letters. And, finally, some beautiful illustrations of the mechanisms of the Analytic Engine with supporting descriptions of how they were supposed to work. I'm not particularly mechanically inclined, nor that fascinated by steampunk, and I still found these diagrams impressive and fun to look at. Someone more interested in such things will be in for a treat. I've been thoroughly enjoying the (sadly infrequent) installments of Lovelace and Babbage for years now, and am utterly delighted by their hardcover appearance. If you already knew about Padua's work and have any interest in nice hardcovers of such things, I don't think you'll regret the purchase. If you haven't heard of her, or this series, before, some quick reading at the web site should quickly reveal whether this is something you'd enjoy. Rating: 9 out of 10

24 October 2015

Mart n Ferrari: Tales from the SRE trenches - Part 1

A few weeks ago, I was offered the opportunity to give a guest talk in the Romanian Association for Better Software. RABS is a group of people interested in improving the trade, and regularly hold events where invited speakers give presentations on a wide array of topics. The speakers are usually pretty high-profile, so this is quite a responsibility! To make things more interesting, much of the target audience works on enterprise software, under Windows platforms. Definitely outside my comfort zone! Considering all this, we decided the topic was going to be about Site Reliability Engineering (SRE), concentrating on some aspects of it which I believe could be useful independently of the kind of company you are working for. I finally gave the talk last Monday, and the audience seemed to enjoy it, so I am going to post here my notes, hopefully some other people will like it too.

Why should I care? I prepared this thinking of an audience of software engineers, so why would anyone want to hear about this idea that only seems to be about making the life of the operations people better? The thing is, having your work as a development team supported by an SRE team will also benefit you. This is not about empowering Ops to hit you harder when things blow apart, but to have a team that is your partner. A partner that will help you grow, handle the complexities of a production environment so you can concentrate on cool features, and that will get out of the way when things are running fine. A development team may seem to only care about adding features that will drive more and more users to your service. But an unreliable service is a service that loses users, so you should care about reliability. And what better to have a team has Reliability on their name?

What is SRE? SRE means Site Reliability Engineering, Reliability Engineering applied to "sites". Wikipedia defines Reliability Engineering as:
[..] engineering that emphasizes dependability in the life-cycle management of a product.
This is, historically, a branch of engineering that made possible to build devices that will work as expected even when their components were inherently unreliable. It focused on improving component reliability, establishing minimum requirements and expectations, and a heavy usage of statistics to predict failures and understand underlying problems. SRE started as a concept at Google about 12 years ago, when Ben Treynor joined the company and created the SRE team from a group of 7 production engineers. There is no good definition of what Site Reliability Engineering means; while the term and some of its ideas are clearly inspired in the more traditional RE, he defines SRE with these words1:
Fundamentally, it's what happens when you ask a software engineer to design an operations function.

Only hire coders After reading that quote it is not surprising that the first item in the SRE checklist2, is to only hire people who can code properly for SRE roles. Writing software is a key part of being SRE. But this does not mean that there is no separation between development and operations, nor that SRE is a fancy(er) name for DevOps3. It means treating operations as a software engineering problem, using software to solve problems that used to be solved by hand, implementing rigorous testing and code reviewing, and taking decisions based on data, not just hunches. It also implies that SREs can understand the product they are supporting, and that there is a common ground and respect between SREs and software engineers (SWEs). There are many things that make SRE what it is, some of these only make sense within a special kind of company like Google: many different development and operations teams, service growth that can't be matched by hiring, and more importantly, firm commitment from top management to implement these drastic rules. Therefore, my focus here is not to preach on how everybody should adopt SRE, but to extract some of the most useful ideas that can be applied in a wider array of situations. Nevertheless, I will first try to give an overview of how SRE works at Google.
That's it for today. In the next post I will talk about how to end the war between developers and SysAdmins. Stay tuned!
  1. http://www.site-reliability-engineering.info/2014/04/what-is-site-reliability-engineering.html
  2. SRE checklist extracted from Treynor's talk at SREcon14: https://www.usenix.org/conference/srecon14/technical-sessions/presentation/keys-sre
  3. By the way, I am still not sure what DevOps mean, it seems that everyone has a different definition for it.
Comment

Dirk Eddelbuettel: rfoaas 0.1.8

A new release rfoaas is now on CRAN following up on the recent upstream release 0.1.8 of FOAAS. The rfoaas package provides an interface for R to the most excellent FOAAS service--which provides a modern, scalable and RESTful web service for the frequent need to tell someone to f$#@ off. Release 0.1.8 brings over a handfull new access points relative to the previous release 0.1.6: bday(), family_(), shutup(), zayn(), dalton(), dosomething(), retard(), and thumbs(). One of these may make a fine addition for beloved post-test comments in testthat: 
R> library(rfoaas)
R> dosomething("Fix", "function", "testthat")
[1] "Fix the fucking function! - testthat"
R>
We did not make a release following version 0.1.7 of FOAAS: my testing revealed somewhat less consistent results for the language / i18n filter added in 0.1.6. So I filed issue ticket #95, but as it still lingers unanswered, I consider this feature to be unsupported for now; the tests for the language filter were disabled for now as well. As usual, CRANberries provides a diff to the previous CRAN release. Questions, comments etc should go to the GitHub issue tracker.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Vincent Sanders: It takes courage to sit on a jury. How many of us want to decide the fate of another person's life or freedom?

I think Regina Brett has a point although having now experienced being a juror in a British crown court I have a much better understanding of both the process and effectiveness of the jury system.

The actual process of becoming a juror on a case is something I had not been aware of previously. You simply receive a letter telling you to be at the court on a specific date and that you are required to be available for at least ten days, possibly more. The only qualification to receive the letter is to be on the electoral roll and it is an invitation with few options to refuse without serious repercussions.

When you arrive at the court you are directed to the Jury lounge (practical hint: take a book) where you notice there are over forty people, which would seem odd until you realise there are three courtrooms and they each need a jury, even then there are an excess of people which is because of the selection process.

The process of jury selection is fairly simple, an usher for a court comes and calls fourteen names which forms the jury in waiting. The group is taken up to the court waiting room (this room gets terribly familiar over the forthcoming weeks) and then twelve names are called.

As each person is called they enter the jury box in order which persists for the entire trial (practical hint:remember your juror number). Before each person is sworn or affirmed there is the possibility they will be found unsuitable and will be replaced by one of the previously unselected jurors. Any unselected jurors are then sent back to the jury lounge and become available for forming another jury in waiting.

Anyone unselected at the end of the process has to remain available to return to the court to form a jury in waiting when a previous trial ends until they have exhausted their duty. There were a few of these unfortunate people who were kept in a state of limbo for several days and I am relieved this did not happen to me.

Being a juror, from a purely practical perspective, felt like working an office job for ten days. Duties consisting of attending a series of meetings with strange rules and the typically understated British approach to mentioning the result of breaking them.

I participated in two cases, both of which were (almost by definition) unpleasant happenings, these were a case of Grievous bodily harm and an offence under section 7 of the Sexual Offences Act 2003.

Both cases were challenging in their own ways the first because of the way the case was presented and the second because of its subject matter. One of the most important rules is "Do not discuss anything with anyone as it might be perjury" so going along with that I will not be discussing any details. Because I cannot be specific this post has become a little impersonal, you will have to forgive me as I found I had to remove a great deal of more content which was not appropriate.

An important thing to note is that the trials bore no resemblance to TV courtroom drama. The trials proceed in a professional manner with very little theatrics. The prosecution barrister commences outlining the case against the accused, calling witnesses and reading into evidence uncontested material. The defence then gets to present their case, again calling witnesses and placing documents into evidence.

One of the striking things about this process is that if the barristers do not call a witness or present evidence that would seem to be pertinent, the jury must not draw inference from that omission, which is especially bizarre when a central witness referred to by almost everyone involved with the case is not called.

Once the case is presented the jury is sequestered in a room and must come to a unanimous decision on each of the charges. This was, for me, the most challenging part of the whole process. Twelve people with unique views on the information presented have to attempt to discuss the evidence and not simply go with their first impressions based on their preconceptions.

The jury is allowed to ask for some evidence to be repeated and if deliberations take some time the jury may be instructed that a majority of 10 to 2 may be accepted. I imagine at some point the jury will run out of time to make a decision and something else will happen but I did not experience this.

Overall the experience was enlightening if not enjoyable, I understand the process a lot more and am happy to have discharged my duty and am equally glad the responsibility will not come around again for at least a few years.

Russ Allbery: Review: Oathbreakers

Review: Oathbreakers, by Mercedes Lackey
Series: Vows and Honor #2
Publisher: DAW
Copyright: January 1989
ISBN: 0-88677-454-3
Format: Mass market
Pages: 318
The Tarma and Kethry stories tend to be stand-alone and are readable out of order, and this isn't an exception. But if you want their background, consider reading Oathblood or (less recommended) The Oathbound before reading this book. (Reading Oathblood first may require a bit of finesse, since some of the stories in that book come after this novel. Unfortunately, there is no good ordering or collection of these stories that maintains internal chronological order.) This is more like it. This is the Tarma and Kethry story that I remembered when calling them my favorite characters in the Valdemar universe. Following the short stories merged into The Oathbound fixup novel, Tarma and Kethry are still trying to gather the resources required to start a school and to rebuild Tarma's clan. That's led to them signing with a highly-respected mercenary company: Idra's Sunhawks. Idra renounced her claims to the Rethwellen line of royal succession to lead the Sunhawks, creating a mercenary band that's legendary for their quality and battlefield capabilities. The story opens with a campaign in Jkatha, on one side of a civil war, which is mostly an opportunity to get to know the Sunhawks and to see Tarma and Kethry show their competence. The real story starts later, when Idra is called back to Rethwellen for family business and something goes very wrong. I think Lackey is best at two types of stories: misunderstood young people who grow into themselves and their place in the world, and competent people displaying their competence. The Tarma and Kethry stories, and particularly Oathbreakers, are of the latter type. This is clearly wish fulfillment: Lackey's stories often lack nuance, there's rarely any doubt as to who the good and bad guys are, and, although very bad things can happen, you're probably going to get some sort of happy ending. But if you're in the mood for that sort of story, it's so satisfying. The Tarma and Kethry we see here are a mature, experienced fighter and mage team (plus Warrl, who provides vitally important magical and combat assistance, as well as some pointed advice). They know what they're doing, they care deeply for each other, and both their relationship patterns and their capabilities are well-understood. Both do a bit of growing over the course of this novel, but that's not really the point. The point is seeing them take on unfamiliar challenges and tricky investigations while being very good at what they do. In other words, this isn't bildungsroman or high fantasy; it's sword and sorcery, and an excellent example of the genre. Reading these books as part of the overall Valdemar series provides some enjoyable moments with the first explicit contact between Valdemar and its Heralds and Tarma and Kethry's world. The maps here firmly establish their home regions as well to the south of Valdemar and multiple kingdoms away, but Rethwellen (as previously established in earlier Herald-focused trilogies) is on Valdemar's southern border. Seeing Lackey's very separate magic and divinity systems cross and meet, with a bit of initial mutual suspicion, is a rather fun moment (if, at least, you're in the mood for a story in which the world has a vested interest in making sure all the good people like each other). Although I'm wondering why Kethry didn't get extremely uncomfortable when she crossed the border into Valdemar due to the trick that Vanyel pulled in his trilogy. (I seem to recall this is explained away at some point.) Be warned that this novel does contain other elements typical of early Lackey. There is, for example, the inevitable rape, although thankfully off-camera and not quite as central to the plot. (Although in a way that makes it worse since it felt gratuitous. I'm unconvinced that the rape was at all necessary to the story that Lackey was telling.) Revenge and eye-for-an-eye justice are hotly defended by the protagonists. This isn't a series to look to for subtle and complex solutions to political problems; instead, everything gets better if you just kill all the evil people. There isn't anything quite as egregious as the actions of the supposed good guys in The Oathbound, but you still have to read past a certain bloodthirstiness in the stated good side of a very black-and-white morality. That means this isn't a novel for all people or all moods. But within those genre conventions, which aren't that unusual for sword and sorcery, Oathbreakers is a lot of fun. It's one of the few Valdemar novels I've read during this re-read that lived up to my memory of it. Recommended if you like this sort of thing. Rating: 8 out of 10

23 October 2015

Petter Reinholdtsen: "Free Culture" by @lessig - The background story for Creative Commons - new edition available

Click here to buy the book. In 2004, as the Creative Commons movement gained momentum, its creator Lawrence Lessig wrote the book Free Culture to explain the problems with increasing copyright regulation and suggest some solutions. I read the book back then and was very moved by it. Reading the book inspired me and changed the way I looked on copyright law, and I would love it if more people would read it too. Because of this, I decided in the summer of 2012 to translate it to Norwegian Bokm l and publish it for those of my friends and family that prefer to read books in Norwegian. I translated the book using docbook and a gettext PO file, and a byproduct of this process is a new edition of the English original. I've been in touch with the author during by work, and he said it was fine with him if I also published an English version. So I decided to do so. Today, I made this edition available for sale on Lulu.com, for those interested in a paper book. This is the cover:

The Norwegian Bokm l version will be available for purchase in a few days. I also plan to publish a French version in a few weeks or months, depending on the amount of people with knowledge of French to join the translation project. So far there is only one active person, but the French book is almost completely translated but need some proof reading. The book is also available in PDF, ePub and MOBI formats from my github project page. Note the ePub and MOBI versions have some formatting problems I believe is due to bugs in the docbook tool dbtoepub (Debian BTS issues #795842 and #796871), but I have not taken the time to investigate. I recommend the PDF and ePub version for now, as they seem to show up fine in the viewers I have available. After the translation to Norwegian Bokm l was complete, I was able to secure some sponsoring from the NUUG Foundation to print the book. This is the reason their logo is located on the back cover. I am very grateful for their contribution, and will use it to give a copy of the Norwegian edition to members of the Norwegian Parliament and other decision makers here in Norway.

21 October 2015

DebConf team: DebConf15 dates are set, come and join us! (Posted by DebConf15 team)

At DebConf14 in Portland, Oregon, USA, next year s DebConf team presented their conference plans and announced the conference dates: DebConf15 will take place from 15 to 22 August 2015 in Heidelberg, Germany. On the Open Weekend on 15/16 August, we invite members of the public to participate in our wide offering of content and events, before we dive into the more technical part of the conference during following week. DebConf15 will also be preceeded by DebCamp, a time and place for teams to gather for intensive collaboration. A set of slides from a quick show-case during the DebConf14 closing ceremony provide a quick overview of what you can expect next year. For more in-depth information, we invite you to watch the video recording of the full session, in which the team provides detailed information on the preparations so far, location and transportation to the venue at Heidelberg, the different rooms and areas at the Youth Hostel (for accommodation, hacking, talks, and social activities), details about the infrastructure that are being worked on, and the plans around the conference schedule. We invite everyone to join us in organising this conference. There are different areas where your help could be very valuable, and we are always looking forward to your ideas. Have a look at our wiki page, join our IRC channels and subscribe to our mailing lists. We are also contacting potential sponsors from all around the globe. If you know any organisation that could be interested, please consider handing them our sponsorship brochure or contact the fundraising team with any leads. Let s work together, as every year, on making the best DebConf ever!

20 October 2015

Dirk Eddelbuettel: Rblpapi 0.3.1

The first update to the Rblpapi package since the initial CRAN upload in August is now available. Rblpapi connects R to the Bloomberg system, giving access to a truly vast array of time series data and custom calculations. This release brings a new beqs() functions to access the results from Bloomberg EQS queries, improves the build system a correct a bug in the retrieval of multiple tick series. The changes are detailed below.
Changes in Rblpapi version 0.3.1 (2015-10-19)
  • Added beqs() to run Bloomberg Equity Screen Search, based on initial PR #79 by Rademeyer Vermaak, reworked in PRs #83 and #84 by Dirk; closes tickets #63 and #80.
  • Bloomberg header and library files are now cached locally instead of being re-downloaded for every build (PR #78 by Dirk addressing issue #65).
  • For multiple ticks, time-stamps are unique-yfied before merging (PR #77 by Dirk addressing issue #76).
  • A compiler warning under new g++ versions is now suppressed (PR #69 by John, fixing #68).
Courtesy of CRANberries, there is also a diffstat report for the this release. As always, more detailed information is on the Rblpapi. Questions, comments etc should go to the issue tickets system at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

17 October 2015

Iustin Pop: Server upgrades and monitoring

Undecided whether the title should be "exercises in Yak shaving" or "paying back technical debt" or "too much complexity for personal systems". Anyway I started hosting my personal website and some other small stuff on a dedicated box (rented from a provider) in early 2008. Even for a relatively cheap box, it worked without issues for a good number of years. A surprising number of years, actually; the only issue was a power supply failure that was solved by the provider automatically and then nothing for many years. Even the harddrive (mechanical) had no issues at all for 7 years (Power_On_Hours: 64380; I probably got it after it had a few months of uptime). I believe it was the longest running harddrive I've ever used (for the record: Seagate Barracuda 7200.10, ST3250310AS). The reason I delayed upgrade for a long time was twofold: first, at the same provider I couldn't get a similar SLA for the same amount of money. I could get better hardware, but with worse SLA and options. This is easily solvable, of course, by just finding a different provider. The other issue was that I never bothered to setup proper configuration management for the host; after all, it was only supposed to run Apache with ikiwiki and some other trivial small other things. The truth was that over time it started pilling up more and more "small things" so actually changing the host is expensive. As the age of the server neared 7 years, I thought to combine upgrade from Wheezy to Jessie with a HW upgrade. Managed to find a different provider that had my desired SLA and HW configuration, got the server and the only thing left was to do the migration. Previous OS upgrades were simple as they were on the same host; i.e. rely on Debian's reliable upgrade and nothing else to, eventually adjust slightly some configs. With a cross-host upgrade (I couldn't just copy the old OS since it was also a 32-to-64 bit change) it's much worse: since there's no previous installation, I had to manually check and port the old configuration for each individual service. This got very tedious, and I realised I have to make it somehow better. "Proper" configuration management aside, I thought that I need proper monitoring first. I already had (for a long while actually) graphing via Munin, but no actual monitoring. Since the host only had few services, this was again supposed to be easy - same mistake again. The problem is that once you have any monitoring system setup, it's very easy to actually add "just one more" host or service to it. First it was only the external box, then it was my firewall, then it was the rest of my home network. Then it was the cloud services that I use for example, checking whether my domain registrar's nameservers still are authoritative for my domain or whether the expiration date it still far in the future. And so on In the end, what was in previous iterations (e.g. Squeeze to Wheezy upgrade) a half-weekend job only, spread out over many weekends (interleaved with other activities, not fully working on it). I had to keep the old machine running for a month more in order to make sure everything was up and running, and I ended up with 80 services monitored across multiple systems; the migrated machine itself has almost half of these. Some of these are light items (e.g. a checking that a single vhost responds) other are aggregates. I still need to add some more checks though, especially more complex (end-to-end) ones. The lesson I learned in all this is that, with or without configuration management in place, having monitoring makes it much easier do to host or service moves, as you know much better when everything is done whether it's "done-done" or just "almost done". The question that remains though: with 80 services for a home network plus external systems (personal use); I'm not sure if I'm doing things right (monitor the stuff I need) or wrong (do I really need these many things)?

Russell Coker: Mail Server Training

Today I ran a hands-on training session on configuring a MTA with Postfix and Dovecot for LUV. I gave each student a virtual machine running Debian/Jessie with full Internet access and instructions on how to configure it as a basic mail server. Here is a slightly modified set of instructions that anyone can do on their own system. Today I learned that documentation that includes passwords on a command-line should have quotes around the password, one student used a semi-colon character in his password which caused some confusion (it s the command separator character in BASH). I also discovered that trying to just tell users which virtual server to login to is prone to errors, in future I ll print out a list of user-names and passwords for virtual servers and tear off one for each student so there s no possibility of 2 users logging in to the same system. I gave each student a sub-domain of unixapropos.com (a zone that I use for various random sysadmin type things). I have changed the instructions to use example.com which is the official address for testing things (or you could use any zone that you use). The test VMs that I setup had a user named auser , the documentation assumes this account name. You could change auser to something else if you wish. Below are all the instructions for anyone who wants to try it at home or setup virtual machines and run their own training session. Basic MTA Configuration
  1. Run apt-get install postfix to install Postfix, select Internet Site for the type of mail configuration and enter the domain name you selected for the mail name.
  2. The main Postfix configuration file is /etc/postfix/main.cf. Change the myhostname setting to the fully qualified name of the system, something like mta.example.com.
    You can edit /etc/postfix/main.cf with vi (or any other editor) or use the postconf command to change it, eg postconf -e myhostname=mta.example.com .
  3. Add home_mailbox=Maildir/ to the Postfix configuration to make it deliver to a Maildir spool in the user s home directory.
  4. Restart Postfix to apply the changes.
  5. Run apt-get install swaks libnet-ssleay-perl to install swaks (a SMTP test tool).
  6. Test delivery by running the command swaks -f auser@example.com -t auser@example.com -s localhost . Note that swaks displays the SMTP data so you can see exactly what happens and if something goes wrong you will see everything about the error.
  7. Inspect /var/log/mail.log to see the messages about the delivery. View the message which is in ~auser/Maildir/new.
  8. When other students get to this stage run the same swaks command but with the -t changed to the address in their domain, check the mail.log to see that the messages were transferred and view the mail with less to see the received lines. If you do this on your own specify a recipient address that s a regular email address of yours (EG a Gmail account).
Basic Pop/IMAP Configuration
  1. Run apt-get install dovecot-pop3d dovecot-imapd to install Dovecot POP and IMAP servers.
    Run netstat -tln to see the ports that have daemons listening on them, observe that ports 110 and 143 are in use.
  2. Edit /etc/dovecot/conf.d/10-mail.conf and change mail_location to maildir:~/Maildir . Then restart Dovecot.
  3. Run the command nc localhost 110 to connect to POP, then run the following commands to get capabilities, login, and retrieve mail:
    user auser
    pass WHATEVERYOUMADEIT
    capa
    list
    retr 1
    quit
  4. Run the command nc localhost 143 to connect to IMAP, then run the following commands to list capabilities, login, and logout:
    a capability
    b login auser WHATEVERYOUMADEIT
    c logout
  5. For the above commands make note of the capabilities, we will refer to that later.
Now you have a basically functional mail server on the Internet! POP/IMAP Over SSL To avoid password sniffing we need to use SSL. To do it properly requires obtaining a signed key for a DNS address but we can do the technical work with the snakeoil certificate that is generated by Debian.
  1. Edit /etc/dovecot/conf.d/10-ssl.conf and change ssl = no to ssl = required . Then add the following 2 lines:
    ssl_cert = </etc/ssl/certs/ssl-cert-snakeoil.pem
    ssl_key = </etc/ssl/private/ssl-cert-snakeoil.key
    1. Run netstat -tln and note that ports 993 and 995 are not in use.
    2. Edit /etc/dovecot/conf.d/10-master.conf and uncomment the following lines:
      port = 993
      ssl = yes
      port = 995
      ssl = yes
    3. Restart Dovecot, run netstat -tln and note that ports 993 and 995 are in use.
  3. Run nc localhost 110 and nc localhost 143 as before, note that the capabilities have changed to include STLS/STARTTLS respectively.
  4. Run gnutls-cli --tofu 127.0.0.1 -p 993 to connect to the server via IMAPS and gnutls-cli --tofu 127.0.0.1 -p 995 to connect via POP3S. The --tofu option means to Trust On First Use , it stores the public key in ~/.gnutls and checks it the next time you connect. This allows you to safely use a snakeoil certificate if all apps can securely get a copy of the key.
Postfix SSL
  1. Edit /etc/postfix/main.cf and add the following 4 lines:
    smtpd_tls_received_header = yes
    smtpd_tls_loglevel = 1
    smtp_tls_loglevel = 1
    smtp_tls_security_level = may
    Then restart Postfix. This makes Postfix log TLS summary messages to syslog and in the Received header. It also permits Postfix to send with TLS.
  2. Run nc localhost 25 to connect to your SMTP port and then enter the following commands:
    ehlo test
    quit
    Note that the response to the EHLO command includes 250-STARTTLS, this is because Postfix was configured with the Snakeoil certificate by default.
  3. Run gnutls-cli --tofu 127.0.0.1 -p 25 -s and enter the following commands:
    ehlo test
    starttls
    ^D
    After the CTRL-D gnutls-cli will establish a SSL connection.
  4. Run swaks -tls -f auser@example.com -t auser@example.com -s localhost to send a message with SSL encryption. Note that swaks doesn t verify the key.
  5. Try using swaks to send messages to other servers with SSL encryption. Gmail is one example of a mail server that supports SSL which can be used, run swaks -tls -f auser@example.com -t YOURADDRESS@gmail.com to send TLS (encapsulated SSL) mail to Gmail via swaks. Also run swaks -tls -f auser@example.com -t YOURADDRESS@gmail.com -s localhost to send via your new mail server (which should log that it was a TLS connection from swaks and a TLS connection to Gmail).
SASL SASL is the system of SMTP authentication for mail relaying. It is needed to permit devices without fixed IP addresses to send mail through a server. The easiest way of configuring Postfix SASL is to have Dovecot provide it s authentication data to Postfix. Among other things if you change Dovecot to authenticate in another way you won t need to make any matching changes to Postfix.
  1. Run mkdir -p /var/spool/postfix/var/spool and ln -s ../.. /var/spool/postfix/var/spool/postfix , this allows parts of Postfix to work with the same configuration regardless of whether they are running in a chroot.
  2. Add the following to /etc/postfix/main.cf and restart Postfix:
    smtpd_sasl_auth_enable = yes
    smtpd_sasl_type = dovecot
    smtpd_sasl_path = /var/spool/postfix/private/auth
    broken_sasl_auth_clients = yes
    smtpd_sasl_authenticated_header = yes
  3. Edit /etc/dovecot/conf.d/10-master.conf, uncomment the following lines, and then restart Dovecot:
    unix_listener /var/spool/postfix/private/auth
    mode = 0666
  4. Edit /etc/postfix/master.cf, uncomment the line for the submission service, and restart Postfix. This makes Postfix listen on port 587 which is allowed through most firewalls.
  5. From another system (IE not the virtual machine you are working on) run swaks -tls -f auser@example.com -t YOURADDRESS@gmail.com -s YOURSERVER and note that the message is rejected with Relay access denied .
  6. Now run swaks -tls --auth-user auser --auth-password WHATEVER -f auser@example.com -t YOURREALADDRESS -s YOURSERVER and observe that the mail is delivered (subject to anti-spam measures at the recipient).
    7. Configuring a MUA If every part of the previous 3 sections is complete then you should be able to setup your favourite MUA. Use auser as the user-name for SMTP and IMAP, mail.example.com for the SMTP/IMAP server and it should just work! Of course you need to use the same DNS server for your MUA to have this just work. But another possibility for testing is to have the MUA talk to the server by IP address not by name.
    Related posts:
    1. Mail Server Security I predict that over the course of the next 10...
    2. I need an LMTP server I am working on a system where a front-end mail...
    3. Moving a Mail Server Nowadays it seems that most serious mail servers (IE mail...

Steve Kemp: Robbing Peter to pay Paul, or location spoofing via DNS

I rarely watched TV online when I was located in the UK, but now I've moved to Finland with appalling local TV choices it has become more common. The biggest problem with trying to watch BBC's iPlayer, and similar services, is the location restrictions. Not a huge problem though: The next part is the harder one - making your traffic pass over the VPN. If you were simple you'd just say "Send everything over the VPN". But that would slow down local traffic, so instead you have to use trickery. My approach was just to run a series of routing additions, similar to this (except I did it in the openvpn configuration, via pushed-routes): 
ip -4 route add .... dev tun0
This works, but it is a pain as you have to add more and more routes. The simpler solution which I switched to after a while was just configuring mitmproxy on the remote OpenVPN end-point, and then configuring that in the browser. With that in use all your traffic goes over the VPN link, if you enable the proxy in your browser, but nothing else will. I've got a network device on-order, which will let me watch netflix, etc, from my TV, and I'm lead to believe this won't let you setup proxies, or similar, to avoid region-bypass. It occurs to me that I can configure my router to give out bogus DNS responses - if the device asks for "iplayer.bbc.com" it can return 10.10.10.10 - which is the remote host running the proxy. I imagine this will be nice and simple, and thought I was being clever: Because I can handle DNS-magic on the router I can essentially spoof my location for all the devices on the internal LAN, which is a good thing. Anyway I was reasonably pleased with the idea of using DNS to route traffic over the VPN, in combination with a transparent proxy. I was even going to blog about it, and say "Hey! This is a cool idea I've never heard of before". Instead I did a quick google(.fi) and discovered that there are companies offering this as a service. They don't mention the proxying bit, but it's clearly what they're doing - for example OverPlay's SmartDNS. So in conclusion I can keep my current setup, or I can use the income I receive from DNS hosting to pay for SmartDNS, or other DNS-based location-fakers. Regardless. DNS. VPN. Good combination. Try it if you get bored.

16 October 2015

Norbert Preining: Debian/TeX Live multiarch update

A big update of all related packages (tex-common 6.04, texlive-bin 2015.20150524.37493-7, texlive-base/lang/extra package 2015.20151016-1) due to the move to support multi-arch. Of course, the regular updates of the TeX Live are included, too. With this change it should be possible to run a multi-arch system with only one TeX Live installed. Debian - TeX Live 2015 Thanks to the excellent support and testing of the Multi-arch guys, in particular Thorsten Glaser, Helmut Grohne, Johannes Schauer, and Wookey, I learned a lot about multi-arch, and I hope that the current setup is safe. All the packages but the various lib* packages are tagged as Multi-Arch: foreign, while the lib packages are tagged Multi-Arch: same. Anyway, if you find a bug concerning multi-arch, that is that some of the programs exhibit architecture information, please let us know via a bug report. Updated packages acro, alegreya, amiri, assoccnt, attachfile, babel-french, babel-hungarian, barr, beebe, biblatex-philosophy, bidi, bnumexpr, caption, chemfig, chemformula, chemmacros, cjk-gs-integrate, csplain, dantelogo, dataref, dtxgen, dvipdfmx-def, dvips, eledmac, elements, fcolumn, fithesis, fontspec, genealogytree, gradstudentresume, gtl, jfontmaps, knuth-local, koma-script, kotex-oblivoir, kotex-plain, kotex-utf, kpathsea, l3build, l3experimental, l3kernel, l3packages, latex, latexconfig, ledmac, ltxfileinfo, lualatex-math, luamplib, luatex, luatexbase, luatexja, luatexko, make4ht, mcf2graph, mflogo, modiagram, multiexpand, newtx, odsfile, old-arrows, paracol, pdfpages, pdftex, plain, pst-stru, pxchfon, randomwalk, reledmac, resumecls, rubik, selnolig, showhyphens, siunitx, suftesi, tetex, teubner, tex4ebook, tex4ht, texlive-scripts, tikzsymbols, tipfr, tools, tudscr, uassign, unicode-math, unravel, visualfaq, xepersian, xetex-def, xint. New packages archaeologie, ctablestack, dynamicnumber, exercises, fibeamer, h2020proposal, imfellenglish, lstbayes, tempora, xellipsis. Enjoy.

15 October 2015

Laura Arjona: Long summer story, Welcome team, and I am a Debian Developer now

Note: 2015/10/16: I need to add some links but I won t delay this more, posting now, will edit later. Summer ended long time ago, but believe me, I m still catching up with all the things that I began in June/July, all the things I left in August when I went holidays, and more things that appeared in August and September. This is a long overdue post, I hope you bear with me for waiting so long, and writing (now) so long too! June In June, I was 100% sure that I would not attend DebConf15 (well, I was 98% sure until then), and when the new Outreach Sponsorship grants were announced, I decided to write some mails to several Debian contributors, so they consider applying for the grant and attend DebConf (and maybe trigger some i18n/l10n meeting ). They kindly declined, and I understood their reasons, but also wondered what would have happened if the proposal would have come from somebody more official instead of a random contributor that they don t know. I also hoped that lots of other Debianites also write to newbies or not-yet-DD-contributors or non-packaging contributors to invite them to DebConf, and I hoped that they had better luck than me in convincing them :) July In July I usually work hard preparing the computer labs for next academic year at my workplace in the University, but I also have more free time in the long afternoons and evenings, since I don t sleep much, and there is not much to do outside with the summer hot. So I used that month to go on contributing to DebConf publicity and think a bit more about Debian and the other free software communities. I didn t put much time in advancing my selfhosting (no SSL yet in *.larjona.net! booooo!) but I decided to deep my toe in Sandstorm.io, and try to selfhost an instance ( http://lacaja.larjona.net ) and try Etherpad inside Sandstorm (since I failed in deploying Etherpad by myself in my jessie+nginx+postgres box). Sandstorm worked, and Etherpad was packaged in Sandstorm so it worked too; and I have my free-software-base pads now for writing and share. So I joined #sandstorm IRC channel since then, and there I learnt that Asheesh Laroia (who works in Sandstorm.io and is also a Debian Developer and was going to give a talk about Sandstorm.io in DebConf15) was offering mentorship for people wanting to learn Sandstorm packaging, and his proposal was to begin packaging Framadate. I also failed in selfhosting Dudle (prepared for Apache + FastCGI, couldn t make it work in my Nginx), so Asheesh s proposal looked suitable for me. We talked and decided to invest the rest of July and first days of August in learning to package Framadate. I learned a lot, but couldn t finish the task. I encountered many issues (setting my dev environment, and later trying to package), and we solved some of them but my time ran out. I posted my work in the list, and I hope that my feedback on the documentation and the issues I encountered helped Asheesh and the Sandstorm community. Framadate is packaged in Sandstorm.io now, Drew Fisher packaged it, not sure if my stuff was useful or not (it s been useful for me, for learning, at least). I ll talk more about Sandstorm.io in a future blog post updating on my selfhosting adventures. What I liked most was the kind of proposal of mentoring that Asheesh made. It was very detailed in every aspect: the task, the things you need to accomplish it, details about his availability for mentorship I try to be welcoming in the teams in which I participate, but the fact is that I fail in actually mentor, maybe because of not making specific proposals to people (until now, I was like Hi, newcomer! Go read this, this and this, and try for yourself any task you feel you like it, and come back if you have issues , la Debian ). This, plus the thoughts about my mails in June for diversity outreach in DebConf, made me feel the need of having a team where people willing to welcome newcomers share tricks and procedures, write together more specific proposals, and follow up the newcomers experiences in a regular way. I talked with Enrico Zini and we wrote down some notes for a Welcome Team in Debian; he said he would spread the word during DebCamp/DebConf and we would see what people thinks about it. August August came, and the day before going on holidays I was really tired: too much luggage to prepare, too many hours in front of the computer, and the usual stress of traveling; and I took the bad decision of signing some GPG keys of several Debianites that I met in July. I say bad decision because the lack of sleep showed its black magic and I accidentally deleted my secring.gpg file. I knew I had a backup but I didn t have too much time to invest and I didn t want to mess it with the backup too, and my laptop was going to stay at home, powered off, during the whole month, so I just went on holidays and left the GPG issue for later. The day after, meanwhile I was waiting in the airport for my boarding time, I received a mail accepting me as Debian Developer. Wow!! Really, I was not expecting that the process was already finished, I had interchanged several mails with my Application Manager (who happens to be the current DPL!) and I thought that his summer could be quite packed of Debian/DebConf work and my process could wait a bit. So it was a very happy news and very motivating after one month (July) full of free software work. On the other side, I was a bit scared: what type of Debian Developer are you, larjona, not capable to sign some GPG keys without breaking your setup?! but I answered myself well, I m the type of Debian Developer that has backups :) and then, with that mixed feelings of excitement and impostor syndrome, I took my plane and went on holidays, not expecting to touch any computer until the end of the month. August is probably the month in the year when I have more free time (holidays), but less time to dedicate to free software. I devote most of the month to visit family and stay with them, with no internet connection available or no free time to look at the mailbox or social networks or IRC But DebCamp and DebConf15 were happening during my holidays. And this DebConf15 was the first one in which I participated in the organization, and the first one in which I felt more than being a consumer of Debian videos . I could not follow the streamings, my only internet-capable device was my Android 2.x phone, but when I had wifi I fetched the mail, and during the nights, while everybody else was sleeping and I was laying on the terrace, below the sky full of stars, I could read batches of hundred of mails from debconf-discuss mailing list. And I could get some feeling from DebConf life, because I learned about the ad-hoc BoFs and discussions, the morning bike rides and swimming proposals, and the dancing classes, the i18m/l10n meeting, and many other things. I could answer some mail from time to time, and I also knew that a fellow Debianite from Madrid was going to bring me some stickers, maybe a t-shirt, and shake hands in my name to some persons. September and October September was about finishing reading all the mails and try to answer the pending ones, and preparing my computer to use my new Debian identity (and stop using larjona-guest). I still have some things to do, pending technical work, and some mails that I should have answered and I ve forgotten, for sure (if you sent me a mail that needs answer or would be fine that I answer (even if it was months ago!), please resend or ping me). I recovered my secring.gpg but and just now I added larjona@debian.org to the ID in my GPG key, but didn t signed the pending keys again (sorry dkg and holger! will catch up there soon). My subkeys expired and I m trying to find out how to proceed (they are in my FSFE SmartCard) :/ About the Debian teams, I ve resumed my work in publicity team (this year I ll try to be more involved, in Debian Project News in particular), partially in the website team, and recently I ve finished catching up with the Spanish translation of the website. I ve also joined the DebConf team again (for DebConf16, no matter I probably won t attend) and documented the Publicity task for DebConf, and I try to engage the mailing list and the IRC meetings. I finally could have time to watch some DebConf15 videos and Andreas Tille s talk ( Creating a more inviting environment for newcomers New experiences from MoM, SoB, Teammetrics ) helped me to step ahead in welcoming people with more useful stuff than Hi, newcomer! Go read this (general URLs), try for yourself whatever you like . I have made specific proposals for two people. In mid September I accepted an interview about Debian for a podcast with quite a lot audience (in Spanish), in which I explained the idea of the Welcome Team and offered myself as first-contact. Since then, two more people have contacted me and I have offered specific tasks I think are suitable for them. I also try to be more available in the IRC and offer some time spans for new contributors to DebConf to explain the git setup, the wiki, and all this stuff that looks more complicated than what it is. And I think that s all. My Debianite friend kindly brought me some stickers and a DebConf t-shirt, plus the organization t-shirt that the team gave me as present for my contributions in DebConf15. Neil McGovern kindly sent me a certificate of my new Debian Developer status (thanks!!), and it s posted in my wall at work. Here you are a photo! larjona_dd.JPG (Note: my wall is full of stickers and pieces of papers with things I need, things I like and things I use to explain my work (sometimes sarcastically/ironically ). Maybe some day I ll make a blog post about that!) I feel very proud and happy. Still, a lot of things to learn and work to do, but my intentions are: to keep on progressing (sometimes fast, sometimes slowly), never give up, and enjoy the multiple flowers I find in my way :) Thanks everybody! October and future Some other ideas/plans for the future (the ones I didn t say yet): Comments? If you want to comment you can use this pump.io thread.
Filed under: My experiences and opinion, News Tagged: Communities, Contributing to libre software, Debian, Developer motivations, encryption, English, Free Software, gpg, libre software

Next.

Previous.