Search Results: "ana"

2 June 2016

Raphaël Hertzog: My Free Software Activities in May 2016

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS Due to some departure and increased workload, I wanted to find out a few new paid contributors for Debian LTS. So I sent a mail to debian-jobs@lists.debian.org and contrary to last time (where I posted the announce only here on my blog), I had plenty of replies I ended up enrolling 6 new contributors and refusing 3 persons that did not have the required profile. All new contributors are supposed to handle at least one LTS update on their free time to get up to speed. But from the 6 new contributors only 3 managed to handle their training update in May. During the month I spent quite some time providing guidance to the new contributors both in private mails and on the debian-lts mailing list. I also reviewed a xen update where I had (rightfully) some doubts about the work done. Packaging work fonts-cantarell. After having diagnosed the problem last month, I got annoyed enough by the lack of a fixed package that I found a way to package a newer upstream release of fonts-cantarell without requiring a fontforge update that was likely to take some time still so I prepared and uploaded 0.0.24-1. cpputest. Bug #823711 reported some license issues with some of the files. I immediately forwarded this upstream (issue 961) and fixed it in Debian by repacking the upstream tarball. Fortunately upstream has been quick to handle his and there s a new upstream release (3.8) where the problematic files have been dropped. live-boot. Kali s live images were no longer booting (stuck in the initrd) and with the help of Ben Hutchings we diagnosed this back to #823069 which I fixed in live-boot 20160511. udev. I filed #824025 to request that the rule defining the MAC-based name of USB network interfaces be isolated in its own file so that it can be easily disabled (we do that in Kali). Misc stuff. I packaged Django 1.8.13 in jessie-backports. I filed
#824165 against sbuild being broken with $apt_allow_unauthenticated = 1; in .sbuildrc. I filed a wishlist bug #824168 against apt-listchanges to suggest that it ignores news from auto-installed packages. I filed #825923 to report a regression in python-nltk (discovered in Kali first). Infrastructure work packages.debian.org. A few months ago, I wrote a patch for packages.debian.org so that it forwards emails to tracker.debian.org instead of packages.qa.debian.org. At that time, I was in touch with Rhonda and was hoping that she would apply it rather quickly (the patch is rather short). After a few more pings, she made it clear that she was not alone and that I should rather file a proper request so that someone else can also process it. So I filed #824085 and tried to find someone else to apply my patch. Most of the members of pkg_maint said that they were part of the group only due to generic webmaster involvement but that they did not want to touch that part. Fortunately, Martin Zobel Helas was more receptive to my request and helped me to deploy my changes. I committed my change and Martin pulled it in the live checkout on picconi.debian.org. This update is also a first step towards the possibility to use foo@packages.debian.org and/or teams+foo@tracker.debian.org in the Maintainer field of a package. With this we can get rid of dedicated mailing lists that just duplicate the work of the package tracker. And we no longer need to care about the fact that the Maintainer is handled differently than Uploaders since all (human) co-maintainers would then be listed in Uploaders only (and the package tracker would deal appropriately with mails sent to the Maintainer). Distro Tracker. I improved the import process to be able to force a new processing of source packages that were already imported. This was useful to let it recognize architectures which were newly added in its database (and that were ignored and thus not displayed up to now). I also made a first review of the AppStream patch submitted by Matthias Klump in #806740. Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Jamie McClelland: Mobile Instant Messaging

Now that I've gone down the signal road, when can I get off of it? The two contenders I've found for more politically conscious mobile-friendly instant messaging are Tox and XMPP. And the problems to solve are: The Tox Project has a great design (peer-to-peer) and handles end-to-end encryption as part of its core design. However, the Antox mobile client has not only failed to solve the battery drain problem but seems to also have a serious bandwidth issue as well. Also, what is up with this drama? How about XMPP? For years, XMPP clients have been using OTR - and many instant messaging applications support it, including ChatSecure. And now, there seems to be quite a bit of excitement around implementing a better protocol called OMEMO. Incidentally, the OMEMO protocol uses the same Double Ratchet (previously referred to as the Axolotl Ratchet) protocol that Signal uses. While there are some irritating hiccups around using the available GPL library for Double Ratchet on iPhone apps, it seems like these issues are being sorted out and soon we'll have some kind of standard way for XMPP clients to exchange end-to-end encrypted messages. Let's move on to notifications and battery drain. If you believe the author of the Android Conversations App there doesn't seem to be a problem. However, client state indication seems to be the only issue related to battery drainage he addresses and I couldn't figure out how to easily install that extension on our Debian Jessie prosody instance since it isn't included in the prosody-modules package in Debian and it requires additional modules for it to work. Chris Ballinger, the author of ChatSecure (iOS and Android XMPP app), is less optimistic about this problem in general. In short, he thinks we need a proper "push" mechanism and has even started to implement one. At the same time, a new (and different) XMPP standard called Push Notifications - XEP-0357 has been released and it is even implemented in Prosody (although not yet available in Debian). So the future seems bright, right? Well, not exactly. All of this "push" activity seems to solve this problem: A federated/decentralized application cannot properly use Apple's APNs or Google's GCM. In the words of Chris Ballinger:
The biggest problem is that there is no easy way to send push messages between my app and your app. For me to send a push to one of your app s users on iOS, I must first obtain an APNs SSL certificate/key pair from you, and one of your user s push token that uniquely identifies their device to Apple. These push tokens are potentially sensitive information because they allow Apple to locate your device (in order to send it a push).
So, even if we manage to get one of these two standards for push notifications up and running, we have only succeeded in solving Signal's centralization problem, not the dependence on Google Play Services and Apple Push Network (in fact it's quite mysterious to me how you could even use the Prosody implementation of Push Notifications with GCM or APN). So... what we really would need would be to figure out how to implement one of these two push standards and then get it to work with an alternative to GCM and APN (perhaps MQTT)? Which, I think would require changes to the XMPP client. Geez. I may be on Signal longer than I planned.

1 June 2016

Antoine Beaupr : (Still) working too much on the computer

I have been using Workrave to try to force me to step away from the computer regularly to work around Repetitive Strain Injury (RSI) issues that have plagued my life on the computer intermittently in the last decade. Workrave itself is only marginally efficient at getting me away from the machine: as any warning systems, it suffers from alarm fatigue as you frenetically click the dismiss button every time a Workrave warning pops up. However, it has other uses.

Analyzing data input In the past, I have used Workrave to document how I work too much on the computer, but never went through more serious processing of the vast data store that Workrave accumulates about mouse movements and keystrokes. Interested in knowing how much my leave from Koumbit affected time spent on the computer, I decided to look into this again. It turns out I am working as much, if not more, on the computer since I took that "time off": per machine keystrokes per day and average We can see here that I type a lot on the computer. Normal days range from 10 000 to 60 000 keystrokes, with extremes at around 100 000 keystrokes per day. The average seem to fluctuate around 30 to 40 000 keystrokes per day, but rises sharply around the end of the second quarter of this very year. For those unfamiliar with the underlying technology, one keystroke is roughly one byte I put on the computer. So the average of 40 000 keystrokes is 40 kilobyte (KB) per day on the computer. That means 15 MB over a year or about 150MB (or 100 MiB if you want to be picky about it) over the course of the last decade. That is a lot of typing. I originally thought this could have been only because I type more now, as opposed to use more the mouse previously. Unfortunately, Workrave also tracks general "active time" which we can also examine: per machine keystrokes per day and average Here we see that I work around 4 hours a day continuously on the computer. That is active time: not just login, logout time. In other words, the time where i look away from the computer and think for a while, jot down notes in my paper agenda or otherwise step away from the computer for small breaks is not counted here. Notice how some days go up to 12 hours and how recently the average went up to 7 hours of continuous activity. So we can clearly see that I basically work more on the computer now than I ever did in the last 7 years. This is a problem - one of the reasons of this time off was to step away from the computer, and it seems I have failed.
Update: it turns out the graph was skewed towards the last samples. I went more easy on the keyboard in the last few days and things have significantly improved: per machine keystrokes per day and average, 3 days later
Another interesting thing we can see is when I switched from using my laptop to using the server as my main workstation, around early 2011, which is about the time marcos was built. Now that marcos has been turned into a home cinema downstairs, I went back to using my laptop as my main computing device, in late 2015. We can also clearly see when I stopped using Koumbit machines near the end of 2015 as well.

Further improvements and struggle for meaning The details of how the graph was produced are explained at the end of this article. This is all quite clunky: it doesn't help that the Workrave data structure is not easily parsable and so easily corruptible. It would be best if each data point was on its own separate line, which would be long, granted, but so easier to parse. Furthermore, I do not like the perl/awk/gnuplot data processing pipeline much. It doesn't allow me to do interesting analysis like averages, means and linear regressions easily. It could be interesting to rewrite the tools in Python to allow better graphs and easier data analysis, using the tools I learned in 2015-09-28-fun-with-batteries. Finally, this looks only at keystrokes and non-idle activity. It could be more interesting to look at idle/active times and generally the amount of time spent on the computer each day. And while it is interesting to know that I basically write a small book on the computer every day (according to Wikipedia, 120KB is about the size of a small pocket book), it is mostly meaningless if all that stuff is machine-readable code. Where is, after all, the meaning in all those shell commands and programs we constantly input on our keyboards, in the grand scheme of human existence? Most of those bytes are bound to be destroyed by garbage collection (my shell's history) or catastrophic backup failures.
While the creative works from the 16th century can still be accessed and used by others, the data in some software programs from the 1990s is already inaccessible. - Lawrence Lessig
But is my shell history relevant? Looking back at old posts on this blog, one has to wonder if the battery life of the Thinkpad 380z laptop or how much e-waste I threw away in 2005 will be of any historical value in 20 years, assuming the data survives that long.

How this graph was made I was happy to find that Workrave has some contrib scripts to do such processing. Unfortunately, those scripts are not shipped with the Debian package, so I requested that to be fixed (#825982). There were also some fixes necessary to make the script work at all: first, there was a syntax error in the Perl script. But then since my data is so old, there was bound to be some data corruption in there: incomplete entries or just plain broken data. I had lines that were all NULL characters, typical of power failures or disk corruptions. So I have made a patch to fix that script (#826021). But this wasn't enough: while this processes data on the current machine fine, it doesn't deal with multiple machines very well. In the last 7 years of data I could find, I was using 3 different machines: this server (marcos), my laptop (angela) and Koumbit's office servers (koumbit). I ended up modifying the contrib scripts to be able to collate that data meaningfully. First, I copied over the data from Koumbit in a local fake-koumbit directory. Second, I mounted marcos home directory locally with SSHFS: 
sshfs anarc.at:/home/anarcat marcos
I also made this script to sum up datasets: 
#!/usr/bin/perl -w
use List::MoreUtils 'pairwise';
$  = 1;
my %data = ();
while (<>)  
    my @fields = split;
    my $date = shift @fields;
    if (defined($data $date ))  
        my @t = pairwise   $a + $b   @ $data $date , @fields;
        $data $date  = \@t;
     
    else  
        $data $date  = \@fields;
     
 
foreach my $d ( sort keys %data )  
    print "$d @ $data $d \n";
Then I made a modified version of the Gnuplot script that processes all those files together: 
#!/usr/bin/gnuplot
set title "Workrave"
set ylabel "Keystrokes per day"
set timefmt "%Y%m%d%H%M"
#set xrange [450000000:*]
set format x "%Y-%m-%d"
set xtics rotate
set xdata time
set terminal svg
set output "workrave.svg"
plot "workrave-angela.dat" using 1:28 title "angela", \
     "workrave-marcos.dat" using 1:28 title "marcos", \
     "workrave-koumbit.dat" using 1:28 title "koumbit", \
     "workrave-sum.dat" using 1:2 smooth sbezier linewidth 3 title "average"
#plot "workrave-angela.dat" using 1:28 smooth sbezier title "angela", \
#     "workrave-marcos.dat" using 1:28 smooth sbezier title "marcos", \
#     "workrave-koumbit.dat" using 1:28 smooth sbezier title "koumbit"
And finally, I made a small shell script to glue this all together: 
#!/bin/sh
perl workrave-dump > workrave-$(hostname).dat
HOME=$HOME/marcos perl workrave-dump > workrave-marcos.dat
HOME=$PWD/fake-koumbit perl workrave-dump > workrave-koumbit.dat
# remove idle days as they skew the average
sed -i '/ 0$/d' workrave-*.dat
# per-day granularity
sed -i 's/^\(........\)....\? /\1 /' workrave-*.dat
# sum up all graphs
cat workrave-*.dat   sort   perl sum.pl > workrave.dat
./gnuplot-workrave-anarcat
I used a different gnuplot script to generate the activity graph: 
#!/usr/bin/gnuplot
set title "Workrave"
set ylabel "Active hours per day"
set timefmt "%Y%m%d%H%M"
#set xrange [450000000:*]
set format x "%Y-%m-%d"
set xtics rotate
set xdata time
set terminal svg
set output "workrave.svg"
plot "workrave-angela.dat"  using 1:($23/3600) title "angela", \
     "workrave-marcos.dat"  using 1:($23/3600) title "marcos", \
     "workrave-koumbit.dat" using 1:($23/3600) title "koumbit", \
     "workrave.dat"         using 1:($23/3600) title "average" smooth sbezier linewidth 3
#plot "workrave-angela.dat" using 1:28 smooth sbezier title "angela", \
#     "workrave-marcos.dat" using 1:28 smooth sbezier title "marcos", \
#     "workrave-koumbit.dat" using 1:28 smooth sbezier title "koumbit"

Keerthana Krishnan: Dealing with the Light Blue Screen Of Doom

Hi! With my third consecutive post of the day, I d like to speak about the login error usually seen on Gnome desktops, aka : Light Blue Screen of Doom . As far as I know, this bug is associated with Gnome and is usually because of faulty installation of gnome files and it was something that tormented me endlessly last week when I tried to install Debian 8 (Jessie) Unfortunately, I never really fixed it myself. What started off as a problem with gnome quickly snowballed into something else that wiped my routing table, making sure I couldn t even update the corrupted files properly. Although I came across a lot of bug reports in my quest to solve it, I never really found a solution.
Reinstallation seems to be only option left here, but there are a few checks you can make
  1. Call the terminal. Use Ctrl+Alt+F2, you should be able to get your command line.
  2. Login as root. This is extreme troubleshooting. You need root
  3. Check and see if startx works. (Yay! problem solved if it does!)
  4. Try to see if the basic gnome packages are installed type apt-get install with basic gnome packages like gnome-desktop-environment, gdm, gnome, etc.. to see if they are installed. If not, try to install them and then run the startx command.
  5. If you can t install them, or if they are already installed, try running the updating the system

    apt-get update
    apt-get upgrade
    apt-get dist-upgrade //If OS is at newest version
  6. If updating doesn t work and gives Failed to Fetch errors for all the source links, try pinging the network to make sure you have a connection.

    ping google.com
    ping 8.8.8.8
  7. If you do have a connection, verify the sources :

    nano /etc/apt/sources.list    deb http://ftp.us.debian.org/debian jessie main contrib non-free
    #deb-src http://ftp.us.debian.org/debian jessie main contrib non-free
    deb http://security.debian.org/ jessie/updates main contrib non-free
    #deb-src http://security.debian.org/ jessie/updates main contrib non-free
    deb http://ftp.us.debian.org/debian testing main contrib non-free
    #deb-src http://ftp.us.debian.org/debian testing main contrib non-free
    deb http://security.debian.org/ testing/updates main contrib non-free
    #deb-src http://security.debian.org/ testing/updates main contrib non-free
  8. If you have no connection, check your routing table with the command netstat -nr . If this routing table is completely empty, then I ve got some bad news for you. Reinstallation is your new friend. Try and see if you can copy your files using command line. cp and ls are basically your new best friends.
Here s a guide to help change your OS. Hopefully, everything works out and you re back to working in no time!

Keerthana Krishnan: Things I wish someone told me about Debian 8 (Jessie) installation

After using Ubuntu for the past 5+ years, I m pretty used to linux. The huge mistake I made was assuming that Debian installation would be as easy as Ubuntu. Truth : It isn t. Ubuntu is much more user friendly and tame. Debian is much more larger with more cuztomisation options, something that might be overwhelming to those used to the slick UX of Unity. Disclaimer : This guide is for people who are used to linux, preferably some other distro like Fedora or Ubuntu and wants to change to Debian. Note : If you want to dual boot Debian with Fedora, install debian first. Fedora is pretty good at the partition adjustment thing. Now go ahead and install all the software you want! A good list is given here. Have fun with your new OS! (Edit: I m still getting used to Debian so any and all comments about the above are welcome! )

Keerthana Krishnan: Changing your OS between two linux distros

Hi guys! Last week, I had installed Debian 8 with Fedora 22 over my then current system of a dual boot between Ubuntu 14.04 and Fedora 22. I had been using these two for a long time and the change was pretty difficult because of all the data I had lying around. This is all the wisdom I understood from it, and hopefully, this helps.
  1. Backup all your important data : Your home folder or any external partition will have tons of data if you have been using your OS for some time. Don t forget to double check all the folders such as your Desktop, localhost (/var/www/html/) and anything else you can think of.
  2. Keep a list of your favorite applications that you ve installed. I wouldn t recommed copying the lib files from the computer. There is a high chance of getting your files corrupted that way (from personal experience). Instead, go to your package manager to find your installed packages and make a note of all the applications that you would want in your new OS.
  3. Backup files from your browser. If you use Google Chrome (Or Chromium) be sure to sign in and save your settings. For firefox users, here s a guide to backup your profile. If your browser is something else, make sure to ask google for help with this,
  4. Download the .iso file of your new OS according to your requirement. I prefer to make a live usb for installation. Make sure you double check and download the right image file.
You re all done! Say your goodbyes to your old system, put in your USB and reboot your system to start installation of your new OS.

31 May 2016

Chris Lamb: Free software activities in May 2016

Here is my monthly update covering a large part of what I have been doing in the free software world (previously):
Debian My work in the Reproducible Builds project was covered in our weekly reports. (#53, #54, #55, #56 & #57)
Debian LTS

This month I have been paid to work 18 hours on Debian Long Term Support (LTS). In that time I did the following:
  • A week of "frontdesk" duties, triaging CVEs, assigning tasks, etc.
  • Issued DLA 464-1 for libav, a multimedia player, server, encoder and transcoder library that fixed a use-after free vulnerability.
  • Issued DLA 469-1 for libgwenhywfar (an OS abstraction layer that allows porting of software to different operating systems like Linux, *BSD, Windows, etc.) correcting the use of an outdated CA certificate bundle.
  • Issued DLA 470-1 for libksba, a X.509 and CMS certificate support library. patching a buffer vulnerability.
  • Issued DLA 474-1 for dosfstools, a collection of utilities for making and checking MS-DOS FAT filesystems, fixing an invalid memory and heap overflow vulnerability.
  • Issued DLA 482-1 for libgd2 graphics library, rectifying a stack consumption vulnerability.
Uploads
  • python-django (1.9.6-1) New upstream bugfix release.
  • redis (3.2.0-1, etc.) New upstream release, correct build on more exotic architectures and minor packaging fixups.
  • gunicorn (19.5.0-1 & 19.6.0-1) New upstream releases and minor packaging fixups.

30 May 2016

Russ Allbery: Review: By the Sword

Review: By the Sword, by Mercedes Lackey
Series: Vows and Honor #4
Publisher: DAW
Copyright: February 1991
ISBN: 0-88677-463-2
Format: Mass market
Pages: 492
By the Sword is the next book in my (slow) Valdemar re-read. This one is a bit hard to classify in the series; it's technically a stand-alone novel, and it doesn't require a lot of prior series knowledge. But the heroine, Kerowyn, is a relative of Tarma and Kethry, and Tarma and Kethry appear in this novel. Most of the book also deals with similar themes as the rest of the Tarma and Kethry books, even though it's also a bridge into Valdemar proper. I'm going to follow Fantastic Fiction and call it book four of the Vows and Honor series, even though the publisher doesn't refer to it that way and it's not strictly correct. I think that creates the right impression, and it's mildly better to read the other Tarma and Kethry novels first. This book is also a bit confusing for reading order. It was published just before the Mage Winds trilogy, and happens before them in series chronological order (between that trilogy and the Talia series). But some of the chronologies in some of the Valdemar books show it after the Mage Winds trilogy. I think I originally read it afterwards, but both natural reading order and publication order puts it first, and that's the ordering I followed this time. Series ordering trivia aside (sometimes the comic book shared universe continuity geek in me raises its head), By the Sword is a hefty, self-contained novel about a very typical Lackey protagonist. Kerowyn is the daughter of a noble house, largely ignored by her father in favor of her brother and tasked with keeping the keep running since her mother died. She wants to learn to fight and ride, but that's not part of her father's plans for her. But those plans become suddenly irrelevant when the keep is attacked during her brother's wedding and the bride kidnapped. Unless someone at least attempts to recover her, this will be taken as an excuse for conquest of the keep by the bride's family. (Spoilers for the start of the book in the following paragraph. I think the outcomes are reasonably obvious given the type of book this is, but skip it if you don't want to know anything about the plot.) If you're familiar with Lackey's musical work (most probably won't be, but you might if you follow filk), "Kerowyn's Ride" is the start of this book. Kerowyn goes to her grandmother Kethry, who is semi-legendary to Kerowyn but well-known to readers of the rest of the series. From Kethry, she acquires Need; with Need's help, she improbably manages to rescue her brother's bride. It seems like a happy ending, but it completely disrupts and destroys her life. Her role as hero does not fit any of the expectations the remaining members of the household have for her. But it also gives her an escape: she ends up as Tarma and Kethry's student, learning all the things about fighting she'd craved to learn and preparing for a life as a mercenary. Quite a few adventures follow, all of which are familiar to Lackey readers and particularly to readers of the Tarma and Kethry books. But I think this is one of Lackey's better-written books. The pacing is reasonably good despite the length of the book, Kerowyn is a likable and interesting character, and I like the pragmatism and practicalities that Lackey brings to sword and sorcery mercenary groups. In style and subject matter, it's the closest to Oathbreakers, which was also my favorite of the Tarma and Kethry novels. By the Sword is both the natural conclusion of the Tarma and Kethry era and arc, and vital foundational material for what I think of as the "core" Valdemar story: Elspeth's adventures during Selany's reign, which start in the Mage Winds trilogy immediately following this. Kerowyn becomes a vital supporting character in the rest of the story, and Need is hugely important in events to come. But even if you're not as invested in the overall Valdemar story arc as I am, this is solid, if a bit predictable and unspectacular, sword and sorcery writing presented in a meaty and satisfying novel with a good coming-of-age story. This is one of my favorites of the Valdemar series as measured by pure story-telling. There are other books that provide more interesting lore and world background, but there are few characters I like as well as Kerowyn, and I find the compromise she reaches with Need delightful. If you liked Oathbreakers, I'm pretty sure you'd like this as well. And, of course, recommended if you're reading the whole Valdemar series as a fairly key link in the plot and a significant bridge between the Heralds and Tarma and Kethry's world, a bridge that Elspeth is about to cross in the other direction. Rating: 8 out of 10

Reproducible builds folks: Reproducible builds: week 57 in Stretch cycle

What happened in the Reproducible Builds effort between May 22nd and May 28th 2016: Media coverage Documentation update Toolchain fixes Packages fixed The following 18 packages have become reproducible due to changes in their build dependencies: canl-c configshell dbus-java dune-common frobby frown installation-guide jexcelapi libjsyntaxpane-java malaga octave-ocs paje.app pd-boids pfstools r-cran-rniftilib scscp-imcce snort vim-addon-manager The following packages have become reproducible after being fixed: Some uploads have fixed some reproducibility issues, but not all of them: Patches submitted that have not made their way to the archive yet: Package reviews 123 reviews have been added, 57 have been updated and 135 have been removed in this week. 21 FTBFS bugs have been reported by Chris Lamb and Santiago Vila. strip-nondeterminism development tests.reproducible-builds.org Misc. This week's edition was written by Reiner Herrmann and Holger Levsen and reviewed by a bunch of Reproducible builds folks on IRC.

Daniel Stender: My work for Debian in May

No double posting this time ;-) I've got not so much spare time this month to spend on Debian, but I could work on the following packages: This series of blog postings also includes little introductions of and into new packages in the archive. This month there is: Pyinfra Pyinfra is a new project which is currently still in development state. It has been already pointed out in an interesting German article1, and is now available as package maintained within the Python Applications Team. It's currently a one man production by Nick Barrett, and eagerly developed in the past weeks (we're currently at 0.1~dev24). Pyinfra is a remote server configuration/provisioning/service deployment tool which belongs in the same software category like Puppet or Ansible2. It's for provisioning one or an array of remote servers with software packages and to configure them. Pyinfra runs agentless like Ansible, that means for using it nothing special (like a daemon) has to run on targeted servers. It's written to be used for provisioning POSIX compatible Linux systems and has alternatives when it comes to special features like package managers (e.g. supports apt as well as yum). The documentation could be found in usr/share/doc/pyinfra/html/. Here's a little crash course on how to use Pyinfra: The pyinfra CLI tool is used on the command line like this, deploy scripts, single operations or facts (see below) could be used on a single server or a multitude of remote servers: 
$ pyinfra -i <inventory script/single host> <deploy script>
$ pyinfra -i <inventory script/single host> --run <operation>
$ pyinfra -i <inventory script/single host> --facts <fact>
Remote servers which are operated on must provide a working shell and they must be reachable by SSH. For connecting, --port, --user, --password, --key/--key-password and --sudo flags are available, --sudo to gain superuser rights. Root access or sudo rights of course have to be already set up. By the way, localhost could be operated on the same way. Single operations are organized in modules like "apt", "files", "init", "server" etc. With the --run option they could be used individually on servers like follows, e.g. server.user adds a new user on a single targeted system (-v adds verbosity to the pyinfra run): 
$ pyinfra -i 192.0.2.10 --run server.user sam --user root --key ~/.ssh/sshkey --key-password 123456 -v
Multiple servers can be grouped in inventories, which hold the targeted hosts and data associated with them, like e.g. an inventory file farm1.py would contain lists like this: 
COMPUTE_SERVERS = ['192.0.2.10', '192.0.2.11']
DATABASE_SERVERS = ['192.0.2.20', '192.0.2.21']
Group designators must be all caps. A higher level of grouping are the file names of inventory scripts, thus COMPUTE_SERVERS and DATABASE_SERVERS can be referenced to at the same time by the group designator farm1. Plus, all servers are automatically added to the group all. And, inventory scripts should be stored in the subfolder inventory/ in the project directory. Inventory files then could be used instead of specific IP addresses like this, the single operation then gets performed on all given machines in farm1.py: 
$ pyinfra -i inventory/farm1.py  --run server.user sam --user root --key ~/.ssh/sshkey --key-password=123456 -v
Deployment scripts could be used together with group data files in the subfolder group_data/ in the project directory. For example, a group_data/farm1.py designates all servers given in inventory/farm1.py (by the way, all.py designates all servers), and contains the random attribute user_name (attributes must be lowercase), next to authentication data for the whole inventory group: 
user_name = 'sam'
ssh_user = 'root'
ssh_key = '~/.ssh/sshkey'
ssh_key_password = '123456'
The random attribute can be picked up by a deployment script using host.data() like follows, user_name could be used again for e.g. server.user(), like this: 
from pyinfra import host
from pyinfra.modules import server
server.user(host.data.user_name)
This deploy, the ensemble of inventory file, group data file and deployment script (usually placed top level in the project folder) then could be run that way: 
$ pyinfra -i inventory/farm1.py deploy.py
You have guessed it, since deployment scripts are Python scripts they are fully programmable (please regard that Pyinfra is build & runs on Python 3 on Debian), and that's the main advantage point with this piece of software. Quite handy for that come Pyinfra facts, functions which check different things on remote systems and return information as Python data. Like e.g. deb_packages returns a dictionary of installed packages from a remote apt based server: 
$ pyinfra -i 192.0.2.10 --fact deb_packages --user root --key ~/.ssh/sshkey --key-password=123456
 
    "192.0.2.10":  
        "libdebconfclient0": "0.192",
        "python-debian": "0.1.27",
        "libavahi-client3": "0.6.31-5",
        "dbus": "1.8.20-0+deb8u1",
        "libustr-1.0-1": "1.0.4-3+b2",
        "sed": "4.2.2-4+b1",
Using facts, Pyinfra reveals its full potential. For example, a deployment script could go like this, linux.distribution() returns a dict containing the installed distribution: 
from pyinfra import host
from pyinfra.modules import apt
if host.fact.linux_distribution['name'] == 'Debian':
   apt.packages(packages='gummi', present=True, update=True)
elif host.fact.linux_distribution['name'] == 'CentOS':
   pass
I'll spare more sophisticated examples to keep this introduction simple. Beyond fancy deployment scripts, Pyinfra features an own API by which it could be programmed from the outside, and much more. But maybe that's enough to introduce Pyinfra. That are the usage basics. Pyinfra is a brand new project and it remains to be seen whether the developer can keep on further developing the tool like he does these days. For a private project it's insane to attempt to become a contender for the established "big" free configuration management tools and frameworks, but, if Puppet has become too complex in the meanwhile or not3, I really don't think that's the point here. Pyinfra follows an own approach in being programmable the way which it is. And it's definitely not harm to have it in the toolbox already, not trying to replace nothing. Brainstorm After the first package has been in experimental, the Brainstorm library from Swiss AI research institute IDSIA4 is now available as python3-brainstorm in unstable. Brainstorm is a lean, easy-to-use library for setting up deep learning networks (multiple layered artificial neural networks) for machine learning applications like for image and speech recognition or natural language processing. To set up a working training network for a classifier for handwritten digits like the MNIST dataset (a usual "hello world") just takes a couple of lines, like an example demonstrates. The package is maintained within the Debian Python Modules Team. The Debian package ships a couple of examples in /usr/share/python3-brainstorm/examples (the data/ and examples/ folders of the upstream tarball are combined here). Among them there are5: The current documentation in /usr/share/doc/python3-brainstorm/html/ isn't complete yet (several chapters are under construction), but there's a walkthrough on the CIFAR-10 example. The MNIST example has been extended by Github user pinae, and has been explained in German C't recently6. What are the perspectives for further development? Like Zhou Mo confirmed, there are a couple of deep learning frameworks around having a rather poor outlook since there have been abandoned after being completed as PhD projects. There's really no point for thriving to have them all in Debian, like the ITP of Minerva has been given up partly for this reason, there weren't any commits since 08/2015 (and because cuDNN isn't available and most likely won't). Brainstorm, 0.5 have been released 05/2015, also was a PhD project as IDSIA. It's stated on Github that the project is "under active development", but the rather sparse project page on the other side expresses the "hope the community will help us to further improve Brainstorm". This sentence much often implies that the developers are not actively working on the project. But there are recent commits and it looks that upstream is active and could be reached when there are problems, and that the project is active. So I don't think we're riding a dead horse, here. The downside for Brainstorm in Debian is, it seems that the libraries which are needed for GPU accelerated processing can't be fully provided. Pycuda is available, but scikit-cuda (an additional library which provides wrappers for CUDA features like CUBLAS, CUFFT and CUSOLVER) is not and won't be, because the CULA Dense Toolkit (scikit-cuda also contains wrappers for also that) is not available freely as source. Because of that, a dependency against pycuda, not even as Suggests (it's non-free), has been spared. Without GPU acceleration, Brainstorm computes the matrices on openBLAS using a Cython wrapper on the NumpyHandler, and the PyCudaHandler couldn't be used. openBLAS makes pretty good use of the available hardware (it distributes over all available CPU cores), but it's not yet possible to run Brainstorm full throttle using available floating point devices to reduce training times, which becomes crucial when the projects are getting bigger. Brainstorm belongs to the number of deep learning frameworks already being or becoming available in Debian. Currently there is: I've checked over Microsoft's CNTK, but although it's also set free recently I have my doubts if that could be included. Apparently there are dependencies against non-free software and most likely other issues. So much for a little update on the state of deep learning in Debian, please excuse if my radar misses something.
  1. Tim Sch rmann: "Schlangen l: Automatisiertes Service-Deployment mit Pyinfra". In: IT-Administrator 05/2016, pp. 90-95.
  2. For a comparison of configuration management software like this, see B wetter/Johannsen/Steig: "Baukastensysteme: Konfigurationsmanagement mit Open-Source-Software". In: iX 04/2016, pp. 94-99 (please excuse the prevalence of German articles in the pointers, I've just have them at hand).
  3. On the points of critique on Puppet, see Martin Loschwitz: "David gegen Goliath Zwei Welten treffen aufeinander: Puppet und Ansible". In Linux-Magazin 01/2016, 50-54.
  4. See the interview with IDSIA's deep learning guru J rgen Schmidhuber in German C't 2014/09, p. 148
  5. The examples scripts need some more finetuning. To run the data creation scripts in place the environment variable BRAINSTORM_DATA_DIR could be set, but the trained networks are currently tried to write in place. So please copy the scripts into some workspace if you want to try them out. I'll patch the example scripts to run out-of-the-box, soon.
  6. Johannes Merkert: "Ziffernlerner. Ein k nstliches neuronales Netz selber gebaut". In: C't 2016/06, p. 142-147. Web: http://www.heise.de/ct/ausgabe/2016-6-Ein-kuenstliches-neuronales-Netz-selbst-gebaut-3118857.html.
  7. See Ramon Wartala: "Tiefensch rfe: Deep learning mit NVIDIAs Jetson-TX1-Board und dem Caffe-Framework". In: iX 06/2016, pp. 100-103
  8. https://lists.debian.org/debian-science/2016/03/msg00016.html

Russ Allbery: Review: Empires of EVE

Review: Empires of EVE, by Andrew Groen
Publisher: Andrew Groen
Copyright: 2015
Printing: 2016
ISBN: 0-9909724-0-2
Format: Hardcover
Pages: 171
The version of this book I read was the hardcover Kickstarter campaign reward, since I was a backer. I believe it's the same as the hardcover currently available for sale from the author's site for those who didn't back the project. There are also softcover and Kindle versions. I've lost track of whether they have less sidebar content, or just less high-quality artwork. EVE Online ("EVE" is not an acronym, just the developer's way of writing the name, so you'll also see "Eve" both for the game and for this book) is a massively multiplayer on-line role-playing game (MMORPG) based on interstellar mining, manufacturing, and combat. Its Icelandic developer takes a different, more emergent approach than most MMORPG developers: rather than fill the world exclusively with pre-scripted adventures and enemies (although there is some of that for those who want it), vast regions of EVE's world are left open to the players to govern, exploit, or fight over as they see fit. Player versus player combat plays a large role in that aspect of the game, and many actions that would be prohibited or made impossible in other games (stealing from other players, betrayals, tricking other players into fatal situations) are permitted and sometimes core components of the game. EVE is best-known for its economy, which is almost entirely player-driven and requires extensive mining and manufacturing work by large teams of players to build the largest and most powerful ships in the game. Empires of EVE is an unusual type of book, one that I'm not sure would have been possible ten years ago. Subtitled "a history of the great wars of EVE Online," it's a history of a virtual world, but not one sponsored by the developer or part of the marketing or lore of the game. I love seeing this (which is also why I backed the Kickstarter). Video games have developed beyond just games to play into games to watch other people play (successfully competing, for me and for many others, for the role previously filled by professional sports), and now into emergent events that are complex enough, and dramatic enough, to warrant their own third-party history. I was quite surprised and delighted by how broad the audience for this sort of writing is. And this is not a shallow effort. Andrew Groen is a freelance writer who does not, himself, play EVE. He approaches the complex political and in-game fighting with the attitude of a reporter and historian, cites sources (to the standards I would expect for long-form journalism, if not quite at the level of academic history), discloses where history has been lost or one side of some fight could not be contacted, and puts substantial effort into explaining the political strengths and weaknesses of the shifting in-game alliances. It's a proper political history of an imaginary world, including objective (so far as I can tell) reporting of times when developers were accused of assisting one of the factions. If, like me, you don't play EVE and are primarily interested in this book to get a feel for the game, there are a few caveats to be aware of. EVE is divided into regions with game-enforced security levels. The ones near the center of the game galaxy are heavily policed by the game to prevent most of the player versus player combat and let new players get their feet. Those regions also offer various built-in missions that don't require interacting with other players. As one moves out from that central area, the game-provided security (and I believe the game-provided interactions) drops off. Empires of EVE deals exclusively with nullsec space: the outer regions of the game where the richest resources are, and where there is no law or policing except what's done by the players themselves. The game here, and as described in the book, is relentlessly blood-thirsty, but this isn't representative of the entire game. Second, most of this book is devoted to ship-to-ship combat and missions of conquest and reprisal. But combat is built on top of a vast "civilian" infrastructure of mining and manufacturing, and there are players who focus on those aspects of the game and rarely, if ever, fight. Groen talks about this in passing, since it can have significant influence on the politics of the game, but spends little time describing the day-to-day life in the game for those players. The focus here is on the resulting combat. Finally, Groen starts his history at the EVE beta in 2003 but ends it in 2009. Maneuvering and wars have, of course, continued ever since, but he has to stop somewhere. As he mentioned during the Kickstarter, it takes some years for events to become history, and for people to be willing to talk about them. The developer has kept changing the game, so some of the mechanics here will be mildly stale and the current political alliances are quite likely far different (although many of the players discussed in this book are still playing). With those caveats, though, this is a fascinating book, even if this isn't your sort of game. Personally, I have a deep-seated dislike of games like Diplomacy where alliances, betrayal, and political maneuvering is sanctioned and encouraged by the game. I've never played EVE and I can't imagine ever wanting to, particularly after reading this book. But it's still a fascinating war history and analysis of slightly skew human politics. The alliances and backstabbing are reminiscent of real human history, but the game setting adds some significant twists: players can just quit if they're not having fun, the largest threat to strong alliances is players just not bothering to show up because they're not having fun or because the risk is too high, and without the stability and momentum of real-world institutions, in-game corporations and alliances can collapse overnight when players lose faith in them. From a distance, it's quite entertaining to see how those factors reshape politics and propaganda. (If sadly somewhat reminiscent of the sort of personalities that killed Usenet. I shouldn't have been surprised that organized Internet trolls made an appearance in EVE, although the nature of EVE politics meant they fit right in rather than doing much to spoil other people's fun, at least in nullsec.) I can't speak to the other formats, but the hardcover is also a gorgeous book. The artwork isn't quite my style, and the in-game screen shots are a bit muddy and confusing (not Groen's fault), but the maps are clear and invaluable, the propaganda posters are amazing, and the hardcover printing is clearly of a very high quality. I felt like I got my $50 worth in terms of presentation and quality printing, and have a book that isn't going to fall apart in a few years. The one quibble that I have is that Groen picked a fairly thin and light font for the main text, which my eyes had some trouble with on high-gloss paper. Something thicker and darker may not have looked as good on the page, but it would have been easier to read. The font choice for the sidebars was a lovely high-contrast white on black, but I would have appreciated something a bit larger. If you have vision issues, you may find the ebook more readable, if not as beautiful. Empires of EVE has apparently been very successful within its niche, which makes me happy. I would love to read more books of this type. I know I'm not the only person who grew up with gaming but has been increasingly drawn into watching other people game or reading about other people game. I love watching or reading about people who have put the effort into becoming very good at something they love, and when that comes with emergent history, creative propaganda, and a lot of assholes to root against (although sadly not many people to root for), the history of EVE makes compelling reading, at least for me. Recommended if this is at all your sort of thing; the ebook version of the book is fairly inexpensive. Rating: 8 out of 10

Sean Whitton: Skype inside Firejail version 0.9.40-rc1

Since my PGP key is on its way into the Debian Maintainers keyring, I feel that I should be more careful about computer security. This week I find that I need to run Skype in order to make some calls to some landlines. With the new release candidate of Firejail, it s really easy to minimise the threat from its non-free code. Firstly, check that the Skype .deb you download from their website merely installs files and does not run any prerm or postinst scripts. You can run dpkg-deb --control skype-debian_4.3.0.37-1_i386.deb and confirm that there s nothing executable in there. You should also list the contents with dpkg-deb --contents skype-debian_4.3.0.37-1_i386.deb, and confirm that it doesn t install anything to places that will be executed by the system, such as to /etc/cron.d. For my own reference the safe .deb has sha256 hash a820e641d1ee3fece3fdf206f384eb65e764d7b1ceff3bc5dee818beb319993c, but you should perform these checks yourself. Then install Firejail and Xephyr. You can hook Firejail and Xephyr together manually, but Firejail version 0.9.40-rc1 can do it for you, which is very convenient, so we install that from the Debian Experimental archive: 
# apt-get install xserver-xephyr firejail/experimental
Here s an invocation to use the jail: 
$ firejail --x11=xephyr --private --private-tmp openbox
$ DISPLAY=$(firemon --x11   grep "DISPLAY"   sed 's/   DISPLAY //') \
  firejail --private --private-tmp skype
This takes advantage of Firejail s existing jail profile for Skype. We get the following: This isn t perfect. An annoyance is that the Xephyr window sticks around when you close Skype. More seriously, computer security is always an attacker s advantage game, so this is just an attempt at reducing (optimistically: minimising) the threat posed by non-free code. Update 2016/vi/1: use openbox

28 May 2016

Evgeni Golov: how to accidentally break DNS for 15 domains or why you maybe could not send mail to me

TL;DR: DNS for golov.de and other (14) domains hosted on my infra was flaky from 15th to 17th of May, which may have resulted in undelivered mail. Yeah, I know, I haven't blogged for quite some time. Even not after I switched the engine of my blog from WordPress to Nikola. Sorry! But this post is not about apologizing or at least not for not blogging. Last Tuesday, mika sent me a direct message on Twitter (around 13:00) that read problem auf deiner Seite? or problem on your side/page? . Given side and page are the same word in German, I thought he meant my (this) website, so I quickly fired up a browser, checked that the site loads (I even checked both, HTTP and HTTPS! :-)) and as everything seemed to be fine and I was at a customer I only briefly replied ? . A couple messages later we found out that mika tried to send a screenshot (from his phone) but that got lost somewhere. A quick protocol change later (yay, Signal!) and I got the screenshot. It said "<evgeni+grml@golov.de>: Host or domain name not found. Name service error for name=golov.de type=AAAA: Host found, but no data record of requested type". Well, yeah, that looks like an useful error message. And here the journey begins. For historical nonsense golov.de currently does not have any AAAA records, so it looked odd that Postfix tried that. Even odder was that dig MX golov.de and dig mail.golov.de worked just fine from my laptop. Still, the message looked worrying and I decided to dig deeper. golov.de is served by three nameservers: ns.die-welt.net, ns2.die-welt and ns.inwx.de and dig was showing proper replies from ns2.die-welt.net and ns.inwx.de but not from ns.die-welt.net, which is the master. That was weird, but gave a direction to look at, and explained why my initial tests were OK. Another interesting data-point was that die-welt.net was served just fine from all three nameservers. Let's quickly SSH into that machine and look what's happening Yeah, but I only have my work laptop with me, which does not have my root key (and I still did not manage to setup a Yubikey/Nitrokey/whatver). Thankfully my key was allowed to access the hypervisor, yay console! Now let's really look. golov.de is served from from the bind backend of my PowerDNS, while die-welt.net is served from the MySQL backend. That explains why one domain didn't work while the other did. The relevant zone file looked fine, but the zones.conf was empty. WTF?! That zones.conf is autogenerated by Froxlor and I had upgraded it during the weekend to get Let's Encrypt support. Oh well, seems I hit a bug, damn. A few PHP hacks later and I got my zones.conf generated properly again and all was good. But what had really happened?

  • On Saturday (around 17:00) I upgraded to Froxlor 0.9.35.1 to get Let's Encrypt support and hit Froxlor bug 1615 without noticing as PowerDNS re-reads zones.conf only when told.

  • On Sunday PowerDNS was restarted because of upgraded packages, thus re-reading zones.conf and properly logging: 

    May 15 08:10:59 shokki pdns[2210]: [bindbackend] Parsing 0 domain(s), will report when done

  • On Tuesday the issue hit a friend who cared and notified me

  • On Tuesday the issue was fixed (first by a quick restore from etckeeper, later by fixing the generating code): 

    May 17 14:56:08 shokki pdns[24422]: [bindbackend] Parsing 15 domain(s), will report when done
And the lessons learned?
  • Monitor all your domains, on all your nameservers. (I didn't)
  • Have emergency access to all you servers. (I did, but it was complicated)
  • Use etckeeper, it's easier to use than backups in such cases.
  • When hitting bugs, look in the bugtracker before solving the issue yourself. (I didn't)
  • Have friends who care :-)

Petter Reinholdtsen: Tor - from its creators mouth 11 years ago

A little more than 11 years ago, one of the creators of Tor, and the current President of the Tor project, Roger Dingledine, gave a talk for the members of the Norwegian Unix User group (NUUG). A video of the talk was recorded, and today, thanks to the great help from David Noble, I finally was able to publish the video of the talk on Frikanalen, the Norwegian open channel TV station where NUUG currently publishes its talks. You can watch the live stream using a web browser with WebM support, or check out the recording on the video on demand page for the talk "Tor: Anonymous communication for the US Department of Defence...and you.". Here is the video included for those of you using browsers with HTML video and Ogg Theora support: I guess the gist of the talk can be summarised quite simply: If you want to help the military in USA (and everyone else), use Tor. :)

26 May 2016

Iustin Pop: First run in 2016

Today I finally ran a bit outside, for the first time in 2016. Actually, for even longer the first run since May 2015. I have been only biking in the last year, so this was a very pleasant change of pace (hah), even if just a short run (below 4K). The funny thing is that since I've been biking consistently (and hard) in the last two months, my fitness level is reasonable, so I managed to beat my all-time personal records for 1 Km and 1 mile (I never sprint, so these are just 'best of' segments out of longer runs). It's probably because I only did ~3.8Km, but still, I was very surprised, since I planned and did an easy run. How could I beat my all-time PR, even better than the times back in 2012 when I was doing regular running? Even the average pace over the entire run was better than my last training runs (~5Km) back in April/May 2015, by 15-45s. I guess cross-training does work after all, at least when competing against myself

Norbert Preining: Shotwell vs. digiKam

How to manage your photos? That is probably the biggest question for anyone doing anything with a photo camera. As resolutions of cameras grow, the data we have to manage is growing ever. In my case I am talking about more than 50000 photos and videos measuring up to about 200Gb of disk space, constantly growing. There are several photo management softwares out there, I guess the most commonly used ones are Shotwell for the Gnome desktop, digiKam for the KDE world, and FotoXX. I have not used Shotwell and digiKam for quite some time, and collect here my experiences of strength and weaknesses of the two programs. FotoXX seems to be very powerful, too, but I haven t tested it till now.
shotwell-digikam There is no clear winner here, unfortunately. Both have their strength and their weaknesses. And as a consequence I am using both in parallel. Before I start a clear declaration: I have been using Shotwell for many years, and have myself contributed considerable code to Shotwell, in particular the whole comment system (comments for photos and events), as well as improved the Piwigo upload features. I started using digiKam some month ago when I started to look for offloading parts of my photo library to external devices. Since then I have used both in parallel. Let us start with what these programs say about themselves: Shotwell is declared as a Photo Manager for Gnome 3, with the following features: digiKam says about itself that it is an advanced digital photo management application for Linux, Windows, and Mac-OSX. It has a very long feature page with a short list at the beginning: Now that sounds like they are very similar, but upon using them it turns out that there are huge differences, that can easily be summed up in a short statement:
Shotwell is Gnome 3 that means get rid of functionality. digiKam is KDE that means provide as much functionality as possible.
Now before you run after me with a knife because you do not agree with me on the above, either read on, or stop reading. I am not interested in flame wars over Gnome versus KDE philosophy. I have been using Gnome since many years, and tried to convince myself to G3 for more than a year until I threw out all of it but selected programs but their number is going down. Let us look at those aspects I am using: organization, offline, sharing, editing. Organization In Shotwell, your photos are organized into events, independent from their location on disk. These events can have title and comment and collect a set of related photos. In my case I often have photos from two or more cameras (my camera and mobile, photos of friends), which I keep in separate directories within a main directory for the event. For example I have a folder 2016/05.21-22.Climbing.Tanigakadake with two sub-folders Norbert (for my photos) and Friend (for my friends photos). In Shotwell all the photos are in the same event, which is shown with title 05.21-22 Tanigakawadake Climbing within the 2016 year and May month.
shotwell-events So in short Shotwell distinguishes between disk layout and album/event names. In digiKam there is a strict connection between disk layout and album names 1:1. Albums are directories. One can adjust the viewer to show all photos of sub-albums in the main album, and by this one can achieve the same effect of merging all photos of my friend and myself. The good thing in this approach is that one can easily have sub-albums: Imagine a trip to three different islands of Hawaii during one trip. This is something easy to achieve in digiKam, but hard in Shotwell.
digikam-albums Other organization methods Both Shotwell and digiKam support tags, including hierarchical tags and rating (0-5 stars). Shotwell has in addition a quick flag action that I used quite often for initial selecting photos, as well as accepted and rejected. digiKam also has so called picks (no pick, reject, pending, accepted), and colors (not used by now). Both programs have some face detection support, but also this I haven t used. So in the organization respect there is no clear winner. I like the Event idea of Shotwell, or better, the separation of events from the disk structure. But on the other hand, Shotwell does not allow for sub-albums, which is also a pain. No clear winner draw Offline storage That is simple: Shotwell: forget it, not reasonably possible. One can move parts to an external HD, then unplug it, and Shotwell will tell you that all the photos are missings. And when you plug the external HD in, it will redetect them. But this is not proper support, just a consequence of hash sum storage. Also separation into several libraries (online and offline) is not supported. On the other hand, digiKam supports multiple libraries, partly offline, without a hinch. I would love to have this feature in Shotwell, because I need to free disk space, urgently!!! Clear winner: digiKam Sharing Again here my testing is very restricted I am using my own Piwigo installation exclusively. Here Shotwell is excellent in providing support for various features: upload to existing category, create new category, resize, optionally remove tags, add comments to albums and photos, etc. (partly implemented by me
shotwell-piwigo On the other hand digiKam has a very barebone interface to Piwigo you can upload photos to an existing album and resize them, but that is already everything. One also needs to say that the list of supported services in digiKam is by far longer than the one in Shotwell, but the main services (usual suspects) are supported in both.
digikam-piwigo Clear winner: Shotwell (but I haven t tested other upload services). Editing The editing capabilities of Shotwell are again very restricted: red eyes, resize, levels, .
shotwell-editing digiKam here is more like a photo editor software with loads of tools and features. I haven t even explored all the options maybe I can get rid of GIMP, too?
digikam-editing Clear winner: digiKam Conclusions While I am still working with both, I actually would love to move completely to digiKam. I simply cannot stand the Gnome 3 philosophy of reducing functionality to a minimal for dummy users. There is a market for that, sure, but I am not one of it. Unfortunately, the missing Event support, and much more the completely minimal support for Piwigo sharing in digiKam is a big show stopper at the moment. Even after testing the upcoming digiKam 5.0 version for some time, I didn t see any improvement wrt to sharing. That leaves me with the only option to continue working with both programs, and hope to get a new bigger SSD before the current one runs out of space. Of course I could start hacking into the digiKam source maybe I do this when I have a bit more time to add proper Piwigo support.

24 May 2016

Thorsten Alteholz: Debian and the Internet of Things

Everybody is talking about the Internet of Things. Unfortunately there is no sign of it in Debian yet. Besides some smaller packages like sispmctl, usbrelay or the 1-wire support in digitemp and owfs, there is not much software to control devices over a network. With the recent upload of alljoyn-core-1504 this might change. The Alljoyn Framework, where the Alljoyn Core is just one of several modules, lets devices and applications detect each other and communicate with one another over a D-Bus like message bus. The development of the framework has been started by Qualcomm some years ago and is meanwhile managed by the AllSeen Alliance, a nonprofit consortium. The software is licensed under the ISC license. This first upload is just the first step of a long journey. Other modules that compose the framework and already have a released tarball are related to lightning products, gateways to overcome the boundaries of the local network and much more. In the near future it is also planned to have modules that attach Z-Wave-, ZigBee- or Bluetooth-devices to the Alljoyn bus. So all in all, this looks like an exciting task and everybody is invited to help maintaining the software in Debian.

Alberto Garc a: I/O bursts with QEMU 2.6

QEMU 2.6 was released a few days ago. One new feature that I have been working on is the new way to configure I/O limits in disk drives to allow bursts and increase the responsiveness of the virtual machine. In this post I ll try to explain how it works. The basic settings First I will summarize the basic settings that were already available in earlier versions of QEMU. Two aspects of the disk I/O can be limited: the number of bytes per second and the number of operations per second (IOPS). For each one of them the user can set a global limit or separate limits for read and write operations. This gives us a total of six different parameters. I/O limits can be set using the throttling.* parameters of -drive, or using the QMP block_set_io_throttle command. These are the names of the parameters for both cases:
-drive block_set_io_throttle
throttling.iops-total iops
throttling.iops-read iops_rd
throttling.iops-write iops_wr
throttling.bps-total bps
throttling.bps-read bps_rd
throttling.bps-write bps_wr
It is possible to set limits for both IOPS and bps at the same time, and for each case we can decide whether to have separate read and write limits or not, but if iops-total is set then neither iops-read nor iops-write can be set. The same applies to bps-total and bps-read/write. The default value of these parameters is 0, and it means unlimited. In its most basic usage, the user can add a drive to QEMU with a limit of, say, 100 IOPS with the following -drive line: 
-drive file=hd0.qcow2,throttling.iops-total=100
We can do the same using QMP. In this case all these parameters are mandatory, so we must set to 0 the ones that we don t want to limit: 
     "execute": "block_set_io_throttle",
     "arguments":  
        "device": "virtio0",
        "iops": 100,
        "iops_rd": 0,
        "iops_wr": 0,
        "bps": 0,
        "bps_rd": 0,
        "bps_wr": 0
I/O bursts While the settings that we have just seen are enough to prevent the virtual machine from performing too much I/O, it can be useful to allow the user to exceed those limits occasionally. This way we can have a more responsive VM that is able to cope better with peaks of activity while keeping the average limits lower the rest of the time. Starting from QEMU 2.6, it is possible to allow the user to do bursts of I/O for a configurable amount of time. A burst is an amount of I/O that can exceed the basic limit, and there are two parameters that control them: their length and the maximum amount of I/O they allow. These two can be configured separately for each one of the six basic parameters described in the previous section, but here we ll use iops-total as an example. The I/O limit during bursts is set using iops-total-max , and the maximum length (in seconds) is set with iops-total-max-length . So if we want to configure a drive with a basic limit of 100 IOPS and allow bursts of 2000 IOPS for 60 seconds, we would do it like this (the line is split for clarity): 
   -drive file=hd0.qcow2,
          throttling.iops-total=100,
          throttling.iops-total-max=2000,
          throttling.iops-total-max-length=60
Or with QMP: 
     "execute": "block_set_io_throttle",
     "arguments":  
        "device": "virtio0",
        "iops": 100,
        "iops_rd": 0,
        "iops_wr": 0,
        "bps": 0,
        "bps_rd": 0,
        "bps_wr": 0,
        "iops_max": 2000,
        "iops_max_length": 60,
With this, the user can perform I/O on hd0.qcow2 at a rate of 2000 IOPS for 1 minute before it s throttled down to 100 IOPS. The user will be able to do bursts again if there s a sufficiently long period of time with unused I/O (see below for details). The default value for iops-total-max is 0 and it means that bursts are not allowed. iops-total-max-length can only be set if iops-total-max is set as well, and its default value is 1 second. Controlling the size of I/O operations When applying IOPS limits all I/O operations are treated equally regardless of their size. This means that the user can take advantage of this in order to circumvent the limits and submit one huge I/O request instead of several smaller ones. QEMU provides a setting called throttling.iops-size to prevent this from happening. This setting specifies the size (in bytes) of an I/O request for accounting purposes. Larger requests will be counted proportionally to this size. For example, if iops-size is set to 4096 then an 8KB request will be counted as two, and a 6KB request will be counted as one and a half. This only applies to requests larger than iops-size: smaller requests will be always counted as one, no matter their size. The default value of iops-size is 0 and it means that the size of the requests is never taken into account when applying IOPS limits. Applying I/O limits to groups of disks In all the examples so far we have seen how to apply limits to the I/O performed on individual drives, but QEMU allows grouping drives so they all share the same limits. This feature is available since QEMU 2.4. Please refer to the post I wrote when it was published for more details. The Leaky Bucket algorithm I/O limits in QEMU are implemented using the leaky bucket algorithm (specifically the Leaky bucket as a meter variant). This algorithm uses the analogy of a bucket that leaks water constantly. The water that gets into the bucket represents the I/O that has been performed, and no more I/O is allowed once the bucket is full. To see the way this corresponds to the throttling parameters in QEMU, consider the following values: 
  iops-total=100
  iops-total-max=2000
  iops-total-max-length=60
bucket The bucket is initially empty, therefore water can be added until it s full at a rate of 2000 IOPS (the burst rate). Once the bucket is full we can only add as much water as it leaks, therefore the I/O rate is reduced to 100 IOPS. If we add less water than it leaks then the bucket will start to empty, allowing for bursts again. Note that since water is leaking from the bucket even during bursts, it will take a bit more than 60 seconds at 2000 IOPS to fill it up. After those 60 seconds the bucket will have leaked 60 x 100 = 6000, allowing for 3 more seconds of I/O at 2000 IOPS. Also, due to the way the algorithm works, longer burst can be done at a lower I/O rate, e.g. 1000 IOPS during 120 seconds. Acknowledgments As usual, my work in QEMU is sponsored by Outscale and has been made possible by Igalia and the help of the QEMU development team. igalia-outscale Enjoy QEMU 2.6!

22 May 2016

Reproducible builds folks: Reproducible builds: week 56 in Stretch cycle

What happened in the Reproducible Builds effort between May 15th and May 21st 2016: Media coverage Blog posts from our GSoC and Outreachy contributors: Documentation update Ximin Luo clarified instructions on how to set SOURCE_DATE_EPOCH. Toolchain fixes Other upstream fixes Packages fixed The following 18 packages have become reproducible due to changes in their build dependencies: abiword angband apt-listbugs asn1c bacula-doc bittornado cdbackup fenix gap-autpgrp gerbv jboss-logging-tools invokebinder modplugtools objenesis pmw r-cran-rniftilib x-loader zsnes The following packages have become reproducible after being fixed: Some uploads have fixed some reproducibility issues, but not all of them: Patches submitted that have not made their way to the archive yet: Reproducibility-related bugs filed: Package reviews 51 reviews have been added, 19 have been updated and 15 have been removed in this week. 22 FTBFS bugs have been reported by Chris Lamb, Santiago Vila, Niko Tyni and Daniel Schepler. tests.reproducible-builds.org Misc. This week's edition was written by Reiner Herrmann and Holger Levsen and reviewed by a bunch of Reproducible builds folks on IRC.

19 May 2016

Antoine Beaupr : My free software activities, May 2016

Debian Long Term Support (LTS) This is my 6th month working on Debian LTS, started by Raphael Hertzog at Freexian. This is my largest month so far, for which I had requested 20 hours of work.

Xen work I spent the largest amount of time working on the Xen packages. We had to re-roll the patches because it turned out we originally just imported the package from Ubuntu as-is. This was a mistake because that package forked off the Debian packaging a while ago and included regressions in the packaging itself, not just security fixes. So I went ahead and rerolled the whole patchset and tested it on Koumbit's test server. Brian May then completed the uploaded, which included about 40 new patches, mostly from Ubuntu.

Frontdesk duties Next up was the frontdesk duties I had taken this week. This was mostly uneventful, although I had forgotten how to do some of the work and thus ended up doing extensive work on the contributor's documentation. This is especially important since new contributors joined the team! I also did a lot of Debian documentation work in my non-sponsored work below. The triage work involved chasing around missing DLAs, triaging away OpenJDK-6 (for which, let me remind you, security support has ended in LTS), raised the question of Mediawiki maintenance.

Other LTS work I also did a bunch of smaller stuff. Of importance, I can note that I uploaded two advisories that were pending from April: NSS and phpMyAdmin. I also reviewed the patches for the ICU update, since I built the one for squeeze (but didn't have time to upload before squeeze hit end-of-life). I have tried to contribute to the NTP security support but that was way too confusing to me, and I have left it to the package maintainer which seemed to be on top of things, even if things mean complete chaos and confusion in the world of NTP. I somehow thought that situation had improved with the recent investments in ntpsec and ntimed, but unfortunately Debian has not switched to the ntpsec codebase, so it seems that the NTP efforts have diverged in three different projects instead of closing into a single, better codebase.

Future LTS work This is likely to be my last month of work on LTS until September. I will try to contribute a few hours in June, but July and August will be very busy for me outside of Debian, so it's unlikely that I contribute much to the project during the summer. My backlog included those packages which might be of interest to other LTS contributors:
  • libxml2: no upstream fix, but needs fixing!
  • tiff ,3 : same mess
  • libgd2: maintainer contacted
  • samba regression: mailed bug #821811 to try to revive the effort
  • policykit-1: to be investigated
  • p7zip: same

Other free software work

Debian documentation I wrote an detailed short guide to Debian package development, something I felt was missing from the existing corpus, which seems to be too focus in covering all alternatives. My guide is opinionated: I believe there is a right and wrong way of doing things, or at least, there are best practices, especially when just patching packages. I ended up retroactively publishing that as a blog post - now I can simply tag an item with blog and it shows up in the blog. (Of course, because of a mis-configuration on my side, I have suffered from long delays publishing to Debian planet, so all the posts dates are off in the Planet RSS feed. This will hopefully be resolved around the time this post is published, but this allowed me to get more familiar with the Planet Venus software, as detailed in that other article.) Apart from the guide, I have also done extensive research to collate information that allowed me to create workflow graphs of the various Debian repositories, which I have published in the Debian Release section of the Debian wiki. Here is the graph: It helps me understand how packages flow between different suites and who uploads what where. This emerged after I realized I didn't really understand how "proposed updates" worked. Since we are looking at implementing a similar process for the security queue, I figured it was useful to show what changes would happen, graphically. I have also published a graph that describes the relations between different software that make up the Debian archive. The idea behind this is also to provide an overview of what happens when you upload a package in the Debian archive, but it is more aimed at Debian developers trying to figure out why things are not working as expected. The graphs were done with Graphviz, which allowed me to link to various components in the graph easily, which is neat. I also prefered Graphviz over Dia or other tools because it is easier to version and I don't have to bother (too much) about the layout and tweaking the looks. The downside is, of course, that when Graphviz makes the wrong decision, it's actually pretty hard to make it do the right thing, but there are various workarounds that I have found that made the graphs look pretty good. The source is of course available in git but I feel all this documentation (including the guide) should go in a more official document somewhere. I couldn't quite figure out where. Advice on this would be of course welcome.

Ikiwiki I have made yet another plugin for Ikiwiki, called irker, which enables wikis to send notifications to IRC channels, thanks to the simple irker bot. I had trouble with Irker in the past, since it was not quite reliable: it would disappear from channels and not return when we'd send it a notification. Unfortunately, the alternative, the KGB bot is much heavier: each repository needs a server-side, centralized configuration to operate properly. Irker's design is simpler and more adapted to a simple plugin like this. Let's hope it will work reliably enough for my needs. I have also suggested improvements to the footnotes styles, since they looked like hell in my Debian guide. It turns out this was an issue with the multimarkdown plugin that doesn't use proper semantic markup to identify footnotes. The proper fix is to enable footnotes in the default Discount plugin, which will require another, separate patch. Finally, I have done some improvements (I hope!) on the layout of this theme. I made the top header much lighter and transparent to work around an issue where followed anchors would be hidden under the top header. I have also removed the top menu made out of the sidebar plugin because it was cluttering the display too much. Those links are all on the frontpage anyways and I suspect people were not using them so much. The code is, as before, available in this git repository although you may want to start from the new ikistrap theme that is based on Bootstrap 4 and that may eventually be merged in ikiwiki directly.

DNS diagnostics Through this interesting overview of various *ping tools, I got found out about the dnsdiag tool which currently allows users to do DNS traces, tampering detection and ping over DNS. In the hope of packaging it into Debian, I have requested clarifications regarding a modification to the DNSpython library the tool uses. But I went even further and boldly opened a discussion about replacing DNSstuff, the venerable DNS diagnostic tools that is now commercial. It is somewhat surprising that there is no software that has even been publicly released that does those sanity checks for DNS, given how old DNS is. Incidentally, I have also requested smtpping to be packaged in Debian as well but httping is already packaged.

Link checking In the process of writing this article, I suddenly remembered that I constantly make mistakes in the various links I post on my site. So I started looking at a link checker, another tool that should be well established but that, surprisingly, is not quite there yet. I have found this neat software written in Python called LinkChecker. Unfortunately, it is basically broken in Debian, so I had to do a non-maintainer upload to fix that old bug. I managed to force myself to not take over maintainership of this orphaned package but I may end up doing just that if no one steps up the next time I find issues in the package. One of the problems I had checking links in my blog is that I constantly refer to sites that are hostile to bots, like the Debian bugtracker and MoinMoin wikis. So I published a patch that adds a --no-robots flag to be able to crawl those sites effectively. I know there is the W3C tool but it's written in Perl, and there's probably zero chance for me to convince those guys to bypass robots exclusion rules, so I am sticking to Linkchecker.

Other Debian packaging work At my request, Drush has finally been removed from Debian. Hopefully someone else will pick up that work, but since it basically needs to be redone from scratch, there was no sense in keeping it in the next release of Debian. Similarly, Semanticscuttle was removed from Debian as well. I have uploaded new versions of tuptime, sopel and smokeping. I have also file a Request For Help for Smokeping. I am happy to report there was a quick response and people will be stepping up to help with the maintenance of that venerable monitoring software.

Background radiation Finally, here's the generic background noise of me running around like a chicken with his head cut off: Finally, I should mention that I will be less active in the coming months, as I will be heading outside as the summer finally came! I somewhat feel uncomfortable documenting publicly my summer here, as I am more protective of my privacy than I was before on this blog. But we'll see how it goes, maybe you'll hear non-technical articles here again soon!

Next.

Previous.