Search Results: "tar"

7 May 2016

Elena 'valhalla' Grandi: Pyra preorders

Pyra preorders

If you've met me at a conference you may have noticed that instead of a laptop I was using a handeld which looks like a laptop scaled down to nintendo DS size, the https://en.wikipedia.org/wiki/Openpandora.

I've used it as my main computing device while travelling for a few years, even for work (as a programmer)so happily that when EvilDragon announced at FOSDEM (link points to youtube video) https://www.youtube.com/watch?v=4T-w1KqrVsM that he was working on a successor device I started saving money for it even before I knew many details about the specs, other that they would have been way better than the Pandora ones (which is getting painful to use a browser on, because of its 256MB RAM).

Immagine/fotohttp://social.gl-como.it/photos/valhalla/image/5a9fdfadf33e40e9e3517b18ca7cba68

Now this successor device is almost ready, they have opened the preorders https://www.dragonbox.de/en/45-pyra, and they have already reached the absolute minimum number of orders for mass production and are almost there for a more reasonable number of 1000 devices, so if you want a chance to get one of the first batch devices now it's time to visit their store.

A few highlights, from my point of view, include:

* It will run Debian with just a custom kernel/bootloader (and a few configuration only packages): most of the kernel mods are being submitted upstream, so maybe one day there won't even be a need for this kernel (but e.g. with Pandora upstream didn't accept the custom way they managed the keyboard; on the Pyra the keyboard is managed in a more standard way, but there may be other similar issues).

* It has been designed with modularity in mind: the CPU board is socketed on the main board and in the future upgrades may require just replacing the CPU board. I haven't read the details on the actual licensing, but it seems that the hardware design will be open enough that 3rd party boards may also be a possibility.

* just like on Pandora: real keyboard. hardware analog volume wheel. Huge user-replaceable battery (I don't think that there are any independent reviews of the pyra battery yet, but the one on the Pandora is still able to go through a day of FOSDEM i.e. alternating often between on with wifi and suspendend and only go down to 50% or so charge). Stylus (and 3d-printed quill) friendly touchscreen. Long term support from the producer.

* The 4G version has been designed in such a way that the GSM modem can be actually turned off (just like on the https://neo900.org/)

There are of course a few bad parts:

* PowerVR. The good news is that there is a risk that no 3d drivers will be available at all, and this means that the Pyra has been tested and considered good enough with just (FOSS) software acceleration.

* The price: yes, it is expensive. I'm happy I've saved money in advance for it, otherwise I wouldn't have been able to afford it. Some of it is a problem of small production, some is actual product quality. If you consider that it can take the place of both a laptop (and small ones are getting quite expensive, now that netbooks have disappeared) and a smartphone (I don't do lots of voice calls) it will start going down from "oh so **** high" to "high, but not unreasonably so"

Disclaimer: I have preordered one, so I am interested in the success of the project because it will mean better software and better support for the device.

Edit: forgot the link to the press kit the images comes from http://cloud.openpandora.org/index.php/s/a0Q0TXV8gh5NLAC?path=%2F, which also includes more infos on specs etc.

Craig Small: Displaying Linux Memory

Memory management is hard, but RAM management may be even harder. Most people know the vague overall concept of how memory usage is displayed within Linux. You have your total memory which is everything inside the box; then there is used and free which is what the system is or is not using respectively. Some people might know that not all used is used and some of it actually is free. It can be very confusing to understand, even for a someone who maintains procps (the package that contains top and free, two programs that display memory usage). So, how does the memory display work? What free shows The free program is part of the procps package. It s central goal is to give a quick overview of how much memory is used where. A typical output (e.g. what I saw when I typed free -h ) could look like this: 
      total   used    free   shared  buff/cache  available
Mem:    15G   3.7G    641M     222M         11G        11G
Swap:   15G   194M     15G
I ve used the -h option for human-readable output here for the sake of brevity and because I hate typing long lists of long numbers. People who have good memories (or old computers) may notice there is a missing -/+ buffers/cache line. This was intentionally removed in mid-2014 because as the memory management of Linux got more and more complicated, these lines became less relevant. These used to help with the not used used memory problem mentioned in the introduction but progress caught up with it. To explain what free is showing, you need to understand some of the underlying statistics that it works with. This isn t a lesson on how Linux its memory (the honest short answer is, I don t fully know) but just enough hopefully to understand what free is doing. Let s start with the two simple columns first; total and free. Total Memory This is what memory you have available to Linux. It is almost, but not quite, the amount of memory you put into a physical host or the amount of memory you allocate for a virtual one. Some memory you just can t have; either due to early reservations or devices shadowing the memory area. Unless you start mucking around with those settings or the virtual host, this number stays the same. Free Memory Memory that nobody at all is using. They haven t reserved it, haven t stashed it away for future use or even just, you know, actually using it. People often obsess about this statistic but its probably the most useless one to use for anything directly. I have even considered removing this column, or replacing it with available (see later what that is) because of the confusion this statistic causes. The reason for its uselessness is that Linux has memory management where it allocates memory it doesn t use. This decrements the free counter but it is not truly used . If you application needs that memory, it can be given back. A very important statistic to know for running a system is how much memory have I got left before I either run out or I start to serious swap stuff to swap drives. Despite its name, this statistic will not tell you that and will probably mislead you. My advice is unless you really understand the Linux memory statistics, ignore this one. Who s Using What Now we come to the components that are using (if that is the right word) the memory within a system. Shared Memory Shared memory is often thought of only in the context of processes (and makes working out how much memory a process uses tricky but that s another story) but the kernel has this as well. The shared column lists this, which is a direct report from the Shmem field in the meminfo file. Slabs For things used a lot within the kernel, it is inefficient to keep going to get small bits of memory here and there all the time. The kernel has this concept of slabs where it creates small caches for objects or in-kernel data strucutures that slabinfo(5) states [such as] buffer heads, inodes and dentries . So basically kernel stuff for the kernel to do kernelly things with. Slab memory comes in two flavours. There is reclaimable and unreclaimable. This is important because unreclaimable cannot be handed back if your system starts to run out of memory. Funny enough, not all reclaimable is, well, reclaimable. A good estimate is you ll only get 50% back, top and free ignore this inconvenient truth and assume it can be 100%. All of the reclaimable slab memory is considered part of the Cached statistic. Unreclaimable is memory that is part of Used. Page Cache and Cached Page caches are used to read and write to storage, such as a disk drive. These are the things that get written out when you use sync and make the second read of the same file much faster. An interesting quirk is that tmpfs is part of the page cache. So the Cached column may increase if you have a few of these. The Cached column may seem like it should only have Page Cache, but the Reclaimable part of the Slab is added to this value. For some older versions of some programs, they will have no or all Slab counted in Cached. Both of these versions are incorrect. Cached makes up part of the buff/cache column with the standard options for free or has a column to itself for the wide option. Buffers The second component to the buff/cache column (or separate with the wide option) is kernel buffers. These are the low-level I/O buffers inside the kernel. Generally they are small compared to the other components and can basically ignored or just considered part of the Cached, which is the default for free. Used Unlike most of the previous statistics that are either directly pulled out of the meminfo file or have some simple addition, the Used column is calculated and completely dependent on the other values. As such it is not telling the whole story here but it is reasonably OK estimate of used memory. Used component is what you have left of your Total memory once you have removed: Notice that the unreclaimable part of slab is not in this calculation, which means it is part of the used memory. Also note this seems a bit of a hack because as the memory management gets more complicated, the estimates used become less and less real. Available In early 2014, the kernel developers took pity on us toolset developers and gave us a much cleaner, simpler way to work out some of these values (or at least I d like to think that s why they did it). The available statistic is the right way to work out how much memory you have left. The commit message explains the gory details about it, but the great thing is that if they change their mind or add some new memory feature the available value should be changed as well. We don t have to worry about should all of slab be in Cached and are they part of Used or not, we have just a number directly out of meminfo. What does this mean for free? Poor old free is now at least 24 years old and it is based upon BSD and SunOS predecessors that go back way before then. People expect that their system tools don t change by default and show the same thing over and over. On the other side, Linux memory management has changed dramatically over those years. Maybe we re all just sheep (see I had to mention sheep or RAMs somewhere in this) and like things to remain the same always. Probably if free was written now; it would only need the total, available and used columns with used merely being total minus available. Possibly with some other columns for the wide option. The code itself (found in libprocps) is not very hard to maintain so its not like this change will same some time but for me I m unsure if free is giving the right and useful result for people that use it.

Keith Packard: Altos1.6.3

AltOS 1.6.3 Bdale and I are pleased to announce the release of AltOS version 1.6.3. AltOS is the core of the software for all of the Altus Metrum products. It consists of firmware for our cc1111, STM32L151, STMF042, LPC11U14 and ATtiny85 based electronics and Java-based ground station software. Version 1.6.3 adds idle mode to AltosDroid and has bug fixes for our host software on desktops, laptops an android devices along with BlueTooth support for Windows. 1.6.3 is in Beta test for Android; if you want to use the beta version, join the AltosDroid beta program AltOS AltOS fixes: AltosUI and TeleGPS Applications AltosUI and TeleGPS New Features: AltosUI and TeleGPS Fixes: AltosDroid AltosDroid new features: AltosDroid bug fixes: Documentation

6 May 2016

Norbert Preining: Yubikey NEO

Two Factor authentication and general improvement of my security infrastructure was long on my todo list. Some month ago I finally purchased a Yubikey NEO from Yubico and try to consistently use it as second factor, as well as gpg signing/encrypting device. yubikey-neo I am trying to get the best out of my Yubikey NEO by using as many of its functionality, in particular: Smartcard for my GNuPG keys, OTP similar to Google Authenticator and similar, as well as challenge-response for additional login security, as well as all that over NFC to not keep keys/passwords on my mobile phone. While there are loads of guides (see the previous article on GnuPG for some of them), many of them are out-of-date for current distributions and GnuPG etc. So I tried to collect all I could find not the least to have a place to look it up in case I forget it again. The Hardware The Yubikey NEO is a great peace of hardware. I not even remotely understand how they manage that this little beast can do all these things and still work out without mixing things up. As far as I understand (please correct me) it has three independent circuits of communication: On top of these circuit of communication there is a variety of applications to make the most out of your Yubikey: Yubikey mode setup There are several modes, and using the ykpersonalize tool (readily available for Windows, Mac, Linux, and in the Debian package yubikey-personalization) one can program the key to work in a variety of modes. I chose to activate all options by passing in -m86 which stand for OTP/U2F/CCID composite device with MODE_FLAG_EJECT. 
$ ykpersonalize -m86
Firmware version 3.4.3 Touch level 1792 Unconfigured
 
The USB mode will be set to: 0x86
 
Commit? (y/n) [n]: y
$
It is a good idea to unplug and replug the key after this operation. Yubikey udev rules for user access To allow users but root to use the Yubikey, additional udev rules are necessary: 
SUBSYSTEMS=="usb", ATTRS idVendor =="1050", ATTRS idProduct =="0116", TAG+="uaccess"
which I put into /etc/udev/rules.d/99-yubikeys.rules on Debian. After that another unplug and replug should allow normal user to access the key. This can be checked by calling getfacl on the newly created /dev/hidraw? device. Using the HID/Challenge-Response mode (slot 2) If you want to secure your login with an additional second factor, there are several options documented on the Yubico site concerning yubico-pam. Since I cannot be sure to be always online with my laptop, I choose Challenge-Response authentication, and followed one-to-one Yubico s docs Local Authentication Using Challenge Response. Basically it boils down to install libpam-yubico, select mode-challenge-response when asked for configuration. Then one needs to personalizing the key (in particular slot 2) for challenge response with: 
$ ykpersonalize -2 -ochal-resp -ochal-hmac -ohmac-lt64 -oserial-api-visible
...
Commit? (y/n) [n]: y
$
Next we need to save the challenge and expected response to the user s directory: 
$ mkdir $HOME/.yubico
$ ykpamcfg -2 -v
...
Stored initial challenge and expected response in '/home/norbert/.yubico/challenge-123456'.
$
It might be a good idea to try this out, and if it works, activate it also for root. But be careful no key no login Challenge: I am currently searching for a method to replace the second factor of they key optionally with a different authentication method, like a very difficult passphrase. This way I could log in even without my key, but in this case would need the complicated passphrase. From my reading of the pam manuals it seems to be possible, and I am planning to use pam_ssh and a specific login key with a complicated passphrase. I will report back when this is done. YubiOATH (TOTP) Time based One Time Passwords (aka Google Authenticator style) Without any setup whatsoever this worked out of the box. I use the Yubico Authenticator on my Android phone, and the dedicated application for the Linux desktop to create second factors for all kind of applications. Currently I am using it with Google login, Github, DropbBox, and WordPress (via the Two Factor plugin which can also be tweaked to use the NEO key as USB key via the FIDO U2F). Challenge: If I start the Yubico Personalization GUI, I see two free slots so where are the TOTPs computed? That also means that I have one slot free and for now I don t know what to do with it Yubikey OpenGPG applet setup The Yubikeys support OpenPGP, and the applet is pre-installed (afaik), meaning you can directly configure the key and upload your keys. Here I use gpg2 (2.1) as it seems to better support card operations. To not interfere with the current gpg setup I use a temporary gpg home: 
$ mkdir gpgtmp
$ chmod go-rwx gpgtmp
$ gpg2 --homedir gpgtmp --list-keys
gpg: keybox 'gpgtmp/pubring.kbx' created
gpg: gpgtmp/trustdb.gpg: trustdb created
Warning: The YubiKey NEO only supports 2048bit keys. If you want 4096bit keys you need to use one of the newer YubiKey 4, which gives you this option, but does not have support for NFC, and thus no way to interact with an Android (or other) mobile phone. Check the correct version of the applet There has been a bug in an older version of the applet, but since 2 years all keys sold should have a correct applet. You can check by: 
$ gpg-connect-agent --homedir gpgtmp --hex "scd apdu 00 f1 00 00" /bye"
gpg-connect-agent: no running gpg-agent - starting '/usr/bin/gpg-agent'
gpg-connect-agent: waiting for the agent to come up ... (5s)
gpg-connect-agent: connection to agent established
D[0000]  01 00 10 90 00                                     .....           
OK
Looking at the output one sees D[0000] 01 00 10 which means applet version 1.0.10, which is the first version fixed. Replace pins of the key The standard pins are 123456 for the user pin, and 12345678 for the admin pin. These need immediate change! Warning: When changing the ping the normal pin must be 6 (at least?) digits, and the admin pin 8 (at least?), other gpg2 cannot use the key anymore. No idea why. 
$ gpg2 --homedir gpgtmp --card-edit
 
Reader ...........: 1050:0116:X:0
Application ID ...: D2760001240102000006036457190000
Version ..........: 2.0
Manufacturer .....: Yubico
Serial number ....: 03645719
Name of cardholder: [not set]
Language prefs ...: [not set]
Sex ..............: unspecified
URL of public key : [not set]
Login data .......: [not set]
Signature PIN ....: forced
Key attributes ...: rsa2048 rsa2048 rsa2048
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 3 3
Signature counter : 0
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]
 
gpg/card> admin
Admin commands are allowed
 
gpg/card> passwd
gpg: OpenPGP card no. D2760001240102000006036457190000 detected
 
1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit
 
Your selection? 3
PIN changed.
 
1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit
 
Your selection? 1
PIN changed.
 
1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit
 
Your selection? q
 
gpg/card> quit
After this you need to use the new pins for all changes. Setup basic data The key can also save some basic data about yourself, like name, sex, language preferences, login name, and url to obtain the public key. As before start gpg2 and then change these infos in the following way> 
gpg/card> name
Cardholder's surname: Preining
Cardholder's given name: Norbert
 
gpg/card> sex
Sex ((M)ale, (F)emale or space): M
 
gpg/card> lang
Language preferences: de
 
gpg/card> login
Login data (account name): norbert
 
gpg/card> url
URL to retrieve public key: https://www.preining.info/preining-norbert.asc
 
gpg/card> list
 
Reader ...........: 1050:0116:X:0
Application ID ...: D2760001240102000006036457190000
Version ..........: 2.0
Manufacturer .....: Yubico
Serial number ....: 03645719
Name of cardholder: Norbert Preining
Language prefs ...: de
Sex ..............: male
URL of public key : https://www.preining.info/preining-norbert.asc
Login data .......: norbert
Signature PIN ....: forced
Key attributes ...: rsa2048 rsa2048 rsa2048
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 3 3
Signature counter : 0
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]
 
gpg/card> quit
Move sub keys to Yubikey As laid out in the article on GnuPG subkeys, we are having three subkeys for signing, encryption, and authentication. In reality I will practically only use the signing key, but upload all three keys to the card. In the following I expect that you have a setup more or less similar to the one described in the article linked before. Again, we use GnuPG v2, mostly because it was the version that worked out of the box. In addition, if you are setting up a similar stage like in my GNuPG article with gpg1 keys on the mail server, then you don t want the gpg1 keys being removed. Basically you must have the Yubikey plugged in and call keytocard after selecting each key in turn (and deselecting it afterwards). Warning: There is another bug in the GnuPG applet that was fixed in later versions (but not in 1.0.10), namely that not all keys are accepted. This is a bit a pain. I needed to recreate a subkey to obtain a key that can be loaded onto the Yubikey. Unfortunately, Yubico has also stopped/disabled the ability to update applets (although I have to say their documentation is an incredible rubbish with respect to applets and upgrades ). As before, assume that $MASTERKEY contains the hex id of your master key. 
$ gpg2 --edit-key $MASTERKEY
gpg (GnuPG) 2.1.11; Copyright (C) 2016 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
 
Secret key is available.
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 2
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb* rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> keytocard
Please select where to store the key:
   (1) Signature key
   (3) Authentication key
Your selection? 1
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb* rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 2
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 3
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb* rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> keytocard
Please select where to store the key:
   (2) Encryption key
Your selection? 2
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb* rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 3
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 4
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb* rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> keytocard
Please select where to store the key:
   (3) Authentication key
Your selection? 3
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb* rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 4
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> save
After that your keys are on the Yubikey (and only there!), and GNuPG will require the PIN (user pin) to sign/encrypt documents. Usage Many things have been said above, but to sum up when and how I am using the YubiKey now: Conclusions With this setup I am now quite content, but not completely. What I still want to do is full disk encryption where I need the Yubikey to boot and again, with an alternative for a very long passphrase. At the end, adding a second factor to the login is not really optimal, and only protects you against quick hacks. If the laptop is actually stolen, only full disc protection helps. Access to the hardware always guarantees that one has access to everything on the disc. Another thing I want to do is re-use the GnuPG key on the Yubikey as ssh key for logging into remote systems. That would mean that I get rid of even more keys on my laptop. But this is still in the work The other open question is what to use the other available slot of the Yubikey for? I thought about some passwords (possible), but I don t feel to happy about having my password issued with the press of a key. But all in all, I like the setup much more than before and not having any GnuPG key on the laptop is a big plus.

Matthias Klumpp: Adventures in D programming

I recently wrote a bigger project in the D programming language, the appstream-generator (asgen). Since I rarely leave the C/C++/Python realm, and came to like many aspects of D, I thought blogging about my experience could be useful for people considering to use D. Disclaimer: I am not an expert on programming language design, and this is not universally valid criticism of D just my personal opinion from building one project with it. Why choose D in the first place? The previous AppStream generator was written in Python, which wasn t ideal for the task for multiple reasons, most notably multiprocessing and LMDB not working well together (and in general, multiprocessing being terrible to work with) and the need to reimplement some already existing C code in Python again. So, I wanted a compiled language which would work well together with the existing C code in libappstream. Using C was an option, but my least favourite one (writing this in C would have been much more cumbersome). I looked at Go and Rust and wrote some small programs performing basic operations that I needed for asgen, to get a feeling for the language. Interfacing C code with Go was relatively hard since libappstream is a GObject-based C library, I expected to be able to auto-generate Go bindings from the GIR, but there were only few outdated projects available which did that. Rust on the other hand required the most time in learning it, and since I only briefly looked into it, I still can t write Rust code without having the coding reference open. I started to implement the same examples in D just for fun, as I didn t plan to use D (I was aiming at Go back then), but the language looked interesting. The D language had the huge advantage of being very familiar to me as a C/C++ programmer, while also having a rich standard library, which included great stuff like std.concurrency.Generator, std.parallelism, etc. Translating Python code into D was incredibly easy, additionally a gir-d-generator which is actively maintained exists (I created a small fork anyway, to be able to directly link against the libappstream library, instead of dynamically loading it). What is great about D? This list is just a huge braindump of things I had on my mind at the time of writing Interfacing with C There are multiple things which make D awesome, for example interfacing with C code and to a limited degree with C++ code is really easy. Also, working with functions from C in D feels natural. Take these C functions imported into D:
extern(C):
nothrow:
struct _mystruct  
alias mystruct_p = _mystruct*;
mystruct_p = mystruct_create ();
mystruct_load_file (mystruct_p my, const(char) *filename);
mystruct_free (mystruct_p my);
You can call them from D code in two ways:
auto test = mystruct_create ();
// treating "test" as function parameter
mystruct_load_file (test, "/tmp/example");
// treating the function as member of "test"
test.mystruct_load_file ("/tmp/example");
test.mystruct_free ();
This allows writing logically sane code, in case the C functions can really be considered member functions of the struct they are acting on. This property of the language is a general concept, so a function which takes a string as first parameter, can also be called like a member function of string. Writing D bindings to existing C code is also really simple, and can even be automatized using tools like dstep. Since D can also easily export C functions, calling D code from C is also possible. Getting rid of C++ cruft There are many things which are bad in C++, some of which are inherited from C. D kills pretty much all of the stuff I found annoying. Some cool stuff from D is now in C++ as well, which makes this point a bit less strong, but it s still valid. E.g. getting rid of the #include preprocessor dance by using symbolic import statements makes sense, and there have IMHO been huge improvements over C++ when it comes to metaprogramming. Incredibly powerful metaprogramming Getting into detail about that would take way too long, but the metaprogramming abilities of D must be mentioned. You can do pretty much anything at compiletime, for example compiling regular expressions to make them run faster at runtime, or mixing in additional code from string constants. The template system is also very well thought out, and never caused me headaches as much as C++ sometimes manages to do. Built-in unit-test support Unittesting with D is really easy: You just add one or more unittest blocks to your code, in which you write your tests. When running the tests, the D compiler will collect the unittest blocks and build a test application out of them. The unittest scope is useful, because you can keep the actual code and the tests close together, and it encourages writing tests and keep them up-to-date. Additionally, D has built-in support for contract programming, which helps to further reduce bugs by validating input/output. Safe D While D gives you the whole power of a low-level system programming language, it also allows you to write safer code and have the compiler check for that, while still being able to use unsafe functions when needed. Unfortunately, @safe is not the default for functions though. Separate operators for addition and concatenation D exclusively uses the + operator for addition, while the ~ operator is used for concatenation. This is likely a personal quirk, but I love it very much that this distinction exists. It s nice for things like addition of two vectors vs. concatenation of vectors, and makes the whole language much more precise in its meaning. Optional garbage collector D has an optional garbage collector. Developing in D without GC is currently a bit cumbersome, but these issues are being addressed. If you can live with a GC though, having it active makes programming much easier. Built-in documentation generator This is almost granted for most new languages, but still something I want to mention: Ddoc is a standard tool to generate code documentation for D code, with a defined syntax for describing function parameters, classes, etc. It will even take the contents of a unittest scope to generate automatic examples for the usage of a function, which is pretty cool. Scope blocks The scope statement allows one to execute a bit of code before the function exists, when it failed or was successful. This is incredibly useful when working with C code, where a free statement needs to be issued when the function is exited, or some arbitrary cleanup needs to be performed on error. Yes, we do have smart pointers in C++ and with some GCC/Clang extensions a similar feature in C too. But the scopes concept in D is much more powerful. See Scope Guard Statement for details. Built-in syntax for parallel programming Working with threads is so much more fun in D compared to C! I recommend taking a look at the parallelism chapter of the Programming in D book. Pure functions D allows to mark functions as purely-functional, which allows the compiler to do optimizations on them, e.g. cache their return value. See pure-functions. D is fast! D matches the speed of C++ in almost all occasions, so you won t lose performance when writing D code that is, unless you have the GC run often in a threaded environment. Very active and friendly community The D community is very active and friendly so far I only had good experience, and I basically came into the community asking some tough questions regarding distro-integration and ABI stability of D. The D community is very enthusiastic about pushing D and especially the metaprogramming features of D to its limits, and consists of very knowledgeable people. Most discussion happens at the forums/newsgroups at forum.dlang.org. What is bad about D? Half-proprietary reference compiler This is probably the biggest issue. Not because the proprietary compiler is bad per se, but because of the implications this has for the D ecosystem. For the reference D compiler, Digital Mars D (DMD), only the frontend is distributed under a free license (Boost), while the backend is proprietary. The FLOSS frontend is what the free compilers, LLVM D Compiler (LDC) and GNU D Compiler (GDC) are based on. But since DMD is the reference compiler, most features land there first, and the Phobos standard library and druntime is tuned to work with DMD first. Since major Linux distributions can t ship with DMD, and the free compilers GDC and LDC lack behind DMD in terms of language, runtime and standard-library compatibility, this creates a split world of code that compiles with LDC, GDC or DMD, but never with all D compilers due to it relying on features not yet in e.g. GDCs Phobos. Especially for Linux distributions, there is no way to say use this compiler to get the best and latest D compatibility . Additionally, if people can t simply apt install latest-d, they are less likely to try the language. This is probably mainly an issue on Linux, but since Linux is the place where web applications are usually written and people are likely to try out new languages, it s really bad that the proprietary reference compiler is hurting D adoption in that way. That being said, I want to make clear DMD is a great compiler, which is very fast and build efficient code. I only criticise the fact that it is the language reference compiler. UPDATE: To clarify the half-proprietary nature of the compiler, let me quote the D FAQ:
The front end for the dmd D compiler is open source. The back end for dmd is licensed from Symantec, and is not compatible with open-source licenses such as the GPL. Nonetheless, the complete source comes with the compiler, and all development takes place publically on github. Compilers using the DMD front end and the GCC and LLVM open source backends are also available. The runtime library is completely open source using the Boost License 1.0. The gdc and ldc D compilers are completely open sourced.
Phobos (standard library) is deprecating features too quickly This basically goes hand in hand with the compiler issue mentioned above. Each D compiler ships its own version of Phobos, which it was tested against. For GDC, which I used to compile my code due to LDC having bugs at that time, this means that it is shipping with a very outdated copy of Phobos. Due to the rapid evolution of Phobos, this meant that the documentation of Phobos and the actual code I was working with were not always in sync, leading to many frustrating experiences. Furthermore, Phobos is sometimes removing deprecated bits about a year after they have been deprecated. Together with the older-Phobos situation, you might find yourself in a place where a feature was dropped, but the cool replacement is not yet available. Or you are unable to import some 3rd-party code because it uses some deprecated-and-removed feature internally. Or you are unable to use other code, because it was developed with a D compiler shipping with a newer Phobos. This is really annoying, and probably the biggest source of unhappiness I had while working with D especially the documentation not matching the actual code is a bad experience for someone new to the language. Incomplete free compilers with varying degrees of maturity LDC and GDC have bugs, and for someone new to the language it s not clear which one to choose. Both LDC and GDC have their own issues at time, but they are rapidly getting better, and I only encountered some actual compiler bugs in LDC (GDC worked fine, but with an incredibly out-of-date Phobos). All issues are fixed meanwhile, but this was a frustrating experience. Some clear advice or explanation which of the free compilers is to prefer when you are new to D would be neat. For GDC in particular, being developed outside of the main GCC project is likely a problem, because distributors need to manually add it to their GCC packaging, instead of having it readily available. I assume this is due to the DRuntime/Phobos not being subjected to the FSF CLA, but I can t actually say anything substantial about this issue. Debian adds GDC to its GCC packaging, but e.g. Fedora does not do that. No ABI compatibility D has a defined ABI too bad that in reality, the compilers are not interoperable. A binary compiled with GDC can t call a library compiled with LDC or DMD. GDC actually doesn t even support building shared libraries yet. For distributions, this is quite terrible, because it means that there must be one default D compiler, without any exception, and that users also need to use that specific compiler to link against distribution-provided D libraries. The different runtimes per compiler complicate that problem further. The D package manager, dub, does not yet play well with distro packaging This is an issue that is important to me, since I want my software to be easily packageable by Linux distributions. The issues causing packaging to be hard are reported as dub issue #838 and issue #839, with quite positive feedback so far, so this might soon be solved. The GC is sometimes an issue The garbage collector in D is quite dated (according to their own docs) and is currently being reworked. While working with asgen, which is a program creating a large amount of interconnected data structures in a threaded environment, I realized that the GC is significantly slowing down the application when threads are used (it also seems to use UNIX signals SIGUSR1 and SIGUSR2 to stop/resume threads, which I still find odd). Also, the GC performed poorly on memory pressure, which did get asgen killed by the OOM killer on some more memory-constrained machines. Triggering a manual collection run after a large amount of these interconnected data structures wasn t needed anymore solved this problem for most systems, but it would of course have been better to not needing to give the GC any hints. The stop-the-world behavior isn t a problem for asgen, but it might be for other applications. These issues are at time being worked on, with a GSoC project laying the foundation for further GC improvements. version is a reserved word Okay, that is admittedly a very tiny nitpick, but when developing an app which works with packages and versions, it s slightly annoying. The version keyword is used for conditional compilation, and needing to abbreviate it to ver in all parts of the code sucks a little (e.g. the Package interface can t have a property version , but now has ver instead). The ecosystem is not (yet) mature In general it can be said that the D ecosystem, while existing for almost 9 years, is not yet that mature. There are various quirks you have to deal with when working with D code on Linux. It s always nothing major, usually you can easily solve these issues and go on, but it s annoying to have these papercuts. This is not something which can be resolved by D itself, this point will solve itself as more people start to use D and D support in Linux distributions gets more polished. Conclusion I like to work with D, and I consider it to be a great language the quirks it has in its toolchain are not that bad to prevent writing great things with it. At time, if I am not writing a shared library or something which uses much existing C++ code, I would prefer D for that task. If a garbage collector is a problem (e.g. for some real-time applications, or when the target architecture can t run a GC), I would not recommend to use D. Rust seems to be the much better choice then. In any case, D s flat learning curve (for C/C++ people) paired with the smart choices taken in language design, the powerful metaprogramming, the rich standard library and helpful community makes it great to try out and to develop software for scenarios where you would otherwise choose C++ or Java. Quite honestly, I think D could be a great language for tasks where you would usually choose Python, Java or C++, and I am seriously considering to replace quite some Python code with D code. For very low-level stuff, C is IMHO still the better choice. As always, choosing the right programming language is only 50% technical aspects, and 50% personal taste UPDATE: To get some idea of D, check out the D tour on the new website tour.dlang.org.

Russ Allbery: Review: The Language of Power

Review: The Language of Power, by Rosemary Kirstein
Series: Steerswomen #4
Publisher: Rosemary Kirstein
Copyright: 2004, 2014
Printing: April 2014
ISBN: 0-9913546-3-X
Format: Kindle
Pages: 400
This is the fourth book in the Steerswomen series and definitely not the place to start. It's also a difficult series to review without spoilers, so I won't be able to provide too many details about the plot. I will say that this is a reunion and a return of sorts to themes from earlier in the series, rather than a direct follow-up to the revelations at the end of The Lost Steersman. Rowan is back in the Inner Lands, continuing to investigate the affairs of wizards. In particular, she's digging into the past of the town of Donner, following up on the report of an earlier steerswoman and investigating a now-dead wizard who seemed to act far different from a typical wizard. And Bel is back at her side again, watching her back. The first half of The Language of Power goes over somewhat familiar ground. Similar to both The Steerswoman and The Lost Steersman, Rowan is poking around in a city, getting to know unfamiliar people, being a steerswoman, and winning people over with her unique charm. But that's a theme I don't mind seeing repeated, since Rowan is one of my favorite protagonists from any series I've read. She's both ethical and respectful in a way that doesn't feel artificial or constructed. She thinks oddly and dives into sudden fascinations, and she does rely on the conventions for interacting with steerswomen, but the more time one spends with her, the better one likes her. This is true of both the reader and the town inhabitants, and Kirstein is brilliant at writing the gradual getting-to-know and growing-respect process. Events in The Language of Power slowly build up to another confrontation with wizards, and this one is full of major revelations about the world. Some of the ambiguity of earlier books is firmly resolved, we find out a lot more about how wizards view themselves and their abilities, and tons of new questions are raised. It's not a conclusion in any way, which is a bit unfortunate given that the next book is still being written (twelve years later, although thankfully it's being actively worked on as I write this). But we get the first clear look at the substratum of the world that Kirstein is building in this series. This sounds satisfying, and to some extent it is, but any regular SFF reader will have guessed at many of the revelations here. I was pretty sure the world was following one of two possible patterns partway through the second book, certain which it was during the third book, and was nodding right along with the revelations in this book. I'm trying to avoid spoilers, but if you read a lot of SFF, chances are you've read about something akin to this background before. That takes a bit of the thrill out of the revelations, unfortunately. What adds the excitement and thrill back in are Rowan's reactions. It's very difficult to write a character who comes from an entirely different perspective than either the author or the reader, and Kirstein does an amazing job. Not perfect, quite, at least for me: there were a few points where I thought Rowan was more baffled or more upset than it felt like she should have been. But they are few and far between, and it's quite possible my expectations are the ones that wouldn't ring true if written into the story. It's just such a delight to see Rowan analyzing the world, incorporating new revelations into her growing world model, and figuring out how to take the most moral action at any point. I would happily read another dozen books of this (but I wish they were all already written). If you've read the previous three books, definitely pick up this one as well. For me, it narrowly misses being the best book of the series (I think that's still The Outskirter's Secret because of the difficulty of the perspective change Kirstein pulls off), but it's a close competition. And the final reveal at the very end of this book points to upcoming adventures that I can hardly wait to read. Rating: 8 out of 10

5 May 2016

Sean Whitton: dh_make_elpa & dh_elpa_test

I recently completed and released some work on Debian s tooling for packaging Emacs Lisp addons to GNU Emacs. Emacs grew a package manager called package.el a few years ago, and last year David Bremner wrote the dh_elpa tool to simplify packaging addons for Debian by leveraging package.el features. Packaging a series of addons for Debian left me with a wishlist of features for dh_elpa and I was recently able to implement them. Debian tooling generally uses Perl, a language I didn t know before starting on this project. I was fortunate enough to receive a free review copy of Perl 5 by Example when I attended a meeting of the Bay Area Linux Users Group while I was visiting San Francisco a few months ago. I accepted the book with the intent of doing this work. dh_make_elpa dh_make_elpa (at present available from Debian experimental) is a Perl script to convert a git repository cloned from the upstream of an Emacs Lisp addon to a rudimentary Debian package. It performs a lot of guesswork, and its simple heuristics are something I hope to improve on. Since I am new to object-oriented program design in Perl and I wanted to leverage object-oriented Debian tooling library code, I took the structure of my project from dh_make_perl. In this manner I found it easy and pleasant to write a maintainable script. dh_elpa_test A lot of Emacs Lisp addon packages use a program called Cask to manage the Emacs Lisp dependencies needed to run their test suites. That meant that dh_auto_test often fails to run Emacs Lisp addon package test suites. Since the Debian packaging toolchain already has advanced dependency management, it s undesirable to involve Cask in the package build pipeline if it can be avoided. I had been copying and pasting the code needed to make the tests run in our environment to the debian/rules files of each package whose test suite I wanted to run. dh_elpa_test tries to detect Emacs Lisp addon package test suites and run them with the workarounds needed in our environment. This avoids boilerplate in debian/rules. dh_elpa_test also disables dh_auto_test to avoid a inadvertent Cask invocation. Future & acknowledgements My hope for this work was to make it easier and faster to package Emacs Lisp addon packages for Debian, for my own sake and for anyone new who is interested in joining the pkg-emacsen team. In the future, I want to have dh_elpa_test generate an autopkgtest definition so that a Testsuite: pkg-emacsen line in debian/control is enough to have an Emacs Lisp addon package test suite run on Debian CI. I m very grateful to David Bremner for reviewing and supporting this work, and also for supporting my Emacs Lisp addon packaging work more generally.

4 May 2016

Petter Reinholdtsen: The Pyra - handheld computer with Debian preinstalled

A friend of mine made me aware of The Pyra, a handheld computer which will be delivered with Debian preinstalled. I would love to get one of those for my birthday. :) The machine is a complete ARM-based PC with micro HDMI, SATA, USB plugs and many others connectors, and include a full keyboard and a 5" LCD touch screen. The 6000mAh battery is claimed to provide a whole day of battery life time, but I have not seen any independent tests confirming this. The vendor is still collecting preorders, and the last I heard last night was that 22 more orders were needed before production started. As far as I know, this is the first handheld preinstalled with Debian. Please let me know if you know of any others. Is it the first computer being sold with Debian preinstalled?

3 May 2016

Carl Chenet: Feed2tweet, your RSS feed to Twitter Python self-hosted app

Feed2tweet is a self-hosted Python app to send you RSS feed to Twitter. Feed2tweet is in production for Le Journal du hacker, a French Hacker News-style FOSS website and LinuxJobs.fr, the job board of the French-speaking FOSS community. linuxjobs-horizontale Feed2tweet 0.3 now only runs with Python 3. It also fixes a nasty bug with RSS feeds modifying the RSS entry orders. Have a look at the Feed2tweet 0.3 changelog: Using Feed2tweet? Send us bug reports/feature requests/push requests/comments about it!

Carl Chenet: Feed2tweet, your RSS feed to Twitter Python self-hosted app

Feed2tweet is a self-hosted Python app to send you RSS feed to Twitter. Feed2tweet is in production for Le Journal du hacker, a French Hacker News-style FOSS website and LinuxJobs.fr, the job board of the French-speaking FOSS community. linuxjobs-horizontale Feed2tweet 0.3 now only runs with Python 3. It also fixes a nasty bug with RSS feeds modifying the RSS entry orders. Have a look at the Feed2tweet 0.3 changelog: Using Feed2tweet? Send us bug reports/feature requests/push requests/comments about it!

Jamie McClelland: Monitoring Deflect

May First/People Link has several members that are targets of politically motivated denial of service attacks (mostly groups that support reproductive justice for women and palestinian rights). To fight off the attacks, we work closely with Deflect - a non-governmental organization based in Canada that fights against this kind of censorship. When a site is down, it's not always easy to understand why. Deflect runs as many as 5 edge servers, any of them could be down. And, of course, the origin server could also be down. I tried using a commericial/free as in beer service for monitoring up time, but when it reported the site being down, I had no idea which part was down. So... httping to the rescue. Unfortunately, it depends on --divert-connect which is only available in Debian Stretch. I run the script via a cron job and output the results to a log file. 
    #!/bin/bash
    # Test all given edges 
    domain="$1"
    origin="$2"
    proto=http
    if [ -n "$3" ]; then
        proto="$3"
    fi
    if [ -z "$domain" ]; then
        printf "Please pass the domain as first argument.\n"
        exit 1
    fi
    if ! ping -c 1 4.2.2.1 >/dev/null; then
        # printf "We are offline. Not running.\n"
        exit 1
    fi
    ips=$(dig +short "$domain")
    if [ "$?" -ne "0" ]; then
        # printf "DNS lookup failure. Not running.\n"
        exit 1
    fi
    if [ -n "$origin" ]; then
        ips="$ips $origin"
    fi
    l=
    if [ "$proto" = "https" ]; then
        l=-l
    fi
    for ip in $ips; do
        date=$(date +%Y.%m.%d-%H:%M)
        for i in 1 2 3; do
            out=$(httping $l -m -t 5 -c 1 --divert-connect "$ip" "$proto://$domain")
            [ -z "$out" ] && out=1
            printf "%s %s %s\n" "$date" "$ip" "$out"
        done
    done

Rapha&#235;l Hertzog: My Free Software Activities in April 2016

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS I handled a new LTS sponsor that wanted to see wheezy keep supporting armel and armhf. This was not part of our initial plans (set during last Debconf) and I thus mailed all teams that were impacted if we were to collectively decide that it was OK to support those architectures. While I was hoping to get a clear answer rather quickly, it turns out that we never managed to get an answer to the question from all parties. Instead the discussion drifted on the more general topic of how we handle sponsorship/funding in the LTS project. Fortunately, the buildd maintainers said they were OK with this and the ftpmasters had no objections, and they both implicitly enacted the decision: Ansgar Burchardt kept the armel/armhf architectures in the wheezy/updates suite when he handled the switch to the LTS team, and Aur lien Jarno also configured wanna-build to keep building armel/armhf for the suite. The DSA team did not confirm that this change was not interfering with one of their plans to decommission some hardware. Build daemons are a shared resource anyway and a single server is likely to handle builds for multiple releases. DebConf 16 This month I registered for DebConf 16 and submitted multiple talk/BoF proposals: I want to share the setup we use in Kali as it can be useful for other derivatives and also for Debian itself to help smooth the relationship with derivatives. I also want to open again the debate on the usage of money within Debian. It s a hard topic but we should really strive to take some official position on what s possible and what s not possible. With Debian LTS and its sponsorship we have seen that we can use money to some extent without hurting the Debian project as a whole. Can this be transposed to other teams or projects? What are the limits? Can we define a framework and clear rules? I expect the discussion to be very interesting in the BoF. Mehdi Dogguy has agreed to handle this BoF with me. Packaging Django. I uploaded 1.8.12 to jessie-backports and 1.9.5 to unstable. I filed two upstream bugs (26473 and 26474) for two problems spotted by lintian. Unfortunately, when I wanted to upload it to unstable, the test suite did not ran. I pinned this down to a sqlite regression. Chris Lamb filed #820225 and I contacted the SQLite and Django upstream developers by email to point them to this issue. I helped the SQLite upstream author (Richard Hipp) to reproduce the issue and he was quick to provide a patch which landed in 3.12.1. Later in the month I made another upload to fix an upgrade bug (#821789). GNOME 3.20. As for each new version, I updated gnome-shell-timer to ensure it works with the new GNOME. This time I spent a bit more time to fix a regression (805347) that dates back to a while and that would never be fixed otherwise since the upstream author orphaned this extension (as he no longer uses GNOME). I have also been bitten by display problems where accented characters would be displayed below the character that follows. With the help of members of the GNOME team, we found out that this was a problem specific to the cantarell font and was only triggered with Harfbuzz 1.2. This is tracked in Debian with #822682 on harfbuzz and #822762 in fonts-cantarell. There s a new upstream release (with the fix) ready to be packaged but unfortunately it is blocked by the lack of a recent fontforge in Debian. I thus mailed debian-mentors in the hope to find volunteers to help the pkg-fonts team to package a newer version Misc Debian/Kali work Distro Tracker. I started to mentor Vladimir Likic who contacted me because he wants to contribute to Distro Tracker. I helped him to setup his development environment and we fixed a few issues in the process. Bug reports. I filed many bug reports, most of them due to my work on Kali: I also investigated #819958 that was affecting testing since it has been reported to Kali as well. And I made an NMU of dh-make-golang to fix #819472 that I reported earlier. Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Russ Allbery: Review: The Effective Engineer

Review: The Effective Engineer, by Edmond Lau
Publisher: Effective Bookshelf
Copyright: 2015
ISBN: 0-9961281-0-7
Format: Trade paperback
Pages: 222
Silicon Valley start-up tech companies have a standard way of thinking about work. Large chunks of this come from Google, which pioneered a wide variety of new, or at least not-yet-mainstream, ways of organizing and thinking about work. The rest accreted through experience with fast-paced start-ups, engineer-focused companies, web delivery of products, and rabid turnover and high job mobility within a hothouse of fairly similar companies. A key part of this mindset is the firm belief that this atmosphere has created a better way to work, at least for software engineers (and systems administrators, although heaven forbid that one call them that any more): more effective, more efficient, more focused on what really matters. I think this is at least partly true, at least from the perspective of a software engineer. This Silicon Valley work structure focuses on data gathering, data-based decision-making, introspection, analysis, and continuous improvement, all of which I think are defensibly pointed in the right direction (if rarely as rigorous as one might want to believe). It absorbs bits and pieces of work organization techniques that are almost certainly improvements for the type of work software engineers do: Agile, Lean, continuous deployment, and fast iteration times. In other cases, though, I'm less convinced that this Silicon Valley consensus is objectively better as opposed to simply different; interviewing, for instance, is a puzzle that I don't think anyone has figured out, and the remarkable consensus in Silicon Valley on how to interview (basically, "like Google except for the bits we thought were obnoxious") feels more like a social fad than a sign of getting it right. But every industry has its culture of good ideas, bad ideas, fads, and fashion, and it's quite valuable to know that culture if you want to work in that industry. The Effective Engineer is a self-published book by Edmund Lau, a Silicon Valley software engineer who also drifted (as is so common in Silicon Valley) into mentoring, organizing, and speaking to other software engineers. Its purpose, per the subtitle, is to tell you "how to leverage your efforts in software engineering to make a disproportionate and meaningful impact." While that's not exactly wrong, and the book contains some useful and valuable tips, I'd tend to give it a slightly different subtitle: "a primer on how a Silicon Valley software engineer is expected to think about their work." This is a bit more practical, a bit less confident, and a bit less convinced of its own correctness than Lau might want to present his work, but it's just as valuable of a purpose if you want to work in the industry. (And is a bit more honest about its applicability outside of that industry.) What this book does extremely well is present, in a condensed, straightforward, and fast-moving form, most of the highlights of how start-ups and web-scale companies approach software engineering and the SWE role in companies (SWE, meaning software engineer, is another bit of Google terminology that's now nearly universal). If you've already worked in or around this industry for a while, you've probably picked up a lot of this via osmosis: prioritize based on impact and be unapologetic about letting other things drop, have a growth mindset, reprioritize regularly, increase your iteration speed, measure everything constantly, check your assumptions against data, derisk your estimates, use code review and automated testing (but not too much), automate operations, and invest heavily in hiring and onboarding. (The preceding list is a chapter list for this book.) If you're working at one of these sorts of companies, you're probably currently somewhere between nodding and rolling your eyes because no one at work will shut up about these topics. But if you've not worked inside one of these companies, even if you've done software engineering elsewhere, this is a great book to read to prepare yourself. You're going to hear about these ideas constantly, and, if it achieves nothing else at all, The Effective Engineer will give you a firm enough grounding in the lingo and mindset that you can have intelligent conversations with people who assume this is the only way to think about software engineering. By this point, you might be detecting a certain cynicism in this review. It's not entirely fair: a lot of these ideas are clearly good ones, and Lau does a good job of describing them quickly and coherently. It's a good job for what it is. But there are a couple of things that limited its appeal for me. First, it's definitely a primer. I read it after having worked at a web-scale start-up for a year and a half. There wasn't much in it that seemed particularly new, and it's somewhat superficial. The whole middle section in particular (build tools for yourself, measure everything, be data-driven) are topics for which the devil is often in the details. Lau gives you the terminology and the expected benefits, but putting any one of these techniques into practice could be a book (or several) by itself. Don't expect to come away from The Effective Engineer with much of a concrete plan for how to do these things in your day-to-day software development projects. But it's a good reminder to be thinking about, say, how to embed metrics and data-gathering hooks into the software you write. This is the nature of a primer; no 222-page book can get into much depth about the fractal complexity of doing good, fast, scalable software development. Second, there's a fundamental question raised by a book like this: effective at what? Lau tackles that in the first chapter with his focus on impact and leverage, and it's good advice as far as it goes. (Regular readers of my book reviews know that I love this sort of time management and prioritization discussion.) But measuring impact is a hard problem that requires a prioritization framework, and this is not really the book for this. The Effective Engineer is written primarily for software developers at start-ups, leaves the whole venture-capital start-up process as unquestioned background material, and accepts without comment the standard measures of value in that world: fast-deployed products, hypergrowth, racing competitors for perceived innovation, and finding ways to extract money. That's as deep into the question of impact as Lau gets: increases in company revenue. There's nothing wrong with this for the kind of book Lau intended to write, and it's not his fault that I find it unsatisfying. But don't expect The Effective Engineer to ask any hard questions about whether that's a meaningful definition of impact, or to talk much about less objective goals: quality of implementation, craftsmanship, giving back to a broader community via free software contributions, impact on the world in ways that can't be measured in market share, or anything else that is unlikely to lead to objective impact for company profits. At best he leaves a bit of wiggle room around using the concept of impact with different goals. If you're a new graduate who wants to work at Silicon-Valley-style start-ups, this is a great orientation, and likewise if you're coming from a different area of software development into that world. If you're not working in that industry, The Effective Engineer may still be moderately interesting, but it's not written for that audience and has little or nothing to say of the challenges of other types of businesses. But if you've already worked in the industry for a while, or if you're more interested in deeper discussions of goals and subjective values, you may not get much out of this. Rating: 7 out of 10

2 May 2016

Reproducible builds folks: Reproducible builds: week 53 in Stretch cycle

What happened in the Reproducible Builds effort between April 24th and 30th 2016. Media coverage Reproducible builds were mentioned explicitly in two talks at the Mini-DebConf in Vienna: Aspiration together with the OTF CommunityLab released their report about the Reproducible Builds summit in December 2015 in Athens. Toolchain fixes Now that the GCC development window has been opened again, the SOURCE_DATE_EPOCH patch by Dhole and Matthias Klose to address the issue timestamps_from_cpp_macros (__DATE__ / __TIME__) has been applied upstream and will be released with GCC 7. Following that Matthias Klose also has uploaded gcc-5/5.3.1-17 and gcc-6/6.1.1-1 to unstable with a backport of that SOURCE_DATE_EPOCH patch. Emmanuel Bourg uploaded maven/3.3.9-4, which uses SOURCE_DATE_EPOCH for the maven.build.timestamp. (SOURCE_DATE_EPOCH specification) Other upstream changes Alexis Bienven e submitted a patch to Sphinx which extends SOURCE_DATE_EPOCH support for copyright years in generated documentation. Packages fixed The following 12 packages have become reproducible due to changes in their build dependencies: hhvm jcsp libfann libflexdock-java libjcommon-java libswingx1-java mobile-atlas-creator not-yet-commons-ssl plexus-utils squareness svnclientadapter The following packages have became reproducible after being fixed: Some uploads have fixed some reproducibility issues, but not all of them: Patches submitted that have not made their way to the archive yet: Package reviews 95 reviews have been added, 15 have been updated and 129 have been removed in this week. 22 FTBFS bugs have been reported by Chris Lamb and Martin Michlmayr. diffoscope development strip-nondeterminism development tests.reproducible-builds.org Misc. Amongst the 29 interns who will work on Debian through GSoC and Outreachy there are four who will be contributing to Reproducible Builds for Debian and Free Software. We are very glad to welcome ceridwen, Satyam Zode, Scarlett Clark and Valerie Young and look forward to working together with them the coming months (and maybe beyond)! This week's edition was written by Reiner Herrmann and Holger Levsen and reviewed by a bunch of Reproducible builds folks on IRC.

Vincent Bernat: Pragmatic Debian packaging

While the creation of Debian packages is abundantly documented, most tutorials are targeted to packages implementing the Debian policy. Moreover, Debian packaging has a reputation of being unnecessarily difficult1 and many people prefer to use less constrained tools2 like fpm or CheckInstall. However, I would like to show how building Debian packages with the official tools can become straightforward if you bend some rules:
  1. No source package will be generated. Packages will be built directly from a checkout of a VCS repository.
  2. Additional dependencies can be downloaded during build. Packaging individually each dependency is a painstaking work, notably when you have to deal with some fast-paced ecosystems like Java, Javascript and Go.
  3. The produced packages may bundle dependencies. This is likely to raise some concerns about security and long-term maintenance, but this is a common trade-off in many ecosystems, notably Java, Javascript and Go.

Pragmatic packages 101 In the Debian archive, you have two kinds of packages: the source packages and the binary packages. Each binary package is built from a source package. You need a name for each package. As stated in the introduction, we won t generate a source package but we will work with its unpacked form which is any source tree containing a debian/ directory. In our examples, we will start with a source tree containing only a debian/ directory but you are free to include this debian/ directory into an existing project. As an example, we will package memcached, a distributed memory cache. There are four files to create:
  • debian/compat,
  • debian/changelog,
  • debian/control, and
  • debian/rules.
The first one is easy. Just put 9 in it: 
echo 9 > debian/compat
The second one has the following content: 
memcached (0-0) UNRELEASED; urgency=medium
  * Fake entry
 -- Happy Packager <happy@example.com>  Tue, 19 Apr 2016 22:27:05 +0200
The only important information is the name of the source package, memcached, on the first line. Everything else can be left as is as it won t influence the generated binary packages.

The control file debian/control describes the metadata of both the source package and the generated binary packages. We have to write a block for each of them. 
Source: memcached
Maintainer: Vincent Bernat <bernat@debian.org>
Package: memcached
Architecture: any
Description: high-performance memory object caching system
The source package is called memcached. We have to use the same name as in debian/changelog. We generate only one binary package: memcached. In the remaining of the example, when you see memcached, this is the name of a binary package. The Architecture field should be set to either any or all. Use all exclusively if the package contains only arch-independent files. In doubt, just stick to any. The Description field contains a short description of the binary package.

The build recipe The last mandatory file is debian/rules. It s the recipe of the package. We need to retrieve memcached, build it and install its file tree in debian/memcached/. It looks like this: 
#!/usr/bin/make -f
DISTRIBUTION = $(shell lsb_release -sr)
VERSION = 1.4.25
PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0
TARBALL = memcached-$(VERSION).tar.gz
URL = http://www.memcached.org/files/$(TARBALL)
%:
    dh $@
override_dh_auto_clean:
override_dh_auto_test:
override_dh_auto_build:
override_dh_auto_install:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)
    ./configure --prefix=/usr
    make
    make install DESTDIR=debian/memcached
override_dh_gencontrol:
    dh_gencontrol -- -v$(PACKAGEVERSION)
The empty targets override_dh_auto_clean, override_dh_auto_test and override_dh_auto_build keep debhelper from being too smart. The override_dh_gencontrol target sets the package version3 without updating debian/changelog. If you ignore the slight boilerplate, the recipe is quite similar to what you would have done with fpm: 
DISTRIBUTION=$(lsb_release -sr)
VERSION=1.4.25
PACKAGEVERSION=$ VERSION -0~$ DISTRIBUTION 0
TARBALL=memcached-$ VERSION .tar.gz
URL=http://www.memcached.org/files/$ TARBALL 
wget -N --progress=dot:mega $ URL 
tar --strip-components=1 -xf $ TARBALL 
./configure --prefix=/usr
make
make install DESTDIR=/tmp/installdir
# Build the final package
fpm -s dir -t deb \
    -n memcached \
    -v $ PACKAGEVERSION  \
    -C /tmp/installdir \
    --description "high-performance memory object caching system"
You can review the whole package tree on GitHub and build it with dpkg-buildpackage -us -uc -b.

Pragmatic packages 102 At this point, we can iterate and add several improvements to our memcached package. None of those are mandatory but they are usually worth the additional effort.

Build dependencies Our initial build recipe only work when several packages are installed, like wget and libevent-dev. They are not present on all Debian systems. You can easily express that you need them by adding a Build-Depends section for the source package in debian/control: 
Source: memcached
Build-Depends: debhelper (>= 9),
               wget, ca-certificates, lsb-release,
               libevent-dev
Always specify the debhelper (>= 9) dependency as we heavily rely on it. We don t require make or a C compiler because it is assumed that the build-essential meta-package is installed and it pulls those. dpkg-buildpackage will complain if the dependencies are not met. If you want to install those packages from your CI system, you can use the following command4: 
mk-build-deps \
    -t 'apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -qqy' \
    -i -r debian/control
You may also want to investigate pbuilder or sbuild, two tools to build Debian packages in a clean isolated environment.

Runtime dependencies If the resulting package is installed on a freshly installed machine, it won t work because it will be missing libevent, a required library for memcached. You can express the dependencies needed by each binary package by adding a Depends field. Moreover, for dynamic libraries, you can automatically get the right dependencies by using some substitution variables: 
Package: memcached
Depends: $ misc:Depends , $ shlibs:Depends
The resulting package will contain the following information: 
$ dpkg -I ../memcached_1.4.25-0\~unstable0_amd64.deb   grep Depends
 Depends: libc6 (>= 2.17), libevent-2.0-5 (>= 2.0.10-stable)

Integration with init system Most packaged daemons come with some integration with the init system. This integration ensures the daemon will be started on boot and restarted on upgrade. For Debian-based distributions, there are several init systems available. The most prominent ones are:
  • System-V init is the historical init system. More modern inits are able to reuse scripts written for this init, so this is a safe common denominator for packaged daemons.
  • Upstart is the less-historical init system for Ubuntu (used in Ubuntu 14.10 and previous releases).
  • systemd is the default init system for Debian since Jessie and for Ubuntu since 15.04.
Writing a correct script for the System-V init is error-prone. Therefore, I usually prefer to provide a native configuration file for the default init system of the targeted distribution (Upstart and systemd).

System-V If you want to provide a System-V init script, have a look at /etc/init.d/skeleton on the most ancient distribution you want to target and adapt it5. Put the result in debian/memcached.init. It will be installed at the right place, invoked on install, upgrade and removal. On Debian-based systems, many init scripts allow user customizations by providing a /etc/default/memcached file. You can ship one by putting its content in debian/memcached.default.

Upstart Providing an Upstart job is similar: put it in debian/memcached.upstart. For example: 
description "memcached daemon"
start on runlevel [2345]
stop on runlevel [!2345]
respawn
respawn limit 5 60
expect daemon
script
  . /etc/default/memcached
  exec memcached -d -u $USER -p $PORT -m $CACHESIZE -c $MAXCONN $OPTIONS
end script
When writing an Upstart job, the most important directive is expect. Be sure to get it right. Here, we use expect daemon and memcached is started with the -d flag.

systemd Providing a systemd unit is a bit more complex. The content of the file should go in debian/memcached.service. For example: 
[Unit]
Description=memcached daemon
After=network.target
[Service]
Type=forking
EnvironmentFile=/etc/default/memcached
ExecStart=/usr/bin/memcached -d -u $USER -p $PORT -m $CACHESIZE -c $MAXCONN $OPTIONS
Restart=on-failure
[Install]
WantedBy=multi-user.target
We reuse /etc/default/memcached even if it is not considered a good practice with systemd6. Like for Upstart, the directive Type is quite important. We used forking as memcached is started with the -d flag. You also need to add a build-dependency to dh-systemd in debian/control: 
Source: memcached
Build-Depends: debhelper (>= 9),
               wget, ca-certificates, lsb-release,
               libevent-dev,
               dh-systemd
And you need to modify the default rule in debian/rules: 
%:
    dh $@ --with systemd
The extra complexity is a bit unfortunate but systemd integration is not part of debhelper7. Without those additional modifications, the unit will get installed but you won t get a proper integration and the service won t be enabled on install or boot.

Dedicated user Many daemons don t need to run as root and it is a good practice to ship a dedicated user. In the case of memcached, we can provide a _memcached user8. Add a debian/memcached.postinst file with the following content: 
#!/bin/sh
set -e
case "$1" in
    configure)
        adduser --system --disabled-password --disabled-login --home /var/empty \
                --no-create-home --quiet --force-badname --group _memcached
        ;;
esac
#DEBHELPER#
exit 0
There is no cleanup of the user when the package is removed for two reasons:
  1. Less stuff to write.
  2. The user could still own some files.
The utility adduser will do the right thing whatever the requested user already exists or not. You need to add it as a dependency in debian/control: 
Package: memcached
Depends: $ misc:Depends , $ shlibs:Depends , adduser
The #DEBHELPER# marker is important as it will be replaced by some code to handle the service configuration files (or some other stuff). You can review the whole package tree on GitHub and build it with dpkg-buildpackage -us -uc -b.

Pragmatic packages 103 It is possible to leverage debhelper to reduce the recipe size and to make it more declarative. This section is quite optional and it requires understanding a bit more how a Debian package is built. Feel free to skip it.

The big picture There are four steps to build a regular Debian package:
  1. debian/rules clean should clean the source tree to make it pristine.
  2. debian/rules build should trigger the build. For an autoconf-based software, like memcached, this step should execute something like ./configure && make.
  3. debian/rules install should install the file tree of each binary package. For an autoconf-based software, this step should execute make install DESTDIR=debian/memcached.
  4. debian/rules binary will pack the different file trees into binary packages.
You don t directly write each of those targets. Instead, you let dh, a component of debhelper, do most of the work. The following debian/rules file should do almost everything correctly with many source packages: 
#!/usr/bin/make -f
%:
    dh $@
For each of the four targets described above, you can run dh with --no-act to see what it would do. For example: 
$ dh build --no-act
   dh_testdir
   dh_update_autotools_config
   dh_auto_configure
   dh_auto_build
   dh_auto_test
Each of those helpers has a manual page. Helpers starting with dh_auto_ are a bit magic . For example, dh_auto_configure will try to automatically configure a package prior to building: it will detect the build system and invoke ./configure, cmake or Makefile.PL. If one of the helpers do not do the right thing, you can replace it by using an override target: 
override_dh_auto_configure:
    ./configure --with-some-grog
Those helpers are also configurable, so you can just alter a bit their behaviour by invoking them with additional options: 
override_dh_auto_configure:
    dh_auto_configure -- --with-some-grog
This way, ./configure will be called with your custom flag but also with a lot of default flags like --prefix=/usr for better integration. In the initial memcached example, we overrode all those magic targets. dh_auto_clean, dh_auto_configure and dh_auto_build are converted to no-ops to avoid any unexpected behaviour. dh_auto_install is hijacked to do all the build process. Additionally, we modified the behavior of the dh_gencontrol helper by forcing the version number instead of using the one from debian/changelog.

Automatic builds As memcached is an autoconf-enabled package, dh knows how to build it: ./configure && make && make install. Therefore, we can let it handle most of the work with this debian/rules file: 
#!/usr/bin/make -f
DISTRIBUTION = $(shell lsb_release -sr)
VERSION = 1.4.25
PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0
TARBALL = memcached-$(VERSION).tar.gz
URL = http://www.memcached.org/files/$(TARBALL)
%:
    dh $@ --with systemd
override_dh_auto_clean:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)
override_dh_auto_test:
    # Don't run the whitespace test
    rm t/whitespace.t
    dh_auto_test
override_dh_gencontrol:
    dh_gencontrol -- -v$(PACKAGEVERSION)
The dh_auto_clean target is hijacked to download and setup the source tree9. We don t override the dh_auto_configure step, so dh will execute the ./configure script with the appropriate options. We don t override the dh_auto_build step either: dh will execute make. dh_auto_test is invoked after the build and it will run the memcached test suite. We need to override it because one of the test is complaining about odd whitespaces in the debian/ directory. We suppress this rogue test and let dh_auto_test executes the test suite. dh_auto_install is not overriden either, so dh will execute some variant of make install. To get a better sense of the difference, here is a diff: 
--- memcached-intermediate/debian/rules 2016-04-30 14:02:37.425593362 +0200
+++ memcached/debian/rules  2016-05-01 14:55:15.815063835 +0200
@@ -12,10 +12,9 @@
 override_dh_auto_clean:
-override_dh_auto_test:
-override_dh_auto_build:
-override_dh_auto_install:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)
-   ./configure --prefix=/usr
-   make
-   make install DESTDIR=debian/memcached
+
+override_dh_auto_test:
+   # Don't run the whitespace test
+   rm t/whitespace.t
+   dh_auto_test
It is up to you to decide if dh can do some work for you, but you could try to start from a minimal debian/rules and only override some targets.

Install additional files While make install installed the essential files for memcached, you may want to put additional files in the binary package. You could use cp in your build recipe, but you can also declare them:
  • files listed in debian/memcached.docs will be copied to /usr/share/doc/memcached by dh_installdocs,
  • files listed in debian/memcached.examples will be copied to /usr/share/doc/memcached/examples by dh_installexamples,
  • files listed in debian/memcached.manpages will be copied to the appropriate subdirectory of /usr/share/man by dh_installman,
Here is an example using wildcards for debian/memcached.docs: 
doc/*.txt
If you need to copy some files to an arbitrary location, you can list them along with their destination directories in debian/memcached.install and dh_install will take care of the copy. Here is an example: 
scripts/memcached-tool usr/bin
Using those files make the build process more declarative. It is a matter of taste and you are free to use cp in debian/rules instead. You can review the whole package tree on GitHub.

Other examples The GitHub repository contains some additional examples. They all follow the same scheme:
  • dh_auto_clean is hijacked to download and setup the source tree
  • dh_gencontrol is modified to use a computed version
Notably, you ll find daemons in Java, Go, Python and Node.js. The goal of those examples is to demonstrate that using Debian tools to build Debian packages can be straightforward. Hope this helps.
  1. People may remember the time before debhelper 7.0.50 (circa 2009) where debian/rules was a daunting beast. However, nowaday, the boilerplate is quite reduced.
  2. The complexity is not the only reason. Those alternative tools enable the creation of RPM packages, something that Debian tools obviously don t.
  3. There are many ways to version a package. Again, if you want to be pragmatic, the proposed solution should be good enough for Ubuntu. On Debian, it doesn t cover upgrade from one distribution version to another, but we assume that nowadays, systems get reinstalled instead of being upgraded.
  4. You also need to install devscripts and equivs package.
  5. It s also possible to use a script provided by upstream. However, there is no such thing as an init script that works on all distributions. Compare the proposed with the skeleton, check if it is using start-stop-daemon and if it sources /lib/lsb/init-functions before considering it. If it seems to fit, you can install it yourself in debian/memcached/etc/init.d/. debhelper will ensure its proper integration.
  6. Instead, a user wanting to customize the options is expected to edit the unit with systemctl edit.
  7. See #822670
  8. The Debian Policy doesn t provide any hint for the naming convention of those system users. A common usage is to prefix the daemon name with an underscore (like _memcached). Another common usage is to use Debian- as a prefix. The main drawback of the latest solution is that the name is likely to be replaced by the UID in ps and top because of its length.
  9. We could call dh_auto_clean at the end of the target to let it invoke make clean. However, it is assumed that a fresh checkout is used before each build.

Russ Allbery: Review: The Girl with the Dragon Tattoo

Review: The Girl with the Dragon Tattoo, by Stieg Larsson
Translator: Reg Keeland
Series: Millennium #1
Publisher: Vintage Crime
Copyright: 2005, 2008
Printing: June 2009
ISBN: 0-307-47347-3
Format: Mass market
Pages: 644
As The Girl with the Dragon Tattoo opens, Mikael Blomkvist is losing a criminal libel suit in Swedish court. His magazine, Millennium, published his hard-hitting piece of investigative journalism that purported to reveal sketchy arms deals and financial crimes by Hans-Erik Wennerstr m, a major Swedish businessman. But the underlying evidence didn't hold up, and Blomkvist could offer no real defense at trial. The result is a short prison stint for him (postponed several months into this book) and serious economic danger for Millennium. Lisbeth Salander is a (very) freelance investigator for Milton Security. Her specialty is research and background checks: remarkably thorough, dispassionate, and comprehensive. She's almost impossible to talk to, tending to meet nearly all questions with stony silence, but Dragan Armansky, the CEO of Milton Security, has taken her partly under his wing. She, and Milton Security, were hired by a lawyer named Dirch Frode to do a comprehensive background check on Mikael Blomkvist, which she and Dragan present near the start of the book. The reason, as the reader discovers in a few more chapters, is that Frode's employer wants to offer Blomkvist a very strange job. Over forty years ago, Harriet Vanger, scion of one of Sweden's richest industrial families, disappeared. Her uncle, Henrik Vanger, has been obsessed with her disappearance ever since, but in forty years of investigation has never been able to discover what happened to her. There are some possibilities for how her body could have been transported off the island the Vangers (mostly) lived, and live, on, but motive and suspects are still complete unknowns. Vanger wants Blomkvist to try his hand under the cover of writing a book about the Vanger family. Payment is generous, but even more compelling is Henrik Vanger's offer to give Blomkvist documented, defensible evidence against Wennerstr m at the end of the year. The Girl with the Dragon Tattoo (the original Swedish title is M n som hatar kvinnor, "Men who hate women") is the first of three mystery novels written at the very end of Stieg Larsson's life, all published posthumously. They made quite a splash when they were published: won multiple awards, sold millions of copies, and have resulted in four movies to date. I've had a copy of the book sitting around for a while and finally picked it up when in the mood for something a bit different. A major disclaimer up front: I read very little crime and mystery fiction. Every genre has its own conventions and patterns, and regular genre readers often look for different things than people new to that genre. My review is from a somewhat outside and inexperienced perspective, which may not be useful for regular genre readers. I'm also a US reader, reading the book in translation. It appears to be a very good translation, but it was also quite obvious to me that The Girl with the Dragon Tattoo was written from a slightly different set of cultural assumptions than I brought to the book. This is one of the merits of reading books from other cultures in translation. It can be eye-opening, and can carry some of the same thrill as science fiction or fantasy, to hit the parts of the book that question your assumptions. But it can also be hard to tell whether some striking aspect of a book is due to a genre convention I wasn't familiar with, a Swedish cultural assumption that I don't share, or just the personal style of the author. A few things do leap out as cultural differences. Blomkvist has to spend a few months in prison in the middle of this book, and that entire experience is completely foreign to an American understanding of what prison is like. The degradation, violence, and awfulness that are synonymous with prison for an American are almost entirely absent. He even enjoys the experience as quiet time to focus on writing a history of the Vangers (Blomkvist early on decides to take his cover story seriously, since he doubts he'll make any inroads into the mystery of Harriet's disappearance but can at least get a book out of it). It's a minor element in the book, glossed over in a few pages, but it's certainly eye-opening for how minimum security prison could be structured in a civilized country. Similarly, as an American reader, I was struck by how hard Larsson has to work to ruin Salander's life. Although much of the book is written from Blomkvist's perspective (in tight third person), Lisbeth Salander is the titular girl with the dragon tattoo and becomes more and more involved in the story as it develops. The story Larsson wanted to tell requires that she be in a very precarious position legally and socially. In the US, this would be relatively easy, particularly for someone who acts like Salander does. In Sweden, Larsson has to go to monumental efforts to find ways for Salander to credibly fall through Sweden's comprehensive social safety net, and still mostly relies on Salander's complete refusal to assist or comply with any form of authority or support. I've read a lot about differences in policies around social support between the US and Scandinavian countries, but I've rarely read something that drove the point home more clearly than the amount of work a novelist has to go to in order to mess up their protagonist's life in Sweden. The actual plot is slow-moving and as much about the psychology of the characters as it is about the mystery. The reader gets inside the thoughts of the characters occasionally, but Larsson shows far more than tells and leaves it to the reader to draw more general conclusions. Blomkvist's relationship with his long-time partner and Millennium co-founder is an excellent example: so much is left unstated that I would have expected other books to lay down in black and white, and the characters seem surprisingly comfortable with ambiguity. (Some of this may be my genre unfamiliarity; SFF tends to be straightforward to a fault, and more literary fiction is more willing to embrace ambiguous relationships.) While the mystery of Harriet's disappearance forms the backbone of the story, rather more pages are spent on Blomkvist navigating the emotional waters of the near-collapse of his career and business, his principles around investigation and journalism, and the murky waters of the Vanger's deeply dysfunctional family. Harriet's disappearance is something of a locked room mystery. The day she disappeared, a huge crash closed the only bridge from the island to the mainland, both limiting suspects and raising significant questions about why her body was never found on the island. It's also forty years into the past, so Blomkvist has to rely on Henrik Vanger's obsessive archives, old photographs, and old police reports. I found the way it unfolded to be quite satisfying: there are just enough clues to let Blomkvist credibly untangle things with some hard work and research, but they're obscure enough to make it plausible that previous investigators missed them. Through most of this novel, I wasn't sure what I thought of it. I have a personal interest in Blomkvist's journalistic focus wrongdoing by rich financiers but I had trouble warming to Blomkvist himself. He's a very passive, inward character, who spends a lot of the early book reacting to things that are happening to him. Salander is more dynamic and honestly more likable, but she's also deeply messed up, self-destructive, and does some viciously awful things in this book. And the first half of the book is very slow: lots of long conversations, lots of character introduction, and lots of Blomkvist wandering somewhat aimlessly. It's only when Larsson gets the two protagonists together that I thought the book started to click. Salander sees Blomkvist's merits more clearly than the reader can, I think. I also need to give a substantial warning: The Girl with the Dragon Tattoo is a very violent novel, and a lot of that violence is sexual. By mid-book, Blomkvist realizes that Harriet's disappearance is somehow linked with a serial killer whose trademark is horrific, sexualized symbolism drawn from Leviticus. There is a lot of rape here, including revenge rape by a protagonist. If that sort of thing is likely to bother you, you may want to steer way clear. That said, despite the slow pace, the nauseating subject matter, the occasionally very questionable ethics of protagonists, and a twist of the knife at the very end of the novel that I thought was gratuitously nasty on Larsson's part and wasn't the conclusion I wanted, I found myself enjoying this. It has a different pace and a different flavor than what I normally read, the characters are deep and complex enough to play off each other in satisfying ways, and Salander is oddly compelling to read about. Given the length, it's a substantial investment of time, but I don't regret reading it, and I'm quite tempted to read the sequel. I'm not sure this is the sort of book I can recommend (or not recommend) given my lack of familiarity with the genre, but I think US readers might get an additional layer of enjoyment out of seeing how different of a slant the Swedish setting puts on some of the stock elements of a crime novel. Followed by The Girl Who Played with Fire. Rating: 7 out of 10

1 May 2016

Lior Kaplan: Backporting of PHP security fixes

4 months ago I wrote my thoughts about PHP support during the PHP 5 support timeline vote:
I think we should limit what we guarantee (meaning keeping only one year of security support till end of 2017), and encourage project members and the eco-system (e.g. Linux distributions) to maintain further security based on best effort. This is already the case for out of official support releases like the 5.3 and 5.4 branches (examples for backports done by Debian: 5.3 and 5.4). And of course, we also have companies that make their money out of long term support (e.g. RedHat). On the other hand, we should help the eco system in doing such extended support, and hosting backported fixes in the project s git repo instead of having each Linux disto do the same patch work on its own.
But suggesting to others what they should do is easy, so I decided to finally find the time to also implement this myself. I ve started with back porting PHP 5.5 fixes to PHP 5.4, resulting in a GitHub repository with all the fixes, including CVE info NEWS file entries and references to the original commits. See https://github.com/kaplanlior/php-src/commits/PHP-5.4-security-backports . I hope this would later on find it s way into PHP LTS packages for Debian Wheezy. Next step would be to start doing the same for PHP 5.3 (back porting from PHP 5.4, and later on also from PHP 5.5). This can be in use for RHEL 6.x (as LTS support for Debian Squeeze was recently finished). The main idea of this repo, is to have a more central location for such work, hoping people would review and contribute fixes that should be taken into consideration. During the process of digging into the CVE information and the commits, I m also filling up a info such as CVE IDs to the NEWS file (e.g. https://github.com/php/php-src/pull/1892/files) and the web changelog (e.g. https://github.com/php/web-php/commits?author=kaplanlior), so users and researchers would find this info where it should be instead of digging themselves.
Filed under: Debian GNU/Linux, PHP

Thorsten Alteholz: My Debian Activities in April 2016

FTP assistant This month I marked 171 packages for accept and rejected 42. I also sent 3 emails to maintainers asking questions. It seems to be that another quiet month is behind us. Nevertheless the flood of strange things in NEW continued this month. Hmm, weird world .. Debian LTS This was my twenty-second month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. This month my all in all workload had been 15.75h. After getting the permission of the security team I changed the temporary-issues to meanwhile assigned CVEs and uploaded fuseiso. This resulted in DSA 3551-1. I also prepared new packages for asterisk and asked for testers on the LTS mailing list. Luckily Gabriel Filion really tried these packages and found a regression with manager connections. Dear reader, the new packages are waiting for your tests now :-) . Further I used the upload of poppler (DLA 446-1) to test the workflow of the new wheezy-security upload. Uploading and building packages worked perfectly. Unfortunately the push to the security mirrors was a bit delayed (it only happened after an upload of the security team). But this seems to be fixed by Ansgar now. Last but not least I had a look at PHP5. I think I will start my regular uploads in May. Other stuff As I had to deal with non-Debian stuff this month, I didn t do lots of other things. I only uploaded node-uml

C.J. Adams-Collier: OMG Maven 3.0.4 on stretch

Why? , you might ask, would one want to run something other than the most recent version of Maven on the very newest and fangledest breed of the linux distribution we have all loved for so long. Because! , I might answer, I m trying to get the nexus-apt-plugin working on nexus.fd.io, and the version of nexus we re running there explained to me in quite uncertain terms that it would talk to no other version of maven than 3.0.4 or something else that is not packaged for debian. So I grabbed the source for version 3.0.4 from wheezy and patched it up to work with stretch: 
$ cd /usr/src/deb
$ dget http.debian.net/debian/pool/main/m/maven/maven_3.0.4-3+deb7u1.dsc
$ cd maven-3.0.4
$ perl -i -pe 's/(libmodello-maven-plugin)1.4(-java)/$1$2/' debian/control
$ quilt pop -a
$ quild push 1
$ perl -i -pe 's/-1.4.x\.jar/.jar/' build.xml
$ perl -i -pe 's/google-collections/guava/' build.xml
$ perl -i -pe 's/\s+$//' build.xml
$ quilt refresh
$ quilt pop
$ quilt push -a
$ debuild -uc -us
$ sudo apt-get remove maven libmaven3-core-java
$ sudo dpkg -i ../maven_3.0.4-3+deb7u1_all.deb
And now I can build the silly nexus-apt-plugin 
$ mkdir -p /usr/src/git/github
$ git clone git@github.com:LLC-Technologies-Collier/nexus-apt-plugin.git /usr/src/git/github/nexus-apt-plugin
$ cd /usr/src/git/github/nexus-apt-plugin
$ mvn compile && mvn -q test

Russ Allbery: Review: The Oath

Review: The Oath, by Jeffrey Toobin
Publisher: Anchor
Copyright: 2012
Printing: June 2013
ISBN: 0-307-39071-3
Format: Trade paperback
Pages: 298
Jeffrey Toobin is a legal analyst for CNN and The New Yorker and plays a similar role for the intricacies of the legal system as popular science writers play for physics. I'd previously read and reviewed his The Nine, an excellent history of the Rehnquist Supreme Court. The Oath is half sequel and half extension, bringing the same analysis to the first four years of the Obama presidency and the appointments of Sonia Sotomayor and Elena Kagan. Sequels to popular history books that are not explicitly multi-volume works are a tricky publishing niche. People expect them to stand alone; I doubt it would work to tell people "read The Nine before reading this book," and regardless, Toobin did not take that approach. But the court profiled in The Oath only differs by two justices than that in The Nine. There was therefore a fair bit of repetition, since Toobin felt obligated to repeat his profiles of the five members of the court he had already deeply analyzed in the previous book. He even retold the story of Sandra Day O'Conner leaving the court despite it falling outside the focus of this book. I think these 300 pages could have been 150 pages of additional material in The Nine if Toobin had started this project later. That said, if you enjoyed The Nine (and I very much did), this is more of the same. Toobin picks up with Obama's inauguration ceremony and a fascinating bit of legal trivia over the oath of office, and then provides a detailed profile of the Roberts court and the major decisions of the first four years of Obama's presidency. His discussion of the nomination process and Obama's judicial philosophy rang very true following the death of Scalia: Obama's nomination of Merrick Garland is exactly what one would predict from Toobin's discussion. And, as with the previous book, I discovered that I had a lot of misconceptions about both Sotomayor and Kagan that Toobin cleared up. He does a great job showing the complexities of the interplay between law, politics, apparently unlikely friendships (such as Scalia and Ginsburg), and the executive and judicial branch. Worth particular mention is Toobin's discussion of the office of Solicitor General of the United States. I had no idea the role it plays in Supreme Court decisions. If I had given it any thought at all, I would have assumed it was essentially a variation on White House Counsel crossed with the Attorney General's office. But it's quite a bit more than that, as Elena Kagan's profile shows. If you, like I, raised an eyebrow at Obama's nomination of Elena Kagan to the Supreme Court from Solicitor General, wondering if that was at all similar to Bush's nomination of Harriet Miers, this section will be very informative. White House Counsel and Solicitor General are very, very different positions. However, The Oath has one major drawback that The Nine didn't: it's partisan. Now, Toobin is a liberal, with a clear preference towards the progressive side of the court. This was also true in The Nine, and I don't think that's a serious problem. Everyone writes from a particular perspective; stating it is more honest than concealing it, and it's the reader's responsibility to weigh multiple sides. But I thought Toobin was largely fair to those he disagreed with in The Nine. Even Thomas received some defense against popular misconceptions. It probably helped that much of that book focuses on conservatives who became liberals as the court shifted, people like Sandra Day O'Conner for whom Toobin has clear respect. I commented in my review of the previous book that it didn't feel quite balanced, but it felt like Toobin was trying hard to be fair. The Oath does not give that same feeling. Toobin hates the direction of the Roberts court, hates most of its 5-4 decisions, and strongly disagrees with the judicial philosophies of both Roberts and Alito. But more than that, he is clearly dubious that they even have coherent judicial philosophies. Maybe that's a legitimate critique, maybe it's not; regardless, I don't think he proves his case. The tone of much of the book is disgusted and angry rather than deliberate and relentless. Where Toobin engages with the thought process of Alito or, particularly, Roberts, the primary focus is to disagree with it rather than explain it. This happens to match my own emotional reaction, but I doubt it will be persuasive to someone who doesn't already agree with Toobin, and it hurts the quality of the history. I suspect this would have been a better book if Toobin had waited ten years before writing it (still covering the same time frame). Some distance from the subject helps provide a more complete and thoughtful history. But, of course, it likely wouldn't have sold as well. That said, one of the themes of this book is how the conservatives on the Roberts court are currently playing the role of radicals from the perspective of the judicial tradition, overturning settled case law and calling into question precedents that have been used to decide numerous cases. The liberals, in contrast, are currently mostly playing the role of conservatives: standing up for the principle of stare decisis, trying to maintain consistency with past decisions, trying to minimize disruptive change. Conservatives will argue (correctly) that this depends on one's time frame and that they're trying to overturn radical past decisions, but those radical decisions, whatever their merits, are now often more than fifty years into the past. I hadn't thought about the current Supreme Court ideological battles from that perspective and found it eye-opening. It also ties in well with Obama's judicial philosophy as Toobin presents it: preferring democracy, laws, and change from the ballot box, and with little appetite for controversial court decisions. Obama is a judicial conservative. He therefore favors the liberal wing as the court is currently constructed, but not because he has much appetite for pushing forward civil rights in the courts. This is not the book The Nine was. It's repetitive if you've read the previous book (which you should, as it's the better book of the two), and I thought Toobin's critical balance was off. But it has a lot of interesting things to say about Obama's approach to the law, how the executive branch interacts with the Supreme Court, and the philosophy and approaches of the newer justices on the court. Recommended, although not as strongly. Rating: 7 out of 10

Next.

Previous.