Search Results: "nate"

9 May 2016

Mike Gabriel: Recent progress in NXv3 development

This is to give a comprehensive overview on the recent progress made in NXv3 (aka nx-libs) development. The upstream sources of nx-libs can be found at / viewed on / cloned from Github:
https://github.com/ArcticaProject/nx-libs A great portion of the current work is sponsored by the Qindel Group [1] in Spain (QINDEL FORMACI N Y SERVICIOS, S.L.). Thanks for making this possible. Planned release date: 2nd July, 2016 We aim at releasing a completely tidied up nx-libs code tree versioned 3.6.0 on July 2nd, 2016. There is still a whole bunch of work to do for this, but I am positive that we can make this release date. Goals of our Efforts There are basically two major goals for spending a considerable amount of time, money and energy on NXv3 hacking: The efforts undertaken always have the various existing use cases in mind (esp. the framework of the coming-up Arctica Project, TheQVD and X2Go). Overview on Recent Development Progress General Code Cleanups Making this beast maintainable means first of all: identifying code redundancies, unused code passages, etc. and remove them. This is where we came from (NoMachine's NX 3.5.x, including nxcomp, nxcompext, nxcompshad, nx-X11 and nxagent): 1,757,743 lines of C/C++ code. 
[mike@minobo nx-libs.35 (3.5.0.x)]$ cloc --match-f '.*\.(c cpp h)$' .
    5624 text files.
    5614 unique files.                                          
    2701 files ignored.
http://cloc.sourceforge.net v 1.60  T=18.59 s (302.0 files/s, 132847.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C                             3134         231180         252893        1326393
C/C++ Header                  2274          78062         116132         349743
C++                            206          20037          13312          81607
-------------------------------------------------------------------------------
SUM:                          5614         329279         382337        1757743
-------------------------------------------------------------------------------
On the current 3.6.x branch of nx-libs (at commit 6c6b6b9), this is where we are now: 662,635 lines of C/C++ code, amount of code reduced to a third of the original code lines. 
[mike@minobo nx-libs (3.6.x)]$ cloc --match-f '.*\.(c cpp h)' .
    2012 text files.
    2011 unique files.                                          
    1898 files ignored.
http://cloc.sourceforge.net v 1.60  T=5.63 s (341.5 files/s, 161351.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C                             1015          74605          81625         463244
C/C++ Header                   785          26992          34354         138063
C++                            122          16984          10804          61328
-------------------------------------------------------------------------------
SUM:                          1922         118581         126783         662635
-------------------------------------------------------------------------------
The latest development branch currently has these statistics: 619,353 lines of C/C++ code, another 40,000 lines could be dropped. 
[mike@minobo nx-libs (pr/libnx-xext-drop-unused-extensions)]$ cloc --match-f '.*\.(c cpp h)' .
    1932 text files.
    1931 unique files.                                          
    1898 files ignored.
http://cloc.sourceforge.net v 1.60  T=5.66 s (325.4 files/s, 150598.1 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C                              983          69474          77186         426564
C/C++ Header                   738          25616          33048         131599
C++                            121          16984          10802          61190
-------------------------------------------------------------------------------
SUM:                          1842         112074         121036         619353
-------------------------------------------------------------------------------
Dropping various libNX_X* shared libraries (and using X.org shared client libraries instead) At first, various bundled libraries could be dropped from the nx-X11 code tree. Secondly, several of the bundled X.org libraries could be dropped, because we managed to build against those libraries as provided system-wide. Then, and this undertaking is much trickier, we could drop nearly all Xlib extension libraries that are used by nxagent with its role of being an X11 client. We could sucessfully drop these Xlib extension libraries from nx-X11, because we managed to build nxagent against the matching libraries in X.org: libNX_Xdmcp, libNX_Xfixes, libNX_XComposite, libNX_Xdamage, libNX_Xtst, libNX_Xinerama, libNX_Xfont, libNX_xkbui, and various others. All these droppings happened without a loss of functionality. However, some shared X client libraries are not easy to remove without loss of functionality, or rather not removable at all. Dropping libNX_Xrender We recently dropped libNX_Xrender [2] and now build nxagent against X.org's libXrender. However, this cost us a compression feature in NX. The libNX_Xrender code has passages that do zero padding of the unused memory portions in non-32bit-depth glyphs (the NX_RENDER_CLEANUP feature). However, we have hope for being able to reintroduce that feature again later, but current efforts [3] still fail at run-time. Dropping libNX_Xext is not possible... ...the libNX_Xext / Xserver Xext code has been cleaned up instead. Quite an amount of research and testing has been spent on removing the libNX_Xext library from the build workflow of nxagent. However, it seems that building against X.org's libXext will require us to update the Xext part of the nxagent Xserver at the same time. While taking this deep look into Xext code, we dropped various Xext extensions from the nx-X11 Xserver code. The extensions that got dropped [5] are all extensions that already have been dropped from X.org's Xserver code, as well. Further investigation, however, showed, that actually the only used client side extension code from libNX_Xext is the XShape extension. Thus, all other client side extension got dropped now in a very recent pull request [4]. Dropping libNX_X11 not easy, research summary given here For the sake of dropping the Xlib library bundled with nx-libs, we have attempted at writing a shared library called libNX_Xproxy. This library is supposed to contain a subset of the NXtrans related Xlib code that NoMachine patched into X.org's libX11 (and libxtrans). Results of this undertaking [6] so far: Over the weekend, I thought this all through once more and I am pretty sure right now, that we can actually make libNX_Xproxy work and drop all of libNX_X11 from nx-libs soon. Although we have to port (i.e. copy) various functions related to the NX transport from libNX_X11 into libNX_Xproxy, this change will allow us to drop all Xlib drawing routines and use those provided by X.org's Xlib shared library directly. Composite extension backport in nxagent to version 0.4 Mihai Moldovan looked at what it needs to make the Composite extension functional in nxagent. Unfortunately, the exact answer cannot be given, yet. As a start, Mihai backported latest Composite extension (v0.4) code from X.org's Xserver into the nxagent Xserver [7]. The currently shipped Composite extension version in nxagent is v0.2. Work on the NX Compression shared library (aka nxcomp v3) Fernando Carvajal and Salvador Fandi o from the Qindel Group [1] filed three pull requests against the nxcomp part of nx-libs recently, two of them have been code cleanups (done by Fernando), the third one is a feature enhancement regarding channels in nxcomp (provided by Salva). Protocol clean-up: drop pre-3.5 support With the release of nx-libs 3.6.x, we will drop support for nxcomp versions earlier than 3.5. Thus, if you are still on nxcomp 3.4, be prepared for upgrading at least to version 3.5.x. The code removal had been issued as pull request #111 ("Remove compatibility code for nxcomp before 3.5.0") [8]. The PR has already been reviewed and merged. Fernando filed another code cleanup PR (#119 [9]) against nx-libs that also already got merged into the 3.6.x branch. UNIX Socket Support for Channels The nxcomp library (and thus, nxproxy) provides a feature called "channels". Channels in nxcomp can be used for forwarding traffic between NX client side and the NX server side (along the graphical X11 transport that nxcomp is designed for). Until version 3.5.x, nxcomp was only able to use local TCP sockets for providing / connecting to channel endpoints. We consider local TCP sockets as insecure and aim at adding UNIX file socket support to nxcomp whereever a connection is established. Salva provided a patch against nxcomp that provides UNIX socket support to channel endpoints. The initial use case for this patch is: connect to client side pulseaudio socket file and avoid enabling the TCP listening socket in pulseaudio. The traffic then is channeled to the server side, so that pulse clients can connect to a UNIX socket file rather than to a local TCP port. The channel support patch has already been reviewed and merged into the 3.6.x branch of nx-libs. Rebasing nxagent against latest X.org Ulrich Sibiller spent a considerable amount of time and energy on providing a build chain that allows building nxagent against a modularized X.org 7.0 (rather than against the monolithic build tree of X.org 6.9, like we still do in nx-libs 3.6.x). We plan to adapt and minimize this build workflow for nx-libs 3.7.x (scheduled for summer 2017). A short howto that shows how to build nxagent with that new workflow will be posted on this blog within the next days. So stay tuned. Further work accomplished Quite a lot of code cleanup PRs have been filed by myself against nx-libs. Most of them target at removal of unnecessary code from the nx-X11 Xserver code base and the nxagent DDX: The third one (PR #120) in the list requires some detailled explanation: We discovered that nxagent ships overrides some symbols from the original Xserver code base. These overrides are induced by copies of some files from some Xserver sub-directory placed into the hw/nxagent/ DDX path. All those files' names match the pattern NX*.c. These copies of code are done in a way that the C compiler suppresses throwing its 'symbol "" redefined: first defined in ""; redefined in ""' errors. The approach taken, however, requires to have quite a few 10.000 lines of redundant code in hw/nxagent/NX*.c that also gets shipped in some Xserver sub-directory (mostly dix/ and render/). With pull request #120, we have identified all code passages in hw/nxagent/NX*.c that must be considered as NX'ish. We differentiated the NX'ish functions from functions that never got changed by NoMachine when developing nxagent. I then came up with four different approaches ([13,14,15,16]) of dropping redundant code from those hw/nxagent/NX*.c files. We (Mihai Moldovan and myself) discussed the various approaches and favoured the disable-Xserver-code-and-include-into-NX*.c variant [14] over the others for the following reasons: In the long run, the Xserver portion of the patches provided via this pull request #120 are required to be upstreamed into X.org's Xserver. The discussion around this will be started when we fully dive into rebasing nxagent's Xserver code base against latest X.org Xserver. Tasks ahead before the 3.6.x Release Various tasks we face before 3.6.x can be released. Here is a probably incomplete list: Tasks ahead after the 3.6.x Release (i.e., for 3.7.x) Here is an even rougher and probably highly incomplete list for tasks after the 3.6.x release: Credits Some people have to be named here that give their heart and love to this project. Thank you guys for supporting the development efforts around nx-libs and the Arctica Project: Thanks to Nico Arenas Alonso from and on behalf of the Qindel Group for coordinating the current funding project around nx-libs. Thanks to Ulrich Sibiller for giving a great amount of spare time to working on the nxagent-rebase-against-X.org effort. Thanks to Mihai Moldovan for doing endless code reviews and being available for contracted work via BAUR-ITCS UG [17] on NXv3, as well. Thanks to Mario Becroft for providing a patch that allows us to hook into nxagent X11 sessions with VNC and have the session fully available over VNC while the NX transport is in suspended state. Also Mario is pouring some fancy UDP ideas into the re-invention of remote desktop computing process performed in the Arctica Project. Mario has been an NX supporter for years, I am glad to have him still around after so many years (although he was close to abandoning NX usage at least once). Thanks to Fernando Carvajal from Qindel (until April 2016) for cleaning up nxcomp code. Thanks to Orion Poplawski from the Fedora Project for working on the first bundled libraries removal patches and being a resource on RPM packaging. Thanks to my friend Lee for working behind the scenes on the Arctica Core code and constantly pouring various of his ideas into my head. Thanks for regularly reminding me on benchmarking things.
Folks, thanks to all of you for all your various efforts on this huge beast of software. You are a great resource of expertise and it's a pleasure and honour working with you all.
New Faces Last but not least, I'd like to let everyone know that the Qindel Group sponsors another developer joining in on NXv3 development: Vadim Troshchinskiy (aka vatral on Github). Vadim has worked on NXv3 before and we are looking forward to having him and his skills in the team soon (probably end of May 2016). Welcome on board of the team, Vadim.
[1] http://www.qindel.com
[2] https://github.com/ArcticaProject/nx-libs/pull/93
[3] https://github.com/sunweaver/nx-libs/commit/be41bde7efc46582b442706dfb85...
[4] https://github.com/ArcticaProject/nx-libs/pull/121
[5] https://github.com/ArcticaProject/nx-libs/pull/106
[6] https://github.com/sunweaver/nx-libs/tree/wip/libnx-x11-full-removal
[7] https://github.com/sunweaver/nx-libs/tree/pr/composite-0_4
[8] https://github.com/ArcticaProject/nx-libs/pull/111
[9] https://github.com/ArcticaProject/nx-libs/pull/119
[10] https://github.com/ArcticaProject/nx-libs/pull/102
[11] https://github.com/ArcticaProject/nx-libs/pull/106
[12] https://github.com/ArcticaProject/nx-libs/pull/120
[13] https://github.com/sunweaver/nx-libs/commit/9692e6a7045b3ab5cb0daaed187e... (include NX'ish code into Xserver)
[14] https://github.com/sunweaver/nx-libs/commit/3d359bfc2b6d021c1ae9c6e19e96... (include Xserver code into NX*.c)
[15] https://github.com/sunweaver/nx-libs/commit/af72ee5624a15d21c610528e37b6... (use weak symbols and non-static symbols)
[16] https://github.com/sunweaver/nx-libs/commit/7205bb8848c49ee3e78a82fde906... (override symbols with interceptions)
[17] http://www.baur-itcs.de/20-x2go/20-x2gosupport/

8 May 2016

Dirk Eddelbuettel: Rblpapi 0.3.4

A new release of Rblpapi is now on CRAN. It provides a direct interface between R and the Bloomberg Terminal via the C++ API provided by Bloomberg Labs (but note that a valid Bloomberg license and installation is required). This marks the fifth release since the package first appeared on CRAN last year. Continued thanks to all contributors for code, suggestions or bug reports. This release contains a lot of internal fixes by Whit, John and myself and should prove to be more resilient to 'odd' representations of data coming back. The NEWS.Rd extract has more details:
Changes in Rblpapi version 0.3.4 (2016-05-08)
  • On startup, the API versions of both the headers and the runtime are displayed (PR #161 and #165).
  • Documentation about extended futures roll notation was added to the bdh manual page.
  • Additional examples for overrides where added to bdh (PR #158).
  • Internal code changes make retrieval of data in unusual variable types more robust (PRs #157 and #153)
  • General improvements and fixes to documentation (PR #156)
  • The bdp function now also supports an option verbose (PR #149).
  • The internal header Rblpapi_types.h was renamed from a lower-cased variant to conform with Rcpp Attributes best practices (PR #145).
Courtesy of CRANberries, there is also a diffstat report for the this release. As always, more detailed information is on the Rblpapi page. Questions, comments etc should go to the issue tickets system at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Niels Thykier: Another Britney patchset

I just submitted another patch series to improve Britney for review. If accepted, they will probably be merged into master within 2 weeks. The changes this time are probably most exciting for people that run/maintain Britney. Key highlights include: I would like to dwell a moment on the faux packages . We have had a helper script generate and inject fake packages into the list of packages (called faux packages ). They generally serve two purposes, which Britney will support:
  1. Whitelist of fake packages to satisfy dependencies of other packages.
    • These are generally stand-in for non-free machine configuration packages, where the end-user system would also fetch packages from the vendor s repository.
    • Packages relying on faux packages are generally not in main as Debian s main component is required to be self-contained.
    • These are (still) be called faux packages in/after the patch series
  2. Ensuring that certain packages are present and installable in testing.
    • We have a lot of d-i related packages here to avoid accidental breakage of d-i.
    • These are now referred to as a constraint (assuming there is no bike-shedding over the name).
Since Britney will now distinguish between these two use cases, I also make Britney enforce the second use case slightly better. Mind you, it can still be overruled by force-hints and BREAK_ARCHES, so there still enough rope to hang yourself. The other exciting part of this patch set (for me, at least) is that Britney will hopefully become simpler to deploy. No doubt there are still some missing features and paper cuts left, but I suspect we are not far from a:
  1. Fill out a template config file pointing Britney to your mirror
  2. Run britney -c britney.conf
  3. Make your archive kit update your target suite based on Britney s output.
  4. Put step 2+3 in crontab/jenkins/task scheduler of choice
  5. Profit
There will certainly be some features that requires extra steps. An example is the anti rc-bugs regression feature, which requires you to feed Britney with the list of RC bugs for your source and target suite. But even without, Britney would still protect your target suite from most installability issues.
Filed under: Debian, Release-Team

7 May 2016

Dirk Eddelbuettel: BH 1.60.0-2

A new minor release of BH is now on CRAN. BH provides a large part of the Boost C++ libraries as a set of template headers for use by R, possibly with Rcpp as well as other packages. This release uses the same Boost 1.60.0 version of Boost as the last release, but adds three more library: bimap, flyweight and icl. A brief summary of changes from the NEWS file is below.
Changes in version 1.60.0-2 (2016-05-06)
  • Added Boost bimap via GH pull request #24 by Jim Hester.
  • Added Boost icl via GH pull request #27 by Jay Hesselbert.
  • Added Boost flyweight as requested in GH ticket #26.
Courtesy of CRANberries, there is also a diffstat report for the most recent release. Comments and suggestions are welcome via the mailing list or the issue tracker at the GitHubGitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

6 May 2016

Norbert Preining: Yubikey NEO

Two Factor authentication and general improvement of my security infrastructure was long on my todo list. Some month ago I finally purchased a Yubikey NEO from Yubico and try to consistently use it as second factor, as well as gpg signing/encrypting device. yubikey-neo I am trying to get the best out of my Yubikey NEO by using as many of its functionality, in particular: Smartcard for my GNuPG keys, OTP similar to Google Authenticator and similar, as well as challenge-response for additional login security, as well as all that over NFC to not keep keys/passwords on my mobile phone. While there are loads of guides (see the previous article on GnuPG for some of them), many of them are out-of-date for current distributions and GnuPG etc. So I tried to collect all I could find not the least to have a place to look it up in case I forget it again. The Hardware The Yubikey NEO is a great peace of hardware. I not even remotely understand how they manage that this little beast can do all these things and still work out without mixing things up. As far as I understand (please correct me) it has three independent circuits of communication: On top of these circuit of communication there is a variety of applications to make the most out of your Yubikey: Yubikey mode setup There are several modes, and using the ykpersonalize tool (readily available for Windows, Mac, Linux, and in the Debian package yubikey-personalization) one can program the key to work in a variety of modes. I chose to activate all options by passing in -m86 which stand for OTP/U2F/CCID composite device with MODE_FLAG_EJECT. 
$ ykpersonalize -m86
Firmware version 3.4.3 Touch level 1792 Unconfigured
 
The USB mode will be set to: 0x86
 
Commit? (y/n) [n]: y
$
It is a good idea to unplug and replug the key after this operation. Yubikey udev rules for user access To allow users but root to use the Yubikey, additional udev rules are necessary: 
SUBSYSTEMS=="usb", ATTRS idVendor =="1050", ATTRS idProduct =="0116", TAG+="uaccess"
which I put into /etc/udev/rules.d/99-yubikeys.rules on Debian. After that another unplug and replug should allow normal user to access the key. This can be checked by calling getfacl on the newly created /dev/hidraw? device. Using the HID/Challenge-Response mode (slot 2) If you want to secure your login with an additional second factor, there are several options documented on the Yubico site concerning yubico-pam. Since I cannot be sure to be always online with my laptop, I choose Challenge-Response authentication, and followed one-to-one Yubico s docs Local Authentication Using Challenge Response. Basically it boils down to install libpam-yubico, select mode-challenge-response when asked for configuration. Then one needs to personalizing the key (in particular slot 2) for challenge response with: 
$ ykpersonalize -2 -ochal-resp -ochal-hmac -ohmac-lt64 -oserial-api-visible
...
Commit? (y/n) [n]: y
$
Next we need to save the challenge and expected response to the user s directory: 
$ mkdir $HOME/.yubico
$ ykpamcfg -2 -v
...
Stored initial challenge and expected response in '/home/norbert/.yubico/challenge-123456'.
$
It might be a good idea to try this out, and if it works, activate it also for root. But be careful no key no login Challenge: I am currently searching for a method to replace the second factor of they key optionally with a different authentication method, like a very difficult passphrase. This way I could log in even without my key, but in this case would need the complicated passphrase. From my reading of the pam manuals it seems to be possible, and I am planning to use pam_ssh and a specific login key with a complicated passphrase. I will report back when this is done. YubiOATH (TOTP) Time based One Time Passwords (aka Google Authenticator style) Without any setup whatsoever this worked out of the box. I use the Yubico Authenticator on my Android phone, and the dedicated application for the Linux desktop to create second factors for all kind of applications. Currently I am using it with Google login, Github, DropbBox, and WordPress (via the Two Factor plugin which can also be tweaked to use the NEO key as USB key via the FIDO U2F). Challenge: If I start the Yubico Personalization GUI, I see two free slots so where are the TOTPs computed? That also means that I have one slot free and for now I don t know what to do with it Yubikey OpenGPG applet setup The Yubikeys support OpenPGP, and the applet is pre-installed (afaik), meaning you can directly configure the key and upload your keys. Here I use gpg2 (2.1) as it seems to better support card operations. To not interfere with the current gpg setup I use a temporary gpg home: 
$ mkdir gpgtmp
$ chmod go-rwx gpgtmp
$ gpg2 --homedir gpgtmp --list-keys
gpg: keybox 'gpgtmp/pubring.kbx' created
gpg: gpgtmp/trustdb.gpg: trustdb created
Warning: The YubiKey NEO only supports 2048bit keys. If you want 4096bit keys you need to use one of the newer YubiKey 4, which gives you this option, but does not have support for NFC, and thus no way to interact with an Android (or other) mobile phone. Check the correct version of the applet There has been a bug in an older version of the applet, but since 2 years all keys sold should have a correct applet. You can check by: 
$ gpg-connect-agent --homedir gpgtmp --hex "scd apdu 00 f1 00 00" /bye"
gpg-connect-agent: no running gpg-agent - starting '/usr/bin/gpg-agent'
gpg-connect-agent: waiting for the agent to come up ... (5s)
gpg-connect-agent: connection to agent established
D[0000]  01 00 10 90 00                                     .....           
OK
Looking at the output one sees D[0000] 01 00 10 which means applet version 1.0.10, which is the first version fixed. Replace pins of the key The standard pins are 123456 for the user pin, and 12345678 for the admin pin. These need immediate change! Warning: When changing the ping the normal pin must be 6 (at least?) digits, and the admin pin 8 (at least?), other gpg2 cannot use the key anymore. No idea why. 
$ gpg2 --homedir gpgtmp --card-edit
 
Reader ...........: 1050:0116:X:0
Application ID ...: D2760001240102000006036457190000
Version ..........: 2.0
Manufacturer .....: Yubico
Serial number ....: 03645719
Name of cardholder: [not set]
Language prefs ...: [not set]
Sex ..............: unspecified
URL of public key : [not set]
Login data .......: [not set]
Signature PIN ....: forced
Key attributes ...: rsa2048 rsa2048 rsa2048
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 3 3
Signature counter : 0
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]
 
gpg/card> admin
Admin commands are allowed
 
gpg/card> passwd
gpg: OpenPGP card no. D2760001240102000006036457190000 detected
 
1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit
 
Your selection? 3
PIN changed.
 
1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit
 
Your selection? 1
PIN changed.
 
1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit
 
Your selection? q
 
gpg/card> quit
After this you need to use the new pins for all changes. Setup basic data The key can also save some basic data about yourself, like name, sex, language preferences, login name, and url to obtain the public key. As before start gpg2 and then change these infos in the following way> 
gpg/card> name
Cardholder's surname: Preining
Cardholder's given name: Norbert
 
gpg/card> sex
Sex ((M)ale, (F)emale or space): M
 
gpg/card> lang
Language preferences: de
 
gpg/card> login
Login data (account name): norbert
 
gpg/card> url
URL to retrieve public key: https://www.preining.info/preining-norbert.asc
 
gpg/card> list
 
Reader ...........: 1050:0116:X:0
Application ID ...: D2760001240102000006036457190000
Version ..........: 2.0
Manufacturer .....: Yubico
Serial number ....: 03645719
Name of cardholder: Norbert Preining
Language prefs ...: de
Sex ..............: male
URL of public key : https://www.preining.info/preining-norbert.asc
Login data .......: norbert
Signature PIN ....: forced
Key attributes ...: rsa2048 rsa2048 rsa2048
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 3 3
Signature counter : 0
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]
 
gpg/card> quit
Move sub keys to Yubikey As laid out in the article on GnuPG subkeys, we are having three subkeys for signing, encryption, and authentication. In reality I will practically only use the signing key, but upload all three keys to the card. In the following I expect that you have a setup more or less similar to the one described in the article linked before. Again, we use GnuPG v2, mostly because it was the version that worked out of the box. In addition, if you are setting up a similar stage like in my GNuPG article with gpg1 keys on the mail server, then you don t want the gpg1 keys being removed. Basically you must have the Yubikey plugged in and call keytocard after selecting each key in turn (and deselecting it afterwards). Warning: There is another bug in the GnuPG applet that was fixed in later versions (but not in 1.0.10), namely that not all keys are accepted. This is a bit a pain. I needed to recreate a subkey to obtain a key that can be loaded onto the Yubikey. Unfortunately, Yubico has also stopped/disabled the ability to update applets (although I have to say their documentation is an incredible rubbish with respect to applets and upgrades ). As before, assume that $MASTERKEY contains the hex id of your master key. 
$ gpg2 --edit-key $MASTERKEY
gpg (GnuPG) 2.1.11; Copyright (C) 2016 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
 
Secret key is available.
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 2
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb* rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> keytocard
Please select where to store the key:
   (1) Signature key
   (3) Authentication key
Your selection? 1
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb* rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 2
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 3
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb* rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> keytocard
Please select where to store the key:
   (2) Encryption key
Your selection? 2
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb* rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 3
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 4
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb* rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> keytocard
Please select where to store the key:
   (3) Authentication key
Your selection? 3
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb* rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 4
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> save
After that your keys are on the Yubikey (and only there!), and GNuPG will require the PIN (user pin) to sign/encrypt documents. Usage Many things have been said above, but to sum up when and how I am using the YubiKey now: Conclusions With this setup I am now quite content, but not completely. What I still want to do is full disk encryption where I need the Yubikey to boot and again, with an alternative for a very long passphrase. At the end, adding a second factor to the login is not really optimal, and only protects you against quick hacks. If the laptop is actually stolen, only full disc protection helps. Access to the hardware always guarantees that one has access to everything on the disc. Another thing I want to do is re-use the GnuPG key on the Yubikey as ssh key for logging into remote systems. That would mean that I get rid of even more keys on my laptop. But this is still in the work The other open question is what to use the other available slot of the Yubikey for? I thought about some passwords (possible), but I don t feel to happy about having my password issued with the press of a key. But all in all, I like the setup much more than before and not having any GnuPG key on the laptop is a big plus.

Matthias Klumpp: Adventures in D programming

I recently wrote a bigger project in the D programming language, the appstream-generator (asgen). Since I rarely leave the C/C++/Python realm, and came to like many aspects of D, I thought blogging about my experience could be useful for people considering to use D. Disclaimer: I am not an expert on programming language design, and this is not universally valid criticism of D just my personal opinion from building one project with it. Why choose D in the first place? The previous AppStream generator was written in Python, which wasn t ideal for the task for multiple reasons, most notably multiprocessing and LMDB not working well together (and in general, multiprocessing being terrible to work with) and the need to reimplement some already existing C code in Python again. So, I wanted a compiled language which would work well together with the existing C code in libappstream. Using C was an option, but my least favourite one (writing this in C would have been much more cumbersome). I looked at Go and Rust and wrote some small programs performing basic operations that I needed for asgen, to get a feeling for the language. Interfacing C code with Go was relatively hard since libappstream is a GObject-based C library, I expected to be able to auto-generate Go bindings from the GIR, but there were only few outdated projects available which did that. Rust on the other hand required the most time in learning it, and since I only briefly looked into it, I still can t write Rust code without having the coding reference open. I started to implement the same examples in D just for fun, as I didn t plan to use D (I was aiming at Go back then), but the language looked interesting. The D language had the huge advantage of being very familiar to me as a C/C++ programmer, while also having a rich standard library, which included great stuff like std.concurrency.Generator, std.parallelism, etc. Translating Python code into D was incredibly easy, additionally a gir-d-generator which is actively maintained exists (I created a small fork anyway, to be able to directly link against the libappstream library, instead of dynamically loading it). What is great about D? This list is just a huge braindump of things I had on my mind at the time of writing Interfacing with C There are multiple things which make D awesome, for example interfacing with C code and to a limited degree with C++ code is really easy. Also, working with functions from C in D feels natural. Take these C functions imported into D:
extern(C):
nothrow:
struct _mystruct  
alias mystruct_p = _mystruct*;
mystruct_p = mystruct_create ();
mystruct_load_file (mystruct_p my, const(char) *filename);
mystruct_free (mystruct_p my);
You can call them from D code in two ways:
auto test = mystruct_create ();
// treating "test" as function parameter
mystruct_load_file (test, "/tmp/example");
// treating the function as member of "test"
test.mystruct_load_file ("/tmp/example");
test.mystruct_free ();
This allows writing logically sane code, in case the C functions can really be considered member functions of the struct they are acting on. This property of the language is a general concept, so a function which takes a string as first parameter, can also be called like a member function of string. Writing D bindings to existing C code is also really simple, and can even be automatized using tools like dstep. Since D can also easily export C functions, calling D code from C is also possible. Getting rid of C++ cruft There are many things which are bad in C++, some of which are inherited from C. D kills pretty much all of the stuff I found annoying. Some cool stuff from D is now in C++ as well, which makes this point a bit less strong, but it s still valid. E.g. getting rid of the #include preprocessor dance by using symbolic import statements makes sense, and there have IMHO been huge improvements over C++ when it comes to metaprogramming. Incredibly powerful metaprogramming Getting into detail about that would take way too long, but the metaprogramming abilities of D must be mentioned. You can do pretty much anything at compiletime, for example compiling regular expressions to make them run faster at runtime, or mixing in additional code from string constants. The template system is also very well thought out, and never caused me headaches as much as C++ sometimes manages to do. Built-in unit-test support Unittesting with D is really easy: You just add one or more unittest blocks to your code, in which you write your tests. When running the tests, the D compiler will collect the unittest blocks and build a test application out of them. The unittest scope is useful, because you can keep the actual code and the tests close together, and it encourages writing tests and keep them up-to-date. Additionally, D has built-in support for contract programming, which helps to further reduce bugs by validating input/output. Safe D While D gives you the whole power of a low-level system programming language, it also allows you to write safer code and have the compiler check for that, while still being able to use unsafe functions when needed. Unfortunately, @safe is not the default for functions though. Separate operators for addition and concatenation D exclusively uses the + operator for addition, while the ~ operator is used for concatenation. This is likely a personal quirk, but I love it very much that this distinction exists. It s nice for things like addition of two vectors vs. concatenation of vectors, and makes the whole language much more precise in its meaning. Optional garbage collector D has an optional garbage collector. Developing in D without GC is currently a bit cumbersome, but these issues are being addressed. If you can live with a GC though, having it active makes programming much easier. Built-in documentation generator This is almost granted for most new languages, but still something I want to mention: Ddoc is a standard tool to generate code documentation for D code, with a defined syntax for describing function parameters, classes, etc. It will even take the contents of a unittest scope to generate automatic examples for the usage of a function, which is pretty cool. Scope blocks The scope statement allows one to execute a bit of code before the function exists, when it failed or was successful. This is incredibly useful when working with C code, where a free statement needs to be issued when the function is exited, or some arbitrary cleanup needs to be performed on error. Yes, we do have smart pointers in C++ and with some GCC/Clang extensions a similar feature in C too. But the scopes concept in D is much more powerful. See Scope Guard Statement for details. Built-in syntax for parallel programming Working with threads is so much more fun in D compared to C! I recommend taking a look at the parallelism chapter of the Programming in D book. Pure functions D allows to mark functions as purely-functional, which allows the compiler to do optimizations on them, e.g. cache their return value. See pure-functions. D is fast! D matches the speed of C++ in almost all occasions, so you won t lose performance when writing D code that is, unless you have the GC run often in a threaded environment. Very active and friendly community The D community is very active and friendly so far I only had good experience, and I basically came into the community asking some tough questions regarding distro-integration and ABI stability of D. The D community is very enthusiastic about pushing D and especially the metaprogramming features of D to its limits, and consists of very knowledgeable people. Most discussion happens at the forums/newsgroups at forum.dlang.org. What is bad about D? Half-proprietary reference compiler This is probably the biggest issue. Not because the proprietary compiler is bad per se, but because of the implications this has for the D ecosystem. For the reference D compiler, Digital Mars D (DMD), only the frontend is distributed under a free license (Boost), while the backend is proprietary. The FLOSS frontend is what the free compilers, LLVM D Compiler (LDC) and GNU D Compiler (GDC) are based on. But since DMD is the reference compiler, most features land there first, and the Phobos standard library and druntime is tuned to work with DMD first. Since major Linux distributions can t ship with DMD, and the free compilers GDC and LDC lack behind DMD in terms of language, runtime and standard-library compatibility, this creates a split world of code that compiles with LDC, GDC or DMD, but never with all D compilers due to it relying on features not yet in e.g. GDCs Phobos. Especially for Linux distributions, there is no way to say use this compiler to get the best and latest D compatibility . Additionally, if people can t simply apt install latest-d, they are less likely to try the language. This is probably mainly an issue on Linux, but since Linux is the place where web applications are usually written and people are likely to try out new languages, it s really bad that the proprietary reference compiler is hurting D adoption in that way. That being said, I want to make clear DMD is a great compiler, which is very fast and build efficient code. I only criticise the fact that it is the language reference compiler. UPDATE: To clarify the half-proprietary nature of the compiler, let me quote the D FAQ:
The front end for the dmd D compiler is open source. The back end for dmd is licensed from Symantec, and is not compatible with open-source licenses such as the GPL. Nonetheless, the complete source comes with the compiler, and all development takes place publically on github. Compilers using the DMD front end and the GCC and LLVM open source backends are also available. The runtime library is completely open source using the Boost License 1.0. The gdc and ldc D compilers are completely open sourced.
Phobos (standard library) is deprecating features too quickly This basically goes hand in hand with the compiler issue mentioned above. Each D compiler ships its own version of Phobos, which it was tested against. For GDC, which I used to compile my code due to LDC having bugs at that time, this means that it is shipping with a very outdated copy of Phobos. Due to the rapid evolution of Phobos, this meant that the documentation of Phobos and the actual code I was working with were not always in sync, leading to many frustrating experiences. Furthermore, Phobos is sometimes removing deprecated bits about a year after they have been deprecated. Together with the older-Phobos situation, you might find yourself in a place where a feature was dropped, but the cool replacement is not yet available. Or you are unable to import some 3rd-party code because it uses some deprecated-and-removed feature internally. Or you are unable to use other code, because it was developed with a D compiler shipping with a newer Phobos. This is really annoying, and probably the biggest source of unhappiness I had while working with D especially the documentation not matching the actual code is a bad experience for someone new to the language. Incomplete free compilers with varying degrees of maturity LDC and GDC have bugs, and for someone new to the language it s not clear which one to choose. Both LDC and GDC have their own issues at time, but they are rapidly getting better, and I only encountered some actual compiler bugs in LDC (GDC worked fine, but with an incredibly out-of-date Phobos). All issues are fixed meanwhile, but this was a frustrating experience. Some clear advice or explanation which of the free compilers is to prefer when you are new to D would be neat. For GDC in particular, being developed outside of the main GCC project is likely a problem, because distributors need to manually add it to their GCC packaging, instead of having it readily available. I assume this is due to the DRuntime/Phobos not being subjected to the FSF CLA, but I can t actually say anything substantial about this issue. Debian adds GDC to its GCC packaging, but e.g. Fedora does not do that. No ABI compatibility D has a defined ABI too bad that in reality, the compilers are not interoperable. A binary compiled with GDC can t call a library compiled with LDC or DMD. GDC actually doesn t even support building shared libraries yet. For distributions, this is quite terrible, because it means that there must be one default D compiler, without any exception, and that users also need to use that specific compiler to link against distribution-provided D libraries. The different runtimes per compiler complicate that problem further. The D package manager, dub, does not yet play well with distro packaging This is an issue that is important to me, since I want my software to be easily packageable by Linux distributions. The issues causing packaging to be hard are reported as dub issue #838 and issue #839, with quite positive feedback so far, so this might soon be solved. The GC is sometimes an issue The garbage collector in D is quite dated (according to their own docs) and is currently being reworked. While working with asgen, which is a program creating a large amount of interconnected data structures in a threaded environment, I realized that the GC is significantly slowing down the application when threads are used (it also seems to use UNIX signals SIGUSR1 and SIGUSR2 to stop/resume threads, which I still find odd). Also, the GC performed poorly on memory pressure, which did get asgen killed by the OOM killer on some more memory-constrained machines. Triggering a manual collection run after a large amount of these interconnected data structures wasn t needed anymore solved this problem for most systems, but it would of course have been better to not needing to give the GC any hints. The stop-the-world behavior isn t a problem for asgen, but it might be for other applications. These issues are at time being worked on, with a GSoC project laying the foundation for further GC improvements. version is a reserved word Okay, that is admittedly a very tiny nitpick, but when developing an app which works with packages and versions, it s slightly annoying. The version keyword is used for conditional compilation, and needing to abbreviate it to ver in all parts of the code sucks a little (e.g. the Package interface can t have a property version , but now has ver instead). The ecosystem is not (yet) mature In general it can be said that the D ecosystem, while existing for almost 9 years, is not yet that mature. There are various quirks you have to deal with when working with D code on Linux. It s always nothing major, usually you can easily solve these issues and go on, but it s annoying to have these papercuts. This is not something which can be resolved by D itself, this point will solve itself as more people start to use D and D support in Linux distributions gets more polished. Conclusion I like to work with D, and I consider it to be a great language the quirks it has in its toolchain are not that bad to prevent writing great things with it. At time, if I am not writing a shared library or something which uses much existing C++ code, I would prefer D for that task. If a garbage collector is a problem (e.g. for some real-time applications, or when the target architecture can t run a GC), I would not recommend to use D. Rust seems to be the much better choice then. In any case, D s flat learning curve (for C/C++ people) paired with the smart choices taken in language design, the powerful metaprogramming, the rich standard library and helpful community makes it great to try out and to develop software for scenarios where you would otherwise choose C++ or Java. Quite honestly, I think D could be a great language for tasks where you would usually choose Python, Java or C++, and I am seriously considering to replace quite some Python code with D code. For very low-level stuff, C is IMHO still the better choice. As always, choosing the right programming language is only 50% technical aspects, and 50% personal taste UPDATE: To get some idea of D, check out the D tour on the new website tour.dlang.org.

Russ Allbery: Review: The Language of Power

Review: The Language of Power, by Rosemary Kirstein
Series: Steerswomen #4
Publisher: Rosemary Kirstein
Copyright: 2004, 2014
Printing: April 2014
ISBN: 0-9913546-3-X
Format: Kindle
Pages: 400
This is the fourth book in the Steerswomen series and definitely not the place to start. It's also a difficult series to review without spoilers, so I won't be able to provide too many details about the plot. I will say that this is a reunion and a return of sorts to themes from earlier in the series, rather than a direct follow-up to the revelations at the end of The Lost Steersman. Rowan is back in the Inner Lands, continuing to investigate the affairs of wizards. In particular, she's digging into the past of the town of Donner, following up on the report of an earlier steerswoman and investigating a now-dead wizard who seemed to act far different from a typical wizard. And Bel is back at her side again, watching her back. The first half of The Language of Power goes over somewhat familiar ground. Similar to both The Steerswoman and The Lost Steersman, Rowan is poking around in a city, getting to know unfamiliar people, being a steerswoman, and winning people over with her unique charm. But that's a theme I don't mind seeing repeated, since Rowan is one of my favorite protagonists from any series I've read. She's both ethical and respectful in a way that doesn't feel artificial or constructed. She thinks oddly and dives into sudden fascinations, and she does rely on the conventions for interacting with steerswomen, but the more time one spends with her, the better one likes her. This is true of both the reader and the town inhabitants, and Kirstein is brilliant at writing the gradual getting-to-know and growing-respect process. Events in The Language of Power slowly build up to another confrontation with wizards, and this one is full of major revelations about the world. Some of the ambiguity of earlier books is firmly resolved, we find out a lot more about how wizards view themselves and their abilities, and tons of new questions are raised. It's not a conclusion in any way, which is a bit unfortunate given that the next book is still being written (twelve years later, although thankfully it's being actively worked on as I write this). But we get the first clear look at the substratum of the world that Kirstein is building in this series. This sounds satisfying, and to some extent it is, but any regular SFF reader will have guessed at many of the revelations here. I was pretty sure the world was following one of two possible patterns partway through the second book, certain which it was during the third book, and was nodding right along with the revelations in this book. I'm trying to avoid spoilers, but if you read a lot of SFF, chances are you've read about something akin to this background before. That takes a bit of the thrill out of the revelations, unfortunately. What adds the excitement and thrill back in are Rowan's reactions. It's very difficult to write a character who comes from an entirely different perspective than either the author or the reader, and Kirstein does an amazing job. Not perfect, quite, at least for me: there were a few points where I thought Rowan was more baffled or more upset than it felt like she should have been. But they are few and far between, and it's quite possible my expectations are the ones that wouldn't ring true if written into the story. It's just such a delight to see Rowan analyzing the world, incorporating new revelations into her growing world model, and figuring out how to take the most moral action at any point. I would happily read another dozen books of this (but I wish they were all already written). If you've read the previous three books, definitely pick up this one as well. For me, it narrowly misses being the best book of the series (I think that's still The Outskirter's Secret because of the difficulty of the perspective change Kirstein pulls off), but it's a close competition. And the final reveal at the very end of this book points to upcoming adventures that I can hardly wait to read. Rating: 8 out of 10

Dirk Eddelbuettel: RcppArmadillo 0.6.700.6.0

armadillo image A second Armadillo release 6.700.6 came out in the 6.700 series, and we uploaded RcppArmadillo 0.6.700.6.0 to CRAN and Debian. This followed the usual thorough reverse-dependecy checking of by now 220 packages using. This release is a little unusual in that it contains both upstream bugfixes in the same series (see below) but also two nice bug fixes from the RcppArmadillo side. Both were squashed by George G. Vega Yon via two focused pull request. The first ensures that we can now use ARMA_64BIT_WORD (provided C++11 is turned on too) allowing for much bigger Armadillo objects. And the second plugs a small leak in the sparse matrix converter I had added a while back. Nice work, all told! Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. Changes in this release are as follows:
Changes in RcppArmadillo version 0.6.700.6.0 (2016-05-05)
  • Upgraded to Armadillo 6.700.6 (Catabolic Amalgamator Deluxe)
    • fix for handling empty matrices by kron()
    • fix for clang warning in advanced matrix constructors
    • fix for false deprecated warning in trunc_log() and trunc_exp()
    • fix for gcc-6.1 warning about misleading indentation
    • corrected documentation for the solve() function
  • Added support for int64_t (ARMA_64BIT_WORD) when required during compilation time. (PR #90 by George G. Vega Yon, fixing #88)
  • Fixed bug in SpMat exporter (PR #91 by George G. Vega Yon, fixing #89 and #72)
Courtesy of CRANberries, there is also a diffstat report for this release. As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

5 May 2016

Sean Whitton: dh_make_elpa & dh_elpa_test

I recently completed and released some work on Debian s tooling for packaging Emacs Lisp addons to GNU Emacs. Emacs grew a package manager called package.el a few years ago, and last year David Bremner wrote the dh_elpa tool to simplify packaging addons for Debian by leveraging package.el features. Packaging a series of addons for Debian left me with a wishlist of features for dh_elpa and I was recently able to implement them. Debian tooling generally uses Perl, a language I didn t know before starting on this project. I was fortunate enough to receive a free review copy of Perl 5 by Example when I attended a meeting of the Bay Area Linux Users Group while I was visiting San Francisco a few months ago. I accepted the book with the intent of doing this work. dh_make_elpa dh_make_elpa (at present available from Debian experimental) is a Perl script to convert a git repository cloned from the upstream of an Emacs Lisp addon to a rudimentary Debian package. It performs a lot of guesswork, and its simple heuristics are something I hope to improve on. Since I am new to object-oriented program design in Perl and I wanted to leverage object-oriented Debian tooling library code, I took the structure of my project from dh_make_perl. In this manner I found it easy and pleasant to write a maintainable script. dh_elpa_test A lot of Emacs Lisp addon packages use a program called Cask to manage the Emacs Lisp dependencies needed to run their test suites. That meant that dh_auto_test often fails to run Emacs Lisp addon package test suites. Since the Debian packaging toolchain already has advanced dependency management, it s undesirable to involve Cask in the package build pipeline if it can be avoided. I had been copying and pasting the code needed to make the tests run in our environment to the debian/rules files of each package whose test suite I wanted to run. dh_elpa_test tries to detect Emacs Lisp addon package test suites and run them with the workarounds needed in our environment. This avoids boilerplate in debian/rules. dh_elpa_test also disables dh_auto_test to avoid a inadvertent Cask invocation. Future & acknowledgements My hope for this work was to make it easier and faster to package Emacs Lisp addon packages for Debian, for my own sake and for anyone new who is interested in joining the pkg-emacsen team. In the future, I want to have dh_elpa_test generate an autopkgtest definition so that a Testsuite: pkg-emacsen line in debian/control is enough to have an Emacs Lisp addon package test suite run on Debian CI. I m very grateful to David Bremner for reviewing and supporting this work, and also for supporting my Emacs Lisp addon packaging work more generally.

3 May 2016

Jamie McClelland: Monitoring Deflect

May First/People Link has several members that are targets of politically motivated denial of service attacks (mostly groups that support reproductive justice for women and palestinian rights). To fight off the attacks, we work closely with Deflect - a non-governmental organization based in Canada that fights against this kind of censorship. When a site is down, it's not always easy to understand why. Deflect runs as many as 5 edge servers, any of them could be down. And, of course, the origin server could also be down. I tried using a commericial/free as in beer service for monitoring up time, but when it reported the site being down, I had no idea which part was down. So... httping to the rescue. Unfortunately, it depends on --divert-connect which is only available in Debian Stretch. I run the script via a cron job and output the results to a log file. 
    #!/bin/bash
    # Test all given edges 
    domain="$1"
    origin="$2"
    proto=http
    if [ -n "$3" ]; then
        proto="$3"
    fi
    if [ -z "$domain" ]; then
        printf "Please pass the domain as first argument.\n"
        exit 1
    fi
    if ! ping -c 1 4.2.2.1 >/dev/null; then
        # printf "We are offline. Not running.\n"
        exit 1
    fi
    ips=$(dig +short "$domain")
    if [ "$?" -ne "0" ]; then
        # printf "DNS lookup failure. Not running.\n"
        exit 1
    fi
    if [ -n "$origin" ]; then
        ips="$ips $origin"
    fi
    l=
    if [ "$proto" = "https" ]; then
        l=-l
    fi
    for ip in $ips; do
        date=$(date +%Y.%m.%d-%H:%M)
        for i in 1 2 3; do
            out=$(httping $l -m -t 5 -c 1 --divert-connect "$ip" "$proto://$domain")
            [ -z "$out" ] && out=1
            printf "%s %s %s\n" "$date" "$ip" "$out"
        done
    done

Rapha&#235;l Hertzog: My Free Software Activities in April 2016

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS I handled a new LTS sponsor that wanted to see wheezy keep supporting armel and armhf. This was not part of our initial plans (set during last Debconf) and I thus mailed all teams that were impacted if we were to collectively decide that it was OK to support those architectures. While I was hoping to get a clear answer rather quickly, it turns out that we never managed to get an answer to the question from all parties. Instead the discussion drifted on the more general topic of how we handle sponsorship/funding in the LTS project. Fortunately, the buildd maintainers said they were OK with this and the ftpmasters had no objections, and they both implicitly enacted the decision: Ansgar Burchardt kept the armel/armhf architectures in the wheezy/updates suite when he handled the switch to the LTS team, and Aur lien Jarno also configured wanna-build to keep building armel/armhf for the suite. The DSA team did not confirm that this change was not interfering with one of their plans to decommission some hardware. Build daemons are a shared resource anyway and a single server is likely to handle builds for multiple releases. DebConf 16 This month I registered for DebConf 16 and submitted multiple talk/BoF proposals: I want to share the setup we use in Kali as it can be useful for other derivatives and also for Debian itself to help smooth the relationship with derivatives. I also want to open again the debate on the usage of money within Debian. It s a hard topic but we should really strive to take some official position on what s possible and what s not possible. With Debian LTS and its sponsorship we have seen that we can use money to some extent without hurting the Debian project as a whole. Can this be transposed to other teams or projects? What are the limits? Can we define a framework and clear rules? I expect the discussion to be very interesting in the BoF. Mehdi Dogguy has agreed to handle this BoF with me. Packaging Django. I uploaded 1.8.12 to jessie-backports and 1.9.5 to unstable. I filed two upstream bugs (26473 and 26474) for two problems spotted by lintian. Unfortunately, when I wanted to upload it to unstable, the test suite did not ran. I pinned this down to a sqlite regression. Chris Lamb filed #820225 and I contacted the SQLite and Django upstream developers by email to point them to this issue. I helped the SQLite upstream author (Richard Hipp) to reproduce the issue and he was quick to provide a patch which landed in 3.12.1. Later in the month I made another upload to fix an upgrade bug (#821789). GNOME 3.20. As for each new version, I updated gnome-shell-timer to ensure it works with the new GNOME. This time I spent a bit more time to fix a regression (805347) that dates back to a while and that would never be fixed otherwise since the upstream author orphaned this extension (as he no longer uses GNOME). I have also been bitten by display problems where accented characters would be displayed below the character that follows. With the help of members of the GNOME team, we found out that this was a problem specific to the cantarell font and was only triggered with Harfbuzz 1.2. This is tracked in Debian with #822682 on harfbuzz and #822762 in fonts-cantarell. There s a new upstream release (with the fix) ready to be packaged but unfortunately it is blocked by the lack of a recent fontforge in Debian. I thus mailed debian-mentors in the hope to find volunteers to help the pkg-fonts team to package a newer version Misc Debian/Kali work Distro Tracker. I started to mentor Vladimir Likic who contacted me because he wants to contribute to Distro Tracker. I helped him to setup his development environment and we fixed a few issues in the process. Bug reports. I filed many bug reports, most of them due to my work on Kali: I also investigated #819958 that was affecting testing since it has been reported to Kali as well. And I made an NMU of dh-make-golang to fix #819472 that I reported earlier. Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Russ Allbery: Review: The Effective Engineer

Review: The Effective Engineer, by Edmond Lau
Publisher: Effective Bookshelf
Copyright: 2015
ISBN: 0-9961281-0-7
Format: Trade paperback
Pages: 222
Silicon Valley start-up tech companies have a standard way of thinking about work. Large chunks of this come from Google, which pioneered a wide variety of new, or at least not-yet-mainstream, ways of organizing and thinking about work. The rest accreted through experience with fast-paced start-ups, engineer-focused companies, web delivery of products, and rabid turnover and high job mobility within a hothouse of fairly similar companies. A key part of this mindset is the firm belief that this atmosphere has created a better way to work, at least for software engineers (and systems administrators, although heaven forbid that one call them that any more): more effective, more efficient, more focused on what really matters. I think this is at least partly true, at least from the perspective of a software engineer. This Silicon Valley work structure focuses on data gathering, data-based decision-making, introspection, analysis, and continuous improvement, all of which I think are defensibly pointed in the right direction (if rarely as rigorous as one might want to believe). It absorbs bits and pieces of work organization techniques that are almost certainly improvements for the type of work software engineers do: Agile, Lean, continuous deployment, and fast iteration times. In other cases, though, I'm less convinced that this Silicon Valley consensus is objectively better as opposed to simply different; interviewing, for instance, is a puzzle that I don't think anyone has figured out, and the remarkable consensus in Silicon Valley on how to interview (basically, "like Google except for the bits we thought were obnoxious") feels more like a social fad than a sign of getting it right. But every industry has its culture of good ideas, bad ideas, fads, and fashion, and it's quite valuable to know that culture if you want to work in that industry. The Effective Engineer is a self-published book by Edmund Lau, a Silicon Valley software engineer who also drifted (as is so common in Silicon Valley) into mentoring, organizing, and speaking to other software engineers. Its purpose, per the subtitle, is to tell you "how to leverage your efforts in software engineering to make a disproportionate and meaningful impact." While that's not exactly wrong, and the book contains some useful and valuable tips, I'd tend to give it a slightly different subtitle: "a primer on how a Silicon Valley software engineer is expected to think about their work." This is a bit more practical, a bit less confident, and a bit less convinced of its own correctness than Lau might want to present his work, but it's just as valuable of a purpose if you want to work in the industry. (And is a bit more honest about its applicability outside of that industry.) What this book does extremely well is present, in a condensed, straightforward, and fast-moving form, most of the highlights of how start-ups and web-scale companies approach software engineering and the SWE role in companies (SWE, meaning software engineer, is another bit of Google terminology that's now nearly universal). If you've already worked in or around this industry for a while, you've probably picked up a lot of this via osmosis: prioritize based on impact and be unapologetic about letting other things drop, have a growth mindset, reprioritize regularly, increase your iteration speed, measure everything constantly, check your assumptions against data, derisk your estimates, use code review and automated testing (but not too much), automate operations, and invest heavily in hiring and onboarding. (The preceding list is a chapter list for this book.) If you're working at one of these sorts of companies, you're probably currently somewhere between nodding and rolling your eyes because no one at work will shut up about these topics. But if you've not worked inside one of these companies, even if you've done software engineering elsewhere, this is a great book to read to prepare yourself. You're going to hear about these ideas constantly, and, if it achieves nothing else at all, The Effective Engineer will give you a firm enough grounding in the lingo and mindset that you can have intelligent conversations with people who assume this is the only way to think about software engineering. By this point, you might be detecting a certain cynicism in this review. It's not entirely fair: a lot of these ideas are clearly good ones, and Lau does a good job of describing them quickly and coherently. It's a good job for what it is. But there are a couple of things that limited its appeal for me. First, it's definitely a primer. I read it after having worked at a web-scale start-up for a year and a half. There wasn't much in it that seemed particularly new, and it's somewhat superficial. The whole middle section in particular (build tools for yourself, measure everything, be data-driven) are topics for which the devil is often in the details. Lau gives you the terminology and the expected benefits, but putting any one of these techniques into practice could be a book (or several) by itself. Don't expect to come away from The Effective Engineer with much of a concrete plan for how to do these things in your day-to-day software development projects. But it's a good reminder to be thinking about, say, how to embed metrics and data-gathering hooks into the software you write. This is the nature of a primer; no 222-page book can get into much depth about the fractal complexity of doing good, fast, scalable software development. Second, there's a fundamental question raised by a book like this: effective at what? Lau tackles that in the first chapter with his focus on impact and leverage, and it's good advice as far as it goes. (Regular readers of my book reviews know that I love this sort of time management and prioritization discussion.) But measuring impact is a hard problem that requires a prioritization framework, and this is not really the book for this. The Effective Engineer is written primarily for software developers at start-ups, leaves the whole venture-capital start-up process as unquestioned background material, and accepts without comment the standard measures of value in that world: fast-deployed products, hypergrowth, racing competitors for perceived innovation, and finding ways to extract money. That's as deep into the question of impact as Lau gets: increases in company revenue. There's nothing wrong with this for the kind of book Lau intended to write, and it's not his fault that I find it unsatisfying. But don't expect The Effective Engineer to ask any hard questions about whether that's a meaningful definition of impact, or to talk much about less objective goals: quality of implementation, craftsmanship, giving back to a broader community via free software contributions, impact on the world in ways that can't be measured in market share, or anything else that is unlikely to lead to objective impact for company profits. At best he leaves a bit of wiggle room around using the concept of impact with different goals. If you're a new graduate who wants to work at Silicon-Valley-style start-ups, this is a great orientation, and likewise if you're coming from a different area of software development into that world. If you're not working in that industry, The Effective Engineer may still be moderately interesting, but it's not written for that audience and has little or nothing to say of the challenges of other types of businesses. But if you've already worked in the industry for a while, or if you're more interested in deeper discussions of goals and subjective values, you may not get much out of this. Rating: 7 out of 10

2 May 2016

Vincent Bernat: Pragmatic Debian packaging

While the creation of Debian packages is abundantly documented, most tutorials are targeted to packages implementing the Debian policy. Moreover, Debian packaging has a reputation of being unnecessarily difficult1 and many people prefer to use less constrained tools2 like fpm or CheckInstall. However, I would like to show how building Debian packages with the official tools can become straightforward if you bend some rules:
  1. No source package will be generated. Packages will be built directly from a checkout of a VCS repository.
  2. Additional dependencies can be downloaded during build. Packaging individually each dependency is a painstaking work, notably when you have to deal with some fast-paced ecosystems like Java, Javascript and Go.
  3. The produced packages may bundle dependencies. This is likely to raise some concerns about security and long-term maintenance, but this is a common trade-off in many ecosystems, notably Java, Javascript and Go.

Pragmatic packages 101 In the Debian archive, you have two kinds of packages: the source packages and the binary packages. Each binary package is built from a source package. You need a name for each package. As stated in the introduction, we won t generate a source package but we will work with its unpacked form which is any source tree containing a debian/ directory. In our examples, we will start with a source tree containing only a debian/ directory but you are free to include this debian/ directory into an existing project. As an example, we will package memcached, a distributed memory cache. There are four files to create:
  • debian/compat,
  • debian/changelog,
  • debian/control, and
  • debian/rules.
The first one is easy. Just put 9 in it: 
echo 9 > debian/compat
The second one has the following content: 
memcached (0-0) UNRELEASED; urgency=medium
  * Fake entry
 -- Happy Packager <happy@example.com>  Tue, 19 Apr 2016 22:27:05 +0200
The only important information is the name of the source package, memcached, on the first line. Everything else can be left as is as it won t influence the generated binary packages.

The control file debian/control describes the metadata of both the source package and the generated binary packages. We have to write a block for each of them. 
Source: memcached
Maintainer: Vincent Bernat <bernat@debian.org>
Package: memcached
Architecture: any
Description: high-performance memory object caching system
The source package is called memcached. We have to use the same name as in debian/changelog. We generate only one binary package: memcached. In the remaining of the example, when you see memcached, this is the name of a binary package. The Architecture field should be set to either any or all. Use all exclusively if the package contains only arch-independent files. In doubt, just stick to any. The Description field contains a short description of the binary package.

The build recipe The last mandatory file is debian/rules. It s the recipe of the package. We need to retrieve memcached, build it and install its file tree in debian/memcached/. It looks like this: 
#!/usr/bin/make -f
DISTRIBUTION = $(shell lsb_release -sr)
VERSION = 1.4.25
PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0
TARBALL = memcached-$(VERSION).tar.gz
URL = http://www.memcached.org/files/$(TARBALL)
%:
    dh $@
override_dh_auto_clean:
override_dh_auto_test:
override_dh_auto_build:
override_dh_auto_install:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)
    ./configure --prefix=/usr
    make
    make install DESTDIR=debian/memcached
override_dh_gencontrol:
    dh_gencontrol -- -v$(PACKAGEVERSION)
The empty targets override_dh_auto_clean, override_dh_auto_test and override_dh_auto_build keep debhelper from being too smart. The override_dh_gencontrol target sets the package version3 without updating debian/changelog. If you ignore the slight boilerplate, the recipe is quite similar to what you would have done with fpm: 
DISTRIBUTION=$(lsb_release -sr)
VERSION=1.4.25
PACKAGEVERSION=$ VERSION -0~$ DISTRIBUTION 0
TARBALL=memcached-$ VERSION .tar.gz
URL=http://www.memcached.org/files/$ TARBALL 
wget -N --progress=dot:mega $ URL 
tar --strip-components=1 -xf $ TARBALL 
./configure --prefix=/usr
make
make install DESTDIR=/tmp/installdir
# Build the final package
fpm -s dir -t deb \
    -n memcached \
    -v $ PACKAGEVERSION  \
    -C /tmp/installdir \
    --description "high-performance memory object caching system"
You can review the whole package tree on GitHub and build it with dpkg-buildpackage -us -uc -b.

Pragmatic packages 102 At this point, we can iterate and add several improvements to our memcached package. None of those are mandatory but they are usually worth the additional effort.

Build dependencies Our initial build recipe only work when several packages are installed, like wget and libevent-dev. They are not present on all Debian systems. You can easily express that you need them by adding a Build-Depends section for the source package in debian/control: 
Source: memcached
Build-Depends: debhelper (>= 9),
               wget, ca-certificates, lsb-release,
               libevent-dev
Always specify the debhelper (>= 9) dependency as we heavily rely on it. We don t require make or a C compiler because it is assumed that the build-essential meta-package is installed and it pulls those. dpkg-buildpackage will complain if the dependencies are not met. If you want to install those packages from your CI system, you can use the following command4: 
mk-build-deps \
    -t 'apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -qqy' \
    -i -r debian/control
You may also want to investigate pbuilder or sbuild, two tools to build Debian packages in a clean isolated environment.

Runtime dependencies If the resulting package is installed on a freshly installed machine, it won t work because it will be missing libevent, a required library for memcached. You can express the dependencies needed by each binary package by adding a Depends field. Moreover, for dynamic libraries, you can automatically get the right dependencies by using some substitution variables: 
Package: memcached
Depends: $ misc:Depends , $ shlibs:Depends
The resulting package will contain the following information: 
$ dpkg -I ../memcached_1.4.25-0\~unstable0_amd64.deb   grep Depends
 Depends: libc6 (>= 2.17), libevent-2.0-5 (>= 2.0.10-stable)

Integration with init system Most packaged daemons come with some integration with the init system. This integration ensures the daemon will be started on boot and restarted on upgrade. For Debian-based distributions, there are several init systems available. The most prominent ones are:
  • System-V init is the historical init system. More modern inits are able to reuse scripts written for this init, so this is a safe common denominator for packaged daemons.
  • Upstart is the less-historical init system for Ubuntu (used in Ubuntu 14.10 and previous releases).
  • systemd is the default init system for Debian since Jessie and for Ubuntu since 15.04.
Writing a correct script for the System-V init is error-prone. Therefore, I usually prefer to provide a native configuration file for the default init system of the targeted distribution (Upstart and systemd).

System-V If you want to provide a System-V init script, have a look at /etc/init.d/skeleton on the most ancient distribution you want to target and adapt it5. Put the result in debian/memcached.init. It will be installed at the right place, invoked on install, upgrade and removal. On Debian-based systems, many init scripts allow user customizations by providing a /etc/default/memcached file. You can ship one by putting its content in debian/memcached.default.

Upstart Providing an Upstart job is similar: put it in debian/memcached.upstart. For example: 
description "memcached daemon"
start on runlevel [2345]
stop on runlevel [!2345]
respawn
respawn limit 5 60
expect daemon
script
  . /etc/default/memcached
  exec memcached -d -u $USER -p $PORT -m $CACHESIZE -c $MAXCONN $OPTIONS
end script
When writing an Upstart job, the most important directive is expect. Be sure to get it right. Here, we use expect daemon and memcached is started with the -d flag.

systemd Providing a systemd unit is a bit more complex. The content of the file should go in debian/memcached.service. For example: 
[Unit]
Description=memcached daemon
After=network.target
[Service]
Type=forking
EnvironmentFile=/etc/default/memcached
ExecStart=/usr/bin/memcached -d -u $USER -p $PORT -m $CACHESIZE -c $MAXCONN $OPTIONS
Restart=on-failure
[Install]
WantedBy=multi-user.target
We reuse /etc/default/memcached even if it is not considered a good practice with systemd6. Like for Upstart, the directive Type is quite important. We used forking as memcached is started with the -d flag. You also need to add a build-dependency to dh-systemd in debian/control: 
Source: memcached
Build-Depends: debhelper (>= 9),
               wget, ca-certificates, lsb-release,
               libevent-dev,
               dh-systemd
And you need to modify the default rule in debian/rules: 
%:
    dh $@ --with systemd
The extra complexity is a bit unfortunate but systemd integration is not part of debhelper7. Without those additional modifications, the unit will get installed but you won t get a proper integration and the service won t be enabled on install or boot.

Dedicated user Many daemons don t need to run as root and it is a good practice to ship a dedicated user. In the case of memcached, we can provide a _memcached user8. Add a debian/memcached.postinst file with the following content: 
#!/bin/sh
set -e
case "$1" in
    configure)
        adduser --system --disabled-password --disabled-login --home /var/empty \
                --no-create-home --quiet --force-badname --group _memcached
        ;;
esac
#DEBHELPER#
exit 0
There is no cleanup of the user when the package is removed for two reasons:
  1. Less stuff to write.
  2. The user could still own some files.
The utility adduser will do the right thing whatever the requested user already exists or not. You need to add it as a dependency in debian/control: 
Package: memcached
Depends: $ misc:Depends , $ shlibs:Depends , adduser
The #DEBHELPER# marker is important as it will be replaced by some code to handle the service configuration files (or some other stuff). You can review the whole package tree on GitHub and build it with dpkg-buildpackage -us -uc -b.

Pragmatic packages 103 It is possible to leverage debhelper to reduce the recipe size and to make it more declarative. This section is quite optional and it requires understanding a bit more how a Debian package is built. Feel free to skip it.

The big picture There are four steps to build a regular Debian package:
  1. debian/rules clean should clean the source tree to make it pristine.
  2. debian/rules build should trigger the build. For an autoconf-based software, like memcached, this step should execute something like ./configure && make.
  3. debian/rules install should install the file tree of each binary package. For an autoconf-based software, this step should execute make install DESTDIR=debian/memcached.
  4. debian/rules binary will pack the different file trees into binary packages.
You don t directly write each of those targets. Instead, you let dh, a component of debhelper, do most of the work. The following debian/rules file should do almost everything correctly with many source packages: 
#!/usr/bin/make -f
%:
    dh $@
For each of the four targets described above, you can run dh with --no-act to see what it would do. For example: 
$ dh build --no-act
   dh_testdir
   dh_update_autotools_config
   dh_auto_configure
   dh_auto_build
   dh_auto_test
Each of those helpers has a manual page. Helpers starting with dh_auto_ are a bit magic . For example, dh_auto_configure will try to automatically configure a package prior to building: it will detect the build system and invoke ./configure, cmake or Makefile.PL. If one of the helpers do not do the right thing, you can replace it by using an override target: 
override_dh_auto_configure:
    ./configure --with-some-grog
Those helpers are also configurable, so you can just alter a bit their behaviour by invoking them with additional options: 
override_dh_auto_configure:
    dh_auto_configure -- --with-some-grog
This way, ./configure will be called with your custom flag but also with a lot of default flags like --prefix=/usr for better integration. In the initial memcached example, we overrode all those magic targets. dh_auto_clean, dh_auto_configure and dh_auto_build are converted to no-ops to avoid any unexpected behaviour. dh_auto_install is hijacked to do all the build process. Additionally, we modified the behavior of the dh_gencontrol helper by forcing the version number instead of using the one from debian/changelog.

Automatic builds As memcached is an autoconf-enabled package, dh knows how to build it: ./configure && make && make install. Therefore, we can let it handle most of the work with this debian/rules file: 
#!/usr/bin/make -f
DISTRIBUTION = $(shell lsb_release -sr)
VERSION = 1.4.25
PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0
TARBALL = memcached-$(VERSION).tar.gz
URL = http://www.memcached.org/files/$(TARBALL)
%:
    dh $@ --with systemd
override_dh_auto_clean:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)
override_dh_auto_test:
    # Don't run the whitespace test
    rm t/whitespace.t
    dh_auto_test
override_dh_gencontrol:
    dh_gencontrol -- -v$(PACKAGEVERSION)
The dh_auto_clean target is hijacked to download and setup the source tree9. We don t override the dh_auto_configure step, so dh will execute the ./configure script with the appropriate options. We don t override the dh_auto_build step either: dh will execute make. dh_auto_test is invoked after the build and it will run the memcached test suite. We need to override it because one of the test is complaining about odd whitespaces in the debian/ directory. We suppress this rogue test and let dh_auto_test executes the test suite. dh_auto_install is not overriden either, so dh will execute some variant of make install. To get a better sense of the difference, here is a diff: 
--- memcached-intermediate/debian/rules 2016-04-30 14:02:37.425593362 +0200
+++ memcached/debian/rules  2016-05-01 14:55:15.815063835 +0200
@@ -12,10 +12,9 @@
 override_dh_auto_clean:
-override_dh_auto_test:
-override_dh_auto_build:
-override_dh_auto_install:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)
-   ./configure --prefix=/usr
-   make
-   make install DESTDIR=debian/memcached
+
+override_dh_auto_test:
+   # Don't run the whitespace test
+   rm t/whitespace.t
+   dh_auto_test
It is up to you to decide if dh can do some work for you, but you could try to start from a minimal debian/rules and only override some targets.

Install additional files While make install installed the essential files for memcached, you may want to put additional files in the binary package. You could use cp in your build recipe, but you can also declare them:
  • files listed in debian/memcached.docs will be copied to /usr/share/doc/memcached by dh_installdocs,
  • files listed in debian/memcached.examples will be copied to /usr/share/doc/memcached/examples by dh_installexamples,
  • files listed in debian/memcached.manpages will be copied to the appropriate subdirectory of /usr/share/man by dh_installman,
Here is an example using wildcards for debian/memcached.docs: 
doc/*.txt
If you need to copy some files to an arbitrary location, you can list them along with their destination directories in debian/memcached.install and dh_install will take care of the copy. Here is an example: 
scripts/memcached-tool usr/bin
Using those files make the build process more declarative. It is a matter of taste and you are free to use cp in debian/rules instead. You can review the whole package tree on GitHub.

Other examples The GitHub repository contains some additional examples. They all follow the same scheme:
  • dh_auto_clean is hijacked to download and setup the source tree
  • dh_gencontrol is modified to use a computed version
Notably, you ll find daemons in Java, Go, Python and Node.js. The goal of those examples is to demonstrate that using Debian tools to build Debian packages can be straightforward. Hope this helps.
  1. People may remember the time before debhelper 7.0.50 (circa 2009) where debian/rules was a daunting beast. However, nowaday, the boilerplate is quite reduced.
  2. The complexity is not the only reason. Those alternative tools enable the creation of RPM packages, something that Debian tools obviously don t.
  3. There are many ways to version a package. Again, if you want to be pragmatic, the proposed solution should be good enough for Ubuntu. On Debian, it doesn t cover upgrade from one distribution version to another, but we assume that nowadays, systems get reinstalled instead of being upgraded.
  4. You also need to install devscripts and equivs package.
  5. It s also possible to use a script provided by upstream. However, there is no such thing as an init script that works on all distributions. Compare the proposed with the skeleton, check if it is using start-stop-daemon and if it sources /lib/lsb/init-functions before considering it. If it seems to fit, you can install it yourself in debian/memcached/etc/init.d/. debhelper will ensure its proper integration.
  6. Instead, a user wanting to customize the options is expected to edit the unit with systemctl edit.
  7. See #822670
  8. The Debian Policy doesn t provide any hint for the naming convention of those system users. A common usage is to prefix the daemon name with an underscore (like _memcached). Another common usage is to use Debian- as a prefix. The main drawback of the latest solution is that the name is likely to be replaced by the UID in ps and top because of its length.
  9. We could call dh_auto_clean at the end of the target to let it invoke make clean. However, it is assumed that a fresh checkout is used before each build.

Russ Allbery: Review: The Girl with the Dragon Tattoo

Review: The Girl with the Dragon Tattoo, by Stieg Larsson
Translator: Reg Keeland
Series: Millennium #1
Publisher: Vintage Crime
Copyright: 2005, 2008
Printing: June 2009
ISBN: 0-307-47347-3
Format: Mass market
Pages: 644
As The Girl with the Dragon Tattoo opens, Mikael Blomkvist is losing a criminal libel suit in Swedish court. His magazine, Millennium, published his hard-hitting piece of investigative journalism that purported to reveal sketchy arms deals and financial crimes by Hans-Erik Wennerstr m, a major Swedish businessman. But the underlying evidence didn't hold up, and Blomkvist could offer no real defense at trial. The result is a short prison stint for him (postponed several months into this book) and serious economic danger for Millennium. Lisbeth Salander is a (very) freelance investigator for Milton Security. Her specialty is research and background checks: remarkably thorough, dispassionate, and comprehensive. She's almost impossible to talk to, tending to meet nearly all questions with stony silence, but Dragan Armansky, the CEO of Milton Security, has taken her partly under his wing. She, and Milton Security, were hired by a lawyer named Dirch Frode to do a comprehensive background check on Mikael Blomkvist, which she and Dragan present near the start of the book. The reason, as the reader discovers in a few more chapters, is that Frode's employer wants to offer Blomkvist a very strange job. Over forty years ago, Harriet Vanger, scion of one of Sweden's richest industrial families, disappeared. Her uncle, Henrik Vanger, has been obsessed with her disappearance ever since, but in forty years of investigation has never been able to discover what happened to her. There are some possibilities for how her body could have been transported off the island the Vangers (mostly) lived, and live, on, but motive and suspects are still complete unknowns. Vanger wants Blomkvist to try his hand under the cover of writing a book about the Vanger family. Payment is generous, but even more compelling is Henrik Vanger's offer to give Blomkvist documented, defensible evidence against Wennerstr m at the end of the year. The Girl with the Dragon Tattoo (the original Swedish title is M n som hatar kvinnor, "Men who hate women") is the first of three mystery novels written at the very end of Stieg Larsson's life, all published posthumously. They made quite a splash when they were published: won multiple awards, sold millions of copies, and have resulted in four movies to date. I've had a copy of the book sitting around for a while and finally picked it up when in the mood for something a bit different. A major disclaimer up front: I read very little crime and mystery fiction. Every genre has its own conventions and patterns, and regular genre readers often look for different things than people new to that genre. My review is from a somewhat outside and inexperienced perspective, which may not be useful for regular genre readers. I'm also a US reader, reading the book in translation. It appears to be a very good translation, but it was also quite obvious to me that The Girl with the Dragon Tattoo was written from a slightly different set of cultural assumptions than I brought to the book. This is one of the merits of reading books from other cultures in translation. It can be eye-opening, and can carry some of the same thrill as science fiction or fantasy, to hit the parts of the book that question your assumptions. But it can also be hard to tell whether some striking aspect of a book is due to a genre convention I wasn't familiar with, a Swedish cultural assumption that I don't share, or just the personal style of the author. A few things do leap out as cultural differences. Blomkvist has to spend a few months in prison in the middle of this book, and that entire experience is completely foreign to an American understanding of what prison is like. The degradation, violence, and awfulness that are synonymous with prison for an American are almost entirely absent. He even enjoys the experience as quiet time to focus on writing a history of the Vangers (Blomkvist early on decides to take his cover story seriously, since he doubts he'll make any inroads into the mystery of Harriet's disappearance but can at least get a book out of it). It's a minor element in the book, glossed over in a few pages, but it's certainly eye-opening for how minimum security prison could be structured in a civilized country. Similarly, as an American reader, I was struck by how hard Larsson has to work to ruin Salander's life. Although much of the book is written from Blomkvist's perspective (in tight third person), Lisbeth Salander is the titular girl with the dragon tattoo and becomes more and more involved in the story as it develops. The story Larsson wanted to tell requires that she be in a very precarious position legally and socially. In the US, this would be relatively easy, particularly for someone who acts like Salander does. In Sweden, Larsson has to go to monumental efforts to find ways for Salander to credibly fall through Sweden's comprehensive social safety net, and still mostly relies on Salander's complete refusal to assist or comply with any form of authority or support. I've read a lot about differences in policies around social support between the US and Scandinavian countries, but I've rarely read something that drove the point home more clearly than the amount of work a novelist has to go to in order to mess up their protagonist's life in Sweden. The actual plot is slow-moving and as much about the psychology of the characters as it is about the mystery. The reader gets inside the thoughts of the characters occasionally, but Larsson shows far more than tells and leaves it to the reader to draw more general conclusions. Blomkvist's relationship with his long-time partner and Millennium co-founder is an excellent example: so much is left unstated that I would have expected other books to lay down in black and white, and the characters seem surprisingly comfortable with ambiguity. (Some of this may be my genre unfamiliarity; SFF tends to be straightforward to a fault, and more literary fiction is more willing to embrace ambiguous relationships.) While the mystery of Harriet's disappearance forms the backbone of the story, rather more pages are spent on Blomkvist navigating the emotional waters of the near-collapse of his career and business, his principles around investigation and journalism, and the murky waters of the Vanger's deeply dysfunctional family. Harriet's disappearance is something of a locked room mystery. The day she disappeared, a huge crash closed the only bridge from the island to the mainland, both limiting suspects and raising significant questions about why her body was never found on the island. It's also forty years into the past, so Blomkvist has to rely on Henrik Vanger's obsessive archives, old photographs, and old police reports. I found the way it unfolded to be quite satisfying: there are just enough clues to let Blomkvist credibly untangle things with some hard work and research, but they're obscure enough to make it plausible that previous investigators missed them. Through most of this novel, I wasn't sure what I thought of it. I have a personal interest in Blomkvist's journalistic focus wrongdoing by rich financiers but I had trouble warming to Blomkvist himself. He's a very passive, inward character, who spends a lot of the early book reacting to things that are happening to him. Salander is more dynamic and honestly more likable, but she's also deeply messed up, self-destructive, and does some viciously awful things in this book. And the first half of the book is very slow: lots of long conversations, lots of character introduction, and lots of Blomkvist wandering somewhat aimlessly. It's only when Larsson gets the two protagonists together that I thought the book started to click. Salander sees Blomkvist's merits more clearly than the reader can, I think. I also need to give a substantial warning: The Girl with the Dragon Tattoo is a very violent novel, and a lot of that violence is sexual. By mid-book, Blomkvist realizes that Harriet's disappearance is somehow linked with a serial killer whose trademark is horrific, sexualized symbolism drawn from Leviticus. There is a lot of rape here, including revenge rape by a protagonist. If that sort of thing is likely to bother you, you may want to steer way clear. That said, despite the slow pace, the nauseating subject matter, the occasionally very questionable ethics of protagonists, and a twist of the knife at the very end of the novel that I thought was gratuitously nasty on Larsson's part and wasn't the conclusion I wanted, I found myself enjoying this. It has a different pace and a different flavor than what I normally read, the characters are deep and complex enough to play off each other in satisfying ways, and Salander is oddly compelling to read about. Given the length, it's a substantial investment of time, but I don't regret reading it, and I'm quite tempted to read the sequel. I'm not sure this is the sort of book I can recommend (or not recommend) given my lack of familiarity with the genre, but I think US readers might get an additional layer of enjoyment out of seeing how different of a slant the Swedish setting puts on some of the stock elements of a crime novel. Followed by The Girl Who Played with Fire. Rating: 7 out of 10

1 May 2016

Thorsten Alteholz: My Debian Activities in April 2016

FTP assistant This month I marked 171 packages for accept and rejected 42. I also sent 3 emails to maintainers asking questions. It seems to be that another quiet month is behind us. Nevertheless the flood of strange things in NEW continued this month. Hmm, weird world .. Debian LTS This was my twenty-second month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. This month my all in all workload had been 15.75h. After getting the permission of the security team I changed the temporary-issues to meanwhile assigned CVEs and uploaded fuseiso. This resulted in DSA 3551-1. I also prepared new packages for asterisk and asked for testers on the LTS mailing list. Luckily Gabriel Filion really tried these packages and found a regression with manager connections. Dear reader, the new packages are waiting for your tests now :-) . Further I used the upload of poppler (DLA 446-1) to test the workflow of the new wheezy-security upload. Uploading and building packages worked perfectly. Unfortunately the push to the security mirrors was a bit delayed (it only happened after an upload of the security team). But this seems to be fixed by Ansgar now. Last but not least I had a look at PHP5. I think I will start my regular uploads in May. Other stuff As I had to deal with non-Debian stuff this month, I didn t do lots of other things. I only uploaded node-uml

30 April 2016

Chris Lamb: Free software activities in April 2016

Here is my monthly update covering a large part of what I have been doing in the free software world (previously):
Debian My work in the Reproducible Builds project was covered in our weekly reports. (#48, #49, #50, #51 & #52)
Uploads
  • redis (2:3.0.7-3) Adding, amongst some other changes, systemd LimitNOFILE support to allow a higher number of open file descriptors.
FTP Team

As a Debian FTP assistant I ACCEPTed 135 packages: aptitude, asm, beagle, blends, btrfs-progs, camitk, cegui-mk2, cmor-tables, containerd, debian-science, debops, debops-playbooks, designate-dashboard, efitools, facedetect, flask-testing, fstl, ganeti-os-noop, gnupg, golang-fsnotify, golang-github-appc-goaci, golang-github-benbjohnson-tmpl, golang-github-dchest-safefile, golang-github-docker-go, golang-github-dylanmei-winrmtest, golang-github-hawkular-hawkular-client-go, golang-github-hlandau-degoutils, golang-github-hpcloud-tail, golang-github-klauspost-pgzip, golang-github-kyokomi-emoji, golang-github-masterminds-semver-dev, golang-github-masterminds-vcs-dev, golang-github-masterzen-xmlpath, golang-github-mitchellh-ioprogress, golang-github-smartystreets-assertions, golang-gopkg-hlandau-configurable.v1, golang-gopkg-hlandau-easyconfig.v1, golang-gopkg-hlandau-service.v2, golang-objx, golang-pty, golang-text, gpaste, gradle-plugin-protobuf, grip, haskell-brick, haskell-hledger-ui, haskell-lambdabot-haskell-plugins, haskell-text-zipper, haskell-werewolf, hkgerman, howdoi, jupyter-client, jupyter-core, letsencrypt.sh, libbpp-phyl, libbpp-raa, libbpp-seq, libbpp-seq-omics, libcbor-xs-perl, libdancer-plugin-email-perl, libdata-page-pageset-perl, libevt, libevtx, libgit-version-compare-perl, libgovirt, libmsiecf, libnet-ldap-server-test-perl, libpgobject-type-datetime-perl, libpgobject-type-json-perl, libpng1.6, librest-client-perl, libsecp256k1, libsmali-java, libtemplates-parser, libtest-requires-git-perl, libtext-xslate-perl, linux, linux-signed, mandelbulber2, netlib-java, nginx, node-rc, node-utml, nvidia-cuda-toolkit, openfst, openjdk-9, openssl, php-cache-integration-tests, pulseaudio, pyfr, pygccxml, pytest-runner, python-adventure, python-arrayfire, python-django-feincms, python-fastimport, python-fitsio, python-imagesize, python-lib389, python-libtrace, python-neovim-gui, python3-proselint, pythonpy, pyzo, r-cran-ca, r-cran-fitbitscraper, r-cran-goftest, r-cran-rnexml, r-cran-rprotobuf, rrdtool, ruby-proxifier, ruby-seamless-database-pool, ruby-syslog-logger, rustc, s5, sahara-dashboard, salt-formula-ceilometer, salt-formula-cinder, salt-formula-glance, salt-formula-heat, salt-formula-horizon, salt-formula-keystone, salt-formula-neutron, salt-formula-nova, seer, simplejson, smrtanalysis, tiles-autotag, tqdm, tran, trove-dashboard, vim, vulkan, xapian-bindings & xapian-core.

Daniel Stender: What I've worked on for Debian this month

This month I've worked on the following things for Debian: To begin with that, I've set up a Debhelper sequencer script for dh-buildinfo1, this add-on now can be used with dh $@ --with buildinfo in deb/rules instead of having to explicitly call it somewhere in an override. Debops I've set up initial Debian packages of Debops2, a collection of fine crafted Ansible roles and playbooks especially for Debian servers, shipped with a couple of convenience and wrapper scripts in Python3. There are two binary packages, one for the toolset (debops), and the other for the playbooks and roles of the project (debops-playbooks). The application is easy to use, just initialize a new project with debops-init foo and add your server(s) to foo/ansible/inventory/hosts belonging to groups representing services and things you want to employ on them. Like the group [debops_gitlab] automatically installs a complete running Gitlab setup on one or a multitude of servers in the same run with the debops command4. Use other groups like [debops_mariadb_server] accordingly in the same host inventory. Ansible runs agent less, so you don't have to prepare freshly setup servers with nothing special to use that tool randomly (like on localhost). The list of things you could deploy with Debops is quite amazing and you've got dozens of services at your hand. The new packages are currently in experimental because they need some more fine tuning, like there are a couple of minor error messages which recently occur using it, but it works well. The (early staged) documentation unfortunately couldn't be packaged because of the scattered resp. collective nature of the project (all parts have their own Github repositories)5, and also how to generate the upstream tarball remains a bit of a challenge (currently, it's the outcome of debops-init)6. I'll have this package in unstable soon. More info on Debops is coming up, then. Hashicorp's Packer I'm very glad to announce that Packer7 is ready being available in unstable, and the two year old RFP bug could be finally closed8. It's another great and much convenient devops tool which does a lot of different things in an automated fashion using only a single "one-argument" CLI tool in combination with a couple of lines in a configuration script (thanks to Yaroslav Halchenko for the tip). Packer helps creating machine images for different platforms. This is like when you use e.g. Debian installations in a Qemu box for testing or development purposes. Instead of setting up a new virtual machine manually like installing Debian on another computer this process could be automated with Packer, like I've written about in this blog entry here9. You just need a template containing instructions for the included Qemu-builder and a preseeding script for the Debian installer, and there you go drinking your coffee while Packer does all the work for you: downloading the installation ISO image, creating the new virtual harddrive, booting the emulator, running the whole installation process automatically like answering questions, selecting things, rebooting without ISO image to complete the installation etc. A couple of minutes and you have a new pre-baked virtual machine image like from a vendoring machine, a fresh one everytime you need it. Packer10 supports a number of builders for different target platforms (desktop virtualization solutions as much as public cloud providers and private cloud software), can build in parallel, and also the full range of common provisioners can be employed in the process to equip the newly installed OSs. Vagrant boxes could be generated by one of the included postprocessors. I'll write more on Packer here on this blog, soon. There were more then two dozens of packages missing to complete Packer11, which is the achievement of combined forces within the pkg-go group. Much thanks esp. to Alexandre Viau who have worked on the most of the needed new packages. Thanks also to the FTP-masters which were always very quick in reviewing the Go packages, so that it could be proceeded to build and package the sub dependent new ones always consecutively. Squirrel3 I've didn't had the most work with it and just sponsored this for Fabian Wolff, but want to highlight here that there's a new package of Squirrel12 now available in Debian13. Squirrel is a lightweight scripting language, somewhat comparable to Lua. It's fully object-oriented and highly embeddable, it's used in a lot of commerical computer games under the hood for implementing intelligence for bots next to other things14, but also for the Internet of Things (it's embedded in hardware from Electric Imp). Squirrel functions could be called from C++15. I've filed an ITP bug for Squirrel already in 2011 (#651195), but always something else got in the way, and it ended up being an RFP. I'm really glad that it got picked up and completed. misc There were a couple of uploads on updated upstream tarballs and for fixing bugs, namely afl/2.10b-1 and 2.11b-1, python-afl/0.5.3-1, pyutilib/5.3.2-1, pyomo/4.3.11327-1, libvigraimpex/1.10.0+git20160211.167be93dfsg-2 (fix of #820429, thanks for Tobias Frost), and gamera/3.4.2+svn1454-1. For the pkg-go group, I've set up a new package of github-mitchellh-ioprogress (which is needed by the official DigitalOcean CLI tool doctl, now RFP #807956 instead of ITP due to the lack of time - again facing a lot of missing packages), and provided a little patch for dh-make-golang updating some standards16. For Packer I've also updated azure-go-autorest and azure-sdk as team upload (#821938, #821832), but it came out that the project which is currently under heavy development towards a new official release broke a lot in the past weeks (and no Git branching have been used), so that Packer as a matter of fact needed a vendored snapshot, although there have been only a couple of commits in between. Docker-registry hat the same problem with the new package of azure-sdk/2.1.1~beta1, so that it needed to be fixed, too (#822146). By the way, the tool ratt17 comes very handy for automatically test building down all reverse dependencies, not only for Go packages (thanks to Tianon Gravi for the tip). Finally, I've posted the needed reverse depencies as RFP bugs for Terraform18 (again quite a lot), Vuls19, and cve-dictionary20, which is needed for Vuls. I'll let them rest a while waiting to get picked up before working anything down.
  1. #570933: dh-buildinfo: should provide a addon file for dh command
  2. https://tracker.debian.org/pkg/debops
  3. http://debops.org/
  4. The servers have to be accessible by SSH. E.g. you could run debops like: $ debops -u root --private-key=~/.ssh/id_digitalocean
  5. https://github.com/debops/docs/issues/132
  6. #819816: ITP: debops -- Ansible based server management utility
  7. https://www.packer.io/
  8. #740753: ITP: packer -- create vm images for multiple platforms
  9. http://www.danielstender.com/blog/packer-qemu.html
  10. https://packages.debian.org/unstable/packer
  11. I've worked on the missing packages this month, namely github-klauspost-pgzip, github-masterzen-xmlpath, github-masterzen-winrm, dylanmei-winrmtest, packer-community-winrmcp (Packer uses WinRM if Windows machines images are created), github-hpcloud-tail, and updated github-rackspace-gophercloud (#822163) and google-api (#822164) to complete it.
  12. http://squirrel-lang.org/
  13. https://tracker.debian.org/pkg/squirrel3
  14. http://www.linux-magazin.de/layout/set/print/content/view/full/62184
  15. http://www.linux-magazin.de/Ausgaben/2011/10/plus/Fremdkoerper-Squirrel-Interpreter-und-Skripte-fuer-C
  16. https://github.com/Debian/dh-make-golang/pull/39
  17. https://packages.debian.org/unstable/ratt
  18. #808940: ITP: terraform -- tool for managing cloud infrastructure
  19. #820614: ITP: vuls -- package inventory scanner for CVE vulnerabilities
  20. #820615: ITP: go-cve-dictionary -- builds a local copy of the NVD/JVN (vulnerability databases)

Daniel Stender: My work for Debian in April

This month I've worked on the these things for Debian: To begin with that, I've set up a Debhelper sequencer script for dh-buildinfo1, this add-on now can be used with dh $@ --with buildinfo in deb/rules instead of having to explicitly call it somewhere in an override. Debops I've set up initial Debian packages of Debops2, a collection of fine crafted Ansible roles and playbooks especially for Debian servers (servers which run on Debian), which are shipped with a couple of helper and wrapper scripts in Python3. There are two binary packages, one for the toolset (debops), and the other for the playbooks and roles of the project (debops-playbooks). The application is easy to use, just initialize a new project with debops-init foo and add your server(s) to foo/ansible/inventory/hosts belonging to groups representing services and things you want to employ on them. For example, the group [debops_gitlab] automatically installs a complete running Gitlab setup on one or a multitude of servers in the same run with the debops command4. Other groups like [debops_mariadb_server] could be used accordingly in the same host inventory. Ansible works without agent, so you don't have to prepare freshly setup servers with nothing special to use that tool randomly (like on localhost). The list of things you could deploy with Debops is quite amazing and dozens of services are at hand. The new Debian packages are currently in experimental because they need some more fine tuning, e.g. there are a couple of minor error messages which recently occur using it, but it works well. The (early staged) documentation unfortunately couldn't be packaged because of the scattered resp. collective nature of the project (all parts have their own Github repositories)5, and also how to generate the upstream tarball remains a bit of a challenge (currently, it's the outcome of debops-init)6. I'll have this package in unstable soon. More info on Debops is coming up, then. HashiCorp's Packer I'm very glad to announce that Packer7 is ready being available in unstable, and the RFP bug could be finally closed after I've taken it over8. It's another great and much convenient devops tool which does a lot of different things in an automated fashion using only a single "one-argument" CLI tool in combination with a couple of lines in a configuration script (thanks to Yaroslav Halchenko for the tip). Packer helps creating machine images for different platforms. This is like when you use e.g. Debian installations in a Qemu box for testing or development purposes. Instead of setting up a new virtual machine manually the same way as installing Debian on another computer this process can be completely automated with Packer, like I've written about in this blog entry here9. You just need a template which contains instructions for the included Qemu builder and a preseeding script for the Debian installer, and there you go drinking your coffee while Packer does all the work: download the ISO image for installation, create the new virtual harddrive, boot the emulator, run the whole installation process automatically like with answering questions, selecting things, reboot without ISO image to complete the installation etc. A couple of minutes and you have a new pre-baked virtual machine image like from a vendoring machine, another fresh one could be created anytime. Packer10 supports a number of builders for different target platforms (desktop virtualization solutions as much as public cloud providers and private cloud software), can build in parallel, and also the full range of common provisioners can be employed in the process to equip the newly installed OSs with services and programs. Vagrant boxes could be generated by one of the included postprocessors. I'll write more on Packer here on this blog, soon. There were more then two dozens of packages missing to complete Packer11, which is the achievement of combined forces within the pkg-go group. Much thanks esp. to Alexandre Viau who have worked on the most of the needed new packages. Thanks also to the FTP masters which were always very quick in reviewing the Go packages, so that it could be proceeded to build and package the sub dependent new ones always consecutively. Squirrel3 I've didn't had the major work of that and just sponsored this for Fabian Wolff, but want to highlight here that there's a new package of Squirrel12 now available in Debian13. Squirrel is a lightweight scripting language, somewhat comparable to Lua. It's fully object-oriented and highly embeddable, it's used in a lot of commerical computer games under the hood for implementing intelligence for bots next to other things14, but also for the Internet of Things (it's embedded in hardware from Electric Imp). Squirrel functions could be called from C++15. I've filed an ITP bug for Squirrel already in 2011 (#651195), but always something else had a higher priority, and it ended up being an RFP. I'm really glad that it got picked up and completed quickly afterwards. misc There were a couple of uploads on updated upstream tarballs and for fixing bugs, namely afl/2.10b-1 and 2.11b-1, python-afl/0.5.3-1, pyutilib/5.3.2-1, pyomo/4.3.11327-1, libvigraimpex/1.10.0+git20160211.167be93dfsg-2 (fix of #820429, thanks to Tobias Frost), and gamera/3.4.2+svn1454-1. For the pkg-go group, I've set up a new package of github-mitchellh-ioprogress (which is needed by the official DigitalOcean CLI tool doctl, now RFP #807956 instead of ITP due to the lack of time, again a lot of missing packages are missing for that), and provided a little patch for dh-make-golang updating some standards16. For Packer I've also updated azure-go-autorest and azure-sdk as team upload (#821938, #821832), but it came out that the project which is currently under heavy development towards a new official release broke a lot in the past weeks (no Git branching have been used), so that Packer as a matter of fact needed a vendored snapshot, although there have been only a couple of commits in between. Docker-registry has the same problem with the new package of azure-sdk/2.1.1~beta1, so that it needed to be fixed, too (#822146). By the way, the tool ratt17 comes very handy for automatically test building down all reverse dependencies, not only for Go packages (thanks to Tianon Gravi for the tip). Finally, I've posted the needed reverse depencies as RFP bugs for Terraform18 (again quite a lot), Vuls19, and cve-dictionary20, which is needed for Vuls. I'll let them rest a while waiting to get picked up before working anything down.
  1. #570933: dh-buildinfo: should provide a addon file for dh command
  2. https://tracker.debian.org/pkg/debops
  3. http://debops.org/
  4. The servers have to be accessible by SSH. E.g. you could run debops like: $ debops -u root --private-key=~/.ssh/id_digitalocean
  5. https://github.com/debops/docs/issues/132
  6. #819816: ITP: debops -- Ansible based server management utility
  7. https://www.packer.io/
  8. #740753: ITP: packer -- create vm images for multiple platforms
  9. http://www.danielstender.com/blog/packer-qemu.html
  10. https://packages.debian.org/unstable/packer
  11. I've worked on the missing packages this month, namely github-klauspost-pgzip, github-masterzen-xmlpath, github-masterzen-winrm, dylanmei-winrmtest, packer-community-winrmcp (Packer uses WinRM if Windows machines images are created), github-hpcloud-tail, and updated github-rackspace-gophercloud (#822163) and google-api (#822164) to complete it.
  12. http://squirrel-lang.org/
  13. https://tracker.debian.org/pkg/squirrel3
  14. http://www.linux-magazin.de/layout/set/print/content/view/full/62184
  15. http://www.linux-magazin.de/Ausgaben/2011/10/plus/Fremdkoerper-Squirrel-Interpreter-und-Skripte-fuer-C
  16. https://github.com/Debian/dh-make-golang/pull/39
  17. https://packages.debian.org/unstable/ratt
  18. #808940: ITP: terraform -- tool for managing cloud infrastructure
  19. #820614: ITP: vuls -- package inventory scanner for CVE vulnerabilities
  20. #820615: ITP: go-cve-dictionary -- builds a local copy of the NVD/JVN (vulnerability databases)

28 April 2016

Michal &#268;iha&#345;: Weblate 2.6

Going back to faster release cycle, Weblate 2.6 has been just released. There is improved support for Python 3 or brand new HTTP REST API. Full list of changes for 2.6: If you are upgrading from older version, please follow our upgrading instructions. You can find more information about Weblate on https://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Weblate is also being used https://hosted.weblate.org/ as official translating service for phpMyAdmin, OsmAnd, Aptoide, FreedomBox, Weblate itself and many other projects. Should you be looking for hosting of translations for your project, I'm happy to host them for you or help with setting it up on your infrastructure. Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far! The roadmap for next release is just being prepared, you can influence this by expressing support for individual issues either by comments or by providing bounty for them.

Filed under: English phpMyAdmin SUSE Weblate 0 comments

Dirk Eddelbuettel: RcppRedis 0.1.7

A new release of RcppRedis arrived on CRAN today. And just like for the previous release, Russell Pierce contributed a lot of changes via several pull requests which make for more robust operations. In addition, we have started to add support for MessagePack by relying on our recently-announced RcppMsgPack package.
Changes in version 0.1.7 (2016-04-27)
  • Added support for timeout constructor argument (PR #14 by Russell Pierce)
  • Added new commands exists, ltrim, expire and pexpire along with unit tests (PR #16 by Russell Pierce)
  • Return NULL for empty keys in serialized get for consistency with lpop and rpop (also PR #16 by Russell Pierce)
  • Minor corrections to get code and hget and hset documentation (also PR #16 by Russell Pierce)
  • Error conditions are now properly forwarded as R errors (PR #22 by Russell Pierce)
  • Results from Redis commands are now checked for NULL (PR #23 by Russell Pierce)
  • MessagePack encoding can now be used which requires MessagePackage headers of version 1.0 or later; the (optional) RcppMsgPack package can be used.
Courtesy of CRANberries, there is also a diffstat report for the most recent release. More information is on the RcppRedis page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Next.

Previous.