Search Results: "spa"

12 May 2016

Ingo Juergensmann: Xen randomly crashing server - part 2

Some weeks ago I blogged about "Xen randomly crashing server". The problem back then was that I couldn't get any information why the server reboots. Using a netconsole was not possible, because netconsole refused to work with the bridge that is used for Xen networking. Luckily my colocation partner rrbone.net connected the second network port of my server to the network so that I could use eth1 instead of the bridged eth0 for netconsole. Today the server crashed several times and I was able to collect some more information than just the screenshots from IPMI/KVM console as shown in my last blog entry (full netconsole output is attached as a file):

May 12 11:56:39 31.172.31.251 [829681.040596] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.16.0-4-amd64 #1 Debian 3.16.7-ckt25-2
May 12 11:56:39 31.172.31.251 [829681.040647] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.0a 01/03/2014
May 12 11:56:39 31.172.31.251 [829681.040701] task: ffffffff8181a460 ti: ffffffff81800000 task.ti: ffffffff81800000
May 12 11:56:39 31.172.31.251 [829681.040749] RIP: e030:[<ffffffff812b7e56>]
May 12 11:56:39 31.172.31.251 [<ffffffff812b7e56>] memcpy+0x6/0x110
May 12 11:56:39 31.172.31.251 [829681.040802] RSP: e02b:ffff880280e03a58 EFLAGS: 00010286
May 12 11:56:39 31.172.31.251 [829681.040834] RAX: ffff88026eec9070 RBX: ffff88023c8f6b00 RCX: 00000000000000ee
May 12 11:56:39 31.172.31.251 [829681.040880] RDX: 00000000000004a0 RSI: ffff88006cd1f000 RDI: ffff88026eec9422
May 12 11:56:39 31.172.31.251 [829681.040927] RBP: ffff880280e03b38 R08: 00000000000006c0 R09: ffff88026eec9062
May 12 11:56:39 31.172.31.251 [829681.040973] R10: 0100000000000000 R11: 00000000af9a2116 R12: ffff88023f440d00
May 12 11:56:39 31.172.31.251 [829681.041020] R13: ffff88006cd1ec66 R14: ffff88025dcf1cc0 R15: 00000000000004a8
May 12 11:56:39 31.172.31.251 [829681.041075] FS: 0000000000000000(0000) GS:ffff880280e00000(0000) knlGS:ffff880280e00000
May 12 11:56:39 31.172.31.251 [829681.041124] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
May 12 11:56:39 31.172.31.251 [829681.041153] CR2: ffff88006cd1f000 CR3: 0000000271ae8000 CR4: 0000000000042660
May 12 11:56:39 31.172.31.251 [829681.041202] Stack:
May 12 11:56:39 31.172.31.251 [829681.041225] ffffffff814d38ff
May 12 11:56:39 31.172.31.251 ffff88025b5fa400
May 12 11:56:39 31.172.31.251 ffff880280e03aa8
May 12 11:56:39 31.172.31.251 9401294600a7012a
May 12 11:56:39 31.172.31.251
May 12 11:56:39 31.172.31.251 [829681.041287] 0100000000000000
May 12 11:56:39 31.172.31.251 ffffffff814a000a
May 12 11:56:39 31.172.31.251 000000008181a460
May 12 11:56:39 31.172.31.251 00000000000080fe
May 12 11:56:39 31.172.31.251
May 12 11:56:39 31.172.31.251 [829681.041346] 1ad902feff7ac40e
May 12 11:56:39 31.172.31.251 ffff88006c5fd980
May 12 11:56:39 31.172.31.251 ffff224afc3e1600
May 12 11:56:39 31.172.31.251 ffff88023f440d00
May 12 11:56:39 31.172.31.251
May 12 11:56:39 31.172.31.251 [829681.041407] Call Trace:
May 12 11:56:39 31.172.31.251 [829681.041435] <IRQ>
May 12 11:56:39 31.172.31.251
May 12 11:56:39 31.172.31.251 [829681.041441]
May 12 11:56:39 31.172.31.251 [<ffffffff814d38ff>] ? ndisc_send_redirect+0x3bf/0x410
May 12 11:56:39 31.172.31.251 [829681.041506] [<ffffffff814a000a>] ? ipmr_device_event+0x7a/0xd0
May 12 11:56:39 31.172.31.251 [829681.041548] [<ffffffff814bc74c>] ? ip6_forward+0x71c/0x850
May 12 11:56:39 31.172.31.251 [829681.041585] [<ffffffff814c9e54>] ? ip6_route_input+0xa4/0xd0
May 12 11:56:39 31.172.31.251 [829681.041621] [<ffffffff8141f1a3>] ? __netif_receive_skb_core+0x543/0x750
May 12 11:56:39 31.172.31.251 [829681.041729] [<ffffffff8141f42f>] ? netif_receive_skb_internal+0x1f/0x80
May 12 11:56:39 31.172.31.251 [829681.041771] [<ffffffffa0585eb2>] ? br_handle_frame_finish+0x1c2/0x3c0 [bridge]
May 12 11:56:39 31.172.31.251 [829681.041821] [<ffffffffa058c757>] ? br_nf_pre_routing_finish_ipv6+0xc7/0x160 [bridge]
May 12 11:56:39 31.172.31.251 [829681.041872] [<ffffffffa058d0e2>] ? br_nf_pre_routing+0x562/0x630 [bridge]
May 12 11:56:39 31.172.31.251 [829681.041907] [<ffffffffa0585cf0>] ? br_handle_local_finish+0x80/0x80 [bridge]
May 12 11:56:39 31.172.31.251 [829681.041955] [<ffffffff8144fb65>] ? nf_iterate+0x65/0xa0
May 12 11:56:39 31.172.31.251 [829681.041987] [<ffffffffa0585cf0>] ? br_handle_local_finish+0x80/0x80 [bridge]
May 12 11:56:39 31.172.31.251 [829681.042035] [<ffffffff8144fc16>] ? nf_hook_slow+0x76/0x130
May 12 11:56:39 31.172.31.251 [829681.042067] [<ffffffffa0585cf0>] ? br_handle_local_finish+0x80/0x80 [bridge]
May 12 11:56:39 31.172.31.251 [829681.042116] [<ffffffffa0586220>] ? br_handle_frame+0x170/0x240 [bridge]
May 12 11:56:39 31.172.31.251 [829681.042148] [<ffffffff8141ee24>] ? __netif_receive_skb_core+0x1c4/0x750
May 12 11:56:39 31.172.31.251 [829681.042185] [<ffffffff81009f9c>] ? xen_clocksource_get_cycles+0x1c/0x20
May 12 11:56:39 31.172.31.251 [829681.042217] [<ffffffff8141f42f>] ? netif_receive_skb_internal+0x1f/0x80
May 12 11:56:39 31.172.31.251 [829681.042251] [<ffffffffa063f50f>] ? xenvif_tx_action+0x49f/0x920 [xen_netback]
May 12 11:56:39 31.172.31.251 [829681.042299] [<ffffffffa06422f8>] ? xenvif_poll+0x28/0x70 [xen_netback]
May 12 11:56:39 31.172.31.251 [829681.042331] [<ffffffff8141f7b0>] ? net_rx_action+0x140/0x240
May 12 11:56:39 31.172.31.251 [829681.042367] [<ffffffff8106c6a1>] ? __do_softirq+0xf1/0x290
May 12 11:56:39 31.172.31.251 [829681.042397] [<ffffffff8106ca75>] ? irq_exit+0x95/0xa0
May 12 11:56:39 31.172.31.251 [829681.042432] [<ffffffff8135a285>] ? xen_evtchn_do_upcall+0x35/0x50
May 12 11:56:39 31.172.31.251 [829681.042469] [<ffffffff8151669e>] ? xen_do_hypervisor_callback+0x1e/0x30
May 12 11:56:39 31.172.31.251 [829681.042499] <EOI>
May 12 11:56:39 31.172.31.251
May 12 11:56:39 31.172.31.251 [829681.042506]
May 12 11:56:39 31.172.31.251 [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
May 12 11:56:39 31.172.31.251 [829681.042561] [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
May 12 11:56:39 31.172.31.251 [829681.042592] [<ffffffff81009e7c>] ? xen_safe_halt+0xc/0x20
May 12 11:56:39 31.172.31.251 [829681.042627] [<ffffffff8101c8c9>] ? default_idle+0x19/0xb0
May 12 11:56:39 31.172.31.251 [829681.042666] [<ffffffff810a83e0>] ? cpu_startup_entry+0x340/0x400
May 12 11:56:39 31.172.31.251 [829681.042705] [<ffffffff81903076>] ? start_kernel+0x497/0x4a2
May 12 11:56:39 31.172.31.251 [829681.042735] [<ffffffff81902a04>] ? set_init_arg+0x4e/0x4e
May 12 11:56:39 31.172.31.251 [829681.042767] [<ffffffff81904f69>] ? xen_start_kernel+0x569/0x573
May 12 11:56:39 31.172.31.251 [829681.042797] Code:
May 12 11:56:39 31.172.31.251 <f3>
May 12 11:56:39 31.172.31.251
May 12 11:56:39 31.172.31.251 [829681.043113] RIP
May 12 11:56:39 31.172.31.251 [<ffffffff812b7e56>] memcpy+0x6/0x110
May 12 11:56:39 31.172.31.251 [829681.043145] RSP <ffff880280e03a58>
May 12 11:56:39 31.172.31.251 [829681.043170] CR2: ffff88006cd1f000
May 12 11:56:39 31.172.31.251 [829681.043488] ---[ end trace 1838cb62fe32daad ]---
May 12 11:56:39 31.172.31.251 [829681.048905] Kernel panic - not syncing: Fatal exception in interrupt
May 12 11:56:39 31.172.31.251 [829681.048978] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff)

I'm not that good at reading this kind of output, but to me it seems that ndisc_send_redirect is at fault. When googling for "ndisc_send_redirect" you can find a patch on lkml.org and Debian bug #804079, both seem to be related to IPv6. When looking at the linux kernel source mentioned in the lkml patch I see that this patch is already applied (line 1510):

if (ha)
ndisc_fill_addr_option(buff, ND_OPT_TARGET_LL_ADDR, ha);

So, when the patch was intended to prevent "leading to data corruption or in the worst case a panic when the skb_put failed" it does not help in my case or in the case of #804079. Any tips are appreciated! PS: I'll contribute to that bug in the BTS, of course!

Attachment	Size
syslog-xen-crash.txt	24.27 KB

Kategorie:

Tags:

11 May 2016

Julian Andres Klode: Backing up with borg and git-annex

I recently found out that I have access to a 1 TB cloud storage drive by 1&1, so I decided to start taking off-site backups of my $HOME (well, backups at all, previously I only mirrored the latest version from my SSD to an HDD). I initially tried obnam. Obnam seems like a good tool, but is insanely slow. Unencrypted it can write about 3 MB/s, which is somewhat OK, but even then it can spend hours forgetting generations (1 generation takes probably 2 minutes, and there might be 22 of them). In encrypted mode, the speed reduces a lot, to about 500 KB/s if I recall correctly, which is just unusable. I found borg backup, a fork of attic. Borg backup achieves speeds of up to 15 MB/s which is really nice. It s also faster with scanning: I can now run my bihourly backups in about 1 min 30s (usually backs up about 30 to 80 MB mostly thanks to Chrome I suppose!). And all those speeds are with encryption turned on. Both borg and obnam use some form of chunks from which they compose files. Obnam stores each chunk in its own file, borg stores multiple chunks (even from different files) in a single pack file which is probably the main reason it is faster. So how am I backing up: My laptop has an internal SSD and an HDD. I backup every 2 hours (at 09,11,13,15,17,19,21,23,01:00 hours) using a systemd timer event, from the SSD to the HDD. The backup includes all of $HOME except for Downloads, .cache, the trash, Android SDK, and the eclipse and IntelliJ IDEA IDEs. Now the magic comes in: The backup repository on the HDD is monitored by git-annex assistant, which automatically encrypts and uploads any new files in there to my 1&1 WebDAV drive and registers them in a git repository hosted on bitbucket. All files are encrypted and checksummed using SHA256, reducing the chance of the backup being corrupted. I m not sure how the WebDAV thing will work once I want to prune things, I suspect it will then delete some pack files and repack things into new files which means it will spend more bandwidth than obnam would. I d also have to convince git-annex to actually drop anything from the WebDAV remote, but that is not really that much of a concern with 1TB storage space in the next 2 years at least I also have an external encrypted HDD which I can take backups on, it currently houses a fuller backup of $HOME that also includes Downloads, the Android SDK, and the IDEs for quicker recovery. Downloads changes a lot, and all of them can be fairly easily re-retrieved from the internet as needed, so there s not much point in painfully uploading them to a WebDAV backup site.
Filed under: Uncategorized

10 May 2016

Lars Wirzenius: Qvarn Platform announcement

In March we started a new company, to develop and support the software whose development I led at my previous job. The software is Qvarn, and it's fully free software, licensed under AGPL3+. The company is QvarnLabs (no website yet). Our plan is to earn a living from this, and our hope is to provide software that is actually useful for helping various organisations handle data securely. The first press release about Qvarn was sent out today. We're still setting up the company and getting operational, but a little publicity never hurts. (Even if it is more marketing-speak and self-promotion than I would normally put on my blog.) So this is what I do for a living now.

The development of the open source Qvarn Platform was led by QvarnLabs CEO Kaius H ggblom (left) and CTO Lars Wirzenius. Helsinki, Finland 10.05.2016 With Privacy by Design, integrated Gluu access management and comprehensive support for regulatory data compliance, Qvarn is set to become the Europe-wide platform of choice for managing workforce identities and providing associated value-added services. Construction industry federations in Sweden, Finland and the Baltic States have been using the Qvarn Platform (http://www.qvarn.org) since October 2015 to securely manage the professional digital identities of close to one million construction workers. Developed on behalf of these same federations, Qvarn is now free and open source software; making it a compelling solution for any organization that needs to manage a secure register of workers data. "There is something universal and fundamental at the core of the Qvarn platform. And that s trust," said Qvarn evangelist Kaius H ggblom. "We decided to make it free, open source and include Gluu access management because we wanted all those using Qvarn or contributing to its continued development to have the freedom to work with the platform in whatever way is best for them." Qvarn has been designed to meet the requirements of the European Union s new General Data Protection Regulation (GDPR), enabling organizations that use the platform to ensure their compliance with the new law. Qvarn has also incorporated the principles of Privacy by Design to minimize the disclosure of non-essential personal information and to give people more control over their data. "Today, Qvarn is used by the construction industry as a way to manage the data of employees, many of whom frequently move across borders. In this way the platform helps to combat the grey economy in the building sector, thereby improving quality and safety, while simultaneously protecting the professional identity data of almost a million individuals," said H ggblom. "Qvarn is so flexible and secure that we envision it becoming the preferred platform for the provision of any value-added services with an identity management component, eventually even supporting monetary transactions." Qvarn is a cloud based solution supported to run on both Amazon Web Services (AWS) and OpenStack. In partnership with Gluu, the platform delivers an out-of-the-box solution that uses open and standard protocols to provide powerful yet flexible identity and access management, including mechanisms for appropriate authentication and authorization. "Qvarn's identity management and governance capabilities perfectly compliment the Gluu Server's access management features," said Founder and CEO of Gluu, Michael Schwartz. "Free open source software (FOSS) is essential to the future of identity and access management. And the FOSS development methodology provides the transparency that is needed to foster the strong sense of community upon which a vibrant ecosystem thrives." Qvarn s development team continues to be led by recognized open source developer and platform architect Lars Wirzenius. He has been developing free and open source software for 30 years and is a renowned expert in the Linux environment, with a particular focus on the Debian distribution. Lars works at all levels of software development from writing code to designing system architecture. About the Qvarn Platform: The Qvarn Platform is free and open source software for managing workforce identities. Qvarn is integrated with the Gluu Server s access management features out of the box, using open and standard protocols to provide the platform with a single common digital identity and mechanisms for appropriate authentication and authorization. A cloud based solution, Qvarn is supported to run on both Amazon Web Services (AWS) and OpenStack. Privacy by Design is central to the architecture of Qvarn and the platform has been third party audited to a security level of HIGH. http://www.qvarn.org For more information, please contact:
Andrew Flowers
andrew.flowers@ellisnichol.com
+358 40 161 5668

9 May 2016

Arturo Borrero Gonz lez: Talk about contributing to FLOSS

The 26th of April I gave a talk in the University of Seville (ETSII) about contributing to FLOS software, focusing in the main projects I contribute to: Debian and Netfilter.

The talk was hosted by SUGUS, which is the university local group of FLOSS users.
The public were other students of the university, all of them young (like me), so the talk was very relaxed and formalities-free :-)

I talked my experiences in contributing to FLOSS, type of projects, how to start and how I integrate this with my full-time job.

Here is a video recording of the talk (in spanish):

I gave a similar talk some months ago to the students of the IES Gonzalo Nazareno.

Mike Gabriel: Recent progress in NXv3 development

This is to give a comprehensive overview on the recent progress made in NXv3 (aka nx-libs) development. The upstream sources of nx-libs can be found at / viewed on / cloned from Github:
https://github.com/ArcticaProject/nx-libs A great portion of the current work is sponsored by the Qindel Group [1] in Spain (QINDEL FORMACI N Y SERVICIOS, S.L.). Thanks for making this possible. Planned release date: 2nd July, 2016 We aim at releasing a completely tidied up nx-libs code tree versioned 3.6.0 on July 2nd, 2016. There is still a whole bunch of work to do for this, but I am positive that we can make this release date. Goals of our Efforts There are basically two major goals for spending a considerable amount of time, money and energy on NXv3 hacking:

make this beast long-term maintainable
make it work with latest X11 desktop environments and applications

The efforts undertaken always have the various existing use cases in mind (esp. the framework of the coming-up Arctica Project, TheQVD and X2Go). Overview on Recent Development Progress General Code Cleanups Making this beast maintainable means first of all: identifying code redundancies, unused code passages, etc. and remove them. This is where we came from (NoMachine's NX 3.5.x, including nxcomp, nxcompext, nxcompshad, nx-X11 and nxagent): 1,757,743 lines of C/C++ code.

[mike@minobo nx-libs.35 (3.5.0.x)]$ cloc --match-f '.*\.(c cpp h)$' .
    5624 text files.
    5614 unique files.                                          
    2701 files ignored.
http://cloc.sourceforge.net v 1.60  T=18.59 s (302.0 files/s, 132847.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C                             3134         231180         252893        1326393
C/C++ Header                  2274          78062         116132         349743
C++                            206          20037          13312          81607
-------------------------------------------------------------------------------
SUM:                          5614         329279         382337        1757743
-------------------------------------------------------------------------------

On the current 3.6.x branch of nx-libs (at commit 6c6b6b9), this is where we are now: 662,635 lines of C/C++ code, amount of code reduced to a third of the original code lines.

[mike@minobo nx-libs (3.6.x)]$ cloc --match-f '.*\.(c cpp h)' .
    2012 text files.
    2011 unique files.                                          
    1898 files ignored.
http://cloc.sourceforge.net v 1.60  T=5.63 s (341.5 files/s, 161351.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C                             1015          74605          81625         463244
C/C++ Header                   785          26992          34354         138063
C++                            122          16984          10804          61328
-------------------------------------------------------------------------------
SUM:                          1922         118581         126783         662635
-------------------------------------------------------------------------------

The latest development branch currently has these statistics: 619,353 lines of C/C++ code, another 40,000 lines could be dropped.

[mike@minobo nx-libs (pr/libnx-xext-drop-unused-extensions)]$ cloc --match-f '.*\.(c cpp h)' .
    1932 text files.
    1931 unique files.                                          
    1898 files ignored.
http://cloc.sourceforge.net v 1.60  T=5.66 s (325.4 files/s, 150598.1 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C                              983          69474          77186         426564
C/C++ Header                   738          25616          33048         131599
C++                            121          16984          10802          61190
-------------------------------------------------------------------------------
SUM:                          1842         112074         121036         619353
-------------------------------------------------------------------------------

Dropping various libNX_X* shared libraries (and using X.org shared client libraries instead) At first, various bundled libraries could be dropped from the nx-X11 code tree. Secondly, several of the bundled X.org libraries could be dropped, because we managed to build against those libraries as provided system-wide. Then, and this undertaking is much trickier, we could drop nearly all Xlib extension libraries that are used by nxagent with its role of being an X11 client. We could sucessfully drop these Xlib extension libraries from nx-X11, because we managed to build nxagent against the matching libraries in X.org: libNX_Xdmcp, libNX_Xfixes, libNX_XComposite, libNX_Xdamage, libNX_Xtst, libNX_Xinerama, libNX_Xfont, libNX_xkbui, and various others. All these droppings happened without a loss of functionality. However, some shared X client libraries are not easy to remove without loss of functionality, or rather not removable at all. Dropping libNX_Xrender We recently dropped libNX_Xrender [2] and now build nxagent against X.org's libXrender. However, this cost us a compression feature in NX. The libNX_Xrender code has passages that do zero padding of the unused memory portions in non-32bit-depth glyphs (the NX_RENDER_CLEANUP feature). However, we have hope for being able to reintroduce that feature again later, but current efforts [3] still fail at run-time. Dropping libNX_Xext is not possible... ...the libNX_Xext / Xserver Xext code has been cleaned up instead. Quite an amount of research and testing has been spent on removing the libNX_Xext library from the build workflow of nxagent. However, it seems that building against X.org's libXext will require us to update the Xext part of the nxagent Xserver at the same time. While taking this deep look into Xext code, we dropped various Xext extensions from the nx-X11 Xserver code. The extensions that got dropped [5] are all extensions that already have been dropped from X.org's Xserver code, as well. Further investigation, however, showed, that actually the only used client side extension code from libNX_Xext is the XShape extension. Thus, all other client side extension got dropped now in a very recent pull request [4]. Dropping libNX_X11 not easy, research summary given here For the sake of dropping the Xlib library bundled with nx-libs, we have attempted at writing a shared library called libNX_Xproxy. This library is supposed to contain a subset of the NXtrans related Xlib code that NoMachine patched into X.org's libX11 (and libxtrans). Results of this undertaking [6] so far:

We managed to build nxagent against Xlib from X.org
Local nxagent sessions (using the normal X11 transport) worked and seemed to perform better than sessions running under the original Xlib library (libNX_X11) bundled with nx-libs.
NXtrans support in libNX_Xproxy is half implemented but far from being finished, yet. The referenced branch [6] is a work-in-progress branch. Don't expect it to work. Expect force-pushes no that branch, too, please.

Over the weekend, I thought this all through once more and I am pretty sure right now, that we can actually make libNX_Xproxy work and drop all of libNX_X11 from nx-libs soon. Although we have to port (i.e. copy) various functions related to the NX transport from libNX_X11 into libNX_Xproxy, this change will allow us to drop all Xlib drawing routines and use those provided by X.org's Xlib shared library directly. Composite extension backport in nxagent to version 0.4 Mihai Moldovan looked at what it needs to make the Composite extension functional in nxagent. Unfortunately, the exact answer cannot be given, yet. As a start, Mihai backported latest Composite extension (v0.4) code from X.org's Xserver into the nxagent Xserver [7]. The currently shipped Composite extension version in nxagent is v0.2. Work on the NX Compression shared library (aka nxcomp v3) Fernando Carvajal and Salvador Fandi o from the Qindel Group [1] filed three pull requests against the nxcomp part of nx-libs recently, two of them have been code cleanups (done by Fernando), the third one is a feature enhancement regarding channels in nxcomp (provided by Salva). Protocol clean-up: drop pre-3.5 support With the release of nx-libs 3.6.x, we will drop support for nxcomp versions earlier than 3.5. Thus, if you are still on nxcomp 3.4, be prepared for upgrading at least to version 3.5.x. The code removal had been issued as pull request #111 ("Remove compatibility code for nxcomp before 3.5.0") [8]. The PR has already been reviewed and merged. Fernando filed another code cleanup PR (#119 [9]) against nx-libs that also already got merged into the 3.6.x branch. UNIX Socket Support for Channels The nxcomp library (and thus, nxproxy) provides a feature called "channels". Channels in nxcomp can be used for forwarding traffic between NX client side and the NX server side (along the graphical X11 transport that nxcomp is designed for). Until version 3.5.x, nxcomp was only able to use local TCP sockets for providing / connecting to channel endpoints. We consider local TCP sockets as insecure and aim at adding UNIX file socket support to nxcomp whereever a connection is established. Salva provided a patch against nxcomp that provides UNIX socket support to channel endpoints. The initial use case for this patch is: connect to client side pulseaudio socket file and avoid enabling the TCP listening socket in pulseaudio. The traffic then is channeled to the server side, so that pulse clients can connect to a UNIX socket file rather than to a local TCP port. The channel support patch has already been reviewed and merged into the 3.6.x branch of nx-libs. Rebasing nxagent against latest X.org Ulrich Sibiller spent a considerable amount of time and energy on providing a build chain that allows building nxagent against a modularized X.org 7.0 (rather than against the monolithic build tree of X.org 6.9, like we still do in nx-libs 3.6.x). We plan to adapt and minimize this build workflow for nx-libs 3.7.x (scheduled for summer 2017). A short howto that shows how to build nxagent with that new workflow will be posted on this blog within the next days. So stay tuned. Further work accomplished Quite a lot of code cleanup PRs have been filed by myself against nx-libs. Most of them target at removal of unnecessary code from the nx-X11 Xserver code base and the nxagent DDX:

Amend all issues generating compiler warnings in progams/Xserver/hw/nxagent/. (PR #102 [10], not yet merged).
nx-X11's Xserver: Drop outdated Xserver extensions. (PR #106 [11], not yet merged).
nxagent: drop duplicate Xserver code and include original Xserver code in nxagent code files. (PR #120 [12], not yet merged).
libNX_Xext: Drop unused extensions. (PR #121 [4], not yet merged).

The third one (PR #120) in the list requires some detailled explanation: We discovered that nxagent ships overrides some symbols from the original Xserver code base. These overrides are induced by copies of some files from some Xserver sub-directory placed into the hw/nxagent/ DDX path. All those files' names match the pattern NX*.c. These copies of code are done in a way that the C compiler suppresses throwing its 'symbol "" redefined: first defined in ""; redefined in ""' errors. The approach taken, however, requires to have quite a few 10.000 lines of redundant code in hw/nxagent/NX*.c that also gets shipped in some Xserver sub-directory (mostly dix/ and render/). With pull request #120, we have identified all code passages in hw/nxagent/NX*.c that must be considered as NX'ish. We differentiated the NX'ish functions from functions that never got changed by NoMachine when developing nxagent. I then came up with four different approaches ([13,14,15,16]) of dropping redundant code from those hw/nxagent/NX*.c files. We (Mihai Moldovan and myself) discussed the various approaches and favoured the disable-Xserver-code-and-include-into-NX*.c variant [14] over the others for the following reasons:

It requires the least invasive change in Xserver code files.
It pulls in Xserver code (MIT/X11) into nxagent code (GPL-2) at build time (vs. [13]).
It does not require weakening symbols, static symbols can stay static symbols [15].
We don't have to use the hacky interception approach as shown in [16].

In the long run, the Xserver portion of the patches provided via this pull request #120 are required to be upstreamed into X.org's Xserver. The discussion around this will be started when we fully dive into rebasing nxagent's Xserver code base against latest X.org Xserver. Tasks ahead before the 3.6.x Release Various tasks we face before 3.6.x can be released. Here is a probably incomplete list:

Drop libNX_X11 from nx-libs (2nd attempt)
Drop libNX_Xau from nx-libs
Fully fix the Composite extension in nxagent (hopefully)
Several QA pull requests to close several of the open issues
Upgrade XRandR extension in the nxagent Xserver to 1.4
Attempt fixing KDE5 launch-up inside nxagent and recent GNOME application failures (due to missing Xinput2 extension in nxagent)
More UNIX file socket support in nxcomp
Fix reparenting in nxagent
Make nxagent run smoothly in x11vnc when suspended

Tasks ahead after the 3.6.x Release (i.e., for 3.7.x) Here is an even rougher and probably highly incomplete list for tasks after the 3.6.x release:

Rename nx-libs to a new name that does not remind us of the original authoring company (NoMachine) that much.
Generalize the channel support in nxcomp (make it possible to fire-up an arbitrary amount of channels with TCP and/or UNIX file socket endpoints.
Proceed with the X.org Rebasing Effort.

Credits Some people have to be named here that give their heart and love to this project. Thank you guys for supporting the development efforts around nx-libs and the Arctica Project: Thanks to Nico Arenas Alonso from and on behalf of the Qindel Group for coordinating the current funding project around nx-libs. Thanks to Ulrich Sibiller for giving a great amount of spare time to working on the nxagent-rebase-against-X.org effort. Thanks to Mihai Moldovan for doing endless code reviews and being available for contracted work via BAUR-ITCS UG [17] on NXv3, as well. Thanks to Mario Becroft for providing a patch that allows us to hook into nxagent X11 sessions with VNC and have the session fully available over VNC while the NX transport is in suspended state. Also Mario is pouring some fancy UDP ideas into the re-invention of remote desktop computing process performed in the Arctica Project. Mario has been an NX supporter for years, I am glad to have him still around after so many years (although he was close to abandoning NX usage at least once). Thanks to Fernando Carvajal from Qindel (until April 2016) for cleaning up nxcomp code. Thanks to Orion Poplawski from the Fedora Project for working on the first bundled libraries removal patches and being a resource on RPM packaging. Thanks to my friend Lee for working behind the scenes on the Arctica Core code and constantly pouring various of his ideas into my head. Thanks for regularly reminding me on benchmarking things.
Folks, thanks to all of you for all your various efforts on this huge beast of software. You are a great resource of expertise and it's a pleasure and honour working with you all.
New Faces Last but not least, I'd like to let everyone know that the Qindel Group sponsors another developer joining in on NXv3 development: Vadim Troshchinskiy (aka vatral on Github). Vadim has worked on NXv3 before and we are looking forward to having him and his skills in the team soon (probably end of May 2016). Welcome on board of the team, Vadim.
[1] http://www.qindel.com
[2] https://github.com/ArcticaProject/nx-libs/pull/93
[3] https://github.com/sunweaver/nx-libs/commit/be41bde7efc46582b442706dfb85...
[4] https://github.com/ArcticaProject/nx-libs/pull/121
[5] https://github.com/ArcticaProject/nx-libs/pull/106
[6] https://github.com/sunweaver/nx-libs/tree/wip/libnx-x11-full-removal
[7] https://github.com/sunweaver/nx-libs/tree/pr/composite-0_4
[8] https://github.com/ArcticaProject/nx-libs/pull/111
[9] https://github.com/ArcticaProject/nx-libs/pull/119
[10] https://github.com/ArcticaProject/nx-libs/pull/102
[11] https://github.com/ArcticaProject/nx-libs/pull/106
[12] https://github.com/ArcticaProject/nx-libs/pull/120
[13] https://github.com/sunweaver/nx-libs/commit/9692e6a7045b3ab5cb0daaed187e... (include NX'ish code into Xserver)
[14] https://github.com/sunweaver/nx-libs/commit/3d359bfc2b6d021c1ae9c6e19e96... (include Xserver code into NX*.c)
[15] https://github.com/sunweaver/nx-libs/commit/af72ee5624a15d21c610528e37b6... (use weak symbols and non-static symbols)
[16] https://github.com/sunweaver/nx-libs/commit/7205bb8848c49ee3e78a82fde906... (override symbols with interceptions)
[17] http://www.baur-itcs.de/20-x2go/20-x2gosupport/

Russell Coker: BIND Configuration Files

I ve recently been setting up more monitoring etc to increase the reliability of servers I run. One ongoing issue with computer reliability is any case where a person enters the same data in multiple locations, often people make mistakes and enter slightly different data which can give bad results. For DNS you need to have at least 2 authoritative servers for each zone. I ve written the below Makefile to extract the zone names from the primary server and generate a config file suitable for use on a secondary server. The next step is to automate this further by having the Makefile copy the config file to secondary servers and run rndc reload . Note that in a typical Debian configuration any user in group bind can write to BIND config files and reload the server configuration so this can be done without granting the script on the primary server root access on the secondary servers. My blog replaces the TAB character with 8 spaces, you need to fix this up if you want to run the Makefile on your own system and also replace 10.10.10.10 with the IP address of your primary server. all: other/secondary.conf other/secondary.conf: named.conf.local Makefile
for n in $$(grep ^zone named.conf.local cut -f2 -d\" sort) ; do echo "zone \"$$n\" \n type slave;\n file \"$$n\";\n masters 10.10.10.10; ;\n ;\n" ; done > other/secondary.conf

Russ Allbery: BookRiot's SF/F by Female Authors

A list by Nikki Steele of the 100 best SF and fantasy novels by female authors (in her subjective take), published on BookRiot, has been making the rounds, with people noting which of those they've read. These crop up from time to time, and I've always been tempted to do the work to track the list over time the way that I track award winners. I had some free time this afternoon, so went ahead and set that up (although I badly need to refactor or rewrite a lot of my review posting code). An extra advantage is that I can publish the list as a separate web page so that I don't spam RSS readers with a huge list. The list, annotated with ratings and reviews where I've read the books, is under the reviews section of my web site. As I have time, I may add more lists. I'm also (slowly) working on the project of adding all the nominees for major awards and annotating those, since often the short lists contain a lot of interesting material too. One big caveat for this list: Steele only lists the first book of series, and in many cases the first book isn't very good and is far from the best of the series. So you'll see some anomalous low ratings here for first books that improve later on.

7 May 2016

Craig Small: Displaying Linux Memory

Memory management is hard, but RAM management may be even harder. Most people know the vague overall concept of how memory usage is displayed within Linux. You have your total memory which is everything inside the box; then there is used and free which is what the system is or is not using respectively. Some people might know that not all used is used and some of it actually is free. It can be very confusing to understand, even for a someone who maintains procps (the package that contains top and free, two programs that display memory usage). So, how does the memory display work? What free shows The free program is part of the procps package. It s central goal is to give a quick overview of how much memory is used where. A typical output (e.g. what I saw when I typed free -h ) could look like this:

      total   used    free   shared  buff/cache  available
Mem:    15G   3.7G    641M     222M         11G        11G
Swap:   15G   194M     15G

I ve used the -h option for human-readable output here for the sake of brevity and because I hate typing long lists of long numbers. People who have good memories (or old computers) may notice there is a missing -/+ buffers/cache line. This was intentionally removed in mid-2014 because as the memory management of Linux got more and more complicated, these lines became less relevant. These used to help with the not used used memory problem mentioned in the introduction but progress caught up with it. To explain what free is showing, you need to understand some of the underlying statistics that it works with. This isn t a lesson on how Linux its memory (the honest short answer is, I don t fully know) but just enough hopefully to understand what free is doing. Let s start with the two simple columns first; total and free. Total Memory This is what memory you have available to Linux. It is almost, but not quite, the amount of memory you put into a physical host or the amount of memory you allocate for a virtual one. Some memory you just can t have; either due to early reservations or devices shadowing the memory area. Unless you start mucking around with those settings or the virtual host, this number stays the same. Free Memory Memory that nobody at all is using. They haven t reserved it, haven t stashed it away for future use or even just, you know, actually using it. People often obsess about this statistic but its probably the most useless one to use for anything directly. I have even considered removing this column, or replacing it with available (see later what that is) because of the confusion this statistic causes. The reason for its uselessness is that Linux has memory management where it allocates memory it doesn t use. This decrements the free counter but it is not truly used . If you application needs that memory, it can be given back. A very important statistic to know for running a system is how much memory have I got left before I either run out or I start to serious swap stuff to swap drives. Despite its name, this statistic will not tell you that and will probably mislead you. My advice is unless you really understand the Linux memory statistics, ignore this one. Who s Using What Now we come to the components that are using (if that is the right word) the memory within a system. Shared Memory Shared memory is often thought of only in the context of processes (and makes working out how much memory a process uses tricky but that s another story) but the kernel has this as well. The shared column lists this, which is a direct report from the Shmem field in the meminfo file. Slabs For things used a lot within the kernel, it is inefficient to keep going to get small bits of memory here and there all the time. The kernel has this concept of slabs where it creates small caches for objects or in-kernel data strucutures that slabinfo(5) states [such as] buffer heads, inodes and dentries . So basically kernel stuff for the kernel to do kernelly things with. Slab memory comes in two flavours. There is reclaimable and unreclaimable. This is important because unreclaimable cannot be handed back if your system starts to run out of memory. Funny enough, not all reclaimable is, well, reclaimable. A good estimate is you ll only get 50% back, top and free ignore this inconvenient truth and assume it can be 100%. All of the reclaimable slab memory is considered part of the Cached statistic. Unreclaimable is memory that is part of Used. Page Cache and Cached Page caches are used to read and write to storage, such as a disk drive. These are the things that get written out when you use sync and make the second read of the same file much faster. An interesting quirk is that tmpfs is part of the page cache. So the Cached column may increase if you have a few of these. The Cached column may seem like it should only have Page Cache, but the Reclaimable part of the Slab is added to this value. For some older versions of some programs, they will have no or all Slab counted in Cached. Both of these versions are incorrect. Cached makes up part of the buff/cache column with the standard options for free or has a column to itself for the wide option. Buffers The second component to the buff/cache column (or separate with the wide option) is kernel buffers. These are the low-level I/O buffers inside the kernel. Generally they are small compared to the other components and can basically ignored or just considered part of the Cached, which is the default for free. Used Unlike most of the previous statistics that are either directly pulled out of the meminfo file or have some simple addition, the Used column is calculated and completely dependent on the other values. As such it is not telling the whole story here but it is reasonably OK estimate of used memory. Used component is what you have left of your Total memory once you have removed:

Free memory because free is not used!
The Cached value recall this is made up of the Page Cache plus the Reclaimable part of Slab
The buffers

Notice that the unreclaimable part of slab is not in this calculation, which means it is part of the used memory. Also note this seems a bit of a hack because as the memory management gets more complicated, the estimates used become less and less real. Available In early 2014, the kernel developers took pity on us toolset developers and gave us a much cleaner, simpler way to work out some of these values (or at least I d like to think that s why they did it). The available statistic is the right way to work out how much memory you have left. The commit message explains the gory details about it, but the great thing is that if they change their mind or add some new memory feature the available value should be changed as well. We don t have to worry about should all of slab be in Cached and are they part of Used or not, we have just a number directly out of meminfo. What does this mean for free? Poor old free is now at least 24 years old and it is based upon BSD and SunOS predecessors that go back way before then. People expect that their system tools don t change by default and show the same thing over and over. On the other side, Linux memory management has changed dramatically over those years. Maybe we re all just sheep (see I had to mention sheep or RAMs somewhere in this) and like things to remain the same always. Probably if free was written now; it would only need the total, available and used columns with used merely being total minus available. Possibly with some other columns for the wide option. The code itself (found in libprocps) is not very hard to maintain so its not like this change will same some time but for me I m unsure if free is giving the right and useful result for people that use it.

Dirk Eddelbuettel: BH 1.60.0-2

A new minor release of BH is now on CRAN. BH provides a large part of the Boost C++ libraries as a set of template headers for use by R, possibly with Rcpp as well as other packages. This release uses the same Boost 1.60.0 version of Boost as the last release, but adds three more library: bimap, flyweight and icl. A brief summary of changes from the NEWS file is below.

Changes in version 1.60.0-2 (2016-05-06)

Added Boost bimap via GH pull request #24 by Jim Hester.

Added Boost icl via GH pull request #27 by Jay Hesselbert.

Added Boost flyweight as requested in GH ticket #26.

Courtesy of CRANberries, there is also a diffstat report for the most recent release. Comments and suggestions are welcome via the mailing list or the issue tracker at the GitHubGitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

6 May 2016

Norbert Preining: Yubikey NEO

Two Factor authentication and general improvement of my security infrastructure was long on my todo list. Some month ago I finally purchased a Yubikey NEO from Yubico and try to consistently use it as second factor, as well as gpg signing/encrypting device. yubikey-neo

I am trying to get the best out of my Yubikey NEO by using as many of its functionality, in particular: Smartcard for my GNuPG keys, OTP similar to Google Authenticator and similar, as well as challenge-response for additional login security, as well as all that over NFC to not keep keys/passwords on my mobile phone. While there are loads of guides (see the previous article on GnuPG for some of them), many of them are out-of-date for current distributions and GnuPG etc. So I tried to collect all I could find not the least to have a place to look it up in case I forget it again. The Hardware The Yubikey NEO is a great peace of hardware. I not even remotely understand how they manage that this little beast can do all these things and still work out without mixing things up. As far as I understand (please correct me) it has three independent circuits of communication:

HID mode working as keyboard and sending keystrokes
CCID mode (smartcard) for PIV / GnuPG / OpenGPG functionality
NFC for communication with your mobile

On top of these circuit of communication there is a variety of applications to make the most out of your Yubikey:

various OTP: Yubico OTP (against a special server), TOTP, OATH-HOTP, Static PW, Challenge-Response
Fido U2F mode universal two-factor authentication mode
OpenGPG smartcard support 4 slots for private keys
PIV

Yubikey mode setup There are several modes, and using the ykpersonalize tool (readily available for Windows, Mac, Linux, and in the Debian package yubikey-personalization) one can program the key to work in a variety of modes. I chose to activate all options by passing in -m86 which stand for OTP/U2F/CCID composite device with MODE_FLAG_EJECT.

$ ykpersonalize -m86
Firmware version 3.4.3 Touch level 1792 Unconfigured
 
The USB mode will be set to: 0x86
 
Commit? (y/n) [n]: y
$

It is a good idea to unplug and replug the key after this operation. Yubikey udev rules for user access To allow users but root to use the Yubikey, additional udev rules are necessary:

SUBSYSTEMS=="usb", ATTRS idVendor =="1050", ATTRS idProduct =="0116", TAG+="uaccess"

which I put into /etc/udev/rules.d/99-yubikeys.rules on Debian. After that another unplug and replug should allow normal user to access the key. This can be checked by calling getfacl on the newly created /dev/hidraw? device. Using the HID/Challenge-Response mode (slot 2) If you want to secure your login with an additional second factor, there are several options documented on the Yubico site concerning yubico-pam. Since I cannot be sure to be always online with my laptop, I choose Challenge-Response authentication, and followed one-to-one Yubico s docs Local Authentication Using Challenge Response. Basically it boils down to install libpam-yubico, select mode-challenge-response when asked for configuration. Then one needs to personalizing the key (in particular slot 2) for challenge response with:

$ ykpersonalize -2 -ochal-resp -ochal-hmac -ohmac-lt64 -oserial-api-visible
...
Commit? (y/n) [n]: y
$

Next we need to save the challenge and expected response to the user s directory:

$ mkdir $HOME/.yubico
$ ykpamcfg -2 -v
...
Stored initial challenge and expected response in '/home/norbert/.yubico/challenge-123456'.
$

It might be a good idea to try this out, and if it works, activate it also for root. But be careful no key no login

Challenge: I am currently searching for a method to replace the second factor of they key optionally with a different authentication method, like a very difficult passphrase. This way I could log in even without my key, but in this case would need the complicated passphrase. From my reading of the pam manuals it seems to be possible, and I am planning to use pam_ssh and a specific login key with a complicated passphrase. I will report back when this is done. YubiOATH (TOTP) Time based One Time Passwords (aka Google Authenticator style) Without any setup whatsoever this worked out of the box. I use the Yubico Authenticator on my Android phone, and the dedicated application for the Linux desktop to create second factors for all kind of applications. Currently I am using it with Google login, Github, DropbBox, and WordPress (via the Two Factor plugin which can also be tweaked to use the NEO key as USB key via the FIDO U2F). Challenge: If I start the Yubico Personalization GUI, I see two free slots so where are the TOTPs computed? That also means that I have one slot free and for now I don t know what to do with it

Yubikey OpenGPG applet setup The Yubikeys support OpenPGP, and the applet is pre-installed (afaik), meaning you can directly configure the key and upload your keys. Here I use gpg2 (2.1) as it seems to better support card operations. To not interfere with the current gpg setup I use a temporary gpg home:

$ mkdir gpgtmp
$ chmod go-rwx gpgtmp
$ gpg2 --homedir gpgtmp --list-keys
gpg: keybox 'gpgtmp/pubring.kbx' created
gpg: gpgtmp/trustdb.gpg: trustdb created

Warning: The YubiKey NEO only supports 2048bit keys. If you want 4096bit keys you need to use one of the newer YubiKey 4, which gives you this option, but does not have support for NFC, and thus no way to interact with an Android (or other) mobile phone. Check the correct version of the applet There has been a bug in an older version of the applet, but since 2 years all keys sold should have a correct applet. You can check by:

$ gpg-connect-agent --homedir gpgtmp --hex "scd apdu 00 f1 00 00" /bye"
gpg-connect-agent: no running gpg-agent - starting '/usr/bin/gpg-agent'
gpg-connect-agent: waiting for the agent to come up ... (5s)
gpg-connect-agent: connection to agent established
D[0000]  01 00 10 90 00                                     .....           
OK

Looking at the output one sees D[0000] 01 00 10 which means applet version 1.0.10, which is the first version fixed. Replace pins of the key The standard pins are 123456 for the user pin, and 12345678 for the admin pin. These need immediate change! Warning: When changing the ping the normal pin must be 6 (at least?) digits, and the admin pin 8 (at least?), other gpg2 cannot use the key anymore. No idea why.

$ gpg2 --homedir gpgtmp --card-edit
 
Reader ...........: 1050:0116:X:0
Application ID ...: D2760001240102000006036457190000
Version ..........: 2.0
Manufacturer .....: Yubico
Serial number ....: 03645719
Name of cardholder: [not set]
Language prefs ...: [not set]
Sex ..............: unspecified
URL of public key : [not set]
Login data .......: [not set]
Signature PIN ....: forced
Key attributes ...: rsa2048 rsa2048 rsa2048
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 3 3
Signature counter : 0
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]
 
gpg/card> admin
Admin commands are allowed
 
gpg/card> passwd
gpg: OpenPGP card no. D2760001240102000006036457190000 detected
 
1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit
 
Your selection? 3
PIN changed.
 
1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit
 
Your selection? 1
PIN changed.
 
1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit
 
Your selection? q
 
gpg/card> quit

After this you need to use the new pins for all changes. Setup basic data The key can also save some basic data about yourself, like name, sex, language preferences, login name, and url to obtain the public key. As before start gpg2 and then change these infos in the following way>

gpg/card> name
Cardholder's surname: Preining
Cardholder's given name: Norbert
 
gpg/card> sex
Sex ((M)ale, (F)emale or space): M
 
gpg/card> lang
Language preferences: de
 
gpg/card> login
Login data (account name): norbert
 
gpg/card> url
URL to retrieve public key: https://www.preining.info/preining-norbert.asc
 
gpg/card> list
 
Reader ...........: 1050:0116:X:0
Application ID ...: D2760001240102000006036457190000
Version ..........: 2.0
Manufacturer .....: Yubico
Serial number ....: 03645719
Name of cardholder: Norbert Preining
Language prefs ...: de
Sex ..............: male
URL of public key : https://www.preining.info/preining-norbert.asc
Login data .......: norbert
Signature PIN ....: forced
Key attributes ...: rsa2048 rsa2048 rsa2048
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 3 3
Signature counter : 0
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]
 
gpg/card> quit

Move sub keys to Yubikey As laid out in the article on GnuPG subkeys, we are having three subkeys for signing, encryption, and authentication. In reality I will practically only use the signing key, but upload all three keys to the card. In the following I expect that you have a setup more or less similar to the one described in the article linked before. Again, we use GnuPG v2, mostly because it was the version that worked out of the box. In addition, if you are setting up a similar stage like in my GNuPG article with gpg1 keys on the mail server, then you don t want the gpg1 keys being removed. Basically you must have the Yubikey plugged in and call keytocard after selecting each key in turn (and deselecting it afterwards). Warning: There is another bug in the GnuPG applet that was fixed in later versions (but not in 1.0.10), namely that not all keys are accepted. This is a bit a pain. I needed to recreate a subkey to obtain a key that can be loaded onto the Yubikey. Unfortunately, Yubico has also stopped/disabled the ability to update applets (although I have to say their documentation is an incredible rubbish with respect to applets and upgrades ). As before, assume that $MASTERKEY contains the hex id of your master key.

$ gpg2 --edit-key $MASTERKEY
gpg (GnuPG) 2.1.11; Copyright (C) 2016 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
 
Secret key is available.
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 2
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb* rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> keytocard
Please select where to store the key:
   (1) Signature key
   (3) Authentication key
Your selection? 1
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb* rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 2
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 3
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb* rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> keytocard
Please select where to store the key:
   (2) Encryption key
Your selection? 2
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb* rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 3
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 4
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb* rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> keytocard
Please select where to store the key:
   (3) Authentication key
Your selection? 3
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb* rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> key 4
 
sec  rsa4096/0x6CACA448860CDC13
     created: 2010-09-14  expires: 2017-02-06  usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/0xD1D2BD14810F62B3
     created: 2010-09-14  expires: 2017-02-06  usage: E   
ssb  rsa2048/0xEC00B8DAD32266AA
     created: 2016-02-07  expires: 2017-02-06  usage: S   
ssb  rsa2048/0xBF361ED434425B4C
     created: 2016-02-07  expires: 2017-02-06  usage: E   
ssb  rsa2048/0x9C7CA4E294F04D49
     created: 2016-02-07  expires: 2017-02-06  usage: A   
[ultimate] (1). Norbert Preining <norbert@preining.info>
[ultimate] (2)  Norbert Preining <preining@logic.at>
[ultimate] (3)  Norbert Preining <preining@debian.org>
[ultimate] (4)  Norbert Preining <preining@jaist.ac.jp>
[ultimate] (5)  [jpeg image of size 4185]
 
gpg> save

After that your keys are on the Yubikey (and only there!), and GNuPG will require the PIN (user pin) to sign/encrypt documents. Usage Many things have been said above, but to sum up when and how I am using the YubiKey now:

Logging into my computer: I need to have the key plugged in, otherwise authentication will not succeed.
GPG activities (signing, encryption): Key needs to be plugged in, GnuPG will ask the User pin.
TOTP (Google, GitHub, WordPress, DropBox login): I use my mobile (Nexus 6p) and the Yubico Authenticator, touch the phone with the Yubikey, and see the TOTPs in the application windows.
OpenKeychain (Android app) integrates with K-9 Mail signing, encryption and decryption is possible on the mobile via NFC (touching the device with the key)

Conclusions With this setup I am now quite content, but not completely. What I still want to do is full disk encryption where I need the Yubikey to boot and again, with an alternative for a very long passphrase. At the end, adding a second factor to the login is not really optimal, and only protects you against quick hacks. If the laptop is actually stolen, only full disc protection helps. Access to the hardware always guarantees that one has access to everything on the disc. Another thing I want to do is re-use the GnuPG key on the Yubikey as ssh key for logging into remote systems. That would mean that I get rid of even more keys on my laptop. But this is still in the work

The other open question is what to use the other available slot of the Yubikey for? I thought about some passwords (possible), but I don t feel to happy about having my password issued with the press of a key. But all in all, I like the setup much more than before and not having any GnuPG key on the laptop is a big plus.

Dirk Eddelbuettel: RcppArmadillo 0.6.700.6.0

A second Armadillo release 6.700.6 came out in the 6.700 series, and we uploaded RcppArmadillo 0.6.700.6.0 to CRAN and Debian. This followed the usual thorough reverse-dependecy checking of by now 220 packages using. This release is a little unusual in that it contains both upstream bugfixes in the same series (see below) but also two nice bug fixes from the RcppArmadillo side. Both were squashed by George G. Vega Yon via two focused pull request. The first ensures that we can now use ARMA_64BIT_WORD (provided C++11 is turned on too) allowing for much bigger Armadillo objects. And the second plugs a small leak in the sparse matrix converter I had added a while back. Nice work, all told! Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. Changes in this release are as follows:

Changes in RcppArmadillo version 0.6.700.6.0 (2016-05-05)

Upgraded to Armadillo 6.700.6 (Catabolic Amalgamator Deluxe)

fix for handling empty matrices by kron()

fix for clang warning in advanced matrix constructors

fix for false deprecated warning in trunc_log() and trunc_exp()

fix for gcc-6.1 warning about misleading indentation

corrected documentation for the solve() function

Added support for int64_t (ARMA_64BIT_WORD) when required during compilation time. (PR #90 by George G. Vega Yon, fixing #88)

Fixed bug in SpMat exporter (PR #91 by George G. Vega Yon, fixing #89 and #72)

Courtesy of CRANberries, there is also a diffstat report for this release. As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

2 May 2016

Vincent Bernat: Pragmatic Debian packaging

While the creation of Debian packages is abundantly documented, most tutorials are targeted to packages implementing the Debian policy. Moreover, Debian packaging has a reputation of being unnecessarily difficult¹ and many people prefer to use less constrained tools² like fpm or CheckInstall. However, I would like to show how building Debian packages with the official tools can become straightforward if you bend some rules:

No source package will be generated. Packages will be built directly from a checkout of a VCS repository.
Additional dependencies can be downloaded during build. Packaging individually each dependency is a painstaking work, notably when you have to deal with some fast-paced ecosystems like Java, Javascript and Go.
The produced packages may bundle dependencies. This is likely to raise some concerns about security and long-term maintenance, but this is a common trade-off in many ecosystems, notably Java, Javascript and Go.

Pragmatic packages 101 In the Debian archive, you have two kinds of packages: the source packages and the binary packages. Each binary package is built from a source package. You need a name for each package. As stated in the introduction, we won t generate a source package but we will work with its unpacked form which is any source tree containing a debian/ directory. In our examples, we will start with a source tree containing only a debian/ directory but you are free to include this debian/ directory into an existing project. As an example, we will package memcached, a distributed memory cache. There are four files to create:

debian/compat,
debian/changelog,
debian/control, and
debian/rules.

The first one is easy. Just put 9 in it:

echo 9 > debian/compat

The second one has the following content:

memcached (0-0) UNRELEASED; urgency=medium
  * Fake entry
 -- Happy Packager <happy@example.com>  Tue, 19 Apr 2016 22:27:05 +0200

The only important information is the name of the source package, memcached, on the first line. Everything else can be left as is as it won t influence the generated binary packages.

The control file debian/control describes the metadata of both the source package and the generated binary packages. We have to write a block for each of them.

Source: memcached
Maintainer: Vincent Bernat <bernat@debian.org>
Package: memcached
Architecture: any
Description: high-performance memory object caching system

The source package is called memcached. We have to use the same name as in debian/changelog. We generate only one binary package: memcached. In the remaining of the example, when you see memcached, this is the name of a binary package. The Architecture field should be set to either any or all. Use all exclusively if the package contains only arch-independent files. In doubt, just stick to any. The Description field contains a short description of the binary package.

The build recipe The last mandatory file is debian/rules. It s the recipe of the package. We need to retrieve memcached, build it and install its file tree in debian/memcached/. It looks like this:

#!/usr/bin/make -f
DISTRIBUTION = $(shell lsb_release -sr)
VERSION = 1.4.25
PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0
TARBALL = memcached-$(VERSION).tar.gz
URL = http://www.memcached.org/files/$(TARBALL)
%:
    dh $@
override_dh_auto_clean:
override_dh_auto_test:
override_dh_auto_build:
override_dh_auto_install:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)
    ./configure --prefix=/usr
    make
    make install DESTDIR=debian/memcached
override_dh_gencontrol:
    dh_gencontrol -- -v$(PACKAGEVERSION)

The empty targets override_dh_auto_clean, override_dh_auto_test and override_dh_auto_build keep debhelper from being too smart. The override_dh_gencontrol target sets the package version³ without updating debian/changelog. If you ignore the slight boilerplate, the recipe is quite similar to what you would have done with fpm:

DISTRIBUTION=$(lsb_release -sr)
VERSION=1.4.25
PACKAGEVERSION=$ VERSION -0~$ DISTRIBUTION 0
TARBALL=memcached-$ VERSION .tar.gz
URL=http://www.memcached.org/files/$ TARBALL 
wget -N --progress=dot:mega $ URL 
tar --strip-components=1 -xf $ TARBALL 
./configure --prefix=/usr
make
make install DESTDIR=/tmp/installdir
# Build the final package
fpm -s dir -t deb \
    -n memcached \
    -v $ PACKAGEVERSION  \
    -C /tmp/installdir \
    --description "high-performance memory object caching system"

You can review the whole package tree on GitHub and build it with dpkg-buildpackage -us -uc -b.

Pragmatic packages 102 At this point, we can iterate and add several improvements to our memcached package. None of those are mandatory but they are usually worth the additional effort.

Build dependencies Our initial build recipe only work when several packages are installed, like wget and libevent-dev. They are not present on all Debian systems. You can easily express that you need them by adding a Build-Depends section for the source package in debian/control:

Source: memcached
Build-Depends: debhelper (>= 9),
               wget, ca-certificates, lsb-release,
               libevent-dev

Always specify the debhelper (>= 9) dependency as we heavily rely on it. We don t require make or a C compiler because it is assumed that the build-essential meta-package is installed and it pulls those. dpkg-buildpackage will complain if the dependencies are not met. If you want to install those packages from your CI system, you can use the following command⁴:

mk-build-deps \
    -t 'apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -qqy' \
    -i -r debian/control

You may also want to investigate pbuilder or sbuild, two tools to build Debian packages in a clean isolated environment.

Runtime dependencies If the resulting package is installed on a freshly installed machine, it won t work because it will be missing libevent, a required library for memcached. You can express the dependencies needed by each binary package by adding a Depends field. Moreover, for dynamic libraries, you can automatically get the right dependencies by using some substitution variables:

Package: memcached
Depends: $ misc:Depends , $ shlibs:Depends

The resulting package will contain the following information:

$ dpkg -I ../memcached_1.4.25-0\~unstable0_amd64.deb   grep Depends
 Depends: libc6 (>= 2.17), libevent-2.0-5 (>= 2.0.10-stable)

Integration with init system Most packaged daemons come with some integration with the init system. This integration ensures the daemon will be started on boot and restarted on upgrade. For Debian-based distributions, there are several init systems available. The most prominent ones are:

System-V init is the historical init system. More modern inits are able to reuse scripts written for this init, so this is a safe common denominator for packaged daemons.

Upstart is the less-historical init system for Ubuntu (used in Ubuntu 14.10 and previous releases).

systemd is the default init system for Debian since Jessie and for Ubuntu since 15.04.

Writing a correct script for the System-V init is error-prone. Therefore, I usually prefer to provide a native configuration file for the default init system of the targeted distribution (Upstart and systemd).

System-V If you want to provide a System-V init script, have a look at `/etc/init.d/skeleton` on the most ancient distribution you want to target and adapt it⁵. Put the result in `debian/memcached.init`. It will be installed at the right place, invoked on install, upgrade and removal. On Debian-based systems, many init scripts allow user customizations by providing a `/etc/default/memcached` file. You can ship one by putting its content in `debian/memcached.default`.

Upstart Providing an Upstart job is similar: put it in debian/memcached.upstart. For example:

description "memcached daemon"
start on runlevel [2345]
stop on runlevel [!2345]
respawn
respawn limit 5 60
expect daemon
script
  . /etc/default/memcached
  exec memcached -d -u $USER -p $PORT -m $CACHESIZE -c $MAXCONN $OPTIONS
end script

When writing an Upstart job, the most important directive is expect. Be sure to get it right. Here, we use expect daemon and memcached is started with the -d flag.

systemd Providing a systemd unit is a bit more complex. The content of the file should go in debian/memcached.service. For example:

[Unit]
Description=memcached daemon
After=network.target
[Service]
Type=forking
EnvironmentFile=/etc/default/memcached
ExecStart=/usr/bin/memcached -d -u $USER -p $PORT -m $CACHESIZE -c $MAXCONN $OPTIONS
Restart=on-failure
[Install]
WantedBy=multi-user.target

We reuse /etc/default/memcached even if it is not considered a good practice with systemd⁶. Like for Upstart, the directive Type is quite important. We used forking as memcached is started with the -d flag. You also need to add a build-dependency to dh-systemd in debian/control:

Source: memcached
Build-Depends: debhelper (>= 9),
               wget, ca-certificates, lsb-release,
               libevent-dev,
               dh-systemd

And you need to modify the default rule in debian/rules:

%:
    dh $@ --with systemd

The extra complexity is a bit unfortunate but systemd integration is not part of debhelper⁷. Without those additional modifications, the unit will get installed but you won t get a proper integration and the service won t be enabled on install or boot.

Dedicated user Many daemons don t need to run as root and it is a good practice to ship a dedicated user. In the case of memcached, we can provide a _memcached user⁸. Add a debian/memcached.postinst file with the following content:

#!/bin/sh
set -e
case "$1" in
    configure)
        adduser --system --disabled-password --disabled-login --home /var/empty \
                --no-create-home --quiet --force-badname --group _memcached
        ;;
esac
#DEBHELPER#
exit 0

There is no cleanup of the user when the package is removed for two reasons:

Less stuff to write.
The user could still own some files.

The utility adduser will do the right thing whatever the requested user already exists or not. You need to add it as a dependency in debian/control:

Package: memcached
Depends: $ misc:Depends , $ shlibs:Depends , adduser

The #DEBHELPER# marker is important as it will be replaced by some code to handle the service configuration files (or some other stuff). You can review the whole package tree on GitHub and build it with dpkg-buildpackage -us -uc -b.

Pragmatic packages 103 It is possible to leverage debhelper to reduce the recipe size and to make it more declarative. This section is quite optional and it requires understanding a bit more how a Debian package is built. Feel free to skip it.

The big picture There are four steps to build a regular Debian package:

debian/rules clean should clean the source tree to make it pristine.
debian/rules build should trigger the build. For an autoconf-based software, like memcached, this step should execute something like ./configure && make.
debian/rules install should install the file tree of each binary package. For an autoconf-based software, this step should execute make install DESTDIR=debian/memcached.
debian/rules binary will pack the different file trees into binary packages.

You don t directly write each of those targets. Instead, you let dh, a component of debhelper, do most of the work. The following debian/rules file should do almost everything correctly with many source packages:

#!/usr/bin/make -f
%:
    dh $@

For each of the four targets described above, you can run dh with --no-act to see what it would do. For example:

$ dh build --no-act
   dh_testdir
   dh_update_autotools_config
   dh_auto_configure
   dh_auto_build
   dh_auto_test

Each of those helpers has a manual page. Helpers starting with dh_auto_ are a bit magic . For example, dh_auto_configure will try to automatically configure a package prior to building: it will detect the build system and invoke ./configure, cmake or Makefile.PL. If one of the helpers do not do the right thing, you can replace it by using an override target:

override_dh_auto_configure:
    ./configure --with-some-grog

Those helpers are also configurable, so you can just alter a bit their behaviour by invoking them with additional options:

override_dh_auto_configure:
    dh_auto_configure -- --with-some-grog

This way, ./configure will be called with your custom flag but also with a lot of default flags like --prefix=/usr for better integration. In the initial memcached example, we overrode all those magic targets. dh_auto_clean, dh_auto_configure and dh_auto_build are converted to no-ops to avoid any unexpected behaviour. dh_auto_install is hijacked to do all the build process. Additionally, we modified the behavior of the dh_gencontrol helper by forcing the version number instead of using the one from debian/changelog.

Automatic builds As memcached is an autoconf-enabled package, dh knows how to build it: ./configure && make && make install. Therefore, we can let it handle most of the work with this debian/rules file:

#!/usr/bin/make -f
DISTRIBUTION = $(shell lsb_release -sr)
VERSION = 1.4.25
PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0
TARBALL = memcached-$(VERSION).tar.gz
URL = http://www.memcached.org/files/$(TARBALL)
%:
    dh $@ --with systemd
override_dh_auto_clean:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)
override_dh_auto_test:
    # Don't run the whitespace test
    rm t/whitespace.t
    dh_auto_test
override_dh_gencontrol:
    dh_gencontrol -- -v$(PACKAGEVERSION)

The dh_auto_clean target is hijacked to download and setup the source tree⁹. We don t override the dh_auto_configure step, so dh will execute the ./configure script with the appropriate options. We don t override the dh_auto_build step either: dh will execute make. dh_auto_test is invoked after the build and it will run the memcached test suite. We need to override it because one of the test is complaining about odd whitespaces in the debian/ directory. We suppress this rogue test and let dh_auto_test executes the test suite. dh_auto_install is not overriden either, so dh will execute some variant of make install. To get a better sense of the difference, here is a diff:

--- memcached-intermediate/debian/rules 2016-04-30 14:02:37.425593362 +0200
+++ memcached/debian/rules  2016-05-01 14:55:15.815063835 +0200
@@ -12,10 +12,9 @@
 override_dh_auto_clean:
-override_dh_auto_test:
-override_dh_auto_build:
-override_dh_auto_install:
    wget -N --progress=dot:mega $(URL)
    tar --strip-components=1 -xf $(TARBALL)
-   ./configure --prefix=/usr
-   make
-   make install DESTDIR=debian/memcached
+
+override_dh_auto_test:
+   # Don't run the whitespace test
+   rm t/whitespace.t
+   dh_auto_test

It is up to you to decide if dh can do some work for you, but you could try to start from a minimal debian/rules and only override some targets.

Install additional files While make install installed the essential files for memcached, you may want to put additional files in the binary package. You could use cp in your build recipe, but you can also declare them:

files listed in debian/memcached.docs will be copied to /usr/share/doc/memcached by dh_installdocs,
files listed in debian/memcached.examples will be copied to /usr/share/doc/memcached/examples by dh_installexamples,
files listed in debian/memcached.manpages will be copied to the appropriate subdirectory of /usr/share/man by dh_installman,

Here is an example using wildcards for debian/memcached.docs:

doc/*.txt

If you need to copy some files to an arbitrary location, you can list them along with their destination directories in debian/memcached.install and dh_install will take care of the copy. Here is an example:

scripts/memcached-tool usr/bin

Using those files make the build process more declarative. It is a matter of taste and you are free to use cp in debian/rules instead. You can review the whole package tree on GitHub.

Other examples The GitHub repository contains some additional examples. They all follow the same scheme:

`dh_auto_clean` is hijacked to download and setup the source tree

`dh_gencontrol` is modified to use a computed version

Notably, you ll find daemons in Java, Go, Python and Node.js. The goal of those examples is to demonstrate that using Debian tools to build Debian packages can be straightforward. Hope this helps.

People may remember the time before debhelper 7.0.50 (circa 2009) where `debian/rules` was a daunting beast. However, nowaday, the boilerplate is quite reduced.

The complexity is not the only reason. Those alternative tools enable the creation of RPM packages, something that Debian tools obviously don t.

There are many ways to version a package. Again, if you want to be pragmatic, the proposed solution should be good enough for Ubuntu. On Debian, it doesn t cover upgrade from one distribution version to another, but we assume that nowadays, systems get reinstalled instead of being upgraded.

You also need to install `devscripts` and `equivs` package.

It s also possible to use a script provided by upstream. However, there is no such thing as an init script that works on all distributions. Compare the proposed with the skeleton, check if it is using `start-stop-daemon` and if it sources `/lib/lsb/init-functions` before considering it. If it seems to fit, you can install it yourself in `debian/memcached/etc/init.d/`. debhelper will ensure its proper integration.

Instead, a user wanting to customize the options is expected to edit the unit with `systemctl edit`.

See #822670

The Debian Policy doesn t provide any hint for the naming convention of those system users. A common usage is to prefix the daemon name with an underscore (like `_memcached`). Another common usage is to use `Debian-` as a prefix. The main drawback of the latest solution is that the name is likely to be replaced by the UID in `ps` and `top` because of its length.

We could call `dh_auto_clean` at the end of the target to let it invoke `make clean`. However, it is assumed that a fresh checkout is used before each build.

Russ Allbery: Review: The Girl with the Dragon Tattoo

Review: The Girl with the Dragon Tattoo, by Stieg Larsson

Translator:	Reg Keeland
Series:	Millennium #1
Publisher:	Vintage Crime
Copyright:	2005, 2008
Printing:	June 2009
ISBN:	0-307-47347-3
Format:	Mass market
Pages:	644

As The Girl with the Dragon Tattoo opens, Mikael Blomkvist is losing a criminal libel suit in Swedish court. His magazine, Millennium, published his hard-hitting piece of investigative journalism that purported to reveal sketchy arms deals and financial crimes by Hans-Erik Wennerstr m, a major Swedish businessman. But the underlying evidence didn't hold up, and Blomkvist could offer no real defense at trial. The result is a short prison stint for him (postponed several months into this book) and serious economic danger for Millennium. Lisbeth Salander is a (very) freelance investigator for Milton Security. Her specialty is research and background checks: remarkably thorough, dispassionate, and comprehensive. She's almost impossible to talk to, tending to meet nearly all questions with stony silence, but Dragan Armansky, the CEO of Milton Security, has taken her partly under his wing. She, and Milton Security, were hired by a lawyer named Dirch Frode to do a comprehensive background check on Mikael Blomkvist, which she and Dragan present near the start of the book. The reason, as the reader discovers in a few more chapters, is that Frode's employer wants to offer Blomkvist a very strange job. Over forty years ago, Harriet Vanger, scion of one of Sweden's richest industrial families, disappeared. Her uncle, Henrik Vanger, has been obsessed with her disappearance ever since, but in forty years of investigation has never been able to discover what happened to her. There are some possibilities for how her body could have been transported off the island the Vangers (mostly) lived, and live, on, but motive and suspects are still complete unknowns. Vanger wants Blomkvist to try his hand under the cover of writing a book about the Vanger family. Payment is generous, but even more compelling is Henrik Vanger's offer to give Blomkvist documented, defensible evidence against Wennerstr m at the end of the year. The Girl with the Dragon Tattoo (the original Swedish title is M n som hatar kvinnor, "Men who hate women") is the first of three mystery novels written at the very end of Stieg Larsson's life, all published posthumously. They made quite a splash when they were published: won multiple awards, sold millions of copies, and have resulted in four movies to date. I've had a copy of the book sitting around for a while and finally picked it up when in the mood for something a bit different. A major disclaimer up front: I read very little crime and mystery fiction. Every genre has its own conventions and patterns, and regular genre readers often look for different things than people new to that genre. My review is from a somewhat outside and inexperienced perspective, which may not be useful for regular genre readers. I'm also a US reader, reading the book in translation. It appears to be a very good translation, but it was also quite obvious to me that The Girl with the Dragon Tattoo was written from a slightly different set of cultural assumptions than I brought to the book. This is one of the merits of reading books from other cultures in translation. It can be eye-opening, and can carry some of the same thrill as science fiction or fantasy, to hit the parts of the book that question your assumptions. But it can also be hard to tell whether some striking aspect of a book is due to a genre convention I wasn't familiar with, a Swedish cultural assumption that I don't share, or just the personal style of the author. A few things do leap out as cultural differences. Blomkvist has to spend a few months in prison in the middle of this book, and that entire experience is completely foreign to an American understanding of what prison is like. The degradation, violence, and awfulness that are synonymous with prison for an American are almost entirely absent. He even enjoys the experience as quiet time to focus on writing a history of the Vangers (Blomkvist early on decides to take his cover story seriously, since he doubts he'll make any inroads into the mystery of Harriet's disappearance but can at least get a book out of it). It's a minor element in the book, glossed over in a few pages, but it's certainly eye-opening for how minimum security prison could be structured in a civilized country. Similarly, as an American reader, I was struck by how hard Larsson has to work to ruin Salander's life. Although much of the book is written from Blomkvist's perspective (in tight third person), Lisbeth Salander is the titular girl with the dragon tattoo and becomes more and more involved in the story as it develops. The story Larsson wanted to tell requires that she be in a very precarious position legally and socially. In the US, this would be relatively easy, particularly for someone who acts like Salander does. In Sweden, Larsson has to go to monumental efforts to find ways for Salander to credibly fall through Sweden's comprehensive social safety net, and still mostly relies on Salander's complete refusal to assist or comply with any form of authority or support. I've read a lot about differences in policies around social support between the US and Scandinavian countries, but I've rarely read something that drove the point home more clearly than the amount of work a novelist has to go to in order to mess up their protagonist's life in Sweden. The actual plot is slow-moving and as much about the psychology of the characters as it is about the mystery. The reader gets inside the thoughts of the characters occasionally, but Larsson shows far more than tells and leaves it to the reader to draw more general conclusions. Blomkvist's relationship with his long-time partner and Millennium co-founder is an excellent example: so much is left unstated that I would have expected other books to lay down in black and white, and the characters seem surprisingly comfortable with ambiguity. (Some of this may be my genre unfamiliarity; SFF tends to be straightforward to a fault, and more literary fiction is more willing to embrace ambiguous relationships.) While the mystery of Harriet's disappearance forms the backbone of the story, rather more pages are spent on Blomkvist navigating the emotional waters of the near-collapse of his career and business, his principles around investigation and journalism, and the murky waters of the Vanger's deeply dysfunctional family. Harriet's disappearance is something of a locked room mystery. The day she disappeared, a huge crash closed the only bridge from the island to the mainland, both limiting suspects and raising significant questions about why her body was never found on the island. It's also forty years into the past, so Blomkvist has to rely on Henrik Vanger's obsessive archives, old photographs, and old police reports. I found the way it unfolded to be quite satisfying: there are just enough clues to let Blomkvist credibly untangle things with some hard work and research, but they're obscure enough to make it plausible that previous investigators missed them. Through most of this novel, I wasn't sure what I thought of it. I have a personal interest in Blomkvist's journalistic focus wrongdoing by rich financiers but I had trouble warming to Blomkvist himself. He's a very passive, inward character, who spends a lot of the early book reacting to things that are happening to him. Salander is more dynamic and honestly more likable, but she's also deeply messed up, self-destructive, and does some viciously awful things in this book. And the first half of the book is very slow: lots of long conversations, lots of character introduction, and lots of Blomkvist wandering somewhat aimlessly. It's only when Larsson gets the two protagonists together that I thought the book started to click. Salander sees Blomkvist's merits more clearly than the reader can, I think. I also need to give a substantial warning: The Girl with the Dragon Tattoo is a very violent novel, and a lot of that violence is sexual. By mid-book, Blomkvist realizes that Harriet's disappearance is somehow linked with a serial killer whose trademark is horrific, sexualized symbolism drawn from Leviticus. There is a lot of rape here, including revenge rape by a protagonist. If that sort of thing is likely to bother you, you may want to steer way clear. That said, despite the slow pace, the nauseating subject matter, the occasionally very questionable ethics of protagonists, and a twist of the knife at the very end of the novel that I thought was gratuitously nasty on Larsson's part and wasn't the conclusion I wanted, I found myself enjoying this. It has a different pace and a different flavor than what I normally read, the characters are deep and complex enough to play off each other in satisfying ways, and Salander is oddly compelling to read about. Given the length, it's a substantial investment of time, but I don't regret reading it, and I'm quite tempted to read the sequel. I'm not sure this is the sort of book I can recommend (or not recommend) given my lack of familiarity with the genre, but I think US readers might get an additional layer of enjoyment out of seeing how different of a slant the Swedish setting puts on some of the stock elements of a crime novel. Followed by The Girl Who Played with Fire. Rating: 7 out of 10

1 May 2016

Ben Hutchings: Debian LTS work, April 2016

This month was still quiet for me in terms of uploads, as "wheezy" was only handed over to the LTS team near the end of the month. I carried over 5.5 hours from March and was assigned another 15 hours of work by Freexian's Debian LTS initiative, but only worked a total of 12.25 hours. I have returned the spare hours to the pool. As last month, I prepared a stable update for Linux 3.2 on kernel.org, which will be released soon as 3.2.80. I also triaged the open security issues and backported a few individual patches to our wheezy-security branch. However I expect to rebase the wheezy-security branch onto Linux 3.2.80 before making the next upload. I also participated in discussion of supporting armel/armhf in wheezy LTS. I don't expect many LTS users to be using the Debian kernel packages, as we only supported a small range of ARM hardware before the introduction of the multiplatform flavours in jessie. However, those architectures rarely require any extra effort to support in linux stable updates so I had no objection to including them.

30 April 2016

Chris Lamb: Free software activities in April 2016

Here is my monthly update covering a large part of what I have been doing in the free software world (previously):

Added Python 3 support to django-template-tests, a tool to perform simple static analysis on Django templates. (#1)
Corrected my Chrome extension for the FastMail web interface to not disable the CTRL+Enter keyboard shortcut when authoring emails. (#3)
Corrected a subtle bug in my django-staticfiles-dotd "staticfiles" library where the Content-Length HTTP header was calculated incorrectly in the presence of Unicode characters resulting in truncated output. (#2)
Various fixes to django-slack, a library to easily post messages to the Slack group-messaging utility from projects using the Django web development framework:
- Don't require an explicit backend import when using the Celery task queue backend. (#41)
- Actually generate and send messages asynchronously when using the Celery backend. (#44)
Fixed an issue with my local-debian-mirror tool where the option to disable DEP-11 mirroring wasn't working. (#1)
Fixed an issue in django-hipchat, a library to easily post messages to the Hipchat group-messaging utility from projects using the Django web development framework where the templates were not includes when installing via PyPI. (#1)
Created a quick-and-dirty tool to scrape a Squarespace blog and convert it to a PDF so I can read them on my Kindle e-reader. (tree)
Updated django-keyerror a library to post exceptions to the KeyError.com error tracking service to silence an AttributeError exception in some error-reporting edge-cases. (commit)
Suggested an improvement to the documentation for the upcoming Twitter Bootstrap version for the deprecated .hidden and .show CSS classes. (#19789)
Submitted a documentation update to the Ansible sever configuration tool's ufw firewall module. (commit)
I also blogged about parsing Jenkins CI output to determine job success or failure.

Debian My work in the Reproducible Builds project was covered in our weekly reports. (#48, #49, #50, #51 & #52)

Uploads

redis (2:3.0.7-3) Adding, amongst some other changes, systemd LimitNOFILE support to allow a higher number of open file descriptors.

FTP Team

As a Debian FTP assistant I ACCEPTed 135 packages: aptitude, asm, beagle, blends, btrfs-progs, camitk, cegui-mk2, cmor-tables, containerd, debian-science, debops, debops-playbooks, designate-dashboard, efitools, facedetect, flask-testing, fstl, ganeti-os-noop, gnupg, golang-fsnotify, golang-github-appc-goaci, golang-github-benbjohnson-tmpl, golang-github-dchest-safefile, golang-github-docker-go, golang-github-dylanmei-winrmtest, golang-github-hawkular-hawkular-client-go, golang-github-hlandau-degoutils, golang-github-hpcloud-tail, golang-github-klauspost-pgzip, golang-github-kyokomi-emoji, golang-github-masterminds-semver-dev, golang-github-masterminds-vcs-dev, golang-github-masterzen-xmlpath, golang-github-mitchellh-ioprogress, golang-github-smartystreets-assertions, golang-gopkg-hlandau-configurable.v1, golang-gopkg-hlandau-easyconfig.v1, golang-gopkg-hlandau-service.v2, golang-objx, golang-pty, golang-text, gpaste, gradle-plugin-protobuf, grip, haskell-brick, haskell-hledger-ui, haskell-lambdabot-haskell-plugins, haskell-text-zipper, haskell-werewolf, hkgerman, howdoi, jupyter-client, jupyter-core, letsencrypt.sh, libbpp-phyl, libbpp-raa, libbpp-seq, libbpp-seq-omics, libcbor-xs-perl, libdancer-plugin-email-perl, libdata-page-pageset-perl, libevt, libevtx, libgit-version-compare-perl, libgovirt, libmsiecf, libnet-ldap-server-test-perl, libpgobject-type-datetime-perl, libpgobject-type-json-perl, libpng1.6, librest-client-perl, libsecp256k1, libsmali-java, libtemplates-parser, libtest-requires-git-perl, libtext-xslate-perl, linux, linux-signed, mandelbulber2, netlib-java, nginx, node-rc, node-utml, nvidia-cuda-toolkit, openfst, openjdk-9, openssl, php-cache-integration-tests, pulseaudio, pyfr, pygccxml, pytest-runner, python-adventure, python-arrayfire, python-django-feincms, python-fastimport, python-fitsio, python-imagesize, python-lib389, python-libtrace, python-neovim-gui, python3-proselint, pythonpy, pyzo, r-cran-ca, r-cran-fitbitscraper, r-cran-goftest, r-cran-rnexml, r-cran-rprotobuf, rrdtool, ruby-proxifier, ruby-seamless-database-pool, ruby-syslog-logger, rustc, s5, sahara-dashboard, salt-formula-ceilometer, salt-formula-cinder, salt-formula-glance, salt-formula-heat, salt-formula-horizon, salt-formula-keystone, salt-formula-neutron, salt-formula-nova, seer, simplejson, smrtanalysis, tiles-autotag, tqdm, tran, trove-dashboard, vim, vulkan, xapian-bindings & xapian-core.

Daniel Stender: What I've worked on for Debian this month

This month I've worked on the following things for Debian: To begin with that, I've set up a Debhelper sequencer script for dh-buildinfo¹, this add-on now can be used with dh $@ --with buildinfo in deb/rules instead of having to explicitly call it somewhere in an override. Debops I've set up initial Debian packages of Debops², a collection of fine crafted Ansible roles and playbooks especially for Debian servers, shipped with a couple of convenience and wrapper scripts in Python³. There are two binary packages, one for the toolset (debops), and the other for the playbooks and roles of the project (debops-playbooks). The application is easy to use, just initialize a new project with debops-init foo and add your server(s) to foo/ansible/inventory/hosts belonging to groups representing services and things you want to employ on them. Like the group [debops_gitlab] automatically installs a complete running Gitlab setup on one or a multitude of servers in the same run with the debops command⁴. Use other groups like [debops_mariadb_server] accordingly in the same host inventory. Ansible runs agent less, so you don't have to prepare freshly setup servers with nothing special to use that tool randomly (like on localhost). The list of things you could deploy with Debops is quite amazing and you've got dozens of services at your hand. The new packages are currently in experimental because they need some more fine tuning, like there are a couple of minor error messages which recently occur using it, but it works well. The (early staged) documentation unfortunately couldn't be packaged because of the scattered resp. collective nature of the project (all parts have their own Github repositories)⁵, and also how to generate the upstream tarball remains a bit of a challenge (currently, it's the outcome of debops-init)⁶. I'll have this package in unstable soon. More info on Debops is coming up, then. Hashicorp's Packer I'm very glad to announce that Packer⁷ is ready being available in unstable, and the two year old RFP bug could be finally closed⁸. It's another great and much convenient devops tool which does a lot of different things in an automated fashion using only a single "one-argument" CLI tool in combination with a couple of lines in a configuration script (thanks to Yaroslav Halchenko for the tip). Packer helps creating machine images for different platforms. This is like when you use e.g. Debian installations in a Qemu box for testing or development purposes. Instead of setting up a new virtual machine manually like installing Debian on another computer this process could be automated with Packer, like I've written about in this blog entry here⁹. You just need a template containing instructions for the included Qemu-builder and a preseeding script for the Debian installer, and there you go drinking your coffee while Packer does all the work for you: downloading the installation ISO image, creating the new virtual harddrive, booting the emulator, running the whole installation process automatically like answering questions, selecting things, rebooting without ISO image to complete the installation etc. A couple of minutes and you have a new pre-baked virtual machine image like from a vendoring machine, a fresh one everytime you need it. Packer¹⁰ supports a number of builders for different target platforms (desktop virtualization solutions as much as public cloud providers and private cloud software), can build in parallel, and also the full range of common provisioners can be employed in the process to equip the newly installed OSs. Vagrant boxes could be generated by one of the included postprocessors. I'll write more on Packer here on this blog, soon. There were more then two dozens of packages missing to complete Packer¹¹, which is the achievement of combined forces within the pkg-go group. Much thanks esp. to Alexandre Viau who have worked on the most of the needed new packages. Thanks also to the FTP-masters which were always very quick in reviewing the Go packages, so that it could be proceeded to build and package the sub dependent new ones always consecutively. Squirrel3 I've didn't had the most work with it and just sponsored this for Fabian Wolff, but want to highlight here that there's a new package of Squirrel¹² now available in Debian¹³. Squirrel is a lightweight scripting language, somewhat comparable to Lua. It's fully object-oriented and highly embeddable, it's used in a lot of commerical computer games under the hood for implementing intelligence for bots next to other things¹⁴, but also for the Internet of Things (it's embedded in hardware from Electric Imp). Squirrel functions could be called from C++¹⁵. I've filed an ITP bug for Squirrel already in 2011 (#651195), but always something else got in the way, and it ended up being an RFP. I'm really glad that it got picked up and completed. misc There were a couple of uploads on updated upstream tarballs and for fixing bugs, namely afl/2.10b-1 and 2.11b-1, python-afl/0.5.3-1, pyutilib/5.3.2-1, pyomo/4.3.11327-1, libvigraimpex/1.10.0+git20160211.167be93dfsg-2 (fix of #820429, thanks for Tobias Frost), and gamera/3.4.2+svn1454-1. For the pkg-go group, I've set up a new package of github-mitchellh-ioprogress (which is needed by the official DigitalOcean CLI tool doctl, now RFP #807956 instead of ITP due to the lack of time - again facing a lot of missing packages), and provided a little patch for dh-make-golang updating some standards¹⁶. For Packer I've also updated azure-go-autorest and azure-sdk as team upload (#821938, #821832), but it came out that the project which is currently under heavy development towards a new official release broke a lot in the past weeks (and no Git branching have been used), so that Packer as a matter of fact needed a vendored snapshot, although there have been only a couple of commits in between. Docker-registry hat the same problem with the new package of azure-sdk/2.1.1~beta1, so that it needed to be fixed, too (#822146). By the way, the tool ratt¹⁷ comes very handy for automatically test building down all reverse dependencies, not only for Go packages (thanks to Tianon Gravi for the tip). Finally, I've posted the needed reverse depencies as RFP bugs for Terraform¹⁸ (again quite a lot), Vuls¹⁹, and cve-dictionary²⁰, which is needed for Vuls. I'll let them rest a while waiting to get picked up before working anything down.

#570933: dh-buildinfo: should provide a addon file for dh command
https://tracker.debian.org/pkg/debops
http://debops.org/
The servers have to be accessible by SSH. E.g. you could run debops like: $ debops -u root --private-key=~/.ssh/id_digitalocean
https://github.com/debops/docs/issues/132
#819816: ITP: debops -- Ansible based server management utility
https://www.packer.io/
#740753: ITP: packer -- create vm images for multiple platforms
http://www.danielstender.com/blog/packer-qemu.html
https://packages.debian.org/unstable/packer
I've worked on the missing packages this month, namely github-klauspost-pgzip, github-masterzen-xmlpath, github-masterzen-winrm, dylanmei-winrmtest, packer-community-winrmcp (Packer uses WinRM if Windows machines images are created), github-hpcloud-tail, and updated github-rackspace-gophercloud (#822163) and google-api (#822164) to complete it.
http://squirrel-lang.org/
https://tracker.debian.org/pkg/squirrel3
http://www.linux-magazin.de/layout/set/print/content/view/full/62184
http://www.linux-magazin.de/Ausgaben/2011/10/plus/Fremdkoerper-Squirrel-Interpreter-und-Skripte-fuer-C
https://github.com/Debian/dh-make-golang/pull/39
https://packages.debian.org/unstable/ratt
#808940: ITP: terraform -- tool for managing cloud infrastructure
#820614: ITP: vuls -- package inventory scanner for CVE vulnerabilities
#820615: ITP: go-cve-dictionary -- builds a local copy of the NVD/JVN (vulnerability databases)

Daniel Stender: My work for Debian in April

This month I've worked on the these things for Debian: To begin with that, I've set up a Debhelper sequencer script for dh-buildinfo¹, this add-on now can be used with dh $@ --with buildinfo in deb/rules instead of having to explicitly call it somewhere in an override. Debops I've set up initial Debian packages of Debops², a collection of fine crafted Ansible roles and playbooks especially for Debian servers (servers which run on Debian), which are shipped with a couple of helper and wrapper scripts in Python³. There are two binary packages, one for the toolset (debops), and the other for the playbooks and roles of the project (debops-playbooks). The application is easy to use, just initialize a new project with debops-init foo and add your server(s) to foo/ansible/inventory/hosts belonging to groups representing services and things you want to employ on them. For example, the group [debops_gitlab] automatically installs a complete running Gitlab setup on one or a multitude of servers in the same run with the debops command⁴. Other groups like [debops_mariadb_server] could be used accordingly in the same host inventory. Ansible works without agent, so you don't have to prepare freshly setup servers with nothing special to use that tool randomly (like on localhost). The list of things you could deploy with Debops is quite amazing and dozens of services are at hand. The new Debian packages are currently in experimental because they need some more fine tuning, e.g. there are a couple of minor error messages which recently occur using it, but it works well. The (early staged) documentation unfortunately couldn't be packaged because of the scattered resp. collective nature of the project (all parts have their own Github repositories)⁵, and also how to generate the upstream tarball remains a bit of a challenge (currently, it's the outcome of debops-init)⁶. I'll have this package in unstable soon. More info on Debops is coming up, then. HashiCorp's Packer I'm very glad to announce that Packer⁷ is ready being available in unstable, and the RFP bug could be finally closed after I've taken it over⁸. It's another great and much convenient devops tool which does a lot of different things in an automated fashion using only a single "one-argument" CLI tool in combination with a couple of lines in a configuration script (thanks to Yaroslav Halchenko for the tip). Packer helps creating machine images for different platforms. This is like when you use e.g. Debian installations in a Qemu box for testing or development purposes. Instead of setting up a new virtual machine manually the same way as installing Debian on another computer this process can be completely automated with Packer, like I've written about in this blog entry here⁹. You just need a template which contains instructions for the included Qemu builder and a preseeding script for the Debian installer, and there you go drinking your coffee while Packer does all the work: download the ISO image for installation, create the new virtual harddrive, boot the emulator, run the whole installation process automatically like with answering questions, selecting things, reboot without ISO image to complete the installation etc. A couple of minutes and you have a new pre-baked virtual machine image like from a vendoring machine, another fresh one could be created anytime. Packer¹⁰ supports a number of builders for different target platforms (desktop virtualization solutions as much as public cloud providers and private cloud software), can build in parallel, and also the full range of common provisioners can be employed in the process to equip the newly installed OSs with services and programs. Vagrant boxes could be generated by one of the included postprocessors. I'll write more on Packer here on this blog, soon. There were more then two dozens of packages missing to complete Packer¹¹, which is the achievement of combined forces within the pkg-go group. Much thanks esp. to Alexandre Viau who have worked on the most of the needed new packages. Thanks also to the FTP masters which were always very quick in reviewing the Go packages, so that it could be proceeded to build and package the sub dependent new ones always consecutively. Squirrel3 I've didn't had the major work of that and just sponsored this for Fabian Wolff, but want to highlight here that there's a new package of Squirrel¹² now available in Debian¹³. Squirrel is a lightweight scripting language, somewhat comparable to Lua. It's fully object-oriented and highly embeddable, it's used in a lot of commerical computer games under the hood for implementing intelligence for bots next to other things¹⁴, but also for the Internet of Things (it's embedded in hardware from Electric Imp). Squirrel functions could be called from C++¹⁵. I've filed an ITP bug for Squirrel already in 2011 (#651195), but always something else had a higher priority, and it ended up being an RFP. I'm really glad that it got picked up and completed quickly afterwards. misc There were a couple of uploads on updated upstream tarballs and for fixing bugs, namely afl/2.10b-1 and 2.11b-1, python-afl/0.5.3-1, pyutilib/5.3.2-1, pyomo/4.3.11327-1, libvigraimpex/1.10.0+git20160211.167be93dfsg-2 (fix of #820429, thanks to Tobias Frost), and gamera/3.4.2+svn1454-1. For the pkg-go group, I've set up a new package of github-mitchellh-ioprogress (which is needed by the official DigitalOcean CLI tool doctl, now RFP #807956 instead of ITP due to the lack of time, again a lot of missing packages are missing for that), and provided a little patch for dh-make-golang updating some standards¹⁶. For Packer I've also updated azure-go-autorest and azure-sdk as team upload (#821938, #821832), but it came out that the project which is currently under heavy development towards a new official release broke a lot in the past weeks (no Git branching have been used), so that Packer as a matter of fact needed a vendored snapshot, although there have been only a couple of commits in between. Docker-registry has the same problem with the new package of azure-sdk/2.1.1~beta1, so that it needed to be fixed, too (#822146). By the way, the tool ratt¹⁷ comes very handy for automatically test building down all reverse dependencies, not only for Go packages (thanks to Tianon Gravi for the tip). Finally, I've posted the needed reverse depencies as RFP bugs for Terraform¹⁸ (again quite a lot), Vuls¹⁹, and cve-dictionary²⁰, which is needed for Vuls. I'll let them rest a while waiting to get picked up before working anything down.

#570933: dh-buildinfo: should provide a addon file for dh command
https://tracker.debian.org/pkg/debops
http://debops.org/
The servers have to be accessible by SSH. E.g. you could run debops like: $ debops -u root --private-key=~/.ssh/id_digitalocean
https://github.com/debops/docs/issues/132
#819816: ITP: debops -- Ansible based server management utility
https://www.packer.io/
#740753: ITP: packer -- create vm images for multiple platforms
http://www.danielstender.com/blog/packer-qemu.html
https://packages.debian.org/unstable/packer
I've worked on the missing packages this month, namely github-klauspost-pgzip, github-masterzen-xmlpath, github-masterzen-winrm, dylanmei-winrmtest, packer-community-winrmcp (Packer uses WinRM if Windows machines images are created), github-hpcloud-tail, and updated github-rackspace-gophercloud (#822163) and google-api (#822164) to complete it.
http://squirrel-lang.org/
https://tracker.debian.org/pkg/squirrel3
http://www.linux-magazin.de/layout/set/print/content/view/full/62184
http://www.linux-magazin.de/Ausgaben/2011/10/plus/Fremdkoerper-Squirrel-Interpreter-und-Skripte-fuer-C
https://github.com/Debian/dh-make-golang/pull/39
https://packages.debian.org/unstable/ratt
#808940: ITP: terraform -- tool for managing cloud infrastructure
#820614: ITP: vuls -- package inventory scanner for CVE vulnerabilities
#820615: ITP: go-cve-dictionary -- builds a local copy of the NVD/JVN (vulnerability databases)

27 April 2016

Stig Sandbeck Mathisen: Using LLDP on Linux. What's on the other side?

On any given server, or workstation, knowing what is at the other end of the network cable is often very useful. There s a protocol for that: LLDP. This is a link layer protocol, so it is not routed. Each end transmits information about itself periodically. You can typically see the type of equipment, the server or switch name, and the network port name of the other end, although there are lots of other bits of information available, too. This is often used between switches and routers in a server centre, but it is useful to enable on server hardware as well. There are a few different packages available. I ve looked at a few of them available for the RedHat OS family (Red Hat Enterprise Linux, CentOS, ) as well as the Debian OS family (Debian, Ubuntu, ) (Updated 2016-04-29, added more recent information about lldpd, and gathered the switch output at the end.)

ladvd A simple daemon, with no configuration needed. This runs as a privilege-separated daemon, and has a command line control utility. You invoke it with a list of interfaces as command line arguments to restrict the interfaces it should use. ladvd is not available on RedHat, but is available on Debian. Install the ladvd package, and run ladvdc to query the daemon for information.

root@turbotape:~# ladvdc
Capability Codes:
    r - Repeater, B - Bridge, H - Host, R - Router, S - Switch,
    W - WLAN Access Point, C - DOCSIS Device, T - Telephone, O - Other
Device ID        Local Intf Proto Hold-time Capability Port ID
office1-switch23 eno1       LLDP  98        B          42

Even better, it has output that can be parsed for scripting:

root@turbotape:~# ladvdc -b eno1
INTERFACE_0='eno1'
HOSTNAME_0='office1-switch23'
PORTNAME_0='42'
PORTDESCR_0='42'
PROTOCOL_0='LLDP'
ADDR_INET4_0=''
ADDR_INET6_0=''
ADDR_802_0='00:11:22:33:44:55
VLAN_ID_0=''
CAPABILITIES_0='B'
TTL_0='120'
HOLDTIME_0='103'

my new favourite :)

lldpd Another package is lldpd , which is also simple to configure and use. lldpd is not available on RedHat, but it is present on Debian. It features a command line interface, lldpcli , which can show output with different level of detail, and on different formats, as well as configure the running daemon.

root@turbotape:~# lldpcli show neighbors
-------------------------------------------------------------------------------
LLDP neighbors:
-------------------------------------------------------------------------------
Interface:    eno1, via: LLDP, RID: 1, Time: 0 day, 00:00:59
  Chassis:
    ChassisID:    mac 00:11:22:33:44:55
    SysName:      office1-switch23
    SysDescr:     ProCurve J9280A Switch 2510G-48, revision Y.11.12, ROM N.10.02 (/sw/code/build/cod(cod11))
    Capability:   Bridge, on
  Port:
    PortID:       local 42
    PortDescr:    42
-------------------------------------------------------------------------------

Among the output formats are json , which is easy to re-use elsewhere.

root@turbotape:~# lldpcli -f json show neighbors

 
  "lldp":  
    "interface":  
      "eno1":  
        "chassis":  
          "office1-switch23":  
            "descr": "ProCurve J9280A Switch 2510G-48, revision Y.11.12, ROM N.10.02 (/sw/code/build/cod(cod11))",
            "id":  
              "type": "mac",
              "value": "00:11:22:33:44:55"
             ,
            "capability":  
              "type": "Bridge",
              "enabled": true
             
           
         ,
        "via": "LLDP",
        "rid": "1",
        "age": "0 day, 00:53:23",
        "port":  
          "descr": "42",
          "id":  
            "type": "local",
            "value": "42"

lldpad A much more featureful LLDP daemon, available for both the Debian and RedHat OS families. This has lots of features, but is less trivial to set up.

Configure lldp for each interface

#!/bin/sh
find /sys/class/net/ -maxdepth 1 -name 'en*'  
    while read device; do
        basename "$device"
    done  
    while read interface; do
         
            lldptool set-lldp -i "$interface" adminStatus=rxtx
            for item in sysName portDesc sysDesc sysCap mngAddr; do
                lldptool set-tlv -i "$interface" -V "$item" enableTx=yes  
                    sed -e "s/^/$item /"
            done
           
            sed -e "s/^/$interface /"
    done

Show LLDP neighbor information

#!/bin/sh
find /sys/class/net/ -maxdepth 1 -name 'en*'  
    while read device; do
        basename "$device"
    done  
    while read interface; do
        printf "%s\n" "$interface"
        ethtool $interface   grep -q 'Link detected: yes'    
            echo "  down"
            echo
            continue
         
        lldptool get-tlv -n -i "$interface"   sed -e "s/^/  /"
        echo
    done

[...]
enp3s0f0
  Chassis ID TLV
    MAC: 01:23:45:67:89:ab
  Port ID TLV
    Local: 588
  Time to Live TLV
    120
  System Name TLV
    site3-row2-rack1
  System Description TLV
    Juniper Networks, Inc. ex2200-48t-4g , version 12.3R12.4 Build date: 2016-01-20 05:03:06 UTC
  System Capabilities TLV
    System capabilities:  Bridge, Router
    Enabled capabilities: Bridge, Router
  Management Address TLV
    IPv4: 10.21.0.40
    Ifindex: 36
    OID: $
  Port Description TLV
    some important server, port 4
  MAC/PHY Configuration Status TLV
    Auto-negotiation supported and enabled
    PMD auto-negotiation capabilities: 0x0001
    MAU type: Unknown [0x0000]
  Link Aggregation TLV
    Aggregation capable
    Currently aggregated
    Aggregated Port ID: 600
  Maximum Frame Size TLV
    9216
  Port VLAN ID TLV
    PVID: 2000
  VLAN Name TLV
    VID 2000: Name bumblebee
  VLAN Name TLV
    VID 2001: Name stumblebee
  VLAN Name TLV
    VID 2002: Name fumblebee
  LLDP-MED Capabilities TLV
    Device Type:  netcon
    Capabilities: LLDP-MED, Network Policy, Location Identification, Extended Power via MDI-PSE
  End of LLDPDU TLV
enp3s0f1
[...]

on the switch side On the switch, it is a bit easier to see what s connected to each interface:

office switch On the switch side, this system looks like:

office1-switch23# show lldp info remote-device
 LLDP Remote Devices Information
  LocalPort   ChassisId                 PortId PortDescr SysName
  --------- + ------------------------- ------ --------- ----------------------
  [...]
  42          22 33 44 55 66 77         eno1   Intel ... turbotape.example.com
  [...]
office1-switch23# show lldp info remote-device 42
 LLDP Remote Device Information Detail
  Local Port   : 42
  ChassisType  : mac-address
  ChassisId    : 00 11 22 33 33 55
  PortType     : interface-name
  PortId       : eno1
  SysName      : turbotape.example.com
  System Descr : Debian GNU/Linux testing (stretch) Linux 4.5.0-1-amd64 #1...
  PortDescr    : Intel Corporation Ethernet Connection I217-LM
  System Capabilities Supported  : bridge, router
  System Capabilities Enabled    : bridge, router
  Remote Management Address
     Type    : ipv4
     Address : 192.0.2.93
     Type    : ipv6
     Address : 20 01 0d b8 00 00 00 00 00 00 00 00 00 00 00 01
     Type    : all802
     Address : 22 33 44 55 66 77

datacenter switch

ssm@site3-row2-rack1> show lldp neighbors
Local Interface    Parent Interface    Chassis Id          Port info          System Name
[...]
ge-0/0/38.0        ae1.0               01:23:45:67:89:58   Interface   2 as enp3s0f0 server.example.com
ge-1/0/38.0        ae1.0               01:23:45:67:89:58   Interface   3 as enp3s0f1 server.example.com
[...]
ssm@site3-row2-rack1> show lldp neighbors interface ge-0/0/38
LLDP Neighbor Information:
Local Information:
Index: 157 Time to live: 120 Time mark: Fri Apr 29 13:00:19 2016 Age: 24 secs
Local Interface    : ge-0/0/38.0
Parent Interface   : ae1.0
Local Port ID      : 588
Ageout Count       : 0
Neighbour Information:
Chassis type       : Mac address
Chassis ID         : 01:23:45:67:89:58
Port type          : Mac address
Port ID            : 01:23:45:67:89:58
Port description   : Interface   2 as enp3s0f0
System name        : server.example.com
System Description : Linux server.example.com 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 4
System capabilities
        Supported  : Station Only
        Enabled    : Station Only
Management Info
        Type              : IPv6
        Address           : 2001:0db8:0000:0000:0000:dead:beef:cafe
        Port ID           : 2
        Subtype           : 2
        Interface Subtype : ifIndex(2)
        OID               : 1.3.6.1.2.1.31.1.1.1.1.2

26 April 2016

Niels Thykier: Putting Debian packages in labelled boxes

Lintian 2.5.44 was released the other day and (to most) the most significant bug fix was probably that Lintian learned about Policy 3.9.8. I would like to thank Axel Beckert for doing that. Notably it also made me update the test suite so to make future policy releases less painful. For others, it might be the fact that Lintian now accepts (valid) versioned provides (which seemed prudent now that Britney accepts them as well). Newcomers might appreciate that we are giving a much more sensible warning when they have extra spaces in their changelog sign off line (rather than pretending it is an improper NMU). But I digress What I am here to talk about is that Lintian 2.5.44 started classifying packages based on various facts or properties , we can determine. Therefore:

Every package will have at least one tag now!
These labels are known as something called classification tags .
The tags are not issues to be fixed! (I will repeat this later to ensure you get this point!)

Here are some of the labelled boxes your packages will be put into[0]:

Source packages will unconditionally have both:
- debian-build-system
- source-format [1]
Binary packages will have one of:
- ctrl-script
- no-ctrl-scripts

The tags themselves are (as mentioned) mere classifications and their primary purpose is to classify or measure certain properties. With them any body can download the data set and come with some bold statement about Debian packages (hopefully without relying too much on lies, damned lies and statistics ). Lets try that immediately!

Almost 75% of all Debian packages do not need to run arbitrary code doing installation[2]!
The dh-sequencer with cdbs is the future![3]

In the next release, we will also add tracking of auto-generated snippets from dh_*-tools. Currently unversioned, but I hope to add versioning to that so we can find and rebuild packages that have been built with buggy autoscripts (like #788098) If you want to see the classification tags for your package, please run lintian with like this:

# Add classification tags
$ lintian -L +classification <pkg-or-changes>
# Or if you want only classification tags$ lintian -L =classification <pkg-or-changes>

Please keep in mind that classification tags ( C ) are not issues in themselves. Lintian is simply attempting to add a visible indicator about a given fact or property in the package nothing more, nothing less. Future work help (read: patches) welcome:

Do something better with classification tags on lintian.d.o (like stacked graphs)
Classify more (useful) things.
Optimising the memory usage of the reporting framework now that we permanently increased the number of tags.
Your idea (with patches) here

[0] Mind you, the reporting framework s handling of these tags could certainly be improved. [1] Please note how it distinguishes 1.0 into native and non-native based on whether the package has a diff.gz. Presumably that can be exploited somehow [2] Disclaimer: At the time of writing, only ~80% of the archive have been processed. This is computed as: NS / (NS + WS), where NS and WS are the number of unique packages with the tags no-ctrl-scripts and ctrl-script respectively. [3] or maybe not, but we got two packages classified as using both CDBS and the dh-sequencer. I have not looked at it in detail. For the curious: libmecab-java and ctioga2.
Filed under: Debian, Lintian

Reproducible builds folks: Reproducible builds: week 52 in Stretch cycle

What happened in the Reproducible Builds effort between April 17th and April 23rd 2016: Toolchain fixes Thomas Weber uploaded lcms2/2.7-1 which will not write uninitialized memory when writing color names. Original patch by Lunar. The GCC 7 development phase has just begun, so Dhole reworked his patch to make gcc use SOURCE_DATE_EPOCH if set which prompted interesting feedback, but it has not been merged yet. Alexis Bienven e submitted a patch for sphinx to strip Python object memory addresses from the generated documentation. Packages fixed The following packages have become reproducible due to changes in their build dependencies: cobertura, commons-pool, easymock, eclipselink, excalibur-logkit, gap-radiroot, gluegen2, jabref, java3d, jcifs, jline, jmock2, josql, jtharness, libfann, libgroboutils-java, libjemmy2-java, libjgoodies-binding-java, libjgrapht0.8-java, libjtds-java, liboptions-java, libpal-java, libzeus-jscl-java, node-transformers, octave-msh, octave-secs2d, openmama, rkward. The following packages have become reproducible after being fixed:

auto-multiple-choice/1.3.0-1 by Alexis Bienven e.
castle-game-engine/5.2.0-3 by Paul Gevers.
gap-autpgrp/1.5-2 by Bill Allombert.
gap-grape/4r7+ds-2 by Jerome Benoit.
letsencrypt.sh/0.1.0-2 by Mattia Rizzolo.
libindicate/0.6.92-3 by Adam Borowski.
ltrsift/1.0.2-7 by Sascha Steinbiss.
perl/5.22.1-10 by Niko Tyni.
php-pear/1:1.10.1+submodules+notgz-7 by Mathieu Parent.
python-pint/0.7.2-2 by Ond ej Nov .
spades/3.7.1+dfsg-3 by Sascha Steinbiss.
tex4ht/20090611-2 by Reiner Herrmann.

Patches submitted that have not made their way to the archive yet:

#821356 against emoslib by boyska: use echo in a portable manner across shells.
#822268 against transdecoder by Dhole: set PERL_HASH_SEED=0 when calling the scripts that generate samples.

tests.reproducible-builds.org

Steven Chamberlain investigated the performance of our armhf boards which also provided a nice overview of our armhf build network.
As i386 has almost been completely tested the order of the architectures displayed has been changed to reflect the fact that i386 is now the 2nd most popular architecture in Debian. (h01ger)
In order to decrease the number of blacklisted packages, the first build is now run with a timeout of 18h (previously: 12h) and the 2nd with 24h timeout (previously: 18h). (h01ger)
We now also vary the CPU model on amd64 (and soon on i386 too) so that one build is performed using a "AMD Opteron 62xx class CPU" while the other is done using a "Intel Core Processor (Haswell)". This is now possible as proftitbricks.com offers VMs running both types of CPU and have generously increased their sponsorship once more. (h01ger)
Profitbricks increased our storage space by 400 GB which will be used to setup a 2nd build node for the coreboot/OpenWrt/NetBSD/Arch Linux/Fedora tests. This 2nd build node will run 398 days in the future for testing reproducibility on a different date.

diffoscope development diffoscope 52 was released with changes from Mattia Rizzolo, h01ger, Satyam Zode and Reiner Herrmann, who also did the release. Notable changes included:

Drop transitional debbindiff package.
Let objdump demangle symbols for better readability.
Install bin/diffoscope instead of auto-generated script. (Closes: #821777)

As usual, diffoscope 52 is available on Debian, Archlinux and PyPI, other distributions will hopefully soon update. Package reviews 28 reviews have been added, 11 have been updated and 94 have been removed in this week. 14 FTBFS bugs were reported by Chris Lamb (one being was a duplicate of a bug filed by Sebastian Ramacher an hour earlier). Misc. This week's edition was written by Lunar, Holger 'h01ger' Levsen and Chris Lamb and reviewed by a bunch of Reproducible builds folks on IRC.

Next.

Previous.