Search Results: "ana"

9 August 2015

Petter Reinholdtsen: First paper version of the Norwegian Free Culture book heading my way

Typesetting a book is harder than I hoped. As the translation is mostly done, and a volunteer proof reader was going to check the text on paper, it was time this summer to focus on formatting my translated docbook based version of the Free Culture book by Lawrence Lessig. I've been trying to get both docboox-xsl+fop and dblatex to give me a good looking PDF, but in the end I went with dblatex, because its Debian maintainer and upstream developer were responsive and very helpful in solving my formatting challenges. Last night, I finally managed to create a PDF that no longer made Lulu.com complain after uploading, and I ordered a text version of the book on paper. It is lacking a proper book cover and is not tagged with the correct ISBN number, but should give me an idea what the finished book will look like. Instead of using Lulu, I did consider printing the book using CreateSpace, but ended up using Lulu because it had smaller book size options (CreateSpace seem to lack pocket book with extended distribution). I looked for a similar service in Norway, but have not seen anything so far. Please let me know if I am missing out on something here. But I still struggle to decide the book size. Should I go for pocket book (4.25x6.875 inches / 10.8x17.5 cm) with 556 pages, Digest (5.5x8.5 inches / 14x21.6 cm) with 323 pages or US Trade (6x8 inches / 15.3x22.9 cm) with 280 pages? Fewer pager give a cheaper book, and a smaller book is easier to carry around. The test book I ordered was pocket book sized, to give me an idea how well that fit in my hand, but I suspect I will end up using a digest sized book in the end to bring the prize down further. My biggest challenge at the moment is making nice cover art. My inkscape skills are not yet up to the task of replicating the original cover in SVG format. I also need to figure out what to write about the book on the back (will most likely use the same text as the description on web based book stores). I would love help with this, if you are willing to license the art source and final version using the same CC license as the book. My artistic skills are not really up to the task. I plan to publish the book in both English and Norwegian and on paper, in PDF form as well as EPUB and MOBI format. The current status can as usual be found on github in the archive/ directory. So far I have spent all time on making the PDF version look good. Someone should probably do the same with the dbtoepub generated e-book. Help is definitely needed here, as I expect to run out of steem before I find time to improve the epub formatting. Please let me know via github if you find typos in the book or discover translations that should be improved. The final proof reading is being done right now, and I expect to publish the finished result in a few months.

7 August 2015

Norbert Preining: Japanese-English dictionary for Kobo

Since ever the Kobo firmwares also allowed downloading of a bunch of dictionaries, most of which I don t need. As I am fluent in most languages I read and write, the only real dictionary I would like to see is a Japanese-English (I don t dare asking for a Japanese-German). Unfortunately, Kobo never shipped one. OTOH, starting with firmware 3.16.10 they ships two different English-Japanese dictionaries, one excellent Japanese-Japanese dictionary, but not one Japanese-English. So I took the liberty to write a script that allows everyone to enrich the shipped Japanese-Japanese dictionary with English definitions. Kobo Logo Till now I have used Tshering s excellent Japanese dictionary which was enriched with English definitions. It was based on the older Japanese-Japanese dictionary shipped with firmwares before 3.16.10. I didn t want to loose the more complete dictionary from the new firmware, so here is a script that updates the dictionary for you Prerequisites The script is neither fool-proof nor does everything by itself. Furthermore, it requires a set of programs. In details: Supported dictionaries At the moment the program can use the following two dictionaries as sources: Edict2 and Japanese3. Edict2 The Edict dictionary is a free dictionary which is the base of most other dictionaries. Created by the JMdict/EDICT Project it provides a very complete Japanese-English dictionary. To use the dict with the current program, one need to download the edict2.gz file and unpack it with gunzip edict2.gz. If you put it into the same directory as the Perl script, nothing else needs to be done, otherwise one can use the command line option --edict PATH-TO-EDICT to specify the location of the edict2 file. Japanese3 The Japanese3 is a dictionary application for iOS which provides another very complete Japanese-English dictionary. My feeling is that it is 90% and more based on Edict, so adding it will not buy you much, but still a bit (see below for how much!). If you have purchased this dictionary/application, and you manage to get access to your iOS device (via jailbreaking or some other tools), then you need to get the file Japanese3.db from the application folder, and then generate a file via sqlite3 as follows: 
$ sqlite3 Japanese3.db
.output japanese3-data
select Entry, Furigana, Summary from entries;
If you save the generated file japanese3-data into the same directory as the Perl script, nothing else needs to be done, otherwise one can use the command line option --japanese3 PATH-TO-J3-DATA to specify the location of the Japanese3 data file. Command line options The program supports the following command line options: Note that in case you pass in the option --unpacked, the files should be properly named (encodings are a problem!). Furthermore, note that the unpacked directory contains lots of .gif and .html files that are actually gzip-compressed files, but lacking the .gz extension. If you want to unpack the dictionary, by advised that the file names in the Zip directory are already encoded in UTF-8, but normal programs (unzip, 7z) assume that they are encoded in some other encoding. Thus, if your are using UTF-8 locale, it is necessary to set LC_CTYPE to C to make sure that the encoding is used, as in 
LC_CTYPE=C 7z x ...
Typical run In the following example we use both dict2 and Japanese3 dictionaries, and prefer Edict2: 
$ perl enhance-dictionary.pl --dicts edict2 --dicts japanese3
Using the following dictionaries as source for translations: edict2 japanese3
loading edict2 ... done
loading Japanese3 data ... done
unpacking original dictionary ... done
loading dict files ... done
searching for words and updating ... done
total words 805521, matches: 130563 (edict: 123388, japanese3: 7175)
creating output html ... done
creating update dictionary in dicthtml-jaxxdjs-201508072201.zip ... done
$
Installing the dictionary After having created the enhanced dictionary, one can install the generated file as KOBO/.kobo/dicts/dicthtml-jaxxdjs.zip (where KOBO is the mount point of the eReader). The dictionary should be picked up automatically. And the next lookup should give you something like the following:
kobo-glohd-enhanced-ja-dict There is only one caveat: Syncing with Kobo will re-download the original dictionary and overwrite the enhanced one. There is at least one solution here, which I am employing, see this post on MobileRead. Download and development place I am using the github repository kobo-ja-dict-enhance to develop the program. Please report bugs, and feature requests there. The file for the last released version can be downloaded from here: enhance-dictionary.pl Future plans I am planning to get rid of the 7z dependency and use Archive::Zip for all the unpacking and packing. This would also allow to do everything in memory and thus make it faster. There is a branch on github with ongoing work on this matter. I also think about including translations for Hiragana words by providing the translations of all possible Kanji readings. Conclusions Even if Kobo does not provide us with a decent Japanese-English dictionary, adding at least a huge amount of translations to the current dictionary is now easily possible. For serious students of Japanese who are reading or starting to read Japanese eBooks, this will be of great help. Enjoy, and don t forget to give feedback, suggestions, and improvements either here or better via the Github page.

Guido G nther: Debian work in July 2015

July was the third month I contributed to Debian LTS under the Freexian umbrella. In total I spent eight hours working on: Besides that I did CVE triaging of 11 CVEs to check if and how they affect oldoldstable security as part of my LTS front desk work.

Christoph Egger: Systemd pitfalls

logind hangs If you just updated systemd and ssh to that host seems to hang, that's just a known Bug (Debian Bug). Don't panic. Wait for the logind timeout and restart logind. restart and stop;start One thing that confused me several times and still confuses people is systemctl restart doing more than systemctl stop ; systemctl start. You will notice the difference once you have a failed service. A restart will try to start the service again. Both stop and start however will just ignore it. Rumors have it this has changed post jessie however. sysvinit-wrapped services and stop While there are certainly bugs with sysvinit services in general (I found myself several times without a local resolver as unbound failed to be started, haven't managed to debug further), the stop behavior of wrapped services is just broken. systemctl stop will block until the sysv initscript finished. It will even note the result of the action in its state. However systemctl will return with exitcode 0 and not output anything on stdout/stderr. This has been reported as Debian Bug. zsh helper I found the following zshrc snipped quite helpful in dealing with non-reported systemctl failures. On root shells it will display a list of failed services as part of the prompt. This will give proper feedback whether your systemctl stop failed, it will give feedback if you still have type=simple services and if the sysv-init script or wrapper is broken. 
precmd ()  
    if [[ $UID == 0 && $+commands[systemctl] != 0 ]]
    then
      use_systemd=true
      systemd_failed=" systemctl --state=failed   grep failed   cut -d \  -f 2   tr '\n' ' ' "
    fi
 
if [[ $UID == 0 && $+commands[systemctl] != 0 ]]
then
  PROMPT=$'% $fg[red]>>  $systemd_failed$reset_color% \n'
else
  PROMPT=""
fi
PROMPT+=whateveryourpromptis
zsh completion Speaking of zsh, there's one problem that bothers me a lot and I don't have any solution for. Tab-completing the service name for service is blazing fast. Tab-completing the service name for systemctl restart takes ages. People traced down to truckloads of dbus communication for the completion but no further fix is known (to me). type=simple services As described in length by Lucas Nussbaum type=simple services are actively harmful. Proper type=forking daemons are strictly superior (they provide feedback of finished initialization and success thereof) and type=notify services are so simple there's no excuse for not using them even for private one-off hacks. Even if you're language doesn't provide libsystemd-daemon bindings: 
(defun sd-notify (event-string)
  (let ((socket (make-instance 'sb-bsd-sockets:local-socket :type :datagram))
        (name (posix-getenv "NOTIFY_SOCKET"))
        (bufferlen (length event-string)))
    (when name
      (sb-bsd-sockets:socket-connect socket name)
      (sb-bsd-sockets:socket-send socket event-string bufferlen))))
This is a stable API guaranteed to not break in the future and implemented in less than ten lines of code with just basic socket functions. And if your language has support it becomes actually trivial: 
    try:
        import systemd
        systemd.daemon.notify("READY=1")
    except ImportError:
        pass
Note that in both cases there is no drawback at all on systemd-free setups. It has the overhead of checking the process' environment for NOTIFY_SOCKET or for the systemd package and behaves like a simple service otherwise. Actually the idea of separating the technical aspect (daemonizing) from the semantic aspect of signalizing "initialization finished, everything's fine" is a pretty good idea and hopefully has the potential to reduce the number of services signalizing the "everything's fine" too early. It could even be ported to non-systemd init systems easily given the API.

Christoph Egger: Systemd pitfalls

logind hangs If you just updated systemd and ssh to that host seems to hang, that's just a known Bug (Debian Bug #770135). Don't panic. Wait for the logind timeout and restart logind. restart and stop;start One thing that confused me several times and still confuses people is systemctl restart doing more than systemctl stop ; systemctl start. You will notice the difference once you have a failed service. A restart will try to start the service again. Both stop and start however will just ignore it. Rumors have it this has changed post jessie however. sysvinit-wrapped services and stop While there are certainly bugs with sysvinit services in general (I found myself several times without a local resolver as unbound failed to be started, haven't managed to debug further), the stop behavior of wrapped services is just broken. systemctl stop will block until the sysv initscript finished. It will even note the result of the action in its state. However systemctl will return with exitcode 0 and not output anything on stdout/stderr. This has been reported as Debian Bug #792045. zsh helper I found the following zshrc snipped quite helpful in dealing with non-reported systemctl failures. On root shells it will display a list of failed services as part of the prompt. This will give proper feedback whether your systemctl stop failed, it will give feedback if you still have type=simple services and if the sysv-init script or wrapper is broken. 
precmd ()  
    if [[ $UID == 0 && $+commands[systemctl] != 0 ]]
    then
      use_systemd=true
      systemd_failed=" systemctl --state=failed   grep failed   cut -d \  -f 2   tr '\n' ' ' "
    fi
 
if [[ $UID == 0 && $+commands[systemctl] != 0 ]]
then
  PROMPT=$'% $fg[red]>>  $systemd_failed$reset_color% \n'
else
  PROMPT=""
fi
PROMPT+=whateveryourpromptis
zsh completion Speaking of zsh, there's one problem that bothers me a lot and I don't have any solution for. Tab-completing the service name for service is blazing fast. Tab-completing the service name for systemctl restart takes ages. People traced down to truckloads of dbus communication for the completion but no further fix is known (to me). type=simple services As described in length by Lucas Nussbaum type=simple services are actively harmful. Proper type=forking daemons are strictly superior (they provide feedback of finished initialization and success thereof) and type=notify services are so simple there's no excuse for not using them even for private one-off hacks. Even if you're language doesn't provide libsystemd-daemon bindings: 
(defun sd-notify (event-string)
  (let ((socket (make-instance 'sb-bsd-sockets:local-socket :type :datagram))
        (name (posix-getenv "NOTIFY_SOCKET"))
        (bufferlen (length event-string)))
    (when name
      (sb-bsd-sockets:socket-connect socket name)
      (sb-bsd-sockets:socket-send socket event-string bufferlen))))
This is a stable API guaranteed to not break in the future and implemented in less than ten lines of code with just basic socket functions. And if your language has support it becomes actually trivial: 
    try:
        import systemd
        systemd.daemon.notify("READY=1")
    except ImportError:
        pass
Note that in both cases there is no drawback at all on systemd-free setups. It has the overhead of checking the process' environment for NOTIFY_SOCKET or for the systemd package and behaves like a simple service otherwise. Actually the idea of separating the technical aspect (daemonizing) from the semantic aspect of signalizing "initialization finished, everything's fine" is a pretty good idea and hopefully has the potential to reduce the number of services signalizing the "everything's fine" too early. It could even be ported to non-systemd init systems easily given the API.

Mike Gabriel: New plugin for GOsa : gosa-plugin-mailaddress

During last week, I hacked a new plugin together for GOsa . Simply quoting parts from debian/control here to inform you on its functionality:
Package: gosa-plugin-mailaddress
Architecture: all
Depends:
$ misc:Depends ,
gosa (>= 2.7),
Conflicts:
gosa-plugin-mail,
Description: Simple plugin to manage user mail addresses in GOsa
This plugin is a very light-weighted version of the GOsa mail plugin.
Whereas gosa-plugin-mail can be used to manage a complete mail server
farm, this tiny plugin only provides means to modify the user's mail
address via a text field.
.
This plugin is useful for people that need to maintain users' email
addresses via GOsa , but do not run their own mailserver(s).
.
GOsa is a combination of system-administrator and end-user web
interface, designed to handle LDAP based setups.
 light+love
Mike (aka sunweaver)

John Goerzen: Detailed Smart Card Cryptographic Token Security Guide

After my first post about smartcards under Linux, I thought I would share some information I ve been gathering. This post is already huge, so I am not going to dive into much specific commands, but I am linking to many sources with detailed instructions. I ve reviewed several types of cards. For this review, I will focus on the OpenPGP card and the Yubikey NEO, since the Cardomatic Smartcard-HSM is not supported by the gpg version in Jessie. Both cards are produced by people with strong support for the Free Software ecosystem and have strong cross-platform support with source code. OpenPGP card: Basics with GnuPG The OpenPGP card is well-known as one of the first smart cards to work well on Linux. It is a single-application card focused on use with GPG. Generally speaking, by the way, you want GPG2 for use with smartcards. Basically, this card contains three slots: decryption, signing, and authentication slots. The concept is that the private key portions of the keys used for these items are stored only on the card, can never be extracted from the card, and the cryptographic operations are performed on the card. There is more information in my original post. In a fairly rare move for smartcards, this card supports 4096-byte RSA keys; most are restricted to 2048-byte keys. The FSF Europe hands these out to people and has a lot of good information about them online, including some HOWTOs. The official GnuPG smart card howto is 10 years old, and although it has some good background, I d suggest using the FSFE instructions instead. As you ll see in a bit, most of this information also pertains to the OpenPGP mode of the Yubikey Neo. OpenPGP card: Other uses Of course, this is already pretty great to enhance your GPG security, but there s a lot more that you can do with this card to add two-factor authentication (2FA) to a lot of other areas. Here are some pointers: OpenPGP card: remote authentication with ssh You can store the private part of your ssh key on the card. Traditionally, this was only done by using the ssh agent emulation mode of gnupg-agent. This is still possible, of course. Now, however, the OpenSC project now supports the OpenPGP card as a PKCS#11 and PKCS#15 card, which means it works natively with ssh-agent as well. Try just ssh-add -s /usr/lib/x86_64-linux-gnu/pkcs11/opensc-pkcs11.so if you ve put a key in the auth slot with GPG. ssh-add -L will list its fingerprint for insertion into authorized_keys. Very simple! As an aside: Comments that you need scute for PKCS#11 support are now outdated. I do not recommend scute. It is quite buggy. OpenPGP card: local authentication with PAM You can authenticate logins to a local machine by using the card with libpam-poldi here are some instructions. Between the use with ssh and the use with PAM, we have now covered 2FA for both local and remote use in Unix environments. OpenPGP card: use on Windows Let s move on to Windows environments. The standard suggestion here seems to be the mysmartlogon OpenPGP mini-driver. It works with some sort of Windows CA system, or the local accounts using EIDAuthenticate. I have not yet tried this. OpenPGP card: Use with X.509 or Windows Active Directory You can use the card in X.509 mode via these gpgsm instructions, which apparently also work with Windows Active Directory in some fashion. You can also use it with web browsers to present a certificate from a client for client authentication. For example, here are OpenSC instructions for Firefox. OpenPGP card: Use with OpenVPN Via the PKCS#11 mode, this card should be usable to authenticate a client to OpenVPN. See the official OpenVPN HOWTO or these other instructions for more. OpenPGP card: a note on PKCS#11 and PKCS#15 support You ll want to install the opensc-pkcs11 package, and then give the path /usr/lib/x86_64-linux-gnu/pkcs11/opensc-pkcs11.so whenever something needs the PKCS#11 library. There seem to be some locking/contention issues between GPG2 and OpenSC, however. Usually killing pcscd and scdaemon will resolve this. I would recommend doing manipulation operations (setting PINs, generating or uploading keys, etc.) via GPG2 only. Use the PKCS#11 tools only to access. OpenPGP card: further reading Kernel Concepts also has some nice readers; you can get this card in a small USB form-factor by getting the mini-card and the Gemalto reader. Yubikey Neo Introduction The Yubikey Neo is a fascinating device. It is a small USB and NFC device, a little smaller than your average USB drive. It is a multi-application device that actually has six distinct modes: There is a ton to digest with this device. Yubikey Neo Basics By default, the Yubikey Neo is locked to only a subset of its features. Using the yubikey-personalization tool (you ll need the version in stretch; jessie is too old), you can use ykpersonalize -m86 to unlock the full possibilities of the card. Run that command, then unplug and replug the device. It will present itself as a USB keyboard as well as a PC/SC-compatible card reader. It has a capacitive button, which is used to have it generate keystrokes to input validation information for HOTP or HMAC validation. It has two slots that can be configured with HMAC and HOTP; a short button press selects the default slot #1 and a long press selects slot #2. But before we get into that, let s step back at some basics. opensc-tool list-algorithms claims this card supports RSA with 1024, 2048, and 3072 sizes, and EC with 256 and 384-bit sizes. I haven t personally verified anything other than RSA-2048 though. Yubikey Neo: OpenPGP support In this mode, the card is mostly compatible with the physical OpenPGP card. I say mostly because there are a few protocol differences I ll get into later. It is also limited to 2048-byte keys. Support for this is built into GnuPG and the GnuPG features described above all work fine. In this mode, it uses firmware from the Yubico fork of the JavaCard OpenPGP Card applet. There are Yubico-specific tutorials available, but again, most of the general GPG stuff applies. You can use gnupg-agent to use the card with SSH as before. However, due to some incompatibilities, the OpenPGP applet on this card cannot be used as a PKCS#11 card with either scute or OpenSC. That is not exactly a huge problem, however, as the card has another applet (PIV) that is compatible with OpenSC and so this still provides an avenue for SSH, OpenVPN, Mozilla, etc. It should be noted that the OpenPGP applet on this card can also be used with NFC on Android with the OpenKeychain app. Together with pass (or its Windows, Mac, or phone ports), this makes a nicely secure system for storing passwords. Yubikey Neo: PKCS#11 with the PIV applet There is also support for the PIV standard on the Yubikey Neo. This is supported by default on Linux (via OpenSC) and Windows and provides a PKCS#11-compabible store. It should, therefore, be compatible with ssh-agent, OpenVPN, Active Directory, and all the other OpenPGP card features described above. The only difference is that it uses storage separate from the OpenPGP applet. You will need one of the Yubico PIV tools to configure the key for it; in Debian, the yubico-piv-tool from stretch does this. Here are some instructions on using the Yubikey Neo in PIV mode: A final note: for security, it s important to change the management key and PINs before deploying the PIV mode. I couldn t get this to work with Firefox, but it worked pretty much everywhere else. Yubikey Neo: HOTP authentication This is the default mode for your Yubikey; all other modes require enabling with ykpersonalize. In this mode, a 128-bit AES key stored on the Yubikey is used to generate one-time passwords (OTP). (This key was shared in advance with the authentication server.) A typical pattern would be for three prompts: username, password, and Yubikey HOTP. The user clicks in the Yubikey HOTP field, touches the Yubikey, and their one-time token is pasted in. In the background, the service being authenticated to contacts an authentication server. This authentication server can be either your own (there are several open source implementations in Debian) or the free Yubicloud. Either way, the server decrypts the encrypted part of the OTP, performs validity checks (making sure that the counter is larger than any counter it s seen before, etc) and returns success or failure back to the service demanding authentication. The first few characters of the posted auth contain the unencrypted key ID, and thus it can also be used to provide username if desired. Yubico has provided quite a few integrations and libraries for this mode. A few highlights: You can also find some details on the OTP mode. Here s another writeup. This mode is simple to implement, but it has a few downsides. One is that it is specific to the Yubico line of products, and thus has a vendor lock-in factor. Another is the dependence on the authentication server; this creates a potential single point of failure and can be undesireable in some circumtances. Yubikey Neo: OATH and HOTP and TOTP First, a quick note: OATH and OAuth are not the same. OATH is an authentication protocol, and OAuth is an authorization protocol. Now then Like Yubikey HOTP, OATH (both HOTP and TOTP) modes rely on a pre-shared key. (See details in the Yubico article.) Let s talk about TOTP first. With TOTP, there is a pre-shared secret with each service. Each time you authenticate to that service, your TOTP generator combines the timestamp with the shared secret using a HMAC algorithm and produces a OTP that changes every 30 seconds. Google Authenticator is a common example of this protocol, and this is a drop-in replacement for it. Gandi has a nice description of it that includes links to software-only solutions on various platforms as well. With the Yubikey, the shared secrets are stored on the card and processed within it. You cannot extract the shared secret from the Yubikey. Of course, if someone obtains physical access to your Yubikey they could use the shared secret stored on it, but there is no way they can steal the shared secret via software, even by compromising your PC or phone. Since the Yubikey does not have a built-in clock, TOTP operations cannot be completed solely on the card. You can use a PC-based app or the Android application (Play store link) with NFC to store secrets on the device and generate your TOTP codes. Command-line users can also use the yubikey-totp tool in the python-yubico package. OATH can also use HOTP. With HOTP, an authentication counter is used instead of a clock. This means that HOTP passwords can be generated entirely within the Yubikey. You can use ykpersonalize to configure either slot 1 or 2 for this mode, but one downside is that it can really only be used with one service per slot. OATH support is all over the place; for instance, there s libpam-oath from the OATH toolkit for Linux platforms. (Some more instructions on this exist.) Note: There is another tool from Yubico (not in Debian) that can apparently store multiple TOTP and HOTP codes in the Yubikey, although ykpersonalize and other documentation cannot. It is therefore unclear to me if multiple HOTP codes are supported, and how.. Yubikey Neo: Challenge-Response Mode This can be useful for doing offline authentication, and is similar to OATH-HOTP in a sense. There is a shared secret to start with, and the service trying to authenticate sends a challenge to the token, which must supply an appropriate response. This makes it only suitable for local authentication, but means it can be done fairly automatically and optionally does not even require a button press. To muddy the waters a bit, it supports both Yubikey OTP and HMAC-SHA1 challenge-response modes. I do not really know the difference. However, it is worth noting that libpam-yubico works with HMAC-SHA1 mode. This makes it suitable, for instance, for logon passwords. Yubikey Neo: U2F U2F is a new protocol for web-based apps. Yubico has some information, but since it is only supported in Chrome, it is not of interest to me right now. Yubikey Neo: Further resources Yubico has a lot of documentation, and in particular a technical manual that is actually fairly detailed. Closing comments Do not think a hardware security token is a panacea. It is best used as part of a multi-factor authentication system; you don t want a lost token itself to lead to a breach, just as you don t want a compromised password due to a keylogger to lead to a breach. These things won t prevent someone that has compromised your PC from abusing your existing ssh session (or even from establishing new ssh sessions from your PC, once you ve unlocked the token with the passphrase). What it will do is prevent them from stealing your ssh private key and using it on a different PC. It won t prevent someone from obtaining a copy of things you decrypt on a PC using the Yubikey, but it will prevent them from decrypting other things that used that private key. Hopefully that makes sense. One also has to consider the security of the hardware. On that point, I am pretty well satisfied with the Yubikey; large parts of it are open source, and they have put a lot of effort into hardening the hardware. It seems pretty much impervious to non-government actors, which is about the best guarantee a person can get about anything these days. I hope this guide has been helpful.

6 August 2015

Eduard Sanou: Reproducible builds on Debian for GSoC 2015, 1st update

This is the second blog post I m writing about my experiences contributing to Debian for Google Summer of Code 2015 (check my first post)

Status update

First month It s been two months and a few days since the GSoC started. During the first month I worked on fixing specific packages, mainly concerning issues with timestamps, which is a very common source of unreproducibility. In many cases, during the build process files are compressed into gzip or zip archives, which store the creation time of files in the metadata. This can lead to unreproducible results when there is timezone variation between builds (easily fixed setting the timezone to UTC before the compression happens). In some cases the compressed files are generated during the build, and thus add build times in the metadata of compressed files (in this case the creation date of the files needs to be normalized somehow). As explained in my application, I finished exams on the end of June, that s why I chose to work on small fixes first, so that I could make the most out of my time between studying and finishing university projects and reports. I m happy with my first month, as I have worked as originally planned. Actually, my estimation of the number of bugs I could submit every week was surpassed in reality!

Second month Once the university was over, I started dedicating myself fully to the project. This allowed me to start working on toolchain fixes, following my original plan on working with timestamp related issues. In particular I have been working a lot in implementing a proposal for deterministic timestamps that appeared in the reproducible builds project. The idea is to define an environment variable called SOURCE_DATE_EPOCH that contains a known timestamp in Unix epoch format. With this variable exported, tools that would embed the localtime in their generated or processed files, can use this externally supplied date. This would happen only if SOURCE_DATE_EPOCH is exported, so the behaviour of the tool wouldn t change if the variable is not set. The first package I patched to implement this behaviour was gcc. The reason behind this is that there are about 420 unreproducible packages due to using the __DATE__, __TIME__ and __TIMESTAMP__ C macros. My patch changes the behavior of the macros __DATE__ and __TIME__ if SOURCE_DATE_EPOCH is exported. I submitted this patch to the gcc-patches list. Even though there was some interesting discussions in the list, the patch has not been accepted yet. Seeing how the reproducible builds idea is gaining momentum and becoming widespread, I m positive that at some point the gcc people will be more receptive for such patch. The second work with SOURCE_DATE_EPOCH was in debhelper; I patched this building tool to export the variable with the latest debian/changelog entry timestamp. With this patch, all the tools that run under dh will be able to use it to embed deterministic timestamps. Unfortunately some parts of the build process of some packages don t happen under debhelper, so the variable needs to be exported in a different way. Having submitted the debhelper patch allowed many packages to become reproducible after the tools that embedded timestamps were patched to honour SOURCE_DATE_EPOCH. As of today, the toolchain packages I have patched to do that are: gcc, libxslt, gettext, ghostscript and qt4-x11 (qhelpgenerator). I have also continued working on fixing individual packages affected by timestamps, random orderings (such as the ones from listing hash keys) and locale depending orderings; I have tagged packages in our infrastructure to note what kind of issue makes them unreproducible; I have updated some parts of the Reproducible Builds Wiki.

Impressions about reproducible builds The work I did during the first month felt a bit tedious sometimes: it didn t require much creativity or thinking as most of the fixes where quite mechanical, following a recipe. After I became free from university duties, I started looking into less trivial issues, which require more deep investigation and feel more rewarding once they are fixed. I also worked on toolchain fixes, which need more work and need more care. Fixing toolchain packages feels particularly rewarding because they can cause many packages to become reproducible at once. There is a very active community in the reproducible builds project! It s great to see so many people contributing to this project spending so many hours and putting so much effort. I ve felt very welcome from the beginning and I have gotten kind replies and helpful answers to all the questions and doubts I ve had, both from my mentor and from the other people in the project. I want to add that I m still amazed by the awesome infrastructure set up for the reproducible builds project. The team is using a Jenkins machine to continuously build packages to check for reproducibility, with irc notifications for changes, and also with a really useful web interface to list all the packages organized by issues that allows exploring them individually with all the available information. Also not only the infrastructure is used to build Debian amd64 packages, but also FreeBSD, NetBSD, OpenWRT, coreboot and lately Debian armhf with the help of a few new arm nodes.

Impressions about working on a free software project This was my first time working on a community driven free software project and I ve learned so many things. Something I learned during the first days, which is even more present in such wide project like the reproducible builds, is that contributing is not just writing patches for the features you want; you also need to convince the maintainer of the package to accept the patch! I was a bit surprised at the beginning because even if this is a Debian project, that aims to make changes to the whole distribution, decisions are not absolute for the whole Debian project. After taking decisions within the reproducible builds teams on how to approach things, we need to convince the rest of the Debian developers (mainly maintainers) to follow them, as they are allowed to disagree. So it is required to work together for solutions that makes everyone comfortable, usually with discussions on mailing lists or irc. There is another fact that I wasn t expecting before getting involved in this project. The kind of teamwork I have done previously involves having a leader who decides how stuff is done, who takes decisions when needed and oversees the whole project. There seems to be a different philosophy in Debian. Instead of having a leader, everyone tries their best, trying to convince the others that their solution is good, often by showing an implementation of the solution and providing proof that it works, rather than trying to get the solution accepted before coding it. Also, solutions and ideas are valued for their quality rather than by the position of the person submitting them, and there is no hierarchy within the group: all comments and advices are taken equally, valuing their usefulness regardless of who gives them. Usually there are no votes when deciding things. Members try their best on their approaches, trying to convince others as best as they can. And even if someone disagrees they may end up accepting the solution if they don t manage to convince the original proposer of doing things differently. The idea is to spend more time working and coding than arguing and deciding on the way to do things. So far I ve seen this approach to be very efficient :) I ve been told by my mentor that for difficult cases there exist a Debian committee that helps mediate on disagreements, but that is only used as a last option, probably when the discussion gets heated up.

Personal experience Overall I m very happy to finally having set my foot in the free software community, where I m able to contribute to the kind of software I have been using for years. The sense of community in Debian is really big, and everyone is invited to participate. I think that Google is doing an awesome job with the Google Summer of Code, not only because it gives a lot to free software but because it helps students to join the free software world as contributors (which is something that may be difficult to get into when you don t know how to begin, as it happened to myself for some time). I plan to continue contributing to the free software world, and I d encourage anyone to find projects to get involved and contribute as well! Happy hacking to everyone!

5 August 2015

Antonio Terceiro: Elixir in Debian, MiniDebconf at FISL, and Debian CI updates

In June I started keeping track of my Debian activities, and this is my July update. Elixir in Debian Elixir is a functional language built on top of the Erlang virtual machine. If features imutable data structures, interesting concurrency primitives, and everything else that Erlang does, but with a syntax inspired by Ruby what makes it much more aproachable in my opinion. Those interested in Elixir for Debian are encouraged to hang around in #debian-elixir on the OFTC IRC servers. There are still a lot of things to figure out, for example how packaging Elixir libraries and applications is going to work. MiniDebconf at FISL, and beyond I helped organize a MiniDebconf at this year s FISL, in Porto Alegre on the 10th of July. The whole program was targetted at getting more people to participate in Debian, so there were talks about translation, packaging, and a few other more specific topics. I myself gave two talks: one about Debian basics, What is Debian, and how it works , and second one on packaging the free software web , which I will also give at Debconf15 later this month. The recordings are available (all talks in Portuguese) at the Debian video archive thanks to Holger Levsen. We are also organizing a new MiniDebconf in October as part of the Latinoware schedule. Ruby We are in the middle of a transition to switch to Ruby 2.2 as default in Debian unstable, and we are almost there. The Ruby transition is now on hold while GCC 5 one is going on, but will be picked up as soon as were are done with GCC 5. ruby-defaults has been uploaded to experimental for those that want to try having Ruby 2.2 as default before that change hits unstable. I myself have been using Ruby 2.2 as default for several weeks without any problem so far, including using vagrant on a daily basis and doing all my development on sid with it. I started taking notes about Ruby interpreter transitions work to make sure that knowledge is registered. I have uploaded minor security updates of both ruby2.1 and ruby2.2 to unstable. They both reached testing earlier today. I have also fixed another bug in redmine, which I hope to get into stable as well as soon as possible. gem2deb has seen several improvements through versions 0.19, 0.20, 0.20.1 and 0.20.2. I have updated a few packages: Two NEW packages, ruby-rack-contrib and ruby-grape-logging ,were ACCEPTED into the Debian archive. Kudos to the ftp-master team who are doing an awesome job reviewing new packages really fast. Debian Continuous Integration This month I have made good progress with the changes needed to make debci work as a distributed system with one master/scheduler node and as many worker nodes (running tests) as possible. While doing my tests, I have submitted a patch to lxc and updated autodep8 in unstable. At some point I plan to upload both autodep8 and autopkgtest to jessie-backports. Sponsoring I have sponsored a few packages:

4 August 2015

Julien Danjou: Ceilometer, Gnocchi & Aodh: Liberty progress

It's been a while since I talked about Ceilometer and its companions, so I thought I'd go ahead and write a bit about what's going on this side of OpenStack. I'm not going to cover new features and fancy stuff today, but rather a shallow overview of the new project processes we initiated. Ceilometer growing Ceilometer has grown a lot since that time when we started it 3 years ago. It has evolved from a system designed to fetch and store measurements, to a more complex system, with agents, alarms, events, databases, APIs, etc. All those features were needed and asked for by users and operators, but let's be honest, some of them should never have ended up in the Ceilometer code repository, especially not all at the same time. The reality is we picked a pragmatic approach due to the rigidity of the OpenStack Technical Committee in regards to new projects to become OpenStack integrated and, therefore, blessed projects. Ceilometer was actually the first project to be incubated and then integrated. We had to go through the very first issues of that process. Fortunately, now that time has passed, and all those constraints have been relaxed. To me, the OpenStack Foundation is turning into something that looks like the Apache Foundation, and there's, therefore, no need to tie technical solutions to political issues. Indeed, the Big Tent now allows much more flexibility to all of that. Back a year ago, we were afraid to bring Gnocchi into Ceilometer. Was the Technical Committee going to review the project? Was the project going to be in the scope of Ceilometer for the Technical Committee? Now we don't have to ask ourselves those questions, now that we have that freedom, it empowers us to actually do what we think is good in term of technical design without worrying too much about political issues.
Ceilometer development activity
Acknowledging Gnocchi The first step in this new process was to continue working on Gnocchi (a timeserie database and resource indexer designed to overcome historical Ceilometer storage issue) and to decide that it was not the right call to merge it into Ceilometer as some REST API v3, but that it was better to keep it standalone. We managed to get traction to Gnocchi, getting a few contributors and users. We're even seeing talks proposed to the next Tokyo Summit where people leverage Gnocchi, such as "Service of predictive analytics on cost and performance in OpenStack", "Suveil" and "Cutting Edge NFV On OpenStack: Healing and Scaling Distributed Applications". We are also doing some progress on pushing Gnocchi outside of the OpenStack community, as it can be a self-sufficient timeserie and resource database that can be used without any OpenStack interaction. Branching Aodh Rather than continuing to grow Ceilometer, during the last summit we all decided that it was time to reorganize and split Ceilometer into the different components it is made of, leveraging a more service-oriented architecture. The alarm subsystem of Ceilometer being mostly untied to the rest of Ceilometer, we decided it was the first and perfect candidate to do that. I personally engaged into doing the work and created a new repository with only the alarm code from Ceilometer, named Aodh.
Aodh is an Irish word meaning fire. A word picked so it also had some relation to Heat, and because we have some Irish influence around the project .
This made sense for a lot of reason. First because Aodh can now work completely standalone, using either Ceilometer or Gnocchi as a backend or any new plugin you'd write. I love the idea that OpenStack projects can work standalone like Swift does for example without implying any other OpenStack component. I think it's a proof of good design. Secondly, because it allows us to resonate on a smaller chunk of software a reason really under-estimated today in OpenStack. I believe that the size of your software should match a certain ratio to the size of your team. Aodh is, therefore, a new project under the OpenStack Telemetry program (or what remains of OpenStack programs now), alongside Ceilometer and Gnocchi, forked from the original Ceilometer alarm feature. We'll deprecate the latter with the Liberty release, and we'll remove it in the Mitaka release. Lessons learned Actually, moving that code out of Ceilometer (in the case of Aodh), or not merging it in (in the case of Gnocchi) had a few side effects that I admit I think we probably under-estimated back then. Indeed, the code size of Gnocchi or Aodh ended up being much smaller than the entire Ceilometer project Gnocchi is 7 smaller and Aodh 5x smaller than Ceilometer and therefore much more easy to manipulate and to hack on. That allowed us to merge dozens of patches in a few weeks, cleaning-up and enhancing a lot of small things in the code. Those tasks are very much harder in Ceilometer, due to the bigger size of the code base and the small size of our team. By having our small team working on smaller chunks of changes even when it meant actually doing more reviews greatly improved our general velocity and the number of bugs fixed and features implemented. On the more sociological side, I think it gave the team the sensation of finally owning the project. Ceilometer was huge, and it was impossible for people to know every side of it. Now, it's getting possible for people inside a team to cover a much larger portion of those smaller project, which gives them a greater sense of ownership and caring. Which ends up being good for the project quality overall. That also means that we technically decided to have different core teams by project (Ceilometer, Gnocchi, and Aodh) as they all serve different purposes and can all be used standalone or with each others. Meaning we could have contributors completely ignoring other projects. All of that reminds me some discussion I heard about projects such as Glance, trying to fit new features in - some that are really orthogonal to the original purpose. It's now clear to me that having different small components interacting together that can be completely owned and taken care of by a (small) team of contributors is the way to go. People that can therefore trust each others and easily bring new people in, makes a project really incredibly more powerful. Having a project covering a too wide set of features make things more difficult if you don't have enough manpower. This is clearly an issue that big projects inside OpenStack are facing now, such as Neutron or Nova.

2 August 2015

Benjamin Mako Hill: Understanding Hydroplane Races for the New Seattleite

It s Seafair weekend in Seattle. As always, the centerpiece is the H1 Unlimited hydroplane races on Lake Washington. EllstromManufacturingHydroplaneIn my social circle, I m nearly the only person I know who grew up in area. None of the newcomers I know had heard of hydroplane racing before moving to Seattle. Even after I explain it to them i.e., boats with 3,000+ horse power airplane engines that fly just above the water at more than 320kph (200mph) leaving 10m+ (30ft) wakes behind them! most people seem more puzzled than interested. I grew up near the shore of Lake Washington and could see (and hear!) the races from my house. I don t follow hydroplane racing throughout the year but I do enjoy watching the races at Seafair. Here s my attempt to explain and make the case for the races to new Seattleites. Before Microsoft, Amazon, Starbucks, etc., there were basically three major Seattle industries: (1) logging and lumber based industries like paper manufacturing; (2) maritime industries like fishing, shipbuilding, shipping, and the navy; (3) aerospace (i.e., Boeing). Vintage hydroplane racing represented the Seattle trifecta: Wooden boats with airplane engines! The wooden U-60 Miss Thriftway circa 1955 (Thriftway is a Washinton-based supermarket that nobody outside has heard of) below is a picture of old-Seattle awesomeness. Modern hydroplanes are now made of fiberglass but two out of three isn t bad. miss_thriftwayAlthough the boats are racing this year in events in Indiana, San Diego, and Detroit in addition to the two races in Washington, hydroplane racing retains deep ties to the region. Most of the drivers are from the Seattle area. Many or most of the teams and boats are based in Washington throughout the year. Many of the sponsors are unknown outside of the state. This parochialness itself cultivates a certain kind of appeal among locals. In addition to old-Seattle/new-Seattle cultural divide, there s a class divide that I think is also worth challenging. Although the demographics of hydro-racing fans is surprisingly broad, it can seem like Formula One or NASCAR on the water. It seems safe to suggest that many of the demographic groups moving to Seattle for jobs in the tech industry are not big into motorsports. Although I m no follower of motorsports in general, I ve written before cultivated disinterest in professional sports, and it remains something that I believe is worth taking on. It s not all great. In particular, the close relationship between Seafair and the military makes me very uneasy. That said, even with the military-heavy airshow, I enjoy the way that Seafair weekend provides a little pocket of old-Seattle that remains effectively unchanged from when I was a kid. I d encourage others to enjoy it as well!

Benjamin Mako Hill: Understanding Hydroplane Races for the New Seattleite

It s Seafair weekend in Seattle. As always, the centerpiece is the H1 Unlimited hydroplane races on Lake Washington. EllstromManufacturingHydroplaneIn my social circle, I m nearly the only person I know who grew up in area. None of the newcomers I know had heard of hydroplane racing before moving to Seattle. Even after I explain it to them i.e., boats with 3,000+ horse power airplane engines that fly just above the water at more than 320kph (200mph) leaving 10m+ (30ft) wakes behind them! most people seem more puzzled than interested. I grew up near the shore of Lake Washington and could see (and hear!) the races from my house. I don t follow hydroplane racing throughout the year but I do enjoy watching the races at Seafair. Here s my attempt to explain and make the case for the races to new Seattleites. Before Microsoft, Amazon, Starbucks, etc., there were basically three major Seattle industries: (1) logging and lumber based industries like paper manufacturing; (2) maritime industries like fishing, shipbuilding, shipping, and the navy; (3) aerospace (i.e., Boeing). Vintage hydroplane racing represented the Seattle trifecta: Wooden boats with airplane engines! The wooden U-60 Miss Thriftway circa 1955 (Thriftway is a Washinton-based supermarket that nobody outside has heard of) below is a picture of old-Seattle awesomeness. Modern hydroplanes are now made of fiberglass but two out of three isn t bad. miss_thriftwayAlthough the boats are racing this year in events in Indiana, San Diego, and Detroit in addition to the two races in Washington, hydroplane racing retains deep ties to the region. Most of the drivers are from the Seattle area. Many or most of the teams and boats are based in Washington throughout the year. Many of the sponsors are unknown outside of the state. This parochialness itself cultivates a certain kind of appeal among locals. In addition to old-Seattle/new-Seattle cultural divide, there s a class divide that I think is also worth challenging. Although the demographics of hydro-racing fans is surprisingly broad, it can seem like Formula One or NASCAR on the water. It seems safe to suggest that many of the demographic groups moving to Seattle for jobs in the tech industry are not big into motorsports. Although I m no follower of motorsports in general, I ve written before cultivated disinterest in professional sports, and it remains something that I believe is worth taking on. It s not all great. In particular, the close relationship between Seafair and the military makes me very uneasy. That said, even with the military-heavy airshow, I enjoy the way that Seafair weekend provides a little pocket of old-Seattle that remains effectively unchanged from when I was a kid. I d encourage others to enjoy it as well!

1 August 2015

Steve McIntyre: New UEFI team in Debian

We've just started a new team in Debian for maintaining our UEFI packages together, with git repositories in a shared project on alioth etc. We're just working out the exact details of how we're going to manage things, but for now we've moved the following packages under the team's umbrella: and in the future we'll clearly end up adding more. We've also started a new IRC channel (#debian-efi) on irc.debian.org aka irc.oftc.net. New members always welcome to help with the work here!

Francois Marier: Setting the wifi regulatory domain on Linux and OpenWRT

The list of available wifi channels is slightly different from country to country. To ensure access to the right channels and transmit power settings, one needs to set the right regulatory domain in the wifi stack.

Linux For most Linux-based computers, you can look and change the current regulatory domain using these commands: 
iw reg get
iw reg set CA
where CA is the two-letter country code when the device is located. On Debian and Ubuntu, you can make this setting permanent by putting the country code in /etc/default/crda. Finally, to see the list of channels that are available in the current config, use: 
iwlist wlan0 frequency

OpenWRT On OpenWRT-based routers (including derivatives like Gargoyle), looking and setting the regulatory domain temporarily works the same way (i.e. the iw commands above). In order to persist your changes though, you need to use the uci command: 
uci set wireless.radio0.country=CA
uci set wireless.radio1.country=CA
uci commit wireless
where wireless.radio0 and wireless.radio1 are the wireless devices specific to your router. You can look them up using: 
uci show wireless
To test that it worked, simply reboot the router and then look at the selected regulatory domain: 
iw reg get

Scanning the local wifi environment Once your devices are set to the right country, you should scan the local environment to pick the least congested wifi channel. You can use the Kismet spectools (free software) if you have the hardware, otherwise WifiAnalyzer (proprietary) is a good choice on Android (remember to manually set the available channels in the settings).

Russ Allbery: Review: The Pyramid Waltz

Review: The Pyramid Waltz, by Barbara Ann Wright
Series: Katya and Starbride #1
Publisher: Bold Strokes
Copyright: September 2012
ISBN: 1-60282-792-3
Format: Kindle
Pages: 264
Princess Katya Nar Umbriel is publicly a bored, womanizing, and difficult daughter to the rulers of Farraday. It's all an act, though, with the full knowledge of her parents. As the second child, she's the leader of the Order of Vestra: the equivalent of the Secret Service, devoted to protecting the royal family and, by extension, the kingdom, particularly against magical attacks. Starbride is new to court and entirely out of place. From a northern neighboring country, and far more comfortable in practical clothing than the frilled court dresses that her mother wants her to wear, she has been sent to court to make contacts. Her people are getting the bad side of various trade contracts and desperately need some political maneuvering space of their own. Starbride's best hope for this is to study law in the palace library when she can manage to avoid the other courtiers. But then she and Katya stumble across each other, outside of the roles they're playing, and might have an opportunity for a deeper connection. One that neither of them want to entangle in their personal worries. This is the last of a set of books I picked up while looking for lesbian romance with fantasy or science fiction elements. On the romance front, it's one of the better entries in that set. Both Katya and Starbride are likeable, in large part due to their mutual exasperation with the trappings of the court. (Making the protagonists more serious, thoughtful, and intelligent than the surrounding characters is an old trick, but it works.) Wright has a good ear for banter, particularly the kind when two people of good will are carefully feeling each other out. And despite Katya's need to keep a deep secret from Starbride for some of the book, The Pyramid Waltz mostly avoids irritating communication failures as a plot driver. The fantasy portion and the plot drivers, alas, are weaker. The world building is not exactly bad, but it's just not that interesting. There are a couple of moderately good ideas, in the form of pyramid magic and secret (and dangerous) magical powers that run in the royal family, but they're not well-developed. Pyramid magic turns out to look much like any other generic fantasy magic system, with training scenes that could have come from a Valdemar or Wheel of Time novel (and without as much dramatic tension). And the royal family's secret, while better-developed and integral to the plot, still felt rather generic and one-sided. Maybe that's something Wright develops better in future novels in this series, but that was another problem: the ending of The Pyramid Waltz was rather weak. Partly, I think, this is because the cast is too large and not well-developed. I cared about Katya and Starbird, and to a lesser extent their servants and one of the Order members. (Wright has a moderately interesting bit of worldbuilding about how servants work in Starbride's culture, which I wish we'd seen more of.) But there are a bunch of other Order of Vesta members, Katya's family, and various other bits of history and hinted world views, none of which seemed to get much depth. The ending climax involved a lot of revelations and twists that primarily concerned characters I didn't care about. It lost something in the process. This book is clearly set up for a sequel. There is an ending, but it's not entirely satisfying. Unfortunately, despite liking Katya and Starbird a lot, the rest of the story wasn't compelling enough to make me want to buy it, particularly since the series apparently goes through another three books before reaching a real ending. I enjoyed parts of this book, particularly Katya and Starbird feeling each other out and discovering similarities in their outlook. Katya teasing Starbird, and Starbird teasing herself, over her mother's choice of her clothing was probably the best part. It's not bad for what it's trying to do, but I think it's a bit too generic and not satisfying enough to really recommend. Followed by For Want of a Fiend. Rating: 6 out of 10

31 July 2015

Steve McIntyre: Linaro VLANd v0.3

VLANd is a python program intended to make it easy to manage port-based VLAN setups across multiple switches in a network. It is designed to be vendor-agnostic, with a clean pluggable driver API to allow for a wide range of different switches to be controlled together. There's more information in the README file. I've just released v0.3, with a lot of changes included since the last release: VLANd is Free Software, released under the GPL version 2 (or any later version). For now, grab it from git; tarballs will be coming shortly.

Raphaël Hertzog: My Free Software Activities in July 2015

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS This month I have been paid to work 15 hours on Debian LTS. In that time I did the following: Kali Linux / Debian Stretch work kaliKali Linux wants to experiment something close to Debian Constantly Usable Testing: we have a kali-rolling release that is based on Debian Testing and we want to take a new snapshot every 4 months (in order to have 3 releases per year). More specifically we have a kali-dev repository which is exactly Debian Stretch + our own Kali packages (the kali package take precedence) updated 4 times a day, just like testing is. And we have a britney2 setup that generates kali-rolling out of kali-dev (without any requirement in terms of delay/RC bugs, it just ensures that dependencies are not broken), also 4 times a day. We have jenkins job that ensures that our metapackages are installable in kali-dev (and kali-rolling) and that we can build our ISO images. When things break, I have to fix them and I try to fix them on the Debian side first. So here are some examples of stuff I did in response to various failures: If you analyze this list, you will see that a large part of the issues we had come down to package getting removed from testing due to RC bugs. We should be able to anticipate those issues and monitor the packages that have an impact on Kali. We will probably add new jenkins job that installs all the metapackages and then run how-can-i-help -s testing-autorm --old I just submitted #794238 as a wishlist against how-can-i-help. At the same time, there are bugs that make it into testing and that I fix / work around on the Kali side. But those fixes / work around might be more useful if they were pushed to testing via testing-proposed-updates. I tried to see whether other derivatives had similar needs to see if derivatives could join their efforts at this level but it does not look like so for now. Last but not least, bugs reported on the Kali side also resulted in Debian improvements: Other Debian work Sponsorship. I sponsored multiple packages for Daniel Stender who is packaging prospector, a software that I requested earlier (through RFP bug). So I reviewed and uploaded python-requirements-detector, python-setoptconf, pylint-celery and pylint-common. During a review I also discovered a nice bug in dh-python (#793609a comment in the middle of a Build-Depends could break a package). I also sponsored an upload of notmuch-addrlookup (new package requested by a Freexian customer). Packaging. I uploaded python-django 1.7.9 in unstable and 1.8.3 in experimental to fix security issues. I uploaded a new upstream release of ditaa through a non-maintainer uploaded (again at the request of a Freexian customer). Distro Tracker. Beside the work to integrate detailed security status, I fixed the code to be compatible with Django 1.8 and modified the tox configuration to ensure that the test suite is regularly run against Django 1.8. I also merged multiple patches of Christophe Siraut (cf #784151 and #754413). Thanks See you next month for a new summary of my activities.

2 comments Liked this article? Click here. My blog is Flattr-enabled.

Simon Kainz: DUCK challenge: week 4

The DUCK challenge is making a quite stable progress: in the last 4 weeks there were approximately 12.25 packages fixed and uploaded per week. In the current week the following packages were fixed and uploaded into unstable: So we had 14 packages fixed and uploaded by 10 different uploaders. A big "Thank You" to you!! Since the start of this challenge, a total of 49 packages, uploaded by 31 different persons were fixed. Here is a quick overview:
Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7
# Packages 10 15 10 14 - - -
Total 10 25 35 49 - - -
The list of the fixed and updated packages is availabe here. I will try to update this ~daily. If I missed one of your uploads, please drop me a line. DebConf15 is approaching quite fast, so please get involved: The DUCK Challenge is running until end of DebConf15! Pevious articles are here: Week 1, Week 2, Week 3.

30 July 2015

Steve Kemp: The differences in Finland start at home.

So we're in Finland, and the differences start out immediately. We're renting a flat, in building ten, on a street. You'd think "10 Streetname" was a single building, but no. It is a pair of buildings: 10A, and 10B. Both of the buildings have 12 flats in them, with 10A having 1-12, and 10B having 13-24. There's a keypad at the main entrance, which I assumed was to let you press a button and talk to the people inside "Hello I'm the postmaster", but no. There is no intercom system, instead you type in a magic number and the door opens. The magic number? Sounds like you want to keep that secret, since it lets people into the common-area? No. Everybody has it. The postman, the cleaners, the DHL delivery man, and all the ex-tenants. We invited somebody over recently and gave it out in advance so that they could knock on our flat-door. Talking of cleaners: In the UK I lived in a flat and once a fortnight somebody would come and sweep the stair-well, since we didn't ever agree to do it ourselves. Here somebody turns up every day, be it to cut the grass, polish the hand-rail, clean the glass on the front-door, or mop the floors of the common area. Sounds awesome. But they cut the grass, right outside our window, at 7:30AM. On the dot. (Or use a leaf-blower, or something equally noisy.) All this communal-care is paid for by the building-association, of which all flat-owners own shares. Sounds like something we see in England, or even like Americas idea of a Home-Owners-Association. (In Scotland you own your own flat, you don't own shares of an entity which owns the complete building. I guess there are pros and cons to both approaches.) Moving onwards other things are often the same, but the differences when you spot them are odd. I'm struggling to think of them right now, somebody woke me up by cutting our grass for the second time this week (!) Anyway I'm registered now with the Finnish government, and have a citizen-number, which will be useful, I've got an appointment booked to register with the police - which is something I had to do as a foreigner within the first three months - and today I've got an appointment with a local bank so that I can have a euro-bank-account. Happily I did find a gym to join, the owner came over one Sunday to give me a tiny-tour, and then gave me a list of other gyms to try if his wasn't good enough - which was a nice touch - I joined a couple of days later, his gym is awesome. (I'm getting paid in UK-pounds, to a UK-bank, so right now I'm getting local money by transferring to my wifes account here, but I want to do that to my own, and open a shared account for paying for rent, electricity, internet, water, & etc). My flat back home is still not rented, because the nice property management company lost my keys. Yeah you can't make that up can you? With a bit of luck the second set of keys I mailed them will arrive soon and the damn thing can be occupied, while I'm not relying on that income I do wish to have it.

27 July 2015

Dirk Eddelbuettel: Evading the "Hadley tax": Faster Travis tests for R

Hadley is a popular figure, and rightly so as he successfully introduced many newcomers to the wonders offered by R. His approach strikes some of us old greybeards as wrong---I particularly take exception with some of his writing which frequently portrays a particular approach as both the best and only one. Real programming, I think, is often a little more nuanced and aware of tradeoffs which need to be balanced. As a book on another language once popularized: "There is more than one way to do things." But let us leave this discussion for another time. As the reach of the Hadleyverse keeps spreading, we sometimes find ourselves at the receiving end of a cost/benefit tradeoff. That is what this post is about, and it uses a very concrete case I encountered yesterday. As blogged earlier, the RcppZiggurat package was updated. I had not touched it in a year, but Brian Ripley had sent a brief and detailed note concerning something flagged by the Solaris compiler (correctly suggesting I replace fabs() with abs() on integer types). (Allow me to stray from the main story line here for a second to stress just how insane a work load he is carrying, essentially for all of us. R and the R community are so just so indebted to him for all his work---which makes the usual social media banter about him so unfortunate. But that too shall be left for another time.) Upon making the simple fix, and submitting to GitHub the usual Travis CI was triggered. And here is what I saw: first travis build in a year
All happy, all green. Previous build a year ago, most recent build yesterday, both passed. But hold on: test time went from 2:54 minutes to 7:47 minutes for an increase of almost five minutes! And I knew that I had not added any new dependencies, or altered any build options. What did happen was that among the dependencies of my package, one had decided to now also depend on ggplot2. Which leads to a chain of sixteen additional packages being loaded besides the four I depend upon---when it used to be just one. And that took five minutes as all those packages are installed from source, and some are big and take a long time to compile. There is however and easy alternative, and for that we have to praise Michael Rutter who looks after a number of things for R on Ubuntu. Among these are the R builds for Ubuntu but also the rrutter PPA as well as the c2d4u PPA. If you have not heard this alphabet soup before, a PPA is a package repository for Ubuntu where anyone (who wants to sign up) can upload (properly setup) source files which are then turned into Ubuntu binaries. With full dependency resolution and all other goodies we have come to expect from the Debian / Ubuntu universe. And Michael uses this facility with great skill and calm to provide us all with Ubuntu binaries for R itself (rebuilding what yours truly uploads into Debian), as well as a number of key packages available via the CRAN mirrors. Less know however is this "c2d4u" which stands for CRAN to Debian for Ubuntu. And this builds on something Charles Blundell once built under my mentorship in a Google Summer of Code. And Michael does a tremdous job covering well over a thousand CRAN source packages---and providing binaries for all. Which we can use for Travis! What all that means is that I could now replace the line 
 - ./travis-tool.sh install_r RcppGSL rbenchmark microbenchmark highlight
which implies source builds of the four listed packages and all their dependencies with the following line implying binary installations of already built packages: 
 - ./travis-tool.sh install_aptget libgsl0-dev r-cran-rcppgsl r-cran-rbenchmark r-cran-microbenchmark r-cran-highlight
In this particular case I also needed to build a binary package of my RcppGSL package as this one is not (yet) handled by Michael. I happen to have (re-)discovered the beauty of PPAs for Travis earlier this year and revitalized an older and largely dormant launchpad account I had for this PPA of mine. How to build a simple .deb package will also have to left for a future post to keep this more concise. This can be used with the existing r-travis setup---but one needs to use the older, initial variant in order to have the ability to install .deb packages. So in the .travis.yml of RcppZiggurat I just use 
before_install:
## PPA for Rcpp and some other packages
- sudo add-apt-repository -y ppa:edd/misc
## r-travis by Craig Citro et al
- curl -OL http://raw.github.com/craigcitro/r-travis/master/scripts/travis-tool.sh
- chmod 755 ./travis-tool.sh
- ./travis-tool.sh bootstrap
to add my own PPA and all is good. If you do not have a PPA, or do not want to create your own packages you can still benefit from the PPAs by Michael and "mix and match" by installing from binary what is available, and from source what is not. Here we were able to use an all-binary approach, so let's see the resulting performance: latest travis build
Now we are at 1:03 to 1:15 minutes---much better. So to conclude, while the every expanding universe of R packages is fantastic for us as users, it can be seen to be placing a burden on us as developers when installing and testing. Fortunately, the packaging infrastructure built on top of Debian / Ubuntu packages can help and dramatically reduce build (and hence test) times. Learning about PPAs can be a helpful complement to learning about Travis and continued integration. So maybe now I need a new reason to blame Hadley? Well, there is always snake case ...

Follow-up: The post got some pretty immediate feedback shortly after I posted it. Craig Citro pointed out (quite correctly) that I could use r_binary_install which would also install the Ubuntu binaries based on their R packages names. Having built R/CRAN packages for Debian for so long, I am simply more used to the r-cran-* notations, and I think I was also the one contributing install_aptget to r-travis ... Yihui Xie spoke up for the "new" Travis approach deploying containers, caching of packages and explicit whitelists. It was in that very (GH-based) discussion that I started to really lose faith in the new Travis approach as they want use to whitelist each and every package. With 6900 and counting at CRAN I fear this simply does not scale. But different approaches are certainly welcome. I posted my 1:03 to 1:15 minutes result. If the "New School" can do it faster, I'd be all ears.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Next.

Previous.