Search Results: "ema"

22 June 2015

Sven Hoexter: Free SSL/TLS snakeoil from wosign.com

I've been a proponet of CaCert.org for a long time and I'm still using those certificates in some places, but lately I gave in and searched for something that validates even on iOS. It's not that I strictly need it, it's more a favour to make life for friends and family easier. I turned down startssl.com because I always manage to somehow lose the client certificate for the portal login. Plus I failed to generate several certificates for subdomains within the primary domain. I want to use different keys on purpose so SANs are not helpful, neither are wildcard certs for which you've to pay anyway. Another point against a wildcard cert from startssl is that I'd like to refrain from sending in my scanned papers for verification. On a sidenote I'm also not a fan of random email address extractions from whois to sent validation codes to. I just don't see why the abuse desk of a registrar should be able to authorize on DV certificates for a domain under my control. startssl abuse desk in dv validation So I decided to pay the self proclaimed leader of the snakeoil industrie (Comodo) via cheapsslshop.com. That made 12USD for a 3 year Comodo DV certificate. Fair enough for the mailsetup I share with a few friends, and the cheapest one I could find at that time. Actually no hassle with logins or verification. It looks a bit like a scam but the payment is done via 2checkout if I remember correctly and the certificate got issued via a voucher code by Comodo directly. Drawback: credit card payment. Now while we're all waiting for letsencrypt.org I learned about the free offer of wosign.com. The CA is issued by the StartSSL Root CA, so technically we're very close to step one. Beside of that I only had to turn off uBlock origin and the rest of the JavaScript worked fine with Iceweasel once I clicked on the validity time selection checkbox. They offer the certificate for up to 3 years, you can paste your own csr and you can add up to 100 SANs. The only drawback is that it took them about 12 hours to issue the certificate and the mails look a hell lot like spam if you sent them through Spamassassin. That provides now a free and validating certificate for sven.stormbind.net in case you'd like to check out the chain. The validation chain is even one certificate shorter then the chain for the certificate I bought from Comodo. So in case anyone else is waiting for letsencrypt to start, you might want to check wosign until Mozilla et al are ready. From my point of view the only reason to pay one of the major CAs is for the service of running a reliable OCSP system. I also pointed that out here. It's more and more about the service you buy and no longer just money for a few ones and zeroes.

Niels Thykier: Introducing dak auto-decruft

Debian now have over 22 000 source packages and 45 500 binary packages. To counter that, the FTP masters and I have created a dak tool to automatically remove packages from unstable! This is also much more efficient than only removing them from testing! :) The primary goal of the auto-decrufter is to remove a regular manual work flow from the FTP masters. Namely, the removal of the common cases of cruft, such as Not Built from Source (NBS) and Newer Version In Unstable (NVIU). With the auto-decrufter in place, such cruft will be automatically removed when there are no reverse dependencies left on any architecture and nothing Build-Depends on it any more. Despite the implication in the opening of this post, this will in fact not substantially reduce the numbers of packages in unstable. :) Nevertheless, it is still very useful for the FTP masters, the release team and packaging Debian contributors. The reason why the release team benefits greatly from this tool, is that almost every transition generates one piece of NBS -cruft. Said piece of cruft currently must be removed from unstable before the transition can progress into its final phase. Until recently that removal has been 100% manual and done by the FTP masters. The restrictions on auto-decrufter means that we will still need manual decrufts. Notably, the release team will often complete transitions even when some reverse dependencies remain on non-release architectures. Nevertheless, it is definitely an improvement. Omelettes and eggs: As an old saying goes You cannot make an omelette without breaking eggs . Less so when the only test suite is production. So here are some of the broken eggs caused by implementation of the auto-decrufter: Of the 3, the boolean inversion was no doubt the worst. By the time we had it fixed, at least 50 (unique) binary packages had lost their override . Fortunately, it was possible to locate these issues using a database query and they have now been fixed. Before I write any more non-trivial patches for dak, I will probably invest some time setting up a basic test framework for dak first.
Filed under: Debian, Release-Team

20 June 2015

Lunar: Reproducible builds: week 4 in Stretch cycle

What happened about the reproducible builds effort for this week: Toolchain fixes Lunar rebased our custom dpkg on the new release, removing a now unneeded patch identified by Guillem Jover. An extra sort in the buildinfo generator prevented a stable order and was quickly fixed once identified. Mattia Rizzolo also rebased our custom debhelper on the latest release. Packages fixed The following 30 packages became reproducible due to changes in their build dependencies: animal-sniffer, asciidoctor, autodock-vina, camping, cookie-monster, downthemall, flashblock, gamera, httpcomponents-core, https-finder, icedove-l10n, istack-commons, jdeb, libmodule-build-perl, libur-perl, livehttpheaders, maven-dependency-plugin, maven-ejb-plugin, mozilla-noscript, nosquint, requestpolicy, ruby-benchmark-ips, ruby-benchmark-suite, ruby-expression-parser, ruby-github-markup, ruby-http-connection, ruby-settingslogic, ruby-uuidtools, webkit2gtk, wot. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which did not make their way to the archive yet: Also, the following bugs have been reported: reproducible.debian.net Holger Levsen made several small bug fixes and a few more visible changes: strip-nondeterminism Version 0.007-1 of strip-nondeterminism the tool to post-process various file formats to normalize them has been uploaded by Holger Levsen. Version 0.006-1 was already in the reproducible repository, the new version mainly improve the detection of Maven's pom.properties files. debbindiff development At the request of Emmanuel Bourg, Reiner Herrmann added a comparator for Java .class files. Documentation update Christoph Berg created a new page for the timestamps in manpages created by Doxygen. Package reviews 93 obsolete reviews have been removed, 76 added and 43 updated this week. New identified issues: timestamps in manpages generated by Doxygen, modification time differences in files extracted by unzip, tstamp task used in Ant build.xml, timestamps in documentation generated by ASDocGen. The description for build id related issues has been clarified. Meetings Holger Levsen announced a first meeting on Wednesday, June 3rd, 2015, 19:00 UTC. The agenda is amendable on the wiki. Misc. Lunar worked on a proof-of-concept script to import the build environment found in .buildinfo files to UDD. Lucas Nussbaum has positively reviewed the proposed schema. Holger Levsen cleaned up various experimental toolchain repositories, marking merged brances as such.

Lunar: Reproducible builds: week 5 in Stretch cycle

What happened about the reproducible builds effort for this week: Toolchain fixes Uploads that should help other packages: Patch submitted for toolchain issues: Some discussions have been started in Debian and with upstream: Packages fixed The following 8 packages became reproducible due to changes in their build dependencies: access-modifier-checker, apache-log4j2, jenkins-xstream, libsdl-perl, maven-shared-incremental, ruby-pygments.rb, ruby-wikicloth, uimaj. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which did not make their way to the archive yet: Discussions that have been started: reproducible.debian.net Holger Levsen added two new package sets: pkg-javascript-devel and pkg-php-pear. The list of packages with and without notes are now sorted by age of the latest build. Mattia Rizzolo added support for email notifications so that maintainers can be warned when a package becomes unreproducible. Please ask Mattia or Holger or in the #debian-reproducible IRC channel if you want to be notified for your packages! strip-nondeterminism development Andrew Ayer fixed the gzip handler so that it skip adding a predetermined timestamp when there was none. Documentation update Lunar added documentation about mtimes of file extracted using unzip being timezone dependent. He also wrote a short example on how to test reproducibility. Stephen Kitt updated the documentation about timestamps in PE binaries. Documentation and scripts to perform weekly reports were published by Lunar. Package reviews 50 obsolete reviews have been removed, 51 added and 29 updated this week. Thanks Chris West and Mathieu Bridon amongst others. New identified issues: Misc. Lunar will be talking (in French) about reproducible builds at Pas Sage en Seine on June 19th, at 15:00 in Paris. Meeting will happen this Wednesday, 19:00 UTC.

17 June 2015

DebConf team: Striving for more diversity at DebConf15 (Posted by DebConf Team)

DebConf is not just for Debian Developers, we welcome all members of our community active in different areas, like translation, documentation, artwork, testing, specialized derivatives, and many other ways that help make Debian better. In fact, we would like to open DebConf to an even broader audience, and we strongly believe that more diversity at DebConf and in the Debian community will significantly help us towards our goal of becoming the Universal Operating System. The DebConf team is proud to announce that we have started designing a specific diversity sponsorship programme to attract people to DebConf that would otherwise not consider attending our conference or not be able to join us. In order to apply for this special sponsorship, please write an email to outreach@debian.org, before July 6th, about your interest in Debian and your sponsorship needs (accomodation, travel). Please include a sentence or two about why you are applying for a Diversity Sponsorship. You can also nominate people you think should be considered for this sponsorship programme. Please feel free to send this announcement on to groups or individuals that could be interested in this sponsorship programme. And we re also looking forward to your feedback. We re just getting started and you can help shape these efforts.

16 June 2015

Julien Danjou: Timezones and Python

Recently, I've been fighting with the never ending issue of timezones. I never thought I would have plunged into this rabbit hole, but hacking on OpenStack and Gnocchi I felt into that trap easily is, thanks to Python. Why you really, really, should never ever deal with timezones To get a glimpse of the complexity of timezones, I recommend that you watch Tom Scott's video on the subject. It's fun and it summarizes remarkably well the nightmare that timezones are and why you should stop thinking that you're smart.
The importance of timezones in applications
Once you've heard what Tom says, I think it gets pretty clear that a timestamp without any timezone attached does not give any useful information. It should be considered irrelevant and useless. Without the necessary context given by the timezone, you cannot infer what point in time your application is really referring to. That means your application should never handle timestamps with no timezone information. It should try to guess or raises an error if no timezone is provided in any input. Of course, you can infer that having no timezone information means UTC. This sounds very handy, but can also be dangerous in certain applications or language such as Python, as we'll see. Indeed, in certain applications, converting timestamps to UTC and losing the timezone information is a terrible idea. Imagine that a user create a recurring event every Wednesday at 10:00 in its local timezone, say CET. If you convert that to UTC, the event will end up being stored as every Wednesday at 09:00. Now imagine that the CET timezone switches from UTC+01:00 to UTC+02:00: your application will compute that the event starts at 11:00 CET every Wednesday. Which is wrong, because as the user told you, the event starts at 10:00 CET, whatever the definition of CET is. Not at 11:00 CET. So CET means CET, not necessarily UTC+1. As for endpoints like REST API, a thing I daily deal with, all timestamps should include a timezone information. It's nearly impossible to know what timezone the timestamps are in otherwise: UTC? Server local? User local? No way to know. Python design & defect
Python comes with a timestamp object named datetime.datetime. It can store date and time precise to the microsecond, and is qualified of timezone "aware" or "unaware", whether it embeds a timezone information or not. To build such an object based on the current time, one can use datetime.datetime.utcnow() to retrieve the date and time for the UTC timezone, and datetime.datetime.now() to retrieve the date and time for the current timezone, whatever it is. 
>>> import datetime
>>> datetime.datetime.utcnow()
datetime.datetime(2015, 6, 15, 13, 24, 48, 27631)
>>> datetime.datetime.now()
datetime.datetime(2015, 6, 15, 15, 24, 52, 276161)

As you can notice, none of these results contains timezone information. Indeed, Python datetime API always returns unaware datetime objects, which is very unfortunate. Indeed, as soon as you get one of this object, there is no way to know what the timezone is, therefore these objects are pretty "useless" on their own. Armin Ronacher proposes that an application always consider that the unaware datetime objects from Python are considered as UTC. As we just saw, that statement cannot be considered true for objects returned by datetime.datetime.now(), so I would not advise doing so. datetime objects with no timezone should be considered as a "bug" in the application.
Recommendations My recommendation list comes down to:
  1. Always use aware datetime object, i.e. with timezone information. That makes sure you can compare them directly (aware and unaware datetime objects are not comparable) and will return them correctly to users. Leverage pytz to have timezone objects.
  2. Use ISO 8601 as input and output string format. Use datetime.datetime.isoformat() to return timestamps as string formatted using that format, which includes the timezone information.
In Python, that's equivalent to having: 
>>> import datetime
>>> import pytz
>>> def utcnow():
    return datetime.datetime.now(tz=pytz.utc)
>>> utcnow()
datetime.datetime(2015, 6, 15, 14, 45, 19, 182703, tzinfo=<UTC>)
>>> utcnow().isoformat()
'2015-06-15T14:45:21.982600+00:00'

If you need to parse strings containing ISO 8601 formatted timestamp, you can rely on the iso8601, which returns timestamps with correct timezone information. This makes timestamps directly comparable: 
>>> import iso8601
>>> iso8601.parse_date(utcnow().isoformat())
datetime.datetime(2015, 6, 15, 14, 46, 43, 945813, tzinfo=<FixedOffset '+00:00' datetime.timedelta(0)>)
>>> iso8601.parse_date(utcnow().isoformat()) < utcnow()
True

If you need to store those timestamps, the same rule should apply. If you rely on MongoDB, it assumes that all the timestamp are in UTC, so be careful when storing them you will have to normalize the timestamp to UTC. For MySQL, nothing is assumed, it's up to the application to insert them in a timezone that makes sense to it. Obviously, if you have multiple applications accessing the same database with different data sources, this can end up being a nightmare. PostgreSQL has a special data type that is recommended called timestamp with timezone, and which can store the timezone associated, and do all the computation for you. That's the recommended way to store them obviously. That does not mean you should not use UTC in most cases; that just means you are sure that the timestamp are stored in UTC since it's written in the database, and you check if any other application inserted timestamps with different timezone. OpenStack status As a side note, I've improved OpenStack situation recently by changing the oslo.utils.timeutils module to deprecate some useless and dangerous functions. I've also added support for returning timezone aware objects when using the oslo_utils.timeutils.utcnow() function. It's not possible to make it a default unfortunately for backward compatibility reason, but it's there nevertheless, and it's advised to use it. Thanks to my colleague Victor for the help! Have a nice day, whatever your timezone is!

Simon Josefsson: SSH Host Certificates with YubiKey NEO

If you manage a bunch of server machines, you will undoubtedly have run into the following OpenSSH question: 
The authenticity of host 'host.example.org (1.2.3.4)' can't be established.
RSA key fingerprint is 1b:9b:b8:5e:74:b1:31:19:35:48:48:ba:7d:d0:01:f5.
Are you sure you want to continue connecting (yes/no)?
If the server is a single-user machine, where you are the only person expected to login on it, answering yes once and then using the ~/.ssh/known_hosts file to record the key fingerprint will (sort-of) work and protect you against future man-in-the-middle attacks. I say sort-of, since if you want to access the server from multiple machines, you will need to sync the known_hosts file somehow. And once your organization grows larger, and you aren t the only person that needs to login, having a policy that everyone just answers yes on first connection on all their machines is bad. The risk that someone is able to successfully MITM attack you grows every time someone types yes to these prompts. Setting up one (or more) SSH Certificate Authority (CA) to create SSH Host Certificates, and have your users trust this CA, will allow you and your users to automatically trust the fingerprint of the host through the indirection of the SSH Host CA. I was surprised (but probably shouldn t have been) to find that deploying this is straightforward. Even setting this up with hardware-backed keys, stored on a YubiKey NEO, is easy. Below I will explain how to set this up for a hypothethical organization where two persons (sysadmins) are responsible for installing and configuring machines. I m going to assume that you already have a couple of hosts up and running and that they run the OpenSSH daemon, so they have a /etc/ssh/ssh_host_rsa_key* public/private keypair, and that you have one YubiKey NEO with the PIV applet and that the NEO is in CCID mode. I don t believe it matters, but I m running a combination of Debian and Ubuntu machines. The Yubico PIV tool is used to configure the YubiKey NEO, and I will be using OpenSC s PKCS#11 library to connect OpenSSH with the YubiKey NEO. Let s install some tools: 
apt-get install yubikey-personalization yubico-piv-tool opensc-pkcs11 pcscd
Every person responsible for signing SSH Host Certificates in your organization needs a YubiKey NEO. For my example, there will only be two persons, but the number could be larger. Each one of them will have to go through the following process. The first step is to prepare the NEO. First mode switch it to CCID using some device configuration tool, like yubikey-personalization. 
ykpersonalize -m1
Then prepare the PIV applet in the YubiKey NEO. This is covered by the YubiKey NEO PIV Introduction but I ll reproduce the commands below. Do this on a disconnected machine, saving all files generated on one or more secure media and store that in a safe. 
user=simon
key= dd if=/dev/random bs=1 count=24 2>/dev/null   hexdump -v -e '/1 "%02X"' 
echo $key > ssh-$user-key.txt
pin= dd if=/dev/random bs=1 count=6 2>/dev/null   hexdump -v -e '/1 "%u"' cut -c1-6 
echo $pin > ssh-$user-pin.txt
puk= dd if=/dev/random bs=1 count=6 2>/dev/null   hexdump -v -e '/1 "%u"' cut -c1-8 
echo $puk > ssh-$user-puk.txt
yubico-piv-tool -a set-mgm-key -n $key
yubico-piv-tool -k $key -a change-pin -P 123456 -N $pin
yubico-piv-tool -k $key -a change-puk -P 12345678 -N $puk
Then generate a RSA private key for the SSH Host CA, and generate a dummy X.509 certificate for that key. The only use for the X.509 certificate is to make PIV/PKCS#11 happy they want to be able to extract the public-key from the smartcard, and do that through the X.509 certificate. 
openssl genrsa -out ssh-$user-ca-key.pem 2048
openssl req -new -x509 -batch -key ssh-$user-ca-key.pem -out ssh-$user-ca-crt.pem
You import the key and certificate to the PIV applet as follows: 
yubico-piv-tool -k $key -a import-key -s 9c < ssh-$user-ca-key.pem
yubico-piv-tool -k $key -a import-certificate -s 9c < ssh-$user-ca-crt.pem
You now have a SSH Host CA ready to go! The first thing you want to do is to extract the public-key for the CA, and you use OpenSSH's ssh-keygen for this, specifying OpenSC's PKCS#11 module. 
ssh-keygen -D /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so -e > ssh-$user-ca-key.pub
If you happen to use YubiKey NEO with OpenPGP using gpg-agent/scdaemon, you may get the following error message: 
no slots
cannot read public key from pkcs11
The reason is that scdaemon exclusively locks the smartcard, so no other application can access it. You need to kill scdaemon, which can be done as follows: 
gpg-connect-agent SCD KILLSCD SCD BYE /bye
The output from ssh-keygen may look like this: 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCp+gbwBHova/OnWMj99A6HbeMAGE7eP3S9lKm4/fk86Qd9bzzNNz2TKHM7V1IMEj0GxeiagDC9FMVIcbg5OaSDkuT0wGzLAJWgY2Fn3AksgA6cjA3fYQCKw0Kq4/ySFX+Zb+A8zhJgCkMWT0ZB0ZEWi4zFbG4D/q6IvCAZBtdRKkj8nJtT5l3D3TGPXCWa2A2pptGVDgs+0FYbHX0ynD0KfB4PmtR4fVQyGJjJ0MbF7fXFzQVcWiBtui8WR/Np9tvYLUJHkAXY/FjLOZf9ye0jLgP1yE10+ihe7BCxkM79GU9BsyRgRt3oArawUuU6tLgkaMN8kZPKAdq0wxNauFtH
Now all your users in your organization needs to add a line to their ~/.ssh/known_hosts as follows: 
@cert-authority *.example.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCp+gbwBHova/OnWMj99A6HbeMAGE7eP3S9lKm4/fk86Qd9bzzNNz2TKHM7V1IMEj0GxeiagDC9FMVIcbg5OaSDkuT0wGzLAJWgY2Fn3AksgA6cjA3fYQCKw0Kq4/ySFX+Zb+A8zhJgCkMWT0ZB0ZEWi4zFbG4D/q6IvCAZBtdRKkj8nJtT5l3D3TGPXCWa2A2pptGVDgs+0FYbHX0ynD0KfB4PmtR4fVQyGJjJ0MbF7fXFzQVcWiBtui8WR/Np9tvYLUJHkAXY/FjLOZf9ye0jLgP1yE10+ihe7BCxkM79GU9BsyRgRt3oArawUuU6tLgkaMN8kZPKAdq0wxNauFtH
Each sysadmin needs to go through this process, and each user needs to add one line for each sysadmin. While you could put the same key/certificate on multiple YubiKey NEOs, to allow users to only have to put one line into their file, dealing with revocation becomes a bit more complicated if you do that. If you have multiple CA keys in use at the same time, you can roll over to new CA keys without disturbing production. Users may also have different policies for different machines, so that not all sysadmins have the power to create host keys for all machines in your organization. The CA setup is now complete, however it isn't doing anything on its own. We need to sign some host keys using the CA, and to configure the hosts' sshd to use them. What you could do is something like this, for every host host.example.com that you want to create keys for: 
h=host.example.com
scp root@$h:/etc/ssh/ssh_host_rsa_key.pub .
gpg-connect-agent "SCD KILLSCD" "SCD BYE" /bye
ssh-keygen -D /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so -s ssh-$user-ca-key.pub -I $h -h -n $h -V +52w ssh_host_rsa_key.pub
scp ssh_host_rsa_key-cert.pub root@$h:/etc/ssh/
The ssh-keygen command will use OpenSC's PKCS#11 library to talk to the PIV applet on the NEO, and it will prompt you for the PIN. Enter the PIN that you set above. The output of the command would be something like this: 
Enter PIN for 'PIV_II (PIV Card Holder pin)': 
Signed host key ssh_host_rsa_key-cert.pub: id "host.example.com" serial 0 for host.example.com valid from 2015-06-16T13:39:00 to 2016-06-14T13:40:58
The host now has a SSH Host Certificate installed. To use it, you must make sure that /etc/ssh/sshd_config has the following line: 
HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub
You need to restart sshd to apply the configuration change. If you now try to connect to the host, you will likely still use the known_hosts fingerprint approach. So remove the fingerprint from your machine: 
ssh-keygen -R $h
Now if you attempt to ssh to the host, and using the -v parameter to ssh, you will see the following: 
debug1: Server host key: RSA-CERT 1b:9b:b8:5e:74:b1:31:19:35:48:48:ba:7d:d0:01:f5
debug1: Host 'host.example.com' is known and matches the RSA-CERT host certificate.
Success! One aspect that may warrant further discussion is the host keys. Here I only created host certificates for the hosts' RSA key. You could create host certificate for the DSA, ECDSA and Ed25519 keys as well. The reason I did not do that was that in this organization, we all used GnuPG's gpg-agent/scdaemon with YubiKey NEO's OpenPGP Card Applet with RSA keys for user authentication. So only the host RSA key is relevant. Revocation of a YubiKey NEO key is implemented by asking users to drop the corresponding line for one of the sysadmins, and regenerate the host certificate for the hosts that the sysadmin had created host certificates for. This is one reason users should have at least two CAs for your organization that they trust for signing host certificates, so they can migrate away from one of them to the other without interrupting operations.

13 June 2015

Craig Small: Linux 4.0 ate my docker images

I have previously written about the gitlab CI runners that use docker. Yesterday I made some changes to procps and pushed them to gitlab which would then start the CI. This morning I checked and it said build failed ok, so that s not terribly unusual. The output from the runner was: 
gitlab-ci-multi-runner 0.3.3 (dbaf96f)
Using Docker executor with image csmall/testdebian ...
Pulling docker image csmall/testdebian ...
Build failed with Error: image csmall/testdebian: not found
Hmm, I know I have that image, it just must be the runner so, let s see what images I have: 
$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
Now, I know I have images, I had about 10 or so of them, where did they go? I even looked in the /var/lib/docker directories and can see the json configs, what have you done with my images docker? Storage Drivers The first hint I got from stackexchange where someone lost their AUFS images and needed to load the aufs kernel module. Now I know there are two places or methods where docker stores its images. They are called aufs and devicemapper. There is some debate around which one is better and to be honest with what I do I don t much care, I just want it to work. The version of kernel is significant. It seems the default storage container was AUFS and this requires the aufs.ko kernel module. Linux 4.0 (the version shipped with Debian) does NOT have that module, or at least I couldn t find it. For new images, this isn t a problem. Docker will just create the new images using devicemapper and everyone is happy. The problem is where you have old aufs images, like me. I want those images. Rescue the Images I m not sure if this is the best or most correct way of getting your images, but for me it worked. I got the idea basically from someone who wanted to switch from aufs to devicemapper images for other reasons. You first need to reboot and select at the grub prompt a 3.x kernel that has aufs support. Then when the system comes up, you should see all your images, like this: 
$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
csmall/testdebian latest 6979033105a4 5 weeks ago 369.4 MB
gcc 5.1 b063030b23b8 5 weeks ago 1.225 GB
gcc 5.1.0 b063030b23b8 5 weeks ago 1.225 GB
gcc latest b063030b23b8 5 weeks ago 1.225 GB
ruby 2.1 236bf35223e7 6 weeks ago 779.8 MB
ruby 2.1.6 236bf35223e7 6 weeks ago 779.8 MB
debian jessie 41b730702607 6 weeks ago 125.1 MB
debian latest 41b730702607 6 weeks ago 125.1 MB
debian 8 41b730702607 6 weeks ago 125.1 MB
debian 8.0 41b730702607 6 weeks ago 125.1 MB
busybox buildroot-2014.02 8c2e06607696 8 weeks ago 2.433 MB
busybox latest 8c2e06607696 8 weeks ago 2.433 MB
What a relief to see this! Work out what images you need to transfer over. In my case it was just the csmall/testdebian one. You need to save it to a tar file. 
$ docker save csmall/testdebian &gt; csmall-testdebian.tar.gz
Once you have all your images you want, reboot back to your 4.x kernel. You then need to load each image back into docker. 
$ docker load csmall-testdebian.tar.gz
and then test to see its there 
$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
csmall/testdebian latest 6979033105a4 5 weeks ago 368.3 MB
The size of my image was slightly different. I m not too sure why this is the case but assume it is to do with the storage types. My CI builds now run, they re failing because my test program is trying to link with the library (it shouldn t be) but at least its not a docker problem.

Dirk Eddelbuettel: pkgKitten 0.1.3: Still creating R Packages that purr

A new release, now at version 0.1.3, of pkgKitten arrived on CRAN this morning. The main change is (optional) support of the excellent whoami package by Gabor which allows us to fill in the Author: and Maintainer: fields of the DESCRIPTION file with automatically discovered values. This is however only a Suggests: and not a Depends: to not force the added dependencies on everywhere. We also alter the default values of Title: and Description: so that they actually pass the current level of tests enforced by R CMD check --as-cran.
Changes in version 0.1.3 (2015-06-12)
  • The fields Title: and Description: in the file DESCRIPTION file are now updated such that they actually pass R CMD check on current versions of R.
  • If installed, the whoami package (version 1.1.0 or later) is now used to discover the username and email in the DESCRIPTION file.
More details about the package are at the pkgKitten webpage and the pkgKitten GitHub repo. Courtesy of CRANberries, there is also a diffstat report for this release

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Gunnar Wolf: Almost free Some experiences with the Raspberry Pi, CI20, BananaPi, CuBox-i... And whatever will follow

I know very little about hardware. I think I have a good understanding on many aspects of what happens inside a computer, but my knowledge is clearly firmer on what happens once an operating system is already running. And even then, my understanding of the lower parts of reality is shaky at most At least according to my self-evaluation, of course, comparing to people I'm honored to call "my peers". During the last ~18 months, my knowledge of this part of reality, while still far from complete, has increased quite a bit Maybe mostly showing that I'm basically very cheap: As I have come across very cheap (or even free for me!) hardware, I have tried to understand and shape what happens in levels below those where I dwell. I have been meaning to do a writeup on the MIPS Creator CI20, which was shipped to me for free (thanks++!) by Imagination Technologies; I still want to get more familiar with the board and have better knowledge before reporting on it. Just as a small advance, as this has been keeping me somewhat busy: I got this board after their offer to Debian Developers, and prompted because I'll be teaching some modules on the Embedded Linux diploma course dictated by Facultad de Ingenier a, UNAM Again, I'll blog about that later. My post today follows Riku's, titled Dystopia of things, where he very clearly finds holes in the Internet of Things offering of one specific product and one specific company, but allows for generalizations on what we will surely see as the model. Riku says:
Today, the GPL sources for hub are available - at least the kernel and a patch for busybox. The proper GPL release is still only through written offer. The sources appeared online April this year while Hub has been sold for two years already. Even if I ordered the GPL CD, it's unlikely I could build a modified system with it - too many proprietary bits. The whole GPL was invented by someone who couldn't make a printer do what he wanted. The dystopian today where I have to rewrite the whole stack running on a Linux-based system if I'm not happy what's running there as provided by OEM.
This is not exactly the situation on the boards/products (it's a disservice to call the cute CuBox-i just a board!) I mention I'm using, but it's neither too far. Being used to the easy x86 world, I am used to bitching on specific hardware that does not get promptly recognized by the Linux kernel But even with the extra work UEFI+SecureBoot introduces, getting the kernel to boot is something we just take for granted. In the MIPS and ARM worlds, this is not so much of a given; I'm still treating the whole SPL and DeviceTree world as a black box, but that's where a lot of the work happens. The boards I am working on try to make a point they are Open Hardware. The CI20 is quite impressive in this regard, as not only it has a much more complete set of on-board peripherials than any other, but a wealth of schematics, datasheets and specifications for the different parts of its components. And, of course, the mere availability of the MIPSfpga program to universities worldwide is noteworthy Completely outside of my skillset, but looks most interesting. However... Despite being so much almost-Free-with-a-capital-F, all those boards fail our definitions of freedom in several ways. And yes, they lead us to a situation similar to what Riku describes, to what Stallman feared... To a situation not really better to where we stand on openly closed-source, commodity x86 hardware: Relying on binary blobs and on non-free portions of code to just use our hardware, or at least to use many of the features that would be available to us otherwise. As an example, both the CI20 and the CuBox-i vendors provide system images able to boot what they describe as a Debian 7 system, based on a 3.0 Linux kernel (which Debian never used; IIRC the CuBox-i site said it was derived from a known-good Android kernel)... Only that it's an image resulting of somebody else installing and configuring it. Why should we trust their image to be sane? Yes, the resulting installation is quite impressive (i.e. the CI20's 3D demos are quite impressive for a system that feels otherwise sluggish, and out of my ARM experience, I'd wager it feels sluggish mostly because of a slow SSD)... I have managed to do clean Debian installs on most of my ARM machines (the CuBox-i as described in my previous blog post; this post from Elena of Valhalla'' prompted me into trying the already well documented way of running the official Debian Installer, which worked like a charm and gave me a very nice and responsive Debian 8 install Modulo yes, the Banana's non-free video interface, which AFAICT uses the non-free Mail binary driver... And which I haven't had the time to play with yet. Of course, my CuBox is in a similar situation, where it works like a charm as a personal server, but is completely worthless as a set-top box. So, with those beautiful, small, cheap SoC systems, we are close to where we stood twenty years ago with x86 Linux: Good support for a small set of peripherials, but a far cry from having a functional system with exclusively free software. ,
Despite claims of being open source, this is not open source hardware. If you are thinking of getting this device, you should also try looking into the hardware from our Community instead.
Still... Playing with these boards has taught me a lot, and has clearly taught me I'm still standing on the first steps of the n00b level. I have a lot to learn to be able to responsibly teach my part of the diploma course, and I'm very thankful for the differences in hardware (and, of course, for the hardware manufacturers, specially for the MIPS Creator CI20 and the Lemaker Banana Pi for giving me boards to work on!) I shall keep posting on this topic.

11 June 2015

Vincent Fourmond: Release 0.13 of ctioga2

Today is ctioga2's release. Unlike most other release, this one does not bring many visible features, but quite a few changes nevertheless, including:
As usual, the new version is available as a gem
~ gem update ctioga
Enjoy !

John Goerzen: Roundup of remote encrypted deduplicated backups in Linux

Since I wrote last about Linux backup tools, back in a 2008 article about BackupPC and similar toools and a 2011 article about dedpulicating filesystems, I ve revisited my personal backup strategy a bit. I still use ZFS, with my tool simplesnap that I wrote about in 2014 to perform local backups to USB drives, which get rotated offsite periodically. This has the advantage of being very fast and very secure, but I also wanted offsite backups over the Internet. I began compiling criteria, which ran like this: So, how did things stack up? Didn t meet criteria A lot of popular tools didn t meet the criteria. Here are some that I considered: Obnam and Attic/Borg Backup Obnam and Attic (and its fork Borg Backup) are both programs that have a similar concept at their heart, which is roughly this: the backup repository stores small chunks of data, indexed by a checksum. Directory trees are composed of files that are assembled out of lists of chunks, so if any given file matches another file already in the repository somewhere, the added cost is just a small amount of metadata. Obnam was eventually my tool of choice. It has built-in support for sftp, but its reliance on local filesystem semantics is very conservative and it works fine atop davfs2 (and, I d imagine, other S3-backed FUSE filesystems). Obnam s repository format is carefully documented and it is very conservatively designed through and through clearly optimized for integrity above all else, including speed. Just what a backup program should be. It has a lot of configurable options, including chunk size, caching information (dedup tables can be RAM-hungry), etc. These default to fairly conservative values, and the performance of Obnam can be significantly improved with a few simple config tweaks. Attic was also a leading contender. It has a few advantages over Obnam, actually. One is that it uses an rsync-like rolling checksum method. This means that if you add 1 byte at the beginning of a 100MB file, Attic will upload a 1-byte chunk and then reference the other chunks after that, while Obnam will have to re-upload the entire file, since its chunks start at the beginning of the file in fixed sizes. (The only time Obnam has chunks smaller than its configured chunk size is with very small files or the last chunk in a file.) Another nice feature of Attic is its use of packs , where it groups chunks together into larger pack files. This can have significant performance advantages when backing up small files, especially over high-latency protocols and links. On the downside, Attic has a hardcoded fairly small chunksize that gives it a heavy metadata load. It is not at all as configurable as Obnam, and unlike Obnam, there is nothing you can do about this. The biggest reason I avoided it though was that it uses a single monolithic index file that would have to be uploaded from scratch after each backup. I calculated that this would be many GB in size, if not even tens of GB, for my intended use, and this is just not practical over the Internet. Attic assumes that if you are going remote, you run Attic on the remote so that the rewrite of this file doesn t have to send all the data across the network. Although it does work atop davfs2, this support seemed like an afterthought and is clearly not very practical. Attic did perform much better than Obnam in some ways, largely thanks to its pack support, but the monolothic index file was going to make it simply impractical to use. There is a new fork of Attic called Borg that may, in the future, address some of these issues. Brief honorable mentions: bup, zbackup, syncany There are a few other backup tools that people are talking about which do dedup. bup is frequently mentioned, but one big problem with it is that it has no way to delete old data! In other words, it is more of an archive than a backup tool. zbackup is a really neat idea it dedups anything you feed at it, such as a tar stream or zfs send stream, and can encrypt, too. But it doesn t (yet) support removing old data either. syncany is fundamentally a syncing tool, but can also be used from the command line to do periodic syncs to a remote. It supports encryption, sftp, webdave, etc. natively, and runs on quite a number of platforms easily. However, it doesn t store a number of POSIX attributes, such as hard links, uid/gid owner, ACL, xattr, etc. This makes it impractical for use for even backing up my home directory; I make fairly frequent use of ln, both with and without -s. If there were some tool to create/restore archives of metadata, that might work out better.

Petter Reinholdtsen: Measuring and adjusting the loudness of a TV channel using bs1770gain

Television loudness is the source of frustration for viewers everywhere. Some channels are very load, others are less loud, and ads tend to shout very high to get the attention of the viewers, and the viewers do not like this. This fact is well known to the TV channels. See for example the BBC white paper "Terminology for loudness and level dBTP, LU, and all that" from 2011 for a summary of the problem domain. To better address the need for even loadness, the TV channels got together several years ago to agree on a new way to measure loudness in digital files as one step in standardizing loudness. From this came the ITU-R standard BS.1770, "Algorithms to measure audio programme loudness and true-peak audio level". The ITU-R BS.1770 specification describe an algorithm to measure loadness in LUFS (Loudness Units, referenced to Full Scale). But having a way to measure is not enough. To get the same loudness across TV channels, one also need to decide which value to standardize on. For European TV channels, this was done in the EBU Recommondaton R128, "Loudness normalisation and permitted maximum level of audio signals", which specifies a recommended level of -23 LUFS. In Norway, I have been told that NRK, TV2, MTG and SBS have decided among themselves to follow the R128 recommondation for playout from 2016-03-01. There are free software available to measure and adjust the loudness level using the LUFS. In Debian, I am aware of a library named libebur128 able to measure the loudness and since yesterday morning a new binary named bs1770gain capable of both measuring and adjusting was uploaded and is waiting for NEW processing. I plan to maintain the latter in Debian under the Debian multimedia umbrella. The free software based TV channel I am involved in, Frikanalen, plan to follow the R128 recommondation ourself as soon as we can adjust the software to do so, and the bs1770gain tool seem like a good fit for that part of the puzzle to measure loudness on new video uploaded to Frikanalen. Personally, I plan to use bs1770gain to adjust the loudness of videos I upload to Frikanalen on behalf of the NUUG member organisation. The program seem to be able to measure the LUFS value of any media file handled by ffmpeg, but I've only successfully adjusted the LUFS value of WAV files. I suspect it should be able to adjust it for all the formats handled by ffmpeg.

Steve McIntyre: Debian-branded USB keys

I've had some 8GB USB keys made, with the Debian swirl and text. By buying a reasonable number, I've got what I think is a good price for nice high-quality keys (metal body with a solid loop for attaching to a keyring). I'm now selling these for 7 pounds each, and I'm planning on bringing some to DebConf 15 too, where they'll be 10 EUR. USB key They're selling faster than I expected - if you're interested in buying one (or several!), please let me know. If there's enough demand, I may order more.

10 June 2015

Norbert Preining: Tribute to Hermann Zapf

Last weekend Hermann Zapf, one of the greatest font designers and typographers of our times, has passed away at the age of 96. Much has been written about his life, his immense sphere of influence, his excellent typefaces, and some of the links to other obituaries can be found here at the end. There is nothing of value I can contribute, but let Hermann Zapf speak for himself with a few images from two books by or dedicated to him. zapf-heart-blood-ink The books I have taken these photos from are: zapf-books In his long life, Hermann Zapf has created a wide range of typefaces, some if which have become the main staple in the printing business, most notably Palatino, Janson, Optima, and Zapf Dingbats. But there are many more: zapf-alphabets While probably not as popular anymore as some years (decades?) ago, Palatino still is one of my favorite typefaces. I often wished to have the freedom to choose my own fonts for publications, but normally that doesn t happen. Palatino has found many offspring (or imitations?), and there are excellent renderings for use with (La)TeX, too. zapf-palatino Hermann Zapf also had connections with Don Knuth and the AMS (American Mathematical Society), and he designed for the AMS the Euler typeface, using Knuth s MetaFont program. zapf-euler Last weekend, when I heard of Hermann Zapf s death, I was at a conference at Tsukuba University. What a coincidence that all the signs in the building (floor numbers, indications, etc) were set optimain another of his great typefaces, the Optima. I had a strange feeling, and reset my presentation in Optima/Euler in honor of Hermann Zapf s great life. Although I never met Hermann Zapf personally I missed the TeX conferences where he participated it was to a great extent his influence that years ago I got started in typography. Living a life like he did, dedicating himself to the beauty and expressiveness of letters, design, calligraphy, seems like a dream for me. But beauty can be found even in the strangest equation in mathematics, the queerest theorem in logic. While not all of us are as gifted as Hermann Zapf in creating beauty, we should strive to increase our ability to appreciate the beauty. The beauty of letters is subtle, evasive, and needs training, that is practice, to be appreciated. Hermann Zapf taught us how even self-training can lead to excellence. His letters, which I meet every day, are a permanent reminder to strive for beauty and clarity, and never relent in our exercise. Thanks Hermann Zapf. zapf-letters Links to tributes around the world English Linotype (Type foundry where Zapf started working)
Bigelow and Holmes (Type designers)
New York Times (Newspaper)
Type Drawers
Quarz (News magazine)
Kaveh Bazargan (next president of TUG)
TeX Overflow (Q&A website concerning TeX) German FontShop (Type foundry and internet shop)
Deutschland Funk (News station)
S ddeutsche (Newspaper)
Echo (Newspaper)
Heise (Techology News page)

9 June 2015

Daniel Silverstone: Sometimes recruiters really miss the point...

I get quite a bit of recruitment spam, especially via my LinkedIn profile, but today's Twitter-madness (recruiter scraped my twitter and then contacted me) really took the biscuit. I include my response (stripped of identifying marks) for your amusement: 
On Tue, Jun 09, 2015 at 10:30:35 +0000, Silly Recruiter wrote:
> I have come across your profile on various social media platforms today and
> after looking through them I feel you are a good fit for a permanent Java
> Developer Role I have available.
Given that you followed me on Twitter I'm assuming you found a tweet or two in
which I mention how much I hate Java?
> I can see you are currently working at Codethink and was wondering if you
> were considering a change of role?
I am not.
> The role on offer is working as a Java Developer for a company based in
> Manchester. You will be maintaining and enhancing the company's core websites
> whilst using the technologies Java, JavaScript, JSP, Struts, Hibernate XML
> and Grails.
This sounds like one of my worst nightmares.
> Are you interested in hearing more about the role? Please feel free to call
> or email me to discuss it further.
Thanks, but no.
> If not, do you know someone that is interested? We offer a  500 referral fee
> for any candidate that is successful.
I wouldn't inflict that kind of Lovecraftian nightmare of a software stack on
anyone I cared about, sorry.
D.
I then decided to take a look back over my Twitter and see if I could find what might have tripped this. There's some discussion of Minecraft modding but nothing which would suggest JavaScript, JSP, Struts, Hibernate XML or Grails. Indeed my most recent tweet regarding Java could hardly be construed as positive towards it. Sigh.

Norbert Preining: Debian/TeX Live 2015 the new layout

While updating the TeX Live package for Debian for the upcoming 2015 release, one of my aims was to convert as much as possible of the TeX configuration infrastructure to use dpkg triggers. We used the trigger mechanism already for the update of the font map configuration file (calls to updmap-sys), but till now neither for files installed into texmf trees, nor for format generation. In this blog I will describe the new management of all TeX related configuration files, while in the following one I will describe how things have been implemented with dpkg triggers. In due time this information will also be codified in the Debian TeX sub-policy. Debian - TeX Live 2015 Let us start with recalling the different actions that have to be taken when TeX packages are installed, removed, or updated. Required actions for TeX packages Filename database The simplest case are packages that only include files ment to be loaded while running TeX or Metafont or similar programs. In this case the file name database of TeX (and the one of ConTeXt) needs to be updated. Usually this is done by calling mktexlsr on the changed trees, plus, if ConTeXt is installed, running mtxrun --generate. Font map activation More complicated are font packages that provide PostScript fonts. These packages need to ship a so called map file which maps TeX font names to PostScript fonts names. Examples are lm.map which defines this mapping for the Latin Modern fonts. All these map files have to be collected in one or more updmap.cfg files, and then a call to updmap(-sys) is necessary to generate configuration files for dvips, pdftex, and some other programs. Format and hyphenation definitions Even more complicated is the case of additional hyphenation patterns and additional format definitions. At the moment, hyphenation patterns are only shipped from the TeX Live packages itself, but additional format definitions are already provided by other packages than the TeX Live ones. Again, in both cases all the different format and hyphenation pattern definitions must be combined, into configuration files fmtutil.cnf and several language.* files, respectively. After this some incantation of fmtutil(-sys) is necessary to update the format dumps of TeX. Status up to Debian/TeX Live 2014 Up to now most of these actions are done in the postinstall script of the providing packages, generated by dh_installtex. The tex-common package shipped besides the program dh_installtex also postinstall and postrm snippets that took care of all the necessary steps, and ensuring that configuration succeeds even if some of the programs were missing. While this worked quite nicely for many years, I saw several disadvantages: Furthermore, /etc/texmf/ was filled with snippet-configuration directories, e.g., /etc/texmf/hyphen.d/ and /etc/texmf/fmt.d/ where the standard configuration of TeX was saved, and system administrators could change the necessary settings. While this is a good thing, i.e., to give system administrators the chance to change their configuration, it carries a big risk: most system administrators, most users, have not the blankest idea how to write a proper format definition for TeX, nor a proper entry for a hyphenation pattern. And the TeX engines are not forgiving in this respect and will bail out immediately. Finally, having so many configuration files hanging around in /etc/texmf/ makes it complicated to update between releases, where these files might have changed completely, files might have been added or removed. New layout for Debian/TeX Live 2015 and onward Thanks to a rewrite of fmtutil in Perl (also one of my endeavors), and adding the capability to read multiple fmtutil.cnf files in the same way updmap can read multiple updmap.cfg files, starting with Debian/TeX Live 2015 we can finally do the following: So let us dive into the details, following the above order: Filename database Update of the filename databases is now done automatically via a trigger on the two directories that are searched by kpathsea and used by Debian packages: /usr/share/texlive/texmf-dist/ (for TeX Live packages) and /usr/share/texmf/ (for all other packages). Thus, if a Debian package installs files into one of these directories, automatically mktexlsr (and, if ConTeXt is installed also mtxrun) is run from the tex-common trigger action. As a consequence for package maintainers of TeX related packages, if you are not adding font maps, formats, or hyphens, there is no need anymore to call dh_installtex or do anything TeX specific besides installing the files into the proper locations. Font map activation Here nothing has changed from the previous status, I just repeat that map snippets should be installed into /var/lib/tex-common/fontmap-cfg/texlive/ (for TeX Live packages) and /var/lib/tex-common/fontmap-cfg/texmf/ (for all other packages). One snippet can contain arbitrary many Map or MixedMap lines. Example: /var/lib/tex-common/fontmap-cfg/texmf/tex-gyre.cfg: 
Map qag.map
Map qbk.map
Map qcr.map
Map qcs.map
Map qhv.map
Map qpl.map
Map qtm.map
Map qzc.map
There is a necessary action to be taken in postinst and postrm, which is calling update-texmf-conf map (but take care when writing your own code in postrm, as update-texmf-conf might not be available!). Format and hyphenation definitions Format definitions and hyphenations are now treated in the very same way as font maps. That is: Package maintainers are advised to use dh_installtex as it also caters for the removal of format dumps under /var/lib/texmf/web2c. Format of the configuration files Files under /var/lib/tex-common/fmtutil-cnf/ must consist of syntactically correct lines for fmtutil.cnf, that is white-space separated entries where the first word is the name of the format, the second the name of the engine, the third the respective hyphenation configuration file or a literal if none, and the rest of the line are additional arguments to the respective initex program, where the last entry is the ini-file. A typical example would be /var/lib/tex-common/fmtutil-cnf/texlive/texlive-omega.cnf: 
aleph aleph - *aleph.ini
lamed aleph language.dat *lambda.ini
Files under /var/lib/tex-common/hyphen-cnf have a slightly changed syntax from the ones we used to have back in /etc/texmf/hyphen.d. The new syntax is taken directly from the TeX Live database, execute statement AddHyphen. That is one line of key=value pairs, where the following keys are allowed: name, synonyms, lefthyphenmin, righthyphenmin, file, file_patterns, file_exceptions, and luaspecial. A typical example would be /var/lib/tex-common/hyphen-cnf/texlive/texlive-lang-polish.cnf: 
name=polish lefthyphenmin=2 righthyphenmin=2 file=loadhyph-pl.tex file_patterns=hyph-pl.pat.txt file_exceptions=hyph-pl.hyp.txt
In addition, package maintainers that have shipped files in /etc/texmf/fmt.d or /etc/texmf/hyphen.d are advised to remove them on package upgrade, using PACKAGE.maintscript and the proper rm_conffile directive (see the musixtex package for an example. Overriding system settings In case an administrator wants to override any of the default setting, it has become much easier in case of font maps and formats, due to the multi-feature of the respective programs. For hyphenations a separate configuration file needs to be provided. Overriding font maps and formats This can be handled by dropping a updmap.cfg or fmtutil.cnf file, respectively, into either /etc/texmf/web2c/ or /usr/local/share/texmf/web2c/ (both will work). The new files will be evaluated by the respective programs (updmap(-sys) and fmtutil(-sys), respectively), and allow by default to override entries in other files. Please see the manual page of the two programs for details, or this entry for updmap and this entry for fmtutil. Overriding hyphenation patterns When generating the various language.* files, update-language evaluates last the file /etc/texmf/web2c/local-hyphen.cnf, which should contain hyphenation patterns in the very same format as described above. Using this entry one can add additional patterns, or override given ones. Current status of uploads Several packages have already been uploaded to experimental: tex-common, musixtex, context, texinfo. What remains are the TeX Live packages itself, but here we are facing two problems: So, in case you cannot wait for the next century to happen (or, that all the packages are accepted), and still want to give it a shot, here are the apt-get lines: 
deb http://people.debian.org/~preining/TeX/ exp/
deb-src http://people.debian.org/~preining/TeX/ exp/
but be warned, there might be more changes coming in. Warnings: First, in case you have already texlive-binaries 2015.20150524.37493-2 installed, please downgrade to the new version -1 which is now on the server. They are the same packages, only the version number has changed. Yes, and if you test it, don t be surprised if suddenly you are left with a not-working TeX system. Don t tell me I didn t warn you Anyway, I hope to hear some feedback and installation reports. Conclusion While packaging has become easier for maintainers, and there is less clutter in /etc/texmf/, we now are depending on dpkg to call the tex-common triggers. Thus, if this happens rather often, then especially fmtutil-sys --all will be called several times. This is a slight regression from the previous setup, but paid off (IMNSHO) by the arguments given in the beginning. That said, there will be of course a certain time of transition until the new setup has proven its merits, and bugs are ironed out. But having bugs that need to be mostly fixed only in one package, namely tex-common, seems like an easier task to me. Enjoy.

8 June 2015

Lunar: Reproducible builds: week 6 in Stretch cycle

What happened about the reproducible builds effort for this week: Presentations On May 26th,Holger Levsen presented reproducible builds in Debian at CCC Berlin for the Datengarten 52. The presentation was in German and the slides in English. Audio and video recordings are available. Toolchain fixes Niels Thykier fixed the experimental support for the automatic creation of debug packages in debhelper that being tested as part of the reproducible toolchain. Lunar added to the reproducible build version of dpkg the normalization of permissions for files in control.tar. The patch has also been submitted based on the main branch. Daniel Kahn Gillmor proposed a patch to add support for externally-supplying build date to help2man. This sparkled a discussion about agreeing on a common name for an environment variable to hold the date that should be used. It seems opinions are converging on using SOURCE_DATE_UTC which would hold a ISO-8601 formatted date in UTC) (e.g. 2015-06-05T01:08:20Z). Kudos to Daniel, Brendan O'Dea, Ximin Luo for pushing this forward. Lunar proposed a patch to Tar upstream adding a --clamp-mtime option as a generic solution for timestamp variations in tarballs which might also be useful for dpkg. The option changes the behavior of --mtime to only use the time specified if the file mtime is newer than the given time. So far, upstream is not convinced that it would make a worthwhile addition to Tar, though. Daniel Kahn Gillmor reached out to the libburnia project to ask for help on how to make ISO created with xorriso reproducible. We should reward Thomas Schmitt with a model upstream trophy as he went through a thorough analysis of possible sources of variations and ways to improve the situation. Most of what is missing with the current version in Debian is available in the latest upstream version, but libisoburn in Debian needs help. Daniel backported the missing option for version 1.3.2-1.1. akira submitted a new issue to Doxygen upstream regarding the timestamps added to the generated manpages. Packages fixed The following 49 packages became reproducible due to changes in their build dependencies: activemq-protobuf, bnfc, bridge-method-injector, commons-exec, console-data, djinn, github-backup, haskell-authenticate-oauth, haskell-authenticate, haskell-blaze-builder, haskell-blaze-textual, haskell-bloomfilter, haskell-brainfuck, haskell-hspec-discover, haskell-pretty-show, haskell-unlambda, haskell-x509-util, haskelldb-hdbc-odbc, haskelldb-hdbc-postgresql, haskelldb-hdbc-sqlite3, hasktags, hedgewars, hscolour, https-everywhere, java-comment-preprocessor, jffi, jgit, jnr-ffi, jnr-netdb, jsoup, lhs2tex, libcolor-calc-perl, libfile-changenotify-perl, libpdl-io-hdf5-perl, libsvn-notify-mirror-perl, localizer, maven-enforcer, pyotherside, python-xlrd, python-xstatic-angular-bootstrap, rt-extension-calendar, ruby-builder, ruby-em-hiredis, ruby-redcloth, shellcheck, sisu-plexus, tomcat-maven-plugin, v4l2loopback, vim-latexsuite. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which did not make their way to the archive yet: Daniel Kahn Gilmor also started discussions for emacs24 and the unsorted lists in generated .el files, the recording of a PID number in lush, and the reproducibility of ISO images in grub2. reproducible.debian.net Notifications are now sent when the build environment for a package has changed between two builds. This is a first step before automatically building the package once more. (Holger Levsen) jenkins.debian.net was upgraded to Debian Jessie. (Holger Levsen) A new variation is now being tested: $PATH. The second build will be done with a /i/capture/the/path added. (Holger Levsen) Holger Levsen with the help of Alexander Couzens wrote extra job to test the reproducibility of coreboot. Thanks James McCoy for helping with certificate issues. Mattia Rizollo made some more internal improvements. strip-nondeterminism development Andrew Ayer released strip-nondeterminism/0.008-1. This new version fixes the gzip handler so that it now skip adding a predetermined timestamp when there was none. Holger Levsen sponsored the upload. Documentation update The pages about timestamps in manpages generated by Doxygen, GHC .hi files, and Jar files have been updated to reflect their status in upstream. Markus Koschany documented an easy way to prevent Doxygen to write timestamps in HTML output. Package reviews 83 obsolete reviews have been removed, 71 added and 48 updated this week. Meetings A meeting was held on 2015-06-03. Minutes and full logs are available. It was agreed to hold such a meeting every two weeks for the time being. The time of the next meeting should be announced soon.

Craig Small: Checking Cloudflare SSL

My website for a while has used CloudFlare as its front-end. It s a rather nice setup and means my real server gets less of a hammering, which is a good thing. A few months ago they enabled a feature called Universal SSL which I have also added to my site. Around the same time, my SSL check scripts started failing for the website, the certificate had expired apparently many many days ago. Something wasn t right. The Problem The problem was simply at first I d get emails saying The SSL certificate for enc.com.au (CN: ) has expired! . I use a program called ssl-cert-check that would check all (web, smtp, imap) of my certificates. It s very easy to forget to renew and this program runs daily and does a simple check. Running the program on the command line gave some more information, but nothing (for me) that really helped: 
$ /usr/bin/ssl-cert-check -s enc.com.au -p 443
Host Status Expires Days
----------------------------------------------- ------------ ------------ ----
unable to load certificate
140364897941136:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: TRUSTED CERTIFICATE
unable to load certificate
139905089558160:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: TRUSTED CERTIFICATE
unable to load certificate
140017829234320:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: TRUSTED CERTIFICATE
unable to load certificate
140567473276560:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: TRUSTED CERTIFICATE
enc.com.au:443 Expired -2457182
So, apparently, there was something wrong with the certificate. The problem was this was CloudFlare who seem to have a good idea on how to handle certificates and all my browsers were happy. ssl-cert-check is a shell script that uses openssl to make the connection, so the next stop was to see what openssl had to say. 
$ echo ""   /usr/bin/openssl s_client -connect enc.com.au:443 CONNECTED(00000003)
140115756086928:error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error:s23_clnt.c:769:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 345 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
---
No peer certificate available. That was the clue I was looking for. Where s my Certificate? CloudFlare Universal SSL uses certificates that have multiple domains in the one certificate. The do this by having one canonical name which is something like sni(numbers).cloudflaressl.com and then multiple Subject Alternative Names (a bit like ServerAlias in apache configurations). This way a single server with a single certificate can serve multiple domains. The way that the client tells the server which website it is looking for is Server Name Indication (SNI). As part of the TLS handshaking the client tells the server I want website www.enc.com.au . The thing is, by default, both openssl s_client and the check script do not use this feature. That was fail the SSL certificate checks were failing. The server was waiting for the client to ask what website it wanted. Modern browsers do this automatically so it just works for them. The Fix For openssl on the command line, there is a flag -servername which does the trick nicely: 
$ echo ""   /usr/bin/openssl s_client -connect enc.com.au:443 -servername enc.com.au 
CONNECTED(00000003)
depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO ECC Certification Authority
verify error:num=20:unable to get local issuer certificate
---
(lots of good SSL type messages)
That was openssl happy now. We asked the server what website we were interested in with the -servername and got the certificate. The fix for ssl-cert-check is even simpler. Like a lot of things once you know the problem, the solution is not only easy to work out but someone has done it for you already. There is a Debian bug report on this problem with a simple fix from Francois Marier. Just edit the check script and change the line that has: 
 TLSSERVERNAME="FALSE"
and change it to true. Then the script is happy too: 
$ ssl-cert-check -s enc.com.au -p https
Host Status Expires Days
----------------------------------------------- ------------ ------------ ----
enc.com.au:https Valid Sep 30 2015 114
All working and as expected! This isn t really a CloudFlare problem as such, it is just that s the first place I had seen these sort of SNI certificates being used in something I administer (or more correctly something behind the something).

7 June 2015

Thomas Goirand

There s a lot of things I d like to blog about. The last version of OpenStack, the OpenStack Liberty design summit, Kilo in the official jessie-backports repositories, etc. Maybe the most interesting part of this blog post is the last bit at the end, about a major change in the packaging workflow for OpenStack in Debian. Please read on OpenStack release names reminder
Just a reminder to make it easier for the average Debian reader who may know Debian well, but not OpenStack. OpenStack 2014.1, is Icehouse, and is the version in Jessie. 2014.2 is Juno and was released right before the freeze of Jessie. 2015.1.0 is what has been released just right after jessie, on the 30th of April. Liberty, which probably will be called 12 (as this will be the 12th release of OpenStack), and not 2015.2 (this has been discussed in Vancouver), will be released in about 5 months form now. The last summit, in Vancouver, BC, Canada, was the Liberty summit, as the OpenStack conventions are always named after the next release (since we are discussing what we will be doing during the next development cycle). OpenStack 2015.1.0, aka Kilo, release in Debian
5 days after the release of Jessie, OpenStack 2015.1.0, aka Kilo, was released. Since I couldn t upload to unstable during the freeze, I was holding a lot of packages, and when I did upload them, there was about 20 packages of mine in the FTP master s NEW queue. Though, since the DSA want to use OpenStack for the Debian infrastructure, the 20 packages were fast track into Sid, thanks to the work of Paultag (thanks man!). OpenStack Kilo in the official Jessie backports
Previously, I was only uploading OpenStack packages to Debian unstable, and maintaining a non-official Debian repositories for backports to Debian stable. However, for multiple reasons, this wasn t satisfying. Then, after packages migrated to Stretch, I started to upload to Debian backports. And right before the summit, almost everything went in. Only python-pysaml2 was missing (as I discovered too late that version 2.0.0 breaks Keystone which needs version 2.4.0). In fact, the last bits of the Kilo release reached jessie-backports in the middle of the OpenStack Liberty summit. Removal of the Debian install-guide from the official site
As there was not enough efforts working on the documentation, unfortunately, the link to the Debian install-guide has been removed from docs.openstack.org. IMO, this is mostly due to a bad communication between myself and the doc team, and also because one person who promised to work on the Debian side of the install-guide failed to warn everyone that he finally couldn t (as his managers assigned him to something else). I hope this will soon be reverted. During the Vancouver summit, I had the opportunity to discuss with the doc team about re-inclusion of the Debian install-guide. Unfortunately, as they are moving away from the XML source format to a more standard RST-based system, the current documentation is frozen, so it seems more realistic to hold on until all of the install-guide is switched to RST. OpenStack Debian image listed on apps.openstack.org
There s a new area on the openstack.org where images and apps for OpenStack are listed. Under the glance image tab, you will see that both the Jessie and the weekly testing image are listed. There s also a nice, easily identifiable Debian logo to link to these images. Also, as there are trademark problems with the Ubuntu images which makes them harder to redistribute, the Murano project (which is shipping a system to automatically install apps that to installed within a few clicks on an OpenStack cloud) decided to switch to Debian for their base image. Debian listed in the OpenStack market place
On the openstack.org site, there s a section called Marketplace. In there, vendors supporting OpenStack are listed. To get there, a vendor needs to 1/ have a defined set of OpenStack project supported by the distribution (Debian already has a way more than the required set), 2/ sign some kind of agreement with the OpenStack foundation, and 3/ pay some sponsoring money. During the summit, I discussed this with Jonathan Bryce, from the OpenStack foundation, and he agreed that Debian would not have to pay for this (since we aren t a big company with big money). I have put Jonathan and Neil (our Debian Project Leader) in touch so that signing the document may happen, though since we were all busy with the summit, I do not expect Jonathan to send the documents right away. Hopefully, this will be fixed before the end of this month of May 2015. Debian (and Ubuntu) packages collaboratively maintained upstream
Since about forever (forever is 5 years in the OpenStack world ), I pushed for more collaboration on OpenStack packaging between Debian package maintainers and Canonical. However, for some reasons which I do not wish to expand on in this blog post, it has been socially hard to do so. Also, Canonical always used BZR, which wasn t to the tastes of everyone. But during the Liberty summit, some very good things happened. First of all, Launchpad is now able to support Git (it s been a few weeks it does in fact). Even though it will take a bit of time before the Canonical server team switches to it, we can consider that this problem is already out of the way. Then it looks like Canonical are now more open than before for collaboration with Debian on the OpenStack packaging. Note that we actually did some work together already, but now we both would like a full alignment of *all* of our packages. I have discussed this with James Page, who is the head of Canonical s server team. We will first start to do so on the dependencies: this includes all of the python-*client libraries, but also all of python-oslo.* (the Oslo libs are use by all of the projects and are kind of unifying the project), plus all the third party dependencies the project relies on. James already pushed new versions of some Oslo libraries to Experimental (in order to not overwrite Kilo), which are adding transition packages needed for Ubuntu. We wont need those in Debian, but we want to welcome them to keep the same source packages. We will then later try to merge the core projects if we can. Unfortunately, since the packaging of the core projects (ie: Nova, Neutron, Cinder, Glance, etc.) was forked, merging probably will be a bit painful. We will have to make some decisions on how this happen. I am however confident that it will be done during the Liberty release cycle. Move of the packaging to upstream Gerrit
A few weeks after the summit, I wrote a proposal to upstream OpenStack dev list, with as subject: Adding packaging as an OpenStack project . What it means is that I have proposed to have Debian/Ubuntu packaging to happen in upstream infrastructure, using Gerrit, and building packages using upstream cloud. We will add all the tests we can, like building with unit tests, lintian, piuparts, adequate, but also maybe a full installation of the packages with functional tests. My proposal is here: http://lists.openstack.org/pipermail/openstack-dev/2015-May/064848.html As everything, this translates into a Gerrit review process: https://review.openstack.org/#/c/185187/ As you can read in the above thread, Fedora/RDO people, which have used a Gerrit work-flow for a long time already, also would like to join. But it looks like we ll be doing 2 teams: one for RPMs and one for debs. The proposal is currently under review by the OpenStack technical committee, which will accept (or not) if the packaging project can be fully considered as an OpenStack project. I expect a final answer next Tuesday. Note that if they deny, we can still use the stackforge namespace instead, their decision is just about the TC blessing the project as being OpenStack or not. What s very nice about this, is that not only we will have a better collaboration between Debian & Ubuntu, better automated testing and Q/A, this also opens contributions to potentially anyone. Especially, we welcome operation people, those who are doing actual big deployments. Sure, it was possible before, but I often had the feedback that many were scared to break anything when trying to contribute. Thanks to the CI/CD form upstream infra, and the Gerrit peer review process, it wont be a problem anymore. So we do expect operation people to contribute more. I will also push more upstream packaging within Mirantis, so that MOS (Mirantis OpenStack) aligns fully with Debian & Ubuntu as well. Another good thing, is that it will be easier for the puppet team to support Debian (they historically were more Ubuntu oriented), and it s going to be super easy for them to request for packaging fixes. I hope we will be able to work hand-to-hand with them, adding puppet deployment checks in the packaging repo, and packaged deployments within the puppet Gerrit review process.

Next.

Previous.