For quite a while, Lintian has been able to create performance logs (--perf-debug --perf-output perf.log) that help diagnose where lintian spends most of its runtime. I decided to make lintian output these logs on lintian.debian.org to help us spot performance issues, though I have not been very good at analysing them regularly.
At the beginning of the month, I finally got around to look a bit into one of them. My findings on IRC triggered Mattia Rizzolo to create this following graph. It shows the accumulated runtime of each check/collection measured in seconds. Find these findings, we set out to solve some of the issues. This lead to the following changes in 2.5.33 (in no particular order):
Increased buffer size in check/cruft.pm to reduce overhead [bc8b3e5] (S)
Reduced overhead in strings(1) extraction [b058fef] (P)
Removed a high overhead check that did not work [2c7b922] (P)
Legend: S: run single threaded (1:1 performance improvement). P: run in parallel.
Overall, I doubt the changes will give a revolutionary change in speed, but it should improve the 3rd, 4th and 5th slowest parts in Lintian.
Beyond runtime performance, we got a few memory optimisations in the pipeline for Lintian 2.5.34:
Remove member from non-dir nodes in the Lintian path graph (2%) [6365635]
Remove two fields at the price of computing them as needed (~5%) [a696197 + 8dacc8e]
Share some memory between package-based caches (18%) [ffc7174]
Combined these 6 commits reduce memory consumption in caches by ~33% compared to 2.5.33, when lintian processes itself. In absolute numbers, we are talking about a drop from 12.53MB to 8.48MB. The mileages can certainly vary depending on the package (mscgen only saw an ~25% improvement). Nevertheless, I was happy to list #715035as being closed in 2.5.34. :) Filed under: Debian, Lintian
Like each month, here comes a report about the work of paid contributors to Debian LTS.
Individual reports
In June, 73.50 work hours have been dispatched among 7 paid contributors. Their reports are available:
Evolution of the situation
July has seen a nice increase in terms of sponsored hours (79.50 hours per month) but the trend is unlikely to continue for the next month, worse it might be negative. While most sponsors who joined us last year in July will renew their support, there are a few where I have no confirmation yet. Many thanks to those who confirmed early: Universit Lille 3, MyTux.
Our first milestone of funding the equivalent of a half-time position is unlikely to be reached before DebConf or even this summer. If you want to prove me wrong, it s time to get in touch with your management and convince your company to contribute a small amount.
In terms of security updates waiting to be handled, the situation is similar to last month: the dla-needed.txt file lists 24 packages awaiting an update (5 more than last month), the list of open vulnerabilities in Squeeze shows about 33 affected packages in total (3 less than last month).
Thanks to our sponsors
There are no new sponsors this month. But I decided to include the number of months that the sponsor has been with us. Since we value long-lasting relations, it seemed quite natural to add this.
Debian is undertaking a huge effort to develop a reproducible builds system.
I'd like to thank you for that. This could be Debian's most important
project, with how badly computer security has been going.
uthash/1.9.9.1+git20150507-2 by Ilias Tsitsimpis, report by Jakub Wilk.
Ben Hutchings upstreamed several patches to fix Linux reproducibility issues which were quickly merged.
Some uploads fixed some reproducibility issues but not all of them:
apt-dater/1.0.2-1 uploaded by Patrick Matth i, original patch by Dhole, fixed upstream by Thomas Liske.
What happened about the reproducible
builds effort this week:
Media coverage
Daniel Stender published an English translation of the article which originally
appeared in Linux Magazin in Admin Magazine.
Toolchain fixes
Fixes landed in the Debian archive:
Lunar uploaded docbook-to-man/1:2.0.0-34 which removes a timestamp in generated manpages. Original patch by Chris Lamb.
Stefano Rivera uploaded dh-python/1.20150628-1 which now sorts namespace files. Original patch by Chris Lamb.
Christian Hofstaedtler uploaded ruby2.2/2.2.2-2 which now uses UTC for the dates in gemspec files. Original patch by Chris Lamb.
reproducible.debian.net
A new package set for the X Strike Force has been added. (h01ger)
Bugs tagged with locale are now visible in the statistics. (h01ger)
Some work has been done add tests for NetBSD. (h01ger)
Many changes by Mattia Rizzolo have been merged on the whole infrastructure:
IRC notifications when known reproducible packages stops buildig successfully.
Last oktober I was involved on behalf of
NUUG with recording the talks at
MakerCon Nordic, a conference for
the Maker movement. Since then it has been the plan to publish the
recordings on Frikanalen, which
finally happened the last few days. A few talks are missing because
the speakers asked the organizers to not publish them, but most of the
talks are available. The talks are being broadcasted on RiksTV
channel 50 and using multicast on Uninett, as well as being available
from the Frikanalen web site. The unedited recordings are
available on
Youtube too.
This is the list of talks available at the moment. Visit the
Frikanalen video
pages to view them.
Evolutionary algorithms as a design tool - from art
to robotics (Kyrre Glette)
Make and break (Hans Gerhard Meier)
Making a one year school course for young makers
(Olav Helland)
Innovation Inspiration - IPR Databases as a Source of
Inspiration (Hege Langlo)
Making a toy for makers (Erik Torstensson)
How to make 3D printer electronics (Elias Bakken)
Hovering Clouds: Looking at online tool offerings for Product
Design and 3D Printing (William Kempton)
Travelling maker stories ( yvind Nydal Dahl)
Making the first Maker Faire in Sweden (Nils Olander)
Breaking the mold: Printing 1000 s of parts (Espen Sivertsen)
Ultimaker and open source 3D printing (Erik de Bruijn)
Autodesk s 3D Printing Platform: Sparking innovation (Hilde
Sevens)
How Making is Changing the World and How You Can Too!
(Jennifer Turliuk)
Open-Source Adventuring: OpenROV, OpenExplorer and the Future of
Connected Exploration (David Lang)
Making in Norway (Haakon Karlsen Jr., Graham Hayward and Jens
Dyvik)
The Impact of the Maker Movement (Mike Senese)
Part of the reason this took so long was that the scripts NUUG had
to prepare a recording for publication were five years old and no
longer worked with the current video processing tools (command line
argument changes). In addition, we needed better audio normalization,
which sent me on a detour to
package
bs1770gain for Debian. Now this is in place and it became a lot
easier to publish NUUG videos on Frikanalen.
What happened about the reproducible
builds effort this week:
Toolchain fixes
Norbert Preining uploaded texinfo/6.0.0.dfsg.1-2 which makes texinfo indices reproducible. Original patch by Chris Lamb.
Lunar submitted recently rebased patches to make the file order of files inside .deb stable.
akira filled #789843 to make tex4ht stop printing timestamps in its HTML output by default.
Dhole wrote a patch for xutils-dev to prevent timestamps when creating gzip compresed files.
Reiner Herrmann sent a follow-up patch for wheel to use UTC as timezone when outputing timestamps.
Mattia Rizzolo started a discussion regarding the failure to build from source of subversion when -Wdate-time is added to CPPFLAGS which happens when asking dpkg-buildflags to use the reproducible profile. SWIG errors out because it doesn't recognize the aforementioned flag.
Trying to get the .buildinfo specification to more definitive state, Lunar started a discussion on storing the checksums of the binary package used in dpkg status database.
akira discovered while proposing a fix for simgrid that CMake internal command to create tarballs would record a timestamp in the gzip header. A way to prevent it is to use the GZIP environment variable to ask gzip not to store timestamps, but this will soon become unsupported. It's up for discussion if the best place to fix the problem would be to fix it for all CMake users at once.
Infrastructure-related work
Andreas Henriksson did a delayed NMU upload of pbuilder which adds minimal support for build profiles and includes several fixes from Mattia Rizzolo affecting reproducibility tests.
Neils Thykier uploaded lintian which both raises the severity of package-contains-timestamped-gzip and avoids false positives for this tag (thanks to Tomasz Buchert).
Petter Reinholdtsen filled #789761 suggesting that how-can-i-help should prompt its users about fixing reproducibility issues.
Packages fixed
The following packages became reproducible due to changes in their
build dependencies:
autorun4linuxcd,
libwildmagic,
lifelines,
plexus-i18n,
texlive-base,
texlive-extra,
texlive-lang.
The following packages became reproducible after getting fixed:
Patches submitted which have not made their way to the archive yet:
#789648 on apt-dater by Dhole: allow the build date to be set externally and set it to the time of the latest debian/changelog entry.
#789715 on simgrid by akira: fix doxygen and patch CMakeLists.txt to give GZIP=-n for tar.
#789728 on aegisub by Juan Picca: get rid of __DATE__ and __TIME__ macros.
#789747 on dipy by Juan Picca: set documentation date for Sphinx.
#789748 on jansson by Juan Picca: set documentation date for Sphinx.
#789799 on tmexpand by Chris Lamb: remove timestamps, hostname and username from the build output.
#789804 on libevocosm by Chris Lamb: removes generated files which include extra information about the build environment.
#789963 on qrfcview by Dhole: removes the timestamps from the the generated PNG icon.
#789965 on xtel by Dhole: removes extra timestamps from compressed files by gzip and from the PNG icon.
#790010 on simbody by akira: set HTML_TIMESTAMP=NO in Doxygen configuration.
#790023 on stx-btree by akira: pass HTML_TIMESTAMP=NO to Doxygen.
#790034 on siscone by akira: removes $datetime from footer.html used by Doxygen.
#790035 on thepeg by akira: set HTML_TIMESTAMP=NO in Doxygen configuration.
#790072 on libxray-spacegroup-perl by Chris Lamb: set $Storable::canonical = 1 to make space_groups.db.PL output deterministic.
#790074 on visp by akira: set HTML_TIMESTAMP=NO in Doxygen configuration.
#790081 on wfmath by akira: set HTML_TIMESTAMP=NO in Doxygen configuration.
#790082 on wreport by akira: set HTML_TIMESTAMP=NO in Doxygen configuration.
#790088 on yudit by Chris Lamb: removes timestamps from the build system by passing a static comment.
#790122 on clblas by akira: set HTML_TIMESTAMP=NO in Doxygen configuration.
#790133 on dcmtk by akira: set HTML_TIMESTAMP=NO in Doxygen configuration.
#790139 on glfw3 by akira: patch for Doxygen timestamps further improved by James Cowgill by removing $datetime from the footer.
#790228 on gtkspellmm by akira: set HTML_TIMESTAMP=NO in Doxygen configuration.
#790232 on ucblogo by Reiner Herrmann: set LC_ALL to C before sorting.
#790235 on basemap by Juan Picca: set documentation date for Sphinx.
#790258 on guymager by Reiner Herrmann: use the date from the latest debian/changelog as build date
#790309 on pelican by Chris Lamb: removes useless (and unreproducible) tests.
debbindiff development
debbindiff/23 includes a few bugfixes by Helmut Grohne that result in a significant speedup (especially on larger files). It used to exhibit the quadratic time string concatenation antipattern.
Version 24 was released on June 23rd in a hurry to fix an undefined variable introduced in the previous version. (Reiner Herrmann)
debbindiff now has a test suite! It is written using the PyTest framework (thanks Isis Lovecruft for the suggestion). The current focus has been on the comparators, and we are now at 93% of code coverage for these modules.
Several problems were identified and fixed in the process: paths appearing in output of javap, readelf, objdump, zipinfo, unsqusahfs; useless MD5 checksum and last modified date in javap output; bad handling of charsets in PO files; the destination path for gzip compressed files not ending in .gz; only metadata of cpio archives were actually compared. stat output was further trimmed to make directory comparison more useful.
Having the test suite enabled a refactoring of how comparators were written, switching from a forest of differences to a single tree. This helped removing dust from the oldest parts of the code.
Together with some other small changes, version 25 was released on June 27th. A follow up release was made the next day to fix a hole in the test suite and the resulting unidentified leftover from the comparator refactoring. (Lunar)
Documentation update
Ximin Luo improved code examples for some proposed environment variables for reference timestamps. Dhole added an example on how to fix timestamps C pre-processor macros by adding a way to set the build date externally. akira documented her fix for tex4ht timestamps.
Package reviews
94 obsolete
reviews have
been removed, 330 added and 153 updated this week.
Hats off for Chris West (Faux) who investigated many fail to build from source issues and reported the relevant bugs.
Slight improvements were made to the scripts for editing the review database, edit-notes and clean-notes. (Mattia Rizzolo)
Meetings
A meeting was held on June 23rd. Minutes are available.
The next meeting will happen on Tuesday 2015-07-07 at 17:00 UTC.
Misc.
The Linux Foundation announced that it was funding the work of Lunar and h01ger on reproducible builds in Debian and other distributions. This was further relayed in a Bits from Debian blog post.
If you manage a bunch of server machines, you will undoubtedly have run into the following OpenSSH question:
The authenticity of host 'host.example.org (1.2.3.4)' can't be established.
RSA key fingerprint is 1b:9b:b8:5e:74:b1:31:19:35:48:48:ba:7d:d0:01:f5.
Are you sure you want to continue connecting (yes/no)?
If the server is a single-user machine, where you are the only person expected to login on it, answering yes once and then using the ~/.ssh/known_hosts file to record the key fingerprint will (sort-of) work and protect you against future man-in-the-middle attacks. I say sort-of, since if you want to access the server from multiple machines, you will need to sync the known_hosts file somehow. And once your organization grows larger, and you aren t the only person that needs to login, having a policy that everyone just answers yes on first connection on all their machines is bad. The risk that someone is able to successfully MITM attack you grows every time someone types yes to these prompts.
Setting up one (or more) SSH Certificate Authority (CA) to create SSH Host Certificates, and have your users trust this CA, will allow you and your users to automatically trust the fingerprint of the host through the indirection of the SSH Host CA. I was surprised (but probably shouldn t have been) to find that deploying this is straightforward. Even setting this up with hardware-backed keys, stored on a YubiKey NEO, is easy. Below I will explain how to set this up for a hypothethical organization where two persons (sysadmins) are responsible for installing and configuring machines.
I m going to assume that you already have a couple of hosts up and running and that they run the OpenSSH daemon, so they have a /etc/ssh/ssh_host_rsa_key* public/private keypair, and that you have one YubiKey NEO with the PIV applet and that the NEO is in CCID mode. I don t believe it matters, but I m running a combination of Debian and Ubuntu machines. The Yubico PIV tool is used to configure the YubiKey NEO, and I will be using OpenSC s PKCS#11 library to connect OpenSSH with the YubiKey NEO. Let s install some tools:
Every person responsible for signing SSH Host Certificates in your organization needs a YubiKey NEO. For my example, there will only be two persons, but the number could be larger. Each one of them will have to go through the following process.
The first step is to prepare the NEO. First mode switch it to CCID using some device configuration tool, like yubikey-personalization.
ykpersonalize -m1
Then prepare the PIV applet in the YubiKey NEO. This is covered by the YubiKey NEO PIV Introduction but I ll reproduce the commands below. Do this on a disconnected machine, saving all files generated on one or more secure media and store that in a safe.
Then generate a RSA private key for the SSH Host CA, and generate a dummy X.509 certificate for that key. The only use for the X.509 certificate is to make PIV/PKCS#11 happy they want to be able to extract the public-key from the smartcard, and do that through the X.509 certificate.
You import the key and certificate to the PIV applet as follows:
yubico-piv-tool -k $key -a import-key -s 9c < ssh-$user-ca-key.pem
yubico-piv-tool -k $key -a import-certificate -s 9c < ssh-$user-ca-crt.pem
You now have a SSH Host CA ready to go! The first thing you want to do is to extract the public-key for the CA, and you use OpenSSH's ssh-keygen for this, specifying OpenSC's PKCS#11 module.
The reason is that scdaemon exclusively locks the smartcard, so no other application can access it. You need to kill scdaemon, which can be done as follows:
Each sysadmin needs to go through this process, and each user needs to add one line for each sysadmin. While you could put the same key/certificate on multiple YubiKey NEOs, to allow users to only have to put one line into their file, dealing with revocation becomes a bit more complicated if you do that. If you have multiple CA keys in use at the same time, you can roll over to new CA keys without disturbing production. Users may also have different policies for different machines, so that not all sysadmins have the power to create host keys for all machines in your organization.
The CA setup is now complete, however it isn't doing anything on its own. We need to sign some host keys using the CA, and to configure the hosts' sshd to use them. What you could do is something like this, for every host host.example.com that you want to create keys for:
The ssh-keygen command will use OpenSC's PKCS#11 library to talk to the PIV applet on the NEO, and it will prompt you for the PIN. Enter the PIN that you set above. The output of the command would be something like this:
Enter PIN for 'PIV_II (PIV Card Holder pin)':
Signed host key ssh_host_rsa_key-cert.pub: id "host.example.com" serial 0 for host.example.com valid from 2015-06-16T13:39:00 to 2016-06-14T13:40:58
The host now has a SSH Host Certificate installed. To use it, you must make sure that /etc/ssh/sshd_config has the following line:
You need to restart sshd to apply the configuration change. If you now try to connect to the host, you will likely still use the known_hosts fingerprint approach. So remove the fingerprint from your machine:
ssh-keygen -R $h
Now if you attempt to ssh to the host, and using the -v parameter to ssh, you will see the following:
debug1: Server host key: RSA-CERT 1b:9b:b8:5e:74:b1:31:19:35:48:48:ba:7d:d0:01:f5
debug1: Host 'host.example.com' is known and matches the RSA-CERT host certificate.
Success!
One aspect that may warrant further discussion is the host keys. Here I only created host certificates for the hosts' RSA key. You could create host certificate for the DSA, ECDSA and Ed25519 keys as well. The reason I did not do that was that in this organization, we all used GnuPG's gpg-agent/scdaemon with YubiKey NEO's OpenPGP Card Applet with RSA keys for user authentication. So only the host RSA key is relevant.
Revocation of a YubiKey NEO key is implemented by asking users to drop the corresponding line for one of the sysadmins, and regenerate the host certificate for the hosts that the sysadmin had created host certificates for. This is one reason users should have at least two CAs for your organization that they trust for signing host certificates, so they can migrate away from one of them to the other without interrupting operations.
Joachim Breitner uploaded ghc/7.8.4-9 which uses a hash of the command line instead of the pid when calculating a random directory name.
Lunar uploaded mozilla-devscripts/0.42 which now properly sets the timezone. Patch by Reiner Herrmann.
Dmitry Shachnev uploaded python-qt4/4.11.4+dfsg-1 which now outputs the list of imported module in a stable order. The issue has been fixed upstream. Original patch by Reiner Herrmann.
Norbert Preining uploaded tex-common/6.00 which tries to ensure reproducible builds in files generated by dh_installtex.
Daniel Kahn Gillmor's report on help2man started a discussion with Brendan O'Dea and Ximin Luo about standardizing a common environment variable that would provide a replacement for an embedded build date. After various proposals and research by Ximin about date handling in several programming languages, the best solution seems to define SOURCE_DATE_EPOCH with a value suitable for gmtime(3).
Martin Borgert wondered if Sphinx could be changed in a way that would avoid having to tweak debian/rules in packages using it to produce HTML documentation.
#788238 on kfreebsd-10 by Steven Chamberlain: fix mtimes in source tarball.
#788246 on pyepr by Juan Picca: set documentation date for Sphinx.
#788247 on grapefruit by Juan Picca: set documentation date for Sphinx.
#788249 on pastescript by Juan Picca: set documentation date for Sphinx.
#788275 on geoip-database by Reiner Herrmann: sets the embedded timestamp to the date from the latest changelog entry and normalizes the used timezone to UTC..
#788336 on irrlicht by akira: removes $datetime from the file footer.html. Now fixed upstream..
#788393 on brian by Juan Picca: set documentation date for Sphinx.
#788402 on linop by Juan Picca: set documentation date for Sphinx.
#788403 on pyevolve by Juan Picca: set documentation date for Sphinx.
#788406 on argvalidate by Juan Picca: set documentation date for Sphinx.
#788467 on astroquery by Juan Picca: set documentation date for Sphinx.
#788476 on mpi4py by Juan Picca: set documentation date for Sphinx.
#788599 on wtforms by Juan Picca: set documentation date for Sphinx.
#788618 on rumor by Holger Levsen: set TZ to UTC when building the documentation.
#788722 on thuban by Reiner Herrmann: set the date embedded in man pages to the changelog date.
reproducible.debian.net
A new variation to better notice when a package captures the environment has been introduced. (h01ger)
The test on Debian packages works by building the package twice in a short time frame. But sometimes, a mirror push can happen between the first and the second build, resulting in a package built in a different build environment. This situation is now properly detected and will run a third build automatically. (h01ger)
OpenWrt, the distribution specialized in embedded devices like small routers, is now being tested for reproducibility. The situation looks very good for their packages which seems mostly affected by timestamps in the tarball. System images will require more work on debbindiff to be better understood. (h01ger)
debbindiff development
Reiner Herrmann added support for decompling Java .class file and .ipk package files (used by OpenWrt). This is now available in version 22 released on 2015-06-14.
Documentation update
Stephen Kitt
documented
the new --insert-timestamp available since
binutils-mingw-w64 version 6.2
available to insert a ready-made date in PE binaries built with
mingw-w64.
Package reviews
195 obsolete
reviews have
been removed, 65 added and 126 updated this week.
New identified issues:
Misc.
Holger Levsen reported an issue with the locales-all package that Provides: locales but is actually missing some of the files provided by locales.
Coreboot upstream has been quick to react after the
announcement
of the tests set up the week
before. Patrick Georgi has fixed all issues in a couple of days and all
Coreboot images are now reproducible (without a payload).
SeaBIOS is one of the most frequently used payload
on PC hardware and can now be made
reproducible
too.
Paul Kocialkowski wrote to the mailing
list
asking for help on getting U-Boot tested for
reproducibility.
Lunar had a chat with maintainers of Open Build
Service to better
understand the difference between their system and what we are doing for
Debian.
The recent AWS introduction of the Elastic File System gives you an automatic grow-and-shrink capability as an NFS mount, an exciting option that takes away the previous overhead in creating shared block file systems for EC2 instances.
However it should be noted that the same auto-management of capacity is not true in the EC2 instance s Elastic Block Store (EBS) block storage disks; sizing (and resizing) is left to the customer. With current 2015 EBS, one cannot simply increase the size of an EBS Volume as the storage becomes full; (as at June 2015) an EBS volume, once created, has fixed size. For many applications, that lack of resize function on its local EBS disks is not a problem; many server instances come into existence for a brief period, process some data and then get Terminated, so long term managment is not needed.
However for a long term data store on an instance (instead of S3, which I would recommend looking closely at from a durability and pricing fit), and where I want to harness the capacity to grow (or shrink) disk for my data, then I will need to leverage some slightly more advanced disk management. And just to make life interesting, I wish to do all this while the data is live and in-use, if possible.
Enter: Logical Volume Management, or LVM. It s been around for a long, long time: LVM 2 made a debut around 2002-2003 (2.00.09 was Mar 2004) and LVM 1 was many years before that so it s pretty mature now. It s a powerful layer that sits between your raw storage block devices (as seen by the operating system), and the partitions and file systems you would normally put on them.
In this post, I ll walk through the process of getting set up with LVM on Debian in the AWS EC2 environment, and how you d do some basic maintenance to add and remove (where possible) storage with minimal interruption.
Getting Started
First a little prep work for a new Debian instance with LVM.
As I d like to give the instance its own ability to manage its storage, I ll want to provision an IAM Role for EC2 Instances for this host. In the AWS console, visit IAM, Roles, and I ll create a new Role I ll name EC2-MyServer (or similar), and at this point I ll skip giving it any actual privileges (later we ll update this). As at this date, we can only associate an instance role/profile at instance launch time.
Now I launch a base image Debian EC2 instance launched with this IAM Role/Profile; the root file system is an EBS Volume. I am going to put data that I ll be managing on a separate disk from the root file system.
First, I need to get the LVM utilities installed. It s a simple package to install: the lvm2 package. From my EC2 instance I need to get root privileges (sudo -i) and run:
apt update && apt install lvm2
After a few moments, the package is installed. I ll choose a location that I want my data to live in, such as /opt/. I want a separate disk for this task for a number of reasons:
Root EBS volumes cannot currently be encrypted using Amazon s Encrypted EBS Volumes at this point in time. If I want to also use AWS encryption option, it ll have to be on a non-root disk. Note that instance-size restrictions also exist for EBS Encrypted Volumes.
It s possibly not worth make a snapshot of the Operating System at the same time as the user content data I am saving. The OS install (except the /etc/ folder) can almost entirely be recreated from a fresh install. so why snapshot that as well (unless that s your strategy for preserving /etc, /home, etc).
The type of EBS volume that you require may be different for different data: today (Apr 2015) there is a choice of Magnetic, General Purpose 2 (GP2) SSD, and Provisioned IO/s (PIOPS) SSD, each with different costs; and depending on our volume, we may want to select one for our root volume (operating system), and something else for our data storage.
I may want to use EBS snapshots to clone the disk to another host, without the base OS bundled in with the data I am cloning.
I will create this extra volume in the AWS console and present it to this host. I ll start by using a web browser (we ll use CLI later) with the EC2 console.
The first piece of information we need to know is where my EC2 instance is running. Specifically, the AWS Region and Availability Zone (AZ). EBS Volumes only exist within the one designated AZ. If I accidentally make the volume(s) in the wrong AZ, then I won t be able to connect them to my instance. It s not a huge issue, as I would just delete the volume and try again.
I navigate to the Instances panel of the EC2 Console, and find my instance in the list:
A (redacted) list of instance from the EC2 console.
Here I can see I have located an instance and it s running in US-East-1A: that s AZ A in Region US-East-1. I can also grab this with a wget from my running Debian instance by asking the MetaData server:
The returned text is simply: us-east-1a .
Time to navigate to Elastic Block Store , choose Volumes and click Create :
Creating a volume in AWS EC2: ensure the AZ is the same as your instance
You ll see I selected that I wanted AWS to encrypt this and as noted above, at this time that doesn t include the t2 family. However, you have an option of using encryption with LVM where the customer looks after the encryption key see LUKS.
What s nice is that I can do both have AWS Encrypted Volumes, and then use encryption on top of this, but I have to manage my own keys with LUKS, and should I lose them, then I can keep all the cyphertext!
I deselected this for my example (with a t2.micro), and continue; I could see the new volume in the list as creating , and then shortly afterwards as available . Time to attach it: select the disk, and either right-click and choose Attach , or from the menu at the top of the list, chose Actions -> Attach (both do the same thing).
Attaching a volume to an instance: you ll be prompted for the compatible instances in the same AZ.
At this point in time your EC2 instance will now notice a new disk; you can confirm this with dmesg tail , and you ll see something like:
[1994151.231815] xvdg: unknown partition table
(Note the time-stamp in square brackets will be different).
Previously at this juncture you would format the entire disk with your favourite file system, mount it in the desired location, and be done. But we re adding in LVM here between this raw device, and the filesystem we are yet to make .
Marking the block device for LVM
Our first operation with LVM is to put a marker on the volume to indicate it s being use for LVM so that when we scan the block device, we know what it s for. It s a really simple command:
pvcreate /dev/xvdg
The device name above (/dev/xvdg) should correspond to the one we saw from the dmesg output above. The output of the above is rather straight forward:
Physical volume "/dev/xvdg" successfully created
Checking our EBS Volume
We can check on the EBS volume which LVM sees as a Physical Volume using the pvs command.
Here we see the entire disk is currently unused.
Creating our First Volume Group
Next step, we need to make an initial LVM Volume Group which will use our Physical volume (xvdg). The Volume Group will then contain one (or more) Logical Volumes that we ll format and use. Again, a simple command to create a volume group by giving it its first physical device that it will use:
# vgcreate OptVG /dev/xvdg
Volume group "OptVG" successfully created
And likewise we can check our set of Volume Groups with vgs :
The Attribute flags here indicate this is writable, resizable, and allocating extents in normal mode. Lets proceed to make our (first) Logical Volume in this Volume Group:
# lvcreate -n OptLV -L 4.9G OptVG
Rounding up size to full physical extent 4.90 GiB
Logical volume "OptLV" created
You ll note that I have created our Logical Volume as almost the same size as the entire Volume Group (which is currently one disk) but I left some space unused: the reason for this comes down to keeping some space available for any jobs that LVM may want to use on the disk and this will be used later when we want to move data between raw disk devices.
If I wanted to use LVM for Snapshots, then I d want to leave more space free (unallocated) again.
We can check on our Logical Volume:
The attribytes indicating that the Logical Volume is writeable, is allocating its data to the disk in inherit mode (ie, as the Volume Group is doing), and that it is active. At this stage you may also discover we have a device /dev/OptVG/OptLV, and this is what we re going to format and mount. But before we do, we should review what file system we ll use.
Filesystems
Popular Linux file systems
Name
Shrink
Grow
Journal
Max File Sz
Max Vol Sz
btrfs
Y
Y
N
16 EB
16 EB
ext3
Y off-line
Y
Y
2 TB
32 TB
ext4
Y off-line
Y
Y
16 TB
1 EB
xfs
N
Y
Y
8 EB
8 EB
zfs*
N
Y
Y
16 EB
256 ZB
For more details see Wikipedia comparison. Note that ZFS requires 3rd party kernel module of FUSE layer, so I ll discount that here. BTRFS only went stable with Linux kernel 3.10, so with Debian Jessie that s a possibility; but for tried and trusted, I ll use ext4.
The selection of ext4 also means that I ll only be able to shrink this file system off-line (unmounted).
I ll make the filesystem:
# mkfs.ext4 /dev/OptVG/OptLV
mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 1285120 4k blocks and 321280 inodes
Filesystem UUID: 4f831d17-2b80-495f-8113-580bd74389dd
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
And now mount this volume and check it out:
# mount /dev/OptVG/OptLV /opt/
# df -HT /opt
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/OptVG-OptLV ext4 5.1G 11M 4.8G 1% /opt
Lastly, we want this to be mounted next time we reboot, so edit /etc/fstab and add the line:
/dev/OptVG/OptLV /opt ext4 noatime,nodiratime 0 0
With this in place, we can now start using this disk. I selected here not to update the filesystem every time I access a file or folder updates get logged as normal but access time is just ignored.
Time to expand
After some time, our 5 GB /opt/ disk is rather full, and we need to make it bigger, but we wish to do so without any downtime. Amazon EBS doesn t support resizing volumes, so our strategy is to add a new larger volume, and remove the older one that no longer suits us; LVM and ext4 s online resize ability will allow us to do this transparently.
For this example, we ll decide that we want a 10 GB volume. It can be a different type of EBS volume to our original we re going to online-migrate all our data from one to the other.
As when we created the original 5 GB EBS volume above, create a new one in the same AZ and attach it to the host (perhaps a /dev/xvdh this time). We can check the new volume is visible with dmesg again:
[1999786.341602] xvdh: unknown partition table
And now we initalise this as a Physical volume for LVM:
# pvcreate /dev/xvdh
Physical volume "/dev/xvdh" successfully created
And then add this disk to our existing OptVG Volume Group:
# vgextend OptVG /dev/xvdh
Volume group "OptVG" successfully extended
We can now review our Volume group with vgs, and see our physical volumes with pvs:
There are now 2 Physical Volumes we have a 4.9 GB filesystem taking up space, so 10.09 GB of unallocated space in the VG.
Now its time to stop using the /dev/xvgd volume for any new requests:
# pvchange -x n /dev/xvdg
Physical volume "/dev/xvdg" changed
1 physical volume changed / 0 physical volumes not changed
At this time, our existing data is on the old disk, and our new data is on the new one. Its now that I d recommend running GNU screen (or similar) so you can detach from this shell session and reconnect, as the process of migrating the existing data can take some time (hours for large volumes):
During the move, checking the Monitoring tab in the AWS EC2 Console for the two volumes should show one with a large data Read metric, and one with a large data Write metric clearly data should be flowing off the old disk, and on to the new.
A note on disk throughput
The above move was a pretty small, and empty volume. Larger disks will take longer, naturally, so getting some speed out of the process maybe key. There s a few things we can do to tweak this:
EBS Optimised: a launch-time option that reserves network throughput from certain instance types back to the EBS service within the AZ. Depending on the size of the instance this is 500 MB/sec up to 4GB/sec. Note that for the c4 family of instances, EBS Optimised is on by default.
Size of GP2 disk: the larger the disk, the longer it can sustain high IO throughput but read this for details.
Size and speed of PIOPs disk: if consistent high IO is required, then moving to Provisioned IO disk may be useful. Looking at the (2 weeks) history of Cloudwatch logs for the old volume will give me some idea of the duty cycle of the disk IO.
Back to the move
Upon completion I can see that the disk in use is the new disk and not the old one, using pvs again:
So all 5 GB is now unused (compare to above, where only 96 MB was PFree). With that disk not containing data, I can tell LVM to remove the disk from the Volume Group:
# vgreduce OptVG /dev/xvdg
Removed "/dev/xvdg" from volume group "OptVG"
Then I cleanly wipe the labels from the volume:
# pvremove /dev/xvdg
Labels on physical volume "/dev/xvdg" successfully wiped
If I really want to clean the disk, I could choose to use shred(1) on the disk to overwrite with random data. This can take a lng time
Now the disk is completely unused and disassociated from the VG, I can return to the AWS EC2 Console, and detach the disk:
Detach an EBS volume from an EC2 instance
Wait for a few seconds, and the disk is then shown as available ; I then chose to delete the disk in the EC2 console (and stop paying for it).
Back to the Logical Volume it s still 4.9 GB, so I add 4.5 GB to it:
# lvresize -L +4.5G /dev/OptVG/OptLV
Size of logical volume OptVG/OptLV changed from 4.90 GiB (1255 extents) to 9.40 GiB (2407 extents).
Logical volume OptLV successfully resized
We now have 0.6GB free space on the physical volume (pvs confirms this).
Finally, its time to expand out ext4 file system:
# resize2fs /dev/OptVG/OptLV
resize2fs 1.42.12 (29-Aug-2014)
Filesystem at /dev/OptVG/OptLV is mounted on /opt; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/OptVG/OptLV is now 2464768 (4k) blocks long.
And with df we can now see:
# df -HT /opt/
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/OptVG-OptLV ext4 9.9G 12M 9.4G 1% /opt
Automating this
The IAM Role I made at the beginning of this post is now going to be useful. I ll start by adding an IAM Policy to the Role to permit me to List Volumes, Create Volumes, Attach Volumes and Detach Volumes to my instance-id. Lets start with creating a volume, with a policy like this:
This policy puts some restrictions on the volumes that this instance can create: only within the given Availability Zone (matching our instance), only GP2 SSD (no PIOPs volumes), and size no more than 250 GB. I ll add another policy to permit this instance role to tag volumes in this AZ that don t yet have a tag called InstanceId:
Now that I can create (and then tag) volumes, this becomes a simple procedure as to what else I can do to this volume. Deleting and creating snapshots of this volume are two obvious options, and the corresponding policy:
Of course it would be lovely if I could use a variable inside the policy condition instead of the literal string of the instance ID, but that s not currently possible.
Clearly some of the more important actions I want to take are to attach and detach a volume to my instance:
Now with this in place, we can start to fire up the AWS CLI we spoke of. We ll let the CLI inherit its credentials form the IAM Instance Role and the polices we just defined.
and at this stage, the above manipulation of the raw block device with LVM can begin. Likewise you can then use the CLI to detach and destroy any unwanted volumes if you are migrating off old block devices.
Long ago, in days of yore, we assumed that any software worth having would be
packaged by the operating system we used. Debian with its enormous pile of
software (over 20,000 sources last time I looked) looked to basically contain
every piece of free software ever. However as more and more people have come
to Linux-based and BSD-based systems, and the proliferation of *NIX-based
systems has become even more diverse, it has become harder and harder to ensure
that everyone has access to all of the software they might choose to use.
Couple that with the rapid development of new projects, who clearly want to get
users involved well before the next release cycle of a Linux-based distribution
such as Debian, and you end up with this recommendation to bypass the operating
system's packaging system and simply curl sudo bash -.
We, the OS-development literati, have come out in droves to say "eww, nasty,
don't do that please" and yet we have brought this upon ourselves. Our
tendency to invent, and reinvent, at the very basic levels of distributions has
resulted in so many operating systems and so many ways to package software (if
not in underlying package format then in policy and process) that third party
application authors simply cannot keep up. Couple that with the desire of the
consumers to not have their chosen platform discounted, and if you provide
Debian packages, you end up needing to provide for Fedora, RHEL, SuSE, SLES,
CentOS, Mint, Gentoo, Arch, etc.etc; let alone supporting all the various BSDs.
This leads to the simple expedience of curl sudo bash -.
Nobody, not even those who are most vehemently against this mechanism of
installing software, can claim that it is not quick, simple for users, easy to
copy/paste out of a web-page, and leaves all the icky complexity of sorting
things out up to a script which the computer can run, rather than the nascent
user of the software in question. As a result, many varieties of software have
ended up using this as a simple installation mechanism, from games to
orchestration frameworks - everyone can acknowledge how easy it is to use.
Now, some providers are wising up a little and ensuring that the url you are
curling is at least an https:// one. Some even omit the sudo from the
copy/paste space and have it in the script, allowing them to display some basic
information and prompting the user that this will occur as root before going
ahead and elevating. All of these myriad little tweaks to the fundamental idea
improve matters but are ultimately just putting lipstick on a fairly sad
looking pig.
So, what can be done? Well we (again the OS-development literati) got
ourselves into this horrendous mess, so it's up to us to get ourselves back
out. We're all too entrenched in our chosen packaging methodologies,
processes, and policies, to back out of those; yet we're clearly not properly
servicing a non-trivial segment of our userbase. We need to do better.
Not everyone who currently honours a curl sudo bash - is capable of
understanding why it's such a bad idea to do so. Some education may reduce
that number but it will never eliminate it.
For a long time I advocated a switch to wget && review && sudo ./script
approach instead, but the above comment, about people who don't understand why
it might be a bad idea, really applies to show how few of those users would
even be capable of starting to review a script they downloaded, let alone able
to usefully judge for themselves if it is really safe to run. Instead we need
something better, something collaborative, something capable of solving the
accessibility issues which led to the curl sudo bash - revolt in the first
place.
I don't pretend to know what that solution might be, and I don't pretend to
think I might be the one to come up with it, but I can hilight a few things I
think we'll need to solve to get there:
Any solution to this problem must be as easy as curl sudo bash - or
easier. This might mean a particular URI format which can have os-specific
ways to handle standardised inputs, or it might mean a pervasive tool which
does something like that.
Any solution must do its best to securely acquire the content the user
actually wanted. This means things like validating SSL certificates,
presenting information to the user which a layman stands a chance of
evaluating to decide if the content is likely to be what they wanted,
and then acting smoothly and cleanly to get that content onto the user's
system.
Any solution should not introduce complex file formats or reliance on any
particular implementation of a tool. Ideally it would be as easy to
implement the solution on FreeBSD in shell, or on Ubuntu as whizzy 3D GUIs
written in Haskell. (modulo the pain of working in shell of course)
The solution must be arrived at in a multi-partisan way. For such a
mechanism to be as usefully pervasive as curl sudo bash - as many platforms
as possible need to get involved. This means not only Debian, Ubuntu,
Fedora and SuSE; but also Arch, FreeBSD, NetBSD, CentOS etc. Maybe even
the OpenSolaris/Illumos people need to get involved.
Given the above, no solution can be "just get all the apps developers to learn
how to package software for all the OS distributions they want their app to run
on" since that way madness lies.
I'm sure there are other minor, and major, requirements on any useful solution
but the simple fact of the matter is that until and unless we have something
which at least meets the above, we will never be rid of curl sudo bash - :-
just like we can never seem to be rid of that one odd person at the party,
noone knows who invited them, and noone wants to tell them to leave because
they do fill a needed role, but noone really seems to like.
Until then, let's suck it up and while we might not like it, let's just let
people keep on curl sudo bash -ing until someone gets hurt.
P.S. I hate curl sudo bash - for the record.
Nota: Este art culo, en espa ol, aqu .
I m not a gamer, although probably I ve played with machines/computers more than most of the girls of my age. My path has been: handheld machine, Pong and tenis in my uncle s console, MSX (with that one, in addition to playing, I learnt what was an algorithm and a program, and I started to write small programs in Basic, and I copied and ran the source code of small games and programs to make graphics, which I found in MSX magazines). My parents considered that arcade machines in bars were like slot machines so they were banned for us (even pinball, only table soccer was saved from them, and only if my father was playing with us).
In the MSX I played Magical Tree, Galaxians, Arkanoid, Konami s Sports Games, Spanish games from Dinamic, and some arcades like Golden Axe, Xenon, and maybe some more. The next computers (PC AT, later a 286) were not so for gaming; let s say that we played more with the printer (Harvard Graphics, Bannermania ). Later, I was interested in other things more than in computer games, and later there were highschool homework, dBase III, and later the University and programming again and more, and it was the end of gaming, computer was for office and Uni homework.
Later, it came the internet and since then, reading and writing and communicating was more interesting for me than playing.
I was not good at playing, and if you are not good, you play less, and you don t get better, so you begin to find other ways to loose your time, or to win it :)
The new generation
My son is 6 years old now, and I m living with him a second adventure about games. Games have changed a lot, and the family computing try to stay in the libre software side whenever I am the one that can decide, so sometimes some challenges arise.
Android (phone and tablet)
The kid has played games in the phone and tablet with Android since he was a baby. We tried some of the last years popular games. I am not so keen of banning things, but I don t feel comfortable with the popular games for Android (advertisements, nonfree software, addictive elements, massive data recollection and possible surveillance ), so I try to control without being Cruella de Vil . Some techniques I use:
We agree in the amount of time for using the table, setting a tomato to control the time (thanks Pomodoro in F-Droid)
I set the airplane mode and shut down internet everytime that I can. I never register account or login to play (if it s mandatory to login in Google or create an account, sorry but we cannot play, or we create a new empty profile).
I put barriers to installing games (for example, I say that he should uninstall 2 or 3 games first if he wants a new one, or he should explain well why he wants that game and why the ones that were installed became boring suddenly).
I ve never bought games for Android, and I m not thinking about buying them in the future. I prefer to donate for some libre game project.
I try to divert attention to other games (not computer games, usually).
If the equivalent non-computer game exists, we play that one (tic-tac-toe, hungman, battleship )
I seldom play in the phone/tablet, unless we play together.
On the other side, in my phone there is no Google Play, so we have been able to discover the section Games of F-Droid.
We have tried (all of them available in F-Droid, emphasis in the ones that he liked best): 2048, AndroFish, Bomber, Coloring for Kids, Core, Dodge, Falling Blocks, Free Fall, Frozen Bubble, HeriSwap (this one in the tablet), Hex, HyperRogue, Meerkat Challenge, Memory, Pixel Dungeon, Robotfindskitten, Slow it!, Tux Memory, Tux Rider, Vector Pinball.
Playing in my phone with CyanogenMod and having downloaded the games from F-Droid provides a relief similar to the one when playing a non-computer game. At least with the games that I have listed above. Maybe it is because they are simpler games, or they make me remember the ones that I played time ago. But it s also because of the peace of mind of knowing that they are libre software, that have been audited by the F-Droid community, that they don t abuse the user.
The same happens with Debian, what takes me to the next part of this blogpost.
Computer games: Debian
The kid has learnt to play with the tablet and the phone before than with the computer, because our computers have no joysticks nor touchscreens. He learnt to use the touchpad before than the mouse, because it s easier and we have no mouse at home. He learnt to use the mouse at school, where they work with educative games via CD or via web, using Flash :(
So Flash player appears and dissapears from my Debian setup depending on his willness to play with the school games .
Until few time ago, in the computer we played with GCompris, ChildsPlay, and TuxPaint.
When he learnt to use the arrow keys, I installed Hannah and he liked a lot, specially when we learnt to do hannah -l 900 :)
Later, in ClanTV there were advertisements about some online computer games about their favorite series, resulting in that they need Flash or a framework called Unity3D (no, it s not Ubuntu s Unity), and after digging a bit I decided that I was not going to install that #@%! in my Debian, so when he insisted in play those games, I booted the Windows 7 partition in his father s laptop and I installed it there.
Windows is slow and sad in the computer, and those web games with that framework are not very light, so luckily they have not become very interesting.
We have not played in the computer much more, maybe some incursion in Minetest, what takes me to the next section.
(Not without stating my eternal thanks to the Games Team in Debian. I think they do a very important work and I think that next year I ll try to get involved in some way, because I know that the future of our family computer games is tied to libre games in Debian).
PlayStation 3 and Minecraft
Some time ago my husband bought a PlayStation 3 for home. The shop had discounts prices and so. He would play together with the kid an so.
The machine came home with some games for free (included in the price), but most of them were classified for +13 or so, so the only two left were Pro Evolution Soccer, and Minecraft.
I decided not to connect the machine to the network. Maybe we are loosing cool things, but I feel safer like that. So, no ethernet cable plugged, no registration in the Sony shop (or whatever its name is).
The controllers are quite complex for the three of us. They are DualShock don t-know-what, and I think there is something (software) that makes the game adaptative to the person playing, because my husband is worse player after the son plays, if they play in turns and use the same controller.
The kid liked Minecraft. I didn t know anything about that game (well, I knew that there was a libre clone called Minetest), so, for learning the basics I had a look at the wiki and searched videos about how to and we began learning. Now, the kid can read a bit so he needs less help, and he has watched a lot of videos about Minecraft, so he is interested in exploring and building.
I had a look at Minetest, and I installed it in Debian. Having to use the keyboard is a disadvantage, and we didn t know how to dig, so it was not much attractive at first sight. I have looked a bit about how to use the PS3 controller in the computer, via USB, and it seems to work, but I suppose I need to write something to match each controller button with the corresponding key and subsequent action in Minetest. This is work, and I am lazy, and the boy seems not very interested in playing with the computer.
Watching the videos we have infered that it s possible to download saved games and worlds to upload them in the videogame console. We have done some tests. I wanted to upload a saved game about an amusement park, but the file was in a folder of name NPEB01899* and even when the PS3 saw it to copy it from USB to the console, later it didn t appear in the list of saved games (our sved games were in folders named BLES01976). And renaming the folder didn t work, of course. I understood that we had met Sony s restrictions, so I searched for more info. The games are saved using an encryption key and you are not able to use saved games from consoles in other world zone or using a media different than ours (the game can be played using a disc or purchasing it in the digital shop, it seems). Very ugly all of this! I read somewhere that there is certain software (libre software, BTW) that allows to break the encryption and re-encrypt the saved game with the zone and type of media of your console, but it seems the program only works for Windows, and it needs a console ID that we have not, because we didn t register the console in the PlayStation network. All these things look shaky grounds for me, unpleasant stuff, I don t want to spend time on this, maybe I should learn a bit more about Minetest and make it work and interesting and tell Sony go fly a kite. Finally, I found a saved game in the same format as ours (BLES01976), it s not an amusement park but it is a world with interesting places to explore and many things already built, so I ve tried to import it and it worked, so my son will be happy for some time, I suppose.
We have tried Minetest in the tablet too, but the touchscreen is not comfortable for this kind of games.
I feel quite frustrated and angry about this issue of Sony s restrictions on saved games. So I suppose that in the next months I ll try to learn more about Minetest in Debian, game controllers in Debian, and games in Debian in general. So I hope to be able to offer cool stuff to my son, and he becomes more interested in playing in a safe environment which does not abuse the user.
And with this, I finish
Libre games in GNU/Linux, Debian, and info about games in internet
When we have searched info about games in internet, I found that many times you need to go out from the secure environment: webpages with links to downloads that who knows if they contain what they say they contain, advertisements, videoblogs with a language not adequate for kids (or any person that loves their mother language) That s why I believe the path is to go into detail about libre games provided by the distro you use (Debian in my case). Here I bookmark a list of website with info that surely will be useful for me, to read in depth:
Now I have an already patched source tree to add the backported
patches to. Especially in cases where the Jessie version is already
fixed this makes it rather quick to get an idea what the affected
versions are and to see how the code evolved over time.
In order for this to work properly I made (on non LTS time) some improvements to
gbp:
git-pbuilder now knows about LTS so it can create chroots like:
DIST=squeeze-lts git-pbuilder create
gbp buildpackage is now clever enough to figure out the distribution
to build for from the current branch name if you adhere to DEP14. So in
case you're building from a git branch named debian/squeeze-lts it will
automatically pass DIST=squeeze-lts to git-pbuilder. This needs
gbp pq now tries harder to preserve patch names. While having
patch names adhere to what git am writes out is nice but renaming
patches just leads to too much noise when importing and exporting
from existing packages (#761161). gbp pq still needs to
improve in preserving DEP-3 header information though (#785274).
I was first elected to the Software in the Public Interest board back in 2009. I was re-elected in 2012. This July I am up for re-election again. For a variety of reasons I ve decided not to stand; mostly a combination of the fact that I think 2 terms (6 years) is enough in a single stretch and an inability to devote as much time to the organization as I d like. I mentioned this at the May board meeting. I m planning to stay involved where I can.
My main reason for posting this here is to cause people to think about whether they might want to stand for the board. Nominations open on July 1st and run until July 13th. The main thing you need to absolutely commit to is being able to attend the monthly board meeting, which is held on IRC at 20:30 UTC on the second Thursday of the month. They tend to last at most 30 minutes. Of course there s a variety of tasks that happen in the background, such as answering queries from prospective associated projects or discussing ongoing matters on the membership or board lists depending on circumstances.
It s my firm belief that SPI do some very important work for the Free software community. Few people realise the wide variety of associated projects. SPI offload the boring admin bits around accepting donations and managing project assets (be those machines, domains, trademarks or whatever), leaving those projects able to concentrate on the actual technical side of things. Most project members don t realise the involvement of SPI, and that s largely a good thing as it indicates the system is working. However it also means that there can sometimes be a lack of people wanting to stand at election time, and an absence of diversity amongst the candidates.
I m happy to answer questions of anyone who might consider standing for the board; #spi on irc.oftc.net is a good place to ask them - I am there as Noodles.
Like each month, here comes a report about the work of paid contributors to Debian LTS.
Individual reports
In April, 81.75 work hours have been dispatched among 5 paid contributors (20.75 hours where unused hours of Ben and Holger that were re-dispatched to other contributors). Their reports are available:
Evolution of the situation
May has seen a small increase in terms of sponsored hours (66.25 hours per month) and June is going to do even better with at least a new gold sponsor. We will have no problems sustaining the increased workload it implies since three Debian developers joined the team of contributors paid by Freexian (Antoine Beaupr , Santiago Ruano Rinc n, Scott Kitterman).
The Jessie release probably shed some light on the Debian LTS project since we announced that Jessie will benefit from 5 years of support. Let s hope that the trend will continue in the following months and that we reach our first milestone of funding the equivalent of a half-time position.
In terms of security updates waiting to be handled, the situation is a bit contrasted: the dla-needed.txt file lists 28 packages awaiting an update (12 less than last month), the list of open vulnerabilities in Squeeze shows about 60 affected packages in total (4 more than last month). The extra hours helped to make a good stride in the packages awaiting an update but there are many new vulnerabilities waiting to be triaged.
Thanks to our sponsors
The new sponsors of the month are in bold.
Like each month, here comes a report about the work of paid contributors to Debian LTS.
Individual reports
In April, 81.75 work hours have been dispatched among 5 paid contributors (20.75 hours where unused hours of Ben and Holger that were re-dispatched to other contributors). Their reports are available:
Evolution of the situation
May has seen a small increase in terms of sponsored hours (66.25 hours per month) and June is going to do even better with at least a new gold sponsor. We will have no problems sustaining the increased workload it implies since three Debian developers joined the team of contributors paid by Freexian (Antoine Beaupr , Santiago Ruano Rinc n, Scott Kitterman).
The Jessie release probably shed some light on the Debian LTS project since we announced that Jessie will benefit from 5 years of support. Let s hope that the trend will continue in the following months and that we reach our first milestone of funding the equivalent of a half-time position.
In terms of security updates waiting to be handled, the situation is a bit contrasted: the dla-needed.txt file lists 28 packages awaiting an update (12 less than last month), the list of open vulnerabilities in Squeeze shows about 60 affected packages in total (4 more than last month). The extra hours helped to make a good stride in the packages awaiting an update but there are many new vulnerabilities waiting to be triaged.
Thanks to our sponsors
The new sponsors of the month are in bold.
Lately, I have been dedicating a lot of my time (well, at least compared to
what I used to) to Free Software projects. In particular, I have spent a
moderate amount of time with two projects written in Python.
In this post, I want to talk about the first, more popular project is called
coursera-dl. To be honest, I think that I may have
devoted much more time to it than to any other project in particular.
With it I started to learn (besides the practices that I already used in
Debian), how to program in Python, how to use unit tests (I started with
Python's built-in unittest framework, then progressed to
nose, and I am now using pytest),
hooking up the results of the tests with a continuous
integration system (in this case, Travis CI).
I must say that I am sold on this idea of testing software (after being a
skeptical for way too long) and I can say that I find hacking on other
projects without proper testing a bit uncomfortable, since I don't know if I
am breaking unrelated parts of the project.
My use/migration to pytest was the result of a campaign from pytest.org
called Adopt Pytest Month which a kind user of the project
let me know about. I got a
very skilled volunteer assigned from pytest to our project.
Besides learning from their pull requests, one side-effect of this whole
story was that I spent a moderate amount of hours trying to understand how
properly package and distribute things on PyPI.
One tip learned along the way: contrary to the official documentation, use
twine, notpython setup.py upload. It is more
flexible for uploading your package to PyPI.
You can see the package on PyPI. Anyway, I made the
first upload of the package to PyPI on the 1st of May and it already has
almost 1500 downloads, which is far more than what I expected.
A word of warning: there are other similarly named projects, but they don't
seem to have as much following as we have. A speculation from my side is
that this may be, perhaps, due to me spending a lot of time interacting
with users in the bug tracker that github provides.
Anyway, installation of the program is now as simple as:
pip install coursera
And all the dependencies will be neatly pulled in, without having to mess
with multi-step procedures. This is a big win for the users.
Also, I even had an offer to package the program to
have it available in Debian!
Well, despite all the time that this project demanded, I think that I have
only good things to say, especially to the original author, John Lehmann.
If you like the project, please let me know and consider
yourselves invited to participate lending a hand, testing/using the
program or triaging some bugs.
I am revamping my XMPP server and I ve written down notes on how to set up certificates to enable TLS.
I will run Debian Jessie with JabberD 2.x, using the recent jabberd2 jessie-backport. The choice of server software is not significant for the rest of this post.
Running XMPP over TLS is a good idea. So I need a X.509 PKI for this purpose. I don t want to use a third-party Certificate Authority, since that gives them the ability to man-in-the-middle my XMPP connection. Therefor I want to create my own CA. I prefer tightly scoped (per-purpose or per-application) CAs, so I will set up a CA purely to issue certificates for my XMPP server.
The current XMPP specification, RFC 6120, includes a long section 13.7 that discuss requirements on Certificates.
One complication is the requirement to include an AIA for OCSP/CRLs fortunately, it is not a strict MUST requirement but a weaker SHOULD . I note that checking revocation using OCSP and CRL is a MUST requirement for certificate validation some specification language impedence mismatch at work there.
The specification demand that the CA certificate MUST have a keyUsage extension with the digitalSignature bit set. This feels odd to me, and I m wondering if keyCertSign was intended instead. Nothing in the XMPP document, nor in any PKIX document as far as I am aware of, will verify that the digitalSignature bit is asserted in a CA certificate. Below I will assert both bits, since a CA needs the keyCertSign bit and the digitalSignature bit seems unnecessary but mostly harmless.
My XMPP/Jabber server will be chat.sjd.se and my JID will be simon@josefsson.org . This means the server certificate need to include references to both these domains. The relevant DNS records for the josefsson.org zone is as follows, see section 3.2.1 of RFC 6120 for more background.
_xmpp-client._tcp.josefsson.org. IN SRV 5 0 5222 chat.sjd.se.
_xmpp-server._tcp.josefsson.org. IN SRV 5 0 5269 chat.sjd.se.
The DNS records or the sjd.se zone is as follows:
chat.sjd.se. IN A ...
chat.sjd.se. IN AAAA ...
The following commands will generate the private key and certificate for the CA. In a production environment, you would keep the CA private key in a protected offline environment. I m asserting a expiration date ~30 years in the future. While I dislike arbitrary limits, I believe this will be many times longer than the anticipated lifelength of this setup.
Let s generate the private key and server certificate for the XMPP server. The wiki page on XMPP certificates is outdated wrt PKIX extensions. I will embed a SRV-ID field, as discussed in RFC 6120 section 13.7.1.2.1 and RFC 4985. I chose to skip the XmppAddr identifier type, even though the specification is somewhat unclear about it: section 13.7.1.2.1 says that it is no longer encouraged in certificates issued by certification authorities while section 13.7.1.4 says Use of the id-on-xmppAddr format is RECOMMENDED in the generation of certificates . The latter quote should probably have been qualified to say client certificates rather than certificates , since the latter can refer to both client and server certificates.
Note the use of a default expiration time of one month: I believe in frequent renewal of entity certificates, rather than use of revocation mechanisms.
With this setup, my XMPP server can be tested by the XMPP IM Observatory. You can see the c2s test results and the s2s test results. Of course, there are warnings regarding the trust anchor issue. It complains about a self-signed certificate in the chain. This is permitted but not recommended however when the trust anchor is not widely known, I find it useful to include it. This allows people to have a mechanism of fetching the trust anchor certificate should they want to. Some weaker cipher suites trigger warnings, which is more of a jabberd2 configuration issue and/or a concern with jabberd2 defaults.
My jabberd2 configuration is simple in c2s.xml I add a <id> entity with the require-starttls , cachain , and pemfile fields. In s2s.xml, I have the <pemfile>, <resolve-ipv6>, and <require-tls> entities.
Some final words are in order. While this setup will result in use of TLS for XMPP connections (c2s and s2s), other servers are unlikely to find my CA trust anchor, let alone be able to trust it for verifying my server certificate. I m happy to read about Peter Saint-Andre s recent SSL/TLS work, and in particular I will follow the POSH effort.
I've recently been contacted by Johannes Hubertz,
who is writing a new book about Python in German called "Softwaretests mit
Python" which will be published by Open Source Press, Munich this
summer. His book will feature some interviews, and he was kind enough to let me
write a bit about software testing. This is the interview that I gave for his
book. Johannes translated to German and it will be included in Johannes' book,
and I decided to publish it on my blog today. Following is the original
version.
How did you come to Python?
I don't recall exactly, but around ten years ago, I saw more and more people
using it and decided to take a look. Back then, I was more used to Perl. I
didn't really like Perl and was not getting a good grip on its object system.
As soon as I found an idea to work on if I remember correctly that was
rebuildd I started to code in
Python, learning the language at the same time.
I liked how Python worked, and how fast I was to able to develop and learn it,
so I decided to keep using it for my next projects. I ended up diving into
Python core for some reasons, even doing things like briefly hacking on
projects like Cython at some point, and finally ended up working on OpenStack.
OpenStack is a cloud computing platform entirely written in Python. So I've
been writing Python every day since working on it.
That's what pushed me to write
The Hacker's Guide to Python in
2013 and then self-publish it a year later in 2014, a book where I talk about
doing smart and efficient Python.
It had a great success, has even been translated in Chinese and Korean, so I'm
currently working on a second edition of the book. It has been an amazing
adventure!
Zen of Python: Which line is the most important for you and why?
I like the "There should be one and preferably only one obvious way to do
it". The opposite is probably something that scared me in languages like Perl.
But having one obvious way to do it is and something I tend to like in
functional languages like Lisp, which are in my humble opinion, even better at
that.
For a python newbie, what are the most difficult subjects in Python?
I haven't been a newbie since a while, so it's hard for me to say. I
don't think the language is hard to learn. There are some subtlety in
the language itself when you deeply dive into the internals, but for
beginners most of the concept are pretty straight-forward. If I had to
pick, in the language basics, the most difficult thing would be around
the generator objects (yield).
Nowadays I think the most difficult subject for new comers is what
version of Python to use, which libraries to rely on, and how to package
and distribute projects. Though things get better, fortunately.
When did you start using Test Driven Development and why?
I learned unit testing and TDD at school where teachers forced me to
learn Java, and I hated it. The frameworks looked complicated, and I had
the impression I was losing my time. Which I actually was, since I was
writing disposable programs that's the only thing you do at school.
Years later, when I started to write real and bigger programs (e.g. rebuildd),
I quickly ended up fixing bugs I already fixed. That recalled me about unit
tests and that it may be a good idea to start using them to stop fixing the
same things over and over again.
For a few years, I wrote less Python and more C code and Lua (for the
awesome window manager), and I didn't use any
testing. I probably lost hundreds of hours testing manually and fixing
regressions that was a good lesson. Though I had good excuses at that time
it is/was way harder to do testing in C/Lua than in Python.
Since that period, I have never stopped writing "tests". When I started to hack
on OpenStack, the project was adopting a "no test? no merge!" policy due to the
high number of regressions it had during the first releases.
I honestly don't think I could work on any project that does not have at
least a minimal test coverage. It's impossible to hack efficiently on a code
base that you're not able to test in just a simple command. It's also a real
problem for new comers in the open source world. When there are no test, you
can hack something and send a patch, and get a "you broke this" in response.
Nowadays, this kind of response sounds unacceptable to me: if there is no test,
then I didn't break anything!
In the end, it's just too much frustration to work on non tested projects as I
demonstrated in
my study of whisper source code.
What do you think to be the most often seen pitfalls of TDD and how to avoid them best?
The biggest problems are when and at what rate writing tests.
On one hand, some people starts to write too precise tests way too soon. Doing
that slows you down, especially when you are prototyping some idea or concept
you just had. That does not mean that you should not do test at all, but you
should probably start with a light coverage, until you are pretty sure that
you're not going to rip every thing and start over. On the other hand, some
people postpone writing tests for ever, and end up with no test all or a too
thin layer of test. Which makes the project with a pretty low coverage.
Basically, your test coverage should reflect the state of your project. If it's
just starting, you should build a thin layer of test so you can hack it on it
easily and remodel it if needed. The more your project grow, the more you
should make it sold and lay more tests.
Having too detailed tests is painful to make the project evolve at the start.
Having not enough in a big project makes it painful to maintain it.
Do you think, TDD fits and scales well for the big projects like OpenStack?
Not only I think it fits and scales well, but I also think it's just impossible
to not use TDD in such big projects.
When unit and functional tests coverage was weak in OpenStack at its
beginning it was just impossible to fix a bug or write a new feature without
breaking a lot of things without even noticing. We would release version N, and
a ton of old bugs present in N-2 but fixed in N-1 were reopened.
For big projects, with a lot of different use cases, configuration options,
etc, you need belt and braces. You cannot throw code in a repository thinking
it's going to work ever, and you can't afford to test everything manually at
each commit. That's just insane.
Hi everyone! As you might know Qt4 has been deprecated (in the sense "you better start to port your code") since Qt5's first release in December 19th 2012. Since that point on Qt4 received only bugfixes. Upstream is about to release the last point release, 4.8.7. This means that only severe bugs like security ones will get a chance to get solved.
Moreover upstream recommended keeping Qt4 until 2017. If we get a Debian release every 2 years that will make Jessie oldstable in 2017 and deprecated in 2018. This means we should really consider starting to port code using Qt4 to Qt5 during Stretch's developing life cycle.
It is important to note that Qt4 depends on a number of dependencies that their maintainers might want to get removed from the archive for similar reasons. In this case we will simply don't hesitate in removing their support as long as Qt4 keeps building. This normally doesn't mean API/ABI breakage but missing plugins that will diminish functionality from your apps, maybe even key ones. As an example let's take the **hypothetical** case in which the libasound2 maintainers are switching to a new libasound3 which is not API-compatible and removing libasound2 in the process. In this case we will have no choice but to remove the dependency and drop the functionality it provides. This is another of the important reasons why you should be switching to Qt5.
Qt4's webkit removalWebkit is definitely not an easy piece of code to maintain. For starters it means having a full copy of the code in the archive for both Qt4 and Qt5. Now add to that the fact that the code evolves quickly and thus having upstream support even for security bugs will be getting harder and harder. So we decided to remove Qt4's webkit from the archive. Of course we still have a lot of KDE stuff using Qt4's webkit, so it won't disappear "soon", but it will at some point.
PortingSome of us where involved in various Qt4 to Qt5 migrations [0] and we know for sure that porting stuff from Qt4 to Qt5 is much much easier and less painful than it was from Qt3 to Qt4.
We also understand that there is still a lot of software still using Qt4. In order to ease the transition time we have provided Wheezy backports for Qt5.
Don't forget to take a look at the C++ API changes page [1] whenever you start porting your application.
Temporarily shipping both Qt4 and Qt5 builds of your libraryIn case you maintain a library chances are that upstream already provides a way to build it using Qt5. Please note there is no point in shipping an application built with both flavours, please use Qt5 whenever possible. This double compilation should be left only for libraries.
You can't mix Qt4 and Qt5 in the same binary, but you may provide libraries compiled against one or the other. For example, your source package foo could provide both libqt4foo1 and libqt5foo1. You need to mangle your debian/rules and/or build system accordingly to achieve this.
A good example both for upstream code allowing both styles of compilation and debian packaging is phonon. Take a look at the CMakeLists.txt files for seeing how a source can be built against both flavours and another to debian/rules to see an example of how to handle the compilation. Just bear in mind that you need to replace $(overridden_command) with the command itself, that variable substitution comes from internal stuff from our team and you should not be using it without a very good reason. If in doubt, feel free to ask us on IRC [2] or on the mailing list [3].
This post is a follow-up to my earlier post on the
"sad state of sysadmin in the age of containers".
While I was drafting this post, that story got picked up by HackerNews, Reddit
and Twitter, sending a lot of comments and emails my way.
Surprisingly many of the comments are supportive of my impression - I would
have expected to see much more insults along the lines "you just don't like
my-favorite-tool, so you rant against using it". But a lot of people seem to share
my concerns. Thanks, you surprised me!
Here is the new rant post, in the slightly different context of big data:
Everybody is doing "big data" these days. Or at least, pretending to do so
to upper management. A lot of the time, there is no big data. People do more
data anylsis than before, and therefore stick the "big data" label on them to
promote themselves and get green light from management, isn't it?
"Big data" is not a technical term. It is a business term, referring to any
attempt to get more value out of your business by analyzing data
you did not use before. From this point of view, most of such projects
are indeed "big data" as in "data-driven revenue generation" projects.
It may be unsatisfactory to those interested in the challenges of volume and
the other "V's", but this is the reality how the term is used.
But even in those cases where the volume and complexity of the data would
warrant the use of all the new toys tools, people overlook
a major problem: security of their systems and of their data.
The currently offered "big data technology stack" is all but secure.
Sure, companies try to earn money with security add-ons such as Kerberos
authentication to sell multi-tenancy, and with offering their version of
Hadoop (their "Hadoop distribution").
The security problem is deep inside the "stack". It comes from the way this
world ticks: the world of people that constantly follow the latest
tool-of-the-day. In many of the projects, you no longer have mostly Linux
developers that co-function as system administrators, but you see a lot of
Apple iFanboys now. They live in a world where technology is outdated after
half a year, so you will not need to support product longer than that. They
love reinstalling their development environment frequently - because each time,
they get to change something. They also live in a world where you would simply
get a new model if your machine breaks down at some point. (Note that this will
not work well for your big data project, restarting it from scratch every half
year...)
And while Mac users have recently been surprisingly unaffected by various attacks
(and unconcerned about e.g. GoToFail, or the fail to fix the
rootpipe exploit)
the operating system is
not
considered to be very secure. Combining this with users who do not care is an
explosive mixture...
This type of developer, who is good at getting a prototype website for a startup
kicking in a short amount of time, rolling out new features every day to beta
test on the live users is what currently makes the Dotcom 2.0 bubble grow.
It's also this type of user that mainstream products aim at - he has already
forgotten what was half a year ago, but is looking for the next tech product
to announced soon, and willing to buy it as soon as it is available...
This attitude causes a problem at the very heart of the stack:
in the way packages are built, upgrades (and safety updates) are handled etc.
- nobody is interested in consistency or reproducability anymore.
Someone commented on my blog that all these tools "seem to be written
by 20 year old" kids. He probably is right. It wouldn't be so bad if we had
some experienced sysadmins with a cluebat around. People that have
experience on how to build systems that can be maintained for 10 years,
and securely deployed automatically, instead of relying on puppet hacks,
wget and unzipping of unsigned binary code.
I know that a lot of people don't want to hear this, but:
Your Hadoop system contains unsigned binary code in a number of
places, that people downloaded, uploaded and redownloaded a countless number of
times. There is no guarantee that .jar ever was what people
think it is.
Hadoop has a huge set of dependencies, and little of this has been
seriously audited for security - and in particular not in a way that would
allow you to check that your binaries are built from this audited code anyway.
There might be functionality hidden in the code that just sits there and waits
for a system with a hostname somewhat like "yourcompany.com" to start looking
for its command and control server to steal some key data from your company.
The way your systems are built they probably do not have much of a firewall
guarding against such. Much of the software may be constantly calling home,
and your DevOps would not notice (nor would they care, anyway).
The mentality of "big data stacks" these days is that of Windows Shareware
in the 90s. People downloading random binaries from the Internet, not
adequately checked for security (ever heard of anybody running an AntiVirus
on his Hadoop cluster?) and installing them everywhere.
And worse: not even keeping track of what they installed over time, or how.
Because the tools change every year. But what if that developer leaves? You may
never be able to get his stuff running properly again!
Fire-and-forget.
I predict that within the next 5 years, we will have a number of security
incidents in various major companies. This is industrial espionage
heaven. A lot of companies will cover it up, but some leaks will reach
mass media, and there will be a major backlash against this hipster way
of stringing together random components.
There is a big "Hadoop bubble" growing, that will eventually burst.
In order to get into a trustworthy state, the big data toolchain needs to:
Consolidate. There are too many tools for every job. There are even
too many tools to manage your too many tools, and frontends for your frontends.
Lose weight. Every project depends on way too many
other projects, each of which only contributes a tiny fragment for a
very specific use case. Get rid of most dependencies!
Modularize. If you can't get rid of a dependency, but it is
still only of interest to a small group of users, make it an optional
extension module that the user only has to install if he needs this
particular functionality.
Buildable. Make sure that everybody can build everything
from scratch, without having to rely on Maven or Ivy or SBT downloading
something automagically in the background. Test your builds offline,
with a clean build directory, and document them! Everything must be
rebuildable by any sysadmin in a reproducible way, so he can ensure a
bug fix is really applied.
Distribute. Do not rely on binary downloads from your CDN
as sole distribution channel. Instead, encourage and support alternate
means of distribution, such as the proper integration in existing
and trusted Linux distributions.
Maintain compatibility. successful big data projects will
not be fire-and-forget. Eventually, they will need to go into
production and then it will be necessary to run them over years.
It will be necessary to migrate them to newer, larger clusters. And you
must not lose all the data while doing so.
Sign. Code needs to be signed, end-of-story.
Authenticate. All downloads need to come with a way of checking the
downloaded files agree with what you uploaded.
Integrate. The key feature that makes Linux systems so very
good at servers is the all-round integrated software management. When you
tell the system to update - and you have different update channels available,
such as a more conservative "stable/LTS" channel, a channel that gets you
the latest version after basic QA, and a channel that gives you the latest
versions shortly after their upload to help with QA. It covers almost all
software on your system, so it does not matter whether the security fix is
in your kernel, web server, library, auxillary service, extension module,
scripting language etc. - it will pull this fix and update you in no time.
Now you may argue that Hortonworks, Cloudera, Bigtop etc. already provide
packages. Well ... they provide crap. They have something they call a "package",
but it fails by any quality standards. Technically, a
Wartburg is a car;
but not one that would pass todays safety regulations...
For example, they only support Ubuntu 12.04 - a three year old Ubuntu is the
latest version they support... Furthermore, these packages are roughly
the same. Cloudera eventually handed over their efforts to "the community" (in
other words, they gave up on doing it themselves, and hoped that someone else
would clean up their mess); and Hortonworks HDP (any maybe Pivotal HD, too)
is derived from these efforts, too.
Much of what they do is offering some extra documentation and training for
the packages they built using Bigtop with minimal effort.
The "spark" .deb packages of Bigtop, for example, are empty. They
forgot to include the .jars in the package. Do I really need to give
more examples of bad packaging decisions? All bigtop packages now depend on
their own version of groovy - for a single script. Instead of rewriting this
script in an already required language - or in a way that it would run
on the distribution-provided groovy version - they decided to make yet another
package, bigtop-groovy.
When I read about Hortonworks and IBM announcing their "Open Data Platform",
I could not care less. As far as I can tell, they are only sticking their label
on the existing tools anyway. Thus, I'm also not surprised that Cloudera and
MapR do not join this rebranding effort - given the low divergence of Hadoop, who
would need such a label anyway?
So why does this matter? Essentially, if anything does not work,
you are currently toast. Say there is a bug in Hadoop that makes it fail to
process your data. Your business is belly-up because of that, no data is processed
anymore, your are vegetable. Who is going to fix it? All these "distributions"
are built from the same, messy, branch. There is probably only a dozen of people
around the world who have figured this out well enough to be able to fully build
this toolchain. Apparently, none of the "Hadoop" companies
are able to support a newer Ubuntu than 2012.04 - are you sure
they have really understood what they are selling? I have doubts. All the
freelancers out there, they know how to download and use Hadoop. But
can they get that business-critical bug fix into the toolchain to get you up
and running again? This is much worse than with Linux distributions. They have
build daemons - servers that continuously check they can compile all the software
that is there. You need to type two well-documented lines to rebuild a typical
Linux package from scratch on your workstation - any experienced developer can
follow the manual, and get a fix into the package. There are even people who
try to recompile complete distributions with
a different compiler to discover compatibility issues early that may
arise in the future.
In other words, the "Hadoop distribution" they are selling you is not
code they compiled themselves. It is mostly .jar files they downloaded
from unsigned, unencrypted, unverified sources on the internet. They have no idea
how to rebuild these parts, who compiled that, and how it was built. At most,
they know for the very last layer. You can figure out how to recompile the
Hadoop .jar. But when doing so, your computer will download a lot of binaries.
It will not warn you of that, and they are included in the Hadoop distributions, too.
As is, I can not recommend to trust your business data into Hadoop.
It is probably okay to copy the data into HDFS and play with it - in particular
if you keep your cluster and development machines isolated with strong
firewalls - but be prepared to toss everything and restart from scratch. It's
not ready yet for prime time, and as they keep on adding more and more unneeded
cruft, it does not look like it will be ready anytime soon.
One more examples of the immaturity of the toolchain:
The scala package from scala-lang.org cannot be cleanly installed as
an upgrade to the old scala package that already exists in Ubuntu and
Debian (and the distributions seem to have given up on compiling a newer Scala
due to a stupid Catch-22 build process, making it very hacky to bootstrap
scala and sbt compilation).
And the "upstream" package also cannot be easily fixed, because it is not built
with standard packaging tools, but with an automagic sbt helper that lacks
important functionality (in particular, access to the Replaces: field,
or even cleaner: a way of splitting the package properly into components)
instead - obviously written by someone with 0 experience in packaging for
Ubuntu or Debian; and instead of using the proven tools, he decided to hack
some wrapper that tries to automatically do things the wrong way...
I'm convinced that most "big data" projects will turn out to be a miserable
failure. Either due to overmanagement or undermanagement, and due to
lack of experience with the data, tools, and project management...
Except that - of course - nobody will be willing to admit these failures.
Since all these projects are political projects, they by definition must
be successful, even if they never go into production, and never earn a single
dollar.
Like 
The recent 
Long ago, in days of yore, we assumed that any software worth having would be
packaged by the operating system we used. Debian with its enormous pile of
software (over 20,000 sources last time I looked) looked to basically contain
every piece of free software ever. However as more and more people have come
to Linux-based and BSD-based systems, and the proliferation of *NIX-based
systems has become even more diverse, it has become harder and harder to ensure
that everyone has access to all of the software they might choose to use.
Couple that with the rapid development of new projects, who clearly want to get
users involved well before the next release cycle of a Linux-based distribution
such as Debian, and you end up with this recommendation to bypass the operating
system's packaging system and simply 
May was the first month I started to contribute to Debian LTS under the
Freexian umbrella. In total I spent six hours working on:
I was first elected to the 
Like 
Lately, I have been dedicating a lot of my time (well, at least compared to
what I used to) to Free Software projects. In particular, I have spent a
moderate amount of time with two projects written in Python.
In this post, I want to talk about the first, more popular project is called
If you like the project, 
I've recently been contacted by 
Hi everyone! As you might know Qt4 has been deprecated (in the sense "you better start to port your code") since Qt5's first release in December 19th 2012. Since that point on Qt4 received only bugfixes. Upstream is about to release the last point release, 4.8.7. This means that only severe bugs like security ones will get a chance to get solved.
