Search Results: "ana"

1 December 2014

Iustin Pop: SSD adventures

SSD adventures and fun times! TL;DR: Oh boy. Things work now, but I'm not sure exactly what happened :( When it all worked fine almost In June, I bought a new laptop, and a new SSD for it. I used that model of SSD before (Samsung 840 Evo), although not for a long time, so I wasn't expecting anything unusual. Since the laptop is a slower one, I installed Debian as follows: connect SSD to my workstation, do an install on it (via Virtualbox connected to the raw device), disconnect and install in the laptop. First sign of trouble was that the SSD didn't boot reliably. I said - maybe my Virtualbox method (new method) wasn't right - so I reinstalled on the laptop, and everything was mostly OK. Mostly OK because rarely the laptop had issues with seeing the SSD. Maybe not cold booting, requiring a restart, or giving some ATA errors on boot (bot nothing afterwards). I didn't use the laptop too much - I mostly use it when travelling - so I didn't investigate further. Trouble begins Fast forward to two weeks ago: I was preparing to leave on a trip, so I booted the laptop (everything OK), apt-get dist-upgrade, synced my git trees, etc. Everything was fine. The next day, in the airport between two flights, the laptop doesn't boot - doesn't see the SSD at all. I tried rebooting a gazillion times, nothing. I was quite upset - at the hardware, and at me for not paying attention to the unreliability signs before. Once I arrived at the destination, I opened the laptop, tried re-seating the SSD, nothing. I bought a SATA-to-USB bridge, and surprise! Boots from the first, no issues. Diagnosis A: Laptop SATA connector has issues. I work with this SATA-to-USB bridge for a couple of days, but it was quite slow (~20MB/s), so I buy a SATA-to-USB3 cradle, which should be much faster. But the SSD was not visible in this cradle. Not only that, but it was causing the laptop to hang in the POST screen - reliably. Turn the cradle off, the laptop passes POST, turn it on, the laptop took 2 minutes to pass the POST. OK, the cradle is broken. Connect the SSD back to the USB2 thing not booting For about five minutes, it was like "dead". After that, it booted and behaved normally. I didn't know what to think, I just put it aside. So worked for the rest of the week on the USB2 bridge, with no issues (once it the SSD dropped off, but I think that was just USB being USB). So at the end of the week, diagnosis (A) still was the main contender. On the flight back home, I worked from the plane for a good number of hours, again no issue. Laptop/SSD were fully powered off before the flight, powered with no issues, worked fine. At the end of the flight I completely shut down my laptop. Diagnosis A still on top. Real trouble now After getting home and sleeping a bit, I wanted to power up the laptop just to transfer the code I wrote on the plane. But it didn't power up. No problem, I said, now I actually have access to running Linux machines and I can check what's happening. And to my surprise, the SSD was behaving erratically: The looked like this (when connected over SATA):
20:22:57 kernel: ata2.00: ATA-9: Samsung SSD 840 EVO 500GB, EXT0BB6Q, max UDMA/133
20:22:57 kernel: ata2.00: 976773168 sectors, multi 1: LBA48 NCQ (depth 31/32), AA
20:22:57 kernel: ata2.00: configured for UDMA/133
20:22:57 kernel: scsi 3:0:0:0: Direct-Access     ATA      Samsung SSD 840  EXT0 PQ: 0 ANSI: 5
20:22:57 kernel: sd 3:0:0:0: [sdc] 488397168 512-byte logical blocks: (250 GB/232 GiB)
20:22:57 kernel: sd 3:0:0:0: [sdc] Write Protect is off
20:22:57 kernel: sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
20:22:57 kernel: sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
20:22:57 kernel: ata2.00: exception Emask 0x0 SAct 0x2 SErr 0x400001 action 0x6 frozen
20:22:57 kernel: ata2: SError:   RecovData Handshk  
20:22:57 kernel: ata2.00: failed command: READ FPDMA QUEUED
20:22:57 kernel: ata2.00: cmd 60/08:08:68:01:00/00:00:00:00:00/40 tag 1 ncq 4096 in
20:22:57 kernel:         res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
20:22:57 kernel: ata2.00: status:   DRDY  
20:22:57 kernel: ata2: hard resetting link
20:22:57 kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
20:22:57 kernel: ata2.00: failed to get NCQ Send/Recv Log Emask 0x1
20:22:57 kernel: ata2.00: failed to get NCQ Send/Recv Log Emask 0x1
20:22:57 kernel: ata2.00: configured for UDMA/133
20:22:57 kernel: ata2.00: device reported invalid CHS sector 0
20:22:57 kernel: ata2: EH complete
20:22:57 kernel: ata2.00: exception Emask 0x0 SAct 0x400000 SErr 0x400001 action 0x6
20:22:57 kernel: ata2.00: irq_stat 0x44000008
20:22:57 kernel: ata2: SError:   RecovData Handshk  
20:22:57 kernel: ata2.00: failed command: READ FPDMA QUEUED
20:22:57 kernel: ata2.00: cmd 60/08:b0:08:03:00/00:00:00:00:00/40 tag 22 ncq 4096 in
20:22:57 kernel:         res 41/84:00:08:03:00/00:00:00:00:00/00 Emask 0x410 (ATA bus error) <F>
20:22:57 kernel: ata2.00: status:   DRDY ERR  
20:22:57 kernel: ata2.00: error:   ICRC ABRT  
20:22:57 kernel: ata2: hard resetting link
20:22:57 kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
20:22:57 kernel: ata2.00: failed to get NCQ Send/Recv Log Emask 0x1
20:22:57 kernel: ata2.00: failed to get NCQ Send/Recv Log Emask 0x1
20:22:57 kernel: ata2.00: configured for UDMA/133
20:22:57 kernel: ata2: EH complete
Note that all of the messages were in short order (during boot, they show the same timestamp but I don't think it was actually the same second). I tried connecting the SSD over USB2 (partially working), over USB3 (not working!), and directly over SATA (initially not working). I connected another SSD I had around (same model, just smaller capacity) over all three, it worked (so the USB3 bridge was working, at least). Diagnosing (A) was out the door now, and the situation was very clear: SSD dying/dead/almost gone. So I connected both broken SSD and empty SSD to my workstation over SATA, and started copying data (once I managed to boot with the old one being visible and working). During the data copy, I saw that the "broken" SSD behaved erratically indeed: it was copying data off it with either ~40MB/s, ~70MB/s, and ~160MB/s. Not other speed, at least not for long time, just cycling between these three; and this is a very slow speed for this SSD model. And then I remembered that there is, for this model (Samsung 840 Evo), an advisory/firmware fix that old data gets harder to access (slower and slower), due to how TLC cells levels are read/etc. I don't know exactly what "old" means, but since the partition table was written only once, it should be the oldest thing written, which would make it the most susceptible to the slowdown, and could explain the cfdisk slowness. So after the data copy, I tested this: So yes, it is something related to this, probably. Diagnosis B: "just" the Samsung Evo bug. So I proceed to (try to) upgrade the firmware: Uh oh, so it's more than this - the SSD seems to be actually broken. I don't have much experience with SSDs, I used for many years an Intel X25-M and more recently I have a couple of Samsung 840 Pro, but I never had issues until now, so maybe the cheap Evo is just cheap and fast Note that during all this drama, SMART gives good info about SSD, and SMART tests are perfect (short/long). Just a number of UDMA CRC errors so thanks for nothing. I try to the firmware upgrade a few times, I give up. Diagnosis C: SSD is just plain broken. I'll send the SSD for a replacement/fix, so (even though it's encrypted), let's try to erase it first (as I can still read/write to it). And since just dd if=/dev/zero of=/dev/sdX is too common, let's try a builtin (ATA) erase! After fighting with hdparm and the fact that my BIOS does indeed "security freeze" on the drives (so you can't change the security settings nor erase the drives), and finding an article on the net that gives a few workaround, I manage to "unfreeze" it by not only live unplugging the SATA cable, but also the power cable, and plugging them back in. Time to erase. Side-note here: So, especially given the very fast ENHANCED erase, I now believe the SSD is truly broken. Diagnosis C rules. And finally, just confusion After the SSD erase, reading (the zeros) from the SSD is really fast: more than 500MB/s, as expected. I also try once more to upgrade the firmware for the Evo bug, and it works flawlessly this time. Even rebooting the machine many times works without problems, the SSD is seen every single time. I still can't make it work well over the USB3 bridge (but I'm not very convinced that device work well, since Linux does apply quirks when it detects it). But it doesn't give any ATA errors this time, and it works (read and write) consistently fast (as per the Evo specs). I still have to let it go through a long burnin with verification, but a short one works fine, so at this point I'm just very confused as what was (is?) the actual problem with this device, and why it behaves so strange Was the laptop broken? Clearly no. Was the SSD broken, or just impacted by the slowdown issue? Time will tell. I'll put it through the paces, and see. But unless I replace it, I'll keep this just for "fast cache" use cases, and not to hold real data. Yep, computers are fun!

Joey Hess: snowdrift - sustainable crowdfunding for free software development

In a recent blog post, I mentioned how lucky I feel to keep finding ways to work on free software. In the past couple years, I've had a successful Kickstarter, and followed that up with a second crowdfunding campaign, and now a grant is funding my work. A lot to be thankful for. A one-off crowdfunding campaign to fund free software development is wonderful, if you can pull it off. It can start a new project, or kick an existing one into a higher gear. But in many ways, free software development is a poor match for kickstarter-type crowdfunding. Especially when it comes to ongoing development, which it's really hard to do a crowdfunding pitch for. That's why I was excited to find Snowdrift.coop, which has a unique approach. Imagine going to a web page for a free software project that you care about, and seeing this button: 1283 patrons will donate MORE when you pledge That's a lot stronger incentive than some paypal donation button or flattr link! The details of how it works are explained on their intro page, or see the ever-insightful and thoughtful Mike Linksvayer's blog post about it. When I found out about this, I immediately sent them a one-off donation. Later, I got to meet one of the developers face to face in Portland. I've also done a small amount of work on the Snowdrift platform, which is itself free software. (My haskell code will actually render that button above!) Free software is important, and its funding should be based, not on how lucky or good we are at kickstarter pitches, but on its quality and how useful it is to everyone. Snowdrift is the most interesting thing I've seen in this space, and I really hope they succeed. If you agree, they're running their own crowdfunding campaign right now.

Olivier Berger: Offline backup/mirror of a Moodle course, using httrack

I havent found much details online on how to perform a Moodle course mirror that could be browsed offline using httrack. This could be useful both for backup purposes, or for distant learners with connectivity issues. In my case, there s a login/password dialog that grants access to the moodle platform, which can be processed by httrack by capturing the POST form results using the catchurl option. The strategy I ve used is to add filters so that everything is excluded and only explicitely mentioned filters are then allowed to be mirrored. This allows to perform the backup connected with a user that may have high privileges, while avoiding to disappear in loops or complex links following for UI rendering variants of Moodle s interface.
Here s an example command line :
httrack -v -z -%F "Mirrored [from host %s [file %s [at %s]]]" -N "%h%p/%n%[id].%t" "http://mymoodle.example.com/login/index.php?>postfile:/home/myself/websites/mycourse/hts-post0" "http://mymoodle.example.com/course/view.php?id=42" -O "/home/myself/websites/mycourse" "-*/*" "+/login/index.php*" "+*/course/view.php*" "+*/mod/resource/view.php*" "+*/mod/page/view.php*" "+*/mod/forum/view.php*" "+*/mod/forum/discuss.php?d=*[0-9]" "+*/mod/url/view.php*" "+*/pluginfile.php/*" "+*/mod/feedback/view.php*" "+*/mod/feedback/analysis.php*" "+*/theme/*" "-*/course/view.php?id=43"
Let s comment on these different parameters : I hope this will be useful.

30 November 2014

Enrico Zini: cxx11-talk-examples

C++11 talk examples On 2014-11-27 I gave a talk about C++ and new features introduced with C++11: these are the examples. They are all licensed under the wtfpli version 2. See cxx11-talk-notes for the talk notes. Note that the wrapper interfaces turns errors from the underlying libraries into exceptions, so the method calls just do what they should, without the need of documenting special return values for error messages, and removing the need for each library to implement yet another way of reporting errors. Also note that all wrapper objects do RAII: you create them and they clean after themselves when they go out of scope. The wrapper objects also have cast operators to make them behave as the pointer or handle that they are wrapping, so that they can be transparently passed to the underlying libraries. (note: I had to add U+2063 INVISIBLE SEPARATOR to prevent noreturn statements to be misinterpreted by the blog formatter. If you copypaste the code and encounter issues, you may want to delete the noreturn statements and retype them) A gcrypt hash class This class is a light wrapper around gcrypt's hashing functions. ezhash.h
#ifndef EZHASH_H
#define EZHASH_H
#include <string>
#include <gcrypt.h>
namespace ezhash  
class Hash
 
protected:
    // members can now be initialized just like this, without needing to repeat
    // their default assignment in every constructor
    gcry_md_hd_t handle = nullptr;
public:
    Hash(int algo, unsigned int flags=0);
    ~Hash();
    // Assign 'delete' to a method to tell the compiler not to generate it
    // automatically. In this case, we make the object non-copiable.
    Hash(const Hash&) = delete;
    Hash(const Hash&&) = delete;
    Hash& operator=(const Hash&) = delete;
    // Add a buffer to the hash
    void hash_buf(const std::string& buf);
    // Add the contents of a file to the hash
    void hash_file(int fd);
    // Get a string with the hexadecimal hash
    std::string read_hex(int algo=0);
    /// Pretend that we are a gcry_md_hd_t handle
    operator gcry_md_hd_t()   return handle;  
 ;
 
#endif
ezhash.cpp
#include "ezhash.h"
#include <unistd.h>
#include <errno.h>
#include <string>
#include <cstring>
#include <sstream>
#include <iomanip>
#include <stdexcept>
using namespace std;
namespace ezhash  
namespace  
// noreturn attribute, to tell the compiler that this function never returns
[ [noreturn]] void throw_gcrypt_error(gcry_error_t err)
 
    string msg;
    msg += gcry_strsource(err);
    msg += "/";
    msg += gcry_strerror(err);
    throw runtime_error(msg);
 
string errno_str(int error)
 
    char buf[256];
#if (_POSIX_C_SOURCE >= 200112L   _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE
    strerror_r(errno, buf, 256);
    string res(buf);
#else
    string res(strerror_r(errno, buf, 256));
#endif
    return res;
 
[ [noreturn]] void throw_libc_error(int error)
 
    throw runtime_error(errno_str(error));
 
 
Hash::Hash(int algo, unsigned int flags)
 
    gcry_error_t err = gcry_md_open(&handle, algo, flags);
    if (err) throw_gcrypt_error(err);
 
Hash::~Hash()
 
    gcry_md_close(handle);
 
void Hash::hash_buf(const std::string& buf)
 
    gcry_md_write(handle, buf.data(), buf.size());
 
void Hash::hash_file(int fd)
 
    char buf[4096];
    while (true)
     
        ssize_t res = ::read(fd, buf, 4096);
        if (res < 0) ezfs::throw_libc_error();
        if (res == 0) break;
        gcry_md_write(handle, buf, res);
     
 
std::string Hash::read_hex(int algo)
 
    unsigned char* res = gcry_md_read(handle, algo);
    unsigned int len = gcry_md_get_algo_dlen(
            algo == 0 ? gcry_md_get_algo(handle) : algo);
    // Format the hash into a hex digit
    stringstream hexbuf;
    hexbuf << hex << setfill('0');
    for (unsigned i = 0; i < len; ++i)
        hexbuf << setw(2) << (unsigned)res[i];
    return hexbuf.str();
 
 
Example usage
        ezhash::Hash sha256(GCRY_MD_SHA256);
        sha256.hash_buf("ciao\n");
        sha256.hash_buf("foo\n");
        cout << sha256.read_hex() << endl;
Simple sqlite bindings Remarkably simple sqlite3 bindings based on lambda callbacks. ezsqlite.h
#ifndef EZSQLITE_H
#define EZSQLITE_H
#include <sqlite3.h>
#include <string>
#include <functional>
#include <stdexcept>
namespace ezsqlite  
/// RAII wrapper around a sqlite3 database handle
class DB
 
protected:
    sqlite3* handle = nullptr;
public:
    // Open a connection to a SQLite database
    DB(const std::string& filename, int flags=SQLITE_OPEN_READWRITE   SQLITE_OPEN_CREATE);
    DB(const DB&) = delete;
    DB(const DB&&) = delete;
    DB& operator=(const DB&) = delete;
    ~DB();
    /**
     * Execute a query, optionally calling 'callback' on every result row
     *
     * The arguments to callback are:
     *  1. number of columns
     *  2. text values of the columns
     *  3. names of the columns
     */
    // std::function can be used to wrap any callable thing in C++
    // see: http://en.cppreference.com/w/cpp/utility/functional/function
    void exec(const std::string& query, std::function<bool(int, char**, char**)> callback=nullptr);
    /// Pretend that we are a sqlite3 pointer
    operator sqlite3*()   return handle;  
 ;
 
#endif
ezsqlite.cpp
#include "ezsqlite.h"
namespace ezsqlite  
DB::DB(const std::string& filename, int flags)
 
    int res = sqlite3_open_v2(filename.c_str(), &handle, flags, nullptr);
    if (res != SQLITE_OK)
     
        // From http://www.sqlite.org/c3ref/open.html
        // Whether or not an error occurs when it is opened, resources
        // associated with the database connection handle should be
        // released by passing it to sqlite3_close() when it is no longer
        // required.
        std::string errmsg(sqlite3_errmsg(handle));
        sqlite3_close(handle);
        throw std::runtime_error(errmsg);
     
 
DB::~DB()
 
    sqlite3_close(handle);
 
namespace  
// Adapter to have sqlite3_exec call a std::function
int exec_callback(void* data, int columns, char** values, char** names)
 
    std::function<bool(int, char**, char**)> cb = *static_cast<std::function<bool(int, char**, char**)>*>(data);
    return cb(columns, values, names);
 
 
void DB::exec(const std::string& query, std::function<bool(int, char**, char**)> callback)
 
    char* errmsg;
    void* cb = callback ? &callback : nullptr;
    int res = sqlite3_exec(handle, query.c_str(), exec_callback, cb, &errmsg);
    if (res != SQLITE_OK && errmsg)
     
        // http://www.sqlite.org/c3ref/exec.html
        //
        // If the 5th parameter to sqlite3_exec() is not NULL then any error
        // message is written into memory obtained from sqlite3_malloc() and
        // passed back through the 5th parameter. To avoid memory leaks, the
        // application should invoke sqlite3_free() on error message strings
        // returned through the 5th parameter of of sqlite3_exec() after the
        // error message string is no longer needed. 
        std::string msg(errmsg);
        sqlite3_free(errmsg);
        throw std::runtime_error(errmsg);
     
 
 
Example usage
    // Connect to the database
    ezsqlite::DB db("erlug.sqlite");
    // Make sure we have a table
    db.exec(R"(
        CREATE TABLE IF NOT EXISTS files (
                name TEXT NOT NULL,
                sha256sum TEXT NOT NULL
        )
    )");
    // Read the list of files that we know
    map<string, string> files;
    db.exec("SELECT name, sha256sum FROM files", [&](int columns, char** vals, char** names)  
        if (columns != 2) return false;
        files.insert(make_pair(vals[0], vals[1]));
        return true;
     );
A fast Directory object This is a lightweight wrapper around O_PATH file descriptors for directories. I'd love to see a library of well-maintained and thin C++ bindings around libc, that do little more than turning errors into exceptions and making it also work with std::string buffers. ezfs.h
#ifndef EZFS_H
#define EZFS_H
#include <string>
#include <functional>
#include <memory>
#include <cerrno>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <dirent.h>
namespace ezfs  
class Directory
 
protected:
    int handle = -1;
public:
    Directory(const std::string& pathname, int flags=0);
    ~Directory();
    Directory(const Directory&) = delete;
    Directory(const Directory&&) = delete;
    Directory& operator=(const Directory&) = delete;
    /// List the directory contents
    void ls(std::function<void(const dirent&)> callback);
    int open(const std::string& relpath, int flags, mode_t mode=0777);
 ;
std::string errno_str(int error=errno);
[ [noreturn]] void throw_libc_error(int error=errno);
 
#endif
ezfs.cpp
#include "ezfs.h"
#include <stdexcept>
#include <memory>
#include <cstring>
#include <cstdlib>
#include <string>
#include <linux/limits.h>
using namespace std;
namespace ezfs  
string errno_str(int error)
 
    char buf[256];
#if (_POSIX_C_SOURCE >= 200112L   _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE
    strerror_r(errno, buf, 256);
    string res(buf);
#else
    string res(strerror_r(errno, buf, 256));
#endif
    return res;
 
[ [noreturn]] void throw_libc_error(int error)
 
    throw runtime_error(errno_str(error));
 
Directory::Directory(const std::string& pathname, int flags)
 
    handle = ::open(pathname.c_str(), O_PATH   O_DIRECTORY   flags);
    if (handle < 0) throw_libc_error();
 
Directory::~Directory()
 
    ::close(handle);
 
void Directory::ls(std::function<void(const dirent&)> callback)
 
    int fd = openat(handle, ".", O_DIRECTORY);
    if (fd < 0) throw_libc_error();
    // RAII Self-cleaning DIR object
    unique_ptr<DIR, std::function<void(DIR*)>> dir(fdopendir(fd), [](DIR* dir)   if (dir) closedir(dir);  );
    if (!dir)
     
        // fdopendir(3): After a successful call to fdopendir(), fd is used
        // internally by the implementation, and should not otherwise be used
        // by the application.
        //
        // but if the fdopendir call was not successful, fd is not managed by
        // DIR, and we still need to close it, otherwise we leak a file
        // descriptor.
        //
        // However, close() may modify errno, so we take note of the errno set
        // by fdopendir and raise the exception based on that.
        int fdopendir_errno = errno;
        close(fd);
        throw_libc_error(fdopendir_errno);
     
    // Size the dirent buffer properly
    const unsigned len = offsetof(dirent, d_name) + PATH_MAX + 1;
    unique_ptr<dirent, std::function<void(void*)>> dirbuf((dirent*)malloc(len), free);
    while (true)
     
        dirent* res;
        int err = readdir_r(dir.get(), dirbuf.get(), &res);
        // End of directory contents
        if (err == 0)
         
            if (res)
                callback(*res);
            else
                break;
          else
            throw_libc_error(err);
     
 
int Directory::open(const std::string& relpath, int flags, mode_t mode)
 
    int res = openat(handle, relpath.c_str(), flags, mode);
    if (res < 0) throw_libc_error();
    return res;
 
 
Example usage
        // This is almost the equivalent of running "sha256sum ."
        ezfs::Directory dir(".");
        dir.ls([&](const dirent& d)  
            if (d.d_type != DT_REG) return;
            ezhash::Hash sha256(GCRY_MD_SHA256);
            // I have no RAII wrapper around file handles at the moment, so
            // I'll have to use a try/catch for cleaning up after errors
            int fd = dir.open(d.d_name, O_RDONLY);
            try  
                sha256.hash_file(fd);
                close(fd);
              catch (...)  
                close(fd);
                throw;
             
            cout << sha256.read_hex() << "  " << d.d_name << endl;
         );

Thorsten Alteholz: My Debian Activities in November 2014

FTP assistant In contrast to the last month, this month has been rather quiet and I really liked that :-) . The stress has moved to the next team. So all in all I marked 101 packages for accept and had to reject 27 packages. As I mostly reviewed really new packages, I didn t have to file any RC bug this month. Squeeze LTS This was my fifth month that I did some work for the Squeeze LTS initiative, started by Raphael Hertzog at Freexian. This month I got assigned a workload of 14.25h and I spent these hours to upload new versions of: I also uploaded [DLA 85-1] libxml-security-java security update, but as nobody of the LTS sponsors had any interest in this package, I did this in my spare time. A package with security in its name should not be affected by security issues. This month my failure of the month has been the binutils package. Although the security team prepared the way for finding the correct patches for all those CVEs, I somehow managed to not find them. This is embarassing I am also a bit disappointed by current LTS users. All important packages have been made available for testing before uploading them to the archive. Apart from some brave fellow DDs, no other feedback was reported on debian-lts. Complaints arrived only when the packages have been finally uploaded. Do admins have enough time nowadays and don t need to use some kind of testbed? Times are changing Other packages This month I even found some time to sponsor uploads, so please welcome a new version of fastaq in experimental and patiently wait for aegaen and kmc to pass NEW. At this point I also want to mention the Debian Med Advent Calendar, which was announced in this email and already mentioned by Andreas in his latest Debian Med bits. Everybody is invited to take care of as much as possible poor souls. Support If you would like to support my Debian work you could either be part of the Freexian initiative (see above) or consider to send some bitcoins to 1JHnNpbgzxkoNexeXsTUGS6qUp5P88vHej. Contact me at donation@alteholz.eu if you prefer another way to donate. Every kind of support is most appreciated.

Enrico Zini: cxx11-talk-notes

C++11 talk notes On 2014-11-27 I gave a talk about C++ and new features introduced with C++11: these are the talk notes. See cxx11-talk-examples for the examples. (note: I had to add U+2063 INVISIBLE SEPARATOR to prevent noreturn statements to be misinterpreted by the blog formatter. If you copypaste the code and encounter issues, you may want to delete the noreturn statements and retype them) Overview of programming languages
It has to be as fast as possible, so interpreted languages are out. You don't want to micro manage memory, so C is out. You don't want to require programmers to have a degree, so C++ is out. You want fast startup and not depend on a big runtime, so Java is out. [...] (Bram Moolenaar)
C++ secret cultist protip
Do not call up what you cannot put down.
C++ is a compiled language It is now possible to use the keyword constexpr to mark functions and objects that can be used at compile time:
/*
 * constexpr tells the compiler that a variable or function can be evaluated at
 * compile time.
 *
 * constexpr functions can also be run at run time, if they are called with
 * values not known at compile time.
 *
 * See http://en.cppreference.com/w/cpp/language/constexpr for more nice examples
 *
 * It can be used to avoid using constants in code, and using instead functions
 * for computing hardware bitfields or physical values, without losing in
 * efficiency.
 */
#include <iostream>
using namespace std;
constexpr int factorial(int n)
 
    return n <= 1 ? 1 : (n * factorial(n-1));
 
int main()
 
    cout << "Compile time factorial of 6: " << factorial(6) << endl;
    cout << "Enter a number: ";
    int a;
    cin >> a;
    cout << "Run time factorial of " << a << ": " << factorial(a) << endl;
 
See also this for more nice examples. See this and this for further discussion. Multiline strings
        const char* code = R"--(
          printf("foo\tbar\n");
          return 0;
        )--";
See this. C++ memory management protip RAII: Resource Acquisition Is Instantiation This is not new in C++11, but in my experience I have rarely seen it mentioned in C++ learning material, and it does make a huge difference in my code. See this and this for details. Constructors and member initializer lists Initializers in curly braces now have their own type: std::initializer_list:
#include <string>
#include <iostream>
#include <unordered_set>
using namespace std;
// std::initializer_list< >
//   will have as its value all the elements inside the curly braces
string join(initializer_list<string> strings)
 
    string res;
    for (auto str: strings)
     
        if (!res.empty())
            res += ", ";
        res += str;
     
    return res;
 
int main()
 
    unordered_set<string> blacklist  ".", "..", ".git", ".gitignore"  ;
    cout << join(  "antani", "blinda"  ) << endl;
 
See this for details, including the new uniform initialization trick of omitting parentesis in constructors so that you can call normal constructors and initializer_list constructors with the same syntax, which looks like an interesting thing when writing generic code in templates. Type inference I can now use auto instead of a type to let the compiler automatically compute the value of something I assign to:
        auto i = 3 + 2;
        // See also https://github.com/esseks/monicelli
        vector<string> names  "antani", "blinda", "supercazzola"  ;
        for (auto i = names.cbegin(); i != names.cend(); ++i)
            cout << i;
        template<typename T>
        T frobble(const T& stuff)
         
             // This will work whatever type is returned by stuff.read()
             auto i = stuff.read();
             //  
         
See this for more details. Range-based for loop C++ now has an equivalent of the various foreach constructs found in several interpreted languages!
        for (auto i: list_of_stuff)
                cout << i << endl;
        for (auto n:  0,1,2,3,4,5 )
                cout << n << endl;
        // This construct:
        for (auto i: stuff)
        // If stuff is an array, it becomes:
        for (i = stuff, i < stuff + sizeof(stuff) / sizeof(stuff[0]); ++i)
        // If stuff has .begin() and .end() methods it becomes:
        for (i = stuff.begin(); i != stuff.end(); ++i)
        // Otherwise it becomes:
        for (i = begin(stuff); i != end(stuff); ++i)
        // And you can define begin() and end() functions for any type you
        // want, at any time
See this and this for details. Lambda functions and expressions Lambdas! Closures! Something like this:
// JavaScript
var add = function(a, b)   return a + b;  
# Python
add = lambda a, b: a + b
Becomes this:
auto add = [](int a, int b)   return a + b;  
And something like this:
// JavaScript
var a = 0;
$.each([1, 2, 3, 4], function(idx, el)   a += el  );
Becomes this:
unsigned a = 0;
std::for_each(  1, 2, 3, 4  , [&a](int el)   return a += el;  );
See this, this and this. Tuple types C++ now has a std::tuple type, that like in Python can be used to implement functions that return multiple values:
        tuple<int, string, vector<string>> parse_stuff()
         
                return make_tuple(id, name, values);
         
        string name; vector<string> values;
        // std::ignore can be used to throw away a result
        tie(ignore, name, values) = parse_stuff();
        // std::tie can also be used to do other kind of
        // multi-operations besides assignment:
        return tie(a, b, c) < tie(a1, b1, c1);
        // Is the same as:
        if (a != a1) return a < a1;
        if (b != b1) return b < b1;
        return c < c1;
See here, here and here. Regular expressions We now have regular expressions!
        std::regex re(R"((\w+)\s+(\w+))");
        string s("antani blinda");
        smatch res;
        if (regex_match(s, res, re))
            cout << "OK " << res[1] << " -- " << res[2] << endl;
The syntax is ECMAScript by default and can be optionally changed to basic, extended, awk, grep, or egrep. See here and here. General-purpose smart pointers There is std::unique_ptr to code memory ownership explicitly, and std::shared_ptr as a reference counted pointer, and smart pointers can have custom destructors:
    unique_ptr<dirent, std::function<void(void*)>> dirbuf((dirent*)malloc(len), free);
See here and here. Miscellaneous other cool things Standard attribute specifiers
string errno_str(int error)
 
    char buf[256];
#if (_POSIX_C_SOURCE >= 200112L   _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE
    strerror_r(errno, buf, 256);
    string res(buf);
#else
    string res(strerror_r(errno, buf, 256));
#endif
    return res;
 
[ [noreturn]] void throw_libc_error(int error)
 
    throw runtime_error(errno_str(error));
 
See here. Hash tables See here and look at the new containers unordered_set, unordered_map, unordered_multiset, and unordered_multimap. Multithreading There is a standard threading model, with quite a bit of library support: see here, here, here, and here for atomic data structures. Variadic templates Templates can now take variable number of arguments, and that opens possibilities for interesting code generation, like implementing a generic, type-safe printf statement, or something like this:
db.query(R"(
   INSERT INTO table NAMES (id, name, description)
     VALUES (?, ?, ?)
)", 4, "genio", "fantasia, intuizione, decisione, e velocit  di esecuzione");
See here and here. Essential tools You need at least g++ 4.8 or clang 3.3 to have full C++11 support. They will be both available in jessie, and for wheezy you can use the nightly clang packages repository. I cannot think of a good excuse not to use -Wall on new code. scan-build from clang is another nice resource for catching even more potential problems at compile time. valgrind is a great tool for runtime code analysis: valgrind --tool=memcheck (the default) will check your program for wrong memory accesses and memory leaks. valgrind --tool=callgrind will trace function calls for profiling, to be analyzed with kcachegrind. valgrind --tool=helgrind can check multi-threaded programs for suspicious concurrent memory accesse patterns. And of course gdb: a nice trick with C++ is to issue catch throw to get a breakpoint at the point where an exception is being thrown. help catch provides a list of other interesting catch examples. Coredump tips: ulimit -c to enable core dumps, triggering a core dump with ^\, opening a core with gdb program core, and more details on man 5 core. An extra gdb tip, which is not related to C++ but helped me considerably recently, is that it can be attached to running python programs to get a live Python traceback.

28 November 2014

Richard Hartmann: Release Critical Bug report for Week 48

Holy bug-count-drop, Batman! Some bugs which need loving: The UDD bugs interface currently knows about the following release critical bugs: How do we compare to the Squeeze release cycle?
Week Squeeze Wheezy Jessie
43 284 (213+71) 468 (332+136) 319 (240+79)
44 261 (201+60) 408 (265+143) 274 (224+50)
45 261 (205+56) 425 (291+134) 295 (229+66)
46 271 (200+71) 401 (258+143) 427 (313+114)
47 283 (209+74) 366 (221+145) 342 (260+82)
48 256 (177+79) 378 (230+148) 274 (189+85)
49 256 (180+76) 360 (216+155)
50 204 (148+56) 339 (195+144)
51 178 (124+54) 323 (190+133)
52 115 (78+37) 289 (190+99)
1 93 (60+33) 287 (171+116)
2 82 (46+36) 271 (162+109)
3 25 (15+10) 249 (165+84)
4 14 (8+6) 244 (176+68)
5 2 (0+2) 224 (132+92)
6 release! 212 (129+83)
7 release+1 194 (128+66)
8 release+2 206 (144+62)
9 release+3 174 (105+69)
10 release+4 120 (72+48)
11 release+5 115 (74+41)
12 release+6 93 (47+46)
13 release+7 50 (24+26)
14 release+8 51 (32+19)
15 release+9 39 (32+7)
16 release+10 20 (12+8)
17 release+11 24 (19+5)
18 release+12 2 (2+0)
Graphical overview of bug stats thanks to azhag:

27 November 2014

Pau Garcia i Quiles: Reminder: Desktops DevRoom @ FOSDEM 2015

We are less than 10 days away from the deadline for the Desktops DevRoom at FOSDEM 2015, the largest Free and Open Source event in Europe. Do you think you can fill a room with 200+ people out of 6,000+ geeks? Prove it! Check the Call for Talks for details on how to submit your talk proposal about anything related to the desktop: http://www.elpauer.org/2014/10/fosdem-2015-desktops-devroom-call-for-talks/

25 November 2014

Erich Schubert: Installing Debian with sysvinit

First let me note that I am using systemd, so these things here are untested by me. See e.g. Petter's and Simon's blog entries on the same overall topic.
According to the Debian installer maintainers, the only accepted way to install Debian with sysvinit is to use preseeding. This can either be done at the installer boot prompt by manually typing the magic spell:
preseed/late_command="in-target apt-get install -y sysvinit-core"
or by using a preseeding file (which is a really nice feature I used for installing my Hadoop nodes) to do the same:
d-i preseed/late_command string in-target apt-get install -y sysvinit-core
If you are a sysadmin, using preseeding can save you a lot of typing. Put all your desired configuration into preseeding files, put them on a webserver (best with a short name resolvable by local DNS). Let's assume you have set up the DNS name d-i.example.com, and your DHCP is configured such that example.com is on the DNS search list. You can also add a vendor extension to DHCP to serve a full URL. Manually enabling preseeding then means adding
auto url=d-i
to the installer boot command line (d-i is the hostname I suggested to set up in your DNS before, and the full URL would then be http://d-i.example.com/d-i/jessie/./preseed.cfg. Preseeding is well documented in Appendix B of the installer manual, but nevertheless will require a number of iterations to get everything work as desired for a fully automatic install like I used for my Hadoop nodes.

There might be an easier option.
I have filed a wishlist bug suggesting to use the tasksel mechanism to allow the user to choose sysvinit at installation time. However, it got turned down by the Debian installer maintainers quire rudely in a "No." - essentially this is a "shut the f... up and go away", which is in my opinion an inappropriate to discard a reasonable user wishlist request.
Since I don't intend to use sysvinit anymore, I will not be pursuing this option further. It is, as far as I can tell, still untested. If it works, it might be the least-effort, least-invasive option to allow the installation of sysvinit Jessie (except for above command line magic).
If you have interest in sysvinit, you (because I don't use sysvinit) should now test if this approach works.
  1. Get the patch proposed to add a task-sysvinit package.
  2. Build an installer CD with this tasksel (maybe this documentation is helpful for this step).
  3. Test whether the patch works. Report results to above bug report, so that others interested in sysvinit can find them easily.
  4. Find and fix bugs if it didn't work. Repeat.
  5. Publish the modified ("forked") installer, and get user feedback.
If you are then still up for a fight, you can try to convince the maintainers (or go the nasty way, and ask the CTTE for their opinion, to start another flamewar and make more maintainers give up) that this option should be added to the mainline installer. And hurry up, or you may at best get this into Jessie reloaded, 8.1. - chance are that the release manager will not accept such patches this late anymore. The sysvinit supporters should have investigated this option much, much earlier instead of losing time on the GR.
Again, I won't be doing this job for you. I'm happy with systemd. But patches and proof-of-concept is what makes open source work, not GRs and MikeeUSA's crap videos spammed to the LKML...
(And yes, I am quite annoyed by the way the Debian installer maintainers handled the bug report. This is not how open-source collaboration is supposed to work. I tried to file a proper wishlist bug reporting, suggesting a solution that I could not find discussed anywhere before and got back just this "No. Shut up." answer. I'm not sure if I will be reporting a bug in debian-installer ever again, if this is the way they handle bug reports ...)
I do care about our users, though. If you look at popcon "vote" results, we have 4179 votes for sysvinit-core and 16918 votes for systemd-sysv (graph) indicating that of those already testing jessie and beyond - neglecting 65 upstart votes, and assuming that there is no bias to not-upgrade if you prefer sysvinit - about 20% appear to prefer sysvinit (in fact, they may even have manually switched back to sysvinit after being upgraded to systemd unintentionally?). These are users that we should listen to, and that we should consider adding an installer option for, too.

Gunnar Wolf: 10 PRINT CHR$(205.5+RND(1)); : GOTO 10 (also known as #10print )

The line of BASIC code that appears as the subject for this post is the title for a book I just finished reading And enjoyed thoroughly. The book is available online for download under a CC-BY-NC-SA 3.0 License, so you can take a good look at it before (or instead of) buying it. Although it's among the books I will enjoy having on my shelf; the printing is of a very enjoyable good quality. And what is this book about? Well, of course, it analizes that very simple line of code, as it ran on the Commodore 64 thirty years ago. And the analysis is made from every possible angle: What do mazes mean in culture? What have they meant in cultures through history? What about regularity in art (mainly 20th century art)? How would this code look (or how it would be adapted) on contemporary non-C64 computers? And in other languages more popular today? What does randomness mean? And what does random() mean? What is BASIC, and how it came to the C64? What is the C64, and where did it come from? And several other beautiful chapters. The book was collaboratively written by ten different authors, in a Wiki-like fashion. And... Well, what else is there to say? I enjoyed so much reading through long chapters of my childhood, of what attracted me to computers, of my cultural traits and values... I really hope that, in due time, I can be a part of such a beautiful project!

24 November 2014

John Goerzen: My boys love 1986 computing

Yesterday, Jacob (age 8) asked to help me put together a 30-year-old computer from parts in my basement. Meanwhile, Oliver (age 5) asked Laura to help him learn cursive. Somehow, this doesn t seem odd for a Saturday at our place. 2014-11-22 18.58.36 Let me tell you how this came about. I ve had a project going on for a while now to load data from old floppies. It s been fun, and had a surprise twist the other day: my parents gave me an old TRS-80 Color Computer II (aka CoCo 2 ). It was, in fact, my first computer, one they got for me when I was in Kindergarten. It is nearly 30 years old. I have been musing lately about the great disservice Apple did the world by making computers easy to learn namely the fact that few people ever bother to learn about them. Who bothers to learn about them when, on the iPhone for instance, the case is sealed shut, the lifespan is 1 or 2 years for many purchasers, and the platform is closed in lots of ways? I had forgotten how finicky computers used to be. But after some days struggling with IDE incompatibilities, booting issues, etc., when I actually managed to get data off a machine that had last booted in 1999, I had quite the sense of accomplishment, which I rarely have lately. I did something that was hard to do in a world where most of the interfaces don t work with equipment that old (even if nominally they are supposed to.) The CoCo is one of those computers normally used with a floppy drive or cassette recorder to store programs. You type DIR, and you feel the clack of the drive heads through the desk. You type CLOAD and you hear the relay click closed to turn on the tape motor. You wiggle cables around until they make contact just right. You power-cycle for the times when the reset button doesn t quite do the job. The details of how it works aren t abstracted away by innumerable layers of controllers, interfaces, operating system modules, etc. It s all right there, literally vibrating your desk. So I thought this could be a great opportunity for Jacob to learn a few more computing concepts, such as the difference between mass storage and RAM, plus a great way to encourage him to practice critical thinking. So we trekked down to the basement and came up with handfulls of parts. We brought up the computer, some joysticks, all sorts of tangled cables. We needed adapters, an old TV. Jacob helped me hook everything up, and then the moment of truth: success! A green BASIC screen! I added more parts, but struck out when I tried to connect the floppy drive. The thing just wouldn t start up right whenever the floppy controller cartridge was installed. I cleaned the cartridge. I took it apart, scrubbed the contacts, even did a re-seat of the chips. No dice. So I fired up my CoCo emulator (xroar), and virtually saved some programs to cassette (a .wav file). I then burned those .wav files to an audio CD, brought up an old CD player from the basement, connected the cassette in plug to the CD player s headphone jack, and presto instant programs. (Well, almost. It takes a couple of minutes to load a program from audio codes.) The picture above is Oliver cackling at one of the very simplest BASIC programs there is: number find. The computer picks a random number between 1 and 2000, and asks the user to guess it, giving a too low or too high clue with each incorrect guess. Oliver delighted in giving invalid input (way too high numbers, or things that weren t numbers at all) and cackled at the sarcastic error messages built into the program. During Jacob s turn, he got very serious about it, and is probably going to be learning about how to calculate halfway points before too long. But imagine my pride when this morning, Jacob found the new CD I had made last night (correcting a couple recordings), found my one-line instruction on just part of how to load a program, and correctly figured out by himself all the steps to do in order (type CLOAD on the CoCo, advance the CD to the proper track, press play on the player, wait for it to load on the CoCo, then type RUN). I ordered a replacement floppy controller off eBay tonight, and paid $5 for a coax adapter that should fix some video quality issues. I rescued some 5.25 floppies from my trash can from another project, so they should have plenty of tools for exploration. It is so much easier for them to learn how a disk drive works, and even what the heck a track is, when you can look at a floppy drive with the cover off and see the heads move. There are other things we can do with more modern equipment Jacob has shown a lot of interest in Arduino projects but I have so far drawn a blank on ways to really let kids discover how a modern PC (let alone a modern phone or tablet) works. Update Nov. 24: Every so often, the world surprises me by deciding to, well, read one of my random blog posts. For the benefit of those of you that don t already know my boys, you might want to know that among their common play activites are turning trees into pretend trains, typing at a manual typewriter, reading, writing their own books, using a cassette recorder, building a PC and learning to use bash or xmonad, making long paper tapes with an adding machine, playing records on a record player, building electric gizmos, and even making mud balls. I am often asked about the role of the computer in the lives, given that my hobby and profession involves computers. The answer: less than that of most of their peers. I look for opportunities for them to learn by doing, discovering, playing, or imagining. I make no presumption that they will develop the passion for computers that I did. What I want is for them to have the curiosity and drive to learn everything there is to know about whatever they do develop a passion for, so they will be great at it.

23 November 2014

Dimitri John Ledkov: Analyzing public OpenPGP keys

OpenPGP Message Format (RFC 4880) well defines key structure and wire formats (openpgp packets). Thus when I looked for public key network (SKS) server setup, I quickly found pointers to dump files in said format for bootstrapping a key server.

I did not feel like experimenting with Python and instead opted for Go and found http://code.google.com/p/go.crypto/openpgp/packet library that has comprehensive support for parsing openpgp low level structures. I've downloaded the SKS dump, verified it's MD5SUM hashes (lolz), and went ahead to process them in Go.

With help from http://github.com/lib/pq and database/sql, I've written a small program to churn through all the dump files, filter for primary RSA keys (not subkeys) and inject them into a database table. The things that I have chosen to inject are fingerprint, N, E. N & E are the modulus of the RSA key pair and the public exponent. Together they form a public part of an RSA keypair. So far, nothing fancy.

Next I've run an SQL query to see how unique things are... and found 92 unique N & E pairs that have from two and up to fifteen duplicates. In total it is 231 unique fingerprints, which use key material with a known duplicate in the public key network. That didn't sound good. And also odd - given that over 940 000 other RSA keys managed to get unique enough entropy to pull out a unique key out of the keyspace haystack (which is humongously huge by the way).

Having the list of the keys, I've fetched them and they do not look like regular keys - their UIDs do not have names & emails, instead they look like something from the monkeysphere. The keys look like they are originally used for TLS and/or SSH authentication, but were converted into OpenPGP format and uploaded into the public key server. This reminded me of the Debian's SSL key generation vulnerability CVE-2008-0166. So these keys might have been generated with bad entropy due to affected tools by that CVE and later converted to OpenPGP.

Looking at the openssl-blacklist package, it should be relatively easy for me to generate all possible RSA key-pairs and I believe all other material that is hashed to generate the fingerprint are also available (RFC 4880#12.2). Thus it should be reasonably possible to generate matching private keys, generate revocation certificates and publish the revocation certificate with pointers to CVE-2008-0166. (Or email it to the people who have signed given monkeysphered keys). When I have a minute I will work on generating openpgp-blacklist type of scripts to address this.

If anyone is interested in the Go source code I've written to process openpgp packets, please drop me a line and I'll publish it on github or something.

Matthew Palmer: You stay classy, Uber

You may have heard that Uber has been under a bit of fire lately for its desires to hire private investigators to dig up dirt on journalists who are critical of Uber. From using users ride data for party entertainment, putting the assistance dogs of blind passengers in the trunk, adding a surcharge to reduce the number of dodgy drivers, or even booking rides with competitors and then cancelling, or using the ride to try and convince the driver to change teams, it s pretty clear that Uber is a pretty good example of how companies are inherently sociopathic. However, most of those examples are internal stupidities that happened to be made public. It s a very rare company that doesn t do all sorts of shady things, on the assumption that the world will never find out about them. Uber goes quite a bit further, though, and is so out-of-touch with the world that it blogs about analysing people s sexual activity for amusement. You ll note that if you follow the above link, it sends you to the Wayback Machine, and not Uber s own site. That s because the original page has recently turned into a 404. Why? Probably because someone at Uber realised that bragging about how Uber employees can amuse themselves by perving on your one night stands might not be a great idea. That still leaves the question open of what sort of a corporate culture makes anyone ever think that inspecting user data for amusement would be a good thing, let alone publicising it? It s horrific. Thankfully, despite Uber s fairly transparent attempt at whitewashing ( clearwashing ?), the good ol Wayback Machine helps us to remember what really went on. It would be amusing if Uber tried to pressure the Internet Archive to remove their copies of this blog post (don t bother, Uber; I ve got a Save As button and I m not afraid to use it). In any event, I ve never used Uber (not that I ve got one-night stands to analyse, anyway), and I ll certainly not be patronising them in the future. If you re not keen on companies amusing themselves with your private data, I suggest you might consider doing the same.

21 November 2014

Gunnar Wolf: Status of the Debian OpenPGP keyring November update

Almost two months ago I posted our keyring status graphs, showing the progress of the transition to >=2048-bit keys for the different active Debian keyrings. So, here are the new figures. First, the Non-uploading keyring: We were already 100% transitioned. You will only notice a numerical increase: That little bump at the right is our dear friend T ssia finally joining as a Debian Developer. Welcome! \o/ As for the Maintainers keyring: We can see a sharp increase in 4096-bit keys. Four 1024-bit DM keys were migrated to 4096R, but we did have eight new DMs coming in To them, also, welcome \o/. Sadly, we had to remove a 1024-bit key, as Peter Miller sadly passed away. So, in a 234-key universe, 12 new 4096R keys is a large bump! Finally, our current-greatest worry If for nothing else, for the size of the beast: The active Debian Developers keyring. We currently have 983 keys in this keyring, so it takes considerably more effort to change it. But we have managed to push it noticeably. This last upload saw a great deal of movement. We received only one new DD (but hey welcome nonetheless! \o/ ). 13 DD keys were retired; as one of the maintainers of the keyring, of course this makes me sad but then again, in most cases it's rather an acknowledgement of fact: Those keys' holders often state they had long not been really involved in the project, and the decision to retire was in fact timely. But the greatest bulk of movement was the key replacements: A massive 62 1024D keys were replaced with stronger ones. And, yes, the graph changed quite abruptly: We still have a bit over one month to go for our cutoff line, where we will retire all 1024D keys. It is important to say we will not retire the affected accounts, mark them as MIA, nor anything like that. If you are a DD and only have a 1024D key, you will still be a DD, but you will be technically unable to do work directly. You can still upload your packages or send announcements to regulated mailing lists via sponsor requests (although you will be unable to vote). Speaking of votes: We have often said that we believe the bulk of the short keys belong to people not really active in the project anymore. Not all of them, sure, but a big proportion. We just had a big, controversial GR vote with one of the highest voter turnouts in Debian's history. I checked the GR's tally sheet, and the results are interesting: Please excuse my ugly bash, but I'm posting this so you can play with similar runs on different votes and points in time using the public keyring Git repository:
  1. $ git checkout 2014.10.10
  2. $ for KEY in $( for i in $( grep '^V:' tally.txt
  3. awk ' print "<" $3 ">" ' )
  4. do
  5. grep $i keyids cut -f 1 -d ' '
  6. done )
  7. do
  8. if [ -f debian-keyring-gpg/$KEY -o -f debian-nonupload-gpg/$KEY ]
  9. then
  10. gpg --keyring /dev/null --keyring debian-keyring-gpg/$KEY \
  11. --keyring debian-nonupload-gpg/$KEY --with-colons \
  12. --list-key $KEY 2>/dev/null \
  13. head -2 tail -1 cut -f 3 -d :
  14. fi
  15. done sort uniq -c
  16. 95 1024
  17. 13 2048
  18. 1 3072
  19. 371 4096
  20. 2 8192
So, as of mid-October: 387 out of the 482 votes (80.3%) were cast by developers with >=2048-bit keys, and 95 (19.7%) were cast by short keys. If we were to run the same vote with the new active keyring, 417 votes would have been cast with >=2048-bit keys (87.2%), and 61 with short keys (12.8%). We would have four less votes, as they retired:
  1. 61 1024
  2. 14 2048
  3. 2 3072
  4. 399 4096
  5. 2 8192
So, lets hear it for November/December. How much can we push down that pesky yellow line? Disclaimer: Any inaccuracy due to bugs in my code is completely my fault!

Daniel Pocock: PostBooks 4.7 packages available, xTupleCon 2014 award

I recently updated the PostBooks packages in Debian and Ubuntu to version 4.7. This is the version that was released in Ubuntu 14.10 (Utopic Unicorn) and is part of the upcoming Debian 8 (jessie) release. Better prospects for Fedora and RHEL/CentOS/EPEL packages As well as getting the packages ready, I've been in contact with xTuple helping them generalize their build system to make packaging easier. This has eliminated the need to patch the makefiles during the build. As well as making it easier to support the Debian/Ubuntu packages, this should make it far easier for somebody to create a spec file for RPM packaging too. Debian wins a prize While visiting xTupleCon 2014 in Norfolk, I was delighted to receive the Community Member of the Year award which I happily accepted not just for my own efforts but for the Debian Project as a whole. Steve Hackbarth, Director of Product Development at xTuple, myself and the impressive Community Member of the Year trophy This is a great example of the productive relationships that exist between Debian, upstream developers and the wider free software community and it is great to be part of a team that can synthesize the work from so many other developers into ready-to-run solutions on a 100% free software platform. Receiving this award really made me think about all the effort that has gone into making it possible to apt-get install postbooks and all the people who have collectively done far more work than myself to make this possible: Here is a screenshot of the xTuple web / JSCommunicator integration, it was one of the highlights of xTupleCon: and gives a preview of the wide range of commercial opportunities that WebRTC is creating for software vendors to displace traditional telecommunications providers. xTupleCon also gave me a great opportunity to see new features (like the xTuple / Drupal web shop integration) and hear about the success of consultants and their clients deploying xTuple/PostBooks in various scenarios. The product is extremely strong in meeting the needs of manufacturing and distribution and has gained a lot of traction in these industries in the US. Many of these features are equally applicable in other markets with a strong manufacturing industry such as Germany or the UK. However, it is also flexible enough to simply disable many of the specialized features and use it as a general purpose accounting solution for consulting and services businesses. This makes it a good option for many IT freelancers and support providers looking for a way to keep their business accounts in a genuinely open source solution with a strong SQL backend and a native Linux desktop interface.

Julien Danjou: Distributed group management and locking in Python with tooz

With OpenStack embracing the Tooz library more and more over the past year, I think it's a good start to write a bit about it. A bit of history A little more than year ago, with my colleague Yassine Lamgarchal and others at eNovance, we investigated on how to solve a problem often encountered inside OpenStack: synchronization of multiple distributed workers. And while many people in our ecosystem continue to drive development by adding new bells and whistles, we made a point of solving new problems with a generic solution able to address the technical debt at the same time. Yassine wrote the first ideas of what should be the group membership service that was needed for OpenStack, identifying several projects that could make use of this. I've presented this concept during the OpenStack Summit in Hong-Kong during an Oslo session. It turned out that the idea was well-received, and the week following the summit we started the tooz project on StackForge. Goals Tooz is a Python library that provides a coordination API. Its primary goal is to handle groups and membership of these groups in distributed systems. Tooz also provides another useful feature which is distributed locking. This allows distributed nodes to acquire and release locks in order to synchronize themselves (for example to access a shared resource). The architecture If you are familiar with distributed systems, you might be thinking that there are a lot of solutions already available to solve these issues: ZooKeeper, the Raft consensus algorithm or even Redis for example. You'll be thrilled to learn that Tooz is not the result of the NIH syndrome, but is an abstraction layer on top of all these solutions. It uses drivers to provide the real functionalities behind, and does not try to do anything fancy. All the drivers do not have the same amount of functionality of robustness, but depending on your environment, any available driver might be suffice. Like most of OpenStack, we let the deployers/operators/developers chose whichever backend they want to use, informing them of the potential trade-offs they will make. So far, Tooz provides drivers based on: All drivers are distributed across processes. Some can be distributed across the network (ZooKeeper, memcached, redis ) and some are only available on the same host (IPC). Also note that the Tooz API is completely asynchronous, allowing it to be more efficient, and potentially included in an event loop. Features Group membership Tooz provides an API to manage group membership. The basic operations provided are: the creation of a group, the ability to join it, leave it and list its members. It's also possible to be notified as soon as a member joins or leaves a group. Leader election Each group can have a leader elected. Each member can decide if it wants to run for the election. If the leader disappears, another one is elected from the list of current candidates. It's possible to be notified of the election result and to retrieve the leader of a group at any moment. Distributed locking When trying to synchronize several workers in a distributed environment, you may need a way to lock access to some resources. That's what a distributed lock can help you with. Adoption in OpenStack Ceilometer is the first project in OpenStack to use Tooz. It has replaced part of the old alarm distribution system, where RPC was used to detect active alarm evaluator workers. The group membership feature of Tooz was leveraged by Ceilometer to coordinate between alarm evaluator workers. Another new feature part of the Juno release of Ceilometer is the distribution of polling tasks of the central agent among multiple workers. There's again a group membership issue to know which nodes are online and available to receive polling tasks, so Tooz is also being used here. The Oslo team has accepted the adoption of Tooz during this release cycle. That means that it will be maintained by more developers, and will be part of the OpenStack release process. This opens the door to push Tooz further in OpenStack. Our next candidate would be write a service group driver for Nova. The complete documentation for Tooz is available online and has examples for the various features described here, go read it if you're curious and adventurous!

20 November 2014

Steve Kemp: An experiment in (re)building Debian

I've rebuilt many Debian packages over the years, largely to fix bugs which affected me, or to add features which didn't make the cut in various releases. For example I made a package of fabric available for Wheezy, since it wasn't in the release. (Happily in that case a wheezy-backport became available. Similar cases involved repackaging gtk-gnutella when the protocol changed and the official package in the lenny release no longer worked.) I generally release a lot of my own software as Debian packages, although I'll admit I've started switching to publishing Perl-based projects on CPAN instead - from which they can be debianized via dh-make-perl. One thing I've not done for many years is a mass-rebuild of Debian packages. I did that once upon a time when I was trying to push for the stack-smashing-protection inclusion all the way back in 2006. Having had a few interesting emails this past week I decided to do the job for real. I picked a random server of mine, rsync.io, which stores backups, and decided to rebuild it using "my own" packages. The host has about 300 packages installed upon it:
root@rsync ~ # dpkg --list   grep ^ii   wc -l
294
I got the source to every package, patched the changelog to bump the version, and rebuild every package from source. That took about three hours. Every package has a "skx1" suffix now, and all the build-dependencies were also determined by magic and rebuilt:
root@rsync ~ # dpkg --list   grep ^ii   awk '  print $2 " " $3 '  head -n 4
acpi 1.6-1skx1
acpi-support-base 0.140-5+deb7u3skx1
acpid 1:2.0.16-1+deb7u1skx1
adduser 3.113+nmu3skx1
The process was pretty quick once I started getting more and more of the packages built. The only shortcut was not explicitly updating the dependencies to rely upon my updages. For example bash has a Debian control file that contains:
Depends: base-files (>= 2.1.12), debianutils (>= 2.15)
That should have been updated to say:
Depends: base-files (>= 2.1.12skx1), debianutils (>= 2.15skx1)
However I didn't do that, because I suspect if I did want to do this decently, and I wanted to share the source-trees, and the generated packages, the way to go would not be messing about with Debian versions instead I'd create a new Debian release "alpha-apple", "beta-bananna", "crunchy-carrot", "dying-dragonfruit", "easy-elderberry", or similar. In conclusion: Importing Debian packages into git, much like Ubuntu did with bzr, is a fun project, and it doesn't take much to mass-rebuild if you're not making huge changes. Whether it is worth doing is an entirely different question of course.

Daniel Pocock: Is Amnesty giving spy victims a false sense of security?

Amnesty International is getting a lot of attention with the launch of a new tool to detect government and corporate spying on your computer. I thought I would try it myself. I went to a computer running Microsoft Windows, an operating system that does not publish its source code for public scrutiny. I used the Chrome browser, users often express concern about Chrome sending data back to the vendor about the web sites the users look for. Without even installing the app, I would expect the Amnesty web site to recognise that I was accessing the site from a combination of proprietary software. Instead, I found a different type of warning. Beware of Amnesty? Instead, the only warning I received was from Amnesty's own cookies: Even before I install the app to find out if the government is monitoring me, Amnesty is keen to monitor my behaviour themselves. While cookies are used widely, their presence on a site like Amnesty's only further desensitizes Internet users to the downside risks of tracking technologies. By using cookies, Amnesty is effectivley saying a little bit of tracking is justified for the greater good. Doesn't that sound eerily like the justification we often hear from governments too? Is Amnesty part of the solution or part of the problem? Amnesty is a well known and widely respected name when human rights are mentioned. However, their advice that you can install an app onto a Windows computer or iPhone to detect spyware is like telling people that putting a seatbelt on a motorbike will eliminate the risk of death. It would be much more credible for Amnesty to tell people to start by avoiding cloud services altogether, browse the web with Tor and only use operating systems and software that come with fully published source code under a free license. Only when 100% of the software on your device is genuinely free and open source can independent experts exercise the freedom to study the code and detect and remove backdoors, spyware and security bugs. It reminds me of the advice Kim Kardashian gave after the Fappening, telling people they can continue trusting companies like Facebook and Apple with their private data just as long as they check the privacy settings (reality check: privacy settings in cloud services are about as effective as a band-aid on a broken leg). Write to Amnesty Amnesty became famous for their letter writing campaigns. Maybe now is the time for people to write to Amnesty themselves, thank them for their efforts and encourage them to take more comprehensive action. Feel free to cut and paste some of the following potential ideas into an email to Amnesty:
I understand you may not be able to respond to every email personally but I would like to ask you to make a statement about these matters on your public web site or blog. I understand it is Amnesty's core objective to end grave abuses of human rights. Electronic surveillence, due to its scale and pervasiveness, has become a grave abuse in itself and in a disturbing number of jurisdictions it is an enabler for other types of grave violations of human rights. I'm concerned that your new app Detekt gives people a false sense of security and that your campaign needs to be more comprehensive to truly help people and humanity in the long term. If Amnesty is serious about solving the problems of electronic surveillance by government, corporations and other bad actors, please consider some of the following:
  • Instead of displaying a cookie warning on Amnesty.org, display a warning to users who access the site from a computer running closed-source software and give them a link to download a free and open source web browser like Firefox.
  • Redirect all visitors to your web site to use the HTTPS encrypted version of the site.
  • Using free software such as the GNU/Linux operating system (using one of the Debian, Fedora or Ubuntu systems is one of the more common ways to achieve this) and LibreOffice for all Amnesty's own operations, making a public statement about your use of free software and mentioning this in the closing paragraph of all press releases relating to surveillance topics.
  • Encouraging Amnesty donors, members and supporters to choose similar software especially when engaging in any political activities.
  • Make a public statement that Amnesty will not use cloud services such as SalesForce or Facebook to store, manage or interact with data relating to members, donors or other supporters.
  • Encouraging the public to move away from centralized cloud services such as those provided by their smartphone or social networks and use de-centralized or federated services such as XMPP chat.
Given the immense threat posed by electronic surveillance, I'd also like to call on Amnesty to allocate at least 10% of annual revenue towards software projects releasing free and open source software that offers the public an alternative to the centralized cloud.

While publicity for electronic privacy is great, I hope Amnesty can go a step further and help people use trustworthy software from the ground up.

Matthew Palmer: Multi-level prefix delegation is not a myth! I've seen it!

Unless you ve been living under a firewalled rock, you know that IPv6 is coming. There s also a good chance that you ve heard that IPv6 doesn t have NAT. Or, if you pay close attention to the minutiae of IPv6 development, you ve heard that IPv6 does have NAT, but you don t have to (and shouldn t) use it. So let s say we ll skip NAT for IPv6. Fair enough. However, let s say you have this use case:
  1. A bunch of containers that need Internet access
  2. That are running in a VM
  3. On your laptop
  4. Behind your home router!
For IPv4, you d just layer on the NAT, right? While SIP and IPsec might have kittens trying to work through three layers of NAT, for most things it ll Just Work. In the Grand Future of IPv6, without NAT, how the hell do you make that happen? The answer is Prefix Delegation , which allows routers to delegate management of a chunk of address space to downstream routers, and allow those downstream routers to, in turn, delegate pieces of that chunk to downstream routers. In the case of our not-so-hypothetical containers-in-VM-on-laptop-at-home scenario, it would look like this:
  1. My border router (a DNS-323 running Debian) asks my ISP for a delegated prefix, using DHCPv6. The ISP delegates a /561. One /64 out of that is allocated to the network directly attached to the internal interface, and the rest goes into the pool , as /60 blocks (so I ve got 15 of them to delegate, if required).
  2. My laptop gets an address on the LAN between itself and the DNS-323 via stateless auto-addressing ( SLAAC ). It also uses DHCPv6 to request one of the /60 blocks from the DNS-323. The laptop puts one /64 from that block as the address space for the virtual LAN (actually a Linux bridge) that connects the laptop to all my VMs, and puts the other 15 /64 blocks into a pool for delegation.
  3. The VM that will be running the set of containers under test gets an address on the all VMs virtual LAN via SLAAC, and then requests a delegated /64 to use for the all containers virtual LAN (another bridge, this one running on the VM itself) that the containers will each connect to themselves.
Now, almost all of this Just Works. The current releases of ISC DHCP support prefix delegation just fine, and a bit of shell script plumbing between the client and server seals the deal the client needs to rewrite the server s config file to tell it the netblock from which it can delegate. Except for one teensy, tiny problem routing. When the DHCP server delegates a netblock to a particular machine, the routing table needs to get updated so that packets going to that netblock actually get sent to the machine the netblock was delegated to. Without that, traffic destined for the containers (or the VM) won t actually make it to its destination, and a one-way Internet connection isn t a whole lot of use. I cannot understand why this problem hasn t been tripped over before. It s absolutely fundamental to the correct operation of the delegation system. Some people advocate running a dynamic routing protocol, but that s a sledgehammer to crack a nut if ever I saw one. Actually, I know this problem has been tripped over before, by OpenWrt. Their solution, however, was to use a PHP script to scan logfiles and add routes. Suffice it to say, that wasn t an option I was keen on exploring. Instead, I decided to patch ISC DHCP so that the server can run an external script to add the necessary routes, and perhaps modify firewall rules and also to reverse the process when the delegation is released (or expired). If anyone else wants to play around with it, I ve put it up on Github. I don t make any promises that it s the right way to do it, necessarily, but it works, and the script I ve added in contrib/prefix-delegation-routing.rb shows how it can be used to good effect. By the way, if anyone knows how pull requests work over at ISC, drop me a line. From the look of their website, they don t appear to accept (or at least encourage) external contributions. So, that s one small patch for DHCP, one giant leap for my home network.
  1. The standard recommendation is for ISPs to delegate each end-user customer a /48 (giving 65,536 /64 networks); my ISP is being a little conservative in only giving me 256 /64s. It works fine for my purposes, but if you re an ISP getting set for deploying IPv6, make life easy on your customers and give them a /48.

Russ Allbery: Interpreting the Debian init system GR

I originally posted this in a thread on debian-private, but on further reflection it seems appropriate for a broader audience. So I'm posting it here, as well as on debian-project. There is quite a lot of discussion in various places about what the recent GR result means. Some are concluding that systemd won in some way that implies Debian is not going to support other init systems, or at least that support for other init systems is in immediate danger. A lot of that analysis concludes that the pro-systemd "side" in Debian won some sort of conclusive victory. I have a different perspective. I think we just had a GR in which the Debian developer community said that we, as a community, would like to work through all of the issues around init systems together, as a community, rather than having any one side of the argument win unambiguously and impose its views on those who disagree. There were options on the ballot that clearly required loose coupling and that clearly required tight coupling. The top two options did neither of those things. The second-highest option said, effectively, that we should feel free to exercise our technical judgement for our own packages, but should do so with an eye to enabling people to make different choices, and should merge their changes and contributions where possible. The highest option said that we don't even want to say that, and would instead prefer to work this whole thing out through discussion, respect, consensus, and mutual support, without giving *anyone* a clear mandate or project-wide blessing for their approach. In other words, the way I choose to look at this GR is that the project as a whole just voted to take away the sticks that we were using to beat each other with. In a way, we just chose the *hardest* option. We didn't make a simplifying technical decision that provides clear guidance to everyone. Instead, we made a complicating social decision that says that, sorry, there's no short cut to avoid having to talk to each other, respect each other's views, and try to reach workable collaborative compromises. Even though it's really hard, even though everyone is raw and upset, that's what the project as a whole is asking us to do. Are we up to the challenge?

Next.

Previous.