Search Results: "cas"

4 December 2014

Olivier Berger: Shell script to connecting to a Shibboleth protected web app with curl

Here s a shell script I ve created (reusing one meant for CAS protected resources), which will allow to connect to a Web application protected by the Shibboleth SSO mechanism.
It uses cURL to navigate through the various jumps required by the protocol, perform the necessary posts, etc.
I haven t read the Shibboleth specs, so it may not be the best way, and may not work in all cases, but that was enough for my case, at least. Feel free to improve it on Github Gists. After the connection is succesful, one may reuse the .cookieJar file to perform further cURL connections, or even some automated content mirroring with httrack, for instance (see a previous experiment of mine with httrack for Moodle).

3 December 2014

Michael Stapelberg: Announcing the new Debian Code Search Instant

For the last few months, I have been working on a new version of Debian Code Search, and today it s going live! I call it Debian Code Search Instant, for multiple reasons, see below. A lot faster The new Debian Code Search is still hosted by Rackspace (thank you!), but our new architecture finally allows us to use their Performance Cloud Servers hardware generation. The old code search architecture was split across 5 different servers, however the search itself was only performed on a single machine with a network attached block volume backed by SSD. The new Code Search spreads out both the trigram index and the source code onto 6 different servers, and thanks to the new hardware generation, we have a locally connected SSD drive in each of them. So, we now get more than 6 times the IOPS we ve had before, at much lower latency :). Grouping by Debian source package The first feature request we ever got for Code Search was to group results by Debian source package. I ve tried tackling that before, but it s a pretty hard problem. With the new architecture and capacity, this finally becomes feasible. After your query completes, there is a checkbox called Group search results by Debian source package . Enable it, and you ll get neatly grouped results. For each package, there is a link at the bottom to refine the search to only consider that package. In case you are more interested in the full list of packages, for example because you are doing large-scale changes across Debian, that s also possible: Click on the symbol right next to the Filter by package list, and a curl command will be revealed with which you can download the full list of packages. Expensive queries (with lots of results) now possible Previously, we had a 60 second timeout during which queries must complete. This timeout has been completely lifted, and long-running queries with tons and tons of results are now possible. We kindly ask you to not abuse this feature it s not very exciting to play with complexity explosion in regular expression engines by preventing others from doing their work. If you re interested in that sort of analysis, go grab the source code and play with it on your own machine :). Instant results While your query is running, you will almost immediately see the top 10 results, even though the search may still take a while. This is useful to figure out if your regular expression matches what you thought it would, before having to wait until your 5 minute query is finished. Also, the new progress bar tells you how far the system is with your search. Instant indexing Previously, Code Search was deployed to a new set of machines from scratch every week in order to update the index. This was necessary because the performance would severely degrade while building the index, so we were temporarily running with twice the amount of machines until the new version was up. In the new architecture, we store an index for each source package and then merge these into one big index shard. This currently takes about 4 minutes with the code I wrote, but I m sure this can be made even faster if necessary. So, whenever new packages are uploaded to the Debian archive, we can just index the new version and trigger a merge. We get notifications about new package uploads from FedMsg. Packages that are not seen on FedMsg for some reason are backfilled every hour. The time between uploading a package and being able to find it in Debian Code Search therefore now ranges from a couple of minutes to about an hour, instead of about a week! New, beautiful User Interface Since we needed to rewrite the User Interface anyway thanks to the new architecture, we also spent some effort on making it modern, polished and beautiful. Smooth CSS animations make you aware of what s going on, and the search results look cleaner than ever. Conclusion At least to me, the new Debian Code Search seems like a significant improvement over the old one. I hope you enjoy the new features into which I put a lot of work. In case you have any feedback, I m happy to hear it (see the contact link at the bottom of every Code Search site).

Diego Escalante Urrelo: Link pack #01

Following the lead of my dear friend Daniel and his fantastic and addictive Summing up series, here s a link pack of recent stuff I read around the web. Link pack is definitely a terrible name, but I m working on it.
How to Silence Negative Thinking
On how to avoid the pitfall of being a Negatron and not an Optimist Prime. You might be your own worst enemy and you might not even know it:
Psychologists use the term automatic negative thoughts to describe the ideas that pop into our heads uninvited, like burglars, and leave behind a mess of uncomfortable emotions. In the 1960s, one of the founders of cognitive therapy, Aaron Beck, concluded that ANTs sabotage our best self, and lead to a vicious circle of misery: creating a general mindset that is variously unhappy or anxious or angry (take your pick) and which is (therefore) all the more likely to generate new ANTs. We get stuck in the same old neural pathways, having the same negative thoughts again and again.
Meet Harlem s Official Street Photographer
A man goes around Harlem with his camera, looking to give instead of taking. Makes you think about your approach to people and photography, things can be simpler. Kinda like Humans of New York, but in Harlem. And grittier, and on film but as touching, or more:
I tell people that my camera is a healing mechanism, Allah says. Let me photograph it and take it away from you.
What Happens When We Let Industry and Government Collect All the Data They Want
Why having nothing to hide is not about the now, but about the later. It s not that someone is going to judge for pushing every detail of your life to Twitter and Instagram, it s just that something you do might be illegal a few years later:
There was a time when it was essentially illegal to be gay. There was a time when it was legal to own people and illegal for them to run away. Sometimes, society gets it wrong. And it s not just nameless bureaucrats; it s men like Thomas Jefferson. When that happens, strong privacy protections including collection controls that let people pick who gets their data, and when allow the persecuted and unpopular to survive.
The Sex-Abuse Scandal Plaguing USA Swimming
Abusive coaches and a bullying culture in sports training are the perfect storm for damaging children. And it s amazing the extent to which a corporation or institution is willing to look the other way, as long as they save face. Very long piece, but intriguing to read. What Cities Would Look Like if Lit Only by the Stars
Thierry Cohen goes around the world and builds beautiful and realistic composite images of how would big cities look like if lit only by stars. The original page has some more cities: Villes teintes (Darkened Cities). On Muppets & Merchandise: How Jim Henson Turned His Art into a Business
Lessons from how Jim Henson managed to juggle both art and business without selling out for the wrong reasons. Really interesting, and reminds you to put Henson in perspective as a very smart man who managed to convince everyone to give him money for playing with muppets. The linked video on How the Muppet Show is Made is also cool. Made me curious enough to get the book. Barbie, Remixed: I (really!) can be a computer engineer
Mattel launched the most misguided book about empowering Barbie to be anything but a computer engineer in a book about being a computer engineer. The internet did not disappoint and fixed the problem within hours. There s now even an app for that (includes user submitted pages).

2 December 2014

Sven Hoexter: Out of the comfort zone: Backporting net-snmp from CentOS 7 to CentOS 5 and 6

Lately I faced the requirement to create a backport of net-snmp 5.7.2, which is part of CentOS 7, to a largely CentOS 5 and 6 based infrastructure. Due to some prior adventures in the rpm world I already knew about Mock (the Fedora equivalent to pbuilder). The easy part was the backporting to CentOS 6. Here the major objectiv is ripping out the newly added systemd support. That boils down to just ripping out a patch, removing a configure flag, some %post, %preun, %postun script snippets and the corresponding %install magic from the spec file. Builds fine but leaves us with the issue that the init script ships in a package net-snmp-sysvinit but the whole chkconfig registration magic is gone. I put that aside for a moment and moved on to CentOS 5. It did not even start to build.
  1. There is no tcp_wrappers-devel in CentOS 5.
  2. The perl libraries are in the main perl package and not in perl-devel.
  3. lm_sensors-devel is not at version 3.
I lowered the versioned build-dep on lm_sensors, disabled via a spec file toggle the tcp-wrapper support and changed the perl build-dep. This time it failed in the configure script due to DTLS support issues with the ancient openssl in CentOS 5. Decided to drop that one as well from the list of configure flags since we do not use it. Tried to build it again, now it failed somewhere down the road post compilation when assembling the package. Not cool. At that time we pushed a few other ideas back and forth. We found an isolated patch for net-snmp 5.6 that could fix our most pressing issue and someone had a look if that could be backported to CentOS 5 easily. Nope, internal structure of net-snmp changed a lot in between. So back to backporting. After staring long enough at the build log I could understand that the variable % RPM_BUILD_ROOT is empty in the CentOS 5 world. And indeed the BuildRoot stanza is also gone from the spec file. So I reintroduced that one and later even found documentation in the Fedora Wiki about this issue. If you know it it's easy to work around it, so I just defined it by hand:
BuildRoot: % _tmppath /% name -% version -% release -root
%global RPM_BUILD_ROOT % buildroot 
About ten minutes later I had a set of rpm files for CentOS 5. Tried to install it, failed due to a depedency on libmysqlclient15 which is part of the mysql package in CentOS 5. Yes, eight years back RedHat did not split off the lib packages and that would mean for us to install a mysql database on every system. So no, we won't do it and we just removed the mysql support. Left with only the init script setup missing I tried to be clever and failed miserable. Be warned, this would've also been a bad idea in the Debian world. Since we have the init script in a seperate binary package I thought I could just use something like
Requires: net-snmp-sysvinit
in the main net-snmp package and re-add the old %post, %preun and %postun magic. Of course that failed at install time for a good reason. In the second (desperate) try to work around it I tried something similar to "PreDepends"
Requires(post): net-snmp-sysvinit 
That works at install time but still breaks on a removal/update. So I think I did the right thing in the end (after reading some more documentation) and just added the scripts to the net-snmp-sysvinit package. It's just about adding the package name to the %post macro invocation. My initial plan included the wish to have one spec file that is usable to build on CentOS 5 and 6 but now I'm left with a number of special cases for CentOS 5. I already knew about the %if %endif conditional macro you can use in rpm spec files, so what was missing was a proper variable to base my decisions on. Every CentOS release ships with a "/etc/rpm/" directory that includes several macro file. Within those one can find the % rhel variable that defines an int we can use.
%if % rhel  <= 5
BuildRequires: perl
%else
BuildRequires: perl-devel
%endif
And finally I could build from one spec file a backport for CentOS 5 and 6. My diff against the CentOS 7 spec is at https://sven.stormbind.net/centos/netsnmp-bp-el56/net-snmp-backport.from_el7.diff Drawbacks of the resulting packages:
  1. No tcp-wrapper support, we don't care but that should be just another package name issue for the conditional Requires, simililar to the perl isssue.
  2. No mysql support. We do not care, one could change that and/or make it conditional or build against newer mysql releases where the -lib package is split.
  3. No DTLS support on CentOS 5. We do not care and that is not easy to fix.
  4. You've to install the net-snmp-sysvinit script to get back the usual setup you'd expect.
  5. We ripped out the systemd support, I did not investigate yet if it's possible to keep that included and just not enable it.
By now CentOS 5 is old. It's painful to backport to it. But it's still possible. Another strategy someone tried was packaging the current net-snmp upstream release based on the CentOS 5 spec file. That would've meant throwing away all 106 patches RedHat includes in that package at the moment and to depend on new upstream releases for your own security support. In case we start using CentOS 7 we would also diverge on the version of snmpd we use. So we decided to stick with the RedHat sources and keep the version consistent on our infrastructure even if we can not keep feature parity. After all the issues one encounters while backporting packages in the rpm world are similar to those in the dpkg world. Still it feels different if you're not familar with the toolchain. For example
  1. If you build a src.rpm with mock you've to copy it out of the result directory before you rebuild it with mock to create the binary package. Mock deletes everything from the result directory when starting a new build. So I guess parallel builds for the same chroot will not work.
  2. Depedencies on files instead of packages are a strange thing.
  3. How on earth do you maintain a patchset of over a hundred patches without something like quilt? I guess you always have to apply them in a for loop on your source directory and then you create a copy of that to implement your modifications and create another patch. I'm pretty sure eight years ago Fedora was far away from using git so I would rule out patch branches in a VCS for the moment. Not sure what they invented inside RedHat to keep track of it.

1 December 2014

Iustin Pop: SSD adventures

SSD adventures and fun times! TL;DR: Oh boy. Things work now, but I'm not sure exactly what happened :( When it all worked fine almost In June, I bought a new laptop, and a new SSD for it. I used that model of SSD before (Samsung 840 Evo), although not for a long time, so I wasn't expecting anything unusual. Since the laptop is a slower one, I installed Debian as follows: connect SSD to my workstation, do an install on it (via Virtualbox connected to the raw device), disconnect and install in the laptop. First sign of trouble was that the SSD didn't boot reliably. I said - maybe my Virtualbox method (new method) wasn't right - so I reinstalled on the laptop, and everything was mostly OK. Mostly OK because rarely the laptop had issues with seeing the SSD. Maybe not cold booting, requiring a restart, or giving some ATA errors on boot (bot nothing afterwards). I didn't use the laptop too much - I mostly use it when travelling - so I didn't investigate further. Trouble begins Fast forward to two weeks ago: I was preparing to leave on a trip, so I booted the laptop (everything OK), apt-get dist-upgrade, synced my git trees, etc. Everything was fine. The next day, in the airport between two flights, the laptop doesn't boot - doesn't see the SSD at all. I tried rebooting a gazillion times, nothing. I was quite upset - at the hardware, and at me for not paying attention to the unreliability signs before. Once I arrived at the destination, I opened the laptop, tried re-seating the SSD, nothing. I bought a SATA-to-USB bridge, and surprise! Boots from the first, no issues. Diagnosis A: Laptop SATA connector has issues. I work with this SATA-to-USB bridge for a couple of days, but it was quite slow (~20MB/s), so I buy a SATA-to-USB3 cradle, which should be much faster. But the SSD was not visible in this cradle. Not only that, but it was causing the laptop to hang in the POST screen - reliably. Turn the cradle off, the laptop passes POST, turn it on, the laptop took 2 minutes to pass the POST. OK, the cradle is broken. Connect the SSD back to the USB2 thing not booting For about five minutes, it was like "dead". After that, it booted and behaved normally. I didn't know what to think, I just put it aside. So worked for the rest of the week on the USB2 bridge, with no issues (once it the SSD dropped off, but I think that was just USB being USB). So at the end of the week, diagnosis (A) still was the main contender. On the flight back home, I worked from the plane for a good number of hours, again no issue. Laptop/SSD were fully powered off before the flight, powered with no issues, worked fine. At the end of the flight I completely shut down my laptop. Diagnosis A still on top. Real trouble now After getting home and sleeping a bit, I wanted to power up the laptop just to transfer the code I wrote on the plane. But it didn't power up. No problem, I said, now I actually have access to running Linux machines and I can check what's happening. And to my surprise, the SSD was behaving erratically: The looked like this (when connected over SATA):
20:22:57 kernel: ata2.00: ATA-9: Samsung SSD 840 EVO 500GB, EXT0BB6Q, max UDMA/133
20:22:57 kernel: ata2.00: 976773168 sectors, multi 1: LBA48 NCQ (depth 31/32), AA
20:22:57 kernel: ata2.00: configured for UDMA/133
20:22:57 kernel: scsi 3:0:0:0: Direct-Access     ATA      Samsung SSD 840  EXT0 PQ: 0 ANSI: 5
20:22:57 kernel: sd 3:0:0:0: [sdc] 488397168 512-byte logical blocks: (250 GB/232 GiB)
20:22:57 kernel: sd 3:0:0:0: [sdc] Write Protect is off
20:22:57 kernel: sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
20:22:57 kernel: sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
20:22:57 kernel: ata2.00: exception Emask 0x0 SAct 0x2 SErr 0x400001 action 0x6 frozen
20:22:57 kernel: ata2: SError:   RecovData Handshk  
20:22:57 kernel: ata2.00: failed command: READ FPDMA QUEUED
20:22:57 kernel: ata2.00: cmd 60/08:08:68:01:00/00:00:00:00:00/40 tag 1 ncq 4096 in
20:22:57 kernel:         res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
20:22:57 kernel: ata2.00: status:   DRDY  
20:22:57 kernel: ata2: hard resetting link
20:22:57 kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
20:22:57 kernel: ata2.00: failed to get NCQ Send/Recv Log Emask 0x1
20:22:57 kernel: ata2.00: failed to get NCQ Send/Recv Log Emask 0x1
20:22:57 kernel: ata2.00: configured for UDMA/133
20:22:57 kernel: ata2.00: device reported invalid CHS sector 0
20:22:57 kernel: ata2: EH complete
20:22:57 kernel: ata2.00: exception Emask 0x0 SAct 0x400000 SErr 0x400001 action 0x6
20:22:57 kernel: ata2.00: irq_stat 0x44000008
20:22:57 kernel: ata2: SError:   RecovData Handshk  
20:22:57 kernel: ata2.00: failed command: READ FPDMA QUEUED
20:22:57 kernel: ata2.00: cmd 60/08:b0:08:03:00/00:00:00:00:00/40 tag 22 ncq 4096 in
20:22:57 kernel:         res 41/84:00:08:03:00/00:00:00:00:00/00 Emask 0x410 (ATA bus error) <F>
20:22:57 kernel: ata2.00: status:   DRDY ERR  
20:22:57 kernel: ata2.00: error:   ICRC ABRT  
20:22:57 kernel: ata2: hard resetting link
20:22:57 kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
20:22:57 kernel: ata2.00: failed to get NCQ Send/Recv Log Emask 0x1
20:22:57 kernel: ata2.00: failed to get NCQ Send/Recv Log Emask 0x1
20:22:57 kernel: ata2.00: configured for UDMA/133
20:22:57 kernel: ata2: EH complete
Note that all of the messages were in short order (during boot, they show the same timestamp but I don't think it was actually the same second). I tried connecting the SSD over USB2 (partially working), over USB3 (not working!), and directly over SATA (initially not working). I connected another SSD I had around (same model, just smaller capacity) over all three, it worked (so the USB3 bridge was working, at least). Diagnosing (A) was out the door now, and the situation was very clear: SSD dying/dead/almost gone. So I connected both broken SSD and empty SSD to my workstation over SATA, and started copying data (once I managed to boot with the old one being visible and working). During the data copy, I saw that the "broken" SSD behaved erratically indeed: it was copying data off it with either ~40MB/s, ~70MB/s, and ~160MB/s. Not other speed, at least not for long time, just cycling between these three; and this is a very slow speed for this SSD model. And then I remembered that there is, for this model (Samsung 840 Evo), an advisory/firmware fix that old data gets harder to access (slower and slower), due to how TLC cells levels are read/etc. I don't know exactly what "old" means, but since the partition table was written only once, it should be the oldest thing written, which would make it the most susceptible to the slowdown, and could explain the cfdisk slowness. So after the data copy, I tested this: So yes, it is something related to this, probably. Diagnosis B: "just" the Samsung Evo bug. So I proceed to (try to) upgrade the firmware: Uh oh, so it's more than this - the SSD seems to be actually broken. I don't have much experience with SSDs, I used for many years an Intel X25-M and more recently I have a couple of Samsung 840 Pro, but I never had issues until now, so maybe the cheap Evo is just cheap and fast Note that during all this drama, SMART gives good info about SSD, and SMART tests are perfect (short/long). Just a number of UDMA CRC errors so thanks for nothing. I try to the firmware upgrade a few times, I give up. Diagnosis C: SSD is just plain broken. I'll send the SSD for a replacement/fix, so (even though it's encrypted), let's try to erase it first (as I can still read/write to it). And since just dd if=/dev/zero of=/dev/sdX is too common, let's try a builtin (ATA) erase! After fighting with hdparm and the fact that my BIOS does indeed "security freeze" on the drives (so you can't change the security settings nor erase the drives), and finding an article on the net that gives a few workaround, I manage to "unfreeze" it by not only live unplugging the SATA cable, but also the power cable, and plugging them back in. Time to erase. Side-note here: So, especially given the very fast ENHANCED erase, I now believe the SSD is truly broken. Diagnosis C rules. And finally, just confusion After the SSD erase, reading (the zeros) from the SSD is really fast: more than 500MB/s, as expected. I also try once more to upgrade the firmware for the Evo bug, and it works flawlessly this time. Even rebooting the machine many times works without problems, the SSD is seen every single time. I still can't make it work well over the USB3 bridge (but I'm not very convinced that device work well, since Linux does apply quirks when it detects it). But it doesn't give any ATA errors this time, and it works (read and write) consistently fast (as per the Evo specs). I still have to let it go through a long burnin with verification, but a short one works fine, so at this point I'm just very confused as what was (is?) the actual problem with this device, and why it behaves so strange Was the laptop broken? Clearly no. Was the SSD broken, or just impacted by the slowdown issue? Time will tell. I'll put it through the paces, and see. But unless I replace it, I'll keep this just for "fast cache" use cases, and not to hold real data. Yep, computers are fun!

Olivier Berger: Offline backup/mirror of a Moodle course, using httrack

I havent found much details online on how to perform a Moodle course mirror that could be browsed offline using httrack. This could be useful both for backup purposes, or for distant learners with connectivity issues. In my case, there s a login/password dialog that grants access to the moodle platform, which can be processed by httrack by capturing the POST form results using the catchurl option. The strategy I ve used is to add filters so that everything is excluded and only explicitely mentioned filters are then allowed to be mirrored. This allows to perform the backup connected with a user that may have high privileges, while avoiding to disappear in loops or complex links following for UI rendering variants of Moodle s interface.
Here s an example command line :
httrack -v -z -%F "Mirrored [from host %s [file %s [at %s]]]" -N "%h%p/%n%[id].%t" "http://mymoodle.example.com/login/index.php?>postfile:/home/myself/websites/mycourse/hts-post0" "http://mymoodle.example.com/course/view.php?id=42" -O "/home/myself/websites/mycourse" "-*/*" "+/login/index.php*" "+*/course/view.php*" "+*/mod/resource/view.php*" "+*/mod/page/view.php*" "+*/mod/forum/view.php*" "+*/mod/forum/discuss.php?d=*[0-9]" "+*/mod/url/view.php*" "+*/pluginfile.php/*" "+*/mod/feedback/view.php*" "+*/mod/feedback/analysis.php*" "+*/theme/*" "-*/course/view.php?id=43"
Let s comment on these different parameters : I hope this will be useful.

30 November 2014

Enrico Zini: cxx11-talk-examples

C++11 talk examples On 2014-11-27 I gave a talk about C++ and new features introduced with C++11: these are the examples. They are all licensed under the wtfpli version 2. See cxx11-talk-notes for the talk notes. Note that the wrapper interfaces turns errors from the underlying libraries into exceptions, so the method calls just do what they should, without the need of documenting special return values for error messages, and removing the need for each library to implement yet another way of reporting errors. Also note that all wrapper objects do RAII: you create them and they clean after themselves when they go out of scope. The wrapper objects also have cast operators to make them behave as the pointer or handle that they are wrapping, so that they can be transparently passed to the underlying libraries. (note: I had to add U+2063 INVISIBLE SEPARATOR to prevent noreturn statements to be misinterpreted by the blog formatter. If you copypaste the code and encounter issues, you may want to delete the noreturn statements and retype them) A gcrypt hash class This class is a light wrapper around gcrypt's hashing functions. ezhash.h
#ifndef EZHASH_H
#define EZHASH_H
#include <string>
#include <gcrypt.h>
namespace ezhash  
class Hash
 
protected:
    // members can now be initialized just like this, without needing to repeat
    // their default assignment in every constructor
    gcry_md_hd_t handle = nullptr;
public:
    Hash(int algo, unsigned int flags=0);
    ~Hash();
    // Assign 'delete' to a method to tell the compiler not to generate it
    // automatically. In this case, we make the object non-copiable.
    Hash(const Hash&) = delete;
    Hash(const Hash&&) = delete;
    Hash& operator=(const Hash&) = delete;
    // Add a buffer to the hash
    void hash_buf(const std::string& buf);
    // Add the contents of a file to the hash
    void hash_file(int fd);
    // Get a string with the hexadecimal hash
    std::string read_hex(int algo=0);
    /// Pretend that we are a gcry_md_hd_t handle
    operator gcry_md_hd_t()   return handle;  
 ;
 
#endif
ezhash.cpp
#include "ezhash.h"
#include <unistd.h>
#include <errno.h>
#include <string>
#include <cstring>
#include <sstream>
#include <iomanip>
#include <stdexcept>
using namespace std;
namespace ezhash  
namespace  
// noreturn attribute, to tell the compiler that this function never returns
[ [noreturn]] void throw_gcrypt_error(gcry_error_t err)
 
    string msg;
    msg += gcry_strsource(err);
    msg += "/";
    msg += gcry_strerror(err);
    throw runtime_error(msg);
 
string errno_str(int error)
 
    char buf[256];
#if (_POSIX_C_SOURCE >= 200112L   _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE
    strerror_r(errno, buf, 256);
    string res(buf);
#else
    string res(strerror_r(errno, buf, 256));
#endif
    return res;
 
[ [noreturn]] void throw_libc_error(int error)
 
    throw runtime_error(errno_str(error));
 
 
Hash::Hash(int algo, unsigned int flags)
 
    gcry_error_t err = gcry_md_open(&handle, algo, flags);
    if (err) throw_gcrypt_error(err);
 
Hash::~Hash()
 
    gcry_md_close(handle);
 
void Hash::hash_buf(const std::string& buf)
 
    gcry_md_write(handle, buf.data(), buf.size());
 
void Hash::hash_file(int fd)
 
    char buf[4096];
    while (true)
     
        ssize_t res = ::read(fd, buf, 4096);
        if (res < 0) ezfs::throw_libc_error();
        if (res == 0) break;
        gcry_md_write(handle, buf, res);
     
 
std::string Hash::read_hex(int algo)
 
    unsigned char* res = gcry_md_read(handle, algo);
    unsigned int len = gcry_md_get_algo_dlen(
            algo == 0 ? gcry_md_get_algo(handle) : algo);
    // Format the hash into a hex digit
    stringstream hexbuf;
    hexbuf << hex << setfill('0');
    for (unsigned i = 0; i < len; ++i)
        hexbuf << setw(2) << (unsigned)res[i];
    return hexbuf.str();
 
 
Example usage
        ezhash::Hash sha256(GCRY_MD_SHA256);
        sha256.hash_buf("ciao\n");
        sha256.hash_buf("foo\n");
        cout << sha256.read_hex() << endl;
Simple sqlite bindings Remarkably simple sqlite3 bindings based on lambda callbacks. ezsqlite.h
#ifndef EZSQLITE_H
#define EZSQLITE_H
#include <sqlite3.h>
#include <string>
#include <functional>
#include <stdexcept>
namespace ezsqlite  
/// RAII wrapper around a sqlite3 database handle
class DB
 
protected:
    sqlite3* handle = nullptr;
public:
    // Open a connection to a SQLite database
    DB(const std::string& filename, int flags=SQLITE_OPEN_READWRITE   SQLITE_OPEN_CREATE);
    DB(const DB&) = delete;
    DB(const DB&&) = delete;
    DB& operator=(const DB&) = delete;
    ~DB();
    /**
     * Execute a query, optionally calling 'callback' on every result row
     *
     * The arguments to callback are:
     *  1. number of columns
     *  2. text values of the columns
     *  3. names of the columns
     */
    // std::function can be used to wrap any callable thing in C++
    // see: http://en.cppreference.com/w/cpp/utility/functional/function
    void exec(const std::string& query, std::function<bool(int, char**, char**)> callback=nullptr);
    /// Pretend that we are a sqlite3 pointer
    operator sqlite3*()   return handle;  
 ;
 
#endif
ezsqlite.cpp
#include "ezsqlite.h"
namespace ezsqlite  
DB::DB(const std::string& filename, int flags)
 
    int res = sqlite3_open_v2(filename.c_str(), &handle, flags, nullptr);
    if (res != SQLITE_OK)
     
        // From http://www.sqlite.org/c3ref/open.html
        // Whether or not an error occurs when it is opened, resources
        // associated with the database connection handle should be
        // released by passing it to sqlite3_close() when it is no longer
        // required.
        std::string errmsg(sqlite3_errmsg(handle));
        sqlite3_close(handle);
        throw std::runtime_error(errmsg);
     
 
DB::~DB()
 
    sqlite3_close(handle);
 
namespace  
// Adapter to have sqlite3_exec call a std::function
int exec_callback(void* data, int columns, char** values, char** names)
 
    std::function<bool(int, char**, char**)> cb = *static_cast<std::function<bool(int, char**, char**)>*>(data);
    return cb(columns, values, names);
 
 
void DB::exec(const std::string& query, std::function<bool(int, char**, char**)> callback)
 
    char* errmsg;
    void* cb = callback ? &callback : nullptr;
    int res = sqlite3_exec(handle, query.c_str(), exec_callback, cb, &errmsg);
    if (res != SQLITE_OK && errmsg)
     
        // http://www.sqlite.org/c3ref/exec.html
        //
        // If the 5th parameter to sqlite3_exec() is not NULL then any error
        // message is written into memory obtained from sqlite3_malloc() and
        // passed back through the 5th parameter. To avoid memory leaks, the
        // application should invoke sqlite3_free() on error message strings
        // returned through the 5th parameter of of sqlite3_exec() after the
        // error message string is no longer needed. 
        std::string msg(errmsg);
        sqlite3_free(errmsg);
        throw std::runtime_error(errmsg);
     
 
 
Example usage
    // Connect to the database
    ezsqlite::DB db("erlug.sqlite");
    // Make sure we have a table
    db.exec(R"(
        CREATE TABLE IF NOT EXISTS files (
                name TEXT NOT NULL,
                sha256sum TEXT NOT NULL
        )
    )");
    // Read the list of files that we know
    map<string, string> files;
    db.exec("SELECT name, sha256sum FROM files", [&](int columns, char** vals, char** names)  
        if (columns != 2) return false;
        files.insert(make_pair(vals[0], vals[1]));
        return true;
     );
A fast Directory object This is a lightweight wrapper around O_PATH file descriptors for directories. I'd love to see a library of well-maintained and thin C++ bindings around libc, that do little more than turning errors into exceptions and making it also work with std::string buffers. ezfs.h
#ifndef EZFS_H
#define EZFS_H
#include <string>
#include <functional>
#include <memory>
#include <cerrno>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <dirent.h>
namespace ezfs  
class Directory
 
protected:
    int handle = -1;
public:
    Directory(const std::string& pathname, int flags=0);
    ~Directory();
    Directory(const Directory&) = delete;
    Directory(const Directory&&) = delete;
    Directory& operator=(const Directory&) = delete;
    /// List the directory contents
    void ls(std::function<void(const dirent&)> callback);
    int open(const std::string& relpath, int flags, mode_t mode=0777);
 ;
std::string errno_str(int error=errno);
[ [noreturn]] void throw_libc_error(int error=errno);
 
#endif
ezfs.cpp
#include "ezfs.h"
#include <stdexcept>
#include <memory>
#include <cstring>
#include <cstdlib>
#include <string>
#include <linux/limits.h>
using namespace std;
namespace ezfs  
string errno_str(int error)
 
    char buf[256];
#if (_POSIX_C_SOURCE >= 200112L   _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE
    strerror_r(errno, buf, 256);
    string res(buf);
#else
    string res(strerror_r(errno, buf, 256));
#endif
    return res;
 
[ [noreturn]] void throw_libc_error(int error)
 
    throw runtime_error(errno_str(error));
 
Directory::Directory(const std::string& pathname, int flags)
 
    handle = ::open(pathname.c_str(), O_PATH   O_DIRECTORY   flags);
    if (handle < 0) throw_libc_error();
 
Directory::~Directory()
 
    ::close(handle);
 
void Directory::ls(std::function<void(const dirent&)> callback)
 
    int fd = openat(handle, ".", O_DIRECTORY);
    if (fd < 0) throw_libc_error();
    // RAII Self-cleaning DIR object
    unique_ptr<DIR, std::function<void(DIR*)>> dir(fdopendir(fd), [](DIR* dir)   if (dir) closedir(dir);  );
    if (!dir)
     
        // fdopendir(3): After a successful call to fdopendir(), fd is used
        // internally by the implementation, and should not otherwise be used
        // by the application.
        //
        // but if the fdopendir call was not successful, fd is not managed by
        // DIR, and we still need to close it, otherwise we leak a file
        // descriptor.
        //
        // However, close() may modify errno, so we take note of the errno set
        // by fdopendir and raise the exception based on that.
        int fdopendir_errno = errno;
        close(fd);
        throw_libc_error(fdopendir_errno);
     
    // Size the dirent buffer properly
    const unsigned len = offsetof(dirent, d_name) + PATH_MAX + 1;
    unique_ptr<dirent, std::function<void(void*)>> dirbuf((dirent*)malloc(len), free);
    while (true)
     
        dirent* res;
        int err = readdir_r(dir.get(), dirbuf.get(), &res);
        // End of directory contents
        if (err == 0)
         
            if (res)
                callback(*res);
            else
                break;
          else
            throw_libc_error(err);
     
 
int Directory::open(const std::string& relpath, int flags, mode_t mode)
 
    int res = openat(handle, relpath.c_str(), flags, mode);
    if (res < 0) throw_libc_error();
    return res;
 
 
Example usage
        // This is almost the equivalent of running "sha256sum ."
        ezfs::Directory dir(".");
        dir.ls([&](const dirent& d)  
            if (d.d_type != DT_REG) return;
            ezhash::Hash sha256(GCRY_MD_SHA256);
            // I have no RAII wrapper around file handles at the moment, so
            // I'll have to use a try/catch for cleaning up after errors
            int fd = dir.open(d.d_name, O_RDONLY);
            try  
                sha256.hash_file(fd);
                close(fd);
              catch (...)  
                close(fd);
                throw;
             
            cout << sha256.read_hex() << "  " << d.d_name << endl;
         );

Enrico Zini: cxx11-talk-notes

C++11 talk notes On 2014-11-27 I gave a talk about C++ and new features introduced with C++11: these are the talk notes. See cxx11-talk-examples for the examples. (note: I had to add U+2063 INVISIBLE SEPARATOR to prevent noreturn statements to be misinterpreted by the blog formatter. If you copypaste the code and encounter issues, you may want to delete the noreturn statements and retype them) Overview of programming languages
It has to be as fast as possible, so interpreted languages are out. You don't want to micro manage memory, so C is out. You don't want to require programmers to have a degree, so C++ is out. You want fast startup and not depend on a big runtime, so Java is out. [...] (Bram Moolenaar)
C++ secret cultist protip
Do not call up what you cannot put down.
C++ is a compiled language It is now possible to use the keyword constexpr to mark functions and objects that can be used at compile time:
/*
 * constexpr tells the compiler that a variable or function can be evaluated at
 * compile time.
 *
 * constexpr functions can also be run at run time, if they are called with
 * values not known at compile time.
 *
 * See http://en.cppreference.com/w/cpp/language/constexpr for more nice examples
 *
 * It can be used to avoid using constants in code, and using instead functions
 * for computing hardware bitfields or physical values, without losing in
 * efficiency.
 */
#include <iostream>
using namespace std;
constexpr int factorial(int n)
 
    return n <= 1 ? 1 : (n * factorial(n-1));
 
int main()
 
    cout << "Compile time factorial of 6: " << factorial(6) << endl;
    cout << "Enter a number: ";
    int a;
    cin >> a;
    cout << "Run time factorial of " << a << ": " << factorial(a) << endl;
 
See also this for more nice examples. See this and this for further discussion. Multiline strings
        const char* code = R"--(
          printf("foo\tbar\n");
          return 0;
        )--";
See this. C++ memory management protip RAII: Resource Acquisition Is Instantiation This is not new in C++11, but in my experience I have rarely seen it mentioned in C++ learning material, and it does make a huge difference in my code. See this and this for details. Constructors and member initializer lists Initializers in curly braces now have their own type: std::initializer_list:
#include <string>
#include <iostream>
#include <unordered_set>
using namespace std;
// std::initializer_list< >
//   will have as its value all the elements inside the curly braces
string join(initializer_list<string> strings)
 
    string res;
    for (auto str: strings)
     
        if (!res.empty())
            res += ", ";
        res += str;
     
    return res;
 
int main()
 
    unordered_set<string> blacklist  ".", "..", ".git", ".gitignore"  ;
    cout << join(  "antani", "blinda"  ) << endl;
 
See this for details, including the new uniform initialization trick of omitting parentesis in constructors so that you can call normal constructors and initializer_list constructors with the same syntax, which looks like an interesting thing when writing generic code in templates. Type inference I can now use auto instead of a type to let the compiler automatically compute the value of something I assign to:
        auto i = 3 + 2;
        // See also https://github.com/esseks/monicelli
        vector<string> names  "antani", "blinda", "supercazzola"  ;
        for (auto i = names.cbegin(); i != names.cend(); ++i)
            cout << i;
        template<typename T>
        T frobble(const T& stuff)
         
             // This will work whatever type is returned by stuff.read()
             auto i = stuff.read();
             //  
         
See this for more details. Range-based for loop C++ now has an equivalent of the various foreach constructs found in several interpreted languages!
        for (auto i: list_of_stuff)
                cout << i << endl;
        for (auto n:  0,1,2,3,4,5 )
                cout << n << endl;
        // This construct:
        for (auto i: stuff)
        // If stuff is an array, it becomes:
        for (i = stuff, i < stuff + sizeof(stuff) / sizeof(stuff[0]); ++i)
        // If stuff has .begin() and .end() methods it becomes:
        for (i = stuff.begin(); i != stuff.end(); ++i)
        // Otherwise it becomes:
        for (i = begin(stuff); i != end(stuff); ++i)
        // And you can define begin() and end() functions for any type you
        // want, at any time
See this and this for details. Lambda functions and expressions Lambdas! Closures! Something like this:
// JavaScript
var add = function(a, b)   return a + b;  
# Python
add = lambda a, b: a + b
Becomes this:
auto add = [](int a, int b)   return a + b;  
And something like this:
// JavaScript
var a = 0;
$.each([1, 2, 3, 4], function(idx, el)   a += el  );
Becomes this:
unsigned a = 0;
std::for_each(  1, 2, 3, 4  , [&a](int el)   return a += el;  );
See this, this and this. Tuple types C++ now has a std::tuple type, that like in Python can be used to implement functions that return multiple values:
        tuple<int, string, vector<string>> parse_stuff()
         
                return make_tuple(id, name, values);
         
        string name; vector<string> values;
        // std::ignore can be used to throw away a result
        tie(ignore, name, values) = parse_stuff();
        // std::tie can also be used to do other kind of
        // multi-operations besides assignment:
        return tie(a, b, c) < tie(a1, b1, c1);
        // Is the same as:
        if (a != a1) return a < a1;
        if (b != b1) return b < b1;
        return c < c1;
See here, here and here. Regular expressions We now have regular expressions!
        std::regex re(R"((\w+)\s+(\w+))");
        string s("antani blinda");
        smatch res;
        if (regex_match(s, res, re))
            cout << "OK " << res[1] << " -- " << res[2] << endl;
The syntax is ECMAScript by default and can be optionally changed to basic, extended, awk, grep, or egrep. See here and here. General-purpose smart pointers There is std::unique_ptr to code memory ownership explicitly, and std::shared_ptr as a reference counted pointer, and smart pointers can have custom destructors:
    unique_ptr<dirent, std::function<void(void*)>> dirbuf((dirent*)malloc(len), free);
See here and here. Miscellaneous other cool things Standard attribute specifiers
string errno_str(int error)
 
    char buf[256];
#if (_POSIX_C_SOURCE >= 200112L   _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE
    strerror_r(errno, buf, 256);
    string res(buf);
#else
    string res(strerror_r(errno, buf, 256));
#endif
    return res;
 
[ [noreturn]] void throw_libc_error(int error)
 
    throw runtime_error(errno_str(error));
 
See here. Hash tables See here and look at the new containers unordered_set, unordered_map, unordered_multiset, and unordered_multimap. Multithreading There is a standard threading model, with quite a bit of library support: see here, here, here, and here for atomic data structures. Variadic templates Templates can now take variable number of arguments, and that opens possibilities for interesting code generation, like implementing a generic, type-safe printf statement, or something like this:
db.query(R"(
   INSERT INTO table NAMES (id, name, description)
     VALUES (?, ?, ?)
)", 4, "genio", "fantasia, intuizione, decisione, e velocit  di esecuzione");
See here and here. Essential tools You need at least g++ 4.8 or clang 3.3 to have full C++11 support. They will be both available in jessie, and for wheezy you can use the nightly clang packages repository. I cannot think of a good excuse not to use -Wall on new code. scan-build from clang is another nice resource for catching even more potential problems at compile time. valgrind is a great tool for runtime code analysis: valgrind --tool=memcheck (the default) will check your program for wrong memory accesses and memory leaks. valgrind --tool=callgrind will trace function calls for profiling, to be analyzed with kcachegrind. valgrind --tool=helgrind can check multi-threaded programs for suspicious concurrent memory accesse patterns. And of course gdb: a nice trick with C++ is to issue catch throw to get a breakpoint at the point where an exception is being thrown. help catch provides a list of other interesting catch examples. Coredump tips: ulimit -c to enable core dumps, triggering a core dump with ^\, opening a core with gdb program core, and more details on man 5 core. An extra gdb tip, which is not related to C++ but helped me considerably recently, is that it can be attached to running python programs to get a live Python traceback.

24 November 2014

Rog rio Brito: Problems with Emacs 24.4

This is, essentially, a call for help, as I don't really know which program is at a fault here. Given that Emacs's upstream converted their repository from bzr to git, all the commits in mirror repositories became "invalid" in relation to the official repository. What does this mean in practical terms, in my case? Well, bear with me while I try to report my steps.

Noticing a regression and reporting a bug There is a regression with Emacs 24.4 relative to 24.3, which I discovered after Emacs 24.4 became available in Debian's sid. The regression in particular is that Emacs 24.4 doesn't seem to respect my Xresources, while 24.3 does (and this is 100% reproducible: I kept the binary packages of version 24.3 of emacs24 and I can install and reinstall things). When I reported this to upstream, I received a reply that it worked fine with another person that was using XFCE with unstable.

Testing various Desktop environments As I am using the MATE desktop environment, I proceeded to test this assertion by installing XFCE. Emacs 24.4 read my Xresources. I went ahead and installed LXDE. It worked again. I tried once more with GNOME 3, but "regular" GNOME 3 just crashed. I tried with GNOME 3 Classic and Emacs 24.4 just worked again.

Going deep into the rabbit's hole Then, I got more curious and I tried to see why things worked the way that they did and given that there was a mirror of the Emacs repo on github, I cloned it and started to git bisect to find the first problematic commit (I have no idea if bzr even offers something like git's bisect and I wouldn't really know how to do it as quickly as I do with git). To cut short a long story, after many recompiles, many wasted hours, a lot of wasted electrical energy, I found a bad commit and reported it. I received no response after that.

The new repo enters in action Of course, all my hard work bisecting things was completely invalidated after the transition to the new repository went live. To make things relevant again, I used the awesome powers of git, restricting the changes of the newly cloned repository to the e-mail of the committer in question (Chong Yidong) and, from there, I proceeded to another painful process of git bisects. And, sure enough, the first bad commit was the same one that I found with the previous tree. Semi-blindly reverting this commit, and also semi-blindly resolving the conflicts make Emacs's from master work again on my system, but I highly suspect that (given the way that I did it), it would not really be appropriate for upstream. But given also that I failed to receive feedback after my original report, I am not too confident that this bug can be solved soon (even if it doesn't qualify for being fixed in Debian 8). After all this, I don't really know what else to do. I even filed a bug report (more like a request for help) to the Debian MATE maintainers. As a side note, I would have filed a bug to upstream MATE, but it is not really clear what the proper procedure to report bugs to them is---they seem to use github's issues system, but given that they have separate repositories for each component of the project, and that I don't know precisely what repository to report to (or even if it applies to MATE after all), I am more or less paralyzed.

A side note I must say that the conversion was well done by Eric Raymond, because the whole .git repository of the new repo is only about 200MB, with history going back to 1985, while the other repository had about 800GB.

John Goerzen: My boys love 1986 computing

Yesterday, Jacob (age 8) asked to help me put together a 30-year-old computer from parts in my basement. Meanwhile, Oliver (age 5) asked Laura to help him learn cursive. Somehow, this doesn t seem odd for a Saturday at our place. 2014-11-22 18.58.36 Let me tell you how this came about. I ve had a project going on for a while now to load data from old floppies. It s been fun, and had a surprise twist the other day: my parents gave me an old TRS-80 Color Computer II (aka CoCo 2 ). It was, in fact, my first computer, one they got for me when I was in Kindergarten. It is nearly 30 years old. I have been musing lately about the great disservice Apple did the world by making computers easy to learn namely the fact that few people ever bother to learn about them. Who bothers to learn about them when, on the iPhone for instance, the case is sealed shut, the lifespan is 1 or 2 years for many purchasers, and the platform is closed in lots of ways? I had forgotten how finicky computers used to be. But after some days struggling with IDE incompatibilities, booting issues, etc., when I actually managed to get data off a machine that had last booted in 1999, I had quite the sense of accomplishment, which I rarely have lately. I did something that was hard to do in a world where most of the interfaces don t work with equipment that old (even if nominally they are supposed to.) The CoCo is one of those computers normally used with a floppy drive or cassette recorder to store programs. You type DIR, and you feel the clack of the drive heads through the desk. You type CLOAD and you hear the relay click closed to turn on the tape motor. You wiggle cables around until they make contact just right. You power-cycle for the times when the reset button doesn t quite do the job. The details of how it works aren t abstracted away by innumerable layers of controllers, interfaces, operating system modules, etc. It s all right there, literally vibrating your desk. So I thought this could be a great opportunity for Jacob to learn a few more computing concepts, such as the difference between mass storage and RAM, plus a great way to encourage him to practice critical thinking. So we trekked down to the basement and came up with handfulls of parts. We brought up the computer, some joysticks, all sorts of tangled cables. We needed adapters, an old TV. Jacob helped me hook everything up, and then the moment of truth: success! A green BASIC screen! I added more parts, but struck out when I tried to connect the floppy drive. The thing just wouldn t start up right whenever the floppy controller cartridge was installed. I cleaned the cartridge. I took it apart, scrubbed the contacts, even did a re-seat of the chips. No dice. So I fired up my CoCo emulator (xroar), and virtually saved some programs to cassette (a .wav file). I then burned those .wav files to an audio CD, brought up an old CD player from the basement, connected the cassette in plug to the CD player s headphone jack, and presto instant programs. (Well, almost. It takes a couple of minutes to load a program from audio codes.) The picture above is Oliver cackling at one of the very simplest BASIC programs there is: number find. The computer picks a random number between 1 and 2000, and asks the user to guess it, giving a too low or too high clue with each incorrect guess. Oliver delighted in giving invalid input (way too high numbers, or things that weren t numbers at all) and cackled at the sarcastic error messages built into the program. During Jacob s turn, he got very serious about it, and is probably going to be learning about how to calculate halfway points before too long. But imagine my pride when this morning, Jacob found the new CD I had made last night (correcting a couple recordings), found my one-line instruction on just part of how to load a program, and correctly figured out by himself all the steps to do in order (type CLOAD on the CoCo, advance the CD to the proper track, press play on the player, wait for it to load on the CoCo, then type RUN). I ordered a replacement floppy controller off eBay tonight, and paid $5 for a coax adapter that should fix some video quality issues. I rescued some 5.25 floppies from my trash can from another project, so they should have plenty of tools for exploration. It is so much easier for them to learn how a disk drive works, and even what the heck a track is, when you can look at a floppy drive with the cover off and see the heads move. There are other things we can do with more modern equipment Jacob has shown a lot of interest in Arduino projects but I have so far drawn a blank on ways to really let kids discover how a modern PC (let alone a modern phone or tablet) works. Update Nov. 24: Every so often, the world surprises me by deciding to, well, read one of my random blog posts. For the benefit of those of you that don t already know my boys, you might want to know that among their common play activites are turning trees into pretend trains, typing at a manual typewriter, reading, writing their own books, using a cassette recorder, building a PC and learning to use bash or xmonad, making long paper tapes with an adding machine, playing records on a record player, building electric gizmos, and even making mud balls. I am often asked about the role of the computer in the lives, given that my hobby and profession involves computers. The answer: less than that of most of their peers. I look for opportunities for them to learn by doing, discovering, playing, or imagining. I make no presumption that they will develop the passion for computers that I did. What I want is for them to have the curiosity and drive to learn everything there is to know about whatever they do develop a passion for, so they will be great at it.

22 November 2014

Sune Vuorela: Is linux about choice?

Occasionally, various quotes from people having an opinion if linux is about choice or not. Even pages like http://www.islinuxaboutchoice.com/ has shown up. My short answer is YES . Linux is about choice. And you get all your choices directly from your f/loss definition of choice (FSF s 4 freedoms / OSI s opensource definition / Debian Free Software Guidelines) It doesn t mean that you get all the gui configuration bits that you want. It doesn t mean that you without any problems can switch out any component. But it does mean that you can get it exactly your way. But it might require you to edit some source code and compile some stuff.

21 November 2014

Jonathan Wiltshire: On kFreeBSD and FOSDEM

Boy I love rumours. Recently I ve heard two, which I ought to put to rest now everybody s calmed down from recent events. kFreeBSD isn t an official Jessie architecture because <insert systemd-related scare story> Not true. Our sprint at ARM (who kindly hosted and caffeinated us for four days) was timed to coincide with the Cambridge Mini-DebConf 2014. The intention was that this would save on travel costs for those members of the Release Team who wanted to attend the conference, and give us a jolly good excuse to actually meet up. Winners all round. It also had an interesting side-effect. The room we used was across the hall from the lecture theatre being used as hack space and, later, the conference venue, which meant everybody attending during those two days could see us locked away there (and yes, we were in there all day for two days solid, except for lunch times and coffee missions). More than one conference attendee remarked to me in person that it was interesting for them to see us working (although of course they couldn t hear what we were discussing), and hadn t appreciated before that how much time and effort goes into our meetings. Most of our first morning was taken up with the last pieces of architecture qualification, and that was largely the yes/no decision we had to make about kFreeBSD. And you know what? I don t recall us talking about systemd in that context at all. Don t forget kFreeBSD already had a waiver for a reduced scope in Jessie because of the difficulty in porting systemd to it. It s sadly impossible to capture the long and detailed discussion we had into a couple of lines of status information in a bits mail. If bits mails were much longer, people would be put off reading them, and we really really want you to take note of what s in there. The little space we do have needs to be factual and to the point, and not include all the background that led us to a decision. So no, the lack of an official Jessie release of kFreeBSD has very little, if anything, to do with systemd. Jessie will be released during (or even before) FOSDEM Not necessarily true. Debian releases are made when they re ready. That sets us apart from lots of other distributions, and is a large factor in our reputation for stability. We may have a target date in mind for a freeze, because that helps both us and the rest of the project plan accordingly. But we do not have a release date in mind, and will not do so until we get much closer to being ready. (Have you squashed an RC bug today?) I think this rumour originated from the office of the DPL, but it s certainly become more concrete than I think Lucas intended. However, it is true that we ve gone into this freeze with a seriously low bug count, because of lots of other factors. So it may indeed be that we end up in good enough shape to think about releasing near (or even at) FOSDEM. But rest assured, Debian 8 Jessie will be released when it s ready, and even we don t know when that will be yet. (Of course, if we do release before then, you could consider throwing us a party. Plenty of the Release Team, FTP masters and CD team will be at FOSDEM, release or none.)
On kFreeBSD and FOSDEM is a post from: jwiltshire.org.uk Flattr

Gunnar Wolf: Status of the Debian OpenPGP keyring November update

Almost two months ago I posted our keyring status graphs, showing the progress of the transition to >=2048-bit keys for the different active Debian keyrings. So, here are the new figures. First, the Non-uploading keyring: We were already 100% transitioned. You will only notice a numerical increase: That little bump at the right is our dear friend T ssia finally joining as a Debian Developer. Welcome! \o/ As for the Maintainers keyring: We can see a sharp increase in 4096-bit keys. Four 1024-bit DM keys were migrated to 4096R, but we did have eight new DMs coming in To them, also, welcome \o/. Sadly, we had to remove a 1024-bit key, as Peter Miller sadly passed away. So, in a 234-key universe, 12 new 4096R keys is a large bump! Finally, our current-greatest worry If for nothing else, for the size of the beast: The active Debian Developers keyring. We currently have 983 keys in this keyring, so it takes considerably more effort to change it. But we have managed to push it noticeably. This last upload saw a great deal of movement. We received only one new DD (but hey welcome nonetheless! \o/ ). 13 DD keys were retired; as one of the maintainers of the keyring, of course this makes me sad but then again, in most cases it's rather an acknowledgement of fact: Those keys' holders often state they had long not been really involved in the project, and the decision to retire was in fact timely. But the greatest bulk of movement was the key replacements: A massive 62 1024D keys were replaced with stronger ones. And, yes, the graph changed quite abruptly: We still have a bit over one month to go for our cutoff line, where we will retire all 1024D keys. It is important to say we will not retire the affected accounts, mark them as MIA, nor anything like that. If you are a DD and only have a 1024D key, you will still be a DD, but you will be technically unable to do work directly. You can still upload your packages or send announcements to regulated mailing lists via sponsor requests (although you will be unable to vote). Speaking of votes: We have often said that we believe the bulk of the short keys belong to people not really active in the project anymore. Not all of them, sure, but a big proportion. We just had a big, controversial GR vote with one of the highest voter turnouts in Debian's history. I checked the GR's tally sheet, and the results are interesting: Please excuse my ugly bash, but I'm posting this so you can play with similar runs on different votes and points in time using the public keyring Git repository:
  1. $ git checkout 2014.10.10
  2. $ for KEY in $( for i in $( grep '^V:' tally.txt
  3. awk ' print "<" $3 ">" ' )
  4. do
  5. grep $i keyids cut -f 1 -d ' '
  6. done )
  7. do
  8. if [ -f debian-keyring-gpg/$KEY -o -f debian-nonupload-gpg/$KEY ]
  9. then
  10. gpg --keyring /dev/null --keyring debian-keyring-gpg/$KEY \
  11. --keyring debian-nonupload-gpg/$KEY --with-colons \
  12. --list-key $KEY 2>/dev/null \
  13. head -2 tail -1 cut -f 3 -d :
  14. fi
  15. done sort uniq -c
  16. 95 1024
  17. 13 2048
  18. 1 3072
  19. 371 4096
  20. 2 8192
So, as of mid-October: 387 out of the 482 votes (80.3%) were cast by developers with >=2048-bit keys, and 95 (19.7%) were cast by short keys. If we were to run the same vote with the new active keyring, 417 votes would have been cast with >=2048-bit keys (87.2%), and 61 with short keys (12.8%). We would have four less votes, as they retired:
  1. 61 1024
  2. 14 2048
  3. 2 3072
  4. 399 4096
  5. 2 8192
So, lets hear it for November/December. How much can we push down that pesky yellow line? Disclaimer: Any inaccuracy due to bugs in my code is completely my fault!

20 November 2014

Gunnar Wolf: UNAM. Viva M xico, viva en paz.

UNAM. Viva M xico, viva en paz.
We have had terrible months in Mexico; I don't know how much has appeared about our country in the international media. The last incidents started on the last days of September, when 43 students at a school for rural teachers were forcefully disappeared (in our Latin American countries, this means they were taken by force and no authority can yet prove whether they are alive or dead; forceful disappearance is one of the saddest and most recognized traits of the brutal military dictatorships South America had in the 1970s) in the Iguala region (Guerrero state, South of the country) and three were killed on site. An Army regiment was stationed few blocks from there and refused to help. And yes, we live in a country where (incredibly) this news by themselves would not seem so unheard of... But in this case, there is ample evidence they were taken by the local police forces, not by a gang of (assumed) wrongdoers. And they were handed over to a very violent gang afterwards. Several weeks later, with far from a thorough investigation, we were told they were killed, burnt and thrown to a river. The Iguala city major ran away, and was later captured, but it's not clear why he was captured at two different places. The Guerrero state governor resigned and a new governor was appointed. But this was not the result of a single person behaving far from what their voters would expect It's a symptom of a broken society where policemen will kill when so ordered, where military personnel will look away when pointed out to the obvious, where the drug dealers have captured vast regions of the country where are stronger than the formal powers. And then, instead of dealing with the issue personally as everybody would expect, the president goes on a commercial mission to China. Oh, to fix some issues with a building company. That coincidentally or not was selling a super-luxury house to his wife. A house that she, several days later, decided to sell because it was tarnishing her family's honor and image. And while the president is in China, the person who dealt with the social pressure and told us about the probable (but not proven!) horrible crime where the "bad guys" for some strange and yet unknown reason (even with tens of them captured already) decided to kill and burn and dissolve and disappear 43 future rural teachers presents his version, and finishes his speech saying that "I'm already tired of this topic". Of course, our University is known for its solidarity with social causes; students in our different schools are the first activists in many protests, and we have had a very tense time as the protests are at home here at the university. This last weekend, supposed policemen entered our main campus with a stupid, unbelievable argument (they were looking for a phone reported as stolen three days earlier), get into an argument with some students, and end up firing shots at the students; one of them was wounded in the leg. And the university is now almost under siege: There are policemen surrounding us. We are working as usual, and will most likely finish the semester with normality, but the intimidation (in a country where seeing a policeman is practically never a good sign) is strong. And... Oh, I could go on a lot. Things feel really desperate and out of place. Today I will join probably tens or hundreds of thousands of Mexicans sick of this simulation, sick of this violence, in a demonstration downtown. What will this achieve? Very little, if anything at all. But we cannot just sit here watching how things go from bad to worse. I do not accept to live in a state of exception. So, this picture is just right: A bit over a month ago, two dear friends from Guadalajara city came, and we had a nice walk in the University. Our national university is not only huge, it's also beautiful and loaded with sights. And being so close to home, it's our favorite place to go with friends to show around. This is a fragment of the beautiful mural in the Central Library. And, yes, the University stands for "Viva M xico". And the university stands for "Peace". And we need it all. Desperately.

Steve Kemp: An experiment in (re)building Debian

I've rebuilt many Debian packages over the years, largely to fix bugs which affected me, or to add features which didn't make the cut in various releases. For example I made a package of fabric available for Wheezy, since it wasn't in the release. (Happily in that case a wheezy-backport became available. Similar cases involved repackaging gtk-gnutella when the protocol changed and the official package in the lenny release no longer worked.) I generally release a lot of my own software as Debian packages, although I'll admit I've started switching to publishing Perl-based projects on CPAN instead - from which they can be debianized via dh-make-perl. One thing I've not done for many years is a mass-rebuild of Debian packages. I did that once upon a time when I was trying to push for the stack-smashing-protection inclusion all the way back in 2006. Having had a few interesting emails this past week I decided to do the job for real. I picked a random server of mine, rsync.io, which stores backups, and decided to rebuild it using "my own" packages. The host has about 300 packages installed upon it:
root@rsync ~ # dpkg --list   grep ^ii   wc -l
294
I got the source to every package, patched the changelog to bump the version, and rebuild every package from source. That took about three hours. Every package has a "skx1" suffix now, and all the build-dependencies were also determined by magic and rebuilt:
root@rsync ~ # dpkg --list   grep ^ii   awk '  print $2 " " $3 '  head -n 4
acpi 1.6-1skx1
acpi-support-base 0.140-5+deb7u3skx1
acpid 1:2.0.16-1+deb7u1skx1
adduser 3.113+nmu3skx1
The process was pretty quick once I started getting more and more of the packages built. The only shortcut was not explicitly updating the dependencies to rely upon my updages. For example bash has a Debian control file that contains:
Depends: base-files (>= 2.1.12), debianutils (>= 2.15)
That should have been updated to say:
Depends: base-files (>= 2.1.12skx1), debianutils (>= 2.15skx1)
However I didn't do that, because I suspect if I did want to do this decently, and I wanted to share the source-trees, and the generated packages, the way to go would not be messing about with Debian versions instead I'd create a new Debian release "alpha-apple", "beta-bananna", "crunchy-carrot", "dying-dragonfruit", "easy-elderberry", or similar. In conclusion: Importing Debian packages into git, much like Ubuntu did with bzr, is a fun project, and it doesn't take much to mass-rebuild if you're not making huge changes. Whether it is worth doing is an entirely different question of course.

Stefano Zacchiroli: Thoughts on the Init System Coupling GR

on perceived hysteria and silent sanity As you probably already know by now, the results of the Debian init system coupling general resolution (GR) look like this:
Init system coupling GR: results (arrow from A to B means that voters preferred A to B by that margin)
results of the init system coupling GR
Some random thoughts about them: My take home message is that we are experiencing a huge gap between the public perception of the state of Debian (both from within and from without the project) and the actual beliefs of the silent majority of people that make Debian with their work, day after day. In part this is old news. The most "senior" members of the project will remember that the topic of "vocal minorities vs silent majority" was a recurrent one in Debian 10+ years ago, when flames were periodically ravaging the project. Since then Debian has grown a lot though, and we are now part of a much larger and varied ecosystem. We are now at a scale at which there are plenty of FOSS "mass-media" covering daily what happens in Debian, inducing feedback loops with our own perception of ourselves which we do not fully grok yet. This is a new factor in the perception gap. This situation is not intrinsically bad, nor there is blame to assign here: after all influential bloggers, news sites, etc., just do their job. And their attention also testifies of the huge interest that there is around Debian and our choices. But we still need to adapt and learn to take perceived hysteria with a pinch (or two) of salt. It might just be time for our decennial check-up. Time to remind ourselves that our ways of doing things might in fact still be much more sane than sometimes we tend to believe. We went on 10+ years ago, after monumental flames. It looks like we are now ready to move on again, putting The Era of the Great systemd Histeria behind us.

Matthew Palmer: Multi-level prefix delegation is not a myth! I've seen it!

Unless you ve been living under a firewalled rock, you know that IPv6 is coming. There s also a good chance that you ve heard that IPv6 doesn t have NAT. Or, if you pay close attention to the minutiae of IPv6 development, you ve heard that IPv6 does have NAT, but you don t have to (and shouldn t) use it. So let s say we ll skip NAT for IPv6. Fair enough. However, let s say you have this use case:
  1. A bunch of containers that need Internet access
  2. That are running in a VM
  3. On your laptop
  4. Behind your home router!
For IPv4, you d just layer on the NAT, right? While SIP and IPsec might have kittens trying to work through three layers of NAT, for most things it ll Just Work. In the Grand Future of IPv6, without NAT, how the hell do you make that happen? The answer is Prefix Delegation , which allows routers to delegate management of a chunk of address space to downstream routers, and allow those downstream routers to, in turn, delegate pieces of that chunk to downstream routers. In the case of our not-so-hypothetical containers-in-VM-on-laptop-at-home scenario, it would look like this:
  1. My border router (a DNS-323 running Debian) asks my ISP for a delegated prefix, using DHCPv6. The ISP delegates a /561. One /64 out of that is allocated to the network directly attached to the internal interface, and the rest goes into the pool , as /60 blocks (so I ve got 15 of them to delegate, if required).
  2. My laptop gets an address on the LAN between itself and the DNS-323 via stateless auto-addressing ( SLAAC ). It also uses DHCPv6 to request one of the /60 blocks from the DNS-323. The laptop puts one /64 from that block as the address space for the virtual LAN (actually a Linux bridge) that connects the laptop to all my VMs, and puts the other 15 /64 blocks into a pool for delegation.
  3. The VM that will be running the set of containers under test gets an address on the all VMs virtual LAN via SLAAC, and then requests a delegated /64 to use for the all containers virtual LAN (another bridge, this one running on the VM itself) that the containers will each connect to themselves.
Now, almost all of this Just Works. The current releases of ISC DHCP support prefix delegation just fine, and a bit of shell script plumbing between the client and server seals the deal the client needs to rewrite the server s config file to tell it the netblock from which it can delegate. Except for one teensy, tiny problem routing. When the DHCP server delegates a netblock to a particular machine, the routing table needs to get updated so that packets going to that netblock actually get sent to the machine the netblock was delegated to. Without that, traffic destined for the containers (or the VM) won t actually make it to its destination, and a one-way Internet connection isn t a whole lot of use. I cannot understand why this problem hasn t been tripped over before. It s absolutely fundamental to the correct operation of the delegation system. Some people advocate running a dynamic routing protocol, but that s a sledgehammer to crack a nut if ever I saw one. Actually, I know this problem has been tripped over before, by OpenWrt. Their solution, however, was to use a PHP script to scan logfiles and add routes. Suffice it to say, that wasn t an option I was keen on exploring. Instead, I decided to patch ISC DHCP so that the server can run an external script to add the necessary routes, and perhaps modify firewall rules and also to reverse the process when the delegation is released (or expired). If anyone else wants to play around with it, I ve put it up on Github. I don t make any promises that it s the right way to do it, necessarily, but it works, and the script I ve added in contrib/prefix-delegation-routing.rb shows how it can be used to good effect. By the way, if anyone knows how pull requests work over at ISC, drop me a line. From the look of their website, they don t appear to accept (or at least encourage) external contributions. So, that s one small patch for DHCP, one giant leap for my home network.
  1. The standard recommendation is for ISPs to delegate each end-user customer a /48 (giving 65,536 /64 networks); my ISP is being a little conservative in only giving me 256 /64s. It works fine for my purposes, but if you re an ISP getting set for deploying IPv6, make life easy on your customers and give them a /48.

19 November 2014

Jonathan Dowland: Moving to Red Hat

I'm changing jobs! From February 2015, I will be joining Red Hat as a Senior Software Engineer. I'll be based in Newcastle and working with the Middleware team. I'm going to be working with virtualisation, containers and Docker in particular. I know a few of the folks in the Newcastle office already, thanks to their relationship with the School of Computing Science, and I'm very excited to work with them, as well as the wider company. It's also going to be great to be contributing to the free software community as part of my day job. This October marked my tenth year working for Newcastle University. I've had a great time, learned a huge amount, and made some great friends. It's going to be sad to leave, especially the School of Computing Science where I've spent the last four years, but it's the right time to move on, It's an area that I've been personally interested in for a long time and I'm very excited to be trying something new.

18 November 2014

Laura Arjona: Translating (reviewing) Debian package descriptions

Some days I feel super lazy but I still would like to go on contributing translations to Debian.
Then, I leave the web translations a bit, and change to translate or review Debian package descriptions. It s something that anybody can do without any knowledge of translation tools, since it is a very simple web interface, as you will see. First you need to create a login account, then, login into the system. And then, go to the page of your mother language (in my case, Spanish, es ). You will see some introductory text, and the list of pending translations:
ddtss_es1
At the end of the page, there is the list of translations pending to review:
ddtss_es2 We should begin with this, so the work that other people already made arrives quickly its destination. And it s the easiest part, as you will see. Let s pick one of them (libvformat1-dev): review1
You see the short description in the original English, and the current translation (if there were changes from a former version, they are coloured too). I didn t know what the package libvformat1-dev does, but here s a nice opportunity to learn aobut it a bit :) The short description looks ok for me. Let s go on to the long description: review2 It also looks correct for me. So I leave the text box as is, and go on until the bottom of the page:
review3
and click Accept as is . That s all!! The system brings you back to the page with pending translations and reviews. Let s pick another one: totem
review4
I found a typo and corrected some other words, so I updated the text in the translation box, left a message to the other translators in the comment box, and clicked Accept with changes . And iterate. When 3 translators agree in a translation, it becomes official, and its propagated to apt-cache, aptitude, synaptic, etc., and the website (packages.debian.org). This is the most difficult part (to get 3 reviews for each package description): many language teams are small, and their workforce is spread in many fronts: translations for the website, news and announcements, debconf templates (the messages that are shown to the user when a package is installed), the Debian installer, the documentation, the package descriptions So your help (even when you only review some translations from time to time) will be appreciated, for sure.
Filed under: Tools Tagged: Contributing to libre software, Debian, English, translations

Michal &#268;iha&#345;: Mercurial support in Weblate

Weblate has started as a translation system tightly bound to Git version control system. This was in no means design decision, but rather it was the version control I've used. But this has shown not to be sufficient and other systems were requested as well. And Mercurial is first of them to be supported. Weblate 2.0 already had separated VCS layer and adding another system to that is quite easy if you know the VCS you're adding. Unfortunately this wasn't the case for me with Mercurial as I've never used it for anything more serious than cloning a repository, committing fixes and pushing it back. Weblate needs a bit more than that, especially in regard to remote branches. But nevertheless I've figured out all operations and the implementation is ready in our Git. In case somebody is interested in adding support for another version control, patches are always welcome!

Filed under: English phpMyAdmin SUSE Weblate 0 comments Flattr this!

Next.

Previous.