Instead of a calculator, I tend to use IPython for those quotidian bits of "mental" arithmetic: However, I often forget to actually start IPython, resulting in me running the following in my shell: Whilst I could learn do this maths within Zsh itself, I would prefer to dump myself into IPython instead being able to use "_" and Python modules generally is just too useful. After following this pattern too many times, I put together the following snippet that will detect whether I have prematurely attempted a calculation inside zsh and pretend that I ran it in IPython all along: For example: (Canonical version from my zshrc.d)

Munin is a great tool. If you can script it, you can monitor it with munin. Unfortunately, however, munin is slow; that is, it will take snapshots once every five minutes, and not look at systems in between. If you have a short load spike that takes just a few seconds, chances are pretty high munin missed it. It also comes with a great webinterfacefrontendthing that allows you to dig deep in the history of what you've been monitoring. By the time munin tells you that your Kerberos KDCs are all down, you've probably had each of your users call you several times to tell you that they can't log in. You could use nagios or one of its brethren, but it takes about a minute before such tools will notice these things, too. Maybe use CollectD then? Rather than check once every several minutes, CollectD will collect information every few seconds. Unfortunately, however, due to the performance requirements to accomplish that (without causing undue server load), writing scripts for CollectD is not as easy as it is for Munin. In addition, webinterfacefrontendthings aren't really part of the CollectD code (there are several, but most that I've looked at are lacking in some respect), so usually if you're using CollectD, you're missing out some. And collectd doesn't do the nagios thing of actually telling you when things go down. So what if you could see it when things go bad? At one customer, I came in contact with Frank, who wrote ExtreMon, an amazing tool that allows you to visualize the CollectD output as things are happening, in a full-screen fully customizable visualization of the data. The problem is that ExtreMon is rather... complex to set up. When I tried to talk Frank into helping me getting things set up for myself so I could play with it, I got a reply along the lines of...

well, extremon requires a lot of work right now... I really want to fix foo and bar and quux before I start documenting things. Oh, and there's also that part which is a dead end, really. Ask me in a few months?

which is fair enough (I can't argue with some things being suboptimal), but the code exists, and (as I can see every day at $CUSTOMER) actually works. So I decided to just figure it out by myself. After all, it's free software, so if it doesn't work I can just read the censored code. As the manual explains, ExtreMon is a plugin-based system; plugins can add information to the "coven", read information from it, or both. A typical setup will run several of them; e.g., you'd have the from_collectd plugin (which parses the binary network protocol used by collectd) to get raw data into the coven; you'd run several aggregator plugins (which take that raw data and interpret it, allowing you do express things along the lines of "if the system's load gets above X, set load.status to warning"; and you'd run at least one output plugin so that you can actually see the damn data somewhere. While setting up ExtreMon as is isn't as easy as one would like, I did manage to get it to work. Here's what I had to do. You will need:

• A monitor with a FullHD (or better) resolution. Currently, the display frontend of ExtreMon assumes it has a FullHD display at all time. Even if you have a lower resolution. Or a higher one.
• Python3
• OpenJDK 6 (or better)

First, we clone the ExtreMon git repository:

git clone https://github.com/m4rienf/ExtreMon.git extremon
cd extremon

There's a README there which explains the bare necessities on getting the coven to work. Read it. Do what it says. It's not wrong. It's not entirely complete, though; it fails to mention that you need to

• install CollectD (which is required for its types.db)
• Configure CollectD to have a line like Hostname "com.example.myhost" rather than the (usual) FQDNLookup true. This is because extremon uses the java-style reverse hostname, rather than the internet-style FQDN.

Make sure the dump.py script outputs something from collectd. You'll know when it shows something not containing "plugin" or "plugins" in the name. If it doesn't, fiddle with the #x3. lines at the top of the from_collectd file until it does. Note that ExtreMon uses inotify to detect whether a plugin has been added to or modified in its plugins directory; so you don't need to do anything special when updating things. Next, we build the java libraries (which we'll need for the display thing later on):

cd java/extremon
mvn install
cd ../client/
mvn install

This will download half the Internet, build some java sources, and drop the precompiled .jar files in your $HOME/.m2/repository. We'll now build the display frontend. This is maintained in a separate repository:

cd ../..
git clone https://github.com/m4rienf/ExtreMon-Display.git display
cd display
mvn install

This will download the other half of the Internet, and then fail, because Frank forgot to add a few repositories. Patch (and push request) on github With that patch, it will build, but things will still fail when trying to sign a .jar file. I know of four ways on how to fix that particular problem:

1. Add your passphrase for your java keystore, in cleartext, to the pom.xml file. This is a terrible idea.
2. Pass your passphrase to maven, in cleartext, by using some command line flags. This is not much better.
3. Ensure you use the maven-jarsigner-plugin 1.3.something or above, and figure out how the maven encrypted passphrase store thing works. I failed at that.
4. Give up on trying to have maven sign your jar file, and do it manually. It's not that hard, after all.

If you're going with 1 through 3, you're on your own. For the last option, however, here's what you do. First, you need a key:

keytool -genkeypair -alias extremontest

after you enter all the information that keytool will ask for, it will generate a self-signed code signing certificate, valid for six months, called extremontest. Producing a code signing certificate with longer validity and/or one which is signed by an actual CA is left as an exercise to the reader. Now, we will sign the .jar file:

jarsigner target/extremon-console-1.0-SNAPSHOT.jar extremontest

There. Who needs help from the internet to sign a .jar file? Well, apart from this blog post, of course. You will now want to copy your freshly-signed .jar file to a location served by HTTPS. Yes, HTTPS, not HTTP; ExtreMon-Display will fail on plain HTTP sites. Download this SVG file, and open it in an editor. Find all references to be.grep as well as those to barbershop and replace them with your own prefix and hostname. Store it along with the .jar file in a useful directory. Download this JNLP file, and store it on the same location (or you might want to actually open it with "javaws" to see the very basic animated idleness of my system). Open it in an editor, and replace any references to barbershop.grep.be by the location where you've stored your signed .jar file. Add the chalice_in_http plugin from the plugins directory. Make sure to configure it correctly (by way of its first few comment lines) so that its input and output filters are set up right. Add the configuration snippet in section 2.1.3 of the manual (or something functionally equivalent) to your webserver's configuration. Make sure to have authentication chalice_in_http is an input mechanism. Add the chalice_out_http plugin from the plugins directory. Make sure to configure it correctly (by way of its first few comment lines) so that its input and output filters are set up right. Add the configuration snippet in section 2.2.1 of the manual (or something functionally equivalent) to your webserver's configuration. Authentication isn't strictly required for the output plugin, but you might wish for it anyway if you care whether the whole internet can see your monitoring. Now run javaws https://url/x3console.jnlp to start Extremon-Display. At this point, I got stuck for several hours. Whenever I tried to run x3mon, this java webstart thing would tell me simply that things failed. When clicking on the "Details" button, I would find an error message along the lines of "Could not connect (name must not be null)". It would appear that the Java people believe this to be a proper error message for a fairly large number of constraints, all of which are slightly related to TLS connectivity. No, it's not the keystore. No, it's not an API issue, either. Or any of the loads of other rabbit holes that I dug myself in. Instead, you should simply make sure you have Server Name Indication enabled. If you don't, the defaults in Java will cause it to refuse to even try to talk to your webserver. The ExtreMon github repository comes with a bunch of extra plugins; some are special-case for the place where I first learned about it (and should therefore probably be considered "examples"), others are general-purpose plugins which implement things like "is the system load within reasonable limits". Be sure to check them out. Note also that while you'll probably be getting most of your data from CollectD, you don't actually need to do that; you can write your own plugins, completely bypassing collectd. Indeed, the from_collectd thing we talked about earlier is, simply, also a plugin. At $CUSTOMER, for instance, we have one plugin which simply downloads a file every so often and checks it against a checksum, to verify that a particular piece of nonlinear software hasn't gone astray yet again. That doesn't need collectd. The example above will get you a small white bar, the width of which is defined by the cpu "idle" statistic, as reported by CollectD. You probably want more. The manual (chapter 4, specifically) explains how to do that. Unfortunately, in order for things to work right, you need to pretty much manually create an SVG file with a fairly strict structure. This is the one thing which Frank tells me is a dead end and needs to be pretty much rewritten. If you don't feel like spending several days manually drawing a schematic representation of your network, you probably want to wait until Frank's finished. If you don't mind, or if you're like me and you're impatient, you'll be happy to know that you can use inkscape to make the SVG file. You'll just have to use dialog behind ctrl+shift+X. A lot. Once you've done that though, you can see when your server is down. Like, now. Before your customers call you.

Radicale and DAVDroid radicale and DAVdroid appeal to me. Let's try to make the whole thing work. A self-signed SSL certificate Generating the certificate:

    openssl req -nodes -x509 -newkey rsa:2048 -keyout cal-key.pem -out cal-cert.pem -days 3650
    [...]
    Country Name (2 letter code) [AU]:IT
    State or Province Name (full name) [Some-State]:Bologna
    Locality Name (eg, city) []:
    Organization Name (eg, company) [Internet Widgits Pty Ltd]:enricozini.org
    Organizational Unit Name (eg, section) []:
    Common Name (e.g. server FQDN or YOUR name) []:cal.enricozini.org
    Email Address []:postmaster@enricozini.org

Installing it on my phone:

    openssl x509 -in cal-cert.pem -outform DER -out cal-cert.crt
    adb push cal-cert.crt /mnt/sdcard/
    enrico --follow-instructions http://davdroid.bitfire.at/faq/entry/importing-a-certificate

Installing radicale in my VPS An updated radicale package, with this patch to make it work with DAVDroid:

    apt-get source radicale
    # I reviewed 063f7de7a2c7c50de5fe3f8382358f9a1124fbb6
    git clone https://github.com/Kozea/Radicale.git
    Move the python code from git to the Debian source
    dch -v 0.10~enrico  "Pulled in the not yet released 0.10 work from upstream"
    debuild -us -uc -rfakeroot

Install the package:

    # dpkg -i python-radicale_0.10~enrico0-1_all.deb
    # dpkg -i radicale_0.10~enrico0-1_all.deb

Create a system user to run it:

    # adduser --system --disabled-password radicale

Configure it for mod_wsgi with auth done by Apache:

    # For brevity, this is my config file with comments removed
    [storage]
    # Storage backend
    # Value: filesystem   multifilesystem   database   custom
    type = filesystem
    # Folder for storing local collections, created if not present
    filesystem_folder = /var/lib/radicale/collections
    [logging]
    config = /etc/radicale/logging

Create the wsgi file to run it:

    # mkdir /srv/radicale
    # cat <<EOT > /srv/radicale/radicale.wsgi
    import radicale
    radicale.log.start()
    application = radicale.Application()
    EOT
    # chown radicale.radicale /srv/radicale/radicale.wsgi
    # chmod 0755 /srv/radicale/radicale.wsgi

Make radicale commit to git

    # apt-get install python-dulwich
    # cd /var/lib/radicale/collections
    # git init
    # chown radicale.radicale -R /var/lib/radicale/collections/.git

Apache configuration Add a new site to apache:

    $ cat /etc/apache2/sites-available/cal.conf
    # For brevity, this is my config file with comments removed
    <IfModule mod_ssl.c>
    <VirtualHost *:443>
            ServerName cal.enricozini.org
            ServerAdmin enrico@enricozini.org
            Alias /robots.txt /srv/radicale/robots.txt
            Alias /favicon.ico /srv/radicale/favicon.ico
            WSGIDaemonProcess radicale user=radicale group=radicale threads=1 umask=0027 display-name=% GROUP 
            WSGIProcessGroup radicale
            WSGIScriptAlias / /srv/radicale/radicale.wsgi
            <Directory /srv/radicale>
                    # WSGIProcessGroup radicale
                    # WSGIApplicationGroup radicale
                    # WSGIPassAuthorization On
                    AllowOverride None
                    Require all granted
            </Directory>
            <Location />
                    AuthType basic
                    AuthName "Enrico's Calendar"
                    AuthBasicProvider file
                    AuthUserFile /usr/local/etc/radicale/htpasswd
                    Require user enrico
            </Location>
            ErrorLog APACHE_LOG_DIR /cal-enricozini-org-error.log
            LogLevel warn
            CustomLog APACHE_LOG_DIR /cal-enricozini-org-access.log combined
            SSLEngine on
            SSLCertificateFile    /etc/ssl/certs/cal.pem
            SSLCertificateKeyFile /etc/ssl/private/cal.key
    </VirtualHost>
    </IfModule>

Then enable it:

    # a2ensite cal.conf
    # service apache2 reload

Create collections DAVdroid seems to want to see existing collections on the server, so we create them:

    $ apt-get install cadaver
    $ cat <<EOT > /tmp/empty.ics
    BEGIN:VCALENDAR
    VERSION:2.0
    END:VCALENDAR
    EOT
    $ cat <<EOT > /tmp/empty.vcf
    BEGIN:VCARD
    VERSION:2.1
    END:VCARD
    EOT
    $ cadaver https://cal.enricozini.org
    WARNING: Untrusted server certificate presented for  cal.enricozini.org':
    [...]
    Do you wish to accept the certificate? (y/n) y
    Authentication required for Enrico's Calendar on server  cal.enricozini.org':
    Username: enrico
    Password: ****
    dav:/> cd enrico/contacts.vcf/
    dav:/> put /tmp/empty.vcf
    dav:/> cd ../calendar.ics/
    dav:/> put /tmp/empty.ics
    dav:/enrico/calendar.ics/> ^D
    Connection to  cal.enricozini.org' closed.

DAVdroid configuration

1. Add a new DAVdroid sync account
2. Use server/username configuration
3. For server, use https:////
4. Add username and password

It should work. Related links

• http://radicale.org/
• https://github.com/Kozea/Radicale
• http://a3nm.net/blog/davdroid.html

I was recently asked about some construct in a shell script starting with a colon(:), leading me into a long monologue about it. Afterwards I realized I had forgotten to mention half of the nice things. So here for your amusement some usage for the colon in the shell: To find the meaning of ":" in the bash manpage[1], you have to look at the start of the SHELL BUILTIN COMMANDS section. There you find:

: [arguments]
	No effect; the command does nothing beyond expanding arguments and performing any specified redirections.  A zero exit code is returned.

If you wonder what the difference to true is: I don't know any difference (except that there is no /bin/:) So what is the colon useful for? You can use it if you need a command that does nothing, but still is a command.

• For example, if you want to avoid using a negation (for fear of history expansion still being on by default on a interactive bash or wanting to support ancient shells), you cannot simply write

  if conditon ; then
  	# this will be an error
  else
  	echo condition is false
  fi
  

  but need some command there, for which the colon can be used:

  if conditon ; then
  	: # nothing to do in this case
  else
  	echo condition is false
  fi
  

  To confuse your reader, you can use the fact that the colon ignores it's arguments and you only have normal words there:

  if conditon ; then
  	: nothing to do in this case # <- this works but is not good style
  else
  	echo condition is false
  fi
  

  though I strongly recommend against it (exercise: why did I use a # there for my remark?).
• This of course also works in other cases:

  while processnext ; do
  	:
  done
  

• The ability to ignore the actual arguments (but still processing them as with every command that ignores it arguments) can also be used, like in:

  : $ VARNAME:=default 
  

  which sets VARNAME to a default if unset or empty. (One could also use that the first time it is used, or $ VARNAME:-default everywhere, but this can be more readable).
• In other cases you do not strictly need a command, but using the colon can clear things up, like creating or truncating a file using a redirection:

  : > /path/to/file
  

Then there is more things you can do with the colon, most I'd put under "abuse":

• misuing it for comments:

  : ====== here
  

  While it has the advantage of also showing up in -x output, the to be expected confusion of the reader and the danger of using any shell active character makes this general a bad idea.
• As it practically the same as true it can be used as a shorter form of true. Given that true is more readable that is a bad idea. (At least it isn't as evil as using the empty string to denote true.)

  # bad style!
  if condition ; then doit= ; doit2=: ; else doit=false ; doit2=false ; fi
  if $doit ; then echo condition true ; fi
  if $doit2 && true ; then echo condition true ; fi
  

• Another way to scare people:

  ignoreornot=
  $ignoreornot echo This you can see.
  ignoreornot=:
  $ignoreornot echo This you cannot see.
  

  While it works, I recommend against it: Easily confusing and any > in there or $(...) will likely rain harvoc over you.
• Last and least, one can shadow the built-in colon with a different one. Only useful for obfuscation, and thus likely always evil. :() :&: ;: anyone?

This is of course not a complete list. But unless I missed something else, those are the most common cases I run into. [1] <rant>If you never looked at it, better don't start: the bash manpage is legendary for being quite useless as hiding all information in other information in a quite absurd order. Unless you look at documentation about how to write a shell script parser, then the bash manpage is really what you want to read.</rant>

Rhoda is guarded. She is secretly in love with her brother. When he gets a girlfriend, she finds a boyfriend. She tells no one what she's really thinking. Rhoda likes to seize the day. In the midst of evening conversation, she will excuse herself to use the bathroom or come right back , then, within literally two minutes, she will go home with an acquaintance or stranger. Often these encounters or the aftermaths thereof do not go according to her liking, and she will make veiled remarks of hostility, should she see those people again. Rhoda was unhappy so she changed everything in her life. Though multiple things remained constant, she concluded that she was, in fact, the problem. Rhoda does not make enough money on which to live. She works part-time, and turns down all other job offers. Other people make up for her financial shortcomings so that she is not homeless and starving. Rhoda is certain that it is more difficult to be female than male.

I do hang out in #debian-women on IRC, which shouldn't be much of a surprise after my last blog entry about my Feminist Year. And for readers of my blog it also shouldn't be much of a surprise that Music is an important part of my life. Recently a colleague from Debian though asked me in said IRC channel about whether I can recommend some female artists or bands. Which got me looking through my recommendations so far, and actually, there weren't many of those in here, unfortunately. So I definitely want to work on that because there are so many female singers, songwriters and bands out there that I totally would like to share with the broader audience. I want to start out with a strong female voice who was introduced to me by another strong woman thanks for that! Fiona Apple definitely has her own style and is something special, she stands out. Here are my suggestions:

• Hot Knife: This was the song I was introduced to her with. And I love the kettledrum rhythm and sound.
• Criminal: Definitely a different sound, but it was the song that won her a Grammy.
• Not About Love: Such a lovely composition. I do love the way she plays the piano.

Like always, enjoy!

/music permanent link Comments: 0 Flattr this

The smell of email This was written in response to a message with a list of demotivating behaviours in email interactions, like fingerpointing, aggressiveness, resistance when being called out for misbehaving, public humiliation for mistakes, and so on There are times when I stumble on an instance of the set of things that were mentioned, and I think "ok, today I feel like doing some paid work rather than working on Debian". If another day I wake up deciding to enjoy working on Debian, which I greatly do, I try and make sure that I can focus on bits of Debian where I don't stumble on any instances of the set of things that were mentioned. Then I stumble on Gregor's GDAC and I feel like I'd happily lose one day of pay right now, and have fun with Debian. I feel like Debian is this big open kitchen populated by a lot of people:

• some dump shit
• some poke the shit with a stick, contributing to the spread of the smell
• some carefully clean up the shit, which in the short term still contributes to the smell, but makes things better in the long term
• some prepare and cook, making a nice smell of food and NOMs
• some try out the food and tell us how good it was

I have fun cooking and tring out the food. I have fun being around people who cook and try out the food. The fun in the kitchen seems to be correlated to several things, one of which is that it seems to be inversely proportional to the stink. I find this metaphore interesting, and I will start thinking about the smell of a mailing list post. I expect it should put posts into perspective, I expect I will develop an instinct for it, so that I won't give a stinky post the same importance of a post that smells of food. I also expect that the more I learn to tell the smell of food from the smell of shit, the more I can help cleaning it, and the more I can help telling people who repeatedly contribute to the stink to please try cooking instead, or failing that, just try and stay out of the kitchen.

My last problem with BTRFS was in August [1]. BTRFS has been running mostly uneventfully for me for the last 4 months, that s a good improvement but the fact that 4 months of no problems is noteworthy for something as important as a filesystem is a cause for ongoing concern. A RAID-1 Array A week ago I had a minor problem with my home file server, one of the 3TB disks in the BTRFS RAID-1 started giving read errors. That s not a big deal, I bought a new disk and did a btrfs replace operation which was quick and easy. The first annoyance was that the output of btrfs device stats reported an error count for the new device, it seems that btrfs device replace copies everything from the old disk including the error count. The solution is to use btrfs device stats -z to reset the count after replacing a device. I replaced the 3TB disk with a 4TB disk (with current prices it doesn t make sense to buy a new 3TB disk). As I was running low on disk space I added a 1TB disk to give it 4TB of RAID-1 capacity, one of the nice features of BTRFS is that a RAID-1 filesystem can support any combination of disks and use them to store 2 copies of every block of data. I started running a btrfs balance to get BTRFS to try and use all the space before learning from the mailing list that I should have done btrfs filesystem resize to make it use all the space. So my balance operation had configured the filesystem to configure itself for 2*3TB+1*1TB disks which wasn t the right configuration when the 4TB disk was fully used. To make it even more annoying the btrfs filesystem resize command takes a devid not a device name. I think that when BTRFS is more stable it would be good to have the btrfs utility warn the user about such potential mistakes. When a replacement device is larger than the old one it will be very common to want to use that space. The btrfs utility could easily suggest the most likely btrfs filesystem resize to make things easier for the user. In a disturbing coincidence a few days after replacing the first 3TB disk the other 3TB disk started giving read errors. So I replaced the second 3TB disk with a 4TB disk and removed the 1TB disk to give a 4TB RAID-1 array. This is when it would be handy to have the metadata duplication feature and copies= option of ZFS. Ctree Corruption 2 weeks ago a basic workstation with a 120G SSD owned by a relative stopped booting, the most significant errors it gave were BTRFS: log replay required on RO media and BTRFS: open_ctree failed . The solution to this is to run the command btrfs-zero-log , but that initially didn t work. I restored the system from a backup (which was 2 months old) and took the SSD home to work on it. A day later btrfs-zero-log worked correctly and I recovered all the data. Note that I didn t even try mounting the filesystem in question read-write, I mounted it read-only to copy all the data off. While in theory the filesystem should have been OK I didn t have a need to keep using it at that time (having already wiped the original device and restored from backup) and I don t have confidence in BTRFS working correctly in that situation. While it was nice to get all the data back it s a concern when commands don t operate consistently. Debian and BTRFS I was concerned when the Debian kernel team chose 3.16 as the kernel for Jessie (the next Debian release). Judging by the way development has been going I wasn t confident that 3.16 would turn out to be stable enough for BTRFS. But 3.16 is working reasonably well on a number of systems so it seems that it s likely to work well in practice. But I m still deploying more ZFS servers. The Value of Anecdotal Evidence When evaluating software based on reports from reliable sources (IE most readers will trust me to run systems well and only report genuine bugs) bad reports have a much higher weight than good reports. The fact that I ve seen kernel 3.16 to work reasonably well on ~6 systems is nice but that doesn t mean it will work well on thousands of other systems although it does indicate that it will work well on more systems than some earlier Linux kernels which had common BTRFS failures. But the annoyances I had with the 3TB array are repeatable and will annoy many other people. The ctree coruption problem MIGHT have been initially caused by a memory error (it s a desktop machine without ECC RAM) but the recovery process was problematic and other users might expect problems in such situations.

• [1] http://etbe.coker.com.au/2014/08/01/more-btrfs-fun/

My first plan was to participate in the Bug Squashing Party (BSP) in Paris, but finally I didn t manage to organize the travel and proper stay in the capital of France. Finally I settled on Munich and below you will find a summary of my stay there. While organizing my trip, the Munich natives offered me a couch to surf during the stay, which was very kind. In the end, however, I stayed at my friend s place - thanks a lot Mehdi! The BSP gathered Debian people interested in preparing Jessie release, but also KDE, Kolab and LibreOffice people. The host for the meeting was LiMux (on of the most known large-scale Linux deployments) which provided a venue, but also food (free!) and other attractions. Organizers, if you read this, I thank you for the amazing work that you have done. I arrived on late Friday s evening and so I didn t participate that day in bug squashing and took a good sleep instead. On Saturday and Sunday we were, well, squashing bugs. I haven t done NMUs before and it was a new experience to me, even though theoretically I know how it works. The DDs present at the BSP were more than helpful to help me when I was stuck. Here is a short description of what I have done:

• #770085 - This is a bug I reported myself before BSP. I picked it as a low-hanging fruit and I had it fixed after 2 hours. It fixes a Python module that couldn t be imported in Python 2.
• #768690 - This was a tricky one. latex-mk depends on tgif which is not going in jessie. A lazy workaround I made in the patch is the disable tgif-related functionality completely. It is not essential to the package, so it should be ok (a shameless publicity here: I use and recommend latex-make).
• #768615 - A simple one - the package testsuite is tightly wired with pygments version (actually, the package is a ruby wrapper around pygments) so it had to be fixed to reflect the version in jessie.
• #768695 - I ve spent nearly a whole day understanding and writing a patch. Long story short - statsmodels uses numpy, pandas and scipy, and its testsuite depends a lot of interfaces of these libraries. For example, it relies on pandas to have DateRange class which is not true for the version in jessie. We uploaded the patch which fixed most of the problems, but the build still failed on i386 due to some precision in floating point operations. The new patch was prepared and awaits upload.
• #768673 - I ve traced the problem to the recent POODLE attack (Wikipedia). Ruby-httpclient uses SSLv3 by default which is deprecated and disabled on the server. The SSL negotiation ends with a nasty ECONNRESET. This problem has been fixed in the newer version of the library so I just backport in my NMU. Note: I suspect that #770616 is related.
• #768905 - keyutils testsuite breaks on newer versions of the kernel (i.e., > 3.16 which is in Jessie). The bug remains, but by testing we confirmed that it won t affect jessie.

I also got my GPG keys signed and vice-versa, so I m much better connected to the web of trust now (7 signatures of DDs so far). All in all, it was a great experience which I recommend to everyone. The atmosphere is very motivating and there are many people who will gladly help you if you don t know or understand something. You don t have to be DD to participate even, but a minimal packaging experience is definitely very useful. I m quite sure that people would be glad to help even a complete newcomer (I certainly would do), but being able to build a package using apt-get source and debuild won t hurt anybody. At the very least you can always triage bugs: reproduce them, find the cause of the bug, propose a way to fix it, etc. As always you should be open-minded and willing to get your hands dirty. I want also thank Debian for sponsoring me which finally convinced me to participate (Munich is around 500 km from where I live and the travel is fairly expensive). Some links of interest:

• Terse report of the BSP
• Website of the BSP
• LiMux
• Squash (not really related, but still interesting)

Many bugs still wait to be squashed (120 at the time of writing this), so let s get back to work!

(If I've linked you to this page, it is my feeble attempt to provide a more convincing justification.)

I often receive instant messages or emails requesting help or guidance at work or on one of my various programming projects. When asked why they asked privately, the responses vary; mostly along the lines of it simply being an accident, not knowing where else to ask, as well as not wishing to "disturb" others with their bespoke question. Some will be more candid and simply admit that they were afraid of looking unknowledgable in front of others. It is always tempting to simply reply with the answer, especially as helping another human is inherently rewarding unless one is a psychopath. However, one can actually do more good overall by insisting the the question is re-asked in a more public forum.

---

This is for many reasons. Most obviously, public questions are simply far more efficient as soon as more than one person asks that question the response can be found in a search engine or linked to in the future. These time savings soon add up, meaning that simply more stuff can be done in any given day. After all, most questions are not as unique as people think. Secondly, a private communication cannot be corrected or elaborated on if someone else notices it is incorrect or incomplete. Even this rather banal point is more subtle that it first appears the lack of possible corrections deprives both the person asking and the person responding of the true and correct answer. Lastly, conversations that happen in private are depriving others of the answer as well. Perhaps someone was curious but hadn't got around to asking? Maybe the answer or even the question! contains a clue to solving some other issue. None of this can happen if this is occurs behind closed doors. (There are lots of subtler reasons too in a large organisation or team, simply knowing what other people are curious about can be curiously valuable information.)

---

Note that this is not as you might immediately suspect simply a way of ensuring that one gets the public recognition or "kudos" from being seen helping others. I wouldn't deny that technical communities work on a gift economy basis to some degree, but to attribute all acts of assistance as "selfish" and value-extracting would be to take the argument too far in the other direction. Saying that, the lure and appeal of public recognition should not be understated and can certainly provide an incentive to elaborate and provide a generally superior response. More philosophically, there's also something fundamentally "honest" about airing issues in an appropriately public and transparent manner. I feel it promotes a culture of egoless conversations, of being able to admit one's mistakes and ultimately a healthy personal mindset. So please, take care not only in the way you phrase and frame your question, but also consider wider context in which you are asking it. And don't take it too personally if I ask you to re-ask elsewhere...

My LTS November In November I resumed work on Debian LTS and worked on the following packages:

• DLA 88-1 for ruby1.8 fixing several CVEs as described in the announcement.
• DLA 91-1 for tomcat6, mostly prepared by one of it's maintainers, Tony Mancill, also fixing several issues by upgrading to a new upstream version. I just did review, testing, release and announcement, and then figured out the proper versioing for Wheezy, which turned out to be problematic because using the recommend versioning breaks the upgrade paths, when new upstream versions are introdued to squeeze-lts which basically have same version as in will be (or are) in wheezy-security. One cause (besides the wrong recommendation which still needs fixing in our wiki page are non enforcable version constraints: the suites wheezy and squeeze-lts reside on ftp-master.debian.org (and wheezy is only updated on point releases), while wheezy-security resides on security.debian.org. Most probably we will still leave things as they are for squeeze-lts and do changes for wheezy-lts only. Oh, and the release of the tomcat6 update for wheezy is currently stalled by #770769.
• DLA 92-1 for tomcat-native was also done in cooperation with Tony and is also a new upstream release, which is needed as the old version of tomcat-native doesn't function with the new tomcat6 version.
• The 2.6.32.64 update of linux-2.6 has not happened yet, but is planned for the coming weekend. So far it has been done in collaboration of Moritz M hlenhoff from the security team, Ben Hutchings from the kernel team, and Rapha l Hertzog and myself from the LTS team, which I consider to be quite nice. As Rapha l had already explained, Ben has joined the LTS team and so far his contribution was to identify a problem in patch related to openvz, so I haven't published this kernel update yet. Also, there was zero feedback from testers for the openvz flavor packages - so if you are using openvz and squeeze kernels, please contact us. For all the other flavors there was positive feedback to the call for testing (thanks!) - so you might want to give these kernels a try too!

Thanks to everyone who is supporting Squeeze LTS in whatever form, according to the wide feedback there are quite many people appreciating the work! (As this was asked on IRC: if you are a maintainer preparing something for squeeze-lts, that's totally great, thanks!, just please tell us about it as this is the assumption we are working under: if noone tells the LTS team, we think we need to prepare upgrades for squeeze-lts.)

Previously I posted a short article about the WordPress package for Debian and how that SID was getting the updated WordPress 4.0.1 which had some security fixes. The question a lot of people were asking was: What about stable (or Wheezy). After way too much time due to other pressing issues, I have just uploaded the patched WordPress debian package for stable. The fixed version has the catchy number of 3.6.1~deb7u5. This package has all of the relevant patches that went in from WordPress 3.7.4 to 3.7.5 and there are even CVE IDs for this package (and 4.0.1 which all this stems from). Stolen from the 3.6.1 changelog, these are the fixes:

• CVE-2014-9031 XSS in wptexturize() via comments or posts
• CVE-2014-9033 CSRF in the password reset process
• CVE-2014-9034 Denial of service for giant passwords
• CVE-2014-9035 XSS in Press This
• CVE-2014-9036 XSS in HTML filtering of CSS in posts
• CVE-2014-9037 Hash comparison vulnerability in old passwords
• CVE-2014-9038 SSRF: Safe HTTP requests did not sufficiently block the loopback IP address space
• CVE-2014-9039 Email address change didn t invalidate previously sent password reset

I d like to thank the Debian security team especially Salvatore for their assistance and checking the package looked ok. Backporting and Git Part of the delay in getting the wordpress stable package out is that backporting is fiddly. I m currently using pdebuild with a custom pbuilderrc file that points to wheezy. Getting things to that point took a lot of trial and error; with one of the errors being that the pbuilder puts the files in a result directory, not the parent. This also means that the wheezy backports are out of the git repository. I see that there is a git-pbuild but to me it looks like yet another workflow which will slow me right down. Anyone got some good and simple suggestions on having a wheezy track (branch?) and requiring backporting that doesn t get complicated or broken quick? sbuild died in a wave of permission denieds within the chroot.

A new major release of RQuantLib is now on CRAN and getting to Debian. All C++ source files have been rewritten to take advantage of newer Rcpp features. Several Fixed Income functions have been added, or refreshed, thanks to contributions by Michele Salvadore. Calendar support was greatly expanded thanks to a contribution by Danilo Dias da Silva. All key changes are listed in detail below.

Changes in RQuantLib version 0.4.0 (2014-12-01)

• Changes in RQuantLib code:
  • All function interfaces have been rewritten using Rcpp Attributes. No SEXP remain in the function signatures. This make the code shorter, more readable and more easily extensible.
  • The header files have been reorganized so that plugin use is possible. An impl.h files is imported once for each compilation unit: for RQuantLib from the file src/dates.cpp directory, from a sourced file via a #define set by the plugin wrapper.
  • as<>() and wrap() converters have added for QuantLib Date types.
  • Plugin support has been added, allowing more ad-hoc use via Rcpp Attributes.
  • Several Fixed Income functions have been added, and/or rewritten to better match the QuantLib signatures; this was done mostly by Michele Salvadore.
  • Several Date and Calendar functions have been added.
  • Calendar support has been greatly expanded thanks to Danilo Dias da Silva.
  • Exported curve objects are now more parsimonious and advance entries in the table object roughly one business month at a time.
  • The DiscountCurve and Bond curve construction has been fixed via a corrected evaluation date and omitted the two-year swap rate, as suggested by Luigi Ballabio.
  • The NAMESPACE file has a tighter rule for export of *.default functions, as suggested by Bill Dunlap
  • Builds now use OpenMP where available.
  • The package now depends on QuantLib 1.4.0 or later.
• Changes in RQuantLib tests:
  • New unit tests for dates have been added.
  • C++ code for the unit tests has also been converted to Rcpp Attributes use; a helper function unitTestSetup() has been added.
  • Continuous Integration via Travis is now enabled from the GitHub repo.
• Changes in RQuantLib documentation:
  • This NEWS file has been added. Better late than never, as they say.

Courtesy of CRANberries, there is also a diffstat report for the this release. As always, more detailed information is on the RQuantLib page. Questions, comments etc should go to the rquantlib-devel mailing list off the R-Forge page. Issue tickets can be filed at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS This month I have been paid to work 18 hours on Debian LTS (14h allocated by Freexian + 4h I did not spend last month). I did the following tasks:

• CVE triage: I pushed 19 commits to the security tracker. I also tried to encourage some maintainers to provide security updates for packages that are not in use by the current LTS sponsors and that are thus not in our priority list.
• DLA 87: dbus update fixing 3 CVE
• DLA 93: libgcrypt11 update fixing 1 CVE
• DLA 96: openjdk-6 security update fixing 21 CVE
• Worked on preparing a security update to linux. It s not released yet.

Updating the linux source package took a good half of the allocated time. We opted to update the kernel to the upstream version 2.6.32.64. I integrated the upstream patches and identified about 130 patches that we had to disable (because they were already integrated upstream). Then I updated our openvz flavor patch to apply on top of the new kernel. This required quite a bit of manual conflict resolution and there are even parts where I was not sure that I took the correct decision. I was not able to find an upstream openvz git tree on this kernel version to to double check. Instead I asked Ben Hutchings to review my patch. He told me that he did not volunteer to work on LTS, but that he would be open to contribute to it for money. Following this remark, as the coordinator of Freexian s offer, I offered him to join to the set of paid LTS contributors to take care of the kernel and he accepted. So hopefully we will be able to wrap this linux upload in the first week of december. We had no uploads of the kernel in Squeeze since July so it s good to know that we now have someone who will be able to handle it in priority. Distro Tracker No new developments this month. Instead I spent some time to import old historic news so that when you lookup removed packages you have some actual content instead of a 404 error. For example you can look at python2.1. Another thing that I did is to tag some bugs with the newly-announced tag newcomer . Those are easy bugs that are ideal targets for new contributors who d like to get started: here s the list. It s up to you now! DEP-14: Recommended layout for Git packaging repositories I have drafted an initial version of a document called Recommended layout for Git packaging repositories and submitted it for discussion on debian-devel. The discussion has been interesting and constructive (yes this is still possible in Debian!). I have a bunch of improvements in my local copy and needs to process a few more feedback before submitting an updated draft. It s not a revolution but it s a good step to try to standardize tags and branches naming conventions. Systemd, the tech-ctte and our mailing lists As an old-timer, I care a lot about the governance of Debian and it s annoying to see how the systemd debate brought back some of our old daemons in terms of hostile atmosphere on our mailing lists. We can disagree on a lot of things, but we must respect each other and we are here to work together on solutions for everybody. As such I wrote to the persons who cross the line to invite them to behave better. And I m glad that our listmasters are backing up our calls with bans when appropriate. I believe we must go further in that direction and I shared an idea (on a debian-private thread that should have never existed, much like most of the traffic on that list) that I shall formalize and share on debian-project@l.d.o at some point. At the same time, we also had another governance-related discussion with the idea to impose some turnover in the technical committee. I m glad to see that we will soon vote on this topic. This is a good thing in general even though we just had 3 tech-ctte members who retired. Misc stuff I sponsored an upload of galette and of 3 of its plugins. I reviewed jitsi-videobridge and jitsi-meet on mentors.debian.net. I filed a few bugs:

• #768256 about huge vim icons in the GNOME contextual menus
• #768540: cdebootstrap: fails to bootstrap old releases with dpkg not supporting data.tar.xz
• #770011: lynx -dump badly converting &#x2026;

Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

In part 1 and part 2 I investigated how to use ccache with clang. That was more than three years ago. I got an email the other day that ccache bug 8118, which I filed while writing part 1, was closed, as ccache 3.2 was released. The release notes of clang 3.2 contain several items related to clang. So it was time to give this another look. Basically, the conclusions from part 2 still stand: You cannot use ccache with clang without using CCACHE_CPP2. And it is now becoming clear to me that this is an issue that is not going to go away, and it s not really even Clang s fault. Warnings! The problem is that clang s -Wall can cause warnings when compiling the preprocessed version of otherwise harmless C code. This can be illustrated by this piece of C code:

int
foo()
 
        int *p, *q;
        p = q = 0;
        p = p;
        if (p == p)
                return 1;
        if ((p == q))
                return 2;
        return 0;
 

When compiled by gcc-4.9 -Wall, this gives no warnings. When compiled by clang-3.5 -Wall, this results in test.c:7:4: warning: explicitly assigning value of variable of type 'int *' to itself [-Wself-assign] test.c:8:8: warning: self-comparison always evaluates to true [-Wtautological-compare] test.c:10:9: warning: equality comparison with extraneous parentheses [-Wparentheses-equality] test.c:10:9: note: remove extraneous parentheses around the comparison to silence this warning test1.c:10:9: note: use '=' to turn this equality comparison into an assignment You wouldn t normally write code like this, but the C preprocessor could create code with self-assignments, self-comparisons, extra parentheses, and so on. This example represents the issues I saw when trying to compile PostgreSQL 9.4 with ccache and clang; there might be others. You can address this issue in two ways:

1. Use CCACHE_CPP2, as discussed in part 2. With ccache 3.2, you can now also put this into a configuration file: run_second_cpp = true in ~/.ccache/ccache.conf
2. Turn off the warnings mentioned above: -Wno-parentheses-equality, -Wno-tautological-compare, -Wno-self-assign (and any others you might find). One might think that these are actually useful warnings that one might want to keep, but GCC doesn t warn about them, and if you develop primarily with GCC, your code might contain these issues anyway. In particular, I have found that -Wno-tautological-compare is necessary for legitimate code.

I think CCACHE_CPP2 is the way to go, for two reasons. Firstly, having to add more and more options to turn off warnings is obviously somewhat stupid. Secondly and more importantly, there is nothing stopping GCC from adding warnings similar to Clang s that would trigger on preprocessed versions of otherwise harmless C code. Unless they come up with a clever way to annotate the preprocessed code to the effect of this code might look wrong to you, but it looked OK before preprocessing, so don t warn about it , in a way that creates no extra warnings and doesn t lose any warnings, I don t think this issue can be solved. Speed! Now the question is, how much would globally setting CCACHE_CPP2 slow things down? To test this, I have built PostgreSQL 9.4rc1 with clang-3.5 and gcc-4.8 (not gcc-4.9 because it creates some unrelated warnings that I don t want to deal with here). I have set export CCACHE_RECACHE=true so that the cache is not read but new cache entries are computed. That way, the overhead of ccache on top of the compiler is measured. Results:

• clang-3.5
  • Using ccache is 10% slower than not using it at all.
  • Using ccache with CCACHE_CPP2 on is another 10% slower.
• gcc-4.8
  • Using ccache is 19% slower than not using it at all.
  • Using ccache with CCACHE_CPP2 is another 9% slower.

(There different percentages between gcc and clang arise because gcc is faster than clang (yes, really, more on that in a future post), but the overhead of ccache doesn t change.) 10% or so is not to be dismissed, but let s remember that this applies only if there is a cache miss. If everything is cached, both methods do the same thing. Also, if you use parallel make, the overhead is divided by the number of parallel jobs. With that in mind, I have decided to put the issue to rest for myself and have made myself a ~/.ccache/ccache.conf containing

run_second_cpp = true

Now Clang or any other compiler should run without problems through ccache. Color! There is one more piece of news in the new ccache release: Another thing I talked about in part 1 was that ccache will disable the colored output of clang, and I suggested workarounds. This was actually fixed in ccache 3.2, so the workarounds are no longer necessary, and the above configuration change is really the only thing to make Clang work smoothly with ccache.

SSD adventures and fun times! TL;DR: Oh boy. Things work now, but I'm not sure exactly what happened When it all worked fine almost In June, I bought a new laptop, and a new SSD for it. I used that model of SSD before (Samsung 840 Evo), although not for a long time, so I wasn't expecting anything unusual. Since the laptop is a slower one, I installed Debian as follows: connect SSD to my workstation, do an install on it (via Virtualbox connected to the raw device), disconnect and install in the laptop. First sign of trouble was that the SSD didn't boot reliably. I said - maybe my Virtualbox method (new method) wasn't right - so I reinstalled on the laptop, and everything was mostly OK. Mostly OK because rarely the laptop had issues with seeing the SSD. Maybe not cold booting, requiring a restart, or giving some ATA errors on boot (bot nothing afterwards). I didn't use the laptop too much - I mostly use it when travelling - so I didn't investigate further. Trouble begins Fast forward to two weeks ago: I was preparing to leave on a trip, so I booted the laptop (everything OK), apt-get dist-upgrade, synced my git trees, etc. Everything was fine. The next day, in the airport between two flights, the laptop doesn't boot - doesn't see the SSD at all. I tried rebooting a gazillion times, nothing. I was quite upset - at the hardware, and at me for not paying attention to the unreliability signs before. Once I arrived at the destination, I opened the laptop, tried re-seating the SSD, nothing. I bought a SATA-to-USB bridge, and surprise! Boots from the first, no issues. Diagnosis A: Laptop SATA connector has issues. I work with this SATA-to-USB bridge for a couple of days, but it was quite slow (~20MB/s), so I buy a SATA-to-USB3 cradle, which should be much faster. But the SSD was not visible in this cradle. Not only that, but it was causing the laptop to hang in the POST screen - reliably. Turn the cradle off, the laptop passes POST, turn it on, the laptop took 2 minutes to pass the POST. OK, the cradle is broken. Connect the SSD back to the USB2 thing not booting For about five minutes, it was like "dead". After that, it booted and behaved normally. I didn't know what to think, I just put it aside. So worked for the rest of the week on the USB2 bridge, with no issues (once it the SSD dropped off, but I think that was just USB being USB). So at the end of the week, diagnosis (A) still was the main contender. On the flight back home, I worked from the plane for a good number of hours, again no issue. Laptop/SSD were fully powered off before the flight, powered with no issues, worked fine. At the end of the flight I completely shut down my laptop. Diagnosis A still on top. Real trouble now After getting home and sleeping a bit, I wanted to power up the laptop just to transfer the code I wrote on the plane. But it didn't power up. No problem, I said, now I actually have access to running Linux machines and I can check what's happening. And to my surprise, the SSD was behaving erratically:

• part of the time it was not seen at all, the block device was a generic kernel: scsi 21:0:0:0: Direct-Access USB TO I DE/SATA Device 0008 PQ: 0 ANSI: 0
• part of the time it was seen, i.e. the block device was visible as kernel: scsi 22:0:0:0: Direct-Access Samsunp w 0 EVO 500G 0008 PQ: 0 ANSI: 0, but with lots of ATA errors
• it took ~10 minutes before I was able to access it at all
• access to the disk when it was visible and not erroring out was slow; e.g. a cfdisk /dev/sdb could take >5 seconds before showing the screen

The looked like this (when connected over SATA):

20:22:57 kernel: ata2.00: ATA-9: Samsung SSD 840 EVO 500GB, EXT0BB6Q, max UDMA/133
20:22:57 kernel: ata2.00: 976773168 sectors, multi 1: LBA48 NCQ (depth 31/32), AA
20:22:57 kernel: ata2.00: configured for UDMA/133
20:22:57 kernel: scsi 3:0:0:0: Direct-Access     ATA      Samsung SSD 840  EXT0 PQ: 0 ANSI: 5
20:22:57 kernel: sd 3:0:0:0: [sdc] 488397168 512-byte logical blocks: (250 GB/232 GiB)
20:22:57 kernel: sd 3:0:0:0: [sdc] Write Protect is off
20:22:57 kernel: sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
20:22:57 kernel: sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
20:22:57 kernel: ata2.00: exception Emask 0x0 SAct 0x2 SErr 0x400001 action 0x6 frozen
20:22:57 kernel: ata2: SError:   RecovData Handshk  
20:22:57 kernel: ata2.00: failed command: READ FPDMA QUEUED
20:22:57 kernel: ata2.00: cmd 60/08:08:68:01:00/00:00:00:00:00/40 tag 1 ncq 4096 in
20:22:57 kernel:         res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
20:22:57 kernel: ata2.00: status:   DRDY  
20:22:57 kernel: ata2: hard resetting link
20:22:57 kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
20:22:57 kernel: ata2.00: failed to get NCQ Send/Recv Log Emask 0x1
20:22:57 kernel: ata2.00: failed to get NCQ Send/Recv Log Emask 0x1
20:22:57 kernel: ata2.00: configured for UDMA/133
20:22:57 kernel: ata2.00: device reported invalid CHS sector 0
20:22:57 kernel: ata2: EH complete
20:22:57 kernel: ata2.00: exception Emask 0x0 SAct 0x400000 SErr 0x400001 action 0x6
20:22:57 kernel: ata2.00: irq_stat 0x44000008
20:22:57 kernel: ata2: SError:   RecovData Handshk  
20:22:57 kernel: ata2.00: failed command: READ FPDMA QUEUED
20:22:57 kernel: ata2.00: cmd 60/08:b0:08:03:00/00:00:00:00:00/40 tag 22 ncq 4096 in
20:22:57 kernel:         res 41/84:00:08:03:00/00:00:00:00:00/00 Emask 0x410 (ATA bus error) <F>
20:22:57 kernel: ata2.00: status:   DRDY ERR  
20:22:57 kernel: ata2.00: error:   ICRC ABRT  
20:22:57 kernel: ata2: hard resetting link
20:22:57 kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
20:22:57 kernel: ata2.00: failed to get NCQ Send/Recv Log Emask 0x1
20:22:57 kernel: ata2.00: failed to get NCQ Send/Recv Log Emask 0x1
20:22:57 kernel: ata2.00: configured for UDMA/133
20:22:57 kernel: ata2: EH complete

Note that all of the messages were in short order (during boot, they show the same timestamp but I don't think it was actually the same second). I tried connecting the SSD over USB2 (partially working), over USB3 (not working!), and directly over SATA (initially not working). I connected another SSD I had around (same model, just smaller capacity) over all three, it worked (so the USB3 bridge was working, at least). Diagnosing (A) was out the door now, and the situation was very clear: SSD dying/dead/almost gone. So I connected both broken SSD and empty SSD to my workstation over SATA, and started copying data (once I managed to boot with the old one being visible and working). During the data copy, I saw that the "broken" SSD behaved erratically indeed: it was copying data off it with either ~40MB/s, ~70MB/s, and ~160MB/s. Not other speed, at least not for long time, just cycling between these three; and this is a very slow speed for this SSD model. And then I remembered that there is, for this model (Samsung 840 Evo), an advisory/firmware fix that old data gets harder to access (slower and slower), due to how TLC cells levels are read/etc. I don't know exactly what "old" means, but since the partition table was written only once, it should be the oldest thing written, which would make it the most susceptible to the slowdown, and could explain the cfdisk slowness. So after the data copy, I tested this:

• read the first ~1GB off the SSD: ~70MB/s
• rewrite it
• read it again: ~110MB/s

So yes, it is something related to this, probably. Diagnosis B: "just" the Samsung Evo bug. So I proceed to (try to) upgrade the firmware:

• the firmware upgrade and data fix on the "new" SSD: firmware upgrade almost instantly, no issues at all
• firmware upgrade on the "old" SSD: Failed to upgrade firmware (or some message like that)

Uh oh, so it's more than this - the SSD seems to be actually broken. I don't have much experience with SSDs, I used for many years an Intel X25-M and more recently I have a couple of Samsung 840 Pro, but I never had issues until now, so maybe the cheap Evo is just cheap and fast Note that during all this drama, SMART gives good info about SSD, and SMART tests are perfect (short/long). Just a number of UDMA CRC errors so thanks for nothing. I try to the firmware upgrade a few times, I give up. Diagnosis C: SSD is just plain broken. I'll send the SSD for a replacement/fix, so (even though it's encrypted), let's try to erase it first (as I can still read/write to it). And since just dd if=/dev/zero of=/dev/sdX is too common, let's try a builtin (ATA) erase! After fighting with hdparm and the fact that my BIOS does indeed "security freeze" on the drives (so you can't change the security settings nor erase the drives), and finding an article on the net that gives a few workaround, I manage to "unfreeze" it by not only live unplugging the SATA cable, but also the power cable, and plugging them back in. Time to erase. Side-note here:

• hdparm says: "2min for SECURITY ERASE UNIT"; this actually takes ~40 seconds
• hdparm also says: "8min for ENHANCED SECURITY ERASE UNIT"; this takes only ~5 seconds!

So, especially given the very fast ENHANCED erase, I now believe the SSD is truly broken. Diagnosis C rules. And finally, just confusion After the SSD erase, reading (the zeros) from the SSD is really fast: more than 500MB/s, as expected. I also try once more to upgrade the firmware for the Evo bug, and it works flawlessly this time. Even rebooting the machine many times works without problems, the SSD is seen every single time. I still can't make it work well over the USB3 bridge (but I'm not very convinced that device work well, since Linux does apply quirks when it detects it). But it doesn't give any ATA errors this time, and it works (read and write) consistently fast (as per the Evo specs). I still have to let it go through a long burnin with verification, but a short one works fine, so at this point I'm just very confused as what was (is?) the actual problem with this device, and why it behaves so strange Was the laptop broken? Clearly no. Was the SSD broken, or just impacted by the slowdown issue? Time will tell. I'll put it through the paces, and see. But unless I replace it, I'll keep this just for "fast cache" use cases, and not to hold real data. Yep, computers are fun!

C++11 talk examples On 2014-11-27 I gave a talk about C++ and new features introduced with C++11: these are the examples. They are all licensed under the wtfpli version 2. See cxx11-talk-notes for the talk notes. Note that the wrapper interfaces turns errors from the underlying libraries into exceptions, so the method calls just do what they should, without the need of documenting special return values for error messages, and removing the need for each library to implement yet another way of reporting errors. Also note that all wrapper objects do RAII: you create them and they clean after themselves when they go out of scope. The wrapper objects also have cast operators to make them behave as the pointer or handle that they are wrapping, so that they can be transparently passed to the underlying libraries. (note: I had to add U+2063 INVISIBLE SEPARATOR to prevent noreturn statements to be misinterpreted by the blog formatter. If you copypaste the code and encounter issues, you may want to delete the noreturn statements and retype them) A gcrypt hash class This class is a light wrapper around gcrypt's hashing functions. ezhash.h ezhash.cpp Example usage Simple sqlite bindings Remarkably simple sqlite3 bindings based on lambda callbacks. ezsqlite.h ezsqlite.cpp Example usage A fast Directory object This is a lightweight wrapper around O_PATH file descriptors for directories. I'd love to see a library of well-maintained and thin C++ bindings around libc, that do little more than turning errors into exceptions and making it also work with std::string buffers. ezfs.h ezfs.cpp Example usage

FTP assistant In contrast to the last month, this month has been rather quiet and I really liked that . The stress has moved to the next team. So all in all I marked 101 packages for accept and had to reject 27 packages. As I mostly reviewed really new packages, I didn t have to file any RC bug this month. Squeeze LTS This was my fifth month that I did some work for the Squeeze LTS initiative, started by Raphael Hertzog at Freexian. This month I got assigned a workload of 14.25h and I spent these hours to upload new versions of:

• [DLA 82-1] wget security update
• [DLA 84-1] curl security update
• [DLA 89-1] nss security update
• [DLA 90-1] imagemagick security update
• [DLA 94-1] php5 security update
• [DLA 97-1] eglibc security update

I also uploaded [DLA 85-1] libxml-security-java security update, but as nobody of the LTS sponsors had any interest in this package, I did this in my spare time. A package with security in its name should not be affected by security issues. This month my failure of the month has been the binutils package. Although the security team prepared the way for finding the correct patches for all those CVEs, I somehow managed to not find them. This is embarassing I am also a bit disappointed by current LTS users. All important packages have been made available for testing before uploading them to the archive. Apart from some brave fellow DDs, no other feedback was reported on debian-lts. Complaints arrived only when the packages have been finally uploaded. Do admins have enough time nowadays and don t need to use some kind of testbed? Times are changing Other packages This month I even found some time to sponsor uploads, so please welcome a new version of fastaq in experimental and patiently wait for aegaen and kmc to pass NEW. At this point I also want to mention the Debian Med Advent Calendar, which was announced in this email and already mentioned by Andreas in his latest Debian Med bits. Everybody is invited to take care of as much as possible poor souls. Support If you would like to support my Debian work you could either be part of the Freexian initiative (see above) or consider to send some bitcoins to 1JHnNpbgzxkoNexeXsTUGS6qUp5P88vHej. Contact me at donation@alteholz.eu if you prefer another way to donate. Every kind of support is most appreciated.

I was unable to find a good place that describes how to create a simple self-hosted email setup. The most surprising discovery was, how much already works after:

apt-get install postfix dovecot-imapd

Right after having finished the installation I was able to receive email (but only in in /var/mail in mbox format) and send email (bot not from any other host). So while I expected a pretty complex setup, it turned out to boil down to just adjusting some configuration parameters.

Postfix The two interesting files to configure postfix are /etc/postfix/main.cf and /etc/postfix/master.cf. A commented version of the former exists in /usr/share/postfix/main.cf.dist. Alternatively, there is the ~600k word strong man page postconf(5). The latter file is documented in master(5).

/etc/postfix/main.cf I changed the following in my main.cf

@@ -37,3 +37,9 @@
 mailbox_size_limit = 0
 recipient_delimiter = +
 inet_interfaces = all
+
+home_mailbox = Mail/
+smtpd_recipient_restrictions = permit_mynetworks reject_unauth_destination permit_sasl_authenticated
+smtpd_sasl_type = dovecot
+smtpd_sasl_path = private/auth
+smtp_helo_name = my.reverse.dns.name.com

At this point, also make sure that the parameters smtpd_tls_cert_file and smtpd_tls_key_file point to the right certificate and private key file. So either change these values or replace the content of /etc/ssl/certs/ssl-cert-snakeoil.pem and /etc/ssl/private/ssl-cert-snakeoil.key. The home_mailbox parameter sets the default path for incoming mail. Since there is no leading slash, this puts mail into $HOME/Mail for each user. The trailing slash is important as it specifies qmail-style delivery'' which means maildir. The default of the smtpd_recipient_restrictions parameter is permit_mynetworks reject_unauth_destination so this just adds the permit_sasl_authenticated option. This is necessary to allow users to send email when they successfully verified their login through dovecot. The dovecot login verification is activated through the smtpd_sasl_type and smtpd_sasl_path parameters. I found it necessary to set the smtp_helo_name parameter to the reverse DNS of my server. This was necessary because many other email servers would only accept email from a server with a valid reverse DNS entry. My hosting provider charges USD 7.50 per month to change the default reverse DNS name, so the easy solution is, to instead just adjust the name announced in the SMTP helo.

/etc/postfix/master.cf The file master.cf is used to enable the submission service. The following diff just removes the comment character from the appropriate section.

@@ -13,12 +13,12 @@
 #smtpd     pass  -       -       -       -       -       smtpd
 #dnsblog   unix  -       -       -       -       0       dnsblog
 #tlsproxy  unix  -       -       -       -       0       tlsproxy
-#submission inet n       -       -       -       -       smtpd
-#  -o syslog_name=postfix/submission
-#  -o smtpd_tls_security_level=encrypt
-#  -o smtpd_sasl_auth_enable=yes
-#  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
-#  -o milter_macro_daemon_name=ORIGINATING
+submission inet n       -       -       -       -       smtpd
+  -o syslog_name=postfix/submission
+  -o smtpd_tls_security_level=encrypt
+  -o smtpd_sasl_auth_enable=yes
+  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
+  -o milter_macro_daemon_name=ORIGINATING
 #smtps     inet  n       -       -       -       -       smtpd
 #  -o syslog_name=postfix/smtps
 #  -o smtpd_tls_wrappermode=yes

Dovecot Since above configuration changes made postfix store email in a different location and format than the default, dovecot has to be informed about these changes as well. This is done in /etc/dovecot/conf.d/10-mail.conf. The second configuration change enables postfix to authenticate users through dovecot in /etc/dovecot/conf.d/10-master.conf. For SSL one should look into /etc/dovecot/conf.d/10-ssl.conf and either adapt the parameters ssl_cert and ssl_key or store the correct certificate and private key in /etc/dovecot/dovecot.pem and /etc/dovecot/private/dovecot.pem, respectively. The dovecot-core package (which dovecot-imapd depends on) ships tons of documentation. The file /usr/share/doc/dovecot-core/dovecot/documentation.txt.gz gives an overview of what resources are available. The path /usr/share/doc/dovecot-core/dovecot/wiki contains a snapshot of the dovecot wiki at http://wiki2.dovecot.org/. The example configurations seem to be the same files as in /etc/ which are already well commented.

/etc/dovecot/conf.d/10-mail.conf The following diff changes the default email location in /var/mail to a maildir in ~/Mail as configured for postfix above.

@@ -27,7 +27,7 @@
 #
 # <doc/wiki/MailLocation.txt>
 #
-mail_location = mbox:~/mail:INBOX=/var/mail/%u
+mail_location = maildir:~/Mail
 
 # If you need to set multiple mailbox locations or want to change default
 # namespace settings, you can do it by defining namespace sections.

/etc/dovecot/conf.d/10-master.conf And this enables the authentication socket for postfix:

@@ -93,9 +93,11 @@
    
 
   # Postfix smtp-auth
-  #unix_listener /var/spool/postfix/private/auth  
-  #  mode = 0666
-  # 
+  unix_listener /var/spool/postfix/private/auth  
+    mode = 0660
+    user = postfix
+    group = postfix
+   
 
   # Auth process is run as this user.
   #user = $default_internal_user

Aliases Now Email will automatically put into the '~/Mail' directory of the receiver. So a user has to be created for whom one wants to receive mail...

$ adduser josch

...and any aliases for it to be configured in /etc/aliases.

@@ -1,2 +1,4 @@
-# See man 5 aliases for format
-postmaster:    root
+root:       josch
+postmaster: josch
+hostmaster: josch
+webmaster:  josch

After editing /etc/aliases, the command

$ newaliases

has to be run. More can be read in the aliases(5) man page.

Finishing up Everything is done and now postfix and dovecot have to be informed about the changes. There are many ways to do that. Either restart the services, reboot or just do:

$ postfix reload
$ doveadm reload

Have fun!