Search Results: "xam"

10 December 2014

Chris Lamb: Starting IPython automatically from zsh

Instead of a calculator, I tend to use IPython for those quotidian bits of "mental" arithmetic: 
In  [1]: 17 * 22.2
Out [1]: 377.4
However, I often forget to actually start IPython, resulting in me running the following in my shell: 
$ 17 * 22.2
zsh: command not found: 17
Whilst I could learn do this maths within Zsh itself, I would prefer to dump myself into IPython instead being able to use "_" and Python modules generally is just too useful. After following this pattern too many times, I put together the following snippet that will detect whether I have prematurely attempted a calculation inside zsh and pretend that I ran it in IPython all along: 
zmodload zsh/pcre
math_regex='^[\d\-][\d\.\s\+\*\/\-]*$'
function math_precmd()  
    if [ "$ ? " = 0 ]
    then
        return
    fi
    if [ -z "$ math_command " ]
    then
        return
    fi
    if whence -- "$math_command" 2>&1 >/dev/null
    then
        return
    fi
    if [ "$ math_command " -pcre-match "$ math_regex " ]
    then
        echo
        ipython -i -c "_=$ math_command ; print _"
    fi
 
function math_preexec()  
    typeset -g math_command="$ 1 "
 
typeset -ga precmd_functions
typeset -ga preexec_functions
precmd_functions+=math_precmd
preexec_functions+=math_preexec
For example: 
lamby@seriouscat:~% 17 * 22.2
zsh: command not found: 17
377.4
In  [1]: _ + 1
Out [1]: 378.4
(Canonical version from my zshrc.d)

9 December 2014

Joey Hess: podcasts that don't suck, 2014 edition

Also, out of the podcasts I listed previously, I still listen to and enjoy Free As In Freedom, Off the Hook, and the Long Now Seminars. PS: A nice podcatcher, for the technically inclined is git-annex importfeed. Featuring list of feeds in a text file, and distributed podcatching!

Wouter Verhelst: Playing with ExtreMon

Munin is a great tool. If you can script it, you can monitor it with munin. Unfortunately, however, munin is slow; that is, it will take snapshots once every five minutes, and not look at systems in between. If you have a short load spike that takes just a few seconds, chances are pretty high munin missed it. It also comes with a great webinterfacefrontendthing that allows you to dig deep in the history of what you've been monitoring. By the time munin tells you that your Kerberos KDCs are all down, you've probably had each of your users call you several times to tell you that they can't log in. You could use nagios or one of its brethren, but it takes about a minute before such tools will notice these things, too. Maybe use CollectD then? Rather than check once every several minutes, CollectD will collect information every few seconds. Unfortunately, however, due to the performance requirements to accomplish that (without causing undue server load), writing scripts for CollectD is not as easy as it is for Munin. In addition, webinterfacefrontendthings aren't really part of the CollectD code (there are several, but most that I've looked at are lacking in some respect), so usually if you're using CollectD, you're missing out some. And collectd doesn't do the nagios thing of actually telling you when things go down. So what if you could see it when things go bad? At one customer, I came in contact with Frank, who wrote ExtreMon, an amazing tool that allows you to visualize the CollectD output as things are happening, in a full-screen fully customizable visualization of the data. The problem is that ExtreMon is rather... complex to set up. When I tried to talk Frank into helping me getting things set up for myself so I could play with it, I got a reply along the lines of...
well, extremon requires a lot of work right now... I really want to fix foo and bar and quux before I start documenting things. Oh, and there's also that part which is a dead end, really. Ask me in a few months?
which is fair enough (I can't argue with some things being suboptimal), but the code exists, and (as I can see every day at $CUSTOMER) actually works. So I decided to just figure it out by myself. After all, it's free software, so if it doesn't work I can just read the censored code. As the manual explains, ExtreMon is a plugin-based system; plugins can add information to the "coven", read information from it, or both. A typical setup will run several of them; e.g., you'd have the from_collectd plugin (which parses the binary network protocol used by collectd) to get raw data into the coven; you'd run several aggregator plugins (which take that raw data and interpret it, allowing you do express things along the lines of "if the system's load gets above X, set load.status to warning"; and you'd run at least one output plugin so that you can actually see the damn data somewhere. While setting up ExtreMon as is isn't as easy as one would like, I did manage to get it to work. Here's what I had to do. You will need: First, we clone the ExtreMon git repository: 
git clone https://github.com/m4rienf/ExtreMon.git extremon
cd extremon
There's a README there which explains the bare necessities on getting the coven to work. Read it. Do what it says. It's not wrong. It's not entirely complete, though; it fails to mention that you need to Make sure the dump.py script outputs something from collectd. You'll know when it shows something not containing "plugin" or "plugins" in the name. If it doesn't, fiddle with the #x3. lines at the top of the from_collectd file until it does. Note that ExtreMon uses inotify to detect whether a plugin has been added to or modified in its plugins directory; so you don't need to do anything special when updating things. Next, we build the java libraries (which we'll need for the display thing later on): 
cd java/extremon
mvn install
cd ../client/
mvn install
This will download half the Internet, build some java sources, and drop the precompiled .jar files in your $HOME/.m2/repository. We'll now build the display frontend. This is maintained in a separate repository: 
cd ../..
git clone https://github.com/m4rienf/ExtreMon-Display.git display
cd display
mvn install
This will download the other half of the Internet, and then fail, because Frank forgot to add a few repositories. Patch (and push request) on github With that patch, it will build, but things will still fail when trying to sign a .jar file. I know of four ways on how to fix that particular problem:
  1. Add your passphrase for your java keystore, in cleartext, to the pom.xml file. This is a terrible idea.
  2. Pass your passphrase to maven, in cleartext, by using some command line flags. This is not much better.
  3. Ensure you use the maven-jarsigner-plugin 1.3.something or above, and figure out how the maven encrypted passphrase store thing works. I failed at that.
  4. Give up on trying to have maven sign your jar file, and do it manually. It's not that hard, after all.
If you're going with 1 through 3, you're on your own. For the last option, however, here's what you do. First, you need a key: 
keytool -genkeypair -alias extremontest
after you enter all the information that keytool will ask for, it will generate a self-signed code signing certificate, valid for six months, called extremontest. Producing a code signing certificate with longer validity and/or one which is signed by an actual CA is left as an exercise to the reader. Now, we will sign the .jar file: 
jarsigner target/extremon-console-1.0-SNAPSHOT.jar extremontest
There. Who needs help from the internet to sign a .jar file? Well, apart from this blog post, of course. You will now want to copy your freshly-signed .jar file to a location served by HTTPS. Yes, HTTPS, not HTTP; ExtreMon-Display will fail on plain HTTP sites. Download this SVG file, and open it in an editor. Find all references to be.grep as well as those to barbershop and replace them with your own prefix and hostname. Store it along with the .jar file in a useful directory. Download this JNLP file, and store it on the same location (or you might want to actually open it with "javaws" to see the very basic animated idleness of my system). Open it in an editor, and replace any references to barbershop.grep.be by the location where you've stored your signed .jar file. Add the chalice_in_http plugin from the plugins directory. Make sure to configure it correctly (by way of its first few comment lines) so that its input and output filters are set up right. Add the configuration snippet in section 2.1.3 of the manual (or something functionally equivalent) to your webserver's configuration. Make sure to have authentication chalice_in_http is an input mechanism. Add the chalice_out_http plugin from the plugins directory. Make sure to configure it correctly (by way of its first few comment lines) so that its input and output filters are set up right. Add the configuration snippet in section 2.2.1 of the manual (or something functionally equivalent) to your webserver's configuration. Authentication isn't strictly required for the output plugin, but you might wish for it anyway if you care whether the whole internet can see your monitoring. Now run javaws https://url/x3console.jnlp to start Extremon-Display. At this point, I got stuck for several hours. Whenever I tried to run x3mon, this java webstart thing would tell me simply that things failed. When clicking on the "Details" button, I would find an error message along the lines of "Could not connect (name must not be null)". It would appear that the Java people believe this to be a proper error message for a fairly large number of constraints, all of which are slightly related to TLS connectivity. No, it's not the keystore. No, it's not an API issue, either. Or any of the loads of other rabbit holes that I dug myself in. Instead, you should simply make sure you have Server Name Indication enabled. If you don't, the defaults in Java will cause it to refuse to even try to talk to your webserver. The ExtreMon github repository comes with a bunch of extra plugins; some are special-case for the place where I first learned about it (and should therefore probably be considered "examples"), others are general-purpose plugins which implement things like "is the system load within reasonable limits". Be sure to check them out. Note also that while you'll probably be getting most of your data from CollectD, you don't actually need to do that; you can write your own plugins, completely bypassing collectd. Indeed, the from_collectd thing we talked about earlier is, simply, also a plugin. At $CUSTOMER, for instance, we have one plugin which simply downloads a file every so often and checks it against a checksum, to verify that a particular piece of nonlinear software hasn't gone astray yet again. That doesn't need collectd. The example above will get you a small white bar, the width of which is defined by the cpu "idle" statistic, as reported by CollectD. You probably want more. The manual (chapter 4, specifically) explains how to do that. Unfortunately, in order for things to work right, you need to pretty much manually create an SVG file with a fairly strict structure. This is the one thing which Frank tells me is a dead end and needs to be pretty much rewritten. If you don't feel like spending several days manually drawing a schematic representation of your network, you probably want to wait until Frank's finished. If you don't mind, or if you're like me and you're impatient, you'll be happy to know that you can use inkscape to make the SVG file. You'll just have to use dialog behind ctrl+shift+X. A lot. Once you've done that though, you can see when your server is down. Like, now. Before your customers call you.

Thorsten Glaser: The colon in the shell: corrigenda

Bernhard s article on Pl net Debian about the colon command in the shell could use a clarification and a security-relevant correcture. There is, indeed, no difference between the : and true built-in commands. St phane Chazelas points out that writing : $ VARNAME:=default is bad, : "$ VARNAME:=default " is correct. Reason: someone could preset $VARNAME with, for example, /*/*/*/*/../../../../*/*/*/*/../../../../*/*/*/* which will exhaust during globbing. Besides that, the article is good. Thanks Bernhard for posting it! PS: I sometimes use the colon as comment leader in the last line of a script or function, because it, unlike the octothorpe, sets $? to 0, which can be useful. Update: As jilles pointed out in IRC, colon ( : ) is a POSIX special built-in (most importantly, it keeps assignments), whereas true is a regular built-in utility.

8 December 2014

Bernhard R. Link: The Colon in the Shell.

I was recently asked about some construct in a shell script starting with a colon(:), leading me into a long monologue about it. Afterwards I realized I had forgotten to mention half of the nice things. So here for your amusement some usage for the colon in the shell: To find the meaning of ":" in the bash manpage[1], you have to look at the start of the SHELL BUILTIN COMMANDS section. There you find: 
: [arguments]
	No effect; the command does nothing beyond expanding arguments and performing any specified redirections.  A zero exit code is returned.
If you wonder what the difference to true is: I don't know any difference (except that there is no /bin/:) So what is the colon useful for? You can use it if you need a command that does nothing, but still is a command. Then there is more things you can do with the colon, most I'd put under "abuse": This is of course not a complete list. But unless I missed something else, those are the most common cases I run into. [1] <rant>If you never looked at it, better don't start: the bash manpage is legendary for being quite useless as hiding all information in other information in a quite absurd order. Unless you look at documentation about how to write a shell script parser, then the bash manpage is really what you want to read.</rant>

7 December 2014

Sven Hoexter: Failing with F5: Implementing HTTP session persistence

In the old days people deployed the Apache HTTPD together with mod_jk and mod_balancer in front of Apache Tomcat instances to achieve scalability and resilience. That sooner or later turns out to not provide features you nowadays would like to have to run a service 24/7. So it happened that I had to jump into the sometimes cold sea of managing services fronted by F5 BigIP devices, a somewhat common load balancing solution. Soon after you start to introduce loadbalancing, because you grow beyond the "one fat webserver" state, you'll have the requirement to implement some way of session persistence. In the old days you might have added a unique jvmRoute value to your mod_balancer pool and to your tomcat configuration to match the connection to the right pool member. If you're clever you've set it on the Tomcat as a variable that automatically builds up the jvmRoute based on the hostname, so you can add new instances without having to touch the Tomcat configuration. In case you were not that clever (like we were when we started) you might have added placeholders in the server.xml to set the jvmRoute value in your deployment script. I think in a somewhat better world your first step usually is to introduce a shared session store, be it the php session handler backed by a memcached or Hazelcast for the Java afficionados, to avoid the persistence requirement on your balancer. But if you grow even bigger the requirement will soon reappear because you might have to add some form of sharding and now have to sent users to the same cluster serving a specific shard. In the end all of that is not cool and your developers have to be aware of those issues. For all cases where fault tolerance within the session is not a priority we aimed for something that we can enable or disable on the balancer without application code changes. The "let the box do its job" solution The nicest solution from our point of view is letting the F5 inject a cookie that points the device on subsequent requests to the right pool node. If you like you can define the cookie name, a timeout, force the injection on replies and some other tunables. Take a look at the help for ltm persistence cookie if you'd like to have a look at the details. The "let's do it by hand but on the box" solution Most Java frameworks provide a session cookie called JSESSIONID. Why not use that one and add a persistence table entry on the BigIP with the cookie as the lookup key? As always if there's no solution provided out of the box you can implement an iRule for it. Kind of an F5 mantra. There are many examples out there, here is what we ended up with: 
when HTTP_REQUEST  
    if   [HTTP::cookie "JSESSIONID"] ne ""  
        persist uie [string tolower [HTTP::cookie "JSESSIONID"]] 1800 
     
 
when HTTP_RESPONSE  
    if   [HTTP::cookie "JSESSIONID"] ne ""  
        persist add uie [string tolower [HTTP::cookie "JSESSIONID"]] 1800
And here is where I failed Now imagine you're running an application that so far handles only machine to machine traffic without session persistence. A few month later you're approached to enable session persistence, because someone introduced a management console for the application as a web user interface. Without further ado you jump on board and provide - showcasing the flexibility of the new balancer - session persistence based on cookies injected by the BigIP. You've done it before, you enable it for the test environment, everything looks fine, you enable it for the production setup. Everything is fine (at least for now ...), the WUI workes fine, sessions are persistent to the node the user hit first. A month or two later someone from the development team approaches you again, this time asking if the balancing setup has some issues because over 90% of the live traffic hit one of the two machines in the pool until it wents down due to a crash or a deployment. Then about 90% hit the other node, even after the crashed one is resurrected. But every time the developer tests the balancing with a "while true" loop sending several hundred requests via curl the balancing is perfectly equal. After beeing puzzled by the behaviour myself we started to look at the live traffic with tcpdump and suddenly it was all clear when I saw the BigIP cookie beeing passed around. Actually everything behaved as we configured it. The application used a proper HTTP library and that one implemented cookies: If you receive one you're polite and pass it back to the sender. The original sender of course also implements HTTP with cookie support and now, due to the natur of the traffic, we have long living connections between two applications passing around the same cookie. On every single HTTP request sent through this connection. And every time the balancer will look at the cookie and happily base the routing decision on the pool member noted in this cookie. When we tested with curl we did not have the usage of cookies on our mind. Because our false assumption was that the applications do not create cookies for machine to machine traffic. Nobody, including myself, actively remembered the decision to base the session persistence on cookies injected by the balancer. We went on and switched to the iRule mentioned above. Luckily we already had that one implemented and tested for a different use case with the requirement to not inject additional cookies.
  1. Can we maybe improve the change process for the balancer to handle a mixed traffic scenario?
  2. Do we need monitoring to monitor the balancing of requests?
  3. Is it a wise idea to mix machine2machine and user2machine traffic on the same port?
  4. Should we maybe, if we seperate properly, place management applications on a different system?
  5. Is our documentation sufficient?
Combine 3. and 4. and ask yourself how resilient you are if someone tries to exhaust the memory of the balancer with a lot of connection attempts that have random JSESSIONID cookies? Or is it even possible to exhaust ressource with faked BigIP cookies you inject? Maybe it's even a vector for something like http://events.ccc.de/congress/2011/Fahrplan/attachments/2007_28C3_Effective_DoS_on_web_application_platforms.pdf? Quite some questions left to think about and draft answers for.

Gregor Herrmann: GDAC 2014/7

creating a free operating system in general & fixing bugs in particular is a collaborative effort. today's example is bug report #766773 (& friends) which involved communication in the BTS & on IRC between the bug submitter, the maintainer of a related package, the release team, & me as a bug triager. & 2.5 hours later, another block for the jessie release is gone. thanks everybody for yet another pleasant collaboration experience!
this posting is part of GDAC (gregoa's debian advent calendar), a project to show the bright side of debian & why it's fun for me to contribute.

6 December 2014

Gregor Herrmann: GDAC 2014/6

positive feedback is motivating (probably not only) for me. a recent example showing that our work on creating a great operating system is appreciated was sent to the debian-project list yesterday. thanks!
this posting is part of GDAC (gregoa's debian advent calendar), a project to show the bright side of debian & why it's fun for me to contribute.

5 December 2014

Richard Hartmann: Release Critical Bug report for Week 49

Look at that bug count! At that pace, Jessy will happen before FOSDEM ;) The UDD bugs interface currently knows about the following release critical bugs: How do we compare to the Squeeze release cycle?
Week Squeeze Wheezy Jessie
43 284 (213+71) 468 (332+136) 319 (240+79)
44 261 (201+60) 408 (265+143) 274 (224+50)
45 261 (205+56) 425 (291+134) 295 (229+66)
46 271 (200+71) 401 (258+143) 427 (313+114)
47 283 (209+74) 366 (221+145) 342 (260+82)
48 256 (177+79) 378 (230+148) 274 (189+85)
49 256 (180+76) 360 (216+155) 226 (147+79)
50 204 (148+56) 339 (195+144)
51 178 (124+54) 323 (190+133)
52 115 (78+37) 289 (190+99)
1 93 (60+33) 287 (171+116)
2 82 (46+36) 271 (162+109)
3 25 (15+10) 249 (165+84)
4 14 (8+6) 244 (176+68)
5 2 (0+2) 224 (132+92)
6 release! 212 (129+83)
7 release+1 194 (128+66)
8 release+2 206 (144+62)
9 release+3 174 (105+69)
10 release+4 120 (72+48)
11 release+5 115 (74+41)
12 release+6 93 (47+46)
13 release+7 50 (24+26)
14 release+8 51 (32+19)
15 release+9 39 (32+7)
16 release+10 20 (12+8)
17 release+11 24 (19+5)
18 release+12 2 (2+0)
Graphical overview of bug stats thanks to azhag:

4 December 2014

Tomasz Buchert: BSP in Munich

My first plan was to participate in the Bug Squashing Party (BSP) in Paris, but finally I didn t manage to organize the travel and proper stay in the capital of France. Finally I settled on Munich and below you will find a summary of my stay there. While organizing my trip, the Munich natives offered me a couch to surf during the stay, which was very kind. In the end, however, I stayed at my friend s place - thanks a lot Mehdi! The BSP gathered Debian people interested in preparing Jessie release, but also KDE, Kolab and LibreOffice people. The host for the meeting was LiMux (on of the most known large-scale Linux deployments) which provided a venue, but also food (free!) and other attractions. Organizers, if you read this, I thank you for the amazing work that you have done. I arrived on late Friday s evening and so I didn t participate that day in bug squashing and took a good sleep instead. On Saturday and Sunday we were, well, squashing bugs. I haven t done NMUs before and it was a new experience to me, even though theoretically I know how it works. The DDs present at the BSP were more than helpful to help me when I was stuck. Here is a short description of what I have done: I also got my GPG keys signed and vice-versa, so I m much better connected to the web of trust now (7 signatures of DDs so far). All in all, it was a great experience which I recommend to everyone. The atmosphere is very motivating and there are many people who will gladly help you if you don t know or understand something. You don t have to be DD to participate even, but a minimal packaging experience is definitely very useful. I m quite sure that people would be glad to help even a complete newcomer (I certainly would do), but being able to build a package using apt-get source and debuild won t hurt anybody. At the very least you can always triage bugs: reproduce them, find the cause of the bug, propose a way to fix it, etc. As always you should be open-minded and willing to get your hands dirty. I want also thank Debian for sponsoring me which finally convinced me to participate (Munich is around 500 km from where I live and the travel is fairly expensive). Some links of interest: Many bugs still wait to be squashed (120 at the time of writing this), so let s get back to work!

3 December 2014

Michael Stapelberg: Announcing the new Debian Code Search Instant

For the last few months, I have been working on a new version of Debian Code Search, and today it s going live! I call it Debian Code Search Instant, for multiple reasons, see below. A lot faster The new Debian Code Search is still hosted by Rackspace (thank you!), but our new architecture finally allows us to use their Performance Cloud Servers hardware generation. The old code search architecture was split across 5 different servers, however the search itself was only performed on a single machine with a network attached block volume backed by SSD. The new Code Search spreads out both the trigram index and the source code onto 6 different servers, and thanks to the new hardware generation, we have a locally connected SSD drive in each of them. So, we now get more than 6 times the IOPS we ve had before, at much lower latency :). Grouping by Debian source package The first feature request we ever got for Code Search was to group results by Debian source package. I ve tried tackling that before, but it s a pretty hard problem. With the new architecture and capacity, this finally becomes feasible. After your query completes, there is a checkbox called Group search results by Debian source package . Enable it, and you ll get neatly grouped results. For each package, there is a link at the bottom to refine the search to only consider that package. In case you are more interested in the full list of packages, for example because you are doing large-scale changes across Debian, that s also possible: Click on the symbol right next to the Filter by package list, and a curl command will be revealed with which you can download the full list of packages. Expensive queries (with lots of results) now possible Previously, we had a 60 second timeout during which queries must complete. This timeout has been completely lifted, and long-running queries with tons and tons of results are now possible. We kindly ask you to not abuse this feature it s not very exciting to play with complexity explosion in regular expression engines by preventing others from doing their work. If you re interested in that sort of analysis, go grab the source code and play with it on your own machine :). Instant results While your query is running, you will almost immediately see the top 10 results, even though the search may still take a while. This is useful to figure out if your regular expression matches what you thought it would, before having to wait until your 5 minute query is finished. Also, the new progress bar tells you how far the system is with your search. Instant indexing Previously, Code Search was deployed to a new set of machines from scratch every week in order to update the index. This was necessary because the performance would severely degrade while building the index, so we were temporarily running with twice the amount of machines until the new version was up. In the new architecture, we store an index for each source package and then merge these into one big index shard. This currently takes about 4 minutes with the code I wrote, but I m sure this can be made even faster if necessary. So, whenever new packages are uploaded to the Debian archive, we can just index the new version and trigger a merge. We get notifications about new package uploads from FedMsg. Packages that are not seen on FedMsg for some reason are backfilled every hour. The time between uploading a package and being able to find it in Debian Code Search therefore now ranges from a couple of minutes to about an hour, instead of about a week! New, beautiful User Interface Since we needed to rewrite the User Interface anyway thanks to the new architecture, we also spent some effort on making it modern, polished and beautiful. Smooth CSS animations make you aware of what s going on, and the search results look cleaner than ever. Conclusion At least to me, the new Debian Code Search seems like a significant improvement over the old one. I hope you enjoy the new features into which I put a lot of work. In case you have any feedback, I m happy to hear it (see the contact link at the bottom of every Code Search site).

Gregor Herrmann: GDAC 2014/3

as russ wrote some weeks ago in an excellent mail to debian-vote, upstreams are the raison d' tre for linux distributions. my experience in collaborating with upstream authors is mostly very positive, the most recent example from today being #728345. thanks to all upstream authors for their passion in writing software & sharing it. & for caring about it later as well!
this posting is part of GDAC (gregoa's debian advent calendar), a project to show the bright side of debian & why it's fun for me to contribute.

2 December 2014

Sven Hoexter: Out of the comfort zone: Backporting net-snmp from CentOS 7 to CentOS 5 and 6

Lately I faced the requirement to create a backport of net-snmp 5.7.2, which is part of CentOS 7, to a largely CentOS 5 and 6 based infrastructure. Due to some prior adventures in the rpm world I already knew about Mock (the Fedora equivalent to pbuilder). The easy part was the backporting to CentOS 6. Here the major objectiv is ripping out the newly added systemd support. That boils down to just ripping out a patch, removing a configure flag, some %post, %preun, %postun script snippets and the corresponding %install magic from the spec file. Builds fine but leaves us with the issue that the init script ships in a package net-snmp-sysvinit but the whole chkconfig registration magic is gone. I put that aside for a moment and moved on to CentOS 5. It did not even start to build.
  1. There is no tcp_wrappers-devel in CentOS 5.
  2. The perl libraries are in the main perl package and not in perl-devel.
  3. lm_sensors-devel is not at version 3.
I lowered the versioned build-dep on lm_sensors, disabled via a spec file toggle the tcp-wrapper support and changed the perl build-dep. This time it failed in the configure script due to DTLS support issues with the ancient openssl in CentOS 5. Decided to drop that one as well from the list of configure flags since we do not use it. Tried to build it again, now it failed somewhere down the road post compilation when assembling the package. Not cool. At that time we pushed a few other ideas back and forth. We found an isolated patch for net-snmp 5.6 that could fix our most pressing issue and someone had a look if that could be backported to CentOS 5 easily. Nope, internal structure of net-snmp changed a lot in between. So back to backporting. After staring long enough at the build log I could understand that the variable % RPM_BUILD_ROOT is empty in the CentOS 5 world. And indeed the BuildRoot stanza is also gone from the spec file. So I reintroduced that one and later even found documentation in the Fedora Wiki about this issue. If you know it it's easy to work around it, so I just defined it by hand: 
BuildRoot: % _tmppath /% name -% version -% release -root
%global RPM_BUILD_ROOT % buildroot
About ten minutes later I had a set of rpm files for CentOS 5. Tried to install it, failed due to a depedency on libmysqlclient15 which is part of the mysql package in CentOS 5. Yes, eight years back RedHat did not split off the lib packages and that would mean for us to install a mysql database on every system. So no, we won't do it and we just removed the mysql support. Left with only the init script setup missing I tried to be clever and failed miserable. Be warned, this would've also been a bad idea in the Debian world. Since we have the init script in a seperate binary package I thought I could just use something like 
Requires: net-snmp-sysvinit
in the main net-snmp package and re-add the old %post, %preun and %postun magic. Of course that failed at install time for a good reason. In the second (desperate) try to work around it I tried something similar to "PreDepends" 
Requires(post): net-snmp-sysvinit
That works at install time but still breaks on a removal/update. So I think I did the right thing in the end (after reading some more documentation) and just added the scripts to the net-snmp-sysvinit package. It's just about adding the package name to the %post macro invocation. My initial plan included the wish to have one spec file that is usable to build on CentOS 5 and 6 but now I'm left with a number of special cases for CentOS 5. I already knew about the %if %endif conditional macro you can use in rpm spec files, so what was missing was a proper variable to base my decisions on. Every CentOS release ships with a "/etc/rpm/" directory that includes several macro file. Within those one can find the % rhel variable that defines an int we can use. 
%if % rhel  <= 5
BuildRequires: perl
%else
BuildRequires: perl-devel
%endif
And finally I could build from one spec file a backport for CentOS 5 and 6. My diff against the CentOS 7 spec is at https://sven.stormbind.net/centos/netsnmp-bp-el56/net-snmp-backport.from_el7.diff Drawbacks of the resulting packages:
  1. No tcp-wrapper support, we don't care but that should be just another package name issue for the conditional Requires, simililar to the perl isssue.
  2. No mysql support. We do not care, one could change that and/or make it conditional or build against newer mysql releases where the -lib package is split.
  3. No DTLS support on CentOS 5. We do not care and that is not easy to fix.
  4. You've to install the net-snmp-sysvinit script to get back the usual setup you'd expect.
  5. We ripped out the systemd support, I did not investigate yet if it's possible to keep that included and just not enable it.
By now CentOS 5 is old. It's painful to backport to it. But it's still possible. Another strategy someone tried was packaging the current net-snmp upstream release based on the CentOS 5 spec file. That would've meant throwing away all 106 patches RedHat includes in that package at the moment and to depend on new upstream releases for your own security support. In case we start using CentOS 7 we would also diverge on the version of snmpd we use. So we decided to stick with the RedHat sources and keep the version consistent on our infrastructure even if we can not keep feature parity. After all the issues one encounters while backporting packages in the rpm world are similar to those in the dpkg world. Still it feels different if you're not familar with the toolchain. For example
  1. If you build a src.rpm with mock you've to copy it out of the result directory before you rebuild it with mock to create the binary package. Mock deletes everything from the result directory when starting a new build. So I guess parallel builds for the same chroot will not work.
  2. Depedencies on files instead of packages are a strange thing.
  3. How on earth do you maintain a patchset of over a hundred patches without something like quilt? I guess you always have to apply them in a for loop on your source directory and then you create a copy of that to implement your modifications and create another patch. I'm pretty sure eight years ago Fedora was far away from using git so I would rule out patch branches in a VCS for the moment. Not sure what they invented inside RedHat to keep track of it.

Rapha&#235;l Hertzog: My Free Software Activities in November 2014

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS This month I have been paid to work 18 hours on Debian LTS (14h allocated by Freexian + 4h I did not spend last month). I did the following tasks: Updating the linux source package took a good half of the allocated time. We opted to update the kernel to the upstream version 2.6.32.64. I integrated the upstream patches and identified about 130 patches that we had to disable (because they were already integrated upstream). Then I updated our openvz flavor patch to apply on top of the new kernel. This required quite a bit of manual conflict resolution and there are even parts where I was not sure that I took the correct decision. I was not able to find an upstream openvz git tree on this kernel version to to double check. Instead I asked Ben Hutchings to review my patch. He told me that he did not volunteer to work on LTS, but that he would be open to contribute to it for money. Following this remark, as the coordinator of Freexian s offer, I offered him to join to the set of paid LTS contributors to take care of the kernel and he accepted. So hopefully we will be able to wrap this linux upload in the first week of december. We had no uploads of the kernel in Squeeze since July so it s good to know that we now have someone who will be able to handle it in priority. Distro Tracker No new developments this month. Instead I spent some time to import old historic news so that when you lookup removed packages you have some actual content instead of a 404 error. For example you can look at python2.1. Another thing that I did is to tag some bugs with the newly-announced tag newcomer . Those are easy bugs that are ideal targets for new contributors who d like to get started: here s the list. It s up to you now! ;-) DEP-14: Recommended layout for Git packaging repositories I have drafted an initial version of a document called Recommended layout for Git packaging repositories and submitted it for discussion on debian-devel. The discussion has been interesting and constructive (yes this is still possible in Debian!). I have a bunch of improvements in my local copy and needs to process a few more feedback before submitting an updated draft. It s not a revolution but it s a good step to try to standardize tags and branches naming conventions. Systemd, the tech-ctte and our mailing lists As an old-timer, I care a lot about the governance of Debian and it s annoying to see how the systemd debate brought back some of our old daemons in terms of hostile atmosphere on our mailing lists. We can disagree on a lot of things, but we must respect each other and we are here to work together on solutions for everybody. As such I wrote to the persons who cross the line to invite them to behave better. And I m glad that our listmasters are backing up our calls with bans when appropriate. I believe we must go further in that direction and I shared an idea (on a debian-private thread that should have never existed, much like most of the traffic on that list) that I shall formalize and share on debian-project@l.d.o at some point. At the same time, we also had another governance-related discussion with the idea to impose some turnover in the technical committee. I m glad to see that we will soon vote on this topic. This is a good thing in general even though we just had 3 tech-ctte members who retired. Misc stuff I sponsored an upload of galette and of 3 of its plugins. I reviewed jitsi-videobridge and jitsi-meet on mentors.debian.net. I filed a few bugs: Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Matthias Klumpp: How to build a cross-distro package with Limba

Disclaimer: Limba is stilllimba-small in a very early stage of development. Bugs happen, and I give to guarantees on API stability yet. Limba is a very simple cross-distro package installer, utilizing OverlayFS found in recent Linux kernels (>= 3.18). As example I created a small Limba package for one of the Qt5 demo applications, and I would like to share the process of creating Limba packages it s quite simple, and I could use some feedback on how well the resulting packages work on multiple distributions. I assume that you have compiled Limba and installed it how that is done is described in its README file. So, let s start. 1. Prepare your application The cool thing about Limba is that you don t really have to do many changes on your application. There are a few things to pay attention to, though: This needs to be done so your application will find its data at runtime. Additionally, you need to write an AppStream metadata file, and find out which stuff your application depends on. 2. Create package metadata & install software 1.1 Basics Now you can create the metadata necessary to build a Limba package. Just run 
cd /path/to/my/project
lipkgen make-template
This will create a pkginstall directory, containing a control file and a metainfo.xml file, which can be a symlink to the AppStream metadata, or be new metadata. Now, configure your application with /opt/bundle as install prefix (-DCMAKE_INSTALL_PREFIX=/opt/bundle, prefix=/opt/bundle, etc.) and install it to the pkginstall/inst_target directory. 1.2 Handling dependencies If your software has dependencies on other packages, just get the Limba packages for these dependencies, or build new ones. Then place the resulting IPK packages in the pkginstall/repo directory. Ideally, you should be able to fetch Limba packages which contain the software components directly from their upstream developers. Then, open the pkginstall/control file and adjust the Requires line. The names of the components you depend on match their AppStream-IDs (<id/> tag in the AppStream XML document). Any version-relation (>=, >>, <<, <=, <>) is supported, and specified in brackets after the component-id. The resulting control-file might look like this: 
Format-Version: 1.0

Requires: Qt5Core (>= 5.3), Qt5DBus (>= 5.3), libpng12
If the specified dependencies are in the repo/ subdirectory, these packages will get installed automatically, if your application package is installed. Otherwise, Limba depends on the user to install these packages manually there is no interaction with the distribution s package-manager (yet?). 3. Building the package In order to build your package, make sure the content in inst_target/ is up to date, then run 
lipkgen build pkginstall/
This will build your package and output it in the pkginstall/ directory. 4. Testing the package You can now test your package, Just run 
sudo lipa install package.ipk
Your software should install successfully. If you provided a .desktop file in $prefix/share/applications, you should find your application in your desktop s application-menu. Otherwise, you can run a binary from the command-line, just append the version of your package to the binary name (bash-comletion helps). Alternatively, you can use the runapp command, which lets you run any binary in your bundle/package, which is quite helpful for debugging (since the environment a Limba-installed application is run is different from the one of other applications). Example: 
runapp $ component_id -$ version :/bin/binary-name
And that s it! :-) I used these steps to create a Limba package for the OpenGL Qt5 demo on Tanglu 2 (Bartholomea), and tested it on Kubuntu 15.04 (Vivid) with KDE, as well as on an up-to-date Fedora 21, with GNOME and without any Qt or KDE stuff installed: qt5demo-limba-kubuntuqt5demo-limba-fedora I encountered a few obstacles when building the packages, e.g. Qt5 initially didn t find the right QPA plugin that has been fixed by adjusting a config file in the Qt5Gui package. Also, on Fedora, a matching libpng was missing, so I included that as well. You can find the packages at Github, currently (but I am planning to move them to a different place soon). The biggest issue with Limba is at time, that it needs Linux 3.18, or an older kernel with OverlayFS support compiled in. Apart from that and a few bugs, the experience is quite smooth. As soon as I am sure there are now hidden fundamental issues, I can think of implementing more features, like signing packages and automatically updating them. Have fun playing around with Limba!

Peter Eisentraut: ccache and clang, part 3

In part 1 and part 2 I investigated how to use ccache with clang. That was more than three years ago. I got an email the other day that ccache bug 8118, which I filed while writing part 1, was closed, as ccache 3.2 was released. The release notes of clang 3.2 contain several items related to clang. So it was time to give this another look. Basically, the conclusions from part 2 still stand: You cannot use ccache with clang without using CCACHE_CPP2. And it is now becoming clear to me that this is an issue that is not going to go away, and it s not really even Clang s fault. Warnings! The problem is that clang s -Wall can cause warnings when compiling the preprocessed version of otherwise harmless C code. This can be illustrated by this piece of C code: 
int
foo()
 
        int *p, *q;
        p = q = 0;
        p = p;
        if (p == p)
                return 1;
        if ((p == q))
                return 2;
        return 0;
When compiled by gcc-4.9 -Wall, this gives no warnings. When compiled by clang-3.5 -Wall, this results in test.c:7:4: warning: explicitly assigning value of variable of type 'int *' to itself [-Wself-assign] test.c:8:8: warning: self-comparison always evaluates to true [-Wtautological-compare] test.c:10:9: warning: equality comparison with extraneous parentheses [-Wparentheses-equality] test.c:10:9: note: remove extraneous parentheses around the comparison to silence this warning test1.c:10:9: note: use '=' to turn this equality comparison into an assignment You wouldn t normally write code like this, but the C preprocessor could create code with self-assignments, self-comparisons, extra parentheses, and so on. This example represents the issues I saw when trying to compile PostgreSQL 9.4 with ccache and clang; there might be others. You can address this issue in two ways:
  1. Use CCACHE_CPP2, as discussed in part 2. With ccache 3.2, you can now also put this into a configuration file: run_second_cpp = true in ~/.ccache/ccache.conf
  2. Turn off the warnings mentioned above: -Wno-parentheses-equality, -Wno-tautological-compare, -Wno-self-assign (and any others you might find). One might think that these are actually useful warnings that one might want to keep, but GCC doesn t warn about them, and if you develop primarily with GCC, your code might contain these issues anyway. In particular, I have found that -Wno-tautological-compare is necessary for legitimate code.
I think CCACHE_CPP2 is the way to go, for two reasons. Firstly, having to add more and more options to turn off warnings is obviously somewhat stupid. Secondly and more importantly, there is nothing stopping GCC from adding warnings similar to Clang s that would trigger on preprocessed versions of otherwise harmless C code. Unless they come up with a clever way to annotate the preprocessed code to the effect of this code might look wrong to you, but it looked OK before preprocessing, so don t warn about it , in a way that creates no extra warnings and doesn t lose any warnings, I don t think this issue can be solved. Speed! Now the question is, how much would globally setting CCACHE_CPP2 slow things down? To test this, I have built PostgreSQL 9.4rc1 with clang-3.5 and gcc-4.8 (not gcc-4.9 because it creates some unrelated warnings that I don t want to deal with here). I have set export CCACHE_RECACHE=true so that the cache is not read but new cache entries are computed. That way, the overhead of ccache on top of the compiler is measured. Results: (There different percentages between gcc and clang arise because gcc is faster than clang (yes, really, more on that in a future post), but the overhead of ccache doesn t change.) 10% or so is not to be dismissed, but let s remember that this applies only if there is a cache miss. If everything is cached, both methods do the same thing. Also, if you use parallel make, the overhead is divided by the number of parallel jobs. With that in mind, I have decided to put the issue to rest for myself and have made myself a ~/.ccache/ccache.conf containing 
run_second_cpp = true
Now Clang or any other compiler should run without problems through ccache. Color! There is one more piece of news in the new ccache release: Another thing I talked about in part 1 was that ccache will disable the colored output of clang, and I suggested workarounds. This was actually fixed in ccache 3.2, so the workarounds are no longer necessary, and the above configuration change is really the only thing to make Clang work smoothly with ccache.

1 December 2014

Olivier Berger: Offline backup/mirror of a Moodle course, using httrack

I havent found much details online on how to perform a Moodle course mirror that could be browsed offline using httrack. This could be useful both for backup purposes, or for distant learners with connectivity issues. In my case, there s a login/password dialog that grants access to the moodle platform, which can be processed by httrack by capturing the POST form results using the catchurl option. The strategy I ve used is to add filters so that everything is excluded and only explicitely mentioned filters are then allowed to be mirrored. This allows to perform the backup connected with a user that may have high privileges, while avoiding to disappear in loops or complex links following for UI rendering variants of Moodle s interface.
Here s an example command line :
httrack -v -z -%F "Mirrored [from host %s [file %s [at %s]]]" -N "%h%p/%n%[id].%t" "http://mymoodle.example.com/login/index.php?>postfile:/home/myself/websites/mycourse/hts-post0" "http://mymoodle.example.com/course/view.php?id=42" -O "/home/myself/websites/mycourse" "-*/*" "+/login/index.php*" "+*/course/view.php*" "+*/mod/resource/view.php*" "+*/mod/page/view.php*" "+*/mod/forum/view.php*" "+*/mod/forum/discuss.php?d=*[0-9]" "+*/mod/url/view.php*" "+*/pluginfile.php/*" "+*/mod/feedback/view.php*" "+*/mod/feedback/analysis.php*" "+*/theme/*" "-*/course/view.php?id=43"
 Let s comment on these different parameters : I hope this will be useful.

30 November 2014

Enrico Zini: cxx11-talk-examples

C++11 talk examples On 2014-11-27 I gave a talk about C++ and new features introduced with C++11: these are the examples. They are all licensed under the wtfpli version 2. See cxx11-talk-notes for the talk notes. Note that the wrapper interfaces turns errors from the underlying libraries into exceptions, so the method calls just do what they should, without the need of documenting special return values for error messages, and removing the need for each library to implement yet another way of reporting errors. Also note that all wrapper objects do RAII: you create them and they clean after themselves when they go out of scope. The wrapper objects also have cast operators to make them behave as the pointer or handle that they are wrapping, so that they can be transparently passed to the underlying libraries. (note: I had to add U+2063 INVISIBLE SEPARATOR to prevent noreturn statements to be misinterpreted by the blog formatter. If you copypaste the code and encounter issues, you may want to delete the noreturn statements and retype them) A gcrypt hash class This class is a light wrapper around gcrypt's hashing functions. ezhash.h 
#ifndef EZHASH_H
#define EZHASH_H
#include <string>
#include <gcrypt.h>
namespace ezhash  
class Hash
 
protected:
    // members can now be initialized just like this, without needing to repeat
    // their default assignment in every constructor
    gcry_md_hd_t handle = nullptr;
public:
    Hash(int algo, unsigned int flags=0);
    ~Hash();
    // Assign 'delete' to a method to tell the compiler not to generate it
    // automatically. In this case, we make the object non-copiable.
    Hash(const Hash&) = delete;
    Hash(const Hash&&) = delete;
    Hash& operator=(const Hash&) = delete;
    // Add a buffer to the hash
    void hash_buf(const std::string& buf);
    // Add the contents of a file to the hash
    void hash_file(int fd);
    // Get a string with the hexadecimal hash
    std::string read_hex(int algo=0);
    /// Pretend that we are a gcry_md_hd_t handle
    operator gcry_md_hd_t()   return handle;  
 ;
 
#endif
ezhash.cpp 
#include "ezhash.h"
#include <unistd.h>
#include <errno.h>
#include <string>
#include <cstring>
#include <sstream>
#include <iomanip>
#include <stdexcept>
using namespace std;
namespace ezhash  
namespace  
// noreturn attribute, to tell the compiler that this function never returns
[ [noreturn]] void throw_gcrypt_error(gcry_error_t err)
 
    string msg;
    msg += gcry_strsource(err);
    msg += "/";
    msg += gcry_strerror(err);
    throw runtime_error(msg);
 
string errno_str(int error)
 
    char buf[256];
#if (_POSIX_C_SOURCE >= 200112L   _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE
    strerror_r(errno, buf, 256);
    string res(buf);
#else
    string res(strerror_r(errno, buf, 256));
#endif
    return res;
 
[ [noreturn]] void throw_libc_error(int error)
 
    throw runtime_error(errno_str(error));
 
 
Hash::Hash(int algo, unsigned int flags)
 
    gcry_error_t err = gcry_md_open(&handle, algo, flags);
    if (err) throw_gcrypt_error(err);
 
Hash::~Hash()
 
    gcry_md_close(handle);
 
void Hash::hash_buf(const std::string& buf)
 
    gcry_md_write(handle, buf.data(), buf.size());
 
void Hash::hash_file(int fd)
 
    char buf[4096];
    while (true)
     
        ssize_t res = ::read(fd, buf, 4096);
        if (res < 0) ezfs::throw_libc_error();
        if (res == 0) break;
        gcry_md_write(handle, buf, res);
     
 
std::string Hash::read_hex(int algo)
 
    unsigned char* res = gcry_md_read(handle, algo);
    unsigned int len = gcry_md_get_algo_dlen(
            algo == 0 ? gcry_md_get_algo(handle) : algo);
    // Format the hash into a hex digit
    stringstream hexbuf;
    hexbuf << hex << setfill('0');
    for (unsigned i = 0; i < len; ++i)
        hexbuf << setw(2) << (unsigned)res[i];
    return hexbuf.str();
Example usage 
        ezhash::Hash sha256(GCRY_MD_SHA256);
        sha256.hash_buf("ciao\n");
        sha256.hash_buf("foo\n");
        cout << sha256.read_hex() << endl;
Simple sqlite bindings Remarkably simple sqlite3 bindings based on lambda callbacks. ezsqlite.h 
#ifndef EZSQLITE_H
#define EZSQLITE_H
#include <sqlite3.h>
#include <string>
#include <functional>
#include <stdexcept>
namespace ezsqlite  
/// RAII wrapper around a sqlite3 database handle
class DB
 
protected:
    sqlite3* handle = nullptr;
public:
    // Open a connection to a SQLite database
    DB(const std::string& filename, int flags=SQLITE_OPEN_READWRITE   SQLITE_OPEN_CREATE);
    DB(const DB&) = delete;
    DB(const DB&&) = delete;
    DB& operator=(const DB&) = delete;
    ~DB();
    /**
     * Execute a query, optionally calling 'callback' on every result row
     *
     * The arguments to callback are:
     *  1. number of columns
     *  2. text values of the columns
     *  3. names of the columns
     */
    // std::function can be used to wrap any callable thing in C++
    // see: http://en.cppreference.com/w/cpp/utility/functional/function
    void exec(const std::string& query, std::function<bool(int, char**, char**)> callback=nullptr);
    /// Pretend that we are a sqlite3 pointer
    operator sqlite3*()   return handle;  
 ;
 
#endif
ezsqlite.cpp 
#include "ezsqlite.h"
namespace ezsqlite  
DB::DB(const std::string& filename, int flags)
 
    int res = sqlite3_open_v2(filename.c_str(), &handle, flags, nullptr);
    if (res != SQLITE_OK)
     
        // From http://www.sqlite.org/c3ref/open.html
        // Whether or not an error occurs when it is opened, resources
        // associated with the database connection handle should be
        // released by passing it to sqlite3_close() when it is no longer
        // required.
        std::string errmsg(sqlite3_errmsg(handle));
        sqlite3_close(handle);
        throw std::runtime_error(errmsg);
     
 
DB::~DB()
 
    sqlite3_close(handle);
 
namespace  
// Adapter to have sqlite3_exec call a std::function
int exec_callback(void* data, int columns, char** values, char** names)
 
    std::function<bool(int, char**, char**)> cb = *static_cast<std::function<bool(int, char**, char**)>*>(data);
    return cb(columns, values, names);
 
 
void DB::exec(const std::string& query, std::function<bool(int, char**, char**)> callback)
 
    char* errmsg;
    void* cb = callback ? &callback : nullptr;
    int res = sqlite3_exec(handle, query.c_str(), exec_callback, cb, &errmsg);
    if (res != SQLITE_OK && errmsg)
     
        // http://www.sqlite.org/c3ref/exec.html
        //
        // If the 5th parameter to sqlite3_exec() is not NULL then any error
        // message is written into memory obtained from sqlite3_malloc() and
        // passed back through the 5th parameter. To avoid memory leaks, the
        // application should invoke sqlite3_free() on error message strings
        // returned through the 5th parameter of of sqlite3_exec() after the
        // error message string is no longer needed. 
        std::string msg(errmsg);
        sqlite3_free(errmsg);
        throw std::runtime_error(errmsg);
Example usage 
    // Connect to the database
    ezsqlite::DB db("erlug.sqlite");
    // Make sure we have a table
    db.exec(R"(
        CREATE TABLE IF NOT EXISTS files (
                name TEXT NOT NULL,
                sha256sum TEXT NOT NULL
        )
    )");
    // Read the list of files that we know
    map<string, string> files;
    db.exec("SELECT name, sha256sum FROM files", [&](int columns, char** vals, char** names)  
        if (columns != 2) return false;
        files.insert(make_pair(vals[0], vals[1]));
        return true;
     );
A fast Directory object This is a lightweight wrapper around O_PATH file descriptors for directories. I'd love to see a library of well-maintained and thin C++ bindings around libc, that do little more than turning errors into exceptions and making it also work with std::string buffers. ezfs.h 
#ifndef EZFS_H
#define EZFS_H
#include <string>
#include <functional>
#include <memory>
#include <cerrno>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <dirent.h>
namespace ezfs  
class Directory
 
protected:
    int handle = -1;
public:
    Directory(const std::string& pathname, int flags=0);
    ~Directory();
    Directory(const Directory&) = delete;
    Directory(const Directory&&) = delete;
    Directory& operator=(const Directory&) = delete;
    /// List the directory contents
    void ls(std::function<void(const dirent&)> callback);
    int open(const std::string& relpath, int flags, mode_t mode=0777);
 ;
std::string errno_str(int error=errno);
[ [noreturn]] void throw_libc_error(int error=errno);
 
#endif
ezfs.cpp 
#include "ezfs.h"
#include <stdexcept>
#include <memory>
#include <cstring>
#include <cstdlib>
#include <string>
#include <linux/limits.h>
using namespace std;
namespace ezfs  
string errno_str(int error)
 
    char buf[256];
#if (_POSIX_C_SOURCE >= 200112L   _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE
    strerror_r(errno, buf, 256);
    string res(buf);
#else
    string res(strerror_r(errno, buf, 256));
#endif
    return res;
 
[ [noreturn]] void throw_libc_error(int error)
 
    throw runtime_error(errno_str(error));
 
Directory::Directory(const std::string& pathname, int flags)
 
    handle = ::open(pathname.c_str(), O_PATH   O_DIRECTORY   flags);
    if (handle < 0) throw_libc_error();
 
Directory::~Directory()
 
    ::close(handle);
 
void Directory::ls(std::function<void(const dirent&)> callback)
 
    int fd = openat(handle, ".", O_DIRECTORY);
    if (fd < 0) throw_libc_error();
    // RAII Self-cleaning DIR object
    unique_ptr<DIR, std::function<void(DIR*)>> dir(fdopendir(fd), [](DIR* dir)   if (dir) closedir(dir);  );
    if (!dir)
     
        // fdopendir(3): After a successful call to fdopendir(), fd is used
        // internally by the implementation, and should not otherwise be used
        // by the application.
        //
        // but if the fdopendir call was not successful, fd is not managed by
        // DIR, and we still need to close it, otherwise we leak a file
        // descriptor.
        //
        // However, close() may modify errno, so we take note of the errno set
        // by fdopendir and raise the exception based on that.
        int fdopendir_errno = errno;
        close(fd);
        throw_libc_error(fdopendir_errno);
     
    // Size the dirent buffer properly
    const unsigned len = offsetof(dirent, d_name) + PATH_MAX + 1;
    unique_ptr<dirent, std::function<void(void*)>> dirbuf((dirent*)malloc(len), free);
    while (true)
     
        dirent* res;
        int err = readdir_r(dir.get(), dirbuf.get(), &res);
        // End of directory contents
        if (err == 0)
         
            if (res)
                callback(*res);
            else
                break;
          else
            throw_libc_error(err);
     
 
int Directory::open(const std::string& relpath, int flags, mode_t mode)
 
    int res = openat(handle, relpath.c_str(), flags, mode);
    if (res < 0) throw_libc_error();
    return res;
Example usage 
        // This is almost the equivalent of running "sha256sum ."
        ezfs::Directory dir(".");
        dir.ls([&](const dirent& d)  
            if (d.d_type != DT_REG) return;
            ezhash::Hash sha256(GCRY_MD_SHA256);
            // I have no RAII wrapper around file handles at the moment, so
            // I'll have to use a try/catch for cleaning up after errors
            int fd = dir.open(d.d_name, O_RDONLY);
            try  
                sha256.hash_file(fd);
                close(fd);
              catch (...)  
                close(fd);
                throw;
             
            cout << sha256.read_hex() << "  " << d.d_name << endl;
         );

Enrico Zini: cxx11-talk-notes

C++11 talk notes On 2014-11-27 I gave a talk about C++ and new features introduced with C++11: these are the talk notes. See cxx11-talk-examples for the examples. (note: I had to add U+2063 INVISIBLE SEPARATOR to prevent noreturn statements to be misinterpreted by the blog formatter. If you copypaste the code and encounter issues, you may want to delete the noreturn statements and retype them) Overview of programming languages
It has to be as fast as possible, so interpreted languages are out. You don't want to micro manage memory, so C is out. You don't want to require programmers to have a degree, so C++ is out. You want fast startup and not depend on a big runtime, so Java is out. [...] (Bram Moolenaar)
C++ secret cultist protip
Do not call up what you cannot put down.
C++ is a compiled language It is now possible to use the keyword constexpr to mark functions and objects that can be used at compile time: 
/*
 * constexpr tells the compiler that a variable or function can be evaluated at
 * compile time.
 *
 * constexpr functions can also be run at run time, if they are called with
 * values not known at compile time.
 *
 * See http://en.cppreference.com/w/cpp/language/constexpr for more nice examples
 *
 * It can be used to avoid using constants in code, and using instead functions
 * for computing hardware bitfields or physical values, without losing in
 * efficiency.
 */
#include <iostream>
using namespace std;
constexpr int factorial(int n)
 
    return n <= 1 ? 1 : (n * factorial(n-1));
 
int main()
 
    cout << "Compile time factorial of 6: " << factorial(6) << endl;
    cout << "Enter a number: ";
    int a;
    cin >> a;
    cout << "Run time factorial of " << a << ": " << factorial(a) << endl;
See also this for more nice examples. See this and this for further discussion. Multiline strings 
        const char* code = R"--(
          printf("foo\tbar\n");
          return 0;
        )--";
See this. C++ memory management protip RAII: Resource Acquisition Is Instantiation This is not new in C++11, but in my experience I have rarely seen it mentioned in C++ learning material, and it does make a huge difference in my code. See this and this for details. Constructors and member initializer lists Initializers in curly braces now have their own type: std::initializer_list: 
#include <string>
#include <iostream>
#include <unordered_set>
using namespace std;
// std::initializer_list< >
//   will have as its value all the elements inside the curly braces
string join(initializer_list<string> strings)
 
    string res;
    for (auto str: strings)
     
        if (!res.empty())
            res += ", ";
        res += str;
     
    return res;
 
int main()
 
    unordered_set<string> blacklist  ".", "..", ".git", ".gitignore"  ;
    cout << join(  "antani", "blinda"  ) << endl;
See this for details, including the new uniform initialization trick of omitting parentesis in constructors so that you can call normal constructors and initializer_list constructors with the same syntax, which looks like an interesting thing when writing generic code in templates. Type inference I can now use auto instead of a type to let the compiler automatically compute the value of something I assign to: 
        auto i = 3 + 2;
        // See also https://github.com/esseks/monicelli
        vector<string> names  "antani", "blinda", "supercazzola"  ;
        for (auto i = names.cbegin(); i != names.cend(); ++i)
            cout << i;
        template<typename T>
        T frobble(const T& stuff)
         
             // This will work whatever type is returned by stuff.read()
             auto i = stuff.read();
             //
See this for more details. Range-based for loop C++ now has an equivalent of the various foreach constructs found in several interpreted languages! 
        for (auto i: list_of_stuff)
                cout << i << endl;
        for (auto n:  0,1,2,3,4,5 )
                cout << n << endl;
        // This construct:
        for (auto i: stuff)
        // If stuff is an array, it becomes:
        for (i = stuff, i < stuff + sizeof(stuff) / sizeof(stuff[0]); ++i)
        // If stuff has .begin() and .end() methods it becomes:
        for (i = stuff.begin(); i != stuff.end(); ++i)
        // Otherwise it becomes:
        for (i = begin(stuff); i != end(stuff); ++i)
        // And you can define begin() and end() functions for any type you
        // want, at any time
See this and this for details. Lambda functions and expressions Lambdas! Closures! Something like this: 
// JavaScript
var add = function(a, b)   return a + b;  

# Python
add = lambda a, b: a + b
Becomes this: 
auto add = [](int a, int b)   return a + b;
And something like this: 
// JavaScript
var a = 0;
$.each([1, 2, 3, 4], function(idx, el)   a += el  );
Becomes this: 
unsigned a = 0;
std::for_each(  1, 2, 3, 4  , [&a](int el)   return a += el;  );
See this, this and this. Tuple types C++ now has a std::tuple type, that like in Python can be used to implement functions that return multiple values: 
        tuple<int, string, vector<string>> parse_stuff()
         
                return make_tuple(id, name, values);
         
        string name; vector<string> values;
        // std::ignore can be used to throw away a result
        tie(ignore, name, values) = parse_stuff();
        // std::tie can also be used to do other kind of
        // multi-operations besides assignment:
        return tie(a, b, c) < tie(a1, b1, c1);
        // Is the same as:
        if (a != a1) return a < a1;
        if (b != b1) return b < b1;
        return c < c1;
See here, here and here. Regular expressions We now have regular expressions! 
        std::regex re(R"((\w+)\s+(\w+))");
        string s("antani blinda");
        smatch res;
        if (regex_match(s, res, re))
            cout << "OK " << res[1] << " -- " << res[2] << endl;
The syntax is ECMAScript by default and can be optionally changed to basic, extended, awk, grep, or egrep. See here and here. General-purpose smart pointers There is std::unique_ptr to code memory ownership explicitly, and std::shared_ptr as a reference counted pointer, and smart pointers can have custom destructors: 
    unique_ptr<dirent, std::function<void(void*)>> dirbuf((dirent*)malloc(len), free);
See here and here. Miscellaneous other cool things Standard attribute specifiers 
string errno_str(int error)
 
    char buf[256];
#if (_POSIX_C_SOURCE >= 200112L   _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE
    strerror_r(errno, buf, 256);
    string res(buf);
#else
    string res(strerror_r(errno, buf, 256));
#endif
    return res;
 
[ [noreturn]] void throw_libc_error(int error)
 
    throw runtime_error(errno_str(error));
See here. Hash tables See here and look at the new containers unordered_set, unordered_map, unordered_multiset, and unordered_multimap. Multithreading There is a standard threading model, with quite a bit of library support: see here, here, here, and here for atomic data structures. Variadic templates Templates can now take variable number of arguments, and that opens possibilities for interesting code generation, like implementing a generic, type-safe printf statement, or something like this: 
db.query(R"(
   INSERT INTO table NAMES (id, name, description)
     VALUES (?, ?, ?)
)", 4, "genio", "fantasia, intuizione, decisione, e velocit  di esecuzione");
See here and here. Essential tools You need at least g++ 4.8 or clang 3.3 to have full C++11 support. They will be both available in jessie, and for wheezy you can use the nightly clang packages repository. I cannot think of a good excuse not to use -Wall on new code. scan-build from clang is another nice resource for catching even more potential problems at compile time. valgrind is a great tool for runtime code analysis: valgrind --tool=memcheck (the default) will check your program for wrong memory accesses and memory leaks. valgrind --tool=callgrind will trace function calls for profiling, to be analyzed with kcachegrind. valgrind --tool=helgrind can check multi-threaded programs for suspicious concurrent memory accesse patterns. And of course gdb: a nice trick with C++ is to issue catch throw to get a breakpoint at the point where an exception is being thrown. help catch provides a list of other interesting catch examples. Coredump tips: ulimit -c to enable core dumps, triggering a core dump with ^\, opening a core with gdb program core, and more details on man 5 core. An extra gdb tip, which is not related to C++ but helped me considerably recently, is that it can be attached to running python programs to get a live Python traceback.

Johannes Schauer: simple email setup

I was unable to find a good place that describes how to create a simple self-hosted email setup. The most surprising discovery was, how much already works after: 
apt-get install postfix dovecot-imapd
Right after having finished the installation I was able to receive email (but only in in /var/mail in mbox format) and send email (bot not from any other host). So while I expected a pretty complex setup, it turned out to boil down to just adjusting some configuration parameters.

Postfix The two interesting files to configure postfix are /etc/postfix/main.cf and /etc/postfix/master.cf. A commented version of the former exists in /usr/share/postfix/main.cf.dist. Alternatively, there is the ~600k word strong man page postconf(5). The latter file is documented in master(5).

/etc/postfix/main.cf I changed the following in my main.cf 
@@ -37,3 +37,9 @@
 mailbox_size_limit = 0
 recipient_delimiter = +
 inet_interfaces = all
+
+home_mailbox = Mail/
+smtpd_recipient_restrictions = permit_mynetworks reject_unauth_destination permit_sasl_authenticated
+smtpd_sasl_type = dovecot
+smtpd_sasl_path = private/auth
+smtp_helo_name = my.reverse.dns.name.com
At this point, also make sure that the parameters smtpd_tls_cert_file and smtpd_tls_key_file point to the right certificate and private key file. So either change these values or replace the content of /etc/ssl/certs/ssl-cert-snakeoil.pem and /etc/ssl/private/ssl-cert-snakeoil.key. The home_mailbox parameter sets the default path for incoming mail. Since there is no leading slash, this puts mail into $HOME/Mail for each user. The trailing slash is important as it specifies qmail-style delivery'' which means maildir. The default of the smtpd_recipient_restrictions parameter is permit_mynetworks reject_unauth_destination so this just adds the permit_sasl_authenticated option. This is necessary to allow users to send email when they successfully verified their login through dovecot. The dovecot login verification is activated through the smtpd_sasl_type and smtpd_sasl_path parameters. I found it necessary to set the smtp_helo_name parameter to the reverse DNS of my server. This was necessary because many other email servers would only accept email from a server with a valid reverse DNS entry. My hosting provider charges USD 7.50 per month to change the default reverse DNS name, so the easy solution is, to instead just adjust the name announced in the SMTP helo.

/etc/postfix/master.cf The file master.cf is used to enable the submission service. The following diff just removes the comment character from the appropriate section. 
@@ -13,12 +13,12 @@
 #smtpd     pass  -       -       -       -       -       smtpd
 #dnsblog   unix  -       -       -       -       0       dnsblog
 #tlsproxy  unix  -       -       -       -       0       tlsproxy
-#submission inet n       -       -       -       -       smtpd
-#  -o syslog_name=postfix/submission
-#  -o smtpd_tls_security_level=encrypt
-#  -o smtpd_sasl_auth_enable=yes
-#  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
-#  -o milter_macro_daemon_name=ORIGINATING
+submission inet n       -       -       -       -       smtpd
+  -o syslog_name=postfix/submission
+  -o smtpd_tls_security_level=encrypt
+  -o smtpd_sasl_auth_enable=yes
+  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
+  -o milter_macro_daemon_name=ORIGINATING
 #smtps     inet  n       -       -       -       -       smtpd
 #  -o syslog_name=postfix/smtps
 #  -o smtpd_tls_wrappermode=yes

Dovecot Since above configuration changes made postfix store email in a different location and format than the default, dovecot has to be informed about these changes as well. This is done in /etc/dovecot/conf.d/10-mail.conf. The second configuration change enables postfix to authenticate users through dovecot in /etc/dovecot/conf.d/10-master.conf. For SSL one should look into /etc/dovecot/conf.d/10-ssl.conf and either adapt the parameters ssl_cert and ssl_key or store the correct certificate and private key in /etc/dovecot/dovecot.pem and /etc/dovecot/private/dovecot.pem, respectively. The dovecot-core package (which dovecot-imapd depends on) ships tons of documentation. The file /usr/share/doc/dovecot-core/dovecot/documentation.txt.gz gives an overview of what resources are available. The path /usr/share/doc/dovecot-core/dovecot/wiki contains a snapshot of the dovecot wiki at http://wiki2.dovecot.org/. The example configurations seem to be the same files as in /etc/ which are already well commented.

/etc/dovecot/conf.d/10-mail.conf The following diff changes the default email location in /var/mail to a maildir in ~/Mail as configured for postfix above. 
@@ -27,7 +27,7 @@
 #
 # <doc/wiki/MailLocation.txt>
 #
-mail_location = mbox:~/mail:INBOX=/var/mail/%u
+mail_location = maildir:~/Mail
 
 # If you need to set multiple mailbox locations or want to change default
 # namespace settings, you can do it by defining namespace sections.

/etc/dovecot/conf.d/10-master.conf And this enables the authentication socket for postfix: 
@@ -93,9 +93,11 @@
    
 
   # Postfix smtp-auth
-  #unix_listener /var/spool/postfix/private/auth  
-  #  mode = 0666
-  # 
+  unix_listener /var/spool/postfix/private/auth  
+    mode = 0660
+    user = postfix
+    group = postfix
+   
 
   # Auth process is run as this user.
   #user = $default_internal_user

Aliases Now Email will automatically put into the '~/Mail' directory of the receiver. So a user has to be created for whom one wants to receive mail... 
$ adduser josch
...and any aliases for it to be configured in /etc/aliases. 
@@ -1,2 +1,4 @@
-# See man 5 aliases for format
-postmaster:    root
+root:       josch
+postmaster: josch
+hostmaster: josch
+webmaster:  josch
After editing /etc/aliases, the command 
$ newaliases
has to be run. More can be read in the aliases(5) man page.

Finishing up Everything is done and now postfix and dovecot have to be informed about the changes. There are many ways to do that. Either restart the services, reboot or just do: 
$ postfix reload
$ doveadm reload
Have fun!

Next.

Previous.