Changes in RcppArmadillo version 0.6.200.2.0 (2015-10-31)

• Upgraded to Armadillo 6.200.0 ("Midnight Blue Deluxe")
  • expanded diagmat() to handle non-square matrices and arbitrary diagonals
  • expanded trace() to handle non-square matrices

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

• Buy paper edition from lulu.com
• Download PDF, size 7.9 MiB (gratis/free)
• Download ePub, size 11 MiB (gratis/free)
• Download MOBI, size 3.8 MiB (gratis/free)

1. A CSV file can be uploaded containing user IDs and new passwords. The CSV file format for this is: comma-separated, no quotes, two columns (<uid>, <userPassword>).
2. An organizational unit of the LDAP tree can be selected and for all user accounts in that section of the LDAP tree (i.e., the selected OU and all sub-OUs), new passwords can be set. The passwords will get auto-generated when using this approach.
3. Other approach needed? Please get in touch and provide me with a description of your potential use case.

• Passwords can be adapted on screen (also the auto-generated passwords).
• Password changes for individual users can be skipped by unticking the user's "Really?" checkbox.
• Expect more features to come...

R> library(littler)
The littler package provides 'r' as a binary.
You could link to the 'r' binary installed in
'/usr/local/lib/R/site-library/littler/bin/r' from
'/usr/local/bin' in order to use 'r' for scripting.
R> 

Changes in littler version 0.3.0 (2015-10-29)

• Changes in build system
  • First CRAN Release as R package following nine years of source releases
  • Script configure, src/Makevars.in and remainder of build system rewritten to take advantage of the R package build infrastructure
  • Reproducible builds are better supported as the (changing) compilation timestamps etc are only inserted for 'verbose builds' directly off the git repo, but not for Debian (or CRAN) builds off the release tarballs
• Changes in littler functionality
  • Also source $R_HOME/etc/Rprofile.site and ~/.Rprofile if present
• Changes in littler documentation
  • Added new vignette with examples

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

SREs are not firefighters As I said before, it is very important that there is trust and respect between Dev and Ops. If Ops is seen as just an army of firefighters that will gladly put away fires at any time of the night, there are less incentives to make good software. Conversely, if Dev is shielded from the realities of the production environment, Ops will tend to think of them as delusional and untrustworthy.

Common staffing pool for SRE and SWE The first tool to combat this at Google -and possibly a difficult one to implement in small organisations- is to have single headcount budgets for Dev and Ops. That means that the more SREs you need to support your service, the less developers you have to write it. Combined with this, Google offers the possibility to SREs to move freely between teams, or even to transfer out to SWE. Because of this, a service that is painful to support will see the most senior SREs leaving and will only be able to afford less experienced hires. All this is a strong incentive to write good quality software, to work closely and to listen to the Ops people.

Share 5% of operational work with the SWE team On the other hand, it is necessary that developers see first hand how the service works in production, understand the problems, and share the pain of things failing. To this end, SWEs are expected to take on a small fraction of the operational work from SRE: handling tickets, being on-call, performing deployments, or managing capacity. This results in better communication among the teams and a common understanding of priorities.

Cap operational load at 50% One very uncommon rule from SRE at Google, is that SREs are not supposed to spend more than half of their time on "operational work". SREs are supposed to be spending their time on automation, monitoring, forecasting growth... Not on repeatedly fixing manually issues that stem from bad systems.

Excess operational work overflows to SWE If an SRE team is found to be spending too much time on operational work, that extra load is automatically reassigned to the development team, so SRE can keep doing their non-operational duties. On extreme cases, a service might be deemed too unstable to maintain, and SRE support is completely removed: it means the development team now has to carry pagers and do all the operational work themselves. It is a nuclear option, but the threat of it happening is a strong incentive to keep things sane.

---

The next post will be less about how to avoid unwanted work and more about the things that SRE actually do, and how these make things better. Comment

> import System.IO
> import Control.Concurrent.Async
> putStrLn (repeat 'A')  concurrently  putStrLn (repeat 'B')
BABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABA
BABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABABA
...

> putStrLn "washed the car"  concurrently  putStrLn "walked the dog"
walwkaesdh etdh et hdeo gc
ar

outputConcurrent :: String -> IO ()
    
createProcessConcurrent :: CreateProcess -> IO whatever

import Control.Concurrent.Async
import System.Console.Concurrent
import System.Process
main = withConcurrentOutput $
    outputConcurrent "washed the car\n"
         concurrently 
    createProcessConcurrent (proc "ls" [])
         concurrently 
    outputConcurrent "walked the dog\n"

$ ./demo
washed the car
walked the dog
Maildir/  bin/  doc/  html/  lib/  mail/  mnt/  src/  tmp/

• It has to do buffering of the output.
• There can be any quantity of output, but program memory use should be reasonably small. Solved by buffering up to 1 mb of output in RAM, and writing excess buffer to temp files.
• Falling off the end of the program is complicated; there can be buffered output to flush and it may have to wait for some processes to finish running etc.
• The locking was tough to get right! I could not have managed to write it correctly without STM.

Dev vs Ops: the eternal battle So, it starts at looking at the problem: how to increase the reliability of the service? It turns out that some of the biggest sources of outages are new launches: a new feature that seemed innocuous somehow managed to bring the whole site down. Devs want to launch, and Ops want to have a quiet weekend, and this is were the struggle begins. When launches are problematic, bureaucracy is put in place to minimise the risks: launch reviews, checklists, long-lived canaries. This is followed by development teams finding ways of side-stepping those hurdles. Nobody is happy. One of the key aspects of SRE is to avoid this conflict completely, by changing the incentives, so these pressures between development and operations disappear. At Google, they achieve this with a few different strategies:

Have an SLA for your service Before any service can be supported by SRE, it has to be determined what is the level of availability that it must achieve to make the users and the company happy: this is called the Service Level Agreement (SLA). The SLA will define how availability is measured (for example, percentage of queries handled successfully in less than 50ms during the last quarter), and what is the minimum acceptable value for it (the Service Level Objective, or SLO). Note that this is a product decision, not a technical one. This number is very important for an SRE team and its relationship with the developers. It is not taken lightly, and it should be measured and enforced rigorously (more on that later). Only a few things on earth really require 100% availability (pacemakers, for example), and achieving really high availability is very costly. Most of us are dealing with more mundane affairs, and in the case of websites, there are many other things that fail pretty often: network glitches, OS freezes, browsers being slow, etc. So an SLO of 100% is almost never a good idea, and in most cases it is impossible to reach. In places like Google an SLO of "five nines" (99.999%) is not uncommon, and this means that the service can't fail completely for more than 5 minutes across a whole year!

Measure and report performance against SLA/SLO Once you have a defined SLA and SLO, it is very important that these are monitored accurately and reported constantly. If you wait for the end of the quarter to produce a hand-made report, the SLA is useless, as you only know you broke it when it is too late. You need automated and accurate monitoring of your service level, and this means that the SLA has to be concrete and actionable. Fuzzy requirements that can't be measured are just a waste of time. This is a very important tool for SRE, as it allows to see the progression of the service over time, detect capacity issues before they become outages, and at the same time show how much downtime can be taken without breaking the SLA. Which brings us to one core aspect of SRE:

Use error budgets and gate launches on them If SLO is the minimum rate of availability, then the result of calculating 1 - SLO is what fraction of the time a service can fail without failing out of the SLA. This is called an error budget, and you get to use it the way you want it. If the service is flaky (e.g. it fails consistently 1 of every 10000 requests), most of that budget is just wasted and you won't have any margin for launching riskier changes. On the other hand, a stable service that does not eat the budget away gives you the chance to bet part of it on releasing more often, and getting your new features quicker to the user. The moment the error budget is spent, no more launches are allowed until the average goes back out of the red. Once everyone can see how the service is performing against this agreed contract, many of the traditional sources of conflict between development and operations just disappear. If the service is working as intended, then SRE does not need to interfere on new feature launches: SRE trusts the developers' judgement. Instead of stopping a launch because it seems risky or under-tested, there are hard numbers that take the decisions for you. Traditionally, Devs get frustrated when they want to release, but Ops won't accept it. Ops thinks there will be problems, but it is difficult to back this feeling with hard data. This fuels resentment and distrust, and management is never pleased. Using error budgets based on already established SLAs means there is nobody to get upset at: SRE does not need to play bad cop, and SWE is free to innovate as much as they want, as long as things don't break. At the same time, this provides a strong incentive for developers to avoid risking their budget in poorly-prepared launches, to perform staged deployments, and to make sure the error budget is not wasted by recurrent issues.

---

That's all for today. The next article will continue delving into how traditional tensions between Devs and Ops are played in the SRE world. Comment

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

• The audio dropping was caused by an older HDMI cable being used. I didn't know this, but apparently there are several versions of HDMI wiring, and if an older cable is used then the amount of data that can be passed over the line is not as high. Since my older TV didn't do 1080p (only 1080i) I didn't notice this before getting the new set, but changing out some of the HDMI cables fixed that issue.
• After searching the settings a bit, I found that the TV does have a setting for making it route audio to the 5.1 amp, so I'm back to using CEC, which has several advantages.
• The hardcoding of one particular type of video to one particular input in the 5.1 amp that I complained about in that other post does seem to have at least some merit: it turns out that this is part of the CEC as well, and so when I change from HDMI input to broadcast data, the TV will automatically switch the amp's input to the "TV" input, too. That's pretty cool, even though I had to buy yet another cable (this time a TOSLINK one) to make it work well.

• Stefano Rivera uploaded python-cffi/1.3.0-1 which makes the generated code order deterministic for anonymous unions and anonymous structs. Reported by Tristan Seligmann, and fixed uptream.

• 389-ds-base/1.3.3.13-1 uploaded by Timo Aaltonen, original patch by Chris Lamb.
• apache2/2.4.17-1 by Stefan Fritsch.
• ben/0.7.3 by Mehdi Dogguy.
• cdo/1.6.9+dfsg.1-3 by Alastair McKinstry.
• epubcheck/4.0.0-2 by Eugene Zhukov.
• grads/2:2.0.2-8 by Alastair McKinstry.
• litl/0.1.7+dfsg-1 uploaded by Samuel Thibault, original patch by Chris Lamb.
• mia/2.2.5-1 by Gert Wollny.
• powerline/2.2-2 by Jerome Charaoui.
• python-oslotest/1:1.11.0-2 by Thomas Goirand.
• tth/4.05+ds-2 uploaded by Jerome Benoit, original patch by Reiner Herrmann.
• xbae/4.60.4-7 uploaded by Nicholas Breen, original patch by Chris Lamb.
• xdmf/2.1.dfsg.1-13 by Alastair McKinstry.

• foxeye/0.10.2-1 by Andriy Grytsenko.
• jaxe/3.5-6 by Samuel Thibault.
• ncurses/6.0+20151017-1 by Sven Joachim, original patch by Esa Peuha.
• olap4j/1.2.0-1 by Emmanuel Bourg.
• tomcat8/8.0.28-1 by Emmanuel Bourg.

Whilst upgrading various projects to Django 1.9, I found myself repeatedly searching for the following code snippet so I offer it below as a more permanent note for myself and to aid others. If you used django.template.base.add_to_builtins to avoid tedious and unsightly % load module % blocks in your template files, under Django 1.9 you will get the following traceback:

Traceback (most recent call last):
  File "django/core/management/__init__.py", line 324, in execute
    django.setup()
  File "django/__init__.py", line 18, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "django/apps/registry.py", line 108, in populate
    app_config.import_models(all_models)
  File "django/apps/config.py", line 202, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "myproject/myproject/utils/models.py", line 1, in <module>
    from django.template.base import add_to_builtins
ImportError: cannot import name add_to_builtins

TEMPLATES = [ 
    'BACKEND': 'django.template.backends.django.DjangoTemplates',
    'DIRS': [
        os.path.join(BASE_DIR, 'templates'),
    ],
    'APP_DIRS': True,
    'OPTIONS':  
        'context_processors': [
            'django.template.context_processors.debug',
            'django.template.context_processors.request',
            'django.contrib.auth.context_processors.auth',
            'django.contrib.messages.context_processors.messages',
            'myproject.utils.context_processors.settings_context',
        ],
        'builtins': [
            'django.contrib.staticfiles.templatetags.staticfiles',
        ],
     ,
 ]

mkdir /sys/fs/cgroup/net_cls/unlocator
/bin/echo 42 > /sys/fs/cgroup/net_cls/unlocator/net_cls.classid
chown md: /sys/fs/cgroup/net_cls/unlocator/tasks

#!/bin/sh -e
CGROUP_NAME=unlocator
if [ ! -d /sys/fs/cgroup/net_cls/$CGROUP_NAME/ ]; then
  echo "The $CGROUP_NAME net_cls cgroup does not exist!" >&2
  exit 1
fi
/bin/echo $$ > /sys/fs/cgroup/net_cls/$CGROUP_NAME/tasks
if [ $# = 0 ]; then
  exec $ SHELL:-/bin/sh 
fi
exec "$@"

port=5354
listen-address=127.0.0.1
bind-interfaces
no-dhcp-interface=*
no-hosts
no-resolv
server=185.37.37.37
server=185.37.37.185

[Unit]
Description=dnsmasq - Second instance
Requires=network.target
[Service]
ExecStartPre=/usr/sbin/dnsmasq --test
ExecStart=/usr/sbin/dnsmasq --keep-in-foreground --conf-file=/etc/dnsmasq2.conf
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/run/dnsmasq/dnsmasq.pid
[Install]
WantedBy=multi-user.target

systemctl enable dnsmasq2
systemctl start dnsmasq2

git clone git://git.netfilter.org/iptables.git
cd iptables
./autogen.sh
./configure
make -k
sudo cp extensions/libxt_cgroup.so /lib/xtables/
sudo chmod -x /lib/xtables/libxt_cgroup.so

iptables -t nat -A OUTPUT -m cgroup --cgroup 42 -p udp --dport 53 -j REDIRECT --to-ports 5354
iptables -t nat -A OUTPUT -m cgroup --cgroup 42 -p tcp --dport 53 -j REDIRECT --to-ports 5354

ip6tables -A OUTPUT -m cgroup --cgroup 42 -j REJECT

iptables -t mangle -A OUTPUT -m cgroup --cgroup 43 -j MARK --set-mark 43

ip rule add fwmark 43 table 43

ip route add default dev tun0 table 43

ip route show scope link proto kernel \
    xargs -I ROUTE ip route add ROUTE table 43

iptables -t nat -A POSTROUTING -m mark --mark 43 --out-interface tun0 -j MASQUERADE

• Github of db2twitter (star it if you like ;)
• Official documentation on readthedocs

dbhost=localhost
database=mytuxjobs
dbuser=mytuxjobs
dbpass=V3rYs3cr3t!
dbtables=jobs,
jobs_rows=title,id

tweet=MyTux hires a   https://www.mytux.fr/jobs/ "

--
-- This method returns the text which is displayed when a maildir is
-- to be show in maildir-mode.
--
function Maildir.to_string(self)
   local total  = self:total_messages()
   local unread = self:unread_messages()
   local path   = self:path()
   local output = string.format( "[%05d / %05d] - %s", unread, total, path );
   if ( unread > 0 ) then
      output = "$[RED]" .. output
   end
   if ( string.find( output, "Automated." ) ) then
      output = strip_colour( output )
      output = "$[YELLOW]" .. output
   end
   return output
end

• https://github.com/lumail/lumail2/

• vejeta (Juan, perhaps the main supporter of the meeting, Debian user)
• Pablo Neira (Netfilter upstream project and Linux kernel maintainer)
• Javier Barroso (IT architect in the local goverment and Debian user)
• me (the only directly involved in the Debian project, I'm DM)