Dev vs Ops: the eternal battle So, it starts at looking at the problem: how to increase the reliability of the service? It turns out that some of the biggest sources of outages are new launches: a new feature that seemed innocuous somehow managed to bring the whole site down. Devs want to launch, and Ops want to have a quiet weekend, and this is were the struggle begins. When launches are problematic, bureaucracy is put in place to minimise the risks: launch reviews, checklists, long-lived canaries. This is followed by development teams finding ways of side-stepping those hurdles. Nobody is happy. One of the key aspects of SRE is to avoid this conflict completely, by changing the incentives, so these pressures between development and operations disappear. At Google, they achieve this with a few different strategies:

Have an SLA for your service Before any service can be supported by SRE, it has to be determined what is the level of availability that it must achieve to make the users and the company happy: this is called the Service Level Agreement (SLA). The SLA will define how availability is measured (for example, percentage of queries handled successfully in less than 50ms during the last quarter), and what is the minimum acceptable value for it (the Service Level Objective, or SLO). Note that this is a product decision, not a technical one. This number is very important for an SRE team and its relationship with the developers. It is not taken lightly, and it should be measured and enforced rigorously (more on that later). Only a few things on earth really require 100% availability (pacemakers, for example), and achieving really high availability is very costly. Most of us are dealing with more mundane affairs, and in the case of websites, there are many other things that fail pretty often: network glitches, OS freezes, browsers being slow, etc. So an SLO of 100% is almost never a good idea, and in most cases it is impossible to reach. In places like Google an SLO of "five nines" (99.999%) is not uncommon, and this means that the service can't fail completely for more than 5 minutes across a whole year!

Measure and report performance against SLA/SLO Once you have a defined SLA and SLO, it is very important that these are monitored accurately and reported constantly. If you wait for the end of the quarter to produce a hand-made report, the SLA is useless, as you only know you broke it when it is too late. You need automated and accurate monitoring of your service level, and this means that the SLA has to be concrete and actionable. Fuzzy requirements that can't be measured are just a waste of time. This is a very important tool for SRE, as it allows to see the progression of the service over time, detect capacity issues before they become outages, and at the same time show how much downtime can be taken without breaking the SLA. Which brings us to one core aspect of SRE:

Use error budgets and gate launches on them If SLO is the minimum rate of availability, then the result of calculating 1 - SLO is what fraction of the time a service can fail without failing out of the SLA. This is called an error budget, and you get to use it the way you want it. If the service is flaky (e.g. it fails consistently 1 of every 10000 requests), most of that budget is just wasted and you won't have any margin for launching riskier changes. On the other hand, a stable service that does not eat the budget away gives you the chance to bet part of it on releasing more often, and getting your new features quicker to the user. The moment the error budget is spent, no more launches are allowed until the average goes back out of the red. Once everyone can see how the service is performing against this agreed contract, many of the traditional sources of conflict between development and operations just disappear. If the service is working as intended, then SRE does not need to interfere on new feature launches: SRE trusts the developers' judgement. Instead of stopping a launch because it seems risky or under-tested, there are hard numbers that take the decisions for you. Traditionally, Devs get frustrated when they want to release, but Ops won't accept it. Ops thinks there will be problems, but it is difficult to back this feeling with hard data. This fuels resentment and distrust, and management is never pleased. Using error budgets based on already established SLAs means there is nobody to get upset at: SRE does not need to play bad cop, and SWE is free to innovate as much as they want, as long as things don't break. At the same time, this provides a strong incentive for developers to avoid risking their budget in poorly-prepared launches, to perform staged deployments, and to make sure the error budget is not wasted by recurrent issues.

---

That's all for today. The next article will continue delving into how traditional tensions between Devs and Ops are played in the SRE world. Comment

Whilst upgrading various projects to Django 1.9, I found myself repeatedly searching for the following code snippet so I offer it below as a more permanent note for myself and to aid others. If you used django.template.base.add_to_builtins to avoid tedious and unsightly % load module % blocks in your template files, under Django 1.9 you will get the following traceback:

Traceback (most recent call last):
  File "django/core/management/__init__.py", line 324, in execute
    django.setup()
  File "django/__init__.py", line 18, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "django/apps/registry.py", line 108, in populate
    app_config.import_models(all_models)
  File "django/apps/config.py", line 202, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "myproject/myproject/utils/models.py", line 1, in <module>
    from django.template.base import add_to_builtins
ImportError: cannot import name add_to_builtins

TEMPLATES = [ 
    'BACKEND': 'django.template.backends.django.DjangoTemplates',
    'DIRS': [
        os.path.join(BASE_DIR, 'templates'),
    ],
    'APP_DIRS': True,
    'OPTIONS':  
        'context_processors': [
            'django.template.context_processors.debug',
            'django.template.context_processors.request',
            'django.contrib.auth.context_processors.auth',
            'django.contrib.messages.context_processors.messages',
            'myproject.utils.context_processors.settings_context',
        ],
        'builtins': [
            'django.contrib.staticfiles.templatetags.staticfiles',
        ],
     ,
 ]

dhcp-authoritative
interface=eth1
read-ethers
dhcp-range=192.168.1.100,192.168.1.150,12h
dhcp-range=2001:9b0:104:42::100,2001:9b0:104:42::1500
dhcp-option=option6:dns-server,[::]
enable-ra

dnssec
trust-anchor=.,19036,8,2,49AAC11D7B6F6446702E54A1607371607A1A41855200FD2CE1CDDE32F24E8FB5
dnssec-check-unsigned

As a default, dnsmasq does not check that unsigned DNS replies are legitimate: they are assumed to be valid and passed on (without the authentic data bit set, of course). This does not protect against an attacker forging unsigned replies for signed DNS zones, but it is fast. If this flag is set, dnsmasq will check the zones of unsigned replies, to ensure that unsigned replies are allowed in those zones. The cost of this is more upstream queries and slower performance.

jow13gw dnsmasq 460 - -  forwarded www.fritidsresor.se to 213.80.101.3
jow13gw dnsmasq 460 - -  validation result is BOGUS

jow13gw dnsmasq 547 - -  reply cloudflare-dnssec.net is BOGUS DNSKEY
jow13gw dnsmasq 547 - -  validation result is BOGUS

jow13gw dnsmasq 547 - -  reply linux.conf.au is BOGUS DS
jow13gw dnsmasq 547 - -  validation result is BOGUS

apt-get install unbound
unbound-control-setup 

server:
	interface: 127.0.0.1
	interface: ::1
	interface: 192.168.1.2
	interface: 2001:9b0:104:42::2
	access-control: 127.0.0.1 allow
	access-control: ::1 allow
	access-control: 192.168.1.2/24 allow
	access-control: 2001:9b0:104:42::2/64 allow
	outgoing-interface: 155.4.17.2
	outgoing-interface: 2001:9b0:1:1a04::2
#	log-queries: yes
#	verbosity: 2

port=0
dhcp-option=option:dns-server,192.168.1.2

$ host linux.conf.au
linux.conf.au has address 192.55.98.190
linux.conf.au mail is handled by 1 linux.org.au.
$ host sigfail.verteiltesysteme.net
;; connection timed out; no servers could be reached
$ 

[unbound*]
user root
env.statefile /var/lib/munin-node/plugin-state/root/unbound.state
env.unbound_conf /etc/unbound/unbound.conf
env.unbound_control /usr/sbin/unbound-control
env.spoof_warn 1000
env.spoof_crit 100000

server:
	extended-statistics: yes
	statistics-cumulative: no
	statistics-interval: 0
remote-control:
	control-enable: yes

$ ps p $$ -o pid=pid,comm=comm
 pid comm
31612 bash

$ ps -p $$ -o pid=pid,comm=comm
pid,comm=comm
 31612

echo 1   sudo tee /proc/sys/kernel/unprivileged_userns_clone > /dev/null

sudo sysctl -w kernel.unprivileged_userns_clone=1

lxc-usernsexec [opts] -- unshare [opts] -- COMMAND

• as a project for me to learn how unprivileged namespaces work
• written in Perl which means:
  • architecture independent (same executable on any architecture)
  • easily inspectable by other curious minds
• tons of code comments to let others understand how things work
• no need to install the lxc package in a minimal environment (perl itself might not be called minimal either but is present in every Debian installation)
• not suffering from being unable to mount proc

$ mkdir -p /tmp/buildroot/proc
$ ./user-unshare --mount-proc=/tmp/buildroot/proc --ipc --pid --net \
    --uts --mount --fork -- sh -c 'ip link set lo up && ip addr && \
    hostname hoothoot-chroot && \
    tar -C /tmp/buildroot -xf /srv/chroot/unstable-amd64.tar.gz; \
    /usr/sbin/chroot /tmp/buildroot /sbin/runuser -s /bin/bash - josch && \
    umount /tmp/buildroot/proc && rm -rf /tmp/buildroot'
(unstable-amd64-sbuild)josch@hoothoot-chroot:/$ whoami
josch
(unstable-amd64-sbuild)josch@hoothoot-chroot:/$ hostname
hoothoot-chroot
(unstable-amd64-sbuild)josch@hoothoot-chroot:/$ ls -lha /proc   head
total 0
dr-xr-xr-x 218 nobody nogroup    0 Oct 25 19:06 .
drwxr-xr-x  22 root   root     440 Oct  1 08:42 ..
dr-xr-xr-x   9 root   root       0 Oct 25 19:06 1
dr-xr-xr-x   9 josch  josch      0 Oct 25 19:06 15
dr-xr-xr-x   9 josch  josch      0 Oct 25 19:06 16
dr-xr-xr-x   9 root   root       0 Oct 25 19:06 7
dr-xr-xr-x   9 josch  josch      0 Oct 25 19:06 8
dr-xr-xr-x   4 nobody nogroup    0 Oct 25 19:06 acpi
dr-xr-xr-x   6 nobody nogroup    0 Oct 25 19:06 asound

#!/bin/sh

set -exu

# I'm using /tmp because I have it mounted as a tmpfs
rootdir="/tmp/buildroot"

# bring the loopback interface up
ip link set lo up

# show that the loopback interface is really up
ip addr

# make use of the UTS namespace being unshared
hostname hoothoot-chroot

# extract the chroot tarball. This must be done inside the user namespace for
# the file permissions to be correct.
#
# tar will fail to call mknod and to change the permissions of /proc but we are
# ignoring that
tar -C "$rootdir" -xf /srv/chroot/unstable-amd64.tar.gz   true

# run chroot and inside, immediately drop permissions to the user "josch" and
# start an interactive shell
/usr/sbin/chroot "$rootdir" /sbin/runuser -s /bin/bash - josch

# unmount /proc and remove the temporary directory
umount "$rootdir/proc"
rm -rf "$rootdir"

$ mkdir -p /tmp/buildroot/proc
$ ./user-unshare --mount-proc=/tmp/buildroot/proc --ipc --pid --net --uts --mount --fork -- ./chroot.sh

$ lxc-usernsexec -m b:0:1000:1 -m b:1:558752:1 -- unshare --mount-proc=/tmp/buildroot/proc --ipc --pid --net --uts --mount --fork -- ./chroot.sh
unshare: mount /tmp/buildroot/proc failed: Invalid argument

• Perhaps you have a lot of hosts, and would like to run propellor on them all concurrently.

    master = host "master.example.com"
        & concurrently conducts alotofhosts
  

• Perhaps you want to run propellor on your dns server last, so when you add a new webserver host, it gets set up and working before the dns is updated to point to it.

    master = host "master.example.com"
        & conducts webservers
             before  conducts dnsserver
  

• Perhaps you have something more complex, with multiple subnets that propellor can run in concurrently, finishing up by updating that dnsserver.

    master = host "master.example.com"
        & concurrently conducts [sub1, sub2]
             before  conducts dnsserver
    sub1 = "master.subnet1.example.com"
        & concurrently conducts webservers
        & conducts loadbalancers
    sub2 = "master.subnet2.example.com"
        & conducts dockerservers
  

• Perhaps you need to first run some command that creates a VPS host, and then want to run propellor on that host to set it up.

    vpscreate h = cmdProperty "vpscreate" [hostName h]
         before  conducts h
  

main = defaultMain (orchestrate hosts)
hosts = 
    [ master
    , webservers 
    , ...
    ]

• Builds up a graph of what conducts what.
• Removes any cycles that might have snuck in by accident, before they cause foot shooting.
• Arranges for the ssh keys to be accepted as necessary.
  Note that you you need to add ssh key properties to all relevant hosts so it knows what keys to trust.
• Arranges for the private data of a host to be provided to the hosts that conduct it, so they can pass it along.

newtype Info = Info [Dynamic]
    deriving (Monoid)

addInfo :: (IsInfo v, Monoid v) => Info -> v -> Info
addInfo (Info l) v = Info (toDyn v : l)

getInfo :: (IsInfo v, Monoid v) => Info -> v
getInfo (Info l) = mconcat (mapMaybe fromDynamic (reverse l))

newtype Controlling = Controlled [Host]
    deriving (Typeable, Monoid)
isControlledBy :: Host -> Controlling -> Bool
h  isControlledBy  (Controlled hs) = any (== hostName h) (map hostName hs)
instance IsInfo Controlling where
    propigateInfo _ = True
mkControllingInfo :: Host -> Info
mkControllingInfo controlled = addInfo mempty (Controlled [controlled])
getControlledBy :: Host -> Controlling
getControlledBy = getInfo . hostInfo
isControllerLoop :: Host -> Host -> Bool
isControllerLoop controller controlled = go S.empty controlled
  where
    go checked h
          controller  isControlledBy  c = True
        -- avoid checking loops that have been checked before
          hostName h  S.member  checked = False
          otherwise = any (go (S.insert (hostName h) checked)) l
      where
        c@(Controlled l) = getControlledBy h

 instance Show Info where
            show (Info l) = "Info " ++ show (map dynTypeRep l)

*Main> hostInfo kite
Info [InfoVal System,PrivInfo,PrivInfo,Controlling,DnsInfo,DnsInfo,DnsInfo,AliasesInfo, ...
*Main> getInfo (hostInfo kite) :: AliasesInfo
AliasesInfo (fromList ["downloads.kitenet.net","git.joeyh.name","imap.kitenet.net","nntp.olduse.net" ...

1. Run apt-get install postfix to install Postfix, select Internet Site for the type of mail configuration and enter the domain name you selected for the mail name.
2. The main Postfix configuration file is /etc/postfix/main.cf. Change the myhostname setting to the fully qualified name of the system, something like mta.example.com.
   You can edit /etc/postfix/main.cf with vi (or any other editor) or use the postconf command to change it, eg postconf -e myhostname=mta.example.com .
3. Add home_mailbox=Maildir/ to the Postfix configuration to make it deliver to a Maildir spool in the user s home directory.
4. Restart Postfix to apply the changes.
5. Run apt-get install swaks libnet-ssleay-perl to install swaks (a SMTP test tool).
6. Test delivery by running the command swaks -f auser@example.com -t auser@example.com -s localhost . Note that swaks displays the SMTP data so you can see exactly what happens and if something goes wrong you will see everything about the error.
7. Inspect /var/log/mail.log to see the messages about the delivery. View the message which is in ~auser/Maildir/new.
8. When other students get to this stage run the same swaks command but with the -t changed to the address in their domain, check the mail.log to see that the messages were transferred and view the mail with less to see the received lines. If you do this on your own specify a recipient address that s a regular email address of yours (EG a Gmail account).

1. Run apt-get install dovecot-pop3d dovecot-imapd to install Dovecot POP and IMAP servers.
   Run netstat -tln to see the ports that have daemons listening on them, observe that ports 110 and 143 are in use.
2. Edit /etc/dovecot/conf.d/10-mail.conf and change mail_location to maildir:~/Maildir . Then restart Dovecot.
3. Run the command nc localhost 110 to connect to POP, then run the following commands to get capabilities, login, and retrieve mail:
   user auser
   pass WHATEVERYOUMADEIT
   capa
   list
   retr 1
   quit
4. Run the command nc localhost 143 to connect to IMAP, then run the following commands to list capabilities, login, and logout:
   a capability
   b login auser WHATEVERYOUMADEIT
   c logout
5. For the above commands make note of the capabilities, we will refer to that later.

1. Edit /etc/dovecot/conf.d/10-ssl.conf and change ssl = no to ssl = required . Then add the following 2 lines:
   ssl_cert = </etc/ssl/certs/ssl-cert-snakeoil.pem
   ssl_key = </etc/ssl/private/ssl-cert-snakeoil.key
2. 1. Run netstat -tln and note that ports 993 and 995 are not in use.
   2. Edit /etc/dovecot/conf.d/10-master.conf and uncomment the following lines:
      port = 993
      ssl = yes
      port = 995
      ssl = yes
   3. Restart Dovecot, run netstat -tln and note that ports 993 and 995 are in use.
3. Run nc localhost 110 and nc localhost 143 as before, note that the capabilities have changed to include STLS/STARTTLS respectively.
4. Run gnutls-cli --tofu 127.0.0.1 -p 993 to connect to the server via IMAPS and gnutls-cli --tofu 127.0.0.1 -p 995 to connect via POP3S. The --tofu option means to Trust On First Use , it stores the public key in ~/.gnutls and checks it the next time you connect. This allows you to safely use a snakeoil certificate if all apps can securely get a copy of the key.

1. Edit /etc/postfix/main.cf and add the following 4 lines:
   smtpd_tls_received_header = yes
   smtpd_tls_loglevel = 1
   smtp_tls_loglevel = 1
   smtp_tls_security_level = may
   Then restart Postfix. This makes Postfix log TLS summary messages to syslog and in the Received header. It also permits Postfix to send with TLS.
2. Run nc localhost 25 to connect to your SMTP port and then enter the following commands:
   ehlo test
   quit
   Note that the response to the EHLO command includes 250-STARTTLS, this is because Postfix was configured with the Snakeoil certificate by default.
3. Run gnutls-cli --tofu 127.0.0.1 -p 25 -s and enter the following commands:
   ehlo test
   starttls
   ^D
   After the CTRL-D gnutls-cli will establish a SSL connection.
4. Run swaks -tls -f auser@example.com -t auser@example.com -s localhost to send a message with SSL encryption. Note that swaks doesn t verify the key.
5. Try using swaks to send messages to other servers with SSL encryption. Gmail is one example of a mail server that supports SSL which can be used, run swaks -tls -f auser@example.com -t YOURADDRESS@gmail.com to send TLS (encapsulated SSL) mail to Gmail via swaks. Also run swaks -tls -f auser@example.com -t YOURADDRESS@gmail.com -s localhost to send via your new mail server (which should log that it was a TLS connection from swaks and a TLS connection to Gmail).

1. Run mkdir -p /var/spool/postfix/var/spool and ln -s ../.. /var/spool/postfix/var/spool/postfix , this allows parts of Postfix to work with the same configuration regardless of whether they are running in a chroot.
2. Add the following to /etc/postfix/main.cf and restart Postfix:
   smtpd_sasl_auth_enable = yes
   smtpd_sasl_type = dovecot
   smtpd_sasl_path = /var/spool/postfix/private/auth
   broken_sasl_auth_clients = yes
   smtpd_sasl_authenticated_header = yes
3. Edit /etc/dovecot/conf.d/10-master.conf, uncomment the following lines, and then restart Dovecot:
   unix_listener /var/spool/postfix/private/auth
   mode = 0666
   
4. Edit /etc/postfix/master.cf, uncomment the line for the submission service, and restart Postfix. This makes Postfix listen on port 587 which is allowed through most firewalls.
5. From another system (IE not the virtual machine you are working on) run swaks -tls -f auser@example.com -t YOURADDRESS@gmail.com -s YOURSERVER and note that the message is rejected with Relay access denied .
6. Now run swaks -tls --auth-user auser --auth-password WHATEVER -f auser@example.com -t YOURREALADDRESS -s YOURSERVER and observe that the mail is delivered (subject to anti-spam measures at the recipient).
Configuring a MUA If every part of the previous 3 sections is complete then you should be able to setup your favourite MUA. Use auser as the user-name for SMTP and IMAP, mail.example.com for the SMTP/IMAP server and it should just work! Of course you need to use the same DNS server for your MUA to have this just work. But another possibility for testing is to have the MUA talk to the server by IP address not by name.
Related posts:
1. Mail Server Security I predict that over the course of the next 10...
2. I need an LMTP server I am working on a system where a front-end mail...
3. Moving a Mail Server Nowadays it seems that most serious mail servers (IE mail...

• Rent a virtual machine.
• Configure an OpenVPN server on it.
• Connect from $current-country to it.

ip -4 route add .... dev tun0

• Remote OpenVPN server.
• MITM proxy on remote VPN-host
  • Which is basically a transparent HTTP/HTTPS proxy.
• Route traffic to it via DNS.
  • e.g. For any DNS request, if it ends in .co.uk return 10.10.10.10.

apt install reboot-notifier

deb http://httpredir.debian.org/debian jessie-backports main

• no case
• no integrated storage
• not having an x86 processor
• being a slow processor
• limited peripheral support
• no real time clock
• not supplying keyboards, mice
• not providing displays.

• Try to catch up with Debian Contributors development/news, adopt some data source (www.debian.org?), try to create some new data source (about translators, for example).
• Clean spam in the archive lists (now that I am DD I may help in not only Mark as spam , but actually remove the spam from the archive)
• Help in the review/updates of the Debian website (ongoing: /users; future: children-distros/derivatives, /partners).
• Try to get more involved in the Debian i18n team, at least in the part of internationalizing the Debian infrastructure (make translatable via PO files/templates many texts that used repetitively in the Debian website, for example) and in welcoming new members in translation teams or helping the weak ones (although I don t know many languages, the workflow and tools are common, so I think I could help new translators at least to get hands on the matter).
• Maybe some short screencasts/videos about the Debian infrastructure, for newcomers? (mailing lists, IRC, wiki, Alioth )

I think Preining mis-characterises what happens on the Linux kernel mailing list, and in Debian, in free software development in general, and what Sarah Sharp has said and done. I continue to think he is wrong about everything on this topic.

Once again, he s complaining about how the fun from Debian has been lost because making sexist jokes, or treating other people like shit is not allowed any more.

• Niels Thykier uploaded debhelper/9.20151004 which exports SOURCE_DATE_EPOCH when running dh_auto_* (patch by Dhole). dh now also calls dh_strip_nondeterminism during build (patch by Andrew Ayer). The only remaining change regarding reproducibility in the custom debhelper is related to mtimes in data.tar which might be fixed directly in dpkg instead.
• Niels Thykier made another upload of debhelper/9.20151005 the next day to sort Build IDs added by dh_strip in control files.

• classified-ads/0.08-1 uploaded by Antti J rvinen, fix merged upstream, original patch by Reiner Herrmann.
• criu/1.7-3 uploaded by Salvatore Bonaccorso, original patch by Chris Lamb.
• doomsday/1.15.4-1 by Michael Gilbert.
• ejabberd/15.07-1~exp1 by Philipp Huebner.
• libxbean-java/4.4-1 by Emmanuel Bourg.
• markdown/1.0.1-8 uploaded by Matt Kraai, original patches by Chris Lamb (#776925) and akira (#793701).
• nedit/1:5.6a-3 by Paul Gevers.
• nvidia-settings/340.93-1 uploaded by Andreas Beckmann, original patch by Lunar.
• ocaml/4.02.3-2 by St phane Glondu.
• polygen/1.0.6.ds2-14 uploaded by Mehdi Dogguy, original patch by Chris Lamb.
• stsci.distutils/0.3.7-4 by Aurelien Jarno.
• vdr/2.2.0-4 by Tobias Grimm.
• vdr-plugin-xineliboutput/1.1.0+cvs20150907-3 by Tobias Grimm.
• w3c-markup-validator/1.3+dfsg-3 by Santiago Vila.
• xcolorsel/1.1a-19 by Santiago Vila.
• xmoto/0.5.11+dfsg-3 by Stephen Kitt.

• ognl/2.7.3-6 by Emmanuel Bourg.
• enblend-enfuse/4.1.4+dfsg-2 by Andreas Metzler.

• nvidia-xconfig/340.93-1 by Andreas Beckmann.
• nvidia-settings-legacy-304xx/304.128-1 by Andreas Beckmann.

• #801212 on unicode-data by Esa Peuha: set TZ=UTC when calling unzip.