Midsummer for me is also known as Noite do Lume Novo (literally New Fire Night ), one of the big calendar events of the year, marking the end of the school year and the beginning of summer. On this day, there are celebrations not very unlike the bonfires in the Guy Fawkes Night in England or Britain ^[1]. It is a bit different in that it is not a single event for the masses, more of a friends and neighbours thing, and that it lasts for a big chunk of the night (sometimes until morning). Perhaps for some people, or outside bigger towns or cities, Guy Fawkes Night is also celebrated in that way and that's why during the first days of November there are fireworks rocketing and cracking in the neighbourhoods all around. Like many other celebrations around the world involving bonfires, many of them also happening around the summer solstice, it is supposed to be a time of renewal of cycles, purification and keeping the evil spirits away; with rituals to that effect like jumping over the fire when the flames are not high and it is safe enough. So it was fitting that, in the middle of June (almost Midsummer in the northern hemisphere), I learnt that I was about to leave my now-previous job, which is a pretty big signal and precursor for renewal (and it might have something to do with purifying and keeping the evil away as well ;-) ). Whatever... But what does all of this have to do with aptitude or Debian, anyway? For one, it was a question of timing. While looking for a new job (and I am still at it), I had more spare time than usual. DebConf 15 @ Heidelberg was within sight, and for the first time circumstances allowed me to attend this event. It also coincided with the time when I re-gained access to commit to aptitude on the 19th of June. Which means Renewal. End of June was also the time of the announcement of the colossal GCC-5/C++11 ABI transition in Debian, that was scheduled to start on the 1st of August, just before the DebConf. Between 2 and 3 thousand source packages in Debian were affected by this transition, which a few months later is not yet finished (although the most important parts were completed by mid-end September). aptitude itself is written in C++, and depends on several libraries written in C++, like Boost, Xapian and SigC++. All of them had to be compiled with the new C++11 ABI of GCC-5, in unison and in a particular order, for aptitude to continue to work (and for minimal breakage). aptitude and some dependencies did not even compile straight away, so this transition meant that aptitude needed attention just to keep working. Having recently being awarded again with the Aptitude Hat, attending DebConf for the first time and sailing towards the Transition Maelstrom, it was a clear sign that Something Had to Be Done (to avoid the sideways looks and consequent shame at DebConf, if nothing else). Happily (or a bit unhappily for me, but let's pretend...), with the unexpected free time in my hands, I changed the plans that I had before re-gaining the Aptitude Hat (some of them involving Debian, but in other ways maybe I will post about that soon). In July I worked to fix the problems before the transition started, so aptitude would be (mostly) ready, or in the worst case broken only for a few days, while the chain of dependencies was rebuilt. But apart from the changes needed for the new GCC-5, it was decided at the last minute that Boost 1.55 would not be rebuilt with the new ABI, and that the only version with the new ABI would be 1.58 (which caused further breakage in aptitude, was added to experimental only a few days before, and was moved to unstable after the transition had started). Later, in the first days of the transition, aptitude was affected for a few days by breakage in the dependencies, due to not being compiled in sequence according to the transition levels (so with a mix of old and new ABI). With the critical intervention of Axel Beckert (abe / XTaran), things were not so bad as they could have been. He was busy testing and uploading in the critical days when I was enjoying a small holiday on my way to DebConf, with minimal internet access and communicating almost exclusively with him; and he promptly tended the complaints arriving in the Bug Tracking System and asked for rebuilds of the dependencies with the new ABI. He also brought the packaging up to shape, which had decayed a bit in the last few years. Gruesome Challenges But not all was solved yet, more storms were brewing and started to appear in the horizon, in the form of clouds of fire coming from nearby realms. The APT Deities, which had long ago spilled out their secret, inner challenge (just the initial paragraphs), were relentless. Moreover, they were present at Heidelberg in full force, in or close to their home grounds, and they were Marching Decidedly towards Victory: In the talk @ DebConf This APT has Super Cow Powers (video available), by David Kalnischkies, they told us about the niceties of apt 1.1 (still in experimental but hopefully coming to unstable soon), and they boasted about getting the lead in our arms race (should I say bugs race?) by a few open bug reports. This act of provocation further escalated the tensions. The fierce competition which had been going on for some time gained new heights. So much so that APT Deities and our team had to sit together in the outdoor areas of the venue and have many a weissbier together, while discussing and fixing bugs. But beneath the calm on the surface, and while pretending to keep good diplomatic relations, I knew that Something Had to Be Done, again. So I could only do one thing jump over the bonfire and Keep the Evil away, be that Keep Evil bugs Away or Keep Evil APT Deities Away from winning the challenge, or both. After returning from DebConf I continued to dedicate time to the project, more than a full time job in some weeks, and this is what happened in the last few months, summarised in another graph, showing the evolution of the BTS for aptitude: The numbers for apt right now (15th November 2015) are:

• 629 open (731 if counting all merged bugs independently)
• 0 Release Critical
• 275 (318 unmerged) with severity Important or Normal
• 354 (413 unmerged) with severity Minor or Wishlist
• 0 marked as Forwarded or Pending

The numbers for aptitude right now are:

• 488 (573 if counting all merged bugs independently)
• 1 Release Critical (but it is an artificial bug to keep it from migrating to testing)
• 197 (239 unmerged) with severity Important or Normal
• 271 (313 unmerged) with severity Minor or Wishlist
• 19 (20 unmerged) marked as Forwarded or Pending

The Aftermath As we can see, for the time being I could keep the Evil at bay, both in terms of bugs themselves and re-gaining the lead in the bugs race the Evil APT Deities were thwarted again in their efforts. ... More seriously, as most of you suspected, the graph above is not the whole truth, so I don't want to boast too much. A big part of the reduction in the number of bugs is because of merging duplicates, closing obsolete bugs, applying translations coming from multiple contributors, or simple fixes like typos and useful suggestions needing minor changes. Many of remaining problems are comparatively more difficult or time consuming that the ones addressed so far (except perhaps avoiding the immediate breakage of the transition, that took weeks to solve), and there are many important problems still there, chief among those is aptitude offering very poor solutions to resolve conflicts. Still, even the simplest of the changes takes effort, and triaging hundreds of bugs is not fun at all and mostly a thankless effort althought there is the occasionally kind soul that thanks you for handling a decade-old bug. If being subjected to the rigours of the BTS and reading and solving hundreds of bug reports is not Purification, I don't know what it is. Apart from the triaging, there were 118 bugs closed (or pending) due to changes made in the upstream part or the packaging in the last few months, and there are many changes that are not reflected in bugs closed (like most of the changes needed due to the C++11 ABI transition, bugs and problems fixed that had no report, and general rejuvenation or improvement of some parts of the code). How long this will last, I cannot know. I hope to find a job at some point, which obviously will reduce the time available to work on this. But in the meantime, for all aptitude users: Enjoy the fixes and new features! Notes [1] ^{^} Some visitors of the recent mini-DebConf @ Cambridge perhaps thought that the fireworks and throngs gathered were in honour of our mighty Universal Operating System, but sadly they were not. They might be, some day. In any case, the reports say that the visitors enjoyed the fireworks.

What happened in the reproducible builds effort this week: Toolchain fixes Emmanuel Bourg uploaded eigenbase-resgen/1.3.0.13768-2 which uses of the scm-safe comment style by default to make them deterministic. Mattia Rizzolo started a new thread on debian-devel to ask a wider audience for issues about the -Wdate-time compile time flag. When enabled, GCC and clang print warnings when __DATE__, __TIME__, or __TIMESTAMP__ are used. Having the flag set by default would prompt maintainers to remove these source of unreproducibility from the sources. Packages fixed The following packages have become reproducible due to changes in their build dependencies: bmake, cyrus-imapd-2.4, drobo-utils, eigenbase-farrago, fhist, fstrcmp, git-dpm, intercal, libexplain, libtemplates-parser, mcl, openimageio, pcal, powstatd, ruby-aggregate, ruby-archive-tar-minitar, ruby-bert, ruby-dbd-odbc, ruby-dbd-pg, ruby-extendmatrix, ruby-rack-mobile-detect, ruby-remcached, ruby-stomp, ruby-test-declarative, ruby-wirble, vtprint. The following packages became reproducible after getting fixed:

• bino/1.6.1-1 by Daniel Schaal.
• debconf/1.5.58 by Colin Watson.
• fte/0.50.2b6-8 uploaded by Axel Beckert, original patch by Reiner Herrmann.
• gnome-desktop3/3.18.2-1 uploaded by Michael Biebl, fix by Laurent Bigonville based on Andreas Henriksson's suggestion.
• libxmu/2:1.1.2-2 uploaded by Julien Cristau, fix by Andreas Boll.
• megaglest/3.11.1-3 by Markus Koschany.
• p4vasp/0.3.29+dfsg-2 uploaded by Graham Inggs, original patch by Reiner Herrmann.
• pd-iemmatrix/0.3-1 by IOhannes m zm lnig.
• performous/1.0+git150721-2 by Markus Koschany.
• python-sqlalchemy-utils/0.30.12-2 uploaded by Thomas Goirand, original patch by Chris Lamb.
• pywavelets/0.3.0-1 uploaded by Daniele Tricoli, original patch by Juan Picca.
• torus-trooper/0.22.dfsg1-10 uploaded by Markus Koschany, original patch by Reiner Herrmann.
• tycho2/2-3 by Ole Streicher.
• xenomai/2.6.4+dfsg-0.1 by Leopold Palomo-Avellaneda.
• xfonts-mona/2.90-8 by Nobuhiro Iwamatsu.

Some uploads fixed some reproducibility issues, but not all of them:

• apache-mime4j/0.7.2-4 by Emmanuel Bourg.
• ispell-lt/1.2.1-5 by Chris Lamb (#778263.
• xscavenger/1.4.5-1 uploaded by Hwei Sheng Teoh, original patch by Chris Lamb.

Patches submitted which have not made their way to the archive yet:

• #804729 on pbuilder by Reiner Herrmann: tell dblatex to build in a deterministic path.

reproducible.debian.net The fifth and sixth armhf build nodes have been set up, resulting in five more builder jobs for armhf. More than 10,000 packages have now been identified as reproducible with the reproducible toolchain on armhf. (Vagrant Cascadian, h01ger) Helmut Grohne and Mattia Rizzolo now have root access on all 12 build nodes used by reproducible.debian.net and jenkins.debian.net. (h01ger) reproducible-builds.org is now linked from all package pages and the reproducible.debian.net dashboard. (h01ger) profitbricks-build5-amd64 and profitbricks-build6-amd64, responsible for running amd64 tests now run 398.26 days in the future. This means that one of the two builds that are being compared will be run on a different minute, hour, day, month, and year. This is not yet the case for armhf. FreeBSD tests are also done with 398.26 days difference. (h01ger) The design of the Arch Linux test page has been greatly improved. (Levente Polyak) diffoscope development Three releases of diffoscope happened this week numbered 39 to 41. It includes support for EPUB files (Reiner Herrmann) and Free Pascal unit files, usually having .ppu as extension (Paul Gevers). The rest of the changes were mostly targetting at making it easier to run diffoscope on other systems. The tlsh, rpm, and debian modules are now all optional. The test suite will properly skip tests that need optional tools or modules when they are not available. As a result, diffosope is now available on PyPI and thanks to the work of Levente Polyak in Arch Linux. Getting these versions in Debian was a bit cumbersome. Version 39 was uploaded with an expired key (according to the keyring on ftp.debian.org which will hopefully be updated soon) which is currently handled by keeping the files in the queue without REJECTing them. This prevented any other Debian Developpers to upload the same version. Version 40 was uploaded as a source-only upload but failed to build from source which had the undesirable side effect of removing the previous version from unstable. The package faild to build from source because it was built passing -I to debbuild. This excluded the ELF object files and static archives used by the test suite from the archive, preventing the test suite to work correctly. Hopefully, in a nearby future it will be possible to implement a sanity check to prevent such mistakes in the future. It has also been identified that ppudump outputs time in the system timezone without considering the TZ environment variable. Zachary Vance and Paul Gevers raised the issue on the appropriate channels. strip-nondeterminism development Chris Lamb released strip-nondeterminism version 0.014-1 which disables stripping Mono binaries as it is too aggressive and the source of the problem is being worked on by Mono upstream. Package reviews 133 reviews have been removed, 115 added and 103 updated this week. Chris West and Chris Lamb reported 57 new FTBFS bugs. Misc. The video of h01ger and Chris Lamb's talk at MiniDebConf Cambridge is now available. h01ger gave a talk at CCC Hamburg on November 13th, which was well received and sparked some interest among Gentoo folks. Slides and video should be available shortly. Frederick Kautz has started to revive Dhiru Kholia's work on testing Fedora packages. Your editor wish to once again thank #debian-reproducible regulars for reviewing these reports weeks after weeks.

Discworld Noir was a superb adventure game, but is also notoriously unreliable, even in Windows on real hardware; using Wine is just not going to work. After many attempts at bringing it back into working order, I've settled on an approach that seems to work: now that qemu and libvirt have made virtualization and emulation easy, I can run it in a version of Windows that was current at the time of its release. Unfortunately, Windows 98 doesn't virtualize particularly well either, so this still became a relatively extensive yak-shaving exercise. These instructions assume that /srv/virt is a suitable place to put disk images, but you can use anywhere you want. The emulated PC After some trial and error, it seems to work if I configure qemu to emulate the following:

• Fully emulated hardware instead of virtualization (qemu-system-i386 -no-kvm)
• Intel Pentium III
• Intel i440fx-based motherboard with ACPI
• Real-time clock in local time
• No HPET
• 256 MiB RAM
• IDE primary master: IDE hard disk (I used 30 GiB, which is massively overkill for this game; qemu can use sparse files so it actually ends up less than 2 GiB on the host system)
• IDE primary slave, secondary master, secondary slave: three CD-ROM drives
• PS/2 keyboard and mouse
• Realtek AC97 sound card
• Cirrus video card with 16 MiB video RAM

A modern laptop CPU is an order of magnitude faster than what Discworld Noir needs, so full emulation isn't a problem, despite being inefficient. There is deliberately no networking, because Discworld Noir doesn't need it, and a 17 year old operating system with no privilege separation is very much not safe to use on the modern Internet! Software needed

• Windows 98 installation CD-ROM as a .iso file (cp /dev/cdrom windows98.iso) - in theory you could also use a real optical drive, but my laptop doesn't usually have one of those. I used the OEM disc, version 4.10.1998 (that's the original Windows 98, not the Second Edition), which came with a long-dead PC, and didn't bother to apply any patches.
• A Windows 98 license key. Again, I used an OEM key from a past PC.
• A complete set of Discworld Noir (English) CD-ROMs as .iso files. I used the UK "Sold Out Software" budget release, on 3 CDs.
• A multi-platform Realtek AC97 audio driver.

Windows 98 installation It seems to be easiest to do this bit by running qemu-system-i386 manually:

qemu-img create -f qcow2 /srv/virt/discworldnoir.qcow2 30G
qemu-system-i386 -hda /srv/virt/discworldnoir.qcow2 \
    -drive media=cdrom,format=raw,file=/srv/virt/windows98.iso \
    -no-kvm -vga cirrus -m 256 -cpu pentium3 -localtime

Don't start the installation immediately. Instead, boot the installation CD to a DOS prompt with CD-ROM support. From here, run

fdisk

and create a single partition filling the emulated hard disk. When finished, hard-reboot the virtual machine (press Ctrl+C on the qemu-system-i386 process and run it again). The DOS FORMAT.COM utility is on the Windows CD-ROM but not in the root directory or the default %PATH%, so you'll have to run:

d:\win98\format c:

to create the FAT filesystem. You might have to reboot again at this point. The reason for doing this the hard way is that the Windows 98 installer doesn't detect qemu as supporting ACPI. You want ACPI support, so that Windows will issue IDLE instructions from its idle loop, instead of occupying a CPU core with a busy-loop. To get that, boot to a DOS prompt again, and use:

setup /p j /iv

/p j forces ACPI support (Thanks to "Richard S" on the Virtualbox forums for this tip.) /iv is unimportant, but it disables the annoying "billboards" during installation, which advertised once-exciting features like support for dial-up modems and JPEG wallpaper. I used a "Typical" installation; there didn't seem to be much point in tweaking the installed package set when everything is so small by modern standards. Windows 98 has built-in support for the Cirrus VGA card that we're emulating, so after a few reboots, it should be able to run in a semi-modern resolution and colour depth. Discworld Noir apparently prefers a 640 480 16-bit video mode, so right-click on the desktop background, choose Properties and set that up. Audio drivers This is the part that took me the longest to get working. Of the sound cards that qemu can emulate, Windows 98 only supports the SoundBlaster 16 out of the box. Unfortunately, the Soundblaster 16 emulation in qemu is incomplete, and in particular version 2.1 (as shipped in Debian 8) has a tendency to make Windows lock up during boot. I've seen advice in various places to emulate an Eqsonic ES1370 (SoundBlaster AWE 64), but that didn't work for me: one of the drivers I tried caused Windows to lock up at a black screen during boot, and the other didn't detect the emulated hardware. The next-oldest sound card that qemu can emulate is a Realtek AC97, which was often found integrated into motherboards in the late 1990s. This one seems to work, with the "A400" driver bundle linked above. For Windows 98 first edition, you need a driver bundle that includes the old "VXD" drivers, not just the "WDM" drivers supported by Second Edition and newer. The easiest way to get that into qemu seems to be to turn it into a CD image:

genisoimage -o /srv/virt/discworldnoir-drivers.iso WDM_A400.exe
qemu-system-i386 -hda /srv/virt/discworldnoir.qcow2 \
    -drive media=cdrom,format=raw,file=/srv/virt/windows98.iso \
    -drive media=cdrom,format=raw,file=/srv/virt/discworldnoir-drivers.iso \
    -no-kvm -vga cirrus -m 256 -cpu pentium3 -localtime -soundhw ac97

Run the installer from E:, then reboot with the Windows 98 CD inserted, and Windows should install the driver. Installing Discworld Noir Boot up the virtual machine with CD 1 in the emulated drive:

qemu-system-i386 -hda /srv/virt/discworldnoir.qcow2 \
    -drive media=cdrom,format=raw,file=/srv/virt/DWN_ENG_1.iso \
    -no-kvm -vga cirrus -m 256 -cpu pentium3 -localtime -soundhw ac97

You might be thinking "... why not insert all three CDs into D:, E: and F:?" but the installer expects subsequent disks to appear in the same drive where CD 1 was initially, so that won't work. Instead, when prompted for a new CD, switch to the qemu monitor with Ctrl+Alt+2 (note that this is 2, not F2). At the (qemu) prompt, use info block to see a list of emulated drives, then issue a command like

change ide0-cd1 /srv/virt/DWN_ENG_2.iso

to swap the CD. Then switch back to Windows' console with Ctrl+Alt+1 and continue installation. I used a Full installation of Discworld Noir. Transferring the virtual machine to GNOME Boxes Having finished the "control freak" phase of installation, I wanted a slightly more user-friendly way to run this game, so I transferred the virtual machine to be used by libvirtd, which is the backend for both GNOME Boxes and virt-manager:

virsh create discworldnoir.xml

Here is the configuration I used. It's a mixture of automatic configuration from virt-manager, and hand-edited configuration to make it match the qemu-system-i386 command-line. Running the game If all goes well, you should now see a discworldnoir virtual machine in GNOME Boxes, in which you can boot Windows 98 and play the game. Have fun!

On the final semester of my MSc program at Columbia SEAS, I was lucky enough to be able to attend a seminar course taught by Alfio Gliozzo entitled Q&A with IBM Watson. A significant part of the course is dedicated to learning how to leverage the services and resources available on the Watson Developer Cloud. This post describes the course project my team developed, the PaperTrail application.

Project Proposal Create an application to assist in the development of future academic papers. Based on a paper s initial proposal, Paper Trail predicts publications to be used as references or acknowledgement of prior art and provides a trend analysis of major topics and methods. The objective is to speed the discovery of relevant papers early in the research process, and allow for early assessment of the depth of prior research concerning the initial proposal.

Meet the Team Wesley Bruning, Software Engineer, MSc. in Computer Science Xavier Gonzalez, Industrial Engineer, MSc. in Data Science Juliana Louback, Software Engineer, MSc. in Computer Science Aaron Zakem, Patent Attorney, MSc. in Computer Science

Prior Art A significant amount of attention has been given to this topic over the past few decades. The table below shows the work the team deemed most relevant due to recency, accuracy and similarity of functionality. The variation in accuracy displayed is a result of experimentation with different dataset sizes and algorithm variations. More information and details can be found in the prior art report. The main differential of PaperTrail is providing a form of access to the citation prediciton and trend analysis algorithm. With the exception of the project by McNee et al., these algorithmns aren t currently available for general use. The application on researchindex.net is open to use but its objective is to rank publications and authors for given topics.

Algorithm Citation Prediction: PaperTrail builds on the work done by Wolski s team in Fall 2014. This algorithmn builds a reference graph used to define research communities, with an associated vector of topic scores generated by an LDA model. The papers in each research community are then ranked by importance within the community with a custom ranking algorithm. When a target document is given to algorithm as input, the LDA model is used to generate a vector of topics that are present in the document. The communities with the most similar topic vectors are selected and the publications within these communities with highest rank and greatest similarity to the input document are recommended as references. A more detailed description can be found here. Trend Analysis: Initially, the idea was to use the AlchemyData News API to obtain statistics pertaining to the amount of publications on a given topic over time. However, with the exception of buzz-words (i.e. big data ), many more specialized topics appeared very infrequently in news articles, if at all. This isn t entirely surprising given the target audience of PaperTrail. As a work around, we use the Alchemy Language API to extract keywords from the abstracts in the dataset, in addition to relevance scores. The PaperTrail database could then be queried for entry counts for a given year and keyword to provide an indication of publication trends in academia. Note that the Alchemy Language API extracts multiple-word keywords as well as single words.

Data To maintain consistency with Wolski s project, we are using the DBLP data as made available on aminer.org. The DBLP-Citation-network V5 dataset contains 1,572,277 entries; we are limited to the use of entries that contain both abstracts and citations, bringing the dataset size down to 265,865 entries.

Architecture A high-level visualization of the project architecture is displayed below. Before launching PaperTrail, it s necessary to train Wolski s algorithm offline. Currently any documentation with regard to the performance of said algorithm is unavailable; the PaperTrail project will include an evaluation phase and report the findings made. The PaperTrail app and database will be hosted on the Bluemix Platform.

Status Report Phases completed:

• Project design
• Prior art research
• Data cleansing
• Development and deployment of an alpha version of the PaperTrail app

Phases under development:

• Algorithm training and evaluation
• Keyword extraction
• MapReduce of publication frequency by year and topic
• Data visualization component

Firefox 42, which was released last week, introduced a new feature in its Private Browsing mode: tracking protection. If you are interested in how this list is put together and then used in Firefox, this post is for you.

Safe Browsing lists There are many possible ways to download URL lists to the browser and check against that list before loading anything. One of those is already implemented as part of our malware and phishing protection. It uses the Safe Browsing v2.2 protocol. In a nutshell, the way that this works is that each URL on the block list is hashed (using SHA-256) and then that list of hashes is downloaded by Firefox and stored into a data structure on disk:

• ~/.cache/mozilla/firefox/XXXX/safebrowsing/mozstd-track* on Linux
• ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/mozstd-track* on Mac
• C:\Users\XXXX\AppData\Local\mozilla\firefox\profiles\XXXX\safebrowsing\mozstd-track* on Windows

This sbdbdump script can be used to extract the hashes contained in these files and will output something like this:

$ ~/sbdbdump/dump.py -v .
- Reading sbstore: mozstd-track-digest256
[mozstd-track-digest256] magic 1231AF3B Version 3 NumAddChunk: 1 NumSubChunk: 0 NumAddPrefix: 0 NumSubPrefix: 0 NumAddComplete: 1696 NumSubComplete: 0
[mozstd-track-digest256] AddChunks: 1445465225
[mozstd-track-digest256] SubChunks:
...
[mozstd-track-digest256] addComplete[chunk:1445465225] e48768b0ce59561e5bc141a52061dd45524e75b66cad7d59dd92e4307625bdc5
...
[mozstd-track-digest256] MD5: 81a8becb0903de19351427b24921a772

The name of the blocklist being dumped here (mozstd-track-digest256) is set in the urlclassifier.trackingTable preference which you can find in about:config. The most important part of the output shown above is the addComplete line which contains a hash that we will see again in a later section.

List lookups Once it's time to load a resource, Firefox hashes the URL, as well as a few variations of it, and then looks for it in the local lists. If there's no match, then the load proceeds. If there's a match, then we do an additional check against a pairwise allowlist. The pairwise allowlist (hardcoded in the urlclassifier.trackingWhitelistTable pref) is designed to encode what we call "entity relationships". The list groups related domains together for the purpose of checking whether a load is first or third party (e.g. twitter.com and twimg.com both belong to the same entity). Entries on this list (named mozstd-trackwhite-digest256) look like this:

twitter.com/?resource=twimg.com

which translates to "if you're on the twitter.com site, then don't block resources from twimg.com. If there's a match on the second list, we don't block the load. It's only when we get a match on the first list and not the second one that we go ahead and cancel the network load. If you visit our test page, you will see tracking protection in action with a shield icon in the URL bar. Opening the developer tool console will expose the URL of the resource that was blocked:

The resource at "https://trackertest.org/tracker.js" was blocked because tracking protection is enabled.

Creating the lists The blocklist is created by Disconnect according to their definition of tracking. The Disconnect list is on their Github page, but the copy we use in Firefox is the copy we have in our own repository. Similarly the Disconnect entity list is from here but our copy is in our repository. Should you wish to be notified of any changes to the lists, you can simply subscribe to this Atom feed. To convert this JSON-formatted list into the binary format needed by the Safe Browsing code, we run a custom list generation script whenever the list changes on GitHub. If you run that script locally using the same configuration as our server stack, you can see the conversion from the original list to the binary hashes. Here's a sample entry from the mozstd-track-digest256.log file:

[m] twimg.com >> twimg.com/
[canonicalized] twimg.com/
[hash] e48768b0ce59561e5bc141a52061dd45524e75b66cad7d59dd92e4307625bdc5

and one from mozstd-trackwhite-digest256.log:

[entity] Twitter >> (canonicalized) twitter.com/?resource=twimg.com, hash a8e9e3456f46dbe49551c7da3860f64393d8f9d96f42b5ae86927722467577df

This in combination with the sbdbdump script mentioned earlier, will allow you to audit the contents of the local lists.

Serving the lists The way that the binary lists are served to Firefox is through a custom server component written by Mozilla: shavar. Every hour, Firefox requests updates from shavar.services.mozilla.com. If new data is available, then the whole list is downloaded again. Otherwise, all it receives in return is an empty 204 response. Should you want to play with it and run your own server, follow the installation instructions and then go into about:config to change these preferences to point to your own instance:

browser.trackingprotection.gethashURL
browser.trackingprotection.updateURL

Note that on Firefox 43 and later, these prefs have been renamed to:

browser.safebrowsing.provider.mozilla.gethashURL
browser.safebrowsing.provider.mozilla.updateURL

Learn more If you want to learn more about how tracking protection works in Firefox, you can find all of the technical details on the Mozilla wiki or you can ask questions on our mailing list. Thanks to Tanvi Vyas for reviewing a draft of this post.

Like each month, here comes a report about the work of paid contributors to Debian LTS. Individual reports In September, 85.50 work hours have been dispatched among 8 paid contributors. Their reports are available:

• Ben Hutchings did 14 hours (13.5h allocated, thus only catching up 0.5 hours out of the 5.5 extra hours he had left from former month).
• Chris Lamb did 11 hours (12h allocated, he will catch up later).
• Guido G nther did 4 hours (out of 8 hours allocated, thus keeping 4 extra hours for November).
• Mike Gabriel did nothing (out of 8 hours allocated, he will catch up in November).
• Rapha l Hertzog did 13.25 hours.
• Santiago Ruano Rinc n did 13.5 hours.
• Scott Kitterman did 8 hours (4 hours allocated and 4 hours remaining from September)
• Thorsten Alteholz did 13.25 hours.

Evolution of the situation November crossed a new record with 114.5 hours funded. This is mainly thanks to our first Platinum sponsor: TOSHIBA (through Toshiba Software Development Vietnam). They don t know yet if they can sponsor us in the long term (they hope so), but it s still a nice news as we jumped from 50% to 65% of the objective of the equivalent of a full-time position with a single new sponsor. Currently no change is expected for next month as we don t have any other new sponsor in the process of joining us. We still need more support to be able to support all the packages we could not afford to support during the squeeze cycle. We are currently discussing which package we can or cannot support on the LTS list, see the thread Unsupported packages for Wheezy LTS for the current situation. In terms of security updates waiting to be handled, the situation is close to last month: the dla-needed.txt file lists 21 packages awaiting an update (6 more than last month), the list of open vulnerabilities in Squeeze shows about 23 affected packages in total (exactly like last month). Thanks to our sponsors The new sponsors are in bold.

• Platinum sponsors:
• TOSHIBA
• Gold sponsors:
• The Positive Internet (for 17 months already)
• Blablacar (for 16 months already)
• Linode LLC (for 6 months already)
• Silver sponsors:
• David Ayers IntarS Austria (for 17 months already)
• Domeneshop AS (for 16 months already)
• Universit Lille 3 (for 16 months already)
• Trollweb Solutions (for 14 months already)
• Gandi SAS (for 11 months already)
• University of Luxembourg (for 8 months already)
• Rentabiliweb Group (for 6 months already)
• Univention GmbH
• Universit Jean Monnet de St Etienne
• Bronze sponsors:
• Offensive Security (for 17 months already)
• Seznam.cz, a.s. (for 17 months already)
• Evolix (for 16 months already)
• Freeside Internet Service (for 16 months already)
• MyTux (for 16 months already)
• Linuxhotel GmbH (for 14 months already)
• Intevation GmbH (for 13 months already)
• Daevel SARL (for 12 months already)
• Bitfolk LTD (for 11 months already)
• Megaspace Internet Services GmbH (for 11 months already)
• Gree, Inc. (for 10 months already)
• Greenbone Networks GmbH (for 10 months already)
• NUMLOG (for 10 months already)
• WinGo AG (for 9 months already)
• Ecole Centrale de Nantes LHEEA (for 5 months already)
• Sig-I/O (for 3 months already)
• Entr ouvert

No comment Liked this article? Click here. My blog is Flattr-enabled.

Now this could just be the prodigious amounts of sugar and ghee I've been eating the past few days, but I'm bursting with energy for working on Debian so here is a list of things I want to do in the upcoming year.

• Finish packaging the latest Dovecot (2.2.19).

This will happen soon. Sooner if you people will stop filing bugs all the time. The rest of the items are more long term.

• Recruit more members for the Dovecot packaging team.
• Undertake to respond to all bug reports and other Debian correspondence within three days.
• Finally get the Minix port to a state where other people can use it and work on it too.
• Relearn Debian development from scratch so that I can be familliar with and apply the latest techniques. Write down what I've learned.
• Arrange (or more realistically get someone to arrange) a Debian event in Northern or Eastern India.
• Watch all the Debconf 15 videos.

To keep all of this on track, I shall adopt the practice lately seen on Debian Planet of periodically posting a report of my Debian activities.

db2twitter 0.2 was just released! If you missed my last post about it, db2twitter automatically extracts fields from your database, use them to feed a template of tweet and send the tweet. db2twitter is developed by and run for LinuxJobs.fr, the job board of th french-speaking Free Software and Opensource community.

• Github of db2twitter (star it if you like ;)
• Official documentation on readthedocs

The main new feature allows to send tweets only during authorized days/hours with the following content in your db2twitter.ini file :

[timer]
days=mon-fri,
hours=0-11,14-17,

• The line days=mon-fri specifies db2twitter can send tweets only from monday to friday.
• The line hours=0-11,14-17 specifies db2twitter can send tweets from midnight to 11AM (included) and from 2PM to 17PM (included)

Given the fact db2twitter will mostly be launched via cron, it allows sending tweets detected since the last db2twitter run during user-defined hours. And that s cool! db2twitter is coded in Python 3.4, uses SQlAlchemy (see supported database types) and Tweepy. The official documentation is available on readthedocs.

Wishing you all a happy Diwali and Gujarati New Year (V.S. 2072 called Plavanga.)Instead of being something sane like a flat disk supported by four elephants standing on a tortoise, the Earth is an oblate spheroid in heliocentric orbit. The Moon too offends by maintaining an unreasonably non-epicyclic orbit. Combined this means that the Americas (plus Iceland if there any Hindus there) observe Diwali today whereas in India and the rest of the world it is tomorrow. Technically in these longitudes the Ashvayuja amavasya tithi pervades both Tuesday and Wednesday but as Lakshmi Puja takes place at the pradosha kala it means it should be celebrated today though no doubt most of the non-astronomically minded people will just follow India any way. The New Year is on Thursday all over the world.Regardless of when you celebrate, may your year be full of happiness, good fortune and prosperity.

It s now official: I m a pilot. This has been one of the most challenging, and also most rewarding, journeys I ve been on. It had its moments of struggle, moments of joy, moments of poetry. I wrote about the poetry of flying at night recently. Here is the story of my first landing on a grass runway, a few months ago. Grass

Where the air is so pure, the zephyrs so free,
The breezes so balmy and light,
That I would not exchange my home on the range
For all of the cities so bright. John A. Lomax

We are used to seeing planes in these massive palaces of infrastructure we call airports. We have huge parking garages, giant terminals, security lines hundreds of people deep, baggage carts, jetways, video screens, restaurants, and miles and miles of concrete. But most of the world s airports are not like that. A pilot of a small plane gets to see the big airports, sure, but we also get to use the smaller airports hidden in plain sight to most. Have you ever taken off from a strip of grass? As I told my flight instructor when I tried it for the first time, I know this will work, but somehow I will still be amazed if it actually does. I took off from a strip of grass not long ago. The airport there had one paved runway, and the rest were grass. Short runways, narrow runways, grass runways. No lights. No paint. No signs. No trucks, no jetways, nothing massive. In fact, no people. Just a mowed path and a couple of yellow or white markers. I taxied down the grass runway, being careful to never let the plane s wheels stop moving lest the nose gear get stuck in a pothole. I felt all the bumps in the ground as we moved. End of runway. Turn the plane around. A little bit of flap for more lift, full throttle, mind the centerline imaginary centerline, this time. It starts picking up speed, slower than usual, bump bump bump. Those buildings at the end of the runway are staring me down. More speed, and suddenly the runway feels smooth; it has enough lift to keep from falling into every bump. Then we lift off just a touch; I carefully keep the plane down until we pick up enough speed to ascend farther, then up we go. I keep a watchful eye on those buildings straight ahead and that water tower just slightly off to the one side. We climb over a lake as I watch that water tower pass safely below and to the side of the plane. It had worked, and I had a smile of amazement. With a half mile of grass, you really can go anywhere. Many times I had driven within half a mile of that runway, but never seen it. Never wondered where people go after using it. Never realizing that, although it s a 45-minute drive from my house, it s really pretty close. Never understanding that where people go after taking off from that runway is everywhere .

Ok, I have been an iPhone user since I moved to Japan and got my first smart-phone ever. First a 3s, then a 4s, then a 5s that I dropped into the toilet, so I switched the SIM back into the 4s to have a working phone. Furthermore, I am a heavy music listener and used iTunes Radio for 3 months. Since I am planning to change phones to an Android phone (being fed up with Apple s super-closed environment), I tried out Google Music Plus for about 3 months, too. Here is my verict Google Music is a big pain, far from iTunes in comfort and user friendliness. My move to Android is not very much in danger. General description of the service In principle, iTunes Radio and Google Music Subscription do the same things:

• allow you to have your own music in the cloud
• stream any music from the respective market place to your device
• provide radio stations, pre-curated or based on artist/genre/etc

There are slight differences, that often create confusion, especially with Apple s services: If you are only signed in, you can listen to music stations and can skip some music (limited), but you cannot listen to arbitrary songs from the whole Apple Music library. Then there is another service from Apple, called iTunes Match, that only allows you to upload your music library to the cloud, but other than that again only listening to normal radio stations. Google Music is much simpler, there are only two options: By default it is free to have your music (up to 25000 songs) in the cloud, but if you want to listen to radio stations or any other music in the Google music store, you need a subscription. Prizes and features overview:

	Apple			Google
	No subscription	iTunes Match	iTunes Radio	No subscription	With subscription
Prize	0	25$/year	10$/month	0	10$/month (early access 8$)
Cloud space	0	100000		25000
Extra service	Radio stations	Radio stations	Radio/all music	Nothing	Radio/all music

Music on the go the application Let us first consider the applications provided to listen to music on your smart phone: Apple Music can of course only be used on Apple devices, and uses the built-in Music application. Start up time is about a few seconds after a cold boot (all on my 4s), and music streaming starts with hardly any delay. Responsiveness is good, and the user interface is clear and easy. Google s application is available on iOS and Android. I have tested the latest version on iOS, but it is a pain in the butt:

• Starting the application, even after cold boot, is successful at a rate of 1/10. Most of the time the application crashes right away. This might be a problem due to my low end device (iPhone 4s with 64Gb), but not due to space problems (half empty) nor internet connectivity (wlan).
• Responsiveness is abysmal
• Access to additional content (radio, songs) and your own library is well done, similar to iMusic.

Managing your library Here iTunes is the way to go, bit of a pain when using Linux, but there are other reasons I have a Windows installation in parallel, so I don t mind to boot now and then into Windows. iTunes gives you very powerful tools to change all kind of data. Google gives you a few options: Use your iTunes library in this case all your playlists and ratings (but see later) are also uploaded. I am not sure what happens if I retag a song in iTunes, or change anything else in there. I guess the song will not be re-uploaded, but who knows for sure. Furthermore, you can edit your library via the web interface, but this is rather poor. Searching your library Over times your library grows and there are hundreds if not thousands of artists. So you want to search them. The natural way would be to scroll to the first letter of the artist you are searching, and then look it up. Well, that works perfectly in both Apple s and Google s application unless you are having artists written in some strange script like Japanese or Korean. iTunes allows you to set a field called something like Artist name for sorting , which allows me to put for example or into the artist field, and into the sort field Tomokawa Kazuki and Kim Doo Soo . This way I find the artists in the correct place. Google on the other hand uses simple Unicode order sorting how could you do that? This is simply plain wrong, and everyone should know that by now. Japanese people will never be able to find anything in their list. And in contrast to Goole Contacts Google Music does not support phonetic name fields (similar to the order sort artist) for artists. What I had to do now is to rename all the artists to include first the phonetic name, followed by the proper name, like in Tomokawa Kazuki ( ) . Something I strongly detest! Radio stations I might have a slightly peculiar music taste, but the radio stations mixed by Google are simply a pain. The reason is easy to explain: I live in Japan, and so what Google does is mixing about 80% of J-Pop into the radio, all those happy yodeling girlies I really dislike (see my Anti J-Pop campaign for alternatives yes, they do exist also in Japan!). iTunes radio is here more relaxed it seems. I appreciate Google s trial to cater to local (dis)taste, but besides voting down each and every song I hear I don t see any other option. And honestly, I cannot go through all this voting down without dying from pain inflicted by J-Pop. Other than this, the two radio stations are probably more or less the same but as I said, due to the local colorit it is hard to compare. Rating Ohhh, what a dire point. So there you have you well curated iTunes library with 5-star ratings. I used the ratings in a way that those songs I like get 1 star, those I even like more get 2 stars, and my absolute favorites got 5 stars. I didn t do any negative ratings. iTunes/iOS Music app allows you to easily adjust rating, and they are synced between devices. All as you would expect. Now for Google they did have a 5-star rating system at some time, but:

Thank you for your feedback. We ve decided to remove the 5-star Rating lab. This decision wasn t made lightly, but Thumbs Up/Down is integral for the future vision for Play Music and will be a central design point for our future releases. Please note that we ll continue to store your ratings that you ve set via the star lab or via iTunes, and we ll translate them to thumbs up/down (1-2 stars = thumbs down, 4-5 stars = thumbs up).

Here we go with the move to a new design they threw away the 5-star system, without any reason but integral for future vision rubbish sales speak. Besides being a very very poor rating system to have only an up and down (good-bad-don t care), the translation from my rating system to the thumbs up-down is just plain wrong. <rant>
Why on earth is the current movement to reduce functionality and rob users of control? Gnome 3 is the prime example of how we stupiditize users by taken any freedom from them in the name of simple design. Google now does the same. I am so sick of getting patronized this way, so Gnome3 was completely banished from my computer and replaced by Cinnamon, which uses the same (good) underlying technique, but takes users seriously!
</rant> Overall verdict As much as I would like to see Google Music a valid alternative to iTunes, by now it makes the impression of a quickly hacked together rotten piece of code that tries to get a share of the market without providing equivalent service. Google is using its market presence and convenience to convert people, not features and quality. I can only hope that this changes in the future. That still leaves me with the question move to Android or remain with iOS

Debian Live has passed on. And it has done so in not happy circumstances. (You can search the list archives for more if you are confused.) I have reposted here my response to this one thread because it s all I really want to say, after all of the years of working with the team. I d like to add as a postscript, that while the focus of this article was to remain positive in the face of Daniel s announcement of the closure of his project, that event has left a lot of users confused about the status of live support in Debian going forward. Read my posts here and here addressing that confusion.

On 09/11/15 12:47 PM, Daniel Baumann wrote:
> So long, and thanks for all the fish[7].
>
> Daniel
>
> [7] http://live.debian.net/project/downstream/ Enough bitter words have been said. I don t want to add any more. So: I m proud. Indeed, that long list of downstreams does speak to the impact you ve had in inspiring and equipping people to make their own live images. I m proud to have been a small part of this project. I m thankful. I m thankful that I was able to, through this project, contribute to something for a while that had a positive impact on many people, and made Debian more awesome. I remember the good times. I remember fondly the good times we had in the project s heyday. I certainly found your enthusiasm and vision for the project, Daniel, personally inspiring. It motivated me to contribute. Debconf10 was a highlight among those experiences, but also I had many good times and made many friendships online, too. I m sad. I m sad, because although I made some attempts to liaise between Debian Live and the CD and Installer teams, I don t feel I did an effective job there, and that contributed to the situation we now find ourselves in. If I did you or the project injury in trying to fulfill that role, please forgive me. I m hopeful. I m hopeful that whichever way we all go from here, that the bitterness will not be forever. That we ll heal. That we ll have learned. That we ll move on to accomplish new things, bigger and better things. Thank you, Daniel. Thank you, Debian Live team. Ben

Last week Simon retweeted a link to Don t Feed the Beast the Great Tech Recruiter Infestation. Which reminded me I d been meaning to comment on my own experiences from earlier in the year. I don t entertain the same level of bile as displayed in the post, but I do have a significant level of disappointment in the recruitment industry. I had conversations with 3 different agencies, all of whom were geographically relevant. One contacted me, the other 2 (one I d dealt with before, one that was recommended to me) I contacted myself. All managed to fail to communicate with any level of acceptability. The agency hat contacted me eventually went quiet, after having asked if they could put my CV forward for a role and pushing very hard about when I could interview. The contact in the agency I d dealt with before replied to say I was being passed to someone else who would get in contact. Who of course didn t. And the final agency, who had been recommended, passed me between 3 different people, said they were confident they could find me something, and then went dark except for signing me up to their generic jobs list which failed to have anything of relevance on it. As it happens my availability and skill set were not conducive to results at that point in time, so my beef isn t with the inability to find a role. Instead it s with the poor levels of communication presented by an industry which seems, to me, to have communication as part of the core value it should be offering. If anyone had said at the start Look, it s going to be tricky, we ll see what we can do or Look, that s not what we really deal in, we can t help , that would have been fine. I m fine with explanations. I get really miffed when I m just left hanging. I d love to be able to say I ll never deal with a recruiter again, but the fact of the matter is they do serve a purpose. There s only so far a company can get with word of mouth recruitment; eventually that network of personal connections from existing employees who are considering moving dries up. Advertising might get you some more people, but it can also result in people who are hugely inappropriate for the role. From the company point of view recruiters nominally fulfil 2 roles. Firstly they connect the prospective employer with a potentially wider base of candidates. Secondly they should be able to do some sort of, at least basic, filtering of whether a candidate is appropriate for a role. From the candidate point of view the recruiter hopefully has a better knowledge of what roles are out there. However the incentives to please each side are hugely unbalanced. The candidate isn t paying the recruiter. If you re not paying for it, you re the product may be bandied around too often, but I believe this is one of the instances where it s very applicable. A recruiter is paid by their ability to deliver viable candidates to prospective employers. The delivery of these candidates is the service. Whether or not the candidate is happy with the job is irrelevant beyond them staying long enough that the placement fee can be claimed. The lengthy commercial relationship is ideally between the company and the recruitment agency, not the candidate and the agency. A recruiter wants to be able to say Look at the fine candidate I provided last time, you should always come to me first in future . There s a certain element of wanting the candidate to come back if/when they are looking for a new role, but it s not a primary concern. It is notable that the recommendations I d received were from people who had been on the hiring side of things. The recruiter has a vested interest in keeping the employer happy, in the hope of a sustained relationship. There is little motivation for keeping the candidate happy, as long as you don t manage to scare them off. And, in fact, if you scare some off, who cares? A recruiter doesn t get paid for providing the best possible candidate. Or indeed a candidate who will fully engage with the role. All they re required to provide is a hire-able candidate who takes the role. I m not sure what the resolution is to this. Word of mouth only scales so far for both employer and candidate. Many of the big job websites seem to be full of recruiters rather than real employers. And I m sure there are some decent recruiters out there doing a good job, keeping both sides happy and earning their significant cut. I m sad to say I can t foresee any big change any time soon. [Note I m not currently looking for employment.] [No recruitment agencies were harmed in the writing of this post. I have deliberately tried to avoid outing anyone in particular.]

For my 11th month working on Debian LTS, I carried over 5.5 hours from September and was assigned another 13.5 hours of work by Freexian's Debian LTS initiative. I worked 14 of a possible 19 hours. As I mentioned in the report for September, I uploaded binutils and issued DLA-324-1 early in October. I fixed a few security issues in the kernel, uploaded and issued DLA-325-1. I had some email discussions about long-term support of the Linux kernel with Willy Tarreau (Linux 2.6.32 maintainer) and Greg Kroah-Hartman (overall stable maintainer). Greg normally selects one version per year to maintain as a 'longterm' branch for 2 years, after which he may hand it over to another maintainer. A few upstream versions have received long-term support entirely from another developer. We were in agreement that it's desirable to have fewer of these branches with more developers and distributions contributing to each, but didn't come to a conclusion about how to coordinate this. The topic came up again at the Kernel Summit, and Greg then agreed to select the first kernel version of each calendar year, starting with Linux 4.4 (expected in January). This doesn't fit well with Debian's current release schedule, but I mean to discuss with the release team whether the freeze date can be set to allow inclusion of 2017's LTS kernel. I spent another week in the 'front desk' role, where I triaged new security issues for squeeze. There was a mixture of serious and trivial, old and new (not affecting squeeze) issues. The many ntp issues announced in October were fixed in unstable by a new upstream release. I spent a long time digging out the specific commits that fixed them and comparing with the older version in squeeze. Several of the issues had been introduced in ntp 4.2.7 or 4.2.8 and therefore didn't affect squeeze (or the newer stable releases). Of the fixes that were needed, most applied with minimal changes. Having prepared an update, I asked the ntp maintainer, Kurt Roeckx, to review my work. (The package has a limited test suite and none of the fixes added new tests.) Following this review he added a few more patches, uploaded and issued DLA-335-1. MySQL 5.1, as shipped in squeeze, no longer receives security support from upstream, and the security fixes they do issue are mixed with other changes that make it impractical to backport them. The LTS team is planning to backport the mysql-5.5 package to squeeze while avoiding conflicts with the binaries built from mysql-5.1. Santiago Ruano Rinc n has prepared a backport and I spent some time reviewing this, but haven't yet sent my review comments.

I've been trying to run containers using systemd-nspawn for quite some time. I was always bumping to one or other dead end. This is not systemd-nspawn's fault, rather my impatience stopping me from reading manual pages properly lack of good tutorial like article available online. Compared to this LXC has a quite a lot of good tutorials and howto's available online. This article is my effort to create a notes putting all required information in one place.

Just had a really good couple of sessions: one from the DPL including Q&A, then a good overview of reproducible builds and where we are today from Holger Levsen (h0lger) and Chris Lamb (lamby)

Then a 2 minute silence at 11.00 as is customary in the UK on this date.

Now Lucy Wayland - who says she has a brainful of IEEE754 from a few years ago working on safety-critical systems so she's unburdening by sharing with all of us - everything you wanted to know about floating point but were afraid to ask (and including a primer on signed/unsigned and fixed point as part of the explanation)

Steve Capper's Java presentation is unfortunately not going to take place, so this afternoon's live stream schedule will be different.

Coffee going down well.

Let's Encrypt is now in beta. Here's how to use the automated CA with Hitch TLS proxy, and Varnish HTTP accelerator as a backend for Hitch.

This is the fifth -and last- part in a series of articles about SRE, based on the talk I gave in the Romanian Association for Better Software. Previous articles can be found here: part 1, part 2, part 3, and part 4.

---

Other lessons learned Apart from these guidelines for SRE I have described in previous posts, there were a few other practices I learned at google that I believe could benefit most operations teams out there (and developers too!).

Automate everything It's worth repeating, it is not sustainable to do things manually. When your manual work grows linearly with your usage, either your service is not growing enough, or you will need to hire more people than there are available in the market.

Involve Ops in systems design Nobody knows better than Ops the challenges of a real production environment, getting early feedback from your operations team can save you a lot of trouble down the road. They might give you insights about scalability, capacity planning, redundancy, and the very real limitations of hardware.

Give Ops time and space to innovate If there is so much emphasis on limiting the amount of operational load put on SRE, that freed time is not only used for creating monitoring dashboards or writing documentation. As I said in the beginning, SRE is an engineering team. And when engineers are not running around putting fires off all day long, they get creative. Many interesting innovations come from SREs even when there is no requirement for them.

Write design documents Design documents are not only useful for writing software. Fleshing out the details of any system on paper is a very useful way to find problems early in the process, communicate with your team exactly what are you building, and convince them why it is a great idea.

Review your code, version your changes This is something that is ubiquitous at Google, and not exclusive to SRE: everything is committed into source control: big systems, small scripts, configuration files. It might not seem worth it at the beginning, but having complete history of all your production environment is invaluable. And with source control, another rule comes into play: no change is committed without first being reviewed by a peer. It can be frustrating having to find a reviewer and wait for the approval for every small change, and some times reviews will be suboptimal, but once you get used to it, the benefits greatly out-weights any annoyances.

Before touching, measure Monitoring is not only for sending alerts, it is an essential tool of SRE. It allows you to gauge your availability, evaluate trends, forecast growth, perform forensic analysis. Also very importantly, it should give you the data needed to take decisions based on reality and not on guesses. Before optimising, find really where the bottleneck is; do not decide which database instance to use until you saw the utilisation of the past few months, etc. And speaking of that... We really need to have a chat about your current monitoring system. But that's for another day.

---

I hope I did not bore you too much with these walls of text! Some people told me directly they were enjoying these posts, so soon I will be writing more about related topics. In particular I would like to write about modern monitoring and Prometheus. I would love to hear your comments, questions, and criticisms. Comment