I've previously written about Canonical's obnoxious IP policy and how Mark Shuttleworth admits it's deliberately vague. After spending some time discussing specific examples with Canonical, I've been explicitly told that while Canonical will gladly give me a cost-free trademark license permitting me to redistribute unmodified Ubuntu binaries, they will not tell me what Any redistribution of modified versions of Ubuntu must be approved, certified or provided by Canonical if you are going to associate it with the Trademarks. Otherwise you must remove and replace the Trademarks and will need to recompile the source code to create your own binaries actually means.

Why does this matter? The free software definition requires that you be able to redistribute software to other people in either unmodified or modified form without needing to ask for permission first. This makes it clear that Ubuntu itself isn't free software - distributing the individual binary packages without permission is forbidden, even if they wouldn't contain any infringing trademarks[1]. This is obnoxious, but not inherently toxic. The source packages for Ubuntu could still be free software, making it fairly straightforward to build a free software equivalent.

Unfortunately, while true in theory, this isn't true in practice. The issue here is the apparently simple phrase you must remove and replace the Trademarks and will need to recompile the source code. "Trademarks" is defined later as being the words "Ubuntu", "Kubuntu", "Juju", "Landscape", "Edubuntu" and "Xubuntu" in either textual or logo form. The naive interpretation of this is that you have to remove trademarks where they'd be infringing - for instance, shipping the Ubuntu bootsplash as part of a modified product would almost certainly be clear trademark infringement, so you shouldn't do that. But that's not what the policy actually says. It insists that all trademarks be removed, whether they would embody an infringement or not. If a README says "To build this software under Ubuntu, install the following packages", a literal reading of Canonical's policy would require you to remove or replace the word "Ubuntu" even though failing to do so wouldn't be a trademark infringement. If an @ubuntu.com email address is present in a changelog, you'd have to change it. You wouldn't be able to ship the juju-core package without renaming it and the application within. If this is what the policy means, it's so impractical to be able to rebuild Ubuntu that it's not free software in any meaningful way.

This seems like a pretty ludicrous interpretation, but it's one that Canonical refuse to explicitly rule out. Compare this to Red Hat's requirements around Fedora - if you replace the fedora-logos, fedora-release and fedora-release-notes packages with your own content, you're good. A policy like this satisfies the concerns that Dustin raised over people misrepresenting their products, but still makes it easy for users to distribute modified code to other users. There's nothing whatsoever stopping Canonical from adopting a similarly unambiguous policy.

Mark has repeatedly asserted that attempts to raise this issue are mere FUD, but he won't answer you if you ask him direct questions about this policy and will insist that it's necessary to protect Ubuntu's brand. The reality is that if Debian had had an identical policy in 2004, Ubuntu wouldn't exist. The effort required to strip all Debian trademarks from the source packages would have been immense[2], and this would have had to be repeated for every release. While this policy is in place, nobody's going to be able to take Ubuntu and build something better. It's grotesquely hypocritical, especially when the Ubuntu website still talks about their belief that people should be able to distribute modifications without licensing fees.

All that's required for Canonical to deal with this problem is to follow Fedora's lead and isolate their trademarks in a small set of packages, then tell users that those packages must be replaced if distributing a modified version of Ubuntu. If they're serious about this being a branding issue, they'll do it. And if I'm right that the policy is deliberately obfuscated so Canonical can encourage people to buy licenses, they won't. It's easy for them to prove me wrong, and I'll be delighted if they do. Let's see what happens.

[1] The policy is quite clear on this. If you want to distribute something other than an unmodified Ubuntu image, you have two choices:

1. Gain approval or certification from Canonical
2. Remove all trademarks and recompile the source code

Note that option 2 requires you to rebuild even if there are no trademarks to remove.

[2] Especially when every source package contains a directory called "debian"

comments

I'm pleased with the way that Lumail2 development is proceeding, and it is reaching a point where there will be a second source-release. I've made a lot of changes to the repository recently, and most of them boil down to moving code from the C++ side of the application, over to the Lua side. This morning, for example, I updated the handing of index.limit to be entirely Lua based.

When you open a Maildir folder you see the list of messages it contains, as you would expect. The notion of the index.limit is that you can limit the messages displayed, for example:

• See all messages: Config:set( "index.limit", "all")
• See only new/unread messages: Config:set( "index.limit", "new")
• See only messages which arrived today: Config:set( "index.limit", "today")
• See only messages which contain "Steve" in their formatted version: Config:set( "index.limit", "steve")

These are just examples that are present as defaults, but they give an idea of how things can work. I guess it isn't so different to Mutt's "limit" facilities - but thanks to the dynamic Lua nature of the application you can add your own with relative ease.

One of the biggest changes, recently, was the ability to display coloured text! That was always possible before, but a single line could only be one colour. Now colours can be mixed within a line, so this works as you might imagine: Panel:append( "$[RED]This is red, $[GREEN]green, $[WHITE]white, and $[CYAN]cyan!" ) Other changes include a persistant cache of "stuff", which is Lua-based, the inclusion of at least one luarocks library to parse Date: headers, and a simple API for all our objects. All good stuff. Perhaps time for a break in the next few weeks, but right now I think I'm making useful updates every other evening or so.

Depending on the application, sometimes you want to have some socket operations going (such as loading a website) and have stdin being read. There are plenty of examples for this in python which usually boil down to making stdin behave like a socket and mixing it into the list of sockets select() cares about. A while ago I asked an email list could I have pysnmp use a different socket map so I could add my own sockets in (UDP, TCP and a zmq to name a few) and the Ilya the author of pysnmp explained how pysnmp can use a foreign socket map. This sample code below is merely an mixture of Ilya s example code and the way stdin gets mixed into the fold. I have also updated to the high-level pysnmp API which explains the slight differences in the calls. Some interesting lines from the above code:

• Lines 8-11 are the stdin class that is called (or rather its handle_read method is) when there is text available on stdin.
• Line 34 is where pysnmp is told to use our socket map and not its inbuilt one
• Line 37 is where we have used the socket map to say if we get input from stdin, what is the handler.
• Lines 39-46 are sending a SNMP query using the high-level API
• Lines 48-51 are my simple socket poller

With all this I can handle keyboard presses and network traffic, such as a simple SNMP poll.

I m excited to see other people thinking about userspace-to-kernel attack surface reduction ideas. Theo de Raadt recently published slides describing Pledge. This uses the same ideas that seccomp implements, but with less granularity. While seccomp works at the individual syscall level and in addition to killing processes, it allows for signaling, tracing, and errno spoofing. As de Raadt mentions, Pledge could be implemented with seccomp very easily: libseccomp would just categorize syscalls. I don t really understand the presentation s mention of Optional Security , though. Pledge, like seccomp, is an opt-in feature. Nothing in the kernel refuses to run unpledged programs. I assume his point was that when it gets ubiquitously built into programs (like stack protector), it s effectively not optional (which is alluded to later as comprehensive applicability ~= mandatory mitigation ). Regardless, this sensible (though optional) design gets me back to his slide on seccomp, which seems to have a number of misunderstandings:

• A Turing complete eBPF program watches your program Strictly speaking, seccomp is implemented using a subset of BPF, not eBPF. And since BPF (and eBPF) programs are guaranteed to halt, it makes seccomp filters not Turing complete.
• Who watches the watcher? I don t even understand this. It s in the kernel. The kernel watches your program. Just like always. If this is a question of BPF program verification, there is literally a program verifier that checks various properties of the BPF program.
• seccomp program is stored elsewhere This, with the next statement, is just totally misunderstood. Programs using seccomp define their program in their own code. It s used the same way as the Pledge examples are shown doing.
• Easy to get desyncronized either program is updated As above, this just isn t the case. The only place where this might be true is when using seccomp on programs that were not written natively with seccomp. In that case, yes, desync is possible. But that s one of the advantages of seccomp s design: a program launcher (like minijail or systemd) can declare a seccomp filter for a program that hasn t yet been ported to use one natively.
• eBPF watcher has no real idea what the program under observation is doing I don t understand this statement. I don t see how Pledge would have a real idea either: they re both doing filtering. If we get AI out of our syscall filters, we re in serious trouble. :)

OpenBSD has some interesting advantages in the syscall filtering department, especially around sockets. Right now, it s hard for Linux syscall filtering to understand why a given socket is being used. Something like SOCK_DNS seems like it could be quite handy. Another nice feature of Pledge is the path whitelist feature. As it s still under development, I hope they expand this to include more things than just paths. Argument inspection is a weak point for seccomp, but under Linux, most of the arguments are ultimately exposed to the LSM layer. Last year I experimented with creating a seccomp LSM for path matching where programs could declare whitelists, similar to standard LSMs. So, yes, Linux could match this API on seccomp . It d just take some extensions to libseccomp to implement pledge(), as I described at the top. With OpenBSD doing a bunch of analysis work on common programs, it d be excellent to see this usable on Linux too. So far on Linux, only a few programs (e.g. Chrome, vsftpd) have bothered to do this using seccomp, and it could be argued that this is ultimately due to how fine grained it is.

2015, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.

Ok, I have been an iPhone user since I moved to Japan and got my first smart-phone ever. First a 3s, then a 4s, then a 5s that I dropped into the toilet, so I switched the SIM back into the 4s to have a working phone. Furthermore, I am a heavy music listener and used iTunes Radio for 3 months. Since I am planning to change phones to an Android phone (being fed up with Apple s super-closed environment), I tried out Google Music Plus for about 3 months, too. Here is my verict Google Music is a big pain, far from iTunes in comfort and user friendliness. My move to Android is not very much in danger. General description of the service In principle, iTunes Radio and Google Music Subscription do the same things:

• allow you to have your own music in the cloud
• stream any music from the respective market place to your device
• provide radio stations, pre-curated or based on artist/genre/etc

There are slight differences, that often create confusion, especially with Apple s services: If you are only signed in, you can listen to music stations and can skip some music (limited), but you cannot listen to arbitrary songs from the whole Apple Music library. Then there is another service from Apple, called iTunes Match, that only allows you to upload your music library to the cloud, but other than that again only listening to normal radio stations. Google Music is much simpler, there are only two options: By default it is free to have your music (up to 25000 songs) in the cloud, but if you want to listen to radio stations or any other music in the Google music store, you need a subscription. Prizes and features overview:

	Apple			Google
	No subscription	iTunes Match	iTunes Radio	No subscription	With subscription
Prize	0	25$/year	10$/month	0	10$/month (early access 8$)
Cloud space	0	100000		25000
Extra service	Radio stations	Radio stations	Radio/all music	Nothing	Radio/all music

Music on the go the application Let us first consider the applications provided to listen to music on your smart phone: Apple Music can of course only be used on Apple devices, and uses the built-in Music application. Start up time is about a few seconds after a cold boot (all on my 4s), and music streaming starts with hardly any delay. Responsiveness is good, and the user interface is clear and easy. Google s application is available on iOS and Android. I have tested the latest version on iOS, but it is a pain in the butt:

• Starting the application, even after cold boot, is successful at a rate of 1/10. Most of the time the application crashes right away. This might be a problem due to my low end device (iPhone 4s with 64Gb), but not due to space problems (half empty) nor internet connectivity (wlan).
• Responsiveness is abysmal
• Access to additional content (radio, songs) and your own library is well done, similar to iMusic.

Managing your library Here iTunes is the way to go, bit of a pain when using Linux, but there are other reasons I have a Windows installation in parallel, so I don t mind to boot now and then into Windows. iTunes gives you very powerful tools to change all kind of data. Google gives you a few options: Use your iTunes library in this case all your playlists and ratings (but see later) are also uploaded. I am not sure what happens if I retag a song in iTunes, or change anything else in there. I guess the song will not be re-uploaded, but who knows for sure. Furthermore, you can edit your library via the web interface, but this is rather poor. Searching your library Over times your library grows and there are hundreds if not thousands of artists. So you want to search them. The natural way would be to scroll to the first letter of the artist you are searching, and then look it up. Well, that works perfectly in both Apple s and Google s application unless you are having artists written in some strange script like Japanese or Korean. iTunes allows you to set a field called something like Artist name for sorting , which allows me to put for example or into the artist field, and into the sort field Tomokawa Kazuki and Kim Doo Soo . This way I find the artists in the correct place. Google on the other hand uses simple Unicode order sorting how could you do that? This is simply plain wrong, and everyone should know that by now. Japanese people will never be able to find anything in their list. And in contrast to Goole Contacts Google Music does not support phonetic name fields (similar to the order sort artist) for artists. What I had to do now is to rename all the artists to include first the phonetic name, followed by the proper name, like in Tomokawa Kazuki ( ) . Something I strongly detest! Radio stations I might have a slightly peculiar music taste, but the radio stations mixed by Google are simply a pain. The reason is easy to explain: I live in Japan, and so what Google does is mixing about 80% of J-Pop into the radio, all those happy yodeling girlies I really dislike (see my Anti J-Pop campaign for alternatives yes, they do exist also in Japan!). iTunes radio is here more relaxed it seems. I appreciate Google s trial to cater to local (dis)taste, but besides voting down each and every song I hear I don t see any other option. And honestly, I cannot go through all this voting down without dying from pain inflicted by J-Pop. Other than this, the two radio stations are probably more or less the same but as I said, due to the local colorit it is hard to compare. Rating Ohhh, what a dire point. So there you have you well curated iTunes library with 5-star ratings. I used the ratings in a way that those songs I like get 1 star, those I even like more get 2 stars, and my absolute favorites got 5 stars. I didn t do any negative ratings. iTunes/iOS Music app allows you to easily adjust rating, and they are synced between devices. All as you would expect. Now for Google they did have a 5-star rating system at some time, but:

Thank you for your feedback. We ve decided to remove the 5-star Rating lab. This decision wasn t made lightly, but Thumbs Up/Down is integral for the future vision for Play Music and will be a central design point for our future releases. Please note that we ll continue to store your ratings that you ve set via the star lab or via iTunes, and we ll translate them to thumbs up/down (1-2 stars = thumbs down, 4-5 stars = thumbs up).

Here we go with the move to a new design they threw away the 5-star system, without any reason but integral for future vision rubbish sales speak. Besides being a very very poor rating system to have only an up and down (good-bad-don t care), the translation from my rating system to the thumbs up-down is just plain wrong. <rant>
Why on earth is the current movement to reduce functionality and rob users of control? Gnome 3 is the prime example of how we stupiditize users by taken any freedom from them in the name of simple design. Google now does the same. I am so sick of getting patronized this way, so Gnome3 was completely banished from my computer and replaced by Cinnamon, which uses the same (good) underlying technique, but takes users seriously!
</rant> Overall verdict As much as I would like to see Google Music a valid alternative to iTunes, by now it makes the impression of a quickly hacked together rotten piece of code that tries to get a share of the market without providing equivalent service. Google is using its market presence and convenience to convert people, not features and quality. I can only hope that this changes in the future. That still leaves me with the question move to Android or remain with iOS

I've been trying to run containers using systemd-nspawn for quite some time. I was always bumping to one or other dead end. This is not systemd-nspawn's fault, rather my impatience stopping me from reading manual pages properly lack of good tutorial like article available online. Compared to this LXC has a quite a lot of good tutorials and howto's available online. This article is my effort to create a notes putting all required information in one place.

Let's Encrypt is now in beta. Here's how to use the automated CA with Hitch TLS proxy, and Varnish HTTP accelerator as a backend for Hitch.

At the miniDebConfUK in Cambridge, November 2015, there was a vmdebootstrap sprint. vmdebootstrap is written in python and the sprint built on the changes I made during DebConf15. The primary aim was to split out the code from a single python script and create modules which would be useful to other tools and make the source code itself easier to follow. Whilst doing this, I worked with Steve McIntyre to implement UEFI support into vmdebootstrap. This version reached experimental shortly after DebConf15. At the sprint, the squashfs support in vmdebootstrap was improved to be more useful in the preparation of live images rather than simply using mksquashfs as a compression algorithm for the entire image. I also improved and extended the documentation for vmdebootstrap. vmdebootstrap is now extensible and modular. Iain Learmonth used this new modular support in the development of live-build-ng which uses vmdebootstrap, live-boot and live-config to replace the role of live-build in the generation of official Debian live images by the debian-cd team. Iain Learmonth demonstrated that the new support in vmdebootstrap can be used to create working live images, including adding support for live images on any architecture supporting the Linux kernel in Debian and adding support for not only Grub but UEFI as well. A UEFI grub live image built by live-build-ng was demonstrated after only two days work on the vmdebootstrap base, wrapping support from live-boot and live-config with customisation for UEFI and grub config. vmdebootstrap and live-build-ng have been explicitly designed within the debian-cd team to remove the need to run live-build to create official Debian live images, replacing live-build with live-build-ng and the vmdebootstrap support. This brings working support for multiple architectures and UEFI to live images. To support the new functionality, the vmdebootstrap and debian-cd team have long sought a test suite for vmdebootstrap and Lars Wirzenius implemented one during the vmdebootstrap sprint. We now have fast tests with a pre-commit hook and build tests which are best with a local mirror. The test suite uses cmdtest & yarn. This test suite will be used for all future changes patches will not be accepted if the test suite fails and any substantially new code must provide working test cases. The fast tests can run without sudo, I expect to be able to sort out ci.debian.net testing for the fast tests too. There is also an outline for testing vmdebootstrap builds in (localhost) LAVA using a lava-submit.py example. All this work will arrive in unstable as vmdebootstrap 1.2 soon and is now in the master branch. The old codehelp/modules branch has been merged and removed.

The Washington Post published an article today which describes the ongoing tension between the security community and Linux kernel developers. This has been roundly denounced as FUD, with Rob Graham going so far as to claim that nobody ever attacks the kernel.

Unfortunately he's entirely and demonstrably wrong, it's not FUD and the state of security in the kernel is currently far short of where it should be.

An example. Recent versions of Android use SELinux to confine applications. Even if you have full control over an application running on Android, the SELinux rules make it very difficult to do anything especially user-hostile. Hacking Team, the GPL-violating Italian company who sells surveillance software to human rights abusers, found that this impeded their ability to drop their spyware onto targets' devices. So they took advantage of the fact that many Android devices shipped a kernel with a flawed copy_from_user() implementation that allowed them to copy arbitrary userspace data over arbitrary kernel code, thus allowing them to disable SELinux.

If we could trust userspace applications, we wouldn't need SELinux. But we assume that userspace code may be buggy, misconfigured or actively hostile, and we use technologies such as SELinux or AppArmor to restrict its behaviour. There's simply too much userspace code for us to guarantee that it's all correct, so we do our best to prevent it from doing harm anyway.

This is significantly less true in the kernel. The model up until now has largely been "Fix security bugs as we find them", an approach that fails on two levels:

1) Once we find them and fix them, there's still a window between the fixed version being available and it actually being deployed
2) The forces of good may not be the first ones to find them

This reactive approach is fine for a world where it's possible to push out software updates without having to perform extensive testing first, a world where the only people hunting for interesting kernel vulnerabilities are nice people. This isn't that world, and this approach isn't fine.

Just as features like SELinux allow us to reduce the harm that can occur if a new userspace vulnerability is found, we can add features to the kernel that make it more difficult (or impossible) for attackers to turn a kernel bug into an exploitable vulnerability. The number of people using Linux systems is increasing every day, and many of these users depend on the security of these systems in critical ways. It's vital that we do what we can to avoid their trust being misplaced.

Many useful mitigation features already exist in the Grsecurity patchset, but a combination of technical disagreements around certain features, personality conflicts and an apparent lack of enthusiasm on the side of upstream kernel developers has resulted in almost none of it landing in the kernels that most people use. Kees Cook has proposed a new project to start making a more concerted effort to migrate components of Grsecurity to upstream. If you rely on the kernel being a secure component, either because you ship a product based on it or because you use it yourself, you should probably be doing what you can to support this.

Microsoft received entirely justifiable criticism for the terrible state of security on their platform. They responded by introducing cutting-edge security features across the OS, including the kernel. Accusing anyone who says we need to do the same of spreading FUD is risking free software being sidelined in favour of proprietary software providing more real-world security. That doesn't seem like a good outcome.

comments

Last week, Senator Omar Fayad presented one of the prime examples of a poorly redacted law that, if enacted, will make basically any way of computer use illegal. And yes, even if he states this is merely a draft, it has so many factual and conceptual errors that there is no way to trust sanity can be regained at any point. Oh, and before I continue with this rant: If the topic interests you, I suggest you to read the 10 key points about Ley Fayad, the worst Internet initiative in history, published by r3d.mx. [update] An English equivalent of the work at r3d, at revolution-news.com: #LeyFayad: The Worst Bill in Internet History The full text (in Spanish, of course) for the law initiative is available at the Senate webpage; the law will be called Ley Federal para Prevenir y Sancionar los Delitos Inform ticos (Federal law to prevent and punish informatic felonies<) A bad name to start with, as there are many laws already in that contested area. I started reading with the preamble (Exposici n de motivos), which already shows bad signs of imprecise redaction and is plagued with factual errors (i.e. asserting that the real danger stems from the Web migrating to the Web 2.0, from which stems that this migration and not any previous one. Or by stating that (quoting+translating a full paragraph):

Activities such as electronic commerce, digital periodism, publicity and the opinions, messages or elements written in social networks can lead to patrimonial, reputation, honor or professional activity losses for people.

He continues by stating that only 16% of the countries have some kind of cybersecurity strategy (and, of course, Mexico doesn't). That... Well, is very hard to believe, as Mexico has two separate policial groups devoted to cybersecurity, and laws regulating from electronic signatures, commerce, identity, privacy, use and abuse, and a long list. Of course, as most law proposals go, it quickly decays into a dry, boring document... And I must admit I didn't fully read it, but picked here and there. I won't copy in full the note I mentioned at the beginning at r3d.mx, but will continue with some strange points, such as:

Article 16

Every person that, without the corresponding authorization or exceeding the authorization confered, accesses, intercepts, interfers or uses an information system, will be punished by one to eight years of prision and fined by 800 to 1000 days of minimum wage

So, yes, borrowing your computer without getting explicit permission, or playing around with the options in kiosks, or tons of whatever we curious people do with systems we encounter are basis for jail. (And yes, fines in this country are expressed in "days of minimum wage", which goes at ~MX$70 per day, which is ~US$4). But it gets funkier quickly:

Article 17

Whoever fraudulently destroys, disables, damages or in any way alters the working of an informatic system or any of its components, will be punished by fice to fifteen years of jail and fined by up to a thousand minimum wage days The same punishment will be given to whoever, without authorization, destroys, damages, modifies, divulges, transfers or disables information contained in any Informatic System or any of its components. The punishment will be ten to twenty years in prision and a fine of up to a thousand days of minimum wage if the effects here mentioned are done by the creation, introduction or fraudulent transmission, by any means, of an informatic weapon or malicious code

This law is meant to protect against cyberfelonies, if such a thing exists. However, here we are putting at risk people even for accidental equipment destructions. I dropped your portable hard disk with my elbow off the table? Accuse me of acting fraudulently, and I'm up for a serious jail time. And yes, laws are meant to be interpreted... And I don't want to be at the receiving end of this one! In this last article, Fayad mentions informatic weapons, which are defined in the preamble as any informatic program, informatic system, or in general, any device or material created or designed with the purpose of committing an informatic crime. So the very next article makes me, as it should make all of my fellow students and researchers, very uneasy:

Article 18

Whoever uses informatic weapons or malicious code will be imprisioned by two to six years, and fined with 200 to 500 days of minimum wage.

Article 19

Whoever builds, distributes, commerces with informatic weapons or malicious codes will be punished by three to seven years of prision and 200 to 500 days of minimum wage.

If we need to analyze malware for our classes (or for paid work, or as a hobby), we clearly fall in article 18. If we write something that can be classified as malware (without even releasing it, as an academic excercise only!), we are covered by article 19. If I give my students code that's known to be malicious (which could be as inofensive as linking to a well-known Web comic), I'm also covered by article 19. I'll jump all the way to article 31 (reproduced only partially):

Article 31

Whoever, by any means, creates, captures, records, copies, alters, duplicates, clones or deletes the information contained in a credit or debit card (...) will be punished by 8 to 14 years of prision and 300 to 500 days of minimum wage. (...)

This clearly disincentivates any way of e-commerce. When I try to buy anything online, I have to capture+copy my (rightfully owned) credit card data. The services provider has to copy, process and then delete said information. Any e-transaction is punished by jail! Well... But thinking about this again, maybe I shouldn't be so worried about the malware distribution issue at my classes. There are clearer and more contundent articles. Say...

Article 35

Whoever convenes, organizes, is part of, or executes a cibernetic attack, will be punished by 20 to 30 years of prision and fined with 100 to 1000 days of minimum wage

Of course we have convened, organized, been part of and executed cibernetic attacks at the computer security lab at ESIME. Why would there be such a lab otherwise? Then, there are clear indications that the Senator didn't understand the topic his team was working on:

Article 37

Who manipulates the digital seals used by command of the public authority will be punished with 240 days of community work

Now... What is a digital seal? It's not a phisical one that does not allow opening the doors to a business found at fault, but something that just proves a document is legitimate and pristine. How can I manipulate them? Of course, if the seals are MD5-based, I can easily forge them (and SHA1-based, it seems they will be broken enough soon to be considered no longer trustable)... But that's about it! And there is more, lots more. I'm swamped with work, and have to get back to it. But chapters the following chapters have a lot of potential for finding holes. PS - And yes, the only use I do of Twitter is via the headlines in my blog ;-) [update] Ley Fayad is dead, yay! \o/ The senator withdrew the proposal.

This is the fourth part in a series of articles about SRE, based on the talk I gave in the Romanian Association for Better Software. Previous articles can be found here: part 1, part 2, and part 3. As a side note, I am writing this from Heathrow Airport, about to board my plane to EZE. Buenos Aires, here I come!

---

So, what does SRE actually do? Enough about how to keep the holy wars under control and how to get work away from Ops. Let's talk about some of the things that SRE does. Obviously, SRE runs your service, performing all the traditional SysAdmin duties needed for your cat pictures to reach their consumers: provisioning, configuration, resource allocation, etc. They use specialised tools to monitor the service, and get alerted as soon as a problem is detected. They are also the ones waking up to fix your bugs in the middle of the night. But that is not the whole story: reliability is not only impacted by new launches. Suddenly usage grows, hardware fails, networks lose packets, solar flares flip your bits... When working at scale, things are going to break more often than not. You want these breakages to affect your availability as little as possible, there are three strategies you can apply to this end: minimise the impact of each failure, recover quickly, and avoid repetition. And of course, the best strategy of them all: preventing outages from happening at all.

Minimizing the impact of failures A single failure that takes the whole service down will affect severely your availability, so the first strategy as an SRE is to make sure your service is fragmented and distributed across what is called "failure domains". If the data center catches fire, you are going to have a bad day, but it is a lot better if only a fraction of your users depend on that one data center, while the others keep happily browsing cat pictures. SREs spend a lot of their time planning and deploying systems that span the globe to maximise redundancy while keeping latencies at reasonable levels.

Recovering quickly Many times, retrying after a failure is actually the best option. So another strategy is to automatically restart in milliseconds any piece of your system that fails. This way, less users are affected, while a human has time to investigate and fix the real problem. If a human needs to intervene, it is crucial that they get notified as fast as possible, and that they have quick access to all the information that is needed to solve the problem: detailed documentation of the production environment, meaningful and extensive monitoring, play-books1, etc. After all, at 2 AM you probably don't remember in which country the database shard lives, or what were the series of commands to redirect all the traffic to a different location. Implementing monitoring and automated alerts, writing documentation and play-books, and practising for disaster are other areas where SREs devote much effort.

Avoiding repetition Outages happen, pages ring, problems get fixed, but it should always be a chance for learning and improving. A page should require a human to think about a problem and find a solution. If the same problem keeps appearing, the human is not needed any more: the problem needs to be fixed at the root. Another key aspect of dealing with incidents is to write post-mortems2. Every incident should have one, and these should be tools for learning, not finger-pointing. Post-mortems can be an excellent tool, if people are honest about their mistakes, they are not used to blame other people, and issues are followed up by bug reports and discussions.

Preventing outages Of course nobody can prevent hard drives from failing, but there are certain classes of outages that can be forecasted with careful analysis. Usage spikes can bring a service down, but an SRE team will ensure that the systems are load-tested at higher-than-normal rates. They could also be prepared to quickly scale the service, provided the monitoring system will alert as soon as a certain threshold is reached. Monitoring is actually the key part in this: measuring relevant metrics for all the components of a system, and following their trends over time, SREs can completely avoid many outages: latencies growing out of acceptable bounds, disks filling up, progressive degradation of components, are all examples of typical problems automatically monitored (and alerted on) in an SRE team.

---

This is getting pretty long, so the next post will be the last one of these series, with some extra tips from SRE experience that I am sure can be applied in many places.

---

1. A play-book is a document containing critical information needed when dealing with a specific alert: possible causes, hints for troubleshooting, links to more documentation. Some monitoring systems will automatically add a play-book URL to every alert sent, and you should be doing it too.
2. Similarly to their real-life counterparts, a post-mortem is the process of examining a dead body (the outage), gutting it out and trying to understand what caused its demise.

Comment

I'm an IT engineering student at the University of Seville, where there are 3 flavours to choose:
* Computers engineering: mostly focused on hardware
* Software engineering: focused on software
* IT engineering: a mixture of things, including subjects dedicated to security and virtualization

Each of these last a total amount of 4 years (better said: courses).

Software Engineering is mostly the traditional IT engineering and it's what I'm doing.

The first course is almost common to all engineering: algebra, calculus, phyisics, electronics and so on.
You may find these subjects in all IT engineering flavours and also in other traditional engineering
like telecommunications, mechanical, industrial or aeronautics.

In the second year, the subjects tend to be specific for IT engineering: networking, databases, algorithms and so on.
The third and fourth courses are almost fully specific for each flavour of IT engineering.

The problem I've found is in the third year of Software engineering.
We have here subjects like the following:
* requirements engineering
* design and testing
* software process management

All of these are focused in what they consider our biggest possibility in our professional career: the regional government.

It turns out this is not Silicon Valley, and the regional government (Junta de Andaluc a) is in fact the biggest IT player in our region.
They run thousands of servers, networks and digital services both for the public and for internal administrative usage.
The government is known for massively running concrete technologies: Oracle and java. Almost all of the biggest applications are running these technologies.

The problem here is that our subjects are fully headed towards these technologies, also using methodologies
that I consider outdated. They think you are going to work in a classical software factory on the classical software consulting business.
And they do this explicitly: the first day in the 'design & testing' subject we were told by the professor that we are likely going to work in the 'local industry', so we need to learn these technologies and no others.

An example: last week I was given by a professor a VirtualBox (sic) virtual machine running Windows XP (yeah...) just to run Eclipse (for java development) in a controlled environment. They expect the host OS to be Windows as well.
Other example: to this day, no subject which requires the usage of a VCS is using git. They stick to svn.

This is very sad!

The other day, it felt event worse because a RedHat internship was announce in a college mailing list, which explicitly excluded students from Software Engineering.... because all the reasons above. How is this possible? RedHat is a software company!

Let's be constructive. These are some things I would like to see in Software Engineering:
* other software workflows (for example, the linux kernel development model or debian package lifecycle)
* modern free libre open source technologies! python! golang!
* packaging (for example: debian, rpm, docker...)
* new database systems (nosql) or at least no more Oracle (please switch to PostgreSQL or whatever)
* no more java! enough of java!

BTW some time ago I read somewhere that no student should graduate without contributing to a major open source project. I can't agree more.

concurrent-output released yesterday got a lot of fun features. It now does full curses-style minimization of the output, to redraw updated lines with optimal efficiency. And supports multiline regions/wrapping too long lines. And allows the user to embed ANSI colors in a region. 3 features that are in some tension and were fun to implement all together. But I have a more interesting feature to blog about... I've added the ability for the content of a Region to be determined by a (STM transaction). Here, for example, is a region that's a clock: There's something magical about this. Whenever a new value is written into the TVar, concurrent-output automatically knows that this region needs to be updated. How does it know how to do that? Magic of STM. Basically, concurrent-output composes all the STM transactions of Regions, and asks STM to wait until there's something new to display. STM keeps track of whatever TVars might be looked at, and so can put the display thread to sleep until there's a change to display. Using STM I've gotten extensability for free, due to the nice ways that STM transactions compose. A few other obvious things to do with this: Compose 2 regions with padding so they display on the same line, left and right aligned. Trim a region's content to the display width. (Handily exported by concurrent-output in a TVar for this kind of thing.)

---

I'm tempted to write a console spreadsheet using this. Each visible cell of the spreadsheet would have its own region, that uses a STM transaction to display. Plain data Cells would just display their current value. Cells that contain a function would read the current values of other Cells, and use that to calculate what to display. Which means that a Cell containing a function would automatically update whenever any of the Cells that it depends on were updated! Do you think that a simple interactive spreadsheet built this way would be more than 100 lines of code?

Building on top of concurrent-output, and some related work Joachim Breitner did earlier, I now have a kind of equivilant to a tiling window manager, except it's managing regions of the console for different parts of a single program. Here's a really silly demo, in an animated gif: Not bad for 23 lines of code, is that? Seems much less tedious to do things this way than using ncurses. Even with its panels, ncurses requires you to think about layout of various things on the screen, and many low-level details. This, by contrast, is compositional, just add another region and a thread to update it, and away it goes. So, here's an apt-like download progress display, in 30 lines of code. Not only does it have regions which are individual lines of the screen, but those can have sub-regions within them as seen here (and so on). And, log-type messages automatically scroll up above the regions. External programs run by createProcessConcurrent will automatically get their output/errors displayed there, too. What I'm working on now is support for multiline regions, which automatically grow/shrink to fit what's placed in them. The hard part, which I'm putting the finishing touches on, is to accurately work out how large a region is before displaying it, in order to lay it out. Requires parsing ANSI codes amoung other things. STM rules There's so much concurrency, with complicated interrelated data being updated by different threads, that I couldn't have possibly built this without Software Transactional Memory. Rather than a nightmare of locks behind locks behind locks, the result is so well behaved that I'm confident that anyone who needs more control over the region layout, or wants to do funky things can dive into to the STM interface and update the data structures, and nothing will ever deadlock or be inconsistent, and as soon as an update completes, it'll display on-screen. An example of how powerful and beuatiful STM is, here's how the main display thread determines when it needs to refresh the display: So, it composes all these STM actions that can wait on various kinds of changes, to get one big action, that waits for all of the above, and builds up a nice sum type to represent what's changed. Another example is that the whole support for sub-regions only involved adding 30 lines of code, all of it using STM, and it worked 100% the first time.

---

Available in concurrent-output 1.1.0.

A new major release of littler is now available. And for the first time in the nine years since 2006 when Jeff started the effort (which I joined not long after) we are now a CRAN package. This required a rewrite of the build system, foregoing the calls to aclocal, autoheader, automake and leaving just a simpler autoconf layer creating a configure script and a simple src/Makevars.in. Not only does R provide such a robust and well-understood build system (which I got to understand reasonably well given all my R packages, but being on CRAN and leveraging its mechanism for installation and upgrades is clearly worth the change. There may be a moment or two of transition. While we can create a binary in an R package, we cannot (generally) copy to /usr/bin or /usr/local/bin as part of the build process (for lack of write-rights to those directories). So if you do not have r in the $PATH and load the package, it makes a suggestion (which needs a linebreak which I added here):

R> library(littler)
The littler package provides 'r' as a binary.
You could link to the 'r' binary installed in
'/usr/local/lib/R/site-library/littler/bin/r' from
'/usr/local/bin' in order to use 'r' for scripting.
R> 

Similarly, you could copy (or softlink) r to ~/bin if that is in your $PATH. The Debian (and Ubuntu) packages will continue to provide /usr/bin/r as before. Note thah these packages will now be called r-cran-littler to match all other CRAN packages. The NEWS file entry is below.

Changes in littler version 0.3.0 (2015-10-29)

• Changes in build system
  • First CRAN Release as R package following nine years of source releases
  • Script configure, src/Makevars.in and remainder of build system rewritten to take advantage of the R package build infrastructure
  • Reproducible builds are better supported as the (changing) compilation timestamps etc are only inserted for 'verbose builds' directly off the git repo, but not for Debian (or CRAN) builds off the release tarballs
• Changes in littler functionality
  • Also source $R_HOME/etc/Rprofile.site and ~/.Rprofile if present
• Changes in littler documentation
  • Added new vignette with examples

Full details for the littler release are provided as usual at the ChangeLog page. The code is available via the GitHub repo, from tarballs off my littler page and the local directory here -- and now of course all from its CRAN page and via install.packages("littler"). A fresh package has gone to the incoming queue at Debian where it will a few days as the binary packages was renamed from littler to r-cran-littler matching all other CRAN packages. Michael Rutter will probably have new Ubuntu binaries at CRAN once the source package gets into Debian proper. Comments and suggestions are welcome at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.