Search Results: "xam"

5 June 2015

Simon McVittie: Why polkit (or, how to mount a disk on modern Linux)

I've recently found myself explaining polkit (formerly PolicyKit) to one of Collabora's clients, and thought that blogging about the same topic might be useful for other people who are confused by it; so, here is why udisks2 and polkit are the way they are. As always, opinions in this blog are my own, not Collabora's. Privileged actions Broadly, there are two ways a process can do something: it can do it directly (i.e. ask the kernel directly), or it can use inter-process communication to ask a service to do that operation on its behalf. If it does it directly, the components that say whether it can succeed are the Linux kernel's normal permissions checks (DAC), and if configured, AppArmor, SELinux or a similar MAC layer. All very simple so far. Unfortunately, the kernel's relatively coarse-grained checks are not sufficient to express the sorts of policies that exist on a desktop/laptop/mobile system. My favourite example for this sort of thing is mounting filesystems. If I plug in a USB stick with a FAT filesystem, it's reasonable to expect my chosen user interface to either mount it automatically, or let me press a button to mount it. Similarly, to avoid data loss, I should be able to unmount it when I'm finished with it. However, mounting and unmounting a USB stick is fundamentally the same system call as mounting and unmounting any other filesystem - and if ordinary users can do arbitrary mount system calls, they can cause all sorts of chaos, for instance by mounting a filesystem that contains setuid executables (privilege escalation), or umounting a critical OS filesystem like /usr (denial of service). Something needs to arbitrate: you can mount filesystems, but only under certain conditions . The kernel developer motto for this sort of thing is mechanism, not policy : they are very keen to avoid encoding particular environments' policies (the sort of thing you could think of as business rules ) in the kernel, because that makes it non-generic and hard to maintain. As a result, direct mount/unmount actions are only allowed by privileged processes, and it's up to user-space processes to arrange for a privileged process to make the desired mount syscall. Here are some other privileged actions which laptop/desktop users can reasonably expect to just work , with or without requiring a sysadmin-like (root-equivalent) user: In environments that use a MAC framework like AppArmor, actions that would normally be allowed can become privileged: for instance, in a framework for sandboxed applications, most apps shouldn't be allowed to record audio. This prevents carrying out these actions directly, again resulting in the only way to achieve them being to ask a service to carry out the action. Ask a system service to do it On to the next design, then: I can submit a request to a privileged process, which does some checks to make sure I'm not trying to break the system (or alternatively, that I have enough sysadmin rights that I'm allowed to break the system if I want to), and then does the privileged action for me. You might think I'm about to start discussing D-Bus and daemons, but actually, a prominent earlier implementation of this was mount(8), which is normally setuid root: 
% ls -l /bin/mount
-rwsr-xr-x 1 root root 40000 May 22 11:37 /bin/mount
If you look at it from an odd angle, this is inter-process communication across a privilege boundary: I run the setuid executable, creating a process. Because the executable has the setuid bit set, the kernel makes the process highly privileged: its effective uid is root, and it has all the necessary capabilities to mount filesystems. I submit the request by passing it in the command-line arguments. mount does some checks - specifically, it looks in /etc/fstab to see whether the filesystem I'm trying to mount has the user or users flag - then carries out the mount system call. There are a few obvious problems with this: Ask a system service to do it, via D-Bus or other IPC To avoid the issues of setuid, we could use inter-process communication in the traditional sense: run a privileged daemon (on boot or on-demand), make it listen for requests, and use the IPC channel as our privilege boundary. udisks2 is one such privileged daemon, which uses D-Bus as its IPC channel. D-Bus is a commonly-used inter-process system; one of its intended/designed uses is to let user processes and system services communicate, especially this sort of communication between a privileged daemon and its less-privileged clients. People sometimes criticize D-Bus as not doing anything you couldn't do yourself with some AF_UNIX sockets. Well, no, of course it doesn't - the important bit of the reference implementation and the various interoperable reimplementations consists of a daemon and some AF_UNIX sockets, and the rest is a simple matter of programming. However, it's sufficient for most uses in its problem space, and is usually better than inventing your own. The advantage of D-Bus over doing your own thing is precisely that you are not doing your own thing: good IPC design is hard, and D-Bus makes some structural decisions so that fewer application authors have to think about them. For instance, it has a central hub daemon (the dbus-daemon, or message bus ) so that n communicating applications don't need O(n ) sockets; it uses the dbus-daemon to provide a total message ordering so you don't have to think about message reordering; it has a distributed naming model (which can also be used as a distributed mutex) so you don't have to design that; it has a serialization format and a type system so you don't have to design one of those; it has a framework for activating" run-on-demand daemons so they don't have to use resources initially, implemented using a setuid helper and/or systemd; and so on. If you have religious objections to D-Bus, you can mentally replace D-Bus with AF_UNIX or something and most of this article will still be true. Is this OK? In either case - exec'ing a privileged helper, or submitting a request to a privileged daemon via IPC - the privileged process has two questions that it needs to answer before it does its work: It needs to make some sort of decision on the latter based on the information available to it. However, before we even get there, there is another layer: In the setuid model, there is a simple security check that you can apply: you can make /bin/mount only executable by a particular group, or only executable by certain AppArmor profiles, or similar. That works up to a point, but cannot distinguish between physically-present and not-physically-present users, or other facts that might be interesting to your local security policy. Similarly, in the IPC model, you can make certain communication channels impossible, for instance by using dbus-daemon's ability to decide which messages to deliver, or AF_UNIX sockets' filesystem permissions, or a MAC framework like AppArmor. Both of these are quite coarse-grained checks which don't really understand the finer details of what is going on. If the answer to "is this safe? is something of the form maybe, it depends on... , then they can't do the right thing: they must either let it through and let the domain-specific privileged process do the check, or deny it and lose potentially useful functionality. For instance, in an AppArmor environment, some applications have absolutely no legitimate reason to talk to udisks2, so the AppArmor policy can just block it altogether. However, once again, this is a coarse-grained check: the kernel has mechanism, not policy, and it doesn't know what the service does or why. If the application does need to be able to talk to the service at all, then finer-grained access control (obeying some, but not all, requests) has to be the service's job. dbus-daemon does have the ability to match messages in a relatively fine-grained way, based on the object path, interface and member in the message, as well as the routing information that it uses itself (i.e. the source and destination). However, it is not clear that this makes a great deal of sense conceptually: these are facts about the mechanics of the IPC, not facts about the domain-specific request (because the mechanics of the IPC are all that dbus-daemon understands). For instance, taking the udisks2 example again, dbus-daemon can't distinguish between an attempt to adjust mount options for a USB stick (probably fine) and an attempt to adjust mount options for /usr (not good). To have a domain-specific security policy, we need a domain-specific component, for instance udisks2, to get involved. Unlike dbus-daemon, udisks2 knows that not all disks are equal, knows which categories make sense to distinguish, and can identify which categories a particular disk is in. So udisks2 can make a more informed decision. So, a naive approach might be to write a function in udisks2 that looks something like this pseudocode: 
may_i_mount_this_disk (user, disk, mount options)   boolean
 
    if (user is root   user is root-equivalent)
        return true;
    if (disk is not removable)
       return false;
    if (mount options are scary)
       return false;
    if (user is in  manipulate non-local disks  group)
        return true;
    if (user is not logged-in locally)
       return false;
    # https://en.wikipedia.org/wiki/Multiseat_configuration
    if (user is not logged-in on the same seat where the disk is
            plugged in)
       return false;
    return true;
Delegating the security policy to something central The pseudocode security policy outlined above is reasonably complicated already, and doesn't necessarily cover everything that you might want to consider. Meanwhile, not every system is the same. A general-purpose Linux distribution like Debian might run on server/mainframe systems with only remote users, personal laptops/desktops with one root-equivalent user, locked-down corporate laptops/desktops, mobile devices and so on; these systems should not necessarily all have the same security policy. Another interesting factor is that for some privileged operations, you might want to carry out interactive authorization: ask the requesting user to confirm that the action (which might have come from a background process) should take place (like Windows' UAC), or to prove that the person currently at the keyboard is the same as the person who logged in by giving their password (like sudo). We could in principle write code for all of this in udisks2, and in NetworkManager, and in systemd, ... - but that clearly doesn't scale, particularly if you want the security policy to be configurable. Enter polkit (formerly PolicyKit), a system service for applying security policies to actions. The way polkit works is that the application does its domain-specific analysis of the request - in the case of udisks2, whether the device to be mounted is removable, whether the mount options are reasonable, etc. - and converts it into an action. The action gives polkit a way to distinguish between things that are conceptually different, without needing to know the specifics. For instance, udisks2 currently divides up filesystem-mounting into org.freedesktop.udisks2.filesystem-mount, org.freedesktop.udisks2.filesystem-mount-fstab, org.freedesktop.udisks2.filesystem-mount-system and org.freedesktop.udisks2.filesystem-mount-other-seat. The application also finds the identity of the user making the request. Next, the application sends the action, the identity of the requesting user, and any other interesting facts to polkit. As currently implemented, polkit is a D-Bus service, so this is an IPC request via D-Bus. polkit consults its database of policies in order to choose one of several results: So how does polkit decide this? The first thing is that it reads the machine-readable description of the actions, in /usr/share/polkit-1/actions, which specifies a default policy. Next, it evaluates a local security policy to see what that says. In the current version of polkit, the local security policy is configured by writing JavaScript in /etc/polkit-1/rules.d (local policy) and /usr/share/polkit-1/rules.d (OS-vendor defaults). In older versions such as the one currently shipped in Debian unstable, there was a plugin architecture; but in practice nobody wrote plugins for it, and instead everyone used the example local authority shipped with polkit, which was configured via files in /etc/polkit-1/localauthority and /etc/polkit-1/localauthority.d. These policies can take into account useful facts like: For instance, gnome-control-center on Debian installs this snippet: 
polkit.addRule(function(action, subject)  
        if ((action.id == "org.freedesktop.locale1.set-locale"  
             action.id == "org.freedesktop.locale1.set-keyboard"  
             action.id == "org.freedesktop.hostname1.set-static-hostname"  
             action.id == "org.freedesktop.hostname1.set-hostname"  
             action.id == "org.gnome.controlcenter.datetime.configure") &&
            subject.local &&
            subject.active &&
            subject.isInGroup ("sudo"))  
                    return polkit.Result.YES;
             
 );
which is reasonably close to being pseudocode for active local users in the sudo group may set the system locale, keyboard layout, hostname and time, without needing to authenticate . A system administrator could of course override that by dropping a higher-priority policy for some or all of these actions into /etc/polkit-1/rules.d. Summary

Patrick Schoenfeld: Testing puppet modules: an overview

When it comes to testing puppet modules, there are lot of options, but for someone entering the world of puppet module testing, the pure variety may seem overwhelming. This is a try to provide some overview. So you ve written a puppet module and would like to add some tests. Now what?As of today, puppet tests basically can be done in two ways, complementing each other: Catalog tests
In most cases you should at least write some catalog tests.
As of writing this (June 2015) the tool of choice is rspec-puppet. There used to be at least one other and you might have heard about it, but it s deprecated. For an introduction to this tool, you are best served by reading its brief but sufficient docs. Function acceptance tests
If catalog testing is not enough for you (e.g. you want to test that your website profile is actually installing and serving your site on port 80 and port 443) the next logical step is to write beaker tests for tests in a real system (as real as a virtual machine can be). This is also what you need, if you are writing custom types. Today s tool of choice for this job is beaker with beaker-rspec. After you ve written some rspec tests, this might feel similar. Since the documentation, might not seem very newbie friendly at first glance, the page Howto Beaker lists the relevant pages in the documentation to get started in a sensible order. Basically it s: Update your modules build depencies (Gemfile), decide for a hypervisor, create (or describe) your test environment, write spec tests and execute them :)
Skeleton of a module If testing puppet modules falls into your lap and you ve already written your puppet code, it s to late too start with a module anatomy as generated by puppet module generate But: It s certainly a good bet to know which technologies are todays common best practice. Further reading A very good guide to setting things up and writing tests of both types is the threepart blog post by Micka l Can vet written for camptocamp. It is a basic guide into test-driven development (writing tests before writing actual code) on a practical example.

4 June 2015

Daniel Kahn Gillmor: Challenge: one reproducible package a week

I encourage anyone interested in debian development to get involved with the Reproducible Builds project. My own project is to try to diagnose (and hopefully provide patches for) two unreproducible packages a week. Maybe you can do one package a week? Reproducible Builds is another example of the kind of audacious project that I celebrated in my last blog post. It's a fun way to learn a little bit about different parts of the archive, and to help on an incrementally-improving process. My workflow below is meant to encourage folks to join in. The documentation on the wiki is certainly more authoritative (and will be more up-to-date in the future). My usual workflow is this: The information at the R-B wiki page is quite detailed if you want more info. Finally, many many thanks to all of the people involved in the project, and particularly to h01ger and Lunar^, who have both contributed a ton of work, and have also made it easy to plug in and help as a less-involved contributor. The nice automatically-updated statistics provided by the team's continuous integration work makes it a fun game to help out. Tags: challenge, reproducible builds

2 June 2015

Julien Danjou: Get back up and try again: retrying in Python

I don't often write about tools I use when for my daily software development tasks. I recently realized that I really should start to share more often my workflows and weapons of choice. One thing that I have a hard time enduring while doing Python code reviews, is people writing utility code that is not directly tied to the core of their business. This looks to me as wasted time maintaining code that should be reused from elsewhere. So today I'd like to start with retrying, a Python package that you can use to retry anything. It's OK to fail
Often in computing, you have to deal with external resources. That means accessing resources you don't control. Resources that can fail, become flapping, unreachable or unavailable. Most applications don't deal with that at all, and explode in flight, leaving a skeptical user in front of the computer. A lot of software engineers refuse to deal with failure, and don't bother handling this kind of scenario in their code. In the best case, applications usually handle simply the case where the external reached system is out of order. They log something, and inform the user that it should try again later. In this cloud computing area, we tend to design software components with service-oriented architecture in mind. That means having a lot of different services talking to each others over the network. And we all know that networks tend to fail, and distributed systems too. Writing software with failing being part of normal operation is a terrific idea. Retrying
In order to help applications with the handling of these potential failures, you need a plan. Leaving to the user the burden to "try again later" is rarely a good choice. Therefore, most of the time you want your application to retry. Retrying an action is a full strategy on its own, with a lot of options. You can retry only on certain condition, and with the number of tries based on time (e.g. every second), based on a number of tentative (e.g. retry 3 times and abort), based on the problem encountered, or even on all of those. For all of that, I use the retrying library that you can retrieve easily on PyPI. retrying provides a decorator called retry that you can use on top of any function or method in Python to make it retry in case of failure. By default, retry calls your function endlessly until it returns rather than raising an error. 
import random
from retrying import retry
 
@retry
def pick_one():
    if random.randint(0, 10) != 1:
        raise Exception("1 was not picked")

This will execute the function pick_one until 1 is returned by random.randint. retry accepts a few arguments, such as the minimum and maximum delays to use, which also can be randomized. Randomizing delay is a good strategy to avoid detectable pattern or congestion. But more over, it supports exponential delay, which can be used to implement exponential backoff, a good solution for retrying tasks while really avoiding congestion. It's especially handy for background tasks. 
@retry(wait_exponential_multiplier=1000, wait_exponential_max=10000)
def wait_exponential_1000():
    print "Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwards"
    raise Exception("Retry!")

You can mix that with a maximum delay, which can give you a good strategy to retry for a while, and then fail anyway: 
# Stop retrying after 30 seconds anyway
>>> @retry(wait_exponential_multiplier=1000, wait_exponential_max=10000, stop_max_delay=30000)
... def wait_exponential_1000():
...     print "Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwards"
...     raise Exception("Retry!")
...
>>> wait_exponential_1000()
Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwards
Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwards
Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwards
Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwards
Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwards
Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwards
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/usr/local/lib/python2.7/site-packages/retrying.py", line 212, in call
    raise attempt.get()
  File "/usr/local/lib/python2.7/site-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/usr/local/lib/python2.7/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "<stdin>", line 4, in wait_exponential_1000
  Exception: Retry!

A pattern I use very often, is the ability to retry only based on some exception type. You can specify a function to filter out exception you want to ignore or the one you want to use to retry. 
def retry_on_ioerror(exc):
    return isinstance(exc, IOError)
 
@retry(retry_on_exception=retry_on_ioerror)
def read_file():
    with open("myfile", "r") as f:
        return f.read()

retry will call the function passed as retry_on_exception with the exception raised as first argument. It's up to the function to then return a boolean indicating if a retry should be performed or not. In the example above, this will only retry to read the file if an IOError occurs; if any other exception type is raised, no retry will be performed. The same pattern can be implemented using the keyword argument retry_on_result, where you can provide a function that analyses the result and retry based on it. 
def retry_if_file_empty(result):
    return len(result) <= 0
 
@retry(retry_on_result=retry_if_file_empty)
def read_file():
    with open("myfile", "r") as f:
        return f.read()

This example will read the file until it stops being empty. If the file does not exist, an IOError is raised, and the default behavior which triggers retry on all exceptions kicks-in the retry is therefore performed. That's it! retry is really a good and small library that you should leverage rather than implementing your own half-baked solution!

1 June 2015

Thorsten Alteholz: My Debian Activities in May 2015

FTP assistant This month I marked 235 packages for accept and rejected 44 of them. I know, the NEW-queue is rather large, but the numbers are showing a downward trend again. In the light of recent events I would like to cite two things. The US Copyright Office Circular 14 says about derivative work:
A typical example of a derivative work received for registration in the Copyright Office is one that is primarily a new work but incorporates some previously published material. This previously published material makes the work a derivative work under the copyright law. To be copyrightable, a derivative work must be different enough from the original to be regarded as a new work or must contain a substantial amount of new material. Making minor changes or additions of little substance to a preexisting work will not qualify the work as a new version for copyright purposes. The new material must be original and copyrightable in itself. Titles, short phrases, and format, for example, are not copyrightable.
Title 17 of the United States Code says in 103:
The copyright in a compilation or derivative work extends only to the material contributed by the author of such work, as distinguished from the preexisting material employed in the work, and does not imply any exclusive right in the preexisting material. The copyright in such work is independent of, and does not affect or enlarge the scope, duration, ownership, or subsistence of, any copyright protection in the preexisting material.
Squeeze LTS This was my eleventh month that I did some work for the Squeeze LTS initiative, started by Raphael Hertzog at Freexian. This month I got assigned a workload of only 10.25h and I spent most of it to prepare a new upload of ruby1.9.1. The other stuff I uploaded was [DLA 222-1 has been only a sponsored upload, where Markus Koschany prepared the patches. [DLA-226] needed two uploads as the first patch turned out to be incomplete. I also marked CVEs in the security tracker as for Squeeze or added notes for future processing.
The next big adventure in June will be another upload of PHP5. Other stuff This month I also uploaded feynmf to take care of the new TDS tree (#766287). Donations Again, thanks alot to all donors. I really appreciate this and hope that everybody is pleased with my commitment. Don t hesitate to make suggestions for improvements.

Russ Allbery: Review: Hild

Review: Hild, by Nicola Griffith
Series: Hild #1
Publisher: Farrar, Straus and Giroux
Copyright: 2013
ISBN: 0-374-28087-8
Format: Hardcover
Pages: 539
Hild was born in seventh-century Britain, daughter of Hereric (would-be king of Daria) and Breguswith. Born, her mother said, to be a light to the land. This much is documented by the Venerable Bede, in The Ecclesiastical History of the English Nation, along with Hild's later rise to become one of the most powerful abbesses in British history. But nearly all of Hild's early life is a cipher. Hild fills in some of that gap with fiction. Specifically, it takes Hild from a child of three, learning her father has been killed, to a young woman, an advisor of Edwin king. It's a coming of age story in part, following her maturation both physically and mentally, her training in when to speak and how, and the dangers of being close to royalty in a fractious, political, and war-torn land. It's also the story of endless maneuvering and care, initially by her mother Breguswith and then by Hild herself sometimes in opposition to her mother, sometimes in alliance as her mother attempts to make a safe place for her daughter and herself in a treacherous court of shifting dangers. But Hild is also a story about Britain. It's a novel about how it felt and how it sounded. How it was organized, primarily among the high-born but with snippets of perspective from the lower classes. And it's a story about women: about weaving, about medicine, about friendships and partnerships and alliances among women, about the politics of marriage and childbirth, but also about the places women held and made in a time when surviving official history is all about the men. Hild is a painstakingly-researched, sprawling, lush, and sensual immersion in a part of history that gets little formal attention: after the Romans, before what we think of as medieval, before England as a country. I've been waiting for a new Nicola Griffith novel for quite a while, since Always in 2007. Hild doesn't disappoint, but it's also different than Griffith's previous writing. It has less of the strong narrative drive and clarity of either her SF or Aud Torvingen stories. Instead, the goal of Hild is to immerse and transport you, to help you feel the shape of Hild's world, to understand her days and tasks, her dreams and dangers. Like all of Griffith's work, it's beautifully written. Griffith puts description at the fore, in the sharp eyes of an observant girl who loves the outdoors, and who has been taught to watch for signs of weather and tools of healing. From early on, Hild's mother sets her up as a seer and prophetess as a way of establishing her value to war leaders and kings, and while some of it is drama and cryptic words, so much of it is careful observation, networks of information-gathering, and sharp deduction about the motives and politics of surrounding kings. Hild is very good at what she does because she has a sharp, quick mind that has been carefully trained, and because she has the aid of her mother's networks and then aid in building her own. Griffith does a wonderful job showing the reader what Hild sees, how she appreciates the world both for its own beauty and for the information she can gather from it, and how to build influence by navigating tense and dangerous moments: waiting for just the right moment and the right word, and taking sudden, impulsive risks and accepting their consequences. Unfortunately for me, this setting is also rich in complex politics and numerous actors, with older and unfamiliar names, and I got lost. Constantly. That's the drawback to immersion: Griffith doesn't hold the reader's hand. We get Hild's thoughts and analysis, and the reader has to keep up. Sometimes I did; sometimes I didn't. There's a dizzying flurry of names here, both personal and place, and while there is a map and a single family tree, neither helped me as much as I wanted them to. At several points, I found myself skimming through the latest shift in the balance between various petty kings because, while I knew I'd seen all the names before, they had come adrift from their context in the story. That was my major frustration with this book. It was all interesting enough that I would have kept thumbing back to a detailed dramatis personae, and indeed I kept checking the family tree, but there just wasn't enough detail there. Even better would have been a brief factual history of the political and military conflicts Hild was living through, keyed by chapter. Hild is startlingly intelligent, leaping from insight to insight, which is wonderful for building character, but which occasionally leaves the reader scrambling to catch up with the connections between her thoughts. I felt like, had I the broader context, I could have understood her insight more readily. All this information is likely available, since Griffith is playing off of documented history, but I'm not the sort of reader who likes doing Internet research while engrossed in a book. So, that's the downside, at least for me. But this book has many strengths, even if you're lost much of the time. Hild as written by Griffith is a fascinating character, full of sharp edges and difficult moods and a powerful belief in what she feels is right. Griffith is at the height of her writing ability when describing Hild making hard choices and taking on burdens that seem too large for her to bear. There are two sections of the book, where Hild is forced by circumstances to lead men in violence, that I think are two of the best bits of writing Griffith has ever published not just because of those scenes themselves, but because of the aftermath, the lingering echos, the way that they shape and inform everything Hild does afterwards. The mingling of reward and loss, maturation and trauma, the sense that the world has shifted both inside and out and it's nearly impossible to say whether the change is for the better or worse. Griffith also knows when not to say too much, and while I found that frustrating for the politics, it does wonders for the characters. Hild's complex and fraught relationship with Gwaldus is the best example. We never know exactly what Gwaldus is thinking; Hild can only guess, and at times one is fairly sure that she guesses wrong. But that doesn't lead to sudden revelations, where the characters finally understand each other. Instead, they both adjust, they maneuver around each other, they find space and understanding where they can, and sometimes they just close off. This book is full of relationships like this: loves that are too complicated for words, bonds that are too dangerous to acknowledge, and characters who can't relax even though they wish the best for each other. At times, it's exhausting reading, but it gives Hild a tension that one wouldn't expect from a sprawling novel full of description and scene-building. Hild is clearly the first book of a series, and leaves quite a lot unresolved. If you want closure in relationships and in politics, there's a lot here that you may find frustrating. And if, like me, you struggle to keep names and politics straight, you're probably going to get lost. But it's well worth the effort for the description, for Hild's thought processes, and for a few haunting scenes that I will be replaying in my head for a very long time. Expect to take your time with this, and wait until you're in the mood for immersion and puzzling out context as you go, but recommended. I suspect it would be even better on a second reading. Rating: 8 out of 10

31 May 2015

Dirk Eddelbuettel: rfoaas 0.1.6

After a few quiet months, a new version of rfoaas is now on CRAN. As before, it shadows upstream release of FOAAS from which carried over the version number 0.1.6. The rfoaas package provides an interface for R to the most excellent FOAAS service--which provides a modern, scalable and RESTful web service for the frequent need to tell someone to f$#@ off. Release 0.1.6 of FOAAS builds on the initial support for filters and now adds working internationalization. This is best shown by example: 
R> library(rfoaas)
R> off("Tom", "Everyone")                                      # standard upstream example
[1] "Fuck off, Tom. - Everyone"
R> off("Tom", "Everyone", language="fr")                       # now in French
[1] "Va te faire foutre, Tom. - Everyone"
R> off("Tom", "Everyone", language="de", filter="shoutcloud")  # or in German and LOUD
[1] "VERPISS DICH, TOM. - EVERYONE"
R>
We start with a standard call to off(), add the (now fully functional upstream) language support and finally illustrate the shoutcloud.io filter added in the preceding 0.1.3 release. As usual, CRANberries provides a diff to the previous CRAN release. Questions, comments etc should go to the GitHub issue tracker.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Dominique Dumont: Improving update of existing debian/copyright file

Hello One of my last blog introduced the new cme update dpkg-copyright command and explained how to use this command to create a debian/copyright file from scratch. We will see in this blog how cme update dpkg-copyright command can also be used if a debian/copyright file already exists. During the lifetime of a package, creating a new debian/copyright copyright file should only happen once. When updating a package to a new upstream version, the content of the copyright file should be modified to reflect upstream changes by: cme update dpkg-copyright can be run with an existing debian/copyright file: the information extracted by licensecheck are merged in debian/copyright. Unfortunately, this merge may bring back the errors coming from licensecheck that you ve cleaned up manually last time. This last part is not optimal as the manual clean up must be repeated after cme update dpkg-copyright is run. Here s a way to avoid the last problem: cme update dpkg-copyright supports a way to modify the content of debian/copyright after the merge: the optional file debian/fix.scanned.copyright contains instructions to tweak the content of debian/copyright. (See Config::Model::Dpkg::Copyright man page for more details) You may remember that Config::Model handles all configuration information as a tree. The information contained in the copyright file is also stored in a tree structure. debian/fix.scanned.copyright contains instructions to navigate this tree and modify the content of the tree. Imagine that there s a robot (or a monkey) in that configuration tree. Each instruction tells the robot either to move in the tree or to perform an operation. The most useful instructions for debian/fix.scanned.copyright are: Otherwise: Let s see a commented example: 
# go to the root of the tree (with '!')
# and remove a bogus Files entry (with ":~")
! Files:~"foo/bogus.c"
// go to a Files entry and append (operator '.=') some info to   owner
! Files:"foo/bar.h"
  Copyright.=" Appended info."
# go back to tree root and assign   info
! Files:"foo/baz.c" Copyright="2014, FooBar"
# add a global license and a file entry missed by licensecheck
# global license must be set before using it in Files entry
# note: MIT text license is provided by cme
! License=MIT
! Files:"3rdparty/missed.c" Copyright="2005 FooBar" License short_name=MIT
Other possible instructions for this file are described in Config::Model::Loader doc. The number of instructions depends on the accuracy of licencecheck command. Hopefully, this number can be reduced by improving licensecheck. Last but not least, the following packages are required: You may want to install also libconfig-model-tkui-perl to update the copyright file with cme GUI. In summary, the command cme updated dpkg-copyright associated with instructions stored in debian/fix.scanned.copyright enable Debian packagers to update with less effort debian/copyright file when updating a package. All the best

28 May 2015

Steve Kemp: A brief examination of tahoe-lafs

Continuing the theme from the last post I made, I've recently started working my way down the list of existing object-storage implementations. tahoe-LAFS is a well-established project which looked like a good fit for my needs: Getting the system up and running, on four nodes, was very simple. Setup a single/simple "introducer" which is a well-known node that all hosts can use to find each other, and then setup four deamons for storage. When files are uploaded they are split into chunks, and these chunks are then distributed amongst the various nodes. There are some configuration settings which determine how many chunks files are split into (10 by default), how many chunks are required to rebuild the file (3 by default) and how many copies of the chunks will be created. The biggest problem I have with tahoe is that there is no rebalancing support: Setup four nodes, and the space becomes full? You can add more nodes, new uploads go to the new nodes, while old ones stay on the old. Similarly if you change your replication-counts because you're suddenly more/less paranoid this doesn't affect existing nodes. In my perfect world you'd distribute blocks around pretty optimistically, and I'd probably run more services: The storage nodes would have the primitives "List all blocks", "Get block", "Put block", and using that you could ensure that each node had sent its data to at least N other nodes. This could be done in the background. The indexer would be responsible for keeping track of which blocks live where, and which blocks are needed to reassemble upload N. There's probably more that it could do.

27 May 2015

Norbert Preining: USB stick update: TAILS 1.4, GParted 0.22, SysResCD 4.5.2, Debian Jessie

I have posted a view times (here and here) about how to get a multi-boot/multi-purpose USB stick working. Now that TAILS has seens a major upgrade, and Debian 8.0 Jessie has been released, I think it is time to update the procedure to reflect the latest releases. That turned out to be a painful experience, in particular since Debian removed support for any reasonable boot method.

(Update: See this post for fixes to get Debian working again) multi-usb-stick So going through these explanations one will end up with a usable USB stick that can boot you into TAILS, System Rescue CD, GNU Parted Live CD, but unfortunately not anymore to boot into an installation of Debian 8.0 Jessie installation. But the USB stick will still be usable as normal media. Let us repeat some things from the original post concerning the wishlist and the main players: I have a long wishlist of items a boot stick should fulfill This time I have added the GNOME/GNU Partition Editor gparted as it came in useful at times. Requirements A USB stick, the iso images of TAILS 1.4, SystemRescueCD 4.5.2, GParted Lice CD 0.22.0, and some tool to access iso images, for example ISOmaster (often available from your friendly Linux distribution). I assume that you have already an USB stick prepared as described previously. If this is not the case, please go there and follow the section on preparing your usb stick. Two types of boot options We will employ two different approaches to boot special systems: the one is directly from an iso image, the other via extraction of the necessary kernels and images. At the moment we have the following status with respect to boot methods: What is a pity is that during the testing phase, booting and installation from testing images worked for Debian as documented previously. But with the final images (and my guess it has to do with systemd, wouldn t surprise me), there is no Debian CD detected as it cannot find the iso image on the USB stick. Bummer. That means that for having a Debian/USB stick you need a dedicated one. Booting from ISO image Grub has gained quite some time ago the ability to boot directly from an ISO image. In this case the iso image is mounted via loopback, and the kernel and initrd are specified relatively to the iso image root. For both SystemRescueCD and GNOME Partition Live CD, just drop the iso files into /boot/iso/, in my case /boot/iso/systemrescuecd-x86-4.5.2.iso and /boot/iso/gparted-live-0.22.0-1-i586.iso. After that, entries like the following have to be added to grub.cfg. For the full list see grub.cfg: 
submenu "System Rescue CD 4.5.2 (via ISO) ---> "  
  set isofile="/boot/iso/systemrescuecd-x86-4.5.2.iso"
  menuentry "SystemRescueCd (64bit, default boot options)"  
        set gfxpayload=keep
        loopback loop (hd0,1)$isofile
        linux   (loop)/isolinux/rescue64 isoloop=$isofile
        initrd  (loop)/isolinux/initram.igz
   
  ...
 
 
submenu "GNU/Gnome Parted Live CD 0.22.0 (via ISO) ---> "  
  set isofile="/boot/iso/gparted-live-0.22.0-1-i586.iso"
  menuentry "GParted Live (Default settings)" 
    loopback loop (hd0,1)$isofile
    linux (loop)/live/vmlinuz boot=live username=user config components quiet noswap noeject  ip=  nosplash findiso=$isofile
    initrd (loop)/live/initrd.img
   
  ...
Note the added isoloop=$isofile and findiso=$isofile that helps the installer find the iso images. Booting via extraction of kernels and images This is a bit more tedious, but still not too bad. Installation of TAILS files Assuming you have access to the files on the TAILS CD via the directory ~/tails, execute the following commands: 
mkdir -p /usbstick/boot/tails
cp -a ~/tails/live/* /usbstick/boot/tails/
The grub.cfg entries look now similar to the following: 
submenu "TAILS Environment 1.4 ---> "  
  menuentry "Tails64 Live System"  
        linux   /boot/tails/vmlinuz2 boot=live live-media-path=/boot/tails config live-media=removable nopersistent noprompt timezone=Etc/UTC block.events_dfl_poll_msecs=1000 splash noautologin module=Tails  libata.force=noncq
        initrd  /boot/tails/initrd2.img
   
  ...
The important part here is the live-media-path=/boot/tails, otherwise TAILS will not find the correct files for booting. The rest of the information was extracted from the boot setup of TAILS itself. Current status of USB stick Just to make sure, the usb stick should contain at the current stage the following files: 
/boot/
    iso/
        gparted-live-0.22.0-1-i586.iso
        systemrescuecd-x86-4.5.2.iso
    tails/
        vmlinuz Tails.module initrd.img ....
    grub/
        fonts/
            lots of files
        locale/
            lots of files
        x86_64-efi/
            lots of files
        font.pf2
        grubenv
        grub.cfg            *this file we create in the next step!!*
/EFI
    BOOT/
        BOOTX64.EFI
The Grub config file grub.cfg The final step is to provide a grub config file in /usbstick/boot/grub/grub.cfg. I created one by looking at the isoboot.cfg files both in the SystemRescueCD, TAILS iso images, GParted iso image, and the Debian/Jessie image, and converting them to grub syntax. Excerpts have been shown above in the various sections. I spare you all the details, grab a copy here: grub.cfg Conclusion That s it. Now you can anonymously provide data about your evil government, rescue your friends computer, fix a forgotten Windows password, and above all, install a proper free operating system. If you have any comments, improvements or suggestions, please drop me a comment. I hope this helps a few people getting a decent USB boot stick running. Enjoy. Postscriptum concerning Debian/Jessie Working method is now described in this post. So I have first tried to boot directly into the Debian Jessie firmware ISO image by dropping the ISO into /boot/iso/firmware-8.0.0-amd64-i386-netinst.iso and adding a grub entry like: 
submenu "Debian 8.0 Jessie NetInstall ---> "  
    set isofile="/boot/iso/firmware-8.0.0-amd64-i386-netinst.iso"
 
    menuentry '64 bit Install'  
        set background_color=black
        loopback loop (hd0,1)$isofile
        linux    (loop)/install.amd/vmlinuz iso-scan/ask_second_pass=true iso-scan/filename=$isofile vga=788 -- quiet 
        initrd   (loop)/install.amd/initrd.gz
     
    ...
That was a no go. I have added the iso-scan/ask_second_pass=true iso-scan/filename=$isofile thingy after some research in forum and web, without any change. Of course I have also tried the official ISO image debian-8.0.0-amd64-netinst.iso, with the same effect. Although I was sure it doesn t make any difference, I have also tried to extract the kernel and initrd, and boot directly from it, i.e., copying the files to /usbstick/boot/debian/ as follows: 
mkdir -p /usbstick/boot/debian
cp -a ~/tails/install.amd /usbstick/boot/debian/
cp -a ~/tails/install.386 /usbstick/boot/debian/
In addition, copy the iso image itself into /usbstick/boot/iso (or directly into the root of the usb stick, didn t change anything), and added a grub.cfg entry as follows: 
submenu "Debian 8.0 Jessie NetInstall ---> "  
  menuentry '64 bit Install'  
    set background_color=black
    linux    /boot/debian/install.amd/vmlinuz vga=788 -- quiet 
    initrd   /boot/debian/install.amd/initrd.gz
   
  ...
All these were without success, always ending up in error messages like: Mounting sdb, this is not a Debian Installation CD, etc etc. I have also submitted a bug report to the installation reports, unfortunately no answer (not that I expected one). In case someone has a better idea, please let me know! It is very sad that these kind of support is removed. The Installation Manual lists some options to install from USB, and the only difference is that there syslinux on a FAT16 system is used. This restricts the USB stick size a lot, and does not allow for easy multi boot via grub. As reported here, I had this actually running. Unfortunately, I stupidly removed the old iso image and replaced it with the current Jessie ISO image, assuming that there will be no regression. Wrong assumption. Now I cannot even investigate the changes by looking into the initrd. Looking at the date of the original post I see that it is more or less one year ago, so before the Systemd introduction. I don t know whether this has any effect, I doubt, as the installer is separate. But something happened in the mean time. Bad for us.

Vincent Bernat: Live patching QEMU for VENOM mitigation

CVE-2015-3456, also known as VENOM, is a security vulnerability in QEMU virtual floppy controller:
The Floppy Disk Controller (FDC) in QEMU, as used in Xen [ ] and KVM, allows local guest users to cause a denial of service (out-of-bounds write and guest crash) or possibly execute arbitrary code via the FD_CMD_READ_ID, FD_CMD_DRIVE_SPECIFICATION_COMMAND, or other unspecified commands.
Even when QEMU has been configured with no floppy drive, the floppy controller code is still active. The vulnerability is easy to test1: 
#define FDC_IOPORT 0x3f5
#define FD_CMD_READ_ID 0x0a
int main()  
    ioperm(FDC_IOPORT, 1, 1);
    outb(FD_CMD_READ_ID, FDC_IOPORT);
    for (size_t i = 0;; i++)
        outb(0x42, FDC_IOPORT);
    return 0;
Once the fix installed, all processes still have to be restarted for the upgrade to be effective. It is possible to minimize the downtime by leveraging virsh save. Another possibility would be to patch the running processes. The Linux kernel attracted a lot of interest in this area, with solutions like Ksplice (mostly killed by Oracle), kGraft (by Red Hat) and kpatch (by Suse) and the inclusion of a common framework in the kernel. The userspace has far less out-of-the-box solutions2. I present here a simple and self-contained way to patch a running QEMU to remove the vulnerability without requiring any sensible downtime. Here is a short demonstration:

Proof of concept First, let s find a workaround that would be simple to implement through live patching: while modifying running code text is possible, it is easier to modify a single variable.

Concept Looking at the code of the floppy controller and the patch, we can avoid the vulnerability by not accepting any command on the FIFO port. Each request would be answered by Invalid command (0x80) and a user won t be able to push more bytes to the FIFO until the answer is read and the FIFO queue reset. Of course, the floppy controller would be rendered useless in this state. But who cares? The list of commands accepted by the controller on the FIFO port is contained in the handlers[] array: 
static const struct  
    uint8_t value;
    uint8_t mask;
    const char* name;
    int parameters;
    void (*handler)(FDCtrl *fdctrl, int direction);
    int direction;
  handlers[] =  
      FD_CMD_READ, 0x1f, "READ", 8, fdctrl_start_transfer, FD_DIR_READ  ,
      FD_CMD_WRITE, 0x3f, "WRITE", 8, fdctrl_start_transfer, FD_DIR_WRITE  ,
    /* [...] */
      0, 0, "unknown", 0, fdctrl_unimplemented  , /* default handler */
 ;
To avoid browsing the array each time a command is received, another array is used to map each command to the appropriate handler: 
/* Associate command to an index in the 'handlers' array */
static uint8_t command_to_handler[256];
static void fdctrl_realize_common(FDCtrl *fdctrl, Error **errp)
 
    int i, j;
    static int command_tables_inited = 0;
    /* Fill 'command_to_handler' lookup table */
    if (!command_tables_inited)  
        command_tables_inited = 1;
        for (i = ARRAY_SIZE(handlers) - 1; i >= 0; i--)  
            for (j = 0; j < sizeof(command_to_handler); j++)  
                if ((j & handlers[i].mask) == handlers[i].value)  
                    command_to_handler[j] = i;
                 
             
         
     
    /* [...] */
Our workaround is to modify the command_to_handler[] array to map all commands to the fdctrl_unimplemented() handler (the last one in the handlers[] array).

Testing with gdb To check if the workaround works as expected, we test it with gdb. Unless you have compiled QEMU yourself, you need to install a package with debug symbols. Unfortunately, on Debian, they are not available, yet3. On Ubuntu, you can install the qemu-system-x86-dbgsym package after enabling the appropriate repositories. The following function for gdb maps every command to the unimplemented handler: 
define patch
  set $handler = sizeof(handlers)/sizeof(*handlers)-1
  set $i = 0
  while ($i < 256)
   set variable command_to_handler[$i++] = $handler
  end
  printf "Done!\n"
end
Attach to the vulnerable process (with attach), call the function (with patch) and detach of the process (with detach). You can check that the exploit is not working anymore. This could be easily automated.

Limitations Using gdb has two main limitations:
  1. It needs to be installed on each host to be patched.
  2. The debug packages need to be installed as well. Moreover, it can be difficult to fetch previous versions of those packages.

Writing a custom patcher To overcome those limitations, we can write a customer patcher using the ptrace() system call without relying on debug symbols being present.

Finding the right memory spot Before being able to modify the command_to_handler[] array, we need to know its location. The first clue is given by the symbol table. To query it, use readelf -s: 
$ readelf -s /usr/lib/debug/.build-id/09/95121eb46e2a4c13747ac2bad982829365c694.debug   \
>   sed -n -e 1,3p -e /command_to_handler/p
Symbol table '.symtab' contains 27066 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
  8485: 00000000009f9d00   256 OBJECT  LOCAL  DEFAULT   26 command_to_handler
This table is usually stripped out of the executable to save space, like shown below: 
$ file -b /usr/bin/qemu-system-x86_64   tr , \\n
ELF 64-bit LSB shared object
 x86-64
 version 1 (SYSV)
 dynamically linked
 interpreter /lib64/ld-linux-x86-64.so.2
 for GNU/Linux 2.6.32
 BuildID[sha1]=0995121eb46e2a4c13747ac2bad982829365c694
 stripped
If your distribution provides a debug package, the debug symbols are installed in /usr/lib/debug. Most modern distributions are now relying on the build ID4 to map an executable to its debugging symbols, like the example above. Without a debug package, you need to recompile the existing package without stripping debug symbols in a clean environment5. On Debian, this can be done by setting the DEB_BUILD_OPTIONS environment variable to nostrip. We have now two possible cases:
  • the easy one, and
  • the hard one.

The easy case On x86, here is the standard layout of a regular Linux process in memory6: Memory layout of a regular process on x86 The random gaps (ASLR) are here to prevent an attacker from reliably jumping to a particular exploited function in memory. On x86-64, the layout is quite similar. The important point is that the base address of the executable is fixed. The memory mapping of a process is also available through /proc/PID/maps. Here is a shortened and annotated example on x86-64: 
$ cat /proc/3609/maps
00400000-00401000         r-xp 00000000 fd:04 483  not-qemu [text segment]
00601000-00602000         r--p 00001000 fd:04 483  not-qemu [data segment]
00602000-00603000         rw-p 00002000 fd:04 483  not-qemu [BSS segment]
[random gap]
02419000-0293d000         rw-p 00000000 00:00 0    [heap]
[random gap]
7f0835543000-7f08356e2000 r-xp 00000000 fd:01 9319 /lib/x86_64-linux-gnu/libc-2.19.so
7f08356e2000-7f08358e2000 ---p 0019f000 fd:01 9319 /lib/x86_64-linux-gnu/libc-2.19.so
7f08358e2000-7f08358e6000 r--p 0019f000 fd:01 9319 /lib/x86_64-linux-gnu/libc-2.19.so
7f08358e6000-7f08358e8000 rw-p 001a3000 fd:01 9319 /lib/x86_64-linux-gnu/libc-2.19.so
7f08358e8000-7f08358ec000 rw-p 00000000 00:00 0
7f08358ec000-7f083590c000 r-xp 00000000 fd:01 5138 /lib/x86_64-linux-gnu/ld-2.19.so
7f0835aca000-7f0835acd000 rw-p 00000000 00:00 0
7f0835b08000-7f0835b0c000 rw-p 00000000 00:00 0
7f0835b0c000-7f0835b0d000 r--p 00020000 fd:01 5138 /lib/x86_64-linux-gnu/ld-2.19.so
7f0835b0d000-7f0835b0e000 rw-p 00021000 fd:01 5138 /lib/x86_64-linux-gnu/ld-2.19.so
7f0835b0e000-7f0835b0f000 rw-p 00000000 00:00 0
[random gap]
7ffdb0f85000-7ffdb0fa6000 rw-p 00000000 00:00 0    [stack]
With a regular executable, the value given in the symbol table is an absolute memory address: 
$ readelf -s not-qemu   \
>   sed -n -e 1,3p -e /command_to_handler/p
Symbol table '.dynsym' contains 9 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
    47: 0000000000602080   256 OBJECT  LOCAL  DEFAULT   25 command_to_handler
So, the address of command_to_handler[], in the above example, is just 0x602080.

The hard case To enhance security, it is possible to load some executables at a random base address, just like a library. Such an executable is called a Position Independent Executable (PIE). An attacker won t be able to rely on a fixed address to find some helpful function. Here is the new memory layout: Memory layout of a PIE process on x86 With a PIE process, the value in the symbol table is now an offset from the base address. 
$ readelf -s not-qemu-pie   sed -n -e 1,3p -e /command_to_handler/p
Symbol table '.dynsym' contains 17 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
    47: 0000000000202080   256 OBJECT  LOCAL  DEFAULT   25 command_to_handler
If we look at /proc/PID/maps, we can figure out where the array is located in memory: 
$ cat /proc/12593/maps
7f6c13565000-7f6c13704000 r-xp 00000000 fd:01 9319  /lib/x86_64-linux-gnu/libc-2.19.so
7f6c13704000-7f6c13904000 ---p 0019f000 fd:01 9319  /lib/x86_64-linux-gnu/libc-2.19.so
7f6c13904000-7f6c13908000 r--p 0019f000 fd:01 9319  /lib/x86_64-linux-gnu/libc-2.19.so
7f6c13908000-7f6c1390a000 rw-p 001a3000 fd:01 9319  /lib/x86_64-linux-gnu/libc-2.19.so
7f6c1390a000-7f6c1390e000 rw-p 00000000 00:00 0
7f6c1390e000-7f6c1392e000 r-xp 00000000 fd:01 5138  /lib/x86_64-linux-gnu/ld-2.19.so
7f6c13b2e000-7f6c13b2f000 r--p 00020000 fd:01 5138  /lib/x86_64-linux-gnu/ld-2.19.so
7f6c13b2f000-7f6c13b30000 rw-p 00021000 fd:01 5138  /lib/x86_64-linux-gnu/ld-2.19.so
7f6c13b30000-7f6c13b31000 rw-p 00000000 00:00 0
7f6c13b31000-7f6c13b33000 r-xp 00000000 fd:04 4594  not-qemu-pie [text segment]
7f6c13cf0000-7f6c13cf3000 rw-p 00000000 00:00 0
7f6c13d2e000-7f6c13d32000 rw-p 00000000 00:00 0
7f6c13d32000-7f6c13d33000 r--p 00001000 fd:04 4594  not-qemu-pie [data segment]
7f6c13d33000-7f6c13d34000 rw-p 00002000 fd:04 4594  not-qemu-pie [BSS segment]
[random gap]
7f6c15c46000-7f6c15c67000 rw-p 00000000 00:00 0     [heap]
[random gap]
7ffe823b0000-7ffe823d1000 rw-p 00000000 00:00 0     [stack]
The base address is 0x7f6c13b31000, the offset is 0x202080 and therefore, the location of the array is 0x7f6c13d33080. We can check with gdb: 
$ print &command_to_handler
$1 = (uint8_t (*)[256]) 0x7f6c13d33080 <command_to_handler>

Patching a memory spot Once we know the location of the command_to_handler[] array in memory, patching it is quite straightforward. First, we start tracing the target process: 
/* Attach to the running process */
static int
patch_attach(pid_t pid)
 
    int status;
    printf("[.] Attaching to PID %d...\n", pid);
    if (ptrace(PTRACE_ATTACH, pid, NULL, NULL) == -1)  
        fprintf(stderr, "[!] Unable to attach to PID %d: %m\n", pid);
        return -1;
     
    if (waitpid(pid, &status, 0) == -1)  
        fprintf(stderr, "[!] Error while attaching to PID %d: %m\n", pid);
        return -1;
     
    assert(WIFSTOPPED(status)); /* Tracee may have died */
    if (ptrace(PTRACE_GETSIGINFO, pid, NULL, &si) == -1)  
        fprintf(stderr, "[!] Unable to read siginfo for PID %d: %m\n", pid);
        return -1;
     
    assert(si.si_signo == SIGSTOP); /* Other signals may have been received */
    printf("[*] Successfully attached to PID %d\n", pid);
    return 0;
Then, we retrieve the command_to_handler[] array, modify it and put it back in memory7. 
static int
patch_doit(pid_t pid, unsigned char *target)
 
    int ret = -1;
    unsigned char *command_to_handler = NULL;
    size_t i;
    /* Get the table */
    printf("[.] Retrieving command_to_handler table...\n");
    command_to_handler = ptrace_read(pid,
                                     target,
                                     QEMU_COMMAND_TO_HANDLER_SIZE);
    if (command_to_handler == NULL)  
        fprintf(stderr, "[!] Unable to read command_to_handler table: %m\n");
        goto out;
     
    /* Check if the table has already been patched. */
    /* [...] */
    /* Patch it */
    printf("[.] Patching QEMU...\n");
    for (i = 0; i < QEMU_COMMAND_TO_HANDLER_SIZE; i++)  
        command_to_handler[i] = QEMU_NOT_IMPLEMENTED_HANDLER;
     
    if (ptrace_write(pid, target, command_to_handler,
           QEMU_COMMAND_TO_HANDLER_SIZE) == -1)  
        fprintf(stderr, "[!] Unable to patch command_to_handler table: %m\n");
        goto out;
     
    printf("[*] QEMU successfully patched!\n");
    ret = 0;
out:
    free(command_to_handler);
    return ret;
Since ptrace() only allows to read or write a word at a time, ptrace_read() and ptrace_write() are wrappers to read or write arbitrary large chunks of memory8. Here is the code for ptrace_read(): 
/* Read memory of the given process */
static void *
ptrace_read(pid_t pid, void *address, size_t size)
 
    /* Allocate the buffer */
    uword_t *buffer = malloc((size/sizeof(uword_t) + 1)*sizeof(uword_t));
    if (!buffer) return NULL;
    /* Read word by word */
    size_t readsz = 0;
    do  
        errno = 0;
        if ((buffer[readsz/sizeof(uword_t)] =
                ptrace(PTRACE_PEEKTEXT, pid,
                       (unsigned char*)address + readsz,
                       0)) && errno)  
            fprintf(stderr, "[!] Unable to peek one word at address %p: %m\n",
                    (unsigned char *)address + readsz);
            free(buffer);
            return NULL;
         
        readsz += sizeof(uword_t);
      while (readsz < size);
    return (unsigned char *)buffer;

Putting the pieces together The patcher is provided with the following information:
  • the PID of the process to be patched,
  • the command_to_handler[] offset from the symbol table, and
  • the build ID of the executable file used to get this offset (as a safety measure).
The main steps are:
  1. Attach to the process with ptrace().
  2. Get the executable name from /proc/PID/exe.
  3. Parse /proc/PID/maps to find the address of the text segment (it s the first one).
  4. Do some sanity checks:
    • check there is a ELF header at this location (4-byte magic number),
    • check the executable type (ET_EXEC for regular executables, ET_DYN for PIE), and
    • get the build ID and compare with the expected one.
  5. From the base address and the provided offset, compute the location of the command_to_handler[] array.
  6. Patch it.
You can find the complete patcher on GitHub. 
$ ./patch --build-id 0995121eb46e2a4c13747ac2bad982829365c694 \
>         --offset 9f9d00 \
>         --pid 16833
[.] Attaching to PID 16833...
[*] Successfully attached to PID 16833
[*] Executable name is /usr/bin/qemu-system-x86_64
[*] Base address is 0x7f7eea912000
[*] Both build IDs match
[.] Retrieving command_to_handler table...
[.] Patching QEMU...
[*] QEMU successfully patched!
  1. The complete code for this test is on GitHub.
  2. An interesting project seems to be Katana. But there are also some insightful hacking papers on the subject.
  3. Some packages come with a -dbg package with debug symbols, some others don t. Fortunately, a proposal to automatically produce debugging symbols for everything is near completion.
  4. The Fedora Wiki contains the rationale behind the build ID.
  5. If the build is incorrectly reproduced, the build ID won t match. The information provided by the debug symbols may or may not be correct. Debian currently has a reproducible builds effort to ensure that each package can be reproduced.
  6. Anatomy of a program in memory is a great blog post explaining in more details how a program lives in memory.
  7. Being an uninitialized static variable, the variable is in the BSS section. This section is mapped to a writable memory segment. If it wasn t the case, with Linux, the ptrace() system call is still allowed to write. Linux will copy the page and mark it as private.
  8. With Linux 3.2 or later, process_vm_readv() and process_vm_writev() can be used to transfer data from/to a remote process without using ptrace() at all. However, ptrace() would still be needed to reliably stop the main thread.

26 May 2015

Julien Danjou: OpenStack Summit Liberty from a Ceilometer & Gnocchi point of view

Last week I was in Vancouver, BC for the OpenStack Summit, discussing the new Liberty version that will be released in 6 months. I've attended the summit mainly to discuss and follow-up new developments on Ceilometer, Gnocchi and Oslo. It has been a pretty good week and we were able to discuss and plan a few interesting things. Ops feedback
We had half a dozen Ceilometer sessions, and the first one was dedicated to getting feedbacks from operators using Ceilometer. We had a few operators present, and a few of the Ceilometer team. We had constructive discussion, and my feeling is that operators struggles with 2 things so far: scaling Ceilometer storage and having Ceilometer not killing the rest of OpenStack. We discussed the first point as being addressed by Gnocchi, and I presented a bit Gnocchi itself, as well as how and why it will fix the storage scalability issue operators encountered so far. Ceilometer putting down the OpenStack installation is more interesting problem. Ceilometer pollsters request information from Nova, Glance to gather statistics. Until Kilo, Ceilometer used to do that regularly and at fixed interval, causing high pike load in OpenStack. With the introduction of jitter in Kilo, this should be less of a problem. However, Ceilometer hits various endpoints in OpenStack that are poorly designed, and hitting those endpoints of Nova or other components triggers a lot of load on the platform. Unfortunately, this makes operators blame Ceilometer rather than blaming the components being guilty of poor designs. We'd like to push forward improving these components, but it's probably going to take a long time. Componentisation
When I started the Gnocchi project last year, I pretty soon realized that we would be able to split Ceilometer itself in different smaller components that could work independently, while being able to leverage each others. For example, Gnocchi can run standalone and store your metrics even if you don't use Ceilometer nor even OpenStack itself. My fellow developer Chris Dent had the same idea about splitting Ceilometer a few months ago and drafted a proposal. The idea is to have Ceilometer split in different parts that people could assemble together or run on their owns. Interestingly enough, we had three 40 minutes sessions planned to talk and debate about this division of Ceilometer, though we all agreed in 5 minutes that this was the good thing to do. Five more minutes later, we agreed on which part to split. The rest of the time was allocated to discuss various details of that split, and I engaged to start doing the work with Ceilometer alarming subsystem. I wrote a specification on the plane bringing me to Vancouver, that should be approved pretty soon now. I already started doing the implementation work. So fingers crossed, Ceilometer should have a new components in Liberty handling alarming on its own. This would allow users for example to only deploys Gnocchi and Ceilometer alarm. They would be able to feed data to Gnocchi using their own system, and build alarms using Ceilometer alarm subsystem relying on Gnocchi's data. Gnocchi
We didn't have a Gnocchi dedicated slot mainly because I indicated I didn't feel we needed one. We anyway discussed a few points around coffee, and I've been able to draw a few new ideas and changes I'd like to see in Gnocchi. Mainly changing the API contract to be more asynchronously so we can support InfluxDB more correctly, and improve Carbonara (the library we created to manipulate timeseries) based drivers to be faster. All of those should plus a few Oslo tasks I'd like to tackle should keep me busy for the next cycle!

21 May 2015

Yves-Alexis Perez: Followup on Debian grsec kernels for Jessie

So, following the previous post, I've indeed updated the way I'm making my grsec kernels. I wanted to upgrade my server to Jessie, and didn't want to keep the 3.2 kernel indefinitely, so I had to update to at least 3.14, and find something to make my life (and maybe some others) easier. In the end, like planned, I've switched to the make deb-pkg way, using some scripts here and there to simplify stuff. The scripts and configs can be found in my debian-grsec-config repository. The repository layout is pretty much self-explaining: The bin/ folder contains two scripts: The configs/ folder contains the various configuration bits: I'm currently building amd64 kernels for Jessie and i386 kernels will follow soon, using config-3.14 + hardening + grsec. I'm hosting them on my apt repository. You're obviously free to use them, but considering how easy it is to rebuild a kernel, you might want to use a personal configuration (instead of mine) and rebuild the kernel yourself, so you don't have to trust my binary packages. Here's a very quick howto (adapt it to your needs): 
mkdir linux-grsec && cd linux-grsec
git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
git clone git://anonscm.debian.org/users/corsac/grsec/debian-grsec-config.git
mkdir build
cd linux-stable
../debian-grsec-config/bin/get-grsec.sh stable2 # for 3.14 branch
../debian-grsec-config/bin/kconfig.py ../build/.config ../debian-grsec-config/configs/config-3.14-2-amd64 ../debian-grsec-config/configs/hardening ../debian-grsec-config/configs/grsec
make KBUILD_OUTPUT=../build -j4 oldconfig
make KBUILD_OUTPUT=../build -j4 deb-pkg
Then you can use the generated Debian binary packages. If you use the Debian config, it'll need a lot of disk space for compilation and generate a huge linux-image debug package, so you might want to unset CONFIG_DEBUG_INFO locally if you're not interested. Right now only the deb files are generated but I've submitted a patch to have a .changes file which can be then used to manipulate them more easily (for example for uploading them a local Debian repository). Note that, obviously, this is not targeted for inclusion to the official Debian archive. This is still not possible for various reasons explained here and there, and I still don't have a solution for that. I hope this (either the scripts and config or the generated binary packages) can be useful. Don't hesitate to drop me a mail if needed.

19 May 2015

Thorsten Alteholz: alpine and UTF-8 and Debian lists

This is a note for my future self: When writing an email with only charset=US-ASCII , alpine creates an email with:

Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII

and everything is fine. In case of UTF-8 characters inside the text, alpine creates something like:

Content-Type: MULTIPART/MIXED; BOUNDARY="705298698-1667814148-1432049085=:28313"

and the only available part contains:

Content-Type: TEXT/PLAIN; format=flowed; charset=UTF-8
Content-Transfer-Encoding: 8BIT
 Google tells me that the reason for this is:
Alpine uses a single part MULTIPART/MIXED to apply a protection wrapper around QUOTED-PRINTABLE and BASE64 content to prevent it from being corrupted by various mail delivery systems that append little (typically advertising) things at the end of the message.
Ok, this behavior might come from bad experiences and it seems to work most of the time. Unfortunately if one sends a signed email to a Debian list that checks whether the signature is valid (like for example debian-lts-announce), such an email will be rejected with:
Failed to understand the email or find a signature: UDFormatError:
Cannot handle multipart messages not of type multipart/signed
*sigh*

17 May 2015

Lunar: Reproducible builds: week 3 in Stretch cycle

What happened about the reproducible builds effort for this week: Toolchain fixes Tomasz Buchert submitted a patch to fix the currently overzealous package-contains-timestamped-gzip warning. Daniel Kahn Gillmor identified #588746 as a source of unreproducibility for packages using python-support. Packages fixed The following 57 packages became reproducible due to changes in their build dependencies: antlr-maven-plugin, aspectj-maven-plugin, build-helper-maven-plugin, clirr-maven-plugin, clojure-maven-plugin, cobertura-maven-plugin, coinor-ipopt, disruptor, doxia-maven-plugin, exec-maven-plugin, gcc-arm-none-eabi, greekocr4gamera, haskell-swish, jarjar-maven-plugin, javacc-maven-plugin, jetty8, latexml, libcgi-application-perl, libnet-ssleay-perl, libtest-yaml-valid-perl, libwiki-toolkit-perl, libwww-csrf-perl, mate-menu, maven-antrun-extended-plugin, maven-antrun-plugin, maven-archiver, maven-bundle-plugin, maven-clean-plugin, maven-compiler-plugin, maven-ear-plugin, maven-install-plugin, maven-invoker-plugin, maven-jar-plugin, maven-javadoc-plugin, maven-processor-plugin, maven-project-info-reports-plugin, maven-replacer-plugin, maven-resources-plugin, maven-shade-plugin, maven-site-plugin, maven-source-plugin, maven-stapler-plugin, modello-maven-plugin1.4, modello-maven-plugin, munge-maven-plugin, ocaml-bitstring, ocr4gamera, plexus-maven-plugin, properties-maven-plugin, ruby-magic, ruby-mocha, sisu-maven-plugin, syncache, vdk2, wvstreams, xml-maven-plugin, xmlbeans-maven-plugin. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Ben Hutchings also improved and merged several changes submitted by Lunar to linux. Currently untested because in contrib: reproducible.debian.net
Thanks to the reproducible-build team for running a buildd from hell. gregor herrmann
Mattia Rizzolo modified the script added last week to reschedule a package from Alioth, a reason can now be optionally specified. Holger Levsen splitted the package sets page so each set now has its own page. He also added new sets for Java packages, Haskell packages, Ruby packages, debian-installer packages, Go packages, and OCaml packages. Reiner Herrmann added locales-all to the set of packages installed in the build environment as its needed to properly identify variations due to the current locale. Holger Levsen improved the scheduling so new uploads get tested sooner. He also changed the .json output that is used by tracker.debian.org to lists FTBFS issues again but only for issues unrelated to the toolchain or our test setup. Amongst many other small fixes and additions, the graph colors should now be more friendly to red-colorblind people. The fix for pbuilder given in #677666 by Tim Landscheidt is now used. This fixed several FTBFS for OCaml packages. Work on rebuilding with different CPU has continued, a kvm-on-kvm build host has been set been set up for this purpose. debbindiff development Version 19 of debbindiff included a fix for a regression when handling info files. Version 20 fixes a bug when diffing files with many differences toward a last line with no newlines. It also now uses the proper encoding when writing the text output to a pipe, and detects info files better. Documentation update Thanks to Santiago Vila, the unneeded -depth option used with find when fixing mtimes has been removed from the examples. Package reviews 113 obsolete reviews have been removed this week while 77 has been added.

16 May 2015

Craig Small: Debian, WordPress and Multi-site

For quite some time, the Debian version of WordPress has had a configuration tweak that made it possible to run multiple websites on the same server. This came from a while ago when multi-site wasn t available. While a useful feature, it does make the initial setup of WordPress for simple sites more complicated. I m looking at changing the Debian package slightly so that for a single-site use it Just Works. I have also looked into the way WordPress handles the content, especially themes and plugins, to see if there is a way of updating them through the website itself. This probably won t suit everyone but I think its a better default. The idea will be to setup Debian packages something like this by default and then if you want more fancier stuff its all still there, just not setup. It s not setup at the moment but the default is a little confusing which I hope to change. Multisite The first step was to get my pair of websites into one. So first it was backing up time and then the removal of my config-websitename.php files in /etc/wordpress. I created a single /etc/wordpress/config-default.php file that used a new database. This initial setup worked ok and I had the primary site going reasonably quickly. The second site was a little trickier. The problem is that multisite does things like foo.example.com and bar.example.com while I wanted example.com and somethingelse.com There is a plugin wordpress-mu-domain-mapping that almost sorta-kinda works. While it let me make the second site with a different name, it didn t like aliases, especially if the alias was the first site. Some evil SQL fixed that nicely. UPDATE wp_domain_mapping SET blog_id=1 WHERE id=2 So now I had: Files and Permissions We really three separate sets of files in wordpress. These files come from three different sources and are updated using three different ways with a different release cycle. The first is the wordpress code which is shipped in the Debian package. All of this code lives in /usr/share/wordpress and is only changed if you update the Debian package, or you fiddle around with it. It needs to be readable to the webserver but not writable. The config files in /etc/wordpress are in this lot too. Secondly, we have the user generated data. This is things like your pictures that you add to the blog. As they are uploaded through the webserver, it needs to be writable to it. These files are located in /var/lib/wordpress/wp-content/uploads Third, is the plugins and themes. These can either be unzipped and placed into a directory or directly loaded in from the webserver. I used to do the first way but are trying the second. These files are located in /var/lib/wordpress/wp-content Ever tried to update your plugins and get the FTP prompt? This is because the wp-content directory is not writable. I adjusted the permissions and now when a plugin wants to update, I click yes and it magically happens! You will have to reference the /var/lib/wordpress/wp-content subdirectory in two places: What broke Images did, in a strange way. My media library is empty, but my images are still there. Something in the export and reimport did not work. For me its a minor inconvenience and due to moving from one system to another, but it still is there.

13 May 2015

Zlatan Todori : How to answer as a master

I have spent some fair amount of time during the life to explore making great responses to generic question (technical one included) and I can say without doubt that it is a pretty simple thing one could learn. First of all, answering question via email or in personal, it is very important that people feel that the person answering is there and is really "getting" their question. So a personal notice at beginning or at the end is not necessary but is a big plus. Particular part of answer should have 3 phases: straight yes or no answer, brief explaining why yes or no, and then explaining the opposite solution. Very important to keep it precise and simple as possible while explaining all what is needed for the person which asked question. For example Joe asks:
Can I install library libfoo1.2 without breaking software foo1.1
And Jane would answer:
Hi Joe, very good question as people often do try things like that and could end up in complicated situation. So the answer is NO. Pulling the libfoo1.2 would break foo1.1 because there were numerous changes from libfoo1.1 that break backward compatibility and there was also rewriting and porting to a newer version of language. Now having that out of way, you can safely pull also foo1.2 and install it with libfoo1.2 which is tested and should work for you without any problems. Best regards, Jane
And that's it. Lean, clean, cyborg.

11 May 2015

Craig Small: procps using GitLab CI

procps-ciThe procps project for a few years has been hosted at Gitorious. With the announcement that Gitorious has been acquired by GitLab and that all repositories need to move there, procps moved along to GitLab. At first I thought it would just be a like for like thing, but then I noticed that GitLab has this GitLab CI feature and had to try it out. CI here stands for Continuous Integration and is a way of automatically testing your program builds using a bunch of test scripts. procps already has a set of tests, with some a level of coverage that has room for improvement, so it was a good candidate to use for CI. The way GitLab works is they have a central control point that is linked to the git repo and you create runners, which are the systems that actually compile the programs and run the tests. The runners then feed back their results and GitLab CI shows it all in pretty red or green. The first problem was building this runner. Most of the documentation assumes you are building testing for Ruby. procps is built using C with the test scripts in TCL, so there was going to be some need to make some adjustments. I chose to use docker containers so that there was at least some isolation between the runners and the base system. I soon found the docker container I used (ruby:2.6 as suggested by GitLab) didn t have autopoint which mean autogen.sh failed so I had no configure script so no joy there. Now my second problem was I had never used docker before and beyond that it was some sort of container thing so like virtual machines lite, I didn t know much about it. The docker docs are very good and soon I had built my own custom docker image that had all the programs I need to compile and test procps. It s basically a cut-down Debian image with a few things like gettext-bin and dejagnu added in. Docker is not a full system With that all behind me and a few oh I need that too (don t forget you need git) moments we had a working CI runner. This was a Good Thing. You then find that your assumptions for your test cases may not always be correct. This is especially noticeable when testing something like procps which needs to work of the proc filesystem. A good example is, what uses session ID 1? Generally its init or systemd, but in Docker this is what everything runs as. A test case which assumes things about SID=1 will fail, as it did. This probably won t be a problem for testing a lot of normal programs that don t need to dig a deep into the host system as procps does, but it is something to remember. The docker environment looks a lot like a real system from the inside, but there are differences, so the lesson here is to write better tests (or fix the ones that failed, like I did). The runner and polling Now, there needs to be communication between the CI website and the runner, so the runner knows there is something for it to do. The gitlab runner has this setup rather well, except the runner is rather aggressive about its polling, hitting the website every 3 seconds. For someone on some crummy Australian Internet with low speeds and download quotas, this can get expensive in network resources. As there is on average an update once a week or so these seems rather excessive. My fix is pretty manual, actually its totally manual. I stop the daemon, then if I notice there are pending jobs I start the daemon, let it do its thing, then shut it down again. This is certainly not an optimal solution but works for now. I will look into doing something more clever later, possibly with webhooks.
Related articles

7 May 2015

Benjamin Mako Hill: DRM on Streaming Services

For the 2015 International Day Against DRM, I wrote a short essay on DRM for streaming services posted on the Defective by Design website. I m republishing it here. Between 2003 and 2009, most music purchased through Apple s iTunes store was locked using Apple s FairPlay digital restrictions management (DRM) software, which is designed to prevent users from copying music they purchased. Apple did not seem particularly concerned by the fact that FairPlay was never effective at stopping unauthorized distribution and was easily removed with publicly available tools. After all, FairPlay was effective at preventing most users from playing their purchased music on devices that were not made by Apple. No user ever requested FairPlay. Apple did not build the system because music buyers complained that CDs purchased from Sony would play on Panasonic players or that discs could be played on an unlimited number of devices (FairPlay allowed five). Like all DRM systems, FairPlay was forced on users by a recording industry paranoid about file sharing and, perhaps more importantly, by technology companies like Apple, who were eager to control the digital infrastructure of music distribution and consumption. In 2007, Apple began charging users 30 percent extra for music files not processed with FairPlay. In 2009, after lawsuits were filed in Europe and the US, and after several years of protests, Apple capitulated to their customers complaints and removed DRM from the vast majority of the iTunes music catalog. Fundamentally, DRM for downloaded music failed because it is what I ve called an antifeature. Like features, antifeatures are functionality created at enormous cost to technology developers. That said, unlike features which users clamor to pay extra for, users pay to have antifeatures removed. You can think of antifeatures as a technological mob protection racket. Apple charges more for music without DRM and independent music distributors often use DRM-free as a primary selling point for their products. Unfortunately, after being defeated a half-decade ago, DRM for digital music is becoming the norm again through the growth of music streaming services like Pandora and Spotify, which nearly all use DRM. Impressed by the convenience of these services, many people have forgotten the lessons we learned in the fight against FairPlay. Once again, the justification for DRM is both familiar and similarly disingenuous. Although the stated goal is still to prevent unauthorized copying, tools for stripping DRM from services continue to be widely available. Of course, the very need for DRM on these services is reduced because users don t normally store copies of music and because the same music is now available for download without DRM on services like iTunes. We should remember that, like ten years ago, the real effect of DRM is to allow technology companies to capture value by creating dependence in their customers and by blocking innovation and competition. For example, DRM in streaming services blocks third-party apps from playing music from services, just as FairPlay ensured that iTunes music would only play on Apple devices. DRM in streaming services means that listening to music requires one to use special proprietary clients. For example, even with a premium account, a subscriber cannot listen to music from their catalog using an alternative or modified music player. It means that their television, car, or mobile device manufacturer must cut deals with their service to allow each paying customer to play the catalog they have subscribed to. Although streaming services are able to capture and control value more effectively, this comes at the cost of reduced freedom, choice, and flexibility for users and at higher prices paid by subscribers. A decade ago, arguments against DRM for downloaded music focused on the claim that users should have control over the music they purchase. Although these arguments may not seem to apply to subscription services, it is worth remembering that DRM is fundamentally a problem because it means that we do not have control of the technology we use to play our music, and because the firms aiming to control us are using DRM to push antifeatures, raise prices, and block innovation. In all of these senses, DRM in streaming services is exactly as bad as FairPlay, and we should continue to demand better.

6 May 2015

John Goerzen: I Give Up on Google: Free is Too Expensive

I am really tired of things Google has done lately. The most recent example being retiring Classic Maps. That s a problem, because the current Maps mysteriously doesn t show most of my saved ( starred ) places. Google has known about this since at least 2013. There are posts all over their forums about it going back to when what is now regular Google Maps was beta. Google employees even knew about it and did nothing. For someone that made heavy use of it, this was quite annoying. But there have been plenty of others: I even used to use Flickr, then moved to Picasa when Yahoo stopped investing in Flickr. Now I m back to Flickr, because Google stopped investing in Picasa. The takeaway is that you can t really rely on Google for anything. Counting on something being there for an upcoming trip and then having it be suddenly yanked away is a level of frustration that just makes the service not so useful. Never knowing when obvious things (7-day calendar view) will be removed means you just can t depend on it. So, are there good alternatives? Things I m thinking of include: Anybody else moving off Google?

Next.

Previous.