Search Results: "tar"

1 August 2013

Julien Danjou: The definitive guide on how to use static, class or abstract methods in Python

Doing code reviews is a great way to discover things that people might struggle to comprehend. While proof-reading OpenStack patches recently, I spotted that people were not using correctly the various decorators Python provides for methods. So here's my attempt at providing me a link to send them to in my next code reviews. :-) How methods work in Python A method is a function that is stored as a class attribute. You can declare and access such a function this way:
>>> class Pizza(object):
... def __init__(self, size):
... self.size = size
... def get_size(self):
... return self.size
...
>>> Pizza.get_size
<unbound method Pizza.get_size>

What Python tells you here, is that the attribute get_size of the class Pizza is a method that is unbound. What does this mean? We'll know as soon as we'll try to call it:
>>> Pizza.get_size()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unbound method get_size() must be called with Pizza instance as first argument (got nothing instead)

We can't call it because it's not bound to any instance of Pizza. And a method wants an instance as its first argument (in Python 2 it must be an instance of that class; in Python 3 it could be anything). Let's try to do that then:
>>> Pizza.get_size(Pizza(42))
42

It worked! We called the method with an instance as its first argument, so everything's fine. But you will agree with me if I say this is not a very handy way to call methods; we have to refer to the class each time we want to call a method. And if we don't know what class is our object, this is not going to work for very long. So what Python does for us, is that it binds all the methods from the class Pizza to any instance of this class. This means that the attribute get_size of an instance of Pizza is a bound method: a method for which the first argument will be the instance itself.
>>> Pizza(42).get_size
<bound method Pizza.get_size of <__main__.Pizza object at 0x7f3138827910>>
>>> Pizza(42).get_size()
42

As expected, we don't have to provide any argument to get_size, since it's bound, its self argument is automatically set to our Pizza instance. Here's a even better proof of that:
>>> m = Pizza(42).get_size
>>> m()
42

Indeed, you don't even have to keep a reference to your Pizza object. Its method is bound to the object, so the method is sufficient to itself. But what if you wanted to know which object this bound method is bound to? Here's a little trick:
>>> m = Pizza(42).get_size
>>> m.__self__
<__main__.Pizza object at 0x7f3138827910>
>>> # You could guess, look at this:
...
>>> m == m.__self__.get_size
True

Obviously, we still have a reference to our object, and we can find it back if we want. In Python 3, the functions attached to a class are not considered as unbound method anymore, but as simple functions, that are bound to an object if required. So the principle stays the same, the model is just simplified.
>>> class Pizza(object):
... def __init__(self, size):
... self.size = size
... def get_size(self):
... return self.size
...
>>> Pizza.get_size
<function Pizza.get_size at 0x7f307f984dd0>

Static methods Static methods are a special case of methods. Sometimes, you'll write code that belongs to a class, but that doesn't use the object itself at all. For example:
class Pizza(object):
@staticmethod
def mix_ingredients(x, y):
return x + y

def cook(self):
return self.mix_ingredients(self.cheese, self.vegetables)

In such a case, writing mix_ingredients as a non-static method would work too, but it would provide it a self argument that would not be used. Here, the decorator @staticmethod buys us several things:
>>> Pizza().cook is Pizza().cook
False
>>> Pizza().mix_ingredients is Pizza.mix_ingredients
True
>>> Pizza().mix_ingredients is Pizza().mix_ingredients
True

Class methods Having said that, what are class methods? Class methods are methods that are not bound to an object, but to a class!
>>> class Pizza(object):
... radius = 42
... @classmethod
... def get_radius(cls):
... return cls.radius
...
>>>
>>> Pizza.get_radius
<bound method type.get_radius of <class '__main__.Pizza'>>
>>> Pizza().get_radius
<bound method type.get_radius of <class '__main__.Pizza'>>
>>> Pizza.get_radius is Pizza().get_radius
True
>>> Pizza.get_radius()
42

Whatever the way you use to access this method, it will be always bound to the class it is attached too, and its first argument will be the class itself (remember that classes are objects too). When to use this kind of methods? Well class methods are mostly useful for two types of methods:
class Pizza(object):
def __init__(self, ingredients):
self.ingredients = ingredients

@classmethod
def from_fridge(cls, fridge):
return cls(fridge.get_cheese() + fridge.get_vegetables())

class Pizza(object):
def __init__(self, radius, height):
self.radius = radius
self.height = height

@staticmethod
def compute_circumference(radius):
return math.pi * (radius ** 2)

@classmethod
def compute_volume(cls, height, radius):
return height * cls.compute_circumference(radius)

def get_volume(self):
return self.compute_volume(self.height, self.radius)

Abstract methods An abstract method is a method defined in a base class, but that may not provide any implementation. In Java, it would describe the methods of an interface. So the simplest way to write an abstract method in Python is:
class Pizza(object):
def get_radius(self):
raise NotImplementedError

Any class inheriting from Pizza should implement and override the get_radius method, otherwise an exception would be raised. This particular way of implementing abstract method has a drawback. If you write a class that inherits from Pizza and forget to implement get_radius, the error will only be raised when you'll try to use that method.
>>> Pizza()
<__main__.Pizza object at 0x7fb747353d90>
>>> Pizza().get_radius()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in get_radius
NotImplementedError

There's a way to triggers this way earlier, when the object is being instantiated, using the abc module that's provided with Python.
import abc

class BasePizza(object):
__metaclass__ = abc.ABCMeta

@abc.abstractmethod
def get_radius(self):
"""Method that should do something."""

Using abc and its special class, as soon as you'll try to instantiate BasePizza or any class inheriting from it, you'll get a TypeError.
>>> BasePizza()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class BasePizza with abstract methods get_radius

Mixing static, class and abstract methods When building classes and inheritances, the time will come where you will have to mix all these methods decorators. So here's some tips about it. Keep in mind that declaring a class as being abstract, doesn't freeze the prototype of that method. That means that it must be implemented, but i can be implemented with any argument list.
import abc

class BasePizza(object):
__metaclass__ = abc.ABCMeta

@abc.abstractmethod
def get_ingredients(self):
"""Returns the ingredient list."""

class Calzone(BasePizza):
def get_ingredients(self, with_egg=False):
egg = Egg() if with_egg else None
return self.ingredients + egg

This is valid, since Calzone fulfil the interface requirement we defined for BasePizza objects. That means that we could also implement it as being a class or a static method, for example:
import abc

class BasePizza(object):
__metaclass__ = abc.ABCMeta

@abc.abstractmethod
def get_ingredients(self):
"""Returns the ingredient list."""

class DietPizza(BasePizza):
@staticmethod
def get_ingredients():
return None

This is also correct and fulfil the contract we have with our abstract BasePizza class. The fact that the get_ingredients method don't need to know about the object to return result is an implementation detail, not a criteria to have our contract fulfilled. Therefore, you can't force an implementation of your abstract method to be a regular, class or static method, and arguably you shouldn't. Starting with Python 3 (this won't work as you would expect in Python 2, see issue5867), it's now possible to use the @staticmethod and @classmethod decorators on top of @abstractmethod:
import abc

class BasePizza(object):
__metaclass__ = abc.ABCMeta

ingredient = ['cheese']

@classmethod
@abc.abstractmethod
def get_ingredients(cls):
"""Returns the ingredient list."""
return cls.ingredients

Don't misread this: if you think this going to force your subclasses to implement get_ingredients as a class method, you are wrong. This simply implies that your implementation of get_ingredients in the BasePizza class is a class method. An implementation in an abstract method? Yes! In Python, contrary to methods in Java interfaces, you can have code in your abstract methods and call it via super():
import abc

class BasePizza(object):
__metaclass__ = abc.ABCMeta

default_ingredients = ['cheese']

@classmethod
@abc.abstractmethod
def get_ingredients(cls):
"""Returns the ingredient list."""
return cls.default_ingredients

class DietPizza(BasePizza):
def get_ingredients(self):
return ['egg'] + super(DietPizza, self).get_ingredients()

In such a case, every pizza you will build by inheriting from BasePizza will have to override the get_ingredients method, but will be able to use the default mechanism to get the ingredient list by using super().

Russ Allbery: Review: And Playing the Role of Herself

Review: And Playing the Role of Herself, by K.E. Lane
Publisher: Regal Crest
Copyright: 2006
Printing: June 2010
ISBN: 1-932300-72-4
Format: Kindle
Pages: 336
Caidence Harris is an actress who has finally gotten her big shot: a starring role in the hit police drama 9th Precinct. She plays half of a police partnership; the other half is played by her best friend, the considerably more famous Liz Stokley. She's quite happy with her life and her work, and only tentatively pondering what to do about coming out (at least to herself) as a lesbian. But matters get considerably more complicated when she starts falling for Robyn Ward, a star in a parallel legal drama. And the very high-profile girlfriend of one of the best male tennis players in the world. Yes, another lesbian romance. This one has won several awards, including the highest recommendation from my friend who's been prefiltering the genre for me and pointing out only the best stories. And despite being set firmly among the rich and famous and involving TV dramas with the serial numbers filed off, he was right. This is a thoroughly enjoyable story. One of its strongest parts is how Lane develops the romance between Caidence and Robyn. There is a bit of the inevitable physical chemistry so common in romances (I still don't believe kisses work that way), and a few descriptions of physical beauty (after all, there's a built-in excuse for everyone to be TV-star beautiful). But Lane avoids putting either at the center of the relationship. Instead, this is a romance built on trust, tentative at first and then growing but rocky. That speaks to me a lot more than all the descriptions of physical chemistry. For example, we don't see Caidence and Robyn meet for the first time in the novel, as is usually the case. They've been around each other before, with Caidence acting rather tongue-tied and a bit star-struck and Robyn thinking she's rather silly. The story begins when, for whatever reason, Caidence finally manages to relax and become genuine, and then both of them become closer as they find themselves lowering barriers around each other that they normally don't lower. Lane does an excellent job using a few key facial expressions as symbols and hallmarks of that, letting the reader follow the way that trust builds through the story. The other great element of this book is Caidence's friend Liz. She could have been a stereotype of a movie star: drop-dead gorgeous, a new boyfriend each week, but not particularly intelligent. That's how she comes off at first. But as the story progresses, Lane shows more and more of their friendship, and the reader gets to see just how much Liz's first impression is a combination of an act and her unapologetic tendency to do what she wants. Underneath, she's a strong female character and an excellent, if meddling friend; by the end of the book, she was in the running for my favorite character. This is characterization of a quality that I rarely see in any genre of novel. Given that it's a romance that I enjoyed, you can correctly expect there not to be a lot of obnoxious communication breakdowns or chapters full of the main characters refusing to communicate with each other. There are, of course, a wide variety of barriers in the way of a relationship for the main characters, not the least of which being that neither of them was looking for something serious. But while they both do occasionally thoughtless or hurtful things, it's always clear just why they were thoughtless, or where their buttons were accidentally pressed, and few of the character actions are outright stupid. And, delightfully, Lane structures the story so that there are multiple, incremental payoffs. One can not only enjoy the characters getting together in the end, as they always do in this genre, but a lot of important and meaningful steps along the way. Lane does a great job putting the reader in Caidence's head, helped by first-person narration, so that one can feel just how much some moments mean to her emotionally. Add in several other fleshed-out and interesting supporting characters, some incredible architecture that was fun to imagine, and rather more in the way of plot twists and dramatic moments than I usually expect in a romance (and although the book worried me once, none of them go to the dark places that would have been less fun), and one gets a novel that kept me engrossed from start to finish. I'm tempted to read it again, and that's rare for me for a romance novel. If you're in the mood for a heart-warming story about intelligent people learning how to be together, I highly recommend this book. It's a great book to read as a stress relief when you want a story that you know will turn out "right." Rating: 9 out of 10

30 July 2013

Rob Bradford: The Waylanders are coming

This GUADEC there will be a couple of sessions on Friday afternoon from 2pm about Wayland. I ll be giving a presentation with a brief introduction to what Wayland is, what new features we ve worked on in the last cycle as well as what s planned for the next one. As this is GUADEC i ll of course be covering how we re doing with getting Wayland integrated into GNOME. There will also be a Wayland panel discussion where you can ask your tricky questions of myself, Owen Taylor, Robert Bragg and Kristian H gsberg to get things started i ve got some already prepared! And if that s not enough Wayland for you, on Monday we ll be BoF ing between 10am and 2pm in room A218. It would be great to see you there.

Daniel Pocock: Switzerland's scariest railway (video)

Although promoted as a funicular railway, the Gelmerbahn funicular has the appearance of a giant roller coaster and all of the fun too. For all those still having nightmares about recent rail tragedies in the age of non-stop surveillance and on-line replays, the 108 degree ascent (or descent) may not be your thing. On the other hand, for those who experience the London Underground each morning, "mind the gap" takes on a whole new meaning as you see your feet hanging out the bottom of the train and the tracks descending vertically into the valley below. <video controls="" height="340" poster="http://danielpocock.com/sites/danielpocock.com/files/gbahn-preview.jpg" width="560">

<source src="http://video.danielpocock.com/Gelmerbahn.webm" type="video/webm"></source>
Click to access video on original page
</video> and here are download links for those who prefer to download now and watch later (about 160MB): At the top At the top, there is a scenic two hour walk around Lake Gelmer, the path takes you across the top of a dam built for hydro-electric power generation. Lake Gelmer - hydro-electric dam Getting there from DebConf13 and the Debian 20th birthday For those coming to Switzerland for DebConf13 next month, the Gelmerbahn is probably a little bit too far for a day trip but makes an excellent place to stop during a 2 or 3 day tour around Switzerland. It is accessible using the all-day bus pass for the tour of 3 or 4 mountain passes or by train to Innertkirchen and then a short ride on the bus. The busses are irregular, use sbb.ch to plan the journey. Walking down the mountain from the lake to the bus stop takes about 90 minutes and for those who choose this option, the ticket is cheaper. More Swiss travel blogs Please click here to access some of my previous blogs about Swiss travel with videos and useful ideas for day trips.

29 July 2013

EvolvisForge blog: SSD

Following the Wiki I put discard entries into my fstab(5) for swap (but not / as it s suggested to use fstrim instead, and I had noatime for /boot and relatime for / already) and changed the scheduler. What wasn t written there was to set vm.swappiness=0 in sysctl.d/local.conf *shrug* but it helps. What they also didn t mention: mount -t tmpfs swap /var/cache/apt/archives And in sid, which my dayjob-laptop is running, APT _finally_ creates the missing partial/ subdirectory itself. Thanks! I had the filesystems already created, so I changed the ext options with tune2fs: tune2fs -E stride=1024,stripe_width=1024 /dev/sda2 # 1 KiB blocks: /boot
tune2fs -E stride=256,stripe_width=256 /dev/sda4 # 4 KiB blocks: /
I wonder how well that works. I also did _not_ manage to find out my device s flash block size (search engine fodder: erase block size), so I assumed 1 MiB (also used for partition alignment already) as worst-case scenario. Input welcome on how to find *that* out.

Thomas Goirand: OpenStack Havana b2 available, openstack-debian-images approved

I have finished preparing the beta 2 of the next release of OpenStack. It is currently only available from out Git on Alioth (in /git/openstack), and directly from my jenkins repository, which creates Wheezy backports for it:
deb ftp://havana.pkgs.enovance.com/debian havana main
deb http://archive.gplhost.com/debian havana-backports main
As for every OpenStack release, a large number of Python modules needed to be packaged and are waiting in the FTP master NEW queue to be approved: oslo-sphinx, python-django-discover-runner, python-hacking, python-jsonrpclib, python-lesscpy, python-neutronclient, python-nosehtmloutput, python-requestbuilder, python-termcolor, sphinxcontrib-httpdomain and sphinxcontrib-pecanwsme. Let s hope that they will be approved before the next beta release in september (time where OpenStack Havana will be in feature freeze). The total number of packages maintained by the OpenStack team (on which I really am the only active maintainer for the moment ), there s 53 packages, plus these 11 packages waiting in the NEW queue. That s a big number of package, and I wouldn t mind some help One thing that annoyed all the community is that Quantum, the OpenStack network virtualization module, had to be renamed as Neutron, because of a trademark from Quantum (you probably remember the Quantum Fireball hard drives? Well, it s the same company ). Another good news is that my openstack-debian-images package has just been approved and landed in Sid. With it, you can automate the process of building a Debian image for OpenStack with a simple shell command (there s a man page that I wrote with it: read it if you need to build images). It is made of a single shell script to build the image, using kpartx, parted, mbr, debootstrap, extlinux and friends. I tried to keep it simple, not involving a huge amount of components. With the release of cloud-init 0.7.2-3, I have fixed a few bugs (3 important bugs, out of which 2 where RC bugs), thanks to contributions in the debian-cloud@lists.debian.org mailing list. This includes adding new init.d scripts, so we now have support for user data. This doesn t only benefits OpenStack images, but anyone willing to start virtual machines in the cloud (nowadays, every cloud implementation needs cloud-init installed in the virtual images). This means you can include a script in the metadata of the virtual machine you start it, and it will be executed at startup. If everything goes as planned (that is, no new RC bug), I will upload an update of cloud-init to backports in 5 days (there is already a version there, but it doesn t have the necessary init.d scripts to execute the user data scripts), and openstack-debian-images in 9 days. Then it will be possible to build OpenStack images with absolutely all the tools available from Wheezy (and backports). I hope to be able to discuss that during Debconf 13.

28 July 2013

Daniel Pocock: This blog was slashdotted

My earlier blog about rigged elections was slashdotted over the weekend. I had an email from the slashdot team giving me advance warning, having heard the stories about other sites getting knocked offline, I immediately logged into my Ganglia console to start the tsunami watch. I then set about enabling a reverse proxy using Squid to accelerate web server performance. This took about 30 minutes to set up. I chose Squid 3 so I could use a logging format similar to Apache. The wave hits A few hours after the warning email, details appeared on slashdot and sure enough, thousands of visitors started coming to this site. As detailed previously, the site is running on Debian servers with XCP. The web site is managed by Drupal/Apache/MySQL. Everything held together very well. Ganglia demonstrated an initial surge lasting about 10 minutes. From there, traffic has tapered off gradually over 24 hours, although it still remains well above average. slashdot comments One of the most interesting things about the slashdot community are the comments. Given that nobody really wanted to defend the indefensible and nothing in my blog surpasses some of what has been in the news recently (150 charges of fraud against a single MP), most of the discussion quickly turned to a comparison of broadband policies. For those not already aware of it, the Australian Government (Labor Party) is currently rolling out fibre to the home. The project is expected to take about 10 years, although it was delivered very quickly in certain areas where the Government needs votes. Opinions are divided: people who got it first think it is great, some people feel it is too expensive (it's much more expensive than a similar service in Switzerland, for example), some people feel it will be too slow when the final stages are completed in 10 years. It appears technically stronger than the solution proposed by the alternative party.

Hideki Yamane: now dpkg-deb uses xz for compression by default

from dpkg 1.17.0 's changelog

 dpkg (1.17.0) unstable; urgency=low
.
[ Guillem Jover ]
(snip)
* Switch dpkg-deb default compressor from gzip to xz. Build dpkg.deb using
gzip to make debootstrap life easier on non-Debian based systems.
Hallelujah, thanks! It'll reduce archive size and cut traffic, download time, too :-)

Then, I hope that dpkg would support "package delta" feature like yum's presto.

27 July 2013

Julien Danjou: OpenStack Ceilometer Havana-2 milestone released

Last week, the second milestone of the Havana developement branch of Ceilometer has been released and is now available for testing and download. This means the first half of the OpenStack Havana development has passed! New features Ten blueprints have been implemented as you can see on the release page. I'm going to talk through some of them here, that are the most interesting for users.
The Ceilometer API now returns all the samples sorted by timestamp. This blueprint is the first one implemented by Terri Yu, our OPW intern! In the same spirit, I've added the ability to limit the number of samples returned. On the alarming front, things evolved a lot. I've implemented the notifier system that will be used to run actions when alarms are triggered. To trigger these alarms, Eoghan Glynn (RedHat) worked on the alarm evaluation system that will use the Ceilometer API to check for alarm states. I've reworked the publisher system so it now uses URL formatted target for publication. That now allows to publish different meters to different target using the same publishing protocol (e.g. via UDP toward different hosts). Sandy Walsh (RackSpace) have been working on the StackTach like functionality and added the ability for the collector to optionally store the notification events received. Finally, Mehdi Abaakouk (eNovance) implemented a TTL system for the database, so you're now able to expire your data whenever you like. Bug fixes Thirty-five bugs were fixed, though most of them might not interest you so I won't elaborate too much on that. Go read the list if you are curious. Toward Havana 3 We now have 30 blueprints targeting the Ceilometer's third Havana milestone, with some of them are already started.I'll try to make sure we'll get there without too much trouble for the 6th September 2013. Stay tuned!

Petter Reinholdtsen: First beta release of Debian Edu/Skolelinux based on Debian Wheezy

The first wheezy based beta release of Debian Edu was wrapped up today. This is the release announcement: New features for Debian Edu 7.1+edu0~b0 released 2013-07-27 These are the release notes for for Debian Edu / Skolelinux 7.1+edu0~b0, based on Debian with codename "Wheezy". About Debian Edu and Skolelinux Debian Edu, also known as Skolelinux, is a Linux distribution based on Debian providing an out-of-the box environment of a completely configured school network. Immediately after installation a school server running all services needed for a school network is set up just waiting for users and machines being added via GOsa , a comfortable Web-UI. A netbooting environment is prepared using PXE, so after initial installation of the main server from CD, DVD or USB stick all other machines can be installed via the network. The provided school server provides LDAP database and Kerberos authentication service, centralized home directories, DHCP server, web proxy and many other services. The desktop contains more than 60 educational software packages and more are available from the Debian archive, and schools can choose between KDE, Gnome, LXDE and Xfce desktop environment. This is the fifth test release based on Debian Wheezy. Basically this is an updated and slightly improved version compared to the Squeeze release. ALERT: Alpha based installations should reinstall or downgrade the versions of gosa and libpam-mklocaluser to the ones used in this beta release. Software updates Other changes Known issues Where to get it To download the multiarch netinstall CD release you can use The MD5SUM of this image is: 55d5de9765b6dccd5d9ec33cf1a07109
The SHA1SUM of this image is: 996a1d9517740e4d627d100de2d12b23dd545a3f To download the multiarch USB stick ISO release you can use The MD5SUM of this image is: d8f0818c51a78d357de794066f289f69
The SHA1SUM of this image is: 49185ca354e8d0543240423746924f76a6cee733 How to report bugs http://wiki.debian.org/DebianEdu/HowTo/ReportBugs

Christian Perrier: [life] Running update July 27th 2013: Bubulle now runs in moutains

My last running update was sent after I had a great April month with two successful trail races (35km then 44km). The next target was Mont-Blanc marathon in Chamonix valley in early July, so it's time for me to send a new update before summer. May and June was mostly training. Huge training. Very long distances with 375km in May and over 430 in June (breaking my former monthly record by over 50 kilometers). Yes, that means over 14 kilometers every day..:-) I indeed achieved that with my "go to work partly by running" habits, which are now deeply included in my daily schedule. Much more convenient than bus ride (with the typical random schedules of RATP, which French people translate by "Rentre Avec Tes Pieds" or "Come back home by walking"....which is exactly what I'm doing). With such training (sometimes including extra rides in the forests, with short but steep ups and downs)....I was well prepared for THE early summer goal : Marathon du Mont-Blanc in Chamonix (not Ultra Trail du Mont-blanc : this one, I'm not ready for....yet).. My first mountain trail race, and what race! Imagine 42km in the moutain lovers heaven, namely the Chamonix Valley, with the Mont-Blanc, Aiguille Verte, Aiguille du Midi, Aiguilles Rouges, etc. above your head. Of course, not only in the valley but goind up and down "slightly"....in short 2200 meters positive climb..:-) I ran it with a female friend of mine, who I met in local races and through runners web forums. Sabine has about the same running skills than me and we developed a good and nice friendship, sharing our love for running in the nature. So, during May and June, we decided to run this race together as we were both registered for it. A first trial in a local 35km trail race two weeks before the MMB was very successful. We ended up in 3h40 and Sabine was ranked 5th female runner. And, moreover, we had a tremendous pleasure by crossing the finishing line together. So, we did it again in Chamonix. And, hell, we did it well: I had set the goal to 6 hours and 30 minutes and we completed the race in..... 6 hours and 34 minutes. Much much better than I and she would have expected. And, believe me, crossing the finishing line was again a special moment, particularly because the crowd at the end of this race make it look like l'Alpe d'Huez on the Tour de France. And, this on a wonderful sunny day, arriving at Planpraz, facing the Mont-Blanc. If you've ever been to Chamonix, you know what I mean. So, first moutain trail and, hell, not the last one! For those of you who can read French...or just want to see some pictures of a running bubulle, here is the link to the story. And these 42 kilometers were indeed so well done that the day after, while coming down from the famous Aiguille du Midi with Elisabeth (who came with me to Chamonix, of course)...I decided to run down from the cable car intermediate stop (2300m altitude) to Chamonix (1050m). Over 1200 meters down in 48 minutes... :-). Just a bit slower than the cable car....:) The remaining of July saw me again running back and forth to work and I'm indeed about to break my monthly record, eventually reaching 450 kilometers in one month and only two days NOT running in the whole month. Yeah, crazy. The upcoming Sunday, August 4th, just before going to Debconf 13, I'll spend a few days in Lanslebourg and will be running the EDF Cenis Tour trail, for 50 kilometers and 2600 meters positive climb. This time, a very "small" race, with no more than 150 runners. So, that will be another experience by running nearly entirely alone between 1400 and 2700 meters altitude. I'll post the outcome (or you'll probably hear about it at DC13....). See you, hopefully not in 3 months, for another update about running Bubulle.

26 July 2013

Justus Winter: Bootstrapping the Hurd

Due to popular demand this weeks report contains another awesome ascii screenshot. I added some lines to /etc/hurd/runsystem.sysv to inspect the environment very early in the boot process:
start ext2fs: Hurd server bootstrap: ext2fs[device:hd0s1] exec init proc auth
[...]
+ echo my pid is 1
my pid is 1
+ ps Ax
+ head
USER       PID TT STAT     TIME COMMAND
-            0  ? R<mo  0:00.01 /hurd/proc
root         1  - Sslow 0:00.02 /bin/sh /etc/hurd/runsystem
-            2  - Sp    0:00.00 /hurd/init root=device:hd0s1 console=com0
-            3  ? D<p   0:00.01 root=device:hd0s1 console=com0
-            4  - S<o   0:00.18 ext2fs --readonly --multiboot-command-line=root
-            5  - S<o   0:00.02 /hurd/exec
-            6  - S<o   0:00.00 /hurd/auth
root         7  - S<o   0:00.04 /hurd/term /dev/console device console
root         8  - S<o   0:00.00 /hurd/pflocal
+ pstree -p
sh(1)-+-init(2)-+-auth(6)
                 -ext2fs(4)-+-exec(5)
                             -pflocal(8)
                             -procfs(12)
                             -term(7)
       -pstree(11)
       -root=device:hd0s1(3)
Those who are familiar with the Hurd might note that /etc/hurd/runsystem.sysv runs as PID 1. At the end of this script /sbin/init is exec(2)uted, so sysvinit gets run as PID 1. This is the result of a fairly small patch series. It still took me like two days to come up with it since bootstrapping a Hurd system (or any microkernel OS, or like bootstrapping in general) is complicated. If you look at the screenshot above you will note the /hurd/pflocal process running with PID 8. This is the server providing pipe(2)s and Unix sockets. Before that server is started, one cannot use pipes to connect two processes with pipes like I did with ps and head (I actually lied a little here, because the way Debian/Hurd is set up, the pflocal server is started on demand courtesy of a Hurd feature called "passive translators", very similar to what the cool kids call "socket activation" today). Process 0 is the /hurd/proc server. It maps Mach tasks to the concept of Unix processes. Before that server is started, the notion of a process does not even exist. Pid 4 is the root filesystem. Without that, no other server can be read from the disk. But if the root filesystem is to be started first, it cannot be registered as a Unix process at the proc server, because that has not been started yet. So the first few processes are started in a weird way and order, and since they need to talk to each other, they need to register at each other. No Unix concepts exist yet, only Mach concepts like tasks and messages can be used. The boot sequence is roughly this (the numbers in parenthesis are the PIDs):
  1. Grub loads GNU Mach(3), the statically linked root filesystem translator (4) and the /hurd/exec (5) server (the exec server loads executables into Mach tasks, implementing most parts of execve(2)).
  2. The rootfs translator is started.
  3. The rootfs initiates the Hurd server bootstrap by starting /hurd/exec and /hurd/init (2).
  4. /hurd/init starts /hurd/proc (0) and /hurd/auth (6).
  5. The translators perform several rendezvouses to introduce themselves to each other.
  6. The proc servers state with respect to the already running processes is fixed.
  7. /hurd/init starts /etc/hurd/runsystem (1) which starts the pager (that pages out memory to disk) and later execs /sbin/init.
  8. From this point on the initialization of Debian/Hurd is no different from Debian/Linux.
Besides the pid one issue I spent my time on: Next week I hope to fully address the killall5 issue and to clean up, polish and submit my patch series for sysvinit. I will then revise my mtab translator prototype which is currently in its third iteration. As always, if there is time left, I will look at the network related issues.

25 July 2013

John Goerzen: Development Freedom in New Platforms

I started writing code when I was somewhere around 1st grade, hacking some rudimentary BASIC on a TRS-80. Since then, I ve used more programming languages than I can count; everything from C to Prolog, Python to Haskell, and probably a dozen others. I enjoy diversity in programming languages and tools. I ve had an interest in both web and mobile development for some time, but have done little with either. Reflecting on that lately, I realize that both platforms severely restrict my freedom to use the tools I choose. Even on an old x86 with 640K of RAM, I had choices; BASIC, Pascal, and C would work under DOS. (Plus assembler, but I had yet to learn that at the time.) On Android, my choices are Java or Java. (OK, or something that compiles to Java, but those selections are still limited.) Plus it seems like I d have to use Eclipse, which seems to have taken the kitchen sync crown from Emacs long ago, and not in a good way. On iOS, the choices are Objective C. And on the web, it s JavaScript. The problem with all this is that there s little room for language innovation on those platforms. If predictions about the decline of development on the PC pan out, how will the next advance in programming languages gain traction if the hot spots for development are locked down to three main languages? Will we find ourselves only willing to consider languages that can compile to Java/Dalvik bytecode? And won t that put limits on the possible space for innovation to occur within? Yes, I know about things like Mono and the Android Scripting Environment, which offer some potential for hope, but I think it s still safe to say that these platforms are pretty closed to development in unofficial languages. I don t know of major apps in the Android market written in Python, for instance. I know of efforts such as Ubuntu s, but even they simply lock down development to a different toolkit. There is no choice there of GTK vs. Qt, or such things. Sadly, I don t think it has to be this way. My phone has way more power than my computer did just a few years ago, and that computer was capable of not just executing, but even compiling, code written in all sorts of different languages. But I don t know if there s much we can do to change things. Even with native code on Android, plenty still has to go through Dalvik (at least that s my understanding). If you wanted to write a Python app with a full graphical touch interface on Android, I don t think this would be all that easy of a task.

John Goerzen: Spring Always Comes

There will come a time when you believe everything is finished. That will be the beginning. Louis L Amour
Last year, I wrote about the difficult times in my life and hope for future, but never really explained why. I have written little since, because there is little I can write without a bit of explanation. It is partly because of the complexity of the task of telling my story without telling too much of others stories. But it is important I tell this story. So often on the Internet, we hear only the brave face, the positive things that happen. This story involves tears and difficulty. And also we often see only anger and bitterness. This story involves joy and celebration. One afternoon last year, I was working as usual (I work from home) when two sheriff deputies arrived on my doorstep. They gave me paperwork showing that my now ex-wife had filed for divorce, had asked them to serve me the paperwork, and that she had been given temporary possession of the house. I had 2 hours to gather up clothes and a little computer equipment (there was a list) and leave the house. Thus began the most difficult time in my life. I went from reading a bedtime story and singing a bedtime song to my young boys every night to seeing them only a little, from living in the house my grandparents and dad had lived in to having no particular plan for where I d sleep that night, from thinking I had a good idea of what the future held to not knowing when, if ever, I d ever be back home. I worried about how the boys would fare (they have done well so far). And it was incomprehensible; I couldn t find answers to why? . In the time since, the divorce became final, I did return home, the boys spend more time with me, and a new normal emerged. At the time, it seemed like a sudden, deep winter blizzard. I couldn t see very far down the road, spring seemed far off, and I couldn t see very well either forward or backward. But I was determined to find positives in the situation. It started almost immediately; I had never been a person to talk about pain, but just a few hours after the divorce was filed, I knew I needed to talk to someone about it, and did. A week later, I shared about it in church. Amazing friends, locally and all over the world, provided support and encouragement. I had less total time with the boys than before the divorce, but more time with just the three of us, and we used it to play together at home, spend days in town, and even take a train trip to Santa Fe, where none of us had ever been before. I realized how much I could forgive, and that my ex-wife probably did the best she could with the persistent legacy of difficult life events that happened to her long before she met me. I understood this, and was never angry, just sad, for everyone. I have always known nobody is perfect, myself included, but can be hard on myself when I m less than perfect. I forgave myself, too, realizing that I did my best to help in the most unfamiliar of waters, and although I sometimes didn t get it right, my conscience is clear because my heart was in the right place and I tried, very hard. Most incredibly, I became a person with a deep sense of inner peace. I always tried to work hard to set life on a good path; I got good grades in school, am a good employee, and have a strong set of values. But where courts are involved, there s a strong sense of powerlessness. At times, there was nothing I could do to make life better for my boys or for me. I finally had to let go of taking on responsibility for all that on my own shoulders. I simply knew that things would be OK, and in fact were OK, and that there is nothing in life that really deserves worries. That doesn t mean worries are never present, but that mostly they are subdued, like a radio quietly playing in another room. When they aren t, I can sit down at the piano, play my penny whistle, sing, walk to my creek, talk to friends, or any of so many things that let them melt away. I stopped searching for happiness and peace, and let those things find me. In religious terms, my faith became not just an intellectual one, but also a spiritual one. An atheist friend asked me, Just what does religion mean to you anyway? My answer: The certainty that spring always comes, for everyone that understands this.
People speak of misfortunes and sufferings, remarked Pierre, but if at this moment I were asked: Would you rather be what you were before you were taken prisoner, or go through all this again? then for heaven s sake let me again have captivity and horseflesh! We imagine that when we are thrown out of our usual ruts all is lost, but it is only then that what is new and good begins. While there is life there is happiness. There is much, much before us. Tolstoy, War and Peace
By Thanksgiving, I had much to be thankful for. Some of it under my nose waiting to be rediscovered after years of distraction, such as the glorious Kansas sunrises. Some things were new, such as roasting a turkey all by myself (or, rather, with Jacob and Oliver) for the first time, and having it come out absolutely perfectly. And some were just the things of everyday life: that I lived in my own house again, that I could walk out to the creek at the edge of my property whenever I wanted, that I could play piano, that the sounds of laughter and little running feet again could often be heard on my wood floors. That s not to say everything was easy; the courts sometimes made decisions sometimes I didn t think were in the boys best interests, legal things dragged on and on, but in the end, peace endured. Happiness endured. I found myself thinking at Thanksgiving that it was the best Thanksgiving ever. Not much later, I considered myself happier than I ve ever been. I was focusing on the daily gifts of life, marveling at the sunrises, looking forward to life s next adventure, confident that it would be far better than the last. And then, to my complete surprise, I found myself in a relationship again. On top of all the wonderful things happening in my life, I met Laura. I never could have imagined a friend so wonderful, a relationship so loving and joyful, something to treasure so deeply. I can look back at events and shake my head in amazement and wonder, that I found myself happier than I d ever been, and then this wonderful relationship on top of that. I have been blessed to have the life I do, and feel almost embarrassingly lucky. I share this story because friends that had been through divorce years ago shared their stories. They gave me hope. And if I didn t share this story in this public way, I would be squandering an opportunity to find more positives from what happened. I hope that this helps, somehow, someone that is in pain know that there is beauty in the valley, and spring always comes, every single year. And I share it because happiness like this can t be repressed for long. Tolstoy was right. While there is life, there is happiness. I ll end with a story from last Thanksgiving. It was 6:10AM that weekend. I was still asleep, and heard this: Jacob, yelling from his room: Dad? Dad! Me, groggy: Yes, Jacob? Jacob: Can I go down and look at the Christmas tree? Me: Sure, and you can turn it on too. Jacob: Great! At that point, I could have gone back to sleep. I was really short on sleep that morning, and Jacob would have been fine. But I gave him a couple of minutes, then I went downstairs too. He was curled up on the piano bench, looking at the tree. I quietly turned up the downstairs thermostat, got a chair, put it next to the piano bench, and sat down by the tree too. Jacob crawled over onto my lap and snuggled up for awhile. Neither of us said anything. Then: Dad? Yes? This is the best Thanksgiving ever. And he gave me a big hug. And he was right. Yes, he was RIGHT! That will be the beginning. Spring comes!

24 July 2013

Gergely Nagy: My git tagging convention

Zigo posted an article earlier, one which I disagree with strongly. First of all, something that has been in use ever since git was invented is hardly a new fashion. Something that is used by the very same person who invented git and by git itself, is hardly surprising to be found so widespread. Especially not when it is used as an example in git's very own documentation! Writing it off as something silly, because of the way GitHub works, without even trying to understand the reason behind it is just plain wrong. Just because GitHub names tarballs the way it does, does not mean that we should change the way we tag.For example, cgit (used by, among others, kernel.org), uses the tag name as-is for its tarballs, unlike GitHub which prepends the project name, so one will have to mangle the filename in a debian/watch file anyway, when downloading from a host that is not GitHub, therefore his arguments for a raw version-only tag are bogus, unless your upstream is using GitHub.But I do not want to criticise only, rather, to provide a reason why a prefix is used, and why I chose the "worst" prefix: the project name itself.One very strong reason to use a prefix, either v or any other prefix is to ease tab completion. If you have a tag that is a bare number, you type the first, press tab, and get a mixture of commits and tags. You can't easily tell your completion system that you want commits or tags. With a prefix outside of the hex range, which v is, you can do that, and that makes working with it a lot easier.Is it for convenience? Yes. But ask yourself this: how many times do you have to write a debian/watch file? Once per package. How much time do you spend looking for a tag, or a commit? A lot more. So which one is most important? The convenience of someone who has to work around the tarball naming of the hosting service once, or the developer who works with the software daily? Obviously the latter!But that is not all. I'm an advocate of prefixing the tag with the project name, actually, and I've been doing that for all new projects for a while now, and I'm not going to change that, because it serves a very practical purpose: if you have a parent repository, with a lot of submodules, if git tag tells you the project name too, that makes it much easier to navigate. I don't have to bake submodule support into my shell prompt (that would be costy), and I won't find it surprising that if I enter a subdirectory, git tag fails to list the version I know I'm working on. Just because I happened to end up in a submodule, which is something I end up doing often, as I have many repositories that have submodules, which in turn have other submodules, and so on and so forth through many layers.A raw version number as a tag name is insufficient for my needs, simple as that. And I, as upstream, don't care that whoever packages my thing, will have to add a line to a watch file. That only needs to be done once, while I work with tags and commits daily.I'm sorry, but calling this naming convention a disease just because GitHub's tarball naming is what it is, without even considering that there may be other reasons behind it than legacy, and that there are hosting sites outside of GitHub, is a mistake. Dear package maintainers, please do not annoy us upstreams with requests to cripple our daily work. Thank you.

Thomas Goirand: The v sikness is spreading

It seems to be a new fashion. Instead of tagging software with a normal version number, many upstream adds a one letter prefix. Instead of version 0.1.2, it becomes version v0.1.2. This sickness has spread all around in Github (to tell only about the biggest one), from one repository to the next, from one author to the next. It has consequences, because Github (and others) conveniently provides tarballs out of Git tags. Then the tarball names becomes packagename-v0.1.2.tar.gz instead of package packagename-0.1.2.tar.gz. I ve even seen worse, like tags called packagename-0.1.2. Then the tarball becomes packagename-packagename-0.1.2. Consequently, we have to go around a lot of problems like mangling in our debian/watch files and so on (probably the debian/gbp.conf if you use that ). This is particularly truth when upstream doesn t make tarball and only provides tags on github (which is really fine to me, but then tags have to be made in a logical way). Worse: I ve seen this v-prefixing disease as examples in some howtos. What s wrong with you guys? From where is coming this v sickness? Have you guys watch too much the v 2009 series on TV, and you are a fan of the visitors? How come a version number isn t just made of numbers? Or is this just a v like the virus of prefixing release names with a v ? So, if you are an upstream author, reading debian planet, with your software packaged in Debian, and caught the bad virus of prefixing your version numbers with a v, please give-up on that. Adding a v to your tags is meaningless anyway, and it s just annoying us downstream. Edit: Some people pointed to me some (IMO wrong) reasons why to prefix version numbers. My original post was only half serious, and responding with facts and common sense breaks the fun! :) Anyway, the most silly one being that Linus has been using it. I wont comment on that one, it s obvious that it s not a solid argumentation. Then the second one is for tab completion. Well, if your bash-completion script is broken, fix it so that it does what you need, rather than going around the problem by polluting your tags. Then the 3rd argument was if you were merging 2 repositories. First this never happened to me to merge 2 completely different repos, and I very much doubt that this is an operation that you have to do often. Second, if you merge the repositories, the tags are loosing all meanings, and I don t really think you will need them anyway. Then the last one would be working with submodules. I haven t done it, and that might be the only case where it makes sense, though this has nothing to do with prefixing with v (you would need a much smarter approach, like prefixing with project names, which in that case makes sense). So I stand by my post: prefixing with v makes no sense.

Jan Wagner: Frag is bigger than frame.

Since 2 weeks I'm faced with some problems at a Xen hosting environment based on Debian wheezy. The issue is, that one specific domU is regularly failing network. The investigations showed, that no network packets from and to this system are seen on the the network interface in the domU and on the bridge interface in the dom0.
The funny part is, that the firewall on the dom0 now is logging packet rejects on the interface of the effected system and one interface of an other specific domU for the destination address. One pack and rejects on both interfaces ... really strange!
Shutting down the domU and recreating them doesn't help. Even if restarting other domUs, those are also effected by now. The only way to fix is to reboot the entire dom0. After several investigations I stumpled upon the following in the dmesg:
vif-xm116: Frag is bigger than frame.
vif-xm116: fatal error; disabling device
xenbr0: port 4(vif-xm116) entering disabled state
A quick search at the famouse search engine revealed, there seems something bigger. Even a Debian bug is open since February. Unfortunately this problem is still unfixed in wheezy and squeeze.
But ... there is hope in sight! Ian Campbell provided packages based on top of the latest kernel packages with a call for testing. Installing them seems to fix the issue in my environment. So we just have to hope this will get fixed soon, as this regression is really annoying. Thanks to all who make this happen in advance!

Keith Packard: present-redirect

Present Extension Redirection Multi-buffered applications have always behaved poorly in the presence of the Composite extension: Owen Taylor suggested that Present should offer a way to redirect operations to the compositing manager as a way to solve these problems. This posting is my attempt to make that idea a bit more concrete given the current Present design. Design Goals Here s a list of features I think we should try to provide:
  1. Provide accurate information to applications about when presentation to the screen actually occurs. In particular, GLX applications using GLX_OML_sync_control should receive the correct information in terms of UST and MSC for each Swap Buffers request.
  2. Ensure that applications still receive correct information as to the contents of their buffers, in particular we want to be able to implement EGL_EXT_buffer_age in a useful manner.
  3. Avoid needing to un-redirect full-screen windows to get page flipping behavior.
  4. Eliminate all extra copies. A windowed application may still perform one copy from back buffer to scanout buffer, but there shouldn t be any reason to copy contents to the composite redirection buffer or the compositing manager back buffer.
Simple Present Redirection With those goals in mind, here s what I see as the sequence of events for a simple windowed application doing a new full-window update without any translucency or window transformation in effect:
  1. Application creates back buffer, draws new frame to it.
  2. Application executes PresentRegion. In this example, the valid and update parameters are None , indicating that the full window should be redrawn.
  3. The server captures the PresentRegion request and constructs a PresentRedirectNotify event containing sufficient information for the compositor to place that image correctly on the screen:
    • target window for the presentation
    • source pixmap containing the new image
    • idle fence to notify when the source pixmap is no longer in use.
    • serial number from the request.
    • target MSC for the presentation. This should probably just be the computed absolute MSC value, and not the original target/divisor/numerator values provided by the application.
  4. The compositing manager receives this event and constructs a new PresentRegion request using the provided source pixmap, but this time targeting the root window, and constructing a valid region which clips the pixmap to the shape of the window on the screen. This request would use the original application s idle fence value so that when complete, the application would get notified. This request would need to also include the original target window and serial number so that a suitable PresentCompleteNotify event can be constructed and delivered when the final presentation is complete.
  5. The server executes this new PresentRegion operation. When complete, it delivers PresentCompleteNotify events to both the compositing manager and the application.
  6. Once the source pixmap is no longer in use (either the copy has completed, or the screen has flipped away from this pixmap), the server triggers the idle fence.
Multiple Application Redirection If multiple applications perform PresentRegion operations within the same frame, then the compositing manager will receive multiple PresentRedirectNotify events, and can simply construct multiple new PresentRegion requests. If these are all queued to the same global MSC, they will execute at the same frame boundary. No inter-operation dependency exists here. Complex Presentations Ok, so the simple case looks like it s pretty darn simple to implement, and satisfies the design goals nicely. Let s look at a couple of more complicated cases in common usage; the first is with translucency, the second with scaling application images down to thumbnails and the third with partial application window updates. Redirection with Translucency If the compositing manager discovers that a portion of the updated region overlays or is overlaid by something translucent (either another window, or drop shadows, or something else), then a composite image for that area must be constructed before the whole can be presented. Starting when the compositing manager receives the event, we have:
  1. The compositing manager receives this event. Using the new pixmap, along with pixmaps for the other involved windows and graphical elements, the compositing manager constructs an updated view for the affected portion of the screen in a back buffer pixmap. Once complete, a PresentRegion operation that uses this back buffer pixmap is sent to the X server. Again, the original target window and serial number are also delivered to the server so that a suitable PresentCompleteNotify event can be delivered to the application.
  2. The server executes this new PresentRegion operation; PresentCompleteNotify events are delivered, and idle fences triggered as appropriate.
Redirection with Transformation Transformation of the window contents means that we cannot always update a portion of the back buffer directly from the provided application pixmap as that will not contain the window border. Contents generated from a region that includes both application pixels and window border pixels must be sourced from a single pixmap containing both sets of pixels. One option that I ve discussed in the past to solve this would be to have the original application allocate the pixmap large enough to hold both the application contents and the surrounding window border. Have it draw the application contents at the correct offset within this pixmap, and then have the window manager contents drawn around that; either automatically by the X server, or even manually by the compositing manager. That would be mighty convenient for the compositing manager, but would require significant additional infrastructure throughout the X server and even harder the drawing system (OpenGL or some other system). There s another reason to want this though, and that s for sub-frame buffer scanout page swapping. The second option would be for the compositing manager to combine these images itself; there s a nice pixmap already containing the window manager image the composite redirect buffer. Taking the provided source pixmap and copying it directly to the target window will construct our composite image, just as if we had no Present redirection in place. This will cost an additional copy though, which we ve promised to avoid. Of course, as it s just for thumb-nailing or other visual effects, perhaps the compositing manager could perform this operation at a reduced frame rate, so that overall system performance didn t suffer. Retaining Access to the Application Buffer Above, I discussed having the idle fence from the redirected PresentRegion operation be sent along with the replacement PresentRegion operation. This ignores the fact that the composting manager may well need the contents of that application frame again in the future, when displaying changes for other applications that involve the same region of the screen. With the goal of making sure the idle fences are triggered as soon as possible so that applications can re-use or free the buffers quickly, let s think about when the triggering can occur.
  1. Full-screen flipped applications. In this case, the application s idle fence can be triggered once the application provides a new frame and the X server has flipped to that new frame, or some other scanout buffer.
  2. Windowed, copied applications. In this case, the application s idle fence can be triggered once the application provides a new frame to the compositing manager, and the X server doesn t have any presentations queued.
In both cases, we require that both the X server and the compositing manager be finished with the buffer before the application s idle fence should be triggered. One easy way to get this behavior is for the composting manager to create a new idle fence for its operations. When that is triggered, it would receive an X event and then trigger the applications idle fence as appropriate. This would add considerable latency to the application s idle fence a round trip through the compositing manager. The alternative would be to construct some additional protocol to make the applications idle fence dependent on the Present operation and some additional state provided by the compositing manager. Some experimentation here is warranted, but my experience with latency in this area is that it causes applications to end up allocating another back buffer as the idle notification arrives just after a buffer allocation request comes down to the rendering library. Definitely sub-optimal. An Aside on Media Stream Counters The GLX_OML_sync_control extension defines the Media Stream Counter (MSC) as a counter unique to the graphics subsystem which is incremented each time a vertical retrace occurs. That would be trivial if we had only one vertical retrace sequence in the world. However, we actually have N+1 such counters, one for each of the N active monitors in the system and a separate fake counter to be used when none of the other counters is available. In the current Present implementation, windows transition among the various Media Stream Counter domains as they move between the various monitors, and those monitors get turned on and off. As they move between these counter domains, Present tracks a global offset from their original domain. This offset ensures that the MSC value remains monotonically increasing as seen by each window. What it does not ensure is that all windows have comparable MSC sequence values; two windows on the same monitor may well have different MSC values for the same physical retrace event. And, even moving a window from one MSC domain to another and back won t make it return to the original MSC sequence values due to differences in refresh rates between the monitors. Internally, Present asserts that each CRTC in the system identifies a unique MSC domain, and it has a driver API which identifies which CRTC a particular window should be associated with. Once a particular CRTC has been identified for a window, client-relative MSC values and CRTC-relative MSC values can be exchanged using an offset between that CRTC MSC domain and the window MSC domain. The Intel driver assigns CRTCs to windows by picking the CRTC showing the greatest number of pixels for a particular window. When two CRTCs show the same number of pixels, the Intel driver picks the first in the list. Vblank Synchronization and Multiple Monitors Ok, so each window lives in a particular MSC domain, clocked by the MSC of the CRTC the driver has associated it with. In an un-composited world, this makes picking when to update the screen pretty simple; Present updates the screen when vblank happens in the CRTC associated with the window. In the composite redirected case, it s a bit harder all of the PresentRegion operations are going to target the root window, and yet we want updates for each window to be synchronized with the monitor containing that window. Of course, the root window belongs to a single MSC domain (likely the largest monitor, using the selection algorithm described above from the Intel driver). So, any PresentRegion requests will be timed relative to that single monitor. I think what is required here is for the PresentRegion extension to take an optional CRTC argument, which would then be used as the MSC domain instead of the window MSC domain. All of the timing arguments would be interpreted relative to that CRTC MSC domain. The PresentRedirectNotify event would then contain the relevant CRTC and the MSC value would be relative to that CRTC. A clever Compositing manager could then decompose a global PresentRegion operation into per-CRTC PresentRegion operations and ensure that multiple monitors were all synchronized correctly. We could take this even further and have the PresentRegion capable of passing a smaller CRTC-sized pixmap down to the kernel, effectively providing per-CRTC pixmaps with no visible explicit protocol Other Composite Users Ok, so the above discussion is clearly focused on getting the correct contents onto the screen with minimal copies along the way. However, what I ve ignored is how to deal with other applications, also using Composite at the same time. They re all going to expect that the composite redirect buffers will contain correct window contents at all times, and yet we ve just spent a bunch of time making that not be the case to avoid copying data into those buffers and instead copying directly to the compositing manager back or front buffers. Obviously the X server is aware of when this happens; the compositing manager will have selected for manual redirection on all top-level windows, while our other application will have only been able to select for automatic redirection. So, we ve got two pretty clear choices here:
  1. Have the X server change how Present redirection works when some other application selects for Automatic redirection on a window. It would copy the source pixmap into the window buffer and then send (a modified?) PresentRedirectNotify event to the compositing manager.
  2. Include a flag in the PresentRedirectNotify event that the composite redirect buffer needs to eventually get the contents of the source pixmap, and then expect the compositing manager to figure out what to do.
Development Plans As usual, I m going to pick the path of least resistance for all of the above options and see how things look; where the easy thing works, we can keep using it. Where the easy thing fails, I ll try something else. The changes required for this are pretty minimal. The PresentRegion request needs to gain a list of window/serial pairs that are also to be notified when the operation completes:
PRESENTNOTIFY  
    window: WINDOW
    serial: CARD32
     
 
    PresentRegion
    window: WINDOW
    pixmap: PIXMAP
    serial: CARD32
    valid-area: REGION or None
    update-area: REGION or None
    x-off, y-off: INT16
    idle-fence: FENCE
    target-crtc: CRTC or None
    target-msc: CARD64
    divisor: CARD64
    remainder: CARD64
    notifies: LISTofPRESENTNOTIFY
 
    Errors: Window, Pixmap, Match
The target-crtc parameter explicitly identifies a CRTC MSC domain. If None, then this request implicitly uses the window MSC domain. notifies provides a list of windows that will also receive PresentCompleteNotify events with the associated serial number when this PresentRegion operation completes.
 
    PresentRedirectNotify
    type: CARD8         XGE event type (35)
    extension: CARD8        Present extension request number
    length: CARD16          2
    evtype: CARD16          Present_RedirectNotify
    eventID: PRESENTEVENTID
    event-window: WINDOW
    window: WINDOW
    pixmap: PIXMAP
    serial: CARD32
    valid-area: REGION
    valid-rect: RECTANGLE
    update-area: REGION
    update-rect: RECTANGLE
    x-off, y-off: INT16
    target-crtc: CRTC
    target-msc: CARD64
    idle-fence: FENCE
    update-window: BOOL
 
The target-crtc identifies which CRTC MSC domain the target-msc value relates to. divisor and remainder have been removed as the target-msc value has been adjusted using the application values. If update-window is True, then the recipient of this event is instructed to provide reasonably up-to-date contents directly to the window by copying the contents of pixmap to the window manually. Beyond these two protocol changes, the compositing manager is expected to receive Sync events when the idle-fence is triggered and then manually perform a Sync operation to trigger the client s idle-fence when appropriate. I m planning to work on these changes, and then go re-work xcompmgr (or perhaps unagi, which certainly looks less messy) to incorporate support for Present redirection. The goal is to have something to demonstrate at Guadec, which doesn t seem impossible, aside from leaving on vacation in four days

23 July 2013

Gunnar Wolf: Sandro Cohen: The cyclist's zen

For all Spanish-speakers that read my blog, specially for the cyclists among you, and most specially for those that dwell in Mexico City's streets: I was recently pointed to a project started inside the Faceboook labrynth by Sandro Cohen, writer and academic: El zen del ciclista urbano. I met Sandro around twenty years ago. He writes in a very good, simple style. What I didn't know until now is that he has also become an urban cyclism promotor, just as me and many of my friends. In this page he started, he posts snippets on the topic of being an urban cyclist: As of today, he has 44 meditations, each of them a joy to read And very instructive as well. Thanks, Sandro, for the great resource! [update] I always find it... almost funny to read comments by so many people saying they'd rather have a lobotomy than to cycle in Mexico City. Hey! Mexico City is among the best places for cycling! Yes, we have to keep our eyes open and our instincts awake, but... Most of the city's area is flat. Many avenues have wide lanes and span a long distance. And yes, although there are some careless or aggressive drivers, after six years with the joy between my legs I can just say that... things are not as bad as you might imagine. I have very few (thankfully!) bad experiences, and so, so many good ones!

Russell Coker: Security is Impossible

The Scope of the Problem Security is inherently complex because of the large number of ways of circumventing it. For example Internet facing servers have been successfully attacked based on vulnerabilities in the OS, the server application, public key generation, DNS, SSL key certificates (and many other programs and algorithms in use), as well as the infrastructure and employees of all companies in the chain. When all those layers work reasonably well (not perfectly but well enough to not obviously be the weakest link) there are attacks on the end user systems that access the servers (such as the trojan horse programs used to attack PCs used for online banking). My Area of Interest The area of security that interests me is Linux software development. There are many related areas such as documentation and default configurations to make it easier for people to secure their systems (instead of insecure systems being the default option) which are all important. There are also many related fields such as ensuring that all people with relevant access are trustworthy. There are many interesting problems to solve in such areas most of which aren t a good match for my skills or just require more time than I have available. I sometimes write blog posts commenting on random security problems in other areas. Sometimes I hope to inspire people to new research, sometimes I hope to just inform users who can consider the issues when implementing solutions to security problems. Bugs In the software development side there are ongoing problems of bugs in code that weaken security. The fact that the main area of concern for people who are interested in securing systems is fixing bugs is an indication that the problem of software quality needs a lot of work at the moment. The other area that gets a reasonable amount of obvious work is in access control. Again it s an area that needs a lot of work, but the fact that we re not done with that is an indication of how far there is to go in generally improving computer security. Authenticating Software Releases There have been cases where source code repositories have been compromised to introduce trojan horse code, the ones I ve read about have been discovered reasonably quickly with little harm done but there could be some which weren t discovered. Of course it s likely that such attacks will be discovered because someone will have the original and the copies can be compared. Repositories of binaries are a bigger problem, it s not always possible to recompile a program and get a binary which checks out as being identical (larger programs often include the build time in the binary). Even for build processes which don t include such data it can be very difficult to determine the integrity of a build process. For example programs compiled with different versions of libraries, header files, or compilers will usually differ slightly. As most developers frequently change the versions of such software they will often be unable to verify their own binaries and any automated verification of such binaries will be impossible for anyone else. So if a developer s workstation was compromised without their knowledge it might be impossible for them to later check whether they released trojan binaries without just running the binaries in question and looking for undesired behavior. The problem of verifying past binaries is solvable for large software companies, Linux distributions, and all other organisations that have the resources to keep old versions of all binaries and libraries used to build software. For proprietary software companies the verification process would have to start with faith in the vendor of their OS and compiler doing the right thing. For Linux distributions and other organisations based on free software it would start by having the source to everything which can then be verified in theory although in practice verifying all source for compilers, the OS, and libraries would be a huge undertaking. Espionage There is a well documented history of military espionage, people who are sworn to secrecy have been subverted by money, blackmail, and by having political beliefs which don t agree with their government. The history of corporate espionage is less well documented but as corporations perform less stringent background checks than military organisations I think it s safe to assume that corporate espionage is much more common. Presumably any government organisation that can have any success at subverting employees of a foreign government can be much more successful in subverting programmers (either in companies such as Microsoft or in the FOSS community). One factor that makes it easier to launch such attacks is the global nature of software development. Government jobs that involve access to secret data have requirements about where the applicant was born and has lived, corporate jobs and volunteer positions in free software development don t have such requirements. The effort involved in subverting an existing employee of a software company or contributor to free software or the effort involved in getting an agent accepted in such a project would be quite small when compared to a nuclear weapons program. Therefore I think we should assume that every country which is capable of developing nuclear weapons (even North Korea) can do such things if they wish. Would the government of such a country want to subvert a major software project that is used by hundreds of millions of people? I can imagine ways that such things could benefit a government and while there would be costs for such actions (both in local politics and international relations) it seems most likely that some governments would consider it to be worth the risk and North Korea doesn t seem to have much to lose. Conclusion We would like every computer to be like a castle with a strong wall separating them from the bad things which can t be breached in ways that aren t obvious. But the way things are progressing with increasingly complex systems depending on more people and other systems it s becoming more like biology than engineering. We can think of important government systems as being comparable to the way people with compromised immune systems are isolated from any risk of catching a disease, the consequences of an infection are worse so greater isolation measures are required. For regular desktop PCs getting infected by a trojan is often regarded as being similar to getting a cold in winter. People just accept that their PC will be infected on occasion and don t bother making any serious effort to prevent it. After an infection is discovered the user (or their management for a corporate PC) tend not to be particularly worried about data loss in spite of some high profile data leaks from companies that do security work and the ongoing attacks against online banking and webcam spying on home PCs. I don t know what it will take for users to start taking security risks seriously. I think that a secure boot is a good step in the right direction, but it s a long way from being able to magically solve all security problems. I ve previously described some of the ways that secure boot won t save you [1]. The problems of subverting developers don t seem to be an immediate concern (although we should consider the possibility that it might be happening already without anyone noticing). The ongoing trend is that the value of computers in society is steadily increasing which therefore increases the rewards for criminals and spy agencies who can compromise them. Therefore it seems that we will definitely face the problems of subverted developers if we can adequately address the current technical problems related to flaws in software and inadequate access control. We just need to fix some of the problems which are exploited more easily to force the attackers to use the more difficult and expensive attacks. Note that it is a really good thing to make attacks more difficult, that decreases the number of organisations that are capable of attack even though it won t stop determined attackers. For end user systems the major problem seems to be related to running random programs from the Internet without a security model that adequately protects the system. Both Android and iOS make good efforts at protecting a system in the face of random hostile applications, but they have both been shown to fail in practice (it might be a good idea to have a phone for games that is separate from the phone used for phone calls etc). More research into OS security is needed to address this. But in the mean time users need to refrain from playing games and viewing porn on systems that are used for work, Internet banking, and other important things. While PCs are small and cheap enough that having separate PCs for important and unimportant tasks is practical it seems that most users don t regard the problems as being serious enough to be worth the effort.

Next.

Previous.