Search Results: "ana"

6 November 2012

Jon Dowland: Chronic

I've been ill for around 18 months. I started off ill, and more recently became more ill, so that I'd mentally divided my illness into two distinct periods. The truth, it turns out, is not quite so clear-cut. My more recent period of malaise spurred me on to seek a different treatment, which has restored me to a state of well-being which I had honestly forgot existed. It's quite possibly a combination of the desired outcome of treatment and also a side-effect of the drugs I'm on at the moment but I feel both mentally and physically the best I have in a long time. My mind is burning with ideas once again. Responsibilities and chores that used to sap me now seem less daunting. Having acclimatised to a steady-seeming state of being, which turns out to not be "well", I'd started to wonder if, in some respects, I'd reached the limits of my abilities. The knock-on effect of that was, to me, to consider what the future held, in many regards, including in terms of career. There were some benefits to my prior state of being. I'd begun to learn how to relax, how to wind down, how to not bite off more than I could chew. In particular I'd stopped worshipping at the alter of multitasking and started to give proper attention to one significant task at a time, with corresponding improvements in my productivity. These useful chunks of self-discipline have fallen to the wayside a little bit now. I'm enjoying a holiday in the body-and-mind landscape of my past. I don't feel too guilty about it. I think I deserve it. But I mustn't linger here too long, lest I forget those hard-won lessons. I've not managed to follow through on a lot of things I've tried to pick up over the last year and a half: software half-finished; Debian committments uncompleted; and I've felt a lingering sense of guilt about that, but at least I now have an explanation. Currently reading: Chronic City, by Jonathan Lethem.

Stefano Zacchiroli: bits from the DPL for October 2012

Freshly baked, bits from the DPL for October 2012.
Dear Project Members,
another month, another periodic report of DPL-ish activities, this time for October 2012. Highlights Debian on public clouds I've spent quite some time to improve Debian presence on the so called "public clouds". Following up to an inquiry of a fellow developer, I've reached out to Microsoft to investigate the possibility of having Debian as an option on Windows Azure. Around the same time, I've been approached by Amazon to have Debian as an option on the AWS marketplace. In both cases, we will need to overcome challenges of various kinds, at the technical (e.g. image preparation), bureaucratic (e.g. terms of the agreements we'll need to accept to be present), and political (e.g. chain of trust, platform freedom) levels. Up to now, discussions have been going on mostly in private, simply because they started as 1-to-1 inquiries and continued from there, but there is no good reason they should remain so. Hence, thanks to the listmasters and in particular Alexander Wirt, we have setup the new debian-cloud mailing list. If you are interested in these topics please join the list. For both Azure and AWS there is good progress on the technical part already; summaries will soon be posted on the list so that we are all on the same page. Similarly, I'll post there status reports about the bureaucratic requirements. And of course there is no reason to focus on specific clouds, if you'd like to support others and are willing to put some work to that end, please join the list and let us know. DPL helpers meeting I've already bothered you at least in my last platform and DebConf13 talk with observations about how non-scalable the DPL job is. After having collected applications of DPL helpers for a while, I've finally sat down and tried to put those applications into good use. The idea is simple: to the extend of possible, we should shift from a one-man-band job to a more "board-like" job, with people sharing an agenda, a list of outstanding tasks, and public communication. We have started slowly, setting up the #debian-dpl IRC channel and running periodic bi-weekly meetings there. You can find the meeting minutes and full logs at the usual place. We are still ramping up, so we don't have yet "fancy" stuff like a mailing list or an issue tracker entry, but they're in the working. Some of the outcome are starting to show, too (e.g. as part of recent discussions on 3rd party orphaning, or on our inbound trademark policy, or even in the forthcoming DMCA policy to make mentors.d.n an official project service). It's an experiment and a big challenge. I'm, for one thing, not yet convinced there are enough people interested in sharing the load of DPL duties (that look boring, for many tech geeks) in the long run. But I'm also convinced that the sustainability of the Debian organization model depends on this, so it's worth trying. If you're interested in the challenge and willing to volunteer some of your time, please join us on #debian-dpl . I'll take care of keeping the project informed of further evolution, in particular about the communication channels we will pick for day to day activities and accountability. Events / public communication I've spent most of my remaining Debian time in October attending events on behalf of the Project, in particular: On the topic of public communication, I've also coordinated with the press team an answer to a press inquiry about Secure Boot (which has become part of this article), and happily vouched for the Ubuntu charity marathon, adding some Debian challenges to it. Delegations As largely overdue matters, I've finalized the delegations mentioned last month, namely: FTP masters, New Members Front Desk, and Policy editors. Assets and legal stuff Miscellanea Now let's all go back to RC Bug squashing to make Wheezy a reality. SPAM-my link of the month is http://udd.debian.org/bugs.cgi and its various "views" at the bottom of the page. Cheers.
PS the day-to-day activity log for October 2012 is available at the usual place master:/srv/leader/news/bits-from-the-DPL.txt.201210

Raphaël Hertzog: My Free Software Activities in October 2012

This is my monthly summary of my free software related activities. If you re among the people who made a donation to support my work (120.46 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Dpkg At the start of the month, I reconfigured dpkg s git repository to use KGB instead of the discontinued CIA to send out commit notices to IRC (on #debian-dpkg on OFTC, aka irc.debian.org). I didn t do anything else that affects dpkg and I must say that Guillem does not make it easy for others to get involved. He keeps all his work hidden in his private for 1.17.x branch and refuses to open an official jessie branch as can be seen from the lack of answer to this mail. On the bright side, he deals with almost all incoming bugs even before I have a chance to take care of them. But it s a pity that I can never review any of his fixes because they are usually pushed shortly before an upload. Misc packaging I helped to get #689336 fixed so that the initrd properly setups the keymap before asking for a passphrase for an encrypted partition. Related to this I filed #689722 so that cryptsetup gains a dependency ensuring that the required tools for keymap setup are available. I packaged a new upstream version of zim (0.57) and also a security update for python-django that affected both Squeeze and Wheezy. I uploaded an NMU of revelation (0.4.13-1.2) so that it doesn t get dropped from Wheezy (it was on the release team list of leaf packages that would be removed if unfixed) since my wife is using it to store her passwords. I sponsored a new upstream version of ledgersmb. Debian France We managed to elect new officers for Debian France. I m taking over the role of president, Sylveste Ledru is the new treasurer and Julien Danjou is the new secretary. Thank you very much to the former officers: Carl Chenet, Aur lien Jarno and Julien Cristau. We re in the process of managing this transition which will be completed during the next mini-Debconf in Paris so that we can exchange some papers and the like. In the first tasks that I have set myself, there s recruiting two new members for the boards of directors since we re only 7 and there are 9 seats. I made a call for volunteers and we have two volunteers. If you want to get involved and help Debian France, please candidate by answering that message as soon as possible. The Debian Handbook I merged the translations contributed on debian.weblate.org (which led me to file this wishlist bug on Weblate itself) and I fixed a number of small issues that had been reported. I made an upload to Debian to incorporate all those fixes But this is still the book covering Squeeze so I started to plan the work to update it for Wheezy and with Roland we have decided who is going to take care of updating each chapter. Librement Progress is annoyingly slow on this project. Handling money for others is highly regulated, at least in the EU apparently. I only wanted an escrow account to secure the money of users of the service but opening this account requires either to be certified as a payment institution by the Autorit de contr le prudentiel or to get an exemption from the same authority (covering only some special cases) or to sign a partnership with an established payment institution. Being certified is out of scope for now since it requires a minimum of 125000 EUR in capital (which I don t have). My bank can t sign the kind of partnership that I would need. So I have to investigate whether I can make it fit in the limited cases of exemption or I need to find another payment institution that is willing to work with me. Gittip uses Balanced a payment service specialized in market places but unfortunately it s US-only if you want to withdraw money from the system. I would love a similar service in Europe If I can t position Librement as a market place for the free software world (and save each contributor the hassle to open a merchant account), then I shall fallback to the solution where Librement only provides the infrastructure but no account, and developers who want to collect donations will have to use either Paypal or any other supported merchant account to collect funds. That s why my latest spec updates concerning the donation service and the payment service mentions Paypal and the possibility of choosing your payment service for your donation form. Thanks See you next month for a new summary of my activities.

5 comments Liked this article? Click here. My blog is Flattr-enabled.

Russ Allbery: Review: Cerebus

Review: Cerebus, by Dave Sim
Series: Cerebus #1
Publisher: Aardvark-Vanaheim
Copyright: August 1987
Printing: July 2003
ISBN: 0-919359-08-6
Format: Graphic novel
Pages: 546
Cerebus is something of a legend in comics. Begun in December of 1977 by Dave Sim, it was one of the first entirely independent, self-published comics in a field dominated by the large work-for-hire companies like Marvel and DC. It ran for 300 issues and nearly 27 years and became one of the most influential independent comic books of all time, in part due to Sim's outspoken views in favor of creator rights and his regular use of the editorial pages in Cerebus issues to air those views. This collection (the first "phonebook") collects issues 1 through 25, with one of the amazing wrap-around covers that makes all of the phonebooks so beautiful (possibly partly by later Cerebus collaborator Gerhard, although if so it's uncredited so far as I can tell). Cerebus reliably has some of the best black-and-white art you will ever see in comics. There is some debate over where to start with Cerebus, and a faction that, for good reasons, argues for starting with the second phonebook (High Society). While these first twenty-five issues do introduce the reader to a bunch of important characters (Elrod, Lord Julius, Jaka, Artemis Roach, and Suenteus Po, for example), all those characters are later reintroduced and nothing that happens here is hugely vital for the overall story. It's also quite rough, starting as Conan parody with almost no depth. The first half or so of this collection features lots of short stories with little or no broader significance, and the early ones are about little other than Cerebus's skills and fighting abilities. That said, when reading the series, I like to start at the beginning. It is nice to follow the characters from their moment of first introduction, and it's delightful to watch Sim's ability grow (surprisingly quickly) through the first few issues. Cerebus #1 is bad: crude, simplistic artwork, almost nothing in the way of a story, and lots of purple narration. But flipping forward even to Cerebus #6 (the first appearance of Jaka), one sees a remarkable difference. By Cerebus #7, Cerebus looks like himself, the plot is getting more complex, and Sim is clearly hitting his stride. And, by the end of this collection, the art has moved from crude past competent and into truly beautiful in places. It's one of the few black-and-white comics where I never miss color. The detailed line work is more enjoyable than I think any coloring could be. The strength of Cerebus as an ongoing character slowly emerges from behind the parody. What I like the most about Cerebus is that he's neither a predestined victor (apart from the early issues that follow the Conan model most closely) nor a pure loner who stands apart from the world. He gets embroiled in political affairs, but almost always for his own reasons (primarily wealth). He has his own moral code, but it's fluid and situational; it's the realistic muddle of impulse and vague principle that most of us fall back on in our everyday life, which is remarkably unlike the typical moral code in comics (or even fiction in general). And while he is in one sense better and more powerful than anyone else in the story, that doesn't mean Cerebus gets what he wants. Most stories here end up going rather poorly for him, forcing daring escapes or frustrating cutting of losses. Sim quickly finds a voice for Cerebus that's irascible, wise, practical, and a bit world-weary, as well as remarkably unflappable. He's one of the best protagonists in comics, and that's already clear by the end of this collection. Parody is the focus of these first issues, which is a mixed bag. The early issues are fairly weak sword-and-sorcery parody (particularly Red Sonja, primarily a vehicle for some tired sexist jokes) and worth reading only for the development in Sim's art style and the growth of Cerebus as a unique voice. Sim gets away from straight parody for the middle of the collection, but then makes an unfortunate return for the final few issues, featuring parodies of Man-Thing and X-Men that I thought were more forced than funny. You have to have some tolerance for this, and (similar to early Pratchett) a lot of it isn't as funny as the author seems to think it is. That said, three of Sim's most brilliant ongoing characters are parodies, just ones that are mixed and inserted into the "wrong" genres in ways that bring them alive. Elrod of Melvinbone, a parody of Moorcock's Elric of Melnibone who speaks exactly like Foghorn Leghorn, should not work and yet does. He's the source of the funniest moments in this collection. His persistant treatment of Cerebus as a kid in a bunny suit shouldn't be as funny as it is, but it reliably makes me laugh each time I re-read this collection. Lord Julius is a straight insertion of Groucho Marx who really comes into his own in the next collection, High Society, but some of the hilarious High Society moments are foreshadowed here. And Artemis Roach, who starts as a parody of Batman and will later parody a huge variety of comic book characters, provides several delightful moments with Cerebus as straight man. I'm not much of a fan of parody, but I still think Cerebus is genuinely funny. High Society is definitely better, but I think one would miss some great bits by skipping over the first collection. Much of what makes it work is the character of Cerebus, who is in turn a wonderful straight man for Sim's wilder characters and an endless source of sharp one-liners. It's easy to really care about and root for Cerebus, even when he's being manipulative and amoral, because he's so straightforward and forthright about it. The world Sim puts him into is full of chaos, ridiculousness, and unfairness, and Cerebus is the sort of character to put his head down, make a few sarcastic comments, and then get on with it. It's fun to watch. One final note: I've always thought the "phonebook" collections were one of Sim's best ideas. Unlike nearly all comic book collections, a Cerebus phonebook provides enough material to be satisfying and has always felt like a good value for the money. I wish more comic book publishers would learn from Sim's example and produce larger collections that aren't hardcover deluxe editions (although Sim has an admitted advantage from not having to reproduce color). Followed by High Society. Rating: 7 out of 10

Cyril Brulebois: Network daemons

While I m not a big fan of clicky-clicky applications, I find it quite nice to be able to select some WEP/WPA settings in a GUI (as opposed to configuring wpa_supplicant manually) once in a while (new home, new couch by a friend s, new phone), and be done with it forever. Having mostly used Xfce over the years, I naturally came to trying out Wicd around 2006 (according to Wikipedia), and it looked like it did the job. It s been on all my laptops since then, and things were quite fine until the last laptop switch. For some (to be determined) reasons, that one comes up with a soft-blocked (rfkill) wireless LAN, which makes Wicd the biggest pain I have ever experienced as far as networking is concerned (see 4th bug report below). So I finally decided to try out that network-manager thing people apparently love to hate. A few bugs were filed in the process: I think nm-applet is the only Gtk3 application I have, meaning slightly bad visuals (no integration with the current Gtk2 theme for other applications), but besides that, everything is now running smoothly. Bottom line: contrary to Wicd, NetworkManager just works! I can t tell for sure since I m not using Gnome, but I m not sure why somebody would want to use Wicd there as it s clearly inferior on a technical level. Maybe something about the GUI/CLI? At least NetworkManager knows about netlink sockets, doesn t waste resources through useless polling, and is able to figure out when connections need kicking. That one just does its job. What triggered this Wicd NetworkManager switch is a look at the linux source, which reminded me of some strange polling noticed while playing around with auditd, as mentioned in #692417; that and the no network at boot-up unless one triggers several actions in a Wicd client bit.

5 November 2012

Joachim Breitner: Somewhere in Israel, someone printed 50 copies

Today, a small package was waiting for me at my door step, containing 52 printed and stapled copies of a paper that I wrote a while ago. To be more precise, two years ago, when I submitted the main results of my math Diploma Thesis to the Israel Journal of Mathematics. Next year at christmas eve, it was accepted and published online. And now, it seems, it finally went into print. So my paper now has page numbers and I have this pile of printouts that outnumbers the set of possibly interested readers by four dozens or more, and they would probably just download the online version instead of asking for this print out. This seems all very anachronistic to me.

Vincent Sanders: Another NetSurf Developer Workshop

The NetSurf Developers - Bringing you alternative internet meme delivery since 2002
The weekend saw some of the NetSurf developers gather once again to do battle against the Red Queen...no, wait. I mean plan and write code.

The successful event was once again kindly hosted by Collabora at their Cambridge office, the facilities remain as good as at the previous event and provided a comfortable environment in which to work.

Four developers managed to attend from around the UK John-Mark Bell, Vincent Sanders, Daniel Silverstone and Rob Kendrick. The turnout was a little lower than previous events, however Michael Drake managed to assist with some bug fixes remotely.

From Friday to Sunday afternoon we each managed to put in 24 hours of work consisting of almost 90 commits changing over 100 files and removing 10,000 lines of code.

Whiteboard with the 3.0 release tasks
The main theme of the event was consolidation ready for the 3.0 release. While not a glamorous target this is a major release for the project which we want to achieve soon, this requires the completion of numerous ongoing tasks (like finally removing the dependency of libxml) and fixing as many reported bugs as possible.

The Major things we achieved were:

Once the final usage of libXML is removed (from the hotlist code) NetSurf will become substantially smaller on several platforms as we will not have two separate parser implementations linked.

One big feature of the 3.0 release which was not originally on the roadmap is that of JavaScript. We initially intend to provide two builds of with and without the JavaScript interpreter compiled into NetSurf. This allows those on very small systems to continue to run NetSurf and those users who want a richer web experience to enjoy it.

Although JavaScript is a major new feature which is frequently requested the initial implementation will be incomplete. To complete the feature several thousand JavaScript to DOM interface binding functions must still be written. In addition it should be noted that we have yet to add dynamic layout or extensions such as Canvas or WebGL

We had a Raspberry Pi which was used to run RISC OS so we could examine the operation on this system.

The 2.9 edition as shipped works OK but is lacking the substantial progress made in the last six months.

The CI system build was run on the system and works well so we are reassured the 3.0 edition will provide an even better browsing experience.

The event was a great success overall and we are now very close to our 3.0 release which we hope to produce before Christmas.

4 November 2012

Russ Allbery: Review: Set This House in Order

Review: Set This House in Order, by Matt Ruff
Publisher: Perennial
Copyright: 2003
Printing: 2004
ISBN: 0-06-095485-X
Format: Trade paperback
Pages: 479
A textbook case history would say that Andy Gage has disassociative identity disorder (referred to here as the more common although medically inaccurate "multiple personality disorder" for reasons of clarity and to match casual speech, according to the author). However, that's a completely inadequate description. A better description would be that Andy Gage's body is inhabited by a large set of different personalities, who take turns being in control. The original Andy Gage is dead, shattered by horrific childhood abuse. Aaron became the dominant personality and gave structure to the internal geography of the mind they all share, with the help of extensive counselling. But Aaron was exhausted by being the public face and maintaining control of the shared body. Hence, Andrew. Andrew Gage is the first-person narrator of half of this book. He is twenty-eight, the same age as the body, but he's only existed for two years. He considers Aaron his father, since Aaron called him out of the lake inside their internal metal geography. His purpose is to run the body and be the public face of the family. He is by turns capable and naive, shy on personal experience, but able to get the help of the others. The one heard from most frequently is Adam, a rude and mischevious teenager who also has an uncanny ability to read other people, and who frequently watches through the body's eyes from what the family calls the "pulpit" and gives Andrew advice. Andrew works for Julie Sivik, a woman he met the day he quit his job at a big-box store because he couldn't explain his behavior to his boss without discussing being a multiple. Julie is a wholly impractical, slightly unethical, well-meaning force of chaos who serves as the thread that draws the plot of the book together. She's running a startup on ingenuity and borrowed money that is attempting to create a comprehensive and customizable virtual reality world. She's the one who finds Penny and hires her, in part for her skills, and in part to push Penny and Andrew together. Penny Driver has the same condition as Andrew, but Penny is only vaguely aware of it. The world for her is full of blackouts, lost time, waking up in strange and sometimes horrific situations, and attempting to structure her life based on notes left for herself that she didn't write. When she meets Andrew, she's barely making it through life; thankfully, one of her personalities is remarkably good at computers, and that has been enough to make a living despite the chaos. But Julie is the unintentional catalyst for her losing that job, the best job she's ever had, leaving her with no real alternative but to take the job with Julie's company. There are a few things that are important to say up-front about this book. First, neither I nor the author are actually multiples (or, put another way, neither of us have DID), and writing about atypical mental states is fraught with peril. It's very easy to misrepresent other people's experiences, and multiple personalities in particular are frequent material for dramas of all sorts. There's something people find inherently fascinating about the idea, and it's easy for that fascination to run away with reality and distort the lived experience. I recommend reading Ruff's FAQ about the book, which spells out what his goals are and what approach he took. I did some quick searches and didn't uncover a lot of anger or dislike of the book among either people who have DID or psychiatrists, and it seems true to the descriptions of internal reality I've heard from people who are multiples. But despite being more respectful of real experience than most fictional treatments of DID, this is still a work of fiction written by someone without first-hand experience, and it's worth keeping that in mind. Second, DID is an intensely controversial diagnosis among psychiatrists. There are, generally speaking, two major controversies. One is over whether or not DID can actually be caused by traumatic experiences such as childhood abuse, or whether it is a creation of psychiatrists, a mental model that they've imposed on their patients or convinced their patients to believe. Set This House in Order comes down firmly on the first side of that controversy: that DID can be created by abuse and is not a creation of psychotherapy. (So, I should note, does every multiple that I've ever talked to or read.) Second, there is significant controversy over treatment, with some psychiatrists believing that integration of the personalities into one is the only true treatment, and others believing that multiples can remain stable and functional as multiple personalities. Set This House in Order leans strongly towards the view that multiples can be stable as multiples and that while integration may be appropriate for some, it's not for all. Both of these viewpoints are presented very convincingly by the narrative in the book, and both are consistent with the beliefs of a lot of multiples, but it's worth remembering that these are active and often quite heated controversies and a novel is not evidence. It's good to be aware of both of those points, and to read Put This House in Order as something of a what-if, where much more is clear about DID than is actually clear in our world. But once one has done that, one is in for an excellent book. I loved this book from start to finish. (Well, okay, I felt some surprise at the nature of the ending, but it grew on me after further consideration.) It's simply a brilliant novel, with a little bit of everything: very good characterization, messy love affairs, the satisfying tension of someone overcoming great odds to put themselves together with the help of friends, a startlingly original perspective of the world, some excellent surprises, and some absolutely wonderful characters. Andrew is a good choice of narrator: he brings an earnest calm to a story that could otherwise be quite chaotic and provides a great vantage point for the reader. And I found Penny (and particularly some of her other souls) impossible not to like. Set This House in Order is a tense book in places, and it's a book where people's lives break down before they can get better. It's also a book about rather horrific abuse, and while nearly all of the specifics happen off-stage, imagination fills in some nasty details. But Ruff manages to write it with a structure and with enough calm assurance that I felt just reassured enough without taking away the tension. That's a tricky balance; few books do it better. And the supporting characters are a delight. One might think, with all the complexity happening for both Andrew and Penny, that Ruff wouldn't have room in the narrative for other characters, but one would be wrong. There are very few saints and demons here, just a lot of complex people, with their own problems and their own perspectives, who are generally trying to do the best they can from day to day. There are some clear-cut villains in the past Ruff wisely makes no attempt to add shades of grey to abusers but even there one can see some clear signs of untreated severe mental illness and chaotic lives rather than mustache-twirling pre-meditated villainy. This is, in short, a book full of people, in all their messy complications, which is what makes it so delightful. I'm not going to say anything more about the plot, or even the various personalities, since discovering the direction of the book was so much of the fun. I will say that, despite the subject matter and some of the events, the book is surprisingly gentle. One of the things about it that I loved the most was the emotional tone Ruff gave to being a multiple: the sense of a squabbling but tight-knit family looking out for each other. I can't judge the reality of this impression, but it makes for a wonderful story. Highly recommended. The best book I've read so far this year. Rating: 10 out of 10

Wouter Verhelst: Why Windows is not ready for general usage

There's this new Windows thing that people keep talking about, and which attempts to replace Linux as the major operating system. I've given it a look, but there's a wide range of problems with using it: In short, while this Windows thing has some promise, the above list shows that there's clearly still quite some work to be done before anyone can even think about starting to use it seriously. Hopefully the Windows developers will understand that and work on the above list of serious issues; otherwise, I'm afraid Windows will not be used by many people. Update: before you go all berserk and start commenting on this list, please read my followup to this post, which explains the point I was trying to make.

Paul Wise: Remembering the dead

Last week was el D a de los Muertos (the Day of the Dead), a Mexican holiday where people gather to remember their dead. Today I would like to remember <predator>. pred was a core member of the cat@lyst collective, a group of folks providing computers, Internet and other tech resources to the activists of Sydney, Australia. I knew pred as cat@lyst's hardware geek, he was always scrounging computers from roadsides, coaxing them back to life and setting them to work supporting cat@lyst. I only knew pred through email and just missed out on meeting him due to the cancer that eventually killed him. pred wrote extensively during his life and of his experiences while dying of cancer. After he died, some friends compiled some of his writing into a book, simply named pred.txt. His full writings also happen to have been archived on the net. Through the book and his writings I came to learn that he was much more than just a hardware geek and gain some more insight into who he was. pred was a geek, activist, anarchist, urban explorer, scientist in the area of molecular genetics, cancer victim and all-around awesome dude. I still miss him.

2 November 2012

Vincent Bernat: Network virtualization with VXLAN

Virtual eXtensible Local Area Network (VXLAN) is a protocol to overlay a virtualized L2 network over an existing IP network with little setup. It is currently described in an Internet-Draft. It adds the following perks to VLANs while still providing isolation:
  1. It uses a 24-bit VXLAN Network Identifier (VNI) which should be enough to address any scale-based concerns of multitenancy.
  2. It wraps L2 frames into UDP datagrams. This allows one to rely on some interesting properties of IP networks like availability and scalability. A VXLAN segment can be extended far beyond the typical reach of today VLANs.
The VXLAN Tunnel End Point (VTEP) originates and terminates VXLAN tunnels. Thanks to a serie of patches from Stephen Hemminger, Linux can now act as a VTEP. Let s see how this works.

About IPv6 When possible, I try to use IPv6 for my labs. This is not the case here for several reasons:
  1. IP multicast is required and PIM-SM implementations for IPv6 are not widespread yet. However, they exist. This explains why I use XORP for this lab: it supports PIM-SM for both IPv4 and IPv6.
  2. VXLAN Internet-Draft specifically addresses only IPv4. This seems a bit odd for a protocol running on top of UDP and I hope this will be fixed soon. This is not a major stopper since some VXLAN implementations support IPv6.
  3. However, the current implementation for Linux does not support IPv6. IPv6 support will be added later.
Once IPv6 support is available, the lab should be easy to adapt.

Lab So, here is the lab used. R1, R2 and R3 will act as VTEPs. They do not make use of PIM-SM. Instead, they have a generic multicast route on eth0. E1, E2 and E3 are edge routers while C1, C2 and C3 are core routers. The proposed lab is not resilient but convenient to explain how things work. It is built on top of KVM hosts. Have a look at my previous article for more details on this. VXLAN lab The lab is hosted on GitHub. I have made the lab easier to try by including the kernel I have used for my tests. XORP comes preconfigured, you just have to configure the VXLAN part. For this, you need a recent version of ip.
$ sudo apt-get install screen vde2 kvm iproute xorp git
$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git
$ cd iproute2
$ ./configure && make
You get  ip' as  ip/ip' and  bridge' as  bridge/bridge'.
$ cd ..
$ git clone git://github.com/vincentbernat/network-lab.git
$ cd network-lab/lab-vxlan
$ ./setup

Unicast routing The first step is to setup unicast routing. OSPF is used for this purpose. The chosen routing daemon is XORP. With xorpsh, we can check if OSPF is working as expected:
root@c1# xorpsh
root@c1$ show ospf4 neighbor   
  Address         Interface             State      ID              Pri  Dead
192.168.11.11    eth0/eth0              Full      3.0.0.1          128    36
192.168.12.22    eth1/eth1              Full      3.0.0.2          128    33
192.168.101.133  eth2/eth2              Full      2.0.0.3          128    36
192.168.102.122  eth3/eth3              Full      2.0.0.2          128    38
root@c1$ show route table ipv4 unicast ospf   
192.168.1.0/24  [ospf(110)/2]
                > to 192.168.11.11 via eth0/eth0
192.168.2.0/24  [ospf(110)/2]
                > to 192.168.12.22 via eth1/eth1
192.168.3.0/24  [ospf(110)/3]
                > to 192.168.102.122 via eth3/eth3
192.168.13.0/24 [ospf(110)/2]
                > to 192.168.102.122 via eth3/eth3
192.168.21.0/24 [ospf(110)/2]
                > to 192.168.101.133 via eth2/eth2
192.168.22.0/24 [ospf(110)/2]
                > to 192.168.12.22 via eth1/eth1
192.168.23.0/24 [ospf(110)/2]
                > to 192.168.101.133 via eth2/eth2
192.168.103.0/24        [ospf(110)/2]
                > to 192.168.102.122 via eth3/eth3

Multicast routing Once unicast routing is up and running, we need to setup multicast routing. There are two protocols for this: IGMP and PIM-SM. The former one allows routers to forward multicast datagrams while the first one allows hosts to subscribe to a multicast group.

IGMP IGMP is used by hosts and adjacent routers to establish multicast group membership. In our case, it will be used by R2 to let E2 know it subscribed to 239.0.0.11 (a multicast group). Configuring XORP to support IGMP is simple. Let s test with iperf to have a multicast listener on R2:
root@r2# iperf -u -s -l 1000 -i 1 -B 239.0.0.11
------------------------------------------------------------
Server listening on UDP port 5001
Binding to local address 239.0.0.11
Joining multicast group  239.0.0.11
Receiving 1000 byte datagrams
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
On E2, we can now check that R2 is properly registered for 239.0.0.11:
root@e2$ show igmp group
Interface    Group           Source          LastReported Timeout V State
eth0         239.0.0.11      0.0.0.0         192.168.2.2      248 2     E
XORP documentation contains a good overview of IGMP.

PIM-SM PIM-SM is far more complex. It does not have its own topology discovery protocol and relies on routing information from other protocols, OSPF in our case. I will describe here a simplified view on how PIM-SM works. XORP documentation contains more details about PIM-SM. The first step for all PIM-SM routers is to elect a rendez-vous point (RP). In our lab, only C1, C2 and C3 have been configured to be elected as a RP. Moreover, we give better priority to C3 to ensure it wins. RP election
root@e1$ show pim rps   
RP              Type      Pri Holdtime Timeout ActiveGroups GroupPrefix       
192.168.101.133 bootstrap 100      150     135            0 239.0.0.0/8
Let s suppose we start iperf on both R2 and R3. Using IGMP, they subscribe to multicast group 239.0.0.11 with E2 and E3 respectively. Then, E2 and E3 send a join message (also known as a (*,G) join) to the RP (C3) for that multicast group. Using the unicast path from E2 and E3 to the RP, the routers along the paths build the RP tree (RPT), rooted at C3. Each router in the tree knows how to send multicast packets to 239.0.0.11: it will send them to the leaves. RP tree
root@e3$ show pim join   
Group           Source          RP              Flags
239.0.0.11      0.0.0.0         192.168.101.133 WC   
    Upstream interface (RP):   eth2
    Upstream MRIB next hop (RP): 192.168.23.133
    Upstream RPF'(*,G):        192.168.23.133
    Upstream state:            Joined 
    Join timer:                5
    Local receiver include WC: O...
    Joins RP:                  ....
    Joins WC:                  ....
    Join state:                ....
    Prune state:               ....
    Prune pending state:       ....
    I am assert winner state:  ....
    I am assert loser state:   ....
    Assert winner WC:          ....
    Assert lost WC:            ....
    Assert tracking WC:        O.O.
    Could assert WC:           O...
    I am DR:                   O..O
    Immediate olist RP:        ....
    Immediate olist WC:        O...
    Inherited olist SG:        O...
    Inherited olist SG_RPT:    O...
    PIM include WC:            O...
Let s suppose that R1 wants to send multicast packets to 239.0.0.11. It sends them to R1 which does not have any information on how to contact all the members of the multicast group because it is not the RP. Therefore, it encapsulates the multicast packets into PIM Register packets and sends them to the RP. The RP decapsulates them and sends them natively. The multicast packets are routed from the RP to R2 and R3 using the reverse path formed by the join messages. Multicast routing via the RP
root@r1# iperf -c 239.0.0.11 -u -b 10k -t 30 -T 10
------------------------------------------------------------
Client connecting to 239.0.0.11, UDP port 5001
Sending 1470 byte datagrams
Setting multicast TTL to 10
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
root@e1# tcpdump -pni eth0
10:58:23.424860 IP 192.168.1.1.35277 > 239.0.0.11.5001: UDP, length 1470
root@c3# tcpdump -pni eth0
10:58:23.552903 IP 192.168.11.11 > 192.168.101.133: PIMv2, Register, length 1480
root@e2# tcpdump -pni eth0
10:58:23.896171 IP 192.168.1.1.35277 > 239.0.0.11.5001: UDP, length 1470
root@e3# tcpdump -pni eth0
10:58:23.824647 IP 192.168.1.1.35277 > 239.0.0.11.5001: UDP, length 1470
As presented here, the routing is not optimal: packets from R1 to R2 could avoid the RP. Moreover, encapsulating multicast packets into unicast packets is not efficient either. At some point, the RP will decide to switch to native multicast1. Rooted at R1, the shortest-path tree (SPT) for the multicast group will be built using source-specific join messages (also known as a (S,G) join). Multicast routing without RP From here, each router in the tree knows how to handle multicast packets from R1 to the group without involving the RP. For example, E1 knows it must duplicate the packet and sends one through the interface to C3 and the other one through the interface to C1:
root@e1$ show pim join   
Group           Source          RP              Flags
239.0.0.11      192.168.1.1     192.168.101.133 SG SPT DirectlyConnectedS 
    Upstream interface (S):    eth0
    Upstream interface (RP):   eth1
    Upstream MRIB next hop (RP): 192.168.11.111
    Upstream MRIB next hop (S):  UNKNOWN
    Upstream RPF'(S,G):        UNKNOWN
    Upstream state:            Joined 
    Register state:            RegisterPrune RegisterCouldRegister 
    Join timer:                7
    KAT(S,G) running:          true
    Local receiver include WC: ....
    Local receiver include SG: ....
    Local receiver exclude SG: ....
    Joins RP:                  ....
    Joins WC:                  ....
    Joins SG:                  .OO.
    Join state:                .OO.
    Prune state:               ....
    Prune pending state:       ....
    I am assert winner state:  ....
    I am assert loser state:   ....
    Assert winner WC:          ....
    Assert winner SG:          ....
    Assert lost WC:            ....
    Assert lost SG:            ....
    Assert lost SG_RPT:        ....
    Assert tracking SG:        OOO.
    Could assert WC:           ....
    Could assert SG:           .OO.
    I am DR:                   O..O
    Immediate olist RP:        ....
    Immediate olist WC:        ....
    Immediate olist SG:        .OO.
    Inherited olist SG:        .OO.
    Inherited olist SG_RPT:    ....
    PIM include WC:            ....
    PIM include SG:            ....
    PIM exclude SG:            ....
root@e1$ show pim mfc  
Group           Source          RP             
239.0.0.11      192.168.1.1     192.168.101.133
    Incoming interface :      eth0
    Outgoing interfaces:      .OO.
root@e1$ exit
[Connection to XORP closed]
root@e1# ip mroute
(192.168.1.1, 239.0.0.11)        Iif: eth0       Oifs: eth1 eth2

Setting up VXLAN Once IP multicast is running, setting up VXLAN is quite easy. Here are the software requirements:
  • A recent kernel. Pick at least 3.7-rc3. You need to enable CONFIG_VXLAN option. You also currently need a patch on top of it to be able to specify a TTL greater than 1 for multicast packets.
  • A recent version of ip. Currently, you need the version from git.
On R1, R2 and R3, we create a vxlan42 interface with the following commands:
root@rX# ./ip link add vxlan42 type vxlan id 42 \
>                               group 239.0.0.42 \
>                               ttl 10 dev eth0
root@rX# ip link set up dev vxlan42
root@rX# ./ip -d link show vxlan42
10: vxlan42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc noqueue state UNKNOWN mode DEFAULT 
link/ether 3e:09:1c:e1:09:2e brd ff:ff:ff:ff:ff:ff
vxlan id 42 group 239.0.0.42 dev eth0 port 32768 61000 ttl 10 ageing 300
Let s assign an IP in 192.168.99.0/24 for each router and check they can ping each other:
root@r1# ip addr add 192.168.99.1/24 dev vxlan42
root@r2# ip addr add 192.168.99.2/24 dev vxlan42
root@r3# ip addr add 192.168.99.3/24 dev vxlan42
root@r1# ping 192.168.99.2                    
PING 192.168.99.2 (192.168.99.2) 56(84) bytes of data.
64 bytes from 192.168.99.2: icmp_req=1 ttl=64 time=3.90 ms
64 bytes from 192.168.99.2: icmp_req=2 ttl=64 time=1.38 ms
64 bytes from 192.168.99.2: icmp_req=3 ttl=64 time=1.82 ms
--- 192.168.99.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 1.389/2.375/3.907/1.098 ms
We can check the packets are encapsulated:
root@r1# tcpdump -pni eth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:30:36.561185 IP 192.168.1.1.43349 > 192.168.2.2.8472: UDP, length 106
11:30:36.563179 IP 192.168.2.2.33894 > 192.168.1.1.8472: UDP, length 106
11:30:37.562677 IP 192.168.1.1.43349 > 192.168.2.2.8472: UDP, length 106
11:30:37.564316 IP 192.168.2.2.33894 > 192.168.1.1.8472: UDP, length 106
Moreover, if we send broadcast packets (with ping -b or ARP requests), they are encapsulated into multicast packets:
root@r1# tcpdump -pni eth0
11:31:27.464198 IP 192.168.1.1.41958 > 239.0.0.42.8472: UDP, length 106
11:31:28.463584 IP 192.168.1.1.41958 > 239.0.0.42.8472: UDP, length 106
Recent versions of iproute also comes with bridge, an utility allowing one to inspect the FDB of bridge-like interfaces:
root@r1# ../bridge/bridge fdb show vxlan42
3e:09:1c:e1:09:2e dev vxlan42 dst 192.168.2.2 self 
0e:98:40:c6:58:10 dev vxlan42 dst 192.168.3.3 self

Demo For a demo, have a look at the following video (it is also available as an Ogg Theora video).
<iframe frameborder="0" height="270" src="http://www.dailymotion.com/embed/video/xusell" width="480"></iframe>

  1. The decision is usually done when the bandwidth used by the follow reachs some threshold. With XORP, this can be controlled with switch-to-spt-threshold. However, I was unable to make this works as expected. XORP never sends the appropriate PIM packets to make the switch. Therefore, for this lab, it has been configured to switch to native multicast at the first received packet.

Erich Schubert: DBSCAN and OPTICS clustering

DBSCAN [wikipedia] and OPTICS [wikipedia] are two of the most well-known density based clustering algorithms. You can read more on them in the Wikipedia articles linked above.
An interesting property of density based clustering is that these algorithms do not assume clusters to have a particular shape. Furthermore, the algorithms allow "noise" objects that do not belong to any of the clusters. K-means for examples partitions the data space in Voronoi cells (some people claim it produces spherical clusters - that is incorrect). See Wikipedia for the true shape of K-means clusters and an example that canot be clustered by K-means. Internal measures for cluster evaluation also usually assume the clusters to be well-separated spheres (and do not allow noise/outlier objects) - not surprisingly, as we tend to experiment with artificial data generated by a number of Gaussian distributions.
The key parameter to DBSCAN and OPTICS is the "minPts" parameter. It roughly controls the minimum size of a cluster. If you set it too low, everything will become clusters (OPTICS with minPts=2 degenerates to a type of single link clustering). If you set it too high, at some point there won't be any clusters anymore, only noise. However, the parameter usually is not hard to choose. If you for example expect clusters to typically have 100 objects, I'd start with a value of 10 or 20. If your clusters are expected to have 10000 objects, then maybe start experimenting with 500.
The more difficult parameter for DBSCAN is the radius. In some cases, it will be very obvious. Say you are clustering users on a map. Then you might know that a good radius is 1 km. Or 10 km. Whatever makes sense for your particular application. In other cases, the parameter will not be obvious, or you might need multiple values. That is when OPTICS comes into play.
OPTICS is based on a very clever idea: instead of fixing MinPts and the Radius, we only fix minpts, and plot the radius at which an object would be considered dense by DBSCAN. In order to sort the objects on this plot, we process them in a priority heap, so that nearby objects are nearby in the plot. this image on Wikipedia shows an example for such a plot.
OPTICS comes at a cost compared to DBSCAN. Largely because of the priority heap, but also as the nearest neighbor queries are more complicated than the radius queries of DBSCAN. So it will be slower, but you no longer need to set the parameter epsilon. However, OPTICS won't produce a strict partitioning. Primarily it produces this plot, and in many situations you will actually want to visually inspect the plot. There are some methods to extract a hierarchical partitioning out of this plot, based on detecting "steep" areas.
The open source ELKI data mining framework (package "elki" in Debian and Ubuntu) has a very fast and flexible implementation of both algorithms. I've benchmarked this against GNU R ("fpc" package") and Weka, and the difference is enormous. ELKI without index support runs in roughly 11 minutes, with index down to 2 minutes for DBSCAN and 3 minutes for OPTICS. Weka takes 11 hours and GNU R/fpc takes 100 minutes (DBSCAN, no OPTICS available). And the implementation of OPTICS in Weka is not even complete (it does not support proper cluster extraction from the plot). Many of the other OPTICS implementations you can find with Google (e.g. in Python or MATLAB) seem to be based on this Weka version ...
ELKI is open source. So if you want to peek at the code, here are direct links: DBSCAN.java, OPTICS.java.
Some part of the code may be a bit confusing at first. The "Parameterizer" classes serve the purpose of allowing automatic UI generation, for example. So there is quite a bit of meta code involved.
Plus, ELKI is quite extensively optimized. For example, it does not use Java Collections much anymore. Java Iterators, for example, require returning an object on next();. The C++ style iterators used by ELKI can have multiple values, and primitive values.
for(DBIDIter id = relation.iterDBIDs(); id.valid(); id.advance())
is a typical for loop in ELKI, iterating over all objects of a relation, but the whole loop requires creating (and GC'ing) a single object. And actually, this is as literal as a for loop can get.
ModifiableDBIDs processedIDs = DBIDUtil.newHashSet(size);
is another example. Essentially, this is like a HashSet<DBID>. Except that it is a lot faster, because the object IDs do not need to live a Java objects, but can internally be stored more efficiently (the only currently available implementation of the DBID layer uses primitive integers).
Java advocates always accuse you of premature optimization when you avoid creating objects for primitives. Yet, in all my benchmarking, I have seen this continuously to have a major impact how many objects you allocate. At least when it is inside a loop that is heavily used. Java collections with boxed primitives just eat a lot of memory, and the memory management overhead does often make a huge difference. Which is why libraries such as Trove (which ELKI uses a lot) exist. Because memory usage does make a difference.
(Avoiding boxing/unboxing systematically in ELKI yielded approximately a 4x speedup. But obviously, ELKI involves a lot of numerical computations.)

Lisandro Dami&aacute;n Nicanor P&eacute;rez Meyer: Is your KDE slooooow? Check your loopback interface

While digging Debian Qt-KDE's bugs, it has become more common to find people reporting that their KDE is very slow. Most of the time (if not every time) it turns out that the loopback interface is down. Sometimes the user did something wrong, sometimes some network manager, but the end result is the same: KDE is terribly slow.

A user filled a bug upstream to ask for a dialogue in KDE that warns the user about this fact. Let's hope it get's implemented :-)

OK, so, how do I check that my lo interface is up?

You can simply run:

$ /sbin/ifconfig
eth0 Link encap:Ethernet HWaddr 00:1f:c6:ba:70:f3

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:1807 errors:0 dropped:0 overruns:0 frame:0
TX packets:1807 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:494525 (482.9 KiB) TX bytes:494525 (482.9 KiB)

The last entry "lo" tells us that the interface is up.

In Debian, the file /etc/network/interfaces should have something like:

# The loopback network interface
auto lo
iface lo inet loopback

So, next time you find a KDE session is slow, you know what to do ;-)

1 November 2012

Antoine Beaupr : My short computing history

This is a long story. I wanted to document which tools I used on my desktop, but then realized it was difficult to do this without an historical context. This therefore, became a history of my computer usage and programming habits.

Before the desktop: vic 20 and apple My first contacts with computers were when my uncle showed me his Apple computers at his home. He was working at UQAM and had access to all sorts of crazy computers. My memory is vague, but I believe he showed me a turtle moving around the screen, which I suspect was Logo, and some rabbit game which I was quite fond of. Later, I hooked up with a Vic-20 (similar to a Commodore 64) and typed in my first programs. Back then, you would painstakingly transcribe the program from a book into the screen, type RUN at the end and really hope that you didn't make a typo. You could also read and write data off audio tape with an external tape drive, which was really amazing. To put things into perspective, the Vic-20 would plug into your television for a display, and the computer was "in the keyboard". Old school shit. The language was BASIC, which was awful. Later, we got luxurious and had the privilege of a Macintosh Plus computer. 3000$ plus the printer, that was quite something. Still, that modern system could run Word, had games, and a 9" black and white screen. But the best part, and where I really started to learn to program, is with Hypercard, the crazy software written by Bill Atkinson that became the inspiration for wikis, the Myst computer game and, to a certain extent, popularized the concept of hyperlinking, which is the basic principle of the World Wide Web.

The early years: afterstep and FreeBSD But my real computer life all started with FreeBSD 2.2.x (probably 2.2.5), around 1997, when I installed my first FLOSS operating system. This was just amazing: source code for everything, down to the compilers themselves, I was amazed, since I was coming from a Mac (still had that Mac Plus lying around) and Windows (95!) world. Back then, X11 (XFree86, it was called) was a fantasy. Something I would compile with the FreeBSD ports system and sometimes would be able to run. Window managers were some weird things I wasn't sure how to use, but I figured out twm (yuck!) but was still amazed. At the university, we had SGI workstations (remember the Indigo machines?) on which I started setting up the Afterstep window manager, which I used for a long time. The web browser was Netscape Communicator 4, and lo and behold, it also did email, although I mostly used Pine over a modem terminal to the university. Internet access at home was spotty and was dependent on students compiling discreetly copies of slirp which were allowing us, in a really clunky way, to run a PPP session over those terminal lines. That actually got my account suspended because I messed up my sendmail configuration and was sending emails that appeared to be coming as , which freaked out the sysadmins that be over there. The hardware was a Pentium 166Mhz with 32MB of ram and 20GB of disk space, which was a huge improvement over the 1MB of ram and 20MB of (external!) hard disk that the Mac Plus was proud of. By then, I started programming in more languages: I learned the basics of shell scripting, C programming and I had learned Pascal in a college class.

The real freedom: Debian and... enlightenment? I then switched to Debian, thanks to my friend scyrma, which showed me the power of dselect which, for a FreeBSD user, was really impressive. Imagine that: you could just install software and all dependencies without compiling anything! ;) I don't exactly remember when that was, but it was probably Woody. I probably still used afterstep, but at that time, changing window managers was like changing t-shirts and we would waste countless hours fiddling around with borders and background images and other silly things. I remember running Enlightenment and writing software for it, for example, but not much else. Back then, email was still over pine, but a new web browser was rearing its ugly head, Mozilla, and was quickly trimmed down to Phoenix, shortly lived as Mozilla Firebird, only to be finally renamed Mozilla Firefox, and was unleashed as a stable release to an unsuspecting Internet Explorer and Microsoft-dominated world. Throughout this chaos, I also used various Mozilla derivatives like Galeon since all of this was pretty unstable and heavy for my poor computers. This was probably around 2002 to 2005. By then, I had formal classes on programming languages, was forced to learn Java, discovered C++, and learned more C as the university assumed that you knew that language in the data structures and algorithm class, which some students really didn't find funny at all. I also learned some Prolog and Haskell back then. Through my experiences with free software, I discovered various eclectic programming languages like TCL, and kept an eye out for other languages like lisp (thanks to Emacs), Forth (thanks to the FreeBSD boot loader), HTML/CSS/Javascript (no thanks to the web). There are probably others I am forgetting here, voluntarily or not.

The years of the desktops: rox, sawfish and gnome Another stable increment on my desktop was the use of the sawfish window manager and the rox desktop, mostly because of the awesome Rox file manager that had no equal in the rest of the free software world. Gnome was an atrocity and KDE even more so: bloated and unstable. Eventually, however, KDE and Gnome not only matured, but simply took over the whole of the graphical environment psyche of the free world. People nowadays don't even know what a window manager is and assume that you now have a "desktop environment", which, for me, is a real desk with a bunch of papers, a radio transceiver and maybe a computer with a screen lying around. So I did use KDE, and then Gnome. I switched from KDE to Gnome because I felt that Gnome left me more space to run my own stuff without starting crazy daemons to preload everything. At this point, Firefox was the new thing and basically stable enough for daily use. This lasted until around 2009, I guess. On the programming side, I had picked up Perl because I had to learn it to work at Ericsson (don't ask) and learned PHP to pay my rent, which I consider one of the biggest professional mistakes I have made so far in that story. It is around that time that Koumbit was founded by me and other friends.

The tiles fall on the gnomes: awesome desktop and sysadmin programming At some point, friends (mostly comar and millette) introduced me to wmii as the best thing since sliced bread. I banged my head against it with great pains, but I really enjoyed the hands off approach to window management. Tiling window managers really got a hold on me there, and in fact we can say on the whole desktop, with recent Gnome 3 taking up some tiling concepts itself. I have since then switched to the Awesome Window Manager which is probably the worst name you can use for software (try googling for "Awesome crash" to get an idea). I have switched from Firefox to Chromium, after having repeated crashes from Firefox on a site that Chrome would load fine. I was expecting Chrome to crash only one tab, because of its design, and it turned out to not crash at all, and I have since given up on Firefox totally. I have also given up on desktop environments. While I have given them a chance in quick setups and still use them when forced to, I find them too bloated and single-minded for my needs. I want control over my desktop, and those tools work very hard to remove control and hide it behind user interfaces that treat me like a brain-dead dog. Finally, I have kept on programming in PHP for work, because that's what we do over there (unfortunately). Mostly because of Drupal and Aegir, in which I was and still am deeply involved. But I am quite fond of Python for my personnal project and still use shell scripting to get myself out of trouble or automate data processing from time to time. I have a rule though: once a program hits 100 lines of shell scripting, it needs to be rewritten, usually in Perl because the syntax is closer. During the years 2009 to 2012, I have started to work more and more on system administration however. In 2010, I became a Debian Developer and a lot of my work at Koumbit revolves around system administration, optimisation and maintenance. I work oncall and sometimes get called during the night to nurse servers (which is why I am up writing this right now).

The future of programming I have become seriously disgruntled with programming. I have seen the emergence of the web, where programmers felt it was relevant to rewrite 30 years of programs into the web framework (webmail? wtf?), and now we are seeing the same thing happen with mobile devices, the tracking devices they call phones and "pads", we should rather say. And again they are trying to make people pay for trivial software. I am tired of seeing our community reinvent the wheel every 10 years because of infatuated egos and marketing sprints. We are wasting great minds on this, a lot of energy and precious time. I would rather maintain 5 year old imperfect software than coordinate a 5 year project to rewrite that software into another imperfect software. That is what is happening right now, and it's going nowhere. So here's the current state of affairs, some perls, some tiles, but mostly system administration and less programming. Through all this time, the common pattern is that I have learned to solve problems and document my work, which are key concepts of the job of a system administrator, which is what I really am now. Somehow, that popped into my life like a natural fit. Maybe, one day, I will retire and become a shepherd in far away mountains, but I will always have in my mind the strong desire to fix shit up and make things right with the world. Maybe get involved in a project like Open Source Ecology and especially their Global Village Construction Set to make sure humankind survives all that stupidity.

31 October 2012

Mirco Bauer: Mono 3.0 Preview Packages for Debian and Ubuntu

Just after the Mono 3.0 release was announced I started working on Debian packages and after a couple of days of hard work I already have some preview packages for you. The preview packages are currently limited to the AMD64 architecture but can be installed on Debian/Testing (Wheezy), Debian/Unstable (Sid) and Ubuntu/Quantal (12.10). After these preview packages have received some successful feedback (please leave feedback via comments!), I will upload them to Debian/Experimental. In case you are shocked why I have packages in just 8 days after the official Mono 3.0 release, there are 3 factors that play a big role:
  1. New packaging workflow
    • Downstream patches are management in git branches
    • Release Early, Release Often: I no longer port and test every patch very carefully (especially with architecture specific patches), instead I port them with my best knowledge and upload them to debian.meebey.net and/or Debian/Experimental and use that as test ground. If regressions happen, let them happen, and make further uploads to address them.
  2. New packaging style
  3. Sponsored packaging
    • As my employer needs Mono packages for an upcoming software project, I was granted to work on Mono packages in worktime.
Install instructions OBLIGATORY WARNING: DO NOT USE THESE PREVIEW PACKAGES ON PRODUCTION SYSTEMS NOR BLAME ME IF YOUR MONO SOFTWARE NO LONGER WORKS! Add this line to your /etc/apt/sources.list file:
deb http://debian.meebey.net/experimental/mono /
Now update the APT database and install mono-complete from that repository:
apt-get update
apt-get install mono-complete

Raphael Geissert: rsync is not enough

Ask how can one create a Debian mirror and you will get a dozen different responses. Those who are used to mirroring content, whether it is distribution packages, software in general, documents, etc., will usually come up with one answer: rsync.

Truth is: for Debian's archive, rsync is not enough.

Due to the design of the archive, a single call to rsync leaves the mirror in an inconsistent state during most of the sync process.
Explanation is simple: index files are stored in dists/, package files are stored in pool/. Sort those directories by name (just like rsync does) and you get that the indexes will be updated before the actual packages are downloaded.

There are multiple scripts out there that do exactly that, one of them in Ubuntu's wiki. Plenty more if you search the web.

Now, addressing that issue shouldn't be so difficult, right? after all, all the index files are in dists/, so syncing in two stages should be enough. It's not that simple.

With the dists/ directory containing over 8.5GBs worth of indexes and, erm, installer files, even a two stages sync will usually leave the mirror in an inconsistent state for a while.

How about only deferring to the second stage the bare minimum?, I hear you ask.
That is the current approach, but it leads to some errors when new index files are added and used. The fact that people insist in writing their own scripts doesn't help.

Hopefully, some ideas like moving the installer stuff out of dists/ and
overhauling the repository layout are being considered. An alternative is to make the users of the mirrors more robust and fault-tolerant, but we would be talking about tenths if not hundreds of tools that would need to be improved.

In all cases, the one script that is actively maintained, is rather portable, and improved from time to time is the ftpsync script. Please, do yourself and your users a favour: don't attempt to reinvent the wheel (and forget about calling rsync just once).

Russell Coker: Links October 2012

The F Word has an informative post about men commenting on Feminist blogs [1]. Most of it applies to any situation where a member of a powerful group comments on an issue related to a minority group. Near the end they say: I ll also paraphrase and flesh out the most useful piece of advice I ever read, when I made the effort to research white privilege: don t expect the minority to trust you. Trust is earned, and you re just another commenter who they can t tell apart from any other commenter. You re entering someone else s space, where different rules apply. You get to have the rest of the world for people to assume you re a wonderful person. Here, you re just another one of them , and given the track record of them , it s up to you to listen, learn and prove that you re being thoughtful and honestly trying to examine your privilege. Sociological Images has an interesting article on people s perception of the sky colour it seems that the blue sky meme is modern [2]. Charles Stross wrote an interesting article about the possible uses for future low power computers [3]. He gets a bit over-excited about the possibilities for sensing making a tiny computer that can sense so many things isn t going to be easy (DSLRs are big for a reason). But having lots of powerful computers everywhere does provide lots of interesting and potentially bad opportunities. Martin Bekkelund has an interesting article about Amazon wiping DRM infected books that it had sold to a customer without giving a refund or an explanation [4]. If you want to buy ebooks it seems that the sensible thing to do would be to immediately crack them or download them from The Pirate Bay so Amazon can t steal them back. Rebecca Saxe gave an interesting talk about trying to develop methods for conflict resolution through neuroscience [5]. Barry Eisler has written an interesting article about the corruption of journalists [6]. It s really worth reading, some of the methods of corruption apply to even the more casual bloggers. Krebs on Security has an informative article about the Microsoft Tech Support phone scams [7]

30 October 2012

Matthew Garrett: A detailed technical description of Shim

Shim is the first stage bootloader we've been implementing for supporting Secure Boot. It's called Shim because it serves as an object to fit the gap between the Microsoft trust root and our own trust root. As originally envisaged it would do nothing other than load and execute appropriately signed binaries, but it's got a little more complicated than that now. It is, however, basically feature complete at this point - I don't expect it to grow significantly further.

(For those of you following along at home, the Shim source code I'm talking about is here)

First, it checks whether the user has disabled local signature verification in check_mok_sb(). This is done by reading the MokSBState UEFI variable and verifying that it has the correct variable attributes, ensuring that it was created in the trusted boot services environment. If so, and if it's set appropriately, signature verification is disabled for the rest of Shim. Shim then pauses for a couple of seconds while displaying a message indicating that it's running in insecure mode.

Next, Shim checks whether the user has set any variables that indicate that a Mok request has been made in check_mok_request(). If so, it launches MokManager. The mechanism by which this is done is described later.

Shim then copies the Mok key database into a runtime variable, which makes it accessible to the kernel (mirror_mok_list()). The kernel can then use the Mok keys when performing module signature verification. Next, as the last step before launching the next stage bootloader, Shim installs a UEFI protocol that permits other applications to call back into Shim. We're now ready to branch off from Shim into the actual bootloader.

To do this, Shim has to do a few things. First, we need to get the bootloader off disk. This involves finding Shim's path, which we can do by opening the loaded image protocol on Shim's image handle, turning that into a string, walking backwards along it until we find a directory separator, stripping the filename and appending the path to the second stage bootloader. This is then turned back into a binary device path, all of this happening in generate_path(). The binary is then read off disk in load_image(), using the simple filesystem protocol. We open the device, open a reference to the file, check how large the file is, allocate a buffer to hold the file and then read it off disk into that buffer. Everything so far has been fairly straightforward, but now things get more complicated.

handle_image() is the real meat of Shim. First it has to examine the header data in read_header(), copying the relevant bits into a context structure that will be used later. Some basic sanity checks on the binary are also performed here. If we're running in secure mode (ie, Secure Boot is enabled and we haven't been toggled into insecure mode) we then need to verify that the binary matches the signature and hasn't been blacklisted.

verify_buffer() has to implement the Microsoft Authenticode specification, which details which sections of the binary have to be hashed and in which order. The actual hashing is done in generate_hash(), which calls into the crypto code to add each binary section to the hashed data. The comments in generate_hash() describe what's going on there clearly enough. The crypto code itself is a copy of the crypto library extracted from the Tiano source tree. It's a relatively thin layer on top of OpenSSL, so the code's been fairly well tested.

Once we have the hash, we need to verify the signature. First, we ensure that the Mok database hasn't been modified from the OS (verify_mok()). Next, we check whether the binary's hash or signature have been blacklisted (check_blacklist()). This is done for each of the SHA1 and SHA256 hashes of the binary (the two hashes implemented in Tiano) and the certificate. If any of these match an entry in the dbx variable or a built-in blacklist, Shim will refuse to run them. Next, we simply do the same for the whitelist. If the binary's hash is either in the db variable or the Mok list, or if it's signed with a certificate that chains back to an entry in either db or the Mok list, we'll run it. Finally, the signature is checked against any built-in keys. Certificate verification is carried out with the AuthenticodeVerify() function, which again comes from Tiano's crypto library and is mostly implemented in OpenSSL.

If the binary passed signature validation we return to handle_image() and now load the binary's sections into their desired addresses. That buffer is handed to relocate_coff() which runs through the binary's relocation entries and fixes them up. There's one last subtlety, which is that because we've been fixing this up by hand, the image handle still refers to the shim binary. We fix the ImageBase and ImageSize values in the loaded image protocol to match the newly relocated binary, which is vital because otherwise grub is unable to find its built in modules. Finally, we jump into the binary's entry point. What the binary does next is up to it - if it's a bootloader, there's a good chance that it'll simply launch an OS and we'll never return. If it does return, we restore ImageBase and ImageSize, uninstall the protocol we installed and exit ourselves. If Shim exits (either because the second stage bootloader wasn't present, didn't verify, or exited) then control will simply pass on to the next boot option in the UEFI priority list. If there's nothing else, the platform will do something platform defined.

And that's how Shim works.

comment count unavailable comments

Matthew Garrett: An overview of our Secure Boot implementation

I've written rather a lot about how we're implementing our Secure Boot support, but there's been a range of subtle changes over time. Here's an overview of how it all fits together.Secure Boot and signingSecure Boot is part of the UEFI specification. When enabled, systems will refuse to boot any UEFI executables (such as an OS bootloader) unless they've been signed by a trusted key. The only trusted key effectively guaranteed to be present is Microsoft's, and Microsoft will sign binaries if you have access to the Windows hardware dev center. You gain access to this by purchasing a certificate from Verisign that verifies your identity and then use that to create a developer account at the Sysdev website. Once you've agreed to the legal contracts, you can submit a UEFI executable and Microsoft will then send you back a signed one. The signature is added to the end of the binary, so it's trivial to verify that your executable hasn't otherwise been modified in the process.
Avoiding signing round tripsWe didn't want to have to get binaries re-signed every time we updated our bootloader or kernel, so we came up with a compromise approach. We implemented a shim bootloader (cunningly called "Shim") which is signed by Microsoft and includes a copy of a public key that's under our control. Shim will launch any binary signed with either a key installed in the system firmware or the public key built into it. This allows us to build and sign all other binaries ourselves.
Preventing Secure Boot circumventionSecure Boot is intended to prevent scenarios where untrusted code can be run before an OS kernel. Locking down the boot environment avoids the simple case of using a modified bootloader, and requiring that the kernel also be signed avoids simply booting arbitrary attack code that looks roughly like a kernel but does nothing to prevent the more complex case of using a running kernel to launch another kernel. To do that we've had to restrict the behaviour of the kernel in Secure Boot environments. Direct hardware access from userspace has been blocked, and modules must be signed by a key the kernel trusts.
Providing user control over trustMicrosoft requires that it be possible to modify the key database on all Windows-certified x86 systems, but the UI to do so may be inconsistent and awkward. Suse developed a plan to avoid this, involving adding an additional key database to the system. The user enrols a key from the running OS and enters a password. On the next reboot, shim detects that a key is awaiting enrolment and drops to a prompt. If the user does nothing or just hits enter, the system continues to boot. If the user chooses to enrol a new key, the user is asked to re-enter the password in order to confirm that they're the same user that initiated the original request. If successful, the key is then copied into a boot services variable which cannot be directly modified from the OS. Anything signed with the user's key will then be trusted.
Providing user control over signature verificationSome users (such as kernel developers) may be hampered by Secure Boot - forcing them to sign every new kernel they build is time consuming and unhelpful. Windows-certified x86 hardware will be required to provide an option to disable Secure Boot, but again that UI may be inconsistent. We've added an option where a user can generate a request to disable signature validation, again requiring a password. On next boot the user must explicitly choose the "Toggle signature validation" menu item and will then be requested to enter three randomly chosen characters from their password. After confirmation, signature validation in Shim will then be disabled.
Supporting other distributionsNot all distributions will want (or be able) to sign via Microsoft. They'll be able to take any other signed version of Shim and put it on their boot media. If their second stage bootloader is signed by a key that Shim doesn't trust, Shim will pop up a menu permitting the user to choose to enrol a new key off the install media. The user can select this, navigate a file browser and select the key. After confirmation, the key will be added to Shim's trusted key list. The user can then install the distribution.
Third party modulesThere's two approaches here. The first is for the third party to provide a module key that the user can install into Shim's database. The kernel will import this at boot time and trust any modules signed with it. The second is for the third party to provide a key signed by Microsoft. David Howells has come up with with the idea of utilising this approach of embedding a public key in a pe binary, with the kernel then having support for importing this key, noticing that it's signed by a trusted key and adding it to the kernel keyring.
How about ARM?We're not planning on supporting Secure Boot on ARM, for two reasons. The first is that Microsoft require that it be impossible to disable Secure Boot on ARM, and we don't want to support systems unless it's guaranteed that the user will be able to run what they want. The second is that there's no strong expectation that the signing key used for signing third party UEFI binaries will be present on ARM systems.
What are other distributions doing?As far as I know, Suse and Fedora will be shipping the same code. Ubuntu is shipping an older version of Shim but should pick up the local key management code in the next release. The only significant difference is that Ubuntu doesn't require that kernel modules be signed.

comment count unavailable comments

29 October 2012

Matthew Garrett: Ted Ts'o is a rape apologist and why this matters

(This post contains some discussion of rape and sexual assault but does not go into any specifics)

There was a brief controversy at Linux.conf.au back in 2011. The final keynote speaker gave a compelling presentation on online privacy, including some slides containing sexualised imagery. This was against the terms of the conference policies, and resulted in an apology from the conference organisers and the speaker. The situation was unfortunate but well handled, and that should have been the end of it.

Afterwards, there was some pushback on the conference mailing list. Concerns were raised about the policy being overly restrictive and the potential for it to be used to stifle speech that influential groups disagreed with. I don't agree with these arguments, but discussion of why policies have been implemented is completely natural and provides an opportunity for a community to determine what its expected standards are.

And then Ted Ts'o effectively called rape victims liars[1]. At first I assumed that this was just some sort of horrific failure to understand the implications of what he was saying, so I emailed him to check. The reply I got drew a pretty clear distinction between the case of a drunk college student raping another drunk college student in their room and the case of knifepoint rape in a dark park. You know, the difference between accidental rape and rape rape. The difference between the one any of us might have done and the one that only bad people do. Legitimate rape and the "rape" that those feminists talk about. The distinction that lets rapists convince themselves that they didn't really rape anyone because they weren't holding a knife at the time.

Ted Ts'o argues that only a small percentage of rape really counts as what people think of as rape. Ted Ts'o is a rape apologist.

There's an ongoing scandal in the UK at the moment. A well known DJ, Jimmy Savile, died last year. He grew up in a working class family, but through hard work and natural talent was one of the most significant figures in promoting pop music in the UK in the 50s and 60s, and worked in various parts of the BBC for the best part of 30 years. He spent significant amounts of time raising money for charity, and it's estimated that he raised over 40 million for various causes. Since his death, around 300 people have accused him of sexually abusing them. The BBC is desperately trying to explain why it cancelled an expose shortly before it aired. Multiple people who worked there at the time claim that everyone knew he was involved in indecent activities, but saying anything would risk both their career and the charities that depended on his fundraising. Nobody said anything, and he was allegedly free to continue his abuse.

Ted Ts'o is a significant figure in the Linux kernel community. He has expressed abhorrent beliefs that damage that community. Condemnation was limited to a mailing list with limited readership, meaning, effectively, that nobody said anything. Last week the Ada Initiative published a blog post pointing out the damage that did, and I realised that my effective silence was not only helping to alienate 50% of the population from involving themselves with Linux, it was also implicitly supporting my community leadership. I was giving the impression that I was basically fine with our community leaders telling people that it wasn't really rape if you were both drunk enough. I was increasing the chances of members of our community being sexually assaulted. Silence is endorsement. Saying nothing is not ok.

In the absence of an apology and explanation from Ted, I'll be interacting with him to the bare minimum that I'm compelled to as a result of my job. I won't be attending any Linux Foundation events he's involved in organising. If I'm running any events, I won't be inviting him. At a time when we're finally making progress in making our community more open and supportive, we don't need leaders who undermine that work. Support organisations who encourage that progress, not the people who help drag us back.

Footnotes

[1]The original archive has vanished. I've put up a copy of the relevant thread here. Throughout, Ted states that he's actually arguing against the idea that women need to be frightened of sexual assault, and not against the definition of rape. Except saying things like This one does a pretty good job of taking apart the Koss / Ms. Magazine study, which is the source for the "1 in 4" number. For example, it points out that over half of those cases were ones where undergraduates were plied with alcohol, and did not otherwise involve using physical force or other forms of coercion is difficult to read in any way other than "Half of the people you're counting as having been raped haven't really been raped", and favourably referring to an article that asserts that the rate of false rape reports is probably close to 50% is pretty strong support for the idea that many rape victims are liars.

(Update 2012/10/30: Adam Williamson suggests in this comment that this mail is a better example of Ted's behaviour - there's some explicit victim blaming and a lot of "Is that rape" questioning with the obvious implication that the answer should be "no". Ted Ts'o is a victim blaming rape apologist.)

(Update 2012/11/05: It's been suggested that I haven't been sufficiently clear about which of Ted's statements justify my claims. So, here we go.

In this mail, Ted links to and endorses this article. He explicitly links to it because of its treatment of rape statistics. Quoting directly from that article:
the rate of false reports is at least 9 percent and probably closer to 50 percent
Ted explicitly endorses an article that claims that a significant percentage of reported rapes are false. The study that generated that figure is held in poor regard by other researchers in the field - Australian police figures indicate that 2.1% of rape accusations were classified as false. Ted asserts that he was trying to argue against poor use of statistics, so it's a fair assumption that he agrees with the alternative statistics that he's citing. Ted believes that many rape victims are making false accusations. Ted believes that many rape victims are liars.

Again in this mail, Ted argues against a claimed figure that 1 in 4 women have been sexually assaulted. One of his arguments is that Also found in the Koss study, although not widely reported, was the statistic that of the women whom she classified as being raped (although 73% refused to self-classify the event as rape), 46% of them had subsequent sex with the reported assailant. Ted disagrees with a statistic because some rape victims subsequently have sex with the reported assailant. This means that Ted believes that this indicates that they were not really raped. Ted is a rape apologist.)

comment count unavailable comments

Next.

Previous.