Search Results: "ed"

5 November 2012

Joachim Breitner: Somewhere in Israel, someone printed 50 copies

Today, a small package was waiting for me at my door step, containing 52 printed and stapled copies of a paper that I wrote a while ago. To be more precise, two years ago, when I submitted the main results of my math Diploma Thesis to the Israel Journal of Mathematics. Next year at christmas eve, it was accepted and published online. And now, it seems, it finally went into print. So my paper now has page numbers and I have this pile of printouts that outnumbers the set of possibly interested readers by four dozens or more, and they would probably just download the online version instead of asking for this print out. This seems all very anachronistic to me.

Vincent Sanders: Another NetSurf Developer Workshop

The NetSurf Developers - Bringing you alternative internet meme delivery since 2002
The weekend saw some of the NetSurf developers gather once again to do battle against the Red Queen...no, wait. I mean plan and write code.

The successful event was once again kindly hosted by Collabora at their Cambridge office, the facilities remain as good as at the previous event and provided a comfortable environment in which to work.

Four developers managed to attend from around the UK John-Mark Bell, Vincent Sanders, Daniel Silverstone and Rob Kendrick. The turnout was a little lower than previous events, however Michael Drake managed to assist with some bug fixes remotely.

From Friday to Sunday afternoon we each managed to put in 24 hours of work consisting of almost 90 commits changing over 100 files and removing 10,000 lines of code.

Whiteboard with the 3.0 release tasks
The main theme of the event was consolidation ready for the 3.0 release. While not a glamorous target this is a major release for the project which we want to achieve soon, this requires the completion of numerous ongoing tasks (like finally removing the dependency of libxml) and fixing as many reported bugs as possible.

The Major things we achieved were:

Once the final usage of libXML is removed (from the hotlist code) NetSurf will become substantially smaller on several platforms as we will not have two separate parser implementations linked.

One big feature of the 3.0 release which was not originally on the roadmap is that of JavaScript. We initially intend to provide two builds of with and without the JavaScript interpreter compiled into NetSurf. This allows those on very small systems to continue to run NetSurf and those users who want a richer web experience to enjoy it.

Although JavaScript is a major new feature which is frequently requested the initial implementation will be incomplete. To complete the feature several thousand JavaScript to DOM interface binding functions must still be written. In addition it should be noted that we have yet to add dynamic layout or extensions such as Canvas or WebGL

We had a Raspberry Pi which was used to run RISC OS so we could examine the operation on this system.

The 2.9 edition as shipped works OK but is lacking the substantial progress made in the last six months.

The CI system build was run on the system and works well so we are reassured the 3.0 edition will provide an even better browsing experience.

The event was a great success overall and we are now very close to our 3.0 release which we hope to produce before Christmas.

Michal Čihař: Weblate 1.3 on l10n.cihar.com

As Weblate release is getting closer, I've decided to give new version more testing in real life and it got deployed at http://l10n.cihar.com/. This release brings quite a lot of new features, most interesting for users might be: You can see full list of changes in our documentation. As usual, I hope this upgrade will go smoothly and won't cause any big problems :-).

Filed under: English Phpmyadmin Website 0 comments Flattr this!

MJ Ray: Seize the Media! @theBoyler @coopsutd

image

1 Angel Square

So when I arrived on Thursday (ten minutes early despite a cancelled train, thanks to help from the cooperative fellow traveller mentioned last post), I was in time to go to a workshop on media cooperatives led by @theBoyler. It wasn t what I expected. It seemed to concentrate on the opportunity presented by the current awful state of news media companies in many countries and the technology-driven changes to their businesses. For example, WordPress is a viable way to start online first, then move into print later. There are already news co-ops in places where there would be no local news media. The challenge well be how to overcome what I d call spoiling or scorched earth tactics from departing media companies. Some local news media owners are looking for a way out. One obvious one is to turn a local title into a minor variation on a regional-or-worse service or publication, maybe by selling it to a bigger media business. That loses some audience but many will put up with it, so it avoids creating an obvious gap in the market for more truly local reporting. Much else was covered and the audience suggested tons of examples. I m not sure we reached many firm conclusions, but there seem quite a few examples to learn from (Morning Star, taz, ) and Dave Boyle s work is continuing

4 November 2012

Wouter Verhelst: Windows usability: followup.

Yesterday, I posted a blog post containing a list of issues with Windows, in response to a similar list for Linux, which has received quite some response; I've seen people +1 it on google+, and I've received a number of replies on the blog post itself. One of these replies was by the author of the original list; and as expected, he totally and utterly failed to see the point. He also has the audacity to claim I haven't installed Linux yet what?!? The last time I used anything other than Linux on my personal machine for any real work was on Windows 98. And no, that was not because I was too cheap to buy the latest and greatest. I have used Windows in some cases since then, but never on my own hardware. First of all, I should note that the list which I posted was only semi-serious (with tongue in cheek); the fact that it refers to Windows as the operating system which "attempts to replace Linux as the major operating system" should have been a pretty big clue to that effect. Like the original list that it was a reply to, my list contains things that are only true in a particular context (e.g., the one about requiring recent hardware), it contains things that don't matter for the large majority of Windows users (e.g., the point about focus-follows-mouse) but that are major arguments for why I don't use Windows, it contains things that are so alien to what most of us are used to that even most Linux users don't use it, while on the other hand some swear by it (like the SELinux argument), and it contains some things that a seasoned Windows user will call bullshit on but that are real actual problems for the non-seasoned Windows user that I am (such as the mouse argument). It also contains some actual facts, like the statement about most crashes being a direct result of faulty drivers that's pulled directly from a scientific study on the subject conducted by Microsoft, together with some independent researchers. Like the original list, I don't expect this list to change anything for the better, precisely because it is flawed in exactly the same way. But whether the items on that list were right or wrong isn't the point. This is: Nobody (except nutcases) ever said Linux is perfect, or that it's useful for everyone; and while the issues Artem lists may be bugs or may be design flaws or any of a number of other things that could be real problems, the fact is that creating a long "list of issues" isn't going to solve anything. If you find that there are problems with Linux, you basically have three options: fix the problem, get someone to fix it for you, or stop using Linux. Really. Rather than better or worse, Linux is different. It will have its own set of problems, and it will have its own set of advantages. Some of the design decisions cause problems, but also have serious advantages, and focussing on only these problems (the existence of which I'm not denying) isn't very helpful. A good example is the library model: under Linux, libraries are typically installed in a system-wide location, and use a SONAME which is (part of) the filename of that library and which defines the binary interface exported by that library. In contrary to the situation under Windows, Linux libraries also must use position-independent code. The result is a more robust library model. The downside to that model, however, is that when you install older applications that require older SONAME versions of a particular library, getting those older versions may be a challenge. While in theory, installing an older version of a library is not going to cause any problem if the SONAME is different, in practice you'll often find that you'll need to go to (unsupported) older packages from older versions of the distribution, which may not install due to missing or conflicting dependencies, or which may expect data files in a different format in the same location that newer versions of the same library do, or which may have any number of other issues. When that happens, the solution is to either use an older version of the distribution in a chroot or some such, or to patch and/or compile the needed libraries yourself, and install them alongside the newer version of the same library. That's perfectly possible, after all, precisely because of the SONAME. Another good example is the development model: where Windows has a central authority (Microsoft) which defines what it is to be "Windows" and what is and isn't part of that, the same simply isn't true about Linux. Yes, Linus defines what the official Linux kernel has, but he doesn't have any say about the user interface, the boot process, or any number of other things; other people make decisions there. As a result, in effect, who decides what it means to be a "Linux system" is, really, you, the user. If you believe that the Linux user space is completely wrong, you can just change it to something else entirely; and that's precisely what Google did when they created Android. Since there's no central authority making decisions about what will and won't go into the system, this means that inevitably, at any given time, there will be many competing options for many subsystems. Usually this will be an advantage; e.g., the competition between KDE and Gnome benefits both. Sometimes, however, the competing options are so confusingly interwoven that even those who should know the system inside out can't make heads or tails of it, and debugging issues then becomes quite a challenge. Unfortunately, the Linux sound subsystem is currently an example of the latter. I'm not saying that all the items in Artem's list are invalid; a minority are, in fact, actual problems for a majority of Linux users. Most however, are personal opinions, issues affecting only a small minority of Linux users, or, as above, things where he only talks about the downsides of a design decision that also has major advantages. As an example of something in the second category: it's easy, for example, to list hundreds if not thousands of issues relating to particular pieces of hardware that don't work under Linux, simply because the manufacturer doesn't provide any information, and the people developing drivers for Linux have to reverse engineer how everything works. If you're a bit careful in what you buy, however much like MacOS users need to be careful, too then using Linux isn't an issue. My Lenovo Thinkpad X220, for instance, has no parts that don't work; and this includes things like a docking station, and the fact that my (9-cell) battery life is over fourteen hours, which (I believe) is more than on the same laptop with the same battery under Windows. So how is making a list of issues useful? The answer to that is simply, not at all. If you want to help, don't look for other people's problems; we can all do that. Instead, try to look for solutions to these problems. That's not always easy, and in many cases requires that you first understand why something is done in a particular way. If you don't want to spend time doing that, that's perfectly fine; but then don't expect that open source developers (who are already strapped for time, usually) will suddenly jump up and fix the problems you think are most important, as opposed to what they think should be worked on most urgently. After all, most Open Source developers have their own sets of priorities what they think requires most work, and that's what they'll work on not what's shown in a particular list by someone they don't know and have no reason to respect, somewhere on the Internet. Finally, I'd like to note that any statement which claims that Linux isn't "ready" to be used on the desktop, when hundreds of thousands (if not millions) of people are using it in exactly that way for their day-to-day work, is insulting these people's intelligence. Since that includes me, it's also insulting my intelligence. And I don't like to have my intelligence insulted...

Gregor Herrmann: RC bugs 2012/44

not much to report this week, still a short summary:

Bartosz Feński: amazing

It rocks!

Petter Reinholdtsen: The European Central Bank (ECB) take a look at bitcoin

Slashdot just ran a story about the European Central Bank (ECB) releasing a report (PDF) about virtual currencies and bitcoin. It is interesting to see how a member of the bitcoin community receive the report. As for the future, I suspect the central banks and the governments will outlaw bitcoin if it gain any popularity, to avoid competition. My thoughts go to the W rgl experiment with negative inflation on cash which was such a success that it was terminated by the Austrian National Bank in 1933. A successful alternative would be a threat to the current money system and gain powerful forces to work against it. While checking out the current status of bitcoin, I also discovered that the community already seem to have experienced its first pyramid game / Ponzi scheme. Not very surprising, given how members of "small" communities tend to trust each other. I guess enterprising crocks will try again and again, as they do anywhere wealth is available.

MJ Ray: The @coopsutd Journey

image I m on my way back from Cooperatives United. I haven t quite posted as much as I planned (just a few microblogs/tweets) because the event was so much bigger and busier than I expected! So I m writing this on my (6 hour) journey home. We ve attended Congress before and it was nothing like that. I was expecting something maybe 50% larger than last time and this seemed more like 300%. I wonder if Cooperatives UK have some numbers Maybe it reflects the growing strength of the cooperative economy that they reported in the summer just gone. Anyway, I was pleasantly surprised on the journey from Norfolk. I made a brief informal travellers co-op with an actor from Micawber Theatre company (tickets on sale now for shows in Ipswich and London) to overcome the challenge of East Midlands Trains. Unprompted by me, he blamed the shocking service on the big train businesses being unaccountable to passengers. Maybe that s it. Cooperatives are accountable and people are ready for more responsible post-capitalist businesses. That s why coops are doing well now and Cooperatives United was such a lively event. I ll write more tomorrow. Now if you ll excuse me, we ve got to get off and change carriages because East Midlands Trains couldn t be bothered to tell us the train divides at Nottingham until long after we boarded!

Jaldhar Vyas: A Tale To Strike Terror Into The Hearts Of All Men, Ninjas, and Disney Princesses

Normally around this time of year I would post a picture of one of my children in their Halloween costumes but alas this All-Hallows Eve, Hurricane Sandy has put an end to such frivolity. For, from out of this hell-born wind comes the tale of a creature so horrifying that even puissant warlocks of dark power cower in their Batman pajamas at the sound of his doom-laden moans rending the night as he wanders pitiously through the blackened streets searching for his forbidden love. I speak of the one known only as... THE MAN WHO WENT FOUR DAYS WITHOUT INTERNET Halloween Pic

Russ Allbery: Review: Set This House in Order

Review: Set This House in Order, by Matt Ruff
Publisher: Perennial
Copyright: 2003
Printing: 2004
ISBN: 0-06-095485-X
Format: Trade paperback
Pages: 479
A textbook case history would say that Andy Gage has disassociative identity disorder (referred to here as the more common although medically inaccurate "multiple personality disorder" for reasons of clarity and to match casual speech, according to the author). However, that's a completely inadequate description. A better description would be that Andy Gage's body is inhabited by a large set of different personalities, who take turns being in control. The original Andy Gage is dead, shattered by horrific childhood abuse. Aaron became the dominant personality and gave structure to the internal geography of the mind they all share, with the help of extensive counselling. But Aaron was exhausted by being the public face and maintaining control of the shared body. Hence, Andrew. Andrew Gage is the first-person narrator of half of this book. He is twenty-eight, the same age as the body, but he's only existed for two years. He considers Aaron his father, since Aaron called him out of the lake inside their internal metal geography. His purpose is to run the body and be the public face of the family. He is by turns capable and naive, shy on personal experience, but able to get the help of the others. The one heard from most frequently is Adam, a rude and mischevious teenager who also has an uncanny ability to read other people, and who frequently watches through the body's eyes from what the family calls the "pulpit" and gives Andrew advice. Andrew works for Julie Sivik, a woman he met the day he quit his job at a big-box store because he couldn't explain his behavior to his boss without discussing being a multiple. Julie is a wholly impractical, slightly unethical, well-meaning force of chaos who serves as the thread that draws the plot of the book together. She's running a startup on ingenuity and borrowed money that is attempting to create a comprehensive and customizable virtual reality world. She's the one who finds Penny and hires her, in part for her skills, and in part to push Penny and Andrew together. Penny Driver has the same condition as Andrew, but Penny is only vaguely aware of it. The world for her is full of blackouts, lost time, waking up in strange and sometimes horrific situations, and attempting to structure her life based on notes left for herself that she didn't write. When she meets Andrew, she's barely making it through life; thankfully, one of her personalities is remarkably good at computers, and that has been enough to make a living despite the chaos. But Julie is the unintentional catalyst for her losing that job, the best job she's ever had, leaving her with no real alternative but to take the job with Julie's company. There are a few things that are important to say up-front about this book. First, neither I nor the author are actually multiples (or, put another way, neither of us have DID), and writing about atypical mental states is fraught with peril. It's very easy to misrepresent other people's experiences, and multiple personalities in particular are frequent material for dramas of all sorts. There's something people find inherently fascinating about the idea, and it's easy for that fascination to run away with reality and distort the lived experience. I recommend reading Ruff's FAQ about the book, which spells out what his goals are and what approach he took. I did some quick searches and didn't uncover a lot of anger or dislike of the book among either people who have DID or psychiatrists, and it seems true to the descriptions of internal reality I've heard from people who are multiples. But despite being more respectful of real experience than most fictional treatments of DID, this is still a work of fiction written by someone without first-hand experience, and it's worth keeping that in mind. Second, DID is an intensely controversial diagnosis among psychiatrists. There are, generally speaking, two major controversies. One is over whether or not DID can actually be caused by traumatic experiences such as childhood abuse, or whether it is a creation of psychiatrists, a mental model that they've imposed on their patients or convinced their patients to believe. Set This House in Order comes down firmly on the first side of that controversy: that DID can be created by abuse and is not a creation of psychotherapy. (So, I should note, does every multiple that I've ever talked to or read.) Second, there is significant controversy over treatment, with some psychiatrists believing that integration of the personalities into one is the only true treatment, and others believing that multiples can remain stable and functional as multiple personalities. Set This House in Order leans strongly towards the view that multiples can be stable as multiples and that while integration may be appropriate for some, it's not for all. Both of these viewpoints are presented very convincingly by the narrative in the book, and both are consistent with the beliefs of a lot of multiples, but it's worth remembering that these are active and often quite heated controversies and a novel is not evidence. It's good to be aware of both of those points, and to read Put This House in Order as something of a what-if, where much more is clear about DID than is actually clear in our world. But once one has done that, one is in for an excellent book. I loved this book from start to finish. (Well, okay, I felt some surprise at the nature of the ending, but it grew on me after further consideration.) It's simply a brilliant novel, with a little bit of everything: very good characterization, messy love affairs, the satisfying tension of someone overcoming great odds to put themselves together with the help of friends, a startlingly original perspective of the world, some excellent surprises, and some absolutely wonderful characters. Andrew is a good choice of narrator: he brings an earnest calm to a story that could otherwise be quite chaotic and provides a great vantage point for the reader. And I found Penny (and particularly some of her other souls) impossible not to like. Set This House in Order is a tense book in places, and it's a book where people's lives break down before they can get better. It's also a book about rather horrific abuse, and while nearly all of the specifics happen off-stage, imagination fills in some nasty details. But Ruff manages to write it with a structure and with enough calm assurance that I felt just reassured enough without taking away the tension. That's a tricky balance; few books do it better. And the supporting characters are a delight. One might think, with all the complexity happening for both Andrew and Penny, that Ruff wouldn't have room in the narrative for other characters, but one would be wrong. There are very few saints and demons here, just a lot of complex people, with their own problems and their own perspectives, who are generally trying to do the best they can from day to day. There are some clear-cut villains in the past Ruff wisely makes no attempt to add shades of grey to abusers but even there one can see some clear signs of untreated severe mental illness and chaotic lives rather than mustache-twirling pre-meditated villainy. This is, in short, a book full of people, in all their messy complications, which is what makes it so delightful. I'm not going to say anything more about the plot, or even the various personalities, since discovering the direction of the book was so much of the fun. I will say that, despite the subject matter and some of the events, the book is surprisingly gentle. One of the things about it that I loved the most was the emotional tone Ruff gave to being a multiple: the sense of a squabbling but tight-knit family looking out for each other. I can't judge the reality of this impression, but it makes for a wonderful story. Highly recommended. The best book I've read so far this year. Rating: 10 out of 10

Wouter Verhelst: Why Windows is not ready for general usage

There's this new Windows thing that people keep talking about, and which attempts to replace Linux as the major operating system. I've given it a look, but there's a wide range of problems with using it: In short, while this Windows thing has some promise, the above list shows that there's clearly still quite some work to be done before anyone can even think about starting to use it seriously. Hopefully the Windows developers will understand that and work on the above list of serious issues; otherwise, I'm afraid Windows will not be used by many people. Update: before you go all berserk and start commenting on this list, please read my followup to this post, which explains the point I was trying to make.

Paul Wise: Remembering the dead

Last week was el D a de los Muertos (the Day of the Dead), a Mexican holiday where people gather to remember their dead. Today I would like to remember <predator>. pred was a core member of the cat@lyst collective, a group of folks providing computers, Internet and other tech resources to the activists of Sydney, Australia. I knew pred as cat@lyst's hardware geek, he was always scrounging computers from roadsides, coaxing them back to life and setting them to work supporting cat@lyst. I only knew pred through email and just missed out on meeting him due to the cancer that eventually killed him. pred wrote extensively during his life and of his experiences while dying of cancer. After he died, some friends compiled some of his writing into a book, simply named pred.txt. His full writings also happen to have been archived on the net. Through the book and his writings I came to learn that he was much more than just a hardware geek and gain some more insight into who he was. pred was a geek, activist, anarchist, urban explorer, scientist in the area of molecular genetics, cancer victim and all-around awesome dude. I still miss him.

3 November 2012

Richard Hartmann: Release Critical Bug report for Week 42

Lucas Nussbaum pointed out that the weekly stats which Alexander Reichle-Schmehl used to provide are AWOL so I decided to take over. As I am patently lazy, I hacked together a little helper in perl which outputs the page I am just adding to by writing this. It's a tad hackish inasmuch I don't bother to transform the HTML into anything and simply embedd it in the Markdown, but it's working well enough for now. In the medium term, I am hoping to add some nice graphs as per azhag's suggestion, but they didn't get back to me, yet. A warning to anyone not subscribed to Planet Debian and reading this via a different aggregator: You will still see weekly stats from now on which may or may not be interesting to you. In case you object, please leave a comment or poke me directly. The UDD bugs interface currently knows about the following release critical bugs: How do we compare to the squeeze cycle ?
Week Squeeze Wheezy Diff
43 284 (213+71) 468 (332+136) +184 (+119 ; +65)
44 261 (201+60)
45 261 (205+56)
46 271 (200+71)
47 283 (209+74)
48 256 (177+79)
49 256 (180+76)
50 204 (148+56)
51 124 (124+54)
52 115 (78+37)
1 93 (60+33)
2 82 (46+36)
3 25 (15+10)
4 14 (8+6)
5 2 (0+2)
6 release!

Russ Allbery: Welcome, Planet Debian

As part of a discussion about something else on debian-project, it seemed clear that people were interested in more than just Debian-relevant posts on the Planet Debian aggregator. I'd previously only pushed Debian-related posts there, but I asked explicitly and people want the full feed. So this is the first untagged post going there (and something of a test). The difference, for Planet Debian readers, is that you'll get a lot more book reviews (of non-technical books), and a smattering of software announcements. You'll also get posts like the below whenever I buy more books. I got into the habit of making haul posts from enjoying a friend's version of that, and now I use them as a way of keeping track of what I've already bought until I get around to putting stuff into a real book database. (I'm currently leaning towards Tellico, for what it's worth.) I'll try to add at least a bit of commentary so that it won't just be a list of books. If I can get back into the habit, you'll also get regular postings of photographs. I've been out of the habit for about a year, though, so we'll see how that goes. Anyway, a small haul: Kevin Barry Dark Lies the Island (mainstream)
J. Robert Lennon Familiar (mainstream)
Ken & Jo Walton GURPS: Celtic Myth (RPG) This is notable because it's my first Indiespensible shipment. Powell's Books, from which I buy nearly all of my books, has a subscription club called Indiespensible, which sends a new novel, usually from independent presses, every six weeks. I've been eyeing this for a long time and finally decided to go ahead and try it for a while. I mostly read science fiction and fantasy, so this is a good way to try to read more mainstream fiction, and since I know absolutely nothing about what's good in mainstream fiction, something curated is a good idea. I haven't cracked open either of the books yet, but I quite enjoyed the chocolate toffee. GURPS: Celtic Myth was because I discovered that there was a role-playing supplement by one of my favorite authors, so that had to be acquired, even if I'm not a GURPS player.

2 November 2012

Neil Williams: Introducing pyBit - Buildd Integration Toolkit

pyBit - cross-platform package building using AMQP


Message queues provide a simple way to create a distributed, cross-platform buildd toolkit to build packages using a collection of buildds, direct from various VCS clients. pyBit is intended to support rapidly evolving software collections and can support multiple VCS frontends and multiple build backends. Cross building is expected to be supported for some backends. The initial backend uses dpkg for Debian with subversion providing the source and sbuild doing the actual build.

pyBit includes support for cancelling selected builds and using multiple buildd clients per architecture, per platform and per suite.

Hooks are available or in development for subversion and git, other VCS hooks can be added. A RESTful web API provides live build reports and can generate build jobs for specific packages using particular VCS branches on selected architectures to support re-building packages at any point in the development process. Build history is stored using postgresql.

Other buildd systems can rebuild long lists of packages or build lots of binary packages from relatively slow moving source packages. PyBit exists to handle much more rapid software development across a wide range of platforms, VCS inputs and architectures. Buildd clients which are under-used can be tasked with building multiple suites or adding cross-build support. Buildd clients which are over-utilised are easily identified and adding new machines to an existing architecture / platform / suite pool should be easy. Hook activation automatically cancels any ongoing build for the same architecture, platform and suite to avoid wasting time on an interim version.

The emphasis in pyBit is to have fast builds with redundant clients, reliable reporting using a flexible and intuitive frontend. To this end, there is no need for a source package to be uploaded. Depending on the VCS hook in use, builds can happen every time a particular file is changed (e.g. debian/changelog for a native Debian package) or at every push (for a distributed VCS) or whatever is appropriate for a particular software collection. Builds are checked for available build-dependencies and packages re-queued if the build-dependencies are not yet met.

In the longer term, it may well be possible to use more than one server / database combination to support more builds and more platforms.

So far, we've got to a working model and tagged 0.1.0 as the first downloadable release. There is a lot more to do, especially adding more documentation, more VCS hooks, support for more VCS methods on the buildd clients and more buildd client scripts for platforms other than Debian. (The git hook and git source client are expected to let pyBit be self-buildable but a certain amount of configuration will be required for the server and each client which makes it not-quite self-hosting.)

pyBit concentrates on preparing collections of binary packages which can be used to build others, rather than trying to rebuild everything every time - this allows more rapid upstream software development and encourages modular, re-usable software. pyBit will also support rebuilding specific versions, architectures, suites or platforms via the RESTful web API. Access to this frontend can be controlled through any of the standard methods.

Components are loosely coupled via JSON encoded messages sent using rabbitMQ and curl. A new client can be added at any time and it will simply pick up buildd jobs from the relevant queue.

Development has now moved to Github and if anyone wants to look at new clients and new hooks, just contact the team by email or via IRC (#pybit on irc.oftc.net). Current development is based on Debian Squeeze 6.0.6 with backports and also on Debian Wheezy - more testing is welcome. Patches are also very welcome, pyBit is licensed under GPL2+.

There is no intrinsic reason why pyBit could not support any buildd platform capable of taking source from a known location and building a set of binaries. Packages are added to the queues whenever the hook is activated, so adding new packages to the collection is simply a case of triggering the existing hook.

Another interesting challenge for pyBit would be to trigger a hook on a new source code repository and just let it work through every package until everything is built. That probably won't work with the current 0.1.0 code but if that is what you'd like to work on, join the team.

Vincent Bernat: Network virtualization with VXLAN

Virtual eXtensible Local Area Network (VXLAN) is a protocol to overlay a virtualized L2 network over an existing IP network with little setup. It is currently described in an Internet-Draft. It adds the following perks to VLANs while still providing isolation:
  1. It uses a 24-bit VXLAN Network Identifier (VNI) which should be enough to address any scale-based concerns of multitenancy.
  2. It wraps L2 frames into UDP datagrams. This allows one to rely on some interesting properties of IP networks like availability and scalability. A VXLAN segment can be extended far beyond the typical reach of today VLANs.
The VXLAN Tunnel End Point (VTEP) originates and terminates VXLAN tunnels. Thanks to a serie of patches from Stephen Hemminger, Linux can now act as a VTEP. Let s see how this works.

About IPv6 When possible, I try to use IPv6 for my labs. This is not the case here for several reasons:
  1. IP multicast is required and PIM-SM implementations for IPv6 are not widespread yet. However, they exist. This explains why I use XORP for this lab: it supports PIM-SM for both IPv4 and IPv6.
  2. VXLAN Internet-Draft specifically addresses only IPv4. This seems a bit odd for a protocol running on top of UDP and I hope this will be fixed soon. This is not a major stopper since some VXLAN implementations support IPv6.
  3. However, the current implementation for Linux does not support IPv6. IPv6 support will be added later.
Once IPv6 support is available, the lab should be easy to adapt.

Lab So, here is the lab used. R1, R2 and R3 will act as VTEPs. They do not make use of PIM-SM. Instead, they have a generic multicast route on eth0. E1, E2 and E3 are edge routers while C1, C2 and C3 are core routers. The proposed lab is not resilient but convenient to explain how things work. It is built on top of KVM hosts. Have a look at my previous article for more details on this. VXLAN lab The lab is hosted on GitHub. I have made the lab easier to try by including the kernel I have used for my tests. XORP comes preconfigured, you just have to configure the VXLAN part. For this, you need a recent version of ip.
$ sudo apt-get install screen vde2 kvm iproute xorp git
$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git
$ cd iproute2
$ ./configure && make
You get  ip' as  ip/ip' and  bridge' as  bridge/bridge'.
$ cd ..
$ git clone git://github.com/vincentbernat/network-lab.git
$ cd network-lab/lab-vxlan
$ ./setup

Unicast routing The first step is to setup unicast routing. OSPF is used for this purpose. The chosen routing daemon is XORP. With xorpsh, we can check if OSPF is working as expected:
root@c1# xorpsh
root@c1$ show ospf4 neighbor   
  Address         Interface             State      ID              Pri  Dead
192.168.11.11    eth0/eth0              Full      3.0.0.1          128    36
192.168.12.22    eth1/eth1              Full      3.0.0.2          128    33
192.168.101.133  eth2/eth2              Full      2.0.0.3          128    36
192.168.102.122  eth3/eth3              Full      2.0.0.2          128    38
root@c1$ show route table ipv4 unicast ospf   
192.168.1.0/24  [ospf(110)/2]
                > to 192.168.11.11 via eth0/eth0
192.168.2.0/24  [ospf(110)/2]
                > to 192.168.12.22 via eth1/eth1
192.168.3.0/24  [ospf(110)/3]
                > to 192.168.102.122 via eth3/eth3
192.168.13.0/24 [ospf(110)/2]
                > to 192.168.102.122 via eth3/eth3
192.168.21.0/24 [ospf(110)/2]
                > to 192.168.101.133 via eth2/eth2
192.168.22.0/24 [ospf(110)/2]
                > to 192.168.12.22 via eth1/eth1
192.168.23.0/24 [ospf(110)/2]
                > to 192.168.101.133 via eth2/eth2
192.168.103.0/24        [ospf(110)/2]
                > to 192.168.102.122 via eth3/eth3

Multicast routing Once unicast routing is up and running, we need to setup multicast routing. There are two protocols for this: IGMP and PIM-SM. The former one allows routers to forward multicast datagrams while the first one allows hosts to subscribe to a multicast group.

IGMP IGMP is used by hosts and adjacent routers to establish multicast group membership. In our case, it will be used by R2 to let E2 know it subscribed to 239.0.0.11 (a multicast group). Configuring XORP to support IGMP is simple. Let s test with iperf to have a multicast listener on R2:
root@r2# iperf -u -s -l 1000 -i 1 -B 239.0.0.11
------------------------------------------------------------
Server listening on UDP port 5001
Binding to local address 239.0.0.11
Joining multicast group  239.0.0.11
Receiving 1000 byte datagrams
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
On E2, we can now check that R2 is properly registered for 239.0.0.11:
root@e2$ show igmp group
Interface    Group           Source          LastReported Timeout V State
eth0         239.0.0.11      0.0.0.0         192.168.2.2      248 2     E
XORP documentation contains a good overview of IGMP.

PIM-SM PIM-SM is far more complex. It does not have its own topology discovery protocol and relies on routing information from other protocols, OSPF in our case. I will describe here a simplified view on how PIM-SM works. XORP documentation contains more details about PIM-SM. The first step for all PIM-SM routers is to elect a rendez-vous point (RP). In our lab, only C1, C2 and C3 have been configured to be elected as a RP. Moreover, we give better priority to C3 to ensure it wins. RP election
root@e1$ show pim rps   
RP              Type      Pri Holdtime Timeout ActiveGroups GroupPrefix       
192.168.101.133 bootstrap 100      150     135            0 239.0.0.0/8
Let s suppose we start iperf on both R2 and R3. Using IGMP, they subscribe to multicast group 239.0.0.11 with E2 and E3 respectively. Then, E2 and E3 send a join message (also known as a (*,G) join) to the RP (C3) for that multicast group. Using the unicast path from E2 and E3 to the RP, the routers along the paths build the RP tree (RPT), rooted at C3. Each router in the tree knows how to send multicast packets to 239.0.0.11: it will send them to the leaves. RP tree
root@e3$ show pim join   
Group           Source          RP              Flags
239.0.0.11      0.0.0.0         192.168.101.133 WC   
    Upstream interface (RP):   eth2
    Upstream MRIB next hop (RP): 192.168.23.133
    Upstream RPF'(*,G):        192.168.23.133
    Upstream state:            Joined 
    Join timer:                5
    Local receiver include WC: O...
    Joins RP:                  ....
    Joins WC:                  ....
    Join state:                ....
    Prune state:               ....
    Prune pending state:       ....
    I am assert winner state:  ....
    I am assert loser state:   ....
    Assert winner WC:          ....
    Assert lost WC:            ....
    Assert tracking WC:        O.O.
    Could assert WC:           O...
    I am DR:                   O..O
    Immediate olist RP:        ....
    Immediate olist WC:        O...
    Inherited olist SG:        O...
    Inherited olist SG_RPT:    O...
    PIM include WC:            O...
Let s suppose that R1 wants to send multicast packets to 239.0.0.11. It sends them to R1 which does not have any information on how to contact all the members of the multicast group because it is not the RP. Therefore, it encapsulates the multicast packets into PIM Register packets and sends them to the RP. The RP decapsulates them and sends them natively. The multicast packets are routed from the RP to R2 and R3 using the reverse path formed by the join messages. Multicast routing via the RP
root@r1# iperf -c 239.0.0.11 -u -b 10k -t 30 -T 10
------------------------------------------------------------
Client connecting to 239.0.0.11, UDP port 5001
Sending 1470 byte datagrams
Setting multicast TTL to 10
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
root@e1# tcpdump -pni eth0
10:58:23.424860 IP 192.168.1.1.35277 > 239.0.0.11.5001: UDP, length 1470
root@c3# tcpdump -pni eth0
10:58:23.552903 IP 192.168.11.11 > 192.168.101.133: PIMv2, Register, length 1480
root@e2# tcpdump -pni eth0
10:58:23.896171 IP 192.168.1.1.35277 > 239.0.0.11.5001: UDP, length 1470
root@e3# tcpdump -pni eth0
10:58:23.824647 IP 192.168.1.1.35277 > 239.0.0.11.5001: UDP, length 1470
As presented here, the routing is not optimal: packets from R1 to R2 could avoid the RP. Moreover, encapsulating multicast packets into unicast packets is not efficient either. At some point, the RP will decide to switch to native multicast1. Rooted at R1, the shortest-path tree (SPT) for the multicast group will be built using source-specific join messages (also known as a (S,G) join). Multicast routing without RP From here, each router in the tree knows how to handle multicast packets from R1 to the group without involving the RP. For example, E1 knows it must duplicate the packet and sends one through the interface to C3 and the other one through the interface to C1:
root@e1$ show pim join   
Group           Source          RP              Flags
239.0.0.11      192.168.1.1     192.168.101.133 SG SPT DirectlyConnectedS 
    Upstream interface (S):    eth0
    Upstream interface (RP):   eth1
    Upstream MRIB next hop (RP): 192.168.11.111
    Upstream MRIB next hop (S):  UNKNOWN
    Upstream RPF'(S,G):        UNKNOWN
    Upstream state:            Joined 
    Register state:            RegisterPrune RegisterCouldRegister 
    Join timer:                7
    KAT(S,G) running:          true
    Local receiver include WC: ....
    Local receiver include SG: ....
    Local receiver exclude SG: ....
    Joins RP:                  ....
    Joins WC:                  ....
    Joins SG:                  .OO.
    Join state:                .OO.
    Prune state:               ....
    Prune pending state:       ....
    I am assert winner state:  ....
    I am assert loser state:   ....
    Assert winner WC:          ....
    Assert winner SG:          ....
    Assert lost WC:            ....
    Assert lost SG:            ....
    Assert lost SG_RPT:        ....
    Assert tracking SG:        OOO.
    Could assert WC:           ....
    Could assert SG:           .OO.
    I am DR:                   O..O
    Immediate olist RP:        ....
    Immediate olist WC:        ....
    Immediate olist SG:        .OO.
    Inherited olist SG:        .OO.
    Inherited olist SG_RPT:    ....
    PIM include WC:            ....
    PIM include SG:            ....
    PIM exclude SG:            ....
root@e1$ show pim mfc  
Group           Source          RP             
239.0.0.11      192.168.1.1     192.168.101.133
    Incoming interface :      eth0
    Outgoing interfaces:      .OO.
root@e1$ exit
[Connection to XORP closed]
root@e1# ip mroute
(192.168.1.1, 239.0.0.11)        Iif: eth0       Oifs: eth1 eth2

Setting up VXLAN Once IP multicast is running, setting up VXLAN is quite easy. Here are the software requirements:
  • A recent kernel. Pick at least 3.7-rc3. You need to enable CONFIG_VXLAN option. You also currently need a patch on top of it to be able to specify a TTL greater than 1 for multicast packets.
  • A recent version of ip. Currently, you need the version from git.
On R1, R2 and R3, we create a vxlan42 interface with the following commands:
root@rX# ./ip link add vxlan42 type vxlan id 42 \
>                               group 239.0.0.42 \
>                               ttl 10 dev eth0
root@rX# ip link set up dev vxlan42
root@rX# ./ip -d link show vxlan42
10: vxlan42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc noqueue state UNKNOWN mode DEFAULT 
link/ether 3e:09:1c:e1:09:2e brd ff:ff:ff:ff:ff:ff
vxlan id 42 group 239.0.0.42 dev eth0 port 32768 61000 ttl 10 ageing 300
Let s assign an IP in 192.168.99.0/24 for each router and check they can ping each other:
root@r1# ip addr add 192.168.99.1/24 dev vxlan42
root@r2# ip addr add 192.168.99.2/24 dev vxlan42
root@r3# ip addr add 192.168.99.3/24 dev vxlan42
root@r1# ping 192.168.99.2                    
PING 192.168.99.2 (192.168.99.2) 56(84) bytes of data.
64 bytes from 192.168.99.2: icmp_req=1 ttl=64 time=3.90 ms
64 bytes from 192.168.99.2: icmp_req=2 ttl=64 time=1.38 ms
64 bytes from 192.168.99.2: icmp_req=3 ttl=64 time=1.82 ms
--- 192.168.99.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 1.389/2.375/3.907/1.098 ms
We can check the packets are encapsulated:
root@r1# tcpdump -pni eth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:30:36.561185 IP 192.168.1.1.43349 > 192.168.2.2.8472: UDP, length 106
11:30:36.563179 IP 192.168.2.2.33894 > 192.168.1.1.8472: UDP, length 106
11:30:37.562677 IP 192.168.1.1.43349 > 192.168.2.2.8472: UDP, length 106
11:30:37.564316 IP 192.168.2.2.33894 > 192.168.1.1.8472: UDP, length 106
Moreover, if we send broadcast packets (with ping -b or ARP requests), they are encapsulated into multicast packets:
root@r1# tcpdump -pni eth0
11:31:27.464198 IP 192.168.1.1.41958 > 239.0.0.42.8472: UDP, length 106
11:31:28.463584 IP 192.168.1.1.41958 > 239.0.0.42.8472: UDP, length 106
Recent versions of iproute also comes with bridge, an utility allowing one to inspect the FDB of bridge-like interfaces:
root@r1# ../bridge/bridge fdb show vxlan42
3e:09:1c:e1:09:2e dev vxlan42 dst 192.168.2.2 self 
0e:98:40:c6:58:10 dev vxlan42 dst 192.168.3.3 self

Demo For a demo, have a look at the following video (it is also available as an Ogg Theora video).
<iframe frameborder="0" height="270" src="http://www.dailymotion.com/embed/video/xusell" width="480"></iframe>

  1. The decision is usually done when the bandwidth used by the follow reachs some threshold. With XORP, this can be controlled with switch-to-spt-threshold. However, I was unable to make this works as expected. XORP never sends the appropriate PIM packets to make the switch. Therefore, for this lab, it has been configured to switch to native multicast at the first received packet.

Erich Schubert: DBSCAN and OPTICS clustering

DBSCAN [wikipedia] and OPTICS [wikipedia] are two of the most well-known density based clustering algorithms. You can read more on them in the Wikipedia articles linked above.
An interesting property of density based clustering is that these algorithms do not assume clusters to have a particular shape. Furthermore, the algorithms allow "noise" objects that do not belong to any of the clusters. K-means for examples partitions the data space in Voronoi cells (some people claim it produces spherical clusters - that is incorrect). See Wikipedia for the true shape of K-means clusters and an example that canot be clustered by K-means. Internal measures for cluster evaluation also usually assume the clusters to be well-separated spheres (and do not allow noise/outlier objects) - not surprisingly, as we tend to experiment with artificial data generated by a number of Gaussian distributions.
The key parameter to DBSCAN and OPTICS is the "minPts" parameter. It roughly controls the minimum size of a cluster. If you set it too low, everything will become clusters (OPTICS with minPts=2 degenerates to a type of single link clustering). If you set it too high, at some point there won't be any clusters anymore, only noise. However, the parameter usually is not hard to choose. If you for example expect clusters to typically have 100 objects, I'd start with a value of 10 or 20. If your clusters are expected to have 10000 objects, then maybe start experimenting with 500.
The more difficult parameter for DBSCAN is the radius. In some cases, it will be very obvious. Say you are clustering users on a map. Then you might know that a good radius is 1 km. Or 10 km. Whatever makes sense for your particular application. In other cases, the parameter will not be obvious, or you might need multiple values. That is when OPTICS comes into play.
OPTICS is based on a very clever idea: instead of fixing MinPts and the Radius, we only fix minpts, and plot the radius at which an object would be considered dense by DBSCAN. In order to sort the objects on this plot, we process them in a priority heap, so that nearby objects are nearby in the plot. this image on Wikipedia shows an example for such a plot.
OPTICS comes at a cost compared to DBSCAN. Largely because of the priority heap, but also as the nearest neighbor queries are more complicated than the radius queries of DBSCAN. So it will be slower, but you no longer need to set the parameter epsilon. However, OPTICS won't produce a strict partitioning. Primarily it produces this plot, and in many situations you will actually want to visually inspect the plot. There are some methods to extract a hierarchical partitioning out of this plot, based on detecting "steep" areas.
The open source ELKI data mining framework (package "elki" in Debian and Ubuntu) has a very fast and flexible implementation of both algorithms. I've benchmarked this against GNU R ("fpc" package") and Weka, and the difference is enormous. ELKI without index support runs in roughly 11 minutes, with index down to 2 minutes for DBSCAN and 3 minutes for OPTICS. Weka takes 11 hours and GNU R/fpc takes 100 minutes (DBSCAN, no OPTICS available). And the implementation of OPTICS in Weka is not even complete (it does not support proper cluster extraction from the plot). Many of the other OPTICS implementations you can find with Google (e.g. in Python or MATLAB) seem to be based on this Weka version ...
ELKI is open source. So if you want to peek at the code, here are direct links: DBSCAN.java, OPTICS.java.
Some part of the code may be a bit confusing at first. The "Parameterizer" classes serve the purpose of allowing automatic UI generation, for example. So there is quite a bit of meta code involved.
Plus, ELKI is quite extensively optimized. For example, it does not use Java Collections much anymore. Java Iterators, for example, require returning an object on next();. The C++ style iterators used by ELKI can have multiple values, and primitive values.
for(DBIDIter id = relation.iterDBIDs(); id.valid(); id.advance())
is a typical for loop in ELKI, iterating over all objects of a relation, but the whole loop requires creating (and GC'ing) a single object. And actually, this is as literal as a for loop can get.
ModifiableDBIDs processedIDs = DBIDUtil.newHashSet(size);
is another example. Essentially, this is like a HashSet<DBID>. Except that it is a lot faster, because the object IDs do not need to live a Java objects, but can internally be stored more efficiently (the only currently available implementation of the DBID layer uses primitive integers).
Java advocates always accuse you of premature optimization when you avoid creating objects for primitives. Yet, in all my benchmarking, I have seen this continuously to have a major impact how many objects you allocate. At least when it is inside a loop that is heavily used. Java collections with boxed primitives just eat a lot of memory, and the memory management overhead does often make a huge difference. Which is why libraries such as Trove (which ELKI uses a lot) exist. Because memory usage does make a difference.
(Avoiding boxing/unboxing systematically in ELKI yielded approximately a 4x speedup. But obviously, ELKI involves a lot of numerical computations.)

Michal &#268;iha&#345;: RSS feeds and feature freeze of Weblate 1.3

Yesterday I've completed last big feature I wanted to have for Weblate 1.3 - improved changes tracking and exporting this as RSS feed. You can find example of this feature on data exports page on demo server, it is also covered in the documentation. I think there is not much to describe here, these are simple RSS feeds which will contain all important things happening to translation. This being last major feature, we're now aiming at stabilization and bug fixes. I expect this phase will take two weeks where I plan to deploy new version to http://l10n.cihar.com/ (probably next week). If you can help with testing, you're welcome to help and report bugs.

Filed under: English Weblate 0 comments Flattr this!

Bastian Venthur: Introducing The Art of Asking

Since October 2011 my flatmate and I where quite busy realizing a little pet project of ours called The Art of Asking. The ultimate goal is to visualize the world s opinion in an intuitive fashion and make it easy for everyone to play around with the data. The idea behind The Art of Asking is that users submit interesting questions which are answered by users around the world. But instead of showing only the boring result, we want to provide interesting insights and statistics about the answers given. For now the users can see the results of the question visualized by geographical regions. For example on the page for the question How are you today? you can see the interactive map with the pie chart. The map shows the average/dominating answer for each continent encoded by color, and the pie chart the distribution of the different answers for that region. If you move the mouse over a continent the pie chart updates and shows the distribution of answers for that continent. You can also click to zoom into the map to see the same for countries and regions. This allows you to investigate how the answers are distributed around the world. This is already quite nice and fancy to play around with, but of course we want much more. Right now we re working on a feature which will users allow to combine two arbitrary questions and see how the answers are related. This doesn t sound like much, but it is very addictive to crawl through the list of questions and find interesting correlations. Here is a little plot how it could look like (the data is from the actual data base of answers). For each possible combination of answers it shows the percentage of people who answered in that combination. Of course those plots only make sense when enough users answered for both questions you want to compare which is right now not very often the case, so I guess we ll roll out that feature sometime later when we have more data. But we have lots of ideas and are working on further ways to investigate the answers. Obviously one low hanging fruit for example would be the distribution of the answers over time. We ve been working for over a year now on this project in our spare time and we built it more or less from scratch. I wrote a small WSGI framework in Python and on top of that the WSGI application which runs the site. We use MongoDB for the storage of the data, uWSGI and nginx for the server, Jinja2 and Bootstrap for the HTML and D3.js for the visualization of the data, where Maci did a wonderful job realizing the interactive map and charts. We re running this site since July 2012 now and are already quite satisfied with the number of users, and the quality of the questions. But of course we could always use more (especially more answers). So if you want to try it out, go to theartofasking.com and fill out the blank spots on the map! We re happy about every answer and question we can get and are eager to hear your suggestions.

Next.

Previous.