Search Results: "ed"

26 January 2013

Vipin Nair: Final Project (1/N): Introducing Simple Realtime Notification Service

I am building a realtime notification system as my final semester project. The end product is a simple API that can be used by third party to push real time notification to client applications running on multiple platforms that include browsers and devices. Ideally, if your device can connect to the internet, it should be able to receive notifications. The non functional system requirements are pretty ambitious with scalability, availability and latency guarantees being the most important. Maybe little too ambitious for someone like me but then I will learn a lot in this process and its more fun when its challenging. To address the scalability issues, we ll have to distribute the system. A single system will not scale beyond a point and we need to be able to distribute the load over nodes effectively to be able to serve clients while maintaining the quality of service guarantees. With distribution comes a whole lot of other problems like latency, coordination and handling failures. We have a theorem that officially states that you cannot have the pie and eat it too; its called the CAP theorem. It says that you cannot have have a system that is Consistent, Available and Partition tolerant at the same time (Partition tolerance here refers to the ability of a system to continue operating in the event of partial failures). You can have any two but not all three in varying degrees. In a way this is good because you do not go chasing the holy grail because you know it doesn t exist. CAP Twelve Years Later: How the Rules Have Changed is a good read for some sensible advice when applying CAP theorem to real world scenarios. Reading Notes on Distributed Systems for Young Bloods by Jeff Hodges is highly recommended for anyone starting to work on distributed systems. All the commons assumptions that you may have when you are building a distributed system is wrong, so wrong that it is sort of standardized now under the title Fallacies of Distributed Computing. A more detailed article is available here. I will be using concepts as given in the Amazon Dynamo paper to make the system highly available. This one is big and will a get a blog post of its own. The notification service that I am building at its core is a Publish/Subscribe system with soft real time guarantees. Right now I am clueless about how to deal with latency guarantees but I hope to figure it out soon. Websockets will be the application layer protocol that ll be used to push to browsers and for the trasport layer protocol I hear the MQTT protocol fits to a T. More on this later. I will updating my blog as the project progresses, documenting all the design decisions and the rationale behind them. The code will be available online. This is not something new and I am aware of tools that can do this. While I will try to improve on what already exists, the point is, as mentioned earlier, to learn something new and build what will be considered good software. I have never attended a distributed systems theory class, so I am reading up on that as well. Vector clocks are really cool, if you ask me. If you think this is too big, to be accomplished by a single student programmer in a semester s worth of effort, I wouldn t deny that but then when you pick the right tools for a job, things become a lot more easier and you can afford to be ambitious. Erlang for the win! Feel free to comment on what I am building as I may be potentially unaware of a lot of things I should be knowing. I could use all your suggestions/advice to make things better.

David Moreno: FOSDEM 2013

One of the conferences that I always wanted to attend is FOSDEM. Now, since I live close enough to Brussels, it'd be a gigantic letdown not to make it there next weekend. I plan to be in Brussels Fri to Sun, see you there!

Ben Hutchings: What's in the Linux kernel for Debian 7.0 'wheezy', part 4

The Debian package of the Linux kernel is based on Linux 3.2, but has some additional features. This continues from parts 1, 2 and 3, and covers new and improved hardware support. DRM drivers from Linux 3.4 (proposed) Some recent Intel and AMD graphics chips were not supported well or at all by the DRM (Direct Rendering Manager) drivers in Linux 3.2. Although many bug fixes have been included in 3.2.y stable updates, we are considering updating them to the versions found in Linux 3.4. (We did something similar for Debian 6.0 'squeeze'.) Julien Cristau has been working on this and has prepared binary packages. To install, run as root:
gpg --no-default-keyring --keyring /usr/share/keyrings/debian-keyring.gpg --export 310180050905E40C   apt-key add -
echo deb http://people.debian.org/~jcristau/wheezy-drm34/ ./ > /etc/apt/sources.list.d/jcristau-wheezy-drm34.list
apt-get update
apt-get install linux-image-3.2.0-4.drm-amd64  # or -486, or -686-pae
Please test these and report your results to bug #687442. I would suggest testing suspend/resume (if applicable), use of internal and external displays on laptops, and 3D accelerated graphics (games and compositing window managers such as GNOME Shell). Remember that AMD/ATI graphics chips require the firmware from the firmware-linux-nonfree package for 3D acceleration and many other features. amilo-rfkill driver I wrote a standard rfkill driver for the Fujitsu-Siemens Amilo A1655 and M7440, based on the out-of-tree fsaa1655g and fsam7440 drivers. Unlike those, it should work with the rfkill command, Network Manager, etc. ALPS touchpads Newer touchpads made by ALPS use different protocols for reporting scroll and pinch gestures. Jonathan Nieder backported the changes to support these. Wacom tablets Jonathan Nieder updated the wacom driver to the version in Linux 3.5, adding support for the Intuos5, Bamboo Connect, Bamboo 16FG and various other models. Emulex OneConnect 'Skyhawk' Sarveshwar Bandi at Emulex contributed a backport of the be2net driver from Linux 3.5, adding support for their 'Skyhawk' chip. Marvell Kirkwood Marvell's Kirkwood SoCs have been supported upstream for some time and in Debian since release 6.0 'squeeze'. However new models and boards generally require specific support. Arnaud Patard backported support for the 6282 rev A1 chip found in QNAP TS-x19P II models, and for the Marvell Dreamplug and Iomega Iconnect. Miscellaneous More to come Missing hardware support is an important bug that can be fixed by kernel updates during a freeze and throughout the lifetime of a stable release. If you know that new hardware isn't supported by the Debian kernel, please open a bug report. I can't promise that it will be fixed, but we need to know what's missing. Hardware vendors that maintain their own drivers upstream (not out-of-tree) are especially welcome to contribute tested backports that add support for the latest hardware. Send mail to debian-kernel@lists.debian.org if you have any questions about this.

Eddy Petrișor: (Serial) console flooded with kernel messages?

(If you want to ignore the explanations and see how to stop the Linux kernel from flooding the console with low importance messages, go straight to the bottom of the article, is the small bit at end with larger font.)

After connecting to the serial console on my Linksys WRT160NL router I was faced with the problem that the console was flooded with all sorts of messages such as:
DROP IN=eth1 OUT= SRC=X.Y.Z.W DST=255.255.255.255 PROTO=UDP SPT=58488 DPT=2008 LEN=26
DROP IN=eth1 OUT= SRC=X.Y.Z.U DST=178.156.183.255 PROTO=UDP SPT=137 DPT=137 LEN=58
DROP IN=eth1 OUT= SRC=X.Y.Z.W DST=255.255.255.255 PROTO=UDP SPT=58488 DPT=2008 LEN=26
ACCEPT IN=br0 OUT=eth1 SRC=a.b.c.d DST=69.171.246.16 PROTO=TCP SPT=3651 DPT=443
DROP IN=eth1 OUT= SRC=X.Y.Z.U DST=178.156.183.255 PROTO=UDP SPT=137 DPT=137 LEN=58
DROP IN=eth1 OUT= SRC=X.Y.Z.W DST=255.255.255.255 PROTO=UDP SPT=58488 DPT=2008 LEN=26
DROP IN=eth1 OUT= SRC=X.Y.Z.W DST=255.255.255.255 PROTO=UDP SPT=58488 DPT=2008 LEN=26
DROP IN=eth1 OUT= SRC=X.Y.Z.W DST=255.255.255.255 PROTO=UDP SPT=58488 DPT=2008 LEN=26
DROP IN=eth1 OUT= SRC=X.Y.Z.W DST=255.255.255.255 PROTO=UDP SPT=58488 DPT=2008 LEN=26
DROP IN=eth1 OUT= SRC=X.Y.Z.W DST=255.255.255.255 PROTO=UDP SPT=58488 DPT=2008 LEN=26
DROP IN=eth1 OUT= SRC=X.Y.Z.W DST=255.255.255.255 PROTO=UDP SPT=58488 DPT=2008 LEN=26
DROP IN=eth1 OUT= SRC=X.Y.Z.W DST=255.255.255.255 PROTO=UDP SPT=58488 DPT=2008 LEN=26
DROP IN=eth1 OUT= SRC=178.156.183.146 DST=255.255.255.255 PROTO=UDP SPT=17500 DPT=17500 LEN=120
DROP IN=eth1 OUT= SRC=178.156.183.146 DST=178.156.183.255 PROTO=UDP SPT=17500 DPT=17500 LEN=120
DROP IN=eth1 OUT= SRC=178.156.183.146 DST=255.255.255.255 PROTO=UDP SPT=17500 DPT=17500 LEN=120
DROP IN=eth1 OUT= SRC=X.Y.Z.W DST=255.255.255.255 PROTO=UDP SPT=58488 DPT=2008 LEN=26
DROP IN=eth1 OUT= SRC=X.Y.Z.W DST=255.255.255.255 PROTO=UDP SPT=58488 DPT=2008 LEN=26
DROP IN=eth1 OUT= SRC=178.156.177.142 DST=255.255.255.255 PROTO=UDP SPT=17500 DPT=17500 LEN=153
DROP IN=eth1 OUT= SRC=178.156.177.142 DST=255.255.255.255 PROTO=UDP SPT=17500 DPT=17500 LEN=153

The serial console was working, but it was impossible to do anything practical in these conditions. I tried looking on the net for 'linux stop console flooding' and similar terms, but didn't get too far, except the fact the problem was the loglevel.

Here is the explanation of what this means (quote from Documentation/kernel-parameters.txt):

        loglevel=       All Kernel Messages with a loglevel smaller than the
                        console loglevel will be printed to the console. It can
                        also be changed with klogd or other programs. The
                        loglevels are defined as follows:
                        0 (KERN_EMERG)          system is unusable
                        1 (KERN_ALERT)          action must be taken immediately
                        2 (KERN_CRIT)           critical conditions
                        3 (KERN_ERR)            error conditions
                        4 (KERN_WARNING)        warning conditions
                        5 (KERN_NOTICE)         normal but significant condition
                        6 (KERN_INFO)           informational
                        7 (KERN_DEBUG)          debug-level messages


This was enough to go through my local git tree of the kernel in the Documentation directory and grep for loglevel. This brought me to this interesting bit from Documentation/sysctl/kernel.txt
==============================================================
printk:
The four values in printk denote: console_loglevel,
default_message_loglevel, minimum_console_loglevel and
default_console_loglevel respectively.
These values influence printk() behavior when printing or
logging error messages. See 'man 2 syslog' for more info on
the different loglevels.
- console_loglevel: messages with a higher priority than
  this will be printed to the console
- default_message_loglevel: messages without an explicit priority
  will be printed with this priority
- minimum_console_loglevel: minimum (highest) value to which
  console_loglevel can be set
- default_console_loglevel: default value for console_loglevel
==============================================================

So I ran 'cat /proc/sys/kernel/printk' and got (I managed to read it through the flood of messages from the firewall):

7       4       1       7
According to the explanations above, that meant that console_loglevel was too, so to fix it I ran:
echo '2 4 1 7' > /proc/sys/kernel/printk
And, behold, the serial console was usable.

Michal Čihař: Weblate 1.4 on SUSE Studio

It took me some time, but finally updated appliances with Weblate are available. As usual, it was built using SUSE Studio. The appliance allows you to test Weblate without installation, everything is setup to work out of the box. However if you want to use it in production setup, few adjustments are recommended.

Filed under: English Suse Weblate 0 comments Flattr this!

25 January 2013

Richard Hartmann: Please don't do this

Update: They got & reported him. Most of you will have seen the news about various private files being hosted on github as this little has taken the Internet by storm yesterday. Obviously most of those files can be found via Google and other means as well, but this exposure is a bit worse inasmuch people will keep on updating their secret information automagically and continuously. Personal favourites that used to work until yesterday include And various other config and history files for AWS, RT, whatnot. Plus, it was really easy to search for certain actions like so. Another fun thing look for were ENV variables setting passwords, hosts, etc. Ironically, having the SSH identity files means anyone is able to change pretty much anything about a person's repositories on github. This includes files like .zshrc which basically allows complete takeover of any account and often machine that file is sourced from. Which reminds me that I still need to add a way for vcsh to only merge commits that have a tag signed with a trusted key and warn for everything else, but I digress. None of this is a problem with github specifically, it's a problem with users who don't think their actions through. And this is non-trivial for github or anyone else to fix as there are potentially endless sources of otherwise secret information. Initially, I sent email to github's quite responsive security team asking them to forbid certain queries and to email users who checked in their private data by accident and left it at that. They got back to me extremely fast, promising to do their best (as of right now, no queries I tried work any more) and after some conversation asked me to link to their help article on removing sensitive data if I were to blog about this. The topic itself has been covered extensively, github did their best to keep user data private, and so the above could have been done without, but... ...I have gotten word of 4chan's /g/ sleuthing their way through various files and at least one incident of people finding direct evidence of child pornography in a Zsh history file. Sadly, this story found its way to me without a link and the person who saw the thread read it on a tablet, preserving neither the URL nor local cache. They saw it on 2013-01-24 at around 2200 UTC. If you, or someone you know, saw anything plainly illegal, immoral, and simply wrong... please report it to the relevant authorities or at least github. As I suspect that there's some overlap between the subscribers of the various planets I am aggregated on and /g/, this may reach the right eyes. Between the account name, a verified email address, access logs with IPs, and possibly a real name, it should be comparatively easy to find anyone you report.

Steve Kemp: Shame there isn't more competition for self-hosted analytics

Today I've been mostly replanting spider-plants, aloe-vera plants, and shuffling trees around inside my flat. Beyond that I've been updating my trivial dashboard skeleton, which was put together as part of this simple introduction article. (So there is a standalone redis&sinatra-using visualization server) After working on the display I was suddenly reminded that I run a cluster now. That means I have four servers each writing a local Apache logfile, and no central way of viewing all my visitor-data. There are several open source analytic packages such as piwik and openwebanalytics - but they require MySQL & PHP at the back-end. Given that node.js is "teh new shiny" it is a surprise there isn't something out there using that, and web sockets perhaps, to collect visitor data. I found a few toy projects, but nothing that seemed to be a clear winner. Adding some javascript to webpages to submit: Is trivial. The hard part is storing that and visualizing it in a neat way. Making data pretty is something I'm notoriously bad at - unless it is turning numbers into graphs using a good library I'm out of luck most of the time. Anyway I will keep digging. Ideally I'll have a scalable node service that'll receive submissions, bung them in redis, and then show real-time activity in a sexy fashion. I can dream?

Tanguy Ortolo: Do not reCAPTCHA!

A book

You probably know reCAPTCHA already: for the webmasters and the end user, it is an antispam system which asks you to read distorted words in order to prove that you are a human and not a spamming bot. This service has another end: instead of generating its distorted texts, it takes them from printed books to help digitalizing them.The problem In theory, this is an excellent idea, which should help preserving and distributing parts of the human culture. And, in the beginning, it was used in such a way, with reCAPTCHA participating in the Gutenberg project which digitalizes books from the public domain to offer them freely (as in free speech) to everyone. In 2009 however, reCAPTCHA was bought by Google. Now they do not participate in the Gutenberg project any more, instead they are digitalizing stuff from the New York Times and from Google Books. They do not provide much detail about that, but it does not seem that they distribute them in a free way afterwards . To put it boldly: they use people's brain power to digitalize books for their own exploitation monopoly, which is wrong. These books should be made available under a free license. Suggestion Do not help reCAPTCHA. If you are a webmaster, try to use another more ethical system. If you are an end user, and you are facing a reCAPTCHA test, avoid putting the exact answer but try inserting small mistakes (not big ones, which get detected and make you fail the test). And ask Google to free the books they digitalized using your brain. Notes

  1. I may be wrong but I learnt to be suspicious in the context of digital culture. Please Google, if you do publish your books digitalized with reCAPTCHA under a free license, tell me, or rather, tell it to everyone and display it proudly in the reCAPTCHA website.

Richard Hartmann: Release Critical Bug report for Week 04

The UDD bugs interface currently knows about the following release critical bugs: How do we compare to the Squeeze release cycle?
Week Squeeze Wheezy Diff
43 284 (213+71) 468 (332+136) +184 (+119/+65)
44 261 (201+60) 408 (265+143) +147 (+64/+83)
45 261 (205+56) 425 (291+134) +164 (+86/+78)
46 271 (200+71) 401 (258+143) +130 (+58/+72)
47 283 (209+74) 366 (221+145) +83 (+12/+71)
48 256 (177+79) 378 (230+148) +122 (+53/+69)
49 256 (180+76) 360 (216+155) +104 (+36/+79)
50 204 (148+56) 339 (195+144) +135 (+47/+90)
51 178 (124+54) 323 (190+133) +145 (+66/+79)
52 115 (78+37) 289 (190+99) +174 (+112/+62)
1 93 (60+33) 287 (171+116) +194 (111/+83)
2 82 (46+36) 271 (162+109) +189 (+116/+73)
3 25 (15+10) 249 (165+84) +224 (+150/+74)
4 14 (8+6) 244 (176+68) +230 (+168/+62)
5 2 (0+2)
6 release!
Graphical overview of bug stats thanks to azhag:

Josselin Mouette: So(lusOS) I herd U liek mudslingingz?

After this announcement about the future of GNOME in Debian, I received several requests to look at SolusOS and their Consort fork of GNOME. On the paper, this doesn t look like a too bad idea, since they only forked gnome-panel and metacity, the two key components that are no longer maintained upstream. Still, starting a plain fork at the very moment when the former maintainer decided to give the key to anyone who wanted to, looked like a very destructive way of acting. Even if the fork gains momentum (which remains speculative), the amount of effort to rename packages makes you think twice before such a switch. I decided to go talk about it with them on IRC nevertheless. Mind you, they work on a GNOME fork and a Debian derivative, but deliberately use their own IRC server (you ll soon understand why). Just in case you would want to cooperate with SolusOS, you d have to use their infrastructure. It is an euphemism to say that the conversation didn t go well. This could have ended there, but thankfully for your already widening eyes, Ikey (the charming person I had the opportunity to discuss with) made the log public, in an attempt at public shaming that would soon gather his followers, chanting out loud how the revolutionary SolusOS would quickly replace every other Linux distribution. (As a side note, of the hundreds of such claims that were made public in the last 10 years, only one came true, and it was from a billionnaire who hired dozens of developers. Just saying.) Then you can understand why the specific IRC server: it is to K-line people who don t behave with enough deference to Ikey, because kickban is not enough. This log and the claims made in the comments look very ironical, if only for presenting a person working in a team of 10+ on the most popular desktop on one of the most popular desktop distributions, as a beggar. Even more ironical when it is the person who spent the most time on making GNOME Classic available and working nicely for wheezy. So when a dozen kids go chanting that Consort is so much better than GNOME and will replace it, while it just re-does the work I already did, let me at least smile. So what is SolusOS, after all? If you want to see where this story is going, there s a place that tells you: the consortium and consort-panel repositories. Here you will find interesting things: I ve stopped looking there. We re facing an ego issue, but not something that can improve the desktop for our users. In the end, I m sorry for those who genuinely thought we could do better by sharing some maintenance load with SolusOS, but it doesn t look like a big loss anyway.

Craig Small: Off to LCA2013

While I ve been involve in Linux and Debian for many (15 or more) years, I ve only ever been to one major Linux thing in all that time and that was manning some stall for Debian about 10 years ago. Well, let s call it two because next week I m off to the Linux.conf.au 2013 conference. I can t get too terribly excited about the location because its Canberra, yep my (now) home town and about 4 blocks from my workplace; so no exotic locales for me! I m hoping to catch up with some Debian folk while the conference is on. There is at the very least a keysigning party where there are some others from Debian on Wednesday. PS WordPress update broke the visual editing, I hope it looks ok.

Enrico Zini: On praising people, and on success

On praising people, and on success This morning I was pointing out to friends how excellent is mako's post on Aaron Swartz, and I thought it'd be nice if we didn't have to wait for people to die before telling the world how awesome and inspirational they are. Then Russ posted an article about work, success and motivation and I went to tell my friends how awesome and inspirational he is. I, too, see myself as somehow successful, and I, too, don't identify in the usual stereotype of success. I don't want to stop being a craftsman to become a manager, I don't get a high from having power over other people, I don't define my value in terms of my profits. At a glance, people don't see me as successful, until they get to know me better. They they realise that I'm not at all unhappy about my life. I have a job that I like, I write Free Software and it gets used and appreciated, my colleagues are friends, who respect me and my opinion, and I respect them and theirs. I can work from home. In fact, I can work from everywhere as long as I have my laptop with me. I can sustain a long distance relationship because I can work from the house of my partner when I'm visiting. Two days ago I worked from the bar of a farm on top of a hill, because I was on the road, it was close by, and what the hell, it's a wonderful place to be. To me success means that I can care about the quality of my life, that I have the luxury of caring about little things that make my day, of trying to make good ideas sustainable, of working a bit more when I'm on fire, and of working a bit less when there's something wonderful in the world to see, or someone interesting in the world to meet. Russ, the way I read your article, you are questioning what "success" means, and you are spot on. People should be able to define "success" as whatever works for them and pursue it freely. Only then success becomes something that is worth praising when it is achieved. Only then it becomes inspirational. I like how you managed to put into words something that has been for a long time in some corner of my mind and I hadn't yet managed or bothered to bring into the spotlight. You have the insight and the confidence of seeing something in an insightful but non-mainstream way, and say "you know what? That actually makes sense." Sometimes I read your post, nod a lot and realise how important something actually is, how that is actually such an important part of myself. And now that you took it out for me to see it, I can appreciate how valuable it is, and make sure I don't accidentally lose it. Thanks! That's another one I owe you. It's just the kind of thing I shouldn't wait before letting you know.

Johannes Schauer: Bootstrappable Debian - New Milestone

This post is about the port bootstrap build ordering tool (naming suggestions welcome) which was started as a Debian Google Summer of Code project in 2012 and continued to be developed afterwards. Sources are available through gitorious. In the end of November 2012, I managed to put down an approximation algorithm to the feedback arc set problem which allowed to break the dependency graph into a directed acyclic graph with only few removed build dependencies. I wrote about this effort on our mailinglist but didnt mention it here because it was still too much of a proof-of-concept. Later, in January 2013, I mentioned the result of this algorithm in an email wookey and me wrote to debian-devel mailinglist. Many things happened since November 2012:

Processing pipeline instead of monolithic tools The tools I developed so far tried to accomplish everything by themselves, reusing functionality implemented in a central library. Therefor, if one wanted to try out even trivial new things, it mostly meant to hack some OCaml code. Pietro Abate suggested to instead develop smaller tools which could work independently of each other, would only execute one algorithm each and could easily be connected together in different ways to achieve different effects. This switch is now done and all functionality from the old tools is moved into a new toolset. The exchange format between the tools is either plain text files in deb822 control format (Packages/Sources files) or a dependency graph. The dependency graph is currently marshalled by OCaml but future versions should work with just passing a GraphML (an XML graph format) representation around. This new way of doing things seems close to the UNIX philosophy (each program does one thing well, data is stored as text, every program is a filter). For example the deb822 control output can easily be manipulated using grep-dctrl(1) and there exist many tools which can read, analyze and manipulate GraphML. It is a big improvement over the old, monolithic tools which did not allow to manipulate any intermediate result by external, existing tools. Currently, a shell script (native.sh) will execute all tools in a meaningful order, connecting them together correctly. The same tools will be used for a future cross.sh but they will be connected differently. I wrote about a first proposal of what the individual tools should do and how they should be connected in this email from which I also linked a confusing overview of the pipeline. This overview has recently been improved to be even more confusing and the current version of the lower half (the native part) now looks like this: native reduced Solid arrows represent a flow of binary packages, dottend arrows represent a flow of source packages, ovals represent a set of packages, boxes with rounded corners represent set operations, rectangular boxes represent filters. There is only one input to a filter, which is the arrow connected to the top of the box. Outgoing arrows from the bottom represent the filtered input. Ingoing arrows to either side are arguments to the filter and control how the filter behaves depending on the algorithm. I will explain this better once the pipeline proved to be less in flux. The pipeline is currently executed like this by native.sh.

Two new ways to break dependency cycles have been discovered So far, we knew of three ways to formally break dependency cycles:
  • remove build dependencies through build profiles
  • find out that the build dependency is only used to build arch:all packages and therefor put it into Build-Depends-Indep
  • cross compile some source packages
Two new methods can be added to the above:

Dependency graph definition changed Back in September when I was visiting irill, Pietro found a flaw in how the dependency graph used to be generated. He supplied a new definition of the dependency graph which does away with the problem he found. After fixing some small issues with his code, I changed all the existing algorithms to use the new graph definition. The old graph is as of today removed from the repository. Thanks to Pietro for supplying the new graph definition - I must still admit that my OCaml foo is not strong enough to have come up with his code.

Added complexity for profile built source packages As mentioned in the introduction, wookey and me addressed the debian-devel list with a proposal on necessary changes for an automated bootstrapping of Debian. During the discussiong, two important things came up which are to be considered by the dependency graph algorithms:
  • profile built source packages may not create all binary packages
  • profile built source packages may need additional build dependencies
Both things make it necessary to alter the dependency graph during the generation of a feedback arc set and feedback vertex set beyond the simple removal of edges. Luckily, the developed approximation algorithms can be extended to support such changes in the graph.

Different feedback arc set algorithms The initially developed feedback arc set algorithm is well suited to discover build dependency edges which should be dropped. It performs far worse when creating the final build order because it only considers edges by itself and not how many edges of a source package can be dropped by profile building it. The adjusted algorithm for generating a build order is more of a feedback vertex set algorithm because instead of greedily finding the edge with most cycles through it, it greedily finds the source package which would break most cycles if it was profile built.

Generating a build order with less profile built source packages After implementing all the features above I now feel more confident to publish the current status of the tools to a wider audience. The following test shows a run of the aforementioned ./native.sh shell script. Its final output is a list of source packages which have to be profile built and a build order. Using the resulting build order, starting from a minimal build system (essential:yes, build-essential and debhelper), all source packages will be compiled which are needed to compile all binary packages in the system. The result will therefor be a list of source and binary packages which fulfill the following property:
  • all binary packages can be built from the available source packages
  • all source packages can be built with the available binary packages
I called this a "reduceded distribution" in earlier posts. The interesting property of this specific selection is, that it contains the biggest problem set of Debian when bootstrapping it: a 900 to 1000 nodes big strongly connected component. Here is a visualization of the problem: hideous mess Source packages do not yet come with build profiles and the cross build situation can not yet be analyzed, so the following assumptions were made: The last point is about 14 build dependencies which were decided to be broken by the feedback arc set algorithm but for which other data sources did not indicate that they are actually breakable. Those 14 are dynamically generated by native.sh. If above assumptions should not be too far from the actual situation, then not more than 73 source packages have to be modified to bootstrap a reduced distribution. This reduced distribution even includes dependency-wise "big" packages like webkit, metacity, iceweasel, network-manager, tracker, gnome-panel, evolution-data-server, kde-runtime, libav and nautilus. By changing one line in native.sh one could easily develop a build order which generates gnome-desktop or really any given (meta-)package selection. All of native.sh takes only 80 seconds to execute on my system (Core i5, 2.5 GHz, singlethreaded). Here is the final build order which creates 2044 binary packages from 613 source packages.
  1. nspr, libio-pty-perl, libmcrypt, unzip, libdbi-perl, cdparanoia, libelf, c-ares, liblocale-gettext-perl, libibverbs, numactl, ilmbase, tbb, check, libogg, libatomic-ops, libnl($), orc($), libaio, tcl8.4, kmod, libgsm, lame, opencore-amr, tcl8.5, exuberant-ctags, mhash, libtext-iconv-perl, libutempter, pciutils, gperf, hspell, recode, tcp-wrappers, fdupes, chrpath, libbsd, zip, procps, wireless-tools, cpufrequtils, ed, libjpeg8, hesiod, pax, less, dietlibc, netkit-telnet, psmisc, docbook-to-man, libhtml-parser-perl, libonig, opensp($), libterm-size-perl, linux86, libxmltok, db-defaults, java-common, sharutils, libgpg-error, hardening-wrapper, cvsps, p11-kit, libyaml, diffstat, m4
  2. openexr, enca, help2man, speex, libvorbis, libid3tag, patch, openjade1.3, openjade, expat, fakeroot, libgcrypt11($), ustr, sysvinit, netcat, libirman, html2text, libmad, pth, clucene-core, libdaemon($), texinfo, popt, net-tools, tar, libsigsegv, gmp, patchutils($), dirac, cunit, bridge-utils, expect, libgc, nettle, elfutils, jade, bison
  3. sed, indent, findutils, fastjar, cpio, chicken, bzip2, aspell, realpath, dctrl-tools, rsync, ctdb, pkg-config($), libarchive, gpgme1.0, exempi, pump, re2c, klibc, gzip, gawk, flex-old, original-awk, mawk, libtasn1-3($), flex($), libtool
  4. libcap2, mksh, readline6, libcdio, libpipeline, libcroco, schroedinger, desktop-file-utils, eina, fribidi, libusb, binfmt-support, silgraphite2.0, atk1.0($), perl, gnutls26($), netcat-openbsd, ossp-uuid, gsl, libnfnetlink, sg3-utils, jbigkit, lua5.1, unixodbc, sqlite, wayland, radvd, open-iscsi, libpcap, linux-atm, gdbm, id3lib3.8.3, vo-aacenc, vo-amrwbenc, fam, faad2, hunspell, dpkg, tslib, libart-lgpl, libidl, dh-exec, giflib, openslp-dfsg, ppl($), xutils-dev, blcr, bc, time, libdatrie, libpthread-stubs, guile-1.8, libev, attr, libsigc++-2.0, pixman, libpng, libssh2, sqlite3, acpica-unix, acl, a52dec
  5. json-c, cloog-ppl, libverto, glibmm2.4, rtmpdump, libdbd-sqlite3-perl, nss, freetds, slang2, libpciaccess, iptables, e2fsprogs, libnetfilter-conntrack, bash, libthai, python2.6($), libice($), libpaper, libfontenc($), libxau, libxdmcp($), openldap($), cyrus-sasl2($), openssl, python2.7($)
  6. psutils, libevent, stunnel4, libnet-ssleay-perl, libffi, readline5, file, libvoikko, gamin, libieee1284, build-essential, libcap-ng($), lcms($), keyutils, libxml2, libxml++2.6, gcc-4.6($), binutils, dbus($), libsm($), libxslt, doxygen($)
  7. liblqr, rarian, xmlto, policykit-1($), libxml-parser-perl, tdb, devscripts, eet, libasyncns, libusbx, icu, linux, libmng, shadow, xmlstarlet, tidy, gavl, flac, dbus-glib($), libxcb, apr, krb5, alsa-lib
  8. usbutils, enchant, neon27, uw-imap, libsndfile, yasm, xcb-util, gconf($), shared-mime-info($), audiofile, ijs($), jbig2dec, libx11($)
  9. esound, freetype, libxkbfile, xvidcore, xcb-util-image, udev($), libxfixes, libxext($), libxt, libxrender
  10. tk8.4, startup-notification, libatasmart, fontconfig, libxp, libdmx, libvdpau, libdrm, libxres, directfb, libxv, libxxf86dga, libxxf86vm, pcsc-lite, libxss, libxcomposite, libxcursor, libxdamage, libxi($), libxinerama, libxrandr, libxmu($), libxpm, libxfont($)
  11. xfonts-utils, libxvmc, xauth, libxtst($), libxaw($)
  12. pmake, corosync, x11-xserver-utils, x11-xkb-utils, coreutils, xft, nas, cairo
  13. libedit, cairomm, tk8.5, openais, pango1.0($)
  14. ocaml, blt, ruby1.8, firebird2.5, heimdal, lvm2($)
  15. cvs($), python-stdlib-extensions, parted, llvm-2.9, ruby1.9.1, findlib, qt4-x11($)
  16. xen, mesa, audit($), avahi($)
  17. x11-utils, xorg-server, freeglut, libva
  18. jasper, tiff3, python3.2($)
  19. openjpeg, qt-assistant-compat, v4l-utils, qca2, jinja2, markupsafe, lcms2, sip4, imlib2, netpbm-free, cracklib2, cups($), postgresql-9.1
  20. py3cairo, pycairo, pam, libgnomecups, gobject-introspection($)
  21. gdk-pixbuf, libgnomeprint, gnome-menus, gsettings-desktop-schemas, pangomm, consolekit, vala-0.16($), colord($), atkmm1.6
  22. libgee, gtk+3.0($), gtk+2.0($)
  23. gtkmm2.4, poppler($), openssh, libglade2, libiodbc2, gcr($), libwmf, systemd($), gcj-4.7, java-atk-wrapper($)
  24. torque, ecj($), vala-0.14, gnome-keyring($), gcc-defaults, gnome-vfs($)
  25. libidn, libgnome-keyring($), openmpi
  26. mpi-defaults, dnsmasq, wget, lynx-cur, ghostscript, curl
  27. fftw3, gnupg, libquvi, xmlrpc-c, raptor, liboauth, groff, fftw, boost1.49, apt
  28. boost-defaults, libsamplerate, cmake, python-apt
  29. qjson, qtzeitgeist, libssh, qimageblitz, pkg-kde-tools, libical, dwarves-dfsg, automoc, attica, yajl, source-highlight, pygobject, pygobject-2, mysql-5.5($)
  30. libdbd-mysql-perl, polkit-qt-1, libdbusmenu-qt, raptor2, dbus-python, apr-util
  31. rasqal, serf, subversion($), apache2
  32. git, redland
  33. xz-utils, util-linux, rpm, man-db, make-dfsg, libvisual, cryptsetup, libgd2, gstreamer0.10
  34. mscgen, texlive-bin
  35. dvipng, luatex
  36. libconfig, transfig, augeas, blas, libcaca, autogen, libdbi, linuxdoc-tools, gdb, gpm
  37. ncurses, python-numpy($), rrdtool, w3m, iproute, gcc-4.7, libtheora($), gcc-4.4
  38. libraw1394, base-passwd, lm-sensors, netcf, eglibc, gst-plugins-base0.10
  39. libiec61883, qtwebkit, libvirt, libdc1394-22, net-snmp, jack-audio-connection-kit($), bluez
  40. redhat-cluster, gvfs($), pulseaudio($)
  41. phonon, libsdl1.2, openjdk-6($)
  42. phonon-backend-gstreamer($), gettext, libbluray, db, swi-prolog, qdbm, swig2.0($)
  43. highlight, libselinux, talloc, libhdate, libftdi, libplist, python-qt4, libprelude, libsemanage, php5
  44. samba, usbmuxd, libvpx, lirc, bsdmainutils, libiptcdata, libgtop2, libgsf, telepathy-glib, libwnck3, libnotify, libunique3, gnome-desktop3, gmime, glib2.0, json-glib, libgnomecanvas, libcanberra, orbit2, udisks, d-conf, libgusb
  45. libgnomeprintui, libimobiledevice, nautilus($), libbonobo, librsvg
  46. evas, wxwidgets2.8, upower, gnome-disk-utility, libgnome
  47. ecore, libbonoboui
  48. libgnomeui
  49. graphviz
  50. exiv2, libexif, lapack, soprano, libnl3, dbus-c++
  51. atlas, libffado, graphicsmagick, libgphoto2, network-manager
  52. pygtk, jackd2, sane-backends, djvulibre
  53. libav($), gpac($), ntrack, python-imaging, imagemagick, dia
  54. x264($), matplotlib, iceweasel
  55. strigi, opencv, libproxy($), ffms2
  56. kde4libs, frei0r, glib-networking
  57. kde-baseapps, kate, libsoup2.4
  58. geoclue, kde-runtime, totem-pl-parser, libgweather, librest, libgdata
  59. webkit
  60. zenity, gnome-online-accounts
  61. metacity, evolution-data-server
  62. gnome-panel
  63. tracker
The final recompilation of profile built source packages is omitted. Source packages marked with a ($) are selected to be profile built. All source packages listed in the same line can be built in parallel as they do not depend upon each other. This order looks convincing as it first compiles a multitude of source packages which have no or only few build dependencies lacking. Later steps allow fewer source packages to be compiled in parallel. The amount of needed build dependencies is highest in the source packages that are built last.

Neil Williams: Hard drive death

My 1Tb laptop drive started misbehaving a few weeks ago, just spending a lot of time spinning when it should have been reading frequently changed files (like the browser cache). I was tempted to blame the browser at that point as no other packages appeared affected.

Wednesday night, a routine package upgrade on unstable brought in a bunch of qt4 updates which I wanted and a virtual box update that I didn't see much point in delaying ... 15 minutes later I couldn't work out why the virtualbox dkms task was still running, spotted it spinning in depmod and found some alarming messages in dmesg about short reads and errors reading from the hard drive. Hmm. The hard drive was out of control at this point, it quickly became apparent that no disc access was going to be possible and I couldn't get to a terminal to kill the current tasks, so I killed the power. The fsck which followed reboot showed more of the errors I'd seen in dmesg and fell back to the manual intervention stage. A few hours of confirming attempts to fix the errors, fsck finally finished. A short amount of usage showed that although fsck had finished, the drive was not happy and was starting to give short reads on other parts of the filesystem, resulting in ~40% of the filesystem appearing to be read-only when the rest was read-write. Somehow, I didn't think this was a welcome feature as the areas affected appeared to be quite random.

The drive in question was a replacement for the original 300Gb drive supplied with the ThinkPad, so a quick bit of switching of drives into a caddy and I could rsync my data off the large drive onto the smaller one. The rsync itself took a lot longer than it should have done because it got lots of short reads too. (Principally in /lib/modules/3.2.0-4/ and in the browser cache directories which had been the original symptom as well as most other places where one could have expected processes to have open files when the drive failed).

Now the 1Tb drive was a pig to fit into the Thinkpad originally because it was too big for the bay but I fitted it anyway. Yes, that was probably a mistake. It certainly meant that in order to fit it I had to not put the drive into the useful caddy provided by Lenovo which makes removal of the drive simple. Indeed the drive was wedged into the bay so tightly that it wasn't going to come out with normal levels of persuasion. This probably contributed to the failure of the drive, so live and learn.

With help from Andy Simpkins (it's always handy to have a hardware engineer on hand at times like this), the keyboard was lifted out, the case was dismantled and just enough room was made to get a screwdriver in behind the sata drive and lever it out of the bay. OK, rebuild laptop, put replacement drive into the caddy (because the smaller capacity drive is also a lot smaller in height than the 1Tb and therefore has plenty of clearance between it and the bay) and move on to the software recovery stage.

Hint: if this happens again, before turning off the broken system for the last time, just remember to download a recent Debian ISO to a USB stick - it saves having to ask someone else or find another machine to do the download. (Thanks Andy...)

OK, so after the usual complaints on reboot that there was no operating system, F12 got the boot order menu up and I was in Debian Installer Rescue Mode. Reinstalling grub failed initially for a few reasons:

  1. I'd deliberately not copied /dev /proc and /sys from the old harddrive but I'd also forgotten to create empty ones on the new drive...
  2. /etc/fstab helpfully referred to / as a UUID which no longer applied, so that had to be edited
  3. grub was confused and couldn't find the root filesystem

A few iterations later, I had a working /dev directory inside the /target chroot, bind mounted from the /dev outside the chroot, I was able to mount proc and sys, so grub was finally happy to reinstall itself and then update the initramfs setup for the new drive.

Reboot, another fsck, all appeared well. I was able to login via the terminal but not in X. Hmmm. Stop xdm, startx manually from the terminal, problems with /tmp/ - permission denied. Oops. Yes, it does help to create /tmp with the right permissions....

The final stage was to complete the dpkg --configure -a from the original failure. I've yet to reinstall the 3.2.0-4 kernel, so I'm back on 3.2.0-3 but that's OK.

So now I'm back on the original drive, albeit temporarily without any swap whatsoever (because I didn't partition the replacement drive to create /dev/sda5 before doing the copy) and now I remember the second reason I wanted to replace the original drive with the 1Tb drive - the original drive is as NOISY as hell. The whole edge of the laptop vibrates constantly, to the point that I can feel the vibration under the keys as I type. It's not that the drive is loose in the bay, it's just a constant vibration.

But, I have kept all my data and I have a usable laptop for the BSP this weekend. I will be looking at an SSD drive to replace this one though and having also found my old Acer laptop with power supply, I can now reference this entry when I transfer the system a second time.

Russ Allbery: The "Why?" of Work

(This is going to be long and rambling. Hopefully at some point I'll be able to distill it into something shorter.) In preparation for a tech leads retreat tomorrow, several of us at work were asked to watch Simon Sinek's TED talk, "How great leaders inspire action". I'll be honest with you: I hated this talk. Sinek lost me right at the start by portraying his idea as the point of commonality among great leaders (don't get me started on survivorship bias; it's a pet peeve) and then compounded the presentation problem with some dubious biology about brain structure. So, after watching it, I ranted a bit about how much I disliked it (to, as it turns out, people who had gotten a lot out of it). (Don't do this, btw. It's nearly always worthwhile to suppress negativity about something someone else enjoyed. I say this to increase the pool of people who can remind me of what I said the next time I forget. Which, if my normal pattern holds, will be about five minutes from now.) Thankfully, I work with tolerant and forgiving people who kindly pointed out the things they saw in the video that I missed, and we ended up having a really good hour and a half discussion, which convinced me that there's an idea under here that's worth talking about. It also helped clarify for me just how much I hate the conventional construction of both leadership and success. This talk is framed around getting other people to do things, which is one of the reasons why I had such a negative reaction to it. It's right there in the title: leaders inspiring action. This feeds into what at least in the United States is an endemic belief that the world consists of leaders and followers, and that the key to success in the world (in business, in politics, in everything else) is to become one of the leaders and accumulate followers (most frequently as customers, since we have a capitalist obsession). This is then defined as success. I think this idea of success is bullshit. Now, that statement requires an immediate qualification. Saying that the societal definition of success is bullshit is a statement from privilege. I have the luxury of saying that because I am successful; I'm in a position where I have a lot of control over my own job, I'm not struggling to make ends meet, and I can spend my time pondering existential questions like how to define success. If I were a little less lucky, success would be whatever put food on the table and kept a roof over my head. I'm making an argument from the top of Maslow's hierarchy. But that is, in a roundabout way, my point: why is defining and constructing success still so hard, and why do we do such a bad job at it, even when we're at the top of the pyramid and able to focus on self-actualization? The context of this talk for my group is pre-work for a discussion about, in Sinek's construction, the "why?" of our group. Why are we here, what is our purpose, and what do we care about? By the mere fact that we are able to ask questions like that, you can correctly surmise that we're already successful. The question, therefore, is what should we do with that success? I normally hear one or more of the following answers, all of which I find unsatisfying or problematic. So, what should I do with success? Or, put another way, since I have the luxury of figuring out a "why?", what's my "why?" This question comes at a good time. As I've mentioned the last couple of days here, I've just come off of two days of the most fun I've had at work in the last several years. I spent about 25 hours total building a log parsing infrastructure that I'm quite fond of, and which may even be useful to other people. And I did that in response to a rather prosaic request: produce a report of user agents by authenticated unique users, rather than by hits, so that we can get an idea of what percentage of our community uses different devices or browsers. This was a problem that I probably could have solved adequately enough for the original request in four hours, maybe less, and then moved on to something else. I spent six times that long on it. That's something I can do because I'm successful: that's the sort of luxury you get when you can define how you want to do your job. So, apparently I have an answer to my question staring me in my face: what I do with success, when I have it, is use that leeway to produce elegant and comprehensive solutions to problems in a way that fully engages me, makes the problem more interesting, and constructs infrastructure that I can reuse for other problems. Huh. That sounds like a "why?" response that's quite common among hackers and software developers. Nothing earth-shattering there... except why is that so rare in a business context? Why isn't it common to answer questions like "what is our group mission statement" with answers like that? This is what I missed in the TED talk, and what the subsequent discussion with my co-workers brought to light for me. I think Sinek was getting at this, but I think he buried the lede. The "why?" should be something that excites you. Something that you're passionate about. Something that you believe in. He says that's because other people will then believe in it too and will buy it from you. I personally don't care about (or, to be honest, have active antipathy towards) that particular outcome, but that's fine; that's not the point. The point is that a "why?" comes from the heart, from something that actually matters, and it creates a motivating and limiting principle. It defines both what you want to do and what you don't want to do. That gives me a personal answer. My "why?" is that I want to build elegant solutions to problems and do work that I find engaging and can be proud of afterwards. I automate common tasks not because I particularly care about being efficient, but because manually doing common tasks is mind-numbing and boring, and I don't like being bored. I write reliable systems not particularly because that helps clients, but primarily because reliable software is more elegant and beautiful and unreliable software offends me. (Being more usable and frustrating for clients is also good; don't get me wrong. It's just not a motive. It's an outcome.) What does that mean for a group mission statement, a group "why?" Usually these exercises produce some sort of distillation of the collective job responsibilities of the people in the group. Our mission is to maintain core infrastructure to let people do their work and to support authentication and authorization services for the university, yadda yadda yadda... this is all true, in its way, but it's also boring. One can work oneself up to caring about things like that, but it requires a lot of effort. But we all have individual "why?" answers, and I bet they look more like my answer than they do like traditional mission statements. If we're in a place where we have the luxury of worrying about self-actualization questions, what gets us up in the morning, what makes it exciting to go into work, is probably some variation on doing interesting and engaging work. But it's probably a different variation for everyone in the group. For example, as you can see from above, I like building things. My happiest moments are when someone gives me a clearly-defined problem statement that fills a real need and then goes away and leaves me in peace to solve it. One thing I've learned is that I'm not very good at coming up with the problem statements myself; I can do it, but usually I end up solving some problem that isn't very important to other people. I love it when my employer can hand me real problems that will make the world better for people, since often they're a lot more interesting (and meaningful) than the problems I come up with on my own. But that's all highly idiosyncratic and is not going to be shared by everyone in my group. I'm an introvert; the "leave me alone" part of that is important. Other people are extroverts; what gets them up in the morning is, in part, engaging with other people. Some people care passionately about UI design. (I also care passionately about UI design, but the UI designs that I'm passionate about are the ones that are natural for my people, who are apparently aliens from another galaxy, so I'm not the person you want doing UI design for things used by humans.) Others might be particularly interested in researching new technology, or coming up with those problem statements, or in smoothly-running production systems, or in metrics and reporting... I don't really know, but I do know that there's no one answer that fits everyone. Which means that none of our individual "why?" responses should become the group "why?". However, I think that leads to an answer, and it's the answer I'm going to advocate for in the meeting tomorrow. I believe the "why?" of our team should be to use the leeway, trust, and credibility that we have because we're successful to try to create an environment in which every individual member of the team can follow their own "why?" responses. In other words, I think the mission of our group should not be about any specific technology, or about any specific set of services, or outcomes. The way we should use our success is to let every member of our team work in a way that lights their fire. That makes them excited to come into work. That lets each of us have as much fun as I had in the past two days. We should have as our goal to create passionate and empowered employees. Nothing more, but nothing less. This is totally not how group mission statements are done. They're always about blending in to some larger overall technological purpose. But I think that's a mistake, and (despite disliking the presentation), I think that's what this TED talk does actually get at. The purpose is the what, or sometimes the how. It's not the why. And the why isn't static; technology is changing fast, and people are using technology in different ways. Any mission statement around technology today is going to be obsolete in short order, and is going to be too narrow. But I think the flip side is that good technological solutions to the problems of the larger organization are outcomes that fall out of having passionate and inspired employees. If people can work in ways that engage and excite them, they will end up solving problems. We're all adults; we know that we're paid to do a job and that job needs to involve solving real problems for the larger organization. All of that is obvious, and therefore none of that belongs in a mission statement. A mission statement should state the inobvious. And while some visionary people can come up with mission statements around technology or around how people use technology that can be a rallying point for a team or organization, I think that's much rarer than people like to think it is. If you stumble across one like that, great, but I think most teams, and certainly our team, would be better served by having the team mission statement be to enable every individual on the team to be passionate about their work. What should our group work on next? Figure out what the university's problems are, line those needs up with the passions of the members of the team, ask the people most excited about each problem how they want to solve that problem, and write down the answers. There's our roadmap and our strategy, all rolled into one.

24 January 2013

Benjamin Mako Hill: Aaron Swartz

I moved to Boston in 2005 at the same time that Aaron Swartz did and we were introduced by a mutual friend. Aaron was one of my first friends in Boston and we became close. When Aaron moved to San Francisco, I moved into his apartment in Somerville where he kept a room for a year or so. Mika and I still live there. His old posters remain on our walls and his old books remain on our shelves. Aaron s brothers Ben and Noah both lived with us and remain close friends. I have spent hours (days?) reading and thinking about Aaron over the last two weeks. It has been disorienting but beautiful to read the descriptions of, and commentaries on, Aaron s life. Although I suspect I may never feel ready, there are several things I want to say about Aaron s death, about Aaron s work, and about what Aaron means to me. 1. Aaron s Death The reaction to Aaron s death has been overwhelming and inspirational. At some point in the near future I plan to join some of the important campaigns already being waged in his name. There are many attempts to understand why Aaron died and many attempts to prevent it from happening to others in the future. Unfortunately, I am familiar with the process of soul-searching and second-guessing that happens when a friend commits suicide. I m sure that every one of his friends has asked themselves, as I have, What could I have done differently? I don t know the answer, but I do know this: Aaron was facing the real risk of losing half his life to prison. But even if one believed that he was only facing the likely loss of ten percent or even one percent of his life I wish that we all, and I wish that I in particular, had reacted with the passion, time, anger, activity, and volume proportional to how we have reacted in the last two weeks when he lost the whole thing. 2. Aaron s Work Of course, Aaron and I worked on related projects and I followed his work. And despite all the incredible things that have been said about Aaron, I feel that Aaron s work was more focused, more ambitious, more transformative, more innovative, and more reckless (in a positive sense) than the outpour online suggests. Although discussion of Aaron has focused his successes, achievements, and victories, the work that inspired me most was not the projects that were most popular or successful. Much of Aaron s work was deeply, and as it turned out overly, ambitious. His best projects were self-conscious attempts to transform knowledge production, organization, and dissemination. Although he moved from project to project, his work was consistently focused on bringing semantic-web concepts and technologies to peer production, to the movement for free culture, and to progressive political activism and on the meta-politics necessary to remove barriers to this work. For example, Aaron created an online collaborative encyclopedia project called the TheInfoNetwork (TIN) several years before Wikipedia was started. I talked to Aaron at length about that project for a research project I am working on. Aaron s work was years ahead of its time; in 2000, TIN embraced more of the Wikimedia Foundation s current goals and principles than Wikipedia did when it was launched. While Wikipedia sought to create a free reference work online, Aaron s effort sought to find out what a reference work online could look like. It turned out to be too ambitious, perhaps, but it taught many, including myself, an enormous amount in that process. When I met Aaron, he was in the process starting a company, Infogami, that was trying to chase many of TIN s goals. Infogami was conceived of as a wiki aware of the structure of data. The model was both simple and profound. Years later, Wikimedia Deutschland s WikiData project is beginning to bring some of these ideas to the mainstream. Infogami merged with Reddit as equal halves of a company with a shared technological foundation based on some of Aaron s other work. But when Reddit took off, Infogami was rarely mentioned, even by Aaron. I think that is too bad. Reddit got traction because it made the most popular stuff more visible; Reddit is popular, fundamentally, because popular things are popular. But popular is not necessary positive. For that reason, Reddit never struck me as either surprising or transformative. But what started as Aaron s half the company, on the other hand, aimed to create a powerful form of democratized information production and dissemination. And although Infogami didn t take off, the ideas and code behind the project found life at the heart of Open Library and will continue to influence and inspire countless other projects. I believe that Infogami s lessons and legacy will undergird a generation of transformative peer production technologies in a way that the Reddit website important as it is will not. 3. What Aaron Means to Me A lot of what has been written about Aaron speaks to his intelligence, his curiosity, his generosity, his ethics and his drive. Although I recognize all these qualities in the Aaron I knew, I ve felt alienated by how abstract some of the discussion of Aaron has been my memories are of particularities.
I remember the time Aaron was hospitalized and I spent two hours on the phone going through my bookshelves arguing with him about the virtues of the books in my library as we tried to decide which books I would bring him. I remember Aaron confronting Peter Singer intellectual founder of the modern animal rights movement at the Boston Vegetarian Food Festival to ask if humans had a moral obligation to stop animals from killing each other. I lurked behind, embarrassed about the question but curious to hear the answer. (Singer sighed and said yes sort of and complemented Aaron on the enormous Marxist commentary he was carrying.) I remember 1-800-INTERNET.com. I remember talking with Aaron about whether being wealthy could be ethical. I argued it could not but Aaron argued uncharacteristically I thought that it could. Aaron told Mika she should slap him if he ever became wealthy. The very next day, it was announced that his company had been acquired and that Aaron was a millionaire. I remember the standing bets I had with Aaron and how he would email me every time news reports favored his claims (but never when they did not). And I remember that I won t hear from him again.
Aaron was a friend and inspiration. I miss him deeply and I am very sad.

Jan Wagner: vsftpd running on IPv4 and IPv6 simultaneous

Today I planed to move my FTP server to a new system running Debian wheezy. Usually I'm running vsftpd, so this was also the plan for the new host. As this system is dual stacked, I wanted to offer the service on IPv6 too.
On the old system I started vsftpd via xinetd, as far as I remember was vsftpd in squeeze not able to bind on the IPv4 and the IPv6. Anyways ... I didn't wanted to use any inetd system ... So I looked if there is any way to solve that. By a quick search I found vsftpd-2.3.4-listen_ipv6.patch, which indicated that just binding to [::] will also accept IPv4 connections. Setting the following in /etc/vsftpd.conf worked out like a charm:
listen=NO
listen_ipv6=YES
I just updated #574837 so hopefully more people can benefit in the future.

Craig Small: Bootstrap The hidden gem in Turbogears

I ve been trying to tidy up my Mako templates within my Turbogears 2 project. As part of that I was looking at some of them that are quickstarted including one which is the About page. What was curious was there was all this CSS work all done already for you, including icons. Digging further I found out that one of the many projects Turbogears takes in, is Bootstrap. It s website describes Bootstrap as Sleek, intuitive, and powerful front-end framework for faster and easier web development. but what it means to me is a bunch of guys who understand HTML and CSS way better than me have made it easy for me to build a decent web application. While the framework won t suit everyone s tastes, it makes a lot of the formatting decisions so much easier. The thing is, in all the Turbogears documentation I have read I ve not heard it mentioned. Not sure why, its pretty awesome (not SQLAlchemy awesome but not many things are).

Russell Coker: Power Supplies and Wires

For some time I ve been wondering how the wire size for power supplies limits the power. So I ve done some quick calculations to determine if it s a problem. The first type that is of interest are the Inverters that are used to convert 12VDC to 240VAC (mains power) to allow electric devices to be operated in a car. I ve seen some reports from dissatisfied user about Inverters not supplying as much power as expected and I ve had problems with my 150W Inverter not always supplying my Thinkpad (which definitely doesn t draw 150W). The second type is phone chargers as charging a phone in a reasonable amount of time is always a problem. Inverter Rating Fine Print vs Laptop PSU My Thinkpad Power Supply claims Efficiency Level IV which according to the US EPA document describing the efficiency marking protocol for external power supplies [1] means that it is at least 85% efficient when supplying 50W+. The peak output of the PSU is 4.5A at 20V which is 90W peak output, 90/0.85 == 106W power drawn. One would hope that wouldn t be a problem from a 150W PSU. But the fine print on the PSU says that it can provide 110W continuously and 150W for 10 minutes. So according to my calculations I m within 4W of overloading the PSU if my Thinkpad uses full power. It also says that it is designed for 13.8V input. I have no idea how the performance of the Inverter changes as the supply Voltage changes between the 12.6V that a 6 cell lead-acid battery is designed to provide and the 13.8V charge from the car alternator. But I have had occasions when my Inverter stopped working correctly presumably due to being unable to supply as much current as my Thinkpad draws. As an aside I measured the Voltage in my car (with the engine off) at 12.85V from the cigarette lighter socket and 13.02V directly from the battery. I wonder if there is some sort of overload protection on the cigarette lighter which has a side effect of reducing the Voltage. Resistance in wires reduces the Voltage, but all Voltage meters are designed to have a high resistance to prevent that from being an issue. If anyone has an explanation for the 0.17Volt drop then please write a comment! Can a Car Provide 130W from the Cigarette Lighter socket? If the Inverter is also 85% efficient (and it might be less as it has no indication of efficiency on the box) then when supplying 110W it would draw 110/0.85 == 129.4W (I ll round it up to 130W). The power in Watts is equal to the Voltage multiplied by the current in Amps (W=V*I). Therefore I=W/V so if the car battery was at 12.85V then 130W/12.85V == 10.12A will flow. The current that goes through a circuit is equal to the Voltage divided by the resistance (see the Wikipedia page on Ohm s law for more information). This also means that the resistance equals the Voltage divided by the current. 12.85V/10.12A == 1.27 Ohms. Note that this is the resistance of the entire circuit, all the wires going to the battery, the circuitry inside the Inverter, and the internal resistance of the battery. The Inverter s cable is 1M long (2 meters of wire) and each wire is about 3.5mm in diameter including the insulation which means that the copper wire is probably equivalent to a single core conductor that is about 1mm in diameter. According to one of the online guides to resistance [2] wire that is 1.02mm in diameter will have a resistance of 0.02 Ohms per meter which gives a resistance of 0.04 Ohms. 0.04 Ohms is 3% of the total resistance of the circuit which doesn t seem like it will be a real problem. In practice I ve noticed that the connector gets extremely hot when it s in use while the cable doesn t get warm enough to notice. I suspect that the quality of the connector limits the power that is available but I don t have an easy way of measuring this. Inverters that are rated at 300W are designed to attach directly to the battery. An Inverter that is rated at 300W would draw 300W/0.85 == 352W from the battery. That needs 352W/13.02V == 27.04A and therefore a circuit resistance of 13.02V/27.04A == 0.48 Ohms total resistance. I wonder whether dirt on the battery terminals would give a significant portion of that. Phone Charging I ve also been wondering about why mobile phones take so long to charge, and now I ve finally done the calculations. The latest standard for mobile phones is to use USB for charging. The Wikipedia page about USB says that the standard is for USB 2.0 to supply up to 500mA at 5V +-5%. That means 0.5A*5V == 2.5W +- 5%. If we assume that the internal power supply in a phone is also 85% efficient then that means 2.5*0.85 == 2.125W going to the battery. My Samsung Galaxy S3 has a battery which is rated at 7.98Wh. According to the Wikipedia page about Lithium Ion batteries the charge/discharge efficiency is 80% to 90% I ll assume that it s 85% for further calculations. If the battery in the phone is 85% efficient and the phone is doing nothing but charging then the charge time for a regular USB port would be 7.98Wh/0.85/2.125W == 4.42 hours (4 hours 25 minutes) of charge time. That probably means something closer to 5 hours to totally charge the phone while it s running. There are dedicated charging ports for USB which can supply up to 1.5A. The 3rd party charger which came with my phone was rated at 1A and would hopefully be capable of completely charging the phone in less than 3 hours (but in practice isn t). It s interesting to note that MacBooks expose the amount of current drawn from a USB port with a GUI, so it should be possible to measure a phone charge rate by connecting it to a MacBook (which is cheaper than cutting up a phone cable). My old Samsung Galaxy S has a battery which is rated at 5.55Wh, by the same calculations it would take slightly more than 3 hours to charge on a standard USB port or 1.5 hours on my newest USB charger. In practice it has never got anywhere close to that, I presume that the phone is designed to draw less than 500mA. Phone Cable Resistance The charger that came with my Galaxy S has a cable that is about 1.75M long, the cable is flat and measures just over 1mm thick and about 2mm wide. Presumably the wire is equivalent to a single core that s about 0.4mm in diameter thus giving it a resistance of about 0.134 Ohm per meter, or 1.75*2*0.134 == 0.469 Ohm for the cable. The charger is rated at 0.7A. To supply 0.7A at 5V the resistance would be 5V/0.7A == 7.143 Ohm so about 6.6% of the total resistance of the circuit would be in the wire from the charger to the phone. The charger that came with my Galaxy S3 has a round cable that s just over 3mm thick and about 90cm long. If each wire in the cable is equivalent to a solid wire that is 0.912mm in diameter then it would be 0.0264 Ohm per meter of wire or 0.9*2*0.0264 == 0.0475 Ohm. The total circuit resistance would be 5V/1A == 5 Ohm. So 0.0475 Ohm is less than 1% of the circuit resistance. Voltage Drop The Voltage across a part of a circuit is proportional to the resistance (see the Wikipedia page on Series and Parallel Circuits for a good explanation). Basically this means that if 1% of the resistance of a circuit is in the wire then 1% of the Voltage drop will also be in the wire, so if we have a 5V supply with my Galaxy S3 cable then each of the two wires in the cable will have a difference of about 0.025V between the ends and the phone will receive a supply of 4.95V, the difference isn t something that is worth worrying about. But the cable from my Galaxy S has a resistance equivalent to 6.6% of the circuit resistance which means that the theoretical charge time will be 6% longer than it might be or 6% more current will be drawn from the mains than should be needed. Conclusion The charger that came with my Samsung Galaxy S isn t much good. Wasting 6.6% of the power in the wire is unreasonable. Phones keep getting more power hungry and batteries keep getting larger. There are third party phone batteries and external batteries that are charged by USB which have more than twice the capacity of the stock phone batteries this means more than twice the charge time. This problem will keep getting worse. The problem of a phone in active use drawing more power than the charger can provide (and running out of battery while on the charger) seems likely to stay with us. So while an Android phone has the potential to be a great little embedded server it seems that hacking the power supply is going to be a required first step for realising that potential. The decision to make 5V the USB power standard was reasonable at the time as it was the voltage used for most things on the motherboard. The decision to use USB as the phone charging standard was also reasonable, it allows phones to be charged anywhere. The combination of those two decisions isn t good for the user. If a higher Voltage such as 12V was used then 5* the power could be supplied through the same wires at the same level of efficiency. It would be really good if cars came with built in Inverters and supplied 240VAC or 110VAC depending on the region they were manufactured for. It s becoming a fairly common feature to have a cigarette lighter port in the car boot as well as at least two ports inside the car. When a car has three sockets and only one device to actually light cigarettes (which I suspect is only provided to fill an empty socket) it s very obvious that people want to connect random devices. Also having USB charging ports inside the car would be a really good idea (one for each seat would be good for Ingress).

Robert Collins: Launchpadlib without gnome-keyring

Recently I ve been doing my personal development SSH d into my personal laptop. I found that launchpadlib (which various projects use for release automation) was failing the gnome keyring API threw an error because the keyring was locked, and python-keyring didn t try to unlock it. I needed a workaround to be able to release stuff, and with a bit of digging and help from #launchpad, came up with this: mkdir ~/.cache/keyring
mkdir ~/.local/share/python_keyring
echo > ~/.local/share/python_keyring/keyringrc.cfg << EOF
[backend]
default-keyring=keyring.backend.UncryptedFileKeyring
keyring-path=/home/robertc/.cache/keyring/
EOF (There is already encryption in place, so I chose an uncrypted store read the keyring source to find other alternatives). With this done, I can now use lp-shell etc over SSH, for when I m not physically at my machine.

Next.

Previous.