Ten years ago, most of us wouldn't have dreamt we'd be managing terabits of storage, tens of megabits of bandwidth, arrays of network-distributed services. The height of a programmer's worry would likely be choice of UI toolkit or finding the right way to indent code, and the height of consumer concern deciding which room to put the new computer in. Now the problems associated with managing large networks are becoming real for everyone, right down to the consumer level. Stupendously large amounts of computing resource are available at an instant. Your household probably has more than a terabyte of storage already. Issues such as single sign-on are going to hit home over the next year, as networked computing and entertainment devices profilerate. Features such as Apple's Time Machine will be increasingly vital — software that makes traditionally gnarly sysadmin tasks consumer-friendly. The rebranding of .Mac into "Mobile Me" is also a step in this direction. The impact on developers As software developers, we also have to cope with the effects of this resource-richness. For small sums of money we can get access to large computing clusters, geographically redundant hosting services. Our programs have left the desktop and found their new home on the web. System administration issues loom large upon us, security concerns lurk auspiciously in the corners of our minds. Although the cost of infrastructure has dropped radically, other costs remain high and are going to stay that way. System administrators are not only grumpy, they demand high wages. Commercial software license fees spiral out of control: traditional per-CPU licensing models make little sense when you can quickly bring up tens of machines. The cost in power is already troubling large companies, and there's no reason to suspect the problems won't ripple down. Help is at hand from a variety of technologies. If they don't yet make massive resource management trivial, they at least make it possible. Some of these also inhabit the weird territory of being both the source of a problem and a solution at the same time: virtualization, for example. Distributed revision control systems Distributed revision control is a technology whose time has finally come in popular circles, thanks in part to Linus Torvald's Git system. DRCS has several important impacts on today's developer:

• Branching and, importantly, merging become much cheaper, allowing agile and flexible iterations of development.
• Loosely connected and geographically diverse development becomes much easier. Even within a single organization it is not uncommon to find teams spanning countries, time zones. Complex multi-site VPN setups aren't necessary when a few SSH keys can do the job.
• Revision control becomes packaging on the cheap. Like it or not, the mere tagging of a source tree has now become a valid option for releasing software.

All these trends lower the barrier to entry, increase collaboration and agility of development. You can the value of this as more software tools become free. Selling such tools is rapidly becoming a thing of the past, the advantages of sharing enable the developers at the sharp end to get their jobs done quicker. However, such increased agility and, well, messiness leave other problems to solve, which the next two technologies address. Virtualization Hardware-as-a-service, infrastructure-as-a-service, call it what you will. The ability to create what we used to call entire machines, pick them up and move them around the network is revolutionary, and it's something that will have a real impact on regular developers. The benefits are at several levels.

• Agile infrastructure — a ready supply of new machines makes it a lot easier and cheaper to test different scenarios, architectures, and to separate concerns. If things go wrong, throw away the image and start over. It's all about cutting the administration load.
• A packaging solution — the new macroeconomics of software distribution mean that distributing entire machine images which communicate exclusively via the network is now a feasible way to distribute your software. We must adjust to the notion of distributing appliances, not code. We may mourn the lost crafts of creating RPMs or installers, but let's face it, it's now a waste of time.
• New business models — your application can now be delivered as a black-box appliance, circumventing compatibility issues, or as a service, with virtualization part of the solution to scaling.

Configuration management Computing is a zero-sum game, and despite our increased ability to create and distribute software, problems still exist. We just pushed them to the next level. In good part, this next level is the problem of configuration management. We now have networks and clusters of (virtual) machines, software so agile we need six decimal places to describe its revision levels, and network and authentication paths that are starting to tangle. How do we manage that? One thing developers crave is repeatability. That's why we love our makefiles, autoconf, Ant, rake and so on. It's the one time even the most imperative-minded programmer writes declarative code. We like to say "let the world be like this." Our new sprawling world lacks this feature, and the best of our old toolkits — .debs, RPMs — address things only at the level of packages in a single environment. So developers must look to the world of operations, a territory we probably thought we needn't enter. In this world the new "make" is called Puppet. You write recipes to describe how things ought to be, and Puppet will make it so. I've been spending some time digging into Puppet, and feel excited by the confidence it's giving me. Now my applications exceed single source trees, and single machines, it gives me the means to tie the whole together. This article was going to be solely about Puppet, but that will have to wait now for another time. It's likely you'll have played with virtual machines and distributed revision control, but have you tried Puppet yet? Give it a spin, and let your mind wander over the benefits for your organization and development approaches. Conclusions
For developers and users alike, our world is changing. Hardware, connectivity and increasingly software is becoming cheap or free. The solidity of the old things we put value on — real things you can touch like disks — is eroding. What really matters is our data, our creations, and their communication. If they don't quite yet exist in a universal "cloud" yet, they're certainly getting frisky. As vendors provide solutions for consumers to manage their new domestic infrastructure, developers must look to network-aware toolkits and operations techniques to manage and get the best from their emergent infrastructures. Also on this topic:

• In search of agile infrastructure for web applications (June 2006)
  

Join the conversation about this post

There has been a lot of talk recently about the cost of petrol, Colin Charles is one of the few people to consider the issue of wages in this discussion [1]. Unfortunately almost no-one seems to consider the overall cost of running a vehicle. While I can’t get the figures for Malaysia (I expect Colin will do that) I can get them for Australia. First I chose a car that’s cheap to buy, reasonably fuel efficient (small) and common (cheap parts from the wreckers) - the Toyota Corolla seemed like a good option.

A search on www.drive.com.au shows adverts for Corollas made in 1990 and 1991 for prices ranging from $3000 to $5000. Assuming some optimism on behalf of the advertisers I could probably buy one for about $2500. The current term deposit rate from the major Australian banks is just over 8%, so spending $2500 on a car (instead of having the money in a term deposit) means a loss of $200 per annum interest, if instead the car was purchased with a bank loan the rate would be significantly higher (car loans have quite high interest rates). Of course the real cost is whenever you sell a car and buy a newer one that costs more - but this is harder to estimate. The next issue is insurance, for a 1990 Corolla of the smallest engine size (as a rule of thumb the smaller the engine the smaller the insurance bill) comprehensive insurance (sans optional extras such as windscreen protection) would cost me $489 per annum, and third-party fire and theft would cost me $398. Of course those rates are for someone who is more than 25 years old and has a good driving record. The majority of people who “flip burgers” appear to be younger than 25 and would therefore pay more. Car registration starts at about $550 per annum (varies by state). Car service would cost at least $200 a year (for scheduled services). So it seems that the cost of a cheap car is around $1,350 per annum before you even drive it! Now with a car that’s 18 years old you have to expect that some parts are wearing out, including parts that make it unroadworthy and must be replaced. Assuming that $300 per annum is spent on parts and labor to repair parts that wear out seems quite conservative (especially if driving a car that costs less than $3,000 - one of the cheaper cars in that age range). That brings the total cost per year to $1,650 if you are lucky (if unlucky then expensive parts could wear out at financially inconvenient times). The amount of driving that I do is around 5,000 to 7,000km per annum (according to the ABS the average passenger vehicle was driven 14,100Km in 2005 [2]). Assuming 7.5L/100Km (one of the results that has been quoted for fuel efficiency of a car of that age) and 7,000Km of driving that would be 525L of petrol per year, at $1.70 that is $892. So at current fuel prices when driving a small car for minimal distances (either for environmental or financial reasons) the cost of petrol may be about 35% of the vehicle operating cost. If driven for the average distance the fuel cost will be about 52% of the total vehicle operating cost (which is about $3,400) - which seems like a reasonable incentive to save fuel. If someone wants to advocate measures to make cars more affordable for poor people then the first thing to consider would be a reduction in the vehicle registration fees (which are a significant part of the vehicle operating costs). Another would be government incentives for car sharing schemes (see my previous post on car sharing for details on what is on offer in Melbourne at the moment [5]), $3,400 would give 340 hours (or more with the weekend deals) of driving a shared car (including fuel) at current prices. A shared car would not be suitable for driving to work but would be good for most other car use. Also it should be noted that share cars are new and have all the latest safety features - I really wouldn’t want to be involved in a crash while driving a 20yo car. Unlike the costs of registration etc for owning a car, all the costs of a shared car will be reduced if you use it less. The tax rate for someone earning $28,579.20 is $2,850 plus 30c for each $1 over $25,000 [3], which is $3,923.76, which leaves a total after tax income of $24,655.44. Someone working 40 hours a week on the minimum income who drives an average distance will spend about 14% of their after-tax income on their car if they have a small old car and were able to buy it without borrowing money. According to the ABS the median income is actually slightly lower than full-time work at the minimum wage [4]. So it seems that there are many people who are working part-time and earning less money than they would if they worked full-time at the minimum wage. So the question for Colin is, how does this compare to Malaysia? Is someone on a minimal income in Malaysia going to spend ~7% of their after-tax income on paying insurance, tax, and basic maintenance on a car without the additional expense of driving it? How does the cost of the basic car ownership compare to the cost of fuel in Malaysia?

• [1] http://www.bytebot.net/blog/archives/2008/06/15/on-fuel-subsidies-and-earningspending-power


• [2] http://www.abs.gov.au/AUSSTATS/abs%40.nsf/mediareleasesbyReleaseDate/66E24E957EB09229CA2571E10077D71D?OpenDocument


• [3] http://www.ato.gov.au/individuals/content.asp?doc=/content/12333.htm&mnu=5053&mfp=001


• [4] http://www.abs.gov.au/AUSSTATS/abs%40.nsf/7d12b0f6763c78caca257061001cc588/96e08af8f4178a45ca257306000d4fd4!OpenDocument


• [5] http://etbe.coker.com.au/2008/04/13/car-sharing-in-melbourne/

Share This

As I've mentioned before, one of the major problems facing reliable suspend/resume is graphics reinitialisation. Modern AMD and Nvidia chipsets contain a set of bytecode scripts making up information on how to program a specific card. At boot time the BIOS executes a small native interpreter that runs these. After boot, the driver can execute the scripts itself. This removes the need to have the driver make BIOS calls, and also means that the driver can filter out commands that it feels are undesirable at the time.

The nouveau project have reverse engineered the majority of the BIOS script format for older nvidia hardware. I've stolenmade use of their code by wedging it into a kernel module. On suspend various bits of register state are saved - on resume, they're poked back into the card and the init scripts are run. This is enough to get me some semblance of graphics back on my nvidia laptop, which previously didn't support this at all.

There's a few caveats. First, it doesn't support nv50 based hardware (8000 series and above). There's a hacky alternative method that basically works there for now. Secondly, it doesn't restore text mode correctly. Your consoles won't work again. Thirdly, it doesn't reinitialise the 3D core at all. If you're using nouveau then this means that it'll wedge itself unless you disable acceleration (option "NoAccel" "true" in xorg.conf). If you do use it with nouveau, you need to load the nv_bios module before the nouveau module is probed, since otherwise it'll grab the PCI device. And fourthly, this obviously won't work at all with the binary nvidia drivers. They ought to be doing much the same thing anyway.

The code's here, and I'm interested in finding out how well it works. Especially on desktops, which I can't test myself.

Thanks go to Stuart Bennett, who not only took a painfully bad laptop off my hands but also helped me hugely in working out why the existing code didn't quite work on my hardware (the ptimer values need saving and restoring, for those who have an excessive degree of interest)

Colin Charles writes about a woman who is selling advertising space on herself [1]. Like Colin I haven’t bought a t-shirt in about 9 years (apart from some Cafepress ones I designed myself). So it seems that the price for getting some significant advertising at a computer conference is to buy a few hundred t-shirts (they cost $7 each when buying one at a time from Cafepress, I assume that the price gets lower than $3 each when buying truck-loads). I have been given boxer-shorts and socks with company logos on them (which I never wore), I think that very few people will show their underwear to enough people to make boxer-shorts a useful advertising mechanism, socks would probably work well in Japan though. It seems to me that many people regard accepting free t-shirts as being an exception to all the usual conventions regarding advertising. Accepting gifts from companies that you do business with is generally regarded as a bad idea, except of course when t-shirts and other apparel are given out then it’s OK. Being paid to wear a placard advertising a product is regarded as degrading by many people, but accepting a free t-shirt (effectively being paid $7 for wearing advertising) is regarded as OK by almost everyone. I don’t mind being a walking advert for a company such as Google. I use many Google products a lot and I can be described as a satisfied customer. There are some companies that have given me shirts which I only wear in winter under a jumper. The Oracle Unbreakable Linux [2] shirt is one that I wear in winter. Now I would not consider accepting an offer to have advertising on my butt (although I’m pretty sure that it doesn’t get enough attention that anyone would make such an offer). I would however be happy to talk with someone who wants to pay me to wear a t-shirt with advertising when giving a lecture at a conference. I am not aware of any conference which has any real dress requirement for speakers (apart from the basic idea of not offending the audience). The standard practice is that if your employer pays you to give a lecture as part of their marketing operation then they give you a shirt to wear (polo more often than t-shirt). I am currently working on some things which could end up as papers for presentation at Linux conferences. If someone wanted to sponsor my work on one of those free software related projects and then get the recognition of having me wear their shirt while giving a lecture and have me listed as being sponsored by that company in the conference proceedings then that seems like a reasonable deal for everyone. One thing that you need to keep in mind when accepting or soliciting for advertising is the effect it has on your reputation. Being known as someone who wants advertising on their butt probably wouldn’t be fun for very long. On the Internet advertising seems to be almost everywhere. It seems that more than half the content on the net (by the number of pages or by the number of hits) either has an option to donate (as Wikipedia does and some blogs are starting to do), has Google advertising (or a similar type of adverts from another company), is a sales site (IE you can buy online), or is a marketing site (IE provides background information and PR to make you want to buy at some other time). Note that my definition of advertising is quite broad, for example the NSA web site [3] has a lot of content that I regard as advertising/marketing - with the apparent aim of encouraging skilled people to apply for jobs. Not that I’m complaining, I’ve visited the National Cryptologic Museum [4] several times and learned many interesting things! I think that Internet advertising that doesn’t intrude on the content (IE no pop-ups, page diversions, or overly large adverts) is fine. If the advertising money either entirely pays people to produce useful content or simply encourages them to do so (as in the case of all the blogs which earn $10 a month) then I’m happy with that. I have previously written about some of my experience advertising on my blog [5] and how I encourage others to do the same. I don’t think that space on a t-shirt is any more or less appropriate for advertising than space on a web site hosting someone’s blog. Finally there is one thing I disagree with in Colin’s post, that is the use of the word “whore“. It’s not uncommon to hear the term “whoring” used as a slang term for doing unreasonable or unworthy things to make money (where “unreasonable” and “unworthy” often merely means doing something that the speaker wouldn’t be prepared to do). But using the term when talking about a woman is quite likely to cause offense and is quite unlikely to do any good. The Wikipedia page about prostitution [6] has some interesting background information.

• [1] http://www.bytebot.net/blog/archives/2008/06/13/on-whoring-oneself


• [2] http://etbe.coker.com.au/2008/02/18/oracle-unbreakable-linux/


• [3] http://www.nsa.gov/


• [4] http://www.nsa.gov/museum/index.cfm


• [5] http://etbe.coker.com.au/2008/04/14/friends-and-adverts/


• [6] http://en.wikipedia.org/wiki/Prostitution

Share This

I have been tracking the Debian release goals progress for a while now, through bug-squashing parties and using various iterations of wiki pages. It's time to write down some thoughts.

First of all, I think the concept of release goals is a fantastic idea. It may be one of the most important conceptual moves in Debian of late. It allows developers to implement distribution-wide changes without having to negotiate with every package maintainer individually and without having to go through the vicious no-change-without-policy/no-policy-without-established-practice maze. It moves the focus back on operating system development instead of package hoarding.

In the future, I think we should come up with some "sexier" release goal descriptions. If you look over the current list, many of these tasks are implementation-level, without any obvious benefit to users: double-build support? no unmet recommends? Or they have incomprehensible descriptions. The pretty much only release goal that has a real user benefit, namely the piuparts cleanness, ensuring clean distribution upgrades, is both the most cryptic and actually the most neglected one in terms of development attention. Don't get me wrong, all these goals are very valid. For better or worse, the press has started to pick up these release goals to put in their various Debian release candidate timeline pre-announcement articles, and next time around we should have something cool in there. If armel support or KDE 4 integration had been release goals, the marketing effect would be better.

The next thing is tracking bugs. If you have some time on your hands, how do you find a release goal bug that you can help with? I had worked up this wiki page to help myself, but it is still too weird. The fact that bts.turmzimmer.net exists shows that the BTS is lacking features or is too hard to use or both. I'm thinking, we should regularly copy the whole BTS into an SQL database and make queries from there. Think aggregation, grouping, full-text search! I'm not familiar with the BTS internals or the LDAP gateway, but I know my SQL. So if anyone knows the other end of this deal, I'd love to chat about setting something like this up.

So then, what happens to the release goals after the release? Has this been thought through?

Obviously, some release goals become technically self-enforcing. Python 2.5 and GCC 4.3 are now the default, and anything that doesn't work with these will have permanent trouble in the future.

Release goals that implement a distribution-wide feature, such as the LSB-based init system enhancements, will really need to become part of policy after the release. Otherwise they would revert to being normal bugs without any NMU powers and the other special attention given to release goals. So over time, the support for the former release goals across packages would again decline. To keep it up, you'd either have to repeat the release goal indefinitely, which would be silly, especially when there are no problems right now, or you'd have to make it a part of policy to enforce it in the future. Essentially, you'd have to consider most release goals trial policy changes, which become permanent if successful.

Near-examples of the above are the eternal release goals of IPv6 support and LFS support. We really have these under control quite well now -- although I still wouldn't dare to call Debian or any general-purpose Linux distribution fully IPv6-ready -- but nowadays these problems are almost all upstream code problems that Debian can't really fix very well on the packaging side. So under the current approach these release goals will exist forever, or worse, will be dropped from release goal status.

And then of course, some release goals fail to be reached. These should reapply for the next release, if interest persists. The dash goal comes to mind.

By the way, we had 13 release goals. 3 are completely done, 5 are pretty much done (less than 10 bugs), 1 more looks like it could get pretty much done before the release, and 4 will most likely fail to be accomplished.

Pity poor Dr. Ari Jaaksi from Nokia. He gave a talk at the Handsets World conference in Berlin on Tuesday, where according to ZDnet, he lectured Open Source Developers that they needed to learn why DRM and other closed technologies were necessary, because of business issues such as subsidized (device) business models. I suspect he wasn’t prepared for the reaction, which took the form of a major fuss on Slashdot, as well as some declarations from a few people on the maemo-users mailing lists that they would never buy another Nokia device. Dr Jaaski then posted today on his blog an entry entitled, “Some learning to do?”, where he stated that while Nokia needs to learn how the open source world works (not just licenses and legal issues, but also the spirit), that the open source world also needed to learn as well — about WHY things are the way they are. He ended the post with “I’m not a teacher, I’m a learner” — which is fair enough. I actually think that Dr. Jaaksi is right; we all should be interested in learning, and having talked with a number of Nokia employees, I can testify that their hearts are very much in the right place, and there are some strong reasons regarding the demands of carriers, the subsidized business model of handsets in the US, the attitudes of suppliers for components in mobile devices, that very much tie their hands — as well the hands of every other handset company out there. Progress in this space will come slowly, and but I believe there is hope that in the long run things will get better. However, at the same time, I would suggest to Dr. Jaaksi that it’s also important how to communicate with the open source community — or with any community that holds views sometimes with great passion. Many people do not react well to being told that they need to learn something — even if it is true that they do. Admitting that they need to learn is for some people tantamount to admitting ignorance, or even worse admitting stupidity, and so they are loath to do this. Telling them this just makes them defensive and angry. My suggestion for how get people to learn without making them defensive is to use a modified Socratic method. It works especially well with the Open Source crowd, because many of them are engineers, and engineers are by nature problem solvers. So if you give them a problem to solve, they will go to work trying to find potential solutions, and thus start thinking and learning about the problem from a different point of view. Hence, a better approach is to give them a series of business constraints (time to market, the demands of carriers, the fact that most end users prefer to buy subsidized phones with 1-2 year lock-in contracts, the intransigence of suppliers, why a single company can’t afford to create their own chipsets from scratch and have to rely on companies like Broadcom, etc.) and then tell them — we’d love to delight our customers in any way we can, and clearly you are very passionate about DRM, open device drivers, and other OSS issues. We however have to create successful products and sell them at a price such that we don’t go out of business — can you help us work through all of these constraints? Let’s work together so we can make an open source mobile platform that meets the spirit as well as the legal requirements of the open source world! Dr. Jaaksi and his team are struggling with a hard problem, and I am very sympathetic to their issues. And, they have to be commended for investing as much time and effort on the Maemo platform, as well as other Linux and Open Source platforms as they have to date. The strength of the Open Source community is that we work together to solve our mutual problems, while celebrating and giving permission to people to “scratch their own itches”. In order to do that, we need to learn how to better communicate with each other, and that is something that goes in both directions. When I was at MIT as an undergraduate, far too many years ago than I care to admit, I took economics classes and Sloan School of Management classes (even though they were not necessary for a Computer Science degree) for that very reason. I encourage open source advocates to learn how to walk a mile or two in business people’s shoes, since if you want to influence them them to change in the ways that you desire, you need to know where they are coming from — just shouting and throwing rotten tomatoes generally won’t get you very far. On the flip side, people who are on the corporate side of the divide also need to understand that certain ways of engaging the community can be more productive than others. At the end of the day, hopefully we can all agree with Dr. Jaaksi that everyone will benefit if we all commit to learning from each other. Update: I’ve fixed Dr. Jaaksi’s name so it is spelled correctly; my apologies for the error.

There has been a lot of fear-mongering about nanotech. The idea is that little robots will eat people (or maybe eat things that we depend on such as essential food crops). It’s unfortunate that fear-mongering has replaced thought and there seems to have been little serious discussion about the issues. If (as some people believe) nanotech has the potential to be more destructive than nuclear weapons then it’s an issue that needs to be discussed in debates before elections and government actions to alleviate the threat need to be reported on the news - as suggested in the Accelerating Future blog [0]. I predict that there will be three things which could be called nanotech in the future:

1. Artifical life forms as described by Craig Venter in his talk for ted.com [1]. I believe that these should be considered along with nanotech because the boundary between creatures and machines can get fuzzy when you talk about self-replicating things devised by humans which are based on biological processes.
   I believe that artificial life forms and tweaked versions of current life forms have significant potential for harm. The BBC has an interesting article on health risks of GM food which suggests that such foods should be given the same level of testing as pharmaceuticals [2]. But that’s only the tip of the iceberg, the potential use of Terminator Gene technology [3] in biological warfare seems obvious.

But generally this form of nanotech has the same potential as bio-warfare (which currently has significantly under-performed when compared to other WMDs) and needs to be handled in the same way.

The more commonly discussed robotic nanotech, self-replicating and which can run around to do things (EG work inside a human body). I doubt that tiny robots can ever be as effective at adapting to their environment as animals, I also doubt that they can self-replicate in the wild. Currently we create CPU cores (the most intricate devices created by humans) from very pure materials in “clean rooms”. Making tiny machines in clean-rooms is not easy, making them in dirty environments is going to be almost impossible. Robots as we know them are based around environments that are artificially clean not natural environments. Robots that can self-replicate in a clean-room when provided with pure supplies of the necessary raw materials is a solvable problem. I predict that this will remain in science-fiction.

Tiny robots manufactured in factories to work as parts of larger machines. This is something that we are getting to today. It’s not going to cause any harm as long as the nano-bots can’t be manufactured on their own and can’t survive in the wild.

In summary, I think that the main area that we should be concerned about in regard to nano-bot technology is as a new development on the biological warfare theme. This seems to be a serious threat which deserves the attention of major governments.

• [0] http://www.acceleratingfuture.com/michael/blog/?p=830


• [1] http://www.ted.com/talks/view/id/227


• [2] http://news.bbc.co.uk/1/hi/health/background_briefings/food_safety/278240.stm


• [3] http://en.wikipedia.org/wiki/Terminator_gene

Share This

In a perfect world, when we design some aspect of our infrastructure, we'd get it right the first time, and would never have to change our design once it had been deployed. Back in this world, however, we regularly have to change things as we expand our infrastructure. Sometimes we made a mistake originally, other times new requirements pop up. In either case we will end up doing things differently in the future than how it was done in the past. The danger here is that if you only create your new machines or services differently, you've got effectively two separate and different systems to maintain. This may not sound like much, until it happens again... and again... and again... and then suddenly you've got 18 different ways of setting up backups, depending on how long ago you set it up and (if you're really unlucky) who did it and what the phase of the moon was at the time. If you've been here before, you probably remember the pain of having some disaster happen and needing to fix it NOW NOW NOW, only to wade into the setup and realise that it's been done in a way that (you thought) died out six months ago -- and so you don't really remember how it works, and yeah, the documentation's been updated so many times since then that it doesn't mention the old way of doing things, and here comes your boss to ask how long it's going to be until it's all working again... If you haven't had this experience (yet?), you'll have to trust me when I say that it's no fun to hit an older, out-of-date system. It's especially painful because the most common reason you change things is because the old way sucked. So you turn up at a critical moment to fix something in a poorly documented and sub-optimally configured system. Fun, fun, funnity fun. There are only two ways to avoid this problem: never change the way you do anything, or else plan your changes so that retrofitting is part of the plan every time you're modifying an existing service. Assuming that never changing anything isn't an option, you need to plan to retrofit. When you're putting together the design for your system change (you do design your changes, right?) you need to audit what's out there, determine what's needed to bring it all up to date, and put together a plan to make that happen. It's like documentation and testing -- if you don't make an explicit effort to make it happen, it'll never get done. That sounds like a lot of work, and it is. Luckily, if you're already working in a well-maintained infrastructure you should only really have one existing config to upgrade, which shouldn't be too painful. However, if you've got a lot of machines to change there's no way around the fact that they've all got to be changed, and that can take time. If you accept that you don't want configuration skew, you can start to think about how to engineer your systems and processes to make it less of a hassle to keep everything up to date. Putting some extra thought into how you set things up in the first place so that it's easier to update is at the heart of this. Automating deployment is an easy way to take some of the pain away from rolling out new configs, but there's also plenty of smaller decisions that help with updating configs: putting everything in revision control, making extensive use of templating and pulling out common config fragments, and making sure that you've got good notes and records of what has been done where. The more you automate your deployment and maintenance, the less pain you feel when you need to retrofit. With something like Puppet, you can eliminate a lot of the pain because you're describing what you want done for all the relevant machines, and then the machines themselves work out what needs to be done to bring themselves up to date. With an advanced automation system in place, you can say "I want all my mail servers to have setting X turned on" and off they go and set the option. Or you might want to have some of your mail servers to have a different config setting, triggered based on a config setting or class of machine. Of course, you might not already have that config setting or class defined, so you'll need to add it. Don't forget to retrofit that new config setting or class to your existing machines. That might involve finding the config for all the existing machines and modifying them. Yep, no matter how much automation you've got you can't get away from the need to retrofit, but given the choice between editing a bunch of config files on my local machine (with all the power of my text editor to back me up) or logging into dozens of machines all over the globe to edit their configs and restart services (with all the risks of making a mistake) I know which one I'd choose...

Yes, Manu Chao was really home yesterday in Paris Bercy. And he was obviously feeling like being home. The concert began at 21:30 and ended up at 00:30. Yes...that makes 3 full hours concert. All that for 25 euros. The crowd support, dance, participation was *amazing*. I often go to concerts in the Bercy arena and never saw the crowd in the upper rows being stood up all concert long. Radio Bemba played their usual tracks first. As usual, the very Manu way to mix songs, themes from all his carreer and mix them up, distord them and make new songs from old ones. This is what always made Mano Negra, then Manu Chao oncerts unique experiences. Just imagine 20,000 people singing Clandestino a capella, most of us not knowing any damn word of Spanish... After about 2 hours of magic without any stop, the very special moments came in. Manu, Gambit, Majiv (what a guitarist when playing rumbas de Barcelona!), Julio (de Barcelona!)...and all others were just feeling like being home with a few friends. So the encores lasted...and lasted...and lasted. They just didn't want to stop (Manu probably said about 3 or 4 times "c'est fini"). Then, they did something I never ever saw in a big 20,000 people concert since my first one back in....long time ago. Manu asked "do you have 5 more minutes?". So, they stopped for 5 minutes to get the guitars re-accorded(sp?)....the crowd just...waited...and the gig re-started again with songs from "Si Sib rie m' tait cont e", the special book-album he made with friends, inspired by the sad winters he spent in Paris in his early years. Soooooo special and, believe me, that *was* special. Seeing all roadies, crew, friends just dancing backstage and partying clearly showed this was not to always happen (this will probably happen again on today's concert). He was just feeling like home. And so were we. Not wanting to leave and shut down the light. Finally, at 00:30, all of us were dead tired and we just hugged around with Manu and left. Just like old friends. I don't know if that will happen everywhere in the upcoming tour (some dates in France, some in Europe, a few in USA....nothing unfortunately in South America as of now)....but if you have the opportunity to go to Radio Bemba concerts, just do it. And (IRC reference), really, Erinn, this was not about "ethnic music for white people" (really wondering where you got this idea...or if you were just kidding me). I dunno what Manu ever did to kill your ears, but just try again and, even more important, go to the gig in your fair city on August 22nd. Radio Bemba concerts will be cheap, about half the price of regular bug size concerts....just use that opportunity.

Jeff Bailey asked:

"Heya Brits! Any of you still driving cars at ~ 1.15 according to the Daily Mail"

Yes, I am. I drove on Monday (at 1.18/l) because it was the least bad option for the journey. I try to avoid it and I felt bad afterwards (literally - it was too damn hot and each part of the journey was too short for the cab to cool down), but the car was available and the other choices involved not attending some events. On Sunday, I used my bike instead, but I was I wondering if the world is full of Sunday drivers today or whether I was really riding that badly. You name a junction on my route and I seemed to get into a conflict with a car at it. Today's bike trip went much better, even getting thanks from a coach for pulling aside halfway up an incline, but I had to take avoiding action as I re-entered the village because of a police car. I'm pretty damn sure that wasn't my fault, but I do wonder when it's the police. I'm still riding without a helmet, without ill effects. Gunnar Wolf was getting a breeze through a different kind of helmet but I think it's telling that cyclists "feel naked" rather than actually being naked (usually, at least). Have we got too used to being cocooned in metal boxes while out on the roads? I've always ridden and walked a lot - is this why I don't miss the hat much? I must remember to drink more in summer without it, though. I share Criag Sanders's scepticism about the protests and Chrisitan Perrier's enthusiasm for bike-pools. I don't agree with many of Russel Coker's views on oil prices but they are interesting reading, even so. I'm taking part in JamBustingJune for the West of England region and BikeWeek 14-21 June 2008

I was like “wtf?!”. Debian.org replaced by a link farm ? debian.org being actually domainsquatted is highly unlikely, so I started searching how this could happen. First, a little explanation of the setup :

• I’m on a bullet train on my laptop, up-to-date Debian Lenny, 2.6.24
• My laptop is connected to a 3G+ (HSDPA in the US) PDA phone with a USB cable
• The phone appears to the computer as a network interface thanks to the usb-rndis-lite kernel module
• The phone does NAT routing between my computer (192.168.0.100), himself (192.168.0.1) and a restricted network from my phone operator
• The only host visible on this restricted network is a HTTP only proxy server that checks the user agent (Nokia is OK, Firefox means GO AWAY)
• I connect through this proxy to a dedicated host through a OpenVPN tunnel masquerading as HTTP with a Nokia user agent
• The dedicated host at the other side has special iptables rules to redirect requests coming from my mobile phone operator netblock on port 80 to the regular OpenVPN port.
• The dedicated host does NAT for my computer to the real Internet.
• The total latency varies from 120ms to 30 secs and the bandwidth from 2mbps to 8kbps depending on the coverage
• The only DNS server on my laptop is set in resolv.conf to 192.168.5.1, the remote OpenVPN endpoint, the dedicated server
• The dedicated server runs Bind and provides recursive resolution

Now there are some peculiarities to the situation :

• It happens exactely once a week on the train from my home in Paris to the campus in Grenoble
• It doesn’t happen the other way around or at any other time for that matter

Some hints :

• My hostname at home is aeris.home.eu
• My hostname on campus is aeris.liuo.res.rhb
• I almost never shut down my computer, only hibernate
• Jonathan Roes

Now for the challenge : how could this happen ?

Yesterday I was in need to requeue several fax messages on a HylaFAX server after it had moved them out of the queue with the note Kill time expired due to a mysterious modem problem. According to the documentation simple faxalter calls should be sufficient. This turned out to become more fun than originally thought, however. The time to send the fax need to be altered as well as the kill time. Also, it may be required to use the ADMIN user of HylaFAX instead of the regular root user. faxalter will ask for the password which needs to be configured in the global configuration file. To make life more interesting the order of command line arguments is significant for the faxalter program. The ordering below did work:

   faxalter -A -a now -k now+5hours -v -r ID

more

It seems that many blog posts related to fuel use (such as my post from yesterday about record oil prices [1]) are getting adverts about perpetual motion [2]. Note that the common usage of the term “Perpetual Motion” does not actually require something to move. A battery that gives out electricity forever would be regarded as fitting the description, as does any power source which doesn’t have an external source of energy. The most common examples of this are claims about Oxyhydrogen [3], this is a mixture of hydrogen and oxygen in a 2:1 ratio. The wikipedia page is interesting, apparently oxyhydrogen is used for welding metals, glass, and plastics, and it was also used to heat lime to provide theatrical lighting (”lime light”). So a mixture of hydrogen and oxygen does have real-world uses. The fraud comes in the issue of the claims about magnecules [4]. Magnecules are supposedly the reason for the “atomic” power of HHO gas (AKA Oxyhydrogen) which are repeated on many web sites. In brief, one mad so-called scientist (of course if he was a real scientist he would have experimental evidence to support his claims and such experiments would be repeatable) has invented entirely new areas of science, one of which involves magnetic bonds between atoms. He claims that such chemical species can be used to obtain free energy. The idea is that you start with water, end with water plus energy - then reuse the water in a closed system. Strangely the web sites promoting water fueled cars don’t seem to mention magnecules and just leave the “atomic energy” claim with no support - maybe magnecules are simply too crazy for them. The water fuelled car wikpedia page is interesting - it lists five different ways that water can actually be used in a car engine (which are based on sound scientific principles and which have been tested) and compares them to the various water fueled car frauds [5]. I’m not accepting any more comments on my blog about perpetual motion solutions to the petrol crisis (they just take up valuable space and distract people who want to discuss science). I’ll allow some comments about such things on this post though.

• [1] http://etbe.coker.com.au/2008/06/09/record-oil-prices/


• [2] http://en.wikipedia.org/wiki/Perpetual_motion


• [3] http://en.wikipedia.org/wiki/Oxyhydrogen


• [4] http://en.wikipedia.org/wiki/Magnecule


• [5] http://en.wikipedia.org/wiki/Water-fuelled_car

Share This

I maintain a number of web shops for our webmaster cooperative and one of our main challenges is to encourage people who put things into their basket/cart to actually buy them. How to avoid shopping cart abandonment by Graham Jones makes some points that I've identified as possible reasons for people not buying in the past: comparisons, robots and not trusting the site enough to give payment details. There's not much we can do about robots or people comparison-shopping at a technical level, but we try to build some trust by publishing the shop owner's geographic address and telephone number (which I think is required by law in England for most web shops now), making sure the SSL certificate and domain registration details are correct, using reputable payment providers and being clear about delivery charges and terms. The point about the slickness of the checkout process is a good one and one that we've only recently started to work on. We've had pretty good results from making the checkout slicker on one site. It looks like two-thirds of people who click the checkout button now continue to buy, putting it comfortably ahead of current UK averages but I need to tweak our stats calculator to make the report directly comparable. Nevertheless, I think those improvements will be added to our other shops as soon as possible. I share Graham's low opinion of the oft-quoted Amazon. We've also been looking at other web shop software besides OSCommerce for a new project, so now would be a good time to change to something new if it improves the checkout a lot. We've made OSCommerce's checkout a lot smoother, but it's still essentially OSC. Is there a good checkout which you'd want to use as an example? The other challenge is getting visitors onto the site in the first place. How To Build Links By Patrick Altoft explains the basics as well as I've seen recently.

Recently I read about a cool bugfix for gdb in the Novell bugtracker on planet.gnome.org. I ported the fix to the ubuntu gdb package, and Martin Pitt promptly extended it to have an amd64 fix as well. I thought I would provide the enhanced patch back to the Novell bugtracker. This required creating new Novell login as my old CNE details are so far back I can't remember them at all. However, hard-stop when I saw this at the bottom of the form: "By completing this form, I am giving Novell and/or Novell's partners permission to contact me regarding Novell products and services." No thank you, I don't want to be contacted. WTF.

I'm a bit slow on the journal-writing front, but belatedly I should mention here that I did upload the new Policy (3.8.0.0) and then a quick bug-fix. If you have a trigger-enabled dpkg, you don't see that you're registering non-existent doc-base files. Lintian should catch that, but doesn't. Yet. I've been trying to find time to kick off the next round of Policy updates. There are several proposals that just need text written for them: #143941 and #241333 are obvious at this point, #416450 should be fairly simple, and I think the due diligence has been done on #470994. There are other fairly straightforward ones too. If anyone beats me to it for any of those (or even better, other bugs), I won't complain. I also closed all of the old Policy bugs that we'd tagged wontfix except for one that I forwarded to the tech-ctte. I also uploaded Lintian 1.24.0 on Saturday. This contains the first steps in a better documented and hopefully faster and cleaner way of extracting information from the lab. Rather than opening and reading files in every check that need data, there are now Lintian::Collect modules that can hold common code and cache the results across multiple different check scripts. Currently, they're not widely used, but they're now available to all checks (hence the version bump). This release also contains some truly fantastic work by Jord Polo Bard s on the web site. Look at lintian.debian.org now. It looks like a real web site! Jord is working on a Google Summer of Code project for Lintian, but it wasn't even the web pages; that was just a bonus. I'm very happy with the outcome. Lintian is still above 100 bugs and I want to start expanding the new infrastructure, so there's more work to be done, but I'm very happy with the progress lately. And it was fun to spend two days solid working on Policy and Lintian for a change, instead of lots of different work projects (or feeling low on energy and barely managing to get a handful of things done).

I just got back from Indiana Jones and the Kingdom of the Crystal Skull, a movie in which Steven Spielberg takes his obsession with the extra terrestrial to the next level.

Please watch it if you feel that times goes too fast, the two hours that you are watching the movie will seem like two years.

And it had to be a UFO in the end :)

Update: I should have kept the spoiler under lj-cut, but now that I think of it, that wouldn't help people aggregating the posts. I suppose I will avoid posting spoilers.

We actually got home last Sunday, but I've been too jetlagged, and then too busy with work to really have a chance to write anything... We got back in at around 2pm, but our bags fell victim to Heathrow's Terminal 5, and didn't arrive until around 11pm. I think I lasted until about 8pm, when I had to crash. Poor Sarah had to stay up until 11pm when they finally got around to delivering our suitcases. The week in Zurich was wonderful. The Zurich office is everything the photos make it out to be, and then some. One weird thing: apparently there's some Swiss regulation about how much you're allowed to cool a building in relation to the outdoor ambient temperature, so there's no air conditioning in the office. Instead, you can open the windows. Unfortunately, there's a fairly busy set of train tracks right beside the office, so it can get a tad noisy... It was a pretty warm week, I think around 30°C. I certainly prefer being colder rather than hotter when indoors, so found the lack of decent cooling to be a shame, given the rest of the building's features. That said, it was pretty amazing sweltering away in Zurich, and then looking up and seeing snow on the mountains. It was surprising how much of a temperature difference there was. On the last evening in Zurich, we went on a reconnoitre to try and find the river that we could hear (and see on the map) behind our apartment. We eventually found some street access to it, and it was another world back there. It was fairly thickly wooded, and the sunlight was heavily filtered through the trees, so it was cool and shady. The river was fairly fast flowing over some rocks in parts, which is what was making it so audible. The whole setting was absolutely beautiful. There were a couple of paths, and the whole thing felt like something out of a fairy tail. We think we stumbled onto a fox, but we're not sure. It's a shame we only discovered the place on the last day, as I'd have liked to have explored it further. I really liked Zurich. It was nice and flat, and had an excellent tram service. Monday Night Skate made me wish I'd packed my roller blades. We were wandering around on Monday night, and it felt like the entire city had donned skates and were going out. Apparently the authorities really get behind it, and close roads, and the police go along behind the pack and reopen the roads behind them and pick up the stragglers. Great way to encourage an active lifestyle. We've got all of our photos up now from the Europe trip, and they're here. This brings the total countries I've visited up to 10, excluding Hong Kong and Macau. I don't like selecting China when I generate this map, as I've never been to mainland China.