Part of the problem that we face in providing Linux hardware support is that we're lucky if there's a spec, and even if there's a spec there almost certainly isn't a test suite. Linux still isn't high on the list of things that vendors test with, so as a result hardware and firmware tend to be written to work with Windows rather than some more ideal notion of what a spec actually says.

This can lead to all kinds of problems. If the spec says we can do something and Windows never does that, we'll often find that there's at least one piece of hardware out there somewhere that breaks in response. But sometimes there'll be several different workarounds, and picking the wrong one may break some other system instead. It's helpful to know what Windows is actually doing, if only because that's the only configuration most systems have been tested with.

The problem is that doing this at the hardware level is next to impossible. I'm sure there are people out there who salivate at the possibility of working out i8042 initialisation sequences by hooking up oscilloscopes to their mouse, but I'm not one of them. There's a much easier alternative involving qemu.

The qemu source base is reasonably large and complex, but that's ok - we don't need to care about most of it. For our purposes we really only want to trace accesses to given bits of hardware. There's three main types of hardware access we're likely to care about: io ports, memory mapped io and pci configuration space. io ports are easy. Each piece of qemu that performs hardware emulation will call register_ioport_read or register_ioport_write. Just grep for those, find the ports that correspond to your hardware and then edit the functions to dump the information you want. PCI configuration space will be handled via pci_default_read_config or pci_default_write_config unless the driver overrides them. Finally, for PCI devices mmio will be handled via pci_register_bar - the last argument is the function called when the memory region is accessed.

All of which makes it pretty easy to watch register reads and writes performed by Windows when it's starting up. Suspend/resume is also an interesting problem, but sadly one that's harder to work with. First of all, you need at least version 0.6c of vgabios in order to indicate to Windows that it's possible to suspend at all. Secondly, for Vista or later you'll also need a WDDM driver for the graphics. Sadly there isn't one for the Cirrus that qemu emulates, which is unsurprising given how ancient it is. So I've had to perform my testing under XP, which was enough to give me an indication as to what Windows does with the SCI_EN bit on resume (answer: Ignores both the bit of the spec that says it should already be enabled and the bit of the spec that says it should never be written by hand). Nice, but if someone would like to write a WDDM driver for Cirrus it'd make my life easier.

The other thing I've been testing is keyboard controller probing. Some Macs deal badly with you banging on the keyboard controller, which is dumb but on the other hand they don't claim to have one. Linux will look at your ACPI tables and use any keyboard controllers it finds there, but if there isn't one it'll go on to try probing the legacy locations. Adding some debug code to the read and write functions of the keyboard driver in qemu and then editing the DSDT in Seabios to remove the declarations for the keyboard showed that Windows will only probe if there's a device with a recognised keyboard PNP ID. No keyboard for you if that's missing. So we probably need to emulate that behaviour as well.

The main reason to do this work is to try to reduce the number of DMI tables in the kernel. These are cases where we alter the kernel's behaviour based on the identity string of the hardware, allowing us to work around functional differences. The problem with this approach is that Windows generally works fine on these machines without knowing anything special about them, and chances are that the tables aren't exhaustive - there may well be some other piece of hardware that's also broken, but the user just gave up and went back to Windows instead of ever letting us know. Using qemu to work out how Windows actually behaves gives us the opportunity to fix things up in a way that will (with luck) work on all machines.

A scandal has been brewing in Latvia over the last half year and yesterday the activity spiked shocking the media and some IT people in the country. I ll go back and explain what happened first, what is happening now and why this could have a heavy impact on IT and journalists in Latvia. At the end of last year, there were rumours that the IT system of Latvia s Internal Revenue System was hacked and millions of documents had been downloaded by multiple organizations. Shortly thereafter more details on the glaring security hole became public (after it was closed). There is a full electronic interface to give all reports to the IRS electronically (at http://eds.vid.gov.lv) and as part of that system you could also view and export monthly report summaries about your organization into XML and PDF files. After the system checked that you are authorized to access the report, you were redirected to the URL to actually download the report by report ID (as a single param in a GET request). Unfortunately, report IDs were predicable and the script that gave the reports for download did not check if you were authorized to get that report. It did not even check if were logged into the system. There were suspicions that the authorization was disabled on purpose to allow to leak data on purpose, but apparently it was an error of forgetting to disable debug code in production environment. The error was discovered only because the firewall administrator noticed an unexplained stable increase of traffic, especially during night hours when typically the traffic fully stopped. Apparently a single hacker (who later identified himself as Neo to the press) discovered the flaw and wrote a script to just try all possible report ids and get as much data out as possible. This had been going on for months, before someone noticed. After the flaw was discovered and a bit of time passed, Neo made his first move he published the list of top salaries in a governmental company, that clearly showed that the top leadership of this company failed to cut their salary by 40%, like everyone elses during harsh budget cuts of 2009. He stripped the names and ids of the specific employees, but named the company which made it pretty easy to figure out who was who. The society was outraged that the top managers in a government owned company failed to comply with the strict pay cut that everyone else in government had to endure. But after a few weeks the outrage subsided and no action followed from the government or law enforcement. Neo continued to release documents detailing salaries of top managers in different Latvian government companies. And each time after short outrage, nothing happened. Neo gave an interview where he said that he was disappointed in the passivity of the Latvian people in face of such blatant injustices. After a few month Neo went silent, promising to return before parliamentary elections this fall. However, this week a new development shocked everyone in the middle of the night two police SWAT teams went into action: one detained Ilm rs Poik ns, a researcher in artificial intelligence at the University of Latvia s Computer Science department and another raided the home of a Latvian TV journalist Ilze Nagle who interviewed Neo. Poik ns confessed of being Neo the next day and was released (with travel restrictions, pending trial) today. Politicians reacted immediately opposition demanded the resignation of the Interior Minister over such blatant disregard of freedom of press and another politician (who is also a famous lawyer) Aleksejs Loskutovs volunteered to defend Neo pro-bono (on Twitter, no less). Almost all Latvian online media have the arrest of Neo and the raid on the home of a journalist as main stories of the day. As a legal titbit, we also know that Neo is being charged with breaking statutes 145 and and 244p2 of the criminal law. Statute 145 is hard to find applicable in this situation as talks about actions done by people authorized (..) to access [private] information . Statute 244p2 will also be hard to pin down as it mentions influencing system resources of (an IT system) and if such action caused severe harm . It looks like the first part talks about at least a DoS attack (which did not happen in this case) and also there was no measurable harm from these leaks. Also Neo was careful to strip all personally identifying information (such as names, social security numbers and addresses of the employees in question), so it will be hard to pin him on that. Also no actual breaking or other modification of an IT system occurred. And no specialized software was used beyond a trivial script such as :

for i in range(0,7000000):
    wget('https://eds.vid.gov.lv/getRep.aspx?id='+str(i))

A lot of commentators on the Internet likened the situation to walking trough an unlocked door and stealing something. I think that analogy is very incorrect there was no door, and nothing went missing after the action. I came up with a different analogy there was this corridor with a lot of doors in IRS, locked steel doors. You were instructed to go to a room with a specified number and given a key to that room to unlock it and see your secret info. However, that corridor opened out to the street on one end, oh and also the walls of the rooms with all the secrets were transparent. So Neo walked into the corridor, looked at some of the secrets, wrote them down (to remember them better) and then went out and discussed the worst examples abuses of power he saw. In the end IRS had to learn their lesson if you have to put naked photos of yourself on the Internet (or something equally embarrassing), then make damn sure you password protect that, but if you don t then don t cry that someone hacked you and stole you pictures. What other people think:
http://freespeechlatvia.blogspot.com/2010/05/neo-released-under-restrictions.html We ll see how the story develops soon.

The last few days a new boot system called systemd has been introduced to the free software world. I have not yet had time to play around with it, but it seem to be a very interesting alternative to upstart, and might prove to be a good alternative for Debian when we are able to switch to an event based boot system. Tollef is in the process of getting systemd into Debian, and I look forward to seeing how well it work. I like the fact that systemd handles init.d scripts with dependency information natively, allowing them to run in parallel where upstart at the moment do not. Unfortunately do systemd have the same problem as upstart regarding platform support. It only work on recent Linux kernels, and also need some new kernel features enabled to function properly. This means kFreeBSD and Hurd ports of Debian will need a port or a different boot system. Not sure how that will be handled if systemd proves to be the way forward. In the mean time, based on the input on debian-devel@ regarding parallel booting in Debian, I have decided to enable full parallel booting as the default in Debian as soon as possible (probably this weekend or early next week), to see if there are any remaining serious bugs in the init.d dependencies. A new version of the sysvinit package implementing this change is already in experimental. If all go well, Squeeze will be released with parallel booting enabled by default.

Since October of 2002, HP has sponsored a "corporate subscription" to LWN on behalf of Debian, and I recently renewed the subscription through April of 2011. As of this moment, 571 Debian Developers and Debian Maintainers are enabled to take advantage of this subscription. From discussion on IRC this morning, I gather one recent change hasn't been adequately communicated: I now allow Debian Maintainers to access the subscription. The process remains unchanged: If you are a Debian Developer or Debian Maintainer and want full LWN access at HP's expense, just go to lwn.net and create an account for yourself (no money is required to create a user account). Then, send email to

lwn-subscription@debian.org

containing your LWN account name. Sign this email with your key on the appropriate Debian keyring. Then, exercise patience. Eventually, I will process your request, and add you to the "Debian Project group" and send you an email acknowledgement. Likewise, if you retire from being a DD or DM, please let me know at the same address so that I can take you off the Debian subscription. I believe I've caught up on all pending requests, but sometimes things get mis-filed, so if you're still waiting and I haven't replied, please re-send your request.

I ve been bicycling to work for awhile now. It s become pretty much routine for dry days. I can reliably ride the 10 miles to work in less than an hour, even with heavy winds. My biggest enemy right now: rain. My first few miles are on gravel and sand roads, which aren t maintained to the highest of standards. When it rains, there will be mud. I ride a Trek 7.3FX, which I bought with the idea of just riding when it s dry. It works quite well for that. But its slick tires don t work so well in mud. Plus our mud is sticky, and the tires pick it up, then deposit it on the frame that goes around the wheel and the brake arm. So I have a mess to clean up. My current dilemma is: do I buy a second bike for use when it s wet, or there s a chance of rain? My 7.3FX can t use tires that are very wide, and a different bike of course could. There would be expense involved, obviously, but I would really like to be able to ride more often than I can now. Winter was pretty damp here and I didn t get to ride much at all over the winter months. Or, would different tires plus fenders do the trick on the 7.3FX? (I m thinking not.) If I do get a different bike, then the question would be whether to sell the 7.3FX. I really like that bike, and imagine it would be faster on the 7 miles or so of paved roads that I ride every day. Perhaps I d keep it as a fair weather or long-distance bike. I plan to visit the bike shop tonight and see what my options are.

I've been a little quieter than usual recently, having spent more time outdoors putting cute people in front of the camera. However that said I've still been doing some things. Most interestingly I've given away my first ever project. The collection of small scripts known as xen-tools (which was initially a sleazy hack to go with a small introduction to Xen article) has now got new developers, and a new home where development is continuing. This isn't the first time I've stopped working on something, but it is the first time I've explicitly "given away" a project. (Mostly on the basis that if I didn't care nobody else did either, or people cared but were too busy/unable to actually do soemthing useful.) I'll be following new updates with interest, even though these days I'm 100% Xen-free. No need to go into huge details about why, but I'm enjoying KVM. Having said that I recently got into a huge mess with a combination of LVM, KVM, and ext3. I've written up the details on ServerFault in the optimistic hope that somebody will report having experienced a similar problem. If you have seen something similar I'd love to hear from you. Otherwise I'm genuinely at a loss to understand what went wrong, and why things failed. I could suspect hardware issues, but that feels like a cop-out, albeit one that has a potential solution (Mad Hatter: All Change!) rather than my current answer and explaination "It broke. I don't know why. It might happen again. It might not. Trust me?". ObFilm: Alice In Wonderland (1951 version.)

Contratulations to Bob Finch and Nathan Dalrymple for becoming our first customers to fly a rocket using a TeleMetrum board last Saturday! The rocket was Nathan's stretched BSD Horizon flying on a CTI Pro29 6xl 305-H226-14A Skidmark motor as shown in this launch photo. I met both Bob and Nathan at the monthly club launch run by the Albuquerque Rocket Society at the same site in Rio Rancho, NM, earlier this year. That was the day I flew my first test flight with on-board GPS. They got excited about what they saw, and became two of our first customers for production boards. Other than some rough edges with our first-generation ground software, Bob reported that everything worked as expected, and the GPS location received in the downlink got them to within about 10 feet of where the rocket set down. In communicating their success to us on IRC, Bob said "MANY thanks to you guys, not only for the product but also for all the help getting me to the point of being able to use it !!". The pleasure is all ours, though, because after Bob asked us lots of new-user questions online, he wrote up his notes for us as "documentation for mere mortals" that will soon form the basis of a "getting started" section in our user manual. Pretty cool stuff!

The jury has announced that Wikimania 2011, the 7th annual Wikimedia Conference will be held in Haifa, Israel. See you next year (:
Filed under: Wikipedia

If anybody is interested or can attend: Tomorrow and the day after (Wednesday and Thursday 12-13/05/2010) FLACSO (a leading Social Sciences faculty spanning countries all over Latin America) and IFE (Mexico's federal electoral authority) are presenting a seminar on e-voting: Electronic vote experiences in Mexico and around the world. Sadly, I will be unable to attend, as I will be on the road once again (Ecuador!). However, I hope those among my readers who are interested in the topic can attend or at least follow the audio and video transmission from the IFE website. The seminar will be held at IFE's auditorium Viaducto Tlalpan No. 100, Col. Arenal Tepepan, Delegaci n Tlalpan, c.p. 14610, M xico, DF. [update] If you don't know what is my stand on this topic (and can read Spanish), please read this short article. In short: I am against e-voting, and hope we are still in time (and can push at the right places) to avoid it becoming the rule in our country.

That is my second day at the UDS in Brussels. I have attended some sessions about server and cloud yesterday. The Ubuntu people are better organized than the Debian people during Debconf. They are collaboratively preparing the sessions in Launchpad and make notes with the Gobby editor during the session. Gobby allows group editing via Internet. The release date helps to keep the focus on stuff that can be done within the time frame and helps to concentrate on getting results.

It will probably be only 5 months for Maverick because Mark asked for a release on 10.10.10 which is an interesting date after removing the dots and converting it to a decimal number. I am comparing it to Debian: Toy Story 3 will be shown in cinemas soon and the Squeeze character is promoting the film. Will we use such a coincidence for more public attention and will will we release our Squeeze soon? I am afraid that it won't happen.

Our last session yesterday was about Tomcat packaging improvements. Debian and Ubuntu are using and maintaining identical packages of tomcat6 using the alioth infrastructure of the pkg-java team. Some of our plans are:

• fixing postinst to read configuration values from /etc/default/tomcat6
• add debconf support to allow modification of /etc/default/tomcat6
• document alternatives to authbind since authbind does not support IPv6
• document 2 ways to package web applications using tomcat6 or tomcat6-user
• transition from the old libservlet2.[34]-java packages to the current libservlet2.5-java package
  

Weeks ago, I was at a TravelWatchSW meeting in Taunton for Cooperatives-SW. The Chair s theme for the meeting was the journey was going so well, but is there trouble ahead? contemplating cuts due to the debt crisis and possible change of government. Lots of good people were present, so it was quite an illuminating meeting. First, we listened to Dr Gabriel Scally, Director of Public Health for the South West NHS, as he spoke on fat, exercise and transport He s a keen cyclist, so I asked for suggestions of how to get better parking at health centres and hospitals. Sadly, he had no easy answer. One tip: Primary Care Trusts control most money, so they re probably the best ones to persuade. The second keynote address was First Great Western s Projects and Planning Director Matthew Golton. He spoke about railway development, unsurprisingly. Rail is vital for co-operatives in the South West, as it s a greener way to travel the 250 miles length of this peninsula and avoids our poor congested roads. The big news is that class 150s (Sprinters) will replace the Exeter area 142s (Pacers/Railbuses) and there will be grade-seperated junctions at Reading by 2016 to reduce congestion between the various routes that converge at that point. There s bad news on the Intercity Express Programme and HST2 being delayed until after the election, but the original High Speed Train fleet should continue as reliably as ever. After lunch, there were a series of Just a minute comments on topics including: west country commuter services, staffing, Taunton nhs parking, axminster peak times, a TransWilts meeting. Finally before the reports and forum came the third and final keynote from Mike Lambden, head of corporate affairs for National Express Group. He outlined their plans including longer coaches, but asked for better coach stations. One strange mistake had been increasing the leg room on some coaches so much that they got complaints from shorter passengers that they couldn t reach the footrest any more oops! The forum was a scary discussion of rail under-provision planned in our region. Even if we only continue to grow rail use at the last decade s rates, overcrowding will increase. The forum ended with applause for all the hard work done by TWSW and the next meetings will be 9 October 2010 in Taunton, 5 March 2011, 1 October 2011.

I'd wager probably more people today don't know what the TERM environment variable really does than do (yes, everyone who used to use ed over a 300-baud acoustic-coupler is laughing, but times have changed!). I recently got hit by what was, at the time, a strange bug. Consider the following trivial Python program, that uses curses and issues the "hide the cursor" command. If you run it in your xterm or linux console, nothing should happen. But change your TERM to vt102 and it won't work: (obvious when you explicitly reset the term, no so obvious when some script is doing it behind your back -- as I can attest!) So, what really happened there? The curs_set man page gives us the first clue:

curs_set returns the previous cursor state, or ERR if the requested visibility is not supported.

How do we know if visibility is supported? For that, curses consults the terminfo library. This is a giant database streching back to the dawn of time that says what certain terminals are capable of, and how to get them to do whatever it is you want them to do. So the TERM environment variable tells the terminfo libraries what database to use (see /usr/share/terminfo/*) when reporting features to people who ask, such as ncurses. man 5 terminfo tells us:

cursor_invisible civis vi make cursor invisible

So curses is asking terminfo how to make the cursor invisible, and getting back "that's not possible here", and hence throwing our error. But how can a person tell if their current terminal supports this feature? infocmp will echo out the features and what escape codes are used to access them. So we can check easily: So we can now clearly see that the vt102 terminal doesn't support civis, and hence has no way to make the cursor invisible; hence the error code. If you're ever looking for a good read, check out the termcap/terminfo master file. The comments are like a little slice of the history of computing!

Eclectic paper on the Ultimate Debian Database A few months ago, I've co-authored with Lucas a paper on UDD, which has just been presented at this year IEEE's Mining Software Repository conference, continuing my recent tradition of eclectic papers. The paper is titled The Ultimate Debian Database: Consolidating Bazaar Metadata for Quality Assurance and Data Mining and is available for download from my publications page. For Debian people already familiar with UDD there is probably not much to learn from it, as the main target of the paper is the community of scientists doing data mining on software repositories. For them, UDD offers a valuable entry point to Debian "facts", as data sources reflected in the database are easily joinable together and to some extent already validated by other UDD users (e.g. QA people). Nevertheless the first two sections of the paper are probably of more broad interest. There we have given our point of view on the so called Debian Data Hell: why it exists, how it's related to the nature of Debian and similar distros, etc. I've already noted in the past how that is also related to the culture of freedom that in Debian we value not only in our software, but also in our infrastructure and procedures. We should just get rid of a bit of inertia, and total world domination will then be just around the corner I'm happy to conclude quoting the acknowledgments section of the paper: Acknowledgments

The authors would like to thank all UDD contributors, and in particular: Christian von Essen and Marc Brockschmidt (student and co-mentor in the Google Summer of Code which witnessed the first UDD implementation), Olivier Berger for his support and FLOSSmole contacts, Andreas Tille who contributed several gatherers, the Debian community at large, the "German cabal" and Debian System Administrators for their UDD hosting and support.

Mark Shuttleworth just kicked off UDS with the Keynote. I have the Audio at just under one hour in length for everyone to enjoy that could not attend the Ubuntu Developer Summit this time.

Keynote in MP3 ( Direct )
Keynote in MP3 ( Torrent )
Keynote in OGG ( Torrent )

He Announces the next radical changes to the desktop, along with Ubuntu "Lite" being installed Dual Boot by Dell and other OEM's that boots to a USABLE web browser in 7 seconds, and we're not talking about something coming up, this is available today by installing "Unity".

The Perfect 10 as Mark describes it will consist of UNE, new icons, the new font and many other desktop improvements. And he is pushing for a Release on 10 Oct, 2010 or 10.10.10 , for those of you geeks out there ( like me ) binary 101010 = decimal 42 ( HHGTTG Reference )

This is going to be an exciting cycle !

UPDATE: Looks Like Dropbox did not like the excessive traffic, I have updated the links.

I woke up with a loud BZZZT It happens every couple of years. The electric transformer for the circuit where my house is located, at the Northern edge of Ciudad Universitaria, decided to die (or at least, to take a break literally). About one hour later, I decided it was time to wake up and start being a useful person. I gave breakfast to my cats and had breakfast myself, and called the electrical company to report this mishap. They told me the report was registered, and I hope to have electricity soon (meanwhile, I'm sitting at a nearby restaurant, as there is some job to do Yes, besides writing blog entries). And they told me, as is often told in Mexico, a general anticorruption phrase With a twist: recuerde que usted no debe renumerar ning n servicio que realice el personal de la Comisi n Federal de Electricidad . (You should not renumber any service done by the Federal Electrical Commission personnel). Yes, remunerado (not renumerado). This shows once again the power of asking people to read things they don't understand.

There s a test code maintenance issue I ve been grappling with, and watching others grapple with for a while now. I ve blogged about some infrastructural things related to it before, but now I think its time to talk about the problem itself. The problem shows up as soon as you start writing setUp functions, or custom assertThing functions. And the problem is where do you put this code?

We added support for Marvell's Kirkwood platform to the Debian installer a few months ago and a lot of people have installed Debian testing (squeeze) on devices such as the SheevaPlug and QNAP TS-11x/TS-21x. While this works great, some people would prefer to run Debian stable (lenny). Until recently, I thought we wouldn't be able to support lenny in the installer since the kernel in stable doesn't have support for Kirkwood. However, some work by Frans Pop showed me that it would be quite trivial to change the installer so it would install Debian stable plus the kernel from an alternative repository. So from now on, it's possible to install Debian stable (lenny) on the SheevaPlug, QNAP TS-11x/TS-21x and OpenRD. This installation mechanism uses the squeeze installer to install Debian lenny (stable) plus the kernel from a repository I maintain. This repository usually contains the kernel from Debian testing (although I sometimes add the kernel from unstable if it has some important features). Since some users might be wondering which version to install, here is an overview of the benefits and downsides of each version. Debian lenny (stable): it is the current stable version of Debian (version 5.0).

• lenny has security support (note that this does not apply to the kernel, which we take from squeeze).
• lenny is stable (i.e. it doesn't change, except for security and other important updates).
• The software in lenny is older (since lenny was released in February 2009).

Debian squeeze (testing): it is currently under development and will be the next stable version of Debian (version 6.0).

• squeeze changes every day, with new software coming in.
• squeeze has fairly new software.
• While there is some security support for testing (squeeze), the security support for stable is better.

I volunteered to manage the Debian booth on the Linuxtag 2010 in Berlin. I asked for booth members a few weeks ago on debian-events-eu and debian-user-german - but unfortunately there aren't that many helpers listed at the organisation wiki. So if you attend the Linuxtag or the Debian Miniconf please lend Debian a little bit of your time and apply for the debian booth. You don't have to be a Debian Developer for that, an interested user is enough. I'll plan the booth in a way that nobody will be alone at the booth ;).

I got my air ticket. So, I m able to say that I will go to DebConf10!