Search Results: "ron"

17 May 2010

Russell Coker: Timing Processes

One thing that happens periodically is that I start a process from an interactive shell, discover that it takes longer than expected, and then want to know how long it took. Basically it s a retrospective need to have run time whatever that I discover after the process has been running for long enough that I don t want to restart it. My current procedure in such situations is to run ps from another session to discover when it started and then type date to display when it ends. A quick test with strace showed that bash uses the wait4() system call to determine when a process ends, but passes NULL as the last parameter. If it passed the pointer to a struct rusage then it would have the necessary data. I think it would be a really good feature for a shell to allow you to type something like echo $TIME_LAST_CMD to see how long the last command took. For the common case where you aren t interested in that data it would only involve an extra parameter to the wait4() system call, a small amount of memory allocated for it, and to store yet another environment variable in it s list. A quick Google search didn t show any way of filing wishlist bugs against Bash and I don t think that this is a real bug as such so I haven t filed a bug report. If anyone reads my blog and has some contact with the Bash people then please pass this idea along if you think it s worthy of being included.

Matt Zimmerman: Ubuntu 10.10 (Maverick) Developer Summit

I spent last week at the Ubuntu Developer Summit in Belgium, where we kicked off the 10.10 development cycle. Due to our time-boxed release cycle, not everything discussed here will necessarily appear in Ubuntu 10.10, but this should provide a reasonable overview of the direction we re taking. Presentations While most of our time at UDS is spent in small group meetings to discuss specific topics, there are also a handful of presentations to convey key information and stimulate creative thinking. A few of the more memorable ones for me were: The talks were all recorded, though they may not all be online yet. Foundations The Foundations team provides essential infrastructure, tools, packages and processes which are central to the development of all Ubuntu products. They make it possible for the desktop and server teams to focus on their areas of expertise, building on a common base system and development procedures. Highlights from their track: Desktop The desktop team manages both Desktop Edition and Netbook Edition, on a mission to provide a top-notch experience to users across a range of client computing devices. Highlights from their track: Server/Cloud The server team is charging ahead with making Ubuntu the premier server OS for cloud computing environments. Highlights from their track: ARM Kiko Reis gave a talk introducing ARM and the corresponding opportunity for Ubuntu. The ARM team ran a full track during the week on all aspects of their work, from the technical details of the kernel and toolchain, to the assembly of a complete port of Netbook Edition 10.10 for several ARM platforms. Kernel The kernel team provided essential input and support for the above efforts, and also held their own track where they selected 2.6.35 as their target version, agreed on a variety of changes to the Ubuntu kernel configuration, and created a plan for providing backports of newer kernels to LTS releases to support deployment on newer hardware. Security Like the kernel team, the security team provided valuable input into the technical plans being developed by other teams, and also organized a security track to tackle some key security topics such as clarifying the duration of maintenance for various collections of packages, and the ongoing development of AppArmor and Ubuntu s AppArmor profiles. QA The QA team focuses on testing, test automation and bug management throughout the project. While quality is everyone s responsibility, the QA team helps to coordinate these activities across different teams, establish common standards, and maintain shared infrastructure and tools. Highlights from their track include: Design The design team organized a track at UDS for the first time this cycle, and team manager Ivanka Majic gave a presentation to help explain its purpose and scope. Toward the end of the week, I joined in a round table discussion about some of the challenges faced by the team in engaging with the Ubuntu community and building support for their work. This is a landmark effort in mating professional design with free software community, and there is still much to learn about how to do this well. Community The community track discussed the usual line-up of events, outreach and advocacy programs, organizational tools, and governance housekeeping for the 10.10 cycle, as well as goals for improving the translation of Ubuntu and related resources into many languages. One notable project is an initiative to aggressively review patches submitted to the bug tracker, to show our appreciation for these contributions by processing them more quickly and completely.

16 May 2010

Enrico Zini: New apt-xapian-index plugins

New apt-xapian-index plugins Besides a fair bit of refactoring and cleanup, I've recently added two new plugins to apt-xapian-index: app-install If app-install-data is installed, information about .desktop files will now enter the index. This allows, for example, to limit query results to only those packages that contain .desktop files, which is quite useful, for example for building desktop-oriented package managers. aliases It reads term->aliases mapping from files in /etc/apt-xapian-index/aliases/ or /usr/share/apt-xapian-index/aliases/, and feeds them as synonyms in the index. apt-xapian-index ships an example alias file, to give people who know the wrong software names a chance to find the right ones:
# Aliases expanding names of popular applications
excel       XToffice::spreadsheet
powerpoint  XToffice::presentation
photoshop   XTworks-with::image:raster
coreldraw   XTworks-with::image:vector
autocad     XTworks-with::3dmodel
Notice how it is possible to use index terms that happen to be Debtags tags as synonyms, which yields better results, language independence and extra coolness.

15 May 2010

Christian Perrier: [life] Running update

I didn't write much about running stuff since my update about Paris marathon. The main point was, in general, to gradually resume running activities, after the marathon peak. Which I actually didn't respect at all, as usual..:-)... I already wrote that my "training program" is mostly made of "run as you feel, with no strongly hardcoded program". So, the week-end after the marathon, I ran 11km on Saturday, followed by an half-marathon (alone) on Sunday which I completed in 1h53'31", very close to my 1h51'31" PB for a "half-marathon alone on trails" run. Definitely not what is generally recommended, but, eh, this is how I was feeling... Then, I could continue during the following weeks with my normal program made of 1 or 2 small runs (between 8 and 14km) at lunch time on weekdays, 1 small 10 or 11 km trail run (on this circuit or a longer variant), and a "long" run on Sunday mornings. The long run usually goes from 15km up to 25km. Now, 6 weeks after the marathon, I'll run a 20km trail named Le Trail des Cerfs ("deer trail"), that starts in my neighbourhood and goes through my beloved Rambouillet Forest. I could have chosen the 35K or even 50K race, but preferred being less ambitious with the distance and rather focus and ahieving what will be my first official trail race. I don't expect much differences with the weekly runs, indeed, as most of these are already done in trails, fields and woods with very few paved roads. My goal for this race is 1h45. Later on, I still have the plan to run a full marathon alone, probably on May 31st. This time, contrary to last year where I ran one by going through a very wide loop in Rambouillet Forest, I'll do 4 loops mostly in the city of Maurepas, on flat roads, with possible pit stops at my home (so, no camel back bag, but more focus on good performance). I plan to target 3h55 if the heat is not too high. Further plans are less precise. I'm not sure I'll be able to run the Sur les Traces du Loup trail in La Ville aux Clercs, 2 hours away from our place. That would mean arranging a small week-end there with Elizabeth. Still, that really looks like a great race, with a start at 18:00 to enjoy the end of the day during the longest days of the year. Also, the race and the barbecue that follows appear to be very friendly events in a small French village. On the other hand, the autumn marathon is decided. This will be the Seine et Eure Marathon, a "small (600 runners) marathon very close to our family house in the Seine valley. The race is one of the flatest marathons in France and has the reputation of leading to great times. That will be on October 17th. Target for me: 3h45 (unless I see changes happening util then).

14 May 2010

Matthew Garrett: Using qemu to instrument Windows

Part of the problem that we face in providing Linux hardware support is that we're lucky if there's a spec, and even if there's a spec there almost certainly isn't a test suite. Linux still isn't high on the list of things that vendors test with, so as a result hardware and firmware tend to be written to work with Windows rather than some more ideal notion of what a spec actually says.

This can lead to all kinds of problems. If the spec says we can do something and Windows never does that, we'll often find that there's at least one piece of hardware out there somewhere that breaks in response. But sometimes there'll be several different workarounds, and picking the wrong one may break some other system instead. It's helpful to know what Windows is actually doing, if only because that's the only configuration most systems have been tested with.

The problem is that doing this at the hardware level is next to impossible. I'm sure there are people out there who salivate at the possibility of working out i8042 initialisation sequences by hooking up oscilloscopes to their mouse, but I'm not one of them. There's a much easier alternative involving qemu.

The qemu source base is reasonably large and complex, but that's ok - we don't need to care about most of it. For our purposes we really only want to trace accesses to given bits of hardware. There's three main types of hardware access we're likely to care about: io ports, memory mapped io and pci configuration space. io ports are easy. Each piece of qemu that performs hardware emulation will call register_ioport_read or register_ioport_write. Just grep for those, find the ports that correspond to your hardware and then edit the functions to dump the information you want. PCI configuration space will be handled via pci_default_read_config or pci_default_write_config unless the driver overrides them. Finally, for PCI devices mmio will be handled via pci_register_bar - the last argument is the function called when the memory region is accessed.

All of which makes it pretty easy to watch register reads and writes performed by Windows when it's starting up. Suspend/resume is also an interesting problem, but sadly one that's harder to work with. First of all, you need at least version 0.6c of vgabios in order to indicate to Windows that it's possible to suspend at all. Secondly, for Vista or later you'll also need a WDDM driver for the graphics. Sadly there isn't one for the Cirrus that qemu emulates, which is unsurprising given how ancient it is. So I've had to perform my testing under XP, which was enough to give me an indication as to what Windows does with the SCI_EN bit on resume (answer: Ignores both the bit of the spec that says it should already be enabled and the bit of the spec that says it should never be written by hand). Nice, but if someone would like to write a WDDM driver for Cirrus it'd make my life easier.

The other thing I've been testing is keyboard controller probing. Some Macs deal badly with you banging on the keyboard controller, which is dumb but on the other hand they don't claim to have one. Linux will look at your ACPI tables and use any keyboard controllers it finds there, but if there isn't one it'll go on to try probing the legacy locations. Adding some debug code to the read and write functions of the keyboard driver in qemu and then editing the DSDT in Seabios to remove the declarations for the keyboard showed that Windows will only probe if there's a device with a recognised keyboard PNP ID. No keyboard for you if that's missing. So we probably need to emulate that behaviour as well.

The main reason to do this work is to try to reduce the number of DMI tables in the kernel. These are cases where we alter the kernel's behaviour based on the identity string of the hardware, allowing us to work around functional differences. The problem with this approach is that Windows generally works fine on these machines without knowing anything special about them, and chances are that the tables aren't exhaustive - there may well be some other piece of hardware that's also broken, but the user just gave up and went back to Windows instead of ever letting us know. Using qemu to work out how Windows actually behaves gives us the opportunity to fix things up in a way that will (with luck) work on all machines.

13 May 2010

Aigars Mahinovs: Hacker Neo caught in Latvia

A scandal has been brewing in Latvia over the last half year and yesterday the activity spiked shocking the media and some IT people in the country. I ll go back and explain what happened first, what is happening now and why this could have a heavy impact on IT and journalists in Latvia. At the end of last year, there were rumours that the IT system of Latvia s Internal Revenue System was hacked and millions of documents had been downloaded by multiple organizations. Shortly thereafter more details on the glaring security hole became public (after it was closed). There is a full electronic interface to give all reports to the IRS electronically (at http://eds.vid.gov.lv) and as part of that system you could also view and export monthly report summaries about your organization into XML and PDF files. After the system checked that you are authorized to access the report, you were redirected to the URL to actually download the report by report ID (as a single param in a GET request). Unfortunately, report IDs were predicable and the script that gave the reports for download did not check if you were authorized to get that report. It did not even check if were logged into the system. There were suspicions that the authorization was disabled on purpose to allow to leak data on purpose, but apparently it was an error of forgetting to disable debug code in production environment. The error was discovered only because the firewall administrator noticed an unexplained stable increase of traffic, especially during night hours when typically the traffic fully stopped. Apparently a single hacker (who later identified himself as Neo to the press) discovered the flaw and wrote a script to just try all possible report ids and get as much data out as possible. This had been going on for months, before someone noticed. After the flaw was discovered and a bit of time passed, Neo made his first move he published the list of top salaries in a governmental company, that clearly showed that the top leadership of this company failed to cut their salary by 40%, like everyone elses during harsh budget cuts of 2009. He stripped the names and ids of the specific employees, but named the company which made it pretty easy to figure out who was who. The society was outraged that the top managers in a government owned company failed to comply with the strict pay cut that everyone else in government had to endure. But after a few weeks the outrage subsided and no action followed from the government or law enforcement. Neo continued to release documents detailing salaries of top managers in different Latvian government companies. And each time after short outrage, nothing happened. Neo gave an interview where he said that he was disappointed in the passivity of the Latvian people in face of such blatant injustices. After a few month Neo went silent, promising to return before parliamentary elections this fall. However, this week a new development shocked everyone in the middle of the night two police SWAT teams went into action: one detained Ilm rs Poik ns, a researcher in artificial intelligence at the University of Latvia s Computer Science department and another raided the home of a Latvian TV journalist Ilze Nagle who interviewed Neo. Poik ns confessed of being Neo the next day and was released (with travel restrictions, pending trial) today. Politicians reacted immediately opposition demanded the resignation of the Interior Minister over such blatant disregard of freedom of press and another politician (who is also a famous lawyer) Aleksejs Loskutovs volunteered to defend Neo pro-bono (on Twitter, no less). Almost all Latvian online media have the arrest of Neo and the raid on the home of a journalist as main stories of the day. As a legal titbit, we also know that Neo is being charged with breaking statutes 145 and and 244p2 of the criminal law. Statute 145 is hard to find applicable in this situation as talks about actions done by people authorized (..) to access [private] information . Statute 244p2 will also be hard to pin down as it mentions influencing system resources of (an IT system) and if such action caused severe harm . It looks like the first part talks about at least a DoS attack (which did not happen in this case) and also there was no measurable harm from these leaks. Also Neo was careful to strip all personally identifying information (such as names, social security numbers and addresses of the employees in question), so it will be hard to pin him on that. Also no actual breaking or other modification of an IT system occurred. And no specialized software was used beyond a trivial script such as :
for i in range(0,7000000):
    wget('https://eds.vid.gov.lv/getRep.aspx?id='+str(i))
A lot of commentators on the Internet likened the situation to walking trough an unlocked door and stealing something. I think that analogy is very incorrect there was no door, and nothing went missing after the action. I came up with a different analogy there was this corridor with a lot of doors in IRS, locked steel doors. You were instructed to go to a room with a specified number and given a key to that room to unlock it and see your secret info. However, that corridor opened out to the street on one end, oh and also the walls of the rooms with all the secrets were transparent. So Neo walked into the corridor, looked at some of the secrets, wrote them down (to remember them better) and then went out and discussed the worst examples abuses of power he saw. In the end IRS had to learn their lesson if you have to put naked photos of yourself on the Internet (or something equally embarrassing), then make damn sure you password protect that, but if you don t then don t cry that someone hacked you and stole you pictures. What other people think:
http://freespeechlatvia.blogspot.com/2010/05/neo-released-under-restrictions.html We ll see how the story develops soon.

Steve Kemp: Can you stand on your head?

I've been a little quieter than usual recently, having spent more time outdoors putting cute people in front of the camera. However that said I've still been doing some things. Most interestingly I've given away my first ever project. The collection of small scripts known as xen-tools (which was initially a sleazy hack to go with a small introduction to Xen article) has now got new developers, and a new home where development is continuing. This isn't the first time I've stopped working on something, but it is the first time I've explicitly "given away" a project. (Mostly on the basis that if I didn't care nobody else did either, or people cared but were too busy/unable to actually do soemthing useful.) I'll be following new updates with interest, even though these days I'm 100% Xen-free. No need to go into huge details about why, but I'm enjoying KVM. Having said that I recently got into a huge mess with a combination of LVM, KVM, and ext3. I've written up the details on ServerFault in the optimistic hope that somebody will report having experienced a similar problem. If you have seen something similar I'd love to hear from you. Otherwise I'm genuinely at a loss to understand what went wrong, and why things failed. I could suspect hardware issues, but that feels like a cop-out, albeit one that has a potential solution (Mad Hatter: All Change!) rather than my current answer and explaination "It broke. I don't know why. It might happen again. It might not. Trust me?". ObFilm: Alice In Wonderland (1951 version.)

11 May 2010

Sune Vuorela: The Debian-KDE specific things ?

So. I was wondering, which nice distro specific tools do exist in debian/gnome or in $other/kde that debian/kde is missing? We have kalternatives for managing alternatives, we have a update notifier frontend in progress and after google summer of code, hopefully a package management frontend, aptitude-qt. (Made by Piotr). But what other distribution specific tools are we missing for Debian-KDE ?

Gunnar Wolf: e-voting seminar, Mexico City, tomorrow

If anybody is interested or can attend: Tomorrow and the day after (Wednesday and Thursday 12-13/05/2010) FLACSO (a leading Social Sciences faculty spanning countries all over Latin America) and IFE (Mexico's federal electoral authority) are presenting a seminar on e-voting: Electronic vote experiences in Mexico and around the world. Sadly, I will be unable to attend, as I will be on the road once again (Ecuador!). However, I hope those among my readers who are interested in the topic can attend or at least follow the audio and video transmission from the IFE website. The seminar will be held at IFE's auditorium Viaducto Tlalpan No. 100, Col. Arenal Tepepan, Delegaci n Tlalpan, c.p. 14610, M xico, DF. [update] If you don't know what is my stand on this topic (and can read Spanish), please read this short article. In short: I am against e-voting, and hope we are still in time (and can push at the right places) to avoid it becoming the rule in our country.

Biella Coleman: The end of a (V)era

The end of (V)era I did not think this day would come anytime soon. When the time came when the time came for my mom to pass on I thought I would be not only be accepting and ready but would welcome it with open arms. After a decade with Alzheimers, after two years bound in bed, and after months of barely able to sculpt even a word, I thought I was ready to see my mom breathe her last breath. But when the end came, the surprise, shock, and sadness were unmistakably there and I was silently urging my mother to keep fighting to breathe. In the end, it was her lungs that took her from this life as most everything else was still running smoothly and strong. She smoked like a chimney, so when she got pneumonia in late April, her diminished lung capacity made it hard for her to breathe, to get the oxygen needed to be present. But I expected a full recovery as she had reacted almost immediately and positively to the antibiotic treatment. She was still physically strong as a horse, for example, during this hospital stay, three of us had to hold her down when we did various tests. But one day to the next her blood pressure plummeted. Since I was unaware of this, I thought she was just (and finally) sleeping deeply after two days of barely doing so. After 8 hours of barely moving, the nurse let me know that in fact her vital signs were dim as were her chances of living. That evening, I crawled into bed with her for one last night together. Finally the next morning, as I was stroking her face, I witnessed her final breath. This is what I read at her service.

Biella Coleman: The end of a (V)era

The end of (V)era I did not think this day would come anytime soon. When the time came when the time came for my mom to pass on I thought I would be not only be accepting and ready but would welcome it with open arms. After a decade with Alzheimers, after two years bound in bed, and after months of barely able to sculpt even a word, I thought I was ready to see my mom breathe her last breath. But when the end came, the surprise, shock, and sadness were unmistakably there and I was silently urging my mother to keep fighting to breathe. In the end, it was her lungs that took her from this life as most everything else was still running smoothly and strong. She smoked like a chimney, so when she got pneumonia in late April, her diminished lung capacity made it hard for her to breathe, to get the oxygen needed to be present. But I expected a full recovery as she had reacted almost immediately and positively to the antibiotic treatment. She was still physically strong as a horse, for example, during this hospital stay, three of us had to hold her down when we did various tests. But one day to the next her blood pressure plummeted. Since I was unaware of this, I thought she was just (and finally) sleeping deeply after two days of barely doing so. After 8 hours of barely moving, the nurse let me know that in fact her vital signs were dim as were her chances of living. That evening, I crawled into bed with her for one last night together. Finally the next morning, as I was stroking her face, I witnessed her final breath. This is what I read at her service.

Ian Wienand: A short tour of TERM

I'd wager probably more people today don't know what the TERM environment variable really does than do (yes, everyone who used to use ed over a 300-baud acoustic-coupler is laughing, but times have changed!). I recently got hit by what was, at the time, a strange bug. Consider the following trivial Python program, that uses curses and issues the "hide the cursor" command.
$ cat curses.py
import curses
curses.initscr()
curses.curs_set(0)
curses.nocbreak()
curses.echo()
curses.endwin()
If you run it in your xterm or linux console, nothing should happen. But change your TERM to vt102 and it won't work:
$ TERM=vt102 python ./test.py
Traceback (most recent call last):test-curses.py
  File "./test-curses.py", line 5, in <module>
    curses.curs_set(0)
_curses.error: curs_set() returned ERR
(obvious when you explicitly reset the term, no so obvious when some script is doing it behind your back -- as I can attest!) So, what really happened there? The curs_set man page gives us the first clue:
curs_set returns the previous cursor state, or ERR if the requested visibility is not supported.
How do we know if visibility is supported? For that, curses consults the terminfo library. This is a giant database streching back to the dawn of time that says what certain terminals are capable of, and how to get them to do whatever it is you want them to do. So the TERM environment variable tells the terminfo libraries what database to use (see /usr/share/terminfo/*) when reporting features to people who ask, such as ncurses. man 5 terminfo tells us:
cursor_invisible civis vi make cursor invisible
So curses is asking terminfo how to make the cursor invisible, and getting back "that's not possible here", and hence throwing our error. But how can a person tell if their current terminal supports this feature? infocmp will echo out the features and what escape codes are used to access them. So we can check easily:
$ echo $TERM
xterm
$ infocmp   grep civis
					   bel=^G, blink=\E[5m, bold=\E[1m, cbt=\E[Z, civis=\E[?25l,
$ TERM=vt102 infocmp   grep civis
$
So we can now clearly see that the vt102 terminal doesn't support civis, and hence has no way to make the cursor invisible; hence the error code. If you're ever looking for a good read, check out the termcap/terminfo master file. The comments are like a little slice of the history of computing!

10 May 2010

Romain Beauxis: Ubuntu, the firm and its employees

This post is not meant to be offensive in any way. I would just like to write down things that came to my mind recently. It all started when I was considering applying for the right to upload the package I maintain into Ubuntu. Looking at the procedure, I realized that this needs to be present during an IRC meeting at a given date online. This stopped me immediately. I am working most of my day-time and, given the time zone I am living now, this was not an option. But, since I had registered to the corresponding mailing list, I started receiving mails of other applicants that where trying to arrange things with their work and private life to be able to participate. Now, today, I stumbled upon these notes of Mark's presentation in Bruxelles, relayed by a friend that is not either working for Ubuntu. Reading it gave me a strong taste of management-oriented presentation, with all this sort of "yeah we want our product in every box next year" propaganda. Not like Steve Ballmer, fortunately, but not far perhaps. So, yeah, there is something that makes me uncomfortable with Ubuntu. I am really happy by its success and I will continue to recommend it to friends around me. However, I wonder if, in a way, Ubuntu is not a model of commercial company where a minimal core of developers are actually payed, while a second layer of developers are fan-boys that are willing to participate as if there were employees but without being actually payed, and a third layer being the open-source developers whose work is actually what makes 90% of the core of the OS. A very good illustration is at the end of the note from Mark's presentation. It reads:
Somebody asks about advertising Ubuntu, looks like there are no plans for the moment to actually advertise Ubuntu across traditional media. Mark uses the Ubuntu Manual as an example of how Ubuntu gets advertised and will reach the mainstream, Mark points out myself and the room applauds the Ubuntu Manual Project! Woohoo!
Now, guess what:
About Us We are a group of individuals trying to promote the use of Ubuntu and its variants. This is a private web site that is not connected in any official or financial way with Ubuntu, Canonical PL or Ubuntu.com. Official Ubuntu guides and downloads are freely available at www.ubuntu.com. Ubuntu is a registered trademark of Canonical PL
Source: http://ubuntumanual.org/about-us

Giuseppe Iuculano: Chromium ready for unstable

I m currently testing the new chromium version for unstable. I tried to remove embedded code copies, and I m pleased with the result. Ubuntu i386 binary: -rwxr-xr-x root/root 35804728 2010-05-05 12:01 ./usr/lib/chromium-browser/chromium-browser Debian i386 binary: -rwxr-xr-x root/root 20693336 2010-05-09 21:33 ./usr/lib/chromium-browser/chromium-browser The latest blocker for an upload in unstable is a minor bug in libv8, but Antonio Radici should upload a [...] No related posts.

Giuseppe Iuculano: Chromium ready for unstable

I m currently testing the new chromium version for unstable. I tried to remove embedded code copies, and I m pleased with the result. Ubuntu i386 binary: -rwxr-xr-x root/root 35804728 2010-05-05 12:01 ./usr/lib/chromium-browser/chromium-browser
Debian i386 binary: -rwxr-xr-x root/root 20693336 2010-05-09 21:33 ./usr/lib/chromium-browser/chromium-browser The latest blocker for an upload in unstable is a minor bug in libv8, but Antonio Radici should upload a fixed version this evening :) No related posts.

Stefano Zacchiroli: tickler file for maildir

snooze your INBOX A few days ago Chris made me realize that my GTD setup was still missing a piece: a proper tickler file implementation. As I wanted an IMAP-free implementation, I've rolled up my own: rotate-tickler. Nothing too fancy: shell script triggered by cron, which moves mails around a set of DELAYED. 1,.. maildir-s. Still, it is careful about clashes (which shouldn't happen according to the maildir specs, but better be paranoid with mails), and properly cleans up the "seen" flag for the final (re)delivery in INBOX. Give it a try and feel free to git format-patch-me your improvements. (BTW, does it make any sense today to publish any piece of software, no matter how small, without shelving it into some DVCS?)

9 May 2010

Robert Collins: Maintainable pyunit test suites

There s a test code maintenance issue I ve been grappling with, and watching others grapple with for a while now. I ve blogged about some infrastructural things related to it before, but now I think its time to talk about the problem itself. The problem shows up as soon as you start writing setUp functions, or custom assertThing functions. And the problem is where do you put this code?
If you have a single TestCase, its easy. But as soon as you have two test classes it becomes more difficult. If you choose either class, the other class cannot use your setUp or assertion code. If you create a base class for your tests and put the code there you end up with a huge base class, and every test paying the total overhead of your test needs, rather than just the overhead needed to test the particular system you want to test. Or with a large and growing list of assertions most of which are irrelevant for most tests.
The reason the choices have to be made is because test code is just code; and all the normal issues there separation of concerns, composition often being better than inheritance, do-one-thing-well all apply to our test code. These issues are exacerbated by pyunit (that is the Python unittest module included with the standard library and extended by various projects)
Lets look some (some) of the concerns involved in a test environment: Test execution, fixture management, outcome decision making. I m using slightly abstract terms here because I don t want to bind the discussion down to an existing implementation. However the down side is that I need to define these terms a little.
Test execution by this I mean the basic machinery of running a single test: the test framework calling into user code and receiving back an outcome with details. E.g. in pyunit your test_method() code is called, success is determined by it returning successfully, and other outcomes by raising specific exceptions. Other languages without exceptions might do this returning an outcome object, or passing some object into the user code to be called by the test.
Fixture management the non trivial code that prepares a situation where you can make assertions. On the small side, creating a few object instances and glueing them together, on the large end, loading data into a database (and creating the database instance at the same time). Isolation issues such as masking out environment variables and creating temp directories are included in this category in my opinion.
Outcome decision making possibly the most obtuse label I ve ever given this, I m referring the process of deciding *what* outcome you wish to have happen. This takes different forms depending on your testing framework. For instance, in Python s doctest:
>>> x
45
provides a specification the test framework calls str(x) and then compares that to the string 45 . In pyunit assertions are typically used:
self.assertEqual(45, x)
Will call 45 == x and if the result is not True, raise an exception indicating a Failure has occured. Unexpected exceptions cause Errors, and in the most recent pyunit, and some extensions, other exceptions can signal that a test should not be run, or should have failed.
So, those are the three concerns that we have when testing; where should each be expressed (in pyunit)? Pragmatically the test execution code is the hardest to separate out: Its partly outside of user control , in that the contract is with the test framework. So lets start by saying that this core facility, which we should very rarely need to change, should be in TestCase.
That leaves fixture management and outcome decision making. Lets tackle decision making if you consider the earlier doctest and assertion examples, I think its fairly clear that there are multiple discrete components at play. Two in particular I d like to highlight are: matching and signalling. In the doctest example the matching is done by string matching the reference object(s) are stringified and compared to an example the test writer provides. In the pyunit example the matching is done by the __eq__ protocol. The signalling in the doctest example is done inside the test framework (so we don t see any evidence of it at all). In the pyunit example the signalling is done by the assertion method calling self.fail(), that being the defined contract for causing a failure. Now for a more complex example: testing a float. In doctest:
>>> %0.3f % x
0.123
In pyunit:
self.assertAlmostEqual(0.123, x, places=3)
This very simple check that a floating point number is effectively 0.123 exposes two problems immediately. The first, in doctest, is that literal string comparisons are extremely limited. A regex or other language would be much more powerful (and there are some extensions to doctest; the point remains though the operator is not enough). The second problem is in pyunit. It is that the contract of assertEqual and assertAlmostEqual are different: you cannot substitute one in where the other was expected without partial function application something that while powerful is not the most obvious thing to reach for, or to read in code. The JUnit folk came up with a nice way to address this: they decoupled /matching/ and /deciding/ with a new assertion called assertThat and a language for matching expressed as classes. The initial matcher library, hamcrest, is pretty ugly in Python; I don t use it because it tries too hard to be english like rather than being honest about being code. (Aside, what would is_() in a python library mean to you? Unless you ve read the hamcrest code, or are not a Python programmer, you ll probably get it wrong. However the concept is totally sound. So, outcome decision making should be done by using a matching language totally seperate from testing, and a small bit of glue for your test framework. In testtools that glue is assertThat , and the matching language is a narrow Matcher contract (in testtools.matchers) which I m going to describe here, in case you cannot or don t want to use the testtools one.
class Matcher:
    def __str__(self):
        "Describe this matcher."""
    def match(self, something):
        """Determine if something is matched.
        :param something: Something to match.
        :return: None if something matched, or a Mismatch object otherwise.
        """
class Mismatch:
    def describe(self):
        """Describe a mismatch that has occured."""
This permits composition and inheritance within your matching code in a pretty clean way. Using == only permits this if you can simultaneously define an __eq__ for your objects that matches with arbitrarily sensitivity (e.g. you might not want to be examining the process_id value for a process a test ran, but do want to check other fields).
Now for fixture management. This one is pretty simple really: stop using setUp (and other similar on-TestCase methods). If you use them, you will end up with a hierarchy like this:
BaseTestCase1
 +TestCase1
 +TestCase2
 +BaseTestCase2
   +TestCase3
   +TestCase4
   +BaseTestCase3
     +TestCase5
     ...
That is, you ll have a tree of base classes, and hanging off them actual test cases. Instead, write on your base TestCase a single glue method e.g.
def useFixture(self, fixture):
      fixture.setUp()
      self.addCleanup(fixture.tearDown)
      return fixture
And then rather than having a setUp function which performs complex operations, define a fixture an object with a setUp and a tearDown method. Use this in tests that need that code::
def test_foo(self):
      server = self.useFixture(NewServerWithUsers())
      self.assertThat(server, HasUser('fred'))
Note that there are some things around that offer this sort of convention already: thats all it is convention. Pick one, and run with it. But please don t use setUp; it was a conflated idea in the first place and is a concrete problem. Something like testresources or testscenarios may fit your needs if it does, great! However they are not the last word they aren t convenient enough to replace just calling a simple helper like I ve presented here.
To conclude, the short story is:
  • use assertThat and have a seperate hierarchy of composable matchers
  • use or create a fixture/resouce framework rather than setUp/tearDown
  • any old TestCase that has the outcomes you want should do at this point (but I love testtools).

6 May 2010

Gunnar Wolf: 5000 Km

In an amazing but meaningless feat of synchronicity, Toxicore's post led me to check how much distance have I done while excercising I have mainly done this cycling, although some running also comes into play. Turns out I was just about to break the same mark he did. From my Nokia Sportstracker records: Before breaking the 5000 Km mark, yesterday at noon:
Yesterday excercise records, at noon After breaking the 5000 Km mark, yesterday at night:
Yesterday excercise records, at night So, following Bubulle's style... If I have done 5000 Km since January 2008, when do you think I should reach 5500? 6000? 7000? Infinity and beyond?

Russ Allbery: git-pbuilder 1.10

This is a minor update of the glue script that I use between git-buildpackage and pbuilder (with cowbuilder). The primary feature it has over other versions of a similar script is that one can set an environment variable to specify the chroot in which to build. This update changes the -i regex to include all of the Git-relevant parts of the standard dpkg-source exclusion regex. Before, the regex tried to exclude only .git directories, but also used insufficient quoting and ended up excluding any file containing git. That should be fixed, and now it should exclude .gitattributes files as well. Thanks to Allard Hoeve for the bug report. You can get the latest version from my miscellaneous scripts page.

4 May 2010

Wouter Verhelst: On clue and opinions

How does one define one's "national identity"? One way is to look at where a person is born. By that definion, I could be called Flemish, Belgian, or European, whatever you think about it. Another way to look at it is how one feels about one's own identity. This, of course, is a much more accurate way to describe a person; and by that definition, I am either Belgian or European. I do not feel any affinity with much of the so-called "flemish" identity, nor do I wish to be associated with it any more than is strictly necessary. Obviously, some people feel different about that for themselves. This I can understand. In fact, the statement to feel more Belgian than Flemish is probably not one that is very common in this general area. I am not stupid; I know that Belgium is not a Nation (it is a state, but that is a different matter). I just do not feel a lot of warm fuzzy feelings about Flanders. Naturally, other people have other feelings about that. This, too, I can understand. The logical result of the above is that my personal opinions about the somewhat convoluted politics in this country are not shared by many; that when I state my opinion, I may expect opposition, or at the very least someone stating why they disagree with me. This, of course, I can also understand. But what I cannot understand is that every time I state my opinion, someone must come out and insult my intelligence by telling me I "do not have a clue". Please do not mistake my disagreement for misunderstanding. Martijn asks me to "check the facts". Let's do that, shall we?
  1. There's been a judgement by the constitutional court that the current situation regarding which party may run for election in the BHV region is unconstitutional.
  2. One group of politicians wishes to resolve this issue by splitting the BHV region in two. That would certainly solve the issue.
  3. Another group of politicians dislikes this solution. They may be willing to go along, but they want other concessions before they will do so. This is what I referred to when I said that there were "a huge heap of all kinds of side issues" in my previous post.
  4. The first group does not wish to make such concessions. Some of their numbers have even stated in the press that there is no need to make them, because, given the first point, there's only one solution and that is to split.
That last point is, of course, complete and utter bullshit. It would hold merit if there was only one way to solve a constitutional crisis; but by definition, there are always at least two ways: one is to change the law to comply with the constitution, the other is to change the constitution to make the law no longer be unconstitutional. So a group that feels strongly about one of these two possible solutions should be prepared to negotiate with the other group so as to hammer out an agreement that benefits both parties. In some cases, this will mean doing things that they do not completely like; but if they're not willing to do that, at all, they have no business being politicians. For the longest time during the early negotiations, however, both parties have stubbornly refused to even acknowledge the other position, let alone talk about it. Now, for clarity, I'm not advocating that the latter of the two options in that previous paragraph is what should happen. Frankly, I don't even care what happens, so long as they manage to resolve the issue at hand in some way. What I do care about, however, is that this country has been in a near-constant state of stalemate over these past few years for an issue that only matters to politicians; I don't care who or what a group of people in a (to me) obscure part of the country can vote for. This country needs a stable government which can act on things that actually matter. Like, say, the international economic crisis of the past two years; I think it matters much more whether the man in the street has a job than whether John Q. Politician will past these next elections.

Next.

Previous.