Search Results: "nate"

27 March 2009

David Welton: CodingHorror and Open Source Economics

Jeff Atwood writes about the economics of open source software: http://www.codinghorror.com/blog/archives/001247.html Well, he actually doesn't say as much, but that's what it's about. Someone really ought to do a book about this subject. I have some ideas - any takers? It's always tough to write about because there's so much material. It's a topic that is near and dear to my heart - almost 10 years ago, I joined the Free Software Business mailing list, as I got curious about the intersection of free software and business, and how the whole thing worked. I've learned a lot in the meantime, including a stint at Linuxcare, which was one of the Linux companies during the dot com boom, and am still fascinated by the subject. Before we get started, let's reiterate something:
  • It's clear that open source software works. There's just no doubt about it - it's been through ups and downs, the dot com boom and bust, and it will trundle happily (well, as happy as anyone is with this economy) through the recession too.
That point bears repeating. Everything else needs to take it into consideration. Could it work better? Differently? Perhaps, but it does work, so you need to understand it very well before you fiddle with it. Jeff writes:
for the average user? Heck, how realistic is this for the average programmer? Even if you're the type of macho, gung-ho programmer who can master an alien code base just to get some small pet feature or bugfix in -- do you honestly have the kind of time it would take to do that? Sometimes, when people say this:
The source code is freely available. If you feel so strongly about this bugfix/tweak/feature/plugin, why don't you code it yourself?
They're really saying this:
F**k you.
Which is of course a terrible analysis of open source, and its pros and cons. I wrote the following comment on Hacker News:
Heck, how realistic is this for the average programmer? ...
"Actually, I have submitted lots of (mostly small and inconsequential) bugs and minor features over the years, to projects such as the Linux Kernel, Tcl, Erlang, Ruby, Rails, Apache, and probably lots of others I'm forgetting right now. I actually enjoy it quite a bit, come to think of it - there's something cool about diving in and studying something new and different, and a thrill that comes with nailing the bug. Sometimes, the knowledge required to actually fix it is too much, and you have to settle for sending in a good bug report, but in other cases it's very much possible to hack up quick, useful things despite not knowing much about the system in question. And, no, I'm not really that great a programmer... I'd consider myself fairly average." And I know I'm not alone, as I see similar sorts of people in Debian and the Apache Software Foundation. Debian, infact, is all about those kinds of tweaks to improve the operating system as a whole. And this sort of exchange goes on all the time, as evidenced by the huge success of a large number of open source projects. Now, granted, most projects are really driven by one or a few really committed hackers, who do the lion's share of the work. However, that large network of people who contribute, test, document, and suggest is certainly important to any open source project. As long as we're looking at my anecdote, let's look at it in a bit more depth - most of those projects are developer tools or closely related. A cynic might note that Jeff comes from the world of Microsoft, where the core developer tools are provided by a company and 'giving back' is limited to payments and bug reports. But leaving that be, I think the enormous success of stuff like Ruby, Perl, PHP, Python, Tcl, Apache, Linux, GCC, and on and on and on indicates that, at least in terms of stuff that developers share between one another, open source works. However, there are problems, one of which is that he (or, in unfortunately rather rarer cases, she) who writes the code does not necessarily get the money. And without money, he or she who writes the code must dedicate themselves to some means to actually accumulate a bit of money in order to pay for living expenses. Most likely, this is time taken away from the open source project, which is a loss for all the people who benefit from it. Proprietary software solves this feedback loop in a neater way (generally) - companies that create good products get money for their efforts, which can then be reinvested in development. Jeff suggests that "sponsorship seems like a constructive way for people who are unable or unwilling to write code to affect the direction of a project.", which is ok as far as it goes, and more or less works here and there, but hopefully he's not thinking about trying it on a larger scale. This sort of thing has been tried before, and never really seems to take off. Part of the problem is that it can work on a person-to-person basis, but is almost impossible to standardize in a way that it makes sense to have some kind of clearinghouse that operates as a business. There are simply too many factors - what kind of open source project it is, what motivates the people behind it, how the transaction costs of defining a specification, monitoring the results, sending money and so on compare to the actual value of a fix... the list goes on. Ultimately, it's a nice idea, but it doesn't seem to work out too well in practice. Anyway, while it's a good thought on Jeff's part, at the end of the day, it is fairly evident that he doesn't have a lot of first hand knowledge of open source business and economics. That said, to some degree, open source took everyone by surprise, and, lacking the nice, simple feedback loop that proprietary software has, can be a lot more difficult to get a handle on. I find it quite pleasant that way - it attracts both hard-headed business types and social-experiment idealist visionaries; both huge corporations (IBM!) and a huge number of individuals; both people who have gotten quite wealthy from open source as well as most of the rest of us who manage to make ends meet but still love it. It's not something that's easy to pigeonhole, and as such it's hard enough to figure out the many ways in which it does work, let alone figure out how to improve on it. Stay tuned for further installments: Software Economics - Public Goods

Matthew Garrett: Reducing disk use

UNIX filesystems generally store three pieces of timing information about files - ctime (when the file was changed in any way), mtime (when the file contents, as opposed to its metadata, was last changed) and atime (when the file was last accessed by any process). This is a usefully flexible system, but the semantics of atime can be troublesome. atime must be updated every time a file is read, causing a read operation to instead become a read/write operation. This results in a surprising amount of io being generated in normal filesystem use, slowing the more relevant io and causing disks to spin up due to atime updates being required even if the file was read out of cache. It also results in a lot of unnecessary activity on flash media which may reduce their lifetime.

One option is to disable atime updates entirely. The problem with this approach is that certain applications depend on atime. This is especially common in mail clients which compare atime to mtime in order to determine whether a mailbox has been read since it was last modified. So, unfortunately, disabling atime entirely is impractical as a default. Back in 2006, Valerie Aurora submitted a patch that worked around this issue. The new relatime option meant that atime would only be updated if it would otherwise be older than ctime or mtime. Mail clients became happy and the world rejoiced.

Unfortunately, it turned out that there was one other common case of atime being used. Applications like tmpwatch monitor files in /tmp and delete them if they appear unused. In this case, "unused" means "has an atime older than a certain date". Since merely reading files doesn't update the ctime or mtime, relatime wouldn't cause the atime on these files to be updated and tmpwatch would happily delete them - even if users were reading them on a daily basis.

Ingo Molnar submitted a patch to add a further heuristic to the relatime behaviour. With it, the atime of a file will be updated if it's older than mtime, older than ctime or (and this is the important one) more than 24 hours in the past. This deals with the tmpwatch case nicely, while still providing a significant reduction in the quantity of atime updates.

Fedora shipped this patch for several releases, and Ubuntu have used it by default since 8.04. Unfortunately there were some concerns over certain aspects of its behaviour (in respect to its interface as opposed to the relatime functionality itself) and it never got merged. I pushed a trimmed down version that purely implements the change to the relatime behaviour, and earlier today Linus merged it and a further patch that makes relatime the default behaviour on Linux.

Most users won't notice this change in behaviour at all, other than as a small improvement in io performance and a reduction in the number of drive spinups. For users that do have issues, a new strictatime mount option has been added - using this will require an updated mount command, but it's a trivial patch. I'd be surprised if there are any real world use cases that are negatively affected by this, especially since it's been default behaviour in several distributions for a while, but there's always the potential that someone will be tripped up by it. We'll see.

26 March 2009

John Goerzen: Time

When I was a child, I was learning to play piano. My parents didn t have one at the time, but my grandparents did. Every evening, we d go over to their house so I could practice. I suspect I was more interested in other things most of the time, though whether Grandma has some cherry moos in her fridge, or whether Grandpa would play chess with me or do something fun in his workshop. Grandma would often be in her curlers, and we d often leave her in the evening saying goodbye in the pink fluorescent lights she had shining on her plants by the door. Other times, we d just go to visit, or I d ride my bike over. I liked helping Grandma cook or bake peppernuts. And when I was trying to learn chess from Grandpa, and Grandma thought he was being too hard on me, she d say something to him in German and I d magically start winning. Grandma volunteered at a local thrift store weaving rugs. These were made by hand on old looms in a traditional way, mainly out of donated clothes and drapes that were too tattered to sell. My mom used to drop me off with grandma while she went shopping. I maybe brought my lunch, and watched grandma, until one time I asked her if I could help. She let me, and eventually the store manager told grandma that I ought to be able to work on my own loom. Pretty soon we had a routine down: in summer, Grandma would pick me up at home, I d read a computing magazine in the car to town, and work at a loom side-by-side with Grandma. That lasted until I got too busy in high school. But ever since then, Grandma delighted in telling this whole story almost every time she saw me, and she remembered word for word what the store manager said something I never quite could remember. I remember watching Grandma care for my brothers when they were sick, or helping out my parents with whatever they needed help with. We had a tradition for a number of years of spending New Year s Eve at Grandma and Grandpa s house, and going back home the next morning. The three of us boys slept in a cold upstairs room, but we didn t care because the bed had more layers on it than we could count, and an electric blanket besides. When my Grandpa Klassen passed away in 1990, Grandma missed him but didn t get too worked up about it. She was convinced that he was in a better place, and appeared to be at peace with it. Death was a normal part of life to her, and it didn t surprise her that it happened. In the last few years, Grandma s health has been failing. Her knees have been bothering her for years, and she has also been struggling with dementia for a few years. It s been hard on me to visit her, because in some ways she hasn t been the same person I remember for awhile now, and in other ways she s exactly the same. Lately she hasn t always remembered where I live, or what Terah and Jacob s names are, we knew to expect that and are respectful of the situation. For years now, Grandma has been saying, I don t know why the Good Lord doesn t take me up to be with him yet. My bags are packed and I m ready! Last time I saw her, I mentioned that we used to do weaving together, and she couldn t remember. That was a sure sign to me that things had taken a turn for the worse. Our last several visits have ended with a big hug, and her still iron grip on my hand, with her saying, Thank you for coming! My family is so good to me. We love each other! She s been battling infections and heart problems the past few weeks. This evening, I got an email from my mom saying It s time to say our goodbyes. She had been to see Grandma today, and Grandma still managed to tell her, We love each other. As much as time has changed her in the past few years, she s still there, the same loving Grandma as ever.

25 March 2009

Bastian Venthur: Reportbug finally has a GUI!

It s good to see that reportbug 4 finally has a graphical user interface it s GTK, but nobody s perfect :) Anyways, I m really glad to see reportbug aiming to improve it s usability. Reportbug finally shows all the bugreports for a package in a list which is easily filterable. So when you re confronted with a huge list of bugreports but searching for a specific problem, you can just enter a few letters in the filter and the buglist will shrink immediately. This missing feature was basically the most important reason why I developed reportbug-ng: Users are expected to don t report bugs which are already filed against a package, but the main tool which was used back then to report the bugs didn t make this task trivial. Now it does and it s a good thing! reportbug and reportbug-ng Compared to reportbug-ng, reportbug shows the bugreports of a package in a tree-view, while rng shows them in a table-view. The difference is that reportbug could theoretically group serveral bugs together, like merged bugs. It currently only uses this feature to group bugs of the same severity and status. Rng does that by coloring the lines. Unfortunately in reportbug all nodes are collapsed by default, which is not very usable since you have to expand every node in order to see all bugreports. This costs the user several clicks in order to see all bugreports. It s also not possible to sort the bugs via the columns of the table, e.g. sort by date of last action which I think is a huge feature in order to quickly find certain kind of bugs (old bugs, bugs sorted by date of last activity) but that s probably easy to fix. Another good news is that Reportbug now brings it s own editor so the user is not forced anymore to use an console based- and maybe unfamiliar editor to compose his bugreport. This is a huge step forward regarding usability. It also moves the concept of writing a mail to the BTS a bit more away from the user which is a good thing. Maybe the BTS will support an alternative way to receive bugreports in the future so mail clients may become superfluous for reporting bugs. Under the hood reportbug still parses the HTML from http://bugs.debian.org/ go gain the bugreport instead of the BTS SOAP interface. This coupling is of course dangerous since even the smallest change in the BTS webdesign could break reportbug. But SOAP support is already on the todo list of reportbug s devs. I can only encourage this step, when reportbug-ng moved away from HTML-parsing to SOAP the actual amount of work it took was much less then I thought, I think it took me just one or two days and as a result there is python-debianbts which provides a bridge between the SOAP interface and Python. In summary I think this is an important improvement for reportbug. There are still some minor issues like the welcome screen always showing, the nodes collapsed by default and the columns not beeing sortable but those are small issues and easy to fix. I m very happy to see this improvement, since it will motivate the not so tech-savvy folks to report bugs, and more bugreports will ultimately lead to a better distribution. Good work guys!

Wouter Verhelst: 93

films=# SELECT COUNT(*) FROM films;
 count
-------
    93
(1 row)
There were several additions, including one which I actually had when I wrote the previous entry but forgot to add to the database. One of the more recent additions is Memoirs of a Geisha, an intriguing and compelling piece about some of the more mysterious parts of Japanese culture, and how World War II affected this wrapped in a love story. It's unfortunate that the movie depicts Geisha as prostitutes (which they are not), but other than that, it's a pretty solid movie. Another movie which I recently acquired is Requiem for a Dream, a somewhat harsh movie (definitely no happy ending here) about addiction in all its forms. I think this movie is a must-see for everyone, really; not only because of its message (and boy, does it have one), but also because of the way the movie is filmed; split screen, snorricam, and other inventive camera and montage techniques all add to the movie's excellence. And then there's the soundtrack. Oh boy, that sountrack. There were more, but those were not as interesting.

Daniel Kahn Gillmor: Leslie Pack Kaelbling for Ada Lovelace Day

So i'm a couple hours late for Ada Lovelace Day (in my time zone at least), but i wanted to mention Leslie Pack Kaelbling, an excellent technologist who has also had a significant impact on my life. Her extremely short biography says (in full):
Leslie Pack Kaelbling is Professor of Computer Science and Engineering at the Computer Science and Artificial Intelligence Laboratory (CSAIL) at the Massachusetts Institute of Technology. Her research focuses on decision-making under uncertainty, learning, and sensing with applications to robotics.
She is also the founder and co-editor-in-chief of the Journal of Machine Learning Research, an active open-access academic collection which encourages the development of open source software for the field of machine learning. Cool stuff! lpk was my professor when i was an undergraduate (before she went to MIT), and she opened my mind to a lot of great ideas about artificial intelligence, machine learning, robotics, and computation in general. Beyond the specifics that she taught, though, she also demonstrated what it means to be a spirited and engaged academic, both in research and in teaching. I think i first read Stanslaw Lem in one of her robotics classes (it was one of the stories from The Cyberiad), where she encouraged us to think not just about the technical possibilities of machinery, but also about the social possibilities. And it wasn't just theorizing: I also had the chance to do significant work with robotics hardware (both real and emulated) in a lab under her friendly and open supervision. She developed a new introductory series of two classes for the school's Computer Science department, which i was unfortunately too late to take. They're designed to reach students who were discouraged by the traditional programming-heavy approach, and focus instead on the issues of theory, abstraction, and collaboration, with programming taking a subordinate place to understanding the nature of computation itself. Later in my scholastic career, I had the privilege of helping her out as a teaching assistant, working with her on various robotics projects, advising a handful of younger students with her, and having her advise my final-year project, where i tested and evaluated variants of the boosting machine-learning algorithm. Throughout all of this, and despite my typical collegiate distractions, Leslie helped me understand technical details and nuance about the possibilities that we have with these incredible machines, and how they might be made to interact with the real world with all of its contradictory input and uncertainty. So here's to Leslie Pack Kaelbling: thank you for all your amazing contributions to our field, and for the things you taught me and the opportunities you shared!Tags: adalovelaceday09

Cyril Brulebois: Things I love

A couple of things I loved in the past few days: : Unfortunately, I can't seem able to find a related webpage. Its member: Inge Hager (violin), Elke Hager (cello), and Enrico Pompili (piano).

22 March 2009

Rob Taylor: Codethink anniversary

Last month, Codethink turned 2 years old, to coincide we had a company-wide hackfest in Brussels straight after FOSDEM. It really brought home how much it d grown. Just a year ago there was just me and Mark. Now we are 8! So i m gonna take a moment to shout out to the great job these guys are doing. In order of appearence . Mark Doffman
Mark has been consistently working on transforming AT-SPI (the GNOME Accessibility and UI automation technology) into a D-Bus based cross-desktop project. From nothing a year ago, working with Mike Gorse from Novell, he s now got something that more or less works and it s getting a lot of focus and testing as we plan for GNOME 3.0. I m looking forward to a brave new future for cross toolkit accessibility on the Linux desktop. John Carr
John came to us from working on Conduit, and he s continued on his dream of making all the devices and web services of the world talk seamlessly to each other. John s been in charge of pushing the Wizbit project along and now the core of this is pretty stable, he is going to be turning his eye to working on using Wizbit for synchronisation. J rg Billeter
A stunning hacker who needs no introduction! J rg s been continuing to hack on Vala and it s getting more and more mature each week. His main work, however has been on the stunning tracker-vstore, a branch of tracker that brings full RDF capabilities, which Philip van Hoof has been blogging about. I ll be writing a biggish blog post on this soon and what RDF could really mean for the desktop experience. Karl Lattimer
Karl came to us from working on UI at Nokia. At Codethink he s continued to focus on graphics technology and user experience, bringing his keen eye to Wizbit, amongst other things. One of the most impressive things I ve seen for a while was Karl s kinetic scrolling widget for the Wizbit timeline view. I m hoping we can turn this into something more general in the future. Ryan Lortie
Another hacker who needs no introduction, Ryan s our low-level infrastructure guy. He s been working on GNIO, a library to do network operations using GIO stream abstractions and continuing to hack on DConf, which is gradually coming together. He s also been helping J rg out with Vala. Mukund Sivaraman
I knew Muks from his GIMP work, and his work on Herb is really impressive. Muks is mostly working on top secret stuff that we can t talk about, unfortunately! Peter Charlton
Our documentation guy. This guy knows how to write and knows how to take something vague and unintelligable and turn it into crystal clarity. Invaluable! Of course, through all this I ve been here, dancing about architecture and generally doing the best I can to hold the whole shebang together. I ve been mostly working on some top secret stuff which hopefully will be getting opened up soon. I ll dance about that then

21 March 2009

Meike Reichle: Chemnitz Linux Days 2009

This is my fourth CLT report and they surely are becoming repetitive! But every good event deserves a good report, so here goes: Contrary to previous years we already arrived on Friday this year. Since we were also not up for another stint with the gym (Aaah the sissifying effects of marriage!) we booked a room with a nice little guesthouse about 15 bus-minutes from the venue. The conference was great as ever: I usually visit the CLT mostly for networking but I also managed to see a few talks. I moderated the session on High Performance and got to see three very nice talks on GPU programing, cluster computing and parallel programming. This year also featured the first talk on (the lack of) women in Free Software. Unusually enough this one was even given by a man. I was quite curious about that talk, which was (how fitting!) scheduled in direct succession to my own. I am kind of torn on that topic myself. On one hand women in Free Software is a topic that is very important to me and deserves any attention it can get. On the other hand instead of talking about how evenly capable women are in Free Software I'd rather just demonstrate it, for instance by giving a good Free Software related talk. And instead of talking about how there should be more women in Free Software I'd rather just be one and try to encourage others with my example. The usual quota of female speakers at linux events is somewhere between 2-4% and I just don't like the idea of having a conference with n men talking about Free Software and 1 woman talking about women. I've had these settings before and it just doesn't sit well with me. Because of this I was quite please to find the topic being tackled by a man this time.
The talk itself, in my impression, left a few open issues though. It was rather brief and focussed mainly on stating the usual numbers, asserting that the Women in FLOSS movement wasn't about affirmative action or discriminating men and explaining how women feel discriminated by sexist behaviour and advertising and how objection to such things should not be mistaken as prudery. It's general advice on how to improve the quota of women in FLOSS mostly boiled down to the linuxchix slogan "Be Polite. Be Helpful.". What I missed most was practical advice to projects wishing to attract more female contributors, such as mentoring programs or low-threshold entry points. Also I felt that the talk lacked a real motivation beyond "gender balance is a Good Thing". However, I was glad the topic finally found its way into the CLT as well and I had a couple of very interesting discussions afterwards. Concerning my own talk I was rather satisfied as well. Attendance was - as usually in Chemnitz - very good, according to the organisers I got around 200 people. Since I designed the talk as a collaborative project and its feedback was predominantly good I'll continue developing it with the feedback I got and submit it again to other events. I think the topic is very worthwile and there's still a lot in it. Some people asked for a more collaborative way of contributing their thoughts and ideas so I'll just create a wiki to collect the new ideas. I'll of course announce it here as soon as it's in place. The slides and audio recording (both German) are as usually available from the talks page at CLT or my own talks section.

19 March 2009

Steve Kemp: Somebody's coming up. Somebody serious

About a year ago I was looking for a "support system", to allow people to report issues and then allow myself and my partner to handle them. I looked at several packages and eventually decided upon roundup. (Minimal dependencies, included in Debian Stable, Simple to setup.) Over time my annoyances with this package have grown, so its definitely time to look again at the support landscape. For the moment I decided to give in to pressure and try RT. Unfortunately this was very quickly the result:
[Tue Mar 17 18:54:23 2009] [crit]: <rt-3.6.7-5189-1237316062-367.1-3-0..>
Could not send mail: Couldn't run /usr/sbin/sendmail:
  Cannot allocate memory at
    /usr/share/request-tracker3.6/lib/RT/Action/SendEmail.pm line 334.
Stack:
  [/usr/share/request-tracker3.6/lib/RT/Action/SendEmail.pm:334]
  [/usr/share/request-tracker3.6/lib/RT/Action/SendEmail.pm:288]
  [/usr/share/request-tracker3.6/lib/RT/Action/SendEmail.pm:107]
How much memory does the system have? 400Mb "real" and 256Mb "swap". I've bumpted it up to 512Mb + 376Mb respectively. Lets see how that helps. I'm reminded, once more, that in theory a support system is a small piece of software. In terms of my RT install all I did was install it, configure it such that when a new user submits a ticket they get back an autoreply with login details and can view/edit/close their own ticket via the web interface. This wiki page helped.
This is a step up from roundup which has a weird idea of security - if you allow a ticket submitter to use the UI they can see all open/closed tickets.
Anyway I'll keep testing it for a couple of days and if the memory helps then I guess its a small price to pay but .. ugh. Maybe the home made solution is the more practical solution.. ObFilm: Leon

Uwe Hermann: Using Debian GNU/Linux on the Lenovo IdeaPad S9e netbook

TuxMobil - Linux on Laptops, Notebooks, PDAs and Mobile Phones Lenovo Ideapad S9e netbook I recently got my hands on a Lenovo IdeaPad S9e netbook for a short amount of time (I don't own it), so I did a few tests with Debian unstable (more or less Lenny right now) and a Linux 2.6.28 kernel on it, see results below. The machine type is 4187-42G, and it features an Intel Atom N270 CPU (with HyperThreading) at 1.6 GHz, 1 GB of DDR2 RAM, an 80 GB SATA drive, an 8.9" WSVGA 1024x600 (glossy) screen, VGA port, LAN, wifi, bluetooth, 2xUSB, SD card slot, PCI ExpressCard slot, built-in microphone, and a webcam. BIOS You can enter the BIOS by pressing F2, the boot menu by pressing F12 during boot. Booting from USB works fine on this netbook. There's a Splashtop installation on the netbook (called "Lenovo Quickstart" here) which you can disable in the BIOS. Installation There's no CD-ROM drive, so the simplest way is to use a USB thumb drive for installation. Here's how you can prepare one containing a Lenny installer (assuming your USB thumb drive is /dev/sda):
  $ wget http://cdimage.debian.org/debian-cd/5.0.0/i386/iso-cd/debian-500-i386-netinst.iso
  $ wget http://ftp.nl.debian.org/debian/dists/lenny/main/installer-i386/current/images/hd-media/boot.img.gz
  $ gunzip boot.img.gz
  $ dd if=boot.img of=/dev/sda1
  $ mount -t vfat /dev/sda1 /mnt
  $ cp debian-500-i386-netinst.iso /mnt
  $ umount /mnt
If the above USB thumb drive doesn't boot correctly (which it did not in my case: GRUB error 17) it's probably because of a messed-up MBR. This is how you can fix it:
  $ apt-get install mbr
  $ install-mbr /dev/sda
 Lenovo Ideapad S9e Debian installation Then insert the USB thumb drive in the Lenovo IdeaPad S9e, choose USB boot in the BIOS, and start the installer. Most of the process works as usual, the only small difference is that you might want to load the "parted" installer module in order to resize the Windows-partition on the disk (if you want to keep it) to make space for Linux. The second (fat32) partition seems to keep a restore image and/or the Splashtop stuff, not sure. Audio Works out of the box using the snd_hda_intel driver. The hardware is onboard audio in the southbridge (82801G / ICH7) and uses the Realtek ALC269 codec. If some programs don't have working audio, try modprobe snd-pcm-oss. Built-in microphone Untested so far. Bluetooth Works out of the box using the bluetooth and btusb driver. The laptop's Bluetooth device is USB-attached internally and shows up in lsusb as:
  $ lsusb
  Bus 003 Device 002: ID 0a5c:2150 Broadcom Corp.
  $ dmesg
  usb 3-2: Product: BCM2046 Bluetooth Device
After modprobe btusb you can use hcitool / hciconfig etc. as usual, and/or enable more related stuff with /etc/init.d/bluetooth start. Sensors The lm-sensors script detects the lm75, eeprom, i2c-dev, and i2c_i801 modules. The following is the 'sensors' output:
  $ sensors
  acpitz-virtual-0
  Adapter: Virtual device
  temp1:       +36.0  C  (crit = +95.0  C)    
The hard drive temperature can be viewed with:
  $ hddtemp /dev/sda
  /dev/sda: FUJITSU MHZ2080BH G1: 44  C
HPET The Intel ICH7 southbridge in this laptop supports High Performance Event Timers (HPET) which allows for more power savings and thus improved battery life.
  $ dmesg   grep -i hpet
  ACPI: HPET 3F6E1E41, 0038 (r1 INTEL  CALISTGA  6040000 LOHR       5A)
  ACPI: HPET id: 0x8086a201 base: 0xfed00000
  hpet clockevent registered
  HPET: 3 timers in total, 0 timers will be used for per-cpu timer
  hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
  hpet0: 3 comparators, 64-bit 14.318180 MHz counter
You can check the wakeups-per-second with powertop. SD card slot Works out of the box. It seems to be attached via USB internally (usb-storage driver).
  $ lsusb
  Bus 001 Device 004: ID 0bda:0158 Realtek Semiconductor Corp. Mass Stroage Device
PCI ExpressCard slot Untested so far. ACPI Works fine, see comments for "acpitool" output. Network card Works out of the box using the tg3 driver.
  $ modprobe tg3
  tg3.c:v3.94 (August 14, 2008)
  tg3 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
  tg3 0000:02:00.0: setting latency timer to 64
  eth0: Tigon3 [partno(BCM95906) rev c002 PHY(5906)] (PCI Express) 10/100Base-TX Ethernet 00:11:22:33:44:55
  eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] WireSpeed[0] TSOcap[0]
  eth0: dma_rwctrl[76180000] dma_mask[64-bit]
Touchpad Works out of the box, both in X as well as in the console using gpm.
  $ dmesg
  Synaptics Touchpad, model: 1, fw: 7.2, id: 0x1c0b1, caps: 0xd04731/0xa40000
Suspend-to-disk and suspend-to-RAM I'm using the hibernate Debian package. You can explicitly force the usage of either method in /etc/hibernate/hibernate.conf by uncommenting the respective lines.
  TryMethod disk.conf
  # TryMethod ram.conf
Suspend does not yet work out of the box, however, as the machine is unknown:
  $ s2ram -n
  Machine unknown
  This machine can be identified by:
      sys_vendor   = "LENOVO                          "
      sys_product  = "418742G         "
      sys_version  = "Lenovo                  "
      bios_version = "14CN51WW  "
  See http://suspend.sf.net/s2ram-support.html for details.
After a few test I found that s2ram -f -a 3 works fine (tested from console only so far). Now this needs to be integrated upstream and in the Debian package (I'll file a bug report). Update: Submitted bug #520848, and an email to the upstream mailing list. Wireless There doesn't seem to be a mainline driver for the Broadcom BCM4312 wifi card in the laptop, yet:
  $ lspci -nn
  05:00.0 Network controller [0280]: Broadcom Corporation BCM4312 802.11b/g [14e4:4315] (rev 01)
Neither the b43 nor the b43legacy drivers work as of 2.6.28. For now, one of two possible options is to build a (partly non-free) driver provided by Broadcom from source (option 2 would be to use ndiswrapper, I guess, but that's untested):
  $ wget http://people.debian.org/~adamm/kernel/linux-kbuild-2.6.28_2.6.28-0.1_i386.deb
  $ dpkg -i linux-kbuild-2.6.28_2.6.28-0.1_i386.deb (currently needed in unstable due to bug #518115)
  $ apt-get install build-essential linux-headers-2.6.28-1-686
  $ mkdir temp; cd temp
  $ wget http://www.broadcom.com/docs/linux_sta/hybrid-portsrc-x86_32-v5_10_79_10.tar.gz
  $ tar xfvz hybrid-portsrc-x86_32-v5_10_79_10.tar.gz
  $ make -C /lib/modules/ uname -r /build M= pwd  clean
  $ make -C /lib/modules/ uname -r /build M= pwd  modules
If that worked, you can load the driver via:
  $ rmmod bcm43xx; rmmod b43; rmmod b43legacy (you could also permanently blacklist these modules)
  $ modprobe ieee80211_crypt_tkip
  $ insmod ./wl.ko
  $ dmesg
  wl: module license '' taints kernel.
  wl 0000:05:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
  wl 0000:05:00.0: setting latency timer to 64
  eth1: Broadcom BCM4315 802.11 Wireless Controller 5.10.79.10
You can now run iwconfig, iwlist, etc. from the command line, or use some GUIs such as kwifimanager. In order to disable wireless, run:
  $ rmmod wl
So far, I only tested WEP (but not WPA). CPU frequency scaling Works out of the box using the acpi_cpufreq driver. Use cpufreq-set -c 0 -g performance if you need full CPU power, cpufreq-set -c 0 -g powersave otherwise. Use -c 1 to do the same with the other CPU/core. PC speaker Works fine out of the box using the pcspkr module, tested with beep. Graphics card Works out of the box using the intel X.org driver.
  $ xrandr
  Screen 0: minimum 320 x 200, current 1024 x 600, maximum 1024 x 1024
  VGA disconnected (normal left inverted right x axis y axis)
  LVDS connected 1024x600+0+0 (normal left inverted right x axis y axis) 195mm x 113mm
     1024x600      60.0*+
     800x600        60.3  
     640x480        59.9  
  TV disconnected (normal left inverted right x axis y axis)
DRI works out of the box with the (mainline, open-source) driver:
  $ glxinfo   grep direct
  direct rendering: Yes
If you attach an external monitor or projector, you can enable it using xrandr as usual:
  $ xrandr --output VGA --auto
You can also use a dual-head setup by adding this to your "Screen" section in /etc/X11/xorg.conf:
  SubSection "Display"
    Virtual 2048 2048
  EndSubSection
After restarting the X server, you can play with xrandr and move the external screen (VGA) "below" the laptop's LCD screen (LVDS) for a simple dual-head setup. The GUI tools arandr or grandr are probably a bit simpler to use than plain command line xrandr. USB Works fine, of course. The only small problem is that there are only two USB ports, more would have been better. Disk drive Works fine, it's an 80 GB SATA drive. Webcam Works out of the box using the uvcvideo driver.
  $ lsusb
  Bus 001 Device 005: ID 5986:0141 Acer, Inc
  $ modprobe uvcvideo
  uvcvideo: Found UVC 1.00 device Lenovo EasyCamera (5986:0141)
  input: Lenovo EasyCamera as /devices/pci0000:00/0000:00:1d.7/usb1/1-3/1-3:1.0/input/input9
  usb 1-3: New USB device found, idVendor=5986, idProduct=0141
  usb 1-3: New USB device strings: Mfr=3, Product=1, SerialNumber=0
  usb 1-3: Product: Lenovo EasyCamera
  usb 1-3: Manufacturer: BISON Corporation
You can use luvcvideo for webcam viewing. Battery Lasts for ca. 3.5 hours, probably less if the system is under high load. Special keys Fn+CursorUp / Fn+CursorDown (brightness), Fn+ESC (enable/disable webcam), Fn+F1 (sleep mode), Fn+F2 (enable/disable TFT backlight), Fn+F6 (enable/disable thouchpad), Fn+F7 (Num lock), Fn+F8 (scroll lock), and Fn+F11 (F12 key) all work fine. Fn+F3, Fn+F5, Fn+F9, Fn+F10, and all other special keys are untested. LEDs The power, disk activity, CAPS lock, Num lock, and battery charging LEDs all work fine out of the box. lspci -tvnn
  -[0000:00]-+-00.0  Intel Corporation Mobile 945GME Express Memory Controller Hub [8086:27ac]
           +-02.0  Intel Corporation Mobile 945GME Express Integrated Graphics Controller [8086:27ae]
           +-02.1  Intel Corporation Mobile 945GM/GMS/GME, 943/940GML Express Integrated Graphics Controller [8086:27a6]
           +-1b.0  Intel Corporation 82801G (ICH7 Family) High Definition Audio Controller [8086:27d8]
           +-1c.0-[0000:02]----00.0  Broadcom Corporation NetLink BCM5906M Fast Ethernet PCI Express [14e4:1713]
           +-1c.1-[0000:03-04]--
           +-1c.2-[0000:05]----00.0  Broadcom Corporation BCM4312 802.11b/g [14e4:4315]
           +-1d.0  Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #1 [8086:27c8]
           +-1d.1  Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #2 [8086:27c9]
           +-1d.2  Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #3 [8086:27ca]
           +-1d.3  Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #4 [8086:27cb]
           +-1d.7  Intel Corporation 82801G (ICH7 Family) USB2 EHCI Controller [8086:27cc]
           +-1e.0-[0000:06]--
           +-1f.0  Intel Corporation 82801GBM (ICH7-M) LPC Interface Bridge [8086:27b9]
           +-1f.1  Intel Corporation 82801G (ICH7 Family) IDE Controller [8086:27df]
           +-1f.2  Intel Corporation 82801GBM/GHM (ICH7 Family) SATA IDE Controller [8086:27c4]
           \-1f.3  Intel Corporation 82801G (ICH7 Family) SMBus Controller [8086:27da]
cat /proc/cpuinfo See comments. Resources All in all it's a really nice hardware, and it works (more or less) flawlessly without much hassle with recent distros/kernels. Update 2009-03-22: Updated various sections, added more info. Added resources section.

Michael Banck: 19 Mar 2009

My experience with LapStore's used-ThinkPad warranty repair service My ThinkPad T40 arrived back from warranty repair today (well, actually yesterday, but I had to run off to the Gnome-2.26 release get-together in Munich so I did not have time to open the box then). I bought it used roughly two years ago at LapStore when my R51 had died. I had bought the R51 new with a one year IBM warranty but unfortunately within two years the graphics chip with got damaged and would freeze the notebook after a couple of minutes. I then decided that I do not really need a new notebook anymore, and opted for a used T4x series (there were no used X40s available at that time). I chose LapStore because they offered a one-year "Garantie" (guarantee? warranty?), which was rather unusual for used notebooks - at best, you would get a one-year "Gewaehrleistung" which is the promise to fix things which were supposedly broken already by delivery. Even better, one could optionally extend the warranty to two years, which I did. The notebook they sent was in pretty good condition (apparently a business out-of-warranty return) and I put in my R51's hard disk, the ipw2100 WLAN card and the RAM (unfortunately, I realized too late that the keyboard and the CDRW/DVD drive do not fit). I was pretty happy with it (and LapStore in general, I recommended it to a couple of friends since, and e.g. my current flatmate bought a T42 there a while ago as well) until the fan started dropping out and making weird noises by the end of 2008. So just before the end of warranty, I sent it (after removing hard disk, optical drive and battery) in to LapStore to see how their service is. I also mentioned a clear bright spot on the display (apparently some fatigue, you see it often mentioned in ThinkPad eBay descriptions) and a crack in the palmrest between the cursor-right key and the hard disk slot. When they sent a mail that the ThinkPad had arrived at their site, I also followed-up via mail that the "indestructible" keyboard caps stickers they used to mod a Scandinavian(?) keyboard into a German one were pretty much destroyed by now and would also need servicing. I assumed that they would service the fan (which looked like a clear-cut warranty issue to me) without arguing, but probably not the display and palmrest (and did not know whether they got the mail about the keyboard stickers), so when they sent another mail two days ago that they sent the notebook back without asking further questions, I became worried about what happened at all. So, long story short, I was totally positively surprised when I opened the box today and read: "Aktion: L fter, Display, Palmrest und Tastatur getauscht" (action: fan, display, palmrest and keyboard replaced) The replacements are still used parts (and the keyboard is still not a real German one, but one with new stickers on it), but they basically changed my almost-totally-broken-will-fall-apart T40 back into a almost-as-good-as-new T40. (not sure whether that is positive or negative, but they also forgot to remove the service-hard disk they put in to test things, I guess I will send it back to them) So all in all, I am very much impressed by their service. I would have expected this kind of service from IBM/Lenovo if I had a manufacturer warranty, but not from some random sell-used-ThinkPads shops on the net. I can now even more strongly recommend LapStore as the place to buy good notebooks. Certainly you can get cheaper prices at some eBay stores, but you do not get real warranty then and what about the service? I recommend geting a T42 - I believe the T43 is inferior to it and the T40s and T41s don't have "LapStore Garantie" anymore. You can get them without operating system and can customize the hard disk, memory and optical drives - unfortunately you cannot downgrade those, which is my only gripe with them.

18 March 2009

Biella Coleman: Hyperparenting: The Ethnography

I have a long list (it exists only in my head unfortunately) of various ethnographic projects I would like to conduct. I might start the real list soon but in the meantime, I will just through the half-baked ideas here and perhaps it will inspire others to take it on. I am currently reading a fantastic book, The Case Against Perfection, and am on a section on hyperparenting and well, while I think at a common sense level we know what psychological havoc such controlled parenting can cause, I think an ethnography of some of the practices and institutions of hyperparenting would, nonetheless, make for a fascinating read. If anyone knows of anything that even resembles this, do drop me a line. Update: Not exactly an ethnography but certainly fun to read.

Biella Coleman: Hyperparenting: The Ethnography

I have a long list (it exists only in my head unfortunately) of various ethnographic projects I would like to conduct. I might start the real list soon but in the meantime, I will just through the half-baked ideas here and perhaps it will inspire others to take it on. I am currently reading a fantastic book, The Case Against Perfection, and am on a section on hyperparenting and well, while I think at a common sense level we know what psychological havoc such controlled parenting can cause, I think an ethnography of some of the practices and institutions of hyperparenting would, nonetheless, make for a fascinating read. If anyone knows of anything that even resembles this, do drop me a line. Update: Not exactly an ethnography but certainly fun to read.

Russell Coker: Hyperthermia and Children in Cars

Bruce Schneier writes about the risks involving children abandoned in cars and cites an article about the tragic deaths of children in hot cars [1]. One unfortunate error that he made was to not cite the following from the end of the last page of the Washington post article he cited [2]:
In hyperthermia cases, he believes, the parents are demonized for much the same reasons. We are vulnerable, but we don t want to be reminded of that. We want to believe that the world is understandable and controllable and unthreatening, that if we follow the rules, we ll be okay. So, when this kind of thing happens to other people, we need to put them in a different category from us. We don t want to resemble them, and the fact that we might is too terrifying to deal with. So, they have to be monsters. I believe that similar thought processes are used in relation to many other situations, and that such thought processes prevent people from taking appropriate actions to minimise the risk. If someone considers that forgetting a child in the back seat to be an accident that could happen to anyone then they would be inclined to take action to minimise the risk (such as spending some money on a sensor). If however they consider such forgetfulness to be proof of being a bad parent , then as they are a good parent they would have to avoid buying a monitor. I m surprised that Bruce didn t draw an analogy between this and the forgetful losses of laptops and guns by people who work for law enforcement agencies (which he has written about before). I wonder how expensive it would be to make a sensor for heart-rate, breathing, and temperature integrated with a GSM modem and a GPS? If it could be small enough to be attached to clothes then the child could wear it at all times. If such a sensor was to detect a sign of a problem it wouldn t matter whether the child was forgotten in a car, at day-care, or even being actively supervised. The data would be sent to the monitoring agency along with GPS data. The monitoring agency could then phone the parents. If the parents don t answer or don t know where the child is then the police could track down the GPS location. Probably most calls would be due to parents leaving a child too close to an air-conditioner or playing outside in the sun in summer which are unlikely to give a fatal result and a phone call would get a quick fix for what would only be a minor health problem. If the device was marketed as monitoring for sleep apnia then parents could buy it without admitting to the possibility that they might do anything wrong. The causes of SIDS are a topic of ongoing research and parents can admit to being worried about their children suffering from it without admitting any possibility that they might make a mistake.

16 March 2009

Theodore Ts'o: Don t fear the fsync!

After reading the comments on my earlier post, Delayed allocation and the zero-length file problem as well as some of the comments on the Slashdot story as well as the Ubuntu bug, it s become very clear to me that there are a lot of myths and misplaced concerns about fsync() and how best to use it. I thought it would be appropriate to correct as many of these misunderstandings about fsync() in one comprehensive blog posting. As the Eat My Data presentation points out very clearly, the only safe way according that POSIX allows for requesting data written to a particular file descriptor be safely stored on stable storage is via the fsync() call. Linux s close(2) man page makes this point very clearly:
A successful close does not guarantee that the data has been successfully saved to disk, as the kernel defers writes. It is not common for a file system to flush the buffers when the stream is closed. If you need to be sure that the data is physically stored use fsync(2).
Why don t application programmers follow these sage words? These three reasons are most often given as excuses:
  1. (Perceived) performance problems with fsync()
  2. The application only needs atomicity, but not durability
  3. The fsync() causing the hard drive to spin up unnecessarily in laptop_mode
Let s examine each of these excuses one at a time, to see how valid they really are. (Perceived) performance problems with fsync() Most of the bad publicity with fsync() originated with the now infamous problem with Firefox 3.0 that showed up about a year ago in May, 2008. What happened with Firefox 3.0 was that the primary user interface thread called the sqllite library each time the user clicked on a link to go to a new page. The sqllite library called fsync(), which in ext3 s data=ordered mode, caused a large, visible latency which was visible to the user if there was a large file copy happening by another process. Nearly all of the reported delays was a few seconds, which would be expected; normally there isn t that much dirty data that needs to be flushed out on a Linux system, even if it is even very busy. For example, consier the example of a laptop downloading an .iso image from a local file server; if the laptop has the exclusive link of a 100 megabit/second ethernet link, and the server has the .iso file in cache, or has a nice fast RAID array so it is not the bottleneck, then in the best case, the laptop will be able to download data at the rate of 10-12 MB/second. Assuming the default 5 second commit interval, that means that in the worst case, there will be at most 60 megabytes which must be written out before the commit can proceed. A reasonably modern 7200 rpm laptop drive can write between 60 and 70 MB/second. (The Seagate Momentus 7200.4 laptop drive is reported to be able to deliver 85-104 MB/second, but I can t find it for sale anywhere for love or money.) In this example, an fsync() will trigger a commit and might need to take a second while the download is going on; perhaps half a second if you have a really fast 7200 rpm drive, and maybe 2-3 seconds if you have a slow 5400 rpm drive. (Jump to Sidebar: What about those 30 second fsync reports?) Obviously, you can create workloads that aren t bottlenecked on the maximum ethernet download speed, or the speed of reading from a local disk drive; for example, dd if=/dev/zero of=big-zero-file will create a very large number of dirty pages that must be written to the hard drive at the next commit or fsync() call. It s important to remember though, fsync() doesn t create any extra I/O (although it may remove some optimization opportunities to avoid double writes); fsync() just pushes around when the I/O gets done, and whether it gets done synchronously or asynchronously. If you create a large number of pages that need to be flushed to disk, sooner or later it will have a significant and unfortunate effect on your system s performance. Fsync() might make things more visible, but if the fsync() is done off the main UI thread, the fact that fsync() triggers a commit won t actually disturb other processes doing normal I/O; in ext3 and ext4, we start a new transaction to take care of new file system operations while the committing transction completes. The final observation I ll make is that part of the problem is that Firefox as an application wants to make a huge number of updates to state files and was concerned about not losing that information even in the face of a crash. Every application writer should be asking themselves whether this sort of thing is really necessary. For example, doing some quick measurements using ext4, I determined that Firefox was responsible for 2.54 megabytes written to the disk for each web page visited by the user (and this doesn t include writes to the Firefox cache; I symlinked the cache directory to a tmpfs directory mounted on /tmp to reduce the write load to my SSD). So these 2.54 megabytes is just for Firefox s cookie cache and Places database to maintain its Awesome bar . Is that really worth it? If you visit 400 web pages in a day, that s 1GB of writes to your SSD, and if you write more than 20GB/day, the Intel SSD will enable its write endurance management feature which slows down the performance of the drive. In light of that, exactly how important is it to update those darned sqllite databases after every web click? What if Firefox saved a list of URL s that has been visited, and only updated every 30 or 60 minutes, instead? Is it really that every last web page that you browse be saved if the system crashes? An fsync() call every 15, 30, or 60 minutes, done by a thread which doesn t block the application s UI, would have never been noticed and would have not started the firestorm on Firefox s bugzilla #421482. Very often, after a little thinking, a small change in the application is all that s necessary for to really optimize the application s fsync() usage. (Skip over the sidebar if you ve already read it). Sidebar: What about those 30 second fsync reports? If you read through the Firefox s bugzilla entry, you ll find reports of fsync delays of 30 seconds or more. That tale has grown in the retelling, and I ve seen some hyperbolic claims of five minute delays. Where did that come from? Well, if you look that those claims, you ll find they were using a very read-heavy workload, and/or they were using the ionice command to set a real-time I/O priority. For example, something like ionice -c 1 -n 0 tar cvf /dev/null big-directory . This will cause some significant delays, first of all because ionice -c 1 causes the process to have a real-time I/O priority, such that any I/O requests issued by that process will be serviced before all others. Secondly, even without the real-time I/O priority, the I/O scheduler naturally prioritizes reads as higher priority than writes because normally processes are waiting for reads to complete, but writes are normally asynchronous. This is not at all realistic workload, and it is even more laughable that some people thought this might be an accurate representation of the I/O workload of a kernel compile. These folks had never tried the experiment, or measured how much I/O goes on during a kernel compile. If you try it, you ll find that a kernel compile sucks up a lot of CPU, and doesn t actually do that much I/O. (In fact, that s why an SSD only speeds up a kernel compile by about 20% or so, and that s in a completely cold cache case. If the commonly used include files are already in the system s page cache, the performance improvement of the SSD is much less.) Jump back to reading Performance problems with fsync. The atomicity not durability argument One argument that has commonly been made on the various comment streams is that when replacing a file by writing a new file and the renaming file.new to file , most applications don t need a guarantee that new contents of the file are committed to stable store at a certain point in time; only that either the new or the old contents of the file will be present on the disk. So the argument is essentially that the sequence:
  • fd = open( foo.new , O_WRONLY);
  • write(fd, buf, bufsize);
  • fsync(fd);
  • close(fd);
  • rename( foo.new , foo );
is too expensive, since it provides atomicity and durability , when in fact all the application needed was atomicity (i.e., either the new or the old contents of foo should be present after a crash), but not durability (i.e., the application doesn t need to need the new version of foo now, but rather at some intermediate time in the future when it s convenient for the OS). This argument is flawed for two reasons. First of all, the squence above exactly provides desired atomicity without durability . It doesn t guarantee which version of the file will appear in the event of an unexpected crash; if the application needs a guarantee that the new version of the file will be present after a crash, it s necessary to fsync the containing directory. Secondly, as we discussed above, fsync() really isn t that expensive, even in the case of ext3 and data=ordered; remember, fsync() doesn t create extra I/O s, although it may introduce latency as the application waits for some of the pending I/O s to complete. If the application doesn t care about exactly when the new contents of the file will be committed to stable store, the simplest thing to do is to execute the above sequence (open-write-fsync-close-rename) in a separate, asynchronous thread. And if the complaint is that this is too complicated, it s not hard to put this in a library. For example, there is currently discussion on the gtk-devel-list on adding the fsync() call to g_file_set_contents(). Maybe if someone asks nicely, the glib developers will add an asynchronous version of this function which runs g_file_set_contents() in a separate thread. Voila! Avoiding hard drive spin-ups with laptop_mode Finally, as Nathaniel Smith said in Comment #111 of of my previous post:
The problem is that I don t, really, want to turn off fsync s, because I like my data. What I want to do is to spin up the drive as little as possible while maintaining data consistency. Really what I want is a knob that says I m willing to lose up to minutes of work, but no more . We even have that knob (laptop mode and all that), but it only works in simple cases.
This is a reasonable concern, and the way to fix this is to enhance laptop_mode in the Linux kernel. Bart Samwel, the author and maintainer of laptop_mode, actually discussed this idea with me last month at FOSDEM. Laptop_mode already adjusts /proc/sys/vm/dirty_expire_centisecs and /proc/sys/vm/dirty_writeback_centisecs based on the configuration parameter MAX_LOST_WORK_SECONDS, and it also adjusts the file system commit time (for ext3; it needs to be taught to do the same thing for ext4, which is a simple patch) to MAX_LOST_WORK_SECONDS as well. All that is necessary is a kernel patch to allow laptop_mode to disable fsync() calls, since the kernel knows that it is in laptop_mode, and it notices that the disk has spun up, it will sync out everything to disk, since once the energy has been spent to spin up the hard drive, we might as well write everything in memory that needs to be written out right away. Hence, a patch which allows fsync() calls to be disabled while in laptop_mode should do pretty much everything Nate has asked. I need to check to see if laptop_mode does this already, but if it doesn t force a file system commit when it detects that the hard drive has been spun up, it should obviously do this as well. (In addition to having a way to globally disable fsync() s, it may also be useful to have a way to selectively disable fsync() s on a per-process basis, or on the flip side, exempt some process from a global fsync-disable flag. This may be useful if there are some system daemons that really do want to wake up the hard drive and once the hard drive is spinning, naturally everything else that needs to pushed out to stable store should be immediately written.) With this relatively minor change to the kernel s support of laptop_mode, it should be possible to achieve the result that Nate desires, without needing force applications to worry about this issue; applications should be able to just simply use fsync() without fear. Summary As we ve seen, the reasons most people think fsync() should be avoided really don t hold water. The fsync() call really is your friend, and it s really not the villain that some have made it out to be. If used intelligently, it can provide your application with a portable way of assuring that your data has been safely written to stable store, without causing a user-visible latency in your application. The problem is getting people to not fear fsync(), understand fsync(), and then learning the techniques to use fsync() optimally. So just as there has been a Don t fear the penguin campaign, maybe we also need to have a Don t fear the fsync() campaign. All we need is a friendly mascot and logo for a Don t fear the fsync() campaign. Anybody want to propose an image? We can make some T-shirts, mugs, bumper stickers Related posts (automatically generated):
  1. Delayed allocation and the zero-length file problem A recent Ubuntu bug has gotten slashdotted, and has started...

15 March 2009

Joerg Jaspert: Section changes, Content Files and some more

And there we have done it - new sections in the archive. Read more about it in my d-d-a post and the whole set of affected packages is also available - and updated slightly, as some small things slipped through. The whole process was less painful than feared. After some discussion, started a few weeks ago we finally went and coordinated the change today. There was no reliable and good way for changes that large, so we had to create our own way. (The last section changes had been done ages ago). Mike provided a little python script which was able to parse our input file with the section matches against our database, so we ended up with a list we could possibly feed back into the database. Except, not really. Thanks to various things the list generated needed some manual work (which I also managed to fuck up a little), but after that was done we were ready to go. And while Mark told the wanna-build admins about the new sections (for some reason our buildd network cares about sections) and prepared a patch for our policy, I prepared the mail for the d-d-a list. Then we all had a break, as a dinstall was still running, and we did not want to break that. The actual changes then took only a few seconds, sending the mail to d-d-a took longer, thankyou gnupg. :) I wonder how much afterwork this will generate. There are at least the policy changes, lintian, the packages control files to adjust. Also some websites, where Frank already changed packages.debian.org to know about the new names. (Before he fixed up me forgetting some PHP packages to move. This way all of the active ftpteam members did something in it :) ).
In other news, we are currently stress-testing our ftp-master host a little. Well, not really, it could take more, but right now we are scanning the full archive once, importing all filenames from all our Debian packages. The goal is to get projectb larger. Err, no, actually we do want to get rid of apt-ftparchive, as that is one of the worst tools in our dinstall runs. It is slow as hell in the mode it has to run for us, making up a large part of our runtime. Step 1 to get rid of it is to generate the content files on our own, and before we can do that we have to import them all Which means we have to unpack every binary package once, reading in all filenames to store them in our database. Thats a lot of data. Currently the associations table, which is nothing more than 4 integers per row, takes up 972MB, and import is still running. (The index on the binary pkg id in this table alone has 700MB). And this table does nothing more than linking the binary package to the filename and filepath. Which are two extra tables, with the filenames having only 87MB, the path entries only taking up 30MB. Yes, anyone using databases will tell me thats not much, and it really isn t that much. It s just that the largest table before this import had 120MB in it. And some of the others used more space for their indexes than for the actual data. :) Anyway, this is only step one on our way to get rid of the slowness apt-ftparchive is. When the contents all work as we want them, the next step will be to get rid of it generating Packages/Sources files. Which means we have some more data to store in projectb. And after that - we also intend to do a Contents file for our source. Each and every source file listed. Guess we really will end up with some more large tables. :) The final goal and main reason why we do this is to save us at least 60 minutes runtime per dinstall run. My own guess is that Contents alone will get us about 20 minutes down. No, before someone wants to yell at me - apt-ftparchive can t be changed as much to make this possible and keep using it. Not in the way the generate option works. It has the disadvantage that it scans the files after they are installed in their pool/ location and stores that in its own database . This approach has to negative points:
  • first, it is using another database (BerkeleyDB). We do have one database already, projectb, thank you.
  • second, and more important, the files only end up in pool/ on dinstall time. Which means the whole scanning is done during dinstall. Which eats time. The new way will surely use the same amount of cpu time in total - but it will spread it over the whole day. 4 runs each hour. Not 4 runs each day. And during dinstall it will boil down to one sql query per installed package moving data from a pending table into the real contents table. Which is a cheap operation.

The title says some more, so here is some more: We aren t only working on what I wrote above. There is also work on changing our database access. Historically dak used the pg module to do database work. While the module itself does work, the way its used isn t really the best one. Way too much groundwork everywhere. Not nearly as much of the nice things used people say python could have. So another largish work area is getting rid of pg. Migrating to psycopg2. And putting a proper OO class around it, which in the end hopefully hides those dark details from the application. And to achieve even more sanity, we (well, honestly, Mark) are looking into SQL Alchemy usage. (We also considered storm from Canonical, but that sucks when you take a look at its documentation. (Which documentation?). SQLA won there easily.) While I m ranting: WTF are people neglecting their docs that much? Ever tried looking at the documentation for psycopg2? Forget it. No, a db api compliance test suite is NOT documentation. If there would be any better python module to connect to postgres, I would veto the usage of psycopg2, but there doesn t seem to be one. Sucks.

Martin Michlmayr: SheevaPlug: the NSLU2 killer

SheevaPlug in my hand I received a SheevaPlug this week, an intriguing device that packs incredible power and functionality into a tiny package. As many of you know, I've been doing a lot of work on Debian for the Linksys NSLU2 in the last few years. The NSLU2 is a key reason why ARM has become the third most popular architecture in Debian (after 32 and 64 bit x86), and I believe a main reason is that the NSLU2 is so incredibly cheap. At a price under $100, most people don't think too long and simply buy a device and do something cool with it. The SheevaPlug is being offered at the same price range but offers considerably more. Riku Voipio asked the right question: "What would you do with something approximately 10x more powerful with same prize/size range?" I believe the SheevaPlug is a killer replacement for the NSLU2 and here's why:
  • Marvell Kirkwood 6281 CPU at 1.2 GHz with 256 KB L2 cache (compared to a 266 MHz CPU)
  • 512 MB RAM (compared to lousy 32 MB)
  • 512 MB flash, enough for a basic Debian installation (compared to 8 MB)
  • Gigabit Ethernet (compared to 100 Mb) and no proprietary microcode required
  • 1x USB and 1x SD (compared to 2x USB: the only advantage the NSLU2 has)
  • ... and finally: mini-USB with serial console and JTAG: while Debian on the NSLU2 works well and is stable, we regularly had users whose system would no longer boot and there was no good way to figure out why. The serial console on the SheevaPlug actually allows users to figure out what's going on.
I'm incredibly excited about the SheevaPlug and the first thing I did was to take the device apart and look at the inside. The results can be found in the SheevaPlug image gallery. My next project will be slightly more productive: porting Debian. As I see it, we should support the following three installation variants for the SheevaPlug:
  • To an external USB drive or stick.
  • To an SD card.
  • To the 512 MB flash memory on the device.
The first two should be relatively straight forward, but of course installing to the internal flash memory is particularly interesting given that 512 MB (plus compression) is enough for a basic installation of Debian. Unfortunately, installations to MTD flash are currently not supported in the Debian installer but I hope we can find a volunteer who wants to implement this functionality. My next steps are to put a kernel for the SheevaPlug into the archive and to get a basic installation going. From there we can look at more sophisticated installation options and other functionality.

13 March 2009

Julien Blache: Kodak scanner drivers for Linux - not there yet

So, Kodak announced the availability of Linux drivers for some of its scanners, together with a frontend application. While the effort is laudable, they re not quite there yet. Let s have a look. First of all, and contrary to what a few people believe after reading the press release, the drivers themselves aren t OpenSource, let alone Free Software. They re all proprietary; read better, the press release clearly mentions that only the frontend application is GPLv2. Now let s look at what Kodak offers in its drivers packages. I ve downloaded LinuxSoftware_i1200_v3.5.tar.gz from the Kodak website, these are the drivers for the Kodak i1200 series. The archive contains a set of RPM packages, one Debian package corresponding to one of the RPM packages, and a setup script. One of the packages contains the OpenUSB library, which is released under the LGPL; sources aren t there, I haven t asked for the sources, don t plan on doing so, and am not implying anything regarding license compliance whatsoever. I ll note that I haven t seen any written offer for the sources, though, be it in the archive (there s not even a README in there) or on the website. So, although Ubuntu is listed as supported, we can say that no Debian packages are provided. The setup script actually relies on alien to convert them if it thinks it s running on a Debian-based system, which is determined by the availability of dpkg in the PATH. Using alien is far from ideal. The packages ship libraries and components under /usr/lib, /usr/local/lib and even /opt, pretty randomly it seems. Let s repeat here that /usr/local and /opt are strictly reserved for the local administrator and packages must not install anything here. There are other oddities in the paths used by the software stack, like /var/kodak. It looks like the SANE backend is only a bridge to Kodak s TWAIN data source, so you d actually need the whole stack to use it. All in all, Kodak does no better, I d even venture to say it does worse, than Epson (Avasys). At least part of Epson s backend is Free Software, even if it relies on proprietary libraries for some scanners, though unfortunately most if not all of the recent ones. I also have to mention that Epson is offering both RPM and Debian packages, and by that I mean proper Debian packages. Kodak are far from that, even the RPM aren t that great (I think the RPM packages even lack dependency information, based on what I ve read in the setup script, though I haven t checked). What Kodak offers here is a GPL frontend for TWAIN that not many people will be using, while keeping the drivers proprietary. Unfortunately their SANE backend isn t a backend so to speak, and even if it was OpenSource it d still rely on the rest of the TWAIN software stack. So, in a nutshell, while I m very happy to see yet another vendor go down the road of Linux support for its products, I am immensely disappointed by the proprietary nature of the drivers and the low quality of the packages. I was, of course, investigating the drivers to possibly package them for Debian; as you ve probably understood, it s not possible, both technically and legally. And I kept a little gem from the setup script for the end:
# We need to create a link for the libdbus-1 requirement for openusb,
# but only on Fedora.
if [ -f /etc/fedora-release ]; then
    ln -s /lib/libdbus-1.so.3 /lib/libdbus-1.so.2 2>> /dev/null
fi
As we all know, sonames and soversions are useless and they just keep getting in the way. Duh.

11 March 2009

Steve Kemp: Oh my God, someone ate my entire pie!

Yesterday was my birthday, and it was full of cookies, pies, magical pixie dust and things made entirely of sugar and spice! The remainder of the day was spent re-installing Debian Lenny upon my EEE PC - Somehow I managed to completely screw the system. Because the EEE PC is one of those ultra-portable machines I mostly used it when I was travelling, or outdoors. That mean I was generally receiving poor connectivity and the system packages weren't up to date. While I was in bed I figured I'd dist-upgrade it to the recently released Lenny. Unfortunately I started the dist-upgrade inside X.org, once I realised this I figured I'd cancel the operation via Ctrl-c. Bad news everbody: I think I was unlucky enough to interrupt an upgrade of libc, or something equally critical. Every single application gave segfaults afterward. I had two open root terminals and I could navigate around via cd .., and "echo *", but all other commands such as sudo, dpkg, strace just gave segfaults. (Even static commands gave errors - so it might have been the dynamic loader that was borked, I admit I didn't look too closely.) I figured reinstalling would be a good solution since the machine has a 4Gb root partition and /home was stored on a separate 16Gb volume. Unfortunately I managed to misjudge the installer's partitioning step and nuke the partition table on the external volume so I ended up losing the whole system. Happily reinstallation was a breeze as my home network is setup to allow installation via PXE network booting (at some point I should document NFS-root PXE-booting). It took me longer to fiddle with the BIOS on the EEE PC to allow network booting than it did to complete a minimal install. Which I guess is good. I still need to restore my backup of /home/, but that can wait a few days. Right now I'm loathe to touch the machine at all - although I did distract myself by getting KVM to PXE boot:
# create 4gb disk image
dd if=/dev/zero of=/tmp/img.img bs=1024 count=4096k
# launch KVM
sudo kvm -no-acpi
  -boot n -tftp /var/lib/tftpboot/ -bootp /pxelinux.0
  -hda /tmp/img.img
  -net nic,macaddr=00:0E:35:be:de:ad  -net user
It seems that KVM wants to have access to the local TFTP root directory so I just pointed it at that. Since my desktop machine is also my TFTP + DHCP host that works out nicely. (A quick scan of the manual suggests that QEMU/KVM has funky built-in TFTP code, so it doesn't actually forward TFTP requests over the network.) DHCP requests were certainly passed around as expected though and were answered via my local dnsmasq installation. I did see errors at every DHCP request in syslog, but they seemed harmleess enough:
gold dnsmasq[29241]: no address range available for DHCP request via qemu0
*shrugs* ObFilm: Never Been Kissed.

Next.

Previous.