After my entry yesterday about MySQL truncating data, several people have pointed out that MySQL 4.1 or later gives you a warning. Yes, this is true. You can even see it in the example I gave:

Query OK, 1 row affected, 1 warning (0.00 sec)

I ignored mentioning this, but perhaps should have said something about it. I reason I didn't mention it was because I didn't feel that a warning really helped anyone. Developers have enough problems remembering to check for errors, let along remembering to check in case there was a warning as well. Plus, they'd then have to work out if the warning was something serious or something they could ignore. There's also the question of how well the language bindings present this information. Take for example, PHP. The mysqli extension gained support for checking for warnings in PHP5 and gives the following code as an example of getting warnings:

mysqli_query($link, $query);
if (mysqli_warning_count($link))  
   if ($result = mysqli_query($link, "SHOW WARNINGS"))  
      $row = mysqli_fetch_row($result);
      printf("%s (%d): %s\n", $row[0], $row[1], $row[2]);
      mysqli_free_result($result);
    
 

Hardly concise code. As of 5.1.0, there is also mysqli_get_warnings(), but is undocumented beyond noting its existence. The MySQL extension does not support getting warning information. The PDO wrapper doesn't provide any way to get this information. In perl, DBD::mysql has a mysql_warning_count() function, but presumably would have to call "SHOW WARNINGS" like in the PHP example. Seems Python's MYSQLdb module will raise an exception on warnings in certain cases. Mostly using the Cursor object. In java, you can set the jdbcCompliantTruncation connection parameter to make the driver throw java.sql.DataTruncation exceptions, as per the JDBC spec, which makes you wonder why this isn't set by default. Unfortunately this setting is usually outside the programmer's control. There is also the java.sql.Statement.getWarnings(), but once again, you need to check this after every statement. Not sure if ORM tools like Hibernate check this or not. So, yes MySQL does give you a warning, but in practice is useless.

When upgrading from older crm114 releases and trying to retain your existing configuration it is important to check that all the configuration options that the new version expects to be set have been set. While some will cause errors if they re omitted others will appear to work but will cause unwanted behaviour at runtime. For example, omitting good_threshold and spam_threshold will cause everything to be flagged as spam in X-CRM114-Status even though the classifier is working well. In practice there are relatively few configuration options that users are expected to configure so it may be easier to redo the configuration based on the example provided. For safety it s best to delete your existing CSS files too in case they ve been invalidated by a configuration or format change. It s all a bit manual but it s worth it for what it does for my inbox.

Well, no, neither FOSDEM nor Debian are dead. But FOSDEM '09 is gone, over, and dealt with. It was a breeze, for the most part. I'm still not very happy with the booth; that needs to be done better next year. Help and suggestions in that area are more than welcome. One thing that more than deserves a follow-up is Lucas' post about FOSDEM:

On a more sad note, the worst talk of the week-end was without any possible doubt Frans Pop's release team bashing. Nobody is claiming that the release management of the lenny release cycle was perfect: there's always room for improvement. But given the context and the constraints, I think that they did a very good job. Frans' talk boiled down to: "The release team doesn't know what they are doing, I would have done much better because I'm so qualified."

As the person responsible for allowing Frans on the schedule in the first place, while being fully aware of the relationship between Frans and the release team (not at first, but certainly in time to kick him off if I'd have wanted to), I'd like to comment on that. First of all, I do not agree that the last statement in the above is true. Frans showed a few stories of things that went on in the release team, and gave his opinion on what he believed went wrong. He also explicitly said, at the beginning of the talk, that none of the talk was meant personally; that he wanted to offer some constructive criticism instead. I believe he did not fail in doing that, but of course YMMV. Secondly, and more importantly, I do not believe it is healthy for Debian (or any project, for that matter) to reject criticism. Indeed, nobody is claiming perfection. I do not believe any venue where talks can be had should be a good news show. People in a position of power not just the release team, but also, say, the DPL, ftp-masters, buildd maintainers and whatnot have received our trust to do what they need to do to the best of their abilities. If someone in the project would believe that I can significantly improve my work as buildd maintainer, it is not just their right, but indeed their duty to inform me of that fact. This is exactly what Frans was attempting to do, and there is nothing wrong with that. It is also not as if he's not tried reasoning with the release team first. He has made suggestions, which have gotten ignored. I feel that talking to more people about what he feels is right, to see whether they agree with him, is the next logical step to take, and that this it is exactly what Frans was attempting to do at FOSDEM. Having said that, I agree that finishing the talk late was a very bad thing. If anything, a talk on a subject like this should have more, not less, time for discussion. So while I do not agree that his intentions were wrong, I do agree that the execution could have been better. There, that was criticism too. Now, what's next? Oh yes, suggestions. If people have suggestions on how to improve the Debian presence at FOSDEM next year (especially, as above, ideas for creative use of the booth would be welcome), then please send me an email, making sure the subject contains the word 'FOSDEM', so my mailfilters know what to do with it. You'll preferably do so now, while FOSDEM is still somewhat fresh in your memory; I'll take notes and use those for doing better next year. Thanks,

Rumour has it that some types of spammer target the secondary MX servers. The concept is that some people have less control over the secondary MX server and less ability to implement anti-spam measures. Therefore if they accept all mail from the secoondary then a spammer will have more success if they attack the secondary server. True secondary servers are becoming increasingly uncommon, the lower priority servers listed in MX records tend to have the same configuration as the primary, so the benefit for the spammer in attacking the secondary server is probably minimal. But it would be good to know whether they do this. I decided to analyse the logs from a mail server that I run to see if I can find evidence of this. I chose a server that I run for a lient which has thousands of accounts and tens of thousands of messages delivered per day, my own server doesn t get enugh traffic to give good results. I analysed the logs for a week for the primary and secondary MX servers to see if the ratio of spam to ham differed. Now this does have some inherent inaccuracy, some spam will slip past the filters and occasionally a legitimate email will be rejected. But I believe that the accuracy required in a spam filter to avoid making the users scream is vastly greater than that which is required to give a noteworthy result. I produced totals of the number of messages delivered, the number rejected by SpamAssassin (which has a number of proprietary additions), the number of message delivery attempts that were prevented due to rate limiting (most of which will be due to spammers), and the number of attempts to deliver to unknown accounts (some of which will be due to spammers having bad addresses in their lists). For each of these rejection criteria I produced a ratio of the number of rejections to the number of delivered messages for each of the servers. The rate limit count didn t seem useful. While the primary server had a ratio of 0.75 messages rejected due to rate limiting to every message accepted the secondary had a ratio of 0.08. It seems that the secondary just didn t get enough traffic to trigger the limits very often. This is an indication that the more aggressive bots might not be targetting the secondary. The ratio of messages rejected by SpamAssassin to legitimate mail was 0.76:1 on the primary server and on the secondary server it was 1.24:1. The ratio of messages addressed to unknown users to successful deliveries was 3.05:1 on the primary and 7.00:1 on the secondary! This seems like strong evidence to show that some spammers are deliberately targetting the secondary server. In this case both the primary and secondary servers are in server rooms hosted by the same major ISP in the same region. The traceroute between the two mail servers is only 7 hops, and there is only one hop between the two server rooms. So it seems unlikely that there would be some connectivity issue that prevents spammers from connecting to the primary. One other factor that may be relevant is that the secondary server has been in service for some years while the primary is only a few months old. Spammers who store the server IP address with the email address (which happens - change the DNS records to send your mail to a different server and you will see some spam go to the old server) will be sending mail to what is now the secondary server. The difference in the rejected mail volume on the secondary server and the amount that would be rejected if it had the same ratio as the primary amounts to 7% of all mail rejected by SpamAssassin and 14% of all mail addressed to unknown users. I think it s unlikely that any significant fraction of that would be due to spammers caching the server IP address for months after the DNS records were changed. So therefore it seems most likely that something between 7% and 14% of spam is specifically targetted at the secondary server. While the ratio of spam to ham seems significantly worse on the secondary it is still a relatively small portion of the overall spam sent to the service. I had been considering setting up secondary mail servers with extra-strict anti-spam measures but the small portion of the overall spam that is targetted in such a way indicates to me that it is not going to be worth the effort. Another thing that has occurred to me (which I have not yet had time to investigate) is the possibility that some spammers will send the same messages to all MX servers. If that happens then the ratio of spam to ham would increase every time the number of MX servers is increased. In that case it would make sense to minimise the number of MX servers to reduce the amount of CPU power devoted to runing SpamAssassin. Note that I have intentionally not given any numbers for the amount of mail received by the service as it is a commercial secret. Update: One thing I realised after publishing this post is that the secondary MX server is also the main server for mail sent between local users. While the number of users who send mail to other users on the service is probably a small portion of the overall traffic (it s not a really big ISP) it will make a difference to the ratios. Therefore the ratio of spam to ham would be even worse on the secondary MX (assuming for the sake of discussion that local users aren t spamming each other).

Some stuff I want to reminder for next year s FOSDEM:

• Carry a power bar with at least 3 outlets. Power plugs are a limited resource. With a power bar at FOSDEM you always will be able to get power for your laptop/gadgets and make new friends. All at the same time!
• Make a backup schedule, in case the talk you wanted to attend run out of space.
• H agen-Dazs next to LUIZA/LOUISE metro station is good for breakfast and dessert. And there is wifi!

Supposedly, my low, casual shoes contain some longish and slender piece of metal in the sole. At least, that's what the X-rays at the airport detected when I was flying back from FOSDEM. While it isn't shaped like a typical knife, I didn't know how to tell it apart from a potentially hidden blade. Well, I guess that's what the people doing the security control are trained for ... Interestingly enough, it was only detected at the Zurich airport - neither the metal detectors in Nuremberg nor those in Brussels seemed to care about it.

I recently received an email from Sigma Xi with various news items. One of them mentioned a Hawking quote:

Science fiction is useful both for stimulating the imagination and for diffusing fear of the future. -- Stephen Hawking?

Now, Sigma Xi is generally a pretty reputable organization, but I wondered: is a scientific organization really encouraging fear of the future? Mind, I think a certain degree of concern about the future is healthy, both individually and societally, and there is a long tradition in science fiction of writing novels about terrifying futures. But Sigma Xi is usually upbeat about the wonders of science and technological progress. Besides, there's a real dissonance in that quote between becoming more creating and imaginative, and becoming more afraid of the future. Maybe it was intentional, but it seems very out-of-place. So, I started to wonder whether this was really what Hawking said. Typing a few key words into Google, I looked at where else this quote showed up. According to the NSF, Sigma Xi's quote is correct.

According to renowned physicist Stephen Hawking, "science fiction is useful both for stimulating the imagination and for diffusing fear of the future." Interest in science fiction may affect the way people think about or relate to science...

ERIC, which is some sort of database of internal government reports, agrees:

According to renowned physicist Stephen Hawking, "science fiction is useful both for stimulating the imagination and for diffusing fear of the future." Indeed, several studies suggest that using science fiction movies as a teaching aid can improve both motivation and achievement.

So, maybe Hawking really does encourage being afraid of the future? Well, here was the third and final reference to this quote that I found, from (of all places) an unofficial transcript of a Larry King Live interview:

HAWKING: I think science fiction is useful, both for stimulating the imagination and for defusing fear of the future. But science fact can be even more amazing. Science fiction never suggested anything as strange as black holes.

So, it looks like this quote comes from a live TV interview. I don't know whether the different interpretations come from different transcripts (this is the only one I can find); defused and diffused are similar enough that it would be easy for either one to be reasonable -- and IIRC, he speaks through some sort of speech-generating machine, which would just make it easier to misunderstand him. But given that Hawking is a crazy-science-guy, I suspect that he would rather defuse fear of the future than spread it around (even if it became more diffuse as it was spread), and that the transcript above is what he meant to say. In which case, I wonder about the versions of the quote that I found everywhere else. Didn't anyone do a double-take when they were typing up those papers, Web pages and news items? Of course, there is a third possibility, which I rather like: the quote could be a play on words, a sort of slant pun (a phrase that I just invented, which is the funnier cousin of a slant rhyme). What's particularly cool about this particular ambiguity is that (a) both sentences are sensible, (b) they have related but opposite meanings, and (c) somewhat oddly, I agree with both of them: science fiction is good at defusing unwarranted fear of the future, while also diffusing fear where it's entirely warranted. I doubt Hawking was going for the double meaning, but it's a nice thought.

---

And for the inevitable people who can't figure out what this blog entry is about, here's a hint: diffuse:

Verb

1. (transitive) To spread over or through as in air, water, or other matter, especially by fluid motion or passive means.
2. (intransitive) To be spread over or through as in air, water, or other matter, especially by fluid motion or passive means.

defuse:

Verb

1. To remove the fuse from a bomb, etc.
2. To make something less dangerous, tense, or hostile.

About one hour ago we have finished the first Openmoko user meeting in Karlsruhe. Twelve FreeRunner (and Neo1973) owners have gatherd in the rooms of Entropia (the local CCC club), and discussed the various distributions, learned aboutt the FSO-alternative PyNeo, which was advocated by Josh, compared GPS applications and talked about various other projects and issues.I collected some statistics about Distribution usage. Most common was SHR, with five users, followed by OM 2008.12 with three users. Single users had Debian, OM testing from pre 2008.12, PyNeo, EmDebian and OM 2008.9 installed. Only counting those who use their FreeRunner as their day-to-day phone, three are using SHR, one OM 2008.12 and one OM testing. I conclude that SHR seems to be a good choice if you want to have a working phone.Asked about their primary use case for the FreeRunner, almost all mentioned telephony and GPS. Half of the participants want to use it to browse the web, a little less think that games are important. Two people, who came from Stuttgart, see CellHunter as an important use caseConsidering that we filled three hours without running out of topics and the good feedback, we will likely have a sequel to this. A date has not been fixed yet, but will be discussed on the openmoko-community mailing list.BTW: I m still planning to package the SHR applications for Debian as soon as possible. Only one dependency (libetk) is missing, but according to Lutin from the pkg-e team, it s almost ready.

I use Google Reader to manage most of the sites that I regularly read. Today it occurred to me that it would be very useful to have some functionality whereby feeds in a common folder could be read in one aggregated display, a bit like how planetplanet works. I then learned that Google Reader does exactly that already: It's fairly obvious to use in the normal reader, a little bit more subtle in the mobile version (you must click the folder once to load it up in a new page, then click it again to see the aggregated contents). That prompted me to try and tidy up my subscriptions and sort them in a way that took advantage of this feature. Deciding that the Google Reader interface was too primitive to properly organise all of my subscriptions (there are about 100 or so of them), I exported my subscriptions as OPML and imported them into liferea. I then dutifully pruned the list and sorted the remainder into a heirarchy of folders. In most cases I categorised things according to their topic: e.g., "computing", "friends", etc.; with two important exceptions: feeds that I track for debugging purposes but don't actually read (such as my own weblog) I put in a folder "don't read". Feeds that I have decided I need to read every post to (and there are very few of those) I moved into a top-level folder called "essential". I note that liferea 1.4.18 at least does not support the planet-style aggregation feature for folders that Google Reader does. On re-importing to Google Reader, I discovered another facet of it's behaviour that I'd neglected to pay attention to: it does not support heirarchical folders. (How could I use this software for so long and not really understand how it works?) In fact, folders are more like tags, with feeds existing in multiple folders. To make matters even more confusing, there is an entirely separate notion of tags, which applies to items inside subscriptions. Regarding the concept of an "essential" list. If you cut down your firehose consumption to just feeds which met the "essential" criteria, would you find yourself short on information, or just short on noise? Sometimes I read a book that for one reason or another (perhaps just one: excellent writing?) I cannot put down. I utterly devour the book and finish it much more quickly than average. Wouldn't it be nice if other information could be consumed so vociforously? When trying to read things such as academic papers or bid proposals, someone unused to them may find them very hard going. Someone with more practise might managed to "read" such a document much more quickly. Perhaps there is a point to all the firehose noise: maybe we need long immersion in the culture or context to train ourselves up so we can optimally consume the good stuff?

DDE, Debian Data Export After testing the idea and the prototype with my presentation at Fosdem, I'm proud to announce DDE, Debian Data Export. The problem In Debian, we publish all sort of information, using all sorts of data formats (often ad hoc and obscure), in obscure places. Try to think of an application, for example, that wants to access all this information together:

• Maintainer <-> Source package mapping
• Popcon rankings
• Changelogs
• .desktop files of packages not installed
• What is in the new queue
• Package screenshots
• Some statistics
• Localisation information
• uscan status
• Buildd logs
• sloccount run results
• Debian Weather
• Debian Pure Blend specific information

A nightmare, uh? The solution DDE is a way to make it simple to publish and download data. The aim is to be able to access all sorts of Debian information without worrying about data formats, protocols and access control, and to make it easy to discover what data is available. DDE exports data as a big virtual tree. You can pick a node in the tree by its URL and download all the data that it contains, in a format of your choice: currently it supports JSON/JSONP, YAML, CSV and Python pickled objects. This means that you can now get Debian data using a trivial HTTP client tool or library, and read it using commonly available decoders: both should be available in almost any programming language nowadays. Since JSON and JSONP are supported, this even includes JavaScript in Ajaxy web pages. DDE is not a competitor to UDD: UDD is about creating a central location where all the data can be accessed, while DDE is about giving people a simple way to access data or subsets of data. In a way, DDE and UDD complete each other: the more data enters UDD, the more data is available for DDE. In turn, DDE gives a simple interface to the most popular and useful UDD queries. The dream Here are some hints at what can be done with this:

• Autocompletion in HTML fields
• Export data to feed external sites like or
• Have a way for package managers to easily access all sorts of data
• Have a way to implement fancy tools that can query massive data sets without needing to download them locally

A call for action You can add data to the DDE tree by just putting a data file in yaml, json or pickle format under ~/.dde (I've written a specific guide to this on the Debian wiki). If you wish to create new and fancy Debian statistics or compute other sorts of data, or if you already maintain tools that generate Debian data, including but not limited to, for example:

• Popcon rankings
• .desktop files of packages not installed
• Content of the new queue
• Screenshots
• All sorts of statistics
• Debian Weather

Then please have a look at http://wiki.debian.org/DDE/StaticData and try to publish your data in ~/.dde on merkel.debian.org. For more complicated cases (like accessing a remote database), it is possible to extend DDE via python plugins. You can get in touch with me if you need to go that way.

The last days I stumbled across a blog post of Lucas Nussbaum, which reminded about another story that was going on for a long time.

I wasn't at FOSDEM so I can't tell whether Lucas' impression of bashing is correct or not. But giving the possibility of being correct, it causes a little bit of different light to the other story. But well, let's resting the past and hope for a better future.

End of last year I migrated from PowerPC arch to i386 which made it necessary to reformat/reparition/relabel the old 250 GB disk from Amiga disk label to PC disk label. Therefor I backupped all necessary files from the 250 GB disk to a 160 GB disk.
After installing the system on the 250 GB disk, I wanted to copy over the backup from the other disk, but accidently issued a pvcreate on the backup partition. Big mistake. ;) No need to tell that lots of files ended in lost+found: 28 GB in total. Of course it's troublesome to sort all those files and directories back to their original places.

Because I don't know any tool yet, I wrote some scripts to address this issue. The first script is to be run to get a list of directories and files and a unique file identifier for the files.



This will create two files: /root/ls-md5sum-dirs.txt and /root/ls-md5sum-files.txt. The above is the second version of the file with openssl sha1 instead of md5sum, because I discovered that md5sum is giving me double hashes:



That makes a difference of ~140000 files. The same script with openssl sha1 is giving a better approximation, but still a gap between those two numbers:



The difference is much smaller with openssl but it takes a lot of more time to generate the resulting file.

Anyway, my second script assumes that the files can be identified by an unique hash. Currently it looks like this:



This script generates a similar list as the first script and compares each hash or file and tries to move it back to its original path. Because of the above mentioned difference between total number of lines and unique liines this approach seems a little bit broken.

So, dear lazyweb, is there something that can generate unique fingerprints in a reasonable amount of time (like md5sum) or is there even an already existing application out there that I'm not aware of? And yes: I know that there are backup applications, but that is another topic. In this case, the backup medium has been corrupt so the problem still exists: how to identify files in lost+found and move them back in an automatic way, if possible?

I'm appreciating help, tips, patches, ideas, ... :)

We recently bought a Canon Vixia HG20 camcorder. The HG20 records in AVCHD format (MPEG-4 h.264) at up to 1920 1080. To get from the camcorder to a DVD (or something we can upload to the web), I need some sort of video editing software. This lets me trim out the boring bits, encode the video for DVD, etc. Background In addition to DVD creation and web uploading, I want the ability to burn high-definition video discs. 1920 1080 is significantly higher resolution than you get from a DVD. There are two main ways to go: a blu-ray format disc, or an AVCHD disc. A blu-ray disc has to be burned onto BD-R media, which costs about $5 each, using a blu-ray burner, which costs about $200. AVCHD discs use the same h.264 encoding that the camcorder does, meaning they have better compression and can be burned onto regular DVD+R media, fitting about 30 minutes onto a DVD. Moreover, it is possible to move AVCHD files directly from a camcorder to an AVCHD disc without re-encoding, resulting in higher quality and lower playing time. The nicer blu-ray players, such as the PS3, can play AVCHD discs. AVCHD seems pretty clearly the direction the industry is moving. Compared to the tape-based HDV, ACVHD has higher quality with lower bitrates, better resolution, and much greater convenience. Hard disk or SD-based AVCHD camcorders are pretty competitive in terms of price by now too, often cheaper than tape-based ones. The downside of AVCHD is that it takes more CPU power to process. Though as video tasks are often done in batch, that wouldn t have to be a huge downside. The bigger problem is that, though all the major video editing software claims to support AVCHD, nobody really supports it well yet. The Contenders Back when I got my first camcorder in about 2001 the one that I m replacing now you pretty much had to have a Mac to do any sort of reasonable consumer or prosumer-level video editing. We bought our first iMac back then to work with that, and it did work well with the MiniDV camera. Today, there s a lot more competition out there. The Mac software stack has not really maintained its lead some would even say that it s regressed and the extremely high cost of buying a Mac capable of working with AVCHD, plus Final Cut Express, makes that option completely out of the question for me. It would be roughly $2500. Nobody really supports AVCHD well yet, even on the Mac. Although most programs advertise support of smart rendering a technique that lets the software merely copy unchanged footage when outputting to the same format as the input none of them have smart rendering that actually works with AVCHD source material. Though this fact is never documented, though discussed on forums. Another annoyance, having used Final Cut Express in the past, is that with these programs you can t just go to the timeline and say delete everything between 1:35 and 3:52 ; you have to go in and split up clips, then select and delete them. They seem to be way too concerned about dealing with individual clips. I briefly used Cinelerra on Linux to do some video editing. It s a very powerful program, but geared at people that are far more immersed in video editing than I. For my needs, it didn t have enough automation and crashed too much and that was with MiniDV footage. It apparently does support AVCHD, but I haven t tried it. I ve tried three programs and considered trying a fourth. Here are my experiences: Ulead/Corel VideoStudio Pro X2 Commonly referenced as the go to program for video editing on Windows, I started with downloading the Free Trial of it from Corel. Corel claims that the free trial is full-featured all over on their website, but I could tell almost instantly that it wasn t. I wound up buying the full version, which came to about $65 after various discounts. I wanted to like this program. Its output options include AVCHD disc, Blu-ray disc, DVD+R, and the like. Its input options include MiniDV, AVCHD, ripping from DVD, ripping from Bluray, and just about every other format you can think of. And it heavily advertised proxy editing , designed to let you edit a scaled-down version of AVCHD video with a low-CPU machine, but refer back to the original high-quality footage for the output. It didn t pan out that way. The biggest problem was the constant crashing. I really do mean constant. It probably crashed on me two dozen times in an hour. If you are thinking that means that it crashes pretty much as soon as I can get it re-opened, you d be correct. Click the Play button and it hangs. Click a clip and it hangs. Do anything and it hangs. It did seem to work better with the parts of the source that had been converted to a low-res version with Smart Proxy, though it didn t eliminate the hangs, just reduced them. And every time I d have to End Task, it would forget what it had already converted via Smart Proxy even if I had recently saved the project and have to start over from scratch. I spent some time trying to figure out why it always thought my project was 720 480 even when it was 1920 1080, and why the project properties box didn t even have an option for 1920 1080. After some forum searching, it turns out that the project properties box is totally irrelevant to the program. Yay for good design, anyone? VideoStudio Pro X2 does have good output options, allowing combination of multiple source files onto a single DVD or AVCHD disc as separate titles. Unfortunately, its DVD/AVCHD rendering process also yes hangs more often than not. The documentation for VideoStudio Pro X2 is of the useless variety. It s the sort of thing that feels like it s saying The trim tool is for trimming your clips without telling you what trimming your clips means, or making it obvious how to remove material from the middle of a clip. The proxy editing feature isn t what it should be either. Instead of being something that just automatically happens and Works in the background, you have to manage its queue in the foreground and it forgets what it was doing whenever the program hangs. On the rare occasion when pressing Play did not cause a hang, the AVCHD footage played back at about 0.5fps far, far worse than PowerDirector manages on the same machine. Bits that had been rendered for proxy editing did appear to play at full framerate. I have applied for a refund for this purchase from Corel under their 30-day return policy, and have already uninstalled it from my disk. What a waste. CyberLink PowerDirector 7 Ultra This was the second program I tried, and the one I eventually bought. Its feature set is not quite as nice as Corel s, especially when it comes to versatility of output options. On the other hand, it feels done. It only crashed two or three times on me apparently that s GOOD on Windows? Things just worked. It appears to have proxy editing support, but it is completely transparent and plays back with a decent framerate even without it. It can output to AVCHD, Bluray, and DVD, though smart rendering doesn t work with AVCHD source material. Its weakness compared to the Corel package is that it doesn t have as many options for formatting these discs. You can have only one title on a disc, though you can have many chapters. You have some, but not much, control over compression parameters. The same goes for exporting files for upload to the web or saving on your disk. The documentation is polished and useful for the basics, though not extensive. Overall, this package works, supports all the basics I wanted from it, so I m using it for now. Adobe Premiere Elements 7 I downloaded the trial of this one too. I opened it up, and up popped a dialog box asking what resolution my project would be, interlacing settings, etc. I thought YES now that s the kind of program I want. As I tried out the interface, I kept thinking the same. This was a program not just for newbies, but for people that wanted a bit more control. Until it came to the question of output. Premiere Elements 7 was the only package I looked at that had no option to burn an AVCHD disc. DVD or Blu-ray only. That s a deal-breaker for me. There s no excuse for a program in this price range to not support the only affordable HD disc option out there. So I didn t investigate very much farther. Another annoying thing is that Adobe seems to treat all of their software as a commercial. I m a user, not an audience, dammit. I do not want to buy some photoshop.net subscription when I buy a video editing program. I do not want to see ads for stuff when I m reading PDFs. LEAVE ME ALONE, ADOBE. I just felt sleazy even giving them my email address, let alone installing the program on my system. I think I will feel like a better person once I reboot into Windows and wipe it off my system. Pinnacle Studio 12 Another program that comes highly rated. But I never installed it because its minimum system requirements state that it needs an Intel Core 2 Quad 2.66GHz or higher for 1920 1080 AVCHD editing. And I have only a Core 2 Duo 2.66GHz half the computing horsepower that it wants. And since they offer no free trial, I didn t bother even trying it, especially since PowerDirector got by fine with my CPU. Conclusions This seems to be a field where we can say all video editing software sucks; some just suck a little less. I m using PowerDirector for now, but all of the above programs should have new versions coming out this year, and I will be keeping a close eye to see if any of them stop being so bad.

From my sigfile:

> There really is no substitute for brute force.
Indeed - I must admit to being a disciple of blessed Saint Makita myself.
        -- Robert Sneddon and Tanuki, in the Monastery

Over the last couple of weeks, I've been teaching myself Erlang by solving the problems over at Project Euler. They're interesting problems, require a bit of math knowledge and logical reasoning (my skills in both have been getting rusty anyway), along with just enough coding on each problem to make it interesting. One of the things that I've found, though, is that algorithmic elegance is required far less often than I had expected. In several cases, I've sat down with the problem and gone, "well, I can brute-force it like this, (clickety click), but I'll bet it'll take forever to run. I'll set that going while I think about a bett... oh, it's done already." Part of this is that I've got a nice shiny-fast computer, and I'm writing the sort of code that Erlang does well at (and using it's sexy-powerful concurrent programming features to hammer away at all my CPU cores, with practically no extra coding effort on my part), but primarily the problem is that I don't have that good a "feel" for how long something's going to actually take to run. I suspect I'm not the only one, and that it's not something that experience is going to completely solve. My take away lesson from all this (apart from a working knowledge of the basics of Erlang), is one that has been said a million times before, but is so easily forgotten in the excitement of coding: premature optimisation is the root of all evil. In other words, first make it work, then make it fast.

Welcome, Congress, to our generation s electric sun. Earlier, I had posted the Wikileaks link to these congressional reports with comments of my own but I thought I would pen down a few thoughts as I finally electronically leafed through some of them. These reports remind me a little bit of another set of reports that are publicly available, which are the Congressional Quarterly reports, which are an excellent resource for research. They are a bit dry but provide a wealth of information and perhaps more important, citations to law cases, journalist articles, and academic pieces (everything that journalistic pieces, in other words, do not do). It does 1/2 the research for you, as I like to think. The few reports I have scanned from the leaks remind me, in fact, of the CQ reports, in content, style and tone. And while I thought CQ reports published on a wide range of topics, these semi-private reports are far more extensive in terms of topics and far more specific as well:

CRS: Humane Treatment of Farm Animals: Overview and Issues, December 10, 2008
CRS: FDA s Authority to Ensure That Drugs Prescribed to Children Are Safe and Effective, December 2, 2008
CRS: African American Members of the United States Congress: 1870-2008, December 4, 2008
CRS: The Pigford Case: USDA Settlement of a Discrimination Suit by Black Farmers, January 13, 2009
CRS: Selected Legal and Policy Issues Related to Coalbed Methane Development, March 9, 2004

I look forward to hearing/learning more about how they are used (I can t imagine that each report is read by many) and why exactly they were hidden away as they don t seem to be the type of information that should be kept classified.

Welcome, Congress, to our generation s electric sun. Earlier, I had posted the Wikileaks link to these congressional reports with comments of my own but I thought I would pen down a few thoughts as I finally electronically leafed through some of them. These reports remind me a little bit of another set of reports that are publicly available, which are the Congressional Quarterly reports, which are an excellent resource for research. They are a bit dry but provide a wealth of information and perhaps more important, citations to law cases, journalist articles, and academic pieces (everything that journalistic pieces, in other words, do not do). It does 1/2 the research for you, as I like to think. The few reports I have scanned from the leaks remind me, in fact, of the CQ reports, in content, style and tone. And while I thought CQ reports published on a wide range of topics, these semi-private reports are far more extensive in terms of topics and far more specific as well:

CRS: Humane Treatment of Farm Animals: Overview and Issues, December 10, 2008
CRS: FDA s Authority to Ensure That Drugs Prescribed to Children Are Safe and Effective, December 2, 2008
CRS: African American Members of the United States Congress: 1870-2008, December 4, 2008
CRS: The Pigford Case: USDA Settlement of a Discrimination Suit by Black Farmers, January 13, 2009
CRS: Selected Legal and Policy Issues Related to Coalbed Methane Development, March 9, 2004

I look forward to hearing/learning more about how they are used (I can t imagine that each report is read by many) and why exactly they were hidden away as they don t seem to be the type of information that should be kept classified.

Today is Monday, what means it is time for Gammu release. Main improvements are this time in DBI driver and dummy phone driver. This allowed increasing number of test cases and they revealed number of bugs in code, which were fixed. The good thing is that we now have covered 30% of code by testcases. There is still place to improve, but I think this is good milestone, because most of untested code lies in drivers for different phones, which are hard to test automatically without having real phone connected to test server. Other thing I would like to highlight is DBI backend in SMSD. It is still new, but you should consider using it instead of old MYSQL or PGSQL backends, because it behaves much more nicely to your database and won't ruin system performance that easily (though there are still things to improve). Unfortunately it is not available on Windows right now, but I hope this will change in future. Full list of changes follows:

• Fix some memory leaks found by cppcheck.
• Implemented reconnecting/retrying in DBI driver.
• Fix memory leaks in SMSD found by valgrind.
• Fix decoding of another Nokia message type.
• Support for passing received message ID in DBI.
• Dummy phone now supports phonebook, calendar, todo, filesystem.
• Experimental SMSD monitor support on Windows.
• Added ID for Nokia 2680 (bug #813).
• Number of retries of message send in SMSD can be configured.
• Updated SQL files to new version (mostly added useful indexes).
• Dropped own logic for generating IDs for SMS and rely on SQL server (only in DBI driver for now).

You can download from usual place: http://cihar.com/gammu/, Debian users will find packages in experimental soon, SUSE, Fedora and Mandriva users can check experimental repository in OpenSuse Build Service.

Of course, I was in Brussels this week-end, for FOSDEM. I gave a talk about Ultimate Debian Database (slides here) in the Debian devroom. The conclusion was UDD is ready, go play with it! so you know what you should do now! FOSDEM in general was great and huge (see the video from Quim Gil if you still doubt that) as usual. On a more sad note, the worst talk of the week-end was without any possible doubt Frans Pop s release team bashing. Nobody is claiming that the release management of the lenny release cycle was perfect: there s always room for improvement. But given the context and the constraints, I think that they did a very good job. Frans talk boiled down to: The release team doesn t know what they are doing, I would have done much better because I m so qualified. Giving such talks at FOSDEM, in front of an audience with many people not involved in Debian development, is really insulting. To make things worse, he finished the talk late, not leaving any time for questions/discussion, so the audience might be left with the impression that his opinions are representative of the opinions of DDs.

Biella asks if I can read novels from a screen. Yes, I've read what has to be getting close to a thousand novels that way, over the past 8+ years. I was thinking about this just the other day as I read The once and future e-book: on reading in the digital age. That article does a good job arguing against many of the common complaints about ebooks, such as "I can't read an ebook in the bathtub".

my pile of paper

Right now, I have a pile of paper books I want to read or am in the middle of reading. But if I want something new to read, I don't reach for that pile, but for my (much larger) directory of ebooks. The pile of paper is reserved for special occasions: plane trips, camping, etc. I started reading ebooks in circumstances where it was too hard to get access to paper ones. Traveling in the third world, hauling along a lot of paper books wasn't practical. Living rural, where every book had to be backpacked in. I've stuck with it because it works for me and because I've enjoyed stretching my muscles of concentration on the screen. To me, the focus required to read an ebook from the screen is similar to the focus required to work on a program. And completly different than skimming through a web page. The best example of this that I have right now is Real World Haskell, which I've almost finished (hurrah!) reading in a combination of paper and web editions. I was only able to stick with reading this book when I got the paper version. I use the web version for reference, or for looking for help on a challenging part of the book. For reading, it's just too much a web page; too easy to skim, or be distracted by the comments that can be attached to every paragraph, or jump off to Wikipedia.

screenshot of my ebook reader

Just as most programmers tune their development environment so that it uniquely suites them, I've put a lot of thought, trial, and error into tuning my ebook reading environment. The most important thing seemed to be getting rid of anything that interfered with focusing on the text. Since I wasn't (easily) able to use that environment for reading Real World Haskell, I wasn't able to read it from the screen.