The last few days I have spent at work here at the University of oslo testing if the new batch of computers will work with Linux. Every year for the last few years the university have organized shared bid of a few thousand computers, and this year HP won the bid. Two different desktops and five different laptops are on the list this year. We in the UNIX group want to know which one of these computers work well with RHEL and Ubuntu, the two Linux distributions we currently handle at the university. My test method is simple, and I share it here to get feedback and perhaps inspire others to test hardware as well. To test, I PXE install the OS version of choice, and log in as my normal user and run a few applications and plug in selected pieces of hardware. When something fail, I make a note about this in the test matrix and move on. If I have some spare time I try to report the bug to the OS vendor, but as I only have the machines for a short time, I rarely have the time to do this for all the problems I find. Anyway, to get to the point of this post. Here is the simple tests I perform on a new model.

• Is PXE installation working? I'm testing with RHEL6, Ubuntu Lucid and Ubuntu Maverik at the moment. If I feel like it, I also test with RHEL5 and Debian Edu/Squeeze.
• Is X.org working? If the graphical login screen show up after installation, X.org is working.
• Is hardware accelerated OpenGL working? Running glxgears (in package mesa-utils on Ubuntu) and writing down the frames per second reported by the program.
• Is sound working? With Gnome and KDE, a sound is played when logging in, and if I can hear this the test is successful. If there are several audio exits on the machine, I try them all and check if the Gnome/KDE audio mixer can control where to send the sound. I normally test this by playing a HTML5 video in Firefox/Iceweasel.
• Is the USB subsystem working? I test this by plugging in a USB memory stick and see if Gnome/KDE notices this.
• Is the CD/DVD player working? I test this by inserting any CD/DVD I have lying around, and see if Gnome/KDE notices this.
• Is any built in camera working? Test using cheese, and see if a picture from the v4l device show up.
• Is bluetooth working? Use the Gnome/KDE browsing tool to see if any bluetooth devices are discovered. In my office, I normally see a few.
• For laptops, is the SD or Compaq Flash reader working. I have memory modules lying around, and stick them in and see if Gnome/KDE notice this.
• For laptops, is suspecd/hibernate working? I'm testing if the special button work, and if the laptop continue to work after resume.
• For laptops, is the extra buttons working, like audio level, adjusting background light, switching on/off external video output, switching on/off wifi, bluetooth, etc? The set of buttons differ from laptop to laptop, so I just write down which are working and which are not.
• Some laptops have smart card readers, finger print readers, acceleration sensors etc. I rarely test these, as I do not know how to quickly test if they are working or not, so I only document their existence.

By now I suspect you are really curious what the test results are for the HP machines I am testing. I'm not done yet, so I will report the test results later. For now I can report that HP 8100 Elite work fine, and hibernation fail with HP EliteBook 8440p on Ubuntu Lucid, and audio fail on RHEL6. Ubuntu Maverik worked with 8440p. As you can see, I have most machines left to test. One interesting observation is that Ubuntu Lucid has almost twice the framerate than RHEL6 with glxgears. No idea why.

A few months ago, my company made the decision to switch from our previous mail provider, to use Google Apps. It was a no-brainer really, most of the company were using it already, for the calendars, and Documents, and well we were a bit fed up with our previous mail provider. So we made the move. As an online retailer, we obviously get a lot of Financial related emails, and our Accountancy Department has an email address setup as a group, where the group sends to all the members in the Accounts Team. A pretty simple setup. However, for the last 2 months, I ve been fighting an uphill battle to get this to work properly. When an email gets delivered to a Google Group, it stops there, and then Google re-sends the email to the people in the group. Somehow, this triggers the receiving account to believe that it s originated from the Google Groups servers, rather than the actual originating server. Google Apps emails have a pretty nifty spam filtering service. For those well-known services on the web, it ll check whether it actually came from them (I think through some combination of DKIM and SPF), and bounce it if not. Can you see where I m going yet? Google Groups has a nifty feature to stop it attempting to send emails to addresses that don t exist. If an email address bounces at a certain rate, it ll flag it as undeliverable. If all the emails in a group are flagged as undeliverable, it ll bounce that email to the original party, so that they know no-one received it. The Accounts team are setup to receive information regarding Paypal Payments, Disputes, etc etc. A couple of days in, I had our Financial controller tell us that the Accounts team had stopped receiving emails. Well, being who I am I sent a test email out guess what I got back? Hello <redacted>, Your message could not be delivered because of previous failures during delivery attempts to this mailing list. Please update the list with valid addresses. This is an example of some of the bounces we received: ----------------------------------------------
Recipient: pa.......@mobilefun.co.uk 550 550 5.7.1 Unauthenticated email is not accepted from this domain. u22si29486554yba.55 (state 18).
Message-Id: <redacted> ----------------------------------------------
Recipient: ni........@mobilefun.co.uk 550 550 5.7.1 Unauthenticated email is not accepted from this domain. u22si29486554yba.55 (state 18).
Message-Id: <redacted> ----------------------------------------------
Recipient: lu..........@mobilefun.co.uk 550 550 5.7.1 Unauthenticated email is not accepted from this domain. u22si29486554yba.55 (state 18).
Message-Id: <redacted> *sigh* time to play with the group. I change the members of the group to use one of our alias domains instead of the @mobilefun.co.uk this gets emails working again. 2 days later . our Financial Controller comes to me and complains that clients are complaining that they re receiving bounce emails when sending to his team I check, and once again, get the same message. Time to open a support ticket. That was in October since then we ve tried everything Changing to user-defined groups and switching off the spam filters, whitelisting the groups, setting up DKIM on our domain, changing the spam levels, clearing bounce statuses all to no avail I ve been back and forthing with Google Enterprise Support for 2 months and have still not found a suitable solution *sigh*. I must say however, that the person dealing with the case at Google (Josephine H) has been professional and helpful all the way along, even with my rising frustration at the issue. Today I ended up calling their service unusable number, after having tried a couple of times to change the groups so that the Accounts Team could receive emails again and finding that none of my previous tricks worked. While on hold, I found that I could use a local part extension to an email address, and the group would recognise it as a new email address and therefore have no bounce status. I ve now made a script using the Provisioning APIs, and a bit of python-fu that will generate a local part extension based on the current date/time, and replace the users in the group for the Accounts team with those. Say for example, the primary email address for a user was test@example.com it d add a user of test+20101221235641@example.com. This is set to run each day so it s pretty much the same as resetting the bounce status (which fixes things for a short while) on a daily basis. According to my latest email from Google Enterprise support:-

At the moment there is an incentive going on to fix this outright as messages from Paypal etc have been causing bounce s for other domains. This fix is supposedly due very early in the New Year and will solve this problem indefinitely.

I wait with baited breath but for now, I m happy with my hack.

As of tonight, all translatable debconf templates for packages in squeeze are translated in German and French, after udev entered testing. Both teams are very happy to join the Russian team who was the very first to reach that goal on November 5th. Unfortunately, for our Russian friends, debian-edu-config is currently missing for them now. We of course hope for Debian Edu developers to correct this before squeeze. The Swedish team should join the 100% gang as soon as recently uploaded console-setup reaches testing. The Portuguese team is now only missing two complete packages to reach that 100%: sympa and (again) udev. udev is likely to be updated but, unfortunately, this is much less sure for sympa. The Czech team is only missing bugzilla. Again, unfortunately, that package needs an upload to testing-proposed-updates (that should fix and RC bug) and I am technically unable to do it (and the maintainer is apparently not motivated for doing it, arguing that a more recent upstream version should go in instead, which is unacceptable for the release team). The Spanish team made a great effort, but I'm not sure that the 16 packages they're missing will make it. Next time, folks! Full list for the "7 languages trying to reach 100%" at http://i18n.debian.net/debian-l10n/l10n-testing.txt

I would appreciate if you could test the proposed fixes for the exim4 privilege escalation bug CVE-2010-4345. Preliminary binary and source packages for squeeze/sid and lenny are available here:
deb http://www.bebt.de/debian/ sid exim4+cve
deb-src http://www.bebt.de/debian/ sid exim4+cve
deb http://www.bebt.de/debian/ lenny exim4+cve
deb-src http://www.bebt.de/debian/ lenny exim4+cve
You can also browse the changes in SVN (lenny and sid) or build your own binaries.

---

Changelog:

• 67_unnecessaryCopt.dpatch: Do not use exim's -C option in utility scripts. This would not work with ALT_CONFIG_PREFIX.
• Pull changes related to fixing CVE-2010-4345 from exim 4.73 rc1. Closes: #606527
  • 1_cfile_norw_eximuid: Don't allow a configure file which is writeable by the Exim user or group.
  • 2_permcheck_configurefile: Check configure file permissions even for non-default files if still privileged.
  • 3_remove_ALT_CONFIG_ROOT_ONLY: Remove ALT_CONFIG_ROOT_ONLY build option, effectively making it always true.
  • 4_FD_CLOEXEC: Set FD_CLOEXEC on SMTP sockets after forking in the daemon, to ensure that rogue child processes cannot use them.
  • 5_TRUSTED_CONFIG_LIST: Add TRUSTED_CONFIG_LIST compile option.
  • 6_nonroot_system_filter_user: If the system filter needs to be run as root, let that be explicitly configured. The default is now the Exim run-time user.
  • 7_filter_D_option: Add a (compiletime) whitelist of acceptable values for the -D option.
  • 8_updatedocumentation: Update documentation to reflect the changes.
• 4_FD_CLOEXEC replaces 80_fdleak.dpatch, drop the latter.
• Build with WHITELIST_D_MACROS=OUTGOING. Post patch 7_filter_D_option exim will not regain root privileges (usually necessary for local delivery) if the -D option was used. Macro identifiers listed in WHITELIST_D_MACROS are exempted from this restriction. mailscanner (4.79.11-2.2) uses -DOUTGOING.
• Build with TRUSTED_CONFIG_LIST=/etc/exim4/trusted_configs. Post patch 3_remove_ALT_CONFIG_ROOT_ONLY exim will not re-gain root privileges (usually necessary for local delivery) if the -C option was used. This makes it impossible to start a fully functional damon with an alternate configuration file. /etc/exim4/trusted_configs (can) contain a list of filenames (one per line, full path given) to which this restriction does not apply.

NEWS entry: Exim versions up to and including 4.72 are vulnerable to CVE-2010-4345. This is a privilege escalation issue that allows the exim user to gain root privileges by specifying an alternate configuration file using the -C option. The macro override facility (-D) might also be misused for this purpose. In reaction to this security vulnerability upstream has made a number of user visible changes. This package includes these changes. If exim is invoked with the -C or -D option the daemon will not regain root privileges though re-execution. This is usually necessary for local delivery, though. Therefore it is generally not possible anymore to run an exim daemon with -D or -C options. However this version of exim has been built with TRUSTED_CONFIG_LIST=/etc/exim4/trusted_configs. TRUSTED_CONFIG_LIST defines a list of configuration files which are trusted; if a config file is owned by root and matches a pathname in the list, then it may be invoked by the Exim build-time user without Exim relinquishing root privileges. As a hotfix to not break existing installations of mailscanner we have also set WHITELIST_D_MACROS=OUTGOING. i.e. it is still possible to start exim with -DOUTGOING while being able to do local deliveries. If you previously were using -D switches you will need to change your setup to use a separate configuration file. The ".include" mechanism makes this easy. The system filter is run as exim_user instead of root by default. If your setup requies root privileges when running the system filter you will need to set the system_filter_user exim main configuration option.

Sven has been very busy getting a new Debian ncurses packages into order. With the multiple changes made and the fact that so many things depend on the shared libraries, we decided that it is best to put the first cut of these packages into experimental.

We're now using the Quilt 3.0 source format for our packages, which means that we should have a better handle on the patches, especially as they all now have Dep-3 patch tags .

So what has changed, why should you go over and try the experimental packages?

• First, it will help us with checking the packages actually work, Sven has tried them out (I will be soon) but the more testing the better.
• We've updated the upstream patch level (basically the upstream version) on this set of packages.
• New package! ncurses-examples which are all the example programs that shows you what ncurses can do.

A new release 0.9.0 of Rcpp is now available at CRAN and has just been uploaded to Debian. As always, sources are also available from my local directory here. With this release, the older API which we have been referring to as the classic Rcpp API has been split off into its own new package RcppClassic to ensure backwards compatibility. Rcpp will now contain only the new API. We also fixes a number a minor bugs and applied a few contributed patches which extended functionality or documentation as detailed below in the NEWS entry:

0.9.0   2010-12-19
    o   The classic API was factored out into its own package RcppClassic which
        is released concurrently with this version.
    o   If an object is created but not initialized, attempting to use
        it now gives a more sensible error message (by forwarding an
        Rcpp::not_initialized exception to R).
    o   SubMatrix fixed, and Matrix types now have a nested ::Sub typedef.
  
    o   New unexported function SHLIB() to aid in creating a shared library on
        the command-line or in Makefile (similar to CxxFlags() / LdFlags()).
    o   Module gets a seven-argument ctor thanks to a patch from Tama Ma.
    o   The (still incomplete) QuickRef vignette has grown thanks to a patch
        by Christian Gunning.
    o   Added a sprintf template intended for logging and error messages.
    o   Date::getYear() corrected (where addition of 1900 was not called for); 
        corresponding change in constructor from three ints made as well.
    o   Date() and Datetime() constructors from string received a missing
        conversion to int and double following strptime. The default format
        string for the Datetime() strptime call was also corrected.
    o   A few minor fixes throughout, see ChangeLog.

Thanks to CRANberries, there is also a diff to the previous release 0.8.9. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

Debian Unstable (also known as sid) is one of the 3 distributions that Debian provides (along with Stable and Testing). It s not conceived as a product for end-users, instead it s the place where contributors are uploading newer packages. Daily. Yes that means that Unstable is a quickly moving target and it s not for everybody. But you can use it and your computer won t explode. 1. It contains mainly stable versions of the software Yes, you read it right. Unstable is not full of development versions of the various software. It happens on some software but then it s usually a conscious decision of the maintainer who believes that this specific version is already better than the previous one. The packages in sid are supposed to migrate to testing, the place where the next Debian stable release is prepared. So maintainers are advised to only upload stuff that is of release quality, the rest should be uploaded to experimental instead. 2. It doesn t break badly every other day Breakages happen but they are not a big deal usually. It has been long time since I could not reboot my computer after an upgrade or since the graphical interface was no longer working. The kind of breakages that you have is that one software stops working, or triggers an annoying bug, or that a few packages are uninstallable. In most cases, you can save yourself by downgrading to the version available in Testing. Or by finding a work-around in the bug tracking system. Or by not upgrading because you have apt-listbugs installed and you have been warned about the problem. 3. It s the basis of other distributions If Debian Unstable was really so bad, it would not be a good basis to build a derivative distribution, isn t it? But Ubuntu and SiduxAptosid (to name only two) are based on Debian Sid. 4. It s not inherently less secure than Stable or Testing High impact security vulnerabilities will usually be quickly fixed in Stable and Unstable. The stable upload is done by the security team while the unstable one is made by the maintainer. Testing will usually get the fix through the package uploaded to Unstable, so testing users get security updates with a delay. For less serious vulnerabilities, it s entirely possible that stable does not get any update at all. In that case, unstable/testing users are better served since they will get the fix with the next upstream version anyway. Of course, it happens that maintainers are busy or that something falls through the cracks, but there are other people watching RC bugs who will fix this if the maintainer doesn t react at all. 5. I use it on my main computer And many other people do the same. And you can do the same if you meet the criteria below:

• you can work on the command-line (enough to downgrade a problematic package, to edit configuration files, etc.);
• you know how to work with APT and multiple distributions in /etc/apt/sources.list;
• you are able to read/write English so that you can read/file bug reports when needed;
• you have another computer connected to the Internet that you can use to lookup documentation (or the bug tracking system, or the support mailing lists) when your usual computer is off-line for a reason that you don t understand.

If you feel you are not ready for the jump, click here to subscribe to this blog (or here via the RSS feed), I ll surely teach some of the required skills in future articles. PS: All that said, if you have a working sid installation, do not upgrade it just before an important presentation, or before a trip. It will always break at the most annoying time. Unless you like to live dangerously, of course.

41 comments Liked this article? Click here. My blog is Flattr-enabled.

A new release 0.9.0 of Rcpp is now available at CRAN and has just been uploaded to Debian. As always, sources are also available from my local directory here. With this release, the older API which we have been referring to as the classic Rcpp API has been split off into its own new package RcppClassic to ensure backwards compatibility. Rcpp will now contain only the new API. We also fixes a number a minor bugs and applied a few contributed patches which extended functionality or documentation as detailed below in the NEWS entry:

0.9.0   2010-12-19
    o   The classic API was factored out into its own package RcppClassic which
        is released concurrently with this version.
    o   If an object is created but not initialized, attempting to use
        it now gives a more sensible error message (by forwarding an
        Rcpp::not_initialized exception to R).
    o   SubMatrix fixed, and Matrix types now have a nested ::Sub typedef.
  
    o   New unexported function SHLIB() to aid in creating a shared library on
        the command-line or in Makefile (similar to CxxFlags() / LdFlags()).
    o   Module gets a seven-argument ctor thanks to a patch from Tama Ma.
    o   The (still incomplete) QuickRef vignette has grown thanks to a patch
        by Christian Gunning.
    o   Added a sprintf template intended for logging and error messages.
    o   Date::getYear() corrected (where addition of 1900 was not called for); 
        corresponding change in constructor from three ints made as well.
    o   Date() and Datetime() constructors from string received a missing
        conversion to int and double following strptime. The default format
        string for the Datetime() strptime call was also corrected.
    o   A few minor fixes throughout, see ChangeLog.

Thanks to CRANberries, there is also a diff to the previous release 0.8.9. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

Tokyo Area Debian Meeting was held on 18 Dec 2010, in Tokyo. We talked about libsane, CACert, and the usual stuff for year-end event. We did look back into 2010, and looked out towards what will happen in 2011, and 2012.

On the last weekend my team and I participated in the Mobile App Camp (link in Lithuanian) organized by Omnitel, which is one of the dominant mobile operators in Lithuania. The event seems to be part of Omnitel's push for increasing the share of smartphones in the Lithuanian market, which has been lagging behind the European trends.

Our team consisted of Povilas Kytra, who is behind the TV.LT project, Mantas Kanaporis from A-Gain and me. In the weekend we built an app that shows the TV programme for the coming day for all the Lithuanian TV channels. (The app is not yet available on the Android Market, but we are working on it.)

Here's a screenshot of the main screen of the application:

Choosing a PhoneI was considering renewing my Three contract and getting a HTC Desire HD [1]. What I need is a phone that is good for being a ssh client on 3G networks, has a good camera, and has all the fancy Google Apps.In the comments Lon recommended a Norwegian review of phone cameras which gave the Sony Ericsson Xperia X10 a much better review than the HTC Desire HD [2] the Xperia was the highest rating Android camera phone while the Nokia N8 was the best overall.Also the Xperia is a lot cheaper, I can get it on a $29 monthly cap from Virgin as opposed to $44 per month from Three. So just on hardware and price the Xperia beats the Desire HD.One of the advantages of the Xperia from Virgin being cheap is that my wife and I can use the same model of phone. This avoids having to solve two sets of phone related problems and also allows us to do things like swap batteries between the phones based on who can most conveniently charge their phone.The Need for RootBut one problem with e Xperia is that the CyanogenMod images for rooted Android phones can t be installed on an Xperia because the boot loader hasn t yet been cracked [3]. I would like to run CyanogenMod so I can get wireless proxy support, and support for tunneling IP over DNS, ICMP, and OpenVPN.The Three web site claims that they have 3G phone and net access in Bendigo, but on a recent holiday my phone said that it was roaming all the time and I couldn t get net access. I ended up having to use McDonalds Wifi net access which had ports such as 22 blocked and thus forced me to use Iodine IP over DNS to get proper net access. To avoid having to talk my mother through rebooting servers in future I need to have a mobile ssh client that can use all possible protocols. I could carry my EeePC with me all the time, but sometimes it s good to travel light.OTOH, as I feel compelled to fiddle with all my computers it would probably give me a more reliable mobile experience if I was unable to mess with my phone.Why Buying a Phone Outright Isn t ViableA $29 monthly plan is probably the cheapest plan that will do for anyone who uses a phone regularly, I have had a Three $29 plan for the last four years which allows up to $150 of calls to be made in a month and typically use about $60. So any plan which doesn t have such a cap will have to be no more than half the price of Three on a per-minute basis to compete. If I m going to pay $29 per month ($696 over a 24 month contract) then I can use a free phone. If I was to buy a phone then it would cost at least $500 for anything that I like and maybe a lot more.Buying a phone independently of a contract would about double the cost of owning a phone. It s really not a viable option.Therefore I am compelled to buy a phone that is on offer from a Telco. Things like the Nokia N900 are nice devices but as the Telcos don t offer them I can t consider them.No Discount if you Don t get a PhoneThe annoying thing is that the Telcos don t offer a discount if you choose not to get a phone. Obviously buying the hardware costs them some money, so a $29 cap with a phone included should have a matching offer of something less than $29 if you choose not to get the phone from them. I currently have a $29 per month contract with Three, I can renew that for another two years at the same rate and get a half-decent phone for free or I can renew for two years on a $19 per month contract and get a low-end phone for free but I can t get a price that is lower than $19 per month if I decide to keep my current phone.If Three was to offer such a discount then I would consider buying a phone outright over the net and staying with them. But as it is they don t provide good deals for buying a phone and give me an economic incentive to go to another provider. So I will probably use Virgin when my contract runs out in January.Locked PhonesMany Telcos still sell locked phones on a contract. When that happens it s really difficult to get a phone unlocked as the Telco employees usually aren t very helpful. There are a variety of web sites claiming to generate unlock codes for phones, most seem to charge $10 or more for this service and the free ones have a very small range of phones, so getting the unlock code from the Telco seems to be the only option for a phone at the end of it s contract period as it s not worth enough to justify the $10 expense.While some Telcos sell unlocked phones on plans the ones that lock their phones have a chilling effect on the industry. Most people don t test whether their old phone can be used with a different provider they just throw it out the phone stores conveniently provide bins for old phones that are apparently recycled for some good cause.ConclusionIf you make serious use of mobile phones (EG being ready to fix errors reported by Nagios 24*7) then choosing a new phone and plan is one of the most difficult things there is to do. All the plans are quite complicated and every Telco offers a different set of phones. The Telco web sites are usually poorly done, most of them don t have an option to search for Android phones or for phones with a certain minimum resolution they usually don t even state the resolution and use terms such as WQVGA which don t even have a fixed meaning in pixels. When it comes to choosing a plan most Telcos don t have a clear comparison of the different plans, writing your own spreadsheet comparing plan costs is a good idea.The fact that Telcos such as Virgin and Three/Vodaphone allow free calls to other people using the same company makes it even more tricky. I have to discuss my phone plans with several relatives as there is a good incentive for everyone to use the same provider.I think that we need government regulation on the way that phones are bundled. The market for phones that aren t associated with Telco contracts has been destroyed by the anti-competitive behavior of the Telcos.

• [1] http://etbe.coker.com.au/2010/11/17/mobile-phone-sysadmin/
• [2] http://www.amobil.no/artikler/norges_beste_kameramobil/79081
• [3] http://www.cyanogenmod.com/

The Google books ngram datasets are a nice resource for playing with text generation. I've created a program that will, given a starting phrase, blather on by connecting phrases from the datasets. The results are much more varied, and seem to more often make sense than most Markov text generators I've seen before, such as dadadodo. It had better be -- it's working with data derived from 4% of all books ever written! Some examples. Do they tell a story? (The starting phrase I entered is in italics.)

• He is assailed they merge
• To be or not to be so deceived ! She denies abdominal scanning
• Beauty with enormous strides . The hospital was under the control of the communal organization of solitary mammals . In other words the mere failure of an individual .
• Time passes . It is the child's first day in town and at his own level ?
• Time above his left shirt pocket for safe keeping . At first I accepted the old man's work . He settled in that part of Paris .
• Until then no ill Accident would generate hydrogen sulfide
• Random nonsense put together huge boxes with ammunition against it itself stops . Working things out afterwards
• A change in voice and piano . He sat down at last at No . 5 in connection with another or with others ; and that by some strange disaster .
• O God ! What is the condition of morality is that they all hold hands in a chain of propositions . These propositions are the primary bearers of meaning making and the rearing of children

I have cheated slightly by picking where to stop quoting the blather each time. It tends to go on for quite a while otherwise. Although it often does manage some kind of sense at length too:

• Worst Case Scenarios in Search of Reality : The ERA and emendation . I can never recover . She became thoughtful . He looked at me gloomily above the tree tops . The women were asked how they used to behave more competitively . Forecasts should be made

---

This was accomplished using only 99 lines of code. I imported the datasets into Xapian databases using this program. And this program is the text generator. I threw out the metadata, to keep the Xapian databases of reasonable size. By which I mean, only a dozen gigabytes. It might be interesting to include the publication year data in Xapian, so that it could prefer ngrams that were published around the same time as the input text. I'm still importing data. I may put up a web interface later if there's interest. PS:

• What about Spam ? Who caused it ? You have said it so loud that the little lady in a black storm .
• What about Spam ? I think it cannot be any . The Church had the right men . They are our friends and relations who could be depended upon with regard to the UN . Some people paint the portraits of the writers .

PPS:

• Joey Hess was not the real author of these lines .

Let's say you want to clone a git repository that has the debian packaging on master and the upstream source is kept on upstream/master (since it tracks upstream's master branch). You can simply use git clone or you can do a

gbp-clone --upstream-branch=upstream/master --pristine-tar git://git.debian.org/git/pkg-lvm/multipath-tools.git

This will additionally set up master, upstream/master and pristine-tar to track the corresponding remote branches. There's no --debian-branch option in the above example since master is the default. If you want to safely update these branches at a later point invoke

gbp-pull --upstream-branch=upstream/master --pristine-tar

from inside the repository. This will only update the branches if they can be fast-forwarded. If the repository has a debian/gbp.conf describing the branch layout a call to

gbp-pull

is enough.

It seems the current trend is to use short-form dh. Some people have even thought that dh has superseded CDBS. Since I prefer CDBS for my own packaging, I will say that no, CDBS is not being deprecated and in fact has been active. Packaging with CDBS is very simple. I'll try to explain how it works, and how to package with it. This may turn into a blog post series, but I won't promise anything.

CDBS is a set of makefiles that do several tasks that are common while packaging software, so that you don't have to repeat them over and over on each package. The makefiles are classified into 2 groups: classes and rules. Classes implement the rules required for building and installing software. These are classes because they can and do inherit from others. For example, there is the makefile class, and the autotools, qmake and cmake classes inherit from it, and the gnome and kde (which is for kde 3) classes inherits from the autotools class. Rules implement several other general purpose rules that don't depend on the toolchain used to build the package. For example, there is the debhelper rules file, which takes care of creating the debian package using the usual dh_* commands. There are several rules files which do all sorts of useful stuff, from running license checks to downloading new upstream releases. If this post transforms into a series, we may look into some of them. For now, I'll demonstrate how to use CDBS to package some simple software (only the debian/rules file, of course).

The first task while running creating debian/rules is determining which build system the software uses. I will use qutecsound package as a guide. So, first things first, we start using CDBS!

#!/usr/bin/make -f



# Uncomment this to turn on verbose mode.

#export DH_VERBOSE=1



include /usr/share/cdbs/1/rules/debhelper.mk

include /usr/share/cdbs/1/class/qmake.mk

Now this would be all that is required if qutecsound where a standard qmake package where nothing needs to be customized. Alas, this is not the case. First, there are some variables that need to be passed to qmake to appropriately configure the build. Also, the projects .pro file is not standard-named. How do we fix this? The CDBS way of overriding this kind of behavior is through the use of variables. Since we are using the qmake class, we will be overriding variables named DEB_QMAKE_*:

DEB_QMAKE_CONFIG_VAL += build64

DEB_QMAKE_ARGS = qcs.pro

We add a configuration value (because CDBS is setting other config values), and we set the extra arguments to qmake.

Now, there is a problem: the resulting binary is not created with the name we want. How do we change that? We could patch the source to avoid that, but it is far easier to add a rule to the makefile to do that. We will modify the build to rename the file at the end (and then clean it up on clean because the upstream makefile will not spot it):

build/qutecsound::

	[ -f bin/qutecsound ]   mv bin/qutecsound-d bin/qutecsound



clean::

	rm -f bin/qutecsound

So what did we just do? We extended the build and clean rules with extra stuff to do. In CDBS each package listed in debian/control gets a build/package rule (and several others at different stages of the build) so that one can add stuff specific to each package (this will be much more useful when dpkg finally learns about build-arch and build-indep).

For the final touch, we want to do a few more things. First, we want to ensure that we are using QT4's qmake, because the build will fail if QT3's is used. Second, since I created a manpage for the command, I want to install it. Finally, we want to use parallel building when the user has specified it, to build the project faster. How to do that? Again, via variables:

QMAKE = qmake-qt4

DEB_BUILD_PARALLEL = 1

DEB_INSTALL_MANPAGES_qutecsound = debian/qutecsound.1

CDBS creates several variables for each of the packages listed in debian/control, so we can customize each package build. In our case, this variable is passed to dh_installman from the debhelper rules file (CDBS will invoke the debhelper tools once for each binary package).

After all this we now have the complete debian/rules of a relatively simple package. How does it look like? Like this:

#!/usr/bin/make -f



# Uncomment this to turn on verbose mode.

#export DH_VERBOSE=1



include /usr/share/cdbs/1/rules/debhelper.mk

include /usr/share/cdbs/1/class/qmake.mk



QMAKE = qmake-qt4

DEB_QMAKE_CONFIG_VAL += build64

DEB_QMAKE_ARGS = qcs.pro

DEB_BUILD_PARALLEL = 1



DEB_INSTALL_MANPAGES_qutecsound = debian/qutecsound.1



build/qutecsound::

        [ -f bin/qutecsound ]   mv bin/qutecsound-d bin/qutecsound



clean::

        rm -f bin/qutecsound

Tags: cdbs, packaging

We had another meeting of the Chicago R User Group last evening. This was scheduled somewhat belatedly once we learned that Drew Conway would be in town. Drew gave a very nice talk about his brand new infochimps package (for accessing the eponymous infochimps data service and marketplace). Slides are available on Drew's blog. In fact, he had already blogged about his talk before I had even started to write my slides... The user group meetings have a meme of showing how to use R with different editors, UIs, IDEs,... It started with a presentation on Eclipse and its StatET plugin. So a while ago I had offered to present on ESS, the wonderful Emacs mode for R (and as well as SAS, Stata, BUGS, JAGS, ...). And now I owe a big thanks to the ESS Core team for keeping all their documentation, talks, papers etc in their SVN archive, and particularly to Stephen Eglen for putting the source code to Tony Rossini's tutorial from useR! 2006 in Vienna there. This allowed me to quickly whip up a few slides though a good part of the presentation did involve a live demo missing from the slides. Again, big thanks to Tony for the old slides and to Stephen for making them accessible when I mentioned the idea of this talk a while back -- it allowed to put this together on short notice. And for those going to useR! 2011 in Warwick next summer, Stephen will present a full three-hour ESS tutorial which will cover ESS in much more detail.

<iframe frameborder="0" marginheight="0" marginwidth="0" scrolling="no" src="http://rcm.amazon.com/e/cm?lt1=_blank&amp;bc1=000000&amp;IS2=1&amp;bg1=FFFFFF&amp;fc1=000000&amp;lc1=0000FF&amp;t=dedasys-20&amp;o=1&amp;p=8&amp;l=as1&amp;m=amazon&amp;f=ifr&amp;md=10FE9736YVPPT7A0FBG2&amp;asins=0470929839" style="width: 120px; height: 240px;"></iframe> I got this to read on my trip to Oregon. Initially I was a bit dubious, after reading the chapter headings on Amazon. Indeed, if you read a lot of articles about tech startups, a lot of the material is not particularly novel, and are things that you've already read elsewhere. If, on the other hand, you're looking for a quick introduction to some of the common ideas behind web/tech startups, it's a pretty good list, written by a variety of people, with a lot of practical advice. The list of concepts includes "execution, not ideas", "fail fast", get feedback early", "get a co-founder", and a lot of similar advice that permeates discussions of early stage internet companies. Other people have different tastes, but for me, this passage was a strikingly effective advertisement for living in Boulder, Colorado:

With this as the backdrop, it was natural to make cycling a central part of my work with Everlater's founders, Nate Abbott and Natty Zola. Nate, Natty, and I scheduled regular weekly rides (sometimes several times a week) and while we didn't keep a formal agenda, each ride had a number of topics planned out for us to cover. In this way, we worked through many of the early challenges the business faced as it moved from idea to reality in the course of the summer of 2009. Our typical rides lasted from 60 to 90 minutes during which time we were able to focus on the business without interruption. And while getting some fresh air helps bring perspective, it's hard to avoid thinking about problems in a new way when you're out of the office, surrounded by the beauty of the Boulder foothills while your mind is completely clear. Startups, cycling, mountains... sign me up! Another tidbit I found interesting:

We often talk about the Number One startup killer at TechStars making a product for which there is no interesting market. TechStars accepts just 10 of more than 600 startups that apply and presumably they are among the best. But still, we find that at least one-third of those startups are attempting to build a product that they want, or that no one wants, instead of what the market wants But wait, I thought "execution is everything", and "ideas are worthless". It seems like avoiding bad ideas is potentially worth a whole lot of money in wasted time and energy though, no? The bit about ideas being worthless is a load of crap. The reason there is no market for ideas is that they are not "excludable". There was never a market for lighthouses either, the classic example in economics of a public good. So the government built them, and they were valuable to people operating ships. Ideas for startups may not have a lot of value compared to people executing them well, but to say they are worthless is hyperbole. In any case, though, if you're interested in startups, or especially if you happen to be interested in the TechStars program, it's book full of good advice, even if none of it will come as a revalation to anyone who follows that sort of thing regularly.

I ve used Delicious (del.icio.us) for some time now for managing bookmarks. I have never really used its social features, just its management features. With word that it is closing (sigh, just after I had to leave Bloglines, too) I m looking for something else. There are a somewhat bewildering array of new options available, and I m wondering if people have had experience with them. My requirements are:

• Completely reliable syncing between multiple devices, even if each one is being used and actively bookmarking things simultaneously. My browser is Firefox. Back when I chose Delicious, nothing else accomplished this.
• The ability to present bookmarks as a toolbar in Firefox, either using its own system or a different one.
• I prefer tagging to hierarchical organization.
• I can run a service on my own server, IF it is Free Software, trivial to set up, and needs little care and feeding. If it s not part of Debian, that s a strong negative here.
• If it s hosted in the cloud, I am concerned about privacy, security, and long-term stability. I must have a way to export my data, preferably automatically. I will be reading ToS and privacy policies carefully.
• A system that can be trusted to encrypt my bookmarks on the server side is another nice to have.
• Being able to create bookmarks directly from my Android phone is nice but not required.
• Being able to access bookmarks from a regular web browser is a feature I use a few times a year; again, nice but not required.
What should I look into? Incidentally, if you need to export your Delicious data, go here.

So, I've been working on multitouch on and off the past few months (which have included a solid ten weeks of holiday), but have finally posted the third patch series, which I think should be pretty close to final, to the list today.

Touch events look kind of similar to standard Xi 2.1 events (DeviceMotionNotify and friends), but the implementation of grabbing and event delivery is quite different from the traditional X model that we all know and love.

Firstly, touch grabs are not strictly grabs as with other input events. When a synchronous pointer or keyboard grab activates, it 'freezes' the device, stopping delivery of all further events until the grabbing client has decided what to do. The client can either pass on the event to the next client (replay), opt to continue receiving all events with the device freezing again after each event (synchronous mode), or continue receiving all events with the device unfrozen (asynchronous). While this is happening, the grabbing client is the only one receiving events, hence the name.

Touch grabs break both of these assumptions. Firstly, the device never freezes: delivery of all events (pointer, keyboard, touch, whatever) continues at full tilt, and a touch grab never affects pointer or keyboard events, with the possible future exception of pointer events that are emulated from touch events. Secondly, all clients with grabs, as well as the client with the first selection we find<super>0</super>, receive the events. An 'owner' flag is set (or unset) to indicate to the client that it does, or does not, own the touch stream in question.

The owner principle is important: the theory is that all touch streams must be considered as a whole (i.e. from init, through all the motion events, to the touch stream ending), and cannot be separated into component parts. So a client which does not own a touch stream can do gesture recognition or whathaveyou in the background, but must not act upon it until it becomes the owner. We added an extra request, XIAllowTouchEvents, to control ownership: clients can either reject ownership, passing it on to the next in line, or assert their ownership and permanently remove all other clients from the delivery list.

So it's somewhat different, but (hopefully) not without reason. The motivation behind this was to keep latency as low as humanly possible, since people seem to demand lower latencies from direct touch-based devices. Sticking to the traditional grab model would have meant a flurry of round trips between all the clients in the stack and the server, as well as maintaining huge buffers in the server for every touch event, which each client would have to transfer in turn.

This model is a compromise between my original design and Ubuntu's suggested design; both of us had far more simple designs, but oriented towards different usecases. The original Ubuntu usecase was for a global gesture recognition system, which apps would only opt out of in rare cases. My primary usecase was exactly the reverse: apps doing their own gesture recognition, with a very small, if any, global system. Hopefully it's actually the best of both worlds, rather than the worst possible intersection: I guess we'll find out. :)

At the moment, it's all largely theoretical, but some of the Nokia Qt guys are working on a port and have provided really helpful feedback, and hopefully the GTK+ multitouch branch, which was written against a much earlier and, er, speculative, version of the spec, gets updated too.

Finally, some thanks: to both Nokia and my excellent employer Collabora for sponsoring my part of this work; to Chase Douglas from Canonical for his contributions and for not just giving us the finger and vanishing when we rejected the X Gesture Extension, but hanging around and working through it; and to Peter Hutterer, for spec work and review. There were also numerous others involved earlier (including, but not limited to, Henrik Rydberg working on mtdev and others) who deserve credit, so sorry if I forgot anyone.

If you want to get it and play around with it, the patches/repositories are all laid out in my initial mail linked to earlier, as well as the follow-up mail noting the evdev repository location. The spec has more details, and will hopefully be cleaned up to be a lot less clunkily-written.

[0]: The delivery order is unchanged. For grabs, we start at the root window and proceed down the window stack to the innermost child window, activating grabs in that order. For events selected for with XISelectEvents, we start at the innermost child window, work our way up towards the root window, and stop as soon as we find one.

Update There is an easier method to do all this using gbp-clone as described here. Ah ! Then to build the package, you just need to suggest git-buildpackage where to find the pristin-tar :

git-buildpackage --git-upstream-branch=upstream/master

or you could simply describe (as suggested) the layout in debian/gbp.conf. Easy !!!

---

I've found a lot of different recipes and howtos about git debian packaging, but I failed to find one simple recipe to create a debian package from scratch when upstream is using git. Of course the following is a big patchwork from many different sources. First we need to do a bit of administrative work to setup the repository :

mkdir yourpackage
cd yourpackage
git init --shared

Then, since I'm interested in tracking upstream development branch I'm going to add a remote branch to my repo:

git remote add upstream git://the.url/here.git

at this point I need to fetch upstream and create a branch for it.

git fetch upstream
git checkout -b upstream upstream/master

Now in my repo I have a master branch and an upstream branch. So far, so good. Let's add the debian branch based on master:

git checkout master
git checkout -b debian master

It's in the debian branch where I'm going to keep the debian related files. I'm finally read for hacking git add / git commit / git remove ... When I'm done, I can switch to master, merge the debian branch into it and use git-buildpackage to build the package.

git checkout master
git branch
debian
* master
upstream

git merge debian
git-buildpackage

Suppose I want to put everything on gitourious for example. I'll create an acocunt, set up my ssh pub key and then I've to add an origin ref in my .git/config . Something like :

[remote "origin"]
url = git@gitorious.org:debian-stuff/yourpackage.git
fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
remote = origin
merge = refs/heads/master

The only thing left to do is to push everything on gitourious. the --all is important.

git push --all

People willing to pull your work from girourious have to follow the following script :

$git clone git@gitorious.org:debian-stuff/yourpackage.git
$cd yourpackage
$git branch -a
* master
remotes/origin/HEAD -> origin/master
remotes/origin/debian
remotes/origin/master
remotes/origin/upstream
$git checkout -t origin/debian
$git checkout -t origin/upstream
$git branch -a
debian
master
* upstream
remotes/origin/HEAD -> origin/master
remotes/origin/debian
remotes/origin/master
remotes/origin/upstream
$git checkout master
$git-buildpackage

Maybe there is an easier way to pull all remote branches at once, but I'm not aware of it. Any better way ?

One of the bad aspects of the the C Programing language is that it silently allows many bad C programs. Together with the widespread use of an architecture that is very bad at catching errors (i386), this sometimes leads to common ideoms that are only working accidentially. This is bad as they often break on other architectures and can break with every new optimisations or new feature the compiler adds. Take for example a look at the code there (I tried to leave a comment there but did not succeed): If you see something like this:

     char buffer[1000];
     struct thingy *header;
     header = (struct thingy *)buffer;

then it is time to run. I hope you do not depend on this software, because it is a pure accident if this is doing anything at all. While you can cast a char * to a struct, that is only allowed if that memory actually was this struct (or one compatible, like a struct with the same initial part and you are only accessing that part). In this case it is obviously not (it's just an array of char), so you might see bus errors or random values if the compiler does no optimisations and you are on a architecture where alignment matters. Or the compiler might optimize it to whatever it wants to, because the C compiler is allowed to do everything with code like that. The next problem is the one that post was about: You are not allowed to access an array after its end. Something like

 struct thingy  
     int magic;
     char data[4];
  

means you may only access the first 4 bytes of data. If you access more than that it may work now on your machine, but it can stop tomorrow with the next revision of the compiler or on another machine. If you have a struct with a variable length data block, then you can use the new C99 feature of char data[] or the old gcc extension of char data[0]. Or you can use unions. (Or in some case use the behaviour of structs with the same initial parts). If you use C code with undefined semantics then every new compiler might break it with some optimisation. There is often the tempting option of just using a slightly different code that currently works. But in the not too distant future the compiler (or even some processor) might again get some new optimisations and the code will break again. Fixing it properly might be harder but it's less likely it will fail to compile again and it also reduces the chances that it will not fail to compile but simply do something you did not expect.