Update for 06/10/2012 Well I had not written the weekly work log for the last week that is because the week was short (oh yeah thanks to all these bandhs our week got fluctuated and to be frank the week was only 4 days long) and second was my lazyness. Here goes the update * After a long discussion with Aravinda on "why productivity is reducing these days." We concluded the social network is eating up most of time. So decision was made and I closed all pinned tabs for Twitter Gmail Identi.ca and Friendica on my browser * After the above resolution I finished almost 4 chapters of Moder Perl within 2 hours!. Indeed Social networks kill the productivity. Update for 13/10/2012 I've really fallen in love with computerised bots, thanks to the wonderful KGB bot :-). So majority of my work on this week is on bots. Jabber Dictionary Redesign After thinking for a while I decided to re-write the dictionary bot which when release got an overwhelming response as see in the comments of above link. Few reason for re-write

1. Generalise the bot framework so single bot can handle multiple languages
2. Improve the data collection on bot side
3. Current code base was not very well organised and trying to add more feature it had become messy.
4. Provide XEP support to help in data collections

Few changes which are already implemented include

1. New code is now using pyxmpp2 library instead of GPLed xmppy used by current code.
2. Implemented XEP-0071 extension to properly format the meaning display by the bot
3. Current implementation was displaying all words in one set without distinguishing between adjectives,verbs proverbs etc. even though wiktionary displays meaning based on this. New implementation gives out meaning in same format as it is displayed on wiktionary.

Things remaining todo

1. Separating wiktionary parsing logic from bot code by providing some sort of intermediate interface between bot and wiktionary parsers.
2. Adding more language wiktionary parsers and teaching the bot to become multilingual :-)
3. Integrating XEP-0004 (Data Forms) for taking the meaning input from user. Current code requires user to enter data in particular format

suckless-tools fix for Wheezy Jakub Wilk suggested me to prepare a minimal version of suckless-tools for Wheezy which includes a patch to the bug #685611. And few minor changes which involve taking over the package and fixes in copyright file other than above mentioned bug. Hopefully release-team will be okay with these changes. I'm waiting for the upload to file an unblock-request. I did face a problem I was halfway through wit suckless-tools_39-1 when Jakub asked me about this change and current repository was fresh one prepared for 39-1 and didn't have history for 38 version. First I thought of preparing separate repository for 38 version which was not an correct option, but I even couldn't play with current repository. So finally I renamed current version of repository to suckless-tools-39.git on collab-maint and prepared fresh repository suckless-tools.git basing it on 38-1 version. From 39 version suckless-tools will be following 3.0 (quilt) source format and will not be working with git-buildpackage as the tool can't handle multi-tarball packages. Yes every tool involved in the package will have separate tarball from 39. More work on Bots Well today I again worked on Jabber bot, but not the dictionary bot. This time its an SMS Gateway bot for Jonas Smedegard and the coding was done in Perl. Thanks to jonas I finally could apply what I learnt in Perl. Code may not be very elegant but it works :-). And after hacking one full day in Perl now I'm not feeling very much interested to go back and hack on my own Python based dictionary bot :-). But I will anyway. Misc

• fonts-kalapi has been uploaded to unstable by Kartik finally :-). Only sad part is he couldn't use the home grown dh-make-font for this purpose.
• Finally Kartik is here in Bangalore and I'm just waiting when I can meet him.
• I ordered a RaspberryPi once it comes some hacking will be done on boxer again :-)
• Watched couple of Star Wars movie :-) (Still have 4 left)
• Expendables 2 movie is in watch list after this post.

So that's it folks. Quite longish post hope you are not bored reading it :-). Well time for movie C'ya all with next weeks log.

There may be a situation where you want to create a Debian source package without having to unpack the upstream source. One case is where you store your debian/ directory in a version control system of its own and to build your package, you pass the .dsc file to a tool like pbuilder or some remote builder. Now a modern debian source package is just a text file (the .dsc file) that references two tarballs, one being the upstream source and the other being the Debian directory. Thanks to Niels Thykier on #debian, I found the magic invocation of dpkg-source that allows you to create the .dsc directly from the two tarballs, the control file and the changelog:

> apt-get source --download-only -t experimental haskell-mtl
Reading package lists... Done
Building dependency tree       
Reading state information... Done
NOTICE: 'haskell-mtl' packaging is maintained in the 'Darcs' version control system at:
http://darcs.debian.org/darcs/pkg-haskell/haskell-mtl
Need to get 18.8 kB of source archives.
Get:1 http://http.debian.net/debian/ experimental/main haskell-mtl 2.1.2-1 (dsc) [1616 B]
Get:2 http://http.debian.net/debian/ experimental/main haskell-mtl 2.1.2-1 (tar) [13.7 kB]
Get:3 http://http.debian.net/debian/ experimental/main haskell-mtl 2.1.2-1 (diff) [3509 B]
Fetched 18.8 kB in 0s (22.3 kB/s)    
Download complete and in download only mode
> cp ./haskell-mtl_2.1.2-1.dsc ./haskell-mtl_2.1.2-1.dsc-orig
> tar xzf haskell-mtl_2.1.2-1.debian.tar.gz
> dpkg-source -cdebian/control -ldebian/changelog --format="3.0 (custom)" --target-format="3.0 (quilt)" -b / haskell-mtl_2.1.2.orig.tar.gz haskell-mtl_2.1.2-1.debian.tar.gz
dpkg-source: info: using source format  3.0 (custom)'
dpkg-source: info: building haskell-mtl in haskell-mtl_2.1.2-1.dsc
> diff  -u haskell-mtl_2.1.2-1.dsc haskell-mtl_2.1.2-1.dsc-orig
--- haskell-mtl_2.1.2-1.dsc	2012-10-13 15:09:00.031695856 +0200
+++ haskell-mtl_2.1.2-1.dsc-orig	2012-10-13 15:08:47.267696329 +0200
@@ -1,3 +1,6 @@
+-----BEGIN PGP SIGNED MESSAGE-----
+Hash: SHA1
+
 Format: 3.0 (quilt)
 Source: haskell-mtl
 Binary: libghc-mtl-dev, libghc-mtl-prof, libghc-mtl-doc
@@ -23,3 +26,11 @@
 Files: 
  943c110524d96126bfa0e61f7df1ebcd 13723 haskell-mtl_2.1.2.orig.tar.gz
  dda5ded58a8d009ecddeed68058ca787 3509 haskell-mtl_2.1.2-1.debian.tar.gz
+
+-----BEGIN PGP SIGNATURE-----
+Version: GnuPG v1.4.12 (GNU/Linux)
+
+iEYEARECAAYFAlBzRQIACgkQ9ijrk0dDIGwymwCfW87jwCv5ZvAXPBwh0LbvYLP9
+n+wAn3TCJwPRuBDEWRYmcv3Opn1Kb9+F
+=/Llc
+-----END PGP SIGNATURE-----

As you can see, the resulting .dsc file is identical to the original one, so this approach works. The parameter to -b does not matter as long as it is some existing directory. Of course you can easily shoot yourself and others in the foot, almost as easily as hand-editing the .dsc file... but if you know what you are doing, then this can be quite handy. Next to do: Write a tool that takes an unpacked Debian directory as an argument, packs it, finds the original tarball using uscan and creates the .dsc file. Would this be something that people would want in the devscripts package?

Postman was on a kind mood yesterday. My OpenPhoenux GTA04 arrived! I had my newer Neo FreeRunner upgraded via the service from Golden Delicious.

First, something rare to behold in phones nowadays - Made in Bavaria:






As a sidenote, also rare now - my Nokia N9 is one of the last phones that had this:

It's sad that's now a thing of the past for both hardware assembly and software! But that's just a hint for new companies to step in.

But back to GTA04 - a quick boot to the pre-installed Debian w/ LXDE:






And then as a shortcut unpacked and booted into QtMoko (well, it's also Debian) instead to test that phone functionality also works - and it does:



Next up: brewing something of my own!

Interactively installing Debian virtual machines with virt-install without having to download anything in advance can already be done by pointing it to a Debian mirror via --location. But you can also add files to the initrd after downloading using --initrd-inject. Upstreams intended use is for kickstart files but we can also feed it a preseed.cfg to automate the whole installation:

virt-install --connect=qemu:///system \
             --location=http://ftp.us.debian.org/debian/dists/stable/main/installer\-i386 \
             --initrd-inject=/path/to/preseed.cfg \
             --extra-args="auto" \
             --name d-i --ram=512 \
             --disk=pool=default,size=5,format=qcow2,bus=virtio

preseed.cfg is a regular preseed file (as described in the Debian Wiki) in your local filesystem. I'm using this one for Squeeze and Wheezy VMs. It must be named preseed.cfg in order for d-i to pick it up from the initrd. This also works for URLs like qemu+ssh://<remotehost>/system/ since virt-install uses libvirt's streaming API to upload the kernel and modified initrd to the remote host. In case you're running this on an i386 you'll need this fix which already sits in experimental. This blog is flattr enabled.

Software

I've been using redis for a while now. It is a fast in-memory storage system which offers persistence (unlike memcached), as well as several primitive data-types such as lists & hashes. Anyway it crossed my mind that I don't have a backup of the data it contains, so I knocked up a simple script to dump the contents in plain-text:

• redis-dump

In other software-news I've had some interesting and useful feedback and made two new releases of my slaughter sysadmin tool - it now contains a wee test suite and more robustness.

Hardware

I received an email last night to say that my Raspberry PI has shipped. Ordered 24/05/2012, and dispatched 12/10/2012 - I'd almost forgotten about it. My plan is to make it a media-serving machine, SNES emulator, or similar. Not 100% decided yet. Finally I've taken the time to repaint my office. When I last wrote about working from home I didn't include pictures - I just described the process of using a "work computer" and a "personal computer". So this is what my office used to look like. As you can see there are two machines and a huge desk. With a few changes I now have an office which looks like this - the two machines are glued-together with a KVM. and I have much more room behind it for another desk, more books, and similar toys. Additionally my dedication is now enforced - I simply cannot play with both computer as the same time. The chair was used to mount the picture - usually I sit on a kneeling chair, which is almost visible. What inspired the painting? Partly the need for more space, but mostly water damage. I had a leaking ceiling. (Local people will know all about my horrible leaking roof situation).

The end?

I ve just downloaded and installed the Debian AMD64 package of the Google Apache Pagespeed module [1]. To see if it worked I used the Google PageSpeed insights tool which gave my blog a rating of 93% (and 88% for mobile) [2]. After installing mod_pagespeed I received the same scores. So it appears that Pagespeed isn t doing any good according to Google s analysis! etbe.coker.com.au 10.11.12.13 - [13/Oct/2012:05:22:31 +0000] "GET /wp-content/plugins/openid/f/W.openid.css,qver=519.pagespeed.cf.Bbu1gxRjUE.css HTTP/1.0" 200 2165 "http://etbe.coker.com.au/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1" 0
etbe.coker.com.au 10.11.12.13 - [13/Oct/2012:05:22:31 +0000] "GET /wp-content/themes/atahualpa/js/DD_roundies.js,qver=0.0.2a.pagespeed.jm.4gw5yluag0.js HTTP/1.0" 200 3679 "http://etbe.coker.com.au/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1" 0
etbe.coker.com.au 10.11.12.13 - [13/Oct/2012:05:22:31 +0000] "GET /wp-includes/js/jquery/jquery.js,qver=1.7.2.pagespeed.jm.XZwfunyK-6.js HTTP/1.0" 200 33587 "http://etbe.coker.com.au/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1" 0
etbe.coker.com.au 10.11.12.13 - [13/Oct/2012:05:22:33 +0000] "GET /wp-content/themes/atahualpa/images/xlogo.png.pagespeed.ic.ICWmaHBME5.png HTTP/1.0" 200 2267 "http://etbe.coker.com.au/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1" 0 The above sample of web logs shows that the string pagespeed is appended to some URLs with a hash of the file contents which apparently allows much longer cache times without making it difficult to change content. So Pagespeed is obviously doing something. Is Google analysis expected to say that there is no change? Note that my speed scores are 93% and 88% so my site is was apparently quite good before mod_pagespeed was installed maybe the analysis would report a difference on a site that didn t perform so well. Now even if mod_pagespeed has given a benefit to real users but not Google tests it still means that I won t get the SEO benefits that Google apparently gives to fast sites. Also to make things even more exciting the W3.org HTML validator [3] now says that there s an error in my blog. So not only has mod_pagespeed failed to improve performance in a way that Google notices but it has also broken something!

• [1] https://developers.google.com/speed/pagespeed/mod
• [2] https://developers.google.com/speed/pagespeed/insights#url=etbe.coker.com.au&mobile=false
• [3] http://validator.w3.org/

Related posts:

1. More About Google Earth I recently wrote about problems with Google Earth [1]. In...
2. A Strange Attempt to Trick Google I just noticed that my post about LED Headlights in...
3. google-bank Currently many people have Google advertising on their web sites,...

deb http://www.coker.com.au wheezy wordpress I maintain Debian packages of a number of WordPress themes and plugins for my personal use which I am not planning to upload to Debian due to the maintenance and security issues. Generally the way things work with WordPress packages (and apparently most things in PHP) is that new versions are released whenever the author feels like it with little documentation and often now way of determining whether it s a security issue. When there is a security issue it s often fixed in a new version that includes new features giving no good option for someone who was happy with the old functionality and just wants a secure system. This isn t the way we like to do things in Debian. The result of this is that I maintain a number of packages for my personal use (and for the benefit of any interested people on the Internet) that often get new updates. I ve written the below script to create a new version of a Debian package. It searches my repository for the most recent .debian.tar.gz file for the package, applies that, runs dch -i to update the changelog, and then builds the package. So far this has only been tested on one package, I expect that I ll have to put a sed command in there to cover the case where the zip file name doesn t match what I want as the package name and I ll probably find other bugs in future, but I think it s good enough to publish now. #!/bin/bash
set -e
REPOSITORY=/home/whatever
unzip $1
FILE=$(basename $1)
PACKAGE=$(echo $FILE sed -e "s/\..*$//")
LEN=$(($(echo $PACKAGE wc -c)+1))
VER=$(echo $FILE cut -c $ LEN -200 sed -e s/.zip//)
DIRNAME=wordpress-$ PACKAGE -$ VER
mv $PACKAGE $DIRNAME
tar czf wordpress-$ PACKAGE _$ VER .orig.tar.gz $ DIRNAME
cd $DIRNAME
tar xzf $(ls -tr $ REPOSITORY /wordpress-$ PACKAGE _*.debian.tar.gz tail -1)
dch -i
dpkg-buildpackage Any suggestions for improvement will be welcome, I don t claim to be the world s greatest shell scripter. But please note that I generally aim to write shell scripts that can be understood by people who aren t experts. So if you can replace the program with a single line of Perl I will be impressed but I won t implement your solution. Related posts:

1. WordPress Maintainability For a while I ve been maintaining my own WordPress packages....
2. permalinks in wordpress, Apache redirection, and other blog stuff When I first put my new blog online I didn t...
3. WordPress Plugins I ve just added the WordPress Minify [1] plugin to my...

Thanks to a blog post by Eddy Petri or, I became aware of yet another "alternative medicine" company using legal intimidation tactics to scare off critics. According to the originating blog post about the detox "cure" ColonHelp and its producers Zenyth Pharmaceuticals actions, the producer sues Wordpress to get rid of the critical information. To check if the story was for real, I contacted Automattic, the company behind wordpress.com, and they reply was "We can confirm that Zenyth is seeking a court order against WordPress / Automattic. However, we don't believe the Terms of Service have been violated in this matter". The story seem to be simply that a blogger checked the scientific foundation for a popular health product in Rumania, ColonHelp, and reported that there was no reason at all to believe it improved the health of its users. This caused the company behind the product, Zenyth Pharmaceuticals, to use legal intimidation to try to silence the critic, instead of presenting its views and scientific foundation to argue its side. This is the usual story, and the Zenyth Pharmaceuticals company deserve everyone to know how it failed to act properly. Lets hope the Streisand effect can make it rethink its strategy. What is the harm, you might think. I suggest you take a look at a list of victims of detoxification.

When I wanted to use my dependency graph analysis tools to analyze earlier states of Debian Sid, I naturally used snapshot.debian.org to retrieve the Packages and Sources files from which my tools retrieve the dependency information. The problem is, that many of those Packages and Sources files contain syntax errors that make the dose3 parser choke. This leads to my tools being unable to parse the affected files. The following script does not only download all Packages and Sources files in a five day interval (4460 MB from 2005/03/12 to 2012/10/11) but also cleans all the syntax errors that were not parsable by dose3. This includes invalid version naming, architecture lists separated by commas, disjunctions in Conflict fields and incorrect braces/bracket usage. Maybe this helps others who also want to profit from Packages and Sources files from the past. Fun fact #1: starting from June 2010, there were no more syntax errors in the Packages and Sources files of Debian Sid. Fun fact #2: starting from December 2009, there are no more mismatches between versions of binary packages in the Packages file and the versions of the corresponding source packages in the Sources file. <script src="https://gist.github.com/3880904.js"> </script>

After 6 months of development, we are proud to release the first release of Ceilometer, the OpenStack Metering project. Ceilometer. This is a first and amazing milestone for us: we follow all other projects by releasing a version for Folsom! Using Ceilometer, you should now be able to meter your OpenStack cloud and retrieve its usage to build statistics or bill your customer! You can read our announcement on the OpenStack mailing list. Architecture We spent a good amount of time defining and refining our architecture. One of its important point, is that it has been designed to work without modifying any of the existing core components. Patching OpenStack components in an intrusive way to meter them was not an option for now, simply because we had no legitimacy to do so. This may change in the future, and this will likely be discussed next week during the OpenStack Summit. Meters Initially, we defined a bunch of meters we'd like to have for a first release, and in the end, most of them are available. Some of them are still missing, like OpenStack Object Storage (Swift) ones, mainly due to lack of interest from the involved parties so far. Anyhow, with this first release, you should be able to meter your instances, their network usage, memory, CPU. Images, networks and volumes and their CRUD operations are metered too. For more detail, you can read the complete list of implemented meters. REST API The HTTP REST API has been partially implemented. The provided methods should allow basic integration with a billing system. DreamHost is using Ceilometer in their deployment architecture and coupling it with their billing system! Towards Grizzly We don't have a clear and established road-map for Grizzly yet. We already have a couple of patches waiting in the queue to be merged, like the use of Keystone to authenticate API request and the removal of Nova DB access. On my side, these last days I've been working on a small debug user interface for the API. Ceilometer API server will return this interface if your do an API request from a browser (i.e. requesting text/html instead of application/json). I hope this will help to discover Ceilometer API more easily for new comers and leverage it to build powerful tools! Anyhow, we have tons of idea and work to do, and I'm sure the upcoming weeks will be very interesting. Also, we hope to be able to become an OpenStack incubated project soon. So stay tuned!

The Problem Some of my clients need systems that store reasonable amounts of data. This is enough data that we can expect some data corruption on disk such that traditional RAID doesn t work, that old fashioned filesystems like Ext3/4 will have unreasonable fsck fimes, and that the number of disks in a small server isn t enough. NetApp is a really good option for bulk reliable storage, but their products are very expensive. BTRFS has a lot of potential, but the currently released versions (as supported in distributions such as Debian/Wheezy) lack significant features. One significant lack in current BTRFS releases is something equivalent to the ZFS send/receive functionality for remote backups, this was a major factor when I analysed the options for hard drive based backup [1], and you should always think about backup before deploying a new system. Currently ZFS is the best choice for bulk storage which is reliable if you can t afford NetApp. Any storage system needs a minimum level of reliability if only to protect it s own metadata and a basic RAID array doesn t protect against media corruption with current data volumes. The combination of performance, lack of fsck (which is a performance feature), large storage support, backup, and significant real-world use makes ZFS a really good option. Now I need to get some servers for more than 8.1TiB of storage (the capacity of a RAID-Z array of 4*3TB disks). One of my clients needs significantly more, probably at least 10 disks in a RAID-Z array so none of the cheaper servers will do. Basically the issue that some of my clients are dealing with (and which I have to solve) is how to provide a relatively cheap ZFS system for storing reasonable amounts of data. For some systems I need to start with about 10 disks and be able to scale to 24 disks or more without excessive expense. Also to make things a little easier and cheaper 24*7 operation is not required, so instead of paying for hot-swap disks we can just schedule down-time outside business hours. The Problem with Dell Dell is really good for small systems, the PowerEdge tower servers that support 2*3.5 or 4*3.5 disks and which have space for an SSD or two are really affordable and easy to order. But even in the mid-size Dell tower servers (which are small by server standards) you have problems with just getting a few disks operating outside a RAID array [2]. The Dell online store is really great for small servers, any time I m buying a server for less than $2500 I check the Dell online store first and usually their price is good enough that there is no need to get a quote from another company. Unfortunately all the servers with bigger storage involve disks that are unreasonably expensive (it seems that Dell makes their profit on the parts) and prices are not available online. I gave my email address and phone number to the Dell web site on Wednesday and they haven t cared to get back to me yet. This is the type of service that makes me avoid IBM and HP for any server deployment where the Dell online store sells something suitable! BackBlaze For some time BackBlaze have been getting interest by describing how they store lots of data in a small amount of space by tightly stacking SATA disks. They don t think that ZFS on Linux is ready for production, but their hardware ideas are useful. They have recently described their latest architecture [3]. They describe it as 135TB for $7,384. Of course the 135TB number is based on the idea of getting the full 3TB capacity out of each disk which they can do as they have redundancy over multiple storage pods. But anyone who wants a single fileserver needs some internal redundancy to cover disk failure. One option might be to have three RAID-Z2 arrays of 15 disks which gives a usable capacity of 42*3TB==126TB==113TiB. Note that while the ZFS documentation recommends between 3 and 9 disks per zpool for performance I don t expect performance problems, when you only have a gigabit Ethernet connection there shouldn t be a problem with three ZFS zpools making the network the bottleneck. For this option the way to go would be to start with an array of 15 disks and then buy a second set of 15 disks when the first storage pool becomes full. It seems likely that 4TB disks will become cheap before a 35TiB array is filled so we can get more efficiency by delaying purchases. The BackBlaze pod isn t cheap, they are sold as a complete system without storage disks for $US5,395 by Protocase [4]. That gives a markup of $US3,411 over the BackBlaze cost which isn t too bad given that BackBlaze are quoting the insane bulk discount hardware prices that I could never get. Protocase also offer the case on it s own for anyone who wants to build a system around it. It seems like the better option is to buy the system from Protocase, but that would end up being over $6,000 when Australian import duty is added and probably close to $7,000 when shipping etc is included. Norco Norco offers a case that takes 24 hot-swap SATA/SAS disks and a regular PC motherboard for $US399 [5]. It s similar to the BackBlaze pod but smaller, cheaper, and there s no obvious option to buy a configured and tested system. 24 disks would allow two RAID-Z2 arrays of 12 disks, the first array could provide 27TiB and the second array could provide something bigger when new disks are released. SuperMicro SuperMicro has a range of storage servers that support from 12 to 36 disks [6]. They seem good, but I d have to deal with a reseller to buy them which would involve pain at best and at worst they wouldn t bother getting me a quote because I only want one server at a time. Conclusion Does anyone know of any other options for affordable systems suitable for running ZFS on SATA disks? Preferably ones that don t involve dealing with resellers. At the moment it seems that the best option is to get a Norco case and build my own system as I don t think that any of my clients needs the capacity of a BackBlaze pod at the moment. Supermicro seems good but I d have to deal with a reseller. In my experience the difference between the resellers of such computer systems and used car dealers is that used car dealers are happy to sell one car at a time and that every used car dealer at least knows how to drive. Also if you are an Australian reader of my blog and you want to build such storage servers to sell to my clients in Melbourne then I d be interested to see an offer. But please make sure that any such offer includes a reference to your contributions to the Linux community if you think I won t recognise your name. If you don t contribute then I probably don t want to do business with you. As an aside, I was recently at a camera store helping a client test a new DSLR when one of the store employees started telling me how good ZFS is for storing RAW images. I totally agree that ZFS is the best filesystem for storing large RAW files and this is what I am working on right now. But it s not the sort of advice I expect to receive at a camera store, not even one that caters to professional photographers.

• [1] http://etbe.coker.com.au/2012/08/08/hard-drives-backup/
• [2] http://etbe.coker.com.au/2012/08/01/cheap-sata-poweredge-t410/
• [3] http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/
• [4] http://protocase.com/products/index.php?e=Backblaze
• [5] http://www.norcotek.com/item_detail.php?categoryid=1&modelno=rpc-4224
• [6] http://www.supermicro.com/products/nfo/storage.cfm

Related posts:

1. Cheap SATA Disks in a Dell PowerEdge T410 A non-profit organisation I support has just bought a Dell...
2. ZFS vs BTRFS on Cheap Dell Servers I previously wrote about my first experiences with BTRFS [1]....
3. Flash Storage and Servers In the comments on my post about the Dell PowerEdge...

I've been continuing to work on Shim this week, and it's now getting pretty close to feature complete. The biggest change has been migrating the MokList variable from a custom format to the one used by the UEFI spec databases, which means it's easier for the kernel to import the MOK entries and use them for validating module signatures.

The other benefit of this format is that it supports hashes as well as certificates, and so I've added support for enrolling hashes through the MokManager UI. This means that distributions can now ship with a signed copy of Shim without having to sign any other components of their distribution. When the user attempts to boot off the media they'll be faced with a menu and a countdown. If the countdown reaches 0, the system will simply fall back to the next entry in the bootlist. If they hit a key, they can choose to enrol a hash. The user then navigates the filesystem explorer, chooses the bootloader, confirms that they want to enrol it and then exits. Shim then verifies the bootloader against the hash and boots successfully.

The big advantage of this over the Linux Foundation approach is that once a hash has been enrolled the need for physical end-user presence is removed - ie, if you enrol the hash, you don't need to hit a key every time you boot. This is still slightly sub-optimal in that if you update your bootloader you'll need to enrol a new hash, but that can be partially automated by calling MokUtil in the postinst - the user then simply needs to confirm that they want to enrol the hash, rather than having to choose it manually. Completely transparent updates are going to require a signed bootloader and an enrolled signing key.

A couple of people have asked whether we're planning on implementing the Linux Foundation approach of simply asking the user whether they want to boot an unsigned file. We've considered it, but at the moment are leaning towards "no" - it's simply too easy to use to trick naive users into running untrusted code. Users are trained to click through pretty much any security prompt that they see, and if an attacker replaces a legitimate bootloader with one that asks them to press "y" to make their computer work, they'll press "y". If that bootloader then launches a trojaned Windows bootloader that launches a trojaned Windows kernel, that's kind of a problem. This could be somewhat mitigated by limiting this feature to removable media, and we're seriously considering that, but there are still some risks associated. We might just end up writing the code but disabling it at build time, and then anyone who wants to distribute with that policy can do so at their own risk.

Meanwhile, Peter Jones is working on tidying up the code we're using for the actual signing and will be publishing that once the last couple of kinks are worked out. We're using hardware cryptography, so even if someone compromises the build systems they won't be able to obtain the private key that we use. However, should something disastrous and unanticipated happen, we do have a plan in place for migrating to new keys with minimal user impact. We'll document the code and infrastructure we're using in order to make it as easy as possible for other distributions to implement equivalent functionality.

As I've mentioned before, our goal is to make it as easy as possible for distributions to implement whatever level of Secure Boot policy they want without having to engage with Microsoft themselves. Shim allows distributions to ship an OS that has no signed binaries at all, or alternatively to ship an OS that uses filesystem-level cryptography to ensure that even userspace is completely signed. I'm expecting to see a range of options available, and I hope that the majority of users will find something to suit their needs.

comments

It has been brought to my attention that a company selling some so-called colon cleansing product wanted to threat with a law suit a Romanian skeptical blogger because he wrote some articles showing that any such products (the one produced by the said company is the most known/popular in Romania) are pure quackery and there is no scientific basis for the claim they make in order to promote their products.

In his articles he also explained how, in fact, the mucoid plaque, the thing that supposedly proves the efficiency of the product, it is a result of taking the product due to its ingredients, and how no such mucoid plaque was ever observed in any colonoscopy, colon surgery or any other situation where you'd expect it to be seen. He also quoted specialists and lots of other scientific references, showing an honest approach to the issue.

As a response to the initial take-down message from the company doing business with people's crap, the blogger said would like to see scientific proof for the claims made for the product, and when that was to happen, he would take down the articles and publish a correction.

The company decided that the best way to continue this was to try to make a legal threat and ask 100.000 euros (one hundred thousands Euros) as damage in a country where, according to the latest data from the National Statistics Institute, the total average monthly personal income is about 180 Euros.

The blogger, as a reply, decided the threat should be made public and wrote another article which probably made the company very unhappy, because they decided to sue Wordpress so they would take down the blog.

And that's exactly what they did, they sued Wordpress, and sent some documents to Wordpress who sent them to the blogger. Among the documents there were 4 pdf files containing each an original article (in Romanian) from the blog and only 3 pdf containg English translation for only 3 of them. The one missing was the one where the blogger himself showed there wasn't any legal basis for the threats they made initially against him.

Here are the translations (ironically, made on the company's own expense):

Initial article entitled "ColonHelp doesn't help the colon. But it empties your wallet!" (original here)
Initial Article.en<iframe class="scribd_iframe_embed" data-aspect-ratio="0.701030927835051" data-auto-height="true" frameborder="0" height="600" id="doc_83859" scrolling="no" src="http://www.scribd.com/embeds/109756494/content?start_page=1&amp;view_mode=scroll&amp;access_key=key-19l4sbejfk4l4na4feoe" width="100%"></iframe>



The second article entitled "Again about ColonHelp and intestinal cleansers" (original here)
Follow Up.en<iframe class="scribd_iframe_embed" data-aspect-ratio="0.70264064293915" data-auto-height="true" frameborder="0" height="600" id="doc_46722" scrolling="no" src="http://www.scribd.com/embeds/109757646/content?start_page=1&amp;view_mode=scroll&amp;access_key=key-1d0z2779liaialmv90r5" width="100%"></iframe>



** Missing translation of the first reply to threats (original Romanian text here)



The second article about the threats entitled "People who clean the colon have filled the fan with shit" (original here)
Threats 2<iframe class="scribd_iframe_embed" data-aspect-ratio="0.705069124423963" data-auto-height="true" frameborder="0" height="600" id="doc_72956" scrolling="no" src="http://www.scribd.com/embeds/109758037/content?start_page=1&amp;view_mode=scroll&amp;access_key=key-2d3c3eq1d6lo313fogho" width="100%"></iframe>


The Romanian blogger explains himself more of the details on this issue in his latest article on his blog.

The company is called Zenyth Pharmaceuticals and Wordpress will probably lose the lawsuit by not presenting themselves in any way in the Romanian courts, but I think some Streisand effect would really help the asses of this company to get them kicked in their rightful place, at the top of the hall of shame.

The product name is called ColonHelp.

Please spread this information as wide as possible.
Do NOT link to the company's site (it would raise its search engine rank), but link to the blogger' article or the translations.


If any Romanian speaker cares to translate the untranslated article and publish it somewhere on the web, I would be more than glad to update this article and add a link to that translation.

For the past weeks I have been working on implementing RFC 6030, also known as Portable Symmetric Key Container (PSKC). So what is PSKC? The Portable Symmetric Key Container (PSKC) format is used to transport and provision symmetric keys to cryptographic devices or software. My PSKC Library allows you to parse, validate and generate PSKC data. The PSKC Library is written in C, uses LibXML, and is licensed under LGPLv2+. In practice, PSKC is most commonly used to transport secret keys for OATH HOTP/TOTP devices (and other OTP devices) between the personalization machine and the OTP validation server. Yesterday I released version 2.0.0 of OATH Toolkit with the new PSKC Library. See my earlier introduction to OATH Toolkit for background. OATH Toolkit is packaged for Debian/Ubuntu and I hope to refresh the package to include libpskc/pskctool soon. To get a feeling for the PSKC data format, consider the most minimal valid PSKC data:

<?xml version="1.0"?>
<KeyContainer xmlns="urn:ietf:params:xml:ns:keyprov:pskc" Version="1.0">
  <KeyPackage/>
</KeyContainer>

The library can easily be used to export PSKC data into a comma-separated value (CSV) format, in fact the PSKC library tutorial concludes with that as an example. There is complete API documentation for the library. The command line tool is more useful for end-users and allows you to parse and inspect PSKC data. Below is an illustration of how you would use it to parse some PSKC data, first we show the content of a file pskc-figure2.xml :

<?xml version="1.0" encoding="UTF-8"?>
<KeyContainer Version="1.0"
	      Id="exampleID1"
	      xmlns="urn:ietf:params:xml:ns:keyprov:pskc">
  <KeyPackage>
    <Key Id="12345678"
         Algorithm="urn:ietf:params:xml:ns:keyprov:pskc:hotp">
      <Issuer>Issuer-A</Issuer>
      <Data>
        <Secret>
          <PlainValue>MTIzNA==
          </PlainValue>
        </Secret>
      </Data>
    </Key>
  </KeyPackage>
</KeyContainer>

Here is how you would parse and pretty print that PSKC data:

jas@latte:~$ pskctool -c pskc-figure2.xml 
Portable Symmetric Key Container (PSKC):
	Version: 1.0
	Id: exampleID1
	KeyPackage 0:
		DeviceInfo:
		Key:
			Id: 12345678
			Issuer: Issuer-A
			Algorithm: urn:ietf:params:xml:ns:keyprov:pskc:hotp
			Key Secret (base64): MTIzNA==
jas@latte:~$

For more information, see the OATH Toolkit website and the PSKC Library Manual.

I found it surprisingly hard to determine in tearDown() whether or not the test that currently ran succeeded or not. I am writing some tests for gnome-settings-daemon and want to show the log output of the daemon if a test failed. I now cobbled together the following hack, but I wonder if there s a more elegant way? The interwebs don t seem to have a good solution for this either.

    def tearDown(self):
        [...]
        # collect log, run() shows it on failures
        with open(self.daemon_log.name) as f:
            self.log_output = f.read()
    def run(self, result=None):
        '''Show log output on failed tests'''
        if result:
            orig_err_fail = result.errors + result.failures
        super().run(result)
        if result and result.errors + result.failures > orig_err_fail:
            print('\n----- daemon log -----\n%s\n------\n' % self.log_output)

As many of you know, I love Ubuntu. I ve given years of my life for Ubuntu. I ve served as much as anyone s expected to, and long upheld the ideology and technical motivations behind Ubuntu as second to none. Until recently. Now, with advertisements in the default DE (and don t tell me they re not, when I search for term and get The Terminator on DVD, it s a ad if you care so much about whatever binding to search applications, why not disable ads by default, and let super-s do your shopping ), and now, begging for a handout when people Download Ubuntu without an option to donate to Debian, which composes over 70% of the Desktop, I get upset. I feel cheated, and I m not even involved on a day-to-day anymore. I m ashamed of what s going on now, and I hope we find a better way to serve our users.

Automatically devising a build order that allows to bootstrap Debian, currently fails (amongst other reasons) because of the lack of metadata information about which build dependencies can potentially be dropped from source packages. If that information was available, an algorithm could decide which build dependencies to drop so that dependency cycles can be broken. Finding droppable build dependencies of a source package is something only humans can do. This is because it involves to manually analyze and test the build system of a source package. Build systems are neither uniform nor do they encode their dependencies in a way that can directly be mapped to Debian packages. Therefor they are not machine readable. One idea to solve the dilemma, is to find a Linux distribution that provides the following:

• allows to do "profile builds" of its source packages with different features enabled or disabled
• stores information about which feature requires which build dependency
• stores everything in a format that can be parsed and analyzed
• covers a similar range of software packages as Debian does

If such a distribution can be found then the information from it can be used to find dependencies that can also be dropped from Debian source packages. Gentoo is a distribution that fulfills above requirements through so called USE flags that allow to enable or disable features during compilation. Dependencies of Gentoo source packages are stored in .ebuild files that control the build process. Since .ebuild files are bash scripts, parsing them is not trivial. I therefor used the emerge software package to extract that information. Thanks to the well written emerge code and to quick help in the Gentoo IRC channel, it didnt take long to make the code run on Debian. My sourcecode is downloadable here: https://gitorious.org/debian-bootstrap/gen2deb Before I list the results of using Gentoo USE flags to determine dependencies that can potentially be dropped from Debian source packages, let me list the problems that this method entails.

Only package name matching, no version matching When writing the mapping from Debian to Gentoo packages and back I discard version information. There are just too many versions that either Debian or Gentoo have and are not present in the other. So the assumption is, that Debian Sid and Gentoo have both the most recent major versions of upstream software which has roughly the same requirements in terms of build dependencies.

Gentoo packages are matched to Debian source packages In Gentoo there are only source packages and no binary packages. So I map Gentoo packages to Debian source packages. But Gentoo source packages build depend on other source packages while Debian source packages depend on binary packages. So at some point I have to translate Gentoo packages to Debian source packages and those source packages to Debian binary packages. I do this by analyzing the original binary package build dependencies of a Debian source package and then filter out those binary packages as being droppable that are built by the Debian source packages that were found to be droppable.

Not the exact same package set There is some software that is only in Gentoo and some that is only in Debian. Debian and Gentoo also split some source packages differently.

Gentoo has more direct dependencies Many build dependencies in Debian are indirectly pulled in through dependencies of direct build dependencies. In Gentoo source packages directly depend on most things they need to build successfully. This leads to the list of dependencies in Gentoo to be much larger than the list of dependencies in the corresponding Debian source package. It also means that lots of dependencies that can be dropped in Gentoo are not found to be droppable in Debian because they are not direct dependencies of that source package.

There are no implicit dependencies Gentoo will often drop dependencies that are essential or build-essential packages in Debian and are therefor implicit build dependencies that cannot be dropped.

Result Despite the many problems, the result doesnt look too wrong. I got some Debian source packages that were found to have droppable build dependencies from Thorsten Glaser and all dependencies that Gentoo found to be droppable were also dropped by him. To put everything into numbers: the current 912 nodes big SCC in Debian Sid can be reduced to 6 individual SCC with 422, 5, 5, 3, 2 and 2 nodes each. So using Gentoo cuts the size of the central component to more than half. Surely, there will be a number of dependencies that were found to be droppable in Gentoo but are actually not droppable in Debian. The point is, that it is better to have "some" data even if it contains false positives than no data at all. It is easier for a human to verify if some suggested droppable build dependencies are actually correct than going through hundreds of source packages with thousands of dependencies manually.

James Bottomley just published a description of the Linux Foundation's Secure Boot plan, which is pretty much as I outlined in the second point here - it's a bootloader that will boot untrusted images as long as a physically present end-user hits a key on every boot, and if a user switches their machine to setup mode it'll enrol the hash of the bootloader in order to avoid prompting again. In other words, it's less useful than shim. Just use shim instead.

comments

Last weekend I was in Southwest Harbour, Maine again for the annual LTSP hackfest (called By The Sea). It s a fun and productive event and as always it s been good catching up with LTSP folk, even though we were missing Oliver, Alkis and Vagrant. Here is a summary of what I can recall from the discussions of the weekend Recent Happenings in LTSP

• New LTSP Website After a really long time, LTSP finally has a new website. It s a big improvement, there s a new success stories page and the wiki is now self-hosted.
• LTSP PNP - This is a new package (ltsp-pnp) that can safely be installed on existing machines to allow a user to log into an LTSP server using LDM.

Going Forward Social

• On Friday, Chuck Liebow took us out for a boat ride around the harbour on the Sea Princess

• On Saturday night we had the big lobster dinner as per BTS tradition (I had steak since I m not a big seafood person). We were to busy eating and telling stories to be taking any pictures
• And last but certainly not least, I finally met Eric Harrisson, who did a lot of work on K12-LTSP in the Portland schools district. He has an awesome hobby where he builds guitars out of well, almost anything he can find.
  He brought along one of the experimental guitars he slapped together recently and told me all about how it s put together. Even more amazing, he said I could have it! I was planning on getting an acoustic guitar anyway so I m very thrilled about it. Not only is it a completely unique guitar but it sounds great too. I m going to have to think of something to make back for him!

It was great seeing everyone again. I first started using LTSP around 9 years ago and never imagined back then that I d get to meet the people behind it. Ron Colcernian sourced us some really cool LTSP tops that you can see us wearing in the group photo. I hope to get to BTS again next year!

Christian's most recent blog post got me wondering if the decline in the bug reporting rate in Debian was something new, or something which often happened during releases. So, lets try to figure that out. In the BTS, when a bug report is filed, the report is written to a file called bugnum.report, and then not touched from then on. Let's look at the modification date on that file to see when each bug was filed; and since we're going to plot this, lets only look at bugs ending in 00:

stat -c '%n %Y' /srv/bugs.debian.org/spool/ archive,db-h /00/*.report > ~/reporting_rate.txt

Now, lets get the data into R and plot it. [For clarity, I'm not showing the R code, but it's available in the source code for this post.] From the plot (Bugs reported per second over time with a red loess fit line), it looks like we do see a decline during certain periods. However, there's an even more alarming trend of a decrease in bug reporting in Debian which has been happening since 2006. (Note that I've truncated the y scale significantly; there are periods in Debian where the bug rate is astronomically high, usually corresponding to mass bug filings; I've also limited the plot to data from 2003 on, as I have to clean up that data significantly before I can plot it like this.) Not sure exactly what that means, but it is troubling.