The Problem Some of my clients need systems that store reasonable amounts of data. This is enough data that we can expect some data corruption on disk such that traditional RAID doesn t work, that old fashioned filesystems like Ext3/4 will have unreasonable fsck fimes, and that the number of disks in a small server isn t enough. NetApp is a really good option for bulk reliable storage, but their products are very expensive. BTRFS has a lot of potential, but the currently released versions (as supported in distributions such as Debian/Wheezy) lack significant features. One significant lack in current BTRFS releases is something equivalent to the ZFS send/receive functionality for remote backups, this was a major factor when I analysed the options for hard drive based backup [1], and you should always think about backup before deploying a new system. Currently ZFS is the best choice for bulk storage which is reliable if you can t afford NetApp. Any storage system needs a minimum level of reliability if only to protect it s own metadata and a basic RAID array doesn t protect against media corruption with current data volumes. The combination of performance, lack of fsck (which is a performance feature), large storage support, backup, and significant real-world use makes ZFS a really good option. Now I need to get some servers for more than 8.1TiB of storage (the capacity of a RAID-Z array of 4*3TB disks). One of my clients needs significantly more, probably at least 10 disks in a RAID-Z array so none of the cheaper servers will do. Basically the issue that some of my clients are dealing with (and which I have to solve) is how to provide a relatively cheap ZFS system for storing reasonable amounts of data. For some systems I need to start with about 10 disks and be able to scale to 24 disks or more without excessive expense. Also to make things a little easier and cheaper 24*7 operation is not required, so instead of paying for hot-swap disks we can just schedule down-time outside business hours. The Problem with Dell Dell is really good for small systems, the PowerEdge tower servers that support 2*3.5 or 4*3.5 disks and which have space for an SSD or two are really affordable and easy to order. But even in the mid-size Dell tower servers (which are small by server standards) you have problems with just getting a few disks operating outside a RAID array [2]. The Dell online store is really great for small servers, any time I m buying a server for less than $2500 I check the Dell online store first and usually their price is good enough that there is no need to get a quote from another company. Unfortunately all the servers with bigger storage involve disks that are unreasonably expensive (it seems that Dell makes their profit on the parts) and prices are not available online. I gave my email address and phone number to the Dell web site on Wednesday and they haven t cared to get back to me yet. This is the type of service that makes me avoid IBM and HP for any server deployment where the Dell online store sells something suitable! BackBlaze For some time BackBlaze have been getting interest by describing how they store lots of data in a small amount of space by tightly stacking SATA disks. They don t think that ZFS on Linux is ready for production, but their hardware ideas are useful. They have recently described their latest architecture [3]. They describe it as 135TB for $7,384. Of course the 135TB number is based on the idea of getting the full 3TB capacity out of each disk which they can do as they have redundancy over multiple storage pods. But anyone who wants a single fileserver needs some internal redundancy to cover disk failure. One option might be to have three RAID-Z2 arrays of 15 disks which gives a usable capacity of 42*3TB==126TB==113TiB. Note that while the ZFS documentation recommends between 3 and 9 disks per zpool for performance I don t expect performance problems, when you only have a gigabit Ethernet connection there shouldn t be a problem with three ZFS zpools making the network the bottleneck. For this option the way to go would be to start with an array of 15 disks and then buy a second set of 15 disks when the first storage pool becomes full. It seems likely that 4TB disks will become cheap before a 35TiB array is filled so we can get more efficiency by delaying purchases. The BackBlaze pod isn t cheap, they are sold as a complete system without storage disks for $US5,395 by Protocase [4]. That gives a markup of $US3,411 over the BackBlaze cost which isn t too bad given that BackBlaze are quoting the insane bulk discount hardware prices that I could never get. Protocase also offer the case on it s own for anyone who wants to build a system around it. It seems like the better option is to buy the system from Protocase, but that would end up being over $6,000 when Australian import duty is added and probably close to $7,000 when shipping etc is included. Norco Norco offers a case that takes 24 hot-swap SATA/SAS disks and a regular PC motherboard for $US399 [5]. It s similar to the BackBlaze pod but smaller, cheaper, and there s no obvious option to buy a configured and tested system. 24 disks would allow two RAID-Z2 arrays of 12 disks, the first array could provide 27TiB and the second array could provide something bigger when new disks are released. SuperMicro SuperMicro has a range of storage servers that support from 12 to 36 disks [6]. They seem good, but I d have to deal with a reseller to buy them which would involve pain at best and at worst they wouldn t bother getting me a quote because I only want one server at a time. Conclusion Does anyone know of any other options for affordable systems suitable for running ZFS on SATA disks? Preferably ones that don t involve dealing with resellers. At the moment it seems that the best option is to get a Norco case and build my own system as I don t think that any of my clients needs the capacity of a BackBlaze pod at the moment. Supermicro seems good but I d have to deal with a reseller. In my experience the difference between the resellers of such computer systems and used car dealers is that used car dealers are happy to sell one car at a time and that every used car dealer at least knows how to drive. Also if you are an Australian reader of my blog and you want to build such storage servers to sell to my clients in Melbourne then I d be interested to see an offer. But please make sure that any such offer includes a reference to your contributions to the Linux community if you think I won t recognise your name. If you don t contribute then I probably don t want to do business with you. As an aside, I was recently at a camera store helping a client test a new DSLR when one of the store employees started telling me how good ZFS is for storing RAW images. I totally agree that ZFS is the best filesystem for storing large RAW files and this is what I am working on right now. But it s not the sort of advice I expect to receive at a camera store, not even one that caters to professional photographers.

• [1] http://etbe.coker.com.au/2012/08/08/hard-drives-backup/
• [2] http://etbe.coker.com.au/2012/08/01/cheap-sata-poweredge-t410/
• [3] http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/
• [4] http://protocase.com/products/index.php?e=Backblaze
• [5] http://www.norcotek.com/item_detail.php?categoryid=1&modelno=rpc-4224
• [6] http://www.supermicro.com/products/nfo/storage.cfm

Related posts:

1. Cheap SATA Disks in a Dell PowerEdge T410 A non-profit organisation I support has just bought a Dell...
2. ZFS vs BTRFS on Cheap Dell Servers I previously wrote about my first experiences with BTRFS [1]....
3. Flash Storage and Servers In the comments on my post about the Dell PowerEdge...

I've been continuing to work on Shim this week, and it's now getting pretty close to feature complete. The biggest change has been migrating the MokList variable from a custom format to the one used by the UEFI spec databases, which means it's easier for the kernel to import the MOK entries and use them for validating module signatures.

The other benefit of this format is that it supports hashes as well as certificates, and so I've added support for enrolling hashes through the MokManager UI. This means that distributions can now ship with a signed copy of Shim without having to sign any other components of their distribution. When the user attempts to boot off the media they'll be faced with a menu and a countdown. If the countdown reaches 0, the system will simply fall back to the next entry in the bootlist. If they hit a key, they can choose to enrol a hash. The user then navigates the filesystem explorer, chooses the bootloader, confirms that they want to enrol it and then exits. Shim then verifies the bootloader against the hash and boots successfully.

The big advantage of this over the Linux Foundation approach is that once a hash has been enrolled the need for physical end-user presence is removed - ie, if you enrol the hash, you don't need to hit a key every time you boot. This is still slightly sub-optimal in that if you update your bootloader you'll need to enrol a new hash, but that can be partially automated by calling MokUtil in the postinst - the user then simply needs to confirm that they want to enrol the hash, rather than having to choose it manually. Completely transparent updates are going to require a signed bootloader and an enrolled signing key.

A couple of people have asked whether we're planning on implementing the Linux Foundation approach of simply asking the user whether they want to boot an unsigned file. We've considered it, but at the moment are leaning towards "no" - it's simply too easy to use to trick naive users into running untrusted code. Users are trained to click through pretty much any security prompt that they see, and if an attacker replaces a legitimate bootloader with one that asks them to press "y" to make their computer work, they'll press "y". If that bootloader then launches a trojaned Windows bootloader that launches a trojaned Windows kernel, that's kind of a problem. This could be somewhat mitigated by limiting this feature to removable media, and we're seriously considering that, but there are still some risks associated. We might just end up writing the code but disabling it at build time, and then anyone who wants to distribute with that policy can do so at their own risk.

Meanwhile, Peter Jones is working on tidying up the code we're using for the actual signing and will be publishing that once the last couple of kinks are worked out. We're using hardware cryptography, so even if someone compromises the build systems they won't be able to obtain the private key that we use. However, should something disastrous and unanticipated happen, we do have a plan in place for migrating to new keys with minimal user impact. We'll document the code and infrastructure we're using in order to make it as easy as possible for other distributions to implement equivalent functionality.

As I've mentioned before, our goal is to make it as easy as possible for distributions to implement whatever level of Secure Boot policy they want without having to engage with Microsoft themselves. Shim allows distributions to ship an OS that has no signed binaries at all, or alternatively to ship an OS that uses filesystem-level cryptography to ensure that even userspace is completely signed. I'm expecting to see a range of options available, and I hope that the majority of users will find something to suit their needs.

comments

It has been brought to my attention that a company selling some so-called colon cleansing product wanted to threat with a law suit a Romanian skeptical blogger because he wrote some articles showing that any such products (the one produced by the said company is the most known/popular in Romania) are pure quackery and there is no scientific basis for the claim they make in order to promote their products.

In his articles he also explained how, in fact, the mucoid plaque, the thing that supposedly proves the efficiency of the product, it is a result of taking the product due to its ingredients, and how no such mucoid plaque was ever observed in any colonoscopy, colon surgery or any other situation where you'd expect it to be seen. He also quoted specialists and lots of other scientific references, showing an honest approach to the issue.

As a response to the initial take-down message from the company doing business with people's crap, the blogger said would like to see scientific proof for the claims made for the product, and when that was to happen, he would take down the articles and publish a correction.

The company decided that the best way to continue this was to try to make a legal threat and ask 100.000 euros (one hundred thousands Euros) as damage in a country where, according to the latest data from the National Statistics Institute, the total average monthly personal income is about 180 Euros.

The blogger, as a reply, decided the threat should be made public and wrote another article which probably made the company very unhappy, because they decided to sue Wordpress so they would take down the blog.

And that's exactly what they did, they sued Wordpress, and sent some documents to Wordpress who sent them to the blogger. Among the documents there were 4 pdf files containing each an original article (in Romanian) from the blog and only 3 pdf containg English translation for only 3 of them. The one missing was the one where the blogger himself showed there wasn't any legal basis for the threats they made initially against him.

Here are the translations (ironically, made on the company's own expense):

Initial article entitled "ColonHelp doesn't help the colon. But it empties your wallet!" (original here)
Initial Article.en<iframe class="scribd_iframe_embed" data-aspect-ratio="0.701030927835051" data-auto-height="true" frameborder="0" height="600" id="doc_83859" scrolling="no" src="http://www.scribd.com/embeds/109756494/content?start_page=1&amp;view_mode=scroll&amp;access_key=key-19l4sbejfk4l4na4feoe" width="100%"></iframe>



The second article entitled "Again about ColonHelp and intestinal cleansers" (original here)
Follow Up.en<iframe class="scribd_iframe_embed" data-aspect-ratio="0.70264064293915" data-auto-height="true" frameborder="0" height="600" id="doc_46722" scrolling="no" src="http://www.scribd.com/embeds/109757646/content?start_page=1&amp;view_mode=scroll&amp;access_key=key-1d0z2779liaialmv90r5" width="100%"></iframe>



** Missing translation of the first reply to threats (original Romanian text here)



The second article about the threats entitled "People who clean the colon have filled the fan with shit" (original here)
Threats 2<iframe class="scribd_iframe_embed" data-aspect-ratio="0.705069124423963" data-auto-height="true" frameborder="0" height="600" id="doc_72956" scrolling="no" src="http://www.scribd.com/embeds/109758037/content?start_page=1&amp;view_mode=scroll&amp;access_key=key-2d3c3eq1d6lo313fogho" width="100%"></iframe>


The Romanian blogger explains himself more of the details on this issue in his latest article on his blog.

The company is called Zenyth Pharmaceuticals and Wordpress will probably lose the lawsuit by not presenting themselves in any way in the Romanian courts, but I think some Streisand effect would really help the asses of this company to get them kicked in their rightful place, at the top of the hall of shame.

The product name is called ColonHelp.

Please spread this information as wide as possible.
Do NOT link to the company's site (it would raise its search engine rank), but link to the blogger' article or the translations.


If any Romanian speaker cares to translate the untranslated article and publish it somewhere on the web, I would be more than glad to update this article and add a link to that translation.

For the past weeks I have been working on implementing RFC 6030, also known as Portable Symmetric Key Container (PSKC). So what is PSKC? The Portable Symmetric Key Container (PSKC) format is used to transport and provision symmetric keys to cryptographic devices or software. My PSKC Library allows you to parse, validate and generate PSKC data. The PSKC Library is written in C, uses LibXML, and is licensed under LGPLv2+. In practice, PSKC is most commonly used to transport secret keys for OATH HOTP/TOTP devices (and other OTP devices) between the personalization machine and the OTP validation server. Yesterday I released version 2.0.0 of OATH Toolkit with the new PSKC Library. See my earlier introduction to OATH Toolkit for background. OATH Toolkit is packaged for Debian/Ubuntu and I hope to refresh the package to include libpskc/pskctool soon. To get a feeling for the PSKC data format, consider the most minimal valid PSKC data:

<?xml version="1.0"?>
<KeyContainer xmlns="urn:ietf:params:xml:ns:keyprov:pskc" Version="1.0">
  <KeyPackage/>
</KeyContainer>

The library can easily be used to export PSKC data into a comma-separated value (CSV) format, in fact the PSKC library tutorial concludes with that as an example. There is complete API documentation for the library. The command line tool is more useful for end-users and allows you to parse and inspect PSKC data. Below is an illustration of how you would use it to parse some PSKC data, first we show the content of a file pskc-figure2.xml :

<?xml version="1.0" encoding="UTF-8"?>
<KeyContainer Version="1.0"
	      Id="exampleID1"
	      xmlns="urn:ietf:params:xml:ns:keyprov:pskc">
  <KeyPackage>
    <Key Id="12345678"
         Algorithm="urn:ietf:params:xml:ns:keyprov:pskc:hotp">
      <Issuer>Issuer-A</Issuer>
      <Data>
        <Secret>
          <PlainValue>MTIzNA==
          </PlainValue>
        </Secret>
      </Data>
    </Key>
  </KeyPackage>
</KeyContainer>

Here is how you would parse and pretty print that PSKC data:

jas@latte:~$ pskctool -c pskc-figure2.xml 
Portable Symmetric Key Container (PSKC):
	Version: 1.0
	Id: exampleID1
	KeyPackage 0:
		DeviceInfo:
		Key:
			Id: 12345678
			Issuer: Issuer-A
			Algorithm: urn:ietf:params:xml:ns:keyprov:pskc:hotp
			Key Secret (base64): MTIzNA==
jas@latte:~$

For more information, see the OATH Toolkit website and the PSKC Library Manual.

As many of you know, I love Ubuntu. I ve given years of my life for Ubuntu. I ve served as much as anyone s expected to, and long upheld the ideology and technical motivations behind Ubuntu as second to none. Until recently. Now, with advertisements in the default DE (and don t tell me they re not, when I search for term and get The Terminator on DVD, it s a ad if you care so much about whatever binding to search applications, why not disable ads by default, and let super-s do your shopping ), and now, begging for a handout when people Download Ubuntu without an option to donate to Debian, which composes over 70% of the Desktop, I get upset. I feel cheated, and I m not even involved on a day-to-day anymore. I m ashamed of what s going on now, and I hope we find a better way to serve our users.

Automatically devising a build order that allows to bootstrap Debian, currently fails (amongst other reasons) because of the lack of metadata information about which build dependencies can potentially be dropped from source packages. If that information was available, an algorithm could decide which build dependencies to drop so that dependency cycles can be broken. Finding droppable build dependencies of a source package is something only humans can do. This is because it involves to manually analyze and test the build system of a source package. Build systems are neither uniform nor do they encode their dependencies in a way that can directly be mapped to Debian packages. Therefor they are not machine readable. One idea to solve the dilemma, is to find a Linux distribution that provides the following:

• allows to do "profile builds" of its source packages with different features enabled or disabled
• stores information about which feature requires which build dependency
• stores everything in a format that can be parsed and analyzed
• covers a similar range of software packages as Debian does

If such a distribution can be found then the information from it can be used to find dependencies that can also be dropped from Debian source packages. Gentoo is a distribution that fulfills above requirements through so called USE flags that allow to enable or disable features during compilation. Dependencies of Gentoo source packages are stored in .ebuild files that control the build process. Since .ebuild files are bash scripts, parsing them is not trivial. I therefor used the emerge software package to extract that information. Thanks to the well written emerge code and to quick help in the Gentoo IRC channel, it didnt take long to make the code run on Debian. My sourcecode is downloadable here: https://gitorious.org/debian-bootstrap/gen2deb Before I list the results of using Gentoo USE flags to determine dependencies that can potentially be dropped from Debian source packages, let me list the problems that this method entails.

Only package name matching, no version matching When writing the mapping from Debian to Gentoo packages and back I discard version information. There are just too many versions that either Debian or Gentoo have and are not present in the other. So the assumption is, that Debian Sid and Gentoo have both the most recent major versions of upstream software which has roughly the same requirements in terms of build dependencies.

Gentoo packages are matched to Debian source packages In Gentoo there are only source packages and no binary packages. So I map Gentoo packages to Debian source packages. But Gentoo source packages build depend on other source packages while Debian source packages depend on binary packages. So at some point I have to translate Gentoo packages to Debian source packages and those source packages to Debian binary packages. I do this by analyzing the original binary package build dependencies of a Debian source package and then filter out those binary packages as being droppable that are built by the Debian source packages that were found to be droppable.

Not the exact same package set There is some software that is only in Gentoo and some that is only in Debian. Debian and Gentoo also split some source packages differently.

Gentoo has more direct dependencies Many build dependencies in Debian are indirectly pulled in through dependencies of direct build dependencies. In Gentoo source packages directly depend on most things they need to build successfully. This leads to the list of dependencies in Gentoo to be much larger than the list of dependencies in the corresponding Debian source package. It also means that lots of dependencies that can be dropped in Gentoo are not found to be droppable in Debian because they are not direct dependencies of that source package.

There are no implicit dependencies Gentoo will often drop dependencies that are essential or build-essential packages in Debian and are therefor implicit build dependencies that cannot be dropped.

Result Despite the many problems, the result doesnt look too wrong. I got some Debian source packages that were found to have droppable build dependencies from Thorsten Glaser and all dependencies that Gentoo found to be droppable were also dropped by him. To put everything into numbers: the current 912 nodes big SCC in Debian Sid can be reduced to 6 individual SCC with 422, 5, 5, 3, 2 and 2 nodes each. So using Gentoo cuts the size of the central component to more than half. Surely, there will be a number of dependencies that were found to be droppable in Gentoo but are actually not droppable in Debian. The point is, that it is better to have "some" data even if it contains false positives than no data at all. It is easier for a human to verify if some suggested droppable build dependencies are actually correct than going through hundreds of source packages with thousands of dependencies manually.

James Bottomley just published a description of the Linux Foundation's Secure Boot plan, which is pretty much as I outlined in the second point here - it's a bootloader that will boot untrusted images as long as a physically present end-user hits a key on every boot, and if a user switches their machine to setup mode it'll enrol the hash of the bootloader in order to avoid prompting again. In other words, it's less useful than shim. Just use shim instead.

comments

Last weekend I was in Southwest Harbour, Maine again for the annual LTSP hackfest (called By The Sea). It s a fun and productive event and as always it s been good catching up with LTSP folk, even though we were missing Oliver, Alkis and Vagrant. Here is a summary of what I can recall from the discussions of the weekend Recent Happenings in LTSP

• New LTSP Website After a really long time, LTSP finally has a new website. It s a big improvement, there s a new success stories page and the wiki is now self-hosted.
• LTSP PNP - This is a new package (ltsp-pnp) that can safely be installed on existing machines to allow a user to log into an LTSP server using LDM.

Going Forward Social

• On Friday, Chuck Liebow took us out for a boat ride around the harbour on the Sea Princess

• On Saturday night we had the big lobster dinner as per BTS tradition (I had steak since I m not a big seafood person). We were to busy eating and telling stories to be taking any pictures
• And last but certainly not least, I finally met Eric Harrisson, who did a lot of work on K12-LTSP in the Portland schools district. He has an awesome hobby where he builds guitars out of well, almost anything he can find.
  He brought along one of the experimental guitars he slapped together recently and told me all about how it s put together. Even more amazing, he said I could have it! I was planning on getting an acoustic guitar anyway so I m very thrilled about it. Not only is it a completely unique guitar but it sounds great too. I m going to have to think of something to make back for him!

It was great seeing everyone again. I first started using LTSP around 9 years ago and never imagined back then that I d get to meet the people behind it. Ron Colcernian sourced us some really cool LTSP tops that you can see us wearing in the group photo. I hope to get to BTS again next year!

Christian's most recent blog post got me wondering if the decline in the bug reporting rate in Debian was something new, or something which often happened during releases. So, lets try to figure that out. In the BTS, when a bug report is filed, the report is written to a file called bugnum.report, and then not touched from then on. Let's look at the modification date on that file to see when each bug was filed; and since we're going to plot this, lets only look at bugs ending in 00:

stat -c '%n %Y' /srv/bugs.debian.org/spool/ archive,db-h /00/*.report > ~/reporting_rate.txt

Now, lets get the data into R and plot it. [For clarity, I'm not showing the R code, but it's available in the source code for this post.] From the plot (Bugs reported per second over time with a red loess fit line), it looks like we do see a decline during certain periods. However, there's an even more alarming trend of a decrease in bug reporting in Debian which has been happening since 2006. (Note that I've truncated the y scale significantly; there are periods in Debian where the bug rate is astronomically high, usually corresponding to mass bug filings; I've also limited the plot to data from 2003 on, as I have to clean up that data significantly before I can plot it like this.) Not sure exactly what that means, but it is troubling.

I am a big fan of Ruby on Rails: it does a lot of things, and it does most of them pretty well. When starting a new web project, it's the first thing I would reach for: most of the time, your problem is going to be figuring out a good product/market fit, or whipping up some internal tool without wasting a lot of programmer time. Once you've got a firm grasp of the problem, then maybe you can consider optimizing. Who cares if you do something that no one wants really really fast? However, Ruby on Rails is not beautiful in terms of being particularly fast or lightweight. No complaints from me: most of the time, I'm happy to have something that does so much for me, leaving me to work on the actual problem at hand. Once in a while, though, you do need fast and relatively lightweight, and that space has been getting more interesting over the past few years, at least in terms of the web. First of all, technologies like "Comet", utilizing web sockets or some other always-on connection are becoming more common, where a socket with the server remains open in order to quickly exchange data from the server to the client - and back. That seems to be a poor fit for something like Rails, where it can tie up a lot of resources if one isn't careful. And while computing costs continue to decline, no one minds getting more for less in terms of what their server can do. Furthermore, with frameworks like Backbone.js, pushing more and more code to the client, the server can afford to be a bit simpler and do less, so it may as well be snappy to boot. Java has long been fairly popular for "heavy lifting" types of applications, partially because it does end up being reasonably fast. But it's not something I've ever had much fun using and is usually kind of wordy, and makes you feel like you need a crew of people in Palo Alto, one in Bangalore, and one in Stockholm just to churn out all the code. And it certainly is no lightweight in terms of memory either. So... while it can certainly do pretty much anything you need, I don't see it as being the strongest player when someone needs "real time web" code, and needs it to be reasonably light weight. Ruby, outside of standard Rails stuff like Passenger, seems to offer some interesting possibilities for this kind of work, like Reel but they don't seem to have the traction other solutions do. Python is in a similar situation: the Twisted framework has been around for a while, and while it has some success stories, never seems to have really 'caught fire'. Neither of these languages was built for "concurrency" from the ground up, and that seems to have, to date, inhibited people from using them extensively for this kind of job. My gut feeling is that the need for speed and "concurrency" (or at least handling a lot of concurrent users) will drive adoption of languages heretofore not so popular on the server. Let's have a look at them. Erlang: This is by no means a new language, having been developed in Ericsson in the late 1980ies; and it implements several interesting concepts. First and foremost, concurrency is handled in the form of many small "processes", which are not actually Unix processes at all, but processes internal to the Erlang VM. The Erlang system contains a scheduler that allocates resources to all of these processes, so even if one of them dies or behaves badly, it's not a problem: the system as a whole can continue to function well. The way Erlang is built, the scheduler is preemptive: the internal "processes" don't need to yield to let other processes run. Beyond the scheduler and simple processes, Erlang gives you the tools to create elaborate trees of supervisors and workers that are quite robust to failures in any one portion of the system, as well as giving you the tools to set up a system to run on multiple, distributed computers. This kind of thinking is necessary when you write applications, such as phone switches, where downtime is really, really not ok. Erlang processes comunicate almost exclusively via message passing, meaning that state is not shared. Altogether, this makes for a fast, rock solid system that can easily handle thousands of concurrent connections without breaking a sweat. The computing world being what it is though, Erlang is likely to be more of a Lisp or Smalltalk: it's a trailblazer that did many things years before their time in other languages, but I don't see it as ever quite catching on amongst 'the masses'. It has a wonky syntax, it's a functional programming language, and because it is used in environments where too much experimentation is not good, it does not have a lot of room to break with its own past and innovate in terms of the language itself: it's slow to change and improve, even where the need to is clearly perceived. Node.js: is the opposite, in some ways: being based on Javascript, it draws on a huge number of potential programmers - orders of magnitude more than Erlang. And thanks to design decisions enforcing the use of asynchronous code and callbacks for anything that could block, it deals quite well with concurrent connections, even if the language and libraries don't really give you much in terms of "true" concurrency. This is a simple model that works pretty well, even if, theoretically, all it would take is one "while (1)" loop in a callback to block the entire system. In practice, this doesn't seem to be a big problem, though. More of a problem is writing maintainable code, when everything is a callback. Keeping the network of callbacks straight can be a bit of a chore, and is probably not an optimal model in terms of programmer productivity. That said, people seem to do make due with it, although Node is young enough that we haven't seem projects that are 5 or 6 years old and maintained by people who didn't write them. One of the advantages of being such a popular language is that lots of people have a strong interest in seeing Javascript being very, very performant. That need begat the V8 Javascript engine, from Google, which they were kind enough to release as free software. So one of the advantages that Node has is that the underlying implementation is extremely fast for a dynamic language. For many people who know Javascript, picking up Node.js is also an easy choice, even though people used to browser side programming with Javascript will have to adjust their way of thinking to succesfully tackle server projects. Go language: this is an interesting one, written by some luminaries who work at Google. It can best be described as something akin to C with some updated features, such as garbage collection, that make it more suitable for working on large projects, where things like memory leaks are going to make life very frustrating. It has the feel of a "real language, for real programmers" in that it doesn't stray far too far from what people are used to - it's not going to cause the "what the hell is this?!" reactions that Erlang might in more close-minded circles. Between the big company backing it, and the approachable syntax and concepts, Go looks like it has a good shot at the mainstream. Where it gets interesting is their concurrency model, which is apparently based on something called "communicating sequential processes" which, superficially at least, looks like it has some things in common with Erlang's "actor model" of concurrency. Under the hood, Go apparently hives off its "goroutines" to different OS level threads, but does not have a preemptive scheduler like Erlang: http://code.google.com/p/go/issues/detail?id=543 - although according to this, they may change that in the future. I'm not enough of a computer science guy to comment much on the details of CSP vs Actors, but both seem like valid models with strengths compared to trying to keep threads straight, which always seems to be a source of problems for programmers. Conclusions So, what's actually going to happen? I see Node.js as the clear front runner. It takes a worse-is-better approach that seems to work well enough as people get started. If they encounter difficulties later, they can always rewrite in something else, if needs be, but by "luring people in", Node.js has gathered a large group of users who continue, in turn, to churn out more code for use with the system, making it more attractive to new users. Programming languages are not winner-take-all markets though, so perhaps there is room for a few more languages to have decent followings in this space. Hopefully the competition will lead to ever better tools for those of us utilizing them! What do you think?

Just posted to d-d-a, bits from the DPL for Septemer 2012.

---

Heya,
here is the periodic report of DPL activities, this time related to September 2012 (posted here with some delay, as I've been traveling oversea on behalf of Debian for the past few days). Help needed: Google Code-In The Google Code-In (GCI) initiative is about to start again. We don't have a great track record of participating into it, and that's a pity. The initiative revolves around small tasks that other Free Software projects have come to call "easy hacks", showing how effective they're in attracting new volunteers. To participate, we need both mentors and admins. If you're interested see Ana's call for help and please volunteer on the soc-coordination list. Related to this, you've probably seen the report of our participation in GSoC 2012. I'd like to thank all students, mentors, and admins for a very well organized edition. Logo relicensing & other assets The DFSG-relicensing of the Debian logo, which I've mentioned in recent updates, has now happened. The so called Debian "Open Use" logo, in both variants (with and without "Debian") is now dually licensed under LGPL3+ / CC-BY-SA 3.0. The change has been made effective with a resolution of the SPI Board of Directors. Note that the so called "official" logo, which we seem to use very little, is still released under the terms of a non-free license. Discussion is ongoing on -project to stop promoting it. If you're using it or if you've arguments in favor of against that, please participate in the discussion. I've got from SFLC an updated draft of the new proposed trademark policy, implementing most of the changes requested during related August discussions on -project. I'll post it for review there shortly. The Japanese Trademark Office (JPO) has sent us the final confirmation that trademark transfer request (from individuals Debian contributors in the area to SPI) has happened. We've paid the needed fee of 457 CHF to conclude the transaction. We finally settled a domain transfer agreement with the current owner of the debian.eu. Technically, the domain will be transferred to FFIS in the coming weeks. Core teams (non-)news Quite some core teams are in the process of changing their formations these weeks, but most are still pending proper delegation (or equivalent). So here are just a few "teasers":

• as you might have read in the ftp-master sprint report, we have a new ftp-master: congrats Ansgar! (delegation update pending, but no suspense here)
• we have a new front-desk member (delegation pending + suspense)
• we have a new debian policy team (delegation pending + suspense)
• the tech-ctte is in the process of selecting a new candidate to fill the vacant seat. I've nominated a couple of candidates myself, and I'm now waiting for tech-ctte recommendations as per Constitution 6.2. On related news, the tech-ctte has now dealt with the conflict on Python interpreter maintenership, which was also the longest standing issue brought before them. Having invested quite some energy in that issue myself, I'm delighted to see it finally addressed.

I hope to finalize all the pending tasks above this month. Future events I've spent quite some time to plan my participation in events I'll be attending during October on behalf of Debian, such as the ACM Reflections conference in Urbana-Champaign (at the time of writing, this has already happened), the Ubuntu Developer Summit in Copenhagen, and the LinuxDay event in Turin. Cheers.

---

PS the day-to-day activity log for September 2012 is available at the usual place master:/srv/leader/news/bits-from-the-DPL.txt.201209

It's quite some time since I didn't report about Debian Installer localization. Indeed, thanks to the tireless activity of Cyril Brulebois, we focused on releasing D-I and I indeed stopped harassing translators for updates around July 2012. Why July? Well, you probably know we have "something" to release as soon as we can and most, if not all, Debian energies should be focused on this! Or, more precisely speaking, I only focused on getting the newly introduced material translated: we had several changes recently in D-I, mostly focused on important features, such as IPv6 support (thanks to Phil Kern who worked on this), better wireless networking support (thanks to Sorina Sandu) and EFI boot support (thanks to Steve McIntyre). And, of course, when people change code in D-I, they want to add questions to users, display error and informative messages, etc, etc. And these need translations..:-) Most translators coped with all this (sometimes with /me hitting them hard on the head to get updates) and D-I beta2 was released with 37 complete translations out of 73 supported languages. Yesterday, I just resumed the activity of trying to get more updates not only for those recent changes, but also for other older changes...or for languages that never got completed in the past. As a result, we bumped from 37 compelte languages up to 45 this morning: look for level 1 here (level 2 has been hit by a change in iso-codes, but that change doesn't really affect D-I). If you language is not 100% in the leftmost column on the stats page, you can probably help. Just get in touch with me and we'll check if somebody is already working on this or not.

(doh, I nearly had it. I just got #690019, #690020 and #690022 for l10n stuff) Bartek Krawczyk reported Debian bug #690000 on Monday October 8th, against guake. And my friend Sylvestre Ledru, the package maintainer, now has to fix it instead of trying to promote the use of Scilab over proprietary alternatives in the French aerospace research organizations..:-) Bug #680000 was reported as of July 2nd: 3 months and 8 days for 10,000 bugs. This is a VERY significant drop in the bug reporting rate in Debian. Last time, I wrote: "How will the wheezy freeze affect this? We'll see in two months!". We have the answer: the wheezy freeze triggerred an important drop in bug reporting rate in Debian. My general feeling is somehow different: for whatever reason, I feel like the *overall* activity in the project has dropped significantly. I seem to have less mails to read, less bugs reported against my packages, even less heated discussions here and there, as well as several very quiet channels on IRC. Am I pessimistic when feeling that the overall momentum is sliding out of Debian? Maybe I am, so, folks, please make me optimistic and move you fingers out of the place where they are and help releasing that damn penguin. Apart from that, our next milestone (apart from the wheezy release!) will be bug #700000. Remember the bet?. It looks like the probability of Kartik Mistry winning it is now away (he bet for Now 8th 2012) and the best position is hel by David Pr vot (he bet for December 12th). On the other hand, my own chances are increasing if the bug rate drop is confirmed and if bug #700000 is reported in more than 3 months (I bet for February 14th 2013, guess why?). We'll see that in a few weeks!

If you have two or more available gpg keys that can be used for the same identity (e.g.: 'John Doe <john@doe.com>') that you might use to add entries to changelogs of packages, you might end up in situations where git-buildpackage or other similar tool might want to sign packages with the wrong key.

It seems debsign (the tool that actually does the signing) just picks up the first key that is still valid (I also have a revoked key) and matches the used identity.

There are many ways to fix this, but the one that will work for most cases is to run this command:

echo 'DEBSIGN_KEYID=0x0123ABCD' >> ~/.devscripts

Of course, you should replace 0x0123ABCD with the keyid which you prefer.

This will create a ~/.devscripts file (if it didn't exist) which will contain the DEBSIGN_KEY variable with the desired value. This file will be sourced by debsign before doing its actual work, so it will do the right thing when ran.

There are also git-buildpackage specific fixes like defining builder in the [DEFAULT] section of ~/.gbp.conf and passing '-k0x0123ABCD' to debuild something like:

[DEFAULT]
builder = debuild -i -I -k1234ABCD
...

But that probably gets ugly if you use a custom builder.


P.S.: I have just deleted my old key 0xDD1F1F9F since I won't be using it anymore. I don't remember where I put the revocation certificate, but I'll revoke the key, once and if I find the revocation certificate. Otherwise, it will expire in July, next year.

From now on, I am going to use only the key 0xE3E083A1 to which I even added a photo and some newer identities and updated with a few signatures I got during DebConf 9.

For those of you back at my university, you may have noticed I'm not there this semester.

Google's 2012 FEP team

I spent this past summer at Google, Inc, developing internal tools for the AdSense team, specifically Google Programmable Ads. Being at Google was great; I loved my team and my intern cohort, the rest of the interns in the Freshman Engineering Practicum.

I'm not currently at Google; my internship ended at the beginning of August. Although it was a blast, all good things must come to an end.

I'm not at Mason, either. I was super excited to return to university, and was just about to buy my books and get ready to move in when I got an email from a former coworker at Ksplice, a startup Oracle acquired while I interned there last year. He was starting a new company which would focus on business communications. I'd be working with a bunch of my former coworkers, and based on what I had heard of the company's plans I was confident in their ability to make an awesome product.

Needless to say, when they decided to offer me a position working there full time, I jumped on it.

From an academic point of view, Mason didn't really have much of a mechanism to support this. Co-ops are uncommon there, and not really supported for more than one semester; a full year away had never been done, according to our career services. My department was fully supportive, however, so we managed to find a way to make it work. This involved filling out some oddly-named forms, such as Special Registration for Graduation Request, which the registrar asserted was the right form, trust us on this one.

Myself and Obey Arthur Liu at a SIPB hackathon

But that's all neither here nor there. I'm now up in Cambridge, MA, working with awesome people (including but not limited to tabbott, wdaher, jesstess, and keegan), and exploring the city. I'm hanging out with MIT SIPB, helping with the maintenance of Sugar Labs' servers in E15 and spending more time working on various open source projects.

To my friends at Mason: I miss you all. I know regardless of how the next year+ turns out, it'll be one one hell of a ride.

I was about to post a different story but things turned out differently, for good. Now I ll post tips that may save the day in the event of encoding issues when dealing with MySQL table data. Hopefully you find it useful. Say you have a MySQL table that has collation set to latin1 but you have utf8-encoded data stored. This happened when importing a dump, or because of Murphy s law. When querying the data from your application or from mysql s tools you get awful results for non-ascii characters such tildes. Changing the table s collation definition will only work for new columns and records. Altering the collation for the said column would make things worse. So what to do? Fortunately MySQL offers a nice way to save the day. You can use a transition conversion and then set the table encoding back to utf8, so it pairs accordingly with the encoding type for the data it has stored. ALTER TABLE table_name CHANGE column_name column_name BLOB; Issuing the previous command will make the table change it s type to BLOB. Why? Because by doing it the stored data is kept untouched. This is important. If you use the CONVERT or MODIFY commands the data will be converted by MySQL. We don t want that since the data is already in the encoding type we want. You can note that column_name is repeated twice, this is because the CHANGE command copies the data and we want the resulting column to have the same name. Now, we have to put the data back on the encoding we want, so everything is under control again. This is done by issuing the following command. ALTER TABLE table_name CHANGE column_name column_name COLUMN_TYPE CHARACTER SET utf8; COLUMN_TYPE depends on the content s column type. For instance, for a TEXT column it should be. ALTER TABLE table_name CHANGE column_name column_name TEXT CHARACTER SET utf8; It works for VARCHAR, LONGTEXT and other data types that are used to store characters.

I have had an opportunity to rework some code to query NTP servers directly in python rather than running ntpq -pn and scraping the result. The over-the-wire protocol format is reasonably straight forward and my little module is now passing all of its nosetests which is wonderful. While its only a single table, ntpq actually does multiple requests to get its results. NTP control messages are all mode 6 which tells the client and server its not time sort of things, but more about queries. A control packet has a sequence number which the server responds with so you can line up your requests with responses. For large responses, the more, offset and count fields work together to knit up a large block of data. The first packet of the response will have the offset field of 0. The count is the size of the data payload in bytes. If the response is too big, the more bit will be set to 1 to say there is more coming. The second packet will have an offset that is the same value as the previous packets count value, so you can stitch them together. The first query is a READSTAT which uses opcode 1. This returns a list of 4 byte responses with the association ID taking the first two bytes and the peer status taking the second. If you look at the output of ntpq you will notice a character in the first column which is the peer selection. The important one for us is asterix, which equates to 110 in the lower 3 bits of the first status byte. This is the system peer which is the main one. There may be others, up to 2, (usually symbol +) which are compared to the system peer. ntpq doesn t really care about the different peers and just displays the status but I want to know what peer the server is synchronised to. Next we need more details about the system peer (or all peers in ntpq). To do this a READVAR or opcode 2 is sent to the server. The association ID is set to the association or peer we are interested in. The response for my setup is sent over two packets which need to be reassembled and it is all in plain ascii which is a comma-separated list with a key=value pair. I use the following python line to split it up. I m sure people that know python better than I will say I am doing it wrong (it looks a bit klunky to me too) but it does work. From that I can query the various values that are returned by the server about its peers, including the source of its clock.

Related articles

• Verifying NTP Reserved Mode Denial of Service Vulnerability (gursevkalra.blogspot.com)
• Time precision between NTP and GPS source (stackoverflow.com)

(This is something I did a few months ago, and I really should have noted down all the details then so I covered everything. However hopefully these notes will remind me enough next time.) When I first wanted to reverse engineer a USB device that only had Windows drivers the "easy" option was to take a Windows machine, install usbsnoop on it and capture the traffic as a bunch of verbose text files. It was a cumbersome procedure. Recently I wanted to do a firmware upgrade of a ZTE 3G modem dongle, partly to provider unlock it and partly to try and enable some voice functionality. I was also hoping to sniff the traffic to see how to drive the voice side of things. These days I don't have a dedicated Windows box, but I do have a Windows 7 KVM virtual machine. I hadn't yet used the USB support in this, but I thought I'd see what it was capable of. Firstly I had to explicitly enable USB2 - the device wasn't happy with the default USB1 only stack that KVM enabled. That involved passing -usb -device usb-ehci,id=ehci to KVM. I also told KVM to grab all the ZTE devices with -device usb-host,bus=ehci.0,vendorid=0x19d2. I dropped a udev rules file into /etc/udev/rules.d to ensure any device nodes created belonged to my normal user. This gave me a Windows setup that could see the USB dongle and install the appropriate drivers. It was also happy to do the firmware update (along with various device resets on the way as it changed USB ID - this is why I needed the udev rule, to ensure every time the device re-appeared it would be seen by the KVM instance without manual intervention). After that was complete I investigated usbmon. modprobe usbmon created the appropriate /dev/usbmon<n> files and I chowned the appropriate bus to my normal user. Once this was done I was able to start Wireshark which rather nicely has full USB decoding support. Firing up the custom app in the Windows KVM guest I could see the traffic going back and forth to each of the device endpoints and work out what was going on. All of this was much more flexible than using a standalone Windows box. Once I figured out that I needed to explicitly enable USB2 I was quite pleased with how simple USB access under KVM was. I also used lvm to create a snapshot of the Windows guest so at the end I could roll back all of the drivers I'd installed for the dongle. And being able to use Wireshark instead of trawling through dense text files helped a lot in seeing the command stream.

So, the other day, I wrote a blog post asking people to make sure their private hacks become published somehow. So in the sprit of that, I should also publish my recent hacks. I have been toying around as a web dude and created two nice pieces of oldschool webby software. So. Time to introductions: Dodoma is a simple online note taking application. I have created a boring homepage for it on http://sune.vuorela.dk/dodoma and I have a screenshot here:
Kadaka is a simple rss reader showing the five newest items from various rss feeds. Has special support for youtube channels.
I have created a boring homepage for it on http://sune.vuorela.dk/kadaka and I have a screenshot here:
Have fun with those two apps. Patches always welcome. Some of you might wonder where does he get those crazy names from . Well. One of them I found on a map. Thats actually a way I name quite many of my projects. Even those that don t even leave my harddrive. And the other one actually also is on a map, but I saw it on a frequently used bus in Tallin, Estonia. And. I am considering naming my next fancy projects after some of the cool words found on the join the game member page. If you want to be part of that, feel free to Join the game!

The plan for supporting UEFI Secure Boot in Fedora is still pretty much as originally planned, but it's dependent upon building a binary which has the Fedora key embedded, and then getting that binary signed by Microsoft. Easy enough for us to do, but not necessarily practical for smaller distributions. There's a few possible solutions for them.

• Require that Secure Boot be disabled
  
  Not ideal. The UI for doing this is going to vary significantly between machines, making it difficult to document. It also means that the security benefits of Secure Boot are lost.
  
  
• Require that the machine be placed in Setup Mode
  
  Clearing the enrolled Platform Key results in the system transitioning into Setup Mode, and from then on new keys can be enrolled into the key database until a new Platform Key is enrolled. Distributions could ship an unsigned bootloader that then writes the distribution keys into the database - James Bottomley has an example here. This means that the distribution can still benefit from Secure Boot, but otherwise has the same downside that the UI for doing this will vary between machines.
  
  
• Ship with a signed bootloader that can add keys to its own database
  
  This is more interesting. Suse's bootloader design involves the bootloader having its own key database, distinct from those provided by the UEFI specification. The bootloader will execute any second stage bootloaders signed with a key in that database. Since the bootloader is in charge of its own key enrolment, the bootloader is free to impose its own policy - including enrolling new keys off a filesystem.

I've taken Suse's code for key management and merged it into my own shim tree with a few changes. The significant difference is a second stage bootloader signed with an untrusted key will cause a UI to appear, rather than simply refusing to boot. This will permit the user to then navigate the available filesystems, choose a key and indicate that they want to enrol it. From then on, the bootloader will trust binaries signed with that key.

Distributions are then able to take an existing signed copy of shim and put it on their install media, along with a file containing their key. If a user attempts to boot then the boot will fail because the second stage bootloader isn't signed with a trusted key, but the user can then use the navigator and select the distribution's key file. After providing confirmation and rebooting, the second stage bootloader's signature will now be recognised and the installer will boot.

This has the advantage over the first two options that the UI is consistent, making it easier to document the install process. The primary disadvantage is that the distribution won't be able to rebuild shim and will have to ship a pre-compiled binary. This may well be unacceptable to distributions like Debian, but should still provide a viable approach for other distributions who are either unwilling or unable to deal with Microsoft themselves.

comments