Just as a little reminder, the Bug Squashing Party in Salzburg will start in three days. We still have sponsored accomodation for five four people left, so don't hesitate to come! Squashing as many RC bugs for Wheezy as possible is on the TODO list for the weekend! Also we'll work on making Debian a proper Spacewalk client. This might be a bit too late for Wheezy, but there will be backports See YOU in Salzburg!

1.5M CPU hours runs on the Vienna Scientific Cluster by my 3D+3V hard expanding loops code. It is simulating the Chromo-Weibel instability of a non-Abelian plasma in an 1d expanding background to account for the expansion of a Quark-Gluon plasma fireball in a heavy ion collision. The code is parallellised using Open MPI and one single run roughly needs 0.5TB memory to hold all the needed 5-dimensional matrices and ~ 72h to complete. The resulting heavy operation is of course matrix multiplication of those physical and auxilliary fields. I tried to use git-annex on the resulting physical fields data output of ~ 150Gb and 300k files. Currently I do prefer git itself for it's speed and it only doubles the amounted space. I'm thrilled to see that git annex is still considering this "semi-extreme" usecase of lots of data files as it would help collaborators to have partial checkouts for laptop usage and quick checks. As the git-annex kickstarter is a rocket, i'd add my wish to see the current symlink mess go: desymlink git-annex. Now it is the time to properly analyse all this data and get the full 3d paper follow up on Instabilities of an anisotropically expanding non-Abelian plasma: 1D+3V discretized hard-loop simulations (previous paper) out maybe soon. (;

Just posted to d-d-a, here are the monthly DPL bits.

---

Dear project members,
here's the periodic report of what has happened in DPL land, this time during May 2012. It's briefer than usual, as this year I've enjoyed the lovely French habit of frequent holidays during the month of May. Highlight First highlight for this month is an invitation to us all. We're now in June and the Wheezy freeze is literally a few days away. The RC bugs count is moving in the right direction, but it's still stellar if we want to ensure a short freeze. And a short freeze is of paramount importance: it'll reduce the time during which we can't implement great plans for the future, increase the "freshness" of software we'll ship with Wheezy, and reduce the inconveniences for those who run the testing suite due to its nice "rolling" feature. So please set out some regular time to do RC bug squashing, by providing patches and doing NMUs. Releasing Wheezy is not something that could be outsourced to the Release Team, it's a collective responsibility that kicks in as soon as our own packages are RC bug free (which they already are, right? The second highlight is more on the internal structure camp. As mentioned last month, I've discussed with the tech-ctte insisting a bit to set up periodic IRC meetings, to ensure outstanding issues get periodically reviewed. At the end of May the first IRC meeting has happened, and has been very productive. See the minutes. Another one has been scheduled, trying to setup a monthly cadence, for the end of June. Many thanks to all tech-ctte members who have took part in and helped with the meeting organization. Communication I've given an interview to iTWire, answering a number of questions about several past and future Debian challenges. Discussions The ongoing discussion to harmonize packaging of multimedia software between the official Debian archive and the unofficial debian-multimedia.org archive (dmo) has progressed. I've tried to help the two groups reaching an agreement on which packages belong where, so that both duplicate packaging efforts and user inconveniences are minimized. That seems not to have worked and dmo maintainers have simply announced that they will move away from the current domain name to a new one that does not include "debian" in its name. Sprints There will be a Debian Science sprint in June, co-located with the broader Debian Science event organized by European Synchrotron Radiation Facility (ESRF) in Grenoble. I've confirmed my attendance for the opening talk of the conference day. ESRF organizers have kindly sponsored travel for all Debian attendees, many thanks to them! Another sprint will happen next week-end in Paris, this time by the i18n/l10n team. I've approved the corresponding tentative budget for travel sponrship for ~2'000 EUR. Other expenses Hardware replacement plans go on. We've ordered SSDs (for ~3'000 CAD) for recently bought machines meant to replace bugs-mirror, bugs-master, and udd. On the "small emergencies" front, we also had to replace failing disks on wagner (1/2 of alioth), for as little as 100 GBP. Miscellanea

• I've helped a bit with the ratification of the Debian diversity statement by seconding it and shortening the discussion period
• on the multimedia front, I've also sought legal advice for the multimedia team, to better understand how to provide a decent multimedia experience for Debian users while respecting the draconian copyright system that exists almost world-wide. There seem to be some solutions in sight for the few missing packages (which I'm still discussing with the respective maintainers), but they'll hardly be ready for Wheezy
• three people replied to my call for help with DPL tasks (thanks!). I'll be shortly contact them to share the list of tasks that can be outsourced to anyone willing to help; then I'll try to setup a calendar of periodic IRC public meetings. There's still time to apply if you are interested (hint hint!)
• on behalf of Debian, SPI has entered in the ISC Forum Membership Agreement, giving access to the Debian security team to security notices for softwares like BIND and DHCP. We previously had a similar agreement for individual maintainers, but it's good to move it at a project level, in order to diminish bus factors

Happy Wheezy freeze,
and RC bugs squashing!

---

PS the boring day-to-day activity log for May is available at master:/srv/leader/news/bits-from-the-DPL.txt.201205

tl;dr: Version 1.0 of Obnam, my snapshotting, de-duplicating, encrypting backup program is released. See the end of this announcement for the details. Where we see the hero in his formative years; parental influence From the very beginning, my computing life has involved backups. In 1984, when I was 14, my father was an independent telecommunications consultant, which meant he needed a personal computer for writing reports. He bought a Luxor ABC-802, a Swedish computer with a Z80 microprocessor and two floppy drives. My father also taught me how to use it. When I needed to save files, he gave me not one, but two floppies, and explained that I should store my files one one, and then copy them to the other one every now and then. Later on, over the years, I've made backups from a hard disk (30 megabytes!) to a stack of floppies, to a tape drive installed into a floppy interface (400 megabytes!), to a DAT drive, and various other media. It was always a bit tedious. The start of the quest; lengthy justification for NIH In 2004, I decided to do a full backup, by burning a copy of all my files onto CD-R disks. It took me most of the day. Afterwards, I sat admiring the large stack of disks, and realized that I would not ever do that again. I'm too lazy for that. That I had done it once was an aberration in the space-time continuum. Switching to DVD-Rs instead CD-Rs would reduce to the number of disks to burn, but not enough: it would still take a stack of them. I needed something much better. I had a little experience with tape drives, and that was enough to convince me that I didn't want them. Tape drives are expensive hardware, and the tapes also cost money. If the drive goes bad, you have to get a compatible one, or all your backups are toast. The price per gigabyte was coming down fast for hard drives, and it was clear that they were about to be very competitive with tapes for price. I looked for backup programs that I could use for disk based backups. rsync, of course, was the obvious choice, but there were others. I ended up doing what many geeks do: I wrote my own wrapper around rsync. There's hundred, possibly thousands, of such wrappers around the Internet. I also got the idea that doing a startup to provide online backup space would be a really cool thing. However, I didn't really do anything about that until 2007. More on that later. The rsync wrapper script I wrote used hardlinked directory trees to provide a backup history, though not in the smart way that backuppc does it. The hardlinks were wonderful, because they were cheap, and provided de-duplication. They were also quite cumbersome, when I needed to move my backups to a new disk the first time. It turned out that a lot of tools deal very badly with directory trees with large numbers of hardlinks. I also decided I wanted encrypted backups. This led me to find duplicity, which is a nice program that does encrypted backups, but I had issues with some of its limitations. To fix those limitations, I would have had to re-design and possibly re-implement the entire program. The biggest limitation was that it treated backups as full backup, plus a sequence of incremental backups, which were deltas against the previous backup. Delta based incrementals make sense for tape drives. You run a full backup once, then incremental deltas for every day. When enough time has passed since the full backup, you do a new full backup, and then future incrementals are based on that. Repeat forever. I decided that this makes no sense for disk based backups. If I already have backed up a file, there's no point in making me backup it again, since it's already there on the same hard disk. It makes even less sense for online backups, since doing a new full backup would require me to transmit all the data all over again, even though it's already on the server. The first battle I could not find a program that did what I wanted to do, and like every good NIHolic, I started writing my own. After various aborted attempts, I started for real in 2006. Here is the first commit message:

revno: 1
committer: Lars Wirzenius <liw@iki.fi>
branch nick: wibbr
timestamp: Wed 2006-09-06 18:35:52 +0300
message:
  Initial commit.

wibbr was the placeholder name for Obnam until we came up with something better. We was myself and Richard Braakman, who was going to be doing the backup startup with me. We eventually founded the company near the end of 2006, and started doing business in 2007. However, we did not do very much business, and ran out of money in September 2007. We ended the backup startup experiment. That's when I took a job with Canonical, and Obnam became a hobby project of mine: I still wanted a good backup tool. In September 2007, Obnam was working, but it was not very good. For example, it was quite slow and wasteful of backup space. That version of Obnam used deltas, based on the rsync algorithm, to backup only changes. It did not require the user to do full and incremental backups manually, but essentially created an endless sequence of incrementals. It was possible to remove any generation, and Obnam would manage the deltas as necessary, keeping the ones needed for the remaining generations, and removing the rest. Obnam made it look as if each generation was independent of each other. The wasteful part was the way in which metadata about files was stored: each generation stored the full list of filenames and their permissions and other inode fields. This turned out to be bigger than my daily delta. The lost years; getting lost in the forest For the next two years, I did a little work on Obnam, but I did not make progress very fast. I changed the way metadata was stored, for example, but I picked another bad way of doing it: the new way was essentially building a tree of directory and file nodes, and any unchanged subtrees were shared between generations. This reduced the space overhead per generation, but made it quite slow to look up the metadata for any one file. The final battle; finding cows in the forest In 2009 I decided to leave Canonical and after that, my Obnam hobby picked up in speed again. Below is a table of the number of commits per year, from the very first commit (bzr log -n0 awk '/timestamp:/ print $3 ' sed 's/-.*//' uniq -c awk ' print $2, $1 ' tac):

2006 466
2007 353
2008 402
2009 467
2010 616
2011 790
2012 282

During most of 2010 and 2011 I was unemployed, and happily hacking Obnam, while moving to another country twice. I don't recommend that as a way to hack on hobby projects, but it worked for me. After Canonical, I decided to tackle the way Obnam stores data from a new angle. Richard told me about the copy-on-write (or COW) B-trees that btrfs uses, originally designed by Ohad Rodeh (see his paper for details), and I started reading about that. It turned out that they're pretty ideal for backups: each B-tree stores data about one generation. To start a new generation, you clone the previous generation's B-tree, and make any modifications you need. I implemented the B-tree library myself, in Python. I wanted something that was flexible about how and where I stored data, which the btrfs implementation did not seem to give me. (Also, I worship at the altar of NIH.) With the B-trees, doing file deltas from the previous generation no longer made any sense. I realized that it was, in any case, a better idea to store file data in chunks, and re-use chunks in different generations as needed. This makes it much easier to manage changes to files: with deltas, you need to keep a long chain of deltas and apply many deltas to reconstruct a particular version. With lists of chunks, you just get the chunks you need. The spin-off franchise; lost in a maze of dependencies, all alike In the process of developing Obnam, I have split off a number of helper programs and libraries:

• genbackupdata generates reproducible test data for backups
• seivot runs benchmarks on backup software (although only Obnam for now)
• cliapp is a Python framework for command line applications
• cmdtest runs black box tests for Unix command line applications
• summain makes diff-able file manifests (md5sum on steroids), useful for verifying that files are restored correctly
• tracing allows run-time selectable debug log messages that is really fast during normal production runs when messages are not printed

I have found it convenient to keep these split off, since I've been able to use them in other projects as well. However, it turns out that those installing Obnam don't like this: it would probably make sense to have a fat release with Obnam and all dependencies, but I haven't bothered to do that yet. The blurb; readers advised about blatant marketing The strong points of Obnam are, I think:

• Snapshot backups, similar to btrfs snapshot subvolumes. Every generation looks like a complete snapshot, so you don't need to care about full versus incremental backups, or rotate real or virtual tapes. The generations share data as much as possible, so only changes are backed up each time.
• Data de-duplication, across files, and backup generations. If the backup repository already contains a particular chunk of data, it will be re-used, even if it was in another file in an older backup generation. This way, you don't need to worry about moving around large files, or modifying them.
• Encrypted backups, using GnuPG.

Backups may be stored on local hard disks (e.g., USB drives), any locally mounted network file shares (NFS, SMB, almost anything with remotely Posix-like semantics), or on any SFTP server you have access to. What's not so strong is backing up online over SFTP, particularly with long round trip times to the server, or many small files to back up. That performance is Obnam's weakest part. I hope to fix that in the future, but I don't want to delay 1.0 for it. The big news; readers sighing in relief I am now ready to release version 1.0 of Obnam. Finally. It's been a long project, much longer than I expected, and much longer than was really sensible. However, it's ready now. It's not bug free, and it's not as fast as I would like, but it's time to declare it ready for general use. If nothing else, this will get more people to use it, and they'll find the remaining problems faster than I can do on my own. I have packaged Obnam for Debian, and it is in unstable, and will hopefully get into wheezy before the Debian freeze. I provide packages built for squeeze on my own repository, see the download page. The changes in the 1.0 release compared to the previous one:

• Fixed bug in finding duplicate files during a backup generation. Thanks to Saint Germain for reporting the problem.
• Changed version number to 1.0.

The future; not including winning lottery numbers I expect to get a flurry of bug reports in the near future as new people try Obnam. It will take a bit of effort dealing with that. Help is, of course, welcome! After that, I expect to be mainly working on Obnam performance for the foreseeable future. There may also be a FUSE filesystem interface for restoring from backups, and a continous backup version of Obnam. Plus other features, too. I make no promises about how fast new features and optimizations will happen: Obnam is a hobby project for me, and I work on it only in my free time. Also, I have a bunch of things that are on hold until I get Obnam into shape, and I may decide to do one of those things before the next big Obnam push. Where; the trail of an errant hacker I've developed Obnam in a number of physical locations, and I thought it might be interesting to list them: Espoo, Helsinki, Vantaa, Kotka, Raahe, Oulu, Tampere, Cambridge, Boston, Plymouth, London, Los Angeles, Auckland, Wellington, Christchurch, Portland, New York, Edinburgh, Manchester, San Giorgio di Piano. I've also hacked on Obnam in trains, on planes, and once on a ship, but only for a few minutes on the ship before I got seasick. Thank you; sincerely

• Richard Braakman, for helping me with ideas, feedback, and some code optimizations, and for doing the startup with me. Even though he has provided little code, he's Obnam's most significant contributor so far.
• Chris Cormack, for helping to build Obnam for Ubuntu. I no longer use Ubuntu at all, so it's a big help to not have to worry about building and testing packages for it.
• Daniel Silverstone, for spending a Saturday with me hacking Obnam, and rewriting the way repository file filters work (compression, encryption), thus making them not suck.
• Tapani Tarvainen for running Obnam for serious amounts of real data, and for being patient while I fixed things.
• Soile Mottisenkangas for believing in me, and helping me overcome periods of despair.
• Everyone else who has tried Obnam and reported bugs or provided any other feedback. I apologize for not listing everyone.

SEE ALSO

• Obnam home page
  • tutorial
  • support
  • NEWS
  • README
  • manual page
  • design documents
  • Debian QA package page
  • bugs
  • bugs in Debian
• Other projects of mine (many are dependencies of Obnam)

• Latvian and Lithuanian complete level 2
• Lithuanian completes level 3
• Tamil completes level 1
• Galician progress in level 1

Status for D-I level 1 (core D-I files):

• 52 languages 100%: ar ast be bg bn bs ca cs da de el eo es et eu fa fi fr gu he hi id it ja kk km kn ko lo lv mr nb nl pa pl pt pt_BR ru si sk sl sr sv ta te th tr ug uk vi zh_CN zh_TW
• 3 languages 97%: ga hu lt
• 4 languages 95%: dz is mk ro
• 1 language 94%: gl
• 1 language 92%: ml
• others are 90% or below

Status for D-I level 2 (packages that have localized material that may appear during default installs, such as iso-codes, tasksel, etc.):

• 33 languages 100%: ast be bg ca cs da de eo es fa fr gu he id is it ja kk km lt lv mr nl pl pt ru si sk sl sv tr uk zh_CN
• 7 languages 99%: eu fi hr nb th vi zh_TW
• 3 languages 98%: dz pt_BR ro
• 1 language 97%: ta
• 1 language 96%: ug
• 3 languages 94%: ar el gl
• 1 language 92%: sr
• 5 languages 92%: bn bs hu ko ne
• others are 90% or below

Status for D-I level 3 (packages that have localized material that may appear during non-default installs, such as win32-loader)

• 39 languages 100%: ast be bg bs ca cs da de el eo es fa fi fr ga gl he hr id is it ja kk km lt nb nl pl pt ru sk sl sr sv th tr vi zh_CN zh_TW
• 3 languages 98%: hu ug uk
• 1 language 95%: lv
• others are 90% or below

Full 100% completeness (hall of fame) for 26 languages: Asturian, Belarusian, Bulgarian, Catalan, Czech, Danish, German, Esperanto, Spanish, Persian, French, Hebrew, Indonesian, Italian, Japanese, Kazakh, Khmer, Dutch, Polish, Portuguese, Russian, Slovak, Slovenian, Swedish, Turkish, Simplified Chinese

If you want to add System z zFCP drives to a Debian GNU/Linux system, you first need to make the HBA known to the system. For this you add an (initially empty) file with the (lowercase) device ID of the zFCP HBA to /etc/sysconfig/hardware. There should already be files for the ECKD/FBA disks and the network adapters.

Then you need to list the drives to initialize by placing the following array in it with a list of WWPN:LUN, seperated by spaces within the parentheses:

ZFCP_DEVICES=(0x1234567890abcdef:0x4000400000000000)

Please keep in mind that the WWPN and LUN need to be specified in hex and again entirely in lower case. This will cause hwup ccw 0.0.4000 (with 4000 being the CHPID) to instruct the HBA to add the drive after setting it online. Then you should regenerate your initrd, so that it happens on-boot, by running update-initramfs -u -k all.

With current kernels the available WWPNs should be probed and listed in /sys/bus/ccw/devices/0.0.4000 without further intervention after the HBA is initialized.

Sadly there's no way to set up a zFCP disk in debian-installer.

Today, as an experiment, I attempted to fix a Debian bug while on the train to work. I use a 3G card from Three.co.uk in my Lenovo Thinkpad x121e, and my commute is from Southampton Central to Fleet (changing at Winchester) - just under an hour. 3G coverage is not 100%, but tends to be better around the major stops.

• First, I found a bug. I used udd.debian.org to browse for a relatively simple RC bug, and found bug #674992 in actionaz. The fix was outlined in the report already, so there was very little thinking required.
• Next, I confirmed the FTBFS using cowbuilder. Unfortunately, this required downloading roughly 120MB of dependencies - I have 1GB of data per month, but I couldn't afford to do this every day. I was lucky in that I was near Basingstoke at the time, so had a good HSDPA signal to get the bulk of this. The build had failed before I reached Fleet.
• In the background, I updated debian/control and debian/changelog with the fix. I was able to set off the build, but had to suspend the laptop until lunchtime before it could finish. Cowbuilder needed to download only a few extra build-deps, as the vast majority were cached from the initial run.
• On the train home, I checked over the result, signed it and uploaded. In this instance, the built package was small enough to upload, but I could see this being a problem with others.
• Finally, I sent the nmudiff, although that was delayed briefly by a drop in connectivity before Southampton Airport.

Thoughts: firstly, part of me is amazed that this is possible. Secondly, there could be a case for a local Debian mirror on my laptop. Otherwise, an interesting experimental extension to UDD would be "Required bandwidth" - the sum of the recursive build-dependencies plus the upload size of the diff/binaries.

Jo and I flew in to HK a few days early for the Linaro conference, so we got some time to ourselves for doing touristy stuff. That was a good plan - we got to see quite a lot of Kowloon for a couple of days, then we wandered over to Hong Kong island itself for a day. Jo also had more time during the week while I was in the sessions at the conference. Hong Kong is a busy place, very densely packed at all times of day and night that we could see. The first impression we got on the taxi from the airport on day one was that it was a city full of tower blocks - huge numbers of apartments stacked very closely together for living space. That makes sense, I guess, with the large population in such a small space. There are houses around with more space for people to live, but they are understandably very expensive. Later on in Kowloon, we got to meet lots of the local street vendors, mostly offering us "Copy Rolex", "Fake handbag" and the like. Even "Sharp trousers" at one point. We decided not to try them out... :-) We saw lots and lots of very expensive shops downtown, crammed into huge shopping centres in the tourist area. As an antidote for all of that, we headed up to the "Golden Computer Arcade" to go browsing around the vast array of tiny technology stalls and shops there. Picked up some nice cheap laptop memory and some SD cards there. Yay! Across on Hong Kong island itself on Sunday, we found my grandfather's grave then rode around on the trams for a while to get a good view of the city, including a democracy protest that happened to be on that day! We had a quick look around Tin Hau Temple, where we found a very specific set of rules for visitors:

I can only guess that they must have been terrified by gangs of kids racing their RC cars at one point! We finished the day by heading up to Victoria Peak for the sunset view and some dinner, then back to the hotel. Killing time in a shopping centre waiting for the bus to turn up, we found a truly astonishing little item on sale, a Solar Queen!

Apparently this gift has a solar panel on the handbag to provide power to a little motor in the Queen's hand to make her wave to her subjects. We decided that we'd somehow have to live without such a delight in our lives... :-) HK was a cool place to visit. It still feels very British in some ways, as you'd expect from an ex-colony (power sockets are still the same as ours, driving on the left, everybody spoke English, etc.) but utterly alien in others (the climate, working public transport *grin*). I look forwards to heading out there again!

I never met my father's father, as he died many years before I was born. He was a soldier in the British Army and was killed in an accident in Burma in the 1950s. I'm told that back then the Army did not repatriate casualties as a matter of course like they do today, so none of the family got to see his funeral or even visit the grave. Like many of the military casualties in the Far East, he was buried in the Colonial Cemetery in Hong Kong. A few photographs were sent to my grandmother of the ceremony and the headstone, but that's all we ever saw.

Fast forward to 2012. I travelled to Hong Kong for a conference (Linaro Connect, more about that later...). I looked up the details of where grandad was laid to rest, and found a major coincidence. He was killed on 27th May 1952, almost exactly 60 years ago! I resolved to go to find him on 27th May 2012 to make the most of this lucky anniversary. Wikipedia told me that in the intervening years the Colonial Cemetery had been renamed to Happy Valley Cemetery, then simply Hong Kong Cemetery, and repurposed from military to civilian use. Jo and I bought some flowers and headed over to Happy Valley on Hong Kong Island on Sunday afternoon. We found a very large cemetery with a helpful map posted to show the different sections, but nothing to tell us where we should be looking. One of the attendants at the office on site looked at the details we had and shrugged to say "sorry, can't help". Ah well, nobody said this was going to be very easy... What we found after a few minutes was that the graves weren't laid out in any obviously logical fashion. In any given small area, you'd find people buried from roughly the same period, but people for any given period could (and would) be scattered across multiple sections at all corners of the site. Great, time for an exhaustive search then. I started combing the site, checking all the markers I could find that looked anything like the tiny grainy black and white photo we had of grandad's headstone.

Almost two hours later, I eventually found him. Two hours of heavy work: the cemetery is built on the side of quite a steep hill, and the prevailing weather was very hot with 100% humidity. But, I forgot all that as we eventually stumbled across the correct grave in (no exaggeration!) the very last part of the last section of the site, 20a.

I was relieved at this point: as things had taken so long, I had started to worry that maybe the grave had been moved or the headstone damaged and lost. But no, I found the man who had gone off to war leaving my dad and aunt as small children. We took some photos and took note of where we had found the final resting place of Sergeant William Alexander McIntyre of the Royal Signals, a man I never met in life but clearly a very important member of our family.

My curiosity was definitely nookd by this story. While this doesn't seem to have been an act of censorship or anything nefarious by Barnes and Noble itself, this does point to just how terrifying ebooks can be if we aren't careful. In the future, when we are dealing with the entirety of human knowledge, we better be careful with the find and replace function. And we better make sure we control our computer systems so that other people can't nook us without our permission. See also:

• War and Nookd
• Nookd

As someone who read War and Peace in paper form, much of it in bed, I definitely empathized with this:

Some weeks ago I decided that I wanted to read Tolstoy's War and Peace. Lou Ann loaned me her copy. At more than 1100 pages, reading it in bed required as much strength as balancing a box of bricks in my hands. In my senior years I have developed arthritis in my thumbs, which made the effort not only difficult, but painful.

A lot of people interested in free software, and user autonomy and network services are very worried about Facebook. Folks are worried for the same reason that so many investors are interested: the networks effects brought by hundreds of millions of folks signed up to use the service. Network effects -- the concept that a good or service increases in value as more people use it -- are not a new problem for free software. Software developers target Microsoft Windows because that is where the large majority of users are. Users with no love for Microsoft and who are otherwise sympathetic to free software use Windows because programs they need will only run there. Folks worried about Facebook are afraid for similar reasons. Sure, you can close down your Facebook account and move to Diaspora. But who will you talk to there? You can already hear people complaining about Facebook the same way they've been complaining about Windows or Office for years. People feel that their hands are tied and that their software, and their social network, will be determined by what everybody is doing. I'm worried about Facebook. But I'm not too intimidated by Facebook's network effects for two reasons. First, using Facebook doesn't preclude using anything else. Twitter has enormous overlapping functionality with Facebook. Sure, people use the systems very differently. But they both ask you to create lists of friends and followers and are designed around sending and receiving short status messages. Millions of people do both and both systems are thriving. For the millions of people who use both Facebook and Twitter, the two services have had to negotiate their marginal utility in a world they share with the other one. People decide that Twitter is for certain types of short messages and Facebook is for others. But these arrangements shift over time. And the relationships between services aren't always peaceful coexistence. Remember Friendster? Remember Orkut? Remember Tribe? Remember MySpace? MySpace, and all the others, are great examples of how social networks die. They very slowly fade away. MySpace users signed up for Facebook accounts and used both. They almost never just switched. Over time, as one platform became more attractive than the other, for many complicated reasons, attention and activity shifted. People logged in on MySpace less and Facebook more and, eventually, realized they were effectively no longer MySpace users. Anyone that has been on the Internet long enough to watch a few of these shifts from one platform to another knows that they're not abrupt -- even if they can be set in motion by a particular event or action. Users of social networking sites simply don't have to choose in the way that a person choosing to boot Windows and GNU/Linux does. I'm sure the vast majority of people with Diaspora accounts use Facebook actively. This is not a problem for Diaspora. It is how Diaspora -- or whatever else eventually achieves what many of us hoped Diaspora would -- could win. Second, Facebook is for the ephemeral. Facebook is primarily used for information that was produced very recently. This week if not today. If not this hour. Facebook has an enormous amount of data that users have fed it that may be hard to get out and move somewhere else. But most people don't care very much about having any regular access to the large majority of this information. What people care deeply about is having access to the data that they and their friends created today. And that data can just as easily be created somewhere else tomorrow. Or, with the right tools, created just as easily in both places. Compare this to something like Windows where moving away would require learning, converting, and perhaps even writing, new software. Perhaps even in new programming languages that most developers don't know yet. Compared to Windows, a migration away from Facebook will be easy. Facebook's photo galleries are an example of an important place where this holds less well. Social network information -- i.e., the list of who is friends with who -- is another example of something that is persistently valuable. That said, people really enjoy the act of finding and friending. Indeed, this process was part of the initial draw of Facebook and other social networks. None of this means that Facebook is over. It doesn't even mean that its ascendancy will be slowed. What it does mean is that Facebook is vulnerable to the next thing more than many technology firms that have benefited from network effects in the past. If users are given compelling reasons to switch to something else, they can with less trouble and they will. That compelling reason might be a new social network with better features or an awesome distributed architecture that allows freedom for users and the ability of those users to benefit from new and fantastic things that Facebook's overseers would never let them have and without the things Facebook's users suffer through today. Or it might be a sexier proprietary box to store users' private information. It doesn't mean that I'm not worried about Facebook. I remain deeply worried. It's just not very hard for me to imagine the end.

since most of the gcc 4.7 bugs are fixed by now, I looked again at the general RC bugs too this week. here's the usual short report:

• #650060 snort: "snort: FTBFS with multiarch libmysqlclient-dev"
  apply patch from Ubuntu / Jean-Louis Dupond, upload to DELAYED/2, then used in a maintainer upload
• #664933 src:fceu: "fceu: FTBFS: file.c:157:20: error: request for member 'have' in something not a structure or union"
  add patch from Sebastian Ramacher, upload to DELAYED/2
• #666225 src:deja-dup: "configure:3028: error: xmllint not found"
  update build dependencies, upload to DELAYED/2
• #666337 src:euler: "euler: FTBFS: gcc: error: unrecognized option '-V'"
  sponsor NMU prepared by Sebastian Ramacher, upload to DELAYED/2
• #667111 bcov: "bcov: ftbfs with GCC-4.7"
  second attempt, extend patch, upload to DELAYED/1
• #667391 telepathy-qt4: "telepathy-qt4: ftbfs with GCC-4.7"
  add patch from Cyril Brulebois, upload to DELAYED/2
• #667886 src:textedit.app: "textedit.app: FTBFS with GCC-4.7"
  apply patch from Yavor Doganov, upload to DELAYED/2
• #672356 libspread-perl: "libspread-perl: DynaLoader now requires versions to be numbers, Spread.pm uses a string."
  add patch from Andrew Ruthven, do a QA upload
• #674330 src:clamav: "clamav: FTBFS: llvm/lib/ExecutionEngine/JIT/Intercept.cpp:69:67: error: 'lseek64' was not declared in this scope"
  apply upstream patch (missing include), upload to DELAYED/2

When packaging software for Debian, there exist two important assumptions:

1. Compilation is done natively
2. Potentially all of Debian is available at compile time

Both assumptions make the life of a package maintainer much easier and they do not create any problem unless you are one of the unlucky few who want to run Debian on an architecture that it does not yet exist for. You will then have to use other distributions like OpenEmbedded or Gentoo which you compiled (or retrieved otherwise) for that new architecture to hack a core of Debian source packages until they build a minimal Debian system that you can chroot into and continue natively building the rest of it. But even if you manage to get that far you will continue to be plagued by cyclic build and runtime dependencies. So you start to hack source packages so that they drop some dependencies and you can break enough cycles to advance step by step. The Debian ports page lists 24 ports of Debian, so despite its unpleasant nature, porting it is something that is not done seldom. The process as laid out above has a number of drawbacks:

• The process is mostly manual and reinvented every time it is done.
• You need another distribution for the bootstrapping process. Debian itself should be sufficient.
• Its complexity and manual nature prevents architectures with little workforce behind them from catching up to the main archive.
• It also avoids that Debian exists in CPU optimized sub-arch builds.

If Debian would provide a set of core packages that are cross-compilable and which suffice for a minimal foreign build system, and if it would also have enough source packages that provide a reduced build dependency set so that all dependency cycles can be broken, building Debian for a yet unknown architecture could be mostly automated. The benefits would be:

• Putting Debian on a foreign architecture would (in the best case) boil down to making the code cross-compile for and native-compile on that architecture.
• Debian would not need any other distribution to be ported to a different architecture. This would make Debian even more "universal".
• Lagging architectures can be more easily updated or rebooted than when they were initially created.
• Debian optimized for specific CPUs (Raspberry Pie, OpenMoko...) would be more attractive.

With three of this year's GSoC projects, this dream seems to come into reach. There is the "Multiarch Cross-Toolchains" project by Thibaut Girka and mentored by Hector Oron and Marcin Juszkiewicz. Cross-compiling toolchains need packages from the foreign architecture to be installed alongside the native libraries. Cross-compiler packages have been available through the emdebian repositories but always were more of a hack. With multiarch, it is now possible to install packages from multiple architectures at once, so that cross-compilation toolchains can be realized in a proper manner and therefor can also enter the main archives. Besides creating multiarch enabled toolchains, he will also be responsible for making them build on the Debian builld system as cross-architecture dependencies are not yet supported. There is also the "Bootstrappable Debian" project by Patrick "P. J." McDermott and mentored by Wookey and Jonathan Austin. He will make a small set of source packages multiarch cross-compilable (using cross-compilers provided by Thibaut Girka) and add a Build-Depends-StageN header to critical packages so that they can be built with reduced build dependencies for breaking dependency cycles. He will also patch tools as necessary to recognize the new control header. And then there is my project: "Port bootstrap build-ordering tool" (Application). It is mentored by Wookey and Pietro Abate. In contrast to the other two, my output will be more on the meta-level as I will not modify any actual Debian package or patch Debian tools with more functionality. Instead the goal of this project is threefold:

1. find the minimal set of source packages that have to be cross compiled
2. help the user to find packages that are good candidates for breaking build dependency cycles through added staged build dependencies or by making them cross-compilable
3. develop a tool that takes the information about packages that can be cross compiled or have staged build dependencies to output an ordering with which packages must be built to go from nothing to a full archive

More on that project in my follow-up post.

Traffic jams suck; but they do give you a chance to think and blog. Truckers hams are telling me that four trucks are totalled and need to be removed by the THW. As the trucker next to me just broke out his gas grill and several people are joining him in a spontaneous BBQ, I don't expect to be moving any time soon. Kicking things along This is coming along nicely RAID 4 Unless you are forced to support legacy systems or playing RAIDemon and gotta try them all, there is no valid use case for RAID 4. So doing so is a patently bad idea. RAID 5 is always the better option as you are not relying on a single disk which contains the checksums, but use striping to distribute them. RAID 4 has a chance of 1/n that a failed disk will force all remaining disks to read all respective sectors and then the controller/CPU to XOR everything. There is a reason why RAID 3 was replaced with RAID 4; just as there's a reason why RAID 4 was replaced by RAID 5. Don't use RAID 4. Seriously. Tom Tom vs OSM In case you have been following the news in the GPS circles, you will have noticed that TomTom pretty gave up on innovating and resigned themselves to dying a slow death. Well that and to start a nice amount of FUD. While they have been rebuked pretty thoroughly already, I wanted to add my own POV. There is a kernel of truth in what TomTom said inasmuch they state that pedestrian paths are better than car paths. Of course, while they got the relative quality right, they are wrong about the absolute quality of the data. They claim pedestrian is OK and motorized is bad. In my experience, maps for motorized traffic are good while pedestrian tracks are mind-boggingly great. These days, most people (and probably all who read this) will be at least somewhat familiar with the quality of OSM in the random cities they visit over the year. The common verdict is usually "very nice". But that's not where OSM shines. It's in the lesser-known places where you don't expect too many tech-minded people to come along where it starts to really impress me. While I found a few stubs on a dead-end street in Akureyri which were not mapped, every foot trail I ever walked along in Iceland has been mapped. ski-doo tracks on Svalbard? Check. Backyard in Novosibirsk? Check, as well. And we are talking "packed earth track in between garages opening up into a 'playground' where a man and a pregnant women are passing a bottle back and forth while smoking and watching over their two kids who play next to a middle-aged man who is lying on the ground wasted, using a rock as a pillow all the while an extremely blingy Mercedes with two muscle-packed thugs is cruising by, all in the time it took us to walk through". Mapped. Horse tracks and hiking trails in Mongolia? Yep. Hutongs in Beijing including alleys so narrow you have to walk single file? Of course. Richard Fairhurst is right that the plural of anecdote is not data, but OSM is good. TomTom is afraid for a reason. As an aside, I find the picture used in TomTom's newsletter extremely funny; I am sure they didn't intend this, but hey. Windows 100-300 MiB for any random driver installation package is par for the course. Be it GFX, audio, or network card. If you plug in a different USB mouse, Windows 7 will take several seconds while "setting up" a standard USB HID device. And hitting ctrl-alt-del in the login screen or while logged in does not give you a quick way to shut down or reboot the machine. As I am breaking out the Windows installation every month to install updates and every six months to actually use this does not affect me a lot. Still, it's amazing that so many people are putting up with this. UEFI It's nice to see that Matthew Garrett is still going strong. I am not sure if this is the best way, but it seems to be a workable solution while in between a rock and a hard place. Not sure about the requirements of GPLv3 with regards to the keys Fedora will use to sign GRUB 2 with. From what I remember without rereading the license, as long as Fedora is not selling a system on which you can not disable secure boot they should be in the clear. IANAL, IIRC, etc. Wrapping up And I have not moved a single meter, other than making way to a police motorbike, in over an hour. Pity I don't have anything to BBQ with me; smells delicious over there.

I'm participating on organizing this year's openSUSE conference and as we've finally finalized place and time and CFP is open, it's time to share some information. This will be special - there is not a single conference, but four of them are sharing same space and the motto - Bootstrapping awesome!!!. The 4th openSUSE conference will be held together with LinuxDays, Czech conference following tradition of canceled LinuxExpo, what should be the biggest free software event in Czech republic. In addition there will be also 12th SUSE Labs conference (so you can meet quite a lot of kernel hackers and other strange guys) and first Gentoo mini summit (the website is empty so far). All that will happen on weekend from 20th October to 21st October, the SUSE conferences will then continue for 22nd and 23rd October. I believe it will be great mixture of conferences and I hope to meet lot of people there.

Filed under: English Gentoo Suse 0 comments Flattr this!

• The "Save the baltic languages" week..:-)
• Latvian translations completed for level 1. Latvian won't be deactivated!
• Also progress in levels 2 and 3 for Latvian
• Lithuanian progress for level 1. Language won't be deactivated!
• Estonian was already safe..:-)

Status for D-I level 1 (core D-I files):

• 51 languages 100%: ar ast be bg bn bs ca cs da de el eo es et eu fa fi fr gu he hi id it ja kk km kn ko lo lv mr nb nl pa pl pt pt_BR ru si sk sl sr sv te th tr ug uk vi zh_CN zh_TW
• 1 languages 99%: ta
• 2 languages 97%: ga hu
• 5 languages 95%: dz is lt mk ro
• 1 language 92%: ml
• others are 90% or below

Status for D-I level 2 (packages that have localized material that may appear during default installs, such as iso-codes, tasksel, etc.):

• 31 languages 100%: ast be bg ca cs da de eo es fa fr gu he id is it ja kk km mr nl pl pt ru si sk sl sv tr uk zh_CN
• 8 languages 99%: eu fi hr lv nb th vi zh_TW
• 3 languages 98%: dz pt_BR ro
• 1 language 97%: ta
• 1 language 96%: ug
• 3 languages 94%: ar el gl
• 1 language 92%: sr
• 5 languages 92%: bn bs hu ko ne
• others are 90% or below

Status for D-I level 3 (packages that have localized material that may appear during non-default installs, such as win32-loader)

• 38 languages 100%: ast be bg bs ca cs da de el eo es fa fi fr ga gl he hr id is it ja kk km nb nl pl pt ru sk sl sr sv th tr vi zh_CN zh_TW
• 3 languages 98%: hu ug uk
• 1 language 95%: lv
• others are 90% or below

Full 100% completeness (hall of fame) for 26 languages: Asturian, Belarusian, Bulgarian, Catalan, Czech, Danish, German, Esperanto, Spanish, Persian, French, Hebrew, Indonesian, Italian, Japanese, Kazakh, Khmer, Dutch, Polish, Portuguese, Russian, Slovak, Slovenian, Swedish, Turkish, Simplified Chinese

Managing the puppet manifest using a vcs is a best practice and there is a lot of material on the web. The easier way to do it, is to use git directly in the directory /etc/puppet and use a simple synchronization strategy with an external repo, either to publish your work, or simply to keep a backup somewhere. Things are a bit more complicated when you would like to co-administer the machine with multiple people. Setting up user accounts, permission and everything can be a pain in the neck. Moreover working from your desktop is always more comfortable then logging in as root on a remote system and make changes there ... The solution I've chosen to make my life a bit easier is to use gitolite, that is a simple git gateway that works using ssh public keys for authentication and does not require the creating of local users on the server machine. Gitolite is available in debian and installing it is super easy : apt-get install gitolite . If you use puppet already you might be tempted to use puppet to manage your gitolite installation. This is all good, but I don't advice you to use modules like this one http://forge.puppetlabs.com/gwmngilfen/gitolite/1.0.0 as it's going to install gitolite from source and on debian it's not necessary ... For my purposes, I didn't find necessary to manage gitolite with puppet as all the default config options where good enough for me. Once your debian package is installed, in order to initialize your repo, you just need to pass to gitolite the admin public key, that is your .ssh/id_rsa.pub key and then run this command: This will create the admin and testing repo in /var/lib/gitolite/repositories and setup few other things. At this point you are ready to test you gitolite installation by cloning the admin repo : Gitolite is engineered to use only the gitolite use to manage all your repositories. To add more repositories and users you should have a look at the documentation and then editing the file conf/gitolite.conf to add your new puppet repository. At this point, you can go two ways. If you use git to manage your puppet directory, you can just make a copy of it somewhere and then add gitolite as a remote If you didn't use git before, you can just copy the manifest in your new git repository, make a first commit and push it on the server. Every authorized users can now use git to clone your puppet repository, hack, commit, push ... One last step is to add a small post-receive hook on the server to synchronize your gitolite repository with the puppet directory in /etc : This will sync your main puppet directory and trigger changes on the nodes for the next puppetd run. First I created a small shell script in /usr/local/bin/puppet-post-receive-hook.sh : This script presuppose that your git repo in /etc/puppet has the gitolite repo as origin ... Then I added a simple hook in the gitolite git repo that calls this script using sudo : And since you are at it you should also add a pre-commit hook to check the manifest syntax. This will save you a lot of useless commits. If you use a more complicated puppet setup using environments (I'm not there yet, and I don't think my setup will evolve in that direction in the near future), you can use puppet-sync that seems a neat script to do the job. For the moment this setup works pretty well. I'm tempted to explore mcollective to trigger puppet runs on my nodes, but I'm there yet...

• Finnish completes level 1 and makes progress in level 2
• Danish, Persian, Hebrew are full 100% (they were indeed forgotten in last list, thanks to Phil Miller for pointing)
• Uyghur now finally listed in stats, as well as Tibetan. Sorry, the language not appearing was because I forgot a commit pending somewhere in my local copy of D-I repository.
• Marathi completes level 2
• console-data dropped from level 2 as it is not used anymore in D-I

Status for D-I level 1 (core D-I files):

• 50 languages 100%: ar ast be bg bn bs ca cs da de el eo es et eu fa fi fr gu he hi id it ja kk km kn ko lo mr nb nl pa pl pt pt_BR ru si sk sl sr sv te th tr ug uk vi zh_CN zh_TW
• 1 languages 99%: ta
• 2 languages 97%: ga hu
• 4 languages 95%: dz is mk ro
• 1 language 92%: ml
• others are 90% or below

Status for D-I level 2 (packages that have localized material that may appear during default installs, such as iso-codes, tasksel, etc.):

• 31 languages 100%: ast be bg ca cs da de eo es fa fr gu he id is it ja kk km mr nl pl pt ru si sk sl sv tr uk zh_CN
• 7 languages 99%: eu fi hr nb th vi zh_TW
• 3 languages 98%: dz pt_BR ro
• 1 language 97%: ta
• 1 language 96%: ug
• 3 languages 94%: ar el gl
• 1 language 92%: sr
• 5 languages 92%: bn bs hu ko ne
• others are 90% or below

Status for D-I level 3 (packages that have localized material that may appear during non-default installs, such as win32-loader)

• 38 languages 100%: ast be bg bs ca cs da de el eo es fa fi fr ga gl he hr id is it ja kk km nb nl pl pt ru sk sl sr sv th tr vi zh_CN zh_TW
• 3 languages 98%: hu ug uk
• others are 90% or below

Full 100% completeness (hall of fame) for 26 languages: Asturian, Belarusian, Bulgarian, Catalan, Czech, Danish, German, Esperanto, Spanish, Persian, French, Hebrew, Indonesian, Italian, Japanese, Kazakh, Khmer, Dutch, Polish, Portuguese, Russian, Slovak, Slovenian, Swedish, Turkish, Simplified Chinese

Third post about ganeti. Ganet-debootstrap-instance contains a nice set of scripts to create a debian (or derivatives) image using debootstrap. Images can be configured and customized by writing simple hooks script to modify various aspects of the default installation. However writing these script is not really fun and pushing it too far can lead to long messy scripts, loosing the overall benefit of automatic configuration. Puppet is my configuration management tool of choice, but installing puppet on a new machine requires few magic incantations that the user should perform manually, or in a semi automatic mode (autosign=true) to make it work. My goal is to install puppet automatically on the newly created instance so it will run and configure the new instance at the first boot. From that moment on I'll forget about ganeti and configure all remaining services of my new VM using puppet. In order to do so, we need to install puppet (and apt-get update/upgrade...), create the ssl certificates for the client and enabling the puppet daemon on the client. We add another hook in /etc/ganeti/instance-debootstrap/hooks/ : This script uses puppetca to create on the puppet (and ganeti) server the client key, sign it, and then copy it to the target machine. Notice that we create the certificate for a fqnd name $INSTANCE_NAME.$DOMAIN or otherwise puppet will complain loudly. This is not strictly needed, but if you want to do otherwise, you'll need to fiddle with the puppet configuration a bit more. The procedure to create a puppet certificate server-side is well documented on the puppet website, so if you are curious about the details duck-duck-it .