I am finally catching up with the remarkably thick goulash of email and blogs entries that comes from traveling for well over a month and today I read one in particular When is Open Source not Open Source? that captivated my interest for it compellingly addresses the dangers that follow from diluting, or one might say hijacking, the term open source. When people learn that I study “free software” one of the most common questions I get asked is: why did I chose free software over open source? The answer is quite simple: given that the bread and butter of my research covers ethics, freedom, and liberalism, free software is the obvious path to follow, yet I also feel like a lot of my work is still relevant to the open source camp because of the affinities between the two. I have long maintained that the ideological gulf between open source and free software is not so great nor impassable, but more modest. As most know, both share a certain strong commitment to access and in a strict technical sense they refer to the same set of licenses. Philosophically there is agreement that openness and, especially, non-discrimination are essential for the quality of software and often by close extension, the vibrancy of community responsible for the software. Of course, when pontificating the ramifications and implications of openness, they do part company and enter into different territories. Free software tends to flag rights and freedoms, while open source meanders into a discussion of markets, business, and competition and in this regard they do craft different visions of the social world and human behavior, etc. But the case that Karl Fogel writes about, where OSI is strongly opposing the use of the term open source for licenses that don’t adhere to the definition demonstrates where the two positions join. As Michael Tiemann from the OSI succinctly put it:

The FSF may have got the orthodoxy wrong, and the OSI may have got the interpretation wrong, but we both agree that prohibition of commercial use without special permission is antithetical to both positions.

There is a unmistakable kernel of agreement and it is great to see the OSI taking such a strong stance in this regard. Now, David Richard’s response, who seeks, I think, to essentially dilute the term open source, is as (or perhaps even more) fascinating for in a nut shell, and using a lot of florid religious imagery, it accuses the OSI of being too rigid! In his own words:

I believe the OSI has a wonderful opportunity to continue being relevant and helping to lead the movement forward. If, however, y’all choose to define your denomination of this religion in a way that we don’t fit in, that’s fine. No hard feelings. It’s your choice. You’ll ultimately be excluding a large congregation and we for one will continue trying to build a church made up of others like ourselves.

In response, I would say that the goal of F/OSS is not to be inclusive of anyone who wants to release bits of source code, but to create the conditions under which software, as it has been defined by the community, can be created. Join the church if you would like to make free/open source software as defined and you can go elsewhere (i.e., create a different term) if you are creating something different, even if it is only slightly different. Integrity matters. And again inclusiveness, if it comes at the expense of the main goal, is not a boon but a danger to F/OSS. The OSI will remain relevant by halting the dilution of the term OSI, not by expanding the definition so that it is left with no substance. And in contradistinction to what David Richard maintains, however, there is a great degree of flexibility within this domain but it does not lie in the strict definition of F/OSS but in the realm of interpretation. You are also free, as Mako and I have argued elsewhere to interpret the significance of F/OSS in multiple ways. And I think this is where the political strength of free software lies. There is interplay between a well-defined goal (in this case for creating free software) and a more flexible realm of interpreting the significance of these technical practice. And we wold lose and I might add, a lot if we became flexible about the strict definition of F/OSS and inflexible about its political significance. I get irked with folks like David Richards who would like to bend open source rules to meet their (often commercial) interests and I find it pretty na ve when folks say the political significance of F/OSS is just x (or worse should be x) for in reality its political significance lies in the fact that it has spawned multiple types of political and economic projects. And there is something almost playfully ironic, (or at least it makes me smile) in this fact. Though there is strict definition contained withing F/OSS, this strictness has, at least to some extent, encouraged by an extreme and very healthy form of political proliferation and promiscuity. More than anyone else I know, Mako has most passionately and thoughtfully argued for the importance of what I would call political clarity and integrity. That is, the importance of having a well articulated definition for social movements, for they act, as he says a rallying point to realize a social movement. Urging the Creative Commons to learn from F/OSS and dare to simultaneously narrow and more clearly define their goals, he states it quite nicely in the following terms:

Free software advocates have been able to use the free software definition as the rallying point for a powerful social movement. Free software, like the concept of freedom in any freedom movement, is something that one can demand, something that one can protest for, and something that one can work toward. Working toward these goals, free and open source software movements have created the GNU/Linux operating system and billions of lines of freely available computer code.

In essence, a definition that people can abide by, respect, and perhaps eventually cherish is the condition of possibility to make working political code. And given how hard it is to make social change happen (at least in comparison to build computer code), we should learn from what F/OSS has to offer. And at the same time there is another lesson embedded in F/OSS. The Free Software Definition is well defined; but it must be emphasized, narrowly so. It does not try to do everything and have everyone pledge allegiance to an inordinately complex set of commitments. Clarity, narrowness, and well-defined goals –> these three attributes have powered it far and wide and I hope it remains so. Now, since the term open source is not trademarked, we are left with the problem of how to challenge the current hijacking of the term. For the solution, I will leave you with Karl Fogel, who I think proposes a good solution:

Note that the OSI s objection is not to the Zimbra license per se. The objection is just to Zimbra s calling that license open source . They can use any license they want, but they shouldn t call it open source unless it actually is. Freedom is freedom, and no amount of spin will change that. So what should we do about this? The term open source isn t trademarked. Years ago, the OSI tried to register it, but it was apparently too generic. …But there is public opinion. What Danese and Michael are proposing doing is organizing a lot of open source developers (and I mean open source according to the traditional definition, the one the OSI and I and most other open source developers I know adhere to) to stand up and, basically, say All of us agree on what the definition of open source is, and we reject as non-open source any license that does not comply with the letter and spirit of the Open Source Definition.

Alex Hudson commented:

"Similar comments about another W3C WG: http://ln.hixie.ch/?start=1181118077&order=-1&count=1 W3C should be about authoring material for the web. It seems at the moment, they're more concerned with the stuff that happens after authoring - e.g., how stuff relates in "the semantic web". IMHO we still don't have any powerful tools for authoring either, so the idea that authoring tools will automate many of the "bolt on accessibility" tasks is pretty naive."

After implementing pingback, I have some respect for hixie. It's disappointing but not surprising to read those comments. Not sure what is meant by powerful authoring tools there. My tools are powerful, but I know they're not easy for new users. As I'm getting old and cynical, I'm not sure whether power tools can ever be correct and truly easy - just look at the pigswill spewing out of some Web 2.0 sites. niq sent me a link to the WCAG Samurai Errata and commented:

"The problem with WCAG 2, as with WCAG 1, is precisely that it's a Work of Committee, and seeks to represent too many irreconcilable viewpoints and egos. In some cases, even the ideas behind it are inconsistent. Many (myself included) regard accessibility as being about ensuring a reader with a physical disability can make use of a site (think Steven Hawking). That's also what HTML is designed to support. Others are concerned with the mentally impaired, also a worthy cause, but one that becomes counterproductive when its advocates fail to distinguish between a site that's inherently inaccessible because it presents a complex and challenging subject, and one that's unnecessarily difficult due to poor presentation. Add the two together, blend in some stupidity, and you get "accessible" sites full of classic "friendly" little illustrations like a "helpful" government booklet, and a horribly intrusive "alt" to explain each pointless "ethnic mummy and child" picture."

I think a Work of Committee could be useful, as long as you have the right committee. Thinking about that, I found the group membership list and I'm not sure that's the right committee. For example, I think the ex-Adobe Google-worker must surely have an irreconcilable conflict of interest (inappropriate use of Adobe's Flash and PDF formats and Google's Web Apps have done more than many to harm accessibility in the past IMO), yet is a leading light of WCAG 2.0 - maybe that's why it's gone so dark? Update: Thijs Kinkhorst comments:

"As I read in the draft, suggestions are still possible until the 29th, not the 27th of June."

I'm sure that's different to the email which started me writing this, but please go comment and see if we can make WCAG-2.0 useful.

It's been a while since I've given a status update for my Google Summer of Code project to create a BitTorrent proxy for downloading packages using APT, so here it is. I've been working hard on integrating support for APT into the DebTorrent program. I've almost got it working perfectly, now it's just a matter of testing to make sure all is well. The functionality works like this:

• DebTorrent listens on a port for HTTP requests
• an http://localhost:port/mirror_name/debian/... entry is added to APT's sources.list file (similar to how apt-cacher works for proxying)
• an apt-get update will then send HTTP requests to the DebTorrent program
  • DebTorrent proxies these requests, downloading files it doesn't have and saving them in a cache before passing them on to APT
  • DebTorrent recognizes requests for Packages files, and also uses them to start the torrents for those files
• an apt-get install will send an HTTP request to DebTorrent for a package (.deb) file
  • rather than getting package files from the mirror, DebTorrent finds the file in one of its running torrents and enables it for download
  • the package file is downloaded (either from other peers, or using the backup HTTP downloader that gets it from the mirror if no peers can be found)
  • once the download is complete, DebTorrent passes the package file to APT

There are two things I really like about this. One of the best is the backup HTTP downloader. It insures that if you're an early adopter and there are no peers, or if the package you're requesting is rare and can't be found in any connected peers, the download will still occur in a reasonable amount of time (taking no more, or less, mirror bandwidth than if you had just been using APT directly). The other thing I like is that you get the BitTorrent-style peer-to-peer downloading, with simple HTTP proxying thrown in for free. You can run DebTorrent on a single computer on your network, and have the others connect to it to initiate downloads and request packages. I haven't done any serious time testing, but I estimate it currently only takes twice as long to use as a regular APT update and download from a mirror. Most of that slowdown is because it currently only processes a single request at a time from APT, which is not very efficient for BitTorrent systems where downloading from multiple peers is how the highest download speeds are achieved. I have been talking to the APT maintainer, Michael Vogt, about a better way to do this, probably by adding a new APT transport method for DebTorrent (i.e. debtorrent:// instead of http://). This will not only speed up the downloads, but also hopefully provide better feedback to the APT user, as currently it will seem nothing is happening until the download comes in all at once at the end. My work has unfortunately been slowed by other commitments, and bugs. I have two papers to submit to a conference by Monday, but after that I should be back full time on DebTorrent. I also spent a long time tracking down and fixing a bug in the underlying BitTornado code (which lead to much rejoicing at 3am), only to find it was fixed in upstream's CVS. (Doh!)

I’ve been gradually pushing on in my free time on gobject-introspection, so I can make the dbus-glib bindings decent. A couple of months ago back I started hacking up a code scanner to generate the introspection xml,using code from gtk-doc, but using perl regexps to parse C just made me feel dirty… Johan Dahlin pointed me at Sparse, a front-end parser for c developed by some of the kernel guys. The only problem is that it doesn’t have language bindings (or even, seemingly, a stable abi). So I’ve just submitted a patch to sparse that adds an executable ‘c2xml‘. c2xml will dump the parse tree of the input file as a simple xml structure, suitable for manipulation by some python or suchlike. That could then be used to, say, generate c api descriptions, documentation and language bindings or make code navigation or refactoring tools. I like it!

Below is a GNUPlot graph of ZCAV output from a 250G SATA disk. The disk has something wrong with it (other disks of the same brand in machines of the same configuration give more regular graphs). The expected graph will start high on the Y scale (MB/s) and steadily drop as the reads go to shorter tracks near the spindle. The ZCAV results page has some examples of the expected results. If you have any ideas as to what might cause this performance result then please make a comment on this blog post.

Share This

Like everyone else with an Intel graphics chipset and some X bug at DebConf, I was happy to have Keith there to debug it. But I'd go one step futher, and recommend anyone who doesn't have an Intel graphics chipset and some X bug, to go find one of each (and Keith). It's worth the price of admission just to experience his debugging process. If your experience is anything like mine, you'll have a lot of fun. You'll learn about some graphics-related thing, such as PCI configuration space and the LCD brightness register. And you might learn some new things about debugging. How twenty years of experience do come in handy at diagnosing what kind of crash happened, at a glance. That it's possible to still have fun working on something after those twenty years of experience. How to keep the bug submitter involved by letting them follow your thoughts as the problem is debugged, and letting them handle some of the skut-work. How to teach them what they need to do to handle the skut-work, without talking down, or wasting any time. How important it can be to have things like good hardware documentation. And probably lots more that I missed.

When I was 12 years old, I discovered free software. That discovery changed my life and I've never recovered. Over what is now more than half of my life, I have looked to the Free Software Foundation for vision, guidance, and an example of a free world and I have rarely been disappointed. The list of directors of the FSF -- Richard Stallman, Eben Moglen, Lawrence Lessig, Henri Poole, Jerry Sussman, Hal Abelson, and Geoffrey Knauth -- doubles as a list of some of my greatest heroes and role models. As such, I lack the words to describe how it feels that, just yesterday, I was elected to the board of directors of the Free Software Foundation. With Moglen having stepped down I have staggeringly large shoes to fill. I'm more than a little intimidated. At 26 years old, I suspect that I'll be the youngest person on the board by quite a bit. This means I'll have to try and make up with hard work and passion what I lack in experience and wisdom. It's a challenge I look forward to. With free software becoming increasingly successful and widespread, we've already begun to see push back. I suspect that in the next years, we'll see much more. We reaching the dangerous part of the, "first they ignore you, then they laugh at you, then they fight you, then you win" progression. I'll do what I can to defend freedom until we've won. In order to ensure that I have the time necessary, I'm going to be resigning from the board of Software Freedom International and will consider reducing and resigning some of my other commitments as well. If you want to support my work with the foundation, you can become an associate member.

We had a great weekend this last weekend here in Montreal. Yesterday, June 24, was Saint-Jean-Baptiste Day here in Quebec -- the national holiday of this cool and fun nation. St. Jean Baptiste is important to me in a lot of ways. Of course, it's cool that it's the national holiday of our adopted new home. But John the Baptist, the patron saint of Quebec, is also our family's name saint. ''Prodromos'' means "forerunner" in Greek, and it's one of the titles of St. John. Most of all, it's the day that Maj and I moved to Montreal. We came in on St. Jean Baptiste Day, 2003, and we've been here ever since. We drove my beautiful 1972 Citro n DS from San Francisco, where my niece Elena had just been born, all the way out here. We arrives on June 24th and wandered around the McGill Ghetto looking for our sight-unseen sublet. It's hard to believe it's only been 4 years -- it seems like so much longer. We had a good day yesterday. The three of us went to Piknic in the afternoon and hung out with our friends Jeremy and Lina and their two great kids Clea and Sylvie. We had a lot of hummus and pasta salad and bread and cheese and generally had a great time. Meg and Catherine came and hung out for a while, and also our friend Kristen, who's 7 months pregnant and has a 10-month-old. Good luck on that one. Most of all a pretty day enjoying the outdoors in our new home town, as well as fun dancing to house under a big Calder statue. How great is that? tags: montreal quebec stjeanbaptiste prodromou piknic jeremy lina

72 hours from Flickr I forgot to mention that we had a really nice time on Friday evening going to the 24 Hours of Flickr party at the SAT. Heather Champ, community leader at Flickr and welcomer to partiers, put it well: "We have three requirements for Flickr parties: the food must be good, the drinks must be copious, and the schwag must be ample." And that's exactly what happened. All the usual Montreal suspects were out in force: YULblog types, Web entrepreneurs, Web pranksters. It was great seeing m-c and c-c, who wore their awesome Flickr Socks -- although they each had a matching pair. I also talked to Allen Morris, which was really fun. He's a developer living now in Saranac Lake, one of the towns in the Adirondacks I find strangely compelling. We exchanged GPG key signatures and talked a lot about the Bay Area (California), where we both once lived. Probably my best talk was with Heather Champ and Derek Powazek. We talked about their joys of creation of JPG Magazine, the despair of leaving it, and looking forward to the future. We also chatted a lot about print on demand technologies, which we're evaluating for Wikitravel. Heather took the photo of my friend Ben Cerveny at Burning Man that's become his avatar icon on about every social software platform I know of. It must be good to capture someone in the way they want to think of themselves. I took my own self-concept picture by putting the camera on the dashboard of my Citroen. Heh. All in all a good party. One thing I was interested to find out was that there's a Yahoo! Quebec, which launched last month and I guess just slipped by me. Sadly, if you click on the "English" button, it takes you to... Yahoo! Canada. Hey, Yahoo! There are actually a lot of anglophones here! Just in case you were wondering. tags: yahooquebec 24heuresflickrmontreal 24hoursflickrmontreal flickr montreal party

Montreal Web Development Book Club Montreal Web scene stalwart Heri Rakotomalala -- the man with the magic name, originally from Madagascar, along with Montevideo and Marfa, one of the M-places on my travel wishlist -- recently started a Montreal Web Development Book Club. It's a meeting for people interested in the practical side of the Web to talk about the books they read and let that drift into talk about the work we do and ideas we have. I think the book club is an awesome idea. Book clubs are a great grass-roots intellectual meetup that we don't often celebrate in the post-flash mob BarCamp-ized world. The great thing about a book club, for Web people, is that it filters out the bullshitters and dilettantes. If you're going to a book club, you're going to have to know what you're talking about, and you're going to have to care about the subject. The book gives focus that keeps the conversation from drifting to tangential topics. Flakes and hangers-on just don't have the stamina for it. Also, we met at La ka, so it was nice to have some beers in a good place on a hot night. Why not? We managed to talk about a lot of good books, too. Jean-Fran ois <something> (I didn't get his last name) talked about In Search of Stupidity, a look at the last 25 years of marketing and business blunders in the high-tech realm. Our industry is subject to a lot of decisions that seem absolutely self-defeating in hindsight, and the book goes in-depth on 15 case studies -- like Wordstar, Ashton-Tate and Borland. It sounds like a great cautionary tale for Web 2.0 startup mavens. Heri talked about the book Information Architecture for the World Wide Web -- the infamous "Polar Bear Book". It sounds like a really good book for doing initial planning for a Web site, but I'm really interested in tools for profiling an IA effort. We get about 2 million unique visitors a month at Wikitravel, for example, which is a huge amount of usability data I could be mining. But I don't really have the tools for it, so I don't do it much. I'd love to know, for example, about people who land on a Wikitravel page from a search engine and then leave. What were they looking for? Why couldn't they find it? Also, users who browse through our pages clicking on links, then resort to a full-text search. What did they think they were clicking on, and what were they looking for instead? Did they find it? How can we put that in front of them in the first place? Most of all, I want to know how and why people start editing Wikitravel pages, and why they stop. Of people who click the "edit" button, only 1 in 17 click "save". (That is one bit of clickpath analysis I've done.) Why? I've often been tempted to put a survey up when someone navigates away from our edit page. "Why didn't you save? Tell us so we can fix it for other people." I talked about the book I just finished reading a few weeks ago, Everything is Miscellaneous. It's the latest book by Dave Weinberger about how we impose order on information to make it easy to understand and retrieve. It's an awesome book -- really fun -- especially for someone married to a knowledge management expert. I don't think I gave a good enough description of the book to ignite the club's interest, but I hope they check it out, because it's really readable and good. At the end of the meeting Heri took a poll and decided that we'll continue doing the book club on the last Monday of the month. I've already picked out my book for next month: RESTful Web Services. I've heard good things about this book (admittedly mostly from the co-author Leonard Richardson) and I want to give it a look-see. So book-club fodder it is. ''Update: Heri has his own book club report here.'' tags: bookclub montreal heri webdev montrealwebdevbookclub

Ok, I'm officially back. Yup, when you have a meeting just over your regular lunch time and are suddenly seen as the guy who will fix the institute's video needs just because you worked manning cameras with the amazing DC7 video team, it just means you are back home. Lets see how several projects that have sprang up in the last couple of weeks (both at DebConf and at work) start materializing.
Anyway... DebConf was great. Just great. This time I had time both to feel I was useful and productive (mainly doing video-related stuff, although on some other points as well), to attend several interesting talks (although manning the camera did not leave my attention as complete as I would have liked), to have several nice chats, to play a good couple of hands of Mao - Hell, thanks to Kitty, I managed even to have some chats _while_ having a hand of Mao!
Of course, I was not productive neither with my Real Life duties nor with with the Debian stuff I wanted to hack on - In fact, the most productive period during the two and a half weeks I was away (excluding probably the two days I spent understanding/hacking Pentabarf with Dami n) was the flight back - During the ~12hr we were on the air, I managed to read the second half of my book, to review two chapters for a thesis, to write a (really simple) article and to work on another one.
Anyway... Thanks to everybody. I had a great time, and am only looking forward towards Argentina.
Ah, and I also got The Debconf Flu.

DebConf7 finished today. This was my first DebConf, and probably not my last one :-) I had a really interesting time in Edinburgh, despite the weather (but at least we are prepared for DebConf8 now!). The organization team really did a fantastic work, the venue was gorgeous, etc, etc, etc. Again, I was positively surprised by how nice everybody was at DebConf. It’s weird to think that the same people are part of the regular flames on the mailing lists. Which proves that most flames are just caused by people being passionate about Debian, and looking for the best for the project, and not by people trying to be annoying, like one could think :) I gave a talk about the usual QA stuff I’m doing: archive-wide rebuilds, piuparts runs, and how to make all this more efficient by working inside the collab-qa project. So far, it’s not like a lot of people have been joining the project, which is kind of a failure. But at least everybody seem to think that the idea is good. I also gave a lightning talk about Debian Package of the Day, which I didn’t plan to give originally (I think I never prepared slides in such a short time). I think it was well received, and we even got a new submission half an hour after the talk. I even managed to clear some tasks on my TODO list during Debconf, the most visible one being the “automated monthly mails to maintainers of packages with serious problems”. I was quite afraid of being flamed, but I sent 250 mails, and only one person replied aggressively. I also got a lot of positive feedback, so the next batch will probably use slightly less strict criterias. And of course, the most important result of debconf was the numerous discussions, that resulted in a lot of good and interesting ideas.

Following my previous entry about playing around with LVM, LUKS and the device mapper, I documented myself about internals involved in pvmove, which is more of a challenge, considering there is no such documentation. I could find no useful documentation for either the device mapper or LVM. A bit of good old UTSL later, I could elaborate how to do what I wanted, and realized it was even possible to do in shell script. So here you are : a shell script to transform an LVM physical volume to a LVM over LUKS physical volume,. I’ll detail in between how it works. As the previous script, use at your own risk, it comes with no warranty.
Note you theorically can still access the filesystems underneath without problems. It’s an in-place and live conversion. Also note this is more a proof of concept than a proper, risk-less and well-written solution.


set -e
dev=$1
luks=$(mktemp)
cryptdev=$(basename $dev)_crypt
pvsize=$(pvs -o pe_start,pv_size --units s --noheadings --nosuffix "$dev" awk ' print $1 + $2 ')
devsize=$(blockdev --getsz "$dev")
mchunk=8


The script takes the physical volume device to convert as an argument. Note there is no check for the validity of its value.

• luks will be the temporary file where we are going to create a temporary LUKS volume.
• pvsize is the full size of the LVM physical volume, i.e. the size of all the physical extents (pv_size) + the LVM headers and metadata (pe_start).
• devsize is the raw device size.
• mchunk is the size in blocks of chunks for the mirror target, see below.


dd of="$luks" seek=$devsize count=0 bs=512 2> /dev/null
luksdev=$(losetup -f)
losetup "$luksdev" "$luks"
trap "losetup -d \"$luksdev\"; rm -f \"$luks\"" EXIT


Next, we create a sparse luks file the same size as the device (in case luksFormat would use the size somehow, but I believe it doesn’t), and a loopback device on this file. The trap is here to avoid leaving the loopback device and the file when an error occurs later (though during the conversion itself, it will be pointless).


cryptsetup luksFormat -q "$luksdev"
cryptsetup luksOpen "$luksdev" $ cryptdev _real
read start length crypt format key IVoff cdev offset <<EOF
$(dmsetup table $ cryptdev _real)
EOF


We create a LUKS device, so that we can get the encryption key ($key), and the size of the LUKS header ($offset). Note you need to add –showkey to the dmsetup table command on sid.


if [ $(expr $devsize - $pvsize) -lt $offset ]; then
  echo Not enough free space after LVM physical volume
  cryptsetup luksClose $ cryptdev _real
  exit
fi


Check we have enough space after the LVM physical volume to offset everything by the size of the LUKS header. If not, you can still try again after you reduce the size of the LVM physical volume by an extent.


if [ $(expr $devsize % $offset % $mchunk) -gt 0 ]; then
  echo Last
  cryptsetup luksClose $ cryptdev _real
  exit
fi


This is another check to avoid surprises at the end, when dealing with the last chunk. As the script is written for the moment, it doesn’t support cases where this last chunk is not a multiple of $mchunk. So we need to abort in these.


read major minor <<EOF
$(stat -t "$dev" awk ' print $10,$11 ')
EOF
maps=$(dmsetup deps awk -F: "/\($major, $minor\)/ print \$1 ")
dmsetup create $cryptdev <<EOF
0 $length linear $dev 0
EOF
dmsetup reload $ cryptdev _real <<EOF
$start $length crypt $format $key $IVoff $dev $offset
EOF
dmsetup resume $ cryptdev _real
for map in $ maps ; do
  dmsetup table "$map" sed s,$major:$minor,/dev/mapper/$cryptdev, dmsetup reload "$map"
  dmsetup resume "$map"
done


Here, we create the dev_crypt device mapper that is our fake LUKS device, which starts as a simple linear mapper and will end as a complete crypt mapper. This fake LUKS device is inserted as an intermediate mapper between the LVM device mapper and the real device. So, the LVM device mapper will have this fake LUKS device as backend, and, at the beginning, the fake LUKS device maps linearly to the real device. Note we look for all device mappers using the real device as backend before creating the fake LUKS device to avoid finding the fake LUKS device in the list. Also note dmsetup reload only loads a new table in the INACTIVE slot, and dmsetup resume makes this inactive table LIVE.


cursor=$length
chunk=$offset
while [ $cursor -gt 0 ]; do
  cursor=$(expr $cursor - $chunk)
  if [ $cursor -lt 0 ]; then
    chunk=$(expr $chunk + $cursor)
    cursor=0
  fi
  (
  [ $cursor -ne 0 ] && echo 0 $cursor linear $dev 0
  echo $cursor $chunk mirror core 1 $mchunk 2 $dev $cursor /dev/mapper/$ cryptdev _real $cursor
  [ $cursor -lt $(expr $length - $chunk) ] && echo $(expr $cursor + $chunk) $(expr $length - $cursor - $chunk) crypt $format $key $(expr $IVoff + $cursor + $chunk) $dev $(expr $offset + $cursor + $chunk)
  ) dmsetup reload "$cryptdev"
  dmsetup resume "$cryptdev"
  chunks=$(expr $chunk / $mchunk)
  while ! dmsetup status "$cryptdev" grep "$chunks/$chunks"; do
    true
  done
done


This is where the main work is done : moving the data around. We actually just let the device mapper deal with the data duplication, $offset blocks by $offset blocks ($offset being the LUKS header size), using a mirror target for the chunk being moved. So our disk looks like the following:

We use the extra dev_crypt_real device (the previously remapped LUKS device) as the encryption backend for the mirror.
I haven’t figured a better way to wait for the end of the mirroring than to do a loop checking with dmsetup status, dmsetup wait doesn’t seem to be very helpful here.
Anyways, this is the part of the script where you don’t want a crash to occur. Because if it does, all you can do is start on a rescue system, and try to find where the encrypted part of the disk start to setup a device mapper by hand.
And you’d better have the luks temporary file in a directory that is neither in RAM (think tmpfs) nor in the LVM you are converting (/tmp in a default etch install is, for instance ; note the script works nevertheless fine in this case). Also note the trap will remove the luks temporary file if the script exits…


dmsetup reload "$cryptdev" <<EOF
$start $length crypt $format $key $IVoff $dev $offset
EOF
dmsetup resume "$cryptdev"
dmsetup remove "$ cryptdev _real"
dd if="$luks" of=$dev count=$offset bs=512 2> /dev/null


Final steps of the conversion : our dev_crypt device becomes a full LUKS volume, so we can remove dev_crypt_real and add the LUKS headers at the top of the device we converted. At this moment, the LUKS volume is setup just as if it had been setup by cryptsetup. For LVM to recognize the change properly, you need to run pvscan. Once you ran it, you can do whatever you want with LVM. Now, you may want to add the following to your /etc/crypttab file:


$cryptdev $dev none luks


i.e. hda5_crypt /dev/hda5 none luks if the device was /dev/hda5. And if the LVM volume you converted contains your root filesystem, you should run (for Debian systems):


update-initramfs -u


I tested this successfully under qemu and will give it a shot on my laptop some time soon. Now, because I had a hard time not finding much about the mirror target of the device mapper, here is what I could gather about it. The target syntax is as follows:


<logical_start_sector> <num_sectors> mirror [ core disk ] <num_params> <param> ... <num_mirrors> [ <destination > <start_sector> ] ...


logical_start_sector, num_sectors, destination and start_sector have the same meaning as in other targets.
core and disk are two different log types (to track differences between mirrors), respectively in memory and on disk.
num_params and params depend on the log type:

• for core, num_params can be either 1 or 2.
  The first parameter is region_size, which is the size (in blocks) of the chunks the mirror log tracks for synchronization. It can’t be less than the page size divided by 512, i.e. 8 on x86, must be a power of 2, and must not exceed num_sectors. Apparently, it is not a problem if num_sectors is not a multiple of region_size.
  The optional second parameter is either sync or nosync, meaning of which I’m not sure. I think it determines whether the mirror should do the initial synchronization (sync) or not (nosync).
• for disk, num_params can be either 2 or 3, with params being log_device (device where the logs are kept), region_size, as above, and an optional sync or nosync, as above too.

num_mirrors is the number of mirrors and for each mirror, we have a pair destination and start_sector. [ Update: after a quick look at the device mapper source code in Linus’s git tree, updated the mirror target description ]

1. The first rule of DebConf: Bring every power adapter you have, regardless of the host country’s standard power outlets
2. The second rule of DebConf: Bring every power adapter you have, regardless of the host country’s standard power outlets

Last December, I went to the Delta de l'Ebre with a good number of friends to spend one of December's three-day weekends. Les Terres de l'Ebre are the southest region of Catalunya, right next to the border with the Valencian Country. The Ebre river forms a big delta as it meets the Mediterranean, and the great amount of wildlife, specially avian, in the Natural Park makes it a fantastic place to visit. We would sleep, more or less, in Cinta's appartment in L'Ampolla. The first night we managed to fit 17 persons for dinner, and about 14 stayed to sleep in any possible corner of the flat. As soon as the Valencian group arrived, we tried to meet the people coming from the North, but apparently they got lost and we stayed on a sightseeing spot admiring how easily the weather changes here. In an hour or so, we went from a sunny day to a quick but heavy shower and back to sunny. When we finally found each other, we decided it was a bit too late to do anything too special, given the short days in December, so we walked South along the beach, on our way to the Salines, the salt processing facility, in the Punta de la Banya.

Agua para todos... During the walk, we witnessed what the people from the Delta have been warning about for decades: the Delta is disappearing. The progressive overuse of the river's flow, excessive irrigation, damm construction and degradation is killing this unique place. The waters are slowly claiming what once was theirs. An electricity line, at the beginning of our trip being at the end of the beach starts approaching the sea, or viceversa, as you move southward, to end up many metres inside the water. The old people in the Delta still remember the 200m tall lighthouse in the Illa de Buda, which the sea managed to devour.

Sunset at the salines del Delta When we reached the Punta we could watch a beautiful sunset over the mountains of Els Ports, and when it got dark and cold we headed to Tortosa, the area's capital, which I had recently visited for the first time for one of Softcatal 's meetings. There, we had dinner at a bar and met Jordi and Anna for a while, before going back to Ampolla, going to bed relatively early due to everyone being very tired.

On Saturday, we decided to have a look at the Delta... from above. So we drove to the top of the Cim del Caro, where we could admire a great view over the flat farmlands of the Delta, and then found a nice trail which took us to a nice forest, where we had lunch. After a visit to Alfara de Carles, we went back to the house, had a great dinner and played fun games at the table. Unfortunately, Sunday morning was dedicated to cleaning up the mess in the appartment and packing, as part of the group, the Jordis from Alcoi and the twins, had to leave early. The remaining four Valencians decided to go to Deltebre to have a local paella. It was quite good, although the bar tender laughed at our request to eat directly from the paella, rejecting dishes. Small cultural clash. ;)

I remember downloading and viewing the pictures of the trip a few days after getting back in Val ncia, and then losing track of them. Last week, while testing an EPIA board with some random hard drive, I suddenly realised there was a big chance the pictures would be on that drive, and after some search, I found them; I had been close to mkfs the disk a few minutes before. I hadn't stopped blaming myself for misplacing these pictures, as they remind me of this wonderful weekend. Luckily, they are back! Going back to the state of the Delta, experts say it will be mostly gone in 200 years. All of the Terres de l'Ebre have fought against the previous government's "PHN" plan to divert part of the river's flow to the North and South, to supply Barcelona, and specially the South of Val ncia, where the developing of the coastal areas is out of control. I fear that whenever PP regains power in Madrid, they will retake this idea. There is a lot of money involved in building a 700 km long scar in our land. The Delta de l'Ebre is a fantastic place which doesn't deserve a terrible fate like this, only for the benefit of the usual suspects. As the good people of this land would say...

Lo riu s vida. No al transvasament!

Last December, I went to the Delta de l'Ebre with a good number of friends to spend one of December's three-day weekends. Les Terres de l'Ebre are the southest region of Catalunya, right next to the border with the Valencian Country. The Ebre river forms a big delta as it meets the Mediterranean, and the great amount of wildlife, specially avian, in the Natural Park makes it a fantastic place to visit. We would sleep, more or less, in Cinta's appartment in L'Ampolla. The first night we managed to fit 17 persons for dinner, and about 14 stayed to sleep in any possible corner of the flat. As soon as the Valencian group arrived, we tried to meet the people coming from the North, but apparently they got lost and we stayed on a sightseeing spot admiring how easily the weather changes here. In an hour or so, we went from a sunny day to a quick but heavy shower and back to sunny. When we finally found each other, we decided it was a bit too late to do anything too special, given the short days in December, so we walked South along the beach, on our way to the Salines, the salt processing facility, in the Punta de la Banya.

Agua para todos... During the walk, we witnessed what the people from the Delta have been warning about for decades: the Delta is disappearing. The progressive overuse of the river's flow, excessive irrigation, damm construction and degradation is killing this unique place. The waters are slowly claiming what once was theirs. An electricity line, at the beginning of our trip being at the end of the beach starts approaching the sea, or viceversa, as you move southward, to end up many metres inside the water. The old people in the Delta still remember the 200m tall lighthouse in the Illa de Buda, which the sea managed to devour.

Sunset at the salines del Delta When we reached the Punta we could watch a beautiful sunset over the mountains of Els Ports, and when it got dark and cold we headed to Tortosa, the area's capital, which I had recently visited for the first time for one of Softcatal 's meetings. There, we had dinner at a bar and met Jordi and Anna for a while, before going back to Ampolla, going to bed relatively early due to everyone being very tired.

On Saturday, we decided to have a look at the Delta... from above. So we drove to the top of the Cim del Caro, where we could admire a great view over the flat farmlands of the Delta, and then found a nice trail which took us to a nice forest, where we had lunch. After a visit to Alfara de Carles, we went back to the house, had a great dinner and played fun games at the table. Unfortunately, Sunday morning was dedicated to cleaning up the mess in the appartment and packing, as part of the group, the Jordis from Alcoi and the twins, had to leave early. The remaining four Valencians decided to go to Deltebre to have a local paella. It was quite good, although the bar tender laughed at our request to eat directly from the paella, rejecting dishes. Small cultural clash. ;)

I remember downloading and viewing the pictures of the trip a few days after getting back in Val ncia, and then losing track of them. Last week, while testing an EPIA board with some random hard drive, I suddenly realised there was a big chance the pictures would be on that drive, and after some search, I found them; I had been close to mkfs the disk a few minutes before. I hadn't stopped blaming myself for misplacing these pictures, as they remind me of this wonderful weekend. Luckily, they are back! Going back to the state of the Delta, experts say it will be mostly gone in 200 years. All of the Terres de l'Ebre have fought against the previous government's "PHN" plan to divert part of the river's flow to the North and South, to supply Barcelona, and specially the South of Val ncia, where the developing of the coastal areas is out of control. I fear that whenever PP regains power in Madrid, they will retake this idea. There is a lot of money involved in building a 700 km long scar in our land. The Delta de l'Ebre is a fantastic place which doesn't deserve a terrible fate like this, only for the benefit of the usual suspects. As the good people of this land would say...

Lo riu s vida. No al transvasament!

So, I'm happy to see that commercial wiki provider WetPaint now supports OpenID for logging into their site. It's great to have more wiki services supporting this single sign-on standard. A couple of nits, though: first, I logged in with my normal OpenID, http://evan.prodromou.name/ , which delegates to http://certifi.ca/evan . WetPaint's OpenID service used the delegated URL, which is just wrong. They need to fix that. The delegate is the "backend", and users should be able to change it over their lifetime. If relying parties store that backend URL instead of the "main" OpenID URL, that undermines the process. Second, they have a signup page after the validation, and it has a lot of fields that could be pre-populated using the OpenID Simple Registration extension. I shouldn't have to type in my preferred user name, real name, or email address. That information is already available through my OpenID provider. Other than that, it seems to work OK. I'm glad that it exists at all, but I wish it was a smoother user experience. tags: wetpaint wiki openid sso

Wikispaces and SourceForge I haven't already mentioned the announcement that SourceForge has implemented Wikispaces wikis for every project on their site. I think that's great news for the wiki community, and probably a good thing for SourceForge users. I feel churlish mentioning it, but it has to be said: with hundreds of available WikiEngines, the vast majority of which are Open Source, and many of which are hosted on SourceForge, why did SourceForge go with a commercial service provider? It doesn't seem like they've justified that decision very well. tags: wikispaces sourceforge wiki announcement

I finally decided myself to switch my laptop to using a LVM over LUKS setup instead of LVM alone. The problem it involves is that there is currently no way that I know to do this in-place. I do have an external disk on which I could pvmove everything and back, but that would not be very challenging ;). So I started wondering how doable that would be. LUKS has a 1032 sectors (a sector being 512 bytes) overhead in which it store the LUKS header. Following the header, encrypted data just takes as much space as its decrypted counterpart, linearly. So all it would take to do in-place LUKS conversion would be to move data by 1032 sectors, and encrypt it. Which means the main problem is to have 1032 sectors free after the current volume. Fortunately, LVM doesn’t take up all the space on the partition a physical volume is setup, except if you’re unlucky or created partitions specifically so that it uses all the space. Because LVM physical volumes use the space by big chunks, 4MB by default), if the size of the partition is not 4MB aligned (modulo the LVM header), you have some free space between the physical volume end and the partition end. We thus need this free space to be greater than 1032 sectors. On my laptop, there is more than 1.8MB free there, so there would be no problem.
Note the debian-installer seems to do more than a simple cryptsetup luksFormat, as LUKS volumes it creates have an overhead of 2056 sectors instead of the 1032 sectors I got creating a volume by hand. Next problem is how to move the data and encrypt it in-place. Bonus points if that can be done without unmounting the filesystems. My first guess was to play around with the device mapper and a specially crafted FUSE filesystem to create a fake LUKS volume that could be half encrypted and half not, and would evolve, with time, into a fully encrypted LUKS volume. Since I didn’t know much about the device mapper, I looked at it a bit, and discovered a FUSE intermediate would be unnecessary: not only can you remap on-the-fly (which I already guessed was somehow possible, considering how LVM can do it, though I didn’t know much about the interaction between LVM and the d-m), but you can also mix dm-crypt and linear chunks in the same d-m table. It also appears LVM over LUKS has an overhead that is actually not necessary. How sad both use the device mapper but yet can’t do an efficient setup with it. When you setup LVM over LUKS, you open a LUKS volume, which will create a dm-crypt device mapper, and use this device mapper as the physical volume for LVM. Logical volumes created with LVM are actually device mappers using the physical volume as destination device. When you access a block on an LVM logical volume, your access is first mapped through the LV device mapper and then mapped again through the dm-crypt mapper. So it goes through 2 device mappers, but it could do with only one: the linear mapping used in most of the cases with LVM could be replaced by crypt mappings, that are just as linear, but also handle the encryption/decryption part. Playing around under qemu with a freshly installed etch system, I was able to implement this and came up with the following script. It should work with most setups, but use at your own risk, it comes with no warranty.


MAJOR=$(awk '/device-mapper/ print $1;exit ' /proc/devices)
dmsetup ls --target linear while read map extra; do
  TABLE=$(dmsetup table $map while IFS=" :" read start length linear major minor offset; do
    map2=
    if [ "$major" -eq "$MAJOR" ]; then
      IFS=":" read map2 extra <<EOF
$(dmsetup info -c --noheadings -j $major -m $minor)
EOF
    fi
    if [ -z "$map2" ] [ $(dmsetup table "$map2" wc -l) -gt 1 ] [ $(dmsetup table --target crypt "$map2" wc -l) -eq 0 ]; then
      echo $start $length $linear $major:$minor $offset
    else
      IFS=" " read cstart clength crypt format key IVoff dev coffset <<EOF
$(dmsetup table "$map2")
EOF
      echo $start $length crypt $format $key $(expr $IVoff + $offset) $dev $(expr $coffset + $offset)
    fi
  done)
  if (dmsetup table $map; echo "$TABLE") sort uniq -c grep -v "^ *2 " > /dev/null 2>&1;
  then
    echo Diverting $map
    dmsetup reload $map <<EOF
$TABLE
EOF
    dmsetup resume $map
  fi
done


[ Update: There was a dmsetup resume missing in the script ] After running this script, you should be able to run cryptsetup luksClose on your LUKS volume. Once you’ve done it, pvdisplay and other LVM tools will show you nothing, because the physical volume will just have disappeared… but everything will still be in place. Note that if you leave the LUKs volume open, you’d better be careful NOT to do operations with LVM (such as pvmove) because it could break everything. Note that on sid, you would need to add a –showkey option into the $(dmsetup table "$map2") line. This option is necessary for dmsetup to display the encryption keys. I wonder if LVM upstream would consider supporting LUKS volumes directly… Anyways, back to the in-place conversion to LVM over LUKs. The idea would be to:

• Create a file the size of the LUKS headers + one block.
• Create a loopback device for this file.
• luksFormat the loopback device.
• luksOpen the loopback device, so that dmsetup can give us the key to use. I wonder if there are ways to get the key not involving the creation of the device mapper.
• luksClose it, because we actually don’t need it
• Create a device mapper that linearly maps the whole LVM physical volume (i.e the real partition, but without the empty tail).
• Remap the LVM device mappers so that they use our own device mapper instead of the partition as physical volume.
• Here is the trickiest part, and I still have to take a look at how pvmove does to know exactly how this can be done (I guess it will require an additional device mapper): move data chunk by chunk (offsetting with the size of the LUKS headers), starting from the end of the LVM physical volume, and encrypt at the same time. Once a chunk is moved, adjust our device mapper so that it maps to the linear unmoved data followed by the moved encrypted data. As I said earlier, this is a setup that works.
• Once everything is moved, we are in a situation where our device mapper is a single crypt mapping, in the same shape as a luksOpen would have made it. We can now copy the LUKS headers at the beginning of the partition, and add the appropriate configuration to /etc/crypttab.

If I got it all right, both LUKS and LVM should be believing they have done this setup themselves, so unmounting, lvchange, etc. should just be fine. It might be necessary to regenerate the initramfs, though. I’ll tell you when I fuck up my hard disk try it. To be continued…

I will be presenting Hecl at OSCON this summer! http://conferences.oreillynet.com/cs/os2007/view/e_sess/12403 I'm pretty excited about it, and have been brushing up Hecl a bit, although I need to do a new release. For the moment, getting it from subversion is your best bet. I'm really looking forward to going home to Oregon, and getting to go to a cool conference and present at it, too! I hope to see you there!

I’m now already 9 days in Edinburgh (started writing this at day 5 already :) ), and all of them have been days of “get up at 8h, breakfast, goto Teviot, work until 23h, go back to Hostel”, with some short breaks in Teviots Bar and for Lunch/Dinner. Sounds stressful but actually was (mostly) a lot of fun, preparing the network, setting up all the core servers together with two other admins while at the same time getting a lot of request to change various parts of our Conference Management System Pentabarf to fit some peoples need at various places. The airline managed to get me to EDI, together with a few other Debian people on Friday, 8th June, around noon, where Mark (mhy) Hymers did wait already, so I didn’t need to take a bus to get into the town. On the way to Teviot I also remembered some of the places from back when we did the Venue visit for the DC7 decision. But not many, of course one forgets a lot over the time. At least I was able to remember some food places, which is always a good sign. :) At Teviot we met with Stephen (sgran) Gran to (unsorted list) have some food, unload the car, think about the way to get our uplink to our server room, check into the hostel - and whatever else we did at that day. Next day, officially called Setup day, we started to get two cables from our server room to the place our uplink is - you can see them if you go to the left site of Teviot. (The other two got there a bit later). After that we started with the setup of the network, which is pretty nice this year. We have a lot of sponsored machines that are doing various tasks, a nice network layout involving a lot of vlans to seperate stuff and a lot of (good) crack. What, you dont want to listen to boring details? Ok, here they come: - The main machine is, again, called Homer. Connected with a 2Gbit link to our master switch. It does those little things noone needs, like firewall, local mirror, dns, nagios, routing between various nets we have here, the schedule bot, etc. - There is also a Marge, who is the central point for the whole wireless network, running chillispot to have authentication for it. - Barney is used as a storage server for the video team, having some 1.5TB (raid5) space for the video data. It has 2 additional disks, 500G each, to copy data on and get them home to a video team after the conference. - Our streaming master machine at DC7 is called Lisa, and does not more than streaming for all the local clients and sending the stream out to our streaming master somewhere in the internet which then relays it to our streaming servers kindly offered by a lot of admins all over the world. - We also have Bart and Maggie, which serve as public build boxes for the attendees, where Maggie even runs a distcc open to the whole local network. - You can add the same amount of servers to this list which all do various tasks for the video team. This stuff is all glued together by a set of managed switches building our core backbone, running a number of vlans to seperate the different network segments. Which actually works quite nicely, as even at the time some expert managed to plug a cable into the same switch twice (yay, loop) - the only affected network part was that where he was in. The rest of the conference didn’t even notice it - until all of the admins jumped out of the admin room at once, looking for that fuckup. :) Sounds like crack? Probably is, but is a nice setup all together, and it was fun to set it all up together with Mark and Stephen, the two main admins for this conference who designed most of the whole structure and are the ones that really know it. I already “volunteered” them for the next DebConf to help as admins and really hope they decide to do it. Later on during DebCamp we also had Peter (weasel) Palfrader joining the local admin team. His main work here is related to the video team by hosting the geodns setup for the streaming and by running a local nagios to check the things we need to check. This enabled us to simply tell everyone Use streams.video.debconf.org:8000 wherever you are and you will get to the right server for you. Explaining this in a short way basically goes as “a monitoring tool keeps track of which servers are up and only enables them to resolve. Our geodns server then replies with the right server IPs, depending on in which region of the world you are.”. Which means streams.video.debconf.org will resolve to the local streaming server lisa while you are here, at the same time resolve to a number (those that are up) servers in Europe if you are in Europe or to those in the US if you are there. Nice. (And while I’m already blogging - I do hate my laptops suspend, err - I mean non-suspend mode. Somehow it got worse in the last days, sometimes needing multiple minutes (like 10) to decide to suspend to disk, and the suspend-to-ram still wont work. I should really get someone here that knows this stuff having a look at it. :) ).

went to Teviot, the main Debconf7 venue, with Megan for a while. Picked up a couple of people and then left for a night in the pub. After chatting to the barmaid, Emma, who wanted her Ubuntu laptop fixed. Looks like that is sorted now. Met Ian Jackson, who surprised me by being both younger and shorter than I expected. Met Hannah, who killed Alfie, and who wasn't pregnant. Excellent night. I also hacked on xen-tools and tracked down copyright bits for debootstrap. Now to bed, via The Hobbit. The tattoo I expect people will recognize when it comes to sauna-time. In other news - I don't have a bear but now I own sandals. TODO: Print groupie badges for people.