Following my previous entry about playing around with LVM, LUKS and the device mapper, I documented myself about internals involved in pvmove, which is more of a challenge, considering there is no such documentation. I could find no useful documentation for either the device mapper or LVM. A bit of good old UTSL later, I could elaborate how to do what I wanted, and realized it was even possible to do in shell script. So here you are : a shell script to transform an LVM physical volume to a LVM over LUKS physical volume,. I’ll detail in between how it works. As the previous script, use at your own risk, it comes with no warranty.
Note you theorically can still access the filesystems underneath without problems. It’s an in-place and live conversion. Also note this is more a proof of concept than a proper, risk-less and well-written solution.


set -e
dev=$1
luks=$(mktemp)
cryptdev=$(basename $dev)_crypt
pvsize=$(pvs -o pe_start,pv_size --units s --noheadings --nosuffix "$dev" awk ' print $1 + $2 ')
devsize=$(blockdev --getsz "$dev")
mchunk=8


The script takes the physical volume device to convert as an argument. Note there is no check for the validity of its value.

• luks will be the temporary file where we are going to create a temporary LUKS volume.
• pvsize is the full size of the LVM physical volume, i.e. the size of all the physical extents (pv_size) + the LVM headers and metadata (pe_start).
• devsize is the raw device size.
• mchunk is the size in blocks of chunks for the mirror target, see below.


dd of="$luks" seek=$devsize count=0 bs=512 2> /dev/null
luksdev=$(losetup -f)
losetup "$luksdev" "$luks"
trap "losetup -d \"$luksdev\"; rm -f \"$luks\"" EXIT


Next, we create a sparse luks file the same size as the device (in case luksFormat would use the size somehow, but I believe it doesn’t), and a loopback device on this file. The trap is here to avoid leaving the loopback device and the file when an error occurs later (though during the conversion itself, it will be pointless).


cryptsetup luksFormat -q "$luksdev"
cryptsetup luksOpen "$luksdev" $ cryptdev _real
read start length crypt format key IVoff cdev offset <<EOF
$(dmsetup table $ cryptdev _real)
EOF


We create a LUKS device, so that we can get the encryption key ($key), and the size of the LUKS header ($offset). Note you need to add –showkey to the dmsetup table command on sid.


if [ $(expr $devsize - $pvsize) -lt $offset ]; then
  echo Not enough free space after LVM physical volume
  cryptsetup luksClose $ cryptdev _real
  exit
fi


Check we have enough space after the LVM physical volume to offset everything by the size of the LUKS header. If not, you can still try again after you reduce the size of the LVM physical volume by an extent.


if [ $(expr $devsize % $offset % $mchunk) -gt 0 ]; then
  echo Last
  cryptsetup luksClose $ cryptdev _real
  exit
fi


This is another check to avoid surprises at the end, when dealing with the last chunk. As the script is written for the moment, it doesn’t support cases where this last chunk is not a multiple of $mchunk. So we need to abort in these.


read major minor <<EOF
$(stat -t "$dev" awk ' print $10,$11 ')
EOF
maps=$(dmsetup deps awk -F: "/\($major, $minor\)/ print \$1 ")
dmsetup create $cryptdev <<EOF
0 $length linear $dev 0
EOF
dmsetup reload $ cryptdev _real <<EOF
$start $length crypt $format $key $IVoff $dev $offset
EOF
dmsetup resume $ cryptdev _real
for map in $ maps ; do
  dmsetup table "$map" sed s,$major:$minor,/dev/mapper/$cryptdev, dmsetup reload "$map"
  dmsetup resume "$map"
done


Here, we create the dev_crypt device mapper that is our fake LUKS device, which starts as a simple linear mapper and will end as a complete crypt mapper. This fake LUKS device is inserted as an intermediate mapper between the LVM device mapper and the real device. So, the LVM device mapper will have this fake LUKS device as backend, and, at the beginning, the fake LUKS device maps linearly to the real device. Note we look for all device mappers using the real device as backend before creating the fake LUKS device to avoid finding the fake LUKS device in the list. Also note dmsetup reload only loads a new table in the INACTIVE slot, and dmsetup resume makes this inactive table LIVE.


cursor=$length
chunk=$offset
while [ $cursor -gt 0 ]; do
  cursor=$(expr $cursor - $chunk)
  if [ $cursor -lt 0 ]; then
    chunk=$(expr $chunk + $cursor)
    cursor=0
  fi
  (
  [ $cursor -ne 0 ] && echo 0 $cursor linear $dev 0
  echo $cursor $chunk mirror core 1 $mchunk 2 $dev $cursor /dev/mapper/$ cryptdev _real $cursor
  [ $cursor -lt $(expr $length - $chunk) ] && echo $(expr $cursor + $chunk) $(expr $length - $cursor - $chunk) crypt $format $key $(expr $IVoff + $cursor + $chunk) $dev $(expr $offset + $cursor + $chunk)
  ) dmsetup reload "$cryptdev"
  dmsetup resume "$cryptdev"
  chunks=$(expr $chunk / $mchunk)
  while ! dmsetup status "$cryptdev" grep "$chunks/$chunks"; do
    true
  done
done


This is where the main work is done : moving the data around. We actually just let the device mapper deal with the data duplication, $offset blocks by $offset blocks ($offset being the LUKS header size), using a mirror target for the chunk being moved. So our disk looks like the following:

We use the extra dev_crypt_real device (the previously remapped LUKS device) as the encryption backend for the mirror.
I haven’t figured a better way to wait for the end of the mirroring than to do a loop checking with dmsetup status, dmsetup wait doesn’t seem to be very helpful here.
Anyways, this is the part of the script where you don’t want a crash to occur. Because if it does, all you can do is start on a rescue system, and try to find where the encrypted part of the disk start to setup a device mapper by hand.
And you’d better have the luks temporary file in a directory that is neither in RAM (think tmpfs) nor in the LVM you are converting (/tmp in a default etch install is, for instance ; note the script works nevertheless fine in this case). Also note the trap will remove the luks temporary file if the script exits…


dmsetup reload "$cryptdev" <<EOF
$start $length crypt $format $key $IVoff $dev $offset
EOF
dmsetup resume "$cryptdev"
dmsetup remove "$ cryptdev _real"
dd if="$luks" of=$dev count=$offset bs=512 2> /dev/null


Final steps of the conversion : our dev_crypt device becomes a full LUKS volume, so we can remove dev_crypt_real and add the LUKS headers at the top of the device we converted. At this moment, the LUKS volume is setup just as if it had been setup by cryptsetup. For LVM to recognize the change properly, you need to run pvscan. Once you ran it, you can do whatever you want with LVM. Now, you may want to add the following to your /etc/crypttab file:


$cryptdev $dev none luks


i.e. hda5_crypt /dev/hda5 none luks if the device was /dev/hda5. And if the LVM volume you converted contains your root filesystem, you should run (for Debian systems):


update-initramfs -u


I tested this successfully under qemu and will give it a shot on my laptop some time soon. Now, because I had a hard time not finding much about the mirror target of the device mapper, here is what I could gather about it. The target syntax is as follows:


<logical_start_sector> <num_sectors> mirror [ core disk ] <num_params> <param> ... <num_mirrors> [ <destination > <start_sector> ] ...


logical_start_sector, num_sectors, destination and start_sector have the same meaning as in other targets.
core and disk are two different log types (to track differences between mirrors), respectively in memory and on disk.
num_params and params depend on the log type:

• for core, num_params can be either 1 or 2.
  The first parameter is region_size, which is the size (in blocks) of the chunks the mirror log tracks for synchronization. It can’t be less than the page size divided by 512, i.e. 8 on x86, must be a power of 2, and must not exceed num_sectors. Apparently, it is not a problem if num_sectors is not a multiple of region_size.
  The optional second parameter is either sync or nosync, meaning of which I’m not sure. I think it determines whether the mirror should do the initial synchronization (sync) or not (nosync).
• for disk, num_params can be either 2 or 3, with params being log_device (device where the logs are kept), region_size, as above, and an optional sync or nosync, as above too.

num_mirrors is the number of mirrors and for each mirror, we have a pair destination and start_sector. [ Update: after a quick look at the device mapper source code in Linus’s git tree, updated the mirror target description ]

Back in 2000, I started to get back into photography. I bought a Canon Elan IIe 35mm SLR camera, some lenses, and a flash. I took color photos various places, and then bought some standard Kodak T-Max black and white film. I shot some photos, and then tried to get this film processed.

Turns out it's not terribly easy to pay someone to process black and white film. Very few of the local photo places will do it. The place I usually used was willing to send it off to Kodak for me. Lacking a darkroom at home, and any interest in doing darkroom work myself, I sent it off to Kodak.

Kodak processed my film just fine, but the prints they made were terrible. It looked as if their enlarger was seriously out of focus. Everything was fuzzy on the prints. The local shop agreed to send them back for re-printing. On try 2, they were somewhat better but not much.

Now, B&W photos should normally be sharper than color photos, so this was nanoying.

I kept looking, and nobody locally could print B&W photos. I even tried one roll of B&W C-41 film (that's color-process film that takes photos in black and white). It stank about as much as I thought it would. I did eventually find one local lab that could take B&W negatives and, via a digital process, print the photos on color paper. They came out far more crisp than Kodak's processing on real B&W paper!

Now, 7 years later, I've been shooting some photos with my Canon Digital Rebel XTi digital SLR camera that I want to print in black and white. There are any number of color labs that I can send them off to, and get results as sharp as one would expect from a color photo. I'd been doing that with reasonable results. But I wasn't satisfied with "reasonable", so I searched some B&W groups on Flickr to see what people were doing to make B&W prints from a digital source.

It was there that I learned of mpix.com, the online service of the USA's largest pro photo lab. They offer printing on true B&W photo paper from digital (or film) sources. They're a pro lab, so they ought to do this really well. They of course also do color printing. Plus all the other things you'd expect from a pro lab, such as red-eye removal, glasses glare removal, color retouching, choice of photo paper, etc.

I sent off my first order for B&W prints to them yesterday. I can't wait to see how they turn out. I'm excited -- I think they'll be great.

Automatic completion for Closes: entry in Vim DebConf is always cool: it's the place where 2 minute chats with people can turn into cool hacks. It just happened to me: 2 minutes with Dancer and Gismo chatting about apt-listbugs and kaboom: in the next upload of Vim you will get for free (as in the free drinks Sun offered us a couple of days ago) completion of Closes: entry in your debian/changelog. Just start typing Closes: # and after the # execute CTRL-X CTRL-O (the key to invoke "omni completion", see ":help omni-completion" in the Vim online help). At that point apt-listbugs will be invoked by Vim, it will contact the BTS using the SOAP interface to retrieve outstanding bugs, and will offer you a list of bugs to choose from. A brief description of the bug report will be shown as well in a preview window (see ":help preview"). Once you've made your choice, the appropriate "#XXXX" entry will be input where your cursor was. Super-cool! Here are a couple of screenshot to wet your appetite: , . If you can't wait for the next Vim upload take my current debchangelog.vim and put it in /usr/share/vim/vimcurrent/ftplugin/.

Those of you that look at photos on my blog may have noticed that I recently switched from a self-hosted Gallery2 installation to the popular photo-sharing site Flickr.

I initially decided to try Flickr for a simple reason: digikam's image upload to Gallery2 has never worked for me, and I hoped that its Flickr upload would work (it did).

I had heard good things about Flickr from friends and colleagues, so I gave it a try.

First Impressions

Almost the first thing I noticed about Flickr was that there are people there that are interested in the same things I'm interested in. I discovered the Anabaptist History group. I've been contacted by a person in North Dakota that is interested German-speaking Russians, as my ancestors were. There are groups about Debian, classic-style black & white photography on digital cameras, and really just about any topic. These groups contain both discussions and a photo "pool" to which any group member can post.

In one sense, Flickr is social networking via photography.

Flickr also seems to be a company with a clue. It has tons of open APIs, permitting everything from flickrfs (mount flickr as a FUSE filesystem) to countless uploaders, mashups, scripts, etc. Despite a couple of recent high-publicity controversies, they seem to be laid back overall and haven't been turned into a corporate drone by Yahoo. There's an RSS feed for just about everything, too.

What is Flickr?

Depends on what you want it to be. It could be that place where you post your cell phone's snapshots, or a place to share photos with family in private, or just a place to find other's work.

I found sharing and talking with others to be infectious. I went from never getting comments on photos on my Gallery2 site to wondering what people will think -- and looking at their photos too.

Let's look at its different aspects in more detail.

Uploading Photos

You can upload photos manually with a HTTP form, or you can use any of the numerous uploading tools. Most of these will let you scale down your images on the fly, should you wish to. Most will also let you assign tags and write descriptions as they're being posted, as well as define who can view the photos. Uploaders are available for every common platform.

Flickr has a AJAX tool called the Organizr that helps you assign tags, add photos to groups, put them in sets, etc. It's fairly nice.

Sharing Photos

Making photos available to the public is easy. You can just them them a link to your photostream. You can also tag photos and give people a link to photos with a specific tag. And you can create sets -- photos around a specific theme.

Security

Each photo can be made visible to: just you, just your contacts marked "family", just your contacts marked "friend", any of your contacts, or anyone at all. This can be defined per photo.

On a global basis, you can define whether you make EXIF information visible on the site, default security for new uploads, who can post comments, who can post photo notes or add tags, and who can use the "all sizes" button.

You can also control how much of your profile to make public. Settings range from making everything public to concealing even your email address.

For sets, you can also send out a "Guest Pass" (GP). A GP lets someone that is not a Flickr member see non-public photos. I use GPs to share family photos, so that family doesn't have to create a Flickr account to see them. GPs can be tied only to sets.

Groups

Flickr groups appear to have been designed for discussions, but I find them most attractive because each group has its own dedicated photo pool. Members of the group can add their photos to the pool. Some groups have tens of thousands of photos, and others a few dozen. It's an interesting way to find images. There are a lot of group management tools as well.

Free vs. Paid Accounts

Free account holders can upload up to 100MB of photos per month. There is no cap on bandwidth consumed by views of photos or management of them. Free users can create up to 3 sets and put photos in up to 10 groups.

The photostream for free account holders will show only their 200 most recent photos. Additionally, the "All Sizes" button will max out at the Large size, which generally will be 1024 pixels along the longest side.

Paid accounts are $25/year and include unlimited upload bandwidth (10MB max per photo though), unlimited photostream size, unlimited sets, and removal of all ads on the site. Additionally, the original upload is made available to you. If you have enabled the All Sizes button, it's also made available to the public. One nice thing is that with a free account, all that data is saved, even if it isn't available. So if you later upgrade to Pro, it's all there.

Prints

Flickr sells prints of your photos for $0.15 each. As with viewing, you can designate who can purchase prints of your photos. I ordered a test batch and they turned out very nice -- approximately the same quality as Shutterfly, which is one of the better consumer-quality labs. They are a bit pricier than Shutterfly when buying in bulk, though the convenience of all the Flickr uploaders may make up for that.

Annoyances

There are a few things that bug me.

One is that the "All Sizes" button is either on or off. Once I upgraded to pro, suddenly my original size images were available. There is no way with a pro account to limit image availability to the 1024xwhatever size like there is with free -- either you give people the medium size only (500xwhatever) or you give them everything. Those that want to have the originals on Flickr generally upload twice: once in original size, kept private; and another resized to 1024xwhatever, made public.

It is also complex for some people to join Flickr. Several of our family and friends had trouble with the signup process, since you first have to create a Yahoo account, then a Flickr one, which can use a different username. It's a nice flexibility, but overly complex for people that just want to sign up quickly. Guest Passes have received rave reviews however -- with the one caveat that the guest pass URLs aren't bookmarkable (they set a cookie and then redirect to the real target).

There is also no built-in way to see all the new photos in the groups you're a member of. However, each group's pool has an RSS feed so I have just subscribed to them with bloglines.

Overall

Overall I am happy with Flickr. I will be migrating my Gallery2 install to it, and will not miss having to maintain Yet Another PHP Script on my server. The best part about Flickr is the community, which has already taught me about mpix.com, which I'll write about tomorrow.

I’m just sitting in the night venue of Debconf7 and do several tasks in the same moment, that includes: * Getting kind of a spanish course on IRC in #debian-qt-kde * Get some news from friends in .ch via MSN (there were several unweathers and other troubles) * Think about tdbackup which is the topic of this blog entry. So what is tdbackup? It is just another backup implementation but it should work kind of distributed. I manage several servers and in that farm are 3 backup servers with a tape drive, every sunday i mount parts of the other machines via NFS and write them with tar to the tape. Until I did that by hand because there was no implementation which fits my needs unfortunately. This is the point where tdbackup jumps in. I want to create a GUI which edits an ini file in the form of: [backupmachine 1] mount = bla1:/home mount = bla2:/home tdbackup will then distribute a specifically generated configfile to each backuphost and a little python script which is called by cron and mounts the file systems, write some information into mysql and generate warning messages via e-mail if the tape is almost full. So, I will be able to sleep better because the stuff is auto-documented in a way. I’m still thinking about how exactly i want to implement that, if there are any interested people who would like to join that project, please just tell me.

Apparently there’s lots of interest in the TiVo-Comcast product that I took a picture of. Already that picture has nearly 350 views, and the blog entry got a couple questions about it. And it’s a crap photo of something that was seen at CES. The two questions that were asked were: When can I get it? I have no idea. All I know is that we got 1.0 done and Comcast currently has it. Most likely they’re going to be doing some testing of it privately, and then start market trials in select areas, but I don’t know when or where. (It’s also going to be available for Cox customers, but I have even less information about that.) Should I wait for it or get a Series 3? I can’t really answer that question. I have a Series 3 at home and it’s awesome. For me, if the choice were between this or a Series 3, I’d take the Series 3, despite the several hundred dollar price tag (especially since it’s currently available). Before I got the Series 3, I had the old Comcast DVR; and if I had to choose between that and the TiVo, I’d definitely choose the TiVo. (I’m biased, but I was a TiVo fan even before I got hired. That’s part of the reason I wanted to work here.) Disclaimer Please keep in mind that I’m not speaking on behalf of TiVo. I only work here.

Web applications are in constant competition for a user's attention. Unlike shrink-wrap software, there's often no captive audience, and web apps must gently guide and woo users to help them to the best experience of the software. This is particularly true when considering the problem of collecting personal details from a user. Collect too little information, and your app might not be able to function effectively. Collect too much information, and you raise the barrier for participation with your application. Users may either not be bothered, be alarmed at the apparent intrusion into their privacy, or fill out bogus information.  One size doesn't fit all
I originally stumbled on the issue in the development of Expectnation. The amount of information a conference speaker needs to give the organizers is significantly more than that of, say, a reviewer. An organizer will often want postal and phone details for a speaker, which are largely irrelevant to the review process. All conferences and events rely on the willingess of participants, so the path to participation needs to be as frictionless as possible.
Thus, a one-size-fits-all personal details form didn't make sense for us. At best, we could have simply made more fields optional, but that would result in missing information that we really did need. Another solution would be to provide a dedicated personal details form per user type, e.g. different forms for speakers and reviewers. This is a non-starter, however, as users can take on many roles within a conference. So, we had a problem on our hands: different levels of interaction with our application requires different levels of personal information. We also had a second problem, which is that not all events and organizers want the same level of information. For example, a user group event is unlikely to need the same amount of information from speakers as an academic conference. Thus we have two axes on which the amount of information required varies: by role, and by event.  Our solution: progressive disclosure We concluded that we wanted to adhere to two principles in the collection of user information for Expectnation:

• collect as little information from users as possible, and
  
• allow organizers to determine the level of information they wish to collect.

The first objective was achieved by constructing a matrix of information items against user role, with the possible values of required, optional, unused for each item. We then achieved the second objective by allowing this matrix to be varied over each event. We've named this concept "progressive disclosure." What this means in practice is that a user is only ever asked for the information required to fulfil the roles they have within Expectnation. If they take on another role, they are asked for the additional information at that point, and not before. Implementation concerns
Extract from disclosure level editing UI It would be untrue to say that progressive disclosure is simple to implement. A user in our system will have disclosure requirements that are the union of all of their roles in all of the events they attend. Within Ruby on Rails, this means the conventional validation logic is useless to us, as it only deals with static requirements. Secondly, disclosure requirements are expensive to compute, so we need to cache them, with the inevitable addition of issues such as cache invalidation. Beyond the data model issues, the user interface issues are, if anything, more troublesome. From an end-user's point of view, the difficulty arises when we need extra information from them, and presenting this in a way that makes sense. From the event organizer's point of view, we have the challenge of presenting a controlling user interface for rather a complex concept. Organizers must be able to change the level of required information either globally, or per-event. We opted for a grid presentation, an extract from which is shown in the screenshot in this article. Over time we'll be able to see how effective this interface is, and what we can do to improve it. Finally, we have chosen sensible defaults for all disclosure levels, as the best result would be that organizers have little need to reconfigure the settings in the first place. Making it look easy Because Expectnation is a general application for organizing events, rather than targetting one specific one, we've had to construct a framework for minimizing user information collection, as opposed to simple implementation of known scenarios. At the end of the day we have added a fairly complicated subsystem and an extra degree of complexity to our codebase. However, the result from the user's point of view is increased simplicity and a streamlined experience. Every time you embark on something that wide-reaching and complex, you have to ask yourself whether the feature is really worth it. In this case, though, the answer is a resounding "yes". The user is the most important factor in our software, and making their life easier in an unobtrusive manner is a priority.

• Expectnation blog: Only ask users for the information you need

The journey to Edinburgh on Wednesday went pretty well. The only problem I ran into was that after landing at Edinburgh airport we had to wait in the plane for a few more minutes because the airport's doors wouldn't open. I felt pretty beaten when I finally arrived at the venue and just exchanged a few greetings and took a look around before going to the hostel and finishing my night's sleep. When I got up again it was almost time to go for dinner and then pick up Tolimar who got here on a later plane, since his appclication for travel sponsorship got accepted much earlier. (He's giving a talk and a BoF, so he had a higher priority.) On Thursday I saw a few talks and attended the debian-women BoF, which I think was very productive and - most importantly - gave us back some of the momentum we'd lost over the last years. We've got a whole lot of plans for new things to do now and I am looking forward to getting to work on them when I get back. The day ended with the DebConf Ceilidh which was pretty fun altough it was quite obvious that geeks may not actually be the best dancers around. Lack of rhythm was easily made up with enthusiasm though and the slight lack of women was also easily compensated. I think that Ceilidh band will remember us. We even contributed a few musicians of our own! Once the Ceilidh was over I pretty much fell right into bed. Although it's good to be here and meet all the people, only being at DebConf for the last few days generally tends to be rather frustrating. Firstly it's always weird to join a group that has already been together for a while and also DebConf really just isn't something that can be condensed to a few days. I've just about started to get into the flow today (Friday) and tomorrow I'll already be leaving again. Shame. Anyway, only being here for a bit is still better than not being here at all and I very much hope to have more time next year when we meet in Argentina. For now I'll just enjoy my remaining last day at DebConf and hope for a better time next year.

Last December, I went to the Delta de l'Ebre with a good number of friends to spend one of December's three-day weekends. Les Terres de l'Ebre are the southest region of Catalunya, right next to the border with the Valencian Country. The Ebre river forms a big delta as it meets the Mediterranean, and the great amount of wildlife, specially avian, in the Natural Park makes it a fantastic place to visit. We would sleep, more or less, in Cinta's appartment in L'Ampolla. The first night we managed to fit 17 persons for dinner, and about 14 stayed to sleep in any possible corner of the flat. As soon as the Valencian group arrived, we tried to meet the people coming from the North, but apparently they got lost and we stayed on a sightseeing spot admiring how easily the weather changes here. In an hour or so, we went from a sunny day to a quick but heavy shower and back to sunny. When we finally found each other, we decided it was a bit too late to do anything too special, given the short days in December, so we walked South along the beach, on our way to the Salines, the salt processing facility, in the Punta de la Banya.

Agua para todos... During the walk, we witnessed what the people from the Delta have been warning about for decades: the Delta is disappearing. The progressive overuse of the river's flow, excessive irrigation, damm construction and degradation is killing this unique place. The waters are slowly claiming what once was theirs. An electricity line, at the beginning of our trip being at the end of the beach starts approaching the sea, or viceversa, as you move southward, to end up many metres inside the water. The old people in the Delta still remember the 200m tall lighthouse in the Illa de Buda, which the sea managed to devour.

Sunset at the salines del Delta When we reached the Punta we could watch a beautiful sunset over the mountains of Els Ports, and when it got dark and cold we headed to Tortosa, the area's capital, which I had recently visited for the first time for one of Softcatal 's meetings. There, we had dinner at a bar and met Jordi and Anna for a while, before going back to Ampolla, going to bed relatively early due to everyone being very tired.

On Saturday, we decided to have a look at the Delta... from above. So we drove to the top of the Cim del Caro, where we could admire a great view over the flat farmlands of the Delta, and then found a nice trail which took us to a nice forest, where we had lunch. After a visit to Alfara de Carles, we went back to the house, had a great dinner and played fun games at the table. Unfortunately, Sunday morning was dedicated to cleaning up the mess in the appartment and packing, as part of the group, the Jordis from Alcoi and the twins, had to leave early. The remaining four Valencians decided to go to Deltebre to have a local paella. It was quite good, although the bar tender laughed at our request to eat directly from the paella, rejecting dishes. Small cultural clash. ;)

I remember downloading and viewing the pictures of the trip a few days after getting back in Val ncia, and then losing track of them. Last week, while testing an EPIA board with some random hard drive, I suddenly realised there was a big chance the pictures would be on that drive, and after some search, I found them; I had been close to mkfs the disk a few minutes before. I hadn't stopped blaming myself for misplacing these pictures, as they remind me of this wonderful weekend. Luckily, they are back! Going back to the state of the Delta, experts say it will be mostly gone in 200 years. All of the Terres de l'Ebre have fought against the previous government's "PHN" plan to divert part of the river's flow to the North and South, to supply Barcelona, and specially the South of Val ncia, where the developing of the coastal areas is out of control. I fear that whenever PP regains power in Madrid, they will retake this idea. There is a lot of money involved in building a 700 km long scar in our land. The Delta de l'Ebre is a fantastic place which doesn't deserve a terrible fate like this, only for the benefit of the usual suspects. As the good people of this land would say...

Lo riu s vida. No al transvasament!

* This is a mandatory question for those who are in NM Queue. tarzeau, I have done it!

The source-searching system I was talking about previously is progressing slowly. So far I've synced the source to Etch to my local machine, total size 29Gb, and this evening I've started unpacking all the source. I'm still in the "a" section at the moment, but thanks to caching I should be able to re-sync the source archive and unpack newer revisions pretty speedily. The big problem at the moment is that the unpacking of all the archives is incredibly slow. Still I do have one new bug to report aatv: Buffer overflow in handling environmental variables.. That was found with:

rgrep getenv /mnt/mirror/unpacked   grep sprintf

(A very very very slow pair of greps. Hopefully once the unpacking has finished it will become faster. ha!) The only issue I see at the moment is that I might not have the disk space to store an unpacked tree. I've got 100Gb allocated, with 29Gb comprised of the source. I'll just have to hope that the source is less than 70Gb unpacked or do this in stages.) I've been working on a list of patterns and processes to run, I think pscan, rats, and its should be the first tools to run on the archive. Then after that some directed use of grep. If anybody else with more disk space and connectivity than myself be interested I can post the script(s) I'm using to sync and unpack .. Failing that I'll shut up now.

For all those people wondering what the Debian Tartan looks like, and what the fuzz about the kilts is: There. Update: people asked for the names. Updated the flickr page to include those I know of.

Last December, I went to the Delta de l'Ebre with a good number of friends to spend one of December's three-day weekends. Les Terres de l'Ebre are the southest region of Catalunya, right next to the border with the Valencian Country. The Ebre river forms a big delta as it meets the Mediterranean, and the great amount of wildlife, specially avian, in the Natural Park makes it a fantastic place to visit. We would sleep, more or less, in Cinta's appartment in L'Ampolla. The first night we managed to fit 17 persons for dinner, and about 14 stayed to sleep in any possible corner of the flat. As soon as the Valencian group arrived, we tried to meet the people coming from the North, but apparently they got lost and we stayed on a sightseeing spot admiring how easily the weather changes here. In an hour or so, we went from a sunny day to a quick but heavy shower and back to sunny. When we finally found each other, we decided it was a bit too late to do anything too special, given the short days in December, so we walked South along the beach, on our way to the Salines, the salt processing facility, in the Punta de la Banya.

Agua para todos... During the walk, we witnessed what the people from the Delta have been warning about for decades: the Delta is disappearing. The progressive overuse of the river's flow, excessive irrigation, damm construction and degradation is killing this unique place. The waters are slowly claiming what once was theirs. An electricity line, at the beginning of our trip being at the end of the beach starts approaching the sea, or viceversa, as you move southward, to end up many metres inside the water. The old people in the Delta still remember the 200m tall lighthouse in the Illa de Buda, which the sea managed to devour.

Sunset at the salines del Delta When we reached the Punta we could watch a beautiful sunset over the mountains of Els Ports, and when it got dark and cold we headed to Tortosa, the area's capital, which I had recently visited for the first time for one of Softcatal 's meetings. There, we had dinner at a bar and met Jordi and Anna for a while, before going back to Ampolla, going to bed relatively early due to everyone being very tired.

On Saturday, we decided to have a look at the Delta... from above. So we drove to the top of the Cim del Caro, where we could admire a great view over the flat farmlands of the Delta, and then found a nice trail which took us to a nice forest, where we had lunch. After a visit to Alfara de Carles, we went back to the house, had a great dinner and played fun games at the table. Unfortunately, Sunday morning was dedicated to cleaning up the mess in the appartment and packing, as part of the group, the Jordis from Alcoi and the twins, had to leave early. The remaining four Valencians decided to go to Deltebre to have a local paella. It was quite good, although the bar tender laughed at our request to eat directly from the paella, rejecting dishes. Small cultural clash. ;)

I remember downloading and viewing the pictures of the trip a few days after getting back in Val ncia, and then losing track of them. Last week, while testing an EPIA board with some random hard drive, I suddenly realised there was a big chance the pictures would be on that drive, and after some search, I found them; I had been close to mkfs the disk a few minutes before. I hadn't stopped blaming myself for misplacing these pictures, as they remind me of this wonderful weekend. Luckily, they are back! Going back to the state of the Delta, experts say it will be mostly gone in 200 years. All of the Terres de l'Ebre have fought against the previous government's "PHN" plan to divert part of the river's flow to the North and South, to supply Barcelona, and specially the South of Val ncia, where the developing of the coastal areas is out of control. I fear that whenever PP regains power in Madrid, they will retake this idea. There is a lot of money involved in building a 700 km long scar in our land. The Delta de l'Ebre is a fantastic place which doesn't deserve a terrible fate like this, only for the benefit of the usual suspects. As the good people of this land would say...

Lo riu s vida. No al transvasament!

One feature I’ve always found terribly useful for team maintained packages are commit emails. They help you know what’s going on without need for extra communication (eg. that somebody started packaging a new releases), and they are useful when there are new maintainers, because their work can be easily reviewed. Centralized version control systems like CVS or Subversion make possible and easy to have such messages sent automatically upon commit, so all fits nicely. Enter distributed version control systems; particularly those like Bazaar that can push over dumb protocols like SFTP: it is not possible to have the server do something (send an email) when new revisions are commited. One has to resort to solutions like bzr-email, where each individual commiter configures a plugin to send out the email from their machines when they commit. The result IME is that, either nobody does that, or not everybody does. Either way, no cookie. So I decided to bite the bullet here at DebConf, and write a tool to solve this problem: a program that monitors the branches in the machine where they are hosted, and sends emails for new revisions. The result is here. If you’d like to use it for an Alioth project, you just need to do:

 alioth% /var/lib/gforge/chroot/home/users/adeodato/bzr-hookless-email/alioth_helper.sh $PROJECT_NAME -d

(And ensure that the process does not die, or gets restarted when it does. I guess I could add code for that in the helper script, hm.)

Today it's been two weeks since I've landed in Edinburgh. On my trip I starting writing a blog post which since then I haven't had time to finish. By now my workload has become a lot less, but it's still to high to finish that post, which is mostly about my involvement in OLPC, if I remember correctly :-)

I'll stop here now and finish that post later, so I can do some videoteam and non-videoteam stuff, like preparing my talk about debian-comunity.org and having fun and chats with other people. And creating another loldebian blog post, which obviously has very high priority ;-)

There is no rain in Edinburgh. Except maybe yesterday.

Last week the Berkeley DB used by the Jabber server of our local linux user group crashed again. It was the second database crash in six months. But this time even db_recover was unable to repair the database, so I did what I had on my TODO list since the first crash: Migrate to ejabberd, which I was already using on two other Jabber servers. Meanwhile the service and all transports are back up again. There's only one visible change compared with the old setup, for security reasons connecting through port 5222 now requires starttls.

Last week the Berkeley DB used by the Jabber server of our local linux user group crashed again. It was the second database crash in six months. But this time even db_recover was unable to repair the database, so I did what I had on my TODO list since the first crash: Migrate to ejabberd, which I was already using on two other Jabber servers.

Meanwhile the service and all transports are back up again. There's only one visible change compared with the old setup, for security reasons connecting through port 5222 now requires starttls.

Some time ago, I needed a small but helpful tool which could start some shell command with bounded time and kill spawned process if it was timed out. There is timeout package in Debian but it has very serious bug #187092. In fact, this tool can't intercept the proper status of called command and can't kill processes spawned by called command. I've tried to fix the Debian's timeout package but the bug is still opened over 4 years. So I decided to code the own Perl implementation. The result was attached to bug report. I've noticed that my script is currently used by Exim packagers. It's nice to see that my work is still useful. I think more people can find this script helpful so I set the Google Code project: timeout-pl. You can download the newest version of this script from Google or CPAN. Have a fun.

I finally decided myself to switch my laptop to using a LVM over LUKS setup instead of LVM alone. The problem it involves is that there is currently no way that I know to do this in-place. I do have an external disk on which I could pvmove everything and back, but that would not be very challenging ;). So I started wondering how doable that would be. LUKS has a 1032 sectors (a sector being 512 bytes) overhead in which it store the LUKS header. Following the header, encrypted data just takes as much space as its decrypted counterpart, linearly. So all it would take to do in-place LUKS conversion would be to move data by 1032 sectors, and encrypt it. Which means the main problem is to have 1032 sectors free after the current volume. Fortunately, LVM doesn’t take up all the space on the partition a physical volume is setup, except if you’re unlucky or created partitions specifically so that it uses all the space. Because LVM physical volumes use the space by big chunks, 4MB by default), if the size of the partition is not 4MB aligned (modulo the LVM header), you have some free space between the physical volume end and the partition end. We thus need this free space to be greater than 1032 sectors. On my laptop, there is more than 1.8MB free there, so there would be no problem.
Note the debian-installer seems to do more than a simple cryptsetup luksFormat, as LUKS volumes it creates have an overhead of 2056 sectors instead of the 1032 sectors I got creating a volume by hand. Next problem is how to move the data and encrypt it in-place. Bonus points if that can be done without unmounting the filesystems. My first guess was to play around with the device mapper and a specially crafted FUSE filesystem to create a fake LUKS volume that could be half encrypted and half not, and would evolve, with time, into a fully encrypted LUKS volume. Since I didn’t know much about the device mapper, I looked at it a bit, and discovered a FUSE intermediate would be unnecessary: not only can you remap on-the-fly (which I already guessed was somehow possible, considering how LVM can do it, though I didn’t know much about the interaction between LVM and the d-m), but you can also mix dm-crypt and linear chunks in the same d-m table. It also appears LVM over LUKS has an overhead that is actually not necessary. How sad both use the device mapper but yet can’t do an efficient setup with it. When you setup LVM over LUKS, you open a LUKS volume, which will create a dm-crypt device mapper, and use this device mapper as the physical volume for LVM. Logical volumes created with LVM are actually device mappers using the physical volume as destination device. When you access a block on an LVM logical volume, your access is first mapped through the LV device mapper and then mapped again through the dm-crypt mapper. So it goes through 2 device mappers, but it could do with only one: the linear mapping used in most of the cases with LVM could be replaced by crypt mappings, that are just as linear, but also handle the encryption/decryption part. Playing around under qemu with a freshly installed etch system, I was able to implement this and came up with the following script. It should work with most setups, but use at your own risk, it comes with no warranty.


MAJOR=$(awk '/device-mapper/ print $1;exit ' /proc/devices)
dmsetup ls --target linear while read map extra; do
  TABLE=$(dmsetup table $map while IFS=" :" read start length linear major minor offset; do
    map2=
    if [ "$major" -eq "$MAJOR" ]; then
      IFS=":" read map2 extra <<EOF
$(dmsetup info -c --noheadings -j $major -m $minor)
EOF
    fi
    if [ -z "$map2" ] [ $(dmsetup table "$map2" wc -l) -gt 1 ] [ $(dmsetup table --target crypt "$map2" wc -l) -eq 0 ]; then
      echo $start $length $linear $major:$minor $offset
    else
      IFS=" " read cstart clength crypt format key IVoff dev coffset <<EOF
$(dmsetup table "$map2")
EOF
      echo $start $length crypt $format $key $(expr $IVoff + $offset) $dev $(expr $coffset + $offset)
    fi
  done)
  if (dmsetup table $map; echo "$TABLE") sort uniq -c grep -v "^ *2 " > /dev/null 2>&1;
  then
    echo Diverting $map
    dmsetup reload $map <<EOF
$TABLE
EOF
    dmsetup resume $map
  fi
done


[ Update: There was a dmsetup resume missing in the script ] After running this script, you should be able to run cryptsetup luksClose on your LUKS volume. Once you’ve done it, pvdisplay and other LVM tools will show you nothing, because the physical volume will just have disappeared… but everything will still be in place. Note that if you leave the LUKs volume open, you’d better be careful NOT to do operations with LVM (such as pvmove) because it could break everything. Note that on sid, you would need to add a –showkey option into the $(dmsetup table "$map2") line. This option is necessary for dmsetup to display the encryption keys. I wonder if LVM upstream would consider supporting LUKS volumes directly… Anyways, back to the in-place conversion to LVM over LUKs. The idea would be to:

• Create a file the size of the LUKS headers + one block.
• Create a loopback device for this file.
• luksFormat the loopback device.
• luksOpen the loopback device, so that dmsetup can give us the key to use. I wonder if there are ways to get the key not involving the creation of the device mapper.
• luksClose it, because we actually don’t need it
• Create a device mapper that linearly maps the whole LVM physical volume (i.e the real partition, but without the empty tail).
• Remap the LVM device mappers so that they use our own device mapper instead of the partition as physical volume.
• Here is the trickiest part, and I still have to take a look at how pvmove does to know exactly how this can be done (I guess it will require an additional device mapper): move data chunk by chunk (offsetting with the size of the LUKS headers), starting from the end of the LVM physical volume, and encrypt at the same time. Once a chunk is moved, adjust our device mapper so that it maps to the linear unmoved data followed by the moved encrypted data. As I said earlier, this is a setup that works.
• Once everything is moved, we are in a situation where our device mapper is a single crypt mapping, in the same shape as a luksOpen would have made it. We can now copy the LUKS headers at the beginning of the partition, and add the appropriate configuration to /etc/crypttab.

If I got it all right, both LUKS and LVM should be believing they have done this setup themselves, so unmounting, lvchange, etc. should just be fine. It might be necessary to regenerate the initramfs, though. I’ll tell you when I fuck up my hard disk try it. To be continued…