Enrique Dans writes about the "The Coming Battle: Apple's iPhone vs. Corporate IT Departments". He makes an interesting point on how this new gadget can be a point of friction between managers of IT departments and their employees using this kind of gadgets, usually corporate executives. Syncing the database with RIM devices, or redirecting email accounts (iPhone is not capable of using corporate Blackberry or MS Exchange servers), can be a problem in corporate world. If you read those articles, they blame iPhone for not supporting all these enterprise-class software. Basically, IT departments are blaming iPhone for not supporting their competition systems, and they are not requiring all them (and themselves) to be using standards, which were just designed to avoid this kind of problems. They are afraid. No more comments.

I was looking for a way of sharing folders, without the need of using Samba but that could be used from Linux and Windows (I actually wanted to share a folder in my Debian server to a Windows client). I found an article on sharing folders using FTP and Avahi. It is quite easy to implement, and works like a charm. [Original post] I think that all these ubiquous network technologies are very interesting as network access trend is to be a commodity, available in every place. One of them is Avahi and the other is XMPP (previously known as Jabber). Mixing both makes creating ad-hoc networks in which people can share files, chat, printing, and use distributed SCM repos painless, and without the need to set up server-like stuff in you laptop. Update (24th July): Emanuele Aina pointed me to Telekinesis project which makes just what I was talking about above, using avahi and XMPP, it makes possible to share folders among peers in meshed networks, without the need to use any kind of server stuff.

I was looking for a way of sharing folders, without the need of using Samba but that could be used from Linux and Windows (I actually wanted to share a folder in my Debian server to a Windows client). I found an article on sharing folders using FTP and Avahi. It is quite easy to implement, and works like a charm. [Original post] I think that all these ubiquous network technologies are very interesting as network access trend is to be a commodity, available in every place. One of them is Avahi and the other is XMPP (previously known as Jabber). Mixing both makes creating ad-hoc networks in which people can share files, chat, printing, and use distributed SCM repos painless, and without the need to set up server-like stuff in you laptop. Update (24th July): Emanuele Aina pointed me to Telekinesis project which makes just what I was talking about above, using avahi and XMPP, it makes possible to share folders among peers in meshed networks, without the need to use any kind of server stuff.

Following my previous entry about playing around with LVM, LUKS and the device mapper, I documented myself about internals involved in pvmove, which is more of a challenge, considering there is no such documentation. I could find no useful documentation for either the device mapper or LVM. A bit of good old UTSL later, I could elaborate how to do what I wanted, and realized it was even possible to do in shell script. So here you are : a shell script to transform an LVM physical volume to a LVM over LUKS physical volume,. I’ll detail in between how it works. As the previous script, use at your own risk, it comes with no warranty.
Note you theorically can still access the filesystems underneath without problems. It’s an in-place and live conversion. Also note this is more a proof of concept than a proper, risk-less and well-written solution.


set -e
dev=$1
luks=$(mktemp)
cryptdev=$(basename $dev)_crypt
pvsize=$(pvs -o pe_start,pv_size --units s --noheadings --nosuffix "$dev" awk ' print $1 + $2 ')
devsize=$(blockdev --getsz "$dev")
mchunk=8


The script takes the physical volume device to convert as an argument. Note there is no check for the validity of its value.

• luks will be the temporary file where we are going to create a temporary LUKS volume.
• pvsize is the full size of the LVM physical volume, i.e. the size of all the physical extents (pv_size) + the LVM headers and metadata (pe_start).
• devsize is the raw device size.
• mchunk is the size in blocks of chunks for the mirror target, see below.


dd of="$luks" seek=$devsize count=0 bs=512 2> /dev/null
luksdev=$(losetup -f)
losetup "$luksdev" "$luks"
trap "losetup -d \"$luksdev\"; rm -f \"$luks\"" EXIT


Next, we create a sparse luks file the same size as the device (in case luksFormat would use the size somehow, but I believe it doesn’t), and a loopback device on this file. The trap is here to avoid leaving the loopback device and the file when an error occurs later (though during the conversion itself, it will be pointless).


cryptsetup luksFormat -q "$luksdev"
cryptsetup luksOpen "$luksdev" $ cryptdev _real
read start length crypt format key IVoff cdev offset <<EOF
$(dmsetup table $ cryptdev _real)
EOF


We create a LUKS device, so that we can get the encryption key ($key), and the size of the LUKS header ($offset). Note you need to add –showkey to the dmsetup table command on sid.


if [ $(expr $devsize - $pvsize) -lt $offset ]; then
  echo Not enough free space after LVM physical volume
  cryptsetup luksClose $ cryptdev _real
  exit
fi


Check we have enough space after the LVM physical volume to offset everything by the size of the LUKS header. If not, you can still try again after you reduce the size of the LVM physical volume by an extent.


if [ $(expr $devsize % $offset % $mchunk) -gt 0 ]; then
  echo Last
  cryptsetup luksClose $ cryptdev _real
  exit
fi


This is another check to avoid surprises at the end, when dealing with the last chunk. As the script is written for the moment, it doesn’t support cases where this last chunk is not a multiple of $mchunk. So we need to abort in these.


read major minor <<EOF
$(stat -t "$dev" awk ' print $10,$11 ')
EOF
maps=$(dmsetup deps awk -F: "/\($major, $minor\)/ print \$1 ")
dmsetup create $cryptdev <<EOF
0 $length linear $dev 0
EOF
dmsetup reload $ cryptdev _real <<EOF
$start $length crypt $format $key $IVoff $dev $offset
EOF
dmsetup resume $ cryptdev _real
for map in $ maps ; do
  dmsetup table "$map" sed s,$major:$minor,/dev/mapper/$cryptdev, dmsetup reload "$map"
  dmsetup resume "$map"
done


Here, we create the dev_crypt device mapper that is our fake LUKS device, which starts as a simple linear mapper and will end as a complete crypt mapper. This fake LUKS device is inserted as an intermediate mapper between the LVM device mapper and the real device. So, the LVM device mapper will have this fake LUKS device as backend, and, at the beginning, the fake LUKS device maps linearly to the real device. Note we look for all device mappers using the real device as backend before creating the fake LUKS device to avoid finding the fake LUKS device in the list. Also note dmsetup reload only loads a new table in the INACTIVE slot, and dmsetup resume makes this inactive table LIVE.


cursor=$length
chunk=$offset
while [ $cursor -gt 0 ]; do
  cursor=$(expr $cursor - $chunk)
  if [ $cursor -lt 0 ]; then
    chunk=$(expr $chunk + $cursor)
    cursor=0
  fi
  (
  [ $cursor -ne 0 ] && echo 0 $cursor linear $dev 0
  echo $cursor $chunk mirror core 1 $mchunk 2 $dev $cursor /dev/mapper/$ cryptdev _real $cursor
  [ $cursor -lt $(expr $length - $chunk) ] && echo $(expr $cursor + $chunk) $(expr $length - $cursor - $chunk) crypt $format $key $(expr $IVoff + $cursor + $chunk) $dev $(expr $offset + $cursor + $chunk)
  ) dmsetup reload "$cryptdev"
  dmsetup resume "$cryptdev"
  chunks=$(expr $chunk / $mchunk)
  while ! dmsetup status "$cryptdev" grep "$chunks/$chunks"; do
    true
  done
done


This is where the main work is done : moving the data around. We actually just let the device mapper deal with the data duplication, $offset blocks by $offset blocks ($offset being the LUKS header size), using a mirror target for the chunk being moved. So our disk looks like the following:

We use the extra dev_crypt_real device (the previously remapped LUKS device) as the encryption backend for the mirror.
I haven’t figured a better way to wait for the end of the mirroring than to do a loop checking with dmsetup status, dmsetup wait doesn’t seem to be very helpful here.
Anyways, this is the part of the script where you don’t want a crash to occur. Because if it does, all you can do is start on a rescue system, and try to find where the encrypted part of the disk start to setup a device mapper by hand.
And you’d better have the luks temporary file in a directory that is neither in RAM (think tmpfs) nor in the LVM you are converting (/tmp in a default etch install is, for instance ; note the script works nevertheless fine in this case). Also note the trap will remove the luks temporary file if the script exits…


dmsetup reload "$cryptdev" <<EOF
$start $length crypt $format $key $IVoff $dev $offset
EOF
dmsetup resume "$cryptdev"
dmsetup remove "$ cryptdev _real"
dd if="$luks" of=$dev count=$offset bs=512 2> /dev/null


Final steps of the conversion : our dev_crypt device becomes a full LUKS volume, so we can remove dev_crypt_real and add the LUKS headers at the top of the device we converted. At this moment, the LUKS volume is setup just as if it had been setup by cryptsetup. For LVM to recognize the change properly, you need to run pvscan. Once you ran it, you can do whatever you want with LVM. Now, you may want to add the following to your /etc/crypttab file:


$cryptdev $dev none luks


i.e. hda5_crypt /dev/hda5 none luks if the device was /dev/hda5. And if the LVM volume you converted contains your root filesystem, you should run (for Debian systems):


update-initramfs -u


I tested this successfully under qemu and will give it a shot on my laptop some time soon. Now, because I had a hard time not finding much about the mirror target of the device mapper, here is what I could gather about it. The target syntax is as follows:


<logical_start_sector> <num_sectors> mirror [ core disk ] <num_params> <param> ... <num_mirrors> [ <destination > <start_sector> ] ...


logical_start_sector, num_sectors, destination and start_sector have the same meaning as in other targets.
core and disk are two different log types (to track differences between mirrors), respectively in memory and on disk.
num_params and params depend on the log type:

• for core, num_params can be either 1 or 2.
  The first parameter is region_size, which is the size (in blocks) of the chunks the mirror log tracks for synchronization. It can’t be less than the page size divided by 512, i.e. 8 on x86, must be a power of 2, and must not exceed num_sectors. Apparently, it is not a problem if num_sectors is not a multiple of region_size.
  The optional second parameter is either sync or nosync, meaning of which I’m not sure. I think it determines whether the mirror should do the initial synchronization (sync) or not (nosync).
• for disk, num_params can be either 2 or 3, with params being log_device (device where the logs are kept), region_size, as above, and an optional sync or nosync, as above too.

num_mirrors is the number of mirrors and for each mirror, we have a pair destination and start_sector. [ Update: after a quick look at the device mapper source code in Linus’s git tree, updated the mirror target description ]

Those of you that look at photos on my blog may have noticed that I recently switched from a self-hosted Gallery2 installation to the popular photo-sharing site Flickr.

I initially decided to try Flickr for a simple reason: digikam's image upload to Gallery2 has never worked for me, and I hoped that its Flickr upload would work (it did).

I had heard good things about Flickr from friends and colleagues, so I gave it a try.

First Impressions

Almost the first thing I noticed about Flickr was that there are people there that are interested in the same things I'm interested in. I discovered the Anabaptist History group. I've been contacted by a person in North Dakota that is interested German-speaking Russians, as my ancestors were. There are groups about Debian, classic-style black & white photography on digital cameras, and really just about any topic. These groups contain both discussions and a photo "pool" to which any group member can post.

In one sense, Flickr is social networking via photography.

Flickr also seems to be a company with a clue. It has tons of open APIs, permitting everything from flickrfs (mount flickr as a FUSE filesystem) to countless uploaders, mashups, scripts, etc. Despite a couple of recent high-publicity controversies, they seem to be laid back overall and haven't been turned into a corporate drone by Yahoo. There's an RSS feed for just about everything, too.

What is Flickr?

Depends on what you want it to be. It could be that place where you post your cell phone's snapshots, or a place to share photos with family in private, or just a place to find other's work.

I found sharing and talking with others to be infectious. I went from never getting comments on photos on my Gallery2 site to wondering what people will think -- and looking at their photos too.

Let's look at its different aspects in more detail.

Uploading Photos

You can upload photos manually with a HTTP form, or you can use any of the numerous uploading tools. Most of these will let you scale down your images on the fly, should you wish to. Most will also let you assign tags and write descriptions as they're being posted, as well as define who can view the photos. Uploaders are available for every common platform.

Flickr has a AJAX tool called the Organizr that helps you assign tags, add photos to groups, put them in sets, etc. It's fairly nice.

Sharing Photos

Making photos available to the public is easy. You can just them them a link to your photostream. You can also tag photos and give people a link to photos with a specific tag. And you can create sets -- photos around a specific theme.

Security

Each photo can be made visible to: just you, just your contacts marked "family", just your contacts marked "friend", any of your contacts, or anyone at all. This can be defined per photo.

On a global basis, you can define whether you make EXIF information visible on the site, default security for new uploads, who can post comments, who can post photo notes or add tags, and who can use the "all sizes" button.

You can also control how much of your profile to make public. Settings range from making everything public to concealing even your email address.

For sets, you can also send out a "Guest Pass" (GP). A GP lets someone that is not a Flickr member see non-public photos. I use GPs to share family photos, so that family doesn't have to create a Flickr account to see them. GPs can be tied only to sets.

Groups

Flickr groups appear to have been designed for discussions, but I find them most attractive because each group has its own dedicated photo pool. Members of the group can add their photos to the pool. Some groups have tens of thousands of photos, and others a few dozen. It's an interesting way to find images. There are a lot of group management tools as well.

Free vs. Paid Accounts

Free account holders can upload up to 100MB of photos per month. There is no cap on bandwidth consumed by views of photos or management of them. Free users can create up to 3 sets and put photos in up to 10 groups.

The photostream for free account holders will show only their 200 most recent photos. Additionally, the "All Sizes" button will max out at the Large size, which generally will be 1024 pixels along the longest side.

Paid accounts are $25/year and include unlimited upload bandwidth (10MB max per photo though), unlimited photostream size, unlimited sets, and removal of all ads on the site. Additionally, the original upload is made available to you. If you have enabled the All Sizes button, it's also made available to the public. One nice thing is that with a free account, all that data is saved, even if it isn't available. So if you later upgrade to Pro, it's all there.

Prints

Flickr sells prints of your photos for $0.15 each. As with viewing, you can designate who can purchase prints of your photos. I ordered a test batch and they turned out very nice -- approximately the same quality as Shutterfly, which is one of the better consumer-quality labs. They are a bit pricier than Shutterfly when buying in bulk, though the convenience of all the Flickr uploaders may make up for that.

Annoyances

There are a few things that bug me.

One is that the "All Sizes" button is either on or off. Once I upgraded to pro, suddenly my original size images were available. There is no way with a pro account to limit image availability to the 1024xwhatever size like there is with free -- either you give people the medium size only (500xwhatever) or you give them everything. Those that want to have the originals on Flickr generally upload twice: once in original size, kept private; and another resized to 1024xwhatever, made public.

It is also complex for some people to join Flickr. Several of our family and friends had trouble with the signup process, since you first have to create a Yahoo account, then a Flickr one, which can use a different username. It's a nice flexibility, but overly complex for people that just want to sign up quickly. Guest Passes have received rave reviews however -- with the one caveat that the guest pass URLs aren't bookmarkable (they set a cookie and then redirect to the real target).

There is also no built-in way to see all the new photos in the groups you're a member of. However, each group's pool has an RSS feed so I have just subscribed to them with bloglines.

Overall

Overall I am happy with Flickr. I will be migrating my Gallery2 install to it, and will not miss having to maintain Yet Another PHP Script on my server. The best part about Flickr is the community, which has already taught me about mpix.com, which I'll write about tomorrow.

The journey to Edinburgh on Wednesday went pretty well. The only problem I ran into was that after landing at Edinburgh airport we had to wait in the plane for a few more minutes because the airport's doors wouldn't open. I felt pretty beaten when I finally arrived at the venue and just exchanged a few greetings and took a look around before going to the hostel and finishing my night's sleep. When I got up again it was almost time to go for dinner and then pick up Tolimar who got here on a later plane, since his appclication for travel sponsorship got accepted much earlier. (He's giving a talk and a BoF, so he had a higher priority.) On Thursday I saw a few talks and attended the debian-women BoF, which I think was very productive and - most importantly - gave us back some of the momentum we'd lost over the last years. We've got a whole lot of plans for new things to do now and I am looking forward to getting to work on them when I get back. The day ended with the DebConf Ceilidh which was pretty fun altough it was quite obvious that geeks may not actually be the best dancers around. Lack of rhythm was easily made up with enthusiasm though and the slight lack of women was also easily compensated. I think that Ceilidh band will remember us. We even contributed a few musicians of our own! Once the Ceilidh was over I pretty much fell right into bed. Although it's good to be here and meet all the people, only being at DebConf for the last few days generally tends to be rather frustrating. Firstly it's always weird to join a group that has already been together for a while and also DebConf really just isn't something that can be condensed to a few days. I've just about started to get into the flow today (Friday) and tomorrow I'll already be leaving again. Shame. Anyway, only being here for a bit is still better than not being here at all and I very much hope to have more time next year when we meet in Argentina. For now I'll just enjoy my remaining last day at DebConf and hope for a better time next year.

1. Being at debconf. 2. Having your blog entry slamming Blue Cross Blue Shield of NJ be on the first page of a google search using both the terms Blue Cross Blue Shield New Jersey as well as Blue Cross Blue Shield Horizon.

Last December, I went to the Delta de l'Ebre with a good number of friends to spend one of December's three-day weekends. Les Terres de l'Ebre are the southest region of Catalunya, right next to the border with the Valencian Country. The Ebre river forms a big delta as it meets the Mediterranean, and the great amount of wildlife, specially avian, in the Natural Park makes it a fantastic place to visit. We would sleep, more or less, in Cinta's appartment in L'Ampolla. The first night we managed to fit 17 persons for dinner, and about 14 stayed to sleep in any possible corner of the flat. As soon as the Valencian group arrived, we tried to meet the people coming from the North, but apparently they got lost and we stayed on a sightseeing spot admiring how easily the weather changes here. In an hour or so, we went from a sunny day to a quick but heavy shower and back to sunny. When we finally found each other, we decided it was a bit too late to do anything too special, given the short days in December, so we walked South along the beach, on our way to the Salines, the salt processing facility, in the Punta de la Banya.

Agua para todos... During the walk, we witnessed what the people from the Delta have been warning about for decades: the Delta is disappearing. The progressive overuse of the river's flow, excessive irrigation, damm construction and degradation is killing this unique place. The waters are slowly claiming what once was theirs. An electricity line, at the beginning of our trip being at the end of the beach starts approaching the sea, or viceversa, as you move southward, to end up many metres inside the water. The old people in the Delta still remember the 200m tall lighthouse in the Illa de Buda, which the sea managed to devour.

Sunset at the salines del Delta When we reached the Punta we could watch a beautiful sunset over the mountains of Els Ports, and when it got dark and cold we headed to Tortosa, the area's capital, which I had recently visited for the first time for one of Softcatal 's meetings. There, we had dinner at a bar and met Jordi and Anna for a while, before going back to Ampolla, going to bed relatively early due to everyone being very tired.

On Saturday, we decided to have a look at the Delta... from above. So we drove to the top of the Cim del Caro, where we could admire a great view over the flat farmlands of the Delta, and then found a nice trail which took us to a nice forest, where we had lunch. After a visit to Alfara de Carles, we went back to the house, had a great dinner and played fun games at the table. Unfortunately, Sunday morning was dedicated to cleaning up the mess in the appartment and packing, as part of the group, the Jordis from Alcoi and the twins, had to leave early. The remaining four Valencians decided to go to Deltebre to have a local paella. It was quite good, although the bar tender laughed at our request to eat directly from the paella, rejecting dishes. Small cultural clash. ;)

I remember downloading and viewing the pictures of the trip a few days after getting back in Val ncia, and then losing track of them. Last week, while testing an EPIA board with some random hard drive, I suddenly realised there was a big chance the pictures would be on that drive, and after some search, I found them; I had been close to mkfs the disk a few minutes before. I hadn't stopped blaming myself for misplacing these pictures, as they remind me of this wonderful weekend. Luckily, they are back! Going back to the state of the Delta, experts say it will be mostly gone in 200 years. All of the Terres de l'Ebre have fought against the previous government's "PHN" plan to divert part of the river's flow to the North and South, to supply Barcelona, and specially the South of Val ncia, where the developing of the coastal areas is out of control. I fear that whenever PP regains power in Madrid, they will retake this idea. There is a lot of money involved in building a 700 km long scar in our land. The Delta de l'Ebre is a fantastic place which doesn't deserve a terrible fate like this, only for the benefit of the usual suspects. As the good people of this land would say...

Lo riu s vida. No al transvasament!

After my brother's wedding, a bunch of us Goerzens got together out at our creek. We had a hot dog roast, some smores, and then eight of us remained to camp out at the creek. That included Terah, Jacob, and me.

My uncle loves camping and has organized camping at the creek many times over the years. This was the first time it worked out for me to join them in quite awhile. Jacob slept well overnight, but woke up 5 minutes late -- 6:35. Never mind that he got to bed hours later than his usual time, he still woke up at the same time as always. He's funny that way.

My uncle brought with him supplies for making pancakes over the campfire in the morning. They were great. After breakfast, it was time to take down the tent and get back home to shower before going to church.

Now, a word about Terah and camping here. Terah had only gone camping with me once since we were married, and that was two years ago. This time, we had a larger tent, warmer weather, and most importantly, a complete lack of raccoon fights all around us.

I had visited a Coleman Factory Outlet store in Wichita in the morning, before the wedding, and bought a new Coleman Crestline tent (the one on the left above) and some other supplies. It's a very nice tent with a lot of ventilation options and plenty of little pockets to store things. Jacob slept in his car seat (which he likes to do) in the tent with us.

I think we might not have to wait two years before going camping again. Though I'm sure Terah will correct me if I'm wrong on that.

Last December, I went to the Delta de l'Ebre with a good number of friends to spend one of December's three-day weekends. Les Terres de l'Ebre are the southest region of Catalunya, right next to the border with the Valencian Country. The Ebre river forms a big delta as it meets the Mediterranean, and the great amount of wildlife, specially avian, in the Natural Park makes it a fantastic place to visit. We would sleep, more or less, in Cinta's appartment in L'Ampolla. The first night we managed to fit 17 persons for dinner, and about 14 stayed to sleep in any possible corner of the flat. As soon as the Valencian group arrived, we tried to meet the people coming from the North, but apparently they got lost and we stayed on a sightseeing spot admiring how easily the weather changes here. In an hour or so, we went from a sunny day to a quick but heavy shower and back to sunny. When we finally found each other, we decided it was a bit too late to do anything too special, given the short days in December, so we walked South along the beach, on our way to the Salines, the salt processing facility, in the Punta de la Banya.

Agua para todos... During the walk, we witnessed what the people from the Delta have been warning about for decades: the Delta is disappearing. The progressive overuse of the river's flow, excessive irrigation, damm construction and degradation is killing this unique place. The waters are slowly claiming what once was theirs. An electricity line, at the beginning of our trip being at the end of the beach starts approaching the sea, or viceversa, as you move southward, to end up many metres inside the water. The old people in the Delta still remember the 200m tall lighthouse in the Illa de Buda, which the sea managed to devour.

Sunset at the salines del Delta When we reached the Punta we could watch a beautiful sunset over the mountains of Els Ports, and when it got dark and cold we headed to Tortosa, the area's capital, which I had recently visited for the first time for one of Softcatal 's meetings. There, we had dinner at a bar and met Jordi and Anna for a while, before going back to Ampolla, going to bed relatively early due to everyone being very tired.

On Saturday, we decided to have a look at the Delta... from above. So we drove to the top of the Cim del Caro, where we could admire a great view over the flat farmlands of the Delta, and then found a nice trail which took us to a nice forest, where we had lunch. After a visit to Alfara de Carles, we went back to the house, had a great dinner and played fun games at the table. Unfortunately, Sunday morning was dedicated to cleaning up the mess in the appartment and packing, as part of the group, the Jordis from Alcoi and the twins, had to leave early. The remaining four Valencians decided to go to Deltebre to have a local paella. It was quite good, although the bar tender laughed at our request to eat directly from the paella, rejecting dishes. Small cultural clash. ;)

I remember downloading and viewing the pictures of the trip a few days after getting back in Val ncia, and then losing track of them. Last week, while testing an EPIA board with some random hard drive, I suddenly realised there was a big chance the pictures would be on that drive, and after some search, I found them; I had been close to mkfs the disk a few minutes before. I hadn't stopped blaming myself for misplacing these pictures, as they remind me of this wonderful weekend. Luckily, they are back! Going back to the state of the Delta, experts say it will be mostly gone in 200 years. All of the Terres de l'Ebre have fought against the previous government's "PHN" plan to divert part of the river's flow to the North and South, to supply Barcelona, and specially the South of Val ncia, where the developing of the coastal areas is out of control. I fear that whenever PP regains power in Madrid, they will retake this idea. There is a lot of money involved in building a 700 km long scar in our land. The Delta de l'Ebre is a fantastic place which doesn't deserve a terrible fate like this, only for the benefit of the usual suspects. As the good people of this land would say...

Lo riu s vida. No al transvasament!

One feature I’ve always found terribly useful for team maintained packages are commit emails. They help you know what’s going on without need for extra communication (eg. that somebody started packaging a new releases), and they are useful when there are new maintainers, because their work can be easily reviewed. Centralized version control systems like CVS or Subversion make possible and easy to have such messages sent automatically upon commit, so all fits nicely. Enter distributed version control systems; particularly those like Bazaar that can push over dumb protocols like SFTP: it is not possible to have the server do something (send an email) when new revisions are commited. One has to resort to solutions like bzr-email, where each individual commiter configures a plugin to send out the email from their machines when they commit. The result IME is that, either nobody does that, or not everybody does. Either way, no cookie. So I decided to bite the bullet here at DebConf, and write a tool to solve this problem: a program that monitors the branches in the machine where they are hosted, and sends emails for new revisions. The result is here. If you’d like to use it for an Alioth project, you just need to do:

 alioth% /var/lib/gforge/chroot/home/users/adeodato/bzr-hookless-email/alioth_helper.sh $PROJECT_NAME -d

(And ensure that the process does not die, or gets restarted when it does. I guess I could add code for that in the helper script, hm.)

So, I'm happy to see that commercial wiki provider WetPaint now supports OpenID for logging into their site. It's great to have more wiki services supporting this single sign-on standard. A couple of nits, though: first, I logged in with my normal OpenID, http://evan.prodromou.name/ , which delegates to http://certifi.ca/evan . WetPaint's OpenID service used the delegated URL, which is just wrong. They need to fix that. The delegate is the "backend", and users should be able to change it over their lifetime. If relying parties store that backend URL instead of the "main" OpenID URL, that undermines the process. Second, they have a signup page after the validation, and it has a lot of fields that could be pre-populated using the OpenID Simple Registration extension. I shouldn't have to type in my preferred user name, real name, or email address. That information is already available through my OpenID provider. Other than that, it seems to work OK. I'm glad that it exists at all, but I wish it was a smoother user experience. tags: wetpaint wiki openid sso

Wikispaces and SourceForge I haven't already mentioned the announcement that SourceForge has implemented Wikispaces wikis for every project on their site. I think that's great news for the wiki community, and probably a good thing for SourceForge users. I feel churlish mentioning it, but it has to be said: with hundreds of available WikiEngines, the vast majority of which are Open Source, and many of which are hosted on SourceForge, why did SourceForge go with a commercial service provider? It doesn't seem like they've justified that decision very well. tags: wikispaces sourceforge wiki announcement

Just to prevent people from thinking I’ve gotten old and stodgy, I’ve joined the Twitter bandwagon. Here’s my feed, which also appears in my sidebar. I’m reminded of a Utah Phillips quote, from a different context:

Well, and I’m open to all those things. If you live in California, you’ve got to be open; if you’re not they pry you open.

I’m not sure whether I really agree with Twitter–it reminds me of this interesting collection of postmodern thoughts from n+1 magazine:

The Decivilizing Process As the specific addressee of any set of remarks becomes less important, in the midst of more and more babble, it will become more and more difficult to remember the special status of listening human beings, in the confusion of shouted orders.

But, hey, it’s worth a try.

Share This

I finally decided myself to switch my laptop to using a LVM over LUKS setup instead of LVM alone. The problem it involves is that there is currently no way that I know to do this in-place. I do have an external disk on which I could pvmove everything and back, but that would not be very challenging ;). So I started wondering how doable that would be. LUKS has a 1032 sectors (a sector being 512 bytes) overhead in which it store the LUKS header. Following the header, encrypted data just takes as much space as its decrypted counterpart, linearly. So all it would take to do in-place LUKS conversion would be to move data by 1032 sectors, and encrypt it. Which means the main problem is to have 1032 sectors free after the current volume. Fortunately, LVM doesn’t take up all the space on the partition a physical volume is setup, except if you’re unlucky or created partitions specifically so that it uses all the space. Because LVM physical volumes use the space by big chunks, 4MB by default), if the size of the partition is not 4MB aligned (modulo the LVM header), you have some free space between the physical volume end and the partition end. We thus need this free space to be greater than 1032 sectors. On my laptop, there is more than 1.8MB free there, so there would be no problem.
Note the debian-installer seems to do more than a simple cryptsetup luksFormat, as LUKS volumes it creates have an overhead of 2056 sectors instead of the 1032 sectors I got creating a volume by hand. Next problem is how to move the data and encrypt it in-place. Bonus points if that can be done without unmounting the filesystems. My first guess was to play around with the device mapper and a specially crafted FUSE filesystem to create a fake LUKS volume that could be half encrypted and half not, and would evolve, with time, into a fully encrypted LUKS volume. Since I didn’t know much about the device mapper, I looked at it a bit, and discovered a FUSE intermediate would be unnecessary: not only can you remap on-the-fly (which I already guessed was somehow possible, considering how LVM can do it, though I didn’t know much about the interaction between LVM and the d-m), but you can also mix dm-crypt and linear chunks in the same d-m table. It also appears LVM over LUKS has an overhead that is actually not necessary. How sad both use the device mapper but yet can’t do an efficient setup with it. When you setup LVM over LUKS, you open a LUKS volume, which will create a dm-crypt device mapper, and use this device mapper as the physical volume for LVM. Logical volumes created with LVM are actually device mappers using the physical volume as destination device. When you access a block on an LVM logical volume, your access is first mapped through the LV device mapper and then mapped again through the dm-crypt mapper. So it goes through 2 device mappers, but it could do with only one: the linear mapping used in most of the cases with LVM could be replaced by crypt mappings, that are just as linear, but also handle the encryption/decryption part. Playing around under qemu with a freshly installed etch system, I was able to implement this and came up with the following script. It should work with most setups, but use at your own risk, it comes with no warranty.


MAJOR=$(awk '/device-mapper/ print $1;exit ' /proc/devices)
dmsetup ls --target linear while read map extra; do
  TABLE=$(dmsetup table $map while IFS=" :" read start length linear major minor offset; do
    map2=
    if [ "$major" -eq "$MAJOR" ]; then
      IFS=":" read map2 extra <<EOF
$(dmsetup info -c --noheadings -j $major -m $minor)
EOF
    fi
    if [ -z "$map2" ] [ $(dmsetup table "$map2" wc -l) -gt 1 ] [ $(dmsetup table --target crypt "$map2" wc -l) -eq 0 ]; then
      echo $start $length $linear $major:$minor $offset
    else
      IFS=" " read cstart clength crypt format key IVoff dev coffset <<EOF
$(dmsetup table "$map2")
EOF
      echo $start $length crypt $format $key $(expr $IVoff + $offset) $dev $(expr $coffset + $offset)
    fi
  done)
  if (dmsetup table $map; echo "$TABLE") sort uniq -c grep -v "^ *2 " > /dev/null 2>&1;
  then
    echo Diverting $map
    dmsetup reload $map <<EOF
$TABLE
EOF
    dmsetup resume $map
  fi
done


[ Update: There was a dmsetup resume missing in the script ] After running this script, you should be able to run cryptsetup luksClose on your LUKS volume. Once you’ve done it, pvdisplay and other LVM tools will show you nothing, because the physical volume will just have disappeared… but everything will still be in place. Note that if you leave the LUKs volume open, you’d better be careful NOT to do operations with LVM (such as pvmove) because it could break everything. Note that on sid, you would need to add a –showkey option into the $(dmsetup table "$map2") line. This option is necessary for dmsetup to display the encryption keys. I wonder if LVM upstream would consider supporting LUKS volumes directly… Anyways, back to the in-place conversion to LVM over LUKs. The idea would be to:

• Create a file the size of the LUKS headers + one block.
• Create a loopback device for this file.
• luksFormat the loopback device.
• luksOpen the loopback device, so that dmsetup can give us the key to use. I wonder if there are ways to get the key not involving the creation of the device mapper.
• luksClose it, because we actually don’t need it
• Create a device mapper that linearly maps the whole LVM physical volume (i.e the real partition, but without the empty tail).
• Remap the LVM device mappers so that they use our own device mapper instead of the partition as physical volume.
• Here is the trickiest part, and I still have to take a look at how pvmove does to know exactly how this can be done (I guess it will require an additional device mapper): move data chunk by chunk (offsetting with the size of the LUKS headers), starting from the end of the LVM physical volume, and encrypt at the same time. Once a chunk is moved, adjust our device mapper so that it maps to the linear unmoved data followed by the moved encrypted data. As I said earlier, this is a setup that works.
• Once everything is moved, we are in a situation where our device mapper is a single crypt mapping, in the same shape as a luksOpen would have made it. We can now copy the LUKS headers at the beginning of the partition, and add the appropriate configuration to /etc/crypttab.

If I got it all right, both LUKS and LVM should be believing they have done this setup themselves, so unmounting, lvchange, etc. should just be fine. It might be necessary to regenerate the initramfs, though. I’ll tell you when I fuck up my hard disk try it. To be continued…

Why are there some people that put "Emailing:" at the start of the subject line of emails that they send out?

I can't figure out why it's useful.

Does anyone know how this got started?

Pete Nuttall writes about the fnreduce function:

fnreduce :: [(a->a)]->a->a
fnreduce [] value = value
fnreduce (a:as) value = freduce as $ a value

A shorter, although not necessarily clearer, expression of this function would be

fnreduce as value = (foldr (flip (.)) id as) value

This just says that to apply a list of functions to a value, we first compose all the functions by folding down the list, then apply the result to the value. The important thing here is (flip (.)), which says to compose backwards; it means that fnreduce [f, g, h] returns (h . g . f . id) instead of (f . g . h . id). An interesting side note is that it doesn't matter (from a semantic point of view) whether I use foldl or foldr, since function composition is associative:

f . (g . (h . id)) = ((id . f) . g) . h

But "flip" here feels a little confusing. A clearer approach would be to reverse the list instead of reversing the operator:

fnreduce as value = foldr (.) id (reverse as) value

This might be less efficient, but it says in a much clearer way what I'm doing.

Yet another option is to eschew the use of function composition:

fnreduce as value = foldl (flip ($)) value as

Here ($) is the application function; (f$x) applies f to x. So if we reverse it (call this $$), we get (x$$f) applies f to x. Folding this from the left down the list [f, g, h] gives us (((value$$f)$$g)$$h), which is exactly what the original fnreduce did.

I think this is maybe the best fold-based solution to the original problem, because it models what's happening (repeatedly transforming a value with a series of functions) directly. To use a right-fold, you have to remove the flip and reverse as (the reason this is necessary is left as an exercise to the reader).

---

Pete also asks whether this can be done for heterogenous lists in a type-safe manner. The core problem is, what is the type of our list? It can't actually be a list: in order to check that it's OK to combine the output of one function with the input of another, we would need the elements of the list to have different types. You can't do this in Haskell, even a crazy ghc-extended Haskell.

So, we could try something like this:

module Test where

data TransformList a b where
    Nil   :: TransformList a a
    Cons  :: (a -> b) -> TransformList b c -> TransformList a c

apply :: TransformList a b -> a -> b
apply Nil         = id
apply (Cons f fs) = (apply fs) . f

This lets us do stuff like

> apply (Cons (\x -> x + 1) (Cons (\x -> show x) Nil)) 5
"6"

This is cute, but I don't know that it's really that useful. After all, how is it any better than the following?

> ((\x -> show x) . (\x -> x + 1)) 5
"6"

Everything the above code lets you do with TransformList could just as well be done with the humble compose operator. The only benefit would be if we could somehow do "list-like" stuff with a transform list. For instance, say we want to write the following function:

traceIntermediates Nil        x = (x, [])
traceIntermediates (Cons f l) x = let x'        = f x
                                      (x'', ss) = traceIntermediates l x'
                                  in
                                    (x'', (show x'):ss)

This won't compile, because we don't know that x' is a member of the Show typeclass. And there is no way to fix this in a reusable way using any technique that relies on building a type parameterized on the input and output types: the problem is that we need to say something about the types of the intermediate values, but their type variables are inaccessible.

In other words, there isn't any way to write down a constraint on the type "TransformList a b" that constrains the intermediates. So I've just wasted a lot of typing duplicating the compose operator. Yay.

On the other hand, it's entirely possible to do this if you don't mind some boilerplate code.

data ShowTransformList a b where
    Nil   :: (Show a) => ShowTransformList a a
    Cons  :: (Show a, Show b, Show c) => (a -> b) -> ShowTransformList b c -> ShowTransformList a c

traceIntermediates :: ShowTransformList a b -> a -> (b, [String])
traceIntermediates Nil        x = (x, [])
traceIntermediates (Cons f l) x = let x'        = f x
                                      (x'', ss) = traceIntermediates l x'
                                  in
                                    (x'', (show x'):ss)

apply :: ShowTransformList a b -> a -> b
apply Nil         = id
apply (Cons f fs) = (apply fs) . f

Now I can do this:

> traceIntermediates
    (Cons (\x -> x + 1)
          (Cons (\x -> x * 2)
                (Cons (\x -> x - 5)
                      (Cons show
                            (Cons (read :: String -> Double)
                                  Nil))))) 5
(7.0,["6","12","7","\"7\"","7.0"])

In a comment on my previous blog post a question was asked as to who benefits when customers are able to use cheap electricity. The answer is that the electricity company benefits the most! When electricity sources such as wind and solar power which can vary in capacity are used the electricity would be cheaper when there is adequate or excess supply. The electricity would be cheaper then to encourage customers to use power hungry devices at those times rather than when the electricity supply is reduced. The larger the capacity of back-up power plants such as gas-fired plants the larger the overall cost of the system (having extra peak-load capacity that is unused most of the time is a waste of money). However the company that I once worked for on a project related to this was not an electricity company. I can’t name the company due to the confidentiality agreement but if the project ever goes into production I’ll blog about it. Recently some of the major investment banks have been focussing on how climate change affects business. I think that with these developments there will be a lot of new investment in environment related technologies. Probably if a few people started work on an embedded Linux box for scheduling power use they could have a good chance of getting some investment. The ideal feature list would include control and monitoring over the Internet, the ability to schedule operations based on power price (received from the grid via a technology similar to X10) and local conditions (how charged the batteries are in your photo-voltaic system). It would control devices via standard X10 modules (which can control the power to the device) as well as directly interfacing with machines that need to be turned on all the time. I would be happy to offer more suggestions via private email to anyone who is interested in implementing this.

Share This

First Draft of the Vim Packaging Policy Available Yay again for DebConf7!! Today the talks I attended were really interesting and in the remaining slices of time I managed to complete ... suspense ... the first draft of the Vim packaging policy \o/. The policy covers packaging of Vim addons and start to tell people about vim-addon-manager. The policy will be shipped in the next upload of the Vim package and is also available on the web. If you're maintaining a package which ships an extension for Vim please have a look, since soon it will be enforced on your package Also an eye from native English speakers will be appreciated, no matter if you speak Vim or not. PS please do not tell bzed about that, he will probably be hating me for "wasting" time on Vim rather then on polishing the first Zenoss package ever

I had a great Father's Day weekend with Maj and Amita June. On Friday afternoon, I spent a couple of hours with Patrick Tanguay to visit potential locations for the incipient Montreal coworking space. We saw a great space near the Provigo on Mont Royal and Clark; there's a definite premium for the location, which has to be weighed against the cost of other places. Then Maj and AJ and I drove up to Mont Tremblant National Park. Mont Tremblant is a beautiful park in the Laurentides a few hours north of Montreal. It's best known for the large ski resort in the town of Mont-Tremblant, but the park area is a real draw on its own. We camped in the Lac Monroe area, in the Secteur du Diable (closest to the resort). We spent most of the weekend on the beach at the lake, swimming and wading. Amita June is definitely hitting her stride as a water baby; she wailed like a banshee any time we left the beach. It was great to stretch our camping legs. Quebec is a great place to live if you like camping. There are lots of good national parks (both Quebec and Canadian) and wildlife reserves. The countryside is a beautiful mix of meadows and eastern temperate forest, and the facilities are of a uniform high quality. Also, the parks aren't particularly crowded, even on holiday weekends. Downsides? The bugs. There are blackflies and deerflies that leave gigantic welts and sometimes draw blood. We spend a lot of time slathering on highly-poisonous insect repellent and still get really bit up. The other thing is the short camping season; many campgrounds don't open for camping until June and close up in September. We woke up to rain on Sunday morning, but it cleared pretty quickly, so we were able to leave in the sunshine. We stopped in Mont-Tremblant on the way home yesterday for a nice Father's Day brunch. Amita June obliging fell asleep in the car, and we had a quiet ride home. Yesterday evening, we went to Piknic for a few hours, and had a great time. There was one of those moon-walk bouncy rooms, which Amita loved. What did I get for Father's Day? Well, the camping trip, of course, plus an IOU for a new BBQ. My old table-top one got a leak in its propane valve during the move, so I'm taking that as a signal to upgrade, and Father's Day is a good time for it. tags: monttremblant fathersday bbq camping travel quebec amitajune

Bad things happen to good people I've been following with interest Biella Coleman's trials and tribulations with the Blue Cross Blue Shield Horizon of New Jersey. Apparently Biella had some dermatological surgery, and the insurance company is not paying up, citing a pre-existing condition. I wonder if they pay out on any dermatological work, since probably all of their customers have skin when they sign up. Biella's extremely resourceful (she doesn't have "healthhacker.org" for nothing) and was able to find a way to put pressure on her insurance company: by filing a complaint with the Department of Insurance. But the most interesting part of the story is that after complaining in her blog, Biella got an email from the Public Relations director of Blue Cross/Blue Shield. Kind of interesting to note which companies are keeping an eye on the blogs, and which aren't. What does it all boil down to? Not much yet, apparently. She's still fighting with the company over whether and how much they'll pay -- although having the PR Director intercede on her behalf has opened some doors. I'm going to continue to follow with interest. tags: biellacoleman healthhackery bluecross blueshield bluedistinction dermatology

BarCampBlackRockCity I just read a Twitter message from Chris Messina about BarCampEarth, and I'm thinking: I'll probably be in Black Rock City at that point. Does a BarCampBlackRockCity make any sense? And then the other side of my head says: no, it does not. tags: blackrockcity barcamp barcampearth

Get these boys some sun…