It seems that in theory cruises can make for quite economical vacations. The cheapest prices tend to be around $100 per person per night for an inside cabin (IE no window) with two people (there is a significant extra fee for having a cabin with only one person). If you book a room in a half-decent hotel with a pool in some moderately desirable place then you will probably pay about $200 per night which might get you free breakfast but won t include lunch or dinner. Cruises include all the free food you can eat for at least 16 hours of the day with maybe an extra fee for getting food in the middle of the night. When a cruise ship stops at the port you can pay for an expensive shore trip arranged by the cruise company or have a cheap trip that involves walking on the beach or taking a taxi to somewhere local. Of course there are lots of extras that can make a cruise really expensive, but if you don t plan to eat at speciality restaurants or drink much alcohol then that shouldn t be a problem. I ve just booked a short luxury cruise, mostly as a trial of the cruise concept. The process of doing so was difficult enough that even if I hadn t previously needed a holiday then I d probably need one now. It would be interesting to compare the amount of time that a reasonable person would be likely to spend choosing a holiday and purchasing it for the different types of holiday, I expect that cruising would be a long way behind everything that involves flying to a foreign city and staying in a hotel. The Issues of Booking When booking a regular holiday you choose where to visit, then choose the time of year, the accommodation, and travel. While the order can vary it s generally a sequential process without too many variables at any stage. With a cruise one of the first decisions is to choose the cruise line, different lines cater to various market segments it s clear what Disney cruises aim for but often not so clear for others. Apart from Disney the other cruise lines don t seem to make clear statements about who they are and aren t targeting. Then there are a variety of ships run by each cruise line which in many cases offer different features, do you want 1 pool or 4? Do you want ice skating? The cruise lines that I have investigated don t offer clear comparisons of their ships. As a cruise ship is essentially a hotel, a collection of restaurants, and some other entertainment you can t just arbitrarily choose one the way you might choose a random hotel with a suitable price and rating. Now while you can choose a hotel and generally get a room when you want a cruise booking has to be made when the ship is in a convenient port. So your holiday needs to be scheduled around the availability of the ship. CruiseCritic.com is an excellent source of information on cruises and has a very active and useful advice forum [1]. The forum appears to be dominated by retirees, presumably because retirees have time to just wait for a ship they like to visit their local port while people who have to schedule holidays around work projects etc. When booking a hotel I have found that Wotif.com is really good for comparing hotels and finding a reasonably cheap one and they also have periodic mail-outs about special deals. I have made a couple of short vacations based on Wotif offering me unusually good prices on a hotel in a place that I was mildly interested in visiting. There are also special deals on cruises, but it s a lot more difficult to take them. Firstly as cruises aren t as interchangeable as hotels it s not as appealing to take an offer of a cruise you didn t previously consider because it s going cheap. The second factor is that according to cruise reviews there is more variation among cabins than there is among hotel rooms. The design of the ship that is needed to cram everything in the available space means that some cabins apparently have noise pollution from various activities on the ship while others are considered to have problems for people prone to sea-sickness. Discounts probably only apply after people who pay non-discount rates have had a chance to book what might be considered the better cabins. Of course it could be a good thing to have most of the ship booked out, CruiseCritic.com has some reviews of individual cabins and presumably there are people who compare hundreds of cabins to discover what is ideal for them. The Problem with Princess I have booked a Princess cruise. I chose it because it was going cheap, but the first two travel agents that advertised it were unable to take my order because Princess only allows US based agencies (which means the discount agents on the net) to sell to US customers I don t think that the people who run the Princess cruise company know what the Internet is about. So I booked the cruise through the Princess web site, they took my money, told me that a good chunk of that money would never be returned if I cancelled, and then sent me to the web page for providing all the information that they need which is a lot. The big problem was when their web site absolutely demanded a passport number and said that I would not be allowed to board without one even though their FAQ (and common sense) indicated that a cruise which doesn t involve any international travel has no need of a passport. Taking someone s money and then telling them that they can t attend due to not having a valid passport is one way of making a future passenger very unhappy. I called the support people (which was Carnival) and had to listen to some really strange hold music it was difficult to determine if the music was produced by someone with unusual tastes in electronic music vastly different from my own preference or whether the computer which manages the phone calls was producing noise instead of music. Eventually I got through to an operator who was very helpful and stored my drivers license number which was adequate. While on the topic of their web site, when booking a cruise with Princess they list which types of cabin (interior, balcony, etc) are available and list the price for the cheapest category of cabin in each price. However if the cheapest category is all sold out then it will still list that cheapest price and thus be advertising a price that can t be booked. This is misleading and annoying. Further Hassle But the difficulty doesn t end here. On a cruise you have to book in advance for which dinner seating you desire which is usually only early or late but some ships offer anytime . Then if you happen to be travelling with people who are in a separate cabin and paying separately (which one would expect to be fairly common) there is no obvious way of synchronising dinner preferences the ship people might assign people to different tables. I admit that I haven t fully explored the post-booking part of the Princess web site, maybe I can arrange a shared table but for the moment my pain threshold for the cruise booking process has been exceeded. Other Cruising Stuff Insight Cruises (formerly Geek Cruises) offers a variety of educational cruises including science, astronomy, chess, art, history, and some other things [2]. From a casual inspection of their web site it seems that the cruise prices are around $300 per night for a 2 person cabin and the registration fees for the conferences are between $1200 and $1500. Most readers of my blog will consider this to be unreasonably expensive as registration for a Linux conference tends to be a lot less than $1000. But when compared to typical for-profit conferences $1500 isn t anywhere near the high end of the range. Also while someone who is choosing a cruise holiday can easily get a cruise that costs less than $200 per day for a 2 person cabin, for commercial conferences it s not uncommon to spend more than $300 per night for two people on accommodation and food. So the only noteworthy part about the cost is that accommodation will be quite expensive if you aren t sharing a room. If you want to run a conference on a cruise there are companies which specialise in such things, Landry and Kling is one company that specialises in planning corporate events [3]. They do seem to aim for the high end of the market, including chartering a small cruise ship or half a large cruise ship that means ~1000 passengers. While there are some conferences with more than 1000 delegates it seems that most conferences top out at about 500 delegates. However I know that some conferences have limited the number of delegates to the maximum capacity of the biggest lecture hall available as some cruise ships have a theater with more than 1000 seats it seems that similar conferences could potentially arrange a half ship charter. A half ship charter apparently allows exclusive access to one of the theaters (at least when it s not needed for evening entertainment) and exclusive seating at the main dining room. It s a standard feature that cruise ships have all the equipment you might need to run a conference. Autism on the Seas is a brand used by the Alumni Cruise company for their holiday packages for special needs children [4]. They sell cruise tickets at the regular list price and the commission that they receive when acting as a travel agency is enough to hire staff to look after the kids while presumably still running at a profit. It seems to me that a similar model could be used for a computer conference, make a profit on cruise ticket sales and then have no extra costs for the conference for people sharing a room that would be cheaper than most Linux conferences I ve attended. For smaller conferences and un-conferences there is no minimum reservation size. One down-side for a computer conference is the cost of Internet access, according to Wikipedia a personal installation of satellite net access costs $5 per megabyte and ship pricing for their Internet access is up around $30-$100 per hour per person! But in the old days when attending a conference meant a week of no net access we survived somehow. It also seems to me that if a company had a small development team it could be productive to put everyone on a cruise ship with a server for testing and version control. They could do 9 hour days on board and still have lots of time for relaxation as it s only a few minutes walk to the pool. Of course this wouldn t work if some members of the team wanted to stay home due to slow and expensive net access.

• [1] http://www.cruisecritic.com/
• [2] http://insightcruises.com/
• [3] http://landrykling.com/
• [4] http://alumnicruises.org/Autism/Autism_Home.htm

See also parts 1, 2, 3, and 4. This is the biggest highlight of our trip to Greece for me. I enjoy having the chance to meet people, visit for awhile, and make new friends and that certainly happened in Greece. In one of the shops we happened to step into, I noticed a radio behind the counter. That s not unusual itself; radios often catch my eye these days. But this radio was tuned to 21.070, an amateur radio frequency. So I asked the shop owner if he happened to be a ham. And indeed he was. He was Lakis (SV5KKU), and Terah and I had a great time visiting with him. Terah took some photos of us, and then we made our purchases and headed out. I had brought my HT (handheld radio, weighs a few ounces and is powered by batteries) with me, and Lakis told me about the repeaters in the area. I had known about some of that since I had emailed Panos of the amatuer radio club in Rhodes before leaving home (I found his name via Google). A couple of days later, on Tuesday, we found ourselves back in Lindos. It was mid-afternoon, so the shops were quiet. After a late lunch, I thought it would be nice to drop in on Lakis one more time, since we were scheduled to fly back the following morning. I m not sure how long we stayed it must have been at least an hour and enjoyed the fresh orange juice he prepared. After we got back to our hotel Tuesday, I learned that our flights on Wednesday were canceled due to a nationwide air traffic controller strike in Greece. After 3 hours on the phone with Delta (more on that experience later), we got rescheduled to fly back Friday. On Wednesday morning, I remembered that the Rhodes amateur radio club meets every Wednesday evening, and now I would be able to go! I knew how to contact Lakis by email and on the radio, and he kindly offered to pick me up and take me there. So that evening, I got a tour of his impressive mountaintop installation, and then it was on to the club the Radio Amateur Association of the Dodecanese (SZ5RDS). There I met Panos, whom I had emailed earlier (and I think surprised him a bit). It was a friendly group, and they translated into English for me every so often so I knew what was being discussed. When I was about to leave, they gave me this: The translation, partly from my memory and partly with the help of Google Translate, is:

Radio Amateur Association of the Dodecanese, SZ5RDS Our friend and radio colleague KR0L JOHN GOERZEN, who visited the island, has our recognition as an HONORARY CLUB MEMBER. PRESIDENT KAVALAKIS PANAGIOTIS SV5AZK (Panos) SECRETARY PAPADIMITRIOU CHRISTOS SV5DDT RHODES Oct. 5, 2011

(I hope that any Greeks reading this will send me corrections.) I truly appreciated that gesture and meeting all the people in the club. On the way back to the hotel, Lakis and I stopped by a restaurant, which I believe had the best souvlaki I ve ever tasted thanks! We brought some back for Terah. She had chosen to stay at the hotel that evening and had a small hotel meal earlier, but enjoyed the souvlaki and pita. Terah had explained to the ma tre d that I wasn t along that evening because I had gone to an amateur radio club meeting. Judging by the surprised reaction, this was probably the first time they had heard that particular comment! Experiences like this make travel fun and worthwhile. Thank you very much, Lakis and everyone in the club I hope to have a chance to visit again.

It s seeming increasingly likely that reports regarding the future of Banshee, Tomboy, and the rest of the Mono stack in the default Ubuntu desktop install are accurate. Ubuntu 12.04 will likely be the first Ubuntu release since 5.10 not to ship with any Mono apps in the default install ending a run of 12 releases over 6 years. The decision seems to have come about during the default apps session at the Ubuntu Developer Summit just ended in Orlando, Florida. Prior to heavy vandalism, the only reasons cited for the change in the UDS session log are Banshee not well maintained and porting music store to GTK3 is blocked on banshee ported to GTK3 . Other reasons mentioned but not in the session logs are complaints that it doesn t work on ARM. I m using a lot of conjecture in this first paragraph because the news about the decision appeared on the blogosphere before anywhere else. The first many Banshee or Tomboy developers read about it was reading a flurry of activity on the Tweetosphere from the anti-Mono activists declaring victory. So first, a word on the cited reasoning. Banshee works fine on ARM, since Mono works fine on ARM. Xamarin, the company behind most upstream Mono work, earns their income almost entirely from ARM versions of Mono, running on the varied ARM implementations found in smartphones. Every major Banshee release is personally tested on my Genesi EfikaMX, an ARM system with a Freescale i.mx51 processor. I ve also demonstrated Banshee running in an Ubuntu chroot on my HP Touchpad, an ARM-based tablet. What is known is that Banshee has some problems running on Texas Instruments OMAP4 processors the target ARM platform for Canonical s ARM work. None of the Banshee upstream developers, Mono upstream developers, or Mono Ubuntu team has ever been able to reproduce the cited problems, since problems specific to an exact ARM chip are impossible to reproduce without the requisite hardware and none of us owns an ARM system matching Canonical s target. That Banshee is still a GTK+2 app is true. A port to GTK+3 is almost complete, but blocking on a single technical bug deep within GTK# s guts, which could be fixed by someone with sufficient knowledge of GTK+ semantics. Nobody with the required GTK+ knowledge has stepped forward with a fix at this point in time. As for the final point, that Banshee is not well maintained, this seems like a directed personal insult against the active and responsive Banshee maintainer, Chow Loong Jin, and upstream bug triager David Nielsen, in addition to the immeasurable hours contributed free of charge for the benefit of Ubuntu users by various other members of related Mono app and library teams, including myself. I need to stress at this point that my annoyance with this decision has nothing to do with the actual app changes. Keeping Tomboy and gbrainy, at a cost of about 25 meg of unsquashed disk space, is a hard argument to make compared with those two plus Banshee for 40 meg. Dropping gbrainy and Tomboy, and switching to Rhythmbox, will save about 30 meg of unsquashed space, all told.I m unconvinced that Rhythmbox is a technically superior app to Banshee several features which were gained by the first app swap will be lost again but that s another long tedious argument to be had. No, what has me deeply angered is the shambolic way the changes were made and announced. Significant accommodations were made by Banshee upstream in order to make life easier for Canonical to integrate Banshee into their OS. For one thing, that s why the Ubuntu One Music Store support is a core Banshee feature, not part of the third-party community extensions package. If Banshee was being considered for replacement due to unresolved technical issues, then perhaps it would have been polite to, I don t know, inform upstream that it was on the cards? Or, if Canonical felt that problems specific to their own itches required scratching, then is it completely beyond the realm of possibility to imagine they might have spent developer resources on bug fixing their OS and sending those fixes upstream? Or even and call me crazy providing access for upstream to specialized hardware such as a $174 Pandaboard to empower upstream to isolate and fix unreproducible bugs specific to Canonical s target hardware? And here s where it gets more astonishing for me Canonical paid money to ship one of the community-based packagers responsible for the stack, Iain Lane, to Orlando for UDS, and didn t think it was worth bothering to perhaps inform him hey, the stuff you work on is in danger of being axed from the default install, maybe you should go to this session . So I m not cross that the stuff I work on has been removed from the default install. I intend to continue working on it as I have for the last 4 years, through my work in Debian. No, why I m cross that I heard about it from fucking Boycott Novell. Regardless of your opinions regarding Banshee or its stack, if you read the above and don t see it as an abysmal failure of community engagement by a company whose community manager wrote a book on the damn topic, then there s something seriously wrong with your understanding of how community labour should be seen as a resource. Maybe someone at Canonical should try reading Jono s book. It s not a first-time offence, and this mail from a PiTiVi developer regarding changes in 11.10 makes for useful further reading. [edit] There is some worthwhile discussion going on on the ubuntu-desktop mailing list covering the technical issues surrounding the decision, I would suggest it s a good place for ongoing technical discussion.

I recently purchased a Seagate FreeAgent GoFlex 1 TB hard drive. It is a 2.5" compact disk that you connect over USB. It draws its power from the USB controller. My previous Extern HDDs were all 2.5" laptop drives, for which I bought a USB enclosure. Those devices always worked perfect, as in:

• They were automatically detected
• Partitions/File Systems understood
• Desktop systems would prompt for actions

The typical kernel messages I got for these devices were:

[56063.268107] usb 1-1: new high speed USB device number 13 using ehci_hcd

[56063.401635] usb 1-1: New USB device found, idVendor=14cd, idProduct=6116

[56063.401645] usb 1-1: New USB device strings: Mfr=1, Product=3, SerialNumber=2

[56063.401652] usb 1-1: Product: USB 2.0 SATA BRIDGE

[56063.401658] usb 1-1: Manufacturer: Super Top

[56063.401663] usb 1-1: SerialNumber: M6116018VF16

[56063.402857] scsi8 : usb-storage 1-1:1.0

[56064.400896] scsi 8:0:0:0: Direct-Access WDC WD50 00BEVT-24A0RT0 PQ: 0 ANSI: 0

[56064.401576] sd 8:0:0:0: Attached scsi generic sg2 type 0

[56064.402102] sd 8:0:0:0: [sdb] 976773168 512-byte logical blocks: (500 GB/465 GiB)

[56064.402615] sd 8:0:0:0: [sdb] Write Protect is off

[56064.402618] sd 8:0:0:0: [sdb] Mode Sense: 03 00 00 00

[56064.403101] sd 8:0:0:0: [sdb] No Caching mode page present

[56064.403105] sd 8:0:0:0: [sdb] Assuming drive cache: write through

[56064.404861] sd 8:0:0:0: [sdb] No Caching mode page present

[56064.404864] sd 8:0:0:0: [sdb] Assuming drive cache: write through

[56064.419657] sdb: sdb1

[56064.421850] sd 8:0:0:0: [sdb] No Caching mode page present

[56064.421854] sd 8:0:0:0: [sdb] Assuming drive cache: write through

[56064.421857] sd 8:0:0:0: [sdb] Attached SCSI disk

But for my Seagate FreeAgent GoFlex HDD, things were different. The kernel would detect but then it would not mount. The error reported is that the device is busy. My FreeAgent's kernel messages look similar to what the regular one has:

[ 168.520140] usb 1-1: new high speed USB device number 5 using ehci_hcd

[ 168.657424] usb 1-1: New USB device found, idVendor=0bc2, idProduct=5021

[ 168.657433] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3

[ 168.657439] usb 1-1: Product: FreeAgent GoFlex

[ 168.657444] usb 1-1: Manufacturer: Seagate

[ 168.657449] usb 1-1: SerialNumber: NA0C1BML

[ 168.659079] scsi5 : usb-storage 1-1:1.0

[ 169.657136] scsi 5:0:0:0: Direct-Access Seagate FreeAgent GoFlex 0148 PQ: 0 ANSI: 4

[ 169.708786] sd 5:0:0:0: Attached scsi generic sg3 type 0

[ 169.709079] sd 5:0:0:0: [sdc] 1953525167 512-byte logical blocks: (1.00 TB/931 GiB)

[ 169.709954] sd 5:0:0:0: [sdc] Write Protect is off

[ 169.709963] sd 5:0:0:0: [sdc] Mode Sense: 1c 00 00 00

[ 169.710567] sd 5:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

[ 169.759942] sdc: sdc1

[ 169.762050] sd 5:0:0:0: [sdc] Attached SCSI disk

As the Device Mapper Multipath Debian Maintainer, I have multipath-tools installed on my laptop. Turns out, for some reason, the device is consumer by the device mapper multipath stack.

20:12:58 rrs@champaran:~$ ls /dev/mapper/

1Seagate@ 1Seagate FreeAgent GoFlex N 1Seagate FreeAgent GoFlex Np1 control LocalDisk-ROOT@ LocalDisk-SWAP@

Also the block device ID listing is enumerated as type SCSI.

19:56:44 rrs@champaran:~$ ls /dev/disk/by-id/

ata-HITACHI_HTS723216L9SA60_091220FC1220NCJASEVG@ scsi-SATA_HITACHI_HTS7232091220FC1220NCJASEVG@

ata-HITACHI_HTS723216L9SA60_091220FC1220NCJASEVG-part1@ scsi-SATA_HITACHI_HTS7232091220FC1220NCJASEVG-part1@

ata-HITACHI_HTS723216L9SA60_091220FC1220NCJASEVG-part2@ scsi-SATA_HITACHI_HTS7232091220FC1220NCJASEVG-part2@

ata-HITACHI_HTS723216L9SA60_091220FC1220NCJASEVG-part3@ scsi-SATA_HITACHI_HTS7232091220FC1220NCJASEVG-part3@

ata-HL-DT-ST_DVDRAM_GU10N_M189CNI1127@ scsi-SSeagate_FreeAgent_GoFle_NA0C1BML@

ata-ST1000LM010-9YH146_W1000ZD8@ usb-WDC_WD12_00BEVE-11UYT0_ST_Killer-0:0@

dm-name-1Seagate@ usb-WDC_WD12_00BEVE-11UYT0_ST_Killer-0:0-part1@

dm-name-LocalDisk-ROOT@ wwn-0x5000c5003d19c8c2@

dm-name-LocalDisk-SWAP@ wwn-0x5000cca586e112bc@

dm-uuid-LVM-buywwzKkpfKG2RegankA2nPkmFBBPFe3D5DepV8w8nLrHfoAjIIQVnakOQJZEqJX@ wwn-0x5000cca586e112bc-part1@

dm-uuid-LVM-buywwzKkpfKG2RegankA2nPkmFBBPFe3k9aJfc9B7wRmVIwfoagffHUZjuN6c4cM@ wwn-0x5000cca586e112bc-part2@

dm-uuid-mpath-1Seagate@ wwn-0x5000cca586e112bc-part3@

raid-1Seagate@

But I'm not sure why the same doesn't happend for my external laptop drive. It gets properly tagged as device type USB. So Laptop or Server, if you have a FreeAgent that you want to connect to your machine, and see the device busy error when accessing the device['s] directly, do the following:First, the ID of the device.

20:17:43 rrs@champaran:~$ /lib/udev/scsi_id --whitelisted --page=0x83 --device=/dev/sdc

1Seagate FreeAgent GoFlex N

Add the ID to /etc/multipath.conf under blacklist section

blacklist

wwid "1Seagate FreeAgent GoFlex*"

wwid "1Seagate FA GoFlex Desk*"

# devnode "^(ram raw loop fd md dm- sr scd st)[0-9]*"

# devnode "^hd[a-z][[0-9]*]"

# device

# vendor DEC.*

# product MSA[15]00

#

Run mulitpath -F to flush the unused maps. Run multipath -v3 to ensure that now the device is blacklisted.

Nov 03 20:19:02 sdc: (Seagate:FreeAgent GoFlex) wwid blacklisted

The front cause for the mis-behavior is:

# Coalesce multipath devices before multipathd is running (initramfs, early

# boot)

ACTION=="add change", SUBSYSTEM=="block", RUN+="/sbin/multipath -v0 /dev/$name"

But something else, maybe the default blacklist table, needs the actual fix.

It's about time that Rapha l is interviewed in the "People behind Debian" series he initiated on his blog. Indeed, when he interviews people, Rapha l asks about other people they could suggest for next interviews. So, during mine, I suggested him to be a next "victim". As he couldn't interview himself, I volunteered for this. As you'll see below, Rapha l (who's a friend of mine as I'm a friend of his) likes to speak and that shows in the length of his answers :-) but you always know more about Debian when reading his blog posts, books, mails, etc. I personnally think that he is among the best promoters of the project for years and it was a pleasure for me to conduct this interview. My questions are in bold, the rest is by Rapha l. Who are you? Hi, Rapha l Hertzog, I'm a 32 years old French Debian developer who is married and who has a 2-year old son. I'm running my own company (Freexian) since 2005, I started it 3 years after the end of my computer science studies. I'm also a very proud author of the Cahier de l'Admin Debian, a French book about Debian. You often wrote about your attempts to make your living partly, if not completely, out of your Debian work. Can you describe the way you're trying to do this? My first try has been with Freexian. I always advertised this company as being specialized in Debian GNU/Linux. While Freexian is successful enough to provide me a decent income, I'm not really satisfied with the result because very few of my contracts are about improving Debian. I use Debian daily for the benefit of my customers, writing new customer specific (embedded) software, deploying a service on Debian servers, etc. But except for the occasional bugfix, all this work does not improve Debian (the only exception has been the dpkg multiarch implementation work sponsored by Linaro). The positive side is that I don't need to fill my entire schedule to earn enough money to live. So I'm regularly taking some days off work to be able to contribute to Debian. This is a freedom that I enjoy... My French book has also been a bestseller and depending on the years the royalties represented between 1 and 2 months of supplementary time that I can spend on Debian (that is between 2000 and 4000 EUR of income). Now since last year, I decided to actively work towards my goal of making a living out of my Debian work. I want to build on what has been most successful for me up to now, that is my book. My strategy has been to build an audience around my blog: with a direct contact with my readers I have the opportunity to sell e-books, and without any intermediary taking the biggest part of the price, I don't need a very large audience to be successful. I have also been experiencing micro-donations with Flattr, people who are enjoying my articles on my blog can use it to give a few cents for each article they find useful. With a large enough userbase, this could fund free documentation and would avoid the need for commercial e-books but we're not there currently and I don't know if it will ever reach the critical mass. Last but not least, I'm soliciting donations for my Debian work on the sidebar of my blog, and I have the chance to a have a few (regular) donators. You're a proud father since last year. How do you manage your commitment to the project with your family life? There are few things that I put above Debian in my life, but my family certainly is. I try to handle most of my Debian duties during work hours so that I can spend time with my family on evenings and during week-ends but in truth I never really disconnect from Debian. It happens quite often that I say to my wife I'll come in a few minutes, let me finish this and then I end up responding to a Debian mail, or an IRC query and take 30 minutes instead of the 5 expected ones. I try hard to avoid this but it's difficult. Luckily for me, my wife is very supportive of my Debian involvement and knows me well... By the way my wife is using Debian on her computer, and my son has already played with DoudouLinux (a Debian derivative!). Have you already been accused of self-promotion in your writings? If that would ever happen, what would you answer to that? Yes, more than once. I am proud of what I do for Debian, I enjoy sharing the result of my work. Because of this, some people believe that I'm selfish and egocentric. And this has somewhat increased since I have been soliciting donations: for me it's important to be transparent towards donators so that they see what I really do for Debian. But some people have the feeling that I'm getting undeserved attention and that I bring everything towards my own person. On the other side, as an author, I'm a public figure who is definitely seeking some attention... I don't have any miraculous answer, we are a large and diverse community, it's next to impossible to please everybody. I listen to all the concerns that people bring forward, I take them into account as much as possible, in particular when I believe they are reasonable/well justified, or when they come from people that I highly respect. But sometimes I have to plainly ignore them too... in particular when they are trying to impose their own political view on a topic that's not directly related to the only value that we all share: the social contract. Contributing to Debian is a challenge, we all have to make efforts to put aside our differences and to concentrate on the work that brings us closer to the best free software operating system ever built. You recently launched a campaign to free out the soon-to-be-published "Debian Administrator Handbook", an English version of your well-known book about Debian in French. Can you tell us more about this project? My French book has been very successful at helping people to get started with Debian, and like I already explained, it was also effective to fund a part of my Debian work. So I wanted to make it available world-wide by publishing an English translation of it. I tried to find an English-speaking editor willing to take on the challenge but I found none interested. Not put off by a setback, Roland (my co-author) and I decided to negotiate with our French editor Eyrolles to recuperate the necessary rights to translate the book into English. Handling everything ourselves represents a lot of work, but it also means that we have the freedom required to decide of the license of the resulting book. We would love to see it under a license compatible with the Debian Free Software Guidelines. But at the same time we firmly believe that we deserve a reasonable monetary compensation for the work on the book, so we conditioned its liberation to a predefined amount of money (25 K ) in what we call the "liberation fund". And since we wanted to be sure that we would have the required means to complete the translation, we used a crowfunding platform to seek support of people interested by the book. With such a platform you're only debited if the minimal requested amount is reached. Anyone can participate, pre-order the book and/or put some money in the liberation fund. As of today, we already reached the minimal funding goal (15 K ) so the book translation will happen. But the liberation target has not yet been reached so we don't know yet if the book will be free from the start... you can follow the progress right on the fundraising page or on the website dedicated to the book. PS: If you want to contribute to this project and also make a donation to Debian at the same time, you should check out this page. You're one of the main developers of dpkg, a critical tool for Debian systems. Can you tell us more about the current development challenges it is facing? What will be the new dpkg features for wheezy? The current challenges are not really technical. dpkg is a relatively mature piece of software and it will continue to work for the foreseeable future without needing much maintenance work. The real challenge is trying to setup a healthy developement community around it so that we can keep tackling new interesting problems (there are many listed in the roadmap and in the 225 wishlist bugs). There is a real problem of leadership and communication in the current team. We used to be three, and we're only two nowadays. Guillem is the legitimate leader since he's involved in dpkg's developement since early 2006 while I joined only in late 2007. But in the last 4 years, we did not manage to recruit anyone else on the team. Some persons tried to contribute significant new features (like Sean Finney with a rework of the way we handle configuration files) but they gave up frustrated after a while because we did not manage to review their work (and discuss the design) in any reasonable timeframe. Another famous case is Ian Jackson with his trigger work. His work got merged, but so late that in the mean time he blew up while trying to hijack the maintenance of dpkg. For a long time I was concentrating my work on the Perl part of dpkg (aka dpkg-dev mainly), so I did not feel qualified to review and merge work related to the C part and I was just a worried observator of this situation. I tried to improve it by setting up some basic review infrastructure, it should have brought some lisibility to the status of each change left to be reviewed... but it has not been used and it changed nothing. Over the years, I became much more interested in the C part. My first big contribution in C has been the rewrite of update-alternatives (from Perl to C). I made other small changes in between, but at the start of this year I had this great opportunity to work on the multiarch implementation (FYI, multiarch is the possibility to mix packages from several architectures on a single system). This really forced me to jump into the C codebase and learn a lot about how dpkg is implemented. Thanks to this I have been able to tackle other small projects (like the improved triggers). This would be all great if my multiarch work was already merged, but it's not. It's a large work, I do not mind waiting a bit in particular since Guillem is a highly skilled C programmer. His sharp analysis of new designs are invaluable, when he reviews code he always finds something to improve. I learnt a lot just by reviewing the code he wrote over the years. That said I have been waiting since April without almost no updates from him. With the release team asking us to hurry up, the situation is getting somewhat strained as I really want to see multiarch in Wheezy and I do not really want to short-circuit Guillem. Hum, I may have drifted a bit from your original question... what great new features can people expect? Well multiarch is supposed to be the big new feature, apart from that there aren't many things that matter to the end users. But there are already quite a few changes that are of interest to package maintainers (like hardened build flags, source package improvements, improved triggers, ). What's the biggest problem of Debian? Manicheism and a tendency to quickly polarize the discussions. In reality, there are very few situations where everything is all good or all bad. Ever since I have read The 7 Habits of Highly Effective People I try hard to put into practice the habits of interdependence . Instead of having only my point of view in mind, I try to understand the motivations from the other party ( Seek First to Understand, Then to be Understood ) in order to be able to put forward solutions acceptable to both parties ( Think Win-Win ). I highly recommend this book to anyone. And I invite everybody to at least try to follow those simple advices. Is there someone in Debian you admire for their contributions? There are many and I can't give an exhaustive list... here are some that I would like to highlight (in no particular order):

• Niels Thykier for the work he puts into Lintian, and his recent work on a britney test-suite (the software that creates testing out of unstable);
• Joachim Breitner for his britney reimplementation using a SAT solver;
• Adam D. Barrat for his work on britney2;
• All the release team members who take care of an endless stream of transitions;
• Christoph Berg for his continuous work over the years on various QA infrastructure (DDPO, Mole, );
• Enrico Zini for the way he helped reform the New Member process without any big disruption;
• Russ Allbery for his Debian Policy work and for being a model in terms of how we should interact on Debian mailing lists;
• Stefano Zacchiroli for being the most active leader that I have ever seen, and for the way he tries to steer important discussions to a constructive conclusion.
• etc.

Most of those people are working on improving Debian's infrastructure so that we can all be more effective and do an even better work. This kind of work is not always very visible but it's crucial to Debian's future. Thank you to Rapha l for the time spent answering my questions. I hope you enjoyed reading his answers as I did. And, anyway, it was fun to just play the game "the other way".

I have been a long-time user of the Galeon web browser, which, while powerful for its time, is getting a bit long in the teeth and has been abandoned for a long time. As a result I need to find something new. As Galeon uses the Mozilla engine I figure switching to Iceweasel/Firefox will be the least amount of pain since they share similar formats for lots of the user configuration and data (with the exception of history). Switching to Firefox also gives me access to a lot of configurability and a vast sea of extensions all written in RDF, JavaScript and zoooool (ahem, XUL). Another plus was that I was already using Firebug for the occasional web development project. Looking at the Mozilla addons site is like entering someone's shed. There will be the few beautiful unfinished projects still being worked on, one polished finished scuplture gathering dust but still admirable, things with power plugs from a bygone era, some things that have a coin slot on them, some cryptic machines with no visible screws or manuals, some spiders and their cobwebs and a few rats and mice chewing through things. A place where you can find some excellent, well documented, useful and Free Software extensions alongside lots of crud. Luckily for me the good stuff that I wanted to use was already in Debian or the friendly Debian Mozilla extension maintainers team was willing to package them for me. In my quest for freedom from Galeon I first noticed that there is no up button in Iceweasel. Bummer, I use that a lot so I went searching for extensions. I soon found one extension, but it hadn't kept up with the ever-changing Mozilla APIs so it fell by the wayside. Thanks to the leavers of breadcrumbs I picked up the trail to a new shiny and working extension. Lo-and-behold I found Uppity, which was all about going up and as it turns out, much better at that than Galeon. Thanks to MozExt team, thats solved, next! The next glaring problem was the lack of Galeon-style smart bookmarks. Before you ask, yes, Firefox smart keyword bookmarks are not the same thing. This was rediculously hard to search for due to the wording used by both projects being same same but different. Some folks switched to Epiphany to get the extra search boxes on their toolbars. Like this guy I was not interested in that, mainly due to the addons I would be missing out on. I tried a few different tacks, even searching for a way to have multiple search boxes in the Firefox toolbars. I soon gave up on finding an extension that would do this like Galeon does so I figured its time to roll up the sleeves and learn some zzzoooool. I already know a little bit about JavaScript and CSS so... First slap a dash of a tutorial about adding toolbar buttons, add a slither of adding extensions without installing them, stare down some Mozilla reference manuals, thrown in a pinch of favicons, give up on a wild goose chase or two, add a big fat blob of zoooool and sautee in fugly hacks. Soon enough you will have something hardcoded that works like Galeon smart bookmarks but looks even better. I may eventually turn this into a proper and functionally equivalent extension for Galeon-stlye smart bookmarks but for now it will remain a useful hack. If you want to get your hands dirty with zzzoooooool and try this out after modifying it to use your personal search URLs, please feel free to contact me. For now the only remaining issue I can see is that the forward/back buttons in Iceweasel don't have the explict menu buttons. This is a minor issue for me so now it is time for me to figure out how to migrate my data and config¹ before permanently switching away from Galeon. Wish me luck! 1. Of course the data and config are fugly, but that is a something for another, much broader and more complicated hack

Wow, a lot (of travel) has happened in the last three months. Zoe has been to Chicago, Langkawi, Brisbane, and is now back in Mountain View. I'm actually feeling really guilty about our carbon footprint for this year. She spent the last about 3.5 weeks in Brisbane with Sarah after the wedding in Langkawi, until I flew there last weekend for my sister's wedding. Losing and finding Smudge helped pass the time, as did having Liam stay with me while he was visiting from Zurich. I could have done without the former, but really appreciated the timing of the latter. There was quite a noticeable vocabulary difference when I finally saw Zoe again. I can't keep up with all the words she can say now, and she's starting to grasp concepts as well. For example, she opened a small cardboard box, and declared it "empty". Apparently we say "uh oh" a lot, as this is one of her favourite things to blurt out whenever anything happens, ranging from my sister's bridal train dragging on the ground at her wedding, to vomiting all over herself in the back of the car. She's definitely not going to be a girly girl. When the flight from Brisbane landed in Los Angeles she yelled out a big "WHOA!" and then "Again!" while furiously doing the baby sign language for "more". Fortunately we were able to oblige for the flight from LAX to San Jose, and she again enjoyed the landing (she got to sit in her own seat for this one) and did a loud "Wow!" and "Again!" I'm looking forward to taking her on her first roller coaster ride. Her canine teeth still haven't come through yet, but I think they've just started to make an appearance. We're still trying to de-jetlag her at the moment, so it's unclear if the wake-ups throughout the night are because of jetlag or teeth issues (or a combination of both). She handled all of the travel fairly well, although she's now too long for a aeroplane bassinet. She didn't get a lot of sleep on the 12.5 hour flight from Brisbane to Los Angeles (which meant we got none), but she was pretty well behaved for the bulk of it. She's good enough on her legs now that she can just tear up and down the aisles without incident, so we made countless laps of the plane to keep her occupied. The return flight was on Halloween, and the cabin crew had gotten into the spirit with temporary face tattoos, some cabin decorations and some "costumes". The trip back home was actually epic in that it was littered with minor annoyances, but I'll save that for a separate blog post. This Christmas should be a fun one, as she'll be old enough to be a bit more interactive about the whole thing. She may also freak out sitting on Santa's knee, so we may not get quite as nice a photo as we got last year.

Some days ago, a hacker group, THC, released a denial of service tool for SSL web servers. As stated in its description, the problem is not really new: a complete SSL handshake implies costly cryptographic computations. There are two different aspects in the presented attack:

• The computation cost of an handshake is more important on the server side than on the client side. The advisory explains that a server will require 15 times the processing power of a client. This means a single average workstation could challenge a multi-core high-end server.
• The use of SSL renegotiation allows to trigger hundreds of handshakes in the same TCP connection. Even a client behind a DSL connection can therefore bomb a server with a lot of renegotiation requests.

Mitigation techniques There is no definitive solution to this attack but there exists some workarounds. Since the DoS tool from THC relies heavily on renegotiation, the most obvious one is to disable this mechanism on the server side but we will explore other possibilities.

Disabling SSL renegotiation Tackling the second problem seems easy: just disable SSL renegotiation. It is hardly needed: a server can trigger a renegotiation to ask a client to present a certificate but a client usually does not have any reason to trigger one. Because of a past vulnerability in SSL renegotiation, recent version of Apache and nginx just forbid it, even when the non-vulnerable version is available. openssl s_client can be used to test if SSL renegotiation is really disabled. Sending R on an empty line trigger renegotiation. Here is an example where renegotiation is disabled (despite being advertised as supported):

$ openssl s_client -connect www.luffy.cx:443 -tls1
[...]
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
[...]
R
RENEGOTIATING
140675659794088:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:591:

Disabling renegotation is not trivial with OpenSSL. As an example, I have pushed a patch to disable renegotiation in stud, the scalable TLS unwrapping daemon.

Rate limiting SSL handshakes Disabling SSL renegotiation on the client side is not always possible. For example, your web server may be too old to propose such an option. Since those renegotiations should not happen often, a workaround is to limit them. When the flaw was first advertised, F5 Networks provided a way to configure such a limitation with an iRule on their load-balancers. We can do something similar with just Netfilter. We can spot most TCP packets triggering such a renegotiation by looking for encrypted TLS handshake record. They may happen in a regular handshake but in this case, they usually are not at the beginning of the TCP payload. There is no field saying if a TLS record is encrypted or not (TLS is stateful for this purpose). Therefore, we have to use some heuristics. If the handshake type is unknown, we assume that this is an encrypted record. Moreover, renegotiation requests are usually encapsulated in a TCP packet flagged with push .

# Access to TCP payload (if not fragmented)
payload="0 >> 22 & 0x3C @ 12 >> 26 & 0x3C @"
iptables -A LIMIT_RENEGOCIATION \
    -p tcp --dport 443 \
    --tcp-flags SYN,FIN,RST,PSH PSH \
    -m u32 \
    --u32 "$payload 0 >> 8 = 0x160300:0x160303 && $payload 2 & 0xFF = 3:10,17:19,21:255" \
    -m hashlimit \
    --hashlimit-above 5/minute --hashlimit-burst 3 \
    --hashlimit-mode srcip --hashlimit-name ssl-reneg \
    -j DROP

The use of u32 match is a bit difficult to read. The manual page gives some insightful examples. $payload allows to seek for the TCP payload. It only works if there is no fragmentation. Then, we check if we have a handshake (0x16) and if we recognise TLS version (0x0300, 0x0301, 0x0302 or 0x0303). At least, we check if the handshake type is not a known value. There is a risk of false positives but since we use hashlimit, we should be safe. This is not a bullet proof solution: TCP fragmentation would allow an attacker to evade detection. Another equivalent solution would be to use CONNMARK to record the fact the initial handshake has been done and forbid any subsequent handshakes. If you happen to disable SSL renegociation, you can still use some Netfilter rule to limit the number of SSL handshakes by limiting the number of TCP connections from one IP:

iptables -A LIMIT_SSL \
    -p tcp --dport 443 \
    --syn -m state --state NEW \
    -m hashlimit \
    --hashlimit-above 120/minute --hashlimit-burst 20 \
    --hashlimit-mode srcip --hashlimit-name ssl-conn \
    -j DROP

Your servers will still be vulnerable to a large botnet but if there is only a handful of source IP, this rule will work just fine¹. I have made all those solutions available in a single file.

Increasing server-side power processing SSL can easily be scaled up and out. Since SSL performance increases linearly with the number of cores, scaling up can be done by throwing in more CPU or more cores per CPU. Adding expensive SSL accelerators would also do the trick. Scaling out is also relatively easy but you should care about SSL session resume.

Putting more work on the client side In their presentation of the denial of service tool, THC explains:

Establishing a secure SSL connection requires 15 more processing power on the server than on the client.

I don t know where this figure comes from. To check it, I built a small tool to measure CPU time of a client and a server doing 1000 handshakes with various parameters (cipher suites and key sizes). The results are summarized on the following plot: For example, with 2048bit RSA certificates and a cipher suite like AES256-SHA, the server needs 6 times more CPU power than the client. However, if we use DHE-RSA-AES256-SHA instead, the server needs 34% less CPU power. The most efficient cipher suite from the server point of view seems to be something like DHE-DSS-AES256-SHA where the server needs half the power of the client. However, you can t really uses those shiny cipher suites:

1. Some browsers do not support them: they are limited to RSA cipher suites².
2. Using them will increase your regular load a lot. Your servers may collapse with just legitimate traffic.
3. They are expensive for some mobile clients: they need more memory, more processing power and will drain battery faster.

Let s dig a bit more on why the server needs more computational power in the case of RSA. Here is a SSL handshake when using a cipher suite like AES256-SHA: When sending the Client Key Exchange message, the client will encrypt TLS version and 46 random bytes with the public key of the certificate sent by the server in its Certificate message. The server will have to decrypt this message with her private key. Those are the two most expensive operations in the handshake. Encryption and decryption are done with RSA (because of the selected cipher suite). To understand why decryption is more expensive than encryption, let me explain how RSA works. First, the server needs a public and a private key. Here are the main steps to generate them:

1. Pick two random distinct prime numbers p and q , each roughly the same size.
2. Compute n=pq . It is the modulus.
3. Compute \varphi(n)=(p-1)(q-1) .
4. Choose an integer e such that 1<e<\varphi(n) and \gcd(\varphi(n),e) = 1 (i.e. e and \varphi(n) are coprime). It is the public exponent.
5. Compute d=e^ -1 \mod\varphi(n) . It is the private key exponent.

The public key is (n,e) while the private key is (n,d) . A message to be encrypted is first turned into an integer m<n (with some appropriate padding). It is then encrypted to a ciphered message c with the public key and should only be decrypted with the private key:

• c=m^e\mod n (encryption)
• m=c^d\mod n (decryption)

So, why is decryption more expensive? In fact, the key pair is not really generated like I said above. Usually, e is a small fixed prime number with a lot of 0, like 17 (0x11) or 65537 (0x10001) and p and q are choosen such that \varphi(n) is coprime with e . This allows encryption to be fast using exponentiation by squaring. On the other hand, its inverse d is a big number with no special property and therefore, exponentiation is more costly and slow. Instead of computing d from e , it is possible to choose d and compute e . We could choose d to be small and coprime with \varphi(n) and then compute e=d^ -1 \mod\varphi(n) and get blazingly fast decryption. Unfortunately, there are two problems with this:

• Most SSL implementation expects e to be a 32bit integer.
• If d is too small (less than one-quarter as many bits as the modulus n ), it can be computed efficiently from the public key.

Therefore, we cannot use a small private exponent. The best we can do is to choose the public exponent to be e =4294967291 (the biggest prime 32bit number and it contains only one 0). However, there is no change as you can see on our comparative plot. To summarize, no real solution here. You need to allow RSA cipher suites and there is no way to improve the computational ratio between the server and the client with such a cipher suite.

Things get worse Shortly after the release the denial of service tool, Eric Rescorla³ published a good analysis on the impact of such a tool. He asks himself about the efficiency to use renegotiation for such an attack:

What you should be asking at this point is whether a computational DoS attack based on renegotiation is any better for the attacker than a computational DoS attack based on multiple connections. The way we measure this is by the ratio of the work the attacker has to do to the work that the server has to do. I ve never seen any actual measurements here (and the THC guys don t present any), but some back of the envelope calculations suggest that the difference is small. If I want to mount the old, multiple connection attack, I need to incur the following costs:

1. Do the TCP handshake (3 packets)
2. Send the SSL/TLS ClientHello (1 packet). This can be a canned message.
3. Send the SSL/TLS ClientKeyExchange, ChangeCipherSpec, Finished messages (1 packet). These can also be canned.

Note that I don t need to parse any SSL/TLS messages from the server, and I don t need to do any cryptography. I m just going to send the server junk anyway, so I can (for instance) send the same bogus ClientKeyExchange and Finished every time. The server can t find out that they are bogus until it s done the expensive part. So, roughly speaking, this attack consists of sending a bunch of canned packets in order to force the server to do one RSA decryption.

I have written a quick proof of concept of such a tool. To avoid any abuse, it will only work if the server supports NULL-MD5 cipher suite. No sane server in the wild will support such a cipher. You need to configure your web server to support it before using this tool. While Eric explains that there is no need to parse any SSL/TLS messages, I have found that if the key exchange message is sent before the server send the answer, the connection will be aborted. Therefore, I quickly parse the server s answer to check if I can continue. Eric also says a bogus key exchange message can be sent since the server will have to decrypt it before discovering it is bogus. I have choosen to build a valid key exchange message during the first handshake (using the certificate presented by the server) and replay it on subsequent handshakes because I think the server may dismiss the message before the computation is complete (for example, if the size does not match the size of the certificate). With such a tool and 2048bit RSA certificate, a server is using 100 times more processing power than the client. Unfortunately, this means that most solutions, except rate limiting, exposed on this page may just be ineffective.

---

1. However, since this rule relies on source IP to identify the attacker, the risk of false positive is real. You can slow down legitimate proxies, networks NATed behind a single IP, mobile users sharing an IP address or people behind a CGN.
2. Cipher suites supported by all browsers are RC4-MD5, RC4-SHA and 3DES-SHA. Support for DHE-DSS-AES256-SHA requires TLS 1.2 (not supported by any browser).
3. Eric is one of the author of several RFC related to TLS. He knows his stuff.

A few years ago I got up on stage and briefly talked about how the Linux community contained far too many people who were willing to engage in entirely inappropriate behaviour, how this discouraged people from getting involved and how we weren't very good at standing up against that sort of behaviour. Despite doing this in front of several hundred people, and despite the video of me doing so then being uploaded to the internet, this got me a sum total of:

• No death threats
• No discussion about any of my physical attributes or lack thereof
• No stalkers
• No accusations that I was selling out the Linux community
• No accusations that I was a traitor to my gender
• No real negative feedback at all[1]

Which is, really, what you'd expect, right? The internet seems intent on telling me otherwise:

Well, she didn't do herself any favors by talking at conferences about women in tech, or setting up a feminist movement. If you wanted to attract abuse, that's a good way to go about it. It should be expected.
(Source)

MikeeUSA is a troll. He has no means to actually harm anyone, and he does it purely for the lulz.

Thus, MikeeUSA trolled a woman, and she took the bait. I just don't get why this is news, I've been trolled before, I don't get a news story.
(Source)

I was going to start a rant about how this behavior is encouraged by the macho men online, but this was just one guy harassing her. "Due to harassment" reads as due to harassment from the community, but she gave in to one idiot. She let him win.
(Source)

The full comment thread has rather more examples. If you stand up and say anything controversial, you should expect abuse. And if you let that abuse change your behaviour in any way, you've let the trolls win.

These attitudes are problematic.

The immediate assumption underlying such advice is that the degree of abuse is related to what you've said, not who you are. I'm reasonably visible in the geek world. I've said a few controversial things. The worst thing that's happened to me has been Ryan Farmer deciding to subscribe me to several thousand mailing lists. Inconvenient, but not really threatening. I haven't, for instance, been sent death threats. Nobody has threatened to rape me. And even if they had done, I wouldn't need to worry too much - there's a rather stronger track record of violent antifeminism being targeted at women than men.

I don't have to worry about this kind of thing. That means I don't get to tell other people that they should have expected it. Nor do I get to tell them that they should ignore it, or that if they don't call the police then they have no grounds to complain. And nor does anyone else.

The trolls don't win because someone decides that getting out of the tech business is more appealing than continuing to face abuse. The trolls win because we consider their behaviour acceptable and then blame the victim for inviting them in the first place. That needs to change.

[1] It was justifiably pointed out that saying all this while standing on stage next to a mostly naked guy wearing a loincloth with a raccoon tail covering his penis may have weakened my message somewhat.

comments

As is natural for an LTS cycle, lots of people are thinking and talking about work focused on quality rather than features. With Canonical extending LTS support to five years on the desktop for 12.04, much of this is quite rightly focused on the desktop. I'm really not a desktop hacker in any way, shape, or form, though. I spent my first few years in Ubuntu working mainly on the installer - I still do, although I do some other things now too - and I used to say only half-jokingly that my job was done once X started. Of course there are plenty of bugs I can fix, but I wanted to see if I could do something with a bit more structure, so I got to thinking about projects we could work on at the foundations level that would make a big difference. Image build pipeline One difficulty we have is that quite a few of our bugs - especially installer bugs, although this goes for some other things too - are only really caught when people are doing coordinated image testing just before a milestone release. Now, it takes a while to do all the builds and then it takes a while to test them. The excellent work of the QA team has meant that testing is much quicker now than it used to be, and a certain amount of smoke-testing is automated (particularly for server images). On the other hand, the build phase has only got longer as we've added more flavours and architectures, particularly as some parts of the process are still serialised per architecture or subarchitecture so ARM builds in particular take a very long time indeed. Exact timings are a bit difficult to get for various reasons, but I think the minimum time between a developer uploading a fix and us having a full set of candidate images on all architectures including that fix is currently somewhere north of eight hours, and that's with people cutting corners and pulling strings which is a suboptimal thing to have to do around release time. This obviously makes us reluctant to respin for anything short of showstopper bugs. If we could get things down to something closer to two hours, respins would be a much less horrible proposition and so we might be able to fix a few bugs that are serious but not showstoppers, not to mention that the release team would feel less burned out. We discussed this problem at the release sprint, and came up with a laundry list of improvements; I've scheduled this for discussion at UDS in case we can think of any more. Please come along if you're interested! One thing in particular that I'm working on is refactoring Germinate, a tool which dates right back to our first meeting before Ubuntu was even called Ubuntu and whose job is to expand dependencies starting from our lists of "seed" packages; we use this, among other things, to generate Task fields in the archive and to decide which packages to copy into our images. This was acceptably quick in 2004, but now that we run it forty times (eight flavours multiplied by five architectures) at the end of every publisher run it's actually become rather a serious performance problem: cron.germinate takes about ten minutes, which is over a third of the typical publisher runtime. It parses Packages files eight times as often as it needs to, Sources files forty times as often as it needs to, and recalculates the dependency tree of the base system five times as often as it needs to. I am confident that we can significantly reduce the runtime here, and I think there's some hope that we might be able to move the publisher back to a 30-minute cycle, which would increase the velocity of Ubuntu development in general. Maintaining the development release Our release cycle always starts with syncing and merging packages from Debian unstable (or testing in the case of LTS cycles). The vast majority of packages in Ubuntu arrive this way, and generally speaking if we didn't do this we would fall behind in ways that would be difficult to recover from later. However, this does mean that we get a "big bang" of changes at the start of the cycle, and it takes a while for the archive to be usable again. Furthermore, even once we've taken care of this, we have a long-established rhythm where the first part of the cycle is mainly about feature development and the second part of the cycle is mainly about stabilisation. As a result, we've got used to the archive being fairly broken for the first few months, and we even tell people that they shouldn't expect things to work reliably until somewhere approaching beta. This makes some kind of sense from the inside. But how are you supposed to do feature development that relies on other things in the development release? In the first few years of Ubuntu, this question didn't matter very much. Nearly all the people doing serious feature development were themselves serious Ubuntu developers; they were capable of fixing problems in the development release as they went along, and while it got in their way a little bit it wasn't all that big a deal. Now, though, we have people focusing on things like Unity development, and we shouldn't assume that just because somebody is (say) an OpenGL expert or a window management expert that they should be able to recover from arbitrary failures in development release upgrades. One of the best things we could do to help the 12.04 desktop be more stable is to have the entire system be less unstable as we go along, so that developers further up the stack don't have to be distracted by things wobbling underneath them. Plus, it's just good software engineering to keep the basics working as you go along: it should always build, it should always install, it should always upgrade. Ubuntu is too big to do something like having everyone stop any time the build breaks, the way you might do in a smaller project, but we shouldn't let things slide for months either. I've been talking to Rick Spencer and the other Ubuntu engineering leads at Canonical about this. Canonical has a system of "rotations", where you can go off to another team for a while if you're in need of a change or want to branch out a bit; so I proposed that we allow our engineers to spend a month or two at a time on what I'm calling the +1 Maintenance Team, whose job is simply to keep the development release buildable, installable, and upgradeable at all times. Rick has been very receptive to this, and we're going to be running this as a trial throughout the 12.04 cycle, with probably about three people at a time. As well as being professional archive gardeners, these people will also work on developing infrastructure to help us keep better track of what we need to do. For instance, we could deploy better tools from Debian QA to help us track uninstallable packages, or we could enhance some of our many existing reports to have bug links and/or comment facilities, or we could spruce up the weather report; there are lots of things we could do to make our own lives easier. By 12.04, I would like, in no particular order:

• Precise to have been more or less continuously usable from Alpha 1 onward for people with reasonable general technical ability
• Canonical engineering teams outside Ubuntu (DX, Ubuntu One, Launchpad, etc.) to be comfortable with running the development release on at least one system from Alpha 2 onward
• Installability problems in daily image builds to be dealt with within one working day, or preferably before they even make it to daily builds
• The archive to be close to consistent as we start milestone preparation, rather than the release team having to scramble to make it so
• A very significant reduction in our long-term backlog of automatically-detected problems

Of course, this overlaps to a certain extent with the kinds of things that the MOTU team have been doing for years, not to mention with what all developers should be doing to keep their own houses in reasonable order, and I'd like us to work together on this; we're trying to provide some extra hands here to make Ubuntu better for everyone, not take over! I would love this to be an opportunity to re-energise MOTU and bring some new people on board. I've registered a couple of blueprints (priorities, infrastructure) for discussion at UDS. These are deliberately open-ended skeleton sessions, and I'll try to make sure they're scheduled fairly early in the week, so that we have time for break-out sessions later on. If you're interested, please come along and give your feedback!

I'm not the first one who thought about managing their dotfiles & scripts in a (D)VCS in order to be able to have them in sync across different machines, & I even haven't re-invented the wheel for doing so. short report on my experiences:

after looking around a bit I decided to try movein (info page with links to downloads & a README, blog post, git repo) because it looked liked it would easily do what I'd like to do without getting in my way too much; which means e.g.: no symlink farm, & only one simple command to manage the whole setup.

movein is a simple shell script that allows to setup & manage multiple modular repos tracking different sets of files quite easily, taking advantage of the powers of git & mr. & being written in shell means I also was able to make a few changes, which stew kindly accepted & applied to his tree already.

in fact I'm just starting to use it for real, but it looks good so far, & I think I'm going to stick with it for the time being.

2 minor observations:

• git in squeeze vs. wheezy/sid behave differently: the former happily & silently overwrites existing files, the latter dies loudly on each existing file. not sure which behaviour I actually prefer.
• running mr up on a couple of repos from a machine without an ssh key on the target machine is kind of boring ...

</promotion_time>

If you have been following Lintian s development closely, you will probably have noticed that I have not really done anything there for the past week. Instead I have turned my focus on our testing migration script, britney2. First, I have created a minimal test suite[1]. It started as 4 simple tests and by now it contains about 30 tests. The size of each test is rather small; the largest tests are about 1600 binary packages in total[2], but most are 2-20 binary packages in total. Thus the test suite is rather fast compared to a live data sample , which easily takes more than 10 minutes for a single run. Unfortunately, hand-crafting the test data is somewhat annoying and easy to get wrong. The test suite has a somewhat unfair focus on auto-hint [3] cases, so the current britney2 fails up to 14 tests. Some of these appears to fail because the auto-hinter (for some reason) receives incomplete information about the situation. To my knowledge we not been able to debug the situation, but Adam has a refactor branch that does not seem to have this issue. Personally I am hoping it will soon be merged into the master branch, especially because it seems to simplify a lot of common operations. Joachim Breitner (who has been working on a SAT-solver based britney) also contributed a couple of test cases[4]. Allegedly, SAT-britney does rather well on the test suite, failing only 2 tests as far as I can tell[5]. On the other hand, it does solve a some of the more interesting cases britney2 does not solve. On a more mathematical note, the britney2 implementation behaves like a function[6] with an attractive fixed point[7]. This is interesting, because for some cases it may take britney2 a couple of iterations to reach the right solution. This fixed point is somewhat simple to find by using the following steps (pseudo-code):

// Runtime complexity O(n * br * diff), where "n" is the number of iterations until
// a fixed point is reached, "br" is the complexity of "run_britney" and "diff" is
// the runtime of the "last != current" comparison.
function find_fixed_point(initial);
    last = run_britney(initial)
    current = run_britney(last)
    while last != current ; do
        last = current
        current = run_britney(last)
    od
    return current
end

This gives us a simple way to test if britney will eventually solve the issue herself (and when she will do it). Currently britney2 is automatically run twice a day, so for every 2 iterations (beyond the first) roughly translates to a 24-hours delay. So far the test suite does not have a lot of problems that requires more than one iteration. Personally I would be pleased if it turned out to stay that way as the test suite coverage grows. If you are interested in playing around with this, you can get sources from:

• britney2
  • Currently only works in stable (i.e. requires python2.5 and python-apt < 0.8 or so)
    • There some patches for this filed against release.debian.org.
  • See the INSTALL file for instructions
  • Adam s branch
    • use the p-u branch.
• SAT-britney
  • I haven t tested this one and I do not know the requirements here
• britney-tests
  • See the README file for instructions

Footnotes: [1] http://lists.debian.org/debian-release/2011/10/msg00178.html [2] These tests are auto-generated, so it is merely an up-scaled pattern . [3] Basically if two (or more) packages needs to migrate into testing at the exact same time, they need to be hinted in. [4] Not to mention all the copy-waste errors he pointed out in mine. Apparently, SAT-britney has stricter requirements to the data than britney2. [5] I assume the test called sat-britney-death (created by Joachim) was named that way for a reason. The second failure is caused by SAT britney not reading hints (yet?), so the approve tpu package test case should fail. [6] A function that maps an archive into another archive erh, I mean, it maps a set of packages into another set of packages [7] http://en.wikipedia.org/wiki/Fixed_point_%28mathematics%29 Assuming my claim to be true, the function will have more than one fixed point. The obtained fixed point depends on the initial state of testing. As an example:
y depends on x
x in testing has RC bugs If x is not in testing, it cannot migrate to testing (due to its RC bugs). If x is not in testing, then y cannot migrate into testing. But if x starts in testing, then y may be able to migrate. This can happen if x migrated to testing before an RC bug was filed against it. (Dis-)Proving my claim is an exercise left for the reader.

We all love stats, don t we? So, here we go! Let s start with a graph: It shows the number of packages in the NEW queue since last year. You can see a big drop during April 2011, and a reasonably low rate during the last six months. You could think fellow Debian Developers stopped to upload NEW packages. Sorry, you re wrong! Since Squeeze release, 3.832 .changes files with NEW components were processed by dak, with an average of 14,85 NEW packages per day. On the FTP Team side, we had 3.732 accepts (14,47 per day), 339 rejects (1,31 per day) and 178 comments to maintainers (0,69 per day).

---

Who were the most prolific maintainers who got a NEW processing? Here is our special top ten:

1. Debian Haskell Group (362 packages)
2. Debian Perl Group (343 packages)
3. Debian Java Maintainers (161 packages)
4. Debian Ruby Extras Maintainers (124 packages)
5. Debian Multimedia Maintainers (100 packages)
6. Debian Fonts Task Force (96 packages)
7. Debian Med Packaging Team (79 packages)
8. Debian Install System Team (61 packages)
9. Debian Javascript Maintainers (54 packages)
10. Debian Python Modules Team (50 packages)

That s bad packaging teams cannot bake cookies!
Let s do the same with Changed By, this time:

1. Ben Hutchings (159 packages)
2. Joachim Breitner (138 packages)
3. Clint Adams (134 packages)
4. Jonas Smedegaard (124 packages)
5. TANIGUCHI Takaki (97 packages)
6. Nicholas Bamber (61 packages)
7. Alessio Treglia (60 packages)
8. maximilian attems (54 packages)
9. David Paleino (51 packages)
10. Torsten Werner (45 packages)

Much better now go and heat up your ovens, we know who you are

---

Another nice aspect to look at is the speed of NEW processing. Some maintainers were very happy for a fast NEW processing, someone even complained for having been too quick! So, let s find out which upload was the quickest ever. Try to gamble a bit before reading the answer, to see whether you are near to the real value Alessio Treglia, you probably already know, because your gwc_0.21.16~dfsg-1 upload has been processed in 41 seconds (yes, forty-one seconds!). Here s an excerpt from ftp-master log to certify it:

20110516120252 process-upload dak Processing changes file gwc_0.21.16~dfsg-1_amd64.changes
20110516120258 process-upload dak Moving to new gwc_0.21.16~dfsg-1_amd64.changes
20110516120339 process-new tolimar NEW ACCEPT: gwc_0.21.16~dfsg-1_amd64.changes

Alex was the super-fast FTP Team member behind the quickest accept, do you want to beat him? Join FTP Team

Since the 3.5.0 release, Openbox automatically starts applications with .desktop files in /etc/xdg/autostart. This was already the case with previous releases when setting the . $GLOBALAUTOSTART instruction in the autostart script. This instruction is now deprecated. This has puzzled me as some applications were started twice in my Openbox session: those launched via my ~/.config/openbox/autostart script which are also launched through xdg autostart. I also like to use notify-osd when in openbox, but it seems the new gnome-fallback session requires notification-daemon. In my Openbox session, I have noticed that notification-daemon was launched whereas my autostart script specifies to use notify-osd. I then discovered that /etc/xdg/autostart/notification-daemon.desktop was the cause of this. I had to drop a notification-daemon.desktop file in my ~/.config/autostart/ directory with the following contents: The last two lines allow me to not start notification-daemon only when using Openbox ie. it still starts automatically when launching a GNOME session. You can check what applications are automatically started through xdg with: The results are quite amazing, most applications being only configured to start in GNOME or KDE, but nothing seems to be very coordinated amongst maintainers. I find this quite misleading, and would prefer openbox to stick with the previous default behaviour. Who said diehard ?!

Rapha l Hertzog recently announced a new dpkg-buildflags interface in dpkg that at long last gives the distribution, the package maintainers, and users the control they want over the build flags used when building packages. The announcement mail gives all the gory details about how to invoke dpkg-buildflags in your build to be compliant; but the nice thing is, if you're using dh(1) with debian/compat=9, debhelper does it for you automatically so long as you're using a build system that it knows how to pass compiler flags to. So for the first time, /usr/share/doc/debhelper/examples/rules.tiny can now be used as-is to provide a policy-compliant package by default (setting -g -O2 or -g -O0 for your build regardless of how debian/rules is invoked). Of course, none of my packages actually work that way; among other things I have a habit of liberally sprinkling DEB_MAINT_CFLAGS_APPEND := -Wall in my rules, and sometimes DEB_LDFLAGS_MAINT_APPEND := -Wl,-z,defs and DEB_CFLAGS_MAINT_APPEND := $(shell getconf LFS_CFLAGS) as well. And my upstreams' build systems rarely work 100% out of the box with dhauto* without one override or another somewhere. So in practice, the shortest debian/rules file in any of my packages seems to be 13 lines currently. But that's 13 lines of almost 100% signal, unlike the bad old days of cut'n'pasted dh_* command lists. The biggest benefit, though, isn't in making it shorter to write a rules file with the old, standard build options. The biggest benefit is that dpkg-buildflags now also outputs build-hardening compiler and linker flags by default on Debian. Specifically, using the new interface lets you pick up all of these hardening flags for free:

-fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security -Wl,-z,relro

It also lets you get -fPIE and -Wl,-z,now by adding this one line to your debian/rules (assuming you're using dh(1) and compat 9):

export DEB_BUILD_MAINT_OPTIONS := hardening=+pie,+bindnow

Converting all my packages to use dh(1) has always been a long-term goal, but some packages are easier to convert than others. This was the tipping point for me, though. Even though debhelper compat level 9 isn't yet frozen, meaning there might still be other behavior changes to it that will make more work for me between now and release, over the past couple of weekends I've been systematically converting all my packages to use it with dh. In particular, pam and samba have been rebuilt to use the default hardening flags, and openldap uses these flags plus PIE support. (Samba already builds with PIE by default courtesy of upstream.) You can't really make samba and openldap out on the graph, but they're there (with their rules files reduced by 50% or more). I cannot overstate the significance of proactive hardening. There have been a number of vulnerabilities over the past few years that have been thwarted on Ubuntu because Ubuntu is using -fstack-protector by default. Debian has a great security team that responds quickly to these issues as soon as they're revealed, but we don't always get to find out about them before they're already being exploited in the wild. In this respect, Debian has lagged behind other distros. With dpkg-buildflags, we now have the tools to correct this. It's just a matter of getting packages to use the new interfaces. If you're a maintainer of a security sensitive package (such as a network-facing daemon or a setuid application), please enable dpkg-buildflags in your package for wheezy! (Preferably with PIE as well.) And if you don't maintain security sensitive packages, you can still help out with the hardening release goal.

Terah and I went to the Greek island Rhodes recently. This is the second in a series about it. I am one to enjoy history. There is something deeply, well, connecting, about standing in an old place. There is a timeless quality to it a feeling of being connected to so many people of the past, and yet still being connected to change, visible in things such as weathering of stones. To gaze at pottery that s 300 years old, walk past 700-year-old walls, or pass through what remains of the grand portico of an ancient temple to Athena stirs a feeling I can barely explain, of timelessness. Although Rhodes doesn t have the famous Greek sites such as the Parthenon or Delphi, I can t help but wonder why the Rhodes sites aren t better known. They were incredible and it is hard to condense all that we saw into a short blog post. I have to start with the medieval Rhodes Old Town. We got off the bus a few blocks from it one bright morning, and our first task was to find a gate across the moat. Oh yes, A GATE ACROSS THE MOAT. It s a dry moat, and that bridge off in the distance is the gate we were headed to. Outside of the outer wall is a nice quiet walking area. The moat and walls completely surround Old Town and, for the most part, date back about 500 years. The round stones you see on that picture, we were told, were likely surplus from catapults and other projectile weapons. Cross one line of walls and you come to another, with original canons still present. The Knights Hospitaller of St. John, which held Rhodes for a few centuries until the Ottomans captured it, sure knew how to build to impress. The gate we happened to use was Amboise, the Grand Master s Gate. Right there is the stunningly rebuilt landmark Palace of the Grand Master. It is absolutely impossible for any photograph to begin to do this building justice. Between its imported Greek and Roman floors, to the grand nature of everything in it, and the archaeological museum in one corner, it was a fitting start to a visit to Old Town. Here s one of the main staircases. Just near the Palace is quiet courtyard with an old door. Pass through that door and suddenly you re in the midst of the busy Old Town. And among the landmarks in Old Town, the most prominent is Ippoton, the Avenue of the Knights. Along this avenue are the buildings built by the various nationalities of knights, many of which are historical sites in their own. Taken together, it is quite clear why Rhodes is said to be one of the world s best-preserved medieval cities. Down at the other end of Ippoton is the Knights Hospital, which is now part of the archaeological museum. Step off the Avenue a few blocks and you get to some quieter narrow streets just as old, in many cases. On Sunday morning, we were able to visit Mount Filerimos. In contrast to the busy Rhodes, Filerimos had an air of quiet and still to it. It was the site of a monastery, two historic churches, and a landmark Italian cross on the mountaintop. We arrived, and begin our visit with a walk up the quiet stone path. When we got to the top, we walked past this peaceful church. As we walked past the outside, we heard the beautiful music of chant from indoors. We got to step in and listen to mass for a few minutes. In typical fashion, directly in front of the church are two much older sites: one, the ruins of a temple to Athena, and the other a 4th-century Christian bapistery. Rhodes is a popular tourist destination, and of course we saw plenty of popular sites (such as the grandmaster s palace). Filerimos had a few tourists too, but not as many. I frequently like to operate on the plan of going wherever all the tourists aren t. And so, on Filerimos, that meant seeing what was behind the monastery. It started with this peaceful tree-lined path. And the deserted, but intentionally open, gate led to the remains of a Byzantine fortress, which had been a staging area for both the Knights and the Ottomans before their campaigns to capture Rhodes. It also provided incredible views of the surrounding countryside. The first historic site we had visited on our trip was the Acropolis of Lindos, parts of which are 2300 years old. Here s a view of the mountain from the rooftop of the Kalypso, our favorite restaurant in Lindos. The columns of the temple to Athena Lindia are visible, and of course so are the walls. The road up to the acropolis is accessible only on foot or by donkey. It is apparently the only road that has ever been used to get to the acropolis. Here is the partially-restored grand portico to the temple. There s an old Christian church (4th century, if memory serves) at the Acropolis too. The Acropolis makes some pretty good use of natural defenses too. Here s a view from one level of it. There s a manmade wall up there at the very top. And, of course, the beautiful Aegean always in the background. There are lots of cats on Rhodes. Here is a kitten napping at the top of the Lindos Acropolis: Lindos itself is a beautiful town. Here s one of the quieter streets: Notice the pebble steps leading into the houses those intricate pieces of artwork are all over. This post won t be complete without the story of our visit to the Acropolis of Rhodes. We walked there from Old Town. At the Acropolis, there are the remains of a temple to Apollo, an ancient theater, and an ancient stadium where qualifying matches for the Olympics were held. As we got closer to the area, we were repeatedly passed by people dressed in uniforms of various types. And as we got there, we joined a stream of people entering the area. The ancient stadium had apparently thousands of people in it, country names were being read off over the loudspeakers, policemen wielding machine guns were standing by, and we had absolutely no idea what was going on. At this point, you can appreciate the difference between Terah and me. Terah thought that we have no idea what is happening, she was tired from the walk, and so thought we should just leave. I thought that we have no idea what is happening, which is a great reason to stay. So Terah opted to sit and read a bit under some trees while I explored. Here s a view of the stadium as it was emptying out, seen from the theater: I explored the temple and theater, and eventually we were ready to head back. We knew there was a bus back to the New Market (from where we could get a bus back to our hotel), but didn t know where the bus stop was. The obvious place to ask were the policemen, which I thought I would do. Terah thought she would just stay sitting under the trees, on the grounds that the policemen nearest us were all carrying machine guns and perhaps wouldn t like to be disturbed. This led to my cryptic tweet:

Only ONE of us is the kind of person that goes up to guys with machine guns to ask what s happening. Me to Terah today

They told me that it was the preparations for the opening ceremony for a global shooting contest, and also gave me directions to the bus stop.

Last weekend Dennis M. Ritchie, creator of the C Programming Language, the UNIX Operating System, The Plan 9 Operating System and many other key contributions to computing, passed away due an extended illness. I learned about it yesterday trough Rob Pike, former dmr s colleague and friend. I m writing this post to honor him and his legacy. While I ve seen people mourn his departure, I ve also noticed most technical people don t really understand why his work is important for today s computing, so I m here to give a refresh. Since I m good with visual tools (mindmaps) I ve made one to describe the extension of dmr s contribution to the world. You can get a glimpse of how this is important. The main topics are their key contributions, the sub-topics are the current technologies that were built upon dmr s work. Click on the image to see a larger resolution version.

The C Programming Language

I m in Paris for the LibreOffice conference, but today after a random talk with a guy that had a Mozilla badge/pin, I found myself in a tour to the Mozilla Europe office (which is also in Paris). Thank you random guy David for the tour (: The office itself looks very nice, and reminds me of the work space of many software companies. But the unique part of the tour was David pointing to people and telling us what they are working on. It s interesting to see the people behind the sciences, although we didn t talk with them in order not to disturb the work. I remember most when we David pointed to a guy and said he s the one doing composer. Maybe because it s very clear what he doesn t, while some people work on more internal parts or more conceptual concepts like performance . After seeing some cool posters on the wall, I got a chance for a short chat with Tristan Nitot and ask him some questions about Mozilla Europe and the office. I m still trying to understand how Mozilla works (as an organization), and this was a good chance to get familiar with the European branch. For me, the highlight of the tour was the chance to say to the people at the office thanks for the software just before leaving and saying goodbye. Not in everyday you get a chance to see the faces behind the software you use daily.
Filed under: Mozilla

This morning, when turning on TV, my ears have been yet another time hurt by the stupidity of a self-appointed economist, trying to intoxicate people with his fantasies about economy being able to regulate itself. Seriously. All neoliberals, please shut the fuck up. Now. The world has been running out of your ideas for more than 30 years. We lowered salaries and taxes, making the world run through credit. We gave everything to those who were already born with everything. Economy was almost brought to the verge of implosion thanks to your crazy ideas. For 4 years since the subprimes crisis, you psychopaths have kept on explaining it happened because we have not listened to you enough, and that the solution is to lower salaries even more. A quick look at the situation in Greece and UK should be enough to understand where this is going. It is time for something new. So shut up now, and let others fix the mess you have left behind.

A month ago, I published an article on the compared performances of stunnel, nginx and stud as SSL terminators. The conclusion was to use stud on a 64-bit system, with session caching and AES. stunnel was unable to scale properly and nginx exhibited important latency issues. I got constructive comments on many aspects. Therefore, here is the second round. The protagonists are the same but both the conditions and the conclusions have changed a bit.

Benchmarks The hardware platform remains unchanged:

• HP DL 380 G7
• 6 GB RAM
• 2 L5630 @ 2.13GHz (8 cores), hyperthreading disabled
• 2 Intel 82576 NIC
• Spirent Avalanche 2900 appliance

We focus on the following environment:

• Ubuntu Lucid, featuring a patched version of OpenSSL 0.9.8k with AES-NI builtin and enabled by default
• 2.6.39 kernel (250 HZ)
• 64-bit
• 1 core for HAProxy, 2 cores for network cards¹, 4 cores for SSL, 1 core for the system
• system limit of 100,000 file descriptors per process

On the SSL side, the main components are:

• 2048-bit certificates
• AES128-SHA cipher suite
• 8 HTTP/1.0 requests per client
• HTTP response payload of 1024 bytes

If you are interested in different conditions (32-bit, number of cores, 1024-bit certificates, proportion of session reuses), have a look at the previous article whose results on the matter are still valid. I only consider 2048-bit certificates because most certificates issued today are 2048-bit. In SP800-57, NIST advises that 1024-bit RSA keys will no longer be viable after 2010 and advises moving to 2048-bit RSA keys:

If information is initially signed in 2009 and needs to remain secure for a maximum of ten years (i.e., from 2009 to 2019), a 1024-bit RSA key would not provide sufficient protection between 2011 and 2019 and, therefore, it is not recommended that 1024-bit RSA be used in this case.

And the protagonists are:

• stud, @e207dbc5a3, with session cache enabled for 20,000 sessions, listen backlog of 1000 connections,
• stunnel 4.44, configured with --disable-libwrap --with-threads=pthread (configuration),
• nginx 1.1.4, configured with --with-http_ssl_module (configuration).

UPDATED: Configuration of nginx has been updated to switch to off both multi_accept and accept_mutex thanks to the feedback of Maxim Dounin on latency issues that were present in the previous article (and still present in the early version of this article). HAProxy (configuration) is used for stunnel and stud to add load-balancing. For each test, we want to maximize the number of HTTP transactions per second (TPS) while minimizing latency and unsucessful transactions. The maximum number of allowed failed transactions is set at 1 transaction out of 1000. The maximum allowed average latency is 100 ms².

Control case Below is the first comparison between stud, nginx and stunnel on plain Ubuntu Lucid. Like in previous benchs, stunnel is not able to scale, with a maximum of 400 TPS. However, early latency issues of nginx has been solved, thanks to the change in configuration. stud and nginx gets similar performance but, as you can see on the bottom plot (whose scale is logarithmic) stud starts with an initial latency of 40 ms. This is pretty annoying. I have tried some micro optimization to solve this issue but without luck. If latency matters for you, stick with stunnel or nginx (or find where the problem lies in stud). UPDATED: After reporting this issue, a fix has been commited. The culprit was Nagle s algorithm which improves bandwidth at the expense of latency. With the fix, performances are mostly the same but there is no latency penalty any more.

OpenSSL 1.0.0e When I published the first round of benchmarks, Micha Trojnara was dubious about the bad results of stunnel. He told me on Twitter:

I solved it! You compiled stunnel against a 4 years old OpenSSL with MT-unsafe SSL_accept. Upgrade OpenSSL and rebuild stunnel. This bug was fixed in OpenSSL 1.0.0b. I implemented a (slow) workaround, so stunnel doesn t crash with earlier versions of OpenSSL.

Therefore, I backported Ubuntu Oneiric OpenSSL 1.0.0e to Ubuntu Lucid³ and, as you can see yourself on the plot below, there is a huge performance boost for stunnel: about 400%! stud features a slight improvement too (about 8%) but still starts with a latency of 40 ms while nginx does not improve.

Intel Accelerator Engine Most Linux distributions are still using OpenSSL 0.9.8. For example, Debian Squeeze ships with OpenSSL 0.9.8o, Ubuntu Lucid with 0.9.8k and RHEL 5.7 with 0.9.8e. Therefore, they are not eligible to recent performance improvements. To solve this, Intel released Intel Accelerator Engine, which also brings, among other things, AES-NI support:

This is essentially a collection of assembler modules from development OpenSSL branch, packed together as a standalone engine. Standalone means that it can be compiled outside OpenSSL source tree without patching the latter. Idea is to provide a way to utilize new code in already released OpenSSL versions constrained by support and release policies limiting their evolvement to genuine bug fixes.

Alas, in my tests, there was no improvement compared to OpenSSL 1.0.0e (already patched with AES-NI support). stunnel comes with out-of-the-box support for additional engines. Just add something like this to your stunnel.conf:

engine=dynamic
engineCtrl=SO_PATH:/WOO/intel-accel-1.4/libintel-accel.so
engineCtrl=ID:intel-accel
engineCtrl=LIST_ADD:1
engineCtrl=LOAD
engineCtrl=INIT

For stud, you should grab support for engine selection from GitHub and for nginx, just specify the engine with ssl_engine.

Intel Performance Primitives In his great article Accelerated SSL , Simon Kuhn shows huge performance improvements by using Intel s IPP library, a library aimed at leveraging power of multi-core processors and including some cryptographic primitives and a patch to make use of them in OpenSSL. Unfortunately, this library is not free, not even as in free beer . The patch provided by Intel only applies on OpenSSL 0.9.8j. Simon updated it for OpenSSL 1.0.0d. It also applies with a slight modification to OpenSSL 1.0.0e. Look at Simon s article for instructions on how to compile OpenSSL with IPP. Simon measured 220% improvement on RSA 1024-bit private key operations by using this library. Unfortunately, we don t get such a performance burst with stud which exhibits a 15% improvement (but remember that private key operations only happen once every 8 transactions thanks to session reuse). Worst, with stunnel, performances are reduced by one third and nginx just segfaults. I think such a library is risky to use in production.

Hyperthreading During the previous tests, only half the capacity of each core were used. For stud, adding more workers per core only slightly improves this situation (less than 10% with 4 workers per core). Therefore, I thought using hyperthreading could rectify the situation here. In this case, stud, stunnel and nginx are allocated 8 virtual cores instead of 4. stud only got a small improvement of 5%, nginx a moderate improvement of 11% while stunnel got a significant improvement of 26%.

Conclusion Here is a quick recap of the maximum TPS reached on each tests (while keeping average response time below 100 ms). All tests were run on 4 physical cores with 2048-bit certificates, AES128-SHA cipher suite. Each client makes 8 HTTP requests and reuse the first SSL session.

Context	nginx 1.1.4	stunnel 4.44	stud @e207dbc5a3
OpenSSL 0.9.8k	1489 TPS	400 TPS	1492 TPS
OpenSSL 1.0.0e	1501 TPS	1561 TPS	1609 TPS
OpenSSL 1.0.0e + IPP	-	1101 TPS	1857 TPS
OpenSSL 1.0.0e + HT	1718 TPS	1973 TPS	1694 TPS

I also would like to stress that below 1500 TPS, stunnel adds an overhead of a few milliseconds while stud adds 40 milliseconds. With OpenSSL 1.0.0e and hyperthreading enabled, stunnel has both good performance and low latency on low loads to moderate loads. You may just want to limit the number of allowed connections to avoid collapsing on high loads. While in the previous round, stud was the champion and stunnel lagged behind, the situation is now reversed : stunnel is on top! UPDATED: As said earlier, the latency issue with stud has been solved. Performances remain mostly the same.

---

1. HAProxy and network cards don t need dedicated core for this kind of load. However, if you also want to handle non SSL traffic, those dedicated cores can get busy.
2. The real condition is more complex. Each point is considered valid if the corresponding average latency is less than 100 ms or if the average on the next 8 points is less than 100 ms. Temporary latency bursts are therefore allowed. We are not so forgiveful with failed transactions.
3. This is pretty straighforward: I just dropped multiarch support. I chose to use Ubuntu version of OpenSSL instead of upstream because there are some interesting patches, like automatic support of AES-NI.