A status update for what changed since Squeeze release and what is the state of Cupt in Wheezy. No new things for those few who follow small blog posts or the changelog, but an overview for, maybe, softly or newly interested.

Cupt, the high-level package manager for Debian with a console interface and therefore APT's competitor, continued its development since the first stable series. In the second major version it's rewritten in C++(11), got many new features such as numerous improvements in the depedency resolver and the dpkg action scheduler, the support of index diffs, the tutorial, wget-based download method, position action override options, logging, colored action preview prompt, treeish detailed error messages if no solutions were found, to name most important. That said, if you want multiarch, CDROM repository support or, say, some exotic download method, -- Cupt won't suit, at least for now. Project-wide support is also still almost completely missing as many developers accept no more than two package managers.

The "bug-freeness" aim still holds, at the moment of writing there is no unfixed runtime bugs of priority normal+. Here I thank again those people who reported bugs, you all definitely made Cupt better. I am sure there are bugs still which wait their time to show up, but that's hardly avoidable.

All in all, 2.x is a big step forward from 1.x. Not a last step -- development continues.

[ once again, I am translating somebody else's material In this case, my good Costa Rican friend Carolina Flores. Please excuse my stylistic mistakes My English is far from native as you well know. But this material is worth sharing, and worth investing some tens of minutes doing a quick translation. If you can read Spanish, go read Caro's original entry ] Have you been to a music record store lately? I did so last Saturday, as a mere excercise. I was not planning on buying anything but I wanted to monitor things and confirm my suspicions. What was I suspicious of? First, that I would only find old records. And so it was: The only recent record I found was ...little broken hearts by Norah Jones. the second, that I would only find music for over 50 year old people. so it was: Were I there to look for a present for my father, I would have walked out with 10 good records. Third, that in the store nothing worth commenting would happen. About that last point, I should point out it was around 10 AM and the store had just opened its doors. Lets concede the benefit of doubt. I don't think many of you will remember, but in Barrio La California (where there is now a beauty parlour, almost in front of AM.PM) there was Auco Disco. In Auco Disco there was a guy specialized in rock (Mauricio Alice) and another one specialized in jazz (I don't remember his name). In that record you could always find rare records, but if they were not there, at least you were sure to find somebody to say: "No, we don't have that, but that's an excellent record, it's the best that [insert group here] have ever recorded because just afterwards they switched their guitar player, they had gone a bit south but with that record they are flying. But no, we don't have it; I can recommend you this record by [insert another group] because it has a guitar solo in track six that is amazing". It would happen more or less like that, which means, one would arrive to Auco Disco at 10 AM and leave around 5 PM with three new records, after having listened to a spectacular music selection. What happened to those stores? Were they killed by The Pirate Bay? That's the simplistic answer from the recording industry! The answer is that those stores never got anything from the industry but an invoice. The industry specially in prescindible markets such as ours was limited to hiring artists, taking care of them recording a sellable product, producing the object called record, and that's it. The more commercial radio stations were paid to program those songs as it cannot be casual that "Mosa, mosa" is the summer hit in all of Latin America, can it? but, record stores? Nothing. Lets carry on with that idea: Radio stations are paid to program said music. This idea should not lead us to believe that recording companies are to blame for bad taste. I won't reveal my sources, but I know the success of the "Locura autom tica" song by La Secta was a real example. Nobody paid for it. That song got to the number one because of its own merits(?) (you don't know the effort it took to find that thing, I cannot recommend it to you). Same thing happens with other stations that don't program reggaet n, that try to save the species, and where they play what we do like. But the thing is, everything we like is not available in any record store in this country. Then, even if we wanted to buy a record or give it as a gift to somebody, it is plain impossible. And don't tell me it's the same to present as a gift a link or a CD full of downloaded MP3 as it is to give a record with cover and booklet, wrapped in gift paper. I might be old-school, but the fetish object record still exists, not only because of its cover, but because of its sound. A 3MB MP3 is akin to drinking coffee dripping from a bag that has been used eight times with the same coffee beans. That format is the worst that has ever happened to music, and if we had any bit dignity we would never purchase digital files in Amazon or iTunes safe for MP3 with an acceptable compression level. That, if we could buy them, because not only that is allowed to us. As the musical industry has no interest in resolving ITS problem (that is not our problem, it is those companies') it has not even been resolved how to charge for a MP3 download that includes import fees (well, if downloading from here a MP3 from a USA-based file server can be considered importing goods into Costa Rica!!!) so we don't have to get dizzy entering into the nineties to Titi Online to discover there is nothing by Muse, Andrew Bird, The Killers, Death Cab for Cutie, Paramore, Bj rk... (believe me, I looked them all up, even Norah jones and La Secta. They also were not there). This all leads me to the question, which I present with all due respect (NOT): What the fuck do they expect us to do??? It is outrageous; above all because in the best case they will sell us a watery coffee download that won't allow us to get all the details a vinyl or less compression would give us. In the worst case, post-MP3 groups will end up recording music with no harmonics or hidden sounds, because, what for? Nobody will hear it. They even admit it: "Some musicians and audio engineers say the MP3 format is changing the way studios mix their recordings. They say the MP3 format "flattens" dynamics differences in tone and volume in a song. As a result, a great deal of new music sounds very much alike, and there is nothing as focusing to create a dynamic listening experience. Why working so hard in creating complex sound if nobody can detect it?" (Rolling Stone, The Death of High fidelity, December 26 2007, taken from here). That's why I am not surprised by Adri n's post regarding the sales of old records. The price has nothing to do with it. The causes are related to the fetish object record and what it means or does not mean for people that have never purchased one. Adri n also asks if somebody here keeps buying records. I answered that I would if the stores sold anything I like. I do it even after the nausea I feel while reading "This phonogram is an intellectual work protected in favor of its producer COPYING IT IN WHOLE OR PARTIALLY IS FORBIDDEN " (like that, uppercased, yelling to whoever is only guilty of having bought a record and defending the producer, not the artist). But I am sure that almost nobody buys records because doing so is no longer a gratifying experience; because if buying a record is clicking to wait 15 days for it to reach the mailbox, we prefer clicking on the download link. But there is another reason for people not to buy records anymore. In one of my talks regarding the dictatorship of the all rights reserved, I asked the 30 twenty-something-year-old students if any of them had ever bought a record. One answered he had, because he is an author and performer (cantautor in Spanish) and understands the effort that producing a record entails. The rest of them had never done so. Is it possible that said young people have never listened to real music? Is it possible that, were it not for concerts, what they consider music is a set of washed-out MP3 that are about to fill up 1TB of their computer? Does people no longer buy records because they cannot differentiate one sound from the other? It is not very clear for me where do I want to get to. The recording industry is despicable. An industry that instead of innovating sets its energy on suing adolescents for downloading songs, trying to pass laws restricting our freedoms in Internet, putting up DRMs making us hostages to our devices* and forcing us to listen just the aroma of music, deserves my whole contempt. If we add ot this that said industry won't allow us to legally download their breadcrumbs because it has not understood that Internet does not need a van crossing borders, besides my contempt they deserve my pity and my heartfelt condolence. But the condolence is also for music, real music, that which is not compressed under the shoe using a terrible format. It is also for independent musicians that have not realized that begging for a bit of space to that industry they just add to themselves the "despicable" tag, given they deserve the fruits of their work to enter their bank account. However, there are good things that have come out of this absurdity. Be it for those that have joined projects such as Aut mata (even if it is in MP3) and for dreams come true such as Musopen (that have achieved that the music that's in theory Public Domain becomes so in practice as well). Good for the Electronic Frontier Foundation and the list of lawyers willing to defend people accused of ilegally downloading music in the USA. Good for the Creative Commons licenses that allow free sharing. All those are growing solutions, although none of them allows me to buy the Panamanian Carlos M ndez's record. Thankfully, a friend of mine who knows I will never give a dime to Apple, bought the files for me in iTunes. I thank him deeply, although I would have prefered to go to Auco Disco and have Mauricio tell me that the 2007 EP I have from Carlos is better than the record he did on 2009. * My devices don't have DRM because I use free software. I also use the ogg file format. Image by verbeeldingskr8

Continuing on from my first EFI progress report... My first alpha release of a Wheezy netinst CD for amd64 EFI worked OK for me and a few others, with reasonable feedback. But it was far from complete, with (at least) three outstanding issues:

• It didn't work as a hybrid (CD or USB) image, just working from CD for EFI
• The initial grub display was really ugly, with no graphics
• I hadn't been able to get grub-efi to work, so it only allowed use of elilo.

In the days since then, I've carried on hacking on things. I've not yet sorted out hybrid boot, but I've fixed the other issues. To make graphics work on the grub boot menu was a simple case of loading the right module from grub.cfg ("insmod png"). Getting grub-efi to work took quite a lot more effort, though. I've been looking through Ubuntu's version of debian-installer code and merged a fair amount of their code into my local builds, adding tweaks as necessary. Result? I've got another CD alpha release ready for testing. Yay! Grab the image from http://cdimage.debian.org/cdimage/unofficial/efi-development/upload2/ if you'd like to help test it. Please do, and let me know via debian-boot/debian-cd how you get on. As previously, the "bits" subdirectory contains all the tweaked d-i packages I've played with, in both source and binary form. My debian-cd changes are still in my branch but are just about ready for merging I think. I'll post more d-i patches to the debian-boot list for review shortly.

IT S FINALLY IN! With this last upload of the MiniUPnP daemon 1.7-3, I ve reached a first milestone: all of the MiniUPnP libraries, daemons and clients are finally in Debian, in a shape which I am not ashamed anymore. What is UPnP for? Back in the days (around Linux 2.0.3x), the only way to share a single Internet connection was to run a Linux box as your gateway. These days are far gone, and nearly every home connection is done through a mini home router, which also often provides a small switch and a WiFi access point. Every computer on the LAN gets its connectivity through the router thanks to NAT, and to be more accurate, masquerading. That works very well for outbound connections (browse the web, send a mail, etc.), but it is then a lot harder that something connects to a server located in the LAN. If you run a permanent server with a fixed LAN IP address, that s fine, you just configure your router to do some port forwarding, so that any inbound connection will be forwarded to the computer of your choice in the LAN. But if one application needs to listen on the public IP address just for a while (for example, IRC using DCC connection), so that something can connect to it, it becomes a way more tricky. That s when UPnP comes into play. UPnP stands for Universal Plug and Play. It s a protocol so that any application on the LAN can open ports on the public IP address of the gateway. Note that an UPnP router is also often referenced as UPnP IGD (Internet Gateway Device). Why using MiniUPnP, don t we have already linux-igd? Before my packaging work, the only UPnP server that was available in Debian was linux-igd, which has a dependency on libupnpX. That library is quite big for what it does (about 500KiB, on top of which you have to install the 200KiB daemon itself), while MiniUPnP is designed to be lightweight (only 158K, with dependencies on iptables, iproute, uuid-runtime and libc6 only). That seems ridiculous, but when you have only 8MiB of flash in your old router, saving few hundreds KiB is really important, and here, we re saving more than 500 KiB, with a full implementation of the UPnP protocol, and the Microsoft NATPmP implemented as well (yes, Microsoft decided that one protocol wasn t enough, and that they should implement their own (sic!)). Who is using MiniUPNP? Probably, you already have the libminiupnpc installed in your SID/wheezy desktop, because the transmission bittorent client which is installed with Gnome by default, depends on it, and minissdpd might be in your desktop too, if you installed the Recommends: packages as well. Which is why these are scoring quite high already in popcon. Previously, transmission was embedding its own version of the MiniUPnP client library, so it s nice to have it as a standalone shared library now. Also, because of its very light weight, and the fact that UPnP is often needed for gaming (both XBOX 360 and PSP need, and can use, the MiniUPnP daemon that runs on your IGD), many ISP who ship a router to their customer, install MiniUPnP in it. For example, at least, the 2nd and 3rd largest Internet access provider in France (Free Telecom and SFR) are shipping MiniUPnPd in their box (one of them using OpenWRT on their router). So probably as well, you are also using MiniUPnP daemon without knowing it. But isn t UPnP unsafe? To this question, I d answer that connecting to Internet is unsafe. It isn t less safe than connecting to Internet on a WiFi hotspot for example. So adding UPnP support on your gateway doesn t make it less safe, especially since MiniUPnP has some features to help with security (like an ACL with IP and port lists, or a secure mode were the router will only open ports to the client requesting it). Also, we are running a distribution where dependencies are easy to check, so it s easy for you to check if a client uses UPnP or not, and decide to trust it (or not). I personally have no issue trusting transmission, x-chat or warzone2010 (more exactly: I don t trust that incoming connectivity will make it less safe for these applications, there are so many other things that could go wrong). MiniUPnP consists of what? MiniUPnP is a set of 4 source packages to handle all of your UPnP needs: - The client library, libminiupnpc (and the corresponding -dev package), with its sidekick miniupnpc client program (you can use it to check the public IP of your gateway, or manually open ports). - The UPnP IGD daemon, miniupnpd, which just reached Debian Experimental. The latest version, 1.7-3, also includes debconf configuration so that you can choose the NIC and IP address to bind on. If you installed previous version, I strongly recommend upgrading to fix previous problems. - The libnatpmp client library (and the -dev package) for implementing the Microsoft protocol as a client (it also has a client package binary for your tests). - The minissdpd daemon to keep memory of all UPnP devices that announced themselves, and speed-up using the UPnP protocol (used by the MiniUPnP client library). A long process to have MiniUPnP in Debian All these 4 pieces of the puzzle were not as easy as it seems to be packaged in Debian. Especially, the IGD daemon, MiniUPnPd, used to depend on some header files that were not packaged in Debian. This was a decision of the iptables maintainers, which even after a long discussion, refused to add the necessary headers because it wasn t supposed to be a kernel API. While this might really be truth, the issue was that there was no public API available in the Linux kernel at all, and other distributions (Fedora, Gentoo, etc.) really did supply these needed files. As a consequence, during a very long time (years literally), shamefully, it was impossible to build the MiniUPnP daemon in Debian, while it was no problem on other distributions. Note that I do not want to start a troll discussion here, I m just stating facts, both parties have very strong and respectable arguments. Anyway, this needed API was finally embedded in MiniUPnPd itself thanks to a contributor to the project, and now it builds on both the Squeeze and the Wheezy (that is Linux 3.2) kernel without any issue. Call for testing Unfortunately, I don t have a Linux box as my Internet gateway. So I did the tests that I could: I installed MiniUPnPd on one computer on my LAN, and checked that I could open ports from another computer. So it seems to work, but I had no way to really test it live , with some real clients. So if you have an Debian SID box as your Internet gateway, feedback would be highly appreciated. What is remaining to do? A transition from libminiupnpc5 to libminiupnpc8 has to happen. Of course, this is too late for Wheezy, so this will happen right after the release. There s not so many reverse dependencies, so it should be ok, but I (yet) have zero experience with transitions. BTW, I had hard time to find out what this transition process was, and maybe some more effort for documenting it would help others (I m thinking about adding some text in the Debian wiki for that). Also, I have called for help on debian-devel, because I m currently the only one maintaining all these 4 source packages, and I m trying to have all my packages checked by others, and team maintained. So if you want to give a hand, or just declare yourself as a backup if I m busy, please raise your hand! Upstream is a friend of mine, and is very reactive to any requests (it very rarely took more than 48 hours to get a reply). Enjoy MiniUPnP and report bugs! Again, feedback would be appreciated. :)

In the process of installing and configuring wanna-build, I had lots of problems. Because there was almost no documentation on the subject (and some of the existing was outdated), I had to figure out how everything works by reading the source code and asking people on IRC and mailing lists. I use the most recent version (from git) and since it s not stable, there were some problems and bugs. I fixed several bugs in order to set the system up. I also documented every step so now there s documentation on everything I have done. I uploaded the documentation on installing and configuring wanna-build [0]. I installed and configured reprepro on a local machine in order to use it with wanna-build. In the process I made step-by step setup instructions [1]. I installed and configured sbuild. The lastest version of sbuild contains filters but they don t work. Their purpose is to keep only the necessary variables in the environment, but apparenly the configuration file which should set them is never read. That leaves the filter list empty which causes the environment to be empty. I had to disable them to keep my work going. I installed buildd. I had some problems setting it up there was a bug in the scripts related to parsing the configuration file. I was pointed to a helpful bug report on the topic and with some additional changes I eventually got it working. In the process I had to read some of the source code of buildd. I made a patch that fixes a bug in the manual page of reprepro and got it approved. I contacted the Alioth admin team via the web site, requesting to use their server for my project Clang for build services . I got approved and I created a git repository where I m now uploading my work. The repository that I use to upload changes to sbuild and buildd is [2]. I add the name of the used compiler to the control file and to the name of the deb package. I wrote a function for parsing the debian/rules file looking for hard-coded compiler there. If there is such compiler, which is different from the one chosen as a default, the building proccess exits with a clear error. I made changes to wanna-build, dpkg, sbuild and buildd to fix bugs in them and make it possible to use a compiler different from gcc for the build process.
After my work a build compiler can be set easily. If a package can not be built with that compiler, the build process fails with a clear error. I also modified the pgstatus web interface to display build logs and create reports on user request. I wrote complete documentation about setting up wanna-build, reprepro and buildd ([0] and [1]). This makes a fully functional buildd infrastructure. Now we have a full step by step tutorial. Me and Sylvestre Ledru who was my great mentor, modified the buildd daemon so it will be possible to be run several instances of it for different architectures on a single physical server. I made a lot of changes to pgstatus this is a web interface to the wanna-build database. I had to add some features and fix several problems. In the process I read some of the code of pgstatus and buildd-tools. Now I have a better understanding of how they work. I added a new feature in pgstatus that allows the user to create a full report of the state of all builds. When a designated reports page is opened, a table with the current state of the builds is shown, together with a list of all the reports saved to files for the current distribution and architecture. The reports can be downloaded directly from the
interface. They are stored in JSON format. The log files were not accessible from pgstatus because of lack of implementation and bugs. I created a script that reads emails sent by wanna-build and extracts log files from them. I made changes to wanna-build so that it feeds the database with the necessary information about package history. I updated the wanna-build patch to include these changes. As a result the buildd logs can now be opened from the web interface of pgstatus. I applied a patch that fixes bug [3] so that config_schroot.sh (this script installs and sets the default compiler) starts with the necessary privileges. I added a new feature to sbuild that allows scripts to be run right after the dependencies are installed in the chroot and before the actual package starts building. Now anybody can configure sbuild to run such a script by modifying the appropriate configuration file. By adding the following lines: $external_commands =
chroot-pre-build-commands => [
['/root/purge-gcc.sh'],
],
; The script /root/purge-gcc.sh is run by sbuild in the chroot. This script can be necessary to set everything in the environment so that packages are built exactly as needed. Me and my mentor used this to replace gcc with a script that removes all gcc-specific packages so that we can be sure whether a package uses clang or not. I have a fully functional buildd infrastructure installed on my development system. In order to get it running, I made several patches: dpkg.patch The main patch with a lot of changes related to compiler support for building.
wanna_build.patch Fixes bugs and makes wanna-build feed the database with package history information.
sbuild.patch Fixes bugs and work-arounds the problem that some sbuild scripts are called without reading the configuration file of sbuild. Adds configuration option that allows scripts to be run right after the dependencies are installed in the chroot and before the actual package starts building.
pgstatus.patch Fixes bugs and adds reports page to pgstatus. Me and my mentor contacted the Debian wanna-build team to connect our new service with Debian. We explained what the current state of the project is and what we need from them. I thank my mentor Sylvestre Ledru for his presentation on DebConf 12 ([4], [5]). He showed our project and it was seen that people are interested in it. A nice resume of his talk has been made by Michael Larabel on Phoronix [6]. I tested everything thoroughly to make sure my changes don t break anything. I am happy to say that the project is now finished. ==== [0] http://wiki.debian.org/DebianWannaBuildInfrastructureOnOneServer
[1] http://wiki.debian.org/SetupBuildServiceForWanna-build
[2] http://anonscm.debian.org/gitweb/?p=pkg-llvm/clang-build.git;a=summary
[3] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=608840
[4] http://sylvestre.ledru.info/blog/media/5/rebuild-debian-presentation-clang-July-10.pdf
[5] http://meetings-archive.debian.net/pub/debian-meetings/2012/debconf12/high/884_Build_Debian_with_another_compiler.ogv
[6] http://www.phoronix.com/scan.php?page=news_item&px=MTEzODQ

I was hired last year ago by Codethink to work on Baserock, a new method and toolset for developing embedded Linux systems. While I don't have a lot of experience developing embedded Linux systems specifically, I have a fair bit of experience devoloping Linux distributions, from Debian and Ubuntu, and that's quite similar. I had already written up some ideas for doing test driven distro development, and system level branching and merging. Then I got hired to do that stuff for real. Baserock is still a work in progress, but we've just made a public development release of the free software version, and I can now talk about it. This blog post is my own; I am not talking for my employer. I won't write about all the aspects and features of Baserock, since that would make this blog post really long. You can read that stuff in the wiki, anyway, and ask questions on the Baserock mailing list and IRC channel if you're curious. I just thought I'd mention what I've been doing this past year at work. However, there's one thing I want to especially mention: I'm excited. Getting to re-invent workflows and tools for system development is fun. I can get up in the morning, decide to try a new version of gcc, make the one line change, and run the one shell command to rebuild everything, and play with the new system pretty soon after. And if it doesn't work, no worries, I'll just not merge that branch into the mainstream. And then I still have time for other things that workday. Baserock is getting rid of the unnecessary friction in system development, all the stupid little things you have to do just to keep working. That is going to make a lot of developers lives better in this corner of the software development universe.

Today I'm in bed, because I have le sniffles, and a painful headache. I'm using epiphany to write this post, via VNC to my main desktop, but I'm hating it as I've somehow evolved into a state where the following crashes my browser:

• Open browser.
• Navigate to gmail.com
• Login.
• Wait for page to complete loading, showing my empty inbox.
• Click "signout".

Running under GDB shows nothing terribly helpful:

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007ffff51a0a46 in ?? () from /usr/lib/libwebkit-1.0.so.2
#2  0x00007ffff3d8f79d in ?? () from /usr/lib/libsoup-2.4.so.1
#3  0x00007ffff2a4947e in g_closure_invoke () from /usr/lib/libgobject-2.0.so.0
#4  0x00007ffff2a5f7f4 in ?? () from /usr/lib/libgobject-2.0.so.0
...

To get more detail I ran "apt-get install epiphany-browser-dbg" - this narrows down the crash, but not in a useful way:

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007ffff51a0a46 in finishedCallback (session=<value optimized out>, msg=0x7fffd801d9c0, data=<value optimized="optimized" out="out">) at ../WebCore/platform/network/soup/ResourceHandleSoup.cpp:329
#2  0x00007ffff3d8f79d in ?? () from /usr/lib/libsoup-2.4.so.1
#3  0x00007ffff2a4947e in g_closure_invoke () from /usr/lib/libgobject-2.0.so.0
#4  0x00007ffff2a5f7f4 in ?? () from /usr/lib/libgobject-2.0.so.0
..

So this crash happens in ResourceHandleSoup.cpp. Slowly I realized that this came from the webkit package. We see that the last call by name is to the function in line ResourceHandleSoup.cpp:329, that puts us at the last line of this function:

// Called at the end of the message, with all the necessary about the last informations.
// Doesn't get called for redirects.
static void finishedCallback(SoupSession *session, SoupMessage* msg, gpointer data)
 
    RefPtr<ResourceHandle> handle = adoptRef(static_cast<ResourceHandle*>(data));
    // TODO: maybe we should run this code even if there's no client?
    if (!handle)
        return;
    ResourceHandleInternal* d = handle->getInternal();
    ResourceHandleClient* client = handle->client();
    if (!client)
       return;
..
..
    client->didFinishLoading(handle.get());
 

So we see there is some validation that happens, then a call to "didFinishLoading" and somewhere shortly after that it dies. didFinishLoading looks trivial:

void WebCoreSynchronousLoader::didFinishLoading(ResourceHandle*)
 
      g_main_loop_quit(m_mainLoop);
      m_finished = true;
 

So my mental-debugging is stymied. I blame my headache. It looks like there is no obvious NULL-pointer deference, if we pretend client cannot be NULL. So the next step is to get the source, the build-dependencies and then build a debug version of webkit. I ran "apt-get source webkit", then editted the file ./debian/rules to add --enable-debug and rebuilt it:

skx@precious:~/Debian/epiphany/webkit-1.2.7$ DEB_BUILD_OPTIONS="nostrip noopt" debuild -sa

*time passes* The build fails:

  CXX    WebCore/svg/libwebkit_1_0_la-SVGUseElement.lo
../WebCore/svg/SVGUseElement.cpp: In member function  virtual void WebCore::SVGUseElement::insertedIntoDocument() :
../WebCore/svg/SVGUseElement.cpp:125: error:  class WebCore::Document  has no member named  isXHTMLDocument 
../WebCore/svg/SVGUseElement.cpp:125: error:  class WebCore::Document  has no member named  parser 
make[2]: *** [WebCore/svg/libwebkit_1_0_la-SVGUseElement.lo] Error 1

Ugh. So I guess we disable that "--enable-debug", and hope that "nostrip noopt" helps instead. *Thorin sits down and starts singing about gold* Finally the debugging build has finished and I've woken up again. Let us do this thing. I'd looked over the webkit tracker and the crashing bugs list in the meantime, but nothing jumped out at me as being similar to my issue. Anyway without the --enable-debug flag present in the call to ../configure the Debian webkit packages were built, eventually, and installed:

skx@precious:~/Debian/epiphany$ mv libwebkit-dev_1.2.7-0+squeeze2_amd64.deb x.deb
skx@precious:~/Debian/epiphany$ sudo dpkg --install libweb*deb
[sudo] password for skx:
(Reading database ... 173767 files and directories currently installed.)
Preparing to replace libwebkit-1.0-2 1.2.7-0+squeeze2 (using libwebkit-1.0-2_1.2.7-0+squeeze2_amd64.deb) ...
Unpacking replacement libwebkit-1.0-2 ...
Preparing to replace libwebkit-1.0-2-dbg 1.2.7-0+squeeze2 (using libwebkit-1.0-2-dbg_1.2.7-0+squeeze2_amd64.deb) ...
Unpacking replacement libwebkit-1.0-2-dbg ...
Preparing to replace libwebkit-1.0-common 1.2.7-0+squeeze2 (using libwebkit-1.0-common_1.2.7-0+squeeze2_all.deb) ...
Unpacking replacement libwebkit-1.0-common ...
Setting up libwebkit-1.0-common (1.2.7-0+squeeze2) ...
Setting up libwebkit-1.0-2 (1.2.7-0+squeeze2) ...
Setting up libwebkit-1.0-2-dbg (1.2.7-0+squeeze2) ...
skx@precious:~/Debian/epiphany$

Good news everybody: The crash still happens! Firing up GDB should hopefully reveal more details - but sadly it didn't.

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007ffff51a0a46 in finishedCallback (session=<value optimized="optimized" out="out">, msg=0xb03420, data=<value optimized="optimized" out="out">) at ../WebCore/platform/network/soup/ResourceHandleSoup.cpp:329
..
(gdb) up
#1  0x00007ffff51a0a46 in finishedCallback (session=<value optimized="optimized" out="out">, msg=0xb03420, data=<value optimized="optimized" out="out">) at ../WebCore/platform/network/soup/ResourceHandleSoup.cpp:329
329     client->didFinishLoading(handle.get());
(gdb) p client
$1 = <value optimized out>

At this point my head hurts too much, and I'm stuck. No quote today, I cannot be bothered.

After some rants about the status of different Qt-based serial port libs, I'm happy to announce that today QExtSerialPort was accepted in Debian experimental (this last link may give a 404 for some time, that's ok).

A great effort was done upstream to put the source in a suitable state for being included in Debian. Thank you all guys!

I packaged a beta version, so I uploaded it to experimental.

In case you are wondering why packaging a lib with (yet) no apps depending on it, the answer is quite simple: there are lots of people already using it in it's home projects (me included). Now those projects have the chance to get into Debian too :-)

As a side note, there is also another nice implementation called QtSerialPort. I have also tested it and works pretty fine too. It aims to become part of Qt as an add-on someday.

Review: Getting Started with Dwarf Fortress, by Peter Tyson

Publisher:	O'Reilly
Copyright:	May 2012
ISBN:	1-4493-1494-5
Format:	Trade paperback
Pages:	219

If you're unfamiliar with Dwarf Fortress, you're in good company. Most people are, even people who play video games regularly. It was heavily inspired by roguelike games: games such as Hack and Rogue and their more recent variations, which have been around since the 1970s and are still frequently played with simple character graphics on 2D maps. Dwarf Fortress even has a roguelike gameplay mode, but the core of the game (and the only part of the game covered in this book) is the fortress mode, where the player controls a band of dwarfs in an attempt to construct and maintain a fortress. All of this is done (by default, at least) in a simulated-text interface filled with a barrage of strange characters and symbols. A Dwarf Fortress game in progress is almost incomprehensible to the uninitiated. (There are graphical tile sets available, but all the illustrations in this book assume the default display mode.) Dwarf Fortress gains its cult popularity (sufficient to support the primary developer solely through voluntary donations; the game is free) from the incredible depth and richness of the world. Almost everything is modeled in startling detail, down to the individual items of clothing that each dwarf wears, their personalities and history, and a considerable part of the industrial lifecycle of any goods that the fortress needs to produce (such as food and all-important alcohol). That's where this book comes in. The game is so massive and involves so many variables, industries, needs, requirements, and perils that the first-time player is likely to see their first games collapse due to resource problems they have no idea how to address. Getting Started with Dwarf Fortress walks the player through the initial fortress setup, the resource cycles, and the fundamentals of the various parts of the game one has to monitor. The fact that this discussion takes over 200 pages should give you an idea of the depth of the game. I should stop here and warn that, similar to my RPG sourcebook reviews, I read this book somewhat against its intended purpose. I haven't played Dwarf Fortress and don't currently intend to. It's not quite my game type: I don't like open world construction games in which it's possible to lose (and thereby lose all the things that one has built). But the idea of the game, the way it's developed, and the fact that O'Reilly would publish a book about it is so delightfully strange that I wanted to get a feel for the game without investing the time required to learn it. Thankfully, while the book is designed to be a tutorial, it also does an excellent job at conveying the feel of a game. Getting Started with Dwarf Fortress is divided into eleven chapters. After a chapter of introduction, it goes over site selection, some of the options for your initial party, and recommended settings for both. It then provides some advice for the initial rooms and initial fortress layout to get started. Subsequent chapters are more topic-oriented, covering food, merchants, resource management (including Strange Moods), digging, and the various chains of industry. The final chapters cover some of the more intricate details of dwarven life, such as justice, vampires, and hospitals, and then provide an introduction to the military and to dwarven engineering. The engineering chapter, in particular, makes it clear that vast complexity and possibility is left unexamined. One drawback of the printed version of the book (I don't know if this also applies to the ebook) is that all of the screen shots here are black and white. This makes Dwarf Fortress's already-forbidding presentation even more difficult to understand. At times, I had a hard time picking out features that were described in the text. A look at on-line screenshots quickly shows that color is massively helpful in making sense of all the little glyphs. I'm sure color printing would have made the printed version prohibitively expensive, but it is a loss. The text, though, is excellent. Not only is it clear and helpful (although with the caveat that I've not personally tested it), it captures much of the humor and feel of the game. Dwarf Fortress incorporates computer-generated storytelling that narrates the events of the game (and the backstory of the world), which has sparked a tradition of delightfully wry commentary on games, game mechanics, and events in the game (particularly the ones that cause the cascading destruction of the whole fortress). Tyson captures that tone brilliantly, weaving it throughout his descriptions. This is a tutorial that's genuinely fun to read. (Although the Tim Denee comics ending chapters sadly didn't work for me.) It's certainly possible to get started with Dwarf Fortress without this book. There are other tutorials on-line, as well as an impressively huge wiki. But this is a great introduction and is entertainment in its own right. If you've thought about playing the game, or if, like me, you want a good feel for what it's like without investing the time in actually playing it, I recommend it. I should probably note that this book covers Dwarf Fortress up to 0.34.07, and the dead tree version that I have obviously won't be further updated. O'Reilly does note on the back cover that if you purchase the ebook edition, it will receive free updates for the "life of the edition" (however long that is). If you're worried about subsequent changes in the game making the book obsolete (given the ongoing active development, this is something of a worry, although this book mostly covers fundamentals), you may want to grab the electronic version. Rating: 8 out of 10

Quite on schedule, Weblate 1.2 has been released today. It comes with lot of improvements, especially for project admins, handling of po files and improved performance. Full list of changes for 1.2:

• Weblate now uses South for database migration, please check upgrade instructions if you are upgrading.
• Fixed minor issues with linked git repos.
• New introduction page for engaging people with translating using Weblate.
• Added widgets which can be used for promoting translation projects.
• Added option to reset repository to origin (for privileged users).
• Project or subproject can now be locked for translations.
• Possibility to disable some translations.
• Configurable options for adding new translations.
• Configuration of git commits per project.
• Simple antispam protection.
• Better layout of main page.
• Support for automatically pushing changes on every commit.
• Support for email notifications of translators.
• List only used languages in preferences.
• Improved handling of not known languages when importing project.
• Support for locking translation by translator.
• Optionally maintain Language-Team header in po file.
• Include some statistics in about page.
• Supports (and requires) django-registration 0.8.
• Caching of counted units with failing checks.
• Checking of requirements during setup.
• Documentation improvements.

You can find more information about Weblate on it's website, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Ready to run appliances can be found in SUSE Studio Gallery. Weblate is also being used https://l10n.cihar.com/ as official translating service for phpMyAdmin, Gammu, Weblate itself and others. If you are free software project which would like to use Weblate, I'm happy to help you with set up or even host Weblate for you (this will be decided case by case as my hosting space is limited). Update: Weblate appliace has been updated to 1.2 as well.

Filed under: English Weblate 0 comments Flattr this!

I've temporarily got hold of a Retina Macbook Pro. Fedora 17 boots fine on it with the following kernel arguments: nointremap drm_kms_helper.poll=0 video=eDP-1:2880x1800@45e, although you'll want an updated install image. The nointremap requirement is fixed by this patch, and the others are being tracked here. The gmux chip that controls the multiple graphics chipsets has changed - patches here. That just leaves Thunderbolt.

Thunderbolt is, to a first approximation, PCIe tunnelled over a Displayport connector. It's a high-bandwidth protocol for connecting external devices. Things are complicated by it also supporting Displayport over the same connection, which I'm sure will be hilarious when we get yet another incompatible display protocol. But anyway. I picked up a Thunderbolt ethernet adapter and started poking.

Booting with the device connected looked promising - an ethernet device appeared. Pulling it out resulted in the pcie hotplug driver noticing that it was missing and cleaning up, although the b44 ethernet driver sat around looking puzzled for a while. So far so good. Unfortunately plugging it in again didn't work, and rebooting without any Thunderbolt devices connected not only did nothing on hotplug, it also meant that the Thunderbolt controller didn't show up in lspci at all.

It turns out that there's several problems at play here. The first is the missing Thunderbolt controller. The problem here is the following code in Apple's ACPI tables:

        Method (RMCR, 0, Serialized)
         
            If (LNot (OSDW))
             
                If (LAnd (LEqual (\_SB.PCI0.PEG1.UPSB.DSB1.UPS0.AVND, 0xFFFF), 
                    LEqual (\_SB.PCI0.PEG1.UPSB.DSB2.UPS0.AVND, 0xFFFF)))
                 
                    Store ("RMCR: Disable Link and Power Off Cactus Ridge Chip",
                           Debug)
                    Store (0x01, \_SB.PCI0.PEG1.LDIS)
                    Sleep (0x07D0)
                    Store (0x00, GP23)
                 
             
         

This checks if the system claims to be Darwin (the core of OS X). If not, it checks whether two PCIe bridges have been configured. If both are still in an unconfigured state, it cuts the PCIe link to the upstream PCIe link and then sets GP23 to 0. Looking further, GP23 turns out to be a GPIO line. Setting it to 0 appears to cut power to the Thunderbolt controller. In summary, unless your OS claims to be OS X, booting without an attached Thunderbolt device will result in the chipset being powered down so hard that it's entirely invisible to the OS.

Fixing that is as easy as adding Darwin to the list of operating systems Linux claims to be. So, now I could boot without a connected device and still see the controller. Since ACPI seemed to be important here, I started looking at the other ACPI methods associated with the device. The first one that drew attention was Name(_GPE, 0x14). The _GPE method (not to be confused with the _GPE object at the top of the global ACPI namespace) is defined as providing the ACPI general purpose event associated with the device. Unfortunately, it's only defined as doing this for embedded controllers - Apple's use of it on a PCI device is massively non-standard. But, still, easy enough to handle. Intel chipsets map GPE 0x10 to 0x1f to GPIO lines 0-15, so this GPE is associated with GPIO 4. Looking through the ACPI code showed a bunch of other references to GP04, and it turns out that if the chip is powered down (via setting GP23 to 0) then plugging a device in to the port will generate a GPE. This makes sense - a powered down controller isn't going to notice hotplug events itself, so you need some kind of out of band notification mechanism. There's also another ACPI method that flips the polarity of the GPIO line so it doesn't keep generating events for the entire time a device is connected. It didn't take long to come up with a stub driver that would power down the controller if nothing was connected at boot, and then wake it up again on a hotplug event.

Unfortunately that's as far as I've got. I'd been hoping that everything beyond this was just PCIe hotplug, but it seems not. Apple's Thunderbolt driver is rather too large to just be handling ACPI events, and this document on Apple's website makes it pretty clear that there's OS involvement in events and device configuration. Booting with a device connected means that the firmware does the setup, but if you want to support hotplug then the OS needs to know how to do that - and Linux doesn't.

Getting this far involved rather a lot of irritation at Apple for conspiring to do things in a range of non-standard ways, but it turns out that the real villains of the piece are Intel. The Thunderbolt controller in the Apples is an Intel part - the 82524EF, according to Apple. Given Intel's enthusiasm for Linux and their generally high levels of engagement with the Linux development community, it's disappointing[1] to discover that this controller has been shipping for over a year with (a) no Linux driver and (b) no documentation that would let anyone write such a driver. It's not even mentioned on Intel's website. So, thanks Intel. You're awful.

Anyway. This was my attempt to spend a few days doing something more relaxing than secure boot, and all I ended up with was eczema and liver pain. Lesson learned, hardware vendors hate you even more than firmware vendors do.

[1] By which I mean grotesquely infuriating

comments

As promised last month at DebConf, I've been working on adding EFI support to debian-cd and debian-installer, hopefully to get it up and running. At the moment, this is basic support only for booting and setting up the installed system via UEFI. I'm not worrying about "Secure" (better named "Restricted"?) Boot yet - that comes later, and depends on the basic stuff working first. So, how is it going? Quite well, really. I've got a debian-cd branch that I'm working on locally, and with a relatively small amount of work there I've got a locally-built CD that boots in EFI mode, using grub-efi as a boot loader. I've borrowed a script called efi-image from Colin Watson which generates the necessary EFI boot image from bits out of the grub-efi-amd64-bin package. For some reason, this grub image isn't working with all the normal graphics code so I've not got a pretty branded startup screen yet. But, nonetheless, I do have a working EFI boot CD. Yay! Next step: the installer. At the moment, generic amd64 and i386 machines are assumed to use need dos-style partitioning, and then either lilo or grub as a bootloader. I've patched and rebuilt some of the d-i packages locally to add a new sub-architecture of "efi" for both amd64 and i386, alongside the existing "generic" and "mac" variants. Using that new sub-architecture, it's possible to add checks for amd/efi and i386/efi in various places to make the necessary changes:

• choose the GPT partition type for blank disks
• tweak the auto-partitioner to add an EFI (fat32) boot partition on the front of the first hard disk
• install the elilo bootloader

Why elilo? It's the default for current ia64 systems, and for EFI Macs. I tried to get grub-efi working with the grub-installer package, but with little success so far. I'll try again with grub-efi, but for now elilo is much simpler and (importantly!) it works for me. So, I have a netinst CD built that works for me in local testing in a qemu VM using James Bottomley's OVMF build of Tianocore. It's not pretty, but it works fine, covering all the normal Debian installation steps and giving me a UEFI system at the end of the process. I'm about to start testing on real hardware now, so I think it's worth sharing this netinst image with others too. Grab it from http://cdimage.debian.org/cdimage/unofficial/efi-development/upload1/ . The "bits" subdirectory contains all the tweaked d-i packages I've played with, in both source and binary form. If you have an EFI system and would like to help us get it supported well in Debian, please download this image and play with it! Feedback to the debian-cd and/or debian-boot lists, please!

Review: Design Patterns, by Erich Gamma, et al.

Author:	Erich Gamma
Author:	Richard Helm
Author:	Ralph Johnson
Author:	John Vlissides
Publisher:	Addison-Wesley
Copyright:	1995
Printing:	September 1999
ISBN:	0-201-63361-2
Format:	Hardcover
Pages:	374

Design Patterns: Elements of Reusable Object-Oriented Software by the so-called "Gang of Four" (Gamma, Helm, Johnson, and Vlissides) is one of the best-known books ever written about software design, and one of the most widely cited. The language introduced here, including the names of specific design patterns, is still in widespread use in the software field, particularly with object-oriented languages. I've had a copy for years, on the grounds that it's one of those books one should have a copy of, but only recently got around to reading it. The goal of this book is to identify patterns of design that are widely used, and widely useful, for designing object-oriented software. It's specific to the object-oriented model; while some of the patterns could be repurposed for writing OO-style programs in non-OO languages, they are about inheritance, encapsulation, and data hiding and make deep use the facilities of object-oriented design. The patterns are very general, aiming for a description that's more general than any specific domain. They're also high-level, describing techniques and methods for constructing a software system, not algorithms. You couldn't encapsulate the ideas here in a library and just use them; they're ideas about program structure that could be applied to any program with the relevant problem. With the benefit of seventeen years of hindsight, I think the primary impact of this book has been on communication within the field. The ideas in here are not new to this book. Every pattern in Design Patterns was already in use in the industry before it was published; the goal was taxonomy, not innovation. One would not come to Design Patterns to learn how to program, although most introductory texts on object-oriented programming now borrow much of the pattern terminology. Rather, Design Patterns is as influential as it is because it introduced a shared terminology and a rigor around that terminology, allowing writers and programmers to put a name to specific program structures and thus talk about them more clearly. This also allows one to take a step back and see a particular structure in multiple programs, compare and contrast how it's used, and draw some general conclusions about where it would be useful. I have the feeling that the authors originally hoped the book would serve as a toolbox, but I think it's instead become more of a dictionary. The pattern names standardized here are widely used even by people who have never read this book, but I doubt many people regularly turn to this book for ideas for how to structure programs. Design Patterns is divided into two parts: a general introduction to and definition of a software pattern followed by a case study, and then a catalog of patterns. The catalog is divided into creational patterns (patterns for creating objects), structural patterns (patterns for composing objects into larger structures), and behavioral patterns (patterns for interactions between objects). Each pattern in turn follows a very rigid presentation structure consisting of the pattern name and high-level classification, its basic intent, other common names, a scenario that motivates the pattern, comments on the applicability of the pattern, the structure and classes or objects that make up the pattern, how those participants collaborate, how the pattern achieves its goals, comments on implementation issues, sample code, known uses of the pattern in real-world software, and related patterns. As with a dictionary, the authors go to great lengths to keep the structure, terminology, and graphical representations uniform throughout, and the cross-referencing is comprehensive (to the point of mild redundancy). As for the patterns themselves, their success, both as terminology and as useful design elements, varies. Some have become part of the core lexicon of object-oriented programming (Factory Method, Builder, Singleton), sometimes to the point of becoming syntactic elements in modern OO languages (Iterator). These are terms that working programmers use daily. Others aren't quite as widespread, but are immediately recognizable as part of the core toolkit of object-oriented programming (Adapter, Decorator, Proxy, Observer, Strategy, Template Method). In some cases, the technique remains widespread, but the name hasn't caught on (Command, for example, which will be immediately familiar but which I rarely hear called by that name outside of specific uses inside UI toolkits due to ambiguity of terms). Other patterns are abstract enough that it felt like a bit of a reach to assign a name to them (Bridge, Composite, Facade), and I don't think use of those names is common, but the entries are still useful for definitional clarity and for comparing similar approaches with different implications. Only one pattern (Interpreter) struck me as insufficiently generic to warrant recording in a catalog of this type. So far, so good, but the obvious question arises: if you've not already read this book, should you read it? I think the answer to that is debatable. The largest problem with Design Patterns is that it's old. It came late enough in the development of object-oriented programming that it does capture much of the foundation, but OO design has continued to change and grow, and some patterns have either been developed subsequently or have become far more important. For example, Model-View-Controller is conspicuous by its absence, mentioned only in passing in the discussion of the Observer pattern. Any pattern catalog written today would have an extensive discussion. Similarly absent are Inversion of Control and Data Access Object, which are much more central to the day-to-day world of the modern programmer than, say, Memento or Visitor. One could easily go on: Lazy Initialization, Mock Object, Null Object... everyone will have their own list. A related problem is that all the implementation examples are shown in either C++ or Smalltalk (occasionally both). Those were probably the best languages to use at the time, but it's doubtful a modern rewrite would choose them. Smalltalk, in particular, I found nearly incomprehensible for the uninitiated, to the point where I ignored the code and only read the surrounding English description. C++ fares better, but Design Patterns occasionally drifts off into tedious discussions of how to work around C++'s limitations in ways that are irrelevant to the pattern itself and would not be necessary in, say, Java or Python. (This is ameliorated by the fact that C++, unlike Smalltalk, is still in widespread use, so those discussions remain moderately helpful for some readers.) Design Patterns is not, therefore, a very good source for a working knowledge of the most common patterns in use today. It has also become somewhat obsolete via its own success: the concept of a design pattern has become so popular that nearly all introductory texts include at least a basic discussion of design patterns and an introduction to the most notable and useful patterns. I think that's a more comfortable and more efficient way to pick up the basics than reading through this book, which is somewhat dense and which expects from the reader a reasonably good working knowledge of object-oriented programming. And, once you have the basics, MVC, DAO, and similar design patterns are probably more important than the more subtle design patterns presented here. That said, I think the rigor of description and the comparisons and discussions here still have some value. Design Patterns encourages the reader to look at patterns from a higher-level perspective, to think about meta-patterns such as the balance between data hiding and access, or between structure designed for the present purpose and structure that's adaptable to future needs. It's also mildly interesting from a historical standpoint; one can see the inspiration for future language designers in how problems are described here, and see how many of the implementation issues and negative consequences have been corrected or simplified by richer language designs. Overall, I would hesitate to recommend buying this book today, particularly at new textbook prices. But if you're a working object-oriented software designer or programmer, I think it's worth checking out from a library (and, thanks to its influence, any library with a decent software design section will almost certainly have a copy). Read the overall discussion, skim the catalog, and read the discussion of the patterns that strike your interest. It may help provide some additional structure and perspective to how you think about OO design. Rating: 6 out of 10

Last week I helped organize the Open and User Innovation Conference at Harvard Business School. One of many interesting papers presented there was an essay on Institutional Change and Information Production by Fabio Landini from the University of Siena. At the core of the paper is an economic model of the relationship between rights protection and technologies that affects the way that cognitive labor can be divided and aggregated. Although that may sound very abstract (and it is in the paper), it is basically a theory that tries to explain the growth of free software. The old story about free software and free culture (at least among economists and many other academics) is that the movements surged to prominence over the last decade because improvements in communication technology made new forms of mass-collaboration -- like GNU/Linux and Wikipedia -- possible. "Possible", for these types of models, usually means profit-maximizing for rational, profit-seeking, actors like capitalist firms. You can basically think of these attempts as trying to explain why open source claims that free licensing leads to "better quality, higher reliability, more flexibility, lower cost" are correct: new technology makes possible an open development process which leads to collaboration which leads to higher quality work which leads to profit. Landini suggests there are problems with this story. One problem is that it treats technology as being taken for granted and technological changes as effectively being dropped in from outside (i.e., exogenous). Landini points out that software businesses build an enormous amount of technology to help organize their work and to help themselves succeed in what they see as their ideal property rights regime. The key feature of Landini's alternate model is that it considers this possibility. What comes out the other end of the model is a prediction for a multiple equilibrium system -- a situation where there are several strategies that can be stable and profitable. This can help explain why, although free software has succeeded in some areas, its success has hardly been total and usually has not led to change within existing proprietary software firms. After all, there are still plenty of companies selling proprietary software. In Landini's model, free is just one of several winning options. But Landini's model raises what might be an even bigger question. If free software can be as efficient as proprietary software, how would anybody ever find out? If all the successful software companies out there are doing proprietary software, which greedy capitalist is going to take the risk of seeing if they could also be successful by throwing exclusive rights out the window? In the early days, new paths are always unclear, unsure, and unproven. Landini suggests that ethically motivated free software hackers provide what he calls a "cultural subsidy." Essentially, a few hackers are motivated enough by the ethical principles behind free software that they are willing to contribute to it even when it isn't clearly better than proprietary alternatives. And in fact, historically speaking, many free software hackers were willing to contribute to free software even when they thought it was likely less profitable than the proprietary alternative models. As Landini suggests, this group was able to build technological platforms and find new social and business arrangements where the free model actually is competitive. I think that the idea of an "cultural subsidy" is a nice way to think about the important role that ethical arguments play in movements like free software and free culture. "Open source" style efficiency arguments persuade a lot of people. Especially when they are true. But those arguments are only ever true because a group of ethically motivated people fought to find a way to make them true. Free software didn't start out as competitive with proprietary software. It became so only because a bunch of ethically motivated hackers were willing to "subsidize" the movement with their with their failed, and successful, attempts at free software and free culture projects and businesses. Of course, the folks attracted by "open source" style superiority arguments can find the ethical motivated folks shrill, off-putting, and annoying. The ethically motivated folks often think the "efficiency" group is shortsighted and mercenary. But as awkward as this marriage might be, it has some huge upsides. In Landini's model, the ethical folks can build their better world without convincing everyone else that they are right and by relying, at least in part, on the self-interest of others who don't share their principles. Just as the free software movement has done. I think that Landini's paper is a good description of the critically important role that the free software movement, and the FSF in particular, can play. The influence and importance of individuals motivated by principles can go far beyond the groups of people who take an ethical stand. They can make involvement possible for large groups of people who do not think that taking a stand on a particular ethical issue is even a good idea.

I feel a sense of pride when I think that I was involved in the development and maintenance of what was probably the first piece of software accepted into Debian which then had and still has direct up-stream support from Microsoft. The world is a better place for having Microsoft in it. The first operating system I ever ran on an 08086-based CPU was MS-DOS 2.x. I remember how thrilled I was when we got to see how my friend s 80286 system ran BBS software that would cause a modem to dial a local system and display the application as if it were running on a local machine. Totally sweet. When we were living at 6162 NE Middle in the nine-eight 292, we got an 80386 which ran Doom. Yeah, the original one, not the fancy new one with the double barrel shotgun, but it would probably run that one, too. It was also totally sweet and all thanks to our armadillo friends down south and partially thanks to their publishers, Apogee. I suckered my brothers into giving me their allowance from Dad one time so that we could all go in on a Sound Blaster Pro 16 sound card for the family s 386. I played a lot of Team Fortress and Q2CTF on that rig. I even attended the Quake 3 Arena launch party that happened at Zoid s place. I recall that he ported the original quake to Linux. I also recall there being naughty remarks included in the README.txt. When my older brother, Aaron turned 16, he was gifted a fancy car. When asked what type of car I would like when I turned 16, I said that I d prefer a computer instead. So I got a high-end 80486 with math co-processor. It could compile the kernel in 15 minutes flat. With all the bits turned on on in /usr/src/linux/.config. But this was later. I hadn t even heard of linux when I got my system. I wanted to be entertained by the thing. I made sure to get a CD-Rom and a sound card. I got on the beta for Ultima Online and spent a summer as a virtual collier. Digging stuff out of mines north of Britannia and hauling them to town to make weapons and armor out of them. And then setting out in said armor only to be PK d because I forgot healing potions and I was no good at fighting. While I was in the middle of all this gaming, my friend Lucas told me that I should try out this lynx thing that they run at the University of Washington. He heard that it was reported to run doom faster than it ran on MS-DOS. It turns out that it did, but that it was not, in fact, called lynx. Or pine. The Doom engine ran so fast that the video couldn t keep up. This was probably because they didn t use double buffering for frame display, since they didn t want to waste the time maintaining and switching context. I think I downloaded the boot/root 3.5 disk pair and was able to get the system to a shell with an on-phone assist from the Rev. I then promptly got lost in bash and the virtual terminals (OMG! I GET SIX CONSOLES!?) and bought a book on the subject. It shipped with slackware. Which I ran. Until Debian came along. Lucas also recommended that I try out this IRC thing, so I did. And I m still doing it on #linpeople just like I did back then. I learned to write Pascal on dos. Then I learned c while they were trying to teach me c++. I learned emacs and vi when I was attending North Kitsap High School. I learned sed and only a little awk when I took Running Start classes in Lynnwood at Edmonds Community College and perl & x.509 while attending Olympic Community College and simultaneously jr-administering Sinclair Communications. I studied TCP/IP, UNP, APUE, C and algorithms & data structures while preparing for an interview with a company whose CEO claimed to have invented SCSI. I learned PGP and PHP while writing web-based adware for this company. I didn t want to write ads and instead wanted to work in security, so took a job with Security Portal. While there, I wrote what one might call a blogging platform. It worked and made it possible for authors to write prose and poetry. Editors didn t have to manage a database in order to review and publish the posts that were ready. Everyone but me was able to avoid html and cgi. Then I sold pizza. Then I helped bring the bombay company onto the interwebs using the Amazon ECS (now AWS) platform. Then I helped support MaxDB. Then I helped develop and maintain the Amazon blogging platform. And then attempted to reduce the load on the Amazon pager system by doing and enforcing code reviews. It turns out that they prefer to run their support team at full bore and a load average of 16. I am now, still, fully employed in an effort to make hard things possible. The hard thing we re working on now is the implementation and ongoing operations of distributed x.500 infrastructure. This includes request handling, processing and delivery of response ( la HTTP, SMTP, IMAP, SIP, RTP, RTSP, OCSP) including authentication, authorization and auditing (AAA) of all transactions. It s a hard thing to get right, but our product development team gets it right. Consistently and reliably. We make mistakes sometimes (sorry Bago), but we correct them and make the product better. I m the newest member of an R and d team (note: big R, little d) called NTR, which sits behind the firewall that is Product Development, out of production space. In a manner that reminds me of Debian Testing. We try new things. Our current project is to allow users to compare their current (cloud-based or iron-based) IT system with what their system would be like with a BIG-IP in front of it. I can probably come up with a demo if anyone s interested in checking it out. I ll go work on that now.

I feel a sense of pride when I think that I was involved in the development and maintenance of what was probably the first piece of software accepted into Debian which then had and still has direct up-stream support from Microsoft. The world is a better place for having Microsoft in it. The first operating system I ever ran on an 08086-based CPU was MS-DOS 2.x. I remember how thrilled I was when we got to see how my friend s 80286 system ran BBS software that would cause a modem to dial a local system and display the application as if it were running on a local machine. Totally sweet. When we were living at 6162 NE Middle in the nine-eight 292, we got an 80386 which ran Doom. Yeah, the original one, not the fancy new one with the double barrel shotgun, but it would probably run that one, too. It was also totally sweet and all thanks to our armadillo friends down south and partially thanks to their publishers, Apogee. I suckered my brothers into giving me their allowance from Dad one time so that we could all go in on a Sound Blaster Pro 16 sound card for the family s 386. I played a lot of Team Fortress and Q2CTF on that rig. I even attended the Quake 3 Arena launch party that happened at Zoid s place. I recall that he ported the original quake to Linux. I also recall there being naughty remarks included in the README.txt. When my older brother, Aaron turned 16, he was gifted a fancy car. When asked what type of car I would like when I turned 16, I said that I d prefer a computer instead. So I got a high-end 80486 with math co-processor. It could compile the kernel in 15 minutes flat. With all the bits turned on on in /usr/src/linux/.config. But this was later. I hadn t even heard of linux when I got my system. I wanted to be entertained by the thing. I made sure to get a CD-Rom and a sound card. I got on the beta for Ultima Online and spent a summer as a virtual collier. Digging stuff out of mines north of Britannia and hauling them to town to make weapons and armor out of them. And then setting out in said armor only to be PK d because I forgot healing potions and I was no good at fighting. While I was in the middle of all this gaming, my friend Lucas told me that I should try out this lynx thing that they run at the University of Washington. He heard that it was reported to run doom faster than it ran on MS-DOS. It turns out that it did, but that it was not, in fact, called lynx. Or pine. The Doom engine ran so fast that the video couldn t keep up. This was probably because they didn t use double buffering for frame display, since they didn t want to waste the time maintaining and switching context. I think I downloaded the boot/root 3.5 disk pair and was able to get the system to a shell with an on-phone assist from the Rev. I then promptly got lost in bash and the virtual terminals (OMG! I GET SIX CONSOLES!?) and bought a book on the subject. It shipped with slackware. Which I ran. Until Debian came along. Lucas also recommended that I try out this IRC thing, so I did. And I m still doing it on #linpeople just like I did back then. I learned to write Pascal on dos. Then I learned c while they were trying to teach me c++. I learned emacs and vi when I was attending North Kitsap High School. I learned sed and only a little awk when I took Running Start classes in Lynnwood at Edmonds Community College and perl & x.509 while attending Olympic Community College and simultaneously jr-administering Sinclair Communications. I studied TCP/IP, UNP, APUE, C and algorithms & data structures while preparing for an interview with a company whose CEO claimed to have invented SCSI. I learned PGP and PHP while writing web-based adware for this company. I didn t want to write ads and instead wanted to work in security, so took a job with Security Portal. While there, I wrote what one might call a blogging platform. It worked and made it possible for authors to write prose and poetry. Editors didn t have to manage a database in order to review and publish the posts that were ready. Everyone but me was able to avoid html and cgi. Then I sold pizza. Then I helped bring the bombay company onto the interwebs using the Amazon ECS (now AWS) platform. Then I helped support MaxDB. Then I helped develop and maintain the Amazon blogging platform. And then attempted to reduce the load on the Amazon pager system by doing and enforcing code reviews. It turns out that they prefer to run their support team at full bore and a load average of 16. I am now, still, fully employed in an effort to make hard things possible. The hard thing we re working on now is the implementation and ongoing operations of distributed x.500 infrastructure. This includes request handling, processing and delivery of response ( la HTTP, SMTP, IMAP, SIP, RTP, RTSP, OCSP) including authentication, authorization and auditing (AAA) of all transactions. It s a hard thing to get right, but our product development team gets it right. Consistently and reliably. We make mistakes sometimes (sorry Bago), but we correct them and make the product better. I m the newest member of an R and d team (note: big R, little d) called NTR, which sits behind the firewall that is Product Development, out of production space. In a manner that reminds me of Debian Testing. We try new things. Our current project is to allow users to compare their current (cloud-based or iron-based) IT system with what their system would be like with a BIG-IP in front of it. I can probably come up with a demo if anyone s interested in checking it out. I ll go work on that now.

Most Debian source packages are using the 3.0 (quilt) format. This means that Debian changes to upstream files are managed in a quilt patch series. Knowledge of quilt is thus a must if you want to get involved in some serious packaging work. Don t worry, this tutorial will teach you how to use quilt in the context of Debian packaging. Pre-requisites Install the packaging-dev package to get a decent set of packages to do Debian packaging. It includes the quilt package.

$ sudo apt-get install packaging-dev

What s quilt? Read the description of the Debian package:

Quilt manages a series of patches by keeping track of the changes each of them makes. They are logically organized as a stack, and you can apply, un-apply, update them easily by traveling into the stack (push/pop). Quilt is good for managing additional patches applied to a package received as a tarball or maintained in another version control system. The stacked organization is proven to be efficient for the management of very large patch sets (more than hundred patches). As matter of fact, it was designed by and for Linux kernel hackers (Andrew Morton, from the -mm branch, is the original author), and its main use by the current upstream maintainer is to manage the (hundreds of) patches against the kernel made for the SUSE distribution.

The various files involved To better understand the various quilt commands, you should have a basic idea of how the tool works. The stack of patches is maintained in a dedicated directory ( patches by default, but in Debian packages we override this value to debian/patches ). This directory contains the patch files and a series file that gives an ordered list of patches to apply. Example:

$ ls debian/patches/
01_use_stdlib_htmlparser_when_possible.diff
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
06_use_debian_geoip_database_as_default.diff
series
$ cat debian/patches/series
01_use_stdlib_htmlparser_when_possible.diff
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
06_use_debian_geoip_database_as_default.diff

When quilt is used, it also maintains some internal files in a directory of its own (it s named .pc ). This directory is used to know what patches are currently applied (.pc/applied-patches) and to keep backup copies of files modified by the various patches. Configuring quilt Before going further, you should put this in your ~/.quiltrc file:

for where in ./ ../ ../../ ../../../ ../../../../ ../../../../../; do
    if [ -e $ where debian/rules -a -d $ where debian/patches ]; then
        export QUILT_PATCHES=debian/patches
        break
    fi
done

This ensures that quilt will always use debian/patches instead of patches when your current directory is within a Debian source package where debian/patches exists. If you only use quilt for debian packaging, then you can be more expedient and put a simple export QUILT_PATCHES=debian/patches in that file. As a matter of personal preferences, I also have those lines in my ~/.quiltrc:

QUILT_PUSH_ARGS="--color=auto"
QUILT_DIFF_ARGS="--no-timestamps --no-index -p ab --color=auto"
QUILT_REFRESH_ARGS="--no-timestamps --no-index -p ab"
QUILT_DIFF_OPTS='-p'

It enables syntax coloring for the output of several commands and customizes the generated patches to get rid of useless information. I recommend you to use those settings too. Applying and unapplying patches, navigating in the stack of patches When you already have a patch series, you can navigate in the stack of patches so that any subset of consecutive patches (starting from the bottom) can be applied. quilt series will list all patches known by quilt. You can apply all patches with quilt push -a or unapply them all with quilt pop -a. You can also verify what patches are applied (quilt applied) or unapplied (quilt unapplied). quilt push applies the next unapplied patch (i.e. the patch returned by quilt next) and quilt pop unapplies the last applied patch (i.e. the patch returned by quilt top). You can give a patch name as parameter to quilt push/pop and it will apply/unapply all the patches required until the given patch is on the top. Here are some examples of navigation in a quilt patch series

$ quilt series
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
06_use_debian_geoip_database_as_default.diff
$ quilt applied
No patches applied
$ quilt next
02_disable-sources-in-sphinxdoc.diff
$ quilt push
Applying patch 02_disable-sources-in-sphinxdoc.diff
patching file docs/conf.py
Now at patch 02_disable-sources-in-sphinxdoc.diff
$ quilt push
Applying patch 03_manpage.diff
patching file docs/man/django-admin.1
Now at patch 03_manpage.diff
$ quilt applied
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
$ quilt unapplied
06_use_debian_geoip_database_as_default.diff
$ quilt pop -a
Removing patch 03_manpage.diff
Restoring docs/man/django-admin.1
Removing patch 02_disable-sources-in-sphinxdoc.diff
Restoring docs/conf.py
No patches applied
$ quilt push 03_manpage.diff
Applying patch 02_disable-sources-in-sphinxdoc.diff
patching file docs/conf.py
Applying patch 03_manpage.diff
patching file docs/man/django-admin.1
Now at patch 03_manpage.diff
$ quilt top
03_manpage.diff

Creating a new patch If there s no quilt series yet, you want to create the debian/patches directory first. If you already have one, you need to decide where to insert the new patch. Quilt will always add the new patch just after the patch which is currently on top. So if you want to add the patch at the end of the series, you need to run quilt push -a first. Then you can use quilt new name-of-my-patch.diff to tell quilt to insert a new empty patch after the current topmost patch. In this operation quilt does almost nothing except updating the series file and recording the fact that the new patch is applied (even if still empty at this point!). Now to add changes in this patch, you re supposed to modify files but only after having informed quilt of your intent to modify those files. You do this with quilt add file-to-modify. At this point quilt will make a backup copy of that file so that it can generate the final patch when you re done with your changes. It s quite common to forget this step and to be unable to generate the patch afterward. That s why I recommend you to use quilt edit file-to-modify which is a shorthand for doing quilt add and then opening the file in your favorite text editor. If you want, you can review your work in progress with quilt diff. When you re done with the changes, you should call quilt refresh to generate the patch (or to update it if it was already existing). And since you re a good packager, you call quilt header --dep3 -e to add DEP-3 meta-information to your patch header. Importing an external patch If someone else already prepared a patch, you can just import it right away with quilt import /tmp/the-patch. If you want to import it under a better name you can use the option -P better-patch-name . Like quilt new, it inserts the patch after the topmost patch. Updating patches for a new upstream version With some luck, your patches will still apply with some offsets in line numbers (quilt displays those offsets) and sometimes with some fuzz:

$ quilt push
[...]
Hunk #1 succeeded at 1362 (offset 11 lines).
Hunk #2 succeeded at 1533 with fuzz 1 (offset 4 lines).
[...]

While offsets are nothing to worry about (it means some lines were added and/or removed before the patched part), fuzz means that patch had to ignore some context lines to find the place where to apply the changes. In that case, you need to double check that patch did the right thing because it might have made changes somewhere where it shouldn t. Also, you ll have to update those patches because dpkg-source doesn t accept any fuzz. If you re confident that all patches are correctly applied by quilt, you can refresh them to get rid of those warnings:

$ quilt pop -a
[...]
$ while quilt push; do quilt refresh; done

That was for the easy case. Now let s deal with the case where some of the patches no longer apply. There s one case that is usually nice to have:

$ quilt push
Applying patch 04_hyphen-manpage.diff
patching file docs/man/django-admin.1
Hunk #1 FAILED at 194.
1 out of 1 hunk FAILED -- rejects in file docs/man/django-admin.1
Patch 04_hyphen-manpage.diff can be reverse-applied

When the patch can be reverse-applied, it means that the upstream authors included the Debian patch (or that they made the same change even though you forgot to forward the patch). You can thus get rid of it:

$ quilt delete -r 04_hyphen-manpage.diff
Removed patch 04_hyphen-manpage.diff

Note that without the -r the patch is only dropped from the series file. With -r the patch file is also removed. But there s a less desirable case where the patch is still relevant but it doesn t apply any longer:

$ quilt push
Applying patch 01_disable_broken_test.diff
patching file tests/regressiontests/test_utils/tests.py
Hunk #1 FAILED at 422.
1 out of 1 hunk FAILED -- rejects in file tests/regressiontests/test_utils/tests.py
Patch 01_disable_broken_test.diff does not apply (enforce with -f)

In that case, you should follow quilt s advice to force the patch application, manually apply the parts of the patch that were rejected, and then refresh the patch.

$ quilt push -f
Applying patch 01_disable_broken_test.diff
patching file tests/regressiontests/test_utils/tests.py
Hunk #1 FAILED at 422.
1 out of 1 hunk FAILED -- saving rejects to file tests/regressiontests/test_utils/tests.py.rej
Applied patch 01_disable_broken_test.diff (forced; needs refresh)
$ vim tests/regressiontests/test_utils/tests.py    
$ quilt refresh
Refreshed patch 01_disable_broken_test.diff

Other quilt commands You should probably read quilt s manual page too to learn about the various other commands and options that exist. There s at least quilt rename new-name that you can also find useful to rename the topmost patch (you can use -P patch-to-rename to rename a patch which is not currently at the top). Feedback Please leave comments if you have suggestions of improvements, or if there are some tips that are good to know. I might incorporate them in this article. Feel free to share this article with newbie packagers which are struggling with quilt. For your convenience, you can also refer to this article with this URL: http://raphaelhertzog.com/go/quilt

Found it useful? Be sure to not miss other packaging tips (or lessons), click here to subscribe to my free newsletter and get new articles by email.

2 comments Liked this article? Click here. My blog is Flattr-enabled.

The general trend seems to be that cheap hard drives are increasing in capacity faster than much of the data that is commonly stored. Back in 1998 I had a 3G disk in my laptop and about 800M was used for my home directory. Now I have 6.2G used for my home directory (and another 2G in ~/src) out of the 100G capacity in my laptop. So my space usage for my home directory has increased by a factor of about 8 while my space available has increased by a factor of about 30. When I had 800M for my home directory I saved space by cropping pictures for my web site and deleting the originals (thus losing some data I would rather have today) but now I just keep everything and it s still doesn t take up much of my hard drive. Similar trends apply to most systems that I use and that I run for my clients. Due to the availability of storage people are gratuitously using a lot of disk space. A relative recently took 10G of pictures on a holiday, her phone has 12G of internal storage so there was nothing stopping her. She might decide that half the pictures aren t that great if she had to save space, but that space is essentially free (she couldn t buy a cheaper phone with less storage) so there s no reason to delete any pictures. When considering backup methods one important factor is the ability to store all of one type of data on one backup device. Having a single backup span multiple disks, tapes, etc has a dramatic impact on the ease of recovery and the potential for data loss. Currently 3TB SATA disks are really cheap and 4TB disks are available but rather expensive. Currently only one of my clients has more than 4TB of data used for one purpose (IE a single filesystem) so apart from that client a single SATA disk can backup anything that I run. Benefits of Hard Drive Backup When using a hard drive there is an option to make it a bootable disk in the same format as the live disk. I haven t done this, but if you want the option of a quick recovery from a hardware failure then having a bootable disk with all the data on it is a good option. For example a server with software RAID-1 could have a backup disk that is configured as a degraded RAID-1 array. The biggest benefit is the ability to read a disk anywhere. I ve read many reports of tape drives being discovered to be defective at the least convenient time. With a SATA disk you can install it in any PC or put it in a USB bay if you have USB 3.0 or the performance penalty of USB 2.0 is bearable a USB 2.0 bay is great if you want to recover a single file, but if you want terabytes in a hurry then it won t do. A backup on a hard drive will typically use a common filesystem. For backing up Linux servers I generally use Ext3, at some future time I will move to BTRFS as having checksums on all data is a good feature for a backup. Using a regular filesystem means that I can access the data anywhere without needing any special software, I can run programs like diff on the backup, and I can export the backup via NFS or Samba if necessary. You never know how you will need to access your backup so it s best to keep your options open. Hard drive backups are the best solution for files that are accidentally deleted. You can have the first line of backups on a local server (or through a filesystem like BTRFS or ZFS that supports snapshots) and files can be recovered quickly. Even a SATA disk in a USB bay is very fast for recovering a single file. LTO tapes have a maximum capacity of 1.5TB at the moment and tape size has been increasing more slowly than disk size. Also LTO tapes have an expected lifetime of only 200 reads/writes of the entire tape. It seems to me that tapes don t provide a great benefit unless you are backing up enough data to need a tape robot. Problems with a Hard Drive Backup Hard drives tend not to survive being dropped so posting a hard drive for remote storage probably isn t a good option. This can be solved by transferring data over the Internet if the data isn t particularly big or doesn t change too much (I have a 400G data set backed up via rsync to another country because most of the data doesn t change over the course of a year). Also if the data is particularly small then solid state storage (which costs about $1 per GB) is a viable option, I run more than a few servers which could be entirely backed up to a 200G SSD. $200 for a single backup of 200G of data is a bit expensive, but the potential for saving time and money on the restore means that it can be financially viable. Some people claim that tape storage will better survive a Carrington Event than hard drives. I m fairly dubious about the benefits of this, if a hard drive in a Faraday Cage (such as a regular safe that is earthed) is going to be destroyed then you will probably worry about security of the food supply instead of your data. Maybe I should just add a disclaimer this backup system won t survive a zombie apocalypse . ;) It s widely regarded that tape storage lasts longer than hard drives. I doubt that this provides a real benefit as some of my personal servers are running on 20G hard drives from back when 20G was big. The fact that drives tend to last for more than 10 years combined with the fact that newer bigger drives are always being released means that important backups can be moved to bigger drives. As a general rule you should assume that anything which isn t regularly tested doesn t work. So whatever your backup method you should test it regularly and have multiple copies of the data to deal with the case when one copy becomes corrupt. The process of testing a backup can involve moving it to newer media. I ve seen it claimed that a benefit of tape storage is that part of the data can be recovered from a damaged tape. One problem with this is that part of a database often isn t particularly useful. Another issue is that in my experience hard drives usually don t fail entirely unless you drop them, drives usually fail a few sectors at a time. How to Implement Hard Drive Backup The most common need for backups is when someone deletes the wrong file. It s usually a small restore and you want it to be an easy process. The best solution to this is to have a filesystem with snapshots such as BTRFS or ZFS. In theory it shouldn t be too difficult to have a cron job manage snapshots, but as I ve only just started putting BTRFS and ZFS on servers I haven t got around to changing my backups. Snapshots won t cover more serious problems such as hardware, software, or user errors that wipe all the disks in a server. For example the only time I lost a significant amount of data from a hosted server was when the data center staff wiped it, so obviously good off-site backups are needed. The easiest way to deal with problems that wipe a server is to have data copied to another system. For remote backups you can rsync to a local system and then use cp -rl or your favorite snapshot system to make a hard linked copy of the tree. A really neat feature is the ZFS ability to send a filesystem snapshot (or the diff between two snapshots) to a remote system [1]. Once you have regular backups on local storage you can then copy them to removable disks as often as you wish, I think I ll have to install ZFS on some of my servers for the sole purpose of getting the send feature! There are NAS devices that provide similar functionality to the ZFS send/receive (maybe implemented with ZFS), but I m not a fan of cheap NAS devices [2]. It seems that the best way to address the first two needs of backup (fast local restore and resilience in the face of site failure) is to use ZFS snapshots on the server and ZFS send/receive to copy the data to another site. The next issue is that the backup server probably won t be big enough for all the archives and you want to be able to recover from a failure on the backup server. This requires some removable storage. The simplest removable backup is to use a SATA drive bay with eSATA and USB connectors. You use a regular filesystem like Ext3 and just copy the files on. It s easy, cheap, and requires no special skill or software. Requiring no special skill is important, you never know who will be called on to recover from backups. When a server is backing up another server by rsync (whether it s in the same rack or another country) you want the backup server to be reliable. However there is no requirement for a single reliable server and sometimes having multiple backup servers will be cheaper. At current purchase prices you can buy two cheap tower systems with 4*3TB disks for less money than a single server that has redundant PSUs and other high end server features. Having two cheap servers die at once seems quite unlikely so getting two backup servers would be the better choice. For filesystems that are bigger than 4TB a disk based backup would require backup software that handles multi part archives. One would hope that any software that is designed for tape backup would work well for this (consider a hard drive as a tape with a very fast seek), but often things don t work as desired. If anyone knows of a good Linux backup program that supports multiple 4TB SATA disks in eSATA or USB bays then please let me know. Conclusion BTRFS or ZFS snapshots are the best way of recovering from simple mistakes. ZFS send/receive seems to be the best way of synchronising updates to filesystems to other systems or sites. ZFS should be used for all servers. Even if you don t currently need send/receive you never know what the future requirements may be. Apart from needing huge amounts of RAM (one of my servers had OOM failures when it had a mere 4G of RAM) there doesn t seem to be any down-side to ZFS. I m unsure of whether to use BTRFS for removable backup disks. The immediate up-sides are checksums on all data and meta-data and the possibility of using built-in RAID-1 so that a random bad sector is unlikely to lose data. There is also the possibility of using snapshots on a removable backup disk (if the disk contains separate files instead of an archive). The down-sides are lack of support on older systems and the fact that BTRFS is fairly new. Have I missed anything?

• [1] http://cuddletech.com/blog/pivot/entry.php?id=984
• [1] http://etbe.coker.com.au/2012/03/23/cheap-nas-devices-suck/

Related posts:

1. New Storage Developments Eweek has an article on a new 1TB Seagate drive....
2. IDE hard drives I just lent two 80G IDE drives to a friend,...
3. Hot-swap Storage I recently had to decommission an old Linux server and...

Great effort of new translators finally forced me to release Ukolovnik 1.4. There are only minor bug fixes besides translations updates. Full list of changes:

• New Spanish translation thanks to Mat as Bellone.
• New Portuguese translation thanks to Everton R.
• Mew Chinese translation thanks to Siramizu.
• New Danish translation thanks to Aputsiaq Niels Janussen.
• Make it work without locales at all.

PS: I don't plan any further development of this tool, in fact I have not touched it in last year and I don't expect this to change.

Filed under: English Ukolovnik 0 comments Flattr this!