Review: Getting Started with Dwarf Fortress, by Peter Tyson

Publisher:	O'Reilly
Copyright:	May 2012
ISBN:	1-4493-1494-5
Format:	Trade paperback
Pages:	219

If you're unfamiliar with Dwarf Fortress, you're in good company. Most people are, even people who play video games regularly. It was heavily inspired by roguelike games: games such as Hack and Rogue and their more recent variations, which have been around since the 1970s and are still frequently played with simple character graphics on 2D maps. Dwarf Fortress even has a roguelike gameplay mode, but the core of the game (and the only part of the game covered in this book) is the fortress mode, where the player controls a band of dwarfs in an attempt to construct and maintain a fortress. All of this is done (by default, at least) in a simulated-text interface filled with a barrage of strange characters and symbols. A Dwarf Fortress game in progress is almost incomprehensible to the uninitiated. (There are graphical tile sets available, but all the illustrations in this book assume the default display mode.) Dwarf Fortress gains its cult popularity (sufficient to support the primary developer solely through voluntary donations; the game is free) from the incredible depth and richness of the world. Almost everything is modeled in startling detail, down to the individual items of clothing that each dwarf wears, their personalities and history, and a considerable part of the industrial lifecycle of any goods that the fortress needs to produce (such as food and all-important alcohol). That's where this book comes in. The game is so massive and involves so many variables, industries, needs, requirements, and perils that the first-time player is likely to see their first games collapse due to resource problems they have no idea how to address. Getting Started with Dwarf Fortress walks the player through the initial fortress setup, the resource cycles, and the fundamentals of the various parts of the game one has to monitor. The fact that this discussion takes over 200 pages should give you an idea of the depth of the game. I should stop here and warn that, similar to my RPG sourcebook reviews, I read this book somewhat against its intended purpose. I haven't played Dwarf Fortress and don't currently intend to. It's not quite my game type: I don't like open world construction games in which it's possible to lose (and thereby lose all the things that one has built). But the idea of the game, the way it's developed, and the fact that O'Reilly would publish a book about it is so delightfully strange that I wanted to get a feel for the game without investing the time required to learn it. Thankfully, while the book is designed to be a tutorial, it also does an excellent job at conveying the feel of a game. Getting Started with Dwarf Fortress is divided into eleven chapters. After a chapter of introduction, it goes over site selection, some of the options for your initial party, and recommended settings for both. It then provides some advice for the initial rooms and initial fortress layout to get started. Subsequent chapters are more topic-oriented, covering food, merchants, resource management (including Strange Moods), digging, and the various chains of industry. The final chapters cover some of the more intricate details of dwarven life, such as justice, vampires, and hospitals, and then provide an introduction to the military and to dwarven engineering. The engineering chapter, in particular, makes it clear that vast complexity and possibility is left unexamined. One drawback of the printed version of the book (I don't know if this also applies to the ebook) is that all of the screen shots here are black and white. This makes Dwarf Fortress's already-forbidding presentation even more difficult to understand. At times, I had a hard time picking out features that were described in the text. A look at on-line screenshots quickly shows that color is massively helpful in making sense of all the little glyphs. I'm sure color printing would have made the printed version prohibitively expensive, but it is a loss. The text, though, is excellent. Not only is it clear and helpful (although with the caveat that I've not personally tested it), it captures much of the humor and feel of the game. Dwarf Fortress incorporates computer-generated storytelling that narrates the events of the game (and the backstory of the world), which has sparked a tradition of delightfully wry commentary on games, game mechanics, and events in the game (particularly the ones that cause the cascading destruction of the whole fortress). Tyson captures that tone brilliantly, weaving it throughout his descriptions. This is a tutorial that's genuinely fun to read. (Although the Tim Denee comics ending chapters sadly didn't work for me.) It's certainly possible to get started with Dwarf Fortress without this book. There are other tutorials on-line, as well as an impressively huge wiki. But this is a great introduction and is entertainment in its own right. If you've thought about playing the game, or if, like me, you want a good feel for what it's like without investing the time in actually playing it, I recommend it. I should probably note that this book covers Dwarf Fortress up to 0.34.07, and the dead tree version that I have obviously won't be further updated. O'Reilly does note on the back cover that if you purchase the ebook edition, it will receive free updates for the "life of the edition" (however long that is). If you're worried about subsequent changes in the game making the book obsolete (given the ongoing active development, this is something of a worry, although this book mostly covers fundamentals), you may want to grab the electronic version. Rating: 8 out of 10

A recent post by Steve Yegge has made quite a splash. The main thesis of his post is:

Software engineering has its own political axis, ranging from conservative to liberal.

Software engineering, like any other social enterprise, does have a political aspect, but I do not think this thesis is apt. The essential property of politics is that you can not have it both ways. A political problem is a coordination problem, and political decisions by its nature apply for an entire political body. Software engineering as a whole does not have these properties. There is absolutely no reason why any two engineering projects have to follow the same ideals and guidelines. Politics in engineering only appears within boundaries of a project (or, at a slightly higher level, a company), at which point it does not really warrant the name of politics, but rather character traits or personal preferences. If Johnny goes for the puck more often than Louie, that does not really warrant a label of a "liberal".

The fundamental quality that Steve is looking at is actually risk aversiveness, as he noted himself. He should have stopped there. Making that next step of conceptualization in political language does not bring any new insights and just muddles the matter.

It seems that the political labels of "liberal" and "conservative" were picked not for their meaning, but rather for their connotations, as political views are perceived to be 1) very stable, part of a person's self-identity and (therefore) hard to change, 2) inherently polarizing, and 3) difficult or impossible to evaluate objectively. Again, these do not apply to software engineering. For example, probably like many engineers, I am clearly a liberal on personal and small projects, and a conservative on larger and more fundamental ones; so much for the idea of a core identity. As for polarization, it may be prevalent in internet echo chambers, but I have not seen much of that in my professional life, especially when working with experienced engineers. When they do have objections, those objections are grounded in engineering. This leads to the third point, claiming that the approaches can not be compared objectively. This one is the most pernicious of all, because it provides an escape hatch from arguments with substance on what is an engineering decision.

Here is my, admittedly less catchy and impressive, counter thesis: Conservativeness of a software engineering project is an objective decision to be based on circumstances. TL;DR: It depends. (Very original, isn't it?) It is a risk management decision, and those are very tricky, but not subjective.

One thing to note is that the architecture of the system should correspond to the implementation style. Schemaless, heavy on magic, untyped approaches are suited for modular systems where each individual module is of limited complexity, and where the modules are isolated from each other and can recover from failures. The more complex the individual modules (and sometimes the complexity is inherent to the problem), the more benefits from "conservative" techniques. (By the way, this lines up nicely with Steve being liberal and a proponent of service-oriented architectures). If you have the freedom to pick between these two approaches, the decision making does get a bit more ideological. However, neither the "liberal" and "conservative" labels, nor risk averseness by itself is enough for a decision here. Say, if you are developing a service, what is more risky: having a "traditional", monolithic service that is inevitably sensitive to failures and is likely to scale poorly, or a cluster of components that are generally more resistant, but have more complicated aggregate behavior?

Despite the criticisms, I have to give fat credit to Steve. His post was thought-provoking, and it brought attention to the subjective side of engineering, one which is often overlooked. That side should not be left to grow - on the contrary - but the only way to reduce its domain it is to be aware of it. I know I will think twice before making another gut decision where risk averseness plays an important factor.

Thanks to the work of Andrej Belym, Debian has now a new C++ standard library. This work has been done in the context of the Google Summer of Code 2012.
Available in Debian Experimental, this new packages provides both the runtime libraries (libc++abi1) and the C++ headers (libc++-dev). With this library, it is possible to build a C++-based program without any dependency on libstdc++. For example, as detailed in README.Debian, with the (amazing) C++ code:

// foo.cpp
#include <iostream> int main()
std::cout < < "plop" << std::endl;
return 0;

build with clang++ (or g++) will give:

$ clang++ -o plop foo.cpp
$ ldd plop grep c++
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f15791cb000)

Using libc++, it will drop the dependency on libstdc++ to use lib++

$ clang++ -stdlib=libc++ -o plop foo.cpp
$ ldd plop grep c++
libc++.so.1 => /usr/lib/x86_64-linux-gnu/libc++.so.1 (0x00007f87464df000)

For the record, it is not that trivial to do with g++. The command being:

g++ -nostdlib -lc++ -lc++abi -std=c++11 -o plop /usr/lib/x86_64-linux-gnu/crt1.o /usr/lib/x86_64-linux-gnu/crti.o foo.cpp /usr/lib/x86_64-linux-gnu/crtn.o -isystem/usr/include/c++/v1 -lc -lgcc_s

Besides this change, libc++ provides a support of C++ 11, considered as "correct" against the C++11 standard by upstream.

// bar.cpp
#include <chrono>
int main()
return 0;

clang++ -stdlib=libc++ -o bar bar.cpp

More information:

• libc++ website
• libc++abi website
• libc++abi Specification

It seems that the UK government recently passed a law that makes it illegal to be unable to decrypt what the law enforcement entities think is encrypted: From http://falkvinge.net/2012/07/12/in-the-uk-you-will-go-to-jail-not-just-for-encryption-but-for-astronomical-noise-too/:

But it s worse than that. Much worse. You re not going to be sent to jail for refusal to give up encryption keys. You re going to be sent to jail for an inability to unlock something that the police think is encrypted. Yes, this is where the hairs rise on our arms: if you have a recorded file with radio noise from the local telescope that you use for generation of random numbers, and the police asks you to produce the decryption key to show them the three documents inside the encrypted container that your radio noise looks like, you will be sent to jail for up to five years for your inability to produce the imagined documents.

This is just insane. Edit: The law was created several years ago, but the blog post somehow made me think it was more recent.

Quite on schedule, Weblate 1.2 has been released today. It comes with lot of improvements, especially for project admins, handling of po files and improved performance. Full list of changes for 1.2:

• Weblate now uses South for database migration, please check upgrade instructions if you are upgrading.
• Fixed minor issues with linked git repos.
• New introduction page for engaging people with translating using Weblate.
• Added widgets which can be used for promoting translation projects.
• Added option to reset repository to origin (for privileged users).
• Project or subproject can now be locked for translations.
• Possibility to disable some translations.
• Configurable options for adding new translations.
• Configuration of git commits per project.
• Simple antispam protection.
• Better layout of main page.
• Support for automatically pushing changes on every commit.
• Support for email notifications of translators.
• List only used languages in preferences.
• Improved handling of not known languages when importing project.
• Support for locking translation by translator.
• Optionally maintain Language-Team header in po file.
• Include some statistics in about page.
• Supports (and requires) django-registration 0.8.
• Caching of counted units with failing checks.
• Checking of requirements during setup.
• Documentation improvements.

You can find more information about Weblate on it's website, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Ready to run appliances can be found in SUSE Studio Gallery. Weblate is also being used https://l10n.cihar.com/ as official translating service for phpMyAdmin, Gammu, Weblate itself and others. If you are free software project which would like to use Weblate, I'm happy to help you with set up or even host Weblate for you (this will be decided case by case as my hosting space is limited). Update: Weblate appliace has been updated to 1.2 as well.

Filed under: English Weblate 0 comments Flattr this!

I've temporarily got hold of a Retina Macbook Pro. Fedora 17 boots fine on it with the following kernel arguments: nointremap drm_kms_helper.poll=0 video=eDP-1:2880x1800@45e, although you'll want an updated install image. The nointremap requirement is fixed by this patch, and the others are being tracked here. The gmux chip that controls the multiple graphics chipsets has changed - patches here. That just leaves Thunderbolt.

Thunderbolt is, to a first approximation, PCIe tunnelled over a Displayport connector. It's a high-bandwidth protocol for connecting external devices. Things are complicated by it also supporting Displayport over the same connection, which I'm sure will be hilarious when we get yet another incompatible display protocol. But anyway. I picked up a Thunderbolt ethernet adapter and started poking.

Booting with the device connected looked promising - an ethernet device appeared. Pulling it out resulted in the pcie hotplug driver noticing that it was missing and cleaning up, although the b44 ethernet driver sat around looking puzzled for a while. So far so good. Unfortunately plugging it in again didn't work, and rebooting without any Thunderbolt devices connected not only did nothing on hotplug, it also meant that the Thunderbolt controller didn't show up in lspci at all.

It turns out that there's several problems at play here. The first is the missing Thunderbolt controller. The problem here is the following code in Apple's ACPI tables:

        Method (RMCR, 0, Serialized)
         
            If (LNot (OSDW))
             
                If (LAnd (LEqual (\_SB.PCI0.PEG1.UPSB.DSB1.UPS0.AVND, 0xFFFF), 
                    LEqual (\_SB.PCI0.PEG1.UPSB.DSB2.UPS0.AVND, 0xFFFF)))
                 
                    Store ("RMCR: Disable Link and Power Off Cactus Ridge Chip",
                           Debug)
                    Store (0x01, \_SB.PCI0.PEG1.LDIS)
                    Sleep (0x07D0)
                    Store (0x00, GP23)
                 
             
         

This checks if the system claims to be Darwin (the core of OS X). If not, it checks whether two PCIe bridges have been configured. If both are still in an unconfigured state, it cuts the PCIe link to the upstream PCIe link and then sets GP23 to 0. Looking further, GP23 turns out to be a GPIO line. Setting it to 0 appears to cut power to the Thunderbolt controller. In summary, unless your OS claims to be OS X, booting without an attached Thunderbolt device will result in the chipset being powered down so hard that it's entirely invisible to the OS.

Fixing that is as easy as adding Darwin to the list of operating systems Linux claims to be. So, now I could boot without a connected device and still see the controller. Since ACPI seemed to be important here, I started looking at the other ACPI methods associated with the device. The first one that drew attention was Name(_GPE, 0x14). The _GPE method (not to be confused with the _GPE object at the top of the global ACPI namespace) is defined as providing the ACPI general purpose event associated with the device. Unfortunately, it's only defined as doing this for embedded controllers - Apple's use of it on a PCI device is massively non-standard. But, still, easy enough to handle. Intel chipsets map GPE 0x10 to 0x1f to GPIO lines 0-15, so this GPE is associated with GPIO 4. Looking through the ACPI code showed a bunch of other references to GP04, and it turns out that if the chip is powered down (via setting GP23 to 0) then plugging a device in to the port will generate a GPE. This makes sense - a powered down controller isn't going to notice hotplug events itself, so you need some kind of out of band notification mechanism. There's also another ACPI method that flips the polarity of the GPIO line so it doesn't keep generating events for the entire time a device is connected. It didn't take long to come up with a stub driver that would power down the controller if nothing was connected at boot, and then wake it up again on a hotplug event.

Unfortunately that's as far as I've got. I'd been hoping that everything beyond this was just PCIe hotplug, but it seems not. Apple's Thunderbolt driver is rather too large to just be handling ACPI events, and this document on Apple's website makes it pretty clear that there's OS involvement in events and device configuration. Booting with a device connected means that the firmware does the setup, but if you want to support hotplug then the OS needs to know how to do that - and Linux doesn't.

Getting this far involved rather a lot of irritation at Apple for conspiring to do things in a range of non-standard ways, but it turns out that the real villains of the piece are Intel. The Thunderbolt controller in the Apples is an Intel part - the 82524EF, according to Apple. Given Intel's enthusiasm for Linux and their generally high levels of engagement with the Linux development community, it's disappointing[1] to discover that this controller has been shipping for over a year with (a) no Linux driver and (b) no documentation that would let anyone write such a driver. It's not even mentioned on Intel's website. So, thanks Intel. You're awful.

Anyway. This was my attempt to spend a few days doing something more relaxing than secure boot, and all I ended up with was eczema and liver pain. Lesson learned, hardware vendors hate you even more than firmware vendors do.

[1] By which I mean grotesquely infuriating

comments

Imagine you have got the following iptables rule set:

*filter
:INPUT ACCEPT [2:130]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [119:14185]
-A INPUT -s 127.0.0.0/8 -j ACCEPT
-A INPUT -p tcp -m tcp dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp tcp-flags FIN,SYN,RST,ACK SYN -j DROP
COMMIT

This would allow all traffic from 127.0.0.0/8, on port 22 and 80. Other (TCP/IP) SYN packages (so on all the other connections) would be dropped.
Now you see, that your counter for the SYN DROP rule is increasing and you want to know what is rejected, but how? The simple answer is ULOG the netfilter userspace logging daemon.
In Debian you have got various implementations/variants of it, the local logging one (which I will use here, just called ulogd) and the -postgres, -mysql and -sqlite3 one (that are not the exact package names), with that you also can log everything to a (remote) database.
An special variant is the -pcap one, it will write the logs in the .pcap format, so you can analyze the full traffic. So for our example it is enough to install the package:

apt-get install ulogd

And then add another rule BEFORE our SYN DROP:

-A INPUT -p tcp -m tcp tcp-flags FIN,SYN,RST,ACK SYN -j ULOG
-A INPUT -p tcp -m tcp tcp-flags FIN,SYN,RST,ACK SYN -j DROP

Now you will find in /var/log/ulog/syslogemu.log a log of all connections, which would be dropped, the log looks like this:

Aug 13 14:42:07 srv1 IN=eth0 OUT= MAC=00:0c:29:8c:2b:6c:00:d0:02:eb:e8:0a:08:00 SRC=75.125.70.194 DST=XXX.XXX.XXX.XXX LEN=40 TOS=00 PREC=0 00 TTL=54 ID=9566 PROTO=TCP SPT=57144 DPT=445 SEQ=2770468863 ACK=0 WINDOW=512 SYN URGP=0
Aug 13 14:45:29 srv1 IN=eth0 OUT= MAC=00:0c:29:8c:2b:6c:00:d0:02:eb:e8:0a:08:00 SRC=75.125.70.194 DST=XXX.XXX.XXX.XXX LEN=40 TOS=00 PREC=0 00 TTL=55 ID=13702 PROTO=TCP SPT=58528 DPT=445 SEQ=1217789951 ACK=0 WINDOW=512 SYN URGP=0

So you have got now the information about the full date, mac address (mostly it will be the one of your gateway), source and destination IP, source and destination port, length, protocol, etc. You also could use it to log outgoing connections to port 80 and the IRC ports:

-A OUTPUT -p tcp -m tcp dport 80 -j ULOG -A OUTPUT -p tcp -m tcp dport 6666:6669 -j ULOG

Whatever you want.

Two years ago Launchpad did schema changes once a month. Everyone would cross their fingers and hope while the system administrators took all the application servers offline, patched the database with a months worth of work and brought up the servers again running the new QA d codebase. This had two problems:

1. due to the complexity of the system something like 300 processes have to be stopped or inhibited to take everything offline the downtime duration was often about 90 minutes long irrespective of the schema patch duration. [Some of the processes don't like being interrupted at all].
2. We simply could not deliver any change in less than 1 week, with the on average latency for something that jumped all the queues still being 2 weeks.

About a year ago we wanted to increase the rate at which schema changes could be carried out the efforts to speed Launchpad up had consumed most low hanging fruit and more and more schema patches were required. We didn t want to introduce additional 90 minute downtime windows though. Adopting incremental migrations the sort of change process described in various places on the internet seemed like a good way to make it possible to apply the schema changes without this slow shutdown-and-restart step, which was required because the pre-patch codebase couldn t speak to the new schema. We could optimise each patch to be very fast by avoiding anything that causes a full table scan or table rewrite (such as adding indices, adding columns with a non-NULL default value). That would let us avoid the 90 minutes of downtime caused by stopping and restarting everything. However, that wasn t sufficient the reason Launchpad ended up doing monthly downtime is that previous attempts to do more frequent schema changes had too high a failure rate. A key reason for patch deployment time blowing out when everything wasn t shut down was due to Launchpad being a very busy system with the use of Slony, schema changes require an exclusive lock on all tables. [More recent versions of Slony only lock some tables, but it still requires very widespread locks for most DDL operations]. We re doing nearly 10 thousand transactions per minute, at any point in time there are always locks open on some table in the system: it was highly improbably and effectively impossible for slonik to get an exclusive lock on all tables in a reasonable timeframe. Background tasks that take many minutes to complete exacerbate this we can t just block new transactions long enough to deliver all the in-flight web pages and let locks clear that way. PGBouncer turns out to be an ideal tool here. If you route all your connections through PGBouncer, you have a single point you can deliberately interrupt to clear all database locks in a second or so (it takes time for backends to all notice that their clients have gone). So we combined these things to get what we called Fast Down Time or FDT. We set the following rules for developers:

1. Any schema patch had to complete in <= 15 seconds in our schema staging environment (which has a full copy of the production DB), or we d roll it back and redesign.
2. Any patch could change either code or schema, never both. schema patches were to land on a separate branch and would be promoted to trunk only after deployment. That branch also receives automated merges from trunk after every commit to trunk, so its running the latest code.

This meant that we could be confident in QA: we would QA the new schema and the application process with the current live code (we deploy trunk multiple times a day). We published some documentation about how to write fast schema patches to help socialise the approach. Then we wrote an automated tool that would:

1. Check for known fragile processes and abort if any were found.
2. Check for very long transactions and abort if any were found.
3. Shutdown pgbouncer, disconnecting all clients instantly.
4. Use slonik to apply one or more schema patches.
5. Start pgbouncer back up again.

The code for this (call it FDTv1) is in the Launchpad source code history its pretty entangled but its there for grabbing if you need it. Read on to see why its only available in the history The result was wonderful we immediately were able to deploy schema changes with <= 90 seconds of downtime, which was significantly less than the 5 minutes our stakeholders had agreed to as a benchmark if we were under 5 minutes, we could schedule downtime once a day rather than once a month. We had to fix some API client code to retry more reliably, and likewise fix a few minor bugs in the database connection handling logic in the appservers, but all in all it was a pretty smooth project. Along the way we spun off a small python helper to run and control pgbouncer, which let us write effective tests for the connection handling code paths. In This gave us the following workflow for making schema changes:

1. Land and deploy an incremental schema change.
2. Land and deploy any indices that need to be added these are deployed live using CREATE INDEX CONCURRENTLY.
3. Land and deploy code changes to populate any additional fields/tables from both application servers, and from cron we do a bulk backfill that does many small transactions while walking over the entire dataset that needs to be updated / populated.
4. Land and deploy code changes to drop references to the old schema, whatever it was.
5. Land and deploy an incremental schema change to finalise the change such as making a new column NOT NULL once the backfill is complete.

This looks long and unwieldy but its worth noting that its actually just repeated applications of a smaller primitive:

1. Make a schema change that is fast and compatible with existing code.
2. Change code to take advantage of the changed schema

Pretty much any change that is desired can be done using this single primitive. We wanted to go further though the multiple stages required for complex migrations became a burden with one change a day. Fortunately PostgreSQL now includes its own replication engine, which replicates the WAL logs rather than installing triggers on all tables like Slony. Stuart, our intrepid DBA migrated Launchpad to PostreSQL 9.1, updated the FDT tool to work with native replication, and migrated Launchpad off of Slony. The result is again wonderful the overhead in doing a schema patch, with all the protection I described above, is now ~5 seconds. We can do incremental changes in less time than it takes your browser to figure out that a given server is offline. We re now negotiating with the Launchpad stakeholders to get multiple downtime windows each day, with this almost unnoticable, super reliable process in place. Reliability wise, FDT has been superb. We ve had 2 failures: one where we believe we encountered a bug in Slony when dropping two tables at once, and one where we landed a patch that worked on staging but led to lock contention in production so the patch applied, but the system was very unhealthy after that until we fixed it. Thats after doing approximately 60 patches over a 1 year period. We re partway through extracting the patching logic from Launchpad s code base into a reusable tool, but the basic principles will apply to any PostgreSQL environment.

It s been a hot year in Kansas this year. Really hot. Our average high for July was 101F / 38C. It s also been extremely dry. So we haven t had too many pleasant opportunities to enjoy a bit of an upcycling project I had with the boys. When we renovated our old farmhouse, we had two chimneys removed. The bricks were saved in a large pile out back, and we haven t really touched them in the last 5 years. I got the idea at some point that it would be nice to have a fire ring on our yard. The boys love campfire-style cooking, and enjoy helping gather kindling and watching the fire grow. I had looked at fire rings in stores, but just couldn t bring myself to pay $60 or $100 or even more for what was really a piece of round metal. I decided we would find a way to build our own fire ring. So the idea of chimney bricks seemed perfect. Some of these bricks still have mortar on them, so the result is imperfect, but it is functional. More importantly, the boys helped. They picked out bricks one at a time, set them in the wheelbarrow (or even carried a few themselves, as Jacob insisted on doing sometimes.) Then we d dump them out on the ground, and I d make some attempt at making the thing round, while the boys would put them on the pile. We did this over the course of several evenings, with me filling in on some of it after they lost interest. When we got it done, they of course loved cooking outside. I made sure that we placed it in a place that will be in the shade every summer evening so we d be comfortable. I made no attempt to mortar it in; this way, it s easy to move or resize. And it s safer for the boys than a metal one, since the outer edge never even gets warm to the touch. Anyhow, it finally got a little cooler last week, so we cooked out there for dinner two days in a row. One day, after eating, the boys came back out to help put out the fire with the hose. After that, Jacob and I went out there to eat dessert. He sat on the grass, and I sat down next to him. He scooted over a bit to be closer to me. Pretty soon, Oliver came running out too, and sat on my lap. The three of us just sat there on the grass, eating our desserts and enjoying the evening. It s the kind of moment that makes a dad happy. The other evening, they again helped me put out the fire with a hose. They d been active that day, so after I finished hosing down the fire ring, I gave them each a small spray with the hose. After a brief flicker of indecision, they both decided this was hilarious. Jacob took off running, yelling You ll never catch me! (And clearly hoping I would.) Oliver copied him, and so I proceeded to chase them around the yard with a hose for quite awhile. There was much laughter from them, and they wound up totally soaked and happy. Another good evening. You never know what will happen outdoors, but so often it is very good.

While many (german) IT magazines write about glusterfs since RedHat has taken it over, here is a short summarzy of the glusterfs Debian packaging status.

• Debian stable/Squeeze delivers 3.0.5-1 and I just updated it on squeeze-backports to 3.2.7
• Squeeze will come with 3.2.7, which is the latest release of the 3.2.x series. 3.3.0 was too new for me to upload it for Whezzy, also if it was released before Whezzy was frozen
• You also could get 3.3.0 from experimental. 3.3.1 should be out soon and I will try to keep the packages up to date

If you are interested, I am still seeking for help!

Just a few photos of my vacation on the Isla de Fuerteventura in july 2012 :) But travelling will be continued in 2012 to Belek in the Turkey on Oct+Nov!

Some of you may have noticed that I am not a concise author. Perhaps that has something to do with the fact that I am not a concise reader. I like facts, details, and lots of them. So as a recent Nexus 7 purchaser, here you go. Genesis I ve long used Android devices, and last year had a company-issued Motorola Xoom, which was the first Google Experience tablet with Honeycomb. That tablet has specs roughly similar to iPads; its 10.1 screen was the same, the 1280 800 screen was better than the iPad available at the time, and its 730g weight identical to the early iPads (though 10% higher than the current iPad). I lost access to it when I changed jobs, and had been without a tablet until recently. Other devices I own are the Galaxy Nexus, sporting a 4.65 screen; and what s now called the Kindle Keyboard, with an eInk screen. I had been somewhat interested in the Kindle Fire, but the closed nature and limited capability of the system kept me away. The Nexus 7 reviews, however, were stunning, as was the price. $200 for a great tablet. I wound up buying the $250 16GB model. But not until after I spent a great deal of time thinking about size. Physical Size My main concern was that the Nexus 7 would be too small to be useful. I had never been particularly pleased with my input speed on the Xoom. I tried to touch type on it, but was just never fast enough to surpass frustratingly slow. I have long been a fast and accurate typist on keyboard; well over 100 words per minute, and it is frustrating when my fingers can t maintain that speed. I figured the situation would be even worse on the Nexus 7, given its smaller size. I also found the Xoom to sometimes feel a little small with the 10 screen, and was concerned about that as well. And finally, 7 doesn t sound all that much larger than 4.65 . However, having actually had the Nexus 7 for a little while now, I m very pleased with the size, and may even prefer it. The 10 tablets are just too big and heavy to comfortably hold in one hand, and I ve realized that part of my Xoom frustration was the fact that I had to set it down and prop it up for anything beyond very brief use. At 340g, the Nexus 7 is less than half the weight of the Xoom or iPad, and it makes a huge difference. While still nowhere near where I d be with a keyboard, two-thumb typing in portrait mode, or even something approaching touch typing in landscape mode, is possible on the Nexus 7. The screen size hasn t been a bother, at all. This may be due to the fact that it s higher resolution (it s 1280 800 like the Xoom, but those pixels are crammed into only 7 ). I think it s also partly due to the fact that the browser in Jelly Bean is significantly better than the one in Honeycomb, and perhaps that websites are better at tablet-friendliness, too. Overall, the Nexus 7 feels a lot farther from the size of a laptop than did the Xoom, and as such is more prone to come with me in lots of situations, I think. It works reasonably well with foldable Bluetooth keyboards, so when thinking about a laptop replacement or alternative, that might be the way I go. A Bluetooth mouse also works with it, though I found it didn t provide near the utility that a BT keyboard does. Display The display is both amazing and disappointing. Browse some photos and some of them will show up in eye-popping clarity. Websites display fine. But the screen can also take on a washed-out appearance at times. I am notoriously picky in my displays, and this bothered me enough that I researched it. Analysis has shown that poor firmware calibration has lead to the compression of highlights, which mirrors what I was seeing. I am mostly used to it by now, but it s a disappointment. Most of the time, though, the screen is excellent. In comparison to my eInk Kindle, however, I don t think any tablet will ever be as good for book reading. The eInk screen truly is easier on the eyes, and the reflection of overhead lights on the Nexus 7 display can be distracting at first. I have had occasional issues with it not registering touches properly. This is always cleared up by touching the power button to put the unit to sleep, then waking it back up. Other Hardware There are three hardware buttons: power and volume up/down. Physically, the device fits my hand well, though I might wish it was a little lighter like my Kindle. Charging is accomplished via high-power 2A micro-USB, and there is, of course, a headphone port. There is no alert LED like my Galaxy Nexus has, and no vibration feature. The speaker is on the back, and the microphones along the left side a position which, it appears, many Nexus 7 cases are blocking. Battery Life I am astonished at how good this device is battery-wise, especially compared to the battery disaster that is the Galaxy Nexus. Google claims the Nexus 7 can survive 8 hours of solid screen-on use, and I don t doubt it. Mine s never gotten low enough to get a solid measurement. Wifi The wifi works well, as far as it goes. The wifi doesn t support 802.11n in 5GHz, which although somewhat common for devices like this, is a bit of a disappointment. Software The big story about the Nexus 7 is Jelly Bean. I had used Honeycomb on the Xoom, and Ice Cream Sandwich on my Galaxy Nexus, so I m familiar with its predecessors. Let s take a look. Project Butter Much has been made of Project Butter, Google s attempt to optimize Android to improve its responsiveness and the smoothness of things like scrolling. I can say they have done quite well. This device is so smooth you don t notice how smooth it is. It wasn t until I had been using it for a bit that I really noticed. That s a job well-done. Chrome The browser in Jelly Bean is now called Chrome. I am not sure if this is just marketing or not. It doesn t really feel all that different from previous versions of the Android browser, and the changes have been along the lines of incremental changes Google has introduced before. One of the very best new features happens when you touch a link that is close to other links on a page. Rather than getting a pretty much random page, Chrome pops up a partial-screen zoom box showing the part of the page near where your finger touched. With everything showing up huge, it is now easy to touch the precise link you want. Do so, and the box goes away, and your page loads. I am amazed at how much improvement this one change brings. Compared to ICS Browser, bookmarks can be brought up quicker, and the tab interface is nicer. All is not perfect in the land of Chrome, however. It contains several regressions from the Ice Cream Sandwich browser. I have two complaints about bookmarks. One is that previous versions of the browser would show thumbnails of sites in the bookmark viewer. This was a nice navigation aid. Chrome shows only favorites icons, if one is available, or a generic icon if not. Also, the bookmarks synced with other Android devices are called, confusingly enough, Desktop Bookmarks now, and require an extra tap to access. I have had occasional trouble with Chrome not wanting to prompt for credentials for servers on my LAN that use HTTP auth. Chrome has also removed the ICS browser s ability to save a page, including all its elements, for offline viewing. Good for things like an airline checkin screen and such. I have no idea why Chrome removed this. I installed the Firefox Beta for Android, which also doesn t have the offline save feature, but it does have a save to PDF feature. Soft Keyboard The on-screen soft keyboard in Jelly Bean is a significant regression from previous versions of Android. My biggest complaint is the lack of visual feedback for keypresses. On earlier versions of Android, when you push a key, you ll see an image of it pop up on the screen, offset a little from the location of the key itself. In JB, all that happens is that the key itself changes colors. Not very helpful, because it is under your finger at the time. This small thing frustrates me to no end. The keyboard in ICS introduced some nice features as well, mainly long-presses as shortcuts to other features. For instance, you can long-press a key on the top row of letters to get numerals without having to switch to the number mode. Similarly, long press the period and you get other common punctuation. The JB keyboard removed both of those features. Thankfully, in the Market, there is an app called Ice Cream Sandwich Keyboard. It appears geared towards people running earlier versions of Android. Sadly, it is also a step up over what we have in JB. Google Now and Voice Recognition The other main headline feature in Jelly Bean is Google Now. The somewhat-competitor to Apple s Siri, Google Now takes a bit of a different approach than Apple. It is said that Siri is better than Google Now at responding to queries, but Google Now is better at predicting what you want to know before you ever ask. I haven t ever used Siri, but I would buy that explanation. Google Now is available with a swipe up from the bottom of the screen, or with a single touch from any Home screen. Bring it up and it shows you current information about what it thinks you need to know. Examples include weather and forecast information, time to get to home or work from your current location, alerts that you need to leave soon to get to a certain place on time, flight schedules, sports scores, etc. Google Now has been mostly a gimmick to me, but that may be because I fall outside its target demographic in significant ways. I live nowhere near a public transportation system, work at home for the most part, haven t flown wince I ve had the Nexus 7, don t follow sports, and already know how long it takes to get places (and when it varies, it s because of muddy roads or harvest neither things that traffic services know about.) The weather widget always seems to show the temperature from a couple of hours ago. It does show the weather in your current location. Well, mostly. I was in Newton, KS one day. I tapped on the icon for more detail. That simply took me to a Google search for weather Newton . Which showed me the weather for Newton, Massachusetts 1600 miles away. Fail. Speech recognition in JB is definitely improved. It is somewhat useful with Google Now. I like being able to simply say set alarm for 30 minutes. And it does it a lot quicker than I could in the interface. It s supposed to be able to let me bring up my contacts in the same way, but it is much more likely to try turning such an attempt into a Google search than an actual display of a contact. It s picky on the precise language used for setting an alarm too; say it slightly differently, and it s another Google search. JB also supports limited offline speech recognition. I say limited because it s a bit strange. I have, for instance, a Remember the Milk widget on my home screen. It has a microphone icon to use to speak a new reminder. Tap it, and you can t use it offline. It also has a button that brings up the on-screen keyboard. Do that, then touch the microphone on the keyboard, and you can use offline recognition. I have no idea how to explain this difference, since both are clearly using Google s engine. The speech recognition is indeed better, and might make it suitable for use instead of a keyboard for composing short texts and such. But it rarely produces even a sentence that I don t have to correct in some way, even now. Conclusion Despite some of its shortcomings, I am very fond of the Nexus 7. It is an excellent device. And at $200-$250, it is an AMAZING device. I am truly impressed with it, and don t regret my purchase at all.

Debian Wheezy will come with otrs 3.1.7, I also backported one important upstream patch from 3.1.8 to it. Also you can get now 3.1.8+dfsg1-1 from experimental :) With the 3.1.7 packaging I also migrated our otrs instance from 2.4.9 to 3.1.7 a few weeks ago, with some small problems:

• We are using Apache as webserver for otrs, which requires the libapache2-reload-perl package. It is just a recommends, because if you run otrs on another webserver, it is not required. So this package was missing after the upgrade and Apache silently died. But having a look at the error.log mod_perl complains about the missing module.
• We are using MySQL as database backend. With MySQL 5.5 Oracle changed the default storage engine from MyISAM to InnoDB, which is a good choice, but OTRS never used the ENGINE option in their installation and upgrade MySQL scripts. So if you upgrade an existing installation, some ALTER TABLE would fail, because of some FK fuck up between MyISAM and InnoDB tables. That was fixed with 3.1.5+dfsg1-2 by adding some default storage engine foo, but our remote MySQL cluster did not knew this option, yet, so I also had to patch out my patches on upgrading :(
  I am already interested in better fixes for the packaging!
• Printing/Exporting tickets resulted in an Internal server error , fixed with 3.1.7+dfsg1-2 by adjusting the font paths. The Perl module has dropped the embedded font copies.

Now after five years of working with otrs 2.x in my company, all our agents accept the new 3.1 one and they are happy with it :-) In the next weeks I also will check a backport of the 3.1.7 release for Squeeze.

Debian packages commonly use or dependencies of the form a b to mean that a or b should be installed, while preferring option a over b. In general, for resolving an or dependency, we will try all options from the left to the right, preferring the left-most option. We also prefer real packages over virtual ones. If one of the alternatives is already installed we use that.

def solve_or(or):
  best_real = None
  best_virtual = None
  for dep in or:
     for target in dep:
        if target.name == dep.name and best_real is None:
           best_real = target
        if target.name != dep.name and best_virtual is None:
           best_virtual = target        
        if target.is_installed():
          return target
  return best_real if best_real is not None else best_virtual

Now, this way of solving dependencies is slightly problematic. Let us consider a package that depends on: a b, b. APT will likely choose to install a to satisfy the first dependency and b to satisfy the second. I currently have draft code around for a future version of APT that will cause it to later on revert unneeded changes, which means that APT will then only install b . This result closely matches the CUDF solvers and cupt s solver. On the topic of solving algorithms, we also have the problem that optimizing solvers like the ones used with apt-cudf do not respect the order of dependencies, rather choosing to minimise the number of packages installed. This causes such a solver to often do stuff like selecting an sqlite database as backend for some service rather then a larger SQL server, as that installs fewer packages. To make such solvers aware of the implicit preferences, we can introduce a new type of dependency category: Weak conflicts, also known as Recommends-Not. If a package P defines a Recommends-Not dependency against a package Q, then this means that Q should not be installed if P is installed. Now, if we have a dependency like: Depends: a b c we can encode this as: Recommends-Not: c, c, b Causing the solver to prefer a, then b, and then c. This should be representable as a pseudo-boolean optimization problem, as is common for the dependency problem, although I have not looked at that yet it should work by taking the standard representation of conflicts, adding a relaxation variable and then minimising [or maximising] the number of relaxation variables.
Filed under: APT2, Debian, General

Review: Design Patterns, by Erich Gamma, et al.

Author:	Erich Gamma
Author:	Richard Helm
Author:	Ralph Johnson
Author:	John Vlissides
Publisher:	Addison-Wesley
Copyright:	1995
Printing:	September 1999
ISBN:	0-201-63361-2
Format:	Hardcover
Pages:	374

Design Patterns: Elements of Reusable Object-Oriented Software by the so-called "Gang of Four" (Gamma, Helm, Johnson, and Vlissides) is one of the best-known books ever written about software design, and one of the most widely cited. The language introduced here, including the names of specific design patterns, is still in widespread use in the software field, particularly with object-oriented languages. I've had a copy for years, on the grounds that it's one of those books one should have a copy of, but only recently got around to reading it. The goal of this book is to identify patterns of design that are widely used, and widely useful, for designing object-oriented software. It's specific to the object-oriented model; while some of the patterns could be repurposed for writing OO-style programs in non-OO languages, they are about inheritance, encapsulation, and data hiding and make deep use the facilities of object-oriented design. The patterns are very general, aiming for a description that's more general than any specific domain. They're also high-level, describing techniques and methods for constructing a software system, not algorithms. You couldn't encapsulate the ideas here in a library and just use them; they're ideas about program structure that could be applied to any program with the relevant problem. With the benefit of seventeen years of hindsight, I think the primary impact of this book has been on communication within the field. The ideas in here are not new to this book. Every pattern in Design Patterns was already in use in the industry before it was published; the goal was taxonomy, not innovation. One would not come to Design Patterns to learn how to program, although most introductory texts on object-oriented programming now borrow much of the pattern terminology. Rather, Design Patterns is as influential as it is because it introduced a shared terminology and a rigor around that terminology, allowing writers and programmers to put a name to specific program structures and thus talk about them more clearly. This also allows one to take a step back and see a particular structure in multiple programs, compare and contrast how it's used, and draw some general conclusions about where it would be useful. I have the feeling that the authors originally hoped the book would serve as a toolbox, but I think it's instead become more of a dictionary. The pattern names standardized here are widely used even by people who have never read this book, but I doubt many people regularly turn to this book for ideas for how to structure programs. Design Patterns is divided into two parts: a general introduction to and definition of a software pattern followed by a case study, and then a catalog of patterns. The catalog is divided into creational patterns (patterns for creating objects), structural patterns (patterns for composing objects into larger structures), and behavioral patterns (patterns for interactions between objects). Each pattern in turn follows a very rigid presentation structure consisting of the pattern name and high-level classification, its basic intent, other common names, a scenario that motivates the pattern, comments on the applicability of the pattern, the structure and classes or objects that make up the pattern, how those participants collaborate, how the pattern achieves its goals, comments on implementation issues, sample code, known uses of the pattern in real-world software, and related patterns. As with a dictionary, the authors go to great lengths to keep the structure, terminology, and graphical representations uniform throughout, and the cross-referencing is comprehensive (to the point of mild redundancy). As for the patterns themselves, their success, both as terminology and as useful design elements, varies. Some have become part of the core lexicon of object-oriented programming (Factory Method, Builder, Singleton), sometimes to the point of becoming syntactic elements in modern OO languages (Iterator). These are terms that working programmers use daily. Others aren't quite as widespread, but are immediately recognizable as part of the core toolkit of object-oriented programming (Adapter, Decorator, Proxy, Observer, Strategy, Template Method). In some cases, the technique remains widespread, but the name hasn't caught on (Command, for example, which will be immediately familiar but which I rarely hear called by that name outside of specific uses inside UI toolkits due to ambiguity of terms). Other patterns are abstract enough that it felt like a bit of a reach to assign a name to them (Bridge, Composite, Facade), and I don't think use of those names is common, but the entries are still useful for definitional clarity and for comparing similar approaches with different implications. Only one pattern (Interpreter) struck me as insufficiently generic to warrant recording in a catalog of this type. So far, so good, but the obvious question arises: if you've not already read this book, should you read it? I think the answer to that is debatable. The largest problem with Design Patterns is that it's old. It came late enough in the development of object-oriented programming that it does capture much of the foundation, but OO design has continued to change and grow, and some patterns have either been developed subsequently or have become far more important. For example, Model-View-Controller is conspicuous by its absence, mentioned only in passing in the discussion of the Observer pattern. Any pattern catalog written today would have an extensive discussion. Similarly absent are Inversion of Control and Data Access Object, which are much more central to the day-to-day world of the modern programmer than, say, Memento or Visitor. One could easily go on: Lazy Initialization, Mock Object, Null Object... everyone will have their own list. A related problem is that all the implementation examples are shown in either C++ or Smalltalk (occasionally both). Those were probably the best languages to use at the time, but it's doubtful a modern rewrite would choose them. Smalltalk, in particular, I found nearly incomprehensible for the uninitiated, to the point where I ignored the code and only read the surrounding English description. C++ fares better, but Design Patterns occasionally drifts off into tedious discussions of how to work around C++'s limitations in ways that are irrelevant to the pattern itself and would not be necessary in, say, Java or Python. (This is ameliorated by the fact that C++, unlike Smalltalk, is still in widespread use, so those discussions remain moderately helpful for some readers.) Design Patterns is not, therefore, a very good source for a working knowledge of the most common patterns in use today. It has also become somewhat obsolete via its own success: the concept of a design pattern has become so popular that nearly all introductory texts include at least a basic discussion of design patterns and an introduction to the most notable and useful patterns. I think that's a more comfortable and more efficient way to pick up the basics than reading through this book, which is somewhat dense and which expects from the reader a reasonably good working knowledge of object-oriented programming. And, once you have the basics, MVC, DAO, and similar design patterns are probably more important than the more subtle design patterns presented here. That said, I think the rigor of description and the comparisons and discussions here still have some value. Design Patterns encourages the reader to look at patterns from a higher-level perspective, to think about meta-patterns such as the balance between data hiding and access, or between structure designed for the present purpose and structure that's adaptable to future needs. It's also mildly interesting from a historical standpoint; one can see the inspiration for future language designers in how problems are described here, and see how many of the implementation issues and negative consequences have been corrected or simplified by richer language designs. Overall, I would hesitate to recommend buying this book today, particularly at new textbook prices. But if you're a working object-oriented software designer or programmer, I think it's worth checking out from a library (and, thanks to its influence, any library with a decent software design section will almost certainly have a copy). Read the overall discussion, skim the catalog, and read the discussion of the patterns that strike your interest. It may help provide some additional structure and perspective to how you think about OO design. Rating: 6 out of 10

In docbook one can specify the language used at the top, and the processing pipeline will use this information to pick the correct translations for 'chapter', 'see also', 'index' etc. And for most languages used with docbook, I guess this work just fine. For example a German user can start the document with <book lang="de">, and the document will show up with the correct content with any of the docbook processors. This is not the case for the language I am working with at the moment, Norwegian Bokm l. For a while, I was confused about which language code to use, because I was unable to find any language code that would work across all tools. I am currently testing dblatex, xmlto, docbook-xsl, and dbtoepub, and they do not handle Norwegian Bokm l the same way. Some of them do not handle it at all. A bit of background information is probably needed to understand this mess. Norwegian is not one, but two written variants. The variants are Norwegian Nynorsk and Norwegian Bokm l. There are three two letter language codes associated with these languages, Norwegian is 'no', Norwegian Nynorsk is 'nn' and Norwegian Bokm l is 'nb'. Historically the 'no' language code was used for Norwegian Bokm l, but many years ago this was found to be bad idea, and the recommendation is to use the most specific language code instead, to avoid confusion. In the transition period it is a good idea to make sure 'no' was an alias for 'nb'. Back to docbook processing tools in Debian. The dblatex tool only understand 'nn'. There are translations for 'no', but not 'nb' (BTS #684391), but due to a bug (BTS #682936) the 'no' language code is not recognised. The docbook-xsl tool chain only recognise 'nn' and 'nb', but not 'no'. The xmlto tool only recognise 'nn' and 'nb', but not 'no'. The end result that there is no language code I can use to get the docbook file working with all of these tools at the same time. :( The correct solution is to use <book lang="nb">, but it will take time before that will work with all the free software docbook processors. :( Oh, the joy of well integrated tools. :/

It s been reported in the mainstream media that the Australian Data Retention Scheme has been referred to another committee, effectively delaying it to after the next election. The Scheme was part of a broad review of the various security agencies powers and how to streamline them. The initial paper originally stated that retention would be for up to 2 years for parts of the data set, without really specifying what the data was. There has been various ideas of what this data is, depending who you ask and when. Some documents state it is only the accounting data; that user X used IP address Y at time Z, while others state it is email logs and a third have been the previous two plus web logs. There are many problems with a scheme such as this. 2 years of everyones web-browsing history is a desirable target for both legitimate and illegitimate access to that data. Leaving aside would the access to the agencies get the right level of access; who else would want this data? I m sure that AFACT (Australia s MPAA effectively) would like this data to go trawling. The recent exposure of AAPT s data by Anonymous shows that there may be some other means of obtaining this data. I actually wonder, if the scheme was in, what they would find? Take email for example; my email leaves my computer and directly communicates with the destination mail server. If someone pulled my email records they d show very little information. If it is too hard to run your own mailserver, I m sure the enterprising no-gooders that they are so interested in finding can work out how to buy a remote mailserver somewhere. If that s too hard, there s always gmail with https, or VPNs, or TOR, or.. well there are plenty of options. It probably comes down to how resourceful (if thats the word) people are. By the way, mentioning https reminded me of the great little plug-in called HTTPS Everywhere by EFF. I m glad to see that this legislation is stalled. To me the paperwork I saw appeared to be saying that they got some bad guys with what powers they have so they will catch more with more powers; therefore more powers are good, OK? It s a rather simplistic view of the world. My only worry is its not stopped, just stalled so it might be re-animated in future.

Related articles

• Anonymous begins dump of stolen ISP data (zdnet.com)
• AAPT confirms data breach as Anonymous claims attack (zdnet.com)
• Roxon stalls web surveillance plans (theage.com.au)
• Anonymous may help, not hinder, data retention laws (zdnet.com)
• Anonymous Dumps Australian Telco Data Online (it.slashdot.org)
• The asymmetry implicit in Internet data retention (go.theregister.com)
• Australian Government Moves to Expand Surveillance Powers (eff.org)

Last week I helped organize the Open and User Innovation Conference at Harvard Business School. One of many interesting papers presented there was an essay on Institutional Change and Information Production by Fabio Landini from the University of Siena. At the core of the paper is an economic model of the relationship between rights protection and technologies that affects the way that cognitive labor can be divided and aggregated. Although that may sound very abstract (and it is in the paper), it is basically a theory that tries to explain the growth of free software. The old story about free software and free culture (at least among economists and many other academics) is that the movements surged to prominence over the last decade because improvements in communication technology made new forms of mass-collaboration -- like GNU/Linux and Wikipedia -- possible. "Possible", for these types of models, usually means profit-maximizing for rational, profit-seeking, actors like capitalist firms. You can basically think of these attempts as trying to explain why open source claims that free licensing leads to "better quality, higher reliability, more flexibility, lower cost" are correct: new technology makes possible an open development process which leads to collaboration which leads to higher quality work which leads to profit. Landini suggests there are problems with this story. One problem is that it treats technology as being taken for granted and technological changes as effectively being dropped in from outside (i.e., exogenous). Landini points out that software businesses build an enormous amount of technology to help organize their work and to help themselves succeed in what they see as their ideal property rights regime. The key feature of Landini's alternate model is that it considers this possibility. What comes out the other end of the model is a prediction for a multiple equilibrium system -- a situation where there are several strategies that can be stable and profitable. This can help explain why, although free software has succeeded in some areas, its success has hardly been total and usually has not led to change within existing proprietary software firms. After all, there are still plenty of companies selling proprietary software. In Landini's model, free is just one of several winning options. But Landini's model raises what might be an even bigger question. If free software can be as efficient as proprietary software, how would anybody ever find out? If all the successful software companies out there are doing proprietary software, which greedy capitalist is going to take the risk of seeing if they could also be successful by throwing exclusive rights out the window? In the early days, new paths are always unclear, unsure, and unproven. Landini suggests that ethically motivated free software hackers provide what he calls a "cultural subsidy." Essentially, a few hackers are motivated enough by the ethical principles behind free software that they are willing to contribute to it even when it isn't clearly better than proprietary alternatives. And in fact, historically speaking, many free software hackers were willing to contribute to free software even when they thought it was likely less profitable than the proprietary alternative models. As Landini suggests, this group was able to build technological platforms and find new social and business arrangements where the free model actually is competitive. I think that the idea of an "cultural subsidy" is a nice way to think about the important role that ethical arguments play in movements like free software and free culture. "Open source" style efficiency arguments persuade a lot of people. Especially when they are true. But those arguments are only ever true because a group of ethically motivated people fought to find a way to make them true. Free software didn't start out as competitive with proprietary software. It became so only because a bunch of ethically motivated hackers were willing to "subsidize" the movement with their with their failed, and successful, attempts at free software and free culture projects and businesses. Of course, the folks attracted by "open source" style superiority arguments can find the ethical motivated folks shrill, off-putting, and annoying. The ethically motivated folks often think the "efficiency" group is shortsighted and mercenary. But as awkward as this marriage might be, it has some huge upsides. In Landini's model, the ethical folks can build their better world without convincing everyone else that they are right and by relying, at least in part, on the self-interest of others who don't share their principles. Just as the free software movement has done. I think that Landini's paper is a good description of the critically important role that the free software movement, and the FSF in particular, can play. The influence and importance of individuals motivated by principles can go far beyond the groups of people who take an ethical stand. They can make involvement possible for large groups of people who do not think that taking a stand on a particular ethical issue is even a good idea.

Recently, I've started converting many of my subversion repositories to git, some of which contain fairly large files (2-3G). However, git can be slow to deal with repositories with large files, and it also isn't able to selectively discard unneeded files when disk space is pressing. Thankfully, git-annex resolves most of these problems with git, but the process required to use git-annex on a converted subversion repository is slightly complicated. Basic conversion of svn to git The basic conversion of svn to git is done using git-svn:

 git svn clone file:///srv/svn/foo --no-metadata -A authors.txt -T trunk foo

where /srv/svn/foo is the subversion repository, authors.txt is a list of login = Full Name <email@example.com> pairs matching each of the subversion commit authors, and foo is the git repository to create. git-svn has a ton of useful options, but the basic invocation above is all I'm concerned with. Migrating large files from git into git-annex In order to migrate from git to a git+git-annex setup, we'll have to walk the entire commit history, and edit each commit to instead store large files in git-annex, replacing the large file with a symlink, and finally eliminate all of the references to the old large objects, and do garbage collection. Because we may have the same file move around, we're going to use the git-annex SHA1 backend instead of the default WORM backend which is based on filename and size, and init git-annex.

  cd foo; echo '* annex.backend=SHA1' > .git/info/attributes
  git annex init

Then, we're going to filter out the large files using git filter-branch. To do that, we'll first, we'll create a little helper script git_annex_add.sh, which will remove the file from the git repository, add to git annex, and fix up the symlinks:

 #!/bin/bash
 f="$1";
 git rm --cached "$ f ";
 git annex add "$ f ";
 annexdest="$(/bin/readlink -v $ f )";
 ln -sf "$ annexdest#../../ " "$ f ";
 echo -n "Added: "
 ls -l "$ f ";

Then we will run filter-branch, and annex all files larger than 5 megabytes. [Tweak the find command if you want to do something different.]

 git filter-branch  --tag-name-filter cat --tree-filter \
'find . -ipath \*.git\* -prune -o -path \*.temp\* -prune -o -size +5M -type f -print0 xargs -0 -r -n1 ~/git_annex_add.sh;
 git reset HEAD .git-rewrite; :' -- master

This operation will take a while. [It would be better to do this during the initial svn git conversion, but since that requires more knowledge of git-svn, svn, git, and git-annex internals than I have, and I only have to do this once for each repository, it's not worth my time.] Now we have successfully switched everything to using git-annex, and we need to clean out the old references to the files:

 rm .git/svn -rf;
 rm -rf .git/refs/original .git/refs/remote/trunk .git/refs/remote/git-svn;
 git reflog expire --expire=now --all
 git gc --prune=now
 git gc --prune=now --aggressive

(I'm not sure if the last two commands need to be separate; I'm cargo culting a bit there.) Storing all git-annex files in a remote repository Because git-annex allows you to easily throw away files which are no longer referred to by the tip of any branch using git annex unneeded (and because I'd like all of the files on my central remote repository), I'm going to shove all of the git annex files into the remote bare repository. Normally, you would use git annex copy --to=remote; to do this, but because that only copies needed files, not everything, we'll have to do it manually. First, create the remote repository:

 git init --bare /srv/git/foo.git
 cd /srv/git/foo.git; git annex init foo.example.com

Add the remote to the local repository, push to the remote, and sync the objects and sync the annex:

 git remote add origin ssh://foo.example.com/srv/git/foo.git
 git push origin master
 rsync -avP .git/annex/objects ssh://foo.example.com/srv/git/foo.git/annex/.;
 git annex sync

Finally, on the remote, run git annex fsck to clean up the links to the imported objects:

 cd /srv/git/foo.git; git annex fsck;

Unresolved issues I don't know if the above works properly for branches. I suspect that it does not. I also have not exhaustively tested this methodology to verify that all of the history is present in every case. But hopefully this post (or some modification of it) will be helpful to you. Credit Many of the methodologies described here I originally found in tyger's git-annex forum post, the git gc stuff came from random google searches about shrinking git repositories, and the rsync suggestion came from joeyh (author of git-annex) and the other helpful denizens of #vcs-home on irc.oftc.net.

Most Debian source packages are using the 3.0 (quilt) format. This means that Debian changes to upstream files are managed in a quilt patch series. Knowledge of quilt is thus a must if you want to get involved in some serious packaging work. Don t worry, this tutorial will teach you how to use quilt in the context of Debian packaging. Pre-requisites Install the packaging-dev package to get a decent set of packages to do Debian packaging. It includes the quilt package.

$ sudo apt-get install packaging-dev

What s quilt? Read the description of the Debian package:

Quilt manages a series of patches by keeping track of the changes each of them makes. They are logically organized as a stack, and you can apply, un-apply, update them easily by traveling into the stack (push/pop). Quilt is good for managing additional patches applied to a package received as a tarball or maintained in another version control system. The stacked organization is proven to be efficient for the management of very large patch sets (more than hundred patches). As matter of fact, it was designed by and for Linux kernel hackers (Andrew Morton, from the -mm branch, is the original author), and its main use by the current upstream maintainer is to manage the (hundreds of) patches against the kernel made for the SUSE distribution.

The various files involved To better understand the various quilt commands, you should have a basic idea of how the tool works. The stack of patches is maintained in a dedicated directory ( patches by default, but in Debian packages we override this value to debian/patches ). This directory contains the patch files and a series file that gives an ordered list of patches to apply. Example:

$ ls debian/patches/
01_use_stdlib_htmlparser_when_possible.diff
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
06_use_debian_geoip_database_as_default.diff
series
$ cat debian/patches/series
01_use_stdlib_htmlparser_when_possible.diff
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
06_use_debian_geoip_database_as_default.diff

When quilt is used, it also maintains some internal files in a directory of its own (it s named .pc ). This directory is used to know what patches are currently applied (.pc/applied-patches) and to keep backup copies of files modified by the various patches. Configuring quilt Before going further, you should put this in your ~/.quiltrc file:

for where in ./ ../ ../../ ../../../ ../../../../ ../../../../../; do
    if [ -e $ where debian/rules -a -d $ where debian/patches ]; then
        export QUILT_PATCHES=debian/patches
        break
    fi
done

This ensures that quilt will always use debian/patches instead of patches when your current directory is within a Debian source package where debian/patches exists. If you only use quilt for debian packaging, then you can be more expedient and put a simple export QUILT_PATCHES=debian/patches in that file. As a matter of personal preferences, I also have those lines in my ~/.quiltrc:

QUILT_PUSH_ARGS="--color=auto"
QUILT_DIFF_ARGS="--no-timestamps --no-index -p ab --color=auto"
QUILT_REFRESH_ARGS="--no-timestamps --no-index -p ab"
QUILT_DIFF_OPTS='-p'

It enables syntax coloring for the output of several commands and customizes the generated patches to get rid of useless information. I recommend you to use those settings too. Applying and unapplying patches, navigating in the stack of patches When you already have a patch series, you can navigate in the stack of patches so that any subset of consecutive patches (starting from the bottom) can be applied. quilt series will list all patches known by quilt. You can apply all patches with quilt push -a or unapply them all with quilt pop -a. You can also verify what patches are applied (quilt applied) or unapplied (quilt unapplied). quilt push applies the next unapplied patch (i.e. the patch returned by quilt next) and quilt pop unapplies the last applied patch (i.e. the patch returned by quilt top). You can give a patch name as parameter to quilt push/pop and it will apply/unapply all the patches required until the given patch is on the top. Here are some examples of navigation in a quilt patch series

$ quilt series
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
06_use_debian_geoip_database_as_default.diff
$ quilt applied
No patches applied
$ quilt next
02_disable-sources-in-sphinxdoc.diff
$ quilt push
Applying patch 02_disable-sources-in-sphinxdoc.diff
patching file docs/conf.py
Now at patch 02_disable-sources-in-sphinxdoc.diff
$ quilt push
Applying patch 03_manpage.diff
patching file docs/man/django-admin.1
Now at patch 03_manpage.diff
$ quilt applied
02_disable-sources-in-sphinxdoc.diff
03_manpage.diff
$ quilt unapplied
06_use_debian_geoip_database_as_default.diff
$ quilt pop -a
Removing patch 03_manpage.diff
Restoring docs/man/django-admin.1
Removing patch 02_disable-sources-in-sphinxdoc.diff
Restoring docs/conf.py
No patches applied
$ quilt push 03_manpage.diff
Applying patch 02_disable-sources-in-sphinxdoc.diff
patching file docs/conf.py
Applying patch 03_manpage.diff
patching file docs/man/django-admin.1
Now at patch 03_manpage.diff
$ quilt top
03_manpage.diff

Creating a new patch If there s no quilt series yet, you want to create the debian/patches directory first. If you already have one, you need to decide where to insert the new patch. Quilt will always add the new patch just after the patch which is currently on top. So if you want to add the patch at the end of the series, you need to run quilt push -a first. Then you can use quilt new name-of-my-patch.diff to tell quilt to insert a new empty patch after the current topmost patch. In this operation quilt does almost nothing except updating the series file and recording the fact that the new patch is applied (even if still empty at this point!). Now to add changes in this patch, you re supposed to modify files but only after having informed quilt of your intent to modify those files. You do this with quilt add file-to-modify. At this point quilt will make a backup copy of that file so that it can generate the final patch when you re done with your changes. It s quite common to forget this step and to be unable to generate the patch afterward. That s why I recommend you to use quilt edit file-to-modify which is a shorthand for doing quilt add and then opening the file in your favorite text editor. If you want, you can review your work in progress with quilt diff. When you re done with the changes, you should call quilt refresh to generate the patch (or to update it if it was already existing). And since you re a good packager, you call quilt header --dep3 -e to add DEP-3 meta-information to your patch header. Importing an external patch If someone else already prepared a patch, you can just import it right away with quilt import /tmp/the-patch. If you want to import it under a better name you can use the option -P better-patch-name . Like quilt new, it inserts the patch after the topmost patch. Updating patches for a new upstream version With some luck, your patches will still apply with some offsets in line numbers (quilt displays those offsets) and sometimes with some fuzz:

$ quilt push
[...]
Hunk #1 succeeded at 1362 (offset 11 lines).
Hunk #2 succeeded at 1533 with fuzz 1 (offset 4 lines).
[...]

While offsets are nothing to worry about (it means some lines were added and/or removed before the patched part), fuzz means that patch had to ignore some context lines to find the place where to apply the changes. In that case, you need to double check that patch did the right thing because it might have made changes somewhere where it shouldn t. Also, you ll have to update those patches because dpkg-source doesn t accept any fuzz. If you re confident that all patches are correctly applied by quilt, you can refresh them to get rid of those warnings:

$ quilt pop -a
[...]
$ while quilt push; do quilt refresh; done

That was for the easy case. Now let s deal with the case where some of the patches no longer apply. There s one case that is usually nice to have:

$ quilt push
Applying patch 04_hyphen-manpage.diff
patching file docs/man/django-admin.1
Hunk #1 FAILED at 194.
1 out of 1 hunk FAILED -- rejects in file docs/man/django-admin.1
Patch 04_hyphen-manpage.diff can be reverse-applied

When the patch can be reverse-applied, it means that the upstream authors included the Debian patch (or that they made the same change even though you forgot to forward the patch). You can thus get rid of it:

$ quilt delete -r 04_hyphen-manpage.diff
Removed patch 04_hyphen-manpage.diff

Note that without the -r the patch is only dropped from the series file. With -r the patch file is also removed. But there s a less desirable case where the patch is still relevant but it doesn t apply any longer:

$ quilt push
Applying patch 01_disable_broken_test.diff
patching file tests/regressiontests/test_utils/tests.py
Hunk #1 FAILED at 422.
1 out of 1 hunk FAILED -- rejects in file tests/regressiontests/test_utils/tests.py
Patch 01_disable_broken_test.diff does not apply (enforce with -f)

In that case, you should follow quilt s advice to force the patch application, manually apply the parts of the patch that were rejected, and then refresh the patch.

$ quilt push -f
Applying patch 01_disable_broken_test.diff
patching file tests/regressiontests/test_utils/tests.py
Hunk #1 FAILED at 422.
1 out of 1 hunk FAILED -- saving rejects to file tests/regressiontests/test_utils/tests.py.rej
Applied patch 01_disable_broken_test.diff (forced; needs refresh)
$ vim tests/regressiontests/test_utils/tests.py    
$ quilt refresh
Refreshed patch 01_disable_broken_test.diff

Other quilt commands You should probably read quilt s manual page too to learn about the various other commands and options that exist. There s at least quilt rename new-name that you can also find useful to rename the topmost patch (you can use -P patch-to-rename to rename a patch which is not currently at the top). Feedback Please leave comments if you have suggestions of improvements, or if there are some tips that are good to know. I might incorporate them in this article. Feel free to share this article with newbie packagers which are struggling with quilt. For your convenience, you can also refer to this article with this URL: http://raphaelhertzog.com/go/quilt

Found it useful? Be sure to not miss other packaging tips (or lessons), click here to subscribe to my free newsletter and get new articles by email.

2 comments Liked this article? Click here. My blog is Flattr-enabled.