I was going to post this yesterday, but some might have thought that it was a joke. April 1st is always an annoying day to communicate real information. If you have been fond of the way Debian and Ubuntu manage multiple PostgreSQL versions in parallel, you can now have the same on OS X with Homebrew:

brew tap petere/postgresql
brew install postgresql-9.2
# etc.
brew install --HEAD postgresql-common

postgresql-common is the same code as in Debian, only mangled a little. Now you have all the client programs symlinked through pg_wrapper, and you can use the server management tools such as:

pg_createcluster 9.2 main
pg_ctlcluster 9.2 main start
pg_lsclusters

Let me know if you find this useful. Links:

• homebrew-postgresql
• postgresql-common homebrew branch

A few months late, perhaps, but I wanted to mention that my team (Codex) competed, once again, in the MIT Mystery Hunt. The prize for winning is the responsibility of writing the hunt next year. After being on the 2012 writing team I have mixed feelings about the fact that we did not win again. Although I did not walk away with another coin, I did manage to make an appearance in a multimedia story that the MIT Tech shot about the hunt which provides nice introductions to those who are not familiar with it. <iframe allowfullscreen="" frameborder="0" height="315" src="http://www.youtube.com/embed/R-tmBfZ8CwQ" width="560"></iframe> It was fun to reflect a little bit on why I find the hunt so fun. I said:

In my day job, I work on a lot of problems that maybe don t have answers. It s really awesome to work in a space where if you put in 1-5 hours of mental energy, you will have a solution. Someone has test-solved for it, you know that it s solvable; you know that there is an answer and that there was designed to be an answer. There s something really satisfying about being able to do that.

If you re looking for a team to hunt on next year, feel free get in contact codex is generally quite open to new members.

A proposal to solve gender imbalance in Debian We've done all we can so far: Debian Women, the Diversity Statement, the anti-harassment contact, gender neutral language, lots of education all round, but we still suffer from a strong gender imbalance in Debian. I think that the reason is that the majority group of cisgender men in the project, although they don't actively work /against/ the rest, still have /no incentive/ to be inclusive, and generally do not understand what bearing a female name online is like. I think it is about time we addressed that, and after a lot of thinking and discussing with many other concerned debianers, I think I have just the right proposal, which is twofold. The first part is this: since the goal is to have an equal gender perception in Debian, we can just decide to only approve one obviously-male-named DD for every obviously-female-named one. That's right: no new obviously-male-named DD unless an obviously-female-named DD has just been approved. It may sound like affirmative action gone wild, but please stop a moment to think about it: this would create precisely the right incentive for the currently dominant group of developers to be inclusive! People shouldn't just assume they can get a Debian account regardless of what happens around them. We already ask NM candidates to fix RC bugs and, well, gender imbalance should be treated as an RC bug, and everyone should feel compelled to join the effort to fix it! Now, of course since there currently are many male names in NM but not a single female name, it would not be reasonable to just stop the flow of new developers into the project: that would just have the effect to make us starve on personpower. So here is the second part of the proposal: the one-female-name-one-male-name policy will not be enforced for, say, a year. But during this year, everyone currently in NM or joining NM will be asked to adopt a female identity. Crazy? No, genius! It's about time people understood what it means to get advances in private every time they make a public contribution! What better way than just trying it out for themselves? On top of that, as more and more female names appear in Debian changelogs, fake or not, people will finally start to understand that it does not matter what name is used to sign the contribution, but the contribution itself. I don't know if this will give us a community where people realise female or male contributions are equally valuable, which is what I hope, or a community where people will think that everyone is a cisgender man even if they have female names. In the end, really, it does not matter. Either way, we finally get to have a community where everyone is /guaranteed/ to be treated the same. But gender imbalance isn't the only imbalance we have in Debian. People accrue a reputation over time, good or bad, and this reputation tends to stick on you for years, regardless of how you may change, for better or for worse. When we evaluate the merit a contribution, we should not be biased by the reputation of the contributor! How can new contributors be taken seriously otherwise? I believe we are loosing lots of fresh, good ideas this way. And how much damage could be wrecked on the project by a well-respected contributor, like a Debian Account Manager, who is having a funny day? I think we can address this just as I propose to address gender imbalance: let's swap identities from time to time, like it usually happens with nametags at the end of Debconfs. Let's see gregoa upload a patched versions of python3.2, and enrico upload a new upstream version of eglibc! See if we won't finally have some peer review at last! Reputation and real identities have many merits, but we have come to rely too much on them, and it is hurting us. It is time we did something about it, before it is too late!

Russ Allbery wrote an informative post about how to determine which charities are worth donating to [1]. He has a link to another article about the charities to which he donates and concentrates on ways of analysing the effectiveness of charities. So someone who has different ideas about which types of charity are worthy of donation could still learn a lot from his post. Adam Green wrote an interesting article for The New Yorker about Apollo Robbins who is one of the world s best pick-pockets [2]. Apollo picks pockets as a magician to entertain people and always returns what he steals. Now he is working with neuroscientists who are devising experiments to determine why his tricks work. Rick Falkvinge wrote an insightful article describing the way that the copyright monopoly is in direct opposition to the freedom to make contracts [3]. It s a good rebuttal of a common argument in favor of copyright law. Seth Godin gave an interesting TED talk about the problems with the education system, how and why it teaches conformity and little else [4]. One of his suggestions for improvement is to have students spend their evenings watching lectures by experts and class time asking questions. He also says that everything should be open book and that there is no value in memorising anything it s a bit of an overstatement but it s essentially correct. Cory Doctorow wrote an interesting article for The Guardian about positive externalities and copyright law [5]. I think that he didn t choose the best way of framing this issue, but he makes some very interesting points anyway. Andrew Norton wrote an interesting article about how to reduce corruption in the police force and other government agencies [6]. A large part of this is based on making them subject to the same laws as everyone else, which seems to be a radical idea. Valerie Aurora wrote an insightful blog post about suicide [7]. Emily Oster gave an interesting TED talk about the factors that determine the spread of AIDS in Africa [8]. It s quite different to what you probably expect.

• [1] http://www.eyrie.org/~eagle/journal/2012-12/006.html
• [2] http://www.newyorker.com/reporting/2013/01/07/130107fa_fact_green
• [3] http://tinyurl.com/apovrka
• [4] http://www.youtube.com/watch?v=sXpbONjV1Jc
• [5] http://tinyurl.com/a7rutft
• [6] http://tinyurl.com/a227fvg
• [7] http://tinyurl.com/a5zo6tn
• [8] http://tinyurl.com/3f7283s

Recently, I was wondering how many Debian Developers are actively working on RC bugs in some way or another in the time period of the last release (squeeze) to now (shortly? before wheezy). I therefore grabbed the mailing list archives of debian-bugs-dist@ from gmane, used only those messages whose X-Debian-PR-Message header matches an RC bug (list retrieved from UDD) and then attributed the message counts to the appropriate Debian Developer. I am sure that there are subtle mistakes in the data I retrieved and that there most likely is a better way to achieve the same results, but this is only to get a trend and should be good enough for that. So, it turns out that 514 different Debian Developers have sent messages regarding RC bugs since squeeze. That s about half of our 985 total active Debian Developers. I would love to show a good histogram of the actual message counts (not sure how well received showing the raw data is ), but I suck at visualizing such data in a compact way. Is there anyone familiar with R (or other free data visualization tools) and willing to help? I can send you the CSV file, just send me an email to stapelberg@.

Four years ago I started using Goodreads to maintain the list of books I've read (which had lived in a flat text file for a decade+ before that). Now it's been aquired by Amazon. I doubt it will survive in its current form for more than 2 years. Anyway, while Goodreads has been a quite good way to find what my friends are reading, I've been increasingly annoyed by the quality of its recommendations, and its paucity of other features I need. It really doesn't seem to help me keep up with new and interesting fiction at all, unless my friends happen to read it. So I looked at LibraryThing. Actually, I seem to have looked at it several times before, since it had accounts named "joey", "joeyh", and "joeyhess" that were all mine. Which is what happens to me on sites that lack Openid or Browserid. Digging a little deeper this time, I am finding its recommendations much better than Goodreads' -- although it seems to sometimes recommend books I've already read. And it has some nice features like tracking series, so you can easily tell when you've read all the books in a series or not. The analytics overall seem quite impressive. The UI is cluttered and it seems to take 5 clicks to add and rate a single book. It supports half stars. Overall I get the feeling this was designed for a set of needs that doesn't quite match mine. For example, it seems it doesn't have a single database entry per book; instead each time I add a book, it seems to pull in data from primary sources (library of congress, Amazon cough) and treat this as a separate (but related) entry somehow. Weird. Perhaps this makes sense to say, librarians. I'm willing to adjust how I think about things if there's an underlying reason that can be grasped. There's a quite interesting thread on LibraryThing where the founder says:

Don't say we should open-source the code. That would be a nightmare! And I have limited confidence in APIs. LibraryThing has the book geeks, but not so much the computers geeks.

I assume that the nightmare is that there would be dozens of clones of the site, all balkanized, with no data transfer, no federation between them. Except, that's the current situation, as every Goodreads user who is now trying to use LibraryThing is discovering. Before I ever started using Goodreads, I made sure it met my minimum criteria for putting my data into a proprietary silo: That I could get the data back out. I can, and have. LibraryThing can import it. But the import process loses data! And it's majorly clunky. If I want to continue using Goodreads due to its better UI, and get the data into LibraryThing, for its better analytics, I have to do periodic dumps and loads of CSV files with manual fixups. This is why we have standards. This is why we're building federated social networks like status.net and the upcoming pump.io that can pass structured data between nodes transparently. It doesn't have to be a nightmare. It doesn't have to rely on proprietary APIs. We have the computer geeks. Thing is, sites like GoodReads and LibraryThing need domain-specific knowledge, and communities to curate data, and stuff like that. Things that work well in a smallish company. (LibraryThing even has a business model that makes sense, yearly payments to store more books in it.) With free software, it's much more appealing to sink the time we have into the most general-purpose solution we can. Why build a LibraryThing when we could build something that tracks not only books but movies and music? Why build that when we could build a generic federated network for structured social data? And that's great, as infrastructure, but if that infrastructure is only used to build a succession of proprietary data silos, what was the point? So, could some computer & book geeks please build a free software alternative to these things, focused on books, that federates using any of the fine APIs we have available? Bear in mind that there is already a nice start at a comprehensive collection of book data in the Open Library. I'd happily contribute to a crowd funded project doing this.

The graph below is generated from popcon submissions. Since they include the version of the popularity-contest package, one can determine the Debian release that was used by the submitter (a new version of the popularity-contest package is generally uploaded just after the release to make that tracking possible). The graph is similar to the one found on popcon, except that versions newer than the latest stable release are aggregated as testing/unstable . Comments:

• Popcon submitters might not be representative of Debian users, of course.
• There s quite a lot of testing/unstable users, and their proportion is quite stable:
  • Today: testing/unstable: 43119 submissions (31.4%); squeeze: 75454 (54.9%); lenny: 13603 (9.9%)
  • 2011-02-04 (just before the squeeze release): testing/unstable: 29012 (29.7%); lenny: 57262 (58.6%); etch: 10032 (10.3%)
  • 2009-02-13 (just before the lenny release): testing/unstable: 30108 (36.9%); etch: 49996 (61.2%)
• I don t understand why the number of Debian stable installations does not increase, except when a new release is made. It s as if people installed Debian, upgraded directly to testing, and switched back to tracking stable after the release. Or maybe people don t update their systems? A more detailed analysis could be done by looking at the raw popcon data.
• Upgrading to the next stable release takes time. Looking at the proportion of users still using oldstable one year after the release, it would be better not to remove oldstable from mirrors too early.

Scripts are available on git.debian.org.

The annoucement below just went to the R-SIG-Finance list. More information is as usual at the R / Finance page:

Now open for registrations: R / Finance 2013: Applied Finance with R
May 17 and 18, 2013
Chicago, IL, USA
The registration for R/Finance 2013 -- which will take place May 17 and 18 in Chicago -- is NOW OPEN! Building on the success of the previous conferences in 2009, 2010, 2011 and 2012, we expect more than 250 attendees from around the world. R users from industry, academia, and government will joining 30+ presenters covering all areas of finance with R. We are very excited about the four keynotes by Sanjiv Das, Attilio Meucci, Ryan Sheftel and Ruey Tsay. The main agenda (currently) includes seventeen full presentations and fifteen shorter "lightning talks". We are also excited to offer five optional pre-conference seminars on Friday morning. To celebrate the fifth year of the conference in style, the dinner will be held at The Terrace of the Trump Hotel. Overlooking the Chicago river and skyline, it is a perfect venue to continue conversations while dining and drinking. More details of the agenda are available at:

http://www.RinFinance.com/agenda/

Registration information is available at

http://www.RinFinance.com/register/

and can also be directly accessed by going to

http://www.regonline.com/RFinance2013

We would to thank our 2013 Sponsors for the continued support enabling us to host such an exciting conference:

International Center for Futures and Derivatives at UIC Revolution Analytics
MS-Computational Finance at University of Washington Google
lemnica
OpenGamma
OneMarketData
RStudio

On behalf of the committee and sponsors, we look forward to seeing you in Chicago!

Gib Bassett, Peter Carl, Dirk Eddelbuettel, Brian Peterson,
Dale Rosenthal, Jeffrey Ryan, Joshua Ulrich

See you in Chicago in May!!

The goal of this article is to illustrate how to manage Python code which includes comments in UTF-8 characters inside a latin-1 source org-mode for LaTeX export. Note that I ve pasted in wordpress the HTML generated by org-mode, so I hope it isn t broken too much. My typical use case is a french lecture on Python where the text is written in french, as well as some of the code comments and examples We ll use org-mode s babel module to include and manage the Python
examples. The goal is to write the source of the Python programs
directly in the same org source as the class book s text, and to extract them into a subdir (with the tangle feature), so that they can be shipped to the students to experiment with. The minted LaTeX environment is used, for babel, to make the Python syntax highlighting.

lbcd is an agent that we run on systems that use software load balancing. It returns a weight and an increment that's used by the lbnamed DNS server. This is a minor feature release. The main point of this release is to add support for an /etc/nolbcd file (or /usr/local/etc/nolbcd in the default build) that's akin to /etc/nologin except it just drops the system from an lbnamed load balance pool rather than denying all logins. Sometimes it's more convenient to do that than to stop the lbcd daemon, particularly when the latter is monitored by configuration management software. Also in this release, the lbcd -t mode, which prints all the data to standard output, is now documented and has been improved, and lbcdclient no longer rewrites maximum weights to -1. You can get the latest release from the lbcd distribution page.

Finally. The wallet is a system management tool for storing and retrieving secure data. I originally wrote it as part of our Kerberos v5 migration project to replace the old Kerberos v4 system we had for distributing srvtabs. It's still mostly used at Stanford for distributing keytabs, but it's extensible and supports storing arbitrary secure data. We also use it internally for storing database passwords, SSL private keys, and anything else that needs to be kept secure and retrieved later. It supports a rich ACL mechanism and namespace enforcement and is built on top of remctl for security and confidentiality. I did a bunch of development on it originally in 2006 and 2007, but then it solved our immediate problems and I didn't have much time to work on it. I used pre-1.0 versions because there was so much more on it that I wanted to do. But we've been running it in production for years, and while there's still tons more work that needs to be done, it really deserves a 1.0 version. Jon Robertson converted the database backend to DBIx::Class (which added PostreSQL support) and implemented schema upgrading, which were two of the major things that were keeping me from putting a 1.0 version on it. So it's time. Other new features since the previous release include a change to the default ACLs so that owners of objects can destroy them by default, a new ldap-attr ACL type that checks whether the caller has a particular attribute set in LDAP (which will let us use our entitlement system to control access), and support for storing and managing key rotation for WebAuth keyrings. There's also a new acl check command to determine whether an ACL exists and a new comment field and command to store an arbitrary comment about an object. Our local naming policy has also gotten much more complex, so I turned it into a module to provide an example to others of how to implement some reasonably detailed namespace and object autocreation controls. You can get the latest version from the wallet distribution page. I'm going to upgrade Debian packages to experimental or unstable (depending on the freeze status) as soon as I get a chance to write a bit more documentation and hopefully figure out debconf integration with the database setup support inside Debian for the server piece.

Ladies and gentlemen, I'm so excited to present you the official blog of the Debian Project. The idea is to make it the voice of the project, in addition to the official press releases and DPN, our bi-weekly newsletter.
I'm really excited about it not only because it will help us to communicate better with our users in letting them know what is going on, but also because it's something we long waited (and planned) for. And it was surely a collective effort: many thanks to Ana Guerrero, who not only is one of the main editors, but also the engine behind the idea and the realization of it; Stefano Zacchiroli, who during his three terms as DPL gave lots of support to this idea; and the Debian System Administrators Team who put all in place (and found the perfect name for it).

We have asked Lucas Nussbaum, one of the three candidates for DPL elections 2013, to tell our readers about himself and his ideas for the Debian Project. You can also read the interviews to the other two candidates: Gergely Nagy and Moray Allan.

---

Please tell us a little about yourself. Hi! I'm a 31 years old french computer geek. In my day job, I'm an assistant professor (Ma tre de Conf rences) of Computer Science at Universit de Lorraine. What do you do in Debian and how did you started contributing? Like many, I started contributing to Debian by creating and maintaining packages for my own software, in the Ruby team. Then, I discovered that, even if it's not so obvious from the outside, there are a lot of areas in Debian that could use more contributors. So I just started to contribute to more and more things. There's a list of things I did in Debian in my platform. What I have been doing recently is:

• rebuild all packages in Debian on a regular basis in order to identify packages that can no longer be built, and file bugs accordingly. In order to do that efficiently, I use cluster and cloud resources (more info)
• develop and maintain Ultimate Debian Database, a data aggregator that collects data in most Debian services so that it is possible to expose it in interesting ways (e.g. find release-critical bugs affecting popular packages).
• write and maintain a Debian Packaging tutorial, (packaging-tutorial package), to provide an easy-to-read introduction to packaging in Debian.

Why did you decide to run as DPL? Two main reasons:

• Most of my Debian contributions aim at addressing problems at the distribution scale (cross-distro collaboration, Quality Assurance, data-mining). Being DPL is a great way to contribute to Debian at this level.
• the DPL campaign is a great time in Debian where we discuss the project's problems, politics and visions. Being a candidate is in itself a way to contribute to Debian (though it would be better if we had those discussions outside DPL campaigns too).

Three keywords to summarise your platform. (re-)make Debian the center of the Free Software ecosystem; foster innovation inside Debian; reduce barriers to contributions What are the biggest challenges that you envision for Debian in the future? I often have the impression that the project is losing momentum, positive energy, and slowing down. It feels like we are living on the benefits of the past. A lot of very cool things happen in the Debian ecosystem, but very often outside the Debian project (in derivative distributions). Debian should aim at reinforcing its position in the center of the Free Software ecosystem: it should be the main active intermediary between upstream projects and final users. To achieve that, we need to reinforce the visibility and the impact of Debian. This is extremely important because the values we fight for as a project are often neglected by our derivatives. What are, in your opinion, the areas of the project more in need of technical and/or social improvements? Fostering innovation inside Debian: we should be more welcoming towards innovation and experiments inside the project. Often, we merely tolerate them, and bureaucracy makes them hard and slow to conduct. As a result, people tends to innovate outside the Debian project. Making it easier to contribute to Debian: we compete with more and more projects to attract contributors. While we are already quite good at welcoming new contributors with good documentation and mentoring (much better than people usually think), there's still a lot of room for improvement. Why should people vote for you? A great thing in Debian's voting system is that you don't vote "for" or "against" a specific candidate. Instead, due to our use of the Condorcet method, you rank candidates (and also indicate those who you consider suitable for the role by ranking a virtual "None of the above" candidate). Why am I a good candidate? My previous contributions to Debian show that I have a pretty good understanding of the inner workings of the project, and that I have a track record of managing projects successfully inside Debian. I think that those are two required qualities for a DPL. Name three tools you couldn't stay without. vim, mutt, ssh. What keep you motivated to work in Debian? Debian is a fantastic project from a technical point of view (focus on technical excellence, lots of interesting challenges), but also from a social point of view: the Debian community is a great community where I have lots of good friends. Also, what's great when you contribute to Debian is that your work has a real impact, and that you see people using stuff you worked on everywhere. Are there any other fields where you call yourself a geek, besides computers? I'm not sure this really qualifies as "besides computers", but I've gotten very interested in the OpenStreetMap project lately. I very much enjoy exploring unmapped areas on a mountain bike. It feels like being Christopher Columbus or Marco Polo, but 20 minutes from home. ;) The OpenStreetMap and Debian projects also share many values, such as a great attention to quality and details.

I recently decided to try out a snapshot of OpenBSD 5.3, intrigued by the notice that

softraid(4) RAID1 and crypto volumes are now bootable on i386 and amd64 (full disk encryption).

I ve always used an encrypted LVM (as setup by the alternate CD) with a fully encrypted root on my netbook and laptop when running Debian/Kubuntu and never noticed much a performance hit. Unfortunately, I m not the only one who has noticed a significant drop in performance with full-disk encryption on OpenBSD. Although the steps required to setup full-disk encryption (as opposed to requiring manual intervention at boot or the equivalent of loop-mounted encrypted images via vnconfig(8)) on OpenBSD are not explicitly documented anywhere, searching through the mailing lists, man pages, and trial and error provided the following steps:

1. Boot from your install media. Select the (S)hell option.
2. Assuming the disk you re installing to is wd0 , run fdisk -i wd0 if you re on an i386. If I m not mistaken, you can skip to the following step otherwise.
3. Run disklabel -E wd0. Create a label (typically b) of the appropriate size of type swap for swap, and then add a label a using the remainder of the space of type RAID. We aren t going to put swap on our crypto device since OpenBSD has encrypted the swap partition by default since 2005.
4. Setup the crypto volume as described in bioctl(8). You will be prompted for a passphrase and presented with a message along the lines of softraid0: SR CRYPTO volume attached as sd0 on successful setup. I used bioctl -c C -l /dev/wd0a softraid0 where - -c C specifies a device with raidlevel CRYPTO , - -l /dev/wd0a specifies the label of type RAID created in step 3 to use for the CRYPTO device, and - softraid0 is the softraid device to configure (this value should work unless you already have a softraid0 device).
5. exit from the shell and (I)nstall as normal, installing to the disk sd0 (or whichever disk the crypto volume got attached as). In the partitioning dialog, you may need to manually adjust things since the auto-partitioning creates an additional swap label on sd0, which we don t need given we already have wd0b.
6. Using ed(1), add the line 1a8bab44e9cc178d.b none swap sw to /mnt/etc/fstab, assuming that 1a8bab44e9cc178d is the DUID associated with wd0 (you can find this out by running sysctl hw.disknames, which will output something like hw.disknames=wd0:1a8bab44e9cc178d,cd0:,rd0:6ce80c78714fa32f,sd0:fce7bfa23c8ec20d; just search for the DUID associated with wd0). Alternatively, wait until you boot into your system, and add it to your /etc/fstab using vi(1) or mg(1).
7. Reboot, and you ll be prompted for your passhprase. You re done!

Here s a screenshot of steps 1 through 4. Here s a screenshot of the passphrase prompt, after successfully entering the passphrase: If I ve made any errors in the steps above, please point them out in the comments below or by sending me an email.

But you can t just tar.gz up the bare repositories on the server and hope for the best. Maybe a given repository will be in a valid state; maybe it won t.

-- Jeff Mitchell in a followup to the recent KDE near git disaster This was a surprising statement to me. I seem to remember that one of (many) selling points for git talked about back in the day was that it avoided the problem that making a simple cp (or backup) of a repository could lead to an inconsistent result. A problem that subversion repositories had, and required annoying commands to work around. (svnadmin $something -- iirc the backend FSFS fixed or avoided most of this issue.) This prompted me to check how I handle it in ikiwiki-hosting. I must have anticipated a problem at some point, since ikisite backup takes care to lock the git repository in a way that prevents eg, incoming pushes while a backup is running. Probably, like the KDE developers, I was simply exercising reasonable caution. The following analysis has probably been written up before (train; limited network availability; can't check), but here are some scenarios to consider:

• A non-bare repository has two parts that can clearly get out of sync during a backup: The work tree and the .git directory.
  • The .git directory will likely be backed up first, since getdirent will typically return it first, since it gets created first . If a change is made to the work tree during that backup, and committed while the work tree is being backed up, the backup won't include that commit -- which is no particular problem and would not be surprising upon restore. Make commit again and get on with life.
  • However, if (part of) the work tree is backed up before .git, then any changes that are committed to git during the backup would not be reflected in the restored work tree, and git diff would show a reversion of those changes. After restore, care would need to be taken to reset the work tree (without losing any legitimate uncommitted changes).
• A non-bare repository can also become broken in other ways if just the wrong state is snapshotted. For example, if a commit is in progress during a backup, .git/index.lock may exist, and prevent future commits from happening, until it's deleted. These problems can also occur if the machine dies at just the right time during a commit. Git tells you how to recover. (git could go further to avoid these problems than it does; for example it could check if .git/index.lock is actually locked using fcntl. Something I do in git-annex to make the .git/annex/index.lock file crash safe.)
• A bare repository could be receiving a push (or a non-bare repository a pull) while the backup occurs. These are fairly similar cases, with the main difference being that a non-bare repository has the reflog, which can be used to recover from some inconsist states that could be backed up. Let's concentrate on pushes to bare repositories.
  • A pack could be in the process of being uploaded during a backup. The KDE developers apparently worried that this could result in a corrupt or inconsistent repository, but TTBOMK it cannot; git transfers the pack to a temp file and atomically renames it into place once the transfer is complete. A backup may include an excess temp file, but this can also happen if the system goes down while a push is in progress. Git cleans these things up.
  • A push first transfers the .git/objects, and then updates .git/refs. A backup might first back up the refs, and then the objects. In this case, it would lose the record that refs were pushed. After being restored, any push from another repository would update the refs, even using the objects that did get backed up. So git recovers from this, and it's not really a concern.
  • Perhaps a backup chooses to first back up the objects, and then the refs. In this case, it could back up a newly changed ref, without having backed up the referenced objects (because they arrived after the backup had finished with the objects). When this happens, your bare repository is inconsistent; you have to somehow hunt down the correct ref for the objects you do have. This is a bad failure mode. git could improve this, perhaps, by maintaining a reflog for bare repositories. (Update: core.logAllRefUpdates can be set to true for bare repositories, but is disabled by default.)
• A "backup" of a git repository can consist of other clones of it. Which do not include .git/hooks/ scripts, .git/config settings, and potentially other valuable information, that strangely, we do not check into revision control despite having this nice revision control system available. This is the most likely failure mode with "git backups". :P

I think that it's important git support naive backups of git repositories as well as possible, because that's probably how most backups of git repositories are made. We don't all have time to carefully tune our backup systems to do something special around our git repositories to ensure we get them in a consistent state like the KDE project did, and as their experience shows, even if we do it, we can easily introduce other, unanticipated problems. Can anyone else think of any other failure modes like these, or find holes in my slightly rushed analysis?

---

PS: git-annex is itself entirely crash-safe, to the best of my abilities, and also safe for naive backups. But inherits any problems with naive backups of git repositories.

The Standalone Sysadmin asks, How do you manage database server instances? :

Do you have one (or a few) centralized database servers, either standalone or clustered, or do you spread the load like we are currently?

His argument for centralisation is one of easing the management burden of configuration and backups, whereas the distributed approach eliminates a central point of failure and performance degradation. I go for distributed, all the way. For a start, we run so many databases for so many customers that there s no way on earth we could stand up a small number of database servers and handle all the load (hell, we ve got single customers who consume a cluster of machines with 384GB of RAM and all the SSDs you can eat). Security and permissions is a whole other kettle of fish; the contortions we d have to do to allow customers the level of management they need with a centralised database system would be immense. Then there s the need of some customers for MySQL, some for PgSQL, different performance tuning for different workloads nope, centralised DBs don t work for us. Given this, we ve bitten the bullet and solved pretty much all of the management problems. Installation and configuration is all handled via Puppet, and backups are trivial the same system that installs the DB server itself also drops a hook script that the backup agent uses to know that it has to dump a database server. Monitoring that this backup is taking place successfully is also automatically provisioned, so we know that we re not missing anything. Ultimately, this same approach applies to practically anything that you re tossing up between centralised and distributed. At scale, you can never rely on centralisation, so you may as well bite the bullet and learn how to do it distributed pretty much from the start. That saves some serious system shock when you discover what your hardware vendor wants for the next step up in big iron hardware

I just pushed out a new python-dbusmock release 0.6. Calling a method on the mock now emits a MethodCalled signal on the org.freedesktop.DBus.Mock interface. In some cases this is easier to track than parsing the mock s log or using GetMethodCalls. Thanks to Lars Uebernickel for this. DBusMockObject.AddTemplate() and DBusTestCase.spawn_server_template() can now load local templates from your own project by specifying a path to a *.py file as template name. Thanks to Lucas De Marchi for this feature. I also wrote a quite comprehensive template for systemd s logind. It stubs out the power management functionality as well as user/seat/session objects, and is convincing enough for loginctl. Some bits like AttachDevice is missing, as this sounds unlikely to be required for D-BUS mock tests, but please let me know if you need anything else. The mock processes now terminate automatically if their connected D-BUS goes down, as advertised in the documentation. You can get the new tarball from Launchpad, and I uploaded it to Debian experimental now. Enjoy!