Long time since I published the last update about Tanglu. So here is a short summary about what we did meanwhile (even if you don t hear that much, there is lots of stuff going on!) Better QA in the archive We now use a modified version of Debian s Britney tool to migrate packages from the newly-created staging suite to our current development branch aequorea . This ensures that all packages are completely built on all architectures and don t break other packages. New uploads and syncs/merges now happen through the staging area and can be tested there as well as being blocked on demand, so our current aequorea development branch stays installable and usable for development. People who want the *very* latest stuff, can embed the staging-sources into their sources.list (but we don t encourage you to do that). Improved syncs from Debian The Synchrotron toolset we use to sync packages with Debian recently gained the ability to sync packages using regular expressions. This makes it possible to sync many packages quickly which match a pattern. Tons of infrastructure bugfixes The infrastructure has been tweaked a lot to remove quirks, and it now works quite smoothly. Also, all Debian tools now work flawless in the Tanglu environment. A few issues are remaining, but nothing really important is affected anymore (and some problems are merely cosmetic). Long term we plan to replace the Jenkins build-infrastructure with the software which is running Paul Tagliamontes debuild.me (only the buildd service, the archive will still be managed by dak). This requires lots of work, but will result in software not only usable by Tanglu, but also by Debian itself and everyone who wants a build-service capable of building a Debian-distro. KDE 4.10 and GNOME 3.8 Tanglu now offers KDE 4.10 by default, a sync of GNOME 3.8 is currently in progress. The packages will be updated depending on our resources and otherwise just be synced from Debian unstable/experimental (and made working for Tanglu). systemd 204 & libudev1 Tanglu now offers systemd 204 as default init system, and we transitioned the whole distribution to the latest version of udev. This even highlighted a few issues which could be fixed before the latest systemd reached Debian experimental. The udev transition went nicely, and hopefully Debian will fix bug#717418 too, soon, so both distributions run with the same udev version (which obviously makes things easier for Tanglu ^^) Artwork We now have a plymouth-bootscreen and wallpapers and stuff is in progress Alpha-release & Live-CD ? This is what we are working on we have some issues in creating a working live-cd, since live-build does have some issues with our stack. We are currently resolving issues, but because of the lack of manpower, this is progressing slowly (all contributors also work on other FLOSS projects and of course also have their work ) As soon as we have working live-media, we can do a first alpha release and offer installation media. You! Tanglu is a large task. And we need every help we can get, right now especially technical help from people who can build packages (Debian Developers/Maintainers, Ubuntu developers, etc.) We especially need someone to take care of the live-cd. But also the website needs some help, and we need more artwork or improved artwork In general, if you have an idea how to make Tanglu better (of course, something which matches it s goals ) and want to realize it, just come back to us! You can reach us on the mailiglists (tanglu-project for stuff related to the project, tanglu-devel for development of the distro) or in #tanglu / #tanglu-devel (on Freenode).

FrOSCon is approaching, and all MirBSD developers will attend but why s there no MirBSD exhibit? The answer to that is a bit complex. First let s state that of course we will participate in the event as well as the Open Source world. We ll also be geocaching around the campus with other interested (mostly OSS) people (including those we won for this sport) and helping out other OSS projects we ve become attached to. MirOS BSD, the operating system, is a niche system. The conference on the other hand got younger and more mainstream. This means that almost all conference visitors do not belong to the target group of MirOS BSD which somewhat is an ancient solution : the most classical BSD around (NetBSD loses because they have rc.d and PAM and lack sendmail(8), sorry guys, your attempt at being not reformable doesn t count) and running on restricted hardware (such as my 486SLC with 12 MiB RAM) and exots (SPARCstation). It s viable even as developer workstation (if your hardware is supported otherwise just virtualise it) but its strength lies with SPARC support and embedded x86 . And being run as virtual machine: we re reportedly more stable and more performant than OpenBSD. MirBSD is not cut off from modern development and occasionally takes a questionable but justified choice (such as using 16-bit Unicode internally) or a weird-looking but beneficial one (such as OPTU encoding saving us locale(1) hassles) or even acts as technological pioneer (64-bit time_t on ILP32 platforms) or, at least, is faster than OpenBSD (newer GNU toolchain, things like that), but usually more conservatively, and yes, this is by design, not by lack of manpower, most of the time. The MirPorts Framework, while technically superiour in enough places, is something that just cannot happen without manpower. I (tg@) am still using it exclusively, continuing to update ports I use and occasionally creating new ones (mupdf is in the works!), but it s not something I d recommend someone (other than an Mac OSX user) to use on a n n-MirBSD system (Interix is not exactly thriving either, and the Interix support was only begun; other OSes are not widely tested). The MirBSD Korn Shell is probably the one thing I will be remembered for. But I have absolutely no idea how one would present it on a booth at such an exhibition. A talk is much more likely. So no on that front too. jupp, the editor which sucks less, is probably something that does deserve mainstream interest (especially considering Natureshadow is using it while teaching computing to kids) but probably more in a workshop setting. And booth space is precious enough in the FH so I think that d be unfair. All the other subprojects and side projects Benny and I have, such as mir c , josef stalin, FreeWRT, Lunix Ewe, Shellsnippets, the fonts, etc. are interesting but share few, if any, common ground. Again, this does not match the vast majority of visitors. While we probably should push a number of these more, but a booth isn t it here, either. MirOS Linux ( MirLinux ) and MirOS Windows are, despite otherwise-saying rumours called W*k*p*d*a, only premature ideas that will not really be worked on (though MirLinux concepts are found in mir c and stalin). As you can see, despite all developers having full-time dayjobs, The MirOS Project is far from being obsolete. We hope that our website visitors understand our reasons to not have an exhibition booth of our own (even if the SPARCstation makes for a way cool one, it s too heavy to lift all the time), and would like to point out that there are several other booths (commercial ones, as well as OSS ones such as AllBSD, Debian and (talking to) others) and other itineries we participate in. This year both Benny and I have been roped into helping out the conference itself, too (not exactly unvoluntarily though). The best way to talk to us is IRC during regular European geek hours (i.e. until way too late into the night which Americans should benefit from), semi-synchronously, or mailing lists. We sort of expect you to not be afraid to RTFM and look up acronyms you don t understand; The MirOS Project is not unfriendly but definitely not suited for your proverbial Aunt Tilly, newbies, desktop users, and people who aren t at least somewhat capable of using written English (this is by design).

... which is probably a good thing. But here is the boot log you all have been waiting for:

start ext2fs: Hurd server bootstrap: ext2fs[device:hd0s1] exec init proc auth
INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
Activating swap...done.
Checking root file system...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
/dev/hd0s1: clean, 44693/181056 files, 291766/723200 blocks
done.
Activating lvm and md swap...(default pager): Already paging to partition hd0s5!
done.
Checking file systems...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
done.
Cleaning up temporary files... /tmp.
Mounting local filesystems...done.
Activating swapfile swap...(default pager): Already paging to partition hd0s5!
done.
df: Warning: cannot read table of mounted file systems: No such file or directory
Cleaning up temporary files....
Configuring network interfaces...Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
Listening on Socket//dev/eth0
Sending on   Socket//dev/eth0
*** stack smashing detected ***: dhclient terminated
Aborted
Failed to bring up /dev/eth0.
done.
Cleaning up temporary files....
Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix.
INIT: Entering runlevel: 2
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Starting deferred execution scheduler: atd.
Starting periodic command scheduler: cron.
Starting system message bus: dbusFailed to set socket option"/var/run/dbus/system_bus_socket": Protocol not available.
Starting OpenBSD Secure Shell server: sshd.
unexpected ACK from keyboard
GNU 0.3 (debian) (console)
login: root
[...]
root@debian:~# ifup /dev/eth0
Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
Listening on Socket//dev/eth0
Sending on   Socket//dev/eth0
*** stack smashing detected ***: dhclient terminated
Aborted
Failed to bring up /dev/eth0.
root@debian:~# dhclient -v -pf /run/dhclient.-dev-eth0.pid -lf /var/lib/dhcp/dhclient.-dev-eth0.leases /dev/eth0
Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
Listening on Socket//dev/eth0
Sending on   Socket//dev/eth0
*** stack smashing detected ***: dhclient terminated
Aborted
root@debian:~# dhclient -pf /run/dhclient.-dev-eth0.pid -lf /var/lib/dhcp/dhclient.-dev-eth0.leases /dev/eth0
root@debian:~# ifup /dev/eth0
Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
Listening on Socket//dev/eth0
Sending on   Socket//dev/eth0
DHCPREQUEST on /dev/eth0 to 255.255.255.255 port 67
DHCPACK from 10.0.2.2
bound to 10.0.2.15 -- renewal in 34108 seconds.
ps: comm: Unknown format spec
root@debian:~# halt
Broadcast message from root@debian (console) (Fri Aug 23 19:42:19 2013):
The system is going down for system halt NOW!
INIT: Switching to runlevel: 0root@debian:~#
INIT: Sending processes the TERM signal
INIT: Sending processes the KILL signal
Using makefile-style concurrent boot in runlevel 0.
Stopping deferred execution scheduler: atd.
task c10f53f8 deallocating an invalid port 2098928, most probably a bug.
Asking all remaining processes to terminate...done.
All processes ended within 1 seconds...done.
Stopping enhanced syslogd: rsyslogd.
Deconfiguring network interfaces...Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
Listening on Socket//dev/eth0
Sending on   Socket//dev/eth0
DHCPRELEASE on /dev/eth0 to 10.0.2.2 port 67
/dev/eth0 (2):
  inet address  0.0.0.0
  netmask       255.255.255.0
  broadcast     10.0.2.255
  flags         BROADCAST ALLMULTI MULTICAST
  mtu           1500
done.
Deactivating swap...swapoff: /dev/hd0s5: 177152k swap space
done.
Unmounting weak filesystems...umount: /etc/mtab: Warning: duplicate entry for device /dev/hd0s1 (/servers/socket/26)
umount: /etc/mtab: Warning: duplicate entry for device /dev/hd0s1 (/dev/cons)
umount: could not find entry for: /dev/cons
umount: could not find entry for: /servers/socket/26
done.
mount: cannot remount /: Device or resource busy
Will now halt.
store a new irq 11init: notifying pfinet of shutdown...init: notifying tmpfs swap of shutdown...init: notifying tmpfs swap of shutdown...init: notifying tmpfs swap of shutdown...init: notifying ext2fs device:hd0s1 of shutdown...init: halting Mach (flags 0x8)...
In tight loop: hit ctl-alt-del to reboot

With some tiny patches for ifupdown I've been able to resolve network related issues. All of them? Of course not, funny thing about developing for the Hurd is that once you fix one thing, then some other thing or code path is executed that has never been run on Hurd before, and therefore something else breaks. In this case I fixed ifupdown to generate valid names for the pid file and leases file and all of the sudden dhclient starts dying. Funny thing about that is, if one drops the -v flag from the dhclient invocation as I did it above, the crash isn't triggered and once the lease file has been successfully written, it is safe to add the -v flag again. Not yet sure what goes on there, then again, looking at the source of isc-dhcp-client it is not so surprising that it crashes :/ When I first looked at ifupdown it was written in noweb, a literate programming tool. It is an interesting idea, even more so since (classic) c can be very verbose and cryptic. But it decouples the control flow from the structure of the program, which makes patching it quite a challenge since it is not as obvious where the changes have to go in. This is how ifupdown looked some weeks ago:

% wc --lines ifupdown.nw
6123 ifupdown.nw
% pdftk ifupdown.pdf dump_data   grep NumberOfPages
NumberOfPages: 113

The ifupdown.nw is the noweb source, from which seven .c, four .h, two .pl and one Makefile are generated. It also contains a redicioulus amount of documentation, to the point that the authors at several points did not now what to write and just drop some nonsensical lines into the file. The source also compiles to a 113 page pdf file, that contains all of the documentation and all of the code, not at all in the order that one would expect a program to be written, but in the order the authors chose to structure the documentation. Fortunately for me the maintainer decided to drop the noweb source and to add the generated files to the source control system. This made my job much easier :) So here are the patches I published this week:

• Fixes for ifupdown: http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00063.html
• Fixes for fakeroot-hurd and remap: http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00064.html

I must admit that I do not know exactly what I will do next week. Obviously fixing the dhclient crash would be nice, I'll look into that. But I'm surely find some useful thing to do.

Eric (a fellow Fedora board member) has a post describing his vision for what Fedora as an end goal should look like. It's essentially an assertion that since we have no idea who our users are or what they want, we should offer them everything on an equal footing.

Shockingly enough, I disagree.

At the most basic level, the output of different Special Interest Groups is not all equal. We've had issues over the past few releases where various spins have shipped in a broken state, because the SIG responsible for producing them doesn't have the resources to actually test them. We're potentially going to end up shipping F20 with old Bluetooth code because the smaller desktops aren't able to port to the new API in time[1]. Promoting these equally implies that they're equal, and doing so when we know it isn't the case is a disservice to our users.

But it's not just about our users. Before I joined the Fedora project, I'd worked on both Debian and Ubuntu. Debian is broadly similar to the current state of Fedora - no strong idea about what is actually being produced, and a desire among many developers to cater to every user's requirements. Ubuntu's pretty much the direct opposite, with a strongly defined goal and a willingness to sacrifice some use cases in order to achieve that goal.

This leads to an interestingly different social dynamic. Ubuntu contributors know what they're working on. If a change furthers the well-defined aim of the project, that change happens. Moving from Ubuntu to Fedora was a shock to me - there were several rough edges in Fedora that simply couldn't be smoothed out because fixing them for one use case would compromise another use case, and nobody could decide which was more important[2]. It's basically unthinkable that such a situation could arise in Ubuntu, not just because there was a self appointed dictator but because there was an explicit goal and people could prioritise based on that[3].

Bluntly, if you have a well-defined goal, people are more likely to either work towards that goal or go and do something else. If you don't, people will just do whatever they want. The risk of defining that goal is that you'll lose some of your existing contributors, but the benefit is that the existing contributors will be more likely to work together rather than heading off in several different directions.

But perhaps more importantly, having a goal can attract people. Ubuntu's Bug #1 was a solid statement of intent. Being freer than Microsoft wasn't enough. Ubuntu had to be better than Microsoft products on every axis, and joining Ubuntu meant that you were going to be part of that. Now it's been closed and Ubuntu's wandered off into convergence land, and signing up to spend your free time on producing something to help someone sell phones is much less compelling than doing it to produce a product you can give to your friends.

Fedora should be the obvious replacement, but it's not because it's unclear to a casual observer what Fedora actually is. The website proudly leads with a description of Fedora as a fast, stable and powerful operating system, but it's obvious that many of the community don't think of Fedora that way - instead it's a playground to produce a range of niche derivatives, with little consideration as to whether contributing to Fedora in that way benefits the project as a whole. Codifying that would actively harm our ability to produce a compelling product, and in turn reduce our ability to attract new contributors even further.

Which is why I think the current proposal to produce three first-class products is exciting. Offering several different desktops on the download page is confusing. Offering distinct desktop, server and cloud products isn't. It makes it clear to our users what we care about, and in turn that makes it easier for users to be excited about contributing to Fedora. Let's not make the mistake of trying to be all things to all people.

[1] Although clearly in this case the absence of a stable ABI in BlueZ despite it having had a dbus interface for the best part of a decade is a pretty fundamental problem.
[2] See the multi-year argument over default firewall rules and the resulting lack of working SMB browsing or mDNS resolving
[3] To be fair, one of the reasons I was happy to jump ship was because of the increasingly autocratic way Ubuntu was being run. By the end of my involvement, significant technical decisions were being made in internal IRC channels - despite being on the project's Technical Board, I had no idea how or why some significant technical changes were being made. I don't think this is a fundamental outcome of having a well-defined goal, though. A goal defined by the community (or their elected representatives) should function just as well.

comments

The second wheezy based beta release of Debian Edu was wrapped up today, slightly delayed because of some bugs in the initial Windows integration fixes . This is the release announcement: New features for Debian Edu 7.1+edu0~b1 released 2013-08-22 These are the release notes for Debian Edu / Skolelinux 7.1+edu0~b1, based on Debian with codename "Wheezy". About Debian Edu and Skolelinux Debian Edu, also known as Skolelinux, is a Linux distribution based on Debian providing an out-of-the box environment of a completely configured school network. Immediately after installation a school server running all services needed for a school network is set up just waiting for users and machines being added via GOsa , a comfortable Web-UI. A netbooting environment is prepared using PXE, so after initial installation of the main server from CD or USB stick all other machines can be installed via the network. The provided school server provides LDAP database and Kerberos authentication service, centralized home directories, DHCP server, web proxy and many other services. The desktop contains more than 60 educational software packages and more are available from the Debian archive, and schools can choose between KDE, Gnome, LXDE and Xfce desktop environment. This is the sixth test release based on Debian Wheezy. Basically this is an updated and slightly improved version compared to the Squeeze release. ALERT: Alpha based installations should reinstall or downgrade the versions of gosa and libpam-mklocaluser to the ones used in this beta release. Both alpha and beta0 based installations should reinstall or deal with gosa.conf manually; there are two options: (1) Keep gosa.conf and edit this file as outlined on the mailing list. (2) Accept the new version of gosa.conf and replace both contained admin password placeholders with the password hashes found in the old one (backup copy!). In both cases every user need to change their their password to make sure a password is set for CIFS access to their home directory. Software updates

• Added ssh askpass packages to default installation, to ensure ssh work also without a attached tty.
• Add the command-not-found package to the default installation to make it easier to figure out where to find missing command line tools. Please note, that the command 'update-command-not-found' has to be run as root to actually make it useful (internet access required).

Other changes

• Adjusted the USB stick ISO image build to include every tool needed for desktop=xfce installations.
• Adjust thin-client-server task to work when installing from USB stick ISO image.
• Made new grub artwork (changed png from indexed to RGB format).
• Minor cleanup in the CUPS setup.
• Make sure that bootstrapping of the Samba domain really happens during installation of the main server and adjust SID handling to cope with this.
• Make Samba passwords changeable (again) via GOsa .
• Fix generation of LM and NT password hashes via GOsa to avoid empty password hashes.
• Adapted Samba machine domain joining to latest change in the smbldap-tools Perl package, fixing bugs blocking Windows machines from joining the Samba domain.

Known issues

• KDE fails to understand the wpad.dat file provided, causing it to not use the http proxy as it should.
• Chromium also fails to use the proxy when using the KDE desktop (using the KDE configuration).

Where to get it To download the multiarch netinstall CD release you can use

• ftp://ftp.skolelinux.org/skolelinux-cd/wheezy/debian-edu-7.1+edu0~b1-CD.iso
• http://ftp.skolelinux.org/skolelinux-cd/wheezy/debian-edu-7.1+edu0~b1-CD.iso
• rsync -avzP ftp.skolelinux.org::skolelinux-cd/wheezy/debian-edu-7.1+edu0~b1-CD.iso .

The MD5SUM of this image is: 1e357f80b55e703523f2254adde6d78b
The SHA1SUM of this image is: 7157f9be5fd27c7694d713c6ecfed61c3edda3b2 To download the multiarch USB stick ISO release you can use

• ftp://ftp.skolelinux.org/skolelinux-cd/wheezy/debian-edu-7.1+edu0~b1-USB.iso
• http://ftp.skolelinux.org/skolelinux-cd/wheezy/debian-edu-7.1+edu0~b1-USB.iso
• rsync -avzP ftp.skolelinux.org::skolelinux-cd/wheezy/debian-edu-7.1+edu0~b1-USB.iso .

The MD5SUM of this image is: 7a8408ead59cf7e3cef25afb6e91590b
The SHA1SUM of this image is: f1817c031f02790d5edb3bfa0dcf8451088ad119 How to report bugs http://wiki.debian.org/DebianEdu/HowTo/ReportBugs

What you end up with is building a layer cake that doesn t add any value and slows down development. Especially when you re starting out and need to stay flexible you don t want to have too much code around and Rails is great for that. But adding a JSON API layer and basically a second application that runs on the client is annihilating this advantage for you.

Source.

As I slowly read my good friends wishing each other a good trip, telling they got home safely, and the IRC channels form thick drops of a bitter-sweet etheral substance, I cannot help feeling DebConf13 is over For me as well, from the distance. Many friends gave me warm greetings, and without being there, gave me that beautiful feeling of real community that Debian has given me for ten years already, since I met in real-life many of its developers at DebConf3 in Oslo. And yes, I have stated this far too many times I have attended every DebConf since (and worked organizing most of them). This year, over 300 people were gathered in Switzerland to enjoy the always most intense weeks of the year. This year, I was unable to attend due to calendar clashes. Even so, without the stress that organizers have, and thanks to the great work of the always-loved Video Team, I think I was able to be present at more sessions than at in any of the last few years. Oh, and for the readers of this blog who were not there Do you want to follow what was presented? You can download already the videos for all of the recorded presentations (that were, due to the planned coverage and the manageable size of the Video Team, about of the total scheduled sessions). And, as always, I was able to follow many very interesting talks and take part of a couple interesting meetings/BoF sessions. I still have a bit of catchup, partly due to the timezone difference (I was only at one of the sessions during the Swiss morning, at 02:30 local time, the pkg-ruby-extras team BoF). Anyway... Not being there, I surely was an avid consumer of the photos posted in the DebConf13 gallery, and will surely follow it for some more time as some of you upload your pending material. It was clear from the beginning that, no matter what your definition of consensus is, the chosen venue was beautiful. A beautiful place between the lake and the mountains where our sportiest guys had a very good share of morning runs, cycling sessions, competition sports of different types, outright plain fun for attendees of all sizes and all species... But, hey, wait! During a chat in the course of DebConf, a friend told me a bit worried that all this beauty and fun might make our dear and very important sponsors they are paying for a geek vacation, is it so? No, not at all. Not by a long stretch. And just looking at those same galleries makes it clear and obvious. After all, it's widely known that Debian is the operating system for the gurus. Simple: It's impossible to have all those geeks without getting amazing work done, in ways that even seem clich s (this last photo had Joey Hess explaining dpkg format version 3.0 (git) ideas, sketched after waking up at 3AM on the first sketching surface available to him). After all, Debian people are famous for their inclination to use any excuse to open their computers and hack away. We can find Debianers hacking in small spaces and also hacking out in the fields. But this time, people were able to hack indoors while enjoying the nature and hack outdoors under a tree. And, yes, one of the things that makes organizing DebConf worth it is, after eleven months having low-bandwidth meetings over IRC, having the opportunity to plan for the next days face to face, in a relaxed but work-full environment. Anyway, here at home I didn't sit idly just longing over them. How could I? We are just celebrating the Debian Project's 20th anniversary! http://gwolf.org/content/jonathan-host-and-organizer-rancho-electr-nico">Jonathan, a Debian enthusiast, student at my university, and collaborator for several free software-related collectives in Mexico City, invited me to the celebration at Rancho Electr nico (which I recently mentioned in this same blog). While I was unable to stay for the whole celebration, we had a very good time; I talked about some ways on how to contribute to Debian. Although I didn't have much of a presentation prepared for it, I feel it was successful and interesting for the attendees I just hope to start seeing some of them get into any of the ways for helping Debian soon. I also stayed as a listener and ocassional commenter for a talk on the Debian Project's history and goals, and to a presentation on a nifty electronic music programming tool called Supercollider (of course, available in Debian). Now, "regular" life should continue. For some value of "regular".

The Taskwarrior Team has just started a blog, kicking it off with a series of articles about development of Taskwarrior itself. Go, team! Almost from the moment I started using Taskwarrior (thanks to Jakub WIlk for an excellent job maintaining this) I knew I had finally found the todo system that I could love. Right away, I started hanging out with the Taskwarrior community at #taskwarrior @ irc.freenode.net and found out what an awesome bunch of people they are, both developers and users alike. I have plunged in with bug reports and feature requests, and am helping get more Taskwarrior-related things into Debian. In NEW right now I have uploaded several of the dependencies needed by my ITP of taskwarrior-web. It s looking like I ll have that finished later this month or early next month. Also, I have uploaded a wheezy backport of Taskwarrior itself (aka task ) which, if all goes well, enters the archive next week.

Laurent Bigonville reported Debian bug #720000 on Saturday August 17th 2013, against the drizzle package. This bug is already marked pending by Tobias Frost, the package maintainer. Bug #710000 was reported as of May 27th: 2 months and 21 days for 10,000 bugs. For once, this is a rate acceleration which we can probably explain by the release of wheezy and the work strongly resumed by many maintainers for the release of jessie. It is indeed interesting to see that this 720000th bug report happened nearly on Debian's 20th birthday. To make it short, we could then say that Debian had 36,000 bug reports every year in average (which is not exactly true as the BTS records start in 1996). Funnily also, this is the first time since I'm doing this recurrent post every 10,000 bugs that one happens *during* a DebConf, a few hours before DebConf 13 officially ends up.

I write a fair number of Unix command line programs, and I want to have test suites for them. A year and a half ago I wrote a tool to make that easier: cmdtest. However, it turned out to not be a particularly good one, primarily because it doesn't make it easy to write good test code. Tests and test code should be obviously correct, so you don't need a test suite for your test suite. Earlier this year, Daniel and I started a slow, but long discussion about this. We'd talk it over during many a lunch, figuring out what we actually want. We eventually converged on an approach based on for Unix command line programs, and based on Markdown documents. We then wrote that up as a README, and then took another month refining the README before writing any code. The naming discussion only took a week.

• We borrowed the given/when/then structure from BDD, and the idea of using that structure so that tests are specified using lightly structured text in a way that allowsnon-developers to participate.
• This borrowed structure gives a way to express sequences of things that happen, including setup and checks. We call these sequences test scenarios, and thus this approach to testing is called scenario testing.
• We embed the test scenarios, and the code that implements them, as code blocks in Markdown documents. This makes it easy to explain what is going on in the tests, when that's needed, right next to the tests. The Markdown files can easily be formatted as web pages or PDFs (among other formats), making it easy to share and discuss them with people who might not be comfortable accessing a git repository.

As a concrete example, here is a real test scenario, from Obnam, for testing its new FUSE plugin:

SCENARIO Browsing backups with FUSE plugin
ASSUMING user is in group fuse
GIVEN a live data directory
AND a 0 byte file called empty
AND a 1 byte file called one
AND a 4096 byte file called 4k
AND a 10485760 byte file called 10meg
AND a hardlink to 4k called 4k-hardlink
AND a symlink to 4k called 4k-symlink
AND a directory called some-dir
AND a symlink to ../4k called some-dir/other-symlink
WHEN live data is backed up
AND repository is fuse mounted
THEN latest generation can be copied correctly from fuse mount
FINALLY unmount repository

Some explanations:

• SCENARIO is the name of the scenario.
• ASSUMING is a condition for the entire scenario: the FUSE plugin can only be tested when the user can use FUSE.
• GIVEN is a setup step, WHEN is the step that does the thing that is being tested, and THEN checks that the right thing happened during WHEN.
• AND is an alias for the keyword on the previous line; it makes the test scenario seem more like normal English text
• FINALLY is for cleaning up afterwards, whether the scenario passed or not

Test scenarios like this are obviously not directly executable. We need to provide the code that actually executs each step. For this, yarn provides an IMPLEMENTS section, of which I will show one example: the creation of the files.

IMPLEMENTS GIVEN a (\d+) byte file called (\S+)
dd if=/dev/zero of="$DATADIR/live/$MATCH_2" bs=1 count="$MATCH_1"

(The other IMPLEMENTS sections for that Obnam test scenario are also only about a line of shell. The longest one is four lines, because it has an if statement that I like to format in a fairly verbose way. I'm skipping the others to avoid needless repetition.) The text after the keyword (GIVEN, WHEN, THEN, etc) in the test scenario is a single line of free-form text, which is matched by a regular expression in the IMPLEMENTS section. There must be exactly one matching IMPLEMENTS for each line in a test scenario. The body of IMPLEMENTS is a shell script snippet. Yarn provides a temporary directory for each scenario, and the path to that is given in the DATADIR environment variable. Yarn also cleans up the directory afterwards, unless the user running yarn requests otherwise, so tests that only change things in the temporary directory don't need any FINALLY steps. The above example uses FINALLY to unmount the FUSE filesystem at the end. The regular expressions can capture parts of the text in the scenario step, and these are provided to the shell script using environment variables: MATCH_1 etc. This is a simple, but powerful mechanism for implementing very flexible steps, which also means the number of steps can be kept down. The IMPLEMENTS sections are shared between all test scenarios. This makes it very easy to write new scenarios using already-existing steps, by choosing which ones to use, in which order, and making use of any variability in the steps that their implementations provide. Yarn is a new tool, and I expect it to mature in the coming months. It's included in the cmdtest Debian package, and is already in Debian jessie and unstable, for those who want to try it. (My personal apt repository has a .deb for wheezy as well; backports are welcome). The yarn scenario testing language itself is not set in stone, but Daniel and I both want to be careful about making changes to it. It's all meant to be very simple: an obviously correct testing tool is better than an intricate one. I've used yarn in several of my own little programs now, as has Daniel, and we've used it at work as well. Undoubtedly we'll find bugs in the implementation, and we're still feeling our way to figuring out how to write really excellent scenarios, and how to structure steps with the best possible taste. We'll also start designing a shell library that yarn provides to the IMPLEMENTS sections, for doing things that we find ourselves doing in many projects. It is already clear to me, however, that yarn is oh so much better than cmdtest could ever be. I'm even starting to express ideas for new tools as yarn test documents, so I'm clear what they should do. If and when I ever get to actually implementing the tools, I'll have a test suite for them already. See, for example, first attempts at specifying how a fast file transfer tool, fling, which is to use Linux-only system calls to reach wire speeds over wired LAN, and a tool to recompress compressed files with another compression tool. We hope yarn will prove to be useful for others as well. If you try it, and have suggestions or comments, please e-mail us.

The OASIS website has not been updated since a while. So I decide to take a shot at making more up to date. This blog post is about the pipeline I have but in place to automatically update the website. It is the first end to end 'continuous deployment' project I have achieved. Among the user visible changes:

• an invitation to circle the OASIS G+ page, which is now the official channel for small updates in OASIS.
• an invitation to fork the project on Github since it is now the official repository for OASIS.
• some link to documentation for the bleeding edge version of OASIS manual and API.

The OASIS website repository is also on Github. Feel free to fork it and send me pull request if you see any mistake. The website is still using a lot of markdown processed by pandoc. But there are some new technical features:

• no more index.php, we use a templating system to point to latest version
• we use a Jenkins job to generate the website daily or after any update in OASIS itself.

Since I start using quite quite a lot Python, I have decided to use it for this project. It has a lot of nice libraries and it helps me to quickly do something about the website (and provides plenty of idea to create equivalent tools in OCaml). The daily generation: Jenkins I have a Jenkins instance running, so I decided to use it to compile once a day the new website with updated documentation and links. This Jenkins instance also monitor changes of the OASIS source code. So I can do something even more precise: regenerate the website after every OASIS changes. I use the Jenkins instance to also generate a documentation tarball for OASIS manual and API. This helps a lot to be able to display the latest manual and API documentation. This way I can quickly browse the documentation and spot errors early. Another good point about Jenkins, is that it allows to store SSH credential. So I created a build user, with its own SSH key, in the OCaml Forge and I use it to publish the website at the end of the build. Right now Jenkins do the following:

• trigger a build of the OASIS website:
  • every day (cron)
  • when a push in OASIS website repository is detected
  • when a successful build of OASIS is achieved.
• get documentation artifact from the latest successful build of OASIS
• build the website
• publish it

Data gathering To build the website I need some data:

• documentation tarballs containing the API (HTML from OCamldoc) and manual (Markdow)
• list of OASIS version published
• links to each tarball (documentation and source tarball)

The OCaml Forge has a nice SOAP API. But one need to be logged in to access it. This is unfortunate, because I just want to access public data. The only way I found to gather my data was to scrape the OCaml Forge. Python has a very nice scraping library for that: beautifulsoup. I use beautifulsoup to parse the HTML downloaded from the Files tab of the OASIS project and extract all the relevant information. I use curl to download the documentation tarball (for released versions) and for the latest development version. Code Template Python has also a very nice library to process template: mako. Using the data I have gathered, I feed them to mako and I process all the .tmpl files in the repository to create matching files. Among the thing that I have transformed into template:

• the index.php has been transformed into a index.mkd.tmpl, it was a hackish PHP script scraping the RSS of updates before, it is now a clean template.
• robots.txt.tmpl, see the following section for explanation
• documentation.mkd.tmpl in order to list all version of documentation.

Fix documentation and indexing One of the problem of providing access to all versions of the documentation, is that people can end up reading an old version of the documentation. In order to prevent that, I use two different techniques:

• prevent search engine to index old version.
• warn the user that he is reading an old version.

To prevent search engine to index the file, I have created a robots.txt that list all URL of old documentation. This should be enough to prevent search engine to index the wrong page. To warn the user that he is reading the wrong version, I have added a box "you are not viewing the latest version". This part was tricky but beautifulsoup v4 provide a nice API to edit HTML in place. I just have to find the right CSS selector to define the position where I want to insert my warning box. Code Publish The ultimate goal of the project is the 'continuous deployment'. Rather than picking what version to deploy and do the process by hand, I let Jenkins deploy every version of it. Deploying the website used to be a simple rsync. But for this project I decided to use a fancier method. I spend a few hours deciding what was the best framework to do the automatic deployment. There are two main frameworks around: capistrano (Ruby) and fabric (Python). Fabric is written in Python, so I pick this one because it was a good fit for the project. Fabric biggest feature is to be a SSH wrapper. The fabric script is quite simple and to understand it, you just have to know that local run a local command and run run a command on the target host. The fabfile.py script do the following:

• create a local tarball using the OASIS website html/ directory
• upload the tarball to ssh.ocamlcore.org
• uncompress it and replace the htdocs/ directory of the oasis project
• remove the oldest tarballs, we keep a few versions to be able to perform a rollback.

Given this new process, the website is updated in 3 minutes automatically after a successful build of OASIS.

Finally, more bootlog-pr0n:

start ext2fs: Hurd server bootstrap: ext2fs[device:hd0s1] exec init proc auth
INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
Activating swap...done.
Checking root file system...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
/dev/hd0s1: clean, 44680/181056 files, 292234/723200 blocks
done.
Activating lvm and md swap...(default pager): Already paging to partition hd0s5!
done.
Checking file systems...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
done.
Cleaning up temporary files... /tmp.
Mounting local filesystems...done.
Activating swapfile swap...(default pager): Already paging to partition hd0s5!
done.
df: Warning: cannot read table of mounted file systems: No such file or directory
Cleaning up temporary files....
Configuring network interfaces...inetutils-ifconfig: invalid arguments
ifup: failed to open pid file /run/network/ifup-/dev/eth0.pid: No such file or directory
Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
Listening on Socket//dev/eth0
Sending on   Socket//dev/eth0
DHCPDISCOVER on /dev/eth0 to 255.255.255.255 port 67 interval 8
DHCPREQUEST on /dev/eth0 to 255.255.255.255 port 67
DHCPOFFER from 10.0.2.2
DHCPACK from 10.0.2.2
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
bound to 10.0.2.15 -- renewal in 34744 seconds.
done.
Cleaning up temporary files....
Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix.
INIT: Entering runlevel: 2
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Starting deferred execution scheduler: atd.
Starting periodic command scheduler: cron.
Starting system message bus: dbusFailed to set socket option"/var/run/dbus/system_bus_socket": Protocol not available.
Starting OpenBSD Secure Shell server: sshd.
unexpected ACK from keyboard
GNU 0.3 (debian) (console)
login: root
root@debian:~# shutdown -h now
Broadcast message from root@debian (console) (Fri Aug 16 20:02:47 2013):
The system is going down for system halt NOW!
INIT: Switching to runlevel: 0root@debian:~#
INIT: Sending processes the TERM signal
INIT: Sending processes the KILL signal
Using makefile-style concurrent boot in runlevel 0.
Stopping deferred execution scheduler: atd.
task c10f72a8 deallocating an invalid port 2098928, most probably a bug.
Asking all remaining processes to terminate...done.
All processes ended within 1 seconds...done.
Stopping enhanced syslogd: rsyslogd.
Deconfiguring network interfaces...ifdown: failed to open pid file /run/network/ifdown-/dev/eth0.pid: No such file or directory
Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
Listening on Socket//dev/eth0
Sending on   Socket//dev/eth0
/bin/sh: 1: ifconfig: not found
done.
Deactivating swap...swapoff: /dev/hd0s5: 177152k swap space
done.
Unmounting weak filesystems...umount: /etc/mtab: Warning: duplicate entry for device /dev/hd0s1 (/dev/cons)
umount: could not find entry for: /dev/cons
done.
Unmounting local filesystems...done.
mount: cannot remount /: Device or resource busy
Will now halt.
init: notifying pfinet of shutdown...init: notifying tmpfs swap of shutdown...init: notifying tmpfs swap of shutdown...init: notifying tmpfs swap of shutdown...init: notifying ext2fs device:hd0s1 of shutdown...init: halting Mach (flags 0x8)...
In tight loop: hit ctl-alt-del to reboot

(You might note that df complains about not being able to read the mtab file. That is because it has been built with _PATH_MOUNTED being /var/run/mtab. This will correct itself when the coreutils package is being rebuilt against a patched libc.) I spent my last two weeks with polishing my patch series, that meant a lot of package rebuilds and that means a lot of waiting (even more so on the Hurd, short version: fakeroot-hurd could be fast but is not yet working properly, fakeroot-tcp is slow) and thus some frustration ;) Also I had to pay special attention so that the upgraded packages could be installed without accidentally breaking anything in the process. Making sysvinit pid 1 is surprisingly tricky in this regard since it breaks the ABI and requires a libc fix that also worked with the current Hurd servers. Here are the patches:

• The mtab translator (v4) provides /proc/mounts: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00259.html
• When I wrote umount I thought I implemented all the flags used by the initscripts package. Turns out I forgot -d. umount.patch (v2): http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00018.html
• Make sysvinit pid 1 (sysvinit kinda requires that), fix killall5 (that was freezing the system before) (v2): http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00020.html
• Making sysvinit pid 1 breaks the ABI, since the glibc assumes /hurd/init to have pid 1: http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00046.html http://sourceware.org/ml/libc-alpha/2013-08/msg00205.html
• So now killall5 is not freezing the system anymore (actually it froze all processes but most importantly the procfs translator providing /proc and it then tried to walk over /proc which obviously blocked killall5 itself). Cool. But then it looks into /proc/*/stat for the fields start_code and end_code. So I dug into the exec server where the ELF decoding happens and handed this to the proc server, but the procfs translator has to make use of this: http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00017.html
• Patch series for the sysvinit package (v2). This contains a runsystem.sysv that can be enabled using update-alternatives --config runsystem and several small fixes: http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00038.html

So I had some spare time on my hand while waiting for numerous package rebuilds and I took this as an opportunity to read papers about Mach and to familiarize myself with mig, the Mach Interface Generator. While I have used it in the past, I had not yet looked at its implementation. And I had to patch the exec server, and there was both code implementing a questionable feature (on-demand unzipping of binaries) and code that was not even compiled (courtesy of the preprocessor) and had probably bit-rot by now. So I figured I could spend my time doing some cleanups:

• Cleanups for the exec server: http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00049.html
• Cleanups for mig: http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00055.html

I have rebuild all the necessary packages and uploaded them into an apt repository:

deb http://teythoon.cryptobitch.de/gsoc/heap/debian unstable main

Please use unstable for now. Also make sure that you have a recovery plan for your Debian/Hurd installation if anything goes wrong. For your convenience there's a seed tarball containing packages with the appropriate sources.list.d snippets and the repository key: https://teythoon.cryptobitch.de/gsoc/heap/debian/seed.tar If you want to switch to the new runsystem.sysv, do:

# update-alternatives --config runsystem

Whenever you switch runsystems, please use reboot-hurd to reboot the system. This is the most robust way. Known issues:

• procfs hardcodes the default kernel pid to 2. This breaks /proc/uptime and any program relying on it, most notably top and friends. Until this is properly fixed, you can do:

  # settrans -apfg /proc /hurd/procfs -c -k 3
  

• The mtab translator should probably try to filter out non-filesystem translators. df complains loudly about /dev/cons for example.

Next week I will address the network related issues. By now they are the source of most of the error messages in the bootlog.

Today is Debian's 20 anniversary. This anniversary would have been impossible without a strong community of users and developers. That's why for its commemoration, we asked the Debian community what Debian meant to them. Below you have a selection of the answers we got.

Since I run Debian on my computers, I do not play anymore to 3D shooting games, not because of the lack of Free 3D drivers, but because developing Debian is more fun and addictive. Debian is a truly community based distro which is devoted to FOSS ideas and standards. It perfectly works on high variety of hardware. Users from all over the world have been using Debian and contributing to it during 20 years. And I am proud to be one of them. Happy Birthday, Debian! When I considered switching to Linux I asked friends which distribution to choose. I was told skip other distributions that were considered more newbie-friendly at the time and go straight to Debian instead. It might be more work initially, they said, but it would save me the hassle of switching to Debian later on, which I would inevitably do. Turns out they were right. I started my Linux experience more than ten years ago with Debian and have yet to see a better Linux distribution. [...] You are a worldwide community of volunteers working together for 20 years now. To me this is an encouraging example and, given the sorry state of the world we live in, more important than the technical quality of the operating system. I wish you another successful twenty years and to stay as independent as you are. I initially started in Debian as it was an interesting technical challenge. Over the years the community and having good standards around what Free Software is has become more important. As a user of Debian for 14 years, and a former Debian Developer for 10, I would like to wish a happy birthday to the best project on the internet, and the best Linux distribution ever. Thanks for all of your support over the years! "Rock Solid Stability and Absolute Freedom" Thats what Debian means to me. Debian gives me a feeling that I am using the best linux has to offer. You know your machine is in safe hands. As a long time Debian user and sometimes supporter [...] I wish Debian all the best for next 20 years and beyond! Debian is the universal operating system. And it's free (and not just as in beer ;-) Debian is an awesome combination of obsessive high quality software and software freedom. It's a pleasure to be able to use and contribute to this project. Thanks to all for their excellent work! Cheers to 2**20 years more! [...] I am so very thankful to all of the people who have contributed to and continue to contribute to such a wonderful ecosystem of tools. I love the commitment to the long term goals of security, freedom and transparency with respect to the computer systems that we all use and rely upon and the information that we store on them. Debian is the operating system that makes me free. Debian is a family gathered around great idea. Its pure love.

Since at my git-dpm talk at debconf13 I got the speed a bit wrong and as the slides I uploaded to penta seem not to work from the html export, I've also uploaded the slides to http://git-dpm.alioth.debian.org/git-dpm-debconf13.pdf

Chromium is the free software build of the Google Chrome web browser. It s essentially the same as the Google code but will often be an older version, particularly when you get Chromium from Debian/Stable (or any other Linux distribution that doesn t track the latest versions all the time) and compare it to getting Chrome straight from Google. My wife is using Chromium on an AMD Opteron 1212 for all the usual web browsing tasks. Recently I ve noticed that it takes a lot of CPU time whenever she leaves a Google web site open, that can be Google+, Gmail, or Youtube. Web standards are complex and it s difficult to do everything the way that one might desire. Making a web browser that doesn t take 100% CPU time when the user is away from their desk may be a difficult technical challenge. Designing a web site that doesn t trigger such unwanted behavior in common web browsers might also be a challenge. But when one company produces both a web browser and some web sites that get a lot of traffic it s rather disappointing that they don t get this right. It could be that Google have fixed this in a more recent version of the Chrome source tree, and it could be that they fixed the browser code before rolling out a new version of Google+ etc which causes problems with the old version (which might explain why I ve never seen this problem). But even if that is the case it s still disappointing that they aren t supporting older versions. There is a real need for computers that don t need to be updated all the time, running a 3 month old Linux distribution such as Debian/Wheezy shouldn t be a problem. There s also a possibility that part of the cause of the problem is that an Opteron 1212 is a relatively slow CPU by today s standards and it s the slowest system I m currently supporting for serious desktop use. I don t even think it was one of the fastest CPUs available when it was released 4 years ago. But I think we should be able to expect systems to remain usable for more than 4 years. The Opteron 1212 system is a Dell PowerEdge tower server that is used as a workstation and a file server, so while I get desktop systems with faster CPUs for free I want to keep using the old PowerEdge server to avoid data corruption. As an aside I ve been storing important data on BTRFS for a year now and the only data loss I ve suffered has been due to a faulty DIMM. The filesystem checksums built in to modern filesystems such as BTRFS and ZFS mean that RAM corruption covers a greater portion of the risk to data integrity and the greater complexity of the data structures in such filesystems gives the possibility of corruption that can t be fixed without mkfs (as happened to me twice on the system with a bad DIMM). The consequences of such wasted CPU use are reduced CPU time for other programs which might be doing something useful, extra electricity use, and more noise from CPU cooling fans (which is particularly annoying for me in this case). Any suggesstions for reducing the CPU use of web browsers, particularly when idle?

The debconf BoF about gitify all things make we to want to report some of my git-annex use cases. The problem Calibre is a ebook manager that is available in debian. I use it to maintain my library, but also to dowload every day an epub version of a French newspaper and then put it on my kobo. Configuring git annex for this I wanted to use git-annex, so

    $ git init
    $ git annex init "some useful name"

But I don't want every thing in annex, because Calibre use some text file to save some metadata, so I used:

    $ git config annex.largefiles "include=* exclude=*.opf exclude=*.json"

then lets add everything

    $ git annex add *
    $ git add *
    $ git commit -m "first commit"

Calibre need read and write access on the its database, so let unlock it:

    $ git annex unlock metadata.db

On my other computer I only need to do

    $ git clone $user@$host:Calibre\ library
    $ cd Calibre\ library
    $ git annex init "another useful name"
    $ git annex get .
    $ git annex unlock metadata.db

The problem is that every time you will git annex sync, git annex will lock again the metadata.db, so lets unlock it automatically. I use git hooks, in .git/hooks/post-commit I have

    #!/bin/bash
    git annex edit metadata.db

don't forget to make this file executable

    $ chmod a+x .git/hooks/post-commit

Day to day operation

    $ git annex add .

Will put new file into the annex

    $ git add .

Will take care of the files that should no go into annex

    $ git annex sync

Will make the repositories exchange informations about all this, and make remote change local

    $ git annex get .

Will make remote book locally available Merge conflict You should not run calibre on the two computer simultaneously, or without syncing before it. If you do, you will have a conflict that git-annex will automatically solve by rename both of the file. You can then either:

• Choose one. If no books have been changed or added on one of the computer, to use the other metadata.db will not make you loose any information
• rebuild it. calibredb restore_database won't do it, but will tell you how to do it.

Checking the library You can use calibredb check_library to check you library is correct. If you use git for it, it will always tell you that it is not correct: there is this author ".git" it doesn't know about. Just don't care about it. Maybe this can be solved by using vcsh but apparently vcsh+git annex it not well tested yet. Automatic stuff I use mr to automatically run all this, but some config could be done (I believe) to have git annex copy --auto do what it should. There are also the git annex assistant for this kind of automatic synchronizations of contents, but I don't know if my automatic unlocking of one file will break this. It might be interesting to find someway to unlock and lock the library only when running calibre, a simple script to launch calibre will do that. Note that each time you will lock and unlock, you will have a new commit in git.