Calypso CalDAV/CardDAV/WebDAV for Android and Evolution
Ever since I bought my first Palm Pilot in 1997, I ve relied upon a
pocket-able device to carry a copy of my calendar and contacts, and for
that same database to be present on my laptop. I went through a long
list of Palm-compatible devices, including both the Palm Treo and Palm
Centro telephones. I even wrote a number of my own
Palm
applications. Years ago, it was pretty obvious that I d have
to find a new phone, but I was stuck looking for something that could
provide the same hot-sync functionality.
SyncML on the Series 40
I bought a Series 40 Nokia phone in Shanghai that promised SyncML
support. Given that I had seen numerous SyncML implementations for
Linux, it seemed like I should be able to get something to work.
SyncML on the Series 40 is a disaster the phone couldn t actually
store all of my contacts, and it couldn t hold half of the data
fields I used. So, that phone landed in the box and back to the Centro
I went.
SyncML on the N900
Nokia kindly sent me an N900 at some point, so I gave SyncML another
try. Given that the N900 runs evolution-data-server, and that I ve had
evolution-data-server running on my laptop, it seemed like I should be
in business. Well, almost. It took several days of hacking to fix bugs
in the evolution SyncML back end, then another several days fussing
with opensync.
I ended up abandoning direct synchronization as unworkable opensync
would sit in an infinite loop, or worse, trash my database
completely. I finally found syncml-ds-tool , which is a debugging
tool that comes with opensync. This tool simply synchronizes a set of
disk files, one per contact or calendar entry, with the phone. That
worked for well over a year. And then, a few months ago, Bluetooth on
the laptop stopped connecting with the phone s SyncML server. I d get
ECONNREFUSED every time I tried to use it. So much for the N900. DUN
still worked, mostly, although it too would get ECONNREFUSED at times,
but retrying seemed to make it work.
However, While the N900 SyncML solution worked, I discovered another
thing I wanted contacts and calendar entries stored in individual
files and revision controlled with git. This makes it reasonable to
delete stale calendar entries and know that they re never really gone,
just left behind in an older version of the calendar. And, if you mess
up, you can recover by poking at the database with git directly.
I switched back to the venerable Palm Centro; it turns out that
calendar and contacts are more important to me than being able to surf
the web on my phone. Alas, my Centro went swimming in August and has
passed on to the great electronics recycling house in the sky. I pulled
the SIM out and switched back to the N900.
I got my contacts imported on the N900 by copying files over the net
work; not a long-term strategy, but at least I had phone numbers
again. There was no hope for my calendar. I started looking for a
solution in earnest.
How about Android?
At this point in my story, I m sure you re asking why I didn t just
use one of the numerous Android phones that came through my hands. The
answer is simple my calendar and contacts are probably some of my
most personal data, and I m not willing to store them outside of my
direct control, for reasons similar to those which are driving the
development of the
FreedomBox.
When Android first came out, it could only talk to Google
services, which didn t meet my hard requirement for personal data
storage.
One of my co-workers had his Google account suspended for violating
the terms-of-service; he asked what he had done, but they wouldn t
say. He asked if he could get his data back, and they said no . They
invited him to create a new account, but it would not ever get any of
the old data. A few days later, he got a nice apologetic email letting
him know that they d made a mistake and that he hadn t, in fact,
violated any of the terms-of-service, and that his old account was
restored with all of his data intact. Wasn t that nice of Google?
WebOS?
I got a WebOS phone over the summer and discovered that while it had
multiple contact/calendar back ends, none of them used standard
protocols and so you only had the choice between multiple corporate
data centers, which isn t actually a choice at all.
Furthermore, the WebOS phone refused route PAN packets over the phone
network, even though I have a data plan which allows this. It s not
that it couldn t support it, it s that it refused.
A couple of weeks after the WebOS phone arrived, HP canceled all of
their WebOS hardware products, which made me less interested in trying
to solve this problem.
Android recovers
About the same time the WebOS phone arrived, I discovered that Google
had published enough information about the calendar/contacts internals
for Marten Gajda to write
CalDAV-Sync and
CardDAV-Sync. And then, Andrew McMillan
wrote
aCal, which is a
complete replacement for the built-in calendar and contacts
applications and supports CalDAV and CardDAV.
With two different standards-compliant solutions available, it seemed
like it might be time to try Android again.
I d love for CardDAV-Sync and CalDAV-Sync to become free software like
aCal is. Andrew makes money from aCal by offering it for sale via the
Android Market, while still publishing the sources for those who want
to build their own copy.
CalDAV/CardDAV on Linux
I think the most widely known CalDAV server for Linux is probably
DAViCal, a huge pile of PHP and SQL sitting
on top of Apache. I m sure it s suitable for running on a server
and being accessed over the internet, but I m not interested in that,
nor am I interested in having my laptop run Apache and PostgreSQL.
I found a tiny little CalDAV server,
Radicale,
which seemed like a lot better fit. It s written in Python and uses
the usual Python HTTP server infrastructure, which provides SSL and
authentication support along with some fairly convenient APIs for
parsing and generating HTML.
Before long, I discovered that Radicale was actually
too simple for
my needs. It stores the whole calendar in a single file, re-parsing it
whenever a request is made, so a calendar with just hundreds of
entries caused the server to slow down enough that evolution would
time-out when talking to it.
Also, Radicale doesn t actually parse the calendar entries completely,
it has some ad-hoc code that finds various pieces of data, but without
dealing with the whole syntax.
I started hacking at Radicale to see how far I could get. I changed
the storage code to store one event per file, then added hooks to use
git for change management. Then, I found a full vcalendar/vcard
parsing library in python,
vobject, which I used to
replace the ad-hoc parsing code. Finally, I added support for VCARD
entries as well, allowing the system to store both calendar and
contact information.
Introducing Calypso, a CardDAV/CalDAV server
With this much divergence from the original project, I ve figured I d
best rename things to avoid confusion, so I decided to call it
calypso , after a brief trip through the dictionary looking for names
starting with ca .
Calypso works with evolution, iceowl and the Android CalDAV/CardDAV
plugins. It does not yet work with aCal; for some reason aCal cannot
find any calendars on the server.
Calypso also supports importing calendar changes from the command
line, allowing you to integrate support into a text-based email
application like
notmuch or
mutt.
Calypso is available via git from git://keithp.com/git/calypso and is
distributed under the GPL (v3 or later). I still consider it a work
derived from Radicale, and so the code retains all of the Radicale
copyrights along with my own.
Using Calypso
Initial setup
Calypso runs as a regular user, all data are stored in
~/.config/calypso. To initialize calypso:
$ mkdir ~/.config/calypso ~/.config/calypso/calendars
$ cat > ~/.config/calypso/config << EOF
[server]
ssl=true
certificate=/home/keithp/.config/calypso/ssl/server.crt
key=/home/keithp/.config/calypso/ssl/server.key
[acl]
;type=htpasswd
type=fake
filename=/home/keithp/.config/calypso/passwd
Running calypso
Then run calypso:
$ python ./calypso.py
No, I haven t figured out how to install it
Creating new calendars
To add a new database:
$ mkdir -p ~/.config/calypso/calendars/private/my_calendar
$ cd ~/.config/calypso/calendars/private/my_calendar
$ git init
$ git commit --allow-empty -m'initialize new calendar'
The new calendar should now be visible as
https://localhost:5233/private/my_calendar
You can add files to the directory at any time; calypso will check the
directory mtime at each operation and update its internal state from
that on disk automatically when the directory changes.
Importing files
Given a set of files with VCALENDAR or VCARD entries, you can import
them with:
$ calypso --import private/my_calendar <filenames...>
This will update any changed entries and add any new ones.
ToDo list for calypso
- Document the config file contents.
- Make it installable
- Figure out what aCal wants
- Support calendar creation
More Android info
- Synker.
This provides a desktop widget which starts the sync process
running on a list of accounts. This makes it easy to manually
synchronize with the laptop when you are connected
- Contact Editor.
This lets you fully edit contacts synchronized over CardDAV. The
built-in contact editor doesn t let you change anything other than
the name for some reason.
I m running
cyanogenmod on my Nexus S
as that provides PAN support. With PAN, I can create a network link
between laptop and phone which doesn t depend on any local WiFi
infrastructure and which gives both phone and laptop static IP
addresses, allowing me to configure the sync URLs statically on the
phone. I d use mDNS, but Android doesn t bother to support that.
Hello, World!
Today s hacks come remotely, on the first day of my long trek to Cleveland. As many debian readers know, debian URLs are usually transmitted over IRC or email with some snappy abbreviations. Usually, you ll see something like:
So there are two servers; let's call them srv1 and srv2. Srv1
contains a bunch of subversion repositories, but these are to be
migrated to srv2. Since the repositories are not (just) used for
ascii-only files, they're fairly big (several tens of gigabytes,
altogether), so copying them from one server to the other would take a
while. In order to make sure this would happen quickly, we had already
copied them over to the new server, so that on the final switch would be
quick (an rsync that would copy over just the new done
transactions).
That final switch was today. Only I didn't know that instead of just
testing, the customer had already started using one of the repositories
(and they'd forgotten to remind me), so the subversion repository
suddenly jumped backwards in time. In addition, the new server wasn't
being backed up yet (at least not for the subversion bits), so restoring
from backups wasn't an option. Oops.
Luckily the solution is fairly simple. You see, fsfs stores each
revision in a unique file; that means that as long as nobody has
committed to the repository yet (which they couldn't, since system users
aren't the same on both servers, so the webserver didn't have write
permissions on the repository after the rsync), nothing is lost. One
only needs to manually change the repository so that whatever subversion
thinks is the latest commit, actually is the latest commit.
That information is stored in a file called db/current
inside the repository. What's in that file depends on the repository
version, which is stored in a file called db/format in the
repository. For versions 1 and 2, the format is a single line with three
space-separated values, of which the first is the last revision number
used in the repository. The other two are counters that are used to give
transactions unique names; and they, too, need to be up-to-date. For
version 3 and above, the file contains only the revision number; there,
the other two are derived from that instead of having their own unique
number.
Figuring out the last used revision number in an fsfs repository is
ridiculously easy:
A recent trip to Switzerland made me dig out my prepaid card for UMTS usage
again. This resulted in some minor enhancements for Prepaid Manager. The 
This blog is 
The great thing about git and other distributed version control systems is
that once you clone (or fork) a repository, you have all the data. You
don't have to trust that Github will preserve it; everyone who develops
the project is a backup.
Github carries this principle quite far amoung the features they provide.
But not all the way. Today I have surveyed their features, and where the
data for each is stored.
[1] The PTS says the 25th. This is because it uses the day it receives the migration -email from trille, which was sent the day after.
Over the Christmas period I've not been doing too much.
December was a month that started out pretty well, in the first 10 days I had five models/friends/random-folk come to pose for me. (I have a few folk lined up for mid-January, but things tailed off quickly this month due to people having little free time).
I spent a while mulling over what I was doing with images, as I've recently been doing a fair number of more NSFW images - In December 
2012 will be the year where we release wheezy. So, it's time to start
fighting for one of my pet projects: bringing as many languages to 100%
completeness for debconf templates translation.
The reference for this is 
Do you believe that there will come a major inflation in the next months or years? If not, you can stop reading this article. Otherwise lets think what would be the right thing to do now. 
thanks to my vacations starting I had a bit more time this week for working
on RC bugs. here's the usual list; again: thanks to all the restless
patch providers!
When attacking a process, one interesting target on the heap is the 
In the face of 
which is only a recent pleasure
One of my side projects for these last months was to enroll on the online 
It s the amount of the wire transfer that I made to 
