Calypso CalDAV/CardDAV/WebDAV for Android and Evolution
Ever since I bought my first Palm Pilot in 1997, I ve relied upon a
pocket-able device to carry a copy of my calendar and contacts, and for
that same database to be present on my laptop. I went through a long
list of Palm-compatible devices, including both the Palm Treo and Palm
Centro telephones. I even wrote a number of my own
Palm
applications. Years ago, it was pretty obvious that I d have
to find a new phone, but I was stuck looking for something that could
provide the same hot-sync functionality.
SyncML on the Series 40
I bought a Series 40 Nokia phone in Shanghai that promised SyncML
support. Given that I had seen numerous SyncML implementations for
Linux, it seemed like I should be able to get something to work.
SyncML on the Series 40 is a disaster the phone couldn t actually
store all of my contacts, and it couldn t hold half of the data
fields I used. So, that phone landed in the box and back to the Centro
I went.
SyncML on the N900
Nokia kindly sent me an N900 at some point, so I gave SyncML another
try. Given that the N900 runs evolution-data-server, and that I ve had
evolution-data-server running on my laptop, it seemed like I should be
in business. Well, almost. It took several days of hacking to fix bugs
in the evolution SyncML back end, then another several days fussing
with opensync.
I ended up abandoning direct synchronization as unworkable opensync
would sit in an infinite loop, or worse, trash my database
completely. I finally found syncml-ds-tool , which is a debugging
tool that comes with opensync. This tool simply synchronizes a set of
disk files, one per contact or calendar entry, with the phone. That
worked for well over a year. And then, a few months ago, Bluetooth on
the laptop stopped connecting with the phone s SyncML server. I d get
ECONNREFUSED every time I tried to use it. So much for the N900. DUN
still worked, mostly, although it too would get ECONNREFUSED at times,
but retrying seemed to make it work.
However, While the N900 SyncML solution worked, I discovered another
thing I wanted contacts and calendar entries stored in individual
files and revision controlled with git. This makes it reasonable to
delete stale calendar entries and know that they re never really gone,
just left behind in an older version of the calendar. And, if you mess
up, you can recover by poking at the database with git directly.
I switched back to the venerable Palm Centro; it turns out that
calendar and contacts are more important to me than being able to surf
the web on my phone. Alas, my Centro went swimming in August and has
passed on to the great electronics recycling house in the sky. I pulled
the SIM out and switched back to the N900.
I got my contacts imported on the N900 by copying files over the net
work; not a long-term strategy, but at least I had phone numbers
again. There was no hope for my calendar. I started looking for a
solution in earnest.
How about Android?
At this point in my story, I m sure you re asking why I didn t just
use one of the numerous Android phones that came through my hands. The
answer is simple my calendar and contacts are probably some of my
most personal data, and I m not willing to store them outside of my
direct control, for reasons similar to those which are driving the
development of the
FreedomBox.
When Android first came out, it could only talk to Google
services, which didn t meet my hard requirement for personal data
storage.
One of my co-workers had his Google account suspended for violating
the terms-of-service; he asked what he had done, but they wouldn t
say. He asked if he could get his data back, and they said no . They
invited him to create a new account, but it would not ever get any of
the old data. A few days later, he got a nice apologetic email letting
him know that they d made a mistake and that he hadn t, in fact,
violated any of the terms-of-service, and that his old account was
restored with all of his data intact. Wasn t that nice of Google?
WebOS?
I got a WebOS phone over the summer and discovered that while it had
multiple contact/calendar back ends, none of them used standard
protocols and so you only had the choice between multiple corporate
data centers, which isn t actually a choice at all.
Furthermore, the WebOS phone refused route PAN packets over the phone
network, even though I have a data plan which allows this. It s not
that it couldn t support it, it s that it refused.
A couple of weeks after the WebOS phone arrived, HP canceled all of
their WebOS hardware products, which made me less interested in trying
to solve this problem.
Android recovers
About the same time the WebOS phone arrived, I discovered that Google
had published enough information about the calendar/contacts internals
for Marten Gajda to write
CalDAV-Sync and
CardDAV-Sync. And then, Andrew McMillan
wrote
aCal, which is a
complete replacement for the built-in calendar and contacts
applications and supports CalDAV and CardDAV.
With two different standards-compliant solutions available, it seemed
like it might be time to try Android again.
I d love for CardDAV-Sync and CalDAV-Sync to become free software like
aCal is. Andrew makes money from aCal by offering it for sale via the
Android Market, while still publishing the sources for those who want
to build their own copy.
CalDAV/CardDAV on Linux
I think the most widely known CalDAV server for Linux is probably
DAViCal, a huge pile of PHP and SQL sitting
on top of Apache. I m sure it s suitable for running on a server
and being accessed over the internet, but I m not interested in that,
nor am I interested in having my laptop run Apache and PostgreSQL.
I found a tiny little CalDAV server,
Radicale,
which seemed like a lot better fit. It s written in Python and uses
the usual Python HTTP server infrastructure, which provides SSL and
authentication support along with some fairly convenient APIs for
parsing and generating HTML.
Before long, I discovered that Radicale was actually
too simple for
my needs. It stores the whole calendar in a single file, re-parsing it
whenever a request is made, so a calendar with just hundreds of
entries caused the server to slow down enough that evolution would
time-out when talking to it.
Also, Radicale doesn t actually parse the calendar entries completely,
it has some ad-hoc code that finds various pieces of data, but without
dealing with the whole syntax.
I started hacking at Radicale to see how far I could get. I changed
the storage code to store one event per file, then added hooks to use
git for change management. Then, I found a full vcalendar/vcard
parsing library in python,
vobject, which I used to
replace the ad-hoc parsing code. Finally, I added support for VCARD
entries as well, allowing the system to store both calendar and
contact information.
Introducing Calypso, a CardDAV/CalDAV server
With this much divergence from the original project, I ve figured I d
best rename things to avoid confusion, so I decided to call it
calypso , after a brief trip through the dictionary looking for names
starting with ca .
Calypso works with evolution, iceowl and the Android CalDAV/CardDAV
plugins. It does not yet work with aCal; for some reason aCal cannot
find any calendars on the server.
Calypso also supports importing calendar changes from the command
line, allowing you to integrate support into a text-based email
application like
notmuch or
mutt.
Calypso is available via git from git://keithp.com/git/calypso and is
distributed under the GPL (v3 or later). I still consider it a work
derived from Radicale, and so the code retains all of the Radicale
copyrights along with my own.
Using Calypso
Initial setup
Calypso runs as a regular user, all data are stored in
~/.config/calypso. To initialize calypso:
$ mkdir ~/.config/calypso ~/.config/calypso/calendars
$ cat > ~/.config/calypso/config << EOF
[server]
ssl=true
certificate=/home/keithp/.config/calypso/ssl/server.crt
key=/home/keithp/.config/calypso/ssl/server.key
[acl]
;type=htpasswd
type=fake
filename=/home/keithp/.config/calypso/passwd
Running calypso
Then run calypso:
$ python ./calypso.py
No, I haven t figured out how to install it
Creating new calendars
To add a new database:
$ mkdir -p ~/.config/calypso/calendars/private/my_calendar
$ cd ~/.config/calypso/calendars/private/my_calendar
$ git init
$ git commit --allow-empty -m'initialize new calendar'
The new calendar should now be visible as
https://localhost:5233/private/my_calendar
You can add files to the directory at any time; calypso will check the
directory mtime at each operation and update its internal state from
that on disk automatically when the directory changes.
Importing files
Given a set of files with VCALENDAR or VCARD entries, you can import
them with:
$ calypso --import private/my_calendar <filenames...>
This will update any changed entries and add any new ones.
ToDo list for calypso
- Document the config file contents.
- Make it installable
- Figure out what aCal wants
- Support calendar creation
More Android info
- Synker.
This provides a desktop widget which starts the sync process
running on a list of accounts. This makes it easy to manually
synchronize with the laptop when you are connected
- Contact Editor.
This lets you fully edit contacts synchronized over CardDAV. The
built-in contact editor doesn t let you change anything other than
the name for some reason.
I m running
cyanogenmod on my Nexus S
as that provides PAN support. With PAN, I can create a network link
between laptop and phone which doesn t depend on any local WiFi
infrastructure and which gives both phone and laptop static IP
addresses, allowing me to configure the sync URLs statically on the
phone. I d use mDNS, but Android doesn t bother to support that.
"Static program analysis" is a technique whereby a program is
verified for errors without actually running it. Finding bugs manually
with a debugger and one's brain is tedious, so every shortcut that can
help you avoid having to do so is great.
There are some commercial tools available to do such analysis, some
of which are rather expensive; but there are also some open source tools
available to do similar things. One of these is built into clang, the C compiler of the
LLVM project.
Using it is fairly simple. Instead of compiling something with
'make', compile it with 'scan-build make'. This will
set the CC (and similar) environment variable(s) so that before the
compiler is ran, the clang static-analysis checker is run over the very
same source code. The output of this checker is an HTML file with your
source, but with comments added to explain the bugs which the tool
found.
What does that mean? well, let's look at an example,
shall we? <iframe height="500px" src="http://barbershop.grep.be/~wouter/scan-view-example#LN1493" width="80%"></iframe>
The iframe above contains one out of three reports produced by a
scan-build run over the nbd source code (if there's no
iframe, someone scrubbed some HTML in your RSS reader. Just follow the
link just above instead). The other two are 'dead assignments', which
might mean that I'm currently depending on some undefined behaviour
(which would be bad), or it might mean that I'm being overly cautious
(which makes my code future-proof, which would be good), or it might
mean something else I still need to investigate. But this one is
pretty interesting.
In the example, there are eight, numbered, comments in the source
code. The first seven show the code path which scan-build took through
my code before getting at the eighth comment; and the eighth is where
things go bad. In this case, when going through the function as shown,
we have a NULL pointer dereference.
When looking at the scan-build output, it's important to realize a
few things. First, the code path shown may be just one
of a number of possible code paths. For instance, the null pointer
dereference would still happen if the phase function parameter
would not contain the NEG_INIT bit with the client
pointer set to NULL. However, clang does not show these other code
paths; this is presumably an optimization ("if we've already shown that
this kind of bug is possible at a particular location through one code
path, don't bother recording future instances of that very same bug at
the exact same location through another code path"). This means that
sometimes, some of the branches shown may be completely irrelevant to
the bug at hand. In this particular case, in fact, it's possible to show
the bug with just the eighth comment; the first seven are in fact
totally irrelevant.
Second, the fact that the clang static analyser found a bug
does not mean that it's possible to crash the application. Yet.
In this particular case, the negotiate function will never be
called with the NEG_INIT or NEG_MODERN bits not set,
and with the client parameter set to NULL. That's an
implicit assertion; there are a few ways in which this function may be
called, but the client parameter may only be NULL if
NEG_INIT and NEG_MODERN are both set at the same
time.
Since nbd-server doesn't currently call the
negotiate function in that way, it is not currently possible to
crash the server by exploiting this bug. But that doesn't mean it won't
ever be possible, nor that there isn't a bug in the code. We
may assume that the above rules are true, but we never check
it. Adding an assertion to that effect should make sure that no future
change to the code will accidentally introduce that error and cause a
NULL pointer dereference.
Is this a silly and useless precautionary measure? Not really.
Usually, bugs happen in code not because someone wasn't thinking
straight, but because there's so much going on inside a piece of
software that it can be too much for any programmer to remember. If a
function assumes that its parameters are within a given subset
of all possible states, but does not check that this is in fact true,
then when (not if) some future change incorrectly introduces a state
that is outside of the assumed states, things will break. And that's
Bad(TM).
(last update for 2011)
When attacking a process, one interesting target on the heap is the 
In the face of 
which is only a recent pleasure
One of my side projects for these last months was to enroll on the online 
A new release 0.9.8 of 
After 
Here's the 
It s the amount of the wire transfer that I made to 
Google Maps is now 
So. Recently I blogged about 
Back in 2007, 