I am
still looking for a solution that allows me to tag my mails similar to what Gmail
offers.
Having thought about this for quite a bit now, I can see three solutions,
implemented at different levels in the mail flow: as filesystem, patch to the
IMAP server (Dovecot), or patch to
mutt. Since this post is about the filesystem, I'll
concentrate on that, but I'll also touch upon the other two solutions briefly.
A bit of theory
First, let's dig into the theory and requirements for a bit. Instead of
sorting mails into folders (one mail can only ever live in one folder), I want
to tag my mails, such that a single message can have multiple tags and hence
be relevant to multiple categories. If you want to think in terms of folders,
then a single message could be attached to multiple folders, e.g. via hard
links. This is what
Mairix
approximates. The problem with this approach is that simple operations, such
as deleting a message (removing it from the enclosing folder) or marking it as
read (renaming the file) only affect the message in the current folder, not
its duplicates/clones in the other folders.
So it's best to stay with a single file per message and to associate tags with
it. Here, two approaches seem plausible: (a) extracting tags from the
individual message files, and (b) storing tags/message pairs in a separate
database. I have a gut feeling that (a) is the better approach, especially
since the files we're dealing with all have the same format. And in fact,
there is a semi-standard header for email messages to store such tags:
X-Label.
One more paragraph of theory: instead of simple tags, like Gmail provides,
I want hierarchical tags, for instance:
X-Label: Debian::pkg::mdadm
X-Label: Debian::popcon, PhD::Debian::stats
Hierarchical tags work similar to folders in that any message tagged as
Debian::pkg::mdadm also belongs to the
Debian::pkg and
Debian
categories. No rocket science.
mutt
Now it would probably be trivial to hack
mutt and add a new layer of
I mentioned earlier that this could be implemented in various places, and I'll
start at the highest level, with
mutt, the mail reader. For quite a while
now,
mutt provides search operators for the
X-Label header, such that
you can search or limit your mailbox e.g. like so:
~y \\<PhD::Debian\\> ~y
\\<popcon\\>. The problem with this approach is that it reuses existing
functionality, which I need on a regular basis: searching and limiting.
Furthermore, I think this approach is hackish and brittle and exposes too much
of the underlying workings.
filters, which would be used in addition to the user-specified limit. On
a high-level, the user would set a tag expression, like
PhD::Debian
::popcon, and may additionally choose to limit the display of messages to
those sent from the
debian.org domain:
~f debian.org$.
mutt would
then assemble the pieces and apply the filter
(~y \\<PhD::Debian\\> ~y
\\<popcon\\>) ~f debian.org$ to the current view.
With a reasonable user interface to specify or pick labels, as well as
a simple label editor, this would almost cut the mustard. Thanks to the
header_cache patch (which also
indexes the
X-Label header),
mutt fares reasonably well on Maildirs
with thousands of messages (well,
not always), and binding keys like
s (save) and
d (delete) to appropriate actions to add and remove labels determined from
the destination or the current tags filter, would make
mutt a force to be
reckoned with.
Dovecot
Unfortunately, implementing this in
mutt would mean that I'd be bound to
the mail reader forever (and I like to pretend that's not the case, haha!).
An alternative would be to implement this on the IMAP server, which already
provides IMAP folders which are not necessarily bound to physical, filesystem
folders and can intercept all commands that manipulate messages or their
status.
But while in theory this works fine, it'll break when offline tools, such as
offlineimap come into play, as those
(need to) instantiate every message they download, which brings us right back
to the problem with duplicate/cloned files discussed before.
A filesystem idea
Trying to stay independent of the tools I use, I started playing with the idea
of implementing mail message tagging at the filesystem level, using
Fuse. The user would run a daemon process against
a given Maildir holding
all mail (the base store) and export a virtual
directory hierarchy in which tags become Maildirs, backed by a cache of
X-Label:filename pairs.
- the root folder contains one Maildir dor every tag present in all messages
in the base store.
- each of those Maildirs, in addition to the new/cur/tmp directories,
provides a subdirectory for every tag minus the ones already in the path,
recursively. Navigating to /PhD/Debian::pkg would result in
a logical-AND query for all messages tagged PhD (and thus PhD::*)
and Debian::pkg. I am not sure how to implement OR and NOT queries yet.
- every Maildir in the entire hierarchy contains the messages satisfying the
query inherent in the path.
- the special directory /INCOMING can be used to write messages to the
hierarchy in such a way that the cache gets updated. Alternatively,
one could register inotify hooks with the kernel on the base store and
let others write to it as usual.
- the special directory /::UNTAGGED might enumerate all messages without
any tags.
The virtual directory hierarchy should also export the familiar filesystem
semantics:
- creating a file automatically adds the tags encoded in the file's path.
- copying, linking, and symlinking a file within the hierarchy are all
equivalent and cause the tag associated with the destination to be added to
the file.
- copying, linking, and symlinking into the hierarchy should probably result
in an error, unless the source is clearly an RFC822 message.
- mkdir creates a new tag in the cache, but without any files associated
with it.
- rmdir removes tags without any associated files.
- rm on a file removes a tag from a file. rm -r on a directory removes
the associated tag from all files (and the cache).
- moving files is equivalent to copy-remove, but needs to be atomic.
Unlike
tagfs, which uses an external
SQLite database for tags, this filesystem should not use an external
database for tags, but rather a run-time cache. That way, tags can be
propagated to other machines with IMAP synchronisation tools.
Initially, I thought such a fileystem could even be deployed "underneath" the
IMAP server, but then we obviously run into the same problems with offline
tools as before.
Potentially, such a filesystem could be incredibly powerful as it could do its
work for any user agent directly dealing with Maildirs (such as
mutt), and
it could also be used as basis for an IMAP server which is only ever accessed
synchronously (in online vs. offline mode). In the latter case, one would have
to ensure that filenames don't change across IMAP operations, or use the
message IDs instead from the start.
But as good as it may sound, getting it to work right will require quite some
attention, I think. Especially guaranteeing filesystem atomicity, as required
for Maildirs, might be impossible through the Fuse layer.
The sad end
Unfortunately, I don't have the time to implement any of the above, but if
I did, I would probably try to patch
mutt first, even though such a patch
may not make it into the upstream source any time soon. The filesystem idea
sounds cooler and cleaner, but it does not yet support OR/NOT queries, and it
will be quite a task to implement and debug. On the other hand, dealing with
Maildirs of several thousand messages in mutt is
a bit of a pain if those Maildirs are being updated
externally (e.g. mail is delivered to them).
At least the idea is out now. If you are interested and want to work on this,
please let me know so that I can link you up with other interested people.
And if you have any other input, please let me know. And no, I don't really
want to install
emacs to read my mail.
NP:
Dream Theater:
Train of Thought
Update: I've created a
mailing list for public discussion of this topic, so please subscribe if
you're interested.
Thomas Viemann suggested to use IMAP flags for the task, as these are
standardised. However, I see two show-stoppers here: first, I am not sure
whether it would be possible to set those flags from
procmail on delivery
(
sieve can supposedly do it, but I consider
sieve not adequate for my
needs). And second, offline tools, such as
offlineimap, which map IMAP
folders to filesystem Maildirs, would have exactly the same problem with
representing affiliation of a single message with multiple tags. You can read
up on the discussion in the
archives of the offlineimap mailing list,
and soon the new
mailtags list.
Martin Scholl had the idea to extend
dbmail with
the concepts of virtual folders (or stored searches). Naturally, as
database-backed mail system, it would be trivial and fast to implement, but
this approach bears two problems for me: first, it also falls short with
respect to offline tools (see above), and second it would require me to
replace my entire mail infrastructure. I am not even considering the
implications of using a database for mail storage.
Having passed all requirements for the Doctor of Philosophy, I had
my graduation today.