Antoine Beaupr : procmail considered harmful
TL;DR: procmail is a security liability and has been abandoned
upstream for the last two decades. If you are still using it, you
should probably drop everything and at least remove its SUID
flag. There are plenty of alternatives to chose from, and conversion
is a one-time, acceptable trade-off.
Procmail is unmaintained
procmail is unmaintained. The "Final release", according to
Wikipedia, dates back to September 10, 2001 (3.22). That release
was shipped in Debian since then, all the way back from Debian 3.0
"woody", twenty years ago.
Debian also ships 25 uploads on top of this, with 3.22-21 shipping the
"3.23pre" release that has been rumored since at least the November
2001, according to
Procmail is unmaintained
procmail is unmaintained. The "Final release", according to
Wikipedia, dates back to September 10, 2001 (3.22). That release
was shipped in Debian since then, all the way back from Debian 3.0
"woody", twenty years ago.
Debian also ships 25 uploads on top of this, with 3.22-21 shipping the
"3.23pre" release that has been rumored since at least the November
2001, according to debian/changelog at least:
procmail (3.22-1) unstable; urgency=low
* New upstream release, which uses the standard' format for Maildir
filenames and retries on name collision. It also contains some
bug fixes from the 3.23pre snapshot dated 2001-09-13.
* Removed sendmail' from the Recommends field, since we already
have exim' (the default Debian MTA) and mail-transport-agent'.
* Removed suidmanager support. Conflicts: suidmanager (<< 0.50).
* Added support for DEB_BUILD_OPTIONS in the source package.
* README.Maildir: Do not use locking on the example recipe,
since it's wrong to do so in this case.
-- Santiago Vila <sanvila@debian.org> Wed, 21 Nov 2001 09:40:20 +0100
All Debian suites from buster onwards ship the 3.22-26 release,
although the maintainer just pushed a 3.22-27 release to fix a seven
year old null pointer dereference, after this article was drafted.
Procmail is also shipped in all major distributions: Fedora
and its derivatives, Debian derivatives, Gentoo, Arch,
FreeBSD, OpenBSD. We all seem to be ignoring this problem.
The upstream website (http://procmail.org/) has been down since
about 2015, according to Debian bug #805864, with no change
since.
In effect, every distribution is currently maintaining its fork of
this dead program.
Note that, after filing a bug to keep Debian from shipping
procmail in a stable release again, I was told that the
Debian maintainer is apparently in contact with the upstream. And,
surprise! they still plan to release that fabled 3.23 release, which
has been now in "pre-release" for all those twenty years.
In fact, it turns out that 3.23 is considered released already, and
that the procmail author actually pushed a 3.24 release, codenamed
"Two decades of fixes". That amounts to 25 commits since 3.23pre
some of which address serious security issues, but none of which
address fundamental issues with the code base.
Procmail is insecure
By default, procmail is installed SUID root:mail in
Debian. There's no debconf or pre-seed setting that can change
this. There has been two bug reports against the Debian to make this
configurable (298058, 264011), but both were closed to say
that, basically, you should use dpkg-statoverride to change the
permissions on the binary.
So if anything, you should immediately run this command on any host
that you have procmail installed on:
dpkg-statoverride --update --add root root 0755 /usr/bin/procmail
Note that this might break email delivery. It might also not work at
all, thanks to usrmerge. Not sure. Yes, everything is on
fire. This is fine.
In my opinion, even assuming we keep procmail in Debian, that default
should be reversed. It should be up to people installing procmail to
assign it those dangerous permissions, after careful consideration of
the risk involved.
The last maintainer of procmail explicitly advised us (in that null
pointer dereference bug) and other projects (e.g. OpenBSD, in [2])
to stop shipping it, back in 2014. Quote:
Executive summary: delete the procmail port; the code is not safe
and should not be used as a basis for any further work.
I just read some of the code again this morning, after the original
author claimed that procmail was active again. It's still littered
with bizarre macros like:
#define bit_set(name,which,value) \
(value?(name[bit_index(which)] =bit_mask(which)):\
(name[bit_index(which)]&=~bit_mask(which)))
... from regexp.c, line 66 (yes, that's a custom regex
engine). Or this one:
#define jj (aleps.au.sopc)
It uses insecure functions like strcpy extensively. malloc()
is thrown around gotos like it's 1984 all over again. (To be fair,
it has been feeling like 1984 a lot lately, but that's another matter
entirely.)
That null pointer deref bug? It's fixed upstream now, in this
commit merged a few hours ago, which I presume might be in response
to my request to remove procmail from Debian.
So while that's nice, this is the just tip of the iceberg. I speculate
that one could easily find an exploitable crash in procmail if only by
running it through a fuzzer. But I don't need to speculate: procmail
had, for years, serious security issues that could possibly lead to
root privilege escalation, remotely exploitable if procmail is (as
it's designed to do) exposed to the network.
Maybe I'm overreacting. Maybe the procmail author will go through the
code base and do a proper rewrite. But I don't think that's what is in
the cards right now. What I expect will happen next is that people
will start fuzzing procmail, throw an uncountable number of bug
reports at it which will get fixed in a trickle while never fixing the
underlying, serious design flaws behind procmail.
Procmail has better alternatives
The reason this is so frustrating is that there are plenty of modern
alternatives to procmail which do not suffer from those problems.
Alternatives to procmail(1) itself are typically part of mail
servers. For example, Dovecot has its own LDA which implements
the standard Sieve language (RFC 5228). (Interestingly, Sieve
was published as RFC 3028 in 2001, before procmail was formally
abandoned.)
Courier also has "maildrop" which has its own filtering mechanism,
and there is fdm (2007) which is a fetchmail and procmail
replacement. Update: there's also mailprocessing, which is not
an LDA, but processing an existing folder. It was, however,
specifically designed to replace complex Procmail rules.
But procmail, of course, doesn't just ship procmail; that would just
be too easy. It ships mailstat(1) which we could probably ignore
because it only parses procmail log files. But more importantly, it
also ships:
lockfile(1) - conditional semaphore-file creatorformail(1) - mail (re)formatter
lockfile(1) already has a somewhat acceptable replacement in the form of
flock(1), part of util-linux (which is Essential, so installed on
any normal Debian system). It might not be a direct drop-in
replacement, but it should be close enough.
formail(1) is similar: the courier maildrop package ships
reformail(1) which is, presumably, a rewrite of formail. It's
unclear if it's a drop-in replacement, but it should probably possible
to port uses of formail to it easily.
Update: the maildrop package ships a SUID root binary (two,
even). So if you want only reformail(1), you might want to disable
that with:
dpkg-statoverride --update --add root root 0755 /usr/bin/lockmail.maildrop
dpkg-statoverride --update --add root root 0755 /usr/bin/maildrop
It would be perhaps better to have reformail(1) as a separate
package, see bug 1006903 for that discussion.
The real challenge is, of course, migrating those old .procmailrc
recipes to Sieve (basically). I added a few examples in the appendix
below. You might notice the Sieve examples are easier to read, which
is a nice added bonus.
Conclusion
There is really, absolutely, no reason to keep procmail in Debian, nor
should it be used anywhere at this point.
It's a great part of our computing history. May it be kept forever in
our museums and historical archives, but not in Debian, and certainly not
in actual release.
It's just a bomb waiting to go off. It is irresponsible for
distributions to keep shipping obsolete and insecure software like
this for unsuspecting users.
Note that I am grateful to the author, I really am: I used procmail
for decades and it served me well. But now, it's time to move, not
bring it back from the dead.
Appendix
Previous work
It's really weird to have to write this blog post. Back in 2016, I
rebuilt my mail setup at home and, to
my horror, discovered that procmail had been abandoned for 15 years at
that point, thanks to that LWN article from 2010. I would have
thought that I was the only weirdo still running procmail after all
those years and felt kind of embarrassed to only "now" switch to the
more modern (and, honestly, awesome) Sieve language.
But no. Since then, Debian shipped three major releases (stretch,
buster, and bullseye), all with the same vulnerable procmail
release.
Then, in early 2022, I found that, at work, we actually had procmail
installed everywhere, possibly because userdir-ldap was using
it for lockfile until 2019. I sent a patch to fix that and scrambled
to remove get rid of procmail everywhere. That took about a day.
But many other sites are now in that situation, possibly not imagining
they have this glaring security hole in their infrastructure.
Procmail to Sieve recipes
I'll collect a few Sieve equivalents to procmail recipes here. If you
have any additions, do contact me.
All Sieve examples below assume you drop the file in ~/.dovecot.sieve.
deliver mail to "plus" extension folder
Say you want to deliver user+foo@example.com to the folder
foo. You might write something like this in procmail:
MAILDIR=$HOME/Maildir/
DEFAULT=$MAILDIR
LOGFILE=$HOME/.procmail.log
VERBOSE=off
EXTENSION=$1 # Need to rename it - ?? does not like $1 nor 1
:0
* EXTENSION ?? [a-zA-Z0-9]+
.$EXTENSION/
That, in sieve language, would be:
require ["variables", "envelope", "fileinto", "subaddress"];
########################################################################
# wildcard +extension
# https://doc.dovecot.org/configuration_manual/sieve/examples/#plus-addressed-mail-filtering
if envelope :matches :detail "to" "*"
# Save name in $ name in all lowercase
set :lower "name" "$ 1 ";
fileinto "$ name ";
stop;
Subject into folder
This would file all mails with a Subject: line having FreshPorts
in it into the freshports folder, and mails from alternc.org
mailing lists into the alternc folder:
:0
## mailing list freshports
* ^Subject.*FreshPorts.*
.freshports/
:0
## mailing list alternc
* ^List-Post.*mailto:.*@alternc.org.*
.alternc/
Equivalent Sieve:
if header :contains "subject" "FreshPorts"
fileinto "freshports";
elsif header :contains "List-Id" "alternc.org"
fileinto "alternc";
Mail sent to root to a reports folder
This double rule:
:0
* ^Subject: Cron
* ^From: .*root@
.rapports/
Would look something like this in Sieve:
if header :comparator "i;octet" :contains "Subject" "Cron"
if header :regex :comparator "i;octet" "From" ".*root@"
fileinto "rapports";
Note that this is what the automated converted does (below). It's not
very readable, but it works.
Bulk email
I didn't have an equivalent of this in procmail, but that's something
I did in Sieve:
if header :contains "Precedence" "bulk"
fileinto "bulk";
Any mailing list
This is another rule I didn't have in procmail but I found handy and
easy to do in Sieve:
if exists "List-Id"
fileinto "lists";
This or that
I wouldn't remember how to do this in procmail either, but that's an
easy one in Sieve:
if anyof (header :contains "from" "example.com",
header :contains ["to", "cc"] "anarcat@example.com")
fileinto "example";
You can even pile up a bunch of options together to have one big rule
with multiple patterns:
if anyof (exists "X-Cron-Env",
header :contains ["subject"] ["security run output",
"monthly run output",
"daily run output",
"weekly run output",
"Debian Package Updates",
"Debian package update",
"daily mail stats",
"Anacron job",
"nagios",
"changes report",
"run output",
"[Systraq]",
"Undelivered mail",
"Postfix SMTP server: errors from",
"backupninja",
"DenyHosts report",
"Debian security status",
"apt-listchanges"
],
header :contains "Auto-Submitted" "auto-generated",
envelope :contains "from" ["nagios@",
"logcheck@",
"root@"])
fileinto "rapports";
Automated script
There is a procmail2sieve.pl script floating around, and
mentioned in the dovecot documentation. It didn't work very well
for me: I could use it for small things, but I mostly wrote the sieve
file from scratch.
Progressive migration
Enrico Zini has progressively migrated his procmail setup to Sieve
using a clever way: he hooked procmail inside sieve so that he could
deliver to the Dovecot LDA and progressively migrate rules one by
one, without having a "flag day".
See this explanatory blog post for the details, which also shows
how to configure Dovecot as an LMTP server with Postfix.
Other examples
The Dovecot sieve examples are numerous and also quite useful. At
the time of writing, they include virus scanning and spam filtering,
vacation auto-replies, includes, archival, and flags.
Harmful considered harmful
I am aware that the "considered harmful" title has a long and
controversial history, being considered harmful in itself (by some
people who are obviously not afraid of contradictions).
I have nevertheless deliberately chosen that title, partly to make
sure this article gets maximum visibility, but more specifically
because I do not have doubts at this moment that procmail is, clearly,
a bad idea at this moment in history.
Developing story
I must also add that, incredibly, this story has changed while writing
it. This article is derived from this bug I filed in Debian to,
quite frankly, kick procmail out of Debian. But filing the bug had the
interesting effect of pushing the upstream into action: as mentioned
above, they have apparently made a new release and merged a bunch of
patches in a new git repository.
This doesn't change much of the above, at this moment. If anything
significant comes out of this effort, I will try to update this
article to reflect the situation. I am actually happy to retract the
claims in this article if it turns out that procmail is a stellar
example of defensive programming and survives fuzzing attacks. But at
this moment, I'm pretty confident that will not happen, at least not
in scope of the next Debian release cycle.
procmail (3.22-1) unstable; urgency=low
* New upstream release, which uses the standard' format for Maildir
filenames and retries on name collision. It also contains some
bug fixes from the 3.23pre snapshot dated 2001-09-13.
* Removed sendmail' from the Recommends field, since we already
have exim' (the default Debian MTA) and mail-transport-agent'.
* Removed suidmanager support. Conflicts: suidmanager (<< 0.50).
* Added support for DEB_BUILD_OPTIONS in the source package.
* README.Maildir: Do not use locking on the example recipe,
since it's wrong to do so in this case.
-- Santiago Vila <sanvila@debian.org> Wed, 21 Nov 2001 09:40:20 +0100
root:mail in
Debian. There's no debconf or pre-seed setting that can change
this. There has been two bug reports against the Debian to make this
configurable (298058, 264011), but both were closed to say
that, basically, you should use dpkg-statoverride to change the
permissions on the binary.
So if anything, you should immediately run this command on any host
that you have procmail installed on:
dpkg-statoverride --update --add root root 0755 /usr/bin/procmail
Executive summary: delete the procmail port; the code is not safe and should not be used as a basis for any further work.I just read some of the code again this morning, after the original author claimed that procmail was active again. It's still littered with bizarre macros like:
#define bit_set(name,which,value) \
(value?(name[bit_index(which)] =bit_mask(which)):\
(name[bit_index(which)]&=~bit_mask(which)))
#define jj (aleps.au.sopc)
strcpy extensively. malloc()
is thrown around gotos like it's 1984 all over again. (To be fair,
it has been feeling like 1984 a lot lately, but that's another matter
entirely.)
That null pointer deref bug? It's fixed upstream now, in this
commit merged a few hours ago, which I presume might be in response
to my request to remove procmail from Debian.
So while that's nice, this is the just tip of the iceberg. I speculate
that one could easily find an exploitable crash in procmail if only by
running it through a fuzzer. But I don't need to speculate: procmail
had, for years, serious security issues that could possibly lead to
root privilege escalation, remotely exploitable if procmail is (as
it's designed to do) exposed to the network.
Maybe I'm overreacting. Maybe the procmail author will go through the
code base and do a proper rewrite. But I don't think that's what is in
the cards right now. What I expect will happen next is that people
will start fuzzing procmail, throw an uncountable number of bug
reports at it which will get fixed in a trickle while never fixing the
underlying, serious design flaws behind procmail.
Procmail has better alternatives
The reason this is so frustrating is that there are plenty of modern
alternatives to procmail which do not suffer from those problems.
Alternatives to procmail(1) itself are typically part of mail
servers. For example, Dovecot has its own LDA which implements
the standard Sieve language (RFC 5228). (Interestingly, Sieve
was published as RFC 3028 in 2001, before procmail was formally
abandoned.)
Courier also has "maildrop" which has its own filtering mechanism,
and there is fdm (2007) which is a fetchmail and procmail
replacement. Update: there's also mailprocessing, which is not
an LDA, but processing an existing folder. It was, however,
specifically designed to replace complex Procmail rules.
But procmail, of course, doesn't just ship procmail; that would just
be too easy. It ships mailstat(1) which we could probably ignore
because it only parses procmail log files. But more importantly, it
also ships:
lockfile(1) - conditional semaphore-file creatorformail(1) - mail (re)formatter
lockfile(1) already has a somewhat acceptable replacement in the form of
flock(1), part of util-linux (which is Essential, so installed on
any normal Debian system). It might not be a direct drop-in
replacement, but it should be close enough.
formail(1) is similar: the courier maildrop package ships
reformail(1) which is, presumably, a rewrite of formail. It's
unclear if it's a drop-in replacement, but it should probably possible
to port uses of formail to it easily.
Update: the maildrop package ships a SUID root binary (two,
even). So if you want only reformail(1), you might want to disable
that with:
dpkg-statoverride --update --add root root 0755 /usr/bin/lockmail.maildrop
dpkg-statoverride --update --add root root 0755 /usr/bin/maildrop
It would be perhaps better to have reformail(1) as a separate
package, see bug 1006903 for that discussion.
The real challenge is, of course, migrating those old .procmailrc
recipes to Sieve (basically). I added a few examples in the appendix
below. You might notice the Sieve examples are easier to read, which
is a nice added bonus.
Conclusion
There is really, absolutely, no reason to keep procmail in Debian, nor
should it be used anywhere at this point.
It's a great part of our computing history. May it be kept forever in
our museums and historical archives, but not in Debian, and certainly not
in actual release.
It's just a bomb waiting to go off. It is irresponsible for
distributions to keep shipping obsolete and insecure software like
this for unsuspecting users.
Note that I am grateful to the author, I really am: I used procmail
for decades and it served me well. But now, it's time to move, not
bring it back from the dead.
Appendix
Previous work
It's really weird to have to write this blog post. Back in 2016, I
rebuilt my mail setup at home and, to
my horror, discovered that procmail had been abandoned for 15 years at
that point, thanks to that LWN article from 2010. I would have
thought that I was the only weirdo still running procmail after all
those years and felt kind of embarrassed to only "now" switch to the
more modern (and, honestly, awesome) Sieve language.
But no. Since then, Debian shipped three major releases (stretch,
buster, and bullseye), all with the same vulnerable procmail
release.
Then, in early 2022, I found that, at work, we actually had procmail
installed everywhere, possibly because userdir-ldap was using
it for lockfile until 2019. I sent a patch to fix that and scrambled
to remove get rid of procmail everywhere. That took about a day.
But many other sites are now in that situation, possibly not imagining
they have this glaring security hole in their infrastructure.
Procmail to Sieve recipes
I'll collect a few Sieve equivalents to procmail recipes here. If you
have any additions, do contact me.
All Sieve examples below assume you drop the file in ~/.dovecot.sieve.
deliver mail to "plus" extension folder
Say you want to deliver user+foo@example.com to the folder
foo. You might write something like this in procmail:
MAILDIR=$HOME/Maildir/
DEFAULT=$MAILDIR
LOGFILE=$HOME/.procmail.log
VERBOSE=off
EXTENSION=$1 # Need to rename it - ?? does not like $1 nor 1
:0
* EXTENSION ?? [a-zA-Z0-9]+
.$EXTENSION/
That, in sieve language, would be:
require ["variables", "envelope", "fileinto", "subaddress"];
########################################################################
# wildcard +extension
# https://doc.dovecot.org/configuration_manual/sieve/examples/#plus-addressed-mail-filtering
if envelope :matches :detail "to" "*"
# Save name in $ name in all lowercase
set :lower "name" "$ 1 ";
fileinto "$ name ";
stop;
Subject into folder
This would file all mails with a Subject: line having FreshPorts
in it into the freshports folder, and mails from alternc.org
mailing lists into the alternc folder:
:0
## mailing list freshports
* ^Subject.*FreshPorts.*
.freshports/
:0
## mailing list alternc
* ^List-Post.*mailto:.*@alternc.org.*
.alternc/
Equivalent Sieve:
if header :contains "subject" "FreshPorts"
fileinto "freshports";
elsif header :contains "List-Id" "alternc.org"
fileinto "alternc";
Mail sent to root to a reports folder
This double rule:
:0
* ^Subject: Cron
* ^From: .*root@
.rapports/
Would look something like this in Sieve:
if header :comparator "i;octet" :contains "Subject" "Cron"
if header :regex :comparator "i;octet" "From" ".*root@"
fileinto "rapports";
Note that this is what the automated converted does (below). It's not
very readable, but it works.
Bulk email
I didn't have an equivalent of this in procmail, but that's something
I did in Sieve:
if header :contains "Precedence" "bulk"
fileinto "bulk";
Any mailing list
This is another rule I didn't have in procmail but I found handy and
easy to do in Sieve:
if exists "List-Id"
fileinto "lists";
This or that
I wouldn't remember how to do this in procmail either, but that's an
easy one in Sieve:
if anyof (header :contains "from" "example.com",
header :contains ["to", "cc"] "anarcat@example.com")
fileinto "example";
You can even pile up a bunch of options together to have one big rule
with multiple patterns:
if anyof (exists "X-Cron-Env",
header :contains ["subject"] ["security run output",
"monthly run output",
"daily run output",
"weekly run output",
"Debian Package Updates",
"Debian package update",
"daily mail stats",
"Anacron job",
"nagios",
"changes report",
"run output",
"[Systraq]",
"Undelivered mail",
"Postfix SMTP server: errors from",
"backupninja",
"DenyHosts report",
"Debian security status",
"apt-listchanges"
],
header :contains "Auto-Submitted" "auto-generated",
envelope :contains "from" ["nagios@",
"logcheck@",
"root@"])
fileinto "rapports";
Automated script
There is a procmail2sieve.pl script floating around, and
mentioned in the dovecot documentation. It didn't work very well
for me: I could use it for small things, but I mostly wrote the sieve
file from scratch.
Progressive migration
Enrico Zini has progressively migrated his procmail setup to Sieve
using a clever way: he hooked procmail inside sieve so that he could
deliver to the Dovecot LDA and progressively migrate rules one by
one, without having a "flag day".
See this explanatory blog post for the details, which also shows
how to configure Dovecot as an LMTP server with Postfix.
Other examples
The Dovecot sieve examples are numerous and also quite useful. At
the time of writing, they include virus scanning and spam filtering,
vacation auto-replies, includes, archival, and flags.
Harmful considered harmful
I am aware that the "considered harmful" title has a long and
controversial history, being considered harmful in itself (by some
people who are obviously not afraid of contradictions).
I have nevertheless deliberately chosen that title, partly to make
sure this article gets maximum visibility, but more specifically
because I do not have doubts at this moment that procmail is, clearly,
a bad idea at this moment in history.
Developing story
I must also add that, incredibly, this story has changed while writing
it. This article is derived from this bug I filed in Debian to,
quite frankly, kick procmail out of Debian. But filing the bug had the
interesting effect of pushing the upstream into action: as mentioned
above, they have apparently made a new release and merged a bunch of
patches in a new git repository.
This doesn't change much of the above, at this moment. If anything
significant comes out of this effort, I will try to update this
article to reflect the situation. I am actually happy to retract the
claims in this article if it turns out that procmail is a stellar
example of defensive programming and survives fuzzing attacks. But at
this moment, I'm pretty confident that will not happen, at least not
in scope of the next Debian release cycle.
lockfile(1) - conditional semaphore-file creatorformail(1) - mail (re)formattermaildrop package ships a SUID root binary (two,
even). So if you want only reformail(1), you might want to disable
that with:
dpkg-statoverride --update --add root root 0755 /usr/bin/lockmail.maildrop
dpkg-statoverride --update --add root root 0755 /usr/bin/maildrop
reformail(1) as a separate
package, see bug 1006903 for that discussion.Appendix
Previous work
It's really weird to have to write this blog post. Back in 2016, I
rebuilt my mail setup at home and, to
my horror, discovered that procmail had been abandoned for 15 years at
that point, thanks to that LWN article from 2010. I would have
thought that I was the only weirdo still running procmail after all
those years and felt kind of embarrassed to only "now" switch to the
more modern (and, honestly, awesome) Sieve language.
But no. Since then, Debian shipped three major releases (stretch,
buster, and bullseye), all with the same vulnerable procmail
release.
Then, in early 2022, I found that, at work, we actually had procmail
installed everywhere, possibly because userdir-ldap was using
it for lockfile until 2019. I sent a patch to fix that and scrambled
to remove get rid of procmail everywhere. That took about a day.
But many other sites are now in that situation, possibly not imagining
they have this glaring security hole in their infrastructure.
Procmail to Sieve recipes
I'll collect a few Sieve equivalents to procmail recipes here. If you
have any additions, do contact me.
All Sieve examples below assume you drop the file in ~/.dovecot.sieve.
deliver mail to "plus" extension folder
Say you want to deliver user+foo@example.com to the folder
foo. You might write something like this in procmail:
MAILDIR=$HOME/Maildir/
DEFAULT=$MAILDIR
LOGFILE=$HOME/.procmail.log
VERBOSE=off
EXTENSION=$1 # Need to rename it - ?? does not like $1 nor 1
:0
* EXTENSION ?? [a-zA-Z0-9]+
.$EXTENSION/
That, in sieve language, would be:
require ["variables", "envelope", "fileinto", "subaddress"];
########################################################################
# wildcard +extension
# https://doc.dovecot.org/configuration_manual/sieve/examples/#plus-addressed-mail-filtering
if envelope :matches :detail "to" "*"
# Save name in $ name in all lowercase
set :lower "name" "$ 1 ";
fileinto "$ name ";
stop;
Subject into folder
This would file all mails with a Subject: line having FreshPorts
in it into the freshports folder, and mails from alternc.org
mailing lists into the alternc folder:
:0
## mailing list freshports
* ^Subject.*FreshPorts.*
.freshports/
:0
## mailing list alternc
* ^List-Post.*mailto:.*@alternc.org.*
.alternc/
Equivalent Sieve:
if header :contains "subject" "FreshPorts"
fileinto "freshports";
elsif header :contains "List-Id" "alternc.org"
fileinto "alternc";
Mail sent to root to a reports folder
This double rule:
:0
* ^Subject: Cron
* ^From: .*root@
.rapports/
Would look something like this in Sieve:
if header :comparator "i;octet" :contains "Subject" "Cron"
if header :regex :comparator "i;octet" "From" ".*root@"
fileinto "rapports";
Note that this is what the automated converted does (below). It's not
very readable, but it works.
Bulk email
I didn't have an equivalent of this in procmail, but that's something
I did in Sieve:
if header :contains "Precedence" "bulk"
fileinto "bulk";
Any mailing list
This is another rule I didn't have in procmail but I found handy and
easy to do in Sieve:
if exists "List-Id"
fileinto "lists";
This or that
I wouldn't remember how to do this in procmail either, but that's an
easy one in Sieve:
if anyof (header :contains "from" "example.com",
header :contains ["to", "cc"] "anarcat@example.com")
fileinto "example";
You can even pile up a bunch of options together to have one big rule
with multiple patterns:
if anyof (exists "X-Cron-Env",
header :contains ["subject"] ["security run output",
"monthly run output",
"daily run output",
"weekly run output",
"Debian Package Updates",
"Debian package update",
"daily mail stats",
"Anacron job",
"nagios",
"changes report",
"run output",
"[Systraq]",
"Undelivered mail",
"Postfix SMTP server: errors from",
"backupninja",
"DenyHosts report",
"Debian security status",
"apt-listchanges"
],
header :contains "Auto-Submitted" "auto-generated",
envelope :contains "from" ["nagios@",
"logcheck@",
"root@"])
fileinto "rapports";
Automated script
There is a procmail2sieve.pl script floating around, and
mentioned in the dovecot documentation. It didn't work very well
for me: I could use it for small things, but I mostly wrote the sieve
file from scratch.
Progressive migration
Enrico Zini has progressively migrated his procmail setup to Sieve
using a clever way: he hooked procmail inside sieve so that he could
deliver to the Dovecot LDA and progressively migrate rules one by
one, without having a "flag day".
See this explanatory blog post for the details, which also shows
how to configure Dovecot as an LMTP server with Postfix.
Other examples
The Dovecot sieve examples are numerous and also quite useful. At
the time of writing, they include virus scanning and spam filtering,
vacation auto-replies, includes, archival, and flags.
Harmful considered harmful
I am aware that the "considered harmful" title has a long and
controversial history, being considered harmful in itself (by some
people who are obviously not afraid of contradictions).
I have nevertheless deliberately chosen that title, partly to make
sure this article gets maximum visibility, but more specifically
because I do not have doubts at this moment that procmail is, clearly,
a bad idea at this moment in history.
Developing story
I must also add that, incredibly, this story has changed while writing
it. This article is derived from this bug I filed in Debian to,
quite frankly, kick procmail out of Debian. But filing the bug had the
interesting effect of pushing the upstream into action: as mentioned
above, they have apparently made a new release and merged a bunch of
patches in a new git repository.
This doesn't change much of the above, at this moment. If anything
significant comes out of this effort, I will try to update this
article to reflect the situation. I am actually happy to retract the
claims in this article if it turns out that procmail is a stellar
example of defensive programming and survives fuzzing attacks. But at
this moment, I'm pretty confident that will not happen, at least not
in scope of the next Debian release cycle.
procmail
installed everywhere, possibly because userdir-ldap was using
it for lockfile until 2019. I sent a patch to fix that and scrambled
to remove get rid of procmail everywhere. That took about a day.
But many other sites are now in that situation, possibly not imagining
they have this glaring security hole in their infrastructure.
Procmail to Sieve recipes
I'll collect a few Sieve equivalents to procmail recipes here. If you
have any additions, do contact me.
All Sieve examples below assume you drop the file in ~/.dovecot.sieve.
deliver mail to "plus" extension folder
Say you want to deliver user+foo@example.com to the folder
foo. You might write something like this in procmail:
MAILDIR=$HOME/Maildir/
DEFAULT=$MAILDIR
LOGFILE=$HOME/.procmail.log
VERBOSE=off
EXTENSION=$1 # Need to rename it - ?? does not like $1 nor 1
:0
* EXTENSION ?? [a-zA-Z0-9]+
.$EXTENSION/
That, in sieve language, would be:
require ["variables", "envelope", "fileinto", "subaddress"];
########################################################################
# wildcard +extension
# https://doc.dovecot.org/configuration_manual/sieve/examples/#plus-addressed-mail-filtering
if envelope :matches :detail "to" "*"
# Save name in $ name in all lowercase
set :lower "name" "$ 1 ";
fileinto "$ name ";
stop;
Subject into folder
This would file all mails with a Subject: line having FreshPorts
in it into the freshports folder, and mails from alternc.org
mailing lists into the alternc folder:
:0
## mailing list freshports
* ^Subject.*FreshPorts.*
.freshports/
:0
## mailing list alternc
* ^List-Post.*mailto:.*@alternc.org.*
.alternc/
Equivalent Sieve:
if header :contains "subject" "FreshPorts"
fileinto "freshports";
elsif header :contains "List-Id" "alternc.org"
fileinto "alternc";
Mail sent to root to a reports folder
This double rule:
:0
* ^Subject: Cron
* ^From: .*root@
.rapports/
Would look something like this in Sieve:
if header :comparator "i;octet" :contains "Subject" "Cron"
if header :regex :comparator "i;octet" "From" ".*root@"
fileinto "rapports";
Note that this is what the automated converted does (below). It's not
very readable, but it works.
Bulk email
I didn't have an equivalent of this in procmail, but that's something
I did in Sieve:
if header :contains "Precedence" "bulk"
fileinto "bulk";
Any mailing list
This is another rule I didn't have in procmail but I found handy and
easy to do in Sieve:
if exists "List-Id"
fileinto "lists";
This or that
I wouldn't remember how to do this in procmail either, but that's an
easy one in Sieve:
if anyof (header :contains "from" "example.com",
header :contains ["to", "cc"] "anarcat@example.com")
fileinto "example";
You can even pile up a bunch of options together to have one big rule
with multiple patterns:
if anyof (exists "X-Cron-Env",
header :contains ["subject"] ["security run output",
"monthly run output",
"daily run output",
"weekly run output",
"Debian Package Updates",
"Debian package update",
"daily mail stats",
"Anacron job",
"nagios",
"changes report",
"run output",
"[Systraq]",
"Undelivered mail",
"Postfix SMTP server: errors from",
"backupninja",
"DenyHosts report",
"Debian security status",
"apt-listchanges"
],
header :contains "Auto-Submitted" "auto-generated",
envelope :contains "from" ["nagios@",
"logcheck@",
"root@"])
fileinto "rapports";
Automated script
There is a procmail2sieve.pl script floating around, and
mentioned in the dovecot documentation. It didn't work very well
for me: I could use it for small things, but I mostly wrote the sieve
file from scratch.
Progressive migration
Enrico Zini has progressively migrated his procmail setup to Sieve
using a clever way: he hooked procmail inside sieve so that he could
deliver to the Dovecot LDA and progressively migrate rules one by
one, without having a "flag day".
See this explanatory blog post for the details, which also shows
how to configure Dovecot as an LMTP server with Postfix.
Other examples
The Dovecot sieve examples are numerous and also quite useful. At
the time of writing, they include virus scanning and spam filtering,
vacation auto-replies, includes, archival, and flags.
Harmful considered harmful
I am aware that the "considered harmful" title has a long and
controversial history, being considered harmful in itself (by some
people who are obviously not afraid of contradictions).
I have nevertheless deliberately chosen that title, partly to make
sure this article gets maximum visibility, but more specifically
because I do not have doubts at this moment that procmail is, clearly,
a bad idea at this moment in history.
Developing story
I must also add that, incredibly, this story has changed while writing
it. This article is derived from this bug I filed in Debian to,
quite frankly, kick procmail out of Debian. But filing the bug had the
interesting effect of pushing the upstream into action: as mentioned
above, they have apparently made a new release and merged a bunch of
patches in a new git repository.
This doesn't change much of the above, at this moment. If anything
significant comes out of this effort, I will try to update this
article to reflect the situation. I am actually happy to retract the
claims in this article if it turns out that procmail is a stellar
example of defensive programming and survives fuzzing attacks. But at
this moment, I'm pretty confident that will not happen, at least not
in scope of the next Debian release cycle.
user+foo@example.com to the folder
foo. You might write something like this in procmail:
MAILDIR=$HOME/Maildir/
DEFAULT=$MAILDIR
LOGFILE=$HOME/.procmail.log
VERBOSE=off
EXTENSION=$1 # Need to rename it - ?? does not like $1 nor 1
:0
* EXTENSION ?? [a-zA-Z0-9]+
.$EXTENSION/
require ["variables", "envelope", "fileinto", "subaddress"];
########################################################################
# wildcard +extension
# https://doc.dovecot.org/configuration_manual/sieve/examples/#plus-addressed-mail-filtering
if envelope :matches :detail "to" "*"
# Save name in $ name in all lowercase
set :lower "name" "$ 1 ";
fileinto "$ name ";
stop;
Subject into folder
This would file all mails with a Subject: line having FreshPorts
in it into the freshports folder, and mails from alternc.org
mailing lists into the alternc folder:
:0
## mailing list freshports
* ^Subject.*FreshPorts.*
.freshports/
:0
## mailing list alternc
* ^List-Post.*mailto:.*@alternc.org.*
.alternc/
Equivalent Sieve:
if header :contains "subject" "FreshPorts"
fileinto "freshports";
elsif header :contains "List-Id" "alternc.org"
fileinto "alternc";
Mail sent to root to a reports folder
This double rule:
:0
* ^Subject: Cron
* ^From: .*root@
.rapports/
Would look something like this in Sieve:
if header :comparator "i;octet" :contains "Subject" "Cron"
if header :regex :comparator "i;octet" "From" ".*root@"
fileinto "rapports";
Note that this is what the automated converted does (below). It's not
very readable, but it works.
Bulk email
I didn't have an equivalent of this in procmail, but that's something
I did in Sieve:
if header :contains "Precedence" "bulk"
fileinto "bulk";
Any mailing list
This is another rule I didn't have in procmail but I found handy and
easy to do in Sieve:
if exists "List-Id"
fileinto "lists";
This or that
I wouldn't remember how to do this in procmail either, but that's an
easy one in Sieve:
if anyof (header :contains "from" "example.com",
header :contains ["to", "cc"] "anarcat@example.com")
fileinto "example";
You can even pile up a bunch of options together to have one big rule
with multiple patterns:
if anyof (exists "X-Cron-Env",
header :contains ["subject"] ["security run output",
"monthly run output",
"daily run output",
"weekly run output",
"Debian Package Updates",
"Debian package update",
"daily mail stats",
"Anacron job",
"nagios",
"changes report",
"run output",
"[Systraq]",
"Undelivered mail",
"Postfix SMTP server: errors from",
"backupninja",
"DenyHosts report",
"Debian security status",
"apt-listchanges"
],
header :contains "Auto-Submitted" "auto-generated",
envelope :contains "from" ["nagios@",
"logcheck@",
"root@"])
fileinto "rapports";
Automated script
There is a procmail2sieve.pl script floating around, and
mentioned in the dovecot documentation. It didn't work very well
for me: I could use it for small things, but I mostly wrote the sieve
file from scratch.
Progressive migration
Enrico Zini has progressively migrated his procmail setup to Sieve
using a clever way: he hooked procmail inside sieve so that he could
deliver to the Dovecot LDA and progressively migrate rules one by
one, without having a "flag day".
See this explanatory blog post for the details, which also shows
how to configure Dovecot as an LMTP server with Postfix.
Other examples
The Dovecot sieve examples are numerous and also quite useful. At
the time of writing, they include virus scanning and spam filtering,
vacation auto-replies, includes, archival, and flags.
Harmful considered harmful
I am aware that the "considered harmful" title has a long and
controversial history, being considered harmful in itself (by some
people who are obviously not afraid of contradictions).
I have nevertheless deliberately chosen that title, partly to make
sure this article gets maximum visibility, but more specifically
because I do not have doubts at this moment that procmail is, clearly,
a bad idea at this moment in history.
Developing story
I must also add that, incredibly, this story has changed while writing
it. This article is derived from this bug I filed in Debian to,
quite frankly, kick procmail out of Debian. But filing the bug had the
interesting effect of pushing the upstream into action: as mentioned
above, they have apparently made a new release and merged a bunch of
patches in a new git repository.
This doesn't change much of the above, at this moment. If anything
significant comes out of this effort, I will try to update this
article to reflect the situation. I am actually happy to retract the
claims in this article if it turns out that procmail is a stellar
example of defensive programming and survives fuzzing attacks. But at
this moment, I'm pretty confident that will not happen, at least not
in scope of the next Debian release cycle.
:0
## mailing list freshports
* ^Subject.*FreshPorts.*
.freshports/
:0
## mailing list alternc
* ^List-Post.*mailto:.*@alternc.org.*
.alternc/
if header :contains "subject" "FreshPorts"
fileinto "freshports";
elsif header :contains "List-Id" "alternc.org"
fileinto "alternc";
:0
* ^Subject: Cron
* ^From: .*root@
.rapports/
if header :comparator "i;octet" :contains "Subject" "Cron"
if header :regex :comparator "i;octet" "From" ".*root@"
fileinto "rapports";
Bulk email
I didn't have an equivalent of this in procmail, but that's something
I did in Sieve:
if header :contains "Precedence" "bulk"
fileinto "bulk";
Any mailing list
This is another rule I didn't have in procmail but I found handy and
easy to do in Sieve:
if exists "List-Id"
fileinto "lists";
This or that
I wouldn't remember how to do this in procmail either, but that's an
easy one in Sieve:
if anyof (header :contains "from" "example.com",
header :contains ["to", "cc"] "anarcat@example.com")
fileinto "example";
You can even pile up a bunch of options together to have one big rule
with multiple patterns:
if anyof (exists "X-Cron-Env",
header :contains ["subject"] ["security run output",
"monthly run output",
"daily run output",
"weekly run output",
"Debian Package Updates",
"Debian package update",
"daily mail stats",
"Anacron job",
"nagios",
"changes report",
"run output",
"[Systraq]",
"Undelivered mail",
"Postfix SMTP server: errors from",
"backupninja",
"DenyHosts report",
"Debian security status",
"apt-listchanges"
],
header :contains "Auto-Submitted" "auto-generated",
envelope :contains "from" ["nagios@",
"logcheck@",
"root@"])
fileinto "rapports";
Automated script
There is a procmail2sieve.pl script floating around, and
mentioned in the dovecot documentation. It didn't work very well
for me: I could use it for small things, but I mostly wrote the sieve
file from scratch.
Progressive migration
Enrico Zini has progressively migrated his procmail setup to Sieve
using a clever way: he hooked procmail inside sieve so that he could
deliver to the Dovecot LDA and progressively migrate rules one by
one, without having a "flag day".
See this explanatory blog post for the details, which also shows
how to configure Dovecot as an LMTP server with Postfix.
Other examples
The Dovecot sieve examples are numerous and also quite useful. At
the time of writing, they include virus scanning and spam filtering,
vacation auto-replies, includes, archival, and flags.
Harmful considered harmful
I am aware that the "considered harmful" title has a long and
controversial history, being considered harmful in itself (by some
people who are obviously not afraid of contradictions).
I have nevertheless deliberately chosen that title, partly to make
sure this article gets maximum visibility, but more specifically
because I do not have doubts at this moment that procmail is, clearly,
a bad idea at this moment in history.
Developing story
I must also add that, incredibly, this story has changed while writing
it. This article is derived from this bug I filed in Debian to,
quite frankly, kick procmail out of Debian. But filing the bug had the
interesting effect of pushing the upstream into action: as mentioned
above, they have apparently made a new release and merged a bunch of
patches in a new git repository.
This doesn't change much of the above, at this moment. If anything
significant comes out of this effort, I will try to update this
article to reflect the situation. I am actually happy to retract the
claims in this article if it turns out that procmail is a stellar
example of defensive programming and survives fuzzing attacks. But at
this moment, I'm pretty confident that will not happen, at least not
in scope of the next Debian release cycle.
if header :contains "Precedence" "bulk"
fileinto "bulk";
if exists "List-Id"
fileinto "lists";
This or that
I wouldn't remember how to do this in procmail either, but that's an
easy one in Sieve:
if anyof (header :contains "from" "example.com",
header :contains ["to", "cc"] "anarcat@example.com")
fileinto "example";
You can even pile up a bunch of options together to have one big rule
with multiple patterns:
if anyof (exists "X-Cron-Env",
header :contains ["subject"] ["security run output",
"monthly run output",
"daily run output",
"weekly run output",
"Debian Package Updates",
"Debian package update",
"daily mail stats",
"Anacron job",
"nagios",
"changes report",
"run output",
"[Systraq]",
"Undelivered mail",
"Postfix SMTP server: errors from",
"backupninja",
"DenyHosts report",
"Debian security status",
"apt-listchanges"
],
header :contains "Auto-Submitted" "auto-generated",
envelope :contains "from" ["nagios@",
"logcheck@",
"root@"])
fileinto "rapports";
Automated script
There is a procmail2sieve.pl script floating around, and
mentioned in the dovecot documentation. It didn't work very well
for me: I could use it for small things, but I mostly wrote the sieve
file from scratch.
Progressive migration
Enrico Zini has progressively migrated his procmail setup to Sieve
using a clever way: he hooked procmail inside sieve so that he could
deliver to the Dovecot LDA and progressively migrate rules one by
one, without having a "flag day".
See this explanatory blog post for the details, which also shows
how to configure Dovecot as an LMTP server with Postfix.
Other examples
The Dovecot sieve examples are numerous and also quite useful. At
the time of writing, they include virus scanning and spam filtering,
vacation auto-replies, includes, archival, and flags.
Harmful considered harmful
I am aware that the "considered harmful" title has a long and
controversial history, being considered harmful in itself (by some
people who are obviously not afraid of contradictions).
I have nevertheless deliberately chosen that title, partly to make
sure this article gets maximum visibility, but more specifically
because I do not have doubts at this moment that procmail is, clearly,
a bad idea at this moment in history.
Developing story
I must also add that, incredibly, this story has changed while writing
it. This article is derived from this bug I filed in Debian to,
quite frankly, kick procmail out of Debian. But filing the bug had the
interesting effect of pushing the upstream into action: as mentioned
above, they have apparently made a new release and merged a bunch of
patches in a new git repository.
This doesn't change much of the above, at this moment. If anything
significant comes out of this effort, I will try to update this
article to reflect the situation. I am actually happy to retract the
claims in this article if it turns out that procmail is a stellar
example of defensive programming and survives fuzzing attacks. But at
this moment, I'm pretty confident that will not happen, at least not
in scope of the next Debian release cycle.
if anyof (header :contains "from" "example.com",
header :contains ["to", "cc"] "anarcat@example.com")
fileinto "example";
if anyof (exists "X-Cron-Env",
header :contains ["subject"] ["security run output",
"monthly run output",
"daily run output",
"weekly run output",
"Debian Package Updates",
"Debian package update",
"daily mail stats",
"Anacron job",
"nagios",
"changes report",
"run output",
"[Systraq]",
"Undelivered mail",
"Postfix SMTP server: errors from",
"backupninja",
"DenyHosts report",
"Debian security status",
"apt-listchanges"
],
header :contains "Auto-Submitted" "auto-generated",
envelope :contains "from" ["nagios@",
"logcheck@",
"root@"])
fileinto "rapports";
OpenStack Newton is released, and uploaded to Sid
OpenStack Newton was released on the Thursday 6th of October. I was able to upload nearly all of it before the week-end, though there was a bit of hick-ups still, as I forgot to upload python-fixtures 3.0.0 to unstable, and only realized it thanks to some bug reports. As this is a build time dependency, it didn t disrupt Sid users too much, but 38 packages wouldn t build without it. Thanks to Santiago Vila for pointing at the issue here.
As of writing, a lot of the Newton packages didn t migrate to Testing yet. It s been migrating in a very messy way. I d love to improve this process, but I m not sure how, if not filling RC bugs against 250 packages (which would be painful to do), so they would migrate at once. Suggestions welcome.
Bye bye Jenkins
For a few years, I was using Jenkins, together with a post-receive hook to build Debian Stable backports of OpenStack packages. Though nearly a year and a half ago, we had that project to build the packages within the OpenStack infrastructure, and use the CI/CD like OpenStack upstream was doing. This is done, and Jenkins is gone, as of OpenStack Newton.
Current status
As of August, almost all of the packages Git repositories were uploaded to OpenStack Gerrit, and the build now happens in OpenStack infrastructure. We ve been able to build all packages a release OpenStack Newton Debian packages using this system. This non-official jessie backports repository has also been validated using Tempest.
Goodies from Gerrit and upstream CI/CD
It is very nice to have it built this way, so we will be able to maintain a full CI/CD in upstream infrastructure using Newton for the life of Stretch, which means we will have the tools to test security patches virtually forever. Another thing is that now, anyone can propose packaging patches without the need for an Alioth account, by sending a patch for review through Gerrit. It is our hope that this will increase the likeliness of external contribution, for example from 3rd party plugins vendors (ie: networking driver vendors, for example), or upstream contributors themselves. They are already used to Gerrit, and they all expected the packaging to work this way. They are all very much welcome.
The upstream infra: nodepool, zuul and friends
To be clear: for Stretch we recommend that reproducible builds are done in the same build path as the "original" build.
Finally, and just for our future references, when we enabled build path variation on Saturday, August 20th 2016, the numbers for unstable were:
But then we also like to confirm the bugs are really gone, which (atm) requires manual
checking, and to look for the words "reproducible" and "deterministic"
(and spelling variations) in debian/changelogs of all uploads, to spot reproducible work
not tracked via the BTS.
And we still need to catch up on the backlog of weekly reports.
Bugs submitted with reproducible usertags
It seems 
What happened in the 