The first alpha of
syslog-ng 3.4 was released more than a
year ago, and the first stable release
earlier this year, development on
syslog-ng 3.5 has been going on steadlily since then, with a lot of interesting features and changes pouring in. While the 3.5 release will not bring as many overwhelming changes as the previous two releases did, the sheer number of improvements is still substantial. Our focus was on smaller developments all over the place this time, instead of big sweeping changes. In this post, we will go over most of these features, with use-cases, examples and a few words about why they are so useful and awesome.
Multi-line support,
new destinations,
new template functions,
type hinting,
unit suffixes, and
other things. Read on, and be amazed!These are only bigger features, there has been a lot of small tweaking going on, and of course, all of the fixes that went into prior versions will also be in 3.5 too, and there's still a chance that something else may find its way in before the feature freeze.
Multi-line supportOne of the major new features in the 3.5 release will be support for multi-line messages, a feature that has been available in
syslog-ng PE for a good while, and which has been ported to and improved upon to the open source edition. Two variants of multi-line are supported, which will be detailed below. Both of them are available for the
file() and
pipe() sources only.Indented multi-lineThe easiest variant is indented multi-line, where each line can be followed by others, indented by whitespace, and the message continues until the first non-indented line. This is the format used by the Linux kernel too, from version 3.5, for
/dev/log. This type of multi-line can be used as follows:
source s_multiline file("/path/to/file" multi-line-mode(indented)); ;Consider that
/path/to/file has the following content:
First line
Continuation 1;
Continuation 2;
Second line
With the
indented multi-line-mode() setting, this would turn into two log messages:
First line\n Continuation 1;\n Continuation 2;
Second line
Regexp-based multilineIf multi-line input is not based on indentation, one can use the
regexp multi-line-mode() instead, which makes two new settings available:
multi-line-prefix() and
multi-line-garbage(). These can be used to define the start and the end of a log message: any string between a the beginning matching prefix and a matching garbage will be considered a single message. That is, the prefix will be included, the garbage will not be: it will be discarded.To illustrate:
source s_multiline
file("/path/to/file" multi-line-mode(regexp)
multi-line-prefix("^prefix")
multi-line-garbage(" garbage$"));
;If the source contains these lines:
prefix message
continuing garbage
This will turn into the following log message:
prefix message\ncontinuing
New destinationsIt would be pretty hard to do a syslog-ng release without new destinations, and with 3.5, we will have three of them!STOMPWhile we had AMQP before, we now have a
STOMP destination too, so we can stream logs to any STOMP server, with a few simple lines:
stomp(
host("localhost")
port(61613)
destination("/topic/syslog")
routing_key("")
persistent(yes)
ack(no)
username("someone")
password("something")
body("This is the body! YAY! Here's a message: $MESSAGE!\n")
value-pairs(scope(nv-pairs, syslog-proto, selected-macros))
)Of course, none of these settings need to be set, you can just use the defaults, and it will just work!RiemannDid you know that syslog-ng is far more than a log collector and processor? No? You do now. When you have access to a lot of logs, and a tremendous amount of power in parsing them, you can use these tools for monitoring easily! And when we're talking monitoring,
riemann is a great asset in our toolbox, and with the new destination, we can easily forward metrics to it:
riemann(
server("localhost")
port(5555)
host("$HOST")
service("$PROGRAM")
description("$PROGRAM pids")
metric(int("$PID"))
ttl(300)
);Of course, the above is a very silly example, it would make much more sense to extract data from a log message instead, but this will serve as an example.
New template functions
$(upper-case TEXT...), $(lower-case TEXT...)-
To turn some text into *upper-* or *lower-case*, these two template
functions come in handy. They simply turn all their arguments into
the respecitve case.
$(delimit DELIMITERS NEW-DELIMITER TEXT)-
Sometimes one has a delimited string, where one wishes to replace
the delimiters. This function does just that: give it a list of
delimiters (you may need to quote it, if it contains whitespace), a
replacement, and a text to replace delimiters in, and it does the
rest.
For example, to replace tabs and spaces with a vertical bar, one
could write a template like this:
template("$(delimit \"\t \" \" \" $MESSAGE)\n")
$(env VARIABLE...)-
Have you ever felt the need to check an environment variable from
within syslog-ng? I did, and now I can.
Type hintingWhile syslog-ng supported sending log messages into various data stores and message queues for a while (SQL at first, MongoDB, JSON and AMQP later), even when those supported different types of data than strings, we could not do anything else. Until now. It is now possible - at places where it makes sense - to annotate templates with type hints, which the destination driver can optionally use. Type hinting is implemented for the
mongodb() destination and the
$(format-json) template function for now. When no type hint is specified, syslog-ng defaults to
string.To add type hints, simply wrap the respective template with the hinted type, like this:
mongodb(
value-pairs(pair("date", datetime("$UNIXDATE"))
pair("pid", int64("$PID")))
);Currently the following type hints exist:
boolean (anything that begins with a
t or
1 is true, anything that begins with
f or
0 is false, everything else is an error),
string,
literal (same as string, but not quoted if it would be quoted otherwise),
int32 (
int is an alias for this),
int64, and
datetime. Only UNIX timestamps can be type-hinted to
datetime, anything else will likely result in a casting error.It is also possible to control what happens when type casting fails: syslog-ng can drop the whole message, drop the property, or fall back to string. It can also do all of these silently:
options
typecast(on-error(silently-drop-property));
;
Using this feature with
$(format-json) is very similary too:
$(format-json date=datetime("$UNIXDATE") pid=int64("$PID"))With this feature in place, you can now store your non-string values with their proper types!
Unit suffixesUnit suffixes make it considerably easier to set limits and describe numbers within the syslog-ng configuration. We no longer need to spell out sizes to the byte precious, it is now enough to write:
log-fifo-size(200MiB). Now, syslog-ng will understand suffixes for kilo-, mega-, and giga-bytes, (
K,
M,
G, respectively) either in base-10 or base-2 (with an extra
i after the suffix). One can also omit the trailing
b from the end.So, to set the
log-fifo-size() to
2097152 bytes, one can simply use
2MiB. Or, to set it to
2000000,
2Mb. That's a whole lot easier, isn't it? No more counting zeros, no more silly typos in a ten-digit number, no more pain, but easily readable units!
Miscellaneous featuresApart from the features above, there have been a lot of other changes and improvements in the code base:
-
We now have a (mostly) non-recursive, quiet build system, which is
not only much faster than what we had before, but more reliable too,
and easier to glance through its logs, too.
-
A few old settings that were never used were removed (the
username() and password() settings of the
mongodb() driver), some were renamed and deprecated:
the replace() key transformation function of
value-pairs() was renamed to
replace-prefix(), as that makes the intent clearer. In
this latter case, the old name is still valid, but obsoleted.
-
Our integration with systemd is
much tighter now: syslog-ng can notify systemd when it is ready.
This also means that when systemd support is enabled, certain
systemd libraries will have to be installed: syslog-ng no longer
carries a convenience copy.
-
A new filter was implemented:
in-list, with which one
can implement efficient white- or blacklists.
To use it, you will need a file with one value a line, and do
something along these lines:
filter f_whitelist
in-list("/path/to/file.list", value("PROGRAM"));
;
To do a blacklist, just negate the filter:
filter f_blacklist
not in-list("/path/to/file.list", value("HOST"));
;
-
syslog-ng now supports Linux 3.5+-style
/dev/kmsg, and
will use that instead of /proc/kmsg when a sufficiently
recent kernel is detected (assuming one is using the
system() source).
The new kernel log format supports structured messages, and
syslog-ng is smart enough to parse them, and make them accessible
like all other message properties (with a .linux. prefix).
Seven hours ago I was handing out how to vote cards for the Greens at the 2013 Australian Federal election. I was hoping that either we would have a Labor/Greens coalition or an outright majority for Labor. Unfortunately we got a Liberal majority in the lower house and it looks like some extreme right wing groups may get into the senate (replacements for Family First the anti-Gay party).
For some reason the polling station where I was working only had volunteers from the three major parties (Greens, Labor, and Liberal) while other polling stations in the same electorate had volunteers from smaller parties such as the Sex Party and the Socialist Alliance.
The volunteers from the Liberal party ate McDonalds outside the polling station and afterwards McDonalds rubbish was left on the ground, the above picture isn t particularly clear because I took it after 6PM when the polls closed. The Liberals didn t care enough to put their rubbish in a bin, it s an externality for them, if they get enough seats in the senate they will surely take the same approach to governing Australia. The Labor people didn t take the effort to clean up the Liberal mess even though it wasn t particularly difficult to do so, I think that s the type of attitude that led to this election defeat. In the case of the McDonalds rubbish in question I put it in the bin so that when the primary school kids return on Monday their school won t be too messy after the election. But in the case of the mess that is being made in Australian politics it will take many more Greens votes to allow us to clean it up.
When it comes to provide redundant services, several options are available:
Let s explain it step by step:
Pretend I run a cluster, for hosting a site. Pretend that I have three-six web-nodes, and each one needs to know which database host to contact.
How do I control that?
Right now I have a /etc/settings.conf file, more or less, deployed by 
The blog engine 
Note: This post will probably not be on the liking of those who think free software is always preferable to closed source software, so if you are such a person, please take this article as an invitation to implement better open source alternatives that can realistically compete with the closed source applications I am mentioning here. I am not going to mention here where the open source alternatives are not up to the same level as the commercial tools, I'll leave that for the readers or for another article.
This is my monthly summary of my free software related activities. If you re among the people who 
On the positive side, Marko has gotten some feedback from people who like the new PTS and are using it daily already. And several persons have expressed their interest to work on the new codebase already.
On my side, I created a package so that it s easy to deploy for derivatives. In this process, I revamped the way we manage the Django settings (for development and for production). The package is not finished yet, but it s mostly usable already. But I still want to do some cleanup/refactoring in the models before others start deploying it. We must also enable South to make it possible to upgrade easily afterwards.
DebConf 13 in Vaumarcus
From August 10th to 17th, I was attending DebConf 13. It matched the only week of vacation that my wife had this summer so we went there with the whole family (that is with a 3 years old son, and 6 months old one). Thus I could not immerse myself in Debconf and missed all the nice things that happen outside of the talk rooms. I picked 3-4 interesting talks per day and I spent the rest with my family.
On the positive side, I was pleased that my wife could meet (or at least see) some other Debian people. She knows quite a few (of you) by name because I have been telling her Debian stories for years now
Debian France
Debian France sold quite some merchandise during Debconf but I didn t take care of that. It was supervised by Sylvestre Ledru but fortunately he got the help of multiple persons, both to bring everything there, to sell it, and to bring back the rest.
The good news of the month is that the 
This DebConf has been the first one where I ran every day. That proved
to be kinda hard on some mornings, but I found a way to circumvent
that by proposing group runs at the end of the relevant days.
In total, I managed to run 12 days in a row, for a grand total of
186.88km, in 21h22', for a total height difference of 6972m. In short,
I nearly climbed Mount Aconcagua, Americas highest peak..:-)
Most of these runs were quite slow ones and rather "short" ones
(around 15km), because of the nature of the paths around (nearly
impossible to find something flat except by running around the
football field).
The longest run was for the DayTrip, with about 25km in about 4 hours
(including some pauses) and 1600m height difference. I'm definitely
working on my moutain running skills more than marathon and other kind
of "speed" races.
That followed a great results for my last race, the "EDF Cenis Tour"
50km trail race in Lanslebourg, French Alps, where I completed the
race in 6h38, nearly the sametime than Mont-Blanc marathon, 1 month
earlier....which was 8 kilometers shorter.
I'm apparently ready for very long distance mountain races, now, as it
seems. Thanks you DebConf 13 organizers for choosing such a great
locations for us, crazy runners.
And, despite that, my next running goal is a "classical" marathon, the
Toulouse marathon in late October, where I expect breaking the 3h30
barrier, that would indeed be a qualifier for the Boston Marathon, one
of my dreams for the future (and, yes, it is still a dream, despite
this year's events).
Animal instincts
With so many candidates to choose from, even our kitty may have found an opportunity in the 
The Senate, a place for convicted hackers?
The next party in the picture is 
For me, DebConf13 in Switzerland was a perfect DebConf, and generally an
great developer meeting and conference. I've attended several past DebConfs
in Canada, Norway, Brazil, Finland, Mexico, Scotland, Spain, NYC, Bosnia,
and Nicaragua. I have special memories of each. DebConf13 was ideal in every
way.
This is part 5 in 