When
syslog-ng 3.5.1 was released, the existence of the
syslog-ng Incubator project was already announced, but I did not go into detail as to why it exists, what is in there, how to build it, or how to use it. The documentation that exists on it is almost non-existent, and what does exist, is usually in the form of a commit message and some example files. But there's nothing on how the things in the Incubator can be used, and what problems they solve.With this post, I will try to ease the situation a little, and provide some insight on how I use some of the things in there, like the
Riemann destination.About the IncubatorThe idea for the
Incubator existed for a long time, the first incarnation of it was called
syslog-ng module collection, and had a different scope: it was a playing field not only for new modules, but updates to existing ones too. The idea was to try out new, possibly experimental features separately from syslog-ng. That didn't work well: it was a nightmare to maintain, and even harder to use. So it never got anywhere, as the developments made there were quickly merged into syslog-ng itself anyway.Then, two years later, I had a need for a new destination, but there was no open development branch to base it against, and forking and maintaining a whole syslog-ng branch for the purpose of a single module sounded like an overkill. So the Incubator was born, from the ashes of the failed module collection.The purpose of the Incubator is to be a place of experimental modules, and an easier way to enter the world syslog-ng development. It's something with far less requirements than syslog-ng (I do not care all that much about portability when it comes to the Incubator, especially not to horrible abominations like HP-UX or AIX), less rules, and more freedom. It's meant to be more developer friendly than syslog-ng proper.It also serves as an example of developing modules completely external to syslog-ng, and parts of it will be used in future posts of mine, to demonstrate the development of new modules.Compiling & InstallationThe first thing we need to do, is to compile whatever we need from the
Incubator, unless we happen to have pre-built binaries available (which are going to appear in your usual third-party syslog-ng repositories over the next few weeks). For this, at a minimum, one will need
syslog-ng 3.5+ and
ivykis installed, along with autotools (autoconf, automake and libtool), pkg-config, bison, and depending on which modules one needs,
riemann-c-client and
libmongo-client too. Most of these are available packaged in better distributions, but only in recent versions, so one may need to compile those too. They follow a very similar scheme, though.If everything is installed at the standard locations, getting the Incubator to compile is as simple as this:
$ git clone https://github.com/balabit/syslog-ng-incubator.git
$ cd syslog-ng-incubator
$ autoreconf -i
$ ./configure && make && sudo make install
If anything happens to be installed in a non-standard location, one will need to adjust
PKG_CONFIG_PATH to help the configure script locate the needed libraries.Once installed (the configure script will figure out where syslog-ng modules are, and modules will be put there), syslog-ng will automatically recognise the new modules. One can make sure that this is the case by running the following command after installation:
$ syslog-ng --module-registry
The PluginsNow that we're over the hard part of compiling and installing the Incubator, lets see what is inside! I will start with the easier things, and move on to the more complicated features as we progress. That is, we'll start with some
template functions, have a glance at the
trigger source, then explore the
rss destination, and finish off with the
riemann destination.
Template functionsThe Incubator gives us three new template functions, some less useful than the others, and one that's a huge, ugly hack for a problem that I ended up solving in a very different way - without the hack.These functions are the
$(or) template function, which takes any number of arguments, and returns the first one that is not empty. The main use case here is that if you have, say, similarly named fields, but some messages have one, the others another, and you want to normalize it,
$(or) is one way to do it:
$(or $ HOST $ HOSTNAME $ HOST_NAME "<unknown>")
Another function is
$(//), which does the same thing as the built-in
$(/) template function: divide its arguments. Except this one works with floating-point numbers only, while the built-in one is for integers exclusively. Using it is simple, too:
$(// $ some_number 3.4)
The last template function provided by the Incubator is
$(state), which can be used to maintain global state, that does not depend on log messages. You can set values in here, like counters, from within a template function. It is possible to count the total amount downloaded data when processing a HTTP server log, for example. But it's slow, and there are better ways to do the same thing, syslog-ng really isn't the best tool for this kind of job. If anyone happens to find a use-case for it, please let me know. As for using it, it has two modes: set (with two arguments) and get (with one):
$(state some-variable $ VALUE )
$(state some-variable)
Trigger sourceThe trigger source has many in common with the built-in mark feature: at given intervals, it sends a message. This is mostly a debugging aid, when you want to generate messages without an external tool. It only has two options:
trigger-freq() and
trigger-message(), which default to
10 and
"Trigger source is trigger happy.", respectively. It also accepts a number of common source options such as
program-override(),
host-override() and
tags().To use it, one just needs to set it up like any other source, and bind it to a destination with a log statement:
source s_trigger
trigger(
program-override("trigger")
tags("trigger-happy")
trigger-freq(5)
trigger-message("Beep.")
);
;
Without a
program-override() option, messages will be attributed to syslog-ng, which is likely not what you want, even while debugging. Internal messages are usually routed somewhere else.
RSS destinationThe RSS destination is an interesting beast. It offers an Atom feed of the last hundred messages routed to the destination. I could very well imagine this being useful in a situation where one already has monitoring set up to listen on various RSS sources - this would be just another. It also works well with most RSS feed readers. The length of the feed is not configurable at this time, and the number of options is limited to
port(),
title(),
entry-title() and
entry-description().The first one specifies which port the destination should listen on (it serves one client at a time!);
title() can be used to set the title of the feed itself, while
entry-title() and
entry-description() can be used with templates to fill in the per-message Atom entries.Once we have a suitable path we want to route to the RSS destination (such as critical error messages only), we can set it up like this:
destination d_rss
rss(
port(8192)
feed-title("Critical errors in the system")
entry-title("Error from $ PROGRAM @ $ HOST_FROM at $ ISODATE ")
entry-description("$ MESSAGE ")
);
;
Riemann destinationBeing the original motivator for the Incubator, I left this last. This module is the interface between your logs and the
Riemann monitoring system. With this, you can take all the legacy applications that are hard to monitor, but provide log files, use syslog-ng's extraordinary log processing power, and send clear and concise events over to Riemann.One can use it to monitor logins, downloads, uploads, exceptions - pretty much anything. Just extract some metric, or state, send it over to Riemann, and that will do the heavy lifting. What exactly can be done, will be worth a separate blog post, so for now, I will give just a very tiny example:
destination d_riemann
riemann(
ttl("120")
description("syslog-ng internal errors")
metric(int("$ SEQNUM "))
);
;
Hook it up to a path that collects syslog-ng internal messages, keeps only error messages, and routes it toward this destination:
log
source internal(); ;
filter level(err..emerg); ;
destination(d_riemann);
;
The destination itself has the following options:
server(): The server to connect to, defaults to localhost.port(): The port the Riemann server is listening on, defaults to 5555.type(): The type of connection: UDP or TCP. Defaults to TCP.host(): The host field of the Riemann event, defaults to $ HOST .service(): The service field of the Riemann event, defaults to $ PROGRAM state(): The state field of the Riemann event, without a default.description(): The description of the event, with no default.ttl(): The time-to-live of the event, no default.metric(): The metric to send to Riemann. This needs to be either an integer or a floating point number. Using type-hinting is advised here. Without one, the destination will try to parse the value of this option as a float.tags(): As the name implies, this adds tags to the Riemann event. By default, all the tags that are set on the message, will be forwarded.attributes(): With this option, one can set custom attributes on the Riemann event. The syntax is the same as for value-pairs(), with a few enhancements. What the difference is, is left as an exercise to the reader: the example config that comes with the Incubator has a hint.
Closing thoughtsThe Incubator contains one more tool, one which can be used to visualise logs in strange ways. But that's not strictly related to syslog-ng, isn't a module, either, so I will not describe it here right now. The above was quite a lot already, I believe.As a closing thought, I would just like to say that while the Incubator is home for experimental modules, some of them are used in production. Don't be afraid to use them, especially when packages start to arrive to your favourite GNU/Linux distribution!
I have written a number of scripts to create Debian foreign architecture
(mostly armel and armhf) rootfs images for SD cards or NAND flashing. I started
with putting Debian on my Openmoko gta01 and gta02 and continued with
devices like the qi nanonote, a marvel kirkwood based device, the Always
Innovating Touchbook (close to the Beagleboard), the Notion Ink Adam and
most recently the Golden Delicious gta04. Once it has been manufactured, I
will surely also get my hands dirty with the Neo900 whose creators are
currently looking for potential donors/customers to increase the size of
the first batch and get the price per unit further down.
Creating a Debian rootfs disk image for all these devices basically follows the
same steps:
Packt Publishing sent me 
The 
I read a lot this year - I'll write more about that and reflections on
Implementing software to replace legacy systems is always a challenge, especially when you're dealing with a system with as much legacy as sendmail, which was first introduced as delivermail in 1979.
Checking if your servers are configured correctly can be done with 
Clicking on any test will display the necessary information to
troubleshoot errors, including the test short description, the
complete test code, the expectation message and the backtrace:
I hope this additional layer will help making serverspec another
feather in the 
Well, it's 2014. The annus horribilus of 2013 is behind me. I'm
entering 2014 in a much better place (on multiple levels) than I did for 2013.
I'm positively excited to see what 2014 brings me.
I was very fortunate to have someone I looked up to reach out to help me when
my marriage fell apart. One of the things he said to me was "A year from now,
you won't recognise yourself". He was right. I am such a different person today
than I was a year ago. I'm happier, fitter, more self-confident. I'm focusing
on different things in my life.
One thing that has been a casualty, has been my geeky pursuits. There were a
few times last year where I questioned if I was still a geek. I am. I've just
deprioritised it. Instead, I've been putting my own self-care and my daughter
above all else, meaning that my running has probably been the one thing I'd
call a "hobby" at the moment.
I've got some stuff in the pipeline that I'm not quite ready to write about
yet, but I'm expecting a rearrangement of my time that will hopefully allow for
some more geeky endeavours in 2014. I'm also hoping I can regain some momentum
on blogging.
Here's what I'd say my goals are, going into 2014:
Good morning all my hungover friends.
New Hy release - sounds like the perfect thing to do while you re waiting for your headaches to go away.
Here s a short-list of the changes (from NEWS) - enjoy!
This week my 
Object Lifetimes under Present Redirection
Present extension redirection passes responsibility for showing
application contents from the X server to the compositing
manager. This eliminates an extra copy to the composite extension
window buffer and also allows the application contents to be shown
in the right frame.
(Currently, the copy from application buffer to window buffer is
synchronized to the application-specified frame, and then a Damage
event is delivered to the compositing manager which constructs the
screen image using the window buffer and presents that to the screen
at least one frame later, which generally adds another frame of delay
for the application.)
The redirection operation itself is simple just wrap the
PresentPixmap request up into an event and send it to the compositing
manager. However, the pixmap to be presented is allocated by the
application, and hence may disappear at any time. We need to carefully
control the lifetime of the pixmap ID and the specific frame contents
of the pixmap so that the compositing manager can reliably construct
complete frames.
We ll separately discuss the lifetime of the specific frame contents
from that of the pixmap itself. By the frame contents , I mean the
image, as a set of pixel values in the pixmap, to be presented for a
specific frame.
Present Pixmap contents lifetime
After the application is finished constructing a particular frame in a
pixmap, it passes the pixmap to the X server with PresentPixmap. With
non-redirected Present, the X server is responsible for generating a
PresentIdleNotify event once the server is finished using the
contents. There are three different cases that the server handles,
matching the three different PresentCompleteModes:
