Continuing my
HOWTO series, in this article, I will explore how to set up syslog-ng to process JSON in various scenarios. We will touch the subject of interoperability, and have a look at multiple scenarios, in the hope that readers will find something similar to their own needs. There are many different ways in which one can plug JSON into their logging system, and it is very likely that I will not be able to cover it all within this HOWTO. However, I will explain the basics, and how to plug the pieces together, which should at least be a useful foundation to build even more complex systems onto.In essence, we will look at the following scenarios:
raw JSON transport,
JSON over syslog,
JSON and syslog payload in the same stream. The documentation will cover both syslog-ng 3.4 and 3.5, where the 3.5 parts will be clearly marked as such. The 3.3 series do not have a JSON parser, and as such, is insufficient for most needs explored in this guide.In the default setting, the JSON parser will parse the MESSAGE part of an incoming log event, and place every key into a macro of the same name. If the incoming JSON is structured, it the keys will be converted to dot notation format:
"deeper": "level": "very much so"
In this case, one can access the value as
$ deeper.level within a syslog-ng template.The parsed keys can be prefixed using the
prefix() option of the json-parser. One can also change what the parser will receive as input by using the
template() option.See the
documentation for more information on these.
Transporting raw JSONThe simplest scenario is to take a transport like TCP, and simply push raw JSON structures through, one each line, and have syslog-ng parse that to its internal representation. One can do all kinds of things with the data then: transport it further as JSON, store it in files, in a database - the possibilities are endless, but we will explore two scenarios.JSON on TCP, using syslog field namesThe first one is receiving JSON on TCP, using syslog field names, store the parsed stuff in a file (using the conventional format), and in mongodb.
source s_tcp_json tcp(port(10514) flags(no-parse)); ;
parser p_json json-parser(); ;
destination d_file file("/var/log/remote.log"); ;
destination d_mongodb mongodb(collection("remote_log")); ;
log
source(s_tcp_json);
parser(p_json);
destination(d_file);
destination(d_mongodb);
;This needs a bit of an explanation: we use the no-parse flag on the source, because by default, the
tcp() source expects a message that conforms to the old syslog protocol. We have raw JSON, and we need to tell the source that. The other elements should be self explanatory, I believe, with the exception of
json-parser(): we give it no parameters, so it will not modify the keys in the JSON it receives, keeping them intact, as-is.For this reason, incoming messages need to have the same keys as syslog-ng normally would use internally: DATE, HOST, MESSAGE, and so on, like in the following example:
"PROGRAM":"prg00000","PRIORITY":"info","PID":"1234","MESSAGE":"seq: 0000000000, thread: 0000, runid: 1374490607, stamp: 2013-07-22T12:56:47 MESSAGE...","HOST":"localhost","FACILITY":"auth","DATE":"Jul 22 12:56:47"
JSON on TCP, custom field namesA more likely scenario is when we receive JSON, but not in a layout syslog-ng uses internally. In this case, we need to rewrite the message a little, and possibly use templates, depending on what kind of output we wish to use.Lets assume the incoming JSON has these fields:
"app":"prg00000","prio":"info","id":"1234","msg":"seq: 0000000000, thread: 0000, runid: 1374490607, stamp: 2013-07-22T12:56:47 MESSAGE...","source":"localhost","timestamp":"Jul 22 12:56:47"
As you can see, this has similar fields, but not quite, and
FACILITY is missing too.In this case, we'll use a different configuration:
source s_tcp_json tcp(port(10514) flags(no-parse)); ;
parser p_json json-parser(prefix(".json.")); ;
destination d_file file("/var/log/remote.log"
template("$ .json.timestamp $ .json.source $ .json.app [$ .json.id ]: $ .json.msg \n")); ;
destination d_mongodb mongodb(collection("remote_log")
value-pairs(
pair("PROGRAM" "$ .json.app ")
pair("HOST" "$ .json.source ")
pair("PID" "$ .json.id ")
pair("MESSAGE" "$ .json.msg ")
pair("DATE" "$ .json.timestamp ")
pair("PRIORITY" "$ .json.prio ")
pair("FACILITY" "auth")
)); ;
log
source(s_tcp_json);
parser(p_json);
destination(d_file);
destination(d_mongodb);
;
We use the prefix() option of
json-parser(), to avoid collisions with the default syslog-ng namespace. Then, we use a template for the file destination, and
value-pairs() for the MongoDB one. Doing all this, we get similar results as we did in the first scenario.
JSON over syslogAn even likelier scenario is when we have JSON payload in the message part of a normal syslog message. This is useful because the logs themselves remain syslog compatible, and can be transported through devices that know no better.Thankfully, if we do not tell syslog-ng that we do not want any parsing on the input, the json-parser will receive the MESSAGE part of the log, so if we have JSON payload there, things will just work:
source s_tcp_json_payload tcp(); ;
parser p_json json-parser(); ;
destination d_file file("/var/log/remote.log"); ;
destination d_mongodb mongodb(collection("remote_log")); ;
log
source(s_tcp_json_payload);
parser(p_json);
destination(d_file);
destination(d_mongodb);
;Again, this assumes that the JSON payloads uses field names compatible with syslog-ng's defaults. Adapting it to use templates or value-pairs can of course be made, as shown in an example above.As an example, here's how a syslog message with JSON payload would look like:
<38>2013-07-22T13:45:54 localhost prg00000[1234]: "MESSAGE": "foo", "PROGRAM": "bar"
This would, of course, override the MESSAGE and PROGRAM fields, but the rest would be kept intact.
Mixed JSON and syslog streamIn order to parse JSON that is intermingled with normal syslog streams, we can use a new feature of syslog-ng 3.4: junctions. These allow us to create branches, and tell syslog-ng something along these lines:
If there's an incoming message, that begins with
@json:, parse it as JSON, otherwise treat it as a normal
syslog message.
We can, of course, skip the part that mandates an in-stream marker, and things will just work, but in that case, for each JSON message, an error will get logged too, which is undesirable. Therefore, if one wants to intermingle JSON and syslog traffic on the same wire, it's best to do so with the JSON having a marker prefix. The marker, however, is not going to be part of the string the json parser receives, so setting it to an opening curly bracket will not work, unless the template is modified accordingly.The config to do this looks like this:
block parser mixed-json-parser()
channel
junction
channel
parser json-parser(marker("@json:")); ;
rewrite set-tag(".json"); ;
flags(final);
;
channel
flags(final);
;
;
;
;
source s_mixed
tcp();
parser mixed-json-parser(); ;
;A more elaborate example can be found in a
post by Bazsi.Closing thoughtsThat covers most of the use cases I saw for the JSON parser, if there are cases uncovered, or questions unanswered, I can be reached via
twitter,
email and a whole host of other ways, and I will happily expand this tutorial if there's enough demand for more.
With the number of female software engineers standing at 20.9% [1], recruitment of females into the industry is a major concern. A friend of mine, Rosemary Francis who's MD of Ellexus, has a great talk about this issue. It mentions various methods you can use to tailor your job applications to be more attractive to females, by simply being careful not to exclude or put off potential applicants.
However, there is also a very poor way of doing so, which is highly sexist and plays to stereotypes. As we can see from Bloopark's hiring page, this is exactly what they have done.
I recently had to migrate the main 
Choosing Embedded Processors for AltOS
When Bdale and I started building rocketry hardware together, we had
already picked out a target processor, the TI
So 
This is my monthly summary of my free software related activities. If you re among the people who 
I also packaged python-django-jsonfield (still in NEW) since Marko has been using this python module in his code, and filed bug 
When the first draft is done, the translated text need to be
proof read, and the remaining formatting problems with images and SVG
drawings need to be fixed. There are probably also some index entries
missing that need to be added. This can be done by comparing the
index entries listed in the SiSU version of the book, or comparing the
English docbook version with the paper version. Last, the colophon
page with ISBN numbers etc need to be wrapped up before the release is
done. I should also figure out how to get correct Norwegian sorting
of the index pages. All docbook tools I have tried so far (xmlto,
docbook-xsl, dblatex) get the order of symbols and the special
Norwegian letters wrong.
There is still need for translators and people with docbook
knowledge, to be able to get a good looking book (I still struggle
with dblatex, xmlto and docbook-xsl) as well as to do the draft
translation and proof reading. And I would like the figures to be
redrawn as SVGs to make it easy to translate them. Any SVG master
around? There are also some legal terms that are unfamiliar to me.
If you want to help, please get in touch with me, and check out the
project files currently available from
Well, my 
I have finished preparing the beta 2 of the next release of OpenStack. It is currently only available from out Git on Alioth (in /git/openstack), and directly from my jenkins repository, which creates Wheezy backports for it:
The wave hits
A few hours after the warning email, details appeared on slashdot and sure enough, thousands of visitors started coming to this site.
As detailed previously, the site is running on 
My last running update was sent after I had a great April month with
two successful trail races (35km then 44km).
The next target was Mont-Blanc marathon in Chamonix valley in early July, so it's time for me to send a new update before summer.
May and June was mostly training. Huge training. Very long distances
with 375km in May and over 430 in June (breaking my former monthly
record by over 50 kilometers). Yes, that means over 14 kilometers
every day..:-)
I indeed achieved that with my "go to work partly by running" habits,
which are now deeply included in my daily schedule. Much more
convenient than bus ride (with the typical random schedules of RATP,
which French people translate by "Rentre Avec Tes Pieds" or "Come back
home by walking"....which is exactly what I'm doing).
With such training (sometimes including extra rides in the forests,
with short but steep ups and downs)....I was well prepared for THE
early summer goal : Marathon du Mont-Blanc in Chamonix (not Ultra
Trail du Mont-blanc : this one, I'm not ready for....yet)..
My first mountain trail race, and what race! Imagine 42km in the
moutain lovers heaven, namely the Chamonix Valley, with the
Mont-Blanc, Aiguille Verte, Aiguille du Midi, Aiguilles Rouges, etc.
above your head. Of course, not only in the valley but goind up and
down "slightly"....in short 2200 meters positive climb..:-)
I ran it with a female friend of mine, who I met in local races and
through runners web forums. Sabine has about the same running skills
than me and we developed a good and nice friendship, sharing our love
for running in the nature. So, during May and June, we decided to run
this race together as we were both registered for it.
A first trial in a local 35km trail race two weeks before the MMB was
very successful. We ended up in 3h40 and Sabine was ranked 5th female
runner. And, moreover, we had a tremendous pleasure by crossing the
finishing line together.
So, we did it again in Chamonix. And, hell, we did it well: I had set
the goal to 6 hours and 30 minutes and we completed the race in..... 6
hours and 34 minutes. Much much better than I and she would have
expected. And, believe me, crossing the finishing line was again a
special moment, particularly because the crowd at the end of this race
make it look like l'Alpe d'Huez on the Tour de France. And, this on a
wonderful sunny day, arriving at Planpraz, facing the Mont-Blanc. If
you've ever been to Chamonix, you know what I mean.
So, first moutain trail and, hell, not the last one! For those of you
who can read French...or just want to see some pictures of a running
bubulle,