Continuing my
HOWTO series, in this article, I will explore how to set up syslog-ng to process JSON in various scenarios. We will touch the subject of interoperability, and have a look at multiple scenarios, in the hope that readers will find something similar to their own needs. There are many different ways in which one can plug JSON into their logging system, and it is very likely that I will not be able to cover it all within this HOWTO. However, I will explain the basics, and how to plug the pieces together, which should at least be a useful foundation to build even more complex systems onto.In essence, we will look at the following scenarios:
raw JSON transport,
JSON over syslog,
JSON and syslog payload in the same stream. The documentation will cover both syslog-ng 3.4 and 3.5, where the 3.5 parts will be clearly marked as such. The 3.3 series do not have a JSON parser, and as such, is insufficient for most needs explored in this guide.In the default setting, the JSON parser will parse the MESSAGE part of an incoming log event, and place every key into a macro of the same name. If the incoming JSON is structured, it the keys will be converted to dot notation format:
"deeper": "level": "very much so"
In this case, one can access the value as
$ deeper.level within a syslog-ng template.The parsed keys can be prefixed using the
prefix() option of the json-parser. One can also change what the parser will receive as input by using the
template() option.See the
documentation for more information on these.
Transporting raw JSONThe simplest scenario is to take a transport like TCP, and simply push raw JSON structures through, one each line, and have syslog-ng parse that to its internal representation. One can do all kinds of things with the data then: transport it further as JSON, store it in files, in a database - the possibilities are endless, but we will explore two scenarios.JSON on TCP, using syslog field namesThe first one is receiving JSON on TCP, using syslog field names, store the parsed stuff in a file (using the conventional format), and in mongodb.
source s_tcp_json tcp(port(10514) flags(no-parse)); ;
parser p_json json-parser(); ;
destination d_file file("/var/log/remote.log"); ;
destination d_mongodb mongodb(collection("remote_log")); ;
log
source(s_tcp_json);
parser(p_json);
destination(d_file);
destination(d_mongodb);
;This needs a bit of an explanation: we use the no-parse flag on the source, because by default, the
tcp() source expects a message that conforms to the old syslog protocol. We have raw JSON, and we need to tell the source that. The other elements should be self explanatory, I believe, with the exception of
json-parser(): we give it no parameters, so it will not modify the keys in the JSON it receives, keeping them intact, as-is.For this reason, incoming messages need to have the same keys as syslog-ng normally would use internally: DATE, HOST, MESSAGE, and so on, like in the following example:
"PROGRAM":"prg00000","PRIORITY":"info","PID":"1234","MESSAGE":"seq: 0000000000, thread: 0000, runid: 1374490607, stamp: 2013-07-22T12:56:47 MESSAGE...","HOST":"localhost","FACILITY":"auth","DATE":"Jul 22 12:56:47"
JSON on TCP, custom field namesA more likely scenario is when we receive JSON, but not in a layout syslog-ng uses internally. In this case, we need to rewrite the message a little, and possibly use templates, depending on what kind of output we wish to use.Lets assume the incoming JSON has these fields:
"app":"prg00000","prio":"info","id":"1234","msg":"seq: 0000000000, thread: 0000, runid: 1374490607, stamp: 2013-07-22T12:56:47 MESSAGE...","source":"localhost","timestamp":"Jul 22 12:56:47"
As you can see, this has similar fields, but not quite, and
FACILITY is missing too.In this case, we'll use a different configuration:
source s_tcp_json tcp(port(10514) flags(no-parse)); ;
parser p_json json-parser(prefix(".json.")); ;
destination d_file file("/var/log/remote.log"
template("$ .json.timestamp $ .json.source $ .json.app [$ .json.id ]: $ .json.msg \n")); ;
destination d_mongodb mongodb(collection("remote_log")
value-pairs(
pair("PROGRAM" "$ .json.app ")
pair("HOST" "$ .json.source ")
pair("PID" "$ .json.id ")
pair("MESSAGE" "$ .json.msg ")
pair("DATE" "$ .json.timestamp ")
pair("PRIORITY" "$ .json.prio ")
pair("FACILITY" "auth")
)); ;
log
source(s_tcp_json);
parser(p_json);
destination(d_file);
destination(d_mongodb);
;
We use the prefix() option of
json-parser(), to avoid collisions with the default syslog-ng namespace. Then, we use a template for the file destination, and
value-pairs() for the MongoDB one. Doing all this, we get similar results as we did in the first scenario.
JSON over syslogAn even likelier scenario is when we have JSON payload in the message part of a normal syslog message. This is useful because the logs themselves remain syslog compatible, and can be transported through devices that know no better.Thankfully, if we do not tell syslog-ng that we do not want any parsing on the input, the json-parser will receive the MESSAGE part of the log, so if we have JSON payload there, things will just work:
source s_tcp_json_payload tcp(); ;
parser p_json json-parser(); ;
destination d_file file("/var/log/remote.log"); ;
destination d_mongodb mongodb(collection("remote_log")); ;
log
source(s_tcp_json_payload);
parser(p_json);
destination(d_file);
destination(d_mongodb);
;Again, this assumes that the JSON payloads uses field names compatible with syslog-ng's defaults. Adapting it to use templates or value-pairs can of course be made, as shown in an example above.As an example, here's how a syslog message with JSON payload would look like:
<38>2013-07-22T13:45:54 localhost prg00000[1234]: "MESSAGE": "foo", "PROGRAM": "bar"
This would, of course, override the MESSAGE and PROGRAM fields, but the rest would be kept intact.
Mixed JSON and syslog streamIn order to parse JSON that is intermingled with normal syslog streams, we can use a new feature of syslog-ng 3.4: junctions. These allow us to create branches, and tell syslog-ng something along these lines:
If there's an incoming message, that begins with
@json:, parse it as JSON, otherwise treat it as a normal
syslog message.
We can, of course, skip the part that mandates an in-stream marker, and things will just work, but in that case, for each JSON message, an error will get logged too, which is undesirable. Therefore, if one wants to intermingle JSON and syslog traffic on the same wire, it's best to do so with the JSON having a marker prefix. The marker, however, is not going to be part of the string the json parser receives, so setting it to an opening curly bracket will not work, unless the template is modified accordingly.The config to do this looks like this:
block parser mixed-json-parser()
channel
junction
channel
parser json-parser(marker("@json:")); ;
rewrite set-tag(".json"); ;
flags(final);
;
channel
flags(final);
;
;
;
;
source s_mixed
tcp();
parser mixed-json-parser(); ;
;A more elaborate example can be found in a
post by Bazsi.Closing thoughtsThat covers most of the use cases I saw for the JSON parser, if there are cases uncovered, or questions unanswered, I can be reached via
twitter,
email and a whole host of other ways, and I will happily expand this tutorial if there's enough demand for more.
So 
Shipping a UEFI-bootable Linux distribution is a touch awkward[1], with the main sticking point being the necessity to produce boot media with multiple El-Torito images. An El-Torito image is either an image of a floppy disk or a small hard drive, embedded into the ISO. This allows BIOS systems to look for an El Torito image, hook some interrupts and then boot without the BIOS having to care about the fact that the image is embedded in an ISO-9660 filesystem. UEFI systems will look for an El-Torito image with a special tag - if they find it they'll mount it as a FAT filesystem and look for a bootloader, and if not they'll fall back to BIOS compatibility.
Following 
I have finished preparing the beta 2 of the next release of OpenStack. It is currently only available from out Git on Alioth (in /git/openstack), and directly from my jenkins repository, which creates Wheezy backports for it:
The wave hits
A few hours after the warning email, details appeared on slashdot and sure enough, thousands of visitors started coming to this site.
As detailed previously, the site is running on 
Given a model:
Last week, the second milestone of the Havana developement branch of
Ceilometer has been released and is now available for testing and download.
This means the first half of the OpenStack Havana development has passed!
New features
Ten blueprints have been implemented as you can see on the
This is time again for my annual report about the number of developers per
country.
This is now the fifth edition of this report, after the 
We are happy to announce that the 
Present Extension Redirection
Multi-buffered applications have always behaved poorly in the presence
of the Composite extension: