This is the 13th part of my syslog-ng tutorial. Last time, we learned about sending log messages to Elasticsearch. Today, we learn about updating syslog-ng, and some of the new features of syslog-ng 4.
You can watch the video or read the text below.
https://youtu.be/205eMGS51XU
Type support in syslog-ng 4
Version 4 of syslog-ng is now available. The good news is that it is fully backwards compatible. If the version string in your configuration is set to a 3.X version, it will work as expected even after updating to version 4. Of course you might run into corner cases, but I had no problems even with complex configurations.
The major new feature of syslog-ng 4 is type support. When using the JSON and PatternDB parsers, syslog-ng stores the type information alongside name-value pairs. You can also set type information using rewrite rules.
Why type is important? You can use it in two ways. Within syslog-ng you can use it for comparisons. For example comparing numbers will work correctly. The other use is type aware output. Previously, even if syslog-ng parsed types properly, all name-value pairs were stored as if they were strings. You could hint the type information on the destination side, but it is a painful manual process. As syslog-ng now stores type information with name-value pairs, the JSON template function and various destinations can now send proper type information. For example if you extract temperature data from logs, you do not have to configure the type manually any more on the syslog-ng side, or configure type in the destination database, as syslog-ng sends temperature data as numbers without additional configuration.
Handling of lists (array) also improved considerably in this major new release.
Today we will use JSON formatted log messages from sudo 1.9.4 (or later) to demonstrate the differences between syslog-ng 3 and 4.
Syslog-ng configuration for JSON sudo logs
The first log path in following configuration collects local log messages and stores them without any further processing or filtering into /var/log/messages. There is also a second log path, which selects sudo logs, parses them using JSON parser, and then creates a JSON formatted message from the result.
@version:3.37
@include "scl.conf"
source s_sys { system(); internal();};
destination d_mesg { file("/var/log/messages"); };
log { source(s_sys); destination(d_mesg); };
filter f_sudo {program(sudo)};
destination d_test {
file("/var/log/sudo.json"
template("$(format-json --scope nv_pairs --scope dot_nv_pairs --rekey .* --shift 1 --exclude *journal* --exclude MESSAGE --scope rfc5424)\n\n"));
};
log {
source(s_sys);
filter(f_sudo);
destination(d_test);
};
The JSON template function has a few excludes to make the log messages shorter, so the MESSAGE macro, which contains the original JSON payload, and any journal macros are discarded. Two new lines at the end make the files more human readable.
Unparsed JSON sudo log
Using this configuration if you run sudo, you will find the JSON formatted log message in /var/log/messages:
Mar 17 15:24:07 localhost sudo[35095]: @cee:{"sudo":{"accept":{"uuid":"9045212067-d7cb-4a8f-ab40-32dc5b7c6f","server_time":{"seconds":1679063047,"nanoseconds":415866617,"iso8601":"20230317142407Z","localtime":"Mar 17 15:24:07"},"submit_time":{"seconds":1679063047,"nanoseconds":399116920,"iso8601":"20230317142407Z","localtime":"Mar 17 15:24:07"},"submituser":"czanik","command":"/usr/bin/ls","runuser":"root","runcwd":"/home/czanik","ttyname":"/dev/pts/0","submithost":"localhost.localdomain","submitcwd":"/home/czanik","runuid":0,"columns":118,"lines":60,"runargv":["ls","/root/"],"runenv":["LANG=en_US.UTF-8","HOSTNAME=localhost.localdomain","TERM=xterm-256color","PATH=/home/czanik/.local/bin:/home/czanik/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin","MAIL=/var/mail/root","LOGNAME=root","USER=root","HOME=/root","SHELL=/bin/bash","SUDO_COMMAND=/usr/bin/ls /root/","SUDO_USER=czanik","SUDO_UID=1000","SUDO_GID=1000"]}}}
Log parsed and recreated by syslog-ng 3.X
Now lets take a look at what syslog-ng can recreate from the original JSON log message. JSON sudo logs are automatically parsed by syslog-ng, this is why you do not see a parser declaration in the above configuration. The JSON template function then tries to recreate the log message from the name-value pairs. For the untrained eye the only difference is that syslog macros, like date or facility are now also part of the JSON:
{"cee":{"sudo":{"accept":{"uuid":"9045212067-d7cb-4a8f-ab40-32dc5b7c6f","ttyname":"/dev/pts/0","submituser":"czanik","submithost":"localhost.localdomain","submitcwd":"/home/czanik","submit_time":{"seconds":"1679063047","nanoseconds":"399116920","localtime":"Mar 17 15:24:07","iso8601":"20230317142407Z"},"server_time":{"seconds":"1679063047","nanoseconds":"415866617","localtime":"Mar 17 15:24:07","iso8601":"20230317142407Z"},"runuser":"root","runuid":"0","runenv":"LANG=en_US.UTF-8,HOSTNAME=localhost.localdomain,TERM=xterm-256color,PATH=/home/czanik/.local/bin:/home/czanik/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin,MAIL=/var/mail/root,LOGNAME=root,USER=root,HOME=/root,SHELL=/bin/bash,\"SUDO_COMMAND=/usr/bin/ls /root/\",SUDO_USER=czanik,SUDO_UID=1000,SUDO_GID=1000","runcwd":"/home/czanik","runargv":"ls,/root/","lines":"60","command":"/usr/bin/ls","columns":"118"}}},"app":{"name":"cee"},"SOURCE":"s_sys","PROGRAM":"sudo","PRIORITY":"notice","PID":"35095","HOST_FROM":"localhost","HOST":"localhost","FACILITY":"authpriv","DATE":"Mar 17 15:24:07"}
If you take a more careful look, or if you use an app like jq to display the JSON message, you will catch a few more differences. The list looks slightly differently, and numbers are enclosed in quotes, meaning that they are forwarded as string instead of as numbers.
"runuser": "root",
"runuid": "0",
"runenv": "LANG=en_US.UTF-8,HOSTNAME=localhost.localdomain,TERM=xterm-256color,PATH=/home/czanik/.local/bin:/home/czanik/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin,MAIL=/var/mail/root,LOGNAME=root,USER=root,HOME=/root,SHELL=/bin/bash,\"SUDO_COMMAND=/usr/bin/ls /root/\",SUDO_USER=czanik,SUDO_UID=1000,SUDO_GID=1000",
"runcwd": "/home/czanik",
"runargv": "ls,/root/",
"lines": "60",
"command": "/usr/bin/ls",
"columns": "118"
Syntax check on update
Now it is time to update syslog-ng from version 3 to version 4. It is different on each and every Linux distribution or BSD variant and thus not the scope of this tutorial. Once you updated syslog-ng, it is time for a syntax check. It shows you a couple warnings, what you should change in your configuration or what changed in the behavior of syslog-ng.
[root@localhost syslog-ng]# syslog-ng -s -f sudo.conf
[2023-03-17T15:50:53.583894] WARNING: Configuration file format is too old, syslog-ng is running in compatibility mode. Please update it to use the syslog-ng 4.1 format at your time of convenience. To upgrade the configuration, please review the warnings about incompatible changes printed by syslog-ng, and once completed change the @version header at the top of the configuration file; config-version='3.37'
[2023-03-17T15:50:53.591649] WARNING: $(format-json) starts using type information associated with name-value pairs in syslog-ng 4.0. This can possibly cause fields in the formatted JSON document to change types if no explicit type hint is specified. This change will cause the type in the output document match the original type that was parsed using json-parser(), add --no-cast argument to $(format-json) to keep the old behavior;
In this particular case it reminds you that the current version is 4.1 while the version is set to 3.37 in the configuration. The second warning is about the changed behavior of the JSON template function and it also shows you how to change the configuration if you want to keep the old behavior with the new configuration version.
Log parsed and recreated by syslog-ng 4.X
On first look you might not notice, but the JSON generated by syslog-ng 4 is very different from the JSON generated by syslog-ng 3:
{"cee":{"sudo":{"accept":{"uuid":"3d3e1f8c9b-d62e-4081-8fff-bc71e80e29","ttyname":"/dev/pts/0","submituser":"czanik","submithost":"localhost.localdomain","submitcwd":"/home/czanik","submit_time":{"seconds":1679063411,"nanoseconds":336097049,"localtime":"Mar 17 15:30:11","iso8601":"20230317143011Z"},"server_time":{"seconds":1679063413,"nanoseconds":696181185,"localtime":"Mar 17 15:30:13","iso8601":"20230317143013Z"},"runuser":"root","runuid":0,"runenv":["LANG=en_US.UTF-8","HOSTNAME=localhost.localdomain","TERM=xterm-256color","PATH=/home/czanik/.local/bin:/home/czanik/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin","MAIL=/var/mail/root","LOGNAME=root","USER=root","HOME=/root","SHELL=/bin/bash","SUDO_COMMAND=/usr/bin/ls /root/","SUDO_USER=czanik","SUDO_UID=1000","SUDO_GID=1000"],"runcwd":"/home/czanik","runargv":["ls","/root/"],"lines":60,"command":"/usr/bin/ls","columns":118}}},"app":{"name":"cee"},"SOURCE":"s_sys","PROGRAM":"sudo","PRIORITY":"notice","PID":"35167","HOST_FROM":"localhost","HOST":"localhost","FACILITY":"authpriv","DATE":"Mar 17 15:30:13"}
It is easier to spot the differences with some formatting:
"runuser": "root",
"runuid": 0,
"runenv": [
"LANG=en_US.UTF-8",
"HOSTNAME=localhost.localdomain",
"TERM=xterm-256color",
"PATH=/home/czanik/.local/bin:/home/czanik/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin",
"MAIL=/var/mail/root",
"LOGNAME=root",
"USER=root",
"HOME=/root",
"SHELL=/bin/bash",
"SUDO_COMMAND=/usr/bin/ls /root/",
"SUDO_USER=czanik",
"SUDO_UID=1000",
"SUDO_GID=1000"
],
"runcwd": "/home/czanik",
"runargv": [
"ls",
"/root/"
],
"lines": 60,
"command": "/usr/bin/ls",
"columns": 118
Both lists and numbers are handled properly.
For more syslog-ng 4 info
By the time of this tutorial the documentation for syslog-ng version 4 is not yet available. You can learn more about the changes from the release notes: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.0.0
If you use Python to extend syslog-ng, you can find information how Python support works in syslog-ng version 4 at: https://github.com/syslog-ng/syslog-ng/tree/master/modules/python-modules
If you have any questions or comments, leave a comment on YouTube or reach out to me on Twitter / Mastodon.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.-
The modules for EL-9 are available for x86_64 and aarch64.
