pimp my syslog server or "Setting up a syslog server with syslog-ng, Elasticsearch and Kibana"

What it looked like before

There are several devices in the jumpNet sending syslog messages to a central syslog server hosted in my part of the network. I have to confess, I never bothered to think about a scalable logging solution before. Therefore, my first syslog configuration was very basic. Just a rsyslogd collecting messages, matching them against facility local2 and then putting them into a single file. After admitting it was a bad setup, I configured rsyslog to split the received syslog messages to different files dependent on the sending IP address.

What a modern syslog server should provide

Last week, a customer appealed me and asked for an easy to use and yet powerful and scalable solution he could send all his syslog messages to. Uhm. Guess what happened - I suggested setting up an ESK stack. Though my own syslog solution looked like an old tractor besides ESK and I refused to dive into Elasticsearach and Kibana, it just blurted out of me.

So I read about the combination of syslog-ng, Elasticsearch and Kibana and to be honest, I was impressed. At least, I suggested a reasonable solution to my customer ;-)

For those of you, who never heared about Elasticsearch and Kibana: Elasticsearch is a powerful NoSQL (I still hate this hipster term) database system which can be searched incredibly fast. Elasticsearch is complemented by its web frontend Kibana (even more hipster!). With Kibana, you can query Elasticsearch, create visualizations and do some other fancy stuff.

Installing the ESK stack on Ubuntu 16.04 LTS

Okay, I know. You're tired of my introductory story. You want to get to the real shit. Here it comes.

Set up an Ubuntu 16.04 LTS server.

If you like to use apt, you need to add the syslog-ng repository and the Elasticsearch and Kibana repository. Start by importing the key for the syslog-ng repositoriy. Check the fingerprint before executing apt-key add!

wget http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUbuntu_17.04/Release.key
apt-key add Release.key

Now create an additional sources file in /etc/apt/sources.list with this line in it:

deb http://download.opensuse.org/repositories/home:/laszlo_budai:/syslog-ng/xUbuntu_16.04 ./

Afterwards you can run apt-get update && apt-get install syslog-ng and syslog-ng will be installed on the system.

Next step: Add the Elasticsearch and Kibana repository. It's the same procedure:

wget https://artifacts.elastic.co/GPG-KEY-elasticsearch
apt-key add GPG-KEY-elasticsearch

Create another additional file in /etc/apt/sources.list with this line in it:

deb https://artifacts.elastic.co/packages/5.x/apt stable main

Afterwards, execute apt-get update && apt-get install elasticsearch kibana to install elasticsearch and kibana on your system.

Configuring syslog-ng

The configuration file for syslog-ng is located in /etc/syslog-ng/syslog-ng.conf. In my case, I installed syslog-ng 3.12. Make sure, your config starts with @version 3.12 as syslog-ng will complain at startup if the configuration version does not match the binary's version. Also make sure, you add @module mod-java.

The header of the file should look like this:

# this is the very beginning of the file
@version: 3.12
@include "scl.conf"
@module mod-java

I used a separate file to configure logging to an Elasticsearch destination. Therefore I created a file elastic.conf in /etc/syslog-ng/conf.d with this content:

##########
# network sources
source s_net {
  udp(port(514));
  tcp(port(514));
  syslog();
};

##########
# Elasticsearch destination
destination d_es {
  elasticsearch2(
    client-lib-dir("/usr/share/elasticsearch/lib/")
    client-mode("http")
    index("syslog-ng_${YEAR}.${MONTH}.${DAY}")
    type("test")
    cluster("syslog")
    template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n")
    time-zone("UTC")
  );
};

###########
# sending logs to ES destionation
log {
  source(s_net);
  destination(d_es);
  flags(flow-control);
};

Unfortunatelly, there is a problem finding the required java libraryies preventing syslog-ng to startup. To circumvent that issue, you need to add the libraries' path to LD_LIBRARY_PATH. As I'm a little unsure wheter the classic init file is used or the systemd unit file, I added the needed environment variable in both, but I think it should be sufficient to just modify the unit file (/etc/systemd/system/multi-user.target.wants/syslog-ng.service). Additionally, make sure, elasticsearch and kibana are already running before syslog-ng is started. You do so by defining Wants in the [Unit] section. All changes to the default file are in bold.

[Unit]
Description=System Logger Daemon
Documentation=man:syslog-ng(8)
Wants=elasticsearch.service
Wants=kibana.service

[Service]
Environment=LD_LIBRARY_PATH=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/:$LD_LIBRARY_PATH
Type=notify
ExecStart=/usr/sbin/syslog-ng -F $SYSLOGNG_OPTS
ExecReload=/bin/kill -HUP $MAINPID
EnvironmentFile=-/etc/default/syslog-ng
EnvironmentFile=-/etc/sysconfig/syslog-ng
StandardOutput=journal
StandardError=journal
Restart=on-failure

[Install]
WantedBy=multi-user.target

Phew, that's it for syslog-ng.

Update: I just corrected a mistake I made: The Wants must be in the [Unit] section of the unit file and not - as I previously did - in the [Install] section.

Configuring elasticsearch

The configuration of elasticsearch is located in /etc/elasticsearch/elasticsearch.yml.

I changed the cluster name, the node name, the data and the log path:

# the clustername must match the name defined in syslog-ng's configuration!
cluster.name: syslog
node.name: syslog01
path.data: /var/lib/elasticsearch/data
path.logs: /var/log/elasticsearch

Finally, create a symbolic link to the unit file:

cd /etc/systemd/system/multi-user.target.wants
ln -s /usr/lib/systemd/system/elasticsearch.service

That was easy, wasn't it?

Configuring Kibana

Kibana's configuration is located in /etc/kibana/kibana.yml. I just made a few changes to the configuration: I made Kibana binding to all available interfaces and explicitly set the listening port to 5601:

server.port: 5601
server.host: 0.0.0.0

As you needed to to for elasticsearch, you propably need to create another symbolic link to the unit file:

cd /etc/systemd/system/multi-user.target.wants
ln -s /etc/systemd/system/kibana.service

That's it.

Testing

Try rebooting the system. If everything is configured the right way, the system should come up along with elasticsearch, syslog-ng and kibana.

Now, configure any syslog capable device to send syslog mesages to your ESK stack. Afterwards, open your browser and navigate to http://your-system:5601. You will be prompted to create an index in Kibana. If there already is some data in Elasticsearch, you can enter "syslog-ng_*" as index and the rest will be discovered automagically.

Have fun exploring the possibilities of Elasticsearch and Kibana.

Security considerations

Stop, stop, stop. Don't quit reading at this point!

The provided setup is granting access to Kibana to everyone (remember server.host 0.0.0.0?) and that's a poor design. You really need to add some sort of access control mechanism.

Either you configure a host firewall on your syslog server to restrict access to certain IP addresses or ranges or you use a HTTP proxy which can provide some sort of authentication.

Keep in mind: Log messages can contain a plenty of sensitive information, for example IP and MAC addresses, usernames or even passwords if something is logging in debug mode (freeradius is a good example for this behavior) and as administrator your are in charge to protect this information against unauthorized access!

Another thing: There is Java on the server. I don't know if Java should generally be deemed as beeing insecure, but I must admit, it feels a little uncomfortable having Java handling syslog messages. *sigh*. I guess this is the tradeoff for having some really cool hipster stuff handling your logs...

Sources

  1. https://www.balabit.com/blog/installing-the-latest-syslog-ng-on-ubuntu-and-other-deb-distributions/
  2. https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html
  3. https://www.balabit.com/documents/pdf/syslog-ng-whitepaper-elasticsearch-kibana.pdf (This link does not take you to the mentioned PDF anymore...)

Go back