Real-time Log Centralization with Rsyslog.
Rsyslog Configuration to centralize and ship your Linux servers logs
Applications and servers generate tons of information through log events. However, the monitoring of heterogeneous logs coming from tens or hundreds of machines is hard, and sometimes impossible, without an efficient Log Management tool.
The good news is that centralizing log data to a remote place from almost any Linux distribution is actually quite easy. And yes, they almost all come with Rsyslog installed. This daemon is able to watch files and publish data through TCP or UDP connections, as well as gather all the system logs.
In this article, we’ll show you how to configure your Rsyslog in 5 minutes and transmit log events to a central syslog daemon
Forward all the syslog messages
If you run Ubuntu, CentOs, Debian, Fedora, openSUSE or any other major linux distribution you should find a rsyslog.log file under your /etc/ directory. To forward all the syslog messages through a TCP socket we will do some edition:
sudo vim /etc/rsyslog.conf
Indicate that you want to send all the messages to a unified syslog serviceon TCP PORT 10514 for instance. The RFC-5424 format is very useful as it is automatically recognized and parsed by many log management solutions. So we want to format all logs as RFC-5424:
$template unifiedFormat,” <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% – – – %msg%\n”
*.* @@unified.syslog.com:10514;unifiedFormat
The *.* before the TCP address is aRsyslog’s selectors which means that you want to send all the messages.
Once you are happy with this configuration don’t forget to restart the service:
sudo service rsyslog restart
Watching your own files
If your applications do not publish their logs natively in syslog format you can also use Rsyslog as a file content forwarder. To do so, follow these steps:
Edit the rsyslog.conf file:
sudo vim /etc/rsyslog.conf
Insert the following configuration block for each file you want to follow:
# Prefix all the followed filed by these 4 lines
$ModLoadimfile
$InputFilePollInterval 10
$PrivDropToGroupadm
$WorkDirectory /var/spool/rsyslog
# Input for FILE1
$InputFileName /path/to/your/log/file.log
$InputFileTag your-app-name
$InputFileStateFile be-sure-that-name-is-unique
$InputFileSeverity info
$InputRunFileMonitor
# Input for FILE2
$InputFileName /path/to/another/log/file.log
$InputFileTag your-secondary-app-name
$InputFileStateFilebe-sure-that-name-is-unique-to-track-offset$InputFileSeverity info
$InputRunFileMonitor
If you need more details about this configuration, please refer to the imfile module documentation.
Again, once you’re done with this rsyslog configuration don’t forget to restart the service:
sudo service rsyslog restart
Going deeper into Rsyslog configuration
Enable TCP SSL security
You can set up encryption for Rsyslog. To do so, you will need to download your log management certificate first. Then install support TLS for Rsyslog. Usually this is done by installing rsyslog-gnutls via apt-get:
sudo apt-get install rsyslog-gnutls
If you followed the previous step, you can now enable TLS by replacing the following:
$template unifiedFormat,” <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% – – – %msg%\n”
*.* @<@unified.syslog.com:10514;unifiedFormat
by:
$template unifiedFormat,” <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% – – – %msg%\n”
$DefaultNetstreamDriverCAFile
$ActionSendStreamDrivergtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode x509/name
*.* @@secure.unified.syslog.com:10515;unifiedFormat
Don’t forget to specify path_to_your_.crt_file, your_api_key and ensure that the port number is the right one (we used 10515 here).
Then you should restart the service.
Filter logs based on the application name
Sometimes, you don’t want to send all your logs to the centralized platform. In order to filter them based on the application name, you have to replace in your rsyslog.conf, the following line:
*.* @@unified.syslog.com:10514;unifiedFormat
by
if $programname == ‘my-app-name’ then @@unified.syslog.com:10514;unifiedFormat
Wrapping up
In this article, we discussed how to set your Rsyslog configuration to centralize and categorize any kind of linux server logs easily. Depending if you are a Java, Node.js or Ruby on Rails developer, if you use Apache or Nginx we encourage you to define a proper logging strategy so you can monitor better and troubleshoot 10x times faster.
