Wazuh Syslog Relay with Rsyslog and Wazuh Agent
In addition to log events collected by Wazuh agents, those generated by syslog sending devices represent a valuable source of log messages for your SIEM to analyze. Wazuh managers can function as network syslog receivers, but their syslog handling is very limited, not supporting rate limiting, message filtering, or message transforms, and requiring network proximity to syslog senders to ensure secure and reliable delivery of syslog messages.
Having used Wazuh for many years to analyze network syslog messages, I have found an especially robust way to funnel these messages into the Wazuh SIEM is to use one or more appropriately located Linux systems as Wazuh syslog relay hosts. Such hosts are configured with rsyslog to locally receive network syslog messages with a Wazuh agent configured to securely and reliably forward them to the Wazuh server for analysis. This brings the following benefits:
- Syslog senders at a given site can, with moderate security and reliability, transmit messages to a Wazuh syslog relay host at the same site, which then can with high levels of privacy and reliability convey them to a Wazuh manager at another site or in the cloud.
- At very low computational cost, rsyslog rules can be used to filter out syslog “noise” messages before they can put any load on the Wazuh server.
- The native Wazuh agent flood protection feature can guard the Wazuh manager from being flooded by excessive syslog messages.
- Other rich features of rsyslog can be employed, such as advanced queueing and rate limiting, message filtering, message transformation, and syslog over TLS.
Setting up Wazuh Syslog Relay Hosts
Location
For the purpose of security and to minimize message loss, it is best to have a syslog relay host at each physical site where you have syslog sending devices, since in most cases those devices will use the cleartext and unreliable UDP protocol for syslog message transport. You do not generally need a syslog relay host on every VLAN or subnet, but it is best to avoid pushing syslog messages across WAN links, VPN tunnels, or congested/lossy network paths. Make sure that any firewalls, routers, or layer 3 switches along the path between the syslog senders and the syslog relay hosts will permit the syslog messages to pass through.
Types of relay hosts and resource requirements
Memory and CPU requirements are quite small for syslog relay hosts. The main disk I/O will involve appending syslog messages to a queue file. Disk storage needs to be large enough to accommodate the queue file as it grows until its next periodic deletion, usually daily. A small Linux virtual machine can serve this purpose quite well. A dedicated physical Linux host, even as small as a Raspberry Pi, should be able to handle relaying even a modest volume of syslog messages. It may also be reasonable to add the syslog relay role to a Linux host that is already serving other purposes.
Local configuration of syslog relay hosts
Linux firewall
Ensure that the Linux firewall (iptables or nftables) on each syslog relay host will allow incoming tcp and udp port 514 packets from on-site devices to reach rsyslog.
Rsyslog configuration
Add or uncomment these lines in /etc/rsyslog.conf to enable udp and tcp port 514 network listeners:
# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")
# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")Rsyslog may have repeated message reduction enabled, which is not well suited for this use case. Make sure it is explicitly disabled in /etc/rsyslog.conf with:
$RepeatedMsgReduction offCreate an additional config file /etc/rsyslog.d/10-wazuh.conf:
template(name="ForwardFormat" type="list") {
property(name="timegenerated" dateFormat="rfc3339")
constant(value=" ")
property(name="hostname")
constant(value=" ")
property(name="syslogtag")
property(name="msg" spifno1stsp="on" )
property(name="msg")
constant(value="\n")
}
# Write network syslog records to disk and process them no further
if $inputname == "imudp" or $inputname == "imtcp" then /var/log/syslogtowazuh;ForwardFormat
& stopThen restart the rsyslog daemon to apply the changes.
systemctl restart rsyslogThe above will cause every syslog message arriving via udp or tcp to be spooled to the queue file /var/log/syslogtowazuh. The special template causes the timestamp in the syslog header to be replaced with a high precision timestamp based on the local system time of the syslog relay host itself. This is much more accurate than the low precision timestamps commonly included in the syslog messages themselves. It is also more reliable because many syslog sending devices can vary with respect to time zone and clock accuracy.
Log rotation of queue file
Create an /etc/cron.d/rotate-syslogtowazuh file that rotates and/or deletes your queue file frequently enough to avoid the disk filling up. This example would clear out the queue file daily at 2am.
0 2 * * * root rm /var/log/syslogtowazuhCentralized configuration of Wazuh agent on the syslog relay hosts
Next we need to configure the Wazuh agents that have the syslog relay role, to collect the syslog messages as they are appended to the /var/log/syslogtowazuh file, and to convey them to the Wazuh server.
This could be done by adding a
Create a “syslog-relay” agent group and configure it.
In Wazuh Dashboard, navigate to ☰ ➜ Agents management ➜ Groups ➜ Add new group.
Enter in the “syslog-relay” agent group name and save it.
If “syslog-relay” is not visible in the group list, then type “syslog-relay” into the Search window and hit Enter.
To the far right of the “syslog-relay” group entry, click on the pencil icon to edit the group configuration.
In the web text editor, entirely replace the content with the following.
<agent_config>
<localfile>
<log_format>syslog</log_format>
<location>/var/log/syslogtowazuh</location>
</localfile>
</agent_config>Member the syslog relay host(s) into the new “syslog-relay” agent group
Navigate to ☰ ➜ Agents management ➜ Groups.
If “syslog-relay” is not already listed on the current page, then type that name into the Search field and hit Enter.
Click directly on the name of the “syslog-relay” group.
Click on “Manage agents”.
On the “Available agents” side, select all the syslog relay hosts and then click “Add selected items”.
Click “Apply changes”.
Wazuh indexer ingest node pipeline configuration
Edit the ingest node pipeline definition file
On your Wazuh server where Filebeat is running, make a backup of the ingest node pipeline definition file /usr/share/filebeat/module/wazuh/alerts/ingest/pipeline.json. Then edit the original, initially finding the following line near the top of the file:
{ "json" : { "field" : "message", "add_to_root": true } },Directly below that line, insert the following:
{
"dissect" : {
"description" : "Extract predecoder.hostname from syslog message",
"field" : "full_log",
"pattern" : "%{?TIMESTAMP_ISO8601} %{predecoder.hostname} %{?GREEDYDATA}",
"ignore_missing" : true,
"ignore_failure" : true,
"if" : "ctx.location == '/var/log/syslogtowazuh'",
"tag" : "dissect_extract_hostname_syslog"
}
},
{
"set": {
"description": "Preserve the original agent name as syslog_relay_host",
"field": "syslog_relay_host",
"value": "{{agent.name}}",
"if" : "ctx.location == '/var/log/syslogtowazuh'"
}
},
{
"set": {
"description": "For relayed syslog events, rename agent.name to name of original syslog sender",
"field": "agent.name",
"value": "{{predecoder.hostname}}",
"if" : "ctx.location == '/var/log/syslogtowazuh'"
}
},
{
"date" : {
"field" : "predecoder.timestamp",
"target_field" : "timestamp",
"formats" : [
"ISO8601"
],
"ignore_failure" : true,
"if" : "ctx.location == '/var/log/syslogtowazuh'"
}
},The three ingest node processors above all act solely on messages log collected from the /var/log/syslogtowazuh queue file.
The “dissect” processor extracts the hostname from the syslog header into the predecoder.hostname field. Wazuh’s native syslog predecoder normally performs this, but due to a bug fails to do so when there is a high precision timestamp in the syslog header.
The “set” processor overwrites the agent.name field with the value in the predcoder.hostname field so that relayed syslog messages are assigned an agent.name that reflects the originator of the syslog message rather than the host that relayed it.
The “date” processor established the official timestamp of the event as the one assigned by rsyslog, reflecting the precise time the event was received.
Push the revised pipeline.json to the Wazuh indexer
Run this command:
filebeat setup --pipelines --modules wazuhTest with a simulated SonicWALL syslog message
For syslog message test generation purposes, install netcat as appropriate for your operating system. The following commands would work on modern Ubuntu/Debian and Redhat varieties of Linux, respectively:
apt install netcat-traditional
yum install netcatOn your syslog relay host, use netcat to simulate the reception of a SonicWALL firewall syslog message.
nc 127.0.0.1 -u 514 -w 1 <<< '<1>'`date "+%b %d %H:%M:%S"`' my_sonicwall id=sonicwall sn=C0EBA485A87C time="'`date "+%Y-%m-%d %H:%M:%S UTC"`'" fw=142.40.29.3 pri=1 c=1024 m=97 app=11 sess="Auto" n=187277068 usr="Unknown (SSO failed)" src=172.12.6.125:60423:X0 dst=40.70.229.150:443:X1 srcMac=54:bf:64:68:fc:65 dstMac=f0:8e:db:00:73:02 proto=tcp/https sent=1882 rcvd=3517 dpi=1 dstname=array501.prod.do.dsp.mp.microsoft.com arg=/ appcat="PROTOCOLS" appid=1279 code=27 Category="Information Technology/Computers" note="Policy: fatal Office Block Policy, Info: 6148 " rule="28 (LAN->WAN)" fw_action="NA"'Confirm it was appended to your new queue file.
grep "my_sonicwall" /var/log/syslogtowazuh | tail -n1Confirm the new syslog event record was log collected from your queue file and indexed by Wazuh, by searching in Wazuh Dashboard with this query string:
decoder.name:sonicwall and data.srcip:"172.12.6.125"The agent.name field for the alert should be “my_sonicwall” instead of the agent name of the syslog relay host.
Configure syslog sending devices
As appropriate for each device type, configure your syslog capable devices to send syslog messages to the IP number or DNS name of the appropriate local syslog relay host.
Remember Wazuh rules
Remember that of all the syslog messages relayed to the Wazuh Server for analysis, only the ones that match a rule of sufficiently high severity (3 or higher by default) will generate an alert that will be stored in Wazuh Indexer and be visible in Wazuh Dashboard. Especially for syslog sending devices not covered in the standard Wazuh ruleset, it is up to you to provide custom decoder and rule families to identify logs coming from those unknown device types and to determine which messages from those devices warrant what kind of alert, if any, to be generated.
Advanced possibilities
Filter out specific incoming syslog messages in rsyslog
It is desirable but not always possible to configure syslog senders to only send the messages you want to collect, but when needed, you can configure rsyslog to discard undesirable messages so that Wazuh does not process them at all.
Insert at the top of /etc/rsyslog.d/10-wazuh.conf one or more lines like this to discard specific syslog network messages:
# example to exclude udp syslog messages from the SonicWALL at 172.16.1.1 of type "m=36"
if $inputname == "imudp" and $fromhost-ip == '172.16.1.1' and $msg contains ' m=36 ' then stopThen restart the rsyslog service
systemctl restart rsyslogSetting up rate limiting and a disk assisted memory queue in rsyslog
If a particular syslog relay host might be subject to very high bursts of syslog messages that exceed the Wazuh agent’s native buffering and rate limiting capabilities to keep up without message loss, then you might want to look into rsyslog’s very advanced rate limiting and disk assisted memory queue capabilities.
https://www.rsyslog.com/doc/concepts/queues.html#disk-assisted-memory-queues
Load balancing syslog messages across multiple syslog relay hosts
If you have certain syslog sending devices that produce very high sustained rates of syslog messages, you may need to put a load balancer between them and multiple syslog relay hosts. Nginx, haproxy, and various network appliances are capable of supporting this, though the details are outside the scope of this article.
Receiving syslog over TLS
There are many cloud based services and devices capable of transmitting their log messages via syslog over TLS, usually using tcp transport for reliable and private delivery of syslog messages. For example
https://cloudone.trendmicro.com/docs/workload-security/event-syslog
Rsyslog can be configured to receive syslog messages of this type, though it is a bit complicated to work through the details of TLS keys and certificates, certificate authorities, and host name verification. The relevant rsyslog documentation is here: