Receive remote syslog into systemd

This article describes how to accept remote syslog messages into the systemd journal, specifically so that router logs can be journalled centrally.

The router being used is a Speedtouch TG585 v7

A simple syslog server

A syslog servers accepts syslog messages from remote hosts. The standard port for the syslog service is UDP port 514. This should be represented in /etc/services:

 syslog            514/udp

Check that the server is listening on the syslog port:

$ sudo netstat -plnut | grep 514

where

  • p shows the PID and name of the program to which the socket belongs.
  • l shows listening sockets.
  • n Show numerical addresses instead of trying to determine symbolic host, port or user names.
  • ut lists tcp and udp services

If the server isn't listening then it isn't providing the service but a simple listener can be implemented with socat, a multipurpose relay or socket cat. In a new terminal:

$ sudo socat -u UDP-RECV:514 STDOUT

which listens UDP port 514 and outputs anything received to standard output. The sudo is required because ports below 1024 are privileged. The -u option is for unidirectional mode reading from UDP-RECV:514 and writing to STDOUT.

netcat can also be used as a listener (netcat -lup 514) but it only works for one connection. The socat method works for multiple connections.

To send to the listening server, from another terminal:

$ netcat -u localhost 514

or

$ socat - UDP:localhost:514

which connects to UDP port 514 on localhost. Anything typed will be displayed by the listener.

To send an ad-hoc log dump from the Speedtouch router CLI, log in and issue this command:

{admin}=> systemlog send hist=enabled dest=10.255.255.255

where the dest is a network broadcast address.

! The Speedtouch firmware being used does not support real-time logging to a remote syslog server. This is discussed at the end of this article.

Systemd journalling

Systemd doesn't listen for inbound syslog requests. A patch was proposed but never applied. A poor man's solution can be provided by piping into systemd-cat:

$ /usr/bin/socat -u UDP-RECV:514 STDOUT | systemd-cat &

That's fine for testing but a complete solution is to implement a systemd service:

/etc/systemd/system/syslog.service
[Unit]
Description=Syslog relay

[Service]
Restart=on-success
ExecStart=/usr/bin/socat -u UDP-RECV:514 STDOUT

And then enable and start it:

# systemctl daemon-reload # to apply the new config file
# systemctl enable syslog
# systemctl start syslog

Which can be watched with

# journalctl -fu syslog

The above provides a method to journal anything received on UDP port 514.

Formatting

The ad-hoc log dump from the router doesn't send line-feeds between the log messages but sends the entire log as one, very long, line and that is journalled as a single, very long, entry. To break this up requires the service to invoke a script to apply some formatting:

#!/bin/sh
# save as /etc/systemd/scripts/syslog and chmod +x
/usr/bin/socat -u UDP-RECV:514 STDOUT | sed --unbuffered -r -e 's:(<[0-9]+>\s):\n\1:g' | systemd-cat -t router-log

Log lines begin with <n> where n is a number. The sed expression detects these and inserts a newline before them.

There is still a problem, however. sed won't process anything until it receives a newline and the router doesn't send any unless a log message contains one.
Solutions are discussed extensively on Stack Exchange.

A similar expression with Ruby instead:

/usr/bin/socat -u UDP-RECV:514 STDOUT | ruby -pe '$stdout.sync = true; $_.gsub!(/\s*(<\d+>\s)/,"\n\")' |systemd-cat -t router-log

The $stdout.sync = true prevents Ruby's standard output from blocking.

To properly solve the problem requires a little bit of coding:

#!/usr/bin/ruby
# save as /etc/systemd/scripts/syslog_linebreak.rb and chmod +x
$stdout.sync                                   # no outbound buffering by Ruby
buf=''
$stdin.each_char do |c| 
  if buf.length>0 || c=='<'                    # buffering starts when '<' received
    buf << c                                   #        and continues until flushed
    buf.gsub!(/(<\d+>)/,"\n\") if (c == '>') # regex transform matching buffer
    unless (buf =~ /<\d*$/)                    # flush buffer when regex fails
      STDOUT << buf 
      buf.replace ''                           # empty buffer stops buffering
    end 
  else
    STDOUT << c;                               # unbuffered output
  end 
  $stdout.flush                                # no buffering, please!
end

and modifying the previous script to use it instead of sed:

/usr/bin/socat -u UDP-RECV:514 STDOUT | /etc/systemd/scripts/syslog_linebreak.rb | systemd-cat -t router-log

The Ruby script works by reading its standard input one character at a time and checking it for the first match character, the <. If a different character was received then it writes it immediately to its standard output. If it did match, it instead writes to a buffer and then continues to do so unless the buffer contents fail a regex match for a partial valid delimiter (< followed by zero or more digits), in which case it flushes the buffer and buffering stops. If, while buffering, it gets a > then it performs the transformation by regex.

Real-time logging

None of the above solves the problem that the router can't be configured to log in real-time to a syslog server. One work-around is a cron job that sends the following commands over telnet:

systemlog send hist=enabled dest=10.255.255.255
systemlog flush

The flush removes logs from the router after sending them. This prevents duplicate messages being received.

My Speedtouch Tools include a stt syslog command that does the above:

$ stt syslog

Another forum post described a way to configure real-time logging by manually editing the router's configuration. This involves performing a backup and adding the following section to the downloaded file before uploading it as a backup restore.

[ syslog.ini ]
config activate=enabled timeout=1 format=standard
ruleadd fac=all dest=10.0.200.1 bootup=enabled
bootup

However, this didn't work on the TG585v7 with router firmware 8.2.6.5.



Some other information that was useful when writing this article is liked below.