Taxonomy Icon

Linux

Overview

In this tutorial, learn to:

  • Configure the syslog daemon
  • Understand standard facilities, priorities, and actions
  • Configure log rotation
  • Understand rsyslog and syslog-ng

What’s happening inside your system

A Linux system has many subsystems and applications running. You use system logging to gather data about your running system from the moment it boots. sometimes you just need to know that all is well. At other times you use this data for auditing, debugging, knowing when a disk or other resource is running out of capacity, and many other purposes. You can collect log data on one system and forward it to another system for processing. Log data can be displayed on a terminal, such as the root user’s terminal, but more often it is saved in files, or forwarded over sockets to a log server. Needless to say, logging is highly configurable.

The traditional syslog facility and its syslogd daemon provide this logging. Nowadays, syslog has been supplemented by other logging facilities such as rsyslog, syslog-ng, and the systemd journal subsystem. I introduce you to these facilities in this tutorial.

This tutorial helps you prepare for Objective 108.2 in Topic 108 of the Linux Server Professional (LPIC-1) exam 102. The objective has a weight of 3.

Prerequisites

To get the most from the tutorials in this series, you need a basic knowledge of Linux and a working Linux system on which you can practice the commands covered in this tutorial. You should be familiar with GNU and UNIX commands. Sometimes different versions of a program format output differently, so your results might not always look exactly like the listings shown here.

In this tutorial, I use Slackware 42.2 for syslogd examples, CentOS 7 for rsyslogd examples, and Fedora 26 for systemd-journalctl and syslog-ng examples.

Traditional syslog and the syslogd daemon

The traditional syslog system logging facility on a Linux system provides system logging and kernel message trapping. You can log data on your local system or send it to a remote system. Use the /etc/syslog.conf configuration file to finely control the level of logging. Logging is performed by the syslogd daemon, which normally receives input through the /dev/log socket, as shown in Listing 1.

Listing 1. /dev/log is a socket
[ian@attic4-sl42 ~]$ #Slackware 42.2 [ian@attic4-sl42 ~]$ /bin/ls -l /dev/log srw-rw-rw- 1 root root 0 Nov 19 16:35 /dev/log

For local logging, the main file is usually /var/log/messages, but many other files are used in most installations, usually located in the /var/log directory or a subdirectory of it. You can customize these extensively. For example, you may want a separate log for messages from the mail system.

The syslog.conf configuration file

The syslog.conf file is the main configuration file for the syslogd daemon. Entries in syslog.conf specify logging rules. Each rule has a selector field and an action field, which are separated by one or more spaces or tabs. The selector field identifies the facility and the priorities that the rule applies to, and the action field identifies the logging action for the facility and priorities.

The defined facilities are auth (or security), authpriv, cron, daemon, ftp, kern, lpr, mail, mark, news, syslog, user, uucp, and local0 through local7. The keyword auth should be used instead of security, and the keyword mark is for internal use.

The priorities (in ascending order of severity) are:

  1. debug
  2. info
  3. notice
  4. warning (or warn)
  5. err (or error)
  6. crit
  7. alert
  8. emerg (or panic)

The parenthesized keywords (warn, error, and panic) are now deprecated.

The default behavior is to take action for the specified level and for all higher levels, although it is possible to limit logging to specific levels. Each selector consists of a facility and a priority separated by a period (dot). Multiple facilities for a given action can be specified by separating them with a comma. Multiple facility/priority pairs for a given action can be specified by separating them with a semicolon. Listing 2 shows an example of a simple syslog.conf file.

Listing 2. Example syslog.conf
#Log anything 'warn' or higher. #Exclude authpriv, cron, mail, and news. These are logged elsewhere. #Don't log private authentication messages! *.warn;\ authpriv.none;cron.none;mail.none;news.none    -/var/log/syslog #The authpriv file has restricted access. authpriv.* /var/log/secure #Log all the mail messages in one place. mail.* -/var/log/maillog #Log cron stuff cron.* /var/log/cron #Everybody gets emergency messages *.emerg * #Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler

Notes:

  • As with many configuration files, lines starting with # and blank lines are ignored.
  • An * may be used to refer to all facilities or all priorities.
  • The special priority keyword none indicates that no logging for this facility should be done with this action.
  • The hyphen before a file name (such as -/var/log/maillog, in this example) indicates that the log file should not be synchronized after every write. You might lose information after a system crash, but you might gain performance by doing this.

The actions are generically referred to as “logfiles,” although they do not have to be real files. Table 1 describe the possible logfiles.

Table 1. Actions in syslog.conf
Action Purpose
Regular file Specify the full pathname, beginning with a slash (/). Prefix it with a hyphen (-) to omit syncing the file after each log entry. This may cause information loss if a crash occurs, but may improve performance.
Named pipes Use a first-in first-out (FIFO) or named pipe as a destination for log messages by including a pipe symbol (|) before the file name. You must create the FIFO using the mkfifo command before starting (or restarting) syslogd. FIFOs are sometimes used for debugging.
Terminal or console Send log messages to a terminal such as /dev/console.
Remote machine Forward messages to another host by putting an at (@) sign before the hostname. Note that messages are not forwarded from the receiving host.
List of users Use a comma-separated list of users to receive messages (if the user is logged in). The root user is frequently included here.
Everyone logged on Specify an asterisk (*) to have everyone logged on notified using the wall command.

Prefix a priority with ! to indicate that the action should not apply to this level and higher. Similarly prefix a priority with = to indicate that the rule applies only to this level or with != to indicate that the rule applies to all except this level. Listing 3 shows some examples, and the man page for syslog.conf has many more examples.

Listing 3. Additional syslog.conf examples

#Kernel messages are first, stored in the kernel
#file, critical messages and higher ones also go
#to another host and to the console
#
kern.*                       /var/adm/kernel
kern.crit                    @log‑server
kern.crit                    /dev/console
kern.info;kern.!err          /var/adm/kernel‑info

#Store all mail messages except info priority in /var/log/mail. 
mail.*;mail.!=info           /var/log/mail

The syslogd command starts the syslogd daemon. It has a numver of options, including -f to specify a different configuration file, and -a for additional sockets to listen to. The daemon responds to several signals, including SIGHUP which causes it to reinitialize. See the man or info pages for more details on running and interacting with the syslogd daemon.

Listing 4 shows some of the messages that might be logged to /var/log/messages using syslogd configuration parameters like those of Listing 2.

Listing 4. Example of messages logged in /var/log/messages


root@attic4‑sl42:~#tail ‑n 20 /var/log/messages     
Nov 19 21:39:57 attic4‑sl42 kernel: [ 1403.274747] usb 1‑1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Nov 19 21:39:57 attic4‑sl42 kernel: [ 1403.274751] usb 1‑1.1: Product: USB DISK 2.0
Nov 19 21:39:57 attic4‑sl42 kernel: [ 1403.274755] usb 1‑1.1: Manufacturer:         
Nov 19 21:39:57 attic4‑sl42 kernel: [ 1403.274759] usb 1‑1.1: SerialNumber: 070B53DF11FC0170
Nov 19 21:39:57 attic4‑sl42 kernel: [ 1403.275036] usb‑storage 1‑1.1:1.0: USB Mass Storage device detected
Nov 19 21:39:57 attic4‑sl42 kernel: [ 1403.275801] scsi host12: usb‑storage 1‑1.1:1.0
Nov 19 21:39:57 attic4‑sl42 mtp‑probe: checking bus 1, device 12: "/sys/devices/pci0000:00/0000:00:12.2/usb1/1‑1/1‑1.1" 
Nov 19 21:39:57 attic4‑sl42 mtp‑probe: bus: 1, device: 12 was not an MTP device 
Nov 19 21:39:58 attic4‑sl42 kernel: [ 1404.301570] scsi 12:0:0:0: Direct‑Access              USB DISK 2.0     PMAP PQ: 0 ANSI: 4
Nov 19 21:39:59 attic4‑sl42 kernel: [ 1405.651626] sd 12:0:0:0: [sdd] 30299520 512‑byte logical blocks: (15.5 GB/14.4 GiB)
Nov 19 21:39:59 attic4‑sl42 kernel: [ 1405.652236] sd 12:0:0:0: [sdd] Write Protect is off
Nov 19 21:40:00 attic4‑sl42 kernel: [ 1405.678810]  sdd: sdd1
Nov 19 21:40:00 attic4‑sl42 kernel: [ 1405.683911] sd 12:0:0:0: [sdd] Attached SCSI removable disk
Nov 19 21:42:36 attic4‑sl42 kernel: [ 1562.844117] usb 1‑1.1: reset high‑speed USB device number 12 using ehci‑pci
Nov 19 21:42:37 attic4‑sl42 kernel: [ 1563.044142] usb 1‑1.1: reset high‑speed USB device number 12 using ehci‑pci
Nov 19 21:42:37 attic4‑sl42 kernel: [ 1563.245138] usb 1‑1.1: reset high‑speed USB device number 12 using ehci‑pci
Nov 19 21:42:37 attic4‑sl42 kernel: [ 1563.619154] usb 1‑1.1: reset high‑speed USB device number 12 using ehci‑pci
Nov 19 21:42:38 attic4‑sl42 kernel: [ 1563.919798] usb 1‑1.1: USB disconnect, device number 12
Nov 19 21:42:38 attic4‑sl42 kernel: <27>[ 1563.950967] udevd[3540]: inotify_add_watch(6, /dev/sdd, 10) failed: No such file or directory
Nov 19 21:56:48 attic4‑sl42 ‑‑ MARK ‑‑

The kernel log daemon – klogd

In Listing 3 you saw some ways to configure kernel message logging. But how do boot-time kernel messages get logged before a file system is even mounted? The kernel stores messages in a ring buffer in memory. The klogd daemon processes these messages directly to a console, or a file such as /var/log/dmesg, or through the syslog facility.

Note that the example in Listing 2 logs all messages at warning level or above in /var/log/syslog. In particular, this includes kernel messages.

As with the syslogd command, the klogd command has a number of options. See the man or info pages for details. There is no configuration file for klogd.

You can also display messages from the kernel ring buffer using the dmesg command. The command also has options to interact with the ring buffer, for example to read and clear messages.

Rotate and archive log files

With the amount of logging that is possible, you need to be able to control the size of log files. This is done using the logrotate command, which is usually run as a cron job. (See our tutorial, Automate system administration tasks by scheduling jobs, for more information on cron jobs).

The general idea behind the logrotate command is that log files are periodically backed up and a new log is started. Several generations of log are kept, and when a log ages to the last generation, it may be archived. For example, it might be mailed to an archival user.

Use the /etc/logrotate.conf configuration file to specify how your log rotating and archiving should happen. You can specify different frequencies, such as daily, weekly, or monthly, for different log files, and you can control the number of generations to maintain and when or whether to mail copies to an archival user. Listing 5 shows a sample /etc/logrotate.conf file.

Listing 5. Sample /etc/logrotate.conf
#/etc/logrotate.conf ##logrotate is designed to ease administration of systems that generate large #numbers of log files. It allows automatic rotation, compression, removal, and #mailing of log files. Each log file may be handled daily, weekly, monthly, or #when it grows too large. ##logrotate is normally run daily from root's crontab. ##For more details, see "man logrotate". #rotate log files weekly: weekly #keep 4 weeks worth of backlogs: rotate 4 #create new (empty) log files after rotating old ones: create #uncomment if you want to use the date as a suffix of the rotated file #dateext #uncomment this if you want your log files compressed: #compress #some packages install log rotation information in this directory: include /etc/logrotate.d #Rotate /var/log/wtmp: /var/log/wtmp { missingok monthly create 0664 root utmp minsize 1M rotate 1 } #Rotate /var/log/btmp: /var/log/btmp { missingok monthly create 0600 root root rotate 1 } #Note that /var/log/lastlog is not rotated. This is intentional, and it should #not be. The lastlog file is a database, and is also a sparse file that takes #up much less space on the drive than it appears. #system-specific logs may be also be configured below:

The logrotate.conf file has global options at the beginning. These are the default options if nothing more specific is specified elsewhere. In this example, log files are rotated weekly, and four weeks worth of backups are kept. Once a log file is rotated, a new one is automatically created in place of the old one. Note that the logrotate.conf file may include specifications from other files. Here, all the files in /etc/logrotate.d are included.

This example also includes specific rules for /var/log/wtmp and /var/log/btmp, which are rotated monthly. No error message is issued if the files are missing. A new file is created, and only one backup is maintained. Access to these files is also restricted by setting permissions.

Note: The files /var/log/wtmp and /var/log/btmp record successful and unsuccessful login attempts, respectively. Unlike most log files, these are not clear text files. You may examine them using the last or lastb commands. See the man pages for these commands for details.

In this example, when a backup reaches the last generation, it is deleted because there is no specification of what else to do with it.

You can back up log files when they reach a specific size. You can also script commands to run before or after the backup operation. Listing 6 shows a more complex example.

Listing 6. Another logrotate configuration example
/var/log/messages { rotate 5 mail logsave@log-server size 100k postrotate /usr/bin/killall -HUP syslogd endscript }

In Listing 6, /var/log/messages is rotated after it reaches 100 KB in size. Five backup files are maintained, and when the oldest backup ages out, it is mailed to logsave@log-server. The postrotate introduces a script that restarts the syslogd daemon after the rotation is complete, by sending it the HUP signal. The endscript statement is required to terminate the script and is also required if a prerotate script is present. See the logrotate man page for more complete information.

As with many such configuration files, some programs provided additional configuration, usually in the directory etc/logrotate.d. Listing 7 shows the files on my Slackware system.

Listing 7. Configuration files for logrotate

root@attic4‑sl42:~#ls /etc/logrot
/etc/logrotate.conf

/etc/logrotate.d:
consolekit  mcelog  mysql.orig  ulogd   wpa_supplicant
httpd       mysql   syslog      vsftpd
root@attic4‑sl42:~#cat /etc/logrotate.d/httpd
/var/log/httpd/_log {
  rotate 10
  notifempty
  missingok
  size=5M
  compress
  delaycompress
  sharedscripts
  postrotate
    /etc/rc.d/rc.httpd restart
  endscript
}

Scan log files for notable activity

Log file entries are usually time stamped and contain the hostname of the reporting process, along with the process name. Listing 8 shows a few lines from /var/log/messages, containing entries from several programs.

Listing 8. Sample log file entries
Nov 19 15:48:31 attic4-sl42 kernel: [ 7.407406] EXT4-fs (sda6): re-mounted. Opts: (null) Nov 19 15:48:32 attic4-sl42 mtp-probe: checking bus 3, device 3: "/sys/devices/pci0000:00/0000:00:12.0/usb3/3-2/3-2.1" Nov 19 15:48:32 attic4-sl42 mtp-probe: bus: 3, device: 3 was not an MTP device Nov 19 15:48:32 attic4-sl42 mtp-probe: checking bus 3, device 4: "/sys/devices/pci0000:00/0000:00:12.0/usb3/3-2/3-2.4" Nov 19 15:48:32 attic4-sl42 mtp-probe: bus: 3, device: 4 was not an MTP device Nov 19 15:48:32 attic4-sl42 mtp-probe: checking bus 1, device 4: "/sys/devices/pci0000:00/0000:00:12.2/usb1/1-1/1-1.1" Nov 19 15:48:32 attic4-sl42 mtp-probe: bus: 1, device: 4 was not an MTP device Nov 19 15:48:36 attic4-sl42 root: /etc/rc.d/rc.inet1: /sbin/ifconfig lo 127.0.0.1 Nov 19 15:48:36 attic4-sl42 root: /etc/rc.d/rc.inet1: /sbin/route add -net 127.0.0.0 netmask 255.0.0.0 lo Nov 19 15:48:36 attic4-sl42 root: /etc/rc.d/rc.inet1: /sbin/dhcpcd -t 10 eth0 Nov 19 15:48:36 attic4-sl42 dhcpcd[1112]: eth0: adding address fe80::4c2a:3f48:e0f7:cc90 Nov 19 15:48:41 attic4-sl42 sshd[1255]: Server listening on 0.0.0.0 port 22. Nov 19 15:48:41 attic4-sl42 sshd[1255]: Server listening on :: port 22. Nov 19 15:48:41 attic4-sl42 ntpd[1262]: ntpd 4.2.8p8@1.3265-o Fri Jun 3 23:08: 22 UTC 2016 (1): Starting Nov 19 15:48:41 attic4-sl42 ntpd[1262]: Command line: /usr/sbin/ntpd -g -p /var /run/ntpd.pid Nov 19 15:48:41 attic4-sl42 ntpd[1264]: proto: precision = 0.230 usec (-22) Nov 19 15:48:41 attic4-sl42 ntpd[1264]: Listen and drop on 0 v6wildcard [::]:12 3 Nov 19 15:48:41 attic4-sl42 ntpd[1264]: Listen and drop on 1 v4wildcard 0.0.0.0:123 Nov 19 15:48:41 attic4-sl42 ntpd[1264]: Listen normally on 2 lo 127.0.0.1:123 Nov 19 15:48:41 attic4-sl42 ntpd[1264]: Listen normally on 3 eth0 192.168.1.24:123 Nov 19 15:48:41 attic4-sl42 ntpd[1264]: Listen normally on 4 lo [::1]:123 Nov 19 15:48:41 attic4-sl42 ntpd[1264]: failed to init interface for address fe80::8616:f9ff:fe04:7a2a%2 Nov 19 15:48:41 attic4-sl42 ntpd[1264]: Listening on routing socket on fd #21 for interface updates Nov 19 15:48:41 attic4-sl42 acpid: starting up with netlink and the input layer Nov 19 15:48:41 attic4-sl42 acpid: 1 rule loaded Nov 19 15:48:41 attic4-sl42 acpid: waiting for events: event logging is off Nov 19 15:48:42 attic4-sl42 dbus[1226]: [system] Activating service name='org.freedesktop.PolicyKit1' (using servicehelper) Nov 19 15:48:42 attic4-sl42 ntpd[1264]: failed to init interface for address fe80::8616:f9ff:fe04:7a2a%2

The last line of Listing 8 shows a failure from the Network Time Protocol daemon (ntpd). In this case, it failed to initialize an IP V6 interface because this system uses only IP V4 connections.

You can scan log files using a pager, such as less, or search for specific entries (such as ntpd messages from host attic4-sl42) using grep as shown in Listing 9.

Listing 9. Scanning log files
root@attic4-sl42:~#grep "attic4-sl42 ntpd" /var/log/messages | tail -9 Nov 19 21:17:12 attic4-sl42 ntpd[1131]: Command line: /usr/sbin/ntpd -g -p /var/run/ntpd.pid Nov 19 21:17:12 attic4-sl42 ntpd[1133]: proto: precision = 0.220 usec (-22) Nov 19 21:17:13 attic4-sl42 ntpd[1133]: Listen and drop on 0 v6wildcard [::]:123 Nov 19 21:17:13 attic4-sl42 ntpd[1133]: Listen and drop on 1 v4wildcard 0.0.0.0:123 Nov 19 21:17:13 attic4-sl42 ntpd[1133]: Listen normally on 2 lo 127.0.0.1:123 Nov 19 21:17:13 attic4-sl42 ntpd[1133]: Listen normally on 3 eth0 192.168.1.24:123 Nov 19 21:17:13 attic4-sl42 ntpd[1133]: Listen normally on 4 lo [::1]:123 Nov 19 21:17:13 attic4-sl42 ntpd[1133]: Listen normally on 5 eth0 [fe80::8616:f9ff:fe04:7a2a%2]:123 Nov 19 21:17:13 attic4-sl42 ntpd[1133]: Listening on routing socket on fd #22 for interface updates

Monitor log files

Occasionally you may need to monitor log files for events. For example, you might be trying to catch an infrequently occurring event at the time it happens. In such a case, you can use the tail command with the -f option to follow the log file. Listing 10 shows an example.

Listing 10. Following log file updates
root@attic4-sl42:~#tail -n 1 -f /var/log/messages Nov 20 09:24:58 attic4-sl42 kernel: [43705.563240] sd 15:0:0:0: [sdd] Attached SCSI removable disk Nov 20 09:26:23 attic4-sl42 kernel: [43790.820125] usb 3-2.4: USB disconnect, device number 5 Nov 20 09:27:13 attic4-sl42 sshd[6059]: Accepted password for ian from 192.168.1.40 port 58184 ssh2 Nov 20 09:29:08 attic4-sl42 kernel: [43955.890670] usb 3-2.4: new low-speed USB device number 6 using ohci-pci Nov 20 09:29:08 attic4-sl42 kernel: [43955.989492] usb 3-2.4: New USB device found, idVendor=046d, idProduct=c50e Nov 20 09:29:08 attic4-sl42 kernel: [43955.989501] usb 3-2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0 Nov 20 09:29:08 attic4-sl42 kernel: [43955.989506] usb 3-2.4: Product: USB Receiver Nov 20 09:29:08 attic4-sl42 kernel: [43955.989509] usb 3-2.4: Manufacturer: Logitech Nov 20 09:29:08 attic4-sl42 kernel: [43956.003175] input: Logitech USB Receiver as /devices/pci0000:00/0000:00:12.0/usb3/3-2/3-2.4/3-2.4:1.0/0003:046D:C50E.0004/input/input19 Nov 20 09:29:08 attic4-sl42 kernel: [43956.054049] hid-generic 0003:046D:C50E.0004: input,hidraw1: USB HID v1.11 Mouse [Logitech USB Receiver] on usb-0000:00:12.0-2.4/input0 Nov 20 09:29:08 attic4-sl42 mtp-probe: checking bus 3, device 6: "/sys/devices/pci0000:00/0000:00:12.0/usb3/3-2/3-2.4" Nov 20 09:29:08 attic4-sl42 mtp-probe: bus: 3, device: 6 was not an MTP device

Track down problems reported in log files

When you find problems in log files, note the time, the host name, and the process that generated the problem. If the message identifies the problem specifically enough for you to resolve it, you are done. If not, you might need to update syslog.conf to specify that more messages be logged for the appropriate facility. For example, you might need to show informational messages instead of warning messages or even debug-level messages. Your application may have additional facilities that you can use.

Finally, if you need to put marks in the log file to help you know what messages were logged at what stage of your debugging activity, you can use the logger command from a terminal window or shell script to send a message of your choice to the syslogd daemon for logging according to the rules in syslog.conf.

Using rsyslogd

Rsyslog is the self-described rocket-fast system for log processing. It is upwards compatible from syslog in the sense that it can process configurations compatible with syslog and also handle the syslog call to log information. It also provides several enhancements that are not backwards compatible. In particular, it supports additional logging protocols, and it can log to databases, such as MySQL or PostgreSQL, as well as files. You can filter any part of a syslog message and can fully configure the output format.

The traditional man and info pages provide basic information about rsyslog. However, there is considerably more documentation provided in the HTML format in the doc directory of your system. You may need to install the rsyslog-doc package if it was not installed in your system. The root of the HTML tree is at /usr/share/doc/rsyslog-8.24.0/html/index.html. The location may differ on your system.

The rsyslog program runs as a daemon, similar to syslogd. The configuration file defaults to rsyslog.conf.

Listing 11. An rsyslog.conf configuration file

[ian@attic4‑ce7 ~]$ cat /etc/rsyslog.conf 
#rsyslog configuration file

#For more information see /usr/share/doc/rsyslog‑*/rsyslog_conf.html
#If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

####MODULES ####

#The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock #provides support for local system logging (e.g. via logger command)
$ModLoad imjournal #provides access to the systemd journal
#$ModLoad imklog #reads kernel messages (the same are read from journald)
#$ModLoad immark  #provides ‑‑MARK‑‑ message capability

#Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

#Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514


####GLOBAL DIRECTIVES ####

#Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

#Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

#File syncing capability is disabled by default. This feature is usually not required,
#not useful and an extreme performance hit
#$ActionFileEnableSync on

#Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

#Turn off message reception via local log socket;
#local messages are retrieved through imjournal now.
$OmitLocalLogging on

#File to store the position in the journal
$IMJournalStateFile imjournal.state


####RULES ####

#Log all kernel messages to the console.
#Logging much else clutters up the screen.
#kern.*                                                 /dev/console

#Log anything (except mail) of level info or higher.
#Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

#The authpriv file has restricted access.
authpriv.*                                             /var/log/secure

#Log all the mail messages in one place.
mail.*                                                  ‑/var/log/maillog


#Log cron stuff
cron.*                                                  /var/log/cron

#Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*

#Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

#Save boot messages also to boot.log
local7.*                                                /var/log/boot.log


####begin forwarding rule ###
#The statement between the begin ... end define a SINGLE forwarding
#rule. They belong together, do NOT split them. If you create multiple
#forwarding rules, duplicate the whole block!
#Remote Logging (we use TCP for reliable delivery)
#
#An on‑disk queue is created for this action. If the remote host is
#down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 #unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   #1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on #save messages to disk on shutdown
#$ActionQueueType LinkedList   #run asynchronously
#$ActionResumeRetryCount ‑1    #infinite retries if host is down
#remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.*  remote‑host:514
####end of the forwarding rule ###

You will recognize familiar syslog.conf entries in the middle of the file, surrounded by additional things that the rsyslog facility understands. See the man or info pages or the HTML documentation for more details.

You can use logrotate with files created by rsyslogd, although SQL databases may require scripting or other tools. Similarly, the logger command still works to place your own mark in the log.

Using the systemd journal service

The systemd journal service examples in this section come from Fedora 26.

The systemd-journald program is a system service daemon that collects and stores logging data. It creates and maintains structured, indexed journals based on logging information that is received from the usual syslog sources as well as structured log messages using a native journal API.

The usual /dev/log socket is a link to /run/systemd/journal/dev-log as shown in Listing 12.

Listing 12. The dev/log and /run/systemd/journal/dev-log sockets

[root@atticf26 ~]#ls ‑l /dev/log
lrwxrwxrwx. 1 root root 28 Nov 20 23:43 /dev/log ‑> /run/systemd/journal/dev‑log
[root@atticf26 ~]#ls ‑l /run/systemd/journal/dev‑log
srw‑rw‑rw‑. 1 root root 0 Nov 20 23:43 /run/systemd/journal/dev‑log

The systemd-journald daemon listens on sockets and other file system entities, including /dev/kmsg, /dev/log, /run/systemd/journal/dev-log, /run/systemd/journal/socket, and /run/systemd/journal/stdout. It can also listen for audit events using netlink, which transfers kernel information to user space using sockets.

Most log data is textual, but binary data is allowed, theoretically up to 2^64-1 bytes in size. The journal stores log data in /run/log/journal/ by default. The /run/ file system is volatile, so log data is lost when the system reboots. To make the data persistent, you create /var/log/journal/ and systemd-journald will then store the data there.

As with other logging systems, there is a configuration file. By default, it is /etc/systemd/journald.conf. Many options are compiled in by default, so most of the options in the configuration file are initially commented out as shown in Listing 13. Uncomment those you wish to change. As usual additional configuration files can be located in the journald.conf.d directory. Packages should install their journal configuration information in /usr/lib/systemd/*.conf.d/.

Listing 13. Initial /etc/systemd/journald.conf example

# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
#Entries in this file show the compile time defaults.
#You can change settings by editing this file.
#Defaults can be restored by simply deleting this file.
#
#See journald.conf(5) for details.

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=no
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg

Use the man or info pages for journald.conf to learn more about the configuration settings that are supported.

Use the journalctl command to display logged information. Listing 14 shows how to display the last 10 lines of logged data and then use the -f option to follow, or continuously display, new lines as they are added. Use the --rotate option to rotate journal files.

Listing 14. Using journalctl to display or follow log messages

[root@atticf26 ~]#journalctl ‑n 10 ‑f
‑‑ Logs begin at Mon 2007‑07‑09 22:14:00 EDT. ‑‑
Nov 21 10:24:47 atticf26 dbus‑daemon[585]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus‑org.freedesktop.nm‑dispatcher.service' requested by ':1.9' (uid=0 pid=650 comm="/usr/sbin/NetworkManager ‑‑no‑daemon " label="system_u:system_r:NetworkManager_t:s0")
Nov 21 10:24:47 atticf26 systemd[1]: Starting Network Manager Script Dispatcher Service...
Nov 21 10:24:47 atticf26 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager‑dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 21 10:24:47 atticf26 dbus‑daemon[585]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 21 10:24:47 atticf26 systemd[1]: Started Network Manager Script Dispatcher Service.
Nov 21 10:24:47 atticf26 nm‑dispatcher[7121]: req:1 'connectivity‑change': new request (5 scripts)
Nov 21 10:24:47 atticf26 nm‑dispatcher[7121]: req:1 'connectivity‑change': start running ordered scripts...
Nov 21 10:24:48 atticf26 gnome‑software‑service.desktop[4641]: 15:24:48:0034 Gs  failed to call gs_plugin_app_install on packagekit: do not know how to install app in state queued
Nov 21 10:24:52 atticf26 dhclient[7110]: DHCPDISCOVER on enp4s0 to 255.255.255.255 port 67 interval 15 (xid=0x36565033)
Nov 21 10:24:57 atticf26 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager‑dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 21 10:25:07 atticf26 dhclient[7110]: DHCPDISCOVER on enp4s0 to 255.255.255.255 port 67 interval 14 (xid=0x36565033)
Nov 21 10:25:21 atticf26 dhclient[7110]: DHCPDISCOVER on enp4s0 to 255.255.255.255 port 67 interval 17 (xid=0x36565033)
Nov 21 10:25:22 atticf26 NetworkManager[650]: <warn>  [1511277922.0280] dhcp4 (enp4s0): request timed out
Nov 21 10:25:22 atticf26 NetworkManager[650]: <info>  [1511277922.0286] dhcp4 (enp4s0): state changed unknown ‑> timeout
Nov 21 10:25:22 atticf26 NetworkManager[650]: <info>  [1511277922.0380] dhcp4 (enp4s0): canceled DHCP transaction, DHCP client pid 7110
Nov 21 10:25:22 atticf26 NetworkManager[650]: <info>  [1511277922.0381] dhcp4 (enp4s0): state changed timeout ‑> done
Nov 21 10:25:22 atticf26 NetworkManager[650]: <info>  [1511277922.0386] device (enp4s0): state change: ip‑config ‑> failed (reason 'ip‑config‑unavailable', internal state 'managed')
Nov 21 10:25:22 atticf26 NetworkManager[650]: <warn>  [1511277922.0393] device (enp4s0): Activation: failed for connection 'enp3s0'
Nov 21 10:25:22 atticf26 NetworkManager[650]: <info>  [1511277922.0401] device (enp4s0): state change: failed ‑> disconnected (reason 'none', internal state 'managed')

...

Nov 21 10:26:37 atticf26 NetworkManager[650]: <info>  [1511277997.7805] device (enp4s0): state change: ip‑config ‑> deactivating (reason 'user‑requested', internal state 'managed')
Nov 21 10:26:37 atticf26 NetworkManager[650]: <info>  [1511277997.7811] audit: op="device‑disconnect" interface="enp4s0" ifindex=3 pid=7167 uid=1000 result="success"
Nov 21 10:26:37 atticf26 NetworkManager[650]: <info>  [1511277997.7817] device (enp4s0): state change: deactivating ‑> disconnected (reason 'user‑requested', internal state 'managed')
Nov 21 10:26:37 atticf26 gnome‑software‑service.desktop[4641]: 15:26:37:0783 Gs  failed to call gs_plugin_app_install on packagekit: do not know how to install app in state queued
Nov 21 10:26:37 atticf26 avahi‑daemon[602]: Withdrawing address record for fe80::3fd7:76aa:e99d:da5d on enp4s0.
Nov 21 10:26:37 atticf26 avahi‑daemon[602]: Leaving mDNS multicast group on interface enp4s0.IPv6 with address fe80::3fd7:76aa:e99d:da5d.
Nov 21 10:26:37 atticf26 avahi‑daemon[602]: Interface enp4s0.IPv6 no longer relevant for mDNS.
Nov 21 10:26:37 atticf26 gnome‑software‑service.desktop[4641]: 15:26:37:0787 Gs  failed to call gs_plugin_app_install on packagekit: do not know how to install app in state queued
Nov 21 10:26:37 atticf26 NetworkManager[650]: <info>  [1511277997.7993] dhcp4 (enp4s0): canceled DHCP transaction, DHCP client pid 7179
Nov 21 10:26:37 atticf26 NetworkManager[650]: <info>  [1511277997.7993] dhcp4 (enp4s0): state changed unknown ‑> done
Nov 21 10:26:37 atticf26 audit: NETFILTER_CFG table=filter family=2 entries=99
Nov 21 10:26:37 atticf26 audit: NETFILTER_CFG table=nat family=2 entries=59
Nov 21 10:26:37 atticf26 audit: NETFILTER_CFG table=mangle family=2 entries=42
Nov 21 10:26:37 atticf26 audit: NETFILTER_CFG table=raw family=2 entries=30
Nov 21 10:26:37 atticf26 dbus‑daemon[585]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus‑org.freedesktop.nm‑dispatcher.service' requested by ':1.9' (uid=0 pid=650 comm="/usr/sbin/NetworkManager ‑‑no‑daemon " label="system_u:system_r:NetworkManager_t:s0")
Nov 21 10:26:37 atticf26 systemd[1]: Starting Network Manager Script Dispatcher Service...
Nov 21 10:26:37 atticf26 audit: NETFILTER_CFG table=filter family=10 entries=90
Nov 21 10:26:37 atticf26 audit: NETFILTER_CFG table=nat family=10 entries=54
Nov 21 10:26:37 atticf26 audit: NETFILTER_CFG table=mangle family=10 entries=41
Nov 21 10:26:37 atticf26 audit: NETFILTER_CFG table=raw family=10 entries=31
Nov 21 10:26:37 atticf26 dbus‑daemon[585]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 21 10:26:37 atticf26 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager‑dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 21 10:26:37 atticf26 nm‑dispatcher[7223]: req:1 'down' [enp4s0]: new request (5 scripts)
Nov 21 10:26:37 atticf26 systemd[1]: Started Network Manager Script Dispatcher Service.
Nov 21 10:26:37 atticf26 nm‑dispatcher[7223]: req:1 'down' [enp4s0]: start running ordered scripts...
Nov 21 10:26:38 atticf26 gnome‑software‑service.desktop[4641]: 15:26:38:0787 Gs  failed to call gs_plugin_app_install on packagekit: do not know how to install app in state queued
Nov 21 10:26:48 atticf26 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager‑dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 21 10:28:35 atticf26 cupsd[4544]: REQUEST localhost ‑ ‑ "POST / HTTP/1.1" 200 182 Renew‑Subscription successful‑ok

Use the systemctl command to display information about, interact with, or control the daemon and related units. Listing 15 shows an example.

Listing 15. Using the systemctl command

[root@atticf26 ~]#systemctl  list‑units "journal" ‑‑no‑pager
UNIT                            LOAD   ACTIVE SUB     DESCRIPTION              
abrt‑journal‑core.service       loaded active running Creates ABRT problems fro
systemd‑journal‑flush.service   loaded active exited  Flush Journal to Persiste
systemd‑journald.service        loaded active running Journal Service          
systemd‑journald‑audit.socket   loaded active running Journal Audit Socket     
systemd‑journald‑dev‑log.socket loaded active running Journal Socket (/dev/log)
systemd‑journald.socket         loaded active running Journal Socket           

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high‑level unit activation state, i.e. generalization of SUB.
SUB    = The low‑level unit activation state, values depend on unit type.

6 loaded units listed. Pass ‑‑all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list‑unit‑files'.

Using syslog-ng

Syslog-ng describes itself as “an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL.” It supports legacy and enhanced syslog protocols and adds support for JavaScript Object Notation (JSON) and journald message formats. Syslog-ng supports extensive capabilities to filter input and format output.

Once installed, there is basic information in the man and info pages. However, you will probably want to use “The syslog-ng Open Source Edition Administrator Guide”, which is available in both HTML and PDF formats. (See resources on the right for more information).

The default configuration file is /etc/syslog-ng/syslog-ng.conf. An example is shown in Listing 16. Additional configuration files can be located in the /etc/syslog-ng/conf.d directory.

Listing 16. Example of syslog-ng.conf

@version:3.9
@include "scl.conf"

#syslog‑ng configuration file.
#
#This should behave pretty much like the original syslog on RedHat. But
#it could be configured a lot smarter.
#
#See syslog‑ng(8) and syslog‑ng.conf(5) for more information.
#
#Note: it also sources additional configuration files (*.conf)
#      located in /etc/syslog‑ng/conf.d/

options {
    flush_lines (0);
    time_reopen (10);
    log_fifo_size (1000);
    chain_hostnames (off);
    use_dns (no);
    use_fqdn (no);
    create_dirs (no);
    keep_hostname (yes);
};

source s_sys {
    system();
    internal();
    #udp(ip(0.0.0.0) port(514));
};

destination d_cons { file("/dev/console"); };
destination d_mesg { file("/var/log/messages"); };
destination d_auth { file("/var/log/secure"); };
destination d_mail { file("/var/log/maillog" flush_lines(10)); };
destination d_spol { file("/var/log/spooler"); };
destination d_boot { file("/var/log/boot.log"); };
destination d_cron { file("/var/log/cron"); };
destination d_kern { file("/var/log/kern"); };
destination d_mlal { usertty(""); };

filter f_kernel     { facility(kern); };
filter f_default    { level(info..emerg) and
                        not (facility(mail)
                        or facility(authpriv) 
                        or facility(cron)); };
filter f_auth       { facility(authpriv); };
filter f_mail       { facility(mail); };
filter f_emergency  { level(emerg); };
filter f_news       { facility(uucp) or
                        (facility(news) 
                        and level(crit..emerg)); };
filter f_boot   { facility(local7); };
filter f_cron   { facility(cron); };

#log { source(s_sys); filter(f_kernel); destination(d_cons); };
log { source(s_sys); filter(f_kernel); destination(d_kern); };
log { source(s_sys); filter(f_default); destination(d_mesg); };
log { source(s_sys); filter(f_auth); destination(d_auth); };
log { source(s_sys); filter(f_mail); destination(d_mail); };
log { source(s_sys); filter(f_emergency); destination(d_mlal); };
log { source(s_sys); filter(f_news); destination(d_spol); };
log { source(s_sys); filter(f_boot); destination(d_boot); };
log { source(s_sys); filter(f_cron); destination(d_cron); };


#Source additional configuration files (.conf extension only)
@include "/etc/syslog‑ng/conf.d/.conf"


#vim:ft=syslog‑ng:ai:si:ts=4:sw=4:et:

You generally don’t need to run both the systemd journal and syslog-ng journalling. If you do wish to use both, you will need to make some configuration changes to both and then restart both. Search the web for current instructions on how to do this.

This concludes your introduction to logging facilities on Linux.