Log files provide summary information that is critical to seeing how a system or a specific server is behaving. On UNIX and UNIX-like systems, log files are simply text files that are continually appended, either by the server processes that they are associated with or by system-logging daemons. All UNIX and UNIX-like systems use a general-purpose system-logging daemon, usually referred to as
syslogd or simply
syslog, which handles log output from many different applications and servers. Linux® systems also run
klogd, a specialized daemon that specifically monitors kernel messages.
Classic log monitoring with the tail command
The classic command used to monitor new output in system or server logs (referred to simply as logs in the rest of this article) is the
tail command, which shows the last few lines in a specified file (the last 10 lines by default). When executed as
tail filename, the
tail command exits after displaying the last lines from the specified file, which isn’t all that helpful when you want to monitor updates to a log file. To actively monitor a log file, the
tail command is most often invoked with the
-f option, which displays the last lines of the file and then continues to display subsequent lines as they are added to the specified file.
It’s often necessary (or at least useful) to monitor multiple log files simultaneously. Specific server processes that you may want to monitor usually write to their own log files, while
syslog sends different types of messages to different log files (depending upon how it is configured in /etc/syslog.conf). As an example, on server systems running Ubuntu, it is often useful to monitor both /var/log/auth.log (where authentication requests and changes are logged) and /var/log/kern.log (where kernel messages are logged). Similarly, on server systems running the Apache Web server, it is useful to monitor both the standard Apache log files: /var/log/apache2/access.log and /var/log/apache2/error.log.
Many systems administrators, including this author in days gone by, simply start multiple xterms (or other graphical console/shell applications), each one of which runs the
tail -f command to monitor a specific log file. That provides the necessary information but sacrifices a tremendous amount of screen real estate regardless of the size of the font that you use. As discussed in the rest of this article, the free and open source community provides a few advanced alternatives to this somewhat primitive file monitoring solution.
Monitoring multiple log files simultaneously
MultiTail is a utility that provides a superset of the basic functionality of the
tail command, enabling you to monitor multiple files at the same time within the same terminal application. MultiTail (executed as
multitail) is a GNU Public License (GPL) Version 2 application that is available in the repositories of most UNIX-like operating systems and can easily be built from source code. The only significant requirement for building
multitail is the ncurses library, which enables pseudo-graphical operations to be done within terminal windows.
The simplest way of executing MultiTail is to specify the names of the files that you want to monitor on the command line. For example, the
multitail access.log error.log command enables you to monitor both the primary log files for an Apache Web server, as shown in Figure 1.
Figure 1. Viewing multiple Apache logs in MultiTail
MultiTail has a huge number of options, all of which are documented in its traditional UNIX man page. When running MultiTail in a terminal session, you can request help, actively modify the display, search for specific text, and so on by using a variety of keyboard commands. You can access MultiTail’s built-in help at any time by clicking Control-h, which displays an overlaid help window, as shown in Figure 2.
Figure 2. Viewing help in MultiTail
You can scroll through this window using the cursor keys and close it by typing q or Control-g. (MultiTail’s help is also bound to the F1 key, but graphical desktops such as GNOME usually intercept that key sequence and display their own online help instead.)
Popular options for the multitail command
Some of the more commonly used MultiTail command-line options are described in Table 1.
Table 1. Popular multitail command options
||Enables you to specify a period of inactivity after which a MultiTail window will close. This can be useful if you are monitoring a large number of files at the same time but only care about those that are actively being updated. Though this option can be convenient, you should use it with care because after a window closes it will not reopen if new output is written to it.|
||Enables you to specify the names of the files that you want to monitor. Ordinarily, you simply specify the names of the files that you want to monitor by listing their names on the command line. Preceding a file name with the
||Enables you to specify a period of time (N seconds) of inactivity that will cause a mark line to be displayed in any MultiTail output window. For example, if you are monitoring the /var/log/auth.log file and no authentication-related messages have been logged to that file in N seconds, MultiTail displays a highlighted line such as
||Causes MultiTail to suppress display of repeated messages in the same window, displaying a summary message (“Last message repeated N times”) instead.|
Figure 3 shows the output from the following command on a sample machine after 60 seconds:
multitail --mark-interval 10 --no-repeat auth.log \ daemon.log kern.log lpr.log mail.err mysql.err \ wpa_supplicant.log messages
Figure 3. More advanced log file display in MultiTail
As you can see from Figure 3, MultiTail divides the available height of your terminal window between all the files that you want to monitor. In most cases, the last one or two lines of log file output is sufficient to identify any truly important messages that are being displayed in one of your log files.
The options used in the previous example are only the tip of the MultiTail iceberg. As you can see from the figures, MultiTail supports various types of highlighting, including pattern-based highlighting that makes it easy to spot messages associated with a specific device or that contain a specific keyword.
multitail command is not limited to simply monitoring existing files. It also provides options that let you monitor the output of a command (
-l command) that produces continuous output. For quick-running commands, the
-r N option enables you to repeat a given command after N seconds.
Monitoring command output
Sometimes it’s more important to know that a given log file is being written, and when, than it is to know what’s being written to it. Apache’s access log is a good example — heavily used Web servers can generate thousands of log messages in a few minutes, each of which provides detailed information that isn’t really useful. In cases such as these, what most systems administrators actually care about is that the log is constantly growing — in other words, that Apache is running and being accessed.
You can certainly do this visually by repeatedly running a command such as
ls -l and comparing file sizes. As mentioned at the end of the previous section, you can also use the
multitail command to monitor command output and repeat a specified command multiple times. Unfortunately, this requires a few command-line options:
multitail -Rc 2 -l "ls -l access.log error.log"
watch command, which is installed by default on most Linux systems, provides a simpler solution. The
watch command automatically runs a specified command at specified intervals, with a default interval of two seconds, and does exactly the right thing when combined with a command such as
ls -l. Figure 4 shows the output from the
watch ls -l access.log error.log command.
Figure 4. Monitoring log file status using watch
multitail command, the
watch command runs until interrupted by a keyboard command such as Control-C.
watch command allows you to watch the output of a program change over time, which can be especially convenient when looking for changes to file sizes that reflect debugging output, download or file creation state, memory or disk usage, and so on. For example, running a command such as
watch cat /proc/meminfo provides a simple way of viewing memory use on your system in real-time. When monitoring a command in which multiple lines differ each time that it is executed, you can specify the
-d option to highlight the differences between successive runs of the command. You can also use the
--differences=cumulative option to accumulate highlighted differences between all runs of the specified command.
Different watch commands on different systems
watch command is part of the procps package, a GPL v2 package that also provides commands such as
vmstat, and so on. Be careful if you are using a UNIX or UNIX-like system other than Linux, because other operating systems provide commands called
watch that are completely different. For example, on Berkeley Software Distribution (BSD)-inspired systems,
watch is usually called
gnuwatch, because these systems have a different
watch command that monitors the processes running on a specified physical or virtual terminal. Similarly, IBM® AIX® operating systems provide a
watch command that monitors the processes that are created by a specified application.
If your system doesn’t provide the standard
watch command and you don’t have the time or inclination to build the procps package for your system, you can easily simulate it with a simple shell script. The shell script shown in Listing 1, simplewatch.sh, provides much of the same functionality as the
Listing 1. A shell equivalent for the watch command
#!/bin/sh while [ 1 ] ; do clear echo "Command: $*" date echo "" ( $* ) sleep 2 done
Figure 5 shows the output from running the
simplewatch.sh ls -lt \*.log command in the /var/log directory on a sample OpenSolaris system. Quoting any wildcards that you pass to this script isn’t necessary but helps simplify the output that it displays.
Figure 5. Output from running the simplewatch.sh ls -lt *.log command
Monitoring log files, file creation and size, and differences between subsequent runs of a specific command are common systems administration tasks. The
watch commands discussed in this article help automate these potentially boring tasks, making it easy to keep an eye on various aspects of your systems while also getting other work done.