Monitoring logs and command output

Log files provide summary information that is critical to seeing how a system or a specific server is behaving. On UNIX and UNIX-like systems, log files are simply text files that are continually appended, either by the server processes that they are associated with or by system-logging daemons. All UNIX and UNIX-like systems use a general-purpose system-logging daemon, usually referred to as syslogd or simply syslog, which handles log output from many different applications and servers. Linux® systems also run klogd, a specialized daemon that specifically monitors kernel messages.

Classic log monitoring with the tail command

The classic command used to monitor new output in system or server logs (referred to simply as logs in the rest of this article) is the tail command, which shows the last few lines in a specified file (the last 10 lines by default). When executed as tail filename, the tail command exits after displaying the last lines from the specified file, which isn’t all that helpful when you want to monitor updates to a log file. To actively monitor a log file, the tail command is most often invoked with the -f option, which displays the last lines of the file and then continues to display subsequent lines as they are added to the specified file.

It’s often necessary (or at least useful) to monitor multiple log files simultaneously. Specific server processes that you may want to monitor usually write to their own log files, while syslog sends different types of messages to different log files (depending upon how it is configured in /etc/syslog.conf). As an example, on server systems running Ubuntu, it is often useful to monitor both /var/log/auth.log (where authentication requests and changes are logged) and /var/log/kern.log (where kernel messages are logged). Similarly, on server systems running the Apache Web server, it is useful to monitor both the standard Apache log files: /var/log/apache2/access.log and /var/log/apache2/error.log.

Many systems administrators, including this author in days gone by, simply start multiple xterms (or other graphical console/shell applications), each one of which runs the tail -f command to monitor a specific log file. That provides the necessary information but sacrifices a tremendous amount of screen real estate regardless of the size of the font that you use. As discussed in the rest of this article, the free and open source community provides a few advanced alternatives to this somewhat primitive file monitoring solution.

Monitoring multiple log files simultaneously

MultiTail is a utility that provides a superset of the basic functionality of the tail command, enabling you to monitor multiple files at the same time within the same terminal application. MultiTail (executed as multitail) is a GNU Public License (GPL) Version 2 application that is available in the repositories of most UNIX-like operating systems and can easily be built from source code. The only significant requirement for building multitail is the ncurses library, which enables pseudo-graphical operations to be done within terminal windows.

The simplest way of executing MultiTail is to specify the names of the files that you want to monitor on the command line. For example, the multitail access.log error.log command enables you to monitor both the primary log files for an Apache Web server, as shown in Figure 1.

Figure 1. Viewing multiple Apache logs in MultiTail

Viewing multiple Apache logs in MultiTail

MultiTail has a huge number of options, all of which are documented in its traditional UNIX man page. When running MultiTail in a terminal session, you can request help, actively modify the display, search for specific text, and so on by using a variety of keyboard commands. You can access MultiTail’s built-in help at any time by clicking Control-h, which displays an overlaid help window, as shown in Figure 2.

Figure 2. Viewing help in MultiTail

Viewing help in MultiTail

You can scroll through this window using the cursor keys and close it by typing q or Control-g. (MultiTail’s help is also bound to the F1 key, but graphical desktops such as GNOME usually intercept that key sequence and display their own online help instead.)

Some of the more commonly used MultiTail command-line options are described in Table 1.

Table 1. Popular multitail command options

Option Description
--closeidle N Enables you to specify a period of inactivity after which a MultiTail window will close. This can be useful if you are monitoring a large number of files at the same time but only care about those that are actively being updated. Though this option can be convenient, you should use it with care because after a window closes it will not reopen if new output is written to it.
-i file Enables you to specify the names of the files that you want to monitor. Ordinarily, you simply specify the names of the files that you want to monitor by listing their names on the command line. Preceding a file name with the -i option enables you to monitor a file whose name begins with a dash (-). Without using the -i option, any file whose name begins with a dash is assumed to be a command-line option. (This is much like the use of the -- option in commands such as rm.)
--mark-interval N Enables you to specify a period of time (N seconds) of inactivity that will cause a mark line to be displayed in any MultiTail output window. For example, if you are monitoring the /var/log/auth.log file and no authentication-related messages have been logged to that file in N seconds, MultiTail displays a highlighted line such as ---auth.log DATE TIME--- in the MultiTail output window for that log file. This is especially handy when you are monitoring multiple log files because it makes it easy to quickly identify the log files to which no new messages have been logged.
--no-repeat Causes MultiTail to suppress display of repeated messages in the same window, displaying a summary message (“Last message repeated N times”) instead.

Figure 3 shows the output from the following command on a sample machine after 60 seconds:

multitail --mark-interval 10 --no-repeat auth.log \
              daemon.log kern.log lpr.log mail.err mysql.err \
              wpa_supplicant.log messages

Figure 3. More advanced log file display in MultiTail

More advanced log file display in MultiTail

As you can see from Figure 3, MultiTail divides the available height of your terminal window between all the files that you want to monitor. In most cases, the last one or two lines of log file output is sufficient to identify any truly important messages that are being displayed in one of your log files.

The options used in the previous example are only the tip of the MultiTail iceberg. As you can see from the figures, MultiTail supports various types of highlighting, including pattern-based highlighting that makes it easy to spot messages associated with a specific device or that contain a specific keyword.

The multitail command is not limited to simply monitoring existing files. It also provides options that let you monitor the output of a command (-l command) that produces continuous output. For quick-running commands, the -r N option enables you to repeat a given command after N seconds.

Monitoring command output

Sometimes it’s more important to know that a given log file is being written, and when, than it is to know what’s being written to it. Apache’s access log is a good example — heavily used Web servers can generate thousands of log messages in a few minutes, each of which provides detailed information that isn’t really useful. In cases such as these, what most systems administrators actually care about is that the log is constantly growing — in other words, that Apache is running and being accessed.

You can certainly do this visually by repeatedly running a command such as ls -l and comparing file sizes. As mentioned at the end of the previous section, you can also use the multitail command to monitor command output and repeat a specified command multiple times. Unfortunately, this requires a few command-line options:

multitail -Rc 2 -l "ls -l access.log error.log"

The watch command, which is installed by default on most Linux systems, provides a simpler solution. The watch command automatically runs a specified command at specified intervals, with a default interval of two seconds, and does exactly the right thing when combined with a command such as ls -l. Figure 4 shows the output from the watch ls -l access.log error.log command.

Figure 4. Monitoring log file status using watch

Monitoring log file status using watch

Unlike the multitail command, the watch command runs until interrupted by a keyboard command such as Control-C.

The watch command allows you to watch the output of a program change over time, which can be especially convenient when looking for changes to file sizes that reflect debugging output, download or file creation state, memory or disk usage, and so on. For example, running a command such as watch cat /proc/meminfo provides a simple way of viewing memory use on your system in real-time. When monitoring a command in which multiple lines differ each time that it is executed, you can specify the -d option to highlight the differences between successive runs of the command. You can also use the --differences=cumulative option to accumulate highlighted differences between all runs of the specified command.

Different watch commands on different systems

The watch command is part of the procps package, a GPL v2 package that also provides commands such as ps, top, vmstat, and so on. Be careful if you are using a UNIX or UNIX-like system other than Linux, because other operating systems provide commands called watch that are completely different. For example, on Berkeley Software Distribution (BSD)-inspired systems, watch is usually called cmdwatch or gnuwatch, because these systems have a different watch command that monitors the processes running on a specified physical or virtual terminal. Similarly, IBM® AIX® operating systems provide a watch command that monitors the processes that are created by a specified application.

If your system doesn’t provide the standard watch command and you don’t have the time or inclination to build the procps package for your system, you can easily simulate it with a simple shell script. The shell script shown in Listing 1,, provides much of the same functionality as the watch command.

Listing 1. A shell equivalent for the watch command


    while [ 1 ] ; do
      echo "Command: $*"
      echo ""
      ( $* )
      sleep 2

Figure 5 shows the output from running the ls -lt \*.log command in the /var/log directory on a sample OpenSolaris system. Quoting any wildcards that you pass to this script isn’t necessary but helps simplify the output that it displays.

Figure 5. Output from running the ls -lt *.log command

The simplewatch,sh Script on OpenSolaris


Monitoring log files, file creation and size, and differences between subsequent runs of a specific command are common systems administration tasks. The tail, multitail, and watch commands discussed in this article help automate these potentially boring tasks, making it easy to keep an eye on various aspects of your systems while also getting other work done.