The following covers using new features of the IBM Application Performance Management (APM) 8.1.3 OS agents to ingest data into IBM Operations Analytics – Log Analysis.
How and why is this useful?
The new APM 8.1.3 OS agent includes log monitoring features that are functionally identical to the IBM Tivoli Monitoring (ITM) 6.3 Log file Agent (LFA) that is bundled with Operations Analytics – Log Analysis. We can use the Log Monitoring feature in the APM OS agent, that runs a sub-process, to monitor and send log events to Log Analysis. This allows a customer to use existing APM OS agents for ingesting log files without having to deploy the Log file Agent (LFA).
Existing Log File Agent configuration and format files can be re-used to work with the APM OS agents for log monitoring.
Since the log monitoring capabilities have been added to the APM OS agent in release 8.1.3, you no longer need the separate Log File Agent. This reduces your installation footprint on the managed system, needing just one agent that takes up less disk space.
If you are planning to migrate or upgrade your environment from existing IBM Tivoli Monitoring agents to the newer APM based agents, the ITM LFA as deployed by the Log Analysis remote install tool can coexist with your APM agents. This makes switching between them a matter of shutting down one agent and starting up the other. This can facilitate an easier migration and upgrade within your environment.
You can now distribute log monitoring configurations to your APM OS agents directly using the APM End User Interface. One can also config the APM agent in autonomous mode if it is being used purely for log monitoring purposes.
Note: Co-existence of both the APM and ITM agents is possible when they are each installed in separate directories (i.e., do not install the APM agent in the Log Analysis directory and vice-versa.)
- An IBM Application Performance Management (APM) 8.1.3 environment, namely
- APM OS agent bundles for deployment
- APM UI for additional configuration features
- An IBM Operations Analytics – Log Analysis server, version 1.3.2 or 1.3.3, for ingesting and analyzing log records.
Configuring the APM agent’s Log Monitoring feature:
The APM 8.1.3 OS agent implements additional log monitoring features as an fcp_daemon sub-process. To use this functionality to monitor logs, it has to be configured to read log files and to send log record updates to Log Analysis. The configuration that drives the log monitoring feature is functionally identical to that of your ITM 6.x Log File Agent (LFA) which is bundled with Log Analysis.
The configuration and format file pairs configure the agent to:
- Monitor the log file(s) you are interested in
- Specify the server you intend to send log records to
- Handle operating system-specific monitoring capabilities (for example, Event Logs on Windows).
Location of the configuration files
The APM agent looks for configuration and format file pairs in a directory specified at agent start time. This directory defaults to:
[Agent_install_path]/localconfig/ux/log_discovery/ on Unix
[Agent_install_path]/localconfig/lz/log_discovery/ on Linux
C:\IBM\APM\localconfig\nt\log_discovery\ on Windows.
Options for creating and using configuration file pairs
You can create and/or reuse existing configuration and format file pairs that can then be added to the configuration location described in the previous section.
1) Reusing previously created config file pairs from an existing log file agent:
- by manually copying them to the APM agent’s log_discovery directory
- by uploading them into the APM UI and distributing them to the agent from the UI itself.
2) By creating the files using the Log Analysis “New Data Source” workflow after shutting down the bundled LFA and then deploying them to the APM agent as in (1) above.
3) By writing the .conf and .fmt files from scratch then deploying them to the APM agent as in (1) above.
NOTE: Configurations that are manually added to the agent’s log_discovery directory will not show up on the APM UI – however they will still be operational.
How do I use Log Analysis to create configuration file pairs for the APM agent?
You can use the Log Analysis “New Data Source” workflow to generate configuration file pairs for local and remote data sources that can then be used with the APM agent installed on the same system as Log Analysis. Once generated, these can be used with either the ITM LFA or the APM OS agent.
If your APM agent is on a remote system, you can still reuse these configuration file pairs with minor modifications to the configuration.
To create local and remote data sources when the APM agent coexists with the ITM LFA on the same system, you should:
– Shut down the ITM Log File agent. You can shut down the bundled ITM Log File Agent by issuing the following command from within the IBM/LogAnalysis/IBM-LFA-6.30/bin/ directory.
‘itmcmd agent -o default_workload_instance stop lo‘
– Use the Log Analysis “New Data Source” UI workflow to create a local or remote data source as usual:
– Move the newly generated configuration file pairs out of the IBM/LogAnalysis/IBM-LFA-6.30/config/lo/ directory into the APM agent log_discovery directory.
Note 1: The same configuration file pairs for a given data source should never be added to both the APM and ITM agents simultaneously. Only one of the agents, either APM or ITM, should be configured with a configuration and format file pair for a given data source at any time.
Note 2: For custom data sources, the configuration file pair is to be created manually. You can also re-use an existing configuration file pair that was created previously.
Troubleshooting and Gotchas
You may find these questions helpful for troubleshooting an environment that’s not working as intended:
- Is your EIF receiver running on a different port than the default (5529)?
- Does the hostname and logpath configured in the agent match that of the Log Analysis datasource?
- Does the user under which the agent is running, have permission to read the log files being monitored?
- Can the agent successfully log into the remote managed system using the configured SSH usernames and passwords?
- Is the log file being monitored a rolling log file?
- Is the same log file being monitored simultaneously by both agents?
Location of log files to help with troubleshooting
- APM agent logs for troubleshooting are in /logs/
- Log Analysis logs are in IBM/LogAnalysis/logs