Managing historical data retention values

Edit online

You can change the number of days that resource monitoring data samples are retained in the Cloud APM server database from the default eight days to the best practices values that were established by the agent and data collector developers. The best practices values retain up to 32 days of data for a subset of the data sets. You can also configure custom retention periods of up to 32 days for data sets. The retention period for transaction tracking data from the Response Time Monitoring Agent or middleware agents and Synthetic transaction data cannot be modified.

About this task

After you change the max data retention period to more than 8 days for resource monitoring data, you can see data up to the max retention period on these tabs in the Cloud APM console:

For dashboard line charts that plot a time range such as the last 4 hours on the Status Overview tab, you can compare the values from the time range today with the same time range from another day up to the max retention period. For example, compare the last 4 hours today with the last 4 hours on another day up to 32 days ago. See Adjusting and comparing metrics over time for more information.
For custom views, you can display data from a time range for any day within the max retention period, for example, you can display a 4 hour time range for any day within the past 32 days. See Viewing custom pages for more information.

Note:

You cannot display data from more than a week ago in the Attribute Details tab even if you change the max retention period to more than 8 days.
After changing the max retention period, in the Status Overview tab, and Custom Views tab, you can select a day in the past up to the max retention period you configured. For example, if you configure the best practices, the Cloud APM console allows you to select a day up to 32 days ago. However, the best practices do not retain 32 days of data for all datasets. So line charts or custom views for datasets that still have a retention period of 8 days will only display data if you select a day within the past week.
After you change the retention period, the Cloud APM server starts retaining data for the configured number of days and older data is not available right away. For example, if you enable the best practices retention today, data from 32 days ago cannot be displayed in the Cloud APM console until at least 3 weeks from today.
Historical data for synthetic transactions and transaction tracking is saved in the DATAMART database. The data retention settings cannot be modified for data in the DATAMART database. To view historical data for synthetic transactions or Response Time Monitoring agent transaction data, you must install Tivoli® Common Reporting and the Cloud APM reports. For more information, see Integrating with Tivoli Common Reporting and Reports.

The Cloud APM server Db2® historical database, WAREHOUS, typically stores 8 days of data samples from the IBM® Cloud APM agents and data collectors. The exact number for each data set is published in the agent or data collector attribute help and in the reference PDF (see PDF documentation).

The Cloud APM server has a get_metrics_retention.sh script that you can run to see the number of days that data samples are retained for the data sets in every monitoring agent and data collector, and a set_metrics_retention.sh script for adjusting those values to adopt the best practices settings or the changes you specify. For complete descriptions of the parameters supported by the historical data retention scripts, see Historical data retention scripts.

The procedure has steps for 3 typical scenarios: Set the data retention to the best practices values, reset the retention values to their default 8 days, and apply customized data retention settings after generating a file with the current historical data retention settings and editing the values.

The scripts require the password for the Db2_server_instance_user db2apm, such as db2Usrpasswd@08 (see Default users and passwords).

Procedure

On the system where the Cloud APM server is installed, change to the /opt/ibm/ccm directory or to the directory that you specified during server installation and complete any of the following steps:

Change the data retention settings to the best practices values:
```
./set_metrics_retention.sh -pw db2_instance_user -retention BEST_PRACTICES
```
After you enter y in response to the prompt, Do you want to continue ? (y/n), the data retention settings are changed to the best practice values on the Db2 historical database.
Return the data retention settings to their default values:
```
./set_metrics_retention.sh -pw db2_instance_user -retention 8DAYS
```
After you enter y in response to the prompt, Do you want to continue ? (y/n), the data retention settings are reset to the default 8 days on the Db2 historical database.
Customize the data retention values of specific data sets:
1. Enter the following command to generate the historical configuration file:
```
./get_metrics_retention.sh -pw db2_instance_user -retention CURRENT
```
  After you enter y in response to the prompt, Do you want to continue ? (y/n), the data retention values of all Cloud APM agents and data collectors are read from the Db2 historical database and output to tmp/history_file.cfg.
2. Change the data retention settings of specific data sets by editing /tmp/history_file.cfg from 2 to 32 days.
  For example, this is the Linux® OS agent Disk IO data set after changing the retention value to 22 days: KLZ,KLZ_Disk_IO,KLZDSKIO,Detailed,22,D.
3. After saving your changes to the history_file.cfg, enter the following command:
```
./set_metrics_retention.sh -pw db2_instance_user -history tmp/history_file.cfg
```
  After you enter y in response to the prompt, Do you want to continue ? (y/n), any changes to the data retention values are applied to the Db2 historical database.

Example

This is the Windows OS agent section of the history_file.cfg after the best practices values were set:

****** PRODUCT CODE: KNT ******

KNT,Knt_LFAProfiles,KNTLFAPFLS,Detailed,32,D
KNT,Knt_Log_File_Status,KNTLOGFST,Detailed,8,D
KNT,Knt_LogfileProfileEvents,KNTLOGPEVT,Detailed,8,D
KNT,NT_Computer_Information,NTCOMPINFO,Detailed,8,D
KNT,NT_IP_Address,NTIPADDR,Detailed,8,D
KNT,NT_Monitored_Logs_Report,NTLOGINFO,Detailed,32,D
KNT,NT_Memory_64,NTMEMORY,Detailed,32,D
KNT,NT_Network_Interface,NTNETWRKIN,Detailed,32,D
KNT,NT_Paging_File,NTPAGEFILE,Detailed,32,D
KNT,NT_Process_64,NTPROCESS,Detailed,32,D
KNT,NT_Processor,NTPROCSSR,Detailed,32,D
KNT,NT_Logical_Disk,WTLOGCLDSK,Detailed,32,D
KNT,NT_Objects,WTOBJECTS,Detailed,32,D
KNT,NT_Physical_Disk,WTPHYSDSK,Detailed,14,D
KNT,NT_System,WTSYSTEM,Detailed,32,D

What to do next

If you change the max retention period, you must logout of the Cloud APM console and log back in. You will then be able to select a day up to the new max retention period in the Status Overview tab or Custom Views tab.