Log Analysis Insight Packs can be written using multiple approaches. Following considerations will help in getting better performance from Insight Packs.
Writing annotators : If you are using multiple REGX across a single log record, use them in the following order:
a) REGX that will pass in most of the cases should be used first in the if/else loop, followed by a check for other REGX. This will ensure optimal processing for most of the log record.
DSV Packs : If the DSV file has more than 10-15 fields:
a) Consider indexing only fields that are relevant for debugging.
b) Reduce the number of fields marked filterable.
c) Don’t set filterable attributes for fields such as message, logRecord or other fields which do not provide any insights by counting them.
d) You could omit indexing a field from the DSV pack by leaving that field in the properties file used to create the pack.
Defining Source Types: If you want to support variations in format of a log record. For example: Depending on the log configuration, there may be an extra field in the log. In this situation, create multiple annotators and use them in creating multiple source types to support each log format versus checking each log record for different formats in the same source type.
Defining Index Config : Index configuration defines the fields extracted from log records and the attribute of each log record.
a) Fields that vary for every log record should NOT be set as filterable. Example: Message field which is large text and varies for each log record.
b) TEXT fields such as Message, Text, LogRecord should NOT be set as filterable and sortable.
c) Limit the number of fields indexed per source type to be less than 10.
d) Do not set true for every attribute for every field. Consider 4-6 fields which you need to count and set filterable as true for these fields. Good candidate for filterable (these fields are counted) are returnCodes, MessageId, HostName, ServerName.
e) Mark fields as searchable only if you are going to perform searches on the field.
Dashboards : While creating dashboards using Dynamic Dashboard functionality or using custom dashboards:
a) Limit the number of charts to 4 per chart.
b) Create different dashboards for different time granularity (Example: One dashboard for months view and another for day view). The monthly view /yearly view will take more time to render than the day view.