After creating the splitter, one needs to write an annotator to parse the required fields from the log record.

Pre-requisites

Determine the fields to be extracted from each log Record

For example in the following Log records sample, extract the highlighted fields. Consider multiple loglines while deciding on the fields to be extracted.

2015-04-18 08:09:35,721 ERROR contentlibrary.exceptions.CLException – Content Library Exception Caught: ERROR CODE: 45012 :: BIPUBLISHER_SERVICE_IS_FOLDER_EXIST_ERROR; message: (404)Not Found

2015-04-18 08:10:53,157 ERROR contentlibrary.exceptions.CLException – Content Library Exception Caught: ERROR CODE: 45012 :: BIPUBLISHER_SERVICE_IS_FOLDER_EXIST_ERROR; message: (404)Not Found

The extracted fields needs to be tagged for future reference. For example tag

Create Annotator in Java Figure 1

Create a new Java package for Annotator

On the src folder, right click –> New –> Package as shown below.  Provide the name of the package and click on Finish.

Create Annotator in Java Figure 3

Create a new class to implement Annotator interface provided by IBM Operations Analytics – Log Analysis product.

On the package created in previous step, right click –> New –> Class. Provide the class name and click Finish.

Create Annotator in Java Figure 2

 

Annotator interface

The Java Annotator interface is defined as follows:


package com.ibm.tivoli.unity.splitterannotator.annotator;
/************************************************************************
* This interface defines the APIs for Java based Annotators and is used
* by third party custom Java Annotator developers
*
***********************************************************************/
public interface IJavaAnnotator
{
/*****************************************************************
* Annotate the input log record & return the output with annotations
*
* @param input
* @return
* @throws JavaAnnotatorException
*****************************************************************/
public JSONObject annotate( JSONObject input ) throws Exception ;
/*****************************************************************
* Data section
* ***************************************************************/
public static final String IBM_COPYRIGHT =
"Licensed Materials - Property of IBM\n"
+ "LK3T-3580\n"
+ "(C)Copyright IBM Corporation 2002.\n"
+ "All Rights Reserved.\n"
+ "US Government Users Restricted Rights - Use, duplication \n"
+ "or disclosure restricted by GSA ADP Schedule Contract with
IBM Corp.\n\n";
}

Implement Annotator interface by copy pasting the following code to MyAppLogAnnotator class created before.

Note:  One can either choose or not choose to use regular expression to parse the required fields from log record. In the below sample regular expression is used to parse fields from the log record.


package com.ibm.la.insightpack.myAppLogAnnotator;

import java.util.ArrayList;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import com.ibm.json.java.JSONArray;
import com.ibm.json.java.JSONObject;
import com.ibm.la.insightpack.myAppLogSplitter.MyAppLogSplitter;
import com.ibm.tivoli.unity.common.logging.LoggerConstants;
import com.ibm.tivoli.unity.common.logging.UnityLogger;
import com.ibm.tivoli.unity.splitterannotator.annotator.IJavaAnnotator;

public class MyAppLogAnnotator implements IJavaAnnotator {

	// Define the fields to be extracted
	private static final String SEVERITY = "Severity";
	private static final String EXCEPTION = "Exception";
	private static final String ERRORCODE = "ErrorCode";
	private static final String DETAIL = "Detail";
	private static final String RETURNCODE = "ReturnCode";
	private static final String RETURNMESSAGE = "ReturnMessage";

	// Define regex to be used
	// 2015-04-18 08:09:35,721 ERROR contentlibrary.exceptions.CLException -
	// Content Library Exception Caught: ERROR CODE: 45012 ::
	// BIPUBLISHER_SERVICE_IS_FOLDER_EXIST_ERROR; message: (404)Not Found
	public static final String LINE_REGEX = "[\\d|-]+\\s(\\w+)\\s(.+)\\s-.+:\\s+ERROR\\sCODE:\\s(\\d+)\\s::\\s(.+);\\smessage:\\s+\\((\\d+)\\)(.+)";

	public static Pattern LINE_PATTERN = Pattern.compile(LINE_REGEX);

	private static UnityLogger logger = (UnityLogger) UnityLogger
			.getLogger(LoggerConstants.GENERIC_RECEIVER_LOG_APPENDER);

	public JSONObject annotate(JSONObject input) throws Exception {

		String lines[];

		if (logger.isDebugEnabled()) {
			logger.debug(getClass(), "-- start of MyAppLog Annotator");
			logger.debug(getClass(), input.toString());
		}

		// read input text
		JSONObject content = (JSONObject) input.get("content");
		String text = (String) content.get("text");

		// start processing
		JSONObject annotations = new JSONObject();
		if (text == null || isEmpty(text))
			return annotations;
		if (!text.endsWith("\n"))
			text = text + "\n";

		// process each line seperately
		// ignore empty lines
		lines = text.split("\\r|\\n");
		int i = 0;
		for (; i < lines.length; i++) {
			if (logger.isDebugEnabled()) {
				logger.debug(getClass(), "Annotating" + lines[i]);
			}

			Matcher matcher = LINE_PATTERN.matcher(text);
			boolean foundMatch = matcher.find();

			// public static final String LINE_REGEX =
			// "[\\d|-]+\\s(\\w+)\\s(.+)\\s.+:\\s+ERROR\\sCODE:\\s(\\d+)\\s::(.+);\\smessage:\\s(\\d+)(\\w+)";
			if (foundMatch) {
				annotations.put(SEVERITY, matcher.group(1));
				annotations.put(EXCEPTION, matcher.group(2));
				annotations.put(ERRORCODE, matcher.group(3));
				annotations.put(DETAIL, matcher.group(4));
				annotations.put(RETURNCODE, matcher.group(5));
				annotations.put(RETURNMESSAGE, matcher.group(6));
			}

		}
		JSONObject Wrapperannotations = new JSONObject();
		Wrapperannotations.put("annotations", annotations);
		Wrapperannotations.put("content", content);
		if (logger.isDebugEnabled()) {
			logger.debug(getClass(), Wrapperannotations.toString());
			logger.debug(getClass(), "--- end of MyAppLog Annotator");
		}
		return Wrapperannotations;
	}

	private boolean isEmpty(String text) {
		return text == null || text.isEmpty();
	}

	public static void main(String[] args) {
		JSONObject object = new JSONObject();
		JSONObject content = new JSONObject();
		JSONObject metadata = new JSONObject();
		String text = "2015-04-18 08:09:35,721 ERROR contentlibrary.exceptions.CLException - Content Library Exception Caught:  ERROR CODE: 45012 :: BIPUBLISHER_SERVICE_IS_FOLDER_EXIST_ERROR; message: (404)Not Found \n";
		text = text
				+ "2015-04-18 08:09:35,721 ERROR contentlibrary.exceptions.CLException - Content Library Exception Caught:  ERROR CODE: 45012 :: BIPUBLISHER_SERVICE_IS_FOLDER_EXIST_ERROR; message: (404)Not Found\n";
		text = text
				+ "2015-04-18 08:09:35,721 ERROR contentlibrary.exceptions.CLException - Content Library Exception Caught:";
		content.put("text", text);
		JSONArray timestampFormats = new JSONArray();
		timestampFormats.add("yyyy-MM-dd HH:mm:ss,SSS");
		object.put("timestampFormats", timestampFormats);
		metadata.put("timestamp", "2015-04-18 08:09:35,721");
		content.put("metadata", metadata);
		object.put("content", content);
		object.put("datasource", "try_applog");

		MyAppLogSplitter splitter = new MyAppLogSplitter();
		MyAppLogAnnotator annotator = new MyAppLogAnnotator();
		ArrayList<JSONObject> splitLogRecords = null;
		System.out.println("-------------Input text");
		System.out.println(object.toString());
		try {
			splitLogRecords = splitter.split(object);
			System.out.println("-------------Splitter output");
			System.out.println(splitLogRecords.toString());
			int i = 0;
			for (; i < splitLogRecords.size(); i++) {
				JSONObject output = annotator.annotate((JSONObject) splitLogRecords.get(i));
				System.out.println("-------------Annotator output");
				System.out.println(output.toString());
			}

		} catch (Exception e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}
}

The main method is present for testing Annotator. After testing the main method can be commented out. The main method is same as what we saw in splitter, methods related to annotators are added.

Test the Annotator

Right click in the Annotator class –> Run As –> Java Application .

If all is good, you would see the following output . Note: the input contained two complete log records and one incomplete log record.

Splitter split them into two Type A records ( i,e complete log records ) and one Type B record ( incomplete record).

Each of these were passed to annotator iteratively were required fields such as Severity, ReturnCode were extracted and annotated (tagged)

————-Input text
{“content”:{“text”:”2015-04-18 08:09:35,721 ERROR contentlibrary.exceptions.CLException – Content Library Exception Caught: ERROR CODE: 45012 :: BIPUBLISHER_SERVICE_IS_FOLDER_EXIST_ERROR; message: (404)Not Found \n2015-04-18 08:09:35,721 ERROR contentlibrary.exceptions.CLExceptionContent Library Exception Caught: ERROR CODE: 45012 :: BIPUBLISHER_SERVICE_IS_FOLDER_EXIST_ERROR; message: (404)Not Found\n2015-04-18 08:09:35,721 ERROR contentlibrary.exceptions.CLExceptionContent Library Exception Caught:”,”metadata”:{“timestamp”:”2015-04-18 08:09:35,721″}},”timestampFormats”:[“yyyy-MM-dd HH:mm:ss,SSS”],”datasource”:”try_applog”}
log4j:WARN No appenders could be found for logger (UnityGenericReceiver).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
————-Splitter output
[{“content”:{“text”:”2015-04-18 08:09:35,721 ERROR contentlibrary.exceptions.CLException – Content Library Exception Caught: ERROR CODE: 45012 :: BIPUBLISHER_SERVICE_IS_FOLDER_EXIST_ERROR; message: (404)Not Found “},”metadata”:{“timestamp”:”2015-04-18 08:09:35,721″,”type”:”A”}}, {“content”:{“text”:”2015-04-18 08:09:35,721 ERROR contentlibrary.exceptions.CLException – Content Library Exception Caught: ERROR CODE: 45012 :: BIPUBLISHER_SERVICE_IS_FOLDER_EXIST_ERROR; message: (404)Not Found”},”metadata”:{“timestamp”:”2015-04-18 08:09:35,721″,”type”:”A”}}, {“content”:{“text”:”2015-04-18 08:09:35,721 ERROR contentlibrary.exceptions.CLException – Content Library Exception Caught:”},”metadata”:{“timestamp”:”2015-04-18 08:09:35,721″,”type”:”B”}}] ————-Annotator output
{“content”:{“text”:”2015-04-18 08:09:35,721 ERROR contentlibrary.exceptions.CLException – Content Library Exception Caught: ERROR CODE: 45012 :: BIPUBLISHER_SERVICE_IS_FOLDER_EXIST_ERROR; message: (404)Not Found “},”annotations”:{“Exception”:”contentlibrary.exceptions.CLException”,”Detail”:”BIPUBLISHER_SERVICE_IS_FOLDER_EXIST_ERROR”,”Severity”:”ERROR”,”ReturnCode”:”404″,”ErrorCode”:”45012″,”ReturnMessage”:”Not Found “}}
————-Annotator output
{“content”:{“text”:”2015-04-18 08:09:35,721 ERROR contentlibrary.exceptions.CLException – Content Library Exception Caught: ERROR CODE: 45012 :: BIPUBLISHER_SERVICE_IS_FOLDER_EXIST_ERROR; message: (404)Not Found”},”annotations”:{“Exception”:”contentlibrary.exceptions.CLException”,”Detail”:”BIPUBLISHER_SERVICE_IS_FOLDER_EXIST_ERROR”,”Severity”:”ERROR”,”ReturnCode”:”404″,”ErrorCode”:”45012″,”ReturnMessage”:”Not Found”}}
————-Annotator output
{“content”:{“text”:”2015-04-18 08:09:35,721 ERROR contentlibrary.exceptions.CLException – Content Library Exception Caught:”},”annotations”:{}}

Join The Discussion

Your email address will not be published. Required fields are marked *