Overview

Skill Level: Any Skill Level

This recipe explains a method to handle retries of sending events to a Netcool system. See recipe "ScienceLogic EM7 to Netcool event integration (JSON)" to set the SL1 to send events to Netcool.

Ingredients

For this recipe the prerequisite is to have the ScienceLogic to Netcool Event Integration implemented (see ScienceLogic EM7 to Netcool event integration (JSON) recipe at https://developer.ibm.com/recipes/tutorials/sciencelogic-em7-to-netcool-event-integration/).

Step-by-step

  1. Add code to the existing runbook action

    In case the Netcool Runbook action fails this code will add required data into a queue to be rerun.

    1) Go to the RunBook/Action and edit the existing Netcool snippet;

    2) Locate part of the code (else) where to add the retry call;

    (code from “ScienceLogic EM7 to Netcool event integration (JSON)” recipe)

       else:
    ¬†¬†¬†¬† err = ‘Netcool Omnibus Request Failed’
         EM7_RESULT = make_display_notes(err)
    else:
    ¬†¬† err = ‘Error Incorrect Credential Type, Quitting’
       EM7_RESULT = make_display_notes(err)

    3) Add the retry call to the snippet

    EVENTID=str(EM7_VALUES[‘%e’])
    JSON=str(JSON_PAYLOAD).replace(“\'”,”\””)
    register_for_retry(EVENTID,JSON)
    NOTE_LIST.append(‘Event scheduled for retry’)

    4) and the original code should be as below

      else:
    ¬†¬†¬† EVENTID=str(EM7_VALUES[‘%e’])
    ¬†¬†¬† JSON=str(JSON_PAYLOAD).replace(“\'”,”\””)
        register_for_retry(EVENTID,JSON)
    ¬†¬†¬† NOTE_LIST.append(‘Event scheduled for retry’)
    ¬†¬†¬† err = ‘Netcool Omnibus Request Failed’
        EM7_RESULT = make_display_notes(err)
    else:
    ¬† EVENTID=str(EM7_VALUES[‘%e’])
    ¬† JSON=str(JSON_PAYLOAD).replace(“\'”,”\””)
      register_for_retry(EVENTID,JSON)
    ¬† NOTE_LIST.append(‘Event scheduled for retry’)
    ¬† err = ‘Error Incorrect Credential Type, Quitting’
      EM7_RESULT = make_display_notes(err)

    5) Add a new function on this snippet (make sure the code indentation is correct, Python depends on it)

    def register_for_retry(EVENT,JSON):
       from silo_common.database import local_db
       dbc = local_db()
    ¬†¬† QUERY= “CREATE TABLE IF NOT EXISTS master.RetryTBL (event_id VARCHAR(10) PRIMARY KEY,json VARCHAR(1000) NOT NULL)”
    ¬†¬† logger_debug(‘Query1’,QUERY)
       dbc.execute(QUERY)
    ¬†¬† QUERY = “INSERT INTO master.RetryTBL (event_id,json) VALUES (‘”+EVENT+”‘,'”+JSON+”‘)”
    ¬†¬† logger_debug(‘Query2’,QUERY)
       dbc.execute(QUERY)
       dbc.close()

    6) This code will basically create a new table (master.RetryTBL) and add the eventID and the Json Payload from all events that failed to be sent to the Netcool.

    Note: Remenber both Databases and Data Collectors components have a DB, so depending on where this code is executed the new table can be created at the Database or the Data Collector level.

     

  2. The retry Runbook Automation

    Now we need a mechanism to execute automatically every 5 min to process the retry event queue.

    Note: Runbook Schedules should handle this very easily but in SL1 current version (8.12.0) schedules seem not to be working correctly. So let’s create something based on events.

    Let’s create an event every N minutes and for this task let’s use SL1 API.

    1) Create a script on the DB linux box as shown below

    #!/usr/bin/python -u

    import requests
    import json
    from requests.auth import HTTPBasicAuth

    #####SL1 Event API definitions
    url = “https://localhost/api/alert”
    payload_header = {“Authorization”: “Basic <your_api_credential>”,”Content-Type”: “application/json”}
    payload_data = { “message”: “API_ALERT”, “message_time”: “0”, “aligned_resource”: “/api/device/2” }
    r2 = requests.post(url, headers=payload_header, data=json.dumps(payload_data), verify=False, timeout=5)

    2) Add the script into the linux crontab

    */5 * * * * /home/em7admin/python/sl1_api_event.py

    3) Create a new Event Policy to handle the new alert

    – Go to Registry>Events>Event Manager
    – Click Create
    – Set Event Source as API
    – Set Event Severity (e.g. Notice)
    – Give a name for the new Policy (e.g. API Event)
    – Set a Expire delay as 1 min
    – Set Occurence Count and Time as Disable
    – Set First Match as “API_ALERT” (to match to the API generated event only)
    – Save

    4) Create a new Runbook Automation

    – Go to Registry>Runbook>Automation
    – Click Create
    – Give a name for the new Automation (e.g. NetcoolRetry)
    – Set criteria logic to match the Event Policy created above (e.g. Severity = Notice)
    – Set Aligh With to Device (the API event is being sent as aligned_resource=/api/device/2)
    – On the available events set the event created above (e.g. API Event)
    – The action will be set to the new action (need to go back to this step as soon as we create the new action)
    – Save

  3. The retry Runbook Action

    Now that we have the Runbook Automation ready, let’s create the Runbook Action to process the retry queue.

    1) go to Registry>Runbook>Actions

    2) Click Create

    3) Give a name (e.g. NetcoolRetry)

    4) Set the Credential and Run Context same as the original Netcool Runbook Action

    5) Use the code below as the snippet

    import json
    import time
    from silo_common.database import local_db
    import requests, base64
    from requests.auth import HTTPBasicAuth

    #Globals:
    RBA_NAME = 'Netcool OMNIbus: Retry'
    RBA_VERSION = '1.0.0'
    DEBUG_LOGGING = True # True is on, False is Off
    LOG_FILE = '/data/tmp/omnibusretry.log' #touch file and chmod 644 and chown s-em7-core.s-em7-core

    def logger_debug(log_message=None, log_var=None):
    if DEBUG_LOGGING is True:
    m = '%s %s\n' % (str(log_message), str(log_var))
    with open(LOG_FILE, "a+") as myfile:
    myfile.write(m)

    def make_display_notes(notes):
    if isinstance(notes, list):
    for note in notes:
    logger_debug(note,'')
    note = '<br>'.join(notes)
    return "<p><font style='font: 9pt Helvetica;'>%s</font>" % (str(note))
    elif isinstance(notes, str):
    logger_debug(notes,'')
    return "<p><font style='font: 9pt Helvetica;'>%s</font>" % (str(notes))

    def isNotBlank (myString):
    return bool(myString and myString.strip())

    def omni_request(json_dict):
    #logger_debug("Inside omni_request function",'') ###DEBUG
    #return True ###DEBUG
    #quit() ###DEBUG
    ##Use credential data to detemine the URL
    if 'http' in str(EM7_ACTION_CRED['cred_host']).lower():
    url = "%s:%s" % (str(EM7_ACTION_CRED['cred_host']).strip(), str(EM7_ACTION_CRED['cred_port']))
    else:
    url = "http://%s:%s" % (str(EM7_ACTION_CRED['cred_host']).strip(), str(EM7_ACTION_CRED['cred_port']))
    logger_debug("URL : " , url)
    ##set base64 password
    if isNotBlank(str(EM7_ACTION_CRED['cred_user'])):
    usrPass = str(EM7_ACTION_CRED['cred_user'])+":"+str(EM7_ACTION_CRED['cred_pwd'])
    b64Val = base64.b64encode(usrPass)
    ##set basic JSON header with Authorization
    headers = {'content-type': 'application/json', "Authorization": "Basic %s" % b64Val}
    logger_debug("Using Authorization in header",'')
    else:
    ##set basic JSON header without Authorization
    headers = {'content-type': 'application/json'}
    logger_debug("NOT using Authorization in header",'')
    ##convert data into JSON format
    try:
    json_data = json.dumps(json_dict)
    except Exception, e:
    logger_debug('JSON Handler Exception in omni_request():', str(e))
    except:
    logger_debug('JSON Handler Exception in omni_request():', '')
    ##make the request to Omnibus WS API
    try:
    r = requests.post(url, headers=headers, data=json_data, verify=False, timeout=5)
    #look for a return status of 200 (Okay)
    if r.status_code == 200:
    return True
    ##handle exceptions if they happen
    except Exception, e:
    logger_debug('HTTP Request Exception in omni_request():', str(e))
    except:
    logger_debug('HTTP Request Exception in omni_request():', '')
    return False

    ##initialize the notes list
    NOTE_LIST = []NOTE_LIST.append('Omnibus Retry Started')

    logger_debug("------------------------------------------------------", '')
    logger_debug(RBA_NAME, RBA_VERSION)

    from silo_common.database import local_db
    dbc = local_db()
    QUERY= "select * from master.RetryTBL"
    dbc.execute(QUERY)
    ROWS = dbc.fetchall()
    for ROW in ROWS:
    EVENTID = ROW[0] JSON = ROW[1] if omni_request(JSON) == True:
    logger_debug("Event sent OK : ",EVENTID)
    logger_debug("JSON",JSON)
    QUERY= "delete from master.RetryTBL where event_id="+EVENTID
    dbc.execute(QUERY)
    NOTE_LIST.append('Omnibus Request Okay for eventid: %s' % (EVENTID))

    else:
    logger_debug("Event retry failed : ",EVENTID)
    NOTE_LIST.append('Omnibus Request Failed for eventid: %s' % (EVENTID))

    NOTE_LIST.append('Omnibus Retry Finished')
    EM7_RESULT = make_display_notes(NOTE_LIST)
    dbc.close()

Join The Discussion