constructing protocols with the app framework

In QRadar terminology a protocol is the mechanism that is used as part of a log source to get data into QRadar. Common examples of this are our Syslog, Log File, and WinCollect protocols. Modern services and applications often allow users to retrieve events or subscribe to event feeds via a REST API. Unfortunately QRadar does not have a good built-in generic way to do this, and instead the QRadar Integration team has to build a Protocol for each individual API. Although there is not a way to create Protocols as driven by the ecs service, you can create an App that acts as an API connector to get events.

From a high level perspective the app you want to create needs to poll your API on some interval and then send those events into QRadar via syslog. There are no specific requirements on the format of the syslog messages you send. Below I will go over the steps to consider designing a data retrieving App for QRadar.

#1 The API

The first thing to consider is the API you want to get events from. A big part of the design is knowing how events should be pulled from the API, is it via subscription? Is it via a poll? If so how does it track of what was polled in the last interval? Knowing your own API and how it should be best used by the App is for you to bring to the development of the App.

#1a The Polling Thread

Provided you need to poll on an interval (a common use case for a lot of REST APIs giving events through a GET) the best way to manage polling is to have your App fire a thread to poll on an interval controlled with a sleep. A quick and dirty look at this would be something like:

    #loop until we hit an exception or kill the app.
    run_forever = True

    while run_forever:

        try:
            #assume we're passing in a user set credentials and proxy settings if applicable
            account_conf = configmanager.ConfManager(APP_CONFIG)
            proxy_conf = configmanager.ConfManager(PROXY_CONFIG)
            account_credentials = account_conf.get_account_credentials()
            proxy_credentials = proxy_conf.get_proxy_credentials()
            
            #basic error check
            if not account_credentials:
                logger.warn("No credentials provided. Will retry after 1 minute.")
            else:
                #Assuming we have an eventCollector class that defines the API call that will get the events.
                account_profile = account_credentials['account_profile']
                events = eventCollector.collect_events(account_credentials,proxy_credentials)  
                #now that we have the events let's pass them to a LEEF Logger
                LEEFLogger.processEvents(events) 
        except Exception as e:
            logger.error("Error starting data collection. Error = {0}. Traceback = {1}".format(e, traceback.format_exc()))

        logger.info("Sleeping for 1 min. Continue data collection in next iteration")
        time.sleep(60)

#2 Formatting the Events

The preferred format for syslog events is the IBM defined Log Event Extended Format (LEEF.) While this is preferred, if your API has returned well formatted JSON or another name-value-pair format it is acceptable to use that. See the folowing guide on how to properly format your events into LEEF: QRadar LEEF Format Guide.

This is merely a preference and for customers, getting events in QRadar is more important than the format of the events.

#3 Sending the events to QRadar using Syslog

After the events are in the App, the best way to get them into QRadar is simply to send them in via syslog. The trick to remember is that you should spoof the syslog header to ensure the App is always sending the events from what appears to be the same location, in case the docker container goes down and comes back up.

WARNING: The syslog channel in QRadar will send the events to the QRadar Console. This is a drawback of using this method.

To actually send the messages into QRadar you will have to create a logger to send your LEEF messages to. A structure like the following will set up your logger. Remember that the APPNAME is the log source identifier.

//start leef_logger code sample

console_address = qpylib.get_console_address()

leef_logger = logging.getLogger('LEEF')
leef_formatter = logging.Formatter("%(asctime)s APPNAME %(message)s", "%b %d %H:%M:%S")
leef_logger.setLevel(logging.INFO)

syslog_handler = logging.handlers.SysLogHandler(address=(console_address, 514), facility=logging.handlers.SysLogHandler.LOG_LOCAL1)
syslog_handler.setFormatter(leef_formatter)
leef_logger.addHandler(syslog_handler)



Then when you have a message to send to QRadar you can call the logger like this:

//calling the logger
leef_logger.info(MSG)



NOTE: When you call the logger info, the MSG is the LEEF formatted message.

Summary

With these examples, you can now send LEEF formatted messages into QRadar that can be used to set up a log source for parsing using the DSM Editor. For more information on data ingestion, see: Processing event data in QRadar.

8 comments on"Blog: Using the App framework to build your own “Protocol”"

  1. Can I access the host Syslog at TCP port 514 from within a Docker container? Seems like the connection timeout or reset.

  2. Does QRadar app(running within the container environment of Qradar) allows to write syslog over TCP?
    I am using:
    syslog_handler = logging.handlers.SysLogHandler(address=(console_address, 514), facility=logging.handlers.SysLogHandler.LOG_LOCAL1)
    This defaults to UDP.
    Tried using socket.SOCK_STREAM as a parameter :
    syslog_handler = logging.handlers.SysLogHandler(address=(console_address, 514), facility=logging.handlers.SysLogHandler.LOG_LOCAL1,socket.SOCK_STREAM)
    But the import socket in the QRadar app code does not seem to work, as the QRadar app deployment fails when I do import socket.

  3. qradarnoob June 04, 2019

    How do you poll data from external API when the container app has no internet access?

  4. I need creating a QRadar app with a Python code that does this, but runs as a background service that does not require user interaction. What kind of app should I create for this? I am using QRadar App Editor and available templates are for user interface elements (buttons, mouse right-click, custom tabs, etc.). I need my app starts when the QRadar server starts and runs every (say) two minutes for polling a REST API and import those events into QRadar without the user having to click on any element in the user interface. I was able to get the events and import using a custom tab app, but need to turn my app into a background service. Can anyone explain me how to do that?

  5. qradar_integration February 12, 2020

    I need to build an app that does this: call a REST API, get events from there and import into QRadar. I was able to do so using a Custom tab app (using the QRadar App Editor and the Hello World – Custom tab template). However, I need to turn that app into a sort of background service that runs without any user interaction (the user should not need clicking on a button, opening a tab or performing any action in the user interface). Instead, the app should start when the QRadar server starts and run every (say) to minutes to check if there are new events, calling the REST API, all this without the user having to do anything. How can I create that type of app in QRadar? I could not find any article explaining how to do so and the templates available are only for user interface elements (mouse right-clicks, buttons, etc.). How can I create an app that runs on the background and uses the same Python code I used for building my custom tab app?

    • I wanted the same functionality as you. I did i through a custom config pop-out. Works like a charm. Added a little bit of html, GET and POST requests and now I can configure my app live.

Join The Discussion

Your email address will not be published. Required fields are marked *