Digital Developer Conference on Data and AI: Essential data science, machine learning, and AI skills and certification Register for free

Build a connector to IBM Cloud Pak for Security with STIX-Shifter

Introduction

IBM Cloud Pak for Security provides a federated search to investigate and analyze security insights across your company. The query is done without moving your data from its source. Cloud Pak for Security provides the Universal Data Insights (UDI) service, which is the primary shared data services layer within the Cloud Pak.

Any application can query or read security data from the variety of shared data sources via the UDI services API. It is built using RESTful principles and integrates with an extensible open source software development kit (SDK) called STIX-Shifter to allow new translation and transmission modules to be contributed by IBM and the worldwide security community. The open source library includes samples and examples of connectors for various security offerings, documentation, and CLIs for testing.

For your data source, build and implement a STIX adapter for it to establish connection between the data source and Cloud Pak for Security. Use the open source library to:

  • Connect to any products that house repositories of cybersecurity data.
  • Convert a STIX 2 pattern to a native data source query.
  • Convert a JSON data source query result to a STIX bundle of observable objects.

Learn more about developing a new STIX-Shifter adapter and the supported query data types.

In this tutorial, you learn how to develop your own UDI connector using STIX-Shifter.

Prerequisites

Before you can start to build a STIX-Shifter connector:

  • Confirm you have login access to the product you want to connect with Cloud Pak for Security.
  • Verify that your selected product is be able to return results in JSON.
  • Collect documentation for your product’s APIs.
  • Familiarize yourself with STIX Version 2.0. Part 4: Cyber Observable Objects and STIX Version 2.0. Part 5: STIX Patterning.
  • Identify the fields within your product that you want to map to STIX objects and properties.
  • Identify any custom objects or properties.

Estimated time

It will take you eight hours to develop a basic hello world connector using the STIX-Shifter SDK.

Steps

Follow these steps to develop and test your connector.

1. Set up your development environment

Your development environment must use Python 3.6.

You can use an integrated development environment of your choice, such as Visual Studio Code.

Use the following steps to configure and test your developer environment:

  • Install Python 3.6.
  • Verify Python 3.6 is installed by opening a terminal window and typing python -version.
  • Fork the STIX-Shifter project.
  • Clone the repo and create a new working branch:

    Copy the repository URL from the Clone menu in GitHub.

    In the same terminal that you cloned the repo, change directory to your newly created repo directory.

  • Create a virtual environment where you will run STIX-Shifter CLI commands. This will install all of the Python packages required by STIX-Shifter.

    Run the following command in the terminal:

     virtualenv -p python3.6 –no-site-packages –distribute virtualenv && source virtualenv/bin/activate && pip install -r requirements-dev.txt
    
  • Test your setup by running the following command in the terminal:

    python main.py translate dummy query '{}' "[ipv4-addr:value = '0.0.0.0']"
    

    If everything is set up correctly, you will see a translated query: {'queries': ["SELECT * FROM tableName WHERE (SourceIpV4 = '0.0.0.0' OR DestinationIpV4 = '0.0.0.0')"]}

2. Build your connector

Create your connector from the open source STIX-Shifter project. Development of a new connector begins with forking the STIX-Shifter project and creating a working branch. The working branch will contain any code changes required by the new connector.

When a new connector is ready for review, a pull request (PR) is submitted. The PR is a request to merge the working branch from the forked project into the main branch of the STIX-Shifter project. When the PR is submitted, a code review is conducted on the proposed changes. The working branch is also cloned and tested in a local development environment using STIX-Shifter’s CLI commands.

Use the following steps to develop your STIX-Shifter connector for a data source:

  • Fork the opencybersecurityalliance/stix-shifter repository to work on your own copy of the library.
  • Create a Translation module.
  • Create a Transmission module.
  • Test your connector.
  • Create a PR to merge your changes in the opencybersecurityalliance/stix-shifter repository.

3. Test your connector

Test your connector’s translation and transmission capabilities. Testing is important to ensure the connector is working properly. The STIX-Shifter developer guide walks you through testing at each step of the development process listed below.

Translate

  • Translate STIX Patterns to native queries with python main.py translate query '{}'.

    python main.py translate dummy query '{}' "[network-traffic:src_port NOT = 37020 AND network-traffic:dst_port != 635] OR [ipv4-addr:value = '333.333.333.0'] OR [url:value = 'www.example.com']" '{"validate_pattern": "true"}'
    

    The validate_pattern option forces validation on the submitted pattern. This check is done before the pattern enters the translation logic. Malformed patterns will throw an error.

  • Translate JSON results to STIX objects with python main.py translate results.

    The identity object represents the data source and is automatically included during the UDI flow, but must be manually included when testing in the CLI. The JSON results would normally come from the data source’s API results call, but a test payload must be included for the CLI to test results translation into STIX.

    python main.py translate dummy results '{"type: "identity","id": "identity–f431f809-377b-45e0-aa1c-6a4751cae5ff", "name": "DummyAdapter","identity_class": "system"}' '[ { "SourcePort": 1234, "EventCount": 1, "Magnitude": 4, "DestinationIpV4": "0.0.0.0", "SourceIpV4": "9.21.123.112", "DestinationPort": 5678, "UserName": "root", "StartTime": 1540312200111}]"{"stix_validator": true}'
    

Transmit

The transmission commands are ping, query, status, results, delete, and is_async.

  • Ping attempts a connection with the data source via the API.

    python main.py transmit dummy '{"host":"some.fake.address.com", "port":"12345"}' '{"auth":{"username": "some_user_name", "password": "some password"}}' ping
    
  • Query submits a query to the data source via the API.

    python main.py transmit dummy ''{"host":"some.fake.address.com", "port":"12345"}' '{"auth":{"username": "some_user_name", "password": "some password"}}' query "SELECT * FROM tableName"
    
  • Status checks the current status of the query (if queries are asynchronous)

    python main.py transmit dummy '{"host":"some.fake.address.com", "port":"12345"}' '{"auth":{"username": "some_user_name", "password": "some password"}}' status"b5751e36-1699-4d57-9f7e-a020b4f91176"
    
  • Results fetches the query results via the API. The page and offset numbers define a range (subset) of the total results returned.

    python main.py transmit dummy '{"host":"some.fake.address.com", "port":"12345"}' '{"auth":{"username": "some_user_name", "password": "some password"}}' results"b5751e36-1699-4d57-9f7e-a020b4f91176"1 3
    
  • Delete deletes the query via the API (if supported by the data source).

    python main.py transmit dummy '{"host":"some.fake.address.com", "port":"12345"}' '{"auth":{"username": "some_user_name", "password": "some password"}}' delete "b5751e36-1699-4d57-9f7e-a020b4f91176"
    
  • Is_async returns a true/false value indicating if the data source uses asynchronous queries.

    python main.py transmit dummy '{"host":"some.fake.address.com", "port":"12345"}' '{"auth":{"username": "some_user_name", "password": "some password"}}' is_async
    

Execute

The execute command provides a means of testing the entire translation-transmission flow. When execute is called from the CLI, the following happens:

  • The submitted pattern is first validated and then translated into a native query.
  • The native query is transmitted to the data source.
  • An is_asynccheck is made. If true, a status check is transmitted to see if the query is complete.
  • Query results are fetched (transmitted) from the data source. This happens right away if the data source is synchronous, otherwise the results are looked up based on the query ID.
  • The results are translated into a bundle of STIX cyber observable objects.
  • STIX validation is run on each of the translated objects.

Summary

In this tutorial, you learned how to build a connector using STIX-Shifter. If you are interested in learning more, check out the other resources located within the STIX-Shifter open source project on GitHub, under the Open Cybersecurity Alliance. To interact with experts on this topic, join the Slack channel by filling out the Open Cybersecurity Alliance Slack Invite Form.