IBM Developer

Tutorial

Integrate Linux logs with IBM Security QRadar Log Insights

Efficient Linux log integration with QRadar Log Insights through Data Collector

By Rahul K P

On this page

IBM Security QRadar Log Insights improves threat visibility and detection in your deployment by providing a workflow to collect and ingest essential event and alert data on all of your threat attack paths.

QRadar Log Insights ingests alerts from multiple sources and enriches them with context to determine priority and then correlate them into a case. Analysts are sent high-priority cases along with recommended actions.

IBM Data Collector sends events to the platform by using a Transport Layer Security over the Transmission Control Protocol connection (TLS over TCP) to Kafka. When the Data Collector uses TLS over TCP, it buffers incoming events if it's disconnected from the platform and sends them when the connection is restored. You can configure buffer capacity, limited by the available memory and disk space.

You can use as many Data Collector instances as you need in your environment. The number of instances that you need depends on the number of events per second (EPS) that you need to send and receive.

Prerequisites

To follow this tutorial, you need:

  • Administrative access to the Linux® system that you want to configure with the Data Collector (see the Data Collector system requirements)
  • Access to the QRadar Log Insights interface with permissions to configure log sources
  • Administrative access to the Linux system that you want to collect logs from

High-level architecture: Ingesting logs to QRadar Log Insights

The following image shows a high-level architecture of ingesting logs to QRadar Log Insights.

Screenshot 2024-02-10 at 1.43.03 AM.png

Steps

  1. Configure Syslog on the Centos machine to forward logs to an external system
  2. Set up and configure the QRadar Data Collector.
  3. Install and register the Data Collector by using the Data Collector installer package.
  4. Add a data source.
  5. Test and validate.
  6. Install and register the Data Collector manually.

Note: For Step 3 and Step 6, you can choose either Step 3 or Step 6. If you prefer not to create an API Key for registering your Data Collector instance, follow the installation and registration procedures that are outlined in Step 6 for IBM Data Collector.

Step 1. Configure Syslog on the Centos machine to forward logs to an external system

To configure Syslog:

  1. Access the CentOS command line as the root user. Ensure that you're logged in to your CentOS machine with root privileges so that you can modify system configurations.

  2. Edit the Syslog configuration file. Open the rsyslog.conf file for editing by running the following command in the terminal.

    vi /etc/rsyslog.conf

  3. Configure log forwarding. Add the following line to the rsyslog.conf file to specify the IP address and port of your QRadar Data Collector.

    *.* @<QRADAR_DATA_COLLECTOR_IP>:514**

    Note: Replace <QRADAR_DATA_COLLECTOR_IP> with the actual IP address of your QRadar Data Collector. The @ symbol indicates that logs should be forwarded using UDP. Use @@ for TCP forwarding instead. If the syslog port has been changed from the default (514) to another port, ensure that you use the correct port in this configuration.

  4. Restart the Syslog service. Apply your changes by restarting the syslog service by running the following command.

    sudo systemctl restart rsyslog

    Screenshot 2024-02-10 at 12.48.44 AM.png Screenshot 2024-02-10 at 12.50.37 AM.png

Step 2. Set up and configure QRadar Data Collector

One method to ingest data sources is to install IBM Data Collector on a virtual machine (VM) and register it with the platform. You can use as many Data Collector instances as you need in your deployment. IBM Data Collector requires IBM SDK, Java™ Technology Edition, Version 8, 64-bit.

  1. Navigate to https://www.ibm.com/support/pages/java-sdk-downloads-version-80.

  2. Select the latest version of the installation package from the downloads.

    Screenshot 2024-02-10 at 1.52.06 AM.png

  3. Scroll down to the license information page, right-click I agree, and select Copy Link Address.

    Screenshot 2024-02-29 at 1.02.40 PM.png

  4. Download the installation package to your computer or VM where you plan to install the Data Collector by using the following command, then press Enter.

    wget <paste the copied link address from step 4>

    Screenshot 2024-02-10 at 1.58.31 AM.png

    Note: InstallAnywhere (.bin and archive.bin) packages of IBM SDK, Java Technology Edition will be discontinued starting with the first quarter release of 2024. For more information, see the Withdrawal of InstallAnywhere packages for IBM SDK, Java Technology Edition and IBM Semeru Runtimes support news article.

  5. Run the following command to install the RPM file.

    rpm -i  ibm-java-x86_64-sdk-8.0-<version>.rpm

Step 3. Install and register a Data Collector by using the Data Collector installer package

The Data Collector installer package includes Data Collector installer files, the RPM file for the latest version of Data Collector, and a script that installs and registers the Data Collector with the platform.

  1. Log in to the instance where you need to configure the Data collector as the root.

  2. Create a temporary directory on your Data Collector instance where you can download the Data Collector installer package, such as /dcinstall.

    Screenshot 2024-02-10 at 2.36.05 PM.png

  3. Navigate to IBM Fix Central.

  4. Select QRadar from the list, and click Continue. Click Continue again on the next screen, keeping the selection as default.

    Screenshot 2024-02-10 at 2.39.41 PM.png

  5. Type dc-installer-1.8.3 in the search bar, and press Enter.

    Screenshot 2024-02-26 at 12.42.36 PM.png

  6. Select the Data collector from the Support & Documentation page.

    Screenshot 2024-02-26 at 12.43.13 PM.png

  7. Click fix pack DLC 1.8.4 under the DLC section.

    Screenshot 2024-02-10 at 2.45.20 PM.png

  8. Right-click on the tgz file, and click Copy Link Address.

    Screenshot 2024-02-10 at 2.49.57 PM.png

  9. Navigate to the new directory that you created in the Data Collector instance in Step 2, provide the following command, and press Enter.

    wget <paste the copied link address>

  10. Unpack and run the Data Collector installer package by running the following command.

    tar -zxvf dc-installer-QRadarSuite-1.8.3.tgz

  11. Run the following script to install and register the Data Collector.

    install-dlc.sh

    The Data Collector installer package contains the following files:

  12. dlc-service.rpm: The RPM file for the most current version of Data Collector

  13. dlc-installer.jar: The Java Data Collector registration program

  14. dlc-installer.conf: The configuration file for the Data Collector registration program

  15. Update the dlc-installer.conf file by populating the ServerUrl field with QRadar Log Insights platform instance URL details.

    vi dlc-installer.conf

  16. Press I on your keyboard to begin editing the file.

    Screenshot 2024-02-10 at 2.57.48 PM.png

  17. Update the file with the QRadar Log Insights Server URL, API Key, and Secret that you generate from the Log Insights console.

  18. Press esc on the keyboard, enter :wq, and press Enter to save the file.

  19. Run the following command from the same location to install the Data collector on your VM.

    ./install-dlc.sh

Step 4. Add a data source

Add new ingestion data sources to receive events from your network devices, servers, workstations, SaaS services, or applications. You must ensure that a Data Collector is configured before you can add data sources.

  1. Click Connections > Ingestion data sources from the main menu.

    Screenshot 2024-02-26 at 2.52.36 PM.png

  2. Click Add a data source on the Ingestion data sources page.

    Screenshot 2024-02-26 at 2.57.03 PM.png

  3. Enter a unique name and description for the data source.

  4. Select a Data source type.
  5. Select the relevant connector type for your data source.
  6. Enter a unique identifier for the system or application that generates the event data in the Data source identifier field.

  7. Ensure that the Enabled parameter is set to On to enable the ingestion data source to be ingested, parsed, and normalized.

    Screenshot 2024-02-26 at 2.58.12 PM.png

  8. Select a Data Collector to collect the events for this data source. (In this case, choose the data collector that you configured from the list). The Data Collector must have visibility to the external system that the data source represents.

  9. Select a Credibility level from the list. You can choose a credibility level from 0 to 10. The higher the credibility, the more certain that you are that this log source sends reliable events.

  10. (Optional) Enter the IP address of the data source whose events are sent to the Data Collector, in the Sending IP field.

  11. Click Add.

    Screenshot 2024-02-26 at 3.05.34 PM.png

    Tip: Data sources that use the Syslog Connector type do not need to be assigned to a particular Data Collector. A Syslog listener is automatically active on all Data Collectors, so a Syslog feed can go to any Data Collector or balance the load across multiple Data Collectors. The events are received and forwarded to QRadar Log Insights.

  12. Complete the parameters for the selected data source type and Connector type combination on the Connector tab (for this case, it is Linux OS logs).

  13. See the instructions for your data source type and Connector type.

  14. Click Add.

    Screenshot 2024-02-26 at 3.06.58 PM.png

  15. Select the ingestion data source that you want to edit, and click Edit.

    Screenshot 2024-02-26 at 3.08.17 PM.png

  16. To delete an ingestion data source, select the ingestion data source that you want to delete, and click Delete > Delete.

Step 5. Test and validate

To test whether the Data Collector registration has occurred and confirm whether QRadar Log Insights is receiving Linux logs:

  1. Navigate to the QRadar Log Insights console.

  2. Click the hamburger menu at the upper left, and go to Connections --> Data collectors.

    You see the Data Collector that you configured in Step 2 using the installer package, and it shows the status as SUCCESS.

    Screenshot 2024-02-26 at 1.59.51 PM.png

  3. Navigate to the hamburger menu again, and go to Data Explorer -> Search.

    Screenshot 2024-02-26 at 2.00.42 PM.png

  4. Construct a sample query, provide the IP address of your Linux instance, select a time frame, and click Run query. The result is displayed as shown in the following image.

    Screenshot 2024-02-26 at 2.04.07 PM.png

  5. Click the KQL Results section to view the exact events.

    Screenshot 2024-02-26 at 2.09.06 PM.png

Step 6. Install and register the Data Collector manually

Use this installation and registration procedure for IBM Data Collector if you do not want to create an API Key to register your Data Collector instance.

  1. Download Data Collector from IBM Fix Central.

  2. Enter disconnected log collector in the Search Fix Central field to find the installation package.

    Screenshot 2024-02-26 at 1.12.07 PM.png

  3. Click DataCollector from the list.

    Screenshot 2024-02-26 at 1.13.07 PM.png

    You are redirected to the DLC section.

  4. Click DLC 1.8.4.

    Screenshot 2024-02-26 at 1.15.01 PM.png

  5. Right-click on the fix pack, and select Copy Link Address.

    Screenshot 2024-02-26 at 1.17.52 PM.png

  6. Log in to the Data Collector computer or VM as the root user, and navigate to the /tmp directory.

  7. Use the wget command to download the fix pack.

    wget <link address copied from step 4>

  8. Install Data Collector by running the following command.

    yum -y install /tmp/<dlc_installer_file>.rpm

  9. Run the following command to see the dlc status after the installation completes.

    systemctl status dlc

    Screenshot 2024-02-29 at 4.00.32 PM.png

An active (running) message indicates that the installation was successful and that Data Collector is running.

Step 6.1. Register Data Collector on the platform

After you install Data Collector, you must register your Data Collector with the platform to ensure that you receive events.

  1. Run the following script on your Data Collector instance.

    /opt/ibm/si/services/dlc/current/script/startRegistration.sh

  2. Open the syslog port on the firewall by typing y when prompted.

    When you open the ports, you can see the following output: firewall-cmd --zone=public --add-port=514/udp --permanent success firewall-cmd --zone=public --add-port=514/tcp --permanent success firewall-cmd --zone=public --add-forward-port=port=514:proto=tcp:toport=1514 --permanent success firewall-cmd --zone=public --add-forward-port=port=514:proto=udp:toport=1514 --permanent success firewall-cmd --reload success firewall changes complete, opened the syslog port

  3. Copy the Data Collector certificate signing request. For example:

    
-----BEGIN CERTIFICATE REQUEST-----
"contents of the certificate request"
-----END CERTIFICATE REQUEST-----

  4. Run the following command to verify that the Data Collector is active.

    systemctl status dlc

  5. Log in to the QRadar Log Insights platform.

  6. Click Connections > Data Collectors, and then click Register a Data Collector in the general settings menu section.

    Screenshot 2024-02-13 at 7.27.50 PM.png

  7. Enter a Name and Description for your Data Collector, then click Next.

    Screenshot 2024-02-13 at 7.32.50 PM.png

  8. Log in as the root user on the computer where the Data Collector is installed.

  9. Run the following command.

    /opt/ibm/si/services/dlc/current/script/startRegistration.sh

    Note: When running the previous script, the syslog port of the Data collector changes to 1514 from 514. Therefore, in the rsyslog configuration, you should provide the port number as 1514.

  10. Paste the certificate signing request from your Data Collector, then click Next.

    Screenshot 2024-02-13 at 7.33.18 PM.png

  11. Download the connection bundle to your Data Collector by clicking Download connection bundle. The connection bundle might take a few minutes to generate before you can click Download connection bundle to download it.

    Screenshot 2024-02-13 at 7.33.37 PM.png

    Screenshot 2024-02-13 at 7.38.01 PM.png

    Note: The connection bundle contains the configuration information that each Data Collector instance needs to complete the connection to the platform.

  12. Click Done.

  13. Copy the connection bundle .json file to your Data Collector instance.

  14. Replace the value for "connection bundle JSON file" with your connection bundle's file name, and then run the following script to process the connection bundle.

    /opt/ibm/si/services/dlc/current/script/GenerateKafkaFiles.sh "connection bundle JSON file"

  15. Run the following command on your Data Collector instance to validate that your Data Collector is connected to the platform.

    /opt/ibm/si/services/dlc/current/script/jmx.sh -p 7787 -b "com.q1labs.sem:application=dlc.dlc,type=destinations,name=CP4SForwardDestination"

Results for successful installation

After you process the connection bundle, you receive a message that says Successfully validated the connection bundle file. After the script restarts the Data Collector, the Data Collector can start receiving event data.

Note: The testing phase for this procedure follows the same steps that are outlined in the Test and validate section.

Step 7. Troubleshooting

  1. Run the following command on your Data Collector instance to validate that your Data Collector is connected to the platform.

    /opt/ibm/si/services/dlc/current/script/jmx.sh -p 7787 -b "com.q1labs.sem:application=dlc.dlc,type=destinations,name=CP4SForwardDestination"

  2. If the Connected value is true, then you have successfully connected your Data Collector.

    Screenshot 2024-02-26 at 3.44.03 PM.png

  3. Issue with the latest Java SDK file

    Screenshot 2024-02-26 at 3.54.47 PM.png

If you encounter the warning displayed in the previous image in the Data Collector configuration, follow these steps:

  1. Open the jmx_ac.properties file located at:

    vi /opt/ibm/si/services/dlc/conf/.jmx_ac.properties

  2. Press the I key on your keyboard to enter insert mode.

  3. Move to the end of the initial line, and press the Enter key to insert a blank line.

    Screenshot 2024-02-23 at 11.39.05 PM.png

  4. Press the ESC key, and type :wq to save and exit the file.

  5. Edit the jmx.properties file located at:

    vi /opt/ibm/si/services/dlc/conf/.jmx.properties

  6. Repeat Steps 2 to 4.

  7. Restart the DLC service by using the following command.

    systemctl restart dlc.service

  1. Navigate to the following location.

    cd /var/log/dlc/

  2. Perform an ls command to locate essential configuration files and error log files in this directory. Important files include:

    • dlc.error
    • dlc.log
    • kafka.error
    • kafka.log

Summary

In this tutorial, you learned how to integrate Linux logs efficiently with IBM QRadar Log Insights using Data Collector. The tutorial guided you through the set up and configuration process, enabling you to use the advanced features of QRadar Log Insights for enhanced log management and security analysis.