A case of Custom Log Source DSM Not Parsing Fields
I'm attempting to send a log from an unsupported device to QRadar and have it parsed correctly. But what I get is fields are not parsed at all and the events are still coming in as unknown.
Here are the steps I've taken thus far...
Add unsupported source STATUS: OK (event are received and seen in QRadar > Log activities)
Wrote a LSX (Log Source Extension) and upload it STATUS: OK (I'm not 100% sure there is no xml error since I use cloud version and can't tail -f /var/log/qradar.error, but upload is successful)
Setup the Log Source Log Source Type: Universal DSM Log Source Extension: [MyLSX] Extension Use Condition: Parsing Override STATUS: OK
Validated parsing for new coming incoming events under QRadar > Log activities STATUS: events are not parsed according to the parser and classified as unknown
Here's a copy of my LSX + an example of an event for pattern
<pattern id="allEventNames">(.*)</pattern>
<!-- [06] Wed, 18 Jan 2017 14:17:55 (0000000) User 'john' login failed! (IP:192.168.37.18) -->
<pattern id="StringPattern1">\[06\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\s\(\d+\)\sUser\s'(.*)'\slogin\sfailed!\s\(IP:(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\)</pattern>
<!-- [01] Thu, 19 Jan 2017 07:39:51 (0001967) The user ABCDEFG logged in successfully via SSH. -->
<pattern id="StringPattern2">\[01\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\s\(\d+\)\sThe\suser\s"?(.*)"?\slogged\sin\ssuccessfully\svia\s(SSH|WEB|FTP).</pattern>
<!-- [02] Thu, 19 Jan 2017 00:30:10 (0001916) Connected from 10.31.17.158 (local address 192.168.48.66, port 22) -->
<pattern id="StringPattern3">\[02\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\s\(\d+\)\sConnected\sfrom\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s\(local\saddress\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),\sport\s(\d+)\)</pattern>
<!-- [02] Thu, 19 Jan 2017 00:30:10 (0001916) Closed session, disconnected from 192.168.17.158 -->
<pattern id="StringPattern4">\[02\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\s\(\d+\)\sClosed\ssession,\sdisconnected\sfrom\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})</pattern>
<!-- [02] Wed, 18 Jan 2017 14:17:59 IP address:192.168.37.18 is blocked for 3000 seconds. -->
<pattern id="StringPattern5">\[02\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\sIP\saddress:(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\sis\sblocked\sfor\s\d+\sseconds\.</pattern>
<!-- [14] Thu, 19 Jan 2017 17:00:14 An error arose on ssh port 22 (error code:995). -->
<pattern id="StringPattern6">\[14\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\sAn\serror\sarose\son\s(ssh|web|ftp)\sport\s(\d+)</pattern>
<match-group description="WingFTP" order="1">
<matcher ext-data="dd MMM YYYY hh:mm:ss" order="1" capture-group="1" pattern-id="StringPattern1" field="DeviceTime"/>
<matcher order="1" capture-group="2" pattern-id="StringPattern1" field="UserName"/>
<matcher order="1" capture-group="3" pattern-id="StringPattern1" field="SourceIp"/>
<matcher ext-data="dd MMM YYYY hh:mm:ss" order="1" capture-group="1" pattern-id="StringPattern2" field="DeviceTime"/>
<matcher order="1" capture-group="2" pattern-id="StringPattern2" field="UserName"/>
<matcher order="1" capture-group="3" pattern-id="StringPattern2" field="Protocol"/>
<matcher ext-data="dd MMM YYYY hh:mm:ss" order="1" capture-group="1" pattern-id="StringPattern3" field="DeviceTime"/>
<matcher order="1" capture-group="2" pattern-id="StringPattern3" field="SourceIp"/>
<matcher order="1" capture-group="3" pattern-id="StringPattern3" field="DestinationIp"/>
<matcher order="1" capture-group="4" pattern-id="StringPattern3" field="DestinationPort"/>
<matcher ext-data="dd MMM YYYY hh:mm:ss" order="1" capture-group="1" pattern-id="StringPattern4" field="DeviceTime"/>
<matcher order="1" capture-group="2" pattern-id="StringPattern4" field="SourceIp"/>
<matcher ext-data="dd MMM YYYY hh:mm:ss" order="1" capture-group="1" pattern-id="StringPattern5" field="DeviceTime"/>
<matcher order="1" capture-group="2" pattern-id="StringPattern5" field="SourceIp"/>
<matcher ext-data="dd MMM YYYY hh:mm:ss" order="1" capture-group="1" pattern-id="StringPattern6" field="DeviceTime"/>
<matcher order="1" capture-group="2" pattern-id="StringPattern6" field="Protocol"/>
<matcher order="1" capture-group="3" pattern-id="StringPattern6" field="DestinationPort"/>
<event-match-multiple device-event-category="unknown" capture-group-index="1" pattern-id="allEventNames"/>
</match-group>
Does anyone see the issue?
Accepted answer
My problem: When you fill a field more than once, you must increment the field named order.
ex:
<matcher order="1" capture-group="2" pattern-id="StringPattern1" field="Username"/>
<matcher order="2" capture-group="2" pattern-id="StringPattern2" field="UserName"/>
If you go to map event what are you getting for an eventID for these? I don't see you setting eventName like referenced in our sample here: https://www.ibm.com/support/knowledgecenter/en/SSKMKU/com.ibm.dsm.doc/c_LogSourceGuide_ExtDocs_write.html
eg:
The issue might be you're not setting something for the event name so the events can't be mapped to a QID.
I'm getting Original QID 20280133, which map to powershell. How weird, my log has nothing related to powershell!
This hypothesis is very probable! I'll ajust the LSX and give it a try tomorrow am
Thank you Chris
You were right, eventName is required:
Field name Description
EventName (Required) The event name to be retrieved from the QID to identify the event.
Here is my revised LSX
<pattern id="allEventNames">(.*)</pattern>
<!-- [06] Wed, 18 Jan 2017 14:17:55 (0000000) User 'john' login failed! (IP:192.168.37.18) -->
<pattern id="StringPattern1">\[06\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\s\(\d+\)\sUser\s'(.*)'\s(login\sfailed)!\s\(IP:(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\)</pattern>
<!-- [01] Thu, 19 Jan 2017 07:39:51 (0001967) The user ABCDEFG logged in successfully via SSH. -->
<pattern id="StringPattern2">\[01\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\s\(\d+\)\sThe\suser\s"?(.*)"?\s(logged\sin\ssuccessfully)\svia\s(SSH|WEB|FTP)\.</pattern>
<!-- [02] Thu, 19 Jan 2017 00:30:10 (0001916) Connected from 10.31.17.158 (local address 192.168.48.66, port 22) -->
<pattern id="StringPattern3">\[02\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\s\(\d+\)\s(Connected)\sfrom\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s\(local\saddress\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),\sport\s(\d+)\)</pattern>
<!-- [02] Thu, 19 Jan 2017 00:30:10 (0001916) Closed session, disconnected from 192.168.17.158 -->
<pattern id="StringPattern4">\[02\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\s\(\d+\)\sClosed\ssession,\s(disconnected)\sfrom\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})</pattern>
<!-- [02] Wed, 18 Jan 2017 14:17:59 IP address:192.168.37.18 is blocked for 3000 seconds. -->
<pattern id="StringPattern5">\[02\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\sIP\saddress:(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\sis\s(blocked)\sfor\s\d+\sseconds\.</pattern>
<!-- [14] Thu, 19 Jan 2017 17:00:14 An error arose on ssh port 22 (error code:995). -->
<pattern id="StringPattern6">\[14\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\sAn\s(error)\sarose\son\s(ssh|web|ftp)\sport\s(\d+)</pattern>
<match-group description="WingFTP" order="1">
<matcher order="1" capture-group="2" pattern-id="StringPattern1" field="Username"/>
<matcher order="1" capture-group="3" pattern-id="StringPattern1" field="EventName"/>
<matcher order="1" capture-group="4" pattern-id="StringPattern1" field="SourceIp"/>
<matcher ext-data="dd MMM YYYY hh:mm:ss" order="1" capture-group="1" pattern-id="StringPattern2" field="DeviceTime"/>
<matcher order="1" capture-group="2" pattern-id="StringPattern2" field="UserName"/>
<matcher order="1" capture-group="3" pattern-id="StringPattern2" field="EventName"/>
<matcher order="1" capture-group="4" pattern-id="StringPattern2" field="Protocol"/>
<matcher ext-data="dd MMM YYYY hh:mm:ss" order="1" capture-group="1" pattern-id="StringPattern3" field="DeviceTime"/>
<matcher order="1" capture-group="2" pattern-id="StringPattern3" field="EventName"/>
<matcher order="1" capture-group="3" pattern-id="StringPattern3" field="SourceIp"/>
<matcher order="1" capture-group="4" pattern-id="StringPattern3" field="DestinationIp"/>
<matcher order="1" capture-group="5" pattern-id="StringPattern3" field="DestinationPort"/>
<matcher ext-data="dd MMM YYYY hh:mm:ss" order="1" capture-group="1" pattern-id="StringPattern4" field="DeviceTime"/>
<matcher order="1" capture-group="2" pattern-id="StringPattern4" field="EventName"/>
<matcher order="1" capture-group="3" pattern-id="StringPattern4" field="SourceIp"/>
<matcher ext-data="dd MMM YYYY hh:mm:ss" order="1" capture-group="1" pattern-id="StringPattern5" field="DeviceTime"/>
<matcher order="1" capture-group="2" pattern-id="StringPattern5" field="SourceIp"/>
<matcher order="1" capture-group="3" pattern-id="StringPattern5" field="EventName"/>
<matcher ext-data="dd MMM YYYY hh:mm:ss" order="1" capture-group="1" pattern-id="StringPattern6" field="DeviceTime"/>
<matcher order="1" capture-group="2" pattern-id="StringPattern6" field="EventName"/>
<matcher order="1" capture-group="3" pattern-id="StringPattern6" field="Protocol"/>
<matcher order="1" capture-group="4" pattern-id="StringPattern6" field="DestinationPort"/>
<event-match-multiple device-event-category="unknown" capture-group-index="1" pattern-id="allEventNames"/>
</match-group>
... but there is still something else. I'll do some more testing