Digital Developer Conference: a FREE half-day online conference focused on AI & Cloud – North America: Nov 2 – India: Nov 9 – Europe: Nov 14 – Asia Nov 23 Register now

Close outline
  • United States
IBM?
  • Site map
IBM?
  • Marketplace

  • Close
    Search
  • Sign in
    • Sign in
    • Register
  • IBM Navigation
IBM Developer Answers
  • Spaces
    • Blockchain
    • IBM Cloud platform
    • Internet of Things
    • Predictive Analytics
    • Watson
    • See all spaces
  • Tags
  • Users
  • Badges
  • FAQ
  • Help
Close

Name

Community

  • Learn
  • Develop
  • Connect

Discover IBM

  • ConnectMarketplace
  • Products
  • Services
  • Industries
  • Careers
  • Partners
  • Support
10.190.13.195

Refine your search by using the following advanced search options.

Criteria Usage
Questions with keyword1 or keyword2 keyword1 keyword2
Questions with a mandatory word, e.g. keyword2 keyword1 +keyword2
Questions excluding a word, e.g. keyword2 keyword1 -keyword2
Questions with keyword(s) and a specific tag keyword1 [tag1]
Questions with keyword(s) and either of two or more specific tags keyword1 [tag1] [tag2]
To search for all posts by a user or all posts with a specific tag, start typing and choose from the suggestion list. Do not use a plus or minus sign with a tag, e.g., +[tag1].
  • Ask a question

A case of Custom Log Source DSM Not Parsing Fields

50RT19P272 gravatar image
Question by 09h33  (28) | Feb 15, 2017 at 02:11 PM qradarparsing

I'm attempting to send a log from an unsupported device to QRadar and have it parsed correctly. But what I get is fields are not parsed at all and the events are still coming in as unknown.

Here are the steps I've taken thus far...

  1. Add unsupported source STATUS: OK (event are received and seen in QRadar > Log activities)

  2. Wrote a LSX (Log Source Extension) and upload it STATUS: OK (I'm not 100% sure there is no xml error since I use cloud version and can't tail -f /var/log/qradar.error, but upload is successful)

  3. Setup the Log Source Log Source Type: Universal DSM Log Source Extension: [MyLSX] Extension Use Condition: Parsing Override STATUS: OK

  4. Validated parsing for new coming incoming events under QRadar > Log activities STATUS: events are not parsed according to the parser and classified as unknown

Here's a copy of my LSX + an example of an event for pattern

 <pattern id="allEventNames">(.*)</pattern>
 <!-- [06] Wed, 18 Jan 2017 14:17:55 (0000000) User 'john' login failed! (IP:192.168.37.18) -->
 <pattern id="StringPattern1">\[06\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\s\(\d+\)\sUser\s'(.*)'\slogin\sfailed!\s\(IP:(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\)</pattern>
 <!-- [01] Thu, 19 Jan 2017 07:39:51 (0001967) The user ABCDEFG logged in successfully via SSH. -->
 <pattern id="StringPattern2">\[01\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\s\(\d+\)\sThe\suser\s&quot;?(.*)&quot;?\slogged\sin\ssuccessfully\svia\s(SSH|WEB|FTP).</pattern>
 <!-- [02] Thu, 19 Jan 2017 00:30:10 (0001916) Connected from 10.31.17.158 (local address 192.168.48.66, port 22) -->
 <pattern id="StringPattern3">\[02\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\s\(\d+\)\sConnected\sfrom\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s\(local\saddress\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),\sport\s(\d+)\)</pattern>
 <!-- [02] Thu, 19 Jan 2017 00:30:10 (0001916) Closed session, disconnected from 192.168.17.158 -->
 <pattern id="StringPattern4">\[02\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\s\(\d+\)\sClosed\ssession,\sdisconnected\sfrom\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})</pattern>
 <!-- [02] Wed, 18 Jan 2017 14:17:59 IP address:192.168.37.18 is blocked for 3000 seconds. -->
 <pattern id="StringPattern5">\[02\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\sIP\saddress:(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\sis\sblocked\sfor\s\d+\sseconds\.</pattern>
 <!-- [14] Thu, 19 Jan 2017 17:00:14 An error arose on ssh port 22 (error code:995). -->
 <pattern id="StringPattern6">\[14\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\sAn\serror\sarose\son\s(ssh|web|ftp)\sport\s(\d+)</pattern>
 <match-group description="WingFTP" order="1">
     <matcher ext-data="dd MMM YYYY hh:mm:ss" order="1" capture-group="1" pattern-id="StringPattern1" field="DeviceTime"/>
     <matcher order="1" capture-group="2" pattern-id="StringPattern1" field="UserName"/>
     <matcher order="1" capture-group="3" pattern-id="StringPattern1" field="SourceIp"/>
     <matcher ext-data="dd MMM YYYY hh:mm:ss" order="1" capture-group="1" pattern-id="StringPattern2" field="DeviceTime"/>
     <matcher order="1" capture-group="2" pattern-id="StringPattern2" field="UserName"/>
     <matcher order="1" capture-group="3" pattern-id="StringPattern2" field="Protocol"/>
     <matcher ext-data="dd MMM YYYY hh:mm:ss" order="1" capture-group="1" pattern-id="StringPattern3" field="DeviceTime"/>
     <matcher order="1" capture-group="2" pattern-id="StringPattern3" field="SourceIp"/>
     <matcher order="1" capture-group="3" pattern-id="StringPattern3" field="DestinationIp"/>
     <matcher order="1" capture-group="4" pattern-id="StringPattern3" field="DestinationPort"/>
     <matcher ext-data="dd MMM YYYY hh:mm:ss" order="1" capture-group="1" pattern-id="StringPattern4" field="DeviceTime"/>
     <matcher order="1" capture-group="2" pattern-id="StringPattern4" field="SourceIp"/>
     <matcher ext-data="dd MMM YYYY hh:mm:ss" order="1" capture-group="1" pattern-id="StringPattern5" field="DeviceTime"/>
     <matcher order="1" capture-group="2" pattern-id="StringPattern5" field="SourceIp"/>
     <matcher ext-data="dd MMM YYYY hh:mm:ss" order="1" capture-group="1" pattern-id="StringPattern6" field="DeviceTime"/>
     <matcher order="1" capture-group="2" pattern-id="StringPattern6" field="Protocol"/>
     <matcher order="1" capture-group="3" pattern-id="StringPattern6" field="DestinationPort"/>
     <event-match-multiple device-event-category="unknown" capture-group-index="1" pattern-id="allEventNames"/>
 </match-group>

Does anyone see the issue?

People who like this

  0
Comment
10 |3000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster

4 answers

  • Sort: 
50RT19P272 gravatar image
Accepted answer

Answer by 09h33 (28) | Feb 22, 2017 at 12:24 PM

My problem: When you fill a field more than once, you must increment the field named order.

ex:

 <matcher order="1" capture-group="2" pattern-id="StringPattern1" field="Username"/>
 <matcher order="2" capture-group="2" pattern-id="StringPattern2" field="UserName"/>
Comment

People who like this

  0   Share
10 |3000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
27000724MT gravatar image

Answer by Chris Collins (IBM) (740) | Feb 15, 2017 at 02:35 PM

If you go to map event what are you getting for an eventID for these? I don't see you setting eventName like referenced in our sample here: https://www.ibm.com/support/knowledgecenter/en/SSKMKU/com.ibm.dsm.doc/c_LogSourceGuide_ExtDocs_write.html

eg:

The issue might be you're not setting something for the event name so the events can't be mapped to a QID.

Comment

People who like this

  0   Show 1   Share
10 |3000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
50RT19P272 gravatar image 09h33 (28)   Feb 15, 2017 at 03:39 PM 0
Share

I'm getting Original QID 20280133, which map to powershell. How weird, my log has nothing related to powershell!

This hypothesis is very probable! I'll ajust the LSX and give it a try tomorrow am

Thank you Chris

50RT19P272 gravatar image

Answer by 09h33 (28) | Feb 21, 2017 at 02:49 PM


You were right, eventName is required:


Field name Description

EventName (Required) The event name to be retrieved from the QID to identify the event.


Reference: https://www.ibm.com/support/knowledgecenter/en/SS42VS_7.2.7/com.ibm.dsm.doc/c_LogSourceGuide_ExtDocs_matcher.html

Here is my revised LSX

  <pattern id="allEventNames">(.*)</pattern>
  <!-- [06] Wed, 18 Jan 2017 14:17:55 (0000000) User 'john' login failed! (IP:192.168.37.18) -->
  <pattern id="StringPattern1">\[06\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\s\(\d+\)\sUser\s'(.*)'\s(login\sfailed)!\s\(IP:(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\)</pattern>
  <!-- [01] Thu, 19 Jan 2017 07:39:51 (0001967) The user ABCDEFG logged in successfully via SSH. -->
  <pattern id="StringPattern2">\[01\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\s\(\d+\)\sThe\suser\s&quot;?(.*)&quot;?\s(logged\sin\ssuccessfully)\svia\s(SSH|WEB|FTP)\.</pattern>
  <!-- [02] Thu, 19 Jan 2017 00:30:10 (0001916) Connected from 10.31.17.158 (local address 192.168.48.66, port 22) -->
  <pattern id="StringPattern3">\[02\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\s\(\d+\)\s(Connected)\sfrom\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s\(local\saddress\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),\sport\s(\d+)\)</pattern>
  <!-- [02] Thu, 19 Jan 2017 00:30:10 (0001916) Closed session, disconnected from 192.168.17.158 -->
  <pattern id="StringPattern4">\[02\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\s\(\d+\)\sClosed\ssession,\s(disconnected)\sfrom\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})</pattern>
  <!-- [02] Wed, 18 Jan 2017 14:17:59 IP address:192.168.37.18 is blocked for 3000 seconds. -->
  <pattern id="StringPattern5">\[02\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\sIP\saddress:(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\sis\s(blocked)\sfor\s\d+\sseconds\.</pattern>
  <!-- [14] Thu, 19 Jan 2017 17:00:14 An error arose on ssh port 22 (error code:995). -->
  <pattern id="StringPattern6">\[14\]\s\w{3},\s(\d{2}\s\w{3}\s\d{4}\s\d{2}:\d{2}:\d{2})\sAn\s(error)\sarose\son\s(ssh|web|ftp)\sport\s(\d+)</pattern>
  <match-group description="WingFTP" order="1">

     <matcher order="1" capture-group="2" pattern-id="StringPattern1" field="Username"/>
     <matcher order="1" capture-group="3" pattern-id="StringPattern1" field="EventName"/>
     <matcher order="1" capture-group="4" pattern-id="StringPattern1" field="SourceIp"/>
     <matcher ext-data="dd MMM YYYY hh:mm:ss" order="1" capture-group="1" pattern-id="StringPattern2" field="DeviceTime"/>
     <matcher order="1" capture-group="2" pattern-id="StringPattern2" field="UserName"/>
     <matcher order="1" capture-group="3" pattern-id="StringPattern2" field="EventName"/>
     <matcher order="1" capture-group="4" pattern-id="StringPattern2" field="Protocol"/>
     <matcher ext-data="dd MMM YYYY hh:mm:ss" order="1" capture-group="1" pattern-id="StringPattern3" field="DeviceTime"/>
     <matcher order="1" capture-group="2" pattern-id="StringPattern3" field="EventName"/>
     <matcher order="1" capture-group="3" pattern-id="StringPattern3" field="SourceIp"/>
     <matcher order="1" capture-group="4" pattern-id="StringPattern3" field="DestinationIp"/>
     <matcher order="1" capture-group="5" pattern-id="StringPattern3" field="DestinationPort"/>
     <matcher ext-data="dd MMM YYYY hh:mm:ss" order="1" capture-group="1" pattern-id="StringPattern4" field="DeviceTime"/>
     <matcher order="1" capture-group="2" pattern-id="StringPattern4" field="EventName"/>
     <matcher order="1" capture-group="3" pattern-id="StringPattern4" field="SourceIp"/>
     <matcher ext-data="dd MMM YYYY hh:mm:ss" order="1" capture-group="1" pattern-id="StringPattern5" field="DeviceTime"/>
     <matcher order="1" capture-group="2" pattern-id="StringPattern5" field="SourceIp"/>
     <matcher order="1" capture-group="3" pattern-id="StringPattern5" field="EventName"/>
     <matcher ext-data="dd MMM YYYY hh:mm:ss" order="1" capture-group="1" pattern-id="StringPattern6" field="DeviceTime"/>
     <matcher order="1" capture-group="2" pattern-id="StringPattern6" field="EventName"/>
     <matcher order="1" capture-group="3" pattern-id="StringPattern6" field="Protocol"/>
     <matcher order="1" capture-group="4" pattern-id="StringPattern6" field="DestinationPort"/>
      <event-match-multiple device-event-category="unknown" capture-group-index="1" pattern-id="allEventNames"/>
  </match-group>



Comment

People who like this

  0   Share
10 |3000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
50RT19P272 gravatar image

Answer by 09h33 (28) | Feb 21, 2017 at 03:58 PM

... but there is still something else. I'll do some more testing

Comment

People who like this

  0   Share
10 |3000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster

Follow this question

104 people are following this question.

Answers

Answers & comments

Related questions

Announcement: QVM Externally Hosted Scans (March 1st - power outtage) 0 Answers

ADFS(Active Directory Federation Services) Events not parsing in QRadar 2 Answers

QRadar parsers for SAP modules 0 Answers

Parsing issue with trendmicro deep secuiry 4 Answers

Qradar Linux firewalld DSM 4 Answers

  • Contact
  • Privacy
  • IBM Developer Terms of use
  • Accessibility
  • Report Abuse
  • Cookie Preferences

Powered by AnswerHub

Authentication check. Please ignore.
  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Spaces
  • API Connect
  • Analytic Hybrid Cloud Core
  • Application Performance Management
  • Appsecdev
  • BPM
  • Blockchain
  • Business Transaction Intelligence
  • CAPI
  • CAPI SNAP
  • CICS
  • Cloud Analytics
  • Cloud Automation
  • Cloud Object Storage
  • Cloud marketplace
  • Collaboration
  • Content Services (ECM)
  • Continuous Testing
  • Courses
  • Customer Experience Analytics
  • DB2 LUW
  • Data and AI
  • DataPower
  • Decision Optimization
  • DevOps Build
  • DevOps Services
  • Developers IBM MX
  • Digital Commerce
  • Digital Experience
  • Finance
  • Global Entrepreneur Program
  • Hadoop
  • Hybrid Cloud Core
  • Hyper Protect
  • IBM Cloud platform
  • IBM Design
  • IBM Forms Experience Builder
  • IBM Maximo Developer
  • IBM StoredIQ
  • IBM StoredIQ-Cartridges
  • IIDR
  • ITOA
  • InformationServer
  • Integration Bus
  • Internet of Things
  • Kenexa
  • Linux on Power
  • LinuxONE
  • MDM
  • Mainframe
  • Messaging
  • Node.js
  • ODM
  • Open
  • PartnerWorld Developer Support
  • PowerAI
  • PowerVC
  • Predictive Analytics
  • Product Insights
  • PureData for Analytics
  • Push
  • QRadar App Development
  • Run Book Automation
  • Search Insights
  • Security Core
  • Storage
  • Storage Core
  • Streamsdev
  • Supply Chain Business Network
  • Supply Chain Insights
  • Swift
  • UBX Capture
  • Universal Behavior Exchange
  • UrbanCode
  • WASdev
  • WSRR
  • Watson
  • Watson Campaign Automation
  • Watson Content Hub
  • Watson Marketing Insights
  • dW Answers Help
  • dW Premium
  • developerWorks Sandbox
  • developerWorks Team
  • Watson Health
  • More
  • Tags
  • Questions
  • Users
  • Badges