Technote – QRadar: Reverse Flow Direction – http://www-01.ibm.com/support/docview.wss?uid=swg21972754
What is a flow?
In QRadar’s terms, a flow represents a report, generated/updated minute by minute, of a session between two endpoints connected to network. Rather than the concept of bytes & packets, which flow from 1 host, to the other, and back, the concept of a flow represents the entire session, a count of the bytes and packets generated in the communication, the flags, protocol used, and the time that it is active.
What identifies or defines a unique flow?
QRadar identifies a unique flow based on 5 properties (5 tuple) – source ip, source port, destination ip, destination port, and protocol. These 5 properties are used to uniquely identify each “flow” or session in QRadar.
How do different flow sources compare?
QRadar supports multiple sources of flow data, including our own “QRadar Flow Collectors”. These devices are normally designed to listen to packet data and are commonly referred to as “internal flow sources”. Packet data coming from a network tap device, or a span or mirror port enabled on a physical network switch is monitored by the flow collector, the output of which is often referred to as “qflows” or “flow data”. One of the benefits of these “packet” sources is that it allows for layer 7 packet analysis, which can help identify the type of application directly.
Other sources are often referred to as “external flow sources”, and are comprised of well known protocols in network monitoring, namely Cisco’s Netflow, IPFix, Juniper’s JFlow, the open standard sFlow & Packeteer data formats. These data formats for the most part, do not include payload. However, since they are often coming from border routers, can provide a different level of visibility. For example, Netflow records can provide us with both the router interface that the packets crossed, as well as the ASN record numbers of the originating network. Also, when using “IPFIX”, additional fields that are not parsed into normalized fields, can be placed into the payload area of the data, as name=value pairs, and be used as custom properties, as required.
Flow Time Stamps
Flow timestamps are created as traffic are detected and recored by the QRadar Flow Collector. Some flows can last seconds, ie, an email message, file upload, etc, while others may far longer – minutes, hours, or even days, such as an interactive remote session, audio/video stream (Netflix, voip call), or database <> application connection. As these sessions/flows continue over time, they are reported into the system. The original “start time” for each session remains the same, when first detected, while the “last packet time” will update as time passes. The best way to see this is to search for the two ip addresses involved in the session/flow, then search over a longer time window – you should see multiple records, one that ends for each minute that the session was active. Each minute will also have the byte & packet count, for each minute the flow was active.
Netflow sourced timestamps
For timestamps on flows generated from external netflow data, these start times can even precede the start/restart time, and even the installation date, of a QRadar flow collector. Netflow sources include 3 time values, in addition to the “time interval” that the flow is reported in. These 3 additional time values are as follows (see attached screen shot for an example):
- System up time (SUT) – number of milliseconds the system (ie router) has been up ( SysUptime )
- Flow Start time (FST)- value of sysuptime when the flow started ( StartTime )
- Last Packet Time (LPT) – value of sysuptime when the last packet was received ( EndTime )
With these 3 time values, the qflow process can then calculate the original start time by subtracting “FST” from “SUT” to get the “seconds ago” value. That is then used in conjuction with the local flow collector time, to calculate a previous start time, which could be days ago for a long running, ongoing application connection.
- “start time delta” = SUT – FST
- qradar start time = current time (on qflow device) – “start time delta”.
- QFlow can also apply a similar calculation for last packet time.
Note, In some user sites, we have seen instances where the “system uptime” is not being updated properly by the external device, and even occasionally the system uptime/time interval is reported as zero. When this occurs, the time can be miscalculated, and even show up in QRadar’s Network Activity area with a start time of “December 31, 1969”, effectively reporting with an “EPOCH 0” value. To diagnose and investigate such an issue, you may want to capture some of the netflow packets arriving at the QRadar flow collector, and review them in a third party application such as wireshark, to review what time values are being reported in the netflow stream.
Note – The “Storage Time” timestamp refers to when the data is written to disk on the QRadar Processor. This would also ‘map” to the time period selected when you create your search.
What is Flow Direction & Flow Bias?
Flow Direction is used to represent the direction of the communication, between two ip addresses.
In the Flow Collector configuration options, the setting “Use Common Destination Port = Yes / No” is used to determine how direction is analyzed, regardless of the flow source type – raw packets via a network card & napatech interface, or an external flow source, such as a router:
- When set to “Yes“, the ports in use by the flow are analyzed, and based on a list of commonly used ports and the port rules below, any flow that is deemed to be “backwards”, has the direction of the flow reversed.
- When set to “No“, the flow is reported, as it detected, be that from a packet based source, such as a span, mirror or tap, or from an external source, such as router via netflow, etc.
When “Use Common Destination Port” is enabled (set to “Yes”), the source and destination port are checked against common destination ports as defined in the QRadar signature set, and the following logic. If a match is found, and the ports are deemed backwards, the direction is reversed, swapping the source and destination.
- If the destination port is NOT a common destination port, then reverse the flow direction, IF:
- the source port is a common destination port OR
- the source port is less than 1024 and the destination port is greater than 1024
- If the destination port IS a common destination port, then reverse the flow direction, IF:
- the source port is a common destination port, AND
- the source port is less than 1024 and the destination port is greater than 1024
- In the absence of any common destination ports, QRadar will determine the direction of the flow based on the arrival of the data
- Only flow data from Packeteer devices, which determines it’s own direction will override this logic.
- These rules are only applied to TCP and UDP flows
- These rules are only applied to flows on flow sources that are NOT marked as asymmetric in the flow source configuration area.
- Byte & packet counts, tcp flags and payloads are NOT used to determine flow direction.
- These rules apply to both packet based sources (span/monitor ports, taps) and external flow sources (netflow, jflow, etc). .
- Flows that are “unidirectional“, ie, do not have both source & destination byte & packet counts, are never adjusted by this logic, and are reported as seen from the original source.
With packet sources (span, mirror, tap), flow direction is normally very reliable, as it is reported directly as detected by the QRadar flow collector. Exceptions to this that are incorrectly detected, can include a) long running sessions, such as application <> database connection, that were active prior to the qflow collector start up, or b) sessions that were active a qradar service restart, such as a full deploy. If you are using a packet source that seems to only be reporting one side of the traffic, verify the configuration of the source device.
With external flow sources, such as netflow, these sources often are limited to reporting the ingress or egress traffic on specific router interfaces, meaning that they only report “one side” of active network communications. With these sources, QRadar can often have an issue properly determining flow direction.
One sided traffic can occur under normal circumstances as well, if there’s a network scan or denial of service attack, that is blocked by a firewall, but the QRadar Flow collector is outside the firewall.
Impacts of Flow Direction
- Flows that are detected with only source source or dest bytes, i.e, “one sided” (commonly referred to as inbound or outbound only, or unidrectional) can show up with what looks like a backwards direction. These can trigger the default DOS, Scanner & Attack rules and create offenses.
- If the sources of flow information are going to different QRadar Flow collectors, or are in separate locations on the network, you may see “both sides” of a communication, but as distinct records under Network Activity.
- superflows can be created prematurely, when both sides of a communication exist, but end up being collected in different intervals and/or on different flow collectors.
When you see this, take note of the flow source & interface, to ensure they are going to the same qflow collector. If not, consider reconfiguring the external sources to go to the same qradar system
Mitigating Unidirectional Flows to reduce direction issues.
There are ways to reduce the possibility and impact of flow direction & unidirectional flows in QRadar:
Consolidate external sources on the same QRadar aappliance
- Both QRadar Flow Collectors and Flow Processors have the ability to attempt to put flows back together. The earlier this can be done in the pipeline, the better.
- Sending all external sources to the same Collector/Processor, increases the probability of getting flows from distinct flow sources/routers, properly assembled.
- If your environment is too geographically separated, or the amount of data is too large, have such a large number of netflow records, that this is not feasible, attempt to send systems in the same region to the same qradar system.
Increase the flow session reporting interval on your network devices
- increase the flow session reporting frequency
- Netflow based reporting works on an interval timer, where flow session information is reported on that timer.
- If you can increase the frequency of this reporting, it should allow for more frequent reporting of all active communications.
- If netflow sources are on different reporting frequency rates, this can increase the possibility of a flow collector only seeing 1 side of the communication.
QRadar configuration changes
Flow Collectors – Flow Carry Over Window
- settings are at: admin / system & license management / choose host / deployment actions / edit host / component management / flow collector >
- This option on the flow collector components in QRadar, tells the qflow process, that any flows it has in the last N seconds of the minute, that are seen as 1 sided, hold those until the next interval, in order to locate the other side of the session, in the next minute.
- Recommended max value is around 6 seconds for this value.
Flow Collectors – Use Common Destination Port
- Settings are at: admin / system & license management / choose host / deployment actions / edit host / component management / flow collector >
- This setting configures the flow collector, that when a “reversed” flow is seen, such as a server responding to a client workstation (port 80 talking to 23245), that this session would be reversed.
- This could have an impact on properly re-assembling flows as well, and setting this to “No” (disabled), may allow the pipeline to put data back together further on in processing.
- Adjust the qflow properties option “remove duplicates = no”. If this setting is set to “Yes”, qflow may drop/ignore the second part of the flow when it comes from the other routers
Flow Sources – Enable Asymmetric Flows
- settings are at: admin / flow sources / edit each flow source.
- This setting is only required when you have multiple netflow data sources going to -multiple- flow collectors in your deployment – dedicated collectors, or directly to flow processors.
- When QFlow sees a 1 sided session, by default, it may consider that to be a legitimate set of information, such as a DDoS, network scan, or port scan. This means that 1 sided sessions may be eligible for coalescing into “super flows”. These superflows effectively can “hide” these 1 sided sessions from being recombined with their corresponding other records.
- To avoid 2 sides of a communication, going to 2 different qflow instances, from being hidden in this way, you can mark those QRadar flow sources as “Asymmetric”. Note however, to do so, means that these sources will no longer create superflows for valid DDoS, Network & Port scans, which can increase the load on your flow processor environment.
Flow Bias looks at the amount of data being sent in each direction, and how it’s balanced to the local & remote system. Here are the 5 levels of “bias” that qradar will identify, the percentages, and a few sample applications that may show up each bias type
- in only
- packets and bytes from the remote side only, no response from local, internal address.
- This could be some remote host, trying to hit address on your network that doesn’t respond , doesn’t exist, or is blocked at a firewall. A network scan could look like this.
- out only
- similar to in only, however a local source address attempting to reach a remote address, but is being blocked or the remote host is unresponsive.
- A common example of this, would be someone on a workstation, that is attempting to reach a port 25 sendmail/smtp server on the internet, but your corporate firewall is blocking it.
- near same
- in/outbound byte counts are closer to equal, where more >30% in one direction, and <70% in the other direction.
- A common example of ‘near same’ communications, could be an instant messaging application, VOIP conversation, or interactive session, such as RDP, ssh, etc.
- mostly out
- any bidirectional flow, with more than 70% of the total bytes outbound to your network
- A example of a mostly-out communication, would be a local web or application server that provides files to end users. A user pulling down a file attachment from a web application would look like a “mostly out” flow.
- mostly in
- any bidirectional flow with more than 70% of the total bytes inbound to your network
- An example of mostly-in communication, would be a end user workstation, connecting to the windows auto-update services, and those update files are pulled down from the autoupdate site for local installation.
- If neither the source or destination ip address are defined in your network, then we classify this as “other” traffic. The reason being, we cannot determine a direction, in or out, when this occurs. For this reason, we cannot assign a bias to the session.
For additional information on flows, please check this Knowledgebase Article on flows. If you have any additional questions, feel free to enter them below, and I’ll expand them in this post.