Your Special Agent: The Importance of NetFlow Reliability
Network Security, Monitoring and Analysis tools can be classified in many ways, including by the way they ingest traffic – some can ingest raw data, a.k.a. the full packet streams, others can ingest a stripped-down version of the raw data. Often, this stripped-down data is called metadata and only contains the most basic parts of the information about the raw traffic, which are essential for the tools to perform their job. Many tools would be overwhelmed by the sheer bandwidth and processing power needed to ingest raw data and have “adapted” to live off metadata just fine.
NetFlow has been an industry standard feature for generating, exporting and ingesting metadata, ever since it was developed by Cisco and adopted by most networking vendors. NetFlow version 9 and version 10, which is also called IPFIX, are the most popular implementations, and they are used by a significant number of forensics, compliance, SIEM and monitoring tools, and even for re-building real network traffic patterns, such as in the case of Keysight’s TrafficREWIND.
In many ways, NetFlow acts like a Special Agent: it stays on your network stealthily and quietly, without leaving any trace or being noticed, and informs you about everything that moves.
NetFlow v9 uses a list of standard, fixed-size flow fields, while IPFIX made it easy to introduce custom, variable-size fields, meaning that anyone could export any information they wanted about the traffic flows, with virtually no limitation. Keysight leveraged on this via AppStack’s IxFlow, which is a v10 implementation with added support for 130+, and counting, custom flow fields, in addition to the standard ones.
Those specialized tools I mentioned above rely solely on metadata information to perform critical tasks such as monitoring QoS and network health or detecting breaches and attacks. If the metadata exported to these tools is incomplete or plain wrong, then the tool’s job is compromised or severely inefficient; as they say, “garbage in – garbage out”.
In their latest report, the Tolly Group evaluates Keysight Vision X on the most common features, including NetFlow implementation, and checks for performance and accuracy. You can read this testing report here: Network Packet Broker Performance & Features: Keysight Vision X.
So, what do the NetFlow/IxFlow results mean?
In order to have accurate NetFlow records, it’s necessary to have protocols and applications identified accurately. Vision X complies with flying colors, both under light and high load.
Consequently, 100% of the filtered application traffic arrives at the NetFlow engine on Vision X.
The immediate effect of this behavior is that Vision X exports NetFlow and IxFlow records for 100% of the application traffic, which is ideal due to the following reasons:
- Tools receive full information about the network. They know the reality, for example that there are 100 active Facebook users, 50 Microsoft Office users and 20 users watching YouTube, with each user’s individual bandwidth, and a total bandwidth of 1 Gbps. Other NetFlow implementations might send just fractions of the real traffic metadata, which can easily miss that important needle in the haystack, such as that one user who hogged the bandwidth for 10 other users.
- Traffic reconstruction from the NetFlow records is reflecting the realistic mix on the production network. Thus, using reconstruction tools to test the network yields extremely relevant results.
- Suspicious and malicious apps and activities are easily seen in the large amount of “known good” traffic, such as that one user who accessed a malware server. Attacks and breaches are detected in due time, preventing data and financial loss.
Next, let’s look at performance. The table below summarizes the rounded maximum rate at which Vision X AppStack can generate and export flow records, reliably, without loss or packet drops. The results are given per CPU, for bi-directional flows, and the units of measurement are Connections Per Second (CPS) for TCP and Frames Per Second (FPS) for UDP. Note that up to 8 CPUs can be plugged into one Vision X chassis to be used simultaneously, therefore increasing performance 8-fold.\
Reliability isn’t only about generating flow records, but also about transporting the exported flows. Most visibility network links are reliable, so NetFlow can easily be transported over User Datagram Protocol (UDP), which is the most common practice. However, when the transport network is unreliable, reliable methods and protocols are needed. Luckily, Vision X can export flows over Stream Control Transmission Protocol (SCTP), a reliable, message-oriented, congestion-aware protocol, which also happens to be good with jumbo frames containing lots of information fields.
Let’s take the example of UDP and see what the actual export looks like on the wire:
When transmitting on the wire, Keysight metadata exporter sends standard length, non-fragmented frames. When it comes to visibility and security metadata, I’m a big fan of non-fragmentation, because of two reasons:
- If one of the IP fragments is lost in transit, the original jumbo packet cannot be re-assembled, and the info is lost or delayed. This is particularly annoying with connectionless protocols such as UDP which have no way to re-transmit failed packets.
- Collector tools are freed from the task of fragment re-assembly. Some tools might not support re-assembly or might have performance penalties, you never know so it’s best to stay away from fragmenting towards tools altogether.
Conclusion
Metadata accuracy and reliability under pressure is essential for a company’s network monitoring and security posture.
Vision X has demonstrated the ability to export accurate and rich flow records with high-performance identification of application traffic. The benefits on business range from meeting QoS/KPI targets and satisfied customers, to fast security response and regulatory compliance.