In-band Network Telemetry—More Insight into the Network

By Kingshuk Mandal |

Overview

In an information-based and service-based economy, the demand for 'always accessible services' is ever increasing. Momentary non-availability of a network is a significant concern for any modern data center. Hence, network monitoring is of paramount importance and acts as a first line of defense to ensure non-disruptive and optimized network functions. Traditionally for network monitoring, operators put a tap in the network and collect packets to monitor. But this structure comes with limitations that include the number of taps you can have in the network, whether data center east-west traffic can flow through the taps, and the inability to track which path a packet has traversed or how well a packet was treated by each switch in the path.

This is where in-band network telemetry (INT) provides a much-needed solution. INT provides the abstractions that enable each data packet to enquire any traversing switch about its internal data state like egress queue occupancy at that point of time, identification of the switches the packet is going through, outgoing link used, and packet processing latency of the switch. This information gets recorded into the data packet itself as it traverses through the network. Because the internal states of the switches are recorded into the data packet, this mechanism is called in-band telemetry.

Telemetry 1

Figure 1: A telemetry-capable network

Referring to the preceding image, a telemetry-capable network has a few components. The first one is a telemetry source. This is a trusted network entity that creates and inserts telemetry headers into the packets it sends. The telemetry headers contain, at a minimum, instruction indicating what to collect. Every telemetry transit hop on the path of the packet is a network node that appends its telemetry data into the telemetry header. In the preceding image, router 1 is the telemetry source for the red and green flow. Router 2 is the telemetry transit hop. Finally, a telemetry sink is the network node that extracts the telemetry headers and collects the path state contained in the headers. The telemetry sink is responsible for removing telemetry headers to make telemetry transparent to upper layers or to the end user. A telemetry sink may send the collected metadata to a central analytic engine to extract actionable insight. In the preceding image, node 4 acts as the telemetry sink.

In any active network, a transient micro-loop can occur for a very short period because of router failure or the network taking time to converge to use a new route. A telemetry-capable network can give an indication for these micro-loops, which traditional flow monitoring may fail to detect. In the telemetry recording, you may see a flow was reaching the destination with N number of telemetry records with node ID(s). Then, for a short duration, a few packets may come with N+P number of telemetry records with same node ID repeated. This indicates a transient micro-loop occurred for a short period. The analytic engine can point out where and when the micro-loop occurred.

Now, let’s discuss in detail some practical applications of data center telemetry. Managing a large data center network is a very daunting task because of traffic volume and complex networking device software stacks. Network faults are unavoidable and diverse in nature, such as packet drops, latency spikes, low throughput, and micro-loops. The existing tools to troubleshoot such scenarios—like ping, traceroute, and device counters—are insufficient.

Let’s study two example scenarios and compare how traditional monitoring tools fail to debug such scenarios and how in-band telemetry does it with precision without much overhead to the network.

Case Study 1: Intermittent and Silent Packet Drop

Often, applications cannot reach the servers, resulting in a timeout that is followed by multiple retries. Shortly after, one of the reties becomes successful and the application starts functioning. This intermittent packet drop may not have a visible impact at the surface, but if ignored, it may aggravate and lead to network failures at a future point of time. So, it’s important as well as tricky to detect where exactly the packets were getting dropped silently in the network.

Telemetry 2

Figure 2: Intermittent and silent packet drops are difficult to find, but often lead to future failure points

Debugging using Existing Tools like Counters and Trace Route

Debugging Intermittent Packet Drop Using In-band Telemetry

In-band network telemetry provides greater visibility into the network, enabling the network operator to be proactive in preventing disruptions.

Case Study 2: Micro Bursts caused by Network Congestion

When there is congestion in the network, the network operator needs to find out the node that has surpassed the latency watermark. For example, if baseline latency in a switch is a few hundred nanoseconds, and then suddenly rises to micro seconds, there is most-likely congestion in the network. When this happens, you’ll need to know the nodes that contributed to the congestion. Which application is the aggressor flow that created the micro bursts and which other application flows are suffering because of this? For example, in Figure 3, the red link between the top of rack (TOR) and spine switch has become congested. The network monitor needs to pinpoint these congestion points with precision.

Telemetry 3

Figure 3: Pinpointing congestion points with precision

Debugging Using Existing Tools (Device Counters, Ping)

Debugging Using In-Band Telemetry

Telemetry Test Scenario using IxNetwork

Before deployment, it is important to thoroughly test and measure the efficiency of a network node implementing telemetry functions. Keysight's IxNetwork is a test solution that generates different types of telemetry data packets that can be configured and transmitted at line rate. IxNetwork currently supports the following encapsulation types with INT headers and metadata:

The IxNetwork test tool can simulate the following INT use cases to validate an INT-capable device under test (DUT):

As the data packet with INT info traverses through the network, every INT transit node pushes its own INT information into the data traffic. The DUT, as shown in Figure 4, is playing the role of an INT transit hop. After receiving a data packet with INT header, it sees the 'Instruction Bit Map' to decide the metadata information that needs to be recorded, and then pushes it on top of the INT stack of metadata accordingly. It must also calculate and fill up the transport protocol’s payload length and INT length in the data packet. It is important to verify whether the DUT can do all these flawlessly under different load condition.

Telemetry 4


Figure 4: Validating a device’s ability to append INT metadata

IxNetwork offers functionality to configure and send data traffic with multi-node INT metadata of different types. After receiving the packet, the DUT inserts its own metadata and forwards it towards the INT sink emulated by another IxNetwork port. The INT header conformance can be validated from the packet capture by IxNetwork (refer to Figure 4).

As an INT transit node, the DUT must do the additional work of inserting INT information in each data packet at a certain rate. This is bound to have some processing overhead that may impact the overall network throughput. This impact can be assessed by using the IxNetwork INT solution. To test this use case, INT should be enabled on the DUT. The user can send traffic with INT instruction mapped for the required metadata insertion, like hop latency, queue occupancy, or ingress/egress port ID (refer to Figure 4). Traffic throughput and latency/jitter should be measured. Repeat the test with INT disabled on the DUT. Now, send the same data traffic without the INT information. Measure the throughput and latency/jitter. Comparing the results with and without INT headers, you can get the actual impact on throughput when the DUT is doing INT processing.

When the DUT is acting as the INT sink, it needs to strip off the INT metadata info from the received data packets and then forward it to the destination of the data traffic. To simulate this test, send INT traffic from the IxNetwork INT Source port as shown in Figure 5. Then capture the traffic on the IxNetwork destination port on the other side of the DUT and verify that the received traffic does not have any INT header/metadata in it.

Telemetry 5

Figure 5: Validating in-band network telemetry sink

Easy Configuration of Telemetry Header/Metadata in IxNetwork

IxNetwork provides full flexibility to craft telemetry packets by using the visual editor with build-in INT template. Users can easily set the instruction bitmap as per the test need, for example, in Figure 6, Switch ID and Hop Latency are enabled. Then optionally add multiple metadata stacks corresponding to multiple hops simulated by the INT ingress port. In Figure 6, three hops are added.

Telemetry 6

Figure 6: Configuring telemetry header/metadata in IxNetwork

IxNetwork also provides the capability to analyze the packets at the bits/bytes level. This gives your visibility to check whether the DUT has inserted its own record according to the Instruction Bitmap set in the INT header.

Telemetry 7

Figure 7: With IxNetwork, you can analyze packets at the bits/bytes level

Conclusion

In-band network telemetry is a game-changer for understanding the treatment of packets as they traverse a network. The information recorded into the data packet itself helps troubleshoot potential problems even before they impact applications. But validation is critical to success in such systems to ensure proper configuration and high performance across the network.

You’ll need pre-deployment validation of a device implementing in-band network telemetry from both the conformance and performance perspectives. Keysight's IxNetwork provides a feature-rich INT test solution that enables user to execute pre-deployment validation. Although specific use-case scenarios have been depicted herein for INT over UDP traffic, IxNetwork also supports similar use cases for INT over Geneve traffic. IxNetwork’s INT solution is designed based on the May 2018 INT draft.

For more details, see our IxNetwork Telemetry emulation demo.

limit
3