Which latency are you actually measuring for AI fabric
In AI data center networks, two metrics both go by "latency"—yet they measure fundamentally different things. Confusing them leads to systematic mischaracterization of Fabric performance.
The traditional transport latency model
Traditional IP/Ethernet is stateless forwarding. One packet in, one packet out. RFC 2544 and TWAMP measure exactly this: the time for a single packet to travel from sender to receiver. On cut-through switches, this typically sits in the tens to hundreds of nanoseconds range.
This is a forwarding-plane metric. The switch maintains no connection state, requires no "readiness" from the peer, and generates no completion signal—the packet arrives, and that's it.
RoCE is stateful transport: the unit of latency is a verb operation
RoCE is not packet forwarding. An application submits a Work Request (WR) to the NIC via libibverbs and waits for a Completion Queue Entry (CQE) confirming the operation completed. The NIC maintains a full QP (Queue Pair) state machine throughout.
The two-sided SEND/RECV verb most clearly exposes this state machine overhead.
The sender posts a SEND WR to its Send Queue (SQ), and the NIC transmits the data. When the receiver NIC gets the packet, it must find a pre-posted RECV WR in its Receive Queue (RQ). If the RQ is empty, the NIC returns an RNR (Receiver Not Ready) NACK, and the sender waits for the RNR timeout before retransmitting.
The full flow requires coordinated QP state machine execution on both ends:
Sender: Fetch WQE from SQ → Packetize → Transmit
↓ Network transit
Receiver: Match RECV WQE in RQ → DMA write to pre-registered buffer → Generate RECV CQE → ACK
↓ ACK return
Sender: Receive ACK → Generate SEND CQE → Application sees send completion
The latency the application observes spans from ibv_post_send to when poll_cq returns the SEND CQE: 2× network latency + both-side NIC state machine processing + DMA + ACK return path.
RDMA WRITE/READ/ATOMIC are one-sided operations—the remote CPU is not involved—but they still depend on the NIC state machine to complete reliable delivery and generate a CQE. Verb RTT still covers the full round trip.
The numbers
ib_write_lat on a back-to-back 400GbE link typically shows a WRITE verb RTT of 5–10 µs for 128KB messages—of which network transmitting contributes ~2.6 µs (128KB ÷ 400Gbps, one-way). The Ethernet cut-through forwarding latency on the same link is ~300–500 ns.
That's a 10–30× gap. The difference does not come from the network itself, but from the NIC completing the full verb state machine: serialization, DMA, remote-side matching, ACK return, local CQE generation.
The real impact on AI Fabric validation
AI workload end-to-end latency is governed by verb RTT, not switch forwarding latency. Using RFC 2544 or TWAMP numbers to evaluate RoCE Fabric latency performance produces systematically wrong conclusions—a switch with extremely low per-packet latency, paired with a slower NIC, may actually yield higher verb RTT.
Validating AI Fabric latency targets requires measurement at the RDMA verb layer: ib_write_lat, ib_read_lat, or equivalent NCCL benchmarks.
Evaluating verb RTT with AI workload emulation
An isolated ib_write_lat run (back-to-back, single flow) captures only the best-case verb RTT. Production verb RTT depends on three dimensions—each addressed by Keysight AI Data Center Builder (KAI DC Builder):
1. Real workload complexity. Production training clusters don't run point-to-point WRITE. They run 3D and expert parallelism layered together, with multiple AllReduce and All-to-All operations concurrently contending for the same switch ports. KAI DC Builder generates real RDMA flows following NCCL ring topology and message scheduling, indistinguishable from actual training traffic at the switch level. Verb RTT measured under this concurrent load reflects what NCCL jobs actually experience—not the minimum from an unloaded direct connection.
2. Congestion control parameters sweep under stress. Verb RTT is highly sensitive to congestion control settings. Under KAI DC Builder's production-grade traffic, you can systematically sweep ECN thresholds and DCQCN Kmin/Kmax values, directly observing each configuration's effect on verb RTT distribution (P50/P99)—a congestion dimension that single-flow ib_write_lat testing cannot provide.
3. Separating network-side from NIC-side overhead. KAI DC Builder measures verb RTT from the network side: pure network round-trip time from WRITE request transmission to ACK return. Comparing this against host-side ib_write_lat, the delta quantifies the additional overhead introduced by the NIC state machine—WQE fetch, DMA, CQE generation, PCIe transit—isolating the NIC's contribution to application-visible latency from the network's.
Then, do you know which latency you are actually measuring now?