From Single Collective to Epoch-Aware Testing: Why Iteration Semantics Matter for AI Fabrics
Motivation
Standard nccl-tests and fabric acceptance tests measure a single collective from a clean state: empty switch buffers, no PFC pause credits consumed, no ECN history. This is not how production of AI training operates.
A training job runs thousands of collective iterations. Each iteration inherits the fabric state — queue occupancy, congestion marks, PFC credit state — left by the previous one. A fabric that achieves 90 GB/s on iteration 1 may deliver 60 GB/s by iteration 50 if congestion builds unchecked or may never converge if a load-balancing mechanism stops functioning under back-to-back load.
Iterated collective testing measures what matters for training: sustained, steady-state collective performance and stability of that performance across iterations.
Without this, networks can appear performant in benchmarks while underperforming in real training.
Why iterative testing beats single-run tests
The fundamental problem with single-run tests is not measuring noise, but a mismatch between test semantics and training semantics.
- Single runs lie: They start from empty buffers and capture only transient peaks. Real training inherits congestion from previous iterations.
- Convergence matters: Only iterative tests show when bus bandwidth stabilizes—and how noisy it is before that.
- Variance is the signal: Bad LB designs cause iteration-to-iteration oscillation (up to 1.5×). P99 > average.
- Congestion needs time: PFC, DCQCN throttling, ECN buildup surface only after many iterations.
- Flowlet LB can silently fail: Back‑to‑back collectives remove gaps, flowlet timeout never fires, paths never rebalance.
- Statistics matter: N=100 gives confidence. N=1 is anecdotal.
How to test with iterations
we moved from one‑off collectives to iteration‑ and epoch‑level execution. That means being explicit about parameters that shape steady‑state behavior, not just peak throughput:
- Total iterations (N) to observe convergence vs. stress conditions
- Warmup (W) to filter out cold‑start artifacts
- Fixed message size to avoid confounding variables
- Inter‑iteration gap calibrated to real compute time
- Collective type and ranks aligned with the target system configuration
This setup lets us reason for stability, variance, and convergence, not just best‑case numbers.
Mapping to Training Epochs
A training step corresponds to one collective iteration in distributed training.
Each step consists of the following phases:
Therefore:
1 iteration = 1 distributed training step = 1 collective instance
1 epoch = M steps = M collective iterations
where M = dataset_size / global_batch_size
At the end of each epoch, two additional events occur:
- Checkpoint AllGather:
Model checkpoints are saved, triggering a large‑scale AllGather burst to collect model weights across GPUs. - Data loading pause:
A prolonged idle period (typically 5–30 seconds, much longer than a normal compute interval) occurs while the next epoch’s data is prepared. This pause effectively resets load‑balancing (LB) state and congestion state in the network.
As a result, cold‑start / congestion initialization is re‑triggered at the beginning of each new epoch, leading to a temporary drop in bus bandwidth (busbw) before performance recovers.
Experimental Results from a Real Testbed
With this setup in place, we now examine results from a real testbed to understand how these effects manifest under realistic training conditions.
All parameters are fixed across experiments unless explicitly stated; values are chosen to reflect realistic training workloads rather than to optimize peak performance.
This result shows how the new traffic pattern drives congestion, with burst accumulation leading to sustained buffering—clearly visible as an ECN‑CE build‑up.
This indicates persistent queue buildup rather than isolated transient congestion, which directly affects collective tail latency.
- Stable steady‑state bandwidth
For most iterations, the bus bandwidth remains close to a stable plateau, indicating that the system has converged to a steady-state under iterated collectives. - Transient congestion events
Periodic spikes in ECN‑CE RX are visible. These spikes correspond to short-lived congestion events, during which the network signals congestion via ECN marking.
By examining 100 iterations within a single epoch, start‑to‑start gaps reveal oscillations, exposing long‑tail latency at the per‑collective level—showing that individual collectives do not maintain perfectly stable performance.
Importantly, this oscillation is visible within a single epoch, showing that even when average busbw is stable, individual collectives experience long-tail delays.
Scope and non-goals
This work focuses on testing methodology, not algorithm comparison. Congestion control, topology, and message size are intentionally fixed to isolate iteration and epoch semantics. Comparative evaluations across different CC or LB designs are deferred as future work.
Wrapping Up
What this study makes clear is that real training behavior only emerges over time. Looking at a single collective can show peak capability, but it hides the dynamics that matter in practice. Once we move to iterated execution and full epoch emulation, a very different picture appears—steady‑state convergence, congestion accumulation, ECN build‑up, and long‑tail latency at the level of individual collectives.
Epoch boundaries add another layer to this story. Checkpoint bursts and long data‑loading gaps reset network state, triggering brief performance drops followed by re‑convergence. These effects are short‑lived, but they happen repeatedly during training and shape the overall efficiency seen by users.
The takeaway is simple: designing and validating AI networks requires an epoch‑aware view, not just isolated microbenchmarks. Stability, variance, and convergence behavior are just as important as peak throughput when it comes to real‑world distributed training.
Peak throughput alone is not a sufficient metric for AI network validation; iteration-aware and epoch-aware behavior must be treated as first-class signals.
