Validating AI Fabric Without GPUs: NCCL Emulation in Practice
What is NCCL
NCCL (NVIDIA Collective Communications Library) is a high-performance communication library optimized for GPU-to-GPU data exchange, providing efficient collective operations (e.g., AllReduce, AllToAll) in multi-GPU and multi-node systems.
nccl-tests is a benchmarking and validation suite for NCCL that measures the performance and correctness of its collective operations across different GPU and network configurations.
GPU for testing fabric?
You don’t need GPUs to validate an AI fabric. What you need is the traffic pattern they generate.
In large-scale AI deployments, the network is often built and brought up before the GPU cluster is available. This creates a gap: the fabric must be validated, but the workloads it is designed for cannot yet run.
Waiting for GPUs delays validation and pushes risk into the final stages of deployment.
We built Keysight AI Data Center Builder (KAI DC Builder) to remove that dependency.
The key observation
Collective communication workloads such as All Reduce are highly structured:
- Deterministic communication patterns (e.g., ring, tree)
- Fixed message sizes per step
- Tight synchronization across ranks
- RDMA-based transport behavior
At the network level, these workloads translate into predictable traffic patterns that are largely independent of GPU compute.
This makes it possible to validate the fabric without executing GPU kernels.
Emulating NCCL without GPUs
When running NCCL All Reduce, GPUs generate coordinated RDMA flows across hosts according to a specific communication schedule.
KAI DC Builder reproduces this schedule directly on the network.
Instead of simulating GPU execution, it generates equivalent RDMA flow characteristics:
- Same message sizes
- Same concurrency across ranks
- Same communication topology (e.g., ring)
- Similar timing and synchronization behavior
From the fabric’s perspective, the resulting traffic is indistinguishable from a real NCCL workload.
This allows us to measure the same metrics used in NCCL tests, including bus bandwidth.
Validation against real hardware
To validate the approach, we compare KAI DC Builder emulation against NCCL measurements on GPUs under equivalent conditions:
- Same number of ranks and hosts
- Same topology and link speed (400 Gbps)
- Same collective algorithm (bidirectional ring)
- Same message size sweep
KAI DCB emulation result
nccl-test difference against H-series GPU
Across most message sizes, emulation tracks NCCL within ±1%.
The higher bandwidth at 256 MB is likely due to differences in pacing and buffering between NCCL and the emulated workload.
These results indicate that matching the communication schedule and flow characteristics is sufficient to reproduce fabric-level behavior.
What this enables
With emulation, fabric validation no longer depends on GPU availability.
Teams can:
- Validate congestion behavior under All Reduce traffic
- Tune ECN and PFC parameters for collective workloads
- Verify load balancing and path utilization
- Identify hotspots and incast scenarios early
This shifts network validation earlier in the deployment cycle, when issues are easier to diagnose and fix.
What this does not model
KAI DCB emulator is designed for fabric-level validation and does not model:
- GPU kernel execution
- Compute/communication overlap
- PCIe or NV Link contention inside a host
- NCCL implementation details beyond communication scheduling
As a result, it is not a substitute for end-to-end training benchmarks.
Instead, it isolates and validates the network component of the system.
Practical workflow
In practice, teams can:
- Deploy and configure the fabric
- Connect emulator test endpoints in place of GPU hosts
- Run NCCL-equivalent collective workloads at scale
- Compare results against known reference baselines
By the time GPUs arrive, the fabric is already characterized and validated.
Summary
AI workloads place unique and demanding requirements on the network.
By reproducing the communication patterns of NCCL directly on the fabric, we can validate these requirements early — without waiting for GPUs.
This reduces deployment risk and shortens the path to production.
Results from Aresone-M emulation platform 16x400Gbps interfaces (fanout).