From Single Collective to Parallelism Benchmark – Part 1
✅ Part 1: DP — Data Parallel traffic pattern under 3D parallelism (this post)
🔜 Coming next: TP — Why Tensor Parallelism have to stay inside the scaleup domain
Data Parallel traffic pattern under 3D parallelism
Back when a single GPU could hold an entire model, Data Parallelism (DP) was simple: every GPU kept a full copy of the parameters and optimizer state, each consumed a different shard of the data, and once backward finished, all N GPUs ran a single AllReduce — a message the size of the full gradient, Ψ, with a communication domain spanning the entire cluster. That's still the default mental model most people carry for "DP traffic," and it's the traffic model baked into a lot of network test cases: one big AllReduce, every node a participant.
Naive DP's real pain point is memory redundancy: under Adam, every GPU stores a full copy of the parameters, gradients, and fp32 momentum/variance backups — none of it unique within a DP group. ZeRO shards optimizer state (ZeRO-1), gradients (ZeRO-2), and parameters (ZeRO-3) across DP ranks, splitting naive DP's AllReduce into a ReduceScatter+AllGather pair. ZeRO-1/2 just swap what rides on the AllGather — gradients become updated parameter shards — so total bytes stay at 2Ψ; only ZeRO-3 pays more, since sharded parameters need a fresh AllGather before every layer's use, pushing volume to roughly 3Ψ (1.5x). PyTorch's native FSDP is, conceptually, ZeRO-3.
There's a precondition here that's easy to miss: the N in ZeRO's sharding formula is the GPU count along the DP dimension, not the cluster's total GPU count. Under pure DP the DP group is the whole cluster, so the two happen to coincide — which is exactly why it's tempting to assume ZeRO shards "across every GPU." That coincidence only holds for pure DP.
Once the model outgrows a single GPU and you have to stack (Tensor Parallel) TP and (Pipeline Parallel) PP on top, that assumption breaks.
Under 3D parallelism, what DP replicates is no longer a single GPU — it's an entire TP×PP shard group. Take an 8-GPU layout with TP=2, PP=2, DP=2: GPU0–3 form the first replica, GPU4–7 the second. DP's AllReduce only happens across replicas, between ranks at matching TP-rank/PP-stage positions — GPU0 syncs only with GPU4, because both sit at "Stage 0, TP-rank 0"; GPU0 never talks directly to GPU5, GPU6, or GPU7. The DP group shrinks from "every one of the N GPUs" to "N / (TP size × PP size)" — which closes the loop from the previous section: the N in ZeRO's sharding formula is exactly this shrunken DP world size, no longer equal to the total GPU count, and ZeRO's ReduceScatter/AllGather communication domain narrows right along with it, down to that same DP sub-group, fully independent of TP's and PP's own domains.
This has two direct consequences. First, the communication domain fragments from one flat plane into many disjoint groups: what used to be a single cluster-wide AllReduce is now TP×PP parallel, independent AllReduces, each connecting only a subset of nodes at matching positions, all running concurrently. Second, each DP AllReduce also moves a smaller message: because the parameters are already split 1/TP-wide by TP, and each PP stage only owns 1/PP of the layers, the gradient volume any single GPU needs to synchronize is inherently Ψ/(TP×PP), not the full Ψ.
More importantly, these communication domains are layered on top of each other and triggered independently — none of them replace another. TP's AllReduce lives in the scale-up domain, firing once per layer on both forward and backward — high-frequency, small messages. PP is point-to-point Send/Recv across stage boundaries. DP fires least often, but it's still a cross-node sub-group AllReduce — just with participants narrowed from "every GPU" down to "the handful sitting at matching positions." The three process groups are fully independent and never interfere with one another: a single GPU is simultaneously a member of one TP group, one PP stage, and one (now-narrowed) DP group.
The implication for network testing and capacity planning is direct: modeling DP traffic as "one cluster-wide AllReduce, message size equal to the full parameter count" gets both the communication-domain scope and the per-message size wrong under 3D parallelism. Real traffic is many concurrent, localized DP sub-group AllReduces, layered on top of intra-node TP's high-frequency small messages and cross-stage PP's point-to-point traffic — not one isolated giant AllReduce. Test matrices need to reconstruct each DP sub-group's real membership from the actual TP×PP split, not assume the DP communication domain equals every node in the cluster.
Want to understand the complete communication picture of 3D training? Stay tuned for Part 2 and check out our KAI DC Builder solution.