From Single Collective to Parallelism Benchmark – Part 2
✅ Part 1: DP — Data Parallel traffic pattern under 3D parallelism
✅ Part 2: TP — Why Tensor Parallelism has to stay inside the scale-up domain (this post)
🔜 Coming next: PP — the Send/Recv traffic pattern of pipeline parallelism
Why Tensor Parallelism has to stay inside the scale-up domain
A Llama-70B-class model (80 layers) running Tensor Parallel (TP) = 8 fires 320 AllReduce calls per micro-batch. Everyone sits on the critical path. None can be hidden behind another micro-batch's compute the way Pipeline Parallelism (PP) traffic can.
That count is why TP is the first parallelism dimension that any scale-up fabric, NVLink or Ethernet, must get right.
What is Tensor Parallel
Here’s a worked example (h=2, d_ff=3): x=[1,2] through up-projection matrix A gives y=[1,2,3],
then through down-projection matrix B gives z=[4,5].
Same example under TP=3: each GPU owns one column of A and one row of B - one AllReduce at the end.
Where the 320 comes from
TP splits the weight matrices inside each layer. It cuts width, not depth. Layer count is PP's job.
In an MLP block, the first matrix is split by column and the second by row. Each GPU computes a partial sum, and one AllReduce is needed to produce the full output. Backward needs one more AllReduce on the input gradient. Attention follows the same pattern, so the count per layer is fixed:
- Attention: 1 forward + 1 backward AllReduce
- MLP: 1 forward + 1 backward AllReduce
- Total: 4 × num_layers per micro-batch
The count does not depend on head count or hidden size. Head count only changes local compute granularity. Layer count is what multiplies the calls.
With Sequence Parallelism (the default pairing in modern Megatron), each AllReduce becomes a ReduceScatter + AllGather. The call count roughly doubles, but the byte volume stays the same.
Each message is one activation tensor, b × s × h.
At b=1, s=4096, h=8192 in bf16, that is 64 MiB.
What happens if TP leaves the node
Back-of-envelope, with the assumptions stated: ring AllReduce, TP=8, line-rate links, no latency term, no overlap.
- Per GPU, one AllReduce moves 2(n−1)/n × 64 MiB = 112 MiB.
- 320 calls move about 37.6 GB per GPU per micro-batch.
- Over NVLink (H100: 900 GB/s total, about 450 GB/s each way): ~0.08 s
- Over a 400G NIC (50 GB/s each way): ~0.75 s
For the same micro-batch, the compute is about 0.5 s per GPU. This assumes about 400 TFLOPS achieved and 6 × 70B × 4096 tokens spread across 8 GPUs. On NVLink, TP communication is a fraction of compute and can be partly overlapped. Across the NIC, it is larger than compute and mostly exposed.
The gap is roughly 9× on bandwidth alone. Latency makes it worse, because 320 serialized collectives pay the per-hop cost 320 times.
Inference makes this harder. Decode has no backward pass, so the call counts halves. But the message shrinks to b × 1 × h, which is 16 KiB at b=1, h=8192 in bf16, and one AllReduce fires per layer per generated token. That is a pure latency problem, and SP stops helping because s=1.
The evidence matches the arithmetic
The Megatron-LM scaling study (Narayanan et al., 2021) gives the same guidance: use TP up to the number of GPUs in a server and use PP to go across servers.
DeepSeek-V3 shows the opposite path. It avoids TP in training altogether by engineering per-layer memory down, using fine-grained experts, FP8 and DualPipe. Teams that can avoid TP do, because of this cost.
What does this mean for your benchmark
TP traffic has a different shape from the single big AllReduce that most collective benchmarks measure. It is hundreds of serialized, latency-critical collectives per step.
- Measure per-collective latency and its tail, not only bus bandwidth. A fabric that wins on 1 GiB AllReduce bandwidth can still lose on TP, where sizes run from 16 KiB (decode at b=1) through a few MiB (decode, batched) to tens of MiB (prefill and training).
- Test by iteration, not by size sweep. TP is a high-frequency collective: hundreds of calls per step, back-to-back. Replay a full iteration (all 4 × num_layers collectives in order, with the compute gaps between them) and report iteration time plus the per-collective latency distribution inside it. A sweep of isolated sizes (16 KiB to tens of MiB) misses queueing, back-to-back contention, and tail effects that only appear under the real call pattern. This is the same argument as iterated collective testing in my earlier post on epoch-aware testing.
- Treat TP as the workload that sets the bar for Ethernet scale-up. If the fabric cannot keep TP at 8 GPUs close to NVLink behavior, the TP group gets pushed out of the domain and the cost shows up in step time.