Benchmarking Collective Operations

AI clusters integrate various compute, interconnect, and network elements, often representing capital expenditures in the tens or hundreds of millions of dollars. Maximizing the efficiency of these investments is therefore critical. Benchmarking—encompassing systematic testing and measurement—plays a pivotal role in uncovering performance bottlenecks and identifying opportunities to enhance throughput without incurring additional hardware costs.

The performance of AI clusters, especially for machine learning (ML) training, depends greatly on the ability to scale computations among a large number of identical neural processing units (NPUs), commonly known as graphics processing units (GPUs) or tensor processing units (TPUs). As NPUs work in parallel to train AI models, they must periodically communicate data, such as gradients and model weights, to maintain synchronization and drive model accuracy. The movement of data among NPUs is known as a collective operation. Key metrics such as data size, collective completion time (CCT), and algorithm bandwidth are examined in detail to illustrate how these measurements can be leveraged to optimize network performance.

The paper further delves into collective benchmarks, the methodology for performing and reporting the results of such measurements in a repeatable and well-understood manner. The paper explores key metrics such as data size, collective completion time (CCT), algorithm bandwidth, and more.

In conclusion, benchmarking collective operations is a foundational methodology for understanding the performance limits of distributed AI infrastructure. By enabling network operators to evaluate available options quickly and accurately, it facilitates the optimal utilization of existing infrastructure resources and contributes to a reduction in AI/ML runtimes.