High-Performance Networking Offloads for AI/ML-Focused Cloud Platforms

The rapid evolution of Artificial Intelligence (AI) and Machine Learning (ML) workloads, driven by the rise of Large Language Models (LLMs) like ChatGPT, has ushered in a new era of technological innovation. LLMs have proven remarkably useful in a wide range of applications, including content creation, audio and video analysis, customer support, education, and cybersecurity. This growing adoption has accelerated networking, storage, and cloud infrastructure advancements while fostering innovative technologies aimed at building customized AI / ML solutions. These innovations are essential for creating a competitive, dynamic market and reducing the risk of monopolization by a few dominant players.

The demands of AI / ML workloads on cloud and networking infrastructure are immense, requiring high-performance solutions to ensure scalability, efficiency, and reliability. Devices like SmartNICs (for example, NVIDIA ConnectX-7) and Data Processing Units (DPUs) (for example, BlueField-3, Pensando DSC ) are now indispensable for offloading compute-intensive tasks, boosting throughput, reducing latency, and optimizing resource utilization — critical for the success of both training and inference workloads.

In this joint paper, Crusoe, a vertically integrated, sustainably powered, and purpose-built AI cloud platform, and Keysight, a leader in test and measurement tools, collaborate to explore the essential elements of building and optimizing LLM infrastructures. The paper covers practical use cases for AI / ML workloads, detailed methodologies for testing and validating performance, and the benefits of data-driven optimization. By integrating real-world insights from testing in actual LLM environments, this paper serves as a comprehensive guide for building robust and scalable LLM infrastructure, offering actionable strategies to meet the stringent demands of next-generation AI / ML applications.