How to Accelerate Your PCIe 5 (PCIe Gen 5) Product Design and Testing
Key takeaways:
- The peripheral component interconnect express 5.0 standard is a crucial milestone that has coincided with the high demand for hardware in the artificial intelligence and data center sectors.
- Although PCIe 5.0 mostly uses the same techniques as 4.0, some clever optimizations enable it to effectively quadruple the maximum possible data transfer.
- PCIe 5.0 design and compliance can be challenging, so highly sophisticated hardware and software solutions are necessary to streamline them.
The peripheral component interconnect express (PCIe) is the preferred bus for adding key peripherals like graphics cards and network cards to modern servers and computers. PCIe is a crucial component in the hardware that's powering the generative artificial intelligence (AI) market, global hyperscaledata centers, and many more sectors.
In this article, learn about PCIe 5 basics, applications, challenges, and solutions that streamline its design and testing.
What is PCIe 5.0 (PCIe Gen 5)?
Figure 1. PCIe 5 specification layers
PCIe 5.0 or PCIe Gen 5 is the fifth version of the specification for PCIe, the most popular interconnect standard used for attaching peripheral devices to servers and computers. The PCIe standards ensure that motherboards, graphics cards, ethernet cards, Wi-Fi cards, storage devices, and other peripherals from different vendors can seamlessly interoperate with each other when integrated into a server or computer.
The logical sub-blocks that constitute the PCIe 5.0 standards are depicted above.
In terms of data transfer, PCIe 5.0 is a serial interface consisting of multiple parallel lanes. Each lane transmits data serially one bit at a time. However, all the lanes transmit and receive simultaneously to increase the net throughput and behave a bit like a parallel bus.
What are the potential use cases for PCIe 5.0 in data centers, artificial intelligence, and gaming?
In this section, we look at some key use cases driving the adoption of PCIe Gen 5 and above.
Artificial intelligence
Artificial intelligence will be a major adopter ofPCIe 5 and above. In particular, large generative AI models (ranging from 20 billion to over 500 billion parameters) require data from system memory or storage to be pumped over the interconnect bus to AI accelerator cards at the maximum possible volume and velocity. AI accelerators for both server and personal use benefit tremendously from the high bandwidths of Gen 5.
Data centers
PCIe 5.0 is a key component in the hyperscale data centers that support modern AI, big data, and high-performance computing applications. PCI Express 5 enables faster technologies, such as:
- Next-generation ethernet: PCIe 5 enables 400-gigabitethernet (400GE) network interface cards and switches in data centers.
- High-speed storage: Enterprise storage technologies like non-volatile memory express solid state storage drives (NVMe SSDs) over fiber channels rely on PCIe 5. Storage controller cards and chipsets that interface various storage devices to motherboards and processors rely on PCIe and benefit from PCIe 5's speed. Such storage technologies include serial-attached small computer system interface (SAS) and serial advanced technology attachment (SATA).
- Accelerator cards: Modern accelerator cards like graphics processing units (GPUs) and tensor processing units (TPUs) for AI or graphics workloads benefit from PCIe 5 and higher standards.
Gaming and graphics
Gamers, graphics professionals, and video editors benefit from PCIe 5's high data transfer speeds between graphics cards (like the Nvidia GeForce RTX GPUs) and system memory using direct memory access.
Additionally, system performance is boosted by PCIe Gen 5 SSD and M.2 NVMe controllers that ensure that storage is less of a bottleneck when working with modern high-core central processing units (CPUs) and the latest double data rate (DDR5) system memories.
What’s the difference between PCIe 5.0 and PCIe Gen 5?
PCIe 5.0 and PCIe Gen 5 refer to the same thing. There is no difference between them—they are simply two ways of naming the fifth generation of the PCI Express standard. “Gen 5" is simply a shorter, more common way to refer to the Peripheral Component Interconnect Express 5.0 standard, which is the fifth generation of the PCIe technology.
Both terms describe the same specification defined by PCI-SIG, offering a raw data rate of 32 GT/s (gigatransfers per second) per lane and up to 128 GB/s total bandwidth in a 16-lane (x16) configuration. Whether labeled as "PCIe 5.0" or "PCIe Gen 5", they are interchangeable in terms of architecture, performance, and compatibility.
What are the key performance improvements of PCIe 5.0?
Some of the key performance capabilities of PCIe 5.0 are listed below.
Increased data rate
PCIe 5.0 or Gen 5 can achieve a maximum bandwidth of 32 gigatransfers per second (GT/s) per lane per direction. Each transfer involves one signal transition in one direction, and each signal transition represents one bit. The effective bandwidth then is 32 gigabits per second (Gbps) per PCIe lane per direction.
For a 16-lane duplex interface (such as the typical GPU's x16 slot), it translates to a net bandwidth of 128 gigabytes per second (GB/s).
A typical ethernet card's eight-lane (x8) duplex PCIe slot delivers a net 64 GB/s, which is more than sufficient for the 50 GB/s throughput required by 400GE networks.
Enhanced signal integrity
PCIe 5's non-return-to-zero (NRZ) signaling runs at 16 gigahertz (GHz) frequency, which is double the 8 GHz of PCIe 4.0. The greater signal attenuation due to channel insertion loss at this higher frequency must be effectively countered.
So PCIe 5.0 employs newequalization and other techniques to maintain signal integrity at these higher speeds. It achieves better performance, higher reliability, and lower error rates, and it also facilitates longer cable lengths.
Reduced error rates
PCIe 5 mandates a low bit error rate (BER) of 10-12 and a total channel insertion loss budget of just 36 decibels (dB). To counter the possibility of more burst errors, it mandates precoding at the transmitters.
Expanded device count
Due to its higher bandwidth, PCIe 5.0 can support more devices simultaneously without any bottlenecks. This is particularly beneficial in data center environments.
How fast is PCIe 5.0 compared to earlier PCIe versions?
While PCIe 4.0 achieves a data rate of 16 GT/s, PCIe 5 doubles it to 32 GT/s per lane in each direction. In terms of bytes, a PCIe Gen 4 x16 device can reach 32 GB/s while PCIe 5.0 can reach 128 GB/s.
Compared to PCIe 3.0's 8 GT/s, PCIe 5 has four times more throughput per lane.
The other parameters are shown in the illustration below.
Figure 2. Comparison of PCIe versions
What is the maximum data transfer rate for PCIe 5.0?
The maximum data transfer rate is 32 GT/s per lane per direction. This translates to the following maximum data rates:
- 128 GB/s for an x16 duplex PCIe slot
- 96 GB/s for x12 duplex
- 64 GB/s for x8 duplex
- 32 GB/s for x4 duplex
- 16 GB/s for x2 duplex
- 8 GB/s for x1 duplex
Can PCIe 5.0 devices work in PCIe 4.0 or earlier slots?
The PCIe 5 specification mandates full backward compatibility with all earlier versions. We can also expect PCIe 5 devices to be future-proof by working successfully with future PCIe versions. Additionally, a PCIe device with fewer lanes can be seated in a wider slot with more lanes. For example, an x8 network card can be seated in an x16 connector.
All this is possible through mechanisms for negotiating mutually compatible lane counts, PCIe versions, data rates, and encoding schemes as outlined below:
- Link training: During link training, the endpoint devices communicate with the root complex. This link initialization phase involves negotiating PCIe versions, data rates, lane widths, and encoding schemes. The stability of the link is then verified using these negotiated parameters.
- Capability registers: Capability registers in PCIe devices provide information about what versions the device supports, the maximum number of lanes, and other parameters.
- Signaling: PCIe 5 uses NRZ signaling like all the earlier versions. The data encoding scheme — like 128b/130b or 8b/10b — is selected to be mutually compatible.
What are the most time-consuming tasks in PCIe 5 design?
PCIe 5's high-performance requirements impose a lot of demands on the design and simulation phases as described below:
- Ensure signal integrity: Designing for flawless signal integrity is essential. Crosstalk, reflection, and mode conversion are significant concerns that can introduce bit errors and must be prevented.
- Simulate transceiver compliance: Comprehensive physical layer testing is required to ensure compliance with the stringent standards. It involves time-consuming analysis and simulations in both time and frequency domains.
- Maintain data rate: High data rates can lead to difficulties in managing the rise times and signal distortions.
- Verify differential signaling: Each PCIe lane consists of two differential pairs — one pair for sending and the other for receiving. Proper differential signaling while avoiding electromagnetic interference (EMI) requires careful designs and simulations.
- Achieve interoperability and compatibility: Ensuring that PCIe 5 endpoints and root complexes can interoperate with earlier-version devices requires extensive simulations.
What are some challenges and limitations of adopting PCIe 5.0 (Gen 5)?
Some of the challenges in adopting PCIe 5.0 are explained below:
- Signal integrity: High-speed interconnects like PCIe 5 result in larger reflections at impedance discontinuities, which degrades the eye diagram, a critical measure of signal quality.
- Printed circuit board (PCB) design: PCIe 5 design requires low-loss dielectric materials, wide signal traces, and techniques like back-drilling of vias or micro-via technology, which add to the design effort and complexity.
- Advanced instruments: The complexity of PCIe 5 digital design requires more sophisticated instruments for analysis and characterization. Design engineers must have a deep understanding of signal propagation properties and utilize equipment like 16-GHz vector network analyzers for complete characterization of physical layer components.
- Jitter management: The shorter clock cycles imply smaller jitter budgets. Reducing jitter in PCIe 5 is more complex compared to earlier generations.
- Return loss: Maintaining a return loss greater than specific limits at various frequencies is crucial. This involves careful channel design and compliance testing.
- Receiver design: Receivers are more susceptible to degraded signals due to high-frequency channel losses. Designing robust receivers that can tolerate degraded signals while achieving acceptable BERs is essential.
- Protocol testing: Various state transitions within the protocol layer as well as link training and status state machine (LTSSM) must be validated using sophisticated emulation and testing setups.
- Labor-intensive testing: The complexity of PCIe 5 makes the testing process very labor-intensive. Companies need robust automated test solutions to reduce test durations from days to hours.
How does EDA software help with PCIe 5.0 design?
Electronic design automation (EDA) tools are crucial for managing the challenges and time-consuming tasks outlined earlier. The various ways in which EDA software helps are explained below:
- Signal and power integrity simulations: As channel topologies diversify and the number of parameters multiplies, ensuring signal and power integrity requires tools that can simulate the electromagnetic effects of high-speed integrated circuits and PCB interconnects. EDA tools can simulate conditions like crosstalk and reflections to avoid signal degradation and timing problems. This includes minimizing jitter, which is critical as clock cycles shorten.
- Channel simulations: EDA software enables the creation and execution of input-output buffering information specification (IBIS) and algorithm modeling interface (AMI) models for simulating the analog channels and end-to-end signal paths.
- Protocol-level analysis: EDA software aids in understanding and optimizing each of the layers — physical, data link, and transaction.
- Integrated co-simulations: EDA software enables end-to-end analyses by co-simulating different areas simultaneously.
- Standards compliance: EDA software is crucial for performing compliance tests required by the PCI Special InterestGroup (PCI-SIG). These tests verify that the electrical performance complies with the PCIe 5.0 standards.
- Automated testing: With EDA tools, designers can automate much of the testing process for maximum efficiency. The automation of compliance tests and the generation of reports help rapid design validations and adjustments.
How does Keysight streamline high-quality PCIe 5.0 design and verification?
Keysight offers a comprehensive set of software solutions and hardware instruments for in-depth PCIe 5 designing, simulating, and testing. These capabilities are explained in detail below.
PCIe 5.0 EDA solutions
The System Designer for PCIe is an end-to-end design environment for modeling and simulating PCIe 5.0 systems. Its simulation-driven compliance testing of PCIe 5.0 designs reduces design iterations and shortens your time-to-market.
System Designer for PCIe also includes IBIS-AMI modeling for simulating analog (electrical) and signal path behaviors.
Transceiver testing
Figure 3. Keysight receiver compliance test automation platform
Keysight's software solutions for PCIe 5 testing include:
- Physical Layer Test System (PLTS) for interconnect signal integrity testing
- N5991 Receiver Compliance Test Automation Platform for verifying PCIe 5 receivers
- Transmitter Electrical Performance Validation and Compliance Software for testing PCIe 5 transmitters
These software support powerful test and verification features like the ones listed below:
- They facilitate both time-domain and frequency-domain analyses, allowing engineers to assess signal integrity accurately.
- Batch-mode automatic fixture removal allows multiple-channel de-embedding with a single fixture model, significantly reducing analysis time.
- Mode-conversion analysis can provide early insights into potential issues related to EMI and system performance.
- Virtual pseudo-random bit sequence pattern generators can accelerate the generation of eye diagrams from S-parameters, reducing the usual processing time from hours to seconds.
- They enable highly accurate characterization of transmission lines to simulate and improve signal integrity.
Protocol analysis
Figure 4. PCIe 5 protocol analysis using Keysight analyzer and exerciser
The PCIe 5 (often referred to as PCIe Gen 5) protocol exerciser and protocol analyzer enable the analysis of all topologies and use cases.
Accelerates your PCIe 5 time-to-market with Keysight
In this article, you discovered the challenges of PCIe 5 and the solutions available to address them.
Contact us for expert advice on using our hardware and software solutions to get your PCIe 5 devices to market quickly with the highest quality.