ESUN Gives Up the IP Header
AI cluster networks typically split into two domains: scale-up, the high-bandwidth interconnect between GPUs inside a single compute unit — small, fixed topology; and scale-out, the interconnect across racks and pods — large, and dependent on routing. Ethernet has long dominated scale-out. But as scale-up domains grow from a handful of GPUs to ≥1000 GPUs spanning multiple racks, covering scale-up with Ethernet has become a live question. ESUN (Ethernet for Scale-Up Networking) is OCP's answer — the 1.0 spec shipped in February 2026. Its approach isn't a patch on top of Ethernet; it re-examines whether the scale-up domain needs an IP layer at all, and lands on a judgment opposite to RoCEv2, the dominant scale-out approach.
RoCEv2 is built for scale-out: crossing racks and pods require L3 routing. It encapsulates InfiniBand's BTH/RETH semantics inside UDP/IP (fixed destination port 4791), at a cost of 28–48 bytes of IP+UDP header per frame. Congestion control rides on the IP header's ECN bit — switches probabilistically mark the CE bit once queue depth exceeds a min-threshold, the receiver generates a CNP packet back to the sender to trigger a rate cut, and without ECN enabled the fallback is hop-by-hop PFC pause frames. The whole mechanism assumes packets traverse multiple hops of routing and that addresses need to aggregate.
ESUN assumes the opposite: in a scale-up domain, GPU count and connectivity are fixed at deployment time — no routing needed, no need for IP-address aggregability. It splits that assumption into four dimensions, each landing on a design opposite to RoCEv2.
Addressing and forwarding: ESUN replaces the IP header with a 4-byte EH header. The address format itself doesn't change — it's still a standard 48-bit MAC, and switches still forward by looking up the destination MAC (DA). What changes is forwarding behavior. Standard Ethernet, where RoCEv2 lives, relies on dynamic MAC learning plus ARP resolution, and floods unknown destination addresses to every port — a way of coping with topology that can change at any time. ESUN assumes that uncertainty doesn't exist in a scale-up domain: the spec states outright that there is "no learning or aging of MAC addresses," forwarding tables must be statically configured, and frames with an unknown destination address must not be broadcast — they're dropped. This is a shift from "discover topology at runtime" to "configure topology deterministically at deployment time."
Congestion and priority signaling: RoCEv2's congestion feedback rides on the IP header's ECN bit — the switch marks it, the receiver generates a CNP packet and sends it back, all traversing a full L3 data path. ESUN's EH-ECN is only 2 bits, but semantically identical to IP ECN; the difference is that it's embedded directly in the 4-byte L2 header, so switches and endpoints can read congestion state without parsing up to IP. Priority classification follows the same logic: RoCEv2 relies on the IP header's DSCP or the 802.1p 3-bit CoS for traffic classification; ESUN's EH-CoS is likewise 3 bits — a compressed version of both — and the spec explicitly states that when an ESUN header is present, EH-CoS takes priority over any 802.1Q tag or CBFC-defined CoS on the same frame.
Load balancing: RoCEv2 relies on ECMP hashing over the source/destination IP and UDP port 5-tuple to spread traffic across equal-cost paths. ESUN uses a 16-bit Flow Label in its header for the same role — switches hash on {DA, SA, Flow Label} to pick an egress port. The outcome is equivalent, but ESUN's hash input no longer needs IP/UDP fields — a direct extension of the same logic as dropping the IP layer for addressing: once forwarding itself no longer depends on the IP header, there's no reason for the load-balancing entropy to come from the IP layer either — a dedicated field in the 4-byte L2 header does the job more directly.
Link reliability: RoCEv2's lossless networks lean mainly on PFC — once a queue exceeds threshold, the switch sends PAUSE frames upstream hop by hop, stalling all traffic at a given priority on the entire link, which risks head-of-line blocking and cascading pause storms. ESUN doesn't reinvent link-layer mechanisms; it directly references two capabilities already defined by the Ultra Ethernet Consortium (UEC): CBFC (Credit-Based Flow Control), which allocates buffer via fine-grained credits instead of bluntly pausing an entire link, and LLR (Link Level Retry), which retransmits lost frames at the link layer instead of waiting for an upper-layer protocol to detect the loss. The spec makes both mandatory on the switch side and optional on the endpoint side — an asymmetric requirement that pushes the primary responsibility for link reliability onto the switch.
These differences buy a quantifiable payoff. RoCEv2's IP+UDP header overhead is fixed at 28–48 bytes; ESUN's header is only 4 bytes, netting 20–40 bytes saved per frame. That saving barely matters for large frames, but scale-up domains carry model-parallel traffic — AllReduce, MoE expert dispatch — small messages with high-frequency synchronization, where header overhead is already a large share of the frame, so the savings get amplified. The spec's own quantified basis: within a 128-accelerator domain, a 20% bandwidth drop causes a 5% end-to-end training performance loss — the concrete reason the scale-up domain is willing to trade away IP-layer routing flexibility for a thinner header and faster congestion feedback.
None of this is free. Giving up address aggregation means the switch's static MAC table entries grow linearly with endpoint count, unlike IP routing, which compresses via subnetting. ESUN also doesn't define a re-routing mechanism for link- or plane-level failures — that responsibility is pushed entirely onto whatever transport protocol runs on top of it.
Pulling this apart, what's genuinely clever about ESUN isn't a new mechanism — it's pinpointing a precondition: the scale-up domain's topology is already fixed at deployment time. The functions the IP header carries — routing, address aggregation, congestion feedback, priority, load balancing — exist fundamentally to handle uncertainty: topology that can change, addresses that need to aggregate, packets that traverse unknown paths. Once topology is fixed a priori, that uncertainty disappears, and OCP simply moves those functions, in place, into a 4-byte Ethernet header — no new protocol stack, no new addressing scheme, just re-implementing what the IP header used to do, at a thinner layer closer to the link. It's a classic "trade an assumption for overhead" design: the firmer the topology-determinism precondition holds, the more valuable the bytes ESUN saves and the sharper congestion feedback it delivers.
This is also why ESUN's significance reaches beyond its own spec. Before this, the AI data center network stack was split: the scale-out domain ran RoCEv2 over Ethernet, while the scale-up domain generally depended on proprietary interconnect standards. ESUN, combined with UEC's LLR/CBFC, gives the scale-up domain its first open, multi-vendor, native option built on standard Ethernet PHY and switch ASICs — in principle, an AI network could now be delivered end to end, from intra-node to cross-rack, on the same Ethernet-ecosystem hardware, without needing a separate interconnect standard just for scale-up.
But "can be delivered" and "has been delivered" are two different things. ESUN today is still a paper spec: the EtherType encoding is unassigned, there are no commercial switch/NIC implementations, and no production performance data exists. The spec itself leaves an obvious gap — it defines no detection or re-routing mechanism for link- or plane-level failure, leaving that entirely to whatever transport runs above it. Mature proprietary interconnects, by contrast, have already been refined across multiple product generations on ultra-low latency and failure recovery. Delivering AI networks entirely over Ethernet is architecturally sound. Whether it can actually displace proprietary interconnects isn't a question of whether Ethernet can do it — it's whether this open ecosystem can catch up on latency, reliability, and operational maturity to interconnects that have been running in production for years. That's the question ESUN needs to answer next, not one this 1.0 spec answers on its own.