← Blog

PCIe vs Ethernet Scale-Up for Inference: When Each One Wins

same GPUs, two ways to join them, different best layouts

November 20265 min readtechnical

contents →

Parts 1 and 2 were about what a single all-reduce costs when GPUs have no NVLink. This part zooms out to the decision people actually make. You have 16 GPUs in two boxes. Do you join the boxes with RoCE Ethernet or with a PCIe switch fabric, and how do you lay the model out on top?

We had both setups side by side. One was two 8-GPU boxes over 400G RoCE through a single Ethernet switch. The other was a rack with the same GPU model, where the boxes share a PCIe switch fabric.

The same boxes joined by Ethernet or by a PCIe switch fabricRoCE Ethernetbox A: PCIe switchbox B: PCIe switchEthernetswitchGPU → PCIe → NICproxy thread, RDMAwire → switch → wireNIC → PCIe → GPUPCIe switch fabricbox A: PCIe switchbox B: PCIe switchPCIefabricGPU → PCIe→ fabric →PCIe → GPUInside a box the two racks are the same; only the hop through the middle differs.
Figure 1. The two setups. Inside a box they are the same; only the hop through the middle differs, and the Ethernet hop has many more steps in it.

#Inside one box, the two are identical

Per-call all-reduce latency on 8 local GPUs matched on both racks, at every size. That's what you'd expect: inside a box, both racks are the same GPUs on the same kind of PCIe switch. Any difference between the racks comes from the cross-box path.

One caveat. End-to-end throughput did differ between the two racks' single boxes. We traced that to the serving harness, not the hardware: one ran vLLM's async scheduling and the other didn't. The aligned rerun never happened, so treat cross-rack throughput comparisons with care.

#Across boxes, PCIe is far cheaper per call at decode sizes

As NCCL sees it, a cross-box hop on the PCIe fabric cost a small fraction of an Ethernet crossing. At large message sizes the two converged, for the same reason as in part 1: each GPU's own PCIe link is the bottleneck either way. The whole difference lives in the decode range.

This is part 1's startup term again, with a much smaller α for the steps that leave the box. And it compounds. A 16-GPU tensor-parallel decode step makes around a hundred all-reduces, and each one crosses between the boxes twice.

#What that does to layouts

TP16, two TP8 replicas, and PP2 by TP8 on two boxesTP16one group, both boxesbox Abox B~100 all-reduces per stepeach crossing twice2 × TP8two independent serversbox Abox Bnothing crosses per stepPP2 × TP8stage 1 in A, stage 2 in Bbox Abox Bone activation hand-off per step
Figure 2. Three ways to use 16 GPUs in two boxes, and what each sends between them per step.
QuestionPCIe switch fabricRoCE Ethernet
Split one TP group across two boxes?Cost was within noiseClear throughput loss at every load we ran (stock NCCL)
TP16 vs one 8-GPU boxTP16 pulls ahead from medium concurrencyTP16 trails one box until that box runs out of KV cache
TP16 vs two independent TP8 replicasNot measured head to head. TP16 gave the best per-user rate in the agentic runTwo replicas win at every concurrency
Communication share of a 16-GPU decode stepA modest sliceComparable to compute at medium load
Prefill-heavy trafficLittle differenceLittle difference

The rule of thumb: over Ethernet, keep each tensor-parallel group inside one box and add boxes as replicas. Over a PCIe fabric, a 16-GPU tensor-parallel group is a real option, and it buys per-request latency that replicas can't.

#When the network stops mattering: KV-bound agentic traffic

We also replayed an agentic coding trace: very long contexts and heavy prefix reuse. There the limit was KV-cache capacity, not communication, and the two fabrics landed at parity on the same layout.

The layout itself turned out to be the interesting part. In MLA models (DeepSeek-style attention, here GLM-5.2), the compressed latent KV cache is replicated on every tensor-parallel rank. So doubling TP doesn't grow the cache pool. Pipeline parallelism does, because each stage holds the cache for its own layers only. For long-context MLA serving across 16 GPUs, PP2 × TP8 beat TP16, and by more than any fabric difference.

Why pipeline parallelism grows an MLA KV pool and tensor parallelism does notTP16one GPU's KV memory6 layers × T tokenslatent KV for all 6 layers,copied on every rankPP2 × TP8one GPU's KV memory3 layers × 2T tokenslatent KV for its own 3 layers:the same memory holds 2× the tokens
Figure 3. Why pipeline parallelism grows an MLA model's KV pool and tensor parallelism doesn't. Same memory per GPU, twice the tokens.

Two more things we learned the hard way here:

  • Every layout collapsed once the working set of contexts outgrew the cache. Prefix-cache hit rate fell off a cliff, and time to first token went with it.
  • A CPU memory tier under two-node TP16 served almost none of the misses. It didn't help.

#Where Ethernet's cost actually lives

It's tempting to read all this as “Ethernet is slow.” The wire wasn't. The fabric ran near line rate, and the switch hop was a small part of each crossing. The cost sits in two places: the host-side path (NCCL's proxy thread, the NIC's PCIe side, turning GPU writes into RDMA), and NCCL's protocol choice, which part 2 fixed in software.

After that fix, TP16 over Ethernet improved a lot at high load. It still trailed two independent replicas, though, and nothing changed at low load, where fixed latency is everything.

That leaves an open question for hardware: how much of the remaining crossing cost can move out of software and into the fabric?

#Decision guide

  1. Latency-sensitive decode on 16+ GPUs: a PCIe fabric lets you run one big TP group. On Ethernet, don't.
  2. Throughput at high concurrency on Ethernet: independent replicas, plus a tuned NCCL rule (part 2) for anything that has to cross boxes.
  3. Long-context MLA models: grow the KV pool with pipeline parallelism before you grow TP. The fabric is secondary.
  4. Prefill-heavy batch work: either fabric. Spend the money elsewhere.

Next: every conclusion in this series came from hundreds of serving runs, and plenty of them were wrong the first time. Part 4 is the field guide to reading a run without fooling yourself. Read part 4.

Thanks to Helia Naeimi, Chris Petersen and Haoyuan Xu at Astera Labs. Shared with permission, with absolute numbers omitted.