PCIe vs Ethernet Scale-Up for Inference: When Each One Wins
same GPUs, two ways to join them, different best layouts
November 20265 min readtechnical
contents →
Parts 1 and 2 were about what a single all-reduce costs when GPUs have no NVLink. This part zooms out to the decision people actually make. You have 16 GPUs in two boxes. Do you join the boxes with RoCE Ethernet or with a PCIe switch fabric, and how do you lay the model out on top?
We had both setups side by side. One was two 8-GPU boxes over 400G RoCE through a single Ethernet switch. The other was a rack with the same GPU model, where the boxes share a PCIe switch fabric.
#Inside one box, the two are identical
Per-call all-reduce latency on 8 local GPUs matched on both racks, at every size. That's what you'd expect: inside a box, both racks are the same GPUs on the same kind of PCIe switch. Any difference between the racks comes from the cross-box path.
One caveat. End-to-end throughput did differ between the two racks' single boxes. We traced that to the serving harness, not the hardware: one ran vLLM's async scheduling and the other didn't. The aligned rerun never happened, so treat cross-rack throughput comparisons with care.
#Across boxes, PCIe is far cheaper per call at decode sizes
As NCCL sees it, a cross-box hop on the PCIe fabric cost a small fraction of an Ethernet crossing. At large message sizes the two converged, for the same reason as in part 1: each GPU's own PCIe link is the bottleneck either way. The whole difference lives in the decode range.
This is part 1's startup term again, with a much smaller α for the steps that leave the box. And it compounds. A 16-GPU tensor-parallel decode step makes around a hundred all-reduces, and each one crosses between the boxes twice.
#What that does to layouts
| Question | PCIe switch fabric | RoCE Ethernet |
|---|---|---|
| Split one TP group across two boxes? | Cost was within noise | Clear throughput loss at every load we ran (stock NCCL) |
| TP16 vs one 8-GPU box | TP16 pulls ahead from medium concurrency | TP16 trails one box until that box runs out of KV cache |
| TP16 vs two independent TP8 replicas | Not measured head to head. TP16 gave the best per-user rate in the agentic run | Two replicas win at every concurrency |
| Communication share of a 16-GPU decode step | A modest slice | Comparable to compute at medium load |
| Prefill-heavy traffic | Little difference | Little difference |
The rule of thumb: over Ethernet, keep each tensor-parallel group inside one box and add boxes as replicas. Over a PCIe fabric, a 16-GPU tensor-parallel group is a real option, and it buys per-request latency that replicas can't.
#When the network stops mattering: KV-bound agentic traffic
We also replayed an agentic coding trace: very long contexts and heavy prefix reuse. There the limit was KV-cache capacity, not communication, and the two fabrics landed at parity on the same layout.
The layout itself turned out to be the interesting part. In MLA models (DeepSeek-style attention, here GLM-5.2), the compressed latent KV cache is replicated on every tensor-parallel rank. So doubling TP doesn't grow the cache pool. Pipeline parallelism does, because each stage holds the cache for its own layers only. For long-context MLA serving across 16 GPUs, PP2 × TP8 beat TP16, and by more than any fabric difference.
Two more things we learned the hard way here:
- Every layout collapsed once the working set of contexts outgrew the cache. Prefix-cache hit rate fell off a cliff, and time to first token went with it.
- A CPU memory tier under two-node TP16 served almost none of the misses. It didn't help.
#Where Ethernet's cost actually lives
It's tempting to read all this as “Ethernet is slow.” The wire wasn't. The fabric ran near line rate, and the switch hop was a small part of each crossing. The cost sits in two places: the host-side path (NCCL's proxy thread, the NIC's PCIe side, turning GPU writes into RDMA), and NCCL's protocol choice, which part 2 fixed in software.
After that fix, TP16 over Ethernet improved a lot at high load. It still trailed two independent replicas, though, and nothing changed at low load, where fixed latency is everything.
That leaves an open question for hardware: how much of the remaining crossing cost can move out of software and into the fabric?
#Decision guide
- Latency-sensitive decode on 16+ GPUs: a PCIe fabric lets you run one big TP group. On Ethernet, don't.
- Throughput at high concurrency on Ethernet: independent replicas, plus a tuned NCCL rule (part 2) for anything that has to cross boxes.
- Long-context MLA models: grow the KV pool with pipeline parallelism before you grow TP. The fabric is secondary.
- Prefill-heavy batch work: either fabric. Spend the money elsewhere.
Next: every conclusion in this series came from hundreds of serving runs, and plenty of them were wrong the first time. Part 4 is the field guide to reading a run without fooling yourself. Read part 4.
Thanks to Helia Naeimi, Chris Petersen and Haoyuan Xu at Astera Labs. Shared with permission, with absolute numbers omitted.