One PCIe Setting, 40% Faster

In the last post I blamed the slow bridge between my two NVLink pairs for holding the rig back. I was right that the bridge was slow and wrong about why. A flag on the PCIe switch was sending every GPU-to-GPU transfer up to the CPU and back down. Clearing it made the same model, on the same software, about 40% faster. No new kernel.

Profile first

The setup hasn't changed: four V100-SXM2-16GB behind one PLX 8749 switch, Qwen3.8-27B with the DFlash2 speculative decoder, TP4 on 1Cat-vLLM. I profiled one decode step with the PyTorch profiler, expecting the matrix maths to dominate.

It didn't. 56% of GPU time was NCCL all-reduce: about 143 calls per step at 100-150 µs each. The 4-bit matrix multiplies were roughly a fifth. My custom all-reduce from last time doesn't even run for DFlash2's 80 KiB verify payload, because it lost to NCCL at that size. So stock NCCL, crossing the bridge 143 times a step, was the real cost.

Someone with the same layout

While looking for ideas I found a small repo from someone running V100-SXM2 carriers behind PLX switches, two NVLink pairs, TP4. Same shape as mine. They measured an 80 KiB all-reduce at 33.5 µs. I measured 110.9 µs. Three times slower on supposedly identical wiring means something on my machine was wrong.

Their README pointed at PCIe ACS. When "P2P request/completion redirect" is on, a switch won't pass peer-to-peer traffic straight between its ports. It sends it to the CPU's root port and back. It's an isolation feature for virtual machines, and Linux often leaves it on. I checked mine:

$ sudo lspci -vvv -s 0b:08.0 | grep ACSCtl
ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-

ReqRedir+ CmpltRedir+, on all four ports above the GPUs. All four GPUs sit behind that one switch, so cross-pair traffic could have stayed inside it. It was being forced out.

flowchart LR
  subgraph B["Before: redirect on"]
    direction TB
    RC1["CPU root port"]
    S1["PLX switch"]
    G1["GPUs 0-3"]
    G1 -- "peer traffic" --> S1 -- "forced up" --> RC1
    RC1 -. "and back down" .-> S1
  end
  subgraph A["After: redirect cleared"]
    direction TB
    S2["PLX switch"]
    G2["GPUs 0-3"]
    G2 <-- "stays inside" --> S2
  end

The fix

Clear the redirect bits on the four ports. The register read 0x001d, I wrote 0x0011:

sudo setpci -s 0b:08.0 ECAP_ACS+6.w=0011   # and 0b:09.0, 0b:10.0, 0b:11.0

The catch is that ACS no longer isolates those ports from each other, which only matters if you pass GPUs through to VMs. I don't. It doesn't survive a reboot, so I put it in a small systemd service that runs before the model server, and the server's unit re-applies it on every start in case a port loses the setting after a PCIe error.

What it did

I timed single all-reduces inside a CUDA graph first, since that's what decode pays for:

80 KiB now lands next to the other rig's 33.5. My custom push kernel got two to four times faster too and now beats NCCL below about 20 KiB, but it still loses at 80 KiB, so the model just uses NCCL.

Then the whole model, nothing changed but the flag:

Prefill got faster as well, because it crosses the same links. My quality battery scored the same as before (it wobbles between 20 and 22 anyway). One side effect confused me for a minute: the GPU utilization number on my dashboard dropped from about 88% to 84% while throughput went up. A communication kernel waiting on another GPU still counts as busy, so slow all-reduces inflate utilization.

The benchmark that confused me

After a reboot I wrote a quicker benchmark and got 181 tok/s on code, not 327. I spent a while checking clocks, throttling, PCIe link width and the CPU governor. All fine. Toggling the flag on the same engine with the new script still showed +47%, so the fix was real, but 181 and 327 can't both describe one machine.

The old script sampled at temperature 0.3. The new one used 0, greedy. Same engine:

Greedy is about 1.8x slower on code here, and I don't know why yet. A tok/s figure without its temperature isn't a figure.

Against the published numbers

1Cat publishes decode speeds on four V100-SXM2 32 GB cards with a full NVLink mesh. I reproduced their single-request setup on mine (real source code as input, top-p 0.95, top-k 20, seed 0, 256 tokens, median of three warm requests):

The time per speculative round now matches within a few percent. The tok/s gap is acceptance: about 2.9-3.5 tokens per round for me against 4.3-4.6 implied by their numbers. I suspect that's the text (real code and an "explain this" prompt against their synthetic input), but I haven't checked. I also time decode from the client stream, not the server.

Wrong turns, and what's left

I was wrong last time that the wire was the limit, and that the push all-reduce couldn't beat NCCL. That second result was only true with the redirect on. I also tried running the model on two GPUs (it doesn't fit: the target takes 9.6 GiB per card and the draft adds about 4.7 more) and rebasing onto upstream's latest code, which was slower on my layout and produced garbage from one sampler path until I switched it off. I haven't reported that upstream yet.

Still open: two simultaneous 32K requests give 109-111 tok/s in total, less than one request alone (139-149), so batched decoding on my layout is off somewhere. I haven't soak-tested a day of real agent use, and the greedy slowdown is a mystery. This is one machine, three warm requests per cell and one seed.

Check your own rig

If you have GPUs behind a PCIe switch, this takes two minutes:

lspci -tv                                        # which switch ports sit above the GPUs
sudo lspci -vvv -s <downstream-port> | grep ACSCtl   # ReqRedir+ CmpltRedir+ means redirect is on
python -c "import torch;print([[torch.cuda.can_device_access_peer(i,j) for j in range(4) if j!=i] for i in range(4)])"

On my box nvidia-smi topo -p2p n says NS between the pairs even though peer access works, so trust the actual test.

The idea comes from the author of vllm-pcie-allreduce, on top of 1Cat-vLLM's kernels.