Four V100s, Two Islands: Six Weeks Running a 27B Model on 2017 GPUs

The Tesla V100 came out in 2017. The SXM2 version - a socketed module rather than a PCIe card - used to be cheap because it doesn't plug into anything a normal PC has, yet each one still has 16 GB of very fast memory. That has changed over the last few months: with capable models like Qwen3.8 coming out, demand has picked up, and the 32 GB version has gone up to around AUD 1,000. I got in early and picked up my 16 GB ones when they were going for as little as AUD 80 each. I wanted four of them running Qwen3.8-27B as my own coding assistant, for Qwen Code, with no code leaving the house. This post is about one thing: how fast I could make it, and what actually moved the number.

The short version: 31.8 → 74.5 tok/s from one small kernel port, most other ideas didn't help, and speculative decoding looked great on benchmarks but made my real sessions slower.

The Rig

The model needs ~54 GB at 16-bit, so it's stored at 4 bits and split across all four GPUs. The wiring between them decides almost every number in this post:

flowchart LR
  subgraph A["Island A"]
    GPU0 <-- "NVLink ~145 GB/s" --> GPU1
  end
  subgraph B["Island B"]
    GPU2 <-- "NVLink ~145 GB/s" --> GPU3
  end
  A <-. "PCIe ~6.4 GB/s" .-> B

Two fast NVLink pairs, joined by a PCIe link about twenty times slower. I call them the two islands.

Step 1: Two GPUs (36–51 tok/s)

On an old Intel board with only two V100s, vLLM ran at ~51 tok/s but could only fit 8K tokens of context, useless for an agent. llama.cpp managed ~36–42 tok/s with the full 262K context, so it served for the first couple of weeks.

Step 2: Four GPUs Online

Four GPUs needed a bigger platform, so I moved to a Threadripper. Getting the V100s to actually power on there was a saga of its own (see Pitfalls); the short answer is that only one slot on the board can wake them.

Step 3: The Slow Bridge (31.8 → 74.5 tok/s)

With tensor parallelism across four GPUs (TP4), every layer ends with an all-reduce: the four GPUs add up their partial results. That happens ~128 times per token. My first TP4 number was 31.8 tok/s, slower than two GPUs:

PCIe-only matching NVLink showed bandwidth wasn't the problem. vLLM only uses its fast all-reduce when every GPU is NVLinked to every other; two islands fail that check, so everything fell back to generic NCCL.

The fork already had a two-level all-reduce for 8 GPUs in two groups of four. My layout is the same shape with groups of two, so the port was a handful of 4 → 2 edits behind an opt-in flag. After an exact-match test against NCCL (100 seeds, 2,000 iterations, zero difference), and narrowing it to the message sizes where it actually wins:

That matches what the fork's authors get on a machine with all four GPUs NVLinked. It's the biggest win of the project.

Step 4: Where the Time Goes

Writing runs at ~70 tok/s, about 14 ms per token:

Reading the prompt is the slow part: a 32K-token prompt takes ~36 s before the first word. About two-thirds of that is the all-reduce: during prefill each one carries 82 MB and takes 42.8 ms to cross the islands. Faster attention code changed nothing (36.4 s), because the bottleneck is the wire.

Prefix caching fixes the common case for agents: resending a 14K-token context dropped from 14.3 s to 0.19 s.

Step 5: One Copy or Two?

If crossing the bridge is expensive, you can avoid it by running one model copy per island (TP2 × 2):

TP4 wins for one agent with full context; TP2 × 2 wins for parallel agents. I kept TP4.

Step 6: Speculative Decoding - Great on Paper

The checkpoint ships an MTP head that guesses the next few tokens; the main model checks them all in one pass. On my coding benchmark:

On real sessions (long reasoning, tool calls, ~20K context) it was slower:

Coding answers accepted ~3.9 tokens per step; my real work only 2.0–2.7. A profile showed why that isn't enough here: a speculative step takes 36.4 ms against 13.9 ms, and the GPU sits idle for 14 ms of it waiting on Python bookkeeping between the draft and verify passes. The weight math itself costs the same for 3 tokens as for 1. MTP is off.

Step 7: Everything Else I Tried

Where It Landed

Production: TP4, hierarchical all-reduce, QUASAR NVFP4, FP8 E5M2 KV cache, prefix caching on, MTP off, full 262K context with ~767K tokens of cache. Cold prompts read at ~900 tok/s; repeated context is near-instant.

Pitfalls

• SXM2 GPUs need an SMBus command to power on. They show up in lspci but report "fallen off the bus" until the carrier gets a command on the slot's SMBus pins. On the X399-A only one slot's SMBus reaches the BIOS.
• That same SMBus reads the RAM. The BIOS reads each stick's ID chip over it during memory training, while the carrier is waking up. Four sticks won't train with the card installed; I'm on two.
• Manual power-on only half works. Booting with the carriers off, then switching them on and rescanning PCIe, brought up 64 GB and powered GPUs - but the driver failed and the RAM corrupted under load.
• First-gen Threadripper plus four non-ECC DIMMs shows instability as random crashes, not logged errors.
• The template default disabled reasoning. The fork's server defaults to enable_thinking: false; set it explicitly.
• Qwen Code gates vision in the client. Add "capabilities": {"vision": true} to the model entry; the server supported images all along.
• Short tests miss long-session bugs. The bigger-KV change passed every short test and crashed within 23 minutes of real use.
• GPUs can be "up" but not ready. Put a real CUDA smoke test in front of the service.
• Benchmark your real workload. MTP was +37% on coding prompts and slower on real sessions.
• A faster kernel isn't a faster server. Measure end to end.
• Correctness before speed. A broken all-reduce can look fast while corrupting output; exact-match against NCCL first.
• Check upstream claims at the source. Read the actual diff, not a summary.

Recommendations

1. Buy a server platform. EPYC 7002/7003 on an ASRock Rack ROMED8-2T or Supermicro H12SSL: 128 lanes, ECC RAM, proper slot SMBus. Skip first-gen Zen.
2. Better: an SXM2 server with every GPU on NVLink (Dell C4140, Supermicro 4029GP). The two-island bridge is what caps prompt reading and speculative decoding here.
3. Use a quantization-aware 4-bit checkpoint. On a V100, fewer bytes is the speed.
4. Fix the all-reduce first if your GPUs aren't fully connected; it can silently cost half your speed.
5. TP4 for one agent, TP2 per NVLink pair for several. Keep prefix caching on.
6. Measure speculative decoding on real sessions before leaving it on.

Next: an all-reduce that sums within each NVLink pair before crossing the bridge, to cut prompt-reading time, and a mixture-of-experts model for fast tool-calling work.