Running Qwen3.8-27B-FP8 on two H100s, MTP speculative decoding was the biggest decode win (49 → 95 tok/s). But there’s a catch: in vLLM, MTP and prefix caching are mutually exclusive. So the question became: which config should actually go into production?

The answer came from running the two configs head-to-head: GPU0 no-MTP + prefix caching, GPU1 MTP (3 draft tokens). Both are TP1, 262K max context, FP8 KV cache, with a LiteLLM proxy load-balancing between them. All numbers are TTFT p50 unless noted.

Decode: MTP takes the point#

First the good news: on decode, MTP wins at every concurrency level, with 1K-token prompts and 512 output tokens:

MTP decode throughput

At a single session the lead is 1.9× (94.9 vs 49.9 tok/s), narrowing to 1.28× at 32 concurrent sessions. So if you’re generating lots of tokens from cold prompts, MTP is your friend.

Single-turn: shared prefix, unique suffixes#

Here the tables start to turn. When multiple sessions share the same prefix (e.g. a common system prompt), the caching config stays flat under concurrency while MTP has to re-prefill the prefix for every request. At 4K prefix and no concurrency, MTP still wins (0.85s vs 1.02s); at 32 sessions it’s already 2.6× behind (13.55s vs 5.20s). At 16K prefix and 16 sessions, the gap is already 8× (27.85s vs 3.49s).

And the cache hit is nothing to sneeze at: on the second request with a 16K prefix, the cache hit is 2.4× faster than MTP’s re-prefill (0.91s vs 2.20s) — though at 4K the hashing overhead eats most of the benefit.

Multi-turn: where it’s decided#

Three-turn sessions, each turn growing the context by ~1K tokens (stepping 4K → 64K). Turn-2 TTFT — the time for the first follow-up — on a linear scale, one panel per base context:

Multi-turn TTFT on a linear scale, per base context

The linear scale makes the story impossible to miss: the caching config stays flat (~1.4–5.5s at every context size up to 8 concurrent sessions — only the ~1K new suffix gets prefilled, the shared prefix is a cache hit, ~65% hit rate during the run), while MTP re-prefills the full accumulated context on every follow-up.

For the large contexts I’m also showing the MTP curves on their own — on a log axis this “wall” barely looks like a wall at all:

The MTP wall on a linear scale

And the caching side up close, so the flatness is actually visible (the 132s point at 64K/16 is the shared KV-capacity ceiling of both configs, not a caching failure):

Prefix caching on a linear scale, zoomed

In numbers: at 8 sessions / 32K, MTP is already 28.6s; at 16 sessions / 64K it’s 80.9s. The ratio climbs to 16.8× (64K, 4 concurrent). The 64K/16 corner is the exception: both configs hit the KV cache capacity ceiling there (16 × 64K of live KV is ~1.05M tokens against a 1.32M pool, and the caching instance degrades worse because cached blocks are pinned) — that’s a capacity limit, not a config effect.

Verdict#

Workload Winner
Single session, long generations, cold prompts MTP (1.3–2.2× decode)
Multi-turn, concurrent sessions, growing context Prefix caching (up to 17× TTFT)
64K+ context at 16+ concurrent sessions Neither — KV capacity bound

Practical takeaway: prefix caching is the right default for conversational / multi-turn workloads with concurrent users sharing context or system prompts. MTP only pays off for decode-heavy, single-session, cold-prompt work. Since vLLM makes the two mutually exclusive, this measurement settled which one I’m putting into production.