MTP vs prompt caching — Qwen3.8-27B-FP8 on 2× H100: which one should you run?

Tue, 18 Aug 2026 12:00:00 +0000

Running Qwen3.8-27B-FP8 on two H100s, MTP speculative decoding was the biggest decode win (49 → 95 tok/s). But there’s a catch: in vLLM, MTP and prefix caching are mutually exclusive. So the question became: which config should actually go into production?

The answer came from running the two configs head-to-head: GPU0 no-MTP + prefix caching, GPU1 MTP (3 draft tokens). Both are TP1, 262K max context, FP8 KV cache, with a LiteLLM proxy load-balancing between them. All numbers are TTFT p50 unless noted.

Vllm on ZoliBen Csupra(Kabra)

MTP vs prompt caching — Qwen3.8-27B-FP8 on 2× H100: which one should you run?