Speculative decoding in production: when it pays and when it does not

Speculative decoding is one of those techniques that gets described as a free lunch. It is not free, and whether it is lunch depends entirely on your workload. We have switched it on and seen inter-token latency drop hard. We have also switched it on and watched throughput fall while the GPUs got hotter. Same technique, different traffic.

This is how we decide, and the harness we use to decide it.

The mechanics in one paragraph#

Decoding is memory-bound at small batch sizes: each new token requires streaming the whole model's weights through the GPU once, and the arithmetic per token is tiny compared to the bandwidth. Speculative decoding exploits that. A cheap draft proposes several tokens at once. The target model then verifies all of them in a single forward pass, which costs about the same as generating one token. Every accepted draft token is a token you got almost for free. Every rejected one is wasted draft work, plus you fall back to the target's own sample at that position. The output distribution is provably identical to plain decoding, so this is a pure speed trade, not a quality trade.

The number that governs everything is the acceptance rate: what fraction of proposed tokens the target agrees with.

Three ways to draft#

N-gram lookup is the one to try first. It has no extra model, and on workloads that quote their input it can be remarkably effective. If it does nothing, that tells you something about the workload before you spend a week on a draft model.

When it pays#

• Small batches. Latency-sensitive traffic, one request at a time or close to it. Decoding is memory-bound and the verify pass is nearly free.
• Predictable text. Structured output, JSON, code, repeated boilerplate, answers that copy from context. Acceptance rates climb and every accepted token is a win.
• Long outputs. The per-request setup cost is amortised over many tokens.

When it does not#

• Large batches. Once the GPU is compute-bound, the verify pass is no longer close to free and the draft work competes with real requests. Throughput can fall.
• High-entropy text. Creative writing, open-ended chat, anything where the next token is genuinely uncertain. Acceptance drops and the wasted draft work dominates.
• Draft and target disagree. A draft model with a different tokeniser, or trained on different data, will propose tokens the target rejects. Acceptance rate tells you this within a minute of testing.

A rough rule we use: if acceptance is comfortably above about two thirds at your real batch size, it is worth it. Below half, switch it off. In between, measure per-token latency and throughput together, because they will move in different directions.

Switching it on#

The flags move between versions, so treat this as the shape rather than the incantation. Recent vLLM takes the configuration as a single JSON argument.

# N-gram lookahead: no extra model, propose up to 5 tokens from prompt spans.
vllm serve /models/your-finetune \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 5, "prompt_lookup_max": 4}' \
  --max-num-seqs 16

# Draft model: same family, same tokeniser, a fraction of the size.
vllm serve /models/target-8b \
  --speculative-config '{"model": "/models/draft-0.5b", "num_speculative_tokens": 4}' \
  --max-num-seqs 16

Start with num_speculative_tokens between three and five. More is not better: each extra proposed token has a lower chance of acceptance, and the whole chain is discarded at the first rejection.

The harness#

We do not trust the metrics endpoint alone. We measure from the client, with the real prompts, at the real concurrency, because that is what the user sees. The script below streams a batch of requests, records time to first token and inter-token latency, and prints the acceptance rate from the server's stats if it exposes them.

import asyncio, time, statistics, json
from openai import AsyncOpenAI
 
client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="local")
 
async def one(prompt: str, model: str):
    t0 = time.perf_counter()
    first, stamps = None, []
    stream = await client.chat.completions.create(
        model=model, messages=[{"role": "user", "content": prompt}],
        max_tokens=400, temperature=0, stream=True,
    )
    async for chunk in stream:
        now = time.perf_counter()
        if chunk.choices and chunk.choices[0].delta.content:
            if first is None:
                first = now - t0
            stamps.append(now)
    gaps = [b - a for a, b in zip(stamps, stamps[1:])]
    return first, gaps
 
async def main(prompts: list[str], model: str, concurrency: int):
    sem = asyncio.Semaphore(concurrency)
    async def guarded(p):
        async with sem:
            return await one(p, model)
    results = await asyncio.gather(*(guarded(p) for p in prompts))
    ttft = [r[0] for r in results if r[0] is not None]
    itl = [g for r in results for g in r[1]]
    print(f"TTFT p50 {statistics.median(ttft)*1000:.0f} ms   "
          f"ITL p50 {statistics.median(itl)*1000:.1f} ms   "
          f"ITL p95 {sorted(itl)[int(len(itl)*0.95)]*1000:.1f} ms")
 
if __name__ == "__main__":
    prompts = [json.loads(l)["prompt"] for l in open("real_prompts.jsonl")]
    asyncio.run(main(prompts, model="your-finetune", concurrency=4))

Run it twice: once with speculation off, once on, at the concurrency you actually serve. Then read the server metrics for acceptance. If the p50 inter-token latency improved and throughput did not fall, ship it. If only one of those is true, you have a decision to make rather than a result.

What we look at in the metrics#

• Acceptance rate, per position if the server exposes it. A steep fall-off after position two means shorten the proposal length.
• Draft time as a share of step time. If the draft is eating more than the tokens it wins, the draft is too big.
• Throughput at the target batch size, not at batch size one. Batch one always looks good.

Two things that bit us#

Structured output with a grammar. Constrained decoding and speculation interact. Some stacks fall back to plain decoding when a grammar is active, silently. Check that the acceptance counters are actually moving in that mode.

Tokeniser mismatch. A draft model that "is the same family" but was released with a slightly different tokeniser will produce acceptance rates that look like a bug. It is not a bug. Check the vocabulary sizes match before you do anything else.

Speculative decoding is a good tool. It is just not a default. Measure your own traffic, keep the harness in the repo, and re-run it every time the model or the workload changes.