Executive summary
Speculative decoding lets a smaller model or specialized prediction mechanism propose several tokens for a larger target model to verify. Accepted proposals reduce the sequential work required from the target. The benefit depends on how accurately and cheaply the draft anticipates the target's behavior.
The report connects token acceptance to end-to-end serving economics. Faster decoding does not accelerate every part of a request: prefill, verification, memory movement, communication and scheduling remain. That distinction explains why a substantial decode-stage improvement can produce a more modest overall throughput gain.
Key findings
- The report's modeled throughput gain rises from approximately 1.22 times at 50% acceptance to 1.67 times at 95% acceptance.
- Draft-model compatibility and workload predictability influence how many proposed tokens are accepted.
- Verification and drafting costs limit the gain even at high acceptance rates.
- Scheduling, batching, rejected-token handling and KV management are part of the implementation.
What the modeled gains mean
The reported acceptance-rate sweep also gives approximately 1.41 times throughput at 70% acceptance and 1.56 times at 85%. These are modeled serving gains, not guarantees for a particular production runtime. They should not be compared directly with a published acceleration figure that measures only the decode stage.
The study assumes decode-stage speedup caps of 3.0 times for structured output, 2.5 times for code and 2.0 times for natural language. Acceleration is applied to the decode-compute portion of the request, leaving other costs in place. That accounting is central to interpreting the smaller end-to-end throughput gains.
Matching the draft and serving system
A draft model can come from the same family as the target, be trained on target outputs, or be purpose-built for short-horizon prediction. The report also discusses approaches with multiple prediction heads or more elaborate drafting strategies. The objective is fast, compatible proposals rather than standalone draft-model capability.
Structured extraction, function calls, templated output and some code-completion workloads can provide more predictable continuations than open-ended conversation. Even for those applications, the benefit depends on the overhead of producing and verifying candidates. A poorly matched draft can add work while contributing few accepted tokens.
Infrastructure implications
Speculative decoding belongs within the serving stack alongside batching, caching and precision choices. Operators should evaluate full-request latency, throughput and cost under representative concurrency, not acceptance rate in isolation. The target model remains in the execution path, so its capacity and memory requirements do not disappear when drafting is introduced.
Assumptions and limitations
Acceptance rates and speedup caps are study assumptions. The modeled results are not measurements of the production implementations discussed in the report. Workload structure, draft cost, verification strategy and serving integration determine whether the expected savings appear in practice.
Source: Speculative Decoding and Inference Throughput, PDF pages 1–5. This summary describes the published report; consult the PDF for figures and references.