Free speed: speculation, grammars and caches
Free speed in AI inference refers to optimisations that improve performance without compromising output quality. These methods focus on reducing redundant computations or reusing prior results in ways that remain semantically sound. The techniques discussed here are used in production systems to accelerate local inference while maintaining correctness.
Speculative decoding and batched verification
Speculative decoding improves efficiency by amortising computational cost. A cheaper “drafter” model proposes multiple tokens, which a larger, more accurate model then verifies in a single batched pass. This process ensures that output quality remains intact because verification is exact. In models using MoE (Mixture-of-Experts), this approach gains an additional performance benefit, as a batch of positions can reuse each fetched expert, reducing redundant computation.
Multi-token prediction and quantisation pitfalls
Multi-token prediction heads are often built into models as auxiliary components. These heads propose token pairs, with measured acceptance rates ranging from 30 to 60 percent when properly configured. A known pitfall in the community involves mis-quantisation of these heads, which can silently reduce acceptance rates to near zero. Diagnosis of such issues requires careful measurement, as the problem is not visually apparent.
Grammar-forced drafting for deterministic spans
When model outputs must conform to a known structure, such as JSON or NDJSON, the grammar can make many output spans deterministic. These spans can be drafted for free and then verified losslessly. An implementation by the course instructor, merged upstream in Colibri PR #70, achieved a speedup of 0.37 to 0.50 tokens per second with 100 percent draft acceptance and byte-identical outputs. A schema-to-grammar compiler is currently under development in PR #192.
Persistent KV caching and prompt reuse
Prefix-KV caching prevents re-computation of earlier tokens in a sequence. When a model processes a long system prompt, the KV cache allows it to resume from a previous state instantly across restarts. This method has been measured to resume 552 tokens in effectively zero time. For workloads involving repeated prompts, such as batch scoring, this is one of the most cost-effective performance gains.
Semantic caching and safety in production
Semantic caching operates above the inference engine by reusing previous outputs for queries that are nearly identical. In production, this is implemented with a conservative one-sided policy: only safe negatives are reused, and a sample of outputs is audited. The instructor’s monitoring pipeline has measured this approach and found zero actionable losses from cache hits. This method demonstrates how safety-critical systems can also improve performance.
What to take away
These optimisations show that performance gains and safety can align through shared mechanisms like exact verification and audited reuse. The techniques described are not only about speed, but also about building robust systems that maintain fidelity under various conditions. Their design reflects the convergence of performance engineering and AI safety principles.
Reference
| Lesson | 10 of 15 |
| Outcome | Explain the lossless acceleration stack for local inference. |
| Worked engine | Colibri disk-streaming MoE engine |
| Gap tracking | Epoch AI open-vs-closed analysis |