Memory & inference / Report summary

Optimizing Memory for Fast Inference

Long-context simulations reveal why substantial decode gains can have limited impact on total inference time.

November 2026 · 11 pages · 1.3 MB · Tolaga Research

Download full report (PDF) ↓

Executive summary

In Tolaga's long-context simulations, reducing weight and KV-cache precision cut decode latency by up to 42.7%, yet delivered only modest prefill improvements. Because prefill dominated these requests, a substantial decode improvement did not translate into a comparable reduction in total response time.

The study also found that doubling projection and feed-forward arithmetic throughput produced negligible improvement in time to first token, while numerical conversion overhead could erase the expected gains. These results show why optimization decisions need to account for the operations that dominate the complete request, rather than focusing solely on hardware specifications or individual stages being accelerated.

Key findings

  • Reducing weight and KV-cache precision produces modeled decode-latency improvements of up to 42.7%; prefill gains are much smaller in the long-context scenarios.
  • Attention dominates the raw compute and memory-transfer demands of the million-token case.
  • Doubling projection and feed-forward arithmetic throughput does not remove the attention bottleneck.
  • Numerical conversion overhead can outweigh the benefit of faster arithmetic.

What the report examines

The baseline is a synthetic approximately one-trillion-parameter MoE model with 96 layers, 386 experts per layer and 16 experts activated per token. It uses grouped-query attention and runs on 64 active accelerators within a 72-accelerator scale-up domain. Each accelerator has 186 GB of HBM and 8 TB/s nominal memory bandwidth.

The study covers input contexts from 2K to one million tokens, batches of 1, 8 and 32, and 512 output tokens. It examines weight and KV-cache precision, separates attention from projection and feed-forward operations, and explores arithmetic throughput, conversion overhead and computational efficiency. Raw compute and memory times are examined separately to expose their relative demands.

Infrastructure implications

Memory capacity determines whether a workload fits, but memory bandwidth and access patterns determine how quickly stored data can be used. Operators should distinguish these constraints before investing in additional capacity or faster devices. Lower-precision storage may release capacity while also reducing transfers, whereas faster arithmetic helps only the operations that use it.

Separating compute-oriented prefill from memory-oriented decode can be attractive, but moving KV state between pools adds communication overhead. The report therefore treats disaggregation as a workload-dependent architectural decision. Shorter prompts or lighter workloads may favor keeping the two phases together.

Assumptions and limitations

These are analytical simulations of a synthetic architecture, not measurements of Kimi K2 or a commercial accelerator deployment. Results depend on attention design, context length, precision and assumed efficiency. Production validation must include conversion costs and the complete serving path.

Source: Optimizing Memory for Fast Inference, PDF pages 2–12 (printed pages 1–11). This summary describes the published report; consult the PDF for figures and references.