You Only Cache Once: Decoder-Decoder Architectures for Language Models

Published on May 8


We introduce a decoder-decoder architecture, YOCO, for large language models, which only caches key-value pairs once. It consists of two components, i.e., a cross-decoder stacked upon a self-decoder. The self-decoder efficiently encodes global key-value (KV) caches that are reused by the cross-decoder via cross-attention. The overall model behaves like a decoder-only Transformer, although YOCO only caches once. The design substantially reduces GPU memory demands, yet retains global attention capability. Additionally, the computation flow enables prefilling to early exit without changing the final output, thereby significantly speeding up the prefill stage. Experimental results demonstrate that YOCO achieves favorable performance compared to Transformer in various settings of scaling up model size and number of training tokens. We also extend YOCO to 1M context length with near-perfect needle retrieval accuracy. The profiling results show that YOCO improves inference memory, prefill latency, and throughput by orders of magnitude across context lengths and model sizes. Code is available at


The evaluation benchmarks use zero-shot, where usually few-shot is used. This raises the question whether the few-shot results weren't as good, compared to similar-sized models.

Hopefully the authors will release few-shot results, aligned with common practice (e.g. HF Open LLM Leaderboard)


We report zero-shot numbers so that we can directly compare the results with stablelm and openllama-v2, which all follow the zero-shot protocol. The trend of few-shot results is similar to zero-shot.

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite in a model to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite in a dataset to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite in a Space to link it from this page.

Collections including this paper 8