A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference (2025)

Abstract: Recently, sharing key-value (KV) cache across layers has been found effective in efficient inference of large language models (LLMs). To systematically investigate different techniques of cross-layer KV sharing, we propose a unified framework that covers several recent methods and their novel variants. We conduct comprehensive experiments on all the configurations of the framework, evaluating their generation throughput and performance in language modeling and downstream tasks. We find that when reducing the size of the KV cache by 2\times , most configurations can achieve higher throughput than standard transformers while maintaining competitive performance.When further reducing the size of the KV cache, however, pairing queries of all layers with KVs of upper layers performs better, at the expense of additional training cost and prefilling latency. We hope that this work will help users make more informed choices of cross-layer KV sharing approaches and facilitate future research on efficient LLM inference.

## introduction a major bottleneck for the deployment of llms is memory consumption, of which the key-value (kv) cache in the transformer architecture occupies a large portion @xcite . various methods have been proposed to reduce the memory consumption of the kv cache in llms. for example, @xcite ; @xcite share the kvs across query heads and @xcite ; @xcite keep the kv cache of only a small portion of tokens. more recently, several methods are proposed in which the kvs are computed only at a subset of transformer layers and shared to the other layers, such as lckv @xcite , yoco @xcite and cla @xcite . these methods not only significantly reduce memory consumption but also improve inference speed, while preserving the performance of llms in language modeling and downstream tasks. however, while all these methods are based on the idea of crosslayer kv sharing, they differ significantly in how the sharing is done. in this study, we consider a unified framework for cross-layer kv sharing, of which lckv, cla, and yoco can be seen as special configurations. we then empirically test all the configurations of the framework, including several novel ones that have never been considered in previous work. our experiments show that, with respect to throughput, all the configurations can achieve significantly higher throughput than the standard transformer when the prompt is short; but when the prompt is long, the throughput of the configurations that compute the kvs at the top layers degrades dramatically. with respect to performance, when only half of the layers rely on the kvs computed by the other layers, the performance of most configurations is comparable with that of the standard transformer; when more layers become reliant on the other layers for the kvs, the configurations that compute the kvs at the bottom layers suffer the greatest performance degradation. we hope our framework and empirical studies would help users interested in cross-layer kv sharing to make more informed choices of methods and configurations according to their throughput and performance requirements. our code is available at https://github.com/whynlp/lckv . ## existing methods layer-condensed kv cache (lckv) @xcite computes the kvs of only the top layer of the transformer, which are paired with queries of all the layers. consequently, lckv omits the kv compu-tation and discards the kv parameters for all the layers other than the top layer. to prevent severe performance degradation, lckv also optionally retains standard attention for a small number of top and bottom layers. you only cache once (yoco) @xcite ) computes the kvs of only the middle layer of the transformer, which are paired with the queries of the top-half of the layers. the bottom-half of the layers uses efficient attention to achieve a constant cache size. @xcite uses a similar sharing pattern to yoco, but further compresses the size of the kv cache. cross-layer attention (cla) @xcite uniformly divides transformer layers into multiple groups of adjacent layers. in each group, it pairs the queries of all the layers with the kvs of the bottom layer. @xcite shares the kvs in the same way as cla, but applies a more efficient training scheme. @xcite groups every two adjacent layers in the middle-to-deep portion and compresses the kv cache in each group. @xcite groups non-adjacent layers and pairs the queries of the upper layer with the kvs of the lower layer in each group. @xcite uses a combination of the sliding window attention and a sharing pattern similar to cla. @xcite ; @xcite ; @xcite apply sharing patterns similar to cla to the computed attention weights instead of kvs. ## a unified framework unifying previous methods, we propose a framework for cross-layer kv sharing that can be applied to any transformer-based model. suppose that the transformer has l layers. we denote kv(i) ∈ {1, ..., l} as the index of the layer whose kvs are paired with the queries of the i-th layer. if kv(i) = i, then layer i is called a kv layer, which computes its own kvs that are paired with its queries just as in a standard transformer. otherwise, layer i does not compute its own kvs and instead uses the kv of layer kv(i) ̸ = i. in this case, we call layer kv(i) the target layer of layer i. since layer i does not need to compute kvs, it does not need weights w k , w v . therefore, the number of kv layers determines the number of weight parameters w k , w v and hence the size of a transformer model. below we define different configurations of our framework assuming the number of kv layers always set to l. we define a configuration by partitioning transformer layers and positioning target layer(s) differently. we choose the layer partitioning from { pizza, sandwich, lasagna } and choose the target layer positioning from { bottom, top, middle } foot_0 . the pizza partitioning sets the first l -1 layers as kv layers. the sandwich partitioning sets the first ⌈ l-1 2 ⌉ layers and the last ⌊ l-1 2 ⌋ layers as kv layers. for the remaining ll + 1 consecutive layers in both pizza and sandwich, their target layer is positioned at either the top, the middle, or the bottom of these layers. the lasagna partitioning uniformly divides the l layers into l groups of consecutive layers. for each group except the first, the target layer of all the layers within the group is positioned at either the top, the middle, or the bottom of these layers. for the first group, however, we always set the bottom layer as the target layer because we empirically find that there is a significant drop in performance if the first layer is not a kv layer. note that for the top and middle positioning of the target layer, there exists a cyclic dependency between the target layer and the lower non-kv layers: for each token, its kvs at the target layer is required for attention computation at lower non-kv layers, but are not computed until computation at all the lower layers is finished. so, we follow @xcite and drop the attention of each token to itself, which is equivalent to masking the diagonal of the attention matrix in each layer. ## experiments we conduct experiments to compare the generation throughput and performance of the standard llama baseline @xcite and the nine configurations with different numbers of kv layers. our implementation is based on huggingface transformers @xcite with kernel replacement with flashattention 2 @xcite , fused rms norm, fused cross-entropy, and fused swiglu. our experiments are conducted on models with 110m and 1.1b parameters, whose configurations are shown in appendix a. we set @xmath0. the sandwich configurations coincide with the pizza configurations when there are only two kv layers and the lasagna-middle configuration coincides with the lasagna-top configuration when the number of kv layers is half of the total number of layers (i.e., 6 and 11 for the 110m and 1.1b models, respectively), therefore omitted in our experiments. ## conclusion in this study, we propose a new framework for llm cross-layer kv sharing that includes previous methods as special cases. we conduct systematic experiments on various configurations of the framework with different kv cache memory budgets and observe their generation throughput and performance in language modeling and downstream tasks. the experimental results show that the pizzabottom and lasagna-bottom configurations can reduce the size of the kv cache by 2× without too much performance degradation or introducing additional training and prefilling time. however, if one wishes to further reduce the size of the kv cache, cares less about additional training time, and needs to generate sequences much longer than prompts, then the sandwich-middle configuration may be a better choice.