Position ID s Matter: An Enhanced Position Layout for Efficient Context Compression in Large Language Models (2025)

Abstract: Using special tokens (e.g., gist, memory, or compressed tokens) to compress context information is a common practice for large language models (LLMs). However, existing approaches often neglect that position encodings inherently induce local inductive biases in models, causing the compression process to ignore holistic contextual dependencies. We propose **Enhanced Position Layout (EPL)**, a simple yet effective method that improves the context compression capability of LLMs by only adjusting position IDs, the numerical identifiers that specify token positions. EPL minimizes the distance between context tokens and their corresponding special tokens and at the same time maintains the sequence order in position IDs between context tokens, special tokens, and the subsequent tokens. Integrating EPL into our best performing context compression model results in 1.9 ROUGE-1 F1 improvement on out-of-domain question answering datasets in average. When extended to multimodal scenarios, EPL brings an average accuracy gain of 2.6 to vision compression LLMs.

## introduction in transformer @xcite architectures, special tokens have been widely adopted as compression carriers of contextual information across natural language processing @xcite @xcite @xcite and computer vision @xcite . for context compression, the so-called soft prompt methods @xcite employ encoders to condense long contexts into few special tokens, enabling decoders to perform inference based on compressed representations rather than raw inputs, thereby significantly reducing memory consumption and inference latency in long-context scenarios @xcite @xcite . we illustrate typical soft prompt architectures in figure 1 where special tokens are appended at the end of the context, in order to capture the context semantics via causal attention mechanism in llms. the design ensures full context visibility for special tokens. however, in transformer architectures position ids do not need to coincide with physical token positions and the model's perceived positional information is primarily determined by position ids rather than physical token positions @xcite . from this viewpoint, the local inductive biases introduced by position encodings @xcite @xcite @xcite weaken the efficacy of context compression under the default position layout (dpl), due to the substantial distance between the special tokens and the context tokens, as shown in figure 1 (dpl). in this paper, we carefully examine position layout designs and propose enhanced position layout (epl) for soft prompt architectures, which comprises uniform position layout (upl) and consistent position layout (cpl). upl redistributes special tokens' position ids to achieve uniform distribution in the context tokens' position id space, as exemplified in figure 1 . by uniformly assigning position ids among context token position ids, a priori, most context tokens would have corresponding special tokens close to them. we assume that such a prior helps the special tokens compress the context. we formalize such intuitions, demonstrating the optimality of the upl in section 3.2.1. during compression, because special tokens are inserted and text chunks are reorganized, the position ids between context, special tokens and subsequent tokens (e.g. reconstructed tokens or subsequent tokens such as qa pairs) can become inconsistent compared to their original positions before compression. our proposed cpl in section 3.2.2 guarantees to maintain the order of the position id sequence for different tokens, between different text chunks in their natural causal order. we empirically apply epl to two dominant context compression frameworks: icae @xcite and 500xcompressor @xcite . for the best model, on the autoencoding (ae) task, epl yields a 1.8 bleu gain and converges 9.7 times faster than dpl; on out-of-domain question answering (qa) tasks, epl gives an average 1.9 rouge-1 f1 gain. to demonstrate the universality of epl, we further conduct experiments by incorporating epl into a more recent fine-grained compression framework @xcite and test our epl equipped compressor on pwc dataset @xcite , an instruction-tuning dataset containing diverse context-related prompts. in both scenarios, incorporating epl brings consistent improvement. when extending our application to multimodality with voco-llama @xcite , epl yields an average 2.6 accuracy gain on multimodal benchmarks. the improvement of epl is consistent across base models of different scales. further analysis shows that both upl (which aims for better context compression) and cpl (which maintains causal sequence ordering) are essential for the final performance improvement across tasks. finally, our upl attention map visualization confirms the usefulness of our specified prior: upl special tokens indeed focus more on tokens close to its assigned position ids. ## method in section 3.1, we review existing llm compression frameworks and their dpls; in section 3.2 we describe our improved position layout. @xcite is a widely used encoderdecoder soft prompt method. its encoder compresses long context into a few memory tokens, after which the decoder performs inference con-ditioned only on the memory tokens to achieve faster inference speed. the left side of figure 3 in the appendix illustrates this process through an example. icae can be trained through two stages: continued pretraining and fine-tuning. continued pretraining trains on a combination of autoencoding (ae) tasks and language modeling (lm) tasks, which trains the llm encoder so that the encoded memory tokens enable a frozen llm to losslessly reconstruct the original context and at the same time predict the subsequent tokens following the context to maintain icae's generation capability. the right side of figure 3 illustrates the ae training process. fine-tuning further trains the icae encoder to adapt to real-world applications such as question answering (qa) that we use in this work. we review the ae and lm pretraining as well as qa fine-tuning in the next subsections. to handle arbitrarily long contexts, we adopt the multichunk version of icae, which divides any long context into chunks for independent compression and then aggregates the resulting memory tokens to represent the complete long context. 500xcompressor @xcite is similar to icae, with the difference of using the kv cache of memory tokens as the compression carrier instead of the output of memory tokens. ## conclusion we examine position layout, an understudied topic in context compression, and propose epl for soft prompt methods. epl improves over default position layout by bringing memory tokens close to its context and at the same time maintains the logical sequence token ordering among context, memory tokens and the subsequent tokens. extensive experiments show epl enhances compression efficiency and downstream performance across architectures and modalities. given llm's widespread use, we believe examining position layout will benefit problems beyond context compression by adopting a prior tailored to tasks that can be expressed in the position layout. we hope epl's success fosters research in this underexplored area.