T able KV : KV Cache Compression for In-Context Table Processing (2025)

Abstract: Processing large tables provided in-context to LLMs is challenging due to token limits and information overload. While Retrieval-Augmented Generation can select relevant subsets externally, this work explores Key-Value (KV) cache compression as an alternative, applied directly to the linearized table during inference. We show that the LLM’s internal attention scores over the table context guides the retention of essential KV pairs, effectively compressing the processing context while preserving crucial relational information needed for complex queries. Experiments on Spider, WikitableQA, and QTSumm datasets validate the compression approach for in-context table processing, offering a promising path for improved table representation learning in LLMs.

## introduction large language models (llms) have demonstrated remarkable capabilities across language tasks. a promising frontier is enabling llms to reason over structured data, such as tables, alongside natural language. this ability is key for applications such as table question answering (tableqa) @xcite and fact-checking using relational data @xcite . while generating sql queries from text (text2sql) is a popular approach @xcite , directly processing tabular data within the llm's context offers a unified framework, leveraging the model's abilities to handle nuances beyond sql's scope @xcite . however, directly feeding large tables into llms faces significant issues. the primary challenge is context length: even moderately sized tables (e.g., thousands of rows and tens of columns) linearized into text easily exceed the token limits of popular models (e.g., >120,000 tokens) @xcite . consequently, full-table inputs are often impractical, necessitating truncation or retrieval mechanisms @xcite . even when models accommodate large contexts, their reasoning accuracy often degrades substantially @xcite . this phenomenon is exacerbated by tables, which inherently mix relevant cells with irrelevant information, diluting the model's attention @xcite . capturing relational patterns that span disparate rows or columns is particularly difficult, hindering accurate aggregation or multi-step reasoning. existing solutions often rely on retrieval-augmented generation (rag) @xcite . while rag effectively reduces the input length by pre-selecting relevant table chunks (rows, columns, or cells), it introduces its own complexities. first, it requires separate retrieval modules and deciding how to optimally partition and retrieve table segments (e.g., by row, column, or semantic blocks) @xcite . second, relying on embedding similarity for retrieval might fail to capture the fine-grained relational dependencies, shifting the bottleneck to the retriever's effectiveness. in this paper, we explore an alternative approach: leveraging key-value (kv) cache compression techniques, originally developed for general text inference @xcite , to handle large tabular data directly within the llm's inference process. our core hypothesis is that the llm's own attention mechanism, as it processes the linearized table, inherently identifies the most salient information. we exploit these attention scores to dynamically prune the kv cache, retaining only the key-value pairs corresponding to the most attended-to tokens. this effectively compresses the table's representation, making the information from the original tables available during inference, mitigating information overload while avoiding the complexities of explicit retrieval. our experiments across datasets, including spider @xcite , wikitableqa @xcite , and qtsumm @xcite , demonstrate the viability of this approach. related work. rag methods retrieve relevant subsets of tabular data to reduce input complexity. tablerag @xcite employs schema and cell retrieval techniques, while tap4llm @xcite uses sampling strategies to focus the model's attention on relevant subsets of data. specialized encoding techniques tailored for structured inputs have also gained attention. spreadsheetllm @xcite exploits structural redundancies within tabular data, compressing input lengths without losing semantic fidelity. however, retrievalbased methods often fall short in capturing the comprehensive relational contexts that are required to handle queries involving multiple tuples. although kv cache compression methods @xcite have demonstrated significant context compression by retaining subsets of relevant tokens, these techniques have yet to be adapted for structured data. this work, therefore, represents the first exploration of kv cache compression tailored to tabular inputs. ## background given a sequence of n tokens @xmath0, each transformer layer produces hidden representations via a multi-head self-attention mechanism: with h representing the hidden states (token embeddings) for the input sequence. the dimension d k is d h where d is the hidden size and h is the number of attention heads. most llms organize their input as a context followed by a prompt. let x denote a sequence of tokens and the input sequence: to introduce compression, we detail a queryaware approach that compresses the kv cache by retaining only the most relevant kv vectors for the query given at inference time @xcite . let m be the chunk length, and let {c 1 , c 2 , . . . } be the segments obtained by slicing the input table x (cont) . at iteration i, the method takes as input after processing all chunks, the final k, @xmath1, at a reduced length. agnostic methods use similar principles but in a single offline computation of the cache, thus without making use of the query. for example, ada expected attention scores are computed by modeling the distribution of queries and estimating their interaction with key vectors at future positions @xcite , in conjunction with head-specific compression @xcite . ## kv compression for tables handling structured data in llms remains a significant challenge due to the quadratic complexity of self-attention and limited context windows. another challenge is capturing interconnections between tuples. for example, consider a table containing sales data. answering a query such as select region, sum(sales) from table group by region requires capturing information across multiple tuples. retrieval methods may fall short by only selecting isolated tuples or columns, missing the holistic relational context necessary for accurate aggregations. kv cache compression, initially introduced for general llm inference, presents a promising opportunity for tabular data. the key insight of kv compression is straightforward: after linearizing a structured table into a textual representation, standard mechanisms within transformers naturally encode relevance and information importance within attention scores. when selecting kv vectors from the cache, these vectors inherently contain latent information representing broader relational context, including information from vectors that have been evicted. thus, rather than employing separate retrieval systems or encoding mechanisms tailored specifically for tables, we hypothesize that llms themselves inherently identify critical elements of linearized tables directly through attention patterns. we explore two types of compression for linearized tabular data. (1) query-aware compression dynamically compresses the cache during inference by retaining kv vectors with the highest attention scores relative to a specific question. (2) queryagnostic compression pre-computes a representation of the cache independently of a specific query, capturing general information from the table. ## results in tables 1a and 1b , we report results for all methods on wikitableqa and qtsumm, respectively. based on the tables' average length in each dataset, we select different target context lengths, obtaining compression rates between 1.7x and 51.39x (average context length in spider is 13158 tokens). overall, kv compression methods outperform the rag-based approach in most scenarios, and in several cases, achieve better results than the fullcontext setup. finch achieves strong results on wikitableqa: with llama-3.1-8b-instruct, it obtains an accuracy of 35.11 at a compression rate of 1.7× (1024 tokens), which is significantly higher than the other approaches, including full context. with qtsumm, compression methods yield results that are either better or very close to those of the full-context case, with compression (e.g., 30.61 for finch with llama-3.1-8b-instruct and 30.01 for ada with qwen-2.5-7b). in the spider dataset's results in figure 2a , queryaware compression reports promising results for llama, outperforming the row retrieval-based strategy in all cases. moving to spider on qwen in figure 2b , rag and ada (agnostic) are more competitive and even surpass the full-context scenario. in this scenario, finch reports strong performance in terms of precision, recall, and tuple constraint, but lower scores for execution accuracy. in terms of execution-time, query-agnostic kv cache compression delivers faster inference than rag, while query-aware compression similarly to full-context decoding @xcite . ## conclusion and future work this work shows that kv cache compression can outperform rag and match full-context performance, offering a promising technique for processing tables directly within llms. future research includes developing table-specific compression strategies beyond adapting existing methods and investigating the interplay between table/query complexity and compression effectiveness. finally, we plan to examine hybrid approaches combining kv compression with lightweight retrieval.