Cache Saver: A Modular Framework for Efficient, Affordable, and Reproducible LLM Inference (2025)

Abstract: Inference constitutes the majority of costs throughout the lifecycle of a large language model (LLM). While numerous LLM inference engines focusing primarily on low-level optimizations have been developed, there is a scarcity of non-intrusive client-side frameworks that perform high-level optimizations. In this paper, we introduce Cache Saver, a modular, plug-and-play, and asynchronous framework that facilitates high-level inference optimizations, thereby integrating cleanly into existing systems without requiring changes to the end-user application logic or the underlying LLM. The key novelty is a *namespace-aware list-valued cache* that ensures *statistical integrity* of LLM responses by generating *i.i.d.* responses within a namespace as well as ensuring *reproducibility*. Moreover, as a direct consequence of operating at a high level, Cache Saver supports both local and online models. We conduct extensive experiments with five representative state-of-the-art reasoning strategies, five diverse benchmark tasks, and three different LLMs. On average across all methods, tasks, and LLMs, Cache Saver reduces cost by \simeq 25\% and CO _2 by \simeq 35\% . Notably, Cache Saver excels in practical machine learning scenarios such as benchmarking across multiple methods or conducting ablation analysis of a specific method, obtaining substantial cost and carbon footprint reduction of \simeq 60\% . Cache Saver is publicly available at [https://github.com/au-clan/cachesaver](https://github.com/au-clan/cachesaver).

## introduction large language models (llms) have taken the world by storm. most mainstream web applications (e.g., facebook, twitter/x, whatsapp, reddit, google/bing search, etc.) now offer some form of llm-based companion or assistant. not surprisingly, llms are estimated to account for ≃ 2% of global electricity consumption and greenhouse emissions @xcite , a figure projected to rise to ≃ 8% by 2030 @xcite . most costs in an llm's lifecycle come from inference @xcite , which is expensive, especially with the rise of autonomous agents @xcite , test-time scaling @xcite , multistep reasoning strategies @xcite @xcite , and native reasoning models @xcite . for instance, openai's o3 costs ≃1000$ per task on the arc-agi benchmark @xcite . ## related work backend optimizations. llm serving engines optimize inference on the server side, with many optimizations focusing on efficient memory management and kv cache reuse @xcite @xcite @xcite @xcite . pagedattention treats the attention key-value cache like a vir-tual memory system, reducing fragmentation and allowing kv cache sharing within and across requests @xcite . kvquant reduces memory usage by quantizing the kv cache @xcite . grouped query attention improves memory and computation efficiency by grouping queries to reduce redundant attention computations @xcite . car introduces a multi-tier caching system to facilitate the reuse and sharing of attention kv caches across different requests @xcite . additional optimizations target prompt batching and scheduling efficiency @xcite @xcite @xcite , as well as improvements in attention and decoding operations @xcite @xcite . for example, flashattention-2 enhances attention computation through improved parallelism and work partitioning @xcite . most production-grade llm serving systems, such as vllm @xcite , hugging face transformers @xcite , and nvidia triton inference server (triton), incorporate several such optimizations. other engines, including deepspeed-fastgen @xcite , llama.cpp (ggml.ai, 2023), mlc llm (mlc-ai, 2023), sglang @xcite , and lightllm (lightllm team, 2023), further specialize for low-latency decoding, efficient edge deployment, and server-side batching and memory management. overall, these optimizations remain largely invisible to the practitioners and aim to reduce costs for the model providers. however, they may increase opacity, potentially client-side caching. client-side caching addresses the challenges of reducing redundant requests, minimizing latency, and lowering operational costs. while retrieval-augmented generation (rag) methods reduce the need for llms to generate answers from scratch by retrieving relevant information from external sources @xcite , their primary focus is not caching. instead, we examine various client-side caching systems that aim to optimize llm performance. helicone implements a key-based caching mechanism by storing llm responses at the edge using cloudflare workers (helicone). gptcache is an open-source semantic cache that transforms queries and llm responses into embeddings, conducting similarity searches to retrieve cached responses @xcite . developer frameworks like langchain offer a framework-specific caching layer that supports both exact and semantic matching (langchain). gpt semantic cache utilizes semantic embedding caching by storing query embeddings in in-memory storage systems, such as redis @xcite . meancache introduces a user-centric semantic caching system that preserves privacy through a learned federated similarity model @xcite . generativecache goes beyond traditional caching by synthesizing new responses from multiple cached entries @xcite . llmproxy is a proxy service that implements a single endpoint for applications, supporting model selection, context management, and caching @xcite . in summary, clientside caching solutions for large language models (llms) are commonly used, primarily addressing generic workloads through semantic caching @xcite . key differences. existing client-side solutions often lack guarantees for statistical integrity and reproducibility. to the best of our knowledge, by employing a namespace-aware list-valued cache, cache saver is the only framework that enables response reuse without sacrificing the statistical integrity of the generative llm. 3 cache saver overview. cache saver is a modular framework composed of three key modules: (1) a batcher ( § 3.2), (2) a deduplicator ( § 3.2), and (3) a cacher ( § 3.1). fig. 3 presents an overview of its inner workings using a toy example, where three search strategies, independently exploring a dynamic search tree in their own namespace (ns 1 through ns 3 ), request responses for the same prompt 𝑄. fig. 3 (top) illustrates the cold-start scenario, where the cache is empty. the asynchronous batcher collates the five incoming requests into a single batch and passes it to the deduplicator, which groups identical prompts for each namespace and emits two aggregate requests, requesting three and two i.i.d. responses to 𝑄 for ns 1 and ns 2 , respectively. the cacher, which relies on a system of asynchronous mutexes to avoid redundant or overlapping requests, receives these two aggregate requests and sends a single request to the llm, asking for three responses to 𝑄. finally, the llm responses ([𝑅 1 , 𝑅 2 , 𝑅 3 ]) are stored in the cache and used to resolve the requests from ns 1 and ns 2 . in the warm-start scenario (fig. 3 bottom), an aggregate request asking for four i.i.d. responses to 𝑄 is triggered from ns 3 , however, since three responses are already cached, the cacher asks for one additional response from the llm and stores it in the cache. the four requests are then resolved by serving ([𝑅 1 , 𝑅 2 , 𝑅 3 ]) from the cache along with the newly generated response (𝑅 4 ). for additional details (pseudocode, practical considerations: cache eviction, consistency, etc.), please see appx. a. ## experiments we assess the effectiveness of cache saver through extensive experiments and analyses comprising 6 reasoning strategies, 5 benchmark tasks, and 3 llms. additional details, e.g., implementation, hyperparameters, additional results, etc. are presented in appx. c. the resources for reproducing our experiments are available at https:// github.com/au-clan/cachesaver .