GPTC ache: An Open-Source Semantic Cache for LLM Applications Enabling Faster Answers and Cost Savings (2023)

Abstract: The rise of ChatGPT1 has led to the development of artificial intelligence (AI) applications, particularly those that rely on large language models (LLMs). However, recalling LLM APIs can be expensive, and the response speed may slow down during LLMs’ peak times, causing frustration among developers. Potential solutions to this problem include using better LLM models or investing in more computing resources. However, these options may increase product development costs and decrease development speed. GPTCache2 is an open-source semantic cache that stores LLM responses to address this issue. When integrating an AI application with GPTCache, user queries are first sent to GPTCache for a response before being sent to LLMs like ChatGPT. If GPTCache has the answer to a query, it quickly returns the answer to the user without having to query the LLM. This approach saves costs on API recalls and makes response times much faster. For instance, integrating GPTCache with the GPT service offered by OpenAI can increase response speed 2-10 times when the cache is hit. Moreover, network fluctuations will not affect GPTCache’s response time, making it highly stable. This paper presents GPTCache and its architecture, how it functions and performs, and the use cases for which it is most advantageous.

## introduction since openai released chatgpt, large language models have impressed many people and have been frequently integrated into our daily work and lives. at the same time, more open-source enthusiasts and tech companies have invested time and effort into developing open-source llms, such as meta's llama @xcite ), google's palm @xcite , stanford's alpaca @xcite , and databrick's dolly @xcite . there are two ways to use large language models: online services provided by companies like openai, claude, and cohere or downloading opensource models and deploying them on your servers. both methods require payment. online services charge you based on tokens, while deploying models on your own server requires purchasing specific computing resources. the choice depends on individual needs. while online services are more expensive, they are more convenient and effective and provide a better user experience than deploying models yourself. costs and user experience are two critical considerations for building llm applications. as your llm application gains popularity and experiences a surge in traffic, the cost of llm api calls will increase significantly. high response latency will also be frustrating, particularly during peak times for llms, directly affecting the user experience. gptcache is an open-source semantic cache designed to improve the efficiency and speed of gpt-based applications by storing and retrieving the responses generated by language models. unlike traditional cache systems such as redis, gpt-cache employs semantic caching, which stores and retrieves data through embeddings. it utilizes embedding algorithms to transform the queries and llms' responses into embeddings and conducts similarity searches on these embeddings using a vector store such as milvus. gptcache allows users to customize the cache to their specific requirements, offering a range of choices for embedding, similarity assessment, storage location, and eviction policies. furthermore, gptcache supports both the openai chatgpt interface and the langchain interface, with plans to support more interfaces in the coming months. through experiments using the paraphrase-albert-small-v2 model @xcite to embed input in the onnx runtime environment and running it on a local mac with i7, 4cpu, and 32g memory, the time consumed when hitting the cache is approximately 0.3 seconds. compared to accessing openai chatgpt with an average response latency of 3 seconds, the time consumed is only 1/10. furthermore, no tokens are consumed when hitting the cache. different embedding models and similarity evaluation algorithms must be selected in real development scenarios based on the tolerance for cache errors. even so, the entire consumption time is about 3-4 times faster. ## gptcache: semantic cache for llms the overall workflow of gptcache follows the general cache pattern -attempting to obtain results from the cache before fetching data or processing requests. if successful, the process terminates immediately. otherwise, the processing path is the same as if the cache did not exist. however, before returning, the corresponding results are stored in the cache so that repeated actions will retrieve results directly from the cache next time. using the cache significantly reduces workflow time, which explains why cache designs are ubiquitous in our lives, such as multi-level caches in computers, dns caches in networks, and redis/memcache in management systems. ## experiments to evaluate gptcache, we randomly scrape some information from the webpage, and then let chatgpt produce a corresponding data (similar or exactly opposite). and then we created a dataset consisting of three types of sentence pairs: first, we cached the keys of all 30,000 positive sample pairs. next, we randomly selected 1,000 samples and used their peer values as queries. we found setting the similarity threshold of gpt-cache to 0.7 achieves a good balance between hit and positive ratios. so we used this for subsequent tests. to determine if a cached result is positive or negative to the query, we used the similarity score from chatgpt with a positive threshold of 0.7. we generated this by prompting: please rate the similarity of the following two questions on a scale from 0 to 1, where 0 means not related and 1 means exactly the same meaning. and questions, "which app lets you watch live football for free?" and "how can i watch a football live match on my phone?" the similarity score is. we issued 1,160 queries with 50% positive and 50% unrelated negative samples. next, we tried to also cache all negative samples and queried with their peers. surprisingly, despite high chatgpt similarity scores (over 0.9) for some pairs, none hit the cache. the cause of the cache error could be the similarity evaluator's fine-tuning on this dataset correctly undervalued the similarity of negative pairs. the initial experiments demonstrate that gpt-cache can effectively utilize semantic similarity to cache llm query-response pairs and achieve significant speedups. we plan to conduct more rigorous evaluations on larger and more diverse datasets. when tuning the similarity threshold, further investigation is required to balance cache hits versus false positives. ## future challenges one core factor affecting gptcache's caching effectiveness is the choice of embedding model. compared to other component selections, the choice of embedding model is crucial because subsequent vector database retrieval relies on the embedding vectors. if the vectors cannot adequately capture the features of the input text, the retrieval results will be very noisy or even counterproductive, returning completely irrelevant cached data. our testing has shown that even the best cache hit rates do not exceed 90% with current embedding models. this means that negative cache hits are noticeable during use. while this may not greatly impact individual users, it would be unacceptable in real production scenarios. although other methods, like more strict similarity evaluation, could improve positive cache hit rates, this would also decrease the overall hit rate. most current embedding models are likely optimized for search scenarios but may not work as well for cache matching. for example, results with semantics opposite to the input text are acceptable in search since they have structural similarity, but this is unacceptable in caching scenarios. naturally, how to obtain embeddings suitable for caching is an open area for exploration. even with a suitable embedding model, positive hit rates are unlikely to reach production requirements, such as 99%, without decreasing cache hits. the similarity evaluation module plays a core role in improving positive cache hit rates by filtering incorrect hits. our current implementations include vector distance, retrieving distance, cohere rerank api, and sbert cross-encoder. however, testing shows these methods do not sufficiently distinguish between positive and negative cache hits. to address this, we are using large models to judge sentence similarity and distill them into a small model to obtain a specialized model for textual similarity. as large language models are widely adopted, their supported token counts have increased from 2k initially to 100k. however, if a single input exceeds the llm's token count limit, it cannot process the request. similarly, conversations with total tokens exceeding the limit must drop some information. large token counts from long texts or conversations pose a challenge for caching, making it difficult to identify key information and generate representative vectors. currently, we utilize summary models to pre-process and shorten long inputs, but this approach increases cache instability, and its effectiveness is not optimistic. therefore, special cache lookup methods may be needed for long texts. as mentioned earlier, is there any alternative to retrieving cache data using vector databases? for example, can we use traditional databases like mysql, postgresql, sql server, or oracle to store cache data, with textual pre-rocessing to standardize user inputs? for instance, when the inputs are "tell me a joke" and "i want to get a joke", can we convert them to a certain string, like "tell a joke" , or a same number? cache hits could then utilize string matching or numeric ranges instead of vectors. ## conclusion gptcache is a caching solution tailored for llm applications. it brings the following benefits to the llm app developers: by utilizing semantic similarity search and vector embeddings, gptcache provides an effective caching solution that enhances performance, reduces costs, and improves scalability for applications that use large language models. our initial experiments have shown great potential, and we plan to conduct more comprehensive evaluations on diverse real-world datasets and application scenarios.