TCRA - LLM : Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction (2023)

Abstract: Since ChatGPT released its API for public use, the number of applications built on top of commercial large language models (LLMs) increase exponentially. One popular usage of such models is leveraging its in-context learning ability and generating responses given user queries leveraging knowledge obtained by retrieval augmentation. One problem of deploying commercial retrieval-augmented LLMs is the cost due to the additionally retrieved context that largely increases the input token size of the LLMs. To mitigate this, we propose a token compression scheme that includes two methods: summarization compression and semantic compression. The first method applies a T5-based model that is fine-tuned by datasets generated using self-instruct containing samples with varying lengths and reduce token size by doing summarization. The second method further compresses the token size by removing words with lower impact on the semantic. In order to adequately evaluate the effectiveness of the proposed methods, we propose and utilize a dataset called Food-Recommendation DB (FRDB) focusing on food recommendation for women around pregnancy period or infants. Our summarization compression can reduce 65% of the retrieval token size with further 0.3% improvement on the accuracy; semantic compression provides a more flexible way to trade-off the token size with performance, for which we can reduce the token size by 20% with only 1.6% of accuracy drop.

## introduction with the increase in computing power and accumulation of enormous text data, large language models (llms) such as chatgpt (openai, 2023b) and gpt-4 (openai, 2023a) have shown impressive performance in dialogue-based question-answering (qa), allowing them to interact with users fluently. in open-domain qa where the models are engaged corresponding author. in casual conversations with users, llms exhibit astonishing performance by leveraging strong incontext learning ability. however llms may produce vague responses or incorrect answers in certain specialized domains, owing to the absence of relevant knowledge or a restricted scope of information acquired during the training stage, which might potentially result in untruthful answers and even cause physical damages to users @xcite . for qa in such domains, retrievalaugmented generation (rag) @xcite ), where the system retrieves external knowledge beforehand and then utilizes llms to generate answers leveraging retrieved knowledge, can greatly reduce the hallucinations generated @xcite . many current commercial llms are black-box models, where the model architectures and the weight information are not disclosed. these llms own superior text comprehension abilities, yet in many cases they can only output desired answers through complicated prompt engineering. on the other hand, deploying open-source llms to local servers is resource-intensive, in contrast to deploying smaller models such as t5 @xcite . some commercial llms like gpt-3.5-turbo (ope-nai, 2023c) and gpt-4 offer access through api calls; however, these models charge users based on the size of input and output foot_0 . for individuals or companies looking to create their own services using llms through api calls, utilizing commercial ones can be resource-consuming if requests are made frequently. therefore, it is necessary to minimize the number of input tokens while maintaining optimal performance during the api calls. in this work, we propose a token compression scheme specifically designed for the retrievalaugmented llms (shown in figure 1 ), namely, token compression retrieval augmented large language model (tcra-llm). our proposed scheme can reduce up to 65% of the token size with additional 0.3% improvement on accuracy when doing qa on our proposed dataset called food-recommendation db (frdb). we propose two approaches to reduce the token size of the llms' input: summarization compression and semantic compression. for summarization compression, we leverage self-instruct @xcite scheme to build multiple summarization datasets with varying lengths to fine-tune the mt5 model @xcite . the samples from the summarization datasets are generated by gpt-3.5-turbo (openai, 2023c), which is instructed to shorten the summary of the input sentences in an iterative manner. the semantic compression approach is based on a simple yet effective intuition, that removing semantically less important words in a sentence won't drastically change its semantic. here, we deploy a multi-lingual sentence-transformer (reimers and gurevych, 2020) to encode sentences into embeddings where the distances between original and perturbed embeddings are used to measure the semantic deviation from the original meaning. larger semantic deviation indicates that the corresponding word owns more important semantic in the sentence. we conduct an iterative process that measures the semantic importance of each word in the sentence and remove less important words. in conclusion, our work has the following three contributions: ## related work llms such as gpt-3 @xcite , palm @xcite , opt @xcite , @xcite , and llama @xcite are trained on massive amounts of data and have demonstrated powerful comprehension capabilities. these models have been deployed in a breadth of tasks and achieve promising results @xcite @xcite @xcite . one major barrier that prevents more people from participating in commercial deployment of the llms is their training and hosting costs. a way to reduce such costs is through training smaller domainspecific models such as biomedlm @xcite , bloomberggpt @xcite , and lawgpt @xcite . such domain-specific training enables smaller llms to be applied to certain fields but still requires huge investment. for instance, bloomberggpt is trained on 512 40gb a100 gpus with the total budget being approximately $2.7 million @xcite . alternatively, llms can be used without finetuning through retrieval augmentation leveraging external data sources, where the retrieved data is used as supplementary information to help llms improve logical reasoning and language generation @xcite . previous experiments @xcite show that additional information can be beneficial for llms across different model sizes. retrieval augmentation eliminates the cost of tuning an in-house llm on new data, and can be easily integrated with commercial llm services such as chatgpt (openai, 2023b) from openai or bard @xcite from google. many studies have shown, applying retrieval augmentation to the commercial llms such as chatgpt allow the models to gain knowledge in specific domains such as natural science and medicine @xcite which is not revealed during their training and retrieval augmentation can be further improved by applying more sophisticated retrievers @xcite . however, commercial llms all have a limitation on input lengths which put an upper ceiling on the amount of information that can be fed into a llm. later models such as gpt-4 has looser restriction but the inference cost increases drastically in comparison with other models. some previous work applies template-based prompt optimization @xcite , which select retrieved context @xcite in an adaptive manner, or uses cascading of llms with different sizes @xcite to reduce the inference costs. our proposed method has no conflict with these methods and can be used with them simultaneously. ## frdb we build a food recommendation dataset in chinese called frdb, for recommending foods that are safe to consume for women before/during/after their pregnancy as well as infants. it contains two parts: multiple-choice (mc) qa pairs and a knowledge database. the qa pairs contain 1,000 samples that cover 200 types of food. the categories of foods are shown in the possible answers to the question falls into three choices based on the increasing degree of recommendations ranging from 1 (avoid) to 3 (highly recommend). each type of food has five recommendation rating corresponding to five groups: pre-pregnancy, pregnancy, postpartum, lactation, and infant. additionally, we build a knowledge database that contains 7,588 entries; details of the entries are shown in ## method typically, a retrieval-augmented llm consists of three components (shown in figure 1 ), a knowledge database, a retriever, and the llm. the knowledge ## conclusion in this paper, we propose two methods that reduce the token size for retrieval-augmented llm. additionally, we propose a food recommendation dataset contains domain-specific knowledge to benchmark the performance of retrieval-augmented llm performance on the gpt-3.5-turbo. we carefully select a subset that focuses on 200 types of food recommendations for maternity and infant people. without retrieval augmentation, the commercial gpt-3.5-turbo model is only able to get