On the Role of Long-tail Knowledge in Retrieval Augmented Large Language Models (2024)

Abstract: Retrieval augmented generation (RAG) exhibits outstanding performance in promoting the knowledge capabilities of large language models (LLMs) with retrieved documents related to user queries. However, RAG only focuses on improving the response quality of LLMs via enhancing queries indiscriminately with retrieved information, paying little attention to what type of knowledge LLMs really need to answer original queries more accurately. In this paper, we suggest that long-tail knowledge is crucial for RAG as LLMs have already remembered common world knowledge during large-scale pre-training. Based on our observation, we propose a simple but effective long-tail knowledge detection method for LLMs. Specifically, the novel Generative Expected Calibration Error (GECE) metric is derived to measure the “long-tailness” of knowledge based on both statistics and semantics. Hence, we retrieve relevant documents and infuse them into the model for patching knowledge loopholes only when the input query relates to long-tail knowledge. Experiments show that, compared to existing RAG pipelines, our method achieves over 4x speedup in average inference time and consistent performance improvement in downstream tasks.

## introduction large language models (llms), equipped with retrieval augmented generation (rag), perform well in various tasks @xcite @xcite . rag retrieves supplement knowledge by retrievers and enhances prompts for llms by retrieved documents, in order to generate more accurate contents @xcite @xcite . however, previous rag works concentrate on improving the task performance, without fine-grained process of knowledge @xcite . in this case, redundant computation is performed on well-learned common knowledge, which does not require further enhancement. therefore, more consideration should be given to longtail knowledge that llms really need, which rarely occurs during pre-training @xcite . 1 in the literature, rag can be roughly divided into two categories: (1) once retrieval. building upon our observation, we explore the role of long-tail knowledge in rag. we suggest that long-tail knowledge is crucial for rag and propose an improved rag pipeline. specifically, to measure the "long-tailness" of knowledge in terms of llms, we largely extend expected calibration error (ece) for classification tasks @xcite @xcite , and propose generative expected calibration error (gece). it leverages meteor @xcite and the output probability of llms to characterize "long-tailness", which considers both continuous gradient-based semantics and discrete frequency-based statistics. based on gece, our pipeline retrieves relevant documents and performs rag only when user queries relate to longtail knowledge. our approach outperforms current rag pipelines, providing a 4x speedup in inference and improved performance in retrieval tasks. ## preliminaries traditional works rely on text frequencies to define whether the instance is long-tail or not; thus, lowfrequency texts tend to be classified into long-tail classes. for llms, computing text frequencies of previously unknown user queries is by no means an easy task. as in @xcite @xcite , expected calibration error (ece) provides a new perspective to measure "long-tailness". ece measures how well a model's estimated probabilities match true (observed) probabilities @xcite . in the calculation of ece, the confidence of each instance is allocated to a specific interval and obtained by the model predicted probability. the accuracy is determined by the comparison of the predicted label and the ground truth. the absolute margin between confidence and accuracy of each instance represents the calibration degree. the expected calibration degree of the whole dataset indicates the reliance of the model. formally, ece can be formulated as: ## experiments in this section, we briefly describe the experimental results and leave detailed experimental settings in appendix a, and supplementary experimental results in appendix b. ## conclusion in summary, our research highlights the significance of long-tail knowledge to enhance the efficacy of rag for llms. we introduced the generative expected calibration error (gece) to identify long-tail knowledge, which accelerates the inference process by more than fourfold in average and improves performance on downstream tasks without compromising the quality of responses. this demonstrates the benefits of selectively augmenting llms with targeted information, paving the way for more efficient and accurate rag systems.