Dynamic Uncertainty Ranking: Enhancing Retrieval-Augmented In-Context Learning for Long-Tail Knowledge in LLM s (2025)

Abstract: Large language models (LLMs) can learn vast amounts of knowledge from diverse domains during pre-training. However, long-tail knowledge from specialized domains is often scarce and underrepresented, rarely appearing in the models’ memorization. Prior work has shown that in-context learning (ICL) with retriever augmentation can help LLMs better capture long-tail knowledge, reducing their reliance on pre-trained data. Despite these advances, we observe that LLM predictions for long-tail questions remain uncertain to variations in retrieved samples. To take advantage of the uncertainty in ICL for guiding LLM predictions toward correct answers on long-tail samples, we propose a reinforcement learning-based dynamic uncertainty ranking method for retrieval-augmented ICL that accounts for the varying impact of each retrieved sample on LLM predictions. Our approach prioritizes more informative and stable samples while demoting misleading ones, updating rankings based on the feedback from the LLM w.r.t. each retrieved sample. To enhance training efficiency and reduce query costs, we introduce a learnable dynamic ranking threshold, adjusted when the model encounters negative prediction shifts. Experimental results on various question-answering datasets from different domains show that our method outperforms the best baseline by 2.76%, with a notable 5.96% boost in accuracy on long-tail questions that elude zero-shot inference. Our code is available at https://github.com/Yu-shuyan/uncertian_ranker .

## introduction pretrained large language models @xcite @xcite have achieved remarkable success across various natural language processing (nlp) tasks, such as summarization @xcite , question answering @xcite , and code generation @xcite . these impressive results are largely due to their pre-training on vast, web-sourced datasets spanning multiple domains. however, these real-world datasets often follow a long-tail distribution @xcite @xcite , where knowledge from less frequent domains is underrepresented. consequently, certain domain-specific information may be rarely or even never included in the llms' memorization @xcite . as a result, llms struggle to provide accurate responses to queries drawn from these long-tail distributions, since the pre-training process fails to capture this sparse information. in-context learning (icl) (brown, 2020) is a few-shot learning method that queries llms by concatenating relevant samples with the test query, without updating the model's parameters. @xcite found that icl, when combined with retriever augmentation, can reduce llms' reliance on pre-training knowledge by retrieving relevant examples related to long-tail queries during inference. common retrieval methods used to select augmentation examples for icl include random selection @xcite , off-the-shelf retrievers (e.g., bm25 @xcite ), and fine-tuned retrievers (e.g., promptpg @xcite ). however, prior works @xcite @xcite have shown that icl with different selection and ordering of the retrieved samples could lead to unstable predictions of llms. in our experiments, we observed a similar pattern: when utilizing existing methods to retrieve relevant samples for icl, the model's predictions for long-tail questions-those not captured by zero-shot inference-exhibited particularly high uncertainty. in some cases, a subset of the retrieved samples led to correct predictions, while the full set misled the model, even with the same retrieval method. in this paper, to enhance the retrieval augmentation for long-tail samples regarding llm's uncertainty, we propose a reinforcement learning-based dynamic uncertainty ranking method motivated by reinforcement learning's capacity to search for optimal retrieved samples based on the llm's feedback @xcite . specifically, our approach trains a retriever to prioritize informative and stable samples while down-ranking misleading ones, enhancing performance on both head and tail distributions. we build on the bert-based retriever architecture @xcite with an appended linear layer. during the training of the retriever, only the linear layer is fine-tuned. initially, bm25 @xcite is used for pre-selection, and the retriever is trained using policy gradients @xcite , guided by feedback from the llm for each retrieved sample. to improve efficiency, we introduce a learnable dynamic threshold as a budget controller for retrieval, selecting only samples with high-ranking scores above this threshold, which adjusts whenever the llm experiences a negative prediction change, i.e., the prediction changes from true to false. to evaluate the proposed approach, we compared our method with the state-of-the-art methods across both multi-choice and open-ended question-answering (qa) datasets from different domains. the experimental results show that our method outperforms the best baseline by 2.76%. long-tail questions failed to be captured by a zero-shot inference benefit particularly from our proposed method. the accuracy of long-tail questions of our method surpasses previous methods with a large margin of up to 5.96%. we summarize our key contributions as follows: in-context learning (icl). icl (brown, 2020) queries the llms with a concatenation of related samples and the test query without parameter updating. to improve the quality of icl, retrievers have been proposed to select related samples, which can be categorized into sparse retrievers (e.g. @xcite ) and dense retrievers (e.g. @xcite ). to further improve the effectiveness of the off-the-shelf retrievers, strategies for finetuning retrievers on specific target domains have been proposed such as promptpg @xcite , udr @xcite , and llm-r @xcite , etc. some works also adopt gpt to help retrieve and rerank samples by providing special prompts and related samples, such as rerank @xcite , sure @xcite , etc. long-tail knowledge learning for icl. @xcite is the first to explore the influence of the long-tail distribution in pre-training data on llm memorization. they find retrieval augmentation as a promising approach to significantly reduce the llm's dependence on pre-training knowledge. several subsequent works have built on this retrieval augmentation approach to address the longtail problem in llms. for example, @xcite propose a retrieve-then-rerank framework leveraging knowledge distillation (kd) from the llm to tackle long-tail qa. however, their method involves tuning the language model, which is computationally expensive and impractical for blackbox llms such as gpt-4 @xcite . another line of research focuses on augmenting the training set using gpt @xcite @xcite , followed by fine-tuning the retriever to enhance its performance. nonetheless, determining which samples should be augmented remains challenging. augmenting the training set based on seed sentences often introduces repetitive rather than diverse information, and incurs significant costs due to gpt queries. therefore, in this paper, rather than augmenting the training set for fine-tuning the retriever, we aim to train an effective retriever capable of selecting the most informative samples to augment the test query during inference. ## problem formulation in this paper, we target in-context learning (icl) for qa tasks including multiple-choice qa and open-ended qa from different domains. suppose we have a training set @xmath0, y i )} n @xmath1, where x is the question and y is the answer. given a query problem p i from a test set p and a k-shot inference budget, we will retrieve k related samples e @xmath2, y i )|e k @xmath3, p i ) as input to feed into the llm: ## motivation: uncertainty of in-context learning due to the lack of knowledge of some specific domains during the pre-training stage, there exists long-tail knowledge that failed to be captured by the llms @xcite . we define easy samples as queries that have been captured during the llm's pre-training stage and are stored in its memorization. in contrast, hard samples refer to queries that the llm failed to capture, which are more likely to represent long-tail data. we classify easy and hard samples using the zero-shot testing results âi = llm 0-shot (p i ): for hard samples, even a tiny variation in retrieved set e can mislead the llm's prediction. one case study for hard sample queries from t-rex @xcite is shown in fig. 2 . in this case, llm gives a correct answer with the first two informative samples in e, effectively compensating for the llm's long-tail knowledge. however, the answer gets wrong when a third sample is added to the prompt, which indicates the newly added knowledge is misleading. other cases to show the uncertain prediction of llm can be found in fig. 7 in section 6.4 and given the uncertainty of in-context learning, our goal is to improve the prediction accuracy of hard samples while maintaining the prediction stability on easy samples. during testing, we lack prior knowledge to determine whether a query falls into the easy or hard category. the primary challenge, therefore, is to prevent the inclusion of misleading information in the retrieved set e, which could lead to incorrect predictions. simultaneously, we must ensure that the retrieved samples are sufficiently informative to address long-tail knowledge gaps and guide the llm toward the correct answer. ## in-context learning with dynamic uncertainty ranking in this section, we introduce a dynamic uncertainty ranking method built on a reinforcement learningbased retriever. this method adjusts the retriever by applying a dynamic threshold, lowering the rankings of misleading samples while elevating the rankings of informative and stable ones. ## experiments in this section, we first introduce the experiment setup and then show the effectiveness of our method through various empirical results. ## conclusion in this paper, to improve the uncertain prediction of llms on long-tail knowledge, we propose a reinforcement learning-based dynamic uncertainty ranking method for retrieval-augmented icl with a budget controller. specifically, it considers the dynamic impact of each retrieved sample based on the llm's feedback. our ranking system system raises the ranks of more informative and stable samples and lower the ranks of misleading samples efficiently. evaluations of various qa datasets from different domains show that our proposed method outperformed all the baselines, and especially improve the llm's prediction on the long-tail questions.