Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation (2025)

Abstract: Recent work suggests that large language models enhanced with retrieval-augmented generation are easily influenced by the order in which the retrieved documents are presented to the model when solving tasks such as question answering (QA).However, there is no method to date that exploits this phenomenon to improve generation.To fill this gap, in this study, we show that the pointwise mutual information between a context and a question is an effective gauge for language model performance.Importantly, this gauge does not depend on knowing the answer to the question a priori .Through experiments on two question-answering datasets using a variety of large language models, we find evidence for an empirical correlation between answer accuracy and pointwise mutual information.Additionally, we propose two methods that use the pointwise mutual information between a document and a question as a gauge for selecting and constructing prompts that lead to better performance, whose effectiveness we demonstrate through experimentation.

## introduction prompt design is an important factor when applying language models (lms) to downstream tasks, including lms that make use of retrieval-augmented generation (rag; @xcite . wellconstructed prompts can improve lms' answers to user-input questions and help generate responses that better align with user expectations @xcite @xcite @xcite . under the rag framework, a prompt typically consists of three components. first, an instruction provides a textual description of the overall task and general guidance for the language model. second, a specific question encodes the precise task or query the model should perform. third, a context figure 1 : for the same question, a permutation of documents with a higher pmi(q, c d (π)) tends to lead to a better answer. while @xcite results are interesting, choosing the optimal permutation of the documents requires knowledge of the answer and, thus, cannot be directly used to improve rag. in this article, we develop a proxy for the optimal permutation: we show that the pointwise mutual information between the question and the context under an lm acts as a useful proxy in determining the optimal permutation. to our knowledge, ours is the first to present in-depth analyses of the relation between question likelihood and model performance under the rag framework. our findings in this paper are summarized in the following list: a language model p is a distribution over σ * , the set of all strings with tokens drawn from σ. with σ ⪯ σ ′ , we denote that σ is a proper prefix of σ ′ . let y be a σ * -valued random variable distributed according to p and σ ∈ σ * . we define the prefix probability 3 -→ p (σ) as the probability that y has σ as a prefix: @xcite for a more in-depth discussion. retrieval-augmented generation. modern language models are often used to perform questionanswering tasks. when solving such a task with a language model, string encoding the question question @xmath0. we assume each question q has a unique correct answer, which we will denote a. this is, of course, a simplifying assumption, but it does jibe with how questionanswering is typically evaluated. we will adorn symbols with •, e.g., a, to indicate an answer generated from -→ p (• | q) that may or may not be correct. generating a from -→ p (• | q) may be done using either a deterministic method, e.g., beam search, or a stochastic method, e.g., ancestral sampling. 4 in rag, the model is additionally given a set of documents @xmath1, where d @xmath2, and a permutation of the documents π : {1, • • • , k} → {1, • • • , k}. given d and π, a context c is constructed by concatenating the documents in the order defined by π, i.e., c d (π) then, we generate an answer from -→ p (• | c • q). ## pmi correlates with performance as discussed in §2, our first goal is to determine how pmi(q, c d (π)) changes as a function of the permutation π of the documents d. due to hypothesis 2.1, we expect a strong correlation between pmi(q, c d (π)) and expected answer accuracy. ## improving rag via reordering in §3.3, we offered evidence for hypothesis 2.1, i.e., that pmi(q, c d (π)) correlates with model performance. in light of this finding, we propose two methods to permute the documents presented to the lm in rag without knowledge of the answer.