MARS : Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLM s (2024)

Abstract: Generative Large Language Models (LLMs) are widely utilized for their excellence in various tasks. However, their tendency to produce inaccurate or misleading outputs poses a potential risk, particularly in high-stakes environments. Therefore, estimating the correctness of generative LLM outputs is an important task for enhanced reliability. Uncertainty Estimation (UE) in generative LLMs is an evolving domain, where SOTA probability-based methods commonly employ length-normalized scoring. In this work, we propose Meaning-Aware Response Scoring (MARS) as an alternative to length-normalized scoring for UE methods. MARS is a novel scoring function that considers the semantic contribution of each token in the generated sequence in the context of the question. We demonstrate that integrating MARS into UE methods results in a universal and significant improvement in UE performance. We conduct experiments using three distinct closed-book question-answering datasets across five popular pre-trained LLMs. Lastly, we validate the efficacy of MARS on a Medical QA dataset. Code can be found here.

## introduction generative large language models (llms) have risen in popularity due to their remarkable ability to understand, generate, and process human language at an unprecedented scale and accuracy @xcite @xcite . these models have become the state-of-the-art in various fields, including machine translation, content generation, and even scientific research @xcite due to their capability to handle diverse tasks such as text summarization, sentiment analysis, and question-answering in a few-shot or zero-shot manner. despite their growing popularity and success, generative llms are not infallible and can sometimes produce erroneous or misleading outputs, especially when dealing with complex reasoning problems or closed-book questions. this limitation becomes particularly critical in question-answering systems used in high-stakes environments. quantifying the uncertainty of generative llm responses in such scenarios is not just beneficial but essential for ensuring trustworthy operation. for example, in a medical advice application, accurately assessing the uncertainty of the responses provided by llms can prevent the provision of incorrect medical advice. this is crucial because erroneous advice may lead to devastating medical missteps or misunderstandings. thus, understanding and quantifying uncertainty helps in reliable risk assessment and in maintaining the overall quality of the answers provided, ensuring that users can assess how much reliance they should place on llm responses. uncertainty estimation (ue) is a well-studied problem in classification scenarios, especially in the computer vision domain @xcite @xcite . the proposed ue methods in classification tasks, which rely on the class probabilities, are not directly applicable to generative llms due to the auto-regressive generative structure of llms @xcite , which implies that llms generate text sequentially by predicting each subsequent word based on the combined context of all preceding words. this process differs significantly from classification tasks, where the output is typically a single label or a set of labels assigned to an entire input, without the sequential and contextaccumulating nature of generative llms. recent work @xcite , formalizes how to adapt popular ue methods developed for classification tasks to the context of generative llms. they propose using length-normalized scoring to estimate the likelihood of a sequence generated by the model, and the subsequent works @xcite @xcite utilize that idea of length-normalized scoring. a downside of these existing ue techniques in the generative llm literature is treating lengthnormalized scoring like the class probabilities in classification tasks. however, better ways may exist for estimating uncertainty than directly using the length-normalized score of a sequence, as it treats all tokens equally. in reality, each word's contribution to the sentence's meaning in the question context might vary. for example, given the question "which planet is known as the red planet?" and with the generated response "mars is the red planet", the tokens of "mars" are the most critical ones in the response because those tokens are the ones actually answering the question. thus, assigning more weight to semantically significant tokens in the response score calculation can improve ue methods, resulting in more accurate predictions. based on this word importance intuition, we propose a novel scoring function for generative llms called meaning-aware response scoring (mars), as outlined in figure 1 . to compute the llm response score as an input to ue methods, we first assign an importance coefficient to each token in the generation. this importance essentially reflects the impact of masking a token in a sequence on the meaning of the generated response, where tokens with a greater influence on the meaning receive higher importance. by leveraging these meaningaware coefficients (w i in figure 1 ), mars returns the multiplication of the weighted probabilities of the tokens in the generated sequence. we list our main contributions as follows: ## background in this section, we will go over probability-based ue methods that our work built on. for a detailed discussion on related works, refer to appendix a. in the literature, ue is used as a proxy for the correctness of the model output @xcite @xcite . for generative llms in the question-answer context, we consider the most probable sequence as the model output and utilize ue to predict the correctness of the response following @xcite . the goal of ue is to assign higher scores to incorrect responses, indicating greater uncertainty, and lower scores to correct responses, signifying less uncertainty. ## understanding generative llm probabilities from a classification perspective in classification tasks, the class probability reflects the model's confidence in assigning a specific class to an input. it is inherently tied to the semantics of the class. for instance, if a well-calibrated classifier gives a 75% probability to the label "cat" for a given image, it suggests a 75% likelihood that the given image is indeed a cat. this output probability is not only a numerical value; it conveys a semantic understanding of the image content as a cat. to understand previous works @xcite ) from a classification perspective, we propose a new random variable that is directly related to the semantics of the output. let y be a random variable with arbitrary dimension corresponding to the meaning of the sequences generated by an llm parametrized with θ. the values of y can be the set of all possible meanings of generated sequences and their contexts. formally, the set is {g(s, x)} @xmath0,@xmath1, where g(•) is the meaning function that takes generated sentence s and context x as inputs and returns the meaning as output. a well-calibrated probability distribution of y ensures that if a model's generation is more likely to be correct than another, the corresponding probability should be higher. mathematically, p (@xmath2, x 1 )|θ) > p (@xmath3, x 2 )|θ) if the model's response s 1 to question x 1 is more likely to be correct than response s 2 to question x 2 . this is indeed similar to the classification example where well-calibrated class probabilities reflect the output's actual probability of correctness @xcite . previous works and mars have heuristic design choices to make y more calibrated to get a better uncertainty estimation. malinin and gales (2021) considers g(•) as a one-to-one function which means that each unique sentence in the given context corresponds to different meanings. to achieve better calibration of y they define its distribution using length-normalized scoring of the generated sequences. this approach ensures a fair comparison between long and short sequences, thereby improving the calibration of y . on the other hand, @xcite claims different sequences can have equal meaning. by considering g(•) as a many-to-one function, we can define the distribution of y with their proposal as follows: with the new random variable y , we essentially write the semantic entropy as the standard monte-carlo approximated entropy over a total of b distinct meanings. notice that the normalization term @xmath4,@xmath5, θ) featured in both (11) and ( 12 ), acts as a constant across all p (@xmath6, ensuring that y conforms to a valid probability distribution. therefore, it only shifts the proposed ue scores which does not affect the performance of accurately predicting the correctness of the model generation. moreover, by introducing the random variable y , we not only provide an alternative explanation of the previous works but also create flexibility to define new distributions for y which may potentially have better calibration and improve the existing ue tools. using the definition of y , we can also rationalize our scoring function mars. we replace the length-normalized scoring function with mars as in (9). mars considers the semantic contribution of tokens, assigning higher weights to those critical for the correctness of the answer. this emphasis on key tokens potentially makes mars a more effective scoring function for achieving better calibration of y . ## limitations the importance function model within mars utilizes an unsupervised methodology, leveraging preexisting models for its formulation. nonetheless, the performance of mars can potentially be further enhanced by using human labelers to assign importance coefficients for training the importance function model. besides, our analysis is limited to the closed-ended question-answering domain in english, where a question has an objective groundtruth answer(s). extensive analysis of mars and other probability-based ue methods on open-ended question-answering tasks and other languages are beyond the scope of the current study and are left as future work. 8 ethics statement although probability-based ue methods combined with mars have a remarkable prediction performance on the correctness of generative llm outputs, it is crucial to acknowledge that these methods do not achieve 100% accuracy. besides, as llms may have biases against gender, ethnicity, age, etc., probability-based methods can carry those biases to ue outputs. thus, one should be aware of these potential risk factors before employing such probabilistic ue methods in real-world systems. ensuring fairness, transparency, and accountability in the deployment of these technologies is important in mitigating risks and fostering trust in their application.