LM -Polygraph: Uncertainty Estimation for Language Models (2023)

Abstract: Recent advancements in the capabilities of large language models (LLMs) have paved the way for a myriad of groundbreaking applications in various fields. However, a significant challenge arises as these models often “hallucinate”, i.e., fabricate facts without providing users an apparent means to discern the veracity of their statements. Uncertainty estimation (UE) methods are one path to safer, more responsible, and more effective use of LLMs. However, to date, research on UE methods for LLMs has been focused primarily on theoretical rather than engineering contributions. In this work, we tackle this issue by introducing LM-Polygraph, a framework with implementations of a battery of state-of-the-art UE methods for LLMs in text generation tasks, with unified program interfaces in Python. Additionally, it introduces an extendable benchmark for consistent evaluation of UE techniques by researchers, and a demo web application that enriches the standard chat dialog with confidence scores, empowering end-users to discern unreliable responses. LM-Polygraph is compatible with the most recent LLMs, including BLOOMz, LLaMA-2, ChatGPT, and GPT-4, and is designed to support future releases of similarly-styled LMs.

## introduction large language models (llms) have demonstrated remarkable performance across a variety of text generation tasks. instruction fine-tuning and reinforcement learning from human feedback (rlhf) have brought the zero-shot performance of these models to a new level @xcite . however, the capabilities of llms, despite their profound power and complexity, are inherently constrained. limitations arise from the finite nature 1 http://lm-polygraph.nlpresearch.group 2 http://lm-polygraph-demo.nlpresearch.group 3 http://lm-polygraph-video.nlpresearch.group ♢ equal contribution of the training data and the model's intrinsic memorization and reasoning capacities. hence, their utility is bounded by the depth and breadth of the knowledge they embed. due to their training objectives, even when the embedded knowledge of an llm on a given topic is limited, it tends to be over-eager to respond to a prompt, sometimes generating misleading or entirely erroneous output. this dangerous behavior of attempting to appease the user with plausiblesounding but potentially false information is known as "hallucination" @xcite . it poses a significant challenge when deploying llms in practical applications. there are several well-known approaches to censoring llm outputs, including: filtering with stopword lists, post-processing with classifiers @xcite , rewriting of toxic outputs @xcite , and longer fine-tuning with rlhf. however, these approaches cannot be relied on to completely resolve hallucinations. since lms are natural (if "unintentional") liars, we propose lm-polygraph -a program framework that, similar to a human polygraph, leverages various hidden signals to reveal when one should not trust the subject. in particular, lm-polygraph provides a comprehensive collection of uncertainty estimation (ue) techniques for llms in text generation tasks. uncertainty estimation refers to the process of quantifying the degree of confidence in the predictions made by a machine learning model. for classification and regression tasks, there is a welldeveloped battery of methods @xcite . there has also been a surge of work investigating ue, particularly in text classification and regression in conjunction with encoder-only lms such as bert @xcite @xcite @xcite @xcite . however, ue for sequence generation tasks, including text generation, is a much more complex problem. to quantify the uncertainty of the whole sequence, we have to aggregate uncertainties of many individual token predictions and deal with non-trivial sampling and pruning techniques like beam search. contrary to classification tasks where the number of possible prediction options is finite, in text generation, the number of possible predictions is infinite or exponential in vocabulary size, complicating the estimation of probabilities and information-based scores. finally, a natural language text is not a simple sum of its tokens; it is a nuanced interposition of context, semantics, and grammar, so two texts can have very diverse surface forms but similar meanings, which should be taken into account during the ue process. several recent studies have delved into developing ue methods for lms in text generation tasks @xcite @xcite @xcite . however, the current landscape of this research is quite fragmented with many non-comparable or even concurrent studies, which makes it challenging to consolidate the findings and draw holistic conclusions. in this work, with the development of lm-polygraph, we strive to bridge these disparate research efforts, fostering more cohesion and synergy in the field. we envision a framework that consolidates the scattered ue techniques within unified frameworks in python, provides an extendable evaluation benchmark, and offers tools to integrate uncertainty quantification in standard llm pipelines seamlessly. this endeavor will not only make the journey less challenging for individual researchers and developers but also set the stage for more robust, reliable, and trustworthy llm deployments for end-users. our contributions are as follows: lm-polygraph implements a set of state-of-the-art ue techniques for llms with unified program interfaces in python. it is compatible with models from the huggingface library and is tested with recent public-domain llms such as bloomz (scao et al., 2022; yong et al., 2023), dolly v2 (conover et al., 2023), alpaca (taori et al., 2023), llama-2 (touvron et al., 2023), and flan-t5 (chung et al., 2022). the framework supports both conditional models with a seq2seq architecture and unconditional decoder-only lms. figure 1 contains a code example of lm-polygraph with bloomz-3b for ue in open-domain question answering. some methods that do not require access to the model itself or its logits could be used in conjunction with web-hosted llms like chatgpt or gpt-4 through apis. we provide a program wrapper for integration with popular online services. ## uncertainty estimation methods here, we summarize ue methods implemented in lm-polygraph, as listed in there are two major technique types: white-box and black-box. the white-box methods require access to logits, internal layer outputs, or the lm itself. the black-box methods require access only to the generated texts, and can easily be integrated with third-party online services like openai lm api. we note that the methods differ by computational requirements: some techniques pose high let us consider the input sequence x and the output sequence @xmath0, where y is a set of all possible output sequences. then the probability of an output sequence given an input sequence for probabilistic autoregressive language models is given by: ## demo we constructed a demo application that can be used to interact with llms and also see confidence scores of model answers (see figure 2 ). a user specifies a ue method and a language model from a number of publicly-available llms with up to 13b parameters, e.g., bloomz, vicuna, and llama-2. there is also the ability to communicate with llms deployed as web services such as chatgpt or gpt-4 and obtain their uncertainty scores based on black-box techniques. for these models, a user should provide an api key. this demo application is potentially helpful for both end-users and researchers. for end-users, it extends the standard ai assistant interface with information about whether it is reasonable to trust a model answer. researchers could use this tool for qualitative analysis of various ue methods and llm responses. ## evaluation benchmark lm-polygraph provides a vast evaluation benchmark. it contains a script for running one or multiple experiments with ue techniques, implemented as python modules. this feature allows the user to easily extend the set of available methods and evaluate novel ue techniques in a unified manner. using this benchmark, we have conducted experiments with most methods implemented in lm-polygraph. below, we provide experimental details. datasets. we experiment with three text generation tasks: machine translation (mt), text summarization (ts), and question answering (qa). for each task, we use two widely-used datasets: wmt-14 german to english and wmt-14 french to english @xcite for mt, xsum @xcite and aeslc @xcite for ts, and coqa @xcite and babi qa @xcite for qa. dataset statistics are presented in appendix d. ## experimental results tables 2 and 3 present the results for vicuna-v1.5-7b and llama-v2-7b correspondingly. for both models, the better performance is usually demonstrated by the white-box methods based on information-theoretic concepts (first 8 rows of the table). these methods are in general also easy to implement and computationally lightweight, with the notable exceptions of semantic entropy, monte carlo sequence entropy, and monte carlo normalized sequence entropy, which require sampling from the model several times to obtain uncertainty scores. when working with llms as web services, usually there is no access to full posterior distributions over tokens, therefore, only black-box methods could be used. among this group of approaches, the best average performance is achieved by eccentricity for vicuna. for llama, there is no clear advantage for any of the methods considered. overall, we see that absolute values for all evaluated methods, models, and datasets are far away from perfect. low performance of current methods is especially evident on more complicated tasks such as xsum and wmt14. our experimental results demonstrate that the task of selective generation is not close to be solved. this once again underlines the importance of further research and development of efficient uncertainty estimation techniques for generative language models. ## conclusion as the community strives to advance the potential of llms, it is critical to be mindful about dangers of their uncontrolled usage. in this work, we propose a tool for making the application of llms safer. enriching model predictions with uncertainty scores helps users and developers to be informed about these risks, encouraging healthy skepticism towards certain outputs generated by these models. we plan to further expand our framework with implementations of new ue methods that emerge in the future. we hope that our work will foster the development of techniques to detect and mitigate llm hallucinations, which we believe is a key to unlocking the safe, responsible, and effective use of llms in real-world applications.