Self-Consistency of Large Language Models under Ambiguity (2023)

Abstract: Large language models (LLMs) that do not give consistent answers across contexts are problematic when used for tasks with expectations of consistency–e.g. question-answering, explanations, etc. Our work presents an evaluation benchmark for self-consistency in cases of under-specification where two or more answers can be correct. We conduct a series of behavioral experiments on the OpenAI model suite using an ambiguous integer sequence completion task. We find that average consistency ranges from 67% to 82%, far higher than would be predicted if a model’s consistency was random, and increases as model capability improves. Furthermore, we show that models tend to maintain self-consistency across a series of robustness checks, including prompting speaker changes and sequence length changes. These results suggest that self-consistency arises as an emergent capability without specifically training for it. Despite this, we find that models are uncalibrated when judging their own consistency, with models displaying both over- and under-confidence. We also propose a nonparametric test for determining from token output distribution whether a model assigns non-trivial probability to alternative answers. Using this test, we find that despite increases in self-consistency, models usually place significant weight on alternative, inconsistent answers. This distribution of probability mass provides evidence that even highly self-consistent models internally compute multiple possible responses.

## introduction language model pre-training approximates a distribution generated by many speakers. as a result, llms learn to express inconsistent beliefs drawn from distinct groups of people @xcite . recent work has investigated the consistency of llms variously as: a logical validity check on model claims @xcite , an explanatory validity check on the simulatability of models' explanations @xcite , and a tool to identify llms representations of truth @xcite . all of these works rest to some degree on the contention that fine-tuned llms can be understood as holding beliefs, an assumption which has recently come under scrutiny @xcite . consistency is particularly of interest in cases of ambiguity. recent work has evaluated llms' ability to identify linguistic and classification-task ambiguity @xcite . our work brings together these threads of research, examining how model explanations can be examined via self-consistency checks. we offer a case study on ambiguity in an arithmetical setting. we ask language models from openai for a continuation of an integer sequence having multiple possible continuations. we then separately ask the models for the formula that generated the initial sequence, which we refer to as the explanation. finally, we evaluate whether modelgenerated continuations are consistent with modelgenerated explanations ( §3). we present the model with the full set of sequence generating functions so that ambiguity is, in principle, recognizable by the model. we find the following across evaluations using davinci (gpt-3), text-davinci-003, gpt-3.5-turbo, and gpt-4: ## dataset: ambiguous integer sequences in order to evaluate self-consistency, we created and open-sourced a dataset of ambiguous integer sequences. 1 . integer sequences were chosen because we can readily identify sequences that have multiple valid completions. this allows us to introduce tasks with ambiguity for measuring properties like model self-consistency. previous work on self-consistency considered open-ended question answering or knowledge probing @xcite which makes measuring consistency difficult (rendering unclear the space of possible answers, and what constitutes distinct answers), whereas in our setting the space of possible answers is rigorously defined via an enumeration of generation functions. our dataset was created as follows: we generate integer sequences, e.g., 7, 11, 15, drawn from a fixed set of generating functions, e.g., lambda x: (4 * x) + 3. ## methodology: evaluating consistency we measure consistency by comparing responses from the completion task to responses from the explanation task, which we call cross-context because the model sees each task in a separate context window. each prompt uses eight demonstrations showing the model how to complete the sequence or explain the sequence using a python function. the demonstrations are drawn randomly foot_0 from the same function space as the ambiguous and unambiguous functions. examples of these prompts are presented in appendix b. the models chosen for evaluation were text-davinci-003, gpt-3.5-turbo, and gpt-4. 3 while we are not entirely sure how these models are trained, these models were chosen because they are commonly used by both researchers and the public, and they represent a sequence of capability increases through data quality improvement, annotations, and innovation in training and inference techniques (see @xcite ). in the below experiments, greedy sampling (temperature set to 0) is used throughout. this choice lets us conduct a best-case analysis of selfconsistency: studying whether a model is capable of self-consistency when the sampling strategy is advantageous. in §5, we move on from greedy decoding and examine what the full output distribution implies about the possible continuation space of models. 4 ## robustness checks for consistency we conducted further experiments to better understand how robust these results were to changes in experimental protocol by using a range of different prompts. we consider: (1) speaker changes in which we prompt the model as if the sequences were generated by different speakers; (2) change of base in which the sequence integers are presented in base 2 instead of base 10; and (2) sequence length changes. full results are given in §c. ## conclusion all tested models behaved more self-consistently across contexts for ambiguous tasks than expected if the models had randomly consistent behaviour. this is surprising given models are not explicitly trained for cross-context self-consistency. we also found that model consistency grows with model capability. we varied the task prompt, as well as the difficulty of the task (using base-2 sequences instead of base-10 sequences and varying the se-quence length), and found that our findings are robust with respect to these changes. across all evaluated models found that they are not well calibrated when it comes to evaluating their own consistency. we also tested that even when a model that chooses relatively consistent answers among several correct answers across contexts, models may still assign non-trivial probability to other correct answers. asking the models to verbalize correct alternatives revealed high precision scores for all models which discern between correct and incorrect answers. in comparison, recall was relatively low where text-davinci-003 surprisingly achieved the highest recall, closely followed by gpt-4, indicating they can retrieve alternative correct answers. the significance of our results is that we shouldn't assume the apparent consistency of llms points to actual internal consistency due to high probability mass placed on alternative answers which may equally be picked using common sampling techniques for natural language generation. as a community we should also be wary of consistency given our results on calibration that show models across capability classes strongly over and under estimate their own consistency.