Reducing Conversational Agents’ Overconfidence Through Linguistic Calibration (2022)

Abstract: While improving neural dialogue agents’ factual accuracy is the object of much research, another important aspect of communication, less studied in the setting of neural dialogue, is transparency about ignorance. In this work, we analyze to what extent state-of-the-art chit-chat models are linguistically calibrated in the sense that their verbalized expression of doubt (or confidence) matches the likelihood that the model’s responses are factually incorrect (or correct). We find that these models are poorly calibrated, yet we show that likelihood of correctness can accurately be predicted. By incorporating such metacognitive features into the training of a controllable generation model, we obtain a dialogue agent with greatly improved linguistic calibration.

## introduction neural generative open-domain english-language dialogue agents have made progress towards the ability to carry on chit-chat conversations with humans @xcite . recent models-trained on large swaths of data from the internet to mimic human-human conversations-can name their favorite sports teams, describe what it's like to be the owner of two dogs, or share their opinions on tacos. however, ask a state-of-the-art chatbot ''which is heavier, 1 kg feathers or 1 kg stone?'', and it might confidently answer: ''feathers, because they are heavier than a kilogram of any other material.'' 1 this amusing overconfidence can become problematic if someone genuinely doesn't know the answer and is misled into believing something false. generative chit-chat dialogue agents have many issues going beyond inaccurate answers @xcite , making them currently generally unsuitable for applications other than entertainement and research. nevertheless, better 1 answer generated by bst 2.7b @xcite . in this work, we seek to understand whether a model's verbalized expression of confidence (''obviously, . . .'') or doubt (''i'm not sure, but. . .'') in its answer-which we refer to throughout as linguistic confidence-corresponds to the likelihood that the answer is correct, and if not, whether we can fine-tune the models with controlled generation techniques to achieve better alignment. in other words, do state-of-the-art open domain dialogue agents ''know'' what they do not know? if yes, can this knowledge inform their responses, to achieve better verbalized metacognition? we thus make three main contributions. (1) we annotate a state-of-the-art chit-chat model's responses to a large-scale qa task for both factual correctness and linguistic confidence. 2 (2) using these annotations, we find that the model figure 1 : proposed method for re-calibrating a generative dialogue agent. this pipeline involves a calibrator that returns the probability that the original dialogue agent's answers are correct, as well as a fine-tuned model which controls for linguistic confidence; the linguistic confidence is adjusted based on the probability returned by the calibrator, yielding a response for which the linguistic confidence aligns with the likelihood that the dialogue agent's answer is correct. this is our proposed calibrator-controlled chatbot. overconfidence humans' assessments of their own accuracy (confidence) routinely exceed their objective accuracy (correctness) @xcite . this overconfidence effect has been well established, robustly showing that humans are poorly calibrated when completing general knowledge tasks @xcite @xcite . @xcite attempt to correct overconfidence in neural models, by training qa models to abstain from answering questions in which they are likely to err, using probabilistic calibration (see next paragraph). we instead focus on getting conversational models to communicate their confidence verbally, that is, still produce an answer, but one less misleading as to its expected correctness. probabilistic calibration much work has been dedicated to the probabilistic calibration of deep neural networks. @xcite show that modern neural networks for classification tasks are poorly calibrated: models' confidence estimate that their answer is correct doesn't match the empirical rate of correctness. this contrasts with previous findings that show that (earlier) neural networks are well-calibrated on binary classification tasks @xcite . we thereafter refer to this notion of calibration as probabilistic calibration to distinguish it from linguistic calibration. more recently, probabilistic calibration has been explored in the space of large-scale language models (lms). @xcite find that the pre-trained transformers roberta @xcite and bert @xcite are well-calibrated in-domain on the tasks of natural language inference (nli), paraphrase detection, and commonsense reasoning. similarly, jagannatha and yu (2020) calibrate bert and distilbert @xcite for part-of-speech tagging (pos), named entity recognition (ner), and qa tasks. rather than using lms as target predictors on classification tasks like nli and ner, @xcite instead focus on lms as natural language generators and analyze t5 @xcite , a large scale transformer with an encoder-decoder architecture. the authors find that it is poorly calibrated in its probability estimates on qa tasks. conversely, @xcite find that gpt2 is reasonably well calibrated on qa tasks, with an accuracy of 63.1% on the 1% of questions it is most confident in on natural questions @xcite . ## re-calibrating chatbots' language given that bst 2.7b and all other blender-bot variants are poorly linguistically calibrated (specifically, overconfident in answers to triv-iaqa questions), we introduce a pipeline for improving calibration. pipeline overview we propose training a calibrator and using controllable generation techniques to allow generative dialogue agents to better ''own their ignorance,'' that is, such that the models' linguistic confidence better aligns with the probability that the answers are correct. the overall pipeline is illustrated 9 in figure 1 . 7 it is worth noting that removing the minimum length requirement and not fine-tuning on bst did improve qa performance slightly (from 5.0% to 6.9% accuracy on the valid set), and increasing the model capacity to 9.4b parameters even raised it to 8.5% accuracy. improving model capacity without suffering losses in engagingness is an important avenue for further research that is orthogonal to our proposal. 8 we also experimented with top-k and nucleus sampling, which slightly reduced accuracies, and looked at correctnesses of the top few beams instead of just the single most likely generation, but those usually were similar to the top-1 answer in terms of correctness. 9 the robot emoji in this figure was drawn by mariella steeb and distributed as part of the openmoji project under cc-by-sa 4.0. the crystal ball illustration was drawn by we first train a calibrator to return the empirical probability that the model's answer is correct (without seeing the gold answer), and fine-tune the generative dialogue model to enable control over linguistic confidence. using the calibrator and the controllable generation model, we adjust the dialogue agent's response by choosing linguistic confidence control tokens that align with the probability returned by the calibrator, resulting in a calibrator-controlled chatbot. training a calibrator the first step involves training a calibrator that predicts the probability that the model's response is correct, given the question and answer, and the vanilla model's internal representations of both. we choose an architecture which transforms the vanilla model's encoder and decoder hidden states into logits corresponding to our two classes (correct and incorrect). 10 vincent le moign and is distributed as part of the streamline emoji project under cc-by 4.0. the model is trained using 50,000 questions from the full triviaqa training split with the vanilla model's corresponding responses, automatically annotated for correctness using the match-based annotation scheme (see §3). ablations in §5.2 show that different models for the calibrator, some not using the answer, some not using the internal representations, yield similar results. ## results we describe data collection and annotation results, as well as experimental results and analysis on the vanilla model and each stage of the pipeline for the calibrator-controlled chatbot. ## conclusion this work has shown that (1) state-of-the-art conversational model blenderbot @xcite is poorly linguistically calibrated, expressing confidence for answers which are very likely incorrect, but (2) correctness likelihood can be well predicted by a trained calibrator, and (3) using those predictions in a controlled generation architecture allows to greatly improve the linguistic calibration of the model. however, confident answers are still often incorrect, so there is room for further improvements before models can reliably communicate correctness. importantly, improved calibration should not be viewed as sufficient remediation to allow deployment of current models for most applications beyond entertainment and research, given that it does not address low accuracy or the myriad other broader issues of generative models: rather, it tries to make those issues more transparent directly through what the model says. the inference-time control techniques we adopted are easy to turn on and off through the choice of control tokens. this allows for flexible adjustments depending on the conversation requirements-for example, being very openly ignorant in settings that require higher sensitivity, or deliberately expressing uncertainty to allow space for the conversation partner to give their own answer, or committing to confident answers even if they are incorrect in low-stakes casual conversation settings where goofy mistakes are acceptable or even funny. if this flexibility is not required, future work could explore ''baking in'' the linguistic calibration so that a vanilla model directly expresses the correct level of confidence, for example, through retraining as in @xcite , or by training the model specifically not to output responses for which confidence and correctness don't match through unlikelihood techniques @xcite . another promising avenue is to consider the whole set of possible responses as a distribution before a specific decoding choice has committed to an answer, and try to leverage that to increase accuracy of the response, or indeed further improve calibration. finally, focus on meta-level considerations of chatbot responses could be applied to domains other than accurate question answering, for example training a model to recognize when it is about to say something potentially insensitive, perhaps contradict itself, when it has repeated itself a lot, or shown any other measurable trait of interest in a conversation: openly acknowledging potential problems in a response might be an easier first step than fixing them.