T ruthful QA : Measuring How Models Mimic Human Falsehoods (2022)

Abstract: We propose a benchmark to measure whether a language model is truthful in generating answers to questions. The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics. We crafted questions that some humans would answer falsely due to a false belief or misconception. To perform well, models must avoid generating false answers learned from imitating human texts. We tested GPT-3, GPT-Neo/J, GPT-2 and a T5-based model. The best model was truthful on 58% of questions, while human performance was 94%. Models generated many false answers that mimic popular misconceptions and have the potential to deceive humans. The largest models were generally the least truthful. This contrasts with other NLP tasks, where performance improves with model size. However, this result is expected if false answers are learned from the training distribution. We suggest that scaling up models alone is less promising for improving truthfulness than fine-tuning using training objectives other than imitation of text from the web.

## introduction there is growing interest in using language models to generate text for practical applications. large companies are deploying their own models @xcite , and hundreds of organizations are deploying gpt-3 via apis from openai and other firms @xcite @xcite . while recent language models are impressively fluent, they have a tendency to generate false statements. these range from subtle inaccuracies to wild hallucinations @xcite @xcite . this leads to three concerns: to address these concerns, it is valuable to quantify how truthful models are. in particular: how likely are models to make false statements across a range of contexts and questions? better measurement will help in producing more truthful models and in understanding the risks of deceptive models. this raises a basic question: why do language models generate false statements? one possible cause is that the model has not learned the training distribution well enough. when asked the question, "what is 1241 × 123?", gpt-3 outputs "14812". gpt-3 fails to reliably generalize from its training data about multiplication @xcite . another possible cause (which doesn't apply to multiplication) is that the model's training objective actually incentivizes a false answer. we call such false answers imitative falsehoods. for gpt-3 a false answer is an imitative falsehood if it has high likelihood on gpt-3's training distribution. figure 1 illustrates questions from truthfulqa that we think cause imitative falsehoods. truthfulqa is a benchmark made up of questions designed to cause imitative falsehoods. one ## discussion the questions in truthfulqa are designed such that correct answers are not incentivized by the standard lm objective. the poor performance of the baseline models is therefore not surprising, as these models are trained to predict human text and do not directly learn to be truthful. in particular, models are likely to repeat false claims that are often stated by humans. we believe that truthfulqa tests for many such claims. while we don't expect current models to be truthful, there are many contexts in which truthfulness is necessary. large language models such as gpt-3 may see widespread use as foundation models for downstream tasks that require robust truthfulness @xcite . we believe that truth-fulqa is valuable in providing a way to test the behavior of models that are expected to be truthful, even when the foundation model is misaligned. ## related work numerous nlp benchmarks test models on factual questions @xcite @xcite . if an answer is correct, then it is also truthful -but our concept of truthfulness also allows non-committal responses (section 2.1). while most benchmarks are multiple choice, some require models to generate short (single-phrase) answers @xcite . concepts related to truthfulness in natural language generation include factuality, veracity, and avoiding hallucinations @xcite . @xcite refine the concept of truthfulness and draw distinctions between truthfulness and honesty. truthfulness is relevant to many applications including generating news stories @xcite , summarization @xcite @xcite , conversational dialog @xcite , and question answering @xcite @xcite . a related line of research is automated fact-checking @xcite @xcite , where the focus is on evaluation of statements rather than generation. the problem of imitative falsehoods is similar to models learning to imitate offensive or prejudiced language @xcite ). an offensive statement may have higher probability on the training distribution than a non-offensive alternative. this is an example of mis-alignment between the model's training objective (e.g. to imitate text on the web) and the goals and values of human users (e.g. to avoid offensive language or to avoid falsehoods). another example is when gpt-3 models trained on github learn to produce buggy code @xcite . increasing the safety and alignment of pre-trained models remains a challenging problem @xcite @xcite @xcite mcguffie and newhouse, 2020). ## conclusion making models more truthful is a major challenge for ai. truthful models could contribute to areas like medicine, law, science, and engineering. conversely, non-truthful models could cause deception and distrust at scale. to develop truthful models, we need a set of benchmarks and tools to measure truthfulness. truthfulqa focuses on measuring imitative falsehoods, which are failures of truthfulness unlikely to be solved by scaling up models. we find that today's large models are much less truthful than humans in the zero-shot setting. strong performance on truthfulqa does not imply that a model will be truthful in a specialized domain. but poor performance does indicate a lack of robustness. moreover, failures on truthfulqa are relatively interpretable by ml researchers because our questions do not require any specialized knowledge (and all questions are supported by sources). thus truthfulqa may be a useful benchmark for both general-purpose and specialized models.