Generating Benchmarks for Factuality Evaluation of Language Models (2024)

Abstract: Before deploying a language model (LM) within a given domain, it is important to measure its tendency to generate factually incorrect information in that domain. Existing methods for factuality evaluation of LLM generation focus on facts sampled from the LM itself, and thus do not control the set of evaluated facts and might under-represent domain specific or rare facts. We propose FACTOR: Factual Assessment via Corpus TransfORmation, a scalable approach for evaluating LM factuality. FACTOR automatically transforms a factual corpus of interest into a benchmark evaluating an LM’s propensity to generate true facts from the corpus vs. similar but incorrect statements. We use our framework to create three benchmarks: Wiki-FACTOR, News-FACTOR and Expert-FACTOR. We show that: (i) our benchmark scores increase with model size and improve when the LM is augmented with retrieval; (ii) benchmark score and perplexity do not always agree on model ranking; (iii) when perplexity and benchmark score disagree, the latter better reflects factuality in open-ended generation, as measured by human annotators.

## introduction despite rapid improvements in their capabilities, large language models (lms) still tend to generate factually inaccurate or erroneous text @xcite @xcite . such phenomena can pose a significant hurdle to deploying lms in important or sensitive settings, motivating the development of methods for evaluating lm factuality in open-ended generation. methods for directly evaluating an lm's propensity towards factual generation were recently proposed by @xcite and @xcite . these methods suggest sampling generations from prefix: steve jobs had a long power struggle with the company's board. completions: (a) in 1985, he was forced out of apple. (b) in 1985, he was forced out of next. (c) in 1985, he quit apple. (d) in 1988, he was forced out of apple. lm ✓ ✗ ✗ ✗ (a)... (b)... (c)... (d)... 1 . the evaluated model assigns likelihood scores to each completion separately. it is considered "correct" if it assigns the highest likelihood to the factually correct completion over all non-factual alternatives. a model, applying an automatic pipeline for fact verification, and then assigning a score corresponding to the percentage of factually correct generated statements. in task-specific domains, such as long-form question answering, evaluation is usually done by assessing the relevance of a sampled generation against a reference text @xcite . however, the sampling approach may introduce bias: by scoring the accuracy of facts that an lm tends to generate in an open-ended setting, high-likelihood facts are overrepresented, while the "long-tail" of rare facts is under-represented. currently, there are no metrics suited to measuring lm factuality with respect to a controlled set of facts in a generation setting. a common proxy is measuring lm perplexity; this was widely adopted to evaluate retrieval-augmented lms @xcite @xcite . however, perplexity is affected by many linguistic phenomena, and so cannot be directly linked to factuality. this paper introduces a novel framework for testing a model's tendency to generate factual information from a given factual corpus: factual assessment via corpus transformation (factor). the key idea is automatically perturbing factual statements taken from the corpus to create a constant number of similar but false variations for each true statement (figure 1 ). we employed instruct-gpt @xcite to generate the false variations for each true statement. the lm's fac-tor accuracy on our benchmark is defined as the percentage of examples for which it assigns higher likelihood to the factual completion than to any of the false variations. we applied factor to the wikipedia and news domains, as well as to a diverse collection of domain specific question-answer pairs (e.g., medicine, technology, law); constructing new benchmarks dubbed wiki-factor, news-factor and expert-factor. we used these datasets to evaluate a large suite of lms from the opt @xcite , gpt-2 @xcite , and gpt-neo @xcite families, ranging from 110m to 66b parameters. we show in §5.1 that, as expected, factor scores increase with model size. however, even the largest models we evaluated achieved scores of only 58% for wiki-factor, 68% for news-factor, and 55% for expert-factor, indicating that these benchmarks are challenging even for large lms. in §5.2 we show that consistent factor score improvements can be achieved by augmenting the lms with the simple retrieval component used by @xcite . this directly demonstrates that retrieval augmentation improves factuality in the lm setting; factor is thus posed as a prominent approach for measuring retrievalaugmented lms. we further show that factor accuracy and lm perplexity are correlted but can sometime induce different orderings between lms ( §5.3). this highlights that factor and perplexity capture different aspects of the lms' performance (see figure 2 ). in §6, we report findings of a manual annotation effort over 1, 200 generated completions, which reinforces factor accuracy as predictive of factuality in open-ended generation. 20b ) and the opt model family (red triangle, 1.3b-66b). labels indicate sizes (in billions). the two may disagree on ranking, e.g., the opt-66b lm has higher perplexity but better wiki-factor accuracy than the gpt-j-6b lm (marked in green circle). in §6 we annotate text generated out of both models and show that better wiki-factor is predictive of more factual text generation. ## related work factuality evaluation the subject of factuality evaluation has been extensively studied in downstream tasks such as summarization, factverification and dialog @xcite @xcite . these works typically focus on factual consistency, evaluating whether a generated text is supported by a reference text or context (e.g., source document and generated summary). another popular approach suggests probing lms' internal factual knowledge by using slot filling tasks, e.g., "barack obama was born is [mask]" @xcite @xcite @xcite @xcite @xcite . these works test lms in a simplified, synthetic setting. factor differs from the above methods as it aims at evaluating factuality in a natural openended text generation setting. in such setting, the context may be needed to reason over the evaluated factual statement, while the factual statement may not be evident in the context (unlike summarization). recent works proposed scoring the factuality of free-form lm generations samples @xcite . however, these approaches lack control over the evaluated facts and are biased towards common facts generated by the lm. contrastive datasets contrastive evaluation, in which a model is tested to discern between similar positive and negative examples, is widely used in various tasks @xcite @xcite . for factuality evaluation, negative examples are obtained by perturbing factual claims. this is done through human annotation, rule-based or model based heuristics @xcite @xcite . following recent works on benchmarks generation @xcite , we employed instruct-gpt to generate non-factual claims, as described in the following section. ## the factor evaluation approach this section outlines our proposed approach: factual assessment via corpus transformation, or factor. given a corpus, we define a multi-choice task where each example is comprised of a multisentence prefix, a single factual next sentence completion, and three non-factual alternative completions (figure 1 ). in §3.1 we present several properties required of a factor benchmark, and describe the error verticals along which we generate non-factual alternatives. we then explain our fac-tor dataset creation pipeline, which automatically generates a factor benchmark from a given corpus ( §3.2). finally, we apply this pipeline to two corpora wikipedia and news, and a long-form question answering dataset, creating wiki-factor, news-factor and expert-factor. we verify the quality of these datasets through manual annotations against the required properties ( §3.3). ## experimental setup we used factor benchmarks to evaluate factual knowledge of llms across varying model families. we describe the experimental setup below. ## factual knowledge evaluation results this section describes the experimental evaluation of llm factuality using our factor benchmarks. in §5.1 we show that factor accuracy increases with model size but also depends on the training data (different model families differ in scores). in §5.2, we show that retrieval augmentation of the lm improves factor accuracy, positioning it as the first automatic measure of factuality improvement for retrieval augmented lms. finally, in §5.3, we show that the pairwise model ranking of corpus perplexity and factor accuracy can differ significantly. this outcome, along with manual validation of the correlation between factor accuracy and factual generation in §6, solidifies factor accuracy as a novel automatic measure for evaluating the proneness of an lm to generate factual information in a certain domain. ## factuality in open-ended generation this section explores the connection between fac-tor accuracy and factuality in open-ended generation, via human annotations. ## discussion this paper introduces factor, a novel way to evaluate lms' factuality. factor creates an evaluation benchmark from a corpus, consisting of factual statements and non-factual variations. by comparing the lm's likelihood of factual claims with non-factual variants, factor score captures the lm's propensity to generate factual information. metrics for measuring factual knowledge over a given corpus are lacking. prior works used perplexity, which may be affected by factors other than factual knowledge and does not contrast facts with false statements. factor focuses the language modeling task on factuality by taking a contrastive approach. our experiments show that factor ranks models differently than perplexity and is more aligned with factuality in open-ended generation. these findings highlight the importance of negative examples for evaluating factuality. moreover, they indicate that incorporating negative examples into training sets might also help optimizing models to be more factual. we leave investigation of training with factor style data to future work. our work joins recent studies on factuality evaluation in a text-generation setting, which proposed to evaluate models by fact-checking the model's generations @xcite . as factor focuses on evaluation over a controlled set of facts, we see these two approaches as complementary; together, they yield a more holistic assessment of lm factuality.