A lign S core: Evaluating Factual Consistency with A Unified Alignment Function (2023)

Abstract: Many text generation applications require the generated text to be factually consistent with input information. Automatic evaluation of factual consistency is challenging. Previous work has developed various metrics that often depend on specific functions, such as natural language inference (NLI) or question answering (QA), trained on limited data. Those metrics thus can hardly assess diverse factual inconsistencies (e.g., contradictions, hallucinations) that occur in varying inputs/outputs (e.g., sentences, documents) from different tasks. In this paper, we propose AlignScore, a new holistic metric that applies to a variety of factual inconsistency scenarios as above. AlignScore is based on a general function of information alignment between two arbitrary text pieces. Crucially, we develop a unified training framework of the alignment function by integrating a large diversity of data sources, resulting in 4.7M training examples from 7 well-established tasks (NLI, QA, paraphrasing, fact verification, information retrieval, semantic similarity, and summarization). We conduct extensive experiments on large-scale benchmarks including 22 evaluation datasets, where 19 of the datasets were never seen in the alignment training. AlignScore achieves substantial improvement over a wide range of previous metrics. Moreover, AlignScore (355M parameters) matches or even outperforms metrics based on ChatGPT and GPT-4 that are orders of magnitude larger.

## introduction recent systems for natural language generation, such as summarization and dialogue systems, can produce fluent and coherent text. however, studies show the generated text can often contain factual consistency errors, such as contradictions with input information or hallucinations irrelevant to the context @xcite @xcite @xcite . it is thus crucial to develop automatic metrics that evaluate factual consistency of a claim (e.g., generated text) with regard to a context (e.g., model input). the evaluation, however, has long been a challenge. recent work has devised various metrics based on specific pretrained functions, such as natural language inference (nli) @xcite @xcite @xcite and question answering (qa) @xcite @xcite . specifically, an nli-based metric measures if the claim is entailed by the context; while a qa-based metric first creates (question, answer) pairs from the claim and then checks if answering the questions with a qa model conditioning on the context will lead to the same answers. however, by relying on specific functions trained with only narrow data (i.e., nli or qa datasets), previous metrics have limited generalizability and fail to apply to diverse evaluation scenarios, including different types of factual consistency errors and varying lengths and characteristics of contexts/claims from different tasks and domains. for instance, a metric trained exclusively with nli data of sentences in a certain domain tends to have difficulty in evaluating summaries of long documents in a different domain @xcite . the limitations motivate a more holistic metric that develops a general understanding of factual consistency and generalizes to diverse evaluation scenarios. in this paper, we propose alignscore, a new general factual consistency metric based on a unified text-to-text information alignment function. in particular, we unify a wide range of data sources, and use the massive diverse data to train a general information alignment model that estimates an alignment score given two arbitrary text pieces. more specifically, we reformat and aggregate 15 datasets from 7 popular language tasks, including nli, qa, paraphrasing, fact verification, information retrieval, semantic similarity, and summarization. this results in a total of 4.7m training examples with diverse characteristics, and yields an alignment function with great generalizability. we then build alignscore using the alignment function as a building block. in particular, to handle long text and accommodate the different roles of context and claim, we develop a splitting strategy that breaks a context into coarse-grained chunks and a claim into fine-grained sentences. aggregating the alignment scores between context-chunks and claim-sentences leads to the final factual consistency score. in our experiments, we build alignscore by finetuning the lightweight roberta models (125m and 355m) for alignment. we evaluate alignscore on the latest large-scale evaluation benchmarks, including summac @xcite , true @xcite , and other testbeds, which contain a total of 22 challenging evaluation datasets. our approach substantially outperforms previous state-of-the-art metrics in terms of different quality measures. notably, our metric (355m) is on par with, and sometimes even much better than latest metrics based on orders-ofmagnitude larger language models (e.g., chatgpt and gpt-4). in particular, alignscore shows strong generalizability on the 19 zero-shot datasets that were never seen during the alignment function training. we also conduct extensive ablation studies to demonstrate the effectiveness of the context splitting strategy and other modeling choices. unified training recent work converts related but different tasks into the same input-output format to train unified models. @xcite propose to unify text generation tasks into a text-to-text conditional generation problem. @xcite further show that the text-to-text generation framework, combined with natural language prompting, improves zero-shot task generalization to unseen tasks. @xcite develop a unified automatic evaluation metric by framing different aspects of nlg evaluation as a boolean question answering problem. recent studies also present task unification as an effective approach to improve model performance and generalizability in multimodal tasks @xcite @xcite . ## methods we introduce the alignscore metric built on top of a unified alignment function. we first train the alignment function by unifying a large diversity of data sources (section 3.1). we then define align-score by combining the alignment function with a new context/claim splitting and aggregation strategy (section 3.2). ## experiments in this section, we evaluate alignscore on a wide range of benchmarks and show it consistently outperforms existing metrics (section 4.1-4.4). we also conduct extensive ablation study in section 4.5. ## conclusion we propose alignscore, a holistic factual consistency metric based on a unified alignment function. to learn the alignment function, we adapt 7 well established language understanding tasks into a unified alignment task , resulting in 4.7m diverse training samples. experiments show align-score achieves state-of-the-art performance on summac and true benchmark, has higher correlation with human judgements than competing metrics, and generalizes well to unseen data.