Language Model Augmented Relevance Score (2021)

Abstract: Although automated metrics are commonly used to evaluate NLG systems, they often correlate poorly with human judgements. Newer metrics such as BERTScore have addressed many weaknesses in prior metrics such as BLEU and ROUGE, which rely on n-gram matching. These newer methods, however, are still limited in that they do not consider the generation context, so they cannot properly reward generated text that is correct but deviates from the given reference. In this paper, we propose Language Model Augmented Relevance Score (MARS), a new context-aware metric for NLG evaluation. MARS leverages off-the-shelf language models, guided by reinforcement learning, to create augmented references that consider both the generation context and available human references, which are then used as additional references to score generated text. Compared with seven existing metrics in three common NLG tasks, MARS not only achieves higher correlation with human reference judgements, but also differentiates well-formed candidates from adversarial samples to a larger degree.

## introduction automated metrics such as bleu @xcite and rouge @xcite are popular methods for evaluating natural language generation (nlg) systems. compared with human evaluation, they are cheaper and faster, and accordingly, they often serve as essential metrics for benchmarking the performance of nlg models @xcite . despite their widespread use, however, these automated metrics often poorly correlate with ratings given by human judges, particularly for datasets in which only a single human reference exists @xcite . moreover, these automated metrics only capture similarities between generated sentences and reference candidates, crucially ignoring provided contexts that are relevant for evaluating the answer in contextual nlg tasks, such as story generation, news summarization, and question-answering @xcite . human reference. she called for help and waited to get her car fixed. ppl bleu-1 rouge-l bertscore mars candidate. her fears were confirmed when her engine was smoking. 75.58 0.223 0.182 0.338 0.574 reorder. her confirmed engine fears her when was were smoking. to address the above issues, prior studies have proposed a number of promising remedies. one line of work has proposed to combine human ratings with automated metrics @xcite . for instance, in huse score, @xcite leverages the differences between perplexity and human judgements to consider both quality and diversity of generated text. another line has proposed training separate neural models to aid automated metrics (mehri and eskenazi, 2020; yuma et al., 2020, inter alia). for instance, bleurt @xcite fine-tunes bert @xcite on synthetic reference-candidate pairs for machine translation. these methods, however, are often limited in practical use, because the high-cost human ratings are not always available for every dataset, and the data-or system-specific training is not easily extended to other domains @xcite , and can even bias the evaluation @xcite . in this paper, we present mars (language model augmented relevance score), a new nlg evaluation metric that requires neither supervision from human ratings nor additional training on specific domains. as shown in figure 1 , instead of comparing candidates only with human written references, as many prior metrics do, mars uses a mixture of both human and augmented references. specifically, mars masks tokens in the reference to create templates, and then uses the context and templates to generate augmented references by infilling the masked parts with an lm guided by reinforcement learning. the augmented references thus incorporate information from both the context and the human reference, and are enriched with lexical and syntactic diversity, facilitating fairer evaluation of candidates. finally, we compute the score as a weighted average of the similarity between the candidate and the set of augmented references in the contextual embedding space. the advantages of mars are three-fold. first, mars correlates highly with human judgements. we apply mars to three diverse nlg tasks, and demonstrate that, compared with seven popular nlg metrics, mars better correlates with human judgements and is robust against adversarial attacks. second, mars is context-aware. unlike existing metrics that only consider the given human reference, we use a constrained nlg approach to incorporate the generation context into augmented references, thus alleviating bias against diverse candidates. third, mars is easy to deploy and extend. built on off-the-shelf lms, mars requires neither human supervision nor additional training for specific domains, and can therefore serve as a general-purpose metric for a broad range of nlg applications, as we will demonstrate for three common nlg tasks: story generation, news summarization, and question-answering. ## tasks & datasets we evaluated mars and compared it with several popular nlg metrics on the following three tasks: story generation. we use the roc stories dataset foot_6 for story generation, which requires candidate nlg systems to generate coherent endings to four-sentence stories @xcite . the dataset consists of 96,198 examples of partially written stories; we take the human-rated subset ( =300) released by huse @xcite , which contains continuances by ( 1 an industry-level system based on apache solr foot_7 , and (2) an open-nmt model with global attention @xcite . news summarization. for the news summarization task, we use the newsroom summary dataset. 8 this dataset contains 1.3 million articles from 38 major publications @xcite and we use the subset with human ratings ( =540) released by the authors. 9 this dataset contains outputs from summarization models: (1) textrank: a sentencelevel summarization system inspired by google pagerank @xcite , (2) a seq2seq model with attention @xcite , and (3) pointer-n: a pointer-based neural model @xcite trained on newsroom dataset. ## human judgement we conducted human evaluation on amazon mechanical turk (mturk) to further study the quality of mars augmentation. in total 150 participants were randomly assigned to evaluate the three tasks. participants (61.3% male and 38.7% female) were all from the united states and above 18 years old, with an average age of 34.7 years old. each participant was paid 75 cents for completing 14 questions in each questionnaire (average completion time per questionnaire was about 5.11 minutes). ## limitations mars can be limited by the lm that it usesfor instance, the total length of context + reference/candidate is limited by the max sequence length of the lm used. additionally, our work has focused on english, and mars may require non-trivial modifications to handle cases where the context and reference/candidate are in different languages, such as machine translation. future work, could potentially extend mars to these scenarios using multi-lingual sequence-to-sequence models such as multilingual-t5 @xcite . we also analyzed errors and found that mars sometimes under-scores candidates that contained unknown tokens or were copied directly from the context (see appendix c for examples and further analysis). ## conclusion we have proposed mars, a context-aware and easy-to-deploy nlg metric built upon an off-theshelf language model (gpt-2). on three contextual nlg tasks, we show that mars better correlates with human judgements compared with seven other unsupervised metrics. requiring neither costly human supervision nor additional training, mars can be applied to a broad range of nlg tasks.