Multilingual Summarization with Factual Consistency Evaluation (2023)

Abstract: Abstractive summarization has enjoyed renewed interest in recent years, thanks to pre-trained language models and the availability of large-scale datasets. Despite promising results, current models still suffer from generating factually inconsistent summaries, reducing their utility for real-world application. Several recent efforts attempt to address this by devising models that automatically detect factual inconsistencies in machine generated summaries. However, they focus exclusively on English, a language with abundant resources. In this work, we leverage factual consistency evaluation models to improve multilingual summarization. We explore two intuitive approaches to mitigate hallucinations based on the signal provided by a multilingual NLI model, namely data filtering and controlled generation. Experimental results in the 45 languages from the XLSum dataset show gains over strong baselines in both automatic and human evaluation. We release models and human judgements of summaries to foster progress towards more factually consistent multilingual summarization.

## introduction the past few years have witnessed a huge leap forward in abstractive summarization thanks to largescale pretraining @xcite and the availability of benchmark datasets. a well-known issue limiting the wider adoption of abstractive summarization models is their tendency to generate factually inconsistent summaries, a.k.a "hallucinations" @xcite . a recently popular line of work explores how to best detect hallucinations in machine generated text, thereby enabling the automatic identification of factually inconsistent summaries @xcite @xcite figure 1 : nli and rouge scores for different models on the arabic development set of xlsum during finetuning. using a multilingual entailment model during training (via data filtering or controlled generation) improves summary quality over a baseline model trained without using the entailment signal. @xcite @xcite @xcite . while such approaches may prove useful for automatic evaluation, it remains unclear how to best leverage them for improving summarization models in multiple languages. while focusing exclusively on english, previous work suggests different techniques to this effect such as discarding "noisy" training examples @xcite , contrastive learning paradigms @xcite , controlled generation and planning @xcite , or reinforcementlearning approaches that use the evaluation model score as a reward function @xcite . despite promising results, no method has emerged as a clear winner in english, let alone across languages with varying amounts of data and resources. in this work, we leverage factual consistency evaluation models to improve summarization systems in multiple languages. specifically, we employ textual entailment models (a.k.a. natural language inference; @xcite in order to determine whether a summary is factually consistent @xcite . we opportunistically opt for nli given the availability of multilingual benchmarks for model training @xcite . approaches based on question generation and answering have been also shown to work well for factuality evaluation (i.e. @xcite @xcite , however, they are not easily portable due to the scarcity of respective resources in languages other than english. we first analyze the quality of the training data for summarization models using a strong multilingual nli model (as evaluated on the xnli dataset, @xcite . in particular, we train our multilingual nli model, following the guidelines from the true survey @xcite for the assessment of factual consistency. focusing on the xlsum foot_0 multilingual summarization dataset @xcite , we find that for some languages up to 70% of training examples are not factually consistent according to the nli model, while such examples are commonly used for training. we use the nli signal to improve the quality of the generated summaries in two ways: (1) data filtering, where we only train on examples whose summaries are predicted to be entailed by the input, and (2) controlled generation, where we also leverage "negative" training examples by conditioning the summarization model on the nli signal. we evaluate the proposed approaches using both automatic and human evaluation in 45 languages, and observe significant gains in the faithfulness of the generated summaries over strong baselines. finally, we show that the human judgments we collected in all languages are useful for training automatic metrics to assess the quality, factual consistency and informativeness of generated summaries. to summarize, the contributions of this work are three-fold: (1) we analyze the quality of the xlsum dataset @xcite using strong multilingual nli models and reveal severe issues with faithfulness in the training data across languages; (2) we explore methods for improving downstream summarization models trained on this data using a multilingual nli signal, and show large gains in both automatic and human evalua-tion; and (3) using the data from our large-scale human evaluation study, we learn metrics for automatically evaluating summaries in multiple languages along the dimensions of quality, factual consistency, and informativeness. 3 to the best of our knowledge, our work is the first to examine the faithfulness of summarization systems in multilingual settings, and we hope it will encourage the development of better metrics and models in multilingual text generation. ## related work despite significant improvements in recent years @xcite @xcite , abstractive summarization models are still prone to "hallucination", i.e., the inclusion of factual errors in the generated summaries @xcite @xcite . a plethora of approaches have been proposed for the automatic detection of factual inconsistencies in machine generated text (see @xcite for overviews) with varying degrees of success. there is growing consensus that techniques based on textual entailment @xcite @xcite and question generation and answering models @xcite @xcite @xcite achieve strong performance across tasks and datasets @xcite . another line of work uses synthetically generated data to train models for evaluating factual consistency @xcite @xcite . aside from assessing system output, several studies have proposed novel model architectures which enforce factuality during training or inference. these include extracting facts from the source and incorporating them as additional input to the model @xcite @xcite , planning using entity chains and avoiding entities that are not in the input @xcite , using reinforcement learning to optimize model training with factual correctness as a reward @xcite @xcite , reranking candidate summaries within a beam using entailment predictions @xcite or quantity verification scores @xcite , using contrastive learning @xcite , modifying the training objective to only maximize the likelihood of factual words @xcite , incorporating factuality into the pretraining objective of models tailored to text summarization tasks @xcite , and adaptively removing examples with high log loss (kang and hashimoto, 2020). other work simply removes noisy training samples @xcite in the hope that factuality will improve by training on better examples. despite promising results, it is unclear whether previous techniques transfer to languages beyond english. our own work aims to improve the factuality of abstractive summarization across languages. leveraging recent progress on multilingual pretrained models @xcite , we show that entailment-based metrics can be trained to detect factually-inconsistent summaries in multiple languages, and that this signal can be leveraged to improve summarization systems in those languages. ## multilingual factual consistency evaluation we cast factual consistency evaluation as a natural language inference (nli) task. the input forms the premise, the summary forms the hypothesis @xcite @xcite , and the nli model is used to predict whether the summary is entailed by the input. more formally, given input document d and summary s, we define an nli model m as a binary classifier, where m(d, s) ≈ p(s is entailed by d). recent studies (honovich et al., 2022) on evaluating factual consistency in summarization and other related tasks in english have obtained promising results when finetuning large pretrained models on nli datasets. specifically, they finetune the t5 pretrained encoder-decoder models @xcite for binary classification where the entailment relation translates to a positive label and contradiction/neutral relations are merged to a negative label. their model encodes the concatenation of the premise (document) and hypothesis (summary) and decodes a single token that represents the class label (entailment or no entailment). 4 4 other work @xcite breaks documents into sentences before running nli models, since we are interested in evaluating factual consistency in multiple languages, we extend the modeling approach of @xcite to a multilingual setting. as our pretrained model, we use mt5-xxl @xcite which was trained on mc4, a dataset drawn from the public common crawl covering 101 languages. we finetuned mt5-xxl on the anli @xcite and @xcite datasets. anli contains 162k english-only examples, while xnli has 37k examples foot_2 in 15 languages. as mentioned above for the english case, the multilingual model is trained to generate a binary label when given the concatenation of a premise and hypothesis, where the positive label corresponds to an entailment relation, and the negative label stands for a neutral/contradiction relation. during inference, we score a premise and hypothesis input by measuring the output probability when force-decoding the positive label, resulting in a score between 0 (no entailment) and 1 (entailment). we measured the quality of our multilingual nli model by evaluating on the xnli @xcite test set and the true benchmark @xcite . the latter is a standardized collection of datasets representing various tasks (summarization, dialog generation, paraphrasing and fact-checking) with manual annotations for factual consistency. on xnli, our model yields an average accuracy of 90.0 over 15 languages in comparison to 87.8 reported in @xcite . 6 we present results for individual languages in appendix a. on the (english-only) true benchmark, our model's average roc auc is 82.4 in comparison to 83.4 reported in @xcite for their best performing english-only, t5-11b model @xcite trained on anli. while our model is trained on both anli (english) and xnli (15 languages, detailed in ## summarization models we next describe two summarization approaches which exploit the factual consistency evaluation however, we refrained from doing so to avoid loss of context. ## experimental setup we focus on xlsum @xcite , a recently introduced summarization dataset. xlsum extends to multiple languages the methodology put forward in @xcite for the creation of the english-only xsum; it contains 1 million bbc article-summary pairs covering 45 languages. ## metric learning for multilingual summary evaluation our large-scale judgment elicitation study (across multiple languages and system outputs) delivered valuable annotations of summary document-quality (31,499 pairs x 3 quality dimensions x 3 raters). we next explore whether it is possible to learn metrics for evaluating quality, attribution, and informativeness automatically. existing metrics (e.g., bleurt; sellam et al. 2020) have not targeted summarization specifically, or considered attribution, and multiple languages. let @xmath0, . . . , s r ) denote a summary of length r where each s i is a token and let @xmath1, . . . , d p ) be its corresponding input document of length p. let {(d i , x i , y i )} n @xmath2. our goal is to learn a function f : (d, x) → y that predicts the human rating. we finetuned three models based on mt5-xxl @xcite , one per dimension (details in appendix f). the input was the concatenation of a document and its summary, and the output the human rating. 10% of the elicited ratings (across languages) were reserved for testing, while the remainder was used for training and validation. overall, we observe that learned metrics correlate best with human ratings (across dimensions). rouge correlates weakly with human judgments but cannot distinguish any dimension in particular, whereas nli scores reliably correlate with attribution. our results underscore the need for better and more fine-grained evaluation of summary quality, and also corroborate well-known issues @xcite with widely adopted lexical overlapbased metrics such as rouge.