Leveraging Entailment Judgements in Cross-Lingual Summarisation (2024)

Abstract: Synthetically created Cross-Lingual Summarisation (CLS) datasets are prone to include document-summary pairs where the reference summary is unfaithful to the corresponding document as it contains content not supported by the document (i.e., hallucinated content). This low data quality misleads model learning and obscures evaluation results. Automatic ways to assess hallucinations and improve training have been proposed for monolingual summarisation, predominantly in English. For CLS, we propose to use off-the-shelf cross-lingual Natural Language Inference (X-NLI) to evaluate faithfulness of reference and model generated summaries. Then, we study training approaches that are aware of faithfulness issues in the training data and propose an approach that uses unlikelihood loss to teach a model about unfaithful summary sequences. Our results show that it is possible to train CLS models that yield more faithful and at the same time informative summaries.

## introduction a widely used method to create abstractive summarisation datasets is to crawl websites from which documents paired with reference summaries can be extracted. examples of this are synthetic datasets created in the news @xcite @xcite and instructional domains @xcite and for descriptive summarisation @xcite @xcite . the potential content misalignment in document-summary pairs created in this way raises concerns about the quality of training and evaluation data @xcite . previous work carried out manual validation of document-summary pairs in automatically created datasets to assess content overlap thereof @xcite @xcite @xcite . this inspection aims at pinpointing whether summaries convey content that cannot be inferred from the document, i.e. hallucinations. @xcite found that 70% of the pairs in the xsum dataset @xcite contain summaries with hallucinations. for multi-and crosslingual summarisation, @xcite found ~30% of the summaries to be unfaithful. taking a causal look into the hallucinations problem in automatically generated summaries, previous work @xcite @xcite @xcite @xcite inspects whether reference summaries in a dataset contain hallucinations in an automatic way. they further exploit this information about the data quality in terms of faithfulness for training with a better signal. their focus is on monolingual summarisation (the input document and output summary are in the same language) and the most studied language is english. in this work, we focus on cross-lingual summarisation where the document is written in one language (e.g., czech) and the corresponding summary is written in a different language (e.g., english). we propose to leverage cross-lingual natural language inference @xcite to supplement human validation and to automatically annotate synthetic cross-lingual datasets with hallucination judgements. our study focuses on the xwikis corpus (perez-beltrachini and lapata, 2021) consisting of descriptive summaries. it is extracted from wikipedia by aligning articles in different languages (e.g., czech and english) and re-combining the lead paragraph of one article with the body of the other to form cross-lingual document-summary pairs. it includes english, french, german, and czech and we extended it with chinese. we study simple training schemes that are aware of hallucinations occurring in reference summaries. these train with a smaller but cleaner dataset where highly unfaithful document-summary pairs are removed, include unfaithful document-summary pairs for training but ignore the unfaithful content or explicitly teach the model about unfaithful summary sub-sequences. our experiments show that by simply fine-tuning with a smaller but more faithful training set it is possible to improve faithfulness on generated summaries while maintaining their informativeness in the context of cls. ## x-nli based faithfulness estimation the xwikis corpus we focus on the xwikis corpus which covers four european languages, english (en), french (fr), german (de), and czech (cs). in addition, we follow the approach proposed in (perez-beltrachini and lapata, 2021) to extend xwikis with chinese (zh). in appendix b.2 we show statistics about the xwikis summarisation task @xcite and the added zh language. thus, xwikis covers high resource pairs, e.g., fr-en and de-en, as well as lower resource and english-distant languages, i.e., cs-en and zh-en. following previous work we focus our experiments on the all-to-english language pairs. efficacy of x-nli we examine the performance of existing multi-lingual nli models in our crosslingual setting, i.e., where the premise and hypothesis are in different languages. we choose x-nli @xcite as multi-lingual nli dataset and an off-the-shelf model, namely mt5-large, finetuned on the x-nli training set, i.e., pairs of english premise and english hypothesis. 2 the nli model will transfer zero-shot to other languages, e.g., french, where the premise and hypothesis are in the same language @xcite . the x-nli test set includes 15 languages with a total of 5,000 premise-hypothesis pairs. however, we need to verify the zero-shot performance in our cross-lingual scenario where the premise and hypothesis are in different languages. to this end, we derive cross-lingual premise-hypothesis pairs from the x-nli test set. we combine the premise in one language (e.g., french) with the hypothesis in another language (e.g., english). ## experimental setup we treat mbart50 @xcite as our backbone model and benchmark those faithfulness aware approaches introduced in section 3.2. vanilla is our baseline model fine-tuned on the original training set assuming all instances are faithful. random is a variant of clean where we remove the same number of document-summary pairs but these are randomly selected. see details about model training in appendix c.1. ## evaluation our evaluation focuses on complementary qualitative aspects, faithfulness and informativeness. ## results and discussion the obtained results @xcite . the other is, that even a model trained on noisy data as vanilla would perform better when the input document is more informative and contains the expected details to form the summary. amongst the faithfulness aware approaches, both in the original test and test f aith , clean and mask perform the closest, while unlike p r performs slightly better than these. the former ones train with less signal while unlike p r explicitly decreases probabilities of unfaithful summary sentences. ## conclusion we propose an nli-based approach to faithfulness evaluation of cross-lingual document-summary pairs. we show that the cross-lingual nli-based evaluation can approximate human judgements of summary faithfulness. we use it to automatically annotate the synthetically created xwikis 3 the cohen kappa inter-annotator agreement on fr-en is 0.25 for faithfulness and 0.43 for informativeness (0.21 and 0.39 on zh-en). we speculate that the low faithfulness agreement relates to annotators using different scoring ranges.