Detecting and Mitigating Hallucinations in Multilingual Summarisation (2023)

Abstract: Hallucinations pose a significant challenge to the reliability of neural models for abstractive summarisation. While automatically generated summaries may be fluent, they often lack faithfulness to the original document. This issue becomes even more pronounced in low-resource languages, where summarisation requires cross-lingual transfer. With the existing faithful metrics focusing on English, even measuring the extent of this phenomenon in cross-lingual settings is hard. To address this, we first develop a novel metric, mFACT, evaluating the faithfulness of non-English summaries, leveraging translation-based transfer from multiple English faithfulness metrics. Through extensive experiments in multiple languages, we demonstrate that mFACT is best suited to detect hallucinations compared to alternative metrics. With mFACT, we assess a broad range of multilingual large language models, and find that they all tend to hallucinate often in languages different from English. We then propose a simple but effective method to reduce hallucinations in cross-lingual transfer, which weighs the loss of each training example by its faithfulness score. This method drastically increases both performance and faithfulness according to both automatic and human evaluation when compared to strong baselines for cross-lingual transfer such as MAD-X. Our code and dataset are available at https://github.com/yfqiu-nlp/mfact-summ.

## introduction recent neural abstractive summarisation models @xcite @xcite @xcite have shown promise in terms of rouge scores @xcite . however, a well-known problem with these models is hallucination @xcite @xcite )-generating summaries that cannot be supported by ground-truth knowl-edge (e.g., news, documents, meeting notes). anecdotally, it was shown that the percentage of generated summaries for cnn/dailymail @xcite and xsum @xcite containing hallucinations amounts to up to 74.8% and 96.9% @xcite , respectively. hallucinations hinder the reliability of abstractive summarisation systems by potentially misleading users with the misinformation they produce. in addition, current summarisation models, opensource or proprietary, struggle in low-resource settings @xcite @xcite , when the target language is under-represented (e.g., vietnamese and urdu). fortunately, cross-lingual transfer methods @xcite @xcite leverage task-specific knowledge learned from a resource-rich source language to summarise documents in many resource-poor target languages, in a zero-shot fashion or only with few annotated examples. nevertheless, it remains unclear to what extent cross-lingual summarisation suffers from the problem of hallucination, compared to monolingual systems where english is the only language. the main challenge in addressing this question is that most faithfulness evaluation metrics are available only for english and do not support lowresource languages. hence, our first contribution (section 2) is a model-based metric (mfact) that measures the factual consistency of multilingual conditional generation, obtained from four diverse english faithfulness metrics @xcite @xcite via 'translate train' knowledge transfer @xcite . as illustrated in figure 1 , we use existing faithfulness metrics to label the english documentsummary pairs as positive (i.e., faithful) or negative (i.e., hallucinated) and translate them into each target language. we then train a classifier in each we average the score of four english metrics to rank the training samples in xsum. we then translate the most faithful and hallucinated samples into each target language and train a classifier to distinguish them. equipped with this new metric, we conduct extensive cross-lingual transfer experiments on xl-sum @xcite for abstractive summarisation in six typologically diverse languages: chinese, spanish, french, hindi, turkish and vietnamese. we find that state-of-the-art cross-lingual transfer methods increase summarisation performance in the target languages, but also introduce more hallucinations compared to english monolingual models in comparable experimental settings, thus further exacerbating this tendency (section 6). we also employ the mfact metric to assess the faithfulness of some recently released multilingual large language models (llms), including phoenix, bloomz, and vicuna @xcite @xcite . we show that llms that use multilingual data for pre-training or conversational finetuning fail to ensure faithfulness in summarisation in various languages, producing more hallucinations in low-resource ones. to overcome this limitation and promote faithful summarisation in multiple languages, we adapt a series of existing methods for reducing hallucinations originally devised for monolingual summarisation (section 3.2). in addition, we introduce a novel, simple but effective method (section 3.3): we weigh the loss for each training example according to their faithful scores. we evaluate our loss-weighting method with automated metrics and human judgements. we observe significant gains in both summarisation performance and faithfulness over a series of strong baselines (section 8). in a nutshell, our main contributions are the following: • we propose mfact, a multilingual faithful metric developed from four english faithfulness metrics. this enables detecting hallucinated summaries in languages other than english. ## mfact: a multilingual metric for faithfulness the lack of faithful metrics in languages other than english greatly limits the evaluation (and hence, the prevention) of hallucinations in cross-lingual transfer for low-resource languages. in this section, we fill this gap by introducing mfact. this is constructed by transferring multiple english faithful metrics into any target language, given the availability of a machine translation model. ## reducing hallucination in cross-lingual transfer we first provide some background on cross-lingual transfer. then, we show how to adapt several methods promoting faithfulness in monolingual summarisation to cross-lingual transfer settings. finally, we describe a new approach based on loss weighting. ## experimental setup evaluation metrics. we use rouge-1/2/l scores @xcite to evaluate the task of abstractive summarisation. we use the four metrics mentioned in section 2 to evaluate the faithfulness of english summaries and our mfact metric for summaries in other languages. dataset. we conduct our experiments on xl-sum, which is a large-scale multilingual summarisation dataset @xcite . xl-sum provides a large collection of annotated document-summary pairs in 45 languages in addition to english. we test our approach on six target languages: chinese, spanish, french, hindi, turkish and vietnamese. ## cross-lingual transfer introduces additional hallucinations the second analysis of this paper aims to corroborate our observation that cross-lingual transfer can introduce additional hallucinations over monolingual fine-tuning, though it improves the task performance for summarisation in the target language. transfer setup. we compare two data scenarios and two styles of fine-tuning. to begin, we investigate the impact of initial training on source data, followed by applying few-shot learning techniques on target data (cross-lingual transfer) instead of direct application. we attribute the difference in faithfulness scores to the additional hallucinations introduced by the training phase in the source language. taking chinese as an example of few-shot cross-lingual transfer, we train the summarisation model first on xl-sum (chinese) and then with 1k randomly sampled xsum (english) examples. secondly, we compare fine-tuning the full model, where all parameters are updated, with parameterefficient fine-tuning, where only the adapters are updated. this allows us to study the effect of different transfer methods on faithfulness. ## hallucinations in multilingual large language models we also assess the summarisation performance of recent multilingual large language models (llms) on xl-sum in bloomz with an additional 267k and 189k instances of multilingual instructions and conversation rounds. we select three languages, aside from english, which are present in the pre-training data for bloomz and the conversational tuning data for vicuna and phoenix. we also report the percentage of examples in each of these languages that these models have been exposed to during their ## reducing hallucinations in this section, we test different methods for crosslingual transfer of summarisation to multiple languages and for promoting faithfulness. we compare our new method of loss weighting based on mfact with mad-x, as well as with a series of approaches for reducing hallucinations (section 4). we evaluate these methods with automated metrics for performance, faithfulness, and abstractiveness (i.e., the ability to rephrase the document instead of copy-pasting spans of text). we also conduct human evaluations to corroborate these results. automatic evaluation. we report rouge scores for performance, faithfulness (mfact), and abstractiveness (novel bigrams and trigrams in the summary) for the test set of each target language in figure 3 illustrates the distributions of abstractiveness and faithfulness for all models in six xl-sum datasets. both positive and negative predictions of mfact scatter with different levels of abstractiveness. we also observe that summaries generated by the weighted loss method generally have a higher level of abstractiveness when they are similarly faithful compared with other baselines. the results in figure 2 indicate that human evaluators prefer the summaries generated by our weighted loss method rather than mad-x, demonstrating that our weighted loss approach improves faithfulness and informativeness for all six languages. finally, we study the correlation between the human preferences from figure 2 and various faithfulness metrics presented in section 5.1. from ## related work while faithfulness in summarisation is a highly researched topic, previous works focused mostly on the english language @xcite @xcite . to evaluate faithfulness, state-of-the-art methods fall into three categories. firstly, validating faithfulness can be cast as a classification problem @xcite @xcite . secondly, faithfulness can be interpreted as answerability, and assessed with existing question answering models @xcite . finally, language models may be adopted to identify extrinsic hallucinations @xcite . to improve faithfulness in summarisation models, an approach related to ours changes the training dynamics, e.g. by filtering out hallucinated data @xcite @xcite . the expert/anti-expert approach aims to learn experts and anti-experts that alter the behaviour of a base generative model @xcite @xcite . other methods include designing the specific neural architecture @xcite @xcite , summary ranking @xcite and posthoc correction @xcite @xcite . however, there is still a limited understanding of the effectiveness of these methods in cross-lingual transfer. ## conclusion we investigate how to measure and mitigate hallucinations of summarisation models in cross-lingual transfer scenarios. we first propose a multilingual metric, mfact, to facilitate the evaluation of faithfulness in low-resource languages. by virtue of this new metric, we find empirical evidence that while common cross-lingual transfer methods benefit summarisation performance, they amplify hallucinations compared to monolingual counterparts. we also point out that faithfulness in summarisation for languages other than english is still challenging for multilingual large language models. finally, with the aim of reducing these hallucinations, we adapt several monolingual methods to crosslingual transfer and propose a new method based on weighting the loss according to the mfact score of each training example. based on both automated metrics and human evaluation, we demonstrate that mfact is the most reliable metric in detecting hallucinations in multiple languages. moreover, compared to a series of state-of-the-art baselines, we find that summaries produced by loss weighting achieve higher performance and abstractiveness, competitive faithfulness, and a higher alignment with human preferences. we hope that this work will attract more attention from the community to the phenomenon of hallucination in languages different from english and facilitate future research by establishing evaluation metrics and baselines.