Evaluating Factuality in Cross-lingual Summarization (2023)

Abstract: Cross-lingual summarization aims to help people efficiently grasp the core idea of the document written in a foreign language. Modern text summarization models generate highly fluent but often factually inconsistent outputs, which has received heightened attention in recent research. However, the factual consistency of cross-lingual summarization has not been investigated yet. In this paper, we propose a cross-lingual factuality dataset by collecting human annotations of reference summaries as well as generated summaries from models at both summary level and sentence level. Furthermore, we perform the fine-grained analysis and observe that over 50% of generated summaries and over 27% of reference summaries contain factual errors with characteristics different from monolingual summarization. Existing evaluation metrics for monolingual summarization require translation to evaluate the factuality of cross-lingual summarization and perform differently at different tasks and levels. Finally, we adapt the monolingual factuality metrics as an initial step towards the automatic evaluation of summarization factuality in cross-lingual settings. Our dataset and code are available at https://github.com/kite99520/Fact_CLS .

## introduction cross-lingual summarization, the task of generating a summary in different languages from the source documents, aims to help people efficiently grain the main point of the original document. it is recognized as a challenging task that combines the difficulties of text summarization as well as machine translation. traditional pipeline methods first translate the document and then summarize it in the target language or vice versa @xcite @xcite @xcite . currently, modern neural cross-lingual summarization models have factuality, a crucial dimension, is absent from the current evaluation of cross-lingual summarization approaches. rouge @xcite is the main automatic evaluation metric. informativeness, fluency, and conciseness are the dimensions of human evaluation. however, many case studies have pointed out that the summaries generated by neural cross-lingual summarization models have factual errors @xcite @xcite . to fill the gap, we collect summaries from six models on a cross-lingual summarization dataset proposed by @xcite and obtain the human judgments of fine-grained factuality. the result of human evaluation suggests that over half of the generated summaries and over 27% of reference summaries contain at least one factual error. during the annotation process, we identify the peculiarity of factual errors in cross-lingual summaries, such as translation-related errors. further, since the existing monolingual factuality metrics require the aid of translation to use in cross-lingual settings, after analyzing their performance, we explore the challenging automatic evaluation of factuality in cross-lingual summarization. in summary, our contributions are as follows: ## typology of factual errors we define a typology of ten factual errors by analyzing both reference summaries and generated summaries. an example for each error type is shown in terminologies error (tere): this occurs when the terminologies in the source document cannot be expressed professionally or accurately in summary. @xcite find that terminologies error ranks first among the high-frequency errors in machine translation. finally, we add the additional type other error (othe) to ensure the completeness of the typology. this occurs when the error does not correspond to any of the above types. ## conclusion in this work, we comprehensively evaluate and analyze the factuality of reference summaries and model-generated summaries in cross-lingual summarization, showing that there are special factual errors in them. automatic evaluation of cross-lingual summarization is yet to be addressed due to the shortcomings of reference summaries and the limitations of monolingual factuality metrics. moreover, our exploration of the automatic factuality evaluation in cross-lingual settings illustrates its challenging nature.