Revisiting Cross-Lingual Summarization: A Corpus-based Study and A New Benchmark with Improved Annotation (2023)

Abstract: Most existing cross-lingual summarization (CLS) work constructs CLS corpora by simply and directly translating pre-annotated summaries from one language to another, which can contain errors from both summarization and translation processes. To address this issue, we propose ConvSumX, a cross-lingual conversation summarization benchmark, through a new annotation schema that explicitly considers source input context. ConvSumX consists of 2 sub-tasks under different real-world scenarios, with each covering 3 language directions. We conduct thorough analysis on ConvSumX and 3 widely-used manually annotated CLS corpora and empirically find that ConvSumX is more faithful towards input text. Additionally, based on the same intuition, we propose a 2-Step method, which takes both conversation and summary as input to simulate human annotation process. Experimental results show that 2-Step method surpasses strong baselines on ConvSumX under both automatic and human evaluation. Analysis shows that both source input text and summary are crucial for modeling cross-lingual summaries.

## introduction with the advance in deep learning and pre-trained language models (plms) @xcite @xcite , much recent progress has been made in text summarization @xcite @xcite . however, most work focuses on english (en) data @xcite @xcite , which does not consider crosslingual sources for summarization @xcite . to address this limitation, cross-lingual summarization (cls) aims to generate summaries in a target language given texts from a source language @xcite , which has shown values pipeline annotation results in errors from both summarization (red: unmentioned content/hallucination) and translation (cyan: incorrect translation) processes. to address this issue, we explicitly annotate targetlanguage summaries with faithfulness rectification (green) based on input context, with the guidance of mono-lingual summaries. most existing work @xcite @xcite constructs cls corpora by translating summaries from existing mono-lingual summarization datasets into other languages, which is de facto a "pipeline" annotation protocol (first summarize, then translate) as shown in figure 1 . however, such an annotation method can suffer from two major problems: first, summaries from mono-lingual summarization corpora (summarization process) can contain errors @xcite , which are likely to be preserved in translated summaries. for example, the english summary in figure 1-(a ) contains unmentioned con-tent/hallucination (red text), which leads to the same discrepancy as in the translated summary (figure 1-(b) , red text). second, the translation process can further introduce errors, in particular for polysemous words. for example, in figure 1 -(b), the term, "ex-viking" (which refers to previous members of the minnesota vikings team), is mistakenly translated into "前海盗" (which means "expirate/buccaneer"). to determine proper translation, it require more information beyond the scope of short summaries. to qualitatively understand the above problems, we conduct human evaluation and error analysis on existing popular cls corpora. empirical results show that existing corpora suffer from the two aforementioned problems, containing significantly many hallucinations and factual errors. 1 in particular, we find that overall 20 ∼ 67% of summaries in cls datasets contain errors, where 7 ∼ 46% and 13 ∼ 47% of summaries suffer from summarization and translation processes, respectively. this suggests that the pipeline protocol, which is widely used in cls research, can result in low-quality data and negatively impact the validity of modeling research. in addition, fine-grained error analysis shows that 55.6 ∼ 89.1% of translation errors can be resolved with the help of input context. motivated by the above findings and to address this issue, we propose the protocol that crosslingual summaries should be sourced from original input text where mono-lingual summaries can serve as a quick review for salient information. with this concept, we annotate cross-lingual summaries (s tgt ) by relying on source text (d src ) and sourcelanguage summaries (s src ) as shown in figure 1-(c ). such an annotation protocol brings three advantages: first, compared with translation only given s src , rich context information from d src helps annotators to disambiguate word senses and comprehend s src accurately, e.g., "前维京人队" (which means "ex-viking team player") in figure 1-(c ); second, d src is more reliable and can provide ground-truth information to correct potential errors in s src , e.g., red text in figure 1-(a) ; third, compared with writing s tgt only given d src , s src can serve as supplement guidance to help annotators be aware of what should be involved in the summaries, ensuring that salient information in s src and s tgt is aligned. using cls protocol, we build convsumx, a new benchmark to facilitate future cls research. con-vsumx focuses on conversational text in a few-shot setting. compared with monologue (e.g., news), conversational text is less explored yet is also practically useful in real-world scenarios @xcite . convsumx contains two sub-tasks, namely dialogsumx and qmsumx, based on two english conversation summarization datasets di-alogsum @xcite and qmsum @xcite , respectively. each covers three language-directions, taking en as the source, and mandarin (zh), french (fr) and ukrainian (ukr) as target languages. we empirically compare different annotations using the pipeline protocol and our cls protocol with human evaluation. analysis shows that by considering input context, our protocol can significantly reduce annotation errors, suggesting convsumx is a high-quality benchmark in terms of cross-lingual faithfulness. based on the same intuition that d src and s src can serve as a critical complement to each other, we propose a 2-step framework for cls, which fine-tunes a multi-lingual plm using concatenated s src and d src as input, and s tgt as output. experimental results show that our conceptual framework yields surprisingly better performance over strong baselines on convsumx. analysis and human evaluation show that our method can effectively generate more faithful cross-lingual summaries in a low-resource setting, and verify that source input text and summaries are supplementary to each other in modeling cross-lingual summaries. to summarize, our contributions are the following: we release convsumx at https://github.com/ cylnlp/convsumx . ## analyzing existing cls corpora we conduct a corpus-based study on existing popular human-annotated cls corpora, namely ncls, xsamsum and xmediasum, covering both monologue and dialogue texts. ncls @xcite is the first large crosslingual news summarization corpus, which is constructed by automatically translating existing monolingual summarization datasets and using a roundtrip strategy with human post-editing on test sets. xsamsum and xmediasum are both from clidsum @xcite , where they manually translate summaries from two english dialogue summarization datasets, namely samsum @xcite and mediasum @xcite , into mandarin and german. ## convsumx to address the aforementioned issues in pipeline annotation, we propose convsumx with a new annotation protocol, focusing on few-shot cls. con-vsumx contains two cross-lingual summarization scenarios, namely daily dialogue summarization, and query-based summarization, covering 3 language directions: en2zh, en2fr and en2ukr. ## conclusion we conducted data analysis on 3 typical corpora and showed that the pipeline annotation protocol suffers from errors from both the summarization and translation processes. to address these issues, we proposed that cross-lingual summaries should be sourced from source input text. based on this principle, we annotated a more faithful cls benchmark, convsumx by relying on both sourcelanguage texts and summaries. based on the same intuition, we proposed a 2-step method that takes both source text and source summaries as input. ex-perimental results showed that 2-step method outperforms strong baselines on convsumx, demonstrating that both source-language texts and summaries are crucial in modeling cross-lingual summaries and are complementary to each other. to our knowledge, we are the first to show that summary translation has limitations for cls, giving a more faithful solution.