Understanding Translationese in Cross-Lingual Summarization (2023)

Abstract: Given a document in a source language, cross-lingual summarization (CLS) aims at generating a concise summary in a different target language. Unlike monolingual summarization (MS), naturally occurring source-language documents paired with target-language summaries are rare. To collect large-scale CLS data, existing datasets typically involve translation in their creation. However, the translated text is distinguished from the text originally written in that language, i.e., translationese. In this paper, we first confirm that different approaches of constructing CLS datasets will lead to different degrees of translationese. Then we systematically investigate how translationese affects CLS model evaluation and performance when it appears in source documents or target summaries. In detail, we find that (1) the translationese in documents or summaries of test sets might lead to the discrepancy between human judgment and automatic evaluation; (2) the translationese in training sets would harm model performance in real-world applications; (3) though machine-translated documents involve translationese, they are very useful for building CLS systems on low-resource languages under specific training strategies. Lastly, we give suggestions for future CLS research including dataset and model developments. We hope that our work could let researchers notice the phenomenon of translationese in CLS and take it into account in the future.

## introduction cross-lingual summarization (cls) aims to generate a summary in a target language from a given document in a different source language. under the globalization background, this task helps people grasp the gist of foreign documents efficiently, and attracts wide research attention from the computational linguistics community @xcite @xcite @xcite @xcite . as pointed by previous literature @xcite @xcite , one of the key challenges lies in cls is data scarcity. in detail, naturally occurring documents in a source language paired with the corresponding summaries in a target language are rare (perez-beltrachini and lapata, 2021), making it difficult to collect large-scale and high-quality cls datasets. for example, it is costly and laborintensive to employ bilingual annotators to create target-language summaries for the given sourcelanguage documents @xcite . generally, to alleviate data scarcity while controlling costs, the source documents or the target summaries in existing large-scale cls datasets @xcite @xcite @xcite are (automatically or manually) translated from other languages rather than the text originally written in that language (section 2.1). distinguished from the text originally written in one language, translated text foot_0 in the same language might involve artifacts which refer to "translationese" @xcite . these artifacts include the usage of simpler, more standardized and more explicit words and grammars @xcite as well as the lexical and word order choices that are influenced by the source language @xcite . it has been observed that the translationese in data can mislead model training as its special stylistic is away from native usage @xcite @xcite . nevertheless, translationese is neglected by previous cls work, leading to unknown impacts and potential risks. grounding the truth that current large-scale cls datasets are typically collected via human or machine translation, in this paper, we investigate the effects of translationese when the translations appear in target summaries (section 3) or source documents (section 4), respectively. we first confirm that the different translation methods (i.e., human translation or machine translation) will lead to different degrees of translationese. in detail, for cls datasets whose source documents (or target summaries) are human-translated texts, we collect their corresponding machine-translated documents (or summaries). the collected documents (or summaries) contain the same semantics as the original ones, but suffer from different translation methods. then, we utilize automatic metrics from various aspects to quantify translationese, and show the different degrees of translationese between the original and collected documents (or summaries). second, we investigate how translationese affects cls model evaluation and performance. to this end, we train and evaluate cls models with the original and the collected data, respectively, and analyze the model performances via both automatic and human evaluation. we find that (1) the translationese in documents or summaries of test sets might lead to the discrepancy between human judgment and automatic evaluation (i.e. rouge and bertscore). thus, the test sets of cls datasets should carefully control their translationese, and avoid directly adopting machine-translated documents or summaries. (2) the translationese in training sets would harm model performance in realworld applications where the translationese should be avoided. for example, a cls model trained with machine-translated documents or summaries shows limited ability to generate informative and fluent summaries. (3) though it is sub-optimal to train a cls model only using machine-translated documents as source documents, they are very useful for building cls systems on low-resource languages under specific training strategies. lastly, since the translationese affects model evaluation and performance, we give suggestions for future cls data and model developments especially on low-resource languages. contributions. (1) to our knowledge, we are the first to investigate the influence of translationese on cls. we confirm that the different translation methods in creating cls datasets will lead to different degrees of translationese. (2) we conduct systematic experiments to show the effects of the translationese in source documents and target summaries, respectively. (3) based on our findings, we discuss and give suggestions for future research. ## translationese in target summaries in this section, we investigate how translationese affects cls evaluation and training when it appears in the target summaries. for cls datasets whose source documents are natural text while target summaries are ht text, we collect another summaries (in mt text) for them via google mt. in this manner, one document will pair with two summaries (containing the same semantics, but one is ht text and the other is mt text). the translationese in these two types of summaries could be quantified. subsequently, we can use the summaries in ht text and mt text as references, respectively, to train cls models and analyze the influence of translationese on model performance. ## translationese in source documents in this section, we explore how translationese affects cls evaluation and training when it appears in the source documents. for cls datasets whose source documents are ht text while target summaries are natural text, we collect another documents (in mt text) for them via google mt service. in this way, one summary will correspond to two documents (containing the same semantics, but one is ht text and the other is mt text). next, we can use the documents in ht text and mt text as source documents, respectively, to train cls models and analyze the influence of translationese. ## discussion and suggestions based on the above investigation and findings, we conclude this work by presenting concrete suggestions to both the dataset and model developments. controlling translationese in test sets. as we discussed in section 3.3 and section 4.3, the translationese in source documents or target summaries would lead to the inconsistency between automatic evaluation and human judgment. in addition, one should avoid directly adopting machinetranslated documents or summaries in the test sets of cls datasets. to make the machine-translated documents or summaries suitable for evaluating model performance, some post-processing strategies should be conducted to reduce translationese. prior work @xcite adopts post-editing strategy to manually correct the machine-translated summaries in their test sets. though post-editing increases productivity and decreases errors compared to translation from scratch @xcite , @xcite finds that post-editing machine translation also has special stylistic which is different from native usage, i.e., post-editese. thus, the postediting strategy cannot reduce translationese. future studies can explore other strategies to control translationese in cls test sets. building mixed-quality or semi-supervised cls datasets. since high-quality cls pairs are difficult to collect (especially for low-resource languages), it is costly to ensure the quality of all samples in a large-scale cls dataset. future work could collect mix-quality cls datasets that involve both high-quality and low-quality samples. in this manner, the collected datasets could encourage model development in more directions (e.g., the curriculum learning and tagged training strategies we discussed in section 4.4) while controlling the cost. in addition, grounding the truth that monolingual summarization samples are much easier than cls samples to collect under the same resources @xcite , semi-supervised datasets, which involve cls samples and in-domain monolingual summarization samples, are also a good choice. designing translationese-aware cls models. it is inevitable to face translationese when training cls models. the degree of translationese might be different in the document or summary of each training sample when faced with one of the following scenarios: (1) training multi-domain cls models based on multiple cls datasets; (2) training cls models in low-resource languages based on mixed-quality datasets. in this situation, it is necessary to build translationese-aware cls models. the tagged training strategy @xcite is a simple solution which only considers two-granularity translationese in source documents. it is more general to model the three-granularity translationese (i.e., natural text, ht text and mt text) in documents as well as summaries. future work could attempt to ( 1 ) explicitly model the dif-ferent degrees of translationese (such as prepending special tokens), or (2) implicitly let the cls model be aware of different degrees of translationese (e.g., designing auxiliary tasks in multi-task learning). cross-lingual summarization. cross-lingual summarization (cls) aims to summarize sourcelanguage documents into a different target language. due to data scarcity, early work typically focuses on pipeline methods @xcite @xcite , i.e., translation and then summarization or summarization and then translation. recently, many large-scale cls datasets are proposed one after another. according to an extensive survey on cls @xcite , they can be divided into synthetic datasets and multi-lingual website datasets. synthetic datasets @xcite @xcite are constructed by translating monolingual summarization (ms) datasets. multi-lingual website datasets @xcite are collected from online resources. based on these large-scale datasets, many researchers explore various ways to build cls systems, including multi-task learning strategies @xcite , knowledge distillation methods @xcite , resource-enhanced frameworks @xcite and pre-training techniques @xcite @xcite . more recently, @xcite explore zeroshot cls by prompting large language models. different from them, we are the first to investigate the influence of translationese on cls. translationese. translated texts are known to have special features which refer to "translationese" @xcite . the phenomenon of translationese has been widely studied in machine translation (mt). some researchers explore the influence of translationese on mt evaluation @xcite @xcite . to control the effect of translationese on mt models, tagged training @xcite is proposed to explicitly tell mt models if the given data is translated texts. besides, @xcite and @xcite mitigate the effect of translationese in cross-lingual transfer learning. ## conclusion in this paper, we investigate the influence of translationese on cls. we design systematic experiments to investigate how translationese affects cls model evaluation and performance when translationese appears in source documents or target summaries. based on our findings, we also give suggestions for future dataset and model developments.