C lid S um: A Benchmark Dataset for Cross-Lingual Dialogue Summarization (2022)

Abstract: We present ClidSum, a benchmark dataset towards building cross-lingual summarization systems on dialogue documents. It consists of 67k+ dialogue documents and 112k+ annotated summaries in different target languages. Based on the proposed ClidSum, we introduce two benchmark settings for supervised and semi-supervised scenarios, respectively. We then build various baseline systems in different paradigms (pipeline and end-to-end) and conduct extensive experiments on ClidSum to provide deeper analyses. Furthermore, we propose mDialBART which extends mBART via further pre-training, where the multiple objectives help the pre-trained model capture the structural characteristics as well as key content in dialogues and the transformation from source to the target language. Experimental results show the superiority of mDialBART, as an end-to-end model, outperforms strong pipeline models on ClidSum. Finally, we discuss specific challenges that current approaches faced with this task and give multiple promising directions for future research. We have released the dataset and code at https://github.com/krystalan/ClidSum.

## introduction cross-lingual summarization (xls) aims at generating a summary in the target language from a given document in a different source language. existing xls datasets mainly focus on singleparticipant documents, such as news reports @xcite @xcite , how-to guides @xcite and encyclopedia articles @xcite . many efforts have been devoted to improving xls for single-participant documents and achieved great success @xcite ; shen figure 1 : an example of cross-lingual dialogue summarization. blue and orange sentences are the german and chinese summaries of the given english dialogue, respectively. @xcite @xcite @xcite @xcite . different from single-participant documents, dialogue is a discourse produced by more than one person @xcite . the multi-participant dialogue documents, which record the communications between human and human/machine, have attracted wide research attention due to their key role in daily interpersonal interaction @xcite . meanwhile, the globalization progress has prompted conversations among interlocutors of different native languages and brought many scenarios, e.g., international academic conferences and business meetings. thus, it is valuable to provide speakers with the summary in their familiar language to help them efficiently grasp the gist of a foreign language dialogue. nevertheless, dialogueoriented xls is still under-explored due to the lack of corresponding datasets. to this end, we introduce cross-lingual dialogue summarization (xlds) task that aims to summarize a dialogue in the source language into a different language. to promote the xlds research, we construct clidsum (cross-lingual dialogue summarization), the first large-scale xlds benchmark dataset with three features: (1) the proposed clidsum is based on two existing monolingual dialogue summarization datasets, i.e., samsum @xcite and mediasum @xcite . we choose these two datasets under the consideration of their quality and diversity. (2) to make these datasets suitable for xlds, we employ professional translators to translate original english summaries of samsum and mediasum to german and chinese. eventually, the translated corpora constitute clidsum, which totally contains ~56.4k en⇒de and ~56.4k en⇒zh xlds samples 1 . (3) besides the supervised benchmark setting that has been discussed by most previous xls work, we argue that it is necessary to utilize large-scale monolingual dialogue summarization pairs for xlds due to the dearth of cross-lingual samples. thus, we design a semi-supervised setting, where a large number of monolingual pairs together with a relatively small scale of cross-lingual pairs are used to build xlds systems. based on clidsum, we build and evaluate various baseline systems, including summarize-thentranslate, translate-then-summarize and end-toend paradigms: (1) for summarize-then-translate baselines, we use a monolingual dialogue summarizer to generate summaries and further translate them to the target language through a machine translation (mt) model/service. (2) for translate-then-summarize baselines, we utilize a mt model/service to translate the dialogue documents from the source language to the target language, and then obtain summaries via a monolingual dialogue summarizer. (3) as for end-to-end paradigm, we adopt mbart-50 @xcite , a multi-lingual sequence-to-sequence (seq2seq) model which has been pre-trained on a large-scale corpora across 50 languages using denoising objectives. though such a general model achieves surprising results in many downstream tasks, its performance will naturally degrade when there is a relatively large gap between the pre-training and fune-tuning stages @xcite . therefore, to narrow the gap and fully use large-scale monolingual dialogue summarization pairs provided in semi-supervised setting, we propose mdialbart that extends mbart-50 through the second stage of pre-training with four objectives: action infilling, utterance permutation, monolingual dialogue summarization and machine translation. specifically, action infilling and utterance permutation encourage the model to 1 en: english; de: german; zh: chinese. effectively capture the structural characteristics in dialogues. monolingual dialogue summarization gives the model the ability to summarize dialogues. machine translation enables the model to learn the transformation from the source language to the target language. the pre-training corpora are provided in the semi-supervised setting, which means we do not use additional data other than clidsum. the experimental results show that mdialbart outperforms strong pipeline baselines on clid-sum. we also conduct human studies to compare generated summaries from different methods and discuss specific challenges that current approaches faced with xlds. we hope that our work could promote the development of xlds. our main contributions are concluded as follows: • we introduce xlds task and present clidsum, the first large-scale xlds benchmark dataset collecting about 56.4k en⇒de and 56.4k en⇒zh samples via crowd-sourcing. furthermore, we design supervised and semi-supervised benchmark settings on clidsum. dialogue summarization. dialogue summarization aims to summarize the dialogue document into a shorter passage. ami @xcite and icsi @xcite are two early meeting corpora that both contain hundreds of samples. recently, the samsum corpus @xcite is introduced, which contains over 16k chat conversations with human-labeled summaries. medi-asum @xcite collects about 464k interview transcripts and uses the overview descriptions as summaries. gupshup @xcite develops the first code-switched dialogue summarization dataset whose dialogues are in hindi-english. based on these datasets, a lot of work @xcite @xcite @xcite models the conversation characteristics and achieves great performance. all these efforts are given to monolingual or code-switched scenarios while we focus on cross-lingual scenario in this paper. ## clidsum in this section, we first discuss how we choose existing monolingual dialogue summarization datasets for the construction of clidsum ( § 3.1). then, we introduce the details of the annotation process that translates original summaries to target languages ( § 3.2). we next give statistic analysis of clidsum ( § 3.3). finally, we formulate xlds task ( § 3.4) and give the details of benchmark settings ( § 3.5). ## mdialbart as suggested by previous work @xcite , building an end-to-end model is preferable to a pipeline one due to: 1) the pipeline models suffer from error propagation; 2) the translation systems in pipeline paradigm require either a large parallel corpus to train mt models or the monetary cost of paid mt services; 3) the pipeline models have a recurring latency during inference @xcite . thus, it is valuable and urgent to explore end-to-end models on xlds. to this end, we propose mdialbart which extends mbart-50 via a second stage of pre-training, where the objectives help the pre-trained model better adapt to the xlds task (cf., figure 2 ). ## experiments we evaluate mdialbart and various baselines on clidsum. four automatic metrics are adopted in our experiments: rouge-1 (r1) and rouge-2 (r2) @xcite evaluate unigram and bigram overlap between the generated summaries and correspondings references, respectively. rouge-l (r-l) @xcite is applied to find the length of the longest common subsequence. bertscore (b-s) @xcite evaluates the semantic similarity of generated sentences against the references. for evaluation toolkits and the implementation details of all models, please refer to appendix a. ## conclusion in this paper, we introduce xlds task and present clidsum, the first large-scale xlds benchmark dataset. we also propose mdialbart, a xldsoriented pre-trained model that extends mbart-50 via the second pre-training stage. the carefully designed pre-training tasks help mdialbart better adapt to the xlds task. experiments on clidsum demonstrate that our mdialbart outperforms strong baseline models.