MSAMS um: Towards Benchmarking Multi-lingual Dialogue Summarization (2022)

Abstract: Dialogue summarization helps users capture salient information from various types of dialogues has received much attention recently. However, current works mainly focus on English dialogue summarization, leaving other languages less well explored. Therefore, we present a multi-lingual dialogue summarization dataset, namely MSAMSum, which covers dialogue-summary pairs in six languages. Specifically, we derive MSAMSum from the standard SAMSum using sophisticated translation techniques and further employ two methods to ensure the integral translation quality and summary factual consistency. Given the proposed MSAMum, we systematically set up five multi-lingual settings for this task, including a novel mix-lingual dialogue summarization setting. To illustrate the utility of our dataset, we benchmark various experiments with pre-trained models under different settings and report results in both supervised and zero-shot manners. We also discuss some future works towards this task to motivate future researches.

## introduction recent years have witnessed increasing interest in dialogue summarization @xcite . it aims to distill the most important information from various types of dialogues, which can alleviate the problem of communication data overload. towards this research direction, various datasets have been proposed to promote this task. the ami @xcite and icsi @xcite datasets provide the initial opportunity for meeting summarization. with the advent of data-hungry neural models and pre-trained language models, @xcite come up with the first high quality large-scale dialogue summarization dataset, namely samsum, which resurges this task. then, various datasets are proposed to meet different needs and scenarios @xcite @xcite @xcite @xcite . despite the encouraging progresses achieved, current works overwhelmingly focused on english. meanwhile, with the help of instantaneous translation systems foot_0 , a dialogue involving multinational participants becomes more and more common and frequent. therefore, it is valuable to provide them with dialogue summaries in a preferred language. to this end, we propose a multi-lingual dialogue summarization task. the practical benefits of this task are twofold: it not only provides rapid access to the salient content, but also enables the dissemination of relevant content across participants of other languages. intuitively, to achieve this goal, we need to answer two key questions, one is where do we get data resources for this multi-lingual research? the other is how do we perform various multi-lingual settings? for the first question, we seek for potential available resources that can support our multi-lingual research. although creating english datasets has proven feasible, the need for dialogues and summary-written experts in different languages makes the collection of multi-lingual datasets highly costing or even intractable. to mitigate this challenge, we devote our efforts to constructing the multi-lingual dataset via sophisticated translation techniques following @xcite . firstly, we select samsum @xcite as our source english dataset because of its large scale and wide domain coverage. then, we translate it into five other official languages of the united nations via high-performance translation api, including chinese, french, arabic, russian and spanish. furthermore, we employ two methods: round-trip translation and textual entailment to filter out lowquality translations and ensure the factual consistency at both the dialogue-level and summary-level. finally, we obtain our msamsum dataset as the data resource for this multi-lingual research. for the second question, given the wellconstructed msamsum dataset, we set up various settings for our multi-lingual dialogue summarization task, including one-to-one, many-to-one, one-to-many and many-to-many. the one-to-one setting can be further divided into mono-lingual and cross-lingual settings. to further boost the research on multi-lingual dialogue summarization, we creatively propose one new setting, namely mix-to-many, which takes a mixlingual dialogue as input and produce summaries in different languages. this setting is in line with the real world scenario that multinational participants can use their mother tongue to communicate with each other by means of instantaneous translation systems (depicted in figure 1 ). to sum up, we set up five settings for the research on the whole scene of multi-lingual dialogue summarization. to illustrate the utility of our msamsum, we conduct extensive experiments under five multilingual settings based on the current multi-lingual pre-trained model mbart-50 @xcite , and evaluate it in both supervised and zero-shot manners. the results reveal the feasibility of multilingual dialogue summarization task. the case study also shows that the multi-lingual model is able to produce fluent and factual consistency summaries in different languages. we further conclude several future works to prompt future researches. ## the msamsum dataset in this section, we introduce our msamsum dataset, including (1) why we choose samsum dataset? (2) how we translate the original sam-sum dataset? (3) how we control the translation quality? and (4) statistics for the newly created msamsum dataset. the whole dataset construction process is shown in figure 2 . ## multi-lingual settings in this section, we introduce various multi-lingual dialogue summarization settings, including a newly proposed mix-to-many setting. all settings are depicted in figure 3 . ## experiments in this section, we first introduce our model mbart-50. after, we describe the evaluation metrics. finally, we show the implementation details. ## results in this section, we describe experimental results and show our analyses for different settings. ## conclusion and future work in this paper, we innovatively explore the multilingual dialogue summarization task. to this end, we carefully create msamsum as our testbed, which covers dialogue-summary pairs in six languages, including english, chinese, russian, french, arabic and spanish. furthermore, we systematically set up five multi-lingual settings to benchmark extensive experiments. our results indicate that various models can achieve impressive performance based on pre-trained models. besides, the newly proposed mix-to-many setting also shows its effectiveness in low-resource scenarios. in the future, we think several concerns need to be addressed for this task. firstly, multi-lingual models tend to underperform mono-lingual models; secondly, low-resource languages tend to perform poorly; thirdly, the difficulty of aligning finegrained information in different languages. future works should pay particular attention to these concerns to facilitate this multi-lingual dialogue summarization research direction.