Neural Label Search for Zero-Shot Multi-Lingual Extractive Summarization (2022)

Abstract: In zero-shot multilingual extractive text summarization, a model is typically trained on English summarization dataset and then applied on summarization datasets of other languages. Given English gold summaries and documents, sentence-level labels for extractive summarization are usually generated using heuristics. However, these monolingual labels created on English datasets may not be optimal on datasets of other languages, for that there is the syntactic or semantic discrepancy between different languages. In this way, it is possible to translate the English dataset to other languages and obtain different sets of labels again using heuristics. To fully leverage the information of these different sets of labels, we propose NLSSum (Neural Label Search for Summarization), which jointly learns hierarchical weights for these different sets of labels together with our summarization model. We conduct multilingual zero-shot summarization experiments on MLSUM and WikiLingua datasets, and we achieve state-of-the-art results using both human and automatic evaluations across these two datasets.

## introduction the zero-shot multilingual tasks, which aim to transfer models learned on a high-resource language (e.g., english) to a relatively low-resource language (e.g., turkish) without further training, are challenging @xcite . recently, large pre-trained multilingual transformers such as m-bert @xcite , xlm @xcite , and xlm-r @xcite have shown remarkable performance on zeroshot multilingual natural language understanding tasks. during pre-training, these transformer models project representations of different languages * work done during the first author's internship at microsoft research asia. ## related work there has been a surge of research on multilingual pretrained models, such as multilingual bert @xcite , xlm @xcite and xlm-roberta @xcite . for multilingual summarization, the summarizethen-translate and translate-then-summarize are widely used approaches in prior studies @xcite . there is another effective multi-lingual data augmentation, a method that replaces a segment of the input text with its translation in another language @xcite . on the other hand, large-scale multilingual summarization datasets have been introduced @xcite , which enable new research directions for the multilingual summarization. nikolov and hahnloser (2020) applies an alignment approach to collect large-scale parallel resources for lowresource domains and languages. in this paper, we aim to advance the multilingual zero-shot transferability, by training extractive summarization on english and inferring on other languages. ## conclusion we first study the monolingual label bias, that when translate the (document, summary) from english into other language, the re-converted labels will change along with the transformation of textual representation. then we propose nlssum to improve the performance of multilingual zero-shot extractive summarization, by introducing multilingual labels. finally, the summarization model is trained on english with the weighted multilingual labels and achieves great improvement on other languages.