Cross-Lingual Summarization with Pseudo-Label Regularization (2024)

Abstract: Cross-Lingual Summarization (XLS) aims to summarize a document in the source language into a condensed version in the target language, effectively removing language barriers for non-native readers. Previous approaches, however, have the same limitation that only a single reference (gold summary) is exploited during model training, making the base model exposed to an underrepresented hypothesis space since the actual number of possible hypotheses is exponentially large. To alleviate this problem, we present a study adopting pseudo-labels in regularizing standard cross-lingual summarization training. We investigate several components leading to the gains in regularization training with verified experiments involving 8 diverse languages from different families. Conclusively, we show that pseudo-labeling is a simple and effective approach that significantly improves over standard gold reference training in XLS.

## introduction with the rise of massive information @xcite , summarization plays an essential part in absorbing and processing the emerging events in a timely and efficient manner @xcite . given the existing large number of natural languages @xcite , the needs for providing information access to widespread groups of audience across nations have necessitated the construction of tasks beyond the monolingual setting @xcite . to facilitate this development, cross-lingual summarization (xls) was introduced where an article in a source language can be summarized to a condensed version in a different target language, effectively conveying key information to non-native readers @xcite . among modeling solutions for xls, a straightforward approach was to adopt pipeline-based systems comprising of translation and summarization networks @xcite . it is, however, not particularly appealing for several reasons including error propagation (takase and okazaki, 2022), loss of context and structure @xcite , and the incurred extra computations along with reduced inference speed. recent research have moved on to adopt end-to-end approaches where a single network executes the whole task @xcite . to empower the cross-lingual performance, existing methods either scale up the training process @xcite or bootstrap learned representations through auxiliary objectives and features @xcite @xcite . despite promising improvements, these methods have the same limitation in that for each training sample, the summarization network only gets exposed to a single gold reference. this mistakenly causes the network to learn skewed, deterministic one-point @xcite and highly uncalibrated summary distributions @xcite ) that easily suffer from error accumulations such as exposure bias @xcite . seeking prospective remedies, in this work, we study the training of end-to-end xls networks in the presence of pseudo-labels i.e. pseudo-references with probability labels obtained from teacher models. in particular, we analyze several important components contributing to the performance of models regularized with pseudo-labels and present these findings as reference practices for future works. to inspect the effectiveness of this training recipe, we conduct empirical experiments spanning 8 typologically diverse languages coupled with comparisons against existing baselines. in summary, our contributions can be listed as follows: early works, due to the lack of parallel supervised corpora, often resort to pipeline methods comprising of separate translation and summarization networks @xcite . @xcite first introduced a large-scale parallel corpus to experiment with end-to-end training where they also proposed a multi-task framework with improved performance. subsequent works experimented with adversarial training @xcite , contrastive learning @xcite , variational inference @xcite , knowledge distillation (nguyen and luu, 2022) and pre-training @xcite @xcite . these works, however, bear the same limitation in that they do not explore alternative hypotheses apart from the gold summary in each training sample which potentially leads to models' underperformance @xcite since they lacked exposure to the summary distribution which includes an exponentially large number of hypotheses. in contrast, we target the setting of training neural crosslingual summarizers with additional supervision from pseudo-labels that are formed from pseudoreferences with probability labels from teacher models. pseudo-labeling. there have been ongoing research on adopting pseudo-labels in sequenceto-sequence training @xcite . @xcite enriched the one-hot annotations with smoothed self-guidance under noised perturbations for summarization on english data. shleifer and rush, 2020 further examined replacing the ground truth target with the teacher's summary. @xcite additionally integrated selective mechanisms in forming training data points for translation tasks. duan et al., 2019 relied on pseudo-sources and pseudo-labels to overcome scarcity in parallel samples for cross-lingual sentence summarization. li et al., 2023 utilised soft-labels for enforcing crosslingual consistency learning. nevertheless, these works either do not consider multiple references during training @xcite or strictly focus on monolingual generation @xcite and machine translation @xcite . this leaves a research gap in cross-lingual summarization where the use of multiple training references remains neglected -which our work aims to fill in. ## empirical analysis in this section, we investigate the choice of the teacher models for obtaining the probability labels as well as the forming of pseudo-references, both of which make up the pseudo-labels for training. additionally, we examine their effects on the xls model's performance and further see how they fare against the gold labels. particularly, we focus on the following research questions: task setup. for experimental purposes, we use the wikilingua foot_1 dataset (ladhak et al., 2020 ) with a focus on 8 languages: english(en), vietnamese(vi), japanese(ja), chinese(zh), arabic(ar), korean(ko), russian(ru) and turkish(tr). these languages come from different families ( for each training run, we use the adamw @xcite optimizer with a learning rate of 1e -5 and a linear decay scheduler for a maximum of 300 000 steps. we use a batch size of 4 for standard training and 2 when adopting pseudo-labels. for evaluation, we primarily use the rouge metrics @xcite . in the multilingual setting, previous works either utilize the multi-lingual rouge toolkit foot_4 for language-specific processing @xcite @xcite or the sentencepiece model for language-agnostic tokenization @xcite @xcite , which we hereinafter refer to as m-rouge and sp-rouge, respectively. upon measuring these metrics with human judgements from the multi-summeval dataset @xcite , we find that sp-rouge generally attains better correlation foot_5 . we therefore report results obtained with the sp-rouge metric 7 . for presentation's sake, we focus on the rouge-l variant and include the remaining results in the appendix. experiments were implemented with the pytorch @xcite and transformer @xcite frameworks, and executed on an a100 gpu. ## benchmarking in this section, we study the effectiveness of pseudo-label regularized models on three settings: fully supervised learning, few-shot learning and parameter-efficient 10 see appendix d fine-tuning. we follow lessons derived from the previous section for the teacher models' choice as well as the construction of pseudo-references. in particular, we use an ensemble of 3 monolingual models for teacher labeling. for pseudo-reference, we use beam decoding with a beam size of 32 from three monolingual models for each language to construct the pseudo-reference pool with a total size of 96. unless explicitly mentioned otherwise, we use the mbart-50 language model as the base architecture, similar to the previous section. for further implementation details, we refer readers to appendix b. ## conclusion in this paper, we study the effectiveness of pseudolabel regularization in standard neural cross-lingual summarization training. we conduct empirical experiments involving 8 diverse languages from different families and study the different components affecting the end performance of regularized models. we further validate the regularization's usage in three distinct learning settings. the results show that pseudo-labeling is a simple but effective companion complementary to the standard training procedure. we hope our study will prove useful for future practitioners working on this task.