S um T ra: A Differentiable Pipeline for Few-Shot Cross-Lingual Summarization (2024)

Abstract: Cross-lingual summarization (XLS) generates summaries in a language different from that of the input documents (e.g., English to Spanish), allowing speakers of the target language to gain a concise view of their content. In the present day, the predominant approach to this task is to take a performing, pretrained multilingual language model (LM) and fine-tune it for XLS on the language pairs of interest. However, the scarcity of fine-tuning samples makes this approach challenging in some cases. For this reason, in this paper we propose revisiting the summarize-and-translate pipeline, where the summarization and translation tasks are performed in a sequence. This approach allows reusing the many, publicly-available resources for monolingual summarization and translation, obtaining a very competitive zero-shot performance. In addition, the proposed pipeline is completely differentiable end-to-end, allowing it to take advantage of few-shot fine-tuning, where available. Experiments over two contemporary and widely adopted XLS datasets (CrossSum and WikiLingua) have shown the remarkable zero-shot performance of the proposed approach, and also its strong few-shot performance compared to an equivalent multilingual LM baseline, that the proposed approach has been able to outperform in many languages with only 10% of the fine-tuning samples.

## introduction cross-lingual summarization (xls) aims to take a document written in a given source language and generate a summary in a chosen target language, providing the speakers of the latter with the ability to concisely understand the content of documents written in foreign languages. however, xls is a challenging task due to the limited training data which are typically available. unlike in monolingual summarization, naturally-occurring crosslingual document-summary pairs are rare, and dedi-cated xls human annotation is demanding since it requires uncommon skills of the annotators @xcite . this has often led to the reuse of existing multilingual data with post-hoc alignments for cross-lingual use @xcite . given the constraints in dedicated training resources, most recent approaches have focused on employing existing multilingual lms @xcite @xcite , pretrained in the typical unsupervised manner over large corpora, and fine-tuning them with the limited xls resources available for the chosen language pairs @xcite . however, these multilingual models suffer from well-known limitations. on the one hand, the uneven pretraining of multilingual lms across languages often results in poor knowledge transfer to low-resource languages @xcite . on the other hand, the superposition of too many languages in a single model can result in a degradation of crosslingual performance in the downstream task (i.e., language interference) @xcite . in addition, it is not trivial to reuse the abundant, existing monolingual summarization data, since finetuning a multilingual lm with monolingual data often compromises its ability to generate text in a language different from the input's @xcite )-a problem known as "catastrophic forgetting" (van de @xcite . the above issues compound in the impossibility of achieving a satisfactory zero-shot and few-shot xls performance out of conventional multilingual lms. for this reason, this work revisits the summarizeand-translate approach to xls @xcite , with the main aim of fully leveraging the existing monolingual summarization resources (i.e., training data, pretrained models) to obtain a performing zero-shot xls pipeline. specifically, we propose combining 1) a monolingual summarizer trained with abundant resources in the source language with 2) a pretrained machine translation model that translates into the target language. if the quality of both models is high, such a pipeline should be able to achieve a significant zero-shot performance. yet, it can also suffer from model misalignment and error propagation. therefore, we modify the summarizer to output "soft" predictions, ensuring that the pipeline remains fully differentiable end-to-end @xcite @xcite . this allows fine-tuning it to improve the coupling of the models, alleviate error propagation, and obtain summaries that are closer to the ideal, joint summarization/translation of the xls task. for immediacy, we refer to the proposed pipeline as sumtra. in particular, in this paper we focus on the less explored english-to-many xls task (most work to date has focused on many-to-english @xcite @xcite or specific language pairs such as english-to-chinese @xcite @xcite . we believe that this is a valuable contribution as it provides access to summaries of the multitude of existing english documents for speakers of other languages around the world. to this aim, we have carried out experiments over two widely used xls datasets (crosssum @xcite and wikilingua @xcite ), with a range of language pairs spanning high-, medium-, and low-resource languages. the results show a strong quantitative performance for the zero-shot pipeline, and a competitive edge over comparable multilingual language model baselines with up to 1000-shot fine-tuning foot_0 . overall, our paper makes the following contributions: ## related work cross-lingual summarization (xls) has been an active research topic for a long time @xcite . pre-neural methods have often combined monolingual summarization and machine translation (mt) modules into pipeline approaches that summarize-and-translate @xcite , or translateand-summarize @xcite @xcite . while conceptually justifiable, these approaches inevitably suffered from error propagation between the modules, and, obviously, the architectural limitations of the models of the day @xcite . with the recent development of multilingual pretrained language models such as mbart @xcite and mt5 @xcite , there has been a surge in xls research that has focused on fine-tuning these models with xls datasets, and as a consequence has relegated pipeline methods to be regarded as mere baselines for comparison @xcite @xcite . however, the current approaches are not exempt from performance limitations at their turn, in particular when applied to low-resource languages foot_1 . to address them, @xcite has attempted to transfer knowledge from high-to low-resource languages by a multi-stage sampling algorithm that aptly up-samples the lowresource languages. other works have explored using language-specific adapter modules in various cross-lingual tasks @xcite to increase the linguistic capacity of the model at a parity of trainable parameters and alleviate language interference @xcite . @xcite have proposed using a combination of monolingual and cross-lingual summarization in an attempt to improve performance on low-resource languages. more recently, @xcite has proposed leveraging various large (>100b parameters) language models for zero-shot cross-lingual summarization. by contrast, in this paper we intentionally focus on the utilization of much smaller, modular, and trainable models in the zero-and fewshot scenario. ## sumtra the proposed sumtra model consists of the cascade of two language models: a monolingual summarization language model, followed by a machine translation language model, which we refer to as sum and tra for summarize and translate, respectively. let us denote the token sequence of the input document as @xmath0, . . . x n }, and the token predicted by the sum module at slot j as s j . we can then express the sequence of probability vectors output by the sum module over the vocabulary as {p 1 , ...p j ..., p m }, with: the probability vectors {p 1 , ...p j ..., p m } are then individually mixed with the embedding layer e of the tra module of size d × v (embedding × vocabulary) to obtain a sequence of expected embeddings, @xmath1...e j ...e m }, with: for fine-tuning the entire sumtra model, we use the standard negative log-likelihood: @xmath2, . . . y t-1 , e, θ, σ) (4) where with {y 1 , . . . y t } we denote the sequence of ground-truth tokens in the target language, and with p(y) the probabilities output by the translator. however, fine-tuning the sum module with only the standard negative log-likelihood of the groundtruth summary in the target language allows for too many degrees of freedom in the generation of the intermediate english summary, and can lead to inaccurate summaries with respect to the source document. for this reason, we add an auxiliary training objective that encourages the predicted summary to adhere to the target more closely. to this aim, we first back-translate the ground-truth sequence, y, into the language of the summarizer (i.e., english) using a reverse tra module, and then use it as auxiliary training objective for the summarizer: the training objectives in equations 4 and 5, are eventually combined in a simple convex combination: ## results and analysis tables 1 and 2 present the results of the proposed approach and comparative baselines over the chosen language pairs, grouped into high-, medium-, and low-resource languages, for the crosssum and wikilingua datasets, respectively. sumtra vs. mbart-50. in both tables, we compare the proposed sumtra model with both mbart-50 and mbart-50-mono (the version with an initial english summarization training), and in both zero-and few-shot configurations (50-1000 examples). the results show that the english training can be beneficial for improving the average zero-and few-shot performance of mbart-50 @xcite ; however, the results are not consistent across languages, even for those that are linguistically similar (e.g., spanish and french). sumtra comparatively displays much stronger average zero-and few-shot performance up to and including 1000 shots, showing the usefulness of the proposed approach. for instance, sumtra (0-shot) outperforms both mbart-50 variants with 1000 shots on average over the cross-sum languages. in a similar fashion, at a parity of fine-tuning samples (1000-shots), the most performant sumtra model outperforms mbart-50 by +1.28 bertscore pp on average over the wikilingua languages. sumtra vs. ## conclusion in this paper, we have proposed sumtra, an xls model that revisits the traditional summarize-andtranslate approach into a more contemporary endto-end differentiable pipeline. given that genuine xls annotation is demanding, the main aim of the proposed model is to provide a competitive zeroand few-shot performance. in the paper, we have evaluated the proposed approach over two mainstream xls datasets and against a set of performing baselines, giving evidence to the competitive performance of the proposed approach. in particular, sumtra's zeroshot performance has proved very strong, and its few-shot performance has been remarkable for a majority of the languages. through various sensitivity, ablation, and qualitative analyses we have shown that the proposed model benefits from the possibility to separately train its component modules, and that its memory and inference time overheads compared to the base model are both manageable. in the future, we aim to test model configurations with different base language models (e.g., pisces) for the summarization and translation modules, and explore alternative fine-tuning strategies such as adversarial training and reinforcement learning.