Language-Independent Representations Improve Zero-Shot Summarization (2024)

Abstract: Finetuning pretrained models on downstream generation tasks often leads to catastrophic forgetting in zero-shot conditions. In this work, we focus on summarization and tackle the problem through the lens of language-independent representations. After training on monolingual summarization, we perform zero-shot transfer to new languages or language pairs. We first show naively finetuned models are highly language-specific in both output behavior and internal representations, resulting in poor zero-shot performance. Next, we propose query-key (QK) finetuning to decouple task-specific knowledge from the pretrained language generation abilities. Then, after showing downsides of the standard adversarial language classifier, we propose a balanced variant that more directly enforces language-agnostic representations. Moreover, our qualitative analyses show removing source language identity correlates to zero-shot summarization performance. Our code is openly available.

## introduction pretrained multilingual models @xcite @xcite have been established as promising sources of transfer learning, where task-specific finetuning benefits from the general knowledge learned on diverse unsupervised data. however, due to data or computational constraints, the task-specific data often only cover a limited subset of the languages in pretraining. therefore, during finetuning it is crucial to retain the knowledge of the pretrained model and to enable zero-shot transfer, i.e., performing the task on more languages covered by the pretrained model. while zero-shot crosslingual transfer has shown very promising results on sequence classification or labeling problems @xcite finetune with summarization data conneau and lample, 2019; @xcite , it remains challenging for generation tasks @xcite @xcite including summarization and translation. a main obstacle is catastrophic forgetting @xcite , where languages supported by the pretrained model but not covered in the finetuning data are forgotten. in this work, we use summarization as a testbed for various types of zero-shot generation. as shown in figure 1 , given a pretrained model and intralingual summarization training data in some languages (@xmath0, @xmath1, we aim for zero-shot intralingual and crosslingual summarization on new languages (@xmath2, @xmath3. to alleviate catastrophic forgetting, one line of work trains on additional unsupervised data @xcite @xcite . besides the computational overhead, this approach raises a theoretical question: as the pretrained language model has already learned extensively on unsupervised data, is it necessary to re-learn language modeling in task-specific finetuning? we therefore explore a more challenging case of only using paired summarization data without relying on any unsupervised data. we identify two challenges when generalizing summarization abilities to new languages. first, decoupling the task-specific knowledge from the train on en train on en+es+ru es-es ru-ru gu-gu gu-gu es-en ru-en es-ru tr-en 0.2 2.3 13.4 99.6 0.0 0.0 0.0 1. 3 for crosslingual zero-shot settings, it is also crucial to decouple language from content, i.e., creating language-agnostic representations. this has been shown to facilitate zero-shot crosslingual generation in general @xcite @xcite . there a prominent approach is adversarial training @xcite , where the model is trained to deceive a language classifier. we show the existing formulation fails to fully achieve language-agnostic representations, and improves it by explicitly incentivizing the model to deceive the classifier into a uniform class distribution. ## current models are highly language-specific we first show that naive finetuning makes the models highly language-specific in their output behavior and internal representations. ## conclusion in this work, we proposed two methods: 1) qk finetuning and 2) balanced adversarial language classifier to improve intralingual and crosslingual zero-shot summarization. we presented evidence that language-independent representations facilitate zero-shot summarization, in both intralingual and crosslingual forms. we are curious to see the applicability of our methods to other generation tasks. we are also curious about additional qualitative comparisons of language-specific and -independent representations. in the current study, we only used probing analyses to assess language-specific versus languageindependent representations. one way to supplement these analyses is to directly analyze the model hidden representations, e.g., compare the similarity between model hidden representations of different languages before and after applying the proposed approaches. this could for instance be achieved by singular vector canonical correlation (svcca) @xcite , which has been used to analyze multilingual representations for translation @xcite @xcite .