Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation (2022)

Abstract: In this paper, we explore the challenging problem of performing a generative task in a target language when labeled data is only available in English, using summarization as a case study. We assume a strict setting with no access to parallel data or machine translation and find that common transfer learning approaches struggle in this setting, as a generative multilingual model fine-tuned purely on English catastrophically forgets how to generate non-English. Given the recent rise of parameter-efficient adaptation techniques, we conduct the first investigation into how one such method, prompt tuning (Lester et al., 2021), can overcome catastrophic forgetting to enable zero-shot cross-lingual generation. Our experiments show that parameter-efficient prompt tuning provides gains over standard fine-tuning when transferring between less-related languages, e.g., from English to Thai. However, a significant gap still remains between these methods and fully-supervised baselines. To improve cross-lingual transfer further, we explore several approaches, including: (1) mixing in unlabeled multilingual data, and (2) explicitly factoring prompts into recombinable language and task components. Our approaches can provide further quality gains, suggesting that robust zero-shot cross-lingual generation is within reach.

## introduction cross-lingual language understanding is an important area of ongoing research @xcite @xcite . with vastly differing amounts of data (both labeled and unlabeled) available across languages, there is significant value to developing techniques that can transfer knowledge from higher-resource languages to improve performance in lower-resource languages. zero-shot cross-lingual benchmarks push on the work done as a student researcher at google brain. ## challenge of zero-shot cross-lingual generation much recent progress in multilingual nlp has been driven by zero-shot cross-lingual benchmarks that require a model to perform classification @xcite , extractive qa @xcite @xcite , or sequence labeling @xcite . 1 here, we are interested in a more challenging task of zero-shot cross-lingual generation (xgen) where a model is trained on a generative task in one language (typically english), and then asked to perform the equivalent task in another language during inference. we construct a novel zeroshot cross-lingual summarization task and show that state-of-the-art text-to-text models adapted using modeltuning and prompttuning techniques are not able to successfully perform our task. our analysis reveals that both techniques suffer from catastrophic forgetting, causing them to often generate text in the wrong language. ## mitigating catastrophic forgetting we have seen that increasing model scale and decreasing tunable parameter capacity are both effective in improving xgen performance. can we obtain further gains by devising methods that explicitly tackle catastrophic forgetting? here, we investigate two approaches: mixing unlabeled training data with english supervised data, and factorizing the learned prompts into composable language and task modules. we show that both methods can provide substantially better results when there is severe catastrophic forgetting. below, we describe each method and analyze our findings in detail. ## qualitative analysis to better understand qualitative differences between the solutions reached by modeltuning and prompttuning, two authors who were native speakers of vietnamese and hindi inspected 50 predictions of each method at the xxl model size. for both languages, we observed that the mod-eltuning predictions were much more likely to include "code-switching", alternating between english and the target language, sometimes several times within a single sentence, as seen in for vietnamese, prompttuning achieved higher sp-rouge than modeltuning (38.0 vs. 34.0), with low variance in both cases (std: ≤ 0.5). on inspection, we found that most prompttuning predictions were entirely in vietnamese, whereas modeltun-ing predictions typically contained at least some english. the prompttuning summaries tended to be shorter, but were often judged to be as good or better than the ground truth summaries. the modeltuning summaries tended to be a bit longer. if mentally translating any english words back to vietnamese, the quality was judged to be similar to the prompt tuning summaries, suggesting that the lower sp-rouge score is primarily due to the presence of intervening english. previous work has also explored intermediate adaptation of pre-trained models, which has been shown to be effective for modeltuning @xcite @xcite and prompttuning @xcite . @xcite apply intermediate adaptation in the multilingual domain, but use english in the adaption instead of the target language. maurya et al. (2021) use a cross-lingual intermediate task. unlike our task, theirs is designed to closely match the downstream task. several works use intermediate adaptation to create a model that is better in the zero-or few-shot settings @xcite @xcite , but these target generalization to new tasks, whereas we focus on generalizing to new languages within one task. many parameter-efficient adaption methods exist @xcite @xcite @xcite and some have shown strong performance under domain shift @xcite . we chose prompttuning due to its simplicity and the localization of parametersmaking the implementation of factorized prompts easy. @xcite , @xcite , and @xcite for detailed discussion of the differences between these methods. other work explores cross-lingual transfer learning with parameter-efficient methods. zhao and schütze (2021) find that soft prompts can effectively be used in cross-lingual settings, but their work is constrained to classification. @xcite use adapters rather than prompts and leverage parameter-efficient learning to create separate language and task understanding modules that can be combined at inference time. there has been recent interest in cross-lingual generation. @xcite and @xcite evaluate their methods using cross-lingual generation, including summarization as we do. however, @xcite use parallel data during pre-training to "align" representations across languages during pre-training while our approach does not. ## conclusion in this work, we explored how different adaptation methods fare on the challenging "xgen" task of zero-shot cross-lingual summarization. while many methods struggled with catastrophic forgetting (outputting english rather than the target language), we observed two factors helped to mitigate this problem: (1) increasing model scale, and (2) decreasing the number of parameters tuned during adaptation. when all of a model's weights are tuned on english (modeltuning), forgetting is quick and severe. by contrast, limiting the tunable parameters to a smaller soft prompt (prompttun-ing) helps to combat forgetting, though prompt size is an important variable to control. to further close the gap with supervised methods, we explored two adaptation techniques-one entirely novel, and one that has been used before, but not in combination with parameter-efficient methods like prompttuning. we find that mixing in unsupervised multilingual data is always helpful. our novel approach, "factorized prompts", is helpful at smaller model sizes, but has no benefit at larger sizes. we hope that future work will continue to explore xgen tasks including wikilingua-0, and develop stronger zero-shot adaptation techniques to allow multilingual models to reliably generate coherent text in any target language.