Why Does Zero-Shot Cross-Lingual Generation Fail? An Explanation and a Solution (2023)

Abstract: Zero-shot cross-lingual transfer is when a multilingual model is trained to perform a task in one language and then is applied to another language. Although the zero-shot cross-lingual transfer approach has achieved success in various classification tasks, its performance on natural language generation tasks falls short in quality and sometimes outputs an incorrect language. In our study, we show that the fine-tuning process learns language invariant representations, which is beneficial for classification tasks but harmful for generation tasks. Motivated by this, we propose a simple method to regularize the model from learning language invariant representations and a method to select model checkpoints without a development set in the target language, both resulting in better generation quality. Experiments on three semantically diverse generation tasks show that our method reduces the accidental translation problem by 68% and improves the ROUGE-L score by 1.5 on average.

## introduction language models (lms) pre-trained on multilingual corpora @xcite @xcite exhibit zero-shot cross-lingual transfer ability @xcite . given only annotated data in one language for a task, multilingual lms are able to perform this task in languages seen only during the pre-training stage. the cross-lingual transferability of multilingual lms reduces the need for annotated data in low-resource languages, which is valuable for building practical multilingual nlp systems. existing studies on cross-lingual transfer select tasks such as word alignment @xcite , pos tagging @xcite , dependency parsing and sentence classification @xcite to investigate cross-lingual transferability of multilingual lms @xcite , and few works focus on cross-lingual transfer in generation tasks @xcite . cross-lingual transfer approach in generation tasks are known to produce incoherent text @xcite , generate in a wrong language @xcite , and suffer from catastrophic forgetting @xcite . we show that the reason why zero-shot crosslingual transfer in text generation fails is because the fine-tuning process learns language invariant representations, which is beneficial for classification tasks, but detrimental to generation tasks. in our paper, we use the cosine similarity between parallel sentence representations in different languages to measure the cross-lingual representation similarity (xlrs). we use a range of tasks from classification to extractive question answering, then to abstractive generation to show that in the best performing model, the xlrs after finetuning decreases as we move from classification to generation. the fact that language invariant representations causes the degradation in generation tasks challenges the common belief that invariant representations generally enhance cross-lingual transfer on all downstream tasks @xcite @xcite . to the best of our knowledge, our work is the first to provide an analysis of how xlrs affects cross-lingual transfer in language generation tasks. motivated by our findings, we propose to use an auxiliary source language that implicitly regularizes the xlrs being too large and results in better generation performance over three complex natural prediction speak to your doctor, understand the dangers of alcohol consumption.. target 与您的医生交谈：改变您对戒酒的想法。 (speak to your doctor: change your thinking towards quitting alcohol) prediction review accounting books periodically. to sum up, our contributions are threefold: ## setup the consensus of the literature @xcite @xcite @xcite is that if a model can produce similar representations for parallel sentences, the model would be able to achieve good cross-lingual transfer performance. intuitively, if a model maps parallel sentences in english and french into nearly identical representations, and is able to predict the sentiment of the english sentence, it will also be able to predict the sentiment of the french sentence. we hypothesize that the fine-tuning process in-creases the similarity between sentence representations of different languages. we use the following setups and tasks to verify our hypothesis. ## learning dynamics of cross-lingual transfer we plot the average cosine similarity between representations of parallel sentences (xlrs) foot_2 for each training iteration in two classification tasks: pos tagging and paraphrase identification (paws-x) at figure 1 and figure 2 , respectively. in both tasks, the plot displays an increasing trend of xlrs between parallel sentences between english and all the other languages. notably, languages that have the same script have a higher similarity. our findings show that the fine-tuning process on classification tasks does make the sentence representations of different languages more similar. we then plotted the xlrs between representations of parallel sentences of a model when finetuned on wikilingua: a summarization dataset in figure 3 . the average similarity gradually increases as we progress further into the training iterations, confirming our hypothesis that fine-tuning on a single source language increases the xlrs between the source and other languages. based on our findings, we conjecture that the model jointly minimizes two metrics, resulting in cross-lingual transfer: and as a result, the cross-entropy loss between the predicted and ground-truth labels, given a context in the target language, is minimized, enabling cross-lingual transfer. ## unified view of tasks with the intuition of how the model does crosslingual transfer in classification tasks, we note that language generation is a classification on a large vocabulary set, rather than a small label set. thus, we point out that the reason why good performance with cross-lingual transfer on generation tasks is harder to achieve is actually caused by increasing xlrs. figure 4 illustrates our intuition. in classification tasks, the model needs to map parallel sentences to the same label. ideally, the model produces identical representations for parallel sentences, resulting in the highest possible xlrs of 1. this is why model cross-lingual transfers better with a high xlrs. however, in generation tasks, if we view it as classifying over the entire vocabulary set, we are mapping parallel sentences to different labels. in an extreme case when xlrs is 1, the model fails to identify the source language, resulting in the common problem of the model producing an incorrect language @xcite . we introduce the notion of label overlap, d, to measure the percentage of examples in a dataset where the model needs to map parallel sentences into the same label. we use c to denote the set of all discrete contexts and c s to denote the set of all discrete contexts in language s. in classification tasks, the model learns to predict the ground-truth label ŷ = l(c) over a set of candidate labels y for context @xmath0. similarly, in a simplified view of generation, the model learns to predict the next word v given context c. therefore, we can essentially view language generation as classification where the label set is the entire vocabulary. in both cases, given context c, the model learns a probability distribution p •|c . the only difference is between their classification label set cardinality. we define cross-lingual label overlap as an indicator of difficulty to cross-lingual transfer for a given task at §5.1. we then use a range of tasks: word-level classification (pos tagging §5.2) -sentence level classification (entailment classification §5.3) -span extraction (extractive generation §5.4) -summarization (abstractive generation §5.5) to show an increasing level of difficulty to perform cross-lingual transfer. ## text generation experiments now that we know xlrs is negatively correlated with cross-lingual transfer in text generation, since calculating xlrs during every iteration is computationally expensive, we wonder if we can regularize xlrs implicitly. motivated by using auxiliary source languages improves machine translation @xcite and few-shot cross-lingual transfer @xcite , we propose to use an additional source language to regularize xlrs. to verify our hypothesis, we plot the xlrs between english and french during training on two different sources (en, de)in the story completion task, compared to training only on one source (en) in figure 6 . we observe that when the model is only given one language as the source, the xlrs increases, whereas using two source languages allows the model to learn to control the xlrs from being too high, resulting in fewer accidental translations and better quality. to show that regularizing xlrs does result in better generation quality, we experiment with three semantically diverse generation tasks: summarization (wikilingua), title generation (tg), and story completion (sg). ## conclusion we show that multilingual lms transfer supervision from one language to another by increasing cross-lingual representation similarity (xlrs). such a learning process results in decent zero-shot cross-lingual transfer performance in classification tasks but is harmful to text generation performance . we demonstrate that regularizing xlrs improves text generation quality and use parallel sentences to guide model selection without annotated data in the target languages. we believe that this is valuable under a practical setting @xcite where we have access to parallel data between the source and target languages, but not task-specific data in the target language.