Zero-shot Cross-lingual Transfer is Under-specified Optimization (2022)

Abstract: Pretrained multilingual encoders enable zero-shot cross-lingual transfer, but often produce unreliable models that exhibit high performance variance on the target language. We postulate that this high variance results from zero-shot cross-lingual transfer solving an under-specified optimization problem. We show that any linear-interpolated model between the source language monolingual model and source + target bilingual model has equally low source language generalization error, yet the target language generalization error reduces smoothly and linearly as we move from the monolingual to bilingual model, suggesting that the model struggles to identify good solutions for both source and target languages using the source language alone. Additionally, we show that zero-shot solution lies in non-flat region of target language error generalization surface, causing the high variance.

## introduction pretrained multilingual encoders like multilingual bert (mbert; @xcite and xlm-roberta (xlm-r; @xcite facilitate zero-shot cross-lingual transfer @xcite ) -training the model on one language then using it on another language without additional task-specific training data. while the generalization performance on the source language has low variance, on the target language the variance is much higher with zero-shot cross-lingual transfer @xcite ), making it difficult to compare different models in the literature. similarly, pretrained monolingual encoders also have unstable performance during fine-tuning @xcite . why are these models so sensitive to the random seed? many theories have been offered: catas-code is available at https://github.com/ shijie-wu/crosslingual-nlp . we offer a new explanation for high variance in target language performance: the zero-shot crosslingual transfer optimization problem is underspecified. based on the well-established linear interpolation of 1-dimensional plot and contour plot @xcite , we empirically show that any linear-interpolated model between the monolingual source model and bilingual source and target model has equally low source language generation error. yet the target language generation error surprisingly reduces smoothly and linearly as we move from a monolingual model to a bilingual model. to the best of our knowledge, no other paper documents this finding. this result provides a new answer to our mystery: only a small subset of the solution space for the source language solves the target language on par with models with actual target language supervision; the optimization could not find such a solution with existing condition (without target language supervision), hence an under-specified optimization problem. if target language supervision were available, as it was in the counterfactual bilingual model, the optimization would find the smaller subset. by comparing both mbert and xlm-r, we find that the generalization error surface of xlm-r is flatter than mbert, contributing to its better performance compared to mbert. thus, zero-shot cross-lingual transfer has high variance, as the solution found by zero-shot cross-lingual transfer lies in the non-flat region of the target language generalization error surface. small turbulence on the parameter space would lead to big generalization error difference, hence the high variance. ## existing hypotheses (related work) prior studies have observed fine-tuning variance with pretrained encoder, and have offered various hypotheses to explain this behavior. catastrophic forgetting -when neural networks trained on one task forget that task after training on a second task @xcite -has been credited as the source of high variance in both monolingual fine-tuning @xcite and zero-shot crosslingual transfer @xcite . @xcite wonder why preserving cloze capability is important. however, in zero-shot cross-lingual transfer, deliberately preserving the multilingual cloze capability with regularization improves performance but does not eliminate the zero-shot transfer gap @xcite . small training data size often seems to have higher variance in performance @xcite , but @xcite found that when controlling the number of gradient updates, smaller data size has the similar variance as larger data size. in the pretraining-then-fine-tune paradigm, random seeds impact the initialization of task-specific layers and data ordering during fine-tuning. @xcite show development set performance has high variance with respect to seeds. additionally, adam optimizer without bias correction-an adam (kingma and ba, 2014) variant (inadvertently) introduced by the implementation of @xcite -has been identified as the source of high variance during monolingual fine-tuning @xcite . however, in zero-shot cross-lingual transfer, while different random seeds lead to high variance in target languages, the source language has much smaller variance in comparison even with standard adam (wu and dredze, 2020). beyond optimizers, @xcite attributes high variance to generalization issues: despite having similar training loss, different models exhibit vastly different development set performance. however, in zero-shot cross-lingual transfer, the development or test performance variance is much smaller on the source language compared ## under-specified optimization existing hypotheses do not explain the high variance of zero-shot cross-lingual transfer: much higher variance on generalization error of the target language compared to the source language. we propose a new explanation: zero-shot cross-lingual transfer is an under-specified optimization problem. foot_0 as in fig. 1 , optimizing a multilingual model for a specific task using only source language annotation allows choices of many good solutions in terms of generalization error. however, unbeknownst to the optimizer, these solutions have wildly different generalization errors on the target language. in fact, a small subset has similar low generalization error as models trained on target language. yet without the guidance of target data, the zero-shot cross-lingual optimization could not find this smaller subset. as we will show in §5, the solution found by zero-shot transfer lies in a non-flat region of target language generalization error, and small turbulence in the parameter space causes big difference in generalization error, causing its high variance. ## experiments we consider four tasks: natural language inference (xnli; @xcite , named entity recognition (ner; @xcite , pos tagging and dependency parsing @xcite . we evaluate xnli and pos tagging with accuracy (acc), ner with span-level f1, and parsing with labeled attachment score (las). we consider two encoders: base mbert and large xlm-r. for the task-specific layer, we use a linear classifier for xnli, ner, and pos tagging, and @xcite for dependency parsing. to avoid english-centric experiments, we consider two source languages: english and arabic. we choose 8 topologically diverse target languages: arabic 4 , german, spanish, french, hindi, russian, vietnamese, and chinese. we train the source language only and target language only monolingual model as well as a source-target bilingual model. 4 arabic is only used when english is the source language. we compute the linear interpolated models as described in §3.1 and test it on both the source and target language development set. we loop over {-0.5, -0.4, • • • , 1.5} for α, α 1 and α 2 . 5 we report the mean and variance of three runs by using different random seeds. we normalized both mean and variance of each interpolated model by the bilingual model performance, allowing us to aggregate across tasks and language pairs. details of fine-tuning can be found in app. a. ## results in fig. 2 , we observe that interpolations between the source monolingual and bilingual model have consistently similar source language performance. in contrast, surprisingly, the target language performance smoothly and linearly improves as the interpolated model moves from the zero-shot model to bilingual model. 6 the only exception is mbert, where the performance drops slightly around 0.1 and 0.9 locally. in contrast, xlm-r has a flatter slope and smoother interpolated models. fig. 3 further demonstrates this finding with a 2d linear interpolation. the generalization error surface of the target language of xlm-r is much flatter compared to mbert, perhaps the fundamental reason why xlm-r performs better than mbert in zero-shot transfer, similar to findings in cv models @xcite ). as we discuss in §3, these two findings support our hypothesis that zero-shot cross-lingual transfer is an underspecified optimization problem. as fig. 3 shows, the solution found by zero-shot transfer lies in a non-flat region of target language generalization error surface, causing the high variance of zeroshot transfer on the target language. in contrast, the same solution lies in a flat region of source language generalization error surface, causing the low variance on the source language. ## discussion we have presented evidence that zero-shot crosslingual transfer is an under-specified optimization problem, and the cause of high variance on target language but not the source language tasks during cross-lingual transfer. this finding holds across 4 tasks, 2 source languages and 8 target languages. training bigger encoders addresses this issue indirectly by producing encoders with flatter crosslingual generalization error surfaces. however, a more robust solution may be found by introducing constraints into the optimization problem. there are a few potential solutions. few-shot cross-lingual transfer is a potential way to further constrain the optimization problem. @xcite finds that it is important to first train on source language then fine-tune with the fewshot target language example. through the lens of our analysis, this finding is intuitive since finetuning with a small amount of target data provides a guidance (gradient direction) to narrow down the solution space, leading to a potentially better solution for the target language. the initial fine-tuning with the source data is also important since it provides a good starting point. additionally, @xcite observes that the choice of shots matters. this is expected as it significantly impacts the quality of the gradient direction. similarly, silver target data is a potential way to further constrain the optimization problem. while @xcite finds that jointly training with gold source data and silver target data benefits cross-lingual transfer, a pipeline fine-tuning approach like few-shot cross-lingual transfer is also worth exploring. unsupervised model selection like chen and ritter (2020) and optimization regularization like @xcite have been proposed in the literature to improve zero-shot cross-lingual transfer. through the lens of our analysis, both solutions attempt to constrain the optimization problem. as none of the existing techniques fully constrain the optimization, future work should study the combination of existing techniques and develop new techniques on top of it instead of studying one technique at a time.