D i TTO : A Feature Representation Imitation Approach for Improving Cross-Lingual Transfer (2023)

Abstract: Zero-shot cross-lingual transfer is promising, however has been shown to be sub-optimal, with inferior transfer performance across low-resource languages. In this work, we envision languages as domains for improving zero-shot transfer by jointly reducing the feature incongruity between the source and the target language and increasing the generalization capabilities of pre-trained multilingual transformers. We show that our approach, DiTTO, significantly outperforms the standard zero-shot fine-tuning method on multiple datasets across all languages using solely unlabeled instances in the target language. Empirical results show that jointly reducing feature incongruity for multiple target languages is vital for successful cross-lingual transfer. Moreover, our model enables better cross-lingual transfer than standard fine-tuning methods, even in the few-shot setting.

## introduction due to the emergence of pre-trained massively multilingual transformers (mmts) such as mbert @xcite , xlm-r @xcite and mt5 @xcite , zero-shot cross-lingual transfer @xcite @xcite @xcite has received significant attention in the nlp community. this approach originated due to the skew in resource distribution in languages @xcite , with most languages of the world having a scarcity of labeled data. zeroshot transfer involves fine-tuning the mmt with task-specific data in one or more source languages, followed by evaluation on target languages whose labeled instances are not used during fine-tuning. accurate zero-shot transfer is crucially important for mmts to be useful for low-resource languages. the performance of mmts drops in the following two cases -when the source and target languages exhibit dissimilar typological features, or when the size of pre-training data in the target language is limited @xcite . two common techniques to improve zero-shot performance include few-shot cross-lingual transfer @xcite and the translate-train approach @xcite . several studies have been conducted comparing these approaches, of which @xcite concludes that if the cost of machine translation is greater than zero, the optimal and lowest-cost performance is achieved with at least some manually labeled data (i.e. the few-shot method). since annotating data is expensive for many languages @xcite @xcite , we investigate improving cross-lingual zero-shot transfer using only unlabelled data in this paper. zero-shot cross-lingual transfer has been identified as an under-specified optimization problem @xcite . a majority of the solutions reports a high performance on the source language but fluctuating performance on target languages. @xcite use linear interpolation to prove that it is possible to obtain a subset of solutions which have optimal performance on both source and target languages. furthermore, they also conclude that current optimization techniques cannot converge to this smaller subset of optimal solutions without the availability of labeled target language data. @xcite and @xcite have observed similar behavior in the zero-shot setup and hypothesize that sub-optimal zero-shot performance may be due to the degradation of generalizable representations of mmts during the finetuning stage. this leads to the model trained on the source language not being able to generalize well to the target languages. mmts have also been shown to be over-parameterized @xcite @xcite , which leads to memorizing the training data (source language) and achieving poor generalization during cross-lingual transfer. similar to @xcite , in our experiments, we also observe that once mmts are fine-tuned on source languages, there is an incongruity between the features of the source and target languages, as shown in figure 2 . we speculate that the mismatch in the feature representation space causes problems in generalization. we also find that this mismatch strongly correlates with zeroshot performance as shown in figure 1 . furthermore, we hypothesize that this instability can be reduced either by finding solutions that can generalize well or learning to match the feature representations. solutions @xcite ) that have been used for improving generalization in other tasks can be considered, so that the model reaches to a better local minima. sharpness-aware optimization (sam) @xcite is one such technique that has been used to improve the generalization of language models @xcite and vision transformers @xcite by smoothing the loss landscape for various adversarial tasks. sam is used to generalize across domains, however, by treating languages as separate domains, we can apply sam for generalizing across languages. while sam looks promising, our experiments (cf. 7.3) showed that it does not guarantee optimal generalization at all times. we need to further reduce the incongruency between language features by aligning target language features to mimic the features of the source language. we propose ditto for improving cross-lingual transfer by source language directed adversarial transition of target language using sharpness aware optimization. the key contributions of this work are: 1) exhibiting the limitations of standard fine-tuning by unveiling the feature incongruity between source and target languages. 2) ditto enhances crosslingual transfer by joint feature transformation of the multiple target languages to mimic the source. 3) ditto makes cross-lingual transfer cost-effective and efficient for distant (typologically different languages), resource-lean and unseen (not present in the pre-training data) languages. 4)ditto exhibits superior performance compared to augmenting the training data for either the source or the target language. a few papers (dziugaite and roy, 2017; @xcite have explored the connection between the flatness of minima and generalization gaps, showing flatter minima leads to better generalization. recently, sam has been proposed to find a smoother minima by minimizing the loss value and its sharpness. sam has been shown to improve the generalization capabilities of vision transformers @xcite . recently, @xcite employed sam in language models such as gpt-3 @xcite and t5 @xcite , showing significant improvements in generalization in english. in this work, we use sam to improve the generalization across other languages. another line of work @xcite hypothesizes that inferior transfer is due to forgetting and degradation of feature representation from pre-trained mmts when they are fine-tuned on the source language data. they propose to preserve the pre-trained features to improve the generalization using regularization and continual learning. unsupervised domain adaptation (uda): various studies have been proposed to reduce the domain shift to perform uda by minimizing discrepancy distances such as maximum mean discrepancy (mmd) @xcite and correlation alignment distance @xcite . adversarial-based feature alignment methods @xcite @xcite have been one of the popular uda methods where the domain discrepancy between the domains is reduced using an adversarial objective. in this work, we use domain-adversarial neural networks (dann) @xcite for performing adversarial adaptation of languages. with n examples. we use the provided mmt m as the encoder and fine-tune it along with the task-specific classifier c by minimizing the crossentropy loss: sharpness-aware minimization (sam): sam seeks to find the parameter w such that even its neighborhood has seemingly similar low training loss l train with minimal variation by optimizing the following objective: ## limitations of zero-shot learning inconsistent cross-lingual transfer: we have reported the average difference (δ) in the zero-shot performance between the target and the source language in we measure centered kernel alignment (cka) @xcite between the source and the target language feature representations to quantify the incongruity. in figure 1 . we have plotted the cka similarity with the zero-shot performance across all the languages. the plot suggests that there is a strong correlation between cka and zeroshot performance, with pearson and spearman correlation coefficients as 0.98 and 0.96, respectively establishing our hypothesis. we have a set of target languages t with each target language t having dataset u @xmath0. in dann, there is one target domain, whereas in our case we have a set of target languages t and we hypothesize and empirically show that performing adaptation for each language separately may cause degradation in other target languages, as seen in ditto consists of an mmt m for encoding the features, a task-specific classifier c and language discriminators d @xmath1. we train these modules using two losses: task-specific classification loss l c , defined in the equation (1) and language discrimination loss l l for distinguishing the target and source language. as we have |t| discriminators, we randomly sample a target language t from a prior distribution p(t) at each training step and train the discriminator {d l t } to accurately distinguish target t and source language using the following loss: we maximize the above loss l l (m, d l t ) for confusing the language discriminator d l t to transform the target features towards the source language. in our initial experiments (reported in ## results in this section, we describe the results of several experiments to analyze the ditto method and compare its performance with the baseline in the zeroshot setting. in order to justify the robustness of our method, we conduct experiments with the varying amount of source language data. in our experiments, en is the default source language and we categorize target languages as follows: 1. distant: languages that are typologically dissimilar to the source language 2. low-resource: languages that have scarcity of data for pre-training 3. unseen: languages that were not included in the pre-training data of mmt. furthermore, we compare the techniques in the few-shot setting with few labeled examples in target languages. then, we perform a thorough ablation study and analyze various variants of ditto . finally, we show evidence in the form of congruity between the source and target language feature representations and t-sne visualization in support of our hypothesis. ## discussion and conclusion in this work, we propose a novel method to improve the cross-lingual transfer capability of pretrained mmts. we find that zero-shot performance is correlated with incongruency between the features of source and target languages. experiments show that our proposed method ditto outperforms the standard fine-tuning approach across multiple setups. in general, the gains from our method are higher on the models (as in mbert) with less cross-lingual transfer. amnli consists only of languages that were not present in the pretrained mmts leading to similar transfer performance to the baseline method. ditto improves cross-lingual transfer using the pre-training features, hence the gains from ditto are similar on both mbert and xlm-r. due to a similar reason, the relative gains for unseen languages do not follow the trend observed on seen languages, where the gains are higher for languages with the lower cross-lingual transfer. we find higher relative gains on unseen and low-resource languages, followed by distant languages. we also notice that the crosslingual transfer improves with the amount of source language data s for seen languages. in contrast, for unseen languages, improvements are limited. due to this, the gains from ditto start to decrease for high values of s for seen languages but remain significant for unseen languages. our method provides similar gains using only unlabeled data compared to the fine-tuned baseline model (using 500 instances for each target language). our ablation study shows that both la and sam are essential components of ditto, with la being the primary contributor to the gains. experiments show that single language adaptation improves on that corresponding target language but may regress on other languages as the feature may remain incongruent to the source. however, ditto that adapts to multiple target languages performs best. ditto tries to exploit the pretraining knowledge for improving the cross-lingual transfer, however few promising works such as adaptors @xcite have been proposed to improve the pre-training features for low-resource and unseen languages. however, task specific adaptors trained on the source language will also face the issue of incongruity in the feature representations. hence, adaptors will not improve the cross-lingual transfer, but only improves the pre-trained features. we plan to extend our method towards integrating with adaptors to take advantage of pre-training features and improve performance.