Generalization Measures for Zero-Shot Cross-Lingual Transfer (2024)

Abstract: Building robust and reliable machine learning systems requires models with the capacity to generalize their knowledge to interpret unseen inputs with different characteristics. Traditional language model evaluation tasks lack informative metrics about model generalization, and their applicability in new settings is often measured using task and language-specific downstream performance, which is lacking in many languages and tasks. To address this gap, we explore a set of efficient and reliable measures that could aid in computing more information related to the generalization capability of language models, particularly in cross-lingual zero-shot settings. Our central hypothesis is that the sharpness of a model’s loss landscape, i.e., the representation of loss values over its weight space, can indicate its generalization potential, with a flatter landscape suggesting better generalization. We propose a novel and stable algorithm to reliably compute the sharpness of a model optimum, and demonstrate its correlation with successful cross-lingual transfer.

## introduction generalization enables models to use prior knowledge to reasonably respond to previously unseen stimuli. although traditional machine learning evaluation is performed based on a preselected set of prediction or generation tasks, accuracy on many public benchmarks may often not be sufficient to extensively assess the ability to perform well in new settings. therefore, a majority of researchers have found it worthwhile to investigate measures that could evaluate the generalization capability of models with properties, such as vc dimension @xcite , crossentropy @xcite , complexity @xcite or variation in parameters during training @xcite . among these, recent findings support the smoothness in the loss curvature to correlate best with generalization capability @xcite @xcite , motivating the development of learning methods that induce smoothness in the learning trajectory such that the model becomes more robust; either through data perturbation @xcite @xcite @xcite @xcite or by integrating the measure directly to the optimization objective @xcite @xcite @xcite @xcite . however it might often not be straightforward to compute such measures in high-dimensional feature space in a stable fashion @xcite . as models get larger and cover more languages, the possibility of improving the applicability of nlp systems in many under-resourced languages gets more promising. an essential requirement in studying the dynamics of cross-lingual knowledge transfer is to have an evaluation methodology that can reliably measure the model's capability in generalization of knowledge under different scenarios. there is a common hypothesis that states that a model demonstrating an extended flat optimum area of low loss value surrounding the minimized loss is indicative of better generalization capability. in this work, we study the above hypothesis and present the first study to provide methods that can be used for measuring the cross-lingual generalization capability of language models. loss-landscape minima one of the most promising indicators of generalization capability to date seems to be related to the form of the loss landscape, in particular, the sharpness in the loss curvature. a potential reason for this fallback is traced to stochastic gradient descent (sgd) @xcite methods which often fall into sharp minima of the loss surface @xcite @xcite . although clear conclusions on the relationship between sharpness and generalization performance, such as whether sharper @xcite vs. flatter @xcite minima would generally yield better generalization, are still due. the main idea behind these methods is that their objective is to explicitly find flat minima, often using stochastic averaging methods @xcite , mini-max or sharpness-aware minimization methods, which can be computed by direct formulation based on the hessian matrix of the loss function @xcite or monte-carlo approximations of the minimizer's neighborhood @xcite . adversarial optimization comparison of two approaches finds that for nlp tasks, mini-max methods are more competitive over averagingbased optimization @xcite . @xcite hypothesize that regularizing the trace of the fisher information matrix amplifies the implicit bias of sgd, which prevents memorization. the fisher information @xcite measures local curvature, so a smaller trace implies a flatter minimum, which gives the model more freedom to reach an optimum. instead of explicitly minimizing the values of parameters, @xcite propose minimizing both loss and sharpness while optimizing the parameters such that they lie in neighborhoods with low loss values. perturbation is an auxiliary objective that encourages the model predictions to be similar in the vicinity of the observed training samples (englesson and azizpour, 2021), usually by penalizing the kl-divergence between the probability distribution of the perturbed and normal model. perturbations can be adversarial inputs @xcite or inputs with gaussian or uniform noise @xcite . to improve cross-lingual generalization, translations of the input generated by machine translation systems were used as perturbed input @xcite . other work also has found the benefit of enforcing consistency for perturbations within the model in addition to the input distribution @xcite . ## methodology in this study, we undertake the development of a methodology that could benefit an accurate assessment of the generalization capability of models for the purpose of cross-lingual knowledge transfer into under-resourced languages. this section first presents approaches to improving generalization performance and the selected measures that provide stable results for measuring zero-shot cross-lingual transfer performance. ## experiments for comparison, we implement each sharpnessbased optimization as a fine-tuning objec-tive on the multilingual mbert base variant (bert-base-multilingual-cased from huggingface) @xcite in addition to mt5 model (google/mt5-small) @xcite . we use a linear classification layer of size 768x3 where the output dimension is equal to the number of labels. we adopt a two-step training approach in our experiments. first, we fine-tune the model on the english language part of the xnli dataset to optimize the model to learn the task specifically in english. subsequently, we perform a zero-shot transfer of the fine-tuned model on the rest of the 14 languages to facilitate an evaluation of the generalization of models. ## conclusion enabling cross-lingual knowledge transfer is an important step towards extending the applicability of nlp models to more languages. despite recent efforts to develop better optimization methods for improving the generalization of language models in new languages or domains; these techniques try different types of methods to achieve higher performance such as sharpness-based minimizations, reducing gradient of loss functions, or consistency regularization. evaluating these techniques thoroughly without a standardized method-ology remains a difficult task. this work aims to uncover insights into how to measure cross-lingual generalization by exploring suitable measures that work well under different settings. our experiments studying model loss landscape and parameter properties find strong relationships between the margin, sharpness in the loss minima neighborhood, and zero-shot cross-lingual downstream task performance, both on validation and test sets, supporting strong applicability to evaluate models before deploying them in new languages.