Understanding the effects of language-specific class imbalance in multilingual fine-tuning (2024)

Abstract: We study the effect of one type of imbalance often present in real-life multilingual classification datasets: an uneven distribution of labels across languages. We show evidence that fine-tuning a transformer-based Large Language Model (LLM) on a dataset with this imbalance leads to worse performance, a more pronounced separation of languages in the latent space, and the promotion of uninformative features. We modify the traditional class weighing approach to imbalance by calculating class weights separately for each language and show that this helps mitigate those detrimental effects. These results create awareness of the negative effects of language-specific class imbalance in multilingual fine-tuning and the way in which the model learns to rely on the separation of languages to perform the task.

## introduction transformer-based large language models (llms) lend themselves well to automatic classification tasks due to their superior performance, ability to be pre-trained on large amounts of data, and easy fine-tuning on downstream tasks. recently, methods like lora @xcite and adapters @xcite have been developed to fine-tune llms using fewer resources, making automation of classification tasks using llms more accessible than ever. multilingual versions of large language models, such as mbert are readily available. they are pre-trained on large multilingual corpora and build latent spaces that have both language-agnostic and language-specific components @xcite . previous works have studied the effect of finetuning on monolingual data on the representation of the multilingual space and cross-lingual transfer performance @xcite and showed that fine-tuning on a specific task with monolingual data reduces language-specificity @xcite . what is relatively understudied is the effect of multilingual fine-tuning on the multilingual space, which is especially interesting because it is not guaranteed that labels are similarly distributed across languages which could create an incentive for the model to rely on language for predictions. oftentimes, curated multilingual datasets will have the same distribution of labels across languages, and it is pointed out as a desirable property @xcite . however, in real-world datasets, data is often heterogeneous and class label distributions can vary significantly between languages. an example of this is the semeval 2018 task 1 dataset @xcite . class imbalance in the monolingual setting has been the focus of many previous works @xcite , some work addresses class imbalance in the multilingual setting @xcite , but to the best of our knowledge, language-specific class imbalance has not been studied in detail. in this paper, we analyse the effect of class imbalance 1 on the model with a number of experiments of multilingual classification on two different datasets. we chose to work with balanced dataset which we artificially imbalance to allow for controlled experiments. more specifically, we create two subsets of the data, one with a uniform joint distribution of language and labels and one with a skewed one. we want to create a better understanding of the influence of imbalance in multilingual fine-tuning. we first show that imbalance has a negative influence on performance and leads to the latent space becoming more separated by language. then, using shap values, we show evidence that the model learns to encode the imbalance even in non-informative tokens, thus effectively learning to classify based on language identity to an extent. we modify the traditional class weighing method to weigh datapoints of different languages separately and show that this mitigates the negative effects of the imbalance. in summary, our main contributions are: ## experimental setup the language model that we use is multilingual bert @xcite . specifically, we use the "bert-base-multilingual-cased" model from huggingface. we use a batch size of 16 with a gradient accumulation step of 8. we select the best model according to the validation loss. we use a linearly decreasing learning rate starting at 5e -5 for the language model and 5e -4 for the classifier head. they both reach 0 at the end of training 3 . we also perform the same experiments with xlm-r (conneau et al., 2020) and report the results in the annex a.2. we use the amazon reviews dataset @xcite in french, german, spanish, english, japanese and chinese, as those are all the available languages in the dataset, and xnli @xcite in french and english, since we wanted to test a bilingual setup. xnli is a text entailment task. for the amazon dataset, we train our models to predict the number of stars given (from 1 to 5). for the language identification experiments, we use the wiki dataset (foundation), specifically the pre-processed wikipedia dataset found on huggingface. the distribution of labels per language for the imbalanced datasets can be seen in tables 1a and 1b . the test sets for both xnli and the amazon reviews dataset are balanced in both marginal and joint distributions of language and labels. ## results and discussion in this section, we discuss results with respect to task performance first, after which we will shed light on the effect on language specificity of the multilingual space by showing results from experiments on language identification. we will show in detail what happens to different sets of features when confronted with class imbalance using shap 3 we make our code available here. ## conclusion in this paper, we showed that a language model trained on a seemingly balanced multilingual dataset, with uniform marginal distributions of languages and of labels, but skewed joint distribution of language and label, will learn this skew. we first showed that the model performs worse in the presence of this imbalance. based on the distribution of the test set predictions, we show that it learns to make predictions based on language, which can negatively impact its out-of-distribution performance. we also showed that the imbalance leads to the latent space being more separated by language. we then analyzed shap values to better understand how the way the model makes predictions changes. shap values showed that features that the model used when trained on balanced data became less important when trained on imbalanced data, and that features that were "neutral", i.e. didn't contribute to the prediction of a given label, became more important. we modify the traditional method of class weighing by calculating class weights separately for each language and train a model on the imbalanced dataset with a weighted loss. we show that this simple method is effective at mitigating the negative effects of the imbalance. this is of high stakes, as training on multiple languages is often done in real-life cases, and preventing the perpetuation of biases is often desirable. it is a reminder that large language models and deep learning architectures in general do not necessarily follow human intuition and will make predictions based on what is available in the data. training a model to build robust features requires careful consideration of not just the marginal distribution of the dataset features but also of their joint distribution.