BERT astic at S em E val-2023 Task 3: Fine-Tuning Pretrained Multilingual Transformers Does Order Matter? (2023)

Abstract: The naive approach for fine-tuning pretrained deep learning models on downstream tasks involves feeding them mini-batches of randomly sampled data. In this paper, we propose a more elaborate method for fine-tuning Pretrained Multilingual Transformers (PMTs) on multilingual data. Inspired by the success of curriculum learning approaches, we investigate the significance of fine-tuning PMTs on multilingual data in a sequential fashion language by language. Unlike the curriculum learning paradigm where the model is presented with increasingly complex examples, we do not adopt a notion of “easy” and “hard” samples. Instead, our experiments draw insight from psychological findings on how the human brain processes new information and the persistence of newly learned concepts. We perform our experiments on a challenging news-framing dataset that contains texts in six languages. Our proposed method outperforms the naïve approach by achieving improvements of 2.57\% in terms of F1 score. Even when we supplement the naïve approach with recency fine-tuning, we still achieve an improvement of 1.34\% with a 3.63\%$ convergence speed-up. Moreover, we are the first to observe an interesting pattern in which deep learning models exhibit a human-like primacy-recency effect.

## introduction deep learning models are state-of-the-art (sota) in many fields including natural language processing (nlp). in nlp, the current sota models @xcite are based on transformers, which are deep-learning models with attention mechanism @xcite . while many transformers are monolingual, there has been increased research and public interest in multilingual transformers @xcite . notably, pretrained transformers, require huge amounts of training data, no matter the domain @xcite . imposing an order on the training data in which pmts are fine-tuned sequentially language by language matters. pmts tend to perform relatively better on languages at either end of the training sequence. the performance improvement is measured against the naïve approach of language-agnostic uniform sampling. consequently, this results in a substantial carbon footprint @xcite , which is against global sustainability objectives. contributions following the success of cl approaches, and inspired by cognitive science, we propose an approach to fine-tuning pmts on multilingual data. we investigate the significance of doing this in a sequential fashion language by language. unlike cl where the model is presented with increasingly complex examples, we do not adopt a notion of "easy" and "hard" examples. instead, our experiments draw insight from psychological findings on how the human brain processes new information and the persistence of newly learned concepts @xcite . we perform our experiments on a multi-label text classification dataset @xcite that contains text in six languages: english, french, german, italian, polish, and russian. our proposed method outperforms the naïve approach by achieving an f1 score gain of 2.57%. even when we supplement the naïve approach with recency finetuning, it achieves an f1 score gain of 1.34% with a 3.63% average convergence speed-up. moreover, we observe an interesting pattern in which deep learning models exhibit a human-like primacyrecency effect, which is also commonly referred to as the serial-position effect @xcite . the effect describes the human tendency to remember the first and the last items in a list more accurately than the ones in the middle. our contributions are as follows: ## system overview we perform all of our experiments using two transformer-based models, xlm-r with 278m parameters @xcite and uncased bert with 110m parameters @xcite , using the setup depicted in figure 3 . we illustrate in the upcoming subsections the different training strategies we adopt in our experiments. ## experimental setup for all of our experiments, we split the task's training set using stratified sampling foot_1 to deal with class imbalance. we adopt a 70/30 train/dev split, and we use the official development set as a test set. in sections 3.2, 3.3, and for bert-tr from section 3.5, we do the split on the combined dataset from all languages, while for the other sections, we do this for each language independently. we use an nvidia quadro rtx 6000 gpu to fine-tune our models. we use wordpiece tokenizer for bert and sentencepiece tokenizer for xlm-r. as for xlm-r-o 1 , we fine-tune xlm-r-bl on english with early stopping, then take the resulting model and fine-tune on french with early stopping, and so on all the way till russian to obtain xlm-r-o 1 . in other words, xlm-r-o 1 involves six finetuning steps with six early stopping convergences. the same methodology is repeated for all other xlm-r-o* models with the only difference being the different order of languages when fine-tuning. ## conclusion and future work we are the first to illustrate the significance of finetuning pmts sequentially language by language. we show that this not only yields sizable performance gains of 2.57% over the naïve approach, but it also converges 3.63% faster when supplemented with recency fine-tuning. moreover, we are the first to observe a human-like primacy-recency effect in deep learning models in general and pmts in particular. we also perform other experiments to study data augmentation via translation, and we study how fine-tuning on any single language benefits all other languages. in the future, we plan to experiment with larger datasets, more languages, multiple runs with different random seeds, different downstream tasks, and more variants of pmts. this work has the potential to give us a better understanding of the inner workings of deep learning models by drawing insight from psychology. it also has the potential to improve the way deep learning models in general, and pmts in particular are pretrained and fine-tuned. in our work, we looked at only six out of 720 possible sequences of languages for fine-tuning. it would be interesting to investigate other sequences. for example, it is interesting to study keeping languages from the same family closer in the sequence or farther apart. it would also be interesting to frame this research as a cl problem and define a notion of "easy" and "hard" languages.