A Comparison of Language Modeling and Translation as Multilingual Pretraining Objectives (2024)

Abstract: Pretrained language models (PLMs) display impressive performances and have captured the attention of the NLP community.Establishing best practices in pretraining has, therefore, become a major focus of NLP research, especially since insights gained from monolingual English models may not necessarily apply to more complex multilingual models.One significant caveat of the current state of the art is that different works are rarely comparable: they often discuss different parameter counts, training data, and evaluation methodology.This paper proposes a comparison of multilingual pretraining objectives in a controlled methodological environment. We ensure that training data and model architectures are comparable, and discuss the downstream performances across 6 languages that we observe in probing and fine-tuning scenarios.We make two key observations: (1) the architecture dictates which pretraining objective is optimal; (2) multilingual translation is a very effective pretraining objective under the right conditions.We make our code, data, and model weights available at https://github.com/Helsinki-NLP/lm-vs-mt.

## introduction the release of bert @xcite has marked a paradigm shift in the nlp landscape and has ushered in a thorough investment of the nlp research community in developing large language models that can readily be adapted to novel situations. the design, training, and evaluation of these models has become a significant enterprise of its own. in recent years, that sustained interest has shifted also to encompass multilingual models (e.g., @xcite . there is considerable variation as to how such models are * equal contribution and corresponding authors. in short, we focus on two variables-model architecture and pretraining objectives-and set out to train five models in strictly comparable conditions and compare their monolingual performances in three downstream applications: sentiment analysis, named entity recognition, and pos-tagging. the scope of our study spans from encoder-decoder machine translation models, to decoder-only causal language models and encoderonly bert-like masked language models. we categorize them into double-stacks (encoder-decoder) and single-stacks (encoder-only or decoder-only) models. we intend to answer two research questions: there are a prima facie reasons to favor either answers to both of these questions. for instance, the success of multilingual pretrained language models (lm) on cross-lingual tasks has been underscored repeatedly @xcite , yet explicit alignments such as linear mapping @xcite and l2 alignment @xcite between source and target languages do not necessarily improve the quality of cross-lingual representations @xcite . our experiments provide tentative evidence that insofar as a bart denoising autoencoder architecture is concerned, models pretrained with a translation objective consistently outperform those trained with a denoising objective. however, for singlestack transformers, we observe causal language models to perform well in probing and masked language models to generally outperform translation and causal objectives when fine-tuned on downstream tasks. this leads us to conjecture that the optimal pretraining objective depends on the architecture. furthermore, the best downstream results we observe appear to stem from a machinetranslation system, highlighting that mt encoderdecoder systems might constitute an understudied but potentially very impactful type of pretrained model. ## methods and settings we start our inquiry by adopting a principled stance: we train strictly comparable models with mt and lm objectives before contrasting their performances on monolingual tasks. ## results double-stack models. we first compare the performance of 2-lm and 2-mt across several key language processing tasks including sa, ner, pos tagging, and nli. results are shown in tables 1a and 1b. the pretraining objectives play a significant role in shaping the models' effectiveness. specifically, 2-mt, which is pretrained with a machine translation objective, consistently outperforms 2-lm, which utilizes a denoising objective. this pattern is consistent across all languages tested after fine-tuning as well as probing. single-stack models. turning to the single-stack models (clm, mlm, tlm), we find a somewhat more complex picture. in a probing context (cf. fine-tuning also tends to minimize the difference between single-stack and double-stack modelswhich suggests that the higher quality of doublestack representations could be an artifact of training limitations. moreover, the relative ranks of the three single-stack models fluctuate much more than what we see for the double-stack models, owing to no little extent to the oftentimes momentous variation across seeds for single-stack models. we therefore conjecture that while a translation objective can yield a clear training signal towards semantically informed representations, this comes with two caveats: first, the signal can only be leveraged with dedicated separate modeling of source and target (viz. double-stack models); second, this advantage is much less consequential when fine-tuning. ## related works multilingual foundation models have flourished in recent years (a.o., @xcite @xcite @xcite @xcite , and with them so have studies of their representations @xcite @xcite @xcite . all of these works, however, fail to control for some of the most crucial factors, such as ensuring that all models are trained on comparable amounts of data. this work is specifically related to @xcite , which also compares mlm, clm, and tlm but does not normalize the training data. another point of comparison is @xcite , which studies the impact of mt continued pretrain-ing in bart on cross-lingual downstream tasks. monolingual evaluation of multilingual systems has also been broached a.o. by @xcite . ## conclusion this paper conducts an empirical study of how pretraining conditions of multilingual models impact downstream performances in probing and finetuning scenarios. despite the inherent limitations that stem from our stringent data requirements, our experiments offer a novel perspective that highlights directions for future inquiry into how multilingual foundation models ought to be pretrained. we observe that double-stack bart-based models fare much better than single-stack models in probing scenarios, but the difference is overall less clear when it comes to fine-tuning. we also find some tentative evidence that translation objectives can be highly effective for model pretraining in precise circumstances: namely, the most effective model on downstream tasks among those we experimented with is an mt-pretrained bart-like model, which outperforms both a more traditional denoising objective for bart as well as decoder-only clm and encoder-only mlm models. this would suggest that translation can serve as a powerful pretraining objective, although it is currently under-explored. 4 another crucial aspect of our study is that we present strictly comparable models, trained on comparable data, with comparable parameter counts and unified implementations. while this entails some limitations, especially with regard to the scale of models and data used, we nonetheless believe that a strict comparison can help discriminate between the various factors at play in other works. here, we find clear evidence that clm pretraining objectives, such as those used in gpt, outperform mlm-based models, such as bert, in probing scenarios; we are also able to isolate and highlight how the optimal choice of pretraining objective is contingent on the architecture being employed. for future work, we recommend exploring multitask learning during pretraining by combining objectives like translation, denoising, and language modeling; in such cases, models could harness the strengths of each task to become more robust and versatile. additionally, investigating training-free evaluation methods can offer insights into a model's inherent capabilities without the variability introduced by fine-tuning.