Not Enough Data to Pre-train Your Language Model? MT to the Rescue! (2023)

Abstract: In recent years, pre-trained transformer-based language models (LM) have become a key resource for implementing most NLP tasks. However, pre-training such models demands large text collections not available in most languages. In this paper, we study the use of machine-translated corpora for pre-training LMs. We answer the following research questions: RQ1: Is MT-based data an alternative to real data for learning a LM?; RQ2: Can real data be complemented with translated data and improve the resulting LM? In order to validate these two questions, several BERT models for Basque have been trained, combining real data and synthetic data translated from Spanish.The evaluation carried out on 9 NLU tasks indicates that models trained exclusively on translated data offer competitive results. Furthermore, models trained with real data can be improved with synthetic data, although further research is needed on the matter.

## introduction since the emergence of the attention-based transformer architecture @xcite and the masking pre-training strategies introduced by bert @xcite , transformer-based language models have become the default approach for many nlp tasks, leading to an impressive performance in high-resource languages, particularly english @xcite @xcite @xcite . as scaling laws dictate @xcite , such competitive models are achievable with big computational budgets and large corpora available, requirements difficult to meet for most languages @xcite . fortunately, lms are being built for lessresourced languages, such as kinyabert for kinyarwanda (390m words) @xcite , elhberteu for basque (351m) @xcite , gabert for irish (161m) @xcite , luxembert for luxembourgish (130m) @xcite , bertinho 45m for galician @xcite , swahbert for swahili (16m) @xcite and qubert for quechua (4m) @xcite . @xcite estimates that 10m-100m words of pre-training data are enough for an lm to acquire the linguistic capacities of syntax and semantics, but the amount of data required to acquire factual knowledge and commonsense is higher. in this work, we propose to tackle the lack of data by using text corpora available in other languages translated via machine translation (mt). to the best of our knowledge, this has been addressed before @xcite but not indepth, and only for a closely related language pair (german-luxembourgish). we selected basque, a language isolate, as the target language, and employ spanish as the auxiliary language. we direct our efforts to answer the following research questions (rq): rq1: can we obtain comparable performance to a native lm by training lms just on synthetic data from mt? rq2: can we improve current lms for lessresourced languages by adding synthetic mt data? ## methodology in order to answer our research questions, we set out the following methodology. we propose two baseline lms: i) elhberteu @xcite as a strong baseline, trained on a corpus of 351m words; and (ii) bert 125m a model trained on a lower data regime. from there on we pre-train various lms with different native/synthetic data combinations. sections 3 and 4 give details of the models pre-trained, including baselines. all models presented in this paper follow the bert base architecture @xcite . we select basque, a language isolate, as a tar-get language, and employ spanish, a romance language, as the auxiliary language since it has huge text corpora available. furthermore, both languages coexist in the same geographical area, therefore, spanish is the language that basque shares the most parallel data with, which is crucial to train mt systems. on the other hand, this is a real case since obtaining a corpus in basque that exceeds 350m words is difficult. ## lm trained solely on synthetic data rq1 aims to prove if it is possible to train a competitive lm with just synthetic text obtained from mt. in order to do that we train a bert model on s_beto2eu (s_bert), and evaluate if the model trained exclusively on synthetic data is able to perform as well as models trained on real data. the results on the basqueglue benchmark for s_bert are reported in in order to improve the results obtained with the synthetic data, we analyse two specific aspects of the data: i) the quality loss during the translation process; ii) the cultural context of the synthetic data. following we analyze each of those factors. ## combining native and synthetic data the objective of rq2 is to test if adding texts translated by mt to a native corpus can boost the performance on downstream nlu tasks of the lm in the target language. with that aim, we trained a new lm on the concatenation of s_beto2eu corpus and the n_elhberteu corpus foot_1 (sn_bert hereinafter). ## conclusions regarding the rq1, we conclude from our experiments that lms trained exclusively on synthetic data from mt can obtain comparable performance to a native lm. we further analyze that other than the quality of mt, the cultural context of the text we select from the auxiliary language do have an effect on the final performance. we conclude that it is better to gather a corpus composed of sources similar to those in the target language, rather than indiscriminately translating vast amounts of data in the auxiliary language. furthermore, with respect to rq2, our experiments show that state-of-the-art models' performance can be improved by adding translated data during the pre-training, albeit it is a small improvement. weighting the native data above synthetic data is key to this improvement. all in all, this approach has a big potential for less-resourced languages, since once you have a proper mt system, there is no limit on the amount of data one can translate from languages with bigger corpora available. data and models are publicly available foot_3 .