Pretraining Language Models Using Translationese (2024)

Abstract: In this paper, we explore the utility of Translationese as synthetic data created using machine translation for pre-training language models (LMs) for low-resource languages (LRLs). Our simple methodology consists of translating large amounts of web-crawled monolingual documents (clean) into the LRLs, followed by filtering the translated documents using tiny LMs trained on small but clean LRL data. Taking the case of Indian languages, we pre-train LMs from scratch with 28M and 85M parameters, and then fine-tune them for 5 downstream natural language understanding (NLU) and 4 generative (NLG) tasks. We observe that pre-training on filtered synthetic data leads to relative performance drops of only 0.87% for NLU and 2.35% for NLG, compared to pre-training on clean data, and this gap further diminishes upon the inclusion of a small amount of clean data. We also study the impact of synthetic data filtering and the choice of source language for synthetic data generation. Furthermore, evaluating continually pre-trained larger models like Gemma-2B and Llama-3-8B in few-shot settings, we observe that using synthetic data is competitive with using clean data. Our findings suggest that synthetic data shows promise for bridging the pre-training gap between English and LRLs.

## introduction large language models (llms) @xcite @xcite have been able to perform very well on downstream tasks like mmlu @xcite , big-bench @xcite , etc, and have even started to reach human potential in many of these tasks. but this performance has very largely been credited to their scale and the vast amount of data that they have been fed. most of these language models (lms) perform well in languages like english where abundant data is available @xcite , but figure 1 : comparing nlu performance in english, hindi, and gujarati shows that filtering synthetic data and adding 10% clean data improves models, approaching the performance of those trained only on clean web data. a vast majority of languages don't have comparable data as compared to english. as a consequence, many llms, both monolingual and multilingual, involving these languages still show poor performance for various downstream tasks. for example, the largest open source multilingual model bloom @xcite covers 46 natural languages spanning 9 language families, but the top 5 languages comprise 74.14% of the data. even for models like mt5 @xcite , the top 10 of 107 languages account for more than 75.48% of the training data. despite the benefits of multilingualism @xcite , this data skew still means that low-resource languages will underperform. fortunately synthetic data is an option and previous works such as, but not limited to, backtranslation @xcite , sequence distillation @xcite , also known as forward translation, etc. have shown that synthetic data obtained using machine translation (mt) can supplement resource scarcity and can significantly enhance model performance @xcite . however, to the best of our knowledge, there has been no work on showing the effectiveness of synthetic data for pre-training lms. furthermore, the quality of synthetic data is also important, which many works take for granted. while round-trip-translation @xcite or referenceless neural quality estimation (qe) @xcite are viable, they either involve twice the compute or a reasonably large model not available for most languages, and this might not be optimal to determine the quality of synthetic documents efficiently. we thus consider tinylms @xcite as an efficient alternative, which have been shown to model documents by their fluent paragraph generation capabilities. in this paper, we focus on indic languages such as hindi, gujarati, and marathi, and present a comprehensive study of the utility of synthetic monolingual data, also called translationese @xcite , obtained using machine translation (mt) for pre-training lms. we propose a simple framework that involves training tiny language models, henceforth tinylms, on original web-crawled data (clean) and then using them to filter synthetic data. we then compare lms of different scales pretrained on clean and synthetic data followed by finetuning on natural language understanding (nlp) and generation (nlg) downstream tasks, where we observe that, while unfiltered synthetic data based lms are inferior compared to lms trained on clean data, filtering leads to performance comparable to the latter. we further show that tuning these synthetic data lms on small clean data leads to further improvements. we also show that these trends hold when continually pre-training llms such as gemma-2b and llama-3-8b. our contributions are: a. a simple framework involving high-quality mt models and tinylms trained on clean web-crawled data to mass-produce and filter synthetic data for lm training. b. demonstrating the efficacy of language models (up to llama-3-8b) trained on filtered synthetic data across a range of nlu and nlg tasks for low resource indic languages. c. a new document-level monolingual corpora (in-dicmonodoc) consisting of 39.5b tokens worth of monolingual clean document-level data spanning 22 scheduled languages and english 1 . ## related work this paper focuses on creating, filtering, and utilizing synthetic data to train tinylms. monolingual data: previous efforts to collect monolingual corpora for indic languages include the emille/ciil corpus @xcite , hindmonocorp @xcite , leipzig corpus @xcite , indiccorpv1 @xcite , and indiccorpv2 @xcite . while indiccorpv2 is large, it is sentence-level and suitable for nlu models but not for longer contexts needed by language models. we extend these corpora and demonstrate the effectiveness of using synthetic data. synthetic data generation and quality estimation: synthetic data aids nlp tasks like back translation for machine translation @xcite @xcite @xcite and native language identification @xcite . however, using synthetic data for pretraining lms is less explored due to hallucination @xcite and ungrounded text @xcite . evaluation methods like rtt bleu scores are computationally intensive, while others like bartscore @xcite , t5score @xcite , mqm, and comet @xcite require large-scale models or human annotations, limiting scalability. approaches like kenlm @xcite have been used to filter monolingual corpora based on perplexity. transfer learning and cross-lingual fine-tuning: approaches like translate-train, as described by @xcite , involve fine-tuning a multilingual plm using machinetranslated training data and evaluating in the target language. oh et al. (2022) combined translate-train and translate-test for improved cross-lingual finetuning. in contrast, our work focuses on pretraining language models and exploring how synthetic text impacts pretraining and various downstream nlu and nlg tasks. tinylms: small lms, even with 10m parameters, produce fluent and consistent text @xcite . challenges like babylm @xcite focus on improving lms within a fixed data budget. we take motivation from this and leverage tinylms for efficient filtering of synthetic documents. figure 2 : overview of our approach to pre-train language models using translationese data. we leverage rich monolingual corpora in the src language and scarce corpora in the tgt language. our method involves employing a pre-trained machine translation model to translate src to tgt. we then filter, using perplexity, the resulting text using a tinylm trained solely on clean tgt monolingual data. the filtered synthetic data can be used to further pretrain larger language models. ## methodology in this section, we describe our framework for leveraging synthetic data for lm training. this process consists of collecting monolingual (clean) data from the web for low-resource languages, training tinylms with it, translating clean data from a high resource language such as english into low-resource languages, using the aforementioned tinylms to filter synthetic data, and then using this filtered data to train lms for downstream tasks. our framework is illustrated in figure 2 . 3.1 collecting clean monolingual corpora following doddapaneni et al. (2023); rae et al. (2022); team et al. ( 2022 ), for all languages of interest, we a. obtain a list of urls to be crawled via word-level n-grams passed to a search engine, b. after url deduplication, we crawl all applicable webpages, c. automatically and manually @xcite filter out unwanted text like html tags and emoticons, d. use language detection-based (lid) filtering using cld3 foot_0 and indiclid-ftn model @xcite to discard languages not of interest, e. perform document filtering to drop documents containing offensive text using toxic words list provided by @xcite , f. merge all the filtered corpus with wikipedia, oscar @xcite and some dumps of mc4 @xcite and finally, g. perform deduplication at paragraph level using murmurhash algorithm foot_1 with a 128-bit unsigned hash for each monolingual split of the corpora. we crawl data for english, with indic context, and 22 indic languages. as a result, we end up with indicmonodoc, with 27.5 billion tokens worth of indic language documents and 12 billion tokens of english documents for a total of 39.5 billion tokens of clean monolingual data. this is larger than the corpora released by @xcite , surpassing it by almost 2 times. we use indicmon-odoc for all experiments with clean data. figure 3 gives an overview of the comparison of indicmon-odoc. note that, creation of indicmonodoc is important since indiccorpv2 is a sentence-level corpus, and training lms need a document-level corpus. it is important to note that we paid special attention to the low-resource languages. in this paper we only use data corresponding to english, hindi, gujarati and marathi. we report additional details of indicmonodoc in appendix e. ## experiments this section outlines the training procedures and datasets for the models described in section 3. we pre-train decoder only lms and fine-tune all models from scratch in monolingual and bilingual settings using the causal language modeling (clm) objective for nlg tasks and a linear classification head for classification tasks. we specify the dataset samples used for pretraining and fine-tuning, and analyze the effects of synthetic corpora on pretraining. ## results we now present our results which help establish the utility of synthetic data for language modeling. ## conclusion in this paper, we performed a first of its kind study showing the promise of using translationese data for training language models for low-resource languages. our simple pipeline involves the translation of high-resource source language documents at scale, followed by perplexity based filtering using small and efficient language models trained on clean target low-resource language data. we then showed on a variety of downstream natural language understanding and generative tasks that both small and large language models pre-trained on clean synthetic data are comparable to those trained on clean data. while we observed that the source language for synthetic data generation matters, it is clear that synthetic data can help bridge the resource scarcity faced by a vast majority of languages for language modeling. future work will focus on better and faster synthetic data generation and filtering mechanisms.