Scaling, Simplification, and Adaptation: Lessons from Pretraining on Machine-Translated Text (2025)

Abstract: Most languages lack sufficient data for largescale monolingual pretraining, creating a “data wall.” Multilingual pretraining helps but is limited by language imbalance and the “curse of multilinguality.” An alternative is to translate high-resource text with machine translation (MT), which raises three questions: (1) How does MT-derived data scale with model capacity? (2) Can source-side transformations (e.g., simplifying English with an LLM) improve generalization to native text? (3) How well do models pretrained on MT-derived data adapt when continually trained on limited native text? We investigate these questions by translating English into Indonesian and Tamil—two typologically distant, lowerresource languages—and pretraining GPT-2 models (124M–774M) on native or MT-derived corpora from raw and LLM-simplified English. We evaluate cross-entropy loss on native text, along with accuracy on syntactic probes and downstream tasks. Our results show that (1) MT-pretrained models benefit from scaling; (2) source-side simplification harms generalization to native text; and (3) adapting MT-pretrained models on native text often yields better performance than native-only models, even with less native data. However, tasks requiring cultural nuance (e.g., toxicity detection) demand more exposure to native data.

## introduction language technologies have advanced rapidly, with large language models (llms) achieving strong performance across an array of tasks @xcite @xcite . scaling studies in pretraining language models show consistent gains with more parameters and more data @xcite . yet for most of the world's languages, the native corpora necessary to realize these pretraining benefits are scarce (üstün et al., 2024), causing models to quickly hit a "data wall"-a performance plateau imposed by limited training data. a common strategy to push past this data wall is multilingual pretraining, which aims to transfer knowledge from high-resource to lowresource languages. however, its effectiveness is constrained by challenges such as language imbalance @xcite , suboptimal multilingual vocabularies @xcite , and the "curse of multilinguality" @xcite . one alternative is to translate data from a highresource language into the target language using machine translation (mt). while this enables largescale corpus creation, it introduces limitations, including reliance on mt quality and the prevalence of "translationese"-literal phrasing, sourcelanguage bias, and cultural mismatches @xcite . nonetheless, its scalability makes mt a practical solution to data scarcity. recent studies investigate the utility of pretraining on mt-derived data (mt pretraining) in both monolingual @xcite and multilingual settings @xcite , consistently reporting downstream performance comparable to models pretrained on native text. we structure our study around three research questions: why these questions aren't obvious and why they matter. to answer these, we conduct controlled experiments by translating english into indonesian and tamil and compare gpt-2 models (124m-774m parameters) pretrained on native corpora against those trained on mt-derived data from both natural and llm-simplified english sources. we evaluate generalization to native text using cross-entropy loss on held-out data, as well as accuracy on syntactic minimal-pair probes and natural language understanding (nlu) tasks including sentiment analysis (sa), toxicity detection (td), natural language inference (nli), and causal reasoning (cr). our findings are as follows: to the best of our knowledge, this is the first systematic study of scaling effects in pretraining on mt-derived data, as well as the first exploration of source-side text manipulation prior to translation as a means of enhancing mt data quality. recent llm breakthroughs have centered on highresource languages like english, where abundant high-quality data is available @xcite . in contrast, low-resource languages still lag due to limited training data and benchmarks. this gap has driven community efforts such as masakhane @xcite , sea-crowd @xcite , and multilingual open-source llms like bloom @xcite and aya @xcite , highlighting the need for data and model development beyond english. pretraining on multilingual data. multilingual pretraining improves performance in low-resource languages @xcite , offering a path beyond the data wall. its promise lies in transferring knowledge across languages, but this comes with the "curse of multilinguality" @xcite , a phenomenon where training on many languages degrades performance on individual languages due to limited capacity and inter-language interference. despite notable successes @xcite @xcite , multilingual models still face challenges such as imbalanced data @xcite , and suboptimal tokenization @xcite . as an alternative for improving monolingual performance with limited native data, we explore leveraging mt models to generate target-language data for monolingual pretraining. pretraining on machine-translated data. pretraining on mt-derived data has been explored in monolingual settings for arabic @xcite and indic languages @xcite , as well as in multilingual settings @xcite , consistently showing downstream performance on par with models pretrained on native text. most related to our work is @xcite , who pretrained 28m and 85m decoder models and explored cpt of larger llms (gemma-2b, llama-3-8b) on translationese and native texts, finding mt-derived data competitive with native data. yet it remains unclear whether mt pretraining benefits larger models and whether cpt on native texts helps when the base model is pretrained on translationese. our study fills this gap by examining model scaling on mt-derived data (124m-774m), source-side manipulation before translation, and cpt on native texts. ## results and discussion we present results by our three research questions, then report translationese fine-tuning outcomes. each subsection starts with a short answer, followed by evidence and a practical takeaway. indonesian tamil model acc. ∆ acc. ∆ small native 53.6 71.5 natural-mt 47.6 66.2 natural-mt-cpt 52.9 +5.3 69.1 +2.9 simplified-mt 46.6 61.3 simplified-mt-cpt 52.4 +5.8 72.1 +10.8 medium native 52.4 62.8 natural-mt 50.5 65.5 natural-mt-cpt 53.7 +3.2 72.8 +7.3 simplified-mt 49.5 65.1 simplified-mt-cpt 52.1 +2.6 76.0 +10.9 large native 57.4 70.9 natural-mt 49.7 62.8 natural-mt-cpt 54.5 +4.8 72.8 +10.0 simplified-mt 49.7 62.8 simplified-mt-cpt 56.3 +6.6 70.9 +8.1 ## conclusion in this work, we asked whether larger models improve generalization to native text when pretraining data is pure machine-translated text, how sourceside complexity affects transfer to native text, and whether mt-pretrained models are good starting points for continually pretraining on native text. we observed three consistent patterns. first, for the 124m to 774m parameters setup, more parameters improve transfer to native text even when pretraining solely on mt-derived data. second, for source-side english texts, higher lexical and syntactic diversity yields mt-derived data that transfers better to native text. avoid operations that reduce this diversity (e.g., simplification) if the goal is native transfer. third, when native data is scarce, mt pretraining followed by continual pretraining on native text often outperforms native-only pretraining. in scenarios with zero or limited fine-tuning data, mt-derived fine-tuning data is useful for tasks like sentiment analysis and nli but has limited value for more culturally nuanced tasks such as toxicity detection. we distill our findings into a recipe for improving monolingual models beyond what is achievable with the available native data: for future work, extending these experiments to larger models, better mt systems, different sourceside and target languages, and more advanced preprocessing that balances mt ease with linguistic diversity will clarify when the effects observed here amplify or taper. furthermore, extending this approach to post-training regimes such as instruction tuning and preference alignment remains an open direction.