T rans BERT : A Framework for Synthetic Translation in Domain-Specific Language Modeling (2025)

Abstract: The scarcity of non-English language data in specialized domains significantly limits the development of effective Natural Language Processing (NLP) tools. We present TransBERT, a novel framework for pre-training language models using exclusively synthetically translated text, and introduce TransCorpus, a scalable translation toolkit. Focusing on the life sciences domain in French, our approach demonstrates that state-of-the-art performance on various downstream tasks can be achieved solely by leveraging synthetically translated data. We release the TransCorpus toolkit, the TransCorpus-bio-fr corpus (36.4GB of French life sciences text), TransBERT-bio-fr, its associated pre-trained language model and reproducible code for both pre-training and fine-tuning. Our results highlight the viability of synthetic translation in a high-resource translation direction for building high-quality NLP resources in low-resource language/domain pairs.

## introduction pre-trained language models (plms) have revolutionized the field of natural language processing (nlp) by leveraging large-scale datasets and powerful neural network architectures to learn rich linguistic representations. these models, such as bert @xcite , gpt @xcite , and t5 @xcite , are pre-trained on vast amounts of text data in an unsupervised manner, enabling them to capture intricate patterns and nuances of human language. plms can be finetuned for specific tasks, such as text classification, named entity recognition (ner), and question answering (qa), by training them on smaller labeled datasets. this transfer learning approach has significantly improved the performance of nlp models across various languages and domains. unfortunately, the success of plms has not been equally distributed across all languages. while high-resource languages like english, chinese, and french have seen significant advancements in nlp applications, many low-resource languages still lack the necessary data and resources to develop effective models. this disparity is particularly evident in specialized domains such as life sciences, where the availability of high-quality datasets is crucial for training accurate models. for example, hindi, which is spoken by over 600m people, has no available plm for the life sciences domain. although @xcite , the first pretrained language model for the life sciences, was released in 2019, significant efforts to gather sufficient domain-specific (ds) data for training life sciences models in other high-resource languages have begun to emerge in recent years. since 2023, life sciences models have emerged for german @xcite , @xcite , and french @xcite . life sciences is only an example of a domain where the lack of data is a significant barrier to the development of nlp tools. other domains, such as legal, finance, and patent, also face similar challenges. in this paper, we introduce transcorpus, an open-source toolkit leveraging the fairseq translation framework @xcite to generate extensive synthetic ds corpora in up to 100 languages, featuring a production-level api/cli setup with multi-gpu and multi-processing capabilities, and reliable checkpoint recovery for scalable corpus management. in the context of a high-resource translation direction, we demonstrate that a language model (lm) trained on transcorpus output can achieve state-of-the-art performance on various downstream tasks by leveraging drbenchmark, a french life sciences benchmark @xcite . our contributions are threefold with (1) the re-lease of an open-source toolkit foot_0 for scalable multilingual corpus translation, (2) a french life sciences corpus foot_1 synthetically translated of 36.4gb along with a tokenizer, and a plm foot_2 released on hugging face, and (3) reproducible code for the pre-training and fine-tuning of the french life sciences plm made publicly available on github 4 . for future research, we are in the process of adding new languages and publishing them in the hugging face datasets repository. to see the current state of the added languages, check out the following link: https://huggingface.co/ jknafou/datasets?@xmath0. ## related work the paradigm of training lms on massive datasets is relatively recent, gaining prominence after the introduction of bert @xcite , and as a result, there is still limited research leveraging translation for training such models, especially in low-resource settings. in @xcite , sentiment analysis in four low-resource scandinavian languages is explored using three strategies: fine-tuning a native monolingual plm, translating the data into english and fine-tuning an english plm, and fine-tuning a multilingual plm on the native data. results generally favor the multilingual approach, though fine-tuning an english model on translated data often outperforms using a monolingual low-resource plm. for luxembourgish, @xcite tackle data scarcity by partially translating unambiguous words from a related high-resource language, evaluating several models including a luxembourgishonly bert, a luxembourgish-german bert, and luxembert, which is trained on mixed corpora. luxembert shows improved performance over mbert, though not to a statistically significant extent. in the basque context, following the introduction of elhberteu @xcite , @xcite use synthetic translated data from spanish to enlarge the basque corpus, finding that while a plm trained solely on synthetic data is competitive, it does not outperform one trained only on native data; however, supplementing native data with synthetic translations does enhance performance. @xcite improve the english-to-vietnamese machine translation (mt) model mtet by injecting synthetic biomedical parallel text via self-training @xcite , resulting in a system that outperforms strong baselines and enables the creation of vipubmed and vimednli datasets. continued pre-training and fine-tuning on these resources lead to vipubmedt5, which achieves state-of-the-art results in several biomedical nlp tasks, further demonstrating the potential of synthetic translation data for advancing low-resource language modeling. ishigaki et al. (2023) pre-train a japanese bert model on 2.5m abstracts from web of science which were translated into japanese via amazon translate, alongside 1.2m native japanese abstracts from wikipedia. although no comparisons are made with existing plms, experiments involving entity and relation extraction tasks indicate that models that use translated data exhibit superior performance over those trained on native data. ## transcorpus: a scalable translation framework in this section, we present transcorpus, a framework designed to facilitate the translation of largescale corpora into multiple languages. first, the selection of the mt toolkit along with its model will be presented, then the model size and context length will be discussed, and finally, the proposed translation workflow will be illustrated. ## transcorpus-bio-fr: a french life sciences corpus as already highlighted, the limited availability of ds plms for certain language/domain pairs presents a notable challenge to the progression of nlp tools. evaluating our framework in a realworld context is complex for several reasons: (1) it is uncommon to find sufficient ds for low-resource language/domain pairs that also have datasets for model evaluation, and (2) even when such benchmarks exist, an appropriate plm for comparison might not be available. fortunately, in the domain of french life sciences, two important papers have recently been published. the first is drbenchmark @xcite a life sciences benchmark that includes multiple datasets supporting the evaluation of in-domain models. the second is dr-bert @xcite , a french life sciences plm. comparing our model against drbert on life sciences tasks allows us to assess the practical effectiveness of our framework. indeed, as drbert is pre-trained from scratch, it does not rely on an advanced general domain plm such as camembert @xcite , which is also the case for most languages. ## transbert pre-training the reasons for pre-training a lm from scratch are twofold. firstly, it enables us to project the usage of our framework on languages that might lack a general domain plm. secondly, it allows the use of our custom tokenizer, which typically provides enhanced performance for ds lms. ## transbert-bio-fr: application to life sciences in french this section details the pre-training of transbertbio-fr and compares it with other french plms. ## experimental setup to compare transbert-bio-fr with other plms, we leveraged drbenchmark which is composed of multiple datasets and tasks. this section describes the experimental setup under which each model was evaluated. ## conclusion & contributions this work establishes a rigorous framework for assessing lms on ds for non-english dataset. it builds upon prior research and extends it to a more comprehensive benchmark that includes a more robust way of evaluating the models by applying hpo, multiple training repetition, 5-folds crossvalidation, and statistical testing on 15 datasets along with their aggregation by task. it illustrates that employing translated synthetic data for training ds lms is a viable approach to address the lack of native language data. our proposed model, transbert-bio-fr, outperforms existing sota models in various life sciences tasks, including classification, ner, pos, and sts. in addition to offering a viable methodology to address data scarcity, we release to the public transcorpus, an adaptative toolkit designed to facilitate the translation of large-scale corpora into multiple languages. the resources generated from this work, including transcorpus-bio-fr, transbertbio-fr and the code for the pre-training and finetuning of the models are also made available on github.