Leveraging High-Resource E nglish Corpora for Cross-lingual Domain Adaptation in Low-Resource J apanese Medicine via Continued Pre-training (2025)

Abstract: Limited low-resource language corpora in professional domains like medicine hinder cross-lingual domain adaptation of pre-trained large language models (PLMs). While abundant English medical corpora could complement this scarcity, the effective mixture of English and target language, including machine-translated content, remains underexplored. We examined how linguistic features (e.g., token sizes and language proportions) affect performance on a Japanese–English medical knowledge benchmark. Through continued pre-training of a bilingual PLM on multilingual corpora with varying proportions of English and Japanese texts (both original and machine-translated), we analyzed correlations between linguistic features and fine-grained task performance. Our findings suggest a practical approach to optimizing multilingual corpora for cross-lingual domain adaptation, which requires leveraging specialized knowledge from English corpora while ensuring sufficient coverage of language-specific expressions in a target language (Japanese). Such insights will contribute to the development of multilingual models that effectively leverage English-language resources in various professional domains with low-resource languages.

## introduction imbalanced language resources pose a significant challenge for pre-trained large language models (plms) in achieving cross-lingual domain adaptation in specific target languages. this imbalance is especially pronounced in professional domains such as medicine, where general biomedical knowledge circulates globally in english, while available resources in the target language remain relatively limited. for example, pubmed hosts over 38 million biomedical papers globally foot_0 , while j-stage, a comparable japanese database, contains only around 5 million foot_1 . while abundant english medical corpora offer a promising avenue for augmenting scarce target-language data, the optimal continued pre-training strategy for acquiring knowledge from well-resourced source languages (often english) to support domain adaptation in less-resourced target languages has yet to be thoroughly explored. here, we investigate the optimal corpus composition for the continued pre-training of a bilingual (japanese-english) plm, with a particular focus on leveraging abundant english-language resources to enhance knowledge acquisition in japanese medicine. continued pre-training usually follows initial pre-training on general corpora, where large language models acquire foundational language abilities such as lexical, syntactic, and semantic patterns, as well as general factual knowledge @xcite . then, continued pre-training leverages additional corpora containing domain-specific or targetlanguage texts, with its effectiveness for domain adaptation demonstrated across multiple studies @xcite @xcite @xcite . nevertheless, several practical considerations have been overlooked for effective continued pretraining aimed at cross-lingual domain adaptation in low-resource professional domains. for example, the optimal mixing ratio of source and target languages for acquiring knowledge from english corpora remains unclear. while current machine translation systems provide reasonable quality, the balance between original and translated content is still not well understood. furthermore, existing studies often lack detailed analyses of how corpus compositional statistics (e.g., token sizes and fig. 1 : study overview. this study comprises three steps. (1) first, we performed continued pre-training on pre-trained large language models using diverse multilingual corpora. (2) next, we computed the difference in scores before and after the continued pre-training using the japanese-english medical knowledge benchmark, jmedbench. in this study, we address the following research questions (rqs): rq1: how do original english and machinetranslated japanese corpora help a bilingual (japanese-english) plm achieve domain adaptation in the japanese medical domain? rq2: what is the optimal corpus configuration and proportion of english and japanese texts for achieving the best performance in the medical domain? rq3: how do specific corpus compsitional statistics in multilingual corpora influence model performance across diverse medical tasks? to investigate these questions, we systematically compared the impact of multilingual corpora containing varying proportions of japanese and english medical content (see the study overview in fig. 1 ). to characterize the language composition of these corpora, we defined seven compositional statistics: total token count, japanese token count, english token count, parallel token count (paragraph-aligned bilingual medical texts), and the ratios of japanese, english, and parallel tokens. then, we employed 13-billion-parameter bilingual (japanese-english) plms and computed the difference in model performance on a comprehensive japanese-english medical knowledge benchmark, jmedbench @xcite , before and after continued pre-training. jmedbench comprises 20 japanese and 7 english tasks, including multiple-choice question answering (mcqa), machine translation (mt), named entity recognition (ner), document classification (dc), and semantic textual similarity (sts) (see appendix a). finally, we applied partial correlation analysis, which estimates the strength and direction of a relationship between two variables while controlling for other covariates. this enabled us to isolate the unique contribution of each compositional statistic to task performance despite inherent mutual correlationsfor example, more japanese tokens automatically raise the total token count. our findings underscore the need to optimize corpus composition so that high-resource english texts can be leveraged effectively for cross-lingual domain adaptation in the low-resource japanese medical domain. our contributions, which correspond to the rqs, can be summarized as follows: ## method this study comprises three steps (see fig. 1 ): (1) continued pre-training of a bilingual (japanese-english) plm on diverse multilingual corpora with various language compositions; (2) computation of task-wise score differences on jmedbench before and after continued pre-training; and (3) partial correlation analysis to examine task-wise correlations with corpus compositional statistics. ## analysis here, we analyze the optimal corpus composition for continued pre-training. rq1: how do original english and machinetranslated japanese corpora help a bilingual (japanese-english) plm achieve domain adaptation in the japanese medical domain? in terms of the corpus-alone effect, incorporating original english texts (as pmc full-text) is generally more beneficial even for japanesedomain tasks than using machine-translated data, as the model using endominant outperformed that using jadominant (see partial correlation analysis indicated that each task is differentially sensitive to certain corpus compositional statistics, including the specific ratio of japanese to english content (see fig. 3 4 5 6 ). therefore, tailoring a corpus composition for particular downstream tasks by balancing language components facilitates effective knowledge acquisition in practice. indeed, in our case, the highest average score across both japanese (20 tasks) and english (7 tasks) was achieved by the model continued pretrained on enja-balance, even surpassing the enja-hybrid model, which was trained on a larger corpus (see ## conclusions we systematically examined how continued pretraining on japanese and english medical domain corpora-at varying proportions-affects task performance to seek optimal corpus composition for the comprehensive japanese-english medical benchmark. the results suggest that effective crosslingual domain adaptation requires (1) leveraging specialized knowledge from well-resourced corpora, (2) ensuring sufficient coverage of languagespecific expressions in the target language, and (3) using parallel corpora in moderation. these findings highlight the importance of balanced corpus design that accounts for both linguistic diversity and domain-specific terminology, particularly in settings involving a well-resourced source language and a low-resource target language. while grounded in the japanese-english medical context, these insights are broadly applicable to multilingual adaptation of plms across diverse domains.