O pen WHO : A Document-Level Parallel Corpus for Health Translation in Low-Resource Languages (2025)

Abstract: Health machine translation (MT) is a high-stakes domain characterised by widespread deployment and domain-specific vocabulary. However, there is a lack of MT evaluation datasets for low-resource languages in the health domain. To address this gap, we introduce OpenWHO, a document-level parallel corpus of 2,978 documents and 26,824 sentences from the World Health Organization’s e-learning platform. Sourced from expert-authored, professionally translated materials shielded from web-crawling, OpenWHO spans a diverse range of over 20 languages, of which nine are low-resource. Leveraging this new resource, we evaluate modern large language models (LLMs) against traditional MT models. Our findings reveal that LLMs consistently outperform traditional MT models, with Gemini 2.5 Flash achieving a +4.79 ChrF point improvement over NLLB-54B on our low-resource test set. Further, we investigate how LLM context utilisation affects accuracy, finding that the benefits of document-level translation are most pronounced in specialised domains like health. We release the OpenWHO corpus to encourage further research into low-resource MT in the health domain.

## introduction translation in the health domain combines clinical risks, widespread demand, and domain-specific complexity @xcite . by offering a timely and resource-efficient complement to human translation, machine translation (mt) can lower the barrier to disseminating health content, from education materials for local health workers @xcite to public safety information during crises @xcite . however, evaluation of mt in the health domain is hampered by a lack of datasets that cover a wide range of languages, particularly low-resource ones. the tico-19 corpus @xcite stands as a notable exception, yet its focus on covid-19 limits its utility on broader health topics, and its age raises the risk of training data contamination. to address this gap, we introduce openwho, a document-level parallel corpus designed for evaluating health mt. sourced from the world health organization's multilingual e-learning platform, its content is expert-authored, professionally translated, and shielded from web-crawling, thus minimising contamination risk. the corpus covers over 20 languages, nine of which are low-resource, including some with low-resource scripts like armenian, georgian, and sinhala. by focusing on health education, a domain fundamental to local quality of care @xcite , openwho provides a realistic benchmark for a high-impact mt use case. leveraging this new resource, we conduct a systematic evaluation comparing modern large language models (llms) against traditional nmt systems. for llms, we study different context strategies (document-level, sentence-level, etc) and to determine whether the benefits of document-level translation are specific to our dataset, we extend our evaluation to the news and literary subsets of the wmt24++ benchmark @xcite . our main contributions are 1 : ## related work document-level mt document-level mt has long been recognised as desirable, as it allows models to leverage broader discourse for improved coherence and accuracy @xcite . early 1 code: github.com/raphaelmerx/openwho-code dataset: huggingface.co/datasets/raphaelmerx/openwho work with traditional nmt models has shown mixed results, with some studies demonstrating that document-level context can significantly improve translation quality @xcite @xcite , while others questioned whether these improvements stemmed from true contextual understanding, arguing that the context encoder was not modeling discourse but acting as a "noise generator" that improves model robustness, rather than leveraging discourse information @xcite . llms for document-level mt llms, given their ability to process extended contexts, are well placed to benefit from document-level context. @xcite explored contextual translation with llama2-13b on english-german, finding mixed results, where document-level context sometimes providing no performance gains over sentence-level translation. @xcite demonstrated that paragraph-level translation outperforms sentence-level approaches in literary fiction using gpt-3.5, though they have noted that their findings might not generalise to low-resource settings. @xcite finetuned llama3-8b for context-aware translation, showing that increasing context window size yields gains, particularly when evaluated with neural metrics. recent mechanistic analysis by @xcite revealed that llms can be surprisingly "context-insensitive," with smaller models showing limited ability to effectively utilise available context, a finding that may explain varying performance observed in earlier works. gaps remain in our understanding of documentlevel translation for low-resource languages in specialised domains. first, there is a shortage of evaluation data for document-level low-resource machine translation in specialised domains, such as healthcare. second, there has been no systematic analysis of how llms utilise document-level context when translating low-resource languages, particularly in specialised domains where coherence and terminological consistency are required. ## experiments having established an evaluation corpus for document-level low-resource mt in the health domain, we now turn to investigating what models perform best on this dataset, and how context utilisation strategies affect llm performance on this dataset. our experimental design addresses a fundamental tension in document-level translation: while broader context can improve coherence and terminological consistency, it may also introduce noise or lead to error propagation. we work with the following research questions: ## discussion our experiments present three key findings: first, modern llms tend to outperform nmt (e.g. gemini outperforms nllb 54b) on low-resource translation in specialised domains (health with open-who, literary text with wmt24++). second, modern llms translate best at the document-level in specialised domains (health and literary), but the advantage of document-level translation is less clear for smaller models and for the general domain. third, other context-utilisation strategies (e.g. sentence window, document context with one sentence at a time) tend to perform less well than whole-document translation. in specialised domains (health, literary fiction), the larger the llm, the more it benefits from doc-level translation. see tables 8 and 11 for scores per language. ## conclusion in this work, we introduced openwho, a highquality parallel corpus for health mt, with a focus on low-resource languages. sourced from the world health organization's expert-authored materials, it addresses a gap in evaluation resources and provides a benchmark for future research at the intersection of health and low-resource languages. the dataset strengths include (1) the grounding of its source english text in evidence-based who guidance (2) its professional translation into various languages and (3) its availability at both the document and sentence level. however, open-who is language imbalanced (not all courses were translated into all languages), which can limit its comparative value. our experiments demonstrate that modern llms, when provided with full document-level context, outperform traditional nmt models on low-resource translation in specialised domains like health and literature. we found that this advantage is most pronounced for the largest models and diminishes in the general (news) domain, highlighting that the utility of context depends on both model capability and domain complexity. our work underscores the potential of documentaware llms to improve translation quality in highimpact settings, while also revealing the critical need for domain-specific evaluation benchmarks and context-aware translation strategies.