Factors Affecting Translation Quality in In-context Learning for Multilingual Medical Domain (2025)

Abstract: Multilingual machine translation in the medical domain presents critical challenges due to limited parallel data, domain-specific terminology, and the high stakes associated with translation accuracy. In this paper, we explore the potential of in-context learning (ICL) with general-purpose large language models (LLMs) as an alternative to fine-tuning. Focusing on the medical domain and low-resource languages, we evaluate an instruction-tuned LLM on a translation task across 16 languages. We address four research questions centered on prompt design, examining the impact of the number of examples, the domain and register of examples, and the example selection strategy. Our results show that prompting with one to three examples from the same register and domain as the test input leads to the largest improvements in translation quality, as measured by automatic metrics, while translation quality gains plateau with an increased number of examples. Furthermore, we find that example selection methods - lexical and embedding based - do not yield significant benefits over random selection if the register of selected examples does not match that of the test input.

## introduction multilingual communication in clinical settings is often hindered by the lack of quality translation tools for low-resource languages @xcite . building machine translation (mt) systems in the medical domain is challenging: parallel corpora is scarce and mistakes can lead to disastrous outcome @xcite . this challenge is intensified when translating into or from low-resource languages @xcite . traditional neural mt models require supervised training on domain-specific data, which is not feasible for many low-resource language pairs. on the other hand, large language models (llms) possess broad world knowledge through pre-training and can be instructed to perform various tasks. yet, generalpurpose llms, when translating only with instructions (in zero-shot setting), often fail to produce adequate translations for specialized domains @xcite . in-context learning (icl) offers a way to guide llms at inference time by providing a few inputoutput pairs examples as part of the prompt @xcite . unlike fine-tuning, icl does not update model parameters; instead, the model learns from examples on the fly. this approach has gained popularity for low-resource scenarios @xcite , since only a handful of examples (as few as 1-5) can significantly improve performances on a given task. prior studies have explored various strategies to optimize icl for mt @xcite . for instance, selection of examples that are similar to the test input yields better translation output. similarity can be defined lexically (word overlap) or semantically (vector distance), and there is ongoing debate on which is more effective @xcite . recent work has also examined the impact of the number of examples: some found that using up to 5 examples is beneficial @xcite , while others observed improvements up to 8 examples before performance plateau @xcite . additionally, domain match is believed to be important: examples from the same domain as the task can guide the model's lexical and stylistic choices @xcite . however, to the best of our knowledge, icl for mt in the medical domain is yet to be explored. our work aims to provide an evaluation of incontext learning for low-resource medical text translation. in particular, our goal is to quantify the impact of three key factors that may influence translation quality: the number of in-context examples, the register of those examples, and the strategy used to select them. we evaluate how each factor affects translation quality using two automatic metrics, namely chrf++ @xcite and comet @xcite . more precisely, we seek answers to the following research questions (rq1, rq2, rq3 and rq4): rq1: effect of number of examples -does increasing the number of in-context examples improve translation quality? rq2: effect of register -do examples with matching registers yield better translations than mismatched ones? rq3: effect of selection strategy -does semantic similarity (content-based) versus lexical similarity (form-based) selection of examples impact translation quality? rq4: effect of linguistic characteristics -which linguistic characteristics of the in-context examples (corpus-and prompt-level) most strongly influence translation quality? our contributions include: (1) an extensive empirical evaluation on 16 diverse languages (covering african, european and asian languages and dialects) in a medical setting (section 4), (2) a statistical analysis of the different factors, individually and in combination, that contribute to translation quality (3) a linguistic analysis of how corpus and prompts characteristics affect translation quality measured by automatic metrics (section 5). ## related work low-resource mt a low-resource language is typically defined as a language for which limited annotated data, such as parallel corpora or monolingual text, is available for training data-driven nlp models. the lack of resources may apply to text quantity, domain coverage, or availability of evaluation benchmarks @xcite . in mt research, a low-resource language pair refers to a translation direction where parallel corpora are insufficient to train reliable mt systems. this limitation may reflect the absolute size (e.g., <1m sentence pairs), the domain (e.g., biomedical), or the bilingual coverage @xcite . in (bio)medical machine translation, even highresource languages can become low-resource indomain, due to the scarcity of domain-specific aligned corpora (e.g., medical records, cochrane reviews, or medical dialogues) @xcite . icl for mt early work on prompting llms for mt showed that models like gpt-3 can perform translation tasks without fine-tuning, particularly for high-resource languages @xcite . however, for low-resource languages and special-ized domains, zero-shot performance is often weak, motivating research into few-shot prompting techniques @xcite . one factor shown to influence translation quality is the number of examples. for instance, @xcite found improvements up to about five examples, while @xcite reported gains up to eight examples before saturation (translation quality scores plateau). these differences suggest that task-or model-specific characteristics can affect translation quality. this research direction allows us to answer rq1. ## effect of factors in translation quality in this section, we describe the results collected during our experiments. we divide the results following the rq1, rq2 and rq3 from section 1 before comparing the effects of all factors on translation quality. ## linguistic analysis based on the previous results, which showed that the domain and register of the examples have the strongest impact on translation quality, we conduct a linguistic evaluation to determine which linguistic characteristics from the examples have the most influence on translation quality (rq4). because our goal is to assess how domain and register alignment shape translation quality, we select three corpuslevel metrics that capture complementary aspects of similarity in texts: to perform this analysis, we included additional corpora for n-shot sampling to provide more data for corpus-level analysis: translations of documents related to public health and disease prevention across different languages within the european union (ecdc, european centre for disease @xcite ; diverse datasets created during the covid-19 period, including a set of wikipedia documents related to health (wikipedia health); a database containing european union law and other public documents generated during covid-19 foot_4 ; and tico-19 for non-european languages @xcite . finally, we included a general-domain corpus, tatoeba foot_5 . while several of these texts share the same domain (medical), they differ in register, ranging from policy documents (eu public documents) to encyclopedic health texts and public health advisories. all corpora were extracted using the opus platform @xcite . we first examine corpus-level characteristics to understand how to select sample pools for icl and to determine which aspects of domain and register are the most impactful when choosing a sample in the next section, we describe the results collected during our experiences to answer rq4, divided by corpus and prompt levels of analysis. ## conclusions this study shows that, in multilingual medical machine translation, the domain and register of incontext examples are the most influential factors affecting translation quality. partial η 2 analysis confirms that aligning the n-shot register and do-main with the test input yields substantially greater improvements than increasing the number of examples. in practice, a small, well-chosen set of domain-relevant shots often yields higher translation quality scores than a larger set of examples sampled from other domains or registers. sentence-level analysis of lexical, syntactic, and semantic similarity confirms that the most predictive features vary by language. in low-resource language pairs, all three similarity types correlate strongly with translation quality, while in higherresource languages pairs such as english to czech and french, syntactic and semantic similarity dominate. semantic similarity is the most consistent predictor across languages. these results suggest that prompt engineering for icl should prioritise register and domain alignment, and adapt exemplar selection criteria to the characteristics of the language pair rather than applying the same similarity heuristics.