Instruction-tuned Large Language Models for Machine Translation in the Medical Domain (2025)

Abstract: Large Language Models (LLMs) have shown promising results on machine translation for high resource language pairs and domains. However, in specialised domains (e.g. medical) LLMs have shown lower performance compared to standard neural machine translation models. The consistency in the machine translation of terminology is crucial for users, researchers, and translators in specialised domains. In this study, we compare the performance between baseline LLMs and instruction-tuned LLMs in the medical domain. In addition, we introduce terminology from specialised medical dictionaries into the instruction formatted datasets for fine-tuning LLMs. The instruction-tuned LLMs significantly outperform the baseline models with automatic metrics. Moreover, the instruction-tuned LLMs produce fewer errors compared to the baseline based on automatic error annotation.

## introduction current state-of-the-art large language models have shown promising results in machine translation for high-resource language pairs and domains @xcite . however, in lowresource domains (e.g. medical) llms have shown lower performance compared to standard neural machine translation (nmt) models @xcite . the accuracy and consistency in the machine translation of terminology, syntax, and document structure is crucial for users, researchers, and translators who post-edit machine translated documents in highrisk domains @xcite . moreover, the introduction of in-domain translation constraints during generation into neural models is currently an open problem @xcite @xcite . llms are trained to perform different natural language processing (nlp) tasks such as summarisation, question answering, and translation, where users interact with the models via instructions (e.g. chat interface) @xcite @xcite . instruction-tuning is a technique that leverages datasets from different nlp tasks, structured as prompts, for fine-tuning llms to enhance generalisation across novel tasks and domains @xcite . for example, in machine translation (mt), translating a segment from the european medicines agency corpus with specialised terminology the prompt can be framed as: "glossary: medicine -> medicamento. translate the source text from english to spanish following the provided translation glossaries. english: the medicine was effective in patients with all three types of homocystinuria. spanish: ". moreover, @xcite instruction-tuned llama-2 @xcite to perform translation related tasks, such as segment and document level translation, post-editing, terminology aware translation, and error annotation. the controlled generation of mt output with the correct terminology, segment length, or syntax can be framed as an instruction-tuning task for llms. thus, improving the workflow of translation during post-editing with an instruction-following (i.e. chat) interface for an llm tuned on a specific domain. we seek to answer the following research question: does instruction-tuning based on terminology rules improve translation quality on llms? in this paper, we show results for adding specialised medical dictionaries into fine-tuning for llms. in particular, we follow the methodology from @xcite by incorporating terminology information into the instruction-tuning datasets. unlike @xcite , our approach relies on openly available medical dictionaries and employs simple heuristics to construct instruction-tuning datasets. an instruction-based interface could facilitate the interaction between professional translators and llms, and enables model customisation via the integration with user-defined terminology dictionaries. our contributions are as follows: we use parameter-efficient fine-tuning (peft) and quantisation of large language models (llms) for indomain translation. we leverage medical dictionary term pairs with parallel data to construct prompts that guide llms in translating specific terminology. we evaluate flan-t5 (chung et al., 2022), llama-3 @xcite for english-spanish, english-german, and english-romanian language pairs in a split of a medical domain dataset. the instruction-tuned models outperformed the baseline models with the automatic metrics bleu @xcite , chrf @xcite , and comet @xcite . moreover, instructiontuning improves the overall accuracy of the terminology. finally, we evaluate the two best models with automatic error annotation @xcite , and quality estimation @xcite . ## background and related work auto-regressive language models predict the next token in a sequence given a prefix context @xcite , where llms are pretrained with large amounts of texts followed by fine-tuning on different downstream tasks @xcite . @xcite propose to fine-tune llms with a mixture of several nlp datasets into an instruction format to improve: generalisation to unseen tasks, and generation given instruction prompts. for a machine translation task, the llm is conditioned on a user defined prompt that consists of a translation instruction along with the source text to be translated @xcite . during testing, zero-shot prompting involves querying an llm with a test input that was not present in the training data. for example, mt instruction includes a prompt asking to translate from a source language to a target language, and the corresponding source text. however, few-shot prompting provides a few examples of the translation task along with the test input to guide the llm generation. in mt, few-shot examples consist of parallel source and human-translated sentences. supervised fine-tuning (sft) is one of the most popular techniques for domain adaptation in llms, where models continue their training with a sample of in-domain data @xcite . however, sft for llms requires large amounts of computational resources, given that during training models update billions of parameters. the goal of parameter-efficient finetuning (peft) is to update (i.e. tune) a minimal set of parameters to achieve a similar performance compared to full sft on downstream tasks. @xcite propose low-rank adaptation (lora) that freezes all the pre-trained model parameters and adds adapter trainable low-rank decomposition matrices of parameters into each layer of the model. moreover, @xcite propose that during fine-tuning to quantise the parameters of the pre-trained model into fewer bits (e.g. 4-bit) and keep the lora adapters with standard precision, thus reducing the memory usage. peft and quantisation with qlora enables academic translation practitioners to fine-tune llms with limited computing resources. llama versions 2 and 3 @xcite are open-source llms with different parameter scales, which are instruction-tuned for multiple natural language processing tasks. moreover, llama has become the base model for the mt related work @xcite @xcite . @xcite compared 15 baseline llms and fine-tuned with qlora on different mt tasks (e.g. segment and document level translation) for the french-english language pair. llama-2 outperformed other llms, fine-tuning improves performance on models that struggle on a few-shot setup, and qlora is potentially superior to full sft in terms of efficiency. alves et al. (2023) compared instruction tuning with lora to fewshot prompting using llama-2 in various language pairs. fine-tuning outperforms the few-shot learning, is comparable to full sft, requires few training data, and tackles over generation. however, llms struggle with translation directions out of english (en-xx). @xcite proposed tower with a focus on translation related tasks, for example, document level translation, post-editing, and terminology-aware prompts. tower is based on the continued training of llama-2 with parallel translation data, and is followed by instruction-tuning for the mt tasks. zheng et al. (2024 ) proposed to fine-tune llms based on prompts, and compared it to lora for domain adaptation in it for chinese-english and english-chinese mt. moreover, zheng et al. (2024) incorporate it terminology by few-shot prompting and chain-of-thought. the template used for the proposed prompt-tuning model has a substantial impact on performance, and the introduction of terminology with simple prompt rephrasing outperforms chain-of-thought. eschbach-dymanus et al. (2024) studied domain adaptation for business it with llms. they compared full sft, lora, different prompting techniques, and standard nmt. finally, eschbach-dymanus et al. (2024) defined guidelines for domain adaptation with llms. moslem et al. (2023) evaluate llms for translation on specialised domains (e.g. medical covid-19), and incorporate terms from glossaries into their prompts to tackle issues with no retrieved matches in few-shot learning. jerpelea et al. ( 2025 ) developed a parallel dataset for the low-resource languages romanian, and aromanian, they instruction-tuned llama-3 for romanian, and compared multilingual nmt, gpt, llama-3, and tower for translation. we followed @xcite for our experimental design. unlike previous work on llm for mt, our approach focuses on the medical domain, relies on openly available medical dictionaries, employs simple heuristics to construct the instruction-tuning datasets, and uses efficient tuning techniques. in particular, we evaluate the impact of instruction tuning for improving terminology translation in llms. ## results we show results with automatic metrics and terminology accuracy for flan-t5, llama-3 and tower for the en-es, en-de and en-ro language pairs. moreover, we show automatic error annotation, and quality estimation scores for the best performing models. ## conclusions and future work in this study, we show a comparison between baseline llms and qlora instruction-tuned models in the medical domain for en-es, en-de, and en-ro. we introduce medical terminology from iate into an instruction-formatted dataset for controlled generation in llms. instruction-tuned models significantly outperform the baseline across automatic evaluation metrics. furthermore, these models show improved accuracy in terminology translation compared to the baseline. in particular, the instruction-tuned tower model presents superior translation quality according to different evaluation methods (automatic metrics, mqm annotation, and qe). additionally, tower requires fewer computational resources and less post-processing compared to llama-3. a limitation of our current evaluation is the reliance on automatic metrics and the limited quality of automatic error annotation. for future work, we will evaluate the baselines with few-shot instead of zero-shot. we will define different prompts for llama-3 to avoid over-generation. we will perform an evaluation on a balanced test split in terms of the number and type of present terms with respect to the training data. finally, we will perform a manual error annotation, as automatic metrics may not test for correct terminology generation on the mt output @xcite .