On Creating an E nglish- T hai Code-switched Machine Translation in Medical Domain (2024)

Abstract: Machine translation (MT) in the medical domain plays a pivotal role in enhancing healthcare quality and disseminating medical knowledge. Despite advancements in English-Thai MT technology, common MT approaches often underperform in the medical field due to their inability to precisely translate medical terminologies. Our research prioritizes not merely improving translation accuracy but also maintaining medical terminology in English within the translated text through code-switched (CS) translation. We developed a method to produce CS medical translation data, fine-tuned a CS translation model with this data, and evaluated its performance against strong baselines, such as Google Neural Machine Translation (NMT) and GPT-3.5/GPT-4. Our model demonstrated competitive performance in automatic metrics and was highly favored in human preference evaluations. Our evaluation result also shows that medical professionals significantly prefer CS translations that maintain critical English terms accurately, even if it slightly compromises fluency. Our code and test set are publicly available https://github.com/preceptorai-org/NLLB_CS_EM_NLP2024.

## introduction medical-domain machine translation (mt) serves as a critical component in enhancing healthcare quality and disseminating medical knowledge. by providing accurate translations of medical research publications, mt enables local medical professionals without english proficiency to overcome the linguistic barrier and have access to more medical academic resources, which are predominantly written in english @xcite . this accessibility is crucial for facilitating continuing medical education, which has been shown to be an effective strategy for healthcare professionals to enhance care quality and patient outcomes @xcite . despite the research in the english-thai mt field, most of the common mt techniques are not yet suitable for the medical domain due to the need for precise translation of medical terminology. translating medical terminology accurately is challenging due to the lack of equivalent thai terms for some english medical keywords. thus, it is understandable why common techniques of machine translation in the medical domain mtsuch as google nmt, no language left behind (nllb) @xcite , gpt-4 @xcite , or gemini-pro @xcite -cannot translate medical keywords precisely, as shown in figure 1 . previous studies have aimed to improve translation accuracy through terminology integration @xcite , yet their application to the thai language and specifi-cally in the medical field remains limited. rather than focusing on enhancing the accuracy of terminology translation, our objective is to preserve medical terms in their original english form within the translated output, thus removing the need to handle terminology translation. this strategy is characterized as code-switching (cs), deviating from conventional monolingual translation practices. apart from reducing the task's complexity, framing this problem as cs is also preferred by medical doctors @xcite . a previous study from the thai covid-19 administrative unit @xcite suggests that medical professionals prefer retaining medical and technical keywords in english, with the rest of the translation in thai, to avoid potential translation inaccuracies. other studies @xcite @xcite also suggest that this phenomenon exists in other languages as well, including arabic, javanese, and spanish. as shown in figure 1 , keeping medical terms in english preserves the original meaning and is preferred by mds. however, cs datasets are usually scarce, preventing researchers from developing a cs translator. several initiatives have been made to address this issue. for example, the lince dataset @xcite is one of the publicly available cs datasets focused on the general domain created to mitigate this problem. additionally, various studies have focused on enhancing cs dataset efficiency through augmentation techniques @xcite @xcite , pre-training techniques @xcite , synthetic cs dataset generation techniques @xcite @xcite . however, these previous works did not focus on medical texts. furthermore, the adaptation of such research to the thai language context remains limited. this, in turn, leads to a significant scarcity of the english-thai cs translation dataset, especially in the medical domain, as shown in in this work, we aim to achieve two objectives: (i) address the data scarcity in medical-domain english-thai mt and (ii) validate our hypothesis that doctors prefer cs translations to monolingual ones in medical contexts. to achieve the first objective, we create a new english-thai cs dataset for the medical domain. our process begins by generating initial cs translations (pseudo-cs) of english medical texts using a widely available monolingual translator. during this translation process, we apply a keyword masking algorithm to preserve key medical terms. we then hire an annotator to post-process and clean a portion of the generated translations, as opposed to doing the whole translation process, to save both time and resources. to achieve our second objective, we conduct comprehensive evaluations using both traditional mt metrics and md evaluations to confirm our hypothesis. this involves evaluating 52 models, including an off-the-shelf translator, large language models (llms), and our fine-tuned cs models based on nllb. furthermore, we assess the translations for factual accuracy and md preference by having mds directly rate them and by distributing preference ranking questionnaires, respectively. our findings reveal that our fine-tuned cs model based on nllb is preferred by mds due to its factual accuracy, even though it achieves a lower score in traditional metrics when compared to off-theshelf translators like google nmt. these results indicate that traditional mt metrics are inadequate for evaluating medical-domain translations and that mds prefer cs translations over monolingual ones. to summarize, our key contributions are: • we propose the first benchmark dataset specifically designed for medical english-thai cs translation. ## main experimental results the full evaluation results for all 52 models on our dataset are presented in human preference. as shown in ## conclusion this paper presents an approach for performing mt in the medical domain using a cs translation to generate translations preferred by medical professionals. we developed a method for generating cs translation data, trained a cs translation model leveraging this data, and evaluated its performance against multiple strong baselines. the experimental results demonstrate that although most automated metrics might be suitable for measuring translation fluency, they are inadequate for assessing factual accuracy or medical doctors' preference in the translations. while current mt technologies may offer monolingual translations with high fluency, medical professionals exhibit a clear preference for cs medical translations that accurately preserve crucial terms in english, even at the expense of fluency.