An Evaluation of E nglish to S panish Medical Translation by Large Language Models (2024)

Abstract: Machine translation (MT) with Large Language Models (LLMs) holds promise as a clinical translation tool with more capabilities than a traditional MT model. This work compares the quality of English to Spanish translation by three LLMs: ChatGPT3.5 Turbo, ChatGPT4o, and Aguila, against Google Translate. The test set used in this study is MedlinePlus, a parallel dataset of educational health information in English and Spanish developed by the National Library of Medicine. ChatGPT4o and Google Translate performed similarly in both automated scoring (BLEU, METEOR, and BERTscore) and human evaluation with ChatGPT3.5 Turbo not far behind. Aguila, the only LLM intended for primarily Spanish and Catalan use, surprisingly performed much worse than the other models. However, qualitative analysis of Aguila’s results revealed the use of Spanish word choice that may reach a broader audience.

## introduction it is well-understood that in today's increasingly diverse america, the healthcare field must overcome barriers of discordant races, ethnicities, cultures, and languages to deliver high-quality care to all patients. according to the 2020 census, approximately 8.3% of american residents speak english less than "very well" (us census bureau, 2020). metropolitan areas are disproportionately home to a large number of immigrants (29% in new york city), many of them not proficient in english (profile of the foreign-born population in new york, new york, 2023). urban settings experience amplified disparities in care for underserved populations, including immigrants, refugees and limited-english proficiency, or lep, patients. lep status has been linked to greater health disparities (e.g. via poorer preventative screening) and worse health outcomes @xcite @xcite . in such cities, medical education institutes and academic health centers are a crucial form of advocacy and social justice that address disparities through service-learning mechanisms @xcite . for instance, several medical schools operate student-run free clinics (srfc) for uninsured patients in their communities. however, language and medical literacy barriers in these patients present a challenge for trainees to ensure their patients, who often have many chronic conditions, understand their diagnosis, medication regime, necessary lifestyle changes, specialist referrals, etc. @xcite . for srfcs and other healthcare settings that deal with a large number of lep patients, artificial-intelligence (ai) or machine translation (mt)-based solutions present a potential low-cost, convenient, and efficient tool to address language barriers in patients. however, maintaining accurate, patient-friendly translations without compromising medical accuracy is a limiting factor of such translation services. thus, phone interpretation services such as cyracom and pacific interpreters remain the standard practice for communicating with lep patients. unfortunately, phone interpretation itself is limited by potentially poor acoustics, lack of visual cues, and lack of context provided to the interpreter @xcite . large language models (llms) are deeplearning algorithms that are trained to accomplish various natural language processing tasks like text classification and text generation, among others. a chatbot, like chatgpt, is a system which has been optimized for conversation with a user. chatgpt, along with other llms, is reported to able to use multiple languages, and has anecdotally been reported as an effective translator between languages @xcite . however, there has not been a formal study looking at the medical translation capabilities of llm chatbots that were not formally trained for translation versus dedicated machine translation algorithms, like google translate (gt). the goal of this research was to quantify the effectiveness of various chatbot llms for translating health information from english to spanish in a patient-friendly manner. at a preliminary stage, we are evaluating the potential of four models with multilingual capabilities to provide translations of english take-home instructions and clinical information into spanish. ## related work automated medical translation has benefited from research in the fields of both machine translation and llms. ## novel contributions to our knowledge, this is the first work that examines the most recent chatgpt model, gpt4o, on medical translation from english to spanish. this work also contributes to the literature by comparing neural mts with translation by llms using commonly used automated scoring metrics, and newly applies these metrics to evaluate patientprovider communication. finally, our study looks at llms developed with primarily english usage compared with one llm that is intended for spanish chat. our qualitative analysis finds small yet important differences in the spanish word choice among different models and highlights areas where medical mts fall short. ## methods in this section we will discuss the selection and cleaning of the dataset, methods applied for automated scoring and human evaluation, and the models tested and corresponding prompts. ## results the results of mt translation will be presented first as automated metrics, human evaluation metrics, and qualitative analysis respectively. following will be an analysis of human evaluation quality. patient-friendliness was a new metric defined in this paper to capture how understandable a medical translation is for the general patient population. this metric is especially important for medical translation where medical terminology provides a unique challenge and patient understanding is especially critical. ## conclusion this study sought to quantify the translation capabilities of llm chatbots like gpt3.5, gpt4o, and aguila for use in healthcare contexts. these models are not specifically designed for translation, but have capabilities that typical mts lack, such being tasked with translating for a particular target audience. to our knowledge, this is the first study to employ automated evaluation metrics to translate a large test set representing clinical patient-provider communication. this is also the first to evaluate the newest chatgpt model, gpt4o, in this manner and context. this work's findings confirm that the widely accessible llm chatbots gpt3.5 turbo and gpt4o indeed have medical mt capabilities on par with gt to translate clinical communication from english to spanish. they hold promise for use in a variety of healthcare settings, from creating public health education texts to explaining physical examinations and inquiring about symptoms to providing take-home patient instructions. the spanish chatbot aguila was less successful at translating from english to spanish, although when successful, its spanish lexicon was much more conversational and accessible than the other models. further studies should seek to evaluate llm chatbots' performances at completing various clinical translation tasks in a real clinical setting, as well as explore more translation prompts.