Evaluation of intralingual machine translation for health communication (2024)

Abstract: In this paper, we describe results of a study on evaluation of intralingual machine translation. The study focuses on machine translations of medical texts into Plain German. The automatically simplified texts were compared with manually simplified texts (i.e., simplified by human experts) as well as with the underlying, unsimplified source texts. We analyse the quality of outputs from three models based on different criteria, such as correctness, readability, and syntactic complexity. We compare the outputs of the three models under analysis between each other, as well as with the existing human translations. The study revealed that system performance depends on the evaluation criteria used and that only one of the three models showed strong similarities to the human translations. Furthermore, we identified various types of errors in all three models. These included not only grammatical mistakes and misspellings, but also incorrect explanations of technical terms and false statements, which in turn led to serious content-related mistakes.

## introduction in germany, according to recent studies in the field of public health, over half of the population reports having difficulties with health-related topics @xcite . for that reason, the promotion of health literacy (knowledge and compe-tences to access, understanding, appraise and apply medical information) has turned into an important task for the german health system @xcite (for an extensive definition of health literacy, see @xcite ). in this context, recent research has underlined the need for accessible communication in the medical domain to effectively promote health literacy and consequently assist patients navigating the health system and improve patient understanding, engagement and compliance with medical recommendations @xcite @xcite . plain german is a prominent form of accessible communication that has gained relevance in health communication scenarios @xcite . although there is an urgent need for translations into plain german, there is also a gap in qualified and experienced human translators @xcite . moreover, there is a lack in computer-aided translation (cat) tools and machine translation systems for this kind of intralingual translation. unfortunately, little is known about existing systems and their performance for different texts that are required to be translated into plain german, as for instance, texts in health communication that we focus on. in our study, we evaluate machine translations of medical texts into plain language. the source texts, as well as reference human translations, are derived from the website of the german health magazine apotheken umschau. we analyse machine-translated texts produced with three models comparing them with human translations from the magazine's website. besides that, we compare all translations with the underlying sources. in the following, we present the results of the qualitative and quantitative analysis. section 2 de-scribes related work. in section 3, we present our research design including corpus and the methods used. section 3.3 presents the results of our analyses, while we discuss those results, as well as limitations and possible extensions in the future in section 4. ## discussion and future work the present paper evaluates three different models of a machine translation system for translating medical texts into plain german. it covers one of the first steps towards the implementation of these tools in accessible health communication in germany and it discusses first methodological ap-proaches, which we intend to expand on in further research. model 2 seems to achieve the best results according to most criteria. at the same time, model 1 seems to be more similar with the human translations at hand. while in terms of syntactic complexity and text readability, the models yielded promising results, the evaluation of the correctness revealed severe misinformation for all three models, the consequence being that the texts cannot be safely used by end users. at the same time, the tool under analysis can be used as a cat tool for professional translators and content providers with an expertise in plain language and post-editing. as our study has clearly revealed that so far machine translated plain language texts cannot do without post-editing, but need intensive revision, professional post-editing competences are more important than ever. this means that translators and experts working on machine translated text must be trained to detect and correct different types of errors, especially those that are critical for user safety. in further steps, a guide for post-editing in intralingual translation will be developed, exposing the necessary competences and factors to be considered when using machine translation into plain german. in a next step, the machine translation system will now be integrated into the editorial workflow of the apotheken umschau on a trial basis. this practice test will serve to assess the time and effort that is needed to post-edit the machine translated texts. adding machine translation to the editorial process could optimize the process and addresses the gap between the need for texts in plain german and the lack of professional translators (see section 1). the metrics from the practice test will be particularly interesting because the tool will only be permanently integrated into the workflow if the time and effort for post-editing the output is lower than for translating from scratch. therefore, these metrics will determine the final decision for or against adapting the status quo. in our future research, we will conduct a thorough analysis and classification of the various error types found in the machine translated texts. for example, we plan to investigate specific linguistic phenomena, such as the translation of compound words. in addition, we also want to test and compare the output from both summ ai and other state-of-theart systems to investigate which of the currently available systems is most suitable for intralingual translation into plain german, in general and for specific subjects and text types. these systems include both freemium tools that offer both free and paid plans, such as chatgpt and google gemini, and commercial tools, such as klartext st. pauli, capito digital and t2k (text2knowledge). in these future studies, we also plan to use other text types and texts from other domains, so that we are able to compare not only different tools but also different datasets.