A Neural Machine Translation Approach to Translate Text to Pictographs in a Medical Speech Translation System - The B abel D r Use Case (2022)

Abstract: The use of images has been shown to positively affect patient comprehension in medical settings, in particular to deliver specific medical instructions. However, tools that automatically translate sentences into pictographs are still scarce due to the lack of resources. Previous studies have focused on the translation of sentences into pictographs by using WordNet combined with rule-based approaches and deep learning methods. In this work, we showed how we leveraged the BabelDr system, a speech to speech translator for medical triage, to build a speech to pictograph translator using UMLS and neural machine translation approaches. We showed that the translation from French sentences to a UMLS gloss can be viewed as a machine translation task and that a Multilingual Neural Machine Translation system achieved the best results.

## introduction patients, especially those with limited health literacy skills, often have trouble understanding health information. one of the ways in which medical communication can be facilitated is through the use of pictographs. in particular, pictographs have been used extensively to deliver specific medical instructions. the use of images has been shown to positively affect patient comprehension by improving attention, recall, satisfaction, and adherence @xcite . although the potential of pictographs has often been recognised, tools that automatically translate sentences into pictographs are still very scarce due to the lack of resources, which also impedes evaluation in this domain. glyph is an automatic healthcare data processing system that automatically converts texts into sets of illustrations using natural language processing and computer graphics techniques @xcite . the system is based on 600 pictographs that are linked to unified medical language system (umls, @xcite ) and has been shown to have a positive impact on information recall, satisfaction, and the understandability of instructions @xcite . some online medical translators also include pictographs, for example, "my symptoms translator" @xcite and "medipicto ap-hp", but they remain very limited in coverage and can only translate predefined sentences. there are generic mt sys- ## system settings in this study, we experiment with two different systems trained on the synthetic data with umls glosses. we compare them to a baseline trained on the variations -core sentences data. in this baseline, the system translates french sentences into a core sentence and uses the grammar to generate the umls gloss, as in the current babeldr architecture (see more, @xcite . in this section, we explain the settings for the systems. ## evaluation methodology the aim of the system is to produce a gloss that has the same meaning as the sentence and the right syntax, so that it can produce the expected pictographic form. we carried out an automatic and human evaluation to assess the systems' performance on real medical dialogue ## conclusion the aim of this paper was to propose an architecture to automatically translate doctor's interactions into pictographs. different generic systems exist, but they are not specialised for the medical domain. the proposed method contains two steps: first, sentences are translated into a umls gloss based on synthetic data and secondly, concepts are mapped to pictographs. the aim of the umls gloss is to characterise the pictographic language, i.e. both the medical concepts and the syntax. this paper focused on the first step. our contributions are twofold. on one hand, we confirmed our hypotheses and showed that mapping to the umls gloss can be seen as a mt task and that nmt models directly trained with umls glosses achieved higher f-scores. the resulting system is still limited in coverage, but results are encouraging, since 30% of out of domain utterances are translated into a potentially useful umls gloss. the human evaluation has also shown that the proposed umls gloss is readable by humans and can characterise pictographic language. on the other hand, this work is the first of its kind and constitutes an initial step in constituting resources (corpus and umls to pictographs database) and testing the impact of pictographs on medical communication. the synthetic data used for training the systems to translate into umls gloss are available upon request. in the future, we plan to build upon this work in different directions. first, we plan to build a pictographic database that links umls concepts to pictographs, based on existing open source pictograph databases, such as arasaac and santébd and other resources for the illustration of concepts (e.g. babelnet). a preliminary study shows that only 69.7% of the babeldr umls concepts can be linked to at least one arasaac pictograph. for medical dialogues, the main problems include the representation of generic words (disease, infection, inflammation), the name of diseases (diabetes, syphilis, etc.) and temporal elements. this step will allow us to build a speech-to-pictographs baseline system for medical dialogues and will allow experiments to be carried out on the medical dialogue task itself. more nmt architectures will be tested. medical dialogues contain a lot of incomplete sentences (ellipsis), as explained in @xcite . in the current version of babeldr, the translation is performed in context (with the previous sentence of the dialogue) when an ellipsis is identified. contextual nmt can also be used when translating into the umls gloss. umls semantic type and babeldr human translations in other languages can also be added as another language in the multilingual system. the umls can also be used in the current architecture as an interlingua to improve translation into low-resource languages @xcite . one of our objectives is to produce more training data with more core sentences. we can easily change the synthetic data to include new core sentences. in the current version, each new grammar rule has to be translated into the 9 babeldr target languages @xcite . the grammar therefore often groups together quasi-paraphrases, as shown in figure 2 to reduce human translation effort (for example, "avez-vous de la fièvre" (do you have fever) and "avezvous de la fièvre maintenant" (do you have fever now) are mapped to the same core sentence, based on the assumption that they allow doctors to collect the same anamnestic information. these rules may be split in order to only include exact paraphrases. other resources will also be added, based on existing hug terminological resources. finally, the selection of training data from the data generated by the grammar is based on n-grams in the source language @xcite . we can try to select the training data based on umls n-grams. our test corpus also contains a high number of in coverage sentences, since it was collected with the babeldr tool. we are in the process of collecting more data.