Investigating the Medical Coverage of a Translation System into Pictographs for Patients with an Intellectual Disability (2022)

Abstract: Communication between physician and patients can lead to misunderstandings, especially for disabled people. An automatic system that translates natural language into a pictographic language is one of the solutions that could help to overcome this issue. In this preliminary study, we present the French version of a translation system using the Arasaac pictographs and we investigate the strategies used by speech therapists to translate into pictographs. We also evaluate the medical coverage of this tool for translating physician questions and patient instructions.

## introduction many people around the world face difficulties to communicate through speech. to overcome this challenge, disabled people, including persons with an intellectual disability (id), resort to augmentative and alternative communication (aac) systems in different forms: objects, visual aids on paper or technologies @xcite . both text and pictographs can be used in aac for enhancing the communication and the social inclusion of individuals with id. images are already used in various medical contexts to increase access to information and communication for all, e.g. in pharmacology on drug leaflets for improving the health literacy or in hospitals for facilitating the medical tourism @xcite . in emergency settings, there are tools such as interpreters and communication technologies for allophones patients @xcite . however, they have drawbacks and are not designed for people with disabilities. recent research focuses on medical applications with images for disabled patients, but does not use nlp techniques @xcite , such as my symptoms translator @xcite or the system of @xcite . this paper focuses on the french version of a translation system using the arasaac pictographs foot_0 @xcite and makes two contributions: investigates the strategies used by speech therapists for translating medical sentences into pictographs and evaluates the lexical coverage of the system. section 2 summarizes related work on the translation systems with images and their evaluation. we describe our system in section 3. section 4 investigates translation strategies. finally, we evaluate the medical coverage of our system in section 5. ## related work pictographs represent one or several concepts: object, verb, feeling, grammatical word, etc. there are several pictograph sets available, such as sclera, beta or arasaac, which are specifically created for people with various disabilities. they can be seen as simplified languages @xcite . therefore, they have been used within aac systems, but many relate to daily communication. as regards the medical language, glyph (bui et al., 2012) -which is not an aac applicationautomatically translates patient instructions from text into pictures with nlp and computer graphics techniques. the babeldr system @xcite proposes pictographs and allows to translate spoken medical utterances in various languages to communicate with migrants and deaf patients in hospitals. for people with id, de @xcite designed a tool with pictographs, called stop-id, to aid the self-reporting of pain (affect, location, intensity and quality). the authors also tested the ability to recognize representations for vocabulary and pain of their tool in adults with id. the comprehension of single pictographs in context is increasingly evaluated with users: e.g. for the patient responses to medical questions @xcite . however, most studies still rely on automated metrics used in mt such as bleu @xcite , nist @xcite , etc. to assess sentences automatically translated into pictographs @xcite @xcite . @xcite tested sentences whose nouns and verbs had been automatically translated into pictures. finally, some evaluations are also carried out by researchers, such as in @xcite , which rated the correctness of 49 patient instructions converted with glyph. more recently, @xcite manually evaluated the comprehension and the lexical coverage of sentences generated into three pictographic languages by their translation system. ## translation system our system was originally designed for the online communication of people with id and was hence optimised for social media context @xcite . it translates texts written in four natural languages into any combination of four pictograph sets. this paper focuses on french and the medical domain with arasaac pictographs (see section 4), as there are fewer medical pictographs in the other sets. the text to translate first undergoes shallow linguistic analysis: sentence detection, tokenization, pos-tagging and lemmatization with treetagger @xcite , simple detection of multi-word expressions (mwe), processing of specific french phenomena based on rules and dictionaries @xcite . then, each word of the text can be translated through two routes: the semantic route and the direct route. in the semantic route, each word is looked up in the wolf database @xcite , a french version of wordnet @xcite . if it is not found, hyperonym and antonym relations of wolf are used to get substitute translation. for example, as there is no pictograph for saumon (salmon), the word is translated by its hyperonym poisson (fish). the word infecter (infect) does not have a pictograph and is translated by its antonym followed by the negative pictograph, désinfecter non (desinfect no). for the direct route, we build a dictionary for the pictographic language that contains the words not covered by wolf (e.g., prepositions, pronouns, etc.). pictograph filenames (i.e. french lemmas) are linked to their identifiers available on the arasaac website. to choose the optimal path while converting a sequence of lemmas to a sequence of pictographs, we use a search algorithm a* @xcite . compared to our previous work @xcite , various improvements were brought to our system. we updated our pictograph database with new pictographs from arasaac api, foot_1 as more medical pictographs have been added due to the covid pandemic. several aac systems with pictographs use a color coding system that informs about the syntactic category of the words represented. this makes it possible to improve the learning of vocabulary and therefore its use. we implemented the coding system of @xcite , which highlights the borders of pictographs with colors, depending on their pos: green for verbs, blue for adjectives, etc. at the beginning of sentences, we also generate a temporal pictograph for past and future tenses (see figure 1 ), as @xcite . we added a wolf relation: eng_derivative, to get similar concepts with a different pos tag, e.g. for the adjective respiratoire (breathing), our system translates it by the verb respirer (breathe). it is the equivalent of xpos_near_synonym relation in other wordnets. we will therefore call it the xpos relation in section 5. finally, we added simple rules of compression for different french phenomena found in our previous evaluation (deletion of some function words, auxiliaries, verb-subject inversion in questions, simplification of some imperative structures for patient instructions). these rules are based on an analysis of our system's output, the advice from a speech therapist and a syntactic analysis carried out on medical sentences from the babeldr system with the berkeley neural parser @xcite ). foot_2 ## translation strategies for pictographs as we aim to automatically translate physician questions and patient instructions into arasaac pictographs, we ran into the issue of the lack of a large authentic medical corpora in pictographs built by aac users and the lack of translation guidelines to create such as a corpus. therefore, we have investigated the actual strategies used by speech therapists to carry out such translation. this section first describes the data to translate, then lists the different translation strategies observed. ## preliminary evaluation we present the system tuning and an automated evaluation (section 5.1), before the manual evaluation to assess the medical coverage (section 5.2). ## conclusion we proposed an original way to investigate the medical coverage of our translation system from french into arasaac pictographs using umls concepts. we also discussed the translation strategies into pictographs for medical sentences. 7 there is room for further improvement to specialize this system to the medical dialogue between physician and patients with id. some linguistic phenomena are not yet taken into account in the french system, such as the word sense disambiguation. both the pictographic representations and the sentence comprehensibility into pictographs by the target users would need to be further investigated in context.