Extending a Text-to-Pictograph System to F rench and to Arasaac (2021)

Abstract: We present an adaptation of the Text-to-Picto system, initially designed for Dutch, and extended to English and Spanish. The original system, aimed at people with an intellectual disability, automatically translates text into pictographs (Sclera and Beta). We extend it to French and add a large set of Arasaac pictographs linked to WordNet 3.1. To carry out this adaptation, we automatically link the pictographs and their metadata to synsets of two French WordNets and leverage this information to translate words into pictographs. We automatically and manually evaluate our system with different corpora corresponding to different use cases, including one for medical communication between doctors and patients. The system is also compared to similar systems in other languages.

## introduction augmentative and alternative communication (aac) is used by disabled people to help them to communicate in daily life and to be more independent in their interactions with others @xcite . as such, aac technologies also improve the social inclusion of disabled people, including those with an intellectual disability (id), which are the focus of this paper. one of the characteristics of aac is to represent the natural language in the form of pictures or pictographs, to support the communication of people with language impairment. several sets of pictographs have been designed specifically for people with an id to express their basic needs to their family, friends, teachers, or healthcare professionals. pictures can be used by these persons for communication in various situations such as for social media @xcite , access to school @xcite , or in the (pre-)hospital setting @xcite . recent research focuses on applications with pictographs for medical settings and people with or without disabilities such as foreigners and allophone patients. this is the case for the smartwatch prototype of @xcite and the ba-beldr project @xcite . current systems are often limited and do not always use nlp techniques, especially the applications available for the general public. my symptoms translator @xcite and medipicto ap-hp foot_0 are examples of mobile app for medical communication with images between doctors and patients. this article focuses on the text-to-picto system, which automatically translates text into pictographs for people with an id @xcite . the system was originally designed for dutch, and later extended to english and spanish @xcite . in this work, we adapt it to french. in addition, we extend the system by linking it to a third pictograph set, namely arasaac foot_1 . until now, two pictograph sets had been used in text-to-picto: sclera foot_2 and beta foot_3 , but adding arasaac was relevant in view of its growing popularity and coverage (more than 15,000 coloured pictographs, which are specifically designed for people with an id). this paper first refers to some related work (section 2), before introducing the methodology used to adapt text-to-picto to french (section 3). then, we automatically and manually evaluate the french translation system with the three pictograph sets, using three corpora corresponding to different use cases of aac. results are also compared to those of similar systems (section 4). finally, we discuss the different evaluations of the system (section 5). ## 2 related work in this section, we present some work about text-topictograph translation systems integrating various nlp techniques for different languages. in their translation system, @xcite used the wordnet resource @xcite , but without exploiting the relations between concepts. in addition, their system aimed to translate only the content words (nouns and verbs). the glyph automatically translated patient instructions using nlp (e.g. preprocessing stage, including sentence splitter, word and synonym normalization, etc.), terminology or medication databases, but also computer graphics techniques @xcite . this application was not designed or tested with disabled people unlike the work of @xcite and @xcite on the dutch text-to-picto system, later extended to english and spanish in the framework of the able to include project. the text-to-picto system had a certain success: @xcite integrated it in an assistive mobile application for travel and communication in russian, intended for people with an id, whereas @xcite adapted it for indian languages. other systems were also developed recently such as aratraductor, an application for spanish using nlp to improve its pictograph translations @xcite or the system of @xcite for english, which uses wordnet and imagenet @xcite . for french, @xcite implemented a speech-to-picto tool with an automatic speech recognition module. it includes a system to automatically translate text into arasaac pictographs, the set of images for aac used for this study. based on the work of @xcite , @xcite also evaluated their prototype by testing word sense disambiguation and automatic simplification for the passive structures or the deletion of some grammatical words. all these techniques were already tested by @xcite , but only for translation from dutch. @xcite developed a rule-based module for simplification of twelve syntactic phenomena (relative clause, non subject-verb-object order, etc.), including compression. for pictograph translation, @xcite also used a dutch word sense disambiguation tool. it is worth mentioning that for the french language, there are not many large resources (e.g. sense-annotated corpus) -compared for example to english -for these nlp tasks. ## methodology in this section, we describe how we automatically linked the pictographs to lexical-semantic resources such as wordnet (section 3.1). then, we present the nlp architecture of the text-to-picto system (section 3.2), followed by a description of our use cases (section 3.3). ## results this section presents how we tuned the system (section 4.1) and describes the results of the automated evaluation (section 4.2), followed by the manual evaluation (section 4.3). ## discussion as regards the efficiency of the different pictograph sets, tested on the email corpus, we see that our results are better for the beta set than for the sclera set. @xcite explained that beta contains less pictographs than sclera. as a result, more paraphrasing translations are possible in sclera, resulting in a less accurate measurement of translation quality by bleu. however, our scores for arasaac, the largest set of pictograph, are the best. compared to others, this set includes more function words (articles, prepositions, etc.) that have been encoded in our dictionary. therefore, they will always be well translated, which improves the results. it should also be mentioned that, despite relatively good results, some pictographs generated by our system are not also easily comprehensible, depending on the context of the sentence. this is especially the case for function words and pain description on a specific body part. in addition, medical words are not always translated because there is no corresponding pictographs, e.g. symptôme (symptom), cancer (tumor), etc. our experiments also aimed at assessing the contribution of different components to our text-to-picto system for french. using only the dictionary clearly yields unsatisfactory results. the system improves when we add the synonym or the relation component, regardless of the pictograph sets and corpora. however, the difference between both components appears marginal. in our reference corpora, a manual inspection of the data reveals that the relation of synonymy is much more frequent than those of hyperonymy and antonymy -which is very rare -, especially for the largest arasaac pictograph set. translating text into pictographs is a meticulous and time-intensive process @xcite . this explains why the corpora are small. it is worth mentioning that the bleu score is very dependent on the reference translation, which may be partially subjective. finally, as explained above, every time we run the french system with the same parameters, we get slightly different translations. this happens when the optimal path calculation step has to choose randomly between several pictographs that have an equal weight, using wordnet. for future work, it would be possible to associate the pictographs with frequency information to regulate this issue. some studies @xcite @xcite also showed that word sense disambiguation improves the results of text-to-picto systems. this would avoid translation errors related to homonyms identified in our manual evaluation, e.g. enceinte (to be pregnant or a speaker) and bleu (the colour or a bruise). ## conclusion we presented the french version of the text-to-picto system, which automatically translates a textual input into pictographs for people with an id. our experiments show that this system is easily extensible to other natural or pictograph languages. 11 11 the source code of the system and the french corpora used for evaluation will be made available for the research community at the following address: https://github. com/vincentccl/picto . compared to the dutch version, we adapted the shallow linguistic analysis by adding new steps (detection of mwe, preprocessing of specific phenomena, and simple ner). data cleaning was performed to link the arasaac pictographs to french semantic resources. the evaluations on the email and the book corpus with wolf show that our results are indeed in line with those of previous studies. however, there is room for further improvement, for instance adding a word sense disambiguation step to select the right pictograph for a given meaning. we also carry out automated and manual evaluations on a new use case: medical data, which raised new challenges related to the translation of technical terms. we have seen above that our system currently tends to poorly handle technical terms, often missing from wolf. we plan to investigate solutions to this limitation, for example by applying automatic text simplification for the medical domain @xcite on the original sentences. we also plan to run tests with target users to tune this text-to-picto system for medical communication between doctors and patients in the hospital setting. the dutch version of the presented system has already been tested in real situations with a focus group of five adults with an id and two coaches in a day centre in belgium @xcite .