PROPICTO : Developing Speech-to-Pictograph Translation Systems to Enhance Communication Accessibility (2023)

Abstract: PROPICTO is a project funded by the French National Research Agency and the Swiss National Science Foundation, that aims at creating Speech-to-Pictograph translation systems, with a special focus on French as an input language. By developing such technologies, we intend to enhance communication access for non-French speaking patients and people with cognitive impairments.

## introduction alternative and augmentative communication (aac) devices have taken an increasingly important role among people with disabilities and their relatives. however, usage of these technologies (i.e., communication boards or electronic media) may be cumbersome (vaschalde et al., 2018). to surmount this problem, we argue that speech-to-pictograph (s2p) translation systems can be helpful for aac users. in addition, we believe that they can improve the accessibility of health services for patients not speaking the local language. developing such tools requires in-depth research on several areas of natural language processing (nlp). in this paper, we present a research project aimed at creating systems that automatically translate spoken french into pictographs. launched in early 2021, propicto 1 (the acronym stands for projecting spoken language into pictographs) is a french-swiss four-year project, funded by both the french national re-search agency foot_1 and the swiss national science foundation. @xcite it is conducted as a collaboration between the department of translation technology at the university of geneva and the study group for machine translation and automated processing of languages and speech, affiliated to the grenoble informatics laboratory. for the purpose of this project, we will examine several nlp-related areas; namely, speech recognition, syntactic parsing, sentence simplification as well as pictograph generation. by integrating this series of tasks into different workflows (depending on the target scenario), we propose novel cross-modal machine translation systems that convert spoken language into pictographic units. using this approach, we intend to tackle societal and communicative needs in the fields of: (1) disability, where an individual seeks to communicate with a person having a cognitive disorder, and (2) medical settings, where a language barrier exists between patient and practitioner. ## architectural overview propicto aims at improving the usability of aac devices by leveraging nlp-based solutions for greater accessibility. we will design new methods and corpora so as to enable spoken utterances to be transcribed directly into sequences of pictographs, either generalpurpose like arasaac, foot_3 or specific-purpose (i.e., santébd foot_4 for health-related concepts). the project will face two major challenges: to better address them, we will adopt a cascade approach for our s2p processing workflow, which will be adapted according to the target setting. thus, a first approach will favor a concept-based strategy to address pictographic generation, and will be integrated within a medical-purpose s2p architecture, 6 consisting of an automatic speech recognition (asr) system and a neural text-to-umls 7 module that will define the pictographs to be produced and the syntax. an alternative pictograph generation strategy will rely on a wordbased approach, and will be preceded by the next stages (as shown in figure 1 ): asr, dependency parsing (dp) and sentence simplification. using a cascade approach is motivated by the expected benefit of one phase over the next. additionally, it helps to ensure greater model explainability. our second proposed cross-modal architecture will start from an asr module, relying on state-of-the-art wav2vec2.0 models. the dp task will be addressed with an end-to-end parser whose input is the raw signal for a given utterance. using the raw signal instead of transcriptions enables us to use prosodic information to better predict syntactic boundaries (pupier et al., 2022). extracting a syntactic-based representation can in turn provide key information for a more effective sentence-level simplification. reducing the linguistic complexity of the input transcript is likely to help the subsequent step, where the translation into pictographs will also be governed by expert grammar rules. @xcite 3 contributions propicto will make available to the scientific community methods and resources enabling a translation from spoken french into pictographs. the licenses will be as permissive as possible and conform to those of the pictographic sets being used. furthermore, several prototypes for different target audiences will be put into production at the end of the project: (1) in emergency settings at the 6 for further details on this architecture, see (mutal et al., 2022). @xcite this acronym refers to unified medical language system (umls) concepts. 8 like multi-word expressions, verb tenses or proper names. geneva university hospitals, foot_5 and (2) in institutions for children and adults with multiple disabilities. these will be tested in real conditions and evaluated using human and automatic methods.