Aiding Non-Verbal Communication: A Bidirectional Language Agnostic Framework for Automating Text to AAC Generation (2024)

Abstract: Persons with severe speech and motor impairments (SSMI), like those with cerebral palsy (CP) experience significant challenges via communication in conventional methods. Many a times they rely on Graphical symbol-based Augmentative and Alternative Communication (AAC) systems to facilitate the communication. Our work aims to support AAC communication by developing specialized datasets for direct translation of Graphical Symbols to Natural Language text. The dataset is enhanced with an automated Text-to-Pictogram generation module. The dataset is enriched with some additive information like tense-based information and subjective information (questionnaires, exclamations). Additionally, we expanded our efforts to include translation into Indian language Bengali, for those individuals with SSMI who are more comfortable communicating in their native language. We aim to develop an end-to-end language agnostic framework for efficient bidirectional communication between non-verbal AAC picture symbols and textual data.

## introduction in today's increasingly connected world, effective communication is vital for personal expression and social interaction. assistive technology @xcite plays a crucial role in promoting inclusion and enabling access for individuals with various disabilities and older adults. a significant number of people with various disbailities including severe speech and motor impairments (ssmi) prefers non-verbal picture symbol based communication. this menthod of communication, known as augmentative and alternative communication (aac) @xcite serves as a system that helps to bridge accessibility gaps for these individuals. picture exchange communication system (pecs) @xcite , is a picture-based communication technology that supports writing, reading, and speech through intrinsic methods like symbolic linguistic systems. in the next section, we have described three types of pecs systems. ## literature survey aac devices are an essential component of assistive technology that assists people with several speech or language problems communicate. this review of the literature looks at the many aac systems @xcite that are used to support persons with ssmi mostly persons with cp as an alternate mode of symbolic communication. in this communication system, some users may use ambiguous symbols for texting, where they must retain and recall all of the meanings associated with each symbolic icon. however, the user needs to have strong linguistic knowledge to obtain the correct output. pictographic communication: a companion system in 1998 @xcite , was created to expand a collection of uninfected content words into a complete phrase or sentence. though many users with limited language proficiency may find it challenging to construct messages using the syntactic sequence @xcite . a web-based translation tool called ara-traductor @xcite can convert plain spanish text into pictograms, primarily matching word-for-word translations. few researchers have worked on text-to-pictogram translation @xcite , @xcite , @xcite , whereas several issues have been mentioned in those systems @xcite . even those methodologies cannot express tenserelated information, or numeric information in the translated text. these limitations of using pictographic language have led to the use of other graphical symbols in communication. bliss symbolic communication: karl blitz @xcite created bliss symbols to address communication gaps. researchers have used this bliss symbolic language for translation @xcite , @xcite and few researchers have modified the method for the users with several speech and motor impairments (ssmi) especially persons with cerebral palsy (cp) @xcite , @xcite , and also people with several speech and hearing impairments @xcite . bliss symbolic language has a limited number (almost 6000 bliss words) of data. thus, most realworld texts lack a description of bliss language. few researchers have considered pictograms and bliss symbols as graphical icon units for augmentative communication (ac) @xcite . previous models mostly rely on proprietary or authorized datasets for translation, many of which are inaccessible for further research. currently, there is no online dataset for direct translation between graphical symbols and natural language text. most available datasets feature either bliss symbols or pictographic languages, but rarely both. the lack of a structured symbolic language database has stalled progress in this area. in our study, we address this gap by creating a reliable dataset for translating between graphical symbols and natural language (english) text. the dataset is further enhanced by including tense-related and subjective information. a 'text-to-pictogram' (picgen) generation module is applied to manage unknown text. this translation system is also extensible to other languages. we extended our research by embedding the indian language bengali into the translation system. our goal is to provide individuals with ssmi a userfriendly tool to advance symbolic communication. ## methodology in this section, we discussed the experiments conducted during our study. the experiment starts with preparing the translation corpus for the symbolic language-english text translation. ## conclusion the study aimed to address the communication needs of individuals with several speech impairments. a graphic symbolic communication system provides an alternative communication tool that makes the environment more accessible to persons with speech and motor impairments (ssmi) especially persons with cerebral palsy (cp). with the help of creating graphical symbolic corpora, the translation datasets were generated to develop a graphical symbolic-natural language text translation system using an emerging nlp-aac domain. the dataset is freely accessible for direct use in graphical language translation and reverse translation to enhance communication. the proposed dataset includes the usefulness of symbolic data like bliss symbols and pictograms. for the persons with ssmi, a unique user-system translation method was designed to overcome practical difficulties. we are aiming to provide those users with an intuitive and user-friendly interface, enhanced by an additive feature such as text-to-pictogram generation (picgen) module to deal with unknown inputs. our dataset is also enriched with contextual information like tense-related information to make the data more reliable. our system is also extensible to other languages, such as bengali. translation into the indian language bengali makes communication easier for individuals who are comfortable with this bengali language. our objective is to complete the project with additional features and resources to promote greater independence for those individuals.