P ic T alky: Augmentative and Alternative Communication for Language Developmental Disabilities (2022)

Abstract: Children with language disabilities face communication difficulties in daily life. They are often deprived of the opportunity to participate in social activities due to their difficulty in understanding or using natural language. In this regard, Augmentative and Alternative Communication (AAC) can be a practical means of communication for children with language disabilities. In this study, we propose PicTalky, which is an AI-based AAC system that helps children with language developmental disabilities to improve their communication skills and language comprehension abilities. PicTalky can process both text and pictograms more accurately by connecting a series of neural-based NLP modules. Additionally, we perform quantitative and qualitative analyses on the modules of PicTalky. By using this service, it is expected that those suffering from language problems will be able to express their intentions or desires more easily and improve their quality of life. We have made the models freely available alongside a demonstration of the web interface. Furthermore, we implemented robotics AAC for the first time by applying PicTalky to the NAO robot.

## introduction the majority of people with language disabilities suffer in their daily lives as they cannot understand or speak the language. as it is a means of communication, they may be deprived of the opportunity to participate in social activities. also, they may experience financial difficulties. in general, people with speech disorders have lower employment rates than people with other types of disabilities. what is worse is that the proportion of people with autism disorder has been increasing every year @xcite . accordingly, a solution is required to ensure their economic freedom. meanwhile, augmentative and alternative communication (aac) has been suggested and applied to solve communication problems for people with language disabilities @xcite . this approach enables nonverbal communication by replacing language. although several aac software resources are available, existing software packages are expensive, difficult to use, and only provide simple functions. to address these problems, we present a novel aac system for children with language developmental disabilities. we refer to our aac software as pictalky. neural-based grammar error correction (gec) and a symbol-based text-to-pictogram (tp) module are utilized in our model. thus, pictalky offers neural-and symbolbased aac for the improvement of communication and language learning, which have not been adopted in existing software. from the perspective of nlp, the speech errors from people with language disabilities can be interpreted as grammatical errors at the morphological and syntactic levels. to handle these errors, neural gec is applied in pictalky. moreover, we consider both text and image processing for aac education and communication. after a sentence is entered as an input through the speech-to-text (stt) module, it is passed through the neural gec and natural language understanding (nlu) modules. finally, the corresponding pictograms are displayed. pictalky is aimed at children aged 0 to 14 years who have language developmental disabilities caused by intellectual or autism disabilities. the first reason that we focus on children is that early treatment during childhood is critical. according to @xcite , language must be acquired during a critical period that ends at approximately the age of puberty with the establishment of the cerebral lateralization of function. unless language is learned during this period, it is difficult for language to be used freely. this may result in social deterioration, contraction, aggression, and other problematic behaviors, which eventually affect the overall quality of life and satisfaction of the person @xcite . the second reason is that there is currently insufficient social support for language therapy. not all children with developmental disabilities can benefit from public systems owing to the limited support. moreover, in addition to the children, their family and caregivers them experience difficulties. therefore, we propose pictalky, which complements the limitation of existing products and increases the accessibility of children with language disabilities to appropriate education and treatment. we expect that not only the people with language disabilities but also their caregivers can have more easier education and communication by using this service. furthermore, in addition to the implementation of the web application, we apply pictalky to the nao robot, thereby providing the first robotics aac. we expect that robotics aac can draw interest of children, so that they can use aac more friendly and easily. our contributions are as follows: ## pictalky with robotics we have distributed pictalky as a web application (see appendix d). however, in the case of the web application, there is a possibility that it is difficult or boring for children to handle. therefore, in addition to the web service, we have applied robotics technology to pictalky for arousing interest in children. the nao robot @xcite is mounted in the communication module of pictalky. nao is the humanoid robot developed by soft-bank robotics foot_7 . nao has eight full-color rgb leds, an inertial sensor, two cameras, and many other sensors. it also has a sonar sensor to check the distance of objects in its vicinity to comprehend its environment with precision and stability. it enables nao to react its body to move when exposed by interaction. nao is also available in social robotics @xcite , which focuses on communicating robots capable of interacting and cooperating with humans. all of these characteristics in nao suit our research pursuit in terms of interacting with a human. we have created a human-robot interaction system whereby the nao robot has a conversation with the end users and the pictograms are printed onto the connected screen. as children show substantial interest in robots, this will aid in more familiar education as opposed to web or other applications @xcite . the video of our demo is also attached with our paper foot_8 . to the best of our knowledge, this study is the first to apply pictalky to the nao robot and to develop robotics aac for the first time. ## conclusion and future work we have proposed pictalky, which is the first ai-based aac service. with the series of deep learning-based modules, it is able to take a speech or text input from the user, correct error, and converts it into a pictogram automatically for more convenient communication and education of the people with language disabilities. the aim of our proposed system is to provide an opportunity of communication and connection among all people, without anyone being excluded. in the future, we will expand the pictalky data to multilingual data for use in various languages and to make it publicly available. in addition, we plan to conduct various ai for accessibility studies to improve the quality of life for the people with disabilities. starting with our research, we look forward to advances in many other studies so that all members of society can get benefits from ai technology without a financial burden.