Empowering AAC Users: A Systematic Integration of Personal Narratives with Conversational AI (2024)

Abstract: Communication barriers have long posed challenges for users of Alternate and Augmentative Communication (AAC). In AAC, effective conversational aids are not solely about harnessing Artificial Intelligence (AI) capabilities but more about ensuring these technologies resonate deeply with AAC user’s unique communication challenges. We aim to bridge the gap between generic outputs and genuine human interactions by integrating advanced Conversational AI with personal narratives. While existing solutions offer generic responses, a considerable gap in tailoring outputs reflecting an AAC user’s intent must be addressed. Thus, we propose to create a custom conversational dataset centered on the experiences and words of a primary AAC user to fine-tune advanced language models. Additionally, we employ a Retrieval-Augmented Generation (RAG) method, drawing context from a summarized version of authored content by the AAC user. This combination ensures that responses are contextually relevant and deeply personal. Preliminary evaluations underscore its transformative potential, with automated metrics and human assessments showcasing significantly enhanced response quality.

## introduction communication is essential for sharing experiences and fostering connections, yet it poses significant challenges for many individuals using aac @xcite . according to recent statistics, about 5 million people in the u.s. and 97 million globally are unable to use speech for communication due to conditions like cerebral palsy and als @xcite . augmentative communication technologies (acts) @xcite have been developed to aid these individuals, offering tools like eye tracking and dynamic screen navigation to facilitate communication through text and pic-ture selection. despite these advancements, traditional aac solutions @xcite often lack the depth to express an individual's personality fully, and the slow communication rates, typically less than 10 words per minute, can lead to frustration and isolation @xcite . traditional aac tools @xcite @xcite have been instrumental in enabling communication for many, yet often lack the finesse @xcite needed to capture the user's personal narratives and unique experiences. recent advancements in ai, deep learning, and language models @xcite @xcite offer new possibilities for creating personalized conversational aids that adapt to the user's background and evolve with their changing needs. this paper, a collaboration between computer scientists and aac practitioners, presents an innovative approach that prioritizes personal narratives by merging modern ai's adaptability with individual user stories. previous studies @xcite prioritized model accuracy over adaptability in aac systems, focusing on technical aspects rather than individuality; this highlights the need for a new approach that values personal narratives and leverages ai to reflect each user's uniqueness. in this study, we introduced a system that does not merely optimize for speed or vocabulary variety; (1) it seeks to resonate deeply with each aac user's individuality. the significant out-performance of our rag approach, validated through human and automatic evaluations, is crucial as it sets a new benchmark in aac, emphasizing the importance of making every interaction deeply personal and contextually rich, thereby enriching the lives of aac users through more meaningful conversations. ## related work recent advancements in aac have leveraged ai to improve communication for those with speech impairments, with applications like voiceitt's talkitt @xcite and livox @xcite providing real-time assistance and bridging communication gaps. these innovations reflect a growing awareness of the challenges faced by differently-abled individuals @xcite and show a shift in aac research towards ai-powered mobile applications, particularly for asd children in developing countries @xcite . this transition from traditional sgd to ai applications indicates a promising direction in enhancing communication abilities for verbally challenged youth. google's project euphonia 1 uses ai to enhance speech recognition for atypical speech patterns by training asr models @xcite and developing speech intelligibility classifiers @xcite on a diverse dataset of disordered speech from conditions like als and cerebral palsy. this initiative improves accessibility to voice-activated technologies and tackles the challenge of understanding non-standard speech patterns. concurrently, brain-machine interfaces (bmi) offers new possibilities in aac, allowing direct brain-to-computer communication, which could transform interaction for those with severe motor impairments @xcite . research highlights that augmentative communication technologies (acts) typically allow communication rates of under 10 words per minute @xcite , with adaptations to improve interaction often leading to misunderstandings @xcite . for individuals with severe motor impairments, options like brain-computer interfaces (bcis) offer textbased communication, though speeds remain below one word per minute @xcite . in contrast, our work enriches aac by focusing on the depth and richness of personal narratives, 1 project euphonia: https://sites.research.google/ euphonia/about/ integrating real-life dialogues to enhance conversational ai. this approach aligns with projects like euphonia and bmi-based systems, aiming to significantly improve communication effectiveness and quality of life for aac users by merging ai advancements with practical communication needs. ## dataset we construct a personalized dataset to enhance communication for aac users. the primary motive of this dataset is to facilitate aac users in sharing their life experiences more effectively and authentically with others. generic language models often lack the nuanced understanding required for personalized interactions; our dataset plays a pivotal role in fine-tuning pre-trained language models. by doing so, we aim to equip these models with the ability to generate communication that is not only contextually rich but also profoundly personal, mirroring the individual experiences and narratives of aac users. information about the aac user and the study setting can be found in appendix a, b.