A utomated Generation of Storytelling Vocabulary from Photographs for use in AAC (2021)

Abstract: Research on the application of NLP in symbol-based Augmentative and Alternative Communication (AAC) tools for improving social interaction support is scarce. We contribute a novel method for generating context-related vocabulary from photographs of personally relevant events aimed at supporting people with language impairments in retelling their past experiences. Performance was calculated with information retrieval concepts on the relevance of vocabulary generated for communicating a corpus of 9730 narrative phrases about events depicted in 1946 photographs. In comparison to a baseline generation composed of frequent English words, our method generated vocabulary with a 4.6 gain in mean average precision, regardless of the level of contextual information in the input photographs, and 6.9 for photographs in which contextual information was extracted correctly. We conclude by discussing how our findings provide insights for system optimization and usage.

## introduction augmentative and alternative communication (aac) tools can enhance communication for nonspeaking individuals, thus offering improved social interaction and independence. well established nlp techniques, such as spell check and word prediction, support those with primarily physical barriers to communication (e.g., adults with als) to compose complex and nuanced sentences in orthographic-based systems more effciently. however, those with developmental disabilities (e.g., autism spectrum disorder, asd) or lexical and semantic processing impairments that limit their ability to spell out words (e.g., adults with aphasia 1 ) must usually rely on less expressive symbol-based systems, for which those techniques offer little sup-1 a language disorder mostly often caused by a stroke. port due to unique characteristics of communication with these systems. users of symbol-based aac typically do not construct full, grammatically correct sentences, complete with prepositions and infections, but rather often only need a few key content words (i.e., nouns, adjectives, verbs)-appearing at any part of the sentence-to supplement other forms of communication, including preserved speech, gestures, or drawings. such scattered use of vocabulary hinders the typical statistical prediction approach, which relies on patterns learnt from a large training corpus. nonetheless, there is much opportunity for improving symbol-based aac, which is often abandoned because it offers too little communication support relative to the effort required to learn and use @xcite . as in many non-orthographic aacs, vocabulary is represented by images that reproduce computer generated speech when selected; however, unlike the status quo, this design eliminates navigation across complicated hierarchies and the need for pre-programming. selecting and organizing vocabularies able to attend user's communication needs in a wide variety of contexts and such that they can fnd words quickly is one of the major challenges (van de @xcite . alphabetical organizations are not useful, and traditional hierarchical schemes based on abstract categories (e.g., @xmath0, making navigation extremely slow for anything but the smallest (least useful) vocabularies. presenting vocabulary as a fat hierarchy is best @xcite @xcite ; however, only a very limited set of options can be displayed, making communication very reliant on having the desired keywords among the available options. providing concise situation-relevant vocabularies currently depends on support from a clinician or caregiver to pre-program the device. but such support is often limited or not available, which consequently limits these devices to supporting generic expressions of wants and needs, i.e., functional communication, and not for social interactions involving spontaneous narratives @xcite . generating vocabulary from user's contextual data through natural language generation (nlg) techniques seems an obvious venue to facilitate social interactions. although nlg has been successfully applied in the context of task-oriented dialogs @xcite , question answering @xcite ), text summarization @xcite , and story generation from photograph sequences @xcite , it is unclear how these techniques can be adapted to the specifc needs of aac support @xcite . in this paper, we call for more research in the nlp community devoted to language generation for symbol-based aac systems. we present an overview of the scarce research on the topic and contribute a method that generates vocabulary automatically from a user's photographs to support autobiographical storytelling, demonstrating how it performs under different combination of the system's controllable parameters and a wide range of input photographs. ## vocabulary generation method our method generates a rank of key words and short narrative phrases from a single foot_1 input photo for scaffolding storytelling. it was designed to be used as the back end of interactive aac systems in which relevant vocabulary is associated with a main photograph, such as mooney et al.'s cochat, or as in the example design shown in fig. 1 . we used vist-train, a sub-set of the visual storytelling dataset vist @xcite as the main source for vocabulary generation. vist-train encompasses 80% of the entire dataset, and is composed of 65,394 photos of personal events, grouped in 16,168 stories. each photo is annotated with descriptions and narrative phrases that are part of a story, created by amazon mechanical turk workers. we judged vist to be a good source of vocabulary because i) photos were extracted from personal flickr albums on a wide range of "storyable" events, related to 69 topics (e.g., graduation, building a house), ii) associated vocabulary is representative of storytelling and, iii) stories and photo descriptions were constructed by a large number (1907) of workers under a rigorous procedure. the generation process is composed of fve steps, as detailed below and illustrated in fig 2 . we explore different implementations for some of the steps, represented by the system's controllable parameters emphasized with bold italic formatting throughout the paper. the different combination of those parameters are evaluated in the next section. ## evaluation experiment the goal of our evaluation is to understand how our design choices, represented by the system controllable parameters, along with uncontrollable factors related to the input photograph (i.e., uncontrollable parameters), affect the system's performance. thus, we compared the relevance of vocabulary generated under different combinations of these parameters to investigate the following specifc research questions: ## discussion the design space for generating aac storytelling vocabulary directly from photographs is vast and under explored. design decisions for individual system components will impact other components and ultimately the overall system effectiveness, and therefore cannot be arbitrary. without a rigorous performance evaluation on different confgurations of parameters, users would be at risk of using a fawed or under optimized system, which could lead to user frustration and abandonment, and cause confounds that obscure whether failures are due to the need for algorithmic tuning or mismatch between the intended support and user needs. the study of controllable parameters (rq1, 5) demonstrated that our method is able to provide relevant vocabulary, and showed how it can be used to optimize the system and identify areas for further improvement. the exploration of uncontrollable parameters (rq2, 3, 4) helped illustrate the likely variation in system performance during real world usage (i.e., wide variety of input photos), allowing us to better anticipate potential problems or pitfalls and understand requirements for use. the similar performance across photos with different levels of contextual information (rq2) suggests that our method is robust to variations in the input photograph. users will not need to be instructed to take photographs following specifc requirements, e.g., "photos should demonstrate an action" or "photos should depict objects only". the similar levels of performance is explained by the pattern observed in the rq4 analysis; the more elements a photo contains, the better knowledge the machine learning has to infer the central aspect of the photo, but at the same time, the harder it is to capture each and every element. in addition, an element wrongly identifed will have less impact on the overall scene understanding since other elements complement the description. an example would be a photo of a birthday party, in which the machine-learning platform is able to infer the central concept (birthday) from the several elements depicted (e.g., cake, candles, balloons), but misses some of the details (e.g. drinks). on the other hand, simplistic photos will rarely lead to elements being cut out, but the computer vision technique will have more variability when performing the inferences, leading to erroneous descriptions more often. on the other hand, the quality of generated vocabulary was strongly dependent on the computer vision technique employed to extract contextual information about the scene (rq3) . when a wrong description is generated, the subsequent steps of the algorithm are misled and therefore generate vocabulary less relevant for retelling the scene depicted in the photograph. nonetheless, even in this case, an aac device using our method would provide vocabulary more relevant than if the most frequent english words were provided. since photos for which the computer vision technique was able to correctly identify all contextual elements resulted in substantial performance gain, we encourage further exploration of this component. an option would be to use a higher number of raw context labels instead of the single human-like description employed in this work. our vocabulary expansion analysis (rq5) provide valuable insights into how the combination of multiple lexicon sources can generate more relevant vocabulary. the most promising approach was to combine the visual-to-story dataset with strongly associated words from a mental-lexicon model, but only when there was high intersection between the two vocabularies. ## conclusion developing a photo-to-story vocabulary aac system presents two challenges; a nlp one in how to generate such vocabularies, and a human-computer-interaction (hci) one in how to use such vocabulary to offer interactive language support. in this work, we tackle the frst challenge. we demonstrated that our method is able to generate vocabulary with reasonable levels of recall and precision, regardless of the level of contextual information in the input photograph, illustrated the likely variation in system performance during real world usage, and provided meaningful insights for fne tuning the algorithm, enabling us to move to the next phase of designing and evaluating, with aac users, our mobile interactive application.