Cue-bot: A Conversational Agent for Assistive Technology (2022)

Abstract: Intelligent conversational assistants have become an integral part of our lives for performing simple tasks. However, such agents, for example, Google bots, Alexa and others are yet to have any social impact on minority population, for example, for people with neurological disorders and people with speech, language and social communication disorders, sometimes with locked-in states where speaking or typing is a challenge. Language model technologies can be very powerful tools in enabling these users to carry out daily communication and social interactions. In this work, we present a system that users with varied levels of disabilties can use to interact with the world, supported by eye-tracking, mouse controls and an intelligent agent Cue-bot, that can represent the user in a conversation. The agent provides relevant controllable ‘cues’ to generate desirable responses quickly for an ongoing dialog context. In the context of usage of such systems for people with degenerative disorders, we present automatic and human evaluation of our cue/keyword predictor and the controllable dialog system and show that our models perform significantly better than models without control and can also reduce user effort (fewer keystrokes) and speed up communication (typing time) significantly.

## introduction conversational agents, especially systems such as alexa and google home, have become commodity items in people's homes. such systems have enabled carrying out one-shot tasks such as setting reminders, playing music and accessing information simpler for the general population. we also have other pc and cloud based chatbots that are designed to perform certain goals or tasks, or to just engage in a casual conversation/chat with a user. the latter class of open-domain conversational agents have not yet seen widespread adoption besides mostly research exploration projects for developing conversational agents @xcite . large language models are being developed today with end-to-end pre-training. large-scale pre-training has attained significant performance gains across many tasks within nlp @xcite , including intent prediction @xcite and dialogue state tracking @xcite . open-domain chatbots are also being trained using generative language modeling objective of minimizing perplexity on next word prediction task using large conversational corpora and transformer based models. these models have demonstrated surprising generality, with models like dialogpt @xcite , meena @xcite and blender @xcite achieving response generation performance competitive with humans in certain settings. these improving systems still suffer from issues such as repeated responses, hallucinated facts, and lack of controllability, grounding and embodiment @xcite . with the availability of these pre-trained language enabling models, novel products and applications are emerging in several domains @xcite . one such accessesibility application we are exploring is aimed towards leveraging language modeling technology to support minority group of people with certain disabilities 1 to communicate with others effectively. one such example is amyotrophic lateral sclerosis (als) or motor neuron disease(mnd), a progressive, degenerative, neurological disorder where people lose their muscle movement, voice and the ability to carry out a normal day-to-day communication. there have been technologies and platforms, one such example is assistive context-aware toolkit (acat) 2 , figure 1: a dialog system for an assistive use-case can listen to a conversation and provide diverse cues to the user. these cues, provide human control to the dialog system that can generate relevant responses that can further be edited. we want to enable full and faster communication and provide interaction support tools for people with such disabilities by having an intelligent agent be their voice and content assistent. the system should use very limited user input (e.g. gaze, single muscle movement, facial gesture, etc) and suggest cues and cue-based responses that can be interactively chosen and edited for near real time social interactions. the goal of such a system is to minimize the effort by minimizing the keystrokes input required for continued coherent interactions. today's response generation systems suffer from several issues and are very hard to use as-is for our usage requirements. the system for our usage needs to be context-aware, personalized, should enable minimal user-intervention and most importantly, be assistive and controllable by the user. fast response generation with response cues, editing and auto complete features can dramatically reduce the silence gap in the conversation resulting from users slower keystroke by keystroke input. our contributions in this work are, i) minority group application: we bring forth a novel usage for open-domain chatbots/response generation systems, i.e., designing a reponse generation system that will represent users with communication disabilities and help them fulfill their day-to-day communication needs. ii) minimal user effort and intervention: we show that the keyword predictor models can speed up communication time by suggesting cues to the user. we also present a tech-nique for controllable response generation using these cues. we present human and automatic evaluation for this approach. iii) demo interface: we showcase a demo where a user can interact only using his/her eyes to control the interface, with minimum effort and time. ## motivation to enable people with mnd and other disabilities to communicate, intel labs developed acat, an open source platform that was originally developed for professor stephen hawking. with acat, users have complete access to the capabitlies of their computers that they can control using various modalities and sensors such as proximity sensors, eye-gaze and further capabilities such as bci-based controls are being developed. acat also includes word prediction, speech synthesis capabilities, this allows users to respond to ongoing conversations, and a range of tasks such as accessing emails, editing documents and browsing the web. in acat, users can choose words that appear from the word predictors, or select letters to create words using the input modality that suits their condition. while this empowers users to communicate, this still involves a lot of effort in terms of the word/letter selections and involves a huge latency. with this work, we aim to reduce the user effort and intervention and also the time in generating a user response, by using the state of the art language modeling technology as will be described in the below sections. our goal is to also integrate this work into the current acat system to enable the user to select entire responses based on input keywords, with minimum effort and intervention. ## system architecture figure 1 shows the interaction flow of the cue-bot system. consider an ongoing conversation between an interlocutor and the user. an automatic speech recognition (asr) system converts the interlocutor's current utterance to text. this, in addition to the dialog context, is input into the cue-generator model (described below), which outputs the possible cues or keywords for a potential response to the input that the user might want to respond with. in the figure, given the utterance "hi peter, what are you upto this friday?", the cue-generator generates "movie", "family", "home" that the user can choose from. the interface also allows the user to enter his/her own cue or keyword, if the user needs, in case none of the suggested words are relevant. the user uses the tobii eye-tracker and the optikey mouse controls to make a keyword selection or to type out a keyword of his/her choice. once the user chooses or enters a custom keyword, this information is sent to the cue/keywordbased response generation system (details below) that generates multiple responses relevant to the cue/keyword. the user can 1) choose one of these responses to use as his/her response to the interlocutor, 2) edit one of the responses or 3) type out the entire response if none of the suggested responses are relevant. these modules are described in detail below. ## experimental setup we initialize the transfertransfo model with weights of dialogpt 'medium' model with 345m parameters. we also use two candidates for the next utterance prediction task. we use a batch_size of 64 for training, nucleus sampling for generation with top_p set to 0.9. we fine-tune the model for 3 epochs. we compare the model trained without any information with a keyword-context model trained with keyword as auxillary input information. ## conclusion and future work in this work, we present a system to support users with mnd and other disabilities to carry out day to day social interactions with lesser effort, time and interventions. we use input modalities such as gaze tracking that allows users to control the entire interface only using their eyes. we build models that utilizes the ongoing conversations and suggests possible cues/keywords that the users can use, and generate relevant responses based on the selected keyword. we show through automatic and human evaluations that our models are better than the models without control and also save significant time and effort in interactions. for future work, we aim to integrate it with the acat toolkit that already supports mnd users, to improve their quality of communication. we also aim to personalize the system by using user's data when available and also build a system that can continually learn through user interactions.