C ue B ot: Cue-Controlled Response Generation for Assistive Interaction Usages (2022)

Abstract: Conversational assistants are ubiquitous among the general population, however, these systems have not had an impact on people with disabilities, or speech and language disorders, for whom basic day-to-day communication and social interaction is a huge struggle. Language model technology can play a huge role in empowering these users and help them interact with others with less effort via interaction support. To enable this population, we build a system that can represent them in a social conversation and generate responses that can be controlled by the users using cues/keywords. We build models that can speed up this communication by suggesting relevant cues in the dialog response context. We also introduce a keyword-loss to lexically constrain the model response output. We present automatic and human evaluation of our cue/keyword predictor and the controllable dialog system to show that our models perform significantly better than models without control. Our evaluation and user study shows that keyword-control on end-to-end response generation models is powerful and can enable and empower users with degenerative disorders to carry out their day-to-day communication.

## introduction conversational agents such as google home and alexa have become almost an integral part of homes and used by people of all ages to carry out tasks such as setting reminders, playing music and accessing information. there are also agents that can simply engage in chit-chat conversations, however, these open domain conversational agents have mostly been research explorations @xcite . large-scale pre-training has attained significant performance gains across many tasks within language modeling (devlin et al., 2019; radford * these authors contributed equally to this work and narasimhan, 2018), including intent prediction @xcite and dialogue state tracking @xcite . these pretrained language models have demonstrated surprising generality in open domain dialog tasks, with models like dialogpt @xcite , meena @xcite and blenderbot @xcite achieving performance competitive with humans in certain settings. with the availability of these models, novel products and applications are emerging @xcite such as communication systems (eg. email response completion @xcite ), creativity tools (story writing assistance @xcite ), human-ai collaboration for software engineering @xcite , biosciences (protein structure prediction @xcite and several others. one such accessibility application we are exploring is aimed towards leveraging language modeling technology to support minority group of people with certain disabilities foot_0 to communicate with others effectively. for example, amyotrophic lateral sclerosis (als) is a progressive, degenerative, neurological disorder, where people lose their muscle movement, voice and the ability to carry out a normal day-to-day conversation. it takes huge effort and time for these patients to use existing systems foot_1 to communicate sentences character by character using various data input mechanisms available to them (gaze, fingers, muscle movements). henceforth, we will use the term 'user' for such patients with disabilities, for whom our system is intended to support. our goal is to empower these users to communicate faster by having an intelligent agent be their voice and reduce the silence gap in the conversation resulting from users slower keystroke inputs. figure 1: a dialog system for an assistive use-case can listen to a conversation and provide diverse cues to the user. these cues, provide human control to the dialog system that can generate relevant responses that could be edited. the system needs to listen to an ongoing conversation (using automatic speech recognition(asr)) and should use be able to use very limited user input and suggest responses that can be interactively chosen and edited for near real time social interactions. such a system needs to be context-aware (contexts such as ongoing conversations, user's emotions, environment), personalized (language usage and style of the user and also be aware of users' interests/likes and dislikes) and most importantly, controllable. in this work, we focus on the controllability aspect and design the control mechanism via keywords in the system. we present the following contributions: i) minority group application: we bring forth a novel usage for response generation systems, i.e., to represent users with disabilities and help them in their day-to-day communication needs. ii) minimal user intervention: we present a human-controllable response generation using keywords/cues. we also build keyword/cue predictor models that further speed up communication time and evaluate these. iii) keyword loss: we introduce a keyword loss to our training objective that further helps in incorporating soft lexical constraints in the form of keywords/similar words in the generated responses, validated through automatic and human evaluation. we also present a user-study to understand the usefullness and the effectivness of our overall system. figure 1 shows the interaction flow of our system. an asr system converts an ongoing conversation (between an interlocutor and a user with disabilities) to text, which is input to the cue/keyword generator that generates possible, relevant cues that the user might want to respond with. the user can choose one of these keywords or also enter his/her own keyword to control the system. this keyword is an input to the response generator model that can generate relevant responses based on the keyword. the user can then either use one of the suggested responses or edit a response with just a few keystrokes, thus drastically reducing communication time. ## experimental setup we initialize the transfertransfo architecture weights of dialogpt 'medium' model with 345m parameters. language modeling and multiclassclassification coefficients, α and β are set to 1 as in the original model. we use a batch_size of 64 for training, nucleus sampling for generation with top_p set to 0.9 to fine-tune the model for 3 epochs. we run an ablation study to determine the effect of different ways of incorporating keywords using 5 main classes of models: i) no-keyword model (no_kw): trained without any keyword information ii) keyword-context (kw_context): trained with keyword as auxillary input + dialog context iii) keyword-loss (kw_loss): incorporates keyword loss + keyword as auxillary information. iv) keyword sim-loss (kw_sim_loss): incorporate similar words (embedding-based techniques such as glove @xcite (kw_sim_loss_glove) and wordnet-based (kw_sim_loss_wordnet) similarity) for loss computation . we experiment with 2 variations, one using the similarity score, and the other using 1. v) multiple-keyword-loss (multi_kw_loss): incorporate multiple keywords into the input as well as into the loss computation. ## conclusion we present a novel usage for open domain conversational models -representing differently abled users and enabling them to communicate. in such a use-case, minimizing the need for user intervention is critical, hence the focus of this work has been to develop controllable response generation models that enable fine-grained human control in the form of keyword inputs from the user. we also introduce keyword-based loss functions that encourages the model to generate the keyword or similar words in the response. to further improve efficiency and time in interaction, we develop keyword predictors and evaluate them. we show with both automatic and human evaluation that our models outperform the baseline model with no control, at the same time maintaining the response quality. we are working with patients to collect feedback and plan to deploy our system as part of an open source tool to impact the quality of life of the patients and help the caregivers. future research direction also involves improving the keyword predictors, and personalization of these controllable models (both speech and linguistic).