S aying N o is A n A rt: C ontextualized F allback R esponses for U nanswerable D ialogue Q ueries (2021)

Abstract: Despite end-to-end neural systems making significant progress in the last decade for task-oriented as well as chit-chat based dialogue systems, most dialogue systems rely on hybrid approaches which use a combination of rule-based, retrieval and generative approaches for generating a set of ranked responses. Such dialogue systems need to rely on a fallback mechanism to respond to out-of-domain or novel user queries which are not answerable within the scope of the dialogue system. While, dialogue systems today rely on static and unnatural responses like “I don’t know the answer to that question” or “I’m not sure about that”, we design a neural approach which generates responses which are contextually aware with the user query as well as say no to the user. Such customized responses provide paraphrasing ability and contextualization as well as improve the interaction with the user and reduce dialogue monotonicity. Our simple approach makes use of rules over dependency parses and a text-to-text transformer fine-tuned on synthetic data of question-response pairs generating highly relevant, grammatical as well as diverse questions. We perform automatic and manual evaluations to demonstrate the efficacy of the system.

## introduction in order to cater to the diversity of questions spanning across various domains, dialogue systems generally follow a hybrid architecture wherein an ensemble of individual response subsystems @xcite are employed from which an appropriate response is presented to the user @xcite @xcite . however, it is common for dialogue systems to encounter queries which are not within their scope of knowledge. while increasing the number of such subsystems would be a good strategy to increase coverage, it can be a never ending process and a default fallback strategy would always be needed. besides, domain specific dialogue systems, especially those deployed in professional settings generally prefer restricting themselves to a fixed set of domains, and purposely refrain from responding to out-of-domain and random or toxic user queries. one approach to acknowledge such queries is to have a fallback mechanism with responses like "i don't know the answer to this question" or "i'm not sure how to answer that." however, such responses are static and unengaging and give an impression that the user's query has gone unacknowledged or is not understood by the system as shown in figure 1 above. @xcite have shown that static and predefined responses lead to lower levels of user engagement and decrease users' interest in interacting with the system. @xcite shows that a system which reacts to system breakdowns and to low user engagement leads to a better user engagement. our fallback approach attempts to address these limitations by generating "don't-know" responses which are engaging and contextually closer with the user query. 1) since there are no publicly available datasets to generate such contextualised responses, we synthetically generate (query, fallback response) pairs using a set of highly accurate handcrafted dependency patterns. 2) we then train a sequence-to-sequence model over synthetic and natural paraphrases of these queries. 3) finally, we measure the grammaticality and relevance of our models using a crowd-sourced setting to assess the generation capability. we have released the code and training dataset used in our experiments publicly. foot_0 ## related work improving the coverage to address out-of-domain queries is not a new problem in designing dialogue systems. the most popular approach has been via presenting the user with chit-chat responses. other systems such as blender @xcite and meena @xcite promise to be successful for open-domain settings. paranjape et al. (2020) finetune a gpt-2 model (radford et al., 2019) on the empatheticdialogues dataset @xcite to generate social talk responses. while this might seem fitting for chit-chat and social talk dialogue systems, domainspecific scenarios often dealing with professional settings would refrain from performing friendly or social talk especially avoiding the possibility of the randomness of generative models. also, multiple subsystem architectures always have the possibility of cascading errors and profane or toxic queries. hence systems should always have a foolproof mechanism in the form of static templates to reply from. @xcite uses an interesting approach for error handling by mapping dialogue acts and intents to templates. besides, like @xcite it is always safer to generate fallback responses on encountering queries which might be toxic, biased or profane. 2 another line of work attempts to handle user queries which are ambiguous by asking back clarification questions @xcite @xcite . while this increases user interaction and coverage to an appreciable extent, it does not eliminate the requirement of a failsafe fallback responder. this paper's contribution is to address this requirement with an enhanced version of a fallback response generator. ## methods we describe two approaches to generate such contextual don't-know responses. ## results most prior generated systems are evaluated on a range of automatic metrics like bleu and rogue @xcite used in the machine translation literature. however, owing to the drawbacks of these metrics, we perform human evaluation of the generated responses using two metrics -namely "relevance" and "grammaticality" as defined in @xcite . we evaluate the performance of both the approaches in a crowd-sourced setting by requesting englishschooled individuals to rate. 6 raters were asked to evaluate grammaticality in a binary setting (grammatical/ungrammatical) and relevance on a likert scale (1 to 5). our human evaluations are shown in table-2. t5 responses tend to be more grammatical than their dependency counterparts by a large margin of 6%. relevance scores drop slightly from 3.97 to 3.66. this can be largely attributed to the model's paraphrastic ability of describing words and connected events outside the knowledge of the user's query. eg. in the second query in in addition, t5 responses on an average generate at least double the number of novel words than their dependency counterparts as shown in the t5 model helped to not only add paraphrastic variations but also scale to user queries outside of the scope of the dependency templates. more importantly, without losing the original ability of saying no, the model was able to generate more natural sounding dont-know-reponses by utilizing it's inherent world-knowledge acquired during pretraining. ## conclusion and future work we describe two simple approaches which enhance user interaction to cater to the necessities of reallife dialogue systems which are generally a tapestry of multiple solitary subsystems. in order to avoid cascading errors from such systems, as well as refrain from answering out-of-domain and toxic queries it is but natural to have a fallback approach to say no. we argue that such a fallback approach could be contextualised to generate engaging responses by having multiple ways of saying no rather than a one common string for all approach. the appeal of our approach is the ease with which it can rightly fit within any larger dialogue design framework. of course, this is not to deny that as we give more paraphrasing power to the fallback system, it would tend to retract from succinctly replying with a no -as is evident from the drop in the relevance scores. nevertheless, we still believe that both our fallback approaches could serve as effective baselines for future work.