On Controlling Fallback Responses for Grounded Dialogue Generation (2022)

Abstract: Dialogue agents can leverage external textual knowledge to generate responses of a higher quality. To our best knowledge, most existing works on knowledge grounded dialogue settings assume that the user intention is always answerable. Unfortunately, this is impractical as there is no guarantee that the knowledge retrievers could always retrieve the desired knowledge. Therefore, this is crucial to incorporate fallback responses to respond to unanswerable contexts appropriately while responding to the answerable contexts in an informative manner. We propose a novel framework that automatically generates a control token with the generator to bias the succeeding response towards informativeness for answerable contexts and fallback for unanswerable contexts in an end-to-end manner. Since no existing knowledge grounded dialogue dataset considers this aim, we augment the existing dataset with unanswerable contexts to conduct our experiments. Automatic and human evaluation results indicate that naively incorporating fallback responses with controlled text generation still hurts informativeness for answerable context. In contrast, our proposed framework effectively mitigates this problem while still appropriately presenting fallback responses to unanswerable contexts. Such a framework also reduces the extra burden of the additional classifier and the overheads introduced in the previous works, which operates in a pipeline manner.

## introduction building knowledge grounded dialogue agents has been an important research line @xcite @xcite @xcite . such incorporation of real-world knowledge @xcite gives rise to consistent, informative and engaging response generation. unfortunately, even with a high-quality knowledge retriever, there is no guarantee that the desired knowledge can always be retrieved. there is indeed even no guarantee for the existence of the desired knowledge in the knowledge database. hence, presenting fallback responses is an essential ability for grounded dialogue agents. we make use of the notion of answerability that represents whether a dialogue context is answerable or not conditioned on the knowledge retrieved. figure 1 depicts an example to illustrate the importance of answerability in grounded dialogue response generation. as in the unanswerable dialogue context, a fallback response is desirable. conversely, as in the answerable dialogue context, the response should be as informative as possible. although the concept of answerability has been well explored in other nlp areas such as question answering @xcite , it is underex-plored in the dialogue community. most existing knowledge grounded dialogue agents @xcite @xcite and knowledge grounded dialogue datasets @xcite ignore the fallback issue. however, this is almost impractical, and it is unlikely to happen in the real world that all the contexts are answerable. one recent approach has been proposed to calibrate responses with the appropriate linguistic confidence @xcite ; however, it overlooks informativeness, or diversity, @xcite @xcite @xcite , which is an important quality metric for a dialogue system. though the previous work mentioned above @xcite employs an additional classifier for answerability, or in their case, linguistic confidence level, we demonstrate that our proposed method can achieve higher accuracy with the response generator. our proposed model employs controlled text generation @xcite @xcite @xcite . its central idea is to bias the generation towards a specific style by placing a control token in the input context. this control token has been investigated via two strategies: manually placed @xcite or model classified @xcite . one can manually place a control token with low offensiveness to prevent the dialogue response generator from generating an offensive context @xcite . one can also use a classifier to determine the linguistic confidence that the generator should present in its response generation @xcite . in contrast to these works, one of our characteristics is that while these works focus on the classification task only, our work turns the classification task into a generative manner and then exploits the classification result for the succeeding generation task within a single autoregressive generator. since no existing dataset is suitable for our task, we derived a dataset by augmenting an existing dialogue dataset with unanswerable tuples of the dialogue context and the knowledge retrieved, and we conducted our experiments on the derived dataset. our experimental results indicate that incorporating controlled text generation @xcite can capture answerability and rigorously replies with a fallback response to unanswerable contexts. however, it still undesirably hurts informativeness for answerable contexts by frequently responding with fallback responses to answerable contexts. our method can achieve higher accuracy in classifying answerability than the traditional controlled text generation. this reduces the chance of responding with fallback to answerable contexts and thus improves the informativeness for responses to answerable contexts while still responding appropriately with fallback to unanswerable contexts. ## experimental setup dataset preparation since no existing dataset is suitable for our aim, we derive our dataset based on the cmu document grounded conver-sations dataset (cmu dog) dataset @xcite . cmu dog is a multi-turn dyadic dialogue dataset in which two crowdsource workers converse and find out more about a specific movie based on that particular film profile. while most of the dialogue datasets focus only on either chitchat @xcite or task-oriented dialogue @xcite , cmu dog interleaves chitchat and task-oriented dialogue @xcite . it thus requires the agent to be both informative and knowledge grounded. such knowledge grounded dialogue agents should appropriately respond with fallbacks to the unanswerable contexts without hurting informativeness on the responses to the answerable contexts. therefore, cmu dog is a suitable dataset to validate the effectiveness of our proposed framework. we label all of the original instances as answerable conditioned on the ground truth knowledge. indeed, the crowdsource workers converse based on the ground truth knowledge @xcite . we then augment with unanswerable dialogues by sampling two instances [k 1 ; c 1 ; r 1 ] and [k 2 ; c 2 ; r 2 ] from the original dataset where k 1 ̸ = k 2 . this results into two unanswerable instances [k 1 ; c 2 ; f ] and [k 2 ; c 1 ; f ], where f represents the fallback responses that typically confess ignorance. this operation derives into a training / development / testing partition with 100,497 / 6,677 / 18,921 instances respectively for the cmu dog dataset. 4 unlike chitchat dialogue datasets @xcite which consist of several dialogue topics that can be irrelevant to each other, the movie profiles from cmu dog guarantees to be within the same domain. this is important as real-world retrievers can be competitive, meaning that irrelevant retrieved knowledge can make the task oversimplified into relevance classification. fortunately, our augmentation strategy can still derive an answerability task with moderate difficulty in which the competitive classifiers report only about 82% test accuracy on the derived cmu dog dataset. baseline and comparison model our baseline adopts a vanilla seq2seq generator as a basic function mapper as described in section 3.1 which maps the concatenation of knowledge retrieved k and dialogue context c to dialogue response r without any control over the fallback response as well as the notion of answerability. one comparison model is derived from the previous work done by @xcite to employ an additional classifier to map the concatenation of knowledge retrieved k and dialogue context c to answerability control token a. it then follows the classical controlled text generation procedure to feed the concatenation of the knowledge, context and control token into the generator for response generation. implementation details for all the generators implemented for the baseline, comparison model and our method, we employ the state-of-the-art gpt2-based @xcite dialogue response generator dialogpt-small @xcite . we also attempted on dialogpt-medium and dialogpt-large. we found all three of them tend to respond inappropriately with fallbacks to the answerable dialogue contexts, and they report similar diversity measurements. therefore, we adopt dialogpt-small for simplicity. we use a learning rate of 5e-4, β 1 = 0.9, β 2 = 0.999 and ϵ = 1e-8. we adopt roberta-base @xcite as the answerability classifier to be used in our comparison model. we also experimented on bert-base, bert-large @xcite and roberta-large, which led to a similar accuracy. therefore, we adopt roberta-base for simplicity. for the classifier, we use a learning rate of 5e-6, β 1 = 0.9, β 2 = 0.999 and ϵ = 1e-8. since we are interested in diversity, or informativeness, we use nucleus sampling, or top-p sampling @xcite as our decoding mechanism throughout our experiments for all our baseline, comparison model, and our method, in which we set @xmath0.95 as the hyper-parameter as in the work done by @xcite . we conduct our experiments with the transformers library @xcite . evaluation metrics in this work, we mainly focus on generation diversity for answerable contexts. we also report our investigation on fallback issues for unanswerable contexts as well as the classification accuracy. we followed previous works to adopt distinct-n @xcite to investigate fallback response generation, we report the number of fallback responses replied to answerable (fr + ) and unanswerable contexts (fr -). 5 the former attains a better quality with lower values and the latter attains a better quality with higher values. for the control token generation, or answerability classification, we report the 5 the scores are obtained by keyword detection. ## conclusion building a grounded dialogue agent is an important research line. however, most previous works have overlooked the situation when the retrieved knowledge cannot help the agent answer the dialogue. under such a situation, fallback answers should be appropriately presented, and such incorporation should not degrade the informativeness in responses to answerable contexts. we demonstrate that a standard language model fails to handle this situation well and degrades the informativeness of responses to answerable dialogue contexts. controlled text generation can be a solution that rigorously replies with fallback to unanswerable contexts. however, naively incorporating controlled text generation still hurts informativeness for the answerable contexts. we propose a novel end-to-end framework that leverages the understanding power of language models for answerability classification that steps into controlled response generation naturally in an autoregressive manner. our experimental results from both automatic and human evaluation demonstrate that our method achieves higher accuracy on dialogue answerability classification than the competitive models specially designed for language understanding. this improves the informativeness for answerable dialogue contexts while still maintaining the ability to reply with fallback to unanswerable dialogue contexts.