Combining Cognitive Modeling and Reinforcement Learning for Clarification in Dialogue (2020)

Abstract: In many domains, dialogue systems need to work collaboratively with users to successfully reconstruct the meaning the user had in mind. In this paper, we show how cognitive models of users’ communicative strategies can be leveraged in a reinforcement learning approach to dialogue planning to enable interactive systems to give targeted, effective feedback about the system’s understanding. We describe a prototype system that collaborates on reference tasks that distinguish arbitrarily varying color patches from similar distractors, and use experiments with crowd workers and analyses of our learned policies to document that our approach leads to context-sensitive clarification strategies that focus on key missing information, elicit correct answers that the system understands, and contribute to increasing dialogue success.

## introduction as dialogue systems move into richer domains, they must increasingly cope with situations where objects don't have generally-known names, where people describe concepts in incompatible ways, and where language use is fundamentally creative and uncertain @xcite . people succeed in such situations thanks to their collaborative interactions, which let the speaker and the audience share responsibility for reaching a satisfactory mutual understanding @xcite . this paper contributes to a broader project of building dialogue systems that can do the same. we focus specifically on the problem of asking targeted, effective clarification questions with creative language. the difficulty of this problem is highlighted by the interactions shown in figure 1 , taken from the human-subjects evaluations reported in section 6. in these dialogues, our system (playing the role of matcher in the referential communication task of @xcite ) uses varying clarification questions to follow up initial color descriptions provided by crowd workers (playing the role of director), and is thereby able to successfully distinguish the target color patches from their contextual distractors. in formulating such questions, the system must resolve the specific uncertainty it faces in context; there are many options about how to present that uncertainty to users. meanwhile, depending on the context and the form and content of the clarification question, the director may not understand the question as intended-as both human-human conversations in this domain @xcite and evaluation data with our system confirm. thus, for the system to synthesize a question with a good outcome, it needs accurate models of the interpretations, interactive strategies and responses of human interlocutors, to guide its behavior. we meet this challenge by integrating a reinforcement learning approach to dialogue policy with cognitive models to describe the actions and outcomes available to the system. in particular, our work relies on a new, probabilistic model of contributions to dialogue, as planned by the user and understood by the system. we construct this model by analyzing human behavior and profiling system performance on a benchmark dataset of human-human conversations, drawing on data collection and modeling work from the recent dialogue literature @xcite . we use this model at planning time to roll out simulated interactions that track user deliberation, anticipate user dialogue moves at the level of utterance content, and predict task outcomes. we use deep q-learning @xcite to estimate the effectiveness of different actions as a function of dialogue state. our approach culminates in a context-sensitive policy that decides whether and how to present clarification options to users, contingent on the ambiguity of user input and the predicted outcomes of different resolution strategies. our work is the first to intelligently deploy a range of clarification strategies, selected to reduce system uncertainty in a targeted and effective way, and exploiting the creative use of diverse vocabulary to describe alternatives. we explain the contrast with previous approaches in section 2. we describe the experimental framework that we build on and extend in section 3, and describe our reinforcement learning approach in section 4. we demonstrate the effectiveness of our approach with two kinds of experiments. in section 5, we analyze reinforcement learning outcomes to document the success of the learning process and characterize the intuitively satisfying policies that result. in section 6, we describe human-subjects evaluations of these strategies that indicate that adaptive clarification improves the system's success and that reinforcement learning improves user satisfaction. these experiments also enable us to quantify the match between simulation predictions and observed human behavior. together, these evaluations substantiate the claim that our approach combines effective communication, targeted clarification, and expressive vocabulary. ## related work our work contributes to the general project of grounding in interactive systems-making sure that system and user have a shared understanding of content in conversation; see @xcite or @xcite . we follow the broadly decision-theoretic approach initiated in the early 2000s by @xcite , @xcite and others. grounding has been a major focus of research in spoken dialogue systems because of the uncertainties associated with speech recognition results, making confirmation strategies particularly important. optimal decisions have long been guided by user simulations that capture the realistic dynamics of slot filling dialogues; see @xcite for review. however, because we work with written rather than spoken language, our concern is uncertainty at deeper levels of interpretation, including such new dimensions of user simulation as the flexible vocabulary and syntax that users adopt, their strategic variability in producing and interpreting utterances, and their deliberation in coordinating with their interlocutors. while a range of work has attempted to model and learn about such referential and speech act ambiguity from dialogue data, including @xcite and devault and stone (2009), these models have not been used to drive decision-theoretic approaches to clarification. reinforcement learning in open-domain referential communication has instead focused on simple decisions, such as whether to wait or move forward in incremental dialogue @xcite or asking yes/no questions to resolve ambiguity in visual dialogue @xcite @xcite . corpus-based modeling work has shown, however, that natural strategies for collaborative reference are typically substantially more flexible @xcite @xcite . our work aims to learn correspondingly more flexible policies. we argue elsewhere (khalid et al., 2020) that formal models of discourse coherence give important resources to do this. although we build on that work here, our prior work has not addressed the problem of learning context-sensitive clarification strategies. appelgren and lascarides (2020) also consider semantic grounding of color terms and exploit formal models of discourse coherence to do so; however, their work so far assumes dialogue policies (and primitive motor skills) to be fixed and known in advance. in rolling out dialogue transcripts using detailed models of interaction, our work may recall utterancelevel models of strategic negotiation, such as @xcite . an important contrast is that these models learn by self-play: two learning agents interact with and adapt to one another. this is not a good fit for human-agent collaboration since agents can easily learn policies by self-play that work well for other agents but cannot be interpreted by human users. this explains the unique role of cognitive modeling in our approach. ## background: task, models and architecture we work in a constrained referential communication domain, inspired by the human subjects experiments of clark and colleagues @xcite . two participants communicate via text chat. in each round, participants both see a set of three color patches, arrayed in possibly different orders. one participant, the director, gets an indication of a target patch to identify. the other participant, the matcher, can click on the patches. the director and matcher can freely exchange text messages to coordinate on the target object. the round is successful if the matcher selects the indicated target. @xcite crowdsource a large dataset of human-human conversations in this setup; we call this the colors-in-context (cic) dataset. figure 2 shows a sample interaction from cic. note the matcher's use of a twoalternative comparative clarification question to pin down a more precise understanding of the director. ## modeling dialogue decisions like many reinforcement learning frameworks, our approach starts from the models of dialogue state and action described in section 4.1 and induces a policy that chooses an action based on dialogue state. choice is based on outcome; we use the user simulation described in section 4.2 to predict the outcomes of actions, and measure the success of the dialogue based on the reward function described in section 4.3. we contrast two configurations: a basic one invites the system to choose whether to ask a confirmatory clarification question or to proceed with its best interpretation; the more substantive one invites the system to choose one of three forms of clarification question: a confirmatory clarification question cq1 involving one description a? (e.g., do you mean the mauve?), a disjunctive clarification question cq2 involving two alternatives a or b? (e.g., bright purple or pink?), and an exhaustive clarification question cq3 describing all the objects in the context a or b or c? (e.g., lime green or grey green or green?). ## rl experiment to optimize the system's policy, we use the dqn algorithm @xcite . in dqn, we have two neural nets: a policy network, θ p , and a target network, θ t . both networks estimate the expected long-term reward of taking action a in state s, but the policy network is adjusted dynamically via backpropagation during training while the target network updates more slowly. we use experience replay memory. we repeatedly sample actions as preferred by the policy network under an ε-greedy policy with exponential decay. this gives a transition from state s to s and an immediate reward r(s, a, s ). this transition is inserted into a experience replay buffer so that a mini-batch of transitions can be sampled from it for training. we make sure each sampled mini-batch always includes the most recent transition @xcite . in training, the target network is used to get an estimate over the future utility of the next state s , which is used to minimize the temporal difference error δ of the policy network: ## human evaluation we ran human evaluation experiments to assess how effective learned policies were in human interactions. we are interested not only in task success, but in the kinds of questions the system asks, the kinds of answers it gets, and users' impressions of the system. protocol. this study was conducted with the approval of our human subjects review committee. we recruited 60 subjects through amazon mechanical turk. participants were all us citizens, gave written consent, and were compensated at an estimated rate of usd 15 an hour. each subject played 4 games with the system out of which 2 were selected from the close condition, 1 from the far condition and 1 from the split condition. after each trial, subjects were asked to rate the performance of the system on a scale of 0 to 5 and leave us feedback. we conducted experiments with a baseline system, which clarifies whenever the probability of the most likely referent is less than 0.95, as well as with the two rl systems reported in section 5. in comparing the simulation model to actual performance, we observed that further breakdowns are necessary to capture the inherent task difficulty. the rl model expects relatively flat performance across the different conditions: 0.884 for close, 0.817 for far and 0.827 for split. in fact the differences are much greater-a natural direction for improvement as we iterate this methodology. ## discussion and conclusion in this paper, we have shown how a model of user utterances, capturing word choice and anticipating both user errors and system difficulties, can be used to optimize the fine-grained interactive strategy to resolve interpretive ambiguities in referential communication. the work has a number of limitations. it captures only some of the clarification strategies people expect in this domain, and doubtless more sophisticated domains involve more complex referential strategies and problem solving. even for color swatches, an important missing case concerns utterances that compare referents rather than characterizing them (brighter rather than bright). the user simulation is only roughly data driven and captures only some of the factors that influence speaker choice; the reward function likewise represents a coarse approximation to user satisfaction. ultimately, we would like the system to derive its user models from its own interactions and learn the features that predict user satisfaction and user choice. more generally, this work is only an initial step towards more flexible learned models of situated, collaborative interaction, and applying the technique to a broader range of problems, including explanation, diagnosis and negotiation, is crucial for the next generation of dialogue applications. we are hopeful about exploring such directions in future work.