I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling (2021)

Abstract: To quantify how well natural language understanding models can capture consistency in a general conversation, we introduce the DialoguE COntradiction DEtection task (DECODE) and a new conversational dataset containing both human-human and human-bot contradictory dialogues. We show that: (i) our newly collected dataset is notably more effective at providing supervision for the dialogue contradiction detection task than existing NLI data including those aimed to cover the dialogue domain; (ii) Transformer models that explicitly hinge on utterance structures for dialogue contradiction detection are more robust and generalize well on both analysis and out-of-distribution dialogues than standard (unstructured) Transformers. We also show that our best contradiction detection model correlates well with human judgments and further provide evidence for its usage in both automatically evaluating and improving the consistency of state-of-the-art generative chatbots.

## introduction recent progress on neural approaches to natural language processing @xcite , and the availability of large amounts of conversational data @xcite have triggered a resurgent interest on building intelligent open-domain chatbots. newly developed end-to-end neural bots @xcite @xcite are claimed to be superior to their predecessors @xcite using various human evaluation techniques @xcite @xcite ) that aim to give a more accurate measure of what makes a good conversation. while the success is indisputable, there is still a long way to go before we figure 1 : contradictory dialogues contained in our new decode dataset. the main train, valid and test sets contain human-written dialogues with deliberate contradictions (example at top), and an out-of-domain test set consists of labeled human-bot dialogues (bottom), involving state-of-the-art bots @xcite . when interacting with chatbots, people carry over many of the same expectations as when interacting with humans @xcite . selfcontradictions by these bots (see fig. 1 , bottom) are often jarring, immediately disrupt the conver-sational flow, and help support arguments about whether generative models could ever really understand what they are saying at all @xcite . from a listener's perspective, such inconsistent bots fail to gain user trust and their long-term communication confidence. from a speaker's perspective, it violates the maxim of quality in grice's cooperative principles @xcite -"do not say what you believe to be false." hence, efforts on reducing contradicting or inconsistent conversations by open-domain chatbots are imperative. prior works @xcite characterized the modeling of persona-related consistency as a natural language inference (nli) problem @xcite , and constructed a dialog nli dataset based on persona-chat @xcite , but so far state-of-the-art chatbots @xcite have not been able to make use of nli techniques in improving dialogue consistency. overall, the challenge remains that we are still unable to answer the simple yet important question-"how good are we at modeling consistency (including persona, logic, causality, etc.) in a general conversation?". the inability to measure this obscures to what degree building new modules or techniques can in turn help prevent contradicting responses during generation. seeking to answer this question, we introduce the dialogue contradiction detection task (de-code) 1 and collect a new conversational dataset containing human written dialogues where one of the speakers deliberately contradicts what they have previously said at a certain point during the conversation. we also collect an out-of-distribution (ood) set of dialogues in human-bot interactive settings which contain human-labeled selfcontradictions made by different chatbots. we then compare a set of state-of-the-art systems, including a standard unstructured approach and a proposed structured approach for utilizing nli models to detect contradictions. in the unstructured approach, a transformer nli model directly takes in the concatenation of all utterances of the input dialogue for prediction, following the paradigm of nlu modeling. in the structured approach, utterances are paired separately before being fed into transformer nli models, explicitly taking account of the natural dialogue structure. results reveal that: (1) our newly collected 1 decode dataset and code are publicly available at https://parl.ai/projects/contradiction . finally, with such improvements on the contradiction detection task, we show that our best resulting detector correlates well with human judgments and can be suitable for use as an automatic metric for checking dialogue consistency. we further provide evidence for its usage in improving the consistency of state-of-the-art generative chatbots. ## related work several prior works on improving dialogue consistency have explored using direct modeling of the dialogue context in generation algorithms. the modeling can be implicit where the dialogue consistency-related information like style @xcite , topics, or personal facts are maintained in distributed embeddings @xcite , neural long-term memories @xcite , hierarchical neural architecture @xcite , latent variables @xcite , topical attention @xcite , or even self-learned feature vectors @xcite . some works have grounded generation models on explicit user input @xcite , or designated personas @xcite . although, improvements on automatic generation metrics were often shown on guided response generation based on the consistency modeling, the issue of contradiction has never been resolved, nor have generally applicable methods to gauge the consistency improvements been developed. further, simply scaling models has not made the problem go away, as is evident in the largest chatbots trained such as blenderbot with up to 9.4b parameter transformers @xcite . more similar to our work is utilizing nli models in dialogue consistency. @xcite attempted to use entailment models trained on syn-thetic datasets for dialogue topic coherence evaluation. particularly, @xcite constructed the dialogue nli dataset and @xcite utilized it to try to reduce inconsistency in generative models via unlikelihood training in a preliminary study that reports perplexity results, but did not measure actual generations or contradiction rates. we note that the dialogue nli dataset is only semi-automatically generated, with limited coverage of only persona-chat data @xcite , whereas our decode is human-written and across multiple domains. our task also involves logical and context-related reasoning beyond personal facts. we show that transfer of de-code is subsequently more robust than dialogue nli on both human-human and human-bot chats. ## models to model the dialogue consistency task, we first employ some of the techniques used in nli sequenceto-label modeling, where the input is a pair of textual sequences and the output is a label. the benefit of such modeling is that we can directly make use of existing nli datasets during training. however, unlike previous work @xcite that directly utilized nli models giving a 3-way output among "entailment", "contradiction", and "neutral", we modify the model with a 2-way output between "contradiction" and "non-contradiction" (either "entailment" or "neutral") labels, as our task is centered around the detection of inconsistency. more formally, we denote the model as ŷpred = f θ (c, u), where ŷpred is the prediction of the label y, i.e. whether the textual response u contradicts some textual context @xmath0, u 1 , ..., u n-1 }, and θ are the parameters of the model. ## conclusion we introduce the dialogue contradiction detection task (decode) and a new conversational dataset containing both human-human and humanbot contradictory dialogues. training models on decode achieves better performance than other existing nli data by a large margin. we further propose a structured utterance-based approach where utterances are paired before being fed into transformer nli models to tackle the dialogue contradiction detection task. we show the superiority of such an approach when transferring to out-ofdistribution dialogues compared to a standard unstructured approach representative of mainstream nlu modeling. we further show that our best contradiction detector correlates with human judgments, and provide evidence for its usage in both automatic checking and improving the consistency of state-of-the-art generative chatbots.