CDC onv: A Benchmark for Contradiction Detection in C hinese Conversations (2022)

Abstract: Dialogue contradiction is a critical issue in open-domain dialogue systems. The contextualization nature of conversations makes dialogue contradiction detection rather challenging. In this work, we propose a benchmark for Contradiction Detection in Chinese Conversations, namely CDConv. It contains 12K multi-turn conversations annotated with three typical contradiction categories: Intra-sentence Contradiction, Role Confusion, and History Contradiction. To efficiently construct the CDConv conversations, we devise a series of methods for automatic conversation generation, which simulate common user behaviors that trigger chatbots to make contradictions. We conduct careful manual quality screening of the constructed conversations and show that state-of-the-art Chinese chatbots can be easily goaded into making contradictions. Experiments on CDConv show that properly modeling contextual information is critical for dialogue contradiction detection, but there are still unresolved challenges that require future research.

## introduction large-scale pre-training for dialogue generation @xcite has advanced the development of engaging and humanlike dialogue systems. unfortunately, state-ofthe-art open-domain chatbots, such as blenderbot @xcite , eva @xcite and plato @xcite , still often behave inconsistently with their role or identity and produce utterances that are self-contradictory non-contradiction 𝑢 1 : 你喜欢吃面条吗？(do you like noodles?) 𝑏 1 : 我喜欢吃米饭! (i love to eat rice!) 𝑢 2 : 你不喜欢面条吗? (don't you like noodles?) 𝑏 2 : 不喜欢! (i dislike!) contradiction 𝑢 1 : 那你喜欢什么小动物呢? (what animals do you like?) 𝑏 1 : 狗,猫 (dogs, cats) 𝑢 2 : 你不喜欢什么小动物? (what animals do you dislike?) 𝑏 2 : 不喜欢猫,其他的都喜欢 (i dislike cats. i like all the other animals) dialogue contradiction detection has shown to be an effective means to improve the consistency of chatbots @xcite , which, however, is always a challenging task. specifically, the contextualization nature of conversations indicates the necessity of considering and modeling contextual information. for instance, in the "contradiction" example in figure 1 , b 2 does not explicitly contradict b 1 . however, given u 1 , the actual meaning of b 1 should be "i like dogs, cats" and b 1 and b 2 are thus contradictory. in contrast, in the "non-contradiction" example, while b 1 and b 2 seem inconsistent ("love" vs. "dislike"), b 2 actually means "i dislike noodles" considering the dialogue context. hence, b 2 is compatible with b 1 and does not make a contradiction. despite the above challenge, existing datasets for contradiction detection @xcite et al., 2019) usually only consider the textual entailment relationship between two isolated sentences @xcite , which is largely insufficient for dialogue contradiction detection due to the neglect of contextual information. a recent work @xcite crowd-sourced a dataset named de-code that contains conversations where the last utterances contradict the dialogue histories. however, decode lacks a wide coverage of typical contradiction categories, and most of its contradiction cases are written by human, which have gap with the real scenario where users trigger chatbots to make contradictions. in this work, we propose a benchmark for contradiction detection in chinese conversations, namely cdconv. it contains 12k multi-turn conversations with human-annotated contradiction labels ( §3). different from previous work (e.g., @xcite ) that only considered the contradiction to dialogue history (i.e., history contradiction), cdconv covers another two typical categories: intra-sentence contradiction and role confusion, which refer to that a reply contradicts itself and that a reply confuses the speaker's role, respectively. since the cases of non-contradiction and contradiction in natural human-bot conversations are extremely unbalanced ( §3, @xcite , we automatically construct the cdconv conversations combined with elaborate manual inspection ( §4.1). specifically, we first devise a series of automatic methods to generate conversations ( §4.2), which simulate the common user behaviors that trigger chatbots to make contradictions. we then conduct careful human screening and annotation for the constructed conversations to ensure the data quality ( §4.3). we validate the effectiveness of the trigger methods and show that state-of-the-art chinese open-domain chatbots (eva and plato) can be easily goaded into making contradictions ( §4.4). finally, we evaluate popular chinese pre-trained models on cdconv ( §5). results show that properly modeling contextual information is critical for dialogue contradiction detection. however, there is still much room for future research in dialogue modeling, integrating commonsense and world knowledge, and reasoning. our contributions are summarized as follows: ## related work ## categories of dialogue contradiction a conversation with n turns is formally denoted as u 1 , b 1 , . . . , u n , b n , where u k and b k denote the kth-turn utterances from the user and the chatbot respectively. we focus on whether b n makes a contradiction in the dialogue context. in the preliminary study, we manually inspected 200 multi-turn human-bot conversations with two chinese open-domain chatbots: eva @xcite and plato @xcite . on average, each conversation contains about 30 turns but only roughly 1 contradiction case. based on the inspected contradiction cases, we identify three typical categories of dialogue contradiction according to the object that b n contradicts, as intuitively illustrated by figure 3: • intra-sentence contradiction: b n is contradictory to itself. in other words, there exist two disjoint subsentences b that is, b n is more likely to be a user's reply to b n-1 rather than a bot's to u n . figure 2 provides the examples of the above three contradiction categories. they occupied 16%, 18%, and 54% in our inspected contradiction cases, 2 we note that the premise of bn making history contradiction is that bn is a bot's reply to un. however, if bn makes role confusion (i.e., bn is more likely to be a user's reply to bn-1 than a bot's reply to un), the premise of history contradiction will not hold and such a case will be judged as role confusion rather than history contradiction. 2 for detailed annotation statistics. ## conclusion in this work, we present cdconv, a benchmark for contradiction detection in chinese conversations. by simulating common user behaviors that trigger chatbots to make contradictions, we collect 12k conversations annotated with three typical contradiction behaviors. experiments show that contextual information plays an important role in dialogue contradiction detection. however, there are still unresolved challenges in cdconv, such as dialogue modeling, commonsense, knowledge grounding and reasoning. we hope that cdconv can inspire and facilitate future research in dialogue contradiction detection and consistent generation.