Mitigating Contradictions in Dialogue Based on Contrastive Learning (2022)

Abstract: Chatbot models have achieved remarkable progress in recent years but tend to yield contradictory responses. In this paper, we exploit the advantage of contrastive learning technique to mitigate this issue. To endow the model with the ability of discriminating contradictory patterns, we minimize the similarity between the target response and contradiction related negative example. The negative example is generated with learnable latent noise, which receives contradiction related feedback from the pretrained critic. Experimental results show that our method helps to avoid contradictions in response generation while preserving response fluency, outperforming existing methods on both automatic and human evaluation.

## introduction in recent years, with the advent of large training corpora and pretrain technology, chatbot models have evolved considerably in open domain @xcite . current chatbots have achieved surprising results in generating fluent, engaging, informative responses, but still occasionally generate responses that are contradictory with history when interacting with human @xcite . such contradiction issues are often jarring and severely disrupt communication. therefore, it is essential to reduce contradiction for chat-bots in multi-turns dialogues. previous work @xcite ) proposes to use the paradigm of rl to mitigate the gap between the training and contradiction avoiding objective. however, the rl-based methods are easy to degrade in deep neural network @xcite , leading to the decoder generates responses that deviate from human language @xcite . other method @xcite aims to address dialogue logical contradictions via unlikelihood training @xcite . while they reduce the probability of the labeled contradicting responses, it is less generalizable to different conversation scenarios with the limited coverage of labeled contradicting data. we argue that one of the reasons behind contradiction is that model lacks the ability to identify contradictory behavior clearly. as shown in fig. 1 , the large pretrained chatbot blenderbot @xcite still has high similarity between the correct and contradictory responses in embedding space. chatbots are likely to cause contradictions when probed with unusual conversations during inference @xcite , while they are commonly trained to mimic human context-response pairs under the teacher-forcing algorithm @xcite . without being exposed to incorrect and contradictory context-response pairs, chatbots fail to learn the ability that discriminating contradictory response patterns directly, which hurts its robustness to avoid contradiction. to tackle this challenging issue, we propose a novel method to mitigate contradiction via contrastive learning, namely mccl. our method explicitly perceives the difference between the self-contradiction negative example and semanticaligned positive example. instead of utilizing welllabeled contradicting examples @xcite , we generate a self-contradiction negative example with a learnable latent noise. to capture contradic-tion actions, we employ the policy gradient method for rewarding the latent noise based on the feedback from a pre-trained critic. furthermore, we construct an additional positive example by adding a small perturbation. the positive example has aligned semantic with the original context, which devotes to the training stability and robustness. overall, our contributions are summarized as follows: 1) to mitigate contradictions in dialogue, we propose a novel method named mccl, which contrasts target response with negative pairs, to make chatbot models discriminate and refrain from contradictory response patterns. 2) experiment results show that our method performs better than baselines in automatic metrics and manual evaluation, especially in contradiction score. ## conclusion in this paper, we propose a new method named mccl to mitigate the contradiction problem in open domain chatbots. our method minimizes the similarity between the target response and selfcontradiction negative example, and maximizes the similarity with semantic-aligned positive example. experiment results show that our contrastive loss helps to avoid contradiction and obtain better response generation metrics on two different datasets. in the future, we will investigate how to improve the interpretability of negative examples. response generation. in proceedings of the 2017 conference on empirical methods in natural language processing, pages 2140-2150. sean welleck, ilia kulikov, stephen roller, emily dinan, kyunghyun cho, and jason weston. 2019. neural text generation with unlikelihood training. in international conference on learning representations. adina williams, nikita nangia, and samuel bowman. 2018. a broad-coverage challenge corpus for sentence understanding through inference. in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long papers), pages 1112-1122. ronald j williams. 1992. simple statistical gradientfollowing algorithms for connectionist reinforcement learning. machine learning, 8(3):229-256. ronald j williams and david zipser. 1989. a learning algorithm for continually running fully recurrent neural networks. neural computation, 1(2):270-280. saizheng zhang, emily dinan, jack urbanek, arthur szlam, douwe kiela, and jason weston. 2018. personalizing dialogue agents: i have a dog, do you have pets too? in proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 2204-2213. tiancheng zhao, kaige xie, and maxine eskenazi. 2019. rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 1208-1218.