Group-wise Contrastive Learning for Neural Dialogue Generation (2020)

Abstract: Neural dialogue response generation has gained much popularity in recent years. Maximum Likelihood Estimation (MLE) objective is widely adopted in existing dialogue model learning. However, models trained with MLE objective function are plagued by the low-diversity issue when it comes to the open-domain conversational setting. Inspired by the observation that humans not only learn from the positive signals but also benefit from correcting behaviors of undesirable actions, in this work, we introduce contrastive learning into dialogue generation, where the model explicitly perceives the difference between the well-chosen positive and negative utterances. Specifically, we employ a pretrained baseline model as a reference. During contrastive learning, the target dialogue model is trained to give higher conditional probabilities for the positive samples, and lower conditional probabilities for those negative samples, compared to the reference model. To manage the multi-mapping relations prevalent in human conversation, we augment contrastive dialogue learning with group-wise dual sampling. Extensive experimental results show that the proposed group-wise contrastive learning framework is suited for training a wide range of neural dialogue generation models with very favorable performance over the baseline training approaches.

## introduction open-domain human-machine dialogue systems, especially the generation-based genre, have attracted extensive attention recently. typically, following the neural encoder-decoder paradigm, contemporary dialogue generation models @xcite @xcite @xcite ), * work done at jd.com. more often than not, are trained with maximum likelihood estimation (mle) principle to mimic human context-response pairs in the training corpus. while notable gains have been achieved under this learning framework, prior art @xcite @xcite suggests that naive mle objective used for training neural dialogue generation models is not that effective enough and tends to result in issues like dull response generation. by optimizing the likelihood of training dialogues, neural models are inclined to assign high probabilities to "safe" responses, due to the fact that vacuous responses like "i don't know" are of relatively high frequencies in conversational datasets @xcite . one promising training framework for neural dialogue generation is adversarial learning @xcite , where a discriminator provides rewards for the generator by contrastively distinguishing dialogues as humangenerated or machine-generated. however, the learning ability of gans in text is drastically limited due to training instability and model collapse @xcite . first, the discriminator is usually unlikely to be fooled very easily, and the generator can hardly learn from those ineffective rewards. second, the generator is sometimes encouraged to mimic the highfrequency generic responses in the training corpus, because in some cases, the discriminator fails to distinguish a good response from a bad one: it can easily recognize contentful but less-grammatical responses as machine-generated, yet treat those human-generated dull responses as the oracle. in this paper, we introduce contrastive learning @xcite into dialogue generation, where the model explicitly perceives the difference between the wellchosen positive and negative utterances. from the perspective of contrastive learning, the discrim- matching degree low high we are both foodies! he would only lay on the porch at night. i enjoy swimming and reading. how about you? i like to read john grisham books. i just hang out with my 10 cats. okie dokie. here you go. i do not like cooking. my hobby is to watch sports. thanks for the offer though. i like it, too! @xmath0 reading is my favorite hobby c + < l a t e x i t s h a 1 _ b a s e 6 4 = " o r o g v i w m 3 q s r r q n v y p z d l t n d 6 f @xmath1. instead, this work deems highly-matched context-response pairs as positive samples and mismatched training pairs as negative samples. in particular, we utilize a pretrained baseline model as a reference. during contrastive learning, for context c and its response r, the target dialogue model is trained to give higher conditional probabilities p(r|c) for the positive samples, and lower conditional probabilities for those negative samples, compared to the reference model. this training paradigm encourages the model to pull the positive data points together and push apart the negative samples, as exemplified in figure 1 . as a result, our proposed training scheme explicitly takes the semantic associations and differences among training examples into account for dialogue modeling. besides, by contrastively characterizing the distinctions relative to a strong reference, our method implicitly enhances the distinctiveness of the generated responses as well, and ensures that the overall performance of the target model is not inferior to the reference. contrastively learning from one pair of positive and negative samples is quite straightforward, however, multi-mapping relations prevail in humanhuman conversations, where there exist multiple appropriate responses for a given context, and a response sometimes fits well to several contexts, known as one-to-many and many-to-one relations. such complex multi-mapping relations are over-looked in previous learning framework, which hampers effective dialogue response learning. furthermore, if a potential highly-matched utterance pair is treated as the negative sample or an outlier is used as the positive sample, the model may be confused. therefore, in order to consider the multi-mapping phenomenon in human conversations and remedy the potential problematic false learning samples, and enhance the training stability, we augment contrastive learning with group-wise dual sampling, where groups of positive and negative instances are sampled regarding both the context and the response, respectively. to further depict subtle differences between instances in the group, we adapt the instance importance with the matching scores, and optimize the weighted loss. we show an illustration case to understand our learning framework in figure 1 . given a training context-response pair (c, r), for context "what are your hobbies? i love to cook", multiple highlymatched responses are organized as the positive samples r + , and the mismatched utterances are deemed as the negatives r -. on the dual direction, regarding the response "reading is my favorite hobby", multiple sampled context utterances are similarly divided into c + and c -. compared with the reference baseline, the target dialogue model is trained to give higher generation probabilities for positive instances, (c, r + ) and (c + , r), and lower probabilities for negatives (c, r -) and (c -, r). by this mean, the target model is actually induced to pull the positive sample pairs together and push the mismatched pairs apart, and thus learns from the distinctions between the positives and negatives. the proposed group-wise contrastive learning framework is suited for training various neural dialogue generation models. we conduct extensive studies on three large-scale conversation datasets using four popular dialogue models to assess the proposed approach. the experimental results confirm the effectiveness of our learning framework with very favorable performance over the baseline training approaches 1 . ## related work learning methods for dialogue generation typically, state-of-the-art neural dialogue generation models adopt maximum likelihood estimation (mle) as their learning approach, maximizing loglikelihood of model parameters on the training data. though effective, well-known issues, including the notorious general dull response problem, are reported by prior art @xcite on dialogue models trained with mle. alternative dialogue learning approaches are proposed to tackle the issues. li et al. (2016a); zhang et al. (2018c) introduce the maximum mutual information as the objective function to promote response diversity. techniques of reinforcement learning (rl) and adversarial learning have been introduced into dialogue generation by @xcite to better approximate the optimization goal of dialogue models. conditional variational framework @xcite @xcite has also shown a promise in dialogue generation. @xcite further introduce a conditional wasserstein autoencoder that employs gan @xcite to model the multimodal latent structures. @xcite design a multi-curriculum learning frame-work to facilitate the dialogue model training. contrasted with existing learning methods for dialogue generation, the proposed framework in this work encourages the model to learn from the difference between well-chosen contrastive pairs, which explicitly models the multi-mapping relations in conversation and promotes the distinctiveness of the generated responses in the meantime. ## conclusion in this work, we propose a group-wise contrastive dialogue learning approach, that explicitly perceives the difference between the well-chosen positive and negative utterances, and manages the multi-mapping relations in human conversations simultaneously. given a training instance, the proposed learning framework first organizes a group of positive samples and negative samples regarding context-response matching degrees, and then trains the target dialogue model to give higher conditional probabilities for positive pairs and lower probabilities for the negatives. extensive experimental results show that the proposed learning framework brings a solid favorable performance boost amongst various strong baseline approaches.