Multi-level Adaptive Contrastive Learning for Knowledge Internalization in Dialogue Generation (2023)

Abstract: Knowledge-grounded dialogue generation aims to mitigate the issue of text degeneration by incorporating external knowledge to supplement the context. However, the model often fails to internalize this information into responses in a human-like manner. Instead, it simply inserts segments of the provided knowledge into generic responses. As a result, the generated responses tend to be tedious, incoherent, and in lack of interactivity which means the degeneration problem is still unsolved. In this work, we first find that such copying-style degeneration is primarily due to the weak likelihood objective, which allows the model to “cheat” the objective by merely duplicating knowledge segments in a superficial pattern matching based on overlap. To overcome this challenge, we then propose a Multi-level Adaptive Contrastive Learning (MACL) framework that dynamically samples negative examples and subsequently penalizes degeneration behaviors at both the token-level and sequence-level. Extensive experiments on the WoW dataset demonstrate the effectiveness of our approach across various pre-trained models and decoding strategies.

## introduction in recent years, pre-trained language models using transformer architectures have made remarkable strides in open-domain generation tasks @xcite @xcite . however, these models still struggle with dull and repetitive outputs, a problem commonly termed as neural text degeneration @xcite . to address text degeneration in dialogue, @xcite proposed to equip interlocutors with external knowledge as additional support to enrich the informativeness of responses, which is known figure 1 : an example shows how current models make rigid use of knowledge and cause the responses incoherent with context. both t5-large and bart-large are finetuned on wow dataset with mle. as the knowledge-grounded dialogue generation (kgdg). recent methods have put much emphasis on the knowledge selection subtask, aiming to provide models with the most suitable knowledge @xcite @xcite . however, simply introducing golden knowledge as a ground source does not indeed mitigate the problem of degeneration. we observe that existing models often duplicate knowledge snippets to construct responses to meet the informativeness requirement, which can lead to unnatural and contextually incoherent responses. for example, the generated contents in figure 1 simply duplicate the provided knowledge about the "blue skies" movie, resulting in a tedious response incoherent with user's utterance. consequently, we need to consider not only the introduction of knowledge, but also how to effectively integrate it into responses. we term the superficially knowledge duplicating "knowledge regurgitation" as it mirrors how the model absorbs an entire knowledge sentence and regurgitates it in the response without genuine comprehension. this issue can be seen as a taskspecific copying-style manifestation of text degeneration. we test some pre-trained models with different parameter scales, and discover that the knowledge regurgitation is common among them. to quantify its severity, we design two automated metrics: pod and kud. based on a series of experiments, we posit that degeneration is caused by the ineffectual design of the training objective, which allows models to "cheat" the mle objective by merely duplicating knowledge snippets to construct responses. the model, therefore, improperly converges to superficial pattern matching based on overlap, given that knowledge sentences share spurious correlations (token overlaps) with groundtruth responses and exhibit high semantic fluency. in this paper, to tackle the aforementioned issues, we propose a novel approach known as multi-level adaptive contrastive learning. this method effectively mitigates knowledge regurgitation through contrastive training at both the token and sequence levels. at the token level, we enhance the negative training paradigm @xcite by employing a dynamic negative token sampling method and reweighting the unlikelihood loss according to model sensitivity. in this way, our approach penalizes tokens that continuously appear in the knowledge but are not the targets to cut off the potential shortcuts. at the sequence level, we first employ a group beam search strategy @xcite to sample negative responses from the degenerator's predictions, then use a novel metric as an oracle function to score them. finally, samples demonstrating obvious degeneration are selected as hard negatives for the infonce loss to distance them from their prefix in the representation space. it can expose the model to potential degeneration mistakes that happen at the inference stage and help the model learn to avoid them. our contributions are summarized as follows: ## preliminaries in this section, we provide a brief introduction to the kgdg task and the unlikelihood training (ut) loss, which serves as the foundation for our tokenlevel contrastive learning loss. ## knowledge regurgitation although current pre-trained language models have demonstrated robust performance in generating fluent dialogue responses, they fail to align with human-like patterns of knowledge integration. we conduct a series of preliminary experiments to demonstrate the presence of text degeneration problems in some pre-trained language models of various sizes. specifically, we quantify the severity of this issue by comparing human and model performance based on our newly proposed metrics. the results are presented in dup-n and proportion of longest common subsequence tokens (plcs) are mainly used to measure knowledge snippets duplication. dup-n represents the proportion of samples with n-grams co-occurring in knowledge and response and is computed as follows: for plcs, it is calculated as ratio of the length of longest common sub-sequence (lcs) of response and knowledge to the length of response as follows: the aforementioned metrics approximately measure the frequency of a degenerated knowledge utilization pattern where only snippets of the provided knowledge are inserted into a generic response without substantial integration. to highlight the gap in knowledge utilization patterns between humans and models, we further employed metrics related to the precision of knowledge grams (mkp-n), which calculates the mean ratio of knowledge tokens in response to all the response tokens. it is performed as follows: a comparison of the performance of each mlefinetuned plm with human dialogue, presented in in chit-chat scenarios, it is important to strike a balance with the number of knowledge tokens used in responses. excessive knowledge tokens can reduce the interactivity of the response, potentially diminishing users' desire to continue the conversation. from the data presented in ## our approach we propose a novel multi-level contrastive learning method that dynamically samples negative examples and subsequently penalizes degeneration behaviors at both the token and sequence levels. token-level contrastive learning the motivation behind the token-level loss is to break the generation inertia more effectively (refer to the figure 3 for a detailed explanation). our improvements over basic unlikelihood training method are mainly twofold: dynamic negative sampling and dynamic unlikelihood loss reweighting. the detailed loss function is presented in the following formula: we empirically investigate the selection strategy of negative examples in our study. given the objective of the knowledge-grounded dialogue task is to effectively integrate knowledge into the response, and that these knowledge tokens are essential potential candidates, punishing all knowledge without distinction is not appropriate. our goal is to eliminate shortcuts leading to knowledge duplication while keep the knowledge integrating ability. to achieve this, we propose to dynamically select negative tokens based on the model's sensitivity, thereby reducing the probability of tokens where the model is more likely to err. specifically, we adopt a strategy of selecting the knowledge token with the highest prediction probability as the negative token. we also explored other strategies, such as sampling by probability, random selection, but the adopted strategy outperforms the others. we conjecture that this is due to our strategy targeting the prediction chains of the knowledge snippet (shortcuts), effectively disrupting it. as for the dynamic reweighting strategy, we considered two reasons. firstly, @xcite highlight the importance of applying differentiable weights to individual token losses by proposing token focal loss inspired by @xcite 's work. in light of this, we enhance the unlikelihood loss by introducing an additional control parameter β(y c ) to dynamically reweight the punishment strength for different tokens. by doing so, we suppress the gradients of easy tokens while amplifying the gradients of hard tokens, leading to faster and improved convergence during training. secondly, @xcite expressed concern that the vanilla negative training method might cause the model to decrease the probability of the target token p * to reduce the gradient norm |∇l a | (10) during the final stages of training, particularly when α is excessively large (refer to the appendix b for a detailed explanation). our approach mitigates the problem of that inverse optimization and allows the loss function to converge to a lower value as the introduction of β(y c ) facilitates the gradual decrease of the weight of l ul (2). ground-truth token (@xmath0: sequence-level contrastive learning regarding the sequence-level contrastive loss, an et al. (2023) point out that contrastive learning provides decoder encoder [context]+[knowledge] 𝒛 𝒙 [response] y 𝒛 𝒚 group beam search lcs rank degenerator 𝒛 𝒚 𝟏 ′ 𝒛 𝒚 𝟐 ′ 𝒛 𝒚 𝟑 ′ 𝒛 𝒚 𝒎 ′ … [response] y' m decoder pull frozen trainable representation space encoder decoder [bos] 𝑦 𝑡-3 𝑦 𝑡-2 𝑦 𝑡-1 𝑦 1 𝑦 𝑡-2 𝑦 𝑡-1 𝑦 𝑡 gt knowledge others vocabulary gt knowledge others vocabulary … [context] … … … … … 𝑘 𝑖-1 𝑘 𝑖 𝒌 𝒊+𝟏 … … … … (𝑘 𝑖-1 ) (𝑘 𝑖 ) (𝑘 𝑖-2 ) (𝑘 𝑖-1 ) (𝑘 𝑖 ) 𝒌 𝒊+𝟏 ෝ 𝒚 𝒕 ෝ 𝒚 𝒕 𝒌 𝒊+𝟏 token-level sequence-level helpful error-prone [context]+[knowledge] [knowledge] owing to the generation inertia, that is, the shortcut, the model predicts an exceptionally high probability for the next knowledge token, which surpasses the probability of the ground-truth token. macl breaks the knowledge snippet chain effectively and outputs more reasonable distributions. z x represents the source sequence (context and knowledge), and z y stands for its target sequence (response). the feature representations are derived from pooling the outputs of the encoder (source sequence) or decoder (target sequence), which are both differentiable in this context. a novel solution to alleviate the exposure bias problem. @xcite suggest that the degeneration problem is a result of exposure bias, which motivates us to address this issue by leveraging the sequence-level contrastive learning method during the training phase. by exposing the model to negative targets exhibiting degeneration, we aim to help the model learn to avoid predicting them. @xmath1,zy ) y′∈b e cos(zx,zy′) + y -∈h µe cos(zx,z y -) , following the collection procedure, we drive the distance between the source sequence representation and the negative target sequence representation in a contrastive way. the l seq loss provides the model with a negative supervision signal on the shortcut paths that generated these sequences. both hard negative samples and from-batch negative samples are retained, but we assign higher weights to the former to emphasize their importance. the final objective function is defined as: see the appendix e for the pseudo-code of the entire training procedure. ## conclusion in this paper, we investigate a distinctive degeneration phenomenon in knowledge-grounded dialogue generation referred to as knowledge regurgitation, which is prevalent in pre-trained language models. to address this challenge, we present a novel solution called multi-level adaptive contrastive learning (macl). our approach tackles the problem by dynamically sampling negative examples and penalizing degeneration behaviors at both the token-level and sequence-level. experiments on the wow dataset demonstrate that our approach significantly mitigates knowledge regurgitation.