Adaptive Bridge between Training and Inference for Dialogue Generation (2021)

Abstract: Although exposure bias has been widely studied in some NLP tasks, it faces its unique challenges in dialogue response generation, the representative one-to-various generation scenario. In real human dialogue, there are many appropriate responses for the same context, not only with different expressions, but also with different topics. Therefore, due to the much bigger gap between various ground-truth responses and the generated synthetic response, exposure bias is more challenging in dialogue generation task. What’s more, as MLE encourages the model to only learn the common words among different ground-truth responses, but ignores the interesting and specific parts, exposure bias may further lead to the common response generation problem, such as “I don’t know” and “HaHa?” In this paper, we propose a novel adaptive switching mechanism, which learns to automatically transit between ground-truth learning and generated learning regarding the word-level matching score, such as the cosine similarity. Experimental results on both Chinese STC dataset and English Reddit dataset, show that our adaptive method achieves a significant improvement in terms of metric-based evaluation and human evaluation, as compared with the state-of-the-art exposure bias approaches. Further analysis on NMT task also shows that our model can achieve a significant improvement.

## introduction auto-regressive models(arm) are widely used for natural language generation(nlg) tasks, such as machine translation @xcite , dialogue response generation @xcite , image captioning @xcite and video description @xcite . they utilize the encoder-decoder framework to predict the next token conditioned on the previous tokens, and minimize the crossentropy between the generation and ground-truths as their objective function. specifically, at training time, the ground-truth is utilized as the previous tokens, which forces the model directly to learn the distribution of ground truths. but at inference, the previous tokens come from the arm decoder itself, which is different from the input distribution at training time. although this discrepancy, named exposure bias, has been studied in some classic nlg tasks, such as neural machine translation(nmt) @xcite @xcite , it faces its unique challenges in dialogue response generation, the representative one-to-various generation scenario. in human dialogue, given the context, people can reply many relevant and appropriate responses, not only with various expressions but also with different topics. take the dialogue 1 in what's more, as maximum likelihood estimation(mle) encourages the model to only learn the common words among different ground-truth responses, but ignores the interesting and specific parts, exposure bias may aggravate the common response problem of the generation model, due to the strict matching between the generated response and the ground-truth responses. take the dialogue 2 in in this paper, we propose a novel adaptive switch mechanism as a bridge(adapbridge), which introduces the generator distribution to the training phase and learns to automatically transit between the ground-truth learning and the generated learning, with respect to the word-level matching scores, such as the cosine similarity. specifically, at each training step, we calculate the cosine similarity for each generated word with respect to all its ground-truths. if the matching score is bigger than the threshold, the generated word is fed to the decoder, while if lower, the ground-truth is fed for training. the threshold is increasing as the training epoch grows. with this adaptive sampling scheme, the switch mechanism can consider the generation quality of every word, i.e., relevance between the generated word and the ground-truth, to decide whether utilizing the generated learning or not. we evaluate the proposed models on two public datasets, i.e. the chinese stc and the english reddit dataset. experimental results show that our models significantly outperform the state-of-the-art exposure bias models with respect to both metricbased evaluations and human judgments. further analysis on nmt task also shows that our model can achieve a significant improvement. the main contributions of this paper include: ## related work this section briefly introduces recent research progresses related to this work in literature. to solve the exposure bias problem in autoregressive or seq2seq models @xcite @xcite , @xcite tried to use data as demonstrator(dad) to augment the training set through the tokens predicted by the model, so as to make the training set to meet the test distribution. the method of scheduled sampling(ss) proposed by @xcite attempted to randomly sample the previous generated words to replace the ground-truth words for the model input during training time. @xcite made a further exploration of this method by sampling the previous words with decay not only from word-level oracle but also from the sentence-level oracle with a semantic metric. the main idea of this kind of method is to introduce the model's prediction information to its input at training time, and reduce the discrepancy between training and inference to alleviate the exposure bias problem. in comparison to those methods and related ideas @xcite , our proposed method adaptively determines whether the input words of model during training are ground truth or predicted by scoring each generated word. alternative based on reinforcement learning(rl) (williams, 1992) methods have been explored for generation tasks, in particular for nmt. mixed incremental cross-entropy reinforce (mixer) @xcite leverage hybrid loss function which combines both cross-entropy and reinforce to directly optimized the metrics used at test time, such as bleu or rouge. there are many other similar works @xcite @xcite . more recently, text generation via generative adversarial networks(gan) @xcite called text gans has attracted of researchers @xcite @xcite . they framed the problem under the gan paradigm, which uses the rl-based (williams, 1992) algorithms to get the gradient estimation, as the text generation is discrete. however, both rl and text gans cannot be avoided the high variance of gradient estimation caused by sparse rewards, which consequently makes the training process unstable and limits improvements. different from traditional methods, our proposed model can adaptively determine whether the current input word is from ground truth or from generation with the word-level matching scores. ## proposed method and a target response sentence y @xmath0, y k 2 , • • • , y k t k }, where s k and t k are the word length of context and response, respectively. the dialogue generation model based on sequence-to-sequence (seq2seq) @xcite framework, directly models the response probability: ## experiments in this section, we evaluate our proposed model on both chinese stc and the english reddit dataset. ## conclusion in this paper, we propose a novel adaptive switch mechanism with word-level matching scores to solve the problem of exposure bias for the dialogue generation task, named adapbridge. our core idea is to utilize the word-level matching scores to determine the input is from ground truth or from prediction at each step of training. experimental results show that our model significantly outperforms pre-vious baseline models. further analysis on nmt also indicates that our model can achieve significant improvement on different generation tasks. in future work, we plan to further design different scoring methods, i.e. bert score or bleu, to guide the model selects better words. it is also interesting to extend our adapbridge model to other generation tasks, such as abstractive summarization.