A Brief Study on the Effects of Training Generative Dialogue Models with a Semantic loss (2021)

Abstract: Neural models trained for next utterance generation in dialogue task learn to mimic the n-gram sequences in the training set with training objectives like negative log-likelihood (NLL) or cross-entropy. Such commonly used training objectives do not foster generating alternate responses to a context. But, the effects of minimizing an alternate training objective that fosters a model to generate alternate response and score it on semantic similarity has not been well studied. We hypothesize that a language generation model can improve on its diversity by learning to generate alternate text during training and minimizing a semantic loss as an auxiliary objective. We explore this idea on two different sized data sets on the task of next utterance generation in goal oriented dialogues. We make two observations (1) minimizing a semantic objective improved diversity in responses in the smaller data set (Frames) but only as-good-as minimizing the NLL in the larger data set (MultiWoZ) (2) large language model embeddings can be more useful as a semantic loss objective than as initialization for token embeddings.

## introduction data for language generation tasks in goaloriented dialogue has semantically diverse samples, where the diversity can be observed from the dialogue topics to the utterances used for getting information on specific slot-values from the user. but, in many niche domains, collecting a large high-quality annotated data set is costly, and often a small data set focused on specific tasks @xcite is used for training. this restricts the model to only learn task-specific frequent contexts and seldom learn semantically similar context due to the lack of sufficient samples @xcite @xcite . optimizing only on objectives like negative log-likelihood (nll), and cross-entropy (ce) losses foster learning by making the models mimic targets at the token level @xcite . the models, hence, mostly generate only the observable patterns in the targets in training set @xcite . this can be attributed to the training procedure being uninformative about the semantic similarity of responses. to better understand, consider target: would you like to travel to paris ?, r1: how about paris as your destination ?, r2: would she like to read to me ? . r2 has 4 tokens in the same position as in the target but r1 is semantically similar to the target. however, the nll/ce loss for predicting r2 will be lower than predicting r1. this is a common occurrence when training a language generation model, and training on a small data set can exacerbate this issue even further. word embeddings from large language models like glove @xcite , bert @xcite or fasttext @xcite have been shown to have nice properties that preserve some of the linguistic structures @xcite ) that help in understanding semantic and temporal structures in dialogue. we make use of the semantics in the large word embeddings by computing a distance heuristic between the sampled text from model distribution and the target during training. this auxiliary semantic loss 1 encourages the model in generating sentences that are similar to the target and thereby potentially diversifying the model responses. although the results are on dialogue generation tasks, the results are comparable to any broad conditional language generation tasks like caption generation @xcite , text summarization @xcite and others @xcite . our contributions in the paper are: ## conditional language generation in an encoder-decoder architecture, the encoder neural network @xcite encodes a textual summary of previous utterance exchanges between a user and an agent, h i-1 , and the current user utterance u i . the encoded summary is used by a decoder network to generate the corresponding agent response (a * @xmath0, w i 2 , . . . , w i t )). language generation models are mostly trained with nll objective as defined in equation 1, ## semantic loss we introduce training with a semantic loss computed with word embeddings from any trained language model. the semantic loss to be minimized is computed in three steps: (1) a sampled @xmath1, w i 2 , . . . , w i t ) is generated by sampling tokens from decoder's distribution over the vocabulary at every step. (2) average the word vectors of the sampled ( ba sampled i ) and ground truth responses ( ba i ) with the embeddings from large language models like bert, glove or fasttext. then, compute l2 distance between the two as shown in equation 2. ## experiments we experiment on two differently sized data sets -frames @xcite and multiwoz 2.0 @xcite -which are relatively small and large. we compute l sem using the commonly used language model embeddings bert-base @xcite , glove @xcite and fasttext @xcite to compare the benefit of using different embeddings. ## conclusion training with a semantic loss has a positive effect in a smaller data set and that reflects on the model's improvement in diversity measuring metrics. but, the semantic loss was not very effective in a large data set due to the lack of diversity within and a hard bias dictated by the samples in the data set. the results obtained in the paper shows that training with semantic loss can be effective in low data setting. sean welleck, ilia kulikov, stephen roller, emily dinan, kyunghyun cho, and jason weston. 2019. neural text generation with unlikelihood training. arxiv. ronald j williams. 1992. simple statistical gradientfollowing algorithms for connectionist reinforcement learning. machine learning.