Generic Mechanism for Reducing Repetitions in Encoder-Decoder Models (2021)

Abstract: Encoder-decoder models have been commonly used for many tasks such as machine translation and response generation. As previous research reported, these models suffer from generating redundant repetition. In this research, we propose a new mechanism for encoder-decoder models that estimates the semantic difference of a source sentence before and after being fed into the encoder-decoder model to capture the consistency between two sides. This mechanism helps reduce repeatedly generated tokens for a variety of tasks. Evaluation results on publicly available machine translation and response generation datasets demonstrate the effectiveness of our proposal.

## introduction sequence-to-sequence (seq2seq) models are a dominant paradigm in various natural language generation tasks, such as machine translation @xcite ), text summarization @xcite , and response generation @xcite . as @xcite reported, however, basic seq2seq models @xcite sometimes suffer from a repetition problem. one reason is that the attention mechanism does not explicitly consider which source side tokens have already been covered in the past attentions. as a result, the encoder repeatedly attends to the same token in the decoding steps, which leads to redundant generation. many researchers have proposed variants of the seq2seq model to tackle the problem. the coverage mechanism @xcite prevented the model from generating redundant outputs by taking into account the coverage of the attention distribution. these approaches can be easily incorporated into the seq2seq model with only a single attention distribution between the encoder and the decoder. however, for seq2seq models with multiple attentions, such as transformer @xcite , we cannot calculate the coverage of attentions, because the encoder attempts to attend to multiple attentions on each layer in the decoder. thus, it is challenging to incorporate the coverage mechanism into the multi-attention based seq2seq models. as another solution, suzuki and nagata (2017) proposed word-frequency estimation (wfe) that predicts the upper-bound frequency for each output token from the given input tokens to control redundancy in the output. furthermore, @xcite proposed a source-side prediction module (spm) that estimates the occurrences of input tokens from the hidden states of the decoder in the seq2seq model to reduce repetition. while wfe and spm have an advantage in not depending on the structure of a seq2seq model, it is difficult to apply these approaches to some tasks other than text summarization because wfe and spm assume that the input sentence contains more tokens than the output. to cope with the above problems, in this work, we propose a generic approach for reducing the repetition, focusing on the differences between the embedding spaces of the source and target sides. based on the assumption of distributional semantics, our approach regards the representations of an input sentence on both sides as word vectors, and attempts to minimize their difference during the training step. hence, the seq2seq model can explicitly take into account the source side context also in the decoder. our experimental results on the iwslt 2014 german-to-english translation task @xcite and the persona-chat response generation task @xcite showed that the proposed method effectively alleviates the repetition problem for both tasks. ## seq2seq model given a source sentence @xmath0, .., x i }, the seq2seq model generates a target sentence @xmath1, ..., y j }, where i and j are the numbers of source and target tokens, respectively. the seq2seq model consists of two main parts: encoder and decoder. the encoder computes the representation of a source sentence x, and the decoder generates a target sentence by decomposing the conditional probability: ## related work to overcome the repetition problem in neural machine translation, @xcite and @xcite introduced the coverage mechanism into a seq2seq model so that the decoder can pay attention to the encoder information without duplication. @xcite extended the coverage model by incorporating a pointer-generator network based on @xcite . however, it is hard to utilize these coverage methods for multi-head attention based models because multi-head attention is a stack of several attention layers, and each layer is trained to capture its own distribution. furthermore, the works of @xcite and @xcite are based on one-to-one correspondence generation, which cannot be applied to a "lossy" compression task such as summarization. suzuki and nagata (2017) proposed wordfrequency estimation (wfe) which used several linear transformations to map the hidden states of the encoder into the upper-bound occurrence of each target vocabulary and controlled the generation by the estimated occurrence. however, we cannot apply wfe for some generation tasks such as the response generation task, in which the frequency of target tokens is irrelevant to the source sentence. @xcite proposed a sourceside prediction module (spm) and assumed that output sentences are always shorter than input sentences (i.e., a summary or a headline of the input). to make sure the lengths of input and output sentences were equal, @xmath2. while this method helps spm to estimate the over-or undergeneration with the euclidean distance, it limits the application of spm. since our approach does not rely on the above assumptions, rrm is more scalable to other downstream tasks, including machine translation and response generation. ## conclusion in this work, we proposed a novel mechanism to suppress repetitions in machine translation and response generation. our model attempts to estimate the semantic vectors from a source sentence on both sides of an encoder-decoder model, which takes semantic repetitions into consideration and does not rely on any attention features. therefore, it is potential to apply our proposal to other sequence-to-sequence models, which is an advantage of our approach compared with previous methods. experimental results on the iwslt 2014 german-to-english translation task and the persona-chat response generation task demonstrated the effectiveness of our proposal. the results of the extensive experiments in the response generation task showed rrm has the ability to handle a concatenated input sequence. because our proposal takes the semantic repetitions into consideration, we believe it might have the ability to reduce the repetitions among semantically similar words. we will verify it as future work. thomas wolf, victor sanh, julien chaumond, and clement delangue. 2019. transfertransfo: a transfer learning approach for neural network based conversational agents. corr, abs/1901.08149. yonghui wu, mike schuster, zhifeng chen, quoc v le, mohammad norouzi, wolfgang macherey, maxim krikun, yuan cao, qin gao, klaus macherey, et al. 2016. google's neural machine translation system: bridging the gap between human and machine translation. arxiv preprint arxiv:1609.08144. saizheng zhang, emily dinan, jack urbanek, arthur szlam, douwe kiela, and jason weston. 2018. personalizing dialogue agents: i have a dog, do you have pets too? in proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 2204-2213, melbourne, australia. association for computational linguistics.