Semantic Label Smoothing for Sequence to Sequence Problems (2020)

Abstract: Label smoothing has been shown to be an effective regularization strategy in classification, that prevents overfitting and helps in label de-noising. However, extending such methods directly to seq2seq settings, such as Machine Translation, is challenging: the large target output space of such problems makes it intractable to apply label smoothing over all possible outputs. Most existing approaches for seq2seq settings either do token level smoothing, or smooth over sequences generated by randomly substituting tokens in the target sequence. Unlike these works, in this paper, we propose a technique that smooths over well formed relevant sequences that not only have sufficient n-gram overlap with the target sequence, but are also semantically similar . Our method shows a consistent and significant improvement over the state-of-the-art techniques on different datasets.

## introduction label smoothing is a regularization technique commonly used in deep learning @xcite @xcite @xcite , that improves calibration @xcite and helps in label de-noising @xcite . here, one smooths labels by introducing a prior in the label space (often just a uniform distribution) in order to prevent overly confident predictions and achieve better model calibration, both of which lead to better generalization. given these benefits, it is natural to consider whether label smoothing can be applied to sequence-to-sequence (seq2seq) prediction tasks in natural language processing. here, inducing a label prior involves smoothing in sequence space. however, this is a challenging task because the output space is exponentially large for sequences, unlike the label space in standard classification. previous works approached this challenge either by smoothing over individual tokens of the target sequence, or by sampling a few nearby targets according to hamming distance or bleu score @xcite . these techniques however do not guarantee that the smoothed targets lie within the space of acceptable targets (i.e., the sampled new target may no longer be grammatically correct or even preserve semantic meaning). in this work, we propose a label smoothing approach for seq2seq problems that overcomes this limitation. given a large-scale corpus of valid sequences, our approach selects a subset of sequences that are not only semantically similar to the target sequence, but also well formed. we achieve this using a pre-trained model to find semantically similar sequences from the corpus, and then use bleu scores to rerank the closest targets. we empirically show that this approach improves over competitive baselines on multiple machine translation tasks. ## related works token-level smoothing a popular approach used in language tasks is so called token level smoothing, where for each position's classification loss, a prior distribution over the entire vocabulary (uniformly or with unigram probability estimates) is used for regularization @xcite . this is similar to the classical label smoothing (e.g. @xcite ), as it smooths each token label independent of their context and position in the sequence. such an approach is thus unlikely to result in semantically related targets. @xcite augment the loss with a term rewarding predictions of sampled sequences. the sampling of sequences is based on their edit distance or hamming distance to the target. this method thus smooths the loss over to similar sequences (in terms of the edit distance) with smoothing rewards. @xcite employ a similar technique, but with a new reward function based on bleu @xcite or cider @xcite score. specifically, @xcite generate a smoothed version of the target sequence, wherein one replaces a token with a random token (with up-sampling of rare words). such newly generated sequences were given a partial reward based on the cosine similarity between the two tokens in a pretrained word-embedding space. this differs from our approach because this context-independent perturbation is limited to generating the same structure for the new sequence as that of the original sequence. @xcite , on the other hand, constructed grammatically correct and meaning preserving sequences. however, unlike our work, their approach relies on having multiple references (target sequences per input sequence) and might not be able to generate sequences where common words or synonyms do not appear in the same order, which is a strong limitation, precluding an augmentation like: yesterday, he scored a 94 on his final (original sequence), he had 94 points in the final test yesterday (augmented sequence). ## method sequence-to-sequence (seq2seq) learning involves learning a mapping from an input sequence x (e.g., a sentence in english) to an output sequence y (e.g., a sentence in french). canonical applications include machine translation and question answering. formally, let x denote the space of input sequences (e.g., all possible english sentences), and y the space of output sequences (e.g., all possible french sentences). we represent by @xmath0, x 2 , ...x n ] an input sequence consisting of n tokens, and similarly @xmath1, y 2 , ...y n ] an output sequence with n tokens. our goal is to learn a function f : @xmath2, given an input sequence, generates a suitable target sequence. to achieve this goal, we have a training set s ⊆ (x × y) n comprising pairs of input and output sequences. we then seek to minimise the objective this may be understood as a maximum likelihood objective, or equivalently the cross-entropy between p θ (•|x; θ) and a one-hot distribution concentrated on y. label smoothing meets seq2seq. intuitively, the cross-entropy objective encourages the model to score the observed sequence y higher than any "competing" sequence @xmath3. while this is a sensible goal, one limitation observed from classification settings is that the loss may lead to models that are overly confident in their predictions, which can hamper generalisation @xcite . label smoothing @xcite @xcite is a simple means of correcting this in classification settings. smoothing involves simply adding a small reward to all possible incorrect labels, i.e., mixing the standard one-hot label with a uniform distribution over all labels. this regularizes the training and generally leads to better predictive performance as well as probabilistic calibration @xcite . given the success of label smoothing in classification settings, it is natural to explore its value in seq2seq problems. however, standard label smoothing is clearly inadmissible: it would require smoothing over all possible outputs @xmath4, which is typically an intractably large set. nonetheless, we may follow the basic intuition of smoothing by adding a subset of related targets to the observed sequence y, yielding a smoothed loss here, r(y) is a set of related sequences that are similar to the ground truth y, and α > 0 is a tuning parameter that controls how much we rely on the observed versus related sequences. the quality of r(y) is important for our task. ideally, we would like an r(y) that: (i) is efficient to compute, and (ii) comprises sequences which meaningfully align with x (e.g., are plausible alternate translations). we now assess several options for constructing r(y) in light of the above. random sequences. one simple choice is to choose a random subset of output sequences from the training set. in the common setting where f is learned by minibatch sgd on randomly drawn minbatches @xmath5, y (i) )}, one may simply pick r(y) to be all output sequences in b. such random sequences contain general target language understanding (e.g., french grammar for an english to french translation task). however, these sequences are unlikely to have any semantic correlation with the true label. token-level smoothing. to ensure greater semantic correlation between the selected sequences and the original y, one idea is to perform token-level smoothing. for example, @xcite proposed to smooth uniformly over all tokens from the vocabulary. @xcite proposed to construct sequences @xmath6, y 2 , . . . , y n ] where for a randomly selected subset of tokens @xmath7, algorithm 1 sampling of related sequences. input: example (x, y); sequences y ref output: related sequences r(y) ## conclusion we propose a novel label smoothing approach for sequence to sequence problems that selects a subset of sequences that are not only semantically similar to the target sequences, but are also well formed. we achieve this by using a pre-trained model to find semantically similar sequences from the training corpus, and then we use bleu score to rerank the closest targets. our method shows a consistent and significant improvement over state-of-the-art techniques across different datasets. in future work, we plan to apply our semantic label smoothing technique to various sequence to sequence problems, including text summarization @xcite ) and text segmentation @xcite . we also plan to study the relation between pretraining and data augmentation techniques.