On Long-Tailed Phenomena in Neural Machine Translation (2020)

Abstract: State-of-the-art Neural Machine Translation (NMT) models struggle with generating low-frequency tokens, tackling which remains a major challenge. The analysis of long-tailed phenomena in the context of structured prediction tasks is further hindered by the added complexities of search during inference. In this work, we quantitatively characterize such long-tailed phenomena at two levels of abstraction, namely, token classification and sequence generation. We propose a new loss function, the Anti-Focal loss, to better adapt model training to the structural dependencies of conditional text generation by incorporating the inductive biases of beam search in the training process. We show the efficacy of the proposed technique on a number of Machine Translation (MT) datasets, demonstrating that it leads to significant gains over cross-entropy across different language pairs, especially on the generation of low-frequency words. We have released the code to reproduce our results.

## introduction autoregressive sequence to sequence (seq2seq) models such as transformers @xcite are trained to maximize the log-likelihood of the target sequence, conditioned on the input sequence. furthermore, approximate inference (search) is typically done using the beam search algorithm @xcite , which allows for a controlled exploration of the exponential search space. however, seq2seq models (or structured prediction models in general) suffer from a discrepancy between token level classification during learning and sequence level inference during search. this discrepancy also manifests itself in the form of the curse of sentence length i.e. the models' proclivity to generate shorter sentences during inference, which the first author is now a researcher at microsoft, usa. 1 https://github.com/vyraun/long-tailed has received considerable attention in the literature @xcite . in this work, we focus on how to better model long-tailed phenomena, i.e. predicting the long-tail of low-frequency words/tokens @xcite , in seq2seq models, on the task of neural machine translation (nmt). essentially, there are two mechanisms by which tokens with low frequency receive lower probabilities during prediction: firstly, the norms of the embeddings of low frequency tokens are smaller, which means that during the dot-product based softmax operation to generate a probability distribution over the vocabulary, they receive less probability. this has been well known in image classification @xcite and neural language models @xcite . since nmt shares the same dot-product softmax operation, we observe that the same phenomenon holds true for nmt as well. for example, we observe a spearman's rank correlation of 0.43 between the norms of the token embeddings and their frequency, when a standard transformer model is trained on the iwslt-14 de-en dataset (more details in section 2). secondly, for transformer based nmt, the embeddings for low frequency tokens lie in a different subregion of space than semantically similar high frequency tokens, due to the different rates of updates @xcite , thereby, making rare words token embeddings ineffective. since these token embeddings have to match to the context vector for getting next-token probabilities, the dot-product similarity score is lower for low frequency tokens, even when they are semantically similar to the high frequency tokens. further, better modeling long-tailed phenomena has significant implications for several text generation tasks, as well as for compositional generalization (lake and baroni, 2018). to this end, we primarily ask and seek answers to the following two fundamental questions in the context of nmt: in this section, we quantitatively characterize the long-tailed phenomena under study at two levels of abstraction, namely at the level of token classification and at the level of sequence generation. to illustrate the phenomena empirically, we use a six-layer transformer model with embedding size 512, ffn layer dimension 1024 and 4 attention heads trained on the iwslt 2014 de-en dataset @xcite , with cross-entropy and label smoothing of 0.1, which achieves a bleu score of 35.14 on the validation set using a beam size of 5. figure 1 : token f-measure bucketed by frequency: fmeasure correlates with the tokens' training frequency. ## related work at a high level, we categorize the solutions to better model long-tailed phenomena into three groups, namely, learning better representations, improving (long-tailed) classification and improvements in sequence inference algorithms. in this work, we will be mainly concerned with the interaction between (long-tailed) classification and sequence inference. better representations many recent works @xcite @xcite propose to either learn better representations for lowfrequency tokens or to integrate pre-trained representations into nmt models. to better capture long range semantic structure, @xcite argue for sequence level supervision during learning. long-tailed classification a number of works, @xcite , have focused on designing algorithms that improve classification of low-frequency classes. below, we list two such algorithms, used as baselines in section 5: focal loss proposed in @xcite , focal loss (fl) increases the relative loss of lowconfidence predictions vis-à-vis high confidence predictions, when compared to cross-entropy. it is described in equation 1, where γ > 0 and p refers to the probability/confidence of the prediction. here, τ is a hyperparameter. the intuition behind τ -normalization is based on the simple observation that the norms of the penultimate layer dictate the feature span of the corresponding class during prediction. at the sequence level, a parallel line of work has explored penalizing overconfident predictions @xcite , e.g., label smoothing has been shown to yield consistent gains in seq2seq tasks @xcite . ## modeling the long tail to improve the generation of the long-tail of low frequency tokens, it is important to study how lowfrequency tokens could appear in the candidate hypotheses during search. subsequently, we could ## experiments and results we evaluate our proposed anti-focal loss against different baselines (ce, fl, τ -norm) on the task of nmt and analyze the results for further insights. ## conclusion and future work in this work, we characterized the long-tailed phenomena in nmt and demonstrated that nmt models aren't able to effectively generate low-frequency tokens in the output. we proposed a new loss function, the anti-focal loss, to incorporate the inductive biases of beam search into the nmt training process. we conducted comprehensive evaluations on 9 language pairs with different amounts of training data from the iwslt and ted corpora. our proposed technique leads to gains across a range of metrics, improving long-tailed nmt at both the token as well as at the sequence level. in future