Long-Short Term Masking Transformer: A Simple but Effective Baseline for Document-level Neural Machine Translation (2020)

Abstract: Many document-level neural machine translation (NMT) systems have explored the utility of context-aware architecture, usually requiring an increasing number of parameters and computational complexity. However, few attention is paid to the baseline model. In this paper, we research extensively the pros and cons of the standard transformer in document-level translation, and find that the auto-regressive property can simultaneously bring both the advantage of the consistency and the disadvantage of error accumulation. Therefore, we propose a surprisingly simple long-short term masking self-attention on top of the standard transformer to both effectively capture the long-range dependence and reduce the propagation of errors. We examine our approach on the two publicly available document-level datasets. We can achieve a strong result in BLEU and capture discourse phenomena.

## introduction recent advances in deep learning have led to significant improvement of neural machine translation (nmt) @xcite @xcite . particularly, the performance on the sentence-level translation of both low-and high-resource language pairs is dramatically improved @xcite @xcite . however, when translating text with long-range dependencies, such as in conversations or documents, the original mode of translating one sentence at a time ignores the discourse phenomena @xcite , introducing undesirable behaviors such as inconsistent pronouns across different translated sentences. document-level nmt, as a more realistic translation task in these scenarios, has been systematically * corresponding author. almost all of the latest studies chose the standard transformer model as their baseline which translates each sentence in the document with the model trained on the sentence-level data. the cohesion and consistency are in general poor. a more reasonable baseline is to train the transformer with the context prepended, and this modification could be simply implemented via data preprocessing. @xcite conducted a detailed analysis of rnn-based nmt models on the topic of whether or not to include the extended context. consistency and precision is often viewed as a trade-off of each other. we conduct a detailed analysis of the effect of document context on consistency in transformer architecture accepting multi-sentence input. when it comes to leveraging the contextual information, the common approach is to model the interaction between the sentence and its context with specially designed attention modules @xcite . such works tend to include more than one encoder or decoder, with a substantial number of parameters and additional computations. in our work, we reduce the contextual and regular attention modules into one single encoder and decoder. our idea is motivated by the one transformer decoder with the two-stream self-attention @xcite . in particular, we maintain two different sets of hidden states and employ two different masking matrices to capture the long and short term dependencies. the contributions of this paper are threefold: i) we extensively research the performance of the standard transformer in the setting of multisentence input and output; ii) we propose a simple but effective modification to adapting the transformer for document nmt with the aim of ameliorating the effect of error accumulation; iii) our experiments demonstrate that even the simple baseline can achieve comparable results. ## the proposed approach the standard transformer nmt follows the typical encoder-decoder architecture with using stacked self-attention, pointwise fully connected layers, and the encoder-decoder attention layers. the selfattention in the decoder allows only those positions from the left up to the current one to be attended to, preventing information flow to the right beyond the current target and preserving the auto-regressive property. the illegal connections will be masked out by setting as -∞ before the softmax operation. the attention probability can be succinctly written in a unified formulation. ## experiments experimental setup we carry out experiments with the chinese-english iwslt ted talks dataset foot_0 and english-russian open-subtitle dataset foot_1 . the widely used zh-en iwslt dataset contains around 200k training sentence pairs divided into 1713 documents. as is the convention, dev2010 and tst2010-2013 are used for validation and testing respectively. the en-ru subtitle dataset contains around 1.5m conversations, where each conversation includes exactly 4 sentences. two randomly selected subsets of 10,000 instances from movies not included in the training are used for development and test foot_2 . the bpe tokenization is separately learnt with 32k operations for each language in the dataset. the resulting source / target vocabulary sizes for en-zh and en-ru datasets are 10296 / 16018 and 12273 / 22642, respectively. the token-level batch sizes are 8192 and 16384 for training the zh-en and en-ru datasets on two and four p-100 gpus. the model hyper-parameters and the optimizer of standard transformer baseline follow the base setting in @xcite . we set the layers in encoder and decoder to 6, and the attention heads to 8. the dimensionality of input and output is 512. in addition, we add a feed-forward layer before the decoder output layer, with dimensionality 1024, to combine the local and global stream. we use the adam optimizer with β1 = 0.9, β2 = 0.98 @xmath0, with 16000 warm-up steps and scale of 4. the batch size for each gpu is 4000. bleu score is calculated with the script mteval-v13a.pl in moses foot_3 . all reported values are evaluated on the test set with the best checkpoint on the development set. ## discussions and conclusions in this work, we present a simple but effective variation with the long-short term masking strategy, and we performed comparative studies with the k-to-k translation model of the standard transformer. just as the big, complex neural network architectures with great many parameters has its power, small but efficient modification like ours to the classical transformer has its unique appeals. other examples of simple but impactful ideas are data augmentation and the round-trip back-translation @xcite , to name just a few. big or small, complex or simple, each has its distinct advantages. we're encouraged by our findings that in tandem with the great machinery that could bring powerful results, simplistic approaches could be just as efficacious.