G -Transformer for Document-Level Machine Translation (2021)

Abstract: Document-level MT models are still far from satisfactory. Existing work extend translation unit from single sentence to multiple sentences. However, study shows that when we further enlarge the translation unit to a whole document, supervised training of Transformer can fail. In this paper, we find such failure is not caused by overfitting, but by sticking around local minima during training. Our analysis shows that the increased complexity of target-to-source attention is a reason for the failure. As a solution, we propose G-Transformer, introducing locality assumption as an inductive bias into Transformer, reducing the hypothesis space of the attention from target to source. Experiments show that G-Transformer converges faster and more stably than Transformer, achieving new state-of-the-art BLEU scores for both nonpretraining and pre-training settings on three benchmark datasets.

## introduction document-level machine translation (mt) has received increasing research attention @xcite @xcite @xcite . it is a more practically useful task compared to sentence-level mt because typical inputs in mt applications are text documents rather than individual sentences. a salient difference between document-level mt and sentence-level mt is that for the former, much larger inter-sentential context should be considered when translating each sentence, which include discourse structures such as anaphora, lexical cohesion, etc. studies show that human translators consider such contexts when conducting document translation @xcite . despite that neural models achieve competitive performances on sentence- * * corresponding author. level mt, the performance of document-level mt is still far from satisfactory. existing methods can be mainly classified into two categories. the first category translates a document sentence by sentence using a sequence-tosequence neural model @xcite @xcite . document-level context is integrated into sentence-translation by introducing additional context encoder. the structure of such a model is shown in figure 1(a) . these methods suffer from two limitations. first, the context needs to be encoded separately for translating each sentence, which adds to the runtime complexity. second, more importantly, information exchange cannot be made between the current sentence and its document context in the same encoding module. the second category extends the translation unit from a single sentence to multiple sentences @xcite @xcite and the whole document @xcite . recently, it has been shown that when the translation unit increases from one sentence to four sentences, the performance improves @xcite . however, when the whole document is encoded as a single unit for sequence to sequence translation, direct supervised training has been shown to fail @xcite . as a solution, either large-scale pre-training @xcite or data augmentation (junczys-dowmunt, 2019) has been used as a solution, leading to improved performance. these methods are shown in figure 1 @xcite . one limitation of such methods is that they require much more training time due to the necessity of data augmentation. intuitively, encoding the whole input document as a single unit allows the best integration of context information when translating the current sentence. however, little work has been done investigating the underlying reason why it is difficult to train such a document-level nmt model. one remote clue is that as the input sequence grows larger, the input becomes more sparse @xcite . to gain more understanding, we make dedicated experiments on the influence of input length, data scale and model size for transformer (section 3), finding that a transformer model can fail to converge when training with long sequences, small datasets, or big model size. we further find that for the failed cases, the model gets stuck at local minima during training. in such situation, the attention weights from the decoder to the encoder are flat, with large entropy values. this can be because that larger input sequences increase the challenge for focusing on a local span to translate when generating each target word. in other words, the hypothesis space for target-to-source attention is increased. given the above observations, we investigate a novel extension of transformer, by restricting selfattention and target-to-source attention to a local context using a guidance mechanism. as shown in figure 1 (c), while we still encode the input document as a single unit, group tags 1 2 3 are assigned to sentences to differentiate their positions. target-to-source attention is guided by matching the tag of target sentence to the tags of source sentences when translating each sentence, so that the hypothesis space of attention is reduced. intuitively, the group tags serve as a constraint on attention, which is useful for differentiating the current sentence and its context sentences. our model, named g-transformer, can be thus viewed as a combination of the method in figure 1(a) and figure 1(b) , which fully separate and fully integrates a sentence being translated with its document level context, respectively. we evaluate our model on three commonly used document-level mt datasets for english-german translation, covering domains of ted talks, news, and europarl from small to large. experiments show that g-transformer converges faster and more stably than transformer on different settings, obtaining the state-of-the-art results under both non-pretraining and pre-training settings. to our knowledge, we are the first to realize a truly document-by-document translation model. we release our code and model at https://github.com/baoguangsheng/g-transformer . ## experimental settings we evaluate transformer and g-transformer on the widely adopted benchmark datasets @xcite , including three domains for english-german (en-de) translation. ted. the corpus is transcriptions of ted talks from iwslt 2017. each talk is used as a document, aligned at the sentence level. tst2016-2017 is used for testing, and the rest for development. news. this corpus uses news commentary v11 for training, which is document-delimited and sentence-aligned. newstest2015 is used for development, and newstest2016 for testing. europarl. the corpus is extracted from europarl v7, where sentences are segmented and aligned using additional information. the train, dev and test sets are randomly split from the corpus. the detailed statistics of these corpora are shown in we consider three standard model configurations. base model. following the standard transformer base model @xcite , we use 6 layers, 8 heads, 512 dimension outputs, and 2048 dimension hidden vectors. big model. we follow the standard transformer big model @xcite , using 6 layers, 16 heads, 1024 dimension outputs, and 4096 dimension hidden vectors. large model. we use the same settings of bart large model @xcite , which involves 12 layers, 16 heads, 1024 dimension outputs, and 4096 dimension hidden vectors. we use s-bleu and d-bleu @xcite ) as the metrics. the detailed descriptions are in appendix a. ## transformer and long inputs we empirically study transformer (see appendix b) on the datasets. we run each experiment five times using different random seeds, reporting the average score for comparison. ## g-transformer an example of g-transformer is shown in figure 6 , where the input document contains more than 3 sentences. as can be seen from the figure, g-transformer extends transformer by augmenting the input and output with group tags @xcite . in particular, each token is assigned a group tag, indicating its sentential index. while source group tags can be assigned deterministically, target tags are assigned dynamically according to whether a generated sentence is complete. starting from 1, target words copy group tags from its predecessor unless the previous token @xmath0, in which case the tag increases by 1. the tags serve as a locality constraint, encouraging target-to-source attention to concentrate on the current source sentence being translated. formally, for a source document x and a target document y , the probability model of transformer can be written as group tags influence the auto-regressive translation process by interfering with the attention mechanism, which we show in the next section. in g-transformer, we use the group-tag sequence g x and g y for representing the alignment between x and y , and for generating the localized contextual representation of x and y . ## g-transformer results we compare g-transformer with transformer baselines and previous document-level nmt models on both non-pretraining and pre-training settings. the detailed descriptions about these training settings are in appendix c.1. we make statistical significance test according to @xcite . ## related work the unit of translation has evolved from word @xcite to phrase @xcite @xcite and further to sentence @xcite @xcite in the mt literature. the trend shows that larger units of translation, when represented properly, can lead to improved translation quality. a line of document-level mt extends translation unit to multiple sentences @xcite @xcite . however, these approaches are limited within a short context of maximum four sentences. recent studies extend the translation unit to whole document @xcite , using large augmented dataset or pretrained models. @xcite shows that transformer trained directly on documentlevel dataset can fail, resulting in unreasonably low bleu scores. following these studies, we also model translation on the whole document. we solve the training challenge using a novel locality bias with group tags. another line of work make document-level machine translation sentence by sentence, using additional components to represent the context @xcite @xcite @xcite . different from these approaches, g-transformer uses a generic design for both source and context, translating whole document in one beam search instead of sentence-by-sentence. some methods use a two-pass strategy, generating sentence translation first, integrating context information through a post-editing model @xcite . in contrast, g-transformer uses a single model, which reduces the complexity for both training and inference. the locality bias we introduce to g-transformer is different from the ones in longformer @xcite and reformer @xcite in the sense that we discuss locality in the context of representing the alignment between source sentences and target sentences in document-level mt. specifically, longformer introduces locality only to self-attention, while g-transformer also introduces locality to cross-attention, which is shown to be the key for the success of g-transformer. reformer, basically same as transformer, searches for attention targets in the whole sequence, while g-transformer mainly restricts the attention inside a local sentence. in addition, the motivations are different. while longformer and reformer focus on the time and memory complexities, we focus on attention patterns in cases where a translation model fails to converge during training. ## conclusion we investigated the main reasons for transformer training failure in document-level mt, finding that target-to-source attention is a key factor. according to the observation, we designed a simple extension of the standard transformer architecture, using group tags for attention guiding. experiments show that the resulting g-transformer converges fast and stably on small and large data, giving the state-of-the-art results compared to existing models under both pre-training and random initialization settings.