Learn To Remember: Transformer with Recurrent Memory for Document-Level Machine Translation (2022)

Abstract: The Transformer architecture has led to significant gains in machine translation. However, most studies focus on only sentence-level translation without considering the context dependency within documents, leading to the inadequacy of document-level coherence. Some recent research tried to mitigate this issue by introducing an additional context encoder or translating with multiple sentences or even the entire document. Such methods may lose the information on the target side or have an increasing computational complexity as documents get longer. To address such problems, we introduce a recurrent memory unit to the vanilla Transformer, which supports the information exchange between the sentence and previous context. The memory unit is recurrently updated by acquiring information from sentences, and passing the aggregated knowledge back to subsequent sentence states. We follow a two-stage training strategy, in which the model is first trained at the sentence level and then finetuned for document-level translation. We conduct experiments on three popular datasets for document-level machine translation and our model has an average improvement of 0.91 s-BLEU over the sentence-level baseline. We also achieve state-of-the-art results on TED and News, outperforming the previous work by 0.36 s-BLEU and 1.49 d-BLEU on average.

## introduction most previous machine translation methods are designed for sentence-level translation. recent studies have shown that the effective use of contextual information between sentences can achieve better performance in document-level machine translation @xcite @xcite @xcite . built on the transformer model @xcite , a general approach is to incorporate neighboring sentence states @xcite into the attention mechanism, which has also been widely used in many long sequence modeling methods @xcite @xcite . @xcite ; @xcite have introduced an additional context encoder to solve the limitation of sentence-level translation, which, however, is separated from the original translation model and context states is only applied on the source side. other works @xcite @xcite concatenated sentences or the entire document and feed into the attention module of the transformer. since more extended contexts may confound attention on meaningful portions of the current sentence, the model is difficult to select valuable inputs from extra contexts to navigate the redundancy of information. such methods also suffer from the quadratically increasing complexity when documents get longer. we solve such problems by introducing a memory mechanism to recurrently integrate contextualized knowledge from intermediate state in transformer layers. as recurrent memory has been widely researched since rnn @xcite , which has been incorporated with transformer by transformer-xl @xcite and further extended by @xcite who compress previous states into a two-layer hidden memory. in our approach, we update the memory through an attention module to select practical information from sentences and reduce the context space into multiple dense vectors in the memory. besides, we use another attention module to pass the knowledge retained in the memory back to the sentence state in the next step. such information exchange is expected to convey contextualized dependency between sentences. this memory mechanism can be applied in each layer for both the source and target documents, and our study shows that incorporating memory only in the last layer achieves the best performance. also, as sentences are ordered in documents, our model reads one sentence pair at each step, keeping the computational cost as same as the sentence-level translation. we experiment across three widely used datasets for document-level translation: ted, news, and europarl, and evaluate our model with s-bleu and d-bleu. we first train a vanilla transformer on sentence-level translation as the baseline and finetune the model for the documents by initializing the memory mechanism to the transformer. our model outperforms previous sota work by 0.5 s-bleu and 2.30 d-bleu on ted, and 0.21 s-bleu and 0.57 d-bleu on news. we do not achieve the sota result on europarl, which might be caused by the different results between the baselines for sentence-level translation. however, we further evaluate the improvement of previous works from their reported baseline transformer, and we achieve the most relative gain on all three datasets. we also analyze our model from the memory usage, long-range effect, context dependency, and computational complexity, and demonstrate the effectiveness and efficiency of our approach in the general understanding of the document machine translation. overall, this paper makes several contributions: (i) our work reduces the contextualized knowledge space of sentences states to multiple dense vectors, and considers the sentence dependency for both source and target documents, while keeping computational complexity in sentence-level. (ii) our model significantly improves the sentence level baseline by 0.91 s-bleu average and achieved the sota results on ted and news. (iii) our model shows the effective use of memory, long-range influence, context-dependency across sentences, and decoding efficiency through convincing analysis. this rnn variant dominated nlp models until the transformer replaced the memory unit with a self-attention mechanism and achieved great success in a wide range of nlp applications. although we cannot deny the robustness and effectiveness of the transformer model, the quadratically increased computational cost as the increase of token numbers makes transformer unable to fit the long-range sequence. some studies @xcite @xcite @xcite @xcite try to mitigate this issue by reducing the complexity of the attention module. however, such work still suffers from the problems by unlimited the document length and the document modeling is hard to solve. transformer-xl @xcite broke this dilemma by introducing the recurrent memory into transformer-based models. it cached previous hidden sentences computation and mapped such states to subsequent sentences states. theoretically, transformer-xl could handle infinite length text but storing uncompressed hidden state requires tremendous memory space, which impeded transformer-xl from good performance on dealing with practical long-sequence tasks. the compressive transformer @xcite further addressed this problem by mapping the evicted hidden state from cached memory to a more compressed representation. however, two-layer caching still requires a huge memory space and may be improved with trainable memories. ## approach our model is shown in figure 1 . additional to the vanilla transformer, we introduce a contextual memory unit and two attention modules to manipulate the memory defined as update attention and output attention. these modules can be applied at each layer in both the encoder and decoder. as input sentences are ordered from left to right in the document, our model only reads one sentence every time. the memory is expected to store contextualized information from the input sentence states and convey such knowledge to the next sentence. at each step, the update attention step maps the contextual information from the sentence state to the memory, and updates the memory to the next step. meanwhile, the output attention step fuses the information from the current sentence and the contextual memory, and outputs the aggregated knowledge to the remaining modules of the layer. formally, we define h t as the sentence state from self-attention module in transformer layer, and m t refers to the contextual memory m at step t, where t refers to the index of t th sentence in the document. m t and h t are updated and outputted as: ## analysis in this section, we discuss our model from memory usage, long-range modeling, context effect, and computational complexity, respectively. experiments are conducted with the model finetuned on the sentence baseline and evaluated on the ted, since ted has the most average sentence number per document, which is more likely to reflect the performance of our model for long documents. ## conclusion this paper introduces a memory unit that recurrently maps information into and out of transformer intermediate states and addresses the limitation about the context dependency and computational complexity in document-level machine translation. we have achieved the sota score on ted and news and a great improvement from the sentence-level baseline. our model demonstrates the effectiveness and efficiency of reduced memory space, context dependency for both source and target document, and long range influence across documents. the limitation of our work is the training cost since we accumulate the update steps and retain the graph for memory update at each step. our work does not conduct experiments for pretrained settings due to the time limitation. however, it should be easy to apply our method to any transformer-based pretrained models, such as @xcite . also, this paper only experiments on document-level machine translation, and future works may apply this approach for other tasks that need long-range sequence modeling.