Context-aware Neural Machine Translation with Mini-batch Embedding (2021)

Abstract: It is crucial to provide an inter-sentence context in Neural Machine Translation (NMT) models for higher-quality translation. With the aim of using a simple approach to incorporate inter-sentence information, we propose mini-batch embedding (MBE) as a way to represent the features of sentences in a mini-batch. We construct a mini-batch by choosing sentences from the same document, and thus the MBE is expected to have contextual information across sentences. Here, we incorporate MBE in an NMT model, and our experiments show that the proposed method consistently outperforms the translation capabilities of strong baselines and improves writing style or terminology to fit the document’s context.

## introduction current standard neural machine translation (nmt) models @xcite @xcite translate sentences in a sentence-by-sentence manner. however, some have argued that it is critical to consider the inter-sentence context in handling discourse phenomena @xcite , which include coherence, cohesion, coreference @xcite , and writing style @xcite . to correctly translate these linguistic features, some works provide additional context information to an nmt model by concatenating the previous sentence @xcite , applying a context encoder @xcite @xcite , or using a cachebased network @xcite . most of the previous studies have considered only a few previous context sentences. several methods, such as the cache-based network, consider long-range context but heavily modify the standard nmt models and require additional training/decoding steps. our goal is to make a simple but effective context-aware nmt model, which does not require heavy modification to standard nmt models and can handle a wider inter-sentence context. to this end, we propose a method to create an embedding that represents the contextual information of a document. to create this embedding, we focused on the mini-batch, which is commonly used in nmt training and decoding for efficient gpu computation. we modified the mini-batch creation algorithm to choose sentences from a single document and created an embedding that represents the features of the mini-batch. we call this embedding mini-batch embedding (mbe) and incorporate it in the nmt model to exploit contextual information across the sentences in the mini-batch. our main contributions can be summarized as follows: (i) we introduce mini-batch embedding to represent the features of sentences in a minibatch. (ii) we incorporate mini-batch embedding in nmt to achieve simple context-aware translation and find that our approach improves translation performance by up to 1.9 bleu points. ## neural machine translation the current nmt model f (•) generates a sequence of target sentence tokens @xmath0, . . . , y t ) given a sequence of source sentence tokens @xmath1, . . . , x s ): @xmath2; ✓), where ✓ is a set of model parameters and s and t are the numbers of source and target sentence tokens. the model parameters are trained by minimizing the loss function: ## context-aware nmt with mini-batch embedding to exploit the inter-sentence context in nmt with a simple modification, we propose mini-batch embedding (mbe) to represent the features of sentences in the mini-batch. figure 1 shows an overview of how we create mini-batch embedding and incorporate it in the nmt model. ## conclusion we proposed mini-batch embedding (mbe), which is a simple but effective method to represent contextual information across documents. we incorporated mbe in the nmt model, which enabled it to outperform competitive baselines. we found that our nmt model could choose the appropriate word and writing style to match the document context. an analysis showed that our model's performance improves with a large context, but it still achieves comparable or even better performance than that of the baseline when translating a single sentence. our future work includes applying mbe to other applications and improving the method to generate embeddings from a mini-batch.