A Document-Level Neural Machine Translation Model with Dynamic Caching Guided by Theme-Rheme Information (2020)

Abstract: Research on document-level Neural Machine Translation (NMT) models has attracted increasing attention in recent years. Although the proposed works have proved that the inter-sentence information is helpful for improving the performance of the NMT models, what information should be regarded as context remains ambiguous. To solve this problem, we proposed a novel cache-based document-level NMT model which conducts dynamic caching guided by theme-rheme information. The experiments on NIST evaluation sets demonstrate that our proposed model achieves substantial improvements over the state-of-the-art baseline NMT models. As far as we know, we are the first to introduce theme-rheme theory into the field of machine translation.

## introduction most state-of-the-art neural machine translation (nmt) models @xcite @xcite regard independent sentence pairs as their training and decoding units without considering the document-level context. due to the ignorance of the discourse connections between sentences and other valuable contextual information like coreference, translations produced by such nmt systems tend to be problematic in coherence and cohesion, e.g. inconsistent translations of the same words, under-translation or mistranslation of the topic words, etc. @xcite @xcite . in the past few years, research has been conducted to incorporate inter-sentence information into nmt and the previous works on context-aware nmt models have shown improvements on the document-level translation @xcite @xcite @xcite @xcite @xcite . recent studies @xcite @xcite introduce an external architecture to produce contextual representation during the translation of a sentence. however, they do not exploit the representations already learned by the nmt encoder and the external structure is inflexible. more recently, @xcite and @xcite propose cache-based nmt models to capture document-level information. in these models, one can define flexible caching rules so that the stored information may be more interpretable. however, these methods usually focus on the target-side context and may suffer from the problem of error propagation, since the target-side context that is used as cache often contains translation errors. besides, these two methods require high-quality and large-scale parallel corpus with document boundaries, which are seldom available. previous works proved that modeling the source-side context is an effective way to improve the performance of a document-level nmt model @xcite . these methods just need small-scale of high-quality corpus with document boundaries. but the current strategies only make use of limited source-side context and are too simple. they usually ignore long-term context outside the k sentences or n words windows and neglect the rest of the discourse. so, we need an extra model to store long-term context while distinguishing important context from noises. in this paper, we proposed an improved document-level nmt model that combines a cache model with source-side context modeling. to model the inter-sentence information, we set a dynamic cache to store the history of encoder states. meanwhile, motivated by the theory of systematic functional linguistics @xcite , the mechanism for updating the context information is guided by theme-rheme information. to integrate cached context into the nmt model, the classical cache-based nmt models usually rely on a query-and-read mechanism @xcite . instead of using the query-and-read mechanism, our model uses multi-head attention @xcite structure and context gate @xcite to get corresponding contextual representation. our key contributions are as follows: ## related work most mt models are built on strong independent assumptions whether it is based on locality assumptions within a sentence as done by phrase-based models or that outside the sentence as done by even the most advanced nmt models today @xcite . text, on the contrary, doesn't consist of isolated, unrelated elements, but of collocated and structured group of sentences bound together by complex linguistic elements. ignoring these linguistic elements, like theme-rheme information @xcite , results in translations which may be perfect at the sentence-level but disappointing at document-level. among many of the document-level translation works, our inspiration comes from the cache-based methods and the source-side context modeling methods. in researches related to cache-based methods, the concept of cache was introduced by @xcite . @xcite proposed cache-based smt model and introduced three cache models: dynamic cache, static cache and topic cache. in recent researches, @xcite integrated dynamic cache and topic cache into the nmt model. @xcite only used a dynamic cache to capture target-side context information but setting their cache model to key-value structure like @xcite , the final context vector from the cache was then combined with the decoder hidden state via a gating mechanism, and the cache had a fixed length and was updated after generating a completed translation sentence. in our model, we set a similar key-value dynamic cache but using very different mechanism. in researches related to source-side context modeling methods, many rnn-based methods have been proposed @xcite . the main idea is using other sentences in the document as context while translating current sentences. in recent researches, @xcite used an extra transformer encoder to model the context directly. @xcite used inter-sentence gate to control the scale of information flowing from context and current sentence. @xcite put forward an attention-based method, using hierarchical attention network to model the context. in our model, we draw lessons from the attention-based method, using a simpler multi-headed attention structure to capture the inter-dependencies between current decoder state and cached context. ## our model in this section, we will describe our model in detail. first, the overall document-level nmt model framework will be proposed in subsection 3.1. then, the internal mechanism including the cache model, the contextual attention structure and the context gating will be presented in subsection 3.2, 3.3 and 3.4, respectively. figure 1 : illustration of the overall architecture of our model. first, we pre-training a transformer encoder with bilingual parallel sentence pairs, then use the encoder to train a theme-rheme labeling model on annotated discourse corpus as a discriminator. in step 0, the discriminator selects the themerheme information of current sentence into the cache, and by step1, the cache will participate in the decoding process of the next sentence. ## results and analysis in this section, we verified the performance of the theme-rheme labeling model, standard accuracy, precision, recall and f1-score was used. we use bleu score @xcite to evaluate the overall performance of our cache-based model (calculated by multi-bleu.per foot_1 ). finally, we used paired bootstrap resampling method @xcite to measure the significance level foot_2 . ## conclusion and future work in this paper, we proposed a document-level nmt model with a dynamic cache guided by the themerheme information. by recognizing topic themes and giving them higher weights, the dynamic cache will keep the important history encoder states as contextual information. then we used a contextual attention layer to integrate the cached context into the nmt model decoder. the experiment results showed that our model has achieved significant improvements over the state-of-the-art baseline model. however, the proposed cache-based model only uses the theme-rheme information to capture linguistic information for translation. in the future, we would like to integrate thematic progression pattern and other linguistic knowledge into document-level nmt models.