A Case Study on Context Encoding in Multi-Encoder based Document-Level Neural Machine Translation (2023)

Abstract: Recent studies have shown that the multi-encoder models are agnostic to the choice of context and the context encoder generates noise which helps in the improvement of the models in terms of BLEU score. In this paper, we further explore this idea by evaluating with context-aware pronoun translation test set by training multi-encoder models trained on three different context settings viz, previous two sentences, random two sentences, and a mix of both as context. Specifically, we evaluate the models on the ContraPro test set to study how different contexts affect pronoun translation accuracy. The results show that the model can perform well on the ContraPro test set even when the context is random. We also analyze the source representations to study whether the context encoder is generating noise or not. Our analysis shows that the context encoder is providing sufficient information to learn discourse-level information. Additionally, we observe that mixing the selected context (the previous two sentences in this case) and the random context is generally better than the other settings.

## introduction document-level neural machine translation (docnmt) has gained a lot of attention due to the ability to incorporate context through different paradigms such as single encoder @xcite , multiple encoders @xcite @xcite , memory networks @xcite and pre-trained language models @xcite . this additional context helps to produce more consistent translations @xcite than sentence-level models. two of the most followed approaches to incorporating context are the concatenation-based and multi-encoderbased approaches. in the concatenation-based method, by concatenating context and current input sentence, a context-aware input sentence is generated @xcite @xcite and use it as the input to the encoder. in the multi-encoder approach, to encode the source or target context, an additional encoder is used @xcite @xcite ) and the entire model is jointly optimized. typically, the current sentence's neighboring sentences (previous or next) are used as the context, whereas models consist of multiple encoders and a single decoder. recent studies on multi-encoder (multienc) based docnmt models @xcite @xcite have shown that the context-encoder is acting as noise generator which improves the robustness of the model and makes the model agnostic to the choice of context. however, the improvement is in terms of bleu @xcite , which might not capture the discourse-level phenomenon effectively @xcite . this phenomenon is not studied well in the existing literature. the context encoder might not generate noise if the model can effectively capture any discourse phenomenon, such as pronoun translation from the source to the target language. modeling the relation between sentences in a given document is essential to capture any discourse phenomenon @xcite . to this end, we hypothesize that, during the training phase, if the model can learn the similarities between all the sentences in a given document given in the form of (context, source) pairs, the context encoder might not be generating noise since all the sentences in the given document are connected via the context. in this work, we aim to study the effect of the context in multienc-based docnmt models and the models' behavior in random context settings but not to introduce a novel technique. we use the 'outside attention multi-encoder' model @xcite with four different context settings to study the effect of the context. we conduct experiments on news-commentary v14 and ted corpora from english-german direction. we report the results on the contrapro test set @xcite , a contrastive test set to evaluate models' performance in translating pronouns. we also report sentence-bleu (s-bleu) @xcite , document-bleu (d-bleu) @xcite , and comet @xcite scores. to summarize, the specific attributes of our current work are as follows: ## related work the performance of document-level nmt is better than that of sentence-level nmt models due to the encoding of context @xcite . towards this goal to represent context, tiedemann and scherrer (2017) concatenate consecutive sentences and use them as input to the single-encoder-based docnmt model. @xcite conducted experiments on varying neighboring contexts and then tied them with the current sentence as input to their model. however, this approach introduced a lot of long-range dependencies. this problem can be alleviated by introducing an additional encoder to encode the context. towards this, zhang et al. (2018) and voita et al. (2018) proposed transformer-based multi-encoder nmt models where the other encoder is used to encode the context. while miculicich et al. (2018) proposed a hierarchical attention network to encode the context and a more recent approach kang et al. (2020) proposed a reinforcement learning-based dynamic context selection module for docnmt. recent studies @xcite @xcite have shown that the improvement in the performance of multi-encoder docnmt models is not due to context encoding but rather the context encoder acting as a noise generator, which improves the robustness of the docnmt model. other approaches such as pre-training (junczys-dowmunt, 2019; @xcite and memory-based approaches @xcite are shown to have improved the performance of docnmt models. recent studies @xcite have demonstrated that single encoder-decoder models can capture long-range dependencies. still, special care must be taken to break a large document into smaller fragments for training. along with document-level translation, multi-encoder models are also commonly used in automatic post-editing @xcite @xcite , multimodal translation @xcite and multitask learning @xcite @xcite scenarios. in this work, we study the effect of context in the multi-encoder @xcite ) based approach. specifically, we verify if the context encoder is generating noise or not. if the context encoder is generating noise, then the model may not be able to effectively capture the discourselevel phenomenon, such as pronoun translation accuracy, even when the model can perform well on other automatic metrics, such as bleu. ## experimental setup this section describes the data sets and the experimental setup used in the experiments. ## results and analysis we test context-aware models with two different contexts viz. previous two and random sentences as context. ## conclusion and future work in this work, we conducted experiments on multi-encoder-based docnmt systems to study how different types of contexts affect context-aware pronoun translation. specifically, we consider three different types of context settings viz, previous two sentences, random two sentences, and a mix of both these settings. we use the contrapro test set as the context-aware test set to analyze the pronoun translation accuracy. our analysis shows that the multi-encoder models can perform well on pronoun translation even when the context is random. we further conduct experiments to study whether the context encoder is generating noise or not by projecting the sentence representations from a single document using t-sne. the analysis shows that the context encoder can encode the context sufficiently enough to capture the relation between the sentences, as these sentences are connected only via context. based on the analysis, we conclude that the random context might not adversely affect the performance of multi-encoderbased docnmt models. choosing context is essential for effectively capturing any discourse phenomenon. the context encoder might not be generating noise. instead, the encoding from the context encoder is dependent on the choice of context. as we observed that mixing selected context (previous two sentences in this case) and random context is performing better than the other settings, we plan to explore effective context encoding through contrastive learning @xcite and dynamic context generation based on the source and target pairs which can help during the inference in round-trip-translation @xcite method.