A Case Study on Context-Aware Neural Machine Translation with Multi-Task Learning (2024)

Abstract: In document-level neural machine translation (DocNMT), multi-encoder approaches are common in encoding context and source sentences. Recent studies (CITATION) have shown that the context encoder generates noise and makes the model robust to the choice of context. This paper further investigates this observation by explicitly modelling context encoding through multi-task learning (MTL) to make the model sensitive to the choice of context. We conduct experiments on cascade MTL architecture, which consists of one encoder and two decoders. Generation of the source from the context is considered an auxiliary task, and generation of the target from the source is the main task. We experimented with German–English language pairs on News, TED, and Europarl corpora. Evaluation results show that the proposed MTL approach performs better than concatenation-based and multi-encoder DocNMT models in low-resource settings and is sensitive to the choice of context. However, we observe that the MTL models are failing to generate the source from the context. These observations align with the previous studies, and this might suggest that the available document-level parallel corpora are not context-aware, and a robust sentence-level model can outperform the context-aware models.

## introduction context-aware neural machine translation gained much attention due to the ability to incorporate context, which helps in producing more consistent translations than sentence-level models @xcite @xcite @xcite @xcite . there are mainly two approaches to incorporating context. the first one is to create a contextaware input sentence by concatenating context and current input sentence @xcite @xcite and using it as the input to the encoder. the second approach uses an additional context-aware component to encode the source or target context @xcite @xcite ) and the entire model is jointly optimized. typically, the current sentence's neighbouring sentences (either previous or next) are used as the context. the context-aware models are trained to maximize the log-likelihood of the target sentence given the source sentence and context. most of the existing works on docnmt @xcite @xcite focus on encoding the context through context-specific encoders. recent studies @xcite show that, in the multi-encoder docnmt models, the performance improvement is not due to specific context encoding but rather the contextencoder acts like a noise generator, which, in turn, improves the robustness of the model. in this work, we explore whether the context encoding can be modelled explicitly through multi-task learning (mtl) @xcite . specifically, we aim to study the effectiveness of the mtl framework for docnmt rather than proposing a state-of-theart system. the availability of document-level corpora is less compared to sentence-level corpora. previous works (junczys-dowmunt, 2019) use the sentence-level corpora to warm-start the document-level model, which can be further tuned with the existing limited amount of documentlevel data. however, in this work, we focus only on improving the performance of docnmt models with available document-level corpora. we consider the source reconstruction from the context as the auxiliary task and the target translation from the source as the main task. we conduct experiments on cascade mtl @xcite architecture. the cascade mtl architecture comprises one encoder and two decoders (figure 1 ). the intermediate (first) decoder attends over the output of the encoder, and the final (second) decoder attends over the output of the intermediate decoder. the input consists of ⟨c x , x, y⟩ triplets, where c x , x and y represents the context, source, and target sentences, respectively. the model is trained to optimize both translation and reconstruction objectives jointly. we also train two baseline models as contrastive models, namely sentence-level vanilla baseline and single encoder-decoder model, by concatenating the context and source @xcite @xcite . we additionally train multiencoder single-decoder models @xcite to study how context affects the docnmt models. we conduct experiments on german-english direction with three different types of contexts (viz. previous two source sentences, previous two target sentences, and previous-next source sentences) on news-commentary v14 and ted corpora. we report bleu @xcite calculated with sacrebleu @xcite and apt (accuracy of pronoun translation) @xcite scores. to summarize, the specific attributes of our current work are as follows: previous studies have proposed various documentlevel nmt models and achieved great success. the main goal of these approaches is to efficiently model context representation, which can lead to better translation quality. towards this goal to represent context, tiedemann and scherrer (2017) concatenate consecutive sentences and use them as input to the single-encoder-based docnmt model. agrawal et al. (2018) conducted experiments on varying neighbouring contexts and concatenated with the current sentence as input to their model. with these similar trends, junczys-dowmunt (2019) conducted experiments considering the entire document as context. further progress on context representation in doc-nmt, zhang et al. (2018) and voita et al. (2018) proposed transformer-based multi-encoder nmt models where the additional encoder is used to encode the context. while miculicich et al. (2018) proposed a hierarchical attention network to encode the context, a more recent approach kang et al. (2020) proposed a reinforcement learningbased dynamic context selection module for doc-nmt. kim et al. (2019) and li et al. (2020) conducted experiments on multi-encoder docnmt models and reported that the performance improvement is not due to context encoding; instead, the context encoder acts as a noise generator, which improves the robustness of the docnmt model. junczys-dowmunt (2019) conducted experiments on a single encoder model with masked language model objective (devlin et al., 2019) to incorporate document-level monolingual sourceside data. since the multi-encoder models are trained to optimize the translation objective only, it might be possible for the model to pay less attention to the context, and li et al. (2020) report the same. mtl strategies in nmt trained on other auxiliary tasks along with the main translation task @xcite @xcite @xcite achieved significant improvements in translation quality so far. the other auxiliary tasks include autoencoding @xcite , denoising autoencoding @xcite , parsing and named entity recognition @xcite . @xcite proposed a cascade mtl network to improve the robustness of the nmt model. they considered denoising the noisy text as an auxiliary task and the translation as the main task. they achieved a significant bleu score improvement (up to 7.1 bleu) on the wmt robustness shared task on the french-english dataset. however, most multi-task models are proposed only for sentence-level nmt models. multi-task learning is relatively unexplored in context-aware nmt settings. @xcite proposed an mtl framework for dialogue translation tasks that jointly correct the sentences having issues such as pronoun dropping, punctuation dropping, and typos and translate them into the target language. @xcite proposed a three-stage training framework for the neural chat translation task. the model is trained on auxiliary tasks such as monolingual cross-lingual response generation tasks to generate coherent translation and the next utterance discrimination task. @xcite proposed an mtl system to force the model to attend over relevant cohesion devices while translating the current sentence. in this work, we propose a multi-task learning objective, i.e., reconstruction of source sentences given the source context in a cascade multi-task learning setting to study the effect of context in document-level nmt systems. ## results and analysis this section discusses the results of the trained models and the context's effect on multi-encoder and mtl settings. we report all models' bleu scores on @xmath0, calculated with sacrebleu @xcite . ## conclusion this work explored the mtl approach for document-level nmt (docnmt). our proposed mtl approach is based on cascade mtl architecture, where the model consists of one encoder (for context encoding) and two decoders (for the representation of the current source and target sentences). reconstruction of the source sentence given the context is considered the auxiliary task, along with the translation of the current source sentence as the main task. we conducted experiments for german-english for news-commentary v14, iwslt'17 ted, and europarl v7 corpora with three different types of contexts viz. two previous sources, two previous targets, and previousnext source sentences with respect to the current input source sentence. our proposed mtl approaches outperform the sentence-level baseline and concatenated-context models in low-resource (for news and ted corpora) settings. however, all models perform well in the high resource setting (europarl corpus), with proposed mtl models slightly underperforming the rest. our mtl models are more sensitive to the choice of context than the multi-encoder mod-els when tested with random context. we observe that the context encoder cannot encode context sufficiently and performs poorly reconstruction tasks. finally, we reported apt (accuracy of pronoun translation) scores, and the proposed mtl models outperformed the sentence-level baseline and concatenated-context models. our empirical analysis concludes that our approach is more sensitive to the choice of context and improves the overall translation performance in low-resource contextaware settings. we plan to explore other tasks, such as gap sentence generation (gsg) @xcite as an auxiliary task for better context encoding, different training curricula to prioritize one objective over the other during the training, and dynamic context selection.