Context-Aware Machine Translation with Source Coreference Explanation (2024)

Abstract: Despite significant improvements in enhancing the quality of translation, context-aware machine translation (MT) models underperform in many cases. One of the main reasons is that they fail to utilize the correct features from context when the context is too long or their models are overly complex. This can lead to the explain-away effect, wherein the models only consider features easier to explain predictions, resulting in inaccurate translations. To address this issue, we propose a model that explains the decisions made for translation by predicting coreference features in the input. We construct a model for input coreference by exploiting contextual features from both the input and translation output representations on top of an existing MT model. We evaluate and analyze our method in the WMT document-level translation task of English-German dataset, the English-Russian dataset, and the multilingual TED talk dataset, demonstrating an improvement of over 1.0 BLEU score when compared with other context-aware models.

## introduction with the rapid development of machine learning techniques, the machine translation (mt) field has witnessed changes from exclusively probabilistic models @xcite to neural network based models, such as simplistic recurrent neural network (rnn) based encoder-decoder models @xcite or higher-level attention-based models @xcite , and finally turn to the current state-of-the-art transformer model @xcite and its variations. the quality of mt models, including rnnbased, attention-based, and transformer models, has been improved by incorporating contextual information @xcite @xcite , or linguistic knowledge @xcite @xcite . in the former context-aware methods, many successful approaches focus on context selection from previous sentences @xcite using multiple steps of translation, including additional module to refine translations produced by context-agnostic mt system, to utilize contextual information @xcite , and encoding all context information as end-to-end frameworks @xcite . although they have demonstrated improved performance, there are still many cases in which their models perform incorrectly for handling, i.e., the ellipsis phenomenon in a long paragraph. one of the reasons is that their models are still unable to select the right features from context when the context is long, or the model is overly complex. therefore, the model will easily suffer from an explain-away effect @xcite @xcite in which a model learns to use only features which are easily exploited for prediction by discarding most of the input features. in order to resolve the problem of selecting the right context features in the context-aware mt, we propose a model which explains decisions of translation by predicting input features. the input prediction model employs the representations of translation outputs as additional features to predict contextual features in the inputs. in this work, we employ coreference as the prediction task since it captures the relation of mentions that are necessary for the context-aware model. the prediction model is constructed on top of an existing mt model without modification in the same manner as done in multi-task learning, but it fuses information from representations used for the decisions of translation in the mt model. under the same settings of the english-russian (en-ru) dataset and the wmt document-level translation task of the english-german (en-de) dataset, our proposed technique outperforms the standard transformer-based neural machine translation (nmt) model in both sentence and contextaware models, as well as the state-of-the-art context-aware model measured by bleu @xcite , bartscore @xcite , and comet @xcite , and the humanannotated test set in a paragraph @xcite . additionally, in the multilingual experiments, our method shows consistent results, paralleling those in the en-ru and en-de datasets, and proving its versatility across languages. further analysis shows that our coreference explanation sub-model consistently enhances the quality of translation, regardless of type of dataset size. notably, the model demonstrates consistent improvement when additional context is incorporated, highlighting its effectiveness in handling larger context sizes. additionally, the analysis highlights a strong correlation between the selfattention heat map and coreference clusters, underscoring the significance of our coreference prediction sub-model in capturing coreference information during the translation process. moreover, our proposed training method proves to be effective in the coreference prediction task. we also provide a suggestion to finetune the contribution of the sub-model to optimize its impact within the overall mt system. we release our code and hyperparameters at https://github.com /hienvuhuy/transcoref . ## context-aware mt with coreference information our motivation stems from the observation that when translating a paragraph, translators are able to pick up precise words and explain why a particular choice of word is better given the context especially by relying on linguistic cues such as discourse structure, verb equivalence, etc. thus, instead of modeling a translation by directly relying on an additional conditional variable of coreference clusters c for x, i.e., p(y|c, x), we propose a model that is akin to the noisy channel framework @xcite , to explain the decision y made by the translation model: architecture the two sub-models, i.e., p(y|x) and p(c|y, x), could be trained separately as done in a noisy channel modeling approach of mt @xcite . this work formulates it as a multi-task setting by predicting two tasks jointly, i.e., translation task and coreference resolution task, by using the representations of the encoder and decoder of @xcite . more specifically, we do not alter translation task p(y|x), but obtain the representation for the coreference task by fusing the representations of the encoder and decoder as follows note that we obtain h coref from an additional decoder layer for the encoder representation h l e enc with cross attention for h l d dec . training we jointly train our two sub-models using the label-smoothing variant of the crossentropy loss function in equation 6 and the marginal log-likelihood loss function in equation 9 , but using h coref in equation 11 as follows inference inference is complex in that the model for the coreference resolution task has to be evaluated every time a target token is generated by the translation model as done in the noisy channel approach of mt @xcite . we resort to a simpler approach of ignoring the term for coreference clusters, i.e., p(c|y, x), and using only the token prediction, i.e., p(y|x); alternatively, we generate a large set of n -best translations from p(y|x) and rerank them using the joint probabilities ## conclusion this study presents a context-aware mt model that explains the translation output by predicting coreference clusters in the source side. the 7 in @xcite , they provided source code without instructions. we tried to reuse and reimplement their method; however, we cannot reproduce their results in any efforts. they did not reply our emails for asking training details. we therefore decided not to include their results in