What Works When in Context-aware Neural Machine Translation? (2023)

Abstract: Document-level Machine Translation has emerged as a promising means to enhance automated translation quality, but it is currently unclear how effectively context-aware models use the available context during translation. This paper aims to provide insight into the current state of models based on input concatenation, with an in-depth evaluation on English–German and English–French standard datasets. We notably evaluate the impact of data bias, antecedent part-of-speech, context complexity, and the syntactic function of the elements involved in discursive phenomena. Our experimental results indicate that the selected models do improve the overall translation in context, with varying sensitivity to the different factors we examined. We notably show that the selected context-aware models operate markedly better on regular syntactic configurations involving subject antecedents and pronouns, with degraded performance as the configurations become more dissimilar.

## introduction neural machine translation (nmt) systems @xcite @xcite have traditionally translated sentences in isolation without considering relations between discourse elements. this leads to translations lacking crucial textual properties such as cohesion, discourse coherence or intersentential anaphora resolution @xcite @xcite . properly handling discourse-related phenomena requires extending the scope of the translation model beyond the sentence level. as a result, many methods have been developed to extend the modeling window beyond isolated sentences. these approaches range from extending the input of standard nmt models @xcite to architectural variants @xcite @xcite . despite the promising results achieved by context-aware nmt, determining the precise use of context remains a significant challenge, leading to contradictory findings, including studies suggesting that context-aware models do not improve intersentential phenomena, but rather act as mere regularisers @xcite @xcite . standard translation metrics have limitations to measure document-level phenomena, whereas contrastive evaluations provide more precise measures but do not delve into how context information is actually used or ignored. we believe that in-depth analyses of context usage by context-aware models could help better understand their current strengths and limitations. in this paper, we analyse the performance of various approaches based on context concatenation, a strong baseline for document-level nmt, examining variations in the use of source and target context. we provide an in-depth analysis of the results achieved by the selected nmt models in terms of data bias, context complexity, as well as part of speech and syntactic functions of the relevant elements in contextual translation. we focus our study on pronoun translation for english-german and english-french, for which there are publicly available annotated datasets. using contextual information to improve machine translation has been a topic of interest in the community for decades @xcite . research within the nmt paradigm, where contextual information may be accessed over extended input windows, has led to a number of new approaches to incorporate intersentential context for more accurate translation. a variety of studies have explored contextaware nmt approaches, analysing the improvements that these models can provide over noncontextual baselines @xcite @xcite . one of the first proposed methods is the concatenation of context sentences to the sentence to be translated @xcite . this simple approach is still efficient, achieving comparable or superior performance to more complex approaches @xcite . other methods involve refining context-agnostic translations @xcite , or modelling context information with specific nmt architectures @xcite . some of these models only use source language context @xcite , while others include target language context as well @xcite . context-aware models have shown to be effective in the translation of context-dependent phenomena @xcite and several test sets have been created to specifically evaluate the ability of models to accurately translate pronouns within their context @xcite @xcite @xcite . @xcite show that inserting small amounts of distracting information is enough to strongly decrease scores in contrastive tests, and @xcite found that only a few sentences are really useful to improve translation quality. a deeper and more thorough analysis is thus still required to draw firm conclusions about the strengths and weaknesses of context-aware models. ## conclusions in this paper, we presented a systematic analysis of various concatenation-based context-aware models to help gain a clearer view of their current strengths and limitations. we compared the performance of three different approaches, using a limited context window of one sentence from the source and/or the target context, in english-german and english-french using the standard contrapro and large-scale contrastive pronoun test, respectively. our experiments focused on several dimensions of analysis: (i) metric results on sentence-level and contrastive sets in terms of bleu and accuracy, (ii) data distribution bias, (iii) part-of-speech of the antecedents, (iv) context complexity in terms of length and number of potential antecedents, and (v) syntactic functions of the pronoun and the antecedent. our results confirm the ability of context-aware models based on concatenation approaches to improve the accuracy of neural machine translation, particularly for pronominal categories affected by bias. integrating target information was shown to be particularly beneficial across experiments, with 2to2 models achieving the best results overall. the part of speech of the antecedent in source sentences was shown to be impactful, once translation bias towards the most frequent pronouns was accounted for. models that made use of source context were thus shown to perform better when the tag of the antecedent was of a nominal type, as opposed to uninformative antecedents, in contrast with models relying on the target context. context complexity, in terms of either length or number of potential antecedents, was shown to be impactful for english-german, but less conclusively so for english-french. further analyses on other datasets would be needed to properly assess the impact of context complexity. we also found that the syntactic function of pronouns and antecedents was a determining factor for all models, with a similar tendency across models and language pairs for context information to be better exploited when both elements shared the same syntactic tag and the antecedent was the subject of the context sentence. function identity with non-subject antecedents performed as a distant second overall, followed by different tags with subject antecedents, and finally by dissimilar tags and non-subject antecedents. these results highlight current limitations of concatenated context-aware models, which seem to mainly capture the most regular and simpler configurations. it might be worth developing new contrastive test sets with higher variability to more precisely assess the strengths and limitations of context-aware models. it is worth noting that our analysis was conducted using the reference target context in our evaluations with target-dependent models, and further analyses would be necessary to determine the impact of using translated target context sentences instead of references. additionally, a more detailed analysis based on different pronominal categories could also be helpful, although this was beyond the scope of this work. we also leave for future work further explorations of the differences between models that use source information and those that use target information, as well as including other types of models that are not based on input concatenation for context modelling.