An Evaluation of Source Factors in Concatenation-Based Context-Aware Neural Machine Translation (2023)

Abstract: We explore the use of source factors in context-aware neural machine translation, specifically concatenation-based models, to improve the translation quality of inter-sentential phenomena. Context sentences are typically concatenated to the sentence to be translated, with string-based markers to separate the latter from the former. Although previous studies have measured the impact of prefixes to identify and mark context information, the use of learnable factors has only been marginally explored. In this study, we evaluate the impact of single and multiple source context factors in English-German and Basque-Spanish contextual translation. We show that this type of factors can significantly enhance translation accuracy for phenomena such as gender and register coherence in Basque-Spanish, while also improving BLEU results in some scenarios. These results demonstrate the potential of factor-based context identification to improve context-aware machine translation in future research.

## introduction machine translation typically operates at the sentence level, leaving aside larger context information. this mode of operation remains dominant within the neural machine translation (nmt) framework @xcite @xcite , although it limits accurate translation for linguistic phenomena that depend on context information, such as cohesion, discourse coherence or intersentential anaphora resolution @xcite @xcite @xcite . addressing discourse-related phenomena in translation requires extending the scope of the translation models to address the relevant information present in the context sentences, in addition to that of the sentence to be translated. several approaches have been proposed within nmt to extend the modelling window beyond isolated sentences, extending the input by including context sentences @xcite or modifying the nmt architecture to model context information @xcite @xcite . despite the marked improvements achievable with the aforementioned approaches, the identification of the relevant contextual information to improve the translation of a given sentence is still an open research topic. within concatenation-based approaches @xcite , a simple yet strong document-level nmt baseline, context sentences are typically prepended to the sentence to be translated, and separated from it by a simple marker. further identification of what belongs to the context or to the sentence to be translated is typically discarded, following in part initial results by @xcite where the use of prefixes to identify context tokens led to degraded results at best. an alternative method that may provide better context identification is the utilization of factors as context markers. factors are learnable embeddings associated to input tokens that provide supplementary information about the token. different approaches, such as addition or concatenation, can be employed to combine token embeddings with factor embeddings. within the context identification process, this supplementary information may serve to indicate whether the token belongs to the context or not. to our knowledge, the use of these markers for context aware nmt has only been partially explored, and the results obtained so far have been inconclusive @xcite . in this work, we present extended results on the use of factors for context-aware nmt, centred on using source factors and measuring their impact on both standard and contrastive datasets. we re-port results on english-german pronoun translation using the contrapro test set @xcite , and on basque-spanish gender selection and register coherence with the tando test sets @xcite . we show that source factors can significantly enhance translation accuracy for phenomena such as gender and register coherence in basque-spanish, while also improving bleu results in some cases. these results demonstrate the potential of factor-based context identification as a research path to improve context-aware machine translation. ## related work the inclusion of contextual information to improve machine translation is a long-standing topic of interest in the field @xcite . within the nmt paradigm in particular, an increasing number of studies have centred on context-aware nmt approaches and the improvements that these models may provide over non-contextual baselines @xcite @xcite @xcite . one of the first methods proposed for the task is the concatenation of context sentences to the sentence to be translated @xcite , a simple approach which provides a robust baseline that often matches or outperforms more sophisticated methods @xcite @xcite . variants of this approach include discounting the loss generated by the context @xcite , extending model capacity @xcite or encoding the specific position of the context sentences @xcite . the latter in particular includes the use of learned embeddings for each sentence position, for which they report mixed results with improvements in english-russian and a negative impact in english-german, using three context sentences. we include a variant of this approach in the form of separate factors for each context sentence, without discounting context loss and applying it to a larger context on english-german and basque-spanish datasets. alternative approaches to input extension notably include refining context-agnostic translations @xcite and modelling context information with specific nmt architectures @xcite . since context-aware models are particularly suited to improve the translation of phenomena that directly depend on context information, several challenge test sets have been created specifically to evaluate the ability of models to adequately translate these phenomena in context @xcite @xcite @xcite . the use of factors was introduced in statistical machine translation as a means to incorporate additional linguistic information @xcite . for nmt, the concurrent work of @xcite and @xcite explored how sentence-level nmt models could benefit from incorporating additional linguistic information via factors in the source language. they thus added morphological features, part-of-speech tags, and syntactic dependency labels as input features, obtaining promising results in terms of perplexity reduction and higher bleu @xcite scores. source factors have only been partially explored for context-aware nmt. in addition to the previously cited work of @xcite on learnable context sentence position embeddings, rikters et al. ( 2020 ) also employ factors to identify tokens as pertaining to the context or to the sentence to be translated. in their experimental results on japanese-english translation, using one context sentence, the use of factors provided only minimal absolute improvements in terms of bleu over simple input concatenation. our work differs from theirs in several respects: we used larger contexts of 5 sentences, evaluated them on two language pairs, used contrastive evaluations on context phenomena in addition to bleu scores, and measured the impact of both unique and multiple context factors. ## conclusions in this work, we explored the use of factors in context-aware neural machine translation to improve the translation quality of inter-sentential phenomena. specifically, we evaluated the impact of source factors in concatenation-based models, using both single factors for all context sentences, and multi-factors, where separate factors are assigned for each context sentence. we conducted our experiments on parallel and contrastive test sets in english-german and basque-spanish, using larger contexts than in previous related studies, and targeting different phenomena such as pronoun translation, gender selection, and coherence in both register and gender. overall, both of the evaluated factor-based approaches improved over the concatenation-based baseline. in terms of bleu, these approaches either matched or improved over the baseline, although the gains were relatively minor and only statistically significant on two test sets in english-german. on the contrastive sets, the largest gains were obtained in basque-spanish on the coherencerelated tests, achieving gains of 10 and 17 percent-age points in accuracy. on the gender selection test, no improvements were observed in this language pair, however. in english-german, the factor approach improved over the baseline overall, but with comparatively smaller gains. the multi-factor approach provided the most consistent benefits across metrics, with additional results showing its increased accuracy in contextbased predictions at a larger distance than the baseline and the single factor approach. this approach might thus be worth exploring further in different contexts or in combination with other approaches. our study mainly aimed to measure the potential of context factors in nmt, on a diverse set of test sets with relatively large contexts. in future work, we will further investigate factor-based context-aware nmt variants, notably by measuring the impact of target-side factors, evaluating the use of factors in combination with other context identification markers, and extending the analyses to more language pairs and contextual phenomena.