On Search Strategies for Document-Level Neural Machine Translation (2023)

Abstract: Compared to sentence-level systems, document-level neural machine translation (NMT) models produce a more consistent output across a document and are able to better resolve ambiguities within the input. There are many works on document-level NMT, mostly focusing on modifying the model architecture or training strategy to better accommodate the additional context-input. On the other hand, in most works, the question on how to perform search with the trained model is scarcely discussed, sometimes not mentioned at all. In this work, we aim to answer the question how to best utilize a context-aware translation model in decoding. We start with the most popular document-level NMT approach and compare different decoding schemes, some from the literature and others proposed by us. In the comparison, we are using both, standard automatic metrics, as well as specific linguistic phenomena on three standard document-level translation benchmarks. We find that most commonly used decoding strategies perform similar to each other and that higher quality context information has the potential to further improve the translation.

## introduction neural machine translation (nmt) @xcite is widely adopted and produces excellent translations for many domains and language pairs. however, when these automatic translations are evaluated on the document level, they reveal shortcomings when it comes to consistency in style, entity-translation or correct inference of the gender, among other things @xcite @xcite . document-level nmt aims to resolve these shortcomings by taking the context of a sentence into account during translation. there exist many works on the topic of document-level nmt, proposing various changes to the standard transformer @xcite architecture and training criteria to improve context incorporation and consequently translation quality. however, while the modeling and training aspects are covered in great detail in these works, the exact decoding strategy is often not very clearly described and sometimes not mentioned at all. in this work, we head out to answer the question, which decoding strategy is most beneficial for document-level nmt systems. we compare all commonly used strategies, as well as some additional ones, on three standard document-level translation benchmarks. we find that most of the analyzed decoding strategies perform similar to each other. also, higher quality context information can lead to better translations in certain scenarios. ## related work the earliest approaches to document-level nmt simply concatenate consecutive sentences without any further changes to the architecture compared to the sentence-level systems @xcite . later, some changes were made to the vanilla transformer architecture, like segment embeddings @xcite or attention masking @xcite and a move was made towards translating longer segments (junczys-dowmunt, 2019; @xcite @xcite . other works employ a separate encoder to include the additional context on the source side @xcite @xcite or make use of the context in a post-editing fashion @xcite . further approaches include the usage of a cache @xcite @xcite or hierarchical attention networks @xcite @xcite . recently, several works have concluded that the simple concatenation approach used with the vanilla transformer architecture performs as good -if not better -than more complicated approaches that modify the model structure @xcite . since we also observed this in our internal comparisons, we decided to focus on this simple approach for our analysis in this work. several works made the argument that the improvements seen in automatic metric scores for document-level nmt systems are from regularization effects rather than from utilizing the additional context information @xcite @xcite . in order to better asses the improvements gained by documentlevel nmt, several targeted test suites have been released @xcite @xcite . however, all of these are based on just scoring contrastive examples without actually translating anything. recently, @xcite and @xcite have released frameworks that allow to score mt systems on their ability to generate contextually correct translations. foot_0 ## search strategies training a document-level nmt system that takes the last k sentences as context is straightforward using the standard concatenation strategy @xcite . given some document level training data (f n , e n ), @xmath0, ..., n , where (f n , e n ) denotes the n-th source-target sentence pair, during training we optimize the parameters θ of the model towards here, e n n-k denotes the concatenation of the sentences e n-k , ..., e n . during search, given a document f m 1 , we want to find the best translation êm 1 according to the model. of course, exact search can not be performed and different works have used different methods to generate a translation: full segment @xcite @xcite : we split the document into non-overlapping parts f k 1 , f 2k k+1 , ..., f m m -k and translate each part separately using ## experiments we perform experiments on three document-level translation benchmarks, called news (@xmath1, ted (@xmath2. for the details regarding data conditions and preparation, as well as model training, we refer to appendix a. for the context-aware systems, we concatenate 3 adjacent sentences (i.e. @xmath3. for the two @xmath4, we also evaluate the systems on the contrapro test set @xcite . instead of scoring and ranking the contrastive examples in contrapro, as the authors have originally envisioned, we translate the source side to calculate bleu and ter as well as to score the pronoun translations according to section 4.1. we can not evaluate the full segment search strategy on con-trapro, because the sentences are not adjacent since they come from different documents. ## conclusion in this work, we analyze decoding strategies for document-level nmt systems. using the most popular document-level translation approach, we compare different search strategies found in the literature against methods developed by us. we find that most of the commonly used decoding strategies result in similar performance, both in terms of common automatic metrics, as well as on specific pronoun evaluation tasks. therefore, we conclude that it is important to include the context information during decoding, but the exact way in which to do this is not as important. also, we find that the document-level systems could actually profit from higher quality context information, in situations where this context is most relevant for translation. ture and training criterion. other approaches exist, which might exhibit a different behavior in decoding. two out of the three document-level translation tasks we use in this work are low resource with less than 500k sentence-pairs as training data. we chose these tasks due to computational limitations and to be better comparable to other works, but higher resource scenarios are more realistic for actual applications. we limit the analysis of pronoun translation to the english-german language pair. also, there are other aspects of documentlevel nmt, like consistent translation of entities, which we did not consider in our analysis.