Encoding Sentence Position in Context-Aware Neural Machine Translation with Concatenation (2023)

Abstract: Context-aware translation can be achieved by processing a concatenation of consecutive sentences with the standard Transformer architecture. This paper investigates the intuitive idea of providing the model with explicit information about the position of the sentences contained in the concatenation window. We compare various methods to encode sentence positions into token representations, including novel methods. Our results show that the Transformer benefits from certain sentence position encoding methods on English to Russian translation, if trained with a context-discounted loss. However, the same benefits are not observed on English to German. Further empirical efforts are necessary to define the conditions under which the proposed approach is beneficial.

## introduction current neural machine translation (nmt) systems have reached human-like quality in translating standalone sentences, but there is still room for improvement when it comes to translating entire documents @xcite . researchers have attempted to close this gap by developing various context-aware nmt (canmt) approaches, where context refers to the sentences preceding or following the current sentence to be translated. a common approach to canmt is sentence concatenation @xcite @xcite . the current sentence and its context are concatenated into a unique sequence that is fed to the standard transformer architecture @xcite . despite its simplicity, the concatenation approach has been shown to achieve competitive or superior performance to more sophisticated, multiencoding systems @xcite . however, learning with long concatenation sequences has been proven challenging for the transformer architecture, because the self-attention can be "distracted" by long context @xcite . recently, @xcite introduced the segment-shifted position embeddings as a way to help concatenation approaches discerning the sentences concatenated in the processed sequence and improve attention's local focus. explicitly telling the model which tokens belong to each sentence is not a new idea, but an intuitive one that was already tested successfully in other tasks and approaches @xcite @xcite . we believe that encoding into token representations explicit information about the position of the sentences in the concatenation sequence can improve translation quality. the temporal structure of the document constitutes essential information for its understanding and for the correct disambiguation of inter-sentential discourse phenomena. this work investigates this intuitive idea by comparing various approaches to encoding sentence position in concatenation approaches. our contributions are the following: (i) we compare segment-shifted position embeddings with three kinds of segment embeddings, evaluating their impact on the performance of the concatenation approach; (ii) we propose and evaluate making sentence position encodings persistent over layers, adding them to the input of every layer in addition to the first; (iii) we propose and evaluate fusing position embeddings and segment embeddings into a single vector where token and sentence positions are encoded in two orthogonal sets of dimensions, allowing a clearer distinction between them, along with memory savings. to the best of our knowledge, this is the first comparative study on the employment of sentence position encodings for canmt. the sentence position encoding variants proposed are not found to improve the performance of the concatenation approach except for one specific setting where a context-discounted training loss is employed @xcite . more empirical studies are needed to clearly define the conditions under which the proposed approaches are beneficial to canmt with concatenation. nonetheless, we find it useful to share these preliminary results with the scientific community. in fact, the proposed approaches are intuitive and easy to implement, hence something that many practitioners would presumably try. we hope that our findings can guide future research on sentence position encodings, by avoiding redundant experiments on failing settings. ## proposed approach a common method for training a concatenation model and translating is by sliding windows @xcite . the sliding concatenation approach sktok translates a window ## experiments we experiment with two models: base, a contextagnostic transformer-base @xcite , and s4to4, a context-sensitive concatenation approach with the same architecture as base. s4to4 process sliding windows of 4 concatenated sentences in input and decodes the whole window into the target language. we equip s4to4 with the sentence position encoding options presented in the previous section, and we evaluate their impact on performance. when experimenting with pse, we allocate 4 dimensions to segment embeddings (d @xmath0, which is enough to encode the position of each of the 4 sentences in the concatenation window, with both one-hot and sinusoidal encodings. since d @xmath1, this leaves d p @xmath2. the models are trained and evaluated on two lan-guage pairs covering different domains: @xmath3, and @xmath4, see voita et al. (2019)'s set for the evaluation of inter-sentential deixis, lexical cohesion, verb-phrase ellipsis, and inflectional ellipsis. for @xmath6, we evaluate the models on the translation of context-dependent ambiguous pronouns with contrapro (müller et al., 2018), a large set of contrastive translations of inter-sentential pronominal anaphora. appendix b includes more setup details. the implementation of our experiments is open-sourced on github. foot_2 ## benchmarking in tables 4 and 5 , we compare our best performing systems with other canmt systems from the literature. for @xmath7, we compare with works that adopted the same experimental conditions as ours. our s4to4 concatenation approach trained with context discounting and persistent segment-shifted positions achieves the best accuracy on voita's contrastive set. for @xmath8, we compare to the works adopting @xcite 's contrastive set for evaluation, even if the training conditions are not comparable. our s4to4+cd trained on the high resource setting (see appendix c) is second of the list, by a negligible margin. notably, huo et al. (2020)'s system is also a concatenation approach, but trained on x10 parallel sentences with respect to our system. this comparison indicates that context discounting @xcite makes training efficient. system foot_5 contrapro maruf et al. (2019) 45.04 voita et al. (2018) foot_6 ## conclusions intending to improve concatenation approaches to context-aware nmt (canmt), we investigated an intuitive idea: encoding into token representations the position of their sentence within the processed sequence. besides adopting existing encoding methods (segment-shifted position embeddings and segment embeddings), we proposed a novel approach to integrate token and sentence position embeddings in a unique vector called position-segment embedding (pse). we also propose to make sentence position encodings persistent throughout the model's layers. we compared these encoding approaches on the @xmath9. consistent improvements were observed on @xmath10. however, results on @xmath11. further research is needed to clearly define the conditions under which the proposed approaches are beneficial to canmt with concatenation. we encourage practitioners to test the most promising sentence-position encodings -persistent segmentshifted positions -should they want to get the most out of their canmt systems, but only in conjunction with context discounting.