Focused Concatenation for Context-Aware Neural Machine Translation (2022)

Abstract: A straightforward approach to context-aware neural machine translation consists in feeding the standard encoder-decoder architecture with a window of consecutive sentences, formed by the current sentence and a number of sentences from its context concatenated to it. In this work, we propose an improved concatenation approach that encourages the model to focus on the translation of the current sentence, discounting the loss generated by target context. We also propose an additional improvement that strengthen the notion of sentence boundaries and of relative sentence distance, facilitating model compliance to the context-discounted objective. We evaluate our approach with both average-translation quality metrics and contrastive test sets for the translation of inter-sentential discourse phenomena, proving its superiority to the vanilla concatenation approach and other sophisticated context-aware systems.

## introduction while current neural machine translation (nmt) systems have reached close-to-human quality in the translation of decontextualized sentences @xcite , they still have a wide margin of improvement ahead when it comes to translating full documents @xcite . many works tried to reduce this margin, proposing various approaches to context-aware nmt (canmt) 1 . a common taxonomy @xcite divides them in two broad categories: multiencoding approaches and concatenation (singleencoding) approaches. despite its simplicity, the concatenation approaches have been shown to achieve competitive or superior performance to more sophisticated, multi-encoding systems @xcite . nonetheless, it has been shown that transformer-based nmt systems @xcite struggle to learn locality properties @xcite of both the language itself and the source-target alignment when the input sequence grows in length, as in the case of concatenation @xcite . unsurprisingly, the presence of context makes learning harder for concatenation models by distracting attention. moreover, we know from recent literature that nmt systems require context for a sparse set of inter-sentential discourse phenomena only @xcite . therefore, it is desirable to make concatenation models more focused on local linguistic phenomena, belonging to the current sentence, while also processing its context for enabling inter-sentential contextualization whenever it is needed. we propose an improved concatenation approach to canmt that is more focused on the translation of the current sentence by means of two simple, parameter-free solutions: we support our solutions with extensive experi-ments, analysis and benchmarking. ## conclusions we presented a simple, parameter-free modification of the nmt objective for context-aware translation with sliding windows of concatenated sentences: context discounting. we analyzed the impact of our approach in the trade-off between current sentence predictions and context sentence predictions, showing that context discounting helps the model to focus on the current sentence, as intended. as a result, the concatenation model significantly improves its ability to disambiguate inter-sentential discourse phenomena, and becomes more robust to different context sizes. as an additional inductive bias towards locality, we equipped our model with segment-shifted positions, marking more explicitly the boundaries between sentences. this solution brings further improvements on targeted evaluation metrics. in the attempt of explaining the empirical functioning of the proposed solutions, we analysed their impact on the distribution of the attention weights, showing that they make it more focused and skewed towards the current sentence, as intended.