Sequence Shortening for Context-Aware Machine Translation (2024)

Abstract: Context-aware Machine Translation aims to improve translations of sentences by incorporating surrounding sentences as context. Towards this task, two main architectures have been applied, namely single-encoder (based on concatenation) and multi-encoder models. In this study, we show that a special case of multi-encoder architecture, where the latent representation of the source sentence is cached and reused as the context in the next step, achieves higher accuracy on the contrastive datasets (where the models have to rank the correct translation among the provided sentences) and comparable BLEU and COMET scores as the single- and multi-encoder approaches. Furthermore, we investigate the application of Sequence Shortening to the cached representations. We test three pooling-based shortening techniques and introduce two novel methods - Latent Grouping and Latent Selecting, where the network learns to group tokens or selects the tokens to be cached as context. Our experiments show that the two methods achieve competitive BLEU and COMET scores and accuracies on the contrastive datasets to the other tested methods while potentially allowing for higher interpretability and reducing the growth of memory requirements with increased context size.

## introduction following the introduction of the transformer model @xcite , sentence-level machine translation, where the task is to translate separate sentences, has seen great success in recent years @xcite @xcite . however, real-world applications of the translation systems are often used to translate a whole document or a longer discourse (e.g. a transcribed speech). in those circumstances, sentence-level machine translation processes each sentence separately and is incapable of leveraging the surround-ing or previous sentences (referred to as the context sentences). this is in contrast to the context-aware machine translation where the context sentences are available to the system. the information in the previous sentences can be helpful to maintain the coherence of the translation and to resolve ambiguities @xcite @xcite . both the sentences of the text in the source language and the previously translated sentences can be used as context. the former is referred to as source-side context and the latter as target-side context. many context-aware machine translation approaches have been proposed including novel architectures that can be broadly categorized into single-encoder and multi-encoder types. in singleencoder architectures, the context sentences are concatenated with the current sentence and processed as a long sequence by a single encoder @xcite @xcite @xcite . in multi-encoder architectures, the context sentences are processed by a separate encoder than the current sentence @xcite @xcite @xcite . several multi-encoder approaches @xcite involve sharing parameters of encoders. this approach reduces the number of parameters and could also increase the speed of translation when translating the whole document sentence-by-sentence. inspired by this idea, we investigate multi-encoder architectures where all the encoder parameters are shared @xcite @xcite , which allows caching the hidden representation of the current sentence and reusing it as the hidden representation of the context when translating subsequent sentences. in this study, we refer to this architecture as caching. we experimentally show that this architecture can achieve comparable results to single-and multi-encoder architectures and is more stable in the realm of larger context sizes. in transformers, the number of tokens does not change during the processing of the sequence through the encoder (and decoder) layers. concurrent to machine translation, several techniques have been proposed to shorten the sequence of tokens in the task of language modeling @xcite @xcite . in particular, the tokens are combined in the shortening modules that are added between a specified number of encoder layers. sequence shortening can lead to the reduction of the computational and memory requirements in the subsequent layers as the requirements of the self-attention module grow quadratically with the number of tokens (although a substantial amount of research is done to mitigate that @xcite ). in this paper, we investigate the application of sequence shortening to context-aware machine translation. specifically, we apply the shortening of the cached hidden representations of the context sentences in the caching multi-encoder architectures. the intuition behind this approach is that a compressed representation of the previously seen sentences should be enough to use as a context while possibly decreasing the computational and memory requirements during inference. sequence shortening can be seen as related to the concept of chunking from psychology @xcite @xcite . to limit the scope, we consider only the source-side context. additionally, we introduce latent grouping and latent selecting -new shortening techniques where the network can learn how to group or select tokens to form a shortened sequence. our experiments indicate that sequence shortening can be leveraged to improve the stability of training for larger context sizes (we tested up to 10 previous sentences as context) while achieving comparable results for smaller context sizes. ## experiments all our experiments are implemented foot_0 in fairseq framework @xcite . we used the code repository of @xcite as the base for our implementation. ## conclusions caching architectures for context-aware machine translation have not been widely explored in the literature so far. in this study, we show that a simple method of remembering the hidden representations of the previous sentences is comparable with more established single-and multi-encoder approaches figure 5 : the mean operation memory of the models when performing inference on the examples from the en-de iwslt 2017 test subset with the varying context sizes. for the context sizes above three, we used the models trained on the context size of three. in terms of bleu and can be more effective in capturing context (up to 6 percentage points of the accuracy on the contrastive dataset for the context size of one) in the relatively low-resource training scenario. furthermore, the caching architectures are more stable to train in the regime of larger context sizes according to our experiments. pooling-based shortening of the cached sentence maintains the comparable results to the caching architecture, while our introduced shortening methods -latent grouping and selecting -show on average a strong performance both in terms of bleu and accuracy while maintaining slower growth of the memory usage during inference, and potential increased interpretability of the model through sparse assignment of tokens into groups. sequence shortening, in general, exhibit stable training in the regime of large context sizes compared to other tested methods. in future work, we will explore the integration of sequence shortening with the target-side context.