Modeling Context With Linear Attention for Scalable Document-Level Translation (2022)

Abstract: Document-level machine translation leverages inter-sentence dependencies to produce more coherent and consistent translations. However, these models, predominantly based on transformers, are difficult to scale to long documents as their attention layers have quadratic complexity in the sequence length. Recent efforts on efficient attention improve scalability, but their effect on document translation remains unexplored. In this work, we investigate the efficacy of a recent linear attention model by Peng et al. (2021) on document translation and augment it with a sentential gate to promote a recency inductive bias. We evaluate the model on IWSLT 2015 and OpenSubtitles 2018 against the transformer, demonstrating substantially increased decoding speed on long sequences with similar or better BLEU scores. We show that sentential gating further improves translation quality on IWSLT.

## introduction sentence-level neural machine translation has seen significant recent progress @xcite . a move to document-level translation opens the possibility of using intersentential context at the scale of paragraphs, documents, or even whole books @xcite @xcite . this opens up new research avenues to improve translation and its evaluation for more consistent anaphora resolution and discourse coherence @xcite @xcite . transformers have enabled state-of-the-art results for sentence-level translation @xcite @xcite and this success has made them the default architecture for document translation. however, they do not scale well in the sequence length due to the quadratic complexity of attention and hence are computationally prohibitive to apply to long text. alternative architectures exist, but most still have quadratic complexity @xcite and/or have extra modules that further add to the inference cost @xcite @xcite . by reducing asymptotic complexity, recent work on efficient attention may pave the way for long text generation. however, these methods' suitability for document translation requires further investigation: some do not focus on decoding speed @xcite @xcite , while others' speedup and quality impact for document translation remains unknown @xcite @xcite . in this work, we consider random feature attention (rfa; @xcite , a representative first model with established accuracy and efficiency in sentencelevel translation. with few additional parameters, it approximates softmax attention in linear time and space with recurrent computations. we explore its effectiveness for document translation and find substantial decoding speedup over a transformer with similar or improved bleu. we also equip rfa with a sentential gate, injecting a recency inductive bias tailored to representing document context. our main contributions are: (i) we study the efficacy of rfa for document translation; (ii) we validate that rfa is competitive with a transformer but substantially faster on long documents; (iii) we augment rfa with a sentential gate designed to promote a recency bias, which brings a 0.5 bleu improvement on iwslt @xcite . to encourage future research, we release our code. foot_0 ## background standard machine translation independently translates each sentence. document translation leverages additional source and target context to produce more coherent translation, improving lexical choice and ambiguity resolution @xcite . ## scalable document-level translation we test rfa as a linear time and space model to improve the efficiency of document translation. we also augment it with a sentential gate to circumvent capacity constraints that come with a long context by injecting a recency inductive bias. figure 2: our sentential gating mechanism. e 1 and e 4 are at the beginnings of two sentences. ## experimental setup datasets and evaluation. we experiment with the iwslt 2015 chinese-to-english (zh-en) dataset @xcite with multilingual ted talk captions and the opensubtitles2018 englishto-russian (en-ru) dataset @xcite with movie and tv subtitles. we measure documentlevel bleu @xcite with sacre-bleu (post, 2018). 3 to quantify discourse consistency, we also use the test sets by @xcite based on opensubtitles. we introduce these datasets and their statistics in more detail in §a.1. ## conclusion we explored the effectiveness of random feature attention on document translation. our model substantially improves its speed over a transformer with similar or improved bleu. our sentential gate also proves effective, especially on long sequences. while our model may potentially be used to produce toxic or fake information, it also enables more efficient detectors toward such content.