Online Versus Offline NMT Quality: An In-depth Analysis on E nglish- G erman and G erman- E nglish (2020)

Abstract: We conduct in this work an evaluation study comparing offline and online neural machine translation architectures. Two sequence-to-sequence models: convolutional Pervasive Attention (Elbayad et al. 2018) and attention-based Transformer (Vaswani et al. 2017) are considered. We investigate, for both architectures, the impact of online decoding constraints on the translation quality through a carefully designed human evaluation on English-German and German-English language pairs, the latter being particularly sensitive to latency constraints. The evaluation results allow us to identify the strengths and shortcomings of each model when we shift to the online setup.

## introduction sequence-to-sequence models are state-of-the-art in a variety of sequence transduction tasks including machine translation (mt). the most widespread models are composed of an encoder that reads the entire source sequence, while a decoder (often equipped with an attention mechanism) iteratively produces the next target token given the full source and the decoded prefix. aside from the conventional offline use case, recent works adapt sequence-to-sequence models for online (also referred to as simultaneous) decoding with low-latency constraints @xcite @xcite . online decoding is desirable for applications such as real-time speech-to-speech interpretation. in such scenarios, the decoding process starts before the entire input sequence is available, and online prediction generally comes at the cost of reduced translation quality. in this work we focus on online neural machine translation (nmt) with deterministic wait-k decoding policies @xcite . with such a policy, we first read k tokens from the source then alternate between producing a target token and reading another source token (see figure 1 ). we consider two sequence-to-sequence models, a position-based convolutional model and a content-based model with self-attention. we specifically use the recent convolutional pervasive attention @xcite and transformer @xcite . we investigate, for both architectures, the impact of online decoding constraints on the translation quality through a carefully designed human evaluation on english )german and german )english language pairs. our contributions are twofold: (1) our work, to the best of our knowledge, is the first human evaluation of online vs. offline nmt systems. (2) we compare transformer and pervasive attention architectures highlighting the advantages and shortcomings of each when we shift to the online setup. the rest of this paper is organized as follows: we present in §2 related work pertaining to online mt and error analysis of nmt systems. we describe our experimental setup for human evaluation and error analysis in §3. we follow with the evaluation results in §4 and summarize our findings in §5. ## experimental setup in this work we train transformer @xcite and pervasive attention @xcite models for the tasks of online and offline translation. following @xcite , we use unidrectional encoders and train the online mt models with k @xmath0, proven to yield better translations across the latency spectrum. we train our models on iwslt'14 de )en (german )english) and en )de (english )german) datasets @xcite . sentences longer than 175 words and pairs with length-ratio exceeding 1.5 are removed. the training set consists of 160k pairs with 7283 held out for development and the test set has 6750 pairs from ted dev2010+tst2010-2013. all data is tokenized using the standard scripts from the moses toolkit @xcite . unlike existing work experimenting with this dataset, we did not lowercase the bitexts so that we can correctly assess typography errors in german. we segment sequences using byte pair encoding @xcite , bpe for short, on the bi-texts resulting in a shared vocabulary of 32k types. we train pervasive attention (pa) with 14 layers and 7-wide filters and transformer (tf) small for offline and online translation. we evaluated our wait-k models with k @xmath1.5, 3.5] (see ## conclusion we have conducted an evaluation of offline and online nmt systems for spoken language translation. our aim was to shed light on the strengths and weaknesses of wait-k decoding under two different architectures, transformer and pervasive attention. we found that transformer models are strongly affected by the shift to online decoding with a significant increase in fluency errors, most of which are duplications. pa on the other hand accrues less degradation in online decoding. our error analysis shows that translation quality in online models can be potentially improved by making read/write decisions based on the model's confidence in order to filter out avoidable additions, mistranslations and duplications. the syntactic asymmetry between german and english remains a challenge for deterministic online decoding. a more detailed analysis of syntactically required long-distance reorderings is left for future work. in this regard, indicators such as lagging difficulty or relative position are more informative for online translation. another line of future work is to assess to what extent these findings carry over to asr outputs.