Beyond Sentence-Level End-to-End Speech Translation: Context Helps (2021)

Abstract: Document-level contextual information has shown benefits to text-based machine translation, but whether and how context helps end-to-end (E2E) speech translation (ST) is still under-studied. We fill this gap through extensive experiments using a simple concatenation-based context-aware ST model, paired with adaptive feature selection on speech encodings for computational efficiency. We investigate several decoding approaches, and introduce in-model ensemble decoding which jointly performs document- and sentence-level translation using the same model. Our results on the MuST-C benchmark with Transformer demonstrate the effectiveness of context to E2E ST. Compared to sentence-level ST, context-aware ST obtains better translation quality (+0.18-2.61 BLEU), improves pronoun and homophone translation, shows better robustness to (artificial) audio segmentation errors, and reduces latency and flicker to deliver higher quality for simultaneous translation.

## introduction document-level context often offers extra informative clues that could improve the understanding of individual sentences. such clues have been proven effective for textual machine translation (mt), particularly in handling translation errors specific to discourse phenomena, such as inaccurate coreference of pronouns @xcite and mistranslation of ambiguous words @xcite . besides, ensuring consistency in translation is virtually impossible without document-level context as well @xcite . analogous to mt, speech translation (st) also suffers from these translation issues, and super-sentential context could in fact be more valuable to st because 1) homophones 1 source code is available at https://github.com/ bzhanggo/zero . figure 1: overview of the concatenation-based contextaware st. y n denotes the n-th target sentence in a document; x n denotes the speech encodings extracted from the n-th audio segment. we use dashed gray box to indicate the concatenation operation. "<@xmath0: sentence separator symbol. and acoustic noise bring additional ambiguity to st, and 2) a common use case in st is simultaneous translation, where the system has to output translations of sentence fragments, and may have to predict future input to account for word order differences between the source and target language @xcite . both for ambiguity from the acoustic signal, and operating on small sentence fragments, we hypothesize that access to extra context foot_0 will be beneficial. although recent studies on st have achieved promising results with end-to-end (e2e) models @xcite @xcite @xcite , nevertheless, they mainly focus on sentence-level translation. one practical challenge when scaling up sentence-level e2e st to the document-level is the encoding of very long audio segments, which can easily hit the computational bottleneck, especially with transformers @xcite . so far, the research question of whether and how contextual information benefits e2e st has received little attention. in this paper, we answer this question through extensive experiments by exploring a concatenation-based context-aware st model. figure 1 illustrates our model, where neighboring source (target) sequences are chained together into one sequence for joint translation. this paradigm only requires data-level manipulation, thus allowing us to reuse any existing sentence-level e2e st models. despite its simplicity, this approach successfully leverages contextual information to improve textual mt @xcite @xcite , and here we adapt it to st. as for the computational bottleneck, we shorten the speech encoding sequence via adaptive feature selection @xcite , which only retains a small subset of encodings (∼16%) for each audio segment. we investigate several decoding methods, including chunk-based decoding and sliding-window based decoding. we also study an extension of the latter with the constraint of target prefix, where the prefix denotes the translation of previous context speeches. we find that using these methods sometimes results in misaligned translations, particularly when using the constraint. this issue manifests itself in mismatching sentence boundaries and producing over-and/or under-translation, which greatly hurts sentence-based evaluation metrics. to avoid such misalignments, we introduce inmodel ensemble decoding (imed) to regularize the document-level translation with its sentence-level counterpart. note that we use the same contextaware st model here for both types of translationthat's why we call it in-model ensemble. we adopt transformer @xcite for experiments with the must-c dataset @xcite . we study the impact of context on translation in different settings. our results demonstrate the effectiveness of contextual modeling. our main findings are summarized below: our work is inspired by pioneer studies on contextaware textual mt. context beyond the current sentence carries information whose importance for translation cohesion and coherence has long been posited @xcite . with the rapid development of neural mt and also available document-level textual datasets, research in this direction gained great popularity. recent efforts often focus on either advanced contextual neural architecture development (tiedemann and scherrer, 2017; @xcite @xcite @xcite @xcite and/or improved analysis and evaluation targeted at specific discourse phenomena @xcite @xcite @xcite . we follow this research line, and adapt the concatenation-based contextual model @xcite @xcite to st. our main interest lies in exploring the impact of context on st. developing dedicated contextual models for st is beyond the scope of this study, which we leave to future work. context-aware st extends the sentence-level st towards streaming st which allows models to access unlimited previous audio inputs. instead of improving contextual modeling, many studies on streaming st aim at developing better sentence/word segmentation policies to avoid segmentation errors that greatly hurt translation @xcite @xcite @xcite . very recently, @xcite proposed a memory augmented transformer encoder for streaming st, where the previous audio features are summarized into a growing continuous memory to improve the model's context awareness. despite its success, this method ignores the target-side context, which turns out to have significant positive impact on st in our experiments. our study still relies on oracle sentence segmentation of the audio. the most related work to ours is @xcite , which also investigated contextualized translation and showed that contextaware st is less sensitive to audio segmentation errors. while they exclusively focus on the robustness to segmentation errors, our study investigates the benefits of context-aware e2e st more broadly. 3 context-aware st via concatenation we extend the sentence-level st with documentlevel context, by modeling up to c previous source/target segments/sentences for translation. formally, given a pre-segmented audio (source document) @xmath1, . . . , a n as well as its paired target document @xmath2, . . . , y n , the model is trained to maximize the following likelihood: ## inference concatenation-based contextual modeling allows for different inference strategies with possible trade-offs between simplicity/efficiency and accuracy. we investigate the following inference strategies (see figure 2 ): chunk-based decoding (cbd) cbd splits all audio segments in one document into nonoverlapping chunks, with each chunk concatenating c + 1 segments, as shown in figure 2a . cbd directly translates each chunk, and then recovers sentence-level translation via the separator symbol "<@xmath3. cbd is the most efficient inference strategy, only encoding/decoding each sentence once, but it might suffer from misaligned translation, producing more or fewer sentences than the input segments. we simply drop the extra generated sentences and replace the missing ones with "<@xmath4. also, cbd introduces an independence assumption between chunks. ## conclusion and future work our experiments confirm the effectiveness of context-aware modeling for end-to-end speech translation. with concatenation-based contextual modeling and appropriate decoding method, we observe positive impact of context on translation. context-aware st improves general translation quality in bleu, and also helps pronoun and homophone translation. st models become less sensitive to (artificial) audio segmentation errors with context. in addition, context also improves simultaneous translation by reducing latency and erasure. we observe overall positive results over different languages and evaluation metrics on the must-c corpus. in the future, we will investigate more dedicated neural architectures to handle long-form speech input. while we relied on a dataset with sentence segmentation in this work, we are interested in removing the reliance on segmentation at inference time to implement the full-fledged streaming translation scenario. results in figure 6 and 7 show that the optimal value of c and λ also differs across evaluation sets. overall, setting @xmath5.5 offers us decent performance. note again, we selected these configurations for generality and simplicity rather than its being optimal.