Start-Before-End and End-to-End: Neural Speech Translation by A pp T ek and RWTH A achen U niversity (2020)

Abstract: AppTek and RWTH Aachen University team together to participate in the offline and simultaneous speech translation tracks of IWSLT 2020. For the offline task, we create both cascaded and end-to-end speech translation systems, paying attention to careful data selection and weighting. In the cascaded approach, we combine high-quality hybrid automatic speech recognition (ASR) with the Transformer-based neural machine translation (NMT). Our end-to-end direct speech translation systems benefit from pretraining of adapted encoder and decoder components, as well as synthetic data and fine-tuning and thus are able to compete with cascaded systems in terms of MT quality. For simultaneous translation, we utilize a novel architecture that makes dynamic decisions, learned from parallel data, to determine when to continue feeding on input or generate output words. Experiments with speech and text input show that even at low latency this architecture leads to superior translation results.

## introduction when developing @xmath0, we had the following goals: ## simultaneous translation in simultaneous translation a stream of source words is translated into a stream of target words without relying on the context of a full sentence. in this process, the system has to make decisions on when to read further input and when to produce partial translations. hence, there is an inherent compromise between latency and mt quality. ## end-to-end direct speech translation the direct speech translation models have been trained using direct speech translation (dst) training data including must-c, iwslt ted, and eu-roparl corpora, i.e. a total of approx. 420 hours of transcribed and translated speech (see the text decoder is based on the decoder of mt models, as illustrated in figure 1 , using either the lstm or the transformer topology. in lstm setups, the decoder is equipped with a 1-layer unidirectional lstm with cell size 1024 and single-head additive attention. all tokens are mapped into a 512-dimensional embedding space. both base and big transformer decoders are based on the architecture explained in section 2.4. to solve the data sparseness problems of dst training, we explore various strategies to augment the data by leveraging weakly supervised data, i.e. asr and mt training data. our high-quality transformer big model has been employed to generate synthetic dst training data by automatically translating the correct transcripts of asr training data @xcite . synth trans refers to machinetranslated asr training data. as listed in to further leverage the weakly supervised data, we apply pre-training of both the encoder and decoder with an adaptor layer in between. initialization of model components using pre-trained asr and mt models is a common transfer learning strategy to reduce dependency on scarce dst training data. we pre-train the encoder using our asr model explained in section 2.1, and the decoder using our mt model, either the lstm attention or transformer, as described in section 2.4. after initialization with pre-trained components, we fine-tune on the dst training data. as proposed in @xcite , in order to familiarize the pre-trained text decoder with the output of the pre-trained speech encoder, we insert an additional adaptor layer which is a bilstm layer between the encoder and decoder. we train the adaptor component jointly without freezing the parameters in the fine-tuning stage. an abstract overview is shown in figure 1 . ## experimental results in this section we report results for offline cascaded and direct speech translation, as well as for simultaneous nmt under various training data conditions. acoustic training of the baseline model and the hmm decoding have been performed with the rwth asr toolkit @xcite . all neural models have been built with re-turnn @xcite using sisyphus framework @xcite . the number of running words of all training corpora is presented in as heldout tuning sets, we use the concatenation of the ted dev2010, tst2014, and must-c dev corpora. as heldout test data, we use ted tst2015, must-c tst-he and must-c tst-common. we report case-sensitive bleu @xcite and ter @xcite scores. for simultaneous nmt, also the average lagging (al) metric @xcite is reported. to measure al, we have integrated our online decoder into the server-client implementation of iwslt 2020 within the fairseq framework @xcite . ## final results compared to last year's submission, the results of both cascade and direct offline speech translation models have improved. the cascade system shows an improvement of 2.0% bleu compared to the 2018 submission. the mt quality of the direct model almost reached the one of the cascade model, obtaining a huge improvement of 12.4% bleu. the performance on the tst2019 and tst2020 test sets is shown in ## conclusions in this paper, we summarize the results of the joint participation of apptek and rwth aachen university in the iwslt 2020 evaluation. for the first time, we present simultaneous translation results on real speech from our hybrid streaming asr system. with a latency of 4 seconds they are only 4 bleu points behind our strong cascaded offline nmt baseline. this baseline still exhibits the best results in the offline speech translation task, but our direct single end-to-end system, with careful architecture selection, pre-training, and data augmentation, is almost able to compete with our best cascaded system, obtaining a bleu score of 29.1 vs 29.7% on must-c tst-common set. on the ted tst2015 set, the ensemble of our direct endto-end systems yields a bleu score of 28.0%, exactly reaching apptek's cascaded system results at iwslt 2018, obtained one and a half years ago. at that time, our first dst prototype scored only 17.1% bleu on the same test set. this shows the fast and tremendous progress of our direct speech translation research.