S imul S peech: End-to-End Simultaneous Speech to Text Translation (2020)

Abstract: In this work, we develop SimulSpeech, an end-to-end simultaneous speech to text translation system which translates speech in source language to text in target language concurrently. SimulSpeech consists of a speech encoder, a speech segmenter and a text decoder, where 1) the segmenter builds upon the encoder and leverages a connectionist temporal classification (CTC) loss to split the input streaming speech in real time, 2) the encoder-decoder attention adopts a wait- k strategy for simultaneous translation. SimulSpeech is more challenging than previous cascaded systems (with simultaneous automatic speech recognition (ASR) and simultaneous neural machine translation (NMT)). We introduce two novel knowledge distillation methods to ensure the performance: 1) Attention-level knowledge distillation transfers the knowledge from the multiplication of the attention matrices of simultaneous NMT and ASR models to help the training of the attention mechanism in SimulSpeech; 2) Data-level knowledge distillation transfers the knowledge from the full-sentence NMT model and also reduces the complexity of data distribution to help on the optimization of SimulSpeech. Experiments on MuST-C English-Spanish and English-German spoken language translation datasets show that SimulSpeech achieves reasonable BLEU scores and lower delay compared to full-sentence end-to-end speech to text translation (without simultaneous translation), and better performance than the two-stage cascaded simultaneous translation model in terms of BLEU scores and translation delay.

## introduction simultaneous speech to text translation @xcite @xcite , which translates source-language speech into targetlanguage text concurrently, is of great importance to the real-time understanding of spoken lectures or conversations and now widely used in many scenarios including live video streaming and international conferences. however, it is widely considered as one of the challenging tasks in machine translation domain because simultaneous speech to text translation has to understand the speech and trade off translation accuracy and delay. conventional approaches to simultaneous speech to text translation @xcite @xcite divide the translation process into two stages: simultaneous automatic speech recognition (asr) @xcite and simultaneous neural machine translation (nmt) @xcite , which cannot be optimized jointly and result in inferior accuracy, and also incurs more translation delay due to two stages. in this paper, we move a step further to translate the source speech to target text simultaneously, and develop simulspeech, an end-to-end simultaneous speech to text translation system. the simulspeech model consists of 1) a speech encoder where each speech frame can only see its previous frames to simulate streaming speech inputs; 2) a text decoder where the encoder-decoder attention follows the wait-k strategy @xcite to decide when to listen and write on the source speech and target text respectively (see figure 1 ); 3) a speech segmenter that builds upon the encoder and leverages a ctc loss to detect the word boundary, which is used to decide when to stop listening according to the figure 1 : the wait-k strategy for simultaneous speech to text translation. the model will wait for the first k source speech segments and then start to translate a target word. after that, once receiving a new source segment, the decoder generates a new target word until there is no more source word, and then the translation degrades to the full-sentence translation. the example shows the case with @xmath0. considering the difficulty of this task, we elaborately design two techniques to boost the performance of simulspeech: 1) attention-level knowledge distillation that transfers the knowledge from the multiplication of the attention matrices of simultaneous nmt and asr model to simulspeech to help the training of its attention mechanism; 2) data-level knowledge distillation that transfers the knowledge from a full-sentence nmt model to simulspeech and also reduces the complexity of data distribution @xcite to help on the optimization of simulspeech model. compared with the cascaded pipeline that trains simultaneous asr and nmt models separately, simulspeech can alleviate the error propagation problem and optimize all model parameters jointly towards the end goal, as well as reduce the delay of simultaneous translation. experiments on must-c 1 english-spanish and english-german spoken language translation datasets demonstrate that simulspeech: 1) achieves reasonable bleu scores and lower delay compared to full-sentence end-to-end speech to text translation (without simultaneous translation), and 2) obtains better performance than the two-stage cascaded simultaneous translation model in terms of bleu scores and translation delay. ## the simulspeech model similar to many sequence to sequence generation tasks, simulspeech adopts the encoder-decoder framework. as shown in figure 2a , both the encoder and decoder follow the basic network structure of transformer @xcite for neural machine translation. simulspeech is different from transformer in several aspects: ## training of simulspeech the training of the simulspeech model is more difficult than that of an nmt model or an asr model, since simulspeech involves multiple modalities (i.e., speech and text) and multiple languages. in this section, we discuss how to train the simul-speech model. as shown in figure 2b , we introduce the ctc loss for the training of the speech segmenter, and attention-level and data-level knowledge distillation for the training of the overall simulspeech model. in simulspeech training, the training data are provided in the format of (source speech, source text, target text) tuples. ## experiments and results in this section, we evaluate simulspeech on must-c corpus @xcite . first we describe experimental settings and details, then we show the experiment results, and further conduct some analyses on our model. ## conclusion in this work, we developed simulspeech, an endto-end simultaneous speech to text translation system that directly translates source speech into target text concurrently. simulspeech consists of a speech encoder, a speech segmenter, and a text decoder with wait-k strategy for simultaneous translation. we further introduced several techniques including data-level and attention-level knowledge distillation to boost the accuracy of simulspeech. experiments on must-c spoken language translation datasets demonstrate the advantages of simul-speech in terms of both translation accuracy and delay. for future work, we will design more flexible policies to achieve better translation quality and lower delay in simultaneous spoken language translation. we will also investigate simultaneous translation from the speech in a source language to the speech in a target.