Learning When to Translate for Streaming Speech (2022)

Abstract: How to find proper moments to generate partial sentence translation given a streaming speech input? Existing approaches waiting-and-translating for a fixed duration often break the acoustic units in speech, since the boundaries between acoustic units in speech are not even. In this paper, we propose MoSST, a simple yet effective method for translating streaming speech content. Given a usually long speech sequence, we develop an efficient monotonic segmentation module inside an encoder-decoder model to accumulate acoustic information incrementally and detect proper speech unit boundaries for the input in speech translation task. Experiments on multiple translation directions of the MuST-C dataset show that outperforms existing methods and achieves the best trade-off between translation quality (BLEU) and latency. Our code is available at https://github.com/dqqcasia/mosst .

## introduction speech translation (st) aims at translating from source language speech into target language text, which is widely helpful in various scenarios such as conference speeches, business meetings, crossborder customer service, and overseas travel. there are two kinds of application scenarios, including the non-streaming translation and the streaming one. the non-streaming models can listen to the complete utterances at one time and then generate the translation afterward. while, the streaming models need to balance the latency and quality and generate translations based on the partial utterance, as shown in figure 1 . recently, end-to-end approaches have achieved remarkable progress in non-streaming st. previous work @xcite @xcite ; alinejad and sarkar, 2020; stoian et al., 2020) @xcite has shown that an end-to-end model achieves even better performance compared to the cascaded competitors. however, attempts at end-to-end streaming st are still not fully explored. traditional streaming st is usually formed by cascading a streaming speech recognition module with a streaming machine translation module @xcite . most of the previous work focuses on simultaneous text translation @xcite . @xcite propose a novel wait-k strategy based on the prefix-to-prefix framework, which is one of the popular research methods of simultaneous text translation. for end-to-end streaming st, @xcite ; @xcite ; @xcite introduce the methodology of streaming machine translation into streaming st and formalize the task, which belongs to the first study to propose simultaneous st in an end-to-end manner. however, those previous streaming st systems generally treat a fixed time-span of audio as a acoustic unit and translate new words based on fixed time segmentation, which might be unfavorable for streaming st translation. since the speaker's speech speed and the length of the phonemes are distinct, previous methods cannot find the best policy to tell whether to continue reading source audio or translate new words when the source audio is streaming in. hence, we expect the model can determine whether the streaming audio information input is enough to translate new words, similar to the manual simultaneous interpretation. this idea inspires monotonic-segmented streaming speech translation (mosst) system. specifically, we design a new module that helps to judge the acoustic boundaries of the input audio. we then propose a translation strategy that enables the model to decide whether to read the audio stream or write new tokens given the audio prefix. with the new module and decoding strategy, the model's performance on streaming speech translation has been significantly improved. we highlight our innovations and findings as follows: the results show that our model significantly outperforms sota baselines. surprisingly, we also find that mosst can rival or even surpass other sota systems in non-streaming speech translation. furthermore, we conduct a comprehensive study to analyze the utility of the proposed module and decoding strategy. ## proposed method: mosst this section first formulates the st task in streaming and non-streaming scenarios. then, we introduce the detailed architecture of mosst, as shown in figure 2 . finally, we give the training and inference strategies of mosst for streaming and nonstreaming cases. ## conclusion we propose mosst, a simple and effective framework for online speech-to-text translation. mosst consists of a pretrained acoustic model, a monotonic segmentation module, and a standard transformer, along with the multitask training strategy and the adaptive decision strategy. the monotonic segmentation module and the adaptive decision strategy tell our method when to translate. moreover, the pre-trained acoustic encoder and the multitask training strategy boost our method's ability to predict what to generate. the experiment on must-c datasets validates the effectiveness of mosst over previous work. the results show that mosst can achieve a better trade-off between quality and latency over prior end-to-end models and cascaded models in diverse latency settings. besides, we also find mosst can rival non-streaming speech translation sota systems given the complete audio waveform.