End-to-End Simultaneous Speech Translation with Differentiable Segmentation (2023)

Abstract: End-to-end simultaneous speech translation (SimulST) outputs translation while receiving the streaming speech inputs (a.k.a. streaming speech translation), and hence needs to segment the speech inputs and then translate based on the current received speech. However, segmenting the speech inputs at unfavorable moments can disrupt the acoustic integrity and adversely affect the performance of the translation model. Therefore, learning to segment the speech inputs at those moments that are beneficial for the translation model to produce high-quality translation is the key to SimulST. Existing SimulST methods, either using the fixed-length segmentation or external segmentation model, always separate segmentation from the underlying translation model, where the gap results in segmentation outcomes that are not necessarily beneficial for the translation process. In this paper, we propose Differentiable Segmentation (DiSeg) for SimulST to directly learn segmentation from the underlying translation model. DiSeg turns hard segmentation into differentiable through the proposed expectation training, enabling it to be jointly trained with the translation model and thereby learn translation-beneficial segmentation. Experimental results demonstrate that DiSeg achieves state-of-the-art performance and exhibits superior segmentation capability.

## introduction end-to-end simultaneous speech translation (simulst) @xcite @xcite @xcite outputs translation when receiving the streaming speech inputs, and is widely used in realtime scenarios such as international conferences, live broadcasts and real-time subtitles. compared with the offline speech translation waiting for the complete speech inputs @xcite , simulst needs to segment the streaming speech inputs and synchronously translate based on the current received speech, aiming to achieve high translation quality under low latency @xcite @xcite . however, it is non-trivial to segment the streaming speech inputs as the speech always lacks explicit boundary @xcite , and segmenting at unfavorable moments will break the acoustic integrity and thereby drop the translation performance @xcite . therefore, the precise segmentation of streaming speech is the core challenge of simulst task @xcite . to ensure that the speech representations derived from the segmentation results can produce highquality translation, simulst model should learn a translation-beneficial segmentation from the underlying translation model. existing simulst methods, involving fixed and adaptive, always fail to learn the segmentation directly from the underlying translation model. the fixed method divides the streaming inputs based on the equal length, e.g., 280ms per segment @xcite @xcite , as shown in figure 1 @xcite . such methods completely ignore the translation model and always break the acoustic integrity @xcite . the adaptive method dynamically decides the segmentation and thereby achieves better simulst performance, as shown in figure 1 @xcite . however, previous adaptive methods often use the external segmentation model @xcite or heuristic detector @xcite @xcite for segmentation, which leave a gap between the segmentation and translation model. this gap hinders learning segmentation directly from the translation model @xcite , hence making them difficult to get segmentation results that are most beneficial to translation quality. under these grounds, we aim to integrate the segmentation into translation model, and thereby directly learn segmentation from the underlying translation model, as shown in figure 1(c ). to this end, we propose differentiable segmentation (diseg) for simulst, which can be jointly trained with the underlying translation model. diseg employs a bernoulli variable to indicate whether the streaming speech inputs should be segmented or not. then, to address the issue that hard segmentation precludes back-propagation (i.e., learning) from the underlying translation model, we propose an expectation training to turn the segmentation into differentiable. owing to powerful segmentation, diseg can handle simultaneous and offline speech translation through a unified model. experiments show that diseg achieves state-of-theart performance on simulst, while also delivering high-quality offline speech translation. simultaneous translation (simulst) unlike offline speech translation, simulst needs to decide when to segment the inputs and then translate based on the received speech features @xcite . since the decisions are often made based on the speech features after downsampling, we use g(t) to denote the number of speech features when the simulst model translates y t , where speech features @xmath0. then, the probability of generating y t is p (y t | ŝ, @xmath1. how to decide g(t) is the key of simulst, which should be beneficial for the translation model to produce the high-quality translation. ## method we propose differentiable segmentation (diseg) to learn segmentation directly from the translation model, aiming to achieve translation-beneficial segmentation. as shown in figure 2 , diseg predicts a bernoulli variable to indicate whether to segment, and then turns the hard segmentation into differentiable through the proposed expectation training, thereby jointly training segmentation with the translation model. we will introduce the segmentation, training and inference of diseg following. ## analysis we conduct extensive analyses to study the effectiveness and specific improvements of diseg. unless otherwise specified, all the results are reported on must-c @xmath2. ## related work early simulst methods segment speech and then use the cascaded model (asr+mt) to translate each segment @xcite @xcite @xcite . recent end-toend simulst methods fall into fixed and adaptive. for fixed, @xcite proposed fixed predecision to divide speech into equal-length segments, and migrated simultaneous mt method, such as wait-k @xcite @xcite and mma @xcite , to simulst. for adaptive, ren et al. (2020) proposed simulspeech to detect the word in speech. chen et al. (2021) used asr results to indicate the word number. zeng et al. (2021) proposed realtrans, which detects the source word and further shrinks the speech length. dong et al. (2022) proposed mosst to translate after the acoustic information exceeding 1. zhang and feng (2022b) proposed itst to judge whether the received information is sufficient for translation. zhang et al. (2022a) proposed mu-st, which constructs the segmentation labels based on meaning unit, and uses it to train a segmentation model. in the previous method, whether using an external segmentation model or the detector, the segmentation cannot receive the gradient (i.e., learning) from the underlying translation model as the hard segmentation is not differentiable. owing to the differentiable property, diseg can be jointly trained with the underlying translation model and directly learn translation-beneficial segmentation. ## conclusion in this study, we propose differentiable segmentation (diseg) for simultaneous speech translation to directly learn segmentation from the underlying translation model. experiments show the superiority of diseg in terms of simulst performance, attention mechanism and segmentation quality. future researches will delve into the untapped potential of differentiable segmentation in such streaming models and long sequence modeling, thereby reducing feedback latency or computational cost without compromising performance.