Direct Segmentation Models for Streaming Speech Translation (2020)

Abstract: The cascade approach to Speech Translation (ST) is based on a pipeline that concatenates an Automatic Speech Recognition (ASR) system followed by a Machine Translation (MT) system. These systems are usually connected by a segmenter that splits the ASR output into hopefully, semantically self-contained chunks to be fed into the MT system. This is specially challenging in the case of streaming ST, where latency requirements must also be taken into account. This work proposes novel segmentation models for streaming ST that incorporate not only textual, but also acoustic information to decide when the ASR output is split into a chunk. An extensive and throughly experimental setup is carried out on the Europarl-ST dataset to prove the contribution of acoustic information to the performance of the segmentation model in terms of BLEU score in a streaming ST scenario. Finally, comparative results with previous work also show the superiority of the segmentation models proposed in this work.

## introduction st is a field that is very closely aligned with asr and mt, as it is their natural evolution to combine the advances in both areas. thus, the goal is to obtain the translation of an utterance that has been spoken in a different language, without necessarily requiring the intermediate transcription. at the same time, it is desirable to have high quality translations without compromising the speed of the system. although research into st started in the nineties @xcite , the field did not really take off until significant breakthroughs were achieved in asr @xcite @xcite and mt @xcite @xcite , mainly due to the introduction of deep neural networks (nn). thanks to this, the field has recently attracted significant amounts of attention from both the research and industry communities, as the field is now mature enough that it has tangible and well-performing applications @xcite . currently, there are two main approaches to st: cascade and end-to-end models. the goal of the end-to-end approach is to train a single system that is able to carry out the the entire translation process @xcite @xcite . this has only recently been possible thanks to advances in neural modeling. due to a lack of st data, different techniques such as pre-training and data augmentation @xcite have been used in order to alleviate this lack of data. it is important to remark that the currently proposed end-to-end models work in an offline manner and must process the entire input sequence. therefore they cannot be used for a streaming scenario. in the cascade approach, an asr system transcribes the input speech signal, and this is fed to a downstream mt system that carries out the translation. the provided input to the mt step can be the 1-best hypothesis, but also n-best lists @xcite or even lattices @xcite . additional techniques can also be used to improve the performance of the pipeline by better adapting the mt system to the expected input, such as training with transcribed text @xcite or chunking @xcite . the cascade approach can be used to take advantage of independent developments in asr and mt, and it is significantly easier to train due to greater data availability. thus, it is very relevant to study improvements for the st cascade pipeline. this work focuses on the effects of segmentation in streaming st, as cascade systems still outperform end-to-end systems in standard setups @xcite . fol-lowing a cascade approach, a streaming st setup can be achieved with individual streaming asr and mt components. advances in neural streaming asr @xcite @xcite allow the training of streaming models whose performance is very similar to offline ones. recent advances in simultaneous mt show promise @xcite @xcite , but current models have additional modelling and training complexity, and are not ready for translation of long streams of input text. for the scenario to be considered (translation of parliamentary speeches, with an average duration of 100s), it is required for the st systems to have a minimum throughput, but simultaneous translation is not required, so the translation of chunks 1 is still acceptable. in this case we prioritize quality over simultaneous translation, with a streaming asr system followed by a standard offline mt system. this way, the resulting st cascade system can provide transcribed words in real-time, that are eventually split into chunks to be translated by the offline mt system. following this approach, it is necessary to incorporate a segmentation component in the middle in order to split the output of the asr system into (hopefully semantically self-contained) chunks that can be successfully processed by the mt model, while maintaining a balance between latency and quality. in @xcite @xcite , the authors approach this problem by training a monolingual mt system that predicts punctuation marks, and then the asr output is segmented into chunks based on this punctuation. another approach is to segment the asr output by using a language model (lm) that estimates the probability of a new chunk to start @xcite @xcite . binary classifiers with part of speech and reordering features have also been proposed @xcite . it is also possible to segment the asr output using handcrafted heuristics such as those based on a fixed number of words per chunk @xcite or acoustic information @xcite . these heuristic approaches present the disadvantage of being very domain and speaker specific. alternatively, segmentation can be integrated into the decoder, so that it is carried out at the target side rather than the source side @xcite . 1 a chunk must be understood as a sequence of words. this work introduces a statistical framework for the problem of segmentation in st, which incorporates both textual and acoustic information. jointly with this, we propose a set of novel models that follow this framework, and a series of extensive experiments are carried out, which show how these new models outperform previously proposed segmentation models. in addition, we study the effect of the preprocessing scheme applied to the input of the mt system, the performance degradation explained by transcription and/or segmentation errors, and the latency due to the components of the st system. this paper is organized as follows. the next section describes the statistical framework of the segmenter in the streaming st scenario. section 3 follows, detailing how our proposed models are instantiated in this framework. then, section 4 describes the europarl-st dataset that is used in the experiments and the three main components of our streaming st system based on a cascade approach: asr and mt systems, and the segmentation models. next, section 5 reports a detailed evaluation in terms of bleu score on the europarl-st dataset and comparative results with previous work, and latency figures. finally, section 6 draws the main conclusions of this work and devises future research lines. ## statistical framework we define the streaming st segmentation as a problem in which a continuous sequence of words provided as the output of an asr system is segmented into chunks. these chunks will then be translated by a downstream mt system. the goal of the segmentation is to maximize the resulting translation accuracy while keeping latency under the responsetime requirements of our streaming scenario. formally, the segmentation problem is the task of dividing a sequence of input words x j 1 into nonoverlapping chunks. we will represent this with a sequence of split/non-split decisions, y j 1 , with y @xmath0; and y @xmath1, otherwise. optionally, additional input features can be used. in this work, we use word-based acoustic features (a j 1 ) aligned with the sequence of words output by the asr system. however, in a streaming setup, we need to bound the sequence to w words into the future (hereafter, future window) to meet latency requirements previous works in the literature can be stated as a particular case of the statistical framework defined above under certain assumptions. lm based segmentation @xcite @xcite . in this approach, an n-gram lm is used to compute the probability monolingual mt segmentation @xcite @xcite . following this setup, a monolingual mt model translates from the original, (unpunctuated) words x j 1 into a new sequence z j 1 that contains segmentation information (via puntuaction marks). each z j can be understood as a pair (x j , y j ), so the segmentation can be defined as an mt problem ## direct segmentation model now that we have introduced the theoretical framework, we are going to describe our proposed segmentation model. this approach has the advantage of allowing a future dependency and consider not only textual, but also acoustic features. this provides the model with additional evidence for taking a better split/non-split decision. first, the text model computes text state vectors s j+w j that consider each word in x j+w j-h using an embedding function e() and one or more recurrent layers, represented by the function f 1 (). in order to incorporate information about previous decisions y j-1 j-h , we create a new sequence xj+w j-h by inserting an end-of-chunk token into the text input sequence every time a split decision has been taken. this sequence is bounded in length by h. next, the split probability is computed by concatenating the state vectors of the current word and those in the future window, and passing them through a series of feedforward layers f 2 () if we include acoustic information, acoustic state vectors are computed using function f 3 () and are concatenated with the text state vectors in order to compute the split/non-split probability in the case of audio information, we assess two variants, depending whether the acoustic sequence is passed through a rnn (audio w/ rnn) or not (audio w/o rnn). these word-based acoustic feature vectors are obtained as follows. the audio w/o rnn (also referred to as copy) option extracts three acoustic features associated to each word: duration of the current word, duration of the previous silence (if any), and duration of the next silence (if any). the three features were selected due to effectiveness (indeed, they improve system performance) as well as being word-based features which therefore can be directly integrated into the proposed model. at training time, these features are obtained by carrying out a forced alignment between the audio and the reference transcription, while at testing time are directly provided by the asr system. the audio w/ rnn option adds an independent rnn as f 3 , to process the sequence a j+w j-h of three-dimensional acoustic feature vectors just described, and the acoustic state vectors are concatenated at word level with the text state vectors. whenever acoustic features are used, first the text model is pre-trained and frozen, and then the feedforward network is updated with the acoustic data. ## experimental setup to study the effects of our streaming st segmenter in terms of bleu score @xcite , state-of-the-art asr and mt systems were trained to perform st from german (de), spanish (es) and french (fr) into english (en), and vice versa. asr and mt systems were treated as black boxes in order to focus our efforts on evaluating the proposed streaming st segmentation models on the recently released and publicly available europarl-st corpus @xcite . basic statistics of the six language pairs of the europarl-st corpus involved in the evaluation are shown in ## evaluation in order to perform an evaluation that simulates real conditions, the asr hypothesis of an entire speech (intervention made by a mep, with an average duration of 100 seconds) is fed to the segmentation model whose generated chunks are translated by the mt system. the chunks are translated independently from each other. the quality of the mt output, in terms of bleu score, provides a clear indication of the performance of the streaming st segmenter and allows us to compare different segmenters. figure 2 shows bleu scores as a function of the length of the future window for the english-german (en-de) and spanish-english (es-en) dev sets. on the left-hand side, the three segmenters (text, audio w/ rnn and audio w/o rnn) are compared averaging their bleu scores over history sizes 5, 10 and 15 for the sake of clarity. on the right-hand side, the effect of history sizes is analysed for the audio w/o rnn segmenter. in both cases, reference transcriptions were used as input to the segmenter. as observed, the length of the future window is a very significant parameter to decide whether to split or not, which validates our decision to use a model that considers not only past history, but also a future window. in the case of en-de, adding a future window significantly improves the results, up to 5.8 and 3.5 bleu points on average, in the text and audio models, respectively. similarly for es-en, but at a lower magnitude, a gain of up to 3.7 and 3.4 bleu points on average in the text and audio models, respectively, is obtained at larger future windows. when comparing the segmenters (figure 2 on the left), the text segmenter provides a performance that is clearly lower than the audio-based segmenters for the english-german pair, and similar or lower for the spanish-english pair. audio-based segmenters offers nearly the same bleu scores for english-german and spanish-english. however, the audio w/o rnn being a simpler model reaches slightly better bleu scores using future window of length 4. this window length presents an appropriate trade-off between system latency and accuracy in our streaming scenario. focusing on the audio w/o rnn segmenter (figure 2 on the right), longer history sizes such as 10 and 15 clearly provide better bleu scores than the shorter history size (@xmath2. a history size of 10 reaches the best bleu scores for english-german, and similar performance is achieved between 10 and 15 in spanish-english for future window of length 4. based on these results, a history size of 10 and a future window of length 4 were selected for the rest of the experiments. ## conclusions this work introduces a statistical framework for the problem of asr output segmentation in stream-