S imul MT to S imul ST : Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation (2020)

Abstract: We investigate how to adapt simultaneous text translation methods such as wait- k and monotonic multihead attention to end-to-end simultaneous speech translation by introducing a pre-decision module. A detailed analysis is provided on the latency-quality trade-offs of combining fixed and flexible pre-decision with fixed and flexible policies. We also design a novel computation-aware latency metric, adapted from Average Lagging.

## introduction simultaneous speech translation (simulst) generates a translation from an input speech utterance before the end of the utterance has been heard. simulst systems aim at generating translations with maximum quality and minimum latency, targeting applications such as video caption translations and real-time language interpreter. while great progress has recently been achieved on both end-to-end speech translation @xcite and simultaneous text translation (simulmt) @xcite @xcite @xcite @xcite @xcite , little work has combined the two tasks together @xcite . end-to-end simulst models feature a smaller model size, greater inference speed and fewer compounding errors compared to their cascade counterpart, which perform streaming speech recognition followed by simultaneous machine translation. in addition, it has been demonstrated that end-to-end simulst systems can have lower latency than cascade systems @xcite . 1 the code is available at https://github.com/ pytorch/fairseq in this paper, we study how to adapt methods developed for simulmt to end-to-end simulst. to this end, we introduce the concept of pre-decision module. such module guides how to group encoder states into meaningful units prior to making a read/write decision. a detailed analysis of the latency-quality trade-offs when combining a fixed or flexible pre-decision module with a fixed or flexible policy is provided. we also introduce a novel computation-aware latency metric, adapted from average lagging (al) @xcite . ## task formalization a simulst model takes as input a sequence of acoustic features @xmath0, ...x |x| ] extracted from speech samples every t s ms, and generates a sequence of text tokens @xmath1, ..., y |y | ] in a target language. additionally, it is able to generate y i with only partial input x 1:n(y i ) = [x 1 , ...x n(y i ) ], where n(y i ) ≤ |x| is the number of frames needed to generate the i-th target token y i . note that n is a monotonic function, i.e. n(y i-1 ) ≤ n(y i ). a simulst model is evaluated with respect to quality, using bleu @xcite , and latency. we introduce two latency evaluation methods for simulst that are adapted from simulmt. we first define two types of delays to generate the word y i , a computation-aware (ca) and a non computation-aware (nca) delay. the ca delay of y i , d ca (y i ), is defined as the time that elapses (speech duration) from the beginning of the process to the prediction of y i , while the nca delay for y i d ca (y i ) is defined by d nca (y i ) = t s • n(y i ). note that d nca is an ideal case for d ca where the computational time for the model is ignored. both delays are measured in milliseconds. two types of latency measurement, l ca and l n ca , are calculated accordingly: @xmath2, ..., d(y |y | )]. to better evaluate the latency for simulst, we introduce a modification to al. we assume an oracle system that can perform perfect simultaneous translation for both latency and quality, while in @xcite the oracle is ideal only from the latency perspective. we evaluate the lagging based on time rather than steps. the modified al metric is defined in eq. ( 1 ): ## experiments we conduct experiments on the english-german portion of the must-c dataset @xcite , where source audio, source transcript and target translation are available. we train on 408 hours of speech and 234k sentences of text data. we use kaldi @xcite to extract 80 dimensional log-mel filter bank features, computed with a 25ms window size and a 10ms window shift. for text, we use sentencepiece @xcite to generate a unigram vocabulary of size 10,000. we use gentle foot_0 to generate the alignment between source text and speech as the label to generate the oracle flexible predecision module. translation quality is evaluated with case-sensitive detokenized bleu with sacrebleu @xcite . the latency is evaluated with our proposed modification of al @xcite . all results are reported on the must-c dev set. all speech translation models are first pretrained on the asr task where the target vocabulary is character-based, in order to initialize the encoder. we follow the same hyperparameter settings from @xcite . we follow the latency regularization method introduced by @xcite , the objective function to optimize is where c is a latency metric (al in this case) and d is described in section 2. only samples with @xmath3. for the models with monotonic multihead attention, we first train a model without latency with λ @xmath4. after the model converges, λ latency is set to a desired value and the model is continue trained until convergence. the latency-quality trade-offs of the 4 types of model from the combination of fixed or flexible predecision with fixed or flexible policy are presented in fig. 2 . the non computation-aware delays are used to calculate the latency metric in order to evaluate those trade-offs from a purely algorithmic perspective. fixed pre-decision + fixed policy foot_1 (fig. 2a ). as expected, both quality and latency increase with step size and lagging. in addition, the latencyquality trade-offs are highly dependent on the step size of the pre-decision module. for example, with step size 120ms, the performance is very poor even with large k because of very limited information being read before writing a target token. large step sizes improve the quality but introduce a lower bound on the latency. note that step size 280ms, which provides an effective latency-quality tradeoff compared to other step sizes, also matches the average word length of 271ms. this motivates the study of a flexible pre-decision module based on word boundaries. ## conclusion we investigated how to adapt simulmt methods to end-to-end simulst by introducing the concept of pre-decision module. we also adapted average lagging to be computation-aware. the effects of combining a fixed or flexible pre-decision module