Streaming Models for Joint Speech Recognition and Translation (2021)

Abstract: Using end-to-end models for speech translation (ST) has increasingly been the focus of the ST community. These models condense the previously cascaded systems by directly converting sound waves into translated text. However, cascaded models have the advantage of including automatic speech recognition output, useful for a variety of practical ST systems that often display transcripts to the user alongside the translations. To bridge this gap, recent work has shown initial progress into the feasibility for end-to-end models to produce both of these outputs. However, all previous work has only looked at this problem from the consecutive perspective, leaving uncertainty on whether these approaches are effective in the more challenging streaming setting. We develop an end-to-end streaming ST model based on a re-translation approach and compare against standard cascading approaches. We also introduce a novel inference method for the joint case, interleaving both transcript and translation in generation and removing the need to use separate decoders. Our evaluation across a range of metrics capturing accuracy, latency, and consistency shows that our end-to-end models are statistically similar to cascading models, while having half the number of parameters. We also find that both systems provide strong translation quality at low latency, keeping 99% of consecutive quality at a lag of just under a second.

## introduction speech translation (st) is the process of translating acoustic sound waves into text in a different language than was originally spoken in. this paper focuses on st in a particular setting, as described by two characteristics: (1) we desire models that translate in a streaming fashion, where * work done during an internship with apple users desire the translation before the speaker has finished. this setting poses additional difficulties compared to consecutive translation, forcing systems to translate without knowing what the speaker will say in the future. (2) furthermore, the speaker may want to verify that their speech is being processed correctly, intuitively seeing a streaming transcript while they speak @xcite . for this reason, we consider models that produce both transcripts and translation jointly. 1 previous approaches to streaming st have typically utilized a cascaded system that pipelines the output of an automatic speech recognition (asr) system through a machine translation (mt) model for the final result. these systems have been the preeminent strategy, taking the top place in recent streaming @xcite @xcite . despite the strong performance of these cascaded systems, there are also some problems: error propagation from asr output to mt input @xcite ; asr/mt training data mismatch and loss of access to prosodic/paralinguistic speech information at the translation stage @xcite ; and potentially sub-optimal latencies in the streaming context. end-to-end (e2e) models for st have been proposed to remedy these problems, leveraging the simplicity of a single model to sidestep these issues. e2e models are also appealing from computational and engineering standpoints, reducing model complexity and decreasing parameter count. although initial research has explored e2e models for joint speech recognition and translation, no previous works have examined them in the streaming case, a crucial step in using them for many real-world applications. to understand this area more fully, we develop an e2e model to compare with its cascading counterpart in this simultaneous joint task. we build off the models proposed by @xcite in the consecutive case, extending them for use in the streaming setting. we also use the re-translation technique introduced by @xcite to maintain simplicity while streaming. to reduce model size, we introduce a new method for e2e inference, producing both transcript and translation in an interleaved fashion with one decoder. as this task requires a multi-faceted evaluation along several axes, we provide a suite of evaluations to highlight the differences of these major design decisions. this suite includes assessing translation quality, transcription quality, lag of the streaming process, output flicker, and consistency between the transcription and translation. we find that our e2e model performs similarly to the cascaded model, indicating that e2e networks are a feasible and promising direction for streaming st. ## proposed method network architecture in the st survey provided by @xcite , they introduce several e2e models that could be used for the joint setting. as our work focuses on providing a simple but effective approach to streaming st, we focus on the concat model, which generates both the transcript and translation in a concatenated fashion. we compare this e2e model against the standard cascading approach, following the architecture and hyperparameter choices used in @xcite . all audio input models use the same multi-layer bidirectional lstm architecture, stacking and downsampling the audio by a factor of three before processing. we note that although bidirectional encoders are unusual with standard asr architectures, re-translation makes them possible. the cascaded model's textual encoder follows the architecture described in @xcite but replaces self-attention blocks with lstms. decoder networks are similar, but use unidirectional lstms. more implementation details can be found in appendix a. in order to reduce model size and inference time for e2e networks, we introduce a novel method for interleaving both transcript and translation in generation, removing the need to use separate decoders. this method extends the concat model proposed by @xcite to jointly decode according to the ratio given by the parameter γ (fig- outputs do wollen you sie want gehen to eos2 go eos1 language tokens src trg src trg src trg src trg src src target @xmath0.5 outputs do you want to go eos1 wollen sie gehen eos2 language tokens src src src src src src trg trg trg trg target @xmath1.0 outputs wollen sie gehen eos2 do you want to go eos1 language tokens trg trg trg trg src src src src src src target @xmath2.0 figure 1: example token representations (en de) for three different interleaving parameters (section 2). language tokens indicate whether the data corresponds to the source transcript or the target translation and are used with a learned embedding that is summed with the word embeddings, as described in sperber et al. (2020). this approach enables us to produce tokens in an interleaving fashion, given the hyperparameter γ. ## experimental settings data we use the must-c corpus @xcite since it is the largest publicly available st corpus, consisting of ted talks with their english transcripts and translations into eight other language pairs. the dataset consists of at least 385 hours of audio for each target language. we utilize the log mel filterbank speech features provided with the corpus as input for the asr and e2e models. to prepare the textual data, we remove non-speech artifacts (e.g. "(laughter)" and speaker identification) and perform subword tokenization using sentencepiece (kudo and richard- model en de incr. en de full en es incr. en es full mean incr. mean full cascade 13.8 13.2 12.2 11.6 14.1 13.4 e2e γ=0.0 17.6 16.7 14.9 13.8 17.0 16.0 e2e γ=0.3 17.2 16.6 14.3 13.7 16.6 15.8 e2e γ=0.5 17.8 16.5 14.8 13.3 17.3 15.7 e2e γ=1.0 17.3 16.8 14.9 13.7 16.9 15.8 ## results results for the quality-latency curves created by the use of constrained decoding and mask-k (section 3) are shown in figure 2 . unconstrained settings are used for all results in table form. for convenience, bold scores indicate the highest performing models in each metric according to a bootstrap permutation test. translation quality we see in transcription quality conversely, ## conclusion we focus on the task of streaming speech translation, producing both a target translation and a source transcript from an audio source. we develop an end-to-end model to avoid problems that arise from the use of cascaded models for streaming st. we further introduce a new method for joint inference for end-to-end models, generating both translation and transcription tokens concurrently. we show that our novel end-to-end model, with only half the number of parameters, is comparable to standard cascaded models across a variety of evaluation categories: transcript and translation quality, lag of streaming, consistency between transcript and translation, and re-translation flicker. we hope that this will spur increased interest in using end-to-end models for practical applications of streaming speech translation. yarin gal and zoubin ghahramani. 2016. dropout as a bayesian approximation: representing model uncertainty in deep learning. in international conference on machine learning (icml). antonino mattia di gangi, roldano cattoni, luisa bentivogli, matteo negri, and marco turchi. 2019. must-c : a multilingual speech translation corpus. in north american chapter of the association for computational linguistics (naacl), minneapolis, usa. xavier glorot and yoshua bengio. 2010. understanding the difficulty of training deep feedforward neural networks. in international conference on artificial intelligence and statistics (aistats), sardinia, italy. alex graves, santiago fernández, faustino gomez, and jürgen schmidhuber. 2006. connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. in proceedings of the 23rd international conference on machine learning, pages 369-376. roger hsiao, ashish venugopal, thilo köhler, ting zhang, paisarn charoenpornsawat, andreas zollmann, stephan vogel, alan w. black, tanja schultz, and alex waibel. 2006. optimizing components for handheld two-way speech translation for an english-iraqi arabic system. in annual conference of the international speech communication association (in-terspeech), pages 765-768, pittsburgh, usa. niehues jan, roldano cattoni, stuker sebastian, matteo negri, marco turchi, salesky elizabeth, sanabria ramon, barrault loic, specia lucia, and marcello federico. 2019. the iwslt 2019 evaluation campaign. in 16th international workshop on spoken language translation 2019. diederik p. kingma and jimmy l. ba. 2014. adam: a method for stochastic optimization. in international conference on learning representations (iclr), banff, canada. taku kudo and john richardson. 2018. sentencepiece: a simple and language independent subword tokenizer and detokenizer for neural text processing. empirical methods in natural language processing (emnlp), pages 66-71. hang le, juan pino, changhan wang, jiatao gu, didier schwab, and laurent besacier. 2020. dualdecoder transformer for joint automatic speech recognition and multilingual speech translation. in