CMU ’s IWSLT 2023 Simultaneous Speech Translation System (2023)

Abstract: This paper describes CMU’s submission to the IWSLT 2023 simultaneous speech translation shared task for translating English speech to both German text and speech in a streaming fashion. We first build offline speech-to-text (ST) models using the joint CTC/attention framework. These models also use WavLM front-end features and mBART decoder initialization. We adapt our offline ST models for simultaneous speech-to-text translation (SST) by 1) incrementally encoding chunks of input speech, re-computing encoder states for each new chunk and 2) incrementally decoding output text, pruning beam search hypotheses to 1-best after processing each chunk. We then build text-to-speech (TTS) models using the VITS framework and achieve simultaneous speech-to-speech translation (SS2ST) by cascading our SST and TTS models.

## introduction in this paper, we present cmu's english to german simultaneous speech translation systems. our iwslt 2023 @xcite shared task submission consists of both simultaneous speechto-text (sst) and simultaneous speech-to-speech (ss2st) systems. our general strategy is to first build large-scale offline speech translation (st) models which leverage unpaired speech data, asr data, and st data. we then adapt these offline models for simultaneous inference. finally, we use a text-to-speech model to achieve ss2st in a cascaded manner. in particular, our system consists of: ## task description the iwslt 2023 simultaneous speech translation track foot_0 is a shared task for streaming speech-totext and speech-to-speech translation of ted talks. this track mandates that systems do not perform re-translation, meaning that the streaming outputs cannot be edited after the system receives more input audio. systems are required to meet a particular latency regime: sst systems must @xmath0.5 seconds start offset (so) @xcite . of the allowed training data, we selected a subset of in-domain data to train our asr and st models: for asr we use tedlium v1 and v2 @xcite and for st we used @xcite . we also use a set of cross-domain data to train our mt and tts models due to the lack of in-domain data: for mt we use europarl, newscommentary, open-subtitles, ted2020, tatoeba, and elrc-cordis news @xcite . for tts we use commonvoice @xcite . the following section describes how each of the asr, st, mt, and tts components fit together in our ultimate systems. ## experimental setup our models were developed using the espnet-st-v2 toolkit @xcite . our st/sst model uses wavlm-large as a front-end @xcite . a linear pre-encoder down-samples from 1024 to 80 feature dim. our encoder is a 12 layer conformer with 1024 attention dim, 8 attention heads, and 2048 linear dim @xcite . a convolutional post-encoder then down-samples along the length dimension by a factor of 2. our decoder follows the mbart architecture and we initialize using the mbart-large-50-many-to-many model @xcite . our st ctc branch uses the same 250k vocabulary as the mbart decoder to enable joint decoding. our tts model consists of 6 transformer encoder layers for text-encoder, 4 normalizing flow layers for duration predictor, 16 residual dilated convolutional blocks as posterior encoder and multi-period hifigan @xcite style decoder. we train vits model for 400 epochs with adamw @xcite optimizer. during inference, we use a chunk size of 2 sec-onds for sst and 2.5 seconds for ss2st. for both sst and ss2st we use beam size 5, ctc weight 0.2, and no length penalty/bonus. to account for incremental outputs which end in a prefix of a word rather than a whole word, we delay outputs for scoring by 1 token. there are two exceptions to this token delay: if the last token is a valid german word or a punctuation, then we do not delay. we evaluate translation quality using bleu score @xcite for st/sst and asr-bleu score for ss2st. st/sst references are case-sensitive and punctuated while ss2st references are case-insensitive and un-punctuated. the asr model used for asr-bleu is whisper-small @xcite . we evaluate translation latency for sst using average lagging (al) @xcite and length-adaptive average lagging (laal) @xcite . we evaluate translation latency for ss2st using start (so) and end-offset (eo) @xcite . ## results from st to sst, we observe a 6% quality degradation. note that the average duration of tst-common utterances is around 5 seconds, meaning the corresponding latency gain is 60%. from sst to ss2st, we observe a 12% quality degradation. note that both the tts model and the whisper asr model powering the asr-bleu metric contribute to this gap. ## conclusion we describe our english to german simultaneous speech-to-text and speech-to-speech translation systems for the iwslt 2023 shared task. we start by building large-scale offline speech-to-text systems which leverage self-supervised speech and text representations. we then adapt these offline models for online inference, enabling simultaneous speech-to-text translation. finally, we feed streaming text outputs to a down-stream tts model, enabling simultaneous speech-to-speech translation.