CMU ’s IWSLT 2024 Offline Speech Translation System: A Cascaded Approach For Long-Form Robustness (2024)

Abstract: This work describes CMU’s submission to the IWSLT 2024 Offline Speech Translation (ST) Shared Task for translating English speech to German, Chinese, and Japanese text. We are the first participants to employ a long-form strategy which directly processes unsegmented recordings without the need for a separate voice-activity detection stage (VAD). We show that the Whisper automatic speech recognition (ASR) model has a hallucination problem when applied out-of-the-box to recordings containing non-speech noises, but a simple noisy fine-tuning approach can greatly enhance Whisper’s long-form robustness across multiple domains. Then, we feed English ASR outputs into fine-tuned NLLB machine translation (MT) models which are decoded using COMET-based Minimum Bayes Risk. Our VAD-free ASR+MT cascade is tested on TED talks, TV series, and workout videos and shown to outperform prior winning IWSLT submissions and large open-source models.

## introduction cmu's submission to the iwslt 2024 offline speech translation shared task is a cascaded automatic speech recognition (asr) and machine translation (mt) system designed to effectively translate english speech from long unsegmented recordings, such as ted talks, tv series, and workout videos, into german, chinese, and japanese text. typically systems are short-form, meaning they are dependent on some voice-activity detection to first convert long recordings which contain speech and non-speech noises into short segments of speech. this makes it relatively easy to train a short-form model and test it on similar clean speech segments. however, these systems exhibit alarming brittleness in the wild; results from recent iterations of the offline st track have shown large fluctuations in performance between different segmentations of the same test set @xcite @xcite . why are these short-form systems brittle in-thewild (or in iwslt by proxy)? our view is that these systems are plagued by train/test mismatch. common training sets, e.g. @xcite , are produced using sentence-level forced alignment. in other words, this training segmentation can only be obtained given a reference. for a blind test set however, forced alignment is not possible. instead, practitioners have resorted to using vad with additional tricks to reduce the train/test mismatch, such as heuristically replicating segment characteristics @xcite or modeling the segmentation pattern of training data @xcite . these methods of approximating the training data segmentation may work within a single domain but are complex to configure for multi-domain scenarios. in this work, we explore long-form processing of unsegmented recordings via a 30 second sliding window as an alternative to segment-dependent speech processing. our system consists of: our experiments first show that whisper out-ofthe-box has a hallucination problem caused by nonspeech noises during long-form inference. we then show that our noisy fine-tuning broadly addresses these hallucinations. finally, we show the ultimate cascaded st performance across multiple domains: ted talks, tv series, and workout videos. the iwslt 2024 offline speech translation shared task consists of three language pairs: english-to-german, english-to-chinese, and english-to-japanese. for all three language pairs, unsegmented ted talks (5-20 min) are given as shared task evaluation data. as a dev set, we use the provided tst2020 (ted'20), tst2021 (ted'21), and tst2022 (ted'22) for english-to-german and tst2022 (ted'22) for english-to-chinese and english-to-japanese. for english-to-german, systems are also tested on additional domains: tv series (45-60 min), workout videos (10-20 min), and accented speech (5-20 min). we therefore use two additional dev sets obtained from the iwslt 2024 subtitling shared task: itv and peloton. we evaluate asr using case-sensitive punctuated word error-rate (wer ↓) against recordinglevel references. we evaluate mt systems using comet ↑ @xcite ) against sentence-level references. we evaluate st systems using comet after first performing minimum wer alignment of our hypothesis to sentence-level references. note that for chinese and japanese, this alignment is done at the character level. we use must-c v3 (di gangi et al., 2019) for fine-tuning asr models on the ted domain. we use ted2020 for fine-tuning english-to-german mt and must-c for english-to-chinese and english-to-japanese. for multi-domain finetuning we also add bazinga tv series asr data @xcite and a 500k subset of opensubtitles mt data @xcite . note that our use of bazinga (as well as the use of whisper) puts our system under the "unconstrained" designation. ## system description figure 1 summarizes the components in our asr+mt cascade. the following section describes the system in greater detail, referring at times to the summary figure. ## conclusion we describe our iwslt 2024 offline speech translation system which is based on long-form processing of unsegmented recordings. our system consists of fine-tuned whisper and nllb components of a cascade. we evaluate our system on ted talks, tv series, and workout videos.