Does Joint Training Really Help Cascaded Speech Translation? (2022)

Abstract: Currently, in speech translation, the straightforward approach - cascading a recognition system with a translation system - delivers state-of-the-art results.However, fundamental challenges such as error propagation from the automatic speech recognition system still remain.To mitigate these problems, recently, people turn their attention to direct data and propose various joint training methods.In this work, we seek to answer the question of whether joint training really helps cascaded speech translation.We review recent papers on the topic and also investigate a joint training criterion by marginalizing the transcription posterior probabilities.Our findings show that a strong cascaded baseline can diminish any improvements obtained using joint training, and we suggest alternatives to joint training.We hope this work can serve as a refresher of the current speech translation landscape, and motivate research in finding more efficient and creative ways to utilize the direct data for speech translation.

## introduction speech translation (st) is the task of automatic translation of speech in some source language into some other target language @xcite . traditionally, a cascaded approach is used, where an automatic speech recognition (asr) system is used to transcribe the speech, followed by a machine translation (mt) system, to translate the transcripts @xcite . the problem of error propagation has been the center of discussion in st literature @xcite @xcite @xcite , and instead of using the discrete symbols in the source languages, ideas like using n-best lists, lattices, and neural network hidden representations are investigated @xcite @xcite @xcite . for a more sys-tematic review of the st development, we refer the readers to @xcite . with recent efforts in the expansion of the st data collection @xcite , more and more direct st data is available. such direct data comes as pairs of source speech and target translation, and often as triplets further including source transcriptions. various joint training methods are proposed to use such data to improve cascaded systems, with the hope that uncertainties during transcription can be passed on to translation to be resolved. here, what we call "joint training" is often referred to as "end-to-end training" in the literature, where the direct st data is utilized in the joint optimization of the asr and mt models @xcite @xcite @xcite @xcite . in this work, we revisit the principal question of whether or not joint training really helps cascaded speech translation. ## cascaded approach in traditional cascaded systems, an asr model p(f j 1 |x t 1 ) and an mt model p(e i 1 |f j 1 ) are trained separately, where we denote speech features as x t 1 , transcriptions as f j 1 , and translations as e i 1 . the decoding is done in two steps: the argmax is approximated using beam search for computational reasons, and we will assume a fixed beam size n for the decoding of both transcriptions and translations. ## experimental results and analysis we focus on the must-c english-german speech translation task @xcite in the domain of ted talks and evaluate on test-he and test-common. we use an in-house filtered subset of the iwslt 2021 english-german dataset as in @xcite , which contains 1.9m segments (2300 hours) of asr data and 24m parallel sentences of mt data. the in-domain asr data comprises must-c, ted-lium, and iwslt ted, while the out-of-domain asr data consists of europarl, how2, librispeech, and mozilla common voice. for translation, the dataset contains 24m parallel sentences of in-domain translation data (must-c, ted-lium, and iwslt ted), as well as out-of-domain translation data (newscommentary, europarl, wikititles, paracrawl, com-moncrawl, rapid, opensubtitles2018). for st data, we only use must-c. we provide further details in appendix a. depending on whether or not fine-tuned on in-domain asr and mt data, we split our experiments into two sets: a1-a5 and b1-b5. model data must-c en-de mt asr st tst-he tst-common bleu ter bleu ter bahar et al. (2021) cascade ✓ ✓ -25.0 59.2 25.9 56.2 tight integrated cascade ✓ ✓ ✓ 26.8 57.2 26.5 54.8 xu et al. (2021) cascade --✓ --23.3 cascade ✓ ✓ ✓ --28.1 direct + pretraining + multi-task --✓ --25.2 direct + pretraining + multi-task ✓ ✓ ✓ --28.1 -dalmia et al. (2021) searchable hidden intermediates --✓ --26.4 -inaguma et al. (2021) cascade ✓ ✓ -26.1 -29.4 direct + kd ✓ ✓ ✓ 27.4 -30.9 searchable hidden intermediates + kd ✓ ✓ ✓ --30.8 this work (no fine-tuning) a1 ground-truth transcript mt ✓ (✓) -30.4 54.0 32.5 48.9 a2 cascade ✓ ✓ -27.7 57.8 29.0 53.6 a3 tight integrated cascade ✓ ✓ ✓ 28.7 56.1 29.2 53.5 a4 top-k-train ✓ ✓ ✓ 28.7 56.9 30.0 52.5 a5 top-k-search ✓ ✓ 28.2 57.1 29.4 53.1 this work (fine-tuned asr and mt) b1 ground-truth transcript mt ✓ (✓) -31.1 53.2 33.7 48.2 b2 cascade ✓ ✓ -29.1 56.6 30.5 52.2 b3 tight integrated cascade ✓ ✓ ✓ 29.4 55.9 30.1 52.5 b4 top-k-train ✓ ✓ ✓ 29.4 55.9 30.5 51.9 b5 top-k-search ✓ ✓ -29.4 56.4 30.8 51.9 without fine-tuning, we observe that both the tight integrated cascade @xcite (a3) and our marginalization approach (a4) outperform the baseline cascade (a2) on both test-he and test-common ( our experimental results suggest that the joint training of cascaded speech translation models does not seem to be effective. this poses the questions: why is that and what were we trying to achieve with joint training anyways? sperber and paulik (2020) highlighted three main issues with st, and in the following, we will discuss these issues from the perspective of joint training. ## conclusion in this work, we analyzed several reasons why joint training does not appear to help when individual automatic speech recognition and machine translation components are stronger. we point out that cascaded models achieve state-of-the-art performance when fine-tuned on in-domain st data. while we do not suggest that joint training is not worth exploring, we want to encourage future research to consider data conditions more carefully and produce strong cascade baselines. concretely, we suggest (1) the inclusion of external asr and mt data, (2) training strong cascade baselines by fine-tuning both asr and mt models on in-domain transcript-like data, if available and (3) the investigation of data conditions where only out-of-domain st data is available, while allowing in-domain asr and mt data.