Effective combination of pretrained models - KIT @ IWSLT 2022 (2022)

Abstract: Pretrained models in acoustic and textual modalities can potentially improve speech translation for both Cascade and End-to-end approaches. In this evaluation, we aim at empirically looking for the answer by using the wav2vec, mBART50 and DeltaLM models to improve text and speech translation models. The experiments showed that the presence of these models together with an advanced audio segmentation method results in an improvement over the previous end-to-end system by up to 7 BLEU points. More importantly, the experiments showed that given enough data and modeling capacity to overcome the training difficulty, we can outperform even very competitive Cascade systems. In our experiments, this gap can be as large as 2.0 BLEU points, the same gap that the Cascade often led over the years.

## introduction speech translation (st) has been the main theme of iwslt for more than a decade and it goes without saying between the traditional cascade approach and the recent end-to-end (e2e) possibility, the former has always been preferred. being able to divide the complicated st to smaller sub-problems: automatic recognition, (often) re-segmentation @xcite and machine translation, the cascade approach has the advantage of using more data to separately optimize the components. the e2e, on the other hand, relies on a single network architecture that requires an explicit speech-translation dataset. over the years of participation, we observed that the performance gap between e2e and cascade is reduced @xcite , and there are three negative factors that outweigh the advantages of having a single architecture without the problem of error propagation @xcite . in our work, we massively improved our endto-end slt systems for @xmath0: the new solution is not only more accurate, but also directly optimized on ted talks giving the translation model more precise and complete segmentations compared to the generic voice activity detectors. moreover, we also applied the same techniques to improve the speech recognition and machine translation components of the cascade system. they also benefit from the above factors, albeit to a limited extent. unlike previous years when the cascade was always the better performing system, for the first time we selected the e2e as our primary submission. for the current evaluation campaign (anastasopoulos et al., 2022), we also expanded the slt systems for two new directions: @xmath1, with both of the approaches available. the resulting system is also used in a simultaneous setting located in the same evaluation campaign @xcite . ## cascade system for offline speech translation we address the offline speech translation task by two main approaches, namely cascade and end-toend. in the cascaded condition, the asr module (section 3.1) receives audio inputs and generates raw transcripts, which will then pass through a segmentation module (section 3.2) to formulate well normalized inputs to our machine translation module (section 3.3). the mt outputs are the final outputs of the cascade system. on the other hand, the end-to-end architecture is trained to directly translate english audio inputs into german text outputs (section 3.4). ## conclusion if the end-to-end models remained as a promising approach in the previous evaluation campaigns, it eventually blooms as the superior solution when the conditions are met to overcome its problems, namely training difficulty, segmentation issues and inefficient data usage. while the performance of the e2e system is now better, we can still believe that its far from being practical given the size of the model and the required presence of an audio segmenter. moreover, the cascade system is still necessary since it can provide a distillation tool for the e2e, via pseudolabels for better data utilization. the development of both approaches remains to be interesting awaiting the future achievement in multilingual and multimodal unsupervised and selfsupervised training.