KIT ’s Multilingual Speech Translation System for IWSLT 2023 (2023)

Abstract: Many existing speech translation benchmarks focus on native-English speech in high-quality recording conditions, which often do not match the conditions in real-life use-cases. In this paper, we describe our speech translation system for the multilingual track of IWSLT 2023, which focuses on the translation of scientific conference talks. The test condition features accented input speech and terminology-dense contents. The tasks requires translation into 10 languages of varying amounts of resources. In absence of training data from the target domain, we use a retrieval-based approach ( k NN-MT) for effective adaptation ( +0.8 BLEU for speech translation). We also use adapters to easily integrate incremental training data from data augmentation, and show that it matches the performance of re-training. We observe that cascaded systems are more easily adaptable towards specific target domains, due to their separate modules. Our cascaded speech system outperforms its end-to-end counterpart on scientific talk translation, although their performance remains similar on TED talks.

## introduction this paper summarizes karlsruhe institute of technology's speech translation system for the multilingual track of iwslt 2023 @xcite . in this track, the task is to translate scientific talks in english into 10 languages: arabic (ar), chinese (zh), dutch (nl), french (fr), german (de), japanese (ja), persian/farsi (fa), portuguese (pt), russian (ru), turkish (tr). the talks are from presentations in the 60th annual meeting of the association for computational linguistics @xcite . translating scientific talks presents several challenges. on the source side, most speakers are non-native, and the recording conditions often vary. this requires acoustic robustness to accents and noise. on the target side, domain-specific terminologies are frequently used, calling for accurate translation of these words that rarely occur in the training data. the styles of the talks, e.g. formality, also differ from other domains. as no training data from the same domain is provided, effective few-shot or zero-shot adaptation is crucial. as the task focuses on one-to-many translation, it is also an interesting testbed for whether multilinguality improves speech translation quality. for text-to-text translation, the gain from multilinguality is mostly concentrated in many-to-one translation @xcite , i.e., multilinguality on the source side. in contrast, for x-to-many translation, it remains unclear whether incorporating more target languages improves translation quality. in this system description paper, we present cascaded and end-to-end systems for the englishto-many speech translation task. we leverage pretrained models, including wavlm @xcite , mbart50 @xcite , and deltalm @xcite . the systems do not use additional data beyond the allowed corpora, and therefore fall under the constrained data condition. for the cascaded system, to handle the unique style of scientific talks, we use knn-mt @xcite to bias the output generation towards the target domain. moreover, as no target monolingual data is provided, we use data diversification @xcite to enrich the existing parallel data. we also use adapters @xcite as a lightweight approach for incremental learning and language adaptation. for the asr model, we improve over last year's performance by using a more recent audio encoder @xcite and adding a dedicated decoder. to adapt the asr system to the target domain, we use n-gram re-weighting and synthesized data for the target domain. for the end-to-end system, we use our machine translation model for knowledge distillation. we also ensemble models trained with and without synthesized speech data. our main findings are as follow: ## data and preprocessing after describing the evaluation data ( §2.1), we outline the training data and preprocessing steps for our automatic speech recognition (asr; §2.2), machine translation (mt; §2.3), casing/punctuation restoration ( §2.4), and speech translation (st; §2.5) models. ## cascaded system for the cascaded system, we introduce our asr ( §3.1) and mt ( §3.2) models. ## end-to-end system for the end-to-end system, similar to our asr model, after seeing initial favourable results of wavlm over wav2vec, we choose wavlm as the audio encoder. following last year's submission @xcite , we use the mbart50 decoder. the results are shown in row (6) of compared to the strongest cascaded system (row (5)), the end-to-end system falls behind 2.6 bleu on acl dev. on ted, however, it appears to slightly outperform the cascaded system. one explanation is that the mt model of the cascaded system has not been separately adapted to ted texts (although parts of the full training data do cover ted data), which was shown essential in improving performance on ted test sets @xcite . the end-to-end system, on the other hand, has seen a larger proportion of ted data in training ( similar to the previous year @xcite , we also adapt our end-to-end offline model for simultaneous track @xcite . ## conclusion in this paper, we described our systems for the multilingual speech translation track of iwslt 2023, which translates english speech into 10 target languages. to tackle the task of translating scientific conference talks, which feature non-native input speech and terminology-dense contents, our systems have several novelties. lacking suitable training data for the target domain, we used knn-mt for inference-time adaptation and showed an improvement of +0.8 bleu for cascaded speech translation system. we also used adapters to integrate incremental data from augmentation, and achieved performance on-par with re-training on all data. in our experiments, we observed that cascaded systems are more easily adaptable towards desired target domains due to their separate modules. our cascaded speech system outperforms its end-to-end counterpart on scientific talk translation, although their performance remains similar on ted talks. for future work, we are interested in the feasibility of applying the adaptation approaches shown effective on mt to end-to-end st.