CMU ’s IWSLT 2022 Dialect Speech Translation System (2022)

Abstract: This paper describes CMU’s submissions to the IWSLT 2022 dialect speech translation (ST) shared task for translating Tunisian-Arabic speech to English text. We use additional paired Modern Standard Arabic data (MSA) to directly improve the speech recognition (ASR) and machine translation (MT) components of our cascaded systems. We also augment the paired ASR data with pseudo translations via sequence-level knowledge distillation from an MT model and use these artificial triplet ST data to improve our end-to-end (E2E) systems. Our E2E models are based on the Multi-Decoder architecture with searchable hidden intermediates. We extend the Multi-Decoder by orienting the speech encoder towards the target language by applying ST supervision as hierarchical connectionist temporal classification (CTC) multi-task. During inference, we apply joint decoding of the ST CTC and ST autoregressive decoder branches of our modified Multi-Decoder. Finally, we apply ROVER voting, posterior combination, and minimum bayes-risk decoding with combined N-best lists to ensemble our various cascaded and E2E systems. Our best systems reached 20.8 and 19.5 BLEU on test2 (blind) and test1 respectively. Without any additional MSA data, we reached 20.4 and 19.2 on the same test sets.

## introduction in this paper, we present cmu's tunisian-arabic to english st systems submitted to the iwslt 2022 dialectal st track @xcite . one of our goals is to investigate dialectal transfer from large msa asr and mt corpora to improve tunisian-arabic st performance. we also view this task as setting for extending the sequence-level knowledge distillation (seqkd) @xcite , e2e multi-decoder architecture @xcite , and system combination methods in our iwslt 2021 offline st systems @xcite . in particular, our contributions are the following: ## task description and data preparation the arabic language is not a monolith. of its estimated 400 million native speakers, many speak in colloquial dialects such as, tunisian-arabic, that have relatively less standard orthographic rules and smaller asr and mt corpora compared to formal msa @xcite . both of these realities present challenges to building effective st systems, and as such the dialectal speech translation shared task is an important venue for tackling these research problems. following the shared task guidelines, punctuation is removed and english text is lowercased. buckwalter one-to-one transliteration of arabic text @xcite was applied to help non-arabic speakers with asr output interpretation. english sentences were tokenized with the tokenizer.perl script in the moses toolkit @xcite for training and detokenized for scoring. language-specific sentencepiece vocabularies were created using the byte pair encoding (bpe) algorithm @xcite with the sentencepiece toolkit. 2 speech data was up-sampled by a factor of 3 using 0.9 and 1.1 speed perturbation ratios @xcite . the iwslt22-dialect data was upsampled to 16khz for consistency using the sox toolkit foot_2 . ## proposed methods in this section, we describe our cascaded ( §3.1) and e2e systems ( §3.2). then we describe methods for integrating both approaches §3.3.