The USTC ’s Dialect Speech Translation System for IWSLT 2023 (2023)

Abstract: This paper presents the USTC system for the IWSLT 2023 Dialectal and Low-resource shared task, which involves translation from Tunisian Arabic to English. We aim to investigate the mutual transfer between Tunisian Arabic and Modern Standard Arabic (MSA) to enhance the performance of speech translation (ST) by following standard pre-training and fine-tuning pipelines. We synthesize a substantial amount of pseudo Tunisian-English paired data using a multi-step pre-training approach. Integrating a Tunisian-MSA translation module into the end-to-end ST model enables the transfer from Tunisian to MSA and facilitates linguistic normalization of the dialect. To increase the robustness of the ST system, we optimize the model’s ability to adapt to ASR errors and propose a model ensemble method. Results indicate that applying the dialect transfer method can increase the BLEU score of dialectal ST. It is shown that the optimal system ensembles both cascaded and end-to-end ST models, achieving BLEU improvements of 2.4 and 2.8 in test1 and test2 sets, respectively, compared to the best published system.

## introduction in this paper, we present the ustc's submission to the dialectal and low-resource track of iwslt 2023 evaluation campaign @xcite , aiming to translate tunisian arabic speech to english text. modern standard arabic (msa) is the official language of arabic-spoken countries. however, arabic dialects like tunisian and egyptian are prevalent in everyday communication, exhibiting a similar relation between chinese and cantonese. msa benefits from an abundant supply of unlabeled speech and text data, as well as relatively adequate automatic speech recognition (asr) and machine translation (mt) paired data. in contrast, dialectical forms of arabic have much less paired data and more irregularities in both pronunciation and writing @xcite . this paper aims to explore the transfer between high-resource msa and low-resource tunisian dialects, as well as effective training and decoding strategies for speech translation (st) tasks related to low-resource dialects. to facilitate dialect transfer, we introduce two approaches. firstly, we pretrain a model using high-resource msa data, which is then fine-tuned using low-resource tunisian data. this approach involves transferring model parameters and can be used to train various models, e.g., asr, mt, end-to-end st. secondly, we also develop two transformation models for explicit dialect transfer. on one hand, for the augmentation of mt data, we build an mt model that translates msa into tunisian, resulting in a vast amount of pseudo tunisian-english paired data. on the other hand, the tunisian-msa mt encoder module is built and then integrated into the end-to-end st model, which can implicitly normalize dialectal expressions. in addition, we also propose robust training and decoding strategies from two perspectives. to improve the robustness of the mt model against asr errors, we fine-tune the mt model with the asr output from the ctc @xcite layer or the asr decoder. the model ensemble method is exploited to decode multiple models synchronously, which is shown to be rather beneficial for the performance. the rest of this paper is organized as follows. section 2 describes data preparation (e.g., datasets, pre-processing). section 3 presents the methods for training and decoding asr, mt and st models. experimental setup and results are given in section 4. finally, section 5 concludes this work. ## conclusion this paper presents the methods and experimental results of the ustc team for the dialect st (tunisian arabic to english) task in iwslt 2023. the proposed forward and reverse dialect transfer methods, which were shown to be effective for augmenting text data and building hybrid sate models. we utilized various model structures for implementing asr, mt and st tasks, and improved the robustness through model ensembling and error adaptation during training. the experiments showed a significant improvement in dialectal st through the use of dialect transfer method. in unconstrained condition, our e2e st system performs better than the cascaded st system but is slightly less effective in constrained condition. future studies might include the exploration of e2e st models for unified modeling of multiple dialects (e.g., tunisian, egyptian) with msa. to 0.2 while it is 1.2 for the ta2msa text encoder of hybrid sate. note that the ctc loss weight for the ta2msa text encoder is much larger because translating tunisian to msa with pseudo ta-msa mt data is challenging.