The NYA ’s Offline Speech Translation System for IWSLT 2024 (2024)

Abstract: This paper reports the NYA’s submissions to IWSLT 2024 Offline Speech Translation (ST) task on the sub-tasks including English to Chinese, Japanese, and German. In detail, we participate in the unconstrained training track using the cascaded ST structure. For the automatic speech recognition (ASR) model, we use the Whisper large-v3 model. For the neural machine translation (NMT) model, the wider and deeper Transformer is adapted as the backbone model. Furthermore, we use data augmentation technologies to augment training data and data filtering strategies to improve the quality of training data. In addition, we explore many MT technologies such as Back Translation, Forward Translation, R-Drop, and Domain Adaptation.

## introduction the offline speech translation (st) task translates the source audio into target text. currently, there are two leading solutions for st. the first is the traditional cascade system @xcite , which decouples the st task into an automatic speech recognition (asr) and a neural machine translation (nmt) task. in the traditional cascade system, when translating, the source speech is recognized into source text, and then the nmt model is used to translate the source text into target text. however, it often leads to higher architectural complexity and error propagation @xcite , affecting subsequent nmt tasks. in order to alleviate this problem, the end-to-end (e2e) st architecture @xcite is proposed. the e2e st combines asr and nmt modeling to establish the map between the source audio and the target text. for the e2e st architecture, one disadvantage is the lack of parallel training data. for the traditional cascade st system, sufficient training can obtain high-accuracy asr and mt systems due to the large asr and mt datasets. therefore, the traditional cascade st system generally achieves better performance than the e2e st. at the same time, in the recent offline track of iwslt evaluation @xcite @xcite , we can see that the cascade st system is better than the e2e st system. thus, in this work, we use the traditional cascaded st scheme. specifically, in the asr task, we directly adopt the whisper @xcite ) large-v3 model, which can achieve a strong comprehensive asr performance. we also explore sharding strategies, such as supervised hybrid audio segmentation (shas) @xcite , to segment the source audio for better st results. in the mt task, we use the transformer architecture @xcite as the backbone model. to ensure the mt model is fully trained, we meticulously collect a large amount of parallel data and monolingual data from various data sources. furthermore, we delve into many mt technologies such as back translation @xcite , forward translation, r-drop @xcite , domain adaptation, and ensemble @xcite . moreover, we compare the two solutions: one-to-one and one-to-many st, and we find that one-to-many is better. through the above explorations, our model finally achieves good st performance. in detail, experimental results on the tst2022 test set demonstrate that our model achieves 36.37, 20.92, and 24.28 bleu in en2zh, en2ja, and en2de, respectively. the rest of this paper is organized as follows. section 2 describes the datasets and data preprocessing. section 3 describes our speech translation system, which includes asr and mt models. ## experiments and results all models are implemented on fairseq toolkits @xcite and trained on four nvidia a100 gpus. the iwslt test sets of tst2022 are used to evaluate the translation performance at sentence level. the mwersegmenter toolkit foot_2 @xcite is used to resegment and align translation results and then sacrebleu foot_3 @xcite is used to compute bleu scores. for the japanese text, tokenization is performed using the mecab, while for the chinese text, tokenization is executed at character level. we apply shas foot_4 @xcite for audio segmentation and try a variety of combinations for min and max segment length, the optimal parameters is 5-30 secs for ted domain. the for the one-to-many system in finally, we integrate the baseline model, which is enhanced by the r-drop mechanism, with finetuned models that leverage additional data, backward translation, and adaptation techniques. the ensemble of model (2, 4) achieves notable improvements, with bleu scores of 36.33 for en2zh, 20.90 for en2ja, and 24.26 for en2de. furthermore, the ensemble of model (2, 4, 5) slightly surpasses the ensemble of model (2, 4), reaching scores of 36.37 for en2zh, 20.92 for en2ja, and 24.28 for en2de. this indicates the effectiveness of model ensemble in boosting translation quality. ## conclusion this paper describes our submission to the iwslt24 offline speech translation task. we collect a large amount of parallel and monolingual data from the public data sources and adopt the traditional cascade st architecture for the unconstrained training track. for the asr model, we use the excellent whisper large-v3 model, which is trained on 680,000 hours of multilingual and multitask supervision data. it shows strong robustness in various audio scenes. for the mt model, we explore a wider and deeper transformer model using fairseq tookit. to make the model fully trained, we carefully experiment many mt technologies, such as back translation, forward translation, domain adaptation, and r-drop. experimental results on the tst2022 test set show that our model achieves 36.37, 20.92, and 24.28 bleu in en2zh, en2ja, and en2de, respectively.