The N iu T rans’s Submission to the IWSLT 22 E nglish-to- C hinese Offline Speech Translation Task (2022)

Abstract: This paper describes NiuTrans’s submission to the IWSLT22 English-to-Chinese (En-Zh) offline speech translation task. The end-to-end and bilingual system is built by constrained English and Chinese data and translates the English speech to Chinese text without intermediate transcription. Our speech translation models are composed of different pre-trained acoustic models and machine translation models by two kinds of adapters. We compared the effect of the standard speech feature (e.g. log Mel-filterbank) and the pre-training speech feature and try to make them interact. The final submission is an ensemble of three potential speech translation models. Our single best and ensemble model achieves 18.66 BLEU and 19.35 BLEU separately on MuST-C En-Zh tst-COMMON set.

## introduction speech translation is the task that transfers the speech input to the target language text. comparing the cascade of automatic speech recognition (asr) and machine translation (mt) systems, recently the end-to-end speech translation (e2e st, for short st) model arises more attention for its low latency and avoiding error propagation @xcite @xcite . on the iwslt21 offline speech translation task, the st has shown its potential ability compared with cascade systems by using asr and mt labeled data to pre-train modules of the st model @xcite . we explore that using different speech features and model architecture for the st model can further lessen the gap with the cascade system. we design a model which fuses the two speech features to enrich speech information. in our submission, we pre-train the machine translations model and choose the deep transformer @xcite , ode transformer @xcite and mbart @xcite as mt backbone architectures. for the acoustic model, we use a progressive down-sampling method (pds) and wav2vec 2.0 (w2v) @xcite . to integrate the pre-trained acoustic and textual model, we use the sate method @xcite which adds an adapter between the acoustic and textual model. to utilize the model pre-trained by unlabeled data, such as w2v, and mbart, we purpose the multi-stage pre-training method toward st (msp) and add the msp-adapter to boost the st performance. manuscripts for the msp and pds are in preparation. we fuse the output feature of the pds encoder and w2v with the multi-head attention of the decoder. the input of the former is a standard speech feature while the latter is a waveform. we evaluate the relation between the effect of the ensemble model and the diversity of model architecture. our best mt model reaches 19.76 bleu and our st model reaches 18.66 bleu on the must-c en-zh tst-common set. while the ensemble model achieves 19.35 which shows the performance of st can be further improved. the model that fuses two strong encoders does not outperform the model with a single encoder. we show the diversity of models is important during the ensemble stage. we find the bottleneck of our st model is the de-noising and translating ability of mt modules. asr following the previous work @xcite , we unify all the audio to the 16000 per second sample rate and single channel. the common voice corpus consists of many noises, so we choose the cleaner part according to the covost corpus. for the must-c v1 corpus, we remove repetitive items comparing the must-c en-zh transcriptions. we use the librispeech set to build the asr system and then score the common voice, ted lium, and st ted three corpora. the sentence that the wer is higher than 75% will be removed. we filter frames with lengths less than 5 or larger than 3000. we remove the utterances with the size of characters exceeding 400. st since st data is scarce, we only filter the data according to the frame lengths and the standard is the same as asr. we segment the final test speech by the webrtc vad tool foot_0 . we control the size of the speech slices to make sure the length distribution is similar to the training set. ## model we explore the performances of different asr, mt, and adapter architectures. we experiment with three mt models, two asr models and two adapters that integrate the mt and asr to the st model. ## fine-tuning and ensemble to adjust the composed model to the st task and a certain domain, we use the whole st data to finetune the model. after coverage, we continue to train the model with only the must-c data set for domain adaptation. we ensemble st models by averaging distributions of model output. we search different combinations and numbers of models on the must-c set to investigate the influence of structural differences on the results of the ensemble model. since the final segmentation on the test set is inconsistent with the training set, we re-segment the training set by the same hyper-parameters as the test set. to get the reference of the audio, we implement the ensemble model to decode all the training audios and use the wer to re-cut the gold training paragraph into sentences. we utilize the new re-segment set to fine-tune the models. ## conclusions this paper describes our submission to the iwslt22 english to chinese offline speech translation task. our system is end-to-end and constrained. we pre-trained three types of machine translation models and two automatic speech recognition models. we integrate the acoustic and translation model on speech translation tasks by two types of adapters msp and sate. we fine-tune models to adapt domain and search for the best ensemble model for our submission. our final system achieves 19.35 bleu on must-c en-zh tst-common set.