The USYD - JD Speech Translation System for IWSLT 2021 (2021)

Abstract: This paper describes the University of Sydney & JD’s joint submission of the IWSLT 2021 low resource speech translation task. We participated in the Swahili->English direction and got the best scareBLEU (25.3) score among all the participants. Our constrained system is based on a pipeline framework, i.e. ASR and NMT. We trained our models with the officially provided ASR and MT datasets. The ASR system is based on the open-sourced tool Kaldi and this work mainly explores how to make the most of the NMT models. To reduce the punctuation errors generated by the ASR model, we employ our previous work SlotRefine to train a punctuation correction model. To achieve better translation performance, we explored the most recent effective strategies, including back translation, knowledge distillation, multi-feature reranking, and transductive finetuning. For model structure, we tried auto-regressive and non-autoregressive models, respectively. In addition, we proposed two novel pre-train approaches, i.e. de-noising training and bidirectional training to fully exploit the data. Extensive experiments show that adding the above techniques consistently improves the BLEU scores, and the final submission system outperforms the baseline (Transformer ensemble model trained with the original parallel data) by approximately 10.8 BLEU score, achieving the SOTA performance.

## introduction recent years have seen a surge of interest in speech translation @xcite ) task, that translates the source-side speech to the target-side text directly. the st task contains two major components, automatic speech recognition (asr, @xcite and machine translation (mt, @xcite . in this year's iwslt low-resource speech translation * work was done when di wu was visiting at jd. task, our usyd-jd translation team participated in the swahili to english track. we break the speech translation task into "@xmath0, and mainly focus on the nmt component. for model frameworks, we tried autoregressive neural machine translation, including transformer-base and -big @xcite , and nonautoregressive translation models @xcite . also, we employ our previous work slotrefine @xcite to tackle the case and punctuation problems after asr. to make the most of the parallel and monolingual data, we proposed two pretrain strategies, i.e. bidirectional pretrain-ing §2.2 and denoising pretraining §2.3, and employed two data augmentation strategies, i.e. bidirectional self-training §2.5 and tagged back translation §2.7. where the data used for tagged back translation are carefully selected with our proposed multi-feature in-domain selection approach in §2.6. for post finetune/ process, we employed transductive fine-tune §2.8 and a simple postprocessing approach §2.10. this paper is structured as follows: section 2 describes the major approaches we used. we present the data descriptions in section 3. the experiments settings and main results are shown in section 4. finnaly, we conclude our work in section 5. ## data preparation for asr task, we downloaded all available swahili speech-to-text data foot_4 , such as openslr foot_5 and iarpa babel foot_6 etc., as training corpus and employ all default settings in kaldi foot_7 to preprocess and train them. to simplify the asr task, we lowercased all swahili sentences and removed punctuation. to rejuvenate these case and punctuation information, we design two pipeline tasks after asr: case correction task and punctuation generation. also, it is worth noting that we design some rules to perform the "voice activity detection" process for the for nmt task, the parallel datasets we utilized are described at to avoid the sub-optimal problem caused by the distribution gap between official validation and training data, we randomly sampled 5k/ 5k sentences from the training set as valid and test sets, respectively. the randomly sampled valid sentences are used to optimize the hype-parameters. mono. corpus for tagged bt #sent. totally collected corpus commoncrawl english 30,513,498 cleaned corpus with criteria in §2.6 in domain english 10,000,000 for nmt task, we tried autoregressive transformer-big (at, §2.1) and non-autoregressive model (nat, §2.4) in preliminary experiments, and found that at performs robust on all settings. thus we employ transformer-big for all mt systems. inspired by @xcite , we empirically adopt large batch strategy @xcite (i.e. 458k tokens/batch) to optimize the performance. the learning rate warms up to 1 × 10 -7 for 10k steps, and then decays for 30k (data volumes range from 2m to 10m) / 50k (data volumes large than 10m) steps with the cosine schedule. for regularization, we tune the dropout rate from [0.1, 0.2, 0.3] based on validation performance, and apply weight decay with 0.01 and label smoothing @xmath1.1. we use adam optimizer @xcite to train models. we evaluate the performance on an ensemble of last 10 checkpoints to avoid stochasticity. for fair comparison, the metric we employed is sacrebleu @xcite enization and truecasing with moses scripts @xcite . in order to limit the size of vocabulary of nmt models, we adopted byte pair encoding (bpe) @xcite with 32k operations. larger beam size may worsen translation quality @xcite , thus we set beam @xmath2.9). all models were trained on 4 16gb nvidia v100 gpus. ## conclusion and future work this paper presents the university of sydney & jd's speech machine translation system for iwslt2021 @xmath3. the whole system is pipelined, containing asr, case correction, punctuation generation and nmt tasks, and we main focused on nmt task. we leveraged multi-dimensional strategies and frameworks to improve the translation qualities, which achieves surprisingly +10.8 bleu scores improvement against baseline and ranks the 1st among all the participants. we find that our proposed bidirectional pretraining ( §2.2) and denoising pretraining ( §2.3) can consistently improves the competitive baselines. also, we employ bidirectional self-training in §2.5 and tagged bt in §2.7 make the most of the existing parallel and monolingual data. in the future, we would like to polish other components in the pipeline to achieve better performance. also, it is worthy to try an end-to-end approach with cross-modal structures to incorporate audio and vision knowledge @xcite . for robust model training and data utilization, we would explore better strategies, e.g. adversarial training @xcite and curriculum learning @xcite .