The USTC - NELSLIP Systems for Simultaneous Speech Translation Task at IWSLT 2021 (2021)

Abstract: This paper describes USTC-NELSLIP’s submissions to the IWSLT2021 Simultaneous Speech Translation task. We proposed a novel simultaneous translation model, Cross-Attention Augmented Transducer (CAAT), which extends conventional RNN-T to sequence-to-sequence tasks without monotonic constraints, e.g., simultaneous translation. Experiments on speech-to-text (S2T) and text-to-text (T2T) simultaneous translation tasks shows CAAT achieves better quality-latency trade-offs compared to wait-k , one of the previous state-of-the-art approaches. Based on CAAT architecture and data augmentation, we build S2T and T2T simultaneous translation systems in this evaluation campaign. Compared to last year’s optimal systems, our S2T simultaneous translation system improves by an average of 11.3 BLEU for all latency regimes, and our T2T simultaneous translation system improves by an average of 4.6 BLEU.

## introduction this paper describes the submission to iwslt 2021 simultaneous speech translation task by national engineering laboratory for speech and language information processing (nelslip), university of science and technology of china, china. recent work in text-to-text simultaneous translation tends to fall into two categories, fixed policy and flexible policy, represented by wait-k @xcite and monotonic attention @xcite respectively. the drawback of fixed policy is that it may introduce over latency for some sentences and under latency for others. meanwhile, flexible policy often leads to difficulties in model optimization. inspired by rnn-t @xcite , we aim to optimize the marginal distribution of all expanded paths in simultaneous translation. however, we found it's impossible to calculate the marginal probability based on conventional attention encoder-decoder @xcite architectures (transformer @xcite included), which is due to the deep coupling between source contexts and target history contexts. to solve this problem, we propose a novel architecture, cross attention augmented transducer (caat), and a latency loss function to ensure caat model works with an appropriate latency. in simultaneous translation, policy is integrated into translation model and learned jointly for caat model. in this work, we build simultaneous translation systems for both text-to-text (t2t) and speechto-text s2t) task. we propose a novel architecture, cross attention augmented transducer (caat), which significantly outperforms wait-k @xcite baseline in both text-to-text and speech-to-text simultaneous translation task. besides, we adopt a variety of data augmentation methods, back-translation @xcite , self-training @xcite and speech synthesis with tacotron2 @xcite . combining all of these and models ensembling, we achieved about 11.3 bleu (in s2t task) and 4.6 bleu (in t2t task) gains compared to the best performance last year. ## conclusion in this paper, we propose a novel simultaneous translation architecture, cross attention augmented transducer (caat), which significantly outperforms wait-k in both s2t and t2t simultaneous translation task. based on caat architecture and data augmentation, we build simultaneous translation systems on text-to-text and speech-totext simultaneous translation tasks. we also build a cascaded speech-to-text simultaneous translation system for comparison. both t2t and s2t systems achieve significant improvements over last year's best-performing systems.