The Xiaomi Text-to-Text Simultaneous Speech Translation System for IWSLT 2022 (2022)

Abstract: This system paper describes the Xiaomi Translation System for the IWSLT 2022 Simultaneous Speech Translation (noted as SST) shared task. We participate in the English-to-Mandarin Chinese Text-to-Text (noted as T2T) track. Our system is built based on the Transformer model with novel techniques borrowed from our recent research work. For the data filtering, language-model-based and rule-based methods are conducted to filter the data to obtain high-quality bilingual parallel corpora. We also strengthen our system with some dominating techniques related to data augmentation, such as knowledge distillation, tagged back-translation, and iterative back-translation. We also incorporate novel training techniques such as R-drop, deep model, and large batch training which have been shown to be beneficial to the naive Transformer model. In the SST scenario, several variations of exttt{wait-k} strategies are explored. Furthermore, in terms of robustness, both data-based and model-based ways are used to reduce the sensitivity of our system to Automatic Speech Recognition (ASR) outputs. We finally design some inference algorithms and use the adaptive-ensemble method based on multiple model variants to further improve the performance of the system. Compared with strong baselines, fusing all techniques can improve our system by 2 extasciitilde3 BLEU scores under different latency regimes.

## introduction in the iwslt 2022 evaluation campaign, our team at xiaomi ai lab participates in one simultaneous speech translation task @xcite , which is the text-to-text track in english to mandarin chinese translation direction. we first introduce the techniques used in our final submitted * equal contribution. † the work was done during the author's internship at xiaomi. data-related techniques are introduced from two perspectives: data augmentation and domainrelated data selection. for data augmentation, we employ technologies such as back-translation (bt) @xcite , knowledge distillation (kd) @xcite , and iterative back-translation @xcite etc. to generate large-scale synthetic bilingual datasets, which have been proved to be very effective in the field of machine translation. we also use the technology of tagged back-translation (taggedbt) @xcite , that is, prepending a reserved @xmath0, so that the model could distinguish the originality of the source sentence. meanwhile, the effects of different combinations of multiple training sets on the model performance are explored. for domain-related data selection, differences in the domains of the training and test sets can have a large negative impact on the results on the test sets. to make the model obtain domain-related knowledge as much as possible, we apply the lm-based data selection technique @xcite to select high-quality and domainrelated data from bilingual corpora. in terms of model, since the submitted systems will be ranked by the translation quality with three latency regimes (low, medium, and high), participants are encouraged to submit multiple systems for each regime to provide more data points for latency-quality tradeoff analyses. besides, we empirically believe that different models have different translation performance and inference latency on t2t tasks, and they can complement each other, so we build various advanced sst models (i.e. basedeep and bigdeep), which are all based on deep transformer model @xcite , but have been empirically proven to outperform the transformer-big model on the sst model. for the t2t track, the output of a streaming asr system (usually prefix of the entire source sentence) will be fed into the sst system as input instead of the gold transcript. so we adopt the wait-k training strategy @xcite to meet the scenario of simulating simultaneous translation. in addition, we also employ the r-drop @xcite and adaptive-ensemble techniques @xcite which have also been proven beneficial for translation models. for inference, we empirically analyze the problems of our system in translation under low latency (e.g. when k is equal to 3) and propose a constrained decoding strategy to wait for some specific words or phrases to appear before translation, which can alleviate some translation issues of the wait-k model in low-latency situations as much as possible. the input fed into the sst model is the output of the asr system, and according to the statistics of previous researchers, the two error types homophones and words with a similar pronunciation account for a large proportion in the output of the asr system. therefore, in order to weaken the model's sensitivity to asr output errors, we introduce methods to enhance the model's robustness to both error types: homophones or words with a similar pronunciation. additionally, a char-to-subwords error correction model is further proposed to correct asr errors before feeding into the translation model. the remainder of this paper is organized as follows. we perform statistics on the data used and introduce pre-processing in section 2. section 3 and 4 elaborate our systems, the techniques employed, and evaluation, followed by the main experimental results and ablation studies reported in section 5. finally, we conclude this paper in section 6. ## data we introduce the data used in our system from the following three aspects: statistics, pre-processing and filtering. statistics. we use the allowed training sets, which include must-c v2.0 foot_0 , covost foot_1 , ted bilingual data size filtered oral must-c v2.0 360k 7.8m covost 870k ted corpus 250k opensubtitles2018 11.2m news wmt2021 61.1m 45.3m total -75.32m 53.1m pre-processing. sacremoses foot_6 is conducted to normalize and tokenize english sentences. we use the traditional and simplified conversion tool to convert traditional chinese text to simplified, use the jieba foot_7 tool to segment chinese sentences, and remove redundant spaces in the text. rule-based filtering. the training set is filtered according to the following rules (the content in parentheses after each item indicates the number of sentence pairs remaining after the current step of filtering is performed): the size statistics of the training set on domains oral and news are shown in ## techniques in this section, we elaborate the models we use and the employed techniques. ## conclusion we elaborate on the xiaomi text-to-text simultaneous speech translation system for the iwslt 2022 in this paper. we first investigate the current mainstream techniques such as deep model and rdrop to construct a relatively strong baseline model, then explore various data augmentation techniques such as taggedbt, kd, and iterative bt to further improve the translation quality of the deep model. then, we adopt the efficient wait-k strategy and the multi-path wait-k strategy to improve the translation quality of the system on the streaming output text which simulates the asr output, and design some rule-based inference algorithms to remedy the obvious translation errors under low latency. in order to alleviate the negative impact of the noise contained in the streaming asr output on our system, we propose two error correction methods to improve the robustness of the model, so that the system has a significant improvement on the noisy inputs. in the future, we will explore the effect of ways to mitigate exposure bias @xcite and pre-trained models, such as bert @xcite and t5 @xcite , on the text-totext simultaneous speech translation task.