The HW - TSC ’s Simultaneous Speech Translation System for IWSLT 2022 Evaluation (2022)

Abstract: This paper presents our work in the participation of IWSLT 2022 simultaneous speech translation evaluation. For the track of text-to-text (T2T), we participate in three language pairs and build wait-k based simultaneous MT (SimulMT) model for the task. The model was pretrained on WMT21 news corpora, and was further improved with in-domain fine-tuning and self-training. For the speech-to-text (S2T) track, we designed both cascade and end-to-end form in three language pairs. The cascade system is composed of a chunking-based streaming ASR model and the SimulMT model used in the T2T track. The end-to-end system is a simultaneous speech translation (SimulST) model based on wait-k strategy, which is directly trained on a synthetic corpus produced by translating all texts of ASR corpora into specific target language with an offline MT model. It also contains a heuristic sentence breaking strategy, preventing it from finishing the translation before the the end of the speech. We evaluate our systems on the MUST-C tst-COMMON dataset and show that the end-to-end system is competitive to the cascade one. Meanwhile, we also demonstrate that the SimulMT model can be efficiently optimized by these approaches, resulting in the improvements of 1-2 BLEU points.

## introduction simultaneous speech/text translation (simulst/simulmt) applications are widely demanded in international communication scenarios such as conferences or live streaming. from the perspective of system architecture, recent works on simulst can be classified into cascade and end-to-end forms. cascade systems are often composed of a streaming automatic speech recognition (asr) module and a steaming textto-text machine translation module (mt). it might also contains other correction modules. the integration of these modules can be challenging, but the training of each can be beneficial from sufficient data resources. end-to-end approach is also a choice for simulst, where translations can be directly generated from a unified model with the speech inputs, but bilingual speech translation datasets are still scarce resources. from the perspective of simultaneous strategy, there is a fixed strategy which is represented by wait-k @xcite and a flexible strategy such as monotonic attention @xcite . the fixed strategy is easier to implement but with inferior performance and the flexible one is more robust to the speed of speech but can be non-trivial in the implementation and training. re-translation is also a strategy proposed recently for simulmt system, which benefits from pre-trained mt models but often encounters with flicker @xcite . the iwslt 2022 simulst shared task @xcite aims to provide a platform for participants to evaluate their approaches on both quality and latency. in this year, there are two sub-tracks, i.e. speech-to-text (s2t) and text-to-text (t2t), and three language directions including en-zh, en-de and en-ja in the evaluation. all submitted systems will be evaluated with the simuleval @xcite tool, where bleu @xcite and average lagging (al) @xcite are used as metrics for ranking. meanwhile, systems will be classified into three latency regimes (low, medium, high) with their al, which are determined differently by the language pairs. the simuleval formulates the simultaneous translation as a process where an agent should take "read" or "write" actions to control the progress of translation. a "read" action allows the agent to get the latest source segments from the server. a "write" action enables the agent to make prediction and send generated tokens back to server for scoring. participants are required to implement their approaches under this framework. in this paper, we present our work on the participation of all language directions for both s2t and t2t sub-tasks. for the t2t task, we start by modeling with the original wait-k model and optimizing it with in-domain fine-tuning and self-training @xcite , resulting in large improvements on their performance. we experiment both cascade and end-to-end systems for the s2t task and find that the end-to-end one is quite competitive especially on the latency metric. ## experiments we conduct experiments on three types of systems including t2t, cascade s2t and e2e s2t. all systems are evaluated on the must-c tst-common dataset for all three languages. ## conclusion in this paper, we report our work in the iwslt-2022 simultaneous speech translation evaluation. we explored 4 solutions with a cascade and end-toend system on two sub-tracks and three language directions: 1) we evaluated the method of training a streaming asr model u2 on the large scale mixed training corpora and inference with the domain controlled generation. 2) we explored the optimization of wait-k t2t models with self-training, and obtained positive results. 3) we tried to build a cascade s2t system by integrating the streaming asr model with the wait-k t2t model, and compared it with our end-to-end approach. 4) we trained our end-to-end s2t model with knowledge distillation and found it to be competitive to our cascade approach. in our future works, we will investigate more in terms of simultaneous strategies, efficient using of pretrained models, as well as better training schema with limited st dataset.