HW - TSC ’s Simultaneous Speech Translation System for IWSLT 2024 (2024)

Abstract: This paper outlines our submission for the IWSLT 2024 Simultaneous Speech-to-Text (SimulS2T) and Speech-to-Speech (SimulS2S) Translation competition. We have engaged in all four language directions and both the SimulS2T and SimulS2S tracks: English-German (EN-DE), English-Chinese (EN-ZH), English-Japanese (EN-JA), and Czech-English (CS-EN). For the S2T track, we have built upon our previous year’s system and further honed the cascade system composed of ASR model and MT model. Concurrently, we have introduced an end-to-end system specifically for the CS-EN direction. This end-to-end (E2E) system primarily employs the pre-trained seamlessM4T model. In relation to the SimulS2S track, we have integrated a novel TTS model into our SimulS2T system. The final submission for the S2T directions of EN-DE, EN-ZH, and EN-JA has been refined over our championship system from last year. Building upon this foundation, the incorporation of the new TTS into our SimulS2S system has resulted in the ASR-BLEU surpassing last year’s best score.

## introduction this paper delineates the hw-tsc's contributions to the simuls2t and simuls2s translation task at iwslt 2024. presently, research on simuls2t translation from a systems architecture standpoint can be segregated into two categories: cascade and end-to-end. cascade systems traditionally encompass a streaming automatic speech recognition (asr) module and a streaming text-to-text machine translation (mt) module, with an additional option of integrating correction modules. despite the complexity of module integration, training each unit with ample data resources can yield significant results. on the other hand, an end-to-end approach for simuls2t is feasible, where translations are directly procured from a unified model with speech input. it's worth mentioning, however, that bilingual speech translation datasets, indispensable for end-to-end models, remain scant. present efforts in simultaneous simuls2t focus on the development of dedicated models customised for this task. this approach, nonetheless, comes with certain limitations, such as the need for an extra model, often accompanying a more complex training and inference process, augmented computational demands, and potential performance decrement when employed in an offline environment. our methodology for simuls2t encompasses the use of a reliable offline asr model and a robust offline mt model as the system's bedrock. we have adapted the onlinization approach of @xcite and introduced an improved technique suitable for integration into the cascade system. on the official development set, our simuls2t achieved a comparable level to the offline models under stringent latency constraints without any alterations to the original models. the disparity between offline and cascade has been further reduced compared to our last year's system. for the new cs-en language pair, we submitted the end-to-end (e2e) system. we anticipate that future research will further enhance the e2e system's performance. lastly, for simuls2s, our system from the previous year had a low-performing tts model. hence, we updated the simuls2s tts model and integrated it with our latest simuls2t system. our achievements is as follows: ## models all models used by our system are offline models and do not use special streaming strategies. the following is an introduction to each model. ## conclusion in summary, this paper presents our efforts for the iwslt 2024 simultaneous speech-to-text and speech-to-speech translation competition. we improved upon our previous system, achieved better translation accuracy and successfully integrated a novel text-to-speech model. our system uses reliable offline models, and we managed to enhance the simulated conversation translation system's quality. our experiments demonstrated that our system performs well across different language pairs. future work will pay more focus on end-toend systems.