The HW - TSC ’s Simultaneous Speech-to-Speech Translation System for IWSLT 2023 Evaluation (2023)

Abstract: In this paper, we present our submission to the IWSLT 2023 Simultaneous Speech-to-Speech Translation competition. Our participation involves three language directions: English-German, English-Chinese, and English-Japanese. Our solution is a cascaded incremental decoding system, consisting of an ASR model, an MT model, and a TTS model. By adopting the strategies used in the Speech-to-Text track, we have managed to generate a more confident target text for each audio segment input, which can guide the next MT incremental decoding process. Additionally, we have integrated the TTS model to seamlessly reproduce audio files from the translation hypothesis. To enhance the effectiveness of our experiment, we have utilized a range of methods to reduce error conditions in the TTS input text and improve the smoothness of the TTS output audio.

## introduction this paper describes the hw-tsc's submission to the simultaneous speech-to-speech translation (simuls2s) task at iwslt 2023 @xcite . simultaneous speech-to-speech translation (simuls2s) is currently being researched using cascade systems. these systems typically involve a streaming automatic speech recognition (asr) module, a streaming text-to-text machine translation (mt) module, and an offline text-to-speech(tts) module, with the option of incorporating additional correction modules. although integrating these modules can be complex, training each module with sufficient data resources can prove to be worthwhile. our study adopts a comprehensive approach that utilizes several key components to build a strong system. we incorporate a formidable offline asr model, a robust offline mt model, and a pre-trained tts model as the foundation for our system. moreover, we introduce a refined onlinization technique based on the approach developed by @xcite , which seamlessly integrates into the cascade system. offline tts models often produce a blank sound at the end of a sentence. as a result, when generating audio results in the simultaneous interpreting mode, it can lead to blank tones between clips, causing the final audio to lack smoothness. to address this issue, we have developed several strategies aimed at mitigating this problem in our work. ## conclusion this paper details our involvement in the iwslt 2023 simultaneous speech-to-speech translation evaluation. our team presents an onlinization strategy that can be utilized by cascaded systems, which we have proven to be effective in three different language directions. additionally, we introduce two strategies that address the disparity between