NAIST Simultaneous Speech-to-speech Translation System for IWSLT 2023 (2023)

Abstract: This paper describes NAIST’s submission to the IWSLT 2023 Simultaneous Speech Translation task: English-to-German, Japanese, Chinese speech-to-text translation and English-to-Japanese speech-to-speech translation. Our speech-to-text system uses an end-to-end multilingual speech translation model based on large-scale pre-trained speech and text models. We add Inter-connections into the model to incorporate the outputs from intermediate layers of the pre-trained speech model and augment prefix-to-prefix text data using Bilingual Prefix Alignment to enhance the simultaneity of the offline speech translation model. Our speech-to-speech system employs an incremental text-to-speech module that consists of a Japanese pronunciation estimation model, an acoustic model, and a neural vocoder.

## introduction this paper presents naist's simultaneous speech translation (simulst) systems for the iwslt 2023 english-to-{german, japanese, chinese} speech-to-text track and the english-to-japanese speech-to-speech track @xcite . many previous studies on end-to-end simulst have focused on training methodologies and architectures specialized for the simultaneous scenario. however, such a specialized system setup for simulst is not trivial and increases the difficulty of the system development and the computational complexity. one recent approach to simulst systems is to use an offline speech translation (st) model for prefix-to-prefix translation required in simulst. in last year's iwslt evaluation campaign @xcite , @xcite demonstrated superior results using such multilingual offline st models. in our last year's systems @xcite , we used an offline model fine-tuned for simulst with data augmentation based on bilingual prefix alignment @xcite . in this year, we use an end-to-end multilingual offline st model based on large-scale pretrained speech and text models for the speechto-text track, following @xcite . we used hidden-unit bert (hubert) @xcite as the speech encoder fine-tuned using english automatic speech recognition (asr) data and mbart50 @xcite as the text decoder fine-tuned using a multilingual machine translation data. we prepare the multilingual st model in the following steps: we use a simulst policy called local agreement @xcite ) that finds the longest common prefixes among successive decoding steps. for the english-to-japanese speech-to-speech track, we developed a cascade of the simulst above and an incremental text-to-speech module using a pronunciation estimation model, an acoustic model, and a neural vocoder. figure 1 illustrates an overview of our system architecture. the following subsections explain our methodologies: inter-connection in 2.1 and bilingual prefix alignment in 2.2, the local agreement in 2.3, and the incremental text-to-speech in 2.4. multilingual st (hubert + mbart50) simuls2t model with (2.1) inter-connection and (2.2) prefix alignment ▁私 は ペン を 買っ (2.3) local agreement pronunciation estimation ワ タ シ ハ ペ ン acoustic model + vocoder wait-k read write read write read write look-ahead (2.4) incremental text-to-speech ## conclusions the numbers above the marks indicates chunk size. circled dots indicate our sumitted system.