NAIST Simultaneous Speech Translation System for IWSLT 2024 (2024)

Abstract: This paper describes NAIST’s submission to the simultaneous track of the IWSLT 2024 Evaluation Campaign: English-to-German, Japanese, Chinese speech-to-text translation and English-to-Japanese speech-to-speech translation. We develop a multilingual end-to-end speech-to-text translation model combining two pre-trained language models, HuBERT and mBART. We trained this model with two decoding policies, Local Agreement (LA) and AlignAtt. The submitted models employ the LA policy because it outperformed the AlignAtt policy in previous models. Our speech-to-speech translation method is a cascade of the above speech-to-text model and an incremental text-to-speech (TTS) module that incorporates a phoneme estimation model, a parallel acoustic model, and a parallel WaveGAN vocoder. We improved our incremental TTS by applying the Transformer architecture with the AlignAtt policy for the estimation model. The results show that our upgraded TTS module contributed to improving the system performance.

## introduction this paper presents naist's simultaneous speech translation (simulst) systems for the english-to-{german, japanese, chinese} speech-to-text track and the english-to-japanese speech-to-speech track within the simultaneous track of the iwslt 2024 evaluation campaign. simultaneous translation involves generating translations incrementally based on partial input, and it requires interpreters who can provide accurate and fluent translations while minimizing delay. early simulst systems are based on a cascade of automatic speech recognition (asr) and machine translation modules (e.g., @xcite @xcite @xcite ), but they suffer from error propagation and added latency imposed by the asr module. recently, an end-toend approach has become popular @xcite , and this approach has been demonstrated to achieve a better quality-latency trade-off. conventional end-to-end simulst models have employed training strategies and architectures designed for a simultaneous setting. however, that approach not only requires additional effort in system development but also results in high computational costs. to alleviate such problems, @xcite proposed a single model trained on offline translation data for the simultaneous scenario. applying a simultaneous decoding policy to an offline speech translation (st) model in simulst inference enables the model to generate outputs similar to simultaneous translation. furthermore, a decoding policy determines whether to generate partial output or wait for more input. using an offline st model with a simultaneous decoding policy has become popular because no specific task adaptation is required for a simulst task. among several simultaneous decoding policies @xcite @xcite @xcite , local agreement (la) @xcite is widely used and won the simulst task at the iwslt 2022 evaluation campaign @xcite . the la policy extracts the longest common prefixes from the 𝑛 consecutive chunks as stable hypotheses. however, it requires a long computation time to obtain the longest common prefix. since simultaneous translation requires real-time translation, a policy that runs fast is desirable. @xcite proposed a decoding policy called alig-natt, which takes the alignments of the source and target tokens using cross attention information. under computation-aware settings, @xcite have shown that alignatt can generate translations with lower latency compared to the la policy, and it is capable of reaching a latency of 2 sec or less. for the iwslt 2024 evaluation campaign, we 1 developed two types of speech-to-text translation models with different decoding policies and compared them. one is based on la and the other on alignatt. the la-based model demonstrates better quality than the alignatt-based one within the given latency constraints, while the alignatt policy works better in a low-latency region in computation-aware settings. for the english-to-japanese speech-to-speech track, we developed a cascade of the above simulst model and an incremental text-to-speech module using a phoneme and prosodic symbol estimation model, a parallel acoustic model, and a parallel wavegan vocoder. in last year's submission, our speech-to-speech translation method suffered from the quality of the synthesized speech and possible asr errors @xcite . the authors reported that the character error rate of the naist 2023 speech-to-speech translation output exceeded that of the simulst text output by over 28%. therefore, we upgraded our tts module by incorporating transformer architecture and alignatt in the estimation model. ## system architecture this section describes the architecture of our simulst systems. first, we explain the decoding policies used for our translation modules. then, we present the details of our simultaneous speechto-text and speech-to-speech translation methods. ## conclusions in this paper, we described our simulst systems for the iwslt 2024 simultaneous speech translation task. experimental results demonstrated the effectiveness of alignatt by comparison to local agreement in terms of computation-aware latency, especially in the low-latency range. our speech-to-speech translation system also showed the effectiveness of applying alignatt to the tts model and resulted in better performance compared to our iwslt 2023 system. this time, our speechto-text method used hubert with the mbart model, while our tts method only used the parallel wavegan vocoder. in the future, we will investigate other methods such as wavlm @xcite and hi-fi gan @xcite .