MLLP - VRAIN UPV systems for the IWSLT 2022 Simultaneous Speech Translation and Speech-to-Speech Translation tasks (2022)

Abstract: This work describes the participation of the MLLP-VRAIN research group in the two shared tasks of the IWSLT 2022 conference: Simultaneous Speech Translation and Speech-to-Speech Translation. We present our streaming-ready ASR, MT and TTS systems for Speech Translation and Synthesis from English into German. Our submission combines these systems by means of a cascade approach paying special attention to data preparation and decoding for streaming inference.

## introduction in this paper we describe the participation of the mllp-vrain research group in the shared tasks of the 19th international conference on spoken language translation (iwslt). we participated in two shared tasks: the simultaneous speech translation and the (offline) speech-to-speech translation tasks. the translation pair for both tasks was english to german. our submission follows the cascade approach, with individual asr, mt and tts components. we use common asr and mt models for both tasks, with additional latency restrictions for the simultaneous task. in short, for the simultaneous s2t task our system comprises a one-pass decoder asr system based on the hmm-dnn approach with a chunk-based blstm am combined with a transformer lm, followed by a multi-k transformer-based mt system. regarding the s2s translation task, the aforementioned systems are followed by a non-autoregressive conformer-based text-to-spectogram module, ending with a multi-band univnet neural vocoder to convert from the spectogram to the final audio wave. this paper is structured as follows. section 2 describes our participation in the simultaneous speech translation (st) task: the architecture and design decisions of the asr and mt components in our cascade system, and the evaluation of the individual components as well as the speech translation system as a whole. section 3 describes our participation in the speech-to-speech (s2s) translation task, paying special attention to the speakeradaptive tts system specifically developed for this task. our conclusions for the shared task are drawn in section 4. ## speech-to-speech translation in this section we describe our submission to the speech-to-speech translation track, in which we include a speaker-adaptive tts module to our previously described cascaded speech translation system. thus, we reuse the asr and mt models developed for the simultaneous speech translation task, though imposing a less restrictive pruning setup. this involves, in brief, more look-ahead context and a wider search space for the asr system described in section 2.1, and using the offline mt system instead of the simultaneous multi-k mt system referred to in section 2.2. therefore, the remaining of this section will describe the additional tts module included to carry out the final text-to-audio conversion of the s2s pipeline. ## conclusions the mllp-vrain research group has participated in the simultaneous speech translation and speech-to-speech translation tasks using our stateof-the art streaming-ready cascade systems. under the cascade approach, each individual component has been described and evaluated, as well as the joint cascade system. the results show that the cascade approach remains a flexible and powerful solution for st tasks, yet at the same time there is a great deal of hyperparameter optimization that needs to be carried out in order to properly integrate the different compo-