The NPU - MSXF Speech-to-Speech Translation System for IWSLT 2023 Speech-to-Speech Translation Task (2023)

Abstract: This paper describes the NPU-MSXF system for the IWSLT 2023 speech-to-speech translation (S2ST) task which aims to translate from English speech of multi-source to Chinese speech. The system is built in a cascaded manner consisting of automatic speech recognition (ASR), machine translation (MT), and text-to-speech (TTS). We make tremendous efforts to handle the challenging multi-source input. Specifically, to improve the robustness to multi-source speech input, we adopt various data augmentation strategies and a ROVER-based score fusion on multiple ASR model outputs. To better handle the noisy ASR transcripts, we introduce a three-stage fine-tuning strategy to improve translation accuracy. Finally, we build a TTS model with high naturalness and sound quality, which leverages a two-stage framework, using network bottleneck features as a robust intermediate representation for speaker timbre and linguistic content disentanglement. Based on the two-stage framework, pre-trained speaker embedding is leveraged as a condition to transfer the speaker timbre in the source English speech to the translated Chinese speech. Experimental results show that our system has high translation accuracy, speech naturalness, sound quality, and speaker similarity. Moreover, it shows good robustness to multi-source data.

## introduction in this paper, we describe npu-msxf team's cascaded speech-to-speech translation (s2st) system submitted to the speech-to-speech (s2s) track 1 of the iwslt 2023 evaluation campaign. the s2s track aims to build an offline system that realizes speech-to-speech translation from english to chinese. particularly, the track allows the use of largescale data, including the data provided in this track as well as all training data from the offline track 2 on speech-to-text translation task. challengingly, the test set contains multi-source speech data, covering a variety of acoustic conditions and speaking styles, designed to examine the robustness of the s2st system. moreover, speaker identities conveyed in the diverse multi-source speech test data are unseen during training, which is called zero-shot s2st and better meets the demands of real-world applications. current mainstream s2st models usually include cascaded and end-to-end systems. cascaded s2st systems, widely used in the speech-to-speech translation task @xcite , usually contain three modules, i.e. automatic speech recognition (asr), machine translation (mt), and textto-speech (tts). meanwhile, end-to-end (e2e) s2st systems @xcite have recently come to the stage by integrating the above modules into a unified model for directly synthesizing target language speech translated from the source language. e2e s2st systems can effectively simplify the overall pipeline and alleviate possible error propagation. cascaded s2st systems may also alleviate the error propagation problem by leveraging the asr outputs for mt model fine-tuning. meanwhile, thanks to the individual training process of sub-modules, cascaded systems can make better use of large-scale text and speech data, which can significantly promote the performance of each module. in this paper, we build a cascaded s2st system aiming at english-to-chinese speech translation with preserving the speaker timbre of the source english speech. the proposed system consists of conformer-based @xcite asr models, a pretrain-finetune schema-based mt model @xcite , and a vits-based tts model @xcite . for asr, model fusion and data augmentation strategies are adopted to improve the recognition accuracy and generalization ability of asr with multi-source input. for mt, we use a three-stage fine-tuning process to adapt the translation model to better facilitate the output of asr. meanwhile, back translation and multi-fold verification strategies are adopted. our tts module is composed of a text-to-bn stage and a bn-to-speech stage, where speakerindependent neural bottleneck (bn) features are utilized as an intermediate representation bridging the two stages. specifically, the bn-to-speech module, conditioned on speaker embedding extracted from the source speech, is to synthesize target language speech with preserving the speaker timbre. combined with a pre-trained speaker encoder to provide speaker embeddings, the tts model can be generalized to unseen speakers, who are not involved in the training process. experimental results demonstrate the proposed s2st system achieves good speech intelligibility, naturalness, sound quality, and speaker similarity. ## automatic speech recognition our asr module employs multiple models for score fusion in the inference. moreover, data augmentation is adopted during training to handle noisy multi-source speech. ## machine translation for the mt module, we first use a pre-trained language model as a basis for initialization and then employ various methods to further enhance translation accuracy. ## conclusion this paper describes the npu-msxf speech-tospeech translation system, which we develop for the iwslt 2023 speech-to-speech translation task. our system is built as a cascaded system that includes asr, mt, and tts modules. to ensure good performance with multi-source data, we improved each module using various techniques such as model fusion and data augmentation in the asr, three-stage fine-tuning, back translation, and cross-validation in the mt, and two-stage training, pre-trained speaker embedding, and audio superresolution in the tts. through extensive experiments, we demonstrate that our system achieves high translation accuracy, naturalness, sound quality, and speaker similarity with multi-source input.