The HW - TSC ’s Speech-to-Speech Translation System for IWSLT 2023 (2023)

Abstract: This paper describes our work on the IWSLT2023 Speech-to-Speech task. Our proposed cascaded system consists of an ensemble of Conformer and S2T-Transformer-based ASR models, a Transformer-based MT model, and a Diffusion-based TTS model. Our primary focus in this competition was to investigate the modeling ability of the Diffusion model for TTS tasks in high-resource scenarios and the role of TTS in the overall S2S task. To this end, we proposed DTS, an end-to-end diffusion-based TTS model that takes raw text as input and generates waveform by iteratively denoising on pure Gaussian noise. Compared to previous TTS models, the speech generated by DTS is more natural and performs better in code-switching scenarios. As the training process is end-to-end, it is relatively straightforward. Our experiments demonstrate that DTS outperforms other TTS models on the GigaS2S benchmark, and also brings positive gains for the entire S2S system.

## introduction compared to previous iterations of the iwslt-s2s task @xcite , this year's task @xcite is distinct, particularly in terms of data. the official training dataset provided is gigas2s @xcite , which is substantially larger than previous s2s datasets, with a data size of 10,000 hours. although the target text and speech are generated by mt and tts systems, their quality is relatively high, making them suitable for initiating research on end-to-end s2s or tts models in highresource scenarios. our strategy is similar to that of last year @xcite , where we used a cascaded s2s system, but our research focus has shifted. in last year's work, we primarily studied the role of asr and mt in the s2s system and attempted to optimize the context consistency of translation results. in this year's competition, we shifted our research focus to the tts component. therefore, we directly used the asr and mt systems in our offline st track @xcite . additionally, we no longer considered the issue of context consistency during inference. given the unprecedented success of the diffusion model @xcite in image generation over the past few years, we sought to explore its potential in speech synthesis. thus, we proposed an end-to-end diffusion tts (dts) model. unlike previous tts models, such as fastspeech2 @xcite , which use phonemes as input and use a duration predictor to determine the duration and generate melspectrograms, dts uses raw text as input, predicts the total audio length, and generates the waveform by iteratively denoising the output. the structure of this paper is as follows: we first introduce the dataset used in this task, followed by a brief introduction to the asr and mt models used. then, we provide a detailed description of our proposed dts model. finally, we showcase the performance of each model on the gigas2s dataset. ## conclusion in this paper, we present the system we developed for the iwslt2023 speech-to-speech competition. the system includes relatively simple and effective asr and mt modules, as well as a tts module proposed by us based on the diffusion model. in the experiments, we demonstrate that the denoising diffusion process can effectively learn end-to-end tts task, simplifying both training and inference. however, its generation speed is relatively slow. in our future work, we will continue to optimize its quality and generation efficiency, and further explore the application of diffusion in end-to-end s2s tasks.