Speech-to-Speech Translation with Discrete-Unit-Based Style Transfer (2024)

Abstract: Direct speech-to-speech translation (S2ST) with discrete self-supervised representations has achieved remarkable accuracy, but is unable to preserve the speaker timbre of the source speech. Meanwhile, the scarcity of high-quality speaker-parallel data poses a challenge for learning style transfer during translation. We design an S2ST pipeline with style-transfer capability on the basis of discrete self-supervised speech representations and codec units. The acoustic language model we introduce for style transfer leverages self-supervised in-context learning, acquiring style transfer ability without relying on any speaker-parallel data, thereby overcoming data scarcity. By using extensive training data, our model achieves zero-shot cross-lingual style transfer on previously unseen source languages. Experiments show that our model generates translated speeches with high fidelity and speaker similarity. Audio samples are available at http://stylelm.github.io/ .

## introduction speech-to-speech translation (s2st) aims to translate spoken utterances from one language to another, which can bring immense convenience to international communication. compared to conventional cascaded systems comprising asr, text translation, and tts models @xcite @xcite , direct s2st models without intermediate text generation have a more concise pipeline with less computation cost and error propagation, and also facilitates application to unwritten languages, and thus spark widespread interest in the community. mainstream approaches of direct s2st @xcite @xcite utilize discrete speech representation from selfsupervised models (such as hubert @xcite ) as prediction target, and then use them to reconstruct the waveform. such representation eliminates speaker identity and prosody of the speeches and retains only semantic contents, which simplifies the target distribution and makes the translation less challenging. however, it also has the drawback of losing the style information of the source speech. extra voice conversion systems are needed if users want to keep the source speaker timbre, which may cause degradation in audio quality. some works propose direct s2st with style transfer @xcite . these methods depend on paired data that source and target speech share the same speakers. however, such data from the real world is extremely scarce as it requires a large number of multilingual speakers, while simulated data from multilingual tts systems suffers from less diversity and extra data collection costs. recent large-scale s2st models @xcite have also incorporated the capability of style transfer, yet their sub-modules are highly coupled and are difficult to apply to other s2st models. inspired by recent progress in spoken language models @xcite , we propose a novel approach for direct s2st with the ability of cross-lingual style transfer, and does not rely on any speaker-parallel data. we utilize two types of discrete representations, namely semantic and acoustic units, from a self-supervised speech model and a neural codec, separately. our method encompasses three stages: 1) speech-tosemantic-unit translation, which translates source speech to target semantic units; 2) acoustic unit modeling, which generates target acoustic units from translated semantic units using style information in the source speech; and 3) unit-to-wave generation, which reconstructs high-fidelity translated speech from the acoustic units. the modules of the three stages are trained independently and decoupled from each other, allowing our framework to be applied to various existing speech-to-unit translation models. for the acoustic unit modeling stage, we introduce an acoustic language model. it employs a selfsupervised training approach and learns style transfer through in-context learning, which relies on no speaker-parallel data, and thus addresses the issue of data scarcity. by utilizing extensive training data, our model achieves zero-shot cross-lingual style transfer with source languages not included in the training. experiments show that our model generates results with superior audio quality and style similarity while maintaining accurate content to a good extent. our contributions can be summarized as follows: the overall inference pipeline of our method is illustrated in fig. 1 (a ). our method comprises three consecutive stages, utilizing two distinct types of discrete units: 1) speech-to-semantic-unit translation stage s 1 , which converts source audio into semantic units of the translated speech; 2) acoustic unit modeling stage s 2 , generating target acoustic units conditioned on the semantic output from the preceding stage and the acoustic units of the source speech as style prompt; 3) unit-to-wave generation stage s 3 , producing translated speech that maintains consistent style with the source. we provide details about these two types of units and the three stages in the following subsections. ## conclusions we propose an s2st approach with style transfer capability by adopting an acoustic language model that learns style transfer through in-context learning. by adopting self-supervised training and large-scale training data, our method addresses the scarcity of speaker-parallel data and achieves crosslingual style transfer with unseen source languages. experiments indicate that our approach achieves outstanding results in terms of speech quality and style similarity while keeping good translation accuracy.