CLAPS peech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-Training (2023)

Abstract: Improving text representation has attracted much attention to achieve expressive text-to-speech (TTS). However, existing works only implicitly learn the prosody with masked token reconstruction tasks, which leads to low training efficiency and difficulty in prosody modeling. We propose CLAPSpeech, a cross-modal contrastive pre-training framework that learns from the prosody variance of the same text token under different contexts. Specifically, 1) with the design of a text encoder and a prosody encoder, we encourage the model to connect the text context with its corresponding prosody pattern in the joint multi-modal space; 2) we introduce a multi-scale pre-training pipeline to capture prosody patterns in multiple levels. 3) we show how to incorporate CLAPSpeech into existing TTS models for better prosody. Experiments on three datasets not only show that CLAPSpeech could improve the prosody prediction for existing TTS methods, but also demonstrate its generalization ability to adapt to multiple languages and multi-speaker text-to-speech. We also deeply analyze the principle behind the performance of CLAPSpeech. Ablation studies demonstrate the necessity of each component in CLAPSpeech. Source code and audio samples are available at https://clapspeech.github.io .

## introduction with the development of deep learning, the audio quality of modern tts systems has been improved, yet prosody modeling is still a challenging problem. previous works on expressive tts have utilized external variation predictors (prediction-based, pb) @xcite and variational generative models (variation-based, vb) @xcite to inject prosody variance into the tts model. another popular direction is to learn better text representation for prosody prediction @xcite . however, the existing text representation learning methods for tts are either based on the masked language model task @xcite @xcite (i.e., learn a bert-like large language model on a text corpus) or masked acoustic model task @xcite (i.e., reconstruct the masked mel-spectrogram based on the input text), which result in two disadvantages. firstly, they only implicitly learn prosody with reconstruction losses, which distracts the model from improving the prosody modeling. secondly, they do not decouple the pronunciation space and prosody space, which leads to low training efficiency and a waste of model capacity. we perform a case study in section 4.3.1, in which we can see that previous text representation used in tts cannot capture the prosody variance under different text contexts. technically, prosody can be regarded as the pitch and duration variance of the same token under different conditions (such as text contexts and speakers) @xcite . this paper mainly studies the prosody correlated to the text context. for instance, for the same word "higher", saying "higher up" or "slightly higher" can lead to different prosodies. inspired by recent cross-modal contrastive learning works in the text-to-image task @xcite , we propose a contrastive learning method that connects the text context and the high-level prosody pattern in the text-speech joint multi-modal space, namely contrastive language-audio pre-training s ! s " s # ... s @xmath0 ... higher up could see ... rising higher and higher ... of a slightly higher age ... token encoding with context information an intuitive example of pre-training clapspeech can be found in figure 1 . we also observe that the prosody pattern can be expressed at multiple levels. therefore, we propose a multi-scale pre-training framework that learns two clapspeech models to capture the prosody information at the phoneme and word levels, respectively. after the pre-training stage, our clapspeech can be regarded as a plugin text encoder applicable to all tts models to provide fine-grained prosody representation. to prove the effectiveness and generalizability of our approach, we use two large-scale automatic speech recognition (asr) datasets (librispeech @xcite for english and wenet-speech @xcite for chinese) to pretrain the clapspeech model. the pre-trained text encoder of clapspeech is then plugged into prediction/variation-based tts baselines to demonstrate the improvement of clapspeech to the exist-ing expressive tts systems. we then evaluate the performance on three tts datasets, including one single-speaker english dataset, one single-speaker chinese corpus, and one multi-speaker english dataset. experiments on all datasets show that clapspeech improves the prosody of the tts models and outperforms previous representation learning methods. to summarize, clapspeech has three prominent advantages: 1) it can provide better prosody representation than previous representation learning methods with a much smaller model scale, thanks to its contrastive objective that explicitly learns the prosody. 2) the text representation of clapspeech can be conveniently used in existing tts systems, only with a minor modification of the front-end network architecture. 3) we also show its potential applications such as fine-grained prosody transfer in section 4.3.2. ## clapspeech we propose clapspeech, a cross-modal contrastive learning approach to provide better text representation for prosody prediction in tts. as shown in figure 1 , clapspeech comprises a text encoder and a prosody encoder, whose training objective is to connect the text token and the speech segment in the joint prosody space. in this section, we first design the network structure and input features of these two encoders. these elaborate designs enable the text encoder to effectively process the text context and ensure that the prosody encoder focuses on extracting the high-level prosody pattern from the speech segment while eliminating other variables, such as timbre. then we introduce the multi-scale contrastive pre-training framework, which enables clapspeech to capture prosody in both phoneme and word levels. finally, we show how the pre-trained text encoder of clapspeech can be conveniently plugged into modern tts systems to improve prosody prediction. we describe these designs in detail in the following subsections and provide more technical details in appendix a. ## conclusion in this paper, we propose clapspeech, a crossmodal contrastive pre-training framework that provides better text representation with rich prosody information for tts. with the design of a text encoder and a prosody encoder, clapspeech learns to connect the text context with its corresponding prosody pattern in the speech. we also introduced multi-scale pre-training to extract prosody patterns at multiple levels. we have demonstrated the performance and generalization ability of clapspeech on three tts datasets (english, chinese, and multispeaker, respectively). we have also deeply analyzed the principle behind the improvement of clapspeech and performed ablation studies to prove the necessity of each component.