Modelling prosody variation is critical for synthesizing natural and expressive speech in endto-end text-to-speech ( TTS ) systems . In this paper , a cross-utterance conditional VAE ( CUC-VAE ) is proposed to estimate a posterior probability distribution of the latent prosody features for each phoneme by conditioning on acoustic features , speaker information , and text features obtained from both past and future sentences . At inference time , instead of the standard Gaussian distribution used by VAE , CUC-VAE allows sampling from an utterance-specific prior distribution conditioned on cross-utterance information , which allows the prosody features generated by the TTS system to be related to the context and is more similar to how humans naturally produce prosody . The performance of CUC-VAE is evaluated via a qualitative listening test for naturalness , intelligibility and quantitative measurements , including word error rates and the standard deviation of prosody attributes . Experimental results on LJ-Speech and LibriTTS data show that the proposed CUC-VAE TTS system improves naturalness and prosody diversity with clear margins . Recently , abundant research have been performed on modelling variations other than the input text in synthesized speech such as background noise , speaker information , and prosody , as those directly influence the naturalness and expressiveness of the generated audio . Prosody , as the focus of this paper , collectively refers to the stress , intonation , and rhythm in speech , and has been an increasingly popular research aspect in end-to-end TTS systems ( van den Oord et al . , 2016 ; Wang et al . , 2017 ; Stanton et al . , 2018 ; Elias et al . , 2021 ; Chen et al . , 2021 ) . Some previous work captured prosody features ex-plicitly using either style tokens or variational autoencoders ( VAEs ) ( Kingma and Welling , 2014 ; Hsu et al . , 2019a ) which encapsulate prosody information into latent representations . Recent work achieved fine-grained prosody modelling and control by extracting prosody features at phoneme or word-level ( Lee and Kim , 2019 ; Sun et al . , 2020a , b ) . However , the VAE-based TTS system lacks control over the latent space where the sampling is performed from a standard Gaussian prior during inference . Therefore , recent research ( Dahmani et al . , 2019 ; Karanasou et al . , 2021 ) employed a conditional VAE ( CVAE ) ( Sohn et al . , 2015 ) to synthesize speech from a conditional prior . Meanwhile , pre-trained language model ( LM ) such as bidirectional encoder representation for Transformers ( BERT ) ( Devlin et al . , 2019 ) has also been applied to TTS systems ( Hayashi et al . , 2019 ; Kenter et al . , 2020 ; Jia et al . , 2021 ; Futamata et al . , 2021 ; Cong et al . , 2021 ) to estimate prosody attributes implicitly from pre-trained text representations within the utterance or the segment . Efforts have been devoted to include cross-utterance information in the input features to improve the prosody modelling of auto-regressive TTS ( Xu et al . , 2021 ) . To generate more expressive prosody , while maintaining high fidelity in synthesized speech , a cross-utterance conditional VAE ( CUC-VAE ) component is proposed , which is integrated into and jointly optimised with FastSpeech 2 ( Ren et al . , 2021 ) , a commonly used non-autoregressive end-toend TTS system . Specifically , the CUC-VAE TTS system consists of cross-utterance embedding ( CUembedding ) and cross-utterance enhanced CVAE ( CU-enhanced CVAE ) . The CU-embedding takes BERT sentence embeddings from surrounding utterances as inputs and generates phoneme-level CUembedding using a multi-head attention ( Vaswani et al . , 2017 ) layer where attention weights are derived from the encoder output of each phoneme as well as the speaker information . The CU-enhanced CVAE is proposed to improve prosody variation and to address the inconsistency between the standard Gaussian prior , which the VAE-based TTS system is sampled from , and the true prior of speech . Specifically , the CU-enhanced CVAE is a fine-grained VAE that estimates the posterior of latent prosody features for each phoneme based on acoustic features , cross-utterance embedding , and speaker information . It improves the encoder of standard VAE with an utterance-specific prior . To match the inference with training , the utterancespecific prior , jointly optimised with the system , is conditioned on the output of CU-embedding . Latent prosody features are sampled from the derived utterance-specific prior instead of a standard Gaussian prior during inference . The proposed CUC-VAE TTS system was evaluated on the LJ-Speech read English data and the LibriTTS English audiobook data . In addition to the sample naturalness measured via subjective listening tests , the intelligibility is measured using word error rate ( WER ) from an automatic speech recognition ( ASR ) system , and diversity in prosody was measured by calculating standard deviations of prosody attributes among all generated audio samples of an utterance . Experimental results showed that the system with CUC-VAE achieved a much better prosody diversity while improving both the naturalness and intelligibility compared to the standard FastSpeech 2 baseline and two variants . The rest of this paper is organised as follows . Section 2 introduces the background and related work . Section 3 illustrates the proposed CUC-VAE TTS system . Experimental setup and results are shown in Section 4 and Section 5 , with conclusions in Section 6 . In this paper , a non-autoregressive CUC-VAE TTS system was proposed to synthesize speech with better naturalness and more prosody diversity . CUC-VAE TTS system estimated the posterior distribution of latent prosody features for each phone based on cross-utterance information in addition to the acoustic features and speaker information . The generated audio was sampled from an utterancespecific prior distribution , approximated based on cross-utterance information . Experiments were conducted to evaluate the proposed CUC-VAE TTS system with metrics including MOS , preference rate , WER , and the standard deviation of prosody attributes . Experiment results showed that the proposed CUC-VAE TTS system improved both the naturalness and prosody diversity in the generated audio samples , which outperformed the baseline in all metrics with clear margins .

Challenge: Variational autoencoder-based text-to-speech systems encapsulate prosody information into latent representations lacking control over prosody at inference time that contribute to naturalness and expressiveness.
Approach: They propose to estimate a posterior probability distribution of latent prosody features conditioned on acoustic features, speaker information, and text features to generate expressive prosody.
Outcome: The proposed system improves naturalness and prosody diversity over the FastSpeech baseline and two variants on a qualitative listening test with LJ-Speech and LibriTTS data.