Cross-Utterance Conditioned VAE for Non-Autoregressive Text-to-Speech (2022)

Abstract: Modelling prosody variation is critical for synthesizing natural and expressive speech in end-to-end text-to-speech (TTS) systems. In this paper, a cross-utterance conditional VAE (CUC-VAE) is proposed to estimate a posterior probability distribution of the latent prosody features for each phoneme by conditioning on acoustic features, speaker information, and text features obtained from both past and future sentences. At inference time, instead of the standard Gaussian distribution used by VAE, CUC-VAE allows sampling from an utterance-specific prior distribution conditioned on cross-utterance information, which allows the prosody features generated by the TTS system to be related to the context and is more similar to how humans naturally produce prosody. The performance of CUC-VAE is evaluated via a qualitative listening test for naturalness, intelligibility and quantitative measurements, including word error rates and the standard deviation of prosody attributes. Experimental results on LJ-Speech and LibriTTS data show that the proposed CUC-VAE TTS system improves naturalness and prosody diversity with clear margins.

## introduction recently, abundant research have been performed on modelling variations other than the input text in synthesized speech such as background noise, speaker information, and prosody, as those directly influence the naturalness and expressiveness of the generated audio. prosody, as the focus of this paper, collectively refers to the stress, intonation, and rhythm in speech, and has been an increasingly popular research aspect in end-to-end tts systems @xcite @xcite @xcite . some previous work captured prosody features ex-plicitly using either style tokens or variational autoencoders (vaes) @xcite which encapsulate prosody information into latent representations. recent work achieved fine-grained prosody modelling and control by extracting prosody features at phoneme or word-level @xcite . however, the vae-based tts system lacks control over the latent space where the sampling is performed from a standard gaussian prior during inference. therefore, recent research @xcite employed a conditional vae (cvae) @xcite to synthesize speech from a conditional prior. meanwhile, pre-trained language model (lm) such as bidirectional encoder representation for transformers (bert) @xcite has also been applied to tts systems @xcite @xcite @xcite to estimate prosody attributes implicitly from pre-trained text representations within the utterance or the segment. efforts have been devoted to include cross-utterance information in the input features to improve the prosody modelling of auto-regressive tts @xcite . to generate more expressive prosody, while maintaining high fidelity in synthesized speech, a cross-utterance conditional vae (cuc-vae) component is proposed, which is integrated into and jointly optimised with fastspeech 2 @xcite , a commonly used non-autoregressive end-toend tts system. specifically, the cuc-vae tts system consists of cross-utterance embedding (cuembedding) and cross-utterance enhanced cvae (cu-enhanced cvae). the cu-embedding takes bert sentence embeddings from surrounding utterances as inputs and generates phoneme-level cuembedding using a multi-head attention @xcite layer where attention weights are derived from the encoder output of each phoneme as well as the speaker information. the cu-enhanced cvae is proposed to improve prosody variation and to address the inconsistency between the standard gaussian prior, which the vae-based tts system is sampled from, and the true prior of speech. specifically, the cu-enhanced cvae is a fine-grained vae that estimates the posterior of latent prosody features for each phoneme based on acoustic features, cross-utterance embedding, and speaker information. it improves the encoder of standard vae with an utterance-specific prior. to match the inference with training, the utterancespecific prior, jointly optimised with the system, is conditioned on the output of cu-embedding. latent prosody features are sampled from the derived utterance-specific prior instead of a standard gaussian prior during inference. the proposed cuc-vae tts system was evaluated on the lj-speech read english data and the libritts english audiobook data. in addition to the sample naturalness measured via subjective listening tests, the intelligibility is measured using word error rate (wer) from an automatic speech recognition (asr) system, and diversity in prosody was measured by calculating standard deviations of prosody attributes among all generated audio samples of an utterance. experimental results showed that the system with cuc-vae achieved a much better prosody diversity while improving both the naturalness and intelligibility compared to the standard fastspeech 2 baseline and two variants. the rest of this paper is organised as follows. section 2 introduces the background and related work. section 3 illustrates the proposed cuc-vae tts system. experimental setup and results are shown in section 4 and section 5, with conclusions in section 6. ## background non-autoregressive tts. promising progress has taken place in non-autoregressive tts systems to synthesize audio with high efficiency and high fidelity thanks to the advancement in deep learning. a non-autoregressive tts system maps the input text sequence into an acoustic feature or waveform sequence without using the autoregressive decomposition of output probabilities. fast-speech @xcite and paranet @xcite requires distillation from an autoregressive model, while more recent non-autoregressive tts systems, including fastpitch @xcite , aligntts @xcite and fastspeech 2 @xcite , do not rely on any form of knowledge distillation from a pre-trained tts system. in this paper, the proposed cuc-vae tts system is based on fastspeech 2. fastspeech 2 replaces the knowledge distillation for the length regulator in fastspeech with mean-squared error training based on duration labels, which are obtained from frame-to-phoneme alignment to simplify the training process. additionally, fastspeech 2 predicts pitch and energy from the encoder output, which is also supervised with pitch contours and l2-norm of signal amplitudes as labels respectively. the pitch and energy prediction injects additional prosody information, which improves the naturalness and expressiveness in the synthesized speech. ## results this section presents the series of experiments for the proposed cuc-vae tts system. first, ablation studies were performed to progressively show the influence of different parts in the cuc-vae tts system based on mos and wer. next, the reconstruction performance of cuc-vae was evaluated by ffe and mcd. then, the naturalness and prosody diversity using cuc-vae were compared to fastspeech 2 and other vae techniques. at last, a case study illustrated the prosody variations with different cross-utterance information as an example. the audio examples are available on the demo page foot_1 . ## conclusion in this paper, a non-autoregressive cuc-vae tts system was proposed to synthesize speech with better naturalness and more prosody diversity. cuc-vae tts system estimated the posterior distribution of latent prosody features for each phone based on cross-utterance information in addition to the acoustic features and speaker information. the generated audio was sampled from an utterancespecific prior distribution, approximated based on cross-utterance information. experiments were conducted to evaluate the proposed cuc-vae tts system with metrics including mos, preference rate, wer, and the standard deviation of prosody attributes. experiment results showed that the proposed cuc-vae tts system improved both the naturalness and prosody diversity in the generated audio samples, which outperformed the baseline in all metrics with clear margins.