Unsupervised Multi-scale Expressive Speaking Style Modeling with Hierarchical Context Information for Audiobook Speech Synthesis (2022)

Abstract: Naturalness and expressiveness are crucial for audiobook speech synthesis, but now are limited by the averaged global-scale speaking style representation. In this paper, we propose an unsupervised multi-scale context-sensitive text-to-speech model for audiobooks. A multi-scale hierarchical context encoder is specially designed to predict both global-scale context style embedding and local-scale context style embedding from a wider context of input text in a hierarchical manner. Likewise, a multi-scale reference encoder is introduced to extract reference style embeddings at both global and local scales from the reference speech, which is used to guide the prediction of speaking styles. On top of these, a bi-reference attention mechanism is used to align both local-scale reference style embedding sequence and local-scale context style embedding sequence with corresponding phoneme embedding sequence. Both objective and subjective experiment results on a real-world multi-speaker Mandarin novel audio dataset demonstrate the excellent performance of our proposed method over all baselines in terms of naturalness and expressiveness of the synthesized speech.

## introduction recently, some text-to-speech (tts) models, such as tacotron @xcite , tacotron 2 @xcite , deep voice @xcite , trans-formertts @xcite have been proposed to generate speech autoregressively from text input, and can achieve performance very close to human quality. in order to increase inference speed and generate more robust speech, non-autoregressive tts models such as fastspeech @xcite and fastspeech 2 @xcite are emerged with robust and fast parallel generation. however, limited expressiveness of synthesized audio persists as one of the major gaps between synthesized speech and real human speech, which draws growing attention to expressive speech synthesis studies. synthesizing long-form expressive datasets (such as audiobooks) is still a challenging task, since wide-ranging voice characteristics are collapsed into an averaged prosodic style. to address this issue, style transfer tts has been a popular strategy in recent years @xcite . the global style token (gst) model @xcite adopts multi-head attention mechanism and several learnable style tokens to extract the global style from reference audio in an unsupervised way. further more, a hierarchical gst architecture is proposed to learn hierarchical embedding information implicitly, which contains several gst layers with residual connection @xcite . to not only learn to represent a wide range of speaking styles, but also synthesize expressive speech without the need of auxiliary inputs at inference time, some methods attempt to predict speaking style directly from text, which is more practical and flexible. the text-predicted global style token (tp-gst) extends the gst by predicting style embedding or style token weights from text only @xcite . considering that style and semantic information of sentences are closely related and bidirectional encoder representations from transformers (bert) @xcite shows its effectiveness in language representation, pre-trained bert embedding is adopted as an additional contextual information input to tacotron 2 model to improve the pronunciation and expressiveness of the generated speech @xcite ) @xcite . in addition, different ways of incorporating linguistic features and bert-based features are investigated and compared in various application domains (news, chat and audiobooks). results show that character embedding is the most effective one @xcite . some works also attempt to predict fine-grained speaking styles from text, such as word level @xcite and phoneme level @xcite . all aforementioned methods only take the single sentence to be synthesized into consideration. some studies demonstrate that considering a wider range of contextual information contributes to expressive speech synthesis @xcite . recently, a hierarchical context encoder that considers adjacent contexts within a fixed-size sliding window is used to predict sentence-level style representation directly from text @xcite . although the overall performance is improved, it is still an averaged result that lacks some local fine-grained expressiveness information and rhythmic fluctuations such as pauses and emphasis. the expressiveness of human speech can be perceived as a compound of multi-scale acoustic factors. one is the global-scale speech style, which includes but not limited to timbre and emotion of the speaker. styles at this level are supposed to be consistent throughout the entire utterance. the other is the local-scale speech style, which consists of speed, energy, pitch, pause and other acoustic features @xcite . therefore, it is insufficient to model speech style from a single aspect. some latest studies are observed to devote efforts to performing multi-scale modeling on some specific tasks, in particular emotional speech synthesis @xcite ) @xcite . this further demonstrates the importance and necessity of multi-scale modeling. with all listed imperfections taken into consideration, this paper proposes an unsupervised multiscale expressive speech synthesis model for audiobooks with hierarchical context information as input. a multi-scale hierarchical context encoder is designed to predict both global-scale context style embedding (gcse) and local-scale context style embedding (lcse) from the context in a hierarchical structure. meanwhile, a multi-scale reference encoder is introduced to extract both global-scale reference style embedding (grse) and local-scale reference style embedding (lrse) from the reference speech, which are used to guide the prediction of speaking styles. on top of these, a bireference attention mechanism is adopted to align both the quasi-phoneme-level lrse sequence and the character-level lcse sequence with the corresponding phoneme embedding (pe) sequence. both objective and subjective experiments on a real-world multi-speaker mandarin novel audio dataset demonstrate the excellent performance of our proposed model over all baseline approaches in terms of naturalness and expressiveness of the synthesized speech. ablation studies are further conducted to investigate the influences of several main modules in our proposed model. ## method the architecture of our proposed model is illustrated in figure 1 . it consists of three major parts: a multi-scale reference encoder, a multi-scale hierarchical context encoder and a sequence-to-sequence expressive tts system based on fastspeech 2 @xcite with extended variance adaptor. the multi-scale reference encoder is used to extract reference style embeddings at both global and local scales (i.e. grse and lrse) from reference speech. while the multi-scale hierarchical context encoder is used to predict the context style embeddings at global and local styles (i.e. gcse and lcse) from hierarchical context. the extended variance adaptor is used to align and fuse the style embeddings at different scales with the phoneme embeddings. ## conclusion in this paper, we propose an unsupervised multiscale context-sensitive text-to-speech model for audiobooks. a multi-scale hierarchical context encoder is designed to predict both the global-scale context style embedding and local-scale context style embedding from hierarchical context with the guidance of a multi-scale reference encoder. both objective and subjective experiment results on a real-world multi-speaker mandarin novel audio dataset demonstrate the excellent performance of our proposed multi-scale context-sensitive model over all baseline approaches in terms of naturalness and expressiveness of the synthesized speech.