Controlling Prosody in End-to-End TTS : A Case Study on Contrastive Focus Generation (2021)

Abstract: While End-2-End Text-to-Speech (TTS) has made significant progresses over the past few years, these systems still lack intuitive user controls over prosody. For instance, generating speech with fine-grained prosody control (prosodic prominence, contextually appropriate emotions) is still an open challenge. In this paper, we investigate whether we can control prosody directly from the input text, in order to code information related to contrastive focus which emphasizes a specific word that is contrary to the presuppositions of the interlocutor. We build and share a specific dataset for this purpose and show that it allows to train a TTS system were this fine-grained prosodic feature can be correctly conveyed using control tokens. Our evaluation compares synthetic and natural utterances and shows that prosodic patterns of contrastive focus (variations of Fo, Intensity and Duration) can be learnt accurately. Such a milestone is important to allow, for example, smart speakers to be programmatically controlled in terms of output prosody.

## introduction text-to-speech (tts) systems attempt to produce human-like speech by processing natural language text inputs. neural network based tts has made rapid progresses and attracted a strong attention in recent years @xcite @xcite . not only synthetic speech quality but also inference speed were improved, the latter benefiting from non-autoregressive tts models @xcite . controlling prosody is another issue, as features such as pitch and duration are difficult to predict because of their large fluctuations over time. while recent works proposed to synthesize speech which closely resembles the prosody of * this study was conducted during the internship at naver labs europe. a provided reference speech using latent representations @xcite @xcite , this work addresses explicit user control of prosody using the symbolic (text) input. our case study concerns control of contrastive focus which emphasizes on a specific word that is contrary to the presuppositions of the interlocutor (figure 1 ). more precisely, we want the tts system to correctly convey the information initially transmitted by the speaker including prosodic prominence (for instance we want to emphasize the word house in the sentence 'sarah closed the house'). we posit that controlling prosody in tts is important for future prosody transfer in speech-to-speech translation systems (carrying the real meaning of a source utterance spoken by a human). for instance, spoken utterances of figure 1 in english (where focus is only prosodically marked) would lead to different word orderings if translated to a language like hungarian (were focus is explicitely marked by putting the verb right after the focused word). the contributions of this paper are the following: the rest of this article goes simply as following: section 2 presents the background in end-to-end tts and controllable prosody synthesis; section 3 introduces the corpus recorded for prosody-controlled tts. section 4 and 5 present the tts model trained and its evaluation respectively. finally section 6 concludes this work and gives some perspectives. ## conclusion and future work in this paper, we proposed a control mechanism for end-to-end tts systems to manipulate fine-grained prosodic features like contrastive focus (as well as affirmative vs interrogative sentences). this mechanism uses local and global interpretable control tags directly in the input sequences to manipulate the generated prosody: f0, intensity, pauses and durations. a specific mono-speaker corpus was recorded for this study and we hope its release will help the speech community to (a) study fine grain prosodic patterns of focus that occur in natural speech, (b) continue investigating the problem of fine grained control of prosody for tts models and (c) address the dual problem of automatically detecting prosodic focus from natural speech. in future work we plan to extend this study with a stronger tts baseline and human evaluations. we would also like to verify how to train a multispeaker tts system using this same dataset. it would allow to create a generic tts model with, as base, a non annotated dataset and, as extensions, many smaller corpora with specific prosodic annotations. naturally the extension of this work to the complementary task seems also straightforward (i.e. building an asr system with automatic prosodic feature extraction) as well as the concrete application of the technology to speech-to-speech neural machine translation with prosodic features transfer between the input signal and the output translation, as was very recently initiated by @xcite . finally, the possibility to emphasize not only word but also subword units is another interesting perspective.