E mo K nob: Enhance Voice Cloning with Fine-Grained Emotion Control (2024)

Abstract: While recent advances in Text-to-Speech (TTS) technology produce natural and expressive speech, they lack the option for users to select emotion and control intensity. We propose EmoKnob, a framework that allows fine-grained emotion control in speech synthesis with few-shot demonstrative samples of arbitrary emotion. Our framework leverages the expressive speaker representation space made possible by recent advances in foundation voice cloning models. Based on the few-shot capability of our emotion control framework, we propose two methods to apply emotion control on emotions described by open-ended text, enabling an intuitive interface for controlling a diverse array of nuanced emotions. To facilitate a more systematic emotional speech synthesis field, we introduce a set of evaluation metrics designed to rigorously assess the faithfulness and recognizability of emotion control frameworks. Through objective and subjective evaluations, we show that our emotion control framework effectively embeds emotions into speech and surpasses emotion expressiveness of commercial TTS services.

## introduction the complexity of human communication extends far beyond mere verbal exchange. vocal inflections and emotional undertones play pivotal roles in conveying meaning. while text alone can be ambiguous in meaning @xcite , different emotions in voices can articulate different messages in the same piece of text @xcite . consider shakespeare's iconic phrase, to be or not to be. this line can express despair, contemplation, defiance, or resignation, depending on the speaker's emotional delivery, illustrating the profound impact of vocal emotions in communication. the ultimate objective in the field of conversational systems is to develop intelligent agents capable of comprehending, deciding, and synthesizing speech with nuanced emotional undertones. while recent advances in text-to-speech (tts) technology have achieved remarkable naturalness and expressiveness in synthesized voices(elevenlabs; openai, 2024b; microsoft), these systems lack the capability for users to select and control the emotional tone and intensity. the emotion conveyed in the generated speech is solely determined by the text, without allowing for variability or intensity control. previous works on emotion control in speech synthesis primarily focus on a few simple emotion categories @xcite @xcite . these methods do not allow control of a more diverse array of emotions. synthesis for more complex and heterogeneous emotions like charisma @xcite and empathy @xcite is not well studied. our work leverages recent breakthroughs in foundation models for voice cloning (metavoice; @xcite @xcite . by exploring the rich expressiveness in these models' latent embedding spaces, we develop methods to extract a representation for any emotion with just a few demonstrative samples. these representations are inherently synergistic with the speech generation capabilities of rapidly advancing voice cloning/tts models, enabling us to generate high quality speech while applying fine-grained emotion controls. this approach proves effective for both simple and complex emotions and includes mechanisms to adjust emotional intensity with a scalar knob. our framework's capability of applying finegrained emotion control for any emotion with a few demonstrative examples enables us to propose two methods for applying emotion control based tts models text input "this is great" audio output ! (a) traditional tts pipeline synthesizes speech whose emotion is entirely decided by input text voice clone models "i am happy" audio output voice clone models text input audio output (b) voice clone models replicate speaker but speech emotion is still decided by input text speaker "this is great" previous frameworks our frameworks few-shot emotion control text input "this is great" audio output (c) we enhance voice cloning with fine-grained control of emotion specified by few-shot samples speaker few-shot emotion example " emotion strength # open-ended emotion control text input "this is great" audio output (d) we enable emotion controls based on text descriptions of emotions speaker text description of emotion romantic, full of desire emotion strength 0.4 # 0.4 on arbitrary text descriptions of emotions. we use a synthetic-data-based and a retrieval-based method to leverage recent advances in large language models (llms) and text embedding models (ope-nai; @xcite , in conjunction with our few-shot emotion control framework, to address a lack of open-ended captioned emotional speech dataset. we recognize that emotion control in speech synthesis is still at its early stage, and traditional evaluation metrics for tts systems cannot comprehensively evaluate emotion control frameworks. we therefore introduce a set of rigorous evaluation metrics designed to systematically measure the effectiveness of an emotion control framework at faithfully conveying recognizable emotions. with a set of subjective and objective evaluations, we show that our framework produces faithful and recognizable emotion control on speech. we find that 83% of the participants consider that speech with emotion enhancement by our framework surpasses leading commercial tts services at conveying these emotions. expressive emotion control few-shot emotion control open-ended emotion control synergetic with tts model advances classifier-based style transfer 1 ✓ ✗ ✗ ✗ domain adversarial training 2 ✓ ✓ ✗ ✗ voice text descriptions 3 ✗ ✗ ✓ ✗ ours ✓ ✓ ✓ ✓ ## methods we apply fine-grained emotion controls by manipulating the speaker embedding space of pre-trained foundation voice cloning models. this framework allows us to apply emotion control with few-shot emotional speech samples. the few-shot capability enables us to design two frameworks for applying control with emotion specified by arbitrary text descriptions. ## ablation studies in this section, we conduct ablation studies that vary the shot (sample) number when obtaining the emotion direction vector and the emotion control strength α when applying emotion control. these ablation studies help users decide how to select these hyper-parameters. we report in ## conclusion and future works we proposed emoknob, a framework that enables fine-grained emotion control in voice cloning with few-shot samples. we also propose a syntheticdata-based and a retrieval-based method to embed emotions described by open-ended text into speech synthesis. given novelty of the emotion control domain, we proposed a set of metrics to rigorously evaluate faithfulness and recognizability of emotion control. our method establishes a new way of extracting emotion representation in foundation speech models thus bypassing data limitations. future works can further explore emotion control paradigms such as synthesizing emotions in conversation turns based on these representations.