Empathic Machines: Using Intermediate Features as Levers to Emulate Emotions in Text-To-Speech Systems (2022)

Abstract: We present a method to control the emotional prosody of Text to Speech (TTS) systems by using phoneme-level intermediate features (pitch, energy, and duration) as levers. As a key idea, we propose Differential Scaling (DS) to disentangle features relating to affective prosody from those arising due to acoustics conditions and speaker identity. With thorough experimental studies, we show that the proposed method improves over the prior art in accurately emulating the desired emotions while retaining the naturalness of speech. We extend the traditional evaluation of using individual sentences for a more complete evaluation of HCI systems. We present a novel experimental setup by replacing an actor with a TTS system in offline and live conversations. The emotion to be rendered is either predicted or manually assigned. The results show that the proposed method is strongly preferred over the state-of-the-art TTS system and adds the much-coveted “human touch” in machine dialogue. Audio samples from our experiments and the code are available at: https://emtts.github.io/tts-demo/

## introduction do you love me juliet juliet romeo what do you think i know what you are thinking do you love me what do you think i know what you are thinking juliet juliet romeo accuracy than even multi-sense modes including facial expressions. @xcite shows that any meaningful spoken dialogue cannot happen without some amount of prosodic matching. as humans, we naturally anticipate and adapt with emotional cues in conversing with others, see figure 1 for an example. celebrated trainer sanford meisner employed this to develop meisner technique for theatre actors to react naturally to others in the environment as opposed to method acting. the importance of emotional prosody in conversations cannot be overstated and tts models need to fill this gap to make humanlike conversations possible in hci systems. mitchell and xu (2015) study the value of emotional prosody in hci and emphasize its role in healthcare dialogue systems, improving social interaction skills in people with autism, augmentative and alternative communication devices and gaming narratives. they explain that successfully incorporating expressive speech into hci, involves two aspects: (a) prosodic emotion recognition and (b) expression of emotional prosody. considerable effort has been made towards recognizing and predicting the emotional nuances in human dialogues @xcite @xcite @xcite . however, current tts systems are yet to improve on rendering emotive or expressive speech for real-world hci systems. state-of-the-art tts systems @xcite tend to exhibit average emotions for a given phoneme sequence by taking the mean of utterances from training data. some efforts towards improving expressiveness (like @xcite provide prosody control using a reference clip. others like @xcite and @xcite further focused on controllability exposing levers that can be manipulated at inference-time to derive the intended expression. however, the quality and stability of synthesized speech heavily depends on various modeling choices. emotion or prosody modeling, for example, could pick from numerous available discrete or continuous space representations. the encoder network module chosen might vary in its ability to disentangle prosody from other acoustic features like speaker identity and adaptability to content. for example, those relying on reference clip to replicate prosody might perform poorly when input text is unsuitable for rendering with prosody of reference. some models feed prosody features with phoneme embeddings directly into the decoder while others use them to predict intermediate features that are used in conditioning the decoder. it is empirically verified (like in @xcite that intermediate features could be suitably manipulated to bring about the desired change in expression. we take this direction forward to endow the intermediate feature prediction module with affective state control over the final rendering. we propose differential scaling (ds) of the predicted intermediates to bring about the required change in emotion. the ds module is aimed to effect only emotion as intended while remaining agnostic to all other features like speakers identities or acoustic conditions as seen in train data. we show that this significantly improves the naturalness of the generated speech, while allowing finer control over prosody. in addition to comparing our model's renderings against various others' from literature for naturalness and emotion control on conventional single utterances drawn from disconnected contexts, we also evaluate them in conversations. we curate data with conversational theatre dialogues and replace an actor with a tts system. we use its response as a proxy to evaluate the empathic accuracy. in another experiment, we had a theatre director con-trol the emotion levers of our tts model in a live conversation with the actor to evaluate controllability. as demonstrated in the results, our proposed method significantly improves over existing methods in producing suitable prosodic variation lending closer to human-like conversations. the rest of this paper will elaborate on the following contributions of this work. ## related work prosody and conversational speech. unlike in written text, spoken words contain additional nonverbal information. these cues are collectively termed prosody @xcite that include variations in tone, pitch, energy, duration, accents, intonation, stress, etc. buchholz (2016) showed that prosodic exchange is unavoidable in human dialogue. various machine learning methods have been proposed to predict emotion in speech from its prosody variations @xcite . variations in pitch accents @xcite , for example, lead to a significant difference in how the receiver perceives the content. a sentence (like i said unlock the door, not lock it from @xcite ) could be delivered both as a statement and a command by merely changing prosody. emotion recognition in conversations has gained increasing attention for developing empathetic machines with emotion-tagged multi-modal data publicly available for modeling like @xcite @xcite . while most methods like @xcite use a combination of text and speech information, some leverage additional side-information from broader context @xcite and the topic of conversation @xcite . in such labeled data, emotion is often represented as a categorical variable over a discrete space following models like ekman's basic emotions @xcite or the wheel of plutchik @xcite . this choice is largely owing to the ease of annotating data. @xcite proposed a continuous two-dimensional space as an alternative called valence-arousal model for human emotions. arousal signifies the intensity of the emotion while valence captures its polarity. it has been extended to add a third dimension of dominance, making it the valence-arousal-dominance (vad) model. vad has since been widely used in modelling emotion in music @xcite , speech @xcite and other content @xcite . we use the continuous space representation as it is richer and more convenient to handle in our model. for instance, a continuous space allows the user to change the level of emotion like happy to delighted, sad to depressed, etc., superlatively during synthesis. expressive and controllable tts. neural tts systems are now increasingly popular, improving upon older concatenative statistical systems @xcite in synthesized speech naturalness. these are broadly sequenceto-sequence networks with an encoder processing the input text or phoneme sequence followed by a decoder that generates the sequence of mel frames for output speech. mel frames are then projected into the time domain by a vocoder @xcite to generate the speech. decoding could be autoregressive with tacotron-like models @xcite or nonautoregressive with fastspeech-like models @xcite . non-autoregressive models are faster at inference than autoregressive models with about comparable naturalness of speech quality @xcite . the trick non-autoregressive models use to generate mel frames in parallel is to predict the relevant features as an intermediate step and condition the independent decoding of mels on them. this technique is now increasingly adopted for au-toregressive models as well @xcite to predict features like phoneme duration that improve decoding stability avoiding alignment issues. our method is compatible with any architecture that predicts prosodic features of pitch, energy, and duration as an intermediate step before decoding. going beyond the naturalness of speech, there has been considerable effort to improve the expressiveness of the renderings. some focused on learning a linear space of variations in speech expressions for selecting a suitable variation at inference time. @xcite learn this space unsupervised by encouraging it to explain all variations in training data not captured in content embedding. a reference encoder maps an input utterance to a style embedding as a linear combination of basis style vectors. manual analysis is required to understand the prosody feature learned into a basis vector that could include variations like vocal depth or pitch, speaking rate, or even background noise as available in training data. while this offers style control, it does not explicitly learn the prosody variations of interest into the style space. our work focuses on the same level of control but specifically over the affective state as labeled in some data for supervision. @xcite propose a model similar to @xcite with style tokens restricted to valence and arousal. however, the absolute (pitch, energy, duration) feature predictions restrict prosody control, leading to unnatural distortions. specifically, it skews more towards retaining the speaker's voice identity than the emotion and entangles emotion with other acoustic features. @xcite replace the linear style space with a variational reference encoder to generate prosody embedding to condition the decoder. @xcite use a similar variational model but instead force its posterior to match that of the reference utterance to copy prosody with a controllable parameter determining the closeness of the match. this trick alleviates certain issues like in pitch-range @xcite and transfer to unrelated sentences but exposes a lower degree of control with no explicit levers to operate, as possible in our work. @xcite propose to learn an explicit latent representation for various prosodic variables, segregating them into explicitly controlable (like affect, speaking rate, etc) and implicit (like intonation, rhythm, stress, etc). while the model offers a higher degree of explicit control, it requires using a proprietary studio recorded data with utterances reflecting prompted emotions at specified arousal. dependence on explicit supervision from studio recorded data makes it harder scale this model across languages and other prosodic variations. in contrast, we use publicly available data with emotion labels to train our models. there are other methods that try to predict suitable prosody features from text content. @xcite add a prosody encoder module to standard tts network that predicts certain handcrafted prosody features from text embedding of input. this prosody encoder is used with a small optional bias for affect variations at inference. @xcite extend this to replace hand-crafting prosody features with explicit training followed by their prediction from text. @xcite further enrich the textual context using bert embeddings and parse-trees. these methods are limited in expressiveness offering no control over rendering emotion that our work focuses on. ## model our network uses a backbone tts that can be borrowed from any model which predicts pitch, energy and duration as intermediates features from input phoneme sequence. this network learns to predict the average features for given phonemes. following the convention in earlier works, we refer to the intermediate features as variances and the module that predicts them as variance adaptor. prior work improves standard variance adaptors in, say fastspeech2, by conditioning on emotion variables of valence-arousal in addition to the phoneme sequence to generate expressive speech. we refer to it as emotional variance adaptor (eva) for which we propose an alternative. our proposed differential scaler (ds) module determines how best to vary the output of the eva to bring the desired change in emotion. we describe the details of these network choices in this section; specifically, the broader backbone network architecture and the different variance adaptor modules from non-emotive baseline, emotive baseline and our proposal. ## training modelling with intermediate features facilitates training the backbone and the variance adaptors independently on different data. we exploit this to train our variance adaptor on scarcely available vad annotated data while reusing backbone models trained on abundant transcribed speech data. backbone. we train two backbone networks fastspeech2π (non-autoregressive) and fcl-taco2 (autoregressive) on blizzard 2013 dataset @xcite . it contains 147 hours of catherine bayers's speech, reading books in american english. due to the style of reading, the dataset is rich in expressiveness and spans different combinations of pitch, energy and duration. both models are trained with mel loss (mean absolute error between predicted and ground truth mels), pitch loss, energy loss and duration loss (mean square error between predicted and ground truth features). both models are trained for 200k iterations using adam optimizer with warm-up learning rate scheduler and batch size of 16. eva. we train eva on msp-podcast corpus @xcite annotated with arousal, valance and dominance values. the corpus consists of around 100 hours speech data but their transcriptions are not available. we generate transcripts using a speech-to-text model. we use montreal-forced-aligner (mfa) (mcauliffe and sonderegger, 2017) for phoneme alignments. those transcripts that mfa fails to find a good alignment for are filtered out. the remaining utterances add up to about 71 hours of emotive speech data which we use to train our eva. we train pitch, energy and duration predictors conditioned on vad values minimizing only the sum of variance losses. for all the experiments, text transcripts are converted to phonemes using @xcite . we generate mel spectrogram from the audio files similar to @xcite . pitch and energy are computed from the mel spectrogram and we use mfa for aligning phonemes to train the duration predictor. ## experiments and user study we present three experiments; comparison with prior-art using conventional evaluation metrics, those for emotional consistency with pre-recorded audio, and finally, live conversations with humans. ## conclusion this work presents a novel method that leverages prosodic features (pitch, energy and duration) to modify emotions in the output of a tts system. our method is model agnostic and can be used with any tts backbone that predicts prosodic features in an intermediate step. this method outperforms existing approaches by a significant margin in its ability to accurately render desired emotions, while preserving the naturalness of speech. we curated theatre conversation data to evaluate and show that our prosody-aware tts better maintains the natural flow of emotions in conversations. our work shows promise in consolidation of prosodic emotional recognition and expression, a coveted pursuit in the field of hci. we present further qualitative experiments involving professional theatre artists and demonstrate that the proposed tts method leads to more human-like conversations. while exposing valence, arousal and dominance values as model levers improves control over the final rendering, in reality it is overwhelming for the user to choose them correctly for a desired output. this is further aggravated by the fact that some sentences cannot be suitably spoken with a chosen set of values, degrading output quality. these are limitations that need to be addressed and appropriately deriving these values from semantics of text input or reference clips could be relevant future directions. affective control is incomplete without explicit levers on the intonations, which is another limitation to be looked upon in the future work.