Textless Speech Emotion Conversion using Discrete & Decomposed Representations (2022)

Abstract: Speech emotion conversion is the task of modifying the perceived emotion of a speech utterance while preserving the lexical content and speaker identity. In this study, we cast the problem of emotion conversion as a spoken language translation task. We use a decomposition of the speech signal into discrete learned representations, consisting of phonetic-content units, prosodic features, speaker, and emotion. First, we modify the speech content by translating the phonetic-content units to a target emotion, and then predict the prosodic features based on these units. Finally, the speech waveform is generated by feeding the predicted representations into a neural vocoder. Such a paradigm allows us to go beyond spectral and parametric changes of the signal, and model non-verbal vocalizations, such as laughter insertion, yawning removal, etc. We demonstrate objectively and subjectively that the proposed method is vastly superior to current approaches and even beats text-based systems in terms of perceived emotion and audio quality. We rigorously evaluate all components of such a complex system and conclude with an extensive model analysis and ablation study to better emphasize the architectural choices, strengths and weaknesses of the proposed method. Samples are available under the following link: https://speechbot.github.io/emotion

## introduction generating spoken utterances and dialogue that sound natural is a fundamental requirement to improve human-computer interaction @xcite . one of the main roadblock in improving naturalness in speech generation is the modeling of expressive and emotional states. the difficulty is that emotion is a phenomenon affecting all linguistic levels simultaneously: when one goes from a happy the input signal is first encoded as a discrete sequence of content units (e c ). next, a sequence to sequence (s2s) model is applied to translate between the sequences corresponding to different emotions. then we predict the duration (e dur ) and f0 (e f 0 ) before feeding these signals to a vocoder (g). the last 4 components are conditioned by emotion (z emo ). we use pink to denote models and green to denote representations. to an angry state, one may use different vocabulary, insert non-verbal vocalizations (cries, grunts, etc), modify prosody (intonation and rhythm), and change voice quality due to stress. vice versa each of the levels contributes to the perception of the emotional state of the speaker, where the non verbal aspects can often override the lexical content @xcite . existing emotion generation or emotion conversion techniques have a hard time producing convincing results because they only manage to tackle a subset of these levels. in a nutshell, signalbased approaches are mainly focused on manipulating parameters of the speech signal @xcite @xcite @xcite @xcite , and can only address changes at the level of voice and prosody. in contrast, text based approaches @xcite @xcite @xcite @xcite can generate expressive speech, but struggle with nonverbal vocalizations because they are typically not annotated in speech corpora. in this work we focus on the task of speech emotion conversion under the parallel dataset setting, modifying the perceived emotion of a speech utterance while preserving the speaker identity and the lexical content @xcite . the proposed pipeline is comprised of four main blocks: speech tokenizer, content translation model, prosody prediction model, and a neural vocoder. we start by extracting discrete representation of the speech signal. we translate these representation to a target emotion (e.g., removing laughter, inserting yawning) while preserving the lexical content. then, we predict prosodic features based on the translated representations. a neural vocoder synthesizes the speech waveform from the translated phonetic-content, predicted prosody, speaker label and target emotion label. the overall system is depicted in figure 1 . our contribution is twofold: we propose a novel textless approach by casting the task of speech emotion conversion as a spoken language translation problem. we demonstrate how such a paradigm can be used to model expressive non-verbal communication cues as well as generating high-quality speech samples. finally, we demonstrate for the first time the coverage of all levels of expressive speech modeling simultaneously. results show that the proposed method is vastly superior to emotion conversion techniques based on signal only, and also beats text-based approaches in terms of generation quality and perceived emotion. ## related work speech emotion conversion. there is an increasing number of studies for speech emotion conversion recently @xcite . early studies on speech emotional conversion include gaussian mixture model (gmm) technique @xcite , sparse representation technique @xcite , or an incorporated framework of hidden markov model (hmm), gmm and fundamental frequency segment selection method @xcite . recently, speech emotion conversion has seen a great leap in performance following the advances in deep nets. this include using multi-layer perceptron @xcite , highway networks @xcite , deep bi-directional lstm networks @xcite , and sequence-to-sequence model @xcite . when considering non-parallel data, new techniques have been proposed to learn the translation between emotional domains with cyclegan @xcite and @xcite , to disentangle the emotional elements from speech with auto-encoders @xcite , and to leverage text-tospeech @xcite or automatic speech recognition (asr) @xcite . studies have also revealed that the emotions can be expressed through universal principles that are shared across different individuals and cultures @xcite . this motivates research on multispeaker @xcite , and speaker-independent emotion conversion @xcite . unsupervised speech representation learning aims to distill features useful for downstream tasks, such as phone discrimination @xcite and semantic prediction @xcite , by constructing pretext tasks that can exploit large quantities of unlabeled speech. pretext tasks in the literature can be roughly divided into two categories: reconstruction and prediction. reconstruction is often implemented in the form of auto-encoding (van den @xcite , where speech is first encoded into a low-dimensional space, and then decoded back to speech. various constraints can be imposed on the encoded space, such as temporal smoothness @xcite @xcite , discreteness @xcite , presence of hierarchy @xcite , and information bottlenecks for speech representation decomposition @xcite . prediction-based approaches task a model with predicting information of unseen speech based on its context. examples of information include spectrograms @xcite @xcite , cluster indices @xcite , derived signal processing features @xcite , and binary labels of whether a candidate is the target unseen spectrogram @xcite @xcite . ## model as emotion manifests itself in multiple aspects of spoken language, to optimally convert emotion one needs to consider all aspects in the conversion process. for example, emotion can be expressed via a prosodic features (high pitch, slow speaking rate, etc.), speaking style (yelling, whispering, etc.), and non-verbal vocalizations @xcite . we suggest to use a decomposed representation of the speech signal to synthesize speech in the target emotion. we consider four components in the decomposition: phoneticcontent, prosodic features (i.e., f0 and duration), speaker identity, and emotion-label, denoted by z c , (z dur , z f 0 ), z spk , z emo respectively. specifically, we propose the following cascaded pipeline: (i) extract z c from the raw waveform using a self-supervised learning (ssl) model; (ii) translate non-verbal vocalizations in z c while preserving the lexical content (e.g., when converting from amused to sleepy, we remove laughter and insert yawning); (iii) predict the prosodic features of the target emotion based on the translated content; (iv) synthesize the speech from the translated content, predicted prosody, target speaker identity and target emotion-label. see figure 1 for a visual description of the method. ## experimental setup we use the emotional voices database (emov) for training and evaluating our model. emov consists of 7000 speech utterances based on transcripts from the cmu arctic database @xcite . each transcript was recorded in multiple acted emotions (neutral, amused, angry, sleepy, disgusted) by multiple native speakers (two males speakers and two females speakers). this allows us to create a dataset of utterance pairs for the task of translation. specifically, we create pairs of utterances that are based on the same transcript but are recorded with different acted emotions. due to the small size of this dataset (∼9 hours), we further augment it by creating additional parallel pairs from different speakers. overall, the size of the entire dataset is 78,324 pairs. we split the data to train/validation/test sets with a ratio of 90/5/5 such that there is no overlap of utterances between the sets. in our experiments, splitting randomly (e.g., overlapping transcripts) led to a memorization of the utterances and failed to generalize to unseen data. we explore two data regimes for pre-training stage: (1) large scale pre-training on a mix of librispeech, blizzard2013 @xcite and emov @xcite . denoted by ours + ; (2) small scale setup in which we pre-train on vctk @xcite and emov. we explore the latter for a fair comparison with the evaluated baselines. we denote this setup by ours -. a full description of architecture hyperparameters and pre-training details can be found in the appendix a.1. ## results we first tune independently the different components of our system using objective metrics: the content units extraction configuration, and the prosodic modeling modules (f0 and duration estimators). next we conduct a subjective evaluation for our best system in terms of audio-quality (mos) and perceived emotions (emoc) and compare the proposed method against the baselines (section 4.1). finally, we run an ablation study where we measure the impact of each component on the perceived emotion. 2021 ) showed that the number of units (k) also has an impact on the overall performance of the system. to better understand the effect of these architectural configurations in our setting, we experimented with extracting hubert features from layers 6 and 9, using 100 and 200 clusters for the k-means post-processing step. additionally, we measure the impact that pre-training the translation model has on performance. as we compare models with different vocabulary sizes, we cannot use metrics such as bleu, hence, we report wer and per. for emotion conversion, we find that using the 9 th hubert layer and 200 tokens performs best. an evaluation of different model design architectures can be found in ## conclusion we presented a novel textless method for speech emotion conversion using decomposed and discrete representations. using such representations we suggested casting speech emotion conversion as a spoken language translation problem. we learn a mapping to translate between discrete speech units from one emotion to another. results suggest the proposed method outperforms the evaluated baselines by a great margin. we demonstrated how the proposed system is able to model expressive non-verbal vocalizations as well as generate highquality expressive speech. we conclude with an ablation study and analysis of the different components composing our system. this study serves as the foundation for improving speech emotion conversion and building general textless expressive speech generation models. for future work, we would like to design and evaluate an end-to-end system where we jointly model content and prosody, as well as extending the current work to the non-parallel dataset setting. edit-distance between the target unit sequence and the generated unit sequence. we additionally report the wer and per metrics extracted using a pretrained sota asr system foot_2 . results are reported on emotions converted to neutral: {amused, angry, sleepy, disgusted}-neutral. the bleu and uer metrics are computed using all emotion pairs from neutral and to neutral. it can be seen that the share-enc and share-none architectures are comparable, while the share-all configuration is inferior. although both shareenc and share-none configurations are similar in terms of lexical reconstruction, in our listening tests share-none generated more expressive speech.