Learning Paralinguistic Features from Audiobooks through Style Voice Conversion (2021)

Abstract: Paralinguistics, the non-lexical components of speech, play a crucial role in human-human interaction. Models designed to recognize paralinguistic information, particularly speech emotion and style, are difficult to train because of the limited labeled datasets available. In this work, we present a new framework that enables a neural network to learn to extract paralinguistic attributes from speech using data that are not annotated for emotion. We assess the utility of the learned embeddings on the downstream tasks of emotion recognition and speaking style detection, demonstrating significant improvements over surface acoustic features as well as over embeddings extracted from other unsupervised approaches. Our work enables future systems to leverage the learned embedding extractor as a separate component capable of highlighting the paralinguistic components of speech.

## introduction an effective speech-based ai system is capable of not only recognizing and interpreting the linguistic content of speech but also recognizing and interpreting its paralinguistic attributes. while the linguistic elements of speech encode what was said (i.e., the content), the paralinguistic elements encode how it was said (i.e., emotion, style, etc.) @xcite . the detection and modeling of paralinguistic attributes have many potential applications; ranging from affect-aware human-computer interaction (hci) systems @xcite to the management of mental health @xcite . one major challenge with building robust paralinguistic models is the limited access to largescale, labeled datasets that are needed for training the machine learning models. for instance, a typical emotion dataset (e.g., iemocap) that is used for building paralinguistic models contains around 12 hours of speech while a modern dataset used for building speaker recognition models contains around 2000 hours of speech @xcite . it is therefore critical that features used in paralinguistic tasks distill relevant information from the original signal to allow the recognizers to effectively detect the target attributes. with this in mind, new methods that can leverage unlabeled data for distilling paralinguistic information from speech should be explored. in this work, we introduce the expressive voice conversion autoencoder (evoca), an unsupervised framework that distills paralinguistic attributes from speech without relying on explicit emotion or style labels. evoca is designed to enable a neural network to learn what it means for speech to be expressive by treating expressive speech as a modulation of neutral speech. evoca is trained using parallel speech inputs: one expressive and one neutral. however, these types of parallel paralinguistic corpora are not available at scale. to address this, we use a large audiobook corpus (i.e., 200 hours) composed of expressive speech and artificially generate the parallel neutral, nonexpressive speech using the available transcriptions (see figure 1 ). we train the evoca model to convert between non-expressive synthetic speech and expressive real speech, demonstrating how this conversion yields an embedding that captures paralinguistic attributes (see figure 2 ). the benefit of the evoca framework is that once trained, the component responsible for producing the paralinguistic embeddings can be used as a front-end speech transformer for a variety of downstream applications. we show that these learned paralinguistic embeddings can be used in downstream emotion recognition and speaking style classification tasks. in summary, the key contributions of this work are the following: parallel data audiobooks (speech + text) text from audiobooks speech synthesis model synthetic speech from synthesizer expressive speech from audiobooks from audiobooks without relying on manual annotations for those attributes. to the best of our knowledge, ours is the first work to demonstrate how one can learn paralinguistic features by training a neural model to convert between non-expressive synthetic speech and expressive real speech. ## related work speech emotion recognition applications rely on an extensive set of acoustic features that have evolved over the years @xcite @xcite @xcite . spectral features are a crucial component of any emotion feature set and are included in the widely used compare and egemaps feature sets @xcite . common surface features that are derived from the speech spectrum include melfrequency cepstral coefficients (mfccs) and melfilterbanks (mfbs). in this work, we propose a framework for learning an mfb transformation that highlights the paralinguistic content of an utterance; we demonstrate the effectiveness of the learned transformation over surface mfb features on emotion and speaking style classification tasks. our work also explores the utility of using both synthetic and real speech to learn paralinguistic information. lotfian and busso have previously demonstrated how speech synthesizers can be used to remove emotion from a speech utterance to provide trained emotion recognizers with a neutral reference to aid in the recognition of expressive speech @xcite . one limitation with their approach, however, is that it relied on having access to a real-time speech synthesizer to generate a neutral version of the input utterance for use by the emotion recognizer. in contrast, we use the speech synthesizer only during the data preparation process (figure 1 ) and not during test time. our approach is related to works that focused on unsupervised and self-supervised speech representation learning. chung et al. introduced two auto-regressive methods for learning mfb transformations for speech applications without relying on explicit labels @xcite . both of the proposed models were trained to predict future frames of the input speech sequence in order to learn global structures represented in the speech signal. they showed that the resulting transformation improved performance over surface features on speaker verification and phone recognition tasks. hsu et al. devised a variational autoencoder that is capable of learning hierarchical information present in speech data @xcite . their approach disentangled frame-level features from utterance-level features in order to provide robust embeddings for both speaker recognition and automatic speech recognition tasks. although several unsupervised learning strategies exist for learning speech transformations, ours is the only approach that is targeted at learning transformations that highlight expressive characteristics in speech. recent works in voice conversion have also inspired our proposed approach. the goal of voice conversion is to convert an utterance from one speaker so that it sounds as if it was spoken by another speaker @xcite . in other words, a voice converter retains all linguistic content and only modulates the paralinguistics of speech. previous works demonstrated that voice conversion techniques can be used to convert between emotional states @xcite . in this work we primarily focus on the use of parallel voice conversion methods and future work will explore the trade-offs between parallel and non-parallel approaches. however, to the best of our knowledge, our work is the first to show that the voice conversion task can be adapted and incorporated into a framework that enables a neural network to learn compact embeddings that capture speech expressiveness. ## results in this section, we provide the results of our four experiments (section 5.1). is the proposed framework capable of inserting expressiveness into synthetic speech? ## concluding remarks we proposed evoca, a framework for learning a surface feature transformation that highlights paralinguistic content needed for detecting emotion and speaking style. we first showed that speech synthesizers can be used to strip away paralinguistic attributes from speech while retaining linguistic information. we demonstrated how a neural voice conversion model can be adapted to facilitate the extraction of paralinguistic features by converting synthetic neutral speech to real expressive speech. finally, we showed that these extracted embeddings improve performance over surface features and can outperform other embeddings extracted from existing unsupervised methods on emotion recognition and speaking style detection tasks. future work will consider how the choice of the synthesis model, the number of speakers in the training set, and the architecture used for the encoder affect the quality of the extracted embeddings.