Text-Free Prosody-Aware Generative Spoken Language Modeling (2022)

Abstract: Speech pre-training has primarily demonstrated efficacy on classification tasks, while its capability of generating novel speech, similar to how GPT-2 can generate coherent paragraphs, has barely been explored. Generative Spoken Language Modeling (GSLM) (CITATION) is the only prior work addressing the generative aspect of speech pre-training, which builds a text-free language model using discovered units. Unfortunately, because the units used in GSLM discard most prosodic information, GSLM fails to leverage prosody for better comprehension and does not generate expressive speech. In this work, we present a prosody-aware generative spoken language model (pGSLM). It is composed of a multi-stream transformer language model (MS-TLM) of speech, represented as discovered unit and prosodic feature streams, and an adapted HiFi-GAN model converting MS-TLM outputs to waveforms. Experimental results show that the pGSLM can utilize prosody to improve both prosody and content modeling, and also generate natural, meaningful, and coherent speech given a spoken prompt. Audio samples can be found at https://speechbot.github.io/pgslm . Codes and models are available at https://github.com/pytorch/fairseq/tree/main/examples/textless_nlp/pgslm .

## introduction natural language processing (nlp) has made tremendous progress recently. one of the most significant findings is that language models (lms) are natural unsupervised multitask learners @xcite @xcite -by simply training a big neural network on next word prediction with a large amount of unlabeled text, it learns to comprehend, answer questions, summarize, and even translate @xcite . fine-tuning such pre-trained models further leads to the stateof-the-art performance on numerous benchmark tasks @xcite , beating tailor-made models trained from scratch only on labeled data. given the impressive performance of pre-trained text language models, it is tempting to approach spoken language processing tasks by first transcribing speech into text with an automatic speech recognition (asr) system and then utilizing text-based models for comprehension and generation. however, there are a number of caveats for such a framework. first, the majority of the world's languages are primarily spoken and do not have associated texts in large quantities @xcite . in practice, this limits the reach of nlp techniques to a fraction of the world's languages that have a large presence on the web and for which there exists a widely available high quality asr system. second, despite sharing the same vocabulary and syntactic rules, the spoken form and the written form of the same language still vary significantly in terms of sentence lengths, word distributions, presence of disfluencies and back-channelings, and so on @xcite . this makes language models pretrained on web text not suitable for processing spoken languages. third, text does not reflect the rich set of features conveyed by oral languages. speech carries not only phonetic information, but also nonverbal vocalizations (laughter, voice clicks, filler vocalization, etc), rhythm and intonation (prosody), and emotional markers. all of these features could help, not only with generating more expressive speech @xcite , but also with the semantic analysis of the content of the message @xcite . to combat these deficiencies, more recently there is increasing interest in exploring speech pretraining using large quantities of unlabeled speech data @xcite @xcite @xcite @xcite @xcite . however, most of the studies evaluate their models on discriminative tasks, such as asr and those in the superb benchmark @xcite . to the best of our knowledge, generative spoken language modelling (gslm) @xcite is the only prior work that evaluates prompted speech completion, a generative tasks that is similar to the text completion task in gpt-2 @xcite . to remove the reliance on text, gslm exploits discovered units from self-supervised models to build a unit language model (ulm) and a unitto-spectrogram (u2s) model. speech completion can be achieved by first sampling a unit sequence from the ulm with a unit prompt inferred from a speech prompt, and then synthesizing the sampled sequence into speech with the u2s model. unfortunately, because those discovered units encode mostly phonetic information @xcite , it suffers from the same prosodic information loss issue as text-based lms. therefore, when using that ulm for speech completion, it fails to continue with a coherent tone to the prompt. in this paper, we introduce a prosody-aware generative spoken language model (pgslm) that jointly models phonetic content and prosody, in order to leverage prosody for comprehension, and to generate speech coherent with the prompt, which is a precursor for building speech-based dialogue systems. in keeping with our aim of liberating nlp from its over-reliance on text, we follow gslm and represent the phonetic content with self-supervised units discovered from raw audio. as for prosody, it is represented by the pattern of quantized fundamental frequency (f0) and duration. pgslm is comprised of two separately trained components: an auto-regressive multi-stream transformer language model (ms-tlm) that predicts the next phonetic and prosodic representation given the past ones, and a unit high-fidelity generative adversarial network (hifi-gan) adapted from @xcite that converts the ms-tlm output into a waveform like a vocoder. to evaluate the proposed model, we adopt metrics from @xcite for content evaluation, and devise a series of metrics for prosody evaluation. experimental results demonstrate that 1) joint modeling of prosody improves phonetic content modeling, 2) pgslm can generate speech continuation co-herent with the prompt in term of the content and the prosody, and 3) proper choices of model and prosodic representation is crucial to synthesizing natural, coherent, and expressive speech. ## related work our work is related to utilizing prosody for comprehension and predicting prosody for speech synthesis, which we discuss in the following sections. ## method in this section, we first describe the phonetic and prosodic representations used in pgslm, and then introduce the two components it is comprised of: a multi-stream transformer language model and an adapted unit hifi-gan. ## conclusion and future work in this work, we propose a text-free prosody-aware generative spoken language model, pgslm, which models textual content and prosodic information explicitly and does not use any text supervision by leveraging self-supervised units. through extensive evaluation on a diverse set of metrics, we demonstrated that prosody not only improves content modeling, but also enables better prompted speech generation that is aware of both the content and the prosody from the prompt for the first time in the literature. we conducted a number of ablation studies to validate the effectiveness of model design choices. as for broader impacts, this work serves as the foundation for building better conditional speech generation applications where prosody is essential, such as in the conversational scenarios. in addition, the proposed model could also serve as a pre-trained model for other classification tasks, such as emotion recognition or syntactic parsing from speech, or as a pre-trained model for generative tasks such as text-to-speech synthesis with more expressive and coherent prosody. finally, the proposed prosody metrics (teacherforcing duration and pitch mae, continuation correctness/consistency/expressiveness) may also be used for evaluation of text-to-speech synthesis systems that can produce diverse prosody for a given text input.