On Generative Spoken Language Modeling from Raw Audio (2021)

Abstract: We introduce Generative Spoken Language Modeling, the task of learning the acoustic and linguistic characteristics of a language from raw audio (no text, no labels), and a set of metrics to automatically evaluate the learned representations at acoustic and linguistic levels for both encoding and generation. We set up baseline systems consisting of a discrete speech encoder (returning pseudo-text units), a generative language model (trained on pseudo- text), and a speech decoder (generating a waveform from pseudo-text) all trained without supervision and validate the proposed metrics with human evaluation. Across 3 speech encoders (CPC, wav2vec 2.0, HuBERT), we find that the number of discrete units (50, 100, or 200) matters in a task-dependent and encoder- dependent way, and that some combinations approach text-based systems.1

## introduction an open question for ai research is creating systems that learn from natural interactions as infants learn their first language(s): from raw uncurated data, and without access to text or expert labels @xcite . natural language processing (nlp) systems are currently far from this requirement. even though great progress has been made in reducing or eliminating the need for expert labels through self-supervised training objectives @xcite @xcite @xcite @xcite , the basic units on which these systems are trained are still textual. young children learn to speak several years before they can read and write, providing a proof of principle that language can be learned without any text. being able to achieve 'textless nlp' would be good news for the majority of the world's languages, which do not have large textual resources or even a widely used standardized orthography (swiss german, dialectal arabic, igbo, etc.), and which, despite being used by millions of users, have little chance of being served by current text-based technology. it would also be good for 'high-resource' languages, where the oral and written forms often mismatch in terms of lexicon and syntax, and where some linguistically relevant signals carried by prosody and intonation are basically absent from text. while text is still the dominant form of language present on the web, a growing amount of audio resources like podcasts, local radios, social audio apps, online video games provide the necessary input data to push nlp to an audio-based future and thereby expand the inclusiveness and expressivity of ai systems. is it possible to build an entire dialogue system from audio inputs only? this is a difficult challenge, but breakthroughs in unsupervised representation learning may address part of it. unsupervised learning techniques applied to speech were shown to learn continuous or discrete representations that capture speaker invariant phonetic content @xcite , despite themselves not being phonemic @xcite . recent developments in self-supervised learning have shown impressive results as a pretraining technique @xcite @xcite , to the extent that automatic speech recognition (asr) on par with the state of the art from two years back can be built with 5000 times less labelled speech @xcite , or even no with no labelled speech at all @xcite . of course, asr still assumes access to text to learn a language model (lm) and the mapping to the audio units. here, we study the case where the lm is directly trained from the audio units without any recourse to text. the high level idea (see figure 1 ) is that automatically discovered discrete units can be used to encode speech into "pseudo-text" (speech-to-unit, s2u), which is used in turn to train a generative language model (unit-based language model, ulm) and to train a speech synthesizer @xcite . this enables learning an lm from scratch without text, and use it to generate speech conditionally or unconditionally, essentially replicating what toddlers achieve before learning to read. early studies using discrete codes learned from an autoencoder show the feasibility of such an approach, but remain at a level of a demo @xcite . in this paper, we address one major conceptual stumbling block which has, thus far, prevented such early studies from having the transformative impact they could have in language technology: model evaluation. we contend that it will be impossible to make progress in this area beyond demos unless proper evaluation methods enabling system comparison are etablished. evaluation for speech generation is difficult due to the continuous, variable and multi-level nature of the speech waveform, and the necessity both to capture fine grained acoustic details to generate intelligible audio and to abstract away from them to learn higher level language concepts. text-based models do not have this problem, since the input is already expressed in terms of mid-level discrete units (characters or words), and are typically evaluated with unsupervised metrics close to the learning objectives like perplexity or log likelihood. here, such an approach is not directly applicable even if we rely on discrete pseudo-text units, since such metrics would depend in an unknown fashion on their granularity (number, duration, and distribution), making the comparison of models that use different units infeasible. conceptually, generative spoken language models can be evaluated at two levels, the acoustic and the language levels, and through two modes of operation, encoding and generation, resulting in 2×2 tasks (see the major contributions of this paper are as follows: (1) we introduce two novel evaluation metrics for the generation mode of spoken language modeling at the acoustic and language levels respectively. our key insight is to use a generic pretrained asr system to establish model-independent assessments of the intelligibility (acoustic level) and meaningfulness (language level) of the produced outputs. the asr system converts the generated waveform back to text, enabling us to adapt standard text-based metrics for these two levels. (2) we validate these metrics through comparison with human evaluation. we show a high degree of concordance between human and machine evaluations of intelligibility and meaningfulness of generated audio. (3) we show that these metrics can be predicted by simpler ones geared to evaluate the encoding mode of the spoken lm. zero-shot metrics borrowed from previous studies in the zero resource speech challenges @xcite correlate well with their generative counterpart, offering an easier proxy to rapidly iterate on model selection. (4) we systematically study the effect of the type of encoding units by factorially crossing three recent speech-to-unit encoders-cpc, wave2vec 2.0, and hubert-with three codebook sizes for the discrete units: 50, 100, 200. we keep constant the rest of the system built from out-of-the-box components (standard transformer for the ulm, tacotron 2 for u2s). we show that both the encoder type and the number of units matter, and that they matter differently depending on the evaluation task. ( 5 ) we open source our evaluation tools and models to help reproducibility and comparability with future work. in section 3, we introduce the asr, zero-shot and human evaluation metrics, in section 4 we present the models, in section 5, we analyze the results and discuss them in section 6. prediction-based approaches which task a model to predict information of unseen speech based on its context, have gained increasing interest recently. examples of information include spectrograms @xcite @xcite @xcite @xcite , cluster indices @xcite , derived signal processing features @xcite , and binary labels of whether a candidate is the target unseen spectrogram @xcite @xcite @xcite . ## evaluation methods we present two sets of automatic evaluation metrics; the first one assesses the output of generative speech models (asr metrics, section 3.1); the second one, the encoded representations (zero-shot probe metrics, section 3.2). finally, we present the human evaluations (section 3.3). ## proposed systems here, we present our s2u (section 4.1), ulm (section 4.2), and u2s (section 4.3) components. ## results in figure 3 , we report the overall results of our models and our logmel baseline as a function of the number of quantized units on our main automated and human metrics. more detailed results follow in the following sections, including two character-based toplines: one uses the oracle transcripts for training the lm, the other uses transcripts produced by the pre-trained asr model. ## discussion and conclusion we introduced generative spoken language modeling as a new unsupervised task bridging the gap between speech and natural language processing and related it conceptually to previously studied unsupervised tasks: acoustic unit discovery, spoken language modeling, discrete speech resynthesis, and text generation. we introduced a suite of metrics, baselines, and first results on lib-rilight that sets the playing field for future work. for comparability, we open source our evaluation stack and the best of our baseline models. our main contributions are as follows. (1) we established a set of easy to use automatic asrbased metrics for model comparison at two critical levels for this task: intelligibility of the speech output and meaningfulness in terms of higher linguistic content. we assessed the first through asr-based per and cer metrics; and the second using text-generation-based metrics (auc for ppx/vert). ( 2 ) we found that these two sets of metrics correlated well with human judgment and (3) that they can be approximated with their inference-mode counterparts, which are faster to compute using zero-shot probe tasks. (4) applying these metrics to pipeline models based on current speech representation learning models and outof-the-box lm and tts components, we found that our basic premise is fulfilled: it is possible to train a language model from quantized units de rived from audio and using it to generate new speech. the generated speech is english-sounding, with recognizable phonemes and words and locally acceptable syntax (see transcribed examples in the appendix and audio snippets here: https:// speechbot.github.io/gslm ). our automatic metrics confirm the quality of the representations and outputs at the acoustic/phonetic level, but show that improvements are needed at the language level. it is to be expected that performance will increase with larger training sets beyond our 6k hours, as has been noted in the case of text. (5) we also uncovered specific issues regarding the number of quantized units. for speech resynthesis, the optimum number of units was always 200 by a large margin, reflecting the well known bitrate/intelligibility trade-off @xcite . however, for language modeling, this was not necessarily the case, as the more detailed acoustic information may introduce too numerous phonetic details that have no impact at the level of lexical and syntactic representations. (6) finally, we found that the choice of units also affected the temperature parameter which is used to control the trade-off between quality and diversity in text-based language model. to address this effect, we proposed a method to normalize the temperature by using an oracle text to build perplexity and diversity anchor points. obviously, this is only a first step towards building textless nlp applications that could be applied to any language, even low resource ones. to reach this long term goal, three important challenges need to be addressed. first, even though we did compare three different encoders and obtained different results, we cannot conclude that one encoder is definitely superior to the others. our point here was merely to use previously published pretrained encoders, and study systematically the effect of number of units on these encoders. a fuller study including a wider set of encoders and a proper hyperparameter search (including the selection of the embedding layer and the clustering algorithm) would be needed in order to determine which of them is most appropriate for speech generation. second, it is to be expected that to further improve generation results, more needs to be done than applying this pipeline to larger training sets. contrary to text, speech unfolds through time and varies continuously in phonetic space. speech also contains multilayered representations (phonetic, prosodic, speaker identity, emotions, background noise, etc.). however, both our tts and our lm were out-of-the-box systems typically used for text applications. more work is needed to adapt these architectures to the richness and variability of the speech signal (see @xcite , for first steps towards integrating prosody into discrete units). the metrics and baselines we introduced here provide landmarks against which we will measure future progress. third, the automatic metrics that we defined here depend on textual resources to build the evaluation asr and lm models, and on linguistic resources to build the zero-shot metrics. how could this ever be applied to low-resource languages? note that the linguistic resources we require are used only for model selection, not model training. our metrics allow for fast iterations in architecture and hyperparameter search, but the overall algorithm is totally unsupervised. therefore, an important next step is to extend this work to other languages, in order to find a common architecture/hyperparameter set that gives good results in held-out languages (high or low resource). the hope is that once good learning models are tuned using a diverse sample of high resource languages, the same models could be deployed in languages where no such resources are available, and work in a purely unsupervised fashion. in figure a1 , we present the pearson correlations between the zero-shot metrics and the human and automatic metrics on downstream tasks. the fact that the abx metric correlates well with these downstream metrics makes it a useful proxy metric for preliminary model and unit size selection, as it is much less costly than generating tts output and running human or asr evaluations.