Generative Spoken Language Model based on continuous word-sized audio tokens (2023)

Abstract: In NLP, text language models based on words or subwords are known to outperform their character-based counterparts. Yet, in the speech community, the standard input of spoken LMs are 20ms or 40ms-long discrete units (shorter than a phoneme). Taking inspiration from word-based LM, we introduce a Generative Spoken Language Model (GSLM) based on word-size continuous-valued audio tokens that can generate diverse and expressive language output. This is obtained by replacing lookup table for lexical types with a Lexical Embedding function, the cross entropy loss by a contrastive loss, and multinomial sampling by k-NN sampling. The resulting model is the first generative language model based on word-size continuous tokens. Its performance is on par with discrete unit GSLMs regarding generation quality as measured by automatic metrics and subjective human judgements. Moreover, it is five times more memory efficient thanks to its large 200ms units. In addition, the embeddings before and after the Lexical Embedder are phonetically and semantically interpretable.

## introduction recent work has opened up the possibility of learning generative language models directly from the raw audio signals, without using either text or automatic speech recognition (asr) @xcite @xcite . the basic idea of these model is to rely on traditional text-based language models (lm), but replace the text input with some other discrete tokens directly learned from audio in an unsupervised fashion. the advantage of learning units from speech instead of relying on asr is that this procedure can capture non-verbal vocalizations (like laughter) or intonation and rhythm, which are typically not transcribed, resulting in more expressive generations @xcite 1 audio examples are available at our website. @xcite . in addition, asr may not be available in many languages that have insufficient textual resources and can make errors, which may then perturb the learning of the lm. the problem of using self-discovered units, however, is that these units are typically very small, in fact, usually smaller than phonemes @xcite . we think that increasing the size of the units will favourably impact the semantic capabilities of a downstream spoken lm. this intuition comes from the nlp literature. among others, @xcite ; @xcite ; @xcite ; @xcite have shown a performance gap between character-based lm and word-based lm. the main reason is that at the level of characters, it is difficult for a text lm to extract long-range syntactic and semantic relationships. this is one of the reasons why recent state-of-the-art text-based lm @xcite typically use a tokenizer representing word or subword units (byte pair encoding @xcite , wordpiece @xcite , unigram @xcite ). another advantage of large units is to save gpu memory at training time that enables to use both larger batches and longer sequences. in speech, building the equivalent of a text-based tokenizer is hampered by two difficulties. first, the boundary problem is that contrary to text in most orthographic systems, speech does not have spaces and punctuation to delimit between word units. finding word boundaries from raw audio is itself a difficult challenge @xcite . second, the clustering problem, is that even if boundaries were available, the clustering of speech fragments is challenging because the same word may surface in a variety of forms depending on speaker, accent, speech rate, etc. this problem may be even more difficult to solve than the first one @xcite because of the highly skewed distribution of word frequencies @xcite . here, we investigate the possibility of building a continuous tokenizer that sidesteps these two problems by using tokens that have neither perfect boundaries nor require a clustering step. in appendix b, we explain in more detail why we wish to avoid the clustering of speech fragments and what methods have been applied to tackle this problem so far. having a continuous tokenizer instead of a discrete one results in drastic changes from the point of view of the downstream lm. with a discrete tokenizer, one can define a finite list of tokens over which the lm can learn a lookup embedding table at the input of the model and use a softmax layer at the output of the model. the softmax is used in training mode to compute the loss function through a cross-entropy with the target token and at inference time to sample sentences. with continuous representations, the list of tokens is unbounded, making these computations intractable. we tackle this problem with a lexical embedder, a semi-learnable function that maps continuous tokens to a practically infinite list of embeddings. the key question addressed in this paper is whether it is possible to generate speech using large (word-size) continuous units instead of short discrete ones. our major technical contribution is to replace the three standard elements of a text-based lm (lookup table, cross-entropy loss function, multinomial sampling) with elements adapted to a virtually infinite list of continuous embeddings. we show that with these changes, it is possible to generate speech of the same quality as discrete unit models. this is interesting because our units are 200ms long which amounts to a 5-time memory reduction compared to regular discrete units @xcite , opening up the possibility to train spoken lms on longer speech sequences. in addition, our model builds interpretable representations thanks to the lexical embedder which learns a mapping between an acoustic space, with phonetic properties, to a lexical space, with semantic and syntactic properties. we call the resulting model tgslm (token-based gslm). ## related work unsupervised speech representations like cpc, wav2vec2.0 and hubert @xcite @xcite are fixed-size representations (10 to 20ms long) that outperform traditional features, like mel-filterbanks and mfccs, in many applications @xcite . in parallel to these works, there is a growing literature on variable-length acoustic encoding called speech sequence embeddings (sse) @xcite @xcite @xcite . sse models take a sequence of speech of any length and return a fixed-size vector. these models encode speech by maximizing phonetic information while minimizing speaker identity and recording conditions. sses are used for spoken term discovery @xcite , speech segmentation into phones or words @xcite but also as input to a bert model @xcite for spoken language modelling. speech generation is often performed with a neural vocoder conditioned on mel-filterbanks @xcite @xcite . in a text-to-speech pipeline, the mel-filterbanks are obtained with another neural network, which is conditioned on text @xcite . in the next step, the mel-filterbanks are decoded into natural-sounding speech by a neural vocoder @xcite @xcite . for the zerospeech challenge 2019, @xcite proposed to remove text and replace it with unsupervised discrete units. this challenge has fueled a large body of works on learning low bitrate speech representations for speech compression, voice conversion and spoken language modelling (chen and hain, 2020; @xcite @xcite @xcite @xcite . for evaluation, the zero-resource challenge used bitrate and human evaluation. spoken language model are neural networks trained to predict missing parts of a spoken sentence with predictive or contrastive losses. gslm @xcite is the first spoken lm able to generate expressive and consistent spoken sentences in a pure textless fashion. it uses a causal transformer lm trained with nll loss on sequences of discrete units obtained with a kmeans clustering (with @xmath0. once trained, gslm can generate a sequence of discrete units by multinomial sampling that is decoded into speech with a separate vocoder. specifically, the sampled hubert units are mapped to mel-filterbanks with tacotron2.0 and decoded into speech with waveglow @xcite , a neural vocoder. @xcite also provides a way to evaluate their spoken lm using an asr to transcribe their spoken generations and an external lm to compute the perplexity of the resulting transcriptions. in addition, the zerospeech challenge 2021 @xcite designed a set of zero-shot metrics to probe what spoken lms learn. a recent paper @xcite , audi-olm, came to our attention, which we did not have the time to include in our experiments. audiolm works similarly to gslm yet with the ability to generate speech that preserves the identity of the speaker. in another line of work, @xcite trained a bert model with a contrastive loss function on sentences represented as a series of sses. they showed the resulting bert is able to model semantics and syntax. this work suggests that discrete tokenizers and the nll loss are not necessary to tackle language modelling on speech. we take inspiration on their work to design our approach. ## conclusion we introduced a generative spoken lm based on continuous word-sized acoustic tokens. our model is able to generate speech with the same level of diversity and accuracy as a model based on discrete units. this shows that building a lexicon of types is not necessary for spoken language modelling, which is encouraging considering the difficulty of clustering large segments of speech without degrading the representation (see appendix b). in addition, this performance was obtained with segments that were not very well aligned with word boundaries (200ms segments). the good result obtained with gold word boundaries indicates that there is room for improvement by using segments better aligned with word boundaries and of course a better speech decoder. further work is also needed to better limit the leakage of low-level acoustic information into the lm through continuous units, which our analysis has shown is detrimental to the performance of the generative model (see also @xcite ). finally, the fact that the units are about 5 times larger than standard gslm units aligns with the nlp literature that is in favour of word-based lms. it opens the possibility to fit larger spans of audio in gpus and capture longdistance relationships.