Augmentation Invariant Discrete Representation for Generative Spoken Language Modeling (2023)

Abstract: Generative Spoken Language Modeling research focuses on optimizing speech Language Models (LMs) using raw audio recordings without accessing any textual supervision. Such speech LMs usually operate over discrete units obtained from quantizing internal representations of self-supervised models. Although such units show impressive modeling results, their robustness capabilities have not been extensively investigated. This work focuses on improving the robustness of discrete input representations for generative spoken language modeling. First, we formally define how to measure the robustness of such representations to various signal variations that do not alter the spoken information (e.g., time-stretch). Next, we empirically demonstrate how current state-of-the-art representation models lack robustness to such variations. To overcome this, we propose an effective and efficient method to learn robust discrete speech representation for generative spoken language modeling. The proposed approach is based on applying a set of signal transformations to the speech signal and optimizing the model using an iterative pseudo-labeling scheme. Our method significantly improves over the evaluated baselines when considering encoding and modeling metrics. We additionally evaluate our method on the speech-to-speech translation task, considering Spanish-English and French-English translations, and show the proposed approach outperforms the evaluated baselines.

## introduction self-supervised speech models were shown to learn effective representations for various downstream tasks @xcite @xcite . these models were mainly evaluated on discriminative tasks, such as automatic speech recognition, speaker verification, intent classification, etc. @xcite . @xcite demonstrated that such self-supervised learning (ssl) representations can be used for generative spoken language modeling. generative spoken language modeling (gslm) is the task of learning the acoustic and linguistic characteristics of a language from raw audio. in other words, a discrete representation of the audio signal is being learned. a common practice is to extract continuous representation using an ssl model, then apply vector quantization, usually using the k-means algorithm @xcite @xcite . then a speech-language model is trained on top of the obtained representation. finally, a neural vocoder converts the output units to raw audio. as the discrete speech representation often operates over units extracted over relatively short windows (e.g., 20ms), sequences can be long and contain repetitions, e.g., 10 11 11 11 21 32 32 32 21. preliminary studies have found that removing sequential repetitions of units improves performance, hence applying it universally @xcite . for example, a pseudo-text 10 11 11 11 21 32 32 32 21 becomes 10 11 21 32 21. this framework was shown to be effective in modeling multiple levels of the speech utterance, namely prosody, and content @xcite @xcite , speech codec @xcite , speech emotion conversion @xcite , spoken dialogue @xcite , and speech-to-speech translation @xcite @xcite . an essential prerequisite for such an audio representation to be used in real-world conditions is robustness to various signal corruptions. although the aforementioned audio representation models have shown effectiveness in many tasks, they were mainly evaluated on academic benchmarks. in this work, we evaluate current state-of-theart self-supervised speech representation models on what are arguably the most basic signal vari-figure 1 : generative spoken language modeling is composed of three components: (i) speech-to-unit, (ii) unit language model, and (iii) unit-to-speech. pre-trained asr and language models are used for evaluation. as a first step, we propose a set of metrics for evaluating the model's robustness. then, we point to the lack of robustness of these models with respect to the aforementioned variations. next, we design a simple and effective method for learning augmentation-invariant discrete representation on top of any speech ssl model. we demonstrate how such a method greatly improves robustness. then, we empirically show that performance improves on several tasks for various ssl models. specifically, we evaluate the newly proposed speech encoders when considering zero-shot evaluation tasks considering encoding and modeling, i.e., abx, swuggy, and sblimp @xcite , together with a high-level downstream task in the form of speech-to-speech translation. ## background the general generative spoken language modeling (gslm) pipeline is comprised of three main modules: (i) speech-to-unit, (ii) unit language model, and (iii) unit-to-speech, where each of these modules is trained separately. speech resynthesis can be achieved while ignoring the language model and directly feeding the quantized units into the unit-to-speech module @xcite (see figure 1 for a visual description). in the following paragraphs, we give detailed background for each of the three components mentioned above, including the standard evaluation methods. speech-to-unit module encodes the raw speech signal into a discrete representation. the com-mon approach is first to encode the speech into a continuous representation and then quantize the representation to achieve a sequence of discrete units @xcite @xcite @xcite @xcite @xcite @xcite . formally, denote the domain of audio samples by x ⊂ r. the representation for a raw signal is therefore a sequence of samples @xmath0, . . . , x t ), where x @xmath1. consider an encoder network, f , that gets as input the speech utterance and outputs a sequence of spectral representations sampled at a low frequency as follows f (x) = (v 1 , . . . , v t ′ ). note that we do not assume anything about the structure of the encoder network f . lakhotia et al. (2021), evaluated several speech encoders, namely, mel-spectrogram, contrastive predictive coding (oord et al., 2018, cpc), wav2vec2 @xcite , and hu-bert @xcite . since the representations learned by such models are usually continuous, a k-means algorithm is applied over the models' outputs to generate discrete units, denoted as @xmath2, . . . , z t ′ ). each element z i in z is a positive integer, z @xmath3, .., k} for 1 ≤ @xmath4, where k is the number of discrete units. we denote the quantization model with e. unit language model is trained on the extracted discrete units, z. such a language model learns a probability distribution of the learned unit sequences, which enables direct modeling of speech data without textual supervision. the language model can be used to generate speech conditionally or unconditionally, replicating what toddlers achieve before learning to read. moreover, such a modeling framework allows for capturing and modeling prosodic fea-tures @xcite , as well as speaker identity @xcite , or even natural dialogues @xcite . this is in contrast to using textual features, as they do not encode such information. unit-to-speech module converts the speech discrete units to a raw waveform. @xcite used a tacotron2.0 @xcite based model followed by waveglow @xcite vocoder. later, @xcite proposed a unit-based vocoder based on the hifi-gan architecture to convert units to speech directly. such a paradigm seems to provide high-quality generations with better efficiency as it uses only one model rather than two. @xcite and @xcite additionally improved the unit based vocoder to include emotional tokens for speech emotion conversion tasks, and duration modeling for direct speech-to-speech translation. zero-shot evaluation. evaluating such a complex pipeline comprised of several components is a challenging task. @xcite proposed a set of zero-shot evaluation tasks aiming for each of the modules. overall the proposed tasks can be divided into four main groups: (i) acoustic encoding using abx, bitrat, (ii) language encoding using swuggy, sblimp @xcite , (iii) resynthesis using phoneme/word error rate; (iv) speech generation using vert @xcite , meaningfulness mean opinion score. ## robustness of speech-to-unit models the first step toward developing an effective spoken language model is to develop a robust representation. the focus of a robust representation should be on the spoken information rather than unrelated signals, such as prosodic features in the form on duration and f0, background noise, or reverberations. in the following section, we propose a metric for quantifying the degree to which augmentations change the resulting encoding. ## invariant discrete representation our findings in section 3 suggest that current stateof-the-art representations may be too sensitive to augmentations that do not alter spoken information. preliminary invariance research focused primarily on noise augmentation. this is convenient since the signal length is not affected by such augmentations. in practice, real-world augmentations may modify the signal length. in order to work with various types of augmentations, we must align the original and augmented sequences. the following section figure 3 : illustration of our method: we forward a clean signal through an encoder followed by a pre-trained quantizer (k-means). next, we forward an augmented signal through the same encoder, followed by a new quantizer (green). the ctc loss between the deduplicated output of the clean signal and the output of the augmented signal is used to learn the parameters of the new quantizer. in the iterative approach, post the convergence of the learned quantizer e 0 , we freeze it and learn a new quantizer e 1 that distills information from e 0 . ## experiments in the following, we assess the efficacy of our method using state-of-the-art self-supervised representations and popular discriminative and generative evaluation tasks. it is important to note that a single metric cannot tell the whole story. for example, similarly to perplexity, all representations can be assigned to the same cluster, which achieves a perfect unit edit distance but a poor representation. we first examine our proposed method using the unit edit distance along with other discriminative and generative performance metrics. then, we show that our method improves downstream tasks. in section 5.1 we use our proposed metric from section 3 to analyze the robustness of our method. in section 5.2 we study the discriminative capabilities of our method using the abx test @xcite . then, we evaluate our methods using generative zero-shot evaluation tasks such as swuggy and sblimp @xcite . finally, we demonstrate the effect of using our invariant quantizer's units in speech-to-speech translation. experimental setup. we study our method using the base versions of hubert, wav2vec2, and wavlm. for readability, we report results for hu-bert in the main paper. the results for wav2vec2 and wavlm are in appendix c. to match the current k-means training set, we use the librispeech-100h to learn our quantizer @xcite . we analyze our metric using the 'clean' and 'other' development sets from librispeech. a detailed setup is provided in appendix b. ## related work this work investigates the robustness of selfsupervised representations for language modeling. this is related to the advancements in speech selfsupervised learning, their robustness, and modern generative spoken language modeling. in the following, we review all three areas. self-supervised learning. the field of deep learning research has significantly benefited from self-supervised learning. commonly, it involves encoding the input data and performing a task that enforces the representation to learn contextual embeddings. speech self-supervised learning can be divided into two lines of research. the first is discriminative, oord et al. (2018) introduced contrastive predictive coding (cpc), which trains a convolutional encoder and a predictor for future embeddings of the encoder using a contrastive loss. on top of it, @xcite propose to use time domain augmentations to improve the cpc model further. wav2vec2 @xcite suggest using a contrastive loss that requires distinguishing between true and false future audio samples. later, wav2vec2 @xcite learn quantized units using gumbel softmax and predict masked spans of the latent speech representation. hu-bert @xcite employ a frame-based masked prediction task. first, it quantizes input frames and then predicts masked frames. the second line of work is generative. an early generative self-supervised work is autoregresstive predictive coding @xcite , which predicts the spectrum of a future frame. later, @xcite introduced mockingjay, which learns its representation by predicting non-causal context. tera @xcite alters time, frequency, and magnitude. then it is required to reconstruct acoustic frames from altered versions. ## conclusions in this work, we first propose a metric for evaluating the robustness of self-supervised speech representations applied for spoken language modeling tasks. equipped with the aforementioned metric, we point out the lack of robustness in current stateof-the-art speech encoders with respect to simple signal variations that do not alter the spoken information. we then propose a simple and effective method to augmentation-invariant discrete representation that boosts the robustness of the current approaches and demonstrate it on three state-of-theart self-supervised speech representation models. we empirically show the efficacy of the proposed approach when considering encoding methods together with a textless speech-to-speech translation.