Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddings (2022)

Abstract: Inducing semantic representations directly from speech signals is a highly challenging task but has many useful applications in speech mining and spoken language understanding. This study tackles the unsupervised learning of semantic representations for spoken utterances. Through converting speech signals into hidden units generated from acoustic unit discovery, we propose WavEmbed, a multimodal sequential autoencoder that predicts hidden units from a dense representation of speech. Secondly, we also propose S-HuBERT to induce meaning through knowledge distillation, in which a sentence embedding model is first trained on hidden units and passes its knowledge to a speech encoder through contrastive learning. The best performing model achieves a moderate correlation (0.5 0.6) with human judgments, without relying on any labels or transcriptions. Furthermore, these models can also be easily extended to leverage textual transcriptions of speech to learn much better speech embeddings that are strongly correlated with human annotations. Our proposed methods are applicable to the development of purely data-driven systems for speech mining, indexing and search.

## introduction in spoken language understanding (slu), a goal is to understand the semantic content of spoken utterances. traditionally, research in speech processing focus on tasks that process the low-level sensory information in speech, such as automatic speech processing (asr), under the assumption that language understanding can be handled by nlp modules after speech is transcribed @xcite @xcite . yet speech-based semantic representations allow us to bypass texts in some scenarios, not only simplifying the pipeline and but also beneficial for certain domains without much transcribed data or some languages without writing systems. for speech processing, the spoken term detection tasks such as keyword detection (e.g., @xcite @xcite and query-by-example search (e.g., @xcite @xcite focus on the exact matching of audio terms in speech databases. yet the speechto-speech search enabled by spoken sentence embeddings can further expand our capacity to search speech in meaning rather than only in form. this capacity marks a significant advancement in the machine's capacity to perform speech mining, voice search and indexing and spoken information retrieval @xcite . while learning textual sentence similarity is a classic task in nlp (e.g., @xcite @xcite , the task is still relatively unexplored in speech research. the main challenge in learning spoken sentence embeddings lies in the lack of labeled data for supervised learning. given the costs associated by creating semantic ratings, it is important to explore unsupervised methods to induce semantic representations directly from speech signals. in this study, we present two approaches to tackle the challenge of inducing semantic representations directly from speech signals without any semantic labeling. the first model, waveform embedding transformer (wavembed) (figure 1 ), is a multimodal sequential autoencoder that encodes a speech signal into a bottleneck vector and reconstructs a sequence of 'hidden units', which are generated using unsupervised acoustic unit discovery. the second model, sentence hubert (s-hubert), learns the semantic representation through aligning with a frozen unsupervised text embedding model, which is trained with the hidden units (figure 2 ). we make the following contributions. with text transcriptions, the performance can further be increased to 0.7∼0.8 in terms of spearman's correlation. we made extensive comparisons and analyses of model performance under different conditions. our code, data and pretrained checkpoints necessary for replicating the experiments are avail-able at https://github.com/lingjzhu/ spoken_sent_embedding . recently, large-scale self-supervised models including cpc @xcite , wav2vec @xcite , wav2vec2 @xcite , hubert @xcite and wavlm @xcite have learned effective speech representations that can benefit a wide range of downstream speech tasks @xcite . in particular, @xcite proposed using clustering algorithm to cluster hidden states of hubert into hidden units, which were then used to create training masks. these clusters were shown to encode rich phonemic information @xcite . later it is found that discretizing speech into 'hidden units' allows the application of nlp algorithms to process speech via the proxy of these discrete hidden units, without the need of actual textual transcriptions ('textless nlp') @xcite . this discovery has greatly benefited a variety of tasks, some of which were traditionally not performed with speech, including unsupervised asr @xcite , spoken language modeling @xcite @xcite , speech resynthesis @xcite , spoken language generation @xcite , speech-tospeech translation @xcite @xcite , and spoken named entity recognition @xcite . following these prior studies, our work also falls in the domain of 'textless nlp'. unsupervised sentence embeddings learning semantic sentence embedding has been extensively studied in nlp community, such as skip-thought vectors @xcite , infersent @xcite , universal sentence encoder @xcite , and sbert @xcite . recently, unsupervised sentence embeddings have considerably narrowed the performance gap between unsupervised and supervised methods. contrastive learning has been utilized to learn a vector space in which semantically similar sentences are close to each other, such as declutr @xcite , simcse @xcite , transencoder @xcite and diffcse @xcite . another approach relies on autoencoders to compress a sentence into a latent vector representation and then reconstruct the original sentence, such as vgvae @xcite and tsdae @xcite . most unsupervised methods are based on textual sentences. in speech, current studies tend to center on acoustic word embeddings (e.g., @xcite @xcite @xcite . despite the progress, learning sentence-level embeddings for speechstill remains under-explored. in superb benchmark for evaluating speech representations @xcite , spoken sentence similarity ranking is not yet listed as a downstream task. in recent works, it has been shown that spoken sentence semantic similarities can be learned via the visually grounded speech models @xcite . multilingual spoken sentence embeddings can also be learned by using supervised multilingual text models as teacher models @xcite . these methods more or less relied on labeled data such as speechimage pairs or multilingual sentence pairs. however, we propose unsupervised methods to induce semantic embeddings from speech signals only, and our methods can also utilize textual transcriptions to improve performance if they are available. it is further assumed that some forms of transcriptions of the original speech signal exist. usually, a transcription of x take the form of a textual sequence @xmath0, y 2 , . . . , y m ], @xmath1. such data are sometimes available as most speech datasets for asr and tts are organized as pairs of speech and texts. however, in most scenarios the textual transcriptions are not available or too costly to create. in these cases, the transcriptions can be in the form of pseudo-units ŷ ∈ r 1×@xmath2, ŷ2 , . . . , ŷl ], @xmath3, which could be generated by an unsupervised system for acoustic unit discovery. during training, these transcripts are used as the targets for the proxy tasks. however, in inference, the pretrained model can directly project speech into semantic embeddings. ## data training data we used the trianing set of the 1700-hour english subset of common voice @xcite for training all speech encoders, which can be assessed through the huggingface datasets library @xcite . speech signals were all downsampled from 44.1khz to 16khz to be compatible with hubert. to avoid memory issues, we limited the maximal length of speech to be 10 seconds, which only excluded less than 1% of data. ## experiments three different types of transcriptions were compared in this study, namely, (ground truth) text transcripts, text transcripts recognized by an asr model, and hidden units. asr transcripts were additionally considered because for some languages asr systems can be used to assist the development of speech retrieval. we used the hubert-large-ls960-ft @xcite via huggingface hub to transcribe the common voice english subset to texts. for wavembed models, we trained multiple variants based on texts, asr transcriptions and hidden units. the audio encoder was initiated with the hubert-base-ls960 checkpoint. the decoder was also initiated with pretrained weights according to their outputs, including bert-base-uncased, gpt2-base, roberta-base, as well as pretrained transformers for hidden units on different vocabulary sizes. for s-hubert, we first trained simcse and ts-dae on common voice texts, asr transcriptions and hidden units, all of which were also initiated with pretrained weights according to respective inputs. specifically for simcse, additionally four existing model checkpoints from @xcite were adopted without any training nor fine-tuning. then the hubert base model with a self-attention pooling layer and a projection head on top was finetuned to aligned with the semantic embeddings of the text models in the same latent space. for hu-simcse and hu-tsdae, we only selected the best performing models to perform model distillation. given that there are several different models implemented here, detailed hyperparameter settings are given at appendix c. the implementation of transformers was modified based on the transformers library @xcite and the sentence_transformer library @xcite . all our models were trained on a single a40 gpu with 48gb of memory. unless otherwise stated, all models were initialized with pretrained models, as pretrained weights were found to improve downstream performance (see @xcite ## results and discussions in general, we show that semantic representations can be learned directly from acoustic signals through the proxy of hidden units. here we summarize the main results and the ablation analyses (see appendix d for additional results). the self-reconstruction approach is more effective in modeling hidden units than contrastive learning. in general, the wavembed method is far more effective than s-hubert when transcriptions are not available supervised training with texts is the most effective method. this holds even if supervised data are in another modality. in training with text transcripts are better than with pseudo-units. in both in order to measure the quality of the learned embeddings, we compute the metrics for alignment and uniformity proposed by @xcite , which are as follows. ℓ unif orm ≜ log e asr transcripts can be a good proxy for ground truth transcripts. modern asr systems can achieve very low word error rate (wer) on certain high resource languages. our experiments with recognized texts of common voice audio indicates that they are slightly inferior to the ground truth transcriptions in performance but the gap is pretty narrow (see wavembed and s-hubert are useful in different scenarios. our results have shown that wavembed works better when only speech signals are available, whereas s-hubert can leverage more powerful text models if transcripts are provided. furthermore, since s-hubert aligns speech embeddings with text embeddings in a common semantic space, it is also capable of performing cross-modal semantic retrieval between speech and texts. ## conclusions and future directions in summary, we investigated two unsupervised approaches, wavembed and s-hubert, to induce semantic embeddings from speech signals, which are significantly correlated with human judgements of semantic similarities. we experimented these approaches with different settings and found that performance can be improved through leveraging transcribed texts, even when the transcriptions were from slightly inaccurate asr systems. we believe that our study is an essential step towards direct speech-to-speech search and simplifying the pipeline of slu. currently, the proposed methods do not depend on any specific features of certain languages, and therefore should also work for other languages. however, it remains unclear how language specific properties can affect the actual performance. in the future, we plan to extend our methods to other languages than english. generally speaking, spoken sentence similarity task is more challenging than its text counterparts, because speech has far more variability than texts, which can be caused by various physiological, psychological and social factors (e.g., vocal tract morphology, gender, accents and emotional states). it remains to be explored how these variabilities (some of which are related to expressing meaning) can be taken into account in computational models. speech conveys information beyond its textual content. for example, prosody can reverse the meaning of a sentence from being non-ironic to ironic. while we only focus on the semantic similarity of textual content in speech, we also envision that spoken sentence similarity tasks in the future should include non-textual features such as prosody and emotion.