What Do Self-Supervised Speech Models Know About Words? (2024)

Abstract: Many self-supervised speech models (S3Ms) have been introduced over the last few years, improving performance and data efficiency on various speech tasks. However, these empirical successes alone do not give a complete picture of what is learned during pre-training. Recent work has begun analyzing how S3Ms encode certain properties, such as phonetic and speaker information, but we still lack a proper understanding of knowledge encoded at the word level and beyond. In this work, we use lightweight analysis methods to study segment-level linguistic properties—word identity, boundaries, pronunciation, syntactic features, and semantic features—encoded in S3Ms. We present a comparative study of layer-wise representations from ten S3Ms and find that (i) the frame-level representations within each word segment are not all equally informative, and (ii) the pre-training objective and model size heavily influence the accessibility and distribution of linguistic information across layers. We also find that on several tasks—word discrimination, word segmentation, and semantic sentence similarity—S3Ms trained with visual grounding outperform their speech-only counterparts. Finally, our task-based analyses demonstrate improved performance on word segmentation and acoustic word discrimination while using simpler methods than prior work.1

## introduction self-supervised speech models (s3ms) are effective across a variety of applications @xcite , including lower-level tasks such as speaker identification @xcite and speech recognition @xcite as well as more linguistically complex spoken language understanding tasks @xcite @xcite @xcite post-processing to use the encoded knowledge. (section 5.1.1) the research community has begun investigating how s3ms encode a number of properties such as speaker identity @xcite @xcite @xcite , para-linguistics @xcite , articulatory and prosodic features @xcite @xcite , and phones @xcite @xcite @xcite . work on generative models based on s3ms also suggests that some s3ms learn phone-like sub-word units @xcite . analysis on higher-level units, such as words in s3ms, has been limited. @xcite analyzed the extent to which different layers of s3ms encode word identity, and @xcite found that pooled s3m representations over word segments perform competitively on the task of acoustic word discrimination. these results indicate that s3ms encode word-identifying information. however, it is not clear from prior work how word information is distributed across frames and what aspects of words are encoded, such as their pronunciation, syntactic properties, or semantic properties; our work addresses this gap. task-specific probing classifiers have been a common analysis tool for speech models @xcite @xcite including s3ms @xcite @xcite @xcite . while these probes provide an intuitive evaluation measure, the design decisions involved in training task-specific classifiers have confounding effects, making the scores hard to interpret @xcite @xcite . we use canonical correlation analysis (cca) @xcite and training-free task-based evaluation, thus bypassing the dependence on task-specific classifiers. cca and its more robust variants have been previously used to compare representations within and across neural networks @xcite , to study text representation models @xcite @xcite , and more recently for the analysis of s3m representations @xcite @xcite . while classifiers require discrete labels, cca can be used with both discrete and continuous-valued labels. cca is also computationally inexpensive and has a closed-form solution. word similarity (wordsim) tasks @xcite have been commonly used for intrinsic evaluation of word vectors. however, previous work has observed some problematic aspects of these tasks @xcite , including that wordsim performance is not well-correlated with extrinsic evaluation, whereas cca-based evaluation more consistently tracks downstream task performance @xcite . our work also shares some motivation with the zero resource speech benchmark @xcite , but tasks in this benchmark require encoding isolated word segments and/or discretizing the representations. our analyses use word segment representations extracted in context, in order to match the most common use cases of s3ms. the task of acoustic word discrimination (awd) @xcite has been commonly used to evaluate segment-level acoustic word embeddings, using both unsupervised @xcite @xcite and supervised models @xcite @xcite . to our knowledge, only @xcite and @xcite have studied the use of s3ms for generating acoustic word embeddings for awd. van staden and @xcite used ''first-generation'' s3ms while @xcite analyzed two of the same modern s3ms we study here. our work provides a more comprehensive study of using multiple approaches (unsupervised mean pooling, dynamic time warping, and supervised models using s3m features), as well as a new state-of-the-art for one of the most commonly used awd benchmarks. word unit discovery and segmentation are common benchmark tasks that have also been used to study speech representations @xcite @xcite @xcite @xcite . previous work studying the segmentation capabilities of s3ms includes @xcite , based on first discovering phone-like units and then using them to discover word segments, and @xcite , based on thresholding the attention map of a visually grounded speech model (vg-hubert, which we also use here). our study complements this prior work by comparing the layer-wise performance of a large number of s3ms for this task and showing that a simple, training-free segmentation algorithm performs very competitively. textual sentence similarity is a classic task in nlp @xcite , but there are only a few studies investigating spoken utterance representations for this task @xcite @xcite . some downstream tasks in the superb benchmark @xcite successfully use spoken utterance representations from frozen s3ms, represented by a single mean-pooled vector. we complement our understanding of the capabilities of pooled utterance representations by performing a broad cross-model comparison. ## analysis methods we extract frame-level and span-level representations from all layers of each s3m. we use cca to compare word segment representations with various linguistic vectors (section 3.1) and also investigate encoded properties using training-free approaches for several tasks: (i) acoustic word discrimination (section 3.2), (ii) word segmentation (section 3.3), and (iii) sentence-level semantic similarity (section 3.4). linguistic property attribute vector (dimension) word identity one-hot embeddings (500) word pronunciation acoustically grounded word embeddings (128) part-of-speech tags attributes derived from ptb 1 (45) semantic attributes attributes derived from semcor foot_2 (41) ## experiment details we present analysis for ten s3ms differing in (i) pre-training objective, (ii) data modality (using either speech or image-speech pairs), and (iii) model size. the pre-trained checkpoints are obtained from publicly available sources. 4 for word-level analysis on librispeech @xcite (sections 4.2 and 4.3), we use ground-truth word alignments generated by the montreal forced aligner @xcite . ## findings we present our findings in two parts. section 5.1 investigates the spoken word knowledge learned by s3ms, and how this information is distributed across the frames of each word segment. section 5.2 looks at specific linguistic propertiespronunciation, syntactic, and semantic-of word-level and sentence-level representations. ## conclusion the analyses presented here further our understanding of s3ms, specifically their representation of word-level properties. some of our findings corroborate patterns found in earlier work; for example, the most linguistically ''deep'' information appears to be encoded best in a small set of intermediate layers, and pre-training objective and model size influence layer-wise trends. we contribute new findings about previously unstudied aspects of s3ms, such as the distribution of word information within word segments and the encoding of syntactic and semantic features. most importantly, the comparison of a large number of models using the same analyses and tasks, and the study of multiple word-level properties, enables a more complete understanding of the space of s3ms. as an additional product of this work, we obtained strong results on multiple benchmark tasks, outperforming prior work using simple models based on frozen s3m representations. our work studies which s3m layers are better (or worse) at encoding certain linguistic properties. previous studies have used similar findings to guide modeling decisions when adapting pre-trained models for downstream tasks @xcite @xcite , including the choice of which layers to drop, distill, foot_11 or reinitialize @xcite @xcite @xcite @xcite . we therefore expect our findings to inform design choices for both model development and their utilization for downstream tasks. for instance, for all the s3ms we study, our analysis reveals that linguistic content is most prominent within the intermediate layers (figure 7 ). since layers encoding semantic content should be particularly beneficial for language understanding tasks, our findings suggest an exploration of alternative strategies to the common practice of adding a prediction head to the topmost layer @xcite . our analyses have addressed several questions about s3ms' word-level representations, thereby providing a foundation to address more challenging questions. for example, a natural next step is to ask how much (and where) phrase-and sentence-level properties, such as constituents, dependencies, and entities, are encoded. for some tasks, such as word segmentation, they are far from solving the task, although our results with s3ms are stronger than prior work. finally, we have noted (as have some prior studies) that larger models are not always better by all measures, raising the question of what the additional model capacity provides and whether there is a better way to train and utilize larger models.