Domain-Informed Probing of wav2vec 2.0 Embeddings for Phonetic Features (2022)

Abstract: In recent years large transformer model architectures have become available which provide a novel means of generating high-quality vector representations of speech audio. These transformers make use of an attention mechanism to generate representations enhanced with contextual and positional information from the input sequence. Previous works have explored the capabilities of these models with regard to performance in tasks such as speech recognition and speaker verification, but there has not been a significant inquiry as to the manner in which the contextual information provided by the transformer architecture impacts the representation of phonetic information within these models. In this paper, we report the results of a number of probing experiments on the representations generated by the wav2vec 2.0 model’s transformer component, with regard to the encoding of phonetic categorization information within the generated embeddings. We find that the contextual information generated by the transformer’s operation results in enhanced capture of phonetic detail by the model, and allows for distinctions to emerge in acoustic data that are otherwise difficult to separate.

## introduction in recent years large transformer models have become available which provide a novel means of generating high-quality vector representations of input speech audio sequences. these transformers aim to exploit feature learning on large unlabelled datasets to perform sequence-to-sequence transformations on audio that capture and preserve salient features from the input sequence in a quantised and contextual output representation. while most work on transformer models in automatic speech recognition focus on performance improvements and applications in down-stream tasks, this paper focuses on whether the internal layers of a transformer model provide any information as to the emergence of phonetic and phonological properties of speech. specifically we interrogate the wav2vec 2.0 model @xcite ) by probing the internal layers of the transformer using domaininformed features. the structure of this paper is as follows. firstly, in section 2 we present some existing research related to our approach followed by a discussion of transformer-based models in section 3. section 4 presents the resources used, and the experimental methodology is described in section 5. in section 6 we present our results, followed by conclusions and future work in section 7. ## related work there has been considerable work in recent years as to the extent and nature of phonetic information captured in the embeddings used by deep learning models. the word2vec model @xcite has been applied below the level of the word to investigate phonological analogies and similarities. @xcite have explored the sound analogies generated by phoneme embeddings. @xcite detailed the ability of embeddings to capture phonemic and allophonic relationships within an artificial language, noting that contrastive elements within the embedding space correlated with articulatory features. @xcite demonstrate that embeddings derived purely from text using a grapheme-to-phoneme mapping and applying a word2vec approach exhibit similarity between phoneme classes. these phoneme embeddings were subsequently integrated with the data-driven acoustic similarities of @xcite to generate a similarity matrix for use in phonemically driven spell checking @xcite . specifically with respect to the capture of phonetic information in the embeddings of automatic speech recognition, @xcite have investigated the internal layers of end-to-end recognition systems using a connectionist temporal classification (ctc) approach with deepspeech2 @xcite . they found significant differences across layers in their architecture with respect to predictive performance of phoneme categories. their work also demonstrated that certain categories became represented in the embedding space of their chosen model such that intra-category separation was significantly more difficult than for other categories. they noted that these categories saw better performance in later layers, at the expense of degraded performance in more easily separable categories. @xcite have investigated the representation of speech in deep neural networks using a 3-layer model trained to distinguish consonants and vowels. they performed a wide-ranging comparison of pca-transformed embedding spaces, and their work saw strong clustering on the basis of the vowel/consonant categorisation and manner of articulation. most recently, @xcite investigated the extent to which phonetic properties emerge from the acoustic representations of transformer-based speech recognition architectures. using four pre-trained acoustic representations from transformer-based speech recognition architectures, they designed probing tasks using linear regression, a support vector machine and a feedforward neural network consisting of two fully-connected layers. their embeddings are associated with high-level categorisations derived from the timit dataset @xcite , perform at a high level and see significant improvements across layers when considering lessseparable classes such as fricatives. @xcite proposed a methodology known as probing as a way to examine what information is present in an embedding. in conneau et al.'s framing a probing task involves training a classification model to predict properties (e.g., length, tense, parse tree depth, and so on) of a sentence based on the embedding of the sentence. probing assumes that the accuracy of the classification model (i.e., a probe) on the task indicates whether the embeddings encode information relevant to task target. there is a growing body of work using probing to examine what types of information are encoded in the embeddings created by trans-former models @xcite @xcite , and also exploring what layer in the transformer architecture different types of information are encoded in @xcite . in this work, we adapt the probing methodology to speech embeddings, and use it to understand and compare the phonetic information encoded in different layers of a transformer model. through this comparision of probing performance across layers on phonetic tasks we hope to better understand whether the information encoded in these speech embeddings, and the sequencing of this encoding across layers, accords with domain-knowledge expectations regarding phonetics. the work presented in this paper focuses specifically on the transformer module of the wav2vec2.0 model @xcite and the representations generated at each layer of the transformer. it will not probe the attention mechanism itself, which is outside the scope of this paper. the primary goal of this investigation is not to deliver an explanation of the operations undertaken by the transformer architecture in generating these representations, but instead to probe the representations generated at different layers across the architecture in order to examine the development of the architecture's ability to delineate between phonetic categories. ## transformer-based models in recent years transformer-based models have reported state-of-the-art results on a range of speech processing tasks, and today pre-trained models are available for a variety of high-demand tasks such as automatic speech recognition (asr). these models leverage the availability of large unlabelled acoustic datasets, in parallel with enhanced architectural features such as attention mechanisms, to produce information-dense distributed vector representations (embeddings) of input audio signals. in the architecture examined herein, embeddings are of a n*t dimensionality, with width n dependent upon input length, and each instance of t representing the dimensionality of the encoded information within a specific time-frame, and specific variance within that dimensionality relating to differences in the acoustic feature space for that frame. the excellent performance of transformer based models on speech processing tasks suggests that these models have the ability to encode within the embeddings they generate aspects of the input sig-nal relating to speech phenomena, while discarding low-information aspects of the input signal such as background noise and variation deemed to be unimportant during the training cycle. furthermore, some architectures such as wav2vec 2.0 have been designed to exploit the high-quality of embeddings generated from unlabelled data by allowing for very small quantities of labelled data to be provided as fine-tuning information during a separate training stage while still achieving high levels of transcription performance. however, while there has been significant inquiry as to the final-level performance of these models, relatively little is known as to the specific information captured within the embedding space, and whether that encoded information accords with domain-knowledge expectations. previous works have explored the use of these embeddings as the basis for higher-order operations, such as accentresilient asr @xcite , identification of speaker emotional state @xcite , and modelling of prosody in speaker input @xcite . for the probing task detailed in section 5, the phoneme embeddings (calculated by averaging the embeddings for the frames within the phoneme interval) for each layer in the multi-layer wav2vec2.0 transformer stack are used as inputs for the training of a multi-layer perceptron (mlp) on the task of identifying an associated timit phonetic label. the performance of this model is taken as indicative of the relative richness of specific phonetic data within the output embeddings from wav2vec 2.0. ## methodology this section sets out the experimental methodology employed in this paper, outlining specifically how the relevant data was generated and the description of the probing task. figure 1 provides an overview of the steps involved. ## results this section presents a discussion of the results of the probing task. firstly, categorisation accuracies for each predicted category per layer were considered. then, heatmap representations of all phoneme confusions for layers of interest were considered in order to focus on the emergence of specific domain-informed categories, in this case a grouping of the consonants categorised with respect to manner of articulation based on hierarchical clustering. ## conclusions and future work while the specific nature of the phonetic information captured by modern large transformer models will require significant further work to adduce, this paper has demonstrated that there is significant evidence to suggest that transformer architectures are capable of capturing significant levels of phonetic detail that accords with domain-informed understandings of phoneme relationships, and that permit distinction between less separable phonemes. future work will look to establish more concretely the nature and effective mechanism of the layer-wise changes to these characteristics and the emergence of phonological generalisations, as well as looking to explore other aspects of the mechanisms assofigure 8: obstruents at layer 12