Uncovering Syllable Constituents in the Self-Attention-Based Speech Representations of Whisper (2024)

Abstract: As intuitive units of speech, syllables have been widely studied in linguistics. A syllable can be defined as a three-constituent unit with a vocalic centre surrounded by two (in some languages optional) consonant clusters. Syllables are also used to design automatic speech recognition (ASR) models. The significance of knowledge-driven syllable-based tokenisation in ASR over data-driven byte-pair encoding has often been debated. However, the emergence of transformer-based ASR models employing self-attention (SA) overshadowed this debate. These models learn the nuances of speech from large corpora without prior knowledge of the domain; yet, they are not interpretable by design. Consequently, it is not clear if the recent performance improvements are related to the extraction of human-interpretable knowledge. We probe such models for syllable constituents and use an SA head pruning method to assess the relevance of the SA weights. We also investigate the role of vowel identification in syllable constituent probing. Our findings show that the general features of syllable constituents are extracted in the earlier layers of the model and the syllable-related features mostly depend on the temporal knowledge incorporated in specific SA heads rather than on vowel identification.

## introduction syllables have long played a central role in phonological theory and are relatively more intuitive to grasp than other phonological entities such as segments @xcite . moreover, syllables represent a language-specific systematic organisation of sounds which allow native speakers of a language to differentiate between well-formed sequences of sounds which may not constitute actual words of the language, and ill-formed sequences which are not permissible in that language. for example, in english, the word blick is considered wellformed (an accidental gap in the lexicon) whereas bnick is ill-formed (a systematic gap); this is because the syllable constituent bl exists at the beginning of a syllable in english but bn does not. this has inspired many automatic speech recognition (asr) developments based on syllables, both in the past @xcite and the present @xcite . one of the main arguments in earlier asr model design was that the use of syllables would offer a limited number of sub-words which in turn makes the coding of the model more efficient @xcite . even though n-gram-based byte pair encoding (bpe) offers a more simplistic approach to a language-agnostic solution, humans can understand syllables as a unit of speech much better than n-grams. furthermore, the results from @xcite show that syllable-based bpe and unigram-language modelling can offer better performance when coupled with a conformer @xcite speech encoder and transformer @xcite language decoder. transformer and conformer architectures rely heavily on the self-attention (sa) weights optimised during the training. the learned parameters in the sa heads define the characteristics of each sa head. one of the core functionalities of the sa mechanism is that it takes the positional dependencies of the input to the output into account, e.g., mapping a segment of the input audio signal to the phonetic localisation of the embedded frame @xcite . given a sufficiently large amount of training data, transformer-based models achieve a high performance. however, the interpretability of the model is not given by design. this leaves the question of whether these models organise sounds systematically into wellformed syllables similar to native speakers of a language, or whether they contextualise based on the acoustic features of the audio alone. previous results suggest that the sa weights can contribute to syllable-based asr. for example, @xcite demonstrated that distilling sa weights for building connectionist temporal classificationbased asr reduces character/kana-syllable error rates for japanese. other recent work by @xcite shows that a transformer module incorporated into a syllable-based asr is superior to context-independent phoneme-based models for mandarin. this implies that sa weights attend to acoustic/contextual information needed for identifying syllables. although methods such as sa weight distillation @xcite hint at the ability of large black box models to extract relevant features for syllable-based asr systems, they do not demonstrate where and how the relevant features are embedded. in this paper, we explore an approach which evaluates the sa weights and consequently the latent representations (also known as embeddings) of openai's transformer-based whisper @xcite via sa head pruning combined with domain-informed probing tasks. first, we measure the capacity of the model to capture the distinctive features of the syllable constituents onset, nucleus, and coda in the english language (see section 2.1) and to the phonetic categories vowel, consonant, and silence. second, we explore the relevance of the phonetic categories in identifying the syllable constituents by using an sa head pruning method. figure 1 illustrates the overall workflow of this study which is explained in section 3. we find that the sa heads in the initial layer of the transformer model extract the general features needed for both probes (syllable constituent and phonetic category probe) and thus pruning any of the sa weights from this layer has a much higher impact than pruning weights from the sa heads in the other layers. however, not all sa heads contribute equally to encoding these features. even where we expect an overlap (e.g., nucleus and vowel), we find that syllable constituents and phonetic categories are not necessarily contextualised in the same way. the paper first provides background on syllable constituents and asr probing in section 2. it then gives details about the probed models, the speech corpus, and the probing tasks, as well as the sa pruning method in section 3. section 4 assesses the probe results and the impact of the sa pruning on syllable constituent encoding. the conclusion and future work are detailed in section 5, while limitations of this work are discussed in section 6. ## background linguistic studies surrounding syllables and asr probing are the two key motivators for the work presented in this paper. we provide further information on these concepts in the following sections. ## materials and methods the details of the overall workflow depicted in figure 1 are presented in the following sections. ## results the experimental results are presented in the following four sections including the probing result (section 4.1) followed by the impact of sa head pruning on syllable constituent (section 4.2) and vowel/nucleus identification (section 4.3), and finally the effect of different sa head types on syllable constituent identification (section 4.4). additionally, graphs showing detailed results for all sa head pruning constellations in whisper-base (appendix a) and whisper-base.en (appendix b) are available on the osf wiki page of this project. 6 ## conclusion and future work in this study, we probed openai's whisper-base multilingual and english-only versions for the syllable constituents onset, nucleus, and coda. the probing results show that the earlier layers of the models already encode the information required to identify syllable constituents, while the later layers improve on this by encoding more relevant features. we observed no substantial differences between the english-only (whisper-base.en) and the multilingual (whisper-base) versions; this could be due to the majority of the data being english speech for training the multilingual model. additionally, the models were probed for phonetic categories (vowel, consonant, and silence) to assess whether there is any connection between identifying a vowel and the nucleus of a syllable. to that end, a self-attention head pruning technique known as zeroing was used in conjunction with the probing tasks. this allowed us to identify the impact of different types of selfattention weight patterns on the embeddings. while pruning the sa heads impacted the performance of the probes to identify a syllable nucleus and vowels, the results showed no direct connection between the two features. however, we found that pruning sa heads with a hybrid temporal and phone-based attention pattern decreased the accuracy of syllable constituent identification more compared to sa heads with purely temporal attention patterns. this was confirmed by calculating the pearson correlation coefficient between the accuracy after pruning and the diagonality score of sa heads. overall, our findings imply that the temporal location of the self-attention weights is a more impactful factor in probing syllable constituents than purely phone-based weights. this approach can be particularly valuable in identifying relevant sa heads which can be used for sa distillation in designing syllable-based asr models, similar to what vitale et al. (2024) suggested regarding distillation of latent representations. in our case, this would involve utilising the relevant sa weights from a larger model as a teacher for a new model. in our future work, we will further study syllable constituents in the scope of latent representations of large transformer-based asr models focusing on the phonetic context. we specifically investigate the phonotactics of onsets and codas in the english language.