Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessment (2025)

Abstract: Allophony refers to the variation in the phonetic realization of a phoneme based on its phonetic environment. Modeling allophones is crucial for atypical pronunciation assessment, which involves distinguishing atypical from typical pronunciations. However, recent phoneme classifier-based approaches often simplify this by treating various realizations as a single phoneme, bypassing the complexity of modeling allophonic variation. Motivated by the acoustic modeling capabilities of frozen self-supervised speech model (S3M) features, we propose MixGoP, a novel approach that leverages Gaussian mixture models to model phoneme distributions with multiple subclusters. Our experiments show that MixGoP achieves state-of-the-art performance across four out of five datasets, including dysarthric and non-native speech. Our analysis further suggests that S3M features capture allophonic variation more effectively than MFCCs and Mel spectrograms, highlighting the benefits of integrating MixGoP with S3M features.

## introduction a phoneme can be phonetically realized differently depending on its environment, a phenomenon known as allophony in phonology @xcite @xcite . for instance, the english phoneme /t/ exhibits various allophonic realizations: [t h ] (aspirated stop) in tap, [t] (unaspirated stop) in stop, [r] (flap) in butter, and [p] (glottal stop) in kitten. accurately capturing these variations is crucial, as it reflects the full spectrum of phonetic realizations within a phoneme. it is particularly important for atypical pronunciation assessment @xcite @xcite , as it has to distinguish atypical (out-of-distribution; ood) from atypical (in-distribution) pronunciations @xcite ). 1 the full codebase is available at https://github.com/ juice500ml/acoustic-units-for-ood typical speech (in-distribution) likelihood p(x) input x allophones of /t/ [t h ] [ɾ] atypical speech (out-of-distribution) p(x 1 ) < p(x 2 ) ⇒ x 1 is more atypical than x 2 x 1 x 2 [t] p(x 1 ) p(x 2 ) figure 1: summary of our method, mixgop. we model the likelihood of each phoneme using a gaussian mixture, trained on typical speech (in-distribution), to capture allophonic variations. we then evaluate on atypical speech (out-of-distribution). the y-axis represents the log-likelihood of a phoneme, where lower values indicate greater atypicality. before the era of deep neural networks (dnns), allophones were modeled for speech recognition @xcite @xcite . however, dnn-based approaches @xcite depend on phoneme classifiers that treat speech segments from a single phoneme as a single cluster, avoiding the complexity of modeling allophones. this is partly due to dnn's strong classification capabilities, which rely on trained hidden features to model individual phonemes well. in recent years, self-supervised speech models (s3ms) have shifted the landscape of acoustic modeling. unlike dnns, s3ms leverage their frozen features directly, without requiring additional training @xcite . their effectiveness motivates us to revisit modeling allophones via gaussian mixture models (gmms) @xcite . consequently, we propose mixgop, a gmm-based approach that models each phoneme as a set of allophonic subclusters (see figure 1 ). by integrating gmms with s3m features, we aim to directly cap-ture the allophonic variations. we evaluate mix-gop with s3ms in atypical pronunciation assessment with dysarthric and non-native speech. furthermore, we analyze the s3m features on how well they capture allophonic variation compared to mel-frequency cepstral coefficients (mfccs) and mel spectrograms. while previous work has shown that s3ms encode phonetic @xcite @xcite and phonemic information @xcite , a detailed investigation of how they capture allophony remains underexplored. in summary, the contributions of our study are: ## method in this section, we briefly review the conventional approach to pronunciation assessment, goodness of pronunciation (gop) @xcite . we highlight its limitations: (i) modeling a phoneme as a single cluster, and (ii) assuming atypical speech are in-distribution with respect to typical speech. we then introduce our method, mixgop, which addresses these limitations by (i) modeling allophonic variation through a mixture distribution and (ii) relaxing in-distribution assumptions by removing the softmax function. ## allophony of s3m features section 3 empirically demonstrates that leveraging s3m features with mixgop helps enhance downstream performance compared to other features, such as mfccs and mel spectrograms. this section aims to further verify the suitability of s3ms for representing individual phonemes with allophonic variations. first, we examine s3m features at the phoneme-level, using the dimensionality reduction technique in section 4.1. next, we design a metric to quantify the ability of capturing allophonic variations, comparing s3m features to mfccs and mel spectrogram in section 4.2. for our analyses, we used the healthy speech recordings from the torgo dataset @xcite , which includes gold-standard phonemic transcriptions and alignments. ## conclusion we demonstrated that improved modeling of allophony can enhance performance in ood detection for the assessment of atypical speech, and that leveraging s3m features can further improve this performance. specifically, our novel approach, mixgop, addresses the limitations of uni-modality and in-distribution assumptions by employing gaussian mixtures, which effectively model allophones and eliminate the need for softmax probabilities. additionally, we show that utilizing s3m features further enhances ood detection performance. our results also confirm that s3m features capture allophonic variation more effectively than traditional features, validating the extension of our approach to include s3ms. we evaluated eight methods across five dysarthric and nonnative speech datasets, with mixgop achieving state-of-the-art performance on four of the datasets. our work provides a deeper understanding of how s3m representations can be hierarchically structured, from allophones to phonemes. further, it sheds new light on the acoustic modeling perspective of speech, expanding the existing kmeans-based speech discretization. it shows the possibility of using atypical speech as a benchmark to measure the quality of s3m features, especially regarding ood robustness.