Probing Subphonemes in Morphology Models (2025)

Abstract: Transformers have achieved state-of-the-art performance in morphological inflection tasks, yet their ability to generalize across languages and morphological rules remains limited. One possible explanation for this behavior can be the degree to which these models are able to capture implicit phenomena at the phonological and subphonemic levels. We introduce a language-agnostic probing method to investigate phonological feature encoding in transformers trained directly on phonemes, and perform it across seven morphologically diverse languages. We show that phonological features which are local, such as final-obstruent devoicing in Turkish, are captured well in phoneme embeddings, whereas long-distance dependencies like vowel harmony are better represented in the transformer’s encoder. Finally, we discuss how these findings inform empirical strategies for training morphological models, particularly regarding the role of subphonemic feature acquisition.

## introduction the transformer architecture has revolutionized natural language processing and computational linguistics since its introduction by @xcite . while it has achieved state-of-the-art results across various tasks, much remains to be understood about its inner workings and representations. investigating how transformers acquire and use linguistic knowledge is crucial for assessing their ability to generalize beyond shallow pattern recognition. one such aspect is morphological knowledge, such as that examined in tasks like morphological inflection @xcite , where a model predicts a word's inflected form given a lemma and morphosyntactic attributes. for example, for the english lemma "hug" and the morphosyntactic attributes verb;past, the model should output "hugged". in many languages, morphology interacts in meaningful ways with phonological attributes, for example through the phe-nomenon of harmony, raising the question of how this correspondence is manifested in model representations. this question is hard to pursue due to the general scarcity of multilingual morphological data, which hinders models' ability to generalize to new lemmas and morphosyntactic attributes @xcite and to adapt to the diversity of morphological processes @xcite . in this work, we present a language-agnostic method for testing phonological features and long-context feature agreement in models trained on a morphological task. 1 we do so by training designated probing classifiers that predict linguistic properties from a model's internal representations @xcite . we show how a morphological transformer implicitly acquires phonological knowledge, complementing previous findings regarding the representations found in neural phoneme embeddings @xcite @xcite @xcite @xcite and the information conveyed by morphological models @xcite @xcite . unlike previous work, we demonstrate via explainability methods that model representations explicitly encode phonological features, and quantify how well they are encoded across languages and features. we test the hypothesis that when trained on reliable phonological representations, models acquire subphonemic features such as voice or round @xcite ) that play a role in morphology, and that this ability depends on a language's reliance on such features in encoding inflectional properties. we find that local phenomena, such as final consonant devoicing, are captured in the character embeddings, while long-distance phenomena are better represented in contextualized embeddings from the transformer encoder. finally, we argue for current practices in language transfer of morphological models based on our results. ## probing phonological features our experiments consist of three stages: training a phonemic transformer model on a morphological task for a language; probing the embeddings for phonological features; and analyzing the probe using minimum description length (mdl). ## results we apply our method to a set of seven languages selected to provide reasonable diversity with respect to morphophonological phenomena. 7 these languages represent different morphological typologies: agglutinative / fusional, prefixing / suffixing / non-concatenative; and exhibit diverse phonological rules, such as palatalization and vowel harmony. we focus on languages with low orthographic depth, which allows us to probe phoneme embeddings in a way comparable to probing character embeddings in standard orthography. we first compare the results to a control probing task where the phonological features (labels) are randomly shuffled @xcite . the results, presented in figure 2 , validate that the compression score is a good indicator of phonological feature representation in the embeddings, as all scores remain below 1.0. we next discuss specific morphophonological phenomena and how they manifest in the data. turkish final-obstruent devoicing. in turkish, a word-final [-continuant] consonant is devoiced. moreover, in accusative case the consonant becomes voiced, and a [+high, -rounded] vowel is suffixed, with the back value subject to vowel harmony. for example, kebap (the kebab) becomes kebabı (acc the kebab). our probe shows that both voice and continuant features have a relatively good compression score in both probes, with more prominence compared to other features in the inflection task. hungarian gemination. nearly every phoneme in hungarian has a corresponding [+long] variant of it in the phonetic inventory. there are two processes that can alter the value of the long feature: gemination, where consonants at the end of a verb become [+long] before a suffix, and degemination, where [+long] consonants become [-long] when preceded or followed by another consonant. among all languages and features in the phoneme probe, the compression score for long in hungarian is the highest in the inflection model. we hypothesize that this is due to two factors: (i) morphological alternations affecting the gemination process, and (ii) the high entropy of long, which effectively separates hungarian's phonetic inventory, allowing the probe to achieve a relatively high compression score. long-context feature agreement. vowel harmony is a rule requiring all vowels in a word to share a specific phonological feature. for example, turkish and hungarian exhibit vowel harmony for round and back. since this rule influences both phonotactics and morphology, we expect these features to have high compression scores. while this is not observed in the phoneme probe (figure 2 ), the harmony probe results (figure 3 ) show high compression scores for context-dependent embeddings in the inflection model. results for probing consonant harmony are provided in appendix a. ## discussion we showed that a morphological transformer can effectively acquire phonological features. the quality of their representation, as reflected by the compression score (c), varies across features and languages, and is influenced by how informative they are per language. features prominent in phonotactics or in short-context environments are represented better in the phoneme embeddings, while those more present across long contexts are represented better through the encoder. higher scores are generally observed for features that are more central to morphology and phonology, though these results may also be influenced by the quality of the datasets or grapheme-to-phoneme tools. surprisingly, in the phoneme probe, the lemma copying model achieves on par with or even better than the inflection model. we believe this might be due to dataset noise, as explored in prior work @xcite . future work could investigate the variance across languages and models. our findings complement work that showed that adding subphonemic features hardly improves model performance, suggesting these are already present in their representations @xcite . our lemma copying findings reinforce the common practice of pretraining models for this task before turning to inflection @xcite @xcite , which has been argued to succeed due to inducing "copy bias" and to coaxing attention modules towards monotonicity @xcite . finally, our results imply that the demonstrated success of transfer learning in morphological inflection, even between typologically unrelated languages @xcite , might stem from the model's ability to acquire subphonemic features, which are approximately universal @xcite and therefore transferable. ## conclusion in this paper we analyze phonological transformers trained on a type-level morphological task, finding that these models acquire subphonemic features. we show that the degree to which these features are embedded in the transformer's representation depends on the feature's importance for the morphology and phonology of the language it is trained for; and on the locality of the feature's importance: in the encoder, long-context features are more salient. we use these results to explain empirical training methods used in the morphology inflection domain. we hope this analysis will add an analytical tool in explaining morphological models using phonology acquisition.