Do transformer models do phonology like a linguist? (2023)

Abstract: Neural sequence-to-sequence models have been very successful at tasks in phonology and morphology that seemingly require a capacity for intricate linguistic generalisations. In this paper, we perform a detailed breakdown of the power of such models to capture various phonological generalisations and to benefit from exposure to one phonological rule to infer the behaviour of another similar rule. We present two types of experiments, one of which establishes the efficacy of the transformer model on 29 different processes. The second experiment type follows a priming and held-out case split where our model is exposed to two (or more) phenomena; one which is used as a primer to make the model aware of a linguistic category (e.g. voiceless stops) and a second one which contains a rule with a withheld case that the model is expected to infer (e.g. word-final devoicing with a missing training example such as b→p) results show that the transformer model can successfully model all 29 phonological phenomena considered, regardless of perceived process difficulty. We also show that the model can generalise linguistic categories and structures, such as vowels and syllables, through priming processes.

## introduction in computational linguistics, neural networks have occupied much of recent work. one prime driver is adaptability to multiple facets of linguistic phenomena. as an example, sequence-to-sequence models have been shown to capture inflection patterns across numerous languages @xcite . while their performance represents significant advances, the abstractions generated during the modelling process warrant further investigation. we experiment with phonological processes on a constructed language to compare the generalisations learned by transformer models with widespread linguistic phenomena. in particular, we address the following questions: we establish that the transformer model successfully models all 29 phonological phenomena we consider, regardless of linguistic complexity. our results show that the model can generalise to linguistic categories with some caveats. by examining the transformer model's generalisation of haplology, we show that the model appears to learn syllables; the model can recognise the difference between vc and cv and generate previously unseen cv sequences. ## related work investigating the cognitive reality of linguistic categories defined within phonology has long been of interest to linguistics. does the natural class of phonemes bear any significance to a cognitive reality? for example, a series of experiments @xcite @xcite examine the natural class of vowels and whether phonological patterns can be extended to previously unseen vowels. the studies suggest that participants were mostly able to generalise. in a similar vein, @xcite presents a study on consonant harmony. the results suggest that learners (human learners) can generalise to novel consonants when the phonological pattern is general. however, the learners failed to generalise when the rule triggering the consonant harmony pattern was highly specific. we adapt this long-standing linguistic question to ask whether transformer-based abstractions are linguistically informed. our experiment setup swaps the human learner with the transformer architecture. previous studies investigating phonological phenomena with transformers include @xcite , where transformers can handle reduplication and gemination. to an extent, foot_0 the sig-morphon shared tasks @xcite also demonstrate the capacity of transformers to represent phonological processes through capturing allomorphs conditioned by phonological environments. there have been extensive studies on various phonological processes and rnns. @xcite shows that encoder-decoder networks (specifically lstm and gru architectures) can learn infixation and reduplication. @xcite explores whether phonological distinctive feature information is required for learning word-level phonotactic generalisations using lstms. the authors find that information about phonological features hinders model performance, and phonotactic patterns are learnable from the distributional characteristics of each segment alone. moreover, distributional information proves to be integral in recovering phonological categories @xcite . another way to investigate neural architecture abstractions is to probe the model internally. @xcite examines whether rnn states encode phonological alternations through experiments on finnish consonant gradation. the authors show that the models often encode consonant gradation in a select number of dimensions. @xcite probes the hidden states of an rnn model which controls turkish vowel harmony. similarly, @xcite establish a correlation between embedding representations and distinctive phonological features for finnish, spanish and turkish. this paper focuses on a model-external interrogation of transformer generalisations by studying the predictions produced. ## language design the phonological phenomena in question are tested on a constructed language. the primary motivation for this is to allow for a controlled experiment and ensure that we can generate enough samples of the required phonological environments for rules to be triggered and thus observed. with this in ## data 2 our data preparation follows three steps: lexicon generation, triplet (lemma, tag, surface form) formation via the finite-state tool foma @xcite and, finally, sampling of these triplets ac-cording to the experiment at hand and formatting for fairseq. (ott et al., 2019) 3 we train the model as a standard 'inflection' task @xcite , but with tags being identifiers of the processes that are to be triggered instead of morphosyntactic information. for example, the input sequence moupi#gemination would be paired with the output mouppi. more example triplets are shown in lexicon generation entails generating viable syllable structures and filling these abstract structures using vowel and consonant inventories. the syllables are concatenated n times, where n is an integer between 1 and 10. we sample from this uniform distribution to produce a gaussian distribution for word length with a mean of 8 symbols. we include a copy tag, where the input is copied to the output, to negate any performance drop by the model when unseen lemmata are encountered @xcite . in other words, the model, at test time, will never encounter a completely unseen lemma on which to perform a phonological change, since it will always have witnessed at least an input-output pair of any lemma used that is simply copied to the output. ## modelling common phonological processes with varying degrees of complexity in this experiment, we establish that seq2seq models can successfully capture a range of phonological processes, including more complex rules such as metathesis. as seen in figure 1 , the transformer model performs reasonably well across all phonological phenomena, with little distinction between the complexity of the process considered. ## linguistic category generalisation we examine whether the transformer model can generalise linguistic categories such as vowels or syllables from examples of alternations. during training, we expose the model to two phenomena at once (priming/held-out cases) of processes where the model could potentially infer relevant categories and extend this knowledge to withheld cases. the first set of experiments focuses on the generalisation of vowels, and the second centres on categorising consonants. ## word-internal representations to test whether seq2seq models can learn a representation of word-internal structures, such as syllables, we experiment with examples of haplology. haplology (@xmath0. for example, if the word haplology were to undergo haplology, it would reduce the sequence lolo to lo, @xmath1. in this experiment, we include two additional processes so the model can witness the contrast between vowels and consonants separately: (1) wordfinal vowel deletion and (2) word-final consonant deletion. ## conclusion the transformer model successfully models all 29 phonological phenomena with slight variation across phenomenon complexity. our results show that the model can generalize linguistic categories and structures. through haplology, we show that the model appears to learn to recognize and generalize syllabic structure and is capable of recognizing the difference between vc and cv and can also generalize the transformation triggered by haplology to unseen cv sequences.