Exploring How Generative Adversarial Networks Learn Phonological Representations (2023)

Abstract: This paper explores how Generative Adversarial Networks (GANs) learn representations of phonological phenomena. We analyze how GANs encode contrastive and non-contrastive nasality in French and English vowels by applying the ciwGAN architecture (Begus, 2021). Begus claims that ciwGAN encodes linguistically meaningful representations with categorical variables in its latent space and manipulating the latent variables shows an almost one to one corresponding control of the phonological features in ciwGAN’s generated outputs. However, our results show an interactive effect of latent variables on the features in the generated outputs, which suggests the learned representations in neural networks are different from the phonological representations proposed by linguists. On the other hand, ciwGAN is able to distinguish contrastive and noncontrastive features in English and French by encoding them differently. Comparing the performance of GANs learning from different languages results in a better understanding of what language specific features contribute to developing language specific phonological representations. We also discuss the role of training data frequencies in phonological feature learning.

## introduction recent studies in natural language processing (nlp) have demonstrated two generic trends: neural networks dominate language-specific machine learning models; the common practice of model training (pre-training and fine-tuning) outperforms many traditional training methods and is particularly suitable for the development of language models used for various downstream tasks. these language models, however, are of black-box nature. the interpretability of these models is limited that the language representation they learned might not align to human language. how, then, to understand the opaque and complex learned representation of language models is an important question in recent studies. phonology, the study of the sound system of human languages, plays an important role in understanding models' inherent biases and their ability to make human-like generalizations. the sound systems of human languages are not organized arbitrarily, but contain structural generalizations and interdependence. thus, learning a sound system involves not only learning to acoustically realize or recognize segments (phonetics), but also mapping them to an inventory characterized by distinctive features, and learning distributional constraints on segment sequences (phonology). just as computational psycholinguists have investigated the degree to which neural network language models learn linguistically motivated features like syntax @xcite @xcite @xcite , they have also investigated the degree to which phonological organization emerges from neural models trained on acoustics @xcite . the degree to which these models learn phonological features is still debatable. recently, a neural network autoencoder seems to successfully learn phoneme-like representations without explicit labels @xcite . while autoencoders seem to acquire some phonological generalizations, their representations of the kind of phonological features used by linguists are both incomplete and distributed across the latent space, requiring probing classifiers to detect. because of this limited success and lack of transparency, it is difficult to tell whether higher-order phonotactic dependencies between different segments are acquired. generative adversarial networks (gans) @xcite @xcite , on the other hand, are claimed to model language acquisition naturally because gans can model phonetic and phonological computation as an almost one to one mapping from random space to generated data of a gan instance trained on raw speech data @xcite . the learned internal representations of gans is claimed to resemble phonological learning in human speech acquisition: gans learn to build their internal latent space via unsupervised phonetic learning from raw acoustic data, which is similar to human constructs underlying phonological representation by listening to the speech sounds in a language. beguš (2021a) proposed ciwgan @xcite which is based on wavegan architecture but with an extra q-network that motivates the generator to produce linguistically categorical and meaningful sounds. @xcite shows that ciwgan can encode allophonic distribution: word-initial pre-vocalic aspiration of voiceless stops ([p h it] v.s. @xcite ). in english, the aspiration of stop consonant t occurs initially before a vowel (#t h v, h refers to the aspiration) while a period of stop closure occurs between the aspiration and the period frication noise of [s] (#stv). ciwgan successfully learned and generated this allophonic distribution in that the generated outputs obey this phonological constraint. moreover, changing a single variable in the latent space is capable of changing generated tokens from stv to t h v, suggesting an almost one-to-one correspondence between latent variables and phonological features. this finding is claimed to prove that gans can model unsupervised phonological representation learning from raw speech data. in this study, we explore the robustness of ciw-gan as a phonological feature learner by testing ciwgan on learning the feature of nasality, which is distinct in french and english. nasality is a contrastive feature for french vowels; nasal vowels can appear independently of nasal consonants @xcite . in english, however, vowel nasality is allophonic, like voiceless stop aspiration -nasal vowels appear only preceding nasal consonants. linguists traditionally analyze this relationship as reflecting a single nasal feature on the consonant, without an independent feature controlling vowel nasality @xcite @xcite @xcite . thus, our experiment provides a more rigorously controlled test of the claims of @xcite . ciw-gan networks are trained on english and french datasets respectively to learn the distinct nasal features of the two languages. analysis of the result ciwgan networks is development to answer the following research questions: (1) what features of the data contribute to learning the nasal representations in english vs. french? (2) how does the training data's distribution affect the learned feature system in wavegan network? results show interactive effects between latent variables in controlling the phonetic and phonological features: multiple to one corresponding mapping is found between latent variables and the phonetic and phonological features, suggesting that the claimed advantage of gans over autoencoders is not as great as was originally claimed. ciwgan do react differently in encoding the different nasal representations in english and french to indicate whether a feature is or is not contrastive, highlighting their potential as phonological learners. moreover, we found that training data's distribution affects the learned feature system in ciwgan; to the extent that gans can be considered cognitively plausible models of human learning, this may lead to predictions about how changes in phonetic distribution can become phonologized into almost-categorical rules. ## related works we review two areas of recent literature. largescale unsupervised models of speech learn words and in some cases phoneme categories, but the degree to which they acquire phonological feature systems is not clear. some smaller-scale models have been specifically analyzed in phonological terms. one recent and successful pre-trained model (wav2vec 2.0) is shown to encode audio sequences with its intermediate representation vectors, which demonstrates superiority in downstream fine-tuning such as automatic speech recognition (asr) tasks, speaker verification tasks, and keyword spotting tasks @xcite . similar to wav2vec, hu-bert @xcite , a pretrain language model that leverages selfsupervised learning for speech, directly processes audio waveform information from raw speech to predict clustering categories for the speech segments. both wav2vec 2.0 and hu-bert have been successful in capturing acoustic information from raw speech and improve the state-of-the-art performance in speech recognition and translation. @xcite introduces a dilated causal convolutional network wavenet which attempts to discover phone units from audios; how-ever, because of the lack of lexical knowledge, wavenet cannot emit explicit phonemes @xcite . moreover, the submissions for the zerospeech challenges @xcite @xcite utilizes generative models like gans @xcite and autoencoders @xcite to learn the lexical or phone-level presentation from raw speech data. however, the learning of phonology features of language from raw speech data is not particularly implemented or evaluated in the above studies. although these models have shown impressive results in speech representation learning that capture phonetic/acoustic content, the degree to which they acquire phonological feature systems is still not clear. some studies have been focused on developing language models that learn phonological representations. in @xcite , an autoencoder neural network is trained on pre-segmented acoustic data and output values that correlates to phonological features. nevertheless, the architecture of autoencoder brings a problem in learning phonological representation: because autoencoders are trained to reproduce their inputs faithfully, their latent representations may contain too much information which is extraneous to phonological categorization, such as speaker-specific information. gans are not trained to strictly reproduce the training data and therefore might not be subject to this issue. recently, @xcite 's study applies the gan architecture based on the dcgan architecture @xcite to learn language features from continuous speech signals (wavegan). gan networks as generative model, is firstly applied in learning allophonic distribution from raw acoustic data in @xcite which also proposes a probing technique to interpret the internal representation of gan networks. the internal language representation is probed and claimed to be interpretable in @xcite ; @xcite which firstly shows that gan networks can learn reduplication and conditional allophonic distribution of voice onset time (vot) duration from the raw speech audio, respectively. @xcite proposes ciwgan (categorical infowavegan) and fiwgan, two gan networks for unsupervised lexical learning from raw acoustic inputs; the two gan networks combine wavegan with infogan, an extension to gan architecture, that includes an additional "q-network" which encourages the model's productions to group into discrete categories @xcite . in these earlier papers, the discrete representational elements in these gan architectures were proposed and interpreted with respect to lexical category learning. in our work, this interpretation does not apply, since our data consists of syllables rather than whole words. while top-down lexical information appears critical to learning many phonological contrasts, the rules governing the distribution of vowel nasality we are studying here are local phonotactic phenomena which can be learned purely by capturing the distribution of vowels and coda consonants. ## model in this paper, we use ciwgan to model phonetic and phonological learning for vowel nasalization in english and french. the gan architecture involves two deep convolutional neural networks: the generator network and the discriminator network @xcite . they are trained against each other to boost their performance. the generator network is trained to generate data from a set of latent variables and maximize the error rate of the discriminator network. the discriminator takes the training data and output of the generator network as input and attempts to determine whether its input comes from the training dataset (actual data) or generator output (fake data). the competition of the two networks against each other makes the generator generate data that is similar to the actual data. the architecture of ciwgan is shown in figure 1 . the generator takes categorical binary latent variables ϕ (size is 3 in figure 1 ) and continuous latent variable z that are uniformly distributed in the interval (-1, 1) as input and outputs a continuous time-series data as audio signal (x). the q-network, extra component in ciwgan than wavegan, also takes audio signals as input, but gives a categorical estimation φ on the audio signal. it is trained to minimize the difference between the categorical estimation φ and the actual latent categorical variables ϕ in the generator's latent space. with the q-network, the generator is motivated to generate audio signals that are categorically distinguishable for the q-network. to interpret the learned phonological features in the generated output, begus and zhou (2022) uses regression analysis. they manually label each generated audio snippet with its phonological features, then measures the strength of correlation between the latent variables (z) and the phonological feature of interest. we also use this technique in our experiments to find the latent variables that correspond to the nasal feature in english and french. beguš (2020) uses regression analysis from the latent variables to the phonetic and phonological features in the generated outputs to reveal the correspondence relations between latent variables and the phonetic and phonological features. however, to avoid expensive manual labeling, we develop a supervised nasal detector (nasaldnn), a deep neural network model adapted from @xcite , to determine whether a generated output carries nasality or not. the nasaldnn is a 1d cnn that takes speech segments as inputs, and calculates the posterior probabilities for the sample at the center point of the segment belongs to nasal phoneme classes [n, m, ng]. for french, we trained the convolutional nasaldnn on the siwis dataset, which has ground truth labels for both nasal consonants and nasal vowels. we used these labels to learn a four-way classifier, which we applied to the sample at the center point of each segment. in english, since timit has no ground truth labeling of nasal vowels, we used a different procedure: we learned independent classifiers for vowels and nasal sounds (using consonants as the gold examples of nasals) and detected nasal vowels by intersecting the predictions. ## data to learn vowel and nasality features in engish and french, two ciwgan instances are trained separately on timit speech corpus @xcite and the siwis french speech synthesis database @xcite . the timit speech corpus includes english raw speech sentences (at 16 khz sampling rate) and their corresponding time-aligned phonetic labels. in the timit corpus, there are 6300 sentences recorded by 630 speakers from eight dialect regions of the united states. we used the entire timit dataset to extract training data for the english experiment. the siwis french speech synthesis database consists of high-quality french speech recordings and associated text files. there are 9750 utterances uttered by french speakers. this french database includes more than ten hours of speech data. ## experiments to explore our first research question: what features of the data contribute to learning the nasal representations in english vs. french, we implement english and french experiments. the results suggest different learned phonetic/phonological representations in ciwgan may be caused by different typology of english and french syllable types for nasal vowels and nasal consonants. ## conclusion our results qualify beguš (2020a)'s claim that gans can learn clearly interpretable representational systems in which single latent variables correspond to identifiable phonological features. while we do find this in the english experiment, we do not find it in the french experiment, englishlike experiment and french-like experiment. this suggests that both the frequencies with which different syllable types in the data occur, and the contrastiveness of the phonological phenomenon, may affect whether the learned representation is simple or distributed across many variables. moreover, as the learned representations in ciwgans involve featural conjunction, this counters beguš (2020a)'s claim of ciwgans having an independent dimension for every phonological feature. in future work, understanding more complicated feature interactions, we plan to use eigendecomposition or other methods which can more easily represent higher-order interactions between features. however, our current methods are still informative about the learned representations, since the regression analyses show that only a few of the learned features are critical to representing nasality. on the other hand, we do find that gans clearly distinguish between the contrastive and noncontrastive status of vowel nasality in english and french. this supports beguš (2020a)'s higher-level claim that gans are good phonological learners by testing it in a more controlled setting in which the same feature is compared across languages. while artificially balancing the frequencies of syllable types in the training data does not erase the difference between english and french, we do observe that the learned representations are more similar between the two, and that the gans learning from english data begins to be able to generate some vt syllables, although with low frequency. this aligns with a widespread theory for the origin of contrastive nasality in languages like french. changing the patterns' frequency will change the feature systems in languages. our results highlight the difficulty of learning featural phonological representations from acoustic data, as well as the interpretational difficulties of detecting such representations once learned. we believe that the question of which architectures successfully acquire these systems is still openmore work needs to be done on larger pretrained models to determine which, if any, of these generalizations they encode. more careful comparisons between smaller-scale systems can also shed light on how well they distinguish between completely predictable (allophonic) distributional properties of segments due to phonotactic constraints, and statistical regularities due to the lexicon or morphology. on the other hand, the observed difficulty of learning these generalizations lends support to theories of phonological change in which mistakes in acquisition lead to the expansion or restructuring of a feature inventory @xcite . by looking at historical corpus of old french, we can observe how the lexicon evolves over time changing the frequency of different vowel-consonant combinations. the fact that changes in frequency result in this kind of change for our model is evidence that this mechanism is plausible, and offers a route to testing its explanatory power for specific historical hypotheses in the future. although the long-term goal of this research is understanding how phonological representation learning works for a variety of models and phenomena, we believe it is necessary to start small, with the treatment of one particular phenomenon. in text linguistics, there are now established benchmarks for understanding linguistic representation in language models, for example, the benchmark of linguistic minimal pairs (blimp) @xcite , but in speech linguistics, we are lagging behind. even doing studies of an individual phenomenon requires identifying a phonological phenomenon, extracting and labeling a corpus and conducting a study of the model's learning behavior. a diverse and comprehensive benchmark dataset for studying phonological learning (beyond phoneme segmentation and categorization) would be an exciting goal for future work.