Identity-Based Patterns in Deep Convolutional Networks: Generative Adversarial Phonology and Reduplication (2021)

Abstract: This paper models unsupervised learning of an identity-based pattern (or copying) in speech called reduplication from raw continuous data with deep convolutional neural networks. We use the ciwGAN architecture (Beguš, 2021a) in which learning of meaningful representations in speech emerges from a requirement that the CNNs generate informative data. We propose a technique to wug-test CNNs trained on speech and, based on four generative tests, argue that the network learns to represent an identity-based pattern in its latent space. By manipulating only two categorical variables in the latent space, we can actively turn an unreduplicated form into a reduplicated form with no other substantial changes to the output in the majority of cases. We also argue that the network extends the identity-based pattern to unobserved data. Exploration of how meaningful representations of identity-based patterns emerge in CNNs and how the latent space variables outside of the training range correlate with identity-based patterns in the output has general implications for neural network interpretability.

## introduction the relationship between symbolic representations and connectionism has been subject to ongoing discussions in computational cognitive science. phonology offers a unique testing ground in this debate as it is concerned with the first discretization that human language users perform: from continuous phonetic data to discretized mental representations of meaning-distinguishing sounds called phonemes. identity-based patterns, repetition, or copying have been at the center of this debate @xcite . reduplication is a morphophonological process where phonological content (phonemes) get copied from a word (also called the base) with some added meaning @xcite . it can be total, which means that all phonemes in a word get copied (e.g., /pula/ → [pula-pula]), or partial, where only a subset of segments gets copied (e.g., /pula/ → [pu-pula]). reduplication is indeed unique among processes in natural language because combining learned entities based on training data distributions does not yield the desired outputs. for example, a learner can be presented with pairs of bare and reduplicated words, such as /pala/ ∼ /papala/ and /tala/ ∼ /tatala/. the learner can then be tested on providing a reduplicated variant of a novel unobserved item with an initial sound /k/ that they have not been exposed to (e.g., /kala/). if the learner learns the reduplication pattern, they will output @xcite . if the learner simply learns that /pa/ and /ta/ are optional constituents that can be attached to words based on data distribution, they will output @xcite or @xcite . reduplication is thus an identity-based pattern (similar to copying), which is computationally more challenging to learn @xcite , both in connectionist @xcite and non-connectionist frameworks @xcite . in /k i a j k i a j la/, the two sounds in the reduplicative morpheme, /k i / and /a j /, need to be in an identity relationship with the first two segments of the base, /k i / and /a j /, and the learner needs to copy rather than recombine learned elements. @xcite argue that connectionist models such as simple neural networks are unable to learn a simple reduplication pattern that 7-month old human infants are able to learn (see also @xcite . according to @xcite , the behavioral outcomes of their experiments cannot be modeled by simple counting, attention to statistical trends in the input, attention to repetition, or connectionist (simple neural network) computational models. instead, they argue, the results support the claim that human infants use ''algebraic rules'' @xcite @xcite to learn reduplication patterns (for a discussion, see, among others, @xcite . with the development of neural network architectures, several studies revisited the claim that neural networks are unable to learn reduplicative patterns @xcite @xcite , arguing that identity-based patterns can indeed be learned with more complex architectures. 1 all these computational experiments, however, operate on an already discretized level and most of these experiments model reduplication with supervised learning. examples like [pu-pula] and [pula-pula] are discretized representations of reduplication, using characters to represent continuous sounds. most, if not all, computational models of reduplication, to the author's knowledge, model reduplication as character or feature manipulation (the inputs to the models are either characters representing phones or phonemes or discrete binary featural representations of phonemes). for example, a seq2seq model treats reduplication as a pairing between the input unreduplicated sequence of ''characters'' (such as /tala/) and an output-a reduplicated sequence (such as /tatala/). already abstracted and discretized phonemes or ''characters'', however, are not the primary input to language-learning infants. the primary linguistic data of most hearing infants is raw continuous speech. hearing infant learners need to acquire reduplication from continuous speech data that is substantially more complex than already discretized characters or binary features. furthermore, most of the existing models of reduplication are also supervised. seq2seq models, for example, are fully supervised: the training consists of pairs of unreduplicated (input) and reduplicated strings of characters or binary features (output). while the performance can be tested on unobserved data or even on unobserved segments, the training is nevertheless supervised. human language learners do not have access to input-output pairings: they are only presented with positive, surface, and continuous acoustic data. while equivalents of copying/identity-based patterns can be constructed in the visual domain, we are not aware of studies that test identity-based visual patterns with deep convolutional neural networks. in this paper, we model reduplication, one of the computationally most challenging processes, from raw unlabeled acoustic data with deep convolutional networks in the gan framework. the advantage of the gan framework for cognitive modeling is that the network has to learn to output raw acoustic data from a latent noise distribution without directly accessing the training data. we argue that cnns discretize continuous phonetic data and encode linguistically meaningful units into individual latent variables. the emergence of a discretized representation of an identity-based pattern (reduplication) is induced by a model that forces the generator network to output informative data (ciwgan; section 4). additionally, we add inductive bias towards symbolic-like representations by binarizing code variables with which the generator encodes meaningful representations. we also test whether a deep convolutional network learns reduplication without the two inductive biases (without the requirement on the generator to output informative data and without binarization of the latent space) in the bare wavegan architecture (section 5). the experiments bear implications for the discussion between symbolic and connectionist approaches to language modelling by testing the emergence of rule-like symbolic representations within the connectionist framework from raw speech in an unsupervised manner. results of the experiments suggest that both models, ciwgan and wavegan learn the identity-based patterns, but inductive biases for informative representation and binarization facilitate learning and yield better results. we discuss properties of symbolic-like representations and how they emerge in the models: discretization, causality (in the sense that manipulation of individual elements results in desired outcome), and categoricity (for discussions on these and other aspects of the debate, see @xcite @xcite @xcite @xcite @xcite . how can we test learning of reduplication in a deep convolutional network that is trained only on raw positive data? we propose a technique to test the ability of the generator to produce forms absent from the training data set. for example, we train the networks on acoustic data of items such as /pala/ ∼ /papala/ and /tala/ ∼ /tatala/, but test reduplication on acoustic forms of items such as /sala/, which is never reduplicated in the training data. using the technique proposed in @xcite , we can identify latent variables that correspond to some phonetic or phonological representation such as reduplication. by manipulating and interpolating a single latent variable, we can actively generate data with and without reduplication. in fact, we can observe a direct relationship between a single latent variable (out of 100) and reduplication that with interpolation gradually disappears from the output. additionally, we can identify latent variables that correspond to [s] in the output. by forcing both reduplication and [s] in the output through latent space manipulation, we can ''wug-test'' the network's learning of reduplication on unobserved data. in other words, we can observe what the network will output if we force it to output reduplication and an [s] at the same time. a comparison of generated outputs with human outputs that were withheld from training reveals a high degree of similarity. we perform an additional computational experiment to replicate the results from the first experiment (from section 4). in the replication experiment, evidence for learning of the reduplicative pattern also emerges. to the author's knowledge, this is the first attempt to model reduplication with neural network architectures trained on raw acoustic speech data. the computational experiments reveal another property about representation learning in deep neural networks: we argue that the network extracts information in the training data and represents a continuous acoustic identity-based pattern with discretized representation. out of 100 variables, the network encodes reduplication with one or two variables, which is suggested by the fact that a small subset of variables are substantially more strongly correlated with presence of reduplication. in other words, there is a near categorical drop in regression estimates between one variable and the rest of the latent space. setting the identified variables to values well beyond the training range results in near categorical presence of a desired variable in the output. this technique (proposed for non-identity-based patterns in beguš, 2020) allows us to directly explore how the networks encode dependencies in data, their underlying values, and interactions between variables, and thus get a better understanding of how exactly deep convolutional networks encode meaningful representations. recent developments in zero-resource speech modeling @xcite @xcite enable modeling of speech processes in an unsupervised manner from raw acoustic data. several proposals exist for modeling unsupervised lexical learning @xcite @xcite that include generative models such as variational autoencoders @xcite @xcite and gans @xcite . this framework allows not only unsupervised lexical term discovery, but also phone-level identification @xcite @xcite . while zero-resource speech modeling has yielded promising results in unsupervised labeling, the proposals generally do not model phonological or morphophonological processes. this paper thus also tests applicability of the unsupervised speech processing framework for cognitive modeling and network interpretability. in the first experiment, we use the ciwgan (categorical infowavegan) model proposed in @xcite . the ciwgan model combines the wavegan and infogan architectures. wave-gan, proposed by @xcite , is a deep convolutional generative adversarial network (dcgan; proposed by @xcite adapted for time-series audio data. the basic architecture is the same as in dcgan, the main difference being that in the wavegan proposal, the deep convolutional networks take onedimensional time-series data as inputs or outputs. the structure of the generator and the discriminator networks in the ciwgan architecture are taken from @xcite . infogan @xcite is an extension of the gan architecture that aims to maximize mutual information between the latent space and generated outputs. the discriminator/q-network learns to retrieve the generator's latent categorical or continuous codes @xcite in addition to estimating realness of generated outputs and real training data. @xcite proposes a model that combines these two proposals and introduces a new latent space structure (in the fiwgan architecture). because we are primarily interested in simple binary classification between bare and reduplicated forms, we use the ciwgan variant of the proposal. the model introduces a separate deep convolutional q-network that learns to retrieve the generator's internal representations. separating the discriminator and the q-network into two net-works is advantageous from the cognitive modeling perspective: the architecture features a separate network that models speech production (the generator) and a separate network that models speech categorization (the q-network). the latter introduces an inductive bias that forces the generator to output informative data and encode linguistically meaningful properties into its code variables. the network learns to generate data such that by manipulating these code variables, we can force the desired linguistic property in the output @xcite . the architecture involves three networks: the generator that takes latent codes (a one-hot vector) and uniformly distributed z-variables and generates waveforms, a discriminator that distinguishes real from generated outputs, and a q-network that takes generated outputs and estimates the latent code (one-hot vector) used by the generator. more specifically, the generator network is a deep convolutional network that takes as its input 100 latent variables (see figure 1 ). 2 two of the 100 variables are code variables (c 1 and c 2 ) that constitute a one-hot vector. the remaining 98 z-variables are uniformly distributed on the interval (-1, 1). the generator learns to take as the input the 2 code variables and the 98 latent variables and output 16,384 samples that constitute just over one second of audio file sampled at 16 khz through five convolutional layers. the discriminator network takes real and generated data (again in the form of 16,384 samples that constitute just over one second of audio file) and learns to estimate the wasserstein distance between generated and real data (according to the proposal in @xcite through five convolutional layers. in the majority of infogan proposals, the discriminator and the q-network share convolutions. beguš (2021a) introduces a separate q-network (also in rodionov 2018). 3 the q-network is in its structure identical to the discriminator network, but the final layer is fully connected to nodes that correspond to the number of categorical variables @xcite . in the ciw-gan architecture, the q-network is trained on estimating the latent code variables with a softmax function @xcite . in other words, the q-network takes the generator's outputs (waveforms) and estimates the generator's latent code variables c 1 and c 2 . weights of both the generator network and the q-network are updated according to the q-network's loss function: to minimize the distance between the actual one-hot vector (c 1 and c 2 ) used by the generator and the one-hot vector estimated with a softmax in the q-network's final layer using cross-entropy. this forces the generator to output informative data. the advantage of the ciwgan architecture is that the network not only learns to output innovative data that resemble speech in the input, but also provides meaningful representations about data in an unsupervised manner. for example, as will be argued in section 4, the ciwgan network encodes reduplication as a meaningful category: it learns to assign a unique code for bare and reduplicated items. this encoding emerges in an unsupervised fashion from the requirement that the generator output data such that unique information is retrievable from its acoustic outputs. given the structure of the training data, the generator is most informative if it encodes presence of reduplication in the code variables. to replicate the results and to test learning of an identity-based pattern without binarization and without the requirement on the generator to output informative data, we run an independent experiment on a bare wavegan @xcite architecture using the same training data. the difference between the two architectures is that the bare gan architecture does not involve a q-network and the latent space only includes 3 for all details about the architecture, see @xcite . ## reduplication in training data the training data was constructed to test a simple reduplication pattern, common in human languages: partial cv reduplication found in languages such as paamese, roviana, tawala, among others @xcite . base items are of the shape c 1 v 2 c 3 v 4 (@xmath0; @xmath1; e.g., /tala/). reduplicated forms are of the shape c 1 v 2 c 1 v 2 c 3 v 4 , where the first syllable (c 1 v 2 ) is repeated. the items were constructed so that c 1 contains a voiceless stop /p, t, k/, a voiced stop /b, d, g/, a labiodental voiced fricative /v/, and nasals /m, n/. the vowels v 2 and v 4 consist of /a (@), i, u/. c 3 consists of /l, ô, j/. all permutations of these elements were created. the stress was always placed on v 2 in the base forms and on the same syllable in reduplicated forms (["p h al@] ∼ [p@"p h al@]). because the reader of the training data was a speaker of american english, the training data is phonetically even more complex. the major phonetic effects in the training data include (i) reduction of the vowel in the unstressed reduplicated forms and in the final syllable (e.g., from [a] to [2/@]) and (ii) deaspiration of voiceless stops in the unstressed reduplication syllable (e.g., from [p h ] to [p]). the training data includes two unique repetitions of each item and two repetitions of the corresponding reduplicated forms. the training data also includes base forms c 1 v 2 c 3 v 4 with the initial consonant c 1 being a fricative @xcite . these items, however, always appear unreduplicated in the training data-the purpose of [s]-initial item is to test how the network extends the reduplicative pattern to novel unobserved data. all 27 permutations of sv 2 c 3 v 4 were included. to increase representation of [s]-initial words, four or five repetitions of each unique [s]-initial base were used in training. 4 altogether, 132 repetitions of the 27 unique unreduplicated words with an initial [s] were used in training. sibilant fricative [s] was chosen as c 1 for testing learning of reduplication because its frication noise is acoustically prominent and sufficiently different from c 1 s in the training data both acoustically and phonologically. this satisfies the requirement that a model learns to generalize to novel segments and feature values @xcite . 5 in phonological terms, the model is tested on a novel feature (sibilant fricative or [±strident]; hayes 2009)-the training data did not consist of any bare or reduplicated forms with other sibilant fricatives. to make the learning even more complex, voiceless fricatives ([f, t, s]) are altogether absent from the training data. all voiced fricatives except for [v] are absent too. spectral properties of the voiced non-sibilant fricative @xcite in the training data (and in standardized american english in general) are so substantially different from a voiceless sibilant fricative [s] that we kept them in the training data. we excluded all items with initial sequences /ti/, /tu/, and /ki/ from the training data, because acoustic properties of these sequences, especially frication of the aspiration of /t/ and /k/, are similar to those of frication noise in /s/. altogether 996 unique sliced items used in training were recorded in a sound attenuated booth by a female speaker of american english with a mixpre 6 (sounddevices) preamp/recorder and the akg c544l head-mounted microphone. 5 for an ''across the board'' generalization, berent (2013) requires that generalization occur to segments fully absent from the inventory. it is challenging to elicit reduplication of segments that are fully absent from the training data in the proposed models. even in human subject experiments testing the ''across the board'' generalization, subjects need to be exposed to the novel segment at least as a prompt. in our case, the novel segment needs to be part of the training data, but only in unreduplicated forms. to test whether the generator encodes reduplication in latent codes, we train the network for 15,920 steps (or approximately 5,114 epochs) with the data described in section 3. the choice of the number of steps is based on two objectives; first, the output data should approximate speech to the degree that allows acoustic analysis. second, the generator network should not be trained to the degree that it replicates data completely. as such, overfitting rarely occurs in the gan architecture @xcite . the best evidence against overfitting in the ciwgan architecture comes from the fact that the generator outputs data that violate training distributions substantially (see section 4.2 below) @xcite . despite these guidelines, the choice of number of steps is somewhat arbitrary (for discussion, see @xcite . ## discussion we perform four generative tests to model learning of reduplication in deep convolutional networks: (i) a test of proportion of outputs when latent codes are manipulated to marginal values, (ii) a test of interpolating latent variables, (iii) a test of reduplication on unobserved data in the ciwgan architecture, and (iv) a replication test of reduplication on unobserved data in the bare wavegan architecture. all four tests suggest that deep convolutional networks learn a simple identity-based pattern in speech called reduplication, that is, a process that copies some phonological material to express new meaning. the ciwgan network learns to encode a meaningful representationpresence of reduplication into its latent codes. there is a near one-to-one correspondence between the two latent codes c 1 and c 2 and reduplication. by interpolating latent codes, we cause the bare form to gradually turn into a reduplicated form with no other major changes in the output in the majority of cases. these results are close to what would be considered appearance of symbolic computation or algebraic rules. additional evidence that an approximation of symbolic computation emerges comes from the bare gan experiment: there is a substantial drop in regression estimates after the first one or two latent variables with highest regression estimates, suggesting that even without the requirement to produce informative data, the network discretizes the continuous and highly variable phonetic feature (presence of reduplication) and uses a small subset of the latent space to represent this morphophonological property. finally, we can force the generator to output reduplication at nearly categorical levels. when latent codes are set to marginal levels outside of training range (e.g., to [5, 0] or [0, 5]), the outputs are almost categorically unreduplicated or reduplicated (at 98% for [5, 0]). @xcite shows that even higher values (e.g., 15) result in performance at 100% for a subset of variables. not all aspects of the models in this paper are categorical (e.g., interpolation of latent codes does not always change an unreduplicated to a reduplicated form without other major changes). improving performance on this particular task is left for future work. inability to derive categorical processes has long been an argument against the connectionist approaches to language modeling. the results of this experiments add to the work suggesting that manipulating variables to extreme marginal values results in near categorical or categorical outputs (depending on the value) of a desired property @xcite . in sum, three properties of rule-like symbolic representations emerge in deep convolutional network tested here: discretized representations, the ability to generate desired property by manipulating a small number of variables, and near categoricity for a subset of representations. these symbolic-like outcomes are facilitated by two inductive biases: the binary nature of latent codes and the requirement on the generator to output informative data (forced by the q-network). at least a subset of these properties also emerges in the bare wavegan architecture that lacks these biases, but at a reduced performance. encoding an identity-based pattern as a meaningful representation in the latent space emerges in a completely unsupervised manner in the ciwgan architecture-only from the requirement that the generator output informative data. reduplicated and unreduplicated forms are never paired in the training data. the network is fed bare and reduplicated forms randomly. this unsupervised training approximates conditions in language acquisition (for hearing learners): the human language learner needs to represent reduplication and to pair bare and reduplicated forms from raw unlabeled acoustic data. the ciwgan learns to group reduplicated and unreduplicated forms and assign a unique representation to the process of reduplication. in fact, the one-hot vector (c 1 and c 2 ) that the generator learns to associate with reduplication in training can be modeled as a representation of the unique meaning/function that reduplication adds, in line with an approach to represent unique semantics with one-hot vectors (e.g., in @xcite . the paper also argues that deep convolutional networks can learn a simple identity-based pattern (copying) from raw continuous data and extend the pattern to novel unobserved data. while the network was not trained on reduplicated items that start with an [s], we were able to elicit reduplication in the output following a technique proposed in @xcite . first, we identify variables that correspond to some phonetic/phonological representation such as presence of @xcite . we argue that setting single variables well above training range can reveal the underlying value for each latent variable and force the desired property in the output. we can thus force both [s] and reduplication in the output simultaneously. for example, the network outputs [s@siju] if we force both reduplication and [s] in the output; however, it never sees @xcite in the training data-only [siju] and other reduplicated forms, none of which included an @xcite . thus, these experiments again confirm that the network uses individual latent variables to represent linguistically meaningful representations @xcite . setting these individual variables to values well above the training interval reveals their underlying values. by manipulating these individual variables, we can explore how the representations are learned as well as how interactions between different variables work (for example, between the representation of reduplication and presence of @xcite ). the results of this study suggest that the deep convolutional network is not only capable of encoding different phonetic properties in individual latent variables, but also processes as abstract as copying or reduplication. one of the advantages of probing learning in deep convolutional neural networks on speech data trained with gans is that the innovative outputs violate training data in structured and highly informative ways. the innovative outputs with reduplication of [s]-initial forms such as [s@siju] can be directly paralleled to acoustic outputs read by l1 speaker of american english that were absent from the training data. acoustic analysis shows a high degree of similarity between the generated reduplicated forms and human recordings, meaning that the network learns to output novel data that are linguistically interpretable and resemble human speech processes even though they are absent from the training data. thus, the results of the experiments have implications for cognitive models of speech acquisition. it appears that one of the processes that has long been held as a hallmark of symbolic computation in language, reduplication, can emerge in deep convolutional network without language-specific components in the model even when they are trained on raw acoustic inputs. the present paper tests a simple partial reduplicative pattern where only cv is copied and appears before the base item. this is perhaps computationally the simplest reduplicative pattern. the training data are also highly controlled and recorded by a single speaker. we can use the well-understood identity-based patterns in speech with various degrees of complexity (longer reduplication, embedding into non-reduplicative patterns) to further test how inductive biases and hyperparameter/architecture choices interact with learning in deep convolutional networks. finally, learning biases in the ciwgan model can be (superficially) compared to learning biases in human subjects in future work. this paper suggests that the generator provides informative outputs even if trained on comparatively small data sets (for a similar conclusion for other processes, see @xcite . this means we can use the same training data to probe learning in cnns and in human artificial grammar learning experiments (for a methodology, see @xcite . although these comparisons are necessarily superficial at this point, they can provide insights into common learning biases between human learners and computational models.