Acoustic barycenters as exemplar production targets (2024)

Abstract: We present a solution to the problem of exemplar-based language production from variable-duration tokens, leveraging algorithms from the domain of time-series clustering and classification. Our model stores and outputs tokens of phonetically rich and temporally variable representations of recorded speech. We show qualitatively and quantitatively that model outputs retain essential acoustic/phonetic characteristics despite the noise introduced by averaging, and also demonstrate the effects of similarity and indexical information as constraints on exemplar cloud selection.

## introduction we present here an exemplar production model that implements solutions to the challenges of measuring between-exemplar distance (i.e. alignment) and fostering phonetic generalization over speech tokens of variable duration @xcite ). 1 our model, mnemorphon, makes use of algorithms for alignment and averaging the domain of time-series clustering and classification. we show qualitatively by direct inspection of model outputs and quantitatively via statistical classification that mnemorphon's outputs retain essential acoustic/phonetic characteristics, despite noise introduced by averaging, and also demonstrate the effects constraining exemplar cloud composition by means of similarity weighting and indexical information. we begin with an overview of exemplar-based approaches to phonetics and phonology, highlighting the core production challenges of temporal variability and generalization. we then introduce mnemorphon and present our experiments and results, and finish with some discussion of planned work and future directions. foot_1 ## exemplar-based phonetics and phonology exemplar-based theories of categorization propose that humans classify percepts by direct comparison to memorized exemplars of previous experiences @xcite @xcite , whereas linguistic theories have traditionally been couched in terms of symbolic categories that abstract away from details of usage and experience. when experiments in speech perception suggested that human word recognition is facilitated by fine details of remembered experiential episodes, e.g. speakers' voices @xcite , phoneticians began to explore the possibilities of memory-based approaches. @xcite presented a pair of exemplar models of phonetic perception that provided elegant and novel accounts of speaker normalization and speech segmentation. soon after @xcite published the first implemented model of exemplar-based phonological production, in the context of a productionperception feedback-loop model of sound change. these initial investigations ushered in a flurry of subsequent research in exemplar-theoretic phonetics and phonology in areas as diverse as sound change, categorical emergence and entrenchment, sociophonetic variation, frequency effects in productivity, the status of abstract phonetic categories, and the induction of morphophonological alternations @xcite @xcite @xcite @xcite @xcite . @xcite provide a recent overview of the theoretical and empirical successes, along with some outstanding potential challenges, of exemplar-based approaches to production. the core theoretical challenges faced by exemplar-based models of production are handling input variability, particularly with respect to temporal variation, and the need for a mechanism for robust generalization from prior experiences. below we discuss the first of these, showing how it can be surmounted with a 50 year old approach to speech recognition, and later we address the latter, introducing a 21 st century algorithm for averaging time series. ## mnemorphon: a bit of progress in exemplar-based production 6 any implemented exemplar model must minimally include tokens of some primitive linguistic unit encoded in a suitable representational format, associated category labels, and a means of computing analogically relevant similarity between exemplars @xcite . for pierrehumbert's model discussed above, these are segments (specifically vowels), the space defined by tuples of the first three formants, and inverse euclidean distance in formant space. for mnemorphon these architectural parameters are as follows: our general task can now be framed as follows: given a seed exemplar and a cloud of tokens of possibly varying lengths from a given category, we seek a procedure by which we can generate an output exemplar as an "average" of the cloud. as it happens, exactly computing the sample mean of a set of sequences with potentially differing lengths corresponds to solving the problem of multiple sequence alignment, which is known to be computationally intractable @xcite . notwithstanding this, there are tractable approximation methods that are theoretically justifiable and empirically suitable; @xcite introduce one such approach, dtw barycenter averaging (dba). ## data for the experiments described below our raw data set is an audio corpus of turkish speech, consisting of microphone recordings (16khz sample rate) 7 mnemorphon uses the implementation of dba available in the tslearn python package @xcite . from 120 speakers (balanced across binarized gender categories; age 19-50 years, @xmath0.9) who each read 40 sentences sampled from a triphonebalanced set of 2462 turkish sentences @xcite . metadata for each speaker includes (binarized) gender, dates of birth and recording, places of birth and residence, and level of education. inspection revealed a subset (@xmath1. these were filtered out, leaving 97 speakers (@xmath2, @xmath3. each recorded sentence is transcribed in standard turkish orthography as well as an asciicompatible phonemic orthography derived from sampa @xcite , called metubet (özgül @xcite . the corpus also includes wordlevel, phone-level, and hmm state alignments, computed with an hmm-gmm acoustic model trained on a subset of the full set of sentences. as with most linguistic corpora, word frequencies follow a roughly zipfian distribution. there are 7412 words in our dataset, the most frequent of which, bir ("one/a"), occurs approximately 897 times, whereas there are 2423 words which occur only once. ## experiment 1: cloud composition in our initial experiments we explore the effect of cloud composition on mnemorphon's outputs. we begin with a maximally unconstrained approach, conditioning cloud selection solely on word category membership. for each of the word categories (i.e. distinct metubet strings) represented in our corpus, we uniform randomly select one token as the seed exemplar and sample progressively larger uniform random subsets of the remaining tokens from the category as the cloud from which mnemorphon computes a barycenter. we illustrate the outcome here in figures 3 and 4 for a representative example, the form belki (gloss: "maybe", corpus freq: 40, rank: 43). we plot full spectrograms and a selection of mel bands, respectively, for the seed token along with from mnemorphon's output barycenter for varying cloud sizes. figure 3 shows clearly that increasing the number of tokens included in the cloud results in mnemorphon's output spectrograms becoming "blurrier", losing most of the fine structure present in individual tokens, particularly with respect to frequency information. notwithstanding this noise, the individual mel bands plotted in figure 4 show that mnemorphon's generation algorithm does find meaningful averages for temporally variable signals, locating and aligning the major peaks and troughs in the energy for each band along the temporal dimension. the relation between output noise and increasing numbers of cloud exemplars found here is not solely due to cloud size, but rather that the clouds' tokens are dispersed in the parametric space. to confirm this we generate outputs from the same seed, this time with two small clouds of the same size (@xmath4, constrained to contain the maximally similar and dissimilar tokens in the category, respectively. the outputs, shown in figure 5 , confirm that dispersion plays a key role in the quality of mnemorphon's generated forms. this in turn raises the question of latent categorical structure or organization within exemplars clouds. 8 as discussed in section 2, some of the early motivation for exploration of exemplar-based speech processing was the apparent storage and use of nonlinguistic information, for example indexical information. the results above suggest that constraining mnemorphon's cloud selection by using any such additional contextually available information would likely serve to further reduce the output variance, resulting in cleaner, and in a sense more representative, output spectrograms. to test this we re-ran the same experiment as above, with the same seed token, this time constraining mnemorphon's clouds by using the (binarized) gender information that is available in our corpus. we used the output from our initial, unconstrained, experiment for a cloud size @xmath5, and then used ten uniform randomly selected tokens from the relevant category that were tagged f ("female") in our corpus. once again, we see in figure 6 that constraining mnemorphon's cloud along dimensions of similarity, linguistic or otherwise, yields cleaner, more representative outputs. notwithstanding the obscuring or blurring of phonetic detail in mnemorphon's outputs, larger scale patterns of energy distribution across different frequency bands and time slices remain visible, hinting at an emergent, transient form of abstraction; a hallmark of exemplar models. in our next experiment we see that there is indeed linguistic categorical information recoverable from these outputs. in addition to the direct visual evaluation here, we use a publicly available pre-trained neural vocoder (lee et al., 2023) 9 to re-synthesize audio from our generated spectrograms for impressionistic auditory evaluation. 10 it is the use of this vocoder that constrained the spectrogram parameters in our data preparation; because bigvgan is trained on narrow-band spectrograms (the standard choice in neural text-to-speech synthesis), these are required for any subsequent synthesis. that said, the finer frequency resolution of narrow-band spectrograms is likely beneficial for the quantitative evaluation in experiment 6. ## experiment 2: latent categorical information in mnemorphon's outputs we have seen that mnemorphon's outputs quickly become noisy as a function of cloud size, although this is somewhat mitigated by heavily weighting the influence of cloud tokens that are close to the seed in dtw distance. despite this noise, we wish to determine whether generated outputs retain any categorically characteristic phonetic signal. we investigate this in the present experiment, in which we train a neural network to take spectral slices as inputs and classify them as front or back vowels. our focus on this particular phonetic characteristic, foreshadows work in progress assessing mnemorphon's ability to learn productive morphophonological generalizations, in particular turkish front/back vowel harmony. 11 although mnemorphon itself has no notion of sub-lexical units, they are useful in the context of this extrinsic analysis. for this experiment, we extracted all vowels from the audio corpus using the included alignments, and converted them directly to mel spectrograms, resulting in a total of 82360 samples, which were randomly shuffled and divided via stratified split into train, development, and test sets representing 80, 10, and 10 percent of the corpus samples. our classifier is a convolution neural network. they are known to perform well on spectrograms and in fact form the backbone of many current speech recognition systems @xcite . our network has 4 layers of 2-d convolutions (5x5 in the first layer and 3x3 for subsequent layer), a max-pooling layer, and a final fully-connected layer projecting to a binary output (modeling [± back]). kernel sizes, learning rate and batch size were tuned on a development split; the final training run was for 25 epochs. ## discussion we have shown here that dynamic time warping and dba barycenter averaging together constitute a viable basis for a production algorithm in an exemplar model, mnemorphon, whose token representations are word-sized mel spectrograms of variable durations, overcoming a core challenge for exemplar production models. we showed both qualitatively and quantitatively that despite noise introduced by averaging over tokens that are dispersed in spectrotemporal space, our model's outputs retain phonetic properties that are characteristic of the exemplars from the generating categories. ## limitations and future work mnemorphon's production algorithm as applied in these experiments generates comparatively noisy outputs, unless the selection of tokens for the exemplar cloud is severely constrained. nonetheless, we see this work as an initial step toward a fully articulated theory and model of exemplarbased (psycho)linguistic knowledge. an eventual goal is to assess how far such a "pure" or "core" model can take us before a hybrid approach becomes necessary (cf. @xcite . in future work will explore further restrictions on cloud construction, exploring e.g. speaker identity, dialect, and speech rate among others. as hinted in section 6, we also intend to extend this work to account for productive morphophonological alternations like turkish vowel harmony (see @xcite , for an exemplar production approach that learns productive vowel harmony on toy data, including patterns of opaque and transparent neutrality), and eventually to data from psycholinguistic research on speech perception and production (e.g. contexts of phonetic reduction and lengthening, and patterns of interlocutor convergence). as the data used here are not widely accessible, we also intend to reproduce these results in the not-too-distant future using data from the mozilla common voice corpus @xcite foot_11 in order to facilitate reproducibility.