Analyzing the Representational Geometry of Acoustic Word Embeddings (2022)

Abstract: Acoustic word embeddings (AWEs) are fixed-dimensionality vector representations in a vector space such that different acoustic exemplars of the same word are projected nearby in the embedding space. In addition to their use in speech technology applications such as spoken term discovery and keyword spotting, AWE models have been adopted as models of spoken-word processing in several cognitively motivated studies and they have shown to exhibit a human-like performance in some auditory processing tasks. Nevertheless, the representation geometry of AWEs remains an under-explored topic that has not been studied in the literature. In this paper, we take a closer analytical look at AWEs and study how the choice of the learning objective and the architecture shapes their representational profile. Our main findings highlight the prominent role of the learning objective on the representational geometry over the architecture.

## introduction due to their ubiquity, word embeddings are nowadays a central component in natural language processing (nlp). inducing word embeddings from text yields representations such that words occurring in similar contexts are nearby in the vector space @xcite . therefore, the representational geometry of text-based word embeddings captures lexical similarity and semantic relatedness at multiple levels of granularity. word embeddings, and their underlying distributional semantic models, have also been adopted as models of human semantic memory in cognitive science research @xcite @xcite . in the speech processing domain, researchers have independently developed representations of acoustic segments that correspond to linguistic units @xcite @xcite , inter alia). a notable example of such representations are acoustic word embeddings (awes)vector representations that encode the sound structure of words, not their semantic and syntactic structure-see fig. 1 . awes support voice-based speech technology applications such as query-byexample spoken term discovery @xcite @xcite and keyword spotting @xcite . in addition, awes can be leveraged to facilitate access to speech recordings of endangered spoken languages that might lack standardized writing systems @xcite however, there are fundamental differences be-tween text-based and speech-based word embeddings that have to do with the degree of variability between the two modalities. contrary to written words which have context-invariant orthographic realizations, foot_0 spoken words are notoriously variable. the underlying sources of variability in speech include speaker-related factors such as vocal tract shape, gender, age, and dialect. in addition, two acoustic instances, or exemplars, of the same word will vary in different phonological and semantic contexts even if they are produced by the same speaker @xcite . therefore, acoustic word embeddings are not static, but have to be computed "on the fly" given a speech segment as input. models of awes need to abstract away from speaker and context variability to project different acoustic exemplars of the same word onto (ideally) the same point of the embedding space. nevertheless, awes have not yet been extensively studied in the literature from a neural network interpretability point of view. we are only aware of a few prior efforts in this direction that have either analyzed the representational geometry of awes from a cognitively motivated angle @xcite or from a cross-linguistic perspective @xcite . in this paper, we make a contribution in this direction and use analytic techniques from machine learning and neuroscience in three different analytic studies: embedding space uniformity ( §4), word discriminability ( §5), and representational consistency ( §6). ## acoustic word embedding models given an acoustic signal that corresponds to a spoken word represented as a temporal sequence of t acoustic feature vectors, i.e., @xmath0, a 2 , ..., a t ), the goal of an awe model is to transform a into a fixed-dimensionality vector representation e. due to the variability in speech production (i.e., speech rate, emotional state, etc), the length of the acoustic segment t varies between different exemplars, or instances, of the same word type. therefore, this task is modeled as a mapping f : a -→ r d , where a is the (continuous) space of acoustic sequences and d is the dimensionality of the embedding. formally, transforming a variable-length acoustic input into a d-dimensional awe is described as ## analysis 1: embedding space uniformity in our first analysis, we take a closer look at how uniform are representational spaces of awe models by analyzing the distribution of cosine similarity for each model type and the degree to which the embeddings are isotropic. ## analysis 2: word discriminability ideally, awe models should project exemplars of the same word category onto the same point in the embedding space. however, there are no strong constraints during training to encourage maximal separability between different word categories. in this analysis, we seek to answer two questions: ## analysis 3: network representational consistency suppose we train two instances of the same architecture and learning objective on the same training samples, but each with different random initializations. do these two neural network instances exhibit differences in their representational geometries? in this section, we shed light on the representational discrepancies caused by different initializations. in other words, we are interested in quantifying the degree to which variability in the initial conditions affects the way two models sepa-rate the same set of speech samples. ## discussion acoustic word embeddings (awes) are vector representations that encode the sound structure and acoustic-phonetic features of spoken words. awes are induced from actual acoustic realizations of speech, and therefore awe models have to abstract away from non-linguistic dimensions of variability in speech signals (e.g., speaker characteristics, speech rate, recording conditions, etc). while analyzing the representational geometry of semantic word embeddings is a topic that has received a substantial attention in the nlp research community, the interpretability of awes remains an under-explored topic and we are aware of a few prior studies in this direction @xcite . in this article, we made a number contributions in analyzing the representational geometry of awes and obtained research findings which we discuss and summarize in this section. learning objective affects the geometry more than architecture. our three analyses in this paper have shown that the learning objective shapes the representational geometry of the awe encoders more than their underlying architectures. this finding suggests that recurrent and convolutional encoders exhibit similar inductive biases while the learning process is mainly guided by the loss function. awe models tend to be anisotropic. our analysis in §4 has shown that awes tend towards being minimally isotropic. however, this behavior is not an emergent property of the training process, but rather an inherent behavior of the neural network. moreover, the degree of isotropy after training the model positively correlates with the acoustic word discrimination evaluation task. since different models have different degree of isotropy and the representation space is not always uniform, we conclude that any comparison between different models based on absolute distance metrics such cosine distance will definitely lead to inaccurate observations. word distinctiveness, but not frequency, predicts category discriminability. while word acoustic distinctiveness has been found in §5 to be a good predictor of the degree to which a word category is compact and well-separated in the embedding space, word frequency does not correlate with category discriminability. in retrospective, this finding should not be surprising as frequent words tend to have shorter lengths. shorter words have more phonological neighbours that are perceptually similar in form and thus they are more confusable with other words. future work could employ more sophisticated linear mixed effects models to analyse the interaction between different lexical properties such as frequency, phonological neighbourhood density, and word length and their effect on word category discriminability. awe models exhibit individual differences. although awe model instances trained with different random initializations are stable with respect to the performance of the evaluation task, they exhibit individual differences in their representational profiles as shown in §6. however, the degree of the network representational consistency across different initializations depends on both the architecture and the learning objective. contrastive objectives are less consistent than decoding-based objectives, while recurrent encoder are less consistent than their convolutional counterparts. ## conclusion in this paper, we have taken a closer, analytical look at the representational geometry of acoustic word embeddings (awes) from three different, but complementary perspectives: (1) embedding space uniformity, (2) word discriminability, and (3) network representational consistency. we have shown that the representational spaces of awes tend to-wards being minimally isotropic, or in other words, they utilize only a few dimensions of the embedding space. another finding was that most awe models rely on word length as a feature to discriminate between lexical categories since the word discriminability index positively correlates with the number of phonemes in a word. furthermore, our representational consistency analysis have shown that awe models exhibit individual differences in their representational profiles, with the contrastive encoders being the most inconsistent across different random initializations. even though we focused on acoustic word embeddings in this paper, our analytic methodology can also be employed for the interpretability of selfsupervised speech representation models such as contrastive predictive coding @xcite and wav2vec @xcite . also, the emergent representations of sublexical units such phonemes and syllables in speech neural networks can be analyzed using the our proposed methodology in this paper. 1 10 100 epoch -log scale 0 0.2 0.4 0.6 0.8 map (a) recurrent 1 10 100 epoch -log scale 0 0.2 0.4 0.6 0.8 map (b) convolutional contrastive siamese encoder (cse) phonologically-guided encoder (pge) correspondence autoencoder (cae) autoencoder (ae) rie ae cae pge cse 0.0 0.2 0.4 0.6 0.8 map rnn cnn (c) best epoch figure 7: evaluation on the same-different acoustic word discrimination task quantified by the word discrimination task and the map metric: learning curves of 100 training epochs for (a) the recurrent encoder and (b) convolutional encoders. (c) map of the best epoch.